You can try adjusting it in your head. If you find the voice is too loud, you can try consciously changing it.
I do this all the time. When I stumble upon a dialogue, if I don't like the voice my mind has come up with, I would reread and change the pitch until it sounds right.
If it feels like too much of a hassle, then I'll just keep on reading regardless of what voice I'm hearing (in that case, just getting the general sense of female and male voices is enough). But in your case, if it's too painful, you can try pausing and changing it.
Another way is to try limiting your subvocalization, basically lowering down the voice in your head or getting rid of it entirely. This often associates with fast reading. When you don't have to sound each word in your head, you can read faster, so the theory goes.
There are many ways to do this. You know, things like listening to music while reading or chewing gum, but these are just the surface stuff. You can search up on the internet and dig deeper.
But be warned, this can make reading fiction less immersive. Doesn't necessarily mean it's going to be less enjoyable, but what you get is mostly an understanding rather than a full immersive experience.
