From three to five is optimal. Its not so much about exact numbers as it is roles. Aim to have 2-3 different combat styles or roles (including the MC), at least one character to stimulate the MCs protective instinct (and maybe trigger a Cursed Powerup if they get kidnapped), and one character with a nose for trouble (or a pile of allowance from Grandpa BigBillions, or just strange luck) to help the MC find plot events (bandits, underground auctions, connections/conflict with the ruling elite, etc). An important consideration is how the harem acts when MC is not around; chapters where two or three members split off without MC or the rest of the harem can be excellent opportunities for character development (Does the MC need to fast inside a mythril dodecahedron for three weeks to purify his ki? Perfect timing for some side chapter character arcs). Keep in mind that if the goal of a harem setup is for the reader to imagine themself as the MC, the MC should have the LEAST developed personality in the story so as to not conflict with a reader's personality.