Technically, if they are just party members with no romantic or sexual interest/involvement between them (except for the actual romantic pair), it's not a harem. It might stray dangerously close, depending on execution, but it's not.
Even in badly written Jap harems, even though the guy is the...