bruh who cares. Like don't people call girls 'he' sometimes? I don't think the situation should change just because you want to be called neither.
If it personally bothers someone and you do it just because, then I suppose could step into less trivial grounds but it's no different from calling...