Unfortunately, yes. Once you've seen enough of them or played with the chatbots enough yourself, you can pick out their very specific phrases and syntax pretty easily.
I actually recommend people play with some of them for that reason. You'll get scammers who use AI to craft their messages...