Not totally save, because it could cause leftover configuration from oh-my-zsh.
The setup of oh-my-zsh adds some lines to your zsh startup files, like ~/.zshrc
The uninstall program you already looked for would remove these added lines.
The problem caused by the leftover configutation could make zsh show error messages at start, or stop running the shell startup scripts somewhere inbetween.
All that can be confusing to fix without knowing the scripts.
So, for now, I propose to keep the directory until you found and used the uninstall script.
If you end up running in errors at zsh start related to this, please show the error messages.
(Please clarify which folder to remove you talk about exactly)