There is the off-chance that you are referring to
uwimapd, an implementation for an IMAP server, when you say "IMAP". In that case you need to make sure none of your mboxes becomes larger than 2GB, that is all emails in the same folder, not counting sub-folders.
If you are not referring to
uwimapd, I believe your problem is the performance of your computer and internet uplink, not IMAP. Therefore:
Only keep emails no older than 3 months and those belonging to ongoing projects. Put the rest in an archive folder. Create the same folder structure under archive as you have under INBOX. That makes it easy to find stuff in archive, if you can't find it in INBOX.
IMAP, the clear text protocol your mail user agent employs to retrieve and manage the emails stored on the (remote) mail server, becomes problematic when it has to handle thousands of emails at once. The key aspect is "has to handle". The overall amount and size of emails is not a problem for IMAP, per se. The problem arises if it has to actualy perform work on a lot of emails.
This happens, for instance, if thousands of emails have their content and/or status (flagged, un-/read, priority, timestamps, ...) changed between two syncs. As you use
offlineimap this may indeed happen, depending on your use case. However, in that case the only thing that will help you are more frequent syncs.