And we use the framework mainly for the ORM, API endpoints via DRF, simple admin UIs, and Celery tasks.

The ORM is, apart from composite keys (sucks if you have external data to import, moot if you don't), very good IMO. The database migration functionality alone is worth it.

We also use the multi-site support.

I agree that adding things on can be a pain if you have to deal with the internals of the ORM or views, though. Some of that code is pretty nasty, although improved in the later versions.