Best way to improve performance with a lot of pawns?

What's the best way to improve performance with a lot of pawns?

I am currently playing with quite a lot of mods, including some performance mods and I get pretty smooth frames with around 20-30 pawns and below but with an additional 10 pawns or so the performance does noticeably drop.

So now I am wondering if there is anything else that I can do to improve performance. As I do want to hit the hundred pawn mark at least and make a large colony.