Network Proximity Checker and major FPS drop with large GC collection

I have 450 objects that use Network Proximity Checker and the servers fps drops by 30 every time it updates. I find it strange that it seems to check every object with the proximity checker if there is a player nearby, rather than checking every player if there are any objects with the proximity checker.

The 450 objects is mainly a test, as my game is going to have a lot more than that. Is this meant only for very small scale games or am I doing something wrong?

I tried with 2,500 objects...

Network Proximity Checker and major FPS drop with large GC collection