Sometime around adding Mars, Phobos and Deimos the game began to show a significant degradation of performance.
Turns out in Unity if you move a Collider2D (gives shape for collisions) that doesn't have a Rigidbody2D (enables physics) the physics system will redraw the collider. If you move a giant complex polygon collider like my surfaces are built of you end up with a very expensive game.
So I set about the arduous task of beating my head against every pain point that exists in Unity's Physics system. In the end I figured out the exact configuration that makes everything happy. I reorganized everything to be as efficient as possible and got back to testing.
The redraw was gone but the performance was identical. Turns out there's always some cost associated with moving colliders in a physics system. Good news was the rigidbodies (physics component) I'd added have a configurable bool simulated I set that to false for all worlds the player isn't on and viola.
In the above graph, orange is cost of physics. The chart shows the last 3000 frames on the x axis against the y axis representing the FPS costs associated with different elements on each frame. You can see I go from around 70fps max on left to around 300 on the right.
Now code in blue is the most expensive problem again. I'm happy and will move on.
I mentioned on discord about stars being the most expensive part of the game. All this, just to draw the background stars, haha oops. I'll be replacing them soon.