From my lovely psychology notes,
photoreceptors fire at a faster rate than baseline when you see one color, once that receptor fires at a lower rate, it is inhibited and signals our brain to see the complementary color. When color receptor is excited or inhibited, we get a rebound effect when we stare at a neutral color. The same color receptor will go into the opposite state causing to one perceive the complementary color.
In your example, when you stare at the orange shirt, your color receptors fire past baseline firing (signaling orange).
But when you stare at a neutral surface, all those cells go into their opposite states and fire less than baseline possibly because pigments are partly bleached out by prolonged exposure to the stimulus causing you to see an afterimage of blue cyan on the white screen.
IDK IF THAT HELPED! I TOTALLY ERASED ALL EDUCATIONAL THINGS FROM MY BRAIN! My explanation seems too simple XD