I think it's because context quite literally changes the way we see things. Imagine you see a breathtaking woman, wind fluttering through her curly brown hair, a cute face, and a fit body. Then you see a child come up to her and say, "Grandma!" Suddenly, she's not so breathtaking. If I find out an attractive girl is a real [bleep], it magically turns her physical attractiveness down a couple notches.