Why does my contact list now have every email address I have ever sent to listed in it?

I used to have my contact list nice and organized. Set up groups and all was good.
But for a couple updates now my contact list has contained every email address I have ever sent to with my gmail account. This is a significant list now, and useless, as they were one off contacts or whatever.

How do I fix this?