Need a way to delete colors not used in the design from the color palate

The color palate has four choices, Select, Design, Favorites and Library.

I could be wrong, but it seems as though once a color is added to the Design, even if that color is later removed, it never disappears from the color palate. This creates an ever increasing number of colors in the Design section. There should be an option for the program to scan for unused colors (the way it can scan for unused images) and then automatically remove them from the Design section of the color palate.