I am switching businesses, and my new one will have large numbers of contacts that I will speak with infrequently, but I'll need to remember our previous conversations etc.
there are a variety of CRM packages out there - for different budgets. Think database - you can build your own on Access for the cost of the software if you're smart enough to write a couple MACROS etc.
goldmine has always been pretty popular. there's some that are industry specific and those tend to be pretty good.
when i had my shop doing loans, I convinced the higher ups to flip the bill for a custom CRM system that was designed by a company that specialized in that (for our industry). was pretty damn good