Maybe your doing too many working sets. I do 4 different exercises of 4 sets each for each muscle. (1 warm up set, then 3 sets of each till failure)...So no more than 16 sets per muscle. And I do a 2 day on 1 day off routine, and my recovery is spot on.
Also, not sure what your doing right now, but don't train more than 2 muscles in one day.
And finally I've noticed that when I'm on recovery speeds up a tad, but just listen to your body, if you feel good and strong then hit it a day early, if not take a day off......