This is the same thing I told someone else on this subject.
I would say dial in your diet and get rid of the handles and see what you look like with out them. Getting rid of fat and gaining muscle are part of the ebb and flow of the sport. I have always had the problem area like you. I hold all of my fat in my upper thigh and lower trunk. I've never wanted to sacrifice muscle size for a few pounds. However, I really want to see what is under my cushion. The leaner I get the more muscular I look. Duh, right. But it is amazing how we hold ourselves back with the things we tell ourselves. (excuses). Even if you lose a little muscle dieting all the way down you can adjust your cutting diet( clean ) by increasing the proper calories for you, to gain back your muscle and then some. Keeping a clean bulk will allow you to monitor where you are more acurately and you will have lees fat to shed next time. But patience and a journal is key and adjusting as needed. You are your own best source of info when it comes to your body. If you're paying attention.