I can't drink milk, eat ice cream or whipped cream any more. However, I do cheese. I use to have those single serving 1oz cheese stick as a snack in my diet. I realized that the cheese was holding me back. Also, I am on a low-carb/keto diet, and I swear the cheese tasted sweet and I wanted more of it. SO, i put the cheese sticks in the freezer (can you freeze cheese???).
If you feel its holding you back, cut it out for 2-4 weeks and see if more fat/weight drops off.
As NYC said: try to eliminate it and see what you get... Milk, skim or otherwise, is inherently pretty high in sugars, and i avoid most dairy for that reason: i would rather get my carbs from things that have texture/crunch/greater mouth fun.
That having been said: does milk fit into your daily macro projections? if it does, enjoy it if you want it, if it doesnt, kick it the hell down the line.