Everyone's different when it comes to lactose
Different people process different nutrients differently.
Some people are lactose intolerant, some people handle it well. Some etyhnic groups tend more towards lactose intolerance, but it's an individual thing. I have two kids and one is chubby and the other one's realy thin, they both eat about the same diet, the doctor said the chubby one can't digest lactose, so there's an example of how people tolerate lactose differntly.
You have to get to know your body and how it reacts to different foods. It won't hurt to cut out dairy, esp w your bodyfat level. It's also good to vary your diet. If you eat the same thing everyday you're body gets accustomed to it and you get in a sort of rut. My advice, would be to (carefuly) vary your diet and you should be able to see in a week or so if you're progressing in the right direction.
Try cutting out dairy for a week (or adding more fiber for example), and then look in the mirror about a week or a week and a half later and you should see subtle differeces.
I think one of the best thing you could do for your diet is to add more fiber.
Good luck.