Unless you're lactose intolerant I don't see any problem with drinking milk in general. However getting skim milk as your only source of carbs wouldn't be the best option. While milk is considered low GI, it is actually pretty high on the Insulin index which is becoming just as recognized as the GI list. When the GI index was created it was assumed that glucose response would have a similar effect to insulin but they found out it doesn't always. Milk is one of the foods that was surprising to see that even though it had a low glucose response it actually boosted insulin very high.
At any rate moderation is the key to everything. Drinking a gallon of skim milk every day for the carbs just isn't the best idea. Where would you get fiber? Fiber comes from carbs mostly so you're not going to be getting much of it unless you get supplementation, even then natural fiber is still better.
Also the 3 grams of fat in oatmeal is nothing. You can't eat a "no fat" diet. Your body needs some fat to survive so 3 grams in oatmeal wouldn't kill you. It's been proven that a diet too low in fat will actually lower testosterone levels. I would never under any circumstance go below 10% fat intake. Also you have to make sure some of that fat is in the form of EFA's. Even if you get 15% fat in your diet, if it's all saturated animal fat then you're still not getting what you need.
At any rate I'd suggest getting oatmeal in the morning with some skim milk in it if you wish. Then maybe for lunch have a banana or apple for your carb. You could have only skim milk for half of your daily meals but try to get some real carbs in there somewhere. Oatmeal and fruit are both very cheap and good sources of carbs. Just try to eat the fruit in the morning so you burn some of your stored liver glycogen during the day. So long as liver glycogen isn't full(guaranteed not to happen in the morning after waking up)then there can't be any spillover from the fructose.
Just to keep up the list; rice, potatoes, oatmeal, fruit are all cheap carbs. Just make sure to stick to the long grain rice and opt for sweet potatoes over normal ones.