"Sugar" generally refers to the high-GI (glycemic index) varities. These may not so much affect your muscles, but they will impair fat loss by entering the blood stream to rapidly, and thereby causing huge insulin surges.
Too much insulin all at once = fat gain or no fat loss