Theoretically, yes, you could have high levels of free test and still not be able to put on muscle easily. You could also have even higher levels of cortisol whih would have a catabolic effect equal to or greater than the anabolic effect of high free test. You could also have naturally low levels of muscle androgen receptor which would limit the amount of growth. You could also have defective receptors that don't respond as well (or even not at all -- a syndrome called complete androgen resistence). Finally, testosterone is not the only determinant of muscle growth.
Seth