It's a good sized house and cats are small (particularly mine, the biggest one weighs around 12 pounds and the smallest weighs less than 6). Besides, they don't like each other very much so you rarely have more than one or two awake in a room for very long. Right now four are sleeping in my bedroom (they stake out a corner of the bed) and one, who is my self appointed puppy dog, is here in my office sleeping under my desk. She follows me everywhere all day. The other two are downstairs somewhere.
Domestic cats are not wild animals, any more than domestic dogs or ferrets are.
Besides, at least two of my cats are ferals and they decided they'd had enough of that shit and wanted to come live with me so I'm not depriving anyone of anything. The one time one of my other cats made the mistake of sneaking outside she was scared shitless and crying to come back in within 10 seconds.