Well only your true self can answer that question...
It depends... I don't know your physiology, do you have a tendency to have sexual addiction... is this a way to keep you out of trouble?
Are you good to your wife, and treat her with respect and love?
Does this happen regularly?
The right or wrong question all comes down to context and intention...