In response, men say they have turned their attention to less attractive and more willing women