So you are looked in the face by someone trying to speak to you, the response given to that person should be one of mutual respect and eye contact.
No other type of response says confidence then that of looking someone in the eye when speaking……What are you gonna do?