Do you think it's wrong?

not to let your spouse show you affection?? my husband really likes to caress me and kiss me like on my neck or my body or hold me . I usually just tell him in an angry voice to stop or push him off...sometimes I just let it happen but try to hint at the fact I don't want it with my body language or not doing anything in return. It seems like he gets saddened by this,but he knows I love him. Once he slept in another room when he tried to be affectionate one night in bed and I yelled at him and moved away.