Honestly, I’m not even mad. I am autistic so I completely understand the urge to avoid all small talk at work.
As long as you are “politely” anti-social, and not giving people a really bad impression of us, I don’t think it is going to hurt anything.
My own fess: I sometimes pretend to be deaf so I don’t have to talk to people in public.