When you dress well with a little flourish and take pleasure in it, most others will pick up on both and take pleasure themselves. This, for me, is part of the pleasure of dressing well. Its positive spirit is infectious.
Unfortunately, the occasional grouch will needle you. There are any number of reasons that some people behave like this, but these reasons have a common theme: they let you know that said grouch's opinions are irrelevant and held only by him or her.
I have never been able to draw, or to paint, or even to sketch. A blank canvas is, to me, an invitation for disaster. But I can, and choose to, make my clothing a visual, rhetorical, communicative statement when I leave for work, or for happy hour, or for a house party. (Sure, it's pastiche, but we live in postmodern times.) A perfectly coordinated and assembled suit is my stab at a personal Mondrian. (Boots with a suit is a velvet Elvis.)
Expand your comfort zone. Dressing well--whether at work or at play--will be its own reward.