I think you could pull it off once summer hits in full force. Christopher Kim's at 20th and M had a great selection when I was there last summer. Be sure to ask for help before trying them on--straw hats are very finicky about where you can and cannot touch them.
I know you're thinking that Panama hats are for golfing and old men, but you are selling yourself short. I've seen you rock different pieces which less sturdy souls would never combine. You are no mere clotheshorse--you are a clotheslion.
Free it from its stereotypical context and see what happens.