Wash 'em every time, unless it's a Number One and I'm in a hurry. I'll take extra care if I spack out and piss all over my hands or something, but since I spend a generous amount of time with my hands down my pants, I figure things will turn out A-OK if I scrimp on the soap 'n' water now and again.
I admit to having a similar quirk to Silv, except it's mostly when I get home from work, not just from being outside. I'm only a supermarket cashier so my hands never turn out especially filthy, but it's like I can feel...I dunno, like, a thin film of workplace all over them, and I can't relax until they're fresh.
I also hate having sticky hands, like when you eat an orange and the juice gets on your fingers and goes kinda tacky. Hate it, so bad. Like, it would drive me to frantic, Lovecraftian madness if I wasn't near a bathroom or in the immediate vicinity of medicated wipes.
And my keyboard smells like hot plastic. I checked.