There was an article about hand dryers in which it discussed safety of use with regards to hygiene.  They said based on the study, using an air dryer can further increase the bacteria on your hands as opposed to the traditional paper towel drying.  Bacteria, germs and viruses get trapped in the dust on the screens which then is blown directly on hands as it's being dried by the air.  Honestly I don't know the real deal with this and somehow it doesn't bother me much because I still prefer the old fashioned hand wiping and drying.  I just don't like the fuss of having to stand there infront the dryer and waiting for your hands to dry.  Paper towel drying on the other hand does the work instantly.

Having said that doesn't mean I never use hand dryers, if that's the only accessible means of hand drying and I have no choice then I just use it anyway than have dripping wet hands.  I hate the long time standing infront of the dryer to totally wipe out the wetness but thanks to the newer dyson hand dryers, drying is quicker than the old models.  I like the dyson air blade hand dryer because it really scrapes the water out so fast, unlike those warm air dyers which take longer and sometimes even get too hot for the skin.  Dyson noted that their airblade hand dryers are certified as hygienic hand-dryers and energy efficient.  

Air drying or paper towel drying, both can be good or bad.  Who really knows?  I believe doing what feels most comfortable for you is the best thing to do.