I think I talk way too much about gaming already to be considered having problems talking about it.
I think the reason people would have problems talking to other non-gaming adults about the subject would be that it's still regarded as a child's pasttime while stamp collection, fishing, gardening is something which have been around for a long time and you have something to show for it. It's not like a game will print out a proof of completeion. Besides there's all the gamers who cheat to complete a game. It's somewhat hard to cheat in gardening, or stamp collection for that matter. Accepted games among adults are usually something physically active which is why it's activity. The thought of games without the social interaction you get and the physical excersice seems strange. It seems like you're wasting your time, when in fact you're more active than the average person sitting in front of the television watching Jerry Lake or Ricky Springer or whatever they're called.