I've got no problems talking with others about my habit. I retort with, would you rather me have a nasty crack habit and come into your homes to steal just to support that habit
Of course with some users, 'Evercrack' really has them by the balls, but I think it's more a matter of choice. You could build models, planes, trains, cars, you name it, how about fishing, boating, skiing, hiking, there are so many hobbies that you could talk about this one subject forever.
If you still feel uncomfortable discussing this subject with other non-gaming people then perhaps you havne't come to grips with this yourself. Perhaps some of the stigma that others label computer/console gamers with is the non-social part of this. Most gamers tend not to interact with others, even online play you are hidden behind an avatar, like we each have one on EAB
But myself I love talking with most people in person as well. I tend to go to as many LAN gaming events as possible, just for this social interaction. There you meet people from all areas and walks of life, they are not just loners and people that can't interact with others, well at least most are this way
I hope you one day see this as a hobby/activity that you enjoy and can feel proud of and not feel ashamed to talk about it with others