I ferment in a 100-year old basement. Mildew, spiders, and dog hair galore.
Point is - the location of your fermentor is relatively unimportant. The beer should be protected by the fermentor, unless you're constantly opening the lid/stopper OR your fermentor has a leak (bucket/lid seal, carboy stopper, empty airlock).
The most probable is that you're letting bugs in when you sample in the freezer OR your airlock is empty.
The liquid inside the airlock can be sucked inside the fermentor during fermentation, if you burp the lid of a bucket, or if you move the fermentor. Make sure your airlocks are full before and after fermentation and periodically during lagering.
Don't open the lid/stopper all the time! If must sample IN the fridge, wipe down the surrounding area and lid, spray with Star-San, quickly sample and cap, then wipe the Star-San off. Or just pull out the fermentor when sampling.