Do you have any insulation on the exterior wall of the closet?
How is the wall finished?