Thanks everyone for the info! I think for now I will simply weather seal the closet door which probably will eliminate 70% of the adjoining room's hot air leak. Next time I have a HVAC pro over I'll get an opinion on sealing the holes.