It could be your body temperature and also your circulation. What also helps is of course getting a good night's rest, and staying away from stimulants, both legal and illegal. All of these things I just mentioned will turtle up your balls (and cock) no matter how hot the day is. Try exercising, that will help you out in numerous ways, not only with testosterone but with getting your heart pumping, improving circulation, and increasing your body temperature.
You want to know something cool, since I started doing PE my balls have been hanging lower ( my flaccid as well, I started 1 month ago) I catch myself looking at my package with my closet mirror, or every time I take a piss. Both my cock and balls have been hanging lower, I can't remember the last time they have both turtled.
If you have access to a jacuzzi or a steam, you should go! I can't wait to go to my gym today, especially after doing some PE. Hope this helps you out some!