When you P.E you are exercising your penis just as you would your muscles if you went to the gym. When you go to the gym, you increase your muscle size and it shows in your build, however, if you stop exercising the muscle will eventually return to fat and be lost. This is not an overnight thing.
Your penis is a muscle and when exercised it will grow stonger and get bigger. When you stop exercising, you may lose some gains. Please do remember that sex is an exercise for your penis and as long as you are having and enjoying a good and active sex life, you should not experience any loss of your hard earned gains.