Don't worry about ruling out the worst first... rule out the easy stuff and make your way down the list. Diet, lack of sleep, porn usage, testosterone levels, anxiety, pelvic floor, overworking during PE, etc etc etc.
It's not that you're being irrational. There are simpler ways to go about finding the cause.