Not sure if this will make you feel any better, but I worry WAY more about my clinic (AKA...all) patients now when I'm not in the office than I ever did as a fellow. It's part of the business. The good news is that I have a lot of support (MA, Triage RN, RN Coordinator, Clinic Pharmacist, Research coordinator) to help me manage it all now.
A more positive way of thinking about things is that, for the vast majority of your patients, they are going to do well or poorly almost no matter what you do, or don't do. And most of the truly important abnormal lab/imaging findings you're going to come across are going to be clinically evident as well, so the patient is unlikely to let you ignore it.
The soul killing stuff is the Stage I ER+ breast cancer patient who demands labs every 3 months (despite just being on an AI) and then pummels you with questions about why her MCHC, % Eosinophils, MPV and Cl levels are 1 point out of range and what you're planning to do about her critically low BUN.