I did a year and a half in IM before switching to Family Medicine. I can mention some differences I found between the two, but of course every training program is different.
In internal medicine, I didn't really have a concept of public health. I spent most of my time admitting patients and getting them well enough to discharge them. I felt my main responsibility was to diagnose and treat illness period. Concepts of patient education, preventive medicine, outpatient continuity clinic all pretty much got lip service where I trained.
Also, The patients in IM generally had more pressing medical problems than what I usually see now in Family Medicine. This made the job more rewarding at times, but also more stressful.
I think in Family Medicine, if you think your job is strictly to diagnose and treat urgent medical issues and get everyone well, then you will be very frustrated. When you write an order in the hospital it (usually) gets done. In family med, where the focus will be on outpatient, you pretty much count on a certain percentage of patients not taking their meds, getting tests done, etc. So you have to look at yourself primarily as an educator, as well as someone who diagnoses and treats acute conditions. Patient interaction in Family medicine is usually more fun, since the people are healthier and (usually) don't smell like bedpans and bedside urinals.
I think if you like Public Health, you should go into FM, or least be very discriminatory about which IM or IM/PEDS programs you look at. The OB/GYN department is a fun place to visit as a FM resident, and the Peds exposure generally is a healthy population in FM, so I wouldn't let those two aspects scare you off. Most of your patients will be adults.
Hope some of this helped.