If you look at the type of stuff we do at our fellowship, it's very much in line with the tenets of family medicine. Our residency is heavily invested in the under-served population. Our sports program decided to focus on alternative sports. Specifically, things like skateboarding, surfing, motor sports, mountain biking, etc. If you take a look at historical skateboarding, for example, many of the athletes come from broken homes, and have never had a coach/trainer/or any other medical professional looking over their shoulder. This is the most under-served population in sports. If you cannot excel in family medicine, it may be difficult to compete with other applicants.
In my opinion, education should be fun. If you are not enjoying your training, you picked the wrong career or the wrong program.