In regards to optimal follow-up:
As a bright-eyed resident my goal is to follow all my cases treated for curative intent for at least 5 years. Even if I was 'just' the adjuvant breast RT or 'just' the neoadjuvant chemoRT for rectal cancer. It can be every 3 months or every 6 months or yearly, but I don't plan to discharge them from my practice. All definitive cases (anal, H&N, esophagus, lung, etc.) I will plan to follow religiously for 5 years or until metastatic progression where they are in-touch with med-onc.
Same for SRS for metastatic disease patients.
Palliative RT patients can get a one-time visit at 3 months and then back to med-onc.
Whether all of these lofty goals can be achieved in a busy clinical practice is to be determined. I agree that the first people to get removed from the follow-up schedule (assuming a generalist practice) will be the cases where the patient had surgery (breast, rectal, pancreas, gastric, etc.) and is having routine follow-up with their surgeon/med-onc. If there's at least 2 other physicians evaluating a patient I'm OK taking a step back.