Have you done IR yet (and I mean in a department running their own clinic/inpatient service)?
You only follow a small subset of patients longitudinally (mostly HCC, biliary strictures, permanent neph-tubes and dialysis access), the ones you do are known to your staff on a first-name basis. Most patients are a one or two-time contact (not different from an subspecialty ophthalmologist for example)
Sure, if patient contact is what you are looking for, there are other specialties that provide more of that than IR. (I didn't want to go into radiology because I wanted to be a 'real doctor' dealing with patients rather than sitting in a dark room all day looking at a screen. After 3 years in the clinical trenches I decided that I didn't care for patient care that much and that sitting in a dark room all day is a perk of radiology, not a drawback. Now, doing mostly IR I have found a field to split the difference and I am happy with it.)