Ask any medical physicist and they'll tell you that your average IR fellow receives on the order of 5 mSv over the course of one year, and attendings (at least at academic centers) significantly less than this.
To put this in perspective, a patient gets approximately 4-8 mSv with a single CT of the abdomen and pelvis, and we all get approximately 2-3 mSv a year just from normal background radiation (depending mostly on what altitude we live at and what the radon concentrations are in our geographic area). Full-time commercial airline pilots and crew probably receive about the same dose per year. An astronaut on the ISS probably receives around 150 mSv per year.
Sieverts, being a measure of equivalent dose, are our best hand-waving way of expressing the likelihood of developing cancer for a given radiation exposure. Even though it is our best guess, it doesn't take into account certain important factors - such as the fact that receiving 5 mSv over a year is likely much less detrimental than receiving it all at once, as the body has much more time to repair any damage. It also doesn't account for lead (dosimeters are worn outside lead shielding), or the fact that we really have no idea what the relationship is between radiation exposure and cancer at doses less than 50 mSv - which is, not coincidentally, the annual limit for exposure in radiation workers. We extrapolate a linear relationship from the data we do have (mostly taken from atomic bomb survivors at Hiroshima and Nagasaki) because it is the most conservative assumption, but it is more than likely that the human body handles low levels of radiation much more efficiently, given that we have been constantly bombarded by cosmic radiation since the earth formed.
Several comments above suggest that IR is a "young field," the implication being that we won't really know until years down the road. Well, fluoroscopy and angio are actually not very new, and if anything the technology, protections, and culture of dosage reduction mean that we probably won't see in our lifetimes the kind of exposures that old-school radiologists were routinely exposed to.
TLDR; it's probably not worth your time worrying about cancer as an IR doc. Cataracts are a different story. So are people who practice bad technique, i.e. not keeping your hands out of the beam.
P.S. that is a pretty typical comment in the OP from ortho on radiation. It is stunning to me that, outside of radiology and radiation oncology, people delivering significant amounts of radiation have little to no education in this stuff. Not to mention the people ordering 10 CT scans in the space of a 2-week admission....