I read subspecialty neuro and get a handful of phone calls or the occasional pop-in visit when I'm at a hospital. In-person consults were much more frequent pre-Covid. On the outpatient side, I may not end up talking to a referrer at all if I'm just pounding through degen spines.
Most interactions are positive. Most of the referring clinicians are grateful you're giving your expert advice and in many cases driving their care. Some of the higher level referrers like neurosurgery, ENT and neurology are pretty good looking at imaging themselves and may catch something you missed.
Sometimes they ask for an unjustified stat read. If there's time I do it. If i've got a stack of 15 traumas on the list, well tough **** unless they come up with a better reason. It's like anything else; I don't call them and tell them how to do their job.
Being accused of being unskilled: that's a pretty bold thing which could have a lot of repercussions. That being said, never seen it. People miss stuff or mis-dictate stuff all the time. Most is pretty meaningless big picture.