Depends on what academic track you are on. As an academic attending, yes you will certainly have administrative duties. After that, it depends on your institution and how their faculty tracks. See for example:
http://medicine.osu.edu/faculty/tracks/Pages/index.aspx and
http://gumc.georgetown.edu/evp/facultyaffairs/facultytracksandapplicationforms/universitytracks and for a paper on the variation in academic faculty tracks:
http://journals.lww.com/academicmed...Career_Tracks_at_U_S__Medical_Schools.11.aspx
So it is conceivable (and I know PM&R attendings who do this...but they work efficiently) to spend 20 hours a week with direct patient care and 10 hours or so a week doing educational/administrative work and be at 180k. They key here is that they are generating beaucoup RVUs with the inpatient service but the resident is doing most of the work (so actual work time is closer to 80 hours as week). Obviously, though stress level can vary depending on academic track...as someone on the research track would have a lot of added pressure to publish.
Again, this is like the lower end, bare minimum type of things. I'd say most academic inpatient PM&R attendings also have an outpatient practice such that they buttress their inpatient RVUs with RVUs from an outpatient practice. If one wants to do private practice general inpatient rehab he can easily be at 200k+ with 40hrs a week of work (again, assuming efficiency). And clearly those who are fellowship trained in certain specialties can commands much higher salaries.
In the end, as DocE mentioned there is a ceiling for physiatrist pay and you're never going to top the Forbes list as just a physiatrist. But the opportunity to be well within the upper middle class of income and work 40 relatively low stress hours and to have very rewarding patient experiences is appealing to a lot of people, but not necessarily those who want to maximize their bank and prestige within medicine.