Changes to the NIH Peer Review Process

[SurfingDoctor]

Changes Coming to Applications and Peer Review in January 2025 – NIH Extramural Nexus

nexus.od.nih.gov

Conceptually, I think it's hard to know how much this impact will have but it's interesting how much the scoring system changed. 5 review criteria to only 2 on the same 1-9 scale. I wonder how this with impact score compression and standard deviation in overall impact scores. Essentially Significant and Innovation are one 1-9 scores and Approach is the other 1-9. The rest are administrative click boxes.

---

Members don't see this ad.

 

[Fencer]

It is also happening for fellowship applications. The reality is that for RPGs, "approach" often drives the score... For fellowships, they are decreasing the value of the mentor and institution. There will be a series of NIH webinars in summer and early fall. Changes will be applicable until January 2025.

 

[SurfingDoctor]

It is also happening for fellowship applications. The reality is that for RPGs, "approach" often drives the score... For fellowships, they are decreasing the value of the mentor and institution. There will be a series of NIH webinars in summer and early fall. Changes will be applicable until January 2025.

Approach has always been a driving score for Ks. That and persistence to some degree.

That being said, it’s probably good to de-emphasize the mentor. There’s good mentors out there, but there are also crappy mentors… and a candidates success is more dependent on them than their good (or crappy) mentor.

 

[tr]

Changes Coming to Applications and Peer Review in January 2025 – NIH Extramural Nexus

nexus.od.nih.gov

Conceptually, I think it's hard to know how much this impact will have but it's interesting how much the scoring system changed. 5 review criteria to only 2 on the same 1-9 scale. I wonder how this with impact score compression and standard deviation in overall impact scores. Essentially Significant and Innovation are one 1-9 scores and Approach is the other 1-9. The rest are administrative click boxes.

I don't have as much reviewing experience as you both but my impression is that Investigator and Environment are functional check-boxes already. Either they are good/acceptable (most of the time) and get a 1, 2, or 3, which is not score-driving, or they are unacceptable, in which case the whole application is unlikely to be scored.

I kind of like the idea of merging Significance and Innovation as well since there are meritorious projects that could be marginal on Innovation but big on Significance.

 

[tr]

Approach has always been a driving score for Ks. That and persistence to some degree.

That being said, it’s probably good to de-emphasize the mentor. There’s good mentors out there, but there are also crappy mentors… and a candidates success is more dependent on them than their good (or crappy) mentor.

Really? My impression is that mentor is actually a pretty big factor in long-term candidate success.

 

[SurfingDoctor]

Really? My impression is that mentor is actually a pretty big factor in long-term candidate success.

On paper? Yes. In reality? Maybe.

I think I met with my K-mentor a total of 5 times over 5 years and would sign on their behalf for progress reports. The person was nice, but it wasn't really that productive of a relationship in isolation (though on paper for the mentor and from the NIH's standpoint, it was rather productive... which was a lie, but whatever). I had other mentors who either 1) provided actually monetary support or 2) provided guidance. So in that regard, I think it is kinda more dependent on the candidate and for the candidate to find the right people and likewise to push themselves to complete the research. Often, at least for Ks, the mentor is usually someone who 1) has NIH funds available and 2) has had previous trainees. They may score that section of the K 1-2 and generally speaking, its pretty hard not to meet the basic criteria. But then you look at the candidate and realize that they've 1) had no career guidance or 2) that they have been in the same lab for like 6 years, making the mentoring on paper look questionable. And then in the Approach, its usually pretty easy to tell who discussed it with the mentor and who didn't.

This is all to say, yes, of course one needs mentorship, but the candidate statement/career development plan and approach can tell you more than you need to know, because at the end of the day, it's the candidate who determines their own success, and not usually the mentor. I've mentored a couple of trainees and junior faculty and its generally easy to figure out who will be successful and who won't because of the effort they put in and the approach they have to science and their own career development.

 

[SurfingDoctor]

I don't have as much reviewing experience as you both but my impression is that Investigator and Environment are functional check-boxes already. Either they are good/acceptable (most of the time) and get a 1, 2, or 3, which is not score-driving, or they are unacceptable, in which case the whole application is unlikely to be scored.

I kind of like the idea of merging Significance and Innovation as well since there are meritorious projects that could be marginal on Innovation but big on Significance.

Yeah, the Environment is usually a checkbox these days, though I have dinged people on rare occasions, especially if large portions of the research is taking place outside the institution. Then it's less environment per se, but more logistics, ie can they organize this properly?. And the Investigator is also kinda a check box, but one needs the right amount of Co-Is to make sure the research can be completed without it being too burdensome. There are definitely times too where someone will slap another Co-I on and you're like "wait, what is this person doing again?". Generally speaking though, these aren't major score drivers.

But I do generally think the changes are good in theory. The overall impact score usually is driven by the Approach>Significance>Innovation anyway, with really more point deduction for the other categories that should be more checkboxes, so it makes sense. And I don't think most grants are technology innovative and even from an theory standpoint, usually more incremental so it makes sense it's combined with significance. I'll be interesting to see if it impacts score compression, which is always a frustration for reviewers and applicants.

 

[Neuronix]

From someone who spends too much time on study sections and has heard a lot of buzz about this...

Significance and approach are what really drives most impact scores. Put simply, is this something worth doing, and are the investigators approaching the problem in a rigorous and appropriate fashion?

Innovation can sort of drive grants, but generally innovation is tied to significance. For some grant mechanisms (e.g. translation), it's not scored as we would consider innovation anyway.

It seems that institution and investigator can only hurt you. These are the biggest things that seemed to need removal. The NIH doesn't want all their money going to just the biggest name institutions. Simply put, I believe they're going to change things to: can tge environment perform the research? Ok check. That should be a "yes/no" or administrative review. Is the investigator qualified? Ok check. Same thing, "yes/no" or administrative review.

Analyses have shown that throwing in institution and investigator as review criteria just biases reviewers towards big name places and senior investigators (who also tend to lack diversity). This was not a good thing.