Pass/ Fail Step 1

[sargon2123]

But then it comes down to imagine if you were 1 point away from the next quintile. 239 to 240 might not be a big difference now but if those are in two separate quintiles that makes all the difference

Imagine missing 200 by a point.

You have to draw the line somewhere. Maybe all this hair splitting isn’t necessary, but you need a way to differentiate applicants

---

Members don't see this ad.

 

[Chibucks15]

Good test taker does not exist. There are people who know content and those who do not.

Eh that's not true. There's a lot of situations where someone reads a question stem differently than others. Some people are able to get into the heads of question writers better. Obviously knowing material helps, but you have to figure out the goal of the question too

 

[Ho0v-man]

Good test taker does not exist. There are people who know content and those who do not.

I’m a pretty crappy test taker. You have to compensate by knowing the content better. Although running through qbanks and listening to boards and beyond did get me better at it.

 

[chlorocruorin]

Looking through your posts, it looks like you’re a premed and were fortunate gain acceptance to several MD schools with kind of crappy stats. Maybe URM, low SES, ESL, lucky state, I don’t know and I don’t care. The point is, you’re gonna be a doctor. Congrats!!

But from these posts it seems like your insecure about being a low stat applicant and projecting them onto this discussion. It took me two tries to break 500 in the mcat and I did it just barely. I had similar insecurities. But while being a good test taker is super important for the mcat, it’ll never save you on step if you don’t know your sh**. In fact, a lot of the complaints about this transition to pass/fail are bc it disproportionately benefits high mcat scorers (which examines test taking ability and some basic sciences) vs high step scorers (an examination of the fundamental sciences related to being a doctor, a knowledge exam). One of the lowest performing students in my class probably had the highest mcat score because test taking skill was carrying him almost all the way.

You can do well on these exams even if you’ve been mediocre up until now. It’s a whole different ball game. But you gotta put in the work.

I know you have no I’ll will but Please do not attribute my success to fortune. It was hard work and I now attend a top 25 school. My mcat was the only low point on my app and my interviewers said that. Also, please do not make assumptions on how I feel just because of my stance on this. You don’t know who I am, what my race is, and what what my SES so don’t guess just for the sole purpose to justify why I got in.

 

[Chibucks15]

I know you have no I’ll will but Please do not attribute my success to fortune. It was hard work and I now attend a top 25 school. My mcat was the only low point on my app and my interviewers said that. Also, please do not make assumptions on how I feel just because of my stance on this. You don’t know who I am, what my race is, and what what my SES so don’t guess just for the sole purpose to justify why I got in. I could easily say for many posters in favor on keeping step 1 the way it is, is you may be “insecure” about your DO degree compared to an MD. I would be wrong for that.

Nice backhanded jab at DOs. Classic! You damn well know anything going P/F would exponentially help t25 students more than anybody else so of course you're all for it. We all work hard, or else nobody would get into or last in medical school. Eventually you either have to perform above and beyond your peers, or adjust what goals you realistically can achieve. Regardless of your race, SES, whatever other measure you wanna throw up. Welcome to the real world. And this is from a lowly average DO student with an average step score and no plans of anything prestigious at all so you know I'm about as neutral as it gets.

Regardless, this nonsense won't pass, and if it does, it won't change anything other than passing the importance onto something else. Instead of large scale overhaul of the system, why not tweak it? I don't understand why people always want to blow up anything they don't agree with, and it happens everywhere in life

 

[chlorocruorin]

Nice backhanded jab at DOs. Classic! You damn well know anything going P/F would exponentially help t25 students more than anybody else so of course you're all for it. We all work hard, or else nobody would get into or last in medical school. Eventually you either have to perform above and beyond your peers, or adjust what goals you realistically can achieve. Regardless of your race, SES, whatever other measure you wanna throw up. Welcome to the real world. And this is from a lowly average DO student with an average step score and no plans of anything prestigious at all so you know I'm about as neutral as it gets.

Regardless, this nonsense won't pass, and if it does, it won't change anything other than passing the importance onto something else. Instead of large scale overhaul of the system, why not tweak it? I don't understand why people always want to blow up anything they don't agree with, and it happens everywhere in life

It wasn’t a backhanded jab? I was saying don’t attribute my stance to something you know nothing about of me. I will edit that part out if you felt like I attacked you

I edited it out. That was the last think I wanted to portray please leave it at that

 

[Chibucks15]

It wasn’t a backhanded jab? I was saying don’t attribute my stance to something you know nothing about of me. I will edit that part out if you felt like I attacked you

I edited it out. That was the last think I wanted to portray please leave it at that

I could care less about anybody on this site 'attacking me'. I don't really base anything in my life off what random people on the internet say. The rest of the point still stands though.

 

[Ho0v-man]

I know you have no I’ll will but Please do not attribute my success to fortune. It was hard work and I now attend a top 25 school. My mcat was the only low point on my app and my interviewers said that. Also, please do not make assumptions on how I feel just because of my stance on this. You don’t know who I am, what my race is, and what what my SES so don’t guess just for the sole purpose to justify why I got in.

I don’t care how you got in. We’re in different stages of training. Your acceptance doesn’t affect me and I sincerely hope you make the most of it. For what it’s worth, I doubt I could have gotten into a top 25 school regardless of circumstance so good for you. I said before the mcat isn’t all that reflective barring extreme ends of the bell curve, so I’m sure you’ll do great if you put in the work like anybody else.

I guess both of our stances at the end of the day are pretty self-serving. I’m trying to do well to overcome my lack of prestige. You’re trying maximize the effects of your schools prestige by eliminating the necessity of doing well. Bend the argument however you want, but I’m sure most people wouldn’t have to think too hard about which system is more likely to make “good doctors.”

 

[Goro]

Good test taker does not exist. There are people who know content and those who do not.

Au contraire, young colleague...standardized test taking is as much a skill as playing the flute or throwing a curveball. If you think about it, standardized tests are exercises in pattern recognition.

 

[7331poas]

Au contraire, young colleague...standardized test taking is as much a skill as playing the flute or throwing a curveball. If you think about it, standardized tests are exercises in pattern recognition.

tosh said it best I believe

 

[Neopolymath]

Au contraire, young colleague...standardized test taking is as much a skill as playing the flute or throwing a curveball. If you think about it, standardized tests are exercises in pattern recognition.

I understand what you are getting at and find it true with some forms of testing like the SAT (although I would still call it G loaded and thus intelligence), but Step 1 rewards knowledge. A lot of questions I know I got the correct answer from simply knowing the other answers could not be correct due to my knowledge. You can call that test taking, but I believe that's a stretch.

 

[sargon2123]

I don’t care how you got in. We’re in different stages of training. Your acceptance doesn’t affect me and I sincerely hope you make the most of it. For what it’s worth, I doubt I could have gotten into a top 25 school regardless of circumstance so good for you. I said before the mcat isn’t all that reflective barring extreme ends of the bell curve, so I’m sure you’ll do great if you put in the work like anybody else.

I guess both of our stances at the end of the day are pretty self-serving. I’m trying to do well to overcome my lack of prestige. You’re trying maximize the effects of your schools prestige by eliminating the necessity of doing well. Bend the argument however you want, but I’m sure most people wouldn’t have to think too hard about which system is more likely to make “good doctors.”

You’re less self serving because you’re more in favor of increasing opportunities for all. The other poster, who we’ve just learned is at a top 25 school, wants to decrease opportunity, and now has lost credibility with the disclosure of his school rank

 

[AnatomyGrey12]

and now has lost credibility with the disclosure of his school rank

They lost credibility when it was discovered they were a pre-med.

 

[deleted600623]

Quartiles/quintiles honestly seem like a great solution and a great way to meet both sides in the middle

I completely agree. Make both Step 1 and Step 2 CK scoring quartiles and for the love of God, don’t touch Step 2 CS. No one has any issues with CS being pass/fail.

 

[olabrador23]

Au contraire, young colleague...standardized test taking is as much a skill as playing the flute or throwing a curveball. If you think about it, standardized tests are exercises in pattern recognition.

are doctor's medical diagnoses exercises in pattern recognition???

 

[Goro]

are doctor's medical diagnoses exercises in pattern recognition???

In Derm they are!
Diagnoses are much more flow charting.

 

[Goro]

I understand what you are getting at and find it true with some forms of testing like the SAT (although I would still call it G loaded and thus intelligence), but Step 1 rewards knowledge. A lot of questions I know I got the correct answer from simply knowing the other answers could not be correct due to my knowledge. You can call that test taking, but I believe that's a stretch.

That is indeed a test taking skill! It's a Dx of exclusion. Not many people master this. In fact, they seem to go for the thing that they never heard of before when they have four knowns.

 

[Ho0v-man]

are doctor's medical diagnoses exercises in pattern recognition???

Yes. Why do you think doctors get stumped when common things present atypically? It doesn’t fit the pattern you’re used to.

 

[olabrador23]

In Derm they are!
Diagnoses are much more flow charting.

lesson: medicine is not about standardized testing!

So when Step goes P/F, I guess it is only Derm people got screwed... it totally makes sense as they make too easy money anyway

Yes. Why do you think doctors get stumped when common things present atypically? It doesn’t fit the pattern you’re used to.

Idk man, ask Goro

 

[kb1900]

Re test taking skills, there is merit to both sides but imo these are not the bulk of differentiation nor the limiting factor for anybody up to a certain score.

1) yes, some people are faster at pattern recognition and picking up clues. But in the context of doing 2700+ uworld questions + 4-10 NBMEs, I think that this is not really as important as the sheer number of questions done over 2 years of medical school. I have yet to see a report of someone who did zanki and Kaplan, rx and uworld and all nbmes and scored below 250.

2) “some people are just generally better at getting into the head of test takers”. I think again this is true to an extent but really is more about how critical people are willing to be when they reflect on each question. Some people do this innately as they practice and some people have to actively make it part of their system. The vast majority of topics only get presented in a handful of ways each and its up to the test taker to be reflective and critical in what they commit to memory.

3) overall fund of knowledge. This is where most stratification happens imo. The volume and depth of the first 2 years is absolutely a limiting factor for most people. to suggest otherwise is absurd when discussing step 1. the exam can not ea be passed based on principles alone and those principles still require excruciating detail. your fund of knowledge is basically the baseline on which you build information - a higher baseline lets you get more out of each question you do compared to someone who doesnt even know what x is before getting a question on x.


There are a plethora of write ups at this point of people with average to mediocre mcats getting incredible step 1 scores. The common thread in all of them is that at the end of the day you have to constantly reflect on your weaknesses with regards to how you study and be willing to change if you want your results to change.


If test taking skills is limiting - and no doubt they are for some people - then like any other skill, improvement comes with proper practice. its my personal opinion and what i tell to students i tutor that in the realm of academic knowledge and testing, you can always push yourself farther along the curve if you put in more work and time than the average scorer in an more self-critical way.

 

[deleted938090]

Doubt MC strategy matters if one knows the answer before reading the choices. Why don't they have an exam with select all choices or short response?

 

[Goro]

lesson: medicine is not about standardized testing!

So when Step goes P/F, I guess it is only Derm people got screwed... it totally makes sense as they make too easy money anyway




Idk man, ask Goro

"If you hear hoofbeats, don't think zebras" is what my clinician colleagues always say.

 

[AnatomyGrey12]

are doctor's medical diagnoses exercises in pattern recognition???

Honestly? A lot of them are actually. Many disease states are very predictable. However yes you have to have the knowledge for when the pattern doesn't fit.

"If you hear hoofbeats, don't think zebras" is what my clinician colleagues always say.

Although we diagnosed 3 zebras within the last week on IM, so yes that holds true but you still have to know the zebras and have them in the back of your mind.

 

[Goro]

Although we diagnosed 3 zebras within the last week on IM, so yes that holds true but you still have to know the zebras and have them in the back of your mind.

1000% true and that's what separates you from the mid levels.

 

[Matthew9Thirtyfive]

1000% true and that's what separates you from the mid levels.

That and the ability to think outside the box. I caught an SVT and cellulitis in a guy on our ship that the other medical personnel dismissed (we were all enlisted) because it didn’t present in a typical fashion. When things don’t quite fit the pattern, that extra knowledge and ability to put atypical things together to get a diagnosis can save lives or prevent delayed care.

 

[FindersFee5]

As much i like to discuss various hypothetical scenarios, we don't know the final decision. And apparently the attendings over in the specialty forums think the change is unlikely going to happen. So it's probably best to wait and see.

Also, from my understanding based on what experienced members/faculty said, it is apparently extremely unlikely that those who took Step 1 and got scores will get retroactively converted to pass/fail. That just gives an opportunity for NBME to get massively sued.

How does that open the NBME up to lawsuits?

Doubt MC strategy matters if one knows the answer before reading the choices. Why don't they have an exam with select all choices or short response?

Short response would be horrendous for a standardized test because different graders would give different scores, introducing even more randomness into the process.

Not sure if you've actually done any practice questions for step yet, but they're not like any other MC question you've done in your life. Many are 3rd or 4th order and paragraphs long, so it's not something you can scan quickly and immediately know what the answer is. Sure, some are that way. But the vast majority are more complicated and you need to get comfortable with working through, figuring out what the question is actually asking about, and then select your answer.

 

[Med Ed]

How does that open the NBME up to lawsuits?

Plaintiffs could argue (convincingly) that retroactively changing Step transcripts has deleterious professional consequences. These organizations are extremely risk averse.

 

[907914]

c) has yet to take step 1 and doesn't want to grind

I am merely an applicant and this is exactly why I want this to go through, because I am super lazy when it comes to standardized test taking and would rather spend time with my family and enjoying life then memorizing a million flash cards...clinicals are a different story in terms of time commitment, but preclinical? IDK I may just not know enough on the subject (as, again, I am merely an applicant), but rote memorization is not my thing.

But a preference is different than an outright refusal to even consider

Gonna agree with the opposition to this. While it is abhorrent that people look down on you for your choice of residency location, I think people are entitled to want to live ONLY in particular places. I will never consider the south because of hurricanes, bugs, and humidity, would never consider the desert because of heat...people are entitled to these things methinks.

Au contraire, young colleague...standardized test taking is as much a skill as playing the flute or throwing a curveball. If you think about it, standardized tests are exercises in pattern recognition.

+1

Edit: I am just a Premed and know nothing. Just wanted to comment on this thread as it is going nowhere anyway. My personal preference would be 8 increments of 12% and the bottom 4% is failure.

 

[AnatomyGrey12]

Gonna agree with the opposition to this. While it is abhorrent that people look down on you for your choice of residency location, I think people are entitled to want to live ONLY in particular places. I will never consider the south because of hurricanes, bugs, and humidity, would never consider the desert because of heat...people are entitled to these things methinks.

Sure they are entitled to it, but they aren't entitled to complaining about the system after they don't match with such a strategy. You can be picky with location when you're looking for an attending job.

I am merely an applicant and this is exactly why I want this to go through, because I am super lazy when it comes to standardized test taking and would rather spend time with my family and enjoying life then memorizing a million flash cards...clinicals are a different story in terms of time commitment, but preclinical? IDK I may just not know enough on the subject (as, again, I am merely an applicant), but rote memorization is not my thing.

Knowing the pre-clinical information is what separates you from the midlevels, and you absolutely have to memorize the foundational info if you ever want to be good at the clinical aspects. Honestly any pre-med opinion on this is worth about as much as the history I took from the high AF meth head the other day.

 

[907914]

Knowing the pre-clinical information is what separates you from the midlevels

Genuine question to the quoted because it doesn’t quite sit right with me and I am likely uninformed on the matter:

is knowing the preclinical information really what separates the physician from the mid-level or is it the thought process that really separates them? During clinicals and residency, you have to be aware of processes, concepts, what you are and should be looking and how to apply them - but you have all of the literature available to you, correct? My understanding is that what separates a fresh-out-of-residency attending from a 30-year-career-NP is not knowledge but thought process. Is my understanding incorrect?

 

[AnatomyGrey12]

Genuine question to the quoted because it doesn’t quite sit right with me and I am likely uninformed on the matter:

is knowing the preclinical information really what separates the physician from the mid-level or is it the thought process that really separates them? During clinicals and residency, you have to be aware of processes, concepts, what you are and should be looking and how to apply them - but you have all of the literature available to you, correct? My understanding is that what separates a fresh-out-of-residency attending from a 30-year-career-NP is not knowledge but thought process. Is my understanding incorrect?

1. You can't have the thought process without the foundational knowledge. You can't simply skip to the clinical part and residency and expect to be a good doctor.
2. Mid-levels have all the literature available to them too....
3. The foundational medical knowledge of a 30 year career NP pales in comparison to an intern, let alone someone who is out of residency. The knowledge gap is huge.

 

[libertyyne]

Genuine question to the quoted because it doesn’t quite sit right with me and I am likely uninformed on the matter:

is knowing the preclinical information really what separates the physician from the mid-level or is it the thought process that really separates them? During clinicals and residency, you have to be aware of processes, concepts, what you are and should be looking and how to apply them - but you have all of the literature available to you, correct? My understanding is that what separates a fresh-out-of-residency attending from a 30-year-career-NP is not knowledge but thought process. Is my understanding incorrect?

There are no shortcuts in life. If you don't have the foundational knowledge you are going to flounder on the wards and the shelves.
Bust your behind ,not for the stupid test ,but because there are aspects of that education that may be beneficial to your future patients. And to get into the habit of busting your behind.

It is easy to pick apart the students who worked hard during preclinicals and those who were p=MD crowd.

And foundational knowledge is key, I would barely be afloat if I had 60-70 points less on step 1.

 

[7331poas]

Edit: I am just a Premed and know nothing. Just wanted to comment on this thread as it is going nowhere anyway. My personal preference would be 8 increments of 12% and the bottom 4% is failure.

You should have led with this so I would know to stop reading.

 

[907914]

You should have led with this so I would know to stop reading.

I am merely an applicant

While I totally agree with your disinterest in Premed opinion, this was in fact my first line.

 

[TheIllusionist]

Let me summarize a few points of crystallization for this thread and provide an argument that I think many of us, who have been through the process, can buy

Points in favor of making STEP1 pass/fail
1. Minorities and low SES people do worse on STEP1
2. STEP1 is not a good measure of clinical ability
3. Making STEP1 pass/fail would incentivize using better markers for clinical ability

Points against making STEP1 pass/fail
1. It would remove the major meritocratic criteria present
2. This would in turn place even heavier emphasis on school prestige
3. It would force more research years, elongating med school in the era of more debt, this would in fact hurt low SES students, who happen to disproportionately be minorities, even more
4. There are ways the exam can be removed or the scoring modified such that less emphasis is placed on the exam, instead of going pass/fail. For example, a quintiles based approach (you are reported if you are in the top 20 the next 20 and so on and so forth). This would screw some people on the borderline of the quintiles but on balance mess with the least number of people will deemphasizing the current trend of OCPD style studying of trying to squeeze out every single last point. So top 20% would roughly be a mid to high 240s on STEP1, the median for even the most competitive specialties, barring a couple that have reached the 250 mark.
5. DO students will get screwed. USMLE is the only objective way for many to "prove" they can be as "good" as their MD counterparts.

Underrepresented Minorities have lower entrance requirements to begin with. The admit rates for a 3.60-3.79 GPA with a 30-32 on the MCAT for the following races
Blacks: 93.7%
Hispanics: 83.4%
Whites: 63%
Asians: 57.7%
All tables publicly available

FACTS

Data on U.S. medical school applicants, matriculants, enrollments, and graduates; as well as data on MD-PhD students, residency, and residency applicants.

www.aamc.org

The MCAT correlates with USMLE weakly to moderately (ranges from 0.4 for the whole thing in some studies to as high as 0.6, when looking at biological reasoning only). Admission to med school means that one has gone through undergraduate education. Undergraduate education completion means that one should be at a college reading level. STEP1, in terms of non medical terminology, can be argued to be written at below college reading level. Some of my classmates would have trouble understanding a tougher article in the Atlantic or WSJ than they would a STEP1 question. The exam is probably calibrated, with regard to non medical terminology, to be at something like a 10th grade reading level, hence why so many foreign grads, including those with poor English backgrounds, can still score in the upper echelons.

The STEP1 is a knowledge exam. It is a licensing test, not an aptitude test. The MCAT is an aptitude test but still with a heavy knowledge component, unlike the LSAT. This is why MENSA only accepted the latter in the past (it no longer accepts either). The rate limiting step the success on the STEP1 is how well you have memorized the information and the relevant connections. If one can meritocratically get into med school, one can do well on the STEP1.

The MCAT is based on years of accumulated knowledge with a big high level reading component. It, like the SAT, has some dependency on access to good prep materials and social class. But even that can be argued, given our internet age and access to a lot of good cheap resources. Also, unlike the SAT, people in tough situations can delay taking the MCAT for years, until they are prepared and "caught up" on the basic skills they need to brush up on. Even then, it can be shown the STEP1 deviates from the MCAT and that good STEP1 scores can be achieved even by very average MCAT scorers. The median MCATs of 1st year residents for following elite specialities

1. ENT: 33
2. Plastics: 32
3. Neurosurg: 32
4. Ortho: 32

Table B1: Test Scores and Experiences of First-Year Residents, by Specialty

Test Scores and Experiences of First-Year Residents, by Specialty

www.aamc.org

These specialties have around 85th percentile average STEP1 scores. Essentially, average level standardized test scorers in the past suddenly became around 1 SD above the mean. Again, this reflects the difference in the exams. STEP1 is largely a knowledge test. Knowledge tests rely more on crystallized intelligence than fluid intelligence. Crystallized intelligence is correlated with fluid intelligence, but it mostly a function of, barring the tiny tiny minority of med students with near photographic memories, consistent hours spent doing quality studying with quality resources. Most school googledrives have all of the resources. UFAP is like $500. People sign contracts that they aren't supposed to do full time jobs in med school, when they sign up for med school. There is enough time for people to use these resources to do really well.

In the end, the exam is really about hard work. What is this nonsense about no clinical correlation? If you don't understand basic pathophys of how action potential works, how can you truly understand an EKG? Do you want doctors to be pattern monkeys like many PAs and NPs? The essence of STEP1 is to test the building blocks that allow one to apply foundation knowledge to novel situations that don't fit neatly into algorithms. That being said, STEP1 is getting more and more clinically oriented. I had questions on first line treatments for various things. Scores are partially a function of that. Are you saying that isn't clinically relevant?

Here are some possible reforms
1. Make the exam more clinically relevant by using lingo that patients in inner city urban areas and rural areas use in the stem. This will actually show the broad variety of patients someone can work with.
2. Put some ultra basic Spanish in the question stem. It is essential to know a little bit in any ED nowadays, especially with the shoddy interpreter services that are often present at many non large academic centers.
3. Maybe some of nitty gritty biochem can really go away, but it is already trending that way. The questions mostly focus on clinical conditions.
4. Test underlying principles more. For example, contrast induced nephropathy can cause a pre renal azotemia type of BUN/creatinine. Instead of testing this by giving that type of ratio and then heaving most people who haven't come across that fact in a book get it wrong because they eliminate the choice, given the ratio is like 32 instead of like 12, as would be expected for an intrarenal phenomena like contrast induced nephropathy, test the underlying pathopahys. The reason this occurs is because contrast not only poisons the tubules directly, resulting in ATN, but also causes spasm of the prerenal vasculature, thus causing the prereanl azotemia type BUN/creatinine in some cases. Put in the stem the part about the spasm and ask what type of ratio would be expected. Force testing of foundational principles rather than specifically memorized cases for more esoteric things.
5. Stop testing two intuitively correct choices, where one was elucidated as the answer just by a recent trial. This makes these questions a 50/50 shot based on who guesses right or who happens to read latest trial data. The latter is what residency is more for. The purpose of STEP1 should be testing the foundation concepts. It is lazy question writing to just put in nit picky question that was only elucidated by a trial. Some of my STEP1 and STEP2 questions I could only find in journals. That is going too far.

My final recommendation would be a quintile based approach. That way people will still care about scores but not obsess over them. It will hurt borderline people that can be on balance argued to be better than the current arms race of turning everything into a one score depression, OCPD, and anxiety encouraging nightmare. Finally, shelf exams should be a thing for preclinical. That will allow better comparison of preclinical grades. NBME should release system based or subject based ones and standardize them well. They already sort of do this. But mandate it. That way the first two years will have a standardized component.

I can see why some elite schools are salivating at P/F. It provides excellent plausible deniability of, at least on the surface, commitment to the social justice vision of fairness. Yet the actual effects will be greater emphasis on school prestige, thus further entrenching and enhancing the advantage of high tier status for residency selection. This will favor their grads and then in turn favor them down the line because even more of their grads will dominate medical leadership. Some elite places have not only gotten rid of preclinical grades but also clinical ones. They want their students to be able to ride the prestige wave all the way.

All in all, I think a quintile based approach and mandating shelf exams for preclinical and clinical years would be a good approach along with reforming the exam to have more relevant questions, not only clinically but also more oriented towards foundation knowledge rather than memorization of esoteric information.

Potential biases and my background
Asian male, upper middle class NE/Cali childhood and adolescence
2330 SAT (99.5%+)
State school undergrad free instead of expensive T20 private schools with much better rank I got into. 3.7 GPA
36 MCAT (97th percentile)
Mid Tier Med School preclinical rank (about 50th percentile but thank god for P/F)
STEP1 247 (82nd percentile)
Honored all rotations and shelf exams were 80th-99th percentile- medicine, peds, neuro, psych were 99th with surgery my lowestv
STEP2CK 261 (86th percentile- based on averaging means and SDs over last 3 years, since 2019 mean and SD unavailable)

 

[sargon2123]

Let me summarize a few points of crystallization for this thread and provide an argument that I think many of us, who have been through the process, can buy

Points in favor of making STEP1 pass/fail
1. Minorities and low SES people do worse on STEP1
2. STEP1 is not a good measure of clinical ability
3. Making STEP1 pass/fail would incentivize using better markers for clinical ability

Points against making STEP1 pass/fail
1. It would remove the major meritocratic criteria present
2. This would in turn place even heavier emphasis on school prestige
3. It would force more research years, elongating med school in the era of more debt, this would in fact hurt low SES students, who happen to disproportionately be minorities, even more
4. There are ways the exam can be removed or the scoring modified such that less emphasis is placed on the exam, instead of going pass/fail. For example, a quintiles based approach (you are reported if you are in the top 20 the next 20 and so on and so forth). This would screw some people on the borderline of the quintiles but on balance mess with the least number of people will deemphasizing the current trend of OCPD style studying of trying to squeeze out every single last point. So top 20% would roughly be a mid to high 240s on STEP1, the median for even the most competitive specialties, barring a couple that have reached the 250 mark.
5. DO students will get screwed. USMLE is the only objective way for many to "prove" they can be as "good" as their MD counterparts.

Underrepresented Minorities have lower entrance requirements to begin with. The admit rates for a 3.60-3.79 GPA with a 30-32 on the MCAT for the following races
Blacks: 93.7%
Hispanics: 83.4%
Whites: 63%
Asians: 57.7%
All tables publicly available

FACTS

Data on U.S. medical school applicants, matriculants, enrollments, and graduates; as well as data on MD-PhD students, residency, and residency applicants.

www.aamc.org

The MCAT correlates with USMLE weakly to moderately (ranges from 0.4 for the whole thing in some studies to as high as 0.6, when looking at biological reasoning only). Admission to med school means that one has gone through undergraduate education. Undergraduate education completion means that one should be at a college reading level. STEP1, in terms of non medical terminology, can be argued to be written at below college reading level. Some of my classmates would have trouble understanding a tougher article in the Atlantic or WSJ than they would a STEP1 question. The exam is probably calibrated, with regard to non medical terminology, to be at something like a 10th grade reading level, hence why so many foreign grads, including those with poor English backgrounds, can still score in the upper echelons.

The STEP1 is a knowledge exam. It is a licensing test, not an aptitude test. The MCAT is an aptitude test but still with a heavy knowledge component, unlike the LSAT. This is why MENSA only accepted the latter in the past (it no longer accepts either). The rate limiting step the success on the STEP1 is how well you have memorized the information and the relevant connections. If one can meritocratically get into med school, one can do well on the STEP1.

The MCAT is based on years of accumulated knowledge with a big high level reading component. It, like the SAT, has some dependency on access to good prep materials and social class. But even that can be argued, given our internet age and access to a lot of good cheap resources. Also, unlike the SAT, people in tough situations can delay taking the MCAT for years, until they are prepared and "caught up" on the basic skills they need to brush up on. Even then, it can be shown the STEP1 deviates from the MCAT and that good STEP1 scores can be achieved even by very average MCAT scorers. The median MCATs of 1st year residents for following elite specialities

1. ENT: 33
2. Plastics: 32
3. Neurosurg: 32
4. Ortho: 32

Table B1: Test Scores and Experiences of First-Year Residents, by Specialty

Test Scores and Experiences of First-Year Residents, by Specialty

www.aamc.org

These specialties have around 85th percentile average STEP1 scores. Essentially, average level standardized test scorers in the past suddenly became around 1 SD above the mean. Again, this reflects the difference in the exams. STEP1 is largely a knowledge test. Knowledge tests rely more on crystallized intelligence than fluid intelligence. Crystallized intelligence is correlated with fluid intelligence, but it mostly a function of, barring the tiny tiny minority of med students with near photographic memories, consistent hours spent doing quality studying with quality resources. Most school googledrives have all of the resources. UFAP is like $500. People sign contracts that they aren't supposed to do full time jobs in med school, when they sign up for med school. There is enough time for people to use these resources to do really well.

In the end, the exam is really about hard work. What is this nonsense about no clinical correlation? If you don't understand basic pathophys of how action potential works, how can you truly understand an EKG? Do you want doctors to be pattern monkeys like many PAs and NPs? The essence of STEP1 is to test the building blocks that allow one to apply foundation knowledge to novel situations that don't fit neatly into algorithms. That being said, STEP1 is getting more and more clinically oriented. I had questions on first line treatments for various things. Scores are partially a function of that. Are you saying that isn't clinically relevant?

Here are some possible reforms
1. Make the exam more clinically relevant by using lingo that patients in inner city urban areas and rural areas use in the stem. This will actually show the broad variety of patients someone can work with.
2. Put some ultra basic Spanish in the question stem. It is essential to know a little bit in any ED nowadays, especially with the shoddy interpreter services that are often present at many non large academic centers.
3. Maybe some of nitty gritty biochem can really go away, but it is already trending that way. The questions mostly focus on clinical conditions.
4. Test underlying principles more. For example, contrast induced nephropathy can cause a pre renal azotemia type of BUN/creatinine. Instead of testing this by giving that type of ratio and then heaving most people who haven't come across that fact in a book get it wrong because they eliminate the choice, given the ratio is like 32 instead of like 12, as would be expected for an intrarenal phenomena like contrast induced nephropathy, test the underlying pathopahys. The reason this occurs is because contrast not only poisons the tubules directly, resulting in ATN, but also causes spasm of the prerenal vasculature, thus causing the prereanl azotemia type BUN/creatinine in some cases. Put in the stem the part about the spasm and ask what type of ratio would be expected. Force testing of foundational principles rather than specifically memorized cases for more esoteric things.
5. Stop testing two intuitively correct choices, where one was elucidated as the answer just by a recent trial. This makes these questions a 50/50 shot based on who guesses right or who happens to read latest trial data. The latter is what residency is more for. The purpose of STEP1 should be testing the foundation concepts. It is lazy question writing to just put in nit picky question that was only elucidated by a trial. Some of my STEP1 and STEP2 questions I could only find in journals. That is going too far.

My final recommendation would be a quintile based approach. That way people will still care about scores but not obsess over them. It will hurt borderline people that can be on balance argued to be better than the current arms race of turning everything into a one score arms race. Finally, shelf exams should be a thing for preclinical. That will allow better comparison of preclinical grades. NBME should release system based or subject based ones and standardize them well. They already sort of do this. But mandate it. That way the first two years will have a standardized component.

I can see why some elite schools are salivating at P/F. It provides excellent plausible deniability of, at least on the surface, commitment to the social justice vision of fairness. Yet the actual effects will be greater emphasis on school prestige, thus further entrenching and enhancing the advantage of high tier status for residency selection. This will favor their grads and then in turn favor them down the line because even more of their grads will dominate medical leadership. Some elite places have not only gotten rid of preclinical grades but also clinical ones. They want their students to be able to ride the prestige wave all the way.

All in all, I think a quintile based approach and mandating shelf exams for preclinical and clinical years would be a good approach along with reforming the exam to have more relevant questions, not only clinically but also more oriented towards foundation knowledge rather than memorization of esoteric information.

Shelf exam weighting and policy is school dependent. It’s sadly moot to bring that up at this very moment.

I’m interested in hearing arguments against specifically a quartile or quintile system.

Lastly, let me clarify and reiterate something. Medical educators are in favor of p/f step 1 probably because it disincentives their efforts to improve preclinical classes, not necessarily because it would increase lecture attendance (like I posited earlier)

 

[7331poas]

Let me summarize a few points of crystallization for this thread and provide an argument that I think many of us, who have been through the process, can buy

Points in favor of making STEP1 pass/fail
1. Minorities and low SES people do worse on STEP1
2. STEP1 is not a good measure of clinical ability
3. Making STEP1 pass/fail would incentivize using better markers for clinical ability

Points against making STEP1 pass/fail
1. It would remove the major meritocratic criteria present
2. This would in turn place even heavier emphasis on school prestige
3. It would force more research years, elongating med school in the era of more debt, this would in fact hurt low SES students, who happen to disproportionately be minorities, even more
4. There are ways the exam can be removed or the scoring modified such that less emphasis is placed on the exam, instead of going pass/fail. For example, a quintiles based approach (you are reported if you are in the top 20 the next 20 and so on and so forth). This would screw some people on the borderline of the quintiles but on balance mess with the least number of people will deemphasizing the current trend of OCPD style studying of trying to squeeze out every single last point. So top 20% would roughly be a mid to high 240s on STEP1, the median for even the most competitive specialties, barring a couple that have reached the 250 mark.
5. DO students will get screwed. USMLE is the only objective way for many to "prove" they can be as "good" as their MD counterparts.

Underrepresented Minorities have lower entrance requirements to begin with. The admit rates for a 3.60-3.79 GPA with a 30-32 on the MCAT for the following races
Blacks: 93.7%
Hispanics: 83.4%
Whites: 63%
Asians: 57.7%
All tables publicly available

FACTS

Data on U.S. medical school applicants, matriculants, enrollments, and graduates; as well as data on MD-PhD students, residency, and residency applicants.

www.aamc.org

The MCAT correlates with USMLE weakly to moderately (ranges from 0.4 for the whole thing in some studies to as high as 0.6, when looking at biological reasoning only). Admission to med school means that one has gone through undergraduate education. Undergraduate education completion means that one should be at a college reading level. STEP1, in terms of non medical terminology, can be argued to be written at below college reading level. Some of my classmates would have trouble understanding a tougher article in the Atlantic or WSJ than they would a STEP1 question. The exam is probably calibrated, with regard to non medical terminology, to be at something like a 10th grade reading level, hence why so many foreign grads, including those with poor English backgrounds, can still score in the upper echelons.

The STEP1 is a knowledge exam. It is a licensing test, not an aptitude test. The MCAT is an aptitude test but still with a heavy knowledge component, unlike the LSAT. This is why MENSA only accepted the latter in the past (it no longer accepts either). The rate limiting step the success on the STEP1 is how well you have memorized the information and the relevant connections. If one can meritocratically get into med school, one can do well on the STEP1.

The MCAT is based on years of accumulated knowledge with a big high level reading component. It, like the SAT, has some dependency on access to good prep materials and social class. But even that can be argued, given our internet age and access to a lot of good cheap resources. Also, unlike the SAT, people in tough situations can delay taking the MCAT for years, until they are prepared and "caught up" on the basic skills they need to brush up on. Even then, it can be shown the STEP1 deviates from the MCAT and that good STEP1 scores can be achieved even by very average MCAT scorers. The median MCATs of 1st year residents for following elite specialities

1. ENT: 33
2. Plastics: 32
3. Neurosurg: 32
4. Ortho: 32

Table B1: Test Scores and Experiences of First-Year Residents, by Specialty

Test Scores and Experiences of First-Year Residents, by Specialty

www.aamc.org

These specialties have around 85th percentile average STEP1 scores. Essentially, average level standardized test scorers in the past suddenly became around 1 SD above the mean. Again, this reflects the difference in the exams. STEP1 is largely a knowledge test. Knowledge tests rely more on crystallized intelligence than fluid intelligence. Crystallized intelligence is correlated with fluid intelligence, but it mostly a function of, barring the tiny tiny minority of med students with near photographic memories, consistent hours spent doing quality studying with quality resources. Most school googledrives have all of the resources. UFAP is like $500. People sign contracts that they aren't supposed to do full time jobs in med school, when they sign up for med school. There is enough time for people to use these resources to do really well.

In the end, the exam is really about hard work. What is this nonsense about no clinical correlation? If you don't understand basic pathophys of how action potential works, how can you truly understand an EKG? Do you want doctors to be pattern monkeys like many PAs and NPs? The essence of STEP1 is to test the building blocks that allow one to apply foundation knowledge to novel situations that don't fit neatly into algorithms. That being said, STEP1 is getting more and more clinically oriented. I had questions on first line treatments for various things. Scores are partially a function of that. Are you saying that isn't clinically relevant?

Here are some possible reforms
1. Make the exam more clinically relevant by using lingo that patients in inner city urban areas and rural areas use in the stem. This will actually show the broad variety of patients someone can work with.
2. Put some ultra basic Spanish in the question stem. It is essential to know a little bit in any ED nowadays, especially with the shoddy interpreter services that are often present at many non large academic centers.
3. Maybe some of nitty gritty biochem can really go away, but it is already trending that way. The questions mostly focus on clinical conditions.
4. Test underlying principles more. For example, contrast induced nephropathy can cause a pre renal azotemia type of BUN/creatinine. Instead of testing this by giving that type of ratio and then heaving most people who haven't come across that fact in a book get it wrong because they eliminate the choice, given the ratio is like 32 instead of like 12, as would be expected for an intrarenal phenomena like contrast induced nephropathy, test the underlying pathopahys. The reason this occurs is because contrast not only poisons the tubules directly, resulting in ATN, but also causes spasm of the prerenal vasculature, thus causing the prereanl azotemia type BUN/creatinine in some cases. Put in the stem the part about the spasm and ask what type of ratio would be expected. Force testing of foundational principles rather than specifically memorized cases for more esoteric things.
5. Stop testing two intuitively correct choices, where one was elucidated as the answer just by a recent trial. This makes these questions a 50/50 shot based on who guesses right or who happens to read latest trial data. The latter is what residency is more for. The purpose of STEP1 should be testing the foundation concepts. It is lazy question writing to just put in nit picky question that was only elucidated by a trial. Some of my STEP1 and STEP2 questions I could only find in journals. That is going too far.

My final recommendation would be a quintile based approach. That way people will still care about scores but not obsess over them. It will hurt borderline people that can be on balance argued to be better than the current arms race of turning everything into a one score arms race. Finally, shelf exams should be a thing for preclinical. That will allow better comparison of preclinical grades. NBME should release system based or subject based ones and standardize them well. They already sort of do this. But mandate it. That way the first two years will have a standardized component.

I can see why some elite schools are salivating at P/F. It provides excellent plausible deniability of, at least on the surface, commitment to the social justice vision of fairness. Yet the actual effects will be greater emphasis on school prestige, thus further entrenching and enhancing the advantage of high tier status for residency selection. This will favor their grads and then in turn favor them down the line because even more of their grads will dominate medical leadership. Some elite places have not only gotten rid of preclinical grades but also clinical ones. They want their students to be able to ride the prestige wave all the way.

All in all, I think a quintile based approach and mandating shelf exams for preclinical and clinical years would be a good approach along with reforming the exam to have more relevant questions, not only clinically but also more oriented towards foundation knowledge rather than memorization of esoteric information.

Lots of typing for poor recommendations imo.

Ill go stepwise.

1) having "lingo" is an awful idea, since in real practice I would just ask a clarifying question once the patient uses the colloquialism.

2) Asking students to learn Spanish will be the new "krebs cycle"

3) The test already does this.

4) The test already tests principles

5) I have personally never seen this.

Final conclusion: quintiles is still a bad choice because it does not differentiate the students in the top quintile who represent the entire population of derm applicants. That change is essential still P/F

The only way your system will work is if we also mandate Step 2 before ERAS.

 

[TheIllusionist]

Lots of typing for poor recommendations imo.

Ill go stepwise.

1) having "lingo" is an awful idea, since in real practice I would just ask a clarifying question once the patient uses the colloquialism.

2) Asking students to learn Spanish will be the new "krebs cycle"

3) The test already does this.

4) The test already tests principles

5) I have personally never seen this.

Final conclusion: quintiles is still a bad choice because it does not differentiate the students in the top quintile who represent the entire population of derm applicants. That change is essential still P/F

The only way your system will work is if we also mandate Step 2 before ERAS.

shouldn't matter even for derm once you hit top qyintile. also derm step average would put maybe 55% in top quintile so not even close to entire population.

and STEP2 should be mandated prior to ERAS. many medicine programs don't begin review till those scores are in.

The issue is extent. Test has to do more of what I stated. It isn't like status quo is doing none at all. You personally not seeing something, doesn't preclude existence. I can give exampls, but I don't want to screwed by the NBME for talking about my exam. Lingo and Spanish and communication issues **** up more medical care than a lot of other things. People often don't ask the clarifying questions or just assume they get it.

 

[TheIllusionist]

Shelf exam weighting and policy is school dependent. It’s sadly moot to bring that up at this very moment.

I’m interested in hearing arguments against specifically a quartile or quintile system.

Lastly, let me clarify and reiterate something. Medical educators are in favor of p/f step 1 probably because it disincentives their efforts to improve preclinical classes, not necessarily because it would increase lecture attendance (like I posited earlier)

LCME needs to work with NBME to mandate min threshold of weightage for clinicial and preclinical shelves

 

[Ho0v-man]

I’m interested in hearing arguments against specifically a quartile or quintile system.

Just the arguments that have already been stated. No real difference between a 240 and a 239 but if a line is arbitrarily drawn between the two it makes it like the difference between a 29 and a 30 on the old mcat which was super stupid.

 

[TheIllusionist]

Just the arguments that have already been stated. No real difference between a 240 and a 239 but if a line is arbitrarily drawn between the two it makes it like the difference between a 29 and a 30 on the old mcat which was super stupid.

again cost benefit ratios being weighed. some people will be hurt no matter what decision

 

[Chibucks15]

Just the arguments that have already been stated. No real difference between a 240 and a 239 but if a line is arbitrarily drawn between the two it makes it like the difference between a 29 and a 30 on the old mcat which was super stupid.

No difference in reality but people love to say that they 'beat' somebody else haha prestige-driven to the max. That's what you get when you have a bunch of type As who have never really came up short before. Its the nature of the beast

 

[Lucca]

Just the arguments that have already been stated. No real difference between a 240 and a 239 but if a line is arbitrarily drawn between the two it makes it like the difference between a 29 and a 30 on the old mcat which was super stupid.

When providing information to humans, it is often more helpful to obscure information and work to patch up corner cases than disinterestedly provide more information that has a high risk of being misapplied.

What do I mean?

Let's say you're a PD and you receive a 250 and a 239. Your program uses a "soft" cutoff of 245 because you're a Big Time Head and Neck program at a Big Time Hospital and you only want to train Big Time Head and Neck Surgeons. Under the current paradigm, ceteris paribus the 239 goes in the trash and the 250 gets a closer look. Restricted to purely the information contained in a Step 1 score, there is no discernible difference between these 2 scores; indeed, it is highly plausible the exact same individual could have received either of these scores (or even slightly lower...or higher) on their single test date.

Now let's say you're a PD living in the future Nightmare Dystopia where Step 1 is scored on quintiles, dogs and cats living together etc. Let's say 250 is the (inclusive) cutoff for top quintile (which, to my knowledge, is about where it would be given current percentiles). You get two applicants with board scores reported as Q1 and Q2. Well, your Big Time program isn't going to settle for second-rate doctors, is it? Luckily for the Q2 applicant, following the Step 1 Apocalypse of 2025 (RIP to the 20,000+ med students who were forced at gunpoint to do FM in Nebraska that year as a result of not attending a Top 20 Medical School) human beings decided to use their brains to think about how to improve residency selection. There is a hard cap on the applications the PD received and now that they dont have 100 apps for every residency slot, they can give both a closer look. Universal SLOE implementation (or anything else like it) has given them more metrics with which to evaluate applicants, perhaps even leading to a second numerical score that takes more than just raw willpower and memorization ability into account.

The bigger point I'm trying to make is that both of the following things are true:

1) if step 1 scoring is to change it cannot be the only thing to change or it will indeed be a bad move
2) if step 1 scoring does not change, nothing else ever will.

 

[Ho0v-man]

When providing information to humans, it is often more helpful to obscure information and work to patch up corner cases than disinterestedly provide more information that has a high risk of being misapplied.

What do I mean?

Let's say you're a PD and you receive a 250 and a 239. Your program uses a "soft" cutoff of 245 because you're a Big Time Head and Neck program at a Big Time Hospital and you only want to train Big Time Head and Neck Surgeons. Under the current paradigm, ceteris paribus the 239 goes in the trash and the 250 gets a closer look. Restricted to purely the information contained in a Step 1 score, there is no discernible difference between these 2 scores; indeed, it is highly plausible the exact same individual could have received either of these scores (or even slightly lower...or higher) on their single test date.

Now let's say you're a PD living in the future Nightmare Dystopia where Step 1 is scored on quintiles, dogs and cats living together etc. Let's say 250 is the (inclusive) cutoff for top quintile (which, to my knowledge, is about where it would be given current percentiles). You get two applicants with board scores reported as Q1 and Q2. Well, your Big Time program isn't going to settle for second-rate doctors, is it? Luckily for the Q2 applicant, following the Step 1 Apocalypse of 2025 (RIP to the 20,000+ med students who were forced at gunpoint to do FM in Nebraska that year as a result of not attending a Top 20 Medical School) human beings decided to use their brains to think about how to improve residency selection. There is a hard cap on the applications the PD received and now that they dont have 100 apps for every residency slot, they can give both a closer look. Universal SLOE implementation (or anything else like it) has given them more metrics with which to evaluate applicants, perhaps even leading to a second numerical score that takes more than just raw willpower and memorization ability into account.

The bigger point I'm trying to make is that both of the following things are true:

1) if step 1 scoring is to change it cannot be the only thing to change or it will indeed be a bad move
2) if step 1 scoring does not change, nothing else ever will.

Get what you’re going for in bold but that just leaves the door open to doing well on something subjective instead of an objective measure which sounds terrible.

I’m for having one score reported that is just an average of the 2 exams. That way people can make up for a low step 1.

 

[sargon2123]

A question I’m wondering is where do underperformers at T25 schools typically go for residency? There can’t be that many T25 students that everyone else is forced into rural residency programs or something like that. Mid/high tier IM programs (but not top 5) for example aren’t dominated by T25 students as it stands

 

[Chibucks15]

A question I’m wondering is where do underperformers at T25 schools typically go for residency? There can’t be that many T25 students that everyone else is forced into rural residency programs or something like that. Mid/high tier IM programs (but not top 5) for example aren’t dominated by T25 students as it stands

Its rare for people to be 'forced into rural FM', people on here love to talk about it and consider everything between Pittsburgh and Denver as rural. There's a reason most of the true rural FM spots are filled by Caribbean, super low performers, etc. Unless you have multiple failures and a ton of red flags you'll match somewhere decent, especially from a T25

 

[sargon2123]

Its rare for people to be 'forced into rural FM', people on here love to talk about it and consider everything between Pittsburgh and Denver as rural. There's a reason most of the true rural FM spots are filled by Caribbean, super low performers, etc. Unless you have multiple failures and a ton of red flags you'll match somewhere decent, especially from a T25

But for non T-25s, if there is a program that students at mid tier schools already match into fairly frequently (e.g. the home program for IM), can we assume the blowback of step 1 p/f will be minimal for matching at that specific program?

 

[Chibucks15]

But for non T-25s, if there is a program that students at mid tier schools already match into fairly frequently (e.g. the home program for IM), can we assume the blowback of step 1 p/f will be minimal for matching at that specific program?

Honestly I would guess so, because then you have networking opportunities through your home program. And if S1 is P/F, networking importance increases also

 

[AnatomyGrey12]

There is a hard cap on the applications the PD received and now that they dont have 100 apps for every residency slot, they can give both a closer look.

Congratulations, you just proposed another way to screw over DOs and low tier MD students.

 

[Espressso]

Congratulations, you just proposed another way to screw over DOs and low tier MD students.

By capping the # of apps people can send out? I don't think capping the # of caps hurt DO students anymore than it does MD students. I think it hurts applicants who don't apply with a strategy