It depends on your scope of practice as well. If you want to do a broader scope of practice (i.e. certain procedures, inpatient, or OB) I would say it matters a lot.
And I would say you at least want to get into a medium-tier residency with good teaching. Many programs churn out docs that don't end up getting good training.