Usually, I would say it "doesn't matter." But if you are looking at working in very competitive areas (e.g. metropolitan areas in California), and especially as a newbie employee, I would say that where you train definitely matters in distinguishing yourself. For example, I remember interviewing at the AAO for a Bay Area associate position. There must have been 30 people interviewing for the same job that same day. So I'm sure the employers have an innate bias toward "more prestigious" residency programs. If you are opening up your own shop, I don't think it matters at all.
But definitely do not go somewhere where you feel like you will not thrive. It is more important to be productive at an "average" residency than to be miserable (and thus less productive) at a better residency. Also, keep in mind that residency is only 3-4 years of your life. It is really such a short time in the grand scheme of things.
As for patients, ~95% of them could care less where I trained. Of course, there are a handful of patients that really research your credentials, but those people are in the minority... and typically are very crazy lol. Plus, the general population does not realize that "lowly" schools like the U of Miami or UCLA are actually some of the best places to train, compared to, say, going to an Ivy League institution.