Potency is a metric of how much of a drug is required to illicit its effect. It isn't a good stand alone term. 5 nanograms of some toxins can kill you, while 500 mgs of some drugs can kill you, etc etc. It is a good comparative term - i.e. "Drug A requires 5g to do what Drug B requires 50g to do - Drug A is more potent".
Non-competitive antagonism by definition acts at an "allosteric" (which just translates as "other site" essentially) site, which isn't the active site of the target (if it is an enzyme, which is usually the premise for these discussions.)
My initial impulse would be a non-competitive antagonist can't change the potency of a drug. The drug still functions in the same dose-range, just achieves less (i.e. has a lower efficacy). A competitive inhibitor, however, I would guess makes a drug seem less potent, since now it may require 6g to accomplish what it once could do in 5g.