A “Potemkin village” signifies any deceptive or false construct, conjured often by cruel regimes, to deceive both those within the land and those peering in from outside.
In AI
AI systems attempt to build a sort of ‘potemkin village’ that “works well on naturally occurring data, but is exposed as fake when one visits points in space that do not have high probability”