Ponzi schemes as a demonstration of out-of-distribution generalization
A Ponzi scheme is fraud where the fraudster induces investors to give the fraudster money with promises of profits, and then uses money from later investors to pay out earlier investors. This pattern, as well as the phrase "Ponzi scheme" have become ubiquitously associated with fraud and grift in modern usage. One might be forgiven for wondering, how is it that anyone ever fell for this type of scam?
We all probably like to think we could never fall for such an obvious con, but I feel that confidently believing so is subject to a two-fold hindsight bias. One, when someone attempts to involve you in a Ponzi scheme, they don't say "by the way, this is a Ponzi scheme". They claim that they are engaged in a legitimate enterprise, and not only that, they have the returns to back it up! They have their previous happy investors who can vouch for the fact that the enterprise really is able to do what they promise. In hindsight, we might be tempted put scare-quotes around "returns", but I don't think this is right. The entire point of a Ponzi scheme is that you really do pay out your early investors! They factual did put in a certain amount of money and got that money back, plus a great return. What could be a more persuasive and valid form of evidence that your business is legit than the actual past performance of your business, with actual cash as proof of that performance? If we avoid using hindsight and put ourselves in the shoes of the scams victims, it actually makes a lot of sense. Without understanding the underlying fundamentals of the business the returns to early investors seem like good evidence that the schemer can produce the promised returns. It is only after the fact, once the nature of the scheme is revealed, that it clicks why those earlier returns weren't necessarily predictive of future returns.
Two, there is a more complicated layer of hindsight that might not be so obvious. There is a reason it's called a "Ponzi" scheme, named for a historical perpetrator of such a fraud. Also commonly mentioned in discussions around Ponzi schemes are cases such as Bernie Madoff. Past examples of Ponzi schemes are common knowledge, to the extent that it is not uncommon for commentators to explicitly invoke the "Ponzi scheme" phrase with regard to enterprises or assets that allegedly bear some similarity to the classic Ponzi scheme. We have had the chance to learn from these historical events, and these lessons have now started to make their way into the culture (just check out the section from the link at the top titled "red flags"). But just because someone is aware of these red flags now, doesn't mean that same person would have spotted a Ponzi scheme if they were in the position of historical victims, without the benefit of this second kind of hindsight.
Evaluating a Ponzi scheme in the making isn't as simple as it might appear after the fact. Initially, the scheme actually is producing good returns for its initial investors, it's just doing so on the backs of later ones. Viewed from a statistical perspective, it is perfectly reasonable that someone would estimate future returns using existing returns given out so far. There is nothing unusual about that. The problem is that at some point there is a shift in returns that the scheme produces. Taking the Madoff case as an example, perhaps an economic downturn spooks investors who suddenly all want there money back, while new investors willing to sign on have dried up. All of a sudden there aren't any new investors to pay previous ones, and the payouts vanish. When such a distributional shift occurs, the distribution of returns from earlier in the life-cycle of scheme no longer reflect the returns after the shift.
I think this is a useful and instructive demonstration of a concept in statistics and machine learning called out-of-distribution generalization. Out-of-distribution generalization address the situation where a model is trained on data generated by one distribution, but it is tested or deployed on data generated by another distribution. This can result in error rates and properties that hold in training failing to hold in testing or deployment, in a manner that is different and more systematic than traditional overfitting. With traditional overfitting, testing on a held-out set with new examples has you covered, but this isn't true for out-of-distribution robustness. The most obvious reason for this is that if you use a test set that has an identical distribution to training (like you would get if you randomly split for train and test sets) you aren't testing out-of-distribution. However, this naturally leads to the question, couldn't you just use a test set that has a distributional shift to test out-of-distribution generalization?
This idea has been raised in the literature as well as in discussions about AI safety. In particular, I think this is relevant to distinctive cultures that exist among those interested in risk from advanced AI. There is a perspective on AI risk, prevalent at leading AI labs, that emphasizes empirical work using frontier AI models. This is a critical part of the argument for these labs that their strategy of building more advanced models is useful for safety. It is also a major source of disagreement with more theoretically minded, If Anyone Builds It Everyone Dies style AI safety. Part of the counterargument that labs make to IABIED style arguments is related to the claimed strong ability of existing AI models to generalize. An example of how this plays out comes from a response to so-called "counting arguments" in the article "Counting arguments provide no evidence for AI doom" from two self-proclaimed AI optimists. Quoting from that article:
The argument also predicts that larger networks— which can express a wider range of functions, most of which perform poorly on the test set— should generalize worse than smaller networks. But empirically, we find the exact opposite result: wider networks usually generalize better, and never generalize worse, than narrow networks. These results strongly suggest that SGD is not doing anything like sampling uniformly at random from the set of representable functions that do well on the training set.
More generally, John Miller and colleagues have found training performance is an excellent predictor of test performance, even when the test set looks fairly different from the training set, across a wide variety of tasks and architectures.
These results clearly show that the conclusion of our parody argument is false. Neural networks almost always learn genuine patterns in the training set which do generalize, albeit imperfectly, to unseen test data.
The article cites this paper "Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization", which argues that in-distribution and out-of-distribution performance are highly correlated. So the argument might go like this. Sure, in theory maybe there is a concern about out-of-distribution generalization, but empirically more advanced models are getting better at this, not worse, and in-distribution performance is also empirically a good predictor of out-of-distribution performance. This shows that theories such as "sharp left turns" and other ideas from the IABIED side aren't actually borne out in practice.
This is what makes out-of-distribution generalization such a pernicious challenge, like the issue of hindsight with Ponzi schemes. Take the case of Bernie Madoff. Madoff operated his scheme for over a decade and perhaps longer, through all sorts of different market conditions during that time. Without using hindsight, it could almost seem anti-empirical to criticize Madoff. Isn't operating successfully for a decade strong empirical evidence? If you're giving your clients satisfactory performance, isn't that the best available evidence that you'll be able to keep offering that performance in the future? Sure you never know what the market will do, "past performance is not indicative of future results" as the disclaimers say, but isn't the best possible empirical evidence about future results?
In the context of out-of-distribution generalization, there isn't just one "out-of-distribution" context. It matters what the future distributional shift is. A model can perform fine under some shifts but terribly under others. If you do some empirical research on "out-of-distribution generalization" of a model but the shifts that the model faces in deployment are different from the ones you studied in your research, that research may not be indicative of the model's performance. In other words, your empirical results face their own out-of-distributional generalization problem! This is kind of like that first layer of hindsight in the Ponzi scheme situation. Those decades of past results didn't protect Madoff's clients when the 2008 financial crisis rolled around.
But researchers don't just study one model and one shift. That paper's abstract says, "we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts". Doesn't studying "a wide range of models and shifts" address this issue? Even beyond that, AI models qualitatively can do pretty impressive things that seem like they require the ability to generalize. You can go ask a model something completely novel right now and get interesting and helpful responses.
This is where things get more complicated, similar to the second layer of hindsight in the context of Ponzi schemes. I can look back at historical Ponzi schemes and learn the patterns and hope I won't fall for a similar scam myself. On the other hand, scammers can also look back at these cases, see how those individuals failed and are aware of what potential victims will look for as warning signs. The next Bernie Madoff might not look like Bernie Madoff. The next big Ponzi schemer might even intentionally change up certain aspects to avoid suspicion. This intentional avoidance could mean that the distributional shift from past schemers to future ones is adversarial designed to fool potential victims and the internal mental models they have build up by hearing about past schemers. That's the tough thing about out-of-distributional generalization. No matter how robust your model is to some class of distributional shifts, if the shift you actually face in practice is outside that class, that robustness counts for nothing.
In my view, reliable out-of-distribution robustness requires some kind of model of what distributional shifts will show up in the future. I have become convinced by certain lines of research that you can't just have general out-of-distribution robustness, you to also have assumptions that restrict the possible distributional shifts in relation to your model. Similarly, I think you need to have transparency into how your model actual works, you need to "open the box". This is needed to understand how the model will be effected by certain distributional shifts. In the Ponzi scheme analogy, this is asking how the enterprise actually achieves its returns. If the returns so far are good but you can see that the enterprise lacks any fundamental way of making money, you can identify the instability. In order to show that the business is a scam, you have to open the books. I have argued before that black-box evaluations can't give us all the answers if we allow any and all possible distributional shifts, including adversarial ones. I hope the Ponzi scheme analogy helps to demonstrate the nature of the problem.
