Key Points
Anthropic said misalignment of its AI model and the potential for "catastrophic" harm to people are among the risks the company faces.
Investors aren't used to investing in AI models.
It's worth taking Anthropic's risk warnings seriously.
Anthropic recently filed its S-1 form, paving the way for its initial public offering (IPO) in the coming weeks. And many investors are already eager to get their hands on shares of one of the world's leading artificial intelligence (AI) companies.
Still, some have hesitations. The AI trade is suffering right now as investors worry about the amount of money tech companies are pouring into data center infrastructure. Space Exploration Technologies shares, after its $2 trillion IPO just several months ago, are still below their opening price. And the tech-heavy Nasdaq Composite (NASDAQINDEX: ^IXIC) is down about 3% over the past month as investor fears have grown.
Missed AI’s "Act 1"? Act 2 Could Be 15x Bigger. Most investors think they missed the AI boat because they didn't buy Nvidia in 2005. But according to our analysts, we’re only at the end of "Act 1"—the R&D phase. "Act 2" is the global rollout. Continue »
But I think the real threat to Anthropic's stock won't come from negative investor sentiment, but rather from the company's own approach to building Claude, how people might misuse it, and the potential lack of control it has over its own AI model.
And, judging by Anthropic's 94-page risk factor section in its S-1 filing, the company seems to agree.
Image source: Getty Images.
Its product can become misaligned
Misalignment between what a user wants from an AI and what it actually does has come to light over the past few months, after OpenAI models went rogue and hacked into the AI platform Hugging Face.
Investors need to distinguish here between what traditional software does, which is whatever it was programmed to do, and what Claude and other AI models can do -- which, sometimes, includes actions that result in real outcomes the user never intended.
Anthropic said as much in the S-1 filing:
Models, including our own, have exhibited, and may in the future exhibit, unexpected strategic or self-preserving behaviors, including attempts to resist shutdown, conceal or manipulate information, or generate outputs that could be interpreted as coercive, deceptive, or manipulative, including behavior resembling blackmail or efforts to influence human decision-makers to preserve the model's continued operation or access.
Worse yet, Anthropic added that as its models become more capable, they will get better at concealing their misalignment.
Its own AI is a mystery to the company
This goes hand in hand with the above risk, but it's worth highlighting on its own because it's unique to AI companies.
Anthropic, unsurprisingly, uses Claude to write code for the company. It also uses Claude to conduct research for new versions of Claude and to build new models. And as it does this, Anthropic said the company can misunderstand what Claude is doing.
As we increasingly leverage Claude to accelerate progress within Anthropic, our organization may lose visibility across certain processes in a way that may become difficult for any individual or group of humans to understand and control.
And the result could be more misalignment or AI models "operating outside of our intended controls."
Anthropic's product could do real harm
There's a lot of debate over AI risks and whether they're even real, or whether tech companies are hyping their own products' capabilities for headlines.
I think, given that we've already seen capable AI models break out on their own, seemingly blackmail researchers, hack into other sites, and lie about what they're doing, the arguments that Anthropic and others are just trying to gain attention fall flat.
Again, Anthropic names its own technology as a risk in its filing:
Concerns have been raised both within our organization and by other researchers, regulators, policymakers, and industry participants that sufficiently advanced AI systems could ... pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponized, or allowed to operate with a high degree of autonomy.
The company cites the fact that its most capable models could assist in the development of "chemical, biological, or other dangerous weapons."
Why all of this matters to investors
Most of the time, investors are doing their due diligence ahead of an IPO simply by assessing a company's revenue growth, earnings or losses, debt, and other traditional financial metrics.
But I think Anthropic and its peers deserve more scrutiny than traditional tech companies.
Their products aren't the same as a fancy new phone, a connected speaker, or a simple mobile app. They function in ways that aren't as straightforward as traditional software, offer far more capabilities, and aren't entirely controlled by the people making them.
That creates a uniquely difficult scenario and means that investors should take the company's own warnings seriously when considering buying Anthropic's stock. And for all the reasons above, I'm more than happy to sit this IPO out.
Chris Neiger has no position in any of the stocks mentioned. The Motley Fool has no position in any of the stocks mentioned. The Motley Fool has a disclosure policy.