Scaling is a huge problem.
Complexity increases quadratically with number of tokens...
https://newsletter.pragmaticengineer.com/p/scaling-chatgpt
They seem to be hitting a wall with this approach.
They're moving monstrous amounts of data around, all over the planet, to do fancy page-level auto-complete and make weird pictures of white women with too many fingers and pointy chins. It's not intelligence.
Yet.
And they may burn up the planet before they get there.
They're setting huge amounts of money on fire, even on their most expensive plan:
That's 4 years from now. I think he's dreaming.
Time will tell, of course.
Best wishes,
Scott.
https://newsletter.pragmaticengineer.com/p/scaling-chatgpt
Scalability challenge from self-attention
Under the hood, we use the Transformer architecture, a key characteristic of which is that each token is aware of every other token. This approach is known as self-attention. A consequence is that the longer your text is – or context – the more math is needed.
Unfortunately, self attention scales quadratically. If you want the model to predict the 100th token, it needs to do about 10,000 operations. If you want the model to predict the 1,000th token, it needs to do about 1 million operations.
At first, this sounds like bad news. However, there are clever workarounds we can use to circumvent the quadratic scale problem. Before we get into how we solve it, we need to talk about the infrastructure powering ChatGPT.
They seem to be hitting a wall with this approach.
They're moving monstrous amounts of data around, all over the planet, to do fancy page-level auto-complete and make weird pictures of white women with too many fingers and pointy chins. It's not intelligence.
Yet.
And they may burn up the planet before they get there.
They're setting huge amounts of money on fire, even on their most expensive plan:
[...]
OpenAI isn’t profitable, despite having raised around $20 billion since its founding. The company reportedly expected losses of about $5 billion on revenue of $3.7 billion last year.
Expenditures like staffing, office rent, and AI training infrastructure are to blame. ChatGPT was at one point costing OpenAI an estimated $700,000 per day.
Recently, OpenAI admitted it needs “more capital than it imagined” as it prepares to undergo a corporate restructuring to attract new investments. To reach profitability, OpenAI is said to be considering increasing the price of its various subscription tiers. Altman also hinted in the Bloomberg interview that OpenAI may explore usage-based pricing for certain services.
OpenAI optimistically projects that its revenue will reach $11.6 billion this year and $100 billion in 2029, matching the current annual sales of Nestlé.
That's 4 years from now. I think he's dreaming.
Time will tell, of course.
Best wishes,
Scott.
