Transferred from: financial sector

This article is derived fromPokertablesforsalenearmeFinancial sector

Morgan Stanley believes that Nvidia is the key to the future exponential growth of computing power, Google, Meta, Amazon and Microsoft will be the biggest beneficiaries of this process, while small companies may be marginalized in the world of larger models, but lower-cost smaller models will create new opportunities for them.

Meta's third-generation big model Llama 3 finally made its official debut this week: the maximum parameter size is over 400 billion, the training token is over 15 trillion, and the success rate is more than 60% compared to GPT-3.5 's various human evaluation. It is officially known as the "strongest open source model on the surface".

In the "volume" of the major technology giants, the big model has finally reached a critical turning point. Morgan Stanley points out that the world is entering a new era of rapid growth of large models driven by hardware and software, which will significantly improve their creativity, strategic thinking and ability to handle complex multi-dimensional tasks.

The report emphasizes that the training of large models in the future will require unprecedented numeracy, which will lead to a significant increase in development costs. A report released this week by a team of Morgan Stanley Stephen C Byrd analysts predicted that the high cost of supercomputers needed to train the next generation of big models is a huge challenge even for tech giants, let alone small companies.

The report further points out that in addition to high capital expenditure, barriers to chip power supply and artificial intelligence technology are also increasing. Together, these factors constitute a major obstacle to entering the field of large models, which may make it difficult for small companies to compete with powerful giants.

As a result, Morgan Stanley has given overweight ratings to large technology companies such as Google, Meta, Amazon and Microsoft, which are expected to take the lead in the development of large models because of their technological, capital and market strengths. At the same time, while small companies may be marginalized in the world of larger models, smaller models with lower costs will create new opportunities for them.

Nvidia is the key to exponential growth in the future.Pokertablesforsalenearme?

Morgan Stanley pointed out that in the near future, the computing power needed to develop large models will grow exponentially, which is closely related to the progress of chip technology. Blackwel, the "strongest chip in history", is one of the key technologies to promote the growth of computing power.

Take the GPT model trained by OpenAI as an example.

Morgan Stanley points out that it currently takes about 100 days to train GPT-4, uses 25000 Nvidia A100 GPU, processes 13 trillion token, and involves about 1.76 trillion parameters.

The total computing power of these A100s (in FP8 teraFLOPs terms) is about 16 million. The teraFLOPs is a measure of floating-point performance, indicating how many trillions of floating-point operations can be performed per second. The total number of floating-point operations required for GPT-4 training is about 137 trillion.

For the upcoming GPT-5, Morgan Stanley estimates that the training of the model will require the deployment of 200000-300000 H100 GPU, which will take 130-200 days.

Supercomputers will make exponential growth expectations easier to achieve. The Morgan Stanley model shows that supercomputers will provide more than 1000 times more computing power for the development of large models later this decade than at current levels.

Using Blackwell supercomputers, it only takes 150-200 days of training time to develop a completely new large model, which provides 1400-1900 times more computing power than the current model, such as GPT-4.

The report also mentioned that the annual calculation power required by GPT-6 in the future will account for a considerable percentage of Nvidia chip annual sales. It is estimated that the cost of a 100MW data center using B100 or H100 GPU may be $1.5 billion.

Morgan Stanley sees Nvidia as a key driver of arithmetic growth.

According to forecasts, Nvidia will grow at a compound annual growth rate of 70% from 2024 to 2026. This growth rate is calculated based on SXM (which may be the code name of a product or service of NVIDIA) and FP8 Tensor Core (a performance metric).

In the era of big models, the tech giants are the biggest beneficiaries?

However, the supercomputers needed to develop super-powerful models and their training involve a series of complex challenges, including capital investment, chip supply, power requirements and software development capabilities. These factors constitute the main barriers to entry into this field, which will give more opportunities to those well-capitalized and technologically advanced technology giants.

In terms of capital investment, Morgan Stanley compared the data center capital expenditures of Google, Meta, Amazon and Microsoft in 2024 for a range of supercomputers of different sizes, of which a 1 gigawatt supercomputer facility is estimated to cost about $30 billion, while a larger supercomputer could cost as much as $100 billion.

Morgan Stanley expects the four US ultra-large computing companies to spend about $155 billion and more than $175 billion on their data centers in 2024 and 2025, respectively. These huge figures will deter small businesses.

The agency also believes that Google, Meta, Amazon and Microsoft will be the direct beneficiaries of computing growth, giving four companies overweight ratings.

Where are the opportunities for small companies?

Although small companies may be marginalized in the development of more complex large models, the development of small models will create new opportunities for them.

Morgan Stanley says small models have lower development costs and are likely to achieve significant benefits in specific industry areas in the future and promote the rapid spread of general artificial intelligence technology.

Our latest general artificial intelligence model includes a tool that calculates the data center costs associated with training small models, which we consider to be a useful starting point for assessing the rate of return (ROIC) that small models may spread in a particular domain.

We believe that the decline in the cost of small models and the improvement of capabilities have strengthened our assessment of the use of general artificial intelligence technology in many areas.

What can the big model of the future do with the blessing of software?

It is worth noting that in addition to advances in hardware such as chips, the innovation of software architecture will also play a key role in promoting the capability of large models, especially the Tree of Thoughts architecture.

The architecture, proposed by researchers at Google DeepMind and Princeton University in December 2023, is inspired by the way human consciousness works, especially the so-called "system 2" thinking. "system 2" is a long-term, highly deliberate cognitive process, as opposed to the rapid, unconscious "system 1" thinking, which is more similar to the way the current large model works.

This shift will allow large models to work in a way more similar to the human thinking process, highlighting AI's greater creativity, strategic thinking and capabilities for complex, multi-dimensional tasks.

Computing costs have dropped significantly

Morgan Stanley's proprietary data center model predicts that the rapid increase in computing power for large models means that computing costs will fall rapidly. Looking at the evolution of a single chip generation (from NVIDIA Hopper to Blackwell), computing costs have dropped by about 50%.

OpenAI CEO Sam Altman previously emphasized the importance of falling computing costs and regarded it as a key resource in the future. He believes that computing power may become the most precious commodity in the world, and is as important as currency.

In addition, the report predicts that a few very large supercomputers will be built, most likely near existing nuclear power plants.

In the United States, Morgan Stanley expects Pennsylvania and Illinois to be the best locations to develop supercomputers because these areas have multiple nuclear power plants that can support the energy needs of multi-gigawatt supercomputers.