The Entire AI Data Center Explained — From Electricity to ChatGPT
A professionally copyedited transcript of Leo Cui’s full-stack tour of AI infrastructure, from electricity and liquid cooling to GPUs, HBM, networking, software, and the financing loop behind the buildout.
Chapter Timestamps
00:00 The two-second path from a phone to an AI data center and back
01:51 Why generation costs more than search: tokens, FLOPs, training, and inference
06:44 The six-stage journey of a prompt: network, gateway, tokenization, prefill, decode, and return
10:10 Power density, grid queues, transformer shortages, and behind-the-meter generation
17:38 Why high-density AI racks require liquid cooling and create water constraints
21:17 CPUs, GPUs, NVIDIA, AMD, Broadcom, and where hardware margins accrue
26:22 Scale-up and scale-out networking: NVLink, InfiniBand, Ethernet, switches, and optics
30:24 Why inference is memory-bandwidth-bound and how HBM and storage capture demand
33:47 CUDA, serving engines, batching, caching, quantization, and enterprise RAG
36:41 Hyperscalers, neoclouds, AI labs, financing loops, depreciation, and end-user revenue
39:14 The full stack revisited and the unresolved return on the AI buildout
Made with: The Transcript Desk Chrome Extension
Full video:
What happens in the two seconds between asking ChatGPT a question and receiving an answer? Leo Cui follows one request through the physical and financial machinery of an AI data center, showing how token generation depends on power plants, grid equipment, liquid cooling, accelerators, high-bandwidth memory, optical networking, storage, serving software, and a still-unresolved chain of capital spending.
Transcript
00:02-04:04
Leo Cui:
Last night, sometime around 7:00 p.m., you pulled out your phone and typed a question. Maybe it was, “What should I make for dinner with chicken and rice?”
About two seconds later, a machine wrote you an answer. Two seconds.
That’s what I want to do in this video: slow those two seconds down—way down.
Because in those two seconds, your question left your phone, traveled hundreds of miles through strands of glass thinner than a human hair, and arrived at a building the size of several football fields—a building that uses as much electricity as a small city.
Inside that building, your question passed through a machine that costs as much as a house, was translated into pure math, and processed by chips running so hot they have to be liquid-cooled like race car engines. Then the answer came back to you, letter by letter, before you even had time to lower your thumb.
And here’s the part that should get your attention as an investor: to make those two seconds possible, the largest companies on Earth are spending roughly $725 billion on infrastructure this year alone.
And that’s just four companies: Amazon, Microsoft, Google, and Meta. That’s more in one year than the inflation-adjusted cost of the entire U.S. interstate highway system.
Goldman Sachs projects a total buildout of $7.6 trillion between 2026 and 2031. Jensen Huang, the CEO of NVIDIA, stood onstage at Davos in January and called it, in his own words, “the largest infrastructure buildout in human history.”
So the question this whole video hangs on is simple: Where does all the money actually go?
By the end of this video, you’ll be able to answer that. You’ll understand every step your question takes: the power plants, cooling systems, chips, memory, fiber optics, and software. You’ll know which companies sit at every step, which companies are printing money, which companies are just telling stories, and where the whole thing could crack.
I’m Leo.
This is for educational purposes only, not financial advice.
Before we trace your question across the country, we need to answer something more basic. The internet has existed for 30 years, and Google has answered trillions of questions. Why did no one need to spend three-quarters of a trillion dollars a year until now? What changed?
The answer comes down to the difference between two people: a librarian and a writer.
Google is a librarian. When you search for a chicken-and-rice recipe, Google doesn’t cook anything. It walks into a giant library it has already organized. It indexed the entire internet years ago and keeps updating it, then hands you pages that already exist. The expensive work happened in advance. Answering your question is just a lookup: fast, cheap, done. A Google search costs a fraction of a cent.
GPT is a writer. When you ask it the same question, there’s no answer sitting on a shelf. There’s no database entry saying, “Here’s what to tell this person.” The model composes your answer from scratch, one word at a time, every single time. Even if a million people ask the same question today, it doesn’t retrieve an answer—it generates one. And generation is expensive.
A single ChatGPT query can require 10 to 100 times more computing power than a Google search. Now multiply that by 900 million weekly users, and that’s the entire reason this video exists.
Search retrieves. AI generates. And generation is a manufacturing process.
That brings me to the analogy I’ll use for the rest of this video: think of an AI data center as a factory—a very strange factory. Raw material goes in one side: electricity. A product comes out the other end: words.
Like any factory, it has departments: a power plant, a cooling system, assembly lines, and a shipping department. We’re going to tour each one. But first, there are three terms you need to know.
Term one: the token. A token is the product this factory makes. Language models don’t actually read words; they read tokens, which are chunks of text roughly three-quarters of a word each. “Chicken and rice” is about four tokens. Your question gets chopped into tokens on the way in, and the answer gets manufactured token by token on the way out.
Here’s why investors care: tokens are the unit of revenue in the AI economy. OpenAI and Anthropic literally price their products per million tokens. When you hear “token,” think of a widget coming off the assembly line.
Term two: the FLOP.
04:04-06:45
Leo Cui:
A FLOP is one floating-point operation—one single arithmetic calculation, one multiplication or one addition. It’s a unit of labor in this factory.
Manufacturing a single token requires a model to do hundreds of billions of these calculations—not per answer, but per word. When people say a chip does a thousand trillion FLOPs per second, they’re telling you how many workers that chip has on the factory floor.
Term three—and this is a big one—training versus inference.
Training is building the factory. You take a model—think of it as a machine with over a hundred billion adjustable knobs, called parameters—and show it a huge portion of the internet, adjusting those knobs until it gets good at predicting language.
This takes months, tens of thousands of chips running around the clock, and on the order of a hundred million dollars or more per frontier model. It happens once per model.
Inference is running the factory. Every time you ask GPT anything, that’s inference: the trained model manufacturing an answer for you.
And here’s the misconception I most want to kill in this video: people assume training is where the money goes. Wrong. By 2026, roughly two-thirds of all AI compute will be inference, because training happens once, but inference happens billions of times a day. OpenAI’s inference bill alone is projected to be around $14 billion this year.
The factory was expensive to build. It’s even more expensive to run.
Okay, so why did all of this suddenly explode after 2022? One discovery—the most economically important discovery of this decade is something most people have never heard of: scaling laws. Around 2020, researchers at OpenAI found something almost embarrassingly simple. If you make a model bigger, give it more data, and use more compute, it gets smarter—not sometimes, but predictably.
On a chart, it is nearly a straight line. If you spend 10 times more, you reliably get a better model. Consider what that means for CEOs. For 50 years, better software meant hiring smarter programmers. Scaling laws turned intelligence into something you could purchase. They converted AI from a research problem into a capital-expenditure problem.
Big companies know exactly how to compete on capital expenditures: outspend everyone. That is the moment software stopped being about code and started being about concrete. That is why we suddenly need factories.
Here is the closing thought for this act: for the entire history of Silicon Valley, software escaped the physical world—zero marginal cost, infinite copies, no factory required. AI reversed that. The frontier of software is now poured in concrete, measured in megawatts, and cooled with water.
Every additional smart answer requires physical machines, physical electricity, and physical heat removal. Software became heavy industry, and that changes who makes money.
06:45-09:16
Leo Cui:
Let’s unfreeze your question. It is 7 p.m. You type, “What should I make for dinner with chicken and rice?” and hit send.
Here is the complete journey, with no steps skipped. Step one: the trip. Your question leaves your phone as radio waves, hitting a cell tower or your Wi-Fi router. Within a few miles, it becomes pulses of light inside fiber-optic cable—glass strands carrying data at two-thirds the speed of light. It is routed to the nearest entry point of the AI company’s network, then often travels hundreds of miles to a data center.
Your question travels hundreds of miles to a data center. Total time so far: a few hundredths of a second.
Step two: the front door. Your question arrives at the API gateway. Think of it as a factory receiving desk. It verifies who you are, makes sure you’re not sending a thousand requests per second, runs a safety check, and assembles everything the model needs: the system instructions, your past conversation, and your new question.
Step three: tokenization. That full task gets broken into tokens—the puzzle pieces from Act One. Your dinner question, plus the context, might be a few hundred tokens. These are converted into numbers, because from here on, everything is math.
Step four: prefill. The model reads. Here’s something almost nobody knows: the model processes your question in two totally different phases. The first is called prefill. The model reads your entire prompt all at once, in parallel, and builds an internal understanding of it.
This is a burst of raw computation—billions of calculations—and it produces something called the KV cache. Don’t let the name scare you. The KV cache is simply the model’s working memory of your conversation: everything said so far, held in super-fast memory right next to the chip.
Ever notice GPT pauses briefly before the first word appears? That pause is prefill. The factory is reading the work order.
Step five: decode. Now the assembly line starts. The model generates the answer one token at a time. It looks at your prompt plus everything it has written so far, runs the entire trillion-parameter network—hundreds of billions of calculations—and produces one word.
Then it does the whole thing again for the next word, and again, one token at a time. Every single word of every ChatGPT answer on Earth is manufactured this way: one at a time, with a full network pass each time.
When you watch the answer type itself onto your screen, that’s not a design flourish. You are literally watching the assembly line run in real time. Each word appears the moment it’s generated.
Each word appears the moment it’s generated. Step six: the trip home. Each token travels back through the same fiber, and two seconds after you hit Send, you’re reading dinner ideas.
09:16-14:43
Leo Cui:
One more thing is happening behind the curtain: you are not alone in there. The factory batches everything you send with hundreds of other people’s questions on the same chip simultaneously, like a delivery driver grouping orders on one route. That batching is the difference between your question costing cents and costing dollars.
Now zoom all the way out, because here’s the whole factory in layers. This is a map for the rest of the video: ten layers.
Here’s the one-sentence version of this entire video: electricity comes in one side, flows through silicon, and becomes computation and heat. The heat is carried away by water. The computation is coordinated by light. And what comes out the door is words. Electrons in, tokens out. That’s the factory.
So, let’s start a tour where every factory tour starts: the power plant. This surprised me most when I first dug into this ecosystem: the story of AI in 2026 is no longer mainly about chips. It’s a story about electricity.
Let me give you the number that framed this whole industry for me. A traditional server rack—the kind that has run the internet for the last 20 years—draws about five to ten kilowatts. Think of a kilowatt as ten old-fashioned 100-watt light bulbs burning at once.
NVIDIA’s flagship AI rack—one refrigerator-sized cabinet—draws 120 kilowatts.
And the next generation, coming later this year, the Rubin racks, is projected to approach 600 kilowatts per rack. That’s a 60- to 100-times jump in power density in under a decade—the electrical demand of an entire neighborhood packed into a phone booth.
This is called power density, and it’s the root cause of nearly everything in the next two acts. Now scale up: a large AI campus today wants a gigawatt or more. A gigawatt is a thousand megawatts.
A gigawatt is a thousand megawatts—roughly the output of a full-size nuclear reactor, enough electricity for about a million homes. Individual companies are now planning multiple campuses of that size.
Data centers consumed about 4% to 5% of U.S. electricity before this boom. Credible projections put that figure at 9% to 17% by 2030.
And here’s the collision: the U.S. electrical grid was built brilliantly decades ago for demand that grew 1% or 2% a year. AI showed up asking for tens of gigawatts. Right now, the grid physically cannot say yes.
There are two bottlenecks, and they are the most important numbers in this story.
Number one: the interconnection queue. To plug a large new facility into the grid, you file a request and wait in line while utilities study whether the grid can handle you. That line is currently four to five years long. This April, about 410 gigawatts of large projects were waiting to connect. Eighty-seven percent of them were data centers. That’s nearly five times the entire Texas grid’s peak demand waiting in line.
Number two: transformers. A large power transformer—the giant grid box that steps voltage down—used to take about a year to order. Today, it takes two and a half to four years, with prices up nearly 80%. You can have your chip in six months; the grid box that powers it might not arrive until 2029.
So what do you do if you’re Microsoft or Meta, and every month of waiting costs you ground in the AI race? You stop waiting for the grid. You go around it.
The industry calls this “behind-the-meter power”: generating electricity on-site or next door so you never have to enter the public queue. That decision—made simultaneously by thousands of companies—is what lit a fire under an entire forgotten sector of the stock market: boring old industrial power companies.
Let me introduce the players, from the most dramatic to the most dependable.
First, the nuclear resurrection. In 2024, Microsoft signed a deal that would have sounded like satire a decade ago: a 20-year agreement with Constellation Energy to restart the undamaged reactor at Three Mile Island, next to the one involved in the 1979 accident. Constellation is spending about $1.6 billion, backed by a $1 billion federal loan, to bring the 835-megawatt unit back online in the second half of 2027. Microsoft will buy every megawatt it produces for 20 years.
Constellation operates the largest nuclear fleet in America—about 22 gigawatts—and suddenly, those aging reactors became some of the most valuable energy assets on Earth. Why? Because AI factories run 24/7, and nuclear is the only carbon-free power source that also runs 24/7.
Constellation’s stock tells the story. It’s now roughly a $90 billion company. Its peer, Vistra, with a 37-gigawatt fleet mixing nuclear and gas, rode the same wave.
The moat here is beautifully simple: you cannot build a new conventional nuclear plant in America this decade. Existing reactors are irreplaceable.
Small modular reactors are lottery tickets. You’ve heard the tickers: Oklo, backed by Sam Altman, with over 14 gigawatts of signed pipeline; NuScale, the only SMR design actually certified by U.S. regulators.
Here’s my skeptical analytical framing, and I’ll be blunt: these are pre-revenue companies whose first commercial electron arrives around 2030 at the earliest. Oklo doesn’t yet have final regulatory approval for its design. NuScale booked about $31 million in revenue against a $356 million loss. Both stocks are down 65% to 78% from their late-2025 peaks. That is not a business yet. It’s an option on the 2030s.
Next: the fastest power in the West. If the grid takes four years and nuclear takes 10, what can you get in 12 to 18 months? Fuel cells. Bloom Energy makes solid-oxide fuel cells—boxes that convert natural gas into electricity chemically, with no combustion. You can park them behind the meter, next to a data center, fast.
14:43-17:28
Leo Cui:
In the first half of this year, Bloom announced $7.65 billion in data center contracts in a single night. But this July, Hindenburg, an investigative outlet whose affiliated funds short the stocks it covers, so weigh the source accordingly, published a report saying that Bloom’s marketed $20 billion backlog is more than 40 times its binding contract obligations, versus roughly 2 times for typical peers.
It also said that scaling to meet its data center ambitions would consume nearly the entire global supply of scandium, a metal China now requires an export license for. Bloom formally rejected the claims as false and misleading. But when a company’s stated backlog and its SEC filings differ by 40 times, the burden of proof is on the company.
Next: the arms dealer set to dominate through 2030. My favorite business in this story is the least glamorous: GE Vernova, the power spinoff of General Electric. They make the giant gas turbines that are realistically the number-one near-term power source for AI, because gas is the only thing you can build at scale before 2030.
GE Vernova’s turbine slots are sold out through the end of this decade. Its backlog is around $163 billion. In the first quarter of 2026 alone, it booked $2.4 billion in data center electrification orders—more than in all of 2025. The stock has risen so much that it is now nearly a $300 billion company.
The only real global rival at scale is Siemens Energy. Between the substation and the chips is a layer of equipment most people never think about: switchgear, busways, and uninterruptible power supplies—essentially giant batteries that catch the load instantly if the grid blinks, because even a half-second outage can ruin a training run that has been running for a month.
Three companies dominate this layer, and remember their names, because two of them show up again in the next act: Vertiv, Schneider Electric, and Eaton.
Eaton’s electrical backlog grew 48% year over year. Vertiv’s backlog more than doubled to $15 billion, and the company joined the S&P 500 in March. These are the companies selling shovels to every miner, regardless of who wins. And the last line of defense is rows of backup generators from Caterpillar and Cummins—diesel engines, like the kind you see on school buses, idling as they wait for the one hour a year the grid fails. Analysts think Caterpillar’s data-center generator business could triple by 2030.
Before we move on, here’s the uncomfortable part, because this is showing up in your inbox: data centers bid for scarce power. They bid against you. In the PJM market—the grid covering 13 states from Illinois to Virginia—data-center demand added more than $9 billion to the latest capacity auction, translating to residential bills rising by $16 to $18 a month in parts of Ohio and Maryland. Communities are noticing. Moratoriums are being proposed. This is becoming a genuine political risk to the buildout, and any honest map of this industry has to include it.
17:28-20:22
Leo Cui:
So the factory has power. One hundred kilowatts are now flowing into a single rack of chips, creating an immediate problem: Physics 101. Every one of those watts becomes heat. The factory is running a fever. A single flagship AI chip today dissipates more than 1,000 watts of heat—a chip the size of a postcard putting out the heat of a full-size space heater. Now stack 72 of them into one rack, plus their memory and networking, and you’ve got 120 kilowatts of heat: the output of about 80 space heaters in a cabinet you could hug.
Why? Because computation is heat. Every one of those trillions of calculations pushes electrons through microscopic wires, and electrical resistance turns into warmth. The factory’s raw material—electricity—doesn’t get consumed making tokens. It gets converted almost entirely into heat. Cooling isn’t a support function of AI data centers; cooling is half the job. For 30 years, the answer was air conditioning—genuinely, just fancy AC. Cold air pushed up through the floor, hot air sucked out the back, giant chillers and cooling towers on the roof.
Giant chillers and cooling towers sit on the roof. Air cooling worked fine up to about 30 to 50 kilowatts per rack, but we’ve permanently crossed that threshold. Air simply cannot carry heat away fast enough from a 120-kilowatt rack. You would need hurricane-force winds blowing through the servers. So, the industry is undergoing its biggest plumbing change in history.
The shift is from air to liquid. Water carries heat roughly 3,000 times more effectively than air per unit volume. Here is the technology ladder in one breath: rear-door heat exchangers—a water-cooled radiator bolted to the back of a rack—are a transitional solution. Direct-to-chip cooling is expected to become mainstream in 2026: metal plates with liquid channels sit directly on top of each chip, connected by hoses to a CDU, or coolant distribution unit. Think of it as the rack’s heart, pumping coolant to every chip and carrying heat to the building’s water loop.
NVIDIA’s flagship racks do not offer this as an option; they require it. At the extreme end is immersion cooling: literally dunking entire servers into tanks of nonconductive fluid, like deep-frying a computer that never burns.
Two quick vocabulary items investors will encounter: PUE, or power usage effectiveness, is a facility-efficiency score. It is total power coming in divided by the power that actually reaches the computers. A perfect score is 1.0. Older data centers can run around 2.0, while modern liquid-cooled facilities can reach 1.1. That efficiency gap, multiplied by a gigawatt and electricity prices, is real money.
Then there is water. Many data centers cool themselves by evaporating millions of gallons, which is becoming a genuine permitting and political fight in dry regions. Closed-loop liquid systems help, but keep an eye on this issue—it will determine where facilities get built.
Who gets paid? Largely the same names as in the power room. Vertiv is the market leader, and one of the rare companies selling both power equipment and liquid cooling: a one-stop shop growing revenue 28% annually at a 20% margin. And then something remarkable happened.
20:22-24:12
Leo Cui:
Then something remarkable happened: two electrical giants each spent billions to buy their way into liquid cooling within months of each other. Eaton paid about $9.5 billion for Boyd Thermal. Schneider Electric bought Motivair. When the industry’s most disciplined acquirers both pay up for the same niche, they’re telling you what they think every future data center will look like.
Smaller pure-play companies are innovating in cooling loops and enclosures, including privately held CoolIT Systems, the specialist whose cold plates ship inside many brand-name servers. The liquid-cooling market was about $5 billion in 2025. Forecasts put it at $15 billion to $27 billion by the early 2030s. It’s the clearest picks-and-shovels growth opportunity in this entire ecosystem, because it doesn’t matter whether NVIDIA, AMD, or Google wins. Heat is heat.
All right, the factory has power. The fever is under control. It’s time to walk onto the factory floor and meet the machine your dinner query actually runs on—and the $3 trillion company that built it.
This is the machine your dinner query runs through: NVIDIA’s GB200 NVL72. Seventy-two GPUs wired together so tightly that they behave as a single giant computer. It weighs about a ton and a half, draws the 120 kilowatts we discussed, and costs roughly $3 million. So let’s open it up.
To keep the parts straight, come back to the factory—specifically, the kitchen. The CPU is the head chef. The central processing unit runs the operating system, takes orders, and coordinates everything. It’s brilliant at complex, sequential tasks, but there are only a handful of them. For decades, the CPU was the star of computing—Intel’s kingdom. In an AI server, it has been demoted to management.
The GPUs are 10,000 line cooks. A graphics processing unit, originally invented to draw video-game graphics, contains thousands of small, simple cores that all perform the same operation simultaneously. It turns out the math inside a neural network is exactly that kind of work: billions of identical multiply-and-add operations. One head chef cannot do that. But 10,000 line cooks, each chopping one onion at the same time, can.
That accident of history—gaming graphics and AI needing the same math—is the foundation of NVIDIA’s empire. HBM, or high-bandwidth memory, is the countertop. Hold that thought; it gets its own act. The SSD is the pantry. The NIC, or network interface card, is a waiter carrying dishes between kitchens. And the power supplies and motherboard are the plumbing and wiring holding it all together.
Now, the company: NVIDIA finished its last fiscal year with $215.9 billion in revenue, up 65%, of which about $194 billion came from data centers. It controls roughly 80% to 86% of the AI accelerator market. Its gross margin in the most recent quarter was about 75%—75% on hardware. Apple, the most admired hardware company in history, runs around 46%.
NVIDIA became the first $5 trillion company last October. Depending on the week, roughly seven cents of every dollar in the S&P 500 is NVIDIA.
How is that margin possible? Everyone says, “Best chips.” And sure—but the real answer is a word we’ll unpack fully in Act Eight: CUDA. It represents 20 years of software that every AI developer on Earth was trained on.
For now, the one-line version is this: NVIDIA doesn’t just sell chips. It sells the only complete factory-floor system the world’s engineers already know how to operate. Buying a competitor’s chip means retraining your entire workforce.
The bear case is that about 40% of NVIDIA’s revenue comes from just four customers, and all four are building their own chips to replace it.
Next, the challenger: AMD. AMD’s Instinct GPUs are genuinely competitive for inference, with more memory per chip and, by some estimates, 25% to 40% better tokens per dollar. Their problem was never silicon; it was software. Their CUDA alternative, called ROCm, now reaches 90% to 95% of NVIDIA’s performance on standard workloads. But being 90% as good with more friction is a hard sell.
24:12-28:09
Leo Cui:
A product that’s 90% as good but comes with more friction is a tough sell when training a model can cost $100 million. AMD holds maybe 5% to 7% of this market—worth watching, but still a distant second. Intel, painful as it is to say, is barely in this race. It still sells plenty of CPUs, but the kitchen is no longer about head chefs.
Now for the quiet assassin: Broadcom. Here’s the plot twist most retail investors miss: the four hyperscalers are building their own chips, but they can’t actually do it alone. Designing a frontier AI chip takes a decade of specialized IP. Broadcom helped build Google’s TPU, Meta’s training chip, and reportedly OpenAI’s chip as well.
Broadcom controls more than 60% of the custom-chip market. Its AI revenue rose 106% last quarter, with a $73 billion backlog, and management says it has line of sight to $100 billion in AI revenue by 2027.
In our factory analogy, NVIDIA sells finished kitchens. Broadcom helps the biggest restaurant chains build their own—and takes a cut either way. It also dominates switching silicon and networking. One company sits on both sides of the war.
Now, who actually builds those racks? Not NVIDIA. NVIDIA designs them. Supermicro integrates full liquid-cooled racks faster than anyone. Revenue was up 123% last quarter, with more than 90% coming from AI—but gross margins are only 6% to 10%, depending on the quarter.
Dell has taken more than $64 billion in AI server orders, with a $43 billion backlog—an astonishing figure—while server-segment margins remain below 9%. HPE is focused on supercomputing and sovereign-AI deals.
And beneath the brand names are the true invisible giants: the Taiwanese ODMs, or original design manufacturers. Foxconn assembles roughly 40% of the world’s AI racks, alongside Quanta, Inventec, and Wistron. The same rack passes through many hands.
NVIDIA captures roughly 75 points of margin on the silicon. The company that physically screws it all together keeps 6% to 10%. In hardware, profits live wherever there’s scarcity. Chips and software are scarce. Assembly is not.
But I’ve been hiding something from you: 72 GPUs can behave like a single computer. Nobody achieved that with a magic wand. Making 10,000 line cooks work as one brain is arguably the hardest engineering problem in the entire building, and it’s where some of the best businesses in the ecosystem are hiding.
So why can’t one GPU do the job? Simple: the model doesn’t fit. A frontier model has over one trillion parameters—those adjustable knobs—requiring terabytes of ultra-fast memory. The biggest GPU carries only a few hundred gigabytes, so the model gets sliced across thousands of chips.
And here’s the consequence: to produce every single token, those chips must constantly exchange intermediate results at unimaginable speed. Back to the kitchen: 10,000 line cooks preparing one dish together. Every chef needs ingredients from other chefs every second. If passing ingredients is slow, your 10,000 chefs stand around waiting—and these are the most expensive chefs in history.
At cluster scale, a network that’s 10% slower can leave billions of dollars’ worth of silicon idle. That’s why networking accounts for roughly 40% to 60% of spending for every dollar spent on GPUs.
Two terms to define: bandwidth is how much data moves per second—the width of the conveyor belt. Latency is the delay for one handoff—how long a single pass takes. AI needs both, everywhere, all at once.
The wiring comes in two flavors. Scale-up, inside the rack, is NVIDIA’s proprietary NVLink: an extremely fast web that makes 72 GPUs operate as one machine. Scale-out, from rack to rack across the building, is where a war is being fought.
For years, serious AI clusters ran on InfiniBand, a specialized, ultra-low-latency networking technology that NVIDIA acquired in 2019 with Mellanox. Premium performance, premium price, one vendor.
Competing against it is Ethernet, the open, universal standard—the same family of technology as your home network. Historically slower, but backed by literally everyone who isn’t NVIDIA. By early 2026, about two-thirds of new AI cluster networking is Ethernet. Open standards usually win over time. They just do.
28:09-31:19
Leo Cui:
This is a proprietary-versus-open story as old as tech. Who profits from the network? Broadcom, again. Its Tomahawk chips are the merchant silicon inside most high-end Ethernet switches.
Arista Networks builds the switches themselves—the best-in-class boxes and software. Hyperscalers standardize on them. It has roughly 65% gross margins and is guiding to $11.5 billion this year, with the risk that its two biggest customers account for 40% of revenue.
Cisco, the incumbent, is still huge and fighting to stay relevant in AI back ends. Marvell follows the Broadcom playbook one tier down: custom chips for Amazon and Microsoft, plus leadership in the digital signal processors inside optical modules.
Astera Labs has one of the most remarkable margin stories in the ecosystem. It makes tiny retimer chips that clean up electrical signals degrading over mere inches of circuit board at these speeds. Boring, invisible, 76% gross margins, and revenue was up 93% last quarter.
When data moves this fast, even the space between two chips becomes a market. Then the plot literally turns to light.
Copper wires can carry these speeds for only a few meters before signals degrade—fine inside a rack, useless across a football-field-sized building. So, between racks, everything converts to light and travels through glass fiber.
The device doing the conversion is the optical transceiver: a thumb-sized gadget that translates electricity into light. You need one at each end of every fiber link.
A single large AI cluster consumes hundreds of thousands of them, at hundreds of dollars each, and they are replaced every upgrade cycle. It’s a razor blade for data centers.
The names are Coherent, the market leader; Lumentum, the Chinese volume champion; Innolight; Fabrinet, the contract manufacturer that assembles for nearly all of them; the arms dealer’s arms dealer, Corning, which draws the glass fiber itself; and Amphenol. The glass fiber itself and the connectors are the knuckles of the entire system. The frontier to watch is co-packaged optics, which moves the light conversion directly onto the switch chip to slash power consumption.
The nervous system is built from 10,000 chips thinking as one. But there’s a dirty secret on the factory floor: most of the time, the most expensive chips in the world are waiting—not for data from across the room, but for data from two centimeters away.
Here’s the secret: during the decode phase—the one-word-at-a-time assembly line from Act Two—the GPU’s massive cores often aren’t the bottleneck. For every token, the chip has to pull the model’s parameters and the conversation’s working memory, the KV cache, from memory into its cores.
Data moves from memory into the cores. The math is fast; fetching the data is slow. Modern inference is what engineers call memory-bandwidth-bound.
The line cooks are lightning-fast, but the countertop cannot feed them ingredients quickly enough. That makes the countertop one of the most valuable pieces of real estate in technology.
The industry’s answer is HBM, or high-bandwidth memory. Instead of laying memory chips flat on a board a few inches from the processor, HBM stacks them vertically—eight to twelve stories high—drills thousands of microscopic elevator shafts through the silicon, and places the whole tower directly next to the GPU in the same package: a skyscraper of memory downtown instead of suburbs of memory across the highway.
31:19-33:57
Leo Cui:
The result is five to six times the bandwidth of conventional memory, at five to six times the price. NVIDIA happily pays.
Memory is now one of the biggest cost components inside every AI chip you’ve heard of, and only three companies on Earth can make it.
SK Hynix, the Korean company, owns roughly 60% of the HBM market and got there by out-executing its giant neighbor. It bet on HBM years before it mattered, shipped each generation first, and locked in the lion’s share of NVIDIA’s next-generation allocation.
Samsung, the largest memory company overall, was embarrassingly late. Micron, the American champion, went from an afterthought to selling out its entire 2026 HBM capacity more than a year in advance. In May 2026, all three memory makers crossed $1 trillion in market value, combining for more than $4 trillion—roughly 16 times their value a decade earlier.
Memory used to be the most brutal commodity business in tech: boom, bust, bankruptcy, repeat. HBM changed the psychology. It was allocated like a scarce resource and priced like a luxury good.
The open question is whether those prices will hold once all three giants complete their capacity expansions at the same time. Memory cycles have broken hearts before.
Now, walk out the back of the factory to the warehouse: storage, the hierarchy in one line.
Storage hierarchy in one line: the closer to the chip, the faster—and more expensive. Cache is on the chip itself; HBM sits beside it; regular DRAM is on the motherboard; then come SSDs and flash drives for hot data; and at the bottom is the technology everyone declared dead 10 years ago: the spinning hard drive, still unbeatable per terabyte for cold, bulk data.
AI turned out to be a huge driver of storage demand: training datasets, model checkpoints saved every few hours, and the part nobody predicted—the output. Every conversation, every log, every generated image, retained forever.
Seagate’s CEO calls it the “inference inflection.” AI doesn’t just consume data; it produces it endlessly. Hard drives are a literal duopoly: Seagate and Western Digital, plus flash-drive players like Kioxia.
Digital flash-storage players Kioxia and Solidigm, after a decade of decline, sold out their entire production through 2027. Western Digital now ships 89% of its revenue to cloud customers and was one of the S&P 500’s top performers. Consumer hard-drive prices jumped 50% because AI consumed the supply.
A dying industry was resurrected by the factory next door needing somewhere to store infinity. The machine is complete: powered, cooled, wired, and fed. But a pile of perfect hardware answers exactly zero questions. Something invisible has to run the place.
Everything we’ve toured so far, you could theoretically buy. The hardware is purchasable. Where does a durable competitive advantage actually live? The stack, briefly, from bottom to top.
33:57-37:11
Leo Cui:
Let’s briefly walk through the stack from the bottom up.
Linux is the free operating system that runs effectively every server on Earth, commercialized by Red Hat and Canonical.
Kubernetes is the invisible foreman: open-source software that schedules work across thousands of machines, restarts what crashes, and keeps the factory floor humming.
Then come the serving layer and the models.
Two stories in this act matter to investors more than all the rest combined. The first is CUDA: the 20-year trap.
In 2006, NVIDIA made a decision Wall Street hated. It spent billions building a programming platform so scientists could use gaming chips for general-purpose computation. For a decade, it looked like an expensive hobby. Then deep learning arrived, and every AI researcher on Earth learned to build on CUDA because it was the only mature option. Twenty years later, CUDA has millions of developers, thousands of specialized libraries, and every framework is optimized for it first.
Understand what this means: when AMD ships a chip with better specs—and sometimes it does—the customer isn’t comparing chips. They’re comparing the chip against the cost of retraining their entire engineering organization and rewriting their code, with a $100 million training run on the line.
That’s why 70% gross margins have survived competition. The moat was never the silicon. The moat is the muscle memory of a million developers.
The muscle memory of a million engineers.
Second, why does your question cost cents rather than dollars? Raw, naïve inference on a trillion-parameter model would be incredibly expensive. The economics only work because of an unglamorous software layer called a serving engine, like vLLM and NVIDIA’s TensorRT, doing three tricks: batching—grouping hundreds of users and processing them through the chip at once; the delivery-route trick from Act Two; caching—reusing the KV working memory instead of recomputing the conversation from scratch for every word; and quantization—rounding the model’s numbers to lower precision, like saving a slightly compressed photo with nearly identical quality at a fraction of the cost. Together, these deliver a three- to tenfold cost reduction. When OpenAI or Anthropic cuts API prices by 80% in a year, most of that comes from this layer, not new chips. Token manufacturing costs are collapsing along the curve. Remember that for the finale, because costs are declining across the board.
One last layer of the stack: your dinner question does not need it, but enterprise AI does. RAG—retrieval-augmented generation. The model is a brilliant writer with a fixed education. RAG hands it your company’s documents at question time through vector databases—search engines that find text by meaning rather than keywords—from players like Pinecone, along with data platforms such as Databricks and Snowflake. It is the librarian and the writer working together. That is the enterprise AI pitch.
Altogether, that’s the enterprise AI pitch in one sentence. And with that, the tour is over. You’ve seen every layer: power, cooling, silicon, light, memory, and software.
Now let’s follow the money—all of it on one map—and ask one question: Does any of this actually pay for itself?
Here’s the whole board. On the demand side are the miners, in three tiers: the hyperscalers—Microsoft, Amazon, Google, Meta, and Oracle—spending a combined $725 billion this year; the new clouds, specialized GPU landlords like CoreWeave, Nebius, Lambda, and Crusoe; and the AI labs—OpenAI, now valued at around $850 billion; Anthropic, which passed it at roughly $965 billion, with about $47 billion in annualized revenue; and xAI, folded into SpaceX. Both major labs have filed for IPOs.
37:11-39:45
Leo Cui:
SpaceX. Both major labs filed for IPO this June. The private market has already priced them as two of the most valuable companies on Earth. The public market is about to vote on the cost.
Bernstein estimates that one gigawatt of AI data-center capacity costs about $35 billion to build. Roughly 39% of that goes to chips—the single largest expense—which is why NVIDIA’s gross profit alone is estimated at nearly 30% of total industry cost.
And that $35 billion machine depreciates quickly. Most operators write chips off over four to six years, and skeptics argue that even this flatters the accounting, since a five-year-old GPU competes against chips that are ten times better.
Every year of depreciation adds or removes billions in reported profit. Watch that debate.
So follow this: NVIDIA invests billions directly into CoreWeave, Nebius, and OpenAI. Those companies use that money to buy NVIDIA chips—revenue for NVIDIA.
The hyperscalers sign enormous contracts with the new clouds. Meta alone committed roughly $21 billion to CoreWeave and reportedly $27 billion to Nebius, which conveniently moves Meta’s data center spending off its balance sheet and into someone else’s debt.
The AI labs sign compute deals with hyperscalers: OpenAI with Microsoft and Oracle; Anthropic with Amazon and Google—paid partly with money those same hyperscalers invested in them. To be fair, the “analytical skeptic” label cuts both ways. This isn’t fraud, and it isn’t new. Vendor financing built the railroads and the telephone network.
The loop has exactly one opening where fresh money is supposed to enter: users and enterprises paying 20 bucks a month, and companies paying for APIs and copilots. That is the only exit that isn’t recycled capital.
And today, that corner is the smallest number on the board. The two biggest labs’ combined annualized run rates top 70 billion. That’s genuinely spectacular—the fastest revenue ramp in software history.
But the revenue they actually booked this calendar year is a fraction of that, set against 725 billion in annual spending, with OpenAI still expected to lose on the order of $14 billion this year. The entire $7 trillion machine is a bet that a small corner grows faster than the big circle spins.
So let’s watch it one more time. In the same two seconds, your thumb hits “send,” and the question becomes light in glass fiber crossing state lines. It arrives at a building drawing the power of a city—power from a restarted nuclear plant, a sold-out turbine, or a fuel cell parked behind the meter.
It’s chopped into tokens and fed into a $3 million rack assembled in Taiwan, sold at a 75-point margin, where 10,000 line cooks fetch a trillion parameters from memory skyscrapers while liquid coolant carries away the heat of 80 space heaters.
39:45-40:16
Leo Cui:
It carries away the heat of 80 space heaters, and light-speed interconnects let 10,000 chips think one thought. The software batches your request with 1,000 others, and the answer streams back token by token, each word manufactured in two seconds. A $700 billion-a-year infrastructure project—the largest our species has ever attempted—so a machine can suggest a one-pan chicken-and-rice recipe.
Whether that’s the most important investment in history or the most expensive, honestly, nobody on Earth knows yet. That’s the whole map. If you found this video helpful, please subscribe, and I’ll see you in the next one.
Made with: The Transcript Desk Chrome Extension