Naveen RaoVP of Generative AI & Former CEO of MosaicML

Training foundation models represents a temporary upfront capital expenditure, whereas inference is the continuous runtime execution of intelligence across every industry. As AI agents execute perpetual reasoning loops, summarize data, generate code, and control physical machines, inference becomes the universal operating cost of global economic productivity.

For fifteen years, software companies enjoyed eighty to ninety percent gross margins because marginal compute and hosting costs were virtually zero. Generative AI fundamentally destroys zero-marginal-cost software economics because every user action, API call, and reasoning token consumes expensive GPU compute and electricity. Software vendors that cannot capture immense enterprise workflow value will watch their margins evaporate.

The ultimate constraint is electrical power and transmission infrastructure rather than silicon supply. Training gigawatt-scale clusters requires immense baseload energy that conventional regional power grids cannot deliver within reasonable timeframes. Securing dedicated clean energy, nuclear power, and advanced liquid cooling is now the defining competitive battleground.

Reinforcement learning allows models to explore reasoning paths, verify intermediate hypotheses, and correct their own errors before producing an answer. Rather than merely predicting the next token based on human text statistics, RL teaches models to discover novel problem-solving strategies through autonomous trial and error.

While software automation and coding assistants are undeniably useful, digital knowledge work represents only a fraction of worldwide economic activity. The vast majority of global GDP is tied to physical labor, agriculture, heavy manufacturing, logistics, and supply chain movement. Bringing generalized spatial intelligence to physical robots will unlock economic value that completely dwarfs the software industry.

Many executives mistakenly believe they must train a massive trillion-parameter model from scratch to maintain competitiveness. In reality, modern enterprise defensibility lies in curating proprietary high-quality corporate data, fine-tuning efficient domain-specific architectures, and integrating models deeply into existing enterprise data pipelines.

NVIDIA will continue to dominate frontier training and flexible research workloads because the CUDA software ecosystem has an insurmountable compounding developer moat. However, high-volume repetitive inference workloads will increasingly shift toward custom ASICs and hyperscaler silicon optimized specifically for cost per token and energy efficiency.

Models are commodities without data, and data is useless without models. Databricks held the enterprise gold standard for unified data warehousing and governance, while MosaicML possessed the premier platform for efficient foundation model training. Combining data intelligence with algorithmic training created an enduring full-stack enterprise advantage that standalone tooling could not match.

Autoregressive language models alone certainly have fundamental limitations regarding planning, world modeling, and physical grounding. However, augmenting transformers with reinforcement learning, external verifiers, memory systems, and test-time compute scaling bridges much of the gap between passive pattern matching and active rational deliberation.

Enterprise customers cannot tolerate non-deterministic errors in regulatory compliance, financial reporting, or healthcare. The solution requires deterministic verification wrappers, retrieval-augmented grounding against authoritative databases, and consensus voting mechanisms across multiple diverse model architectures.

Software engineers will evolve from manual syntax typists into architectural orchestrators and systems verifiers. The premium shifts away from knowing specific framework syntax toward first-principles problem decomposition, distributed systems architecture, and rigorous quality validation.

Frontier models are too slow, computationally expensive, and power-hungry for high-throughput operational tasks. By using massive frontier models to generate synthetic reasoning data and teach compact, specialized models, enterprises achieve near-frontier accuracy at a fraction of the inference latency and cost.

Avoid investing in thin wrappers around closed proprietary APIs, and be deeply skeptical of startups competing on raw hardware leasing margins. Durable venture returns will flow to teams building proprietary scientific breakthroughs, defensible domain data flywheels, and physical automation platforms.

Always skate toward the physical realities of the technology stack rather than chasing ephemeral software trends. Physics, thermodynamics, and computational efficiency ultimately dictate what can scale economically; if you master the underlying infrastructure constraints, you will consistently anticipate the next major technological inflection.