AWS Innovations at re:Invent 2024: A Leap in AI and Cloud Technology

Peter DeSantis, Senior Vice President, welcomes attendees to re:Invent 2024, expressing gratitude for their presence at Monday Night Live, and humorously apologizes for the lack of IPAs, promising to rectify it next year.

DeSantis outlines the evening's agenda, emphasizing a deep dive into AWS's operational strategies and innovations, highlighting the importance of understanding the 'how' behind their offerings, which are not merely features but essential tools for customers.

Inspired by AI assistants, DeSantis shares his thought process for visualizing AWS's structure, initially considering an iceberg to represent visible features versus hidden complexities, then shifting to a space flight analogy to illustrate the extensive behind-the-scenes efforts, ultimately settling on trees to symbolize long-term investments and foundational support.

DeSantis introduces the concept of taproots, explaining their role in accessing deep water sources, which parallels how Amazon leaders engage with details to understand operational realities, enabling proactive problem-solving and decision-making, contrasting this with other organizations where information may be siloed.

He highlights the significance of AWS-wide operations meetings as a critical mechanism for discussing issues and sharing learnings, emphasizing the importance of being involved in details to make informed long-term decisions, citing the decision to invest in Nitro technology as a pivotal moment that shaped AWS's performance capabilities.

DeSantis transitions to discussing buttress roots, which support large trees through extensive above-ground systems, drawing a parallel to AWS's innovative capabilities across various domains, including data center power, networking, and database internals, asserting that few companies invest as comprehensively in critical components as AWS does.

He concludes by teasing upcoming presentations that will demonstrate how AWS's unique investments enable the creation of innovative solutions tailored to customer needs, reinforcing the theme of deep-rooted support for sustained growth and performance.

The discussion begins with the fascinating interconnectedness of trees through their buttress root systems, which are linked to a vast underground organism, often mistaken for mere mushrooms. This symbiotic relationship allows trees to communicate and share resources, enhancing the overall health and resilience of the forest ecosystem.

The speaker reflects on their experience joining AWS, noting the company's strong emphasis on building a robust culture that supports scalability. Senior leaders prioritized defining cultural values and mechanisms, such as a weekly review process, to ensure accountability and maintain a focus on cost and innovation. This unique culture is seen as a critical element in AWS's evolution.

Transitioning to the innovations at AWS, the speaker introduces their journey as part of a small 14-person team tasked with developing the Elastic Compute Cloud. They highlight the ambitious mission of architecting a service that would revolutionize cloud computing, leading to the introduction of custom silicon development, starting with the Graviton processor aimed at enhancing developer collaboration and performance.

The evolution of the Graviton processors is discussed, with Graviton2 focusing on specialized workloads such as web servers and containerized applications, while Graviton3 delivered significant performance improvements for demanding tasks like machine learning inference and video transcoding. The speaker emphasizes that Graviton4, the latest iteration, is the most powerful chip yet, offering three times the performance of its predecessors, marking a significant advancement for large database applications.

The speaker elaborates on the complexities of evaluating CPU performance, likening modern CPUs to sophisticated systems with distinct frontends and backends. They critique traditional microbenchmarks for oversimplifying performance assessments, arguing that real-world workloads are messier and more unpredictable, which necessitates a deeper understanding of how processors handle various tasks beyond mere benchmark results.

Finally, the speaker presents the performance improvements of Graviton3 over Graviton2, showcasing the advancements made in handling real-world applications. They stress that AWS's design philosophy focuses on excelling in practical workloads rather than just winning benchmarks, highlighting the importance of understanding the intricacies of frontend and backend performance in achieving superior results.

The testing of NGINX revealed significant performance improvements with Graviton4, particularly in MySQL workloads, which has led to increased customer satisfaction. These results are not merely theoretical; they reflect real-world benefits experienced by customers, especially during events like Amazon Prime Day, where over 250,000 Graviton processors were utilized.

The AWS Nitro System represents a revolutionary shift in server architecture, enhancing security and agility. Nitro's design allows for the seamless operation of various environments, including bare metal EC2 instances and even Apple Macs, showcasing its versatility and the significant improvements it brings to cloud security.

AWS has transformed its approach to cloud security through the Nitro System, which ensures the integrity of software running across its global infrastructure. This involves a detailed boot process that includes cryptographic proof at each step, from the read-only memory to the applications, creating a robust chain of trust that prevents unauthorized code execution.

The boot process in AWS servers is critical for maintaining security, beginning with a unique secret generated during manufacturing. This secret forms the basis of a public-private key pair, with the private key acting as an anchor for the chain of trust. Each stage of the boot process involves creating new keys while securely destroying the previous ones, ensuring that any failure in this chain results in immediate system access denial.

Building on the Nitro System, Graviton4 processors extend attestation capabilities, allowing for secure communication between processors. This cryptographic verification ensures that every critical connection, including CPU-to-CPU communication, is protected by hardware-based security, creating a continuous security framework that is unprecedented in traditional server environments.

The discussion shifts to the evolution of hard drive capacity, highlighting how manufacturers have consistently increased data storage capabilities over the years, which is essential for accommodating the growing demands of cloud computing and data management.

The discussion begins by reflecting on the evolution of storage drives, highlighting a significant drop in costs due to innovations in design and manufacturing processes. This evolution is crucial for ensuring efficiency and readiness for future storage innovations.

A detailed examination of the storage architecture is presented, which consists of a frontend fleet of web servers managing authentication requests, a backend service that tracks data, and the actual data storage area. The head node, equipped with CPU and memory, runs specialized software for critical functions, including drive health monitoring.

As drive capacities have increased, maintaining the fixed ratio of compute resources to storage has become increasingly challenging. Initially, servers housed 12 to 24 drives, but advancements in technology allowed for configurations with up to 288 drives, leading to significant increases in storage capacity.

The introduction of the Barge storage server, which contained 288 drives, is noted as a major milestone, capable of storing nearly six petabytes of data with today's 20 terabyte drives. This ambitious project provided valuable lessons, particularly regarding the physical challenges of managing such heavy and dense configurations.

The operational challenges faced with the Barge server are discussed, including the significant weight of each rack (over two tons), which necessitated reinforced floors and careful planning for deployment. Additionally, the vibration from 288 drives impacted performance, and managing such a large number of drives pushed software systems to their limits.

The lessons learned from the Barge server experience led to a reevaluation of operational efficiency and agility in storage management. The need for a more flexible approach to storage services, such as S3 and EBS, became apparent, prompting a shift towards disaggregation in storage architecture.

The concept of disaggregation in storage is introduced, aiming to separate storage from compute resources to enhance performance and scalability. This approach allows for independent scaling of storage and compute, addressing the limitations of tightly coupled systems.

The implementation of Nitro cards is highlighted as a transformative step, providing intelligence to storage systems. Each drive is securely virtualized, preserving direct access while enabling enhanced encryption and security. This innovation allows for efficient data access without the constraints of physical limitations.

The discussion highlights key differences in storage architecture, particularly focusing on Nitro cards and drives. The design allows for quick replacement of failed drives due to its disaggregated nature, enabling technicians to service units without significant downtime. This innovation has transformed drive failures from major events into manageable occurrences, as seen in the Barge example, which impacted 288 drives. With disaggregated storage, a head node failure can be resolved by simply launching and reattaching drives, eliminating the need for data movement and allowing for seamless operations.

The architecture's ability to decouple compute and storage has led to significant operational improvements. It allows for temporary scaling during data hydration and rebalancing periods, enhancing flexibility and delivering better value. This separation enables high performance while scaling, which is crucial as drive capacities continue to grow. The architecture not only simplifies maintenance but also fosters faster innovation, ultimately leading to more reliable storage services.

The speaker introduces two distinct workloads: AI model training and big data applications. Unlike traditional scale-out workloads, AI workloads are characterized as scale-up workloads due to the increasing size of models. The discussion references the 'Scaling Laws' published in 2020, which suggest that as certain parameters scale up, such as dataset size, the models become more compute-intensive. This trend has led to a significant push in the industry towards building better AI infrastructure, as larger models require exponentially more resources to achieve marginal improvements.

The speaker explains the implications of log-log graphs in understanding AI model growth. Unlike linear relationships, straight lines on log-log graphs indicate that to achieve a 50% improvement in model performance, a million times more resources may be required. This relationship underscores the industry's challenge in developing better AI infrastructure, as the exponential growth in model size necessitates a corresponding increase in computational power and resources.

The process of training a predictive model begins with prompting it with a set of tokens, leading to the prediction of the next token. This basic skill allows for the emergence of remarkable properties. To create such a model, extensive training is required to minimize prediction error, which necessitates massive computational resources. Training the largest models often exceeds the capabilities of even the most powerful single servers.

Implementing data parallelism involves splitting data and distributing it across multiple servers. However, simply dividing the data does not yield effective results; all servers must share and combine their outputs to create a unified model. This process is constrained by the concept of global batch size, which dictates the maximum data set size before combining results. The practical limitation of this approach means that scaling is typically restricted to a few thousand servers, beyond which the efficiency diminishes, leading to increased costs without proportional benefits.

To effectively scale up computational resources, it is essential to create a coherent system that maximizes compute power and high-speed memory. The proximity of components is crucial, as closer arrangements allow for shorter wiring, which reduces latency and enhances efficiency. This design consideration is vital for optimizing performance in large-scale computing environments.

The announcement of Trainium2 marks a significant advancement in chip technology aimed at building the most powerful computational models. The development of Trainium2 involves sophisticated manufacturing processes that leverage advanced packaging techniques to optimize performance. The chip fabrication process is limited by the size of the reticle used in production, which is approximately 800 square millimeters, although the final package appears larger due to its enclosing structure.

The evolution of chip packaging has transformed from a simple enclosure to a complex system that integrates multiple chips within a single package. Advanced packaging now utilizes interposers, which function like tiny motherboards, allowing for enhanced connectivity and performance. This innovation enables the integration of multiple chips, as seen in the Graviton3 and Graviton4 models, which utilize advanced packaging to achieve significant performance improvements.

The discussion begins with an overview of the chip architecture, highlighting the central chip and smaller chips that facilitate memory access. The speaker notes that by separating compute functions, they achieved a 50% cost reduction on the Graviton4 processor, emphasizing that this approach is now standard practice in the industry.

The speaker introduces the Trainium2 package, showcasing its design which includes two chips positioned centrally. These chips are High Bandwidth Memory (HBM) modules, which are stacked to enhance power efficiency. The speaker explains that this stacking is made possible by advancements in technology that allow for reduced power consumption.

A detailed cross-section of the chip is presented, illustrating the layers of the HBM module and the interposer beneath. The speaker describes the intricate electrical connections, noting that each connection is approximately 100 microns, smaller than the finest grain of sand. This precision is crucial for maintaining the integrity of the chip's performance amidst heat and power fluctuations.

The speaker elaborates on the importance of voltage management in semiconductor operations. They explain that semiconductors require specific voltage levels to function efficiently, and data centers progressively step down voltage as it approaches the chip. The final voltage adjustment occurs very close to the chip to minimize voltage drop, which is critical for optimal performance.

The Trainium2 board is highlighted, showcasing the placement of voltage regulators around the package perimeter. This strategic positioning allows for shorter wiring, which enhances performance by reducing latency. The speaker contrasts this with the previous Trainium1 model, demonstrating how the new design mitigates load variability, thereby improving the chip's lifespan and overall efficiency.

The discussion shifts to the server configuration, specifically the Trainium2 servers, which are described as large and powerful. Each server consists of multiple trays, each housing two chips with dedicated resources. The speaker notes that while Trainium servers act as accelerators for computational tasks, they do not support traditional programming environments, which is a key consideration in their engineering design.

The Trainium2 server is touted as the most powerful AI server offered by AWS, capable of delivering 20 petaflops of processing power. This performance metric is highlighted as being seven times greater than previous models, underscoring the significant advancements made in AI server technology.

The Trainium2 server boasts 1.5 times the performance of its predecessor, Trainium1, making it a significant upgrade with two and a half times more processing power. This scale-up server is designed to meet the immediate demands of customers who want access to powerful AI capabilities right from day one, reflecting a shift from traditional adoption curves seen with new chip technologies.

The design of Trainium2 incorporates a reduction in the number of cables used, opting instead for wire traces. This innovation is crucial as each cable connection can introduce defects during manufacturing. The server is engineered for high-level automation in manufacturing and assembly, enhancing reliability and efficiency.

Trainium2 is not just powerful; it is a specialized tool with a unique architecture that diverges from traditional CPU and GPU designs. The architecture employs a systolic array, which allows for efficient data processing by minimizing memory bandwidth and optimizing tensor operations, thus enhancing performance over conventional hardware.

The Neuron Kernel Interface (NKI) is introduced as a means to optimize performance and facilitate cost-effective experimentation. This interface is designed to leverage the novel hardware capabilities of Trainium, allowing researchers to build and innovate on demanding AI workloads.

In a recent announcement, access to Trainium hardware was extended to researchers from prestigious institutions such as Carnegie Mellon, UT Austin, and Oxford. This collaboration aims to explore the novel capabilities of Trainium and drive innovation in AI technology.

NeuronLink, a proprietary technology, enables the combination of multiple Trainium servers into a single logical server, referred to as an ultra-server. This innovative protocol allows for enhanced connectivity and performance, surpassing the capabilities of traditional server architectures.

The presentation culminated in the unveiling of an UltraServer, which integrates multiple Trainium servers to deliver unprecedented performance, exceeding that of any current EC2 AI server. This development signifies a major leap forward in server technology for AI applications.

The discussion begins with an overview of AI inference, particularly focusing on large model inference and its demanding workloads. It highlights two main workloads: input encoding, which prepares prompts for token generation, and token generation itself. The speaker explains that during token generation, the entire model must be read each time a token is generated, creating a high demand for resources despite the small amount of compute required.

As AI models evolve, customer expectations shift towards faster token generation, especially in applications like chatbots where users experience delays during prefill. The speaker notes that while prefill requires significant memory bandwidth, the demand for rapid inference is growing, prompting customers to seek solutions that can deliver both prefill and token generation efficiently.

The speaker announces the launch of Amazon Bedrock, a service designed to provide access to optimized AI inference capabilities. This service is currently in preview and aims to enhance latency-optimized inference. The introduction of models like the smaller Llama 70B is highlighted, which reportedly offers superior performance compared to other offerings in the market.

The speaker introduces the Claude 3.5 model, which is noted for its impressive speed, running 60% faster than previous models. This model is part of the latency-optimized offerings and is designed to provide the fastest inference capabilities. The performance metrics are discussed, emphasizing the importance of lower response times in AI applications.

Tom Brown, Co-founder and Chief Compute Officer of Anthropic, joins the discussion to elaborate on their collaboration with Amazon. He emphasizes the significance of Claude in providing trustworthy AI solutions, noting that millions rely on it for various tasks. The partnership allows businesses to utilize Claude on a secure cloud platform without requiring changes on their end, streamlining the integration process.

Tom Brown discusses the technical specifications that contribute to Claude 3.5's performance, highlighting the powerful hardware that supports over a petaflop of compute and ample memory bandwidth. He stresses the importance of maintaining peak performance by ensuring that the system is consistently fed with data, which is crucial for achieving optimal results in AI applications.

The discussion begins with the challenge of sequencing work and waiting for inputs, likened to a game of Tetris. Anthropic has been collaborating with Amazon and Annapurna to enhance performance. They discovered that a single performance optimization can significantly unlock compute resources, making it worthwhile to write low-level kernels, similar to transitioning from Python to C for critical components. The design of Trainium is particularly suited for this low-level coding, allowing developers to better understand instruction execution and optimize kernel writing.

Excitement builds as the speaker announces Project Rainier, a new Amazon cluster featuring hundreds of thousands of chips capable of delivering exaFLOPs, which is over five times more than previous capabilities. This project aims to enhance the performance of AI models, including the Claude 3.5 Sonnet, which has been recognized as one of the smartest models globally. Project Rainier is expected to accelerate research and provide customers with smarter, more reliable agents.

The speaker emphasizes the importance of scaling up and scaling out in AI training. AWS's extensive experience in high-performance scale-out is highlighted, with a focus on building a robust AI network. The discussion includes the need for a massive network capable of handling simultaneous server operations during training, ensuring no delays occur that could lead to idle capacity. The introduction of the 10p10u network, which boasts tens of petabits of capacity and elastic scalability, is presented as a solution to these challenges.

The 10p10u network is described as a highly elastic network fabric that can be scaled down to fit various cluster sizes. The speaker shares a light-hearted anecdote about the network's green switches, which were inspired by the 2017 Pantone color of the year, illustrating the importance of investing in meaningful aspects of technology rather than superficial details. This approach reflects a broader philosophy of prioritizing spending on critical components while minimizing costs on less significant features.

The discussion begins with the importance of network patch cables in building a dense network fabric, emphasizing the need to interconnect switches effectively. The complexity of the 10p10u network is highlighted, showcasing the innovations that have emerged to streamline this process.

A proprietary trunk connector innovation is introduced, which consists of 16 separate fiber optic cables. This game-changing solution is implemented at the factory, dramatically streamlining the installation process and virtually eliminating previous complexities. The speaker notes that while this may seem modest, it significantly speeds up operations, enhancing the visual appeal of the green switches.

Another innovation, the Firefly Optic Plug, is presented as a low-cost device that allows for comprehensive testing before server racks arrive. This innovation is crucial as it prevents waste of time and resources, underscoring the idea that time is literally money in this context. Additionally, the Firefly Plug serves a dual purpose by preventing dust particles from entering the system, which can degrade performance and create network issues.

The speaker mentions the extensive implementation of the 10p10u network, noting that over 3 million links have been installed, even before the full rollout. This network aims to deliver higher performance, addressing the biggest source of failure in optical links, which are the miniature components responsible for sending and receiving signals.

The discussion shifts to the challenges of optimizing massive networks, particularly in detecting failures and updating switches. The speaker explains that traditional protocols like BGP and OSPF are used for health sharing among switches, but these can be slow in large networks when links fail, requiring significant time to find new optimal paths.

To address these challenges, the speaker introduces the Scalable Intelligent Data Routing (SIDR) protocol, which combines central planning with decentralized speed. This innovative approach allows for rapid responses to failures, achieving a response time that is ten times faster than other network fabrics, ensuring that the 10p10u network can quickly return to operational status.

In closing, the speaker reflects on the core innovations discussed, including Nitro, Graviton, and storage solutions, emphasizing the powerful AI server capabilities and the overall benefits of these innovations. The audience is encouraged to appreciate the advancements being made to create differentiated solutions, concluding with a warm farewell and an invitation to enjoy the re:Invent event.