Taming the AI Hardware Cambrian Explosion — Abdul Dakkak, Modular

So, let's get started . Hello, my name is . Hello, my name is . Hello, my name is Abdul. I'm from Abdul. I'm from Abdul. I'm from Modular. Today I Modular. Today I Modular. Today I will tell you what Modular is will tell you what Modular is will tell you what Modular is and what benefits it and what benefits it and what benefits it brings you. As you brings you. As you brings you. As you probably know, there probably know, there probably know, there is a is a is a huge shortage of huge shortage of huge shortage of computing computing computing power today. And there are power today. And there are no signs of improvement in the situation. no signs of improvement in the situation. Many of them are already Many of them are already Many of them are already booked for the booked for the booked for the next few next few next few years. But as a result years. But as a result years. But as a result , there is a , there is a , there is a real explosion in the real explosion in the real explosion in the world of equipment. We world of equipment. We world of equipment. We call this the " call this the " Cambrian explosion Cambrian explosion ." We have Nvidia, which ." We have Nvidia, which ." We have Nvidia, which offers several offers several offers several generations of GPUs, Intel, ARM, Apple— generations of GPUs, Intel, ARM, Apple— it all exists. All this is it all exists. All this is it all exists. All this is presented on the presented on the presented on the market. Each of them market. Each of them market. Each of them occupies its own niche, but the occupies its own niche, but the occupies its own niche, but the question arises: how question arises: how question arises: how to program all these to program all these to program all these systems? If systems? If systems? If you look at you look at you look at modern modern modern software, it's software, it's software, it's also a kind of chaos. also a kind of chaos. This is because many hardware features hardware features trickle down from the bottom up the trickle down from the bottom up the trickle down from the bottom up the stack. Here is a stack. Here is a stack. Here is a diagram showing diagram showing how vLLM supports Nvidia how vLLM supports Nvidia how vLLM supports Nvidia and AMD. If you add more and AMD. If you add more and AMD. If you add more equipment, the diagram equipment, the diagram equipment, the diagram becomes even more complicated.

becomes even more complicated. Today, people Today, people use not use not use not only vLLM, but also SGELang, TRTLM, only vLLM, but also SGELang, TRTLM, only vLLM, but also SGELang, TRTLM, Llama.cpp, etc. So at Modular Llama.cpp, etc. So at Modular Llama.cpp, etc. So at Modular we thought: this is a we thought: this is a we thought: this is a complete mess. complete mess. Let's Let's redesign the stack redesign the stack redesign the stack from scratch and create a from scratch and create a from scratch and create a unified layer unified layer unified layer for the entire for the entire for the entire hardware. To make everything hardware. To make everything hardware. To make everything work on our work on our work on our stack. This stack. This stack. This requires going back requires going back requires going back to basics and building a to basics and building a to basics and building a full stack, full stack, full stack, considering considering considering portability, portability, portability, performance, and performance, and performance, and programmability. At the programmability. At the programmability. At the lowest level, we lowest level, we lowest level, we developed Mojo—a developed Mojo—a heterogeneous heterogeneous heterogeneous programming language for AI programming language for AI programming language for AI and and and general- general- general- purpose programming. It is purpose programming. It is purpose programming. It is similar in style to Python. similar in style to Python. similar in style to Python. So those who know Python So those who know Python So those who know Python already know 90% of Mojo. We already know 90% of Mojo. We already know 90% of Mojo. We created this language, and created this language, and created this language, and it works with CPUs, GPUs, and it works with CPUs, GPUs, and it works with CPUs, GPUs, and all other all other all other accelerators.

accelerators. Using Mojo, we Using Mojo, we built MAX—our built MAX—our model inference engine. It model inference engine. It provides provides provides AI output for many AI output for many AI output for many different data types, different data types, different data types, such as images, such as images, such as images, videos, speech, and text. videos, speech, and text. We built all of our kernels We built all of our kernels from scratch based on from scratch based on from scratch based on Mojo. In addition, Mojo. In addition, Mojo. In addition, modern AI solutions are modern AI solutions are modern AI solutions are distributed. Therefore, distributed. Therefore, distributed. Therefore, you also need a you also need a you also need a cloud solution. You cloud solution. You cloud solution. You need to choose need to choose need to choose which hardware which hardware which hardware to deploy your to deploy your to deploy your model on, how model on, how model on, how to route to route to route user requests to user requests to user requests to different machines, how different machines, how different machines, how to perform to perform to perform KB cache offloading, KB cache offloading, KB cache offloading, and distributed and distributed and distributed and disaggregated and disaggregated and disaggregated inference to inference to inference to ensure your ensure your ensure your model runs at the expected model runs at the expected model runs at the expected speed and speed and speed and cost. So, cost. So, cost. So, to summarize, we to summarize, we to summarize, we have created a have created a have created a Modular platform that is fully Modular platform that is fully Modular platform that is fully programmable, programmable, programmable, hardware hardware hardware abstracted, has abstracted, has abstracted, has high high high performance, and performance, and performance, and can be deployed both can be deployed both can be deployed both on-premises and in on-premises and in on-premises and in our cloud our cloud our cloud environment. That is, environment. That is, environment. That is, if you have a cloud if you have a cloud if you have a cloud infrastructure, you infrastructure, you infrastructure, you can download can download can download Modular cloud and deploy Modular cloud and deploy Modular cloud and deploy it there. Let it there. Let it there. Let me look at me look at me look at each component of the each component of the each component of the stack in turn. So, the stack in turn. So, the stack in turn. So, the first one is Mojo. Mojo, as I first one is Mojo. Mojo, as I first one is Mojo. Mojo, as I mentioned, is a Python- mentioned, is a Python- like like like systems systems systems programming language. It is programming language. It is programming language. It is similar to C++, Rust, or CUDA similar to C++, Rust, or CUDA similar to C++, Rust, or CUDA C++. Its main C++. Its main C++. Its main advantage is the

advantage is the advantage is the ability ability ability to create programs to create programs to create programs for different for different for different accelerators. She is accelerators. She is accelerators. She is incredibly fast. incredibly fast. incredibly fast. It again It again It again really competes with CUDA really competes with CUDA really competes with CUDA in terms of in terms of in terms of performance. And you performance. And you performance. And you can consider it the can consider it the can consider it the best way best way best way to program to program to program processors and processors and processors and graphics cards graphics cards graphics cards using a language using a language using a language similar to Python. Regarding similar to Python. Regarding similar to Python. Regarding how we evaluate how we evaluate how we evaluate Mojo, there are Mojo, there are Mojo, there are three different vectors: three different vectors: usability, usability, performance, and performance, and performance, and mobility. ROCm and CUDA mobility. ROCm and CUDA mobility. ROCm and CUDA provide provide provide tremendous tremendous tremendous performance, but performance, but performance, but they lose out in they lose out in usability and usability and application development. application development. application development. Today, many people Today, many people Today, many people use agents use agents use agents to program to program to program such systems. OpenCL such systems. OpenCL was popular at one time a few years ago was popular at one time a few years ago . It . It . It provides excellent provides excellent provides excellent mobility, but mobility, but mobility, but sacrifices sacrifices sacrifices performance and performance and performance and convenience. Triton occupies a convenience. Triton occupies a convenience. Triton occupies a successful niche. It provides successful niche. It provides successful niche. It provides 70% of the productivity, but 70% of the productivity, but 70% of the productivity, but ensures ensures ensures mobility and mobility and mobility and ease of ease of ease of use. And Mojo use. And Mojo use. And Mojo is trying to fill is trying to fill is trying to fill this niche this niche this niche by combining the best of by combining the best of by combining the best of both worlds. It both worlds. It both worlds. It provides an abstraction that provides an abstraction that provides an abstraction that allows you to write allows you to write allows you to write mobile and user-friendly mobile and user-friendly mobile and user-friendly kernels while kernels while kernels while exposing exposing exposing low-level low-level low-level access for access for access for maximum maximum maximum performance. We've performance. We've performance. We've made made made almost all of almost all of almost all of our kernels open source, so you our kernels open source, so you our kernels open source, so you can can can test it all yourself.

test it all yourself. test it all yourself. You can run You can run You can run benchmarks on your own benchmarks on your own benchmarks on your own machine. We have the machine. We have the machine. We have the most modern kernels most modern kernels most modern kernels for Nvidia, AMD, for Nvidia, AMD, for Nvidia, AMD, and other hardware. To and other hardware. To and other hardware. To explain what " explain what " state-of-the-art" means: our state-of-the-art" means: our state-of-the-art" means: our matrix multiplication matrix multiplication matrix multiplication is faster than cuBLAS, hipBLAS, and is faster than cuBLAS, hipBLAS, and is faster than cuBLAS, hipBLAS, and everything else out there. And everything else out there. And everything else out there. And it's not by 1% or, in some it's not by 1% or, in some it's not by 1% or, in some cases, by 2%. We cases, by 2%. We cases, by 2%. We have the fastest have the fastest have the fastest attention core, faster than flash attention core, faster than flash attention core, faster than flash attention, and also for AI attention, and also for AI editor and editor and editor and things like that. One things like that. One things like that. One thing we thing we thing we announced about announced about announced about a month ago is Mojo 1.0 a month ago is Mojo 1.0 , a stabilization of our , a stabilization of our , a stabilization of our APIs, and we will be releasing Mojo APIs, and we will be releasing Mojo APIs, and we will be releasing Mojo 1.0 in the coming 1.0 in the coming 1.0 in the coming months. And we months. And we months. And we will also make Mojo open-source in the will also make Mojo open-source in the will also make Mojo open-source in the next few next few next few months. months. months. Max is based on Mojo. So, Max Max is based on Mojo. So, Max Max is based on Mojo. So, Max is trying to address is trying to address is trying to address some of the assumptions that some of the assumptions that some of the assumptions that were made at the were made at the were made at the time for PyTorch. PyTorch has time for PyTorch. PyTorch has time for PyTorch. PyTorch has been around for about 10 been around for about 10 been around for about 10 years. It was not years. It was not years. It was not designed for designed for designed for modern GenAI inference modern GenAI inference . As a result, . As a result, . As a result, VLM and SG Lang emerged, VLM and SG Lang emerged, VLM and SG Lang emerged, filling filling filling the gap by providing the gap by providing the gap by providing things like KV cache, things like KV cache, things like KV cache, paged attention, etc. But all paged attention, etc. But all paged attention, etc. But all this has somewhat this has somewhat this has somewhat complicated the entire complicated the entire complicated the entire PyTorch ecosystem.

PyTorch ecosystem. PyTorch ecosystem. Many people Many people Many people don't actually use PyTorch don't actually use PyTorch don't actually use PyTorch in its pure form. in its pure form. in its pure form. They are forced They are forced They are forced to use these to use these to use these add-ons. So add-ons. So add-ons. So consider a new consider a new consider a new approach we approach we approach we call the call the call the Max framework. It is Max framework. It is Max framework. It is built from the ground up built from the ground up built from the ground up using the using the using the best technologies best technologies , state-of-the-art graph , state-of-the-art graph compilers, compilers, compilers, eager execution mode, is eager execution mode, is eager execution mode, is portable across portable across portable across devices, understands devices, understands devices, understands your programs on Mojo your programs on Mojo your programs on Mojo for a new type of for a new type of for a new type of merging, and gives you merging, and gives you merging, and gives you maximum maximum maximum performance and performance and deployability that you would deployability that you would n't be able to get n't be able to get n't be able to get in regular PyTorch. in regular PyTorch. Compared to other Compared to other stacks, this is the only stacks, this is the only stacks, this is the only stack that allows stack that allows stack that allows you to service, you to service, you to service, develop models, and develop models, and develop models, and also optimize also optimize also optimize all in one holistic all in one holistic all in one holistic system. In Max, we system. In Max, we system. In Max, we support thousands of support thousands of support thousands of models, in fact models, in fact models, in fact thousands of architectures. thousands of architectures. Everything from Gemma 4 to Mistral, Everything from Gemma 4 to Mistral, MiniMax 3 and GLM. So, MiniMax 3 and GLM. So, any model you can any model you can any model you can think of is practically think of is practically think of is practically supported in Max. And supported in Max. And supported in Max. And it has the same it has the same it has the same interface you would interface you would interface you would expect by expect by expect by today's today's today's standards. We also standards. We also standards. We also provide provide provide OpenAI endpoints that you OpenAI endpoints that you OpenAI endpoints that you can can can use in use in use in this stack. If you are this stack. If you are this stack. If you are familiar with familiar with familiar with programming in programming in programming in PyTorch, the diagram on PyTorch, the diagram on PyTorch, the diagram on the right or left

the right or left the right or left will not be a big will not be a big will not be a big departure from departure from departure from your mental your mental your mental model. You can model. You can model. You can build the model as you build the model as you build the model as you expect, you know, expect, you know, expect, you know, you can you can you can import modules. The import modules. The import modules. The programming language programming language programming language for developing models for developing models is Python, so it is is Python, so it is is Python, so it is familiar to many. You familiar to many. You familiar to many. You can combine can combine can combine things as you things as you things as you create them. We also create them. We also create them. We also support eager support eager execution to execution to execution to simplify simplify simplify debugging. Another debugging. Another debugging. Another thing to thing to thing to understand is that understand is that understand is that modern LLMs run modern LLMs run modern LLMs run on multiple GPUs, and in on multiple GPUs, and in on multiple GPUs, and in many cases, many cases, on multiple nodes. on multiple nodes. So, the Max APIs give you a So, the Max APIs give you a way to program way to program way to program both multi-graphics both multi-graphics both multi-graphics and multi-node and multi-node and multi-node systems systems systems directly through directly through directly through the API so that the model the API so that the model the API so that the model is natively supported on is natively supported on is natively supported on different GPUs and nodes. And different GPUs and nodes. And different GPUs and nodes. And it can also be it can also be it can also be heterogeneous. So, heterogeneous. So, heterogeneous. So, there's nothing stopping you from there's nothing stopping you from there's nothing stopping you from setting up setting up prefill on hardware X prefill on hardware X and decoding on and decoding on and decoding on hardware Y. Finally, hardware Y. Finally, hardware Y. Finally, on top of all this on top of all this on top of all this is a modular is a modular is a modular cloud. As I cloud. As I cloud. As I mentioned, mentioned, mentioned, today's AI today's AI inference is inference is inference is distributed. It distributed. It distributed. It works on many works on many works on many different machines. It is different machines. It is different machines. It is heterogeneous. So we heterogeneous. So we heterogeneous. So we offer Mojo cloud, offer Mojo cloud, offer Mojo cloud, which is which is which is productivity-focused. We

productivity-focused. We productivity-focused. We call this a " call this a " glass box" glass box" glass box" because you can because you can because you can change all the variables change all the variables change all the variables in MAX, Mojo, and everything else in in MAX, Mojo, and everything else in in MAX, Mojo, and everything else in your endpoints your endpoints your endpoints . It . It . It provides both provides both provides both endpoint deployment endpoint deployment endpoint deployment and and and the ability to the ability to the ability to use your own use your own use your own cloud cloud cloud infrastructure. We infrastructure. We infrastructure. We also have Mojo cloud lab, also have Mojo cloud lab, also have Mojo cloud lab, which allows you which allows you which allows you to experiment with to experiment with to experiment with different parameters different parameters different parameters and choose the best and choose the best and choose the best point on the Pareto frontier point on the Pareto frontier point on the Pareto frontier for your model or for your model or for your model or workloads. workloads. As for the interface, As for the interface, if you are familiar with if you are familiar with if you are familiar with any cloud any cloud any cloud provider, it should provider, it should provider, it should be quite familiar. be quite familiar. You simply go to the You simply go to the site, choose a model site, choose a model site, choose a model to launch, to launch, to launch, get an API key, get an API key, get an API key, insert it into insert it into insert it into your HTTP request, and your HTTP request, and your HTTP request, and launch. You can launch. You can launch. You can have dedicated have dedicated have dedicated endpoints, and we endpoints, and we endpoints, and we can service can service can service your own model. your own model. your own model. Often people Often people Often people use use use pre-trained models or pre-trained models or pre-trained models or models with their own models with their own models with their own speculative speculative speculative decoders. We can decoders. We can decoders. We can do all this for do all this for do all this for you too. Why should you you too. Why should you you too. Why should you care? That's a lot of care? That's a lot of care? That's a lot of technology. Why is this technology. Why is this technology. Why is this important to people?

important to people? Because thanks to this Because thanks to this system you can system you can system you can achieve much achieve much achieve much more. You can more. You can more. You can really reduce really reduce really reduce costs. You get costs. You get costs. You get more flexibility more flexibility more flexibility because you can because you can because you can use use use different equipment. You are different equipment. You are different equipment. You are not dependent on a not dependent on a not dependent on a single single single iron supplier. You get iron supplier. You get iron supplier. You get portability. You portability. You portability. You can can can use use use many different many different many different models and so on. So what models and so on. So what models and so on. So what about performance? about performance? about performance? We tested We tested We tested many of these many of these many of these options. And this is options. And this is options. And this is a comparison with the a comparison with the a comparison with the best solutions best solutions , such as vLLM , such as vLLM , such as vLLM or SG Lang combinations. In many or SG Lang combinations. In many or SG Lang combinations. In many cases, these cases, these cases, these indicators indicators indicators are also independently are also independently are also independently verified. So, in verified. So, in verified. So, in the case of Qwen 2.5, we are 5.5 the case of Qwen 2.5, we are 5.5 the case of Qwen 2.5, we are 5.5 times faster with TTFT times faster with TTFT times faster with TTFT compared to vLLM. We compared to vLLM. We compared to vLLM. We can also can also can also get a 1.5x get a 1.5x get a 1.5x improvement in improvement in improvement in throughput throughput . So, the number of . So, the number of . So, the number of tokens per minute on tokens per minute on tokens per minute on GPU is 1.5 times greater on GPU is 1.5 times greater on GPU is 1.5 times greater on B200 than on vLLM. As B200 than on vLLM. As B200 than on vLLM. As for Flux 2, I for Flux 2, I for Flux 2, I advise everyone to visit the advise everyone to visit the advise everyone to visit the booth behind this curtain. There booth behind this curtain. There 's a great 's a great Flux 2 demo there. We're Flux 2 demo there. We're Flux 2 demo there. We're about 7x about 7x about 7x faster compared to faster compared to faster compared to torch.compile with diffusers torch.compile with diffusers . So, again, MAX . So, again, MAX . So, again, MAX supports different supports different supports different modalities within the modalities within the modalities within the same platform. You same platform. You same platform. You don't need to don't need to don't need to use use use some special omni- some special omni- version of version of version of software or anything software or anything software or anything like that. We

like that. We like that. We support support support video, video, video, audio, image generation, as well audio, image generation, as well audio, image generation, as well as text as text as text generation. We also generation. We also generation. We also support, you know, support, you know, support, you know, AMD systems, where you can AMD systems, where you can AMD systems, where you can get a get a get a TCO advantage by being 4x TCO advantage by being 4x TCO advantage by being 4x faster than the counterpart on faster than the counterpart on ... on MI355 systems. ... on MI355 systems. Actually, some of Actually, some of these comparisons are not these comparisons are not these comparisons are not entirely correct, because entirely correct, because entirely correct, because some people might say, "Oh some people might say, "Oh , 7x is much more than , 7x is much more than , 7x is much more than 4x." But this comparison 4x." But this comparison 4x." But this comparison cannot be made cannot be made cannot be made so simply. You need so simply. You need so simply. You need to consider the cost of to consider the cost of to consider the cost of one GPU. And one GPU. And one GPU. And AMD products are usually 30% AMD products are usually 30% cheaper than cheaper than cheaper than Nvidia counterparts. Artificial Analysis—we Nvidia counterparts. Artificial Analysis—we Nvidia counterparts. Artificial Analysis—we work with them, work with them, work with them, and they independently and they independently and they independently verify our verify our verify our results. These are our results. These are our results. These are our results as of results as of results as of today. We are today. We are today. We are about 3 times about 3 times about 3 times faster than the faster than the faster than the closest closest closest competitor on the competitor on the competitor on the Flux 2 dev model. And that's without any Flux 2 dev model. And that's without any Flux 2 dev model. And that's without any tricks with tricks with tricks with distillation or distillation or distillation or quantization or quantization or quantization or anything like that. This is anything like that. This is anything like that. This is simply "pure" simply "pure" simply "pure" speed, as we speed, as we speed, as we utilize the utilize the hardware capabilities from the ground up, hardware capabilities from the ground up, ensuring ensuring ensuring maximum maximum model performance. You can model performance. You can visit the Artificial visit the Artificial visit the Artificial Analysis website today and Analysis website today and Analysis website today and see these see these see these results. So what does this

results. So what does this results. So what does this mean for people? This mean for people? This mean for people? This means you can means you can means you can build build build applications that applications that applications that were previously were previously were previously impossible due to impossible due to impossible due to slow output. slow output. This is, in fact, a kind of real-time demo. Someone real-time demo. Someone just inserts just inserts just inserts prompts. But prompts. But prompts. But the generation is the generation is the generation is accurate. You can accurate. You can accurate. You can generate generate generate images in less than a images in less than a images in less than a second. In second. In second. In many cases, this many cases, this many cases, this takes 0.3 seconds. takes 0.3 seconds. takes 0.3 seconds. Again, you can Again, you can Again, you can come to our come to our come to our stand. There's a nice stand. There's a nice stand. There's a nice demonstration there: you demonstration there: you demonstration there: you enter a prompt, enter a prompt, enter a prompt, the system reads the system reads the system reads the video stream and the video stream and the video stream and visualizes it. visualizes it. visualizes it. Another data point is Another data point is Another data point is Gemma 4. So Gemma 4 is a Gemma 4. So Gemma 4 is a Gemma 4. So Gemma 4 is a model that was model that was model that was introduced very introduced very introduced very recently, about a recently, about a recently, about a month or a month and a half month or a month and a half month or a month and a half ago. Again, we are ago. Again, we are ago. Again, we are significantly faster than any significantly faster than any other other other competitor in the market. competitor in the market. You know, in many You know, in many cases it's even cases it's even cases it's even hardware companies.

hardware companies. For example, SambaNova is a For example, SambaNova is a hardware hardware hardware accelerator. We are, accelerator. We are, accelerator. We are, say, 6 times say, 6 times say, 6 times faster than them, and in faster than them, and in faster than them, and in general we are much, general we are much, general we are much, much faster than much faster than much faster than others. And, by the way, for others. And, by the way, for others. And, by the way, for all these cases, we are all these cases, we are all these cases, we are just getting started. Over just getting started. Over just getting started. Over time, we time, we time, we are implementing more and are implementing more and are implementing more and more optimizations. more optimizations. So we never, ever So we never, ever stop stop stop improving these improving these improving these things. We have many things. We have many things. We have many places where you can get places where you can get places where you can get involved. You can involved. You can involved. You can join our join our open source community. open source community. Much of Much of Much of our code is our code is our code is open source open source open source software. We software. We software. We exposed over 700,000 exposed over 700,000 exposed over 700,000 lines of kernel code. We lines of kernel code. We lines of kernel code. We have made have made have made all of our all of our all of our model definitions and our entire Max model definitions and our entire Max model definitions and our entire Max Serve open. We have AI skills. So, Serve open. We have AI skills. So, Serve open. We have AI skills. So, if you are not familiar with if you are not familiar with if you are not familiar with Mojo or Max, you can Mojo or Max, you can Mojo or Max, you can use AI skills use AI skills use AI skills to port code to port code to port code from languages ​​you know to from languages ​​you know to from languages ​​you know to Mojo or Max. You Mojo or Max. You Mojo or Max. You can also join can also join can also join us. We are hiring us. We are hiring us. We are hiring employees.

employees. employees. Our stand has Our stand has Our stand has brochures with brochures with brochures with information about information about information about vacancies for various vacancies for various vacancies for various positions. As you positions. As you positions. As you probably know, we probably know, we probably know, we hire both hire both low-level kernel developers low-level kernel developers and and and high-level high-level high-level Kubernetes specialists, and Kubernetes specialists, and Kubernetes specialists, and everyone in between. You can everyone in between. You can everyone in between. You can try all this in try all this in try all this in practice. This is not practice. This is not practice. This is not just a demonstration just a demonstration just a demonstration or a presentation. You or a presentation. You or a presentation. You can install can install can install modular via pip right modular via pip right modular via pip right now if you want to now if you want to now if you want to try all of this out, and I try all of this out, and I try all of this out, and I invite everyone invite everyone invite everyone to check out these to check out these to check out these results. That's results. That's results. That's probably all, but I'm probably all, but I'm probably all, but I'm ready to answer ready to answer ready to answer questions if there are any questions if there are any questions if there are any , and I'll be on the , and I'll be on the , and I'll be on the stand for a few more stand for a few more stand for a few more hours.