How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs

Hello everyone. My Hello everyone. My name is Daniel. I name is Daniel. I name is Daniel. I work on AI and AI work on AI and AI agents here at Cerebras. agents here at Cerebras. agents here at Cerebras. What is your name? What is your name? What is your name? I am Michelle. I am I am Michelle. I am I am Michelle. I am the co-founder of Gimlet the co-founder of Gimlet the co-founder of Gimlet Labs, where we build a Labs, where we build a Labs, where we build a cloud based on different cloud based on different cloud based on different types of processors to types of processors to types of processors to optimize optimize optimize inference at every inference at every inference at every level of the stack. level of the stack. level of the stack. We are very excited We are very excited We are very excited to present our to present our to present our report on what report on what report on what we have to we have to we have to do to do to do to satisfy your satisfy your satisfy your insatiable need for insatiable need for insatiable need for tokens. As you know, tokens. As you know, tokens. As you know, spending on AI is spending on AI is spending on AI is crazy right now. This crazy right now. This crazy right now. This year we plan to year we plan to year we plan to spend spend spend $765 billion on $765 billion on $765 billion on computing, and computing, and computing, and up to $1.6 trillion by 2031 up to $1.6 trillion by 2031 up to $1.6 trillion by 2031 . Leading . Leading . Leading laboratories laboratories laboratories are trying to get are trying to get are trying to get as much as much as much computing computing computing power as possible from power as possible from power as possible from different different different equipment manufacturers both for equipment manufacturers both for equipment manufacturers both for training models and training models and training models and for serving for serving for serving users. And all users. And all users. And all this power goes this power goes this power goes to satisfy to satisfy everyone's insatiable need for tokens. At everyone's insatiable need for tokens. At last year's last year's last year's IO conference, IO conference, IO conference, Google said it was Google said it was Google said it was servicing 480 servicing 480 servicing 480 trillion tokens trillion tokens trillion tokens per month. And last per month. And last per month. And last month they month they month they reported that this reported that this reported that this figure had increased to 3.2 figure had increased to 3.2 figure had increased to 3.2 quadrillion tokens quadrillion tokens quadrillion tokens per month. So, all of per month. So, all of per month. So, all of this is happening this is happening this is happening at the same time as at the same time as at the same time as models are getting models are getting models are getting cheaper and cheaper and cheaper and faster. Llama 3.3 7B at the faster. Llama 3.3 7B at the faster. Llama 3.3 7B at the time of its release time of its release was 25% cheaper than today, was 25% cheaper than today, while remaining while remaining while remaining 3.6 times 3.6 times 3.6 times faster. So today faster. So today faster. So today we're going to talk about two we're going to talk about two we're going to talk about two things. First: what have we already things. First: what have we already things. First: what have we already done to done to done to optimize optimize optimize inference to

inference to inference to make it make it make it cheaper and faster? And cheaper and faster? And cheaper and faster? And what does the roadmap look like what does the roadmap look like what does the roadmap look like for even greater for even greater performance and performance and throughput improvements throughput improvements ? To do this, ? To do this, ? To do this, let's jump let's jump let's jump to 2022 and learn to 2022 and learn to 2022 and learn about all the optimizations about all the optimizations about all the optimizations that have brought us to where we are that have brought us to where we are that have brought us to where we are today today today and, and, and, hopefully, what the hopefully, what the future will look like. LLMs future will look like. LLMs create tokens one at a time create tokens one at a time create tokens one at a time in a cycle in a cycle in a cycle called called called autoregressive. autoregressive. So, to generate So, to generate one additional one additional one additional token, you need token, you need token, you need to read all the weights from to read all the weights from to read all the weights from memory into the memory into the memory into the computing unit. computing unit. computing unit. So most of the So most of the So most of the time the computer time the computer time the computer is idle. It waits for is idle. It waits for is idle. It waits for memory memory memory to transfer all the weights to transfer all the weights to transfer all the weights needed to needed to needed to perform perform perform the calculations themselves. So, up to 90% of the calculations themselves. So, up to 90% of the time, the computer is simply the time, the computer is simply the time, the computer is simply waiting to waiting to waiting to load data load data load data from memory. So how do we from memory. So how do we from memory. So how do we optimize this so that there optimize this so that there optimize this so that there is less is less is less downtime and you can downtime and you can downtime and you can fully fully fully utilize utilize utilize resources and resources and resources and issue tokens faster? You issue tokens faster? You issue tokens faster? You can simply add can simply add can simply add more more more computing computing computing power to power to power to solve this solve this solve this problem. Instead of problem. Instead of problem. Instead of passing the passing the passing the entire set of weights, what entire set of weights, what entire set of weights, what if we could if we could if we could split the model into split the model into split the model into four parts and four parts and four parts and let each GPU let each GPU let each GPU process a quarter of process a quarter of process a quarter of the model? Then the model? Then the model? Then theoretically you could theoretically you could theoretically you could reduce the time it reduce the time it reduce the time it takes to takes to takes to process a single process a single process a single token by a factor of four.

token by a factor of four. token by a factor of four. However, if you want to However, if you want to However, if you want to do this, you do this, you do this, you will have to pay a will have to pay a will have to pay a "tax". Before the "tax". Before the "tax". Before the attention block, after attention block, after attention block, after it, and after the it, and after the it, and after the FFM block, you need to FFM block, you need to FFM block, you need to perform an perform an perform an all-reduce operation, where you take the all-reduce operation, where you take the all-reduce operation, where you take the individual calculations from individual calculations from individual calculations from each GPU and each GPU and each GPU and combine them together. combine them together. combine them together. So, you can So, you can So, you can get the result get the result get the result because none of the because none of the because none of the GPUs GPUs GPUs has the full has the full has the full picture. It picture. It picture. It only processes only processes only processes a quarter of the model. Therefore, a quarter of the model. Therefore, a quarter of the model. Therefore, this needs to be done this needs to be done twice for each layer, which can twice for each layer, which can only result in a only result in a only result in a twofold twofold twofold speedup with a speedup with a speedup with a fourfold fourfold fourfold increase in increase in increase in computing computing computing resources. If I were to resources. If I were to resources. If I were to plot this on a plot this on a plot this on a graph, you would graph, you would graph, you would see that see that see that adding more adding more adding more GPUs GPUs GPUs speeds up tokens, speeds up tokens, speeds up tokens, but requires but requires but requires exponentially exponentially exponentially more energy per more energy per more energy per token due to the token due to the token due to the communication communication communication overhead we overhead we overhead we just talked about. So just talked about. So just talked about. So if we flip the if we flip the if we flip the Y axis so that it's Y axis so that it's Y axis so that it's bandwidth, bandwidth, bandwidth, you'll see that it you'll see that it you'll see that it resembles a Pareto frontier, resembles a Pareto frontier, resembles a Pareto frontier, where it becomes clear that where it becomes clear that where it becomes clear that as an as an as an inference provider you're forced to inference provider you're forced to inference provider you're forced to sacrifice either sacrifice either sacrifice either bandwidth bandwidth or speed.

or speed. or speed. A great example of A great example of A great example of this is the fast mode of this is the fast mode of this is the fast mode of Claude code, where you Claude code, where you Claude code, where you spend what seems to be spend what seems to be spend what seems to be 6 times as many 6 times as many 6 times as many tokens to process a tokens to process a tokens to process a single token single token single token compared to compared to compared to regular Claude, while regular Claude, while regular Claude, while getting double the getting double the getting double the speed. So, for speed. So, for speed. So, for this product, they this product, they this product, they optimize towards the optimize towards the optimize towards the right side of the right side of the right side of the Pareto frontier, Pareto frontier, Pareto frontier, gaining speed gaining speed gaining speed at the cost of not being able to at the cost of not being able to at the cost of not being able to process a large process a large process a large number of tokens. number of tokens. number of tokens. So in So in So in today's today's today's conversation, we'll conversation, we'll conversation, we'll discuss the innovations discuss the innovations we've implemented in we've implemented in we've implemented in the past and plan to implement the past and plan to implement the past and plan to implement in the future to in the future to in the future to shift the Pareto frontier shift the Pareto frontier shift the Pareto frontier to the right. You already to the right. You already to the right. You already know from our know from our know from our previous slides previous slides previous slides that the entire model needs to be read to generate that the entire model needs to be read to generate that the entire model needs to be read to generate each token each token . So, what if . So, what if we could use a we could use a we could use a single pass that single pass that single pass that we've already paid for to we've already paid for to we've already paid for to serve serve serve multiple requests? This is where multiple requests? This is where multiple requests? This is where the idea of the idea of batching and continuous batching and continuous batch processing comes in. batch processing comes in. batch processing comes in. The idea of ​​batch The idea of ​​batch The idea of ​​batch processing is to processing is to processing is to use an use an use an already completed already completed already completed scale reading to scale reading to scale reading to service service service multiple multiple multiple user requests. user requests. user requests. The problem, however, The problem, however, The problem, however, is that is that is that some requests take longer than some requests take longer than some requests take longer than others, which means there are others, which means there are others, which means there are periods of periods of computing computing power downtime when power downtime when power downtime when nothing nothing nothing happens. So how happens. So how happens. So how can we solve this can we solve this can we solve this problem and problem and problem and squeeze out more squeeze out more squeeze out more productivity? This is where productivity? This is where continuous batch continuous batch processing comes in. So, the idea processing comes in. So, the idea processing comes in. So, the idea is to is to add a new add a new user request for each free slot to user request for each free slot to be served be served during the same during the same during the same weight reading in order to weight reading in order to weight reading in order to process as process as process as many requests as possible. However, many requests as possible. However, many requests as possible. However, this has a slight this has a slight this has a slight downside, as each downside, as each downside, as each token token token needs to be saved needs to be saved needs to be saved to the KV cache when it is withdrawn, as this is how the

to the KV cache when it is withdrawn, as this is how the attention block works. You need to attention block works. You need to understand each token, understand each token, understand each token, for example, what the K for example, what the K for example, what the K and V vectors are. This can and V vectors are. This can add up very quickly. So add up very quickly. So if you run an if you run an if you run an agent to encode agent to encode agent to encode with a very long with a very long with a very long context, context, context, say 128k say 128k say 128k tokens, you will save tokens, you will save tokens, you will save 42GB of KV cache 42GB of KV cache 42GB of KV cache needed to needed to needed to handle that handle that handle that single request. So single request. So , the rule of thumb , the rule of thumb , the rule of thumb for a model with 70 for a model with 70 for a model with 70 billion billion billion parameters is: for parameters is: for parameters is: for every 3000 tokens, you every 3000 tokens, you every 3000 tokens, you need 1 GB of need 1 GB of need 1 GB of additional KV cache. And additional KV cache. And additional KV cache. And that's for one that's for one that's for one user request. Imagine user request. Imagine user request. Imagine that you use that you use that you use batch processing and batch processing and batch processing and execute eight execute eight execute eight different queries different queries different queries at the same time—then your at the same time—then your at the same time—then your KV cache will overflow KV cache will overflow KV cache will overflow and you won't have enough and you won't have enough and you won't have enough memory to memory to memory to service all the service all the service all the queries. So, what queries. So, what queries. So, what innovations have we innovations have we innovations have we implemented to implemented to implemented to optimize optimize optimize memory usage? memory usage? memory usage? This is where the This is where the paged paged KV cache comes to the rescue. Traditionally, the KV cache comes to the rescue. Traditionally, the KV cache comes to the rescue. Traditionally, the KV cache worked like this: you KV cache worked like this: you KV cache worked like this: you reserved physical reserved physical reserved physical memory to memory to memory to guarantee guarantee guarantee enough space to enough space to enough space to store all the store all the store all the KV information for KV information for KV information for each request. However, each request. However, each request. However, this had the disadvantage of being this had the disadvantage of being this had the disadvantage of being significantly significantly significantly inefficient, inefficient, inefficient, as there were as there were as there were many memory blocks many memory blocks that were that were that were reserved but reserved but reserved but not actually not actually not actually used.

used. used. The idea behind The idea behind The idea behind KV page cache is to KV page cache is to KV page cache is to use an approach use an approach similar to how an OS similar to how an OS similar to how an OS or virtual or virtual or virtual machine works, where you machine works, where you machine works, where you allocate an entire block of allocate an entire block of allocate an entire block of virtual memory virtual memory virtual memory and store the KV cache and store the KV cache and store the KV cache for each request for each request for each request wherever there is free space wherever there is free space . This allows you . This allows you . This allows you to serve three times to serve three times to serve three times more requests than more requests than more requests than without the without the without the KV page cache, simply KV page cache, simply KV page cache, simply by doing the work as by doing the work as by doing the work as before. This significantly before. This significantly before. This significantly reduces the amount of reduces the amount of reduces the amount of wasted wasted wasted space when you space when you space when you reserve reserve reserve memory slots without memory slots without memory slots without using them. using them. Immediately after that, Immediately after that, an innovation an innovation an innovation called called called prompt caching appeared. The idea prompt caching appeared. The idea prompt caching appeared. The idea is that is that is that if you have a if you have a if you have a coding agent or chat coding agent or chat application with a very application with a very application with a very long system long system long system prompt, you don't prompt, you don't prompt, you don't need to need to need to re-list the KV cache re-list the KV cache re-list the KV cache every time a every time a every time a new request comes in new request comes in new request comes in . You can . You can reuse the cache reuse the cache KV that you have already calculated KV that you have already calculated KV that you have already calculated for a previous for a previous for a previous query for the query for the query for the next one. This next one. This next one. This way, you can way, you can way, you can continue working from where you continue working from where you left off. Here you left off. Here you can skip the can skip the can skip the very, very long very, very long very, very long prefill phase prefill phase prefill phase . And you . And you might ask: might ask: might ask: what is prefill? So, what is prefill? So, what is prefill? So, during output, the process during output, the process during output, the process can be divided into can be divided into can be divided into two main phases: prefill two main phases: prefill two main phases: prefill and decoding. The and decoding. The and decoding. The prefill phase consists of prefill phase consists of prefill phase consists of all operations that all operations that all operations that occur after occur after occur after the request is received and the request is received and the request is received and ends when the ends when the ends when the first token is generated first token is generated first token is generated . You can . You can . You can see this when see this when see this when you type something into ChatGPT: it you type something into ChatGPT: it you type something into ChatGPT: it takes a few takes a few takes a few seconds before the seconds before the seconds before the first first first token appears, and then everything goes token appears, and then everything goes token appears, and then everything goes pretty smoothly. In these pretty smoothly. In these pretty smoothly. In these few seconds, while few seconds, while few seconds, while nothing is output,

nothing is output, nothing is output, the system calculates the system calculates the system calculates the vectors K and V for the vectors K and V for the vectors K and V for each input each input each input query token and query token and query token and creates the first creates the first creates the first output token. This is output token. This is output token. This is very dependent on very dependent on very dependent on computing computing computing power, as there power, as there power, as there is huge is huge is huge room for room for room for optimization here. This can be optimization here. This can be parallelized extremely efficiently. However, parallelized extremely efficiently. However, after generating the after generating the after generating the first token, you first token, you first token, you decode one decode one decode one token at a time, token at a time, reusing the KV cache reusing the KV cache created in the created in the created in the prefill phase. So, it prefill phase. So, it prefill phase. So, it depends more on depends more on depends more on memory. I'm talking memory. I'm talking memory. I'm talking about this because about this because something has recently emerged that is seriously something has recently emerged that is seriously slowing down many slowing down many slowing down many output pipelines: output pipelines: output pipelines: extremely token- extremely token- hungry hungry programming agents. programming agents. Let's say you have an Let's say you have an Let's say you have an agent to write agent to write agent to write code that code that code that uses a uses a uses a context of 200 thousand, 500 context of 200 thousand, 500 context of 200 thousand, 500 thousand or a million thousand or a million thousand or a million tokens, right? As tokens, right? As tokens, right? As a result, it a result, it a result, it starts to clog the starts to clog the starts to clog the entire conveyor. While entire conveyor. While entire conveyor. While prefill is in progress, prefill is in progress, prefill is in progress, no other no other no other prefill or decoding requests prefill or decoding requests prefill or decoding requests can pass. This is a can pass. This is a can pass. This is a real disaster real disaster real disaster if you have a lot of if you have a lot of if you have a lot of people sending people sending people sending requests to write requests to write requests to write code and a few code and a few code and a few users users users just trying to just trying to just trying to solve solve solve math problems in the math problems in the math problems in the chat. To avoid chat. To avoid chat. To avoid this this this congestion problem, there is congestion problem, there is congestion problem, there is an innovation called an innovation called an innovation called "chunked prefill" "chunked prefill" that solves that solves that solves this issue. The idea here is this issue. The idea here is this issue. The idea here is quite simple. It quite simple. It quite simple. It consists of consists of consists of taking a very long taking a very long taking a very long prefill fragment and prefill fragment and prefill fragment and breaking it into smaller breaking it into smaller breaking it into smaller parts so that parts so that parts so that different different decoding and prefill requests can be interleaved.

decoding and prefill requests can be interleaved. This way, no single This way, no single This way, no single request takes request takes request takes too long and too long and too long and blocks the entire pipeline. blocks the entire pipeline. blocks the entire pipeline. So, Michelle, tell So, Michelle, tell So, Michelle, tell us more about what us more about what us more about what else we can else we can else we can optimize. optimize. optimize. Yes, all the optimizations that Yes, all the optimizations that Daniel just talked about are at the Daniel just talked about are at the runtime level runtime level . They explain how . They explain how . They explain how to make output to make output to make output more efficient within a more efficient within a more efficient within a single worker. But single worker. But single worker. But to continue to to continue to to continue to push these boundaries, push these boundaries, push these boundaries, we really need to we really need to we really need to start looking at start looking at system-level optimizations: how to system-level optimizations: how to distribute these distribute these distribute these workloads workloads workloads across many different across many different across many different workers. And the first thing to workers. And the first thing to workers. And the first thing to realize here is that realize here is that realize here is that each request is each request is each request is unique. So, unique. So, unique. So, let's take a let's take a use case for use case for summarization as an example. You summarization as an example. You summarization as an example. You provide LLM with a bunch of provide LLM with a bunch of provide LLM with a bunch of different documents, but different documents, but different documents, but you only want to get you only want to get you only want to get an answer to an answer to an answer to one question. So one question. So one question. So this this this workload will have a workload will have a workload will have a long request and a long request and a long request and a short response. short response. short response. Therefore, it is due to the Therefore, it is due to the Therefore, it is due to the pre- pre- pre- fill phase and, therefore, fill phase and, therefore, fill phase and, therefore, is limited by is limited by is limited by computational computational computational resources. On the other resources. On the other resources. On the other hand, there may be a hand, there may be a hand, there may be a case where the case where the case where the opposite is true. Let's say you opposite is true. Let's say you opposite is true. Let's say you want to write a want to write a want to write a long story or long story or long story or book, and you just book, and you just book, and you just want to convey want to convey want to convey a few themes to a few themes to a few themes to jump off of jump off of . So here will be a . So here will be a . So here will be a short question and a short question and a short question and a long answer. And this long answer. And this long answer. And this will be due to will be due to will be due to decoding decoding decoding because you have a lot of because you have a lot of because you have a lot of output tokens output tokens output tokens compared to the compared to the compared to the number of input number of input number of input tokens. And it will be tokens. And it will be tokens. And it will be limited by memory. And limited by memory. And limited by memory. And finally, there might be a finally, there might be a finally, there might be a case that is case that is case that is something in between, something in between, something in between, right? Here, both the right? Here, both the right? Here, both the query and query and query and the response are long, and you the response are long, and you the response are long, and you can see that the can see that the can see that the pre- pre- pre- fill and fill and fill and decoding phases, and therefore both decoding phases, and therefore both decoding phases, and therefore both computation and memory, computation and memory, computation and memory, matter.

matter. Looking at these Looking at these examples, you can examples, you can examples, you can see that the see that the see that the resource requirements of your resource requirements of your resource requirements of your hardware vary significantly hardware vary significantly depending on your workload. So how do workload. So how do you avoid you avoid you avoid computational or computational or computational or memory limitations? One memory limitations? One memory limitations? One way to do this way to do this way to do this is called is called is called disaggregation. That is, disaggregation. That is, disaggregation. That is, the distribution of the the distribution of the the distribution of the workload between workload between workload between different working different working different working nodes. nodes. nodes. The most common form The most common form The most common form is called is called profile and profile and decode disaggregation, where you decode disaggregation, where you decode disaggregation, where you actually separate actually separate actually separate this workload into this workload into this workload into phases. This allows phases. This allows phases. This allows you to essentially have two you to essentially have two you to essentially have two pools. So you have a pools. So you have a pools. So you have a pre- pre- pre- populated pool that populated pool that populated pool that will handle this will handle this will handle this part of your part of your part of your workload. The decoding pool workload. The decoding pool will handle the will handle the appropriate portion of appropriate portion of appropriate portion of your workload. your workload. your workload. In the In the summing use case, you'll summing use case, you'll want want want to scale the to scale the prefill pool so you're not computationally constrained, and computationally constrained, and scale down the scale down the scale down the decode pool because you don't decode pool because you don't decode pool because you don't want to waste want to waste want to waste resources when you have resources when you have resources when you have little little little decode work to do. This is decode work to do. This is decode work to do. This is convenient because now convenient because now convenient because now you can you can you can scale them scale them scale them independently and neither independently and neither independently and neither part slows down the part slows down the part slows down the other. We can go other. We can go other. We can go even further, right? We have even further, right? We have even further, right? We have already said that already said that already said that pre-filling itself is pre-filling itself is inherently inherently computationally limited.

computationally limited. computationally limited. Decoding Decoding Decoding is limited by memory. To is limited by memory. To is limited by memory. To quickly recap: quickly recap: quickly recap: the reason is that the reason is that the reason is that when decoding, you when decoding, you when decoding, you go through go through go through each token in turn, each token in turn, each token in turn, and at each step you and at each step you and at each step you reread the entire reread the entire reread the entire KB cache, right? This will be KB cache, right? This will be KB cache, right? This will be limited by memory. limited by memory. So, yes, what So, yes, what Michelle just said Michelle just said Michelle just said about about about memory limitations can be clearly memory limitations can be clearly memory limitations can be clearly seen in the seen in the seen in the capabilities of the capabilities of the capabilities of the hardware itself. So, hardware itself. So, hardware itself. So, GPUs, GPUs, GPUs, right? The speed at which right? The speed at which data can be transferred from memory to data can be transferred from memory to the computer is limited by a the computer is limited by a the computer is limited by a parameter parameter parameter called called memory bandwidth. For the H100, this is memory bandwidth. For the H100, this is 3.35 3.35 3.35 terabytes per second. terabytes per second. terabytes per second. However, Cerebras has a However, Cerebras has a However, Cerebras has a throughput throughput throughput 7,000 times higher than the 7,000 times higher than the 7,000 times higher than the H100, at 21,000 terabytes H100, at 21,000 terabytes H100, at 21,000 terabytes per second. This is because per second. This is because per second. This is because we use a we use a we use a very specialized very specialized SRAM-based memory architecture on SRAM-based memory architecture on our chip. This our chip. This our chip. This means that Cerebras is means that Cerebras is means that Cerebras is truly built truly built truly built for decoding. So for decoding. So for decoding. So we can process we can process we can process 3000 tokens per second 3000 tokens per second , while the H100 outputs 600 , while the H100 outputs 600 , while the H100 outputs 600 tokens per second tokens per second tokens per second during decoding.

during decoding. We are very excited We are very excited to partner with AWS to partner with AWS to partner with AWS to bring this to to bring this to to bring this to production. We will production. We will production. We will partner with partner with partner with AWS Trainium to AWS Trainium to AWS Trainium to perform perform perform prefill on prefill on prefill on their AWS Trainium chips, their AWS Trainium chips, their AWS Trainium chips, using the using the using the EFA service to EFA service to EFA service to transfer the KV cache to transfer the KV cache to transfer the KV cache to our Cerebras CS-3 systems our Cerebras CS-3 systems our Cerebras CS-3 systems for final for final for final token withdrawal. This will token withdrawal. This will token withdrawal. This will provide us with 5 provide us with 5 provide us with 5 times faster times faster times faster token processing speed for the token processing speed for the token processing speed for the same physical same physical same physical system size. This is system size. This is system size. This is one of the steps we are taking one of the steps we are taking one of the steps we are taking to make to make to make rapid inference more rapid inference more rapid inference more accessible and accessible and accessible and affordable for affordable for affordable for developers. We are very developers. We are very developers. We are very excited about the opportunity excited about the opportunity excited about the opportunity to develop this to develop this to develop this partnership. partnership. partnership. And at Gimlet, we extend And at Gimlet, we extend And at Gimlet, we extend this approach far this approach far this approach far beyond collaboration with just beyond collaboration with just beyond collaboration with just Trainium and Cerebras. Gimlet Trainium and Cerebras. Gimlet Trainium and Cerebras. Gimlet essentially creates a essentially creates a essentially creates a multi-chip multi-chip multi-chip inference cloud that inference cloud that inference cloud that takes your takes your takes your workload, workload, workload, breaks it down into breaks it down into breaks it down into smaller pieces, smaller pieces, smaller pieces, determines the determines the resource characteristics for each of resource characteristics for each of them, and then them, and then them, and then finds the best finds the best finds the best way to distribute it across the way to distribute it across the way to distribute it across the underlying hardware, underlying hardware, underlying hardware, which can be which can be anything. Nvidia, AMD, Trainium, Cerebras, anything. Nvidia, AMD, Trainium, Cerebras, anything. Nvidia, AMD, Trainium, Cerebras, D-Matrix, Intel Gaudi. This D-Matrix, Intel Gaudi. This D-Matrix, Intel Gaudi. This list could list could list could go on for a long time. go on for a long time. go on for a long time. By running these By running these By running these tasks on the tasks on the tasks on the devices that are devices that are devices that are most efficient for most efficient for most efficient for them, we get them, we get them, we get simply incredible simply incredible simply incredible results. We results. We results. We observe a observe a three to tenfold speedup in inference compared to compared to homogeneous homogeneous homogeneous systems with the same systems with the same power consumption.

power consumption. This goes beyond This goes beyond This goes beyond simply separating the simply separating the simply separating the filling and filling and filling and decoding phases, and some decoding phases, and some decoding phases, and some of these methods of these methods of these methods include, for example, include, for example, model-based rather than model-based rather than phase-based separation. This is called phase-based separation. This is called phase-based separation. This is called speculative speculative speculative decoding. decoding. decoding. Going back to the Going back to the Going back to the decoding phase: decoding phase: decoding phase: you usually you usually you usually do a direct do a direct do a direct pass for each pass for each pass for each token you token you token you get on the output. get on the output. get on the output. This is the standard This is the standard This is the standard way of doing it. way of doing it. way of doing it. Speculative Speculative Speculative decoding is where decoding is where decoding is where you you you use a use a use a draft model. It is draft model. It is draft model. It is simply a scaled-down simply a scaled-down simply a scaled-down version of the full model, version of the full model, version of the full model, which can generally which can generally which can generally approximate approximate the results produced by the the results produced by the main model. And the main model. And the main model. And the nice thing here is nice thing here is nice thing here is that because that because that because it's smaller, it it's smaller, it it's smaller, it runs much runs much runs much faster and also faster and also faster and also consumes fewer consumes fewer consumes fewer resources. So, resources. So, resources. So, during speculative during speculative during speculative decoding, you decoding, you decoding, you can, for example, can, for example, can, for example, get the get the get the next nine tokens next nine tokens next nine tokens by first running by first running by first running the decoding on this the decoding on this the decoding on this rough model. It will rough model. It will rough model. It will issue you with so-called issue you with so-called issue you with so-called candidate tokens. You are candidate tokens. You are candidate tokens. You are not 100% sure that not 100% sure that not 100% sure that all of these tokens are all of these tokens are all of these tokens are correct, but correct, but correct, but because they have been frequently because they have been frequently because they have been frequently tuned or tuned or tuned or trained on the full trained on the full trained on the full model, they are model, they are model, they are approximately correct. approximately correct. approximately correct. And you want And you want And you want to check if these are the to check if these are the to check if these are the tokens I tokens I tokens I actually want to actually want to actually want to withdraw? So you withdraw? So you withdraw? So you run the next run the next run the next stage on this full stage on this full stage on this full model, model, model, called the called the called the verification stage, to verification stage, to verification stage, to confirm which of confirm which of confirm which of these tokens are these tokens are these tokens are correct. And, in fact, correct. And, in fact, correct. And, in fact, you can you can you can keep keep keep doing this until you've doing this until you've doing this until you've tested all of these tested all of these tested all of these tokens from the draft tokens from the draft tokens from the draft model. After all, you model. After all, you model. After all, you see here at the top that see here at the top that see here at the top that with the standard with the standard with the standard way of starting, you would have had to way of starting, you would have had to do a straight do a straight pass on the full pass on the full pass on the full model nine times model nine times model nine times to get those to get those to get those nine tokens. On nine tokens. On nine tokens. On the other hand, with the other hand, with the other hand, with speculative speculative speculative decoding, you decoding, you only needed to run it on the full model

only needed to run it on the full model three times. And if you three times. And if you three times. And if you can get this can get this can get this rough model rough model rough model working very, very working very, very working very, very quickly, then this time quickly, then this time quickly, then this time will be minimal, and will be minimal, and will be minimal, and you can significantly you can significantly you can significantly reduce the time reduce the time reduce the time to get tokens. to get tokens. Looking at the more Looking at the more specific specific specific results, this is results, this is results, this is essentially the Pareto frontier essentially the Pareto frontier essentially the Pareto frontier that Daniel showed earlier that Daniel showed earlier that Daniel showed earlier , where the X-axis is the , where the X-axis is the token acquisition rate and the Y-axis is the throughput. the throughput. the throughput. So the basic option So the basic option So the basic option we are we are we are comparing here is comparing here is comparing here is split profile split profile decoding on decoding on decoding on homogeneous homogeneous homogeneous GPUs. You can see GPUs. You can see that the that the that the throughput is pretty throughput is pretty throughput is pretty good, but as soon as good, but as soon as good, but as soon as you increase you increase you increase the speed, it the speed, it the speed, it drops dramatically. So what if you drops dramatically. So what if you drops dramatically. So what if you run run run speculative speculative speculative decoding on those decoding on those decoding on those same same same GPUs? Now you GPUs? Now you GPUs? Now you run this run this run this rough model and rough model and rough model and see that the curve see that the curve see that the curve shifts to the right. shifts to the right. shifts to the right. The reason for this is The reason for this is The reason for this is that you run that you run that you run the decoding on the the decoding on the the decoding on the full model fewer full model fewer full model fewer times. But times. But times. But we can go even we can go even we can go even further, right? further, right? further, right? Returning to Returning to Returning to our topic of running our topic of running our topic of running workloads workloads workloads on the hardware that is on the hardware that is on the hardware that is best suited for it best suited for it best suited for it . On something . On something . On something like Matrix or like Matrix or like Matrix or Cerebrus, where you can Cerebrus, where you can Cerebrus, where you can run small run small run small models very, very models very, very models very, very quickly, you'll see a quickly, you'll see a quickly, you'll see a significant significant significant performance boost. So performance boost. So performance boost. So here you see these here you see these here you see these numbers for the same numbers for the same numbers for the same bandwidth bandwidth , right? You can , right? You can , right? You can get 10x get 10x get 10x better better better interactivity interactivity interactivity compared to if compared to if compared to if you were to run this only you were to run this only you were to run this only on homogeneous on homogeneous on homogeneous machines. The last machines. The last machines. The last form of disaggregation form of disaggregation form of disaggregation we wanted to talk about we wanted to talk about is called attention-FFN disaggregation. And it gets attention-FFN disaggregation. And it gets quite interesting when quite interesting when quite interesting when you think about you think about you think about servicing servicing servicing large models. I'm large models. I'm large models. I'm talking about models with 3 talking about models with 3 talking about models with 3 trillion trillion trillion parameters, where these parameters, where these parameters, where these models are so models are so models are so large that even large that even large that even running just one

running just one running just one layer can be quite layer can be quite layer can be quite resource-intensive. So resource-intensive. So resource-intensive. So the concept is essentially the same the concept is essentially the same the concept is essentially the same , right? For a , right? For a , right? For a specific task specific task that you need to that you need to that you need to perform, whether it's an perform, whether it's an perform, whether it's an attention layer or a feedforward attention layer or a feedforward attention layer or a feedforward neural network ( neural network ( neural network ( FFN) layer, FFN) layer, FFN) layer, you're essentially you're essentially you're essentially trying trying trying to figure out: what are the to figure out: what are the resource characteristics, and what resource characteristics, and what hardware is best hardware is best hardware is best suited for it? The suited for it? The suited for it? The attention layer will be more attention layer will be more attention layer will be more memory constrained. memory constrained. memory constrained. It makes many It makes many It makes many KB cache hits. The KB cache hits. The KB cache hits. The FFN layer performs many FFN layer performs many FFN layer performs many dense dense dense matrix multiplications. So that makes matrix multiplications. So that makes matrix multiplications. So that makes sense, right? You sense, right? You sense, right? You will run attention will run attention will run attention on a machine with on a machine with memory bandwidth, and the FFN layer on memory bandwidth, and the FFN layer on machines that can machines that can machines that can perform these perform these perform these matrix multiplications matrix multiplications matrix multiplications as quickly as possible. Essentially, what as quickly as possible. Essentially, what as quickly as possible. Essentially, what happens is this: happens is this: happens is this: even within a even within a even within a model, you're ping- model, you're ping- ponging between ponging between ponging between many different many different many different machines to machines to machines to get an output get an output get an output from that model, really from that model, really disaggregating each disaggregating each individual layer. And one more individual layer. And one more individual layer. And one more thing we wanted to point out thing we wanted to point out thing we wanted to point out : there : there : there is no single is no single is no single best form of best form of best form of disaggregation. disaggregation. disaggregation. In fact, there is no such thing as the In fact, there is no such thing as the In fact, there is no such thing as the best best best equipment. It equipment. It equipment. It really depends on the really depends on the really depends on the tradeoffs tradeoffs tradeoffs you make depending on you make depending on you make depending on your model, your model, workload, workload, latency requirements, and latency requirements, and latency requirements, and bandwidth guarantees bandwidth guarantees bandwidth guarantees . Looking . Looking . Looking to the future, we see to the future, we see to the future, we see agents increasingly agents increasingly embedded in embedded in our daily our daily our daily lives. They are lives. They are lives. They are being implemented into being implemented into being implemented into our work. Everyone our work. Everyone our work. Everyone uses them. And uses them. And uses them. And this continues to this continues to this continues to drive the need drive the need drive the need for more for more for more faster tokens.

faster tokens. faster tokens. Yes, this is indeed Yes, this is indeed Yes, this is indeed headed in the headed in the headed in the right direction. right direction. right direction. I told Daniel not to I told Daniel not to I told Daniel not to say that. But the say that. But the say that. But the good thing is that while this good thing is that while this good thing is that while this seems like a crazy seems like a crazy seems like a crazy goal, we can actually goal, we can actually use use agents to agents to agents to help us help us help us solve these problems solve these problems solve these problems and keep and keep and keep moving forward. One moving forward. One moving forward. One of the things we do at of the things we do at of the things we do at Gimlet is Gimlet is Gimlet is use an agent use an agent use an agent to generate to generate to generate optimized kernels. optimized kernels. optimized kernels. Especially for our Especially for our Especially for our use case use case , where we want to , where we want to , where we want to support a wide support a wide support a wide range of different range of different range of different hardware, we hardware, we hardware, we need to figure out need to figure out need to figure out how to make this how to make this how to make this problem much problem much problem much simpler. Because right simpler. Because right simpler. Because right now people generate now people generate now people generate and optimize kernels and optimize kernels and optimize kernels manually, spending manually, spending manually, spending a lot of time a lot of time a lot of time tweaking and tweaking and tweaking and trying to figure out how to trying to figure out how to trying to figure out how to optimize them. The thing is, optimize them. The thing is, optimize them. The thing is, these these these optimizations can optimizations can vary quite significantly depending vary quite significantly depending on the model you're on the model you're on the model you're trying trying trying to run, the to run, the to run, the hardware hardware hardware it's running on, and even it's running on, and even it's running on, and even the generations of that the generations of that the generations of that hardware or the hardware or the hardware or the workload structure itself workload structure itself workload structure itself . So, at . So, at . So, at Gimlet, we created KForge—an Gimlet, we created KForge—an agent for generating agent for generating agent for generating such kernels. It such kernels. It such kernels. It collects a bunch of collects a bunch of collects a bunch of information: profiles of information: profiles of information: profiles of your hardware, your hardware, your hardware, profiles of profiles of profiles of workloads—really workloads—really workloads—really anything that will help anything that will help anything that will help figure out how figure out how figure out how to optimize the cores, and to optimize the cores, and to optimize the cores, and we solve it as a we solve it as a we solve it as a brute force problem brute force problem , you know?

, you know? , you know? Here's an example, right? Here's an example, right? Here's an example, right? We have a We have a We have a forward pass and a kernel that forward pass and a kernel that forward pass and a kernel that KForge has optimized KForge has optimized KForge has optimized based on it. And we based on it. And we based on it. And we see quite interesting see quite interesting see quite interesting results. Yes, results. Yes, results. Yes, last week we last week we last week we ran the agent for ran the agent for ran the agent for about less than about less than about less than a week, and during that a week, and during that a week, and during that time it was able to time it was able to time it was able to speed up GPT-Neo 120B speed up GPT-Neo 120B speed up GPT-Neo 120B on AMD hardware by 33% on AMD hardware by 33% from start to finish. from start to finish. from start to finish. And it's interesting to note And it's interesting to note And it's interesting to note that this is based on that this is based on that this is based on kernels that have already been kernels that have already been kernels that have already been optimized before. optimized before. You can take You can take agents to the next agents to the next agents to the next level and level and level and use them use them use them to help us to help us to help us create and create and create and design design design specialized specialized specialized chips that are chips that are chips that are hyper-tuned to hyper-tuned to hyper-tuned to specific models or specific models or specific models or your specific your specific your specific workloads. workloads. workloads. This can be This can be This can be used to used to used to train even train even train even better, smarter, better, smarter, better, smarter, and faster AI, which in and faster AI, which in and faster AI, which in turn can be turn can be turn can be used to used to used to develop the next develop the next develop the next generation of such generation of such generation of such specialized specialized specialized chips. And, you know, chips. And, you know, chips. And, you know, this cycle this cycle this cycle goes on goes on goes on forever, it's a forever, it's a forever, it's a recursive cycle— recursive cycle— congratulations to the Recursive team. congratulations to the Recursive team. I want to end my I want to end my talk by saying that talk by saying that talk by saying that we are just beginning we are just beginning we are just beginning our journey towards our journey towards our journey towards optimizing optimizing optimizing throughput throughput throughput and and and inference speed. Because inference speed. Because inference speed. Because at the very top of the stack, at the very top of the stack, at the very top of the stack, you don't really you don't really you don't really think about think about output optimization when you're output optimization when you're talking about talking about incident response, right? But incident response, right? But when you have agents when you have agents when you have agents that make our that make our that make our systems much more systems much more systems much more reliable and reliable and reliable and increase increase increase uptime, uptime, uptime, we get higher we get higher we get higher throughput throughput throughput and can generate and can generate and can generate more tokens.

more tokens. more tokens. Or you can Or you can Or you can use agents use agents use agents to solve to solve model launch problems. As soon as a model model launch problems. As soon as a model appears on the market, appears on the market, appears on the market, ensure that it ensure that it ensure that it works on all types of works on all types of works on all types of equipment on the very equipment on the very equipment on the very first day. first day. first day. You can even go even You can even go even You can even go even bolder with bolder with bolder with compilers. compilers. compilers. Imagine a world where the Imagine a world where the Imagine a world where the output stack can be output stack can be output stack can be individually individually individually designed for your designed for your designed for your workload and the workload and the workload and the model you are model you are model you are running. This is already running. This is already running. This is already being demonstrated by colleagues being demonstrated by colleagues being demonstrated by colleagues from Anthropic, right? They from Anthropic, right? They from Anthropic, right? They created the compiler created the compiler created the compiler in 2 weeks. So I in 2 weeks. So I in 2 weeks. So I believe there are a believe there are a believe there are a lot of optimizations lot of optimizations lot of optimizations that can be made that can be made that can be made when you build when you build when you build specialized specialized software on software on the fly to support the fly to support the fly to support the model you want to the model you want to the model you want to run, on the run, on the workload you workload you need, and on your need, and on your need, and on your own hardware. own hardware. own hardware. Going back to the Going back to the Going back to the previous slide, previous slide, previous slide, we use we use we use agents to continue to agents to continue to agents to continue to build more build more build more specialized specialized specialized chips and chips and chips and speed up the process of speed up the process of speed up the process of manufacturing them. manufacturing them. manufacturing them. And finally, this is And finally, this is And finally, this is not the technical part at all, not the technical part at all, not the technical part at all, but it is all but it is all but it is all about about inference maintenance. All inference maintenance. All procurement, all procurement, all procurement, all legal issues, everything that legal issues, everything that legal issues, everything that happens in the happens in the happens in the back office. All of this, back office. All of this, back office. All of this, combined with combined with combined with agent agent agent workflows, can be workflows, can be workflows, can be optimized. So once optimized. So once optimized. So once we do all of we do all of we do all of this, I'll be very this, I'll be very this, I'll be very optimistic optimistic optimistic about the future and about the future and about the future and what we can what we can what we can achieve to provide achieve to provide achieve to provide tokens faster and tokens faster and tokens faster and cheaper. So, cheaper. So, cheaper. So, next slide, next slide, next slide, please. Thank you very much for please. Thank you very much for please. Thank you very much for coming to our coming to our coming to our talk. We are both talk. We are both working very, very hard to get you get you tokens faster and cheaper. And yes, let me tokens faster and cheaper. And yes, let me know if you have any know if you have any know if you have any questions. We will be there questions. We will be there questions. We will be there after the performance.

after the performance. after the performance. Yes, and go Yes, and go Yes, and go consume tokens. We are consume tokens. We are consume tokens. We are creating creating creating more tokens for you so that more tokens for you so that more tokens for you so that you can also you can also you can also generate such GIF generate such GIF animations.