Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod - Chris Vela, Runpod

[music] My name is Chris Bella. I work at RunPod. I'm a forward deploy engineer. Today I'm going to be talking about benchmarking VLMs. Kind of like the struggle that people have when like especially like a new model comes out. Just some like learnings that we've had like internally at RunPod and with like customers. So today I'm going to be showing you essentially like a dashboard that is built on top of like VLM. It's using our like pods actually. So to give you like a little more context of like what RunPod does is we're in the old cloud. We make it really really easy for you to rent GPUs and starting from like R&D to like more service production workflows. A lot of times what I see with customers especially with VLM is like someone wants you know the latest model. They don't know how to like spin it up or they they want to start going to production they don't know the the knobs to like touch. They're just using whatever like the base config and I'm guessing a lot of you probably seen like VLM recipes before.

You know it's a good start especially like but like especially in this world we're always moving fast. So how do you get to more production level style of code and at the same time not just like vibing it. It's more like how do you actually have data? How do you show you know your manager like your staff engineer that hey this is actually better. This is actually working correctly. So really what I'm trying to say is like I've seen a lot everyone's just guessing. Like you know you're a lot of times I see people are using the wrong VLM version. They might they're probably not using like the nightly version. They have the wrong size for their context window. It really depends like I think always a step back. It's like what are you trying to achieve with your your LM model? Are you is it going to be a coding agent? Is it more kind of like you're just you know doing summaries like very very basic tasks. That's usually the first question I'd be asking before you know getting Kim K 3 having a you know a million long context. Like that's not always necessary.

And at the same time it's like how many users are going to be using this? Um a lot of times people are like going everyone's trying to always get like the the smallest amount of GPU for the cheapest price. But if you're going to be going to a more concurrent users, you need to start thinking about like how am I going to be utilizing the KB cache. So all this is there's all these little like optimization knobs that you need to like figure out before you go to production. And a lot of times we're trying to move fast. So um yeah like like I said like RunPod like we like have very much like you know we make it very very easy it's like a VM. It's dockerized. You can bring your own image. But at the end of the day it like like who here has used like VLM >> before. >> Okay, so you know, like, when you're looking for, like, like especially like Kim K 3 like, if you you use, like, a version of, like, you know, 0.27, it's not going to be working. Like, like, we're moving so fast, the code is not getting as fast; like, you need to have the right version, and a lot of times that turns into kind of, like, a CD/CICD process that, especially if people who don't understand what's happening, just start struggling. They they start seeing errors and they're like why is this not working you know it the the layers within it are not they haven't been written or they're it's very in development code.

And at the same time that's actually kind of a problem is like Kim K 3 comes out you know the next day I would not put that on production just yet. You're going to be seeing a lot of probably hallucinations. It's just not ready in that state I would say. Um so what I propose so I created this thing called it's called BenchPod. It's going to be a template. It hasn't come out yet. So it's essentially built on top of like VLM. It's a whole metric system. You you very easy. And a lot of you who are familiar with like v l m you can benchmark already, but this kind of um centralizes everything, so you can be able to see everything, be able to change the config very quickly. Uh that's like a very important thing that I I notice is that uh especially with people who use like our uh our pods is you set up a config, but then you have to restart the image. You have to wait for cold starts to happen. It's all this loading, and you're not moving as fast. You're waiting like 30 minutes like for Kim 83 to like download the weights, load into GPU. Uh it's kind of wasted time. I mean, if you have an agent, you know, you really don't care, but um that is a pain point, especially when we're trying to move fast. And at the same time it's like the cool thing about Runpod is you can find these machines.

So, if you have that question, it's like, "I want to run Kim 83 on a B300." We have B300s, and you can start going from like not just like reading about it. You can actually start testing it and seeing if, you know, there's truth to all these metrics. So, yeah, so like bench like like I said, like Benchpod is more like um it's it's built like a of a a Docker image of um um v l m open API. We're going to be having every version already. Um we just have a little like like payload that like pretty much creates this whole fast API application. Uh you can use it for your agents. Uh the cool thing is you can serve uh you can pretty much specify which uh models you want to have. We have a whole watchdog, too, so it like self-heals, too. So, uh if you are actually starting to go to more production levels, uh you can actually uh you don't have to worry about it. And then like I will show you uh like this past Sunday, uh Kim went down on our instance, and it uh self-healed. And, you know, we have we have right now like four nodes running it, so none of like our internal users saw any problem.

It was a production ready. Um didn't have to like think about it too much. And at the end of day, yeah, like it's it it makes it much more sim-uh simple process to uh have production work workflows. So, I'm guessing a lot of people I'm I'm going to show you Kim and K3 because we've been using that a lot. Um I'm guessing a lot of people here are very familiar with Kim and K3. Uh so, you know, 1 1 million contacts. Uh one thing that's interesting on 1B 300 uh can't get to the 1 million. You got to use half of it. But one one thing I've noticed especially with our team like we're using it a lot for coding uh agents. Um that is very much um enough to do long task. Um It's utilizing KB cache uh very well, too. So, um in terms of speed of like time to first second um it is very very very quick. And like in the demo I'll show you too with like my coding agent running this Kim and uh model on uh RunPod.

So, this is like our dashboard. Uh um It has a lot of metrics. You know, it can it look looks like a lot, but uh essentially, you know, we're always thinking about like the GPU like, you know, have a lot of like GPU uh metrics, you know, and what CUDA version you're on, which is very very important. You know, if you're running Kim and it doesn't work on a 12 8 CUDA. It works on 13.0. Uh and really it's what we have is like the best we can we we can look at benchmarking, look at all our data. We can like playground with it. So, like before you even put it into your like coding agent or uh just hitting the OpenAI uh response, you can start testing real quick if this is working. And just a lot of like more CICD process of like, you know, how long are my cold starts happening? Um how are how are my metrics looking um through actual traffic? So, I'm going to show you guys the demo. So, So, really like I'm going to show you real quick like RunPod itself. So, um this will So, we don't have the sample it up yet. Like I said, um it'll be up probably at the end of the month. Uh but really all we have is just like a Docker image. We can be able to uh put that in right there.

If it loads correctly. But uh Uh, yeah, so right now there's no B300s, but uh, I have one running right now. That's the fun thing about working in this industry. Uh, you know, uh, supply can be uh, dependent. Uh, but [laughter] But yeah, it's in in the sense of that like, you know, we make it really easy. You can put storage volume. Uh, you deploy it, you know, we have a part two so you can deploy when you're available. So, you know, if you were seeing the screen like this, it will deploy whenever it get it get started. Uh, so that's like literally the process. You click that button, benchpod like starts up. Um, and I can show you like right here. I have it on the wrong one, but I'll show you guys here. So, like this is like this is our actual like Kimi running currently. Um, so a cool thing is like you know, you you start like it has a whole air scan so you have actual data to know like what is happening. Um, cool thing is we have live metrics too as like things are running. Uh, all of this is also like backed up in like storage data. So like it's all um, um, backed up no matter what. Um, it's all face fast API calls so your agent could easily just call all this data.

You can actually just serve everything. Like you don't even have to use the UI. I I think I'm a big proponent of like as we go to agents it's like I still want to have like some awareness of what's happening and having this gives me a good like like eagle's eye view of like what is happening with my uh, Kimi K3 K3 instance. Uh, but yeah, like you can see here like I was going to show you like uh, this past weekend we had a failure with a GPU. Uh, we were able to fix it, but uh, being able to just do a cold start time like 9 minutes on it like K3 is pretty pretty good. Um, and the cool thing here is that we have the serve so you can start out experiments and essentially like apply them. It'll restart real quick. Um, you can it's literally just like the config YAML file from like VLM. And um, so like for this like little demo what I tested was we were just changing the max number of batch tokens uh, that were sent through. Uh, And I kind of wanted to show you guys of like, you know, what what kind of like speed ups would happen with this. Um I'll go right here.

So, I run these before because we're not going to spend like 20 minutes just looking at benchmarks. So, these benchmarks are running like concurrency. So, they're they're running um you know, starting with like just one concurrent user and then eight, 64, 192. And the cool thing is like we can be at we can actually compare multiple runs against each other. You can kind of see like even just increasing the match the the number of batch tokens doesn't really change that much the like time to first second. But, interesting thing that I did notice of increasing that is like the inner token latency actually decreases quite a bit when you increase that batch tokens. Which means it's like the request itself, if it's especially running like a long um like job essentially, it's actually much faster finishing the job than the base like config that like you find on VM recipe. The cool thing is is like you can add this to an agent and just start going crazy with it. And you can start getting those answers much much quicker, have the data, and move from this. If you have it like on a pod, you can start moving to, you know, a multi-node cluster. You can move to serverless, too. So, if you're going to be like scaling up and down, that that this is a perfect way of like actually getting data. And just to give you like a context on our serverless,

like serverless a lot of people will do the same thing. They they just put whatever the base config is and then are struggling as like they're waiting for workers to start up. This actually gets you the real data. You're now ready to go to production. Makes it really really simple. And just gives It's like I always say, it's basing everything in facts. And I can like show you guys right here. Like this is actually like Kenny writing. So I was just like, what is a bench pod? I misspelled it. Sorry. But that this is little command instance running through it. Um yeah, and um you know, it is pretty pretty quick. Like like I would say like there's like a probably like 50 users right now on our internal one and we most of us are getting off cloud and just using the and we have other instances with like GLM, uh deep seek, uh quant 3.8. Um but yeah, like I think in the future we're looking to more at like smart routers. Like how how do we host all this without being reliant on someone else?

Um Yeah, so let me go back to our uh thing real quick. So some like important data like we were just talking about it. If anyone wants to screenshot that. But uh like I said, like you can you can specify the experiment yourself too. So if you were like, I'm going to have like 500 people using this all the time. Like I want to like see how well this is going to be. Um but at the end of the day, like I said, it is how do we go from R&D to production? And I think this is where like bench pod will make it a lot more easier for people to use. Um So like at the end of the day, like what I want you guys to like know is like um you know, pick a pod, get like a GPU machine. You have your model, compare configs. Don't just start with like the VLN recipe. The VLN recipe is proven, but it's not production ready. Um and then test. Have your agent tested. And then whatever, you know, based on what you need, if it's ITL, if you need TTF, TPOT, like metric, like optimize on that. And then ship that winner. Go to service, scale out, and now you can like um now you have a working LLM model that can be used internally, can be used for our Um Yeah, it's it just makes it very very simple.

But yeah, one thing I would say especially with like Kimmy that I've noticed is um it's not always necessary for a team. Like like a big thing that I think like most people don't think about is that um you can use a smaller model. You know, it it's a beta 100 right now like a node. That's like on our on our platform like that's going to cost you 40K a month essentially. And it might go up, it might go down a little, but you know, it's a very powerful machine and I think when you're thinking about like how you're going to be um serving stuff if you want to be doing this internally or for yourselves is like what makes sense the best. And just because it's the the frontier model, maybe it doesn't make sense. Um like I said like like I run a deep six on a 4 H 200s and does a really good job. Kimmy's more for like long tasks that I'm trying to solve something like very very detailed.

But like I said, the other thing is like making sure your like tooling choice is like on there. That's a big thing. Context like I I'm not going to be going through too much about like context like wikis and all that, but the ability to have all that and like AI governance is the next step that we are going to be going with RunPod, but just something to be always be thinking about especially if you're going to be self-hosting your um your LLM models. But yeah, is there any questions? Like And we'll also be at the booths. So like downstairs. But yeah, add me on LinkedIn if you want to talk more. I'm always exploring every new model like every week. So always have some like gotchas that I've been learning through the through the time I've been doing all this work. And yeah, also scan the card QR code so you can get you get some credits and you can test out RunPod. And and said, the template will be out like at the end of the month. So you can go play around with it.