Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

I am Nicholas Arcolano. I am the I am Nicholas Arcolano. I am the head of head of head of artificial intelligence artificial intelligence artificial intelligence and research at Jellyfish. and research at Jellyfish. and research at Jellyfish. It's great to be here again It's great to be here again at AI Engineer. And today at AI Engineer. And today at AI Engineer. And today I want to talk about I want to talk about I want to talk about why, if you're not why, if you're not why, if you're not getting the results you want getting the results you want with AI, um, the problem is probably not with the model, problem is probably not with the model, it's with you. Um, since it's with you. Um, since it's with you. Um, since we're here, um, you know, we're we're here, um, you know, we're we're here, um, you know, we're dealing with this dealing with this dealing with this financial topic financial topic , we're dealing with , we're dealing with , we're dealing with leadership, we'll leadership, we'll leadership, we'll start with a start with a start with a golf analogy. It seems golf analogy. It seems golf analogy. It seems appropriate. Um, is there appropriate. Um, is there appropriate. Um, is there anyone here who plays anyone here who plays anyone here who plays golf? Is anyone into golf golf? Is anyone into golf ? If you are, fine. Um, ? If you are, fine. Um, ? If you are, fine. Um, yeah, if you don't play yeah, if you don't play yeah, if you don't play golf, bear with golf, bear with golf, bear with me. I apologize, me. I apologize, me. I apologize, if you play if you play if you play golf, I golf, I golf, I apologize too. This is a very apologize too. This is a very apologize too. This is a very painful sport. painful sport. painful sport. Um, I like golf. Um, Um, I like golf. Um, Um, I like golf. Um, I'm a bad golfer. I'm a bad golfer. I'm a bad golfer. Um, it's part of the Um, it's part of the Um, it's part of the culture—saying culture—saying culture—saying you're a bad golfer, you're a bad golfer, you're a bad golfer, but I'm really a bad but I'm really a bad but I'm really a bad golfer, and when you're out golfer, and when you're out golfer, and when you're out on the course, you on the course, you on the course, you just want to just want to just want to get rid of the pain, and get rid of the pain, and get rid of the pain, and one of the things one of the things one of the things you think about is, "Oh, you think about is, "Oh, you think about is, "Oh, well, shouldn't well, shouldn't well, shouldn't I just buy I just buy I just buy better clubs, right?" better clubs, right?" better clubs, right?" That would simplify things That would simplify things . And then you have an . And then you have an . And then you have an instructor and he instructor and he instructor and he says, "No, like you're the says, "No, like you're the says, "No, like you're the problem. You don't problem. You don't problem. You don't need better need better need better clubs, do you?" Well, it's about the clubs, do you?" Well, it's about the clubs, do you?" Well, it's about the basics. But, basics. But, basics. But, you know, I'm a you know, I'm a you know, I'm a data scientist, so I'm data scientist, so I'm data scientist, so I'm like, "I don't know if like, "I don't know if like, "I don't know if that's really true?" Um that's really true?" Um , if you look at , if you look at , if you look at the data, the clubs the data, the clubs the data, the clubs matter. The sticks matter. The sticks matter. The sticks have become better. So what have become better. So what have become better. So what you see here you see here you see here is the average is the average is the average distance distance distance traveled by a traveled by a traveled by a PGA Tour ball, uh, from 1980 PGA Tour ball, uh, from 1980 PGA Tour ball, uh, from 1980 to the present. In to the present. In to the present. In 1980, I was 1980, I was 1980, I was shocked. I did shocked. I did shocked. I did n't actually n't actually n't actually know that. The know that. The know that. The average was 257 average was 257 average was 257 yards. For example, I yards. For example, I yards. For example, I hit the ball so hit the ball so hit the ball so far, and as I far, and as I far, and as I mentioned, I'm a terrible mentioned, I'm a terrible mentioned, I'm a terrible golfer. Er, golfer. Er, golfer. Er, so since 1980, its so since 1980, its so since 1980, its number has increased

number has increased number has increased by about 20%. And by about 20%. And by about 20%. And you know, one of the you know, one of the you know, one of the proofs that proofs that proofs that clubs clubs clubs matter is the era of balls matter is the era of balls matter is the era of balls with a hard with a hard with a hard core core core in the middle, uh, before in the middle, uh, before in the middle, uh, before 2000. There were a lot of 2000. There were a lot of 2000. There were a lot of technical and technical and technological changes, and that's when the technological changes, and that's when the technological changes, and that's when the 15- 15- yard climb happened. So yard climb happened. So yard climb happened. So people haven't become better in people haven't become better in people haven't become better in any way in these 3 any way in these 3 any way in these 3 years, as if it was all years, as if it was all years, as if it was all due to due to due to technology. And this technology. And this technology. And this chart chart chart reminded me a lot of what reminded me a lot of what reminded me a lot of what we see with models we see with models we see with models at the turn, right? at the turn, right? at the turn, right? We see a steady We see a steady We see a steady progression of models that are progression of models that are progression of models that are getting better and getting better and getting better and better. This is a better. This is a better. This is a synthetic analysis chart. I synthetic analysis chart. I synthetic analysis chart. I assume everyone assume everyone assume everyone has seen one or many has seen one or many has seen one or many of these and checks them out of these and checks them out of these and checks them out every time a every time a every time a new model comes out, right new model comes out, right ? So, the models are getting ? So, the models are getting ? So, the models are getting better and better better and better . I think that's . I think that's . I think that's for sure. We can for sure. We can for sure. We can do things we do things we do things we just couldn't just couldn't just couldn't do a year ago, do a year ago, do a year ago, right? So, models are of right? So, models are of right? So, models are of great great great importance. Isn't that right? importance. Isn't that right? importance. Isn't that right? Yes? Well, I...don't know. That's Yes? Well, I...don't know. That's Yes? Well, I...don't know. That's interesting, isn't it? I interesting, isn't it? I interesting, isn't it? I work...I work for a work...I work for a work...I work for a company called company called company called Jellyfish. We combine an Jellyfish. We combine an AI-based coding tool AI-based coding tool and and and agent telemetry with agent telemetry with engineering workflow data, engineering outcomes. We have the engineering outcomes. We have the largest largest largest dataset of its kind, dataset of its kind, dataset of its kind, hundreds of thousands of hundreds of thousands of hundreds of thousands of developers, about developers, about developers, about 1,300 companies. And we 1,300 companies. And we 1,300 companies. And we track everything track everything track everything they do with they do with they do with artificial intelligence in the artificial intelligence in the artificial intelligence in the development space. And development space. And development space. And one thing I one thing I one thing I noticed in our data noticed in our data is the data from around the is the data from around the is the data from around the time that SpaceX time that SpaceX time that SpaceX announced the announced the announced the acquisition of Cursor, and we acquisition of Cursor, and we acquisition of Cursor, and we saw a huge saw a huge saw a huge spike in spike in spike in Grok usage and the biggest

Grok usage and the biggest Grok usage and the biggest model change we've ever model change we've ever seen, seen, seen, much faster than much faster than much faster than regular engineers regular engineers regular engineers changing their changing their changing their preferences and preferences and preferences and trying to find the trying to find the trying to find the best model to best model to best model to work with. And when we work with. And when we work with. And when we dug into it, what dug into it, what dug into it, what happened was that Cursor happened was that Cursor happened was that Cursor was doing a lot of things to the was doing a lot of things to the UI, UI, pushing Grok to the top of the pushing Grok to the top of the pushing Grok to the top of the UI, changing how UI, changing how UI, changing how auto auto auto mode worked. And you know, it mode worked. And you know, it mode worked. And you know, it grew really quickly, and grew really quickly, and grew really quickly, and what was interesting about what was interesting about what was interesting about it for a it for a data scientist was what data scientist was what we call a we call a we call a kind of natural kind of natural kind of natural experiment. There was experiment. There was experiment. There was this external shock, and the this external shock, and the this external shock, and the natural question to natural question to ask was: what ask was: what happened? Has happened? Has happened? Has the result changed, for example? the result changed, for example? So what you see So what you see here in this chart here in this chart here in this chart is the change in is the change in is the change in merger PR metrics per merger PR metrics per merger PR metrics per engineer. And so, over engineer. And so, over engineer. And so, over this time period this time period this time period that I showed you, that I showed you, that I showed you, every dot here is an every dot here is an every dot here is an engineer. And the X-axis is engineer. And the X-axis is engineer. And the X-axis is how much their how much their how much their share of the model share of the model share of the model changed from changed from changed from not using Grok to not using Grok to not using Grok to using Grok on a using Grok on a using Grok on a scale from 0 to 100%. The scale from 0 to 100%. The scale from 0 to 100%. The Y-axis is the relative Y-axis is the relative Y-axis is the relative change in their change in their PR throughput relative to their PR throughput relative to their peers in the same peers in the same peers in the same company. And the story company. And the story company. And the story here is not a story. Here here is not a story. Here here is not a story. Here the cloud is centered the cloud is centered the cloud is centered around zero. There is no around zero. There is no around zero. There is no statistical statistical statistical correlation. The result did correlation. The result did correlation. The result did not change.

not change. not change. Switching everyone to Switching everyone to Switching everyone to Grok didn't really Grok didn't really Grok didn't really matter. So I matter. So I matter. So I left this left this left this experiment experiment experiment confused and wanted confused and wanted confused and wanted to know how to know how to know how important the model was. And important the model was. And important the model was. And if that doesn't matter if that doesn't matter if that doesn't matter much, then much, then much, then what other things what other things what other things really really really matter? So, that's what matter? So, that's what matter? So, that's what we'll devote the we'll devote the we'll devote the rest of our conversation to. rest of our conversation to. So, here's what we're So, here's what we're going to do. going to do. going to do. We will continue to We will continue to We will continue to focus on focus on focus on this metric of this metric of this metric of merger performance – merger performance – headcount headcount headcount per engineer per per engineer per per engineer per week. We week. We week. We will use a will use a will use a curated dataset here curated dataset here curated dataset here . This is a subset of . This is a subset of . This is a subset of our data where we our data where we our data where we have all the necessary have all the necessary have all the necessary elements to elements to elements to run this run this run this experiment. experiment. experiment. About 50,000 About 50,000 About 50,000 engineers in 447 engineers in 447 engineers in 447 companies. companies. companies. This year's data is for This year's data is for This year's data is for code users code users code users Cursor and Claude. And some Cursor and Claude. And some Cursor and Claude. And some important things. So, important things. So, important things. So, you have a tool. We're you have a tool. We're you have a tool. We're going to combine going to combine going to combine the system and the model the system and the model the system and the model because they are very because they are very because they are very interconnected. It's interconnected. It's interconnected. It's like your clubs, like your clubs, like your clubs, okay? It's okay? It's okay? It's how well you how well you how well you use them. By use them. By use them. By that I mean that I mean that I mean things like, you know, things like, you know, things like, you know, using skills using skills , cursor rules, , cursor rules, , cursor rules, system tuning system tuning , the tricks that we all , the tricks that we all , the tricks that we all learn as AI engineers learn as AI engineers to improve the way we to improve the way we use these use these use these tools. The tools. The tools. The golfing equivalent would be golfing equivalent would be golfing equivalent would be knowing knowing knowing how to get out of a how to get out of a how to get out of a bunker, which is not bunker, which is not bunker, which is not something that comes naturally. This is something that something that comes naturally. This is something that something that comes naturally. This is something that needs to needs to needs to be learned. It's be learned. It's be learned. It's how much you how much you how much you use it. So, use it. So, use it. So, apart from those things, apart from those things, apart from those things, just the amount of just the amount of just the amount of tokens spent.

tokens spent. tokens spent. So, golf time So, golf time So, golf time on the course, right? Uh, on the course, right? Uh, on the course, right? Uh, and then there are and then there are and then there are factors like where you factors like where you factors like where you work, your work, your work, your environment. So, environment. So, environment. So, your company and your your company and your your company and your team, it's the course itself, team, it's the course itself, team, it's the course itself, something you something you something you have no control over as a have no control over as a have no control over as a golfer. And golfer. And golfer. And then there's you, the individual, then there's you, the individual, then there's you, the individual, all the things you all the things you all the things you carry with you as an carry with you as an carry with you as an engineer that, engineer that, engineer that, holding all these holding all these holding all these other factors other factors other factors constant, constant, constant, still make you different still make you different . So, that's the map for . So, that's the map for . So, that's the map for the rest of our conversation. the rest of our conversation. the rest of our conversation. You put it all You put it all You put it all together, you do together, you do together, you do a bunch of big a bunch of big a bunch of big statistical statistical statistical analyses, the details of which analyses, the details of which analyses, the details of which I would love to I would love to I would love to go into here, but that would be go into here, but that would be go into here, but that would be boring for anyone but boring for anyone but boring for anyone but statisticians like statisticians like statisticians like me. But this is a map. You me. But this is a map. You me. But this is a map. You have approximately less than 1% have approximately less than 1% of how of how of how your your your PR bandwidth changes, PR bandwidth changes, PR bandwidth changes, the tool explained. the tool explained. How well you How well you use it, use it, use it, i.e. skills, is i.e. skills, is i.e. skills, is about 3%. Uh, about 3%. Uh, about 3%. Uh, how often you how often you how often you use it, use it, use it, it's about 10 it's about 10 it's about 10 times more than the times more than the times more than the tool, it's 9%. Where tool, it's 9%. Where tool, it's 9%. Where you work, it's 12%. The you work, it's 12%. The you work, it's 12%. The biggest part biggest part is you, the person. We is you, the person. We is you, the person. We will look at each of will look at each of will look at each of them and explain a little about them and explain a little about them and explain a little about why it looks like this why it looks like this why it looks like this . Hmm. No, there is no . Hmm. No, there is no . Hmm. No, there is no quality indicator here, but quality indicator here, but quality indicator here, but quality is something quality is something quality is something we pay a lot of we pay a lot of we pay a lot of attention to. I'll be attention to. I'll be attention to. I'll be happy happy happy to talk about that to talk about that to talk about that later, but it's a good later, but it's a good later, but it's a good question. So,

question. So, question. So, let's let's let's look at look at look at the tool first. So we'll the tool first. So we'll the tool first. So we'll go go go through this quickly. This is the least through this quickly. This is the least through this quickly. This is the least interesting, as you can interesting, as you can interesting, as you can guess, uh, from the guess, uh, from the guess, uh, from the title of my report. title of my report. title of my report. But what happens But what happens But what happens when engineers when engineers when engineers change tools change tools ? We've seen what ? We've seen what ? We've seen what happens when they're happens when they're happens when they're encouraged to, say, encouraged to, say, encouraged to, say, change change change tools, but tools, but tools, but engineers change the engineers change the engineers change the models they models they models they use use use all the time, they all the time, they all the time, they change the equipment. change the equipment. change the equipment. So you can do So you can do So you can do things like things like things like view changes in view changes in view changes in weekly weekly weekly merge reports. You can merge reports. You can merge reports. You can compare an engineer to compare an engineer to compare an engineer to himself and himself and himself and see what see what see what happens when happens when happens when he is fully he is fully he is fully invested in Opus invested in Opus invested in Opus versus Sonnet, versus Sonnet, versus Sonnet, switches to Chat switches to Chat switches to Chat GPT. What happens GPT. What happens GPT. What happens if he moves from if he moves from if he moves from Cursor to Cloud Code, and we have Cursor to Cloud Code, and we have Cursor to Cloud Code, and we have a lot of data about the a lot of data about the a lot of data about the people who made people who made people who made that transition? What that transition? What that transition? What happens when, uh happens when, uh , you go from , you go from , you go from working entirely in working entirely in autocomplete mode to autocomplete mode to agent mode? agent mode? agent mode? The story is about the The story is about the The story is about the same. Statistically, it's same. Statistically, it's same. Statistically, it's something like a no-brainer something like a no-brainer something like a no-brainer . There's not much to . There's not much to . There's not much to explain here to explain here to explain here to explain any explain any explain any meaningful change in meaningful change in meaningful change in the amount of the amount of the amount of PR bandwidth you PR bandwidth you PR bandwidth you get. What about get. What about get. What about how well how well how well you use it?

you use it? you use it? That's 3%, which is a much That's 3%, which is a much That's 3%, which is a much bigger piece of the bigger piece of the bigger piece of the pie. Let's pie. Let's pie. Let's talk about talk about talk about using skills using skills . This is interesting because . This is interesting because . This is interesting because the skills need to be the skills need to be the skills need to be greater. In fact, greater. In fact, greater. In fact, we find that we find that we find that skills skills skills matter a lot. One of the matter a lot. One of the matter a lot. One of the reasons it doesn't reasons it doesn't reasons it doesn't matter more is matter more is matter more is because, and I because, and I because, and I don't think this might don't think this might don't think this might surprise some surprise some surprise some people. People who people. People who people. People who use use use skills just like they would skills just like they would skills just like they would in the real world, in the real world, in the real world, outside of this outside of this outside of this wonderful land of wonderful land of AI engineering conferences AI engineering conferences that we that we that we inhabit. Uh, this inhabit. Uh, this inhabit. Uh, this chart shows, chart shows, chart shows, you know, weeks of you know, weeks of you know, weeks of skill usage by skill usage by skill usage by an engineer over an engineer over an engineer over time. So the proportion of time. So the proportion of time. So the proportion of engineering weeks that engineering weeks that engineering weeks that have, you know, have, you know, have, you know, low low low skill utilization skill utilization , zero , zero , zero skill utilization skill utilization , low level, , low level, , low level, medium level, and medium level, and medium level, and high high high skill utilization skill utilization . And as of September . And as of September , you know, almost , you know, almost , you know, almost half of the weeks half of the weeks half of the weeks shown here shown here shown here essentially didn't essentially didn't essentially didn't use any use any use any skills. Uh, and only 12% skills. Uh, and only 12% used used used the skills at a level the skills at a level the skills at a level that I think we would that I think we would that I think we would consider to be something that consider to be something that consider to be something that should be done from a should be done from a should be done from a modern modern modern AI engineering perspective AI engineering perspective AI engineering perspective . And the reason . And the reason . And the reason why this is so important is why this is so important is why this is so important is that that that the use of skills the use of skills , when widespread, has a , when widespread, has a , when widespread, has a much much much greater impact than the choice of greater impact than the choice of greater impact than the choice of model. So I'm model. So I'm model. So I'm showing here again those showing here again those showing here again those results that I results that I results that I mentioned earlier about mentioned earlier about mentioned earlier about changing the changing the changing the model family, as a kind of model family, as a kind of model family, as a kind of reference. But below reference. But below reference. But below here shows the here shows the here shows the skill usage level skill usage level skill usage level and its effect on and its effect on and its effect on changing the changing the PR throughput level, holding these other PR throughput level, holding these other factors constant. And factors constant. And factors constant. And you can see what you can see what you can see what you get, you know, you get, you know, you get, you know, when you get to when you get to when you get to 30+ 30+ 30+ skill usage runs

skill usage runs skill usage runs per engineer per per engineer per per engineer per week, that's a 30% week, that's a 30% growth just from growth just from growth just from that factor, that factor, that factor, holding other things holding other things holding other things constant. So, that's a constant. So, that's a constant. So, that's a big effect. Okay, big effect. Okay, big effect. Okay, let's talk let's talk let's talk about how much you about how much you about how much you use it. use it. use it. This is consumption. These are This is consumption. These are This is consumption. These are tokens. These are days of tokens. These are days of tokens. These are days of active active active use. Uh, use. Uh, use. Uh, I could write a I could write a I could write a whole report about this. whole report about this. whole report about this. In fact, I gave a In fact, I gave a In fact, I gave a whole talk about it whole talk about it whole talk about it at the World at the World AI Engineering Expo. AI Engineering Expo. So I encourage you, So I encourage you, So I encourage you, if you're interested in this, uh if you're interested in this, uh , to take a look at this. Uh , to take a look at this. Uh , but I'll give you a , but I'll give you a , but I'll give you a one-slide version. So, the one-slide version. So, the one-slide version. So, the data you data you data you see here is taken from this see here is taken from this see here is taken from this report. So that's report. So that's report. So that's about 500,000 weeks of about 500,000 weeks of about 500,000 weeks of developer behavior developer behavior . Each point is a . Each point is a . Each point is a two percentile two percentile two percentile interval of people interval of people interval of people using using using the same level of the same level of the same level of consumption during consumption during consumption during that week, and this is that week, and this is that week, and this is their average their average their average range of tokens and range of tokens and range of tokens and their combined PR per their combined PR per their combined PR per developer per developer per developer per week. And the basic week. And the basic week. And the basic story is that story is that story is that the more the more the more tokens, the greater the tokens, the greater the tokens, the greater the throughput. throughput. throughput. And there is diminishing returns. And there is diminishing returns. And there is diminishing returns. So, there is something like a So, there is something like a So, there is something like a saturation curve. I saturation curve. I saturation curve. I compared it to the compared it to the compared it to the rocket equation in rocket equation in rocket equation in that report. It's like that report. It's like that report. It's like rocket fuel. And what rocket fuel. And what rocket fuel. And what you get if you get if you get if you look at the top you look at the top you look at the top 10% versus the 10% versus the 10% versus the median, uh, you median, uh, you median, uh, you can actually can actually can actually double your output double your output double your output between the median and the between the median and the between the median and the top 10%. This costs top 10%. This costs top 10%. This costs you about 10 you about 10 you about 10 times more tokens.

times more tokens. times more tokens. But more tokens But more tokens But more tokens means more means more means more output. Now we output. Now we output. Now we move on to the move on to the move on to the bigger pieces of the bigger pieces of the bigger pieces of the pie. They are interesting. pie. They are interesting. pie. They are interesting. So, there's a factor of, you know So, there's a factor of, you know , where you work, your , where you work, your , where you work, your company, your company, your company, your team. In fact, team. In fact, team. In fact, if you break down if you break down if you break down the factors, about the factors, about the factors, about a third of them are the a third of them are the a third of them are the team, two-thirds team, two-thirds are the company. You know, are the company. You know, are the company. You know, your codebase, your your codebase, your your codebase, your review process, review process, review process, your culture. So your culture. So your culture. So once you capture once you capture once you capture the volume of consumption, the volume of consumption, the volume of consumption, you know, do these other you know, do these other you know, do these other things still things still things still matter? One matter? One matter? One way to look way to look way to look at it is, you know, if you at it is, you know, if you at it is, you know, if you correct for these factors correct for these factors correct for these factors and just look and just look and just look at the volatility, you at the volatility, you at the volatility, you look at the spread of look at the spread of look at the spread of companies relative to the companies relative to the companies relative to the average company, average company, average company, holding these other holding these other holding these other things fixed. Uh, things fixed. Uh, things fixed. Uh, this is what it looks like. And this is what it looks like. And this is what it looks like. And one way one way one way to interpret this is that to interpret this is that to interpret this is that I I I just showed you this just showed you this just showed you this token curve. So, token curve. So, token curve. So, you can spend you can spend you can spend 10 times as many 10 times as many 10 times as many tokens to move tokens to move tokens to move from the median to the from the median to the from the median to the top 10 top 10 top 10 percentiles and percentiles and percentiles and get twice as get twice as get twice as much output. much output. much output. There is also a 2x There is also a 2x There is also a 2x gap between the top 10% of companies gap between the top 10% of companies and the bottom 10% of companies companies companies in this in this in this distribution, which keeps distribution, which keeps distribution, which keeps your consumption your consumption your consumption fixed. In other fixed. In other fixed. In other words, your words, your words, your culture, your culture, your culture, your team, your team, your team, your company– company– company– there’s a lot of variation in these things, there’s a lot of variation in these things, there’s a lot of variation in these things, even in terms of even in terms of even in terms of fixed costs for fixed costs for fixed costs for AI. This matters a lot AI. This matters a lot AI. This matters a lot , and it's interesting , and it's interesting , and it's interesting for this group on the for this group on the for this group on the leadership path because it's something that leadership path because it's something that leaders can control.

leaders can control. Like, it's our Like, it's our Like, it's our job to control job to control job to control these things, even if these things, even if these things, even if maybe our maybe our maybe our CFO CFO CFO tells us we can't tells us we can't tells us we can't have more have more have more tokens than we have. tokens than we have. tokens than we have. Hmm, which I hope Hmm, which I hope Hmm, which I hope they don't have. So what they don't have. So what they don't have. So what about the biggest about the biggest about the biggest part? You are an part? You are an part? You are an individual, right? This is three individual, right? This is three individual, right? This is three quarters of this quarters of this quarters of this statistical model. statistical model. statistical model. So this is of great So this is of great So this is of great importance. Hmm, one importance. Hmm, one importance. Hmm, one way to think way to think way to think about it is this question about it is this question about it is this question that a lot of people have asked that a lot of people have asked that a lot of people have asked : does AI mean : does AI mean : does AI mean you need different you need different you need different engineers? Do we engineers? Do we engineers? Do we need different people, need different people, need different people, right? Is this a change in right? Is this a change in right? Is this a change in ratings? Does ratings? Does ratings? Does this change the meaning of the concept of " this change the meaning of the concept of " good engineer"? good engineer"? So, uh, you saw So, uh, you saw that distribution that I that distribution that I that distribution that I just showed you for just showed you for just showed you for companies. If you companies. If you companies. If you make the same make the same make the same distribution for distribution for distribution for engineers, it will be engineers, it will be engineers, it will be even wider. So there's even wider. So there's even wider. So there's even more volatility, even more volatility, even more volatility, and that's why it's such an and that's why it's such an and that's why it's such an important factor in this important factor in this important factor in this statistical map statistical map statistical map that we've put together for that we've put together for that we've put together for ourselves to explain the ourselves to explain the ourselves to explain the different levers available to us different levers available to us different levers available to us . Uh, it's even . Uh, it's even . Uh, it's even wider. This is, you know, a 3.5- wider. This is, you know, a 3.5- fold spread between fold spread between fold spread between the bottom 10% and the top the bottom 10% and the top the bottom 10% and the top 10%, uh, instead of a 2- 10%, uh, instead of a 2- fold spread like for fold spread like for fold spread like for companies. And there is a companies. And there is a companies. And there is a lot to lot to lot to sort out here. It would be a sort out here. It would be a sort out here. It would be a whole other conversation whole other conversation whole other conversation to sort out all the to sort out all the to sort out all the things that make things that make things that make different, uh, different, uh, different, uh, AI engineers, uh, better than AI engineers, uh, better than AI engineers, uh, better than others. One thing others. One thing others. One thing we can we can we can answer a little bit in the few answer a little bit in the few answer a little bit in the few minutes I have minutes I have minutes I have left is how left is how left is how AI is changing who the AI is changing who the AI is changing who the most productive most productive most productive engineers are. So, engineers are. So, engineers are. So, one way one way one way to look at it, here's a to look at it, here's a to look at it, here's a little study little study that ranks 10,000 that ranks 10,000 that ranks 10,000 engineers. We have engineers. We have engineers. We have long, uh, longitudinal

long, uh, longitudinal long, uh, longitudinal data, a long time data, a long time data, a long time scale. Jellyfish has been around for scale. Jellyfish has been around for scale. Jellyfish has been around for about 10 years. So about 10 years. So about 10 years. So we have a huge we have a huge we have a huge amount of history amount of history amount of history about how about how about how engineers behave. And if we engineers behave. And if we engineers behave. And if we look at how the look at how the look at how the top engineers top engineers top engineers rank, quarter rank, quarter rank, quarter by quarter, 6 by quarter, 6 by quarter, 6 months by 6 months, months by 6 months, months by 6 months, we can see we can see we can see how the top how the top how the top engineers are performing in terms of engineers are performing in terms of productivity. We are productivity. We are still talking about still talking about PR bandwidth. How much do they PR bandwidth. How much do they get out of the upper get out of the upper get out of the upper quartile. And if quartile. And if quartile. And if you compare that to, you compare that to, you compare that to, about 44% move out about 44% move out about 44% move out of the top quartile of the top quartile of the top quartile every six months. If every six months. If every six months. If you look at the period you look at the period you look at the period before and after AI, the before and after AI, the before and after AI, the situation is slightly situation is slightly situation is slightly less clear. People less clear. People less clear. People are becoming a little less are becoming a little less are becoming a little less mobile. This is 41%. So mobile. This is 41%. So mobile. This is 41%. So we conducted other we conducted other we conducted other similar studies. similar studies. similar studies. We do not see a major We do not see a major We do not see a major restructuring of the world restructuring of the world restructuring of the world order. Your order. Your order. Your best AI engineers best AI engineers best AI engineers generally generally generally remain the remain the remain the best best best engineers. So, in the engineers. So, in the engineers. So, in the narrative that narrative that narrative that AI AI AI levels the playing field, levels the playing field, levels the playing field, uh, skill still uh, skill still matters a lot, and that's partly why matters a lot, and that's partly why that factor is that factor is that factor is so big. So, so big. So, so big. So, you know, going back you know, going back you know, going back to this picture to this picture to this picture I was showing you. When I was showing you. When I was showing you. When you sum it all up, you sum it all up, you sum it all up, you know what I want you know what I want you know what I want you to take away is that you to take away is that you to take away is that these things on the left, these things on the left, these things on the left, you know, the tool, you know, the tool, you know, the tool, especially that one is especially that one is especially that one is really interesting to talk about.

really interesting to talk about. really interesting to talk about. This is cool new This is cool new This is cool new technology. It technology. It technology. It takes up almost all takes up almost all takes up almost all the air in the room, the air in the room, the air in the room, all the air on all the air on all the air on Twitter. And it's Twitter. And it's Twitter. And it's exciting, and I exciting, and I exciting, and I like it. Um, but like it. Um, but like it. Um, but we don't have much we don't have much we don't have much control over that. I don't control over that. I don't control over that. I don't work in the work in the work in the foundation's lab. I can't foundation's lab. I can't foundation's lab. I can't improve the improve the improve the tools. Uh, and with the tools. Uh, and with the tools. Uh, and with the cost reduction, cost reduction, cost reduction, we have, uh, less and we have, uh, less and we have, uh, less and less freedom to less freedom to less freedom to maybe buy the maybe buy the maybe buy the tools we tools we tools we want. But as leaders, want. But as leaders, want. But as leaders, the things we can the things we can the things we can control control control are on the other are on the other are on the other side of the, uh, equation. These are side of the, uh, equation. These are side of the, uh, equation. These are people, this is an organization. people, this is an organization. people, this is an organization. And I think, you know, And I think, you know, And I think, you know, going back to going back to going back to our our our golf analogy, what's really golf analogy, what's really golf analogy, what's really exciting is that exciting is that exciting is that unlike golf, unlike golf, unlike golf, you can change the you can change the you can change the course. You can course. You can course. You can change your change your change your environment. So you environment. So you environment. So you can, for example, can, for example, can, for example, fill in water fill in water fill in water hazards or do what hazards or do what hazards or do what you're yelled at you're yelled at you're yelled at if you're trying to if you're trying to if you're trying to do on a do on a do on a golf course. But, um, these are golf course. But, um, these are golf course. But, um, these are things that are available to things that are available to things that are available to us as leaders. So, us as leaders. So, us as leaders. So, in conclusion, um, in conclusion, um, in conclusion, um, you know, if you're not you know, if you're not you know, if you're not getting the getting the getting the results you want, I would results you want, I would results you want, I would encourage you encourage you encourage you to think, you know, to think, you know, to think, you know, maybe it's not the maybe it's not the maybe it's not the model, you know, model, you know, model, you know, maybe it's you. maybe it's you.

So. Thank you. I appreciate it. So. Thank you. I appreciate it. This is all I have. This is all I have. This is all I have. If you're interested, we If you're interested, we If you're interested, we publish monthly publish monthly publish monthly benchmarks benchmarks benchmarks with all this data with all this data with all this data and more. Go and more. Go and more. Go to the section " to the section " Artificial Intelligence Engineering Trends Artificial Intelligence Engineering Trends for Jellyfish". Uh, for Jellyfish". Uh, for Jellyfish". Uh, a lot of our a lot of our a lot of our research research research is published in is published in is published in our Substack. Uh, our Substack. Uh, our Substack. Uh, thank you.