0:03Thanks for coming to our panel, we've got a short amount of time here, so we're going to jump in and go rapid fire with the questions, but excited to be joined by three esteemed panelists and we're about to jump into the latest of everything happening right now in the CodeGen era, and how that's impacting DevOps. So I'll just quickly start off with a quick data point here, a recent survey by JetBrains showed that almost 50% of code is now being written by agents, and you have Frontier labs like Anthropic now disclosing, you know, 90% of cloud code was written by the agents. So we'd love to just run down the line here, just have each panelist just quickly introduce themselves and then tell us, do these stats line up with your internal usage, as well as what you can see at your end customers in terms of how your end customers are using code agents? We'll start with you, Dan. Yeah, I'm Dan Connect, Cloudflare CTO.

0:50You know, I think at this point, almost, I would say by 98 % of every MRPR cloud. software has some AI involvement whether that is planning or the generating the code or the review at this point it's pretty much you know ubiquitous across the team and

1:12so yeah Madison yeah so hey I'm Madison had a forward deployment at factory which is building autonomous software factories for enterprises for the most part it's an important station because we are also building coding agents for organizations. So we also run on a software factory. And so I would say we tend to try to build and dog food that system as much as possible ahead of driving that out to organizations. I was actually able to look into our data and we started off the year around, I think, 30 to 40 percent of agents driving most the code base and the changes. And now I think we're around. 60 to 70 percent. I think that percent being as high as possible is not always the target, right?

1:59I think the goal is really to have good verification systems and really trust the agents that are making changes on your behalf. And so I think we're doing that in a really healthy way and we're trying to teach organizations how to do that as well. And Chris, how about at Sentry? Yeah, so my name is Chris Vilder. I'm the CRO at Sentry, so the go-to-market organization. So I've never deployed code to production. So by default, I'm the safest engineer in this room.

2:27So at Century, we probably are in the same 50, 60% range, right? Like we have a whole bunch of people using it. Our founder is fanatic. So he's probably at like 98% utilization. And at least what we see, so we are essentially an observability tool for developers. What we see is that, you know, the bottleneck has... It really shifts it, not just from writing code, but to the reviews. And then in production, you know, what happens and how do you monitor the sheer amount of code that's in generator, right? That's kind of what was kind of visible now. So I got so excited about the questions, I didn't get to introduce myself. So I'm Aaron Jacobson, part of NEA, a multi -huge venture capital firm, which happens to have been an early investor in all three of these companies. That may give you a sense of how this panel came together.

3:13So let's now double-click into how this. eyes of co -gen agents is actually really impacting workflows. You obviously have an increasing volume of code being written by agents. You know, Madison, you are an AI-native company. You get to build things from scratch. I mean, how are you seeing this impact to other workflows? How is it impacting CICD, review, you know, the whole development chain when you start to have this kind of volume of code being written by agents? What's breaking? Yeah, so a fun story that Aaron is very aware of is I was actually the lead investor in factory about a year and a half. a half ago and underpinning that investment was a belief that the biggest bottlenecks for agent -native software development would not be the IDE and autocomplete, but more about the organizational context and ultimately the software factory. And again, that's what factory is now building.

4:02And so after seeing that and recognizing that a lot of code would be driven by agents in the future, I recognize that one of the next pieces that would really break was CICD, because we have a system in traditional DevOps that was really not built for agentic software. It is extremely difficult to use at rapid pace, and it does not handle bursts and spikes very well. And it's very difficult to take out lots of different versions of the code base and be able to merge them back together. So we're seeing GitHub obviously break in terms of reliability, but also just the process through which it works for agentic. and that's why we're starting to see a lot of these CICD acceleration companies like Namespace and Blacksmith start to take off because they're really making it so much easier for agents to use existing tools in CICD runners.

4:52Another space that we're seeing a lot of is code review and bringing autonomy to code review, making that a really agent native process. That's something that we have a lot of internal tooling around, but ultimately identifying find the opportunities to drive more agent-native code review, but then also find ways to surface where should this actually become autonomous and how and under what circumstances is the hardest part. So, Dane, how does that line up with what you're seeing in Cloudflare given the amount of code that agents are touching? Do you agree with CICD in review or what broke first for you guys? I mean, absolutely, the review side. I see some conference engineers in here, I'm sure. would agree uh you know obviously the writing the code is the cheap part now it's uh the planning planning and uh and the review part you know the bookends of it uh that really still need a human input uh and so you know we put a lot of effort into building a a review agent that helps at least speed up that process that encodes uh uh how we build how we like to build a cloudflare kind of takes the dna of cloudflare and uh taught an agent how to understand that, taught an agent to look for what our past incidents look like, those spot patterns.

6:10It's very good at putting all that context together and can even find things that humans can't find, but we're still not ready to let go of the final human step of merging it either. So, you know, right now, an ordinal amount of time is spent reviewing a code. I'm going to come back to that. How long do you mean? are going to be in the loop, but let's save that for later. So I'm glad you brought up a review as a scenario I'd really have to double-click into further. Chris, you know, Sentry, you're seeing all this code actually be shipped in production. You guys, I believe, also have a review agent that you built yourselves to. So how are you seeing this code behave? Is quality going up, or is AI sloppification also happening in the code world too?

6:55I mean, I think the evidence is kind of mixed. One of the things we see is we do see more, if you're a person, line of code written there are more issues but it's more in the order of 10 to 15 maybe 20 percent that we've seen i think the major issue is really that there is just way more code being written so there's just way more code that needs to be reviewed in the first place so there is a clear bottleneck there and then there is way more code that ends up in production and there's still a lot of issues in production they they actually just tend to uh accelerate with just code development right um at least what i see from my perspective from the customer customers I talk to, the people that have decent practices in place that just get better. Those that are less evolved have and end up getting worse over time as well. So the ones that do it well, they tend to identify kind of categories of PRs and they say, oh, there's low risk PRs, there is middle and there is high.

7:46The ones that are high, they go to the human. The ones that are low, they get automatically reviewed. They get pushed to production and it's a large category. So you can kind of accelerate it that way. And then once it gets to production, obviously. uh i mean talking my own book to some extent but like you have to monitor what's going on if you don't get signal about what's going on you're not going to know what's going on and so uh i think one of the benefits of coding agents is that they've made instrumentation a lot easier so yeah even if you go down a path of open telemetry like you need to migrate to that like putting that instrumentation in place is now much more cost effective so you can actually get the signal that you need to figure out what's going on with your coding production and so So we see that shifting in that direction. Chris, a quick follow-up. Are you seeing the nature of the errors change or failure mode changing in terms of what's happening with the agents now writing the code?

8:34I think a little bit. The obvious errors are less prevalent, right? Like syntax errors are non-existent anymore because these things have learned to write code. And so the syntax is good. So there is a little bit of an element of fewer simple things and maybe a handful of more complex. things that are tending to surface, but I don't have great statistics on that. Dane, what about your experience being a practitioner and what you're seeing in the engineering team at Cloudflare? Does this match up in terms of, you know, quality not necessarily getting better, seeing any new errors, any new things that were unexpected as code agents are now taking over the world? Well, I mean, so I think we're still in the, you know, just because of the type of code we write, still very much a human-centric engineering org. Humans are responsible for the planning and iterating with the agent on that plan.

9:26And so, you know, I think engineers that were great engineers before are still going to be the great engineers now. You know, if they're pushing slop, it's probably because they want to get a quick script out versus something that they knew was high value and high risk. And, you know, in most cases, I would hope that, you know, it's code that... putting their name on um you know we kind of said early on uh you can't blame ai for uh issues uh because we still haven't i mean maybe one day when we take the uh we take a human out of the loop but right now humans in the loop so you know uh uh own the mistake there uh but also own the quality and uh for most part i think quality actually has gone up in a lot of places uh velocity um you know uh uh you know kenton varda who uh It's our distinguished engineer who works on the worker's platform.

10:23He's going from writing 3 ,000 to 5 ,000 lines a day to 20 ,000 to 30 ,000 lines a day.

10:33And he believes it's the same quality that he had before. Sometimes he has to go some circles to bang it into shape. But, you know, even that... is getting solved over time. I mean, given that amazing quantity of code that's going up, what's life like right now for an on-call engineer? I know that's particularly important for Cloudflare, give me your uptimes. I mean, are on-call engineers just getting crushed right now? Or are they themselves also now starting to use agents to make their lives easier? Well, the head of our SREs is here, so another question we can bring in. Yeah, I think we're probably early in our journey here. We have a lot of tools. around uh incident management yeah uh but uh you know uh and part of the problem is i mean we are a security company at hark and we segregate and compartmentalize data quite a bit uh um and uh you know it's not easy for an engineer to get access to the production data sometimes that's needed uh which actually is a great opportunity for agents because agents can actually investigate something for you as an engineer without actually even showing you the data uh showing you the the so i think there actually is opportunity to actually use it actually to make that part of the job better for Cloudflare and Cloudflare is very unique in that perspective.

11:45But, you know, on more of the pre -production side, I think things like, you know, Sentry and, you know, observability, agents love a loop where they can see a reward and optimize. And so, you know, I think that is where kind of data and pulling in a lot of information can be really helpful. Yeah. Chris, what are you seeing in your customers can you're you know you're selling the tools to I think by definition if you're generating three times as much code then you assume that the quality is the same you're going to get three times more incidents or five times more incidents all right so we do see that a lot of our customers are very concerned about it because you can only wake up the senior people so many times they can only drink so much coffee on any given night before they decide to take another job somewhere else but I think that's that's really the main concern I don't think the solutions are like put some agent on top and trying to figure out like what you can get out of the telemetry to get to a root cause faster but there's still a big concern there in terms of like relying on these people and on top of that obviously people do not know if they haven't written the code themselves a lot of that kind of tribal knowledge becomes less efficient as well in the process i think i mean there's some truth to the more code is you know potentially more incidents uh uh The code that actually I think we're fighting the most incidents is actually code that was written, you know, sometimes five, 10 years ago.

13:13The agents can now find those vulnerabilities, you know. And so if they can find those vulnerabilities now, I mean, the hope is that they're writing code and their review process now is not leaving those same vulnerabilities in the code. So, you know, I think the actual the incidents are going up more from the technical debt and legacy code of the past. And, you know, maybe it's wishful thinking that the code we're writing now is somewhat more incident -free. Yeah, something I would add is we're doing, we're seeing this probably come up as the single biggest use cases for our enterprises who are deploying autonomous software factories. They really want to establish, we have a product called Agent Readiness, where you're able to actually measure the readiness of your code base.

14:02opportunities to proactively get on top of security issues uh documentation issues making sure that you basically have this like knowledge base of your code that works for agents and i actually think some of the security issues we're seeing now with um legacy code languages are going to be um this is the beginning of what a tornado of issues like this and we're going to see this effort around proactive security remediation and bung triaging you one of the biggest investments and focuses of the next couple of months into 2027. So Madison, we talked a lot about a lot of the challenges that the world of code agents is introducing to folks. Obviously, the quantity is going way up. The quality, not clear, it's actually improving.

14:47Maybe discovering a lot more incidents historically from legacy code. Who's responsible for solving these challenges? I mean, do the agents need to get better producing code? Or do you think it's the review and the air hand? handling process that needs to change to keep up with the agents. Yeah. So I would say engineers for the first time ever are finally getting to be proactive in their development and start to think less about reactive work. I think that's one of the biggest things that we're seeing now. So what needs to happen is we need to have engineers up level as much as possible to be thinking about review quality and managing their systems, managing their code bases. Thinking about this as an artifact in the company that is fairly liquid in nature. How do we modernize and change the right languages and applications that are required to really effectively work?

15:37So I think the responsibility really is always on the engineer. You are the one setting in place the verifications, the tests, the QA, the evals. But using agents as much as possible to automate that work and bring you into this proactive mindset is what's going to really unlock autonomous, the next era. on a software development. Chris, you're on the other side. You're downstream from factory. What's your view in terms of who's ultimately responsible for solving some of these new challenges? I think it's all of the above, right? Like, you want to write better code, you want to review the code more effectively, you want to have better telemetry in place, you want to feed it to an agent who can solve as much as possible. I do think that, like, getting more guardrails in place in terms of how you deploy software and getting more kind of, you know, effective at, like, splitting up.

16:22populations you want to launch your software to deciding whether you want to roll back very rapidly based on some very specific criteria and having that practice which you know has existed for like 10 years at least if you're 15 like we have some of our customers have done this with other agents right but you know getting into these practices for every deploy and deciding what he putting everything behind a feature flag rolling down to a very small set of people seeing how it reacts and then kind of gradually expanding and rolling back is probably and these guardrails is where I see the best upside. Yeah, I mean, we've talked about this whole cycle, right, of agents writing code, reviewing code, testing it and deploying it, and ultimately I think the holy grail is getting that back, looping that back, making it continuously, maybe even pulling the human out of the loop.

17:08I mean, where do you guys see the opportunity between, you know, factory, sentry, you know, CICD, other things in the middle, to actually close the loop and have everything working more simultaneously? together without violating some of the data and security issues that dame was talking about i mean it's at the end of the day you gotta measure something right and then you have a goal seek algorithm that goes and finds the best way to get there and as long as you put the guardrails in such a way that the algorithm cannot deviate too much from uh from the road right like if you're not saying go eliminate my test so that i can pass the test like that kind of stuff uh that's really go, you see this with some of our customers, like you find a problem in production, say the latency of the, I click on this button, the latency when it exceeds a certain, you know, 0 .5 seconds or whatever, then go investigate, go see what's going on, find all the evidence you can find from Sentry, from other telemetry solutions, bring it together, do some kind of a root cause analysis, feed that to the agent and let the agent kind of address the So it's like a close loop measurement system that the feedback goes in.

18:14but you have to make sure that, again, the agent stays on the rails. Madison, does that resonate with you? What does the autonomous software factory look like in the future? So we like to say that your software factory is always ingesting signals from every system it's connected to, all the APIs, all the MSCPs, all the existing. I mean, Sentry is something that we partner with, and we see a lot of organizations think about alerts from Sentry. as being automatic triggers for their software factory. So defining those signals, I think ultimately being as data-driven as possible is really the key to creating autonomy in a closed loop. And so we are very much on the same page and think a lot about signal in software factory, signal out is new software.

19:05So we're going to have to close things out here with our last minutes, but I want to come back to Dane as well as the other panelists. on this question of how long until humans are completely out of the loop. I mean, it seems like the ground is shifting constantly. The models are getting better and better. Where do you see agent-of-code development DevOps going from here? And how long until we actually, everyone just has to hang out while their agents write and deploy and fix code without any human intervention? I mean, I still believe a lot in human judgment and human ingenuity and creativity. So, I mean, I think humans will be... part of the process uh for uh quite some time uh the you know where they sit it where they sit in it it might shift uh but i i don't you know in the next couple years i don't see uh us removing humans uh from uh uh the software devolver process madison yeah we obviously are the ones pushing for what we call the autonomy maturity model at these organizations but it is it's quite wide to see um you know the dichotomy between okay you have uh the best ai models in the world able to do crazy tasks but then you have organizations that are stuck on extremely legacy software and bridging the gap there is still very very they're so far to go um so having these systems that really bring humans into the process and build autonomy slowly with verification is um going to be a it's to be a very long process no matter how fast ai evolves chris you want

20:40to finish this off what does the future hold same same thing i think judgment and the much overused taste are still going to be important um and you can teach agents to to some extent but you know if you think about it in the ultimate limit this becomes kind of a self -organizing organic system software right and if you just let it do whatever the user you know directs it to do god knows where it will end up so you still need some of again, you know, guardrails in this case, judgment by humans, like, hey, this is what I want my software to do. And this is where it is to ultimately end up, right? So for a while still. Amazing. Thank you, Dane, Madison, and Chris for this awesome panel. Please give our panelists a hand in the applause.