0:03my name is jonas and i lead the audio video team at discord and i want to start by telling you a small story i will make sense in a bit it is february 2025 it's 9 30 at night i'm in my basement sitting on my trainer bike thought about bringing a picture but i don't think you want to see that so i cut that out but i'm in this virtual training thing called swift i'm pushing up this hill you know music in my ears really in the zone and And the music fades down. I get the Discord notification sound. And it's super annoying because I'm one of those people who do not have the notifications for virtually anything. I have notifications for my wife, my very close friends, my CTO and my VP. And my wife is asleep and my friends are asleep because they're in Sweden. So it must be either of those two's last one. So then sort of curiosity gets the best of me. And I was like, oh, do I really like stop my workout to figure out what's going on?

0:51So I get my phone out of my pocket and it's a five minute long voice message from my VP. at 9.30. And that wouldn't be so weird to get a five minute long voice message. But I happen to know that my VP is with Cloudflare in the East Coast. So it's a bit later for him. And I also happen to know that there may have been a couple of cold ones involved in the situation. So you kind of know that something is coming out of this one. So I'm like, okay, let's listen to what he has to say. So, you know, play the voice memo and just, you know, a lot of back and forth and eventually gets to the point. And the point he's sort of come to is like, what if we took the voice engine, and we moved it to run on the Cloudflare edge instead of what we're currently running on, because we're currently running this on a bare metal cloud combo. And at the time when he said this, like, Discord is a heavy user of Cloudflare. We use their Ingress products, we use workers, we use their CDN, we use tons of Cloudflare already, but hadn't really been on or even radar to run this workload on the Cloudflare edge, if that makes sense.

1:41So I was like, okay, you're crazy. And I was like, we are, you know, the Cloudflare is mostly TCP, our workload is UDP. There's tons of reasons. why this is not going to work and you know you initially start with this like this is not going to work if that makes sense and today i want to tell you a story like how we ended up making this possible and why that voice message wasn't that crazy at the scene right so if you don't know what discord is if you have not used discord uh hopefully you have but if you haven't discord is where people come to sort of talk before during and after they play games we're pretty big at this point we have over 200 million monthly active users and the kind of things people do if you have to use Discord. So you kind of find a voice channel with your friends and you go into voice channels and you sit with them for a long amount of time when you play games or just hang out.

2:28You can think of it as like a way to talk to your friends over long periods of times. So instead of like Google Meets or something like that, where it's a meeting that's 30 minutes long, these are very long sessions. They can go five hours. Some sessions are days. We've had sessions go for months. So there are these voice sessions that can be very, very long. So we have to build our infrastructure to handle like very long sessions. if that makes sense. When I say the lead audio video team, I'm not really talking about videos of webcams because the sort of secondary thing people do when they use Discord is use our feature called GoLive. And GoLive is this feature we built many years ago that allows you to just click a button and we hardware encode. We do like a lot of accelerated stuff to sort of shoot that video feed from your game to someone across the internet, your friends often. And we're doing it at super low latency because the sort of experience we want to give to people is...

3:16being in the same room so you can think of what we're trying to optimize for is like you want to feel like you're sitting next to someone when they're playing the game if that makes sense and that's the bread of butter of our application right we do voice and video and people use this in all kind of weird ways that we hadn't expected like when we built this feature we thought like people are going to use it for streaming games that people stream homework they stream web browsers on mobile like a very big use case we have is people actually stream themselves watching instagram so people can follow along in your feed so this use cases I don't think we had imagined when we built this, but that's the kind of scale of what we're supporting with this today. And I think that's been true since we built Discord. Like you always start by building a product and you have assumptions of how are people going to use your product. And then when you sort of shoot it into real life, you quickly learn that like most of assumptions you have are wrong. But the good part about our voice and video is that we've done this since we started the product. So we've done this for 10 years at this point.

4:04So we've learned a lot about what works and how are users using the product, if that makes sense. So going into something like this, we could really look at the data to figure out what are we actually solving for, which is a much better position to be than like, hey, we're solving for unknown, if that makes sense. So how do we do it, or how does this code work? And like, how do we do voice and video under the hood? As I said before, we use cloud for a lot for the application, but for the voice and video, we are today a WebRTC shop. And if you don't know what WebRTC is, WebRTC is a protocol that allows you to sort of communicate between web browsers and applications. applications it was initially sort of funded and developed by google and ericsson and they took a lot of telephony standards and just shoved it into a protocol and then released to the world so it's a bit archaic but it's very stable and it's virtually support everywhere so the cool thing it gives us is we can support browsers we can support apps you know it kind of just works out of the box and the most common thing i hear when i talk about a product is people are like oh of course that allows you to do peer-to-peer so you know this will mean that like every client data to each other because WebRTC allows this.

5:08If you set up a browser simply, it's like most people do. And people assume that we do it, but we don't do peer -to -peer. What we do is we use a relay. And we use relay because there's a couple of reasons we figured out very early on as we were building Discord. The first one is stability. Like if you do peer-to-peer, you're often enough in these situations where like you have asynchronous peering. So you might be able to send packets to your friend, but they cannot send packets back. So when you try to talk to them like they will hear you but you cannot hear them and you kind of get into all these weird issues so you have to usually the server to like control the network if that makes sense but the second one for us which has been a selling point of what we're doing so we want to give users the privacy now you're thinking about like what do you mean privacy like a big reason actually that this could got popular is that the back in the days 10 years ago the alternatives that were peer-to-peer allowed people to discover your ip and you think this is an edge case when i tell you what would happen in games is that, you know, you will play against someone and you will beat them in the game and they would rent a small bot farm to DDoS you. So you would lose in the game, right?

6:09This would happen in reality. And especially with streamers that, you know, had sort of a larger audience, this would be a very common thing. So they just stopped using tools like Skype and other tools. And they sort of came to Discord because what we offer is this guarantee that when you use Discord, you sort of, we keep your IP and like you behind a wall, if that makes sense.

6:33So, the problem if you build a relay and you're sort of naive about it is that if I stream video to the relay and the relay fans it out to everyone, that's 12 streams of video, and that might not make sense if not everyone is watching it. But you can think of, like, you have four people streaming video, and then one person is watching it. Then you're fanning this video out four times, 12 times. That's a lot of bandwidth you're wasting. And especially when you're trying to give away video for free to everyone on the internet, that quickly breaks the bag. So what I think a lot of people, and we do, to build a relay. what's called an SFU. It's a smart relay, stands for Selective Forwarding Unit. And it's this like sort of server that figures out like, hey, who is watching? Who needs the videos? If you're on a call with someone and you send video up, then the SFU knows exactly who needs that packet and decides in real time where this gets found out. And it's a much better use of bandwidth, but it comes at a cost in terms of processing because WebRTC is a very stateful protocol.

7:21So in order to know exactly what packets you need, you need to keep track of every packet you've sent. and like where they are in the stream and SSRCs. So you're kind of trading the decrease in bandwidth for more compute that you have to spend, if that makes sense. You can't have the niceties of an HTTP CDN. And you might ask at this point, like why not use a CDN? Like there's a lot of companies that are trying to solve these problems. There's tons of commodity CDNs, there's technology like HLS and Dash. Like we've sold this problem as an industry. I used to work at Twitch. Like they've sold that problem. YouTube have sold it, Matt have sold it. it. So why would we not use a commodity CDN to solve this problem, given they're optimized for video? And I think it's sort of important to go through this because it kind of makes sense later how we ended up going to the edge in the first place.

8:07The first one is that CDNs are often just optimized for like a one-to-many situation. So like when you build a CDN, what you want is a big fan out. And the fan out has to be economically sensible in that way you can give it to as many users possible. Our users is few to few, so that the value you get from a CDN is less than fanout is not really worth it to us and we actually end up just paying more for that fanout than we would gain. And the second one, which I talked about earlier, is latency. What we're trying to give users is like a real-time experience. Even if you use something like LLHLS, then it's really hard to get under two seconds, like getting under one second or sub -second. It can be done in a test lab, but in reality, it's very hard to do with HP CDNs. So if you think about it, if you're sort of watching a friend and they scream like oh my god and then it takes two seconds for you to see that video that breaks the immersion we're trying to give users right so we need it to be low latency to kind of give that immersion and i think you know there didn't used to be good cdns that did web rtc that has changed 10 years later but like in 2015 if you really went and tried to find someone who was doing web rtc cdns that didn't really exist most people pointed you to the lib web rtc project if you have ever worked with that and tried to build a server you know it's a scam but like it's one of these things where to the industry has advanced but even as the industry has advanced it's really hard to make the control plane scalable across these different cdms so you end up having to build your own control plane anyway and like when you

9:37build your own control plane then like the value of the data plane is not really that good anymore so the problem we're about to see that i think we're facing and the industry is facing is it's really hard to standardize this right and we're solving this industry like media over quick step side note like actually developing to fix this problem so we can commoditize low latency video and audio on the internet but it's not really there yet and like for our case since we've been doing this for 10 years we're still on webrc so i mentioned we run on bare metal and our goal even before we started this project with cloudflare has been to get off bare metal and there's a couple of reasons why you get off bare metal you can probably imagine a lot of them but i also will say that this was before the sort of ai thing happen like before stockouts happen daily. So in 2023, this seemed like a very sensible decision that might not be as sensible today as it was back then.

10:24But it was sensible then to say like, hey, the public clouds have an excessive compute that you can use. And we really wanted to make use of that. Not specifically because we believe that there will be value in getting better CPUs and RAM, but because we believe that if we could get access to better network accelerators and better network, we would be able to do a better job of serving our users. So we saw that like both, you know, AWS and Google had perfected how to sort of do accelerated networking from the Linux kernel all the way down through their host. And that was not as easy to get on a bare metal host, especially what we were running on before. So we wanted to be able to make use of this, and it seemed like public clouds would be where we could do that, if that makes sense.

11:03So when this moist message showed up, right, we were already in this mind of like, we want to move to something different. We are prepared to re -architect our sort of like crafty bare metal thing to fit on these public clouds. But we hadn't really imagined Cloudflare being that choice, if that makes sense. And then this voice message shows up. And I use Cloudflare a lot personally. I built a lot of personal projects. And again, we use it a lot as a company, but we didn't see it as like a CDN. I think if you think about the CDNs, by the way, and you ask them like, hey, can I have like low latency? They will happily sort of give you low latency, but they won't give you the low latency with Robotic C. So the problem is that we came back to Cloudflare with after we started talking to them. It's like, first of all, you know, if you think about what they run, like a worker runs for milliseconds, but I said before, a call can run for an hour, Can run for many hours, can run for days.

11:54So like, how do we bridge this gap between what we are trying to do and what they are offering, right? You have durable objects, but at the time, durable objects didn't support UDP, so we couldn't run the SFU. And specifically with the sort of data volumes that we're talking about, it wasn't really cost competitive for us either to run on something like durable objects. So we really had to answer the question for ourselves. Why would we pick going with Cloudflare or an established vendor when we knew that we could get what we want? And there was something as we started looking at the data because I would admit that I was sort of initially pretty dismissive of this idea, but I gave it a fair shot and we looked at the data and when we started looking at the data, the thing that stuck out to us that made us be like, okay, this might actually be a better choice than we thought is that 75% of all Discord calls are local. They're within the same geografia. region and sure if you meet friends on the internet you play world of warcraft or whatever you create a guild you might end up talking to people that are not local for most users they meet their friends in real life they meet like people from school people from clubs etc and they end up building connections where people live close to them so with that in mind if we look at data it's sort of like obvious to us that we could if we could do the work to be more hyper local we would serve this use case much better and that's what's kind of like made us into didn't go in McLaughlin.

13:06Because the thing about the problem, right, if I am here and I sort of send a packet and it goes across the internet for the Wi-Fi, the ISP, peers, and then eventually we show SFU, and then it has to go all the way back only to come out four houses down, that's a very long round trip for a packet just to like sort of come down the street, if that makes sense. So we can shrink this distance, we can do a better job. But now you might be wondering, like, if we sort of shrink this zone, what do you get from this? Why would this? matter to the user. Because hyperscale and networks, they're pretty good. If you've used AWS and if you use GCP, you know that the networks are super good. But the thing for us that we know with our traffic, because we're not TCP, we're UDP, is that more distance means higher packet loss. So for us, if you're not familiar with it, I'll sort of go over. The thing with UDP is you don't have the sort of error correction and the negotiation back and forth.

13:55So you just kind of send the packet over the internet and hope for the rest, right? And that's... how you get the low latency, but for us, because the more steps that is in between you and SFU, the more chances there are for you to drop that packet on the floor. And we fix this problem we're about to see by doing stuff like buffering, we do expand, we sort of paint over the audio, but we can't really fix the actual issue, which is that like higher loss means that we have to at least like higher latency and it's bad for users. And you might be like, well, it's fine if it's small glitches here and there, but what we've found working with our users is that it's fine if you're on a call for 30 minutes on a meeting and you speak for like five minutes because it's sort of easy to paint over that with your head. And it's fine if you're talking to your parents because you kind of know they're going to tell you to drink more water. Like it doesn't really matter.

14:41But if you're on a call for five, six hours, these small imperfections starts mattering because they become friction. And that friction over time, especially looking at user sentiments, become a problem for users. So we can literally see a correlation between user sentiment and packet loss. packet loss go up by 1% and your sentiment moves in another direction. So for us, improving the loss is the most important thing for us. So what we really wanted to get is if we can have fewer hops, we can have less loss, and that leads to way better calls. So the kind of idea was that hey, maybe it makes sense to shrink the zones because it gives us something we cannot get from the big hyperscales. As I mentioned, they will happily tell you if you want to get close to the user, you can use their CDM product, but their CDM product cannot. run the SFU and we really believed that Cloudflare was the same and we talked to them and like we kind of went back and forth and they basically said like well you know we don't do what you want but we kind of make it happen because we also want this to be the future if that makes sense so yes we worked back and forth and it took a while you know we tried to initially figure out like can we use their existing concepts is there things that you do that we could use and re-architect our system we thought about durable objects but one thing that was very important to me throughout this entire experience we didn't want them to create something that was a bespoke this if you've ever been on the other side of this you're a customer and like a vendor or if you've been on the side of building for something for a for a customer and you sort of build something that is bespoke you know that the second that customer is is happy you'll start thinking stop thinking about it and like no one touches that for four years and that kind of falls apart so we didn't want to be on like a side par

16:15car of like the sort of network we wanted to use something that cloud for invested in and believed in and we sort of told this to them and they're like you know we've been thinking of something that might fit the bill. It might not have what you need right now, but it's like sort of in the direction of where you want to go. And that's what today is known as cloud for containers. Back then, I think it was called cloud chamber, but today's known as cloud for containers. And there was a lot of things they hadn't really imagined. I don't think they had imagined someone running this with UDP traffic. There was tons of other things. I think they thought about these as like sort of short-lived, not living for hours on end. So there was some sort of like friction we had to fit. out but you know to their credit we figured out with them and we started deploying our our traffic on these sort of on cloud for containers and the cool part about this is like it literally took us three weeks from the voice memo to we starting to get actual traffic on production right so the voice memo and then we had sort of like back and forth like will this work out and february 18 we sat down with the team and said like let's do this together they gave us access to the alpha and we started working with them and we got it done and on february 26 what we trying to figure out with them was not specifically like, will it work?

17:18We're trying to validate our thesis. Because at the time we were like, are we crazy to do this? You know, is this going to work? It's one thing to sit on your bike in the basement and sort of ideate these like wild ideas. And then there's another thing to take them into production. So we wanted to figure out like, is there a way we can test this hypothesis in the real world without impacting users? So we wanted to find a region we could do this on. We need a region that had its own language. So if has its own language, we know that traffic are more likely to be in a certain zone and kind of spread out. You know, if you speak English and you're next to another English-speaking country, there's a high certainty that that traffic moves further. But if you have an isolated language, it's a high certainty people don't know the language and just naturally don't speak outside of that. Makes sense. We want it to be a geographic isolation. So, you know, just naturally from a network point of view, we could get captured data in a smaller sense.

18:06And lastly, we wanted to have lots of Discord users because if we didn't have Discord users, we wouldn't get the data so it had to be a place that had high sort of penetration of our product and a lot of people in games and there's one place in the world that fits this bill that is Iceland happens to be an island they have their own language and they play a lot of games so it's perfect and the good thing is you might be like oh they used them as a sacrificial goat we did not because by the time Iceland was transiting all the traffic from Iceland all the way to Rotterdam so they were they were already not having good experience so what we thought is like hey we can actually test to do this and hopefully would make their lives better so it's kind of win-win like we get the data and they get a better experience so we work with them and on February 26th we turn on Reykjavik and as we hope we saw a 9 % decrease in ping and 11 % decrease in packet loss which was just huge by the way like that might not seem like much but we would celebrate if we saw a 1 % decrease of packet loss so 11 % was just like oh but the problem that quickly became apparent to us is that we saw a 2% ping increase for everyone else outside of Iceland that were talking to people on the calls.

19:15And it was a bit like, you know, in hindsight, it makes a lot of sense, but it was a bit of like a head turner for us in the beginning. Because what we used to do is that the first person who would join a call would place the So if we have one person from Iceland join a voice channel, then the survey gets placed in Reykjavik. And then if you have three people from Germany, then all the other people have to go back to Germany to get that data. The problem for us is that the case improved for Iceland. But, you know, the average got worse for everyone else. So now we have this question of, like, if that's the case, then, like, maybe this local thing won't really work out. So what we kind of figure out from that is, like, this coverage isn't placement. And we have to be very careful when we went forward to figure out, like, how do we do this placement in a way that works for our customer? So instead, what we did, we kind of went back to the thing and we looked at what we could do. And we decided to do, like, Rotterdam and the surrounding region because we believe that, like, you would have a better experience if we did it in a place.

20:07were used when you weren't geographically distributed. And we had sort of also gotten more confident that it was going to work from Triadon Islands. We felt more confident doing it. So we deployed it on Rotterdam, and it was great for almost all ISPs. We saw similar gains, but we started seeing these other weird outliers. There was specifically one French ISP who saw over one second of Latest's peak. And if you're ever talking to someone and suddenly the responses come a second later, just imagine how bad that feels and then we dug in and tried to figure out what was going on and it turned out it's like not the servers they were not doing well you know there was nothing with the software and it was nothing with the network and the thing that we then realized is that before when you have a hyperscaler let's say you have us east one all the isps optimized to get into us east one if that makes sense but if you have cloudfer and you have a massive distribution then it might be that a small isp they optimize to get into the small one that is more local than in they don't optimized to get into the one that is in Germany, if that makes sense, if you're France.

21:03So we kind of ran into this problem that we created more front doors that goes into our traffic, and we didn't do a good job of figuring out how to route the traffic backwards. So like, you know, if the traffic coming to France, it would go across the Claufer's backbone, but then it would be dropped on the floor like a hot potato and go back over the public internet on the way back. So we kind of had this asymmetrical routing problem as we were talking about before. But Claufer has solved this problem. It only turns out that you know for all the stuff they do they've household this problem they have a great backbone but we didn't get to write on this backbone because we're sort of dumping udp and they didn't really deal with this so another thing we had to work with them to figure out is like how can we do symmetric routing for udp as in like how can they keep track of like where packets go in and out so they translate it or their backbone so you get the same network quality for everyone without sort of getting into the public intimate problems because then you're just back to the peer to peer problem that we had before and then the sort of second question we had to figure out is how do we run this right we said earlier that we didn't want to have anything bespoke and we were promised this magical land of cloud for containers so we said let's go for this but at the same time we were rebuilding our software and we were building our software to fit on the public cloud because again there was a lot of things when if you run bare metal yourself you know that you can like ignore a lot of the problems and just like hope that they go away but when you're on a public cloud like stuff like maintenance out of your control does not go away you just have to figure it out so we are already starting to rebuild the rest of you to solve some of these problems so luckily for us we had built an sfu that was like built for the modern kernel it has all this packet acceleration we're talking about uh it was small

22:36and flexible so like it could run on one core it could run four cores eight cores and it was a single threaded runtime per core so we just knew that sort of we believed that it was scaled linearly with cores of course that turned out to not be true and then one of the last things we wanted to make sure is that calls can move so when we're on bare metal a call placed like you know we know that server is going to be for a couple of months but in our case when we run on containers they can come and go so that the sfu itself has to be willing and able to say i know that i'm going to get shut down i'm going to move this call over to somewhere else and move that traffic in real time without user noticing this is a huge undertaking by the team but they managed to get this done which unlocked us being able to run this on containers if that makes sense so we did all this and it went really well and we started starting scaling it over the year and by the end of the year in december 2025 we had moved a lot of traffic to Cloudflare.

23:26Actually, it was almost 80% of that point. And I remember we had like an event in the office and we were celebrating. We were like, we're finally done. And we celebrated and we sort of, you know, made a ceremony where we turned off the last bare metal pop. And we were all super happy. And, you know, we're like, we're going to celebrate this. And immediately it was sort of seeing issues. We learned it's like the fun doesn't end when you sort of finish your migration. And it was really weird for us because we've been scaling slowly and it's been great so far. But as we kind of cranked the lever, like issues started popping up a lot. And it was really weird. So we tried to figure out like what was going on. Because what we started seeing was that like when we were off peak, the traffic was fine. We're doing really well. We saw great improvements. But as soon as we went on these peaks, we saw latency spikes that were like, you know, very, very bad for our users.

24:11And it went so bad that like users started figuring out because you can see in the application, if you're on a Cloudflare host and a DSP host, it started. figuring out that the CF's hosts were bad, so they started disconnecting many, many times to get another host, which just made everything worse. So we ran into this really adversarial problem with ourselves, if that makes sense. So what was going on here? We saw over 500 milliseconds of ping speak at peak, and at this point, seeing this, I just literally told myself, what have we done? Have we messed this one up? Have we sort of bet on the wrong horse, if that makes sense? But I think we went back to Cloudflare and we said down with our internal team who did a fantastic job. And we were like, how do we figure this out? Because initially we were like, you know, it's probably us. It's probably not Cloudflare, but like we couldn't be figure out what was going on and we couldn't be certain.

24:57We don't want to blame the vendor, if that makes sense. You don't want to be like, hey, you are the error here. So we have this problem where we're trying to figure out how do we figure out what is going on? We went back and forth. We added logging and we tried different logging methods, but it's really hard to see the issue. But then what Cloudflare came up with that we implemented with them was really smart. So we have this health check packet that gets sent between us and back and forth that has a very specific signature. So what Cloudflare did is they basically wrote an eBPF filter to drone kernel that would timestamp this packet. So that allowed us to understand, like, where is the latency coming from? Like, is it coming from the network card to the host OS? Is it coming from the host OS to the VM? Or is it coming from the VM to the R-app or then R-app out? And of course, after they wrote this, and we saw the statistics, it was very obvious that the problem was inside our app.

25:47So that was like, okay, great, it's our fault. As you know, you feel a bit deflated, but to the credit of the team, they took this with like great spirits and they're like, let's get to fixing this problem. And the good thing is it pointed us to where the problem was happening in our application. Because what we would see is that sort of one core would spike and when the traffic sort of started scaling up, the core would spike and it was very clear correlation. it was like the way we run it like i said like we have a single threaded runtime per core but the old holes we're running on bare metal they were not the most modern and they didn't have fast cpus and they didn't have fast network cards so when we had written as a few and deployed it on the old network you know it kind of just worked but the problem we run into in this case is we'll have a loop that will basically do like receive and then send and then flush but the new networks were so fast that we just kept getting packets on receive and we never got to the send and flush so we're kind of starving ourselves the loop would never continue.

26:41So we hadn't really anticipated of like how much traffic we'll be able to push on these modern CPUs with these modern network cards. And it was kind of like a learning for us and we had to provide it. So we fixed this problem and we felt very good about like, okay, you know, we got it, we're great. And we started to deploy it again and we did two, two, two, two cores, sorry, two cores scaling and it went fine. And we did four cores scaling, And then we did eight cores and it wasn't fine and 16 cores was absolutely not fine. And then we're back to the same. problem of like okay what is the error here but this time we were sort of like you know last time we blamed cloudfares this time it cannot be them it must be us so i need team was like they really went through every single thing they looked at every line of code they went through you know all different test cases and eventually we're like at this time maybe it's not us you know we don't know so we make the cloudfares sort of head down and we talked to them and they took it seriously to their to their credit cloudfares took it super seriously as seriously as the first time and they worked through with us and we learned looking into the metrics and kind of what like led us on to that was outside of our control was that we would see a core on the box sort of what we did is like we sort of wrote a tool that will sample the course some like h top but like for ourselves and we would see that like one core could be continuously pegged and then every other core would be fine but this core just like sit pegged if that makes sense kind of similar to the thing before but in this case we were like we know it's not us doing it so like what is pegging the core and it's turns out, as I said before, that Cloudflare Containers is built on Firecracker.

28:04And Firecracker is just like, it was built by AWS for Lambda, basically. So it's this idea of micro VMs. You have many VMs on a host and you're all small, but we were kind of doing a macro VM. So they hadn't really intended for this kind of workload to be run the way it is in Firecracker. So the problem we run out with is that in Firecracker, you sort of get an assigned core that takes all the network traffic in and then sort of fans it out to drive a course. And that's where scaling the other cores that core got extremely hot just processing all the packets that have to go to every other core if that makes sense so we're kind of just running into this kind of like it was a single pipe in and a single pipe out out of our out of our out of our containers and that made us impossible for us to scale above four cpus without sort of burning that core out and once we kind of learned that and we talked to cloudflare to the credit we came up with a fix which is like we're just going to do four core distribution and we're going to do many of them rather than like do one because it kind of fixed the problem.

29:03And then longer term, they entered the packet steering so they can now have multiple of these and figure it out. So we don't run into this issue. But those were like the kind of problems we faced when we were scaling this. But they fixed it. So we were super happy. And today it's a solved problem. We're good.

29:20I'm really proud of what we ended up here. We have this review running in production. We managed to get this transition over. And the results of this. is really where I think we are extremely happy with where we end up. We've gotten to 80% of voice and video that runs on Cloudflare today. And if we take one region, Frankfurt, we saw a 34% decrease in ping and a 42% decrease in packet loss. I said before that we would celebrate 1%. So 42% decreases for us is like magic. Like we worked for 10 years to get this thing better. And this kind of just like unlocked that for us in ways we couldn't have imagined. For Europe and Latin America, saw between 20 % and 60 % decrease in packet loss. Again, crazy numbers for us. If you told me this up front, I would have dropped everything I did and worked on this immediately. It wouldn't have been a question.

30:06And then the last thing that I think is very interesting, when we measure our user sentiment, we look at Santiago because we use the thing I told about before, the algorithm that pads over missing bytes. We call it expand. It's like a thing that's built into Opus and we're about to see. We saw a 40% decrease of distance antiaxia. which means that we know that the user experience is improved it's not just metrics right it's like the actual padding that we do that's kind of like weird i should have brought a recording with but like the kind of weird sound you hear when you're losing packets on a voice call that is down by 40 so we know that this is impacting users and it's good so this is what you can do if you run computer at the edge like and we were super happy where we ended up here but the thing that i think for us was so magical it's like there are products that we couldn't even before that we can build now we've had ideas for years of like what could we do with our client like we run a client on on people's computers right what could we do with this client if we had a lower lids network and i think now we're in this idea we're kind of building on two things that i can't talk about but we're building this idea that like it unlocks products that we just couldn't even imagine before and i think that is what's so empowering for us and why we feel so happy that we did this so the last thing i wanted to deal sort of leave you with like if you're I think take time to rethink your traffic.

31:23Like if you believe one thing about your traffic and you really start looking into how your traffic really works, you might learn that you have the wrong shape of the optimization for your entire world, if that makes sense. And then as you do this, you kind of have to build the software for new configurations. I think the mistake that we run into often is that we try to take our own software and sort of slowly shift it. I think we should have done earlier is to be very clear about like we have to take the new software and kind of build it from the primitives as we learned. And I think the third one, important like you're going to find a partner that is as crazy as you to do these things and with that that is all i have for your talk thank you very much