0:03Welcome. I'm Steve Goldsmith. I'm joined here by Maddy Toa from Uber. I was just about to say the wrong company. Yeah, don't say the wrong company. From Uber. And we're starting a new series here for Cloudflare where we're trying to talk basically about with technical leaders who are customers of ours, how do they use the platform and get into some real detail so we can understand collectively how different global brands like Uber are stretching how Cloudflare runs, but also how you might learn from how they're using these platforms and otherwise. Do you want to do a quick introduction, Matty? Sure, yeah. So I'm Matty Toya. I'm the Vice President of Infrastructure at Uber. I've been there about five months now, so relatively new. So maybe I would forgive you if you got the wrong company. I know it's a change for me. But I've been a longtime CloudBart customer and really excited to talk about the partnership, particularly for large enterprises and for infrastructure scale.

0:51Well, let's start with that. So five, six months in, what's your first impressions of sort of like how this... platform relies on Cloudflare or doesn't and sort of interesting historical choices that have been made around how this is working. Yeah, it's been really interesting to learn about a new business and the new way that the infrastructure kind of comes together. Obviously, Uber is a very high -scale business. We are mobile first, so a lot of the traffic is coming from mobile devices. We have multiple apps, so a lot of the traffic is coming from those apps themselves. You know, coming from a background with a lot of web experience, this was sort of a new thing for me, and I got the chance to learn a lot more about the nuances of dealing with traffic coming from devices. And also, it's a lot of real-time data as well.

1:37So it's not just like you render a web page and you're done. A lot of the data is constantly coming in, telemetry, information about a trip, information about earners on the platform, and so on. And as the business. has grown we've seen the kind of diversity of the requests that uber serves has also grown um and having cloudflare be a significant part of what our front door looks like has been a development that's really helped us reach the scale we had and i think also deal with the modern state of the internet today so if you start thinking about like tracing a request through the system it's your first day and you're like i open my phone i've used uber before right but as you start to unpack that how do you can you walk us through a little bit about like learning how the work and sort of like the connect-the-dots version of how things are flowing?

2:23Sure, yeah. So obviously with all good infrastructure, there's many layers, right? You always give anything to be solved with one more layer is usually what people say. And it really does start with the device itself, the SDK that's in the application. We do a lot of work on the client side to try to make sure that the requests are low latency, that they are using, you know, the right the right set of parameters to make those connections to our front door. And then once the request comes from there, it hits Cloudflare. Then at Cloudflare's layer, we have our DDoS protection, the dynamic WAF, all set up, which is a big part of how we make sure that we secure our network and deal with the threats of rogue scrapers or large -scale traffic hitting one of our endpoints on one of our...

3:16properties, and then also provides us with a good amount of telemetry. One of my actual secret favorite features, I think, of Cloudflare is LogPush. I think there's a ton of great data that Cloudflare provides to you about the traffic coming over the network. And in addition to sort of the way that the tools work online, you can actually get that data into your systems, which is a big part of what kind of comes in the second layer at Uber. After Cloudflare, it comes to one of our data centers. We have a mixture of... on -prem and cloud locations where we run our workloads and there is a Envoy proxy that sits at that layer. This unpacks and takes a look at what the message is to try to figure out where the right backend inside of Uber is to go to and then also sends off a bunch of the fingerprinting for that particular message so we can know a little bit about where it's coming from, who the client is, the versions of the client connecting.

4:12And then also the data that we get from Cloudflare about things like the job for fingerprint and so on, which is going to become important later down in the stack as well. And so that gets forked off to Kafka, where it goes into a logging pipeline. And then the request itself gets forward on to our service mesh. So we run our stateless workloads inside of Kubernetes. We have an internally developed service mesh that then routes those requests through. to the individual's backends that provide the actual support for the particular request that's happening. And then that actually does some rate limiting. So we have some internal rate limiting at that point. So if we do see traffic spikes or behavior that's undesirable or a system under load, we can either route around it or push back as needed. And then finally, that gets to the service, whether it's the thing that's providing you with an estimate for your fare on Uber or...

5:08potentially information about a merchant that you're looking at where you want to order some food for dinner. Whatever it is, it gets to that ultimate service at the end. And then we provide back the response upstream from there. And in that particular background, there is also a couple other safety measures we have layered in where it can do on the server side some pushback or connection termination if there is an overload situation. And then the error propagates and gets handled on the way back up. That's an excellent description. So if you think about which dimension most operationally in terms of scaling the system. It's the number of clients, it's the number of requests coming through. It's like, which one do you worry the most about? Yeah, you know, we think about this a lot. And one of the interesting things that I've come to learn about Uber's business is that we just have a lot of different products that kind of scale differently.

5:54You know, we have our core rides side of our business, which really we're thinking a lot about, OK, how many trips are ongoing at a particular time? You know, where are the errs on the platform? What's the data that's coming in? If you've used the app recently, you might note that sometimes there's an option to record your trip. So it could be like video or audio data that's coming in, support requests, et cetera. And so that's like a lot of data exhaust that's happening during the trip in real time. And there's also sort of this other side of the business, which is our delivery side, where there's a lot more browsing and selection and learning about, you know, what products are offered to you before you even start your purchase. You know, you're constructing a basket and then ordering a delivery from that way. Or even, you know, if it's not food, it could be a retail thing as well, where you're getting a local delivery from a local business.

6:44Those have fundamentally different dimensions, right? We think a lot about the concurrent trips, but we also think about visits, which is much more traditionally, I think, where Cloudflare comes in. And we, you know, as the business has shifted since COVID, we've seen that sort of overlap in the way you think about Internet traffic really. converge a lot more with the technologies that Cloudflare has been providing. I mean, that's really interesting to think about this collection of different use cases and how the answer varies on different. When you think about, I guess, the next year or two of your roadmap, are you consolidating those into certain kind of patterns, or is that diversity likely to remain? So there's always going to be diversity and probably even more diversity over time. I think one of the kind of powers that Uber provides is just a lot of different services to... bring these products that are accessible to people like on their phone and accessible to people who are earning on our platform, um, and then connecting them in this marketplace.

7:38And, uh, you know, there's always going to be an evolution of that. So I expect that that's going to diversify. I think what continues to get refined and continues to get, um, we're just saying is I just talked about all that telemetry data. We can use all that data to create the forecasting models to really drive the rhythm of our business. So we know what to expect, come say, halloween or something like that where there's going to be a lot of activity on our platform or um how many folks here watch the world cup this year you know world cup's a big thing for delivery around the world um and an interesting impact on supply when everyone's watching the exciting game you know it's like suddenly no one's driving cars anymore um so so you know understanding the dynamics of those events along with the historical trends in our data from the traffic coming in uh really allows us to prepare from the edge in in terms of how we provision, how we ensure that our capacity is safe, and then how do we make sure that our systems are efficient and performant in order to deliver the best results.

8:36So Cloudflare is a source of continual innovation. You'll see it around here for next few days. One of the pieces of feedback I get from lots of customers is you guys give us too many things and it's hard to figure out where to use them safely or new or whatever. So do you have a point of view on how you guide your teams to? try our new products or just generally products from other vendors and which part of the stack and where you're willing to take more or less risk? Yeah. Well, I think that it's important to always have some budget and appetite for risk. I think if you don't take, especially in infrastructure, I think traditionally, if you're working in infrastructure, a lot of people look at you as like the people who say no. And I say, we have to meet the people who sometimes say no. We can't always say no. But there's always an opportunity to innovate on how you deliver your services and how you incorporate new technology into your stack. Like we've done a lot of things. lot of work on DDoS with Cloudflare. And, you know, we still run a core set of DDoS protections internal to our systems for defense in depth.

9:29But we also recognize that there's a lot of efficiency we gain by being able to do DDoS protection at the Cloudflare level. And additionally, there's a lot of additional intelligence and insight that Cloudflare can layer on in terms of detection for some of the very large scale botnets that might be coming in. so it took a program to sort of work our way up to that. We also have the ability to sort of split workloads, right, where we can take a particular part of the market or a particular percentage of traffic and experiment with that independently of the rest. And that's a really important way for us to gain insights into how new products are being developed and also how new technology from an infrastructure perspective can evolve.

10:14You built that with us, didn't you? Yeah, we did. Actually, that's... I think there's a talk later. There's a plug that some of my folks, I think, are going to do a talk about config deployment, things like that. But yeah, having the ability to do a change where you don't impact the global world is just such an important fundamental construct when it comes to reliability and safety when you're dealing with high -scale infrastructure. And having that built into the platform itself really has enabled us to do more and move faster with the kind of work that we're doing. So I have two kind of related follow-ups. Everything breaks. all the time right yeah this is the reality breakage is real i think there's multiple incidents happening on my phone right now yeah it wouldn't be a conference without serious one not serious ones but there's been a couple where we're still here i'm still here yeah if it might really start you'll know um so just graceful degradation and how you think about like failure modes of the system and like at what point does feel is uber really broken versus degraded and how do you manage that through the whole pipeline yeah so you know Obviously, it does start at the edge, right?

11:17No one could reach the website, then we're super broken. And so we do have some contingency plans for if there's a major outage. And often, a major outage might not be that Cloudflare is having a problem. Sometimes it's like, I think I was in New York a couple weeks ago, and T-Mobile just had this massive outage. And I was standing outside with my phone like, well, okay, this is a brick. So how does that happen? And what can we do about it? There's other paths. costs to get to people if it's like over Wi-Fi, it's over a different provider, things like that. And sometimes we can use those to work around a network problem, but getting them to their front door. Once you're inside of our network, we do run redundant systems. We run active, active setups for all of our critical workloads. And then on top of that, we have a tiering system where we think about different workloads, like which are ones that absolutely must run, which are ones that we can break for a period of time.

12:05And so if there is a failure, what we'll typically do is We'll bring all the services online in one of the other active regions and locations and then take down some of the lower tier capacity to make room for it and then slowly fill back in and replace those lower tier services with new capacity as we can begin to scale behind it. And this is kind of a good balance, I think, of like having a good resiliency plan when you can drill. We drill this regularly, usually once a week or so we have a small drill associated with this. control costs, which I think is on everyone's mind, right? You can't, if you're just running tons of infrastructure all the time at max capacity, you're going to have a hard time with your business, I think. So then the inverse of that is performance testing and scale. It's not degraded. You just want to make it faster. And for complex systems like this, that's one of the hardest things to do is figure out where it's slow and then how to make it faster.

12:53How do y'all think about solving that problem? Yeah, we've been doing a lot of work actually on like squeezing out every ounce of latency, you know, reduce the spinner time like if you've ever tried to get your car and then you see a spinner while you're waiting to get home on like a cold day you know the frustration grows very rapidly um so reducing that time is a very big important part of how we try to think about the quality of the app um the quality of experience for the the people who are using it the earners on the platform and the merchants in the marketplace and um you know it it does it's an end -to -end problem right like it's as much about what's in the client as much about what's the network transit to the edge and then getting through you know cloudflare through the smart routing to our our servers like every leg of that can have um room for variability and then you know we're dealing with millions and millions i think something like you know tens of millions like 40 million or something like that trips per day or being having on the platform something like that um and and these things you know you will see the tail like you will have those tail experiences we're a global company operating in parts of the world that don't have as great internet as San Francisco.

14:04And so those things do have an impact on that experience, but if you can't see those details, if you can't section the data, then it's difficult for you to begin to optimize and to talk about what can be done. Like, what can we do to reduce the tail latency to get to the front door, right? Like, we know very well how fast the backends are, so we know that we can actually deliver to the network a really fast response. But then how do we get there? back to the end user who's on their phone someplace because that's ultimately what drives that experience of the spinner uh are there some example projects you could share that you've been working on uh to to address those kind of upstream legs oh let's see what we can talk about um we've been doing uh a bunch of work on the specifics of the rpcs i've been using in the sdk um i don't want to talk too much about it mostly because like i don't have all the details on my fingertips so my team is here they'll call me on it if i if i say um but um but really you know like you trying to use um like all the new quick parameters like really get the traffic off the off the phone into the back end with as limited handshakes as possible and then also taking a look at the behavior inside of the client to make sure we're using connections and caching things locally as much as possible um we've also done a just a pretty big migration just in the past few weeks to get all of our static content to be cloudflare's cn which is also a huge help um because there's a lot that can be done with like um uh cash shielding and like the way that uh you can deliver

15:37the content even if it's not just yet available locally for that particular user we can get it from a nearby region and then push it out that way so slightly increasing the cash hit rate of our static assets like those things each add up and when we're talking about you know a p you know 99 .9 latency right it really matters to do those types of things that's fascinating um so the the i'm going to tee it up for a little bit of like cloudflare specific stuff so of course yeah that's what we're here for right so yeah how do you decide when we are the right partner to solve these problems and when we're not and then i have a bunch of follow-up questions yeah i think you know i think a lot of times it's not a zero -sum question right i think that a lot of times we we look at the capabilities um that cloudflare has and it's a lot of the discussion is around that like how can we incorporate the best without giving up what we already like and i think that there's a lot of times where you can engineer yourself into a place where you actually can have your cake and eat it too with some of these things so i talked a little bit about the way that our um ddos protection works with waft right we knew that we needed to help with this sort of volumetric attack type problem because As we were seeing, I think there was a, if you saw the keynote, there was this great slide of like, traffic go like this, revenue go like this, and you know, on the internet, right?

16:58And I mean, like, if you're here, you probably are aware of this trend, right? And you start to see this sort of rise of non -human traffic on your platform, you need to have solutions for it. And Cloudflare has some really great solutions for it. But we also know that we have like a core set of identity associated with your Uber account, a lot of telemetry associated with your behavior in an app. et cetera, which is not necessarily present on the Cloudflare platform. But we've been using that for years to actually protect our platform and have a number of services that actually depend on that data to build it, right? And so that's the layered approach where we can say, okay, we build our WAF that uses all the tools that Cloudflare provides. We keep our system that maintains our understanding of the nuance of traffic with our internal data.

17:46stitch that data together in our pipeline to then feedback loop to both systems a configuration that represents our intent in terms of blocking traffic that we believe is illegitimate or useless on the platform um and so that layered approach kind of is this sort of you know goldilocks zone of being able to do both at the same time um similarly when we talk about um the way we think about uh you know some of the more compute and storage products that that cloud floor has like we have our own data centers we have a number of cloud footprints so we have a lot of resources from that perspective um but there are still like use cases where like r2 um is a good solution for certain types of data to store where we're wanting to either pull it from cloud floor or make it accessible on the edge and the web um and so having options i think is a really powerful thing and so when we talk about building infrastructure for the future having the ability to maintain um a level set of the complexity of your infrastructure while still being able to try new things is kind of the name of the game and that's where sort of this internal platforms really comes into play and where i think that you can drive a lot of value from your engineering team focusing on that commonality and what's the substrate you use to manage all of this it's actually a good segue to my next question if you think about a partner like us is it you use cloudflare or you use an abstraction that's powered by cloudflare like paved roads or like the actual native solution that's an interesting question like i would i mean i think it would be

19:21weird to say we don't use cloudflare i think that like we definitely use cloudflare but you know one of the reasons why working with cloudflare has been great is because you do provide a lot of apis you provide a lot of um configuration primitives where you don't need to click ops your way to success you can if you want to but like ultimately when you're a large enterprise running large infrastructure that's not necessarily the approach that you want to take to driving your business. And so having the ability to build the commonality in that API layer, and so you're like, okay, yeah, there's three different APIs. Some are internal, some are provided by vendors, but I have one system that I actually use to push out into it, that makes a big difference. And so that, to me, is sort of the way that I see us moving forward, and it's been kind of interesting with the way that.

20:08agents have been developed over the past couple years now that we're seeing those become much more richer interfaces because we want to make them accessible for agents but like it's just generally good and useful for any type of automation that you might have whether it's you know traditional whether it's an orchestration layer whether it's something where there's an agentic experience associated with it in any of those cases ultimately you need those hooks and having those hooks allows you to build what you need for your business you interesting um what is the biggest product gap that we have that we have yet to fill oh i'm gonna complain about my least favorite thing it's gonna it's gonna sound petty but i'm just gonna bring it up because i just drive me crazy um so we're a lot of us are cloud customers we have big big cloud footprints things like that and as part of the sort of evolution of what it means to run infrastructure on it internet you know we've all been developing this idea of like fin ops and like how we manage the costs when you're dealing with cloud services and sas and things like that um one of the hardest things i have to deal with is the billing with cloud player the billing data and managing it is just a seriously complicated thing and you know as i mentioned we do a lot of complex forecasting we do a lot of uh forward-looking planning and so you know being able to really understand that aspect of our business in the same way we think about our cloud spend etc is is a powerful thing And I know there's a lot of people working on it, so I don't want to pretend like that's not happening.

21:35But it is a place where I'd love to see more just because, you know, whenever you're doing an investment with a partner like this, cost is always a part of it, right? The commercial side of it has always got to be part of the equation because ultimately you have to make your business make sense. We've seen tremendous value, which is, you know, why I'm here on the stage. You know, it's easy to talk about things that are showing a lot of value. But we also want to measure and make sure that we're planning properly for the work. the system is running in the way that the traffic is evolving over time right um that's very fair feedback and something we've heard before yes a couple people in this room are in a pile i'm sure of a new billing platform that okay well maybe i should uh try to get myself signed up if you know somebody you'll cook you up um so i'm gonna switch gears a little bit and this this is sort of like based on the products you offer the customers sure switching gears talking about like managing engineers and managing engineering teams and building things these days yeah What's the most interesting thing you've seen in your first five months of maybe how Uber's doing something differently about like building products than they would have built before?

22:35Yeah, you know, it's I think one of the things I really appreciate is that despite having both a diverse set of infrastructure platforms, as well as a wide variety of microservices, like Uber is kind of a little bit famous for having a lot of microservices. The teams that are really amazing. job at figuring out ways to manage that diversity of workloads and platforms and i think that like that has actually been kind of the key on lock as to why this is sustainable like how do you actually run across so many different clouds how do you actually orchestrate so many different teams across the company who are building their own services and kind of have like full ownership of them in a way that then still allows you to do platform things like we're changing your front door to the internet We're changing your service match behind the scenes.

23:26You know, those types of changes are pretty difficult if you don't have the right layers and hooks. And at Uber, we really think about that as like a first principle part of the problem. And bringing that investment in those foundations has been a big part of the success that's got Uber here. And I think it's a big part about what we're going to need going forward as we think about the realities of what it means to have, you know, capacity in. 2026 looking at 2027 what it means to um you know continue to integrate with you know the new and innovative companies that are out there so i think that like those platforms are a really big part about what makes infrastructure great some of the things that we're working on around uh percent rollouts and safe changes and stuff like that yeah we've partnered with you on but you want to talk a little bit more about how you're thinking about using a partner like us to do percent like i want to change it just for this one region this one slice of traffic yeah You know, just to sort of start with, I think I'd like to say that one of the things that we've really appreciated is the willingness to, like, work and partner together to do these types of things.

24:26I think when you're making a big investment with a provider, then you really want to have someone that, you know, not just is available for help, but, like, actually wants to do that sort of co -engineering type things for these, like, specific types of programs. And then also, I think the sort of values around, you know, scale, latency, and performance. Like, we feel there's... a shared set of values there when we talk to Cloudflare in terms of how we think of our own infrastructure. So it's been really great to have those kinds of things. But about config rollout, I think that it started out as a conversation around safety. Like really, we were looking at all of our systems in terms of like, what is the global blast radius, right? Because, you know, there's a, it's been many years as an SRE. And if something was a global rollout, it wasn't like, you know, when is it going to blow up?

25:15blow up it's like eventually someone's going to take down everything because it's there right it's just it's there for you so um being able to do these types of rollouts and these types of changes which affect you know uber .com which is you know kind of a big deal is is scary if you don't have the ability to slice the traffic and the sort of techniques that we were able to um work with your team to deploy to really have our configurations focus on like slices of traffic in regions or based on header information or based on other parameters that we could set allows us to do similar sort of progressive rollout that we're doing inside of our environment on the edge as well. So that was just a really big unlock to allow us to be safe. But then also now when we have a conversation about like, okay, we want to make this change to our configuration to adopt a new feature or there's a deprecation for a configuration format.

26:07We want to go to this new one. We have a way to do it such that we're not going to immediately take. on all of Uber just because we do this migration. And that's actually, I think, a really important tool to have no matter what your system is, but especially for your edge. Because again, like no front door, like no one's coming over, right? So that's a big deal. So if you think about like just the places beyond billing where you're working around Cloudflare, what is your biggest workaround right now that you say, well, we have to do this differently because Cloudflare made a choice that's frustrating to us? Yeah, I think one of the challenges is always... always this sort of like um the global internet's kind of a complicated place and i think that we do see that in certain countries in certain regions connectivity is not as good as we'd like it to be for certain providers and like we can because we have this like amazing distributed pingdom quote-unquote pingdom type setup here where like it's mobile phones all around the world we get to see a lot of like what's happening um globally at any given time and you can see that like there places where um the connectivity between cloud flare and this provider is not as good as like other paths that we might be able to take and so we we have worked around just in like specific regions um in order to increase the reliability we've actually put in like a sort of fail fast method where we can like you know instantly flip between you know one path or another if we need to just to work around those without you know much interruption to someone who's using the app on the client on the client side yeah yeah it's Being able to own the client has been new for me.

27:38It's been a really powerful thing to be able to actually solve some of these problems. You know, Chrome won't do that for you, right? So this is a really useful tool in the toolkit. But, like, ultimately, you know, I like to think that Cloudflare is a company that thinks big. It'd be interesting to think about, like, what does the Internet look like in the future? Like, is there other ways for the ISPs and the mobile networks to integrate better so that these aren't just persistent problems? problems like the way that they currently are today it does feel very i don't know i'll say like early 2000s internet sometimes i'm just like okay i guess that someone didn't plug a cable in but it's real it's kind of how it works right there's still switches and routers and things like that but um there's a real there's a latency penalty when it happens and then potentially depending on um the specifics you may or may not have an alternate path for that particular city or region or whatever it is that's being impacted at that moment we do have some work on the h3 quick congestion control that last leg yes but what it's missing is a partner who has the client side telemetry to tell me if it's a wonder who might so maybe i didn't realize you know my folks are nodding their heads so yeah yeah yeah this is good yeah okay last question then we'll turn it over to the audience um if you had one magic wand to have us build one thing on cloudflare what was the one thing you would just say it doesn't exist today i wish you guys did this if there's one

29:02thing i think

29:06this is a really good question okay so i talked about log push being like an awesome an awesome feature i think there's like a real interesting challenge because getting the logs out of cloud flare means that you have to have some log system on your side that actually can ingest it and i think that's actually you know we are doing it we're a big company but like i really do think Having long horizon aggregate statistics about your traffic is like super useful. And I think that like having more of that provided like inside the box in Cloudflare could be a really awesome feature because, you know, there are a lot of problems that you just can't see unless you look at specifically where your traffic is coming from and what are the headers, what are the response codes, like how are the clients behaving.

29:53That kind of, even in aggregate, like that kind of thing back. seven days can be a big difference like you might not detect a small problem for a week um but when you want to go look at those logs if you don't have them already then you're kind of in trouble um you know if that's something that could be in the box that would be really great to have so that'd be my push that's a good my log push my log push yeah we did announce a log transform product earlier this year where you can do like on the fly coalescing and transformation of formats from our format to yours so that for that ingestion process yeah yeah i think it's i just i think that ingestion is its own like hardball of lacks to deal with and um i don't know i'd say that you could enable a lot of a lot of people if that could be a thing awesome thank you thanks very much thanks maddie very much for being here so much this is wonderful yeah