0:03Thanks for joining today. My name is Marc. I lead solutions here at Cloudflare. But actually my job is really easy. My job is to make Alex look good. So give it up for Alex. Thank you. Alex, tell us who are you, where do you work, what do you do? Sure. I mean, Marc, we know for quite some time now. Probably four years ago. I think as long as you work for Cloudflare probably, right?

0:34Yeah, I'm Alex Braunreuther, Chief Architect at Contentful, now Contentful by Salesforce, got acquired just a few weeks ago. I actually used Cloudflare through my previous company that got acquired by Contentful, which is Ninetild, a personalization software that runs completely on Cloudflare and Roblox. objects. That's why the title of this talk, 100 billion profiles. Yeah, and I'm doing software architecture mainly. Okay, that's dope. So you had your startup tiny. How many people were you at the time? 9 tired? Roughly 30. 30. That got acquired by? Contentful. How big is that? 800. How big is Salesforce now? 85 ,000. Nice. Okay, good journey right there. Okay, cool.

1:21Tell us about that. So you have it present. software and the talk title says like you have a hundred billion profiles last time i checked we have like eight billion people on the planet so what is a profile what do you use it for and what do you need dribble objects for yeah so um when we when we started building our nintel we wanted to to build a truly api first personalization software so personalization software and experimentation at the same time normally work like this. You get some HTML diffs, which you fumble into your website somewhere in a client, which is pretty annoying because it's slow, you've got flicker effects, and it's simply not API-first.

2:08So when you think about agents, for example, today, they could just work on raw JSON -stored content or you simply don't have HTML, you work in. apps or in a kiosk or wherever and this was was very important to us on the other hand performance was to me extremely important so i i come from the gem stack era of of web development basically and good times good times like like get squeeze out the last bit of performance you can get out of a site and then you had a had a script on your site that runs for two seconds until you flicker something and that was pretty annoying so we wanted to build a solution that is extremely low latency anywhere in the world and that's why we use durable objects for that so basically any profile that you got is one durable object so that makes for a very nice scaling effect in the very very low blast radius so why is it 100 million durable objects or somewhere in this area even if there are less people because our system works a little bit like at the end So you could now have a profile here.

3:16When you open the contentful .com website, you get a profile here. When you travel over to Berlin, for example, then it creates a new profile there and takes the data over to a durable object in Berlin. When now Cloudflare uses Salesforce, for example, the Salesforce server will probably inject some data somewhere in the middle of the US. Then it will generate a durable object. the middle of the US and then redirect the data to Berlin, basically where you are right now. So your profile travels with you, but it's profile partials that build up a system that talks to each other. Dang, that sounds pretty cool. So basically my user profile travels with me around the world. And basically you have a small database for everyone who visits your customer's website?

4:05Yes, so every user is one or even multiple small databases. Basically, they're Dribbble object partials of their profile. And yeah, it travels with you because it has to be absolutely minimal latency as much as you can. When you started building on Dribbble objects, so we met in 2022. Dribbble objects were super new. So like we now have merged with Dribbble objects on it at Scarflet. I can tell you in 2022, we did not have merged with Dribbble objects on it. I would take one of those. I'll get you one, actually. I'll get you one. Don't worry, don't worry. Why did you bet on it? I mean, like AWS already at the time in like 35 regions or whatever. Why use StrubleObjects? So the problem with a low latency system that is fully distributed is eventually consistency, ultimately.

4:57So we have put our bets on workers in 2020 when we started. The problem there was like, Distributing your state to multiple regions is insanely complex because the system becomes eventually consistent. Let's take this traveling problem. You would travel over from here to Berlin, for example. Now I have to synchronize. Let's say you have a Redis, for example, here and have a Redis in Berlin. Now I have to synchronize somehow over a bus, then your profile also is available there. Okay, when you travel, that's probably not that big of a problem. your plane is fairly slow and for sure a lot slower than speed of light. But when you think about the problem that I said, okay Salesforce server sits here somewhere and updates your profile, this leads to inconsistencies and then out of a sudden when you're in a website you just flick around through different content because the consensus mechanism of this whole distributed system becomes very complex.

5:59And the interesting part here with DurableObject is they play very nicely to the problem because they are transactional in themselves. So basically, I like to think about durable objects like the cap theory. It's actually not possible to have all the three parts of the cap theory. But as soon as you take the durable object as your atomic instance of something, it's basically in itself consistent. available and you can scale it out to the get now because at least we can because you do that for us thankfully and this is basically a perfect symbiosis of what we have needed at the time okay well you can tell we have a serious talk here it's your technical talk because the cap theory was mentioned so I'm really happy thank you so much for like checkbox of my bucket list I appreciate that and I'll try to like take that back right so you're globally distributed was important but also making sure that my user profile is always consistent with what it should be basically right like strong consistency guarantees but now you have like 100 billion small databases yep i personally know how much of a pain it is already to manage one database now you have 100 billion of them yep oh how tough was it to shift your mental model to say okay how do i go from i feel like engineers maybe like you better new technology, most likely there's not a lot of experts in the industry that were like,

7:33oh yeah, I worked with dual objects for 10 years, you know, like these take CV requirements, whereas like 10 years of dual objects, they're not there yet, right? So what was the biggest challenge for your team, maybe in the mental model of how to work there? Yeah, I think before we go into the mental model, like one factor that is extremely interesting there is like we were working with some of the biggest enterprises in this world, which have… a ton of traffic. And one big problem for most of our competitors always was blast radius. A classic personalization software works like this. You send in events, they go into a big data warehouse, BigQuery, Redshift, whatever. You have batch mechanisms that produce new profiles, put them into a cache, and the cache is basically live. Let's say you get too much traffic and your redis cache.

8:25runs over or gets too much traffic and simply blows out. So you have a multi-tenant system. One big enterprise kills your whole region. Or when you have sharded it to this enterprise, it kills itself. With the durable objects, I don't care. If they basically come in and send 100,000 requests to a single profile, this profile is dead. But there are a few hundred million profiles. They don't care about it. They just run. So like, I think since we run the software, we had never a single download. time which which is great and like the scale you can put on the system so it basically slept pretty much through every black friday and nothing works so now we how how how do you i think your team needs needs really a mental mental shift for that that's that's extremely important um the most important thing here is to not fall back into what you have learned So an example would be, I need to do GDPR deletion, for example, for my profiles.

9:31So I need to delete them after, I don't know, let's say 90 days of inactivity or something like that. I need to delete my profile because a customer has set up this. So how would you normally do this? So you would have probably a central index where you list your... or you have all your profiles in a Postgres database, and then every day or every hour or whatever, there's a crontag, you have a cron drop, that other thing, yeah. Works great when you have a few thousand or a few hundred thousand or maybe a few million, but when you have like hundreds of billions of these profiles, now all of a sudden, this query for this deletion can simply kill your database. That's pretty annoying. It sounds like it. So you basically now need to shift your mind.

10:17When you think about durable objects, you don't manage all durable objects. Every durable object manages itself, so they have a great mechanism, which I think actually was the moment where I thought, okay, these things are brilliant, or durable object alarms. Because with this primitive, you can solve any complicated distributed system problem So durable object, in our case, is just like, whenever it gets updated… It sets a new alarm for one year, and in one year it kills itself. That's it. It's very, very simple. On the other hand, what the team then needs to learn is you always have to apply this thinking. For example, there could be the case of a customer comes in and says, I need you to delete all my data.

11:07Now you're in a pickle because the Cloudflare APIs are actually slower in rate limits. our profiles increase. So basically, let's say you have 100 requests which give you 100 profiles per second. When your profile rate increases by more than 100 profiles a second, then you cannot query them anymore. So the naive approach which our engineers directly went was like, okay let's index everything into BigQuery and then we have a huge list and we can fetch this and go through all of them. It's pretty annoying. You can think the opposite direction. You just have a helper database where you say, I have my tenant, and my tenant says, I am deleted. And every day, every durable object has alarm, looks into this database and says, should I delete myself?

11:56No? Okay, I'm going to stay. And I mean, as basically a database read doesn't cost much because you can put a cache in front of this, and the durable object just wakes up and takes three milliseconds of, or. whatever like a very very small amount of time it's not very costly and they kill themselves so you just turn the problem around and this is i think the most important aspect here like make it clear to the team to not take the obvious solution here cool before i open up to the audience i have two more questions for you but if you already like want to think of questions that's my signal to make you guys think a little bit uh So, I'm curious, you jumped on this really, really early on this primitive, most likely when it was, I don't know, better.

12:46How was the journey from jumping on the primitive early to where it is today? Did it get better? Yeah, for sure. I mean, if it wouldn't have, then we wouldn't be able to run it, actually. Because in the early days, betting on workers was… really a hard bet i have to say um and and on durable objects i mean in the very early days there were no locks so there was no lock push so you basically ran a worker input and you could not run it on on on dev basically on your local machine you deployed it to the cloud and sent back errors by a header or something like that so that's how we developed in the very early days which was a bit crazy but yeah then another topic is like how you do tracing so like just uh um the specter attack problem so that you don't have timing.

13:36We have built a complete own system before now, the workers' observability came somewhere to aggregate logs into batches so we can ship them off to Grafana, for example, without killing Grafana. That's another problem when you are at the scale that you just kill downstream systems, basically.

14:02like there was no event bus or there was nothing like R2 or I think KV was there at the time when we started for sure but that was pretty much the only thing yeah and I said I think it's actually it was great that there was not a lot of things there because you have built the stuff the way you need it so that's maybe I like that you create all this new new primitives but I think you don't need them so you could build everything yourself pretty easily. I'm happy to hear. Now we have the options. You can still do the hardcore path like you guys did, build everything yourself, or use the managed features from Cloudflare now, I guess. Maybe last question before I hand it over to the audience.

14:48I know we met more than once, so you talked to Cloudflare. I think you were not always happy. I don't know. How did the collaboration with Cloudflare happen? teamwork, what can we do better? I think it's very normal that you're not always happy. So that's for sure. It's like otherwise there wouldn't be innovation to move forward. I think it's very important to be vocal about your expectations and what you're building. So I think this was probably the best things that we have. always transparent with your teams and basically this idea where they said in the beginning that our system works like a DNS actually came from a solution architect from Cloudflare because this was a central piece to solve for us and we didn't know how to solve it on global objects and we basically sat with a person like for two days in a call until we were like have you thought about this and we're like oh my god that's brilliant that's cool yeah but this only works by trusting you as a vendor enough that we basically get you early into our plans and really let us help us, so let you in.

16:04But I think that that was great. And I mean, we have talked about so many things. I think we were always very blunt about what doesn't work and what we need. Can't confirm, can't confirm. Bluntness was present. Yeah, yeah. And I mean, you delivered over time. So some things for sure. It's always nice to have every nice thing from the get -go, but yeah, cool. That was the important part for us. Alex, thank you so much for sharing what it actually means to run something at massive scale in production. Now that you're acquired by Salesforce, I can't wait for your talk next year. 100 million trillion DOs on Cloudflare, that'll be great. So thank you so much, everyone. Thanks, Alex.