0:03I'm Ram. I'm a software engineer at Rippling, and we'll be talking about how we leverage Cloudflare workers and the rest of the workers' platform. But first, a little bit about me and Rippling. I focus on IT products at Rippling. If you are not familiar, we actually build, at the very core, we have... employment data but then we attach a whole bunch of things to that employment information everything from single sign -on and surveys and payroll etc etc it's a unique model but i think it'll start to make sense as we go further and a fun fact about me i was actually an early employee at cloudflare way back when 2011 so this is almost 15 years ago so it's a bit of a full circle moment for me coming back here.

0:56And also, it was a very different Cloudflow then than now. But it's also a little bit similar in the sense that at the very core, there's the servers that run every single thing as you heard during the keynote this morning. And then you attach more things. I think Rippling has some similarities there that you'll find. But let's let's jump into the talk. So I'm going to cover a few themes here. One is how do we go how we went over the years from serving human scale traffic to then serving machine scale traffic as part of our IT suite and then finally as we are living in the agentic era how we're serving agent scale traffic first we'll start with the people

1:44so the core thesis here with the people problem is we have companies that use us for their services and basically on data residency requirements they can choose which region they want to host it in as an example a company in eu might choose we want to host our data in the eu what that means is you need to route traffic for employees of that company to eu regardless of where they are in the globe so it can't be to the closest region which is typically comes for free with a network like cloudflare but you need to uh do a special routing here based on which company they're accessing the data for. And to illustrate this, I made a slide, but I wanna apologize in advance for this.

2:29But I wanted to convey the concept of an employee in the EU trying to visit a client in the US. This was AI's impression of it, of what that could look like. So when someone travels, you still need to send their traffic back to their companies. region that where they have established data residency so what that looks like if we take a two region example we end up leveraging cloud for workers along with a simple kv lookup i think it's probably the simplest of the use cases you can think of for for a setup like this the data doesn't change often and once a company chooses the region it remains the same so that's kind of where we were a few years ago Now, let's move on to the next use case, which is our device management product, where we, as part of our IT suite, we have companies that can set policies on computers.

3:29So an example of that could be, I want to lock someone's screen when it's been inactive for 10 minutes, as an example. And also, people could move in and out of groups based on where they are in the organization, and that could trigger more commands as well. or a company might write new policies. All of this needs to then be sent back to the device so that the command can be executed there, right? And when we initially built this, we built it around polling. So each device started polling for new commands at a set interval. However, as the number of devices grew, it caused a bunch of challenges for us, especially around scalability and costs. Because as you can imagine, change someone's policy doesn't happen often.

4:16However, the device is asking extremely frequently in order to stay up to date, what is my next command? And the other problem we were even more concerned about is certain sensitive commands need to go out to the device more frequently or very quickly instead of waiting for that one to two minute late polling intro we had set in order to meet the scaling demands. So, This is another illustration powered by AI of an ID admin trying to initiate a remote desktop session. I don't know why the ID admin is in a lawyer's office, but career goals, I suppose.

4:56So they're essentially trying to, as an example scenario, they're trying to help an employee with a computer issue. It would be super awkward for them. get on that call and say oh let's stare at the computer and watch for two minutes and see if it initiates you really want this to happen in a matter of less than a second or a very small number of seconds so what you truly want the architecture to look like is you want to push commands from your control plane over to the devices right but the moment you say push what you're really saying is i want persistent connections right you can't really push unless the device always connected and when you really say persistent connections you're really saying i want to do one of these things it could be long polling it could be web sockets or you could even go a little niche and do something like mqtt which is more common in the iot world but the issue with doing one of those is you end up with a bunch of persistent connections back to your control plane this and i'm i think i'm only showing 12 or 15 but imagine hundreds of thousands going into the millions of connections just staying open all the time, right?

6:11And the reality here is it's not even just your control plane. If you look at a single machine's request path, it looks a little bit like this, right? You have your edge slash CDN or whichever service we use Cloudflare. Then you have your load balancer. For us, that's AWS. So each machine is now opening a persistent connection to each of these. layers and that leads to a bunch of problems one of them is cost right each of those services will charge you for a persistent connection if you ever look up alb's pricing they charge you for how many hours the connection was open for which is not common with http requests when someone's browsing on a on their browser right because if you just send the data and it's done so that's one of the challenges and then your app servers how to keep it open then you have to run enough app servers to do that and then there's a bunch of technical complexity that comes with uh with that setup which is more often than not which is this was the case for us we have a pretty massive 20 million line monolith and trying to keep that running to just to hold connections is extremely wasteful because what you truly just want is keep the connection open right but let's say you solve You still have the issue of which of these servers should I send my command to?

7:34Which specific device is connected with server? You need to know the mapping. And if you don't want to keep track of the mapping, you then have to do a broadcast to every server. Or there's other challenges, like if you do a deployment to the service that's holding connections open, you have thundering herds. So where we were. going towards is essentially what if we wrote a small web socket server in front and we had extremely enthusiastic people that were like oh go is async io friendly and we should it's lightweight or some people were like rust doesn't have a garbage collector it's even faster but there is there is a few challenges with that but first i i want to say both are great languages we actually use them in our stuff We use Rust for our device client.

8:27We just go in parts of our control plane. But the issue is not the language. It's actually the part where you're running the WebSocket server, right? And this is maybe a bit of a spoiler alert. Hope you can cross over the hump and not be stuck like we were in the middle, where you consider running a WebSocket server, which is a very tempting technical challenge, but maybe not the right. right business problem to focus on. And I'll show you where we landed, which is our goal was, let's push commands to the very edge of our network and have a network that has millions of, or I don't know how many servers Cloudflare has, but I know it's a lot more than how many we do. So maybe someone that has 100 to 1,000 X more servers should hold the connection open.

9:17Whereas we just sit there, push commands. And the architecture we landed on is use a combination of durable objects plus worker. So the worker helps you act as a management interface where you send the commands from your control plane. And durable objects serve as essentially a addressable state for each of your devices. So the way we did it is each device has an ID. You have a durable object for each of those IDs. And the moment you send a command, uh the durable object ships it over the web socket connection and back to the cost part i was talking about earlier uh durable objects have this nice feature called hibernation so if you do websockets to durable objects but if you're not really uh sending anything into the do itself keeping the web socket connection open is actually almost free it doesn't keep the deal on uh if you haven't come across this i think we didn't know this feature either until we started doing some research definitely check it out hibernation is a nice uh add -on for durable objects so that's that's the story of how we solved our uh devices problem uh now let's get into agents um and and how we are tackling agent scale traffic so first just to give you an intro we we have a chat interface just like everyone else i feel like this is the new way of uh software point and click is the old school way now you can talk to software and

10:52more specifically I think I think there's a lot of chatter around LLMs and GPUs and innovations but I think the real innovation here is the human computer interface that has gotten so much more advanced here I'm asking it three different things which is I want to take some time off and I want to see how much time I've taken and I also want to how many meetings or interviews i would miss this is not an interface that existed before for any of us you had to kind of go piece this together right but how do you actually actually before we get into how do you make it happen and spoiler alert we'll talk about the google calendar connection but briefly just to show you it showed me what is the time off taken and then it showed me your list of meetings like standard stuff you've all seen this but also i got a little more curious asked it is this time off i'm taking is that too much like what's my manager doing or what are my peers doing and it instantly blocked me from showing that information hey i i figured out who your manager and peers are by traveling the org chart but i don't think you should see that information and so that's that's our view of how inference is pluggable but once you connect your tool calls and the permissions and the and your system of record data is the powerful thing.

12:09That's a review on how we're approaching AI. But let's get into how we're handling agent -scale traffic. So the way you connect your agent to third-party data is through MCP. Whether you like it or not, that's the protocol that's the most common. And there's a little bit of irony there. We spent a decade or so talking about REST APIs with weird formats like posts. slash v1 slash calendar and all of a sudden when its machines were saying oh we'll give you a nice tool called create calendar event and we'll write a description and so on and so forth but that's aside aside from me but let's move on so so there's mcp but then mcp at a company which i think are very different levels of how you deploy this more often than not at a company you want to know which mcp should be used by people or maybe even which tool calls maybe a company is okay with i want people to run slack reads but not writes they don't i don't want them sending messages or maybe you want to do different settings by department and at the very least you want to have an audit log of what's happening uh and the way we built this is because of our org data we're able to give our customers the interface however we still need to run this high -traffic agent to third -party traffic, right?

13:39And the way we architected that is you can go one of two ways. You can say, check with my control plane each time which things are allowed, or you can say, I want to snapshot those policies, push them to my data plane, which is running along the top, and you essentially end up turning this problem into an HTTP proxy problem, right? Essentially, that's what... it is there's http traffic going left to right at the top and then you have your customers giving you policies and so on and so forth right the moment you do that you start to see a familiar shape you can put that traffic on workers and durable objects one more time right um and i think this architecture has served us really well because oftentimes this traffic is very spiky uh and we've just two weeks ago we've seen someone write a crazy automation that was doing a ton of MCP calls to Google Workspace, which I think if it hit our control plane, we probably would have had a downtime versus having workers and durable objects handle that has been a thing we haven't had to think about.

14:47And once you have this setup, you can open that to not just our agent, you can do it for any agent, right? It could be Claude, Codex, and so on. forth and this is where it starts to become a very general mcp gateway and how do you then bring the audit logs back from the edge now that your traffic is actually living at the edge we end up using pipelines and then r2 which is the equivalent of s3 and it cheaply stored there and then bring it back into our control plane for querying so that we can apply the right permissions so on. And then as we started working on this and had customers looking at this, one of the questions we got was how do I host my own MCPs because my engineers internally have built their own custom code to expose tool calls.

15:40And oftentimes this is untrusted code. We could have them hosted but then you need to tell them how, where do you host it? How do you do auth or could have them give you the code but where do you run the untrusted code and and also remember we're running all of this on the edge so it was timing was amazing when cloudflare came out with the sandbox feature which is you just give it an image and the sandbox runs closest to wherever your request is happening and you get to serve customer traffic that took away another problem for us and then lastly while this was all happening we had another team that was building rippling's own mcp server uh which from once we heard of it from the get -go we wanted to put it behind the gateway reason is the same way a company might want to control which tools are allowed for who they might want to do the same thing for what's allowed on the rippling mcp server as well uh and uh we had a massive amount of tool call sprawl which was like we have 30 to 50 skews where uh we sell many many many services so instead of sending all of those we ended up leveraging code mode which is a way for agent to discover agents to discover and write code but again you then end up with dynamic code that's getting generated on the fly and dynamic workers nicely help encapsulate that as a runtime so that ended up solving that as well so in

17:15essence we ended up with this architecture We had our own MCP server, we had third-party MCPs and customer-hosted MCPs, all behind the MCP gateway, using pretty much Cloudflare's infrastructure, while we focus all of our efforts on our control plane, which is what we think we bring to the table. So that's on MCPs. And then you'll see a similar shape here.

17:46Similar to how your agents are making tool calls, they're also making inference calls, right? And those, again, are HTTP requests going back and forth. And you have many providers on the right-hand side and very similar to how a company would want to figure out what rules they want to apply to tool calls, there's a very high chance they want to figure out which models are a lot. As an example, maybe Fable and Astra are not allowed for... people at my company or how much costs or how much can they spend on LLM budgets and so on and again once you push policies up to your data plane you get a very similar shape one more time with workers and durable objects so I think we're really glad we invested in that like years ago on routing our people traffic however with AI Gateway there's a lot more to it you actually need need a pretty large model catalog, especially with open weight models being able to be served for extremely cheap.

18:50And also, you want some guardrails, and you probably want some smart model routing. So these are not really pure HTTP traffic things. These are like deep technical trenches. And we're working with the team at Cloudflare to figure out how to bring this all together. More to come on that. the next few weeks. But that's about agent scale traffic. So thank you. I think we got to cover human scale traffic, then the machine scale and finally agent scale. So thank you all for listening.