0:03I'm Lai Yi and this is Doug and we're going to talk about how to measure real world network quality for end users. Oh, you should probably get through them. Yeah, thank you. I'll start with it. Okay, great. Okay, so quick last session, crowd participation is mandatory. Who here has taken a speed test? Okay, great. So chances are the speed test that you took, Doug or I had something to do with it. it so sorry uh so i'm like i'm the head of radar at cloud flare um you might know about radar from our outage reporting we also report on uh bots activity ai trends um but we also report on internet quality and that's kind of my background i came from measurement lab which runs the speed test embedded in google search so whenever you google uh how fast is my internet it's the one that pops up and so um a lot of what i'll talk about today and what both of us will talk about today is our you know shared experience with that.

0:57We'll let you introduce yourself. Hi, I'm Doug Suttles, most known as the founder of Ookla, one of the creators of Speedtest. It was the first thing in Google search until they decided to do that, and it became the second. I always like to point out that if you look on internet archive at Ookla.com, you'll see a picture of my cat from 2005 and prior before we started the business. We named it Ucla because of the cat, because I own the domain, and it was just a catchy domain to use for that purpose. I also like to point out this italicized quote here, where I think I'm probably as guilty as anyone for convincing the world that throughput or max bandwidth is equivalent to user experience on the Internet.

1:45That's just not the case. Today we apologize for our samples. Yes, yes. when i first started chatting with ii it was i said this this company orb that i've started now this part of my apology tour for all the years that i spent uh not lying because in the early days as we know um i just jumped into my slides you know we had this heavy focus on speed you know back then actually my next slide is about that so let me make sure i get in the right order here and don't ramble too much but uh you know isps have always priced things fix fix broadband by balance tiers they needed something to differentiate getting more money or extracting more dollars out of you and you know this whole question of am i getting what i'm paying for is common in anything that a consumer purchases but for the internet they needed something to do exactly that and speedtest .net was born out of that but what it fed into was this flywheel of marketing You know, I like to say the biggest component of Ookla's and the speed test business is getting all this data to then allow an ISP to use it to say, hey, we're the fastest, we're the best, something along those lines.

3:04And, you know, the perception again of what is network quality has just, it's long been, go take a speed test and see what the number says.

3:15Graphic here, I don't know. me if you know uh iron man 3 i always like to point out when he um i i got a call from marvel one morning and uh i thought it was i thought i was being pranked and uh they said we have an upcoming marvel film we'd like to you you know film speedtest.net for it and i said oh no not with all those ads i put on there that's awful um that's it and they let me make a custom render for them which got dropped into the movie i believe that the phrasing in the movie i didn't make the i got a two second clip with no audio and no effects and so we just built the little thing that dropped into the laptop screen i'm at the movie theater and all of a sudden you get tony stark saying um we need to adjust the isdns is what i believe the phrasing was i'm like what okay anyways times have changed significantly i left ukla three years ago you know it's owned by a public company i sold it a while ago and our ability to innovate and something new was shackled by you know a lot of typical public company needs of growth revenue all of that um but nowadays is i like to say you know the available bandwidth that you can get with connectivity often exceeds what you need to do most things and the chart at the bottom here this only goes through 23 but it's 20 years it's covered i like to say that since the beginning of speed test you know measure throughput has gone 100x.

4:44I think the only application, aside from like massive file transfers, if you're into like big media or something, the only application that really needs this is speed data. And it's, you know, I was on an event, the cable show event once, and one of the operators, CEOs up there bragging about their new gigabit over Doxis, you know, and that was the question from the audience. It's like, what's it for? And he said, it's for speed testing.

5:11But, you know, the thing at Ukwa and the thing that we did with Speedtest that was hugely advantageous to them was basically built our own CDN of sorts. The Speedtest server network is a bunch of infrastructure that is sitting inside of ISPs managed by Ukwa, but it's only for that purpose. And it's, you know, can lead to some questionable things, as I would say. of traffic you know you name it it's it's not always the answer that you're seeking it's always the optimal result so um well we we uh oh i gotta make this point it's uh my favorite analogy these days is you know the car analogy of do you care more about top speed or every other thing about the car you know if you say top speed then okay maybe you're one of those people that transfers files all the time as well so we need something that orion's you know lives is a real world experience orb is that for me i took all the things that were asked for in the ukla days that i just couldn't do and said let's build that let's build a software probe type orchestration solution that lets you measure any leg of a network measure anything but uh you know we we wanted it to be available to the layman as well as the hardcore nerd and be equally useful and so orb is a uh it's a three -tiered measurement you know

6:41solutions we'd call it where we measure responsiveness which we think is the most important component reliability is responsiveness over time that's the other thing with orb is it's always measuring you know we got out of like the spot checking type solutions that we had in the past and then speed we still do speed because it is expected and so Orp can run on almost anything. I even run it on my unified doorbell at my house, as an example. You know, it is a lightweight software sensor that is constantly measuring lag. And so, let's talk about lag. You know, lag is application layer latency. You know, this isn't ICMP. It's technically running over H3. And anything that happens to the connection, packet loss.

7:29you name it, you see it in a single metric with lag.

7:33It has no impact on the connection itself. Since we're running with basically an empty calorie set of packets that are flowing to these endpoints, you can do this all the time with no impact, which is another thing. You take a speed test, anyone else on your network feels it. And the last point here, I said it in the prior slide as well, but, uh... that, you know, that network, you know, when ISPs are chasing to be, you know, to get a marketing message, they'll do whatever they need to, to make sure that it's a delivering masterfacity.

8:11And so what I was going to do here, I made a video, but instead of showing a video, I like doing demos. So I was going to hop out to actually show what Orb is. And so, as you might've seen before, there is an application, Orb runs as. and by the way it's all free for consumers you know i'm a big believer in this play it's a brand play just give it away difference is we're not a data business we don't mind your data you can even turn that off you can send it locally to uh some local endpoint we have push and pull apis from every orb it is meant for even being able to run air gapped off the internet so it is all the things that we weren't before data sovereignty is a big deal to us um and like i said runs on almost anything so this is my orb account you know and i've opened the orb app on my on on my mac and the big orb here in the top left is the local network that's the one running in it's a sensor running inside of the app on this laptop and so if i switch back just a couple views here this is the default layman view as i would call it And so optimal scoring, it's 96.

9:25That's probably about as good as you can get over Wi-Fi. You have to be, you know, all of our stuff has been architected with a lot of key people involved that, you know, I think our scoring is probably about as good as anyone's out there.

9:41But let's do a speed test. And this is testing the cloud player. This is what Lai will talk about predominantly next. But if I... you know wrap this up we had to have a new ui i was tired of speedometers and whatnot it will measure full capacity we know how to do this this my business partner that i co -founded ukla with the technical guy that did all the engine work behind it he's working on this with me so this is a hundred percent people that used to work at ukla work at orb but have a very different purpose and a different mission um that test is finished but now what i was going to point we don't automate that test we don't want to because like i said it thrashes the network it consumes so much data we just we just use two gigs of data to run that one test i mean it's just ridiculous what you have to do in order to measure capacity in the old -fashioned way and so what we do is as you can see something here called check score we run speed checks on a scheduled base And these are just 10 megabyte transfers in each direction to give you a baseline of how are things going.

10:54And you can easily see if you have a problem, if something changes based on that baseline. And so the history, here I'll do a couple speed checks. It'll add another 10 meg on each side of the up and down in the calculations here. And then I'm going to shift into what I call, ooh, I had a little interesting impact on the local network. That's the other point I was going to make. is you can measure from any orb to any orb. We also measure to local router. That is an ICMP measurement to the local router, which is why I think it's not very happy with me and not behaving, but data flowing through it is still working. But now let's shift into here. This is NerdView, as we call it. Everything imaginable that we would have wanted to collect, also all available in the local API or in our cloud API.

11:41you can get to all this data, every little detail that we're collecting. We just want to make it easy and make it so you can build your own things and do your own stuff with it. Notice the router lag really got impacted, but even the internet lag during the speed test portion, I can highlight it by floating over this section here.

12:01It is so brutal to a network to shove enough data through it to measure capacity. So the two things we're trying to do is get people out of that mindset. You can see if something's wrong with your connection, just looking at lag, just looking at the responsiveness metric. If it's always running, if you notice something a little off with your connection, you want to go see it, just go back and look at it. You actually have the data collected already without having to go take a speed test. Again, we always still give you that capability, but wow, look at all the, yeah, the router here is, or the Wi-Fi access. or router at the other end is basically not very, it's deprioritizing ICMP like crazy and probably is not even wanting to respond to us. That's why we see that there.

12:47But these little two highlights here, my last point before I hand this over to Laie, there's a couple of speed checks that went on here and they have almost no impact to the connection. It's like we can run them all the time. We run them on a scheduled basis, again, to Cloudflare to give you that baseline. something going on that's how we score your connection and uh with that all said i'm gonna hand this over to lai and i can come back to questions later but uh yeah orb .net check it out and i'm not here to sell it's all free so thank you oh let's go to the next one yeah

13:28okay there we go so doug mentioned a couple times orb uses cloud players we'll talk a bit about talking about oh my god the end of the world okay um so how does cloudflare play into this but also just how uh you know doug also talked about sort of how or ukla informed the journey at orb and i'm going to talk about how mlab and just speed test measurement in general has informed how i've developed products at cloudflare so you know the definition of speed is actually a really hard thing to define doug talked about how you know i certainly think that the use of it has been largely driven by the marketing sort of potential from it and the way that it's caught on. It's really intuitive. Also, it makes sense, right? Like when you're trying to load a page, speed is what comes to mind.

14:14But that use of that term, I also attribute to that the actual act of figuring out how to measure network quality is actually a really difficult problem to solve. So, you know, when you're thinking about how to define network quality, and define a speed test or define any sort of internet measurement. A lot of what Doug talked through and what Orb is thinking about is the actual methodology. So how are you pulling the metrics? How are you calculating from them? How are you aggregating them? What are you showing the user? And I think of this as a measurement metrics themselves. But there's also the question of where are the vantage points? So when you're measuring end user quality, that's typically pretty easy. You're going to, on the client side, you're going to be measuring from a mobile device, you know, a desktop, a laptop. but where is it measuring to this gets into a really existential question when you really think about it which is what is the internet i want to measure my internet connection where is that what do i use to represent that so um you know various speed tests make different choices about how to answer this uh ukla runs you know an infrastructure largely uh either in or close to access networks meaning the isps that you and i subscribe to measurement lab answered this by putting uh server in interconnection points.

15:26And the idea there is to measure the degree to which peering impacts the end user performance. But the point being that no matter what, as a speed test provider or anyone sort of devising a speed test measurement, you have to think about what it is you're measuring to and how representative that is of the user experience. So this is something that I thought all about a lot at Measurement Lab. Again, we measured from the interconnection point, which was a really informative measurement for what we were trying. to measure. But when thinking about what is the most representative of end user performance, and I'm not just saying this, even before I joined, I would often think about measuring to CDNs and specifically Cloudflare. You know, Cloudflare represents 20 % of the internet or facilitates 20 % of the world's traffic. So in many ways, connecting to Cloudflare, you can think of as a proxy for connecting to the internet, or at least that's my premise or proposition.

16:15So when I'm thinking about what is the most. representative way to measure end user connectivity measuring to Cloudflare a lot of the time that is what you're actually accessing as a user so measuring to Cloudflare can be a proxy for that experience of connecting to the internet and so this is the approach that speed .cloudflare takes and you know when I go to speed .cloudflare .com we can run a test you are just connecting essentially to the closest Cloudflare pop our anycast network and um you know we run uh the typical speed test measurements our particular methodology um is running uh basically increasingly uh larger sizes of data ah you have the reverse scroll and that really messes me up um and so uh we also then use latency measurements so idle and loaded latency the difference being are you um actively doing something with your connection or not so you know the latency can really differ depending on what your connection is trying to do so you measure both jitter etc um this speed test itself uh is all done on the client side so it's all client side measurements this is the browser essentially doing these calculations and this is open source and will remain so this is uh our open source speed test that you can look at on um on github look at the calculations and then the last bit that i'll mention too is that we have a network quality score that basically translates these metrics into use cases.

17:46So how does my connection work for streaming? How well does it work for gaming and, you know, and video calls, which are things that people are typically trying to understand when they take a speed test. So this is all, again, open source and going to continue to be. But the thing I want to point out is going back to that question of, you know, what is the internet and what are we going to use to represent it? The way that speed .cloudflare is it tests again to the Cloudflare network. And so over the years, this was something that I really wished I could do as another speed test provider. And what I'm excited to share today is that we're making that possible.

18:31So introducing Cloudflare's network quality API. So we're introducing this today in private beta. And what essentially that means is that any speed test can now use Cloudflare. entire network to facilitate their test on the server side. So again, legally obligated to show this slide in a Cloudflare presentation. But the point being that you get access to this entire global network when you're using any Cloudflare product, but specifically the network quality API. So when I am wanting to create a new speed test or use someone else, but run on devices and i have then the question of okay but where am i going to test to now cloud player is an option um so we'll talk a bit about how it works and just the features um you know one of the core features is that you can measure network quality any way you want so orb has a solution cloud player has our solution but the something that i always come back to in internet measurement work is i think the more methodologies the better of course you want uh to be able to compare over time and, you know, between different data collections.

19:39So, you know, longitudinal data is helpful for that. Standardization is helpful for that. But I do think that when you're thinking about how to measure network quality, having different measurements from different ways of thinking about from different definitions can be really useful. So the premise and what we're trying to create here, again, largely based off of my previous experience, is thinking about, you know, when you have. speed test or when you want to use a speed testing or networking network quality product often you will have to use both their client and their server this is separating the two and meaning that you can test to cloud blur as your server but test in any way that you want so you could also so you can test in any way that you want and you can also test in multiple ways and all of this would just use our api to facilitate the server side of the measurements usage tracking so we make it really easy to I don't walk over here we can track in Dash so you can track your API or your you know API usage directly in Dash alongside all of your other Cloudflare products and you know you can look at the number of bytes transferred so you know in the orb case looking at all of the different orb devices and how much they're transferring data you know egress and ingress and then also where is the traffic coming from and how many requests all of this makes it really easy essentially to you know facilitate this kind of measurement at scale which is the other sort of core feature you know again if I today

21:14just wanted to go ahead and set up my own speed testing platform I would have to make that choice again about where I'm measuring to but even say let's say I figure that out I would if not using something like this Cloudflare API I would have to maintain all that infrastructure myself. This is a really big challenge in the internet measurement community and academic research is, you know, you have a question, you have a thing you want to measure, but where are you going to measure to and how are you going to maintain all of those servers? So we're trying to make that as easy as possible. And again, doing this at scale. So, you know, being able to test around the entire world and all of these different locations, that's now just done for you instead of having to, you know, sort of build up a platform one by one.

21:59as I previously did.

22:04And then the last core feature I would say is just to, you know, really drive this home that when you are measuring to our, or using our API, you're essentially measuring to Cloudflare. And that's, I think, the biggest kind of feature is that you're able to measure to this entire representative, entire network that is representative of user experience. So how does it work? You know, fairly simple at the end of the day. really what you're doing is you're getting an API token and then with that token you will be able to give your users so you know in Orbs case so let's say that Doug's apartment in Chelsea wants to request a speed test then that will get a short -lived token that will then authorize that client or that device to test a Cloudflare and then using that it will measure the connection.

22:54Potential use we're thinking this will be really helpful for network quality monitoring services, but also potentially mesh network device providers. And then a lot of my background comes from measuring for digital inclusion and broadband mapping purposes. So, you know, trying to understand the landscape of who has proper connectivity and who doesn't. So I think this solution could be helpful for any of those use cases, but also really wondering who else. So if this is, you know, something that is of interest and you're not kind of in any of these categories, I would love to know about. that and you know just to say why do we care why is cloudflare interested in this why is this something that we're pursuing the agentic internet as we know i've been you know talking about is here and it only makes internet connectivity that much more important only more and more of our lives are becoming dependent on this critical infrastructure and so you know with that then we have to think about okay who are then we uh uh helping advance into that brave new world and who are we leaving behind and you know i i cite the FCC here because it's widely known to be a very optimistic measurement and is also neither one of our former employers.

23:57But according to the FCC, 10 .6 million Americans lack access to even 100 upload or 100 download, 20 upload. And so, you know, when marketing plans are pushing us to bigger and bigger plans, 1G, 5G, 10G, and there's still this critical mass of people in this country that are not able to access even the bare minimum to do the things that they need to. work remotely, go to school online, et cetera, then I think these types of measurements are that much more critical to understand where the gaps are and what it is to really have 100 % connectivity, not only in this country, but elsewhere. And so we want to make it possible to get those measurements and to understand the landscape of internet quality in all regions by supporting customers like Orb and other providers of these measurements so that we can truly measure network quality meaningfully at scale.

24:50So just a preview of what's on our API roadmap, we're launching today in private beta. And again, I would love to know if you're interested, I'll have my contact in a second, but what's coming down the roadmap is being able to access finer grained server side metrics. So right now what the API is primarily returning is just timing information, which is more or less what you need to run a network quality test. But I'm thinking about what are the, metrics on the lower level kernel side. So TCP info, we could imagine congestion control metrics as well. And so giving more fine -grained access to that on the server side is something that we're interested in. Enable measurements of higher throughputs, because even though we said everything we just said, people still will always ask, you know, can I measure up to 1G? Can I measure up to 10G?

25:38And we want to be able to say yes, talk about all the things we all really align. on that that's not the most important thing to measure, but we want to ensure that Cloudflare is never the bottleneck. And then finally, we're interested also in being able to analyze and monitor your users' results inside of Dash. So basically creating an interface that's as informative to you as possible about all of your different users and the people using your measurements without kind of having to go outside of the Dash interface. So with that, thank you so