People forget AWS is 20 years old. When I routinely remind people of that fact they often remark (incredulously) with "WHAT? Really?" When I think about it the entire paradigm of "regions" itself seems completely antiquated in the grand scheme of things. AWS has made a few moves on this but in the end you're largely still tied to this fundamental regional concept in terms of control planes, etc and you incur the ridiculousness that is VPCs, wild billing and availability issues, all kinds of hacks, stacks, and effort all over the place to have multiple regions/AZs, etc.
I suspect that later starters like Fly, Cloudflare, etc with their approaches of (more or less) compute at edge with no concept of region etc will grow more and more to dominate the cloud space with AWS region-based approaches primarily relegated to "legacy applications" that are stuck there. It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.
When I think about it the region based approaches are more akin to "hosted elastic datacenter" than they are what could be described as a "true cloud" - where it's just everywhere and not even a consideration.
The realities of regions don’t go away. It’s a primitive to save you money. If a new entrant doesn’t have it they are either just operating in one region or you’re paying cross-region prices for everything.
Cloudflare products and pricing (what I'm most familiar with) are in wild opposition to this view.
I've never seen anything even remotely hinting at region or geography in their product line other than geo-routing for load balancing products, headers with geo info for you to do something with, etc. They include the serving "POP" in headers for diagnostic purposes but other than that you have no idea.
Where do my Workers run? Don't know, don't care. They are substantially cheaper and offer better availability and response times than their region-based "cloud" competitors. Same for KV, D1, R2, and anything else they come up with as they move further and further in (out?).
One would expect with them having the ability to allocate supporting hardware dynamically and globally their cost basis is substantially better than having customer facing and controlled region-based resources that still need to be built out for product support (regardless of usage) AND maintain the excess capacity local to each region to be anywhere near "elastic".
It's important to qualify this though. E.g. Amazon was not profitable for a very long time, but only because they furiously reinvested everything. They could've turned profitable at any time.
Assuming Cloudflare is a similar case to Amazon: ie they could turn profitable at any time but choose to reinvest those profits immediately into more business, that is a very good spot to be in.
It really doesn't though. Sure, it would if the difference is only "16 mega-regions (regional-model) versus 300 mini-regions (edge)". But that's not the difference.
The actual difference, at least how Cloudflare pushes it, is in the software; its a higher layer of abstraction; that customers get to stop thinking about regions and locations. Your software is global; its deployed everywhere; it runs everywhere.
This is ENTIRELY antithetical to everything on AWS except CloudFront, Route53, IAM(ish), maybe a couple other minor services. Every Single Thing Every Single AWS Customer configures beyond those global services forces you into a region. Want to deploy a lambda function? Which region would you like? Pricing and available products is different in every region. Fronting it with an API? Where would you like that API Gateway? We'll put CloudFront in front of it so its a bit faster for ya, its global(ish), don't worry.
In a no-true-scottsman true-edge world, this goes out the window. Cloudflare Workers is like this; D1 is kinda-ish-there-ish; R2 isn't. Here's the code; spider it across the world; automatically route clients to your closest PoP. No regions. No locations.
Its not obvious that AWS will ever make meaningful moves to a world like this, for a ton of reasons, but most meaningfully: it requires customer demand; but AWS's existing customers aren't demanding it, because AWS's customers are old CTOs worried about past quarters too much to think about the next one. The younger, smarter, potential customers who recognize how valuable this is aren't AWS's customers; so AWS never hears from them (which, by the way, should more generally concern Amazon's investors; their "relentless customer driven" approach to their products is great for a few decades, but its absolutely true that most of the time customers just want a faster horse, and you're not hearing from the people who aren't your customers because you've never solved their problems before). So, Amazon has dumped billions upon billions into their couple-dozen giga-regions; you cannot pivot that into hundreds upon hundreds of mini-regions like what Cloudflare (and to a lesser but still awesome degree Fly) has.
Vercel is the best case study on this. They are exploding in popularity. They don't run any of their own infra afaik, to be clear, but they whitelabel products from Cloudflare, Neon, and others; all Edge infra providers. No one on Vercel is cross-shopping AWS, and AWS has nothing that feels competitive to what Vercel offers. They can't compete.
The outcome, in maybe 10-20 years, of this landscape feels really obvious right now, but AWS's size and the insane scale of their investment into "how we did things in 2010" will turn them into the next IBM. Some would say they already are. There's nothing they can do about it. They're not making the right investments. Even if they wanted to, they can't at the necessary scale, because what they're doing is working right now and their existing customers keep forcing them to spend more money building more structures, more racks, more blades, all in Ashburn VA.
Stateless compute is relatively simple to be region-less, but even fly’s homepage has a map of regions.
When state is introduced, then CAP/PACELC distributed systems issues arise, and figuring out approaches to deal with them are a must. Read fly’s Postgres docs, and regions come right back in.
Only the most trivial business systems can operate without any state.
If your application has any sort of shared, mutable state, you absolutely need to pick a region, or go really deep into the EC/consensus/clustering rabbit hole. With a dedicated region with 1 big DB, you can achieve latency figures (over the shared state) that would be infeasible with other schemes. Serializing transactions in a distributed manner is a circus compared to what Postgres has to do to achieve the same.
For better or worse, sticking a big SQL database in exactly one region will almost always be the best path for most forms of business. If latency/edge are still a concern at this point, that's when we maybe start talking about more specific geographies and standing up read replicas in those regions. In my experience, if you are peddling B2B webapps, no one will ever complain about east vs west coast latency unless you actually screwed something up in code.
This is Fly - a much earlier stage company and them receiving early funding is the genesis of this entire discussion.
I point to Cloudflare (what I'm most familiar with). Across their entire stack of compute, storage, db, etc product lines you won't see a hint of region anywhere. It just works, everywhere. Same with Fastly/Oracle (although they like Fly are quite a ways behind Cloudflare).
I suspect (hope) that as Fly matures their apparent true goal of edge and region-less applications will mature more. Their compromises on region awareness for DB has always struck me as a more of a resource restraint compromise on their part than end goal. As you say these challenges are hard (expensive).
If I were at Cloudflare I'd suggest calling these "geographies" where the magic still applies (any one of those geographies has at least dozens of component POPs that are invisible).
The fact remains this is somewhat down in the docs because it is automatic and invisible to the point it works well enough where most developers don't know, don't care, and aren't exposed to it in any meaningful way. Contrast this with AWS where the first thing you are presented with is picking a region where everything lives, while not clearly and explicitly explaining that many things are ultimately dependent on us-east. Cue the shock and confusion when us-east has yet another issue and those in all other regions don't understand how and why they are impacted.
You are correct about the free lunch, information theory, and physics at the end. I don't think a reasonable person can think when you write data to a pop in Tokyo that it could possibly show up in Chicago instantly. That is fundamentally impossible and always will be. I think CF does a pretty good job making it "magic" for most devs and use cases while explaining these fundamental realities and limitations to those who need to know (potentially) for whatever their application is.
"Won't see a hint of region anywhere" is an exaggeration.
Here's the Durable Objects documentation [1]. Don't those "location hints" look like regions?
Also, here's the "Data location" page for Cloudflare D1 [2]. See "available hints" at the bottom.
I think it would be more accurate to say that Cloudflare's architecture is designed around having lots of regions, automatic migrations between them, and no guarantee about which one you'll get. But they're still there.
These are offered more as an escape hatch in case the system doesn't do what you want automatically. But the goal is definitely for the system to do what you want automatically, so that no one ever has to provide these hints.
I actually argued against adding these hints at all, since it confuses the messaging and most people don't need them... but they were easy to add and solved an immediate problem for a few customers, so we did.
In other cases we've held firm. For example, some people have wanted to restrict their Workers to run in specific regions because the Worker made lots of requests to a specific back-end in that region. Instead of actually offering explicit hints, we built a system to detect this automatically: https://blog.cloudflare.com/announcing-workers-smart-placeme...
In any case, most people don't have to think about regions on Workers, and for the few that do, our goal is to change that.
Thank you for the context and this is what I was getting at - if you really want to dig in on things like KV, D1 and DO you will see occasional mentions of regions (I'd really call it geography) to add context on things like "KV is eventually consistent, instant in the pop where the write takes place but takes X ms to propagate everywhere". For many (most?) applications it doesn't really matter, I get the sense for these you clearly document these things so people understand the more-or-less inescapable fundamental reasons why things are what they are. As other commenters have noted "there is no free lunch" when it comes to things like the physics of getting data from a pop in Tokyo to Chicago but Cloudflare does a good job and making it happen more-or-less invisible.
> In other cases we've held firm. For example, some people have wanted to restrict their Workers to run in specific regions because the Worker made lots of requests to a specific back-end in that region. Instead of actually offering explicit hints, we built a system to detect this automatically: https://blog.cloudflare.com/announcing-workers-smart-placeme...
That is an excellent way to design a product. Great job.
The question of whether "the edge" takes off will come down to whether or not the culture at large will swallow the illusion at its heart. You cannot remove the concept of a region any more than you can remove the concept of "the computer" in a cloud environment.
> any more than you can remove the concept of "the computer" in a cloud environment
But that's just not true. Tons of cloud services completely abstract the computer out of the picture. You are paying for capacity or throughput as an abstract unit of cost and the cloud configures as many computers as needed to run your request. You never interact with anything resembling a computer in this situation.
Except, you always interact with computerS no matter what provider or architecture you use. It's inescapable as long as you're doing business in software. The trade-offs in architecture and regionality do NOT go away just because a company abstracts over it. They may be able to set up read replicas and consensus on the fly and be able to detect when various approaches make the most sense, but you are still at the mercy of the accompanying constraints. For most use cases, none of this may matter. When you do run into a use case where it matters, you should be fully aware of what trade-offs you're making and have control over them.
Not to mention, regions matter when you're serious. GDPR/DSA/DMA/India/China -- the list goes on. Certain data must live in certain bubbles.
> It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.
I'm currently at a startup that started on Aptible a few years ago because there was a lack of expertise in systems development. We're now in the process of migrating to AWS natively because of cost savings (it's why I was hired). Aptible is costing us 3-4x what native AWS will. Yes, we're incurring some operational overhead, but the automation and tooling for managing infra is reasonably mature (Terragrunt & Terraform aren't perfect, but they work well enough).
Ultimately, startups may start with higher-level cloud providers, but they'll eventually migrate to lower-level providers eventually. At the higher end of the growth curve, companies likely wind up getting back to on-prem in some way.
AWS does have local zones and has Lambda at Edge. I will be the first to admit though that the local zones are tied to regional zones and that some global services are tied to us-east-1 like IAM.
These days, "edge" just means "has more than N data centers" (where N depends on the speaker). Fly and Cloudflare have regions too, try `flyctl platform regions` or see https://www.cloudflare.com/network/
No. That’s wrong. We don’t “manage regions”. 300 data centers. Everything running everything. Only time you’d need to think of a region is if you have some compliance situation (“keep all my data in North America” or “only terminate TLS in the EU”) and ask to subdivide our network using Regional Services. But there’s no decision on here you say “I’ll use Cloudflare-EU-North-42” and get stuck with it as no such things exist.
THIS. No one cares about regions, you just care that the app is fast and globally available. The way we did things is a relic of the past and everyone mimics it because trying to guide new user behaviour is hard. The reality is we just need the app to be globally deployed with a globally distributed database and anycast DNS to route you to the local most DC. There's a lot more to it, have built this architecture more than a couple times, but it all needs to be in the service of something and if you're UX isn't better than the next guy, it won't matter much.
As a counterpoint, anyone working with compliance data needs to know and be certain that data does not cross geo-boundaries. Intelligence data for US gov’t agencies must typically be kept stateside: AWS uses the GovCloud region specifically for satisfying gov compliance requirements.
Passing HIPAA data through other countries when not needed to is generally frowned upon - data should be kept as local as reasonably possible.
I haven’t worked with GDPR requirements but I have heard they are similar.
AWS solved this problem well with regions. I’m curious what other providers that are “regionless” do to solve the compliance problem.
Cloudflare does this although to my knowledge it's not available without an Enterprise subscription which is the only thing they offer with this level of compliance anyway.
All lower plans maybe get PCI DSS and that's it.
It's a valid counter to my point - while they clearly can do it locking it behind an enterprise "talk to us" subscription is lame. That said they are generally customer, market, and product savvy and I suspect they know that any/most orgs that are able to pull off the broader process, etc for actual compliance with these requirements are in fundamentally different positions than some random startup with a bunch of consumer data.
We're also seeing more and more ambiguity with coaching platform startups, etc who don't have a covered entity in sight but that's a different topic for a different day.
I've spent most of my career in HIPAA-relevant startups. Generally, I think it's actually a disservice all around for Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant". That's not even what it's supposed to mean, let alone the virtually endless issues and process that need to happen throughout the org for "compliant" to actually mean anything. I haven't reviewed it but I'm certain the AWS BAA is so specific and exclusionary it's trivial to step outside of what it covers.
Yes these agreements are a piece of the puzzle but it's completely reckless to throw something together in AWS, present the BAA to a customer, and call yourselves "HIPAA compliant". Savvy customers, investors, etc will call you out but otherwise it's mostly just a matter of time before you discover your standalone AWS BAA is actually just one of the things on a 50 point (or whatever) checklist.
> Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant".
N-2 companies ago I was a tech lead in 2015 where we were migrating to AWS. AWS was constantly explaining which underlying services were HIPAA compliant and what you build on top of it was your responsibility. I had to answer to separate non AWS compliance folks about my architecture.
AWS also gave guidance. But made no promises that your implementation was compliant. That’s where the entire “shared responsibility model” comes in.
Now, I work at AWS in ProServe. I am very familiar with our messaging regarding HIPAA compliance. We are very careful not to rubber stamp things as compliant. I know the best practices in and out. But when I’m asked if something is compliant, I say this is the guidance we are given. I’ll do a presentation before an Architecture Review board. But I would never make anything that hints as a guarantee.
How large were your employers? If you have Enterprise support, you definitely get one or more TAMs dedicated to helping your company.
If you are a smaller organization, you may be able to request an SA.
My personal experience is from when companies pay for consulting hours from ProServe.
At the n-2 company I referred to above, the only thing we got from AWS is a BAA and we had an outside consulting company and compliance auditors. I was just learning AWS then.
Compliance is a completely different conversation.
In the Cloudflare Workers platform, we have a number of features to control "jurisdictions" where compute runs or where data is stored. These are meant for compliance, not for performance. Our goal is that no one has to think about regions for performance reasons, but certainly compliance issues are not going to go away -- in fact, they are getting more onerous over time. So features to control jurisdiction are likely to expand.
This is MAY be true consumer applications and even then not all.
But it is rarely true for enterprise customers.
They want the guarantees their data never leaves certain regions even in a request along with other security and compliance guarantees.
I suspect that later starters like Fly, Cloudflare, etc with their approaches of (more or less) compute at edge with no concept of region etc will grow more and more to dominate the cloud space with AWS region-based approaches primarily relegated to "legacy applications" that are stuck there. It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.
When I think about it the region based approaches are more akin to "hosted elastic datacenter" than they are what could be described as a "true cloud" - where it's just everywhere and not even a consideration.