Insightful piece from Marc Brooker on Aurora DSQL, which was announced at AWS re:invent this week. DSQL stands for “distributed sql”. The idea is to get ACID semantics at gigantic scale with Postgres compatibility (psql works with Aurora DSQL as a backend):

We built a team to go do something audacious: build a new distributed database system, with SQL and ACID, global active-active, scalability both up and down (with independent scaling of compute, reads, writes, and storage), PostgreSQL compatibility, and a serverless operational model.

To get there, a few key design choices have been made, firstly by thinking carefully about cross-region behaviours:

Latency scales with the number of statements in a transaction. For cross-region active-active, latency is all about round-trip times. Even if you’re 20ms away from the quorum of regions, making a round trip (such as to a lock server) on every statement really hurts latency. In DSQL local in-region reads are as fast as 1.2ms, so 20ms on top of that would really hurt.

From the beginning, we took avoiding this as a key design goal for our transaction protocol, and have achieved our goals. In Aurora DSQL, you only incur additional cross-region latency on COMMIT, not for each individual SELECT, UPDATE, or INSERT in your transaction (from any of the endpoints in an active-active setup). That’s important, because even in the relatively simple world of OLTP, having 10s or even 100s of statements in a transaction is common. It’s only when you COMMIT (and then only when you COMMIT a read-write transaction) that you incur cross-region latency. Read-only transactions, and read-only autocommit SELECTs are always in-region and fast (and strongly consistent and isolated).

In designing DSQL, we wanted to make sure that developers can take advantage of the full power of transactions, and the full power of SQL.

And then secondly by making a distinction about consistency within region versus cross region:

Having observed teams at Amazon build systems for over twenty years, we’ve found that application programmers find dealing with eventual consistency difficult, and exposing eventual consistency by default leads to application bugs. Eventual consistency absolutely does have its place in distributed systems8, but strong consistency is a good default. We’ve designed DSQL for strongly consistent in-region (and in-AZ) reads, giving many applications strong consistency with few trade-offs.

Snapshot isolation is also used by default. Snapshot isolation is a concurrency control mechanism that means each transaction can operate on a “consistent snapshot” of the database taken at the point the transaction starts. This means you’re isolated from changes made by other transactions so you have a consistent view of the data:

We’ve also picked snapshot isolation by default. We believe that snapshot isolation is, in distributed databases, a sweet spot that offers both a high level of isolation and few performance surprises. Again, our goal here is to simplify the lives of operators and application programmers. Higher isolation levels push a lot of performance tuning complexity onto the application programmer, and lower levels tend to be hard to reason about.

The piece is great end to end, so well worth a read. The footnotes are also brilliant - a tonne of useful links to papers with underlying theory. I’ll finish with one last pull quote which I thought was interesting and highlight the team’s use of formal methods - you don’t see formal methods pop up often so it’s worth calling out:

The fourth was AWS’s strong formal methods and automated reasoning tool set. Formal methods allow us to explore the space of design and implementation choices quickly, and also helps us build reliable and dependable distributed system implementations6. Distributed databases, and especially fast distributed transactions, are a famously hard design problem, with tons of interesting trade-offs, lots of subtle traps, and a need for a strong correctness argument. Formal methods allowed us to move faster and think bigger about what we wanted to build.