Skip to main content

The Shift to S3 - Every Database Will Be Rearchitected for S3 in the Next Few Years

· 7 min read
Wei Cao
Wei Cao
Founder at ApeCloud

In recent years, we’ve seen a profound shift in how databases handle storage, with more and more systems transitioning to S3 (or S3-compatible object storage) as their primary storage backend. This trend is not isolated to any single type of database but is happening across the spectrum— from OLAP data warehouses, to streaming systems, to OLTP databases, and even embedded databases. It’s becoming increasingly clear that object storage will be the future of database architecture, and in the next two to three years, we can expect nearly every database to be rearchitected to utilize S3 or similar services.

Early Adopters: OLAP Data Warehouses​

The shift to S3 started in the OLAP (Online Analytical Processing) domain. Object storage, with its unlimited scalability, pay-as-you-go pricing, and high read&write bandwidth, is a perfect fit for the I/O-heavy, large-scale data operations typical of OLAP systems.

Take Snowflake, for example. Snowflake was one of the earliest products to adopt S3 as its backend storage, revolutionizing how data warehouses could scale in the cloud. One of the biggest advantages Snowflake gained from adopting S3 was the ability to separate compute from storage. This separation enabled Snowflake to offer elastic scaling of compute resources, allowing users to independently scale compute nodes as needed. This architecture also supports features like virtual data warehouses, where multiple teams or workloads can operate on the same underlying data without resource contention. By leveraging S3, Snowflake not only achieved cost savings and reliability but also unlocked powerful cloud-native capabilities.

Another player, Rockset also embraced S3 for its data storage, achieving compute-storage separation. It uses RocksDB as its storage engine, storing data on S3 to provide data durability, while leveraging a tiered storage architecture to offer better cost efficiency.

Beyond OLAP: Streaming Systems and Kafka Alternatives​

The shift to S3 is not limited to OLAP databases. Even streaming systems are exploring object storage as a primary backend. Kafka was once challenged by WarpStream, a Kafka alternative that leveraged S3 for handling streams of data. WarpStream claimed a No Disk architecture, persisting all data directly to S3 to provide durability and scalability in ways that traditional Kafka architectures—relying on local disks or expensive block storage—could not match. Additionally, brokers across multiple Availability Zones (AZs) could share the same data via S3, avoiding the costly cross-AZ replication traffic typically required for synchronizing WAL logs. WarpStream has since been acquired by Confluent, consolidating its innovations into the broader Kafka ecosystem.

PostgreSQL and the Rise of S3 Storage Engines​

One of the most exciting developments is Neon, an open-source alternative to Aurora PostgreSQL, that brings compute-storage separation to PostgreSQL by leveraging S3 as the storage layer. Neon's Page Server converts the database's random writes into log-structured sequential writes stored on S3, transforming PostgreSQL into a serverless database with compute-storage separation, where compute nodes can be restarted, migrated, and recovered from failures extremely quickly.

But Neon is not alone. Another project, OrioleDB(acquired by Supabase), is also developing an experimental S3 storage engine for PostgreSQL, for increasing data safety, and for scaling and changing the architecture of compute instances preserving all data.

Embedded Databases: SQLite and DuckDB Embrace S3​

Even embedded databases are not immune to the growing influence of S3. Cloudflare, for instance, has completely replaced the persistent layer of its SQLite storage backend for DO (Durable Objects) with R2, Cloudflare’s S3-compatible object storage service. This move highlights that even lightweight, embedded databases can benefit by using a local disk as a cache on top of durable, cheap object storage, combining fast access with the reliability of S3-like storage.

Meanwhile, MotherDuck is using Differential Storage to store DuckDB data on S3. DuckDB is a high-performance embedded analytical database, often described as SQLite for OLAP workloads. By shifting the storage layer to S3, MotherDuck ensures that DuckDB can take on the role of a central data warehouse, scaling efficiently and handling large datasets without being constrained by local storage limits.

Enterprise Databases: DB2 Integrates with S3​

The trend toward S3 is not limited to modern, open-source databases. Even traditional enterprise databases are being rearchitected to take advantage of object storage. A recent paper presented at VLDB 2024 revealed that IBM DB2 Warehouse is undergoing a significant transformation. IBM is replacing DB2’s traditional storage engine with RocksDB, an LSM-based storage engine, and moving data to S3. This change allows DB2 to handle the massive scale of contemporary data workloads, benefiting from object storage not only for cost efficiency but also for improved performance.

Why S3?​

So, why are so many databases moving to S3 or S3-compatible storage systems?

Some of the advantages come from S3 itself, which the database can benefit from without requiring any modifications:

  • Scalability: S3 offers virtually unlimited storage capacity, allowing databases to grow without the headaches of managing and provisioning storage manually.
  • Cost Efficiency: S3’s pay-as-you-go pricing model makes it much more affordable than traditional block storage, especially for databases with large datasets or archival needs. It eliminates the need for expensive, pre-provisioned disk space.
  • Durability and Reliability: S3 guarantees 99.999999999% (11 nines) of durability, which means data is incredibly safe from loss. This level of reliability is difficult to achieve with traditional storage systems without significant overhead.
  • High Bandwidth: S3 provides high write bandwidth, ensuring that database I/O operations, such as flushing dirty pages, no longer become bottlenecks. Additionally, S3's high read bandwidth allows databases to load data quickly during startup and enables concurrent scanning of multiple files during queries, reducing latency.
  • Disaster Recovery and Multi-AZ Support: S3’s architecture inherently supports database replication across different Availability Zones (AZs), providing robust disaster recovery options out of the box.

Other reasons are that databases, through modifications, can gain functional and elasticity advantages:

  • Separation of Compute and Storage: By moving data to S3, databases can separate compute resources from storage resources. Compute resources can scale independently, and data (such as partitions or cache hotspots) can be rebalanced between nodes much faster without the need to physically move the data.
  • Shared Data Access: An S3 bucket can be accessed simultaneously by multiple virtual machines. This allows databases to support advanced functionalities such as shared access to data files and the creation of instant, low-cost clones.

The Future: Every Database Will Use S3​

As we look ahead, it’s clear that the trend of using S3 for database storage is only accelerating. From OLAP systems like Snowflake and Rockset, to streaming platforms like WarpStream, to relational databases like PostgreSQL (with Neon and OrioleDB), and even embedded databases with DuckDB and SQLite, the entire database ecosystem is moving towards object storage.

The flexibility, scalability, and cost advantages of S3 make it an irresistible choice for modern databases. As more and more projects adopt S3 as their storage backend, it’s only a matter of time before every database—from enterprise systems like DB2 to lightweight embedded databases— will be rearchitected to use S3 or S3-compatible storage solutions.

In the next two to three years, this will no longer be a trend—it will be the norm.

What We Did​

WeSQL, to the best of our knowledge, is the first open-source, S3-based database built within the MySQL ecosystem. We replaced InnoDB, the traditional B+ tree-based storage engine, with SmartEngine, an LSM-tree-based storage engine. To overcome S3's write latency, we implemented Raft replication, ensuring that transaction latency is decoupled from S3 write delays. Additionally, we addressed read latency by introducing a multi-tier caching system.

References​