In recent years, we’ve seen a profound shift in how databases handle storage, with more and more systems transitioning to S3 (or S3-compatible object storage) as their primary storage backend. This trend is not isolated to any single type of database but is happening across the spectrum— from OLAP data warehouses, to streaming systems, to OLTP databases, and even embedded databases. It’s becoming increasingly clear that object storage will be the future of database architecture, and in the next two to three years, we can expect nearly every database to be rearchitected to utilize S3 or similar services.

Early Adopters: OLAP Data Warehouses​

The shift to S3 started in the OLAP (Online Analytical Processing) domain. Object storage, with its unlimited scalability, pay-as-you-go pricing, and high read&write bandwidth, is a perfect fit for the I/O-heavy, large-scale data operations typical of OLAP systems.

Take Snowflake, for example. Snowflake was one of the earliest products to adopt S3 as its backend storage, revolutionizing how data warehouses could scale in the cloud. One of the biggest advantages Snowflake gained from adopting S3 was the ability to separate compute from storage. This separation enabled Snowflake to offer elastic scaling of compute resources, allowing users to independently scale compute nodes as needed. This architecture also supports features like virtual data warehouses, where multiple teams or workloads can operate on the same underlying data without resource contention. By leveraging S3, Snowflake not only achieved cost savings and reliability but also unlocked powerful cloud-native capabilities.

Another player, Rockset also embraced S3 for its data storage, achieving compute-storage separation. It uses RocksDB as its storage engine, storing data on S3 to provide data durability, while leveraging a tiered storage architecture to offer better cost efficiency.

Beyond OLAP: Streaming Systems and Kafka Alternatives​

The shift to S3 is not limited to OLAP databases. Even streaming systems are exploring object storage as a primary backend. Kafka was once challenged by WarpStream, a Kafka alternative that leveraged S3 for handling streams of data. WarpStream claimed a No Disk architecture, persisting all data directly to S3 to provide durability and scalability in ways that traditional Kafka architectures—relying on local disks or expensive block storage—could not match. Additionally, brokers across multiple Availability Zones (AZs) could share the same data via S3, avoiding the costly cross-AZ replication traffic typically required for synchronizing WAL logs. WarpStream has since been acquired by Confluent, consolidating its innovations into the broader Kafka ecosystem.

PostgreSQL and the Rise of S3 Storage Engines​

One of the most exciting developments is Neon, an open-source alternative to Aurora PostgreSQL, that brings compute-storage separation to PostgreSQL by leveraging S3 as the storage layer. Neon's Page Server converts the database's random writes into log-structured sequential writes stored on S3, transforming PostgreSQL into a serverless database with compute-storage separation, where compute nodes can be restarted, migrated, and recovered from failures extremely quickly.

But Neon is not alone. Another project, OrioleDB(acquired by Supabase), is also developing an experimental S3 storage engine for PostgreSQL, for increasing data safety, and for scaling and changing the architecture of compute instances preserving all data.

Embedded Databases: SQLite and DuckDB Embrace S3​

Even embedded databases are not immune to the growing influence of S3. Cloudflare, for instance, has completely replaced the persistent layer of its SQLite storage backend for DO (Durable Objects) with R2, Cloudflare’s S3-compatible object storage service. This move highlights that even lightweight, embedded databases can benefit by using a local disk as a cache on top of durable, cheap object storage, combining fast access with the reliability of S3-like storage.

Meanwhile, MotherDuck is using Differential Storage to store DuckDB data on S3. DuckDB is a high-performance embedded analytical database, often described as SQLite for OLAP workloads. By shifting the storage layer to S3, MotherDuck ensures that DuckDB can take on the role of a central data warehouse, scaling efficiently and handling large datasets without being constrained by local storage limits.

Enterprise Databases: DB2 Integrates with S3​

The trend toward S3 is not limited to modern, open-source databases. Even traditional enterprise databases are being rearchitected to take advantage of object storage. A recent paper presented at VLDB 2024 revealed that IBM DB2 Warehouse is undergoing a significant transformation. IBM is replacing DB2’s traditional storage engine with RocksDB, an LSM-based storage engine, and moving data to S3. This change allows DB2 to handle the massive scale of contemporary data workloads, benefiting from object storage not only for cost efficiency but also for improved performance.

Why S3?​

So, why are so many databases moving to S3 or S3-compatible storage systems?

Some of the advantages come from S3 itself, which the database can benefit from without requiring any modifications:

• Scalability: S3 offers virtually unlimited storage capacity, allowing databases to grow without the headaches of managing and provisioning storage manually.
• Cost Efficiency: S3’s pay-as-you-go pricing model makes it much more affordable than traditional block storage, especially for databases with large datasets or archival needs. It eliminates the need for expensive, pre-provisioned disk space.
• Durability and Reliability: S3 guarantees 99.999999999% (11 nines) of durability, which means data is incredibly safe from loss. This level of reliability is difficult to achieve with traditional storage systems without significant overhead.
• High Bandwidth: S3 provides high write bandwidth, ensuring that database I/O operations, such as flushing dirty pages, no longer become bottlenecks. Additionally, S3's high read bandwidth allows databases to load data quickly during startup and enables concurrent scanning of multiple files during queries, reducing latency.
• Disaster Recovery and Multi-AZ Support: S3’s architecture inherently supports database replication across different Availability Zones (AZs), providing robust disaster recovery options out of the box.

Other reasons are that databases, through modifications, can gain functional and elasticity advantages:

• Separation of Compute and Storage: By moving data to S3, databases can separate compute resources from storage resources. Compute resources can scale independently, and data (such as partitions or cache hotspots) can be rebalanced between nodes much faster without the need to physically move the data.
• Shared Data Access: An S3 bucket can be accessed simultaneously by multiple virtual machines. This allows databases to support advanced functionalities such as shared access to data files and the creation of instant, low-cost clones.

The Future: Every Database Will Use S3​

As we look ahead, it’s clear that the trend of using S3 for database storage is only accelerating. From OLAP systems like Snowflake and Rockset, to streaming platforms like WarpStream, to relational databases like PostgreSQL (with Neon and OrioleDB), and even embedded databases with DuckDB and SQLite, the entire database ecosystem is moving towards object storage.

The flexibility, scalability, and cost advantages of S3 make it an irresistible choice for modern databases. As more and more projects adopt S3 as their storage backend, it’s only a matter of time before every database—from enterprise systems like DB2 to lightweight embedded databases— will be rearchitected to use S3 or S3-compatible storage solutions.

In the next two to three years, this will no longer be a trend—it will be the norm.

What We Did​

WeSQL, to the best of our knowledge, is the first open-source, S3-based database built within the MySQL ecosystem. We replaced InnoDB, the traditional B+ tree-based storage engine, with SmartEngine, an LSM-tree-based storage engine. To overcome S3's write latency, we implemented Raft replication, ensuring that transaction latency is decoupled from S3 write delays. Additionally, we addressed read latency by introducing a multi-tier caching system.

References​

• https://www.snowflake.com/resource/sigmod-2016-paper-snowflake-elastic-data-warehouse/
• https://rockset.com/blog/separate-compute-storage-rocksdb/
• https://neon.tech/blog/architecture-decisions-in-neon
• https://github.com/orioledb/orioledb/blob/main/doc/usage.md#s3-database-storage-experimental
• https://blog.cloudflare.com/sqlite-in-durable-objects/
• https://motherduck.com/blog/differential-storage-building-block-for-data-warehouse/
• Native Cloud Object Storage in Db2 Warehouse: Implementing a Fast and Cost-Efficient Cloud Storage Architecture

MySQL is a popular open-source database with a large user base. Building on top of MySQL, cloud providers like AWS have developed cloud-native MySQL services, such as Aurora MySQL, which adopt a compute-storage separation architecture. These cloud-native services offer features like serverless operation, rapid scaling, and fast snapshots. However, they are proprietary and vendor-locked, meaning Aurora can only be used on AWS. This creates challenges for multi-cloud or on-premise users who might use Aurora in AWS environments but are forced to revert to RDS MySQL or open-source MySQL binaries in others.

We believe that an open-source MySQL distribution with a compute-storage separation architecture, capable of running on any cloud—including on-premise environments—is essential. This belief led to the development of WeSQL.

Similar to Neon, an open-source alternative to Aurora PostgreSQL, WeSQL also chose S3 as the storage backend due to its scalability, pay-as-you-go pricing model, exceptional reliability, and high bandwidth. For a detailed analysis of why S3 was chosen, you can refer to this article. In addition, WeSQL also supports MinIO, a S3-compatible object storage system often used for on-premise deployments. Looking ahead, WeSQL plans to expand its compatibility with more S3-compatible object storage systems.

WeSQL retains an almost unmodified MySQL Server layer, replacing InnoDB with SmartEngine, a Log-Structured Merge tree (LSM) storage engine derived from LevelDB and RocksDB. The choice of LSM is driven by its superior compatibility with S3-based storage, as it optimally handles sequential writes and bulk operations, which are well-suited for object storage systems like S3 that don’t support random writes.

At present, WeSQL is based on MySQL 8.0.35. However, due to the minimal nature of our patches, WeSQL can be easily adapted to newer MySQL versions, including MySQL 8.4, 9.0, and future releases.

WeSQL is still in its early stages and evolving rapidly. We welcome your support and feedback as we continue to build and improve it. Thank you for your interest, and we hope you enjoy exploring what WeSQL has to offer!