Skip to main content

Storage Layout

Each WeSQL database instance persists its data in S3. This article introduces the directory structure and file organization.

Overall, WeSQL's database persisted in S3 is divided into three main types:

  • Binlog: The transaction log that captures all changes made to the database, supporting replication and recovery.
  • Metadata: This includes InnoDB's system tables, SmartEngine’s MetaSnapshot, and Manifest log. These components store essential information regarding the schema (e.g., database, table, index) and metadata of the LSM tree structure.
  • Data: A collection of SmartEngine Extents, which store the actual data in the system, along with SmartEngine's WAL. The WAL contains data not yet written to LSM Level 0, and is used to recover the MemTable during database recovery.

The directory structure is as follows: each instance corresponds to a directory under a S3 bucket. The objects within the directory include the following categories:

  • Multiple InnoDB snapshot files.
  • One index that lists all InnoDB snapshot files.
  • Multiple binlog files, each storing a segment of incrementally uploaded binlog files.
  • One index that lists all binlog files.
  • SmartEngine data, where each object stores an LSM extent.
  • One snapshot index that stores all snapshots. A snapshot represents a complete snapshot of the database. Each snapshot contains the corresponding InnoDB snapshot file object names, binlog file names, and the list of SmartEngine LSM extent objects.

alt text