ARKDB: The Key-Value Engine for Alibaba Cloud Storage Services
Alibaba Cloud offers a comprehensive set of storage services, including Object Storage Service (OSS), File Storage Service (NAS) and NoSQL Tablestore with high durability, high availability, high scalability and strong consistency. All these services are built with a layered architecture, including a scalable index layer built with ArkDB key-value engine, and a persistent storage layer provided by Pangu distributed storage system. In this talk, we present the detailed design and implementation of ArkDB, addressing challenges in storage disaggregation, supporting write and range query-heavy workloads, and balancing of scalability, availability and resource usage. ArkDB combines some of key advantages from both LSM tree and recently proposed B-tree variants e.g., Bw-tree and Bε -tree. Its main contributions include shrinkable page mapping table, separation of system and user states for fast recovery, write amplification reduction, efficient garbage collection and lightweight partition split and merge. We share some of our lessons and experiences in building ArkDB key-value engine and demonstrate its performance improvements with experimental results.