Optimizing LSM-Tree Key-Value Stores with Adaptive Lower-level Driven Compaction

PROJECT TITLE :

Adaptive Lower-level Driven Compaction to Optimize LSM-Tree Key-Value Stores

ABSTRACT:

Log-structured merge (LSM) tree key-value stores have been widely implemented in many NoSQL and SQL systems. These stores are used to serve online Big Data applications such as social NetWorking, graph processing, Machine Learning, and many more. In LSM-tree key-value stores, the batch processing of sorted data merging (also known as compaction) improves write efficiency. Additionally, some lazy compaction methods have been proposed to accumulate more data within a batch. These methods of batch writing, on the other hand, result in a significant amount of tail latency, which is unacceptable for online processing. We propose a novel method called Lower-level Driven Compaction (LDC), with the goal of optimizing both latency and throughput. This method breaks the limitations of the traditional upper-level driven compaction manner and triggers practical compaction actions bottom-up. This has the benefits of both decreasing the compaction granularity for smaller latency and reducing write amplification for higher throughput. In addition, this method breaks the limitations of the traditional upper-level driven compaction manner. In addition, we extend LDC to Adaptive LDC (ALDC) by including an adaptive policy that can adjust the key compaction threshold to fit the changes in the characteristics of the workloads. This allows LDC to become more flexible. According to the findings of the experiments, ALDC is capable of significantly lowering the tail latency while simultaneously achieving a significantly higher and more stable throughput than other approaches currently in use.

Did you like this research project?

To get this research project Guidelines, Training and Code... Click Here