Large-scale, high-performance computing — supercomputing — is essential to solving both complex and large questions. But storage platforms essential for these advanced computer systems have been stuck in a rigid framework that required users to either choose between customization of features or high availability.
A framework called BespoKV was developed that incorporates key value (KV) systems that store and retrieve important data from very fast memory-based storage instead of slower disks. These systems are increasingly used in high-performance applications that use distributed systems made up of many computers to solve a problem. High-performance computing relies on having computers intake, process, and analyze huge amounts of data at unprecedented speeds. Currently, the best systems operate at a quadrillion calculations per second, or a petaflop.
When continually accessing certain content, KV systems can be far more efficient as a storage medium because content loads from the faster in-memory store nearby, not the faraway storage server. This allows the system to provide very high performance in completing tasks or requests.
The main innovation of BespoKV is that it supports composing a range of KV stores with desirable features. It works by taking a single-server KV store called a datalet and enables immediate and ready-to-use distributed KV stores. Instead of redesigning a system from scratch to accomplish a specific task, a developer can drop a datalet into BespoKV and offload distributed systems to the framework. BespoKV decouples the KV store design into the control plane for distributed management and the data plane for local data storage.
One of the major limiting effects of current state-of-the-art KV stores is that they are designed with pre-existing distributed services in mind and are often specialized for one specific setting. Another limiting factor is the inflexible monolithic design where distributed features are deeply baked into a system with back-end data stores that manage inventory, orders, and supply. The rigid design of these KV stores is not adaptive to changing user demands for myriad back-end, topology, consistency, and a host of other services.
BespoKV allows an arbitrary mapping between desired services and available components while supporting distributed management services to realize and enable the distributed KV stores associated with the datalet.