Ask questionsEngine abstraction tracking issue

This issue is to organize and track progress on abstracting TiKV over generic storage engines (previous issue: There are many tasks to be done and contributions are welcome.

In scope: the process of abstracting the entire RocksDB engine API into the engine_traits and engine_rocks crates, decoupling TiKV and its test suite from RocksDB., and prototyping new engine bindings. Out of scope: refactoring engine_traits to better support with non-RocksDB engines.

The engine_traits crate docs describe the design and migration process.

The issue is updated to contain a checklist of tasks that definitely need to happen, along with name of who is working on it. When you decide to take a task, say so on the issue before attempting it so that someone else doesn't duplicate your work.

It's hard to know in what order these need to happen, nor exactly how to complete each of them, as each tends to present new challenges. Instead I have documented here, in the engine_traits crate docs, and in a status report, my experiences so far and suggestions for how to proceed.

The most important thing to keep in mind is:

Make small changes: small commits, small PRs. When you find that something else needs to change first, stop what you are doing, and make that change, and submit that PR. Don't let your patches grow out of control. Don't be afraid to throw away your work and try something else.

Task list

Migrating the engine abstractions

Take some abstraction defined in engine, implement it in engine_traits and engine_rocks, change all callers to use the new version, delete the old version. This abstraction can be a single function, an implementation of a trait, a type, or an entire trait.

  • [x] SstReader/Writer (brson)
  • [x] IOLimiter (brson) (replaced by the async-speed-limit crate in 965e8bfbca1aa1043a4785a419972bbe9d5c957c)
  • [x] Snapshot (brson)
  • [x] Mutable for WriteBatch
  • [x] Peekable for DB
  • [x] Iterable for DB (hicqu)
  • [x] cf module
  • [x] util module
  • [ ] rocks/util module
  • [ ] Errors
  • [ ] Engines
  • [x] MetricsFlusher (@Rustin-Liu )
  • [ ] move MetricsFlusher into engine_traits (cc

Eliminating direct-use of rocksdb re-exports

engine reexports many types from rocksdb. Find one of those types, add an equivalent abstraction to engine_traits, implement it for engine_rocks, migrate all callers to use engine_rocks, delete the reexport so that it cannot be used again.

  • [x] TablePropertiesCollection (brson)
  • [x] TablePropertiesCollector (brson)
  • [x] TablePropertiesCollectorFactory (brson)
  • [x] BlockBasedOptions
  • [x] CColumnFamilyDescriptor
  • [ ] CFHandle
  • [ ] Cache
  • [ ] ColumnFamilyOptions
  • [ ] CompactOptions
  • [ ] CompactionJobInfo
  • [ ] CompactionOptions
  • [ ] CompactionPriority
  • [ ] DB
  • [x] DBBackgroundErrorReason
  • [ ] DBBottommostLevelCompaction
  • [ ] DBCompactionStyle
  • [ ] DBCompressionType
  • [ ] DBEntryType
  • [ ] DBIterator
  • [ ] DBOptions
  • [ ] DBRateLimiterMode
  • [ ] DBRecoveryMode
  • [x] DBStatisticsHistogramType
  • [ ] DBStatisticsTickerType
  • [ ] DBTitanDBBlobRunMode
  • [x] DBVector
  • [ ] Env
  • [x] EnvOptions
  • [ ] EventListener
  • [x] ExternalSstFileInfo
  • [x] FlushJobInfo
  • [x] HistogramData
  • [ ] IngestExternalFileOptions
  • [x] IngestionInfo
  • [x] Kv
  • [ ] LRUCacheOptions
  • [x] MapProperty
  • [ ] MemoryAllocator
  • [ ] PerfContext
  • [ ] Range
  • [x] RateLimiter
  • [ ] ReadOptions
  • [ ] SeekKey
  • [x] SequentialFile
  • [ ] SliceTransform
  • [ ] TitanBlobIndex
  • [ ] TitanDBOptions
  • [x] UserCollectedProperties
  • [ ] Writable
  • [x] WriteBatch
  • [ ] WriteOptions
  • [x] WriteStallCondition
  • [x] WriteStallInfo
  • [x] load_latest_options,
  • [x] supported_compression,
  • [ ] run_ldb_tool,
  • [x] set_external_sst_file_global_seq_no,

"Pulling up" the generic abstractions though TiKV

Find the concrete users of these types and make them generic over the appropriate traits.

This is easiest to do by crate/subsystem

  • [ ] raftstore (brson)
  • [ ] tikv (brson)

Isolating test cases from RocksDB

  • [ ] Create an engine_test crate that simply wraps engine_rocks. Don't use typedefs. Actually define wrapping types. This creates a layer of indirection between TiKV test cases and RocksDB so that TiKV can be fully-decoupled from RocksDB.
  • [ ] Migrate test module X to engine_test (TODO expand this)
  • [ ] Write a engine_traits test suite that is parameterized over implementations

New engines

  • [x] Create an engine_panic crate that contains no behavior but may be used as a template for creating new engines, avoiding the time it takes to redifine all the impls required by an engine and keeping engine organization consistent. (brson) (
  • [ ] Create engine_mem, a simple in-memory engine. Use engine_panic as a template.


  • [ ] Document engine_traits
  • [ ] Remove protobuf dependency
  • [ ] Remove distinction between Snapshot/SyncSnapshot? Remove SyncSnapshot in favor of Arc<Snapshot>

Answer questions brson

Now that raftstore is its own crate (thanks @overvenus), we have a nice intermediate goal: remove the engine dependency from raftstore. That simplifies the problem quite a bit.


Related questions

Test failed with error `too many open files` hot 1
combine all binaries into one hot 1
PCP: test hot 1
Deploy tidb with docker swarm, fail to send snap between tikv of different hosts hot 1
Fix design of engine_traits for mutual associated types / type bound equality hot 1
Needs protoc dependency hot 1
Failpoint tests will not run when make dev hot 1
Running cargo bench causes panic hot 1
Better memory observation for tidb_query executors hot 1
how to disable gRPC batch mechanism hot 1
Github User Rank List