Contributor
Bingchang Chen
View Code

TiKV: High-performance Data Import Tool

Mentors
Andy Lok, kennytm
Organization
CNCF

Lightning is a tool used to import large amounts of data into TiDB. In addition to the logic of preparation and improving performance, the main logic of lightning can be divided into two parts:

  1. Translate SQL files to KV data by different encoders according to different backends.
  2. Import KV data into TiKV cluster.

These two steps are in the logical layer. However, in the implementation layer, these two steps are coupled. Therefore, when support for KV database like HBase, lightning still needs to convert KV like data to SQL first, and then translate SQL to KV like data again.

I will try to implement this feature by enhancing the ability of local backend, refactoring the logic of the import table, and so on.