profile
viewpoint
Nick Cameron nrc @pingcap Christchurch, New Zealand https://www.ncameron.org Software engineer at PingCAP; @rust-lang core team.

nrc/derive-new 233

derive simple constructor functions for Rust structs

nrc/apr-intro 64

An alternate introdcution to the APR book

nrc/callgraph.rs 26

Callgraphs for Rust programs

nrc/find-work 22

find something Rusty to work on

GSam/rust-refactor 19

Rust refactoring project

nrc/box-error 3

A library for error handling using boxed errors

nrc/clyde 3

wip

nrc/gh-velocity 3

measure the velocity of PRs landing in GitHub

nrc/cargo-edit 1

A utility for managing cargo dependencies from the command line.

PR opened pingcap-incubator/tinykv

DNM WIP: Implement latching for MVCC transactions

Just posting for feedback, not ready to merge.

+167 -37

0 comment

13 changed files

pr created time in 2 days

create barnchnrc/tinykv

branch : latches

created branch time in 2 days

push eventnrc/tinykv

Nick Cameron

commit sha ce79b9c994b22896f6864e50d029771ad3398c90

Remove keys and txn-infos from resolve lock command Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 3 days

pull request commentpingcap-incubator/tinykv

Implement resolve lock

This seems to be failing on CI due to a test failure in raftstore

nrc

comment created time in 3 days

Pull request review commenttikv/tikv

*: fix some test errors on AMD and/or modern Linux kernels

 slog-term = "2.4" slog_derive = "0.1" parking_lot = "0.10" sst_importer = { path = "components/sst_importer" }-sysinfo = { git = "https://github.com/tikv/sysinfo.git", branch = "tikv"}+sysinfo = { git = "https://github.com/tikv/sysinfo.git", branch = "tikv2"}

GitHub will garbage-collect commits which are not on a branch. It will also cause problems if anyone tries to cargo update.

nrc

comment created time in 3 days

pull request commentpingcap-incubator/tinykv

Implement resolve lock

the logic LGTM. But to make it simple, let's remove TxnInfos(BatchResolveLocks only used by GC) and Keys in ResolveLockRequest.

Done. @Connor1996 please take another look

nrc

comment created time in 3 days

push eventnrc/tinykv

YangKeao

commit sha 3744254dfbf5681d484e4d76604a883231c73eea

Simplify scheduler codes (#62) * remove persistent config Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove label property Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove label & location related score Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove unused codes Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove leveldb_kv Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove snapshotcount filter and peer count filter Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove unused code Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove unused cache Signed-off-by: Yang Keao <keao.yang@yahoo.com> * change Schedule function signature Signed-off-by: Yang Keao <keao.yang@yahoo.com> * simplify balance region source calc Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove unused code Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove dynamic config setting Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove push driver Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove range_cluster Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove cluster_version Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove unused function and variable Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove unused config Signed-off-by: Yang Keao <keao.yang@yahoo.com> * format Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove unused functions Signed-off-by: Yang Keao <keao.yang@yahoo.com> * restore some tests Signed-off-by: Yang Keao <keao.yang@yahoo.com> * format Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove SplitMergeInterval Signed-off-by: Yang Keao <keao.yang@yahoo.com>

view details

Nick Cameron

commit sha 4328d38bf0dcc06dbc99af2f5287c6719a35f975

Implement resolve lock Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

Nick Cameron

commit sha a159415ba5384c5b0d4fec4a5d010f6cfa81bd2f

Remove keys and txn infos from resolve lock command Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 3 days

push eventnrc/derive-new

Olivier Pinon

commit sha 612b7fc2c7211e00db3217360cb4329faf4fc2c2

Add LICENSE file

view details

Nick Cameron

commit sha 3e05ab99a2b2710275f3eb0063d1b7448d4c6d33

Merge pull request #48 from AlisCode/license Add LICENSE file

view details

push time in 3 days

PR merged nrc/derive-new

Add LICENSE file

Some companies require an explicit license file in order to allow usage of a library as a dependency.

This PR adds an MIT license file (as derive-new is licensed under those terms, according to the Cargo.toml file), giving credit to @nrc and the derive-new contributors.

+21 -0

0 comment

1 changed file

AlisCode

pr closed time in 3 days

push eventnrc/tikv

Qinxuan Chen

commit sha 4761287587c9dcbcec1d86257a9c9ef1eab50f74

Replace the fork version with upstream murmur3 (#6559)

view details

Pengcheng Xu

commit sha 66d3b534987a2bb0a83b7c4ce23fa02ceb095a26

copr: Add vectorize day_of_week (#6645) Signed-off-by: Poytr1 <pooytr1@gmail.com>

view details

Jay

commit sha faa49ac0d22b7c1063ea3bac1f40a9ac33d0386b

apply/yield: remove unnecessary log (#6654) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

3pointer

commit sha 40534f59c015710247cfdd4f6db41b9d38305da2

backup: break one key scan when met delete version (#6655) Signed-off-by: luancheng <luancheng@pingcap.com>

view details

Neil Shen

commit sha a3588deb1746a670b062d7dc3494fc770ee974e8

*: reduce duplicate into_other (#6616) Signed-off-by: Neil Shen <overvenus@gmail.com>

view details

Jay

commit sha 271fe98872de2c7d491821dafbf5cf92cf34a571

tests: make test quick and stable (#6653) - Adjust make file to run integration tests first - Fix backup test that always timeout - Check lock result to avoid double panic - Disable transport stats for tests to save CPU Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

Nick Cameron

commit sha b8fa4c54d1ae3d8868b42874b920464bb5e681f9

*: TODO Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 3 days

issue commenttikv/tikv

2020-02-18: 25% performance regression on TPCC

The offending PR was backed out, so this is fixed. I'm redoing the PR in #6644

mahjonp

comment created time in 3 days

issue closedtikv/tikv

2020-02-18: 25% performance regression on TPCC

Bug Report

There is a 25% performance regression on our TPCC daily benchmarking.

  1. What did you do?

Run performance test daily

  1. What did you expect to see?

None.

  1. What did you see instead?

Version: tidb: 6d50a47a8b70df8b6d43bc9b593c7602fe997f3c(master) 2020-02-16 11:55:03 tikv: aa786e1c3cd20260ce3258779438b29afbd933fc(master) 2020-02-17 08:10:55 pd: a1e13640572e3d6eadabf76c2869541c0c443956(master) 2020-02-17 01:50:09

Compare: tidb: https://github.com/pingcap/tidb/compare/6d50a47...6d50a47 tikv: https://github.com/tikv/tikv/compare/36dab75...aa786e1 pd: https://github.com/pingcap/pd/compare/a1e1364...a1e1364

tidb_max_cpu: 16.16, delta: -15.90% tikv_max_cpu: 18.91, delta: 33.37% tidb_max_memory: 1679.57 MiB, delta: -14.28% tikv_max_memory: 2121.40 MiB, delta: -96.43%

Measured tpmC (NewOrders): 17772.85 ± 0.00% (std=0.00), delta: -25.22% (p=nan)

  1. What version of TiKV are you using?

aa786e1c3cd20260ce3258779438b29afbd933fc

closed time in 3 days

mahjonp

PR opened pingcap-incubator/tinykv

Reviewers
Implement resolve lock

Closes #38

PTAL @Connor1996

+486 -29

0 comment

7 changed files

pr created time in 3 days

create barnchnrc/tinykv

branch : resolve

created branch time in 3 days

pull request commenttikv/tikv

*: compile and test on modern Linux and AMD v2

/bench+tpcc

nrc

comment created time in 4 days

pull request commenttikv/tikv

*: compile and test on modern Linux and AMD v2

/bench +tpcc

nrc

comment created time in 4 days

Pull request review commenttikv/tikv

tests: make test quick and stable

 pub fn new_server_config(cluster_id: u64) -> ServerConfig {         // Considering connection selection algo is involved, maybe         // use 2 or larger value here?         grpc_raft_conn_num: 1,+        // Disable stats concurrency. procinfo performs too bad without optimization,+        // disable it to save CPU for real tests.+        stats_concurrency: 0,

could this be a bool rather than an int?

BusyJay

comment created time in 4 days

Pull request review commenttikv/tikv

tests: make test quick and stable

 impl<T: Simulator> Cluster<T> {     pub fn shutdown(&mut self) {         debug!("about to shutdown cluster");         let keys;-        match self.sim.try_read() {+        match self.sim.read() {             Ok(s) => keys = s.get_node_ids(),-            Err(sync::TryLockError::Poisoned(e)) => {-                let s = e.into_inner();-                keys = s.get_node_ids();+            Err(_) => {+                if thread::panicking() {

Why do you want to avoid the double panic? Do we recover from the initial panic elsewhere?

As far as I know for this sort of thing, the best practice is to have the destructor never panic and never return an error (Including from methods it calls). A shutdown method which can return an error or panic should only ever be called explicitly. I'm not sure if that works here.

BusyJay

comment created time in 4 days

pull request commenttikv/sysinfo

update cache-size to 0.5.0

I've ported this to the tikv2 branch too.

kennytm

comment created time in 4 days

push eventtikv/sysinfo

Nick Cameron

commit sha 6970da18c7f03c1fdb93af08fbe42f6dba3340fd

Update cache-size to 0.5.0 Port of https://github.com/tikv/sysinfo/pull/4 from @kennytm Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 4 days

push eventtikv/sysinfo

kennytm

commit sha d42ff00af7a02c3cd7da84398dc69e658711e447

Cargo.toml: update cache-size to 0.5.0 Signed-off-by: kennytm <kennytm@gmail.com>

view details

Nick Cameron

commit sha 762badf918f1e0b36fce032f9f81274d79fdff7d

Merge pull request #4 from tikv/kennytm/make-cache-size-test-only update cache-size to 0.5.0

view details

push time in 4 days

PR merged tikv/sysinfo

update cache-size to 0.5.0

Include the fix to lovesegfault/cache-size#3, allowing sysinfo to be built outside of x86.

+1 -1

1 comment

1 changed file

kennytm

pr closed time in 4 days

push eventpingcap-incubator/tinykv

Nick Cameron

commit sha 564a4ce1deb195df5125862497c64867138fb350

Implement CheckTxnStatus (#66) Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 4 days

PR merged pingcap-incubator/tinykv

Implement CheckTxnStatus

PTAL @Connor1996

+403 -266

0 comment

15 changed files

nrc

pr closed time in 4 days

pull request commenttikv/tikv

*: compile and test on modern Linux and AMD v2

/bench

nrc

comment created time in 4 days

PR opened tikv/tikv

*: compile and test on modern Linux and AMD v2 C: Build S: DNM S: WIP

What have you changed?

Running the test suite on an AMD processor with Linux kernel 5.3 leads to a few errors.

This PR contains a number of small changes:

update sysinfo so that we can correctly read /sys/block/.../stat files (format is changed in modern Linux kernels)
read the chip vendor_id from (the updated) sysinfo, and where this indicates an AMD chip, skip some parts of a test (due to a different data layout returned by CPUID).
change test_reserve_space_for_recover to not rely on space available on a disk. Because other processes might write to or delete from the disk, this test was very unreliable.
fix a log rotation test where the test set the 'accessed' time but the rotation code ignores that if it can read the file's 'created' time (presumably this test currently passes because the 'created' time can never be read).
fix some typos

What is the type of the changes?

Bugfix (a change which fixes an issue)
Engineering (engineering change which doesn't change any feature or fix any issue)

How is the PR tested?

Unit test
Integration test

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

no Does this PR affect tidb-ansible?

no

+94 -60

0 comment

10 changed files

pr created time in 4 days

create barnchnrc/tikv

branch : sys

created branch time in 4 days

Pull request review commenttikv/tikv

Revert "*: fix some test errors on AMD and/or modern Linux kernels

 impl Write for RotatingFileLogger {                 fs::rename(&self.path, &new_path)?;                 self.file = open_log_file(&self.path)?; -                // Updates all rotators' states.+                // Updates all roators' states.

I intend to resubmit the original PR which includes this typo fix once I deal with the performance issue

nrc

comment created time in 4 days

push eventnrc/tinykv

Andy Lok

commit sha 0607e589fb63797186a3269e6510e84ff62f8554

test_raftstore: implement TestNodePartitionWrite (#50) * test_raftstore: implement TestNodePartitionWrite Signed-off-by: andylokandy <andylokandy@hotmail.com>

view details

Nick Cameron

commit sha c34dfd9c71d92810e799c61633cd09d181ba41aa

Merge branch 'master' into check

view details

push time in 4 days

push eventtikv/sysinfo

Nick Cameron

commit sha 56f224c0435cc74ea53ae6d2f12b778528781269

restore get_logical_cores Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

Nick Cameron

commit sha b3b907474386fdb300eac4c1e4b6d9aa488e676c

Merge pull request #5 from nrc/logical restore get_logical_cores

view details

push time in 4 days

PR merged tikv/sysinfo

restore get_logical_cores

Restore get_logical_cores. Since using the length of the processors array may have caused a performance regression (maybe by being inaccurate, or maybe the call itself is slow).

PTAL @lonng

+7 -5

0 comment

1 changed file

nrc

pr closed time in 4 days

push eventnrc/tikv

Nick Cameron

commit sha c8be83ccdf59afd0bdc41951caf3f7d47a04edd0

Revert "*: fix some test errors on AMD and/or modern Linux kernels (#6497)" This reverts commit aa786e1c3cd20260ce3258779438b29afbd933fc. Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 4 days

PR opened tikv/tikv

Reviewers
Revert "*: fix some test errors on AMD and/or modern Linux kernels C: Build-Time T: BugFix

What have you changed?

this is a simple revert of #6497

What is the type of the changes?

Bugfix (a change which fixes an issue)

How is the PR tested?

  • Unit test
  • Integration test

PTAL @zhouqiang-cl @siddontang

+65 -117

0 comment

8 changed files

pr created time in 4 days

create barnchnrc/tikv

branch : revert-sysinfo

created branch time in 4 days

PR opened tikv/sysinfo

Reviewers
restore get_logical_cores

Restore get_logical_cores. Since using the length of the processors array may have caused a performance regression (maybe by being inaccurate, or maybe the call itself is slow).

PTAL @lonng

+7 -5

0 comment

1 changed file

pr created time in 4 days

create barnchnrc/sysinfo

branch : logical

created branch time in 4 days

PR opened pingcap/blog

Add a blog post on Go and Rust

PTAL @Wenting0905 @CaitinChen @dcalvin

+89 -0

0 comment

1 changed file

pr created time in 5 days

create barnchnrc/blog

branch : go

created branch time in 5 days

PR opened pingcap-incubator/tinykv

Reviewers
Implement CheckTxnStatus

PTAL @Connor1996

+403 -266

0 comment

15 changed files

pr created time in 5 days

pull request commenttikv/tikv

Make fuzzing work.

/merge

nrc

comment created time in 5 days

create barnchnrc/tinykv

branch : check

created branch time in 5 days

push eventpingcap-incubator/tinykv

Nick Cameron

commit sha 60bf036a68275b4f4393da167227d23f43111e56

Refactor interfaces to allow moving command tests (#64) Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 5 days

PR merged pingcap-incubator/tinykv

Refactor interfaces to allow moving command tests

Trying to simplify the unit tests and move them into the commands directory (i.e., next to the commands they are testing).

PTAL @Connor1996

+1225 -1280

1 comment

22 changed files

nrc

pr closed time in 5 days

pull request commenttikv/tikv

Make fuzzing work.

/merge

nrc

comment created time in 5 days

push eventnrc/tikv

Iosmanthus Teng

commit sha 36dab75da84ec57374d364a4a4af9146ec31df07

copr: Fix fuzz error in `codec::Duration` (#6636) Signed-off-by: Iosmanthus Teng <myosmanthustree@gmail.com>

view details

Nick Cameron

commit sha aa786e1c3cd20260ce3258779438b29afbd933fc

*: fix some test errors on AMD and/or modern Linux kernels (#6497) Signed-off-by: Nick Cameron <nick@ncameron.org>

view details

5kbpers

commit sha 03f2dbe132f4c48d5e9dda460d62bf2ed269eb10

raftstore: add cmd observer (#6602) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

Nick Cameron

commit sha 0444a6c7bf535be455e8baca904ee07518c6c8bd

Merge branch 'master' into fix-fuzz

view details

push time in 5 days

pull request commentpingcap-incubator/tinykv

Refactor interfaces to allow moving command tests

I suggest extract tikv/storage/kvstore into tikv/transaction, LGTM anyway

This sounds like a good idea. I think I will finish the transaction work, then do another refactor to move things into their 'right' places.

It's too weird to define InnerServer in storage

Why is it weird? It seems to me to make sense because the interface is defined where it is used. We could also pull it further up

nrc

comment created time in 5 days

push eventnrc/tinykv

NingLin-P

commit sha 42d471f9242a2fc2d579581fc06e4f4ec3f7acf4

raft: cleanup node (#65) * remove optimisticUpdate Signed-off-by: linning <linningde25@gmail.com> * remove MaxSizePerMsg Signed-off-by: linning <linningde25@gmail.com> * rename Signed-off-by: linning <linningde25@gmail.com> * remove node Signed-off-by: linning <linningde25@gmail.com> * remove MsgUnreachable Signed-off-by: linning <linningde25@gmail.com> * remove MustSync Signed-off-by: linning <linningde25@gmail.com>

view details

Nick Cameron

commit sha 6a0253dada781bca06f29e6e61fdc431ba98ffc4

Merge branch 'master' into tests

view details

push time in 5 days

Pull request review commenttikv/website

add monthly for January 2020

+---+title: This Month in TiKV - January 2020+date: 2020-02-18+author: Calvin Weng+---++<!-- Fill in the below from the API: https://developer.github.com/v4/explorer/ -->++Welcome to the first monthly wrap-up for 2020. As folks in China are fighting against the outbreak of coronavirus, we keep our fingers crossed and hope this could end soon. In the meantime, we stay healthy and focus on driving forward our project.+++## Releases++This month our team made 2 TiKV releases (3.0.9 and 4.0.0-beta). The 3.0.9 release includes the configuration change to speed up the Region scattering, some optimized configuration items and a fixed issue. The 4.0.0-beta release includes major updates such as quick backup and restoration support, RocksDB version upgrade,and also bugs fixes.++You can review the changelogs here:++*   [3.0.9](https://github.com/tikv/tikv/releases/tag/v3.0.9)+*   [4.0.0-beta](https://github.com/tikv/tikv/releases/tag/v4.0.0-beta)++## Reading materials++[@brson](https://github.com/brson) published his first episode of the Rust Compile time series, [The Rust Compilation Calamity](https://tikv.org/blog/rust-compilation-calamity/), and shared his research and experiences with Rust compile times, using the TiKV project as a case study.++[@BusyJay](https://github.com/BusyJay) (the first author) together with [@zhexuany](https://github.com/zhexuany) and [@bb7133](https://github.com/bb7133) (second authors) wrote [an overview on Raftstore in the series of A TiKV Source Code Walkthrough (in Chinese)](https://pingcap.com/blog-cn/tikv-source-code-reading-17/). This article uses the code of TiKV 3.0 version as an example to present the key definitions and designs in the raftstore source code.
[@BusyJay](https://github.com/BusyJay) with [@zhexuany](https://github.com/zhexuany) and [@bb7133](https://github.com/bb7133) wrote [an overview on Raftstore in the series of A TiKV Source Code Walkthrough (in Chinese)](https://pingcap.com/blog-cn/tikv-source-code-reading-17/). This article uses the code of TiKV 3.0 version as an example to present the key definitions and designs in the raftstore source code.

I think the ordering of authors is clear from the context

dcalvin

comment created time in 5 days

Pull request review commenttikv/website

add monthly for January 2020

+---+title: This Month in TiKV - January 2020+date: 2020-02-18+author: Calvin Weng+---++<!-- Fill in the below from the API: https://developer.github.com/v4/explorer/ -->++Welcome to the first monthly wrap-up for 2020. As folks in China are fighting against the outbreak of coronavirus, we keep our fingers crossed and hope this could end soon. In the meantime, we stay healthy and focus on driving forward our project.+++## Releases++This month our team made 2 TiKV releases (3.0.9 and 4.0.0-beta). The 3.0.9 release includes the configuration change to speed up the Region scattering, some optimized configuration items and a fixed issue. The 4.0.0-beta release includes major updates such as quick backup and restoration support, RocksDB version upgrade,and also bugs fixes.

"a fixed issue" - could you link the issue or explain it?

dcalvin

comment created time in 5 days

Pull request review commenttikv/website

add monthly for January 2020

+---+title: This Month in TiKV - January 2020+date: 2020-02-18+author: Calvin Weng+---++<!-- Fill in the below from the API: https://developer.github.com/v4/explorer/ -->++Welcome to the first monthly wrap-up for 2020. As folks in China are fighting against the outbreak of coronavirus, we keep our fingers crossed and hope this could end soon. In the meantime, we stay healthy and focus on driving forward our project.+++## Releases++This month our team made 2 TiKV releases (3.0.9 and 4.0.0-beta). The 3.0.9 release includes the configuration change to speed up the Region scattering, some optimized configuration items and a fixed issue. The 4.0.0-beta release includes major updates such as quick backup and restoration support, RocksDB version upgrade,and also bugs fixes.++You can review the changelogs here:++*   [3.0.9](https://github.com/tikv/tikv/releases/tag/v3.0.9)+*   [4.0.0-beta](https://github.com/tikv/tikv/releases/tag/v4.0.0-beta)++## Reading materials++[@brson](https://github.com/brson) published his first episode of the Rust Compile time series, [The Rust Compilation Calamity](https://tikv.org/blog/rust-compilation-calamity/), and shared his research and experiences with Rust compile times, using the TiKV project as a case study.++[@BusyJay](https://github.com/BusyJay) (the first author) together with [@zhexuany](https://github.com/zhexuany) and [@bb7133](https://github.com/bb7133) (second authors) wrote [an overview on Raftstore in the series of A TiKV Source Code Walkthrough (in Chinese)](https://pingcap.com/blog-cn/tikv-source-code-reading-17/). This article uses the code of TiKV 3.0 version as an example to present the key definitions and designs in the raftstore source code.++## Notable PRs

If there is time, then it would be nice to explain all the terms here which readers might not understand: yatp, green GC, and the external_storage crate

dcalvin

comment created time in 5 days

Pull request review commenttikv/website

add monthly for January 2020

+---+title: This Month in TiKV - January 2020+date: 2020-02-18+author: Calvin Weng+---++<!-- Fill in the below from the API: https://developer.github.com/v4/explorer/ -->++Welcome to the first monthly wrap-up for 2020. As folks in China are fighting against the outbreak of coronavirus, we keep our fingers crossed and hope this could end soon. In the meantime, we stay healthy and focus on driving forward our project.
Welcome to the first monthly wrap-up of 2020. As folks in China are fighting against the outbreak of coronavirus, we keep our fingers crossed and hope this could end soon. In the meantime, we stay healthy and focus on driving forward our project.
dcalvin

comment created time in 5 days

Pull request review commenttikv/website

add monthly for January 2020

+---+title: This Month in TiKV - January 2020+date: 2020-02-18+author: Calvin Weng+---++<!-- Fill in the below from the API: https://developer.github.com/v4/explorer/ -->++Welcome to the first monthly wrap-up for 2020. As folks in China are fighting against the outbreak of coronavirus, we keep our fingers crossed and hope this could end soon. In the meantime, we stay healthy and focus on driving forward our project.+++## Releases++This month our team made 2 TiKV releases (3.0.9 and 4.0.0-beta). The 3.0.9 release includes the configuration change to speed up the Region scattering, some optimized configuration items and a fixed issue. The 4.0.0-beta release includes major updates such as quick backup and restoration support, RocksDB version upgrade,and also bugs fixes.++You can review the changelogs here:++*   [3.0.9](https://github.com/tikv/tikv/releases/tag/v3.0.9)+*   [4.0.0-beta](https://github.com/tikv/tikv/releases/tag/v4.0.0-beta)++## Reading materials++[@brson](https://github.com/brson) published his first episode of the Rust Compile time series, [The Rust Compilation Calamity](https://tikv.org/blog/rust-compilation-calamity/), and shared his research and experiences with Rust compile times, using the TiKV project as a case study.++[@BusyJay](https://github.com/BusyJay) (the first author) together with [@zhexuany](https://github.com/zhexuany) and [@bb7133](https://github.com/bb7133) (second authors) wrote [an overview on Raftstore in the series of A TiKV Source Code Walkthrough (in Chinese)](https://pingcap.com/blog-cn/tikv-source-code-reading-17/). This article uses the code of TiKV 3.0 version as an example to present the key definitions and designs in the raftstore source code.++## Notable PRs++- [@yiwu-arbug](https://github.com/yiwu-arbug) added S3 support to external_storage crate in [#6209](https://github.com/tikv/tikv/pull/6209).++- [@breeswish](https://github.com/breeswish) integrated dashboard into PD in [#2086](https://github.com/pingcap/pd/pull/2086).++- [@sticnarf](https://github.com/sticnarf) replaced all the readpools with [yatp](https://github.com/tikv/yatp) in [#6375](https://github.com/tikv/tikv/pull/6375) and [#6401](https://github.com/tikv/tikv/pull/6401).++- [@MyonKeminta](https://github.com/MyonKeminta) implemented green GC in [#6138](https://github.com/tikv/tikv/pull/6138), [#6070](https://github.com/tikv/tikv/pull/6070) and [#6333](https://github.com/tikv/tikv/pull/6333).++## Notable issues++**Help wanted issues** (mentoring available)++- [@brson](https://github.com/brson) opened the issue [#6402](https://github.com/tikv/tikv/issues/6402) regarding organizing and tracking progress on abstracting TiKV over generic storage engines.++- [@yiwu-arbug](https://github.com/yiwu-arbug) created the issue [#6496](https://github.com/tikv/tikv/issues/6496), requesting a feature to make RocksDB issues easy to investigate.++- [@yiwu-arbug](https://github.com/yiwu-arbug) suggested enabling `rate-bytes-per-sec` in [#6484](https://github.com/tikv/tikv/issues/6484) because it would keep RocksDB background compaction steady, smoothing both IO and CPU usage.++**Call for participation**++- [@yeya24](https://github.com/yeya24) suggested in [#6407](https://github.com/tikv/tikv/issues/6407) that some counter metrics don’t follow  the[ Prometheus Label Naming](https://prometheus.io/docs/practices/naming/) conventions and we should consider renaming them. While we agreed it is worth regulating the metrics, there would be some concerns along the way. What do you think?

Could you explain a little bit what kind of concerns we have?

dcalvin

comment created time in 5 days

Pull request review commenttikv/website

add monthly for January 2020

+---+title: This Month in TiKV - January 2020+date: 2020-02-18+author: Calvin Weng+---++<!-- Fill in the below from the API: https://developer.github.com/v4/explorer/ -->++Welcome to the first monthly wrap-up for 2020. As folks in China are fighting against the outbreak of coronavirus, we keep our fingers crossed and hope this could end soon. In the meantime, we stay healthy and focus on driving forward our project.+++## Releases++This month our team made 2 TiKV releases (3.0.9 and 4.0.0-beta). The 3.0.9 release includes the configuration change to speed up the Region scattering, some optimized configuration items and a fixed issue. The 4.0.0-beta release includes major updates such as quick backup and restoration support, RocksDB version upgrade,and also bugs fixes.

And maybe comment on how production-ready the 4.0 beta is.

dcalvin

comment created time in 5 days

Pull request review commenttikv/website

add monthly for January 2020

+---+title: This Month in TiKV - January 2020+date: 2020-02-18+author: Calvin Weng+---++<!-- Fill in the below from the API: https://developer.github.com/v4/explorer/ -->++Welcome to the first monthly wrap-up for 2020. As folks in China are fighting against the outbreak of coronavirus, we keep our fingers crossed and hope this could end soon. In the meantime, we stay healthy and focus on driving forward our project.

Maybe mention that you're all working remotely?

dcalvin

comment created time in 5 days

PR opened pingcap-incubator/tinykv

Reviewers
Refactor interfaces to allow moving command tests

Trying to simplify the unit tests and move them into the commands directory (i.e., next to the commands they are testing).

PTAL @Connor1996

+1225 -1280

0 comment

22 changed files

pr created time in 5 days

push eventnrc/tinykv

Nick Cameron

commit sha 15ada78643906d2fc3d8b061a9c290bd748cea1a

Refactor interfaces to allow moving command tests Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 5 days

create barnchnrc/tinykv

branch : tests

created branch time in 5 days

push eventnrc/tinykv

Nick Cameron

commit sha 9bae03c732457a1658f00dd9cefea493b5b2c5e4

Post-review changes Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 5 days

push eventnrc/tinykv

Nick Cameron

commit sha eaf724e6a98deee5d041834da945b42e26cc97eb

Post-review changes Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 6 days

Pull request review commentpingcap-incubator/tinykv

Implement rollback and scan

+package commands++import (+	"fmt"+	"github.com/pingcap-incubator/tinykv/kv/tikv/storage/kvstore"+	"github.com/pingcap-incubator/tinykv/proto/pkg/kvrpcpb"+)++type Rollback struct {+	request *kvrpcpb.BatchRollbackRequest+}++func NewRollback(request *kvrpcpb.BatchRollbackRequest) Rollback {+	return Rollback{request}+}++func (r *Rollback) BuildTxn(txn *kvstore.MvccTxn) error {+	txn.StartTS = &r.request.StartVersion+	for _, k := range r.request.Keys {+		err := rollbackKey(k, txn)+		if err != nil {+			return err+		}+	}+	return nil+}++func rollbackKey(key []byte, txn *kvstore.MvccTxn) error {+	lock, err := txn.GetLock(key)+	if err != nil {+		return err+	}++	if lock == nil || lock.TS != *txn.StartTS {+		// There is no lock, check the write status.+		existingWrite, err := txn.FindWrite(key, *txn.StartTS)+		if err != nil {+			return err+		}+		if existingWrite == nil {+			// There is no write either, presumably the prewrite was lost. We insert a rollback write anyway.+			write := kvstore.Write{StartTS: *txn.StartTS, Kind: kvstore.WriteKindRollback}+			txn.PutWrite(key, &write, *txn.StartTS)++			return nil+		} else {+			if existingWrite.Kind == kvstore.WriteKindRollback {+				// The key has already been rolled back, so nothing to do.+				return nil+			}++			// The key has already been committed. This should not happen since the client should never send both+			// commit and rollback requests.+			return fmt.Errorf("storage: rollback: key has already been committed %v", key)

This is an error that should never happen and if it happens in real life should end up in a panic somewhere, rather than an error which is possible and is sent back to the client. Given that, I thought it was better practice in Go to use a string error?

nrc

comment created time in 6 days

Pull request review commentpingcap-incubator/tinykv

Implement rollback and scan

 func (lock *Lock) IsLockedFor(key []byte, txnStartTs uint64) bool { 	} 	return lock.TS <= txnStartTs }++// LockedError occurs when a key or keys are locked. The protobuf representation of the locked keys is stored as Info.+type LockedError struct {

This error can't be with the others in commands because that would create an import cycle. I could pull it out into its own file in kvstore, but since there is just one error I thought it was better to put it here. I don't mind pulling it out though if you prefer.

nrc

comment created time in 6 days

pull request commenttikv/tikv

*: fix some test errors on AMD and/or modern Linux kernels

/merge

nrc

comment created time in 6 days

push eventnrc/tikv

Iosmanthus Teng

commit sha 36dab75da84ec57374d364a4a4af9146ec31df07

copr: Fix fuzz error in `codec::Duration` (#6636) Signed-off-by: Iosmanthus Teng <myosmanthustree@gmail.com>

view details

Nick Cameron

commit sha d6539904a2a5fa1b3c7d415fa8302529c10f1410

Merge branch 'master' into amd

view details

push time in 6 days

Pull request review commenttikv/tikv

*: fix some test errors on AMD and/or modern Linux kernels

 slog-term = "2.4" slog_derive = "0.1" parking_lot = "0.10" sst_importer = { path = "components/sst_importer" }-sysinfo = { git = "https://github.com/tikv/sysinfo.git", branch = "tikv"}+sysinfo = { git = "https://github.com/tikv/sysinfo.git", branch = "tikv2"}

Not really. It is a rebase not a merge, so it would be bad practice to replace a branch which is public. More practically, it would mean old versions of TiKV (any version without this PR) would no longer build because they would pull the tikv2 branch which is not backwards compatible with the tikv branch due to changes to sysinfo.

nrc

comment created time in 6 days

push eventnrc/tikv

Yilin Chen

commit sha f5b7dc8c9321b13274fced3aaa2b6801a715f2ae

*: add stack size and max task number limit to unified read pool (#6597) Signed-off-by: Yilin Chen <sticnarf@gmail.com>

view details

Yilin Chen

commit sha d2af249c4ca6c55b110b0c34461b0afe6b6bf581

*: unify read pools by default (#6593)

view details

Nick Cameron

commit sha a041903875f0c4a291f7c493c0a083cc15f26823

Merge branch 'master' into fix-fuzz

view details

push time in 6 days

Pull request review commenttikv/tikv

Make fuzzing work.

 test: 	export DYLD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}:${LOCAL_DIR}/lib" && \ 	export LOG_LEVEL=DEBUG && \ 	export RUST_BACKTRACE=1 && \-	cargo test --no-default-features --features "${ENABLE_FEATURES}" --all --exclude tests --exclude cdc ${EXTRA_CARGO_ARGS} -- --nocapture && \+	cargo test --no-default-features --features "${ENABLE_FEATURES}" --all --exclude tests --exclude \

No idea. It was excluded before so I left it excluded

nrc

comment created time in 6 days

push eventnrc/tikv

Yilin Chen

commit sha f5b7dc8c9321b13274fced3aaa2b6801a715f2ae

*: add stack size and max task number limit to unified read pool (#6597) Signed-off-by: Yilin Chen <sticnarf@gmail.com>

view details

Yilin Chen

commit sha d2af249c4ca6c55b110b0c34461b0afe6b6bf581

*: unify read pools by default (#6593)

view details

Nick Cameron

commit sha 1d3e104d8faedffe02494b2158f6a85faab4f645

Merge branch 'master' into amd

view details

push time in 6 days

pull request commentpingcap-incubator/tinykv

Implement rollback and scan

@Connor1996 Please take another look. I've addressed review comments and rebased.

nrc

comment created time in 6 days

push eventnrc/tinykv

Iosmanthus Teng

commit sha 19b5b739b989a518d0ae8e74f0fd3e4a1d89e81d

Extract standalone server (#18) * Impl standalone inner server Signed-off-by: Iosmanthus Teng <myosmanthustree@gmail.com>

view details

NingLin-P

commit sha a645a1c4fbb43560a24825543ab32afa97097250

raft: cleanup some feature (#60) * remove skipBcastCommit Signed-off-by: linning <linningde25@gmail.com> * remove MaxUncommittedEntriesSize Signed-off-by: linning <linningde25@gmail.com> * remove DisableProposalForwarding Signed-off-by: linning <linningde25@gmail.com> * remove matchBuf Signed-off-by: linning <linningde25@gmail.com> * remove TickQuiesced Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Connor <zbk602423539@gmail.com>

view details

YangKeao

commit sha 056c4cf4c52c8888e72b9bed594856fd741fb1da

Simplify scheduler codes (#61) * remove waiting merge operation Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove waiting queue Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove unused function Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove label scheduler Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove some operators Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove placement Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove influence and store limit Signed-off-by: Yang Keao <keao.yang@yahoo.com> * format Signed-off-by: Yang Keao <keao.yang@yahoo.com> * remove store limit filter Signed-off-by: Yang Keao <keao.yang@yahoo.com>

view details

Nick Cameron

commit sha 8048fc53684b6a5f1ae1eaae5d350d4e852e5934

Implement rollback Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

Nick Cameron

commit sha ff2bc66982ea2815b177c7413bedef19d6c9f6bc

Implement scan request Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

Nick Cameron

commit sha c42f752fbd51121ebd39e96bdc9d17624a1418fe

Post-review changes Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 6 days

pull request commenttikv/tikv

*: fix some test errors on AMD and/or modern Linux kernels

The new version uses the tikv2 branch from tikv/sysinfo which is the tikv branch rebased on to current master of sysinfo. There are some changes in this PR to take account of changes in the sysinfo API. The rest is the same as before.

PTAL @lonng @overvenus

nrc

comment created time in 6 days

push eventnrc/tikv

庄天翼

commit sha 7c679639885172814c4ef28f00d190337ce284a4

copr: chunk time use new 8 byte format (#6418) Signed-off-by: TennyZhuang <zty0826@gmail.com>

view details

Nick Cameron

commit sha 4620cd89ec8cba752b2c47a20253701272ee112a

build: fix builds on Mac OS (#6510) Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

5kbpers

commit sha ed953bd3feffce8c51a3c562ca4b78c24e3a4dcc

raftstore: refactor raftstore coprocessor (#6479) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

disksing

commit sha b383f12416adbf11aaa6eec362bf41cc7b94c675

pd: minor log adjustment (#6512) Signed-off-by: disksing <i@disksing.com>

view details

MyonKeminta

commit sha daa31e73bfaf91aaa1f5025c76015e2e6cad52d2

CDC: Implement Initializer of Endpoint (#6490) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

xinhua5

commit sha e4e3f067dd496f66fbae7407853c800f6f6f8f4f

backup raw kv (#6308) Signed-off-by: xinhua5 <xinhua5@163.com> Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

pingcap-github-bot

commit sha 20606553a653de34100b982913996d6644f4cd60

rust-rocksdb: Travis mac build with xcode11.3 (#6520) Signed-off-by: sre-bot <sre-bot@pingcap.com>

view details

Neil Shen

commit sha d0a03d2e7019d722193acbba192c75680bc05c7d

storage, raftstore: move RegionInfoProvider into raftstore (#6514) Signed-off-by: Neil Shen <overvenus@gmail.com>

view details

Neil Shen

commit sha 7e60f11d47cd952d4ff3848920834dc345b54626

*: move properties and compact_listener into engine_rocks (#6515) Signed-off-by: Neil Shen <overvenus@gmail.com>

view details

Iosmanthus Teng

commit sha 0f08b6beafe17825329d4a1c00a22db6b6699e6c

tidb_query: fix parse 'yyyy-mm-dd HH:MM:SS' format to Duration (#6492) Signed-off-by: Iosmanthus Teng <myosmanthustree@gmail.com>

view details

Renkai

commit sha 34b6e6c114f82d44e373867d2604f75f496f35ef

Optimize Chunk Encoder (#6341) Signed-off-by: Renkai <gaelookair@gmail.com> Co-authored-by: Wenxuan <hi@breeswish.org>

view details

Neil Shen

commit sha 35c8c7d720b336d8944a35ed134e99523ad202b2

raftstore, storage: move storage tests from raftstore to other mods (#6527) Signed-off-by: Neil Shen <overvenus@gmail.com>

view details

Jay

commit sha c784d3485973104192291072403d993d0a27630c

makefile: simplify test threads (#6524) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

Calvin Weng

commit sha 02ac19facbdca76025f9eb16005707f2812b3403

Remove governance in TiKV repo (#6416) * Remove governance in TiKV repo Signed-off-by: Calvin Weng <wenghao@pingcap.com>

view details

MyonKeminta

commit sha b95583a719bd51bd68b8033a35afccb5616e8c92

gc_worker: Split config and GCManager out to make the file smaller (#6530) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

Jay

commit sha 15693cede0e626fc9878b7667a6b21e4e894b5e9

apply: support yield (#6487) Signed-off-by: Jay Lee <busyjaylee@gmail.com> Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

YangKeao

commit sha 3ba72ab98ed8f3155bca4bd8fed13d1ddbf1a02a

update pprof version to 0.3.15 (#6529) Signed-off-by: Yang Keao <keao.yang@yahoo.com>

view details

Jay

commit sha bdd97f0338377c29b2cd8e89e73c17aeb8a0b997

batch-system: add benchmark (#6477) Signed-off-by: Jay Lee <busyjaylee@gmail.com> Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

Lonng

commit sha ce9bf269f17a11836e3acff5e68eaa43788996a9

diagnostics: refine the load information of memory (#6535) Signed-off-by: Lonng <heng@lonng.org>

view details

Neil Shen

commit sha d15463937f0fe0aa8bf7fc934783e4201f91a355

cdc: add CDC delegate (#6522) * cdc: add CDC delegate Signed-off-by: Neil Shen <overvenus@gmail.com> * temporarily disable clippy check for tests Signed-off-by: Neil Shen <overvenus@gmail.com>

view details

push time in 6 days

PR closed tikv/sysinfo

Rebase the TiKV branch on to master

So, some of our changes got upstreamed, but did not (in particular, io and mem load). So rather than use the old tikv branch, I rebased it to current sysinfo master. I've removed some code which has alternatives in upstream. I think that we can work to upstream the other stuff, but in the meantime I need something so I can run TiKV tests on my machine.

Unfortunately, since this is a rebase, it cannot be merged with the tikv branch. If this PR passes review I propose we create a new branch, tikv2 or something, to land it rather than merging this PR into our master.

PTAL @lonng

+5810 -2714

4 comments

50 changed files

nrc

pr closed time in 6 days

pull request commenttikv/sysinfo

Rebase the TiKV branch on to master

This is now https://github.com/tikv/sysinfo/tree/tikv2

nrc

comment created time in 6 days

create barnchtikv/sysinfo

branch : tikv2

created branch time in 6 days

delete branch tikv/sysinfo

delete branch : tikv-rb

delete time in 6 days

create barnchtikv/sysinfo

branch : tikv-rb

created branch time in 6 days

pull request commenttikv/tikv

Make fuzzing work.

@breeswish @zhouqiang-cl PTAL. Passing CI now, but the Makefile got a bit more complicated

nrc

comment created time in 7 days

push eventnrc/tikv

pingcap-github-bot

commit sha b0241e858f36f689c53856b67f0f6fe9b93bea63

rust-rocksdb: Add reference-counted TablePropertiesCollection bindings (#6550) Signed-off-by: sre-bot <sre-bot@pingcap.com>

view details

Nick Cameron

commit sha 521d88f4d0375bb57e957c6b094876ceb2918a71

Merge branch 'master' into fix-fuzz

view details

push time in 7 days

push eventnrc/tikv

Nick Cameron

commit sha ac18e9f83d490d345cffc4069fb8931a42445749

Make fuzzing work. Adds a new make target, fuzz. Closes #6591 Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 7 days

pull request commenttikv/sysinfo

Rebase the TiKV branch on to master

Hi, @nrc Can we move some changes which cannot be merged to upstream to an individual repository?

We could, but I think not now - this PR is an improvement on the current situation, and the better long-term solution is to upstream everything. If that is not possible (I think it should be), then we can make another repo.

nrc

comment created time in 8 days

PR opened tikv/sysinfo

Reviewers
Rebase the TiKV branch on to master

So, some of our changes got upstreamed, but did not (in particular, io and mem load). So rather than use the old tikv branch, I rebased it to current sysinfo master. I've removed some code which has alternatives in upstream. I think that we can work to upstream the other stuff, but in the meantime I need something so I can run TiKV tests on my machine.

Unfortunately, since this is a rebase, it cannot be merged with the tikv branch. If this PR passes review I propose we create a new branch, tikv2 or something, to land it rather than merging this PR into our master.

PTAL @lonng

+5810 -2714

0 comment

50 changed files

pr created time in 8 days

create barnchnrc/sysinfo

branch : tikv-rb

created branch time in 8 days

PR opened tikv/tikv

Reviewers
Make fuzzing work. C: Build C: Test/Bench T: BugFix

Fixes a Cargo error and adds a new make target, fuzz.

What have you changed?

There is a bit of complexity passing the right features to dependencies when fuzzing. This PR fixes that by adding dependencies to fuzz-targets. I also add a make target to make fuzzing more convenient.

What is the type of the changes?

  • Bugfix (a change which fixes an issue)
  • Engineering (engineering change which doesn't change any feature or fix any issue)

How is the PR tested?

  • Manual test (add detailed scripts or steps below)

make test and FUZZ_TARGET=fuzz_codec_bytes make test

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

no

Does this PR affect tidb-ansible?

no

Refer to a related PR or issue link (optional)

Fixes #6591

PTAL @zhouqiang-cl @breeswish

+19 -5

0 comment

4 changed files

pr created time in 8 days

create barnchnrc/tikv

branch : fix-fuzz

created branch time in 8 days

Pull request review commentpingcap-incubator/tinykv

Add split check test

+package codec++import "github.com/pingcap/errors"++const (+	encGroupSize = 8+	encMarker    = byte(0xFF)+	encPad       = byte(0x0)+)++var pads = make([]byte, encGroupSize)++// EncodeBytes guarantees the encoded value is in ascending order for comparison,+// encoding with the following rule:+//  [group1][marker1]...[groupN][markerN]+//  group is 8 bytes slice which is padding with 0.+//  marker is `0xFF - padding 0 count`+// For example:+//   [] -> [0, 0, 0, 0, 0, 0, 0, 0, 247]+//   [1, 2, 3] -> [1, 2, 3, 0, 0, 0, 0, 0, 250]+//   [1, 2, 3, 0] -> [1, 2, 3, 0, 0, 0, 0, 0, 251]+//   [1, 2, 3, 4, 5, 6, 7, 8] -> [1, 2, 3, 4, 5, 6, 7, 8, 255, 0, 0, 0, 0, 0, 0, 0, 0, 247]+// Refer: https://github.com/facebook/mysql-5.6/wiki/MyRocks-record-format#memcomparable-format+func EncodeBytes(data []byte) []byte {+	// Allocate more space to avoid unnecessary slice growing.+	// Assume that the byte slice size is about `(len(data) / encGroupSize + 1) * (encGroupSize + 1)` bytes,+	// that is `(len(data) / 8 + 1) * 9` in our implement.+	dLen := len(data)+	result := make([]byte, 0, (dLen/encGroupSize+1)*(encGroupSize+1)+8) // make extra room for appending ts+	for idx := 0; idx <= dLen; idx += encGroupSize {+		remain := dLen - idx+		padCount := 0+		if remain >= encGroupSize {+			result = append(result, data[idx:idx+encGroupSize]...)+		} else {+			padCount = encGroupSize - remain+			result = append(result, data[idx:]...)+			result = append(result, pads[:padCount]...)+		}++		marker := encMarker - byte(padCount)+		result = append(result, marker)+	}+	return result+}++// DecodeBytes decodes bytes which is encoded by EncodeBytes before,+// returns the leftover bytes and decoded value if no error.+func DecodeBytes(b []byte) ([]byte, []byte, error) {

It is a bit sad that we cannot always know whether a key is raw or encoded. I suppose the problem is that the client can use the raw API to write raw keys to the DB? But this makes me wonder that one could write a raw key which can be validly decoded, but isn't in fact an encoded key and then things would break. Maybe we should not offer the raw API in TinyKV at all? Does TinySQL use it?

Connor1996

comment created time in 10 days

push eventnrc/tinykv

Nick Cameron

commit sha 8a3ba39622cd264924697ceff31f61735a51edad

Implement scan request Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 10 days

push eventnrc/tinykv

Connor

commit sha 9dc002c33fdb8c9be7bb15ac1d9b146f3d995c06

Add split check test (#52) * add split check test Signed-off-by: Connor1996 <zbk602423539@gmail.com>

view details

Nick Cameron

commit sha e0d3499c1dd6910943afd854d027938d213cf659

Implement rollback Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

Nick Cameron

commit sha b3759bd27e59945294ef6aa722f2134556edbd2d

Implement scan request Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

push time in 10 days

PR closed tikv/sysinfo

Fix some issues causing errors in TiKV on my machine

I'm running an AMD machine with Linux kernel version 5.3. This causes two issues:

  • The stat entries in /sys/block/ have 15 columns instead of 11. This is a backwards compatible change, so we can just ignore the extra four columns.
  • The layout of data returned by the cpuid instruction is completely different. The cache-size crate is based on the rust-cpuid crate, which does not yet support AMD. In order to distinguish this state I need to know the vendor_id of the CPU.

These are addressed in the first and second commits, respectively.

PTAL @lonng

+36 -6

5 comments

10 changed files

nrc

pr closed time in 10 days

pull request commenttikv/tikv

*: fix some test errors on AMD and/or modern Linux kernels

Update: most changes to sysinfo have been upstreamed, so I will adapt this PR to use upstream sysinfo rather than our fork

nrc

comment created time in 10 days

pull request commentGuillaumeGomez/sysinfo

Implement load_average, CPU's frequency and vendor_id

Nice! Thanks for pushing this! Sorry I didn't have time to look it over

GuillaumeGomez

comment created time in 10 days

Pull request review commentpingcap-incubator/tinykv

Add split check test

+package codec++import "github.com/pingcap/errors"++const (+	encGroupSize = 8+	encMarker    = byte(0xFF)+	encPad       = byte(0x0)+)++var pads = make([]byte, encGroupSize)++// EncodeBytes guarantees the encoded value is in ascending order for comparison,+// encoding with the following rule:+//  [group1][marker1]...[groupN][markerN]+//  group is 8 bytes slice which is padding with 0.+//  marker is `0xFF - padding 0 count`+// For example:+//   [] -> [0, 0, 0, 0, 0, 0, 0, 0, 247]+//   [1, 2, 3] -> [1, 2, 3, 0, 0, 0, 0, 0, 250]+//   [1, 2, 3, 0] -> [1, 2, 3, 0, 0, 0, 0, 0, 251]+//   [1, 2, 3, 4, 5, 6, 7, 8] -> [1, 2, 3, 4, 5, 6, 7, 8, 255, 0, 0, 0, 0, 0, 0, 0, 0, 247]+// Refer: https://github.com/facebook/mysql-5.6/wiki/MyRocks-record-format#memcomparable-format+func EncodeBytes(data []byte) []byte {+	// Allocate more space to avoid unnecessary slice growing.+	// Assume that the byte slice size is about `(len(data) / encGroupSize + 1) * (encGroupSize + 1)` bytes,+	// that is `(len(data) / 8 + 1) * 9` in our implement.+	dLen := len(data)+	result := make([]byte, 0, (dLen/encGroupSize+1)*(encGroupSize+1)+8) // make extra room for appending ts+	for idx := 0; idx <= dLen; idx += encGroupSize {+		remain := dLen - idx+		padCount := 0+		if remain >= encGroupSize {+			result = append(result, data[idx:idx+encGroupSize]...)+		} else {+			padCount = encGroupSize - remain+			result = append(result, data[idx:]...)+			result = append(result, pads[:padCount]...)+		}++		marker := encMarker - byte(padCount)+		result = append(result, marker)+	}+	return result+}++// DecodeBytes decodes bytes which is encoded by EncodeBytes before,+// returns the leftover bytes and decoded value if no error.+func DecodeBytes(b []byte) ([]byte, []byte, error) {

Yeah, I mean it would be nicer to panic in DecodeBytes than DecodeKey. We can return nil, nil if there is an error

Connor1996

comment created time in 10 days

Pull request review commentpingcap-incubator/tinykv

Add split check test

 func (txn *MvccTxn) PutValue(key []byte, value []byte) { // timestamped keys are sorted first by key (ascending), then by timestamp (descending). The encoding is based on // https://github.com/facebook/mysql-5.6/wiki/MyRocks-record-format#memcomparable-format. func EncodeKey(key []byte, ts uint64) []byte {-	newLen := (len(key)/8+1)*9 + 8-	newKey := make([]byte, 0, newLen)--	for i := 0; i <= len(key); i += 8 {-		remaining := len(key) - i-		var padCount byte-		if remaining >= 8 {-			padCount = 0-			newKey = append(newKey, key[i:i+8]...)-		} else {-			padCount = 8 - byte(remaining)-			newKey = append(newKey, key[i:]...)-			newKey = append(newKey, make([]byte, padCount)...)-		}--		newKey = append(newKey, 0xff-padCount)-	}--	// Append the timestamp, Note we invert the timestamp so that when sorted, they are in descending order.-	newKey = append(newKey, make([]byte, 8)...)-	binary.BigEndian.PutUint64(newKey[newLen-8:], ^ts)+	encodedKey := codec.EncodeBytes(key)+	newKey := append(encodedKey, make([]byte, 8)...)+	binary.BigEndian.PutUint64(newKey[len(newKey)-8:], ^ts)

Could use len(encodedKey) instead of len(newKey)-8, I think it's clearer.

Connor1996

comment created time in 10 days

Pull request review commentpingcap-incubator/tinykv

Add split check test

+package codec++import "github.com/pingcap/errors"++const (+	encGroupSize = 8+	encMarker    = byte(0xFF)+	encPad       = byte(0x0)+)++var pads = make([]byte, encGroupSize)++// EncodeBytes guarantees the encoded value is in ascending order for comparison,+// encoding with the following rule:+//  [group1][marker1]...[groupN][markerN]+//  group is 8 bytes slice which is padding with 0.+//  marker is `0xFF - padding 0 count`+// For example:+//   [] -> [0, 0, 0, 0, 0, 0, 0, 0, 247]+//   [1, 2, 3] -> [1, 2, 3, 0, 0, 0, 0, 0, 250]+//   [1, 2, 3, 0] -> [1, 2, 3, 0, 0, 0, 0, 0, 251]+//   [1, 2, 3, 4, 5, 6, 7, 8] -> [1, 2, 3, 4, 5, 6, 7, 8, 255, 0, 0, 0, 0, 0, 0, 0, 0, 247]+// Refer: https://github.com/facebook/mysql-5.6/wiki/MyRocks-record-format#memcomparable-format+func EncodeBytes(data []byte) []byte {+	// Allocate more space to avoid unnecessary slice growing.+	// Assume that the byte slice size is about `(len(data) / encGroupSize + 1) * (encGroupSize + 1)` bytes,+	// that is `(len(data) / 8 + 1) * 9` in our implement.+	dLen := len(data)+	result := make([]byte, 0, (dLen/encGroupSize+1)*(encGroupSize+1)+8) // make extra room for appending ts+	for idx := 0; idx <= dLen; idx += encGroupSize {+		remain := dLen - idx+		padCount := 0+		if remain >= encGroupSize {+			result = append(result, data[idx:idx+encGroupSize]...)+		} else {+			padCount = encGroupSize - remain+			result = append(result, data[idx:]...)+			result = append(result, pads[:padCount]...)+		}++		marker := encMarker - byte(padCount)+		result = append(result, marker)+	}+	return result+}++// DecodeBytes decodes bytes which is encoded by EncodeBytes before,+// returns the leftover bytes and decoded value if no error.+func DecodeBytes(b []byte) ([]byte, []byte, error) {

Can we panic here instead of returning the error? I think we do not need proper error handling for TinyKV.

Connor1996

comment created time in 10 days

pull request commentpingcap-incubator/tinykv

Add split check test

No, EncodeBytes is only used by transaction package. But region split should make a clean cut for mvcc key that the different versions of same user key should locate in a same region. That's why split check use DecodeBytes to check if the key is encoded.

OK, thanks for the explanation. Longer term I think we should use some kind of reader to check this rather than use DecodeBytes, but that doesn't need to block this PR

Connor1996

comment created time in 10 days

pull request commenttikv/client-rust

Convert part of futures to `async/await`

@sticnarf PTAL

silathdiir

comment created time in 11 days

issue commenttikv/client-rust

Convert futures to async/await

And I also tried to update in file src/pd/cluster.rs, but could not compile successfully, the below error was reported.

How did you try to change retry.rs? I don't see any obvious futures to remove.

nrc

comment created time in 12 days

Pull request review commenttikv/client-rust

Convert part of futures to `async/await`

 impl Client {                 requests::new_raw_scan_request(range, limit, self.key_only, self.cf.clone())                     .execute(self.rpc.clone()),             )-        }+        };++        request.await

I think this function can be further simplified by directly returning the error in the first case and using ...execute.await in the second.

silathdiir

comment created time in 12 days

Pull request review commenttikv/client-rust

Convert part of futures to `async/await`

 impl Client {     /// let result = req.await;     /// # });     /// ```-    pub fn batch_scan(+    pub async fn batch_scan(         &self,         ranges: impl IntoIterator<Item = impl Into<BoundRange>>,         each_limit: u32,-    ) -> impl Future<Output = Result<Vec<KvPair>>> {-        if each_limit > MAX_RAW_KV_SCAN_LIMIT {+    ) -> Result<Vec<KvPair>> {+        let request = if each_limit > MAX_RAW_KV_SCAN_LIMIT {

Similarly to above

silathdiir

comment created time in 12 days

Pull request review commenttikv/client-rust

Convert part of futures to `async/await`

 impl KvClient for KvRpcClient {     } } -fn map_errors_and_trace<Resp, RpcFuture>(+async fn map_errors_and_trace<Resp, RpcFuture>(     request_name: &'static str,     fut: ::grpcio::Result<RpcFuture>,-) -> impl Future<Output = Result<Resp>>+) -> Result<Resp> where     Compat01As03<RpcFuture>: Future<Output = std::result::Result<Resp, ::grpcio::Error>>,     Resp: HasError + Sized + Clone + Send + 'static, {     let context = tikv_stats(request_name); -    // FIXME should handle the error, not unwrap.-    Compat01As03::new(fut.unwrap())-        .map(|r| match r {-            Err(e) => Err(ErrorKind::Grpc(e).into()),-            Ok(r) => Ok(r),-        })-        .map(move |r| context.done(r))+    let fut = match fut {+        Err(e) => return context.done(Err(ErrorKind::Grpc(e).into())),+        Ok(f) => f,+    };

Could this be written as context.done(fut.and_then(async |f| Compat01As03::new(fut).await).map_err(|e| ErrorKind::Grpc(e).into())) or something similar?

silathdiir

comment created time in 12 days

more