profile
viewpoint
Brian Anderson brson PingCAP https://brson.github.io I write Rust code. Also, I do other things — things that aren't even related to Rust.

alexcrichton/flate2-rs 318

FLATE, Gzip, and Zlib bindings for Rust

brson/basic-http-server 257

A simple static HTTP server in Rust, for learning and local doc development

brson/annotated-std-rs 66

An annotation of the Rust standard library

brson/being-rust 15

Intro to Rust talk

brson/big_s 14

Rust's missing `String` literal

brson/archaea 12

Historic Rust code browsing

alegalle/rustgl_4_2_core 8

Opengl 3.2 - 4.2 bindings for rust

startedu2zv1wx/neut

started time in 4 days

pull request commenttikv/tikv

Remove engine::rocks::util

/build

brson

comment created time in 5 days

push eventbrson/mem-basics

push time in 5 days

PR opened tikv/tikv

Reviewers
Remove engine::rocks::util

What problem does this PR solve?

Work toward deleting engine crate in favor of engine_traits

What is changed and how it works?

This creates a few new abstractions for things in engine::rocks::util, and updates code to use them. Then it moves the remaining DB constructors to engine_rocks::raw_util. The contents of this module reflect a lower-level version of functions already defined in engine_rocks::util. These constructors are still used extensively across the codebase, particularly in tests, but will eventually disappear in favor of RocksEngine constructors. Then this module will be deleted entirely.

After this there are only a few more steps before the engine crate is deleted.

Related changes

https://github.com/tikv/tikv/issues/6402

Check List <!--REMOVE the items that are not applicable-->

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+281 -261

0 comment

38 changed files

pr created time in 5 days

create barnchbrson/tikv

branch : rm-rocks-util

created branch time in 5 days

create barnchbrson/tikv

branch : no-engine-rocks

created branch time in 5 days

create barnchbrson/tikv

branch : rm-engine-crate

created branch time in 5 days

push eventbrson/my-rust-lists

Brian Anderson

commit sha 99565d7c176edda32d1fa445a0948075c0703b1b

.

view details

push time in 6 days

push eventbrson/mem-basics

Brian Anderson

commit sha ade0e8fe4a50e88f24b3877a71f49dd0e017dc13

.

view details

push time in 6 days

push eventbrson/mem-basics

Brian Anderson

commit sha fea3ff0999db89fe68d9c4bd1a2767afb66ad0b5

.

view details

push time in 6 days

push eventbrson/mem-basics

Brian Anderson

commit sha f08dbff99f248526b20e85b71c68bb2fbcf6b024

.

view details

push time in 6 days

push eventbrson/mem-basics

Brian Anderson

commit sha 279aebf5778e696f9f0176ff2d502a53286c089e

.

view details

Brian Anderson

commit sha 5d2c79994897a3a9ad2ca9d323b725f647407a11

.

view details

Brian Anderson

commit sha 8013fe8ee56826a4b50a20cf603d2624fa514826

.

view details

push time in 6 days

push eventbrson/mem-basics

Brian Anderson

commit sha c5b7d9fb99229fe14dcbac51ed8cbb7ce446435f

.

view details

Brian Anderson

commit sha 894a726312b6f2c51cf23abbc6aa06ce3126d329

.

view details

push time in 6 days

push eventbrson/mem-basics

Brian Anderson

commit sha 4303f4527c60fb841fdeb759eb1afbb37d10d7c2

.

view details

Brian Anderson

commit sha 922617ef346bcb0c1596d7626423e0c13e032d26

.

view details

Brian Anderson

commit sha 00387901b833fd359b5bba7b5fe881d21f7b1265

.

view details

push time in 6 days

push eventbrson/mem-basics

Brian Anderson

commit sha 5308601c3476e4537333d8838398b53c6b8c47a7

.

view details

Brian Anderson

commit sha aa9b582c4152beb845b9f3874f9e538ac78ffa4a

.

view details

Brian Anderson

commit sha 9925e01b2b83a527f82b18aa3aae1cc2b3ea2477

.

view details

Brian Anderson

commit sha 4290539d6ac474942d78519291bf34053dafc7d8

.

view details

push time in 6 days

push eventbrson/mem-basics

Brian Anderson

commit sha c381042497e8a7faa1d058b5868015f246151f2c

.

view details

Brian Anderson

commit sha 410d7eb7755c3d798cab76cff05600bd83edcb70

.

view details

Brian Anderson

commit sha 8f286843add35a70f798c4f17786e797bdcc2472

.

view details

Brian Anderson

commit sha 3ad763a082a63df60a5121a1868918d4dc1652b0

.

view details

Brian Anderson

commit sha b25557d2f7e6a9892ed569d4959bc6d69e26cde2

.

view details

Brian Anderson

commit sha 804accc5a387b67290316b77cfb6f3a22123c00d

.

view details

push time in 6 days

create barnchbrson/mem-basics

branch : master

created branch time in 7 days

created repositorybrson/mem-basics

created time in 7 days

push eventbrson/my-rust-lists

Brian Anderson

commit sha 25e7d21ff6b833010a8c9a153c9dd49362b7a3b6

.

view details

push time in 7 days

push eventbrson/my-rust-lists

Brian Anderson

commit sha cdc491ffed2744af3c0d91b87ed84c99d0275446

.

view details

push time in 7 days

pull request commenttikv/tikv

Add exists function to engine_traits

/merge

brson

comment created time in 8 days

push eventbrson/tikv

二手掉包工程师

commit sha badbf76a8e5d57d1682dbb53328bd6fd615526fd

raftstore: fix typo and remove useless msg enum (#7862) Signed-off-by: Rustin-liu <rustin.liu@gmail.com>

view details

Brian Anderson

commit sha 58ca498ff7bb59975b517b4e02d495cff8c74de0

Move `engine::stats` into `engine_traits` (#7805) Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

MyonKeminta

commit sha 6b09cadbf55311ac07fc51e1b43e3b57b54e71f2

Fix the issue that end key of a partial rawkv-restore range is inclusive (#7196) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

Liqi Geng

commit sha cc79a5e49cbe96ac7ab909bed23db6b02572c60c

test: fix check split bug in pd (#7841) Signed-off-by: Liqi Geng <gengliqiii@gmail.com>

view details

WT

commit sha c0613a069181a4de0553c214a463a8308fdcdc30

update community meeting time (#7867) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

JmPotato

commit sha 6c1f28f0c08088645b34271036dfb9775bcfe211

UCP: Migrate scalar function `GreatestReal` (Vectorize) from TiDB (#7860) Signed-off-by: JmPotato <ghzpotato@gmail.com>

view details

Brian Anderson

commit sha 1ddfff19b239680ea2d00ca411b8548e47f571fe

Merge remote-tracking branch 'origin/master' into no-db-exists

view details

push time in 8 days

pull request commenttikv/tikv

raftstore: More engine_traits generics cleanup

/merge

brson

comment created time in 8 days

push eventbrson/tikv

Qiannan

commit sha 8bcb0a32115424b93efe8f053c1bef1d387b3688

UCP: Migrate scalar function `Rpad` from TiDB (#7668) Signed-off-by: Qiannan Lyu <lvqiannan@gmail.com>

view details

Wallace

commit sha 6c326d1b5a412d831047c43fa7e881b7a7a3c575

Fix some case of #6683 (#7826) Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

view details

WT

commit sha ff16adc678ab92d58d3d6b4fa7b9c3e22c4cf8e7

update content (#7851) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

JmPotato

commit sha aa65d34b4ccbd2df66ba13880aa10e4595350420

UCP: Migrate `GreatestInt` (Vectorize) from TiDB (#7847) Signed-off-by: JmPotato <ghzpotato@gmail.com>

view details

trabbart

commit sha 00fe91fcf8bde4233d15b754f6fca7c4f17b7579

copr: add non-vectorized time_to_sec (#7203) Signed-off-by: Bartosz Podkanowicz <podkanowicz.bartosz@gmail.com>

view details

Qiannan

commit sha ff6bdcfd266a22a259f6224e802a6493df27b085

copr: Migrate scalar function `LpadUtf8` from TiDB (#7514) Signed-off-by: Qiannan Lyu <lvqiannan@gmail.com>

view details

Brian Anderson

commit sha e5afd630e35b42e970e6184e8b71eb98263a9363

Remove `engine::iterable` (#7801) Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

二手掉包工程师

commit sha badbf76a8e5d57d1682dbb53328bd6fd615526fd

raftstore: fix typo and remove useless msg enum (#7862) Signed-off-by: Rustin-liu <rustin.liu@gmail.com>

view details

Brian Anderson

commit sha 58ca498ff7bb59975b517b4e02d495cff8c74de0

Move `engine::stats` into `engine_traits` (#7805) Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

MyonKeminta

commit sha 6b09cadbf55311ac07fc51e1b43e3b57b54e71f2

Fix the issue that end key of a partial rawkv-restore range is inclusive (#7196) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

Liqi Geng

commit sha cc79a5e49cbe96ac7ab909bed23db6b02572c60c

test: fix check split bug in pd (#7841) Signed-off-by: Liqi Geng <gengliqiii@gmail.com>

view details

WT

commit sha c0613a069181a4de0553c214a463a8308fdcdc30

update community meeting time (#7867) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

JmPotato

commit sha 6c1f28f0c08088645b34271036dfb9775bcfe211

UCP: Migrate scalar function `GreatestReal` (Vectorize) from TiDB (#7860) Signed-off-by: JmPotato <ghzpotato@gmail.com>

view details

Brian Anderson

commit sha 497b7fb8364700d3d6b0d1c047b82e63ffa9ff88

Merge remote-tracking branch 'origin/master' into engine-traits-more-raftstore-generics-cleanup

view details

push time in 8 days

PR opened tikv/tikv

Reviewers
engine: Remove duplicate DATA_KEY_PREFIX_LEN definition

What problem does this PR solve?

Remove duplicate definition of DATA_KEY_PREFIX_LEN from engine crate in favor of engine_traits crate.

What is changed and how it works?

Related changes

https://github.com/tikv/tikv/issues/6402

Check List <!--REMOVE the items that are not applicable-->

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+3 -6

0 comment

3 changed files

pr created time in 8 days

create barnchbrson/tikv

branch : rm-dupe-data-key-prefix

created branch time in 8 days

starteddohliam/dropin-minimal-css

started time in 8 days

issue commenttikv/tikv

tokio-threadpool reads stale stack frames in status_service

afaik hyper does not require tokio, but it does use tokio by default.

brson

comment created time in 9 days

issue openedtikv/tikv

tokio-threadpool reads stale stack frames in status_service

Bug Report

Address sanitizer reports invalid reads inside tokio-threadpool during status_service test cases. The code in question is pretty complex, and on a drop path, so I haven't been able to verify that it is definitely wrong, but there are unsafe thread locals involved, so it seems plausibly wrong.

tokio-threadpool is deprecated so it seems the best solution is just to replace it completely, presumably with yatp cc @sticnarf

What version of TiKV are you using?

e5afd630e35b42e970e6184e8b71eb98263a9363

What operating system and CPU are you using?

Linux

Steps to reproduce

ASAN_OPTIONS=allow_addr2line=true RUSTFLAGS="-Zsanitizer=address" cargo -Zbuild-std test --target=x86_64-unknown-linux-gnu --lib -p tikv -- security_status_service_without_cn

What did you expect?

What did happened?

https://gist.github.com/brson/049fbb51e7cf6a7e373a652e4bffbaf2

created time in 9 days

push eventbrson/my-rust-lists

Brian Anderson

commit sha 9de862415baa672be34a94dfd4f681a4a3e01db6

.

view details

push time in 9 days

pull request commenttikv/tikv

Fix two instances of undefined behavior in codec crate

Waiting on further review from @breeswish

brson

comment created time in 10 days

PR opened tikv/tikv

Reviewers
Add exists function to engine_traits

What problem does this PR solve?

This is another step toward deleting the engine crate. Add an exists method to MiscExt in engine_traits, and use that in the config module. The old db_exist function is still used internally, so mark it deprecated for now.

What is changed and how it works?

Related changes

https://github.com/tikv/tikv/issues/6402

Check List <!--REMOVE the items that are not applicable-->

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+18 -3

0 comment

5 changed files

pr created time in 10 days

create barnchbrson/tikv

branch : no-db-exists

created branch time in 10 days

PR opened tikv/tikv

tikv_util: Fix two leaks in mpsc test cases

As reported by asan.

Signed-off-by: Brian Anderson andersrb@gmail.com

<!-- Thank you for contributing to TiKV!

If you haven't already, please read TiKV's CONTRIBUTING document.

If you're unsure about anything, just ask; somebody should be along to answer within a day or two.

PR Title Format:

  1. module [, module2, module3]: what's changed
  2. *: what's changed -->

What problem does this PR solve?

Two test cases in tikv_util use the forget method of CpuFuture, which seems to do what it says and leak memory. This changes the test cases to not do that.

Found with asan.

On its own this doesn't have much impact, but it will make future dynamic analysis easier to not have these leaks.

What is changed and how it works?

Related changes

Check List <!--REMOVE the items that are not applicable-->

  • Unit test

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+4 -6

0 comment

1 changed file

pr created time in 10 days

create barnchbrson/tikv

branch : tikv-util-leaks

created branch time in 10 days

pull request commenttikv/tikv

Remove `engine::iterable`

/merge

brson

comment created time in 10 days

pull request commenttikv/tikv

Remove `engine::iterable`

/merge

brson

comment created time in 10 days

push eventbrson/rust-doc-tool

Brian Anderson

commit sha 64a4d3f5fd13ee53c765506b20ef697a30dfb0ef

.

view details

push time in 11 days

push eventbrson/rust-anthology

Brian Anderson

commit sha 29c07f5017a3705d8e0316a7ea1705b36bc5ca3c

.

view details

push time in 11 days

PR opened tikv/tikv

Reviewers
raftstore: More engine_traits generics cleanup

What problem does this PR solve?

Replace various KvEngine bounds with Snapshot bounds. This simplifies parts of raftstore.

There are still big parts of raftstore that depend on the full KvEngine trait, but I don't think it's worth doing any more cleanup in that direction right now. From here I'll continue removing the engine_rocks dependency from raftstore.

The first few commits here are previous revisions of an already-merged PR to remove the SyncSnapshot trait.

What is changed and how it works?

Related changes

https://github.com/tikv/tikv/issues/6402

Check List <!--REMOVE the items that are not applicable-->

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+286 -236

0 comment

36 changed files

pr created time in 12 days

push eventbrson/tikv

Wallace

commit sha a935b3b01479c1a7fa51d3b28e11ab0c9a21ea99

batch multiple write request into an entry (#6600) (#6683) * batch multiple write request into an entry (#6600) * batch multiple write request into an entry Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix name Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * refactor code for better to understand Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * add metrics Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * change metric of batch request to local metric Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix lint Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix wake up Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * wake up when batch entry Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * move code to fsm/peer.rs Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * not batch delete range Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix write batch size Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * remove unused variable Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix conflict of trait Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

view details

Jay

commit sha d10871fc80eef53a11fef4620dc5859cfc7319e8

tests: check tombstone safety (#7476) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

Hexilee

commit sha 817445dbb73edab2af7a49618df9161740deb611

status_server: support dumping region meta by http (#7650) Signed-off-by: Hexilee <i@hexilee.me>

view details

Alex Chi

commit sha 5b3b1e0ee454876ca08033aa4d06ad3df4cf38d7

copr: Add vectorized FromBase64 (#7767) Signed-off-by: Alex Chi <iskyzh@gmail.com>

view details

zhangjinpeng1987

commit sha 9e83b1045768e4421664d83ecdbce339eb6656cb

github action: assign bugs and questions to project 38 (#7752) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Shirly <AndreMouche@126.com>

view details

Wenxuan

commit sha 7f6620f9110d4fad57024ab77c3993da45323fae

Improve memory defragmentation (#7788) Signed-off-by: Breezewish <breezewish@pingcap.com>

view details

MyonKeminta

commit sha 8a6aa40512591d79250312840def9230f6db8e12

*: Remove incorrect alert rule (#7787) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

Connor

commit sha 39b243217f5fe2c3f37f9283350b89ccb11f6238

engine: Add titan customized statistics (#6328) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

view details

yiwu-arbug

commit sha e26bc916c330fec127ca250fc065c6ffb56f766c

encryption: move encryption config to under security config (#7800) Signed-off-by: Yi Wu <yiwu@pingcap.com>

view details

Nick Cameron

commit sha ffea7ec19bfd8495c2d7fff4371aa49d51e0a3d0

storage,config: check size of lock cf when considering large CFs (#7676) Signed-off-by: Nick Cameron <nrc@ncameron.org> Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

Lei Zhao

commit sha 9eb70c42397ad04d88b705d10f4d576bc1b401f9

storage: add tests for lock manager (#7663) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

Yilin Chen

commit sha cc9ac0d176e66bdcd86c7a123eac60c3e647d9c4

*: use given task id or random number as task id (#7521) Signed-off-by: Yilin Chen <sticnarf@gmail.com> Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

yiwu-arbug

commit sha 3adc3dab0b445efddeea4dd678eae1d2733215e5

encryption: Fix errors related to file ingestion (#7806) Signed-off-by: Yi Wu <yiwu@pingcap.com>

view details

NingLin-P

commit sha 0649dc17e0edfea8619ffac958cf6f4b0c45f72a

config: fix write config change to config file (#7795) Signed-off-by: linning <linningde25@gmail.com>

view details

Shuyang Wu

commit sha 87cada4de668ddf6f5330ea188d80ab703554908

Fix broken URL for "how-to/deploy" (#7809) Signed-off-by: Yiyiyimu <wosoyoung@gmail.com>

view details

WT

commit sha 58e5b3a1f39bf6473b06ed41c3371b64c7c374be

update community meeting with video links (#7783) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

qupeng

commit sha 4d39c5ecd7b3553de44c02c3f3b92ff714c2c37e

raft client: make grpc message size limit configurable (#7816) Signed-off-by: qupeng <qupeng@pingcap.com>

view details

TXXT

commit sha 18675bf7f8c6d06ee55dcafdb11c89bb596f2152

update grpcio version (#7819) Signed-off-by: TXXT <hunterlxt@live.com> Co-authored-by: qupeng <qupeng@pingcap.com>

view details

Lei Zhao

commit sha 3a1002b5ee9c943bad0ed70ad9620b3d354bd742

storage: add tests for pipelined pessimistic lock (#7706) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

TXXT

commit sha 6e33e71c6948f4c29b932ee428fd5f9e6ae51664

encryption: Update grafana json file to add encryption panel (#7754) * update json file Signed-off-by: TXXT <hunterlxt@live.com>

view details

push time in 12 days

PR opened tikv/tikv

engine_traits: Add some doc comments

What problem does this PR solve?

Add some API docs to engine_traits.

Related changes

https://github.com/tikv/tikv/issues/6402

Check List <!--REMOVE the items that are not applicable-->

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+211 -76

0 comment

5 changed files

pr created time in 12 days

push eventbrson/tikv

Wallace

commit sha a935b3b01479c1a7fa51d3b28e11ab0c9a21ea99

batch multiple write request into an entry (#6600) (#6683) * batch multiple write request into an entry (#6600) * batch multiple write request into an entry Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix name Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * refactor code for better to understand Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * add metrics Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * change metric of batch request to local metric Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix lint Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix wake up Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * wake up when batch entry Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * move code to fsm/peer.rs Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * not batch delete range Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix write batch size Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * remove unused variable Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix conflict of trait Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

view details

Jay

commit sha d10871fc80eef53a11fef4620dc5859cfc7319e8

tests: check tombstone safety (#7476) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

Hexilee

commit sha 817445dbb73edab2af7a49618df9161740deb611

status_server: support dumping region meta by http (#7650) Signed-off-by: Hexilee <i@hexilee.me>

view details

Alex Chi

commit sha 5b3b1e0ee454876ca08033aa4d06ad3df4cf38d7

copr: Add vectorized FromBase64 (#7767) Signed-off-by: Alex Chi <iskyzh@gmail.com>

view details

zhangjinpeng1987

commit sha 9e83b1045768e4421664d83ecdbce339eb6656cb

github action: assign bugs and questions to project 38 (#7752) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Shirly <AndreMouche@126.com>

view details

Wenxuan

commit sha 7f6620f9110d4fad57024ab77c3993da45323fae

Improve memory defragmentation (#7788) Signed-off-by: Breezewish <breezewish@pingcap.com>

view details

MyonKeminta

commit sha 8a6aa40512591d79250312840def9230f6db8e12

*: Remove incorrect alert rule (#7787) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

Connor

commit sha 39b243217f5fe2c3f37f9283350b89ccb11f6238

engine: Add titan customized statistics (#6328) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

view details

yiwu-arbug

commit sha e26bc916c330fec127ca250fc065c6ffb56f766c

encryption: move encryption config to under security config (#7800) Signed-off-by: Yi Wu <yiwu@pingcap.com>

view details

Nick Cameron

commit sha ffea7ec19bfd8495c2d7fff4371aa49d51e0a3d0

storage,config: check size of lock cf when considering large CFs (#7676) Signed-off-by: Nick Cameron <nrc@ncameron.org> Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

Lei Zhao

commit sha 9eb70c42397ad04d88b705d10f4d576bc1b401f9

storage: add tests for lock manager (#7663) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

Yilin Chen

commit sha cc9ac0d176e66bdcd86c7a123eac60c3e647d9c4

*: use given task id or random number as task id (#7521) Signed-off-by: Yilin Chen <sticnarf@gmail.com> Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

yiwu-arbug

commit sha 3adc3dab0b445efddeea4dd678eae1d2733215e5

encryption: Fix errors related to file ingestion (#7806) Signed-off-by: Yi Wu <yiwu@pingcap.com>

view details

NingLin-P

commit sha 0649dc17e0edfea8619ffac958cf6f4b0c45f72a

config: fix write config change to config file (#7795) Signed-off-by: linning <linningde25@gmail.com>

view details

Shuyang Wu

commit sha 87cada4de668ddf6f5330ea188d80ab703554908

Fix broken URL for "how-to/deploy" (#7809) Signed-off-by: Yiyiyimu <wosoyoung@gmail.com>

view details

WT

commit sha 58e5b3a1f39bf6473b06ed41c3371b64c7c374be

update community meeting with video links (#7783) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

qupeng

commit sha 4d39c5ecd7b3553de44c02c3f3b92ff714c2c37e

raft client: make grpc message size limit configurable (#7816) Signed-off-by: qupeng <qupeng@pingcap.com>

view details

TXXT

commit sha 18675bf7f8c6d06ee55dcafdb11c89bb596f2152

update grpcio version (#7819) Signed-off-by: TXXT <hunterlxt@live.com> Co-authored-by: qupeng <qupeng@pingcap.com>

view details

Lei Zhao

commit sha 3a1002b5ee9c943bad0ed70ad9620b3d354bd742

storage: add tests for pipelined pessimistic lock (#7706) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

TXXT

commit sha 6e33e71c6948f4c29b932ee428fd5f9e6ae51664

encryption: Update grafana json file to add encryption panel (#7754) * update json file Signed-off-by: TXXT <hunterlxt@live.com>

view details

push time in 12 days

pull request commenttikv/tikv

Remove `engine::iterable`

OK, the bug should be fixed. I was using rocks iterators incorrectly. I understand them better now.

brson

comment created time in 12 days

push eventbrson/tikv

Connor

commit sha 39b243217f5fe2c3f37f9283350b89ccb11f6238

engine: Add titan customized statistics (#6328) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

view details

yiwu-arbug

commit sha e26bc916c330fec127ca250fc065c6ffb56f766c

encryption: move encryption config to under security config (#7800) Signed-off-by: Yi Wu <yiwu@pingcap.com>

view details

Nick Cameron

commit sha ffea7ec19bfd8495c2d7fff4371aa49d51e0a3d0

storage,config: check size of lock cf when considering large CFs (#7676) Signed-off-by: Nick Cameron <nrc@ncameron.org> Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

Lei Zhao

commit sha 9eb70c42397ad04d88b705d10f4d576bc1b401f9

storage: add tests for lock manager (#7663) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

Yilin Chen

commit sha cc9ac0d176e66bdcd86c7a123eac60c3e647d9c4

*: use given task id or random number as task id (#7521) Signed-off-by: Yilin Chen <sticnarf@gmail.com> Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

yiwu-arbug

commit sha 3adc3dab0b445efddeea4dd678eae1d2733215e5

encryption: Fix errors related to file ingestion (#7806) Signed-off-by: Yi Wu <yiwu@pingcap.com>

view details

NingLin-P

commit sha 0649dc17e0edfea8619ffac958cf6f4b0c45f72a

config: fix write config change to config file (#7795) Signed-off-by: linning <linningde25@gmail.com>

view details

Shuyang Wu

commit sha 87cada4de668ddf6f5330ea188d80ab703554908

Fix broken URL for "how-to/deploy" (#7809) Signed-off-by: Yiyiyimu <wosoyoung@gmail.com>

view details

WT

commit sha 58e5b3a1f39bf6473b06ed41c3371b64c7c374be

update community meeting with video links (#7783) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

qupeng

commit sha 4d39c5ecd7b3553de44c02c3f3b92ff714c2c37e

raft client: make grpc message size limit configurable (#7816) Signed-off-by: qupeng <qupeng@pingcap.com>

view details

TXXT

commit sha 18675bf7f8c6d06ee55dcafdb11c89bb596f2152

update grpcio version (#7819) Signed-off-by: TXXT <hunterlxt@live.com> Co-authored-by: qupeng <qupeng@pingcap.com>

view details

Lei Zhao

commit sha 3a1002b5ee9c943bad0ed70ad9620b3d354bd742

storage: add tests for pipelined pessimistic lock (#7706) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

TXXT

commit sha 6e33e71c6948f4c29b932ee428fd5f9e6ae51664

encryption: Update grafana json file to add encryption panel (#7754) * update json file Signed-off-by: TXXT <hunterlxt@live.com>

view details

WT

commit sha 13d138f7d70b0e9ed68bdee5c4b7fe20b9773331

Update- security (#7820) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

Jay

commit sha 9ff03c1980bb379ea727836aa0ccca3792594d45

raftstore: change replication mode online (#7657) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

loxp

commit sha 61782edfae2eeb0f024fa6fde03c431ea0fef94a

copr: add vectorized truncate_int (#7813) Signed-off-by: loxp <loxppqqxol@gmail.com>

view details

zhangjinpeng1987

commit sha 007556d3bc0ae058d08caed39e71c4ddd4de4195

add edition field for the version information (#7821) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

view details

5kbpers

commit sha 53088c1286d6a14e4fefaf82d3c71a5162fca31e

raftstore: mark peer as pending remove after destroying it (#7492) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

Ping Yu

commit sha ffb73ae6218dc2f00f2be49acfae0c9734ea02e3

copr: Add vectorize Trim3Args (#7195) Signed-off-by: pingyu <shui.yu@126.com>

view details

5kbpers

commit sha 015d7ac0cebc6343f5efc3b0a3b65621cdae7392

raftstore: change yield condition into duration (#7763) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

push time in 12 days

pull request commenttikv/tikv

Remove `engine::iterable`

Trying to figure out what I broke here.

brson

comment created time in 13 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 13 days

push eventbrson/tikv

kennytm

commit sha 1bdab7bf652b1663d707725decc15114cf7d4c5a

external_storage: fix GCS download error, support GCS endpoints, and refactoring (#7734) Signed-off-by: kennytm <kennytm@gmail.com>

view details

5kbpers

commit sha 82d180d120e115e69512ea7f944e93e6dc5022a0

*: upgrade rust version (#7716) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

glorv

commit sha 039406403d554988c56fc0241ecefb93434e389d

build: remove unstable build option -Zconfig-profile (#7743) Signed-off-by: glorv <glorvs@163.com>

view details

Jay

commit sha cee0f390c18a6145d19f26b4f6b97d16d38dae7e

tests: make read_on_replica stable (#7744) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

Fu Chen

commit sha eeecc09ac261961216c8449969d12410c4f52e90

add a new versioncf for VerKV (#7607) Signed-off-by: fredchenbj <cfworking@163.com>

view details

pingcap-github-bot

commit sha fed97edf793fed736eb9cee30930b490db7a4db4

rust-rocksdb: titan: Add interface MultiBatchWrite for TitanDB (#7750) Signed-off-by: sre-bot <sre-bot@pingcap.com>

view details

Rustin

commit sha 98c58d98df8c2711de0e99f707c58984a7c16d05

raftstore: add RAFT_ENTRIES_CACHES_GAUGE metric (#7671) Signed-off-by: Rustin-liu <rustin.liu@gmail.com>

view details

Nick Cameron

commit sha dea16c8d4e6a36f3247399cead2758515e30372d

*: Move from our fork of sysinfo to upstream and get test suite passing on AMD cpus (#7682) * server: move diagnostics to its own directory in service Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

Brian Anderson

commit sha c7ded910fbf46338c6b8fa7abdd0d8a8fa461bd3

Fix some future clippy warnings (#7729) Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

Jay

commit sha 957655cacab1bd48100bf0b252fa93a9758848dc

raftstore: support dr auto sync (#7648) Support initializing DrAutoSync replication mode. In such case, logs will not be committed before they are replicated to at least two nodes with different label values. It utilizes raft's group commit algorithm under the hook. Dynamically change will be supported in next PR. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

zhangjinpeng1987

commit sha 2d6acde569ec1246b5baa6dcda70fbafab1ba886

Assign issues to SIG project automatically (#7764)

view details

Zhongyang Wu

commit sha 464164ddd52a4c6b8708e5c87b7721cf8aee8083

UCP: use key only support of Titan for rawkv scan (#7673) Signed-off-by: zhongyang.wu <zhongyang.wu@outlook.com>

view details

Wallace

commit sha a935b3b01479c1a7fa51d3b28e11ab0c9a21ea99

batch multiple write request into an entry (#6600) (#6683) * batch multiple write request into an entry (#6600) * batch multiple write request into an entry Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix name Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * refactor code for better to understand Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * add metrics Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * change metric of batch request to local metric Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix lint Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix wake up Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * wake up when batch entry Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * move code to fsm/peer.rs Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * not batch delete range Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix write batch size Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * remove unused variable Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix conflict of trait Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

view details

Jay

commit sha d10871fc80eef53a11fef4620dc5859cfc7319e8

tests: check tombstone safety (#7476) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

Hexilee

commit sha 817445dbb73edab2af7a49618df9161740deb611

status_server: support dumping region meta by http (#7650) Signed-off-by: Hexilee <i@hexilee.me>

view details

Alex Chi

commit sha 5b3b1e0ee454876ca08033aa4d06ad3df4cf38d7

copr: Add vectorized FromBase64 (#7767) Signed-off-by: Alex Chi <iskyzh@gmail.com>

view details

zhangjinpeng1987

commit sha 9e83b1045768e4421664d83ecdbce339eb6656cb

github action: assign bugs and questions to project 38 (#7752) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Shirly <AndreMouche@126.com>

view details

Wenxuan

commit sha 7f6620f9110d4fad57024ab77c3993da45323fae

Improve memory defragmentation (#7788) Signed-off-by: Breezewish <breezewish@pingcap.com>

view details

MyonKeminta

commit sha 8a6aa40512591d79250312840def9230f6db8e12

*: Remove incorrect alert rule (#7787) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

Connor

commit sha 39b243217f5fe2c3f37f9283350b89ccb11f6238

engine: Add titan customized statistics (#6328) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

view details

push time in 13 days

pull request commenttikv/tikv

Move `engine::stats` into `engine_traits`

Rebased and fixed rustfmt errors.

brson

comment created time in 13 days

push eventbrson/tikv

Connor

commit sha 39b243217f5fe2c3f37f9283350b89ccb11f6238

engine: Add titan customized statistics (#6328) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

view details

yiwu-arbug

commit sha e26bc916c330fec127ca250fc065c6ffb56f766c

encryption: move encryption config to under security config (#7800) Signed-off-by: Yi Wu <yiwu@pingcap.com>

view details

Nick Cameron

commit sha ffea7ec19bfd8495c2d7fff4371aa49d51e0a3d0

storage,config: check size of lock cf when considering large CFs (#7676) Signed-off-by: Nick Cameron <nrc@ncameron.org> Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

Lei Zhao

commit sha 9eb70c42397ad04d88b705d10f4d576bc1b401f9

storage: add tests for lock manager (#7663) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

Yilin Chen

commit sha cc9ac0d176e66bdcd86c7a123eac60c3e647d9c4

*: use given task id or random number as task id (#7521) Signed-off-by: Yilin Chen <sticnarf@gmail.com> Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

yiwu-arbug

commit sha 3adc3dab0b445efddeea4dd678eae1d2733215e5

encryption: Fix errors related to file ingestion (#7806) Signed-off-by: Yi Wu <yiwu@pingcap.com>

view details

NingLin-P

commit sha 0649dc17e0edfea8619ffac958cf6f4b0c45f72a

config: fix write config change to config file (#7795) Signed-off-by: linning <linningde25@gmail.com>

view details

Shuyang Wu

commit sha 87cada4de668ddf6f5330ea188d80ab703554908

Fix broken URL for "how-to/deploy" (#7809) Signed-off-by: Yiyiyimu <wosoyoung@gmail.com>

view details

WT

commit sha 58e5b3a1f39bf6473b06ed41c3371b64c7c374be

update community meeting with video links (#7783) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

qupeng

commit sha 4d39c5ecd7b3553de44c02c3f3b92ff714c2c37e

raft client: make grpc message size limit configurable (#7816) Signed-off-by: qupeng <qupeng@pingcap.com>

view details

TXXT

commit sha 18675bf7f8c6d06ee55dcafdb11c89bb596f2152

update grpcio version (#7819) Signed-off-by: TXXT <hunterlxt@live.com> Co-authored-by: qupeng <qupeng@pingcap.com>

view details

Lei Zhao

commit sha 3a1002b5ee9c943bad0ed70ad9620b3d354bd742

storage: add tests for pipelined pessimistic lock (#7706) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

TXXT

commit sha 6e33e71c6948f4c29b932ee428fd5f9e6ae51664

encryption: Update grafana json file to add encryption panel (#7754) * update json file Signed-off-by: TXXT <hunterlxt@live.com>

view details

WT

commit sha 13d138f7d70b0e9ed68bdee5c4b7fe20b9773331

Update- security (#7820) Signed-off-by: Wenting Liu <liuwenting@pingcap.com>

view details

Jay

commit sha 9ff03c1980bb379ea727836aa0ccca3792594d45

raftstore: change replication mode online (#7657) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

loxp

commit sha 61782edfae2eeb0f024fa6fde03c431ea0fef94a

copr: add vectorized truncate_int (#7813) Signed-off-by: loxp <loxppqqxol@gmail.com>

view details

zhangjinpeng1987

commit sha 007556d3bc0ae058d08caed39e71c4ddd4de4195

add edition field for the version information (#7821) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

view details

5kbpers

commit sha 53088c1286d6a14e4fefaf82d3c71a5162fca31e

raftstore: mark peer as pending remove after destroying it (#7492) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

Ping Yu

commit sha ffb73ae6218dc2f00f2be49acfae0c9734ea02e3

copr: Add vectorize Trim3Args (#7195) Signed-off-by: pingyu <shui.yu@126.com>

view details

5kbpers

commit sha 015d7ac0cebc6343f5efc3b0a3b65621cdae7392

raftstore: change yield condition into duration (#7763) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

push time in 13 days

pull request commenttikv/tikv

Cherry-pick two UB fixes into 4.0

Fixed merge conflict.

brson

comment created time in 13 days

push eventbrson/tikv

Fullstop000

commit sha 911bb62c9ba3a0a651c235d76db9437f3d339b91

Cherry-pick #7085 #7273 #7541 #7437 (#7776) Signed-off-by: Fullstop000 <fullstop1005@gmail.com>

view details

Wenxuan

commit sha c5721d64686f23ea635374479aef2858a743b19e

[release-4.0] *: reduce sys_getdents syscall (#7306) (#7556) Signed-off-by: Breezewish <breezewish@pingcap.com>

view details

Shenghui Wu

commit sha bf6ba902ed0b84dd2cba9e99bc18c4cc8bc827fa

copr: shouldn't check invalid dates when using chunk format. (#7218) (#7268) Signed-off-by: wshwsh12 <793703860@qq.com>

view details

pingcap-github-bot

commit sha 98111ae00df42a41cd96a7b5a31d1853a691d656

*: Remove incorrect alert rule about coprocessor wait second (#7787) (#7793) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

view details

pingcap-github-bot

commit sha 4a8d6d6c2bdf29613016be22c1ad4cc4d085ed1b

external_storage: fix GCS download error, support GCS endpoints, and refactoring (#7734) (#7739) Signed-off-by: sre-bot <sre-bot@pingcap.com>

view details

pingcap-github-bot

commit sha 8919c97cb9bbd1b99c3031c691c33610f563a659

encryption: validate KMS key id (#7665) (#7719) Signed-off-by: Neil Shen <overvenus@gmail.com>

view details

pingcap-github-bot

commit sha 19690cbf670c57a645a845488008e73da0107942

tidb_query_vec_executors: Fixed stack-borrowing undefined-behavior (#7709) (#7714) Signed-off-by: Brian Anderson <andersrb@gmail.com> Co-authored-by: Brian Anderson <andersrb@gmail.com>

view details

pingcap-github-bot

commit sha 19d5c3ebacef8df66a2612aed8f03da308410a00

encryption: move encryption config to under security config (#7800) (#7810) Signed-off-by: Yi Wu <yiwu@pingcap.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com>

view details

pingcap-github-bot

commit sha 9ff1330b013b2e340bdea3e704b35cc4612eb811

encryption: Fix errors related to file ingestion (#7806) (#7815) Signed-off-by: Yi Wu <yiwu@pingcap.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com>

view details

pingcap-github-bot

commit sha 9e7fd5499a4af14e360d37a9108cd4beea6163ea

config: fix write config change to config file (#7795) (#7817) Signed-off-by: linning <linningde25@gmail.com>

view details

pingcap-github-bot

commit sha 7884da7cefbc295c97703c205489602b9d6ae343

raft client: make grpc message size limit configurable (#7816) (#7824) Signed-off-by: qupeng <qupeng@pingcap.com>

view details

TXXT

commit sha 4834c5716cd2fd5d1fe8c7871153b5dfbf2bf6a2

Add encryption panel into grafana template (#7827) Signed-off-by: TXXT <hunterlxt@live.com>

view details

pingcap-github-bot

commit sha 642ffbb8b284b602b37607f16e37369d4fb5dbc1

Update grpcio to enable ALPN (#7819) (#7825) Signed-off-by: TXXT <hunterlxt@live.com>

view details

pingcap-github-bot

commit sha d7525fac92484417a18dcfc79aa4966e25629a03

add edition field for the version information (#7821) (#7831) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

view details

Connor

commit sha f69ee269e27fcf8b4a5aa6e5a56ab69ad7938c0e

engine: Add titan customized statistics (#6328) (#7818) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Connor <zbk602423539@gmail.com>

view details

pingcap-github-bot

commit sha 535b2fdbe9ff380a688075ddbdc11b0de7aea3eb

storage,config: check size of lock cf when considering large CFs (#7676) (#7812) Signed-off-by: Nick Cameron <nrc@ncameron.org> Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

view details

pingcap-github-bot

commit sha 234cdda54b16bff3feae77625c05d7c927288b56

tidb_query: fix UB causing by `from_raw_parts` (#7635) (#7730) Signed-off-by: zhongzc <zhongzc_arch@outlook.com>

view details

pingcap-github-bot

commit sha 2e06572e4c3dc59d114db09205dfd78f680579ba

*: use given task id or random number as task id (#7521) (#7814) Signed-off-by: sre-bot <sre-bot@pingcap.com> Signed-off-by: Yilin Chen <sticnarf@gmail.com>

view details

Wallace

commit sha 981ca2cbe66131424a2bc9d51c8f5af84ab228fa

raftstore: Use multi-batch-write for apply (#7111) (#7718) Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

view details

pingcap-github-bot

commit sha 035a1bf6db9cae406ac9ff6c3fe68f67797fb4d4

raftstore: mark peer as pending remove after destroying it (#7492) (#7836) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

push time in 13 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 13 days

push eventbrson/rust-doc-tool

Brian Anderson

commit sha f90d7bb5337f28ce1f30c717fcf4bb3a820531ab

.

view details

push time in 15 days

push eventbrson/rust-anthology

Brian Anderson

commit sha e0e067a286650184a8dd44e55461197cceaa55bb

.

view details

push time in 15 days

PR opened tikv/tikv

Reviewers
Move `engine::stats` into `engine_traits`

What problem does this PR solve?

This moves the dump function into the engine_traits crate and deletes it from the engine crate. Part of engine abstraction.

Related changes

https://github.com/tikv/tikv/issues/6402

Check List <!--REMOVE the items that are not applicable-->

cargo test --no-run

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+43 -40

0 comment

7 changed files

pr created time in 15 days

pull request commenttikv/tikv

Remove `engine::iterable`

Looks like the CI failures are real.

brson

comment created time in 15 days

create barnchbrson/tikv

branch : rm-engine-stats

created branch time in 15 days

PR opened tikv/tikv

Reviewers
Remove `engine::iterable`

What problem does this PR solve?

This removes the iterable module from the engine crate, which is being removed entirely in favor of engine_rocks.

In order to do this I converted src/server/debug to consistently use the RocksEngineIterator type. I believe the resulting iterator behavior should be identical, but don't know how to test manually.

Related changes

https://github.com/tikv/tikv/issues/6402

Check List <!--REMOVE the items that are not applicable-->

Type checking

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+120 -138

0 comment

4 changed files

pr created time in 15 days

create barnchbrson/tikv

branch : rm-engine-iterable

created branch time in 15 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

I assume the build failures are not from this patch.

brson

comment created time in 16 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 16 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 16 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 16 days

pull request commenttikv/tikv

Fix two instances of undefined behavior in codec crate

/merge

brson

comment created time in 16 days

push eventbrson/rust-anthology

Brian Anderson

commit sha fbef5dab4155eda0df217a2d48f896196c7727a8

.

view details

push time in 16 days

push eventbrson/rust-doc-tool

Brian Anderson

commit sha 1335b605a8b69bf9c235202f764d992bf97125d8

.

view details

push time in 16 days

push eventbrson/tikv

Brian Anderson

commit sha 6d60db4220f40ef1dced3f4455a85feb9931aef9

. Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

Brian Anderson

commit sha 6b8e458c191958e9436c14f3b02bd7a9313a7e02

*: fix build Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

Brian Anderson

commit sha 0367db35f73c7fd8ff8dd6378cf5be94c3f9ca63

tikv: Change KvEngine bounds to Snapshot Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

Brian Anderson

commit sha 3dbd80d5df2a4d4d7fb3b0cdb19194f97f3195e0

*: rustfmt Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

Brian Anderson

commit sha 0e61a0c8056cfd8a78155d9013f3de68272941c8

raftstore: clippy Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

push time in 17 days

push eventbrson/rust-doc-tool

Brian Anderson

commit sha d90f1f65d666328a9d956dc40a858d5f2d297d21

.

view details

push time in 17 days

push eventbrson/rust-anthology

Brian Anderson

commit sha d0c7c4f4fd2c30bc31e1f4a7e914209311578a09

.

view details

push time in 17 days

create barnchbrson/tikv

branch : engine-traits-more-raftstore-generics-cleanup

created branch time in 17 days

push eventbrson/tikv

Brian Anderson

commit sha 3a9ddbb2ae70ebbabdc38c407fac4b98643b8a99

engine_traits: Add some doc comments Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

push time in 17 days

push eventbrson/tikv

Brian Anderson

commit sha 26ca170208ba3f4afd818b89a3dbce5a9d6fb5ed

raftstore: Replace KvEngine type parameters with Snapshot typarams (#7662) Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

kennytm

commit sha 1bdab7bf652b1663d707725decc15114cf7d4c5a

external_storage: fix GCS download error, support GCS endpoints, and refactoring (#7734) Signed-off-by: kennytm <kennytm@gmail.com>

view details

5kbpers

commit sha 82d180d120e115e69512ea7f944e93e6dc5022a0

*: upgrade rust version (#7716) Signed-off-by: 5kbpers <tangminghua@pingcap.com>

view details

glorv

commit sha 039406403d554988c56fc0241ecefb93434e389d

build: remove unstable build option -Zconfig-profile (#7743) Signed-off-by: glorv <glorvs@163.com>

view details

Jay

commit sha cee0f390c18a6145d19f26b4f6b97d16d38dae7e

tests: make read_on_replica stable (#7744) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

Fu Chen

commit sha eeecc09ac261961216c8449969d12410c4f52e90

add a new versioncf for VerKV (#7607) Signed-off-by: fredchenbj <cfworking@163.com>

view details

pingcap-github-bot

commit sha fed97edf793fed736eb9cee30930b490db7a4db4

rust-rocksdb: titan: Add interface MultiBatchWrite for TitanDB (#7750) Signed-off-by: sre-bot <sre-bot@pingcap.com>

view details

Rustin

commit sha 98c58d98df8c2711de0e99f707c58984a7c16d05

raftstore: add RAFT_ENTRIES_CACHES_GAUGE metric (#7671) Signed-off-by: Rustin-liu <rustin.liu@gmail.com>

view details

Nick Cameron

commit sha dea16c8d4e6a36f3247399cead2758515e30372d

*: Move from our fork of sysinfo to upstream and get test suite passing on AMD cpus (#7682) * server: move diagnostics to its own directory in service Signed-off-by: Nick Cameron <nrc@ncameron.org>

view details

Brian Anderson

commit sha c7ded910fbf46338c6b8fa7abdd0d8a8fa461bd3

Fix some future clippy warnings (#7729) Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

Jay

commit sha 957655cacab1bd48100bf0b252fa93a9758848dc

raftstore: support dr auto sync (#7648) Support initializing DrAutoSync replication mode. In such case, logs will not be committed before they are replicated to at least two nodes with different label values. It utilizes raft's group commit algorithm under the hook. Dynamically change will be supported in next PR. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

view details

zhangjinpeng1987

commit sha 2d6acde569ec1246b5baa6dcda70fbafab1ba886

Assign issues to SIG project automatically (#7764)

view details

Zhongyang Wu

commit sha 464164ddd52a4c6b8708e5c87b7721cf8aee8083

UCP: use key only support of Titan for rawkv scan (#7673) Signed-off-by: zhongyang.wu <zhongyang.wu@outlook.com>

view details

Brian Anderson

commit sha d105e285af86cbee4ca2a0748f988b3af321a356

engine_traits: Add some doc comments Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

push time in 17 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 17 days

pull request commenttikv/tikv

Fix two instances of undefined behavior in codec crate

/merge

brson

comment created time in 17 days

push eventbrson/my-rust-lists

Brian Anderson

commit sha b6ba52e1c3b984cccba162127869b4c6bd8ff25f

.

view details

push time in 17 days

PR opened tikv/tikv

Cherry-pick two UB fixes into 4.0

What problem does this PR solve?

Apply two fixes for undefined-behavior to the 4.0 branch.

Release note <!-- bugfixes or new feature need a release note -->

  • No release note.
+2 -21

0 comment

4 changed files

pr created time in 17 days

pull request commenttikv/tikv

Fix two instances of undefined behavior in codec crate

/merge

brson

comment created time in 17 days

create barnchbrson/tikv

branch : ub-40

created branch time in 17 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 17 days

Pull request review commentpingcap-incubator/tiup

Manifest and server organisation design doc

+# Manifest format and repository layout++See [#120](https://github.com/pingcap-incubator/tiup/issues/120).++## The Update Framework++[The Update Framework (TUF)](https://theupdateframework.io/overview/) is the state of the art for update systems. It has a strong focus on security.++TUF is a [specification](https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification) for building update systems, not an implementation (though reference implementations exist). It does not fully specify how such a system should work, e.g., it does not discuss how updates are applied or how servers are managed. However, it defines what metadata should be kept, how data and metadata should be signed, and how to ensure downloads are secure.++We build on and adapt TUF, all places where we deviate from TUF are enumerated below. I believe we still fulfil all TUF security guarantees.+++## Components++A component is a program which can be run by TiUp. In this document, we do not consider what constitutes a component, for our purposes it is an opaque file.++A component may have many versions. We do not discuss how versions are specified or applied, only that a client may specify a version to download.++A component may be supplied by PingCAP or a third party, we detail how components are *uploaded*, but how components are packaged and checked by the client is out of scope. We assume that a component is not 'opened', i.e., a component is uploaded to a server and later downloaded, the server does not inspect the contents of the component. Note that this design document only addresses part of the security issue with handling third party code or binaries - we ensure that the code that a user downloads is the same code that the publisher uploads, however, we cannot ensure that the code uploaded is not malicious.++We do not support channels or groupings of components. We expect components will support something here, but it will be up to the client to implement.++TiUp itself can be treated as a component and I think does not need to be treated any differently on the server side.++## Other terms++* repository: a source of components and manifests (aka server, mirror)+* manifest: a JSON metadata file+* platform: the hardware platform and operating system where a binary will run (c.f., built).+++## Principles++* URLs are persistent.+* All data is immutable+* Most metadata is immutable, achieved using versioning.+* Components cannot be removed or deleted.+* Zero-trust access control.+* Minimise the number of keys and the frequency with which they must be replaced.+* Uploads and downloads should be secure.+* All operations should be idempotent or at least repeatable. If an operation is interupted, the user can just retry until it succeeds.+* There should be no 'transaction' mechanism between client and server, groups of operations should not require atomicity.+* Minimise required knowledge in the client - the metadata should direct the client how to perform operations with minimal knowledge of the server's data layout, etc.+* We should not require consistency between files in the repo (beyond hashes and keys).+++## Assumptions++* Updates are per-component, updating all components is interpreted by the client.+* There is a secure channel to distribute initial and updated keys for each server+* Don't need to change the owner of a component.+* Don't need to delegate ownership of a component (owners can delegate themselves with client-side tooling).+++## Repository organisation++We require snapshot.json and root.json to exist at the root of the repository. There will be many n.root.json files, these must also be at the repository root.++The rest of the repository organisation is here for completeness but can be changed freely since all other locations are specified by URLs.++n.index.json files and snapshot.json are at the root of the repository.++For each component there is a directory named with the component's id. Within this directory are the n.id.json files (where id is the id of the component; alternative: all these manifests could be called n.component.json). Within the component directory, there is a sub-directory for each platform named with the target triple. All data files are within these platform subdirectories. Each file has the form version.platform.id.tar.gz for tarballs and version.platform.id.ext or version.platform.id for other single files, where id is the component id, platform is a target triple, and version is a semver version containing exactly two periods (`.`, e.g., `1.0.0`).

There are conceivably platformless components too. Consider how a component with no target-triple would fit into this scheme.

nrc

comment created time in 17 days

Pull request review commentpingcap-incubator/tiup

Manifest and server organisation design doc

+# Manifest format and repository layout++See [#120](https://github.com/pingcap-incubator/tiup/issues/120).++## The Update Framework++[The Update Framework (TUF)](https://theupdateframework.io/overview/) is the state of the art for update systems. It has a strong focus on security.++TUF is a [specification](https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification) for building update systems, not an implementation (though reference implementations exist). It does not fully specify how such a system should work, e.g., it does not discuss how updates are applied or how servers are managed. However, it defines what metadata should be kept, how data and metadata should be signed, and how to ensure downloads are secure.++We build on and adapt TUF, all places where we deviate from TUF are enumerated below. I believe we still fulfil all TUF security guarantees.+++## Components++A component is a program which can be run by TiUp. In this document, we do not consider what constitutes a component, for our purposes it is an opaque file.++A component may have many versions. We do not discuss how versions are specified or applied, only that a client may specify a version to download.++A component may be supplied by PingCAP or a third party, we detail how components are *uploaded*, but how components are packaged and checked by the client is out of scope. We assume that a component is not 'opened', i.e., a component is uploaded to a server and later downloaded, the server does not inspect the contents of the component. Note that this design document only addresses part of the security issue with handling third party code or binaries - we ensure that the code that a user downloads is the same code that the publisher uploads, however, we cannot ensure that the code uploaded is not malicious.++We do not support channels or groupings of components. We expect components will support something here, but it will be up to the client to implement.++TiUp itself can be treated as a component and I think does not need to be treated any differently on the server side.++## Other terms++* repository: a source of components and manifests (aka server, mirror)+* manifest: a JSON metadata file+* platform: the hardware platform and operating system where a binary will run (c.f., built).+++## Principles++* URLs are persistent.+* All data is immutable+* Most metadata is immutable, achieved using versioning.+* Components cannot be removed or deleted.+* Zero-trust access control.+* Minimise the number of keys and the frequency with which they must be replaced.+* Uploads and downloads should be secure.+* All operations should be idempotent or at least repeatable. If an operation is interupted, the user can just retry until it succeeds.+* There should be no 'transaction' mechanism between client and server, groups of operations should not require atomicity.+* Minimise required knowledge in the client - the metadata should direct the client how to perform operations with minimal knowledge of the server's data layout, etc.+* We should not require consistency between files in the repo (beyond hashes and keys).+++## Assumptions++* Updates are per-component, updating all components is interpreted by the client.+* There is a secure channel to distribute initial and updated keys for each server+* Don't need to change the owner of a component.+* Don't need to delegate ownership of a component (owners can delegate themselves with client-side tooling).+++## Repository organisation++We require snapshot.json and root.json to exist at the root of the repository. There will be many n.root.json files, these must also be at the repository root.++The rest of the repository organisation is here for completeness but can be changed freely since all other locations are specified by URLs.++n.index.json files and snapshot.json are at the root of the repository.++For each component there is a directory named with the component's id. Within this directory are the n.id.json files (where id is the id of the component; alternative: all these manifests could be called n.component.json). Within the component directory, there is a sub-directory for each platform named with the target triple. All data files are within these platform subdirectories. Each file has the form version.platform.id.tar.gz for tarballs and version.platform.id.ext or version.platform.id for other single files, where id is the component id, platform is a target triple, and version is a semver version containing exactly two periods (`.`, e.g., `1.0.0`).+++## Manifests++Each manifest is a [canonical json](http://wiki.laptop.org/go/Canonical_JSON) file with a `.json` extension.++Manifest format:++```+{+    "signatures": [+        {+            "keyid": "",+            "sig": "",+        }+    ],+    "signed": {+        "_type": "TY",+        "spec_version": "",+        "expires": "TS",+        // version is only present for versioned manifests.+        "version": n,+        ROLE+    },+}+```++where++* `""` is a string.+* `n` is an integer > 0.+* `"TS"` is a timestamp (date and time) in ISO 8601 format; timezone should always be `Z`.+* `"TY"` is one of `"root"`, `"index"`, `"component"`, `"snapshot"`, or `"timestamp"`.+* `ROLE` is specific to the type of manifest.+* The value of `"spec_version"` should be valid semver.++The timestamp and snapshot manifests should expire after one day. All other manifests should expire after one year.++URLs are relative to the repository root, which allows a repository to be cloned to a new location without changing URLs.++The spec version should be the same for all files in a given snapshot, if the client finds a spec version which is inconsistent, or that the client cannot interpret, it should abort its operation.++### Keys++A key definition is:++```json+"KEYID": {+    "keyid_hash_algorithms": [+        "sha256",+        "sha512",+    ],+    "keytype": "ed25519",+    "keyval": {+        "public": "KEY",+    },+    "scheme": "ed25519",+},+```++where++* `"KEYID"` is a globally unique identifier (i.e., keys must not share key ids); it is the hexdigest of the SHA-256 hash of the canonical JSON form of the key.+* We support `"sha256"` and `"sha512"` hashing (only SHA256 in first iteration).+* We support `"rsa"`, `"ed25519"`, and `"ecdsa-sha2-nistp256"` key types (only `"rsa"` in first iteration).+* We support `"rsassa-pss-sha256"`, `"ed25519"`, and `"ecdsa-sha2-nistp256"` schemes (only `"rsassa-pss-sha256"` in first iteration)+  - "rsassa-pss-sha256" : RSA Probabilistic signature scheme with appendix. The underlying hash function is SHA256. https://tools.ietf.org/html/rfc3447#page-29+  - "ed25519" : Elliptic curve digital signature algorithm based on Twisted Edwards curves. https://ed25519.cr.yp.to/+  - "ecdsa-sha2-nistp256" : Elliptic Curve Digital Signature Algorithm with NIST P-256 curve signing and SHA-256 hashing. https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm

There's no need to claim to support all these options, unless perhaps we're developing this as a general-purpose library for others to use.

nrc

comment created time in 17 days

Pull request review commentpingcap-incubator/tiup

Manifest and server organisation design doc

+# Manifest format and repository layout++See [#120](https://github.com/pingcap-incubator/tiup/issues/120).++## The Update Framework++[The Update Framework (TUF)](https://theupdateframework.io/overview/) is the state of the art for update systems. It has a strong focus on security.++TUF is a [specification](https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification) for building update systems, not an implementation (though reference implementations exist). It does not fully specify how such a system should work, e.g., it does not discuss how updates are applied or how servers are managed. However, it defines what metadata should be kept, how data and metadata should be signed, and how to ensure downloads are secure.++We build on and adapt TUF, all places where we deviate from TUF are enumerated below. I believe we still fulfil all TUF security guarantees.+++## Components++A component is a program which can be run by TiUp. In this document, we do not consider what constitutes a component, for our purposes it is an opaque file.++A component may have many versions. We do not discuss how versions are specified or applied, only that a client may specify a version to download.++A component may be supplied by PingCAP or a third party, we detail how components are *uploaded*, but how components are packaged and checked by the client is out of scope. We assume that a component is not 'opened', i.e., a component is uploaded to a server and later downloaded, the server does not inspect the contents of the component. Note that this design document only addresses part of the security issue with handling third party code or binaries - we ensure that the code that a user downloads is the same code that the publisher uploads, however, we cannot ensure that the code uploaded is not malicious.++We do not support channels or groupings of components. We expect components will support something here, but it will be up to the client to implement.++TiUp itself can be treated as a component and I think does not need to be treated any differently on the server side.++## Other terms++* repository: a source of components and manifests (aka server, mirror)+* manifest: a JSON metadata file+* platform: the hardware platform and operating system where a binary will run (c.f., built).+++## Principles++* URLs are persistent.+* All data is immutable+* Most metadata is immutable, achieved using versioning.+* Components cannot be removed or deleted.+* Zero-trust access control.+* Minimise the number of keys and the frequency with which they must be replaced.+* Uploads and downloads should be secure.+* All operations should be idempotent or at least repeatable. If an operation is interupted, the user can just retry until it succeeds.+* There should be no 'transaction' mechanism between client and server, groups of operations should not require atomicity.+* Minimise required knowledge in the client - the metadata should direct the client how to perform operations with minimal knowledge of the server's data layout, etc.+* We should not require consistency between files in the repo (beyond hashes and keys).+++## Assumptions++* Updates are per-component, updating all components is interpreted by the client.+* There is a secure channel to distribute initial and updated keys for each server+* Don't need to change the owner of a component.+* Don't need to delegate ownership of a component (owners can delegate themselves with client-side tooling).+++## Repository organisation++We require snapshot.json and root.json to exist at the root of the repository. There will be many n.root.json files, these must also be at the repository root.++The rest of the repository organisation is here for completeness but can be changed freely since all other locations are specified by URLs.++n.index.json files and snapshot.json are at the root of the repository.++For each component there is a directory named with the component's id. Within this directory are the n.id.json files (where id is the id of the component; alternative: all these manifests could be called n.component.json). Within the component directory, there is a sub-directory for each platform named with the target triple. All data files are within these platform subdirectories. Each file has the form version.platform.id.tar.gz for tarballs and version.platform.id.ext or version.platform.id for other single files, where id is the component id, platform is a target triple, and version is a semver version containing exactly two periods (`.`, e.g., `1.0.0`).

Perhaps the version-first scheme is related to ease of parsing?

nrc

comment created time in 17 days

Pull request review commentpingcap-incubator/tiup

Manifest and server organisation design doc

+# Manifest format and repository layout++See [#120](https://github.com/pingcap-incubator/tiup/issues/120).++## The Update Framework++[The Update Framework (TUF)](https://theupdateframework.io/overview/) is the state of the art for update systems. It has a strong focus on security.++TUF is a [specification](https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#the-update-framework-specification) for building update systems, not an implementation (though reference implementations exist). It does not fully specify how such a system should work, e.g., it does not discuss how updates are applied or how servers are managed. However, it defines what metadata should be kept, how data and metadata should be signed, and how to ensure downloads are secure.++We build on and adapt TUF, all places where we deviate from TUF are enumerated below. I believe we still fulfil all TUF security guarantees.+++## Components++A component is a program which can be run by TiUp. In this document, we do not consider what constitutes a component, for our purposes it is an opaque file.++A component may have many versions. We do not discuss how versions are specified or applied, only that a client may specify a version to download.++A component may be supplied by PingCAP or a third party, we detail how components are *uploaded*, but how components are packaged and checked by the client is out of scope. We assume that a component is not 'opened', i.e., a component is uploaded to a server and later downloaded, the server does not inspect the contents of the component. Note that this design document only addresses part of the security issue with handling third party code or binaries - we ensure that the code that a user downloads is the same code that the publisher uploads, however, we cannot ensure that the code uploaded is not malicious.++We do not support channels or groupings of components. We expect components will support something here, but it will be up to the client to implement.++TiUp itself can be treated as a component and I think does not need to be treated any differently on the server side.++## Other terms++* repository: a source of components and manifests (aka server, mirror)+* manifest: a JSON metadata file+* platform: the hardware platform and operating system where a binary will run (c.f., built).+++## Principles++* URLs are persistent.+* All data is immutable+* Most metadata is immutable, achieved using versioning.+* Components cannot be removed or deleted.+* Zero-trust access control.+* Minimise the number of keys and the frequency with which they must be replaced.+* Uploads and downloads should be secure.+* All operations should be idempotent or at least repeatable. If an operation is interupted, the user can just retry until it succeeds.+* There should be no 'transaction' mechanism between client and server, groups of operations should not require atomicity.+* Minimise required knowledge in the client - the metadata should direct the client how to perform operations with minimal knowledge of the server's data layout, etc.+* We should not require consistency between files in the repo (beyond hashes and keys).+++## Assumptions++* Updates are per-component, updating all components is interpreted by the client.+* There is a secure channel to distribute initial and updated keys for each server+* Don't need to change the owner of a component.+* Don't need to delegate ownership of a component (owners can delegate themselves with client-side tooling).+++## Repository organisation++We require snapshot.json and root.json to exist at the root of the repository. There will be many n.root.json files, these must also be at the repository root.++The rest of the repository organisation is here for completeness but can be changed freely since all other locations are specified by URLs.++n.index.json files and snapshot.json are at the root of the repository.++For each component there is a directory named with the component's id. Within this directory are the n.id.json files (where id is the id of the component; alternative: all these manifests could be called n.component.json). Within the component directory, there is a sub-directory for each platform named with the target triple. All data files are within these platform subdirectories. Each file has the form version.platform.id.tar.gz for tarballs and version.platform.id.ext or version.platform.id for other single files, where id is the component id, platform is a target triple, and version is a semver version containing exactly two periods (`.`, e.g., `1.0.0`).

The version-first naming scheme seems very strange. Id-first provides a more natural sorted order. I can't think of an advantage to version-first. id.version.platform makes more sense to me.

The subdirectory-per platform is redundant with the file-naming scheme. That's not necessarily a problem, since it could e.g. aid human comprehension.

Some rationale for both these choices would help me.

nrc

comment created time in 17 days

pull request commentpingcap-incubator/tiup

Manifest and server organisation design doc

It looks good in general, simply by following tuf. I haven't carefully reviewed all the flows, but the parts I have read seem plausibly correct.

The lack of channels and grouping seems like a big problem, but from the language in the intro I guess you see that as something to layer on top of the base protocol. e.g. I might have a meta-component called that is nothing but a manifest of component-versions.

nrc

comment created time in 17 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 17 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 17 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/rebuild

brson

comment created time in 17 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 17 days

Pull request review commenttikv/tikv

Fix two instances of undefined behavior in codec crate

 impl MemComparableByteCodec {             let src_ptr_end = src_ptr.add(src_len);              loop {-                let src_ptr_next = src_ptr.add(MEMCMP_GROUP_SIZE + 1);-                if std::intrinsics::unlikely(src_ptr_next > src_ptr_end) {+                // Make sure the source buffer is the expected length before+                // constructing the next source pointer. Note that it is UB to+                // even _create_ a pointer that points beyond its allocation.+                let ptrdiff = src_ptr_end.offset_from(src_ptr);+                let ptroob = ptrdiff < MEMCMP_GROUP_SIZE as isize + 1;++                if std::intrinsics::unlikely(ptroob) {                     return Err(ErrorInner::eof().into());                 } +                let src_ptr_next = src_ptr.add(MEMCMP_GROUP_SIZE + 1);

Thanks for the help @RalfJung @oli-obk

brson

comment created time in 17 days

pull request commenttikv/tikv

Fix two instances of undefined behavior in codec crate

/rebuild

brson

comment created time in 17 days

Pull request review commenttikv/tikv

Fix two instances of undefined behavior in codec crate

 impl MemComparableByteCodec {     ///     /// Please refer to `try_decode_first` for the meaning of return values, panics and errors.     #[inline]-    fn try_decode_first_internal<T: MemComparableCodecHelper>(+    unsafe fn try_decode_first_internal<T: MemComparableCodecHelper>(

Done.

brson

comment created time in 17 days

push eventbrson/tikv

Brian Anderson

commit sha 868f1e3e1d828b1f66d4164ce5b91ee458a5a13d

codec: Add safety doc comment sto try_decode_first_internal Signed-off-by: Brian Anderson <andersrb@gmail.com>

view details

push time in 17 days

Pull request review commenttikv/tikv

Fix two instances of undefined behavior in codec crate

 impl MemComparableByteCodec {         let src_ptr_untouched = src_ptr;         let dest_ptr_untouched = dest_ptr; -        unsafe {-            let src_ptr_end = src_ptr.add(src_len);+        let src_ptr_end = src_ptr.add(src_len); -            loop {-                let src_ptr_next = src_ptr.add(MEMCMP_GROUP_SIZE + 1);-                if std::intrinsics::unlikely(src_ptr_next > src_ptr_end) {-                    return Err(ErrorInner::eof().into());-                }+        loop {+            // Make sure the source buffer is the expected length before+            // constructing the next source pointer. Note that it is UB to+            // even _create_ a pointer that points beyond its allocation.+            let ptrdiff = src_ptr_end.offset_from(src_ptr);+            let ptroob = ptrdiff < MEMCMP_GROUP_SIZE as isize + 1; -                // Copy `MEMCMP_GROUP_SIZE` bytes any way. However we will truncate the returned-                // length according to padding size if it is the last block.-                std::ptr::copy(src_ptr, dest_ptr, MEMCMP_GROUP_SIZE);+            if std::intrinsics::unlikely(ptroob) {+                return Err(ErrorInner::eof().into());+            } -                let padding_size = T::parse_padding_size(*src_ptr.add(MEMCMP_GROUP_SIZE));-                src_ptr = src_ptr_next;-                dest_ptr = dest_ptr.add(MEMCMP_GROUP_SIZE);+            let src_ptr_next = src_ptr.add(MEMCMP_GROUP_SIZE + 1); -                // If there is a padding, check whether or not it is correct.-                if std::intrinsics::unlikely(padding_size > 0) {-                    // First check padding size.-                    if std::intrinsics::unlikely(padding_size > MEMCMP_GROUP_SIZE) {-                        return Err(ErrorInner::bad_padding().into());-                    }+            // Copy `MEMCMP_GROUP_SIZE` bytes any way. However we will truncate the returned

I believe the comment is true and is unrelated to the error-handling path here. By my reading, padding can be applied to any record entry, and this is just saying to "copy all the padding, we'll truncate it out later".

brson

comment created time in 17 days

pull request commenttikv/tikv

engine_traits: Remove the SyncSnapshot trait

/merge

brson

comment created time in 17 days

push eventbrson/rust-doc-tool

Brian Anderson

commit sha cd2a9197a6555ab07e70475a6be5d63542173a5a

.

view details

push time in 19 days

push eventbrson/rust-doc-tool

Brian Anderson

commit sha 581211da609fb51901e74f7d10da0925d03854a0

.

view details

Brian Anderson

commit sha ed16d6b57a69b92aba4fbd85cec794da271bef1a

.

view details

Brian Anderson

commit sha c7db60d4af0515e6203dbb6e16aa7ae04275b8d6

.

view details

Brian Anderson

commit sha b4efd9791a71244ddca8c0a6fc366665324069e0

.

view details

Brian Anderson

commit sha dfd3a7e22bb64a030149f82fceacbd5c3fc9aefa

.

view details

push time in 19 days

push eventbrson/my-rust-lists

Brian Anderson

commit sha 3dec9344b1e53fee86a63254fcc7dc663a1b8faa

.

view details

push time in 20 days

Pull request review commenttikv/tikv

Fix two instances of undefined behavior in codec crate

 impl MemComparableByteCodec {             let src_ptr_end = src_ptr.add(src_len);              loop {-                let src_ptr_next = src_ptr.add(MEMCMP_GROUP_SIZE + 1);-                if std::intrinsics::unlikely(src_ptr_next > src_ptr_end) {+                // Make sure the source buffer is the expected length before+                // constructing the next source pointer. Note that it is UB to+                // even _create_ a pointer that points beyond its allocation.+                let ptrdiff = src_ptr_end.offset_from(src_ptr);+                let ptroob = ptrdiff < MEMCMP_GROUP_SIZE as isize + 1;++                if std::intrinsics::unlikely(ptroob) {                     return Err(ErrorInner::eof().into());                 } +                let src_ptr_next = src_ptr.add(MEMCMP_GROUP_SIZE + 1);

I don't understand the rules enough to say for sure, but here's what miri says about this code: https://gist.github.com/brson/26d2111ce4c1a555d0e9a817c37ff646

I take this to mean that even creating the pointer is UB.

I can't find language in https://doc.rust-lang.org/beta/reference/behavior-considered-undefined.html to support the idea that even doing pointer math to create out-of-bounds pointers is undefined. The strongest it says is that dereferencing dangling pointers is undefined, which is obvious to me.

cc @oli-obk @RalfJung sorry for pinging you so much, but maybe you can explain why it's not ok to do pointer math that produces a bad pointer here.

brson

comment created time in 20 days

pull request commenttikv/tikv

Fix two instances of undefined behavior in codec crate

This also marks the try_decode_first_internal function as unsafe.

brson

comment created time in 20 days

more