profile
viewpoint
Akanksha Mahajan akankshamahajan15 Facebook Menlo Park

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 15fd491dbeb666ffc10088afc93e7a0594e8d486

Add 6.14 branch to check_format_compatible.sh Summary: Add 6.14 to check_format_compatible.sh Test Plan: ./tools/check_format_compatible.sh

view details

push time in 2 days

pull request commentfacebook/rocksdb

Add 6.14 branch to check_format_compatible.sh

Thanks @akankshamahajan15 . The PR description can be cleaned up.

I am sorry I didn't get it. Cleaned up as in?

akankshamahajan15

comment created time in 2 days

PR opened facebook/rocksdb

Add 6.14 branch to check_format_compatible.sh

Summary: Add 6.14 to check_format_compatible.sh

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

+2 -2

0 comment

1 changed file

pr created time in 2 days

create barnchakankshamahajan15/rocksdb

branch : format_compatible

created branch time in 2 days

push eventakankshamahajan15/rocksdb

Cheng Chang

commit sha 5227b315ec214e6be1aa25d5c04374e41aad7cc2

Fix unchecked statuses for transaction_test (#7572) Summary: When `ASSERT_STATUS_CHECKED` is enabled, `transaction_test` does not pass without this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7572 Test Plan: `ASSERT_STATUS_CHECKED=1 make -j32 transaction_test && ./transaction_test` Reviewed By: zhichao-cao Differential Revision: D24404319 Pulled By: cheng-chang fbshipit-source-id: 13689035995366ab06d8eada3ea404e45fef8bc5

view details

Akanksha Mahajan

commit sha eef27d00486c45a52eaaa2b628415ade5603c6c9

Bug fix to remove function calling in assert statement (#7581) Summary: Remove function calling in assert statement as assert is a no op in opt build and that function might not be called. This causes hang in closing RocksDB when refit level is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7581 Test Plan: make check -j64 Reviewed By: riversand963 Differential Revision: D24466420 Pulled By: akankshamahajan15 fbshipit-source-id: 97db4ec5a95ae693c3290e176a3c12a9b1ad2f6d

view details

Ziyue Yang

commit sha 1c78e4b235d3df167b3856dd4fc3b105f7a840fb

Make parallel compression optimization code tidier (#6888) Summary: This commit makes https://github.com/facebook/rocksdb/issues/6262's code change tidier and easier to understand by: 1. Wrapping parallel compression initialization and termination into common methods; 2. Wrapping BlockRep initialization, push/pop into common methods; 3. Wrapping file size estimation into common methods; 4. Fixing function declarations that use non-const reference; 5. Fixing some uninitialized variables; 6. Fixing first_block data race; 7. Making BlockRep::status check in BlockBasedTableBuilder::Finish only present if ok(); 8. Making assert(ok()) in BlockBasedTableBuilder::CompressAndVerifyBlock only present in non-parallel compression mode. In parallel compression mode, compression will abort if status is not OK; 9. Eliminating potential data race caused by BlockBasedTableBuilder::GetStatus() and BlockBasedTableBuilder::GetIOStatus() by returning status copy instead of unprotected reference. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6888 Reviewed By: ajkr Differential Revision: D21957110 Pulled By: jay-zhuang fbshipit-source-id: 3a29892f249209513f030349756cecd7736eae80

view details

Zhichao Cao

commit sha d8ec0a760ab19a2ea1ce8249420cfe5b7ab79387

Make FileType Public and Replace kLogFile with kWalFile (#7580) Summary: As suggested by pdillinger ,The name of kLogFile is misleading, in some tests, kLogFile is defined as info log. Replace it with kWalFile and move it to public, which will be used in https://github.com/facebook/rocksdb/issues/7523 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7580 Test Plan: make check Reviewed By: riversand963 Differential Revision: D24485420 Pulled By: zhichao-cao fbshipit-source-id: 955e3dacc1021bb590fde93b0a568ffe9ad80799

view details

jmn

commit sha b1cdb8cc86f18e7cad7b219c4c496ab967044383

add StartTrace and EndTrace to stackable_db (#7585) Summary: In addition to trace block cache access, we want to support trace queries on MySQL. To achieve that StartTrace and EndTrace need to be added to the stackable_db.h Pull Request resolved: https://github.com/facebook/rocksdb/pull/7585 Reviewed By: zhichao-cao Differential Revision: D24482306 Pulled By: nmjnmjnmj fbshipit-source-id: de641b4837c64cd33b44b5bebaeae5d1527c8c31

view details

Peter Dillinger

commit sha a16d1b2fd36a2fe036b0aed1008f60570c981773

Add Encode/DecodeFixedGeneric, coding_lean.h (#7587) Summary: To minimize dependencies for Ribbon filter code in progress, core part of coding.h for fixed sizes has been moved to coding_lean.h. Also, generic versions of these functions have been added to math128.h (since the generic versions are likely only to be used along with Unsigned128). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7587 Test Plan: Unit tests added for new functions Reviewed By: jay-zhuang Differential Revision: D24486718 Pulled By: pdillinger fbshipit-source-id: a69768f742379689442135fa52237c01dfe2647e

view details

Cheng Chang

commit sha 1b224324b58b925efbd6e765dd345b4650d57fb1

Track WAL in MANIFEST: persist WALs to and recover WALs from MANIFEST (#7256) Summary: This PR makes it able to `LogAndApply` `VersionEdit`s related to WALs, and also be able to `Recover` from MANIFEST with WAL related `VersionEdit`s. The `VersionEdit`s related to WAL are treated similarly as those related to column family operations, they are not applied to versions, but can be in a commit group. Mixing WAL related `VersionEdit`s with other types of edits will make logic in `ProcessManifestWrite` more complicated, so `VersionEdit`s related to WAL can either be WAL additions or deletions, like column family add and drop. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7256 Test Plan: a set of unit tests are added in `version_set_test.cc` Reviewed By: riversand963 Differential Revision: D23123238 Pulled By: cheng-chang fbshipit-source-id: 246be2ed4744fd03fa2738aba408aaa611d0379c

view details

Yanqin Jin

commit sha 6595267980781bd872d2ecbf4f017c1d4068b381

Allow compaction iterator to perform garbage collection (#7556) Summary: Add a threshold timestamp, full_history_ts_low_ of type `std::string*` to `CompactionIterator`, so that RocksDB can also perform garbage collection during compaction. * If full_history_ts_low_ is nullptr, then compaction iterator does not perform GC, preserving all timestamp history for all keys. Compaction iterator will treat user key with different timestamps as different user keys. * If full_history_ts_low_ is not nullptr, then compaction iterator performs GC. GC will look at keys older than `*full_history_ts_low_` and determine their eligibility based on factors including snapshots. Current rules of GC: * If an internal key is in the same snapshot as a previous counterpart with the same user key, and this key is eligible for GC, and the key is not single-delete or merge operand, then this key can be dropped. Note that the previous internal key cannot be a merge operand either. * If a tombstone is the most recent one in the earliest snapshot and it is eligible for GC, and keyNotExistsBeyondLevel() is true, then this tombstone can be dropped. * If a tombstone is the most recent one in a snapshot and it is eligible for GC, and the compaction is at bottommost level, then all other older internal keys of the same user key must also be eligible for GC, thus can be dropped * Single-delete, delete-range and merge are not currently supported. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7556 Test Plan: make check Reviewed By: ltamasi Differential Revision: D24507728 Pulled By: riversand963 fbshipit-source-id: 3c09c7301f41eed76dfcf4d1527e68cf6e0a8bb3

view details

Peter Dillinger

commit sha 25d54c799ce5fb6887a151f4f584bf8d84c1256e

Ribbon: initial (general) algorithms and basic unit test (#7491) Summary: This is intended as the first commit toward a near-optimal alternative to static Bloom filters for SSTs. Stephan Walzer and I have agreed upon the name "Ribbon" for a PHSF based on his linear system construction in "Efficient Gauss Elimination for Near-Quadratic Matrices with One Short Random Block per Row, with Applications" ("SGauss") and my much faster "on the fly" algorithm for gaussian elimination (or for this linear system, "banding"), which can be faster than peeling while also more compact and flexible. See util/ribbon_alg.h for more detailed introduction and background. RIBBON = Rapid Incremental Boolean Banding ON-the-fly This commit just adds generic (templatized) core algorithms and a basic unit test showing some features, including the ability to construct structures within 2.5% space overhead vs. information theoretic lower bound. (Compare to cache-local Bloom filter's ~50% space overhead -> ~30% reduction anticipated.) This commit does not include the storage scheme necessary to make queries fast, especially for filter queries, nor fractional "result bits", but there is some description already and those implementations will come soon. Nor does this commit add FilterPolicy support, for use in SST files, but that will also come soon. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7491 Reviewed By: jay-zhuang Differential Revision: D24517954 Pulled By: pdillinger fbshipit-source-id: 0119ee597e250d7e0edd38ada2ba50d755606fa7

view details

Levi Tamasi

commit sha a7a04b6898be9e62c16042b9a89df3774291be77

Integrate BlobFileBuilder into the compaction process (#7573) Summary: Similarly to how https://github.com/facebook/rocksdb/issues/7345 integrated blob file writing into the flush process, the patch adds support for writing blob files to the compaction logic. Namely, if `enable_blob_files` is set, large values encountered during compaction are extracted to blob files and replaced with blob indexes. The resulting blob files are then logged to the MANIFEST as part of the compaction job's `VersionEdit` and added to the `Version` alongside any table files written by the compaction. Any errors during blob file building fail the compaction job. There will be a separate follow-up patch to perform blob garbage collection during compactions. In addition, the patch continues to chip away at the mess around computing various compaction related statistics by eliminating some code duplication and by making the `num_output_files` and `bytes_written` stats more consistent for flushes, compactions, and recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7573 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24404696 Pulled By: ltamasi fbshipit-source-id: 21216af3a172ad3ce8f85d11cd30923784ae426c

view details

Yanqin Jin

commit sha 6134ce6444898f8ad432045ad73a2e54ed820af2

Perform post-flush updates of memtable list in a callback (#6069) Summary: Currently, the following interleaving of events can lead to SuperVersion containing both immutable memtables as well as the resulting L0. This can cause Get to return incorrect result if there are merge operands. This may also affect other operations such as single deletes. ``` time main_thr bg_flush_thr bg_compact_thr compact_thr set_opts_thr 0 | WriteManifest:0 1 | issue compact 2 | wait 3 | Merge(counter) 4 | issue flush 5 | wait 6 | WriteManifest:1 7 | wake up 8 | write manifest 9 | wake up 10 | Get(counter) 11 | remove imm V ``` The reason behind is that: one bg flush thread's installing new `Version` can be batched and performed by another thread that is the "leader" MANIFEST writer. This bg thread removes the memtables from current super version only after `LogAndApply` returns. After the leader MANIFEST writer signals (releasing mutex) this bg flush thread, it is possible that another thread sees this cf with both memtables (whose data have been flushed to the newest L0) and the L0 before this bg flush thread removes the memtables. To address this issue, each bg flush thread can pass a callback function to `LogAndApply`. The callback is responsible for removing the memtables. Therefore, the leader MANIFEST writer can call this callback and remove the memtables before releasing the mutex. Test plan (devserver) ``` $make merge_test $./merge_test --gtest_filter=MergeTest.MergeWithCompactionAndFlush $make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6069 Reviewed By: cheng-chang Differential Revision: D18790894 Pulled By: riversand963 fbshipit-source-id: e41bd600c0448b4f4b2deb3f7677f95e3076b4ed

view details

mrambacher

commit sha f35f7f2704d9803908de7eb8864f260312d173b4

Fix many tests to run with MEM_ENV and ENCRYPTED_ENV; Introduce a MemoryFileSystem class (#7566) Summary: This PR does a few things: 1. The MockFileSystem class was split out from the MockEnv. This change would theoretically allow a MockFileSystem to be used by other Environments as well (if we created a means of constructing one). The MockFileSystem implements a FileSystem in its entirety and does not rely on any Wrapper implementation. 2. Make the RocksDB test suite work when MOCK_ENV=1 and ENCRYPTED_ENV=1 are set. To accomplish this, a few things were needed: - The tests that tried to use the "wrong" environment (Env::Default() instead of env_) were updated - The MockFileSystem was changed to support the features it was missing or mishandled (such as recursively deleting files in a directory or supporting renaming of a directory). 3. Updated the test framework to have a ROCKSDB_GTEST_SKIP macro. This can be used to flag tests that are skipped. Currently, this defaults to doing nothing (marks the test as SUCCESS) but will mark the tests as SKIPPED when RocksDB is upgraded to a version of gtest that supports this (gtest-1.10). I have run a full "make check" with MEM_ENV, ENCRYPTED_ENV, both, and neither under both MacOS and RedHat. A few tests were disabled/skipped for the MEM/ENCRYPTED cases. The error_handler_fs_test fails/hangs for MEM_ENV (presumably a timing problem) and I will introduce another PR/issue to track that problem. (I will also push a change to disable those tests soon). There is one more test in DBTest2 that also fails which I need to investigate or skip before this PR is merged. Theoretically, this PR should also allow the test suite to run against an Env loaded from the registry, though I do not have one to try it with currently. Finally, once this is accepted, it would be nice if there was a CircleCI job to run these tests on a checkin so this effort does not become stale. I do not know how to do that, so if someone could write that job, it would be appreciated :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7566 Reviewed By: zhichao-cao Differential Revision: D24408980 Pulled By: jay-zhuang fbshipit-source-id: 911b1554a4d0da06fd51feca0c090a4abdcb4a5f

view details

Peter Dillinger

commit sha 0e2e67562f3de67c088890ef9f0798551bfc225e

Give instructions instead of broken 2to3 for clang-format-diff.py (#7603) Summary: My previous change to use lib2to3 to migrate clang-format-diff.py for Python 2 only works if there's nothing to reformat. Instead, give instructions to download to REPO_ROOT. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7603 Test Plan: Try the instructions on a fresh CentOS 8 devserver Reviewed By: riversand963 Differential Revision: D24569608 Pulled By: pdillinger fbshipit-source-id: 1410ba163e016b226e883dec93fae3df9ed0eab2

view details

Andrew Kryczka

commit sha 6c2c0635c927be9c14d57e1592c2c42079c07067

Require only one `Logger::Logv()` implementation (#7605) Summary: A user who extended `Logger` recently pointed out it is unusual to require they implement the two-argument `Logv()` overload when they've already implemented the three-argument `Logv()` overload. I agree with that and think we can fix it by only calling the two-argument overload from the default implementation of the three-argument overload. Then when the three-argument overload is overridden, RocksDB would not rely on the two-argument overload. Only `Logger::LogHeader()` needed adjustment to achieve this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7605 Reviewed By: riversand963 Differential Revision: D24584749 Pulled By: ajkr fbshipit-source-id: 9aabe040ac761c4c0dbebc4be046967403ecaf21

view details

Ramkumar Vadivelu

commit sha 9a690a74e1010fd0ba60fc7d4f25a765fcddda59

In ParseInternalKey(), include corrupt key info in Status (#7515) Summary: Fixes Issue https://github.com/facebook/rocksdb/issues/7497 When allow_data_in_errors db_options is set, log error key details in `ParseInternalKey()` Have fixed most of the calls. Have few TODOs still pending - because have to make more deeper changes to pass in the allow_data_in_errors flag. Will do those in a separate PR later. Tests: - make check - some of the existing tests that exercise the "internal key too small" condition are: dbformat_test, cuckoo_table_builder_test - some of the existing tests that exercise the corrupted key path are: corruption_test, merge_helper_test, compaction_iterator_test Example of new status returns: - Key too small - `Corrupted Key: Internal Key too small. Size=5` - Corrupt key with allow_data_in_errors option set to false: `Corrupted Key: '<redacted>' seq:3, type:3` - Corrupt key with allow_data_in_errors option set to true: `Corrupted Key: '61' seq:3, type:3` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7515 Reviewed By: ajkr Differential Revision: D24240264 Pulled By: ramvadiv fbshipit-source-id: bc48f5d4475ac19d7713e16df37505b31aac42e7

view details

push time in 3 days

GollumEvent
GollumEvent
GollumEvent

push eventfacebook/rocksdb

akankshamahajan

commit sha 91f3b72ebc96e8083f23aca0aa516406c3e0df43

Update History.md and version.h for 6.14.2 Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in 10 days

push eventfacebook/rocksdb

Akanksha Mahajan

commit sha 3ddd79e983b342785dfdefe5076e3746cd7ed67f

Bug fix to remove function calling in assert statement (#7581) Summary: Remove function calling in assert statement as assert is a no op in opt build and that function might not be called. This causes hang in closing RocksDB when refit level is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7581 Test Plan: make check -j64 Reviewed By: riversand963 Differential Revision: D24466420 Pulled By: akankshamahajan15 fbshipit-source-id: 97db4ec5a95ae693c3290e176a3c12a9b1ad2f6d

view details

push time in 10 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha dcc12fecbb4199619ecc71762e234dca05202607

Fix format errors Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in 10 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 777d119d2b37d63048f5dd84fb099dad345a4cc6

Update History Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

akankshamahajan

commit sha ea0b6d828bb7a9db0896e693748c21b2091dbf5e

Fix format errors Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in 10 days

PR opened facebook/rocksdb

Bug fix to remove function calling in assert statement

Summary: Remove function calling in assert statement as assert is a no op in opt build and that function might not be called. This causes hang in closing RocksDB when refit level is set.

Test Plan: make check -j64

Reviewers:

Subscribers:

Tasks:

Tags:

+12 -10

0 comment

3 changed files

pr created time in 10 days

push eventakankshamahajan15/rocksdb

Andrew Kryczka

commit sha 8efb5cfb6fed6c07d31df3529db2517f3d8b7894

add SstFileManager to crash test (#6993) Summary: SstFileManager is already supported in the stress test as of https://github.com/facebook/rocksdb/issues/6454. This PR enables the SstFileManager in some of the crash test runs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6993 Reviewed By: riversand963 Differential Revision: D22084406 Pulled By: ajkr fbshipit-source-id: 78b8642682e7570ff6ec3a1c3ccd9940f4362289

view details

Zhichao Cao

commit sha 83a4dd1a67c9169b04f5a5dbce90c7999a2cfa46

Fix the memory leak in Env_basic_test (#7017) Summary: Fix the memory leak broken asan and other test introduced by https://github.com/facebook/rocksdb/issues/6830 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7017 Test Plan: pass asan_check Reviewed By: siying Differential Revision: D22190289 Pulled By: zhichao-cao fbshipit-source-id: 03a095f698b4f9d72fd9374191b17c890d7c2b56

view details

sdong

commit sha 50d696981664d20870890da634d4c2102f14b8c6

Fix unity build broken by #7007 (#7024) Summary: https://github.com/facebook/rocksdb/pull/7007 broken the unity build. Fix it by moving the const inside the function Pull Request resolved: https://github.com/facebook/rocksdb/pull/7024 Test Plan: make unity and see it to build. Reviewed By: zhichao-cao Differential Revision: D22212028 fbshipit-source-id: 5daff7383b691808164d4745ab543238502d946b

view details

sdong

commit sha 9cc25190e1ba6fc66bbd0d47caaf043d75839b6f

Test CircleCI with CLANG-10 (#7025) Summary: It's useful to build RocksDB using a more recent clang version in CI. Add a CircleCI build and fix some issues with it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7025 Test Plan: See all tests pass. Reviewed By: pdillinger Differential Revision: D22215700 fbshipit-source-id: 914a729c2cd3f3ac4a627cc0ac58d4691dca2168

view details

Yanqin Jin

commit sha e66199d848cd484b816d07359f1dc0f0b99e5351

First step towards handling MANIFEST write error (#6949) Summary: This PR provides preliminary support for handling IO error during MANIFEST write. File write/sync is not guaranteed to be atomic. If we encounter an IOError while writing/syncing to the MANIFEST file, we cannot be sure about the state of the MANIFEST file. The version edits may or may not have reached the file. During cleanup, if we delete the newly-generated SST files referenced by the pending version edit(s), but the version edit(s) actually are persistent in the MANIFEST, then next recovery attempt will process the version edits(s) and then fail since the SST files have already been deleted. One approach is to truncate the MANIFEST after write/sync error, so that it is safe to delete the SST files. However, file truncation may not be supported on certain file systems. Therefore, we take the following approach. If an IOError is detected during MANIFEST write/sync, we disable file deletions for the faulty database. Depending on whether the IOError is retryable (set by underlying file system), either RocksDB or application can call `DB::Resume()`, or simply shutdown and restart. During `Resume()`, RocksDB will try to switch to a new MANIFEST and write all existing in-memory version storage in the new file. If this succeeds, then RocksDB may proceed. If all recovery is completed, then file deletions will be re-enabled. Note that multiple threads can call `LogAndApply()` at the same time, though only one of them will be going through the process MANIFEST write, possibly batching the version edits of other threads. When the leading MANIFEST writer finishes, all of the MANIFEST writing threads in this batch will have the same IOError. They will all call `ErrorHandler::SetBGError()` in which file deletion will be disabled. Possible future directions: - Add an `ErrorContext` structure so that it is easier to pass more info to `ErrorHandler`. Currently, as in this example, a new `BackgroundErrorReason` has to be added. Test plan (dev server): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6949 Reviewed By: anand1976 Differential Revision: D22026020 Pulled By: riversand963 fbshipit-source-id: f3c68a2ef45d9b505d0d625c7c5e0c88495b91c8

view details

Zitan Chen

commit sha be41c61f22ad6195e6b3a004762d54ee6c19c3ed

Add a new option for BackupEngine to store table files under shared_checksum using DB session id in the backup filenames (#6997) Summary: `BackupableDBOptions::new_naming_for_backup_files` is added. This option is false by default. When it is true, backup table filenames under directory shared_checksum are of the form `<file_number>_<crc32c>_<db_session_id>.sst`. Note that when this option is true, it comes into effect only when both `share_files_with_checksum` and `share_table_files` are true. Three new test cases are added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6997 Test Plan: Passed make check. Reviewed By: ajkr Differential Revision: D22098895 Pulled By: gg814 fbshipit-source-id: a1d9145e7fe562d71cde7ac995e17cb24fd42e76

view details

Zitan Chen

commit sha 95fbb62c449cda22085a19bfd8d93bca64283780

Update HISTORY.md to include the Public API Change for DB::OpenForReadonly introduced earlier (#7023) Summary: `DB::OpenForReadOnly()` now returns `Status::NotFound` when the specified DB directory does not exist. Previously the error returned depended on the underlying `Env`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7023 Reviewed By: ajkr Differential Revision: D22207845 Pulled By: gg814 fbshipit-source-id: f35830811a0e67efb0ee82eda3a9739bc526baba

view details

Siying Dong

commit sha 63b59f21d9971fc941e3ee2244783a3c16bd5ad4

Add CircleCI gadget (#7028) Summary: CircleCI is stably running. Need to add a gadget. Also since Circle builds some Windows and Linux, rename Travis and Appveyor builds to their names. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7028 Reviewed By: ajkr Differential Revision: D22223756 fbshipit-source-id: 8f9dcea0b614f083e6d2a2ebf84dc6ab7c4d1601

view details

sdong

commit sha d64cf0e4eebe004a9e801fc3b5e625b784fabaa6

Move away from direct TmpDir() call in some tests (#7030) Summary: Some tests directly uses TmpDir() as temporary directory without adding any randomize factor. This would cause failures when tests run in parallel. Fix it by moving some of them to test::PerThreadDBPath() Pull Request resolved: https://github.com/facebook/rocksdb/pull/7030 Test Plan: Watch existing tests pass Reviewed By: zhichao-cao Differential Revision: D22224710 fbshipit-source-id: 28c9932fede0a4a64670e5b5fdb08f4fb5dccdd0

view details

Adam Retter

commit sha 82d98447e4493fa32d1f44ffa111b7dd547fb6e3

Use official CMake snap on Travis Linux (#6892) Summary: Switch to using the official CMake snap rather than our own from S3. See https://gitlab.kitware.com/cmake/cmake/-/issues/20122#note_769266 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6892 Reviewed By: ajkr Differential Revision: D22211822 Pulled By: pdillinger fbshipit-source-id: baf20005a4064d7f86b46087a41af4dfdea5510f

view details

Stanislav Tkach

commit sha 70b5d95dc7ad597742417cb9e81653f5d4cfe403

Add (more) getters for options to the C API (#6998) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6998 Reviewed By: ajkr Differential Revision: D22211700 Pulled By: pdillinger fbshipit-source-id: 1141c20527dee5e13205059bf8e83927063c4c1e

view details

Adam Retter

commit sha df5fbe6408e694987591a1c258af5aa330083963

Portable backward compatibility with MacOS 10.12+ (#7016) Summary: When `PORTABLE=1` is set, RocksDB will now be built with backwards compatibility for MacOS as far back as 10.12 (i.e. 2016). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7016 Reviewed By: ajkr Differential Revision: D22211312 Pulled By: pdillinger fbshipit-source-id: 7b0858d9b55d6265d3ea27bf5ea1673639b6538c

view details

Daniel Black

commit sha 741b9ba96bb103f57380768508fa961f03427cb7

gflags: freebsd include path + links (#7011) Summary: The include path from find_package(gflags) needed to be included to compile. Because gflags got included in THIRDPARTY_LIBS as a PRIVATE library to ROCKSDB_{SHARED|STATIC}_LIB, its functions aren't accessible to the all the tools an utilities that use gflags directly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7011 Reviewed By: siying Differential Revision: D22176303 Pulled By: ajkr fbshipit-source-id: 0a94523fc69e82d8f686bc0b43dc3eafc51ad84f

view details

Daniel Black

commit sha c2b0b696c4bbe510ea0ba871fba7bd659994e02a

filelock_test: add freebsd headers for waitpid (#7010) Summary: Per manual https://www.unix.com/man-page/FreeBSD/2/waitpid Pull Request resolved: https://github.com/facebook/rocksdb/pull/7010 Reviewed By: siying Differential Revision: D22176164 Pulled By: ajkr fbshipit-source-id: 0a850ae6f1791d10951d5e4a79cfee01a3981d5a

view details

Daniel Black

commit sha ce332f8c5e81db5cf4edd740e62059eed20dec9a

freebsd: malloc_usable_size check malloc_np.h (#7009) Summary: Per https://www.unix.com/man-page/freebsd/3/malloc_usable_size/ malloc_usable_size is in malloc_np.h as its a non-standard API. Without patch it just fails to detect from ./CMakeFiles/CMakeError.log In file included from /home/dan/build-rocksdb/CMakeFiles/CMakeTmp/CheckSymbolExists.cxx:2: /usr/include/malloc.h:3:2: error: "<malloc.h> has been replaced by <stdlib.h>" ^ /home/dan/build-rocksdb/CMakeFiles/CMakeTmp/CheckSymbolExists.cxx:8:19: error: use of undeclared identifier 'malloc_usable_size' return ((int*)(&malloc_usable_size))[argc]; ^ 2 errors generated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7009 Reviewed By: riversand963 Differential Revision: D22176093 Pulled By: ajkr fbshipit-source-id: da980f3d343b6d9b0c70d7827c6df495f3fb1ade

view details

sdong

commit sha 7006997e1286dffb73e1b73ed7773cbe5e94780a

Add ASAN CircleCI Run (#7027) Summary: ASAN run is powerful in finding memory leak bugs. Running it as a part of the pre-merge CI can help contributors avoid to merge some code with bugs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7027 Test Plan: Watch the test result. Reviewed By: pdillinger Differential Revision: D22222371 fbshipit-source-id: 92f9ce19e01a94ba5f9b765e154f7bcdece5c2a9

view details

sdong

commit sha f9817201aff23e1c0c30d84949660d272283e20f

Add unity build to CircleCI (#7026) Summary: We are still keeping unity build working. So it's a good idea to add to a pre-commit CI. A latest GCC docker image just to get a little bit more coverage. Fix three small issues to make it pass. Also make unity_test to run db_basic_test rather than db_test to cut the test time. There is no point to run expensive tests here. It was set to run db_test before db_basic_test was separated out. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7026 Test Plan: watch tests to pass. Reviewed By: zhichao-cao Differential Revision: D22223197 fbshipit-source-id: baa3b6cbb623bf359829b63ce35715c75bcb0ed4

view details

Zitan Chen

commit sha 1569dc48f5016eac0817f9cba72f2944a67929a7

`BackupEngine::VerifyBackup` verifies checksum by default (#7014) Summary: A parameter `verify_with_checksum` is added to `BackupEngine::VerifyBackup`, which is true by default. So now `BackupEngine::VerifyBackup` verifies backup files with checksum AND file size by default. When `verify_with_checksum` is false, `BackupEngine::VerifyBackup` only compares file sizes to verify backup files. Also add a test for the case when corruption does not change the file size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7014 Test Plan: Passed backupable_db_test Reviewed By: zhichao-cao Differential Revision: D22165590 Pulled By: gg814 fbshipit-source-id: 606a7450714e868bceb38598c89fd356c6004f4f

view details

Akanksha Mahajan

commit sha b9d51b86842b8432d740538c31fd8596f7765f0f

Fix for TSAN failure in DeleteScheduler (#7029) Summary: TSAN failure caused by setting statistics in SstFileManager and DeleteScheduler. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7029 Test Plan: 1. make check -j64 2. COMPILE_WITH_TSAN=1 make check -j64 Reviewed By: siying, zhichao-cao Differential Revision: D22223418 Pulled By: akankshamahajan15 fbshipit-source-id: c5bf336d711b787908dfeb6166cab4aa2e494d61

view details

Yanqin Jin

commit sha d47c8711908e84e6048d7a1233b20adc831c8590

Fix data race to VersionSet::io_status_ (#7034) Summary: After https://github.com/facebook/rocksdb/issues/6949 , VersionSet::io_status_ can be concurrently accessed by multiple threads without lock, causing tsan test to fail. For example, a bg flush thread resets io_status_ before calling LogAndApply(), while another thread already in the process of LogAndApply() reads io_status_. This is a bug. We do not have to reset io_status_ each time we call LogAndApply(). io_status_ is part of the state of VersionSet, and it indicates the outcome of preceding MANIFEST/CURRENT files IO operations. Its value should be updated only when: 1. MANIFEST/CURRENT files IO fail for the first time. 2. MANIFEST/CURRENT files IO succeed as part of recovering from a prior failure without process restart, e.g. calling Resume(). Test Plan (devserver): COMPILE_WITH_TSAN=1 make check COMPILE_WITH_TSAN=1 make db_test2 ./db_test2 --gtest_filter=DBTest2.CompactionStall Pull Request resolved: https://github.com/facebook/rocksdb/pull/7034 Reviewed By: zhichao-cao Differential Revision: D22247137 Pulled By: riversand963 fbshipit-source-id: 77b83e05390f3ee3cd2d96d3fdd6fe4f225e3216

view details

push time in 10 days

push eventakankshamahajan15/rocksdb

Cheng Chang

commit sha 0ea7db768e4332fa81f234e99c650f6f3244745d

Abstract out LockManager interface (#7532) Summary: In order to be able to introduce more locking protocols, we need to abstract out the locking subsystem in TransactionDB into a set of interfaces. PR https://github.com/facebook/rocksdb/pull/7013 introduces interface `LockTracker`. This PR is a follow up to take the first step to abstract out a `LockManager` interface. Further modifications to the interface may be needed when introducing the first implementation of range lock. But the idea here is to put the range lock implementation based on range tree under the `utilities/transactions/lock/range/range_tree`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7532 Test Plan: point_lock_manager_test Reviewed By: ajkr Differential Revision: D24238731 Pulled By: cheng-chang fbshipit-source-id: 2a9458cd8b3fb008d9529dbc4d3b28c24631f463

view details

anand76

commit sha 00751e4292e55c1604b28b7b93fe7a538fa05f29

Add a host location property to TableProperties (#7479) Summary: This PR adds support for writing a location identifier of the DB host to SST files as a table property. By default, the hostname is used, but can be overridden by the user. There have been some recent corruptions in files written by ```SstFileWriter``` before checksumming, so this property can be used to trace it back to the writing host and checking the host for hardware isues. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7479 Test Plan: Add new unit tests Reviewed By: pdillinger Differential Revision: D24340671 Pulled By: anand1976 fbshipit-source-id: 2038949fd8d160c0633ccb4f9da77740f19fa2a2

view details

Akanksha Mahajan

commit sha b4cd51d847bb41bb80df999393b2acac4f67219d

Fix for stress test failure (#7574) Summary: Ignore read error in 'FilePrefetchBuffer::TryReadFromCache' as status is ignored and bool value is returned. Return error if prefetch fails in 'PrefetchTail' as we have planned to return Prefetch failures to users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7574 Test Plan: make check -j64, python -u tools/db_crashtest.py blackbox Reviewed By: zhichao-cao Differential Revision: D24408825 Pulled By: akankshamahajan15 fbshipit-source-id: feebda771415998253fbe54632f13e6e75b7a243

view details

Cheng Chang

commit sha fc9b4160139c66f50fd5ad42d05e9111483b2e69

Fix typo in db_wal_test (#7571) Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7571 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D24392577 Pulled By: cheng-chang fbshipit-source-id: c94f92db48270d0e215aa0f2782b0ff2e31bf708

view details

mrambacher

commit sha 1eda625eab491402efaa2cad17c5964008ad7d10

Revert `Status`es returned from pre-`Configurable` options functions (#7563) Summary: Further refinement of the earlier PR. Now the Status is NotFound with a subcode of PathNotFound. Also the existing functions for options parsing/loading are reverted to return InvalidArgument no matter in which way the user-provided arguments are deemed invalid. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7563 Reviewed By: zhichao-cao Differential Revision: D24422491 Pulled By: ajkr fbshipit-source-id: ba6b237cd0584d3f925c5ba0d349aeb8c250af67

view details

Cheng Chang

commit sha 73dbe10bbf34f3eeea6cb2d525fb7309d704399b

Fix write_batch_test when ASSERT_STATUS_CHECKED=1 (#7575) Summary: Without this PR, `ASSERT_STATUS_CHECKED=1 make -j32 write_batch_test && ./write_batch_test` fails. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7575 Test Plan: ASSERT_STATUS_CHECKED=1 make -j32 write_batch_test && ./write_batch_test Reviewed By: zhichao-cao Differential Revision: D24411442 Pulled By: cheng-chang fbshipit-source-id: f67dc43c44d6afcc6d7e5ff15c6ae9bbf4dfc943

view details

push time in 10 days

push eventfacebook/rocksdb

mrambacher

commit sha 59ddf7808143e1015207ca619e1f09310aea2bfb

Revert `Status`es returned from pre-`Configurable` options functions (#7563) Summary: Further refinement of the earlier PR. Now the Status is NotFound with a subcode of PathNotFound. Also the existing functions for options parsing/loading are reverted to return InvalidArgument no matter in which way the user-provided arguments are deemed invalid. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7563 Reviewed By: zhichao-cao Differential Revision: D24422491 Pulled By: ajkr fbshipit-source-id: ba6b237cd0584d3f925c5ba0d349aeb8c250af67

view details

push time in 11 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 02cb4d8224470bf606a8ec7addc15977108363c8

Update "ReadOptions.value_size_soft_limit" to also include keys size. Summary: Update "ReadOptions.value_size_soft_limit" to include keys size (keys found) along with size of their values. Initally this option considered only the size of values. Test Plan: make check -j64, Tests related to value_size_soft_limit Reviewers: Subscribers: Tasks: T77798449 Tags:

view details

push time in 11 days

PR opened facebook/rocksdb

Update "ReadOptions.value_size_soft_limit" to also include keys size.

Summary: Update "ReadOptions.value_size_soft_limit" to include keys size (keys found) along with size of their values. Initally this option considered only the size of values.

Test Plan: make check -j64, Tests related to value_size_soft_limit

Reviewers:

Subscribers:

Tasks: T77798449

Tags:

+39 -33

0 comment

6 changed files

pr created time in 11 days

push eventakankshamahajan15/rocksdb

Akanksha Mahajan

commit sha b4cd51d847bb41bb80df999393b2acac4f67219d

Fix for stress test failure (#7574) Summary: Ignore read error in 'FilePrefetchBuffer::TryReadFromCache' as status is ignored and bool value is returned. Return error if prefetch fails in 'PrefetchTail' as we have planned to return Prefetch failures to users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7574 Test Plan: make check -j64, python -u tools/db_crashtest.py blackbox Reviewed By: zhichao-cao Differential Revision: D24408825 Pulled By: akankshamahajan15 fbshipit-source-id: feebda771415998253fbe54632f13e6e75b7a243

view details

akankshamahajan

commit sha 3e483358b0660bc1dff9eb0c7994711ef5d6e6e8

Update "ReadOptions.value_size_soft_limit" to also include keys size. Summary: Update "ReadOptions.value_size_soft_limit" to include keys size (keys found) along with size of their values. Initally this option considered only the size of values. Test Plan: make check -j64, Tests related to value_size_soft_limit Reviewers: Subscribers: Tasks: T77798449 Tags:

view details

push time in 11 days

create barnchakankshamahajan15/rocksdb

branch : value_size

created branch time in 11 days

PR opened facebook/rocksdb

Fix for stress test failure

Summary: Ignore read error in 'FilePrefetchBuffer::TryReadFromCache' as status is ignored and bool value is returned. Return error if prefetch fails in 'PrefetchTail' as we have planned to return Prefetch failures to users.

Test Plan: make check -j64, python -u tools/db_crashtest.py blackbox

Reviewers:

Subscribers:

Tasks:

Tags:

+7 -0

0 comment

2 changed files

pr created time in 12 days

create barnchakankshamahajan15/rocksdb

branch : fix_stress_error

created branch time in 12 days

push eventakankshamahajan15/rocksdb

Levi Tamasi

commit sha e8cb32ed67f4e8971c16264ed921f857da38f45a

Introduce BlobFileCache and add support for blob files to Get() (#7540) Summary: The patch adds blob file support to the `Get` API by extending `Version` so that whenever a blob reference is read from a file, the blob is retrieved from the corresponding blob file and passed back to the caller. (This is assuming the blob reference is valid and the blob file is actually part of the given `Version`.) It also introduces a cache of `BlobFileReader`s called `BlobFileCache` that enables sharing `BlobFileReader`s between callers. `BlobFileCache` uses the same backing cache as `TableCache`, so `max_open_files` (if specified) limits the total number of open (table + blob) files. TODO: proactively open/cache blob files and pin the cache handles of the readers in the metadata objects similarly to what `VersionBuilder::LoadTableHandlers` does for table files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7540 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24260219 Pulled By: ltamasi fbshipit-source-id: a8a2a4f11d3d04d6082201b52184bc4d7b0857ba

view details

Adam Retter

commit sha f4ade82ad29790b1e0d99188e0b26e5b805c7243

Fix the CI badge for ppc64le Jenkins (#7561) Summary: Fixes the URL for the badge and link. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7561 Reviewed By: ltamasi Differential Revision: D24336928 Pulled By: pdillinger fbshipit-source-id: 41e04a5b036458e303aac3ae3b6129f572f0e9ce

view details

Stanislav Tkach

commit sha ed90e2a45043a107a594d72df5a0ecbed7b0ebb7

Add getters to the C API for env, universal compaction options and fifo compaction options (#7501) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7501 Reviewed By: ltamasi Differential Revision: D24344109 Pulled By: pdillinger fbshipit-source-id: d9a2b1b1cc8c8d8a96f13b8ae6814380caa10c96

view details

push time in 15 days

push eventakankshamahajan15/rocksdb

Jay Zhuang

commit sha c87c3a48afe1849caad5f10d352d334ca237d54a

Add a missing bug fix in HISTORY.md (#7549) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7549 Reviewed By: ajkr, zhichao-cao Differential Revision: D24292032 Pulled By: jay-zhuang fbshipit-source-id: 0442283386ae20d10410a8d013a431d7cd282b22

view details

Akanksha Mahajan

commit sha 7b65666cf148685bdc22fc578001f10fbd13757f

Update IOTrace operations in stackable_db.h (#7514) Summary: Update IOTrace operations in stackabledb.h and also trace few other IO operations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7514 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D24151202 Pulled By: akankshamahajan15 fbshipit-source-id: 112cd3d2041f8c6398b7b0ba1a783b8c93224d4a

view details

Akanksha Mahajan

commit sha db87afbcb36f88a4595c3328974b129821682126

Return error if Get/Multi() fails in Prefetching Filter blocks (#7543) Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails. On returning error in PartitionedFilterBlockReader::CacheDependencies was causing stress test failure because PrefetchBuffer is initialized with enable_ = true, as result when PosixMmapReadableFile::Read is called from Prefetch, scratch is ignored causing buffer to fill with garbage values. Initializing prefetch buffer by CreatePrefetchBuffer that sets enable_ with !ioptions.allow_mmap_reads fixed the problem as it returns without prefetching data if allow_mmap_reads is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7543 Test Plan: make check -j64; python -u tools/db_crashtest.py --simple blackbox Reviewed By: anand1976 Differential Revision: D24284596 Pulled By: akankshamahajan15 fbshipit-source-id: f3f0fd44b59dcf60645730436f28564a07884868

view details

Zhichao Cao

commit sha b99fe1ab74ba2b29223aa40eac5141912ca676ff

Remove the status.PermitUncheckedError() from WriteGroup Destructor (#7555) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7555 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 error_handler_fs_test Reviewed By: riversand963 Differential Revision: D24299387 Pulled By: zhichao-cao fbshipit-source-id: 6c8aa91c4b6e2bc82580b8d2264c177068f5a32c

view details

Tomasz Posluszny

commit sha 6528ecc8000c8a62664fd78eaf00578c7d8d5906

Add event listeners to RocksJava (#7425) Summary: Allows adding event listeners in RocksJava. * Adds listeners getter and setter in `Options` and `DBOptions` classes. * Adds `EventListener` Java interface and base class for implementing custom event listener callbacks - `AbstractEventListener`, which has an underlying native callback class implementing C++ `EventListener` class. * `AbstractEventListener` class has mechanism for selectively enabling its callback methods in order to prevent invoking Java method if it is not implemented. This decreases performance cost in case only subset of event listener callback methods is needed - the JNI code for remaining "no-op" callbacks is not executed. * The code is covered by unit tests in `EventListenerTest.java`, there are also tests added for setting/getting listeners field in `OptionsTest.java` and `DBOptionsTest.java`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7425 Reviewed By: pdillinger Differential Revision: D24063390 Pulled By: jay-zhuang fbshipit-source-id: 508c359538983d6b765e70d9989c351794a944ee

view details

Akanksha Mahajan

commit sha 850cc0dbed30ae2956d2e4570b557bcdf9af1151

Fix for clang_analyzer build failure in table_test (#7553) Summary: fix for clang_analyzer build failure in table_test because of potential memory leak of memtable in case of ASSERT failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7553 Test Plan: USE_CLANG=1 make analyze; make check -j64 Reviewed By: jay-zhuang Differential Revision: D24295042 Pulled By: akankshamahajan15 fbshipit-source-id: e9ea184367970fff3b520e33f3ceebf28d66ac8d

view details

Tomasz Posluszny

commit sha 05fba969278306bbc7edd61af1b8cf638a369c8a

Make RocksDB instance responsible for closing associated ColumnFamilyHandle instances (#7428) Summary: - Takes the burden off developer to close ColumnFamilyHandle instances before closing RocksDB instance - The change is backward-compatible ---- Previously the pattern for working with Column Families was: ```java try (final ColumnFamilyOptions cfOpts = new ColumnFamilyOptions().optimizeUniversalStyleCompaction()) { // list of column family descriptors, first entry must always be default column family final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList( new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, cfOpts), new ColumnFamilyDescriptor("my-first-columnfamily".getBytes(), cfOpts) ); // a list which will hold the handles for the column families once the db is opened final List<ColumnFamilyHandle> columnFamilyHandleList = new ArrayList<>(); try (final DBOptions options = new DBOptions() .setCreateIfMissing(true) .setCreateMissingColumnFamilies(true); final RocksDB db = RocksDB.open(options, "path/to/do", cfDescriptors, columnFamilyHandleList)) { try { // do something } finally { // NOTE user must explicitly frees the column family handles before freeing the db for (final ColumnFamilyHandle columnFamilyHandle : columnFamilyHandleList) { columnFamilyHandle.close(); } } // frees the column family options } } // frees the db and the db options ``` With the changes in this PR, the Java user no longer has to worry about manually closing the Column Families, which allows them to write simpler symmetrical create/free oriented code like this: ```java try (final ColumnFamilyOptions cfOpts = new ColumnFamilyOptions().optimizeUniversalStyleCompaction()) { // list of column family descriptors, first entry must always be default column family final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList( new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, cfOpts), new ColumnFamilyDescriptor("my-first-columnfamily".getBytes(), cfOpts) ); // a list which will hold the handles for the column families once the db is opened final List<ColumnFamilyHandle> columnFamilyHandleList = new ArrayList<>(); try (final DBOptions options = new DBOptions() .setCreateIfMissing(true) .setCreateMissingColumnFamilies(true); final RocksDB db = RocksDB.open(options, "path/to/do", cfDescriptors, columnFamilyHandleList)) { // do something } // frees the column family options, then frees the db and the db options } } ``` **NOTE**: The changes in this PR are backwards API compatible, which means existing code using the original approach will also continue to function correctly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7428 Reviewed By: cheng-chang Differential Revision: D24063348 Pulled By: jay-zhuang fbshipit-source-id: 648d7526669923128c863ead94516bf4d50ac658

view details

Stanislav Tkach

commit sha 1a83f5a8ac0adc1fd0428a721a3493131baaa5be

Expose BackupableDBOptions in the C API (#7550) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7550 Reviewed By: jay-zhuang Differential Revision: D24315343 Pulled By: ajkr fbshipit-source-id: fc7855b630a50c00dcb940241942295932732f39

view details

Adam Retter

commit sha ccbf468cb1e3ba5255b1977e8daf10d5e64f68eb

Small JNI improvements (#7371) Summary: * Avoid some unnecessary array copy operations on read/write * Remove some duplicated code * Don't leak arrays on some exceptions * Fixed some doc comments Pull Request resolved: https://github.com/facebook/rocksdb/pull/7371 Reviewed By: jay-zhuang Differential Revision: D24312932 Pulled By: pdillinger fbshipit-source-id: 422fe6b98bbdb922a148922ac0d2d965c715176e

view details

mrambacher

commit sha a8c89cc96983be8884bf93de484c13f76685bcf9

Test for LoadLatestOptions (#7554) Summary: Make LoadLatestOptions return PathNotFound if the options file does not exist. Added tests for the LoadOptions related methods. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7554 Reviewed By: akankshamahajan15 Differential Revision: D24298985 Pulled By: zhichao-cao fbshipit-source-id: c9ae3cb12fc4a5bbef07743e1c1300f98a2441b3

view details

Adam Retter

commit sha fa2a8cda7bf5a5af1176fff044645b631fd76a64

Update zstd dependency to 1.4.5 (#7560) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7560 Reviewed By: akankshamahajan15 Differential Revision: D24323048 Pulled By: zhichao-cao fbshipit-source-id: 2094ef089c37f570d4ea30d28d2f46f9fa1ce0f1

view details

push time in 16 days

push eventfacebook/rocksdb

mrambacher

commit sha d043387bddbf4bbd77fc44402d232db4793878c0

Test for LoadLatestOptions (#7554) Summary: Make LoadLatestOptions return PathNotFound if the options file does not exist. Added tests for the LoadOptions related methods. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7554 Reviewed By: akankshamahajan15 Differential Revision: D24298985 Pulled By: zhichao-cao fbshipit-source-id: c9ae3cb12fc4a5bbef07743e1c1300f98a2441b3

view details

push time in 16 days

pull request commentfacebook/rocksdb

Update IOTrace operations in stackable_db.h

Are there any tests to cover this code?

I added tests in io_tracer_test to cover testing of writing records in IOTracer and io_tracer_parser_test to cover writing the records through calling DB::StartIOTracer and then DB::EndIOTracer and parsing those records.

akankshamahajan15

comment created time in 17 days

issue commentfacebook/rocksdb

The FileSystemTracingWrapper makes an extra ElapsedNanos call on every traced operation, and other issues.

I have fixed the issue 2 mentioned above. Working on the remaining two issues

mrambacher

comment created time in 17 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha c1436f6efbd7729c46c3a81f051fa06037fe6498

Fix for clang_analyzer build failure in table_test Summary: fix for clang_analyzer build failure in table_test because of potential memory leak of memtable in case of ASSERT_FALSE. Test Plan: USE_CLANG=1 make analyze; make check -j64 Reviewers: Subscribers: Tasks: Tags:

view details

push time in 18 days

push eventakankshamahajan15/rocksdb

Jay Zhuang

commit sha c87c3a48afe1849caad5f10d352d334ca237d54a

Add a missing bug fix in HISTORY.md (#7549) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7549 Reviewed By: ajkr, zhichao-cao Differential Revision: D24292032 Pulled By: jay-zhuang fbshipit-source-id: 0442283386ae20d10410a8d013a431d7cd282b22

view details

akankshamahajan

commit sha c5f0796919c903e8ca1f7fdd0d8df1dae43d16da

Fix for clang_analyzer build failure in table_test Summary: fix for clang_analyzer build failure in table_test because of potential memory leak of memtable in case of ASSERT_FALSE. Test Plan: USE_CLANG=1 make analyze; make check -j64 Reviewers: Subscribers: Tasks: Tags:

view details

push time in 18 days

PR opened facebook/rocksdb

Fix for clang_analyzer build failure in table_test

Summary: fix for clang_analyzer build failure in table_test because of potential memory leak of memtable in case of ASSERT failure.

Test Plan: USE_CLANG=1 make analyze; make check -j64

Reviewers:

Subscribers:

Tasks:

Tags:

+23 -16

0 comment

1 changed file

pr created time in 18 days

create barnchakankshamahajan15/rocksdb

branch : fix_bug

created branch time in 18 days

push eventakankshamahajan15/rocksdb

Jay Zhuang

commit sha 4a6840bd004c840112e263cf96c4a21852f03f60

db_stress prints key in Hex (#7533) Summary: db_stress prints key in both id and hex. https://github.com/facebook/rocksdb/issues/7531 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7533 Test Plan: local test Reviewed By: akankshamahajan15 Differential Revision: D24252725 Pulled By: jay-zhuang fbshipit-source-id: f0c1409a0568874df36949d5da139316d978fa98

view details

push time in 18 days

push eventakankshamahajan15/rocksdb

Akanksha Mahajan

commit sha 9dd25487ccdc61fecc57514072d17dec174034a7

Update release history 6.14 (#7525) Summary: Update release history for 6.14 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7525 Test Plan: No code change Reviewed By: jay-zhuang Differential Revision: D24224690 Pulled By: akankshamahajan15 fbshipit-source-id: 95441aefde96672fea5a6af5d7e67cdafb1ebdd2

view details

Cheng Chang

commit sha 12b78e40bdba980f3692fd2018cc905eef2127f8

Track WAL in MANIFEST: add option track_and_verify_wals_in_manifest (#7275) Summary: This option determines whether WALs will be tracked in MANIFEST and verified on recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7275 Test Plan: db_options_test options_test Reviewed By: pdillinger Differential Revision: D23181418 Pulled By: cheng-chang fbshipit-source-id: 5dd1cdc166f3dfc1c93c094df4a2f7734e3b4547

view details

Andrew Kryczka

commit sha 75d3b6fdf0aa1007c4d26382f65be0adf4519a37

Redesign block cache pinning API (#7520) Summary: The old flag-based APIs (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache` and `BlockBasedTableOptions::pin_top_level_index_and_filter`) were insufficient for our needs. For example, it was impossible to pin only unpartitioned meta-blocks, which could prevent block cache contention when turning on dictionary compression or during a migration to partitioned indexes/filters. It was also impossible to pin all meta-blocks in memory while having predictable memory usage via block cache. If we had continued adding flags to address these scenarios, they would have had significant overlap causing confusion. Instead, this PR deprecates the flags and starts a new API with non-overlapping options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7520 Test Plan: - new unit test - added new options to stress/crash test and ran for a while: `$ python tools/db_crashtest.py blackbox --simple --max_key=1000000 -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 --interval=10 -value_size_mult=33 -column_families=1 -reopen=0` Reviewed By: pdillinger Differential Revision: D24200034 Pulled By: ajkr fbshipit-source-id: 3fa7cfc71e7960f7a867511dd6ae5834dd73b13e

view details

Cheng Chang

commit sha cb2581031a53adabc5e60937266bcee041f9eea0

Add macos tests to circleci (#7536) Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7536 Test Plan: see the new `build-macos` tests pass in circleci Reviewed By: jay-zhuang Differential Revision: D24243218 Pulled By: cheng-chang fbshipit-source-id: 9b5f8a859e54c99a9ebe7efff6f336458a5d42de

view details

Jay Zhuang

commit sha f548a2a03cacdf1463bb4c63340569f846c73128

Fix a flaky tsan test for DBTest2 (#7526) Summary: ThreadSanitizer: data race for `DummyOldStats.num_rt`. Failed build: https://app.circleci.com/pipelines/github/facebook/rocksdb/3991/workflows/b47c3ae1-5531-4489-ac51-11854abdfd0f/jobs/42305 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7526 Reviewed By: akankshamahajan15 Differential Revision: D24226736 Pulled By: jay-zhuang fbshipit-source-id: e05ce354d0c0db0eba242d59d4b0e89ce7c25acf

view details

Jay Zhuang

commit sha 9e03e4dd52ead7875b407d72106c5d23594065b4

db_crashtest preserves expected values file for failed crash tests (#7534) Summary: If crash test fails, don't delete the `expected_values_file` for later debug. More details: https://github.com/facebook/rocksdb/issues/7530 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7534 Test Plan: local host Reviewed By: ajkr Differential Revision: D24239655 Pulled By: jay-zhuang fbshipit-source-id: 3566f91a30aae1e27d2f51d910cddd08edb7d4cf

view details

Zhichao Cao

commit sha 861e5443355c3de88bb0a180a09fba818cd238ee

Add db_flush_test to ASSERT_STATUS_CHECKED list (#7476) Summary: Added status check enforcement for db_flush_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7476 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 db_flush_test Reviewed By: akankshamahajan15 Differential Revision: D24033752 Pulled By: zhichao-cao fbshipit-source-id: d957934e1666d0043bebdd8a4149e94cdcbbb89b

view details

mrambacher

commit sha bf342394b6fbf5f988ea134ca7bfb615bb4bc935

Add tests for paranoid checks with range deletion (#7521) Summary: Added unit tests that have paranoid_check = true that perform range deletions. At the moment, the deleted ranges do not appear to be checked as part of the paranoid checks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7521 Reviewed By: zhichao-cao Differential Revision: D24262175 Pulled By: ajkr fbshipit-source-id: 1035e968f7ab8ccaa7af086b835a4e72c7e56743

view details

Zhichao Cao

commit sha 16bff5370d6d660de8e07c354db5a8d3b2fc6bc0

Add plain_table_db_test to ASSERT_STATUS_CHECKED list (#7482) Summary: Add plain_table_db_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7482 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 plain_table_db_test Reviewed By: riversand963 Differential Revision: D24034987 Pulled By: zhichao-cao fbshipit-source-id: e61c937d55ded0947cc8936937362dafed572a60

view details

Andrew Kryczka

commit sha 3dc823212d73da455812fb39d6923cb2fec9cbed

add missing release notes to HISTORY.md (#7545) Summary: These notes existed on the release branches where they were backported, but were never added on master branch. Added them now and mentioned what minor release the fix originally appeared. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7545 Reviewed By: riversand963 Differential Revision: D24281759 Pulled By: ajkr fbshipit-source-id: 7422e984b667793d6260dd32a7492afcb2ff1c4b

view details

push time in 18 days

PR opened facebook/rocksdb

Return error if Get/Multi() fails in Prefetching Filter blocks

Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails.

On returning error in PartitionedFilterBlockReader::CacheDependencies was causing stress test failure because in PrefetchBuffer is initialized with enable_ = true, as result when PosixMmapReadableFile::Read is called in Prefetch, scratch is ignored causing buffer to fill with garbage values. Initializing by CreatePrefetchBuffer that sets enable_ with !ioptions.allow_mmap_reads fixed the problem as it returns without prefetch if allow_mmap_reads is set.

Test Plan: make check -j64; python -u tools/db_crashtest.py --simple blackbox

Reviewers:

Subscribers:

Tasks:

Tags:

+33 -21

0 comment

5 changed files

pr created time in 19 days

push eventakankshamahajan15/rocksdb

Zhichao Cao

commit sha b7062f0b2cf05917edb21ff4e2e9fed4fc44284f

Status check enforcement for error_handler_fs_test (#7342) Summary: Added status check enforcement for error_test_fs_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7342 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 error_test_fs_test Reviewed By: akankshamahajan15 Differential Revision: D23972231 Pulled By: zhichao-cao fbshipit-source-id: fa41bfe440012e0c55f2c9507c1d0104e5e93f84

view details

sdong

commit sha 668ee08915fb753ecd5c0dd78fa998bd4898e80b

Fix prefix_test for status check (#7495) Summary: Fix prefix_test so that it passes when ASSERT_STATUS_CHECKED=1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7495 Test Plan: Run the test with the option Reviewed By: anand1976 Differential Revision: D24069715 fbshipit-source-id: 54f74b58575a1b49dbdee9ea2d24751fa956b620

view details

Akanksha Mahajan

commit sha 7d503e66a9a9d9bdf17e2d35038bfe8f3e12f5dc

Return error if Get() fails in Prefetching Filter blocks (#7463) Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7463 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D24008226 Pulled By: akankshamahajan15 fbshipit-source-id: b65d63b2d01465db92500b78de7ad58650ec9b3b

view details

Yanqin Jin

commit sha 758ead5df771d8ac2972c9f2c4d6074c97272151

Enforce status check for corruption_test (#7453) Summary: Enforce status check for corruption_test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7453 Test Plan: ``` ASSERT_STATUS_CHECKED=1 make corruption_test ./corruption_test ``` Reviewed By: jay-zhuang Differential Revision: D24006862 Pulled By: riversand963 fbshipit-source-id: 664677caf4c3007a25cf565cec3d677f2dcea130

view details

Levi Tamasi

commit sha 5d16325ce33384de974560648434473031639018

Revert "Return error if Get() fails in Prefetching Filter blocks (#7463)" (#7505) Summary: This reverts commit 7d503e66a9a9d9bdf17e2d35038bfe8f3e12f5dc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7505 Reviewed By: ajkr Differential Revision: D24100875 Pulled By: ltamasi fbshipit-source-id: 8705e3e6e8be4b4fd175ffdb031baa6530b61151

view details

sdong

commit sha 2496d3d4b849af53ef3dbb03801c1e359f4d63ff

Avoid to suppress status code in ~BlockBasedTableBuilder (#7507) Summary: We just used a hacky way to fix db_basic_test: suppress status code in ~BlockBasedTableBuilder. Rather, we should pass them back in Finish() and suppress them in Abandon(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7507 Test Plan: Watch existing tests to succeed. Reviewed By: riversand963 Differential Revision: D24119527 fbshipit-source-id: 71c4d4a81c0fd1c5595224692275f20f7759973a

view details

Jay Zhuang

commit sha 53089038def1a5ffa05afb27beaa63ed64dff764

Fix StallWrite crash with mixed of slowdown/no_slowdown writes (#7508) Summary: `BeginWriteStall()` removes no_slowdown write from the write list and updates `link_newer`, which makes `CreateMissingNewerLinks()` thought all write list has valid `link_newer` and failed to create link for all writers. It caused flaky test and SegFault for release build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7508 Test Plan: Add unittest to reproduce the issue. Reviewed By: anand1976 Differential Revision: D24126601 Pulled By: jay-zhuang fbshipit-source-id: f8ac5dba653f7ee1b0950296427d4f5f8ee34a06

view details

Yanqin Jin

commit sha 1bcef3d83c668afa690dd40c89bb1e955be0b1da

Make sure assert(false) handles failures too (#7483) Summary: In opt mode, assertions are just no-ops. Therefore, we need to report errors instead of just doing an `assert(false)`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7483 Test Plan: make check Reviewed By: anand1976 Differential Revision: D24142725 Pulled By: riversand963 fbshipit-source-id: 5629556dbe29f00dd09e30a7d5df5e6cf09ee435

view details

anand76

commit sha a242a5830116c2fdd476af79873297a7e0d02410

Enable ASSERT_STATUS_CHECKED for db_universal_compaction_test (#7460) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7460 Reviewed By: riversand963 Differential Revision: D24057636 Pulled By: anand1976 fbshipit-source-id: bfb13da6993a5e407be20073e4d6751dfb38e442

view details

peterpaule

commit sha 60649aa01b65ecb998d3f19c0327a5ce515455a8

Fix wrong comments about function TruncateToPageBoundary. (#6975) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6975 Reviewed By: ajkr Differential Revision: D22914797 Pulled By: riversand963 fbshipit-source-id: 5be2cd322d41447f638dba1336e96dcdc090f9dd

view details

Jay Zhuang

commit sha 8891e9a0eb302c980792619fee641f31ecf19c86

Disallow trivial move if BottommostLevelCompaction is kForce* (#7368) Summary: If `BottommostLevelCompaction.kForce*` is set, compaction should avoid trivial move and always compact the sst to the target size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7368 Reviewed By: ajkr Differential Revision: D23629525 Pulled By: jay-zhuang fbshipit-source-id: 79f23c79ecb31587e0593b28cce43131107bbcd0

view details

Akanksha Mahajan

commit sha 38d0a365e3d57003b11d34a92a65299a13b46699

Add Stats for MultiGet (#7366) Summary: Add following stats for MultiGet in Histogram to get more insight on MultiGet. 1. Number of index and filter blocks read from file as part of MultiGet request per level. 2. Number of data blocks read from file per level. 3. Number of SST files loaded from file system per level. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7366 Reviewed By: anand1976 Differential Revision: D24127040 Pulled By: akankshamahajan15 fbshipit-source-id: e63a003056b833729b277edc0639c08fb432756b

view details

Levi Tamasi

commit sha 22655a398b840978bdff49ccdca7d26c947ed416

Introduce a blob file reader class (#7461) Summary: The patch adds a class called `BlobFileReader` that can be used to retrieve blobs using the information available in blob references (e.g. blob file number, offset, and size). This will come in handy when implementing blob support for `Get`, `MultiGet`, and iterators, and also for compaction/garbage collection. When a `BlobFileReader` object is created (using the factory method `Create`), it first checks whether the specified file is potentially valid by comparing the file size against the combined size of the blob file header and footer (files smaller than the threshold are considered malformed). Then, it opens the file, and reads and verifies the header and footer. The verification involves magic number/CRC checks as well as checking for unexpected header/footer fields, e.g. incorrect column family ID or TTL blob files. Blobs can be retrieved using `GetBlob`. `GetBlob` validates the offset and compression type passed by the caller (because of the presence of the header and footer, the specified offset cannot be too close to the start/end of the file; also, the compression type has to match the one in the blob file header), and retrieves and potentially verifies and uncompresses the blob. In particular, when `ReadOptions::verify_checksums` is set, `BlobFileReader` reads the blob record header as well (as opposed to just the blob itself) and verifies the key/value size, the key itself, as well as the CRC of the blob record header and the key/value pair. In addition, the patch exposes the compression type from `BlobIndex` (both using an accessor and via `DebugString`), and adds a blob file read latency histogram to `InternalStats` that can be used with `BlobFileReader`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7461 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23999219 Pulled By: ltamasi fbshipit-source-id: deb6b1160d251258b308d5156e2ec063c3e12e5e

view details

Levi Tamasi

commit sha 1f84611e5dce7ccbc4545454dc25047e6c6bfe90

Clean up BlobLogReader and rename it to BlobLogSequentialReader (#7517) Summary: The patch does some cleanup in and around the legacy `BlobLogReader` class: * It renames the class to `BlobLogSequentialReader` to emphasize that it is for sequentially iterating through blobs in a blob file, as opposed to doing random point reads using `BlobIndex`es (which is `BlobFileReader`'s jurisdiction). * It removes some dead code from the old BlobDB implementation that references `BlobLogReader` (namely the method `BlobFile::OpenRandomAccessReader`). * It cleans up some `#include`s and forward declarations. * It fixes some incorrect/outdated comments related to the reader class. * It adds a few assertions to the `Read` methods of the class. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7517 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24172611 Pulled By: ltamasi fbshipit-source-id: 43e2ae1eba5c3dd30c1070cb00f217edc45bd64f

view details

Yanqin Jin

commit sha 002b30c967a077464a560775434d8493b64fca9f

Fix clang analyzer (#7518) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7518 Test Plan: ``` $USE_CLANG=1 make analyze ``` Reviewed By: zhichao-cao Differential Revision: D24175390 Pulled By: riversand963 fbshipit-source-id: c70121652908cf5d450120c38ab65cc595332ca7

view details

Zhichao Cao

commit sha 41462768859fe16c0aecfbe4c7fc38026fb3a486

Add ldb_cmd_test to ASSERT_STATUS_CHECKED list (#7499) Summary: Add ldb_cmd_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7499 Test Plan: pass ASSERT_STATUS_CHECKED=1 make -j48 ldb_cmd_test Reviewed By: cheng-chang Differential Revision: D24086203 Pulled By: zhichao-cao fbshipit-source-id: 29592202b1d4335e566de15e7937269d98d57841

view details

Jay Zhuang

commit sha 98c133380604b0dc7dbcf161b63c2af80d63c169

Disable a known flaky test: RandomAccessUniqueIDDeletes (#7511) Summary: It's a known issue, which is tracked in https://github.com/facebook/rocksdb/issues/7405, https://github.com/facebook/rocksdb/issues/7470. Disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7511 Reviewed By: zhichao-cao Differential Revision: D24145075 Pulled By: jay-zhuang fbshipit-source-id: 1858497972f2baba617867aaeac30d93b8305c80

view details

Levi Tamasi

commit sha 810ab34ede79f9fea6ad1bd751366846c875adf8

Do not rely on the two-argument std::pair constructor being constexpr (#7519) Summary: The `std::pair(const T1& x, const T2& y);` constructor is `constexpr` only starting from C++14; relying on this breaks compilation on certain compilers/platforms we need to support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7519 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D24195747 Pulled By: ltamasi fbshipit-source-id: 665e8fbc9747675bb49c5d895aad3dcf2714750f

view details

Akanksha Mahajan

commit sha 24498ab1eceeab5b1eba65f64a9b78a272efc42f

Add few unit test cases in ASSERT_STATUS_CHECKED (#7500) Summary: Add status enforcement for following tests: 1. import_column_family_test 2. memory_test 3. table_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7500 Reviewed By: zhichao-cao Differential Revision: D24095887 Pulled By: akankshamahajan15 fbshipit-source-id: db8e1ec595852df143fad78a0c07bfdd27dc3c84

view details

sdong

commit sha 82d42606ad2d1f576f25aac0a90c5d854c4f7af3

Enable paranoid_file_checks in crash test (#7489) Summary: Cover paranoid_file_checks in crash test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7489 Test Plan: Run crash tests for hours and didn't see any failure. Reviewed By: ajkr Differential Revision: D24063868 fbshipit-source-id: 7b48b110e66ce78ae5d0c99a9f32af86edd34c1e

view details

push time in 19 days

PullRequestReviewEvent
PullRequestReviewEvent

create barnchfacebook/rocksdb

branch : 6.14.fb

created branch time in 22 days

push eventakankshamahajan15/rocksdb

push time in 22 days

PR opened facebook/rocksdb

Update release history 6.14

Summary: Update release history for 6.14

Test Plan: No code change

Reviewers:

Subscribers:

Tasks:

Tags:

+2 -2

0 comment

2 changed files

pr created time in 22 days

create barnchakankshamahajan15/rocksdb

branch : release_history_6.14

created branch time in 22 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 62a7f30add4807d42bd1286ec144fcd88da9f638

Update release version to 6.14 Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in 22 days

push eventakankshamahajan15/rocksdb

anand76

commit sha a242a5830116c2fdd476af79873297a7e0d02410

Enable ASSERT_STATUS_CHECKED for db_universal_compaction_test (#7460) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7460 Reviewed By: riversand963 Differential Revision: D24057636 Pulled By: anand1976 fbshipit-source-id: bfb13da6993a5e407be20073e4d6751dfb38e442

view details

peterpaule

commit sha 60649aa01b65ecb998d3f19c0327a5ce515455a8

Fix wrong comments about function TruncateToPageBoundary. (#6975) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6975 Reviewed By: ajkr Differential Revision: D22914797 Pulled By: riversand963 fbshipit-source-id: 5be2cd322d41447f638dba1336e96dcdc090f9dd

view details

Jay Zhuang

commit sha 8891e9a0eb302c980792619fee641f31ecf19c86

Disallow trivial move if BottommostLevelCompaction is kForce* (#7368) Summary: If `BottommostLevelCompaction.kForce*` is set, compaction should avoid trivial move and always compact the sst to the target size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7368 Reviewed By: ajkr Differential Revision: D23629525 Pulled By: jay-zhuang fbshipit-source-id: 79f23c79ecb31587e0593b28cce43131107bbcd0

view details

Akanksha Mahajan

commit sha 38d0a365e3d57003b11d34a92a65299a13b46699

Add Stats for MultiGet (#7366) Summary: Add following stats for MultiGet in Histogram to get more insight on MultiGet. 1. Number of index and filter blocks read from file as part of MultiGet request per level. 2. Number of data blocks read from file per level. 3. Number of SST files loaded from file system per level. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7366 Reviewed By: anand1976 Differential Revision: D24127040 Pulled By: akankshamahajan15 fbshipit-source-id: e63a003056b833729b277edc0639c08fb432756b

view details

Levi Tamasi

commit sha 22655a398b840978bdff49ccdca7d26c947ed416

Introduce a blob file reader class (#7461) Summary: The patch adds a class called `BlobFileReader` that can be used to retrieve blobs using the information available in blob references (e.g. blob file number, offset, and size). This will come in handy when implementing blob support for `Get`, `MultiGet`, and iterators, and also for compaction/garbage collection. When a `BlobFileReader` object is created (using the factory method `Create`), it first checks whether the specified file is potentially valid by comparing the file size against the combined size of the blob file header and footer (files smaller than the threshold are considered malformed). Then, it opens the file, and reads and verifies the header and footer. The verification involves magic number/CRC checks as well as checking for unexpected header/footer fields, e.g. incorrect column family ID or TTL blob files. Blobs can be retrieved using `GetBlob`. `GetBlob` validates the offset and compression type passed by the caller (because of the presence of the header and footer, the specified offset cannot be too close to the start/end of the file; also, the compression type has to match the one in the blob file header), and retrieves and potentially verifies and uncompresses the blob. In particular, when `ReadOptions::verify_checksums` is set, `BlobFileReader` reads the blob record header as well (as opposed to just the blob itself) and verifies the key/value size, the key itself, as well as the CRC of the blob record header and the key/value pair. In addition, the patch exposes the compression type from `BlobIndex` (both using an accessor and via `DebugString`), and adds a blob file read latency histogram to `InternalStats` that can be used with `BlobFileReader`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7461 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23999219 Pulled By: ltamasi fbshipit-source-id: deb6b1160d251258b308d5156e2ec063c3e12e5e

view details

Levi Tamasi

commit sha 1f84611e5dce7ccbc4545454dc25047e6c6bfe90

Clean up BlobLogReader and rename it to BlobLogSequentialReader (#7517) Summary: The patch does some cleanup in and around the legacy `BlobLogReader` class: * It renames the class to `BlobLogSequentialReader` to emphasize that it is for sequentially iterating through blobs in a blob file, as opposed to doing random point reads using `BlobIndex`es (which is `BlobFileReader`'s jurisdiction). * It removes some dead code from the old BlobDB implementation that references `BlobLogReader` (namely the method `BlobFile::OpenRandomAccessReader`). * It cleans up some `#include`s and forward declarations. * It fixes some incorrect/outdated comments related to the reader class. * It adds a few assertions to the `Read` methods of the class. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7517 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24172611 Pulled By: ltamasi fbshipit-source-id: 43e2ae1eba5c3dd30c1070cb00f217edc45bd64f

view details

Yanqin Jin

commit sha 002b30c967a077464a560775434d8493b64fca9f

Fix clang analyzer (#7518) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7518 Test Plan: ``` $USE_CLANG=1 make analyze ``` Reviewed By: zhichao-cao Differential Revision: D24175390 Pulled By: riversand963 fbshipit-source-id: c70121652908cf5d450120c38ab65cc595332ca7

view details

Zhichao Cao

commit sha 41462768859fe16c0aecfbe4c7fc38026fb3a486

Add ldb_cmd_test to ASSERT_STATUS_CHECKED list (#7499) Summary: Add ldb_cmd_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7499 Test Plan: pass ASSERT_STATUS_CHECKED=1 make -j48 ldb_cmd_test Reviewed By: cheng-chang Differential Revision: D24086203 Pulled By: zhichao-cao fbshipit-source-id: 29592202b1d4335e566de15e7937269d98d57841

view details

Jay Zhuang

commit sha 98c133380604b0dc7dbcf161b63c2af80d63c169

Disable a known flaky test: RandomAccessUniqueIDDeletes (#7511) Summary: It's a known issue, which is tracked in https://github.com/facebook/rocksdb/issues/7405, https://github.com/facebook/rocksdb/issues/7470. Disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7511 Reviewed By: zhichao-cao Differential Revision: D24145075 Pulled By: jay-zhuang fbshipit-source-id: 1858497972f2baba617867aaeac30d93b8305c80

view details

Levi Tamasi

commit sha 810ab34ede79f9fea6ad1bd751366846c875adf8

Do not rely on the two-argument std::pair constructor being constexpr (#7519) Summary: The `std::pair(const T1& x, const T2& y);` constructor is `constexpr` only starting from C++14; relying on this breaks compilation on certain compilers/platforms we need to support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7519 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D24195747 Pulled By: ltamasi fbshipit-source-id: 665e8fbc9747675bb49c5d895aad3dcf2714750f

view details

Akanksha Mahajan

commit sha 24498ab1eceeab5b1eba65f64a9b78a272efc42f

Add few unit test cases in ASSERT_STATUS_CHECKED (#7500) Summary: Add status enforcement for following tests: 1. import_column_family_test 2. memory_test 3. table_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7500 Reviewed By: zhichao-cao Differential Revision: D24095887 Pulled By: akankshamahajan15 fbshipit-source-id: db8e1ec595852df143fad78a0c07bfdd27dc3c84

view details

sdong

commit sha 82d42606ad2d1f576f25aac0a90c5d854c4f7af3

Enable paranoid_file_checks in crash test (#7489) Summary: Cover paranoid_file_checks in crash test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7489 Test Plan: Run crash tests for hours and didn't see any failure. Reviewed By: ajkr Differential Revision: D24063868 fbshipit-source-id: 7b48b110e66ce78ae5d0c99a9f32af86edd34c1e

view details

push time in 22 days

create barnchakankshamahajan15/rocksdb

branch : maste

created branch time in 22 days

push eventakankshamahajan15/rocksdb

Akanksha Mahajan

commit sha 7d503e66a9a9d9bdf17e2d35038bfe8f3e12f5dc

Return error if Get() fails in Prefetching Filter blocks (#7463) Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7463 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D24008226 Pulled By: akankshamahajan15 fbshipit-source-id: b65d63b2d01465db92500b78de7ad58650ec9b3b

view details

Yanqin Jin

commit sha 758ead5df771d8ac2972c9f2c4d6074c97272151

Enforce status check for corruption_test (#7453) Summary: Enforce status check for corruption_test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7453 Test Plan: ``` ASSERT_STATUS_CHECKED=1 make corruption_test ./corruption_test ``` Reviewed By: jay-zhuang Differential Revision: D24006862 Pulled By: riversand963 fbshipit-source-id: 664677caf4c3007a25cf565cec3d677f2dcea130

view details

Levi Tamasi

commit sha 5d16325ce33384de974560648434473031639018

Revert "Return error if Get() fails in Prefetching Filter blocks (#7463)" (#7505) Summary: This reverts commit 7d503e66a9a9d9bdf17e2d35038bfe8f3e12f5dc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7505 Reviewed By: ajkr Differential Revision: D24100875 Pulled By: ltamasi fbshipit-source-id: 8705e3e6e8be4b4fd175ffdb031baa6530b61151

view details

sdong

commit sha 2496d3d4b849af53ef3dbb03801c1e359f4d63ff

Avoid to suppress status code in ~BlockBasedTableBuilder (#7507) Summary: We just used a hacky way to fix db_basic_test: suppress status code in ~BlockBasedTableBuilder. Rather, we should pass them back in Finish() and suppress them in Abandon(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7507 Test Plan: Watch existing tests to succeed. Reviewed By: riversand963 Differential Revision: D24119527 fbshipit-source-id: 71c4d4a81c0fd1c5595224692275f20f7759973a

view details

Jay Zhuang

commit sha 53089038def1a5ffa05afb27beaa63ed64dff764

Fix StallWrite crash with mixed of slowdown/no_slowdown writes (#7508) Summary: `BeginWriteStall()` removes no_slowdown write from the write list and updates `link_newer`, which makes `CreateMissingNewerLinks()` thought all write list has valid `link_newer` and failed to create link for all writers. It caused flaky test and SegFault for release build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7508 Test Plan: Add unittest to reproduce the issue. Reviewed By: anand1976 Differential Revision: D24126601 Pulled By: jay-zhuang fbshipit-source-id: f8ac5dba653f7ee1b0950296427d4f5f8ee34a06

view details

Yanqin Jin

commit sha 1bcef3d83c668afa690dd40c89bb1e955be0b1da

Make sure assert(false) handles failures too (#7483) Summary: In opt mode, assertions are just no-ops. Therefore, we need to report errors instead of just doing an `assert(false)`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7483 Test Plan: make check Reviewed By: anand1976 Differential Revision: D24142725 Pulled By: riversand963 fbshipit-source-id: 5629556dbe29f00dd09e30a7d5df5e6cf09ee435

view details

anand76

commit sha a242a5830116c2fdd476af79873297a7e0d02410

Enable ASSERT_STATUS_CHECKED for db_universal_compaction_test (#7460) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7460 Reviewed By: riversand963 Differential Revision: D24057636 Pulled By: anand1976 fbshipit-source-id: bfb13da6993a5e407be20073e4d6751dfb38e442

view details

akankshamahajan

commit sha 3494d4bf6da4ff44d226ecd06738cf50a5780fcf

Add unit test cases in ASSERT_STATUS_CHECKED Summary: Add status enforcement for following tests: 1. import_column_family_test 2. memory_test 3. table_test Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

akankshamahajan

commit sha 45ab8af6d9f0f0079b01bea375f102df3b042112

Addressed comments Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in 24 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 1a6e51db2869c49ea54843e2a5b35b5832b95b8a

Fix build issues Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in 24 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 6ab6c9d5fc8625f47ad7b9b2fd1883dcfd96c0dd

Update IOTrace operations in stackabledb.h Summary: Update IOTrace operations in stackabledb.h and also trace few other IO operations. Test Plan: make check -j64 Reviewers: Subscribers: Tasks: Tags:

view details

push time in 24 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha dd62585594c8f3a42abec0eef8a38f4716c98d15

Update History Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in 24 days

push eventakankshamahajan15/rocksdb

Jay Zhuang

commit sha 53089038def1a5ffa05afb27beaa63ed64dff764

Fix StallWrite crash with mixed of slowdown/no_slowdown writes (#7508) Summary: `BeginWriteStall()` removes no_slowdown write from the write list and updates `link_newer`, which makes `CreateMissingNewerLinks()` thought all write list has valid `link_newer` and failed to create link for all writers. It caused flaky test and SegFault for release build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7508 Test Plan: Add unittest to reproduce the issue. Reviewed By: anand1976 Differential Revision: D24126601 Pulled By: jay-zhuang fbshipit-source-id: f8ac5dba653f7ee1b0950296427d4f5f8ee34a06

view details

Yanqin Jin

commit sha 1bcef3d83c668afa690dd40c89bb1e955be0b1da

Make sure assert(false) handles failures too (#7483) Summary: In opt mode, assertions are just no-ops. Therefore, we need to report errors instead of just doing an `assert(false)`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7483 Test Plan: make check Reviewed By: anand1976 Differential Revision: D24142725 Pulled By: riversand963 fbshipit-source-id: 5629556dbe29f00dd09e30a7d5df5e6cf09ee435

view details

anand76

commit sha a242a5830116c2fdd476af79873297a7e0d02410

Enable ASSERT_STATUS_CHECKED for db_universal_compaction_test (#7460) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7460 Reviewed By: riversand963 Differential Revision: D24057636 Pulled By: anand1976 fbshipit-source-id: bfb13da6993a5e407be20073e4d6751dfb38e442

view details

akankshamahajan

commit sha 1586b1ccf2309d3be355815ef5f455d918faa28b

Add Stats for MultiGet Summary: Add following stats for MultiGet in Histogram to get more insight on MultiGet. 1. Number of index and filter blocks read from file as part of MultiGet request per level. 2. Number of data blocks read from file per level. 3. Number of SST files loaded from file system per level. Test Plan: Reviewers: Subscribers: Tasks: T69090067 Tags:

view details

akankshamahajan

commit sha 372715ff2b9090265e67dbeaaa01d8ea1a860857

Add unit test Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in 24 days

PR opened facebook/rocksdb

Update IOTrace operations in stackable_db.h

Summary: Update IOTrace operations in stackabledb.h and also trace few other IO operations.

Test Plan: make check -j64

Reviewers:

Subscribers:

Tasks:

Tags:

+177 -3

0 comment

4 changed files

pr created time in 25 days

push eventakankshamahajan15/rocksdb

sdong

commit sha 43cc622d099fc9f1244e39c671d4d75c09984ca8

Add CLANG analyze to CircleCI (#7114) Summary: CLANG analyze is useful before pull request. Add it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7114 Test Plan: Watch the CI results to succeed. Reviewed By: riversand963 Differential Revision: D22491942 fbshipit-source-id: 9ccad91c6142fedc3d3dd491cf55054827908f36

view details

Jay Zhuang

commit sha 0ff752cf0dc678efb116eeb23425237057cea556

Add circleci java build (#7119) Summary: Add circleci java build workflow. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7119 Reviewed By: siying Differential Revision: D22512426 Pulled By: jay-zhuang fbshipit-source-id: 45a7445c861fee48017bae42cdb7172c8b091475

view details

Akanksha Mahajan

commit sha d93bd3ce2534196f4e34a4513c355c16ad7417ab

Add FileSystem wrapper classes for IO tracing. (#7002) Summary: 1. Add the wrapper classes FileSystemTracingWrapper, FSSequentialFileTracingWrapper, FSRandomAccessFileTracingWrapper, FSWritableFileTracingWrapper, FSRandomRWFileTracingWrapper that forward the calls to underlying storage system and then pass the file operation information to IOTracer. IOTracer dumps the record in binary format for tracing. 2. Add the wrapper classes FileSystemPtr, FSSequentialFilePtr, FSRandomAccessFilePtr, FSWritableFilePtr and FSRandomRWFilePtr that overload operator-> and return ptr to underlying storage system or Tracing wrapper class based on enabling/disabling of IO tracing. These classes are added to bypass Tracing Wrapper classes when we disable tracing. 3. Add enums in trace.h that distinguish which options need to be added for different file operations(Read, close, write etc) as part of tracing record. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7002 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D22127897 Pulled By: akankshamahajan15 fbshipit-source-id: 74cff58ce5661c9a3832dfaa52483f3b2d8565e0

view details

sdong

commit sha 2f32d504692dcf7818c9507a78a5009a735297b1

Add VS2017 to CircleCI Runs (#7120) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7120 Test Plan: Watch CI results. Reviewed By: cheng-chang Differential Revision: D22516995 fbshipit-source-id: ddf7c0482dfac30d9044b833852348eda987cb67

view details

Yanqin Jin

commit sha c628fae6d1e05163de321b06fcc7bde34989a129

Report corruption on unrecognized value type (#7121) Summary: During memtable lookup, an unrecognized value type should be reported as Status::Corruption. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7121 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D22512124 Pulled By: riversand963 fbshipit-source-id: 9b97be7d9b230c5aae9205f96054420e5ea09066

view details

Jay Zhuang

commit sha fc4d5f5065d3e9059af7453295007dddacff3d1c

Add stress test for GetProperty (#7111) Summary: Add stress test coverage for `DB::GetProperty()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7111 Test Plan: ``` ./db_stress -get_property_one_in=1 make crash_test ``` Reviewed By: ajkr Differential Revision: D22487906 Pulled By: jay-zhuang fbshipit-source-id: c118d95cc9b4e2fa669a06e6aa531541fa885dc5

view details

Levi Tamasi

commit sha bdf4de6cb90c4f2a8e87f706bd85d8e57ba7eb80

Remove some dead code from BlobLogWriter (#7125) Summary: Periodic syncing of blob files is performed by `WritableFileWriter`; `bytes_per_sync_` and `next_sync_offset_` in `BlobLogWriter` are actually unused (or more precisely, only used by methods that are themselves unused). The patch removes all this dead code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7125 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D22531021 Pulled By: ltamasi fbshipit-source-id: 6b293ad5a79d3e6bf15c5c68f7aedd7ce7a15f10

view details

sdong

commit sha e930ba1d18c83a241b4dc77b23d4b3c152d73b86

Run a subset of TSAN tests in CIrcleCI (#7122) Summary: It is helpful to add some TSAN coverage before a pull request is committed. This diff adds some of them. Some slow tests are excluded for the running speed. Some are blacklisted because they show warnings. Will investigate these warnings and see whether we can fix or suppress them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7122 Test Plan: Watch CIrcleCI runs Reviewed By: riversand963 Differential Revision: D22532133 fbshipit-source-id: 81ddd02d9df19c513a12811979e8ddabae911354

view details

Adam Retter

commit sha de8c92a5967f64acfcf4bc93b2eb789d81b4c0a8

Only check for python location once (#7123) Summary: This fixes an issue introduced in 0c56fc4 whereby the location of Python is evaluated many times and leads to excessive logging of unknown python locations of CentOS 6. The location is now only checked once. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7123 Reviewed By: zhichao-cao Differential Revision: D22532274 Pulled By: ajkr fbshipit-source-id: cade71b4b46e9a23d63ecb4dd36a4ac8ae217970

view details

Levi Tamasi

commit sha 687fbd0270b92fab5958d7a8f2a45f1feb763df1

Update some log messages in BlobDB to account for compaction filters (#7128) Summary: https://github.com/facebook/rocksdb/pull/6850, which added compaction filter support to BlobDB, reused elements of the BlobDB GC mechanism. This patch updates some log messages in this logic to account for this fact; namely, it replaces mentions of "GC" with "compaction/GC" to avoid confusion in cases when GC is not enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7128 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D22535371 Pulled By: ltamasi fbshipit-source-id: 1f14f3b02ab9983728bbca1cf680420208d9a195

view details

Yanqin Jin

commit sha 27735dea9adfe95fdf282b4ba41b790f1969baa6

Report corrupted keys during compaction (#7124) Summary: Currently, RocksDB lets compaction to go through even in case of corrupted keys, the number of which is reported in CompactionJobStats. However, RocksDB does not check this value. We should let compaction run in a stricter mode. Temporarily disable two tests that allow corrupted keys in compaction. With this PR, the two tests will assert(false) and terminate. Still need to investigate what is the recommended google-test way of doing it. Death test (EXPECT_DEATH) in gtest has warnings now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7124 Test Plan: make check Reviewed By: ajkr Differential Revision: D22530722 Pulled By: riversand963 fbshipit-source-id: 6a5a6a992028c6d4f92cb74693c92db462ae4ad6

view details

yxj25245

commit sha e8d5a248155268d5d5473a45c505fa193d31741a

Fix typo in ThreadData comment (#7131) Summary: Fix typo in ThreadData comment Pull Request resolved: https://github.com/facebook/rocksdb/pull/7131 Reviewed By: riversand963 Differential Revision: D22543135 Pulled By: jay-zhuang fbshipit-source-id: 39c9d0e8cd5a364af9a2f05fd3783e8482dea976

view details

Zhichao Cao

commit sha a10f12eda1c5800b3bcf3f135904b574cbe9ed51

Auto resume the DB from Retryable IO Error (#6765) Summary: In current codebase, in write path, if Retryable IO Error happens, SetBGError is called. The retryable IO Error is converted to hard error and DB is in read only mode. User or application needs to resume it. In this PR, if Retryable IO Error happens in one DB, SetBGError will create a new thread to call Resume (auto resume). otpions.max_bgerror_resume_count controls if auto resume is enabled or not (if max_bgerror_resume_count<=0, auto resume will not be enabled). options.bgerror_resume_retry_interval controls the time interval to call Resume again if the previous resume fails due to the Retryable IO Error. If non-retryable error happens during resume, auto resume will terminate. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6765 Test Plan: Added the unit test cases in error_handler_fs_test and pass make asan_check Reviewed By: anand1976 Differential Revision: D21916789 Pulled By: zhichao-cao fbshipit-source-id: acb8b5e5dc3167adfa9425a5b7fc104f6b95cb0b

view details

Jay Zhuang

commit sha afb6bb1df231764a2ad4ea7615f289d21a765f5e

CircleCI: Pipe java-build to ignore EAGAIN errors (#7135) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7135 Reviewed By: siying Differential Revision: D22553594 Pulled By: jay-zhuang fbshipit-source-id: 5b40621ffad05c4fffe3ac536a11d6c87e7ef512

view details

Levi Tamasi

commit sha ee8c79d40d8bcbd13ce6e479fe36851da97947ac

Turn the compression_type check in BlobDBImpl::DecompressSlice into an assertion (#7127) Summary: In both cases where `BlobDBImpl::DecompressSlice` is called, `compression_type` is already checked at the call site; thus, the check inside the method is redundant and can be turned into an assertion. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7127 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D22533454 Pulled By: ltamasi fbshipit-source-id: ae524443fc6abe0a5fb12327a3fe761a9cd2c831

view details

sdong

commit sha ca5a069a79e5a03a5c9e6bff196c016285e6bbd1

Suppress a TSAN warning (#7126) Summary: TSAN shows warning with clang with warning similar to this: WARNING: ThreadSanitizer: data race (pid=10159) Atomic write of size 8 at 0x7b5000002890 by thread T33: #0 __tsan_atomic64_store <null> (db_test+0x4ca2b5) https://github.com/facebook/rocksdb/issues/1 std::__atomic_base<unsigned long>::store(unsigned long, std::memory_order) /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/atomic_base.h:374:2 (db_test+0x774fde) https://github.com/facebook/rocksdb/issues/2 rocksdb::VersionSet::SetLastSequence(unsigned long) /home/circleci/project/./db/version_set.h:1057:20 (db_test+0x774fde) https://github.com/facebook/rocksdb/issues/3 rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*) /home/circleci/project/db/db_impl/db_impl_write.cc:449:18 (db_test+0x774fde) ...... Previous read of size 8 at 0x7b5000002890 by thread T5 (mutexes: write M1044689462619020832): #0 rocksdb::DBImpl::ReleaseSnapshot(rocksdb::Snapshot const*) /home/circleci/project/db/db_impl/db_impl.cc (db_test+0x6f4ae7) https://github.com/facebook/rocksdb/issues/1 rocksdb::(anonymous namespace)::MTThreadBody(void*) /home/circleci/project/db/db_test.cc:2514:13 (db_test+0x56ac59) https://github.com/facebook/rocksdb/issues/2 rocksdb::(anonymous namespace)::StartThreadWrapper(void*) /home/circleci/project/env/env_posix.cc:443:3 (db_test+0x88c4cd) It is not limited to ReleaseSnapshot() and rocksdb::DBImpl::MultiCFSnapshot(). While we are not 100% sure it doesn't indicate any correctness violation, we suppress them for now to keep TSAN clean with more tests so that we can cover more bugs with CI. In the gcc runs we have been running, this warning rarely shows up. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7126 Test Plan: See the mini-TSAN test to pass with reasonable run time. Reviewed By: ajkr Differential Revision: D22552375 fbshipit-source-id: ebdd3854cb3becec3403970326a1ca961db2ab00

view details

Akanksha Mahajan

commit sha a7feebd67096496abaf631d434113d5654c01257

Add "build-examples" in CircleCI (#7136) Summary: Add "examples" build (which build examples folder in rocksdb) in TravisCI to CircleCI. This is helpful before pull request. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7136 Test Plan: Watch for CircleCI results to succeed Reviewed By: jay-zhuang Differential Revision: D22555528 Pulled By: akankshamahajan15 fbshipit-source-id: 6bca16647760d5f0131f064765fe9e88e034c578

view details

Levi Tamasi

commit sha c5ddeceba0404d2f64a1187e1f7eb55d7b70df49

Remove some more dead code around syncing blob files (#7138) Summary: Periodic syncing of blob files is handled by a lower layer, namely by `WritableFileWriter`; the `NeedsFsync` method of `BlobFile` and the `last_fsync_` member variable are actually unused and thus can be removed. See also https://github.com/facebook/rocksdb/pull/7125 . Pull Request resolved: https://github.com/facebook/rocksdb/pull/7138 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D22562981 Pulled By: ltamasi fbshipit-source-id: c235aad94a7c27120528c9ec270a7a5b9154e49f

view details

mrambacher

commit sha ec711b23152ca4e426057557784af322a2065fbc

Add Support for saving CompressionOptions to Options File (#6817) Summary: This PR does a few things: - The "compression_opts" and "bottom_compression_opts" can now be read/written as name/value pairs of options (instead of only a colon-separated list; - These options can now be read/written to the Options file; - The parallel_threads value can now be set (either in the colon or name-value format). The compression options are now stored and treated as a OptionTypeInfo::Struct by the options system, meaning they can be read and written like the other structs. This change allows them to be read/written easily to the options file. Additionally, the colon-format was extended to allow support for setting parallel threads. Tests were added to test all of the option settings via the optional parameters in the colon format. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6817 Reviewed By: ajkr Differential Revision: D22396004 Pulled By: zhichao-cao fbshipit-source-id: 38bcf74b7e9cd5bc2a84540fac2e9ba4f765b2c8

view details

Andrew Kryczka

commit sha 9a83fd21e6e668b0ad9f6b4473f53e50e34ea76d

stagger first DumpMallocStats after opening DB (#7145) Summary: Previously when running `db_bench` with large value for `num_multi_dbs` and enabled `Options::dump_malloc_stats`, we would see most CPU spent in jemalloc locking. After this PR that no longer shows up at the top of the profile. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7145 Reviewed By: riversand963 Differential Revision: D22593031 Pulled By: ajkr fbshipit-source-id: 3b3fc91f93249c6afee53f59f34c487c3fc5add6

view details

push time in 25 days

push eventakankshamahajan15/rocksdb

sdong

commit sha 2496d3d4b849af53ef3dbb03801c1e359f4d63ff

Avoid to suppress status code in ~BlockBasedTableBuilder (#7507) Summary: We just used a hacky way to fix db_basic_test: suppress status code in ~BlockBasedTableBuilder. Rather, we should pass them back in Finish() and suppress them in Abandon(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7507 Test Plan: Watch existing tests to succeed. Reviewed By: riversand963 Differential Revision: D24119527 fbshipit-source-id: 71c4d4a81c0fd1c5595224692275f20f7759973a

view details

Jay Zhuang

commit sha 53089038def1a5ffa05afb27beaa63ed64dff764

Fix StallWrite crash with mixed of slowdown/no_slowdown writes (#7508) Summary: `BeginWriteStall()` removes no_slowdown write from the write list and updates `link_newer`, which makes `CreateMissingNewerLinks()` thought all write list has valid `link_newer` and failed to create link for all writers. It caused flaky test and SegFault for release build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7508 Test Plan: Add unittest to reproduce the issue. Reviewed By: anand1976 Differential Revision: D24126601 Pulled By: jay-zhuang fbshipit-source-id: f8ac5dba653f7ee1b0950296427d4f5f8ee34a06

view details

Yanqin Jin

commit sha 1bcef3d83c668afa690dd40c89bb1e955be0b1da

Make sure assert(false) handles failures too (#7483) Summary: In opt mode, assertions are just no-ops. Therefore, we need to report errors instead of just doing an `assert(false)`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7483 Test Plan: make check Reviewed By: anand1976 Differential Revision: D24142725 Pulled By: riversand963 fbshipit-source-id: 5629556dbe29f00dd09e30a7d5df5e6cf09ee435

view details

push time in 25 days

Pull request review commentfacebook/rocksdb

Add Stats for MultiGet

 Status BlockBasedTable::RetrieveBlock(         GetMemoryAllocator(rep_->table_options), for_compaction,         rep_->blocks_definitely_zstd_compressed,         rep_->table_options.filter_policy.get());++    if (get_context) {

Thanks for clarifying. I have unit test ready. I will upload it by EOD.

akankshamahajan15

comment created time in 25 days

PullRequestReviewEvent

Pull request review commentfacebook/rocksdb

Add Stats for MultiGet

 Status BlockBasedTable::RetrieveBlock(         GetMemoryAllocator(rep_->table_options), for_compaction,         rep_->blocks_definitely_zstd_compressed,         rep_->table_options.filter_policy.get());++    if (get_context) {

@anand1976 In TableCache::MultiGet, we first get the TableReader and then call the BlockBasedTable::MultiGet. In TableCahe::GetTableReader flow, it creates Index reader which call RetrieveBlock for index block type with get_context = nullptr. So that won't be logged as of now. Is this ok for the experiment or that should also be recorded?

akankshamahajan15

comment created time in 25 days

PullRequestReviewEvent

pull request commentfacebook/rocksdb

Add Stats for MultiGet

Addressed comments

akankshamahajan15

comment created time in 25 days

push eventakankshamahajan15/rocksdb

sdong

commit sha 2496d3d4b849af53ef3dbb03801c1e359f4d63ff

Avoid to suppress status code in ~BlockBasedTableBuilder (#7507) Summary: We just used a hacky way to fix db_basic_test: suppress status code in ~BlockBasedTableBuilder. Rather, we should pass them back in Finish() and suppress them in Abandon(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7507 Test Plan: Watch existing tests to succeed. Reviewed By: riversand963 Differential Revision: D24119527 fbshipit-source-id: 71c4d4a81c0fd1c5595224692275f20f7759973a

view details

akankshamahajan

commit sha a8d319cf94e8f39fe13f9e3a4460c6cd58c97d60

Add Stats for MultiGet Summary: Add following stats for MultiGet in Histogram to get more insight on MultiGet. 1. Number of index and filter blocks read from file as part of MultiGet request per level. 2. Number of data blocks read from file per level. 3. Number of SST files loaded from file system per level. Test Plan: Reviewers: Subscribers: Tasks: T69090067 Tags:

view details

push time in 25 days

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 74b5d92c765975e3da0d7ba1cf0712962f7e15a6

Add Stats for MultiGet Summary: Add following stats for MutliGet in Histogram to get more insight on MultiGet. 1. Number of index and filter blocks read from file as part of MultiGet request per level. 2. Number of data blocks read from file per level. 3. Number of SST files loaded from file system per level. Test Plan: Reviewers: Subscribers: Tasks: T69090067 Tags:

view details

push time in a month

push eventakankshamahajan15/rocksdb

Peter Dillinger

commit sha 9de912de3f96f5bf0b5c96d62e64dc07fd202b5d

Fix some errors showing up in Travis builds (#7359) Summary: Also enables a pull request to trigger all the Travis configurations by writing FULL_CI in the commit message. (See what I did there?) First issue make: *** No rule to make target 'jl/util/crc32c_ppc_asm.o', needed by 'rocksdbjava'. Stop. Second issue tools/db_bench_tool.cc:5514:38: error: ‘gen_exp.rocksdb::Benchmark::GenerateTwoTermExpKeys::keyrange_size_’ may be used uninitialized in this function Pull Request resolved: https://github.com/facebook/rocksdb/pull/7359 Test Plan: CI Reviewed By: zhichao-cao Differential Revision: D23582132 Pulled By: pdillinger fbshipit-source-id: 06d794673fd522ba11cf6398385387e6bd97ef89

view details

Jay Zhuang

commit sha f1e99b36f55a1861c6fbb711666450f6ec776996

tests need linked with third_party libs (#7351) Summary: To fix the cmake build with third_party libs, like: `mkdir build && cd build && cmake .. -DWITH_SNAPPY=1 && make` Error: ``` Undefined symbols for architecture x86_64: "snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)" ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7351 Reviewed By: pdillinger Differential Revision: D23553705 Pulled By: jay-zhuang fbshipit-source-id: 19b45c6763c7256107583e8af4c01d370ca06128

view details

mrambacher

commit sha a6ac51b99ab3f1fa1846796a928db25820efae37

Fix db_bench_tool_test. Fixes 7341 (#7344) Summary: 1. Failed to compile because of use of FileSystem* instead of Env* to some methods; 2. Failed to compile with addition of ConfigOptions to some methods 3. Failed to run successfully because the database and/or db_bench would change some of the options, invalidating the comparison 4. Failed to run successfully if Snappy was not available. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7344 Reviewed By: siying Differential Revision: D23501093 Pulled By: jay-zhuang fbshipit-source-id: 81fd947e95fff9db8a4c5ff419d69d4c36bef23f

view details

Peter Dillinger

commit sha a7fde8727beee384c51ce32f188f8d5ef8436ee8

Fix platform_dependent in Travis, rebalance groups (#7360) Summary: Was broken by https://github.com/facebook/rocksdb/issues/6660 Travis times before this change, after 6660: platform_dependent: 17 min group 1: 15 min group 2: 44 min (often timeout on non-x86 or non-Linux) group 3: 31 min group 4: 21 min After this change: TODO Pull Request resolved: https://github.com/facebook/rocksdb/pull/7360 Test Plan: CI inspection Reviewed By: ajkr Differential Revision: D23586917 Pulled By: pdillinger fbshipit-source-id: 4c67cf33180b0b833c39a817e6c1f128727941d2

view details

Levi Tamasi

commit sha 7b1d6c438a71bab04680e066c101eabdac14540a

Fix the handling of the case when a blob file with a lower number gets added in VersionBuilder (#7349) Summary: When multiple background jobs are generating blob files in parallel, it is actually possible for a blob file to be added with a file number that is lower than the highest one in the base version. (This is a harmless race condition.) The patch fixes the handling of this case and adds a unit test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7349 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23542453 Pulled By: ltamasi fbshipit-source-id: 4ff6f3654bc58c391d10b9870e1cc40b5e3fa8e4

view details

Peter Dillinger

commit sha e3149358a569d515289ba07648a9b710da4698fd

More backup/restore stress test fixes (#7361) Summary: (a) Missed a case in updating handling of rand_keys (b) Only opening restored db with DB::Open so don't (yet) attempt to open restored BlobDB or TransactionDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7361 Test Plan: better than being broken Reviewed By: ajkr Differential Revision: D23592570 Pulled By: pdillinger fbshipit-source-id: dd1d999bcc0c852ee77cb6041964ec4abc0fd4fd

view details

Stanislav Tkach

commit sha 5c39d8df691a3096cf826ff639773a3dd574fcd2

Add getters to the C API for flush, write, cache and compact options (#7321) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7321 Reviewed By: ajkr Differential Revision: D23590160 fbshipit-source-id: 35d106e732ac37f674222759cdb1dbb31e005ca7

view details

Tomasz Posluszny

commit sha ec5add398c27b07595f583f1834da78b51490906

Implement Java API for ConcurrentTaskLimiter class and compaction_thread_limiter field in ColumnFamilyOptions (#7347) Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7347 Test Plan: unit tests included Reviewed By: jay-zhuang Differential Revision: D23592552 Pulled By: pdillinger fbshipit-source-id: 1c3571b6f42bfd0cfd723ff49d01fbc02a1be45b

view details

Levi Tamasi

commit sha 5ce246c716bbe7787150e3ced964b27a93995f6b

Expose the start of the expiration range for TTL blob files through LiveFileMetaData (#7365) Summary: The patch adds support for exposing the start of the expiration range for TTL blob files through the `GetLiveFilesMetaData` API. This can be used for monitoring purposes, i.e. to make sure TTL blob files are deleted in a timely manner. The patch also fixes a couple of uninitialized variable issues. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7365 Test Plan: `make check` Reviewed By: pdillinger Differential Revision: D23605465 Pulled By: ltamasi fbshipit-source-id: 97a9612bf5f4b058423debdd3f28f576bb23a70f

view details

Peter Dillinger

commit sha 92639b93a67c7ec28fd7d2de611eac759120ffce

Fix checkpoint file deletion race with avoid_unnecessary_blocking_io (#7369) Summary: https://github.com/facebook/rocksdb/issues/3341 guaranteed that upon return of `GetSortedWalFiles` after `DisableFileDeletions`, all pending purges of previously obsolete WAL files will have finished. However, the addition of avoid_unnecessary_blocking_io in https://github.com/facebook/rocksdb/issues/5043 opened a hole in the code making that assurance, which can lead to files to be copied for checkpoint or backup going missing before being copied, with that option enabled. This change patches the hole. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7369 Test Plan: apparent fix to backups in crash test observed. Will work on a unit test for another commit Reviewed By: ajkr Differential Revision: D23620258 Pulled By: pdillinger fbshipit-source-id: bea36b461a5b719c3e3ef802f967bc3e8ae71614

view details

Peter Dillinger

commit sha c4e2066dbd52696215c12adfae1fdee8b2b21744

Fix cf_consistency_stress for backup/restore, harmonize (#7373) Summary: We can only check key on restored backup if in a stress test configuration locking the key. (Fixes mismatch seen in backup/restore with atomic flush.) TestCheckpoint used a very ugly solution to the same problem: copy-paste dozens of lines of code with some changes and removals. I removed the unnecessary implementation and made the existing one simply adaptive, like TestBackupRestore. Also made TestBackupRestore clean up dead backup/restore directories on success. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7373 Test Plan: blackbox_crash_test_with_atomic_flush for a while, blackbox_crash_test for a while, with backup and checkpoint 1 in 5k and only 1k max_keys to stress this area Reviewed By: ajkr Differential Revision: D23629057 Pulled By: pdillinger fbshipit-source-id: d7fe7e2be75aaf3cf974be9540a7c5c5de8b371b

view details

Peter Dillinger

commit sha be8445eea8ebf01a5e10ce40195636be41b275c1

Assert valid linked list for write group (#7375) Summary: We've seen some segfaults in db_write_test, with at least one suggesting corruption of a write group linked list. Adding an assertion to have this fail in a more specific way if that is the broken invariant. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7375 Test Plan: make check Reviewed By: jay-zhuang Differential Revision: D23638477 Pulled By: pdillinger fbshipit-source-id: a76fd677cad60a3a516bd363947bfd9ce418edc1

view details

Yanqin Jin

commit sha 205e5776945398c623089e269cfef94d2e455b20

Cancel tombstone skipping during bottommost compaction (#7356) Summary: During bottommost compaction, RocksDB cannot simply drop a tombstone if this tombstone is not in the earliest snapshot. The current behavior is: RocksDB skips other internal keys (of the same user key) in the same snapshot range. In the meantime, RocksDB should check for the `shutting_down` flag. Otherwise, it is possible for a bottommost compaction that has already started running to take a long time to finish, even if the application has tried to cancel all background jobs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7356 Test Plan: make check Reviewed By: ltamasi Differential Revision: D23663241 Pulled By: riversand963 fbshipit-source-id: 25f8e9b51bc3bfa3353cdf87557800f9d90ee0b5

view details

Peter Dillinger

commit sha ecc8ffe17b5d72993a0dfc34ddf694099b30decc

Update master to version 6.13 (#7378) Summary: for release fork Pull Request resolved: https://github.com/facebook/rocksdb/pull/7378 Test Plan: make check + CI Reviewed By: jay-zhuang Differential Revision: D23669163 Pulled By: pdillinger fbshipit-source-id: 14cbf95b32717c28418c71cc8e10f06733bbc49f

view details

Tomasz Posluszny

commit sha 6b72342a128500bfbab307c28ca0ffa86be377e1

Implement missing Java API for ColumnFamilyOptions (#7372) Summary: Covered methods: - OldDefaults() - OptimizeForSmallDb(std::shared_ptr<Cache>) Covered fields: - cf_paths Pull Request resolved: https://github.com/facebook/rocksdb/pull/7372 Reviewed By: pdillinger Differential Revision: D23683449 Pulled By: jay-zhuang fbshipit-source-id: 3e5a8b657cc382c19de3a48c666a3b0e8d96968d

view details

Peter Dillinger

commit sha a0ac71aae1d1530137a126aa3a3175b389defea1

Disable sst_file_manager in stress testing backup restore (#7384) Summary: This is potentially the cause of failures: Failure in Destroy restore dir with: IO error: file rmdir: /dev/shm/rocksdb/rocksdb_crashtest_whitebox/.restore13: Directory not empty Pull Request resolved: https://github.com/facebook/rocksdb/pull/7384 Test Plan: smoke test blackbox_crash_test Reviewed By: jay-zhuang Differential Revision: D23685087 Pulled By: pdillinger fbshipit-source-id: 55f62e9853ce84be1d5ca7d856de867f0f2596ee

view details

Peter Dillinger

commit sha 7e09750790ad2e0fe961017ac4ab5f9e3c383fd5

Update Travis config for broken snapd on ppc (#7381) Summary: snapd update has been failing on ppc for ~a week. Disabling it for now in pull requests. Also, https://github.com/facebook/rocksdb/issues/6653 seems to be fixed, so re-enabling standard unit tests for PPC on pull requests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7381 Test Plan: CI Reviewed By: jay-zhuang Differential Revision: D23684962 Pulled By: pdillinger fbshipit-source-id: 96ec9487b714c4741bb1653dae90b24118830cb5

view details

anand76

commit sha 18a3227b122d528a45bae842a9bbbe3403fff100

Add a new IOStatus subcode to indicate that writes are fenced off (#7374) Summary: In a distributed file system, directory ownership is enforced by fencing off the previous owner once they've been preempted by a new owner. This PR adds a IOStatus subcode for ```StatusCode::IOError``` to indicate this. Once this error is returned for a file write, the DB is put in read-only mode and not allowed to resume in read-write mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7374 Test Plan: Add new unit tests in ```error_handler_fs_test``` Reviewed By: riversand963 Differential Revision: D23687777 Pulled By: anand1976 fbshipit-source-id: bef948642089dc0af399057864d9a8ca339e8b2f

view details

mrambacher

commit sha 7d472accdca996d7d83ae8ce78ad17f799696926

Bring the Configurable options together (#5753) Summary: This PR merges the functionality of making the ColumnFamilyOptions, TableFactory, and DBOptions into Configurable into a single PR, resolving any merge conflicts Pull Request resolved: https://github.com/facebook/rocksdb/pull/5753 Reviewed By: ajkr Differential Revision: D23385030 Pulled By: zhichao-cao fbshipit-source-id: 8b977a7731556230b9b8c5a081b98e49ee4f160a

view details

Andrew Kryczka

commit sha d4993b9b6026570de04212b197373fe7f5c85629

Makefile support subset/individual valgrind tests (#7379) Summary: Introduced `valgrind_check_some`, which is analogous to the `check_some` target for non-valgrind tests. It simplifies the process for running a single valgrind test or subset of valgrind tests when trying to repro a failure. I also added a `ROCKSDBTESTS_ONLY` parameter, which simplifies selecting a single test to run. Previously the user would have to use `ROCKSDBTESTS_START` and `ROCKSDBTESTS_END`, but it was difficult to determine the end variable since it is an exclusive endpoint and must match an actual test name. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7379 Reviewed By: pdillinger Differential Revision: D23673608 Pulled By: ajkr fbshipit-source-id: 87ed81f1a671d46c2dff6a701f85f1891c725b3f

view details

push time in a month

push eventakankshamahajan15/rocksdb

Koby Kahane

commit sha 3e745053b7c3a5453f823b74d8479beba07d50a8

Fix MSVC-related build issues (#7439) Summary: This PR addresses some build and functional issues on MSVC targets, as a step towards an eventual goal of having RocksDB build successfully for Windows on ARM64. Addressed issues include: - BitsSetToOne and CountTrailingZeroBits do not compile on non-x64 MSVC targets. A fallback implementation of BitsSetToOne when Intel intrinsics are not available is added, based on the C++20 `<bit>` popcount implementation in Microsoft's STL. - The implementation of FloorLog2 for MSVC targets (including x64) gives incorrect results. The unit test easily detects this, but CircleCI is currently configured to only run a specific set of tests for Windows CMake builds, so this seems to have been unnoticed. - AsmVolatilePause does not use YieldProcessor on Windows ARM64 targets, even though it is available. - When CondVar::TimedWait calls Microsoft STL's condition_variable::wait_for, it can potentially trigger a bug (just recently fixed in the upcoming VS 16.8's STL) that deadlocks various tests that wait for a timer to execute, since `Timer::Run` doesn't get a chance to execute before being blocked by the test function acquiring the mutex. - In c_test, `GetTempDir` assumes a POSIX-style temp path. - `NormalizePath` did not eliminate consecutive POSIX-style path separators on Windows, resulting in test failures in e.g., wal_manager_test. - Various other test failures. In a followup PR I hope to modify CircleCI's config.yml to invoke all RocksDB unit tests in Windows CMake builds with CTest, instead of the current use of `run_ci_db_test.ps1` which requires individual tests to be specified and is missing many of the existing tests. Notes from peterd: FloorLog2 is not yet used in production code (it's for something in progress). I also added a few more inexpensive platform-dependent tests to Windows CircleCI runs. And included facebook/folly#1461 as requested Pull Request resolved: https://github.com/facebook/rocksdb/pull/7439 Reviewed By: jay-zhuang Differential Revision: D24021563 Pulled By: pdillinger fbshipit-source-id: 0ec2027c0d6a494d8a0fe38d9667fc2f7e29f7e7

view details

sdong

commit sha 750817555867a43f0e7b73dffa44756a9136c808

Introduce options.check_flush_compaction_key_order (#7467) Summary: Introduce an new option options.check_flush_compaction_key_order, by default set to true, which checks key order of flush and compaction, and fail the operation if the order is violated. Also did minor refactor hash checking code, which consolidates the hashing logic to a vlidation class, where the key ordering logic is added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7467 Test Plan: Add unit tests to validate the check can catch reordering in flush and compaction, and can be properly disabled. Reviewed By: riversand963 Differential Revision: D24010683 fbshipit-source-id: 8dd6292d2cda8006054e9ded7cfa4bf405f0527c

view details

Levi Tamasi

commit sha 786c1a2cc475b31449fed9cd17309b09f89cc22c

Reduce the number of iterations in DBTest.FileCreationRandomFailure (#7481) Summary: `DBTest.FileCreationRandomFailure` frequently times out during our continuous test runs. (It's a case of "stress test posing as unit test.") The patch reduces the number of iterations to avoid this. Note that the lower numbers are still sufficient to trigger both flushes and compactions, so test coverage is still the same. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7481 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24034712 Pulled By: ltamasi fbshipit-source-id: 8731a9446e5a121a1041b00f0df473b9f714935a

view details

Peter Dillinger

commit sha 9082771b8641f0d2c82db3b9d87a399767d04921

Add is_full_compaction to CompactionJobStats, cleanup (#7451) Summary: This exposes to the listener interface whether a compaction was full or not. Also cleaned up API comment for CompactionJobInfo::stats, which is not of a nullable type. And since CompactionJob is always created with non-null CompactionJobStats, removed conditionals on it being nullptr and instead assert non-null. TODO later: update C and Java interfaces Pull Request resolved: https://github.com/facebook/rocksdb/pull/7451 Test Plan: updated existing unit tests to check new field, make check Reviewed By: ltamasi Differential Revision: D23977796 Pulled By: pdillinger fbshipit-source-id: 1ae7e26cb949631c2b2fb9e696710daf53cc378d

view details

darionyaphet

commit sha f5e22ce607c0734b32b76ed8fc29dcbdbff75c35

fix dummy collector's name (#7442) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7442 Reviewed By: jay-zhuang Differential Revision: D24011201 Pulled By: ajkr fbshipit-source-id: 54f24c29875f7575612a780f15a42cda918d6641

view details

Zhichao Cao

commit sha 685cabdafa0722395f419cea248f44bbb6d4be93

Add trace_analyzer_test to ASSERT_STATUS_CHECKED list (#7480) Summary: Add trace_analyzer_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7480 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 trace_analyzer_test Reviewed By: riversand963 Differential Revision: D24033768 Pulled By: zhichao-cao fbshipit-source-id: b415045e6fab01d6193448650772368c21c6dba6

view details

sdong

commit sha 94fc676d3f77adb1eada5578032b960ec314c2b0

Fix db_properties_test for ASSERT_STATUS_CHECKED (#7490) Summary: Add all status handling in db_properties_test so that it can pass ASSERT_STATUS_CHECKED. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7490 Test Plan: Run the test with ASSERT_STATUS_CHECKED Reviewed By: jay-zhuang Differential Revision: D24065382 fbshipit-source-id: e008916155196891478c964df0226545308ca71d

view details

Andrew Kryczka

commit sha 1e00909730e552fe9fe41af1404f055423120204

Periodically flush info log out of application buffer (#7488) Summary: This PR schedules a background thread (shared across all DB instances) to flush info log every ten seconds. This improves debuggability in case of RocksDB hanging since it ensures the log messages leading up to the hang will eventually become visible in the log. The bulk of this PR is moving monitoring/stats_dump_scheduler* to db/periodic_work_scheduler* and making the corresponding name changes since now the scheduler handles info log flushing, not just stats dumping. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7488 Reviewed By: riversand963 Differential Revision: D24065165 Pulled By: ajkr fbshipit-source-id: 339c47a0ff43b79fdbd055fbd9fefbb6f9d8d3b5

view details

Andrew Kryczka

commit sha 29ed7661933fb8fe035d6fee0ad31e3f9f43d5f5

add Status check enforcement for stats_history_test (#7496) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7496 Reviewed By: zhichao-cao Differential Revision: D24070007 Pulled By: ajkr fbshipit-source-id: 4320413a4d7707774ee23a7e6232714d7ee7a57f

view details

Yanqin Jin

commit sha 48d5aa9bab7ba84e796719b9ecfdc14ef0c30708

Enable status check for db_secondary_test (#7487) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7487 Test Plan: ASSERT_STATUS_CHECKED=1 make db_secondary_test ./db_secondary_test Reviewed By: zhichao-cao Differential Revision: D24071038 Pulled By: riversand963 fbshipit-source-id: e6600c0aecab71c1326b22af263e92bddee5f7ac

view details

Akanksha Mahajan

commit sha 7cd760dfdfd865052955088e3bf49e806b3da46a

Add status check enforcement for column_family_test.cc (#7484) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7484 Reviewed By: jay-zhuang Differential Revision: D24037616 Pulled By: akankshamahajan15 fbshipit-source-id: 0f63281f81046bcb1b95a7578783285cc6346ece

view details

Zhichao Cao

commit sha b7062f0b2cf05917edb21ff4e2e9fed4fc44284f

Status check enforcement for error_handler_fs_test (#7342) Summary: Added status check enforcement for error_test_fs_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7342 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 error_test_fs_test Reviewed By: akankshamahajan15 Differential Revision: D23972231 Pulled By: zhichao-cao fbshipit-source-id: fa41bfe440012e0c55f2c9507c1d0104e5e93f84

view details

sdong

commit sha 668ee08915fb753ecd5c0dd78fa998bd4898e80b

Fix prefix_test for status check (#7495) Summary: Fix prefix_test so that it passes when ASSERT_STATUS_CHECKED=1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7495 Test Plan: Run the test with the option Reviewed By: anand1976 Differential Revision: D24069715 fbshipit-source-id: 54f74b58575a1b49dbdee9ea2d24751fa956b620

view details

Akanksha Mahajan

commit sha 7d503e66a9a9d9bdf17e2d35038bfe8f3e12f5dc

Return error if Get() fails in Prefetching Filter blocks (#7463) Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7463 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D24008226 Pulled By: akankshamahajan15 fbshipit-source-id: b65d63b2d01465db92500b78de7ad58650ec9b3b

view details

Yanqin Jin

commit sha 758ead5df771d8ac2972c9f2c4d6074c97272151

Enforce status check for corruption_test (#7453) Summary: Enforce status check for corruption_test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7453 Test Plan: ``` ASSERT_STATUS_CHECKED=1 make corruption_test ./corruption_test ``` Reviewed By: jay-zhuang Differential Revision: D24006862 Pulled By: riversand963 fbshipit-source-id: 664677caf4c3007a25cf565cec3d677f2dcea130

view details

Levi Tamasi

commit sha 5d16325ce33384de974560648434473031639018

Revert "Return error if Get() fails in Prefetching Filter blocks (#7463)" (#7505) Summary: This reverts commit 7d503e66a9a9d9bdf17e2d35038bfe8f3e12f5dc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7505 Reviewed By: ajkr Differential Revision: D24100875 Pulled By: ltamasi fbshipit-source-id: 8705e3e6e8be4b4fd175ffdb031baa6530b61151

view details

push time in a month

push eventakankshamahajan15/rocksdb

Koby Kahane

commit sha 3e745053b7c3a5453f823b74d8479beba07d50a8

Fix MSVC-related build issues (#7439) Summary: This PR addresses some build and functional issues on MSVC targets, as a step towards an eventual goal of having RocksDB build successfully for Windows on ARM64. Addressed issues include: - BitsSetToOne and CountTrailingZeroBits do not compile on non-x64 MSVC targets. A fallback implementation of BitsSetToOne when Intel intrinsics are not available is added, based on the C++20 `<bit>` popcount implementation in Microsoft's STL. - The implementation of FloorLog2 for MSVC targets (including x64) gives incorrect results. The unit test easily detects this, but CircleCI is currently configured to only run a specific set of tests for Windows CMake builds, so this seems to have been unnoticed. - AsmVolatilePause does not use YieldProcessor on Windows ARM64 targets, even though it is available. - When CondVar::TimedWait calls Microsoft STL's condition_variable::wait_for, it can potentially trigger a bug (just recently fixed in the upcoming VS 16.8's STL) that deadlocks various tests that wait for a timer to execute, since `Timer::Run` doesn't get a chance to execute before being blocked by the test function acquiring the mutex. - In c_test, `GetTempDir` assumes a POSIX-style temp path. - `NormalizePath` did not eliminate consecutive POSIX-style path separators on Windows, resulting in test failures in e.g., wal_manager_test. - Various other test failures. In a followup PR I hope to modify CircleCI's config.yml to invoke all RocksDB unit tests in Windows CMake builds with CTest, instead of the current use of `run_ci_db_test.ps1` which requires individual tests to be specified and is missing many of the existing tests. Notes from peterd: FloorLog2 is not yet used in production code (it's for something in progress). I also added a few more inexpensive platform-dependent tests to Windows CircleCI runs. And included facebook/folly#1461 as requested Pull Request resolved: https://github.com/facebook/rocksdb/pull/7439 Reviewed By: jay-zhuang Differential Revision: D24021563 Pulled By: pdillinger fbshipit-source-id: 0ec2027c0d6a494d8a0fe38d9667fc2f7e29f7e7

view details

sdong

commit sha 750817555867a43f0e7b73dffa44756a9136c808

Introduce options.check_flush_compaction_key_order (#7467) Summary: Introduce an new option options.check_flush_compaction_key_order, by default set to true, which checks key order of flush and compaction, and fail the operation if the order is violated. Also did minor refactor hash checking code, which consolidates the hashing logic to a vlidation class, where the key ordering logic is added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7467 Test Plan: Add unit tests to validate the check can catch reordering in flush and compaction, and can be properly disabled. Reviewed By: riversand963 Differential Revision: D24010683 fbshipit-source-id: 8dd6292d2cda8006054e9ded7cfa4bf405f0527c

view details

Levi Tamasi

commit sha 786c1a2cc475b31449fed9cd17309b09f89cc22c

Reduce the number of iterations in DBTest.FileCreationRandomFailure (#7481) Summary: `DBTest.FileCreationRandomFailure` frequently times out during our continuous test runs. (It's a case of "stress test posing as unit test.") The patch reduces the number of iterations to avoid this. Note that the lower numbers are still sufficient to trigger both flushes and compactions, so test coverage is still the same. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7481 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24034712 Pulled By: ltamasi fbshipit-source-id: 8731a9446e5a121a1041b00f0df473b9f714935a

view details

Peter Dillinger

commit sha 9082771b8641f0d2c82db3b9d87a399767d04921

Add is_full_compaction to CompactionJobStats, cleanup (#7451) Summary: This exposes to the listener interface whether a compaction was full or not. Also cleaned up API comment for CompactionJobInfo::stats, which is not of a nullable type. And since CompactionJob is always created with non-null CompactionJobStats, removed conditionals on it being nullptr and instead assert non-null. TODO later: update C and Java interfaces Pull Request resolved: https://github.com/facebook/rocksdb/pull/7451 Test Plan: updated existing unit tests to check new field, make check Reviewed By: ltamasi Differential Revision: D23977796 Pulled By: pdillinger fbshipit-source-id: 1ae7e26cb949631c2b2fb9e696710daf53cc378d

view details

darionyaphet

commit sha f5e22ce607c0734b32b76ed8fc29dcbdbff75c35

fix dummy collector's name (#7442) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7442 Reviewed By: jay-zhuang Differential Revision: D24011201 Pulled By: ajkr fbshipit-source-id: 54f24c29875f7575612a780f15a42cda918d6641

view details

Zhichao Cao

commit sha 685cabdafa0722395f419cea248f44bbb6d4be93

Add trace_analyzer_test to ASSERT_STATUS_CHECKED list (#7480) Summary: Add trace_analyzer_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7480 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 trace_analyzer_test Reviewed By: riversand963 Differential Revision: D24033768 Pulled By: zhichao-cao fbshipit-source-id: b415045e6fab01d6193448650772368c21c6dba6

view details

sdong

commit sha 94fc676d3f77adb1eada5578032b960ec314c2b0

Fix db_properties_test for ASSERT_STATUS_CHECKED (#7490) Summary: Add all status handling in db_properties_test so that it can pass ASSERT_STATUS_CHECKED. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7490 Test Plan: Run the test with ASSERT_STATUS_CHECKED Reviewed By: jay-zhuang Differential Revision: D24065382 fbshipit-source-id: e008916155196891478c964df0226545308ca71d

view details

Andrew Kryczka

commit sha 1e00909730e552fe9fe41af1404f055423120204

Periodically flush info log out of application buffer (#7488) Summary: This PR schedules a background thread (shared across all DB instances) to flush info log every ten seconds. This improves debuggability in case of RocksDB hanging since it ensures the log messages leading up to the hang will eventually become visible in the log. The bulk of this PR is moving monitoring/stats_dump_scheduler* to db/periodic_work_scheduler* and making the corresponding name changes since now the scheduler handles info log flushing, not just stats dumping. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7488 Reviewed By: riversand963 Differential Revision: D24065165 Pulled By: ajkr fbshipit-source-id: 339c47a0ff43b79fdbd055fbd9fefbb6f9d8d3b5

view details

Andrew Kryczka

commit sha 29ed7661933fb8fe035d6fee0ad31e3f9f43d5f5

add Status check enforcement for stats_history_test (#7496) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7496 Reviewed By: zhichao-cao Differential Revision: D24070007 Pulled By: ajkr fbshipit-source-id: 4320413a4d7707774ee23a7e6232714d7ee7a57f

view details

Yanqin Jin

commit sha 48d5aa9bab7ba84e796719b9ecfdc14ef0c30708

Enable status check for db_secondary_test (#7487) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7487 Test Plan: ASSERT_STATUS_CHECKED=1 make db_secondary_test ./db_secondary_test Reviewed By: zhichao-cao Differential Revision: D24071038 Pulled By: riversand963 fbshipit-source-id: e6600c0aecab71c1326b22af263e92bddee5f7ac

view details

Akanksha Mahajan

commit sha 7cd760dfdfd865052955088e3bf49e806b3da46a

Add status check enforcement for column_family_test.cc (#7484) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7484 Reviewed By: jay-zhuang Differential Revision: D24037616 Pulled By: akankshamahajan15 fbshipit-source-id: 0f63281f81046bcb1b95a7578783285cc6346ece

view details

Zhichao Cao

commit sha b7062f0b2cf05917edb21ff4e2e9fed4fc44284f

Status check enforcement for error_handler_fs_test (#7342) Summary: Added status check enforcement for error_test_fs_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7342 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 error_test_fs_test Reviewed By: akankshamahajan15 Differential Revision: D23972231 Pulled By: zhichao-cao fbshipit-source-id: fa41bfe440012e0c55f2c9507c1d0104e5e93f84

view details

sdong

commit sha 668ee08915fb753ecd5c0dd78fa998bd4898e80b

Fix prefix_test for status check (#7495) Summary: Fix prefix_test so that it passes when ASSERT_STATUS_CHECKED=1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7495 Test Plan: Run the test with the option Reviewed By: anand1976 Differential Revision: D24069715 fbshipit-source-id: 54f74b58575a1b49dbdee9ea2d24751fa956b620

view details

akankshamahajan

commit sha 92255c2c9700d90362273fb38536e7946bc8721a

Add unit test cases in ASSERT_STATUS_CHECKED Summary: Add status enforcement for following tests: 1. import_column_family_test 2. memory_test 3. table_test Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

akankshamahajan

commit sha d558ed2fe5742a3d97557f54f5857d03af64f36c

Addressed comments Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

Pull request review commentfacebook/rocksdb

Add few unit test cases in ASSERT_STATUS_CHECKED

 Status DBImpl::CreateColumnFamilyWithImport(    import_job.Cleanup(status);   if (!status.ok()) {-    DropColumnFamily(*handle);-    DestroyColumnFamilyHandle(*handle);+    // TODO: Should handle this error or log this?+    DropColumnFamily(*handle).PermitUncheckedError();

If I add status = DropColumnFamily(*handle), then it will overwrite the previous errored status and new error will be returned(instead of returning previous error). Shouldn't we log the error instead as we want to drop and delete the column handle after status is not ok?

akankshamahajan15

comment created time in a month

PullRequestReviewEvent

push eventakankshamahajan15/rocksdb

Yanqin Jin

commit sha 07dc955a1f774e9f5699cde7cbd24f342297dc3c

Report error of GetChildren (#7459) Summary: As title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7459 Test Plan: make check Reviewed By: anand1976 Differential Revision: D23999393 Pulled By: riversand963 fbshipit-source-id: 09df8e1637f4df3616c63ee314de397b35be4e4a

view details

Andrew Kryczka

commit sha 1600aac46f64f906631f64d036b2ea5359c7ef5f

Flush info log for warning and higher severity (#7462) Summary: After unclean crash, the tail of the log could look as follows due to block buffering, even when the call to `ROCKSDB_LOG_ERROR()` finished. ``` 2020/09/29-13:54:39.596710 7f67025fe700 [ERROR] [/db_impl/db_impl_compaction_flush.cc:2500] Waiting after background compaction err ``` This PR forces the flush while logging warning severity or higher to prevent that case. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7462 Reviewed By: riversand963 Differential Revision: D24000154 Pulled By: ajkr fbshipit-source-id: 3bf5f1e69a62ee10e84095cebc88937a8f81b4ad

view details

anand76

commit sha bd5d9e2a1d79fa8f49965499a1d8228fd7ae191e

Fix misspelling of PartitionedIndexIterator (#7450) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7450 Reviewed By: zhichao-cao Differential Revision: D23992811 Pulled By: anand1976 fbshipit-source-id: 71bd898aafce6a3add3c8cd86d9f0e0fb63860cf

view details

Andrew Kryczka

commit sha 8115eb520de5329977f0430d6f4f6e3309fd233f

add Status check assertions for repair_test (#7455) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7455 Reviewed By: riversand963 Differential Revision: D23985283 Pulled By: ajkr fbshipit-source-id: 5dd2be62350f6e31d13a1e7821cb848a37699c93

view details

Jay Zhuang

commit sha 1bdaef7a06ee2fc147f1188cec6781905849c71b

Status check enforcement for timestamp_basic_test (#7454) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7454 Reviewed By: riversand963 Differential Revision: D23981719 Pulled By: jay-zhuang fbshipit-source-id: 01073f73e54c17067b886c4a2f179b2804198399

view details

Akanksha Mahajan

commit sha 9d212d3f0ecee2e6f72286497dc2b7a3e2d3ba58

Provide users with option to opt-in to get corrupt data in logs/messages (#7420) Summary: Add a new Option "allow_data_in_errors". When it's set by users, it allows them to opt-in to get error messages containing corrupted keys/values. Corrupt keys, values will be logged in the messages, logs, status etc. that will help users with the useful information regarding affected data. By default value is set false to prevent users data to be exposed in the messages. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7420 Test Plan: 1. make check -j64 2. Add a new test case Reviewed By: ajkr Differential Revision: D23835028 Pulled By: akankshamahajan15 fbshipit-source-id: 8d2eba8fb898e79fcf1fccc07295065a75eb59b1

view details

sdong

commit sha 5f33436285c532936de8870bf22badf1568366dd

Revert an uncessary status code check skipping (#7458) Summary: https://github.com/facebook/rocksdb/pull/7452 added an uncessary skip for status code checking. Revert it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7458 Test Plan: Watch CI to finish Reviewed By: jay-zhuang Differential Revision: D23994390 fbshipit-source-id: a2b50a6326d8073db3386bff3d32acc5a6666e9b

view details

Peter Dillinger

commit sha ddbc5dad0505ba88c60975f146704f04c359d4c3

Enable force_consistency_checks by default (#7446) Summary: This has been running in production on some key workloads, so we believe it to be safe and extremely low cost. Nevertheless, I've added code to ensure that "force_consistency_checks" is mentioned in any corruption reports so that people know how to disable in case of false positive corruption reports. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7446 Test Plan: make check, CI, temporary debug print new message with ./version_builder_test Reviewed By: ajkr Differential Revision: D23972101 Pulled By: pdillinger fbshipit-source-id: 9623e400f3752577c0ecf977e6d0915562cf9968

view details

Akanksha Mahajan

commit sha 5997f6b17bfe571a5882f2f490d81cf5c6b44732

Add Status check enforcement for unit tests (#7464) Summary: Add status check for unit tests : block_based_filter_block_test, block_fetcher_test, full_filter_block_test and partitioned_filter_block_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7464 Reviewed By: zhichao-cao Differential Revision: D24011309 Pulled By: akankshamahajan15 fbshipit-source-id: d814803f94e8bb8b811ef170d20b22d52c1a3ff2

view details

sdong

commit sha aedcaaef99a3ed55de799219d9e30fcea53189cc

Stress test to support paranoid_file_checks (#7473) Summary: It's important to make sure no false positive is reported when options.paranoid_file_checks is used. Add it to stress test and a place holder in crash test. It is disabled in crash test as there appears to be a bug causing false positive. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7473 Test Plan: Run crash test Reviewed By: ajkr Differential Revision: D24026939 fbshipit-source-id: 89102acb45cf041776775ce44a4eef4b0f3a380c

view details

Jay Zhuang

commit sha 58905f31c18aa79b6f278601dd824c3d50ee4f54

Remove a gtest-parallel workaround (#7433) Summary: As the issue is fixed in official repo: https://github.com/google/gtest-parallel/pull/79 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7433 Reviewed By: ajkr Differential Revision: D24023271 Pulled By: jay-zhuang fbshipit-source-id: 0babf5e4c59cd61ded5a64cf9aa2d457deeeaa47

view details

Andrew Kryczka

commit sha 718e192965cabb311feb7718d9cb8a2e34958072

Fix flaky intra-L0 consistency failure regression tests (#7477) Summary: Do not assert the number of files after intra-L0 compaction is eligible to run since it could complete (and reduce the number of files) before the assertion executes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7477 Reviewed By: pdillinger Differential Revision: D24032049 Pulled By: ajkr fbshipit-source-id: e838ac7a24651ebd643b9e5a9d39d2e789c46929

view details

Jay Zhuang

commit sha e127fe18c3ee0d99b4507f1e2b7cfcb38a653e74

Fix TSAN failure for backupable_db_test (#7478) Summary: It's a transient failure, but can be reproduce with running the test 100 times: https://app.circleci.com/pipelines/github/facebook/rocksdb/3760/workflows/de909685-f22b-45ba-a8f3-6ebb78a54e96/jobs/37039 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7478 Test Plan: re-run the test 100 times Reviewed By: ajkr Differential Revision: D24035758 Pulled By: jay-zhuang fbshipit-source-id: 6b31983d5c3f7faa8d5481306098513485d0d69d

view details

Ramkumar Vadivelu

commit sha e04a50923dd404f9990a1cfd20f926e1111fd067

Change ParseInternalKey() to return Status instead of bool (#7457) Summary: Fixes https://github.com/facebook/rocksdb/issues/7430 Change ParseInternalKey() to return Status instead of bool. db_bench (seekrandom) based before/after results with value size of 100 bytes and 16 bytes can be found at (tests ran on an udb server): https://www.dropbox.com/s/47bwamdy5ozngph/PIK_ret_Status_results.xlsx?dl=0 ![db_bench_results](https://user-images.githubusercontent.com/62277872/94642825-2a21a800-029a-11eb-88f2-124136c83fd3.png) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7457 Reviewed By: ajkr Differential Revision: D24002433 Pulled By: ramvadiv fbshipit-source-id: ac253ecf577a29044c47c3fe254a01e71404c44c

view details

darionyaphet

commit sha 2af46f10112f91298a66d513269705fe609a2461

Move break into block (#7468) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7468 Reviewed By: riversand963 Differential Revision: D24028736 Pulled By: ajkr fbshipit-source-id: bd2b4b8d069491a16373d3d2705fddf7ebfe6723

view details

Yanqin Jin

commit sha 2f2e6e1e2c695bf8fac53b3bd162626de71df5b4

Add a rocksdb lib target with link_whole=True (#7466) Summary: We would like to build a shared library with all fbcode dependencies statically linked within. This resulting .so should not drop any symbols definitions in the building process. To ensure that, we use `link_whole=True` according to https://buck.build/rule/cxx_library.html#link_whole. Since `link_whole` is `False` by default, adding a `link_whole=False` to existing libraries won't change any behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7466 Test Plan: build a .so and test internally. Reviewed By: pdillinger Differential Revision: D24009780 Pulled By: riversand963 fbshipit-source-id: d18804d495da7195ed72a2040e1a5de4fd336519

view details

Koby Kahane

commit sha 3e745053b7c3a5453f823b74d8479beba07d50a8

Fix MSVC-related build issues (#7439) Summary: This PR addresses some build and functional issues on MSVC targets, as a step towards an eventual goal of having RocksDB build successfully for Windows on ARM64. Addressed issues include: - BitsSetToOne and CountTrailingZeroBits do not compile on non-x64 MSVC targets. A fallback implementation of BitsSetToOne when Intel intrinsics are not available is added, based on the C++20 `<bit>` popcount implementation in Microsoft's STL. - The implementation of FloorLog2 for MSVC targets (including x64) gives incorrect results. The unit test easily detects this, but CircleCI is currently configured to only run a specific set of tests for Windows CMake builds, so this seems to have been unnoticed. - AsmVolatilePause does not use YieldProcessor on Windows ARM64 targets, even though it is available. - When CondVar::TimedWait calls Microsoft STL's condition_variable::wait_for, it can potentially trigger a bug (just recently fixed in the upcoming VS 16.8's STL) that deadlocks various tests that wait for a timer to execute, since `Timer::Run` doesn't get a chance to execute before being blocked by the test function acquiring the mutex. - In c_test, `GetTempDir` assumes a POSIX-style temp path. - `NormalizePath` did not eliminate consecutive POSIX-style path separators on Windows, resulting in test failures in e.g., wal_manager_test. - Various other test failures. In a followup PR I hope to modify CircleCI's config.yml to invoke all RocksDB unit tests in Windows CMake builds with CTest, instead of the current use of `run_ci_db_test.ps1` which requires individual tests to be specified and is missing many of the existing tests. Notes from peterd: FloorLog2 is not yet used in production code (it's for something in progress). I also added a few more inexpensive platform-dependent tests to Windows CircleCI runs. And included facebook/folly#1461 as requested Pull Request resolved: https://github.com/facebook/rocksdb/pull/7439 Reviewed By: jay-zhuang Differential Revision: D24021563 Pulled By: pdillinger fbshipit-source-id: 0ec2027c0d6a494d8a0fe38d9667fc2f7e29f7e7

view details

sdong

commit sha 750817555867a43f0e7b73dffa44756a9136c808

Introduce options.check_flush_compaction_key_order (#7467) Summary: Introduce an new option options.check_flush_compaction_key_order, by default set to true, which checks key order of flush and compaction, and fail the operation if the order is violated. Also did minor refactor hash checking code, which consolidates the hashing logic to a vlidation class, where the key ordering logic is added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7467 Test Plan: Add unit tests to validate the check can catch reordering in flush and compaction, and can be properly disabled. Reviewed By: riversand963 Differential Revision: D24010683 fbshipit-source-id: 8dd6292d2cda8006054e9ded7cfa4bf405f0527c

view details

Levi Tamasi

commit sha 786c1a2cc475b31449fed9cd17309b09f89cc22c

Reduce the number of iterations in DBTest.FileCreationRandomFailure (#7481) Summary: `DBTest.FileCreationRandomFailure` frequently times out during our continuous test runs. (It's a case of "stress test posing as unit test.") The patch reduces the number of iterations to avoid this. Note that the lower numbers are still sufficient to trigger both flushes and compactions, so test coverage is still the same. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7481 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24034712 Pulled By: ltamasi fbshipit-source-id: 8731a9446e5a121a1041b00f0df473b9f714935a

view details

Peter Dillinger

commit sha 9082771b8641f0d2c82db3b9d87a399767d04921

Add is_full_compaction to CompactionJobStats, cleanup (#7451) Summary: This exposes to the listener interface whether a compaction was full or not. Also cleaned up API comment for CompactionJobInfo::stats, which is not of a nullable type. And since CompactionJob is always created with non-null CompactionJobStats, removed conditionals on it being nullptr and instead assert non-null. TODO later: update C and Java interfaces Pull Request resolved: https://github.com/facebook/rocksdb/pull/7451 Test Plan: updated existing unit tests to check new field, make check Reviewed By: ltamasi Differential Revision: D23977796 Pulled By: pdillinger fbshipit-source-id: 1ae7e26cb949631c2b2fb9e696710daf53cc378d

view details

push time in a month

Pull request review commentfacebook/rocksdb

Return error if Get() fails in Prefetching Filter blocks

 void PartitionedFilterBlockReader::CacheDependencies(const ReadOptions& ro,         prefetch_buffer.get(), ro, handle, UncompressionDict::GetEmptyDict(),         &block, BlockType::kFilter, nullptr /* get_context */, &lookup_context,         nullptr /* contents */);--    assert(s.ok() || block.GetValue() == nullptr);

yes it is. I recently got to know that assertions are excluded in optimized builds.

akankshamahajan15

comment created time in a month

PullRequestReviewEvent

PR opened facebook/rocksdb

Add few unit test cases in ASSERT_STATUS_CHECKED

Summary: Add status enforcement for following tests:

  1. import_column_family_test
  2. memory_test
  3. table_test

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

+107 -72

0 comment

8 changed files

pr created time in a month

create barnchakankshamahajan15/rocksdb

branch : status_enforcement

created branch time in a month

pull request commentfacebook/rocksdb

Add ldb_cmd_test to ASSERT_STATUS_CHECKED list

LGTM!

zhichao-cao

comment created time in a month

Pull request review commentfacebook/rocksdb

Add ldb_cmd_test to ASSERT_STATUS_CHECKED list

 Status VersionSet::ReduceNumberOfLevels(const std::string& dbname, // metadata from Manifest to VersionSet before calling this function. Status VersionSet::GetLiveFilesChecksumInfo(FileChecksumList* checksum_list) {   // Clean the previously stored checksum information if any.+  Status s = Status::OK();

Status s is initialized to Status::OK by default.

zhichao-cao

comment created time in a month

push eventakankshamahajan15/rocksdb

darionyaphet

commit sha f5e22ce607c0734b32b76ed8fc29dcbdbff75c35

fix dummy collector's name (#7442) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7442 Reviewed By: jay-zhuang Differential Revision: D24011201 Pulled By: ajkr fbshipit-source-id: 54f24c29875f7575612a780f15a42cda918d6641

view details

Zhichao Cao

commit sha 685cabdafa0722395f419cea248f44bbb6d4be93

Add trace_analyzer_test to ASSERT_STATUS_CHECKED list (#7480) Summary: Add trace_analyzer_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7480 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 trace_analyzer_test Reviewed By: riversand963 Differential Revision: D24033768 Pulled By: zhichao-cao fbshipit-source-id: b415045e6fab01d6193448650772368c21c6dba6

view details

sdong

commit sha 94fc676d3f77adb1eada5578032b960ec314c2b0

Fix db_properties_test for ASSERT_STATUS_CHECKED (#7490) Summary: Add all status handling in db_properties_test so that it can pass ASSERT_STATUS_CHECKED. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7490 Test Plan: Run the test with ASSERT_STATUS_CHECKED Reviewed By: jay-zhuang Differential Revision: D24065382 fbshipit-source-id: e008916155196891478c964df0226545308ca71d

view details

Andrew Kryczka

commit sha 1e00909730e552fe9fe41af1404f055423120204

Periodically flush info log out of application buffer (#7488) Summary: This PR schedules a background thread (shared across all DB instances) to flush info log every ten seconds. This improves debuggability in case of RocksDB hanging since it ensures the log messages leading up to the hang will eventually become visible in the log. The bulk of this PR is moving monitoring/stats_dump_scheduler* to db/periodic_work_scheduler* and making the corresponding name changes since now the scheduler handles info log flushing, not just stats dumping. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7488 Reviewed By: riversand963 Differential Revision: D24065165 Pulled By: ajkr fbshipit-source-id: 339c47a0ff43b79fdbd055fbd9fefbb6f9d8d3b5

view details

Andrew Kryczka

commit sha 29ed7661933fb8fe035d6fee0ad31e3f9f43d5f5

add Status check enforcement for stats_history_test (#7496) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7496 Reviewed By: zhichao-cao Differential Revision: D24070007 Pulled By: ajkr fbshipit-source-id: 4320413a4d7707774ee23a7e6232714d7ee7a57f

view details

akankshamahajan

commit sha 54d3cdfa973ce43a161a93fe7507ce9c64d4f350

Add status check enforcement for column_family_test.cc Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

akankshamahajan

commit sha f065b8043335f9207f7b974a192014eeaf743ad6

Addressed comments Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

push eventakankshamahajan15/rocksdb

Ramkumar Vadivelu

commit sha e04a50923dd404f9990a1cfd20f926e1111fd067

Change ParseInternalKey() to return Status instead of bool (#7457) Summary: Fixes https://github.com/facebook/rocksdb/issues/7430 Change ParseInternalKey() to return Status instead of bool. db_bench (seekrandom) based before/after results with value size of 100 bytes and 16 bytes can be found at (tests ran on an udb server): https://www.dropbox.com/s/47bwamdy5ozngph/PIK_ret_Status_results.xlsx?dl=0 ![db_bench_results](https://user-images.githubusercontent.com/62277872/94642825-2a21a800-029a-11eb-88f2-124136c83fd3.png) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7457 Reviewed By: ajkr Differential Revision: D24002433 Pulled By: ramvadiv fbshipit-source-id: ac253ecf577a29044c47c3fe254a01e71404c44c

view details

darionyaphet

commit sha 2af46f10112f91298a66d513269705fe609a2461

Move break into block (#7468) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7468 Reviewed By: riversand963 Differential Revision: D24028736 Pulled By: ajkr fbshipit-source-id: bd2b4b8d069491a16373d3d2705fddf7ebfe6723

view details

Yanqin Jin

commit sha 2f2e6e1e2c695bf8fac53b3bd162626de71df5b4

Add a rocksdb lib target with link_whole=True (#7466) Summary: We would like to build a shared library with all fbcode dependencies statically linked within. This resulting .so should not drop any symbols definitions in the building process. To ensure that, we use `link_whole=True` according to https://buck.build/rule/cxx_library.html#link_whole. Since `link_whole` is `False` by default, adding a `link_whole=False` to existing libraries won't change any behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7466 Test Plan: build a .so and test internally. Reviewed By: pdillinger Differential Revision: D24009780 Pulled By: riversand963 fbshipit-source-id: d18804d495da7195ed72a2040e1a5de4fd336519

view details

Koby Kahane

commit sha 3e745053b7c3a5453f823b74d8479beba07d50a8

Fix MSVC-related build issues (#7439) Summary: This PR addresses some build and functional issues on MSVC targets, as a step towards an eventual goal of having RocksDB build successfully for Windows on ARM64. Addressed issues include: - BitsSetToOne and CountTrailingZeroBits do not compile on non-x64 MSVC targets. A fallback implementation of BitsSetToOne when Intel intrinsics are not available is added, based on the C++20 `<bit>` popcount implementation in Microsoft's STL. - The implementation of FloorLog2 for MSVC targets (including x64) gives incorrect results. The unit test easily detects this, but CircleCI is currently configured to only run a specific set of tests for Windows CMake builds, so this seems to have been unnoticed. - AsmVolatilePause does not use YieldProcessor on Windows ARM64 targets, even though it is available. - When CondVar::TimedWait calls Microsoft STL's condition_variable::wait_for, it can potentially trigger a bug (just recently fixed in the upcoming VS 16.8's STL) that deadlocks various tests that wait for a timer to execute, since `Timer::Run` doesn't get a chance to execute before being blocked by the test function acquiring the mutex. - In c_test, `GetTempDir` assumes a POSIX-style temp path. - `NormalizePath` did not eliminate consecutive POSIX-style path separators on Windows, resulting in test failures in e.g., wal_manager_test. - Various other test failures. In a followup PR I hope to modify CircleCI's config.yml to invoke all RocksDB unit tests in Windows CMake builds with CTest, instead of the current use of `run_ci_db_test.ps1` which requires individual tests to be specified and is missing many of the existing tests. Notes from peterd: FloorLog2 is not yet used in production code (it's for something in progress). I also added a few more inexpensive platform-dependent tests to Windows CircleCI runs. And included facebook/folly#1461 as requested Pull Request resolved: https://github.com/facebook/rocksdb/pull/7439 Reviewed By: jay-zhuang Differential Revision: D24021563 Pulled By: pdillinger fbshipit-source-id: 0ec2027c0d6a494d8a0fe38d9667fc2f7e29f7e7

view details

sdong

commit sha 750817555867a43f0e7b73dffa44756a9136c808

Introduce options.check_flush_compaction_key_order (#7467) Summary: Introduce an new option options.check_flush_compaction_key_order, by default set to true, which checks key order of flush and compaction, and fail the operation if the order is violated. Also did minor refactor hash checking code, which consolidates the hashing logic to a vlidation class, where the key ordering logic is added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7467 Test Plan: Add unit tests to validate the check can catch reordering in flush and compaction, and can be properly disabled. Reviewed By: riversand963 Differential Revision: D24010683 fbshipit-source-id: 8dd6292d2cda8006054e9ded7cfa4bf405f0527c

view details

Levi Tamasi

commit sha 786c1a2cc475b31449fed9cd17309b09f89cc22c

Reduce the number of iterations in DBTest.FileCreationRandomFailure (#7481) Summary: `DBTest.FileCreationRandomFailure` frequently times out during our continuous test runs. (It's a case of "stress test posing as unit test.") The patch reduces the number of iterations to avoid this. Note that the lower numbers are still sufficient to trigger both flushes and compactions, so test coverage is still the same. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7481 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24034712 Pulled By: ltamasi fbshipit-source-id: 8731a9446e5a121a1041b00f0df473b9f714935a

view details

Peter Dillinger

commit sha 9082771b8641f0d2c82db3b9d87a399767d04921

Add is_full_compaction to CompactionJobStats, cleanup (#7451) Summary: This exposes to the listener interface whether a compaction was full or not. Also cleaned up API comment for CompactionJobInfo::stats, which is not of a nullable type. And since CompactionJob is always created with non-null CompactionJobStats, removed conditionals on it being nullptr and instead assert non-null. TODO later: update C and Java interfaces Pull Request resolved: https://github.com/facebook/rocksdb/pull/7451 Test Plan: updated existing unit tests to check new field, make check Reviewed By: ltamasi Differential Revision: D23977796 Pulled By: pdillinger fbshipit-source-id: 1ae7e26cb949631c2b2fb9e696710daf53cc378d

view details

akankshamahajan

commit sha e692387a6c1239791aa8fc52c7f5783deef47138

Add status check enforcement for column_family_test.cc Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

akankshamahajan

commit sha ba4a86a090e1588d210d13e0f4b4829057289678

Addressed comments Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 7f748eddbc2a3c7f732ba35e0344af1541e3a8a3

Addressed comments Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

Pull request review commentfacebook/rocksdb

Add status check enforcement for column_family_test.cc

 class ColumnFamilyTestBase : public testing::Test {       ColumnFamilyDescriptor desc;       ASSERT_OK(handles_[cfi]->GetDescriptor(&desc));       RocksDBOptionsParser::VerifyCFOptions(ConfigOptions(), desc.options,-                                            current_cf_opt);+                                            current_cf_opt)+          .PermitUncheckedError();

Yes that works. Thanks a lot!

akankshamahajan15

comment created time in a month

PullRequestReviewEvent

Pull request review commentfacebook/rocksdb

Add status check enforcement for column_family_test.cc

 class ColumnFamilyTestBase : public testing::Test {       ColumnFamilyDescriptor desc;       ASSERT_OK(handles_[cfi]->GetDescriptor(&desc));       RocksDBOptionsParser::VerifyCFOptions(ConfigOptions(), desc.options,-                                            current_cf_opt);+                                            current_cf_opt)+          .PermitUncheckedError();

I am not sure if this is expected or not.

akankshamahajan15

comment created time in a month

PullRequestReviewEvent

Pull request review commentfacebook/rocksdb

Add status check enforcement for column_family_test.cc

 class ColumnFamilyTestBase : public testing::Test {       ColumnFamilyDescriptor desc;       ASSERT_OK(handles_[cfi]->GetDescriptor(&desc));       RocksDBOptionsParser::VerifyCFOptions(ConfigOptions(), desc.options,-                                            current_cf_opt);+                                            current_cf_opt)+          .PermitUncheckedError();

@mrambacher, Oh my bad. This is different. It gives an error that "RocksDBOptionsParser::VerifyCFOptions( ConfigOptions(), desc.options, current_cf_opt) Invalid argument: [RocksDBOptionsParser]: failed the verification on ColumnFamilyOptions::periodic_compaction_seconds--- The specified one is 0 while the persisted one is 18446744073709551614." Line: https://github.com/facebook/rocksdb/blob/7d472accdca996d7d83ae8ce78ad17f799696926/options/options_parser.cc#L679

akankshamahajan15

comment created time in a month

PullRequestReviewEvent

pull request commentfacebook/rocksdb

Add status check enforcement for column_family_test.cc

LGTM.

There are several other PRs that are also changing db_impl_write at the same time. I have suggested a new signature to the "BGError" so that the status does not need to be ignored everywhere.

Sounds good. I will add the comment as TODO and when its confirmed to add a new signature to return void, I will change the signature here as well.

akankshamahajan15

comment created time in a month

Pull request review commentfacebook/rocksdb

Add status check enforcement for column_family_test.cc

 class ColumnFamilyTestBase : public testing::Test {       ColumnFamilyDescriptor desc;       ASSERT_OK(handles_[cfi]->GetDescriptor(&desc));       RocksDBOptionsParser::VerifyCFOptions(ConfigOptions(), desc.options,-                                            current_cf_opt);+                                            current_cf_opt)+          .PermitUncheckedError();

In the test itself it expects the arguments to be invalid so it returns invalid in few cases and OK in other.

akankshamahajan15

comment created time in a month

PullRequestReviewEvent

push eventakankshamahajan15/rocksdb

Peter Dillinger

commit sha 08552b19d3276c75147f84b9a8e217e2dec7e897

Genericize and clean up FastRange (#7436) Summary: A generic algorithm in progress depends on a templatized version of fastrange, so this change generalizes it and renames it to fit our style guidelines, FastRange32, FastRange64, and now FastRangeGeneric. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7436 Test Plan: added a few more test cases Reviewed By: jay-zhuang Differential Revision: D23958153 Pulled By: pdillinger fbshipit-source-id: 8c3b76101653417804997e5f076623a25586f3e8

view details

Ramkumar Vadivelu

commit sha c203e01773e59c48ff2b0eaae14f18f1bc37d73f

reset refitting_level_ flag to false in error paths (#7403) Summary: Reset refitting_level_ flag to false in error paths in DBImpl::ReFitLevel() Pull Request resolved: https://github.com/facebook/rocksdb/pull/7403 Reviewed By: ajkr Differential Revision: D23909028 Pulled By: ramvadiv fbshipit-source-id: 521ad9aadc1b734bef9ef9119d1e1ee1fa8126e9

view details

Levi Tamasi

commit sha 1abbc56abac20c3ed646532fdbbfb1daa4cf0ded

Add version_builder_test to the list of ASSERT_STATUS_CHECKED tests (#7444) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7444 Test Plan: `ASSERT_STATUS_CHECKED=1 make version_builder_test -j24` Reviewed By: jay-zhuang Differential Revision: D23965793 Pulled By: ltamasi fbshipit-source-id: 8beaf66548379f21146189cda699d5f6fbb35a1b

view details

Zhichao Cao

commit sha d71cfe04e4990209612e0c388bfda3676d97a3ed

Add flush_job_test to the list of ASSERT_STATUS_CHECKED tests (#7445) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7445 Test Plan: pass ASSERT_STATUS_CHECKED=1 make -j48 flush_job_test Reviewed By: akankshamahajan15 Differential Revision: D23969372 Pulled By: zhichao-cao fbshipit-source-id: 498ff45ef84e07ec27a8f35d0874d3371412afe9

view details

Andrew Kryczka

commit sha 7a23a2175bacc4bd30998e503544d066edc6147e

enable Status check assertions for sst_file_reader_test (#7448) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7448 Reviewed By: jay-zhuang Differential Revision: D23973281 Pulled By: ajkr fbshipit-source-id: 90fe55f1db904f1a236f3f7994d0806b8d24282c

view details

Yanqin Jin

commit sha 8c7bac6491ea94a3c431ecb83eec6b9477c6709e

Check status for file_reader_writer_test (#7449) Summary: Check the status for file_reader_writer_test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7449 Test Plan: ``` ASSERT_STATUS_CHECKED=1 make -j20 file_reader_writer_test ./file_reader_writer_test ``` Reviewed By: zhichao-cao Differential Revision: D23975609 Pulled By: riversand963 fbshipit-source-id: a468eb04b386967fcc0478a56e4f0a19bdf81cdf

view details

Levi Tamasi

commit sha 5e221a98b5cbe65a40d945685821877b43f259bc

Support injecting read errors for RandomAccessFile when using FaultInjectionTestEnv (#7447) Summary: The patch adds support for injecting errors when reading from `RandomAccessFile` using `FaultInjectionTestEnv`. (This functionality was curiously missing w/r/t `RandomAccessFile`, even though it was implemented for `RandomRWFile`.) The patch also fixes up a test case in `blob_db_test` which uses `FaultInjectionTestEnv` but has so far relied on reads from `RandomAccessFile`s succeeding even after deactivating the filesystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7447 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D23971740 Pulled By: ltamasi fbshipit-source-id: 8492736cb64b1ee138c658822535f3ff4fe560c6

view details

sdong

commit sha d08a9005b76a7c96d25326ecd98890c6d2b77511

Make db_basic_test pass assert status checked (#7452) Summary: Add db_basic_test status check list. Some of the warnings are suppressed. It is possible that some of them are due to real bugs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7452 Test Plan: See CI tests pass. Reviewed By: zhichao-cao Differential Revision: D23979764 fbshipit-source-id: 6151570c2a9b931b0fbb3fe939a94b2bd1583cbe

view details

anand76

commit sha 12ede5ed7cb64ba9d5f4a848be1d940d8085f69f

Remove invalid assertion in compaction_picker_universal.cc (#7421) Summary: The assertion checks that there is no overlap in sequence numbers across levels in universal compaction. However, this assumption doesn't hold when there is a delete triggered compaction or a trivial move, as they operate on a subset of a level. Tests - make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/7421 Reviewed By: ajkr Differential Revision: D23872672 Pulled By: anand1976 fbshipit-source-id: c386deab8e01a5746ca996ff1f4ebcae3b15b7d2

view details

Yanqin Jin

commit sha 07dc955a1f774e9f5699cde7cbd24f342297dc3c

Report error of GetChildren (#7459) Summary: As title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7459 Test Plan: make check Reviewed By: anand1976 Differential Revision: D23999393 Pulled By: riversand963 fbshipit-source-id: 09df8e1637f4df3616c63ee314de397b35be4e4a

view details

Andrew Kryczka

commit sha 1600aac46f64f906631f64d036b2ea5359c7ef5f

Flush info log for warning and higher severity (#7462) Summary: After unclean crash, the tail of the log could look as follows due to block buffering, even when the call to `ROCKSDB_LOG_ERROR()` finished. ``` 2020/09/29-13:54:39.596710 7f67025fe700 [ERROR] [/db_impl/db_impl_compaction_flush.cc:2500] Waiting after background compaction err ``` This PR forces the flush while logging warning severity or higher to prevent that case. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7462 Reviewed By: riversand963 Differential Revision: D24000154 Pulled By: ajkr fbshipit-source-id: 3bf5f1e69a62ee10e84095cebc88937a8f81b4ad

view details

anand76

commit sha bd5d9e2a1d79fa8f49965499a1d8228fd7ae191e

Fix misspelling of PartitionedIndexIterator (#7450) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7450 Reviewed By: zhichao-cao Differential Revision: D23992811 Pulled By: anand1976 fbshipit-source-id: 71bd898aafce6a3add3c8cd86d9f0e0fb63860cf

view details

Andrew Kryczka

commit sha 8115eb520de5329977f0430d6f4f6e3309fd233f

add Status check assertions for repair_test (#7455) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7455 Reviewed By: riversand963 Differential Revision: D23985283 Pulled By: ajkr fbshipit-source-id: 5dd2be62350f6e31d13a1e7821cb848a37699c93

view details

Jay Zhuang

commit sha 1bdaef7a06ee2fc147f1188cec6781905849c71b

Status check enforcement for timestamp_basic_test (#7454) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7454 Reviewed By: riversand963 Differential Revision: D23981719 Pulled By: jay-zhuang fbshipit-source-id: 01073f73e54c17067b886c4a2f179b2804198399

view details

Akanksha Mahajan

commit sha 9d212d3f0ecee2e6f72286497dc2b7a3e2d3ba58

Provide users with option to opt-in to get corrupt data in logs/messages (#7420) Summary: Add a new Option "allow_data_in_errors". When it's set by users, it allows them to opt-in to get error messages containing corrupted keys/values. Corrupt keys, values will be logged in the messages, logs, status etc. that will help users with the useful information regarding affected data. By default value is set false to prevent users data to be exposed in the messages. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7420 Test Plan: 1. make check -j64 2. Add a new test case Reviewed By: ajkr Differential Revision: D23835028 Pulled By: akankshamahajan15 fbshipit-source-id: 8d2eba8fb898e79fcf1fccc07295065a75eb59b1

view details

sdong

commit sha 5f33436285c532936de8870bf22badf1568366dd

Revert an uncessary status code check skipping (#7458) Summary: https://github.com/facebook/rocksdb/pull/7452 added an uncessary skip for status code checking. Revert it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7458 Test Plan: Watch CI to finish Reviewed By: jay-zhuang Differential Revision: D23994390 fbshipit-source-id: a2b50a6326d8073db3386bff3d32acc5a6666e9b

view details

Peter Dillinger

commit sha ddbc5dad0505ba88c60975f146704f04c359d4c3

Enable force_consistency_checks by default (#7446) Summary: This has been running in production on some key workloads, so we believe it to be safe and extremely low cost. Nevertheless, I've added code to ensure that "force_consistency_checks" is mentioned in any corruption reports so that people know how to disable in case of false positive corruption reports. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7446 Test Plan: make check, CI, temporary debug print new message with ./version_builder_test Reviewed By: ajkr Differential Revision: D23972101 Pulled By: pdillinger fbshipit-source-id: 9623e400f3752577c0ecf977e6d0915562cf9968

view details

Akanksha Mahajan

commit sha 5997f6b17bfe571a5882f2f490d81cf5c6b44732

Add Status check enforcement for unit tests (#7464) Summary: Add status check for unit tests : block_based_filter_block_test, block_fetcher_test, full_filter_block_test and partitioned_filter_block_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7464 Reviewed By: zhichao-cao Differential Revision: D24011309 Pulled By: akankshamahajan15 fbshipit-source-id: d814803f94e8bb8b811ef170d20b22d52c1a3ff2

view details

sdong

commit sha aedcaaef99a3ed55de799219d9e30fcea53189cc

Stress test to support paranoid_file_checks (#7473) Summary: It's important to make sure no false positive is reported when options.paranoid_file_checks is used. Add it to stress test and a place holder in crash test. It is disabled in crash test as there appears to be a bug causing false positive. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7473 Test Plan: Run crash test Reviewed By: ajkr Differential Revision: D24026939 fbshipit-source-id: 89102acb45cf041776775ce44a4eef4b0f3a380c

view details

Jay Zhuang

commit sha 58905f31c18aa79b6f278601dd824c3d50ee4f54

Remove a gtest-parallel workaround (#7433) Summary: As the issue is fixed in official repo: https://github.com/google/gtest-parallel/pull/79 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7433 Reviewed By: ajkr Differential Revision: D24023271 Pulled By: jay-zhuang fbshipit-source-id: 0babf5e4c59cd61ded5a64cf9aa2d457deeeaa47

view details

push time in a month

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha bf8eb649b3803bf89544fd616ef0b2ae21aa0e17

Add status check enforcement for column_family_test.cc Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

PR opened facebook/rocksdb

Add status check enforcement for column_family_test.cc

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

+49 -29

0 comment

4 changed files

pr created time in a month

push eventakankshamahajan15/rocksdb

Akanksha Mahajan

commit sha 9d212d3f0ecee2e6f72286497dc2b7a3e2d3ba58

Provide users with option to opt-in to get corrupt data in logs/messages (#7420) Summary: Add a new Option "allow_data_in_errors". When it's set by users, it allows them to opt-in to get error messages containing corrupted keys/values. Corrupt keys, values will be logged in the messages, logs, status etc. that will help users with the useful information regarding affected data. By default value is set false to prevent users data to be exposed in the messages. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7420 Test Plan: 1. make check -j64 2. Add a new test case Reviewed By: ajkr Differential Revision: D23835028 Pulled By: akankshamahajan15 fbshipit-source-id: 8d2eba8fb898e79fcf1fccc07295065a75eb59b1

view details

sdong

commit sha 5f33436285c532936de8870bf22badf1568366dd

Revert an uncessary status code check skipping (#7458) Summary: https://github.com/facebook/rocksdb/pull/7452 added an uncessary skip for status code checking. Revert it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7458 Test Plan: Watch CI to finish Reviewed By: jay-zhuang Differential Revision: D23994390 fbshipit-source-id: a2b50a6326d8073db3386bff3d32acc5a6666e9b

view details

Peter Dillinger

commit sha ddbc5dad0505ba88c60975f146704f04c359d4c3

Enable force_consistency_checks by default (#7446) Summary: This has been running in production on some key workloads, so we believe it to be safe and extremely low cost. Nevertheless, I've added code to ensure that "force_consistency_checks" is mentioned in any corruption reports so that people know how to disable in case of false positive corruption reports. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7446 Test Plan: make check, CI, temporary debug print new message with ./version_builder_test Reviewed By: ajkr Differential Revision: D23972101 Pulled By: pdillinger fbshipit-source-id: 9623e400f3752577c0ecf977e6d0915562cf9968

view details

Akanksha Mahajan

commit sha 5997f6b17bfe571a5882f2f490d81cf5c6b44732

Add Status check enforcement for unit tests (#7464) Summary: Add status check for unit tests : block_based_filter_block_test, block_fetcher_test, full_filter_block_test and partitioned_filter_block_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7464 Reviewed By: zhichao-cao Differential Revision: D24011309 Pulled By: akankshamahajan15 fbshipit-source-id: d814803f94e8bb8b811ef170d20b22d52c1a3ff2

view details

sdong

commit sha aedcaaef99a3ed55de799219d9e30fcea53189cc

Stress test to support paranoid_file_checks (#7473) Summary: It's important to make sure no false positive is reported when options.paranoid_file_checks is used. Add it to stress test and a place holder in crash test. It is disabled in crash test as there appears to be a bug causing false positive. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7473 Test Plan: Run crash test Reviewed By: ajkr Differential Revision: D24026939 fbshipit-source-id: 89102acb45cf041776775ce44a4eef4b0f3a380c

view details

Jay Zhuang

commit sha 58905f31c18aa79b6f278601dd824c3d50ee4f54

Remove a gtest-parallel workaround (#7433) Summary: As the issue is fixed in official repo: https://github.com/google/gtest-parallel/pull/79 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7433 Reviewed By: ajkr Differential Revision: D24023271 Pulled By: jay-zhuang fbshipit-source-id: 0babf5e4c59cd61ded5a64cf9aa2d457deeeaa47

view details

Andrew Kryczka

commit sha 718e192965cabb311feb7718d9cb8a2e34958072

Fix flaky intra-L0 consistency failure regression tests (#7477) Summary: Do not assert the number of files after intra-L0 compaction is eligible to run since it could complete (and reduce the number of files) before the assertion executes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7477 Reviewed By: pdillinger Differential Revision: D24032049 Pulled By: ajkr fbshipit-source-id: e838ac7a24651ebd643b9e5a9d39d2e789c46929

view details

Jay Zhuang

commit sha e127fe18c3ee0d99b4507f1e2b7cfcb38a653e74

Fix TSAN failure for backupable_db_test (#7478) Summary: It's a transient failure, but can be reproduce with running the test 100 times: https://app.circleci.com/pipelines/github/facebook/rocksdb/3760/workflows/de909685-f22b-45ba-a8f3-6ebb78a54e96/jobs/37039 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7478 Test Plan: re-run the test 100 times Reviewed By: ajkr Differential Revision: D24035758 Pulled By: jay-zhuang fbshipit-source-id: 6b31983d5c3f7faa8d5481306098513485d0d69d

view details

akankshamahajan

commit sha 57b9247946f801cc6152df2d34e6f4ea161ae01e

Add status check enforcement for column_family_test.cc Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

Pull request review commentfacebook/rocksdb

Status check enforcement for error_handler_fs_test

 Status DBImpl::ResumeImpl(DBRecoverContext context) {     // during previous error handling.     if (file_deletion_disabled) {       // Always return ok-      EnableFileDeletions(/*force=*/true);+      EnableFileDeletions(/*force=*/true).PermitUncheckedError();

Can we add assert or check status here in case if different status is returned in future, PermitUncheckedError might ignore it?

zhichao-cao

comment created time in a month

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 0bb4f3edcd59de1dc755ab37be8826b3f9c9b6b4

Add Status check enforcement for unit tests Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 5bf7e61beb2d3b7f9eddf26d8fb477f79f392ec2

Add Status check enforcement for unit tests Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

PR opened facebook/rocksdb

Add Status check enforcement for unit tests

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

+2 -2

0 comment

1 changed file

pr created time in a month

push eventakankshamahajan15/rocksdb

Cheng Chang

commit sha ada700b9062560dff1614fe2fe934bfaf6a9ca94

Re-read the whole request in direct IO mode when IO uring returns partial result (#6853) Summary: If both direct IO and IO uring are enabled, when IO uring returns partial result, we'll try to read the remaining part of the request, but the starting address/offset of the remaining part might not be aligned to the block size, in direct IO mode, the unaligned offset causes bug. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6853 Test Plan: run make check with both direct IO and IO uring enabled, this is covered by one of the continuous tests. Reviewed By: anand1976 Differential Revision: D21603023 Pulled By: cheng-chang fbshipit-source-id: 942f6a11ff21e1892af6c4464e02bab4c707787c

view details

mrambacher

commit sha 0ac0098705761e0b7486b1c1637410e516e9e97c

Make options length longer for sst_dump_test (#6846) Summary: Under MacOS when running with make -j 8 check, the temporary directory generated was > 100 characters. This caused the tests to do nothing under MacOS. Most of them still reported success for doing nothing, but ReadaheadSize was expecting the test to run. By making the option name longer, the tests will no run successfully (and do something!) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6846 Reviewed By: ajkr Differential Revision: D21576032 fbshipit-source-id: b089cde0d598137b572aa8527cc5459085252af7

view details

Levi Tamasi

commit sha 1551f1011ab419d21ac50be4b894cf3f688b30d7

Refactor the blob file related logic in VersionBuilder (#6835) Summary: This patch is groundwork for an upcoming change to store the set of linked SSTs in `BlobFileMetaData`. With the current code, a new `BlobFileMetaData` object is created each time a `VersionEdit` touches a certain blob file. This is fine as long as these objects are lightweight and cheap to create; however, with the addition of the linked SST set, it would be very inefficient since the set would have to be copied over and over again. Note that this is the same kind of problem that `VersionBuilder` is solving w/r/t `Version`s and files, and we can apply the same solution; that is, we can accumulate the changes in a different mutable object, and apply the delta in one shot when the changes are committed. The patch does exactly that by adding a new `BlobFileMetaDataDelta` class to `VersionBuilder`. In addition, it turns the existing `GetBlobFileMetaData` helper into `IsBlobFileInVersion` (which is fine since that's the only thing the method's clients care about now), and adds a couple of helper methods that can create a `BlobFileMetaData` object from the `BlobFileMetaData` in the base (if applicable) and the delta when the `Version` is saved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6835 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21505187 Pulled By: ltamasi fbshipit-source-id: d81a48c5f2ca7b79d7124c935332a6bcf3d5d988

view details

anand76

commit sha 39b24432d42dd6eef63af29ad850b618b0ec65f6

Strengthen MultiGet correctness verification in NoBatchedOpsStress (#6849) Summary: Add MultiGet to VerifyDb and check consistency with Get in TestMultiGet. Test plan - make crash_test ASAN crash test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6849 Reviewed By: pdillinger Differential Revision: D21635011 Pulled By: anand1976 fbshipit-source-id: deb5a79d08fefd8d8010204f1f20b83adc92310e

view details

Akanksha Mahajan

commit sha a1523efcdf2f0e8133b9a9f6e170a0dad49f928f

Status check enforcement for io_posix_test and options_settable_test (#6857) Summary: Added status check enforcement for io_posix_test and options_settable_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6857 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 check Reviewed By: ajkr Differential Revision: D21647904 Pulled By: akankshamahajan15 fbshipit-source-id: b7f2321eb6c141a88cd5e1270ecb7d58f00341af

view details

Peter Dillinger

commit sha aaafcb80ab305321783c9b9e5f701f11ee12038a

Use in-repo gtest in buck build (#6858) Summary: ... so that we have freedom to upgrade it (see https://github.com/facebook/rocksdb/issues/6808). As a side benefit, gtest will no longer be linked into main library in buck build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6858 Test Plan: fb internal build & link Reviewed By: riversand963 Differential Revision: D21652061 Pulled By: pdillinger fbshipit-source-id: 6018104af944debde576b5beda6c134e737acedb

view details

Zhichao Cao

commit sha 545e14b53be58a282eb3ae05fe627589f96d9de0

Generate file checksum in SstFileWriter (#6859) Summary: If Option.file_checksum_gen_factory is set, rocksdb generates the file checksum during flush and compaction based on the checksum generator created by the factory and store the checksum and function name in vstorage and Manifest. This PR enable file checksum generation in SstFileWrite and store the checksum and checksum function name in the ExternalSstFileInfo, such that application can use them for other purpose, for example, ingest the file checksum with files in IngestExternalFile(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6859 Test Plan: add unit test and pass make asan_check. Reviewed By: ajkr Differential Revision: D21656247 Pulled By: zhichao-cao fbshipit-source-id: 78a3570c76031d8832e3d2de3d6c79cdf2b675d0

view details

anand76

commit sha eb04bb86c6e713230ab07ee7ab919feeb9f891e0

Fix a bug in crash_test_with_txn (#6860) Summary: In NoBatchedOpsStress::TestMultiGet, call txn->Get() when transactions are in use. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6860 Test Plan: make crash_test_with_txn Reviewed By: pdillinger Differential Revision: D21667249 Pulled By: anand1976 fbshipit-source-id: 194bd7b9630a8efc3ae29d85422a61214e9e200e

view details

Peter Dillinger

commit sha c7aedf1b48d8dfd9a2726b20bec9daee7be833f4

Clean up some code related to file checksums (#6861) Summary: * Add missing unit test for schema stability of FileChecksumGenCrc32c (previously was only comparing to itself) * A lot of clarifying comments * Add some assertions for preconditions * Rename WritableFileWriter::CalculateFileChecksum -> UpdateFileChecksum * Simplify FileChecksumGenCrc32c with shared functions * Implement EndianSwapValue to replace unused EndianTransform And incidentally since I had trouble with 'make check-format' GitHub action disagreeing with local run, * Output full diagnostic information when 'make check-format' fails in CI Pull Request resolved: https://github.com/facebook/rocksdb/pull/6861 Test Plan: new unit test passes before & after other changes Reviewed By: zhichao-cao Differential Revision: D21667115 Pulled By: pdillinger fbshipit-source-id: 6a99970f87605aa024fa540c78cd519ff322c3e6

view details

mrambacher

commit sha 38be6861601a378dd98e2d83da328ecbb61cafc9

Add Struct Type to OptionsTypeInfo (#6425) Summary: Added code for generically handing structs to OptionTypeInfo. A struct is a collection of variables handled by their own map of OptionTypeInfos. Examples of structs include Compaction and Cache options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6425 Reviewed By: siying Differential Revision: D21668789 Pulled By: zhichao-cao fbshipit-source-id: 064b110de39dadf82361ed4663f7ac1a535b0b07

view details

Andrew Kryczka

commit sha 292bcf6227e7cf39e5cd3ff9d588ff860151d131

skip direct I/O tests in rocksdb lite (#6867) Summary: Fix a couple places where direct I/O was used even though it is unsupported in lite builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6867 Test Plan: `LITE=1 make check -j48` Reviewed By: pdillinger Differential Revision: D21689185 Pulled By: ajkr fbshipit-source-id: 3eaa3abf69cd7d0bcaabbcad3bb5a26fb8dd7301

view details

mrambacher

commit sha 826295a5e9d62e90d6773e1b1221d650de2f3fe4

Change autovector to have a reserved size in LITE mode (#6868) Summary: Previously in LITE mode, an autovector did not have a reserved size. When elements were added to the vector, the underlying array could be reallocated. There was a set of code that never expands the autovector and was doing &autovector::back(). When the vector is resized, the old addresses may become invalid, causing a later exception to be thrown. By reserving space in the autovector up front, this problem is eliminated for those uses where the vector will never exceed the initial size. the resize happens, these pointers become invalid, leading to SEGV or other exceptions. This change allows the autovector to be fully populated before we take the address of any of its elements, thereby elminating the potential for a resize. There is comparable code to this change in Version::MultiGet for dealing with the context objects. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6868 Reviewed By: ajkr Differential Revision: D21693505 Pulled By: cheng-chang fbshipit-source-id: e71d516b15e08f202593cb80f2a42f048fc95768

view details

Peter Dillinger

commit sha 35a25a3fb9b5a61a389e853f8be37932487edd03

Fix/expand ASSERT_STATUS_CHECKED build, add to Travis (#6870) Summary: Fixed some option handling code that recently broke the ASSERT_STATUS_CHECKED build for options_test. Added all other existing tests that pass under ASSERT_STATUS_CHECKED to the whitelist. Added a Travis configuration to run all whitelisted tests with ASSERT_STATUS_CHECKED. (Someday we might enable this check by default in debug builds.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6870 Test Plan: ASSERT_STATUS_CHECKED=1 make check, Travis Reviewed By: ajkr Differential Revision: D21704374 Pulled By: pdillinger fbshipit-source-id: 15daef98136a19d7a6843fa0c9ec08738c2ac693

view details

Marek Kurdej

commit sha bcd32560dd5898956b9d24553c2bb3c1b1d2319f

Fix warning -Wextra-semi. NFC. (#6869) Summary: Minor fix. CLA signed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6869 Reviewed By: ajkr Differential Revision: D21704001 Pulled By: pdillinger fbshipit-source-id: 57fd08114f3234f51f34758e25e708cc70962582

view details

Peter Dillinger

commit sha 803a517b48ddd01689cc9c6feb0f4b7b19c317e6

Misc things for ASSERT_STATUS_CHECKED, also gcc 4.8.5 (#6871) Summary: * Print stack trace on status checked failure * Make folly_synchronization_distributed_mutex_test a parallel test * Disable ldb_test.py and rocksdb_dump_test.sh with ASSERT_STATUS_CHECKED (broken) * Fix shadow warning in random_access_file_reader.h reported by gcc 4.8.5 (ROCKSDB_NO_FBCODE), also https://github.com/facebook/rocksdb/issues/6866 * Work around compiler bug on max_align_t for gcc < 4.9 * Remove an apparently wrong comment in status.h * Use check_some in Travis config (for proper diagnostic output) * Fix ignored Status in loop in options_helper.cc Pull Request resolved: https://github.com/facebook/rocksdb/pull/6871 Test Plan: manual, CI Reviewed By: ajkr Differential Revision: D21706619 Pulled By: pdillinger fbshipit-source-id: daf6364173d6689904eb394461a69a11f5bee2cb

view details

Andrew Kryczka

commit sha b559dba817bdb2e23d6576acfae3ca3378ebb29d

pin image version in circle CI builds (#6876) Summary: somehow the windows-server-2019-vs2019 image changed in a way that made VS 14 2015 the default. This caused an error when we specify VS 16 2019 as the cmake generator. I could not figure out the right arguments/env vars to get the latest VS working so pinned the image to the previous version instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6876 Reviewed By: pdillinger Differential Revision: D21709679 Pulled By: ajkr fbshipit-source-id: 2d16819ad239b4611fa199547744e1c101dc9da0

view details

Andrew Kryczka

commit sha b464a85e337ea801eabff0595a933c4e6fc793dd

fix transaction rollback in db_stress TestMultiGet (#6873) Summary: There were further uses of `txn` after `RollbackTxn(txn)` leading to stress test errors. Moved the rollback to the end of the function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6873 Test Plan: found a command from the crash test that previously failed immediately under TSAN; verified now it succeeds. ``` ./db_stress --acquire_snapshot_one_in=10000 --allow_concurrent_memtable_write=1 --avoid_flush_during_recovery=0 --avoid_unnecessary_blocking_io=1 --block_size=16384 --bloom_bits=222.913637674 --bottommost_compression_type=none --cache_index_and_filter_blocks=1 --cache_size=1048576 --checkpoint_one_in=0 --checksum_type=kCRC32c --clear_column_family_one_in=0 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_style=1 --compaction_ttl=0 --compression_max_dict_bytes=0 --compression_parallel_threads=1 --compression_type=zstd --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --db=/dev/shm/rocksdb/rocksdb_crashtest_whitebox --db_write_buffer_size=1048576 --delpercent=5 --delrangepercent=0 --destroy_db_initially=0 --disable_wal=0 --enable_pipelined_write=0 --flush_one_in=1000000 --format_version=5 --get_current_wal_file_one_in=0 --get_live_files_one_in=1000000 --get_sorted_wal_files_one_in=0 --index_block_restart_interval=12 --index_type=2 --key_len_percent_dist=1,30,69 --level_compaction_dynamic_level_bytes=True --log2_keys_per_lock=22 --long_running_snapshots=0 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --max_key=1000000 --max_key_len=3 --max_manifest_file_size=1073741824 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=3 --memtablerep=skip_list --mmap_read=1 --mock_direct_io=False --nooverwritepercent=1 --num_levels=1 --open_files=100 --ops_per_thread=200000 --partition_filters=1 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefixpercent=5 --progress_reports=0 --read_fault_one_in=0 --readpercent=45 --recycle_log_file_num=0 --reopen=20 --snapshot_hold_ops=100000 --subcompactions=4 --sync=0 --sync_fault_injection=False --target_file_size_base=2097152 --target_file_size_multiplier=2 --test_batches_snapshots=0 --txn_write_policy=1 --unordered_write=1 --use_block_based_filter=0 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=1 --use_merge=1 --use_multiget=1 --use_txn=1 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_db_one_in=100000 --write_buffer_size=4194304 --write_dbid_to_manifest=0 --writepercent=35 ``` Reviewed By: pdillinger Differential Revision: D21708338 Pulled By: ajkr fbshipit-source-id: dcf55cddee0a14f429a75e7a8a505acf8025f2b1

view details

Kefu Chai

commit sha bd68bfb41b58dba4332045392cdbda4664b01e81

cmake: link env_librados_test against rados (#6855) Summary: otherwise we have FTBFS like: 2020-05-18T15:12:06.400 INFO:tasks.workunit.client.0.smithi032.stdout:[100%] Linking CXX executable env_librados_test 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:/usr/bin/ld: CMakeFiles/rocksdb_env_librados_test.dir/utilities/env_librados_test.cc.o: undefined reference to symbol '_ZN8librados7v14_2_05Rados4initEPKc@LIBRADOS_14.2.0' 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:/usr/bin/ld: /lib/librados.so.2: error adding symbols: DSO missing from command line 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:collect2: error: ld returned 1 exit status this addresses the regression introduced by 07204837ce8d66e1e6e4893178f3fd040f9c1044, which hides the symbols exposed by `${THIRDPARTY_LIBS}` from consumers of librocksdb Signed-off-by: Kefu Chai <tchaikov@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6855 Reviewed By: riversand963 Differential Revision: D21621904 fbshipit-source-id: 7022ba4dc0003504401fce6f06547e4d74a32ac0

view details

Cheng Chang

commit sha 82a82c76e702176bd5bfb92dafb354597af099c1

Fix potential memory leak of scratch buffer (#6879) Summary: If `req.scratch` is an internally allocated buffer, but `raw_block_contents` is not constructed to own `req.scratch`, then `req.scratch` will be leaked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6879 Test Plan: make asan_check Reviewed By: anand1976 Differential Revision: D21728498 Pulled By: cheng-chang fbshipit-source-id: 8fc6a4f2543918c565ddc16ecfad1807eb9a42cf

view details

Adam Retter

commit sha 9060e6fa793d93d410bf0a265da3a6936d3b6a20

Add newer WBWI::NewIteratorWithBase functions to RocksJava (#6872) Summary: Exposes the `ReadOptions` arguments to `NewIteratorWithBase`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6872 Reviewed By: ajkr Differential Revision: D21725867 Pulled By: pdillinger fbshipit-source-id: 4079ba590cc13ba7a6244ed91439d89c40a543b6

view details

push time in a month

pull request commentfacebook/rocksdb

Return error if Get() fails in Prefetching Filter blocks

Failure: ci/circleci: build-linux-clang10-ubsan not related to this change.

akankshamahajan15

comment created time in a month

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha 2bb4652504360190fd64db88e9c8c6fee276f6ca

Update unit test Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

push eventakankshamahajan15/rocksdb

akankshamahajan

commit sha d641a158dd4b6aaf1dca294538b33bbd9d31bd54

Return error if Get() fails in Prefetching Filter blocks Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails. Test Plan: make check -j64 Reviewers: Subscribers: Tasks: Tags:

view details

push time in a month

PR opened facebook/rocksdb

Return error if Get() fails in Prefetching Filter blocks

Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails.

Test Plan: make check -j64

Reviewers:

Subscribers:

Tasks:

Tags:

+27 -23

0 comment

5 changed files

pr created time in a month

create barnchakankshamahajan15/rocksdb

branch : return_error

created branch time in a month

more