tqcq/leveldb
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
434 Commits 3 Branches 21 Tags 42 MiB
Commit Graph

434 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
leveldb Team
068d5ee1a3 leveldb: Check slice length in Footer::DecodeFrom() 
Without this check decoding the footer in Table::Open() can read
uninitialized bytes from a buffer allocated on the stack if the file
was unexpectedly short.

In practice this is probably fine since this function validates a magic
number but MSan complains about branching on uninitialized data.

PiperOrigin-RevId: 525271012
 2023-04-20 18:09:06 +00:00
leveldb Team
c61238dcf3 Support Zstd compression level in Leveldb 
PiperOrigin-RevId: 520556840
 2023-04-20 18:08:55 +00:00
Victor Costan
77d66aaf3e Fix GitHub CI on Linux. 
This PR temporarily removes a package that is currently broken on
GitHub's Ubuntu 22.04 installation. This is the most expedient way to
make the CI green again, so we can test any other changes we may want to
land.

PiperOrigin-RevId: 520206940
 2023-03-28 20:17:54 -07:00
Victor Costan
9cbbc5fb75 Merge pull request #1104 from reillyeon:chromium_env 
PiperOrigin-RevId: 520172744
 2023-03-28 20:17:45 -07:00
Victor Costan
80d858fb2a Merge pull request #1106 from reillyeon:run_many 
PiperOrigin-RevId: 520171344
 2023-03-28 16:49:26 -07:00
leveldb Team
1d6e8d64ee Add support for Zstd-based compression in LevelDB. 
This change implements support for Zstd-based compression in LevelDB. Building
up from the Snappy compression (which has been supported since inception), this
change adds Zstd as an alternate compression algorithm.

We are implementing this to provide alternative options for users who might
have different performance and efficiency requirements. For instance, the
Zstandard website (https://facebook.github.io/zstd/) claims that the Zstd
algorithm can achieve around 30% higher compression ratios than Snappy, with
relatively smaller (~10%) slowdowns in de/compression speeds.

Benchmarking results:

$ blaze-bin/third_party/leveldb/db_bench
LevelDB:    version 1.23
Date:       Thu Feb  2 18:50:06 2023
CPU:        56 * Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
CPUCache:   35840 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
------------------------------------------------
fillseq      :       2.613 micros/op;   42.3 MB/s
fillsync     :    3924.432 micros/op;    0.0 MB/s (1000 ops)
fillrandom   :       3.609 micros/op;   30.7 MB/s
overwrite    :       4.508 micros/op;   24.5 MB/s
readrandom   :       6.136 micros/op; (864322 of 1000000 found)
readrandom   :       5.446 micros/op; (864083 of 1000000 found)
readseq      :       0.180 micros/op;  613.3 MB/s
readreverse  :       0.321 micros/op;  344.7 MB/s
compact      :  827043.000 micros/op;
readrandom   :       4.603 micros/op; (864105 of 1000000 found)
readseq      :       0.169 micros/op;  656.3 MB/s
readreverse  :       0.315 micros/op;  350.8 MB/s
fill100K     :     854.009 micros/op;  111.7 MB/s (1000 ops)
crc32c       :       1.227 micros/op; 3184.0 MB/s (4K per op)
snappycomp   :       3.610 micros/op; 1081.9 MB/s (output: 55.2%)
snappyuncomp :       0.691 micros/op; 5656.3 MB/s
zstdcomp     :      15.731 micros/op;  248.3 MB/s (output: 44.1%)
zstduncomp   :       4.218 micros/op;  926.2 MB/s
PiperOrigin-RevId: 509957778
 2023-03-28 16:49:13 -07:00
Reilly Grant
13ebad24dc Address comments. 2023-03-28 16:28:35 -07:00
Reilly Grant
df68d9578c Fix EnvTest.RunMany to allow parallel execution 
As allowed by the documentation for Env::Schedule(), ChromiumEnv may
execute functions on multiple threads and guarantees no sequencing.
EnvTest.RunMany assumed that functions ran in order, is the case for the
stock PosixEnv and WindowsEnv implementations. This change updates the
test to not assume sequential execution.
 2023-03-28 16:17:21 -07:00
Reilly Grant
bfae97ff7d Roll third_party/benchmark to f7547e29ccaed7b64ef4f7495ecfff1c9f6f3d03 
Fixes an unused variable warning.
 2023-03-28 15:13:57 -07:00
Reilly Grant
89ea7f2643 Fix tests when run against ChromiumEnv 
There are a couple differences between ChromiumEnv and
PosixEnv/WindowsEnv which cause test failures that are fixed (or at
least patched over) in this change:

* NewSequentialFile() and NewRandomAccessFile() return Status::IOError
  rather than Status::NotFound when a file is not found, due to
  https://crbug.com/760362. This means a few tests need to expect a
  different error result.
* GetChildren() never returns the '.' or '..' entries.
* As allowed by the documentation for Env::Schedule(), ChromiumEnv may
  execute functions on multiple threads and guarantees no sequencing.
  EnvTest.RunMany assumed that functions ran in order. The test has been
  updated.
 2023-03-28 14:37:48 -07:00
Sanjay Ghemawat
fb644cb445 Stop future writes if a log file Close() fails. 
See https://github.com/google/leveldb/issues/1081

PiperOrigin-RevId: 499519182
 2023-01-04 20:41:21 +00:00
leveldb Team
aa5479bbf4 Fix maintenance text 
PiperOrigin-RevId: 461725664
 2022-07-18 22:11:43 +00:00
Victor Costan
ca684d00b5 Fix Markdown formatting in README. 
PiperOrigin-RevId: 461722304
 2022-07-18 21:59:46 +00:00
leveldb Team
7b650f85de Add note on current development state 
PiperOrigin-RevId: 461695246
 2022-07-18 21:20:02 +00:00
Victor Costan
0a9b7b8e95 Merge pull request #1036 from chjj:benchmark-compression 
PiperOrigin-RevId: 461612590
 2022-07-18 21:19:56 +00:00
leveldb Team
fff74f20ff Use GTEST_SKIP in leveldb:table_test 
This replaces a usage of fprintf to stderr

PiperOrigin-RevId: 454620969
 2022-07-18 21:19:46 +00:00
Christopher Jeffrey
9e1c274074
Add compression flag to benchmarks. 2022-06-19 20:45:14 -04:00
Victor Costan
d019e3605f Merge pull request #1008 from pkasting:main 
PiperOrigin-RevId: 447466466
 2022-05-09 16:20:11 +00:00
Peter Kasting
8b5b1ca96c Fixes for C++20 support. 
Structs with user-declared constructors are no longer considered
aggregates.  Just remove the constructor declaration where applicable.

Bug: chromium:1284275
 2022-05-05 07:44:11 -07:00
Victor Costan
4fb146810c The master branch was renamed to main. 
PiperOrigin-RevId: 422409116
 2022-01-17 21:37:53 +00:00
Victor Costan
479a1f4e9b Update contributing guidelines. 
* Align CONTRIBUTING.md with the google/new-project template.
* Explain the support story for the CMake config.

PiperOrigin-RevId: 421120645
 2022-01-11 22:37:16 +00:00
Victor Costan
1b51a3a968 Merge pull request #506 from lingbin:fix_issue_505 
PiperOrigin-RevId: 420787858
 2022-01-10 18:01:30 +00:00
Victor Costan
a797000713 Merge pull request #893 from myccccccc:master 
PiperOrigin-RevId: 420783873
 2022-01-10 17:56:16 +00:00
Victor Costan
bf4fcd85b5 Merge pull request #888 from JayiceZ:fix_typo 
PiperOrigin-RevId: 420783487
 2022-01-10 17:56:09 +00:00
Victor Costan
8a68093c84 Merge pull request #652 from caodhuan:master 
PiperOrigin-RevId: 420782536
 2022-01-10 17:56:02 +00:00
Victor Costan
bfea90d883 Merge pull request #744 from HenryRLee:patch-1 
PiperOrigin-RevId: 420781374
 2022-01-10 17:55:55 +00:00
Victor Costan
bda46dd00d Merge pull request #602 from andyli029:feature_fix_lack_tag_comment 
PiperOrigin-RevId: 420781095
 2022-01-10 17:55:49 +00:00
Victor Costan
ec4e3a5cb3 Merge pull request #747 from zltl:patch-1 
PiperOrigin-RevId: 420778907
 2022-01-10 17:55:39 +00:00
Victor Costan
3ab94e7da8
Merge branch 'master' into fix_issue_505 2022-01-10 09:48:52 -08:00
Victor Costan
3180f9cb40
Merge branch 'master' into patch-1 2022-01-09 23:08:24 -08:00
Victor Costan
8ccb79b57e Merge pull request #901 from mapleFU:opt-using-move 
PiperOrigin-RevId: 420662938
 2022-01-10 06:35:14 +00:00
Victor Costan
74f0be238f Merge pull request #897 from raynolmenezes:patch-1 
PiperOrigin-RevId: 420662891
 2022-01-10 06:35:06 +00:00
Victor Costan
56f2394250 Merge pull request #945 from xiong-ang:master 
PiperOrigin-RevId: 420645727
 2022-01-10 02:29:23 +00:00
Victor Costan
068a0f1214 Merge pull request #934 from BilyZ98:master 
PiperOrigin-RevId: 420645080
 2022-01-10 02:29:16 +00:00
Victor Costan
8f5aa6375e Merge pull request #919 from wineway:fix_posix_test 
PiperOrigin-RevId: 420644954
 2022-01-10 02:29:08 +00:00
Victor Costan
7a2f90460a Merge pull request #928 from ehds:fix-comment 
PiperOrigin-RevId: 420541137
 2022-01-09 07:31:01 +00:00
Victor Costan
4db0eaccf1 Merge pull request #960 from ericuni:dedup 
PiperOrigin-RevId: 420534594
 2022-01-09 03:18:18 +00:00
Victor Costan
7ee3889a61 VersionSet::Builder::Apply() does not mutate its argument. 
PiperOrigin-RevId: 420533763
 2022-01-09 03:15:31 +00:00
Victor Costan
42cf899927 Merge pull request #903 from LazyWolfLin:dev_random 
PiperOrigin-RevId: 420532625
 2022-01-09 03:15:22 +00:00
Victor Costan
8796c44772 Merge pull request #902 from ehds:update-table-cache 
PiperOrigin-RevId: 420517390
 2022-01-09 03:15:10 +00:00
Victor Costan
e4ccaa0c9c Merge pull request #965 from ShawnZhong:cpp20 
PiperOrigin-RevId: 420504266
 2022-01-08 20:55:42 +00:00
Victor Costan
c8b708d496 Merge pull request #967 from rex4539:typos 
PiperOrigin-RevId: 420403341
 2022-01-08 02:55:46 +00:00
Victor Costan
639195221c Merge pull request #968 from xindubawukong:dxy_remove_code 
PiperOrigin-RevId: 420399272
 2022-01-08 01:29:06 +00:00
xindubawukong
87b3a371b1 remove useless code in cache.h 2022-01-06 03:11:11 +08:00
Dimitris Apostolou
0e8aa26c4e
Fix typos 2022-01-05 11:04:16 +02:00
Victor Costan
8f464e7f68 Remove main() from most tests. 
This gives some flexibility to embedders.

Currently, embedders have to build a binary for each test file.

After this CL, embedders can still choose to have a binary for each test
file, by linking each test file with a googletest target that includes
main() (usually "gtest_main"). Embedders can also choose to build a
single binary for almost all test files, and link with a googletest
target that includes main(). The latter is more convenient for projects
that have very few test binaries, like Chromium.

PiperOrigin-RevId: 419470798
 2022-01-03 21:05:04 +00:00
Shawn Zhong
7a2f64ed50 Update env_posix.cc 2021-12-30 18:33:55 -06:00
Victor Costan
b2801ee1a0 Extract benchmark from db_test.cc. 
The benchmark in db/db_test.cc is extracted to its own file,
benchmarks/db_bench_log.cc.

PiperOrigin-RevId: 418713499
 2021-12-29 03:49:16 +00:00
Victor Costan
335876a133 Add invariant checks to Limiter in Env implementations. 
PiperOrigin-RevId: 417853172
 2021-12-22 19:26:31 +00:00
Eric Wang
42d00a80cc rm redundant code: SetNextFile has already been called before in this function 2021-12-05 11:44:55 +08:00