156 Commits

Author SHA1 Message Date
Victor Costan
8f464e7f68 Remove main() from most tests.
This gives some flexibility to embedders.

Currently, embedders have to build a binary for each test file.

After this CL, embedders can still choose to have a binary for each test
file, by linking each test file with a googletest target that includes
main() (usually "gtest_main"). Embedders can also choose to build a
single binary for almost all test files, and link with a googletest
target that includes main(). The latter is more convenient for projects
that have very few test binaries, like Chromium.

PiperOrigin-RevId: 419470798
2022-01-03 21:05:04 +00:00
Victor Costan
b2801ee1a0 Extract benchmark from db_test.cc.
The benchmark in db/db_test.cc is extracted to its own file,
benchmarks/db_bench_log.cc.

PiperOrigin-RevId: 418713499
2021-12-29 03:49:16 +00:00
leveldb Team
8e62cc5124 Remove the / prefix from the recovery_test test file to prevent a double /.
PiperOrigin-RevId: 388341429
2021-08-03 01:12:08 +00:00
Sanjay Ghemawat
13e3c4efc6 Fix compactions that could end up breaking a run of the same user
key across multiple files.

As reported in Github issue #339, it is incorrect to split the
same user key across multiple compacted files since it causes
tombstones/newer-versions to be dropped, thereby exposing obsolete
data. There was a fix for #339, but it ended up not fully fixing
the problem. (It checked for boundary problems in the first level
being compacted, but not the second). This problem was revealed
by Github issue 887.

We now adjust boundaries to avoid splitting user keys in both the
first level and the second level.

PiperOrigin-RevId: 374921082
2021-05-20 19:13:04 +00:00
Chris Mumford
2a47801868 Use partial path to benchmark/benchmark.h.
Using the partial path offers more flexibility to projects which
may checkout google/benchmark to a different location.

PiperOrigin-RevId: 357819911
2021-02-16 16:51:09 -08:00
Victor Costan
4a919ea4f7 IWYU fixes in db/c.cc.
Fixes https://github.com/google/leveldb/issues/872

PiperOrigin-RevId: 353657701
2021-01-25 17:18:22 +00:00
leveldb Team
8f1861462b Sync MANIFEST before closing in db_impl when creating a new DB.
Add logging with debugging information when failing to load a version set.

PiperOrigin-RevId: 351432332
2021-01-12 21:58:08 +00:00
leveldb Team
8cce47e450 Optimize leveldb block seeks to utilize the current iterator location.
This is beneficial when iterators are reused and seeks are not random
but increasing. It is additionally beneficial with larger block sizes and keys with common prefixes.

Add a benchmark "seekordered" to db_bench that reuses iterators across
increasing seeks.  Add support to the benchmark to count comparisons made and to support common key prefix length. Change benchmark random seeds to be reproducible for entire benchmark suite executions but unique for threads in different benchmarks runs.  This changes a benchmark suite of readrandom,seekrandom from having a 100% found ratio as previously it had the same seed used for fillrandom.

./db_bench --benchmarks=fillrandom,compact,seekordered --block_size=262144 --comparisons=1 --key_prefix=100

without this change (though with benchmark changes):
seekrandom   :      55.309 micros/op; (631820 of 1000000 found)
Comparisons: 27001049
seekordered  :       1.732 micros/op; (631882 of 1000000 found)
Comparisons: 26998402

with this change:
seekrandom   :      55.866 micros/op; (631820 of 1000000 found)
Comparisons: 26952143
seekordered  :       1.686 micros/op; (631882 of 1000000 found)
Comparisons: 25549369

For ordered seeking, this is a reduction of 5% comparisons and a 3% speedup. For random seeking (with single use iterators) the comparisons and speed are less than 1% and likely noise.

PiperOrigin-RevId: 351149832
2021-01-11 15:41:38 +00:00
Chris Mumford
b754fdca72 Fixed fprintf of 64-bit value. 2020-11-30 10:48:17 -08:00
leveldb Team
99ab4730d6 Use external benchmark API header
PiperOrigin-RevId: 339310928
2020-11-30 09:10:59 -08:00
leveldb Team
ed781070b4 Internal test cleanup
PiperOrigin-RevId: 339287832
2020-11-30 09:10:46 -08:00
Victor Costan
c46e79c760 Merge pull request #819 from wzk784533:master
PiperOrigin-RevId: 321000544
2020-07-13 19:14:04 +00:00
wzk784533
28602d3625 avoid unnecessary memory copy 2020-07-11 13:44:11 +08:00
Victor Costan
a6b3a2012e Add some std:: qualifiers to types and functions.
PiperOrigin-RevId: 309110431
2020-04-29 22:33:14 +00:00
Victor Costan
3f934e3705 Switch from C headers to C++ headers.
This CL makes the following substitutions.

* assert.h -> cassert
* math.h -> cmath
* stdarg.h -> cstdarg
* stddef.h -> cstddef
* stdint.h -> cstdint
* stdio.h -> cstdio
* stdlib.h -> cstdlib
* string.h -> cstring

PiperOrigin-RevId: 309080151
2020-04-29 20:51:13 +00:00
Victor Costan
23d67e7c1f Fix C++11 build.
PiperOrigin-RevId: 308839805
2020-04-28 18:05:22 +00:00
leveldb Team
98a3b8cf65 change const to constexpr
PiperOrigin-RevId: 307113877
2020-04-28 00:17:51 +00:00
Victor Costan
5903e7a112 Remove Windows workarounds in some tests.
leveldb::Env::DeleteFile was replaced with leveldb::Env::RemoveFile in
all tests. This allows us to remove workarounds for windows.h #defining
DeleteFile.
PiperOrigin-RevId: 289121105
2020-01-14 18:31:37 -08:00
Victor Costan
a0191e5563 Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}.
The "DeleteFile" method name causes pain for Windows developers, because
<windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA.
Current code uses workarounds, like #undefining DeleteFile everywhere an
Env is declared, implemented, or used.

This CL removes the need for workarounds by renaming Env::DeleteFile to
Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to
Env::RemoveDir. A few internal methods are also renamed for consistency.
Software that supports Windows is expected to migrate any Env
implementations and usage to Remove{File,Dir}, and never use the name
Env::Delete{File,Dir} in its code.

The renaming is done in a backwards-compatible way, at the risk of
making it slightly more difficult to build a new correct Env
implementation. The backwards compatibility is achieved using the
following hacks:

1) Env::Remove{File,Dir} methods are added, with a default
    implementation that calls into Env::Delete{File,Dir}. This makes old
    Env implementations compatible with code that calls into the updated
    API.
2) The Env::Delete{File,Dir} methods are no longer pure virtuals.
    Instead, they gain a default implementation that calls into
    Env::Remove{File,Dir}. This makes updated Env implementations
    compatible with code that calls into the old API.

The cost of this approach is that it's possible to write an Env without
overriding either Rename{File,Dir} or Delete{File,Dir}, without getting
a compiler warning. However, attempting to run the test suite will
immediately fail with an infinite call stack ending in
{Remove,Delete}{File,Dir}, making developers aware of the problem.

PiperOrigin-RevId: 288710907
2020-01-09 09:18:14 -08:00
leveldb Team
d152b23f3b Defend against inclusion of windows.h in tests that invoke
Env::DeleteFile.

PiperOrigin-RevId: 283607548
2020-01-09 09:17:59 -08:00
leveldb Team
583a42b596 Internal change.
PiperOrigin-RevId: 282373286
2019-12-02 11:44:39 -08:00
Victor Costan
1c58902bdc Switch testing harness to googletest.
PiperOrigin-RevId: 281815695
2019-11-21 13:11:40 -08:00
Chris Mumford
95d0ba1cb0 Renamed local variable in DBImpl::Write.
The local variable `updates` in DBImpl::Write was hiding the
`updates` parameter. Renamed to avoid this conflict.

PiperOrigin-RevId: 277089971
2019-10-28 13:24:08 -07:00
Chris Mumford
657ba51429 Added return in Version::Get::State::Match to quiet warning.
Added unreached return at the end of Version::Get::State::Match
to stop this _incorrect_ warning:

    version_set.cc:376:5: warning: control reaches end of
    non-void function [-Wreturn-type]

This warning was being emitted when building with clang 6.0.1-10
and also emitted by lgtm.com when statically analyzing leveldb even
though all SaverState enumeration values were handled.

PiperOrigin-RevId: 272455474
2019-10-28 13:23:53 -07:00
Victor Costan
21304d41f7 Merge pull request #698 from neal-zhu:master
PiperOrigin-RevId: 266001777
2019-08-28 15:05:07 -07:00
neal-zhu
5e921896ee drop fileds in State that are duplicates of fileds in Saver and fix typo 2019-08-28 23:43:34 +08:00
Chris Mumford
53e280b568 Simplify unlocking in DeleteObsoleteFiles.
A recent change (4cb80b7ddce6f) to DBImpl::DeleteObsoleteFiles
unlocked DBImpl::mutex_ while deleting files to allow for
greater concurrency. This change improves on the prior in
a few areas:

1. The table is evicted from the table cache before unlocking
   the mutex. This should only improve performance.
2. This implementation is slightly simpler, but at the cost of
   a bit more memory usage.
3. A comment adding more detail as to why the mutex is being
   unlocked and why it is safe to do so.

PiperOrigin-RevId: 253111645
2019-06-13 15:22:52 -07:00
neal-zhu
107a75b62c cache Saver in State object 2019-06-12 07:05:00 +08:00
neal-zhu
76ca116276 fix bug(uninitialized options pointer in State) 2019-06-12 05:58:00 +08:00
neal-zhu
f668239bb2 remove TODO in Version::ForEachOverlapping 2019-06-11 20:33:18 +08:00
neal-zhu
177cd08629 format 2019-06-11 20:30:54 +08:00
neal-zhu
8fa7a937ee fix bug 2019-06-11 20:20:58 +08:00
neal-zhu
6a90bb91ee use ForEachOverlapping to impl Get 2019-06-11 19:16:49 +08:00
Chris Mumford
4cb80b7ddc Merge pull request #386 from ivanabc:master
PiperOrigin-RevId: 250702492
2019-05-30 09:56:34 -07:00
Victor Costan
72a38ff7f2 Replace "> >" with ">>"
PiperOrigin-RevId: 250383036
2019-05-30 09:55:43 -07:00
Victor Costan
863f185970 unsigned char -> uint8_t
PiperOrigin-RevId: 250309603
2019-05-28 15:44:32 -07:00
ivan
63d5315e1c
Merge branch 'master' into master 2019-05-23 14:02:04 +08:00
Chris Mumford
c00e177f36 Guard DBImpl::versions_ by mutex_.
mutex_ was already acquired before accessing DBImpl::versions_ in all
but one place: DBImpl::GetApproximateSizes. This change requires mutex_
to be held before accessing versions_.

PiperOrigin-RevId: 248390814
2019-05-16 12:07:21 -07:00
Chris Mumford
1d0b101165 Converted two for-loops to while-loops.
Converted `for (;<condition>;)` to `while (<condition>)`.

PiperOrigin-RevId: 247950510
2019-05-13 13:51:11 -07:00
Chris Mumford
28e6d238be Switch to using C++ 11 override specifier.
PiperOrigin-RevId: 247491163
2019-05-09 14:11:06 -07:00
Chris Mumford
85cd40d108 Added unit test for InternalKey::DecodeFrom with empty string.
PiperOrigin-RevId: 247483339
2019-05-09 14:10:55 -07:00
Chris Mumford
1aae5c9f29 Merge pull request #411 from proller:assert1
PiperOrigin-RevId: 247424040
2019-05-09 08:37:49 -07:00
Chris Mumford
b7b86baec9 Using std::ostringstream in key DebugString.
Switching from snprintf to std::ostringstream eliminates
cast warning for (unsigned long long).

PiperOrigin-RevId: 247326681
2019-05-08 17:36:35 -07:00
Victor Costan
4bd052d7e8 Consolidate benchmark code to benchmarks/.
Currently, the benchmark used to assess leveldb changes lives in db/. The codebase also contains two benchmarks against other database engines in doc/bench/. Moving all the benchmarks in one place opens up the way for extracting common code.

PiperOrigin-RevId: 246737541
2019-05-05 12:59:23 -07:00
Victor Costan
506b1722ef Convert missed virtual -> override in db_test.cc.
PiperOrigin-RevId: 246680419
2019-05-04 18:05:53 -07:00
Victor Costan
24424a1ef2 Style cleanup.
1) Convert iterator-based for loops to C++11 foreach loops.
2) Convert "void operator=" to "T& operator=".
3) Switch from copy operators from private to public deleted.
4) Switch from empty ctors / dtors to "= default" where appropriate.

PiperOrigin-RevId: 246679195
2019-05-04 17:42:20 -07:00
Victor Costan
9a56c49ed4 Merge pull request #679 from smartxworks:optimize-readseq
PiperOrigin-RevId: 246668103
2019-05-04 16:53:17 -07:00
Chris Mumford
9bd23c7676 Correct class/structure declaration order.
1. Correct the class/struct declaration order to be IAW
   the Google C++ style guide[1].
2. For non-copyable classes, switched from non-implemented
   private methods to explicitly deleted[2] methods.
3. Minor const and member initialization fixes.

[1] https://google.github.io/styleguide/cppguide.html#Declaration_Order
[2] http://eel.is/c++draft/dcl.fct.def.delete

PiperOrigin-RevId: 246521844
2019-05-03 09:48:57 -07:00
Chris Mumford
297e66afc1 Format all files IAW the Google C++ Style Guide.
Use clang-format to correct formatting to be in agreement with the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). Doing this simplifies the process of accepting changes. Also fixed a few warnings flagged by clang-tidy.

PiperOrigin-RevId: 246350737
2019-05-02 19:04:50 -07:00
Kyle Zhang
d3d1c8a0f4 don't check current key in DBIter::Next()
When iter_ is pointing to current key, we can safely move to the next
key to avoid checking current key, which is of course not necessary.

Benchmark shows that 'readseq' has about 8% performance improvement.

Without patch:

>./db_bench --benchmarks=readseq --num=$((4<<20)) --db=/tmp/db --use_existing_db=1
LevelDB:    version 1.21
Date:       Thu Apr 25 09:37:21 2019
CPU:        32 * Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    4194304
RawSize:    464.0 MB (estimated)
FileSize:   264.0 MB (estimated)
------------------------------------------------
readseq      :       0.196 micros/op;  565.7 MB/s

With patch:

>./db_bench --benchmarks=readseq --num=$((4<<20)) --db=/tmp/db --use_existing_db=1
LevelDB:    version 1.21
Date:       Thu Apr 25 09:38:20 2019
CPU:        32 * Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    4194304
RawSize:    464.0 MB (estimated)
FileSize:   264.0 MB (estimated)
------------------------------------------------
readseq      :       0.181 micros/op;  612.3 MB/s

Signed-off-by: Kyle Zhang <kyle@smartx.com>
2019-04-25 09:54:05 +08:00