148 Commits

Author SHA1 Message Date
costan
201f77d137 Inline defaults in options.
This CL moves default values for
leveldb::{Options,ReadOptions,WriteOptions} from constructors to member
declarations, and removes now-redundant comments stating the defaults.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239271242
2019-03-20 13:56:22 -07:00
costan
7d8e41e49b leveldb: Replace AtomicPointer with std::atomic.
This CL removes AtomicPointer from leveldb's port interface. Its usage is replaced with std::atomic<> from the C++11 standard library.

AtomicPointer was used to wrap flags, numbers, and pointers, so its instances are replaced with std::atomic<bool>, std::atomic<int>, std::atomic<size_t> and std::atomic<Node*>.

This CL does not revise the memory ordering. AtomicPointer's methods are replaced mechanically with their std::atomic equivalents, even when the underlying usage is incorrect. (Example: DBImpl::has_imm_ is written using release stores, even though it is always read using relaxed ordering.) Revising the memory ordering is left for future CLs.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237865146
2019-03-11 13:41:25 -07:00
Dimitris Apostolou
a20508dc6a Fix typo (#565) 2019-03-11 10:36:11 -07:00
cmumford
c69d33b0ec Added native support for Windows.
This change adds a native Windows port (port_windows.h) and a
Windows Env (WindowsEnv).

Note1: "small" is defined when including <Windows.h> so some
parameters were renamed to avoid conflict.

Note2: leveldb::Env defines the method: "DeleteFile" which is
also a constant defined when including <Windows.h>. The solution
was to ensure this macro is defined in env.h which forces
the function, when compiled, to be either DeleteFileA or
DeleteFileW when building for MBCS or UNICODE respectively.

This resolves #519 on GitHub.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=236364778
2019-03-01 18:00:35 -08:00
costan
fe4494804f leveldb: Make WriteBatch::ApproximateSize() const.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229395810
2019-01-15 18:43:13 +00:00
costan
89af27bde5 Remove ssize_t from code that is not POSIX-specific.
ssize_t is not standard C++. It is a POSIX extension. Therefore, it does
not belong in generic code.

This change tweaks the logic in DBIter to remove the need for signed
integers, so ssize_t can be replaced with size_t. The impacted method
and private member are renamed to better express their purpose.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211471606
2018-09-04 10:37:22 -07:00
costan
16a2b8bb3a Expose WriteBatch::Append in the C API.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209345072
2018-08-19 19:54:34 -07:00
costan
f7b0e1d901 Expose WriteBatch::Append().
WriteBatchInternal has a method for efficiently concatenating two
WriteBatches. This commit exposes the method to the public API.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208724311
2018-08-14 15:30:29 -07:00
costan
1868398150 Clean up SnapshotImpl.
* Omit SnapshotImpl::list_ when assert() isn't on
* Make SnapshotImpl::number_ const and set it in the constructor
* Make SnapshotImpl::number_ private and access it via a getter
* Rename SnapshotImpl::number_ to SnapshotImpl::sequence_number_
* Rename SnapshotList::list_ to SnapshotList::head_
* Wrap casting from Snapshot* to SnapshotImpl* in ToSnapshotImpl()

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194852828
2018-04-30 16:01:39 -07:00
MarcoFalke
14cce848e7 Fix sign mismatch warnings in GCC.
This was contributed in https://github.com/google/leveldb/pull/492

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=193080913
2018-04-16 18:13:09 -07:00
costan
09217fd067 Replace NULL with nullptr in C++ files.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=192365747
2018-04-10 16:26:43 -07:00
costan
0db30413a4 leveldb: Add more thread safety annotations.
After this CL, all classes with Mutex members should be covered by annotations. Exceptions are atomic members, which shouldn't need locking, and DBImpl members that cause errors when annotated, which will be tackled separately.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=190260865
2018-03-23 12:56:14 -07:00
costan
739c25100e Add CMake build support.
Fixes https://github.com/google/leveldb/issues/466

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189425354
2018-03-16 19:17:27 -07:00
costan
0fa5a4f7b1 Extend thread safety annotations.
This CL makes it easier to reason about thread safety by:

1) Adding Clang thread safety annotations according to comments.
2) Expanding a couple of variable names, without adding extra lines of code.
3) Adding const in a couple of places.
4) Replacing an always-non-null const pointer with a reference.
5) Fixing style warnings in the modified files.

This CL does not annotate the DBImpl members that claim to be protected
by the instance mutex, but are accessed without the mutex being held.
Those members (and their unprotected accesses) will be addressed in
future CLs.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189354657
2018-03-16 10:32:40 -07:00
costan
8143c12f3f Fix includes in util/testharness.h.
This CL removes unused headers included by util/testharness.h, adds
precise includes where the build breaks, and fixes style errors in the
edited files.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189331061
2018-03-16 10:31:48 -07:00
costan
aece2068d7 Remove extern from function declarations.
External linkage is the default for function declarations in C++.

This also fixes ClangTidy errors generated by removing the "extern"
keyword as described above.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=188730416
2018-03-12 09:24:48 -07:00
costan
ddab751002 Add tests for {Old}InfoLogFileName().
This change was contributed by GitHub user @LopatkinEvgeniy in
https://github.com/google/leveldb/pull/559.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=188728845
2018-03-12 09:24:25 -07:00
costan
7fd7c00721 Remove unused function ExtractValueType.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=188728505
2018-03-12 09:24:07 -07:00
costan
623d014a54 Expose Env::GetTempDirectory() for use in C test.
This removes the use of the non-portable headers <sys/types.h> and <unistd.h> in c_test.c.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=188503102
2018-03-09 10:38:04 -08:00
cmumford
47cb9e2a21 Add leveldb_options_set_max_file_size to the C API.
When the max file size option was added in CL 134391640 the C API
was not modified to support this.

This change was contributed by GitHub user @olt and fixes issue #439.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=173466388
2017-11-03 15:04:26 -07:00
cmumford
3da4d8b989 Deleted unused assignments in Reader.
Deleted two unused assignments:

1. offset_in_block in Reader::SkipToInitialBlock().
2. in_fragmented_record in Reader::ReadRecord().

Reasons for the change:
1. offset_in_block is not read again after the if condition.
2. The kFullRecordType switch branch returns, so
   in_fragmented_record isn't read again.
3. The kFirstType switch branch sets in_fragmented_record to
   true after the if, so the write in the if is ignored.

Change contributed by @C0deAi on GitHub.

This fixes https://github.com/google/leveldb/issues/517

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172763897
2017-11-03 15:03:44 -07:00
cmumford
0509414f85 leveldb::DestroyDB will now delete empty directories.
Env's that filtered out dot files ("." and "..") would return an
empty vector of children causing DestroyDB to do nothing. This fixes
https://github.com/google/leveldb/issues/215

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=172501335
2017-11-03 15:03:20 -07:00
costan
23162ca1c6 Fix typo (forgotten reference operator) in test.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=171708408
2017-10-10 11:47:13 -07:00
sanjay
7e12c00ecf Fix issue 474: a race between the f*_unlocked() STDIO calls in
env_posix.cc and concurrent application calls to fflush(NULL).

The fix is to avoid using stdio in env_posix.cc but add our own
buffering where we need it.

Added a test to reproduce the bug.

Added a test for Env reads/writes.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=170738066
2017-10-03 11:27:09 -07:00
costan
ea0a7586b8 Remove confusing and unnecessary if.
12 lines above, there is an "if (!s.ok()) { return s; }" block of code.
"s" is never modified between that block and the "if" removed by this
CL, so "s.ok()" must be true.

The code most likely intended to say "if (!builder->ok())", because the
builder->Add() call above can modify the TableBuilder's status, as a
side-effect. However, this approach would have required setting "s =
builder.status()" in the "else" branch, near the "builder.Abandon()"
call. So, removing the "if" outright is simpler than following that line
of thought.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=167326229
2017-09-01 14:41:28 -07:00
davidair
02f43c0fcd Remove dead code.
The dead code has been in the codebase since the initial commit and is
generating a compiler warning when used in Xcode.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=164174594
2017-08-24 15:00:52 -07:00
costan
8415f00eee leveldb: Report missing CURRENT manifest file as database corruption.
BTRFS reorders rename and write operations, so it is possible that a filesystem crash and recovery results in a situation where the file pointed to by CURRENT does not exist. DB::Open currently reports an I/O error in this case. Reporting database corruption is a better hint to the caller, which can attempt to recover the database or erase it and start over.

This issue is not merely theoretical. It was reported as having showed up in the wild at https://github.com/google/leveldb/issues/195 and at https://crbug.com/738961. Also, asides from the BTRFS case described above, incorrect data in CURRENT seems like a possible corruption case that should be handled gracefully.

The Env API changes here can be considered backwards compatible, because an implementation that returns Status::IOError instead of Status::NotFound will still get the same functionality as before.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=161432630
2017-07-10 14:14:00 -07:00
costan
69e2bd224b LevelDB: Add WriteBatch::ApproximateSize().
This can be used to report metrics on LevelDB usage.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=156934930
2017-07-10 14:13:30 -07:00
cmumford
7fa20948d5 Convert documentation to markdown.
Markdown is more readable in a text editor and when hosted
on GitHub is more readable than HTML.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=148830423
2017-03-01 09:42:25 -08:00
proller
7d060117fa broken db: fix assertion in leveldb::InternalKey::Encode, mark base as corrupt 2016-10-03 19:40:07 +03:00
corrado
a2fb086d07 Add option for max file size. The currend hard-coded value of 2M is inefficient in colossus.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=134391640
2016-09-28 10:52:24 -07:00
ivanabc
2883fcd849 set const property 2016-06-21 16:57:57 +08:00
ivanabc
e5f0a51fa4 reduce lock's range in DeleteObsoleteFiles 2016-06-20 13:18:07 +08:00
Nicholas Westlake
ea992b467b Change std::uint64_t to uint64_t (#354)
-This fixes compile errors with default setup on RHEL 6 systems.
2016-04-12 15:38:09 -07:00
mjwiacek
e84b5bdb5a This CL fixes a bug encountered when reading records from leveldb files that have been split, as in a [] input task split.
Detailed description:

Suppose an input split is generated between two leveldb record blocks and the preceding block ends with null padding.

A reader that previously read at least 1 record within the first block (before encountering the padding) upon trying to read the next record, will successfully and correctly read the next logical record from the subsequent block, but will return a last record offset pointing to the padding in the first block.

When this happened in a [], it resulted in duplicate records being handled at what appeared to be different offsets that were separated by only a few bytes.

This behavior is only observed when at least 1 record was read from the first block before encountering the padding. If the initial offset for a reader was within the padding, the correct record offset would be reported, namely the offset within the second block.

The tests failed to catch this scenario/bug, because each read test only read a single record with an initial offset. This CL adds an explicit test case for this scenario, and modifies the test structure to read all remaining records in the test case after an initial offset is specified.  Thus an initial offset that jumps to record #3, with 5 total records in the test file, will result in reading 2 records, and validating the offset of each of them in order to pass successfully.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115338487
2016-03-31 15:53:34 -07:00
Bruce Dawson
6b18316d01 Fix signed/unsigned mismatch on VC++ builds 2016-02-19 13:59:19 -08:00
cmumford
adbe3eb073 Putting build artifacts in subdirectory.
1. Object files, libraries, and compiled executables are put
   into subdirectories.
2. The shared library is linked from individual object files.
   This provides for greater parallelism on large desktops
   while at the same time making for easier builds on small
   (i.e. embedded) systems. Fixes issue #279.
3. One program, db_bench, is compiled using the shared library.
4. The source file for "leveldbutil" was renamed from
   leveldb_main.cc to leveldbutil.cc. This provides for simpler
   makefile rules.
5. Because all targets placed the library (libleveldb.a) at the top
   level, the last platform built (desktop/device) always overwrote
   any prior artifact.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=113407013
2016-01-29 16:10:00 -08:00
Chris Mumford
dac40d25f6 Merge pull request #284 from ideawu/master
log compaction output file's level along with number
2016-01-12 11:30:32 -08:00
ssid
706b7f8d43 Resolve race when getting approximate-memory-usage property
The write operations in the table happens without holding the mutex
lock, but concurrent writes are avoided using "writers_" queue.
The Arena::MemoryUsage could access the blocks when write happens.
So, the memory usage is cached in atomic word and can be loaded
from any thread safely.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=107573379
2015-12-09 11:27:50 -08:00
cmumford
3c9ff3c691 Only compiling TrimSpace on linux.
Incorporated change by zmodem at https://github.com/google/leveldb/pull/310
to fix issue #310.

This change will only build TrimSace on linux to avoid unused function
warning/error.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=105323419
2015-12-09 10:35:07 -08:00
ssid
528c2bc6ad Add "approximate-memory-usage" property to leveldb::DB::GetProperty
The approximate RAM usage of the database is calculated from the memory
allocated for write buffers and the block cache. This is to give an
estimate of memory usage to leveldb clients.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=104222307
2015-12-09 10:34:58 -08:00
Mike Wiacek
ce45404bba Suppress error reporting after seeking but before a valid First or Full record is encountered.
Fix a spelling mistake.
2015-12-09 10:34:57 -08:00
Chris Mumford
65190ac48b Will not reuse manifest if reuse_logs options is false.
Prior implementation would always try to reuse the manifest, even if reuse_logs
was false (the default). This was missed because the stock
Env::NewAppendableFile implementation returns false forcing the creation of a
new log.
2015-08-11 14:59:48 -07:00
Sanjay Ghemawat
ac1d69da31 LevelDB now attempts to reuse the preceding MANIFEST and log file when re-opened.
(Based on a suggestion by cmumford.)

"open" benchmark on my workstation speeds up significantly since we
can now avoid three fdatasync calls and a compaction per open:

  Before: ~80000 microseconds
  After:    ~130 microseconds

Details:

(1) Added Options::reuse_logs (currently defaults to false) to control
new behavior.  The intention is to change the default to true after some
baking.

(2) Added Env::NewAppendableFile() whose default implementation returns
a not-supported error.

(3) VersionSet::Recovery attempts to reuse the MANIFEST from which
it is recovering.

(4) DBImpl recovery attempts to reuse the last log file and memtable.

(5) db_test.cc now tests a new configuration that sets reuse_logs to true.

(6) fault_injection_test also tests a reuse_logs==true config.

(7) Added a new recovery_test.
2015-08-11 14:56:39 -07:00
ideawu
76bba139c0 fix indent 2015-04-20 12:41:01 +08:00
ideawu
8fcceb2a6f log compaction output file's level along with number 2015-04-20 12:39:14 +08:00
Sanjay Ghemawat
77948e7eec Add benchmark that measures cost of repeatedly opening the database. 2014-12-11 08:08:57 -08:00
Chris Mumford
34ad72e3e9 Move header guard below copyright banner. 2014-12-11 08:04:40 -08:00
Chris Mumford
a75d435d1d Clean up layering of storage/leveldb/...
With these changes, this package should be properly cleaned up and not require
any further changes.
2014-12-11 08:02:45 -08:00
Chris Mumford
b234f65b34 Added a new fault injection test.
This test is intended to ensure leveldb properly detects and recovers from
faults - specifically unwritten file data lost as a result of a system reset.
2014-12-11 07:59:38 -08:00
Chris Mumford
803d69203a Release 1.18
Changes are:

* Update version number to 1.18
* Replace the basic fprintf call with a call to fwrite in order to
  work around the apparent compiler optimization/rewrite failure that we are
  seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8.
* Fix ALL the header guards.
* Createed a README.md with the LevelDB project description.
* A new CONTRIBUTING file.
* Don't implicitly convert uint64_t to size_t or int.  Either preserve it as
  uint64_t, or explicitly cast. This fixes MSVC warnings about possible value
  truncation when compiling this code in Chromium.
* Added a DumpFile() library function that encapsulates the guts of the
  "leveldbutil dump" command. This will allow clients to dump
  data to their log files instead of stdout. It will also allow clients to
  supply their own environment.
* leveldb: Remove unused function 'ConsumeChar'.
* leveldbutil: Remove unused member variables from WriteBatchItemPrinter.
* OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define
  PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes:
   * issue #143
   * issue #198
   * issue #249
* Switch from <cstdatomic> to <atomic>. The former never made it into the
  standard and doesn't exist in modern gcc versions at all.  The later contains
  everything that leveldb was using from the former.
  This problem was noticed when porting to Portable Native Client where no memory
  barrier is defined.  The fact that <cstdatomic> is missing normally goes
  unnoticed since memory barriers are defined for most architectures.
* Make Hash() treat its input as unsigned.  Before this change LevelDB files
  from platforms with different signedness of char were not compatible. This
  change fixes: issue #243
* Verify checksums of index/meta/filter blocks when paranoid_checks set.
* Invoke all tools for iOS with xcrun. (This was causing problems with the new
  XCode 5.1.1 image on pulse.)
* include <sys/stat.h> only once, and fix the following linter warning:
  "Found C system header after C++ system header"
* When encountering a corrupted table file, return Status::Corruption instead of
  Status::InvalidArgument.
* Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188
* Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159
* Fix typos and comments, and address the following two issues:
  * issue #166
  * issue #241
* Add missing db synchronize after "fillseq" in the benchmark.
* Removed unused variable in SeekRandom: value (issue #201)
2014-09-16 14:19:52 -07:00
Chris Mumford
e353fbc7ea Release LevelDB 1.17
- Cleanup: delete unused IntSetToString

  It was added in http://cr/19491949 (and was referenced at the time).
  The last reference was removed in http://cr/19507363.

  This fixes warning/error with pre-release crosstoolv18:
    'std::string leveldb::{anonymous}::IntSetToString(const std::set<long unsigned int>&)' defined but not used [-Werror=unused-function]

- Added arm64 and and armv7s to IOS build as suggested on leveldb mailing list.

- Changed local variable type from int to size_t

  This eliminates compiler warning/error and resolves
  https://code.google.com/p/leveldb/issues/detail?id=140
2014-05-01 13:44:03 -07:00
David Grogan
269fc6ca94 Release LevelDB 1.16
- Make Log::Reader not report a corruption when the last record in a
  log file is truncated.
- Fix issue 224: variable created but not utilized.
- Remove comment that referenced a removed feature.
2014-02-10 11:36:06 -08:00
David Grogan
0cfb990d58 Release LevelDB 1.15
- switched from mmap based writing to simpler stdio based writing. Has a
  minor impact (0.5 microseconds) on microbenchmarks for asynchronous
  writes. Synchronous writes speed up from 30ms to 10ms on linux/ext4.
  Should be much more reliable on diverse platforms.
- compaction errors now immediately put the database into a read-only
  mode (until it is re-opened). As a downside, a disk going out of
  space and then space being created will require a re-open to recover
  from, whereas previously that would happen automatically. On the
  plus side, many corruption possibilities go away.
- force the DB to enter an error-state so that all future writes fail
  when a synchronous log write succeeds but the sync fails.
- repair now regenerates sstables that exhibit problems
- fix issue 218 - Use native memory barriers on OSX
- fix issue 212 - QNX build is broken
- fix build on iOS with xcode 5
- make tests compile and pass on windows
2013-12-10 10:36:31 -08:00
David Grogan
0b9a89f40e Release LevelDB 1.14
Fix issues 200, 201

Also,
* Fix link to bigtable paper in docs.
* New sstables will have the file extension .ldb. .sst files will
continue to be recognized.
* When building for iOS, use xcrun to execute the compiler. This may
affect issue 177.
2013-09-19 13:49:19 -07:00
David Grogan
748539c183 LevelDB 1.13
Fix issues 77, 87, 182, 190.

Additionally, fix the bug described in
https://groups.google.com/d/msg/leveldb/yL6h1mAOc20/vLU64RylIdMJ
where a large contiguous keyspace of deleted data was not getting
compacted.

Also fix a bug where options.max_open_files was not getting clamped
properly.
2013-08-21 11:12:47 -07:00
David Grogan
7b094f12e4 Release leveldb 1.11
Fixes issues
161
174
178

As well as the issue reported by edouarda14@gmail.com about
MissingSSTFile unit test failing on windows.
2013-06-13 16:14:06 -07:00
David Grogan
28dad918f2 Release leveldb 1.10
Fixes issues
147 - thanks feniksgordonfreeman
153
156
166

Additionally,
* Remove calls to exit(1).
* Fix unused-variable warnings from clang.
* Fix possible overflow error related to num_restart value >= (2^32/4).
* Add leveldbutil to .gitignore.
* Add better log messages when Write is stalled on a compaction.
2013-05-14 17:03:07 -07:00
David Grogan
514c943a8e Make DB::Open fail if sst files are missing.
Also, cleanup for Clang's -Wimplicit-fallthrough warning.
2013-02-06 18:03:32 -08:00
David Grogan
d84c825a70 Fix corruption bug found and analyzed by dhruba@gmail.com
https://groups.google.com/d/msg/leveldb/Kc9JxuIUu5A/9P0N9RL4ar8J
2013-01-07 13:17:43 -08:00
Sanjay Ghemawat
ea2e9195fc added utility to dump leveldb files 2012-12-27 10:38:48 -08:00
Sanjay Ghemawat
40768657bc Small fixes.
Details:
* Fix shared library building.
* Reorganize linking commands so flags like --as-needed can be passed.
* C binding exports version numbers.
* Fix small typos in documention.
2012-10-16 16:17:53 -07:00
David Grogan
946e5b5a4c Update to leveldb 1.6
Highlights
----------
Mmap at most 1000 files on Posix to improve performance for large databases.
Support for more architectures (thanks to Alexander K.)

Building and porting
--------------------
HP/UX support (issue 126)
AtomicPointer for ia64 (issue 123)
Sparc v9 support (issue 124)
Atomic ops for powerpc
Use -fno-builtin-memcmp only when using g++
Simplify IOS build rules (issue 114)
Use CXXFLAGS instead of CFLAGS when invoking C++ compiler (issue 118)
Fix snappy shared library problem (issue 94)
Fix shared library installation path regression
Endian-ness detection tweak for FreeBSD

Bug fixes
---------
Stop ignoring FLAGS_open_files in db_bench
Make bloom test behavior agnostic to endian-ness

Performance
-----------
Limit number of mmapped files to 1000 to improve perf for large dbs
Do not delay for 1 second on shutdown path (issue 125)

Misc
----
Make InMemoryEnv return a no-op logger
C binding now has a wrapper for free (issue 117)
Add thread-safety annotations
Added an in-process lock table (issue 120)
Make RandomAccessFile and SequentialFile non-copyable
2012-10-12 11:53:12 -07:00
Sanjay Ghemawat
075a35a6d3 Remove static initializer; fix endian-ness detection; fix build on
various platforms; improve android port speed.

Avoid static initializer by using a new portability interface for
thread-safe lazy initialization.  Custom ports will need to be
extended to implement InitOnce/OnceType/LEVELDB_ONCE_INIT.

Fix endian-ness detection (fixes Powerpc builds).

Build related fixes:
- Support platforms that have unversioned shared libraries.
- Fix IOS build rules.

Android improvements
- Speed up atomic pointers
- Share more code with port_posix.

Do not spin in a tight loop attempting compactions if the file system
is inaccessible (e.g., if kerberos tickets have expired or if it is out
of space).
2012-05-30 09:45:46 -07:00
Sanjay Ghemawat
85584d497e Added bloom filter support.
In particular, we add a new FilterPolicy class.  An instance
of this class can be supplied in Options when opening a
database.  If supplied, the instance is used to generate
summaries of keys (e.g., a bloom filter) which are placed in
sstables.  These summaries are consulted by DB::Get() so we
can avoid reading sstable blocks that are guaranteed to not
contain the key we are looking for.

This change provides one implementation of FilterPolicy
based on bloom filters.

Other changes:
- Updated version number to 1.4.
- Some build tweaks.
- C binding for CompactRange.
- A few more benchmarks: deleteseq, deleterandom, readmissing, seekrandom.
- Minor .gitignore update.
2012-04-17 08:36:46 -07:00
Sanjay Ghemawat
bc1ee4d25e build shared libraries; updated version to 1.3; add Status accessors 2012-03-30 13:15:49 -07:00
Sanjay Ghemawat
583f1499c0 fix LOCK file deletion to prevent crash on windows 2012-03-09 07:51:04 -08:00
Sanjay Ghemawat
d79762e273 added group commit; drastically speeds up mult-threaded synchronous write workloads 2012-03-08 16:23:21 -08:00
Sanjay Ghemawat
239ac9d2de avoid very large compactions; fix build on Linux 2012-02-02 09:34:14 -08:00
Sanjay Ghemawat
3c8be108bf fixed issues 66 (leaking files on disk error) and 68 (no sync of CURRENT file) 2012-01-25 14:56:52 -08:00
Hans Wennborg
42fb47f6ed Pass system's CFLAGS, remove exit time destructor, sstable bug fix.
- Pass system's values of CFLAGS,LDFLAGS.
  Don't override OPT if it's already set.
  Original patch by Alessio Treglia <alessio@debian.org>:
  http://code.google.com/p/leveldb/issues/detail?id=27#c6

- Remove 1 exit time destructor from leveldb.
  See http://crbug.com/101600

- Fix problem where sstable building code would pass an
  internal key to the user comparator.

(Sync with uptream at 25436817.)
2011-11-14 17:06:16 +00:00
Hans Wennborg
36a5f8ed7f A number of fixes:
- Replace raw slice comparison with a call to user comparator.
  Added test for custom comparators.

- Fix end of namespace comments.

- Fixed bug in picking inputs for a level-0 compaction.

  When finding overlapping files, the covered range may expand
  as files are added to the input set.  We now correctly expand
  the range when this happens instead of continuing to use the
  old range.  For example, suppose L0 contains files with the
  following ranges:

      F1: a .. d
      F2:    c .. g
      F3:       f .. j

  and the initial compaction target is F3.  We used to search
  for range f..j which yielded {F2,F3}.  However we now expand
  the range as soon as another file is added.  In this case,
  when F2 is added, we expand the range to c..j and restart the
  search.  That picks up file F1 as well.

  This change fixes a bug related to deleted keys showing up
  incorrectly after a compaction as described in Issue 44.

(Sync with upstream @25072954)
2011-10-31 17:22:06 +00:00
Gabor Cselle
299ccedfec A number of bugfixes:
- Added DB::CompactRange() method.

  Changed manual compaction code so it breaks up compactions of
  big ranges into smaller compactions.

  Changed the code that pushes the output of memtable compactions
  to higher levels to obey the grandparent constraint: i.e., we
  must never have a single file in level L that overlaps too
  much data in level L+1 (to avoid very expensive L-1 compactions).

  Added code to pretty-print internal keys.

- Fixed bug where we would not detect overlap with files in
  level-0 because we were incorrectly using binary search
  on an array of files with overlapping ranges.

  Added "leveldb.sstables" property that can be used to dump
  all of the sstables and ranges that make up the db state.

- Removing post_write_snapshot support.  Email to leveldb mailing
  list brought up no users, just confusion from one person about
  what it meant.

- Fixing static_cast char to unsigned on BIG_ENDIAN platforms.

  Fixes	Issue 35 and Issue 36.

- Comment clarification to address leveldb Issue 37.

- Change license in posix_logger.h to match other files.

- A build problem where uint32 was used instead of uint32_t.

Sync with upstream @24408625
2011-10-05 16:30:28 -07:00
gabor@google.com
7263023651 Bugfixes: for Get(), don't hold mutex while writing log.
- Fix bug in Get: when it triggers a compaction, it could sometimes
  mark the compaction with the wrong level (if there was a gap
  in the set of levels examined for the Get).

- Do not hold mutex while writing to the log file or to the
  MANIFEST file.

  Added a new benchmark that runs a writer thread concurrently with
  reader threads.

  Percentiles
  ------------------------------
  micros/op: avg  median 99   99.9  99.99  99.999 max
  ------------------------------------------------------
  before:    42   38     110  225   32000  42000  48000
  after:     24   20     55   65    130    1100   7000

- Fixed race in optimized Get.  It should have been using the
  pinned memtables, not the current memtables.



git-svn-id: https://leveldb.googlecode.com/svn/trunk@50 62dab493-f737-651d-591e-8d6aee1b9529
2011-09-01 19:08:02 +00:00
gabor@google.com
e3584f9c28 Bugfix for issue 33; reduce lock contention in Get(), parallel benchmarks.
- Fix for issue 33 (non-null-terminated result from
  leveldb_property_value())

- Support for running multiple instances of a benchmark in parallel.

- Reduce lock contention on Get():
  (1) Do not hold the lock while searching memtables.
  (2) Shard block and table caches 16-ways.

  Benchmark for evaluating this change:
  $ db_bench --benchmarks=fillseq1,readrandom --threads=$n
  (fillseq1 is a small hack to make sure fillseq runs once regardless
  of number of threads specified on the command line).



git-svn-id: https://leveldb.googlecode.com/svn/trunk@49 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-22 21:08:51 +00:00
gabor@google.com
ab323f7e1e Bugfixes for iterator and documentation.
- Fix bug in Iterator::Prev where it would return the wrong key.
  Fixes issues 29 and 30.

- Added a tweak to testharness to allow running just some tests.

- Fixing two minor documentation errors based on issues 28 and 25.

- Cleanup; fix namespaces of export-to-C code.
  Also fix one "const char*" vs "char*" mismatch.



git-svn-id: https://leveldb.googlecode.com/svn/trunk@48 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-16 01:21:01 +00:00
dgrogan@chromium.org
a05525d13b @23023120
git-svn-id: https://leveldb.googlecode.com/svn/trunk@47 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-06 00:19:37 +00:00
gabor@google.com
021ee9c32b C binding for leveldb, better readseq benchmark for SQLite.
- Added a C binding for LevelDB.
  May be useful as a stable ABI that can be used by 
  programs that keep leveldb in a shared library, 
  or for JNI API.

- Replaced SQLite's readseq benchmark to a more efficient version. 
  SQLite readseq speeds increased by about a factor of 2x 
  from the previous version. Also updated benchmark page to
  reflect readseq speed up.



git-svn-id: https://leveldb.googlecode.com/svn/trunk@46 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-05 20:40:49 +00:00
gabor@google.com
60bd8015f2 Speed up Snappy uncompression, new Logger interface.
- Removed one copy of an uncompressed block contents changing
  the signature of Snappy_Uncompress() so it uncompresses into a
  flat array instead of a std::string.
        
  Speeds up readrandom ~10%.

- Instead of a combination of Env/WritableFile, we now have a
  Logger interface that can be easily overridden applications
  that want to supply their own logging.

- Separated out the gcc and Sun Studio parts of atomic_pointer.h
  so we can use 'asm', 'volatile' keywords for Sun Studio.




git-svn-id: https://leveldb.googlecode.com/svn/trunk@39 62dab493-f737-651d-591e-8d6aee1b9529
2011-07-21 02:40:18 +00:00
gabor@google.com
6872ace901 Sun Studio support, and fix for test related memory fixes.
- LevelDB patch for Sun Studio
  Based on a patch submitted by Theo Schlossnagle - thanks!
  This fixes Issue 17.

- Fix a couple of test related memory leaks.



git-svn-id: https://leveldb.googlecode.com/svn/trunk@38 62dab493-f737-651d-591e-8d6aee1b9529
2011-07-19 23:36:47 +00:00
gabor@google.com
6699c7ebe6 Small tweaks and bugfixes for Issue 18 and 19.
Slight tweak to the no-overlap optimization: only push to
level 2 to reduce the amount of wasted space when the same
small key range is being repeatedly overwritten.

Fix for Issue 18: Avoid failure on Windows by avoiding
deletion of lock file until the end of DestroyDB().

Fix for Issue 19: Disregard sequence numbers when checking for 
overlap in sstable ranges. This fixes issue 19: when writing 
the same key over and over again, we would generate a sequence 
of sstables that were never merged together since their sequence
numbers were disjoint.

Don't ignore map/unmap error checks.

Miscellaneous fixes for small problems Sanjay found while diagnosing
issue/9 and issue/16 (corruption_testr failures).
- log::Reader reports the record type when it finds an unexpected type.
- log::Reader no longer reports an error when it encounters an expected
  zero record regardless of the setting of the "checksum" flag.
- Added a missing forward declaration.
- Documented a side-effects of larger write buffer sizes
  (longer recovery time).



git-svn-id: https://leveldb.googlecode.com/svn/trunk@37 62dab493-f737-651d-591e-8d6aee1b9529
2011-07-15 00:20:57 +00:00
gabor@google.com
e0cbd242cb Fixing issue 11: version_set_test.cc was missing
git-svn-id: https://leveldb.googlecode.com/svn/trunk@33 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-22 18:45:39 +00:00
gabor@google.com
ccf0fcd5c2 A number of smaller fixes and performance improvements:
- Implemented Get() directly instead of building on top of a full
  merging iterator stack.  This speeds up the "readrandom" benchmark
  by up to 15-30%.

- Fixed an opensource compilation problem.
  Added --db=<name> flag to control where the database is placed.

- Automatically compact a file when we have done enough
  overlapping seeks to that file.

- Fixed a performance bug where we would read from at least one
  file in a level even if none of the files overlapped the key
  being read.

- Makefile fix for Mac OSX installations that have XCode 4 without XCode 3.

- Unified the two occurrences of binary search in a file-list
  into one routine.

- Found and fixed a bug where we would unnecessarily search the
  last file when looking for a key larger than all data in the
  level.

- A fix to avoid the need for trivial move compactions and
  therefore gets rid of two out of five syncs in "fillseq".

- Removed the MANIFEST file write when switching to a new
  memtable/log-file for a 10-20% improvement on fill speed on ext4.

- Adding a SNAPPY setting in the Makefile for folks who have
  Snappy installed. Snappy compresses values and speeds up writes.



git-svn-id: https://leveldb.googlecode.com/svn/trunk@32 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-22 02:36:45 +00:00
hans@chromium.org
80e5b0d944 sync with upstream @21706995
Fixed race condition reported by Dave Smit (dizzyd@dizzyd,com)
on the leveldb mailing list.  We were not signalling
waiters after a trivial move from level-0.  The result was
that in some cases (hard to reproduce), a write would get
stuck forever waiting for the number of level-0 files to drop
below its hard limit.

The new code is simpler: there is just one condition variable
instead of two, and the condition variable is signalled after
every piece of background work finishes.  Also, all compaction
work (including for manual compactions) is done in the
background thread, and therefore we can remove the
"compacting_" variable.



git-svn-id: https://leveldb.googlecode.com/svn/trunk@31 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-07 14:40:26 +00:00
dgrogan@chromium.org
740d8b3d00 Update from upstream @21551990
* Patch LevelDB to build for OSX and iOS
* Fix race condition in memtable iterator deletion.
* Other small fixes.

git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 00:53:58 +00:00
dgrogan@chromium.org
da79909507 sync with upstream @ 21409451
Check the NEWS file for details of what changed.

git-svn-id: https://leveldb.googlecode.com/svn/trunk@28 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-21 02:17:43 +00:00
dgrogan@chromium.org
ba6dac0e80 @20776309
* env_chromium.cc should not export symbols.
* Fix MSVC warnings.
* Removed large value support.
* Fix broken reference to documentation file

git-svn-id: https://leveldb.googlecode.com/svn/trunk@24 62dab493-f737-651d-591e-8d6aee1b9529
2011-04-20 22:48:11 +00:00
dgrogan@chromium.org
69c6d38342 reverting disastrous MOE commit, returning to r21
git-svn-id: https://leveldb.googlecode.com/svn/trunk@23 62dab493-f737-651d-591e-8d6aee1b9529
2011-04-19 23:11:15 +00:00
dgrogan@chromium.org
b743906eea Revision created by MOE tool push_codebase.
MOE_MIGRATION=


git-svn-id: https://leveldb.googlecode.com/svn/trunk@22 62dab493-f737-651d-591e-8d6aee1b9529
2011-04-19 23:01:25 +00:00
dgrogan@chromium.org
b409afe968 chmod a-x
git-svn-id: https://leveldb.googlecode.com/svn/trunk@21 62dab493-f737-651d-591e-8d6aee1b9529
2011-04-18 23:15:58 +00:00
dgrogan@chromium.org
f779e7a5d8 @20602303. Default file permission is now 755.
git-svn-id: https://leveldb.googlecode.com/svn/trunk@20 62dab493-f737-651d-591e-8d6aee1b9529
2011-04-12 19:38:58 +00:00
jorlow@chromium.org
4671a695fc Move include files into a leveldb subdir.
git-svn-id: https://leveldb.googlecode.com/svn/trunk@18 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-30 18:35:40 +00:00
jorlow@chromium.org
e2da744e12 Upstream changes.
git-svn-id: https://leveldb.googlecode.com/svn/trunk@16 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-28 20:43:44 +00:00
jorlow@chromium.org
e11bdf1935 Upstream changes
git-svn-id: https://leveldb.googlecode.com/svn/trunk@15 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-25 20:27:43 +00:00
jorlow@chromium.org
8303bb1b33 Pull from upstream.
git-svn-id: https://leveldb.googlecode.com/svn/trunk@14 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-22 23:24:02 +00:00
jorlow@chromium.org
13b72af77b More changes from upstream.
git-svn-id: https://leveldb.googlecode.com/svn/trunk@12 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-22 18:32:49 +00:00
jorlow@chromium.org
0e38925490 Sync in bug fixes
git-svn-id: https://leveldb.googlecode.com/svn/trunk@9 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-21 19:40:57 +00:00
jorlow@chromium.org
f67e15e50f Initial checkin.
git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 22:37:00 +00:00