Browse Source
a31c8aa Add NewAppendableFile for win32 environment 1913d71 Merge upstream LevelDB 1.19 3080a45 Increase leveldb version to 1.19. fa6dc01 A zippy change broke test assumptions about the size of compressed output. Fix the tests by allowing more slop in zippy's behavior. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=123432472 06a191b fix problems in LevelDB's caching code a7bff69 Fix LevelDB build when asserts are enabled in release builds. (#367) ea992b4 Change std::uint64_t to uint64_t (#354) e84b5bd This CL fixes a bug encountered when reading records from leveldb files that have been split, as in a [] input task split. 3211343 Deleted redundant null ptr check prior to delete. 7306ef8 Merge pull request #348 from randomascii/master 6b18316 Fix signed/unsigned mismatch on VC++ builds adbe3eb Putting build artifacts in subdirectory. 2d0320a Merge pull request #329 from ralphtheninja/travis-badge dd1c3c3 add travis build badge 43fcf23 Merge pull request #328 from cmumford/master 9fcae61 Added a Travis CI build file. dac40d2 Merge pull request #284 from ideawu/master 8ec241a Merge pull request #317 from falvojr/patch-1 5d36bed Merge pull request #272 from vapier/master 4753c9b Added a contributors section to README.md e2446d0 Merge pull request #275 from paulirish/patch-1 706b7f8 Resolve race when getting approximate-memory-usage property 3c9ff3c Only compiling TrimSpace on linux. f8d205c Including atomic_pointer.h in port_posix 889de31 Let LevelDB use xcrun to determine Xcode.app path instead of using a hardcoded path. 528c2bc Add "approximate-memory-usage" property to leveldb::DB::GetProperty 359b6bc Add leveldb::Cache::Prune 50e77a8 Fix size_t/int comparison/conversion issues in leveldb. 5208e79 Added leveldb::Status::IsInvalidArgument() method. ce45404 Suppress error reporting after seeking but before a valid First or Full record is encountered. b9afa1f include <assert> -> <cassert> edf2939 Update README.md 65190ac Will not reuse manifest if reuse_logs options is false. ac1d69d LevelDB now attempts to reuse the preceding MANIFEST and log file when re-opened. 76bba13 fix indent 8fcceb2 log compaction output file's level along with number 0e0f074 documentation. improved link c85addc readme: improved documentation link ceff6f1 Fix Android/MIPS build. 77948e7 Add benchmark that measures cost of repeatedly opening the database. 34ad72e Move header guard below copyright banner. a75d435 Clean up layering of storage/leveldb/... b234f65 Added a new fault injection test. c4c38f9 Add arm64 support to leveldb. cea9b10 Fixed incorrect comment wording for Iterator::Seek. c00c569 Deleted old README file. git-subtree-dir: src/leveldb git-subtree-split: a31c8aa408d5594830f7cb20ead1ef1dff51b79e0.15
Pieter Wuille
8 years ago
52 changed files with 1942 additions and 372 deletions
@ -0,0 +1,13 @@ |
|||||||
|
language: cpp |
||||||
|
compiler: |
||||||
|
- clang |
||||||
|
- gcc |
||||||
|
os: |
||||||
|
- linux |
||||||
|
- osx |
||||||
|
sudo: false |
||||||
|
before_install: |
||||||
|
- echo $LANG |
||||||
|
- echo $LC_ALL |
||||||
|
script: |
||||||
|
- make -j 4 check |
||||||
@ -1,51 +0,0 @@ |
|||||||
leveldb: A key-value store |
|
||||||
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com) |
|
||||||
|
|
||||||
The code under this directory implements a system for maintaining a |
|
||||||
persistent key/value store. |
|
||||||
|
|
||||||
See doc/index.html for more explanation. |
|
||||||
See doc/impl.html for a brief overview of the implementation. |
|
||||||
|
|
||||||
The public interface is in include/*.h. Callers should not include or |
|
||||||
rely on the details of any other header files in this package. Those |
|
||||||
internal APIs may be changed without warning. |
|
||||||
|
|
||||||
Guide to header files: |
|
||||||
|
|
||||||
include/db.h |
|
||||||
Main interface to the DB: Start here |
|
||||||
|
|
||||||
include/options.h |
|
||||||
Control over the behavior of an entire database, and also |
|
||||||
control over the behavior of individual reads and writes. |
|
||||||
|
|
||||||
include/comparator.h |
|
||||||
Abstraction for user-specified comparison function. If you want |
|
||||||
just bytewise comparison of keys, you can use the default comparator, |
|
||||||
but clients can write their own comparator implementations if they |
|
||||||
want custom ordering (e.g. to handle different character |
|
||||||
encodings, etc.) |
|
||||||
|
|
||||||
include/iterator.h |
|
||||||
Interface for iterating over data. You can get an iterator |
|
||||||
from a DB object. |
|
||||||
|
|
||||||
include/write_batch.h |
|
||||||
Interface for atomically applying multiple updates to a database. |
|
||||||
|
|
||||||
include/slice.h |
|
||||||
A simple module for maintaining a pointer and a length into some |
|
||||||
other byte array. |
|
||||||
|
|
||||||
include/status.h |
|
||||||
Status is returned from many of the public interfaces and is used |
|
||||||
to report success and various kinds of errors. |
|
||||||
|
|
||||||
include/env.h |
|
||||||
Abstraction of the OS environment. A posix implementation of |
|
||||||
this interface is in util/env_posix.cc |
|
||||||
|
|
||||||
include/table.h |
|
||||||
include/table_builder.h |
|
||||||
Lower-level modules that most clients probably won't use directly |
|
||||||
@ -0,0 +1,554 @@ |
|||||||
|
// Copyright 2014 The LevelDB Authors. All rights reserved.
|
||||||
|
// Use of this source code is governed by a BSD-style license that can be
|
||||||
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
||||||
|
|
||||||
|
// This test uses a custom Env to keep track of the state of a filesystem as of
|
||||||
|
// the last "sync". It then checks for data loss errors by purposely dropping
|
||||||
|
// file data (or entire files) not protected by a "sync".
|
||||||
|
|
||||||
|
#include "leveldb/db.h" |
||||||
|
|
||||||
|
#include <map> |
||||||
|
#include <set> |
||||||
|
#include "db/db_impl.h" |
||||||
|
#include "db/filename.h" |
||||||
|
#include "db/log_format.h" |
||||||
|
#include "db/version_set.h" |
||||||
|
#include "leveldb/cache.h" |
||||||
|
#include "leveldb/env.h" |
||||||
|
#include "leveldb/table.h" |
||||||
|
#include "leveldb/write_batch.h" |
||||||
|
#include "util/logging.h" |
||||||
|
#include "util/mutexlock.h" |
||||||
|
#include "util/testharness.h" |
||||||
|
#include "util/testutil.h" |
||||||
|
|
||||||
|
namespace leveldb { |
||||||
|
|
||||||
|
static const int kValueSize = 1000; |
||||||
|
static const int kMaxNumValues = 2000; |
||||||
|
static const size_t kNumIterations = 3; |
||||||
|
|
||||||
|
class FaultInjectionTestEnv; |
||||||
|
|
||||||
|
namespace { |
||||||
|
|
||||||
|
// Assume a filename, and not a directory name like "/foo/bar/"
|
||||||
|
static std::string GetDirName(const std::string filename) { |
||||||
|
size_t found = filename.find_last_of("/\\"); |
||||||
|
if (found == std::string::npos) { |
||||||
|
return ""; |
||||||
|
} else { |
||||||
|
return filename.substr(0, found); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
Status SyncDir(const std::string& dir) { |
||||||
|
// As this is a test it isn't required to *actually* sync this directory.
|
||||||
|
return Status::OK(); |
||||||
|
} |
||||||
|
|
||||||
|
// A basic file truncation function suitable for this test.
|
||||||
|
Status Truncate(const std::string& filename, uint64_t length) { |
||||||
|
leveldb::Env* env = leveldb::Env::Default(); |
||||||
|
|
||||||
|
SequentialFile* orig_file; |
||||||
|
Status s = env->NewSequentialFile(filename, &orig_file); |
||||||
|
if (!s.ok()) |
||||||
|
return s; |
||||||
|
|
||||||
|
char* scratch = new char[length]; |
||||||
|
leveldb::Slice result; |
||||||
|
s = orig_file->Read(length, &result, scratch); |
||||||
|
delete orig_file; |
||||||
|
if (s.ok()) { |
||||||
|
std::string tmp_name = GetDirName(filename) + "/truncate.tmp"; |
||||||
|
WritableFile* tmp_file; |
||||||
|
s = env->NewWritableFile(tmp_name, &tmp_file); |
||||||
|
if (s.ok()) { |
||||||
|
s = tmp_file->Append(result); |
||||||
|
delete tmp_file; |
||||||
|
if (s.ok()) { |
||||||
|
s = env->RenameFile(tmp_name, filename); |
||||||
|
} else { |
||||||
|
env->DeleteFile(tmp_name); |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
delete[] scratch; |
||||||
|
|
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
struct FileState { |
||||||
|
std::string filename_; |
||||||
|
ssize_t pos_; |
||||||
|
ssize_t pos_at_last_sync_; |
||||||
|
ssize_t pos_at_last_flush_; |
||||||
|
|
||||||
|
FileState(const std::string& filename) |
||||||
|
: filename_(filename), |
||||||
|
pos_(-1), |
||||||
|
pos_at_last_sync_(-1), |
||||||
|
pos_at_last_flush_(-1) { } |
||||||
|
|
||||||
|
FileState() : pos_(-1), pos_at_last_sync_(-1), pos_at_last_flush_(-1) {} |
||||||
|
|
||||||
|
bool IsFullySynced() const { return pos_ <= 0 || pos_ == pos_at_last_sync_; } |
||||||
|
|
||||||
|
Status DropUnsyncedData() const; |
||||||
|
}; |
||||||
|
|
||||||
|
} // anonymous namespace
|
||||||
|
|
||||||
|
// A wrapper around WritableFile which informs another Env whenever this file
|
||||||
|
// is written to or sync'ed.
|
||||||
|
class TestWritableFile : public WritableFile { |
||||||
|
public: |
||||||
|
TestWritableFile(const FileState& state, |
||||||
|
WritableFile* f, |
||||||
|
FaultInjectionTestEnv* env); |
||||||
|
virtual ~TestWritableFile(); |
||||||
|
virtual Status Append(const Slice& data); |
||||||
|
virtual Status Close(); |
||||||
|
virtual Status Flush(); |
||||||
|
virtual Status Sync(); |
||||||
|
|
||||||
|
private: |
||||||
|
FileState state_; |
||||||
|
WritableFile* target_; |
||||||
|
bool writable_file_opened_; |
||||||
|
FaultInjectionTestEnv* env_; |
||||||
|
|
||||||
|
Status SyncParent(); |
||||||
|
}; |
||||||
|
|
||||||
|
class FaultInjectionTestEnv : public EnvWrapper { |
||||||
|
public: |
||||||
|
FaultInjectionTestEnv() : EnvWrapper(Env::Default()), filesystem_active_(true) {} |
||||||
|
virtual ~FaultInjectionTestEnv() { } |
||||||
|
virtual Status NewWritableFile(const std::string& fname, |
||||||
|
WritableFile** result); |
||||||
|
virtual Status NewAppendableFile(const std::string& fname, |
||||||
|
WritableFile** result); |
||||||
|
virtual Status DeleteFile(const std::string& f); |
||||||
|
virtual Status RenameFile(const std::string& s, const std::string& t); |
||||||
|
|
||||||
|
void WritableFileClosed(const FileState& state); |
||||||
|
Status DropUnsyncedFileData(); |
||||||
|
Status DeleteFilesCreatedAfterLastDirSync(); |
||||||
|
void DirWasSynced(); |
||||||
|
bool IsFileCreatedSinceLastDirSync(const std::string& filename); |
||||||
|
void ResetState(); |
||||||
|
void UntrackFile(const std::string& f); |
||||||
|
// Setting the filesystem to inactive is the test equivalent to simulating a
|
||||||
|
// system reset. Setting to inactive will freeze our saved filesystem state so
|
||||||
|
// that it will stop being recorded. It can then be reset back to the state at
|
||||||
|
// the time of the reset.
|
||||||
|
bool IsFilesystemActive() const { return filesystem_active_; } |
||||||
|
void SetFilesystemActive(bool active) { filesystem_active_ = active; } |
||||||
|
|
||||||
|
private: |
||||||
|
port::Mutex mutex_; |
||||||
|
std::map<std::string, FileState> db_file_state_; |
||||||
|
std::set<std::string> new_files_since_last_dir_sync_; |
||||||
|
bool filesystem_active_; // Record flushes, syncs, writes
|
||||||
|
}; |
||||||
|
|
||||||
|
TestWritableFile::TestWritableFile(const FileState& state, |
||||||
|
WritableFile* f, |
||||||
|
FaultInjectionTestEnv* env) |
||||||
|
: state_(state), |
||||||
|
target_(f), |
||||||
|
writable_file_opened_(true), |
||||||
|
env_(env) { |
||||||
|
assert(f != NULL); |
||||||
|
} |
||||||
|
|
||||||
|
TestWritableFile::~TestWritableFile() { |
||||||
|
if (writable_file_opened_) { |
||||||
|
Close(); |
||||||
|
} |
||||||
|
delete target_; |
||||||
|
} |
||||||
|
|
||||||
|
Status TestWritableFile::Append(const Slice& data) { |
||||||
|
Status s = target_->Append(data); |
||||||
|
if (s.ok() && env_->IsFilesystemActive()) { |
||||||
|
state_.pos_ += data.size(); |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
Status TestWritableFile::Close() { |
||||||
|
writable_file_opened_ = false; |
||||||
|
Status s = target_->Close(); |
||||||
|
if (s.ok()) { |
||||||
|
env_->WritableFileClosed(state_); |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
Status TestWritableFile::Flush() { |
||||||
|
Status s = target_->Flush(); |
||||||
|
if (s.ok() && env_->IsFilesystemActive()) { |
||||||
|
state_.pos_at_last_flush_ = state_.pos_; |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
Status TestWritableFile::SyncParent() { |
||||||
|
Status s = SyncDir(GetDirName(state_.filename_)); |
||||||
|
if (s.ok()) { |
||||||
|
env_->DirWasSynced(); |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
Status TestWritableFile::Sync() { |
||||||
|
if (!env_->IsFilesystemActive()) { |
||||||
|
return Status::OK(); |
||||||
|
} |
||||||
|
// Ensure new files referred to by the manifest are in the filesystem.
|
||||||
|
Status s = target_->Sync(); |
||||||
|
if (s.ok()) { |
||||||
|
state_.pos_at_last_sync_ = state_.pos_; |
||||||
|
} |
||||||
|
if (env_->IsFileCreatedSinceLastDirSync(state_.filename_)) { |
||||||
|
Status ps = SyncParent(); |
||||||
|
if (s.ok() && !ps.ok()) { |
||||||
|
s = ps; |
||||||
|
} |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
Status FaultInjectionTestEnv::NewWritableFile(const std::string& fname, |
||||||
|
WritableFile** result) { |
||||||
|
WritableFile* actual_writable_file; |
||||||
|
Status s = target()->NewWritableFile(fname, &actual_writable_file); |
||||||
|
if (s.ok()) { |
||||||
|
FileState state(fname); |
||||||
|
state.pos_ = 0; |
||||||
|
*result = new TestWritableFile(state, actual_writable_file, this); |
||||||
|
// NewWritableFile doesn't append to files, so if the same file is
|
||||||
|
// opened again then it will be truncated - so forget our saved
|
||||||
|
// state.
|
||||||
|
UntrackFile(fname); |
||||||
|
MutexLock l(&mutex_); |
||||||
|
new_files_since_last_dir_sync_.insert(fname); |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
Status FaultInjectionTestEnv::NewAppendableFile(const std::string& fname, |
||||||
|
WritableFile** result) { |
||||||
|
WritableFile* actual_writable_file; |
||||||
|
Status s = target()->NewAppendableFile(fname, &actual_writable_file); |
||||||
|
if (s.ok()) { |
||||||
|
FileState state(fname); |
||||||
|
state.pos_ = 0; |
||||||
|
{ |
||||||
|
MutexLock l(&mutex_); |
||||||
|
if (db_file_state_.count(fname) == 0) { |
||||||
|
new_files_since_last_dir_sync_.insert(fname); |
||||||
|
} else { |
||||||
|
state = db_file_state_[fname]; |
||||||
|
} |
||||||
|
} |
||||||
|
*result = new TestWritableFile(state, actual_writable_file, this); |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
Status FaultInjectionTestEnv::DropUnsyncedFileData() { |
||||||
|
Status s; |
||||||
|
MutexLock l(&mutex_); |
||||||
|
for (std::map<std::string, FileState>::const_iterator it = |
||||||
|
db_file_state_.begin(); |
||||||
|
s.ok() && it != db_file_state_.end(); ++it) { |
||||||
|
const FileState& state = it->second; |
||||||
|
if (!state.IsFullySynced()) { |
||||||
|
s = state.DropUnsyncedData(); |
||||||
|
} |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
void FaultInjectionTestEnv::DirWasSynced() { |
||||||
|
MutexLock l(&mutex_); |
||||||
|
new_files_since_last_dir_sync_.clear(); |
||||||
|
} |
||||||
|
|
||||||
|
bool FaultInjectionTestEnv::IsFileCreatedSinceLastDirSync( |
||||||
|
const std::string& filename) { |
||||||
|
MutexLock l(&mutex_); |
||||||
|
return new_files_since_last_dir_sync_.find(filename) != |
||||||
|
new_files_since_last_dir_sync_.end(); |
||||||
|
} |
||||||
|
|
||||||
|
void FaultInjectionTestEnv::UntrackFile(const std::string& f) { |
||||||
|
MutexLock l(&mutex_); |
||||||
|
db_file_state_.erase(f); |
||||||
|
new_files_since_last_dir_sync_.erase(f); |
||||||
|
} |
||||||
|
|
||||||
|
Status FaultInjectionTestEnv::DeleteFile(const std::string& f) { |
||||||
|
Status s = EnvWrapper::DeleteFile(f); |
||||||
|
ASSERT_OK(s); |
||||||
|
if (s.ok()) { |
||||||
|
UntrackFile(f); |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
Status FaultInjectionTestEnv::RenameFile(const std::string& s, |
||||||
|
const std::string& t) { |
||||||
|
Status ret = EnvWrapper::RenameFile(s, t); |
||||||
|
|
||||||
|
if (ret.ok()) { |
||||||
|
MutexLock l(&mutex_); |
||||||
|
if (db_file_state_.find(s) != db_file_state_.end()) { |
||||||
|
db_file_state_[t] = db_file_state_[s]; |
||||||
|
db_file_state_.erase(s); |
||||||
|
} |
||||||
|
|
||||||
|
if (new_files_since_last_dir_sync_.erase(s) != 0) { |
||||||
|
assert(new_files_since_last_dir_sync_.find(t) == |
||||||
|
new_files_since_last_dir_sync_.end()); |
||||||
|
new_files_since_last_dir_sync_.insert(t); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return ret; |
||||||
|
} |
||||||
|
|
||||||
|
void FaultInjectionTestEnv::ResetState() { |
||||||
|
// Since we are not destroying the database, the existing files
|
||||||
|
// should keep their recorded synced/flushed state. Therefore
|
||||||
|
// we do not reset db_file_state_ and new_files_since_last_dir_sync_.
|
||||||
|
MutexLock l(&mutex_); |
||||||
|
SetFilesystemActive(true); |
||||||
|
} |
||||||
|
|
||||||
|
Status FaultInjectionTestEnv::DeleteFilesCreatedAfterLastDirSync() { |
||||||
|
// Because DeleteFile access this container make a copy to avoid deadlock
|
||||||
|
mutex_.Lock(); |
||||||
|
std::set<std::string> new_files(new_files_since_last_dir_sync_.begin(), |
||||||
|
new_files_since_last_dir_sync_.end()); |
||||||
|
mutex_.Unlock(); |
||||||
|
Status s; |
||||||
|
std::set<std::string>::const_iterator it; |
||||||
|
for (it = new_files.begin(); s.ok() && it != new_files.end(); ++it) { |
||||||
|
s = DeleteFile(*it); |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
void FaultInjectionTestEnv::WritableFileClosed(const FileState& state) { |
||||||
|
MutexLock l(&mutex_); |
||||||
|
db_file_state_[state.filename_] = state; |
||||||
|
} |
||||||
|
|
||||||
|
Status FileState::DropUnsyncedData() const { |
||||||
|
ssize_t sync_pos = pos_at_last_sync_ == -1 ? 0 : pos_at_last_sync_; |
||||||
|
return Truncate(filename_, sync_pos); |
||||||
|
} |
||||||
|
|
||||||
|
class FaultInjectionTest { |
||||||
|
public: |
||||||
|
enum ExpectedVerifResult { VAL_EXPECT_NO_ERROR, VAL_EXPECT_ERROR }; |
||||||
|
enum ResetMethod { RESET_DROP_UNSYNCED_DATA, RESET_DELETE_UNSYNCED_FILES }; |
||||||
|
|
||||||
|
FaultInjectionTestEnv* env_; |
||||||
|
std::string dbname_; |
||||||
|
Cache* tiny_cache_; |
||||||
|
Options options_; |
||||||
|
DB* db_; |
||||||
|
|
||||||
|
FaultInjectionTest() |
||||||
|
: env_(new FaultInjectionTestEnv), |
||||||
|
tiny_cache_(NewLRUCache(100)), |
||||||
|
db_(NULL) { |
||||||
|
dbname_ = test::TmpDir() + "/fault_test"; |
||||||
|
DestroyDB(dbname_, Options()); // Destroy any db from earlier run
|
||||||
|
options_.reuse_logs = true; |
||||||
|
options_.env = env_; |
||||||
|
options_.paranoid_checks = true; |
||||||
|
options_.block_cache = tiny_cache_; |
||||||
|
options_.create_if_missing = true; |
||||||
|
} |
||||||
|
|
||||||
|
~FaultInjectionTest() { |
||||||
|
CloseDB(); |
||||||
|
DestroyDB(dbname_, Options()); |
||||||
|
delete tiny_cache_; |
||||||
|
delete env_; |
||||||
|
} |
||||||
|
|
||||||
|
void ReuseLogs(bool reuse) { |
||||||
|
options_.reuse_logs = reuse; |
||||||
|
} |
||||||
|
|
||||||
|
void Build(int start_idx, int num_vals) { |
||||||
|
std::string key_space, value_space; |
||||||
|
WriteBatch batch; |
||||||
|
for (int i = start_idx; i < start_idx + num_vals; i++) { |
||||||
|
Slice key = Key(i, &key_space); |
||||||
|
batch.Clear(); |
||||||
|
batch.Put(key, Value(i, &value_space)); |
||||||
|
WriteOptions options; |
||||||
|
ASSERT_OK(db_->Write(options, &batch)); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
Status ReadValue(int i, std::string* val) const { |
||||||
|
std::string key_space, value_space; |
||||||
|
Slice key = Key(i, &key_space); |
||||||
|
Value(i, &value_space); |
||||||
|
ReadOptions options; |
||||||
|
return db_->Get(options, key, val); |
||||||
|
} |
||||||
|
|
||||||
|
Status Verify(int start_idx, int num_vals, |
||||||
|
ExpectedVerifResult expected) const { |
||||||
|
std::string val; |
||||||
|
std::string value_space; |
||||||
|
Status s; |
||||||
|
for (int i = start_idx; i < start_idx + num_vals && s.ok(); i++) { |
||||||
|
Value(i, &value_space); |
||||||
|
s = ReadValue(i, &val); |
||||||
|
if (expected == VAL_EXPECT_NO_ERROR) { |
||||||
|
if (s.ok()) { |
||||||
|
ASSERT_EQ(value_space, val); |
||||||
|
} |
||||||
|
} else if (s.ok()) { |
||||||
|
fprintf(stderr, "Expected an error at %d, but was OK\n", i); |
||||||
|
s = Status::IOError(dbname_, "Expected value error:"); |
||||||
|
} else { |
||||||
|
s = Status::OK(); // An expected error
|
||||||
|
} |
||||||
|
} |
||||||
|
return s; |
||||||
|
} |
||||||
|
|
||||||
|
// Return the ith key
|
||||||
|
Slice Key(int i, std::string* storage) const { |
||||||
|
char buf[100]; |
||||||
|
snprintf(buf, sizeof(buf), "%016d", i); |
||||||
|
storage->assign(buf, strlen(buf)); |
||||||
|
return Slice(*storage); |
||||||
|
} |
||||||
|
|
||||||
|
// Return the value to associate with the specified key
|
||||||
|
Slice Value(int k, std::string* storage) const { |
||||||
|
Random r(k); |
||||||
|
return test::RandomString(&r, kValueSize, storage); |
||||||
|
} |
||||||
|
|
||||||
|
Status OpenDB() { |
||||||
|
delete db_; |
||||||
|
db_ = NULL; |
||||||
|
env_->ResetState(); |
||||||
|
return DB::Open(options_, dbname_, &db_); |
||||||
|
} |
||||||
|
|
||||||
|
void CloseDB() { |
||||||
|
delete db_; |
||||||
|
db_ = NULL; |
||||||
|
} |
||||||
|
|
||||||
|
void DeleteAllData() { |
||||||
|
Iterator* iter = db_->NewIterator(ReadOptions()); |
||||||
|
WriteOptions options; |
||||||
|
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) { |
||||||
|
ASSERT_OK(db_->Delete(WriteOptions(), iter->key())); |
||||||
|
} |
||||||
|
|
||||||
|
delete iter; |
||||||
|
} |
||||||
|
|
||||||
|
void ResetDBState(ResetMethod reset_method) { |
||||||
|
switch (reset_method) { |
||||||
|
case RESET_DROP_UNSYNCED_DATA: |
||||||
|
ASSERT_OK(env_->DropUnsyncedFileData()); |
||||||
|
break; |
||||||
|
case RESET_DELETE_UNSYNCED_FILES: |
||||||
|
ASSERT_OK(env_->DeleteFilesCreatedAfterLastDirSync()); |
||||||
|
break; |
||||||
|
default: |
||||||
|
assert(false); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
void PartialCompactTestPreFault(int num_pre_sync, int num_post_sync) { |
||||||
|
DeleteAllData(); |
||||||
|
Build(0, num_pre_sync); |
||||||
|
db_->CompactRange(NULL, NULL); |
||||||
|
Build(num_pre_sync, num_post_sync); |
||||||
|
} |
||||||
|
|
||||||
|
void PartialCompactTestReopenWithFault(ResetMethod reset_method, |
||||||
|
int num_pre_sync, |
||||||
|
int num_post_sync) { |
||||||
|
env_->SetFilesystemActive(false); |
||||||
|
CloseDB(); |
||||||
|
ResetDBState(reset_method); |
||||||
|
ASSERT_OK(OpenDB()); |
||||||
|
ASSERT_OK(Verify(0, num_pre_sync, FaultInjectionTest::VAL_EXPECT_NO_ERROR)); |
||||||
|
ASSERT_OK(Verify(num_pre_sync, num_post_sync, FaultInjectionTest::VAL_EXPECT_ERROR)); |
||||||
|
} |
||||||
|
|
||||||
|
void NoWriteTestPreFault() { |
||||||
|
} |
||||||
|
|
||||||
|
void NoWriteTestReopenWithFault(ResetMethod reset_method) { |
||||||
|
CloseDB(); |
||||||
|
ResetDBState(reset_method); |
||||||
|
ASSERT_OK(OpenDB()); |
||||||
|
} |
||||||
|
|
||||||
|
void DoTest() { |
||||||
|
Random rnd(0); |
||||||
|
ASSERT_OK(OpenDB()); |
||||||
|
for (size_t idx = 0; idx < kNumIterations; idx++) { |
||||||
|
int num_pre_sync = rnd.Uniform(kMaxNumValues); |
||||||
|
int num_post_sync = rnd.Uniform(kMaxNumValues); |
||||||
|
|
||||||
|
PartialCompactTestPreFault(num_pre_sync, num_post_sync); |
||||||
|
PartialCompactTestReopenWithFault(RESET_DROP_UNSYNCED_DATA, |
||||||
|
num_pre_sync, |
||||||
|
num_post_sync); |
||||||
|
|
||||||
|
NoWriteTestPreFault(); |
||||||
|
NoWriteTestReopenWithFault(RESET_DROP_UNSYNCED_DATA); |
||||||
|
|
||||||
|
PartialCompactTestPreFault(num_pre_sync, num_post_sync); |
||||||
|
// No new files created so we expect all values since no files will be
|
||||||
|
// dropped.
|
||||||
|
PartialCompactTestReopenWithFault(RESET_DELETE_UNSYNCED_FILES, |
||||||
|
num_pre_sync + num_post_sync, |
||||||
|
0); |
||||||
|
|
||||||
|
NoWriteTestPreFault(); |
||||||
|
NoWriteTestReopenWithFault(RESET_DELETE_UNSYNCED_FILES); |
||||||
|
} |
||||||
|
} |
||||||
|
}; |
||||||
|
|
||||||
|
TEST(FaultInjectionTest, FaultTestNoLogReuse) { |
||||||
|
ReuseLogs(false); |
||||||
|
DoTest(); |
||||||
|
} |
||||||
|
|
||||||
|
TEST(FaultInjectionTest, FaultTestWithLogReuse) { |
||||||
|
ReuseLogs(true); |
||||||
|
DoTest(); |
||||||
|
} |
||||||
|
|
||||||
|
} // namespace leveldb
|
||||||
|
|
||||||
|
int main(int argc, char** argv) { |
||||||
|
return leveldb::test::RunAllTests(); |
||||||
|
} |
||||||
@ -0,0 +1,324 @@ |
|||||||
|
// Copyright (c) 2014 The LevelDB Authors. All rights reserved.
|
||||||
|
// Use of this source code is governed by a BSD-style license that can be
|
||||||
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
||||||
|
|
||||||
|
#include "db/db_impl.h" |
||||||
|
#include "db/filename.h" |
||||||
|
#include "db/version_set.h" |
||||||
|
#include "db/write_batch_internal.h" |
||||||
|
#include "leveldb/db.h" |
||||||
|
#include "leveldb/env.h" |
||||||
|
#include "leveldb/write_batch.h" |
||||||
|
#include "util/logging.h" |
||||||
|
#include "util/testharness.h" |
||||||
|
#include "util/testutil.h" |
||||||
|
|
||||||
|
namespace leveldb { |
||||||
|
|
||||||
|
class RecoveryTest { |
||||||
|
public: |
||||||
|
RecoveryTest() : env_(Env::Default()), db_(NULL) { |
||||||
|
dbname_ = test::TmpDir() + "/recovery_test"; |
||||||
|
DestroyDB(dbname_, Options()); |
||||||
|
Open(); |
||||||
|
} |
||||||
|
|
||||||
|
~RecoveryTest() { |
||||||
|
Close(); |
||||||
|
DestroyDB(dbname_, Options()); |
||||||
|
} |
||||||
|
|
||||||
|
DBImpl* dbfull() const { return reinterpret_cast<DBImpl*>(db_); } |
||||||
|
Env* env() const { return env_; } |
||||||
|
|
||||||
|
bool CanAppend() { |
||||||
|
WritableFile* tmp; |
||||||
|
Status s = env_->NewAppendableFile(CurrentFileName(dbname_), &tmp); |
||||||
|
delete tmp; |
||||||
|
if (s.IsNotSupportedError()) { |
||||||
|
return false; |
||||||
|
} else { |
||||||
|
return true; |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
void Close() { |
||||||
|
delete db_; |
||||||
|
db_ = NULL; |
||||||
|
} |
||||||
|
|
||||||
|
void Open(Options* options = NULL) { |
||||||
|
Close(); |
||||||
|
Options opts; |
||||||
|
if (options != NULL) { |
||||||
|
opts = *options; |
||||||
|
} else { |
||||||
|
opts.reuse_logs = true; // TODO(sanjay): test both ways
|
||||||
|
opts.create_if_missing = true; |
||||||
|
} |
||||||
|
if (opts.env == NULL) { |
||||||
|
opts.env = env_; |
||||||
|
} |
||||||
|
ASSERT_OK(DB::Open(opts, dbname_, &db_)); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
} |
||||||
|
|
||||||
|
Status Put(const std::string& k, const std::string& v) { |
||||||
|
return db_->Put(WriteOptions(), k, v); |
||||||
|
} |
||||||
|
|
||||||
|
std::string Get(const std::string& k, const Snapshot* snapshot = NULL) { |
||||||
|
std::string result; |
||||||
|
Status s = db_->Get(ReadOptions(), k, &result); |
||||||
|
if (s.IsNotFound()) { |
||||||
|
result = "NOT_FOUND"; |
||||||
|
} else if (!s.ok()) { |
||||||
|
result = s.ToString(); |
||||||
|
} |
||||||
|
return result; |
||||||
|
} |
||||||
|
|
||||||
|
std::string ManifestFileName() { |
||||||
|
std::string current; |
||||||
|
ASSERT_OK(ReadFileToString(env_, CurrentFileName(dbname_), ¤t)); |
||||||
|
size_t len = current.size(); |
||||||
|
if (len > 0 && current[len-1] == '\n') { |
||||||
|
current.resize(len - 1); |
||||||
|
} |
||||||
|
return dbname_ + "/" + current; |
||||||
|
} |
||||||
|
|
||||||
|
std::string LogName(uint64_t number) { |
||||||
|
return LogFileName(dbname_, number); |
||||||
|
} |
||||||
|
|
||||||
|
size_t DeleteLogFiles() { |
||||||
|
std::vector<uint64_t> logs = GetFiles(kLogFile); |
||||||
|
for (size_t i = 0; i < logs.size(); i++) { |
||||||
|
ASSERT_OK(env_->DeleteFile(LogName(logs[i]))) << LogName(logs[i]); |
||||||
|
} |
||||||
|
return logs.size(); |
||||||
|
} |
||||||
|
|
||||||
|
uint64_t FirstLogFile() { |
||||||
|
return GetFiles(kLogFile)[0]; |
||||||
|
} |
||||||
|
|
||||||
|
std::vector<uint64_t> GetFiles(FileType t) { |
||||||
|
std::vector<std::string> filenames; |
||||||
|
ASSERT_OK(env_->GetChildren(dbname_, &filenames)); |
||||||
|
std::vector<uint64_t> result; |
||||||
|
for (size_t i = 0; i < filenames.size(); i++) { |
||||||
|
uint64_t number; |
||||||
|
FileType type; |
||||||
|
if (ParseFileName(filenames[i], &number, &type) && type == t) { |
||||||
|
result.push_back(number); |
||||||
|
} |
||||||
|
} |
||||||
|
return result; |
||||||
|
} |
||||||
|
|
||||||
|
int NumLogs() { |
||||||
|
return GetFiles(kLogFile).size(); |
||||||
|
} |
||||||
|
|
||||||
|
int NumTables() { |
||||||
|
return GetFiles(kTableFile).size(); |
||||||
|
} |
||||||
|
|
||||||
|
uint64_t FileSize(const std::string& fname) { |
||||||
|
uint64_t result; |
||||||
|
ASSERT_OK(env_->GetFileSize(fname, &result)) << fname; |
||||||
|
return result; |
||||||
|
} |
||||||
|
|
||||||
|
void CompactMemTable() { |
||||||
|
dbfull()->TEST_CompactMemTable(); |
||||||
|
} |
||||||
|
|
||||||
|
// Directly construct a log file that sets key to val.
|
||||||
|
void MakeLogFile(uint64_t lognum, SequenceNumber seq, Slice key, Slice val) { |
||||||
|
std::string fname = LogFileName(dbname_, lognum); |
||||||
|
WritableFile* file; |
||||||
|
ASSERT_OK(env_->NewWritableFile(fname, &file)); |
||||||
|
log::Writer writer(file); |
||||||
|
WriteBatch batch; |
||||||
|
batch.Put(key, val); |
||||||
|
WriteBatchInternal::SetSequence(&batch, seq); |
||||||
|
ASSERT_OK(writer.AddRecord(WriteBatchInternal::Contents(&batch))); |
||||||
|
ASSERT_OK(file->Flush()); |
||||||
|
delete file; |
||||||
|
} |
||||||
|
|
||||||
|
private: |
||||||
|
std::string dbname_; |
||||||
|
Env* env_; |
||||||
|
DB* db_; |
||||||
|
}; |
||||||
|
|
||||||
|
TEST(RecoveryTest, ManifestReused) { |
||||||
|
if (!CanAppend()) { |
||||||
|
fprintf(stderr, "skipping test because env does not support appending\n"); |
||||||
|
return; |
||||||
|
} |
||||||
|
ASSERT_OK(Put("foo", "bar")); |
||||||
|
Close(); |
||||||
|
std::string old_manifest = ManifestFileName(); |
||||||
|
Open(); |
||||||
|
ASSERT_EQ(old_manifest, ManifestFileName()); |
||||||
|
ASSERT_EQ("bar", Get("foo")); |
||||||
|
Open(); |
||||||
|
ASSERT_EQ(old_manifest, ManifestFileName()); |
||||||
|
ASSERT_EQ("bar", Get("foo")); |
||||||
|
} |
||||||
|
|
||||||
|
TEST(RecoveryTest, LargeManifestCompacted) { |
||||||
|
if (!CanAppend()) { |
||||||
|
fprintf(stderr, "skipping test because env does not support appending\n"); |
||||||
|
return; |
||||||
|
} |
||||||
|
ASSERT_OK(Put("foo", "bar")); |
||||||
|
Close(); |
||||||
|
std::string old_manifest = ManifestFileName(); |
||||||
|
|
||||||
|
// Pad with zeroes to make manifest file very big.
|
||||||
|
{ |
||||||
|
uint64_t len = FileSize(old_manifest); |
||||||
|
WritableFile* file; |
||||||
|
ASSERT_OK(env()->NewAppendableFile(old_manifest, &file)); |
||||||
|
std::string zeroes(3*1048576 - static_cast<size_t>(len), 0); |
||||||
|
ASSERT_OK(file->Append(zeroes)); |
||||||
|
ASSERT_OK(file->Flush()); |
||||||
|
delete file; |
||||||
|
} |
||||||
|
|
||||||
|
Open(); |
||||||
|
std::string new_manifest = ManifestFileName(); |
||||||
|
ASSERT_NE(old_manifest, new_manifest); |
||||||
|
ASSERT_GT(10000, FileSize(new_manifest)); |
||||||
|
ASSERT_EQ("bar", Get("foo")); |
||||||
|
|
||||||
|
Open(); |
||||||
|
ASSERT_EQ(new_manifest, ManifestFileName()); |
||||||
|
ASSERT_EQ("bar", Get("foo")); |
||||||
|
} |
||||||
|
|
||||||
|
TEST(RecoveryTest, NoLogFiles) { |
||||||
|
ASSERT_OK(Put("foo", "bar")); |
||||||
|
ASSERT_EQ(1, DeleteLogFiles()); |
||||||
|
Open(); |
||||||
|
ASSERT_EQ("NOT_FOUND", Get("foo")); |
||||||
|
Open(); |
||||||
|
ASSERT_EQ("NOT_FOUND", Get("foo")); |
||||||
|
} |
||||||
|
|
||||||
|
TEST(RecoveryTest, LogFileReuse) { |
||||||
|
if (!CanAppend()) { |
||||||
|
fprintf(stderr, "skipping test because env does not support appending\n"); |
||||||
|
return; |
||||||
|
} |
||||||
|
for (int i = 0; i < 2; i++) { |
||||||
|
ASSERT_OK(Put("foo", "bar")); |
||||||
|
if (i == 0) { |
||||||
|
// Compact to ensure current log is empty
|
||||||
|
CompactMemTable(); |
||||||
|
} |
||||||
|
Close(); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
uint64_t number = FirstLogFile(); |
||||||
|
if (i == 0) { |
||||||
|
ASSERT_EQ(0, FileSize(LogName(number))); |
||||||
|
} else { |
||||||
|
ASSERT_LT(0, FileSize(LogName(number))); |
||||||
|
} |
||||||
|
Open(); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
ASSERT_EQ(number, FirstLogFile()) << "did not reuse log file"; |
||||||
|
ASSERT_EQ("bar", Get("foo")); |
||||||
|
Open(); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
ASSERT_EQ(number, FirstLogFile()) << "did not reuse log file"; |
||||||
|
ASSERT_EQ("bar", Get("foo")); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
TEST(RecoveryTest, MultipleMemTables) { |
||||||
|
// Make a large log.
|
||||||
|
const int kNum = 1000; |
||||||
|
for (int i = 0; i < kNum; i++) { |
||||||
|
char buf[100]; |
||||||
|
snprintf(buf, sizeof(buf), "%050d", i); |
||||||
|
ASSERT_OK(Put(buf, buf)); |
||||||
|
} |
||||||
|
ASSERT_EQ(0, NumTables()); |
||||||
|
Close(); |
||||||
|
ASSERT_EQ(0, NumTables()); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
uint64_t old_log_file = FirstLogFile(); |
||||||
|
|
||||||
|
// Force creation of multiple memtables by reducing the write buffer size.
|
||||||
|
Options opt; |
||||||
|
opt.reuse_logs = true; |
||||||
|
opt.write_buffer_size = (kNum*100) / 2; |
||||||
|
Open(&opt); |
||||||
|
ASSERT_LE(2, NumTables()); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
ASSERT_NE(old_log_file, FirstLogFile()) << "must not reuse log"; |
||||||
|
for (int i = 0; i < kNum; i++) { |
||||||
|
char buf[100]; |
||||||
|
snprintf(buf, sizeof(buf), "%050d", i); |
||||||
|
ASSERT_EQ(buf, Get(buf)); |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
TEST(RecoveryTest, MultipleLogFiles) { |
||||||
|
ASSERT_OK(Put("foo", "bar")); |
||||||
|
Close(); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
|
||||||
|
// Make a bunch of uncompacted log files.
|
||||||
|
uint64_t old_log = FirstLogFile(); |
||||||
|
MakeLogFile(old_log+1, 1000, "hello", "world"); |
||||||
|
MakeLogFile(old_log+2, 1001, "hi", "there"); |
||||||
|
MakeLogFile(old_log+3, 1002, "foo", "bar2"); |
||||||
|
|
||||||
|
// Recover and check that all log files were processed.
|
||||||
|
Open(); |
||||||
|
ASSERT_LE(1, NumTables()); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
uint64_t new_log = FirstLogFile(); |
||||||
|
ASSERT_LE(old_log+3, new_log); |
||||||
|
ASSERT_EQ("bar2", Get("foo")); |
||||||
|
ASSERT_EQ("world", Get("hello")); |
||||||
|
ASSERT_EQ("there", Get("hi")); |
||||||
|
|
||||||
|
// Test that previous recovery produced recoverable state.
|
||||||
|
Open(); |
||||||
|
ASSERT_LE(1, NumTables()); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
if (CanAppend()) { |
||||||
|
ASSERT_EQ(new_log, FirstLogFile()); |
||||||
|
} |
||||||
|
ASSERT_EQ("bar2", Get("foo")); |
||||||
|
ASSERT_EQ("world", Get("hello")); |
||||||
|
ASSERT_EQ("there", Get("hi")); |
||||||
|
|
||||||
|
// Check that introducing an older log file does not cause it to be re-read.
|
||||||
|
Close(); |
||||||
|
MakeLogFile(old_log+1, 2000, "hello", "stale write"); |
||||||
|
Open(); |
||||||
|
ASSERT_LE(1, NumTables()); |
||||||
|
ASSERT_EQ(1, NumLogs()); |
||||||
|
if (CanAppend()) { |
||||||
|
ASSERT_EQ(new_log, FirstLogFile()); |
||||||
|
} |
||||||
|
ASSERT_EQ("bar2", Get("foo")); |
||||||
|
ASSERT_EQ("world", Get("hello")); |
||||||
|
ASSERT_EQ("there", Get("hi")); |
||||||
|
} |
||||||
|
|
||||||
|
} // namespace leveldb
|
||||||
|
|
||||||
|
int main(int argc, char** argv) { |
||||||
|
return leveldb::test::RunAllTests(); |
||||||
|
} |
||||||
Loading…
Reference in new issue