summaryrefslogtreecommitdiffstats
path: root/common/storage_lmdb.cpp
Commit message (Collapse)AuthorAge
* Separate UIDs and Revisions in main databasesRémi Nicole2018-08-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: - Change revision type from `qint64` to `size_t` for LMDB in a couple of places (LMDB supports `unsigned int` or `size_t` which are `long unsigned int` on my machine) - Better support for database flags (duplicate, integer keys, integer values for now but is extensible) - Main databases' keys are now revisions - Some databases switched to integer keys databases: - Main databases - the revision to uid mapping database - the revision to entity type mapping database - Refactor the entity type's `typeDatabases` method (if in the future we need to change the main databases' flags again) - New uid to revision mapping database (`uidsToRevisions`): - Stores all revisions (not uid to latest revision) because we need it for cleaning old revisions - Flags are: duplicates + integer values (so findLatest finds the latest revision for the given uid) ~~Problems to fix before merging:~~ All Fixed! - ~~Sometimes Sink can't read what has just been written to the database (maybe because of transactions race conditions)~~ - ~~Most of the times, this results in Sink not able to find the uid for a given revision by reading the `revisions` database~~ - ~~`pipelinetest`'s `testModifyWithConflict` fails because the local changes are overridden~~ ~~The first problem prevents me from running benchmarks~~ Reviewers: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D14974
* Actually store the flags instead of the db name.Christian Mollekopf2018-08-08
| | | | | | | The reason why we didn't notice was probably: * we only use this table nowadays when we have no db layout. * The only flag we ever set is the dupsort flag, and the return from a failed int conversion of 0 is otherwise correct.
* Fixed threading issues with dynamic db creation.Christian Mollekopf2018-08-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch addresses two problems: * A potential deadlock. We had the following code inside a separately protected section: dbiLocker.unlock(); //Here we could loos the readlock QWriteLocker dbiWriteLocker(&sDbisLock); If we lost the lock in between the two lines, the second thread that was now holding a read-lock on sDbisLock could not enter the protected section, which was a requirement to release the read-lock, and we'd thus end up in a deadlock. This is solved using tryLock with intermediate releases of the read-lock, allowing the original thread to finish. * When failing to validate a dbi for the current transacation we simply returned an invalid db (which then in this particular case broke reading of revision uid's and type's), leading to queries not executing as they should. Both problems are unfortunately hard to reproduce, the adjusted test at leaset allowed me to reproduce the deadlock situation sometimes. To fix this cleanly we should probably just get rid of dynamic dbi allocation for good.
* Fixed the case when a dbi would leak through to a transaction where itChristian Mollekopf2018-07-12
| | | | | | shouldn't be visible yet. Was reproducible in the initial sync of the caldav resource.
* Reduce the maximum database size to 200MB until we have fixed the lmdbChristian Mollekopf2018-06-01
| | | | | | | | | | | issues. https://phabricator.kde.org/T8723 With 200MB we can both deal with the 200MB files on disk, and we could even load all of them (the 5 databases the resource uses), into memory. Once the open problems are resolved we should be able to bump it back to at least 20GB.
* Reduce the maximum database size to 10GB on windows.Christian Mollekopf2018-05-29
|
* Only call the result handler if reading succeededChristian Mollekopf2018-05-29
|
* Document why we might be creating a db several times.Christian Mollekopf2018-05-27
|
* We can't recover from not being able to open the database.Christian Mollekopf2018-05-24
| | | | | We'll only end up with defunctional processes that may or may not do anything useful.
* Get the mapsize from a separate function and check the error whenChristian Mollekopf2018-05-24
| | | | setting it.
* Fixed a bunch of warningsChristian Mollekopf2018-05-24
|
* Return number of retrieved values.Christian Mollekopf2018-05-24
|
* Add findAllInRange function in the storage layerRémi Nicole2018-05-24
| | | | | | | | | | | | | | | | | Summary: In preparation of the support for ranged queries. Notes: Since they are pretty similar, it could be nice to refactor `scan` and `findAllInRange` to use common 3rd function Test Plan: This is tested in storagetest.cpp Reviewers: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D13066
* We were off by a factor of 10Christian Mollekopf2018-05-24
| | | | | It's possible that we therefore went over the virtual address space limit on windows which is 128GB.
* Opening dbis may fail in read-only modeChristian Mollekopf2018-05-24
|
* I think those variables should just be staticChristian Mollekopf2018-05-24
|
* We need an extern declaration for non-static variables.Christian Mollekopf2018-05-23
| | | | Or so says the compiler.
* Fixed use of mdb_dbi_openChristian Mollekopf2018-05-23
| | | | | | | | | | | | | | | | | | | | There can only ever be one transaction using mdb_dbi_open running, and that transaction must commit or abort before any other transaction attempts to use mdb_dbi_open. Use delayed dbi merging with write transactions and a temporary transaction for read transactions. We now protect dbi initialization with a mutex and immediately update the sDbis hash. This assumes that the created dbis are indeed We can still violate the only one transaction may use mdb_dbi_open rule if we start a read-only transaction after the write transaction, before the write transaction commits. It does not seem to be something we actually do though. Opening dbis on environment init is further separated out, so we don't end up in the regular openDatabase codepath at all.
* Actually close the lmdb environment before removing from disk.Christian Mollekopf2018-05-20
| | | | Otherwise removal doesn't work on windows due to open file handles.
* mdb_env_set_mapsize should be called after create but before openChristian Mollekopf2018-05-15
| | | | according to the docs.
* Try a bit harder to ensure that env always has a sane value.Christian Mollekopf2018-05-15
|
* Windows compatChristian Mollekopf2018-04-19
|
* Make sure we initialize the environment correctly and only set the db ↵Christian Mollekopf2018-03-26
| | | | version when creating it
* Don't just set the db version, only when we create the db first.Christian Mollekopf2018-03-26
|
* Optional printing of transaction spanChristian Mollekopf2018-01-30
|
* Use read-write locks for finer grained control to sDbi and sEnvironmentsChristian Mollekopf2018-01-03
| | | | | | | | | | There are only a few cases where have to access the list of dbis or environments, so we can normally get away with just read-locking. This seems to fix a segfault that was possibly caused be an environment being reused that has already been freed in another thread. The read-only lock when initially retrieving the environment seems to fix that.
* Trying to open a db that doesn't exist in read-only mode is not an errorChristian Mollekopf2018-01-03
|
* Print transaction spanChristian Mollekopf2017-11-21
|
* Storage debugging codeChristian Mollekopf2017-11-21
|
* Print key/value when put failsChristian Mollekopf2017-11-20
|
* Simplified codeChristian Mollekopf2017-07-16
| | | | The while loop is executed at least once, so advanced is always true.
* Better error messagesChristian Mollekopf2017-07-16
|
* initEnvironment with double-checked lockingChristian Mollekopf2017-05-11
|
* Starting to get rid of the SINK_DEBUG_AREAChristian Mollekopf2017-05-11
|
* A defined table layoutChristian Mollekopf2017-05-06
|
* Don't leak transactions when copying them.Christian Mollekopf2017-05-05
| | | | Previsouly we would hit the maxreaders limit
* Take sink down if we're not assertingChristian Mollekopf2017-05-04
|
* CleanupChristian Mollekopf2017-05-04
|
* Sanity check db namesChristian Mollekopf2017-05-03
| | | | | | lmdb and sink deal badly with e.g. a string containing a null in the millde as db name. Thus we try to protect better against it. This is an actual problem we triggered: https://phabricator.kde.org/T5880
* lmdb cleanupChristian Mollekopf2017-03-06
|
* Ensure we don't try to open db's that are not available to ourChristian Mollekopf2017-03-06
| | | | transaction.
* Less debug outputChristian Mollekopf2017-02-28
|
* Make opening dbis non-racyChristian Mollekopf2017-02-27
| | | | | | | | | | | | | | Dbis can only be opened by one thread and should then be shared accross all threads after committing the transaction to create the dbi. This requires us to initially open all db's, which in turn requires us to know the correct flags. This patch stores the flags to open each db in a separate db, and then opens up all databases on initial start. If a new database is created that dbi is as well shared as soon as the transaction is committed (before the dbi is private to the transaction).
* Attempt to fix the issue of opening the wrong databaseChristian Mollekopf2017-02-26
|
* Better debug outputChristian Mollekopf2017-02-24
|
* Bring the application down in a controlled fashion if we experience aChristian Mollekopf2017-02-23
| | | | critical error.
* Fixed index removalsChristian Mollekopf2017-02-13
|
* Fixed build issues with pedantic enabledChristian Mollekopf2017-01-12
|
* Copy command and proper moveChristian Mollekopf2016-12-04
|
* Refactor how the storage is used.Christian Mollekopf2016-10-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is the initial refactoring to improve how we deal with the storage. It does a couple of things: * Rename Sink::Storage to Sink::Storage::DataStore to free up the Sink::Storage namespace * Introduce a Sink::ResourceContext to have a single object that can be passed around containing everything that is necessary to operate on a resource. This is a lot better than the multiple separate parameters that we used to pass around all over the place, while still allowing for dependency injection for tests. * Tie storage access together using the new EntityStore that directly works with ApplicationDomainTypes. This gives us a central place where main storage, indexes and buffer adaptors are tied together, which will also give us a place to implement external indexes, such as a fulltextindex using xapian. * Use ApplicationDomainTypes as the default way to pass around entities. Instead of using various ways to pass around entities (buffers, buffer adaptors, ApplicationDomainTypes), only use a single way. The old approach was confusing, and was only done as: * optimization; really shouldn't be necessary and otherwise I'm sure we can find better ways to optimize ApplicationDomainType itself. * a way to account for entities that have multiple buffers, a concept that I no longer deem relevant. While this commit does the bulk of the work to get there, the following commits will refactor more stuff to get things back to normal.
generated by cgit v1.2.3 (git 2.25.1) at 2025-04-28 21:27:24 +0000