summaryrefslogtreecommitdiffstats
path: root/common/storage
Commit message (Collapse)AuthorAge
* Separate UIDs and Revisions in main databasesRémi Nicole2018-08-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: - Change revision type from `qint64` to `size_t` for LMDB in a couple of places (LMDB supports `unsigned int` or `size_t` which are `long unsigned int` on my machine) - Better support for database flags (duplicate, integer keys, integer values for now but is extensible) - Main databases' keys are now revisions - Some databases switched to integer keys databases: - Main databases - the revision to uid mapping database - the revision to entity type mapping database - Refactor the entity type's `typeDatabases` method (if in the future we need to change the main databases' flags again) - New uid to revision mapping database (`uidsToRevisions`): - Stores all revisions (not uid to latest revision) because we need it for cleaning old revisions - Flags are: duplicates + integer values (so findLatest finds the latest revision for the given uid) ~~Problems to fix before merging:~~ All Fixed! - ~~Sometimes Sink can't read what has just been written to the database (maybe because of transactions race conditions)~~ - ~~Most of the times, this results in Sink not able to find the uid for a given revision by reading the `revisions` database~~ - ~~`pipelinetest`'s `testModifyWithConflict` fails because the local changes are overridden~~ ~~The first problem prevents me from running benchmarks~~ Reviewers: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D14974
* Print parsed keys that we can read.Christian Mollekopf2018-08-17
|
* Avoid creating an instance just to check for instanceChristian Mollekopf2018-08-06
|
* Avoid unnecessary Identifier conversions in performance ciritical code.Christian Mollekopf2018-07-28
| | | | | This fixes the performance regressions to a state where we are roughly at the same performance as pre Identifier (but not any better either).
* Fixed buildChristian Mollekopf2018-07-27
|
* Use Key API in SinkSHRémi Nicole2018-07-27
| | | | | | | | | | | | | | | | | | Summary: Depends on D14289 - Fixes the `sinksh inspect …` command - Introduces `isValid`, `isValidInternal` and `isValidDisplay` static functions in Key, Identifier and Revision I still have to do a more extensive search for induced bugs in other commands Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D14404
* Use Key API in DataStoreQueryRémi Nicole2018-07-27
| | | | | | | | | | Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D14099
* Use Key API in indexesRémi Nicole2018-07-27
| | | | | | | | | | | | | | | | | | Summary: - Only in TypeIndex, not in Index (since we might want to store something other than identifiers as values) - We might want to do the same in the `SynchronizerStore` for localId ↔ remoteId indexes Depends on D13735 Some quick benchmarks (against develop and D13735): {F6022279} Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D13902
* New Key API in storage layerRémi Nicole2018-07-27
| | | | | | | | | | | | | | | | | Summary: - Use object oriented paradigm for Keys / Identifiers /Revisions - "Compress" keys by using byte representation of Uuids - Still some cleaning left to do - Also run some benchmarks - I'm questioning whether files other than entitystore (tests excluded) are allowed to access this API Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D13735
* Avoid overwriting local changes with remote modifications.Christian Mollekopf2018-06-26
| | | | | | | | | | | | | | | | | | | The case we ran into is the following: * Fetching the full payload and marking all messages of a thread as read happens simultaneously. * The local modification to mark as read gets immediately overwritten when the full payload arrives. * Eventually the modification gets replayed to the server though (and the reversal isn't because coming from the source), so on next sync the situation fixes itself. To be able to improve this we try to protect local modifications in that properties that have been modified since baseRevision (which currently isn't, but should be equal to the last to the server replayed revision) are not overwritten. This conflict resolution strategy thus always prefers local modifications. baseRevision is currently set to the current maximum revision of the store at the time when the resource creates the modification.
* Fixed the thread index.Christian Mollekopf2018-06-19
| | | | | | | | * Modifications could result in index changes because we lost the threadId due to remove + add. A modify was necessary (although we can ignore it for the email case). * The ThreadIndexer would try to lookup and potentially index threads for empty parent ids, which is clearly wrong.
* Asserts, debug messages and other cleanupChristian Mollekopf2018-06-19
|
* Implement Overlap queriesRémi Nicole2018-06-19
| | | | | | | | | | | | | | | | | | | | Summary: Notes: - Introduces the concept of queries on multiple properties (which meant changing query's internals a bit) - Dates are stored as well as the "reference" in the index to allow quick filtering without fetching the whole entity - Buckets are weeks starting on Monday (guaranteed by the use of the Julian calendar) - Some size improvements are definitely possible (dates are padded numbers again, not using integer databases, Julian calendar starts at a very old date, etc.) Test Plan: Tested in querytest Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D13477
* Improved error messagesChristian Mollekopf2018-05-18
|
* A lot less namespace typing.Christian Mollekopf2018-05-17
|
* Copy the entity when we return it.Christian Mollekopf2018-05-17
| | | | | Otherwise we easily end up copying it and then have an entity that points into nowhere. Callback -> no copy, no callback -> copy.
* Using findLatest with an empty uid is an error.Christian Mollekopf2018-05-14
| | | | | Seems to be happening sometimes, so that needs to be fixed once we find the cause.
* Fix CalDAV test in Docker + add Calendar and Todo in the database layoutRémi Nicole2018-05-14
| | | | | | | | | | | | | Summary: Like the title says. Also removed a "/" that got doubled between the collection URI and the item path, because it made Cyrus freak out (returning unauthorized errors) Reviewers: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D12755
* Make sure we initialize the environment correctly and only set the db ↵Christian Mollekopf2018-03-26
| | | | version when creating it
* Don't just set the db version, only when we create the db first.Christian Mollekopf2018-03-26
|
* Deal with removals in reduced queriesChristian Mollekopf2018-02-22
|
* Fixed crashes due to concurrently running queries.Christian Mollekopf2018-02-15
| | | | | | | | | | | | | | | | | | | | | | | | A single QueryRunner should never have multiple workers running at the same time. We did not properly enforce this in case of incremental updates coming in. The only way I managed to reproduce the crash: * Open a large folder with lots of unread mail in kube * Select a mail in the maillist and hold the down button * This will: * Repeatedly call fetch more * Trigger lot's of mark as read modifications that result in notifications. * Eventually it crashes somewhere in EntityStore, likely because of concurrent access of the filter structure which is shared through the state. We now ensure in the single threaded portion of the code that we only ever run one worker at a time. If we did receive an update during, we remember that change and fetch more once we're done. To be able to call fetch again that portion was also factored out into a separate function.
* Xapian based fulltext indexingChristian Mollekopf2018-02-11
| | | | | This cuts into the sync performance by about 40%, but gives us fast fulltext searching for all local content.
* Removed all traces of BLOB propertiesChristian Mollekopf2018-02-06
|
* Fixed getUids by type filtering.Christian Mollekopf2017-09-07
| | | | | We used to simply return all uids. Requires "sinksh upgrade"
* No return value needed here.Christian Mollekopf2017-07-16
|
* We don't need the layout just to check for existenceChristian Mollekopf2017-05-20
|
* Avoid redoing stuff over and over that we can easily avoid.Christian Mollekopf2017-05-12
|
* Avoid copyingChristian Mollekopf2017-05-12
|
* Don't store blobs in directories.Christian Mollekopf2017-05-11
| | | | | Creating the directories is way more expensive than searching through the files on removal.
* Fixed merge implementationChristian Mollekopf2017-05-08
|
* Gather required databases from index definitions.Christian Mollekopf2017-05-08
|
* A defined table layoutChristian Mollekopf2017-05-06
|
* CleanupChristian Mollekopf2017-05-04
|
* Sanity check db namesChristian Mollekopf2017-05-03
| | | | | | lmdb and sink deal badly with e.g. a string containing a null in the millde as db name. Thus we try to protect better against it. This is an actual problem we triggered: https://phabricator.kde.org/T5880
* Moved all preprocessing back into the pipelineChristian Mollekopf2017-04-11
|
* Move the preprocssing back out of entitystore into the pipeline.Christian Mollekopf2017-04-11
| | | | | This is where this really belongs, only the indexing is part of storage. This is necessary so preprocessors can move entities as well.
* Added uid indexChristian Mollekopf2017-04-10
|
* Fixed readAllUids and readAllChristian Mollekopf2017-04-10
|
* Move type implementations in one placeChristian Mollekopf2017-03-21
| | | | | | | Having them separated is rather pointless (since we need one for every type, and all types are the interface of sink, as one), and caused quite a bit of friction when adding new types. This will also make it easier to change things for all types.
* Make opening dbis non-racyChristian Mollekopf2017-02-27
| | | | | | | | | | | | | | Dbis can only be opened by one thread and should then be shared accross all threads after committing the transaction to create the dbi. This requires us to initially open all db's, which in turn requires us to know the correct flags. This patch stores the flags to open each db in a separate db, and then opens up all databases on initial start. If a new database is created that dbi is as well shared as soon as the transaction is committed (before the dbi is private to the transaction).
* Remember whether a blob property is external or not.Christian Mollekopf2017-02-16
| | | | | ...we used to accidentally move external blobs after the property was lost when storing the modificatoin in the queue.
* Ensure we don't accidentally spawn more transactions than necessary.Christian Mollekopf2017-01-23
|
* Debug outputChristian Mollekopf2017-01-23
|
* Fixed revision cleanupChristian Mollekopf2017-01-22
|
* Fixed build issues with pedantic enabledChristian Mollekopf2017-01-12
|
* Fixed revision cleanupChristian Mollekopf2017-01-10
|
* Introduced Log::ContextChristian Mollekopf2016-12-22
| | | | | | | To have hierarchical debug output we have to pass around something at run-time, there is no reasonable alternative. Log::Context provides the identifier to do just that and largely replaces the debug component idea.
* CleanupChristian Mollekopf2016-12-20
|
* Avoid unnecessary warnings if the db is not existing.Christian Mollekopf2016-12-16
| | | | | It is expected that a query returns nothing if the db is not existing yet.
generated by cgit v1.2.3 (git 2.25.1) at 2025-04-28 22:22:08 +0000