| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
- Change revision type from `qint64` to `size_t` for LMDB in a couple of places (LMDB supports `unsigned int` or `size_t` which are `long unsigned int` on my machine)
- Better support for database flags (duplicate, integer keys, integer values for now but is extensible)
- Main databases' keys are now revisions
- Some databases switched to integer keys databases:
- Main databases
- the revision to uid mapping database
- the revision to entity type mapping database
- Refactor the entity type's `typeDatabases` method (if in the future we need to change the main databases' flags again)
- New uid to revision mapping database (`uidsToRevisions`):
- Stores all revisions (not uid to latest revision) because we need it for cleaning old revisions
- Flags are: duplicates + integer values (so findLatest finds the latest revision for the given uid)
~~Problems to fix before merging:~~
All Fixed!
- ~~Sometimes Sink can't read what has just been written to the database (maybe because of transactions race conditions)~~
- ~~Most of the times, this results in Sink not able to find the uid for a given revision by reading the `revisions` database~~
- ~~`pipelinetest`'s `testModifyWithConflict` fails because the local changes are overridden~~
~~The first problem prevents me from running benchmarks~~
Reviewers: cmollekopf
Tags: #sink
Differential Revision: https://phabricator.kde.org/D14974
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
- Use object oriented paradigm for Keys / Identifiers /Revisions
- "Compress" keys by using byte representation of Uuids
- Still some cleaning left to do
- Also run some benchmarks
- I'm questioning whether files other than entitystore (tests excluded) are allowed to access this API
Reviewers: cmollekopf
Reviewed By: cmollekopf
Tags: #sink
Differential Revision: https://phabricator.kde.org/D13735
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The case we ran into is the following:
* Fetching the full payload and marking all messages of a thread as read
happens simultaneously.
* The local modification to mark as read gets immediately overwritten
when the full payload arrives.
* Eventually the modification gets replayed to the server though (and
the reversal isn't because coming from the source), so on next sync the
situation fixes itself.
To be able to improve this we try to protect local modifications in that
properties that have been modified since baseRevision (which currently
isn't, but should be equal to the last to the server replayed revision)
are not overwritten. This conflict resolution strategy thus always
prefers local modifications. baseRevision is currently set to the
current maximum revision of the store at the time when the resource
creates the modification.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is the initial refactoring to improve how we deal with the storage.
It does a couple of things:
* Rename Sink::Storage to Sink::Storage::DataStore to free up the
Sink::Storage namespace
* Introduce a Sink::ResourceContext to have a single object that can be
passed around containing everything that is necessary to operate on a
resource. This is a lot better than the multiple separate parameters
that we used to pass around all over the place, while still allowing
for dependency injection for tests.
* Tie storage access together using the new EntityStore that directly
works with ApplicationDomainTypes. This gives us a central place where
main storage, indexes and buffer adaptors are tied together, which
will also give us a place to implement external indexes, such as a
fulltextindex using xapian.
* Use ApplicationDomainTypes as the default way to pass around entities.
Instead of using various ways to pass around entities (buffers,
buffer adaptors, ApplicationDomainTypes), only use a single way.
The old approach was confusing, and was only done as:
* optimization; really shouldn't be necessary and otherwise I'm sure
we can find better ways to optimize ApplicationDomainType itself.
* a way to account for entities that have multiple buffers, a concept
that I no longer deem relevant.
While this commit does the bulk of the work to get there, the following
commits will refactor more stuff to get things back to normal.
|
| |
|
|
|
| |
The org.kde prefix is useless and possibly misleading.
Simply prefixing with sink is more unique and shorter.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
changereplay and synchronization.
This cleans up the API and avoids the excessive passing around of
transactions. It also provides more flexibility in eventually using
different synchronization strategies for different resources.
|
| |
|
|
| |
And allow preprocessors to modify the result.
|
| |
|
|
| |
clang-format -i */**{.cpp,.h}
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
(except for documentation).
|
| |
|
|
| |
dummyresourcetest
|
| | |
|
| |
|
|
| |
... also if there are intermediate revisions.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of having the asynchronous preprocessor concept with different
pipelines for new/modify/delete we have a single pipeline with
synchronous preprocessors that act upon new/modify/delete.
This keeps the code simpler due to lack of asynchronity and keeps the
new/modify/delete operations together (which at least for the indexing
makes a lot of sense).
Not supporting asynchronity is ok because the tasks done in
preprocessing are not cpu intensive (if they were we had a problem
since they are directly involved in the round-trip time), and the main
cost comes from i/o, meaning we don't gain much by doing multithreading.
Costly tasks (such as full-text indexing) should rather be implemented
as post-processing, since that doesn't increase the round-trip time directly,
and eventually consistent is typically good enough for that.
|
| | |
|
| | |
|
| |
|
|
| |
So we can replay the change.
|
| | |
|
|
|
Cleanup of revisions, and revision for removed entity is yet missing.
|
