summaryrefslogtreecommitdiffstats
path: root/common
Commit message (Collapse)AuthorAge
* Actually store the flags instead of the db name.Christian Mollekopf2018-08-08
| | | | | | | The reason why we didn't notice was probably: * we only use this table nowadays when we have no db layout. * The only flag we ever set is the dupsort flag, and the return from a failed int conversion of 0 is otherwise correct.
* Removed unused codeChristian Mollekopf2018-08-06
|
* Fixed threading issues with dynamic db creation.Christian Mollekopf2018-08-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch addresses two problems: * A potential deadlock. We had the following code inside a separately protected section: dbiLocker.unlock(); //Here we could loos the readlock QWriteLocker dbiWriteLocker(&sDbisLock); If we lost the lock in between the two lines, the second thread that was now holding a read-lock on sDbisLock could not enter the protected section, which was a requirement to release the read-lock, and we'd thus end up in a deadlock. This is solved using tryLock with intermediate releases of the read-lock, allowing the original thread to finish. * When failing to validate a dbi for the current transacation we simply returned an invalid db (which then in this particular case broke reading of revision uid's and type's), leading to queries not executing as they should. Both problems are unfortunately hard to reproduce, the adjusted test at leaset allowed me to reproduce the deadlock situation sometimes. To fix this cleanly we should probably just get rid of dynamic dbi allocation for good.
* Asserts in methods where we always expect a result.Christian Mollekopf2018-08-06
|
* Avoid creating an instance just to check for instanceChristian Mollekopf2018-08-06
|
* Calendar colorChristian Mollekopf2018-08-02
|
* Email debug serializerChristian Mollekopf2018-08-02
|
* Cleanup.Christian Mollekopf2018-08-02
|
* Fixed signed/unsigned comparisonChristian Mollekopf2018-07-30
|
* Prevent phrase searches from spanning different indexed partsChristian Mollekopf2018-07-30
|
* An adaptive query limit and a warning if we exceed itChristian Mollekopf2018-07-30
|
* Small optimizationChristian Mollekopf2018-07-30
|
* Avoid unnecessary Identifier conversions in performance ciritical code.Christian Mollekopf2018-07-28
| | | | | This fixes the performance regressions to a state where we are roughly at the same performance as pre Identifier (but not any better either).
* Fixed warningsChristian Mollekopf2018-07-28
|
* Don't print all results.Christian Mollekopf2018-07-27
|
* Debug messages and a higher limit for queriesChristian Mollekopf2018-07-27
|
* Fixed buildChristian Mollekopf2018-07-27
|
* Use Key API in SinkSHRémi Nicole2018-07-27
| | | | | | | | | | | | | | | | | | Summary: Depends on D14289 - Fixes the `sinksh inspect …` command - Introduces `isValid`, `isValidInternal` and `isValidDisplay` static functions in Key, Identifier and Revision I still have to do a more extensive search for induced bugs in other commands Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D14404
* Use Key API in ResultSetRémi Nicole2018-07-27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Depends on D14099 Notes: - Tests pass without many modifications outside of resultset.cpp/.h - `mGenerator` doesn't seem to be used? Benchmarks ========= Run benchmarks: | Develop | D14099 | This patch | | ---------------------------------- | ---------------------------------- | ---------------------------------- | | Current Rss usage [kb]: 40700 | Current Rss usage [kb]: 38564 | Current Rss usage [kb]: 39112 | | Peak Rss usage [kb]: 40700 | Peak Rss usage [kb]: 38564 | Peak Rss usage [kb]: 39112 | | Rss growth [kb]: 15920 | Rss growth [kb]: 13352 | Rss growth [kb]: 13432 | | Rss growth per entity [byte]: 3260 | Rss growth per entity [byte]: 2734 | Rss growth per entity [byte]: 2750 | | Rss without db [kb]: 29736 | Rss without db [kb]: 29248 | Rss without db [kb]: 30100 | | Percentage peak rss error: 0 | Percentage peak rss error: 0 | Percentage peak rss error: 0 | | On disk [kb]: 10788 | On disk [kb]: 9140 | On disk [kb]: 8836 | | Buffer size total [kb]: 898 | Buffer size total [kb]: 898 | Buffer size total [kb]: 898 | | Write amplification: 12.0075 | Write amplification: 10.1732 | Write amplification: 9.83485 | Test Disk Usage: | Develop | D14099 | This patch | | ----------------------------------- | ----------------------------------- | ----------------------------------- | | Free pages: 412 | Free pages: 309 | Free pages: 312 | | Total pages: 760 | Total pages: 599 | Total pages: 603 | | Used size: 1425408 | Used size: 1187840 | Used size: 1191936 | | Calculated key + value size: 856932 | Calculated key + value size: 702866 | Calculated key + value size: 702866 | | Calculated total db sizes: 970752 | Calculated total db sizes: 954368 | Calculated total db sizes: 933888 | | Main store on disk: 3112960 | Main store on disk: 2453504 | Main store on disk: 2469888 | | Total on disk: 3293184 | Total on disk: 2633728 | Total on disk: 2650112 | | Used size amplification: 1.66339 | Used size amplification: 1.68999 | Used size amplification: 1.69582 | | Write amplification: 3.63268 | Write amplification: 3.49071 | Write amplification: 3.51402 | Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D14289
* Use Key API in DataStoreQueryRémi Nicole2018-07-27
| | | | | | | | | | Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D14099
* Use Key API in indexesRémi Nicole2018-07-27
| | | | | | | | | | | | | | | | | | Summary: - Only in TypeIndex, not in Index (since we might want to store something other than identifiers as values) - We might want to do the same in the `SynchronizerStore` for localId ↔ remoteId indexes Depends on D13735 Some quick benchmarks (against develop and D13735): {F6022279} Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D13902
* New Key API in storage layerRémi Nicole2018-07-27
| | | | | | | | | | | | | | | | | Summary: - Use object oriented paradigm for Keys / Identifiers /Revisions - "Compress" keys by using byte representation of Uuids - Still some cleaning left to do - Also run some benchmarks - I'm questioning whether files other than entitystore (tests excluded) are allowed to access this API Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D13735
* Shouldn't be a warning if we expect to run into it.Christian Mollekopf2018-07-27
| | | | We always run into that when starting a resource.
* Only do partial matching (instead of wildcard) and limit expansion.Christian Mollekopf2018-07-27
|
* All xapian stuff in a central placeChristian Mollekopf2018-07-27
|
* Avoid the socket probing and move the shutdown logic intoChristian Mollekopf2018-07-25
| | | | | | | | | | | | | | | resourceaccess. The problem was (as excercised by the last test in resourcecontroltest), that in this scenario we would: * trigger a synchronization that starts the resource, and then goes into a loop trying to connecting (KAsync::wait -> singleshot timer) * trigger a shutdown that would probe for the socket, not find it, and thus do nothing. * exit the testfunction, which somehow stops qtimer processing, meaning we are stuck in KAsync::wait. For now this is fixed by simply not probing for the socket.
* FixupChristian Mollekopf2018-07-25
|
* Same fix as last commit.Christian Mollekopf2018-07-25
|
* Fixed crash.Christian Mollekopf2018-07-25
| | | | Could be triggered by running the composerviewtest in kube.
* Deal with modifications to filtered entitiesChristian Mollekopf2018-07-16
|
* Fixed the case when a dbi would leak through to a transaction where itChristian Mollekopf2018-07-12
| | | | | | shouldn't be visible yet. Was reproducible in the initial sync of the caldav resource.
* Document what copies and what doesn'tChristian Mollekopf2018-07-03
|
* Fixed yet another reduction update codepathChristian Mollekopf2018-07-02
|
* Generate globally unique messageids without leaking the hostnameChristian Mollekopf2018-07-02
|
* Contains with an empty key doesn't make much sense.Christian Mollekopf2018-06-26
|
* Only modifications have changed propertiesChristian Mollekopf2018-06-26
|
* Avoid overwriting local changes with remote modifications.Christian Mollekopf2018-06-26
| | | | | | | | | | | | | | | | | | | The case we ran into is the following: * Fetching the full payload and marking all messages of a thread as read happens simultaneously. * The local modification to mark as read gets immediately overwritten when the full payload arrives. * Eventually the modification gets replayed to the server though (and the reversal isn't because coming from the source), so on next sync the situation fixes itself. To be able to improve this we try to protect local modifications in that properties that have been modified since baseRevision (which currently isn't, but should be equal to the last to the server replayed revision) are not overwritten. This conflict resolution strategy thus always prefers local modifications. baseRevision is currently set to the current maximum revision of the store at the time when the resource creates the modification.
* Fixed the lost update scenarioChristian Mollekopf2018-06-25
| | | | | | | If we get a fetchMore right between when the revision was updated and the incrementalQuery actually running, we ended up loosing the update because the result provider ended up with a too recent revision after the additional initial query.
* Try harder to avoid storing a revision that is too high in the resultChristian Mollekopf2018-06-22
| | | | | | | | | | | | set. We might miss some updates. This should not normally ever happen if we assume that we have a transaction from start to finish of the query (the maxRevision() call should be equivalent. We do have some cornercases in our lmdb implementation that breaks transactions when new databases are opened, so we try to be extra safe this way.... Let's see if it works.
* Log query resultsChristian Mollekopf2018-06-22
|
* Fixed the thread index.Christian Mollekopf2018-06-19
| | | | | | | | * Modifications could result in index changes because we lost the threadId due to remove + add. A modify was necessary (although we can ignore it for the email case). * The ThreadIndexer would try to lookup and potentially index threads for empty parent ids, which is clearly wrong.
* Asserts, debug messages and other cleanupChristian Mollekopf2018-06-19
|
* Implement Overlap queriesRémi Nicole2018-06-19
| | | | | | | | | | | | | | | | | | | | Summary: Notes: - Introduces the concept of queries on multiple properties (which meant changing query's internals a bit) - Dates are stored as well as the "reference" in the index to allow quick filtering without fetching the whole entity - Buckets are weeks starting on Monday (guaranteed by the use of the Julian calendar) - Some size improvements are definitely possible (dates are padded numbers again, not using integer databases, Julian calendar starts at a very old date, etc.) Test Plan: Tested in querytest Reviewers: cmollekopf Reviewed By: cmollekopf Tags: #sink Differential Revision: https://phabricator.kde.org/D13477
* Debug outputChristian Mollekopf2018-06-18
|
* Reduce the maximum database size to 200MB until we have fixed the lmdbChristian Mollekopf2018-06-01
| | | | | | | | | | | issues. https://phabricator.kde.org/T8723 With 200MB we can both deal with the 200MB files on disk, and we could even load all of them (the 5 databases the resource uses), into memory. Once the open problems are resolved we should be able to bump it back to at least 20GB.
* Avoid trying to remove an item that we don't have in the model.Christian Mollekopf2018-06-01
| | | | This triggered the assert in createIndexFromId.
* CleanupChristian Mollekopf2018-05-29
|
* Reduce the maximum database size to 10GB on windows.Christian Mollekopf2018-05-29
|
* Only call the result handler if reading succeededChristian Mollekopf2018-05-29
|
* Fixed build with extra warnings.Christian Mollekopf2018-05-28
|
generated by cgit v1.2.3 (git 2.25.1) at 2025-06-23 17:48:39 +0000