Documentation

1 files changed, 38 insertions, 21 deletions

diff --git a/docs/resource.md b/docs/resource.md
index defbf9a..8c87522 100644
--- a/docs/resource.md
+++ b/docs/resource.md
@@ -4,7 +4,7 @@ The resource consists of:
 * a plugin providing the client-api facade
 * a configuration setting of the filters
 
-# Synchronizer
+## Synchronizer
 The synchronizer process is responsible for processing all commands, executing synchronizations with the source, and replaying changes to the source.
 
 Processing of commands happens in the pipeline which executes all preprocessors ebfore the entity is persisted.
@@ -16,7 +16,15 @@ The synchronizer process has the following primary components:
 * Listener: Opens a socket and listens for incoming connections. On connection all incoming commands are read and entered into command queues. Control commands (i.e. a sync) don't require persistency and are therefore processed directly.
 * Synchronization: Handles synchronization to the source, as well as change-replay to the source. The modification commands generated by the synchronization enter the command queue as well.
 
-# Preprocessors
+A resource can:
+
+* provide a full mirror of the source.
+* provide metadata for efficient access to the source.
+
+In the former case the local mirror is fully functional locally and changes can be replayed to the source once a connection is established again.
+It the latter case the resource is only functional if a connection to the source is available (which is i.e. not a problem if the source is a local maildir on disk).
+
+## Preprocessors
 Preprocessors are small processors that are guaranteed to be processed before an new/modified/deleted entity reaches storage. They can therefore be used for various tasks that need to be executed on every entity.
 
 Usecases:
@@ -33,16 +41,29 @@ The following kinds of preprocessors exist:
 
 Preprocessors are typically read-only, to i.e. not break signatures of emails. Extra flags that are accessible through the sink domain model, can therefore be stored in the local buffer of each resource.
 
-## Requirements
+### Requirements
 * A preprocessor must work with batch processing. Because batch-processing is vital for efficient writing to the database, all preprocessors have to be included in the batch processing.
 * Preprocessors need to be fast, since they directly affect how fast a message is processed by the system.
 
-## Design
+### Design
 Commands are processed in batches. Each preprocessor thus has the following workflow:
 * startBatch is called: The preprocessor can do necessary preparation steps to prepare for the batch (like starting a transaction on an external database)
 * add/modify/remove is called for every command in the batch: The preprocessor executes the desired actions.
 * endBatch is called: If the preprocessor wrote to an external database it can now commit the transaction.
 
+### Generic Preprocessors
+Most preprocessors will likely be used by several resources, and are either completely generic, or domain specific (such as only for mail).
+It is therefore desirable to have default implementations for common preprocessors that are ready to be plugged in.
+
+The domain type adaptors provide a generic interface to access most properties of the entities, on top of which generic preprocessors can be implemented.
+It is that way trivial to i.e. implement a preprocessor that populates a hierarchy index of collections.
+
+### Preprocessors generating additional entities
+A preprocessor, such as an email threading preprocessors, might generate additional entities (A thread entity is a regular entity, just like the mail that spawned the thread).
+
+In such a case the preprocessor must invoke the complete pipeline for the new entity.
+
+
 ## Indexes
 Most indexes are implemented as preprocessors to guarantee that they are always updated together with the data.
 
@@ -65,6 +86,9 @@ Index types:
     * sort indexes (i.e. sorted by date)
         * Could also be a lookup in the range index (increase date range until sufficient matches are available)
 
+### Default implementations
+Since only properties of the domain types can be queried, default implementations for commonly used indexes can be provided. These indexes are populated by generic preprocessors that use the domain-type interface to extract properties from individual entites.
+
 ### Example index implementations
 * uid lookup
     * add:
@@ -106,25 +130,14 @@ Building the index on-demand is a matter of replaying the relevant dataset and u
 
 The indexes status information can be recorded using the latest revision the index has been updated with.
 
-## Generic Preprocessors
-Most preprocessors will likely be used by several resources, and are either completely generic, or domain specific (such as only for mail).
-It is therefore desirable to have default implementations for common preprocessors that are ready to be plugged in.
-
-The domain type adaptors provide a generic interface to access most properties of the entities, on top of which generic preprocessors can be implemented.
-It is that way trivial to i.e. implement a preprocessor that populates a hierarchy index of collections.
-
-## Preprocessors generating additional entities
-A preprocessor, such as an email threading preprocessors, might generate additional entities (A thread entity is a regular entity, just like the mail that spawned the thread).
-
-In such a case the preprocessor must invoke the complete pipeline for the new entity.
-
 # Pipeline
 A pipeline is an assembly of a set of preprocessors with a defined order. A modification is always persisted at the end of the pipeline once all preprocessors have been processed.
 
-# Synchronization / Change Replay
-* The synchronization can either:
-    * Generate a full diff directly on top of the db. The diffing process can work against a single revision/snapshot (using transactions). It then generates a necessary changeset for the store.
-    * If the source supports incremental changes the changeset can directly be generated from that information.
+# Synchronization
+The synchronization can either:
+
+* Generate a full diff directly on top of the db. The diffing process can work against a single revision/snapshot (using transactions). It then generates a necessary changeset for the store.
+* If the source supports incremental changes the changeset can directly be generated from that information.
 
 The changeset is then simply inserted in the regular modification queue and processed like all other modifications. The synchronizer has to ensure only changes are replayed to the source that didn't come from it already. This is done by marking changes that don't require changereplay to the source.
 
@@ -142,8 +155,12 @@ The remoteid mapping has to be updated in two places:
 * New entities that are synchronized immediately get a localid assinged, that is then recorded together with the remoteid. This is required to be able to reference other entities directly in the command queue (i.e. for parent folders).
 * Entities created by clients get a remoteid assigned during change replay, so the entity can be recognized during the next sync.
 
+## Change Replay
+To replay local changes to the source the synchronizer replays all revisions of the store and maintains the current replay state in the synchronization store.
+Changes that already come from the source via synchronizer are not replayed to the source again.
+
 # Testing / Inspection
-Resources new to be tested, which often requires inspections into the current state of the resource. This is difficult in an asynchronous system where the whole backend logic is encapsulated in a separate process without running tests in a vastly different setup from how it will be run in production.
+Resources have to be tested, which often requires inspections into the current state of the resource. This is difficult in an asynchronous system where the whole backend logic is encapsulated in a separate process without running tests in a vastly different setup from how it will be run in production.
 
 To alleviate this inspection commands are introduced. Inspection commands are special commands that the resource processes just like all other commands, and that have the sole purpose of inspecting the current resource state. Because the command is processed with the same mechanism as other commands we can rely on ordering of commands in a way that a prior command is guaranteed to be executed once the inspection command is processed.