5 files changed, 105 insertions, 96 deletions
diff --git a/docs/design.md b/docs/design.md
index 4451b49..499f527 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -1,44 +1,7 @@
-# Design Goals
-## Axioms
-1. Personal information is stored in multiple sources (address books, email stores, calendar files, ...)
-2. These sources may local, remote or a mix of local and remote
-## Requirements
-1. Local mirrors of these sources must be available to 1..N local clients simultaneously
-2. Local clients must be able to make (or at least request) changes to the data in the local mirrors
-3. Local mirrors must be usable without network, even if the source is remote
-4. Local mirrors must be able to syncronoize local changes to their sources (local or remote)
-5. Local mirrors must be able to syncronize remote changes and propagate those to local clients
-6. Content must be searchable by a number of terms (dates, identities, body text ...)
-7. This must all run with acceptable performance on a moderate consumer-grade desktop system
-Nice to haves:
-1. As-close-to-zero-copy-as-possible for data
-2. Simple change notification semantics
-3. Resource-specific syncronization techniques
-4. Data agnostic storage
-Immediate goals:
-1. Ease development of new features in existing resources
-2. Ease maintenance of existing resources
-3. Make adding new resources easy
-4. Make adding new types of data or data relations easy
-5. Improve performance relative to existing Akonadi implementation
-Long-term goals:
-1. Project view: given a query, show all items in all stores that match that query easily and quickly
-Implications of the above:
-* Local mirrors must support multi-reader, but are probably best served with single-writer semantics as this simplifies both local change recording as well as remote synchronization by keeping it in one process which can process write requests (local or remote) in sequential fashion.
-* There is no requirement for a central server if the readers can concurrently access the local mirror directly
-* A storage system which requires a schema (e.g. relational databases) are a poor fit given the desire for data agnosticism and low memory copying
 # Overview
+Sink is a data access layer that additionally handles synchronization with external sources and indexing of data for efficient queries.
 ## Client API
 The client facing API hides all Sink internals from the applications and emulates a unified store that provides data through a standardized interface. 
 This allows applications to transparently use various data sources with various data source formats.
@@ -63,39 +26,3 @@ The change replay is based on the revisions in the store. Clients (as well as al
 ### Preprocessor pipeline
 Each resource has an internal pipeline of preprocessors that can be used for tasks such as indexing or filtering. The pipeline guarantees that the preprocessor steps are executed before the entity is persisted.
-# Tradeoffs/Design Decisions
-* Key-Value store instead of relational
-    * `+` Schemaless, easier to evolve
-    * `-` No need to fully normalize the data in order to make it queriable. And without full normalization SQL is not really useful and bad performance wise. 
-    * `-` We need to maintain our own indexes
-* Individual store per resource
-    * Storage format defined by resource individually
-        * `-` Each resource needs to define it's own schema
-        * `+` Resources can adjust storage format to map well on what it has to synchronize
-        * `+` Synchronization state can directly be embedded into messages
-    * `+` Individual resources could switch to another store technology
-    * `+` Easier maintenance
-    * `+` Resource is only responsible for it's own store and doesn't accidentaly break another resources store
-    * `-` Inter`-`resource moves are both more complicated and more expensive from a client perspective
-    * `+` Inter`-`resource moves become simple additions and removals from a resource perspective
-    * `-` No system`-`wide unique id per message (only resource/id tuple identifies a message uniquely) 
-    * `+` Stores can work fully concurrently (also for writing)
-* Indexes defined and maintained by resources
-    * `-` Relational queries accross resources are expensive (depending on the query perhaps not even feasible)
-    * `-` Each resource needs to define it's own set of indexes
-    * `+` Flexible design as it allows to change indexes on a per resource level
-    * `+` Indexes can be optimized towards resources main usecases
-    * `+` Indexes can be shared with the source (IMAP serverside threading)
-* Shared domain types as common interface for client applications
-    * `-` yet another abstraction layer that requires translation to other layers and maintenance
-    * `+` decoupling of domain logic from data access
-    * `+` allows to evolve types according to needs (not coupled to specific application domain types)
-# Risks
-* key-value store does not perform with large amounts of data
-* query performance is not sufficient
-* turnaround time for modifications is too high to feel responsive
-* design turns out similarly complex as Akonadi
diff --git a/docs/designgoals.md b/docs/designgoals.md
new file mode 100644
index 0000000..4ffeeac
--- /dev/null
+++ b/docs/designgoals.md
@@ -0,0 +1,39 @@
+# Design Goals
+## Axioms
+1. Personal information is stored in multiple sources (address books, email stores, calendar files, ...)
+2. These sources may local, remote or a mix of local and remote
+## Requirements
+1. Local mirrors of these sources must be available to 1..N local clients simultaneously
+2. Local clients must be able to make (or at least request) changes to the data in the local mirrors
+3. Local mirrors must be usable without network, even if the source is remote
+4. Local mirrors must be able to syncronoize local changes to their sources (local or remote)
+5. Local mirrors must be able to syncronize remote changes and propagate those to local clients
+6. Content must be searchable by a number of terms (dates, identities, body text ...)
+7. This must all run with acceptable performance on a moderate consumer-grade desktop system
+Nice to haves:
+1. As-close-to-zero-copy-as-possible for data
+2. Simple change notification semantics
+3. Resource-specific syncronization techniques
+4. Data agnostic storage
+Immediate goals:
+1. Ease development of new features in existing resources
+2. Ease maintenance of existing resources
+3. Make adding new resources easy
+4. Make adding new types of data or data relations easy
+5. Improve performance relative to existing Akonadi implementation
+Long-term goals:
+1. Project view: given a query, show all items in all stores that match that query easily and quickly
+Implications of the above:
+* Local mirrors must support multi-reader, but are probably best served with single-writer semantics as this simplifies both local change recording as well as remote synchronization by keeping it in one process which can process write requests (local or remote) in sequential fashion.
+* There is no requirement for a central server if the readers can concurrently access the local mirror directly
+* A storage system which requires a schema (e.g. relational databases) are a poor fit given the desire for data agnosticism and low memory copying
diff --git a/docs/index.md b/docs/index.md
index 3019cfd..36e309c 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,15 +1,18 @@
-# Index
+Sink is a data access layer handling synchronization, caching and indexing.
-* Design
-    * Design Goals
+Discussion of the code should be done on the kde-pim at kde.org mailing list
-    * Overview
+or in #kontact on IRC.
-    * Client API
-    * Storage
+Note that all feature development should happen in feature branches, and that
-    * Resource
+the mainline development branch is "develop". Master is for releases. It is
-    * Facade
+recommended (though not required) to use the "git flow" tools to make branched
-    * Logging
+development easy (and easy for others to coordinate with):
-* Extending Akoandi Next
-    * Steps to add support for new types
+    https://github.com/nvie/gitflow
-    * Steps for adding support for a type to a resource
+For further information see:
+    https://phabricator.kde.org/project/view/5/
 # Documentation
 This documentation is built using [mkdocs.org](http://mkdocs.org).
diff --git a/docs/tradeoffs.md b/docs/tradeoffs.md
new file mode 100644
index 0000000..d0e32c1
--- /dev/null
+++ b/docs/tradeoffs.md
@@ -0,0 +1,36 @@
+# Tradeoffs/Design Decisions
+* Key-Value store instead of relational
+    * `+` Schemaless, easier to evolve
+    * `-` No need to fully normalize the data in order to make it queriable. And without full normalization SQL is not really useful and bad performance wise. 
+    * `-` We need to maintain our own indexes
+* Individual store per resource
+    * Storage format defined by resource individually
+        * `-` Each resource needs to define it's own schema
+        * `+` Resources can adjust storage format to map well on what it has to synchronize
+        * `+` Synchronization state can directly be embedded into messages
+    * `+` Individual resources could switch to another store technology
+    * `+` Easier maintenance
+    * `+` Resource is only responsible for it's own store and doesn't accidentaly break another resources store
+    * `-` Inter`-`resource moves are both more complicated and more expensive from a client perspective
+    * `+` Inter`-`resource moves become simple additions and removals from a resource perspective
+    * `-` No system`-`wide unique id per message (only resource/id tuple identifies a message uniquely) 
+    * `+` Stores can work fully concurrently (also for writing)
+* Indexes defined and maintained by resources
+    * `-` Relational queries accross resources are expensive (depending on the query perhaps not even feasible)
+    * `-` Each resource needs to define it's own set of indexes
+    * `+` Flexible design as it allows to change indexes on a per resource level
+    * `+` Indexes can be optimized towards resources main usecases
+    * `+` Indexes can be shared with the source (IMAP serverside threading)
+* Shared domain types as common interface for client applications
+    * `-` yet another abstraction layer that requires translation to other layers and maintenance
+    * `+` decoupling of domain logic from data access
+    * `+` allows to evolve types according to needs (not coupled to specific application domain types)
+# Risks
+* key-value store does not perform with large amounts of data
+* query performance is not sufficient
+* turnaround time for modifications is too high to feel responsive
+* design turns out similarly complex as Akonadi
diff --git a/mkdocs.yml b/mkdocs.yml
index 21f9471..9f71214 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -1,14 +1,18 @@
 site_name: Sink
 pages:
 - Home: index.md
- Terminology: terminology.md
+- Design:
- Overview: design.md
+    - Terminology: terminology.md
- Resource: resource.md
+    - "Design Goals": designgoals.md
- Storage: storage.md
+    - Overview: design.md
- Logging: logging.md
+    - Resource: resource.md
- "Client API": clientapi.md
+    - Storage: storage.md
- sinksh: sinksh.md
+    - Logging: logging.md
- "Application Domain Types": applicationdomaintypes.md
+    - "Client API": clientapi.md
- "Extending Sink": extending.md
+    - "Application Domain Types": applicationdomaintypes.md
+    - "Tradeoffs and Design Decisions": tradeoffs.md
+- Development:
+    - "Extending Sink": extending.md
 - Building: building.md
+- "Sink Shell": sinksh.md
 theme: readthedocs

diff --git a/docs/design.md b/docs/design.md index 4451b49..499f527 100644 --- a/docs/design.md +++ b/docs/design.md
@@ -1,44 +1,7 @@
1	# Design Goals
2	## Axioms
3	1. Personal information is stored in multiple sources (address books, email stores, calendar files, ...)
4	2. These sources may local, remote or a mix of local and remote
5
6	## Requirements
7	1. Local mirrors of these sources must be available to 1..N local clients simultaneously
8	2. Local clients must be able to make (or at least request) changes to the data in the local mirrors
9	3. Local mirrors must be usable without network, even if the source is remote
10	4. Local mirrors must be able to syncronoize local changes to their sources (local or remote)
11	5. Local mirrors must be able to syncronize remote changes and propagate those to local clients
12	6. Content must be searchable by a number of terms (dates, identities, body text ...)
13	7. This must all run with acceptable performance on a moderate consumer-grade desktop system
14
15	Nice to haves:
16
17	1. As-close-to-zero-copy-as-possible for data
18	2. Simple change notification semantics
19	3. Resource-specific syncronization techniques
20	4. Data agnostic storage
21
22	Immediate goals:
23
24	1. Ease development of new features in existing resources
25	2. Ease maintenance of existing resources
26	3. Make adding new resources easy
27	4. Make adding new types of data or data relations easy
28	5. Improve performance relative to existing Akonadi implementation
29
30	Long-term goals:
31
32	1. Project view: given a query, show all items in all stores that match that query easily and quickly
33
34	Implications of the above:
35
36	* Local mirrors must support multi-reader, but are probably best served with single-writer semantics as this simplifies both local change recording as well as remote synchronization by keeping it in one process which can process write requests (local or remote) in sequential fashion.
37	* There is no requirement for a central server if the readers can concurrently access the local mirror directly
38	* A storage system which requires a schema (e.g. relational databases) are a poor fit given the desire for data agnosticism and low memory copying
39
40	# Overview	1	# Overview
41		2
		3	Sink is a data access layer that additionally handles synchronization with external sources and indexing of data for efficient queries.
		4
42	## Client API	5	## Client API
43	The client facing API hides all Sink internals from the applications and emulates a unified store that provides data through a standardized interface.	6	The client facing API hides all Sink internals from the applications and emulates a unified store that provides data through a standardized interface.
44	This allows applications to transparently use various data sources with various data source formats.	7	This allows applications to transparently use various data sources with various data source formats.
@@ -63,39 +26,3 @@ The change replay is based on the revisions in the store. Clients (as well as al
63	### Preprocessor pipeline	26	### Preprocessor pipeline
64	Each resource has an internal pipeline of preprocessors that can be used for tasks such as indexing or filtering. The pipeline guarantees that the preprocessor steps are executed before the entity is persisted.	27	Each resource has an internal pipeline of preprocessors that can be used for tasks such as indexing or filtering. The pipeline guarantees that the preprocessor steps are executed before the entity is persisted.
65		28
66	# Tradeoffs/Design Decisions
67	* Key-Value store instead of relational
68	* `+` Schemaless, easier to evolve
69	* `-` No need to fully normalize the data in order to make it queriable. And without full normalization SQL is not really useful and bad performance wise.
70	* `-` We need to maintain our own indexes
71
72	* Individual store per resource
73	* Storage format defined by resource individually
74	* `-` Each resource needs to define it's own schema
75	* `+` Resources can adjust storage format to map well on what it has to synchronize
76	* `+` Synchronization state can directly be embedded into messages
77	* `+` Individual resources could switch to another store technology
78	* `+` Easier maintenance
79	* `+` Resource is only responsible for it's own store and doesn't accidentaly break another resources store
80	* `-` Inter`-`resource moves are both more complicated and more expensive from a client perspective
81	* `+` Inter`-`resource moves become simple additions and removals from a resource perspective
82	* `-` No system`-`wide unique id per message (only resource/id tuple identifies a message uniquely)
83	* `+` Stores can work fully concurrently (also for writing)
84
85	* Indexes defined and maintained by resources
86	* `-` Relational queries accross resources are expensive (depending on the query perhaps not even feasible)
87	* `-` Each resource needs to define it's own set of indexes
88	* `+` Flexible design as it allows to change indexes on a per resource level
89	* `+` Indexes can be optimized towards resources main usecases
90	* `+` Indexes can be shared with the source (IMAP serverside threading)
91
92	* Shared domain types as common interface for client applications
93	* `-` yet another abstraction layer that requires translation to other layers and maintenance
94	* `+` decoupling of domain logic from data access
95	* `+` allows to evolve types according to needs (not coupled to specific application domain types)
96
97	# Risks
98	* key-value store does not perform with large amounts of data
99	* query performance is not sufficient
100	* turnaround time for modifications is too high to feel responsive
101	* design turns out similarly complex as Akonadi


diff --git a/docs/designgoals.md b/docs/designgoals.md new file mode 100644 index 0000000..4ffeeac --- /dev/null +++ b/docs/designgoals.md
@@ -0,0 +1,39 @@
		1	# Design Goals
		2	## Axioms
		3	1. Personal information is stored in multiple sources (address books, email stores, calendar files, ...)
		4	2. These sources may local, remote or a mix of local and remote
		5
		6	## Requirements
		7	1. Local mirrors of these sources must be available to 1..N local clients simultaneously
		8	2. Local clients must be able to make (or at least request) changes to the data in the local mirrors
		9	3. Local mirrors must be usable without network, even if the source is remote
		10	4. Local mirrors must be able to syncronoize local changes to their sources (local or remote)
		11	5. Local mirrors must be able to syncronize remote changes and propagate those to local clients
		12	6. Content must be searchable by a number of terms (dates, identities, body text ...)
		13	7. This must all run with acceptable performance on a moderate consumer-grade desktop system
		14
		15	Nice to haves:
		16
		17	1. As-close-to-zero-copy-as-possible for data
		18	2. Simple change notification semantics
		19	3. Resource-specific syncronization techniques
		20	4. Data agnostic storage
		21
		22	Immediate goals:
		23
		24	1. Ease development of new features in existing resources
		25	2. Ease maintenance of existing resources
		26	3. Make adding new resources easy
		27	4. Make adding new types of data or data relations easy
		28	5. Improve performance relative to existing Akonadi implementation
		29
		30	Long-term goals:
		31
		32	1. Project view: given a query, show all items in all stores that match that query easily and quickly
		33
		34	Implications of the above:
		35
		36	* Local mirrors must support multi-reader, but are probably best served with single-writer semantics as this simplifies both local change recording as well as remote synchronization by keeping it in one process which can process write requests (local or remote) in sequential fashion.
		37	* There is no requirement for a central server if the readers can concurrently access the local mirror directly
		38	* A storage system which requires a schema (e.g. relational databases) are a poor fit given the desire for data agnosticism and low memory copying
		39


diff --git a/docs/index.md b/docs/index.md index 3019cfd..36e309c 100644 --- a/docs/index.md +++ b/docs/index.md
@@ -1,15 +1,18 @@
1	# Index	1	Sink is a data access layer handling synchronization, caching and indexing.
2	* Design	2
3	* Design Goals	3	Discussion of the code should be done on the kde-pim at kde.org mailing list
4	* Overview	4	or in #kontact on IRC.
5	* Client API	5
6	* Storage	6	Note that all feature development should happen in feature branches, and that
7	* Resource	7	the mainline development branch is "develop". Master is for releases. It is
8	* Facade	8	recommended (though not required) to use the "git flow" tools to make branched
9	* Logging	9	development easy (and easy for others to coordinate with):
10	* Extending Akoandi Next	10
11	* Steps to add support for new types	11	https://github.com/nvie/gitflow
12	* Steps for adding support for a type to a resource	12
		13	For further information see:
		14
		15	https://phabricator.kde.org/project/view/5/
13		16
14	# Documentation	17	# Documentation
15	This documentation is built using [mkdocs.org](http://mkdocs.org).	18	This documentation is built using [mkdocs.org](http://mkdocs.org).


diff --git a/docs/tradeoffs.md b/docs/tradeoffs.md new file mode 100644 index 0000000..d0e32c1 --- /dev/null +++ b/docs/tradeoffs.md
@@ -0,0 +1,36 @@
		1	# Tradeoffs/Design Decisions
		2	* Key-Value store instead of relational
		3	* `+` Schemaless, easier to evolve
		4	* `-` No need to fully normalize the data in order to make it queriable. And without full normalization SQL is not really useful and bad performance wise.
		5	* `-` We need to maintain our own indexes
		6
		7	* Individual store per resource
		8	* Storage format defined by resource individually
		9	* `-` Each resource needs to define it's own schema
		10	* `+` Resources can adjust storage format to map well on what it has to synchronize
		11	* `+` Synchronization state can directly be embedded into messages
		12	* `+` Individual resources could switch to another store technology
		13	* `+` Easier maintenance
		14	* `+` Resource is only responsible for it's own store and doesn't accidentaly break another resources store
		15	* `-` Inter`-`resource moves are both more complicated and more expensive from a client perspective
		16	* `+` Inter`-`resource moves become simple additions and removals from a resource perspective
		17	* `-` No system`-`wide unique id per message (only resource/id tuple identifies a message uniquely)
		18	* `+` Stores can work fully concurrently (also for writing)
		19
		20	* Indexes defined and maintained by resources
		21	* `-` Relational queries accross resources are expensive (depending on the query perhaps not even feasible)
		22	* `-` Each resource needs to define it's own set of indexes
		23	* `+` Flexible design as it allows to change indexes on a per resource level
		24	* `+` Indexes can be optimized towards resources main usecases
		25	* `+` Indexes can be shared with the source (IMAP serverside threading)
		26
		27	* Shared domain types as common interface for client applications
		28	* `-` yet another abstraction layer that requires translation to other layers and maintenance
		29	* `+` decoupling of domain logic from data access
		30	* `+` allows to evolve types according to needs (not coupled to specific application domain types)
		31
		32	# Risks
		33	* key-value store does not perform with large amounts of data
		34	* query performance is not sufficient
		35	* turnaround time for modifications is too high to feel responsive
		36	* design turns out similarly complex as Akonadi


diff --git a/mkdocs.yml b/mkdocs.yml index 21f9471..9f71214 100644 --- a/mkdocs.yml +++ b/mkdocs.yml
@@ -1,14 +1,18 @@
1	site_name: Sink	1	site_name: Sink
2	pages:	2	pages:
3	- Home: index.md	3	- Home: index.md
4	- Terminology: terminology.md	4	- Design:
5	- Overview: design.md	5	- Terminology: terminology.md
6	- Resource: resource.md	6	- "Design Goals": designgoals.md
7	- Storage: storage.md	7	- Overview: design.md
8	- Logging: logging.md	8	- Resource: resource.md
9	- "Client API": clientapi.md	9	- Storage: storage.md
10	- sinksh: sinksh.md	10	- Logging: logging.md
11	- "Application Domain Types": applicationdomaintypes.md	11	- "Client API": clientapi.md
12	- "Extending Sink": extending.md	12	- "Application Domain Types": applicationdomaintypes.md
		13	- "Tradeoffs and Design Decisions": tradeoffs.md
		14	- Development:
		15	- "Extending Sink": extending.md
13	- Building: building.md	16	- Building: building.md
		17	- "Sink Shell": sinksh.md
14	theme: readthedocs	18	theme: readthedocs