diff options
Diffstat (limited to 'docs')
-rw-r--r-- | docs/clientapi.md | 91 | ||||
-rw-r--r-- | docs/design.md | 2 | ||||
-rw-r--r-- | docs/resource.md | 4 | ||||
-rw-r--r-- | docs/terminology.md | 25 |
4 files changed, 65 insertions, 57 deletions
diff --git a/docs/clientapi.md b/docs/clientapi.md index a2ac18e..59998ca 100644 --- a/docs/clientapi.md +++ b/docs/clientapi.md | |||
@@ -1,18 +1,17 @@ | |||
1 | The client API consists of: | 1 | The client API consists of: |
2 | 2 | ||
3 | * a modification API for messages (Create/Modify/Delete) | 3 | * a modification API for entities (Create/Modify/Delete) |
4 | * a query API to retrieve messages | 4 | * a query API to retrieve entities |
5 | * a resource facade to abstract the resource implementation details | 5 | * a resource facade to abstract the resource implementation details |
6 | * a set of standardized domain types | 6 | * a set of standardized domain types |
7 | * a notification mechanism to be notified about changes from individual stores | 7 | * a notification mechanism to be notified about changes from individual stores |
8 | 8 | ||
9 | ## Requirements/Design goals | 9 | ## Requirements/Design goals |
10 | * zero-copy should be possible (mmap support) | 10 | * zero-copy should be possible (mmap support) |
11 | * Likely only possible until application domain until we rewrite portions of the applications | 11 | * A single copy is more realistic |
12 | * Most importantly we should hide how the data is stored (in parts, or one mmapped buffer) | 12 | * Most importantly we should hide how the data is stored (in parts, or one mmapped buffer) |
13 | * Support for mmapped buffers implies that we keep track of the lifetime of the loaded values. | 13 | * property-level on-demand loading of data |
14 | * property-level on-demand loading | 14 | * streaming support for large properties (attachments) |
15 | * streaming support for certain properties (attachments) | ||
16 | 15 | ||
17 | ## Domain Types | 16 | ## Domain Types |
18 | A set of standardized domain types is defined. This is necessary to decouple applications from resources (so a calendar can access events from all resources), and to have a "language" for queries. | 17 | A set of standardized domain types is defined. This is necessary to decouple applications from resources (so a calendar can access events from all resources), and to have a "language" for queries. |
@@ -24,42 +23,35 @@ The definition of the domain model directly affects: | |||
24 | 23 | ||
25 | The purpose of these domain types is strictly to be the interface and the types are not meant to be used by applications directly, or to be restricted by any other specifications (such as ical). By nature these types will be part of the evolving interface, and will need to be adjusted for every new property that an application must understand. | 24 | The purpose of these domain types is strictly to be the interface and the types are not meant to be used by applications directly, or to be restricted by any other specifications (such as ical). By nature these types will be part of the evolving interface, and will need to be adjusted for every new property that an application must understand. |
26 | 25 | ||
27 | ### Akonadi Domain Types | 26 | ### Application Domain Types |
28 | This is a proposed set of types that we will need to evolve into what we actually require. Hierarchical types are required to be able to query for a result set of mixed types. | 27 | This is a proposed set of types that we will need to evolve into what we actually require. Hierarchical types are required to be able to query for a result set of mixed types. |
29 | 28 | ||
30 | Items: | 29 | * Entity |
31 | 30 | * Item | |
32 | * Item | 31 | * Incidence |
33 | * incidence | 32 | * Event |
34 | * Event | 33 | * Todo |
35 | * Todo | 34 | * Journal |
35 | * Freebusy | ||
36 | * Note | ||
37 | |||
38 | * Contact | ||
39 | * Collection | ||
40 | * Mail Folder | ||
41 | * Calendar | ||
42 | * Tasklist | ||
36 | * Journal | 43 | * Journal |
37 | * Freebusy | 44 | * Contact Group |
38 | * Note | 45 | * Address Book |
39 | 46 | * Relation | |
40 | * Contact | 47 | * Tag |
41 | |||
42 | Collections: | ||
43 | |||
44 | * Collection | ||
45 | * Mail Folder | ||
46 | * Calendar | ||
47 | * Tasklist | ||
48 | * Journal | ||
49 | * Contact Group | ||
50 | * Address Book | ||
51 | |||
52 | Relations: | ||
53 | |||
54 | * Relation | ||
55 | * Tag | ||
56 | 48 | ||
57 | ## Store Facade | 49 | ## Store Facade |
58 | The store is always accessed through a store specific facade, which hides: | 50 | The store is always accessed through a store specific facade, which hides: |
59 | * store access (one store could use a database, and another one plain files) | 51 | * store access (one store could use a database, and another one plain files) |
60 | * message type (flatbuffers, ...) | 52 | * message type (flatbuffers, ...) |
61 | * indexes | 53 | * indexes |
62 | * syncronizer communication | 54 | * synchronizer communication |
63 | * notifications | 55 | * notifications |
64 | 56 | ||
65 | This abstraction layer allows each resource to separately define how data is stored and retrieved. Therefore tradeoffs can be defined to suit the expected access patters or structure of source data. Further it allows individual resources to choose different technologies as suitable. Logic can still be shared among resources, while keeping the maintenance effort reasonable, by providing default implementations that are suitable for most workloads. | 57 | This abstraction layer allows each resource to separately define how data is stored and retrieved. Therefore tradeoffs can be defined to suit the expected access patters or structure of source data. Further it allows individual resources to choose different technologies as suitable. Logic can still be shared among resources, while keeping the maintenance effort reasonable, by providing default implementations that are suitable for most workloads. |
@@ -67,7 +59,7 @@ This abstraction layer allows each resource to separately define how data is sto | |||
67 | Because the facade also implements querying of indexes, a resource my use server-side searching to fullfill the query, and fallback to local searches when the server is not available. | 59 | Because the facade also implements querying of indexes, a resource my use server-side searching to fullfill the query, and fallback to local searches when the server is not available. |
68 | 60 | ||
69 | ## Modifications | 61 | ## Modifications |
70 | Modifications are stored by the client sending modification commands to the syncronizer. The syncronizer is responsible for ensuring that modification are not lost and eventually persistet. A small window exists therefore where a modification is transferred to the syncronizer where a modifications can get lost. | 62 | Modifications are stored by the client sending modification commands to the synchronizer. The synchronizer is responsible for ensuring that modification are not lost and eventually persistet. A small window exists therefore where a modification is transferred to the synchronizer where a modifications can get lost. |
71 | 63 | ||
72 | The API consists of the following calls: | 64 | The API consists of the following calls: |
73 | 65 | ||
@@ -75,23 +67,23 @@ The API consists of the following calls: | |||
75 | * modify(domainObject, resource) | 67 | * modify(domainObject, resource) |
76 | * remove(domainObject, resource) | 68 | * remove(domainObject, resource) |
77 | 69 | ||
78 | The changeset can be recorded by the domain object adapter while the properties are set, and are then sent to the syncronizer once modify is called. | 70 | The changeset can be recorded by the domain object adapter while the properties are set, and are then sent to the synchronizer once modify is called. |
79 | 71 | ||
80 | Each modification is associated with a specific revision, which allows the syncronizer to do automatic conflict resolution. | 72 | Each modification is associated with a specific revision, which allows the synchronizer to do automatic conflict resolution. |
81 | 73 | ||
82 | ### Conflict Resolution | 74 | ### Conflict Resolution |
83 | Conflicts can occur at two points in the client: | 75 | Conflicts can occur at two points in the client: |
84 | 76 | ||
85 | * While i.e. an editor is open and we receive an update for the same entity | 77 | * While i.e. an editor is open and we receive an update for the same entity |
86 | * After a modification is sent to the syncronizer but before it's processed | 78 | * After a modification is sent to the synchronizer but before it's processed |
87 | 79 | ||
88 | In the first case the client is repsonsible to resolve the conflict, in the latter case it's the syncronizer's responsibility. | 80 | In the first case the client is repsonsible to resolve the conflict, in the latter case it's the synchronizer's responsibility. |
89 | A small window exists where the client has already started the modification (i.e. command is in socket), and a notification has not yet arrived that the same entity has been changed. In such a case the syncronizer may reject the modification because it has the revision the modification refers to no longer available. | 81 | A small window exists where the client has already started the modification (i.e. command is in socket), and a notification has not yet arrived that the same entity has been changed. In such a case the synchronizer may reject the modification because it has the revision the modification refers to no longer available. |
90 | 82 | ||
91 | This design allows the syncronizer to be in control of the revisions, and keeps it from having to wait for all clients to update until it can drop revisions. | 83 | This design allows the synchronizer to be in control of the revisions, and keeps it from having to wait for all clients to update until it can drop revisions. |
92 | 84 | ||
93 | ## Query System | 85 | ## Query System |
94 | The query system should allow for efficient retrieval for just the amount of data required by the client. Efficient querying will be supported by the indexes povided by the resources. | 86 | The query system should allow for efficient retrieval for just the amount of data required by the client. Efficient querying will be supported by the indexes provided by the resources. |
95 | 87 | ||
96 | The query always retrieves a set of entities matching the query, while not necessarily all properties of the entity need to be populated. | 88 | The query always retrieves a set of entities matching the query, while not necessarily all properties of the entity need to be populated. |
97 | 89 | ||
@@ -187,12 +179,12 @@ Generic: | |||
187 | ### Lazy Loading ### | 179 | ### Lazy Loading ### |
188 | The system provides property-level lazy loading. This allows i.e. to defer downloading of attachments until the attachments is accessed, at the expense of having to have access to the source (which could be connected via internet). | 180 | The system provides property-level lazy loading. This allows i.e. to defer downloading of attachments until the attachments is accessed, at the expense of having to have access to the source (which could be connected via internet). |
189 | 181 | ||
190 | To achieve this, the query system must check for the availability of all requested properties on all matched entities. If a property is not available the a command should be sent to the synchronizer to retrieve said properties. Once all properties are available the query can complete. | 182 | To achieve this, the query system must check for the availability of all requested properties on all matched entities. If a property is not available, a command must be sent to the synchronizer to retrieve said properties. Once all properties are available the query can complete. |
191 | 183 | ||
192 | Note: We should perhaps define a minimum set of properties that *must* be available. Otherwise local search will not work. On the other hand, if a resource implements server-side search, it may not care if local search doesn't work. | 184 | Note: We should perhaps define a minimum set of properties that *must* be available. Otherwise local search will not work. On the other hand, if a resource implements server-side search, it may not care if local search doesn't work. |
193 | 185 | ||
194 | ### Data streaming ### | 186 | ### Data streaming ### |
195 | Large objects such as attachments should be streamable. An API that allows to retrieve a single property of a defined entity in a streamable fashion is probably enough. | 187 | Large properties such as attachments should be streamable. An API that allows to retrieve a single property of a defined entity in a streamable fashion is probably enough. |
196 | 188 | ||
197 | ### Indexes ### | 189 | ### Indexes ### |
198 | Since only properties of the domain types can be queried, default implementations for commonly used indexes can be provided. These indexes are populated by generic preprocessors that use the domain-type interface to extract properties from individual entites. | 190 | Since only properties of the domain types can be queried, default implementations for commonly used indexes can be provided. These indexes are populated by generic preprocessors that use the domain-type interface to extract properties from individual entites. |
@@ -200,14 +192,5 @@ Since only properties of the domain types can be queried, default implementation | |||
200 | ## Notifications ## | 192 | ## Notifications ## |
201 | A notification mechanism is required to inform clients about changes. Running queries will automatically update the result-set if a notification is received. | 193 | A notification mechanism is required to inform clients about changes. Running queries will automatically update the result-set if a notification is received. |
202 | 194 | ||
203 | A notification constist of: | 195 | Note: A notification could supply a hint on what changed, allowing clients to ignore revisions with irrelevant changes. |
204 | 196 | A running query can do all of that transparently behind the scenes. Note that the hints should indeed only hint what has changed, and not supply the actual changeset. These hints should be tailored to what we see as useful, and must therefore be easy to modify. | |
205 | * The latest revision of the store | ||
206 | * A hint what properties changed | ||
207 | |||
208 | The revision allows the client to only fetch the data that changed. | ||
209 | The hint allows the client to avoid fetching that it's not interested in. | ||
210 | A running query can do all of that transparently behind the scenes. | ||
211 | |||
212 | Note that the hints should indeed only hint what has changed, and not supply the actual changeset. These hints should be tailored to what we see as useful, and must therefore be easy to modify. | ||
213 | |||
diff --git a/docs/design.md b/docs/design.md index 9b074b1..59cdf74 100644 --- a/docs/design.md +++ b/docs/design.md | |||
@@ -89,7 +89,7 @@ The change replay is based on the revisions in the store. Clients (and also the | |||
89 | * Shared domain types as common interface for client applications | 89 | * Shared domain types as common interface for client applications |
90 | * `-` yet another abstraction layer that requires translation to other layers and maintenance | 90 | * `-` yet another abstraction layer that requires translation to other layers and maintenance |
91 | * `+` decoupling of domain logic from data access | 91 | * `+` decoupling of domain logic from data access |
92 | * `+` allows to evolve types according to needs (not coupled to specific applicatoins domain types) | 92 | * `+` allows to evolve types according to needs (not coupled to specific application domain types) |
93 | 93 | ||
94 | # Risks | 94 | # Risks |
95 | * key-value store does not perform with large amounts of data | 95 | * key-value store does not perform with large amounts of data |
diff --git a/docs/resource.md b/docs/resource.md index 847831d..bba4dbb 100644 --- a/docs/resource.md +++ b/docs/resource.md | |||
@@ -4,9 +4,9 @@ The resource consists of: | |||
4 | * a plugin providing the client-api facade | 4 | * a plugin providing the client-api facade |
5 | * a configuration setting up the filters | 5 | * a configuration setting up the filters |
6 | 6 | ||
7 | # Syncronizer | 7 | # Synchronizer |
8 | * The synchronization can either: | 8 | * The synchronization can either: |
9 | * Generate a full diff directly on top of the db. The diffing process can work against a single revision, and could even stop writing other changes to disk while the process is ongoing (but doesn't have to due to the revision). It then generates a necessary changeset for the store. | 9 | * Generate a full diff directly on top of the db. The diffing process can work against a single revision/snapshot (using transactions). It then generates a necessary changeset for the store. |
10 | * If the source supports incremental changes the changeset can directly be generated from that information. | 10 | * If the source supports incremental changes the changeset can directly be generated from that information. |
11 | 11 | ||
12 | The changeset is then simply inserted in the regular modification queue and processed like all other modifications. | 12 | The changeset is then simply inserted in the regular modification queue and processed like all other modifications. |
diff --git a/docs/terminology.md b/docs/terminology.md new file mode 100644 index 0000000..4b49f2d --- /dev/null +++ b/docs/terminology.md | |||
@@ -0,0 +1,25 @@ | |||
1 | Consistent, agreed-upon terminology is key to being able to work together efficiently on a complex software project such as this, particularly as we are building on the earlier Akonadi efforts which itself has established terminology. You can find the current glossary here. | ||
2 | |||
3 | It is recommended to familiarize yourself with the terms before going further into the design of Akonadi-next as it will make things clearer for you and easier to ask the questions you have in a way that others will understand immediately. | ||
4 | |||
5 | ## Glossary of Akonadi-Next Terms | ||
6 | * akonadi1: The current akonadi implementation that uses a central server and an SQL database | ||
7 | * akonadi-next: This is the codename for the project. In the long run this is supposed to be folded into regular akonadi, so we will never release a product called akonadi-next. | ||
8 | * client: any application which accesses data using akonadi | ||
9 | * entity: The atomic unit for a given type of data. An email is an entity; an email folder is an entity; a calendar event is an entity; a contact is an entity; etc. Different kinds of entities may have their own data structure, but conceptually they are equivalent in most other ways. | ||
10 | * revision: A version of the store. One entity may have multiple revisions in a store, representing (for instance) the local state and the synchronized state of the entity. | ||
11 | * source: The canonical data set, which may be a remote IMAP server, a local iCal file, a local maildir, etc. | ||
12 | * store: The local, persistent (e.g. on disk) record of entities belonging to a source. This may be a full mirror of the data or simply metadata, a detail left up to the resource. The format of the data in the store is defined by the resource that owns it. | ||
13 | * resource: A plugin which provides client command processing, a store facade and synchronization for a given type of store. The resource also manages the configuration for a given source including server settings, local paths, etc. | ||
14 | * store facade: An object provided by resources which provides transformations between domain objects and the store. | ||
15 | * synchronizer: The operating system process responsible for overseeing the process of modifying and synchronizing a store. To accomplish this, a synchronizer loads the correct resource plugin, manages pipelines and handles client communication. One synchronizer is created for each source that is accessed by clients; these processes are shared by all clients. | ||
16 | * Preprocessor: A component that takes an entity and performs some modification of it (e.g. changes the folder an email is in) or processes it in some way (e.g. indexes it) | ||
17 | * pipeline: A run-time definable set of filters which are applied to an entity after a resource has performed a specific kind of function on it (add, update, remove) | ||
18 | * query: A declarative method for requesting entities from one or more sources that match a given set of constraints | ||
19 | * command: Clients request modifications, additions and deletions to the store by sending commands to a synchronizer for processing | ||
20 | * command queue: A queue of commands kept by the synchronizer to ensure durability and, when necessary, replayability | ||
21 | * notification: A message sent from a synchronizer to inform the client of a change in the store | ||
22 | * domain object: An application domain object, i.e. an event. | ||
23 | * domain type: The type of a domain object. i.e. Akonadi2::ApplicationDomain::Event | ||
24 | * buffer: The buffers used by the resources to persist data in the datastore. | ||
25 | * buffer type: The individual buffer types as specified by the resource. These are internal types that don't necessarily have a 1:1 mapping to the domain types, although that is the default case that the default implementations expect. | ||