Event Sourcing PoC backend
System name used: Federated zone event store -> F.Z.E.S. -> F6 F6 is the collection of components that make up the event store.
Concept
Modeling data surrounding human more closely
The current status quo is storing state and history. This approach has some issues when historical data has to be reinterpreted.
In this proof of concept a system is built to explore and demonstrate an event based alternative. We want to be able to:
- Not save the current state, but construct it from events
- Time travel to see what the state would be at any point in time
- To change or remove past events, and their effects on legal consequences of those events
- To build views on the data based on need of the consumers
- Keep data separated by responsibility / ownership of events
- Make automatic systems able to react on events
Federation
Separated based on responsibilities / ownership is done by federating by organisation and zone.
We see municipalities as organisations, and divisions of the municipalities as zones.
An example of this can be seen below:
C4Context Boundary(b1, "F6 system", "Federated zone event store system") { Boundary(b2, "Gemeente Zonnendaal", "organisation") { System(SystemA, "Burgelijke stand") System(SystemB, "Adres huishouding") } Boundary(b3, "Gemeente Woudendijk", "organisation") { System(SystemC, "Burgelijke stand") System(SystemD, "Adres huishouding") } }
Events
Event flow
To design the system we have made an example of the event flow involved in the birth of a human
.
Legend:
- pink: facts
- yellow: actions / real world events
- purple: event claim
- blue: event claim validation
Event data structure
We expose events as follows:
struct EventValue {
id: Uuid,
created_at: DateTime<Utc>,
occurred_at: DateTime<Utc>,
object_ids: Vec<Uuid>,
event_type: String,
event_version: String,
event_kind: EventKind,
event_source_id: Uuid,
event_source_type: String,
event_reference: Vec<Uuid>,
event_data: Value,
}
Id
Uuid used to identify the event
Object ids
- Uuids used to identify the objects involved in the event, in this PoC
humans
. humans
do not have to be born before an id for them may be createdhumans
do not have to be citizens
Event type
Type of the event (e.g. Geboorte
orVerhuizing
)
Event version
Events are versioned so that changes to their format do not require data migrations
Event kind
- Claim: A claim that an event has occurred, claims do not have any legal consequences.
- Correction: A claim that a past event contained incorrect data
- Validation: A claim that is verified by a civil servant, within the civil servants domain of responsibility. Validation must be linked to the
Claim
event that is being validated.
Event source id
The uuid of the event source
creating event, intended for use in audits
Event source type
The type of authority (burger
, ouder
, ambtenaar
) that the event source
creating the event has
Event reference
If the event is based on one or more previous events, this must contain the ids of those events
Event data
Event specific data, the data format is set in the write_config
file, read from the write_config_path
given to the write server
Material vs Formal history
! todo: improve description in associated issue
Material history: what has changed in reality Formal history: what has changed in registry
Dates available:
- creation date
- occur date
- event data date
System architecture
System diagram
An overview of the planned system can be seen below, this has not yet been completely implemented. This design is not final, and may require changes as the PoC progresses.
Event database
Responsible for the storage of events
Read model
Responsible for building a view of the data in the system, retrieves data from all F6s that have relevant data.
Command API
A CQRS command API, commands can be sent, these commands should be validated, and result in events being added to the event database.
Command validation will require a read model for the objects in the events object_ids
, to ensure that the command is valid for the objects current state.
! note: This seems like a possible location for race conditions
Query API
A CQRS query API, exposes a way to query the event database.
Not yet implemented:
F6 directory
Read models need to query every F6 that contains data about an object_id
to make this efficient a directory is required.
In the diagram the directory is split into parts:
- A global directory containing F6s and the events they store
- A local F6 directory containing F6s that we have a
contract
with allowing data exchange - A local
object_id
directory containing links to the F6s with data concerning theobject_id
! note: we do not yet have a definition of what contracts are exactly
! note: there is no global directory linking
object_ids
to F6s, this is done because knowledge of inclusion in some zones may be privacy sensitive (e.g.dienst justitiële inrichtingen
)
Event subscription API
Allows automatic systems to take actions when new events are added, to provide better service to citizens (e.g. automatically generate a BSN when a human
is born)
Management API
To keep management burden of the system low an API and UI should be made to manage the F6 system
- request and allow connections from other F6s
- manage event types / commands supported by the system
- audit activity
FSC
To ensure a safe data connection between multiple organisations FSC will be used
Development environment
Tooling used
- docker compose
https://docs.docker.com/compose/install/#scenario-two-install-the-compose-plugin
- cargo watch
cargo install cargo-watch
- cargo udeps
cargo install cargo-udeps
- sea-orm-cli
cargo install sea-orm-cli
Setup
- run
make setup
to startup a database, run migrations, and build schema based on the db - run
make write
ormake watch
to start up the graphql write api