Event Sourcing PoC backend

System name used: Federated zone event store -> F.Z.E.S. -> F6 F6 is the collection of components that make up the event store.

Concept

Modeling data surrounding human more closely

The current status quo is storing state and history. This approach has some issues when historical data has to be reinterpreted.

In this proof of concept a system is built to explore and demonstrate an event based alternative. We want to be able to:

  • Not save the current state, but construct it from events
  • Time travel to see what the state would be at any point in time
  • To change or remove past events, and their effects on legal consequences of those events
  • To build views on the data based on need of the consumers
  • Keep data separated by responsibility / ownership of events
  • Make automatic systems able to react on events

Federation

Separated based on responsibilities / ownership is done by federating by organisation and zone. We see municipalities as organisations, and divisions of the municipalities as zones.
An example of this can be seen below:

C4Context

Boundary(b1, "F6 system", "Federated zone event store system") {
	Boundary(b2, "Gemeente Zonnendaal", "organisation") {
		System(SystemA, "Burgelijke stand")
		System(SystemB, "Adres huishouding")
	}
	
	Boundary(b3, "Gemeente Woudendijk", "organisation") {
		System(SystemC, "Burgelijke stand")
		System(SystemD, "Adres huishouding")
	}
}

Events

Event flow

To design the system we have made an example of the event flow involved in the birth of a human.

Legend:

  • pink: facts
  • yellow: actions / real world events
  • purple: event claim
  • blue: event claim validation

System diagram

Event data structure

We expose events as follows:

struct EventValue {
	id: Uuid,
	created_at: DateTime<Utc>,
	occurred_at: DateTime<Utc>,
	object_ids: Vec<Uuid>,
	event_type: String,
	event_version: String,
	event_kind: EventKind,
	event_source_id: Uuid,
	event_source_type: String,
	event_reference: Vec<Uuid>,
	event_data: Value,
}

Id

Uuid used to identify the event

Object ids

  • Uuids used to identify the objects involved in the event, in this PoC humans.
  • humans do not have to be born before an id for them may be created
  • humans do not have to be citizens

Event type

Type of the event (e.g. Geboorte orVerhuizing )

Event version

Events are versioned so that changes to their format do not require data migrations

Event kind

  • Claim: A claim that an event has occurred, claims do not have any legal consequences.
  • Correction: A claim that a past event contained incorrect data
  • Validation: A claim that is verified by a civil servant, within the civil servants domain of responsibility. Validation must be linked to the Claim event that is being validated.

Event source id

The uuid of the event source creating event, intended for use in audits

Event source type

The type of authority (burger, ouder, ambtenaar) that the event source creating the event has

Event reference

If the event is based on one or more previous events, this must contain the ids of those events

Event data

Event specific data, the data format is set in the write_config file, read from the write_config_path given to the write server

Material vs Formal history

! todo: improve description in associated issue

Material history: what has changed in reality Formal history: what has changed in registry

Dates available:

  • creation date
  • occur date
  • event data date

System architecture

System diagram

An overview of the planned system can be seen below, this has not yet been completely implemented. This design is not final, and may require changes as the PoC progresses.

System diagram

Event database

Responsible for the storage of events

Read model

Responsible for building a view of the data in the system, retrieves data from all F6s that have relevant data.

Command API

A CQRS command API, commands can be sent, these commands should be validated, and result in events being added to the event database.

Command validation will require a read model for the objects in the events object_ids, to ensure that the command is valid for the objects current state.

! note: This seems like a possible location for race conditions

Query API

A CQRS query API, exposes a way to query the event database.

Not yet implemented:

F6 directory

Read models need to query every F6 that contains data about an object_id to make this efficient a directory is required.

In the diagram the directory is split into parts:

  • A global directory containing F6s and the events they store
  • A local F6 directory containing F6s that we have a contract with allowing data exchange
  • A local object_id directory containing links to the F6s with data concerning the object_id

! note: we do not yet have a definition of what contracts are exactly

! note: there is no global directory linking object_ids to F6s, this is done because knowledge of inclusion in some zones may be privacy sensitive (e.g. dienst justitiƫle inrichtingen)

Event subscription API

Allows automatic systems to take actions when new events are added, to provide better service to citizens (e.g. automatically generate a BSN when a human is born)

Management API

To keep management burden of the system low an API and UI should be made to manage the F6 system

  • request and allow connections from other F6s
  • manage event types / commands supported by the system
  • audit activity

FSC

To ensure a safe data connection between multiple organisations FSC will be used

Development environment

Tooling used

  • docker compose https://docs.docker.com/compose/install/#scenario-two-install-the-compose-plugin
  • cargo watch cargo install cargo-watch
  • cargo udeps cargo install cargo-udeps
  • sea-orm-cli cargo install sea-orm-cli

Setup

  • run make setup to startup a database, run migrations, and build schema based on the db
  • run make write or make watch to start up the graphql write api