UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Upload
Sign in
Uploaded bySander Mak (@Sander_Mak)
PDF, PPTX27,949 views

Event-sourced architectures with Akka

The document discusses event-sourced architectures using Akka, focusing on designing applications with a three-layer architecture consisting of a UI, service layer, and database. It emphasizes the importance of capturing all changes as a sequence of events, enabling features like event replay, integration mechanisms, and audit trails. Additionally, it examines the use of actors for managing application state and persistence, addressing challenges and methods for effective command and event handling in a distributed system.

Software◦
Download as PDF, PPTX
1 / 81
2 / 81
Most read
3 / 81
4 / 81
5 / 81
6 / 81
7 / 81
8 / 81
9 / 81
10 / 81
11 / 81
12 / 81
13 / 81
14 / 81
15 / 81
16 / 81
17 / 81
18 / 81
19 / 81
20 / 81
Event-sourced architectures with Akka @Sander_Mak Luminis Technologies
Today's journey Event-sourcing Actors Akka Persistence Design for ES
Event-sourcing Is all about getting the facts straight
Typical 3 layer architecture UI/Client Service layer Database fetch ↝ modify ↝ store
Typical 3 layer architecture UI/Client Service layer Database fetch ↝ modify ↝ store Databases are shared mutable state
Typical entity modelling Concert ! artist: String date: Date availableTickets: int price: int ... TicketOrder ! noOfTickets: int userId: String 1 *
Typical entity modelling Concert ! artist = Aerosmith availableTickets = 100 price = 10 ... ! TicketOrder TicketOrder ! noOfTickets = 3 userId = 1 TicketOrder ! noOfTickets = 3 userId = 1 noOfTickets = 3 userId = 1
Typical entity modelling Changing the price Concert ! artist = Aerosmith availableTickets = 100 price = 10 ... ! TicketOrder TicketOrder ! noOfTickets = 3 userId = 1 TicketOrder ! noOfTickets = 3 userId = 1 noOfTickets = 3 userId = 1
Typical entity modelling Changing the price Concert ! artist = Aerosmith availableTickets = 100 price = 10 ... ! TicketOrder TicketOrder ! noOfTickets = 3 userId = 1 TicketOrder ! noOfTickets = 3 userId = 1 noOfTickets = 3 userId = 1 ✘100
Typical entity modelling Canceling an order Concert ! artist = Aerosmith availableTickets = 100 price = 10 ... ! TicketOrder TicketOrder ! noOfTickets = 3 userId = 1 TicketOrder ! noOfTickets = 3 userId = 1 noOfTickets = 3 userId = 1
Typical entity modelling Canceling an order Concert ! artist = Aerosmith availableTickets = 100 price = 10 ... ! TicketOrder TicketOrder ! noOfTickets = 3 userId = 1 ! noOfTickets = 3 userId = 1
Update or delete statements in your app? Congratulations, you are ! LOSING DATA EVERY DAY
Event-sourced modelling ConcertCreated ! artist = Aerosmith availableTickets = 100 price = 10 ... ! TicketsOrdered TicketsOrdered ! noOfTickets = 3 userId = 1 TicketsOrdered ! noOfTickets = 3 ! userId = 1 noOfTickets = 3 userId = 1 time
Event-sourced modelling TicketsOrdered TicketsOrdered Changing the price ConcertCreated ! artist = Aerosmith availableTickets = 100 price = 10 ... ! TicketsOrdered ! noOfTickets = 3 userId = 1 ! noOfTickets = 3 ! userId = 1 noOfTickets = 3 userId = 1 time
Event-sourced modelling TicketsOrdered TicketsOrdered Changing the price ConcertCreated ! artist = Aerosmith availableTickets = 100 price = 10 ... ! PriceChanged ! price = 100 TicketsOrdered ! noOfTickets = 3 userId = 1 ! noOfTickets = 3 ! userId = 1 noOfTickets = 3 userId = 1 time
Event-sourced modelling ConcertCreated ! artist = Aerosmith availableTickets = 100 price = 10 ... ! PriceChanged ! price = 100 TicketsOrdered TicketsOrdered ! noOfTickets = 3 userId = 1 TicketsOrdered ! noOfTickets = 3 ! userId = 1 noOfTickets = 3 userId = 1 Canceling an order time
Event-sourced modelling ConcertCreated ! artist = Aerosmith availableTickets = 100 price = 10 ... ! PriceChanged ! price = 100 OrderCancelled ! userId = 1 TicketsOrdered TicketsOrdered ! noOfTickets = 3 userId = 1 TicketsOrdered ! noOfTickets = 3 ! userId = 1 noOfTickets = 3 userId = 1 Canceling an order time
Event-sourced modelling ‣ Immutable events ‣ Append-only storage (scalable) ‣ Replay events: reconstruct historic state ‣ Events as integration mechanism ‣ Events as audit mechanism
Event-sourcing: capture all changes to application state as a sequence of events Events: where from?
Commands & Events Do something (active) It happened. Deal with it. (facts) Can be rejected (validation) Can be responded to
Querying & event-sourcing How do you query a log?
Querying & event-sourcing How do you query a log? Command Query Responsibility Segregation
CQRS without ES Command Query UI/Client Service layer Database
CQRS without ES Command Query UI/Client Service layer Database Command Query UI/Client Command Model Datastore Query Model(s) DaDtaastatosrtoere Datastore ?
Event-sourced CQRS Command Query UI/Client Command Model Journal Query Model(s) DaDtaastatosrtoere Datastore Events
Actors ‣ Mature and open source ‣ Scala & Java API ‣ Akka Cluster
Actors "an island of consistency in a sea of concurrency" Actor ! ! ! mailbox state behavior async message send Process message: ‣ update state ‣ send messages ‣ change behavior Don't worry about concurrency
Actors A good fit for event-sourcing? Actor ! ! ! mailbox state behavior mailbox is non-durable (lost messages) state is transient
Actors Just store all incoming messages? Actor ! ! ! mailbox state behavior async message send store in journal Problems with command-sourcing: ‣ side-effects ‣ poisonous (failing) messages
Persistence ‣ Experimental Akka module ‣ Scala & Java API ‣ Actor state persistence based on event-sourcing
Persistent Actor PersistentActor actor-id ! ! state ! event async message send (command) ‣ Derive events from commands ‣ Store events ‣ Update state ‣ Perform side-effects event journal (actor-id)
Persistent Actor PersistentActor actor-id ! ! state ! event ! Recover by replaying events, that update the state (no side-effects) ! event journal (actor-id)
Persistent Actor ! case object Increment // command case object Incremented // event ! class CounterActor extends PersistentActor { def persistenceId = "counter" ! var state = 0 ! val receiveCommand: Receive = { case Increment => persist(Incremented) { evt => state += 1 println("incremented") } } ! val receiveRecover: Receive = { case Incremented => state += 1 } }
Persistent Actor ! case object Increment // command case object Incremented // event ! class CounterActor extends PersistentActor { def persistenceId = "counter" ! var state = 0 ! val receiveCommand: Receive = { case Increment => persist(Incremented) { evt => state += 1 println("incremented") } } ! val receiveRecover: Receive = { case Incremented => state += 1 } }
Persistent Actor ! case object Increment // command case object Incremented // event ! class CounterActor extends PersistentActor { def persistenceId = "counter" ! var state = 0 ! val receiveCommand: Receive = { case Increment => persist(Incremented) { evt => state += 1 println("incremented") } } ! val receiveRecover: Receive = { case Incremented => state += 1 } } async callback (but safe to close over state)
Persistent Actor ! case object Increment // command case object Incremented // event ! class CounterActor extends PersistentActor { def persistenceId = "counter" ! var state = 0 ! val receiveCommand: Receive = { case Increment => persist(Incremented) { evt => state += 1 println("incremented") } } ! val receiveRecover: Receive = { case Incremented => state += 1 } }
Persistent Actor ! case object Increment // command case object Incremented // event ! class CounterActor extends PersistentActor { def persistenceId = "counter" ! var state = 0 ! val receiveCommand: Receive = { case Increment => persist(Incremented) { evt => state += 1 println("incremented") } } ! val receiveRecover: Receive = { case Incremented => state += 1 } } Isn't recovery with lots of events slow?
Snapshots class SnapshottingCounterActor extends PersistentActor { def persistenceId = "snapshotting-counter" ! var state = 0 ! val receiveCommand: Receive = { case Increment => persist(Incremented) { evt => state += 1 println("incremented") } case "takesnapshot" => saveSnapshot(state) } ! val receiveRecover: Receive = { case Incremented => state += 1 case SnapshotOffer(_, snapshotState: Int) => state = snapshotState } }
Snapshots class SnapshottingCounterActor extends PersistentActor { def persistenceId = "snapshotting-counter" ! var state = 0 ! val receiveCommand: Receive = { case Increment => persist(Incremented) { evt => state += 1 println("incremented") } case "takesnapshot" => saveSnapshot(state) } ! val receiveRecover: Receive = { case Incremented => state += 1 case SnapshotOffer(_, snapshotState: Int) => state = snapshotState } }
Journal & Snapshot Cassandra Cassandra Kafka Kafka MongoDB HBase DynamoDB MongoDB HBase MapDB JDBC JDBC Plugins:
Plugins: Serialization Default: Java serialization Pluggable through Akka: ‣ Protobuf ‣ Kryo ‣ Avro ‣ Your own
Persistent View Persistent Actor journal Persistent View Persistent View Views poll the journal ‣ Eventually consistent ‣ Polling configurable ‣ Actor may be inactive ‣ Views track single persistence-id ‣ Views can have own snapshots snapshot store other datastore
Persistent View ! "The Database is a cache of a subset of the log" - Pat Helland Persistent Actor journal Persistent View Persistent View snapshot store other datastore
Persistent View ! case object ComplexQuery class CounterView extends PersistentView { override def persistenceId: String = "counter" override def viewId: String = "counter-view" var queryState = 0 def receive: Receive = { case Incremented if isPersistent => { queryState = someVeryComplicatedCalculation(queryState) // Or update a document/graph/relational database } case ComplexQuery => { sender() ! queryState; // Or perform specialized query on datastore } } }
Persistent View ! case object ComplexQuery class CounterView extends PersistentView { override def persistenceId: String = "counter" override def viewId: String = "counter-view" var queryState = 0 def receive: Receive = { case Incremented if isPersistent => { queryState = someVeryComplicatedCalculation(queryState) // Or update a document/graph/relational database } case ComplexQuery => { sender() ! queryState; // Or perform specialized query on datastore } } }
Persistent View ! case object ComplexQuery class CounterView extends PersistentView { override def persistenceId: String = "counter" override def viewId: String = "counter-view" var queryState = 0 def receive: Receive = { case Incremented if isPersistent => { queryState = someVeryComplicatedCalculation(queryState) // Or update a document/graph/relational database } case ComplexQuery => { sender() ! queryState; // Or perform specialized query on datastore } } }
Sell concert tickets ConcertActor ! ! price availableTickets startTime salesRecords ! Commands: CreateConcert BuyTickets ChangePrice AddCapacity journal ConcertHistoryView ! ! ! ! 60 45 30 15 0 100 50 0 $50 $75 $100 code @ bit.ly/akka-es
Scaling out: Akka Cluster Single writer: persistent actor must be singleton, views may be anywhere Cluster node Cluster node Cluster node Persistent Persistent Actor id = "1" Actor id = "1" Persistent Actor id = "1" Persistent Actor id = "2" Persistent Actor id = "1" Persistent Actor id = "3" distributed journal
Scaling out: Akka Cluster Single writer: persistent actor must be singleton, views may be anywhere How are persistent actors distributed over cluster? Cluster node Cluster node Cluster node Persistent Persistent Actor id = "1" Actor id = "1" Persistent Actor id = "1" Persistent Actor id = "2" Persistent Actor id = "1" Persistent Actor id = "3" distributed journal
Scaling out: Akka Cluster Sharding: Coordinator assigns ShardRegions to nodes (consistent hashing) Actors in shard can be activated/passivated, rebalanced Cluster node Cluster node Cluster node Persistent Persistent Actor id = "1" Actor id = "1" Persistent Actor id = "1" Persistent Actor id = "2" Persistent Actor id = "1" Persistent Actor id = "3" Shard Region Shard Region Shard Region Coordinator distributed journal
Scaling out: Sharding val idExtractor: ShardRegion.IdExtractor = { case cmd: Command => (cmd.concertId, cmd) } IdExtractor allows ShardRegion to route commands to actors
Scaling out: Sharding val idExtractor: ShardRegion.IdExtractor = { case cmd: Command => (cmd.concertId, cmd) } IdExtractor allows ShardRegion to route commands to actors val shardResolver: ShardRegion.ShardResolver = msg => msg match { case cmd: Command => hash(cmd.concert) } ShardResolver assigns new actors to shards
Scaling out: Sharding val idExtractor: ShardRegion.IdExtractor = { case cmd: Command => (cmd.concertId, cmd) } IdExtractor allows ShardRegion to route commands to actors val shardResolver: ShardRegion.ShardResolver = msg => msg match { case cmd: Command => hash(cmd.concert) } ShardResolver assigns new actors to shards ClusterSharding(system).start( typeName = "Concert", entryProps = Some(ConcertActor.props()), idExtractor = ConcertActor.idExtractor, shardResolver = ConcertActor.shardResolver) Initialize ClusterSharding extension
Scaling out: Sharding val idExtractor: ShardRegion.IdExtractor = { case cmd: Command => (cmd.concertId, cmd) } IdExtractor allows ShardRegion to route commands to actors val shardResolver: ShardRegion.ShardResolver = msg => msg match { case cmd: Command => hash(cmd.concert) } ShardResolver assigns new actors to shards ClusterSharding(system).start( typeName = "Concert", entryProps = Some(ConcertActor.props()), idExtractor = ConcertActor.idExtractor, shardResolver = ConcertActor.shardResolver) Initialize ClusterSharding extension val concertRegion: ActorRef = ClusterSharding(system).shardRegion("Concert") concertRegion ! BuyTickets(concertId = 123, user = "Sander", quantity = 1)
Design for event-sourcing
DDD+CQRS+ES DDD: Domain Driven Design Fully consistent Fully consistent Aggregate ! Aggregate ! Eventual Consistency Root entity enteitnytity entity Root entity entity entity
DDD+CQRS+ES DDD: Domain Driven Design Fully consistent Fully consistent Aggregate ! Aggregate ! Eventual Consistency Root entity enteitnytity entity Root entity entity entity Persistent Actor Persistent Actor Message passing
DDD+CQRS+ES DDD: Domain Driven Design Fully consistent Fully consistent Aggregate ! Aggregate ! Eventual Consistency Akka Persistence is not a DDD/CQRS framework But it comes awfully close Root entity enteitnytity entity Root entity entity entity Persistent Actor Persistent Actor Message passing
Designing aggregates Focus on events Structural representation(s) follow ERD
Designing aggregates Focus on events Structural representation(s) follow ERD Size matters. Faster replay, less write contention Don't store derived info (use views)
Designing aggregates Focus on events Structural representation(s) follow ERD Size matters. Faster replay, less write contention Don't store derived info (use views) With CQRS read-your-writes is not the default When you need this, model it in a single Aggregate
Between aggregates ‣ Send commands to other aggregates by id ! ‣ No distributed transactions ‣ Use business level ack/nacks ‣ SendInvoice -> InvoiceSent/InvoiceCouldNotBeSent ‣ Compensating actions for failure ‣ No guaranteed delivery ‣ Alternative: AtLeastOnceDelivery
Between systems System integration through event-sourcing topic 1 topic 2 derived Kafka Application Akka Persistence Spark Streaming External Application
Designing commands ‣ Self-contained ‣ Unit of atomic change ‣ Granularity and intent
Designing commands ‣ Self-contained ‣ Unit of atomic change ‣ Granularity and intent UpdateAddress street = ... city = ... vs ChangeStreet street = ... ChangeCity street = ...
Designing commands ‣ Self-contained ‣ Unit of atomic change ‣ Granularity and intent UpdateAddress street = ... city = ... vs ChangeStreet street = ... ChangeCity street = ... Move street = ... city = ... vs
Designing events CreateConcert
Designing events CreateConcert ConcertCreated Past tense Irrefutable
Designing events CreateConcert ConcertCreated Past tense Irrefutable TicketsBought newCapacity = 99
Designing events CreateConcert ConcertCreated Past tense Irrefutable TicketsBought newCapacity = 99 TicketsBought quantity = 1 Delta-based No derived info
Designing events CreateConcert ConcertCreated ConcertCreated Past tense Irrefutable TicketsBought newCapacity = 99 TicketsBought quantity = 1 Delta-based No derived info
Designing events CreateConcert ConcertCreated ConcertCreated who/when/.. Add metadata to all events ConcertCreated Past tense Irrefutable TicketsBought newCapacity = 99 TicketsBought quantity = 1 Delta-based No derived info
Versioning Actor logic: v1 and v2 event v2 event v2 event v1 event v1
Versioning Actor logic: v1 and v2 event v2 event v2 event v1 event v1 Actor logic: v2 event v2 event v2 event v2 event v1 event v2 event v1
Versioning Actor logic: v1 and v2 event v2 event v2 event v1 event v1 Actor logic: v2 event v2 event v2 event v2 event v1 event v2 event v1 Actor logic: v2 event v2 event v2 snapshot event v1 event v1
Versioning Actor logic: v1 and v2 event v2 event v2 event v1 event v1 Actor logic: v2 event v2 event v2 event v2 event v1 event v2 event v1 Actor logic: v2 event v2 event v2 snapshot event v1 event v1 ‣ Be backwards compatible ‣ Avro/Protobuf ‣ Serializer can do translation ! ‣ Snapshot versioning: harder
In conclusion Event-sourcing is... Powerful but unfamiliar
In conclusion Event-sourcing is... Powerful but unfamiliar Combines well with UI/Client Command Model Journal Query Model DaDtaastatosrtoere Datastore Events DDD/CQRS
In conclusion Event-sourcing is... Powerful but unfamiliar Combines well with UI/Client Command Model Journal Query Model DaDtaastatosrtoere Datastore Events DDD/CQRS Not a
In conclusion Akka Actors & Akka Persistence A good fit for event-sourcing PersistentActor actor-id ! ! state ! async message send (command) event event journal (actor-id) Experimental, view improvements needed
Thank you. ! code @ bit.ly/akka-es @Sander_Mak Luminis Technologies