Event storming is a process of modelling software that works well with domain driven design principles. During an event storming session all involved parties get together and help model a domain, this helps everyone from the developer to the business decision makers understand the system and from a ubiquitous language ( common language). Event storming is just another modelling tool to help with quick design and development.
Here is another short article about event sourcing. This article includes many useful links at the bottom.
Alberto Brandolini – 50,000 Orange Stickies Later
This video is from the man who created the concept of Event Storming. In this video Alberto discusses how he invented this technique of Event Storming with PostIt notes. He describes and demonstrates how this sort of Brain Storming session is a great way to get all involved parties together. This technique will help you more quickly and completely model your software. I’ll be writing about this technique in my articles about microservices with Scala.
When learning about Domain Driven Design one of the harder things is wrapping your head around what aggregates are. In this post I list some facts, some may be redundant, but redundancy helps reinforce concepts.
Here is my personal list of facts and notes about domain driven design aggregates in no general order. This list will be updated as time goes on.
List of Aggregate facts
An aggregate is a consistency boundary that decomposes large models into smaller clusters of domain objects that are technically easier to manage. Basically I refer to these as modules/services, modules should be easily hot swappable.
An Aggregate is a DDD pattern, a cluster of related domain objects that work together as a single unit
Aggregates will have one component object that acts as an interface to the aggregate known as the Aggregate root All outside interaction with the aggregate should be through the aggregate root. An aggregate root is the contract for the entire aggregate, the API interface for the Aggregate/Module/Service
Aggregates are the inner bounded contexts that build the complete bounded context of the entire domain, like building blocks.
An aggregate is a way to group objects inside of a specific domain bounded context. They are a way to modularize things that have to work together to perform a specific task of the system
Aggregates are collections of objects that perform the functions of a particular bounded context.
Aggregates are stored in a repository, aka database
Aggregates are the only thing that can be persisted and retrieved ( re-hydrated )
Repositories are used to manage the persistence of aggregates and ensure a clear separation between the data and the domain
Aggregates communicate with other aggregates via Events, creating event driven systems
Try to make aggregates as small and specific as possible, this creates less room for bugs and makes the system easier to maintain. Stick to the Single Responsibility Principle
Large aggregates can suffer from performance issues when they span multiple entities, data stores, database tables Only make aggregates as large as they absolutely have to be, eliminate all unrelated code and actions.
An Aggregate lives in a Bounded Context. Each bounded context is made up of multiple aggregates. Each a aggregate within a bounded context should have a Single Responsibility.
When two or more objects/entities need to interact with each other to perform a task, make them part of the same aggregate.
Aggregates represent concepts in the larger domain.
Aggregates should be behavior focused
Each object within an aggregate should be required to do some action for the aggregate to be included. Don’t have redundant aggregates within other bounded contexts, this is a clue that a new aggregate needs to be born into its own bounded context. Create another module and make the redundant aggregate it’s own.
An aggregate root should be the only entry point for an aggregate. The aggregate root is like an application programmers interface to the aggregate API.
The aggregate root is the coordinator for the aggregate. It handles all of the events by calling the aggregate objects to perform the required tasks/actions.
The aggregate root should expose only the behaviors required by other aggregates. The aggregate root is the contract of the aggregate. Removing behaviors makes breaking changes and can destroy a system. However, adding new behaviors any time is safe.
Objects within aggregates should not hold references to objects in other aggregates, this creates tight coupling, exactly what we are trying to prevent. This means no object in one module/service/microservice should directly call an object or refer to one inside another module/service/microservice. This means no communication outside one bounded context to another, the aggregate root is the only way to communicate.
An aggregate can span several database tables when persisted, but this can cause problems.
Domain objects within an aggregate can have direct object references to each other. This means that aggregates ( objects ) within a single bounded context (module/service) can have references to each other. Meaning within a service you can call other objects within the service.
Repositories should hold aggregates. Aggregates can be reconstructed at any time from a repository.
DDD repositories are not the same as code repositories like github
The aggregate roots are AKKA actors that should accept commands and produce events
Repositories are used to manage domain object/aggregate root persistence
A repository manages the retrieval of aggregate/domain objects while ensuring a separation between the domain and data models
A repository is a pattern for storing and retrieving the pieces of an aggregate in a database
A repository enforces the aggregate roots contract by providing the interface to store and retrieve the aggregate parts
Repositories map to the aggregates not the actual data-store such as database tables
Developing microservices with aggregates – Chris Richardson
I found this to be one of the best explanations of aggregates and how they relate in microservice based systems. Here is a link to the site mentioned in the video microservices.io where you can find a lot more useful information about microservices.
Most of us older people have been building monoliths for most of our careers. Now BAM! a new buzzyword is born Microservices. Microservices are all about tearing apart the dreaded monolith into smaller services that all function together to perform the actions of the monolith. Microservices have many benefits over Monoliths, however they have many disadvantages as well. Microservice systems are inherently distributed. Technology has changed so much recently. In the past we had to rent entire servers or virtual private servers etc.
Now we have the cloud, we can buy server power for super cheap, my favorite host Vultr.com has hosting packages for less than $5 USD. Vultr has the most awesome hosting interface for us Developers too. No intrusive demands for a credit card, you can pay with Paypal, Bitcoin and more.
This cheap hosting combined with containerization technology such as Docker/Kubernetes, makes getting into microservices easier than ever. The problem may be deciding if you need microservices or not.
I’ve spent about two years reading and researching Microservices below I list some of the Articles, Books and Videos that helped me along the way. I list these because there is a ton of old and bad information floating around the internettens and it took some effort to get to the bottom of what is currently the best accepted practices. This also gives me the ability to look back for information I can’t quite remember where I saw it since all I do is read, day and night.
Microservices – Not A Free Lunch! — This is a good article to start with. This highlights the fact that microservices are not always appropriate for every situation and you should need them more than you want them.
Microservice pattern explained. This article covers the idea of the microservice pattern. It covers the uses of the microservice pattern and compares it to the traditional Microservice pattern. This article glosses over communication between services. This article also covers the pros and cons of the microservice pattern. This article has good information about how to split your application into microservices, it has quite a few good links too.
Message Broker or Bus – what’s the difference? This is a good article that explains the minute differences between a Message Bus and a Message Broker. One of the worst things about the Software Industry is all the gimmicky terms and buzzwords, it is enough to make you want to move on to Physics where bosons, quarks and leptons are the buzzwords.
The Saga Pattern This article describes a useful microservice communication pattern called the Saga pattern. This pattern has it’s uses but I would not use it everywhere. This pattern allows a series of events to be reversed if a condition is not met or something along the line of communication fails.
This article has some good information about transactions between microservices. This article also covers choreography vs orchestration. Orchestration leads to a highly coordinated system which can create more tight coupling between microservices because some central authority orchestrates everything it must know about everything. A choreographed system decreases coupling by using an event sourced approach allowing microservices to publish events and decide what to do when an even they are listening to occurs. See the section about Choreographed vs Orchestrated in the article.
While researching Kafka as a message broker I wanted to know what the difference between Kafka and an ESB was, this excellent article covers just that topic. It covers event sourced/based systems and how Kafka is used in this type of setup.
This article talks about SOA Software Oriented Architecture which was the prelude to Microservices. This article is from 2015, this is about the time the word Microservice was born. This shift from SOA to microservices has made researching microservices a pain in the ass since during the shift many things were copied over or renamed etc. It has been hard to find any sort of consensus on many of the patterns and software used to build microservices.
The article mentions how SOA uses Enterprise Service Buses (ESB) as a way to integrate services. It mentions that before ESB the services connected to each other directly. This article is actually an interesting read for those of us who have not been in the industry for 20 years, it is a great recent history lesson. Before SOA it was the monolithic architecture, the mega pains of which is what spawned the SOA culture. Once you are bitten by a monolithic pile of crap you never ever want to see or create one ever again, it is the worst horror filled nightmare one can endure.
This article quickly mentions some of the bad habits of building Microservices as mini monoliths which are hard to scale. It quickly covers the basics of what makes a good microservice. This article contains many handy links worth reading. The article is a summary of a Talk from QCon Here is a link to the slides From Monoliths to Microsystems. You an find the video and summary below under the same name.
Implementing Microservices with Scala and Akka
GOTO 2014 • Microservices • Martin Fowler
Martin begins this talk by asking “What is a microservice” and compares it to a monolith giving a nice definition of Microservices. This video gives a good overview of microservices. Here is Martin Fowlers website link to microservices, lots of good links here, the same as the top link of the page. This video contains a lot of good information, Martin talks about the past and how we ended up with Microservice architecture. In this video he explains the move away from Enterprise Service Bus in the idea of dumb pipelines and smart end points. This is a very good very short explanation of Microservices and the rules of Microservices. Martin also talks about how we should design our microservices to fail and use monitoring for self healing. He also covers Microservices vs Service Oriented Architecture (SOA). At about the 18:00 mark Martin Talks about Microservices vs Monoliths and when you should choose which architecture, this topic is covered fairly thoroughly. Personally I would say Always avoid the monolith Always unless you like pain and misery. Also this video is from 2014, things have changed especially with Container Technology like Kubernetes and Docker, which make microservices much easier now.
Greg Young – The Long Sad History of MicroServices
The talk starts at about the 9:00 minute mark. Greg makes a great point about microservices – everyone has their own definition it seems. This video makes the point that the software industry repeats everything over and over, and likes to rebuild the wheel. This video mentions smart endpoints and dumb pipes as well if you are wondering what that is.
This video talks about Software Oriented Architecture (SOA) and how we ended up with Microservices which are basically SOA done correctly, or applying best concepts to SOA. Turns out SOA was a buzzword too, just like Microservices. Microservices is basically just a new buzzwordy term for SOA.
The concepts of Microservices are very old, if you are an older developer trying to wrap your head around these microservice concepts then this video will help clarify a lot of things for you. Greg talks about many of the misunderstood concepts of Microservices like how many lines of code and microservice cohesion. This video also covers the Database per service idea vs a single database per company.
|Greg also talks about how Microservices should be asynchronous and event driven and why. At 46:00 greg talks about securing events, very interesting concept. He makes the point that microservices are not a cure for everything and it is sort of a fad.
GOTO 2016 • The Entity Microservice Trap You May Be Doing It Wrong • Fred George
I like this video because Fred takes the time to explain the history of software design and development and slowly leads up to Microservices. This video will help you avoid some pitfalls of Microservice design and help clarify what Microservices really are and how they work.
In this video Vaughn talks about some of the misconceptions we have when we first start learning about Microservices. Through a series of funny stories he covers how most people think of microservices and DDD and how we should think of the two. Microservices are distributed systems and they require modeling uncertainty. This video will also help clarify what is going on with microservices, how do they communicate with commands and events etc. I like the coverage of how the commands, events and microservices should work in this video. This video covers a lot and will help clarify some concepts of DDD and how they are used in Microservices.
Eric Evans DDD and Microservices: At Last, Some Boundaries!
This video covers Domain Driven Design and boundaries between microservices. Eric also covers some of the decisions that need to be made on how microservices communicate with each other.
Developing microservices with aggregates – Chris Richardson
Chris breaks down how microservices and aggregates relate in this video. This video answers lots of questions about how Microservices are built and communicate. It also covers aggregates and how to pick what goes into an aggregate. This video does a good job explaining how Event sourcing works as well. This is actually one of the most useful videos because it covers Microservices and DDD concepts at a very high view level, giving you an overall idea of what everything means. This video really helps tie many of the Microservice and DDD concepts together and will give you aha moments. microservices.io mentioned in the video half way through.
The Enterprise Architects Intro To Microservices Part 1 Kevin Webber
This video starts by covering some of the history of software design and how the industry arrived at Microservices. Kevin then covers how users expectations have changed since the internet was born and how these days people expect things nearly immediately. This video is an excellent explanation of why the microservice architecture is so important in the Enterprise world.
Kevin soon covers the history of SOA, how and why it was a failure because ESB was a single point of failure. This gives good coverage of Enterprise service bus failings. This video gives a great explanation of what a microservice is with a nice diagram. This is an absolutely excellent explanation of the Microservices Architecture or MSA.
The Enterprise Architects Intro To Microservices Part 2 – Jonas Boner
This is the second video in the Lightbend Microservice series. The talk begins at about the 3:00 mark, before is an introduction. This video starts with the differences between hardware in the past compared to today. Jonas talks about how microservices allow for better isolation between services and why this is good for preventing entire cascading system failure when one service fails. He then compares Microservices to the Titanic and how microservices allow natural bulkheading because of their isolation by nature. This isolation allows for self healing microservices.
Why Asynchronous communication?(15:00 mark)
Jonas then talks about asynchronous communication, why synchronous is bad and asynchronous is better. If you are wondering why everyone is saying you should use asynchronous then this video gives an excellent explanation with a nice diagram. You shouldn’t always use asynchronous communication, you can also use a Message Broker and pass messages.
What is microservice state? (~26:00 mark)
Jonas talks about microservices should own their state, they should own their data or database that is. This is one of the most important points of Microservices. One microservice should not reach into another microservices database either to write or read. He talks about the fact that there is no such thing as stateless architectures (~28:00 mark) scammy buzzword.
Bounded Contexts (~29:00)Jonas talks about bounded contexts in relation to microservices. Event Logging and Event Sourcing explained(~31:00 mark)
The Enterprise Architects Intro To Microservices Part 3 – Jonas Boner
This is the third video in the Enterprise Architects series. This video starts out by giving a good description of microservices and how they should work. Jonas talks about how microservices are distributed systems by nature and the sort of things that need to be considered when designing your architecture. Jonas makes the point that no matter what in a microservices system you will be dealing with stale data and inconsistencies and that is just the way it is. Inside data vs Outside data (~10:00 mark) this is a good concept to understand. One microservice communicates to another it wants something done via a command. Commands should have a side effect. The challenges of microservices (~11:30 mark ) Service discovery (~13:00 mark) is covered. Anti-corruption layer discussed (~16:15 mark) Api gateway discussed (~17:00 mark) Communication challenges discussed (~19:00 mark) Integrating with other systems discussed (~21:00 mark ) Circuit breakers explained (~22:00 mark) Back pressure explained (~24:32 mark ) Securing microservices discussed quickly (~26:00 mark ) Saga Pattern discussed (~35:45 mark) More about Saga pattern here.
GOTO 2019 • 3 Common Pitfalls in Microservice Integration & How to Avoid Them • Bernd Rücker
Bernd is an entertaining person and he is quite funny in this video at times. Here is a link to the article mentioned in the video. He mentions in this video one of the biggest problems with microservices is they are distributed systems and many developers don’t know how distributed systems work. Developers building distributed systems should plan for failure (3:00). At about 5:30 Bernd mentions using REST for service communication and the issues involved. At about 8:30 he talks about circuit breakers with REST communication. Handling service state ~15:30. Workflow engines ~18:00 . Idempotency ~24:30 and why it is important. Asynchronous communication ~32:30. Message Bus 34:30 and monitoring dead messages. Distributed transactions ~38:30 Saga pattern mentioned ~41:00
Lost in Transaction – Strategies to deal with consistency in distributed systems (QCon London 2019)Bernd Rücker
Bernd talks about transactions, as in when you have two or more things that depend on each other. This is the video mentioned in the video above 3 Common Pitfalls… At ~3:30 he mentions Two Phase Commit (2pc) and why it doesn’t work with distributed transactions. Idempotentcy and why it is important ~10:00 and it’s basic implementation. Stateful retry strategy ~15:45 describes retry strategies with a state machine. An example of using a message queue pattern ~22:30 Saga pattern described ~27:30 and how it got it’s name, I hate the name. Choreography vs Orchestration event-driven pattern ~29:00 He goes on to talk about why fully 100% event driven systems are not the best way to build microservices. At ~35:00 he starts to talk about alternatives involving orchestration using commands. Orchestration with Saga workflows and BPMN(business process model and notation) pattern ~36:30
Distributed Sagas: A Protocol for Coordinating Microservices – Caitie McCaffrey – JOTB17
Sagas are a way to orchestrate transactions between microservices that need to either all succeed or all fail. This video was mentioned in the video above “Lost in Transaction”. Caitie speaks a little too fast, you may need to adjust the video speed down some. This is a great video that does a good job of explaining Sagas and what a Distributed Saga is . She covers what 2pc ( two phase commits) are and why they don’t work because they don’t scale well. She then goes on to talk about distributed Sagas. A saga is a way to reverse transactions when one fails in a series. The talk about Distributed Sagas starts at ~15:00, this is the definition. How to define a distributed saga ~ 21:30. Distributed Saga log ~23:00, basically using a log to move a multi-transaction process forward or reverse it in error. Failures in Distributed Sagas ~27:00, explains how to perform a Saga Rollback. Each of your microservices need a way to reverse transactions.
Avoiding Microservice Megadisasters – Jimmy Bogard
There are some things to avoid when designing a Microservice Architecture (MSA). In this excellent video Jimmy highlights some of the “do not do’s” of Microservice design. He does this by telling a story of history to help us understand how we got to where we are and the mistakes we made along the way. At ~10:00 he starts to cover API hell, and how it is created by microservices calling microservices calling microservices via HTTP requests tying up the whole network. This is naturally how most of us think of structuring microservice communication until we dig much deeper and find videos like this that explain why it is bad. At ~14:00 Jimmy explains how we can be naive as developers by only locally testing our microservices with mock outs and never tapping into a live system for testing, this is a very important point. At ~15:00 he points out an older Microservice design that failed. So many of the ideas on how to design Microservice architectures is wrong even though they seem logical. At ~17:00 he points out what a mega failure the management of the IT industry is, basically how not to run a company. At ~19:45 he gives the definition of a Microservice. At ~20:30 he defines and explains what a DDD bounded context is and how it relates to Microservice Architectures (MSA) At ~24:00 Jimmy gets to data duplication and why it is needed in Microservice architectures. At ~27:00 he explains Service Dependency Inversion and why it is needed in a Microservice Architecture. This dependency inversion is a great design for searches especially. At ~38:00 he again points out how not to run a business, especially not a software business. Idiot managers. At ~39:00 Jimmy’s law.
GOTO 2017 • The Seven (More) Deadly Sins of Microservices • Daniel Bryant
Daniel starts out by telling a short story about microservices, a very good talk about some of the history. At ~6:30 he makes an excellent point about moving to Microservices, you really need to plan your architecture well. You need to think about what technologies you will need in your architecture to make it function, the more you plan the better. At ~8:45 to containerize or not, he talks about Docker and containerization. GRPC ~12:00 quickly mentioned. Service Mesh’s covered ~14:00 More about RPC ~15:00 comparing it to event sourcing Link to the article mentioned at ~15:30 From Microliths to Microsystems: Jonas Bonér at QCon London Next up ESB ( Enterprise Service Buses) ~16:30 these are used in SOA (Service Oriented Architectures) which were popular before Microservices became a word. Microservices are a continuation of the SOA concepts, taking the good leaving the bad basically. Non functional requirements ~23:00 He makes another good point of things you need to consider upfront when designing your microservice system. At ~26:00 he mentions Performance and Load testing with Gatling, JMeter and Flood.io. He then covers security testing ~27:15 Owasp Dependency Check, Docker Bench etc. The discussion about Wrath begins ~28:30 He talks about Devops starting ~30:30 some useful information about books there. PAAS Platforms as a Service ~35:00 Data Models ~36:00 Domain Driven Design DDD mentioned ~36:30 Context Mapping and Event Storming ~38:00 Data bases and Data Stores, RDBMS, NoSQL etc. ~39:00 At ~40:00 he mentions testing At ~41:30 he mentions API Simulation and Service Virtualization this is a very interesting concept. At ~43:00 Service Virtualization software information listed.
From Microliths to Microsystems GOTO 2017 – Jonas Boner
This video is mentioned in articles and videos above. Jonas starts the video with a discussion about Monoliths and how they don’t work. This is an excellent video that points out some bad habits to avoid, things that seem intuitive in Microservice systems is actually the wrong way to do it. At ~5:00 he starts talking about the most common way we developers begin creating microservices. At ~6:00 he begins to define what a Microlith is. Reactive Programing vs Reactive Systems ~16:30 Async discussed ~18:30 Back pressure ~20:00 Microservices come as Systems ~23:00 Separating the Stateless behavior from the Stateful entity ~23:30 Asynchronous Message Passing ~28:00 An excellent book on this subject is Reactive Messaging Patterns with the Actor Model: Applications and Integration in Scala and Akka by Vaugh Vernon . Thinking in terms of Consistent Boundaries ~34:00 Inside Data vs Outside Data ~35:00 DDD and Events first design ~36:00 Defining Bounded contexts by Events ~37:00 Event Storming ~38:00 Event Logging ~41:30 CQRS and Event Sourcing ~43:30 decouple read from write. What about Transactions? ~47:30
Microservices, Kubernetes, and Application Modernization Done Right
This video covers multiple subjects, many listed individually above. This is sort of like tying it all together. It talks about the mistakes of the SOA era.
Domain Driven Design and it’s related concepts can take some time to understand. Below are links to articles, books and videos I have found helpful.
Here is a free guide to Domain Driven Design written by microsoft. This guide covers CQRS and Event Sourcing complete with an app that can be downloaded and inspected while reading the guide. This covers bounded contexts and suggestions for finding them and much more. This is an excellent free source.
Exploring CQRS and Event Sourcing This link is from a Microsoft project and is an excellent resource about CQRS and Event Sourcing. It walks you through the entire decision process of breaking down an entire application domain with basic Domain Driven Design concepts. It is very thorough and a great read to help you understand what goes into CQRS, Event sourcing and breaking your domain into smaller more useable chunks.
Patterns, Principles and Practices of Domain-Driven Design by Scott Millett and Nick Tune This book covers Event Sourcing and CQRS as well as all of the other principles of Domain Driven Design. The nice thing about this book is it has code examples, the code is C#, but you don’t have to down load it to view it and basically follow along. The examples often come after long explanations of the concepts, if you have time or know C# you can download and try the example code, a nice extra if you ask me. This is one that stays in a pile next to my desk.
Greg Young – CQRS and Event Sourcing – Code on the Beach 2014
Greg tells some great stories that relate to why CQRS and Event Sourcing are useful and gives interesting insights into use cases. For many of us who have been doing the old fashioned CRUD architecture of storing and reading everything from a database, this video will make sense of why that was a bad way of doing things. After this video you will really want to dig into CQRS and Event Sourcing in depth.
CQRS and Event Sourcing with Lagom by Miel Donkers
An Introduction to CQRS and Event Sourcing Patterns – Mathew McLoughlin
In this video Mathew explains how an Event Sourced system that utilizes CQRS works. One thing covered is why having separate read and write data stores is a good idea and how it works in an abstracted way.
Event Sourcing You are doing it wrong by David Schmitz
I like this video this guy is funny. David quickly covers Microservices, event sourcing and other Domain Driven Design concepts, tying things together in a very entertaining animated and funny way. This is one of my favorite videos about event sourcing, it answers a lot of questions and makes many good points. This video is worth watching more than once.
Allard Buijze – Talk Session: The Inconveniences of Event Sourcing
This video has some good abstract information about events and event sourcing in microservices. It points out some of the issues with event sourcing and how to overcome them. Allard covers the types of Events, commands, events, queries and how they are used in an event driven system. The most interesting and for me useful part of this video is the part about GDPR and how to comply in an Event Sourced system at about the 38:00 mark.
Martin Kleppmann — Event Sourcing and Stream Processing at Scale
Udi Dahan – If (domain logic) then CQRS, or Saga?
Udi tells a story that builds up to CQRS and Sagas. He says in CQRS that Commands can’t be allowed to fail. At about 27:00 he explains what this means. At ~31:00 he talks about eventual consistency and how to deal with it in a system. At about 40:00 give them the Rick Sanchez slow ramp. LOL He finishes up by talking about bounded contexts.