EDA Mistakes

Who am I

Martin Skovbo Faartoft

Lead Engineer @

Slides: ftft.dk/talks/eda

Why am I here?

I built a big distributed system
Made every mistake in the book
millions of events per day

Getting started is easy

Quickstarts / Tutorials / Template repos are everywhere

Reliable events are hard

Don't repeat my mistakes!

Distributed systems 101

Your app will crash

(at the worst possible time)

The network is not reliable

Synchronous

Things I screwed up

in no particular order

Reinvented the wheel

[Picture of wheel] notes:

Seemed smart in the beginning, because I didn't know what I didn't know.
Migrating while running was painful.
Changing frameworks was painful - abstractions didn't exactly fit.

Use a library

[Picture standing on the shoulder of giants] [Pictures library logos] notes: Use MassTransit/Rebus/NServiceBus

Configuration

 public async Task MessageHandler(MyEvent e)
{
  var result = ComplicatedCalculation(e);
  //crash?
  await _broker.Publish(result);
  //crash?
}

ReceiveMode.ReceiveAndDelete ReceiveMode.PeekLock

Learn the language

PeekLock, ReceiveAndDelete
Complete
(Auto)Commit
Ack
Prefetch
{AUTO|CLIENT|UNORDERED}_ACKNOWLEDGE
What happened? A bunch of events disappeared

Dual-Writes

 public async Task MessageHandler(MyEvent e)
{
  var result = ComplicatedCalculation(e);
  await _dbContext.SaveChanges();
  //crash?
  await _broker.Publish(result);
}

notes:

What happened? Two systems were slightly out of sync

1 - Transactional Outbox

 public async Task MessageHandler(OrderUpdated e)
{
  await var order = _dbContext.Orders.Find(e.Id);
  var result = UpdateOrder(order, e);
  _dbContext.Outbox.Add(result);
  await _dbContext.SaveChanges();
}

notes: - doesn't have to be a db write - distributed transactions

2 - Change Data Capture (CDC)

Let the database track changes

 public async Task MessageHandler(OrderUpdated e)
{
  await var order = _dbContext.Orders.Find(e.Id);
  var result = UpdateOrder(order, e);
  //_dbContext.Outbox.Add(result);
  await _dbContext.SaveChanges();
}

3 - Nothing

Let retry deal with it

Trusted message ordering

Detour: dead-letters
Dead letters
Competing consumers
Envelope timestamps
Tombstones

Didn't apply Idempotency

What is idempotency
Detour: delivery semantics
Sent a few too many emails

Concurrency

Lost update is really easy with wrong isolation level

Didn't store processed events

Explaining current state is really hard without it

No kill-switch

Piling up >1M dead-letters over night.

services.AddKillSwitch(5, true, myClass)

Accepted bad event formats

Stitching partial events together
Timestamps in local time
Events with important data in envelope / headers
PUSH BACK - LET THE PRODUCER KNOW

Assumed ServiceBus pricing tier was easy to change

Resources

Wordcloud

Delivery_Semantics Failure_Semantics Idempotency Dual_Write Change_Data_Capture Two-Phase_Commit Distributed_Transaction Transactional_Outbox Dead-letter Peek-lock Topics Queues Subscriptions Partitioning FIFO LIFO Exactly_once_delivery At_least_once_delivery At_most_once_delivery Exactly_once_processing Two_Generals_problem Failure_Semantics Sync Async Eventual_Consistency Messages Events Commands Prefetch Tombstones Consumer Producer Publisher Subscription Subscriber Duplicate_detection Event-streaming Message_broker MQTT AMQP Event_sourcing

	public async Task MessageHandler(MyEvent e)
	{
	var result = ComplicatedCalculation(e);
	//crash?
	await _broker.Publish(result);
	//crash?
	}

	public async Task MessageHandler(MyEvent e)
	{
	var result = ComplicatedCalculation(e);
	await _dbContext.SaveChanges();
	//crash?
	await _broker.Publish(result);
	}

	public async Task MessageHandler(OrderUpdated e)
	{
	await var order = _dbContext.Orders.Find(e.Id);
	var result = UpdateOrder(order, e);
	_dbContext.Outbox.Add(result);
	await _dbContext.SaveChanges();
	}

Event-Driven Architecture