How to manage state in Apache Kafka

Event logs often have the full state within them, but that doesn’t necessarily mean that reading the event log from the start to build the current state is the best way to store and manage it. When considering event log based systems, it is vital to think carefully not only about functional considerations, but also non-functional considerations.

This talk is about the importance of considering performance, data consistency and latency from the start when architecturally planning event log systems. It will give examples how developers who were new to Apache Kafka tried various different approaches to managing state in such systems. It shows the successes, the failures and the lessons learned and how an approach to managing state in an event log evolved, and where we hope to take it.

Does Kafka’s long living, persistent event log mean that we can now realise a utopia of fully stateless systems? Look before you leap, says Ben Abramson …

Ben Abramson is a Technical Lead at William Hill, working in the Trading Technology department. He has been developing software for over 20 years and has worked across many industries and in many companies, and he has seen the good, the bad and the ugly.

He has worked primarily in Java, ranging from classic WebApps to Big Data systems with Hadoop, Kafka, Spark, Storm, Hive, Cassandra and many more. Ben has been working with Apache Kafka for several years on a variety of use cases.

He has spoken about Apache Kafka at various conferences and meet-ups and enjoys being an evangelist of good practices with emerging technologies.

Ben also has extensive experience in DevOps and automation and is passionate about creating durable software of high quality.

When not developing software, he is happiest at home with his family, usually kicking a ball around with his football-mad son.