Apache Kafka 2.4 lets consumers fetch from closest replica

It’s been more than eight years since Apache Kafka was first released by business network LinkedIn. A short time later it became part of the Apache Software Foundation ecosystem and has been open source ever since. The development of the big data framework is being advanced significantly by Confluent, a company founded by the original Apache Kafka developers.

Apache Kafka 2.4: New features and improvements

Version 2.4 is a relatively big update for the platform and brings some improvements and new features for the framework. These are discussed in the form of KIPs (Kafka Improvement Proposals) by the community and the developers, a similar concept to Java’s JEPs.

With KIP-392, Kafka now supports reading from follower replicas. This gives the broker the ability to redirect users to adjacent replicas to save costs. This could have a big impact on cloud environments by helping to reduce the amount of data transferred across longer or more expensive links.

KIP-504 is a Java authorizer API that is compatible with other pluggable interfaces in the broker. Several limitations in the current Scala authorizer API that could not be fixed without affecting compatibility have been removed with the new API. The API also supports asynchronous ACL updates with batching and only requires a dependency on the client’s JAR.

Kafka Connect and Kafka Stream

Also new is MirrorMaker 2.0 – a new multi-cluster and cross-data center replication engine based on the Connect Framework. This tool includes several features designed for disaster recovery, including cross-cluster consumer checkpoints and offset synchronization.

Finally, Kafka 2.4 includes stream support for non-key connections in KTable. Previously, the domain-specific stream language (DSL) allowed only table-table connections based on the primary key of the connecting KTables. Now, if a KTable (left input) is linked to another KTable (right input) based on a specified foreign key as part of the value fields, the link result is a new KTable encoded on the original key of the left KTable.

A complete list of the changes can be found in the release notes on the homepage of the Apache Software Foundation, more details about the highlights can be found in the official blog post and in the following YouTube video by Tim Berglund, Apache Kafka Developer Advocate.