Kafka offset commit failed (CommitFailedException)

During one of my professional adventures I’ve observed a lot of errors like the infamous CommitFailedException presented below. There were many of them or similar in log messages generated by one of micro-services.

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

At the same time I’ve observed that the system was not consuming new messages from Kafka topic… but log messages were showing it was consuming constantly. Actually it looked like the micro-service was receiving the same messages from Kafka over and over again.

The exception message is very verbal and it suggests to tweak Kafka client parameters like “session timeout”, max.poll.records or max.poll.interval.ms.

Solution

I’ve investigated default values of above-mentioned Kafka client parameters. It turned out the default value for max.poll.records is 500. And as processing of these messages was rather heavy it was quite likely all 500 messages won’t be done in 5 minutes (the default value for max.poll.interval.ms).

Setting the max.poll.records parameter to value of 5 solved the issue. When using Spring Boot 2 this can be done by setting this application property:

spring.kafka.consumer.max-poll-records: 5

Conclusions

In my opinion the default value of 500 for max.poll.records parameter is rather crazy for general purpose usage. And this story is a good lesson for Kafka lovers who see it as a general purpose messaging technology. The truth is Kafka client configuration is complex and without good understanding and quite impressive knowledge of its configuration parameters one can easily shots his own feet.

About krzysztoftomaszewski

I've got M.Sc. in software engineering. I graduated in 2005 at Institute of Computer Science, Warsaw University of Technology, Faculty of Electronics and Information Technology. I'm working on computer software design and engineering continuously since 2004.
This entry was posted in Java, messaging, Spring and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s