Consumers Basics and Configuration in Kafka

This guide covers the basics of Kafka consumers and their configuration. Consumers are responsible for reading records from Kafka topics. Understanding how to configure consumers is essential for processing and managing data retrieved from Kafka.

1. What is a Kafka Consumer?

A Kafka consumer is an application or component that reads records from Kafka topics. Consumers play a crucial role in processing the data produced by Kafka producers. They subscribe to topics and pull data from the Kafka cluster.

2. Basic Configuration Options

Kafka consumers can be configured using various properties to control their behavior. Key configuration options include:

3. Example: Basic Kafka Consumer Configuration

Below is a Java example demonstrating how to configure and use a Kafka consumer to read messages from a Kafka topic.


import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.Collections;
import java.util.Properties;

public class KafkaConsumerExample {
    public static void main(String[] args) {
        // Consumer configuration
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-consumer-group");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
        
        // Create KafkaConsumer
        KafkaConsumer consumer = new KafkaConsumer<>(props);
        
        // Subscribe to topic
        consumer.subscribe(Collections.singletonList("my-topic"));
        
        // Poll for new records
        while (true) {
            ConsumerRecords records = consumer.poll(1000);
            
            records.forEach(record -> {
                System.out.printf("Consumed record with key %s and value %s from topic %s%n",
                                  record.key(), record.value(), record.topic());
            });
        }
        
        // Close the consumer (not reached in this example)
        // consumer.close();
    }
}
    

4. Kafka Consumer Configuration Diagram

The following diagram illustrates the key components involved in Kafka consumer configuration, including the Kafka broker connection, deserialization, and offset management.

Kafka Consumer Configuration Diagram

Diagram: Kafka Consumer Configuration Components

5. Conclusion

Kafka consumers are essential for reading and processing data from Kafka topics. Properly configuring consumers is important for efficient data consumption and processing. By understanding and applying the various configuration options, you can optimize your Kafka consumers to meet the needs of your application.