This guide covers the basics of Kafka consumers and their configuration. Consumers are responsible for reading records from Kafka topics. Understanding how to configure consumers is essential for processing and managing data retrieved from Kafka.
A Kafka consumer is an application or component that reads records from Kafka topics. Consumers play a crucial role in processing the data produced by Kafka producers. They subscribe to topics and pull data from the Kafka cluster.
Kafka consumers can be configured using various properties to control their behavior. Key configuration options include:
Below is a Java example demonstrating how to configure and use a Kafka consumer to read messages from a Kafka topic.
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Collections;
import java.util.Properties;
public class KafkaConsumerExample {
public static void main(String[] args) {
// Consumer configuration
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-consumer-group");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
// Create KafkaConsumer
KafkaConsumer consumer = new KafkaConsumer<>(props);
// Subscribe to topic
consumer.subscribe(Collections.singletonList("my-topic"));
// Poll for new records
while (true) {
ConsumerRecords records = consumer.poll(1000);
records.forEach(record -> {
System.out.printf("Consumed record with key %s and value %s from topic %s%n",
record.key(), record.value(), record.topic());
});
}
// Close the consumer (not reached in this example)
// consumer.close();
}
}
The following diagram illustrates the key components involved in Kafka consumer configuration, including the Kafka broker connection, deserialization, and offset management.
Diagram: Kafka Consumer Configuration Components
Kafka consumers are essential for reading and processing data from Kafka topics. Properly configuring consumers is important for efficient data consumption and processing. By understanding and applying the various configuration options, you can optimize your Kafka consumers to meet the needs of your application.