Kafka Advanced Topics Management

Managing Kafka topics effectively involves advanced configurations and operations to optimize performance and ensure reliability. Below are key aspects of advanced topics management.

1. Configuring Topic-Level Settings

Kafka allows fine-grained control over topic settings to optimize performance based on specific use cases.

Example Topic Configuration

# Alter topic configuration
kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --config retention.ms=604800000 # 7 days

# Set the number of partitions
kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --partitions 10

# Configure replication factor
kafka-configs.sh --alter --zookeeper localhost:2181 --entity-type topics --entity-name my-topic --add-config replication.factor=3

2. Managing Topic Partitions

Partition management is crucial for balancing load and optimizing performance.

Reassigning Partitions

# Generate reassignment JSON file
kafka-reassign-partitions.sh --generate --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2" > reassignment.json

# Execute reassignment
kafka-reassign-partitions.sh --execute --zookeeper localhost:2181 --reassignment-json-file reassignment.json

# Verify reassignment
kafka-reassign-partitions.sh --verify --zookeeper localhost:2181 --reassignment-json-file reassignment.json

3. Topic Deletion

Deleting topics should be handled with caution. Ensure you have backups or that the data is no longer needed.

Enable Topic Deletion

# Enable topic deletion in broker configuration
# Set `delete.topic.enable` to `true` in server.properties
delete.topic.enable=true

Delete a Topic

# Delete a topic
kafka-topics.sh --delete --zookeeper localhost:2181 --topic my-topic

4. Topic Cleanup and Maintenance

Regular maintenance tasks help manage disk usage and topic performance.

Cleanup Policy

# Set cleanup policy to delete for a topic
kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --config cleanup.policy=delete

Checking Topic Size

# Check topic size
kafka-log-dirs.sh --describe --broker-id 0