Kafka topic分区增加副本

Kafka中topic的每个分区可以设置多个副本。如果副本数为1，当该分区副本的leader节点宕机后，会导致该分区不可用。故需要设置多副本来保证可用性。

实际项目中，存在项目初期创建了副本数为1的topic，但是后期又需要扩大副本数的场景。通常不能直接删除topic重建，可以通过如下操作实现。

准备工作

创建副本为1的topic

kafka-topics --zookeeper mdw:2181/kafka --create --replication-factor 1 --partitions 3 --topic test_topic

查看topic信息

kafka-topics --describe --zookeeper mdw:2181/kafka --topic test_topic
输出：
Topic:test_topic        PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: test_topic       Partition: 0    Leader: 364     Replicas: 364   Isr: 364
        Topic: test_topic       Partition: 1    Leader: 365     Replicas: 365   Isr: 365
        Topic: test_topic       Partition: 2    Leader: 366     Replicas: 366   Isr: 366

可以看出3个分区，各自都是只有一个副本。当Leader对应节点挂掉后，分区就不可用了。

增加为3副本

使用官方自带的

kafka-reassign-partitions.sh

脚本实现。该脚本用来移动分区的副本位置。除了可以实现增加副本，还可以实现将分区的副本移动到指定机器上。该脚本的help见附录。

以下步骤实现将test_topic的各个分区增加为三副本。

1. 制定分区及副本分配策略创建json文件，写入如下内容：填写说明：topic为topic名称，partition为分区号，replicas为broker id的数组{"version":1,"partitions":[{"topic":"test_topic","partition":0,"replicas":[364,365,366]},{"topic":"test_topic","partition":1,"replicas":[364,365,366]},{"topic":"test_topic","partition":2,"replicas":[364,365,366]}]}
2. 执行扩副本操作本例中创建的json文件为test_topic.json执行如下命令：kafka-reassign-partitions --zookeeper mdw:2181/kafka --reassignment-json-file test_topic.json --execute输出内容：Current partition replica assignment{"version":1,"partitions":[{"topic":"test_topic","partition":2,"replicas":[366],"log_dirs":["any"]},{"topic":"test_topic","partition":1,"replicas":[365],"log_dirs":["any"]},{"topic":"test_topic","partition":0,"replicas":[364],"log_dirs":["any"]}]}Save this to use as the --reassignment-json-file option during rollbackSuccessfully started reassignment of partitions.这是个异步操作，执行命令后开始执行扩副本。输出内容中有之前的副本策略，可以保存下来，用于回滚。
3. 查看执行进度kafka-reassign-partitions --zookeeper mdw:2181/kafka --reassignment-json-file test_topic.json --verify出现如下内容说明执行完成Status of partition reassignment: Reassignment of partition test_topic-0 completed successfullyReassignment of partition test_topic-1 completed successfullyReassignment of partition test_topic-2 completed successfully
4. 验证是否成功kafka-topics --describe --zookeeper mdw:2181/kafka --topic test_topic输出：Topic:test_topic PartitionCount:3 ReplicationFactor:3 Configs: Topic: test_topic Partition: 0 Leader: 364 Replicas: 364,365,366 Isr: 364,366,365 Topic: test_topic Partition: 1 Leader: 365 Replicas: 364,365,366 Isr: 365,366,364 Topic: test_topic Partition: 2 Leader: 366 Replicas: 364,365,366 Isr: 366,365,364从输出中可以看出每个分区的副本都变成了3个。

附录

# kafka-reassign-partitions
This tool helps to moves topic partitions between replicas.
Option                                  Description                           
------                                  -----------                           
--bootstrap-server <String: Server(s)   the server(s) to use for              
  to use for bootstrapping>               bootstrapping. REQUIRED if an       
                                          absolute path of the log directory  
                                          is specified for any replica in the 
                                          reassignment json file              
--broker-list <String: brokerlist>      The list of brokers to which the      
                                          partitions need to be reassigned in 
                                          the form "0,1,2". This is required  
                                          if --topics-to-move-json-file is    
                                          used to generate reassignment       
                                          configuration                       
--command-config <String: Admin client  Property file containing configs to be
  property file>                          passed to Admin Client.             
--disable-rack-aware                    Disable rack aware replica assignment 
--execute                               Kick off the reassignment as specified
                                          by the --reassignment-json-file     
                                          option.                             
--generate                              Generate a candidate partition        
                                          reassignment configuration. Note    
                                          that this only generates a candidate
                                          assignment, it does not execute it. 
--help                                  Print usage information.              
--reassignment-json-file <String:       The JSON file with the partition      
  manual assignment json file path>       reassignment configurationThe format
                                          to use is -                         
                                        {"partitions":                        
                                                [{"topic": "foo",                    
                                                  "partition": 1,                    
                                                  "replicas": [1,2,3],               
                                                  "log_dirs": ["dir1","dir2","dir3"] 
                                          }],                                 
                                        "version":1                           
                                        }                                     
                                        Note that "log_dirs" is optional. When
                                          it is specified, its length must    
                                          equal the length of the replicas    
                                          list. The value in this list can be 
                                          either "any" or the absolution path 
                                          of the log directory on the broker. 
                                          If absolute log directory path is   
                                          specified, the replica will be moved
                                          to the specified log directory on   
                                          the broker.                         
--replica-alter-log-dirs-throttle       The movement of replicas between log  
  <Long: replicaAlterLogDirsThrottle>     directories on the same broker will 
                                          be throttled to this value          
                                          (bytes/sec). Rerunning with this    
                                          option, whilst a rebalance is in    
                                          progress, will alter the throttle   
                                          value. The throttle rate should be  
                                          at least 1 KB/s. (default: -1)      
--throttle <Long: throttle>             The movement of partitions between    
                                          brokers will be throttled to this   
                                          value (bytes/sec). Rerunning with   
                                          this option, whilst a rebalance is  
                                          in progress, will alter the throttle
                                          value. The throttle rate should be  
                                          at least 1 KB/s. (default: -1)      
--timeout <Long: timeout>               The maximum time in ms allowed to wait
                                          for partition reassignment execution
                                          to be successfully initiated        
                                          (default: 10000)                    
--topics-to-move-json-file <String:     Generate a reassignment configuration 
  topics to reassign json file path>      to move the partitions of the       
                                          specified topics to the list of     
                                          brokers specified by the --broker-  
                                          list option. The format to use is - 
                                        {"topics":                            
                                                [{"topic": "foo"},{"topic": "foo1"}],
                                        "version":1                           
                                        }                                     
--verify                                Verify if the reassignment completed  
                                          as specified by the --reassignment- 
                                          json-file option. If there is a     
                                          throttle engaged for the replicas   
                                          specified, and the rebalance has    
                                          completed, the throttle will be     
                                          removed                             
--zookeeper <String: urls>              REQUIRED: The connection string for   
                                          the zookeeper connection in the form
                                          host:port. Multiple URLS can be     
                                          given to allow fail-over.