kafka 集群环境搭建weixin45749011的博客-

1. 初始化环境准备

安装jdk，安装zookeeper并保证zk服务正常启动

2. 下载安装包并上传解压

node01执行以下命令，下载并解压

[root@node01 ~]# cd /export/soft [root@node01 soft]# wget https://archive.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz [root@node01 soft]# tar –zxvf  kafka_2.11-1.0.0.tgz -C /export/servers/ 

3. node01服务器修改kafka配置文件

node01执行以下命令进入到kafka的配置文件目录，修改配置文件
node01执行以下命令创建数据文件存放目录

[root@node01 ~]# mkdir -p  /export/servers/kafka_2.11-1.0.0/logs  [root@node01 ~]# cd /export/servers/kafka_2.11-1.0.0/config [root@node01 config]# vim server.properties 

broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/export/servers/kafka_2.11-1.0.0/logs
num.partitions=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=node01:2181,node02:2181,node03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
host.name=node01

4. 安装包分发到其他服务器上面去

node01执行以下命令，将node01服务器的kafka安装包发送到node02和node03服务器上面去

[root@node01 ~]# cd /export/servers/ [root@node01 servers]# scp -r kafka_2.11-1.0.0/ node02:$PWD [root@node01 servers]# scp -r kafka_2.11-1.0.0/ node03:$PWD 

5. node02与node03服务器修改配置文件

node02与node03服务器修改kafka配置文件
node02使用以下命令修改kafka配置文件

[root@node02 ~]# cd /export/servers/kafka_2.11-1.0.0/config [root@node02 config]# vim server.properties 

broker.id=1
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/export/servers/kafka_2.11-1.0.0/logs
num.partitions=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=node01:2181,node02:2181,node03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
host.name=node02

node03使用以下命令修改kafka配置文件

[root@node03 ~]# cd /export/servers/kafka_2.11-1.0.0/config [root@node03 config]# vim server.properties 

broker.id=2
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/export/servers/kafka_2.11-1.0.0/logs
num.partitions=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=node01:2181,node02:2181,node03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
host.name=node03

6. kafka集群启动与停止

注意事项：在kafka启动前，一定要让zookeeper启动起来。
node01执行以下命令将kafka进程启动在后台

[root@node01 ~]# cd /export/servers/kafka_2.11-1.0.0 [root@node01 kafka_2.11-1.0.0]# nohup bin/kafka-server-start.sh config/server.properties 2>&1 & 

node02执行以下命令将kafka进程启动在后台

[root@node02 ~]# cd /export/servers/kafka_2.11-1.0.0 [root@node02 kafka_2.11-1.0.0]# nohup bin/kafka-server-start.sh config/server.properties 2>&1 & 

node03执行以下命令将kafka进程启动在后台

[root@node03 ~]# cd /export/servers/kafka_2.11-1.0.0 [root@node03 kafka_2.11-1.0.0]# nohup bin/kafka-server-start.sh config/server.properties 2>&1 & 

三台机器也可以执行以下命令停止kafka集群

[root@node01 ~]# cd /export/servers/kafka_2.11-1.0.0 [root@node02 kafka_2.11-1.0.0]# bin/kafka-server-stop.sh 

此操作过于繁琐，简便方式 —> Kafka一键启动脚本