消息队列apache rocketmq4.2入门(一)

原创 2018-03-09 10:56 阅读(1700)次
RocketMQ是一款分布式、队列模型的消息中间件,具有以下特点:
    支持严格的消息顺序
    支持Topic与Queue两种模式
    亿级消息堆积能力
    比较友好的分布式特性
    同时支持Push与Pull方式消费消息
    历经多次天猫双十一海量消息考验

下面给大家分享我的入门笔记:

可以配置单Master,双(多)Master,Master-Slave三种模式
这里我搭建的是双Master模式
优点:配置简单,单个Master宕机或重启维护对应用无影响,在磁盘配置为RAID10时,即使机器宕机不可恢复情况下,由于RAID10磁盘非常可靠,消息也不会丢(异步刷盘丢失少量消息,同步刷盘一条不丢)。性能最高。
缺点:单台机器宕机期间,这台机器上未被消费的消息在机器恢复之前不可订阅,消息实时性会收到影响

官网:
http://rocketmq.apache.org/
下载:
https://www.apache.org/dyn/closer.cgi?path=rocketmq/4.2.0/rocketmq-all-4.2.0-bin-release.zip
一、linux下安装:
这里示范的是双主模式,两台机的安装和配置基本一致:

1.准备了两台虚拟机
设置两台机的hosts文件如下,vi /etc/hosts:
192.168.10.128 rocketmq-nameserver1
192.168.10.128 rocketmq-master1
192.168.10.130 rocketmq-nameserver2
192.168.10.130 rocketmq-master2

2.重启两台机的网卡
service network restart
检查是否能ping通各自
192.168.10.128:
ping rocketmq-nameserver2
ping rocketmq-master2

192.168.10.130:
ping rocketmq-nameserver1
ping rocketmq-master1

3.上传下载好的RocketMQ包到两台服务器并解压
下载地址:https://www.apache.org/dyn/closer.cgi?path=rocketmq/4.2.0/rocketmq-all-4.2.0-bin-release.zip
我下载的是zip包,上传到/home/apps下解压:
unzip rocketmq-all-4.2.0-bin-release.zip -d rocketmq-4.2.0

4.创建两台存储目录
mkdir /home/apps/rocketmq-4.2.0/store
mkdir /home/apps/rocketmq-4.2.0/store/commitlog
mkdir /home/apps/rocketmq-4.2.0/store/consumequeue
mkdir /home/apps/rocketmq-4.2.0/store/index

5.配置两台broker
由于我们搭建的是双主模式,只要配置主节点就可以了
cd /home/apps/rocketmq-4.2.0/conf/2m-noslave
配置broker-a.properties,broker-b.properties (双主的broker只要brokerName不一样就可以了,例如broker-b.properties中的brokerName=broker-b,如果搭建的是三主就多加一配置即可,以此类推)
配置如下:


#所属集群名字
borkerClusterName=rocketmq-cluster
#broker名字,注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示Master, >0 表示Slave
brokerId=0
#nameServer地址,分号分割,此例中只有两台主机,所以写两台就可以,以此类推
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
#在发消息时,自动创建服务器不存在的topic,默认创建的队列数
defaultTopicQueueNums=4
#是否允许Broker自动创建Topic,建议线下开启,线上关闭
autoCreateTopicEnable=true
#是否允许Broker自动创建订阅组,建议线下开启,线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点,默认凌晨4点
deleteWhen=04
#文件保留时间,默认48小时
fileReservedTime=120
#commitLog每个文件的大小默认1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/apps/rocketmq-4.2.0/store
#commitLog存储路径
storePathCommitLog=/home/apps/rocketmq-4.2.0/store/commitlog
#消费队列存储路径
storePathConsumeQueue=/home/apps/rocketmq-4.2.0/store/consumequeue
#消息索引存储路径
storePathIndex=/home/apps/rocketmq-4.2.0/store/index
#checkpoint 文件存储路径
storeCheckPoint=/home/apps/rocketmq-4.2.0/store/checkpoint
#abort 文件存储路径
abortFile=/home/apps/rocketmq-4.2.0/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制Master
#- SYNC_MASTER 同步双写Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH  异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128
6.配置两台日志文件
创建日志目录
mkdir /home/apps/rocketmq-4.2.0/logs
修改日志配置文件,替换conf目录下的logback_*.xml中的${user.home}为我们的安装目录/home/apps/rocketmq-4.2.0
cd conf
sed -i 's#${user.home}#/home/apps/rocketmq-4.2.0#g' *.xml

查看logback_*.xml中的${user.home}有没有被替换成功:

<appender name="DefaultAppender"
		  class="ch.qos.logback.core.rolling.RollingFileAppender">
	<file>/home/apps/rocketmq-4.2.0/logs/rocketmqlogs/broker_default.log</file>
	<append>true</append>
	<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
		<fileNamePattern>/home/apps/rocketmq-4.2.0/logs/rocketmqlogs/otherdays/broker_default.%i.log.gz</fileNamePattern>
		<minIndex>1</minIndex>
		<maxIndex>10</maxIndex>
	</rollingPolicy>
	<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
		<maxFileSize>100MB</maxFileSize>
	</triggeringPolicy>
	<encoder>
		<pattern>%d{yyy-MM-dd HH:mm:ss,GMT+8} %p %t - %m%n</pattern>
		<charset class="java.nio.charset.Charset">UTF-8</charset>
	</encoder>
</appender>
7.修改启动脚本JVM调优
由于默认的JVM内存占用都会比较高,在我的虚拟机里没有这么足的内存,需要调低点,生产环境中要根据自己的内存设置
vi /home/apps/rocketmq-4.2.0/bin/runbroker.sh
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

vi /home/apps/rocketmq-4.2.0/bin/runserver.sh
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

注意在老版本有人可能会设置PermSize与MaxPermSize这两个参数,但我用的是rocketmq-4.2.0,必须是jdk8,所以这里就不能设置PermSize与MaxPermSize这两个参数了,jdk8不支持

8.启动两台NameServer
进入bin目录
cd /home/apps/rocketmq-4.2.0/bin
nohup sh mqnamesrv >start.log 2>&1 &
查看日志是不是启动成功
[root@jary bin]# cat start.log 
Java HotSpot(TM) 64-Bit Server VM warning: Using the DefNew young collector with the CMS collector is deprecated and will likely be removed in a future release
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
The Name Server boot success. serializeType=JSON

也可以用jps查看启动进程
jps

9.启动两台BrokerServer
Master1:
cd /home/apps/rocketmq-4.2.0/bin
nohup sh mqbroker -c /home/apps/rocketmq-4.2.0/conf/2m-noslave/broker-a.properties >/dev/null 2>&1 &

Master2:
cd /home/apps/rocketmq-4.2.0/bin
nohup sh mqbroker -c /home/apps/rocketmq-4.2.0/conf/2m-noslave/broker-b.properties >/dev/null 2>&1 &

可以查看日志:
/home/apps/rocketmq-4.2.0/logs/rocketmqlogs/broker.log
Master1:

2018-03-07 17:13:59 INFO main - Set user specified name server address: rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
2018-03-07 17:13:59 INFO PullRequestHoldService - PullRequestHoldService service started
2018-03-07 17:13:59 INFO main - register broker to name server rocketmq-nameserver2:9876 OK
2018-03-07 17:14:00 INFO main - register broker to name server rocketmq-nameserver1:9876 OK
2018-03-07 17:14:00 INFO main - The broker[broker-a, 192.168.10.128:10911] boot success. serializeType=JSON and name server is rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
2018-03-07 17:14:09 INFO BrokerControllerScheduledThread1 - dispatch behind commit log 0 bytes
2018-03-07 17:14:09 INFO BrokerControllerScheduledThread1 - Slave fall behind master: 0 bytes
2018-03-07 17:14:10 INFO BrokerControllerScheduledThread1 - register broker to name server rocketmq-nameserver2:9876 OK
2018-03-07 17:14:10 INFO BrokerControllerScheduledThread1 - register broker to name server rocketmq-nameserver1:9876 OK
Master2:

2018-03-07 17:18:49 INFO main - Set user specified name server address: rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
2018-03-07 17:18:50 INFO PullRequestHoldService - PullRequestHoldService service started
2018-03-07 17:18:50 INFO main - register broker to name server rocketmq-nameserver2:9876 OK
2018-03-07 17:18:50 INFO main - register broker to name server rocketmq-nameserver1:9876 OK
2018-03-07 17:18:50 INFO main - The broker[broker-b, 192.168.10.130:10911] boot success. serializeType=JSON and name server is rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
2018-03-07 17:19:00 INFO BrokerControllerScheduledThread1 - dispatch behind commit log 0 bytes
2018-03-07 17:19:00 INFO BrokerControllerScheduledThread1 - Slave fall behind master: 0 bytes
2018-03-07 17:19:00 INFO BrokerControllerScheduledThread1 - register broker to name server rocketmq-nameserver2:9876 OK
2018-03-07 17:19:01 INFO BrokerControllerScheduledThread1 - register broker to name server rocketmq-nameserver1:9876 OK
也可以用jps查看到进程:
[root@jary bin]# jps
4050 BrokerStartup
3987 NamesrvStartup
4106 Jps

10停止服务,先停broker,再停NameServer
停止broker
./mqshutdown broker
停止NameServer
./mqshutdown namesrv

下篇文章我会给大家分享在java端的代码:消息队列apache rocketmq4.2入门(二)