副标题[/!--empirenews.page--]
简介
最近一个 MongoDB 集群环境中的某节点异常下电了,导致业务出现了中断,随即又恢复了正常。
通过ELK 告警也监测到了业务报错日志。
运维部对于节点下电的原因进行了排查,发现仅仅是资源分配上的一个失误导致。 在解决了问题之后,大家也对这次中断的也提出了一些问题:
"当前的 MongoDB集群 采用了分片副本集的架构,其中主节点发生故障会产生多大的影响?"
"MongoDB 副本集不是能自动倒换吗,这个是不是秒级的?"
带着这些问题,下面针对副本集的自动Failover机制做一些分析。
日志分析
首先可以确认的是,这次掉电的是一个副本集上的主节点,在掉电的时候,主备关系发生了切换。
从另外的两个备节点找到了对应的日志:
备节点1的日志
- 2019-05-06T16:51:11.766+0800 I REPL [ReplicationExecutor] Starting an election, since we've seen no PRIMARY in the past 10000ms
- 2019-05-06T16:51:11.766+0800 I REPL [ReplicationExecutor] conducting a dry run election to see if we could be elected
- 2019-05-06T16:51:11.766+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 172.30.129.78:30071
- 2019-05-06T16:51:11.767+0800 I REPL [ReplicationExecutor] VoteRequester(term 3 dry run) received a yes vote from 172.30.129.7:30071; response message: { term: 3, voteGranted: true, reason: "", ok: 1.0 }
- 2019-05-06T16:51:11.767+0800 I REPL [ReplicationExecutor] dry election run succeeded, running for election
- 2019-05-06T16:51:11.768+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 172.30.129.78:30071
- 2019-05-06T16:51:11.771+0800 I REPL [ReplicationExecutor] VoteRequester(term 4) received a yes vote from 172.30.129.7:30071; response message: { term: 4, voteGranted: true, reason: "", ok: 1.0 }
- 2019-05-06T16:51:11.771+0800 I REPL [ReplicationExecutor] election succeeded, assuming primary role in term 4
- 2019-05-06T16:51:11.771+0800 I REPL [ReplicationExecutor] transition to PRIMARY
- 2019-05-06T16:51:11.771+0800 I REPL [ReplicationExecutor] Entering primary catch-up mode.
- 2019-05-06T16:51:11.771+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Ending connection to host 172.30.129.78:30071 due to bad connection status; 2 connections to that host remain open
- 2019-05-06T16:51:11.771+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 172.30.129.78:30071
- 2019-05-06T16:51:13.350+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 172.30.129.78:30071; ExceededTimeLimit: Couldn't get a connection within the time limit
备节点2的日志
- 2019-05-06T16:51:12.816+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Ending connection to host 172.30.129.78:30071 due to bad connection status; 0 connections to that host remain open
- 2019-05-06T16:51:12.816+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 172.30.129.78:30071; ExceededTimeLimit: Operation timed out, request was RemoteCommand 72553 -- target:172.30.129.78:30071 db:admin expDate:2019-05-06T16:51:12.816+0800 cmd:{ replSetHeartbeat: "shard0", configVersion: 96911, from: "172.30.129.7:30071", fromId: 1, term: 3 }
- 2019-05-06T16:51:12.821+0800 I REPL [ReplicationExecutor] Member 172.30.129.160:30071 is now in state PRIMARY
可以看到,备节点1在 16:51:11 时主动发起了选举,并成为了新的主节点,随即备节点2在 16:51:12 获知了最新的主节点信息,因此可以确认此时主备切换已经完成。
同时在日志中出现的,还有对于原主节点(172.30.129.78:30071)大量心跳失败的信息。
那么,备节点具体是怎么感知到主节点已经 Down 掉的,主备节点之间的心跳是如何运作的,这对数据的同步复制又有什么影响?
下面,我们挖掘一下 ** 副本集的 自动故障转移(Failover)** 机制
副本集 如何实现 Failover
(编辑:湘西站长网)
【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容!
|