replication block when network down

[iampastor]

when the network is down (such as unplug the network cable) and recover after, the replication client will block at ReadPacket, this is the info of pprof:

goroutine 19 [IO wait, 14 minutes]:
net.runtime_pollWait(0x7fe48c176b70, 0x72, 0xc82177e000)
    /home/cdyf/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82176a760, 0x72, 0x0, 0x0)
    /home/cdyf/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82176a760, 0x0, 0x0)
    /home/cdyf/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82176a700, 0xc82177e000, 0x1000, 0x1000, 0x0, 0x7fe48c171050, 0xc82000a0f8)
    /home/cdyf/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc821768120, 0xc82177e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/cdyf/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc8217783c0)
    /home/cdyf/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8217783c0, 0xc821c66148, 0x4, 0x4, 0x8, 0x0, 0x0)
    /home/cdyf/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fe48c131078, 0xc8217783c0, 0xc821c66148, 0x4, 0x4, 0x4, 0x0, 0x0, 0x0)
    /home/cdyf/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fe48c131078, 0xc8217783c0, 0xc821c66148, 0x4, 0x4, 0x20, 0x0, 0x0)
    /home/cdyf/go/src/io/io.go:315 +0x62
mysql_byroad/vendor/github.com/siddontang/go-mysql/packet.(*Conn).ReadPacketTo(0xc82101e8c0, 0x7fe48c175d78, 0xc821a388c0, 0x0, 0x0)
    /home/cdyf/goworkspace/src/mysql_byroad/vendor/github.com/siddontang/go-mysql/packet/conn.go:81 +0xf1
mysql_byroad/vendor/github.com/siddontang/go-mysql/packet.(*Conn).ReadPacket(0xc82101e8c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/cdyf/goworkspace/src/mysql_byroad/vendor/github.com/siddontang/go-mysql/packet/conn.go:35 +0x92
mysql_byroad/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).onStream(0xc821717480, 0xc82101eca0)
    /home/cdyf/goworkspace/src/mysql_byroad/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:479 +0xa8
created by mysql_byroad/vendor/github.com/siddontang/go-mysql/replication.(*BinlogSyncer).startDumpStream
    /home/cdyf/goworkspace/src/mysql_byroad/vendor/github.com/siddontang/go-mysql/replication/binlogsyncer.go:220 +0x111

I think this is because mysql server closed the connection and replication client didn't receive any close data because of bad network, so it blocked at reading data from this connection. Is there any solution for this condition?

[siddontang]

Hi @iampastor

I think we can add a read timeout here. But if the connection is good and we don't receive any binlog from MySQL after the timeout, the connection will be closed too and go-mysql will reconnect it.
Another way is to update the TCP heartbeat interval parameters with sysctrl.

[iampastor]

@siddontang ye, this is what I did, adding a read timeout, but I think this is not a very good solution, Does mysql replication have any heart beat message or other keep alive mechanism so that we can check whether the connection is bad?

[siddontang]

@iampastor
MySQL replication has no heartbeat message, so I can't figure out other ways.

[iampastor]

@siddontang
ok, I see. thanks a lot. I will use time out to check the connection.

[siddontang]

Thanks @iampastor
You can send me a PR.

[3manuek]

@iampastor You can keep the coordinates, retry the connection after timeout and set a limit to the loop.

@siddontang I see that you can enable heartbeat by using semi-sync through Slave_heartbeat_period, don't know how the underlying protocol works tho :( .

[siddontang]

Hi @3manuek , I don't find any protocol document on how to support Slave_heartbeat_period 😭

[3manuek]

Yeah, the only entry on the docs is this one. So, I dug into the Binlog_sender::send_heartbeat_event at sql/rpl_binlog_sender.cc and I see that it is just an "empty" event (it only has the checksum if enabled).

But mostly it just sets the header and nothing more:

 /* Timestamp field */
  int4store(header, 0);
  header[EVENT_TYPE_OFFSET] = binary_log::HEARTBEAT_LOG_EVENT;
  int4store(header + SERVER_ID_OFFSET, server_id);
  int4store(header + EVENT_LEN_OFFSET, event_len);
  int4store(header + LOG_POS_OFFSET, static_cast<uint32>(log_pos));
  int2store(header + FLAGS_OFFSET, 0);

So dealing with it shouldn't be more issue as it is a generic event \o/

The tricky part I guess will be how to deal with the timeout tho. The MASTER_HEARTBEAT_PERIOD can be set at replication.BinlogSyncerConfig, but a timer should be set at the beginning of the streamer and reset on each heartbeat event. Seems like it can be handled manually outside the library by just providing the necessary interface to handle the event (it can be up to the user on how to deal with it).

[siddontang]

Hi @3manuek

The HEARTBEAT_EVENT is sent by the master periodically. We can add MASTER_HEARTBEAT_PERIOD configuration.
If set (>0, maybe should better > the time configured in MySQL), we will set read timeout for the connection, and we will retry after the connection is closed.

[picago]

just use "SELECT 'x'" to keep the coordinates?

[lkzz]

if client.conn is closed periodically，connection maybe not close on master.

issue like this: alibaba/canal#139

[siddontang]

Hi @lkpjj

Can mysql keepalive not close the tombstone connection? Seem this is the problem for all client connections.