Replication protocol

[lukescott]

I found a fork that implements the replication protocol.

I'm not sure how complete it is. Unfortunately it works off a version of the MySQL driver that is at least a year out of date. My knowledge of the MySQL wire protocol is lacking. I've noticed the writeCommandPacket is quite a bit different.

I'm interested in this because I need a way to update ElasticSearch in real time. Using the replication protocol is attractive because I can turn on/off the syncing and have it pick up where it left off. The current solutions poll MySQL for data, and are more or less straight copies. Also my target (search optimized) data for ElasticSearch is quite a bit different than my MySQL source table.

Would there be any interest in having this feature?

[arnehormann]

Speaking only for myself: I'm open to a performant implementation that doesn't worsen the performance of other parts of the driver (read: should ideally be independent of the rest of the code).
But it will be a very big pull request which is next to impossible to review.
If you can submit something in bite-sized chunks fullfilling the outlined criteria of no performance regressions and a high degree of independence, I'd be reluctantly ok with it.
It will probably not be testable with our CI server. If you do find a way around that, I'm more than willing to drop the "reluctantly" part.

[lukescott]

Agreed. There are some commonality, but I wouldn't change anything that would impact the performance. Do you know if there is a way with the existing functions to write a command packet with more than 1 argument? The writeCommandPacket between the version used in the fork and now is quite a bit different. If I can get that done I can then worry about parsing the data.

I may need some more guidance on "bite-sized chunks".

I'll find out if there is a way to enable binary logging on the CI server.

[arnehormann]

Bite sized chunks (here): a series of commits, each adding up to the whole and each adding just one part of the functionality. I can't say anything more than I want to be able to read and criticise each commit in the pull request on its own and fully understand what it does. That can be:

• add a file with support functions and a bunch of consts / vars
• change funcs in some existing files to prepare usage by new code
• add that usage

I think it's best if you fork the driver and add parts of the functionality with feature branches in your own code so we can keep the noise to a minimum in the main repo.
Write an outline of what you want to do as an issue in your fork. Ask us for comments with @... so we get notified. When we ok your approach, add a checklist you'll keep current by editing it and referencing other pull requests per item.
Per item, create a branch adding what's described in the issue, make a pull request from that branch into your own main branch (didn't try - that's possible, right?) and add a comment referencing us ("@julienschmidt @arnehormann please review") so we get notified. When we say a part of this is finished in the pull request discussion, merge, cross it off and start the cycle with the next one.
When all is done, start a pull request here.

But first, wait for input from @julienschmidt - is this ok, do we want it at all? Give him some time, he's pretty swamped right now. If he says it is: thank you in advance for undertaking an herculean effort.

[julienschmidt]

I must admit that I have no experience with binlog / the replication protocol.
As long as the implementation is nice and doesn't slow down the rest of the driver, I'm open to it.

[lukescott]

If I merge the changes into my own master, won't it be difficult to pull from upstream? Ideally my master branch's history should match upstream. Would it be better to create a main feature branch w/ sub-feature branches? That way I can pull from upstream and merge changes from master and stay up to date.

[arnehormann]

Well, master is just a name, right? You could also merge with a branch called "upstream" tracking our master. That doesn't matter much - unless tools mandate it or I'm missing something else here. Your approach is fine, too. I just want the whole thing to be reviewed during its development and finished before our watchers get flooded with notifications.

[lukescott]

For those that are interested, I have a working implementation. See lukescott#5.

Feedback appreciated!