PYTHON-1232: protocol v5 out of beta

[aboudreault]

No description provided.

[aboudreault]

@beobal so with those changes, things look good for C* <4. However, I'm having some failures with a secure 4.0 cluster. There is probably something missing in the driver. If I send bad creds, I can see that the auth failed response is not wrapped (checksumming). When sending good creds, it seems that the auth succeed response is wrapped, is that correct?

[beobal]

@aboudreault that doesn't sound right, any response sent from server to client after the server has received the STARTUP should be checksummed (for v5). Could be a server side bug, I'll check it out and file a JIRA if so.

[beobal]

@aboudreault it's a driver issue, looks like I missed enabling checksumming for v5 in the authentication case. This patch fixes it:

diff --git a/cassandra/connection.py b/cassandra/connection.py
index 477eaf2f..ca1e7531 100644
--- a/cassandra/connection.py
+++ b/cassandra/connection.py
@@ -1345,6 +1345,9 @@ class Connection(object):
                           "if DSE authentication is configured with transitional mode" % (self.host,))
                 raise AuthenticationFailed('Remote end requires authentication')

+            if ProtocolVersion.has_checksumming_support(self.protocol_version):
+                self._enable_checksumming()
+
             if isinstance(self.authenticator, dict):
                 log.debug("Sending credentials-based auth response on %s", self)
                 cm = CredentialsMessage(creds=self.authenticator)

[aboudreault]

That is indeed something I have already tried. I get a CrcMismatchException when enabling checksumming just before sending the auth message. I will retest everything with the CASSANDRA-14973's branch today.

[beobal]

That is indeed something I have already tried. I get a CrcMismatchException when enabling checksumming just before sending the auth message. I will retest everything with the CASSANDRA-14973's branch today.

Interesting, I definitely see an error without that patch, but not with it. When you say "just before sending the auth message", do you mean the AuthResponseMessage? Where is the exception being thrown from?

[aboudreault]

I just push your change in this branch. So with this driver branch and this C* branch: https://github.com/beobal/cassandra/commits/14973-trunk

I create a cluster with ccm and this config:

authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer

I'm getting this error:

cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'127.0.0.1:9042': CrcMismatchException('CRC mismatch on header e20070. Received a0400", computed 46c363.')})

which is raised by the server:

ERROR [epollEventLoopGroup-5-4] 2021-02-10 08:33:58,753 NoSpamLogger.java:98 - 60c62165 invalid, unrecoverable CRC mismatch detected in frame header. Read 1421, Computed 5104812

Were you testing the same auth configuration?

[beobal]

I wasn't setting the authorizer, but that shouldn't matter anyway as it's not used during authentication. Either way, I've followed your steps and created a single node ccm cluster from that branch, with those settings. Using this driver branch I see this:

>>> from cassandra.cluster import Cluster
>>> from cassandra.auth import PlainTextAuthProvider
>>> auth_provider=PlainTextAuthProvider(username='cassandra', password='cassandra')
>>> c=Cluster(auth_provider=auth_provider)
>>> s=c.connect()
>>>
>>> auth_provider=PlainTextAuthProvider(username='cassandra', password='cassandrax')
>>> c=Cluster(auth_provider=auth_provider)
>>> s=c.connect()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cassandra/cluster.py", line 1690, in connect
    self.control_connection.connect()
  File "cassandra/cluster.py", line 3488, in connect
    self._set_new_connection(self._reconnect_internal())
  File "cassandra/cluster.py", line 3533, in _reconnect_internal
    raise NoHostAvailable("Unable to connect to any servers", errors)
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'127.0.0.1:9042': AuthenticationFailed('Failed to authenticate to 127.0.0.1:9042: Error from server: code=0100 [Bad credentials] message="Provided username cassandra and/or password are incorrect"',)})
>>>

The fact that it's your server logging the CRC error means the client isn't properly encoding the request. Am I doing something different here?

[beobal]

Also, just to be sure, I verified that my client is actually connecting over v5:

❯ bin/nodetool -p 7100  clientstats --by-protocol
Clients by protocol version

Protocol-Version IP-Address Last-Seen
5/v5             /127.0.0.1 Feb 10, 2021 15:07:18

[aboudreault]

looks like we are doing the same thing then. that must be something in my env... I will continue to debug to understand what's going on..

[aboudreault]

Ok. I found the issue in the driver. You can also reproduce it if you install the lz4 package in your env, to enable compression. The compression wasn't enabled properly before sending the auth message. I should be good to merge this PR today.

[beobal]

nice catch! confirmed I can repro with compression enabled.

[aboudreault]

Merging. Thanks!