The second node can't be joined in the innodb-cluster (MySQL 8.0.11)

[hashstone]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

MySQL Operator Version:
d9b3b4f

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• mysql:
  8.0.11

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"

• Kernel (e.g. uname -a):

10.5.6.13   Ready         4d        v1.10.3           CentOS Linux 7 (Core)   3.10.0-693.5.2.el7.x86_64   docker://17.3.1
10.5.6.14   Ready     master    6d        v1.10.3           CentOS Linux 7 (Core)   3.10.0-693.5.2.el7.x86_64   docker://17.3.1
10.5.6.17   Ready         6d        v1.10.3           CentOS Linux 7 (Core)   3.10.0-693.5.2.el7.x86_64   docker://17.3.1

• Others:

cluster-with-3-replicas.yaml

apiVersion: "mysql.oracle.com/v1alpha1"
kind: Cluster
metadata:
  name: mysql-cluster
spec:
  members: 3
  baseServerId: 2000

00-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: mysql-operator

01-resources.yaml

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: mysqlclusters.mysql.oracle.com
spec:
  group: mysql.oracle.com
  version: v1alpha1
  scope: Namespaced
  names:
    kind: Cluster
    singular: mysqlcluster
    plural: mysqlclusters
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: mysqlbackups.mysql.oracle.com
spec:
  group: mysql.oracle.com
  version: v1alpha1
  scope: Namespaced
  names:
    kind: Backup
    singular: mysqlbackup
    plural: mysqlbackups
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: mysqlrestores.mysql.oracle.com
spec:
  group: mysql.oracle.com
  version: v1alpha1
  scope: Namespaced
  names:
    kind: Restore
    singular: mysqlrestore
    plural: mysqlrestores
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: mysqlbackupschedules.mysql.oracle.com
spec:
  group: mysql.oracle.com
  version: v1alpha1
  scope: Namespaced
  names:
    kind: BackupSchedule
    singular: mysqlbackupschedule
    plural: mysqlbackupschedules

02-rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mysql-operator
  namespace: mysql-operator
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mysql-agent
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: mysql-operator
rules:
  - apiGroups:
    - "*"
    resources:
    - "*"
    verbs:
    - "*"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: mysql-operator
  namespace: mysql-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind:  ClusterRole
  name: mysql-operator
subjects:
- kind: ServiceAccount
  name: mysql-operator
  namespace: mysql-operator
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: mysql-agent
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mysql-operator
subjects:
- kind: ServiceAccount
  name: mysql-agent
  namespace: default

03-deployment.yaml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mysql-operator
  namespace: mysql-operator
  labels:
    release: 0.1.0
    app: mysql-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-operator
  template:
    metadata:
      labels:
        app: mysql-operator
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      serviceAccountName: mysql-operator
      containers:
      - name: mysql-operator-controller
        imagePullPolicy: Always
        image: 10.5.6.10/mysql/mysql-operator:d9b3b4ff-2018062814
        ports:
        - containerPort: 10254
        args:
          - --v=4
          - --mysql-agent-image=10.5.6.10/mysql/mysql-agent
          - --mysql-server-image=10.5.6.10/mysql/mysql-server

What happened?

The second mysql node can't be joined in cluster, the following is error msg in the second mysql node:

2018-06-28T07:24:22.399692Z 0 [ERROR] [MY-011526] [Repl] Plugin group_replication reported: 'This member has more executed transactions than those present in the group. Local transactions: 41200517-7aa0-11e8-b7f2-0a58ac6000d9:1-5 > Group transactions: 2a7148a3-7aa0-11e8-ab02-0a58ac6003c8:1-19,
373b0f6e-7aa0-11e8-aa88-0a58ac6003c8:1-442'
2018-06-28T07:24:22.399742Z 0 [ERROR] [MY-011522] [Repl] Plugin group_replication reported: 'The member contains transactions not present in the group. The member will now exit the group.'
2018-06-28T07:24:22.399773Z 0 [Note] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to mysql-cluster-0.mysql-cluster:3306,  on view 15301689195655916:126.'
2018-06-28T07:24:22.399800Z 1097 [Note] [MY-011647] [Repl] Plugin group_replication reported: 'Going to wait for view modification'

What you expected to happen?

The second mysql node could be joined in the group

How to reproduce it (as minimally and precisely as possible)?

1. kubectl create -f 00-namespace.yaml
2. kubectl create -f 01-resources.yaml
3. kubectl create -f 02-rbac.yaml
4. kubectl create -f 03-deployment.yaml
5. kubectl create -f cluster-with-3-replicas.yaml

Anything else we need to know?

[prydie]

Hi @hashstone, apologies for the delay. Are you still seeing the issue? Unfortunately, I haven't been able to reproduce it.

[hashstone]

Hi @prydie, I check it again with mysql (8.0.11) whose imageID is a8a59477268d. Have reproduced.

clue

1. Error log:
   fail-mysql.log
2. show master status:

mysql> show master status;
+----------------------------+----------+--------------+------------------+------------------------------------------+
| File                       | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                        |
+----------------------------+----------+--------------+------------------+------------------------------------------+
| mysql-cluster-1-bin.000003 |      191 |              |                  | 9fc2420b-866c-11e8-b809-0a58ac10006c:1-5 |
+----------------------------+----------+--------------+------------------+------------------------------------------+

3. binlog:

[root@localhost mysql_operator_8.0]# kubectl exec -it mysql-cluster-1 -- mysqlbinlog /var/lib/mysql/mysql-cluster-1-bin.000003
Defaulting container name to mysql.
Use 'kubectl describe pod/mysql-cluster-1 -n default' to see all of the containers in this pod.
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#180713  7:16:21 server id 2001  end_log_pos 124 	Start: binlog v 4, server v 8.0.11 created 180713  7:16:21 at startup
# Warning: this binlog is either in use or was not closed properly.
ROLLBACK/*!*/;
BINLOG '
xVFIWw/RBwAAeAAAAHwAAAABAAQAOC4wLjExAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAADFUUhbEwANAAgAAAAABAAEAAAAYAAEGggAAAAICAgCAAAACgoKKioAEjQA
CgA3osjb
'/*!*/;
# at 124
#180713  7:16:21 server id 2001  end_log_pos 191 	Previous-GTIDs
# 9fc2420b-866c-11e8-b809-0a58ac10006c:1-5
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

4. mysql status:

mysql> status
--------------
mysql  Ver 8.0.11 for Linux on x86_64 (MySQL Community Server - GPL)

Connection id:		1299
Current database:	
Current user:		root@localhost
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		8.0.11 MySQL Community Server - GPL
Protocol version:	10
Connection:		Localhost via UNIX socket
Server characterset:	utf8mb4
Db     characterset:	utf8mb4
Client characterset:	latin1
Conn.  characterset:	latin1
UNIX socket:		/var/run/mysqld/mysqld.sock
Uptime:			49 min 53 sec

Threads: 8  Questions: 14896  Slow queries: 0  Opens: 472  Flush tables: 2  Open tables: 448  Queries per second avg: 4.976
--------------

Some attempts

1. Notice that SET @@SESSION.GTID_NEXT= 'AUTOMATIC' is not done by operator, I update mysql-image to 02d081b9c73e
2. Then restart cluster, it turns be OK.
3. So what is the problem? Any advice would be greatly appreciated! THX~!