PressLabs/mysql-operator

Flavius

@AMecea

Yes, you can build you own image, you can inspire from the percona image based on centos, or you can extend the percona mysql 5.7 based on debian. But I'm not sure that backups will work, because, I'm not sure that percona xtrabackup works with MyRocks.

bluven

@bluven

Hello, sorry to bother you. Can anyone tell me does presslabs/mysql-operator support recovery at present?

Flavius

@AMecea

Hi @bluven, the operator uses orchestrator to manage topology recoveries.

So the operator has auto recovery of a cluster indirectly because of using orchestrator.

bluven

@bluven

@AMecea Got it, Thanks.

bluven

@bluven

Hello, everyone. I was trying presslabs/mysql-operator on a k8s 1.12.2 cluster. But after executing helm install presslabs/mysql-operator --name mysql-operator，I found a pvc error with data-mysql-operator-orchestrator-0: no persistent volumes available for this claim and no storage class is set。My cluster didnt have any pv plugins installed. I can install one, But I don't know how to specify storageclass, can any one tell me how to do it? Thanks a lot.

bluven

@bluven

Sorry. Forget to say I was trying mysql-operator 0.2.2.

Jerry Londergaard

@kalium99

If you create a default storage class then it should just pick it up automatically ?

bluven

@bluven

@kalium99 I installed nfs-client-provisioner and set it as default storage class, but I still got the same error. I'll try specify storageClassName next time.

bluven

@bluven

@kalium99 After specifying storageClass，it worked. But there is a new problem: I made a cluster of 4 members, then I changed replica to 3 and memory to 512MB as of being lack of memory. I noticed orchestrator recover the cluster, but one node is lost, later I found there are 2 cluster shown on orchestrator. Is that a orchestrator problem?

smnp

@smanpathak

hey @AMecea in the latest mysql-operator deployed via helm chart (0.2.2?) I see very few prometheus metrics published. All the innodb stats from old operator are missing. Any idea why?

mysql_exporter_last_scrape_error 1
mysql_exporter_scrapes_total 2506
mysql_up 0

smnp

@smanpathak

I also see this following from metrics-exporter logs

time="2019-01-10T21:26:15Z" level=error msg="Error pinging mysqld: Error 1049: Unknown database '&lock_wait_timeout=2'" source="exporter.go:119"

smnp

@smanpathak

looks like i am hitting prometheus/mysqld_exporter#303

smnp

@smanpathak

filed presslabs/mysql-operator#195

Jerry Londergaard

@kalium99

@bluven Sorry I'm not sure, I haven't personally come across that. @AMecea might know ?

bluven

@bluven

Hello, everyone. How do you handle slow query log in production? I want to collect slow query log and save it somewhere. But when I check the spec, there is only one VolumeSpec which is used for datadir.

bluven

@bluven

Hello, everyone. I encountered a blocking delete. I checked mysql-operator log, it kept outputing this: E0111 07:20:58.953852 1 reflector.go:205] github.com/presslabs/mysql-operator/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.MysqlCluster: v1alpha1.MysqlClusterList.Items: []v1alpha1.MysqlCluster: v1alpha1.MysqlCluster.Spec: v1alpha1.MysqlClusterSpec.PodSpec: v1alpha1.PodSpec.MysqlConf: ReadString: expects " or n, but found 1, error found in #10 byte of ...|ry_time":1},"podSpec|..., bigger context ...|ries.log","log_output":"table","long_query_time":1},"podSpec":{"affinity":{},"resources":{"requests"|...
There was indeed a config error. But when I try to delete the cluster, the operation is blocked.

Flavius

@AMecea

Hi @kalium99, when a cluster is partitioned then orchestrator shows it as two clusters, that's the orchestrator way of showing it. I saw that you filled an issue, thanks! When the configs are updated the pods get restarted and the orchestartor does the failover but after few seconds the node should be configure by orchestrator as slave. I will try to reproduce it and update the issue.

Flavius

@AMecea

A cluster cannot be deleted because of the finalizers that are set on the cluster for safety removal. But the issue is with improper configs that tries to be unmarshal into strings but they are numbers and failed and the operator cannot remove the finalizers. I will try to figure out a solution for this cases, maybe using validation at API level will help.

Thanks @smanpathak and @bluven for the issues, nice catch!

Flavius

@AMecea

@bluven for slow logs I think you can redirect them to stdout

Flavius

@AMecea

Hey @/all, we are moving the conversation on the official Kubernetes Slack workspace, under #mysql-operator channel. We invite you to pick-up the discussion there.

Giovanbattista Amato

@gioamato

Hi there, what happens if i update cluster yaml? (precisely i want to change resources.requests) Will the cluster be recreated? And what about pvc? Will i lose data?

Flavius

@AMecea

Hi @gioamato, the cluster will be updated this means that the pods will be restarted (because you want to change resources) and the PVC will NOT be deleted.

So you won't lose data, if the cluster has more than 1 replica then you wont have downtime either.

Giovanbattista Amato

@gioamato

Thank you! As expected.

Just one last question: we have a deployed cluster with 2 replicas (1 master). No issues for more that a month, suddenly this morning the slave replica started to fail in state Waiting: CrashLoopBackOff with the error

Readiness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) /bin/sh: line 0: test: -eq: unary operator expected

I've tried to delete the pod but it keeps restarting in this state

Any advice on recovering it?

(I'm on latest mysql-operator version)

Flavius

@AMecea

do you use some extra or custom configurations for the cluster?

are you using other mysql version?

Giovanbattista Amato

@gioamato

Pretty "stock" config: https://pastebin.com/jqUnkMdg

Flavius

@AMecea

please provide me a describe on that pod. I don't know what possible generate that error.

did you migrate from an older version of the operator?

Giovanbattista Amato

@gioamato

Nothing changed cluster wide. Suddenly started to act like that. All the other pods are healthy. I'm trying to restart the cluster. Will get back with the describe asap.

Giovanbattista Amato

@gioamato

Ok everything is healthy again now. What i've noticed: during cluster restart, the slave was restarted first and became healthy, then was restarted the master and started to fail on the same error. The slave was not elected new master (How should this work? How much downtime should we have?). Then the slave was restarted again, the master became healthy and then the slave became healthy too.

Flavius

@AMecea

usually should be with no downtime but I suppose that the cluster had unacknowledged recoveries in the past this will block a new failover. Those recoveries are reset by the operator if the cluster is healthy for more than 10 min, but in your case, the cluster was not ready because of that pod.

Giovanbattista Amato

@gioamato

Thank you.

magreenberg

@magreenberg

I am trying to run the Presslabs mysql-operator on OpenShift 3.11.

After running the Helm installation and then running:
oc adm policy add-scc-to-user anyuid -z mysql-operator
I tried to create a generic cluster. The status ends up as follows:
Status:
Conditions:
Last Transition Time: 2020-01-27T19:19:39Z
Message: cluster is in read only
Reason: ClusterReadOnlyTrue
Status: True
Type: ReadOnly
Last Transition Time: 2020-01-27T19:19:39Z
Message: StatefulSet is not ready
Reason: StatefulSetNotReady
Status: False
Type: Ready
Last Transition Time: 2020-01-27T19:19:39Z
Message: no pending ack
Reason: NoPendingFailoverAckExists
Status: False
Type: PendingFailoverAck
And the following warning message appears:
Type Reason Age From Message

Warning ConfigMapSyncFailed 3m (x21 over 25m) controller.mysqlcluster *v1.ConfigMap mysql/my-cluster-mysql failed syncing: configmaps "my-cluster-mysql" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: no RBAC policy matched, <nil>

A similar warning message appears related to a Redis Operator that was missing an RBAC policy setting for OpenShift.
Has anyone succeeded in using the mysql-operator on OpenShift?

Shyam Biradar

@shyam-biradar

mysql cluster failing to start with following error on openshift 4.3

Status:
Conditions:
Last Transition Time: 2020-03-13T09:59:23Z
Message: cluster is in read only
Reason: ClusterReadOnlyTrue
Status: True
Type: ReadOnly
Last Transition Time: 2020-03-13T09:59:23Z
Message: StatefulSet is not ready
Reason: StatefulSetNotReady
Status: False
Type: Ready
Last Transition Time: 2020-03-13T09:59:23Z
Message: no pending ack
Reason: NoPendingFailoverAckExists
Status: False
Type: PendingFailoverAck
Events:
Type Reason Age From Message

Warning ConfigMapSyncFailed 73s (x17 over 6m46s) controller.mysqlcluster *v1.ConfigMap openshift-marketplace/sample-mysqlcluster-mysql failed syncing: configmaps "sample-mysqlcluster-mysql" is forbidden: cannot set blockOwnerDeletionif an ownerReference refers to a resource you can't set finalizers on: , <nil>

Any thoughts/

kozachenko1203

@kozachenko1203

Hi team.
How can I create several databases by using operator as the app database?

In the mysqlcluster-secret, I can only specify one app database name.

Is there any way to provision multiple databases when the pod ready?

Thank you.

Where communities thrive

People

Repo info

Activity