https://danubecloud.org | https://github.com/erigones/esdc-ce/wiki/Release-Notes#30
Monday 09 December 2019 21:33:13 +0000 (0:00:00.422) 0:00:00.422 *
fatal: [mgmt01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).", "unreachable": true}
- pubkey of node01: allows accessing the 5 man
/var/log/headnode*.log file
Although i must say i have a ubuntu 16.04 20171122 b2da7f6e-7ef5-454c-9d76-d15e2ef8abf1 running at an OVH server with SmartOS 5.11 joyent_20181011T004530Z and that one is running fine and also has KVM-Clock a just saq
Thu Dec 19 16:03:55 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:18:43 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:03:59 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:04:00 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:04:01 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:18:47 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:06:41 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:53 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:53 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:55 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:56 UTC 2019
ok, at least i found a solution to keep the clock at hpet after reboots:
sudo apt-get install sysfsutils
Set clocksource to hpet
sudo tee -a /etc/sysfs.d/clocksource.conf <<-EOF
devices/system/clocksource/clocksource0/current_clocksource = tsc
EOF
---------------------------------
sudo systemctl enable sysfsutils.service
sudo systemctl start sysfsutils.service
hwclock -c if available
@YanChii Hi! Could you give some details on proper way of updating whole DC infrastructure?
I have all hw nodes booted from USB (2 USB Flash drives on each for redundancy) I have HA deployed therefore I have mgmt01-03, and on the first node01 USB flash drive contains first node image (not CN).
Now I see that recommended way is to update mgmt01 (bin/esdc-git-update, and bin/esdc-appliance-update) and then update first node (but how to do it in HA, and more than that, I have some changes in templates of mgmt01, which I guess I'll have to make by hands after upgrade of mgmt)
Then I guess I'll have to execute: /opt/erigones/bin/esdc-platform-upgrade v4.2 on all nodes, and it seems upgrade image on USB, but on which one? They all unmounted, and there are 2 of them. How to update another one (or all of them at once?)
esdc-platform-upgrade rewrites the first USB key. Then after successful node reboot, you can find the second USB key by command rmformat and use dd to copy the first USB over the second (use rdsk device names)
Greetings, everyone!
some feedback.
1) If you have more than one backup to delete - action fails. Reproduce: make retention rate 10, wait for all backups to fill-up, and then just change retention to 5. You will see backup deletion always fails after that. Action should create queue and delete backups one-by-one though.
2) When making appliance-update, I had to run it on every mgmt0X by hands. And I could definitely say that 3 mgmt machines are not enough for fault tolerance. I suggest making session-share between mgmt machines, transaction-share and using haproxy, rather than VIP, corosync and other questionable stuff.
3) mon0 has a lot of issues, for example no partitioning and then a lot of housekeeper busy more than 75% every day, and also some false-positives with agent unavailable more than 5 min on nodes (without any real issues). And when mon0 is loaded with housekeepers you may hit timeouts.
Also when looking large list like Servers cpu consumption, it's possible but usually not clear which color belongs to which machine.
Maybe it worth looking at monit/grafana, rather than zabbix?
thank you for your reports and feedback.
- I have created issue for this: erigones/esdc-ce#478
- The HA can be improved indeed. The main problem is the HA of postgresql master. It would require massive schema changes to support multi-master DB setup (using pub/sub replication). Therefore I'm not sure whether haproxy would help because you still need to wait for a DB failover. But no doubt that session sharing can be useful anyway during failover/switchover. Regarding running
appliance-updateon each node - what happended? It should be enough to run it just once. - Database partitioning is not supported by Zabbix LLC (even thou everybody is using it). You might have a look at this doc: https://github.com/erigones/esdc-ce/wiki/Postgresql-partitioning-on-zabbix-monitoring-server
We have a preparation for partitioning so you can just enable it. Same thing for housekeeper tuning.
On top of that, DC supports horizontal Zabbix scaling - deploying a separatemonserver per virtual datacenter. This way you will avoid a massive monolithic Zabbix VM that is hard to manage and has its own issues. See here: https://docs.danubecloud.org/user-guide/monitoring/dc-monitoring-server.html