https://danubecloud.org | https://github.com/erigones/esdc-ce/wiki/Release-Notes#30
Although i must say i have a ubuntu 16.04 20171122 b2da7f6e-7ef5-454c-9d76-d15e2ef8abf1 running at an OVH server with SmartOS 5.11 joyent_20181011T004530Z and that one is running fine and also has KVM-Clock a just saq
Thu Dec 19 16:03:55 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:18:43 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:03:59 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:04:00 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:04:01 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:18:47 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:06:41 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:53 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:53 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:55 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:56 UTC 2019
ok, at least i found a solution to keep the clock at hpet after reboots:
sudo apt-get install sysfsutils
Set clocksource to hpet
sudo tee -a /etc/sysfs.d/clocksource.conf <<-EOF
devices/system/clocksource/clocksource0/current_clocksource = tsc
EOF
---------------------------------
sudo systemctl enable sysfsutils.service
sudo systemctl start sysfsutils.service
hwclock -c if available
@YanChii Hi! Could you give some details on proper way of updating whole DC infrastructure?
I have all hw nodes booted from USB (2 USB Flash drives on each for redundancy) I have HA deployed therefore I have mgmt01-03, and on the first node01 USB flash drive contains first node image (not CN).
Now I see that recommended way is to update mgmt01 (bin/esdc-git-update, and bin/esdc-appliance-update) and then update first node (but how to do it in HA, and more than that, I have some changes in templates of mgmt01, which I guess I'll have to make by hands after upgrade of mgmt)
Then I guess I'll have to execute: /opt/erigones/bin/esdc-platform-upgrade v4.2 on all nodes, and it seems upgrade image on USB, but on which one? They all unmounted, and there are 2 of them. How to update another one (or all of them at once?)
esdc-platform-upgrade rewrites the first USB key. Then after successful node reboot, you can find the second USB key by command rmformat and use dd to copy the first USB over the second (use rdsk device names)
Greetings, everyone!
some feedback.
1) If you have more than one backup to delete - action fails. Reproduce: make retention rate 10, wait for all backups to fill-up, and then just change retention to 5. You will see backup deletion always fails after that. Action should create queue and delete backups one-by-one though.
2) When making appliance-update, I had to run it on every mgmt0X by hands. And I could definitely say that 3 mgmt machines are not enough for fault tolerance. I suggest making session-share between mgmt machines, transaction-share and using haproxy, rather than VIP, corosync and other questionable stuff.
3) mon0 has a lot of issues, for example no partitioning and then a lot of housekeeper busy more than 75% every day, and also some false-positives with agent unavailable more than 5 min on nodes (without any real issues). And when mon0 is loaded with housekeepers you may hit timeouts.
Also when looking large list like Servers cpu consumption, it's possible but usually not clear which color belongs to which machine.
Maybe it worth looking at monit/grafana, rather than zabbix?
thank you for your reports and feedback.
- I have created issue for this: erigones/esdc-ce#478
- The HA can be improved indeed. The main problem is the HA of postgresql master. It would require massive schema changes to support multi-master DB setup (using pub/sub replication). Therefore I'm not sure whether haproxy would help because you still need to wait for a DB failover. But no doubt that session sharing can be useful anyway during failover/switchover. Regarding running
appliance-updateon each node - what happended? It should be enough to run it just once. - Database partitioning is not supported by Zabbix LLC (even thou everybody is using it). You might have a look at this doc: https://github.com/erigones/esdc-ce/wiki/Postgresql-partitioning-on-zabbix-monitoring-server
We have a preparation for partitioning so you can just enable it. Same thing for housekeeper tuning.
On top of that, DC supports horizontal Zabbix scaling - deploying a separatemonserver per virtual datacenter. This way you will avoid a massive monolithic Zabbix VM that is hard to manage and has its own issues. See here: https://docs.danubecloud.org/user-guide/monitoring/dc-monitoring-server.html
Hi everyone! How's your pandemic lockdown goes? Hope everyone is ok!
Couple of questions:
1) zpool status -v gives following:
pool: zones
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: none requested
config:
so is it worth upgrading zpool features?
2) I had one of the backup pools down due to sudden errors... it was happened unnoticed, so I thinking: It's cruitial to monitor health of storage on the nodes... I believe admin panel could also be enriched with some data about disks and pools health, and it probably worth including smart monitoring into node images by default.
thank you for your feedback. We are quite fine. Some folks here are a bit overwhelmed by children at home but otherwise we're doing good :).
1) If you don't plan downgrading your platform to older version (probably not), you can run
zpool upgrade <pool> to enable new zfs features.2) Good point. Would you mind creating an issue in
esdc-ce? I'll reference it in the fix.Apart from that, the upcoming version will also contain the
kvm-clock fix. It is already done and tested.
2) Good point. Would you mind creating an issue in
esdc-ce? I'll reference it in the fix.
Apart from that, the upcoming version will also contain thekvm-clockfix. It is already done and tested.
That's very cool! I was struggleing with that a lot for last months... every machine has to be manually fixed to hpet, which wastes time...
Would you mind creating an issue in esdc-ce?
I'll create an issue, sure...
@klebed thank you. If you have suggestions what to change to make the zabbix notifications more usable after install, pls share. We can adjust the defaults.
Honestly I don't think it worth polishing zabbix, since it has so many backlashes and architecture issues. It is probably good for almost zero effort monitoring for standard old-fashioned infrastructures, but it's geting worse with all other stuff. And problem of devaluating notifications is one of the very important and recognizeable story about zabbix.
If you really looking into docker, k8s, container zones, etc, then mind thinking of Prometheus since it natively support all the modern stuff. Docker exports prometheus metrics, for instance. And notification and alert system of it is way more efficient, which allows you to control the severiness and make proper differenciation of alerts. And despite the zabbix it's drastically more efficient on dealing with time-series data and you could benefit from flexible and feature rich grafana either to make monitoring more informative.
I understand that it might look like too much efforts at that moment, but in the end it would be worth it. You have certain control over the infrastructure, and your management service is more or less smartly built, so automation over the monitoring could do more than just a zabbix with some defaults.