erigones/DanubeCloud

https://www.danube.cloud | https://github.com/erigones/esdc-ce/wiki/Release-Notes

otis

@otis:slackhive.net

[m]

on that node

then click "refresh" in mgmt

matthewddunlap

@matthewddunlap

Thanks, but that's not fixing it seeing.

Seeing a lot of repeating:

[2021-03-02 00:07:04,058: WARNING/Worker-23] Ping [1] of all queue "mgmt" workers: ('mgmt@mgmt01.local',) failed ([])

[2021-03-02 00:07:04,058: WARNING/Worker-23] Sending task "api.vm.status.tasks.vm_status_all_cb" by 7e7d1-034862bf-ec78-40da-8352 failed. Error: Task queue "mgmt" worker is not responding!
[2021-03-02 00:07:04,058: WARNING/Worker-23] Task "api.vm.status.tasks.vm_status_all_cb" sent by 7e7d1-034862bf-ec78-40da-8352 must run! Retrying (63) in 3 seconds...

Running erigonesd mgmt inspect ping from this compute node and mgmt01 both look good.

Jan Poctavek

@YanChii

also putting the node to maintenance mode and back forces the update of keys and img sources

if it wont work, then the erigonesd doesn't get commands from mgmt… is is a newly installed node? Does gui say it's online?

none of 4 erigonesd should be in maintenance mode… if it is, use svcadm clear <name>

check also whether node install finished successfully… /var/log/computenode*.log

matthewddunlap

@matthewddunlap

This is happening with existing nodes that had been working. I also did try rebuilding a problematic one, and it's still happening.

I think it's a bigger issue, becuase also even for compute nodes that are correct in regards to imgadm/authorized_keys, their VM status are failing to update.

Looking at mgmt01, the process beam.smp is running with extremely high CPU non-stop. haproxy is also quite high non-stop.

The mgmt01 VM had been sized to 4xCPU/8GB and I doubled it last night. The probelms persists.

This was all fresh deployed 4.4 in the past month or so.

9 replies

otis

@otis:slackhive.net

[m]

@YanChii: i'm seeing very similar behaviour as @matthewdunlap (intermittent network connectivity outages causes weird behaviour)

Jan Poctavek

@YanChii

https://www.datacenterknowledge.com/uptime/ceo-says-fire-has-destroyed-ovh-s-strasbourg-data-center-sbg2

FilipFrancis

@FilipFrancis

yeah saw that also even the other datacenter in the same area are also down for the moment

infinity202

@infinity202

Good morning! Has anyone ever tried to upgrade OPNSense to the new 21.1 release ? And how did you do it ?

Paolo Marcheschi

@marcheschi

I'd like to make some cleanup on images , How is possible to remove unused images from DC?

Jan Poctavek

@YanChii

@infinity202 does normal procedure using opnsense gui work?

opnsense upgrade is not (yet?) integrated into DC

@marcheschi Nodes->nodename->Images tab and delete what you need

they must not be used

also removing them from main image list should work

Paolo Marcheschi

@marcheschi

Thank you That's What I intended to know if they are used ?

Jan Poctavek

@YanChii

@matthewddunlap did you manage to fix it?

infinity202

@infinity202

Hi @YanChii : The GUI says "This software release has reached its designated end of life. The next major release is: 21.1".
It seems one has to use a ISO boot to upgrade the OPNsense VM to this new release. https://docs.opnsense.org/manual/updates.html#major-upgrades

but..... during this update you won't have a functioning OPNSense server and (in my case) no connection to the server

Jan Poctavek

@YanChii

you still have VGA console

@marcheschi not sure I understand the question… if you can detach the image from all datacenters (or directly delete it), then its surely not used.

Paolo Marcheschi

@marcheschi

Ok I forgot it, thank you

There is also a button for it

Jan Poctavek

@YanChii

thumb up forgot about that one, too :)

Paolo Marcheschi

@marcheschi

DC is great !!!

infinity202

@infinity202

I just booted a new linux VM with the OPNsense 21.1 ISO and will configure it to use my spare IP-adress on the WAN. Everything seems to be working for now.

Jan Poctavek

@YanChii

maybe try to import the config.xml from previous appliance to test whether it works

infinity202

@infinity202

yes, I wil ltry that. But I will examine the file first to avoid IP conflicts

Jan Poctavek

@YanChii

yup… it has DHCP on interfaces thanks to KVM so it shouldn't be much problem. Bhyve doesn't have such functions. If you have an idea how to autoconfigure opnsense interfaces in bhyve, let me know. I don't want to do too much opnsense image customizations because it can break after upgrade.

infinity202

@infinity202

I've never played with bhyve. I will try to find some time to get this new KVM machine up and running in decent manner first. Importing the XML won't work in my case. But that's not a problem.

Paolo Marcheschi

@marcheschi

Hi Is there a way to add documentation about the virtual Data-center? (network maps , compute nodes schema, logical connections, ) I think this could be useful, in order to see all at glance. I think could be enough a place to show a web page inside Datacenter administration.

Jan Poctavek

@YanChii

Paolo: erigones/esdc-ce#536

scheker13

@scheker13

Hello everyone, just by chance does anyone know why the website is not accessible?

Jan Poctavek

@YanChii

Hetzner is having an outage :-/

I hope it will be fixed soon

Jan Poctavek

@YanChii

@scheker13 fixed

scheker13

@scheker13

@YanChii Thank you :)

infinity202

@infinity202

crap... I am using CE 4.3 and have a centos CentOS Linux release 8.3.2011 VM which I updates yesterday.

after the update I am experiencing the bloody clock problems again. The system time goes up and down, causing the services to crash

at this moment I am updating the danube cloud machine to version 4.4. Will see what that brings me.

Jan Poctavek

@YanChii

pls update the platform and let me know… I've not seen clock problems on new platforms so far

infinity202

@infinity202

hmz, I managed to update the compute node to ce 4.4 and rebooted. The clock problem seems to be solved. But now my mgmt node won't update to ce 4.4

infinity202

@infinity202

ok, manualy running git update to v4.4

ah, sorry. user error. I disabled the zabbix zone mon01 some time ago due to excesive power usage. And the update script fails due to not getting in touch with mon01

Jan Poctavek

@YanChii

cool

Where communities thrive

People

Repo info

Activity