Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 27 2019 23:44

    YanChii on master

    workaround new bootparams varia… (compare)

  • Jan 25 2019 16:02

    YanChii on new_release_20190117

    fixed boot.manifest (compare)

  • Jan 25 2019 16:00

    YanChii on new_release_20190117

    fixed manifest (compare)

  • Jan 25 2019 15:42

    YanChii on new_release_20190117

    OS-7436 vminfod can be overwhel… OS-6795 illumos-joyent build to… OS-7488 vm delete fails after O… and 8 more (compare)

  • Jan 25 2019 15:39

    YanChii on master

    removed VM.js (TSC workaround n… (compare)

  • Jan 25 2019 15:34

    YanChii on master

    moved postboot-rc manifests (compare)

  • Jan 25 2019 15:33

    YanChii on merge-sysinfo

    (compare)

  • Jan 25 2019 15:33

    YanChii on master

    merge sysinfo upstream changes (compare)

  • Jan 25 2019 15:33
    YanChii closed #10
  • Jan 25 2019 14:46

    YanChii on new_release_20190103

    updated boot copyright year (compare)

  • Jan 23 2019 13:48

    YanChii on master

    allow consistently reexecute he… (compare)

  • Jan 23 2019 13:48
    YanChii closed #119
  • Jan 23 2019 13:46
    YanChii synchronize #385
  • Jan 22 2019 21:58
    YanChii synchronize #384
  • Jan 22 2019 21:43
    YanChii synchronize #384
  • Jan 22 2019 19:58
    YanChii synchronize #385
  • Jan 21 2019 09:50

    dn0 on master

    Fixed links to issues (compare)

  • Jan 21 2019 09:08
    YanChii synchronize #385
  • Jan 21 2019 09:02
    YanChii review_requested #385
  • Jan 21 2019 09:02
    YanChii milestoned #385
witte1000
@witte1000
http://dtrace.org/blogs/bmc/2019/12/02/the-soul-of-a-new-computer-company/
balwo
@balwo

I've got a question about 4.1 to 4.2.1 upgrade.
The upgrade keeps failing with the error shown further below.
Looks like I've got a problem with a ket or so. Although I can't remember messing around with keys in the Admin DC.
What should I do to get the upgrade working ?

main* Task Log
Time Status Object Message User Detail
21:16:31 2019-12-08
FAILURE 1m1u1-16618c5e-6ece-45c8-89c7 main System Update admin

Running git fetch origin
Running git checkout to commit: 19838ddcfcc7905bca3e6228ccd8c223aac750e3

#

Repository is on commit: 19838ddcfcc7905bca3e6228ccd8c223aac750e3
Starting 2nd run of /opt/erigones/bin/esdc-git-update --esdc-service-restart v4.2.1

#

Upgrading appliances (please wait) ...

PLAY [Running appliance upgrade of hosts mgmt??] *

TASK [Extracting current appliance version] **
Sunday 08 December 2019 20:16:31 +0000 (0:00:00.131) 0:00:00.131 *
fatal: [mgmt01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).", "unreachable": true}

NO MORE HOSTS LEFT *

PLAY RECAP *
mgmt01 : ok=0 changed=0 unreachable=1 failed=0

Sunday 08 December 2019 20:16:31 +0000 (0:00:00.213) 0:00:00.344 *

Extracting current appliance version ------------------------------------ 0.21s
Playbook run took 0 days, 0 hours, 0 minutes, 0 seconds
ERROR: Appliance upgrade failed

#

Running emergency rollback
Running git checkout to commit before upgrade: deb8f7a9b430b75980d90d26034a054fcec84dbb
Previous HEAD position was 19838dd... bump patchlevel to v4.2.1
HEAD is now at deb8f7a... v4.1 version bump

Jan Poctavek
@YanChii
@balwo it looks like for some reason you don't have installed .ssh/authorized_keys on management VM(s)
you can check/fix it by ssh-ing into mgmt01 from the first compute node and make sure the .ssh/id_rsa.pub is present in .ssh/authorized_keys
then check also other management VMs like dns01, mon01 etc… pubkey from mgmt01 must be present in all authorized keys
then you can re-run the upgrade
pls let us know how it went
balwo
@balwo
@YanChii I check authorized_hosts on all mgmt VMs. Some VMs had a different key. If key was different, then I changed it to the value of id_rsa.pub of node01 . I can now access all mgmt VMs from node01, without having to enter a password.
@YanChii However, when I restart the upgrade to 4.2.1, I'm still getting the same error.
...
TASK [Extracting current appliance version] **
Monday 09 December 2019 21:33:13 +0000 (0:00:00.422) 0:00:00.422 *
fatal: [mgmt01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).", "unreachable": true}
...
What am I missing ?
Jan Poctavek
@YanChii
@balwo there are two different keys for two different things
  1. pubkey of node01: allows accessing the 5 man
management VMs from node01
2:
pubkey of mgmt01: allows accessing the 5 management VMs from mgmt01 - THIS IS USED FOR UPGRADES
so to sum up: all 5 management VMs should contain both keys - mgmt01 and node01
Jan Poctavek
@YanChii
these keys are added automatically during install… so if there's some other key or the key is missing, I suspect the initial DC install has either failed (and maybe re-started) or did not for some reason perform all steps
the answer probably lies in /var/log/headnode*.log file
(there's only one headnode logfile)
balwo
@balwo
@YanChii Your instructions worked like a charm ! Nodes are now running 4.2.1. Thanks for your quick responses and outstanding help !
Jan Poctavek
@YanChii
glad to help
enjoy the Danube ;)
infinity202
@infinity202
Hi, i tried several Ubuntu installtions (which are provided by Joyent) and i still see that all KVM's will get KVM-clock settings and that performance is totally poor.
Jan Poctavek
@YanChii
Hi @infinity202, do you observe this behavior also on vanilla SmartOS? We don't modify the SmartOS platform so heavily to have such issues. If we can replicate it on vanilla, we can create an issue for Joyent.
infinity202
@infinity202
it is time for me to startup my old server and reinstall SmartOS on it. I will see what happens there.
Although i must say i have a ubuntu 16.04 20171122 b2da7f6e-7ef5-454c-9d76-d15e2ef8abf1 running at an OVH server with SmartOS 5.11 joyent_20181011T004530Z and that one is running fine and also has KVM-Clock a just saq
saq
saw
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:03:55 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:18:43 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:03:59 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:04:00 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:04:01 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:18:47 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:06:41 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:52 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:53 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:53 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:55 UTC 2019
ubuntu@--on danube Cloud--:/opt$ date
Thu Dec 19 16:21:56 UTC 2019
infinity202
@infinity202

ok, at least i found a solution to keep the clock at hpet after reboots:
sudo apt-get install sysfsutils

Set clocksource to hpet

sudo tee -a /etc/sysfs.d/clocksource.conf <<-EOF
devices/system/clocksource/clocksource0/current_clocksource = tsc
EOF

---------------------------------

sudo systemctl enable sysfsutils.service
sudo systemctl start sysfsutils.service

Jan Poctavek
@YanChii
so the vanilla smartos does the same... that was my suspicion
you can use hwclock -c if available
to see the diff between os clock (vm) and hw clock (hypervisor)
BTW you don't have to reinstall Danube to test vanilla smartos... just reboot with smartos USB stick and then reboot back with Danube stick
all VMs will be kept
Jan Poctavek
@YanChii
BTW2 I've added your howto to our known issues https://github.com/erigones/esdc-ce/wiki/Clock-unstable-in-KVM-VMs
infinity202
@infinity202
Crap i see i posted the wrong copy of the code.
Now it states "tsc" in stead of "hpet"
I found it on a forum where someone needed tsc, and i thought let me try this with hpet. It worked but i copied the lines from the forum to gitter without propper checking
(original reference: https://gist.github.com/flyinprogrammer/6e1cdd7cbdd32d607c07a9ed8059dc1e)
Jan Poctavek
@YanChii
no prob, I've corrected it
tsc might also work… but it also might not (kvm-clock is a normalized tsc) and with tsc you will for sure have time issues during live migration
anyway, I think we should replicate this issue and create a SmartOS bug report… too many people suffer from this bug
klebed
@klebed

@YanChii Hi! Could you give some details on proper way of updating whole DC infrastructure?
I have all hw nodes booted from USB (2 USB Flash drives on each for redundancy) I have HA deployed therefore I have mgmt01-03, and on the first node01 USB flash drive contains first node image (not CN).
Now I see that recommended way is to update mgmt01 (bin/esdc-git-update, and bin/esdc-appliance-update) and then update first node (but how to do it in HA, and more than that, I have some changes in templates of mgmt01, which I guess I'll have to make by hands after upgrade of mgmt)

Then I guess I'll have to execute: /opt/erigones/bin/esdc-platform-upgrade v4.2 on all nodes, and it seems upgrade image on USB, but on which one? They all unmounted, and there are 2 of them. How to update another one (or all of them at once?)

Jan Poctavek
@YanChii
Hi. The normal way of upgrading is to run esdc-git-update which will automatically call also esdc-appliance-update. The latter one applies update on all numbered management instances (e.g. mgmt??, dns??, etc). So it should upgrade also HA instances.
If you encounter problems during HA upgrade, let us know.
Not sure what you mean by templates of mgmt01.
esdc-platform-upgrade rewrites the first USB key. Then after successful node reboot, you can find the second USB key by command rmformat and use dd to copy the first USB over the second (use rdsk device names)