prometheusmonitoring: 2020-04-16 13:29:09 Drone: {'site_name': 'NEMO', 'machine_type': 'tardis_c40m120', 'remote_resource_uuid': 8065030, 'created': datetime.datetime(2020, 4, 16, 12, 2, 52, 850664), 'updated': datetime.datetime(2020, 4, 16, 13, 29, 8, 464074), 'drone_uuid': 'nemo-8065030', 'resource_status': <ResourceStatus.Stopped: 3>} has changed state to CleanupState
prometheusmonitoring: 2020-04-16 13:29:09 TTTT: <enum 'ResourceStatus'>
ah I think I know what may be happening
"crosstalk" from the Elasticsearch plugin
In the
ElasticsearchPlugin, I'm turning state into a str so that ElasticSearch isn't confused, but since resource_attributes is passed by reference it affects everything else
(Sorry for not being available to give hints earlier, by the way. Trying to catch up now.)
But I think I will completely redesign this anyway
so I think it is not necessary that you waste any time on this :)
Sorry, for being late to the party. I was stuck in a zoom meeting. Good that found the problem. Dictionaries are mutable objects in Python. Passing mutable objects to a function behaves similar to passed by reference in other languages. Everything that is not mutable behaves similar to passed by value in other languages.
For reference, a simple/fast way to convert-with-copy some keys is to use a dict literal (
{}) with unpacking (**) of the original dict.resource_attributes = {
**resource_attributes,
"state": str(state),
"meta": self._meta,
"timestamp": int(time() * 1000),
"resource_status": resource_attributes["resource_status"],
}
BTW: I'm using PR #141 in production for a while now and it does not solve the problem that tardis tries to cancel drones which are long gone. I guess this may be related to @giffels comment here: https://github.com/MatterMiners/tardis/pull/141#discussion_r409591396
Okay, thanks for the information. @rfvc could you address the remaining issue in #141, please? I found that
CANCELLED is also not yet handled there.
@giffels @rfvc: I just wanted to let you know that mapping
TimeLimit and Vacated to ResourceStatus.Deleted in the MOAB site adapter seems to solve our problem of zombie drones outlined above. The zombie drones caused a couple of problems which required us to restart the service at least once a day. I thought this was related to a bug in the Slurm batchsystem adapter, but apparently it wasn't. Anyway, with the updated mapping it has been running for more than a week without needing to be restarted.
@giffels, I think so, I’m on it
@giffels: Thanks! unfortunately it looks like it didn't work. At least the diff seems to be missing the last commit :/
So to introduce myself, I am a sys-admin at the University of Exeter.
- We are setting up a 2500 core OpenStack system, which mostly will be used to serve researchers bespoke interactive VMs. I will admit I am a very new sys-admin so a lot of this stuff is confusing to me.
- I would love to use Cobald/Tardis to utilize an idle cpu resources for batch compute. But the docs feel a bit light in setting up a new pool.
Right, we've wanted to brush up the github.io page for a while but there are still some parts missing.
In case you want other ways of contacting us as well:
You can reach the core team at matterminers@lists.kit.edu at any time for practically any questions.
We've got a mattermost at https://chat.eudat.eu/ where you can reach other users of C/T as well – though it's mostly in German, as usual for the community people can switch to English at any time and will gladly do so.
Finally, a lot of us are also on the CERN mattermost.
In case you want other ways of contacting us as well:
You can reach the core team at matterminers@lists.kit.edu at any time for practically any questions.
We've got a mattermost at https://chat.eudat.eu/ where you can reach other users of C/T as well – though it's mostly in German, as usual for the community people can switch to English at any time and will gladly do so.
Finally, a lot of us are also on the CERN mattermost.
Now that we've got that covered... :sweat_smile:
@giffels has more experience using C/T with OpenStack, but I should be able to walk you through the rough steps and we can work our way up from there on. ;)
Before we dive into C/T itself, do you have an existing batch system through which users will access the resources? I see you've been active on the HTCondor mailing, so I assume you're familiar with running an HTCondor Cluster?
3 replies
I am an experienced HTCondor user, not admin.
So would have to spin up a central manager and submit hosts. How my boss wants storage managed, I am not sure.
We just had the machines installed last week, a bunch is still in flux. I just wanted to reach out before hand. This also helps me make a clearer case to supervisor if I might need an extra node for running the submit &/or CM.
- You'll need some HTCondor submit hosts to get jobs into the system. C/T doesn't really care about these, but be prepared that they must be able to communicate with your opportunistic resources. A public IP address is advisable since that means the resources don't need one then.
- You'll need a central manager for the Collector and Negotiator. This one must be reachable from all opportunistic resources. I recommend configuring the Collector as a Condor Connection Broker as well, since that means only one of submit/worker node must be public.
- You'll need a machine that runs COBalD/TARDIS. This one must be able to reach your resource provider (i.e. OpenStack) and the Collector. It's fine to just use the central manager for this as well; C/T is pretty flexible about being redeployed later on, so you don't lock-in yourself.
- You can have static worker nodes (
STARTD) but don't need them. C/T will opportunistically add them to your system. There is no problem having multiple C/T instances add resources, nor in having resources from other sources as well.
If you need recommendations on any of these, feel free to ask. Generally, C/T doesn't care much about how HTCondor is setup so we try to avoid giving people wrong ideas. But we do have the expertise for the entire HTCondor stack so we can give you more information as needed.
As for getting started with C/T: I recommend to go through our tutorial first, which will walk you through setting up COBalD and TARDIS with a dummy resource pool. We'll walk you through using the OpenStack resource pool afterwards.