prometheusmonitoring: 2020-04-16 13:29:09 Drone: {'site_name': 'NEMO', 'machine_type': 'tardis_c40m120', 'remote_resource_uuid': 8065030, 'created': datetime.datetime(2020, 4, 16, 12, 2, 52, 850664), 'updated': datetime.datetime(2020, 4, 16, 13, 29, 8, 464074), 'drone_uuid': 'nemo-8065030', 'resource_status': <ResourceStatus.Stopped: 3>} has changed state to CleanupState
prometheusmonitoring: 2020-04-16 13:29:09 TTTT: <enum 'ResourceStatus'>
ah I think I know what may be happening
"crosstalk" from the Elasticsearch plugin
In the
ElasticsearchPlugin, I'm turning state into a str so that ElasticSearch isn't confused, but since resource_attributes is passed by reference it affects everything else
(Sorry for not being available to give hints earlier, by the way. Trying to catch up now.)
But I think I will completely redesign this anyway
so I think it is not necessary that you waste any time on this :)
Sorry, for being late to the party. I was stuck in a zoom meeting. Good that found the problem. Dictionaries are mutable objects in Python. Passing mutable objects to a function behaves similar to passed by reference in other languages. Everything that is not mutable behaves similar to passed by value in other languages.
For reference, a simple/fast way to convert-with-copy some keys is to use a dict literal (
{}) with unpacking (**) of the original dict.resource_attributes = {
**resource_attributes,
"state": str(state),
"meta": self._meta,
"timestamp": int(time() * 1000),
"resource_status": resource_attributes["resource_status"],
}
BTW: I'm using PR #141 in production for a while now and it does not solve the problem that tardis tries to cancel drones which are long gone. I guess this may be related to @giffels comment here: https://github.com/MatterMiners/tardis/pull/141#discussion_r409591396
Okay, thanks for the information. @rfvc could you address the remaining issue in #141, please? I found that
CANCELLED is also not yet handled there.
@giffels @rfvc: I just wanted to let you know that mapping
TimeLimit and Vacated to ResourceStatus.Deleted in the MOAB site adapter seems to solve our problem of zombie drones outlined above. The zombie drones caused a couple of problems which required us to restart the service at least once a day. I thought this was related to a bug in the Slurm batchsystem adapter, but apparently it wasn't. Anyway, with the updated mapping it has been running for more than a week without needing to be restarted.
@giffels, I think so, I’m on it
@giffels: Thanks! unfortunately it looks like it didn't work. At least the diff seems to be missing the last commit :/