@channel I'm thinking about implementing Deamon process for better asynchronous process execution.
NOW:
Async process is executed, and looks into the queue, if there is not wating anything to be executed again.
Cons: sharing memory, if something really fails, next processes is not taken out from the queue, process instance train
Planned:
Async processes are just stored in the dblog
Daemon checks every e.g. 30 sec, if there is something to be executed, creates new processes in seperate thread (multiprocessing) and waits another 30sec
Pros: new process instance does have nothing in common with any previews one. the acts of storing requests into dblog and processing them are independant
Questions:
- Should user start the deamon manually, similar to Service instance?
- should the deamon be lounched as part of Service?
@cehbrecht if I understand correctly, this code could do the job in the daemon process:
stored_request = dblog.pop_first_stored()
value = {
'process': json.dumps(store_request.process),
'wps_request': json.dumps(stored_request.wps_request)
}
job = Job.from_json(value)
processing_process = pywps.processing.Process(
process=job.process,
wps_request=job.wps_request
wps_response=job.wps_response)
process.start()this should
- take stored request (and stored jsonised description of a process) from the database
- create request, response and Process objects (using
Job.from_json) - run the execute request on selected Job using configured Sheduler
looks usable?
pywps.processing module could be a common interface for the different job execution implementations. I think you already figured out how the scheduler part works: dump the job status and run the joblauncher with this status document (json) on a remote batch node. A shared file-system and the postgres DB are used to get the outputs and update the job status. The drmaa library is only the interface to schedulers like slurm. We might even skip it because it doesn’t look well maintained. We would then call slurm directly (skipping support for other scheduler systems like GridEngine).
https://pypi.org/project/pywps/4.2.3/
@jachym thanks for the 4.2.4 release :)
https://github.com/geopython/pywps/releases/tag/4.2.4
Could you please upload it to pypi?
The conda-forge package is build from github and a build for 4.2.4 is triggered:
https://github.com/conda-forge/pywps-feedstock
