Running sos run test combined with test.sos having the following workflow,
[single]
input: for_each=dict(i=range(2))
output: f'single_{i}.bam'
_output.touch()
[double]
input: for_each=dict(i=range(2))
output: f'single_{i}.bam'
_output.touch()
[combined]
input: output_from('single'), output_from('double')
print(_input)You will see that the two groups from single and double are combined to form two groups with one output from single and one output from double.
single]
input: for_each=dict(i=range(2))
output: f'single_{i}.bam'
_output.touch()
[double]
input: for_each=dict(i=range(3))
output: f'single_{i}.bam'
_output.touch()
[combined]
input: output_from('single', group_by='all'), output_from('double', group_by='all'), group_by=1
print(_input)output_from into a single group before separating them into groups with one file (group_by=).
Hello, another quick question. The login nodes for my cluster get pretty congested, and during peak hours I start to see a lot of
ERROR: ERROR workflow_executor.py:1206 - Failed to connect to yale_hpc_slurm: ssh connection to pgc29@xxx.xxx.xxx.xxx time out with prompt: b'' - None errors. Is there a way to adjust the timeout to make it longer?
My dataset has gotten to be so large (several thousand genomes) that for every step, there will likely be one or two failures due to subtle race conditions, or an outlier requiring much more memory or runtime so the cluster kills it, or the initial file was a contaminant etc. So my workflow has started to bog down into a cycle of 1) submit job 2) check in a few hours later and see which substeps failed 3) adjust parameters or just resubmit (it often just works the second time) 4) check in a few hours later to see how step 2 failed 5) resubmit 6) repeat. With a pipeline of >10 steps, this is tedious. I'd prefer if I wouldn't have to babysit runs as much. Is there a way to have SoS continue to the next step, even if some substeps fail? That way 99% of my samples will make it from fastq file to a final VCF and then I can tweak things for a second run to finish up the failed 1%. Any other suggestions on how to improve robustness would be welcome.
@pgcudahy
I'm having some issues with running steps with
remote inputs and outputs and SoS not noticing changed files, so ignoring steps with saved signatures. Where exactly are signatures stored for jobs run remotely and how can I clear them? I've tried !sos remove -s from within my notebook, but I still get steps skipped.
@pgcudahy The content of "remote" files are not checked at this point and this is why signatures related to remote files are inaccurate. The current mechanism for remote files work for some cases, but broken for others (I have a ticket for returning remote files with
workdir is set) and certainly needs improvement.
This is now vatlab/sos#1411
0.22.3 seems to have broken my scripts, so I'm trying to figure out what's wrong
@BoPeng ?
Hello Bo, when submitting workflows and tasks to a cluster (eg
%run check_validation -q yale_hpc_task_spooler -r yale_hpc_task_spooler) is there a way to synchronize the output back to my local computer? Using a named path like #scratch fails with WARNING: Error from step check_validation is ignored: [check_validation]: Failed to process step output (f'#scratch/helen_mixed_infection/data/fqtools/good_files.txt'): 'NoneType' object has no attribute 'expanduser'
The second issue on anchored path looks more like a bug but I need more background on how it happened.