These are chat archives for nextflow-io/nextflow

1st
Mar 2017
Félix C. Morency
@fmorency
Mar 01 2017 15:06
@pditommaso i just pulled a repo and when I pull again, I get a ... contains uncommitted changes -- cannot pull from repository
Paolo Di Tommaso
@pditommaso
Mar 01 2017 15:06
have you modified the code in the repo ?
Félix C. Morency
@fmorency
Mar 01 2017 15:06
no
Paolo Di Tommaso
@pditommaso
Mar 01 2017 15:07
weird
Félix C. Morency
@fmorency
Mar 01 2017 15:07
i tried pulling a few seconds after the first pull
Maxime Garcia
@MaxUlysse
Mar 01 2017 15:07
@fmorency That happened to me before too, I remove the repo from .nextflow/assets/repo
Paolo Di Tommaso
@pditommaso
Mar 01 2017 15:07
try to change in to repo directory and run git status what it reports ?
Félix C. Morency
@fmorency
Mar 01 2017 15:08
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean
@MaxUlysse yeah that's a workaround but I would prefer not having to do that all the time :)
@pditommaso Mmmm I did nothing and now it tells me it's Already-up-to-date
oh well... nvm
Paolo Di Tommaso
@pditommaso
Mar 01 2017 15:10
nvm ?
Félix C. Morency
@fmorency
Mar 01 2017 15:10
nvm == nevermind :)
Paolo Di Tommaso
@pditommaso
Mar 01 2017 15:10
:)
Maxime Garcia
@MaxUlysse
Mar 01 2017 15:11
@fmorency It had only happened to me a couple of times
But yes, I can see why it's problematic
And I never seem to be able to reproduce the problem
Matthieu Foll
@mfoll
Mar 01 2017 15:33
Same here, we are having this problem from time to time and deleting in .nextflow/assets/repo
Paolo Di Tommaso
@pditommaso
Mar 01 2017 16:12
If any of you can manage to replicate this problem in a deterministic manner, it would be great
Félix C. Morency
@fmorency
Mar 01 2017 19:52
@pditommaso how can those hash be different:
  732e13c4638783bd04bde23b5d9cf26c [nextflow.util.ArrayBag] [FileHolder(sourceObj:/mnt/developers/fmorency/Imeka/20170228-731-t1crop/work/9f/2a804388a48d0267ac0e1495bbbf25/DCC05__dwi_nlm.nii.gz, storePath:/mnt/developers/fmorency/Imeka/20170228-731-t1crop/work/9f/2a804388a48d0267ac0e1495bbbf25/DCC05__dwi_nlm.nii.gz, stageName:DCC05__dwi_nlm.nii.gz)] 

  3b5ad2aad6e89b77fceebed339aba720 [nextflow.util.ArrayBag] [FileHolder(sourceObj:/mnt/developers/fmorency/Imeka/20170228-731-t1crop/work/9f/2a804388a48d0267ac0e1495bbbf25/DCC05__dwi_nlm.nii.gz, storePath:/mnt/developers/fmorency/Imeka/20170228-731-t1crop/work/9f/2a804388a48d0267ac0e1495bbbf25/DCC05__dwi_nlm.nii.gz, stageName:DCC05__dwi_nlm.nii.gz)]
Paolo Di Tommaso
@pditommaso
Mar 01 2017 20:42
has the file size or last update time changed?
Mike Smoot
@mes5k
Mar 01 2017 20:44
Hi @pditommaso I'm getting a SocketTimeoutException right here with a file in S3 (I'm running in AWS) and this crashes my pipeline. Any idea how I can make my code resilient to this? I'd be fine with a process crashing and then let the normal error/retry handling deal with things, but I'm guessing the normalization happens outside of the process execution.
Paolo Di Tommaso
@pditommaso
Mar 01 2017 20:47
good point, yes it should re-execute the task
Please, report on GH. I need to improve that part.
Mike Smoot
@mes5k
Mar 01 2017 20:48
Will do, thanks! In the meantime I'll just curse the network gremlins.
Paolo Di Tommaso
@pditommaso
Mar 01 2017 20:49
It is related to the S3 failure I guess
Mike Smoot
@mes5k
Mar 01 2017 20:49
Yeah, presumably S3 is being flaky
I'll include the full stacktrace
Paolo Di Tommaso
@pditommaso
Mar 01 2017 20:50
Has this happened yesterday or today ?
Mike Smoot
@mes5k
Mar 01 2017 20:51
It happened just a few minutes ago
Paolo Di Tommaso
@pditommaso
Mar 01 2017 20:52
weird
you can try to tune the S3 client settings
Mike Smoot
@mes5k
Mar 01 2017 20:55
Cool, I'll give that a try. Earlier today the pipeline ran fine on the same data, so I'm not sure what's going on.
Paolo Di Tommaso
@pditommaso
Mar 01 2017 20:55
ah no, it report a problem yesterday
today looks fine
maybe S3 is crashing again, right now
:)
Mike Smoot
@mes5k
Mar 01 2017 20:59
Could be.
Paolo Di Tommaso
@pditommaso
Mar 01 2017 21:01
actually, a SocketTimeoutException in their network hardly could be something else . .
Félix C. Morency
@fmorency
Mar 01 2017 21:13
@pditommaso same file. one of the DCC05__dwi_nlm.nii.gz is from an actual run and the other is the cached version
Félix C. Morency
@fmorency
Mar 01 2017 21:24
in other words, the state of the task producing the file DCC05__dwi_nlm.nii.gz in run 1) is [9f/2a8043] Submitted process > MyTask (XXX) while run 2) prints [9f/2a8043] Cached process > MyTask (XXX)
fyi im investigating the issue of tasks being rescheduled for no apparent reason