These are chat archives for nextflow-io/nextflow

26th
Mar 2018
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:19
Hi I work on a cluster and I want run a Nextflow pipeline with a singularity but I have an error: /bin/bash: .command.sh: No such file or directory. I tried to set SINGULARITY_BINDPATH with the path of my scratch but change anything. Moreover I tried to set the work dir to my $HOME and it work. Do you have any ideas ?
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:21
are bind mounts managed by your sysadmins or you are supposed to mount them manually ?
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:23
I think it's managed
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:24
if your run singularity shell <image>
are you able to list the content of host paths ?
for example your home ?
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:25
yes. By default I have an access to my home but I have not access to the scratch. but if I set SINGULARITY_BINDPATH to the scratch, I have access to the scratch in my singularity.
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:26
if so should world also NF
how is the complete error message ?
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:27

``N E X T F L O W ~ version 0.27.6 Launching/home/theg3301/src/theaud-nextflow/preprocess_stride_flip/main.nf` [gloomy_galileo] - revision: 1e076b264f

Preprocess pipeline

Start time: Mon Mar 26 12:15:55 PDT 2018

Input: raw/
[warm up] executor > local
[aa/f8a1db] Submitted process > correctstride (raw-_dwi-3)
[66/0696b4] Submitted process > correctstride (raw-_dwi-1)
[11/ef6864] Submitted process > correctstride (raw-_dwi-2)
[b6/ca2c28] Submitted process > flipgradients (raw-_dwi-3)
[9b/be24eb] Submitted process > flipgradients (raw-_dwi-2)
[e8/b9248f] Submitted process > flipgradients (raw-_dwi-1)
ERROR ~ Error executing process > 'flipgradients (raw-_dwi-3)'

Caused by:
Process flip_gradients (raw_-_dwi-3) terminated with an error exit status (1)

Command executed:

scil_flip_grad.py bvec bvec x --fsl -f

Command exit status:
1

Command output:
(empty)

Command error:
/bin/bash: .command.sh: No such file or directory

Work dir:
/scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '.nextflow.log' file for details
Pipeline completed at: Mon Mar 26 12:16:02 PDT 2018
Execution status: failed
Execution duration: 7.3s```

Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:28
to format code/text use: triple `
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:28
Sorry for the display
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:28
then new-line
triple ` new-line
your stuff
new-line
triple `
that's '
not ```
anyhow change into the directory /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839
cat the content of .command.run
how is the singularity command at the bottom ?
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:35
'''
#!/bin/bash
# NEXTFLOW TASK: flip_gradients (raw_-_dwi-3)
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x

nxf_env() {
    echo '============= task environment ============='
    env | sort | sed "s/\(.*\)AWS\(.*\)=\(.\{6\}\).*/\1AWS\2=\3xxxxxxxxxxxxx/"
    echo '============= task output =================='
}

nxf_kill() {
    declare -a ALL_CHILD
    while read P PP;do
        ALL_CHILD[$PP]+=" $P"
    done < <(ps -e -o pid= -o ppid=)

    walk() {
        [[ $1 != $$ ]] && kill $1 2>/dev/null || true
        for i in ${ALL_CHILD[$1]:=}; do walk $i; done
    }

    walk $1
}

nxf_mktemp() {
    local base=${1:-/tmp}
    if [[ $(uname) = Darwin ]]; then mktemp -d $base/nxf.XXXXXXXXXX
    else TMPDIR="$base" mktemp -d -t nxf.XXXXXXXXXX
    fi
}

on_exit() {
  exit_status=${ret:=$?}
  printf $exit_status > /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839/.exitcode
  set +u
  [[ "$tee1" ]] && kill $tee1 2>/dev/null
  [[ "$tee2" ]] && kill $tee2 2>/dev/null
  [[ "$COUT" ]] && rm -f "$COUT" || true
  [[ "$CERR" ]] && rm -f "$CERR" || true
  (sudo -n true && sudo rm -rf "$NXF_SCRATCH" || rm -rf "$NXF_SCRATCH")&>/dev/null || true
  exit $exit_status
}

on_term() {
    set +e
    [[ "$pid" ]] && nxf_kill $pid
}

trap on_exit EXIT
trap on_term TERM INT USR1 USR2

export NXF_BOXID="nxf-$(dd bs=18 count=1 if=/dev/urandom 2>/dev/null | base64 | tr +/ 0A)"
NXF_SCRATCH="$(set +u; nxf_mktemp $TMPDIR)"
[[ $NXF_DEBUG > 0 ]] && nxf_env
touch /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839/.command.begin
[[ $NXF_SCRATCH ]] && echo "nxf-scratch-dir $HOSTNAME:$NXF_SCRATCH" && cd $NXF_SCRATCH
# stage input files
rm -f bval
rm -f bvec
cp -fRL /scratch/theg3301/test_nextflow/raw/dwi-3/bval bval
cp -fRL /scratch/theg3301/test_nextflow/raw/dwi-3/bvec bvec

set +e
COUT=$PWD/.command.po; mkfifo "$COUT"
CERR=$PWD/.command.pe; mkfifo "$CERR"
tee .command.out < "$COUT" &
tee1=$!
tee .command.err < "$CERR" >&2 &
tee2=$!
(
set +u; env - PATH="$PATH" SINGULARITYENV_TMP="$TMP" SINGULARITYENV_TMPDIR="$TMPDIR" singularity exec /scratch/theg3301/test_nextflow/./singularity.img /bin/bash -c "cd $PWD; /bin/bash -ue /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839/.command.sh"
) >"$COUT" 2>"$CERR" &
pid=$!
wait $pid || ret=$?
wait $tee1 $tee2
cp .command.out /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839/.command.out || true
cp .command.err /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839/.command.err || true
# copies output files to target
if [[ ${ret:=0} == 0 ]]; then
  mkdir -p /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839
  rsync -rRl bval /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839 || true
  rsync -rRl bvec /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839 || true
fi
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:35
good
the singularity command is this
set +u; env - PATH="$PATH" SINGULARITYENV_TMP="$TMP" SINGULARITYENV_TMPDIR="$TMPDIR" singularity exec /scratch/theg3301/test_nextflow/./singularity.img /bin/bash -c "cd $PWD; /bin/bash -ue /scratch/theg3301/test_nextflow/work/b6/ca2c289203e1cf2313e1f473be3839/.command.sh"
what's the output is you run this:
env - PATH="$PATH" SINGULARITYENV_TMP="$TMP" SINGULARITYENV_TMPDIR="$TMPDIR" singularity exec /scratch/theg3301/test_nextflow/./singularity.img /bin/bash -c "cd $PWD; ls -la "
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:39
total 116
drwx------ 9 theg3301 theg3301  4096 Mar 26 19:01 .
drwxr-xr-x 1 root     root        60 Mar 26 19:38 ..
-rw------- 1 theg3301 theg3301 26022 Mar 26 19:37 .bash_history
-rw-r--r-- 1 theg3301 theg3301    18 Aug 16  2017 .bash_logout
-rw-r--r-- 1 theg3301 theg3301   193 Aug 16  2017 .bash_profile
-rw-r--r-- 1 theg3301 theg3301   293 Mar 23 16:41 .bashrc
drwx------ 3 theg3301 theg3301  4096 Mar 22 19:51 .config
drwxrwxr-x 2 theg3301 theg3301  4096 Mar 22 18:08 .licenses
drwxrwxr-x 7 theg3301 theg3301  4096 Mar 26 17:58 .nextflow
-rw-rw-r-- 1 theg3301 theg3301  6268 Mar 26 17:58 .nextflow.log
-rw-rw-r-- 1 theg3301 theg3301  9204 Mar 26 17:29 .nextflow.log.1
-rw-rw-r-- 1 theg3301 theg3301  6350 Mar 26 14:27 .nextflow.log.2
-rw-rw-r-- 1 theg3301 theg3301    95 Mar 26 14:26 .nextflow.log.3
-rw-rw-r-- 1 theg3301 theg3301    95 Mar 23 19:05 .nextflow.log.4
-rw-rw-r-- 1 theg3301 theg3301  1239 Mar 22 20:29 .nwchemrc
drwxrwxr-x 2 theg3301 theg3301  4096 Mar 22 20:30 .oracle_jre_usage
drwx------ 2 theg3301 theg3301  4096 Mar 23 18:39 .ssh
-rw-rw-r-- 1 theg3301 theg3301   165 Mar 23 15:25 .wget-hsts
lrwxrwxrwx 1 theg3301 theg3301    16 Aug 16  2017 project -> /project/6001802
drwx------ 2 theg3301 theg3301  4096 Aug 16  2017 projects
lrwxrwxrwx 1 theg3301 theg3301    17 Aug 16  2017 scratch -> /scratch/theg3301
drwxrwxr-x 3 theg3301 theg3301  4096 Mar 26 17:18 src
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:41
does this match with the content of directory where you run that command ?
ie. does that match with the result of a ls -la ?
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:41
Yes it's my home. But in the scratch I have an error
/bin/bash: line 0: cd: /home/theg3301/scratch: No such file or directory
total 116
drwx------ 9 theg3301 theg3301  4096 Mar 26 19:01 .
drwxr-xr-x 1 root     root        60 Mar 26 19:41 ..
-rw------- 1 theg3301 theg3301 26022 Mar 26 19:37 .bash_history
-rw-r--r-- 1 theg3301 theg3301    18 Aug 16  2017 .bash_logout
-rw-r--r-- 1 theg3301 theg3301   193 Aug 16  2017 .bash_profile
-rw-r--r-- 1 theg3301 theg3301   293 Mar 23 16:41 .bashrc
drwx------ 3 theg3301 theg3301  4096 Mar 22 19:51 .config
drwxrwxr-x 2 theg3301 theg3301  4096 Mar 22 18:08 .licenses
drwxrwxr-x 7 theg3301 theg3301  4096 Mar 26 17:58 .nextflow
-rw-rw-r-- 1 theg3301 theg3301  6268 Mar 26 17:58 .nextflow.log
-rw-rw-r-- 1 theg3301 theg3301  9204 Mar 26 17:29 .nextflow.log.1
-rw-rw-r-- 1 theg3301 theg3301  6350 Mar 26 14:27 .nextflow.log.2
-rw-rw-r-- 1 theg3301 theg3301    95 Mar 26 14:26 .nextflow.log.3
-rw-rw-r-- 1 theg3301 theg3301    95 Mar 23 19:05 .nextflow.log.4
-rw-rw-r-- 1 theg3301 theg3301  1239 Mar 22 20:29 .nwchemrc
drwxrwxr-x 2 theg3301 theg3301  4096 Mar 22 20:30 .oracle_jre_usage
drwx------ 2 theg3301 theg3301  4096 Mar 23 18:39 .ssh
-rw-rw-r-- 1 theg3301 theg3301   165 Mar 23 15:25 .wget-hsts
lrwxrwxrwx 1 theg3301 theg3301    16 Aug 16  2017 project -> /project/6001802
drwx------ 2 theg3301 theg3301  4096 Aug 16  2017 projects
lrwxrwxrwx 1 theg3301 theg3301    17 Aug 16  2017 scratch -> /scratch/theg3301
drwxrwxr-x 3 theg3301 theg3301  4096 Mar 26 17:18 src
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:42
it looks working fine
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:42
But the first line ?
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:43
ah true
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:44
I don't know if it's a nextflow or singularity problem
Paolo Di Tommaso
@pditommaso
Mar 26 2018 19:44
because that's a link to a a separate path
you need to check with your sysadmins that /scratch/ is automatically mounted by singularity, I guess no
same for /projects
Guillaume Theaud
@GuillaumeTh
Mar 26 2018 19:49
ok thanks I will check