These are chat archives for nextflow-io/nextflow

15th
Apr 2016
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 09:14
Hi Paolo, are u there?
Evan Floden
@evanfloden
Apr 15 2016 09:15
@JoseEspinosa maybe we can help?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 09:16
It was something I found yesterday night, when reading files from a DB
Files containing parenthesis inside the name crashed because nextflow was not able of doing the symbolic link to the file
Evan Floden
@evanfloden
Apr 15 2016 09:17
Ahh, parenthesis in filenames sounds nasty
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 09:17
I don't know whether he would like to change it or just kill any person that put parenthesis in file names :P
Evan Floden
@evanfloden
Apr 15 2016 09:18
:gun: I think
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 09:19
Moreover in a DB O_o
Emilio Palumbo
@emi80
Apr 15 2016 09:21
HI @JoseEspinosa, it looks bad. Did you try using single quotes around the file names?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 09:23
mat_files_path = "${params.path_files}*.mat"
mat_files = Channel.fromPath(mat_files_path)

process get_speed {

  input:
  file file_worm from mat_files

...
}
Emilio Palumbo
@emi80
Apr 15 2016 09:23
this one '
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 09:25
This was the code and inside the folder I had the files downloaded from the DB
I just deleted all parenthesis from file names
OK
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 09:50
I have another question, I have a nf script with several processes, the last process crashes but I would like to trace back which is the input file that is producing the file (i.e the file input in the first process)
Is there a way I can do this?
Evan Floden
@evanfloden
Apr 15 2016 10:25
If you have sym linked files in the work directory of the process that crashed, they should link to the previous processess work directory. I have backtracked succesfully like this several times
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 11:17
OK thanks
Now I have another issue...
I am resuming a nextflow execution
And I don't know why I am getting a buffer underflow exception
ERROR ~ Error executing process > 'speed_to_pergola (828)'

Caused by:
  Buffer underflow.
Paolo Di Tommaso
@pditommaso
Apr 15 2016 11:19
José..... :/
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 11:19
I went to the folder and everything seems to be ok
and I can run the .command.sh without problems
Hi Paolo!
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 11:29
I found the problem sorry :pensive:
Paolo Di Tommaso
@pditommaso
Apr 15 2016 11:32
And what was causing Buffer underflow?!
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 11:35
I changed the name of the output file in order to solve the previous issue, know from which original input file the file of the process was coming, I guess that when I resumed the old file name and the new file name did not match and it is crashing
Paolo Di Tommaso
@pditommaso
Apr 15 2016 11:36
Buffer underflow looks odd in any case open an issue including the stack trace please
Also, scientists using parentheses in file names should not be entitled to get a PhD .. :)
but nxf should not break, please open an issue also for that
Emilio Palumbo
@emi80
Apr 15 2016 12:30
@JoseEspinosa I make a little test for the issue with parentheses
the following works (I know it's not very straightforward...)
mat_files_path = "${params.path_files}*.mat"
mat_files = Channel.fromPath(mat_files_path)

process get_speed {
    echo true

  input:
  file file_worm from mat_files.map { file("${it.name.replaceAll(/(\(|\))/) { all,p -> "\\${p}" }}") }

  """
  echo ${file_worm}
  """
}
basically it escapes parentheses
Paolo Di Tommaso
@pditommaso
Apr 15 2016 12:32
yep, but nxf should manage it
Emilio Palumbo
@emi80
Apr 15 2016 12:32
I see
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 12:33
It was not a big deal once you know it because you can also just change name files with a pattern matching.
Emilio Palumbo
@emi80
Apr 15 2016 12:34
Sure, but I agree with Paolo this should be managed by nxf
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 12:35
And the main problem is that it was not straightforward to know what the problem was, I realized because I got both files with and without parentheses
Jason Byars
@jbyars
Apr 15 2016 17:59
Is there any option to throw an error when you declare a file that doesn't? I'm looking for a cleaner way to make sure all references exist at the beginning of a pipeline. I.E.
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:02
for example?
Jason Byars
@jbyars
Apr 15 2016 18:03
ref = file("/path/typo", mustexist= true)
// instead of 
ref = file("/path/typo")
if (!ref.exists()) 
  long winded error message
  exit code
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:04
ah, no
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:04
I have this process:
process zeros_bed_and_bedGraph {
  input:
  set file ('bed_file'), val(body_part), val(name_file) from bed_speed
  set file ('bedGraph_file'), val(body_part), val(name_file) from bedGraph_speed


  output:
  set '*.no_na.bed', body_part, name_file into bed_speed_no_nas
  set '*.no_tr.bed', body_part, name_file into bed_speed_no_track_line


  """
  cat ${bed_file} | sed 's/track name=\"1_a\"/track name=\"${body_part}_speed\"/g' > bed_file.tmp
  cat bed_file.tmp | grep -v "\\-10000" > bed_file.no_na.bed
  echo "#no track file" >  bed_file.no_tr.bed
  cat bed_file.no_na.bed | grep -v "track name" >>  bed_file.no_tr.bed
  """


}
and is returning always an exception
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:05
what exception ?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:05
Error executing process > 'zeros_bed_and_bedGraph (6)'

Caused by:
  Process 'zeros_bed_and_bedGraph (6)' terminated with an error exit status

Command executed:

  cat bed_file | sed 's/track name="1_a"/track name="tailTip_speed"/g' > bed_file.tmp
  cat bed_file.tmp | grep -v "\-10000" > bed_file.no_na.bed

  echo "#no track file" >  bed_file.no_tr.bed
  cat bed_file.no_na.bed | grep -v "track name" >>  bed_file.no_tr.bed

Command exit status:
  1

Command output:
  (empty)

Work dir:
  /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option '-resume'
the problem is the last line
Jason Byars
@jbyars
Apr 15 2016 18:05
@pditommaso ok, thanks!
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:06
@jbyars welcome
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:06
but if I enter in the work folder everything is OK and I can run the code
I have tried several changes in the script but nothing works
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:07
not possible
move into work/af/67dc7d64b42037b335a9a7c4270c17
and run it using bash -x .command.run
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:08
exit 1
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:09
there's no error message?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:09
but bash .command.sh works
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:10
that's weird
there's must be an error message when you run it with bash -x .command.run
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:11
+ set -e
+ set -u
+ NXF_DEBUG=0
+ [[ 0 > 2 ]]
+ trap on_exit EXIT
+ trap on_term TERM INT USR1 USR2
+ [[ 0 > 0 ]]
+ touch /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.begin
+ '[' -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.env ']'
+ source /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.env
++ export PATH=/nfs/software/cn/el6.5/python/envs/.virtualenvs/cpython279/bin/pergola_rules.py:/Users/jespinosa/git/phecomp/lib/nxf/:/Users/jespinosa/software/bedTools/bedtools2/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/jespinosa/software/ucscUtils:/Users/jespinosa/software/apache-ant-1.9.4/bin:/Users/jespinosa/software/sesMail/bin:/Users/jespinosa/lib/bash:/Users/jespinosa/software/bin:/Users/jespinosa/phecomp/bin/C:/opt/local/bin:/opt/local/sbin:/Users/jespinosa/git/phecomp/lib/R:/Users/jespinosa/workspace/hmm/:/Users/jespinosa/git/mwm/lib/perl:/Users/jespinosa/phecomp/lib/bash:/Users/jespinosa/git/phecomp/lib/perl:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin
++ PATH=/nfs/software/cn/el6.5/python/envs/.virtualenvs/cpython279/bin/pergola_rules.py:/Users/jespinosa/git/phecomp/lib/nxf/:/Users/jespinosa/software/bedTools/bedtools2/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/jespinosa/software/ucscUtils:/Users/jespinosa/software/apache-ant-1.9.4/bin:/Users/jespinosa/software/sesMail/bin:/Users/jespinosa/lib/bash:/Users/jespinosa/software/bin:/Users/jespinosa/phecomp/bin/C:/opt/local/bin:/opt/local/sbin:/Users/jespinosa/git/phecomp/lib/R:/Users/jespinosa/workspace/hmm/:/Users/jespinosa/git/mwm/lib/perl:/Users/jespinosa/phecomp/lib/bash:/Users/jespinosa/git/phecomp/lib/perl:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin
+ rm -f bed_file
+ rm -f bedGraph_file
+ ln -s /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/d6/0f7b84d58b8d67b01aa095fb39cf74/tr_1_dt_a.bed bed_file
+ ln -s /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/d6/0f7b84d58b8d67b01aa095fb39cf74/tr_1_dt_a.bedGraph bedGraph_file
+ set +e
+ COUT=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.po
+ mkfifo /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.po
+ CERR=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.pe
+ mkfifo /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.pe
+ tee1=72863
+ tee .command.out
+ tee2=72864
+ tee .command.err
+ pid=72865
+ wait 72865
+ /bin/bash /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.run.1
+ ret=1
+ wait 72863 72864
+ on_exit
+ exit_status=1
+ printf 1
+ rm -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.po
+ rm -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.pe
+ exit 1
this is the complete output
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:12
what if NXF_DEBUG=3 bash .command.run
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:13
+ trap on_exit EXIT
+ trap on_term TERM INT USR1 USR2
+ [[ 3 > 0 ]]
+ nxf_env
+ echo '============= task environment ============='
============= task environment =============
+ env
+ sort
+ sed 's/\(.*\)AWS\(.*\)=\(.\{6\}\).*/\1AWS\2=\3xxxxxxxxxxxxx/'
Apple_PubSub_Socket_Render=/tmp/launch-mFQgNI/Render
DBUS_LAUNCHD_SESSION_BUS_SOCKET=/tmp/launch-YYP9dC/unix_domain_listener
DISPLAY=/tmp/launch-2J10rb/org.macosforge.xquartz:0
HOME=/Users/jespinosa
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_31.jdk/Contents/Home
LANG=en_US.UTF-8
LANGUAGE=en
LC_ALL=C
LC_CTYPE=UTF-8
LOGNAME=jespinosa
LS_COLORS=
LS_OPTIONS=--color=auto
MANPATH=/opt/local/share/man:
NXF_DEBUG=3
PATH=/Users/jespinosa/git/phecomp/lib/nxf/:/Users/jespinosa/software/bedTools/bedtools2/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/jespinosa/software/ucscUtils:/Users/jespinosa/software/apache-ant-1.9.4/bin:/Users/jespinosa/software/sesMail/bin:/Users/jespinosa/lib/bash:/Users/jespinosa/software/bin:/Users/jespinosa/phecomp/bin/C:/opt/local/bin:/opt/local/sbin:/Users/jespinosa/git/phecomp/lib/R:/Users/jespinosa/workspace/hmm/:/Users/jespinosa/git/mwm/lib/perl:/Users/jespinosa/phecomp/lib/bash:/Users/jespinosa/git/phecomp/lib/perl:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin
PERL5LIB=/opt/local/lib/perl5/site_perl/5.12.4/:
PWD=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17
SECURITYSESSIONID=186a6
SHELL=/bin/bash
SHLVL=2
SSH_AUTH_SOCK=/tmp/launch-ftSzHJ/Listeners
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=326
TERM_SESSION_ID=E73D7BBC-1AF1-410A-B7FC-8653583BF8E2
TMPDIR=/var/folders/3b/0cr04q0s3vq_0389xtsl6tvh0000gp/T/
USER=jespinosa
_=/usr/bin/env
_JAVA_OPTIONS=-Xmx2G
__CF_USER_TEXT_ENCODING=0x1F6:0:0
__CHECKFIX1436934=1
+ echo '============= task output =================='
============= task output ==================
+ touch /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.begin
+ '[' -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.env ']'
+ source /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.env
++ export PATH=/nfs/software/cn/el6.5/python/envs/.virtualenvs/cpython279/bin/pergola_rules.py:/Users/jespinosa/git/phecomp/lib/nxf/:/Users/jespinosa/software/bedTools/bedtools2/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/jespinosa/software/ucscUtils:/Users/jespinosa/software/apache-ant-1.9.4/bin:/Users/jespinosa/software/sesMail/bin:/Users/jespinosa/lib/bash:/Users/jespinosa/software/bin:/Users/jespinosa/phecomp/bin/C:/opt/local/bin:/opt/local/sbin:/Users/jespinosa/git/phecomp/lib/R:/Users/jespinosa/workspace/hmm/:/Users/jespinosa/git/mwm/lib/perl:/Users/jespinosa/phecomp/lib/bash:/Users/jespinosa/git/phecomp/lib/perl:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin
++ PATH=/nfs/software/cn/el6.5/python/envs/.virtualenvs/cpython279/bin/pergola_rules.py:/Users/jespinosa/git/phecomp/lib/nxf/:/Users/jespinosa/software/bedTools/bedtools2/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/jespinosa/software/ucscUtils:/Users/jespinosa/software/apache-ant-1.9.4/bin:/Users/jespinosa/software/sesMail/bin:/Users/jespinosa/lib/bash:/Users/jespinosa/software/bin:/Users/jespinosa/phecomp/bin/C:/opt/local/bin:/opt/local/sbin:/Users/jespinosa/git/phecomp/lib/R:/Users/jespinosa/workspace/hmm/:/Users/jespinosa/git/mwm/lib/perl:/Users/jespinosa/phecomp/lib/bash:/Users/jespinosa/git/phecomp/lib/perl:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin
+ rm -f bed_file
+ rm -f bedGraph_file
+ ln -s /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/d6/0f7b84d58b8d67b01aa095fb39cf74/tr_1_dt_a.bed bed_file
+ ln -s /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/d6/0f7b84d58b8d67b01aa095fb39cf74/tr_1_dt_a.bedGraph bedGraph_file
+ set +e
+ COUT=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.po
+ mkfifo /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.po
+ CERR=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.pe
+ mkfifo /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.pe
+ tee1=73051
+ tee .command.out
+ tee2=73052
+ tee .command.err
+ pid=73053
+ wait 73053
+ /bin/bash /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.run.1
+ ret=1
+ wait 73051 73052
+ on_exit
+ exit_status=1
+ printf 1
+ rm -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.po
+ rm -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/af/67dc7d64b42037b335a9a7c4270c17/.command.pe
+ exit 1
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:16
there's no .command.err or .command.out that can help ?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:17
nothing
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:17
what is the content of .command.run.1 ?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:18
#!/bin/bash
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 3 ]] && set -x

nxf_tree() {
    declare -a ALL_CHILD
    while read P PP;do
        ALL_CHILD[$PP]+=" $P"
    done < <(ps -e -o pid= -o ppid=)

    stat() {
        local x_ps=$(ps -o pid=,state=,pcpu=,pmem=,vsz=,rss= $1)
        local x_io=$(cat /proc/$1/io 2> /dev/null | sed 's/^.*:\s*//' | tr '\n' ' ')
        local x_vm=$(cat /proc/$1/status 2> /dev/null | egrep 'VmPeak|VmHWM' | sed 's/^.*:\s*//' | sed 's/[\sa-zA-Z]*$//' | tr '\n' ' ')
        [[ ! $x_ps ]] && return 0

        printf "$x_ps"
        if [[ $x_vm ]]; then printf " $x_vm"; else printf " 0 0"; fi
        if [[ $x_io ]]; then printf " $x_io"; fi
        printf "\n"
    }

    walk() {
        stat $1
        for i in ${ALL_CHILD[$1]:=}; do walk $i; done
    }

    walk $1
}

nxf_pstat() {
    local data=$(nxf_tree $1)
    local tot=''
    if [[ "$data" ]]; then
      tot=$(awk '{ t3+=($3*10); t4+=($4*10); t5+=$5; t6+=$6; t7+=$7; t8+=$8; t9+=$9; t10+=$10; t11+=$11; t12+=$12; t13+=$13; t14+=$14 } END { printf "%d 0 %.0f %.0f %.0f %.0f %.0f %.0f %.0f %.0f %.0f %.0f %.0f %.0f\n", NR,t3,t4,t5,t6,t7,t8,t9,t10,t11,t12,t13,t14 }' <<< "$data")
      printf "$tot\n"
    fi
}

nxf_sleep() {
  if [[ $1 < 0 ]]; then sleep 5;
  elif [[ $1 < 10 ]]; then sleep 0.1;
  elif [[ $1 < 130 ]]; then sleep 1;
  else sleep 5; fi
}

nxf_date() {
    case `uname` in
        Darwin) if hash gdate 2>/dev/null; then echo 'gdate +%s%3N'; else echo 'date +%s000'; fi;;
        *) echo 'date +%s%3N';;
    esac
}

NXF_DATE=$(nxf_date)

nxf_trace() {
  local pid=$1; local trg=$2;
  local tot;
  local count=0;
  declare -a max=(); for i in {0..13}; do max[i]=0; done
  while [[ true ]]; do
    tot=$(nxf_pstat $pid)
    [[ ! $tot ]] && break
    IFS=' ' read -a val <<< "$tot"; unset IFS
    for i in {0..13}; do
      [ ${val[i]} -gt ${max[i]} ] && max[i]=${val[i]}
    done
    echo "pid state %cpu %mem vmem rss peak_vmem peak_rss rchar wchar syscr syscw read_bytes write_bytes" > $trg
    echo "${max[@]}" >> $trg
    nxf_sleep $count
    count=$((count+1))
  done
}

trap 'exit ${ret:=$?}' EXIT
touch .command.trace
start_millis=$($NXF_DATE)
(
/bin/bash -ue /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/5c/af73865b6c41f03b1152dae29f46e9/.command.sh
) &
pid=$!
nxf_trace "$pid" .command.trace &
mon=$!
wait $pid || ret=$?
end_millis=$($NXF_DATE)
kill $mon || wait $mon
echo $((end_millis-start_millis)) >> .command.trace
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:19
are you using the -with-trace option ?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:19
not now but I can do it because is a toy example to develop the script
wait
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:20
no
are you using -with-timeline option?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:21
no but I have a config file
maybe there is something inside
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:22
ok, remove both timeline and trace if any
and relaunch it and try to debug it again
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:22
  profiles {cluster {
    process {
      executor = 'sge'
  queue = 'cn-el7'
  // This line send mail when something is not working
      clusterOptions = '-m ae -M jespinosa@ant-login5.linux.crg.es'
    }
  }
}
trace {
  enabled = true
    file = 'pipeline_trace.csv'
    fields = 'task_id,name,status,exit,realtime,%cpu,rss'
}
env {
    PATH="/nfs/software/cn/el6.5/python/envs/.virtualenvs/cpython279/bin/pergola_rules.py:$PATH"
}
//Old version
//executor {
//    name = 'sge'
//    queue = 'cn-el6'
//}
so yes trace is enabled
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:23
put trace = false
em, no
enabled = false
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:23
voy
I keep having the same problem, how should it help to have the trace set to false?
It is very strange because when I go to the working folder all files are there...
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:27
this command what does it print now?
NXF_DEBUG=3 bash .command.run
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:29
+ trap on_exit EXIT
+ trap on_term TERM INT USR1 USR2
+ [[ 3 > 0 ]]
+ nxf_env
+ echo '============= task environment ============='
============= task environment =============
+ env
+ sort
+ sed 's/\(.*\)AWS\(.*\)=\(.\{6\}\).*/\1AWS\2=\3xxxxxxxxxxxxx/'
Apple_PubSub_Socket_Render=/tmp/launch-mFQgNI/Render
DBUS_LAUNCHD_SESSION_BUS_SOCKET=/tmp/launch-YYP9dC/unix_domain_listener
DISPLAY=/tmp/launch-2J10rb/org.macosforge.xquartz:0
HOME=/Users/jespinosa
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_31.jdk/Contents/Home
LANG=en_US.UTF-8
LANGUAGE=en
LC_ALL=C
LC_CTYPE=UTF-8
LOGNAME=jespinosa
LS_COLORS=
LS_OPTIONS=--color=auto
MANPATH=/opt/local/share/man:
NXF_DEBUG=3
PATH=/Users/jespinosa/git/phecomp/lib/nxf/:/Users/jespinosa/software/bedTools/bedtools2/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/jespinosa/software/ucscUtils:/Users/jespinosa/software/apache-ant-1.9.4/bin:/Users/jespinosa/software/sesMail/bin:/Users/jespinosa/lib/bash:/Users/jespinosa/software/bin:/Users/jespinosa/phecomp/bin/C:/opt/local/bin:/opt/local/sbin:/Users/jespinosa/git/phecomp/lib/R:/Users/jespinosa/workspace/hmm/:/Users/jespinosa/git/mwm/lib/perl:/Users/jespinosa/phecomp/lib/bash:/Users/jespinosa/git/phecomp/lib/perl:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin
PERL5LIB=/opt/local/lib/perl5/site_perl/5.12.4/:
PWD=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64
SECURITYSESSIONID=186a6
SHELL=/bin/bash
SHLVL=2
SSH_AUTH_SOCK=/tmp/launch-ftSzHJ/Listeners
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=326
TERM_SESSION_ID=E73D7BBC-1AF1-410A-B7FC-8653583BF8E2
TMPDIR=/var/folders/3b/0cr04q0s3vq_0389xtsl6tvh0000gp/T/
USER=jespinosa
_=/usr/bin/env
_JAVA_OPTIONS=-Xmx2G
__CF_USER_TEXT_ENCODING=0x1F6:0:0
__CHECKFIX1436934=1
+ echo '============= task output =================='
============= task output ==================
+ touch /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.begin
+ '[' -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.env ']'
+ source /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.env
++ export PATH=/nfs/software/cn/el6.5/python/envs/.virtualenvs/cpython279/bin/pergola_rules.py:/Users/jespinosa/git/phecomp/lib/nxf/:/Users/jespinosa/software/bedTools/bedtools2/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/jespinosa/software/ucscUtils:/Users/jespinosa/software/apache-ant-1.9.4/bin:/Users/jespinosa/software/sesMail/bin:/Users/jespinosa/lib/bash:/Users/jespinosa/software/bin:/Users/jespinosa/phecomp/bin/C:/opt/local/bin:/opt/local/sbin:/Users/jespinosa/git/phecomp/lib/R:/Users/jespinosa/workspace/hmm/:/Users/jespinosa/git/mwm/lib/perl:/Users/jespinosa/phecomp/lib/bash:/Users/jespinosa/git/phecomp/lib/perl:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin
++ PATH=/nfs/software/cn/el6.5/python/envs/.virtualenvs/cpython279/bin/pergola_rules.py:/Users/jespinosa/git/phecomp/lib/nxf/:/Users/jespinosa/software/bedTools/bedtools2/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/jespinosa/software/ucscUtils:/Users/jespinosa/software/apache-ant-1.9.4/bin:/Users/jespinosa/software/sesMail/bin:/Users/jespinosa/lib/bash:/Users/jespinosa/software/bin:/Users/jespinosa/phecomp/bin/C:/opt/local/bin:/opt/local/sbin:/Users/jespinosa/git/phecomp/lib/R:/Users/jespinosa/workspace/hmm/:/Users/jespinosa/git/mwm/lib/perl:/Users/jespinosa/phecomp/lib/bash:/Users/jespinosa/git/phecomp/lib/perl:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Library/TeX/texbin
+ rm -f bed_file
+ rm -f bedGraph_file
+ ln -s /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/8b/c968879d811baa8f93140ab9757f51/tr_1_dt_a.bed bed_file
+ ln -s /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/8b/c968879d811baa8f93140ab9757f51/tr_1_a.bedGraph bedGraph_file
+ set +e
+ COUT=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.po
+ mkfifo /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.po
+ CERR=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.pe
+ mkfifo /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.pe
+ tee1=79622
+ tee .command.out
+ tee2=79623
+ tee .command.err
+ pid=79624
+ wait 79624
+ /bin/bash -ue /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.sh
+ ret=1
+ wait 79622 79623
+ on_exit
+ exit_status=1
+ printf 1
+ rm -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.po
+ rm -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/f5/0fcd9b2a4552fe80151f1072908a64/.command.pe
+ exit 1
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:32
delete the file .command.env
and run it again with bash -x .command.run
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:34
+ set -e
+ set -u
+ NXF_DEBUG=0
+ [[ 0 > 2 ]]
+ trap on_exit EXIT
+ trap on_term TERM INT USR1 USR2
+ [[ 0 > 0 ]]
+ touch /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.begin
+ '[' -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.env ']'
+ rm -f bed_file
+ rm -f bedGraph_file
+ ln -s /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/d3/6f47877671a76e62492d76f8059a47/tr_1_dt_a.bed bed_file
+ ln -s /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/d3/6f47877671a76e62492d76f8059a47/tr_1_dt_a.bedGraph bedGraph_file
+ set +e
+ COUT=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.po
+ mkfifo /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.po
+ CERR=/Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.pe
+ mkfifo /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.pe
+ tee1=83782
+ tee .command.out
+ tee2=83783
+ tee .command.err
+ pid=83784
+ wait 83784
+ /bin/bash -ue /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.sh
+ ret=1
+ wait 83782 83783
+ on_exit
+ exit_status=1
+ printf 1
+ rm -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.po
+ rm -f /Users/jespinosa/git/pergola/test/c_elegans_data_test/work/82/86f4ba65ef9e92f0e8019c038fe781/.command.pe
+ exit 1
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:34
ok, i give up :)
anyway there's something wrong with your script
it returns 1
thus error
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:36
I was trying some stuff this afternoon and by only deleting the line `cat bed_file.no_na.bed | grep -v "track name" >> bed_file.no_tr.bed
it worked
but when you go to the folder the files are there and have content
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:39
it is not
I guess if you run echo $? it prints 1 AFTER that command
right?
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:40
it is not?
101
after the command
of after the nextflow execution
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:41
no after nextflow
after running your bash script
only that fragment
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:42
yeah
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:42
so you need to pay a :beer: to me
:)
grep returns an error code when it finds no matches
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:43
I think I got it...
no that's not the problem, wait 1 sec
we are close
you are right...
buaaaaaaah
hijo de %$%&&*
grep I mean
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:49
There is any easy fix for this you can easily think about
without changing the code too much
I got it! Yeah I have to pay you a :beer:
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:51
with the fix two :beer: :beer:
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:51
Thanks a lot!!!!
Paolo Di Tommaso
@pditommaso
Apr 15 2016 18:51
:)
append || true to the command
Jose Espinosa-Carrasco
@JoseEspinosa
Apr 15 2016 18:53
Two :beers: done!!!! :smile: