These are chat archives for dereneaton/ipyrad

26th
Apr 2017
susanmiller-uct
@susanmiller-uct
Apr 26 2017 08:21
Hi, I'm new to ipyrad and I'm trying to get it to work on my university's HPC and I keep getting the following error: "ValueError: Unable to configure handler 'ipyrad': [Errno 13] Permission denied: '/data/ipyrad_log.txt'" I have changed the permission of the "ipyrad_log.txt" file so that it can be read, written to, and executed by owner, group and others, but this doesn't seem to help. Any suggestions would be much appreciated.
susanmiller-uct
@susanmiller-uct
Apr 26 2017 09:19

here's the full error report in case it is useful: Traceback (most recent call last):
File "/opt/exp_soft/anaconda/python2.7/bin/ipyrad", line 11, in <module>
load_entry_point('ipyrad==0.6.15', 'console_scripts', 'ipyrad')()
File "/opt/exp_soft/anaconda/python2.7/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/init.py", line 542, in load_entry_point

File "/opt/exp_soft/anaconda/python2.7/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/init.py", line 2569, in load_entry_point

File "/opt/exp_soft/anaconda/python2.7/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/init.py", line 2229, in load

File "/opt/exp_soft/anaconda/python2.7/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/init.py", line 2235, in resolve

File "/opt/exp_soft/anaconda/python2.7/lib/python2.7/site-packages/ipyrad/init.py", line 152, in <module>
_set_debug_dict(loglevel)
File "/opt/exp_soft/anaconda/python2.7/lib/python2.7/site-packages/ipyrad/init.py", line 147, in _set_debug_dict
'propogate': True
File "/opt/exp_soft/anaconda/python2.7/lib/python2.7/logging/config.py", line 794, in dictConfig
dictConfigClass(config).configure()
File "/opt/exp_soft/anaconda/python2.7/lib/python2.7/logging/config.py", line 576, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'ipyrad': [Errno 13] Permission denied: '/data/ipyrad_log.txt'

Isaac Overcast
@isaacovercast
Apr 26 2017 11:04
@susanmiller-uct Well, it really does look like there's a problem writing to ipyrad_log.txt. It's hard to know what's going on without knowing more about the configuration of the cluster.... When you say you changed permission for this file did you do this on the head node or on the compute node the ipyrad process is running on? Is the /data directory shared across compute nodes? By default ipyrad tries to create this log file in the directory it is executed in, so if the /data directory isn't writeable you'll need to change directory to somewhere that is prior to calling ipyrad.
susanmiller-uct
@susanmiller-uct
Apr 26 2017 12:06

@isaacovercast the "ipyrad_log.txt" file was created in the directory that I have my data and parameter files in (my personal space within the HPC cluster) so I changed the permissions of the file that was created by ipyrad (the default wasn't writable if I remember correctly) . This /data directory should be writeable as I can add files to it? Unless there is something I don't understand? (I'm very new to Linux, so apologies if this something obvious that I need to change!) This is the script that I ran: #PBS -N ipyrad_test

PBS -M susan.miller@uct.ac.za

PBS -m ae

PBS -q UCTlong

PBS -l nodes=1:ppn=1:series600

Please leave the hostname command here for troubleshooting purposes.

hostname

change into the directory with your params file

cd /data/
module load python/anaconda-python-2.7 #this is what the HPC admin said I needed to include to access ipython on the system

call some ipyrad commands

ipyrad -p params-iplark.txt -s 1

susanmiller-uct
@susanmiller-uct
Apr 26 2017 12:36
@isaacovercast I think I have solved the problem - still not sure why it wasn't working in the /data folder, but I made a new folder (in the same directory as I had the "data" one), copied my files over and started again and it is now working! Thanks for your help!
Isaac Overcast
@isaacovercast
Apr 26 2017 13:20
@susanmiller-uct Glad you got it sorted. It could be that the /data directory is mounted differently on the compute nodes. When you created a new directory it probably gave it more permissive read/write access.
rtm79
@rtm79
Apr 26 2017 15:55
Hi,
does anyone have any advice on restarting step 3? my run was terminated on the cluster i'm using because i didn't allocate enough time, and i don't know if there's a way i can restart the job and have it pick up where it left off. thanks...
joqb
@joqb
Apr 26 2017 19:35

Hi @isaacovercast @dereneaton,

I wanted to update to the newest version but it seems something is wrong

-bash-4.2$ ipyrad -v

 -------------------------------------------------------------
  ipyrad [v.0.6.15]
  Interactive assembly and analysis of RAD-seq data
 -------------------------------------------------------------

  A new version of ipyrad is available (v.0.6.16). To upgrade run:

    conda install -c ipyrad ipyrad

ipyrad 0.6.15
-bash-4.2$ conda install -c ipyrad ipyrad
Fetching package metadata ...........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /data/filer-5-2/brassac/miniconda2:
#
ipyrad                    0.6.15                        0    ipyrad
-bash-4.2$

Very excited about the return of the .alleles file ;-)

Deren Eaton
@dereneaton
Apr 26 2017 22:15
@joqb I think I screwed up the upload. If you check now the latest version should be up (on linux, it will be a little while longer until it's up on OSX).
Deren Eaton
@dereneaton
Apr 26 2017 22:22

@rtm79 it depends on how far it was when it stopped whether you will be able to restart from the middle or from the beginning. If some of the samples are finished then they will not be run again when you restart step3 (as long as you do not use the --force flag), but if none of the samples were completely finished with step3 then all of the samples will all run again from the start. We have been meaning to put better checkpointing inside of step3 but we haven't done it yet.

One way to check whether it will restart all of the samples from scratch or only the ones that are not finished -- and to make sure that the existing files do not get removed no matter what -- would be to create a new branch and try rerunning step3 on that one.

### make new branch from old branch
ipyrad -p params-old.txt -b new

## restart step3 *without* using the -f flag (you do not want to overwrite)
ipyrad -p params-new.txt -s 3
rtm79
@rtm79
Apr 26 2017 22:55
@dereneaton thanks, step 3 was on ~individual 795 out of 864, so i think i'm good to go