These are chat archives for dereneaton/ipyrad

27th
Jul 2018
Isaac Overcast
@isaacovercast
Jul 27 2018 01:55
@tommydevitt The PATH variable needs to include the directory where the structure binary lives. Unless you have the structure binary in $WORK/Applications, this is not the right format. Where exactly is the structure binary that you compiled?
@LinaValencia85 Have you run step 6 previously? What version of ipyrad are you running? Did you do any branching to remove samples? Did you try running step 6 with the -f flag?
tommydevitt
@tommydevitt
Jul 27 2018 13:45
@isaacovercast Sorry, should have been more explicit. The structure binary is inside the folder called 'structure' within Applications: /WORK/Applications/structure
LinaValencia85
@LinaValencia85
Jul 27 2018 13:47
@isaacovercast I am using version 0.6.21 (I know its old but is the one I have used for previous samples from other libraries and want to keep it consistent). This step has work perfectly for other jobs. I have re-run it with -f and got the same error. I did not do any branching but merge some samples.
Isaac Overcast
@isaacovercast
Jul 27 2018 14:49
@tommydevitt Did you put the export in your .bashrc/.bash_profile file? What happens when you run structureby hand at the command line?
tommydevitt
@tommydevitt
Jul 27 2018 14:52
@isaacovercast No, I didn't, and I see now that ~/.bashrc is not automatically sourced on login shells for my HPC. I'll try it. I can run Structure fine from the command line though.
Isaac Overcast
@isaacovercast
Jul 27 2018 14:52
@LinaValencia85 What step did you merge samples during? Version 0.6.21 is quite old, and I don't think we'll be able to very effectively debug this problem, unless it's something simple. It might be worth trying merging samples at an earlier step... I would definitely recommend upgrading. The newest version is the only version guaranteed to be accurate to the best of our ability.
@tommydevitt Yes, you have to make sure the structure binary is available in the PATH on the compute nodes.
tommydevitt
@tommydevitt
Jul 27 2018 15:49
@isaacovercast I added the export to my .bashrc and .profile files. When I run the ipa.structure call, input files are created as before, but the analysis still fails.
LinaValencia85
@LinaValencia85
Jul 27 2018 15:50
@isaacovercast I have merged after s3, the reason I do it, is because my samples have so many reads that they only run successfully independently as I am unable to partition the job in the cluster I used. Has the clustering algorithm changed between versions?
LinaValencia85
@LinaValencia85
Jul 27 2018 16:29
I think the problem is that after s6 the .clust.hdf5 is not being created. My only concern of using another version of ipyrad is that I am not sure whether the .clustS files generated with the old version are compatible with the newest version. And I dont want to re-run hundreds of samples.
Isaac Overcast
@isaacovercast
Jul 27 2018 16:33
@LinaValencia85 How many samples do you have? The clustering algo didn't change between versions, and the format of the .clustS files after step 3 hasn't changed.
Isaac Overcast
@isaacovercast
Jul 27 2018 17:17
@tommydevitt It's still either a path issue or a structure binary glibc error, are you sure the compute nodes use the same version of glibc? You can try getting an interactive session on a compute node to run by hand. These are all really cluster specific issues so it might be easier for one of the cluster admins to help track down these issues.
tommydevitt
@tommydevitt
Jul 27 2018 17:19
@isaacovercast I assume the compute nodes use the same glibc, but I don't know. You're right though, this isn't ipyrad-specific so I'll take it to my HPC admins. Thanks for all your help troubleshooting though, I really appreciate it.
tommydevitt
@tommydevitt
Jul 27 2018 20:40
@isaacovercast @dereneaton Hey guys, to reload an assembly in the API is it ip.load_json or ip.load_assembly? Only the former worked for me but the docs say ip.load_assembly (or ip.load.load_assembly in one place, which I assume is a typo). Thanks.