Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

resbaz/Arizona

All things Research Bazaar Arizona! ☀️🌵💻 Need help with computing? Your fellow researchers are here to help/commiserate.

People

Repo info

Activity

Julian Pistorius

@julianpistorius

Kinda tricky to explain. It involves Docker. :D

Devin Bayly

@DevinBayly

your*
oh interesting! I've never had success with webby docker things but maybe this could change all that. I'll get the scoop on thursday eh?

Julian Pistorius

@julianpistorius

Yep! Maybe even a proof of concept!

Devin Bayly

@DevinBayly

@julianpistorius just ordered my first zero phone part .. here we go side project hahah

Elaheh Alizadeh

@Elaheh-Alizadeh

Has anyone used job array to run matlab code on cluster that launches parpool?

I am using UAHPC . I am having issues for some of the jobs. To me it looks like only limited number of jobs can use parpool at the same time!

Here is examples of errors for some jobs:

Caused by:
Error using parallel.internal.pool.InteractiveClient>iThrowWithCause (line
678)
Failed to start pool.
Error using parallel.Cluster/createConcurrentJob (line 1145)
Unable to write to MAT-file
/home/u5/elahehalizadeh/.matlab/local_cluster_jobs/R2019a/Job265.in.mat.
The file may be corrupt.

==> ThirdStep_TISMorph.e2586980.21 <==
Parallel pool failed to start with the following error. For more detailed
information, validate the profile 'local' in the Cluster Profile Manager.

Caused by:
Error using parallel.internal.pool.InteractiveClient>iThrowWithCause (line
678)
Failed to start pool.
Error using parallel.Job/createTask (line 320)
Only one task may be created on a concurrent Job.

==> ThirdStep_TISMorph.e2586978.38 <==
Caused by:
Error using parallel.internal.pool.InteractiveClient>iThrowWithCause (line
678)
Failed to start pool.
Error using
parallel.internal.customattr.CustomGetSet/hSetAllowNonPublic (line 87)
Cannot create 'Task1.in.mat' because
'/home/u5/elahehalizadeh/.matlab/local_cluster_jobs/R2019a/Job253' does
not exist.

Julian Pistorius

@julianpistorius

Hi Elaheh. Let me check.

Julian Pistorius

@julianpistorius

Somebody at Hacky Hour says:

Tell her to contact HPC consult, may be a limit we set on the number of jobs. Tell her to make sure to include code

Elaheh Alizadeh

@Elaheh-Alizadeh

Thanks. I have. No response yet! I emailed them again.

Julian Pistorius

@julianpistorius

Hmm... @bjoyce3?

Julian Pistorius

@julianpistorius

@Elaheh-Alizadeh, Blake says he'll get in touch with you. Apparently this involves some code specific stuff. 🤓💻

Elaheh Alizadeh

@Elaheh-Alizadeh

okay thanks!

Blake Joyce

@bjoyce3

Hey Elaheh, we're looking into the issue. The first of the year has been kinda crazy for us.

We think it may be a licensing issue....in that you're asking for a number of processes that exceeds how many Matlab licenses we have. But we're not sure

Blake Joyce

@bjoyce3

@Elaheh-Alizadeh this line has me a bit confused: /home/u5/elahehalizadeh/.matlab/local_cluster_jobs/R2019a/Job265.in.mat.
The file may be corrupt.

Can you check the matlab code you're using to make sure that file transferred properly and isn't corrupt?

Elaheh Alizadeh

@Elaheh-Alizadeh

I am running a job array and it pulls the same code for all the jobs. The code processes bunch of images and for different jobs it just pulls different set of images. I do not think the code is corrupted since it ran successfully for some jobs. The job is supposed to launch parpool and run the images. So, if there is an issue with launching the parpool it will stop in first 2-3 minutes. I just reran that specific job with the error you mentioned and it is running with no error. I feel like I can only launch parpool once at a given time. That's because I see the jobs that start at the same time fail launching parpool except one job. Is there a way to delay array jobs so each starts 4 minutes after the job before?

Blake Joyce

@bjoyce3

I would assume there's a way to do that

I'm not familiar with a wait function off the top of head for PBS arrays, but I'll ask the HPC consultants tomorrow to get in touch with you

It is interesting that rerunning the job meant it worked fine. Have you sent us the PBS array file too?

Elaheh Alizadeh

@Elaheh-Alizadeh

I have in my first email.

Elaheh Alizadeh

@Elaheh-Alizadeh

Now I am 100% sure that it is some kind of license issue. I ran one job at a time (10 jobs that had error before) with no issue.

Blake Joyce

@bjoyce3

We'll dig in. It'll take some time though

Elaheh Alizadeh

@Elaheh-Alizadeh

Okay. Let me know when you know how to delay the each job for couple of minutes. This should solve my problem for now.

Blake Joyce

@bjoyce3

Will do. Thanks for being patient with us. Things will get better on our response time once the first month is over!

Elaheh Alizadeh

@Elaheh-Alizadeh

Thanks

aconkli4

@aconkli4

Blake, Julian, do either of you know where I could get help implementing locality sensitive hashing?

Julian Pistorius

@julianpistorius

Hi @aconkli4! What's the context?

aconkli4

@aconkli4

Hi Julian!
I am trying to implement an implicit correlation network analysis (finding modules in a network without calculating all the edge weights). One of the steps is to convert negative edge weights into positive edge weights. This is accomplished by squaring the edge weights. The goal (I think) is to utilize a set of hashing functions so that I can approximate the squared dot products implicitly.

Julian Pistorius

@julianpistorius

Interesting! What have you looked at so far? Are there any libraries which do approximately what you need?

aconkli4

@aconkli4

Maybe. My primary problem is figuring out how to sausage grind the math into code. There may be some library that can help me, but I probably wouldn't know if I saw it.
Unfortunately, my Youtube level education on LSH is not measuring up well to the math in this dissertation I am reading.

So far I have googled a lot of LSH tutorials and had min-hashing and Jaccard distance between text documents explained to me a half dozen times.

aconkli4

@aconkli4

So maybe a better question is, do you know really a patient math/computer science tutor that would be willing to put up with my math inadequacy for a while?

@Elaheh-Alizadeh I just realized you are in this group! Would you maybe be willing to help me some time this week?

Julian Pistorius

@julianpistorius

Hmm... I wonder if Ian Bertolacci might be able to help.

I've asked on the Python Meetup Slack.

Alex Pakalniskis

@alex-pakalniskis

@aconkli4 I have weekly Python drop-in hours on Monday evenings (https://new.library.arizona.edu/events/python-programming-drop-hours) and I'm patient. Happy to help however I can. Feel free to message me.

https://docs.python.org/3/library/hashlib.html

Elaheh Alizadeh

@Elaheh-Alizadeh

Sure, How about this afternoon?

aconkli4

@aconkli4

Thanks @alex-pakalniskis and @Elaheh-Alizadeh ! Elaheh, that sounds good. Are you free after seminar this afternoon?
Alex, I might take you up on that. I won't make it this coming Monday, but the following drop-in you may see me

Elaheh Alizadeh

@Elaheh-Alizadeh

I will not be at the seminar. You can find me at cancer center #4906B after the seminar.

aconkli4

@aconkli4

Okay thanks