Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

ArkoudaProject/community

People

Repo info

Activity

Michael Merrill

@mhmerrill

@rp98njit we’ve got someone looking into it, I might have to do with a PR we merged yesterday which made Parquet/Arrow a non optional part of the build

Ethan-DeBandi99

@Ethan-DeBandi99

@mhmerrill /@rp98njit it definitely is related to that. I am looking to see what specifically is not configured to cause that.

@rp98njit Can you confirm a few things for me? Do you have the LD_LIBRARY_PATH environment variable set?

2 replies

Ethan-DeBandi99

@Ethan-DeBandi99

@rp98njit Also, have you run make install-arrow?

Ethan-DeBandi99

@Ethan-DeBandi99

Ok - I believe the issue is that arrow is not installed running make install-arrow should resolve this. I have added an issue to update our install documentation to reflect this

15 replies

Ethan-DeBandi99

@Ethan-DeBandi99

@rp98njit I created a bare bones conda environment to run the install in and make sure everything is functional. Here is the list of packages that need to be installed using conda before make install-arrow will run.

boost-cpp
snappy
thrift-cpp
re2
utf8proc

Please let me know if it gives you any additional problems after installing the packages listed above.

Ethan-DeBandi99

@Ethan-DeBandi99

Additionally, here is the command if using pip,
pip install boost snappy thrift re2 utf8proc

pierce

@pierce314159

Arkouda v2022.06.06 was just released! Thanks to everyone who contributed!
https://github.com/Bears-R-Us/arkouda/releases/tag/v2022.06.06

Michael Merrill

@mhmerrill

Today's Arkouda Weekly Call is cancelled.

Engin Kayraklioglu

@e-kayrakli

@/all In case you missed it, CHIUW is today and we just started. Just register here and hop in! https://hpe.zoom.us/meeting/register/tJEoc-2orzotH9QYrnGAJOIGveblfnGKtNw0 (Ooh, Brad is talking about Arkouda right now)

Rajendra Prasad Patil

@rp98njit

Hi all,
Thanks in advance.

How can I verify? I installed the chapel with the "LLVM=bundled" configuration.

Michael Merrill

@mhmerrill

@rp98njit you should put that question in the chapel gitter channel https://gitter.im/chapel-lang/chapel

pierce

@pierce314159

Arkouda v2022.06.13 was just released!
https://github.com/Bears-R-Us/arkouda/releases/tag/v2022.06.13

Michael Merrill

@mhmerrill

Anyone have a topic for today's Arkouda weekly call? https://github.com/Bears-R-Us/ArkoudaWeeklyCall

@/all Anyone have a topic for today's Arkouda weekly call? https://github.com/Bears-R-Us/ArkoudaWeeklyCall

Michael Merrill

@mhmerrill

@/all if no takers to today's call we will cancel, any takers...

@/all we will spend today's meeting talking about and recapping CHIUW.

pierce

@pierce314159

just circling back on @rp98njit's issue, it was resolved by running conda install -c conda-forge pyarrow

Zhihui Du

@zhihuidu

addEntry problem.
In my tests for a large graph amazon0601.txt, the st.addEntry function will be blocked somewhere after giving the following information
"2022-06-18:11:58:59 [ServerConfig] overMemLimit Line 202 INFO [Chapel] memory high watermark = 267023064 memory limit = 486862464614"
No error information is given. It seems that it enters into an infinite loop.

For small graphs, everything is good.

Any suggestions on solving this problem? Thanks!

Michael Merrill

@mhmerrill

@zhihuidu this message is printed when you allocate more memory that at any point before in the run. From the message it appears that you still have plenty of memory left. I don’t think the message is tied to the behavior you are seeing. Are you using standard Arkouda I/O to read in your data?

Zhihui Du

@zhihuidu

Thanks, Mike!
In the previous tests, I used writeln and it could not show output immediately.
After using the standard Arkouda I/O, I have located the position of the problem.
I am looking into the problem to see why a large graph will have an infinite loop.

Michael Merrill

@mhmerrill

@/all anyone have a topic for today's Arkouda Weekly Call?

Michael Merrill

@mhmerrill

@/all if there are no topics to discuss then we will cancel today call.

Michael Merrill

@mhmerrill

@/all would people be interested in me talking about a new/experimental way of interpreting arkouda array computation?

pierce

@pierce314159

Hi everyone! The Arkouda weekly call for today is canceled

Chris Long

@compiling-is-winning

Does Arkouda have a supported uint32 datatype? I don't see one here: https://bears-r-us.github.io/arkouda/autoapi/arkouda/dtypes/index.html?highlight=dtype#module-arkouda.dtypes

I ask because I'm currently working on a server-side algorithm that will require concatenating two integers (one a set element index, the other a hash index) into a single uint64 that will be used to seed a random number generator. So it would be easiest if both indices above were uint32's, though not necessarily a deal-breaker if not.

Michael Merrill

@mhmerrill

Not in memory but I think you can read and write 32-bit in file I/O. You should ask Bill about this.

There is really no reason to support it other than to conserve memory.

Chris Long

@compiling-is-winning

Thanks! I'm guessing a uint8 would suffice for the hash index (and looks to be supported), so I could probably just put in a check on the set element index to make sure that the concatenated value will still be representable as a uint64

I suppose that I could also use a Cantor pairing function, though I haven't thought this through much either

Chris Long

@compiling-is-winning

The pairing function would also give me an actual uniqueness guarantee (even though I expect that the number of hashes will be something small like 10 or less, so that this isn't a big issue in practice)

pierce

@pierce314159

Hi everyone! The meeting today is canceled since mike is on vacation. Hope everyone had a good 4th!

pierce

@pierce314159

Arkouda v2022.07.08 was just released! Thanks to everyone who contributed!
https://github.com/Bears-R-Us/arkouda/releases/tag/v2022.07.08

Michael Merrill

@mhmerrill

Any topics for the call this afternoon?

Chris Long

@compiling-is-winning

So I've moved from my ancient creaking laptop to the Ubuntu partition on my new desktop (so no Brew) and am now encountering some new build errors. Compilation appears to be successful, but the python tests hang on parquet_test.py (after skipping one). If I do a keyboard interrupt, I get the message "279 passed, 1 skipped". And if I run the Chapel tests, I get an error (appears to be from "UnitTestParquetCpp".

I built the bundled version of arrow, hdf5, and zmq (using "make install-deps") and got their dependencies (I think just Boost) and those of Chapel from the Ubuntu package manager. I'm using chpl version 1.27.0 built with LLVM version 14.0.0 (from the Ubuntu repo, not the bundled Chapel version).

I tried pointing Makefile.paths to my top-level Anaconda directory, but that fails due to missing arrow. I've tried installing arrow via conda-forge, but that seems to take all day, so I've interrupted it every time.

Any ideas what could be causing these issue(s)? Should I go back and try an earlier version of Chapel?

Michael Merrill

@mhmerrill

@compiling-is-winning i’ve got time today to talk about this if you’re available.

Ethan-DeBandi99

@Ethan-DeBandi99

@compiling-is-winning Let me know if you have time to work through the build issues. I think there are a few quick tweaks we can make to get you squared away.

Chris Long

@compiling-is-winning

@mhmerrill @Ethan-DeBandi99 thanks! Sorry for the late response, but I should be available all of tomorrow if either of you are.

Zhihui Du

@zhihuidu

@bmcdonald3 Now, I can compile the arkouda-njit package based on the latest arkouda version. One thing I want to confirm with you. About the user module, why must I give the absolute path of the modules (otherwise it will give an error)? I have given the absolute path in the `ARKOUDA_SERVER_USER_MODULES' part. Or I make something wrong in the command line configuration? Thanks!

Zhihui Du

@zhihuidu

When I use the following configuration to compile chapel
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=udp
export CHPL_LAUNCHER= amudprun
And use the following script to submit the job

!/bin/bash

SBATCH --exclusive

SBATCH -N 2

PortNum=5555
numNodes=2

SBATCH -o $output.log-%j-%a

Get a list of the nodes allocated to this job

temp=scontrol show hostname | paste -d, -s
export SSH_SERVERS=$temp
export LD_LIBRARY_PATH=$HOME/anaconda3/lib:${LD_LIBRARY_PATH}

./arkouda_server -nl $numNodes --ServerConfig.ServerPort=$PortNum 2>&1

It gives me an error

~/Mike/arkouda/arkouda_server_real: error while loading shared libraries: libthrift.so.0.15.0: cannot open shared object file: No such file or directory

under $HOME/anaconda3/lib we have the libthrift.so.0.15.0
Why export LD_LIBRARY_PATH=$HOME/anaconda3/lib:${LD_LIBRARY_PATH} does not work?
Can anyone tell me the reason? Thanks!

Michael Merrill

@mhmerrill

@zhihuidu Maybe the home dir is not mounted on the compute nodes?

Arkouda weekly call is canceled today.

@/all Arkouda weekly call is canceled today.

pierce

@pierce314159

@zhihuidu i'll message you directly about this

Zhihui Du

@zhihuidu

@pierce314159 Yes, your conda env create -f arkouda-env-dev.yml method can work well for a new platform.
I really appreciate your help!

pierce

@pierce314159

excellent!

Chat via Matrix