Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Chris Long
    @compiling-is-winning
    @mhmerrill @pierce314159 thanks, that cleared up all but the dataframe_test issue with the Python tests on my Mac. Still fighting cmake on Sherlock
    Michael Merrill
    @mhmerrill
    Anyone have a topic for today’s call?
    Michael Merrill
    @mhmerrill
    Today's Arkouda Call will be further discussion of Radix sort structure and mason packages , are people ok with these topics?
    Rajendra Prasad Patil
    @rp98njit

    Hi everyone,

    I am trying to build arkouda
    I am getting this error.

    make: Entering directory /home/r/rp98/arkouda' make compile-arrow-cpp make[1]: Entering directory/home/r/rp98/arkouda'
    g++ -O3 -std=c++11 -c /home/r/rp98/arkouda//src/ArrowFunctions.cpp -o /home/r/rp98/arkouda//src/ArrowFunctions.o -I/home/r/rp98/anaconda3/envs/arkouda/include -L/home/r/rp98/anaconda3/envs/arkouda/lib
    In file included from /home/r/rp98/arkouda//src/ArrowFunctions.cpp:1:0:
    /home/r/rp98/arkouda//src/ArrowFunctions.h:7:10: fatal error: arrow/api.h: No such file or directory

    include <arrow/api.h>

          ^~~~~~~~~~~~~

    compilation terminated.
    make[1]: [compile-arrow-cpp] Error 1
    make[1]: Leaving directory `/home/r/rp98/arkouda'
    make:
    [/home/r/rp98/arkouda//src/ArrowFunctions.o] Error 2
    make: Leaving directory `/home/r/rp98/arkouda'
    (arkouda) login-1-57 arkouda-njit >:

    Thank you in advance

    Michael Merrill
    @mhmerrill
    @rp98njit we’ve got someone looking into it, I might have to do with a PR we merged yesterday which made Parquet/Arrow a non optional part of the build
    Ethan-DeBandi99
    @Ethan-DeBandi99
    @mhmerrill /@rp98njit it definitely is related to that. I am looking to see what specifically is not configured to cause that.
    @rp98njit Can you confirm a few things for me? Do you have the LD_LIBRARY_PATH environment variable set?
    2 replies
    Ethan-DeBandi99
    @Ethan-DeBandi99
    @rp98njit Also, have you run make install-arrow?
    Ethan-DeBandi99
    @Ethan-DeBandi99
    Ok - I believe the issue is that arrow is not installed running make install-arrow should resolve this. I have added an issue to update our install documentation to reflect this
    15 replies
    Ethan-DeBandi99
    @Ethan-DeBandi99

    @rp98njit I created a bare bones conda environment to run the install in and make sure everything is functional. Here is the list of packages that need to be installed using conda before make install-arrow will run.

    • boost-cpp
    • snappy
    • thrift-cpp
    • re2
    • utf8proc

    Please let me know if it gives you any additional problems after installing the packages listed above.

    Ethan-DeBandi99
    @Ethan-DeBandi99
    Additionally, here is the command if using pip,
    pip install boost snappy thrift re2 utf8proc
    pierce
    @pierce314159
    Arkouda v2022.06.06 was just released! Thanks to everyone who contributed!
    https://github.com/Bears-R-Us/arkouda/releases/tag/v2022.06.06
    Michael Merrill
    @mhmerrill
    Today's Arkouda Weekly Call is cancelled.
    Engin Kayraklioglu
    @e-kayrakli
    @/all In case you missed it, CHIUW is today and we just started. Just register here and hop in! https://hpe.zoom.us/meeting/register/tJEoc-2orzotH9QYrnGAJOIGveblfnGKtNw0 (Ooh, Brad is talking about Arkouda right now)
    Rajendra Prasad Patil
    @rp98njit

    Hi all,
    Thanks in advance.

    How can I verify? I installed the chapel with the "LLVM=bundled" configuration.

    Michael Merrill
    @mhmerrill
    @rp98njit you should put that question in the chapel gitter channel https://gitter.im/chapel-lang/chapel
    pierce
    @pierce314159
    Michael Merrill
    @mhmerrill
    Anyone have a topic for today's Arkouda weekly call? https://github.com/Bears-R-Us/ArkoudaWeeklyCall
    @/all Anyone have a topic for today's Arkouda weekly call? https://github.com/Bears-R-Us/ArkoudaWeeklyCall
    Michael Merrill
    @mhmerrill
    @/all if no takers to today's call we will cancel, any takers...
    @/all we will spend today's meeting talking about and recapping CHIUW.
    pierce
    @pierce314159
    just circling back on @rp98njit's issue, it was resolved by running conda install -c conda-forge pyarrow
    Zhihui Du
    @zhihuidu

    addEntry problem.
    In my tests for a large graph amazon0601.txt, the st.addEntry function will be blocked somewhere after giving the following information
    "2022-06-18:11:58:59 [ServerConfig] overMemLimit Line 202 INFO [Chapel] memory high watermark = 267023064 memory limit = 486862464614"
    No error information is given. It seems that it enters into an infinite loop.

    For small graphs, everything is good.

    Any suggestions on solving this problem? Thanks!

    Michael Merrill
    @mhmerrill
    @zhihuidu this message is printed when you allocate more memory that at any point before in the run. From the message it appears that you still have plenty of memory left. I don’t think the message is tied to the behavior you are seeing. Are you using standard Arkouda I/O to read in your data?
    Zhihui Du
    @zhihuidu
    Thanks, Mike!
    In the previous tests, I used writeln and it could not show output immediately.
    After using the standard Arkouda I/O, I have located the position of the problem.
    I am looking into the problem to see why a large graph will have an infinite loop.
    Michael Merrill
    @mhmerrill
    @/all anyone have a topic for today's Arkouda Weekly Call?
    Michael Merrill
    @mhmerrill
    @/all if there are no topics to discuss then we will cancel today call.
    Michael Merrill
    @mhmerrill
    @/all would people be interested in me talking about a new/experimental way of interpreting arkouda array computation?
    pierce
    @pierce314159
    Hi everyone! The Arkouda weekly call for today is canceled
    Chris Long
    @compiling-is-winning
    I ask because I'm currently working on a server-side algorithm that will require concatenating two integers (one a set element index, the other a hash index) into a single uint64 that will be used to seed a random number generator. So it would be easiest if both indices above were uint32's, though not necessarily a deal-breaker if not.
    Michael Merrill
    @mhmerrill
    Not in memory but I think you can read and write 32-bit in file I/O. You should ask Bill about this.
    There is really no reason to support it other than to conserve memory.
    Chris Long
    @compiling-is-winning
    Thanks! I'm guessing a uint8 would suffice for the hash index (and looks to be supported), so I could probably just put in a check on the set element index to make sure that the concatenated value will still be representable as a uint64
    I suppose that I could also use a Cantor pairing function, though I haven't thought this through much either
    Chris Long
    @compiling-is-winning
    The pairing function would also give me an actual uniqueness guarantee (even though I expect that the number of hashes will be something small like 10 or less, so that this isn't a big issue in practice)
    pierce
    @pierce314159
    Hi everyone! The meeting today is canceled since mike is on vacation. Hope everyone had a good 4th!
    pierce
    @pierce314159
    Arkouda v2022.07.08 was just released! Thanks to everyone who contributed!
    https://github.com/Bears-R-Us/arkouda/releases/tag/v2022.07.08
    Michael Merrill
    @mhmerrill
    Any topics for the call this afternoon?
    Chris Long
    @compiling-is-winning
    So I've moved from my ancient creaking laptop to the Ubuntu partition on my new desktop (so no Brew) and am now encountering some new build errors. Compilation appears to be successful, but the python tests hang on parquet_test.py (after skipping one). If I do a keyboard interrupt, I get the message "279 passed, 1 skipped". And if I run the Chapel tests, I get an error (appears to be from "UnitTestParquetCpp".
    I built the bundled version of arrow, hdf5, and zmq (using "make install-deps") and got their dependencies (I think just Boost) and those of Chapel from the Ubuntu package manager. I'm using chpl version 1.27.0 built with LLVM version 14.0.0 (from the Ubuntu repo, not the bundled Chapel version).
    I tried pointing Makefile.paths to my top-level Anaconda directory, but that fails due to missing arrow. I've tried installing arrow via conda-forge, but that seems to take all day, so I've interrupted it every time.
    Any ideas what could be causing these issue(s)? Should I go back and try an earlier version of Chapel?
    Michael Merrill
    @mhmerrill
    @compiling-is-winning i’ve got time today to talk about this if you’re available.
    Ethan-DeBandi99
    @Ethan-DeBandi99
    @compiling-is-winning Let me know if you have time to work through the build issues. I think there are a few quick tweaks we can make to get you squared away.
    Chris Long
    @compiling-is-winning
    @mhmerrill @Ethan-DeBandi99 thanks! Sorry for the late response, but I should be available all of tomorrow if either of you are.
    Zhihui Du
    @zhihuidu
    @bmcdonald3 Now, I can compile the arkouda-njit package based on the latest arkouda version. One thing I want to confirm with you. About the user module, why must I give the absolute path of the modules (otherwise it will give an error)? I have given the absolute path in the `ARKOUDA_SERVER_USER_MODULES' part. Or I make something wrong in the command line configuration? Thanks!
    Zhihui Du
    @zhihuidu

    When I use the following configuration to compile chapel
    export CHPL_COMM=gasnet
    export CHPL_COMM_SUBSTRATE=udp
    export CHPL_LAUNCHER= amudprun
    And use the following script to submit the job

    !/bin/bash

    SBATCH --exclusive

    SBATCH -N 2

    PortNum=5555
    numNodes=2

    SBATCH -o $output.log-%j-%a

    Get a list of the nodes allocated to this job

    temp=scontrol show hostname | paste -d, -s
    export SSH_SERVERS=$temp
    export LD_LIBRARY_PATH=$HOME/anaconda3/lib:${LD_LIBRARY_PATH}

    ./arkouda_server -nl $numNodes --ServerConfig.ServerPort=$PortNum 2>&1

    It gives me an error

    ~/Mike/arkouda/arkouda_server_real: error while loading shared libraries: libthrift.so.0.15.0: cannot open shared object file: No such file or directory

    under $HOME/anaconda3/lib we have the libthrift.so.0.15.0
    Why export LD_LIBRARY_PATH=$HOME/anaconda3/lib:${LD_LIBRARY_PATH} does not work?
    Can anyone tell me the reason? Thanks!

    Michael Merrill
    @mhmerrill
    @zhihuidu Maybe the home dir is not mounted on the compute nodes?
    Arkouda weekly call is canceled today.