Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    dgrichardson
    @dgrichardson
    I don't think anything sent a SIGTERM to the server.
    Elliot Ronaghan
    @ronawho
    Any chance you hit a workload manager timelimit?
    Michael Merrill
    @mhmerrill
    do you use SLURM? it could be what Elliot said.
    https://bears-r-us.github.io/arkouda/autoapi/arkouda/client/index.html#arkouda.client.get_mem_used
    Michael Merrill
    @mhmerrill
    @dgrichardson there are client-side queries to the server, documentation is found here https://bears-r-us.github.io/arkouda/autoapi/arkouda/client/index.html
    dgrichardson
    @dgrichardson
    I'm using ssh. I took the nodes out of our cluster so I'm the only one using them.
    Elliot Ronaghan
    @ronawho

    Hmm, we've had one other user getting a SIGTERM like this, but we haven't found the root cause yet. Part of the problem is just how vague the error message is.

    Do you have any more information about what was happening when this error occurred? (Time since you launched, was this in the middle of an operation or while the server was idle?)

    Michael Merrill
    @mhmerrill
    @dgrichardson what version of HDF5 are you using? we are working on support for 1.12.x , we currently require 1.10.x, @glitch
    glitch
    @glitch
    and I'm really close to getting it working for HDF5 1.12.x
    Bears-R-Us/arkouda#975
    at least from our unit/functional test perspective
    dgrichardson
    @dgrichardson
    I let arkouda's makefile download whatever version of hdf it wanted.
    glitch
    @glitch
    ah, ok, you probably got 1.10.5
    dgrichardson
    @dgrichardson
    @ronawho: The server had been up for less than an hour, there was one user connected. They were in the process of creating arrays to see what happened if we used to much memory. It wasn't clear if it happened as part of a command or in between a command. All the command messages were complete, so it didn't look like two threads printing at the same time. Do you think the "Spawner" part was related? That looks like it could be a socket getting closed and someone getting a 0 byte read. Who knows how likely that is to be the problem, but it might be easy to just look for socket operations (my guess is arkouda does not have that many, but maybe there are lots in lower layers like chapel and gasnet).
    The server has been up for ~18 hours with no problems, but we also haven't been using it as much. It feels kind of load or memory use related....
    Elliot Ronaghan
    @ronawho
    Spawner part could be related. For communication between locales in the steady-state gasnet-ibv is using InfiniBand. However, to launch onto the compute nodes and exchange information prior to IB being set up gasnet has to use "out-of-band" communication. If you're using the ssh-spawner launching is done with ssh and out-of-band comm is done with sockets.
    The spawner seeing read of 0 likely just indicates the launcher on the login node can't communicate with the server anymore, which doesn't really tell us too much. I was starting to wonder if maybe we're hitting an ssh timeout or something somewhere, but just taking a stab in the dark.
    For OOMs our runtime should produce an error message and halt, I'm not sure I'd expect that to produce a SIGTERM. But if you see that error again, any additional info you could provide would be really helpful
    glitch
    @glitch
    Hi folks, just a brief note, we released Arkouda v2021.12.02 today. You can check out the release notes here: https://github.com/Bears-R-Us/arkouda/releases/tag/v2021.12.02 Cheers!
    Louis Jenkins
    @LouisJenkinsCS
    2021-12-07:11:25:33 [arkouda_server] main Line 266 INFO [Chapel] >>> "readAllHdf" "True 1 0 False False [\"srcIP\"] | []"
/localdisk/ljenkin4/arkouda//src/GenSymIO.chpl:521: error: halt reached - array index out of bounds
note: index was 0 but array bounds are 0..-1
    glitch
    @glitch
    Hi @LouisJenkinsCS can you give us some details about what was going on when you ran into this error?
    • What version of Arkouda are you running on the server
    • What version of HDF5 is installed
    • What is the internal data structure of the files you were trying to read? (i.e. where they generated by Arkouda or generated somewhere else?)
      Thanks!
    5 replies
    Michael Merrill
    @mhmerrill
    We will be discussing multi-dim array views today on the Arkouda weekly call.
    Hope @bmcdonald3 and @glitch can make the call today along with anyone else who would like to be in on the conversation!
    pierce314159
    @pierce314159
    Join Zoom Meeting https://us04web.zoom.us/j/77717000423?pwd=TGlmaUN3L2hScFovTy9NRXNnUTE5dz09
    Louis Jenkins
    @LouisJenkinsCS
    @glitch I responded in thread form btw
    glitch
    @glitch
    @LouisJenkinsCS gotcha, thanks!
    Michael Merrill
    @mhmerrill
    ATTENTION: We are no longer generating Arkouda Documentation on the Read-The-Docs site, we are only generating documentation onto GitHub Pages
    glitch
    @glitch
    Here's a direct link to the docs: https://bears-r-us.github.io/arkouda/
    You can also find the link in the README.md :thumbsup:
    Louis Jenkins
    @LouisJenkinsCS
    https://gitter.im/chapel-lang/chapel?at=61ba97dde002586c752aa448
    Michael Merrill
    @mhmerrill
    We are NOT having an Arkouda Weekly Call today
    Happy Holidays!
    Michael Merrill
    @mhmerrill
    sorry I missed having the weekly call today
    Michael Merrill
    @mhmerrill
    Today on the Arkouda Weekly Call we will discuss the code that handles binary operators and its structure. We will also take a minute to discuss the next PRs we are going to merge.
    Michael Merrill
    @mhmerrill
    I am cancelling todays Arkouda Weekly Call
    glitch
    @glitch
    got it
    Zhihui Du
    @zhihuidu
    @glitch The BFS pull request code has been updated based on your suggestions. If you have time, please have a look and let us know your comments. If you have time, we may have a zoom discussion about it. Thanks!
    Michael Merrill
    @mhmerrill
    Today's Arkouda weekly call @bmcdonald3 will talk about his PR #1034 to refactor the binary operators
    Bears-R-Us/arkouda#1034
    Michael Merrill
    @mhmerrill
    I am cancelling todays Arkouda Weekly Call
    Michael Merrill
    @mhmerrill
    I am cancelling today's Arkouda Weekly Call ... again... unless someone wants a meeting?
    Zhihui Du
    @zhihuidu

    For the latest Arkouda version. I have the following error

    /home/z/zd4/Mike/arkouda//dep/checkRE2.chpl:3: In function 'main':
    /home/z/zd4/Mike/arkouda//dep/checkRE2.chpl:9: error: Cannot find module or enum 'Regex'
    make: * [check-re2] Error 1

    I have installed regex using conda.
    How to fix this checking problem?

    pierce314159
    @pierce314159
    you need to set the CHPL_RE2 env variable and rebuild chapel
    export CHPL_RE2=bundled
    Zhihui Du
    @zhihuidu
    Thanks!
    login-1-84 chapel >: export CHPL_RE2=bundled
    login-1-85 chapel >: make
    ....
    make[3]: Nothing to be done for `re2'.
    .....
    It seems I missed something else?
    I need to
    make clean
    first?
    Must I build it from clean? If I rebuild the chapel, it will need a long time.
    pierce314159
    @pierce314159
    if arkouda is still failing to make, then try make clean and then make
    Brad Chamberlain
    @bradcray
    I would expect you not to have to re-build if re2 is already built.
    What version of Chapel are you using? (chpl --version)
    Zhihui Du
    @zhihuidu
    chpl version 1.24.0 pre-release (8ca0d00)
    I have make clean and make