Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    Louis Jenkins
    I'd recommend using something like conda or python's venv to manage the development environment. It makes it much easier to control and test versioning of various dependencies.
    I know this isn't ideal, but on the bright side, I think we have a reasonable idea of why the tests are failing on your build.
    Louis Jenkins
    I'll try to take another crack at it
    Louis Jenkins
    See above for current error when trying to run the test
    Name: numpy
    Version: 1.19.3
    Summary: NumPy is the fundamental package for array computing with Python.
    Home-page: https://www.numpy.org
    Author: Travis E. Oliphant et al.
    Author-email: None
    License: BSD
    Location: /home/users/p02405/.local/lib/python3.8/site-packages
    Required-by: tifffile, tables, statsmodels, seaborn, scipy, scikit-learn, scikit-image, PyWavelets, pyerfa, patsy, pandas, numexpr, numba, mkl-random, mkl-fft, matplotlib, imageio, hdflow, h5py, Bottleneck, bokeh, bkcharts, astropy
    __________________________ CategoricalTest.testBinop ___________________________
    self = <categorical_test.CategoricalTest testMethod=testBinop>
        def testBinop(self):
            cat = self._getCategorical()
            catDupe = self._getCategorical()
            catNonDupe = self._getRandomizedCategorical()
                                       True,True,True]) == cat._binop(catDupe,'==')).all())
                                       False,False,False,False]) == cat._binop(catDupe,'!=')).all())
                                       False,False,False,False]) ==
                                       cat._binop('string 1', '==')).all())
                                       False,False,False,False]) ==
                                       cat._binop(np.str_('string 1'), '==')).all())
            self.assertTrue((ak.array([False,True,True,True,True,True,True,True,True,True]) ==
                       cat._binop('string 1', '!=')).all())
            self.assertTrue((ak.array([False,True,True,True,True,True,True,True,True,True]) ==
                       cat._binop(np.str_('string 1'), '!=')).all())
            with self.assertRaises(NotImplementedError):
                cat._binop('string 1', '===')
            with self.assertRaises(TypeError) as cm:
                cat._binop(1, '==')
    >       self.assertEqual(('type of argument "other" must be one of (Categorical, str, str_);' +
                              ' got int instead'),
    E       AssertionError: 'type[32 chars] of (Categorical, str, str_); got int instead' != 'type[32 chars] of (arkouda.categorical.Categorical, str, num[21 chars]tead'
    E       - type of argument "other" must be one of (Categorical, str, str_); got int instead
    E       + type of argument "other" must be one of (arkouda.categorical.Categorical, str, numpy.str_); got int instead
    E       ?                                          ++++++++++++++++++++                  ++++++
    tests/categorical_test.py:129: AssertionError
    This is unmodified Arkouda
    I'm also using the chapel module on Swan (Cray-XC)
    I filed an issue for this: Bears-R-Us/arkouda#955
    I think you are fine to comment out the failed assertion here until we fix it since the types are actually correct, just fully qualified versus the short name versions.
    Michael Merrill
    We will NOT be having the Arkouda Weekly Call today because of a conflicting meeting.
    ATTENTION: We will NOT be having the Arkouda Weekly Call today because of a conflicting meeting.
    Louis Jenkins
    Another error, this time in tests/categorical_test.py::CategoricalTest::testSaveAndLoadCategorical
    Does h5ls exist on the machine you're running the test on? i.e. which h5ls ?
    Louis Jenkins
    :o I don't have that
    Oh wait
    I didn't have module loaded
    Shall I try it again?
    For clarity: I am running on both a Cray-XC and an older infiniband machine
    I would expect it to work as long as h5ls is installed and accessible
    Also because it's relevant I was considering taking on this ticket Bears-R-Us/arkouda#452 which would remove the need for h5ls installation
    Louis Jenkins
    Cool, it worked!
    Louis Jenkins
    It is unfortunate that the client does not have a way of backing out when the server is unavailable in between tests. It seems to happen at random times, but client is just hanging.
    If you are running the unit tests on a multi-locale system the hangling client is generally indicative of the server dying for some reason. How many locales are you running on? (There are a few known issues with running unit tests with small amounts of data on systems with num_locales > num_data_elements and I'm wondering if that's what you're running into.
    I'd also advocate for running a specific test file / class with the -v flag on first to get an idea of what test is failing, then running it with the -s flag to capture the output.
    python3 -m pytest tests/categorical_test.py::CategoricalTest -v
    Louis Jenkins
    The default number of locales which is 2
    I'll try the -s flag
    Louis Jenkins
    Unfortunately I can't seem to drag-and-drop here... so here is a pastebin: https://pastebin.com/xvPa8ZHB
    It works fine down to testEquality
    Louis Jenkins
    This is weird, I am having so many problems running on the supercomputers, but it works fine on my laptop. Now I can't even do a pip install to build the arkouda package...
    Building wheels for collected packages: arkouda
      Building wheel for arkouda (setup.py) ... error
      ERROR: Command errored out with exit status 1:
       command: /home/users/p02405/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/lus/scratch/tmp/pip-req-build-mra5diqu/setup.py'"'"'; __file__='"'"'/lus/scratch/tmp/pip-req-build-mra5diqu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /lus/scratch/tmp/pip-wheel-2cq41ur_
           cwd: /lus/scratch/tmp/pip-req-build-mra5diqu/
      Complete output (30 lines):
      /home/users/p02405/anaconda3/lib/python3.8/site-packages/setuptools/dist.py:461: UserWarning: Normalizing 'v2021.08.20+35.g6f5616b.dirty' to '2021.8.20+35.g6f5616b.dirty'
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib
      creating build/lib/arkouda
      copying arkouda/groupbyclass.py -> build/lib/arkouda
      copying arkouda/io_util.py -> build/lib/arkouda
      copying arkouda/pdarrayclass.py -> build/lib/arkouda
      copying arkouda/timeclass.py -> build/lib/arkouda
      copying arkouda/message.py -> build/lib/arkouda
      copying arkouda/sorting.py -> build/lib/arkouda
      copying arkouda/logger.py -> build/lib/arkouda
      copying arkouda/numeric.py -> build/lib/arkouda
      copying arkouda/infoclass.py -> build/lib/arkouda
      copying arkouda/strings.py -> build/lib/arkouda
      copying arkouda/client.py -> build/lib/arkouda
      copying arkouda/security.py -> build/lib/arkouda
      copying arkouda/pdarrayIO.py -> build/lib/arkouda
      copying arkouda/pdarraysetops.py -> build/lib/arkouda
      copying arkouda/_version.py -> build/lib/arkouda
      copying arkouda/pdarraycreation.py -> build/lib/arkouda
      copying arkouda/__init__.py -> build/lib/arkouda
      copying arkouda/dtypes.py -> build/lib/arkouda
      copying arkouda/categorical.py -> build/lib/arkouda
      copying arkouda/join.py -> build/lib/arkouda
      UPDATING build/lib/dist/arkouda/_version.py
      error: [Errno 2] No such file or directory: 'build/lib/dist/arkouda/_version.py'
      ERROR: Failed building wheel for arkouda
    This one is on a Cray-XC

    Are you pip installing from pypi or are you building your own client package to install (https://github.com/Bears-R-Us/arkouda#install-ak).

    You shouldn't install from pypi as we have not updated that in quite some time and it will not work with the current server.

    Louis Jenkins
    :o I see, I got it now, thanks
    Brett Eiffert

    hi all - new to this project. i have just been playing with arkouda for the past month or so.

    i am having an issue with using a system where 256 threads are available (2 x 64 core processors with hyperthreading) and am only getting 128ish arkouda threads. how is the number of arkouda threads spun up decided for a given system? it seems there is a default value that applies to smaller machines but not this machine i am trying. Is there a max number of threads? Is there a flag to specify how many threads to spin up when starting the server? Thanks!

    Elliot Ronaghan
    Arkouda is backed by Chapel, and by default Chapel uses physical cores only and not hyperthreads (which generally do not help performance of HPC Applications). You can control this value manually with CHPL_RT_NUM_THREADS_PER_LOCALE, but generally speaking using logical threads will not help performance and may in fact hurt. See https://chapel-lang.org/docs/usingchapel/tasks.html#controlling-the-number-of-threads for more info.
    Brett Eiffert
    Thanks Elliot! that’s helpful
    Oliver Alvarado Rodriguez
    @mhmerrill Hey Mike! Hope all is well. Where do we find the link with the weekly Arkouda meeting presentations on GitHub? I want to reference a student to one of the first ones you gave, but I am having trouble finding it on GitHub, thanks!
    I found it
    Hi All, Mike asked me to pass on the following message:
    ATTENTION: We will NOT be having the Arkouda Weekly Call today.
    He expects things to resume as normal next week. Thanks!
    Louis Jenkins
    I think I found reason why it was hanging during those tests from before; the walltime is set to a default 300 seconds; is there a way to specify the test use a walltime for each test?
    Louis Jenkins
    Also interestingly enough, the memory limit does not seem to increase per-locale Nevermind

    I'm fairly certain the reason for hanging/failure was the server process dying and the client ZMQ socket waiting for reconnection. If that's the case the test will never actually complete and should be considered failed as soon as the server process dies.

    However, if you want to try to increase the time out you can install the pytest-timeout plugin and it enables the cmd line arg --timeout=300 where the value is in seconds. An alternative is to try and use pytest marks to set something on a specific test @pytest.mark.timeout(10, "slow", method-"thread") (from the pytest docs)

    Michael Merrill
    Does anyone have a subject they would like to discuss at today's Arkouda weekly call?
    Michael Merrill
    Sorry for the late notice but since we don't have a topic... ATTENTION: We will NOT be having the Arkouda Weekly Call today.
    kay doh
    @glitch can you let me know if you are online?