ArkoudaProject/community

Louis Jenkins

@LouisJenkinsCS

Also can confirm that Chapel compiler works for simple hello.chpl example, and runs, so I don't think it has to do with CHPL_TARGET_CPU or some kind of illegal opcode.

glitch

@glitch

Hmm.. this is a bit beyond my skills. I'd start with a really stripped down build and remove the gasnet / comm settings and see if you can get that built. Run top or something to watch system resources during the build and see if you can spot something off.
For comparison my resolve time is around 125s, the lowerIterators is 3s (and pretty much everything after that is low seconds until it makeBinary)

On the surface it almost looks like you ran into swap issues towards the end and the bus error leads me to believe something is wrong with your system's memory.

Louis Jenkins

@LouisJenkinsCS

I guess other people could be using the head node. I suppose I should reserve a compute node this time and have it handle compilation.

glitch

@glitch

If there is light usage now, I'd try kicking off another build and keeping an eye on the system resource usage. Honestly I'd expect it to take 10-15 minutes.

Looking at my output stats, resolve and makeBinary are really the spots that take the majority of the time.

Louis Jenkins

@LouisJenkinsCS

I'm running it again on a compute node this time.

glitch

@glitch

So it's been over 45 minutes, did it complete yet? If not what does the memory usage look like?

Louis Jenkins

@LouisJenkinsCS

Took only 10 min

I used a compute node this time

Turns out the problem was the headnode

glitch

@glitch

:thumbsup:

Louis Jenkins

@LouisJenkinsCS

Thanks for the help!

Louis Jenkins

@LouisJenkinsCS

This message was deleted

tests/categorical_test.py .F......FFF... [ 5%]
tests/check.py ................... [ 12%]
tests/client_test.py ......... [ 15%]
tests/coargsort_test.py ...... [ 18%]
tests/compare_test.py ............... [ 23%]
tests/datetime_test.py ............. [ 28%]
tests/dtypes_tests.py ........... [ 32%]
tests/groupby_test.py .............. [ 37%]
tests/nan_test.py . [ 38%]
tests/io_test.py F.F.FF.FFF..FF.FF.sF.. [ 46%]
tests/io_util_test.py ... [ 47%]
tests/join_test.py ....... [ 50%]
tests/logger_test.py ....... [ 53%]
tests/message_test.py .... [ 54%]
tests/numeric_test.py .......F..... [ 59%]
tests/operator_tests.py ............. [ 64%]
tests/pdarray_creation_test.py .......F...F.F.F.F.. [ 71%]
tests/regex_test.py ........ [ 74%]
tests/registration_test.py ................ [ 80%]
tests/security_test.py .... [ 82%]
tests/setops_test.py ...... [ 84%]
tests/sort_test.py ....

So the "F" means failure? "s" means?

glitch

@glitch

correct, s means skipped, currently there should only be 1 skipped test

Louis Jenkins

@LouisJenkinsCS

Is each test different or are these trials?

glitch

@glitch

they are all individual, you should have gotten a listing of the specific tests that failed

Louis Jenkins

@LouisJenkinsCS

Gotcha, I'll wait until they finish then.

glitch

@glitch

The make test target is really running pytest under the hood which you can invoke manually. i.e.

python3 -m pytest tests/numeric_test.py

You can run a specific one via

python3 -m pytest tests/numeric_test.py::NumericTest::testHistogram

Passing the -s option will also write all of the output to stdout

You can see other optoins etc. in the pytest.ini configuration file

Louis Jenkins

@LouisJenkinsCS

Gotcha. I ran make test-all but right now I'm waiting on the test suite to complete so I can see precisely which tests failed; its stalling right now

I'm wondering, will it fail if it times out? Also, does it create/shutdown the server in each test? I.E. is it possible that it fails because it takes too long to start the server in each test?

glitch

@glitch

From past experience if you're running in multi-locale mode and you think it stalled, do a ps -ef|grep arkouda_server to see if anything is running. Chances are the server process died out and the client in the test is waiting to reconnect ... which is never going to happen.

Everything should pass and it looks like you have enough failures where I'd kill the make test-all process and then re-run just one of the test files manually with the -soption to see what's going on.

Louis Jenkins

@LouisJenkinsCS

Weird its hanging but... ps -ef|grep arkouda_server yields a running server

Categorial Test, which ran perfectly the first time around, hangs on the 4th test

Oh nvm its still running even after the test ended

OH NVM

I was looking at the command itself; the server did indeed die

How do I handle this issue? Is it killed due to default timeout?

glitch

@glitch

No, the server likely seg-faulted during the test. I'd start with a single test class and single test and run it with the -s option so you can see the output

Also you should probably re-compile Arkouda with the ARKOUDA_DEVELOPER flag turned on (i.e. export ARKOUDA_DEVELOPER=1)

glitch

@glitch

This should display the name of the test running, which can help narrow down to a specific test that is hanging and then we can debug from there:

python3 -m pytest tests/categorical_test.py::CategoricalTest -v

Louis Jenkins

@LouisJenkinsCS

E       AssertionError: 'type[32 chars] of (Categorical, str, str_); got int instead' != 'type[32 chars] of (arkouda.categorical.Categorical, str, num[21 chars]tead'
E       - type of argument "other" must be one of (Categorical, str, str_); got int instead
E       + type of argument "other" must be one of (arkouda.categorical.Categorical, str, numpy.str_); got int instead

Hmm

========================================================================================== FAILURES ==========================================================================================
_________________________________________________________________________________ CategoricalTest.testBinop __________________________________________________________________________________

self = <categorical_test.CategoricalTest testMethod=testBinop>

    def testBinop(self):
        cat = self._getCategorical()
        catDupe = self._getCategorical()
        catNonDupe = self._getRandomizedCategorical()


        self.assertTrue((cat._binop(catDupe,'==')).all())
        self.assertTrue((cat._binop(catNonDupe,'!=')).all())

        self.assertTrue((ak.array([True,True,True,True,True,True,True,
                                   True,True,True]) == cat._binop(catDupe,'==')).all())

        self.assertTrue((ak.array([False,False,False,False,False,False,
                                   False,False,False,False]) == cat._binop(catDupe,'!=')).all())

        self.assertTrue((ak.array([True,False,False,False,False,False,
                                   False,False,False,False]) ==
                                   cat._binop('string 1', '==')).all())
        self.assertTrue((ak.array([True,False,False,False,False,False,
                                   False,False,False,False]) ==
                                   cat._binop(np.str_('string 1'), '==')).all())

        self.assertTrue((ak.array([False,True,True,True,True,True,True,True,True,True]) ==
                   cat._binop('string 1', '!=')).all())
        self.assertTrue((ak.array([False,True,True,True,True,True,True,True,True,True]) ==
                   cat._binop(np.str_('string 1'), '!=')).all())

        with self.assertRaises(NotImplementedError):
            cat._binop('string 1', '===')

        with self.assertRaises(TypeError) as cm:
            cat._binop(1, '==')
>       self.assertEqual(('type of argument "other" must be one of (Categorical, str, str_);' +
                          ' got int instead'),
                         cm.exception.args[0])
E       AssertionError: 'type[32 chars] of (Categorical, str, str_); got int instead' != 'type[32 chars] of (arkouda.categorical.Categorical, str, num[21 chars]tead'
E       - type of argument "other" must be one of (Categorical, str, str_); got int instead
E       + type of argument "other" must be one of (arkouda.categorical.Categorical, str, numpy.str_); got int instead
E       ?                                          ++++++++++++++++++++                  ++++++

tests/categorical_test.py:129: AssertionError
================================================================================== short test summary info ===================================================================================

7c180623e234acc691e04947c5a5faf3b0812faf

Hash is above

glitch

@glitch

What's your version of numpy?

Louis Jenkins

@LouisJenkinsCS

Name: numpy
Version: 1.20.1
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /home/users/p02405/anaconda3/lib/python3.8/site-packages
Requires: 
Required-by: tifffile, tables, statsmodels, seaborn, scipy, scikit-learn, scikit-image, PyWavelets, pyerfa, patsy, pandas, numexpr, numba, mkl-random, mkl-fft, matplotlib, imageio, hdflow, h5py, Bottleneck, bokeh, bkcharts, astropy

glitch

@glitch

Ah, ok, we're not compatible with 1.20.x yet (Bears-R-Us/arkouda#670), you'll need to downgrade to the 1.19.x line

Louis Jenkins

@LouisJenkinsCS

glitch

@glitch

https://github.com/Bears-R-Us/arkouda/blob/7c180623e234acc691e04947c5a5faf3b0812faf/setup.py#L134

I'd recommend using something like conda or python's venv to manage the development environment. It makes it much easier to control and test versioning of various dependencies.

I know this isn't ideal, but on the bright side, I think we have a reasonable idea of why the tests are failing on your build.

Louis Jenkins

@LouisJenkinsCS

I'll try to take another crack at it

Louis Jenkins

@LouisJenkinsCS

out.txt

See above for current error when trying to run the test

Where communities thrive

People

Repo info

Activity