I am sorry. I've been reading this. This library provides I/O:
https://stacresearch.com/system/files/resource/files/STAC-Summit-1-Nov-2017-Intel-Totoni.pdf
But this only would work in intel processors, is that right?
can anyone give a tip about this?
That didn't answer the question :) However, HPAT, which is what that link is talking about is now Intel SDC https://github.com/IntelPython/sdc, and I don't recall seeing anything that'd prevent it being compiled for any CPU, perhaps just try building it?
Fundamentally, it's just calling out to some C code to do the IO, have you considered just doing this in your case?
Numba has pretty good ctypes support http://numba.pydata.org/numba-doc/latest/reference/pysupported.html#ctypes
If it doesn't need to be cross platform, and you are just reading a line of a file that is sometimes updated, you can probably just fopen() fread() fclose() it on linux.
but, there is a control signal of the hardware I am using. This control signal is in a file that cannot be mmaped,so i have to open as a regular file. This takes to much time. 10-100 times more.
if you can express the read you want to do in terms of
fopen, fread, fclose you can probably just do it with ctypes
GLOBAL_VAR_NAME = 'mypackage.mygv'
result_type = MyType()
def caching_wrapper(arg):
candidate = _load_global(result_type, GLOBAL_VAR_NAME) # returns Optional(result_type)
if candidate is None:
result = expensive_function(arg)
_store_global(result, GLOBAL_VAR_NAME)
else:
result = _cast(candidate, result_type)
return result
where
_load_global and _store_global load and store from an ir.GlobalVariable with linkage 'linkonce'. Will the global variable be shared between different jitted functions that call caching wrapper?
In an unbox function I have:
pyaddr = c.pyapi.object_getattr_string(obj, "_wrapper_address")
...If obj does not have the required attribute, pyaddr will contain NULL and an exception should be raised.
How to accomplish that within the unbox function? I found guard_null but I was not able to get it working..
grep for
NativeValue.*is_error in the code base for example
you can use
cgutis.is_null() to check if pyaddr is null
The
guard_null assume a different calling convention then the code in @unbox
This is what I currently have:
@unbox(FunctionType)
def unbox_function_type(typ, obj, c):
print(f'UNBOX_function_type({typ}, {obj})')
fnty = lower_nbtype(typ)
# Assume obj is CFunc
pyaddr = c.pyapi.object_getattr_string(obj, "_wrapper_address")
# TODO: pyaddr == NULL, e.g. when obj is pure Python function
ptr = c.pyapi.long_as_voidptr(pyaddr)
# TODO: decref pyaddr?
fptr = c.builder.bitcast(ptr, fnty.as_pointer())
return NativeValue(fptr, is_error=c.pyapi.c_api_error())
you’ll need to have a if-else branch when there’s an error
where
_load_globaland_store_globalload and store from anir.GlobalVariablewith linkage 'linkonce'. Will the global variable be shared between different jitted functions that call caching wrapper?
@asodeur, linkonce_odr linkage should merge the symbols. LLVM linkage is quite tricky and I usually rely on trial-and-error
Do we expect
python -m numba.runtests numba.cuda.tests.cudadrvto work? As a whole, the testsuite works for me (running numba.runtests only) but with the aforementioned invocation I reliably get:test_broadcast (numba.cuda.tests.cudadrv.test_cuda_array_slicing.CudaArraySetting) ... /home/nfs/gmarkall/numbadev/numba/numba/cuda/compiler.py:272: RuntimeWarning: divide by zero encountered in long_scalars
blkct = (self.ntasks + tpbm1) // tpb
Fatal Python error: Segmentation fault
Current thread 0x00007fdc68cb1700 (most recent call first):
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/cudadrv/driver.py", line 293 in safe_cuda_api_call
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/cudadrv/driver.py", line 1622 in launch_kernel
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/cudadrv/driver.py", line 1578 in __call__
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/compiler.py", line 611 in _kernel_call
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/compiler.py", line 537 in __call__
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/compiler.py", line 275 in __call__
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/cudadrv/devicearray.py", line 581 in _do_setitem
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/cudadrv/devicearray.py", line 530 in __setitem__
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/cudadrv/devices.py", line 225 in _require_cuda_context
File "/home/nfs/gmarkall/numbadev/numba/numba/cuda/tests/cudadrv/test_cuda_array_slicing.py", line 209 in test_broadcast
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/unittest/case.py", line 628 in run
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/unittest/case.py", line 676 in __call__
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/unittest/suite.py", line 122 in run
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/unittest/suite.py", line 84 in __call__
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/unittest/runner.py", line 176 in run
File "/home/nfs/gmarkall/numbadev/numba/numba/testing/main.py", line 123 in run
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/unittest/main.py", line 271 in runTests
File "/home/nfs/gmarkall/numbadev/numba/numba/testing/main.py", line 354 in run_tests_real
File "/home/nfs/gmarkall/numbadev/numba/numba/testing/main.py", line 369 in runTests
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/unittest/main.py", line 101 in __init__
File "/home/nfs/gmarkall/numbadev/numba/numba/testing/main.py", line 163 in __init__
File "/home/nfs/gmarkall/numbadev/numba/numba/testing/__init__.py", line 75 in run_tests
File "/home/nfs/gmarkall/numbadev/numba/numba/_runtests.py", line 28 in _main
File "/home/nfs/gmarkall/numbadev/numba/numba/runtests.py", line 9 in <module>
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/runpy.py", line 85 in _run_code
File "/home/nfs/gmarkall/miniconda3/envs/numbaenv/lib/python3.7/runpy.py", line 193 in _run_module_as_main
Segmentation fault
It's a bit of a heisenbug... What I can determine so far is that the call to determine the threads per block (
cuOccupancyMaxPotentialBlockSize?) is somehow returning 0
then the kernel gets configured with 0 threads, which the runtime doesn't like
wonder if its due to you having newer hardware?
I see:
`
numba/cuda/tests/cudadrv/test_linker.py", line 93, in test_set_registers_57
self.assertEquals(57, compiled._func.get().attrs.regs)
AssertionError: 57 != 56
hmm, will poke around some more
adding a check:
diff --git a/numba/cuda/compiler.py b/numba/cuda/compiler.py
index d242e11..0b3ea5f 100644
--- a/numba/cuda/compiler.py
+++ b/numba/cuda/compiler.py
@@ -268,6 +268,8 @@ class ForAll(object):
kernel = self.kernel
tpb = self._compute_thread_per_block(kernel)
+ if tpb == 0:
+ raise RuntimeError("AWWW")
tpbm1 = tpb - 1
blkct = (self.ntasks + tpbm1) // tpb
does get me through the test suite... didn't think of that before, but knowing it works for you led me down the right direction
also, I then get:
test_set_registers_57 (numba.cuda.tests.cudadrv.test_linker.TestLinker) ... ok
what hardware do you have, @stuartarchibald ?
I wonder if it's related to the fact that some tests are
CUDATestCase instances, which calls numba.cuda.reset() in tearDown()
I'm not sure, I'd have to go digging in the code :)
Going to open an issue to keep track of this. There seems to be various ways to hit this, and I suspect given that slightly different conditions make different tests fail, a valgrind run would be informative but I need to find a relatively low-pain way to do that and get far enough to trigger the bug