Public channel for discussing Numba usage. Don't post confidential info here! Consider posting questions to: https://numba.discourse.group/ !
1 reply
gmarkall (Graham Markall):
Hey Graham, I implemented a VCD parser and a class to represent a signal as you suggested (See below)
Basically a FastVCD object holds a dict of Signal objects. For my usecase, I have about 100k VCDs, each with about 30-100 signals each. I wish to iterate through all VCDs and collect information from all signals. To do this with numba, should I decorate these two classes with @jitclass ? The Signal class is pretty simple, just holds two integer arrays.
pseudocode of what I want to do :
vcds = [List of FastVCD objs]
for i, v in enumerate(vcds):
for t in range(0, 1000): # iterate through time
sum = 0
# Sum up values of all signals at time t
for s in v.signals:
sum += s[t]
arr[i, t] = sum
return arrMy current code:
from bisect import bisect_right
class Signal:
def __init__(self) -> None:
self.indices = []
self.data = []
def add(self, i, d):
self.indices.append(i)
self.data.append(d)
def __getitem__(self, i):
x = bisect_right(self.indices, i)
return self.data[x-1]
class FastVCD:
def __init__(self, vcd):
self.signals = {} # Dict of signal objects
import os
import cv2
import time
import numpy as np
from numba import jit,cuda
from google.colab import drive
drive.mount('/content/gdrive')
#@cuda.jit('f8[:,:](u1[:,:])', parallel=True, cache=True, device=True)
@cuda.jit(device=True)
def normalize_mat(depth_src):
depth_min = depth_src.min()
depth_max = depth_src.max()
depth = (depth_src - depth_min) / (depth_max - depth_min)
return depth
def generate_stereo(depth_dir, depth_prefix, out_dir, f):
filename = f.split(".")[0]
print("=== Start processing:", filename, "===")
depth_src = cv2.imread(os.path.join(depth_dir, depth_prefix + filename + ".jpg"), cv2.IMREAD_GRAYSCALE)
depth = normalize_mat(depth_src)
depth = depth * 255
cv2.imwrite(os.path.join(out_dir, "depth_" + filename + ".jpg"), depth)
def file_processing_imdepth_dir, depth_prefix, out_dir):
for f in os.listdir(depth_dir):
filename = f.split(".")[0]
generate_stereo( depth_dir, depth_prefix, out_dir, filename)
def main():
start_time = time.time()
depth_dir = 'gdrive/MyDrive/depth/'
depth_prefix = 'Depth_'
out_dir = 'gdrive/MyDrive/output/'
if not os.path.exists(out_dir):
os.mkdir(out_dir)
file_processing_im(depth_dir, depth_prefix, out_dir)
print(time.time() - start_time, "seconds for base generation")
if __name__ == "__main__":
main()
3 replies
Hello, the issue is about indexing numpy array in a jited function.
@jit(nopython=True)
def foo(index):
x = np.array([1,2,3])
y = x[index]
return y
print(foo(4))This returns some random number rather than raising an error.
This only happens with numpy arrays. With tuples or numba typed lists, an IndexError: list index out of range is raised
here's a code example
import numba
from numba import cuda
from numba.cuda.random import create_xoroshiro128p_states
from numba.cuda.random import xoroshiro128p_uniform_float64
@cuda.jit('void(float32[:,:])', device=True)
def device(rng_states):
thread_id = cuda.grid(1)
probability = xoroshiro128p_uniform_float64(rng_states, thread_id)
@cuda.jit()
def kernel(rng_states):
device(rng_states)
BPG = 10
TPB = 10
rng_states = create_xoroshiro128p_states(BPG * TPB, seed=42069)
kernelTPB, BPG
Error message:
numba.core.errors.TypingError: Failed in cuda mode pipeline (step: nopython frontend)
Internal error at <numba.core.typeinfer.CallConstraint object at 0x2ad3ee333970>.
module, class, method, function, traceback, frame, or code object was expected, got CPUDispatcher
During: resolving callee type: Function(<numba.cuda.compiler.DeviceDispatcher object at 0x2ad3ee333820>)
During: typing of call at /home/rudiejd/cse620c_finalproject/test.py (21)
Enable logging at debug level for details.
File "test.py", line 21:
def device(rng_states):
<source elided>
thread_id = cuda.grid(1)
probability = xoroshiro128p_uniform_float64(rng_states, thread_id)
^
@cuda.jit(device=True)
def device(rng_states):
thread_id = cuda.grid(1)
probability = xoroshiro128p_uniform_float64(rng_states, thread_id)Traceback (most recent call last):
File "/home/rudiejd/cse620c_finalproject/milestone3.py", line 734, in <module>
cuda_main(agent_count, graph_type, comm_algo)
File "/home/rudiejd/cse620c_finalproject/milestone3.py", line 498, in cuda_main
hm.run_parallel(BPG, TPB)
File "/home/rudiejd/cse620c_finalproject/milestone3.py", line 280, in run_parallel
kernel[BPG, TPB](
File "/home/rudiejd/.conda/envs/env/lib/python3.9/site-packages/numba/cuda/compiler.py", line 868, in __call__
return self.dispatcher.call(args, self.griddim, self.blockdim,
File "/home/rudiejd/.conda/envs/env/lib/python3.9/site-packages/numba/cuda/compiler.py", line 1003, in call
kernel.launch(args, griddim, blockdim, stream, sharedmem)
File "/home/rudiejd/.conda/envs/env/lib/python3.9/site-packages/numba/cuda/compiler.py", line 752, in launch
exccls, exc_args, loc = self.call_helper.get_exception(code)
ValueError: not enough values to unpack (expected 3, got 2)