Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Mario Roy
@marioroy
Blessings and grace to all. NUMBA is incredibly fun. I witnessed set_parallel_chunksize working wonderfully using the tbb layer. Yeah!
Mario Roy
@marioroy
This message was deleted
Mario Roy
@marioroy
I moved the reply to thread.
Dyuman Aditya
@DyumanAditya

Hi, I'm trying to initialize a numba.typed.Dict() in the @lower_builtin decorated constructor but am not able to do so because it's saying labels is not an iterable:

@lower_builtin(World, types.ListType(types.string))
def impl_world(context, builder, sig, args):
    typ = sig.return_type
    world = cgutils.create_struct_proxy(typ)(context, builder)
    labels = args[0]
    world.labels = labels
    for l in numba.typed.List(labels):
        world.world[l] = 0

    return world._getvalue()

How can I make labels iterable and thereby initialize world.world in this constructor?

14 replies
nelson2005
@nelson2005
The documentation states that the exception constructor argument must be a compile-time constant
5 replies
Dyuman Aditya
@DyumanAditya
@sklam would you be able to provide any insight into the @lower_builtin question above?
LI Da
@dlee992
Hi, guys. I found some code using @register_jitable(_nrt=False), I feel I know what is register_jitable, but what is _nrt=False? I only found a comment: disable runtime allocation, which means (1) the decorated function used in nopython mode doesn't need memory allocation from NRT, or (2) if the decorated function wants to allocate memory, we use another way, rather than NRT?
3 replies
LI Da
@dlee992
Wow, I tried to dump SSA and optimized ir, if _nrt=False, then SSA is dumped and no ir is dumped at all; if _nrt=True, the ir will be dumped. It seems more confusing.
11 replies
LI Da
@dlee992
Hi, guys. How can I let the below dump to stdout? I tried with setting environ["NUMBA_DEBUG_NRT"] = "1", uncomment #undef NDEBUG, and changing to #if 1 in the line below, then recompiled but not worked, only print out other NRT info, e.g., NRT_Allocate_External, not any meminfo refct info.
void NRT_MemInfo_dump(NRT_MemInfo *mi, FILE *out) {
    fprintf(out, "MemInfo %p refcount %zu\n", mi, mi->refct);
}
2 replies
Dyuman Aditya
@DyumanAditya
Hi everyone, is there a way to use the python networkx package inside a jitted function? If not is there any other way to represent a graph inside a @jit function?
6 replies
Dyuman Aditya
@DyumanAditya
Are numba typed dicts normally very slow?
10 replies
nelson2005
@nelson2005
0.56.0 release notes mention that this is the last release for python 3.7
11 replies
smorken
@smorken
hi all, wanted to check if there is a better/more robust way to map values in an array with a dictionary... or maybe if there are any obvious improvements, gotchas or mistakes in my approach.. In my case I only really care about basic int, float types
@numba.njit
import numba
from numba.typed import Dict as NumbaDict

def _map_1D_nb(a: np.ndarray, out: np.ndarray, d: dict) -> np.ndarray:

    for i in np.arange(a.shape[0]):
        out[i] = d[a[i]]


@numba.njit
def _map_2D_nb(a: np.ndarray, out: np.ndarray, d: dict) -> np.ndarray:

    for i in np.arange(a.shape[0]):
        for j in np.arange(a.shape[1]):
            out[i, j] = d[a[i, j]]


def _map(a: np.ndarray, d: dict) -> np.ndarray:
    out_value_type = numba.typeof(next(iter(d.values())))
    nb_d = NumbaDict.empty(
        key_type=numba.from_dtype(a.dtype),
        value_type=out_value_type,
    )
    for k, v in d.items():
        nb_d[k] = v
    out = np.empty_like(a, dtype=str(out_value_type))
    if a.ndim == 1:
        _map_1D_nb(a, out, nb_d)
    elif a.ndim == 2:
        _map_2D_nb(a, out, nb_d)
    else:
        raise ValueError("ndim=1 or ndim=2 supported")
    return out
1 reply
Dyuman Aditya
@DyumanAditya

Hi, I'm facing a small problem: I have extended numba for a custom class node. node has 1 attribute that is a numba TypedDict. When construct a node object inside a jitted function and I pass a typed Dict into the constructor, the object gets instantiated correctly. But then at the end of the function, when I return node, I get a segmentation fault. This is because the typed Dict seems to have gone out of scope and therefore I can no longer access the node's attribute anymore. How can I prevent the typed Dict from going out of scope? Here's a sample:

@numba.njit
def f():
        d = numba.typed.Dict.empty(key_type=label_type, value_type=interval_type)
        n = Node(d)
        return n

a = f()

I get a segmentation fault. (label_type and interval_type are classes that have been extended for numba as well)

But If I do this, I don't get a segmentation fault. Clearly the dict is going out of scope right?
@numba.njit
def f(d):
        n = Node(d)
        return n

d = numba.typed.Dict.empty(key_type=label_type, value_type=interval_type)
a = f(d)
4 replies
Christopher
@Hellgrammite00_twitter
I have been building my own @cuda.jit kernals, but been trying to figure out to get a small section of my code to work. Basically I need to pass into my kernal an array/tuple of strings, and to have the kernal extract the position of the string. So if I have (A,B,C,D), some way to have C return 2. However since I can't use dictionaries, and passing to device functions isn't working since looping through each element, and seeing if it == Str is a different type, and won't work. Enum I haven't gotten to work to well on Cuda, so maybe that is an option?
Even as something ineffecient like this won't work because of the mixed types ``` @njit(device=True)
def test(arr,string):
    y = 0
    for x in arr:
        if x ==string:
            c1 = y
            break
        y += 1                
    return c1       ```
26 replies
luk-f-a
@luk-f-a

should I be worried about this?

NumbaIRAssumptionWarning: variable 'bund_idx.8081' is not in scope.

This warning came from an internal pedantic check. Please report the warning
message and traceback, along with a minimal reproducer at:

Compilation finishes, and calculations seem fine at the end.

3 replies
luk-f-a
@luk-f-a
can numba types be recreated using their repr() string?
6 replies
Yanjie Xu
@xubenji
hello, everyone.
Michael Kummer
@randompast_twitter
I want to create a mixed array so I can print it to a csv file. ex: rows += [i, i*i, string, ...] Is there a way I can speed this up with numba?
16 replies
Nik
@Nikorasu
So I've been trying out numba on some various math functions, to familiarize myself with using it.. And everything seemed to be working great, until I tried using it to speed up a generator function that calculates digits of Pi.. For some reason, when I try it on that, instead of the intended digits of Pi, the function only outputs/yields 0s. And I'm not sure if I'm just missing something simple, or if I've encountered a bug.. (I'm on Linux, using Python 3.8.10, numba 0.56.0, numpy 1.22.4)
from numba import njit

@njit
def calcPi(n):
    k, a, b, a1, b1 = 2, 4, 1, 12, 4
    while n > 0:
        p, q, k = k * k, 2 * k + 1, k + 1
        a, b, a1, b1 = a1, b1, p * a + q * a1, p * b + q * b1
        d, d1 = a / b, a1 / b1
        while d == d1 and n > 0:
            yield int(d)
            n -= 1
            a, a1 = 10 * (a % b), 10 * (a1 % b1)
            d, d1 = a / b, a1 / b1

pi_digits = calcPi(100000)

for d in pi_digits:
    print(str(d), end='', flush=True)
10 replies
bairdlampard534
@bairdlampard534

I would like to AOT compile some jit functions, and use the functions in another jit functions on the fly. I generated a module by AOT compilation. The exported jit function signature is <class 'builtin_function_or_method'> rather than <class 'numba.core.registry.CPUDispatcher'>, so I cannot use it in another jit function.

import numba
from numba.pycc import CC
cc = CC('test')
cc.verbose = True

@cc.export("test", numba.typeof(0)(numba.typeof(0)))
def test(a):
    return a + 10

cc.compile()

import test

test.test(10) # works.

TEST = test.test

@numba.njit(numba.typeof(0)(numba.typeof(0)))
def test3(x):
    return TEST(x)

"""
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name 'TEST': Cannot determine Numba type of <class 'builtin_function_or_method'>
"""
test3(10)

is there a way to make this work? like import via ctypes or more straightforward way?

16 replies
Sumanth S Raikar
@sumanthraikar
Hi guys, I want to store few (like 7) shallow neural net models from PyTorch and use it’s predictions over a big for loop. I want to accelerate the code using GPU. Any ideas how to do it?
11 replies
Tobias Sargeant
@folded
Does anyone know much about numba + cuda + LLVM versions? it seems like cuda's libnvvm embeds some version of llvm for parsing LLVM IR, and if the version of LLVM that numba is compiled to use is too skewed from that then tests fail with compilation errors like 'unterminated attribute group'.
8 replies
Yanjie Xu
@xubenji
Hello everyone, Does somebody knows how to handle the "nan" (Not a number) in LLVMLITE?
1 reply
Jacob Schreiber
@jmschrei
Hi everyone. I'm noticing that np.argsort(x, kind='quicksort') gives a different result on the same x if x has ties on it when it's in a numba accelerated function versus outside one. I know that quicksort is unstable, but it should give the same output for the same input, right? Is quicksort implemented differently in numba than in numpy?
1 reply
Ali_Sh
@alisheikholeslam

How can I signature the following reflected list?

lst = [np.array([[1, 2],
                 [3, 4]]),
       np.array([[1, 2, 3],
                 [4, 5, 6]]),
       np.array([[1, 2],
                 [3, 4],
                 [5, 6]])]


# @nb.njit(nb.types.ListType(nb.int_[:, ::1])(nb.float64[:, :, ::1]))
@nb.njit(nb.types.List(nb.int_[:, ::1], reflected=True)(nb.float64[:, :, ::1]))
def numba_(lst):
    maxx = 0
    maxy = 0
    for x in lst:
       maxx = max(x.shape[0], maxx)
       maxy = max(x.shape[1], maxy)

    arr = np.zeros((len(lst), maxx, maxy))
    for i in range(len(lst)):
        arr[i, :lst[i].shape[0], :lst[i].shape[1]] = lst[i]
    return arr

which get error:

numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No conversion from array(float64, 3d, C) to ListType[array(int32, 2d, C)] for '$138return_value.1', defined at None
20 replies
Jordan Gunn
@githubpsyche
Should I be using anaconda's distribution of scipy rather than the one I get when I pip install scipy if I want the performance benefits from the Intel SVML when using numpy.linalg?
LI Da
@dlee992
Hi, guys. Say I have an numba extension project, and I want to re-typeof specific python type, e.g., numba has an unicodetype for python str, but I re-implment another MyStringType for str. How to do the re-typeof thing in my exntension project?
9 replies
xxauroraxx
@xxauroraxx
I recently ran into https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types
Currently, is it possible to have a collection within an @njit decorator that contains different shaped np.array elements?
10 replies
@njit
def build_net(layer_settings):
    new_net = List()
    for index, layer_size in enumerate(layer_settings):
        if index == 0:
            new_net.append(np.zeros((layer_size)))
            continue
        x = np.random.randint(low=-10, high=10, size=(layer_size, 3))
        y = np.random.rand(layer_size, 3)
        new_layer = np.add(x, y)
        new_net.append(new_layer)
    return new_net
callejon97
@callejon97

Hi guys! I am trying to enhance some linear square estimation that will use some ~26000x10 matrix multiplication. The code mainly uses numpy arrays from pandas, and some user-defined dictionaries that can be translated to lists if needed. Twoquestions:
1) How fast will numba speed up my code? At the end of each matrix value, there is a call to a C code to compute a numerical jacobian from a model. Asking in case there is something easier that I am not seeing. The numpy calls are vectorized.
2) I am running into the following error, in a call to the function that contains the main loop, even with @jit(nopython=False):

Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: global name 'length_of_iterator' is not defined

I saw numba/numba#7395, but I can't really understand the underlaying code meaning. How should I solve the error?

16 replies
Kihwang Kim
@kihwangkim

Hi guys!!
I'm trying to type-cast dataframe column to numba's types to inplace-update data of the dataframe column via numba's njit function.
It works well in a case of integer type as I intended that the data on dataframe is updated.

import pandas
from numba.core import types as numba_types
from numba.typed import Dict as numba_dict

df = pandas.DataFrame(data = {
    'col_int': [1, 2, 3],
})
kv_int = numba_dict.empty(
    key_type=numba_types.unicode_type,
    value_type=numba_types.int64[:],
)
kv_int['col_int'] = df['col_int'].to_numpy(copy=False)

print(kv_int['col_int'])
print(df)
kv_int['col_int'][1] = 11
print(kv_int['col_int'])
print(df)
[1 2 3]
   col_int
0        1
1        2
2        3
[ 1 11  3]
   col_int
0        1
1       11
2        3

However, in a case of string type, dataframe data didn't be updated.

import pandas
from numba.core import types as numba_types
from numba.typed import Dict as numba_dict

df = pandas.DataFrame(data = {
    'col_str': ['a', 'b', 'c']
})
kv_str = numba_dict.empty(
    key_type=numba_types.unicode_type,
    value_type=numba_types.UnicodeCharSeq(32)[:],
)
kv_str['col_str'] = df['col_str'].to_numpy(dtype='U32', copy=False)

print(kv_str['col_str'])
print(df)
kv_str['col_str'][1] = 'k'
print(kv_str['col_str'])
print(df)
['a' 'b' 'c']
  col_str
0       a
1       b
2       c
['a' 'k' 'c']
  col_str
0       a
1       b
2       c

Could anyone please give any idea how to do it in string case, or give any information?

Jack Miller
@NeutralKaon
Thanks for Numba's help in the past on this channel – I don't know if you have a policy about this or if it would be useful, but I have acknowledged the core developers in the resulting publication in Science Advances: https://www.science.org/doi/10.1126/sciadv.abm7935
5 replies
Ali_Sh
@alisheikholeslam
Is there any suggestion or future plan for handling reflected lists or dictionaries instead converting lists (reflected) or dictionaries to numba lists or dictionaries when using numba njit. These additional conversion related loopings will get adverse effect on the performance in such codes. These conversions make numba inefficient based on my little experiences. Thanks for any advise or opinion.
12 replies
Dyuman Aditya
@DyumanAditya
Hi, I have a custom class extended for numba that takes a numba int8 as parameter. How can I instantiate this class? Because when I pass an int to the constructor, I get an error that an int64 has been passed. How can I cast 64 to 8?
5 replies
Dyuman Aditya
@DyumanAditya
I tried defining a class member as type numba.types.float16 in the datamodel of the class, but I get a Not Implemented Error for the float16, Are float16s not supported as class members?
1 reply
stemill
@stemill
I can't get the following function to jit. What am i doing wrong? I want to end up with an array of strings
@jit(nopython=True)
def test():
    chars = np.array([[97, 98, 99, 0, 0],[99, 98, 97, 0, 0]], dtype=np.uint8)
    return chars.view(dtype='<S5').astype(str).squeeze()
27 replies
bairdlampard534
@bairdlampard534
@box(types.Record)
def box_record(typ, val, c):
    # Note we will create a copy of the record
    # This is the only safe way.
    size = ir.Constant(ir.IntType(32), val.type.pointee.count)
    ptr = c.builder.bitcast(val, ir.PointerType(ir.IntType(8)))
    return c.pyapi.recreate_record(ptr, size, typ.dtype, c.env_manager)
in the boxing code of types.Record, it mentions that the copy is the only safe way. Would there be some more explanation of this somewhere? why view cannot be returned?
Allen Xiang
@allenxiangxin

What I did wrong heree?

from numba import njit
import numpy as np

a = np.random.normal(1, 1, 10000)

@njit
def find_quantile(a):
    qx = np.quantile(a, [0.16, 0.5, 0.84])
    return qx
find_quantile(a)

The error says:
TypeError: expected dtype object, got 'numpy.dtype[float64]'

Thanks a lot

5 replies
Timothy Dijamco
@timothydijamco

Hi, is there a way to return an instance of a jitted class from a jitted function?

import numba as nb
import numpy as np

@nb.experimental.jitclass({'val': nb.float64})
class FooClass:
    def __init__(self, val):
        self.val = val

@nb.njit
def jitted_func(foo_class):
    return foo_class(3.0)

jitted_func(FooClass)

gives me

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
During: typing of argument at /tmp/ipykernel_2376236/1061734744.py (11)

File "../../../../../tmp/ipykernel_2376236/1061734744.py", line 11:
<source missing, REPL/exec in use?>

This error may have been caused by the following argument(s):
- argument 0: Cannot determine Numba type of <class 'numba.experimental.jitclass.base.JitClassType'>
7 replies
scbgm
@scbgm
Hi, I am running my python codes with Numba on AWS and have found that Numba doesn't work on machines with higher computation power. Particularly, I'm running an EC2 c6i instance and found that c6i.8xlarge(32 CPU core, 64GB Memory) is the highest-spec AWS instance that guarantees normal operations. From c6i.12xlarge(48 cores/96GB) or higher-spec instances, my code is terminated with a "Process finished with exit code -1073741571 (0xC00000FD)" message (Not all Numba functions cause this error, though.) I guess that this error is related to the memory, but I haven't found any statement for the hardware spec constraints of Numba. Any ideas? Thanks a lot!
2 replies
NicholasFrancken
@NicholasFrancken
image.png
24 replies
Hello, I am trying to run a cuda jit function which gives me the following error. I wasn't able to find much help on this online, any suggestions to what I could be doing wrong are greatly appreciated! Thanks in advance :)
NicholasFrancken
@NicholasFrancken
Fyi, I am using python 3.6.13 and Numba version 0.53.1
NicholasFrancken
@NicholasFrancken
I reproduced the error using python 3.10 and Numba version 0.55.2
Olivier Gauthé
@ogauthe
Hi! Is there a simple way to initialize an array with a shape that is obtained at run time? I get an error when I try to do
at = (np.array([1, 2]), np.array([3,4,5]))

@numba.njit
 def f(at):
    sh = [x.size for x in at]
    r = np.zeros(sh)
    return r
4 replies
NicholasFrancken
@NicholasFrancken
image.png