I've an array with shape (4, 2, 12)
and ive got a function
def cos(v1 : Vec, v2 : Vec) -> float:
return jnp.sum(v1 * v2) / (jnp.sqrt(jnp.sum(v1**2)) * (jnp.sqrt(jnp.sum(v2**2))) + 0.000001)for each (2, 12) of the 4 elements, id like to apply the cos function, how can i vectorize it?
e.g this is what it looks like
(4, 2, 12)
[[[0. 0. 0. 0. 0. 0. 2. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]and result should be something like:
[.9 1 1 1]
so like behaviiour of
[cos2(x) for x in arr] where cos2 = def cos2(v : Vec) -> float:
print(v.shape)
v1 = v[0]
v2 = v[1]
return jnp.sum(v1 * v2) / (jnp.sqrt(jnp.sum(v1**2)) * (jnp.sqrt(jnp.sum(v2**2))) + 0.000001)
why is this?
ie depending on the index for the last row in the array, the array will either add up to 0.0 or it will add up with a diff of around 1e-14
numba's sum function does the same thing. Kahan summation solved the issue
Hello folks, I am looking for options to make my unit package even more numpy-compatible. I already use
and
__array_ufunc__and
__array_function__ interfaces (which are really great) but I have seen many ideas here and there (extensible dtypes, duckarray, context-local and global override API and more). My question is which of those solutions is implemented and available for testing ?
Hey all :),
I would really appreciate your help on optimizing the following code:
_cells = np.copy(self._cells)
for pcells, pstate in self._patterns:
height, width = pcells.shape
for cy, cx in np.ndindex(_cells.shape):
selection = _cells[cy : cy + height, cx : cx + width]
if np.array_equal(selection, pcells):
for sy, sx in np.ndindex(selection.shape):
if selection[sy, sx] == Cell.Alive:
self._cells[cy + sy, cx + sx] = pstateWhat it basically does is that it marks certain patterns (of different size) within a big matrix. I wonder if there is a more purely numpy way of doing this? Any ideas how to optimize this? Because this is currently very computational heavy :/
Thank you very much in advance <3
Best,
Felix
e.g. include dummy data of the same size/shape to test the algorithm
Hello, I'm hoping someone can point me towards reference material regarding the underlying BLAS's threading capabilities. Specifically I'd like to understand this timing difference (numpy 1.22.3 installed from PyPI, so OpenBLAS in this case):
In [1]: from threadpoolctl import ThreadpoolController
...: import numpy as np
...: import time
...:
...: controller = ThreadpoolController()
...: a = np.random.random((500, 321, 321))
...: b = np.random.random((500, 321))
In [2]: st = time.perf_counter()
...: _ = np.linalg.solve(a, b)
...: ed = time.perf_counter()
...: print(ed - st)
5.766736300000002
In [3]: st = time.perf_counter()
...: with controller.limit(limits=1, user_api='blas'):
...: _ = np.linalg.solve(a, b)
...: ed = time.perf_counter()
...: print(ed - st)
0.5034510000000019
... and what operating system
Hey all, I'm trying to pip install the requirements.txt specified here, but I'm running into an issue. When pip gets to the point where scipy-1.4.1 is being installed, the build fails with an error:
Collecting scipy==1.4.1
Using cached scipy-1.4.1.tar.gz (24.6 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [60 lines of output]
...
Original error was: /tmp/pip-build-env-z8oxljl3/overlay/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so: undefined symbol: cblas_sgemmThe error message indicated that there were many ways in which this issue could arise, so here's some more info for context:
- I'm using
pyenvto manage python versions. I'm running this on python version 3.9.12, and this error happens with a totally fresh install. - This is being run on arch linux, with
uname -rms:Linux 5.17.8-arch1-1 x86_64 - I do currently have multiple versions of python (system version; 3.9.12; and 3.10.2). I've set
pyenv global 3.9.12, however, and I've confirmed that I canpip install scipywithout issue. - If it matters, I'm running gcc 12.1.0
Anyone have any advice?
I'm processing analog signals from high-bitrate audio files into digital data, and am reaching RAM limits. (40 minutes of 192ksamples/s represented as a complex float each are almost 4GB, I can fit that in RAM once but not multiple times).
I'm looking to do things more efficiently along two approaches, for none of which I found support in numpy so far:
I'm looking to do things more efficiently along two approaches, for none of which I found support in numpy so far:
- Can I do things like convolution (I'm low-pass filtering the signal after multiplying in a complex carrier) in-place? (
numpy.convolvehad no hints) - Is there a format in which I can program sequentially, but it'll stash out data into memory maps, or even evaluate lazily? (In the image space I think that GEGL might do that for me). The idea is that I'd write
but the objects are either backed by file storage (effectively similar to swapping them out, but some of the data is already on disk so that wouldn't need to go through swap space, and accessors might be a bit more wary of preferring sequential access) or even creating a chain of lazily performed operations that get run over windows of the original data when accessed / saved.data = load_data(...) data = data[:,1] dc_part = gaussian(data, sigma) rectified = abs(data - dc_part) ... save_dat(...)
What I do right now is to do in-place what can be done in-place, and (around the big costly low-pass filtering of the IQ multiplied signal) del everything and reload after I've dropped the input data to the filtering, but that's a rather tedious manual process
sorry, @peytondmurray ^^^^
@chrysn:matrix.org one technique is to use chunking and process the data in smaller pieces. You need to be careful about edge effects.
yeah, chunking would create edges for my gaussians. i'd need to work concurrently on several chunks, and that's something i'd rather do with an existing library than doing it locally.
i might need to do my gaussians manually (as scipy's ndimage.gaussian_filter1d appears to create some numpy arrays internally), but that shouldn't be an issue -- there seems to be a path toward .convolve() there
It seems that in NumPy that it is more natural to right-multiply matrices vs left-multiply, e.g., x.T @ A.T vs A @ x. In particular, when vectorizing linear algebra operations, there is often the need to reshape or add dimensions. Row vectors seem to be a more natural fit with the assumptions in the NumPy API than column vectors for the types of operations. E.g., adding dimensions pre-pends rather than post-pends. For easy array reshaping and slicing, and speed consideration, and for interoperability with other libraries, should row vector representation be preferred?
[array([1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2], dtype=int32)
array([1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 2, 2], dtype=int32)
array([1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 0], dtype=int32)
array([0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 2, 2, 0, 2, 0], dtype=int32)
array([0, 1, 0, 1, 1, 0, 0, 0, 0, 2, 0, 0, 2, 2, 0, 0], dtype=int32)
array([0, 0, 1, 1, 0, 0, 1, 2, 0, 0, 0, 2, 2, 0, 0, 0], dtype=int32)
array([0, 0, 0, 1, 1, 0, 0, 0, 1, 2, 2, 0, 2, 0, 0, 0], dtype=int32)
array([0, 0, 0, 1, 2, 1, 1, 2, 0, 0, 0, 2, 0, 0, 0, 0], dtype=int32)
array([0, 0, 0, 0, 1, 0, 0, 2, 1, 2, 0, 1, 2, 0, 0, 0], dtype=int32)
array([0, 0, 0, 1, 2, 0, 1, 2, 1, 0, 2, 0, 0, 0, 0, 0], dtype=int32)]
@bigyankarki: where are you getting this list from? You probably should modify that so that you generate the correct shape to begin with. But, what do you mean by "it didn't work"? Did you get an error?
Hello - I was wondering if anyone had experience with calling a Python module using numpy from C++. Specifically - I am trying to do some simple task parallelism from C++ using std::thread and while (I'd like to believe) my coupling works appropriately when I do not use numpy - I seem to have a deadlock issue when calling any numpy function (such as np.sum(x)) etc.
I'd be happy to send across a minimum working example as an attachment
Thanks in advance!
Hi, guys. I want to know the meaning of
NPY_NO_EXPORT, which is defined by #define NPY_NO_EXPORT NPY_VISIBILITY_HIDDEN. I saw a lot of usage of NPY_NO_EXPORT in definition of various c functions and python types, e.g., NPY_NO_EXPORT PyTypeObject PyArray_Type = {. Any explaination is good for me! Thanks. I'd guess this MACRO means the defined funcs or types are only used in NumPy C world, and not in NumPy Python wrold?
NPY_NO_EXPORT means the functions are internal to NumPy and are not available for general use. The functions marked with NUMPY_API are exported via a mechanism that overrides the NPY_NO_EXPORT macro at build time. All the non-static functions should be marked with NPY_NO_EXPORT, if there are any public ones they must be expressly declared as such and become part of the NumPy C-API.
Hi, guys. I want to compile NumPy extensions with flags
-O0 -g to disable compiler optimization and enable debugging info. Where should I modify in the setup.py process? NumPy building process and the distutitls package is kind of overwhelming for me... Emmm... Or any document about this is helpful.
1 reply
Hi, guys. I found an old NEP about
Deferred UFunc evalulation using C++ expression template, https://numpy.org/neps/nep-0011-deferred-ufunc-evaluation.html. But its status is deferred, I wonder if numpy's already implemented lazy ufunc evalulation using another way? I searched in the numpy source code, I feel numpy doesn't impl this feature.
I feel it's a key difference about ufunc & loop fusion between Numba and NumPy.