HSF/PyHEP

Discussion of Python in High Energy Physics https://hepsoftwarefoundation.org/activities/pyhep.html

Eduardo Rodrigues

@eduardo-rodrigues

Morning @mayou36. It's still good to have it accepted as a poster. The package may have been perceived as more specific? Probably. Congrats too!

Henry Schreiner

@henryiii

Hi everyone, I’d like to introduce @LovelyBuggies (Nino), who has been selected as our Google Summer of Code student for this summer, working on Hist! (cross posting from Scikit-HEP)

N!no

@LovelyBuggies

Nice to meet all of you!

Chris Tunnell

@tunnell

howdy

Eduardo Rodrigues

@eduardo-rodrigues

PyHEP 2020 Workshop, July 13-17 2020 - registration is open!

Dear colleague,

The PyHEP 2020 workshop will be a virtual workshop given the worldwide conditions related to the COVID-19 pandemic. The dates have been adapted and the workshop now runs the week of July 13‒17; it still follows SciPy 2020 (July 6‒12, also virtual).

Registration for the PyHEP 2020 virtual workshop is open, see https://indico.cern.ch/e/PyHEP2020. There are no registration fees.

The agenda is shaping up with tutorials and standard talks. Topics under discussion include data access and manipulation (Uproot/Awkward and related), distributed computing, auto-differentiation, histogramming and fitting, among others. We welcome submissions of abstracts for live tutorials and shorter Jupyter-notebook talks, both of which are intended to target the strengths of live, online communication. Details on how "notebook talks" will be organised will be provided in due time. It is highly likely that we will collect all notebooks together and will promote 1‒2 notebooks to a standard presentation as well.

More details can be found on the Indico page https://indico.cern.ch/e/PyHEP2020 or from the PyHEP WG homepage http://hepsoftwarefoundation.org/activities/pyhep.html.
You can also join the PyHEP WG Gitter channel (https://gitter.im/HSF/PyHEP) and/or the HSF forum (https://groups.google.com/forum/#!forum/hsf-forum) to get more information about the workshop and community.

We are directly reachable via pyhep2020-organisation@cern.ch.

Looking forward to your participation!

Organising Committee
Eduardo Rodrigues - University of Liverpool (Chair)
Ben Krikler - University of Bristol (Co-chair)
Jim Pivarski - Princeton University (Co-chair)
Chris Tunnell - Rice University
Matthew Feickert - University of Illinois at Urbana-Champaign
Peter Onyisi - The University of Texas at Austin

Lana

@Lana-B

Hi all, I've have been nudged in the way of this channel to ask my python fitting question. I want to fit ~10^5 histograms, which aren't correlated, with gaussians. I have been using scipy.curve_fit in a loop which is slow and I'd prefer to find a method that uses matrix operations instead, minimising per histogram. I haven't been able to find an appropriate python package so far. Does anyone have a suggestion?

Jonas Eschle

@mayou36

Hi @Lana-B if I understand correctly, you can just let them simultaneously fit and use the normal "curve_fit", just create a loss with the sum of all the individual terms.

But I would maybe recommend you to use a more specific tool, pyhf I think would be perfectly well suited for this (you just fit in 10^5 channels?), otherwise zfit, which is not specialized on binned fits but allows to define flexible loss functions would maybe be a better suit (as they also provide more advanced statistics tools that you _may_need; in the end, they're written especially having HEP in mind).

Hans Dembinski

@HDembinski

Hi @Lana-B , if you still have the original data before the binning: the maximum-likelihood estimators for mu, sigma for a gaussian can be calculated analytically. Those are just the arithmetic mean and the standard deviation

This fixes the shape of Gaussians, now you only need to compute the amplitude, and that you could to with a linear fit that uses matrix operations.

It is not well known, but scipy can fit all standard distributions to unbinned data and will use analytical results when available, e.g. for norm it should as simple as norm.fit(data).
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html#scipy.stats.norm

Linear least-squares fit in scipy:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.lsq_linear.html#scipy.optimize.lsq_linear

Hans Dembinski

@HDembinski

I think in this case the amplitude of the histogram is simply np.sum(norm(mu, sigma).pdf(x))

where x is your unbinned data

Hans Dembinski

@HDembinski

Nevermind the last formula, it is not correct

Adrian Thompson

@athompson-tamu

Hello! Are there any PyHEP talks planned for writing performance python code or writing wrappers to C++ extensions?

Eduardo Rodrigues

@eduardo-rodrigues

Hi. Please use the workshop mailing list pyhep2020-organisation@cern.ch as it's a bit premature for the organisers to disclose just now what is presently being discussed, I mean the contents. This being said, the topic is totally of relevance so if you are considering sending an abstract you should not feel shy ;-).

Adrian Thompson

@athompson-tamu

@eduardo-rodrigues thank you! I am still too inexperienced to give a decent talk on the subject, but if I become an expert between now and July I will certainly consider it! :D

Eduardo Rodrigues

@eduardo-rodrigues

No worries. Welcome to this nice community!
I take note of the interest :-).

Henry Schreiner

@henryiii

@athompson-tamu The three minicouses listed here might be of interest to you: https://iscinumpy.gitlab.io (Under “My books and workshops", near the top)

Adrian Thompson

@athompson-tamu

@henryiii thanks!! these look great.

Lana

@Lana-B

Thanks for your advice @mayou36 and @HDembinski ! I am looking to fit pixel values, to find their noise and dark values, but with sporadic gamma hits through giving higher values. This is pretty simple with curve_fit and it works well but perhaps the covariance matrix will be too large if I tried to create a loss function for all the terms. The higher valued hits through the pixels pull norm.fit's mean higher and the sigma too wide. I will have a look at the other options suggested, such as zfit. Cheers

Eduardo Rodrigues

@eduardo-rodrigues

This may be of interest to many of you:
Python for HPC workshop at the upcoming Scientific Computing conference this year, see https://www.pyhpc.io/ (thanks to Ilektra-Athanasia Christidi for the info)

Eduardo Rodrigues

@eduardo-rodrigues

Hi all, reminder that registration for the PyHEP 2020 virtual (and free) workshop (https://indico.cern.ch/e/PyHEP2020) closes tomorrow Tuesday 30th.

Hans Dembinski

@HDembinski

853 participants, that's crazy

Good job advertising @eduardo-rodrigues et al :)

Eduardo Rodrigues

@eduardo-rodrigues

Hi. Yep, crazy ... but great :-)! Thanks.

geophysics91

@geophysics91

Good morning.. I would like to minimize the test_function. basically, test_function() takes 3 variables - m, n and p. The goal is to find such values of these 3 variables that the function returns the minimal possible value.I am using nelder-mead minimization problem. while running my script for x0 = [25.0, 45.0, 10.0] i am getting error like Maximum number of function evaluations has been exceeded. Anyway I followed the stackoverflow link to write my code. https://stackoverflow.com/questions/55751317/minimize-multivariable-function please help me on this.Thanks.My data and script is attached here https://i.fluffy.cc/HLR1jCJLLV8lX4NfjSqbjRKG6DsB4bwS.html stackoverflow link is blocked in my area..please help

Adrian Oeftiger

@aoeftiger

853 participants, that's crazy

slight increase from last year's PyHEP :O

alexander-held

@alexander-held

will the PyHEP talks be recorded?

Henry Schreiner

@henryiii

SciPy talks have gone live, including my boost-histogram one! https://youtu.be/ERraTfHkPd0

I still hate what they did to my audio. The others don’t sound that bad.

Matthew Feickert

@matthewfeickert

will the PyHEP talks be recorded?

@alexander-held Yes, the PyHEP talks will be recorded and then after the fact be uploaded with captions. I'm still working on the logistics of all of this, but there will be some public record after the fact of all talks (if not, then I've messed up and all complaints should go to me).

@henryiii Even with the audio business (which I agree is bad) you talk is so good! :D

For reference, Jim's talk: https://www.youtube.com/watch?v=WlnUF3LRBj4

and my talk: https://www.youtube.com/watch?v=FrH9s3eB6fU (my audio sounds awful at the start, but it gets better once I switch to slides)

Henry Schreiner

@henryiii

@jpivarski's audio was just fine - don’t know what they had against me and maybe a few others… Jim’s talk is fantastic, by the way, finally watched it fully last night. The “copy” live demo was effective. Not sure I’m going to get though all the talks (or even half the talks) before the session today. I wish we had more than 1-2 days with the videos. ML only had a bit of Sunday and part of Monday - I hadn’t watched any of the talks by the time the session started.

HotPopRobot: Space, Robotics, Machine Learning, AI

@wonrobot_twitter

Saw the slides shared by Matthew Feickert at #SciPy2020 Conference today about the PyHEP event Is it still possible to register?

@matthewfeickert Thanks for sharing the slides on #SciPy2020 Lightning Talks today. I was one of the speakers. Came to know about the PyHEP through your slides and looks very interesting. Would it be still possible to register? Thanks - Artash.

5 replies

Chris Burr

@chrisburr

@matthewfeickert Have you considered live streaming the entirety of PyHEP to youtube? I haven't tried myself but I've heard it's quite easy to set up with zoom

Eduardo Rodrigues

@eduardo-rodrigues

@chrisburr this is indeed under consideration. We will be using a special Fermilab Zoom account to be able to accommodate 1000 people and we have to see if live streaming will be available ...

Andry Rakotozafindrabe

@arakotoz

Hi @eduardo-rodrigues, my zoom saif that the maximum of 300 participants is reached (not 1000 participants), and that it is not possible to join anymore

Jonas Eschle

@mayou36

Same for me. It's a technical difficulty and currently being fixed

Andry Rakotozafindrabe

@arakotoz

Thanks for the info

Matthieu Marinangeli

@marinang

This should also be lived here https://www.youtube.com/watch?v=9_6XHHFexqA

but doesn't work :(

Eduardo Rodrigues

@eduardo-rodrigues

The limitation at 300 has been sorted out quickly. For some weird reason the settings were not as we thought the were!

Note that we never officially announced that live streaming was up and running. It isn't, at least for now.

Henry Schreiner

@henryiii

Does anyone know a good way to automate checking for print statements? Either a nice automated check in pre-commit or a way to have pytest error out if some output is generated (ideally by default / for most functions, as functions really shouldn’t be writing to stdout/stderr)

Henry Schreiner

@henryiii

flake8-print works!

Where communities thrive

People

Repo info

Activity