HSF/PyHEP

Discussion of Python in High Energy Physics https://hepsoftwarefoundation.org/activities/pyhep.html

Eduardo Rodrigues

@eduardo-rodrigues

All fine with the examples above.
hepunits package: I had given from hepunits import units as u as an example in the README, as I doubt anyone will use u. in general. But I'm also fine with ... as hu.

Eduardo Rodrigues

@eduardo-rodrigues

Hi @/all, a little announcement:
Software Carpentry Workshop to teach fundamental research computing skills at CERN from 27-29 Nov, 2019. The indico page to register is here: https://indico.cern.ch/event/834411/.
It is jointly organized by the Hep Software Foundation (HSF, https://hepsoftwarefoundation.org/) and IRIS-HEP (https://iris-hep.org/).
The organisers are looking for volunteers to act as helpers, saying that "you do not need to be a super expert (remembering difficulties you have had will help you relate to the students). If you are a comfortable intermediate user that will be enough."
Like teaching and have some time? Sign-up for 1/2/3 days via the following doodle https://doodle.com/poll/7n9dpbhhmyk9hcm2. I just did.
Thanks.

Jim Pivarski

@jpivarski

Not every library needs a two-letter abbreviation. I think that hepunits loses usefulness if its usage involves weird u.MeV type things. I think there's an argument for

from hepunits import MeV, GeV, mm, cm

0.510 * MeV   # electron mass
125.2 * GeV   # Higgs mass
0.456 * mm    # B0 ctau
2.685 * cm    # K0 ctau

since these objects exist primarily for readability.

By the way, Awkward's official two-letter abbreviation will be ak:

import awkward1 as ak

as in "ak! ak! this array is so awkward!". I've seen some

import awkward as awk

in the wild and I want to avoid mental overlap with Awk (which is a rather nice language for its domain, but still).

Eduardo Rodrigues

@eduardo-rodrigues

Jim, I've been using hepunits exactly as you describe above but reckon u.MeV is clear too. Clearly the above works just fine for small things but bigger files may want to be safe(r) with u.MeV and alike. Anyway, largely a matter of taste.

Agreed that ... as awk is not a good option ...

Jim Pivarski

@jpivarski

These are about recommendations. I don't think I'd recommend

import hepunits as u

which is an extra step (as u) for the sake of having a name around that's more likely to get clobbered than MeV or GeV. For instance, there could easily be a step in a calculation like

u, v  = math.cos(theta), math.sin(theta)

that silently clobbers the u, but

MeV = ...   # ?

would not be likely at all. It's about the names humans are likely to use and notice as being different. I wouldn't recommend

from hepunits import *

though. A given script is probably only going to use a couple of units, and pulling them into the namespace deliberately is not an abuse.

Eduardo Rodrigues

@eduardo-rodrigues

For sure from hepunits import * is to be banned. No doubt. I explicitly import exclusively what's needed.

Jonas Eschle

@mayou36

MeV and GeV may be not a problem, but mm, cm? I agree it looks nicer, but for larger files having a nice abbreviation that can be used would be indeed helpful.

Henry Schreiner

@henryiii

I’m working on setting up something nice with PyForest, and (depending on the final implementation), it needs clear names. Making all unit abreviations would be a no-go.

benkrikler

@benkrikler

That package scares me a bit! Seems very "unpythonic" in terms of "explicit is better than implicit"

Henry Schreiner

@henryiii

It’s only for interactive work - it’s fantastic with IPython. It saves most of the typing of quick interactive work, and when you are done and you want to make a script out of what you’ve done, you can just call active_imports() and get the list of imports you need to paste into your script. :)

Jonas Eschle

@mayou36

It has the from pylab import * touch, which is now, for good reasons, depreceated. For IPython, I have a startup script full of import package as pkg with suppressed ImporError warnings, because yeah, it's otherwise pain.

It’s only for interactive work

I see the use case, and it's nice. But I also see the abuse cases and prefer e.g. template python modules combined with an "optimize import" in my IDE that removes all unnecessary imports. No magic, just clean...
(or better, what I actually do: an editor that imports modules like np, pd with a single shortcut when I use them)

Henry Schreiner

@henryiii

For IPython, I have a startup script full of import package as pkg

This is exactly what it can and should be used for - and your start up script slows down your IPython startup time, even if you don’t use all the packages. If you copy your start up script to ~/.pyforest/user_imports.py, you will have instant startup times again, and things get imported when you use them. And you gain the ability to quickly see what you have used. I don’t use the from pyforst import * at all, it’s just an extention to IPython. I rather which that wasn’t there, for misusage issues.

Jonas Eschle

@mayou36

...copy your start up script to ~/.pyforest/user_imports.py

I was in fact not aware of that, thanks! That sounds really useful then (also because this can't be abused in a script).

Henry Schreiner

@henryiii

I haven’t been using it very long (and user_imports was just added), but it’s great to quickly use bh, plt, and np without having to type them every time I start an IPython session. When testing things like reprs I only write about 3 lines of IPython, which can become 5-6 lines with imports.

Luke Kreczko

@kreczko

@henryiii What happened to https://bitbucket.org/henryiii/pygdml/src btw?
We are currently discussion a pythonic GDML implementation to do comparisons and came across your package

there are some other packages that try to solve this problem (describing geometry in python, exporting/importing GDML), but nothing that is published on PyPi

Henry Schreiner

@henryiii

It became good enough to export my GDML datafile from blender so I could use it in Geant, then I forgot about it, I believe.

I used blender to convert a topographic map into a mesh that I then converted to GDML so it would work in Geant.

It also produces C++ code, which is the one I think I ended up using.

Luke Kreczko

@kreczko

Ah, did not notice the C++code

wait, it produces C++ code - I need to have another look

anyway, such a package would be nice for GDML import/export, (Blender, SketchUp etc) and validation.
Would you be interested to put this on Github for further development or is it better to start from scratch?

Henry Schreiner

@henryiii

I think there are some useful parts, it would probably be better than starting from scratch. I don’t remember all the details, though. Yes, I can move it to github.

Matthieu Marinangeli

@marinang

Hi all, the first release of scikit-stats, part of the scikit-hep toolset, has just been published.

It contains 2 submodules:

modeling: with the Bayesian Blocks algorithm that moved from scikit-hep to scikit-stats.
hypotest: aims to provide tools do likelihood-based hypothesis tests such as discovery test, computations of upper limits or confidences intervals. Currently discovery test
using asymptotic formulae is available. More functionalities will be added in the future.

Quick documentation is available in the README and notebook examples in binder , a proper documentation will be added in future releases.

Suggestions are welcome, and feel free to give it a try.

Henry Schreiner

@henryiii

As a community, we need to decide on our support for Python versions, especially Python 2. The maintainers of Matplotlib, scikit-learn, IPython, Jupyter, yt, SciPy, NumPy, and scikit-image have come together and agreed on a plan for Python and Numpy version support. I hope to discuss this plan within Scikit-HEP so we can come to a recommendation in the near future. For now, here is the community policy those packages have adopted: Numpy NEP 29

Eduardo Rodrigues

@eduardo-rodrigues

It makes complete sense to start a discussion soon-ish, for some planning. But we cannot afford to be as aggressive as some, because HEP still relies on Python 2 for so much, unfortunately. Otherwise we will loose users. There will be a compromise for everything that relies on numpy & co. As ever, there's no trivial solution. On the positive side, we can be more forward-looking at stick to Python 3 for high-level analysis tools, those which are likely to be loosely connected with experiments software stacks.
(I will stop here as otherwise we will have the whole discussion just now LOL.)

Henry Schreiner

@henryiii

Otherwise we will loose users.

I would argue not many or maybe any. If we add a new package, regardless of the requirements, we can’t loose a user. If we add a new release of a package that is Python 3 only, existing users will still get the old Python 2 version, and again, won’t be lost. If experiments are using our code, they would fall into this category. The only ones I think we would risk loosing would be a new users, who are Python 2 only and want to start using our code - but a) I don’t think we have that many, b) they will still be able use older versions, and c) they already have to be using older versions of numpy, SciPy, matplotlib, and IPython, so our packages would just be one more thing.

Keep in mind, the cost of keeping Python 2 compatibility is non-zero. It requires more complex code, limits use of time saving features, adds extra checks, increases binary build time, etc. It can also hamper the package API and user experience in Python 3. We could use the time we spend fiddling with Python 2 code or writing code in a way to be compatible with both instead developing new libraries and features.

Obviously, the final decision for any package has done on a per-case basis by the core maintainers of that package. I still support Python 2.6 in Plumbum, for example.

(For those not in the mailing list, I'm being a bit more aggressive here than in the plan I've suggested)

But we cannot afford to be as aggressive as some

I don't think we can afford to be less aggressive than numpy - I don't think the HEP community should take over maintaining numpy. Similarly with Python - Over time, the old versions will stop working with anything new, or on new systems, etc. I already had to drop Pandas from Particle/DecayLanguage, because they are missing some variants of Python 2 wheels and will not produce them - their hands are washed clean of Python 2 already.

Eduardo Rodrigues

@eduardo-rodrigues

I'm not saying (1) that I plan to support Py2.7 for long and (2) that HEP will even for a second maintain numpy. I think we pretty much agree on most points, actually, as you should know. Just saying we cannot say now "well, after Christmas all Python 2.7 support is gone. That's it.". We should rather do things smoothly having users in mind, balancing the requirements it imposes on us maintainers.

Henry Schreiner

@henryiii

Agreed, we should be using python_requires in our setups, so that pip 9+ will automatically find the last supported Python 2 version, etc. The idea is not to cause Python 2 to stop working, but to stop producing new features for Python 2 (more or less). It will be very interesting after January, though, to see how long things hold together...

Eduardo Rodrigues

@eduardo-rodrigues

:+1:

Henry Schreiner

@henryiii

My talk from PyHEP on Python 3.8 is now a post: https://iscinumpy.gitlab.io/post/python-38/

Doug Davis

@douglasdavis

Thanks! Small correction, cp38 wheels for NumPy have been up for about a week (started with 1.17.3)

Henry Schreiner

@henryiii

The original talk was written before that. :) Thanks, will update. GitHub Actions is supposed to be 2-3 days away from supporting Python 3.8 now, too

I didn't know the two snakes in the Python logo make up a P and a Y.

Matthew Feickert

@matthewfeickert

Yup. Everyone seems happy on that end: https://github.com/actions/setup-python/issues/30#issuecomment-546974397

Do they? If so that seems kinda forced. :/

Hans Dembinski

@HDembinski

I am looking at the PyHEP survey, really nice to have such feedback and quite useful for planning the next workshop.

It reduces the amount of guessing what people want/need.

I think we needed to give one more minute to the lightning talks and have fewer of them.

I disagree with this, I think the lightning talks were just fine and we should have had more of them.

For me the lightning talks are not only about the content, but they also give everyone an opportunity to introduce themselves

Hans Dembinski

@HDembinski

Deconstructing existing monoliths: focus on small tools with specific purpose instead

Yes :D

DON'T USE PYTHON 2!

Lol

this field has evolved from enthusiast working in isolation 2 real interactivity in just a year or two. It's amazing how quickly that happened.

Luke Kreczko

@kreczko

the last one is what made a real difference though - e.g. I've seen many plotting tools rise and fall in isolation. Now we can rally around a few focused projects and stop wasting time ;)

Matthew Feickert

@matthewfeickert

agreed. It is nice to have projects that we can all be developers and contributors to (even if I just do some drive-by-developing PRs every few months)

Henry Schreiner

@henryiii

I was happy to see how many mentioned histogramming :)

Where communities thrive

People

Repo info

Activity