Scikit-HEP/community

Scikit-HEP project - general community channel

Henry Schreiner

@henryiii

This is new and likely very useful: composite actions! You can bundle actions together, just like Azure templates, but far more powerful - you also have inputs and outputs! I had a feeling they had a plan for a templates replacement when they didn’t bring templates to Actions right away, and looks like this was it.

Of course, you could do this with javascript/typescript, but this is far simpler and easier to maintain for simple jobs. Remember you can also have local actions in ./.github/actions/*/action.yml!

There are a few limitations. You have to have a single shell per step (no automatic selection based on OS, so bash or powershell core), and they have to be run steps, no actions-in-actions. But still quite useful! Actions-in-actions would have been killer, though...

Hans Dembinski

@HDembinski

I looked at it and it is not clear to me how I should use this. I certainly did not miss that feature so far, but I haven't written configs for Azure, which I generally hated (too complex for doing simple things).

Henry Schreiner

@henryiii

Here’s an example It simplifies the workflows slighly by combining two steps, but the key improvement is that now the cibuildwheel version is pinned in exactly one place instead of 5 or so places. (Note: not actually making this change, working on something different upstream in cibuildwheel, but you can see how it could be useful)

1 reply

Basically, look for places where you have to repeat the same or similar steps and now those can be pulled out into an “action” without having to write docker or javascript, but just with a pretty normal looking actions file. (Haven’t done more than basic testing yet anywhere, so don’t have it nailed down yet)

And unlike Azure templates, you can put them in a remote repository (as action.yml) then pull them in, so you can even share across all of github (like any JS / docker action).

1 reply

Henry Schreiner

@henryiii

One use I’m playing with is to bundle together the HSF training CI into a centerally managed action across 10 or more lesson repos. But no “actions in actions” really limits that. :(

Henry Schreiner

@henryiii

YES! PEP 604 is in. :) MyPy, please support it soon! Union[int, str] is now int | str, and Optional[str] is now str | None (well, functionally identical). We can pretty much ignore importing simple stuff from typing now! Also happy to see PEP 563 (postponed evaluation of annotations) in too, but actually rather surpised, it was supposed to hit in 4.0, and they kept saying 3.10 was not 4.0. Hmmm…. If you are interested in what’s in so far, visit: https://docs.python.org/3.10/whatsnew/3.10.html

1 reply

Hans Dembinski

@HDembinski

I find it difficult to be as excited about this as you are. I haven't read the PEP, so while I am sure this was discussed, I wonder at first glance about readability. Union[int, str] is more descriptive that int | str. Sometimes a bit of additional verbosity is a good thing.

5 replies

Hans Dembinski

@HDembinski

Readability is not discussed in the PEP, perhaps it was not an issue. It is good that it works also in isinstance and not only in type annotations.

1 reply

Henry Schreiner

@henryiii

GitHub is turning on automerge! I’ve seen it in a repo now. To use it, make sure you set the “required” status checks that have to pass before merging on your main/master/develop branch. Then go to settings, and enable the "Allow auto-merge" checkbox under Merge button. Now, you can tell a PR to automerge when the status checks pass, rather than having to come back to the page after they pass to do the merge! (Yes, GitLab has had this for years…)

Hans Dembinski

@HDembinski

Can you decide this for each PR individually or do you have to turn it on globally? I can see some workflows where this is useful, but just because a PR passes all tests does not mean it is done.

Eduardo Rodrigues

@eduardo-rodrigues

I agree. Indeed the most useful thing about a PR is the implicit need of it being reviewed. The fact that tests pass is in that respect a minimal requirement. Put another way, I don't think I ever merged a green PR without looking at the actual content no matter how trivial the PR was.

Henry Schreiner

@henryiii

You have to click (as far as I can tell) the button to enable - so you can’t fully automate (which is good, I think - you must review). So anyone who has the permissions to merge a PR now has a choice in the dropdown “Merge when greeen” so they can click that instead of merging before checks pass or coming back to the page later. That’s how I understand it, I haven’t seen it in action yet.

I could be wrong, in which case you’d have to also add “one required review” to get the proper, one person sees it behavior. I’ll update here when I see what it does!

Hans Dembinski

@HDembinski

Same here, I also usually do dummy messages in the commits to the PR in my own repos and then write a meaningful commit message when I squash the PR.

Ok, the feature "merge when green" is useful

Henry Schreiner

@henryiii

I’ll update as soon as I see it in action. I’m partially assuming it’s like gitlab, where you can write the squashed message, etc. then “merge when green” and head off to do something else.

Hans Dembinski

@HDembinski

Yeah, if it is like that then it is very good

Eduardo Rodrigues

@eduardo-rodrigues

Thanks for the explanations. Makes more sense.

Henry Schreiner

@henryiii

Screen Shot 2021-01-08 at 12.22.12 PM.png

So the “merge” button is replaced with “Enable auto-merge” if th status checks haven’t passed. And there’s a “admin override and merge now” checkbox above; checking it gives you back the normal merge button. You get the normal dropdown, and clicking “Enable auto-merge: squash” gives you the normal edit box for changing the commit message. What happes to the message if it fails, that I don’t know.

Once you enable it, it is replaced with a “disable auto-merge” until the checks pass and it is merged. I tried to get a screenshot, but the checks passed and it merged before I could snap it. :)

I forgot to / didn’t enable required checks for three wheel builds, so those didn’t need to complete for it to merge.

Hans Dembinski

@HDembinski

excellent, thanks for the report

on a side note, one of my students recently brought up poetry, it seems to get more and more users

Henry Schreiner

@henryiii

Did I share Rich here? I don’t see it, but it uses poetry, maybe the largest (20K stars) project I’ve seen using it. It is an amazing project, by the way (Rich).

Just look through the readme. :) https://github.com/willmcgugan/rich (and it has a protocol for making your objects Rich printable ;) )

Andrzej Novak

@andrzejnovak

agreed, I tried using it to visualize a histogram being filled as it's processed, but I got a weird flicker with more then 5-6 lines, never figured out how to fix it

Henry Schreiner

@henryiii

Peotry has a bit of a learning curve, but then it does everything, it’s pip + venv + setuptools + twine. It’s like bundle from Ruby or Yarn from Node.JS. Biggest issue; no compiled project support. But I think they are developing a plugin system - if that could be used for a CMake plugin, I’d be sold! :)

I think, also not sure how well it interacts with conda.

Andrzej Novak

@andrzejnovak

(sorry i meant rich)

Henry Schreiner

@henryiii

I guessed, we probalby should be using threads, but that’s okay. ;)

If poetry could visualize histograms being filled, that’s too all in one! ;)

Andrzej Novak

@andrzejnovak

yeah, that's my main gripe with gitter, even the threads are hard to navigate

hehe

Henry Schreiner

@henryiii

PS: You have to stick in a token setup.py file to get GitHub to recognize your repo as a Python project, at least that was true 8 months ago when Rich added one :disappointed:

Henry Schreiner

@henryiii

This is my first (real) foray into Poetry: https://github.com/henryiii/hypernewsviewer. A single config file, no MANIFEST.in, no setup.py, no setup.cfg, no requirements.txt, no setting up a venv, integrated dev requirements that automatically get installed with poetry install, since it knows you are obviously trying to develop (fantastic, why does no one else do this?). All nice.

They all produce normal wheels, since the metadata format in a wheel is defined in a PEP, so once you make a pure python package, no one downstream even knows you used something to put together the files and metadata.

Poetry doesn’t handle versions very elegantly, requires 2 places. Flit requires 1 but requires importability (but handles installing and importing more elegantly than setuptools). Neither can use git to do it. :(

Hans Dembinski

@HDembinski

You should also mention that poetry only works well with projects that are pure python

If you praise something you should also point out the caveats.

From my point of view, poetry just does everything I can already do with specialized tools, so I don't really get the hype.

Henry Schreiner

@henryiii

I thought that’s exactly what I did first?

Biggest issue; no compiled project support. But I think they are developing a plugin system - if that could be used for a CMake plugin, I’d be sold! :)

From my point of view, poetry just does everything I can already do with specialized tools

Yes, but that’s exactly it; it’s all in one, and when you are working with a project, not having to deal with venvs and setup.* hassles and all the weird caveats is really refreshing, and much easier to teach and use. “Did I remember to add this file to the SDist in my MANIFEST.in”, did I turn on find packages, did find grab by test suite by default, etc. Dealing with setuptools has gotten a lot better, but I’d not call it “fun”. I might call poetry “fun”. The seperate tools are still more powerful/configurable.

I am not yet sold on poetry, primarly due the main caveat I’ve listed above and interaction with Conda. I am finding it pretty useful for setting up quick projects though (cookie solves that for setuptools too, as well, including C++ ones ;) )

Henry Schreiner

@henryiii

Since we are mentioning caveats, if you are on Big Sur, Poetry doesn’t get binary wheels yet - so no NumPy. It’s a problem with PyPA/packaging, and the fix is tricking down now, but something to be aware of, pip releases happen a little quicker after pypa/packaging ones, it takes longer to get into Poetry. And if you are on a Apple Silicon Mac, you are stuck with pip too, it can’t grab properly named universal2 wheels, you need pypa/packaging master for that.

Henry Schreiner

@henryiii

Put another way, https://github.com/scikit-hep/cookie gives you many of the the benefits of poetry with the power of setuptools! :P (and you can pick poetry with it too if you want)

Hans Dembinski

@HDembinski

Since we are mentioning caveats, if you are on Big Sur, Poetry doesn’t get binary wheels yet - so no NumPy [...]

Good that you mention that, I was not aware.

alexander-held

@alexander-held

Hi, is anyone aware of python implementations of smoothing algorithms specifically designed for smoothing histograms with a small number (~3-30) of bins? Such algorithms are used for smoothing templates in statistical models. ROOT's TH1:Smooth is an example. scipy.signal has some algorithms that seem more suited to >>100 bins, and I have not found much else so far.

Henry Schreiner

@henryiii

Just a reminder, it is never okay to use open without with for a quick “one liner” - things can break when you try to assume the garbage collector will run, and it’s a bad example. Maybe always run python with -Wd! Blosc/python-blosc#243

Hans Dembinski

@HDembinski

Just as a reminder, statements that begin with "it is never okay" are almost never correct. Using open without with is fine if you know what you are doing.

15 replies

@alexander-held Smoothing is a bad idea in any statistical analysis. You usually don't have control over the uncertainties after a smoothing operation. Instead of smoothing, fit a low order polynomial or spline to your data. This allows you to capture the uncertainties in the data in the covariance matrix of the fitted parameters.

9 replies

Where communities thrive

People

Repo info

Activity