Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Henry Schreiner
    @henryiii
    The @ line must be right before a function
    Skagamaster
    @Skagamaster
    I've run other things in VS Code, so I know it at least functions.
    Henry Schreiner
    @henryiii
    It’s a decorator, it only applies to functions.
    Well, classes and functions, but not normal lines of code. Delete that line and it should work.
    Skagamaster
    @Skagamaster
    OK, that must be the issue; I'll have to research more how that works. I'm looping over a lot of iterations and want to run them all at the same time.
    That did it!
    Henry Schreiner
    @henryiii
    Always get it to work 1 at a time first, then you can make it faster. :)
    Skagamaster
    @Skagamaster
    That's sound reasoning.
    Is it easy to modify a loop to run in parallel once you have it working? I'd like to code it with correct format from the start if possible. Also, Python makes it a lot easier to put data in an array!
    Henry Schreiner
    @henryiii
    Fairly easy if you do things with simple data types and arrays and fairly basic operations. If you can do things an “array at a time”, that’s also a good way to go.
    Skagamaster
    @Skagamaster
    OK, I'll proceed in that manner. Thanks again! This had me going in circles.
    Henry Schreiner
    @henryiii
    a = np.array([1,2,3,4,5])
    b = np.array([1,2,3,4,5])
    
    # Method 1: Array at a time
    c = a + b
    
    # Method 2: Numba vectorize
    @numba.vectorize
    def vsum(a, b):
        return a + b # a and b are just single values!
    
    c = vsum(a, b)
    
    # Method 3: Manual parallel loop
    @numba.njit(parallel=True)
    def jsum(a, b):
        c = np.empty_like(a)
        for i in numba.prange(len(a)):
            c[i] = a[i] + b[i]
    Code may not be quite correct.
    Skagamaster
    @Skagamaster
    I had read that you simply change "range" to "prange" once the @numba.njit bit was set.
    Example I found online:
    from numba import njit, prange
    import numpy as np
    
    @njit(parallel=True)
    def two_d_array_reduction_prod(n):
        shp = (13, 17)
        result1 = 2 * np.ones(shp, np.int_)
        tmp = 2 * np.ones_like(result1)
    
        for i in prange(n):
            result1 *= tmp
    
        return result1
    Henry Schreiner
    @henryiii
    That’s what I wrote above (third example)
    Except I forgot to return c
    Skagamaster
    @Skagamaster
    Does it matter that you had "numba.prange" as opposed to simply "prange"?
    On a completely unrelated note, I'm trying to plot with errors and then fit that plot to its average value with the error from the fit. I have an array with many rows (one for each data point) and six columns. I'd like to plot using that array, but subsets. For example, I would like to plot the value in the 5th column (its error is in the 6th) for only values of {1,3,20} in the 2nd, 3rd, and 4th columns and I would like the x-axis to be the values in the 1st column (to be clear; that would technically be the 0th column and all others, technically, one less).
    Skagamaster
    @Skagamaster

    This is for energy values for a detector that had multiple tiles, each tile is in a sector and there are two sectors (East and West). It is for runs over multiple days, so each row corresponds to a tile in the form:
    [Day, East/West, Sector, Tile, Energy, Error]
    Just to make it more clear what I was saying above. My goal is to get a graph for a particular tile over all the range of days, with error bars showing, and then to get an average value from that tile (again, with the associated error). I'm having trouble getting started; the standard code of:

    import matplotlib.pyplot as plt
    
    plt.errorbar(x,y,xerr,yerr)

    I'm not able to use well; I don't really know what should go in where (I've tried a few things without much success). Anyone perhaps have a link to a relevant tutorial or some other such?

    Doug Davis
    @douglasdavis
    @Skagamaster prange is a function in thenumba module. numba.prange(...) is required if you only import the numba module. If you use from numba import prange (which you do) then you have the prange(...) function available without having to tell python that you want it from the numba module (since you already imported it)
    For fitting (x,y) data I'd like a look at the scipy.optimize module fitting functions
    They have quite a few good examples
    The documentation for scipy.optimize.curve_fit shows an example with some plotting
    Henry Schreiner
    @henryiii
    @Skagamaster You might find the materials for the course I taught last year useful: https://github.com/henryiii/compclass
    I assume you are using pandas; pandas has lots of flexibility for setting up the data for this.
    Skagamaster
    @Skagamaster
    Thank you! I need all the instruction I can find.
    Eduardo Rodrigues
    @eduardo-rodrigues

    Dear colleague,

    We are pleased to announce the second "Python in HEP" workshop organised by the HEP Software Foundation (HSF). The PyHEP, "Python in HEP", workshops aim to provide an environment to discuss and promote the usage of Python in the HEP community at large.
    PyHEP 2019 will be held in Abingdon, near Oxford, United Kingdom, from 16-18 October 2019.

    The workshop will be a forum for the participants and the community at large to discuss developments of Python packages and tools, exchange experiences, and steer where the community needs and wants to go. There will be ample time for discussion.

    The agenda will be composed of plenary sessions, a highlight of which is the following:
    1) A keynote presentation from the Data Science domain.
    2) A topical session on histogramming including a talk and a hands-on tutorial.
    3) Lightning talks from participants.
    4) Presentations following up from topics discussed at PyHEP 2018.

    We encourage community members to propose presentations on any topic (email: pyhep2019-organisation@cern.ch). We are particularly interested in new(-ish) packages of broad relevance.

    The agenda will be made available on the workshop indico page (https://indico.cern.ch/event/833895/) in due time. It is also linked from the PyHEP WG homepage http://hepsoftwarefoundation.org/activities/pyhep.html.

    Registration will open very soon, and we will provide detailed travel and accommodation information at that time.
    Travel funds may be available at a modest level. To be confirmed once registration opens.

    You are encouraged to register to the PyHEP WG Gitter channel (https://gitter.im/HSF/PyHEP) and/or to the HSF forum (https://groups.google.com/forum/#!forum/hsf-forum) to receive further information concerning the organisation of the workshop.

    Looking forward to your participation!
    Eduardo Rodrigues & Ben Krikler, for the organising committee

    Skagamaster
    @Skagamaster
    Making some progress with Python, and I have a little error that's not affecting anything but my sanity. My line of code is:
    '''python
    plt.hist2d(nMIP, refMult, bins=[150, 50], cmap=plt.cm.jet)
    '''
    The code runs just fine, so that's nice. But when I save it, I get the error:
    "Module 'matplotlib.cm' has no 'jet' memberpylint(no-member)"
    I think the issue is just in how VSCode checks for issues, but it's annoying as it could mask any real issues (as that *.py file is always flagged as having an issue). Any ideas on where to start for solutions? Thanks!
    Shoot; I meant:
    plt.hist2d(nMIP, refMult, bins=[150, 50], cmap=plt.cm.jet)
    Henry Schreiner
    @henryiii
    plt.hist2d(nMIP, refMult, bins=[150, 50], cmap=plt.cm.get_cmap("jet"))
    That should quiet the linter. The colormaps aren’t actually normal python objects in the file, and that’s tripping up the linter.
    Skagamaster
    @Skagamaster
    Thank you!
    Skagamaster
    @Skagamaster

    All right, next problem (sorry if I'm overly bugging you all): histogram fitting. I can do this fairly easily on ROOT, but I'm having a lot of trouble on Python. For whatever reason, I can't seem to find a tutorial that includes this. All the fitting tutorials give errors when I try to fit a 2D histogram. I'm making the histogram as I did in the above post:

    plt.hist2d(mipVref[0], mipVref[1], bins=[150, 50],
               cmap=plt.cm.get_cmap("afmhot"))

    I've tried curve_fit and Model, but to no avail. Any pointers to a specific method to fit 2D histos? Thanks!

    Doug Davis
    @douglasdavis
    Check out the returns of hist2d (here); you get (h, xedges, yedges, image); you can define your function f(x, y) and make the points h the dependent variable in your fit using curve_fit (don't forget to convert xedges and yedges to the center of the bins); here's an example of a 2D gaussian fit.
    Skagamaster
    @Skagamaster

    I checked out a few things, and am hitting a snag when it comes to getting the function portion down. Here's my relevant code (much of which I adapted from others on Stack Exchange):

    H, xedges, yedges = np.histogram2d(mipVref[0], mipVref[1], bins=[100, 100])
    
    
    def centers(edges):
        return edges[:-1] + np.diff(edges[:2])/2
    
    
    xcenters = centers(xedges)
    ycenters = centers(yedges)
    
    pdf = interp2d(xcenters, ycenters, H)
    
    plt.pcolor(xedges, yedges, pdf(xedges, yedges), cmap=plt.cm.get_cmap("hot"))

    The issue is the plot is almost an inverse relationship to the actual plot. It should look like this:
    image1
    But it looks like this:
    image2

    Skagamaster
    @Skagamaster
    alt
    That worked! OK, so that's the "incorrect" image. Here's the "correct" one:
    alt
    Skagamaster
    @Skagamaster

    Sorry for the long post!
    New question: I have a working model using Keras with Tensorflow background. My goal is to get the final function coded into ROOT as that's what we would need for the metric we're making. So, how do I make sense of the weights? Here's the relevant code:

    model = Sequential()
    model.add(Dense(16, input_dim=16, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit(data[:, 1:], data[:, 0], epochs=30, batch_size=50)

    I'm trying it with only one relu layer with 2 neurons in order to get a feel for how it all works, and it returns the following array sizes:
    (16, 16), (1, 16), (16, 1), (1, 1)

    Just from dimensional analysis (I'm using 16 inputs to get 1 output), I would think that the 16 inputs will combine with the (16, 16) array to give a (1, 16) array which adds the (16, 1) array elements and then the last (16, 1) gives a single element to use in the sigmoid (with the (1, 1) value being the additional term). Taking weights times inputs as w.x, and added to their modifiers. So this would be:
    w1.x+b1 = x1, for all b1 indexed
    w1 -> (16, 16), b1 -> (1, 16)
    1/(1+exp(-w2.x1+b2))
    w2 -> (16, 1), b2 -> (1, 1)

    Do I understand this correctly? I'm trying to hand-reproduce the final predictions from the model so I can run them in ROOT.

    To clarify: the array sizes returned there are for the code I shared. Please ignore the "I'm trying it with only one relu layer with 2 neurons in order to get a feel for how it all works." I'll be working that to reproduce initially from a smaller data set to get a feel for the code; I should have omitted that. Sorry!
    Skagamaster
    @Skagamaster
    The above doesn't recover what the model predicts for smaller sets.
    Chris Tunnell
    @tunnell
    As the question is more about how to get things into ROOT, you might have more luck on a ROOT channel. There are ways to export your models (https://machinelearningmastery.com/save-load-keras-deep-learning-models/) but I don't remember ROOT's neural network codes being anywhere nearly as advanced as TensorFlow so you might hit issues.
    It lets you print out things about the shapes too.
    The math is sort of hard to parse in the chat, but when you just print the model description, it should tell you about the shapes at each step
    I'm sort of lost on the question though
    @Skagamaster
    Henry Schreiner
    @henryiii
    @Skagamaster that general idea should work. this is what is done here, if you would like an example: https://github.com/lwtnn/lwtnn
    There they use Eigen to do the matrix calcs to avoid ROOT dependencies. Model format is JSON.
    Skagamaster
    @Skagamaster

    @henryiii I think that's what I'm looking for; I was trying to reconstruct the mathematics of the handoffs, but I think I was missing a layer or two in the exchange. It seems it's not quite as simple as I had it (thought reLU would be like a delta plus and sigmoid like a Boltzman). I'm reading through that git now. Thanks!

    @tunnell It's not nearly as ambitious as running TensorFlow on ROOT; I'm just looking for a way to execute the prediction model that was generated. I thought it could be exported in a simple, mathematical formula (as all it's doing, in the end, is putting weights on inputs and using those for an output). I have the shape and weights printed out; basically, I'm looking to reconstruct the prediction algorithm (not further refine it or anything like that; I'm considering all training done once I get out of Python) for use in ROOT. If it's just a mathematical formula, which I should think it would have to be, it ought to be readily programmable to any language without much fuss, no?

    Raahul Singh
    @Raahul-Singh
    Hi everyone!
    I'm a sophomore computer science and engineering undergrad from the Indian Institute of Information Technology Sri City, India.
    I'm new to Open Source Contributions. Though I really wish to do my part and contribute in anyway to CERN.
    Could anyone please guide me here? 😅