Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Laurenz Rettig
    @rettigl
    we are running a time-resolved ARPES lab
    I'm not quite sure what's wrong with the second solution, but the first one now does what I want, if I add the 'hinted' kind to the target channel. That's excellent, thank you!
    Laurenz Rettig
    @rettigl
    I would think it would be good to add examples like this to the tutorials
    Thomas A Caswell
    @tacaswell
    can you weigh in on that issue with that opinion?
    Laurenz Rettig
    @rettigl
    where should I do that?
    Thomas A Caswell
    @tacaswell
    Laurenz Rettig
    @rettigl
    Sure, I'll do that. Thanks again for your help, I think I get the hang of it more or less now. I'm off for tonight.
    Thomas A Caswell
    @tacaswell
    night!
    Till Stensitzki
    @Tillsten
    @rettigl You did study at the FU-Berlin, right?
    Laurenz Rettig
    @rettigl
    @Tillsten Yes, I did my diploma nd PhD in Martin Wolf's group before he moved to the FHI. What are you using bluesky for?
    Till Stensitzki
    @Tillsten
    I was one of your study colleges
    Not using it yet, while I coded my own labcontrol framework in python, I am thinking of switching to somehing more sophisticated for a long time
    OTH bluesky seems a little bit overengieered in some aspectes, at least in my domain of experimental setups (talking about ultrafast spectroscopy here) .
    Especially if someone is not already using a epics backend
    Laurenz Rettig
    @rettigl
    yes, I remember
    The EPICS backend is probably needed or at least helps a lot
    but I find that a good investment anyways in terms of meta data recording and archiving and decentralized data aquisition
    plus you can interface with virtually everything
    we also control a lot of things through labview interfaces that talk to epics
    but that is less flexible, off course
    Till Stensitzki
    @Tillsten
    I agree, on the other hand saving data you want is not really the problem in python. I do my hardest to fully earse labview. If the resource are there, using something like bluesky is properly the correct way to go
    Laurenz Rettig
    @rettigl
    yes, I agree to that mostly. On the other hand user-friendly interfaces are also important
    Till Stensitzki
    @Tillsten
    Fully agree, i spent definitivly too much time programming trying to be user friedly. It is a shame, that there no clear go-to package for experimental control in python. Likley since it is too easy to do on most stuff on your own. But most of the project are not active anymore, since the responsible coders left science for the industry. Therefore I am hopeful for bluesky since they put some funding into it.
    pshafer-als
    @pshafer-als

    Hi, How can I get databroker to use my custom Handler?

    I have tried the following steps so far based on the Event Model documentation:

    1. Define a class called FitsHandler
    2. Add an entry to the handler registry
    3. Create a filler that uses the updated handler registry

    When I try accessing an event from the catalog, it uses the default filler. Can I change it at runtime to use the new filler?

    I've seen the documentation about registering entry points during setup, but I'm not yet at the point of creating a package.

    Here's the code...

    from databroker import catalog, catalog_search_path
    import databroker.core
    import event_model
    from astropy.io import fits
    
    class FitsHandler:
        def __init__(self, file_path, **resource_kwargs):
            if os.path.exists(file_path) and os.path.isfile(file_path):
                self._fits_file_path = file_path
                return self
            raise OSError  # FileNotFoundError
    
        def __call__(self, **datum_kwargs):
            hdulist = fits.open(self._fits_file_path)
    
            # Image data is generally the last entry in the FITS file
            ccd_image_data = hdulist[-1]
    
            return ccd_image_data
    
    handler_registry = databroker.core.discover_handlers()
    handler_registry["FITS"] = FitsHandler
    filler = event_model.Filler(handler_registry)
    
    runs = catalog["My_Catalog"]
    print(list(runs))
    run = runs[-1]  # How to use custom handler / filler here?
    >>> databroker.__version__
    '1.2.0'

    Thanks for reading this!

    Dan Allan
    @danielballan

    The dict returned by discover_handlers() is just a copy; mutating it won't have any effect on other parts of the program. (I should change that return an immutable DictView....) Modifying the handler registry at run time is fragile because of some caching that happens internally. Can you add:

    handler_registry:
         FITS: "your_module.FitsHandler"

    to the configuration file that is producing catalog["My_Catalog"]?

    pshafer-als
    @pshafer-als

    @danielballan Thanks! I did not know about this option. A small hurdle is that my module is not installed...and not really structured to be a proper package. I suppose pip install -e . could be used temporarily though.

    In the interim, Dylan has pointed me to a FITS handler that is ready to be installed. I will give that a try next.

    Thomas A Caswell
    @tacaswell
    it is a bit meta-programmy, but you can do
    import types
    import sys
    my_module = types.ModuleType('my_module')
    my_module.FitsHandler = FitsHandler
    sys.modules['my_module'] = my_module
    
    from my_module import FitsHandler as _FitsHandler
    you just have to do that before you interrogate databroker / intake
    pshafer-als
    @pshafer-als

    @tacaswell Cool, thank you

    Eventually I will need to do this the right way, but always nice to know some workarounds to help bridge that gap

    Thomas A Caswell
    @tacaswell
    yeah, that is the sort of thing that if I saw in a code-review I would be like "....no", but for development it may be useful
    pshafer-als
    @pshafer-als

    UPDATE: Databroker is now using the FITS handler that Dylan shared with me

    Hooray!

    I ended up needing to rewrite the stored data (ummm, 'datum's?) with "frame_per_point" passed as a keyword argument when creating the resource. Otherwise extract_shape() would raise RuntimeError(f"Could not figure out shape of {key}") for key="ccd", even though the shape is explicitly set in the event descriptor.

    This is the relevant section where the documents are being generated from existing data...

    ...
    descriptor_keys['ccd']['shape'][0] = sample_fits['Images'].data.shape[0]
    descriptor_keys['ccd']['shape'][1] = sample_fits['Images'].data.shape[1]
    
    primary_stream_bundle = run_bundle.compose_descriptor(
        data_keys=descriptor_keys,
        name=primary_stream_bundle_name)
    yield 'descriptor', primary_stream_bundle.descriptor_doc
    ...

    It looks like extract_shape() might not be expecting the simple case of descriptor['data_keys'][key]['shape'] already defined correctly; so it raises an exception instead of simply returning return descriptor['data_keys'][key]['shape']?

    This is probably just my ignorance of databroker in general (and extract_shape() in particular), but I'm curious whether you see this as an issue in the logic of extract_shape()?

    Dan Allan
    @danielballan
    Oh yikes. If the shape is correct we should be getting out of your way.
    pshafer-als
    @pshafer-als

    I finally managed to get debug information on the recursion condition. It appears that the recursion loop is caused by intake forcing a reload of information every time it checks the property/attribute of an Entry/Catalog.

    It turns out that the inifite recursion I was experiencing (see my post from Jan. 11) was related to an outdated (and beta) version of databroker / its dependencies. Upgrading to databroker 1.2.0 fixed this problem. Thanks, Dylan!

    It seems strange that conda pulled databroker 1.0.0b9 on 2021-01-05, when issuing the following commands.

    conda create --name fits-env python=3.8 pandas astropy requests event-model suitcase-mongo -c lightsource2-tag
    conda activate fits-env
    conda install conda-build
    conda install -c nsls2forge databroker
    conda install scikit-image

    Just to clarify, databroker now works as expected. I'm posting this simply for informational purposes.

    Kartik Cating-Subramanian
    @ksubrama_gitlab
    hi! Is this the right channel for yaq questions? If there is a better channel I want to go there before I poke Blaise.
    Thomas A Caswell
    @tacaswell
    You will need to raise @untzag or @ksunden_gitlab
    Kartik Cating-Subramanian
    @ksubrama_gitlab
    Thank you for @- ing them for me :). I'm trying to run through the yaq "let's just run fake hardware tutorial" and am running into issues that I suspect step from spaces in user names and passwordless users..
    I'm trying to locally develop a driver that I wrote inside C:\Users\my_wanky_user_name\repo\my_yaq_repo and I am trying to enable it inside yaq - so if any of the yaq developers are here, it would be awesome if you could help out/let me bounce ideas off of you.
    Kyle Sunden
    @ksunden_gitlab

    HI @ksubrama_gitlab

    Yes, spaces in paths was a problem that we found and fixed on master of yaqd-control (needed to be quoted in a subprocess call):
    https://gitlab.com/yaq/yaqd-control/-/blob/master/CHANGELOG.md

    I thought we had done a release since then, but apparently not (I think we have one other thing that we were planning on putting in the release, that just kept getting bumped down the priority list)

    Anyway, I pinged you on our matrix channel if you want to talk more.

    Kartik Cating-Subramanian
    @ksubrama_gitlab
    Uhoh - I'm not using a gitter client with notifications and just check this channel manually in the morning - so I probably missed your ping?
    @ksunden_gitlab Can you pull me into this other channel.. I'm still trying to figure out how to sensibly enable a daemon that I am actively developing without having to pip install it. I tried using yaqd edit-config with some port that was between 20k and 30k and but it didn't seem to work :(
    Kyle Sunden
    @ksunden:matrix.org
    [m]
    (Hey, gitter rooms can now be accessed via element, the "first party" matrix client, which does have notifications... I just connected in)

    anywho https://matrix.to/#/#yaq:matrix.org?via=matrix.org

    you came to this chatroom in the past, with username selfification

    Kartik Cating-Subramanian
    @ksubrama_gitlab
    Yep I did - but 2020 being 2020, I can barely remember how to do things people explained to me last week :) 2021 is following in that tradition it seeems.
    Kyle Sunden
    @ksunden_gitlab
    perhaps to add to the confusion, I think when you last were in here, the client was called "Riot" and they have rebranded to "element" (same people behind it, just a new name)
    Kartik Cating-Subramanian
    @ksubrama_gitlab
    Anyway - so how does one actively develop a yaqd-daemon. I managed to slowly follow through yaqd-control and it seems like yaqd has a couple of places it will read configs from (it'll pick up toml files from the parent directory of... wherever it can find a daemon that's pip installed or from AppData/Local/something?/yaq/config.toml?
    But I can't seem to figure out how, besides just hoping a yaq module is already installed in your python path, yaqd-control actually picks up the code for a yaq driver. I want to be able to iterate on developing a driver without having to pip install it every time.
    Kyle Sunden
    @ksunden:matrix.org
    [m]

    To get the entry points they do need to be installed. yaqd-control is more for production machines than dev (though it can be useful in some dev circumstances)

    for dev work I tend to just run the entry point directly in a terminal e.g. $ yaqd-horiba-micro-hr -v (-v to get verbose outputs)

    that prints logs to stderr so I can see what is going on.

    you can install in editable mode to be able to make changes without having to re pip install every time

    to install in editable mode with flit (which is what we mostly use) is flit install --pth-file (or on systems like linux/mac that have symlinks flit install -s)
    the default config file location (which uses appdirs to get) is in that AppData folder. yaqd edit-config horiba-micro-hr will open up notepad (or actually whatever is in $EDITOR) with that file directly