Hi, How can I get databroker
to use my custom Handler?
I have tried the following steps so far based on the Event Model documentation:
FitsHandler
When I try accessing an event from the catalog, it uses the default filler. Can I change it at runtime to use the new filler?
I've seen the documentation about registering entry points during setup, but I'm not yet at the point of creating a package.
Here's the code...
from databroker import catalog, catalog_search_path
import databroker.core
import event_model
from astropy.io import fits
class FitsHandler:
def __init__(self, file_path, **resource_kwargs):
if os.path.exists(file_path) and os.path.isfile(file_path):
self._fits_file_path = file_path
return self
raise OSError # FileNotFoundError
def __call__(self, **datum_kwargs):
hdulist = fits.open(self._fits_file_path)
# Image data is generally the last entry in the FITS file
ccd_image_data = hdulist[-1]
return ccd_image_data
handler_registry = databroker.core.discover_handlers()
handler_registry["FITS"] = FitsHandler
filler = event_model.Filler(handler_registry)
runs = catalog["My_Catalog"]
print(list(runs))
run = runs[-1] # How to use custom handler / filler here?
>>> databroker.__version__
'1.2.0'
Thanks for reading this!
The dict returned by discover_handlers() is just a copy; mutating it won't have any effect on other parts of the program. (I should change that return an immutable DictView....) Modifying the handler registry at run time is fragile because of some caching that happens internally. Can you add:
handler_registry:
FITS: "your_module.FitsHandler"
to the configuration file that is producing catalog["My_Catalog"]
?
@danielballan Thanks! I did not know about this option. A small hurdle is that my module is not installed...and not really structured to be a proper package. I suppose pip install -e .
could be used temporarily though.
In the interim, Dylan has pointed me to a FITS handler that is ready to be installed. I will give that a try next.
import types
import sys
my_module = types.ModuleType('my_module')
my_module.FitsHandler = FitsHandler
sys.modules['my_module'] = my_module
from my_module import FitsHandler as _FitsHandler
UPDATE: Databroker is now using the FITS handler that Dylan shared with me
Hooray!
I ended up needing to rewrite the stored data (ummm, 'datum's?) with "frame_per_point" passed as a keyword argument when creating the resource. Otherwise extract_shape()
would raise RuntimeError(f"Could not figure out shape of {key}")
for key="ccd", even though the shape is explicitly set in the event descriptor.
This is the relevant section where the documents are being generated from existing data...
...
descriptor_keys['ccd']['shape'][0] = sample_fits['Images'].data.shape[0]
descriptor_keys['ccd']['shape'][1] = sample_fits['Images'].data.shape[1]
primary_stream_bundle = run_bundle.compose_descriptor(
data_keys=descriptor_keys,
name=primary_stream_bundle_name)
yield 'descriptor', primary_stream_bundle.descriptor_doc
...
It looks like extract_shape()
might not be expecting the simple case of descriptor['data_keys'][key]['shape']
already defined correctly; so it raises an exception instead of simply returning return descriptor['data_keys'][key]['shape']
?
This is probably just my ignorance of databroker
in general (and extract_shape()
in particular), but I'm curious whether you see this as an issue in the logic of extract_shape()
?
I finally managed to get debug information on the recursion condition. It appears that the recursion loop is caused by
intake
forcing a reload of information every time it checks the property/attribute of an Entry/Catalog.
It turns out that the inifite recursion I was experiencing (see my post from Jan. 11) was related to an outdated (and beta) version of databroker / its dependencies. Upgrading to databroker 1.2.0 fixed this problem. Thanks, Dylan!
It seems strange that conda
pulled databroker 1.0.0b9
on 2021-01-05, when issuing the following commands.
conda create --name fits-env python=3.8 pandas astropy requests event-model suitcase-mongo -c lightsource2-tag
conda activate fits-env
conda install conda-build
conda install -c nsls2forge databroker
conda install scikit-image
Just to clarify, databroker now works as expected. I'm posting this simply for informational purposes.
HI @ksubrama_gitlab
Yes, spaces in paths was a problem that we found and fixed on master of yaqd-control (needed to be quoted in a subprocess call):
https://gitlab.com/yaq/yaqd-control/-/blob/master/CHANGELOG.md
I thought we had done a release since then, but apparently not (I think we have one other thing that we were planning on putting in the release, that just kept getting bumped down the priority list)
Anyway, I pinged you on our matrix channel if you want to talk more.
anywho https://matrix.to/#/#yaq:matrix.org?via=matrix.org
you came to this chatroom in the past, with username selfification
To get the entry points they do need to be installed. yaqd-control is more for production machines than dev (though it can be useful in some dev circumstances)
for dev work I tend to just run the entry point directly in a terminal e.g. $ yaqd-horiba-micro-hr -v
(-v to get verbose outputs)
that prints logs to stderr so I can see what is going on.
you can install in editable mode to be able to make changes without having to re pip install every time
flit install --pth-file
(or on systems like linux/mac that have symlinks flit install -s
)
yaqd edit-config horiba-micro-hr
will open up notepad (or actually whatever is in $EDITOR) with that file directly