For your footprint issue, there are a few things going on here.
(1) If you actually retrieve the data (using e.g. get_footprint) rather than search I think you'll see that you are getting both of the footprints (2017, 2018) you added. The start_date and end_date currently returned are a bit misleading but I think otherwise this is actually doing the right thing
(2) For your specific case of having the two sets of footprint files with inlet="10magl" and another set with inlet="10m" Gareth is just finalising an update which will allow you to delete the extra one(s) (the 10magl footprints)
(1) If you actually retrieve the data (using e.g. get_footprint) rather than search I think you'll see that you are getting both of the footprints (2017, 2018) you added. The start_date and end_date currently returned are a bit misleading but I think otherwise this is actually doing the right thing
(2) For your specific case of having the two sets of footprint files with inlet="10magl" and another set with inlet="10m" Gareth is just finalising an update which will allow you to delete the extra one(s) (the 10magl footprints)
Finally, I've added in an update already around the inlet as "10m" versus "10magl" (versus "10") problem to hopefully stop that from happening again. It should try and format this to always be of the form "10m" etc. even if you input "10magl" and if you search for "10magl" this should also work as well. However, let me know if you spot any issues with that
get_footprint(...) is basically the same as search(...).retrieve(...) where you're grabbing the data from the object store
So no, I'm just talking about processes you've already been doing rather than getting any new data etc.
And you don't need to use
get_footprint() directly if you're using the ModelScenario set up as this will do it for you. I was just saying if you actually grabbed the data you found through the search then you'd probably see this did contain the whole date range you'd added for the 2 footprints.
For reference by the way this was the issue we raised about this with the discussion etc.: openghg/openghg#469
So, I should wait for PR #468 to be merged (and other modifications too?), or is that all done already?
You'll need to wait for PR #468 as you'll need to be able to modify your pre-existing object store (unless you wanted to just delete your whole object store and try again). @gareth-j should be able to give a timeline on that.
10 replies
Then it should be ready to go
For standardise.standardise_surface I'm now getting an error: ValueError: could not convert string to float: '185m'
I've put inlet = 185 so I think it must be changing it in the code to '185m' by default (we asked for it to do this for the other inputs so that it always uses 'm' instead of 'magl')
I've put inlet = 185 so I think it must be changing it in the code to '185m' by default (we asked for it to do this for the other inputs so that it always uses 'm' instead of 'magl')
Would you be able to send over a code snippet of what you're running?
Ok - I've moved that to a separate issue as I think it'll be related but perhaps distinct - openghg/openghg#478
Thanks for the details, that's helpful
So I have a PR openghg/openghg#480 for Issue #477 but we're just having some issues with underlying version changes which are causing a few extra test fails. So I'll hold off on merging that just yet
However, a short term fix is to not pass an inlet at all for your "openghg" source_format files, as this can be inferred from the attributes in the file and shouldn't produce the error you're seeing
But if you can ignore the large error box the rest of the tutorial is up: https://docs.openghg.org/tutorials/local/Modifying_and_deleting_data.html
Hi all, a quick question about the
openghg.retrieve.get_bc() function. If a start date and end date are specified when calling this function, are the boundary condition files that are closest to the start date retrieved from the objectstore? e.g. If only boundary condition files for "2012-01-01" are in the objectstore, a specified start_date="2019-01-01" will retrieve the 2012 data in this case?
6 replies
It looks like ModelScenario step works for me now! :-
In [2]: scenario=ModelScenario(site='WAO',inlet='20m',domain='EUROPE',species='co2',source='edgar-ukghg-ff',start_date='2018-01-01',end_date='2018
...: -01-31')
Adding obs_surface to model scenario
Updating any inputs based on observation data
site: wao, species: co2, inlet: 20m
Adding footprint to model scenario
Adding flux to model scenario
Adding boundary_conditions to model scenario
I deleted the data store and started again using the new code - I had to drop the date keyword from the standardise_flux stage, which I guess was the change you made. Thanks for all the help on this!
N.B. - I tried this requesting 30GB of memory, but it seems it wasn't enough. When I requested 50GB it worked. So, it is pretty memory hungry still!
ModelScenario is still quite memory hungry due to it pulling everything from the object store and loading each of the Datasets into memory
I need to have a look at how we can reduce the memory usage there
Thanks, I'll give that at go. At the moment I'm having a small issue when running using a Jupyter notebook on the SPICE computers. Using an ipython session on Spice seems to work fine with the data store set in ~/.config/openghg/openghg.conf. However, using Jupyter it is saying :-
"No environment variable OPENGHG_PATH found, please set to use the local object store"
... as though it is not picking up the path from the .conf file. Any ideas how to fix this?
"No environment variable OPENGHG_PATH found, please set to use the local object store"
... as though it is not picking up the path from the .conf file. Any ideas how to fix this?
Have you pulled the latest changes to devel?
Hi all, bit of an odd error I got this morning that's stumped me
has anyone encountered this before or have any suggestions
Thanks Hannah and Rachel! I was able to see the BC files I'd added using
search but they had data_type=NaN (indeed no results were given for search_bc). Is there a way to change the datatype of these files or would this require having to remake the BC files and add them to the OS again?
In the meantime for the files you have added, you can edit metadata for this data to add this in to include the data_type as "boundary_conditions". Have a look at the "Modifying_and_deleting_data.ipynb" tutorial - https://github.com/openghg/openghg/blob/devel/notebooks/tutorials/local/Modifying_and_deleting_data.ipynb
I've raised an issue for this - openghg/openghg#491. @EricSaboya - if you could add any more relevant details (e.g.
standardise_bc line run, file added etc.) that would be helpful for debugging.
