Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Rachel Tunnicliffe
    @rt17603
    For your footprint issue, there are a few things going on here.
    (1) If you actually retrieve the data (using e.g. get_footprint) rather than search I think you'll see that you are getting both of the footprints (2017, 2018) you added. The start_date and end_date currently returned are a bit misleading but I think otherwise this is actually doing the right thing
    (2) For your specific case of having the two sets of footprint files with inlet="10magl" and another set with inlet="10m" Gareth is just finalising an update which will allow you to delete the extra one(s) (the 10magl footprints)
    Finally, I've added in an update already around the inlet as "10m" versus "10magl" (versus "10") problem to hopefully stop that from happening again. It should try and format this to always be of the form "10m" etc. even if you input "10magl" and if you search for "10magl" this should also work as well. However, let me know if you spot any issues with that
    dangrosvenor
    @dangrosvenor
    Thanks. So far I've been using "standardise_footprint" to add in two files given to me by Hannah. Are you suggesting that I should use get_footprint to download the data myself instead?
    Rachel Tunnicliffe
    @rt17603
    So standardise_footprintis the way the data gets added to the object store (standardise because this gets checked to make sure this is in the right format etc.)
    get_footprint(...) is basically the same as search(...).retrieve(...) where you're grabbing the data from the object store
    So no, I'm just talking about processes you've already been doing rather than getting any new data etc.
    And you don't need to use get_footprint() directly if you're using the ModelScenario set up as this will do it for you. I was just saying if you actually grabbed the data you found through the search then you'd probably see this did contain the whole date range you'd added for the 2 footprints.
    Rachel Tunnicliffe
    @rt17603
    For reference by the way this was the issue we raised about this with the discussion etc.: openghg/openghg#469
    dangrosvenor
    @dangrosvenor
    Ok, thanks Rachel. And, yes, it seems that the issue with ModelScenario is for the fluxes rather than footprints to add more confusion!
    So, I should wait for PR #468 to be merged (and other modifications too?), or is that all done already?
    Rachel Tunnicliffe
    @rt17603
    You'll need to wait for PR #468 as you'll need to be able to modify your pre-existing object store (unless you wanted to just delete your whole object store and try again). @gareth-j should be able to give a timeline on that.
    10 replies
    Gareth Jones
    @gareth-j
    I’m just running some final checks
    Then it should be ready to go
    hanchawn
    @hanchawn
    For standardise.standardise_surface I'm now getting an error: ValueError: could not convert string to float: '185m'
    I've put inlet = 185 so I think it must be changing it in the code to '185m' by default (we asked for it to do this for the other inputs so that it always uses 'm' instead of 'magl')
    Rachel Tunnicliffe
    @rt17603
    Ok I'll take a quick look at that
    Would you be able to send over a code snippet of what you're running?
    Rachel Tunnicliffe
    @rt17603
    Link to issue: openghg/openghg#477
    hanchawn
    @hanchawn
    Thanks Rachel, I'll add the code snippet to the ticket
    hanchawn
    @hanchawn
    I've added another comment to that issue because I think this is related, using search_surface I can't find results if an inlet is defined, so ModelScenario also doesn't return any obs data
    Rachel Tunnicliffe
    @rt17603
    Ok - I've moved that to a separate issue as I think it'll be related but perhaps distinct - openghg/openghg#478
    Thanks for the details, that's helpful
    Rachel Tunnicliffe
    @rt17603
    So I have a PR openghg/openghg#480 for Issue #477 but we're just having some issues with underlying version changes which are causing a few extra test fails. So I'll hold off on merging that just yet
    However, a short term fix is to not pass an inlet at all for your "openghg" source_format files, as this can be inferred from the attributes in the file and shouldn't produce the error you're seeing
    Gareth Jones
    @gareth-j
    OK @dangrosvenor @hanchawn I’ve added in the ability to modify metadata and delete data from the object store
    Rachel Tunnicliffe
    @rt17603
    It's worth pointing out there's also a new tutorial for this functionality that Gareth has added to the notebooks/tutorials/local/ folder called "Modifying_and_deleting_data.ipynb" (or .md) - @gareth-j is this included on the Tutorial pages as well now?
    Gareth Jones
    @gareth-j
    For some reason the version in the docs contains a Python error caused by xarray, I’m just looking into it now
    But if you can ignore the large error box the rest of the tutorial is up: https://docs.openghg.org/tutorials/local/Modifying_and_deleting_data.html
    Eric Saboya
    @EricSaboya
    Hi all, a quick question about the openghg.retrieve.get_bc() function. If a start date and end date are specified when calling this function, are the boundary condition files that are closest to the start date retrieved from the objectstore? e.g. If only boundary condition files for "2012-01-01" are in the objectstore, a specified start_date="2019-01-01" will retrieve the 2012 data in this case?
    6 replies
    dangrosvenor
    @dangrosvenor

    It looks like ModelScenario step works for me now! :-
    In [2]: scenario=ModelScenario(site='WAO',inlet='20m',domain='EUROPE',species='co2',source='edgar-ukghg-ff',start_date='2018-01-01',end_date='2018
    ...: -01-31')
    Adding obs_surface to model scenario
    Updating any inputs based on observation data
    site: wao, species: co2, inlet: 20m
    Adding footprint to model scenario
    Adding flux to model scenario
    Adding boundary_conditions to model scenario

    I deleted the data store and started again using the new code - I had to drop the date keyword from the standardise_flux stage, which I guess was the change you made. Thanks for all the help on this!
    N.B. - I tried this requesting 30GB of memory, but it seems it wasn't enough. When I requested 50GB it worked. So, it is pretty memory hungry still!

    Gareth Jones
    @gareth-j
    Ah I’m glad it’s working!
    ModelScenario is still quite memory hungry due to it pulling everything from the object store and loading each of the Datasets into memory
    I need to have a look at how we can reduce the memory usage there
    Rachel Tunnicliffe
    @rt17603
    One thing you could try Dan is running any calculation steps (e.g. ModelScenario.footprints_data_merge) on ModelScenario with cache=False. This may reduce the memory a bit though not sure how much this would help yet.
    dangrosvenor
    @dangrosvenor
    Thanks, I'll give that at go. At the moment I'm having a small issue when running using a Jupyter notebook on the SPICE computers. Using an ipython session on Spice seems to work fine with the data store set in ~/.config/openghg/openghg.conf. However, using Jupyter it is saying :-
    "No environment variable OPENGHG_PATH found, please set to use the local object store"
    ... as though it is not picking up the path from the .conf file. Any ideas how to fix this?
    Gareth Jones
    @gareth-j
    It looks like you might be using an older version of the code
    Have you pulled the latest changes to devel?
    dangrosvenor
    @dangrosvenor
    Thanks Gareth, that fixed it. Sorry, only just saw your message...
    Gareth Jones
    @gareth-j
    No worries! Glad that worked
    Eric Saboya
    @EricSaboya
    Screenshot 2022-11-25 at 10.03.52.png
    Hi all, bit of an odd error I got this morning that's stumped me
    has anyone encountered this before or have any suggestions
    hanchawn
    @hanchawn
    I had a similar thing when the data_type on my obs data wasn't right, if you try retrieve.search you can check that the meta data looks like it should
    Rachel Tunnicliffe
    @rt17603
    Yes good idea Hannah. Eric: if you do a search_bc (or search) and keep the terms as loose as possible that should give you information about the data stored
    Eric Saboya
    @EricSaboya
    Thanks Hannah and Rachel! I was able to see the BC files I'd added using search but they had data_type=NaN (indeed no results were given for search_bc). Is there a way to change the datatype of these files or would this require having to remake the BC files and add them to the OS again?
    Rachel Tunnicliffe
    @rt17603
    Ok thanks for looking into this - that sounds like a bug where the data_type isn't being assigned when the file is being added so I'll raise an issue for this.
    In the meantime for the files you have added, you can edit metadata for this data to add this in to include the data_type as "boundary_conditions". Have a look at the "Modifying_and_deleting_data.ipynb" tutorial - https://github.com/openghg/openghg/blob/devel/notebooks/tutorials/local/Modifying_and_deleting_data.ipynb
    Rachel Tunnicliffe
    @rt17603
    I've raised an issue for this - openghg/openghg#491. @EricSaboya - if you could add any more relevant details (e.g. standardise_bc line run, file added etc.) that would be helpful for debugging.
    Eric Saboya
    @EricSaboya
    Thanks Rachel, will have a look at the tutorial and the issue to see if there's anything else I can add
    abcj123cour
    @abcj123cour
    Excuse me, I am trying to download the notebooks on windows. Do the tutorials run on windows ? If not how do I run the notebooks ?
    Gareth Jones
    @gareth-j
    Hi @abcj123cour, sorry for the delay in getting back to you. Did you have any luck getting the tutorials working?