Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Eliot McIntire
    @eliotmcintire
    This message was deleted
    Eliot McIntire
    @eliotmcintire
    Welcome! We are experimenting with this gitter app for disucssions about the SpaDES-core package in R. For immediate issues, please create an issue on GitHub.com/PredictiveEcology/SpaDES-core
    Eliot McIntire
    @eliotmcintire
    @achubaty I have pushed a PR with the work around. It doesn't not identify the line in Copy that causes the problem.
    Eliot McIntire
    @eliotmcintire

    To reproduce problem with debugging, checkout SpaDES.core@ShowEnvironmentError

    devtools::test(filter = "checkpoint$")

    That will open a browser() call and then the next 5 lines or so can be run interactively... the all.equal is TRUE, then change the environment and it goes all weird.

    Eliot McIntire
    @eliotmcintire
    Should there be a check during each do.event call at the SpaDES.core level for "enormous" objects that shouldn't be, like formula and function classes?
    ianmseddy
    @ianmseddy

    I'm not sure I like the solution to this name collision issue. I'm posting the example here instead of scfm because it could happen with anything, really. In this case, one object (from scfm) is a spatialPolygonsDataFrame, and the other is a list (whatever Tati was referring to, presumably fireSense_dataPrep). I'm not sure why she needs both objects in the sim list, but that's irrelevant. In this case, the fix was this:

     if (is(sim$firePoints, "list")){
        tmp <- do.call(what = bind, args = list(sim$firePoints))
      } else {
        tmp <- sim$firePoints
      }
    
    # sim$firePoints <- tmp # TM ~ Terrible terrible idea. Other modules might use this object and
      # espect it to be the list of fire points
      # You shouldn't modify it unless its a dataPrep module or smth and even so, its dangerous.
      # Just use your local variable, or save as a new sim.Object if this needs to go across modules.
      # I checked all SCFM modules, and only regime uses firePoints, so I am keeping the local
      # Object
    #several lines later 
      # firePolys <- unlist(sim$firePoints) # TM ~ Use tmp, as its already unlisted!
      firePolys <- tmp # TM ~ Use tmp, as its already unlisted!

    sim$firePoints could be anything supplied by another module, including objects that can't be coerced into something usable by 'scfm', unlike in this example. So while the workaround is correct, I don't think the code that does this should happen in scfm. A user who only uses scfm might wonder "why can firePoints be a list if the metadata stipulates SPDF, and the module supplies an SPDF". It feels like an integration module should be used instead, but this is tricky because presumably both modules using 'sim$firePoints' are parameterization modules that need it during some init event. So whatever integration module is created, it would have to schedule this conversion to happen in between the two init events. But either way it is an idiosyncratic integration issue that belongs in an idiosyncratic integration module. And even that doesn't resolve the general problem of having two completely irreconcilable objects with the same name. That's my take anyway.

    Eliot McIntire
    @eliotmcintire
    Like you, I don't like this solution. I feel like a possibility is for the module developers who use this object to talk and agree on a single object type/class. It would involve some code changing but it would be better than this solution which is trying to be more than one class/type.
    Alex Chubaty
    @achubaty
    i agree -- in this particular case we have the power to change the name of the object so that's what we should do.
    Eliot McIntire
    @eliotmcintire
    After discussion, it seems like the best solution to this specific case is to rename both objects, with slightly more precise names.
    Ceres Barros
    @CeresBarros

    I'm having a strange behaviour during save events ever since I started using the last reproducible big changes - not sure if this is a reproducible issue or not though
    the save event seems to be saving sim$speciesLayers even though they are not in the outputs data.frame I pass to the modules. my simulation seems to be doing this at each save of every module.

    1) I'm not sure why this is happening in Biomass_core, given that sim$speciesLayers is not specified in outputs(sim) and the module doens't use this object during simulations
    2) I'm not sure why it's using a temporary folder in Biomass_core, but not when it does this in other modules that effectively use this object (like Biomass_borealDataPrep)

    DuringBiomass_core:

    2021-02-23 23:30:53 INFO:: total elpsd: 8.5 mins | 2001 Biomass_core plotSum
    2021-02-23 23:30:53 INFO:: total elpsd: 8.5 mins | 2002 save later 1
    2021-02-23 23:30:54 INFO::Hardlinked version of file created at: /tmp/Rtmpg5dfT5/Require/m2YPnTgZNVt/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/m2YPnTgZNVt/speciesLayers_3cda4aed011c168c.gri, which points to /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.gri; no copy was made.
    2021-02-23 23:30:54 INFO:: total elpsd: 8.5 mins | 2002 Biomass_core mortali
    2021-02-23 23:30:54 INFO::LandR::vegTypeMapGenerator: NROW(cohortData) == 11328
    2021-02-23 23:30:56 INFO::Hardlinked version of file created at: /tmp/Rtmpg5dfT5/Require/O8k5dwfGbPw/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/O8k5dwfGbPw/speciesLayers_3cda4aed011c168c.gri, which points to /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.gri; no copy was made.
    2021-02-23 23:30:56 INFO:: total elpsd: 8.6 mins | 2002 Biomass_core plotSum
    2021-02-23 23:30:56 INFO:: total elpsd: 8.6 mins | 2003 save later 1
    2021-02-23 23:30:56 INFO::Hardlinked version of file created at: /tmp/Rtmpg5dfT5/Require/T0tAfvxgrIr/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/T0tAfvxgrIr/speciesLayers_3cda4aed011c168c.gri, which points to /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.gri; no copy was made.
    during Biomass_borealDataPrep the difference is that it hardlinks the file to inputPath(sim)
    Eliot McIntire
    @eliotmcintire
    I don't really have any ideas of how this is happening. If you can give a repository that I can reproduce the error... I could try to look at it.
    Debugging:
    What is outputs(yourOutputSim)?
    Ceres Barros
    @CeresBarros
    everything should be up to date, so hopefully you only have to pull.
    Alex Chubaty
    @achubaty
    note that filename2 = NULL there, which, if you're hitting that block, could explain the tmp files used for those layres
    Ceres Barros
    @CeresBarros
    @achubaty this is way past that prepInputs
    it's happening at every save during simulation (so after .inputObjects and init)
    @eliotmcintire outputs(sim) ^^
    If you do try to reproduce @eliotmcintire, no need to go all the way down to running the validation. this behaviour is visible during experiment2 call
    Alex Chubaty
    @achubaty
    ^^ ah! that seems like an important point
    to confirm, you don't see this behaivour running the module using spades() but only during experiment2()?
    Ceres Barros
    @CeresBarros
    haven't tried yet
    Eliot McIntire
    @eliotmcintire
    According to the outputs, that object is not being saved...

    NOTE:
    NOTE: When writing scripts put this at the start:

    if (!require("Require")) {install.packages("Require"); library(Require)}

    then load all libraries with e.g.,

    Require(c("SpaDES", "PredictiveEcology/SpaDES.experiment"))

    all in one line... and don't "pre-load" packages that modules need. Let them do that themselves.

    Ceres Barros
    @CeresBarros
    ok thanks

    According to the outputs, that object is not being saved...

    I know... that's what I said in my initial post

    Eliot McIntire
    @eliotmcintire
    :+1:
    Ceres Barros
    @CeresBarros

    and don't "pre-load" packages that modules need. Let them do that themselves.

    I'm only preloading packages that my sourced scripts need - that includes LandR yes...

    But I'll change that to the Require call that you sugested
    Ceres Barros
    @CeresBarros

    to confirm, you don't see this behaivour running the module using spades() but only during experiment2()?

    just tried using spades() and I can confirm that I DON'T see the strange save behaviour

    Ceres Barros
    @CeresBarros
    Not sure what's going on, but spades is not running my simList generated using simInitAndSpades(..., events = "init"). Here's whats happening:
    ## This is how I generate my `simList`
    Browse[2]>   simInitList <- Cache(simInitAndSpades
    +                    , times = simTimes
    +                    , params = simParams
    +                    , modules = simModules2
    +                    , loadOrder = unlist(simModules2)
    +                    , paths = simPaths2
    +                    , objects = simObjects
    +                    , outputs = outputs
    +                    , events = "init"
    +                    , debug = TRUE
    +                    , .plotInitialTime = NA
    +                    , cacheRepo = simPaths2$cachePath
    +                    , userTags = c("simInitAndInits", scenario)
    +                    , omitArgs = c("userTags", ".plotInitialTime", "debug"))
      ...(Object to retrieve (1ba83c088049ea42.rds) is large: 259.8 Mb)
         loaded cached result from previous simInitAndSpades call,
    
    ## check end time and change to 2 for `spades` test
    Browse[2]> end(simInitList)
    [1] 100
    attr(,"unit")
    [1] "year"
    Browse[2]> end(simInitList) <- 2
    
    ## nothing happens whn calling  `spades`
    Browse[2]> spades(simInitList)
    simList saved in
     SpaDES.core:::.pkgEnv$.sim
    It will be deleted at next spades() call.
    the simList has scheduled events, and the inits ran fine:
    Browse[2]> events(simInitList)
        eventTime                moduleName                 eventType eventPriority
     1:         1                      save                    spades          1.00
     2:         1     fireSense_IgnitionFit                       run          5.00
     3:         1 fireSense_IgnitionPredict                       run          5.00
     4:         1              Biomass_core        mortalityAndGrowth          6.00
     5:         1              Biomass_core                summaryBGM          8.25
     6:         1              Biomass_core      plotSummaryBySpecies          9.00
     7:         5          Biomass_fuelsPFG               doFuelTypes          1.00
     8:         5    Biomass_fireProperties        calcFireProperties          2.25
     9:         5                fireSpread              doFireSpread          2.50
    10:         5      Biomass_regeneration           fireDisturbance          3.00
    11:        11              Biomass_core           summaryBGMstart          4.00
    12:        11              Biomass_core                 Dispersal          5.00
    13:        11              Biomass_core cohortAgeReclassification          7.00
    14:        11              Biomass_core              summaryRegen          8.00
    15:       100              Biomass_core      plotSummaryBySpecies          9.00
    
    Browse[2]> > completed(simInitList)
        eventTime                moduleName eventType eventPriority
     1:         1                checkpoint      init             0
     2:         1                      save      init             0
     3:         1                  progress      init             0
     4:         1                      load      init             0
     5:         1    Biomass_borealDataPrep      init             1
     6:         1 Biomass_speciesParameters      init             1
     7:         1              Biomass_core      init             1
     8:         1          Biomass_fuelsPFG      init             1
     9:         1        fireSense_dataPrep      init             1
    10:         1     fireSense_IgnitionFit      init             1
    11:         1 fireSense_IgnitionPredict      init             1
    12:         1    Biomass_fireProperties      init             1
    13:         1                fireSpread      init             1
    14:         1      Biomass_regeneration      init             1
    
    Browse[2]> start(simInitList)
    [1] 1
    attr(,"unit")
    [1] "year"
    Ceres Barros
    @CeresBarros
    this, however, works if not for a problem in fireSense_IgnitionPredict
    simInitList <- Cache(simInitAndSpades
                         , times = list(start = 1, end = 2)
                         , params = simParams
                         , modules = simModules2
                         , loadOrder = unlist(simModules2)
                         , paths = simPaths2
                         , objects = simObjects
                         , outputs = outputs
                         , debug = TRUE
                         , .plotInitialTime = NA
                         , cacheRepo = simPaths2$cachePath
                         , useCache = FALSE
                         , userTags = c("simInitAndInits", scenario)
                         , omitArgs = c("userTags", ".plotInitialTime", "debug"))
    ....
     18:     435.15456: 0.570331 1.00000e-14 0.545362 0.152339 0.385196
     19:     435.15456: 0.570330 1.00000e-14 0.545359 0.152339 0.385196
    Mar13 22:41:11 frSns_ 1         fireSense_IgnitionPredict run       5
    Error in frequencyPredictRun(sim) :
      fireSense_IgnitionPredict> 'julMDC' not found in data objects.
    In addition: There were 50 or more warnings (use warnings() to see the first 
    
    Browse[2]> completed(SpaDES.core:::.pkgEnv$.sim)
        eventTime                moduleName eventType eventPriority
     1:         1                checkpoint      init             0
     2:         1                      save      init             0
     3:         1                  progress      init             0
     4:         1                      load      init             0
     5:         1    Biomass_borealDataPrep      init             1
     6:         1 Biomass_speciesParameters      init             1
     7:         1              Biomass_core      init             1
     8:         1          Biomass_fuelsPFG      init             1
     9:         1        fireSense_dataPrep      init             1
    10:         1     fireSense_IgnitionFit      init             1
    11:         1 fireSense_IgnitionPredict      init             1
    12:         1    Biomass_fireProperties      init             1
    13:         1                fireSpread      init             1
    14:         1      Biomass_regeneration      init             1
    15:         1                      save    spades             1
    16:         1     fireSense_IgnitionFit       run             5
    
    Browse[2]> events(SpaDES.core:::.pkgEnv$.sim)
        eventTime                moduleName                 eventType eventPriority
     1:         1 fireSense_IgnitionPredict                       run          5.00
     2:         1              Biomass_core        mortalityAndGrowth          6.00
     3:         1              Biomass_core                summaryBGM          8.25
     4:         1              Biomass_core      plotSummaryBySpecies          9.00
     5:         2              Biomass_core      plotSummaryBySpecies          9.00
     6:         5          Biomass_fuelsPFG               doFuelTypes          1.00
     7:         5    Biomass_fireProperties        calcFireProperties          2.25
     8:         5                fireSpread              doFireSpread          2.50
     9:         5      Biomass_regeneration           fireDisturbance          3.00
    10:         5                      save                     later         10.00
    11:        11              Biomass_core           summaryBGMstart          4.00
    12:        11              Biomass_core                 Dispersal          5.00
    13:        11              Biomass_core cohortAgeReclassification          7.00
    14:        11              Biomass_core              summaryRegen          8.00
    will try to fix the fireSense_IgnitionPredict issue and see if that's the culprit...
    Eliot McIntire
    @eliotmcintire
    I think the work around is to set times = list(start = 1, end = 1) for the events = "init" ... it looks like there is a weird thing with end time being way past the init events, but spades only running the init events because events = "init"...
    Ceres Barros
    @CeresBarros
    I'll try, but I don't think that's it, because I used the exact same configuration on my LandRPub sims and it worked fine
    Ceres Barros
    @CeresBarros
    update: spades call on that simList worked fine on my local machine, which makes me think that there's something going wrong in the simLists I have in BC
    I'll start by comparing package versions, as in both the local machine and BC the simLists were generated from cleared Caches
    Ceres Barros
    @CeresBarros
    update #2: SpaDES.core and reproducible were at versions 1.0.6.9015 and 1.2.6.9009, respectively. After updating both to current @development versions, spades call is working again
    Eliot McIntire
    @eliotmcintire

    Note for best practices: Do not put "random" elements in metadata, such as this:

        defineParameter(".tempdir", "character", tempdir(), NA, NA,
                        "Temporary (scratch) directory to use for transient files (e.g., GIS intermediates)."),

    This means that a module is unique every time it is run; this therefore affects Caching, i.e., every time a session is restarted, then all caching at the event or module level will require a new run of the event or module.
    Suggestion -- set to NULL in the metadata, and in the module, test for is.null, then set to tempdir() e.g.,:

        defineParameter(".tempdir", "character", NULL, NA, NA,
                        "Temporary (scratch) directory to use for transient files (e.g., GIS intermediates)."),
    Ceres Barros
    @CeresBarros
    great tip, thanks Eliot
    this should go to the SpaDES book that you/we should write - I suggest pandas in the cover
    Tati Micheletti
    @tati-micheletti
    I agree with the panda's cover!