https://github.com/PredictiveEcology/SpaDES.core/blob/b2ca257d659e3b7ed4996c777d2add7f63398ee8/R/copy.R#L85
To reproduce problem with debugging, checkout SpaDES.core@ShowEnvironmentError
devtools::test(filter = "checkpoint$")That will open a browser() call and then the next 5 lines or so can be run interactively... the all.equal is TRUE, then change the environment and it goes all weird.
I'm not sure I like the solution to this name collision issue. I'm posting the example here instead of scfm because it could happen with anything, really. In this case, one object (from scfm) is a spatialPolygonsDataFrame, and the other is a list (whatever Tati was referring to, presumably fireSense_dataPrep). I'm not sure why she needs both objects in the sim list, but that's irrelevant. In this case, the fix was this:
if (is(sim$firePoints, "list")){
tmp <- do.call(what = bind, args = list(sim$firePoints))
} else {
tmp <- sim$firePoints
}
# sim$firePoints <- tmp # TM ~ Terrible terrible idea. Other modules might use this object and
# espect it to be the list of fire points
# You shouldn't modify it unless its a dataPrep module or smth and even so, its dangerous.
# Just use your local variable, or save as a new sim.Object if this needs to go across modules.
# I checked all SCFM modules, and only regime uses firePoints, so I am keeping the local
# Object
#several lines later
# firePolys <- unlist(sim$firePoints) # TM ~ Use tmp, as its already unlisted!
firePolys <- tmp # TM ~ Use tmp, as its already unlisted!sim$firePoints could be anything supplied by another module, including objects that can't be coerced into something usable by 'scfm', unlike in this example. So while the workaround is correct, I don't think the code that does this should happen in scfm. A user who only uses scfm might wonder "why can firePoints be a list if the metadata stipulates SPDF, and the module supplies an SPDF". It feels like an integration module should be used instead, but this is tricky because presumably both modules using 'sim$firePoints' are parameterization modules that need it during some init event. So whatever integration module is created, it would have to schedule this conversion to happen in between the two init events. But either way it is an idiosyncratic integration issue that belongs in an idiosyncratic integration module. And even that doesn't resolve the general problem of having two completely irreconcilable objects with the same name. That's my take anyway.
I'm having a strange behaviour during save events ever since I started using the last reproducible big changes - not sure if this is a reproducible issue or not though
the save event seems to be saving sim$speciesLayers even though they are not in the outputs data.frame I pass to the modules. my simulation seems to be doing this at each save of every module.
1) I'm not sure why this is happening in Biomass_core, given that sim$speciesLayers is not specified in outputs(sim) and the module doens't use this object during simulations
2) I'm not sure why it's using a temporary folder in Biomass_core, but not when it does this in other modules that effectively use this object (like Biomass_borealDataPrep)
DuringBiomass_core:
2021-02-23 23:30:53 INFO:: total elpsd: 8.5 mins | 2001 Biomass_core plotSum
2021-02-23 23:30:53 INFO:: total elpsd: 8.5 mins | 2002 save later 1
2021-02-23 23:30:54 INFO::Hardlinked version of file created at: /tmp/Rtmpg5dfT5/Require/m2YPnTgZNVt/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/m2YPnTgZNVt/speciesLayers_3cda4aed011c168c.gri, which points to /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.gri; no copy was made.
2021-02-23 23:30:54 INFO:: total elpsd: 8.5 mins | 2002 Biomass_core mortali
2021-02-23 23:30:54 INFO::LandR::vegTypeMapGenerator: NROW(cohortData) == 11328
2021-02-23 23:30:56 INFO::Hardlinked version of file created at: /tmp/Rtmpg5dfT5/Require/O8k5dwfGbPw/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/O8k5dwfGbPw/speciesLayers_3cda4aed011c168c.gri, which points to /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.gri; no copy was made.
2021-02-23 23:30:56 INFO:: total elpsd: 8.6 mins | 2002 Biomass_core plotSum
2021-02-23 23:30:56 INFO:: total elpsd: 8.6 mins | 2003 save later 1
2021-02-23 23:30:56 INFO::Hardlinked version of file created at: /tmp/Rtmpg5dfT5/Require/T0tAfvxgrIr/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/T0tAfvxgrIr/speciesLayers_3cda4aed011c168c.gri, which points to /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.grd, /tmp/Rtmpg5dfT5/Require/Lty0oSST0Lx/speciesLayers_3cda4aed011c168c.gri; no copy was made.Biomass_borealDataPrep the difference is that it hardlinks the file to inputPath(sim)
What is
outputs(yourOutputSim)?
prepInputs touches those layers herehttps://github.com/CeresBarros/Biomass_core/blob/e72285307d1c50f63b549f74244cbb52b2285372/Biomass_core.R#L538-L542
filename2 = NULL there, which, if you're hitting that block, could explain the tmp files used for those layres
save during simulation (so after .inputObjects and init)
outputs(sim) ^^
experiment2 call
spades() but only during experiment2()?
NOTE:
NOTE: When writing scripts put this at the start:
if (!require("Require")) {install.packages("Require"); library(Require)}then load all libraries with e.g.,
Require(c("SpaDES", "PredictiveEcology/SpaDES.experiment"))all in one line... and don't "pre-load" packages that modules need. Let them do that themselves.
According to the
outputs, that object is not being saved...
I know... that's what I said in my initial post
Require call that you sugested
spades is not running my simList generated using simInitAndSpades(..., events = "init"). Here's whats happening:## This is how I generate my `simList`
Browse[2]> simInitList <- Cache(simInitAndSpades
+ , times = simTimes
+ , params = simParams
+ , modules = simModules2
+ , loadOrder = unlist(simModules2)
+ , paths = simPaths2
+ , objects = simObjects
+ , outputs = outputs
+ , events = "init"
+ , debug = TRUE
+ , .plotInitialTime = NA
+ , cacheRepo = simPaths2$cachePath
+ , userTags = c("simInitAndInits", scenario)
+ , omitArgs = c("userTags", ".plotInitialTime", "debug"))
...(Object to retrieve (1ba83c088049ea42.rds) is large: 259.8 Mb)
loaded cached result from previous simInitAndSpades call,
## check end time and change to 2 for `spades` test
Browse[2]> end(simInitList)
[1] 100
attr(,"unit")
[1] "year"
Browse[2]> end(simInitList) <- 2
## nothing happens whn calling `spades`
Browse[2]> spades(simInitList)
simList saved in
SpaDES.core:::.pkgEnv$.sim
It will be deleted at next spades() call.simList has scheduled events, and the inits ran fine:Browse[2]> events(simInitList)
eventTime moduleName eventType eventPriority
1: 1 save spades 1.00
2: 1 fireSense_IgnitionFit run 5.00
3: 1 fireSense_IgnitionPredict run 5.00
4: 1 Biomass_core mortalityAndGrowth 6.00
5: 1 Biomass_core summaryBGM 8.25
6: 1 Biomass_core plotSummaryBySpecies 9.00
7: 5 Biomass_fuelsPFG doFuelTypes 1.00
8: 5 Biomass_fireProperties calcFireProperties 2.25
9: 5 fireSpread doFireSpread 2.50
10: 5 Biomass_regeneration fireDisturbance 3.00
11: 11 Biomass_core summaryBGMstart 4.00
12: 11 Biomass_core Dispersal 5.00
13: 11 Biomass_core cohortAgeReclassification 7.00
14: 11 Biomass_core summaryRegen 8.00
15: 100 Biomass_core plotSummaryBySpecies 9.00
Browse[2]> > completed(simInitList)
eventTime moduleName eventType eventPriority
1: 1 checkpoint init 0
2: 1 save init 0
3: 1 progress init 0
4: 1 load init 0
5: 1 Biomass_borealDataPrep init 1
6: 1 Biomass_speciesParameters init 1
7: 1 Biomass_core init 1
8: 1 Biomass_fuelsPFG init 1
9: 1 fireSense_dataPrep init 1
10: 1 fireSense_IgnitionFit init 1
11: 1 fireSense_IgnitionPredict init 1
12: 1 Biomass_fireProperties init 1
13: 1 fireSpread init 1
14: 1 Biomass_regeneration init 1
Browse[2]> start(simInitList)
[1] 1
attr(,"unit")
[1] "year"fireSense_IgnitionPredictsimInitList <- Cache(simInitAndSpades
, times = list(start = 1, end = 2)
, params = simParams
, modules = simModules2
, loadOrder = unlist(simModules2)
, paths = simPaths2
, objects = simObjects
, outputs = outputs
, debug = TRUE
, .plotInitialTime = NA
, cacheRepo = simPaths2$cachePath
, useCache = FALSE
, userTags = c("simInitAndInits", scenario)
, omitArgs = c("userTags", ".plotInitialTime", "debug"))
....
18: 435.15456: 0.570331 1.00000e-14 0.545362 0.152339 0.385196
19: 435.15456: 0.570330 1.00000e-14 0.545359 0.152339 0.385196
Mar13 22:41:11 frSns_ 1 fireSense_IgnitionPredict run 5
Error in frequencyPredictRun(sim) :
fireSense_IgnitionPredict> 'julMDC' not found in data objects.
In addition: There were 50 or more warnings (use warnings() to see the first
Browse[2]> completed(SpaDES.core:::.pkgEnv$.sim)
eventTime moduleName eventType eventPriority
1: 1 checkpoint init 0
2: 1 save init 0
3: 1 progress init 0
4: 1 load init 0
5: 1 Biomass_borealDataPrep init 1
6: 1 Biomass_speciesParameters init 1
7: 1 Biomass_core init 1
8: 1 Biomass_fuelsPFG init 1
9: 1 fireSense_dataPrep init 1
10: 1 fireSense_IgnitionFit init 1
11: 1 fireSense_IgnitionPredict init 1
12: 1 Biomass_fireProperties init 1
13: 1 fireSpread init 1
14: 1 Biomass_regeneration init 1
15: 1 save spades 1
16: 1 fireSense_IgnitionFit run 5
Browse[2]> events(SpaDES.core:::.pkgEnv$.sim)
eventTime moduleName eventType eventPriority
1: 1 fireSense_IgnitionPredict run 5.00
2: 1 Biomass_core mortalityAndGrowth 6.00
3: 1 Biomass_core summaryBGM 8.25
4: 1 Biomass_core plotSummaryBySpecies 9.00
5: 2 Biomass_core plotSummaryBySpecies 9.00
6: 5 Biomass_fuelsPFG doFuelTypes 1.00
7: 5 Biomass_fireProperties calcFireProperties 2.25
8: 5 fireSpread doFireSpread 2.50
9: 5 Biomass_regeneration fireDisturbance 3.00
10: 5 save later 10.00
11: 11 Biomass_core summaryBGMstart 4.00
12: 11 Biomass_core Dispersal 5.00
13: 11 Biomass_core cohortAgeReclassification 7.00
14: 11 Biomass_core summaryRegen 8.00fireSense_IgnitionPredict issue and see if that's the culprit...
@development versions, spades call is working again
Note for best practices: Do not put "random" elements in metadata, such as this:
defineParameter(".tempdir", "character", tempdir(), NA, NA,
"Temporary (scratch) directory to use for transient files (e.g., GIS intermediates)."),This means that a module is unique every time it is run; this therefore affects Caching, i.e., every time a session is restarted, then all caching at the event or module level will require a new run of the event or module.
Suggestion -- set to NULL in the metadata, and in the module, test for is.null, then set to tempdir() e.g.,:
defineParameter(".tempdir", "character", NULL, NA, NA,
"Temporary (scratch) directory to use for transient files (e.g., GIS intermediates)."),