diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml new file mode 100644 index 0000000..a685168 --- /dev/null +++ b/.github/workflows/docs.yaml @@ -0,0 +1,75 @@ +# Build the Quarto docs site in docs/ and publish it to GitHub +# Pages. Same pattern as sciSpatialR's pkgdown workflow. +# +# The site is rebuilt from docs/ on every run and handed to Pages +# as a build artifact, so no rendered HTML is ever committed and +# there is no gh-pages branch to keep in sync. Pull requests build +# but do not publish, which surfaces a broken page during review +# rather than after merge. +# +# The pages are plain Markdown, so Quarto renders them without R. +# Which pages are built is set by `project: render` in +# docs/_quarto.yml. +# +# Repository prerequisite: Settings > Pages > Source = "GitHub +# Actions". Without that, the deploy job fails with a 404. + +name: docs + +on: + push: + branches: [main] + paths: + - "docs/**" + - ".github/workflows/docs.yaml" + pull_request: + paths: + - "docs/**" + - ".github/workflows/docs.yaml" + workflow_dispatch: + +# Deny by default; each job requests only the scopes it needs. +permissions: {} + +# Serialize publishing. Queued runs collapse to the newest, but an +# in-flight deployment is allowed to finish so the live site is +# never left half-written. +concurrency: + group: pages + cancel-in-progress: false + +jobs: + build: + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@v7 + + - uses: quarto-dev/quarto-actions/setup@v2 + + # Renders to docs/_site, as set in docs/_quarto.yml + - name: Build site + run: quarto render docs + + # No configure-pages step: Quarto reads its base URL from + # `site-url` in docs/_quarto.yml, so this job stays on + # contents:read alone. + - uses: actions/upload-pages-artifact@v5 + with: + path: docs/_site + + deploy: + # Publishing is a main-branch-only consequence of a green build. + if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' + needs: build + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + permissions: + pages: write + id-token: write + steps: + - id: deployment + uses: actions/deploy-pages@v5 diff --git a/.gitignore b/.gitignore index 60171cd..3b432fd 100644 --- a/.gitignore +++ b/.gitignore @@ -66,6 +66,7 @@ vignettes/*.pdf !3_output/**/tables/parity_summary.csv !3_output/**/tables/coverage.csv !3_output/**/tables/metric_summary.csv +!3_output/**/tables/grid_summary.csv !3_output/**/tables/v2_self_agreement_summary.csv AGENTS.md diff --git a/1_code/.gitkeep b/1_code/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/1_code/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/1_code/_deprecated/01_bootstrap_ids.R b/1_code/_deprecated/01_bootstrap_ids.R deleted file mode 100644 index 5899935..0000000 --- a/1_code/_deprecated/01_bootstrap_ids.R +++ /dev/null @@ -1,233 +0,0 @@ -# --- -# title: Generate Bootstrap Site Ids (Plant Groups) -# author: Brendan Casey -# created: 2026-09-08 -# inputs: -# set by the calling experiment, or defaulted in 1.2: -# - taxon, data_dir, run_dir, species_subset, boot_iter, -# n_clusters -# in data_dir: -# - .csv, sites.csv, covariates/, lookup/ -# in this repository: -# - 1_code/modules/plants/load_model_data.R -# - 1_code/modules/plants/functions/bootstrapping_functions.R -# outputs: -# in run_dir: -# - bootstrap/-bootstrap-ids.Rdata (bootstrap.ids) -# notes: -# - Rewrite of the v2 script 01b_bootstrapping.R, which held -# the same four blocks one after another in a single file and -# wrote back over the model-data files. This one takes the -# taxon as a parameter and writes only the bootstrap ids, so -# nothing in 0_data/ is modified by a run. -# - Each species gets its own set of ids because the thinning -# depends on where that species was detected. Iteration 1 is -# the complete data set by construction, and 2 onward -# resample sites within spatial blocks with replacement. -# - The block grid is the v2 grid: longitude cut at -121, -116, -# -112 and -109, latitude at 48, 51, 54, 57 and 61, with the -# interaction relabelled a, b, c and so on. -# - The ids are SiteYearQu values, which is what the -# hierarchical models and the validation both index on. -# - Bootstrapping resamples, so a run is only reproducible if -# the seed is fixed. v2 set none. `boot_seed` defaults to -# NULL, which reproduces v2 exactly; set it to make a run -# repeatable. -# - The species selection is `species_subset`, not `species`. -# The v2 code loops with `for (species in species.list)`, -# which would otherwise overwrite the selection and leave the -# next stage running on one species. -# - Future improvement - ids are generated for every modelled -# species even when a later stage runs on a subset, because -# the v2 code reads them as boot.data[[species]] and ignores -# extras. Passing `species` here narrows both. -# --- - -# 1. Setup ---- - -## 1.1 Load packages ---- -library(foreach) # sequential %dopar% loop, as in v2 (version: 1.5.2) -library(parallel) # species-level parallelism (version: 4.5.0) - -## 1.2 Configure the run ---- -# Values a calling experiment has already set are kept, so this -# script runs either from 1_code/experiments/ or on its own. -if (!exists("project_root", inherits = FALSE)) { - project_root <- normalizePath(getwd(), winslash = "/") -} - -if (!exists("taxon", inherits = FALSE)) { - taxon <- "bryophyte" -} - -if (!exists("data_dir", inherits = FALSE)) { - data_dir <- file.path(project_root, "0_data/test_dataset") -} - -if (!exists("run_dir", inherits = FALSE)) { - run_dir <- file.path(project_root, "2_pipeline/plants") -} - -# NULL generates ids for every species the snapshot declares as -# modelled. -if (!exists("species_subset", inherits = FALSE)) { - species_subset <- NULL -} - -# The v2 defaults. 01b used 16 cores where 02 used 14. -if (!exists("boot_iter", inherits = FALSE)) { - boot_iter <- 1:100 -} - -if (!exists("n_clusters", inherits = FALSE)) { - n_clusters <- 16 -} - -# Minimum detections for a region's model to be considered -# viable, as in v2. -if (!exists("boot_threshold", inherits = FALSE)) { - boot_threshold <- 20 -} - -# NULL leaves the stream unseeded, which is what v2 did. -if (!exists("boot_seed", inherits = FALSE)) { - boot_seed <- NULL -} - -## 1.3 Load functions and data ---- -source(file.path( - project_root, "1_code/modules/plants/load_model_data.R" -)) -source(file.path( - project_root, - "1_code/modules/plants/functions/bootstrapping_functions.R" -)) - -model_data <- load_plant_model_data( - taxon = taxon, - data_dir = data_dir, - species = species_subset -) - -climate.data <- model_data$climate.data - -species.list <- unique(c( - model_data$veg.species.list, - model_data$soil.species.list -)) - -## 1.4 Create the output folder ---- -# save() does not create a missing folder. -dir.create( - file.path(run_dir, "bootstrap"), - recursive = TRUE, showWarnings = FALSE -) - -# 2. Define the bootstrap blocks ---- -# Sites are resampled within coarse latitude and longitude -# blocks, so a bootstrap sample keeps the geographic spread of -# the original data rather than drifting toward whichever region -# happens to be best surveyed. - -## 2.1 Cut the study area into blocks ---- -site.block <- data.frame( - LongBlock = cut( - climate.data$Long, c(-121, -116, -112, -109) - ), - LatBlock = cut( - climate.data$Lat, c(48, 51, 54, 57, 61) - ) -) - -site.block$Block <- interaction( - droplevels(site.block$LongBlock), - droplevels(site.block$LatBlock), - sep = "::", - drop = TRUE -) - -## 2.2 Relabel the blocks ---- -# bootstrap_data() selects a block with grep(), so the labels -# have to be free of the regex metacharacters that the interval -# notation above is full of. -reclass.site <- data.frame( - Orig = unique(site.block$Block), - Update = letters[1:length(unique(site.block$Block))] -) - -climate.data$Block <- reclass.site$Update[ - match(site.block$Block, reclass.site$Orig) -] - -rm(site.block, reclass.site) - -# 3. Generate the bootstrap ids ---- -# One set of ids per species, because the viability checks inside -# bootstrap_data() depend on where that species was detected. - -## 3.1 Define cores and objects for parallel processing ---- -if (!is.null(boot_seed)) { - set.seed(boot_seed) -} - -core.input <- makeCluster(n_clusters) - -if (!is.null(boot_seed)) { - clusterSetRNGStream(core.input, boot_seed) -} - -clusterExport( - core.input, - c("climate.data", "species.list", "boot_iter", "boot_threshold", - "bootstrap_data") -) - -cat( - "Bootstrapping ", taxon, ": ", length(species.list), - " species x ", length(boot_iter), " iterations on ", - n_clusters, " cores\n", - sep = "" -) - -## 3.2 Loop through each species ---- -bootstrap.list <- foreach(species = species.list) %dopar% - - parSapply( - core.input, - as.list(boot_iter), - FUN = function(boot) { - tryCatch( - bootstrap_data( - data = climate.data, - species = species, - threshold = boot_threshold, - boot = boot - ), - error = function(e) e - ) - } - ) - -names(bootstrap.list) <- species.list - -stopCluster(core.input) - -## 3.3 Simplify to one array per species ---- -bootstrap.ids <- list() - -for (one_species in species.list) { - bootstrap.ids[[one_species]] <- simplify2array( - bootstrap.list[[one_species]] - ) -} - -# 4. Save the results ---- -bootstrap_path <- file.path( - run_dir, "bootstrap", paste0(taxon, "-bootstrap-ids.Rdata") -) - -save(bootstrap.ids, file = bootstrap_path) - -cat("Wrote ", bootstrap_path, "\n", sep = "") - -# End of script ---- diff --git a/1_code/_deprecated/02_hierarchical_models.R b/1_code/_deprecated/02_hierarchical_models.R deleted file mode 100644 index 7892984..0000000 --- a/1_code/_deprecated/02_hierarchical_models.R +++ /dev/null @@ -1,525 +0,0 @@ -# --- -# title: Fit Bootstrapped Hierarchical Models (Plant Groups) -# author: Brendan Casey -# created: 2026-09-08 -# inputs: -# set by the calling experiment, or defaulted in 1.2: -# - taxon, data_dir, run_dir, species_subset, boot_iter, -# n_clusters -# in data_dir: -# - .csv, sites.csv, covariates/, lookup/ -# in run_dir: -# - bootstrap/-bootstrap-ids.Rdata, from 01 -# in this repository: -# - 1_code/modules/plants/load_model_data.R -# - 1_code/modules/plants/functions/ -# hierarchical-model_functions.R -# outputs: -# in run_dir: -# - models/-species-models.Rdata -# (climate.coef, vegetation.coef, soil.coef) -# notes: -# - Rewrite of the v2 scripts 02a-02d, which were four -# near-identical files. They differed only in the taxon they -# loaded and in whether they fitted a Protocol term, so they -# are one script here, parameterized by `taxon`. The formula -# text in section 2 is the v2 text verbatim, and the Protocol -# term is inserted for the taxa whose v2 script carried it, -# which reproduces all 42 formulas of both variants exactly. -# - The v2 scripts read a -model-data.Rdata that 01a -# built against paths on the v2 project. This one reads the -# harmonized test dataset, so a run needs only this -# repository and 0_data/test_dataset/. -# - Output location is the caller's to choose. v2 hard-coded -# 3_output/models/; here run_dir comes from the experiment. -# - All three model sets are defined in section 2, before any -# of them is fitted. v2 defined each set immediately before -# fitting it, which meant a model naming a habitat type its -# prediction matrix had no column for failed only after the -# climate models had already run. -# - The species selection is `species_subset`, not `species`. -# The v2 code loops with `for (species in species.list)`, -# which would otherwise overwrite the selection and leave the -# next stage running on one species. -# - boot_iter and n_clusters were hard-coded in v2 as 1:100 and -# 14. They are parameters here, but they default to the v2 -# values, so a parity run sets neither. -# - The v2 code took its vegetation coefficient names -# positionally, as colnames(veg.data)[403:489] and three -# other per-taxon spans. load_model_data.R names them -# instead; see its header for why. -# - foreach() runs with no backend registered, exactly as in -# v2, so the outer %dopar% is sequential and the parallelism -# is the inner parSapply over bootstraps. Left as it is: -# registering a backend would change the run, not just its -# speed. -# - Future improvement - the three model sets differ only in -# their data, formulas and fitting function, so sections 3 to -# 5 could share one helper rather than repeating the cluster -# setup. -# --- - -# 1. Setup ---- - -## 1.1 Load packages ---- -library(foreach) # sequential %dopar% loop, as in v2 (version: 1.5.2) -library(parallel) # bootstrap-level parallelism (version: 4.5.0) - -## 1.2 Configure the run ---- -# Values a calling experiment has already set are kept, so this -# script runs either from 1_code/experiments/ or on its own. -if (!exists("project_root", inherits = FALSE)) { - project_root <- normalizePath(getwd(), winslash = "/") -} - -if (!exists("taxon", inherits = FALSE)) { - taxon <- "bryophyte" -} - -if (!exists("data_dir", inherits = FALSE)) { - data_dir <- file.path(project_root, "0_data/test_dataset") -} - -if (!exists("run_dir", inherits = FALSE)) { - run_dir <- file.path(project_root, "2_pipeline/plants") -} - -# NULL models every species the snapshot declares as modelled. -if (!exists("species_subset", inherits = FALSE)) { - species_subset <- NULL -} - -# The v2 defaults. Lower them for a smoke test. -if (!exists("boot_iter", inherits = FALSE)) { - boot_iter <- 1:100 -} - -if (!exists("n_clusters", inherits = FALSE)) { - n_clusters <- 14 -} - -## 1.3 Load functions and data ---- -source(file.path( - project_root, "1_code/modules/plants/load_model_data.R" -)) -source(file.path( - project_root, - "1_code/modules/plants/functions/hierarchical-model_functions.R" -)) - -model_data <- load_plant_model_data( - taxon = taxon, - data_dir = data_dir, - species = species_subset -) - -climate.data <- model_data$climate.data -veg.data <- model_data$veg.data -soil.data <- model_data$soil.data - -veg.species.list <- model_data$veg.species.list -soil.species.list <- model_data$soil.species.list -species.list <- unique(c(veg.species.list, soil.species.list)) - -## 1.4 Load the bootstrap ids ---- -# Written by 01_bootstrap_ids.R into the same run_dir. -bootstrap_path <- file.path( - run_dir, "bootstrap", paste0(taxon, "-bootstrap-ids.Rdata") -) - -if (!file.exists(bootstrap_path)) { - stop( - "No bootstrap ids at ", - bootstrap_path, - "\nRun 1_code/modules/plants/01_bootstrap_ids.R first.", - call. = FALSE - ) -} - -load(bootstrap_path) - -## 1.5 Set the Protocol term for this taxon ---- -# v2 fitted Protocol for bryophytes and lichens, and not for -# mites or vascular plants. plant_taxa() is the record of which. -use_protocol <- taxon_uses_protocol(taxon) - -# In v2 the two Protocol scripts were their non-Protocol -# counterparts with the term inserted directly after Climate, and -# nowhere else. Applying that rule to the formula text in section -# 2 reproduces both variants exactly. -# -# The environment is set to the global environment, matching v2, -# where every formula was built at the top level. -build_models <- function(model_text, protocol = use_protocol) { - lapply( - model_text, - function(one) { - if (protocol) { - one <- sub( - "pcount ~ Climate", - "pcount ~ Climate + Protocol", - one, - fixed = TRUE - ) - } - - as.formula(one, env = globalenv()) - } - ) -} - -# Prefix for the coefficient templates, for the same reason. -coef_prefix <- if (use_protocol) { - c("Intercept", "Climate", "Protocol") -} else { - c("Intercept", "Climate") -} - -## 1.6 Create the output folder ---- -# save() does not create a missing folder, so a run would -# otherwise fail after the modelling rather than before it. -dir.create( - file.path(run_dir, "models"), - recursive = TRUE, showWarnings = FALSE -) - -cat( - "Fitting ", taxon, ": ", length(species.list), " species x ", - length(boot_iter), " bootstraps on ", n_clusters, " cores\n", - sep = "" -) - -# 2. Define the model sets ---- -# Everything the three fitting sections need, built before any of -# them runs, so a mismatch between a model and its prediction -# matrix is found in seconds rather than after the climate models -# have been fitted. - -## 2.1 Define climate models ---- -# The climate formulas carry no Protocol term in either v2 -# variant, so they are built without one. -climate.model.text <- c( - "pcount ~ 1", - "pcount ~ PET", - "pcount ~ CMD", - "pcount ~ MAT", - "pcount ~ FFP", - "pcount ~ MAP + FFP", - "pcount ~ MAP + FFP + CMD", - "pcount ~ MAP + PET + CMD + MAPPET", - "pcount ~ MAT + MAP + CMD + CMDMAT", - "pcount ~ MAT + MAP", - "pcount ~ MWMT + TD", - "pcount ~ CMD + PET", - "pcount ~ MAT + MAT2 + MWMT + MWMT2", - "pcount ~ TD + FFP + MAT" -) - -climate.models <- build_models( - climate.model.text, protocol = FALSE -) - -bioclim.model.text <- c( - ".~.+ bio9 + bio15" -) - -bioclim.models <- build_models( - bioclim.model.text, protocol = FALSE -) - -## 2.2 Define spatial models ---- -space.model.text <- c( - ".~.+ Easting + Northing", - ".~.+ Easting + Northing + EastingNorthing", - ".~.+ Easting + Northing + Easting2 + Northing2 + EastingNorthing" -) - -space.models <- build_models(space.model.text, protocol = FALSE) - -## 2.3 Define the prediction matrices ---- -# Both are needed for the checks below, so unlike v2 they are -# held under separate names rather than one reused -# prediction.matrix. -veg.prediction.matrix <- model_data$veg.pm -rownames(veg.prediction.matrix) <- veg.prediction.matrix$VegType -veg.prediction.matrix <- veg.prediction.matrix[, -1] - -soil.prediction.matrix <- model_data$soil.pm -rownames(soil.prediction.matrix) <- soil.prediction.matrix$VegType -soil.prediction.matrix <- soil.prediction.matrix[, -1] - -## 2.4 Define vegetation models ---- -vegetation.model.text <- c( - "pcount ~ Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + BlackSpruce + TreedFen + TreedSwamp + GraminoidFen + ShrubbyFen + ShrubbyBog + ShrubbySwamp + Marsh + Grass + Shrub + CCWhiteSpruceR + CCWhiteSpruce1 + CCWhiteSpruce234 + CCPineR + CCPine1234 + CCDecidMixedR + CCDecidMixed1 + CCDecidMixed234 + HardLin + EnSeismic + EnSoftLin + TrSoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", - "pcount ~ Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + Peatland + Mineral + Grass + Shrub + CCWhiteSpruceR + CCWhiteSpruce1 + CCWhiteSpruce234 + CCPineR + CCPine1234 + CCDecidMixedR + CCDecidMixed1 + CCDecidMixed234 + HardLin + EnSeismic + EnSoftLin + TrSoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", - "pcount ~ Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + Bog + TreedFen + + NonTreedFen + TreedSwamp + ShrubbySwamp + Marsh + Grass + Shrub + CCWhiteSpruceR + CCWhiteSpruce1 + CCWhiteSpruce234 + CCPineR + CCPine1234 + CCDecidMixedR + CCDecidMixed1 + CCDecidMixed234 + HardLin + EnSeismic + EnSoftLin + TrSoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", - "pcount ~ Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + Bog + Fen + TreedSwamp + ShrubbySwamp + Marsh + Grass + Shrub + CCWhiteSpruceR + CCWhiteSpruce1 + CCWhiteSpruce234 + CCPineR + CCPine1234 + CCDecidMixedR + CCDecidMixed1 + CCDecidMixed234 + HardLin + EnSeismic + EnSoftLin + TrSoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", - "pcount ~ Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + Bog + Fen + Swamp + Marsh + GrassShrub + CCWhiteSprucePineR + CCWhiteSprucePine1 + CCWhiteSprucePine234 + CCDecidMixedR + CCDecidMixed1 + CCDecidMixed234 + HardLin + SoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", - "pcount ~ Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + Peatland + Mineral + GrassShrub + CCWhiteSprucePineR1234 + CCDecidMixedR1234 + HardLin + SoftLin + UrbIndWellsites + Crop + Pasture", - "pcount ~ Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + Lowland + GrassShrub + CCWhiteSprucePineR1234 + CCDecidMixedR1234 + SoftLin + Alien", - "pcount ~ Climate + Upland + BlackSpruce + TreedFen + TreedSwamp + GraminoidFen + ShrubbyFen + ShrubbyBog + ShrubbySwamp + Marsh + Grass + Shrub + CCR1234 + HardLin + EnSeismic + EnSoftLin + TrSoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", - "pcount ~ Climate + Upland + Bog + Fen + Swamp + Marsh + Grass + Shrub + CCR1234 + HardLin + EnSeismic + EnSoftLin + TrSoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", - "pcount ~ Climate + Upland + Peatland + Swamp + Marsh + GrassShrub + CCR1234 + HardLin + SoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", - "pcount ~ Climate + Upland + Peatland + Mineral + GrassShrub + CCR1234 + HardLin + SoftLin + UrbIndWellsites + Crop + Pasture", - "pcount ~ Climate + Upland + Lowland + GrassShrub + CCR1234 + SoftLin + Alien" -) - -vegetation.models <- build_models(vegetation.model.text) - -## 2.5 Define soil models ---- -soil.model.text <- c( - "pcount ~ Climate + Blowout + ClaySub + Loamy + RapidDrain + SandyLoam + ThinBreak + Other + UrbInd + Wellsites + Crop + TameP + RoughP + EnSoftLin + EnSeismic + TrSoftLin + HardLin", - "pcount ~ Climate + Blowout + ClaySubThin + Loamy + SandyRapid + Other + UrbInd+ Wellsites + Crop + TameP + RoughP + EnSoftLin + EnSeismic + TrSoftLin + HardLin", - "pcount ~ Climate + Productive + Nonproductive + Other + UrbInd + Wellsites + Crop + TameP + RoughP + EnSoftLin + EnSeismic + TrSoftLin + HardLin", - "pcount ~ Climate + Blowout + ClaySub + Loamy + RapidDrain + SandyLoam + ThinBreak + Other + UrbIndWellsites + Crop + Pasture + SoftLin + HardLin", - "pcount ~ Climate + Blowout + ClaySubThin + Loamy + SandyRapid + Other + UrbIndWellsites + Crop + Pasture + SoftLin + HardLin", - "pcount ~ Climate + Productive + Nonproductive + Other + UrbIndWellsites + Crop + Pasture + SoftLin + HardLin", - "pcount ~ Climate + Productive + Nonproductive + Other + UrbIndWellsites + Cult + SoftLin + HardLin", - "pcount ~ Climate + Productive + Nonproductive + Other + Alien + SoftLin", - "pcount ~ Climate + Blowout + ClaySub + Loamy + RapidDrain + SandyLoam + ThinBreak + Other + UrbInd + Wellsites + Crop + TameP + RoughP + EnSoftLin + EnSeismic + TrSoftLin + HardLin + paspen", - "pcount ~ Climate + Blowout + ClaySubThin + Loamy + SandyRapid + Other + UrbInd + Wellsites + Crop + TameP + RoughP + EnSoftLin + EnSeismic + TrSoftLin + HardLin + paspen", - "pcount ~ Climate + Productive + Nonproductive + Other + UrbInd + Wellsites + Crop + TameP + RoughP + EnSoftLin + EnSeismic + TrSoftLin + HardLin + paspen", - "pcount ~ Climate + Blowout + ClaySub + Loamy +RapidDrain + SandyLoam + ThinBreak + Other + UrbIndWellsites + Crop + Pasture + SoftLin + HardLin + paspen", - "pcount ~ Climate + Blowout + ClaySubThin + Loamy + SandyRapid + Other + UrbIndWellsites + Crop + Pasture + SoftLin + HardLin + paspen", - "pcount ~ Climate + Productive + Nonproductive + Other + UrbIndWellsites + Crop + Pasture + SoftLin + HardLin + paspen", - "pcount ~ Climate + Productive + Nonproductive + Other + UrbIndWellsites + Cult + SoftLin + HardLin + paspen", - "pcount ~ Climate + Productive + Nonproductive + Other + Alien + SoftLin + paspen" -) - -soil.models <- build_models(soil.model.text) - -## 2.6 Check the models against the prediction matrices ---- -# Every habitat term is predicted onto a column of its prediction -# matrix after fitting, so a term with no column cannot be -# predicted. In v2 that surfaced as `object '' not found` -# raised inside every bootstrap of every species. -pm_source_note <- paste0( - "\nThe matrices in ", data_dir, "/lookup/ are written from ", - "the\nsnapshot's veg.pm and soil.pm. The v2 scripts instead ", - "read\nveg-prediction-matrix-CC_2024.csv and\n", - "soil-prediction-matrix_2024.csv from the v2 project, which\n", - "carry the aggregate habitat columns the snapshot copies do\n", - "not. Set SDM_V2_LOOKUP to the folder holding those two files\n", - "and re-run 1_code/_setup/01_harmonize_model_ready_v2.R." -) - -check_prediction_terms( - vegetation.models, veg.prediction.matrix, "vegetation", - pm_source_note -) - -check_prediction_terms( - soil.models, soil.prediction.matrix, "soil", pm_source_note -) - -## 2.7 Define the coefficient templates ---- -# The vegetation template is the 87 habitat types, named by -# load_model_data.R rather than taken positionally. -veg.coef.names <- c(coef_prefix, model_data$veg_coef_names) -veg.coef.template <- rep(NA, length(veg.coef.names)) -names(veg.coef.template) <- veg.coef.names - -soil.coef.names <- c( - "paspen", - "Loamy", - "SandyLoam", - "ClaySub", - "RapidDrain", - "Blowout", - "ThinBreak", - "Other", - "EnSeismic", - "EnSoftLin", - "TrSoftLin", - "HardLin", - "UrbInd", - "Wellsites", - "Crop", - "TameP", - "RoughP" -) - -soil.coef.template.names <- c(coef_prefix, soil.coef.names) -soil.coef.template <- rep( - NA, length(soil.coef.template.names) -) -names(soil.coef.template) <- soil.coef.template.names - -# 3. Fit the climate models ---- -# Run over every bootstrap iteration on the spatially thinned -# data. The fitted climate coefficients feed both habitat model -# sets, so this section runs first. - -## 3.1 Define cores and objects for parallel processing ---- -core.input <- makeCluster(n_clusters) -clusterExport( - core.input, - c("climate.data", "species.list", "bootstrap.ids", - "climate.models", "bioclim.models", "space.models", - "boot_iter", "climate_models") -) -clusterEvalQ(core.input, { - - # Load relevant libraries - library(AICcmodavg) # Model averaging - library(arm) # Allows for the use of bayesglm function - library(binom) # For exact binomial confidence intervals - library(mapproj) # For projected maps - library(mgcv) # For binomial GAM - library(MuMIn) - library(pROC) - library(RcmdrMisc) - -}) - -## 3.2 Loop through each species ---- -climate.coef <- foreach(species = species.list) %dopar% - - t(parSapply( - core.input, - as.list(boot_iter), - FUN = function(boot) { - tryCatch( - climate_models( - species = species, - data = climate.data, - boot.data = bootstrap.ids, - climate.models = climate.models, - bioclim.models = bioclim.models, - space.models = space.models, - boot = boot - ), - error = function(e) e - ) - } - )) - -names(climate.coef) <- species.list - -stopCluster(core.input) - -# 4. Fit the vegetation models ---- -# The north model set, fitted for veg.species.list. - -## 4.1 Define cores and objects for parallel processing ---- -# use_protocol is read inside the worker closure, so it has to be -# exported. v2 passed a literal TRUE or FALSE and did not. -core.input <- makeCluster(n_clusters) -clusterExport( - core.input, - c("veg.data", "species.list", "bootstrap.ids", - "vegetation.models", "veg.prediction.matrix", "climate.coef", - "boot_iter", "veg.coef.template", "use_protocol", - "vegetation_models") -) -clusterEvalQ(core.input, { - - # Load relevant libraries - library(AICcmodavg) # Model averaging - library(arm) # Allows for the use of bayesglm function - library(binom) # For exact binomial confidence intervals - library(mapproj) # For projected maps - library(mgcv) # For binomial GAM - library(MuMIn) - library(pROC) - library(RcmdrMisc) - -}) - -## 4.2 Loop through each species ---- -vegetation.coef <- foreach(species = veg.species.list) %dopar% - - t(parSapply( - core.input, - as.list(boot_iter), - FUN = function(boot) { - tryCatch( - vegetation_models( - species = species, - data = veg.data, - boot.data = bootstrap.ids, - habitat.models = vegetation.models, - prediction.matrix = veg.prediction.matrix, - climate.coef = climate.coef, - coef.template = veg.coef.template, - weight.method = "IVW", - coef.adjust = TRUE, - protocol.flag = use_protocol, - boot = boot - ), - error = function(e) e - ) - } - )) - -names(vegetation.coef) <- veg.species.list - -stopCluster(core.input) - -# 5. Fit the soil models ---- -# The south model set, fitted for soil.species.list. A species -# can be common enough to model in one region and not the other, -# so the two lists differ. - -## 5.1 Define cores and objects for parallel processing ---- -core.input <- makeCluster(n_clusters) -clusterExport( - core.input, - c("soil.data", "species.list", "bootstrap.ids", "soil.models", - "soil.prediction.matrix", "climate.coef", "boot_iter", - "soil.coef.template", "use_protocol", "soil_models") -) -clusterEvalQ(core.input, { - - # Load relevant libraries - library(AICcmodavg) # Model averaging - library(arm) # Allows for the use of bayesglm function - library(binom) # For exact binomial confidence intervals - library(mapproj) # For projected maps - library(mgcv) # For binomial GAM - library(MuMIn) - library(pROC) - library(RcmdrMisc) - -}) - -## 5.2 Loop through each species ---- -soil.coef <- foreach(species = soil.species.list) %dopar% - - t(parSapply( - core.input, - as.list(boot_iter), - FUN = function(boot) { - tryCatch( - soil_models( - species = species, - data = soil.data, - boot.data = bootstrap.ids, - habitat.models = soil.models, - prediction.matrix = soil.prediction.matrix, - climate.coef = climate.coef, - coef.template = soil.coef.template, - weight.method = "IVW", - coef.adjust = TRUE, - protocol.flag = use_protocol, - boot = boot - ), - error = function(e) e - ) - } - )) - -names(soil.coef) <- soil.species.list - -stopCluster(core.input) - -# 6. Save the results from all three model sets ---- -models_path <- file.path( - run_dir, "models", paste0(taxon, "-species-models.Rdata") -) - -save(climate.coef, soil.coef, vegetation.coef, file = models_path) - -cat("Wrote ", models_path, "\n", sep = "") - -# End of script ---- diff --git a/1_code/_deprecated/03_model_validation.R b/1_code/_deprecated/03_model_validation.R deleted file mode 100644 index e5eff25..0000000 --- a/1_code/_deprecated/03_model_validation.R +++ /dev/null @@ -1,285 +0,0 @@ -# --- -# title: Validate Bootstrapped Models (Plant Groups) -# author: Brendan Casey -# created: 2026-09-08 -# inputs: -# set by the calling experiment, or defaulted in 1.2: -# - taxon, data_dir, run_dir, species_subset, boot_iter, -# n_clusters -# in data_dir: -# - .csv, sites.csv, covariates/, lookup/ -# in run_dir: -# - bootstrap/-bootstrap-ids.Rdata, from 01 -# - models/-species-models.Rdata, from 02 -# in this repository: -# - 1_code/modules/plants/load_model_data.R -# - 1_code/modules/plants/functions/ -# model-validation_functions.R -# outputs: -# in run_dir: -# - validation/-models-validation.Rdata -# (vegetation.fit, soil.fit) -# notes: -# - Rewrite of the v2 scripts 03a-03d, which were four -# near-identical files differing only in the taxon they -# loaded and in protocol.flag. They are one script here, -# parameterized by `taxon`, with the flag read from -# plant_taxa() so it cannot drift from what 02 fitted. -# - The species queue comes from the fitted model object rather -# than from the data, exactly as v2 did, so validation covers -# what was actually fitted even when 02 ran on a subset. -# - NA coefficients are set to qlogis(0.001) before scoring and -# the standard error columns are dropped, which is v2's -# handling of a bootstrap that failed to converge: an -# unestimated coefficient is scored as near-zero probability -# rather than dropping the iteration. -# - v2's 03a header names its output byrophyte-...Rdata while -# the code writes bryophyte-...Rdata. The name is built from -# `taxon` here, so the two cannot disagree. -# - The species selection is `species_subset`, not `species`. -# The v2 code loops with `for (species in species.list)`, -# which would otherwise overwrite the selection and leave the -# next stage running on one species. -# - Future improvement - the two model sets differ only in -# their data, coefficients and landcover.type, so they could -# share one helper rather than repeating the cluster setup. -# --- - -# 1. Setup ---- - -## 1.1 Load packages ---- -library(foreach) # sequential %dopar% loop, as in v2 (version: 1.5.2) -library(parallel) # bootstrap-level parallelism (version: 4.5.0) - -## 1.2 Configure the run ---- -# Values a calling experiment has already set are kept, so this -# script runs either from 1_code/experiments/ or on its own. -if (!exists("project_root", inherits = FALSE)) { - project_root <- normalizePath(getwd(), winslash = "/") -} - -if (!exists("taxon", inherits = FALSE)) { - taxon <- "bryophyte" -} - -if (!exists("data_dir", inherits = FALSE)) { - data_dir <- file.path(project_root, "0_data/test_dataset") -} - -if (!exists("run_dir", inherits = FALSE)) { - run_dir <- file.path(project_root, "2_pipeline/plants") -} - -if (!exists("species_subset", inherits = FALSE)) { - species_subset <- NULL -} - -# The v2 defaults. -if (!exists("boot_iter", inherits = FALSE)) { - boot_iter <- 1:100 -} - -if (!exists("n_clusters", inherits = FALSE)) { - n_clusters <- 14 -} - -## 1.3 Load functions and data ---- -source(file.path( - project_root, "1_code/modules/plants/load_model_data.R" -)) -source(file.path( - project_root, - "1_code/modules/plants/functions/model-validation_functions.R" -)) - -model_data <- load_plant_model_data( - taxon = taxon, - data_dir = data_dir, - species = species_subset -) - -veg.data <- model_data$veg.data -soil.data <- model_data$soil.data - -use_protocol <- taxon_uses_protocol(taxon) - -## 1.4 Load the bootstrap ids and the fitted models ---- -bootstrap_path <- file.path( - run_dir, "bootstrap", paste0(taxon, "-bootstrap-ids.Rdata") -) - -models_path <- file.path( - run_dir, "models", paste0(taxon, "-species-models.Rdata") -) - -missing_inputs <- c(bootstrap_path, models_path) -missing_inputs <- missing_inputs[!file.exists(missing_inputs)] - -if (length(missing_inputs) > 0) { - stop( - "Validation inputs not found:\n ", - paste(missing_inputs, collapse = "\n "), - "\nRun 01_bootstrap_ids.R and 02_hierarchical_models.R first.", - call. = FALSE - ) -} - -load(bootstrap_path) -load(models_path) - -## 1.5 Take the species queue from the fitted models ---- -# Not from the data. 02 may have run on a subset, and only what -# it fitted can be scored. -species.list <- names(climate.coef) -veg.species.list <- names(vegetation.coef) -soil.species.list <- names(soil.coef) - -dir.create( - file.path(run_dir, "validation"), - recursive = TRUE, showWarnings = FALSE -) - -cat( - "Validating ", taxon, ": ", length(veg.species.list), - " vegetation and ", length(soil.species.list), - " soil species x ", length(boot_iter), " bootstraps\n", - sep = "" -) - -## 1.6 Prepare coefficients for model fit evaluation ---- -# A bootstrap that failed to estimate a coefficient leaves NA. -# v2 scores those as qlogis(0.001), a near-zero probability, -# rather than dropping the iteration, and drops the standard -# error columns, which are not used in scoring. -for (one_species in veg.species.list) { - - climate.coef[[one_species]][ - is.na(climate.coef[[one_species]]) - ] <- qlogis(0.001) - - vegetation.coef[[one_species]][ - is.na(vegetation.coef[[one_species]]) - ] <- qlogis(0.001) - vegetation.coef[[one_species]] <- vegetation.coef[[one_species]][ - , -grep(".SE", colnames(vegetation.coef[[one_species]])) - ] - -} - -for (one_species in soil.species.list) { - - climate.coef[[one_species]][ - is.na(climate.coef[[one_species]]) - ] <- qlogis(0.001) - - soil.coef[[one_species]][ - is.na(soil.coef[[one_species]]) - ] <- qlogis(0.001) - soil.coef[[one_species]] <- soil.coef[[one_species]][ - , -grep(".SE", colnames(soil.coef[[one_species]])) - ] - -} - -# 2. Vegetation model fit ---- -# Scores the north model set per species per bootstrap, on AUC -# and the related measures model_validation() returns. - -## 2.1 Define cores and objects for parallel processing ---- -core.input <- makeCluster(n_clusters) -clusterExport( - core.input, - c("veg.data", "veg.species.list", "bootstrap.ids", - "vegetation.coef", "climate.coef", "boot_iter", - "use_protocol", "model_validation") -) -clusterEvalQ(core.input, { - - # Load relevant libraries - library(pROC) - -}) - -## 2.2 Loop through each species ---- -vegetation.fit <- foreach(species = veg.species.list) %dopar% - - t(parSapply( - core.input, - as.list(boot_iter), - FUN = function(boot) { - tryCatch( - model_validation( - species = species, - data = veg.data, - boot.data = bootstrap.ids, - climate.coef = climate.coef, - landcover.coef = vegetation.coef, - landcover.type = "Vegetation", - protocol.flag = use_protocol, - boot = boot - ), - error = function(e) e - ) - } - )) - -names(vegetation.fit) <- veg.species.list - -stopCluster(core.input) - -# 3. Soil model fit ---- -# The same scoring for the south model set. - -## 3.1 Define cores and objects for parallel processing ---- -core.input <- makeCluster(n_clusters) -clusterExport( - core.input, - c("soil.data", "soil.species.list", "bootstrap.ids", - "soil.coef", "climate.coef", "boot_iter", "use_protocol", - "model_validation") -) -clusterEvalQ(core.input, { - - # Load relevant libraries - library(pROC) - -}) - -## 3.2 Loop through each species ---- -soil.fit <- foreach(species = soil.species.list) %dopar% - - t(parSapply( - core.input, - as.list(boot_iter), - FUN = function(boot) { - tryCatch( - model_validation( - species = species, - data = soil.data, - boot.data = bootstrap.ids, - climate.coef = climate.coef, - landcover.coef = soil.coef, - landcover.type = "Soil", - protocol.flag = use_protocol, - boot = boot - ), - error = function(e) e - ) - } - )) - -names(soil.fit) <- soil.species.list - -stopCluster(core.input) - -# 4. Save the results ---- -validation_path <- file.path( - run_dir, "validation", - paste0(taxon, "-models-validation.Rdata") -) - -save(vegetation.fit, soil.fit, file = validation_path) - -cat("Wrote ", validation_path, "\n", sep = "") - -# End of script ---- diff --git a/1_code/_deprecated/README.md b/1_code/_deprecated/README.md deleted file mode 100644 index 3f46efc..0000000 --- a/1_code/_deprecated/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# _deprecated - -Code superseded by the framework described in -[`docs/framework_design.md`](../../docs/framework_design.md). Kept for -reference during the transition, and not sourced by anything. - -Delete once the taxon specs in `1_code/modules/` reproduce v2 through the -new harness. - -## What is here and why - -| File | Superseded by | Note | -| --- | --- | --- | -| `01_bootstrap_ids.R` | `harness/resample.R` | Plant-only bootstrap generation | -| `02_hierarchical_models.R` | `harness/engines.R`, `harness/selection.R` | Plant-only two-stage fitting | -| `03_model_validation.R` | `harness/eval_metrics.R` | Plant-only scoring | -| `load_model_data.R` | `harness/data_load.R` | Read `covariates/_*.csv`, a layout the dataset no longer has | -| `covar_attach.R`, `data_load.R`, `data_split.R`, `eval_metrics.R` | the harness files of the same names | Placeholders that stopped rather than returning anything | -| `plants_functions/` | — | Byte-identical copies of `0_data/v2_scripts/plants/` | -| `exp_000_parity_v2/` | `1_code/experiments/exp_000_parity_v2/` | The gate rebuilt on the harness; the old version drove the deleted plants module | - -Two of these are worth knowing about. - -**The plants module was already broken.** `load_model_data.R` reads -`covariates/_climate.csv` and its siblings. The harmonizer now -writes one `covariates.csv` keyed on taxon instead, so that module could -not have run against the current test dataset regardless of the rewrite. - -**`plants_functions/` was pure duplication.** All three files were -byte-identical to the frozen originals in `0_data/v2_scripts/plants/`, -which are tracked and have recorded provenance. A second copy could only -drift. When the plant spec needs `climate_models()`, `vegetation_models()` -or `model_validation()` for parity, it should source the frozen originals -rather than a copy. - -## plants_module/spec.R - -The combined plant-group spec, superseded on 2026-09-10 when the -four plant-group taxa were split into their own modules: -`bryophytes/`, `lichens/`, `soil_mites/` and `vascular_plants/`. -Its `plant_spec(taxon)` became `plant_group_spec()` in -`1_code/modules/_shared/plant_group.R`, called by four thin taxon -files that each state only what is true of that taxon. - -Kept because it is what the split was verified against: every one -of the eight taxon-by-region model frames built from -`0_data/test_dataset` is identical between the old spec and the -new ones. The spec objects themselves differ only in the -environments captured by the region-filter formulas, which the -harness discards, evaluating the right-hand side against the data -frame instead. diff --git a/1_code/_deprecated/covar_attach.R b/1_code/_deprecated/covar_attach.R deleted file mode 100644 index 975c60a..0000000 --- a/1_code/_deprecated/covar_attach.R +++ /dev/null @@ -1,46 +0,0 @@ -# --- -# title: Attach Covariates to Records -# author: Brendan Casey -# created: 2026-09-05 -# inputs: none -# outputs: none -# notes: -# - Placeholder. 0_data/covariates/catalogue_v1.csv holds -# metadata and server paths only; the rasters live on the -# ABMI server. -# - The catalogue is extensible - a new catalogue version can -# add covariates without invalidating the test dataset. -# --- - -# 1. attach_covariates() ---- - -#' Attach Covariates to Records -#' -#' Attaches covariates from the versioned catalogue to records. -#' -#' @param data Data frame of response records with -#' coordinates. -#' @param catalogue Character. Catalogue version, e.g. -#' "catalogue_v1". -#' @param covariates Character vector of covariate names to -#' attach, or NULL for every covariate in the catalogue. -#' @return `data` with one column per attached covariate. -#' -#' @example -#' # Example usage of the function -#' # attach_covariates() -attach_covariates <- function( - data, - catalogue = "catalogue_v1", - covariates = NULL -) { - # Step 1: Read the catalogue and resolve the layer paths - # Step 2: Extract at the record coordinates and bind - - stop( - "attach_covariates() is not implemented yet.", - call. = FALSE - ) -} - -# End of script ---- diff --git a/1_code/_deprecated/data_load.R b/1_code/_deprecated/data_load.R deleted file mode 100644 index b294470..0000000 --- a/1_code/_deprecated/data_load.R +++ /dev/null @@ -1,39 +0,0 @@ -# --- -# title: Load the Frozen Test Dataset -# author: Brendan Casey -# created: 2026-09-05 -# inputs: none -# outputs: none -# notes: -# - Placeholder. The frozen test dataset is not yet -# assembled, so this stops rather than returning -# a partial answer. -# - 0_data/manifest.md is the source of truth for the -# dataset version and paths. -# --- - -# 1. load_test_data() ---- - -#' Load the Frozen Test Dataset -#' -#' Loads the frozen cross-taxa test dataset for one taxon. -#' -#' @param taxon Character. Taxon slug (e.g. "mite"). -#' @param version Character. Dataset version from -#' 0_data/manifest.md. -#' @return A data frame of response records. -#' -#' @example -#' # Example usage of the function -#' # load_test_data() -load_test_data <- function(taxon, version = NULL) { - # Step 1: Resolve the dataset path from the manifest - # Step 2: Read the records and validate the columns - - stop( - "load_test_data() is not implemented yet.", - call. = FALSE - ) -} - -# End of script ---- diff --git a/1_code/_deprecated/data_split.R b/1_code/_deprecated/data_split.R deleted file mode 100644 index 3a3201f..0000000 --- a/1_code/_deprecated/data_split.R +++ /dev/null @@ -1,39 +0,0 @@ -# --- -# title: Apply the Evaluation Split -# author: Brendan Casey -# created: 2026-09-05 -# inputs: none -# outputs: none -# notes: -# - Placeholder. The split is part of the frozen dataset, -# so it is read, never recomputed - recomputing it would -# break comparability across experiments. -# - The v2 pipeline does not use this; the v2 scripts carry -# their own bootstrap design in bootstrap.ids. -# --- - -# 1. apply_split() ---- - -#' Apply the Evaluation Split -#' -#' Applies the frozen evaluation split to a set of records. -#' -#' @param data Data frame of response records. -#' @param split Character. Which split to return, one of -#' "train", "test", or "all". -#' @return A data frame holding the requested split. -#' -#' @example -#' # Example usage of the function -#' # apply_split() -apply_split <- function(data, split = c("train", "test", "all")) { - # Step 1: Match the requested split - # Step 2: Join the frozen split assignment and subset - - stop( - "apply_split() is not implemented yet.", - call. = FALSE - ) -} - -# End of script ---- diff --git a/1_code/_deprecated/eval_metrics.R b/1_code/_deprecated/eval_metrics.R deleted file mode 100644 index 4b54222..0000000 --- a/1_code/_deprecated/eval_metrics.R +++ /dev/null @@ -1,41 +0,0 @@ -# --- -# title: Compute Evaluation Metrics -# author: Brendan Casey -# created: 2026-09-05 -# inputs: none -# outputs: none -# notes: -# - Placeholder. Metrics computed here are what experiments -# are compared on, so changes affect every past and future -# experiment and should be reviewed accordingly. -# - The v2 validation scripts compute their own AUC through -# model_validation(); this is for methods work that does -# not run the v2 code. -# --- - -# 1. eval_metrics() ---- - -#' Compute Evaluation Metrics -#' -#' Computes the shared evaluation metrics for a model's predictions. -#' -#' @param observed Numeric vector of observed values. -#' @param predicted Numeric vector of predicted values. -#' @param metrics Character vector of metric names to -#' compute, or NULL for the full set. -#' @return A named numeric vector of metric values. -#' -#' @example -#' # Example usage of the function -#' # eval_metrics() -eval_metrics <- function(observed, predicted, metrics = NULL) { - # Step 1: Validate that the two vectors align - # Step 2: Compute each requested metric - - stop( - "eval_metrics() is not implemented yet.", - call. = FALSE - ) -} - -# End of script ---- diff --git a/1_code/_deprecated/exp_000_parity_v2/01_collect_results.R b/1_code/_deprecated/exp_000_parity_v2/01_collect_results.R deleted file mode 100644 index 929dfe9..0000000 --- a/1_code/_deprecated/exp_000_parity_v2/01_collect_results.R +++ /dev/null @@ -1,63 +0,0 @@ -# --- -# title: Collect v2 Model Output (Experiment 000) -# author: Brendan Casey -# created: 2026-09-05 -# inputs: -# in pipeline_dir//: -# - models/-species-models.Rdata -# - validation/-models-validation.Rdata -# outputs: -# in out_dir/tables/: -# - species_.csv -# - validation_summary.csv -# notes: -# - Placeholder. Reads the model output each taxon left in -# pipeline_dir and writes flat summaries to out_dir. -# - Raw model objects stay in pipeline_dir on purpose. They are -# 100 bootstraps by every species by three model families, -# which is too large to commit; only the derived summaries -# belong in 3_output/. -# - Expects exp_id, pipeline_dir and out_dir from run.R. Both -# paths are taken from there rather than rebuilt here, so a -# run pointed at a scratch disk is still read correctly. -# --- - -# 1. Setup ---- - -## 1.1 Resolve paths ---- -# Sourced by run.R, so the experiment configuration is already -# in the environment. -tables_dir <- file.path(out_dir, "tables") -dir.create(tables_dir, recursive = TRUE, showWarnings = FALSE) - -## 1.2 Identify the taxa that produced output ---- -# A taxon appears here only once its stages have run into this -# pipeline_dir, in this pass or an earlier one. -run_taxa <- list.dirs( - pipeline_dir, full.names = FALSE, recursive = FALSE -) -run_taxa <- setdiff(run_taxa, "logs") - -# 2. Collect model coefficients ---- -# TODO: load each -species-models.Rdata, reduce the -# bootstrap coefficient arrays to per-species summaries, and -# write one table per taxon. - -# 3. Collect validation metrics ---- -# TODO: load each -models-validation.Rdata and write the -# per-species AUC and fit measures to validation_summary.csv. - -# 4. Report what was found ---- -cat( - "01_collect_results.R is a placeholder. Taxa found in ", - pipeline_dir, ": ", - if (length(run_taxa) > 0) { - paste(run_taxa, collapse = ", ") - } else { - "none" - }, - "\n", - sep = "" -) - -# End of script ---- diff --git a/1_code/_deprecated/exp_000_parity_v2/02_compare_to_v2.R b/1_code/_deprecated/exp_000_parity_v2/02_compare_to_v2.R deleted file mode 100644 index fbb1292..0000000 --- a/1_code/_deprecated/exp_000_parity_v2/02_compare_to_v2.R +++ /dev/null @@ -1,52 +0,0 @@ -# --- -# title: Compare Against the v2.0 Reference (Experiment 000) -# author: Brendan Casey -# created: 2026-09-05 -# inputs: -# in 3_output/exp_000_parity_v2/tables/: -# - species_.csv -# - validation_summary.csv -# reference: -# - the published v2.0 model output (source to be agreed) -# outputs: -# in 3_output/exp_000_parity_v2/: -# - tables/parity_.csv -# - figures/parity_.png -# notes: -# - Placeholder. This is the parity gate - it compares what -# this repository produced against the species models v2.0 -# results for the same species. -# - Open question: where the v2.0 reference output comes from. -# Either a frozen snapshot under 0_data/ with a pointer in -# manifest.md, or the external v2 project. The comparison -# cannot be written until that is settled. -# - The minimum acceptable parity target is not yet defined and -# will need to be agreed per taxon. -# - Expects exp_id, project_root, and out_dir from run.R. -# --- - -# 1. Setup ---- - -## 1.1 Resolve paths ---- -tables_dir <- file.path(out_dir, "tables") - -figures_dir <- file.path(out_dir, "figures") -dir.create(figures_dir, recursive = TRUE, showWarnings = FALSE) - -# 2. Compare coefficients ---- -# TODO: join this run's per-species coefficients to the v2.0 -# reference and record the differences per model family. - -# 3. Compare validation metrics ---- -# TODO: compare AUC and fit measures per species and bootstrap. - -# 4. Summarize parity ---- -# TODO: write parity_.csv and the matching figure, then -# state the result against the agreed target. -cat( - "02_compare_to_v2.R is a placeholder. The v2.0 reference ", - "source and the parity target are still to be agreed.\n", - sep = "" -) - -# End of script ---- diff --git a/1_code/_deprecated/exp_000_parity_v2/03_build_report.R b/1_code/_deprecated/exp_000_parity_v2/03_build_report.R deleted file mode 100644 index 71e50d0..0000000 --- a/1_code/_deprecated/exp_000_parity_v2/03_build_report.R +++ /dev/null @@ -1,50 +0,0 @@ -# --- -# title: Build the Experiment 000 Report -# author: Brendan Casey -# created: 2026-09-05 -# inputs: -# in 3_output/exp_000_parity_v2/: -# - tables/parity_.csv -# - figures/parity_.png -# outputs: -# in 3_output/exp_000_parity_v2/: -# - report.md -# notes: -# - Placeholder. Assembles the tables and figures into the -# committed report that states whether the parity gate -# passed, per taxon. -# - Writing a stub report.md on every run keeps the experiment -# folder complete, so a partial run is still legible. -# - Expects exp_id, project_root, and out_dir from run.R. -# --- - -# 1. Setup ---- - -## 1.1 Resolve paths ---- -report_file <- file.path(out_dir, "report.md") - -dir.create(out_dir, recursive = TRUE, showWarnings = FALSE) - -# 2. Assemble the report ---- -# TODO: read the parity tables and figures and write the results -# section. For now the file records that the run happened and -# which species vector produced it. -report_lines <- c( - paste0("# ", exp_id), - "", - paste0("Generated ", format(Sys.time(), "%Y-%m-%d %H:%M:%S")), - "", - "## Status", - "", - "Placeholder report. The parity comparison in", - "`02_compare_to_v2.R` is not yet implemented, so no result is", - "stated here yet.", - "" -) - -# 3. Write the report ---- -writeLines(report_lines, report_file) - -cat("Wrote ", report_file, "\n", sep = "") - -# End of script ---- diff --git a/1_code/_deprecated/exp_000_parity_v2/README.md b/1_code/_deprecated/exp_000_parity_v2/README.md deleted file mode 100644 index 2a2cc59..0000000 --- a/1_code/_deprecated/exp_000_parity_v2/README.md +++ /dev/null @@ -1,97 +0,0 @@ -# exp_000_parity_v2 - -![Status](https://img.shields.io/badge/Status-Partially%20running-yellow) -![Languages](https://img.shields.io/badge/Languages-R-blue) - -The validation gate. Runs the v2.0 models through this repository's -pipeline and compares the results against the published species models -v2.0 outputs. - -## Question - -Does running the rewritten v2 modelling code, on the harmonized test -dataset, through this repository's pipeline reproduce the v2.0 results? - -Everything downstream depends on the answer. Later experiments are -compared against this one, so a difference here is a difference in the -plumbing, not in the method being tested. - -## Design - -| Element | Value | -| --- | --- | -| Taxa | Set per run in `run.R` section 1.4 (`mites <- TRUE`, and so on) | -| Species | Set per taxon in `utils/species_lists.R`; `NULL` runs all | -| Bootstraps | `boot_iter` in `run.R`; must stay at the v2 value of `1:100` for parity | -| Reference | The published v2.0 output — source still to be agreed | -| Target | Not yet defined; to be agreed per taxon | - -## Files - -| File | Purpose | -| --- | --- | -| `run.R` | Entry point. Paths, taxon flags, stage flags, run length | -| `utils/species_lists.R` | The species vector for each taxon | -| `01_collect_results.R` | Model output → flat summary tables | -| `02_compare_to_v2.R` | The parity comparison itself | -| `03_build_report.R` | Tables and figures → `report.md` | - -## How to run - -1. Build the test dataset first, if it is not already there — see - [Setup](../../../README.md#setup) in the repository README. A run needs - only `0_data/test_dataset/`, not the v2 project. -2. Edit `run.R`: - - **1.3** where the run reads and writes. The defaults follow the - repository conventions; point `pipeline_dir` at a scratch disk for a - long run. - - **1.4** the taxon flags. - - **1.5** the stage flags. `02` needs what `01` wrote and `03` needs what - `02` wrote, so a stage can be run alone only once the ones before it - have run into the same `pipeline_dir`. - - **1.6** the run length. Leave it at the v2 values for parity. -3. Name the species in `utils/species_lists.R`. `NULL` runs every modelled - species; a short vector is the smoke test, and is the right way to - shorten a trial run since `boot_iter` has to stay at `1:100`. -4. Run from the repository root: - -```r -source("1_code/experiments/exp_000_parity_v2/run.R") -``` - -## Outputs - -Intermediates, per taxon, in `pipeline_dir` -(`2_pipeline/exp_000_parity_v2/` by default): - -- `/bootstrap/` — the per-species bootstrap site ids. -- `/models/` — the fitted climate, vegetation and soil - coefficients. Gitignored; 100 bootstraps by every species by three - model families is too large to commit. -- `/validation/` — the per-bootstrap fit measures. -- `logs/` — one timestamped log per stage. - -Deliverables in `out_dir` (`3_output/exp_000_parity_v2/` by default): - -- `tables/`, `figures/`, `report.md` — committed, so comparisons - across experiments can be made from the repository alone. - -## Status - -The climate and soil model sets run end to end. Three things are -outstanding: - -- **The vegetation models cannot be fitted.** The prediction matrix in the - snapshot is missing four columns the v2 vegetation models are predicted - onto. See [Known gaps](../../../README.md#known-gaps) — it is one file - away from working, and `02_hierarchical_models.R` stops with the missing - columns named rather than failing partway through a run. -- **The v2.0 reference source** is undecided: a frozen snapshot under - `0_data/` or the external v2 project. -- **The parity target** is undefined, per taxon. - -`01`–`03` are placeholders pending the last two. - -Note that the v2 bootstrap draws without a fixed seed, so successive runs -of identical code differ. `boot_seed` in `run.R` fixes the stream when a -run needs to be repeatable; `NULL` reproduces v2. diff --git a/1_code/_deprecated/exp_000_parity_v2/run.R b/1_code/_deprecated/exp_000_parity_v2/run.R deleted file mode 100644 index d344c7a..0000000 --- a/1_code/_deprecated/exp_000_parity_v2/run.R +++ /dev/null @@ -1,253 +0,0 @@ -# --- -# title: Run Experiment 000 - v2.0 Parity Check -# author: Brendan Casey -# created: 2026-09-05 -# inputs: -# harness, in 1_code/harness/utils/: -# - step_runner.R -# experiment: -# - 1_code/experiments/exp_000_parity_v2/utils/species_lists.R -# modules, in 1_code/modules/plants/: -# - 01_bootstrap_ids.R -# - 02_hierarchical_models.R -# - 03_model_validation.R -# data: -# - 0_data/test_dataset/ -# outputs: -# in pipeline_dir (2_pipeline/exp_000_parity_v2/ by default): -# - /bootstrap/, /models/, /validation/ -# - logs/ -# in out_dir (3_output/exp_000_parity_v2/ by default): -# - tables/, figures/, report.md -# notes: -# - Entry point for the experiment, and the only file that has -# to be edited to change what runs. Sections 1.2 to 1.6 are -# the whole configuration: where output goes, which taxa run, -# on which species, and at what run length. -# - The modelling scripts in 1_code/modules/plants/ take every -# path as an argument. Nothing is hard-coded to a project -# layout, so pipeline_dir and out_dir can point anywhere -# writable; the defaults follow this repository's -# conventions. -# - Reads 0_data/test_dataset/ only, which is the harmonized -# snapshot 1_code/_setup/ builds. The v2 project itself is -# not needed at run time, and 0_data/ is not written to. -# - Species vectors live in utils/species_lists.R. NULL models -# every species the snapshot declares as modelled. -# - The three stages are ordered: 02 reads what 01 wrote and 03 -# reads what 02 wrote, all from pipeline_dir. Re-running one -# stage means setting its flag alone, so long as the earlier -# stages have run at least once. -# --- - -# 1. Setup ---- - -## 1.1 Set the working directory ---- -# Every path below is relative to this repository's root. In -# RStudio or Positron, uncomment to set it from this script's -# location. -# setwd(dirname(rstudioapi::getActiveDocumentContext()$path)) -# setwd("../../..") - -## 1.2 Configure the experiment ---- -# The identifier is used verbatim in 1_code/experiments/, -# 2_pipeline/ and 3_output/, so all three paths derive from one -# variable rather than being typed per stage. -exp_id <- "exp_000_parity_v2" - -project_root <- normalizePath(getwd(), winslash = "/") - -exp_code_dir <- file.path( - project_root, "1_code/experiments", exp_id -) - -module_dir <- file.path(project_root, "1_code/modules/plants") - -## 1.3 Choose where the run reads and writes ---- -# The three paths a run needs. Point them anywhere writable - a -# scratch disk for a long run, say - and nothing else has to -# change. The defaults follow this repository's conventions. -# -# data_dir is read only. It holds the frozen test dataset, and -# no stage writes to it. -data_dir <- file.path(project_root, "0_data/test_dataset") - -# pipeline_dir takes the intermediates: bootstrap ids, fitted -# models, validation objects, and the logs. These are large and -# gitignored. -pipeline_dir <- file.path(project_root, "2_pipeline", exp_id) - -# out_dir takes the committed deliverables: tables, figures and -# the report. -out_dir <- file.path(project_root, "3_output", exp_id) - -log_path <- file.path(pipeline_dir, "logs") - -## 1.4 Set the taxa to run ---- -# Set to TRUE to model that taxon. Toggling a flag re-runs one -# taxon without commenting out code. -bryophytes <- FALSE -lichens <- FALSE -mites <- TRUE -vascular_plants <- FALSE - -## 1.5 Set the stages to run ---- -# The three modelling stages, in order. 02 needs what 01 wrote -# and 03 needs what 02 wrote, so a stage can be run alone only -# once the ones before it have run at least once into the same -# pipeline_dir. -run_bootstrap <- TRUE # 01_bootstrap_ids.R -run_models <- TRUE # 02_hierarchical_models.R -run_validation <- TRUE # 03_model_validation.R - -# The steps that turn raw model output into deliverables. -run_collect <- TRUE -run_compare <- TRUE -run_report <- TRUE - -## 1.6 Set the run length ---- -# v2 hard-coded 100 bootstraps and 14 cores. Both are parameters -# now, but parity means running at the v2 values, so leave these -# as they are for exp_000 and shorten a smoke test with the -# species vectors instead. -boot_iter <- 1:100 -n_clusters <- 14 - -# NULL leaves the bootstrap stream unseeded, as v2 did. Set an -# integer to make a run repeatable. -boot_seed <- NULL - -## 1.7 Load the species vectors ---- -# Defines bryophyte_species, lichen_species, mite_species and -# vascular_plant_species. NULL means every modelled species. -source(file.path(exp_code_dir, "utils/species_lists.R")) - -## 1.8 Load the step runner ---- -# Supplies run_step(), which times each stage and mirrors its -# output to a log. -source(file.path( - project_root, "1_code/harness/utils/step_runner.R" -)) - -# 2. Run the modelling stages ---- -# One block per taxon. Each sets the parameters the module -# scripts read - taxon, species and the output root - then runs -# the three stages in order. Nothing outside pipeline_dir is -# written. - -## 2.1 run_taxon() ---- - -#' Run the Three Modelling Stages for One Taxon -#' -#' Sets the configuration the module scripts read from the global -#' environment, then runs the stages whose flags are TRUE. The -#' scripts are sourced rather than called because they read their -#' parameters by name from there, which is also where their -#' parallel workers export from. -#' -#' @param taxon_name Character. Taxon slug, as used in the test -#' dataset file names. -#' @param taxon_species Character vector of species to model, or -#' NULL for every modelled species. -#' @param label Character. The taxon as it reads in a log line. -#' @return NULL, invisibly. -#' -#' @example -#' # Example usage of the function -#' # run_taxon("mite", mite_species, "mites") -run_taxon <- function(taxon_name, taxon_species, label) { - # Step 1: Publish the parameters the module scripts read. They - # are assigned to the global environment because that is where - # the scripts are sourced and where clusterExport() looks. - # - # run_dir is the module's own output root, kept distinct from - # out_dir so the intermediates and the deliverables cannot be - # written to the same place by accident. - taxon <<- taxon_name - species_subset <<- taxon_species - run_dir <<- file.path(pipeline_dir, taxon_name) - - # Step 2: Run each requested stage in order - stages <- list( - list( - flag = run_bootstrap, - label = paste0("01: Bootstrap ids (", label, ")"), - script = file.path(module_dir, "01_bootstrap_ids.R") - ), - list( - flag = run_models, - label = paste0("02: Hierarchical models (", label, ")"), - script = file.path(module_dir, "02_hierarchical_models.R") - ), - list( - flag = run_validation, - label = paste0("03: Model validation (", label, ")"), - script = file.path(module_dir, "03_model_validation.R") - ) - ) - - for (stage in stages) { - if (stage$flag) { - run_step( - label = stage$label, - script = stage$script, - log_dir = log_path - ) - } - } - - invisible(NULL) -} - -## 2.2 Run each requested taxon ---- -experiment_start <- Sys.time() - -if (bryophytes) { - run_taxon("bryophyte", bryophyte_species, "bryophytes") -} - -if (lichens) { - run_taxon("lichen", lichen_species, "lichens") -} - -if (mites) { - run_taxon("mite", mite_species, "mites") -} - -if (vascular_plants) { - run_taxon( - "vascular_plant", vascular_plant_species, "vascular plants" - ) -} - -# 3. Summarize the results ---- -# These read the model output left in pipeline_dir and write the -# committed deliverables to out_dir. They are placeholders until -# the parity targets are agreed. -dir.create(out_dir, recursive = TRUE, showWarnings = FALSE) - -## 3.1 Collect model output ---- -if (run_collect) { - source(file.path(exp_code_dir, "01_collect_results.R")) -} - -## 3.2 Compare against the v2.0 reference ---- -if (run_compare) { - source(file.path(exp_code_dir, "02_compare_to_v2.R")) -} - -## 3.3 Build the report ---- -if (run_report) { - source(file.path(exp_code_dir, "03_build_report.R")) -} - -# 4. Experiment complete ---- -experiment_time <- format(round(Sys.time() - experiment_start, 1)) -cat("\n========================================\n") -cat("EXPERIMENT ", exp_id, " COMPLETE\n", sep = "") -cat("Total run time: ", experiment_time, "\n", sep = "") -cat("========================================\n") -cat("Intermediates are in ", pipeline_dir, "\n", sep = "") -cat("Deliverables are in ", out_dir, "\n", sep = "") - -# End of script ---- diff --git a/1_code/_deprecated/exp_000_parity_v2/utils/species_lists.R b/1_code/_deprecated/exp_000_parity_v2/utils/species_lists.R deleted file mode 100644 index 9ca6a2d..0000000 --- a/1_code/_deprecated/exp_000_parity_v2/utils/species_lists.R +++ /dev/null @@ -1,49 +0,0 @@ -# --- -# title: Species Vectors for Experiment 000 -# author: Brendan Casey -# created: 2026-09-05 -# inputs: none -# outputs: none -# notes: -# - Defines the species each taxon is modelled on. Sourced by -# run.R and passed to the module scripts, which narrow the -# work queue to these names. -# - NULL models every species the snapshot declares as -# modelled, which is the parity setting: exp_000 has to cover -# what v2 covered. -# - A short vector is the smoke test. Species count and -# boot_iter are both adjustable now, but boot_iter has to -# stay at the v2 value for a parity run, so the species -# vector is what a trial run should shorten. -# - Names must match the species columns exactly. -# load_plant_model_data() stops and lists any name it cannot -# find, so a typo fails before modelling starts. -# - The names are those in lookup/modelled_species.csv, which -# is written from the snapshot's own veg.species.list and -# soil.species.list. To see the options for a taxon: -# -# modelled <- read.csv( -# "0_data/test_dataset/lookup/modelled_species.csv" -# ) -# head(modelled[modelled$taxon == "mite", ]) -# --- - -# 1. Species vectors ---- - -## 1.1 Bryophytes ---- -# 134 modelled species: 133 north, 32 south. -bryophyte_species <- NULL - -## 1.2 Lichens ---- -# 191 modelled species: 175 north, 55 south. -lichen_species <- NULL - -## 1.3 Soil mites ---- -# 125 modelled species: 116 north, 34 south. -mite_species <- NULL - -## 1.4 Vascular plants ---- -# 486 modelled species: 353 north, 269 south. -vascular_plant_species <- NULL - -# End of script ---- diff --git a/1_code/_deprecated/load_model_data.R b/1_code/_deprecated/load_model_data.R deleted file mode 100644 index e771a50..0000000 --- a/1_code/_deprecated/load_model_data.R +++ /dev/null @@ -1,443 +0,0 @@ -# --- -# title: Load Plant-Group Model Data from the Test Dataset -# author: Brendan Casey -# created: 2026-09-08 -# inputs: -# in 0_data/test_dataset/: -# - sites.csv -# - .csv -# - covariates/_climate.csv -# - covariates/_veg.csv -# - covariates/_soil.csv -# - lookup/veg_prediction_matrix.csv -# - lookup/soil_prediction_matrix.csv -# - lookup/modelled_species.csv -# outputs: none; returns objects in memory -# notes: -# - Rebuilds the three model frames the v2 plant scripts used to -# load from -model-data.Rdata, from the harmonized CSVs -# instead. The frames it returns are equivalent to the v2 -# objects of the same name, so the v2 function files in -# functions/ run against them unmodified. -# - The v2 code took its vegetation coefficient names -# positionally, as colnames(veg.data)[403:489] and three other -# per-taxon spans. Those spans are all the same 87 habitat -# types, and they are the first 87 columns of the vegetation -# covariate table, so veg_coef_names names them instead. That -# removes the one place where a reordered snapshot would have -# silently renamed every vegetation coefficient. -# - Protocol is carried only for bryophytes and lichens, because -# only those two v2 scripts fit it. plant_taxa() is the single -# record of which taxa do, and 02 and 03 read the flag from -# there rather than repeating it. -# - Both row-addressing styles the v2 functions use are -# supported: the hierarchical models index by rownames -# (data[site.id, ]) and the validation indexes by column -# (data$SiteYearQu %in% site.id), so SiteYearQu is kept as a -# column and as the rownames. -# - Future improvement - the same reader would serve mammals -# once their covariates are harmonized; nothing here is -# specific to the plant taxa except plant_taxa() and the site -# field mapping. -# --- - -# 1. Setup ---- - -## 1.1 Load packages ---- -library(data.table) # fast CSV reading (version: 1.16.4) - -# 2. plant_taxa() ---- - -#' Describe the Plant-Group Taxa -#' -#' The one record of which taxa exist and which of them fit a -#' Protocol term. In v2 that distinction was implicit in four -#' near-identical scripts: 02a and 02b carried `Protocol` in every -#' habitat formula and passed `protocol.flag = TRUE`, while 02c -#' and 02d carried neither. It is stated once here so the single -#' modelling script can reproduce both variants. -#' -#' @return A data frame with `taxon`, `use_protocol`, and -#' `label` (the taxon as it reads in a log line). -#' -#' @example # Example usage of the function -#' # plant_taxa() -#' # subset(plant_taxa(), taxon == "mite")$use_protocol -plant_taxa <- function() { - data.frame( - taxon = c("bryophyte", "lichen", "mite", "vascular_plant"), - use_protocol = c(TRUE, TRUE, FALSE, FALSE), - label = c("bryophytes", "lichens", "mites", "vascular plants"), - stringsAsFactors = FALSE - ) -} - -# 3. taxon_uses_protocol() ---- - -#' Look Up One Taxon's Protocol Flag -#' -#' @param taxon Character. One of the taxa in `plant_taxa()`. -#' @return Logical. TRUE when that taxon's v2 scripts fit a -#' Protocol term. -#' -#' @example # Example usage of the function -#' # taxon_uses_protocol("lichen") -taxon_uses_protocol <- function(taxon) { - taxa <- plant_taxa() - match_row <- match(taxon, taxa$taxon) - - if (is.na(match_row)) { - stop( - "Unknown taxon '", - taxon, - "'. Expected one of: ", - paste(taxa$taxon, collapse = ", "), - call. = FALSE - ) - } - - taxa$use_protocol[match_row] -} - -# 4. check_prediction_terms() ---- - -#' Check a Model Set Against Its Prediction Matrix -#' -#' Every habitat term a model fits is later predicted onto a row -#' of the prediction matrix, so a term the matrix has no column -#' for cannot be predicted. In v2 that surfaced only once fitting -#' reached the prediction step, as `object '' not found` -#' raised separately inside every bootstrap of every species. -#' Checking up front turns it into one readable failure. -#' -#' `Climate`, `Protocol` and `paspen` are excluded: they are -#' columns the model functions attach to the data themselves, and -#' the prediction matrix is not expected to carry them. -#' -#' @param models List of model formulas. -#' @param prediction_matrix Data frame. The prediction matrix the -#' fitted models are predicted onto, with `VegType` already -#' moved to the rownames. -#' @param what Character. Label for the model set, used in the -#' error message (e.g. "vegetation"). -#' @param source_note Character. Where the matrix came from, so -#' the message says what to fix. -#' @return NULL, invisibly. Stops when a term has no column. -#' -#' @example # Example usage of the function -#' # check_prediction_terms( -#' # vegetation.models, prediction.matrix, "vegetation" -#' # ) -check_prediction_terms <- function( - models, - prediction_matrix, - what, - source_note = "" -) { - # Step 1: Collect the habitat terms across the whole model set - attached_by_model_code <- c( - "pcount", "Climate", "Protocol", "paspen" - ) - - terms <- unique(unlist(lapply(models, all.vars))) - terms <- setdiff(terms, attached_by_model_code) - - # Step 2: Report every missing column at once - missing_terms <- setdiff(terms, colnames(prediction_matrix)) - - if (length(missing_terms) > 0) { - stop( - "The ", what, " prediction matrix has no column for ", - length(missing_terms), " term(s) the models fit:\n ", - paste(missing_terms, collapse = "\n "), - "\n", source_note, - call. = FALSE - ) - } - - invisible(NULL) -} - -# 5. load_plant_model_data() ---- - -#' Rebuild One Taxon's v2 Model Frames from the Test Dataset -#' -#' Joins the response, site and covariate tables into the three -#' frames the v2 plant scripts expect - `climate.data`, -#' `veg.data` and `soil.data` - and returns them alongside the -#' prediction matrices and species lists. -#' -#' `climate.data` is identity + response + climate covariates. -#' `veg.data` and `soil.data` are that frame plus their own -#' habitat block, which is how the v2 files were built: each -#' habitat frame is a superset of the climate frame, so a model -#' formula can name a species, a climate term and a habitat term -#' at once. -#' -#' @param taxon Character. One of the taxa in `plant_taxa()`. -#' @param data_dir Character. Path to 0_data/test_dataset. -#' @param species Character vector of species to model, or NULL -#' to keep every species the snapshot declares as modelled. -#' @return A list with `climate.data`, `veg.data`, `soil.data`, -#' `veg.pm`, `soil.pm`, `veg.species.list`, `soil.species.list` -#' and `veg_coef_names`. -#' -#' @example # Example usage of the function -#' # model_data <- load_plant_model_data( -#' # taxon = "bryophyte", -#' # data_dir = "0_data/test_dataset" -#' # ) -#' # dim(model_data$veg.data) -load_plant_model_data <- function( - taxon, - data_dir, - species = NULL -) { - taxon_uses_protocol(taxon) - - # Step 1: Confirm every file is present, so a missing covariate - # table fails here rather than as a cryptic formula error hours - # into a bootstrap run. - paths <- c( - sites = file.path(data_dir, "sites.csv"), - response = file.path(data_dir, paste0(taxon, ".csv")), - climate = file.path( - data_dir, "covariates", paste0(taxon, "_climate.csv") - ), - veg = file.path( - data_dir, "covariates", paste0(taxon, "_veg.csv") - ), - soil = file.path( - data_dir, "covariates", paste0(taxon, "_soil.csv") - ), - veg_pm = file.path( - data_dir, "lookup", "veg_prediction_matrix.csv" - ), - soil_pm = file.path( - data_dir, "lookup", "soil_prediction_matrix.csv" - ), - modelled = file.path( - data_dir, "lookup", "modelled_species.csv" - ) - ) - - missing_paths <- paths[!file.exists(paths)] - - if (length(missing_paths) > 0) { - stop( - "Test dataset files not found:\n ", - paste(missing_paths, collapse = "\n "), - "\nRun 1_code/_setup/01_harmonize_model_ready_v2.R first.", - call. = FALSE - ) - } - - read_table <- function(path) { - as.data.frame(fread(path, na.strings = "")) - } - - response <- read_table(paths[["response"]]) - climate_covariates <- read_table(paths[["climate"]]) - veg_covariates <- read_table(paths[["veg"]]) - soil_covariates <- read_table(paths[["soil"]]) - - # Step 2: Take the site fields back under their v2 names. The - # harmonizer factored them out of the covariate tables into - # sites.csv, so each value has one home; the v2 functions read - # them as NR, Lat and so on, and reference them by name. - site_fields <- c( - SiteYearQu = "survey_unit_id", - SiteYear = "site_year", - Site = "site", - Year = "year", - QUAD = "quadrant", - OnOffGrid = "on_off_grid", - Lat = "lat", - Long = "long", - Elevation = "elevation", - NR = "nr", - NSR = "nsr", - LufName = "luf", - Easting = "easting", - Northing = "northing" - ) - - sites <- read_table(paths[["sites"]])[, site_fields] - names(sites) <- names(site_fields) - - # Step 3: Put every table in the response table's row order, so - # the three frames stay aligned and reproduce the v2 ordering. - unit_ids <- as.character(response$survey_unit_id) - - align <- function(frame, what) { - row_order <- match(unit_ids, as.character(frame$survey_unit_id)) - - if (anyNA(row_order)) { - stop( - taxon, - ": ", - sum(is.na(row_order)), - " survey unit(s) in the response have no ", - what, - " row.", - call. = FALSE - ) - } - - frame[row_order, setdiff(names(frame), "survey_unit_id"), - drop = FALSE] - } - - site_rows <- sites[ - match(unit_ids, sites$SiteYearQu), , drop = FALSE - ] - - if (anyNA(site_rows$SiteYearQu)) { - stop( - taxon, - ": ", - sum(is.na(site_rows$SiteYearQu)), - " survey unit(s) in the response are absent from sites.csv.", - call. = FALSE - ) - } - - # Step 4: Assemble the climate frame - identity, response and - # climate covariates - then extend it with each habitat block. - species_columns <- response[ - , setdiff(names(response), "survey_unit_id"), drop = FALSE - ] - - climate.data <- cbind( - site_rows, - species_columns, - align(climate_covariates, "climate covariate") - ) - - # Step 4b: Recompute the terms that are products of other - # columns, using the same expressions - # 01a_data-standardization.R used. They are derived rather than - # stored so they stay exactly consistent with the columns they - # come from: a CSV carries about 15 significant digits, and a - # separately stored Northing2 would not equal Northing squared. - climate.data$Easting2 <- - climate.data$Easting * climate.data$Easting - climate.data$Northing2 <- - climate.data$Northing * climate.data$Northing - climate.data$EastingNorthing <- - climate.data$Northing * climate.data$Easting - climate.data$MAPPET <- climate.data$MAP * climate.data$PET - climate.data$MAT2 <- climate.data$MAT * climate.data$MAT - climate.data$CMDMAT <- climate.data$CMD * climate.data$MAT - climate.data$MWMT2 <- climate.data$MWMT * climate.data$MWMT - - veg_block <- align(veg_covariates, "vegetation covariate") - soil_block <- align(soil_covariates, "soil covariate") - - veg.data <- cbind(climate.data, veg_block) - soil.data <- cbind(climate.data, soil_block) - - # Step 5: Address rows both ways. The hierarchical models index - # by rownames and the validation matches on the SiteYearQu - # column, so the identifier has to be reachable as both. - for (frame_name in c("climate.data", "veg.data", "soil.data")) { - frame <- get(frame_name) - rownames(frame) <- frame$SiteYearQu - assign(frame_name, frame) - } - - # Step 6: Protocol is a factor in the v2 data, and the habitat - # formulas fit it as one. fread reads it as character, so it is - # converted back for the two taxa that carry it. - if (taxon_uses_protocol(taxon)) { - if (!"Protocol" %in% names(climate.data)) { - stop( - taxon, - " fits a Protocol term but no Protocol column was found ", - "in its climate covariates.", - call. = FALSE - ) - } - - climate.data$Protocol <- as.factor(climate.data$Protocol) - veg.data$Protocol <- as.factor(veg.data$Protocol) - soil.data$Protocol <- as.factor(soil.data$Protocol) - } - - # Step 7: Name the 87 habitat types the vegetation coefficient - # template is built from. They are the head of the vegetation - # block, which is the same set the v2 scripts reached by - # position; the rest of the block holds the aggregates the - # formulas call for, such as Peatland and CCDecidMixed234. - veg_coef_names <- names(veg_block)[1:87] - - # Step 8: Read the work queue, narrowed to `species` when the - # caller names one. The lists say which species each model set - # is fitted for, and they differ: a species can be common - # enough to model in the north and not in the south. - modelled <- read_table(paths[["modelled"]]) - modelled <- modelled[modelled$taxon == taxon, ] - - if (nrow(modelled) == 0) { - stop( - "No modelled species listed for ", - taxon, - " in lookup/modelled_species.csv.", - call. = FALSE - ) - } - - if (!is.null(species)) { - unknown <- setdiff(species, modelled$species) - - if (length(unknown) > 0) { - stop( - "Species not modelled for ", - taxon, - ":\n ", - paste(unknown, collapse = "\n "), - call. = FALSE - ) - } - - # Subsetting rather than reordering keeps the v2 queue order, - # so a narrowed run visits species in the same sequence. - modelled <- modelled[modelled$species %in% species, ] - } - - # Each list keeps its own v2 ordering, recorded per species by - # the harmonizer. The two differ, so neither can be recovered - # from the row order of modelled_species.csv alone. - veg_rows <- modelled[modelled$in_veg_models, ] - soil_rows <- modelled[modelled$in_soil_models, ] - - veg.species.list <- - veg_rows$species[order(veg_rows$veg_order)] - soil.species.list <- - soil_rows$species[order(soil_rows$soil_order)] - - message( - "Loaded ", - taxon, - ": ", - nrow(climate.data), - " survey units, ", - length(veg.species.list), - " vegetation and ", - length(soil.species.list), - " soil species" - ) - - return(list( - climate.data = climate.data, - veg.data = veg.data, - soil.data = soil.data, - veg.pm = read_table(paths[["veg_pm"]]), - soil.pm = read_table(paths[["soil_pm"]]), - veg.species.list = veg.species.list, - soil.species.list = soil.species.list, - veg_coef_names = veg_coef_names - )) -} - -# End of script ---- diff --git a/1_code/_deprecated/plants_functions/bootstrapping_functions.R b/1_code/_deprecated/plants_functions/bootstrapping_functions.R deleted file mode 100644 index fd87be9..0000000 --- a/1_code/_deprecated/plants_functions/bootstrapping_functions.R +++ /dev/null @@ -1,130 +0,0 @@ -# --- -# title: Bootstrapping functions -# author: Brandon Allen -# created: 2025-02-11 -# notes: Define functions that are used generating the bootstrap site list -# --- - -#' [Bootstrap data] -#' -#' [Creates a vector of site codes after thinning the recording data and performing a spatially explicate bootstrap sample.] -#' -#' @param [data] [Cleaned species and landcover data.] -#' @param [species] [Species code.] -#' @param [threshold] [Number of unique sites required to be considered sufficient for modeling.] -#' @param [boot] [Bootstrap iteration.] -#' @return [Creates a vector of site codes.] -#' - -bootstrap_data <- function(data, species, threshold = 20, boot) { - - # - # 1.0 Data standardization - # - - # Since we are treating each recording individually, we fix visit to 1 as we no longer need to apply this weight. - data["nQuadrant"] <- 1 - data$visit <- 1 - - # Create new column which is the observed count data (used for weighting in the age relationships), one for the converted probabilities - data["Count"] <- as.integer(ifelse(data[, species] > 0, 1, 0)) - data["pcount"] <- as.integer(ifelse(data[, species] > 0, 1, 0) / data$nQuadrant) - - # - # 2.0 Bootstrap - # If the boot index is 1, use the complete data set, - # otherwise perform subsampling with replacement based on block ID. - # - - # Check if there are at least 20 detections in each modeling region. - veg.detect <- sum(data$pcount[data$NR != "Grassland"]) - soil.detect <- sum(data$pcount[data$NR %in% c("Grassland", "Parkland")]) - - if (boot != 1) { - - unique.blocks <- unique(data$Block) - veg.site <- 0 - soil.site <- 0 - - # If both models are viable originally - if(veg.detect >= threshold & soil.detect >= threshold) { - - while (veg.site < threshold | soil.site < threshold) { - - data.in <- NULL - - for (sample_block in 1:length(unique.blocks)) { - - data.block <- data[grep(unique.blocks[sample_block], data$Block), ] - data.block <- data.block[sample(1:nrow(data.block), - nrow(data.block), replace=TRUE), ] - data.in <- rbind(data.in, data.block) - - } - - veg.site <- sum(ifelse(data.in[, "Count"] > 0, 1, 0)) - soil.site <- sum(ifelse(data.in[, "Count"] > 0, 1, 0)) - - } - - } - - # If only veg model is viable - if(veg.detect >= threshold & soil.detect < threshold) { - - while (veg.site < threshold) { - - data.in <- NULL - - for (sample_block in 1:length(unique.blocks)) { - - data.block <- data[grep(unique.blocks[sample_block], data$Block), ] - data.block <- data.block[sample(1:nrow(data.block), - nrow(data.block), replace=TRUE), ] - data.in <- rbind(data.in, data.block) - - } - - veg.site <- sum(ifelse(data.in[, "Count"] > 0, 1, 0)) - - } - - } - - # If only soil model is viable - if(veg.detect < threshold & soil.detect >= threshold) { - - while (soil.site < threshold) { - - data.in <- NULL - - for (sample_block in 1:length(unique.blocks)) { - - data.block <- data[grep(unique.blocks[sample_block], data$Block), ] - data.block <- data.block[sample(1:nrow(data.block), - nrow(data.block), replace=TRUE), ] - data.in <- rbind(data.in, data.block) - - } - - soil.site <- sum(ifelse(data.in[, "Count"] > 0, 1, 0)) - - } - - } - - - - } else { - - data.in <- data - - } - - # Return the unique IDs for the bootstrap iteration - boot.id <- data.frame(Boot = data.in$SiteYearQu) - colnames(boot.id)[1] <- paste0("Boot_", boot) - - return(boot.id) - -} diff --git a/1_code/_deprecated/plants_functions/hierarchical-model_functions.R b/1_code/_deprecated/plants_functions/hierarchical-model_functions.R deleted file mode 100644 index b38632b..0000000 --- a/1_code/_deprecated/plants_functions/hierarchical-model_functions.R +++ /dev/null @@ -1,981 +0,0 @@ -# --- -# title: Species hierarchical model functions -# author: Brandon Allen -# created: 2025-02-11 -# notes: Define functions that are used for performing the hierarchical modeling process. -# --- - -#' [Climate-space models] -#' -#' [Climate and spatial models that uses the spatially thinned data.] -#' -#' @param [species] [Focal species of interest.] -#' @param [data] [Cleaned species, climate, and landcover data.] -#' @param [boot.data] [List of site codes to be included for each species and bootstrap iteration.] -#' @param [climate.models] [List of climate model formulas to be assessed.] -#' @param [bioclim.models] [List of bioclim model formulas to be assessed.] -#' @param [space.models] [List of spatial model formulas to be assessed.] -#' @param [boot] [Defines bootstrap iteration.] -#' @return [Outputs vector of model coefficients defined in climate.models, bioclim.models, and space.models.] -#' - -climate_models <- function(species, data, boot.data, climate.models, bioclim.models, space.models, boot) { - - # - # 1.0 Data standardization - # - - # Since we are treating each recording individually, we fix visit to 1 as we no longer need to apply this weight. - data["nQuadrant"] <- 1 - data$visit <- 1 - - # Create new column which is the observed count data (used for weighting in the age relationships), one for the converted probabilities - data["Count"] <- as.integer(ifelse(data[, species] > 0, 1, 0)) - data["pcount"] <- as.integer(ifelse(data[, species] > 0, 1, 0) / data$nQuadrant) - data["TotalNoOfQU"] <- rep (1, nrow(data)) #Just add 1 - data["wt"] <- rep (1, nrow(data)) #Just add 1 - wt1<-data$TotalNoOfQU*data$wt # Kept for legacy purposes - - # 2.0 Bootstrap Selection - boot.data <- boot.data[[species]] - site.id <- boot.data[, paste0("Boot_", boot)] - data <- data[site.id, ] - - # - # 3.0 Climate Models - # - - space.climate.store <- list(NULL) - - for (model in 1:length(climate.models)) { - - space.climate.store[[model]] <- try(bayesglm(climate.models[[model]], - family = "binomial", - data = data, - maxit = 250)) - - } - - # Need to make sure all climate models get an update for these two variables - if(!is.null(bioclim.models)) { - - for (model.bioclim in 1:length(bioclim.models)) { - - # Calculate the length of the current model object and add to it - model.list.length <- length(climate.models) - - for (model.clim in 1:length(climate.models)) { - - # Define modifier for storing - space.climate.store[[model.clim + model.list.length]] <- try(update(space.climate.store[[model.clim]], bioclim.models[[model.bioclim]])) - - } - - } - - } - - # - # 4.0 Spatial Models - # - - if(!is.null(space.models)) { - - # Only add spatial updates for models without MAT, TD, PET (keep 1,3,5,6,7,17,19,20,21) - model.update.id <- c(1, 3, 5, 6, 7, 15, 17, 19, 20, 21) - - for (model.clim in model.update.id) { - - # Calculate the length of the current model object and add to it - model.list.length <- length(space.climate.store) - - for (model.space in 1:length(space.models)) { - - # Define modifier for storing - space.climate.store[[model.space + model.list.length]] <- try(update(space.climate.store[[model.clim]], space.models[[model.space]])) - - } - - } - - } - - # - # 5.0 Create ensemble and store - # - - modavg.sc <- model.avg(space.climate.store , rank = "AICc") - aicWtd.sc <- modavg.sc$coefficients["full", ] - - # Formatting to align variable names and keep intercept first - intercept.only <- aicWtd.sc[1] - names(intercept.only) <- "Intercept" - aicWtd.sc <- aicWtd.sc[-1] - aicWtd.sc <- aicWtd.sc[order(names(aicWtd.sc))] - aicWtd.sc <- c(intercept.only, aicWtd.sc) - - return(aicWtd.sc) - -} - -#' [Vegetation models] -#' -#' [Vegetation models that uses the spatially thinned data.] -#' -#' @param [species] [Focal species of interest.] -#' @param [data] [Cleaned species, climate, and landcover data.] -#' @param [boot.data] [List of site codes to be included for each species and bootstrap iteration.] -#' @param [habitat.models] [List of habitat model formulas to be assessed.] -#' @param [prediction.matrix] [Prediction matrix used for converting model coefficients to standardized habitat classes.] -#' @param [climate.coef] [List of climate coefficients calculated from the detection_models function.] -#' @param [weight.method] [Method used for weighting model coefficients. Inverse variance weighted is the only available method currently.] -#' @param [coef.template] [Vector template used to define valid model coefficients] -#' @param [protocol.flag] [Flag that determines if we need to account for differences in sample protocols] -#' @param [coef.adjust] [Flag used to determine if poorly sampled human footprint coefficients are adjusted based on similar feature types.] -#' @param [boot] [Defines bootstrap iteration.] -#' @return [Outputs vector of model coefficients defined in habitat.models.] -#' - -vegetation_models <- function(species, data, boot.data, habitat.models, prediction.matrix, - climate.coef, weight.method = "IVW", coef.template, - protocol.flag, coef.adjust = TRUE, boot) { - - # - # 1.0 Data standardization - # - - # Since we are treating each recording individually, we fix visit to 1 as we no longer need to apply this weight. - data["nQuadrant"] <- 1 - data$visit <- 1 - - # Create new column which is the observed count data (used for weighting in the age relationships), one for the converted probabilities - data["Count"] <- as.integer(ifelse(data[, species] > 0, 1, 0)) - data["pcount"] <- as.integer(ifelse(data[, species] > 0, 1, 0) / data$nQuadrant) - - # 1.1 Bootstrap Selection - boot.data <- boot.data[[species]] - site.id <- boot.data[, paste0("Boot_", boot)] - data <- data[site.id, ] - - # 1.2 Regional filter - data <- data[!(data$NR %in% c("Grassland")), ] - - # If the number of detections for the species is less than 20, skip - if(sum(data$Count) < 20) { - - coef.results <- coef.template - coef.se.results <- coef.results - names(coef.se.results) <- paste0(names(coef.se.results), ".SE") - coef.results <- c(coef.results, coef.se.results) - return(coef.results) - - } - - # 1.3 Define the templates for coefficients - coef.results <- coef.template - coef.se.results <- coef.template - - # - # 3.0 Climate Predictions - # - - # Using the matching climate model, create the prediction that will be used - # Add the intercept and extract climate coefficients - data$Intercept <- 1 - climate.dect.coef <- climate.coef[[species]][boot, ] - climate.veg <- as.matrix(data[, names(climate.dect.coef)]) - - # Predict space/climate component (Probability scale) - data$Climate <- plogis(drop(climate.veg %*% climate.dect.coef)) - - # - # 4.0 Vegetation Model - # - - # Loop through the models - landscape.store <- list(NULL) - - for (model in 1:length(habitat.models)) { - - landscape.store[[model]] <- try(bayesglm(habitat.models[[model]], family = "binomial", - data = data, - maxit = 250)) - - } - - # - # 5.0 Store the coefficients - # - - nModels <- length(habitat.models) - p1 <- p1.se <- array(0, c(length(habitat.models), nrow(prediction.matrix))) # Predictions for each model for each soil and HF type type. These are the main coefficients - colnames(p1) <- colnames(p1.se) <- rownames(prediction.matrix) - p.site1 <- p.site1.se <- array(0, c(nModels, nrow(data))) # Predictions for each model and each site. These are used as the offsets in the next stage - - # If there is a protocol coefficient, add it to the list, otherwise only store the climate coefficient - if(protocol.flag == TRUE) { - - p1.climate <- p1.climate.se <- array(0, c(length(habitat.models), 3)) - colnames(p1.climate) <- colnames(p1.climate.se) <- c("Intercept", "Climate", "Protocol") - - } else { - - p1.climate <- p1.climate.se <- array(0, c(length(habitat.models), 2)) - colnames(p1.climate) <- colnames(p1.climate.se) <- c("Intercept", "Climate") - - } - - for (i in 1:nModels) { - - if (class(landscape.store[[i]])[1] != "try-error") { # Prediction is 0 if model failed, but this is not used because AIC wt would equal 0 - - if(protocol.flag == TRUE) { - - plot.data <- data.frame(prediction.matrix, - Climate = 0, - Protocol = as.factor("New")) - - } else { - - plot.data <- data.frame(prediction.matrix, - Climate = 0) - - } - - p <- predict(landscape.store[[i]], newdata = plot.data, se.fit = TRUE) # For each type. Predictions made at 0% Aspen. All predictions made with new protocol. Aspen effect added later, and plotted as separate points - p1[i,] <- p$fit - p1.se[i,] <- p$se.fit - - model.fit <- predict(landscape.store[[i]], se.fit = TRUE) - p.site1[i,] <- model.fit$fit # For each site (using the original d.sp data frame) - p.site1.se[i,] <- model.fit$se.fit # For each site SE - - # Extract the climate and intercept coefficients - # We weight them separately, but then add them together to create the final coefficient - - # Coef storage (Extracts only the number of coefficients equal to object) - p <- summary(landscape.store[[i]])$coefficients[1:ncol(p1.climate),1] # For each type. - names(p)[1] <- "Intercept" - p1.climate[i, ] <- p - - # Standard error storage - p <- summary(landscape.store[[i]])$coefficients[1:ncol(p1.climate),2] # For each type. - names(p)[1] <- "Intercept" - p1.climate.se[i, ] <- p - - } - - } - - # - # 6.0 Weight the models - # - - if (weight.method == "IVW") { - - # - # Landcover - # - - tTypeMean <- tTypeVar <- NULL # Logit-scaled IVW for each veg type - - mod.converged <- NULL - for (i in 1:nModels) {mod.converged[i] <- landscape.store[[i]]$converged} - - # If there SE is 0 for a coefficient, fix it to a small value to avoid INF - p1.se[p1.se == 0] <- 0.0001 - - for (i in 1:nrow(prediction.matrix)) { - - # Note for me, the SE should be the mean not the sum. Otherwise the SE will converge to 0 - tTypeMean[i] <- sum(p1[mod.converged ,i] / p1.se[mod.converged ,i]^2) / sum(1/p1.se[mod.converged ,i]^2)# IVW mean of nModels for each veg type, on transformed scale - # Double check this tTypeVar[i]<- 1/mean(1/p1.se[mod.converged ,i]^2 ) # IVW variance of mean... - tTypeVar[i]<- 1/sum(1/p1.se[mod.converged ,i]^2 ) # IVW variance of mean... - - } - - # Weight the site level predictions - data$prediction <- colSums(p.site1/ p.site1.se^2) / colSums(1/p.site1.se^2) # Add logit-scaled model average prediction to data frame - - # Add names - names(tTypeMean) <- names(tTypeVar) <- rownames(prediction.matrix) - - # Put coefficients in Coef matrix (and SE's) - coef.results[na.omit(match(names(tTypeMean), names(coef.results)))] <- tTypeMean[names(coef.results)[na.omit(match(names(tTypeMean), names(coef.results)))]] # On logit scale - coef.se.results[na.omit(match(names(tTypeVar), names(coef.se.results)))] <- sqrt(tTypeVar[names(coef.se.results)[na.omit(match(names(tTypeVar), names(coef.se.results)))]]) # On logit scale - - # - # Climate - # - - tClimateMean <- tClimateVar <- NULL # Logit-scaled IVW for each climate - - mod.converged <- NULL - for (i in 1:nModels) {mod.converged[i] <- landscape.store[[i]]$converged} - - # If there SE is 0 for a coefficient, fix it to a small value to avoid INF - p1.climate.se[p1.climate.se == 0] <- 0.0001 - - for (i in 1:ncol(p1.climate)) { - - # Note for me, the SE should be the mean not the sum. Otherwise the SE will converge to 0 - tClimateMean[i] <- sum(p1.climate[mod.converged ,i] / p1.climate.se[mod.converged ,i]^2) / sum(1/p1.climate.se[mod.converged ,i]^2)# IVW mean of nModels for each veg type, on transformed scale - # Double check this tTypeVar[i]<- 1/mean(1/p1.se[mod.converged ,i]^2 ) # IVW variance of mean... - tClimateVar[i]<- 1/sum(1/p1.climate.se[mod.converged ,i]^2 ) # IVW variance of mean... - - } - - - # Add the intercept coefficient to the table - coef.results["Intercept"] <- tClimateMean[1] # On logit scale - coef.se.results["Intercept"] <- tClimateVar[1] # On logit scale - - # Add the climate coefficient to the table - coef.results["Climate"] <- tClimateMean[2] # On logit scale - coef.se.results["Climate"] <- tClimateVar[2] # On logit scale - - # Store protocol coefficient if modeled - if (protocol.flag == TRUE) { - - coef.results["Protocol"] <- tClimateMean[3] # On logit scale - coef.se.results["Protocol"] <- tClimateVar[3] # On logit scale - - } - - - } - - # # Currently only implement the IWV - # if (weight.method == "AIC") { - # - # # AIC calculation (I'm using AIC here, because this is primarily for prediction, rather than finding a minimial best model) - # aic.ta <- rep(999999999, (nModels)) - # - # for (i in 1:(nModels)) { - # - # if (!is.null(landscape.store[[i]]) & class(landscape.store[[i]])[1] != "try-error") { # last part is to not used non-converged models, unless none converged - # aic.ta[i] <- AICc(landscape.store[[i]]) - # } - # - # } - # - # aic.delta <- aic.ta - min(aic.ta) - # aic.exp <- exp(-1 / 2 * aic.delta) - # aic.wt.ta <- aic.exp / sum(aic.exp) - # - # tTypeMean <- tTypeVar <- NULL # Logit-scaled model averages for each veg type - # - # aic.wt.ta.adj <- ifelse(aic.wt.ta < 0.01, 0, aic.wt.ta) # Use adjusted weight to avoid poor fitting models with extreme values for certain types (usually numerically equivalent to 0's or 1's) - # aic.wt.ta.adj <- aic.wt.ta.adj / sum(aic.wt.ta.adj) - # - # # Add the site level prediction based on the ensemble - # # Convert to probabilities for the age estimation - # data$prediction <- colSums(p.site1 * aic.wt.ta.adj) - # - # for (i in 1:nrow(prediction.matrix)) { - # - # tTypeMean[i] <- sum(aic.wt.ta.adj * p1[, i]) # Mean of nModels for each veg type, on transformed scale - # #tTypeVar[i] <- sum(aic.wt.ta.adj * sqrt(p1.se[, i]^2 + (rep(tTypeMean[i], nModels) - p1[, i])^2))^2 # AIC-weighted variance of mean... - # - # } - # - # names(tTypeMean) <- rownames(prediction.matrix) # Rename the final intercept coefficient to productive - # - # # Put coefficients in Coef matrix (and SE's) - # coef.results[na.omit(match(names(tTypeMean), names(coef.results)))] <- tTypeMean[names(coef.results)[na.omit(match(names(tTypeMean), names(coef.results)))]] # On logit scale - # coef.se.results[na.omit(match(names(tTypeVar), names(coef.se.results)))] <- sqrt(tTypeVar[names(coef.se.results)[na.omit(match(names(tTypeVar), names(coef.se.results)))]]) # On logit scale - # - # - # } - - # - # 7.0 Create the forest age relationships - # - - # 7.1 Set up separate dataframes for sites containing a minimum amount of each broad stand type - stand.data <- list(WhiteSpruce = NULL, Pine = NULL, - Deciduous = NULL, Mixedwood = NULL, - BlackSpruce = NULL) - - cutoff.pc.for.age <- 0.1 # Set the cut-off for the minimum proportion of a stand type for a site to be included in the age analysis for a stand type - pSoftLin <- c(0.049, 0.0893, 0.434, 0.396) # Values based on overlap of softlin and the stand types in the 1km summary file. Lowland spruce includes other wetlands. Updated proportions 2020-11-17 - pSoftLin <- pSoftLin / sum(pSoftLin) # Make sure they sum to zero - aic.age <- array(NA, c(1, 4)) - stand.name <- c("WhiteSpruce", "Pine", "Deciduous", "Mixedwood", "BlackSpruce") - - for (i in 0:8) { # Add sites with each age class of the stand type to a separate data frames for each stand type - - i1 <- ifelse(i == 0, "R", i) # For variable name - i2 <- ifelse(i == 0, 0.5, i) # For twenty-year age - - for (stand.id in 1:length(stand.name)) { - - temp.name <- paste(stand.name[stand.id], i1, sep = "") # Col name - if (sum(data[ ,temp.name] > cutoff.pc.for.age) > 0) { - - stand.data[[stand.name[stand.id]]] <- rbind(stand.data[[stand.name[stand.id]]], - data.frame(pCount = data[data[, temp.name] > cutoff.pc.for.age, "Count"] / data$nQuadrant[data[, temp.name] > cutoff.pc.for.age], - age = i2, - wt1 = data[data[, temp.name] > cutoff.pc.for.age, temp.name]*data$visit[data[, temp.name] > cutoff.pc.for.age]*data$nQuadrant[data[, temp.name] > cutoff.pc.for.age], - p = data$prediction[data[, temp.name] > cutoff.pc.for.age])) # Weight is the proportion of the site of that age class and stand type, multiplied by the original weight (which accounts for revisited sites) and the number of binomial trials - - } - - } - - } - - rm(stand.name) - - # For upland forests, add grass sites as age 1, but with half the normal weight - d.grass <- data[data[, "Grass"] > cutoff.pc.for.age, ] # Select sites with grass (to make lines below less unruly) - pWhiteSpruce <- d.grass$WhiteSpruce / (d.grass$WhiteSpruce + d.grass$Pine + d.grass$Deciduous + d.grass$Mixedwood + d.grass$BlackSpruce + 0.01) - pPine <- d.grass$Pine / (d.grass$WhiteSpruce + d.grass$Pine + d.grass$Deciduous + d.grass$Mixedwood + d.grass$BlackSpruce + 0.01) - pDeciduous <- d.grass$Deciduous / (d.grass$WhiteSpruce + d.grass$Pine + d.grass$Deciduous + d.grass$Mixedwood + d.grass$BlackSpruce + 0.01) - pMixedwood <- d.grass$WhiteSpruce / (d.grass$WhiteSpruce + d.grass$Pine + d.grass$Deciduous + d.grass$Mixedwood + d.grass$BlackSpruce + 0.01) - - # If there are no sites to add, don't bind - if(sum(pWhiteSpruce) != 0) {stand.data$WhiteSpruce <- rbind(stand.data$WhiteSpruce, data.frame(pCount = d.grass[pWhiteSpruce > 0, "Count"] / d.grass$nQuadrant[pWhiteSpruce > 0], age = 0.5, wt1 = d.grass[pWhiteSpruce > 0, "Grass"]/2*d.grass$visit[pWhiteSpruce > 0]*d.grass$nQuadrant[pWhiteSpruce > 0]*pWhiteSpruce[pWhiteSpruce > 0], p = d.grass$prediction[pWhiteSpruce > 0]))} - if(sum(pPine) != 0) {stand.data$Pine <- rbind(stand.data$Pine, data.frame(pCount = d.grass[pPine > 0, "Count"] / d.grass$nQuadrant[pPine>0], age = 0.5, wt1 = d.grass[pPine>0,"Grass"]/2*d.grass$visit[pPine>0]*d.grass$nQuadrant[pPine>0]*pPine[pPine>0], p = d.grass$prediction[pPine>0]))} - if(sum(pDeciduous) != 0) {stand.data$Deciduous <- rbind(stand.data$Deciduous, data.frame(pCount = d.grass[pDeciduous > 0, "Count"] / d.grass$nQuadrant[pDeciduous > 0], age = 0.5, wt1 = d.grass[pDeciduous > 0, "Grass"] / 2*d.grass$visit[pDeciduous > 0]*d.grass$nQuadrant[pDeciduous > 0]*pDeciduous[pDeciduous > 0], p = d.grass$prediction[pDeciduous > 0]))} - if(sum(pMixedwood) != 0) {stand.data$Mixedwood <- rbind(stand.data$Mixedwood, data.frame(pCount = d.grass[pMixedwood > 0, "Count"] / d.grass$nQuadrant[pMixedwood > 0], age = 0.5, wt1 = d.grass[pMixedwood > 0, "Grass"] / 2*d.grass$visit[pMixedwood > 0]*d.grass$nQuadrant[pMixedwood > 0]*pMixedwood[pMixedwood > 0], p = d.grass$prediction[pMixedwood > 0]))} - - # And same for sites with shrubs - d.shrub <- data[data[, "Shrub"] > cutoff.pc.for.age,] # Select sites with shrubs (to make lines below less unruly) - pWhiteSpruce <- d.shrub$WhiteSpruce / (d.shrub$WhiteSpruce + d.shrub$Pine + d.shrub$Deciduous + d.shrub$Mixedwood + d.shrub$BlackSpruce + 0.01) - pPine <- d.shrub$Pine / (d.shrub$WhiteSpruce + d.shrub$Pine + d.shrub$Deciduous + d.shrub$Mixedwood + d.shrub$BlackSpruce + 0.01) - pDeciduous <- d.shrub$Deciduous / (d.shrub$WhiteSpruce + d.shrub$Pine + d.shrub$Deciduous + d.shrub$Mixedwood + d.shrub$BlackSpruce + 0.01) - pMixedwood <- d.shrub$WhiteSpruce / (d.shrub$WhiteSpruce + d.shrub$Pine + d.shrub$Deciduous + d.shrub$Mixedwood + d.shrub$BlackSpruce + 0.01) - - if(sum(pWhiteSpruce) != 0) {stand.data$WhiteSpruce <- rbind(stand.data$WhiteSpruce, data.frame(pCount = d.shrub[pWhiteSpruce > 0 , "Count"] / d.shrub$nQuadrant[pWhiteSpruce > 0], age = 1, wt1 = d.shrub[pWhiteSpruce > 0, "Shrub"] / 2*d.shrub$visit[pWhiteSpruce > 0]*d.shrub$nQuadrant[pWhiteSpruce > 0]*pWhiteSpruce[pWhiteSpruce > 0], p = d.shrub$prediction[pWhiteSpruce > 0]))} - if(sum(pPine) != 0) {stand.data$Pine <- rbind(stand.data$Pine, data.frame(pCount = d.shrub[pPine > 0, "Count"] / d.shrub$nQuadrant[pPine > 0], age = 1, wt1 = d.shrub[pPine > 0, "Shrub"] / 2*d.shrub$visit[pPine > 0]*d.shrub$nQuadrant[pPine > 0]*pPine[pPine > 0], p = d.shrub$prediction[pPine > 0]))} - if(sum(pDeciduous) != 0) {stand.data$Deciduous <- rbind(stand.data$Deciduous, data.frame(pCount = d.shrub[pDeciduous > 0, "Count"] / d.shrub$nQuadrant[pDeciduous > 0], age = 1, wt1 = d.shrub[pDeciduous > 0, "Shrub"] / 2*d.shrub$visit[pDeciduous > 0]*d.shrub$nQuadrant[pDeciduous > 0]*pDeciduous[pDeciduous > 0], p = d.shrub$prediction[pDeciduous > 0]))} - if(sum(pMixedwood) != 0) {stand.data$Mixedwood <- rbind(stand.data$Mixedwood, data.frame(pCount = d.shrub[pMixedwood > 0, "Count"] / d.shrub$nQuadrant[pMixedwood > 0], age = 1, wt1 = d.shrub[pMixedwood > 0, "Shrub"] / 2*d.shrub$visit[pMixedwood > 0]*d.shrub$nQuadrant[pMixedwood > 0]*pMixedwood[pMixedwood > 0], p = d.shrub$prediction[pMixedwood > 0]))} - - # Combined data frames for models using more than one stand type - stand.data$Conifpine <- rbind(cbind(stand.data$WhiteSpruce, Sp = "WhiteSpruce"), cbind(stand.data$Pine, Sp = "Pine")) - stand.data$Decidmixed <- rbind(cbind(stand.data$Deciduous, Sp = "Decid"), cbind(stand.data$Mixedwood, Sp = "Mixed")) - stand.data$All <- rbind(cbind(stand.data$WhiteSpruce, Sp = "WhiteSpruce"), cbind(stand.data$Pine, Sp = "Pine"), cbind(stand.data$Deciduous, Sp = "Decid"), cbind(stand.data$Mixedwood, Sp = "Mixed"), cbind(stand.data$BlackSpruce, Sp = "BlackSpruce")) - - # 6.2 Then fit models of age functions - ndet <- 8 - m.age <- list(NULL) # Age models for each of the 5 stand types plus 3 combinations - if (sum(sign(stand.data$WhiteSpruce$pCount)) > ndet) { - m.age[[1]] <- gam(pCount~s(sqrt(age),k = 3,m = 2) + offset(p), data = stand.data$WhiteSpruce, family = "binomial", weights = wt1) # Fit spline through age data for that stand type if enough records - } else { - m.age[[1]] <- gam(pCount~1 + offset(p), data = stand.data$WhiteSpruce, family = "binomial", weights = wt1) # Fit constant only if too few records in that stand type - } - if (sum(sign(stand.data$Pine$pCount)) > ndet) { - m.age[[2]] <- gam(pCount~s(sqrt(age),k = 3,m = 2) + offset(p), data = stand.data$Pine, family = "binomial", weights = wt1) - } else { - m.age[[2]] <- gam(pCount~1 + offset(p), data = stand.data$Pine, family = "binomial", weights = wt1) - } - if (sum(sign(stand.data$Deciduous$pCount)) > ndet) { - m.age[[3]] <- gam(pCount~s(sqrt(age),k = 3,m = 2) + offset(p), data = stand.data$Deciduous, family = "binomial", weights = wt1) - } else { - m.age[[3]] <- gam(pCount~1 + offset(p), data = stand.data$Deciduous, family = "binomial", weights = wt1) - } - if (sum(sign(stand.data$Mixedwood$pCount)) > ndet) { - m.age[[4]] <- gam(pCount~s(sqrt(age),k = 3,m = 2) + offset(p), data = stand.data$Mixedwood, family = "binomial", weights = wt1) - } else { - m.age[[4]] <- gam(pCount~1 + offset(p), data = stand.data$Mixedwood, family = "binomial", weights = wt1) - } - if (sum(sign(stand.data$BlackSpruce$pCount)) > ndet) { - m.age[[5]] <- gam(pCount~s(sqrt(age),k = 3,m = 2) + offset(p), data = stand.data$BlackSpruce, family = "binomial", weights = wt1) - } else { - m.age[[5]] <- gam(pCount~1 + offset(p), data = stand.data$BlackSpruce, family = "binomial", weights = wt1) - } - if (sum(sign(stand.data$Conifpine$pCount)) > ndet) { - m.age[[6]] <- gam(pCount~s(sqrt(age),k = 3,m = 2) + offset(p), data = stand.data$Conifpine, family = "binomial", weights = wt1) - } else { - m.age[[6]] <- gam(pCount~1 + offset(p), data = stand.data$Conifpine, family = "binomial", weights = wt1) - } - if (sum(sign(stand.data$Decidmixed$pCount)) > ndet) { - m.age[[7]] <- gam(pCount~s(sqrt(age),k = 3,m = 2) + offset(p), data = stand.data$Decidmixed, family = "binomial", weights = wt1) - } else { - m.age[[7]] <- gam(pCount~1 + offset(p), data = stand.data$Decidmixed, family = "binomial", weights = wt1) - } - if (sum(sign(stand.data$All$pCount)) > ndet) { - m.age[[8]] <- gam(pCount~s(sqrt(age),k = 3,m = 2) + offset(p), data = stand.data$All, family = "binomial", weights = wt1) - } else { - m.age[[8]] <- gam(pCount~1 + offset(p), data = stand.data$All, family = "binomial", weights = wt1) - } - - # Added an intercept only model. Prevents the simple model 8 from producing curves - # when we effectively have no age relationships. Even if we have enough data to attempt a curve. - m.age[[9]] <- gam(pCount~1 + offset(p), data = stand.data$All, family = "binomial", weights = wt1) - - # AIC for options - each separate, pairwise combinations, combine all - aic.age[1,1] <- AIC(m.age[[1]]) + AIC(m.age[[2]]) + AIC(m.age[[3]]) + AIC(m.age[[4]]) + AIC(m.age[[5]])-4 # Minus 4 for the 4 extra variances - aic.age[1,2] <- AIC(m.age[[5]]) + AIC(m.age[[6]]) + AIC(m.age[[7]])-2 - aic.age[1,3] <- AIC(m.age[[8]]) - aic.age[1,4] <- AIC(m.age[[9]]) - - # 7.3. Predict coefficients (and logit-scale SE) for each age class for each stand type - if (which.min(aic.age[1, ]) == 1) { - - p3 <- predict(m.age[[1]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["WhiteSpruce"]), se.fit = T) - coef.results[which(names(coef.results) == "WhiteSpruceR"):which(names(coef.results) == "WhiteSpruce8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "WhiteSpruceR"):which(names(coef.results) == "WhiteSpruce8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[2]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Pine"]), se.fit = T) - coef.results[which(names(coef.results) == "PineR"):which(names(coef.results) == "Pine8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "PineR"):which(names(coef.results) == "Pine8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[3]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Deciduous"]), se.fit = T) - coef.results[which(names(coef.results) == "DeciduousR"):which(names(coef.results) == "Deciduous8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "DeciduousR"):which(names(coef.results) == "Deciduous8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[4]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Mixedwood"]), se.fit = T) - coef.results[which(names(coef.results) == "MixedwoodR"):which(names(coef.results) == "Mixedwood8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "MixedwoodR"):which(names(coef.results) == "Mixedwood8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[5]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["BlackSpruce"]), se.fit = T) - coef.results[which(names(coef.results) == "BlackSpruceR"):which(names(coef.results) == "BlackSpruce8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "BlackSpruceR"):which(names(coef.results) == "BlackSpruce8")] <- p3$se.fit # SE still on the logit scale - - } - - if (which.min(aic.age[1, ]) == 2) { - - p3 <- predict(m.age[[6]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["WhiteSpruce"]), se.fit = T) - coef.results[which(names(coef.results) == "WhiteSpruceR"):which(names(coef.results) == "WhiteSpruce8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "WhiteSpruceR"):which(names(coef.results) == "WhiteSpruce8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[6]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Pine"]), se.fit = T) - coef.results[which(names(coef.results) == "PineR"):which(names(coef.results) == "Pine8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "PineR"):which(names(coef.results) == "Pine8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[7]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Deciduous"]), se.fit = T) - coef.results[which(names(coef.results) == "DeciduousR"):which(names(coef.results) == "Deciduous8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "DeciduousR"):which(names(coef.results) == "Deciduous8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[7]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Mixedwood"]), se.fit = T) - coef.results[which(names(coef.results) == "MixedwoodR"):which(names(coef.results) == "Mixedwood8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "MixedwoodR"):which(names(coef.results) == "Mixedwood8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[5]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["BlackSpruce"]), se.fit = T) - coef.results[which(names(coef.results) == "BlackSpruceR"):which(names(coef.results) == "BlackSpruce8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "BlackSpruceR"):which(names(coef.results) == "BlackSpruce8")] <- p3$se.fit # SE still on the logit scale - - } - - if (which.min(aic.age[1, ]) == 3) { - - p3 <- predict(m.age[[8]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["WhiteSpruce"]), se.fit = T) - coef.results[which(names(coef.results) == "WhiteSpruceR"):which(names(coef.results) == "WhiteSpruce8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "WhiteSpruceR"):which(names(coef.results) == "WhiteSpruce8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[8]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Pine"]), se.fit = T) - coef.results[which(names(coef.results) == "PineR"):which(names(coef.results) == "Pine8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "PineR"):which(names(coef.results) == "Pine8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[8]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Deciduous"]), se.fit = T) - coef.results[which(names(coef.results) == "DeciduousR"):which(names(coef.results) == "Deciduous8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "DeciduousR"):which(names(coef.results) == "Deciduous8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[8]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Mixedwood"]), se.fit = T) - coef.results[which(names(coef.results) == "MixedwoodR"):which(names(coef.results) == "Mixedwood8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "MixedwoodR"):which(names(coef.results) == "Mixedwood8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[8]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["BlackSpruce"]), se.fit = T) - coef.results[which(names(coef.results) == "BlackSpruceR"):which(names(coef.results) == "BlackSpruce8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "BlackSpruceR"):which(names(coef.results) == "BlackSpruce8")] <- p3$se.fit # SE still on the logit scale - - } - - - if (which.min(aic.age[1, ]) == 4) { - - p3 <- predict(m.age[[9]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["WhiteSpruce"]), se.fit = T) - coef.results[which(names(coef.results) == "WhiteSpruceR"):which(names(coef.results) == "WhiteSpruce8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "WhiteSpruceR"):which(names(coef.results) == "WhiteSpruce8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[9]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Pine"]), se.fit = T) - coef.results[which(names(coef.results) == "PineR"):which(names(coef.results) == "Pine8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "PineR"):which(names(coef.results) == "Pine8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[9]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Deciduous"]), se.fit = T) - coef.results[which(names(coef.results) == "DeciduousR"):which(names(coef.results) == "Deciduous8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "DeciduousR"):which(names(coef.results) == "Deciduous8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[9]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["Mixedwood"]), se.fit = T) - coef.results[which(names(coef.results) == "MixedwoodR"):which(names(coef.results) == "Mixedwood8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "MixedwoodR"):which(names(coef.results) == "Mixedwood8")] <- p3$se.fit # SE still on the logit scale - - p3 <- predict(m.age[[9]], newdata = data.frame(age = c(0.5,1:8), p = tTypeMean["BlackSpruce"]), se.fit = T) - coef.results[which(names(coef.results) == "BlackSpruceR"):which(names(coef.results) == "BlackSpruce8")] <- p3$fit # Coefficient on logit scale - coef.se.results[which(names(coef.results) == "BlackSpruceR"):which(names(coef.results) == "BlackSpruce8")] <- p3$se.fit # SE still on the logit scale - - } - - # Will need to think of something else - # # 7.4 Adjust SE for estimates that are numerically equivalent to 0 - use exact binomial CI - # i4 <- which(is.na(coef.results) | coef.results < 0.0001) - # coef.results[i4] <- 0.0001 - # ntrials <- (colSums(data[, match(names(coef.results), names(data), nomatch = 0)])*2) + 1 # *2 because many rare types (which have the 0's are estimated from groups of types), + 1 to avoid failure for old cutblocks, - # for (i5 in i4) coef.se.results[i5] <- (qlogis(binom.confint(0, ntrials[i5], conf.level = 0.9)[5,6]) - qlogis(0.0001)) / 1.65 # That conf/level gives 1SE. Value is on logit scale, assuming estimate of 0.0001. This is set up so that when 90% CI's are ploted below, they will return the exact 90% binom confints - - # 7.5 Converge CC to natural trajectory - # Weighting is done on the probabilities - # Smooth CC trajectories into natural ones - coef.results["CCWhiteSpruce2"] <- qlogis(plogis(coef.results["WhiteSpruce2"])*0.50 + plogis(coef.results["CCWhiteSpruce2"])*(1-0.50)) # WhiteSpruce CC 20-40yr is 50% of way to 20-40yr natural forest - see document on convergence rates - coef.results["CCWhiteSpruce3"] <- qlogis(plogis(coef.results["WhiteSpruce3"])*0.849 + plogis(coef.results["CCWhiteSpruce3"])*(1-0.849)) # WhiteSpruce CC 40-60yr is 84.9% of way to 40-60yr natural forest - coef.results["CCWhiteSpruce4"] <- qlogis(plogis(coef.results["WhiteSpruce4"])*0.96 + plogis(coef.results["CCWhiteSpruce4"])*(1-0.96)) # WhiteSpruce CC 60-80yr is 96% of way to 60-80yr natural forest - coef.results["CCPine2"] <- qlogis(plogis(coef.results["Pine2"])*0.50 + plogis(coef.results["CCPine2"])*(1-0.50)) # Pine CC 20-40 is 50% of way to 20-40yr natural forest - see document on convergence rates - coef.results["CCPine3"] <- qlogis(plogis(coef.results["Pine3"])*0.849 + plogis(coef.results["CCPine3"])*(1-0.849)) # Pine CC 40-60 is 84.9% of way to 40-60yr natural forest - coef.results["CCPine4"] <- qlogis(plogis(coef.results["Pine4"])*0.96 + plogis(coef.results["CCPine4"])*(1-0.96)) # Pine CC 60-80 is 96% of way to 60-80yr natural forest - coef.results["CCDeciduous2"] <- qlogis(plogis(coef.results["Deciduous2"])*0.705 + plogis(coef.results["CCDeciduous2"])*(1-0.705)) # Decid CC 20-40yr is 70.5% of way to 20-40yr natural forest - see document on convergence rates - coef.results["CCDeciduous3"] <- qlogis(plogis(coef.results["Deciduous3"])*0.912 + plogis(coef.results["CCDeciduous3"])*(1-0.912)) # Decid CC 40-60yr is 91.2% of way to 40-60yr natural forest - coef.results["CCDeciduous4"] <- qlogis(plogis(coef.results["Deciduous4"])*0.97 + plogis(coef.results["CCDeciduous4"])*(1-0.97)) # Decid CC 60-80yr is 97% of way to 60-80yr natural forest - coef.results["CCMixedwood2"] <- qlogis(plogis(coef.results["Mixedwood2"])*0.705 + plogis(coef.results["CCMixedwood2"])*(1-0.705)) # Mixedwood CC 20-40yr is 70.5% of way to 20-40yr natural forest - see document on convergence rates - coef.results["CCMixedwood3"] <- qlogis(plogis(coef.results["Mixedwood3"])*0.912 + plogis(coef.results["CCMixedwood3"])*(1-0.912)) # Mixedwood CC 40-60yr is 91.2% of way to 40-60yr natural forest - coef.results["CCMixedwood4"] <- qlogis(plogis(coef.results["Mixedwood4"])*0.97 + plogis(coef.results["CCMixedwood4"])*(1-0.97)) # Mixedwood CC 60-80yr is 97% of way to 60-80yr natural forest - - # And same for SE's - coef.se.results["CCWhiteSpruce2"] <- coef.se.results["WhiteSpruce2"]*0.50 + coef.se.results["CCWhiteSpruce2"]*(1-0.50) - coef.se.results["CCWhiteSpruce3"] <- coef.se.results["WhiteSpruce3"]*0.849 + coef.se.results["CCWhiteSpruce3"]*(1-0.849) - coef.se.results["CCWhiteSpruce4"] <- coef.se.results["WhiteSpruce4"]*0.96 + coef.se.results["CCWhiteSpruce4"]*(1-0.96) - coef.se.results["CCPine2"] <- coef.se.results["Pine2"]*0.50 + coef.se.results["CCPine2"]*(1-0.50) - coef.se.results["CCPine3"] <- coef.se.results["Pine3"]*0.849 + coef.se.results["CCPine3"]*(1-0.849) - coef.se.results["CCPine4"] <- coef.se.results["Pine4"]*0.96 + coef.se.results["CCPine4"]*(1-0.96) - coef.se.results["CCDeciduous2"] <- coef.se.results["Deciduous2"]*0.705 + coef.se.results["CCDeciduous2"]*(1-0.705) - coef.se.results["CCDeciduous3"] <- coef.se.results["Deciduous3"]*0.912 + coef.se.results["CCDeciduous3"]*(1-0.912) - coef.se.results["CCDeciduous4"] <- coef.se.results["Deciduous4"]*0.97 + coef.se.results["CCDeciduous4"]*(1-0.97) - coef.se.results["CCMixedwood2"] <- coef.se.results["Mixedwood2"]*0.705 + coef.se.results["CCMixedwood2"]*(1-0.705) - coef.se.results["CCMixedwood3"] <- coef.se.results["Mixedwood3"]*0.912 + coef.se.results["CCMixedwood3"]*(1-0.912) - coef.se.results["CCMixedwood4"] <- coef.se.results["Mixedwood4"]*0.97 + coef.se.results["CCMixedwood4"]*(1-0.97) - - # - # 8.0 Adjust coefficients that are often estimated poorly. - # - - if (coef.adjust == TRUE) { # If true, adjust coefficients, otherwise leave as is. Can help to determine how much the adjustment is needed - - coef.results["HardLin"] <- (coef.results["HardLin"] / coef.se.results["HardLin"]^2 + coef.results["UrbInd"]/coef.se.results["UrbInd"]^2) / (1 / coef.se.results["HardLin"]^2 + 1 / coef.se.results["UrbInd"]^2) # Inverse-variance weighted, done on logit scale - coef.se.results["HardLin"] <- sqrt(1 / (1 / coef.se.results["HardLin"]^2 + 1 / coef.se.results["UrbInd"]^2)) - - young <- (pSoftLin[1]*coef.results["CCWhiteSpruceR"] + pSoftLin[2]*coef.results["CCPineR"] + pSoftLin[3]*coef.results["CCDeciduousR"] + pSoftLin[4]*coef.results["BlackSpruce1"]) / sum(pSoftLin) - young.se <- sqrt( ( pSoftLin[1]*coef.se.results["CCWhiteSpruceR"]^2 + pSoftLin[2]*coef.se.results["CCPineR"]^2 + pSoftLin[3]*coef.se.results["CCDeciduousR"]^2 + pSoftLin[4]*coef.se.results["BlackSpruce1"]^2 ) / sum(pSoftLin) ) # Weighted se (on logit scale) - does not include component of variance among types, because fixed weighting - - # Adjust for the three soft linear type coefficients (EnSoftLin, EnSeismic, TrSoftLin) - coef.results["EnSoftLin"] <- (coef.results["EnSoftLin"]/coef.se.results["EnSoftLin"]^2 + young / young.se^2) / (1 / coef.se.results["EnSoftLin"]^2 + 1 / young.se^2) # Inverse-variance weighted - coef.se.results["EnSoftLin"] <- sqrt(1 / (1 / coef.se.results["EnSoftLin"]^2 + 1 / young.se^2)) - - coef.results["EnSeismic"] <- (coef.results["EnSeismic"]/coef.se.results["EnSeismic"]^2 + young / young.se^2) / (1 / coef.se.results["EnSeismic"]^2 + 1 / young.se^2) # Inverse-variance weighted - coef.se.results["EnSeismic"] <- sqrt(1 / (1 / coef.se.results["EnSeismic"]^2 + 1 / young.se^2)) - - coef.results["TrSoftLin"] <- (coef.results["TrSoftLin"]/coef.se.results["TrSoftLin"]^2 + young / young.se^2) / (1 / coef.se.results["TrSoftLin"]^2 + 1 / young.se^2) # Inverse-variance weighted - coef.se.results["TrSoftLin"] <- sqrt(1 / (1 / coef.se.results["TrSoftLin"]^2 + 1 / young.se^2)) - - rm(young, young.se) - - } - - # Store the coefficients as a single object - names(coef.se.results) <- paste0(names(coef.se.results), ".SE") - coef.results <- c(coef.results, coef.se.results) - return(coef.results) - - -} - -#' [Soil models] -#' -#' [soil models that uses the spatially thinned data.] -#' -#' @param [species] [Focal species of interest.] -#' @param [data] [Cleaned species, climate, and landcover data.] -#' @param [boot.data] [List of site codes to be included for each species and bootstrap iteration.] -#' @param [habitat.models] [List of habitat model formulas to be assessed.] -#' @param [prediction.matrix] [Prediction matrix used for converting model coefficients to standardized habitat classes.] -#' @param [climate.coef] [List of climate coefficients calculated from the detection_models function.] -#' @param [weight.method] [Method used for weighting model coefficients. Inverse variance weighted is the only available method currently.] -#' @param [coef.template] [Vector template used to define valid model coefficients] -#' @param [protocol.flag] [Flag that determines if we need to account for differences in sample protocols] -#' @param [coef.adjust] [Flag used to determine if poorly sampled human footprint coefficients are adjusted based on similar feature types.] -#' @param [boot] [Defines bootstrap iteration.] -#' @return [Outputs vector of model coefficients defined in habitat.models.] -#' - -soil_models <- function(species, data, boot.data, habitat.models, prediction.matrix, - climate.coef, weight.method = "IVW", coef.template, - protocol.flag, coef.adjust = TRUE, boot) { - - # - # 1.0 Data standardization - # - - # Since we are treating each recording individually, we fix visit to 1 as we no longer need to apply this weight. - data["nQuadrant"] <- 1 - data$visit <- 1 - - # Create new column which is the observed count data (used for weighting in the age relationships), one for the converted probabilities - data["Count"] <- as.integer(ifelse(data[, species] > 0, 1, 0)) - data["pcount"] <- as.integer(ifelse(data[, species] > 0, 1, 0) / data$nQuadrant) - - # 1.1 Bootstrap Selection - boot.data <- boot.data[[species]] - site.id <- boot.data[, paste0("Boot_", boot)] - data <- data[site.id, ] - - # 1.2 Regional filter - data <- data[data$NR %in% c("Grassland", "Parkland"), ] - - # If the number of detections for the species is less than 20, return empty matrix - if(sum(data$Count) < 20) { - - coef.results <- coef.template - coef.se.results <- coef.results - names(coef.se.results) <- paste0(names(coef.se.results), ".SE") - coef.results <- c(coef.results, coef.se.results) - return(coef.results) - - } - - # 1.3 Define the templates for coefficients - coef.results <- coef.template - coef.se.results <- coef.template - - # - # 3.0 Climate Predictions - # - - # Using the matching climate model, create the prediction that will be used - # Add the intercept and extract climate coefficients - data$Intercept <- 1 - climate.dect.coef <- climate.coef[[species]][boot, ] - climate.veg <- as.matrix(data[, names(climate.dect.coef)]) - - # Predict space/climate component (Probability scale) - data$Climate <- plogis(drop(climate.veg %*% climate.dect.coef)) - - # - # 4.0 Soil Model - # - - # Loop through the models - landscape.store <- list(NULL) - - for (model in 1:length(habitat.models)) { - - landscape.store[[model]] <- try(bayesglm(habitat.models[[model]], family = "binomial", - data = data, - maxit = 250)) - - } - - # - # 4.0 Store the coefficients - # - - # South model needs to account for the addition of the probability of aspen coefficient, otherwise the processes is similar. - nModels <- length(habitat.models) - p1 <- p1.se <- array(0, c(length(habitat.models), nrow(prediction.matrix))) # Predictions for each model for each soil and HF type type. These are the main coefficients - colnames(p1) <- colnames(p1.se) <- rownames(prediction.matrix) - p.site1 <- p.site1.se <- array(0, c(nModels, nrow(data))) # Predictions for each model and each site. These are used as the offsets in the next stage - - # If there is a protocol coefficient, add it to the list, otherwise only store the climate coefficient - if(protocol.flag == TRUE) { - - p1.climate <- p1.climate.se <- array(0, c(length(habitat.models), 3)) - colnames(p1.climate) <- colnames(p1.climate.se) <- c("Intercept", "Climate", "Protocol") - - } else { - - p1.climate <- p1.climate.se <- array(0, c(length(habitat.models), 2)) - colnames(p1.climate) <- colnames(p1.climate.se) <- c("Intercept", "Climate") - - } - - p1.aspen <- p1.se.aspen <- rep(0, nModels) # Predictions for each model for aspen coefficient - - for (i in 1:nModels) { - - if (class(landscape.store[[i]])[1] != "try-error") { # Prediction is 0 if model failed, but this is not used because AIC wt would equal 0 - - if(protocol.flag == TRUE) { - - plot.data <- data.frame(prediction.matrix, - Climate = 0, - paspen = 0, - Protocol = as.factor("New")) - - } else { - - plot.data <- data.frame(prediction.matrix, - paspen = 0, - Climate = 0) - - } - - p <- predict(landscape.store[[i]], newdata = plot.data, se.fit = TRUE) # For each type. Predictions made at 0% Aspen. All predictions made with new protocol. Aspen effect added later, and plotted as separate points - p1[i,] <- p$fit - p1.se[i,] <- p$se.fit - - model.fit <- predict(landscape.store[[i]], se.fit = TRUE) - p.site1[i,] <- model.fit$fit # For each site (using the original d.sp data frame) - p.site1.se[i,] <- model.fit$se.fit # For each site SE - - # Extract the climate and intercept coefficients - # We weight them separately, but then add them together to create the final coefficient - - # Coef storage - p <- summary(landscape.store[[i]])$coefficients[1:ncol(p1.climate),1] # For each type. Predictions made at 0% Aspen. All predictions made with new protocol. Aspen effect added later, and plotted as separate points - names(p)[1] <- "Intercept" - p1.climate[i, ] <- p - - # Standard error storage - p <- summary(landscape.store[[i]])$coefficients[1:ncol(p1.climate),2] # For each type. Predictions made at 0% Aspen. All predictions made with new protocol. Aspen effect added later, and plotted as separate points - names(p)[1] <- "Intercept" - p1.climate.se[i, ] <- p - - # If pasen is present, store the coefficient - if ("paspen" %in% names(coef(landscape.store[[i]]))) { - p1.aspen[i] <- coefficients(summary(landscape.store[[i]]))["paspen","Estimate"] - p1.se.aspen[i] <- coefficients(summary(landscape.store[[i]]))["paspen","Std. Error"] - } - - } - - } - - # - # 5.0 Weight the models - # - - if (weight.method == "IVW") { - - # - # Landcover - # - - tTypeMean <- tTypeVar <- NULL # Logit-scaled IVW for each veg type - - mod.converged <- NULL - for (i in 1:nModels) {mod.converged[i] <- landscape.store[[i]]$converged} - - for (i in 1:nrow(prediction.matrix)) { - - tTypeMean[i] <- sum(p1[mod.converged ,i] / p1.se[mod.converged ,i]^2) / sum(1/p1.se[mod.converged ,i]^2)# IVW mean of nModels for each veg type, on transformed scale - tTypeVar[i]<- 1/sum(1/p1.se[mod.converged ,i]^2 ) # IVW variance of mean... - - } - - # Add names - names(tTypeMean) <- names(tTypeVar) <- rownames(prediction.matrix) - - # If paspen coefficients / SE is 0, remove - p1.aspen <- p1.aspen[p1.aspen > 0] - p1.se.aspen <- p1.se.aspen[p1.se.aspen > 0] - - tTypeMean["paspen"] <- sum(p1.aspen / p1.se.aspen^2) / sum(1 / p1.se.aspen^2) - tTypeVar["paspen"] <- 1/sum(1/p1.se.aspen^2 ) # IVWvariance of mean... - - # Put coefficients in Coef matrix (and SE's) - coef.results[na.omit(match(names(tTypeMean), names(coef.results)))] <- tTypeMean[names(coef.results)[na.omit(match(names(tTypeMean), names(coef.results)))]] # On logit scale - coef.se.results[na.omit(match(names(tTypeVar), names(coef.se.results)))] <- sqrt(tTypeVar[names(coef.se.results)[na.omit(match(names(tTypeVar), names(coef.se.results)))]]) # On logit scale - - - # - # Climate - # - - tClimateMean <- tClimateVar <- NULL # Logit-scaled IVW for each climate - - mod.converged <- NULL - for (i in 1:nModels) {mod.converged[i] <- landscape.store[[i]]$converged} - - # If there SE is 0 for a coefficient, fix it to a small value to avoid INF - p1.climate.se[p1.climate.se == 0] <- 0.0001 - - for (i in 1:ncol(p1.climate)) { - - # Note for me, the SE should be the mean not the sum. Otherwise the SE will converge to 0 - tClimateMean[i] <- sum(p1.climate[mod.converged ,i] / p1.climate.se[mod.converged ,i]^2) / sum(1/p1.climate.se[mod.converged ,i]^2)# IVW mean of nModels for each veg type, on transformed scale - # Double check this tTypeVar[i]<- 1/mean(1/p1.se[mod.converged ,i]^2 ) # IVW variance of mean... - tClimateVar[i]<- 1/sum(1/p1.climate.se[mod.converged ,i]^2 ) # IVW variance of mean... - - } - - # Add the intercept coefficient to the table - coef.results["Intercept"] <- tClimateMean[1] # On logit scale - coef.se.results["Intercept"] <- tClimateVar[1] # On logit scale - - # Add the climate coefficient to the table - coef.results["Climate"] <- tClimateMean[2] # On logit scale - coef.se.results["Climate"] <- tClimateVar[2] # On logit scale - - # Store protocol coefficient if modeled - if (protocol.flag == TRUE) { - - coef.results["Protocol"] <- tClimateMean[3] # On logit scale - coef.se.results["Protocol"] <- tClimateVar[3] # On logit scale - - } - - } - - # Don't currently use this method of weighting - # if (weight.method == "AIC") { - # - # - # # AIC calculation (I'm using AIC here, because this is primarily for prediction, rather than finding a minimial best model) - # aic.ta <- rep(999999999, (nModels)) - # - # for (i in 1:(nModels)) { - # - # if (!is.null(landscape.store[[i]]) & class(landscape.store[[i]])[1] != "try-error") { # last part is to not used non-converged models, unless none converged - # aic.ta[i] <- AICc(landscape.store[[i]]) - # } - # - # } - # - # aic.delta <- aic.ta - min(aic.ta) - # aic.exp <- exp(-1 / 2 * aic.delta) - # aic.wt.ta <- aic.exp / sum(aic.exp) - # - # tTypeMean <- tTypeVar <- NULL # Logit-scaled model averages for each veg type - # - # aic.wt.ta.adj <- ifelse(aic.wt.ta < 0.01, 0, aic.wt.ta) # Use adjusted weight to avoid poor fitting models with extreme values for certain types (usually numerically equivalent to 0's or 1's) - # aic.wt.ta.adj <- aic.wt.ta.adj / sum(aic.wt.ta.adj) - # - # for (i in 1:ncol(p1)) { - # - # tTypeMean[i] <- sum(aic.wt.ta.adj * p1[, i]) - # tTypeVar[i] <- sum(aic.wt.ta.adj * p1.se[, i]) - # - # } - # - # # Add names - # names(tTypeMean) <- names(tTypeVar) <- rownames(prediction.matrix) - # - # # Adjust the paspen variable as it isn't in all of the models - # mod.converged[p1[, "paspen"] == 0] <- FALSE - # tTypeMean["paspen"] <- sum(p1[mod.converged ,"paspen"] / p1.se[mod.converged ,"paspen"]^2) / sum(1/p1.se[mod.converged ,"paspen"]^2)# IVW mean of nModels for each veg type, on transformed scale - # tTypeVar["paspen"] <- 1/sum(1/p1.se[mod.converged ,"paspen"]^2 ) # IVW variance of mean... - # - # # Put coefficients in Coef matrix (and SE's) - # coef.results[na.omit(match(names(tTypeMean), names(coef.results)))] <- tTypeMean[names(coef.results)[na.omit(match(names(tTypeMean), names(coef.results)))]] # On logit scale - # coef.se.results[na.omit(match(names(tTypeVar), names(coef.se.results)))] <- sqrt(tTypeVar[names(coef.se.results)[na.omit(match(names(tTypeVar), names(coef.se.results)))]]) # On logit scale - # - # - # } - - # - # 7.0 Adjust coefficients that are often estimated poorly. - # - - if (coef.adjust == TRUE) { # If true, adjust coefficients, otherwise leave as is. Can help to determine how much the adjustment is needed - - coef.results["HardLin"] <- (coef.results["HardLin"] / coef.se.results["HardLin"]^2 + coef.results["UrbInd"]/coef.se.results["UrbInd"]^2) / (1 / coef.se.results["HardLin"]^2 + 1 / coef.se.results["UrbInd"]^2) # Inverse-variance weighted, done on logit scale - coef.se.results["HardLin"] <- sqrt(1 / (1 / coef.se.results["HardLin"]^2 + 1 / coef.se.results["UrbInd"]^2)) - - } - - # Store the coefficients as a single object - names(coef.se.results) <- paste0(names(coef.se.results), ".SE") - coef.results <- c(coef.results, coef.se.results) - return(coef.results) - -} diff --git a/1_code/_deprecated/plants_functions/model-validation_functions.R b/1_code/_deprecated/plants_functions/model-validation_functions.R deleted file mode 100644 index f4a31c1..0000000 --- a/1_code/_deprecated/plants_functions/model-validation_functions.R +++ /dev/null @@ -1,232 +0,0 @@ -# --- -# title: Model validation functions -# author: Brandon Allen -# created: 2025-02-11 -# notes: Define functions that are used for model validation. -# --- - -#' [Model validation] -#' -#' [Calculation of AUC statistics based on the final hierachical model. Implements two prediction approaches and climate truncation.] -#' -#' @param [species] [Focal species of interest.] -#' @param [data] [Cleaned species, climate, and landcover data.] -#' @param [boot.data] [List of site codes to be included for each species and bootstrap iteration.] -#' @param [climate.coef] [List of space-climate coefficients calculated from the detection_models function.] -#' @param [landcover.coef] [List of landcover coefficients calculated from the detection_models function.] -#' @param [landcover.type] [Defines if the provided landcover coefficients are Vegetation or Soil models.] -#' @param [protocol.flag] [Flag that determines if we need to account for differences in sample protocols] -#' @param [boot] [Defines bootstrap iteration.] -#' @return [Outputs vector of model coefficients defined in climate.models and space.models.] -#' - -model_validation <- function(species, data, boot.data, - climate.coef, landcover.coef, - landcover.type, protocol.flag, boot) { - - # - # 1.0 Libraries - # - - require(pROC) - - # - # 2.0 Data standardization - # - - # Since we are treating each recording individually, we fix visit to 1 as we no longer need to apply this weight. - data["nQuadrant"] <- 1 - data$visit <- 1 - - # Create new column which is the observed count data (used for weighting in the age relationships), one for the converted probabilities - data["Count"] <- as.integer(ifelse(data[, species] > 0, 1, 0)) - data["pcount"] <- as.integer(ifelse(data[, species] > 0, 1, 0) / data$nQuadrant) - - # 3.0 Bootstrap Selection - boot.data <- boot.data[[species]] - site.id <- boot.data[, paste0("Boot_", boot)] - data <- data[data$SiteYearQu %in% site.id, ] - - # - # 4.0 Climate Predictions - # - - # Using the matching climate model, create the prediction that will be used - # Add the intercept and extract climate coefficients - data$Intercept <- 1 - climate.dect.coef <- climate.coef[[species]][boot, ] - climate.veg <- as.matrix(data[, names(climate.dect.coef)]) - - # Predict space/climate component (Probability scale) - climate.prediction <- plogis(drop(climate.veg %*% climate.dect.coef)) - climate.prediction.truncated <- ifelse(climate.prediction >= quantile(climate.prediction, 0.99), - quantile(climate.prediction, 0.99), - climate.prediction) - - # - # 5.0 Landcover Predictions and statistics - # - - if(landcover.type == "Vegetation") { - - # - # Evaluation using landcover + climate - # - - veg.dect.coef <- vegetation.coef[[species]][boot, ] - - # If protocol flag is present, subtract protocol coefficient from the values - # for "Old" surveys. This is done because the coefficients are based on the - # "New" protocol. - if(protocol.flag == TRUE) { - - # Align the vegetation information and coefficients - column.remove <- -(1:3) - veg.in <- as.matrix(data[, names(veg.dect.coef)[column.remove]]) - - # Predict without space climate component - data$Vegetation <- drop(veg.in %*% plogis(veg.dect.coef[column.remove])) - - # If protocol == "old" remove the protocol coefficient - data$Vegetation[data$Protocol == "Old"] <- data$Vegetation[data$Protocol == "Old"] - as.numeric(veg.dect.coef["Protocol"]) - - } else { - - # Align the vegetation information and coefficients - column.remove <- -(1:2) - veg.in <- as.matrix(data[, names(veg.dect.coef)[column.remove]]) - - # Predict without space climate component - data$Vegetation <- drop(veg.in %*% plogis(veg.dect.coef[column.remove])) - - } - - # Predict with space climate component - climate.component <- climate.prediction * veg.dect.coef["Climate"] - climate.component.truncated <- climate.prediction.truncated * veg.dect.coef["Climate"] - data$VegetationClim <- plogis(qlogis(drop(veg.in %*% plogis(veg.dect.coef[column.remove]))) + climate.component) - data$VegetationClimTrunc <- plogis(qlogis(drop(veg.in %*% plogis(veg.dect.coef[column.remove]))) + climate.component.truncated) - - # Statistics - # Climate uses the entire grid - climate.auc <- auc(data$pcount, climate.prediction) - climate.truncated.auc <- auc(data$pcount, climate.prediction.truncated) - - # Landcover only uses the section included in the model - data.region <- data[!(data$NR %in% c("Grassland")), ] - landcover.auc <- auc(data.region$pcount, data.region$Vegetation) - full.auc <- auc(data.region$pcount, data.region$VegetationClim) - full.truncated.auc <- auc(data.region$pcount, data.region$VegetationClimTrunc) - - # - # Evaluation using modified climate + landcover coefficients - # - - # Convert climate component to a climate matrix with same dimentions as the landcover - climate.component <- matrix(climate.component, nrow = nrow(veg.in), ncol = ncol(veg.in)) - climate.component.truncated <- matrix(climate.component.truncated, nrow = nrow(veg.in), ncol = ncol(veg.in)) - - # Isolate the landcover coefficients - veg.dect.coef <- veg.dect.coef[column.remove] - - # Joint landcover and climate prediction - veg.joint.coef <- t(t(climate.component) + veg.dect.coef) - veg.joint.truncated.coef <- t(t(climate.component.truncated) + veg.dect.coef) - - data$VegetationClimJoint <- rowSums(veg.in * plogis(veg.joint.coef)) - data$VegetationClimJointTrunc <- rowSums(veg.in * plogis(veg.joint.truncated.coef)) - - # Evaluation in the focal region - data.region <- data[!(data$NR %in% c("Grassland")), ] - full.joint.auc <- auc(data.region$pcount, data.region$VegetationClimJoint) - full.joint.truncated.auc <- auc(data.region$pcount, data.region$VegetationClimJointTrunc) - - } - - if(landcover.type == "Soil") { - - # - # Evaluation using landcover + climate - # - - soil.dect.coef <- soil.coef[[species]][boot, ] - - # If protocol flag is present, subtract protocol coefficient from the values - # for "Old" surveys. This is done because the coefficients are based on the - # "New" protocol. - if(protocol.flag == TRUE) { - - # Align the vegetation information and coefficients - column.remove <- -(1:4) - soil.in <- as.matrix(data[, names(soil.dect.coef)[column.remove]]) - - # Predict without space climate component - data$Soil <- drop(soil.in %*% plogis(soil.dect.coef[column.remove])) - - # If protocol == "old" remove the protocol coefficient - data$Soil[data$Protocol == "Old"] <- data$Soil[data$Protocol == "Old"] - as.numeric(soil.dect.coef["Protocol"]) - - } else { - - # Align the vegetation information and coefficients - column.remove <- -(1:3) - soil.in <- as.matrix(data[, names(soil.dect.coef)[column.remove]]) - - # Predict without space climate component - data$Soil <- drop(soil.in %*% plogis(soil.dect.coef[column.remove])) - - } - - # Predict the pAspen and climate components - paspen.component <- data$paspen * soil.dect.coef["paspen"] - climate.component <- climate.prediction * soil.dect.coef["Climate"] - climate.component.truncated <- climate.prediction.truncated * soil.dect.coef["Climate"] - - # Predict with space climate component - data$SoilClim <- plogis(qlogis(drop(soil.in %*% plogis(soil.dect.coef[column.remove]))) + paspen.component + climate.component) - data$SoilClimTrunc <- plogis(qlogis(drop(soil.in %*% plogis(soil.dect.coef[column.remove]))) + paspen.component + climate.component.truncated) - - # Statistics - # Climate uses the entire grid - climate.auc <- auc(data$pcount, climate.prediction) - climate.truncated.auc <- auc(data$pcount, climate.prediction.truncated) - - # Landcover only uses the section included in the model - data.region <- data[data$NR %in% c("Grassland", "Parkland"), ] - landcover.auc <- auc(data.region$pcount, data.region$Soil) - full.auc <- auc(data.region$pcount, data.region$SoilClim) - full.truncated.auc <- auc(data.region$pcount, data.region$SoilClimTrunc) - - # - # Evaluation using modified climate + landcover coefficients - # - - # Convert climate component to a climate matrix with same dimentions as the landcover - climate.component <- matrix((climate.component + paspen.component), nrow = nrow(soil.in), ncol = ncol(soil.in)) - climate.component.truncated <- matrix((climate.component.truncated + paspen.component), nrow = nrow(soil.in), ncol = ncol(soil.in)) - - # soil.dect.coef the landcover coefficients - soil.dect.coef <- soil.dect.coef[column.remove] - - # Joint landcover and climate prediction - soil.joint.coef <- t(t(climate.component) + soil.dect.coef) - soil.joint.truncated.coef <- t(t(climate.component.truncated) + soil.dect.coef) - - data$SoilClimJoint <- rowSums(soil.in * plogis(soil.joint.coef)) - data$SoilClimJointTrunc <- rowSums(soil.in * plogis(soil.joint.truncated.coef)) - - # Evaluation in the focal region - data.region <- data[data$NR %in% c("Grassland", "Parkland"), ] - full.joint.auc <- auc(data.region$pcount, data.region$SoilClimJoint) - full.joint.truncated.auc <- auc(data.region$pcount, data.region$SoilClimJointTrunc) - - } - - model.fit <- c(climate.auc, climate.truncated.auc, landcover.auc, - full.auc, full.truncated.auc, full.joint.auc, full.joint.truncated.auc) - names(model.fit) <- c("Climate", "Climate_Truncated", "Landcover", - "Full", "Full_Truncated", "Full_Joint", "Full_Joint_Truncated") - return(model.fit) - -} - diff --git a/1_code/_deprecated/plants_module/spec.R b/1_code/_deprecated/plants_module/spec.R deleted file mode 100644 index f65f29c..0000000 --- a/1_code/_deprecated/plants_module/spec.R +++ /dev/null @@ -1,207 +0,0 @@ -# --- -# title: Plant-Group Taxon Specs -# author: Brendan Casey -# created: 2026-09-09 -# inputs: none -# outputs: none; returns objects in memory -# notes: -# - The v2 configuration for the four plant-group taxa, as data. -# "Plant group" names a survey design rather than a taxonomy: -# vascular plants, bryophytes, lichens and soil mites share -# one design and one set of scripts, soil mites included. -# - Parity coverage is the climate stage. The v2 habitat stage -# uses inverse-variance model averaging with a coefficient -# adjustment and post-hoc GAM age splines, none of which the -# harness implements yet; `habitat_v2_ready` records that per -# taxon so a run cannot quietly claim more than it did. -# - Protocol is fitted for bryophytes and lichens and not for -# mites or vascular plants. That asymmetry is v2's, preserved -# here for parity, and `use_protocol` exposes it so an -# experiment can equalize across taxa. See the parity ledger -# in docs/framework_design.md. -# - The engine is bayesglm, which the v2 plant scripts use -# because rare species fitted against 30-term habitat -# formulas separate completely and plain glm then returns -# infinite coefficients. The regularization is load-bearing. -# - Regions are the two model sets: north is fitted off the -# grassland, south on grassland and parkland. v2 applies -# those filters inside its model functions; they are stated -# here instead, which is numerically the same. -# --- - -# 1. Setup ---- - -## 1.1 Load packages ---- -# Base R only; the harness supplies everything else. - -# 2. plant_spec() ---- - -#' Build the v2 Spec for One Plant-Group Taxon -#' -#' @param taxon Character. One of "bryophyte", "lichen", "mite" -#' or "vascular_plant". -#' @param use_protocol Logical, or NULL to take the v2 value for -#' that taxon. Setting it equalizes a term v2 applies -#' inconsistently, which is what a cross-taxa experiment wants. -#' @return A spec list, as run_spec() consumes. -#' -#' @example # Example usage of the function -#' # spec <- plant_spec("bryophyte") -#' # spec$stages[[1]]$engine -plant_spec <- function(taxon, use_protocol = NULL) { - known <- c("bryophyte", "lichen", "mite", "vascular_plant") - - if (!taxon %in% known) { - stop( - "Unknown plant-group taxon `", taxon, "`. Known: ", - paste(known, collapse = ", "), - call. = FALSE - ) - } - - # v2 fits Protocol for bryophytes and lichens only. - v2_protocol <- taxon %in% c("bryophyte", "lichen") - - if (is.null(use_protocol)) { - use_protocol <- v2_protocol - } - - # The habitat sets are spliced from the mite script, which - # fits no Protocol. In the bryophyte and lichen scripts the - # same formulas carry the term directly after Climate and - # nowhere else, so inserting it there reproduces them. The - # climate set carries no Protocol in any variant. - with_protocol <- function(set) { - models <- get_model_set(set) - - if (!use_protocol) { - return(models) - } - - sub( - ". ~ . + Climate", ". ~ . + Climate + Protocol", - models, fixed = TRUE - ) - } - - list( - taxon = taxon, - response_name = "response", - - # Plant responses are cover or abundance values; v2 models - # presence, so anything above zero is a detection. - response_transform = "detection", - family = "binomial", - weight_column = NULL, - tier = NULL, - - regions = list( - north = list( - filter = ~ nr != "Grassland", - grid = "veg", - term_block = "veg", - habitat_models = with_protocol("habitat_veg_plant_v2") - ), - south = list( - filter = ~ nr %in% c("Grassland", "Parkland"), - grid = "soil", - term_block = "soil", - habitat_models = with_protocol("habitat_soil_plant_v2") - ) - ), - - stages = list( - list( - name = "climate", - models = "climate_plant_v2_full", - engine = "bayesglm", - selection = "aic_average", - ic = "AICc", - control = list(maxit = 250), - # The habitat stage takes this stage's prediction as a - # term called Climate, which is how v2 composes them. - carry_as = "Climate" - ), - list( - name = "habitat", - # Set per region below, because north fits vegetation - # types and south fits soil types. - models = NULL, - engine = "bayesglm", - selection = "ivw_grid", - ic = "AICc", - control = list(maxit = 250), - carry_from = "climate", - carry_from_as = "Climate", - # v2 predicts each candidate onto the grid holding - # Climate at zero, and at the new protocol where Protocol - # is fitted, so the habitat effects are read at an - # average climate under one protocol. - grid_constants = if (use_protocol) { - list( - Climate = 0, - Protocol = factor("New", levels = c("New", "Old")) - ) - } else { - list(Climate = 0) - }, - head_terms = if (use_protocol) { - c("Intercept", "Climate", "Protocol") - } else { - c("Intercept", "Climate") - }, - coef_adjust = TRUE - ) - ), - - resample = list( - scheme = "spatial_block", - min_detections = 20L, - seed = NULL - ), - - # The habitat stage runs, but not all of it: v2 overwrites 45 - # of the 87 vegetation effects with GAM age splines over - # stand age, which the harness does not fit. See the notes. - habitat_v2_ready = "partial", - use_protocol = use_protocol, - protocol_is_v2 = identical(use_protocol, v2_protocol), - - notes = paste0( - "v2 climate stage, and the habitat stage less its age ", - "splines. v2 refits the five aged stand types - white ", - "spruce, pine, deciduous, mixedwood and black spruce, nine ", - "age classes each - with GAM splines over stand age and ", - "overwrites 45 of the 87 vegetation effects with them, so ", - "those 45 will not match. The other 42, the intercept, ", - "climate and the soil effects are reproduced. ", - "Protocol ", if (use_protocol) "fitted" else "not fitted", - if (identical(use_protocol, v2_protocol)) { - " (the v2 setting for this taxon)." - } else { - " (overridden; v2 does the opposite for this taxon)." - } - ) - ) -} - -# 3. plant_specs() ---- - -#' Every Plant-Group Spec -#' -#' @param use_protocol Logical, or NULL for the v2 setting per -#' taxon. -#' @return A named list of specs. -#' -#' @example # Example usage of the function -#' # names(plant_specs()) -plant_specs <- function(use_protocol = NULL) { - taxa <- c("bryophyte", "lichen", "mite", "vascular_plant") - - stats::setNames( - lapply(taxa, plant_spec, use_protocol = use_protocol), - taxa - ) -} - -# End of script ---- diff --git a/1_code/_scratch.R b/1_code/_scratch.R index 9bf49b2..a3df377 100644 --- a/1_code/_scratch.R +++ b/1_code/_scratch.R @@ -26,8 +26,3 @@ # --- # End of script ---- - -## 2026-09-09 ---- -load( - "\\\\abmi-data2\\science\\sc\\sdmMethodsDev\\0_data\\data_snapshots\\model_ready_v2\\mite-model-data.Rdata" -) diff --git a/1_code/_setup/00_mirror_setup_inputs.R b/1_code/_setup/00_mirror_setup_inputs.R new file mode 100644 index 0000000..4a2430a --- /dev/null +++ b/1_code/_setup/00_mirror_setup_inputs.R @@ -0,0 +1,268 @@ +# --- +# title: Mirror the _setup Inputs onto ABMI-DATA2 +# author: Brendan Casey +# created: 2026-10-05 +# inputs: +# the original locations of every external file _setup/ reads, +# listed in section 1.3: +# - //ABMI-DATA2/science/sc/sdmMethodsDev/0_data/ +# data_snapshots/model_ready_v2/ +# - //ABMI-DATA2/science/sc/AB_data_v2023/sites/processed/ +# climate/ +# - //ABMI-DATA2/science/sc/ToEmily/VegetationModels/ +# - the work_abmi, BirdModels and ABMI Mammals Google drives +# outputs: +# in //ABMI-DATA2/science/sdmMethodsDev/0_data/setup_inputs/ +# (SDM_SETUP_INPUT_ROOT overrides it): +# - /..., byte-for-byte copies, under each source's +# own relative paths +# - inputs_manifest.csv: per file, the original path, the +# mirror path, bytes, md5, source modification time and +# the date it was copied +# - README.md: what the folder is and the rules for it +# notes: +# - Makes _setup/ runnable by anyone with ABMI-DATA2 access. +# Originals are never modified. +# - Never overwrites the mirror: mirrored files are md5-checked +# and skipped, and a changed source stops the script (move the +# old copy aside to re-mirror). Copies go via a .partial name. +# - Not mirrored, deliberately: +# - the v2 coefficient outputs on the drives (the bird +# coefficient CSVs, COEFS.RData, the mammal coefficient +# tables). Parity is scored against ABMIexploreR (08), so +# nothing in _setup/ reads them. +# - _setup/08's species-coefs.RData, which 08 downloads from +# the public ABMIexploreR repository at a pinned commit. +# - 0_data/v2_scripts/, which is tracked in git. +# - Run once, from the repository root, by someone who can +# reach every source. Reads are serial: the Google Drive +# mount refuses concurrent reads. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # manifest reading and writing (version: 1.16.4) + +## 1.2 Resolve the mirror ---- +project_root <- normalizePath(getwd(), winslash = "/") +helper <- file.path(project_root, "1_code/_setup/utils/input_paths.R") + +if (!file.exists(helper)) { + stop("Run this from the repository root.", call. = FALSE) +} + +source(helper) + +manifest_path <- setup_input("inputs_manifest.csv") + +## 1.3 The inventory ---- +# One entry per source: where it lives, and the paths below it +# that _setup/ reads. A path that is a folder is copied whole. +# The mirror keeps each path below a folder named for the source, +# so the scripts' roots (SDM_V2_PROJECT and the like) carry over. +sources <- list( + model_ready_v2 = list( + from = paste0( + "//ABMI-DATA2/science/sc/sdmMethodsDev/0_data/", + "data_snapshots/model_ready_v2" + ), + paths = "." + ), + ab_data_v2023 = list( + from = paste0( + "//ABMI-DATA2/science/sc/AB_data_v2023/sites/processed/", + "climate" + ), + paths = "abmi-camera-climate_2023.Rdata" + ), + vegetation_models = list( + from = "//ABMI-DATA2/science/sc/ToEmily/VegetationModels", + paths = c( + "0_data/lookup/prediction-matrix", + "0_data/species/processed/bryophyte-model-data.Rdata", + "0_data/bootstrap/bryophyte-bootstrap-ids.Rdata", + "0_data/bootstrap/lichen-bootstrap-ids.Rdata", + "0_data/bootstrap/mite-bootstrap-ids.Rdata", + "0_data/bootstrap/vascular-plant-bootstrap-ids.Rdata" + ) + ), + birds_data_v2 = list( + from = paste0( + "G:/Shared drives/work_abmi/1_projects/active/", + "sdmMethodsDev/remote/birds_data_v2" + ), + paths = "Stratified.Rdata" + ), + bird_models = list( + from = paste0( + "G:/.shortcut-targets-by-id/", + "17Ymt13eHfKvIiuoMl6x-Kn74Z2uVbbzS/BirdModels" + ), + paths = c( + "Data/lookups/Xn-veg-v2024.Rdata", + "Data/lookups/birdlist.csv" + ) + ), + abmi_mammals = list( + from = "G:/Shared drives/ABMI Mammals", + paths = c( + paste0( + "Results/Habitat Modeling/2024/Climate/Predictions/", + "All Species Climate Predictions.csv" + ), + "Data/Lookup Tables/WildTrax Species Strings.RData" + ) + ) +) + +# 2. List the files ---- +# Stops before copying if any source is unreachable + +## 2.1 Expand each source ---- +files <- rbindlist(lapply(names(sources), function(name) { + src <- sources[[name]] + + rel <- unlist(lapply(src$paths, function(path) { + full <- file.path(src$from, path) + + if (dir.exists(full)) { + inside <- list.files(full, recursive = TRUE) + if (path == ".") inside else file.path(path, inside) + } else { + path + } + })) + + data.table( + source = name, + from = file.path(src$from, rel), + to = file.path(name, rel) + ) +})) + +## 2.2 Check every source is reachable ---- +missing <- files$from[!file.exists(files$from)] + +if (length(missing) > 0) { + stop( + "Source file(s) not found:\n ", + paste(missing, collapse = "\n "), + call. = FALSE + ) +} + +cat( + nrow(files), " files, ", + round(sum(file.size(files$from)) / 1e9, 2), " GB, from ", + length(sources), " sources\n", + sep = "" +) + +# 3. Copy ---- + +## 3.1 The previous manifest, to keep first-copy dates ---- +previous <- if (file.exists(manifest_path)) { + fread(manifest_path, colClasses = "character") +} else { + NULL +} + +## 3.2 Copy each file ---- +rows <- vector("list", nrow(files)) + +for (i in seq_len(nrow(files))) { + from <- files$from[i] + to <- setup_input(files$to[i]) + source_md5 <- unname(tools::md5sum(from)) + + if (file.exists(to)) { + # Already mirrored: the copy must still match its source + if (!identical(unname(tools::md5sum(to)), source_md5)) { + stop( + "The source has changed since it was mirrored:\n ", + from, "\nThe mirror is never overwritten. Move\n ", to, + "\naside first if the new version should replace it.", + call. = FALSE + ) + } + + copied <- if (is.null(previous)) { + NA_character_ + } else { + previous$copied[match(files$to[i], previous$path)] + } + status <- "present" + } else { + dir.create(dirname(to), recursive = TRUE, showWarnings = FALSE) + partial <- paste0(to, ".partial") + + if (!file.copy(from, partial, overwrite = TRUE, + copy.date = TRUE)) { + stop("Copy failed:\n ", from, call. = FALSE) + } + + if (!identical(unname(tools::md5sum(partial)), source_md5)) { + unlink(partial) + stop("md5 mismatch after copying:\n ", from, call. = FALSE) + } + + if (!file.rename(partial, to)) { + stop("Could not rename into place:\n ", to, call. = FALSE) + } + + copied <- NA_character_ + status <- "copied" + } + + rows[[i]] <- data.table( + source = files$source[i], + path = files$to[i], + original = from, + bytes = file.size(from), + md5 = source_md5, + source_modified = format( + file.mtime(from), "%Y-%m-%d %H:%M:%S" + ), + copied = if (is.na(copied)) format(Sys.Date()) else copied + ) + + cat(sprintf("[%d/%d] %s %s\n", i, nrow(files), status, files$to[i])) +} + +# 4. Record ---- + +## 4.1 The manifest ---- +# Rows for files dropped from the inventory are kept as +# provenance for copies still in the mirror +manifest <- rbindlist(rows) + +if (!is.null(previous)) { + kept <- previous[!path %in% manifest$path] + manifest <- rbind(manifest, kept, fill = TRUE) +} + +setorder(manifest, source, path) +fwrite(manifest, manifest_path) + +## 4.2 The README ---- +writeLines(c( + "# _setup inputs for sdmMethodsDev", + "", + "Byte-for-byte copies of every external file that", + "`1_code/_setup/` in the sdmMethodsDev repository reads, so", + "anyone with access to this share can rebuild the test", + "dataset. Written by `1_code/_setup/00_mirror_setup_inputs.R`.", + "", + "- Do not edit, rename or delete anything here. The _setup", + " scripts read these paths by default.", + "- `inputs_manifest.csv` gives each file's original location,", + " size, md5 and the date it was copied.", + "- The originals were left in place; the manifest names them.", + "- 04 and 05's inputs are not here: their output,", + " `v2_results/`, is published frozen under `../v2_results/`." +), setup_input("README.md")) + +cat("\nWrote", manifest_path, "\n") + +# End of script ---- diff --git a/1_code/_setup/01_harmonize_model_ready_v2.R b/1_code/_setup/01_harmonize_model_ready_v2.R index 018db80..523fcd8 100644 --- a/1_code/_setup/01_harmonize_model_ready_v2.R +++ b/1_code/_setup/01_harmonize_model_ready_v2.R @@ -3,6 +3,8 @@ # author: Brendan Casey # created: 2026-09-05 # inputs: +# Read via setup_input() (utils/input_paths.R); the original +# locations are listed, and SDM_* variables can point back. # from 0_data/data_snapshots/model_ready_v2/ on ABMI-DATA2: # - vascular-plant-model-data.Rdata # - bryophyte-model-data.Rdata @@ -43,107 +45,30 @@ # - covariate_columns.csv which block each covariates # column came from, per taxon # notes: -# Source files use two incompatible survey designs, so -# they are harmonized into one shared convention rather than -# one table: every taxon CSV is `survey_unit_id` followed by -# that taxon's species columns, and every identity, design and -# location field is factored out into sites.csv. Join any taxon -# CSV to sites.csv on survey_unit_id. -# -# The response tables above carry detections only. The -# covariate and lookup tables carry everything else the v2 -# hierarchical models regress against, so the modelling code in -# the plant-group modules can run from 0_data/test_dataset/ -# alone rather than reaching back to the snapshot. -# -# covariates.csv is keyed on survey_unit_id AND taxon, where -# sites.csv is keyed on survey_unit_id alone. That is forced by -# the source: on the 7001 quadrats vascular plant and mite -# share, 154 of 162 vegetation columns and 58 of 59 soil -# columns hold different values, while climate agrees to the -# last digit. Bryophyte and lichen match vascular plant -# exactly. -# -# The covariates were never shared to begin with: -# 01a_data-standardization.R lifts each taxon's veg and soil -# blocks wholesale from its own raw SpTable *_Quad.RData file, -# so the four are four separate extractions. The quadrats -# themselves are the same places - Lat, Long, Easting, -# Northing and NSR are identical on every shared unit. -# -# Where mite differs is concentrated in the Surr* surrounding -# buffer terms: 20 columns carry 67 per cent of the differing -# veg cells and 80 per cent of the soil ones. It is not a -# footprint vintage, since every year from 2007 to 2022 -# disagrees. That points at a different spatial extraction -# upstream, but the raw files are in neither this repository -# nor the snapshot, so it cannot be confirmed here. -# -# Either way they are two descriptions of one quadrat, and -# choosing between them is not this script's call. Filter on -# taxon before joining. -# -# Its columns are the union across taxa, so each taxon fills -# its own blocks and leaves the rest empty - about 16 per cent -# of cells are populated. Drop the empty columns after -# filtering to a taxon. -# -# A name that more than one block carries is suffixed with its -# block: the plant files spell 42 footprint terms the same way -# in the veg and soil blocks, and 14 of those hold different -# values because each is computed under its own landcover -# framework. So `Crop_veg` and `Crop_soil` are both kept. -# lookup/covariate_columns.csv maps every column back to the -# block and the v2 spelling it came from. -# -# Mammal covariates come from the same two SpTable files as -# the mammal response, because those files are themselves the -# output of the v2 scripts in 0_data/v2_scripts/mammals/. Their -# embedded pred_matrix objects were checked against -# prediction-matrix_north.csv and Prediction matrix for ABMI -# South coefficients 2020.csv on the ABMI Mammals drive and are -# identical, so no separate lookup is needed for them. -# -# Mammals contribute two taxon values, mammal_north and -# mammal_south, rather than one. They are separate models on -# overlapping deployments - 512 location_project values appear -# in both files - so their covariates cannot be pooled the way -# the response is. -# -# Birds come from one file, Stratified.Rdata, rather than from -# the snapshot. It is the output of a stratification stage that -# is not in 0_data/v2_scripts/birds/ - those scripts start at -# 06 and take it as given - so this script reads it where it -# sits on the BirdModels drive and never writes there. -# -# Birds need two things no other taxon does. Every survey has a -# QPAD offset per species, written as bird_offsets.csv, which -# is response-shaped rather than a covariate and is by far the -# largest table here. And every survey is separately eligible -# for the north and south models, so eligibility is two flags -# in covariates.csv rather than one region in sites.csv; a -# survey can feed both models, or neither. -# -# Bird counts are integer point counts. Plants are 0/1 and -# mammals are densities, so the three responses are on three -# different scales and are not comparable across taxa without -# the offsets and weights their own models apply. -# -# Set SDM_BIRD_DATA to read Stratified.Rdata from elsewhere. -# -# The prediction matrices come from the vegetation models -# project on ABMI-DATA2, not from the snapshot, because the -# snapshot's veg.pm lacks the four aggregate columns four of -# the twelve vegetation models are predicted onto. Set -# SDM_V2_LOOKUP to read them from elsewhere. If the folder is -# unreachable the snapshot objects are used instead and -# section 7.4 says so, because such a run cannot fit the -# vegetation models. -# -# Future improvement - the plant-group files are read twice, -# once for the response and once for the covariates. That is -# simpler to follow than threading one environment through both -# readers, and this script runs once per snapshot. +# - Every taxon CSV is survey_unit_id + species columns; +# identity, design and location fields go to sites.csv. The +# covariate and lookup tables let the modules run from +# 0_data/test_dataset/ alone. +# - covariates.csv is keyed on survey_unit_id AND taxon (taxa +# disagree on shared quadrats; see section 5.4). Filter on +# taxon before joining, then drop empty columns. +# - Names in more than one block are block-suffixed (e.g. +# Crop_veg, Crop_soil); lookup/covariate_columns.csv maps +# each column to its block and v2 spelling. +# - The plant-group mite covariates differ from vascular plant +# mostly in the Surr* buffer terms (20 columns carry 67% of +# differing veg cells, 80% of soil), in every year 2007-2022: +# likely a different upstream extraction, unconfirmable here. +# - Mammal covariates and prediction matrices come from the +# SpTable files (matrices checked identical to the ABMI +# Mammals drive CSVs); the climate block is replaced (1.6). +# - Birds come from Stratified.Rdata (1.10), read-only +# (SDM_BIRD_DATA overrides). Prediction matrices come from +# the vegetation models project (1.8; SDM_V2_LOOKUP +# overrides). +# - TODO: the plant-group files are read twice (response, then +# covariates); simpler than threading one environment, and +# this runs once per snapshot. # --- # 1. Setup ---- @@ -154,33 +79,23 @@ library(data.table) ## 1.2 Configure paths ---- project_root <- normalizePath(getwd(), winslash = "/") -# Directory containing the v2 model ready snapshot files on ABMI-DATA2 +# The _setup inputs mirrored on ABMI-DATA2 (setup_input()) +source(file.path(project_root, "1_code/_setup/utils/input_paths.R")) + snapshot_dir <- Sys.getenv( "SDM_SNAPSHOT_V2", - unset = paste0( - "//ABMI-DATA2/science/sc/sdmMethodsDev/0_data/", - "data_snapshots/model_ready_v2" - ) + unset = setup_input("model_ready_v2") ) -# Output is the frozen test dataset itself, so it lands in -# 0_data/test_dataset/. That folder is otherwise read-only: this -# script is what writes it, by hand and once per snapshot, and -# every experiment then treats the result as fixed. The CSVs are -# gitignored along with the rest of 0_data/. +# The frozen test dataset; only this script writes it output_dir <- file.path(project_root, "0_data/test_dataset") -# Lookups sit in their own folder so the response and covariate -# tables stay the obvious contents of test_dataset/. lookup_dir <- file.path(output_dir, "lookup") ## 1.3 Name the source files ---- -# Plant-group files share one schema, mammal files another. -# -# "Plant group" names a file layout, not a taxonomy. It covers -# vascular plants, bryophytes, lichens and soil mites, because -# all four are surveyed on the same site-year-quadrant design -# and arrive in the same three-frame layout. +# "Plant group" is a file layout (the site-year-quadrant design), +# not a taxonomy: vascular plants, bryophytes, lichens and soil +# mites. Mammal files have another schema. plant_files <- c( vascular_plant = "vascular-plant-model-data.Rdata", bryophyte = "bryophyte-model-data.Rdata", @@ -200,12 +115,9 @@ mammal_files <- c( ) ## 1.4 Name the plant-group non-species columns ---- -# Plant-group frames are identity + species + site and climate. -# Species are whatever is left once the other two are named, so -# both are listed here rather than found by position: column -# order is an accident of how the files were built, and `Protocol` -# already sits last in two of the four files rather than with the -# site block it belongs to. +# Species are whatever is left once identity and site columns +# are named; column order is not reliable (`Protocol` sits last +# in two of the four files). # The six identity columns every plant-group frame opens with. plant_id_cols <- c( @@ -274,10 +186,8 @@ plant_site_cols <- c( "Protocol" ) -# Of those, the fields sites.csv already carries. They are -# dropped from the climate covariate table so each value has one -# home, and the model-data loader joins them back from sites.csv -# under their v2 names. +# Fields sites.csv carries, dropped from the climate table; the +# loader joins them back under their v2 names plant_site_cols_in_sites <- c( "Lat", "Long", @@ -289,17 +199,9 @@ plant_site_cols_in_sites <- c( "Northing" ) -# And the ones that are products of other stored columns. They -# are not written: the model-data loader recomputes them with the -# same expressions 01a_data-standardization.R used. -# -# Storing them would be worse than redundant. A CSV holds about -# 15 significant digits, so Northing lands within ~2e-9 of the -# source value; squaring it lifts that to ~0.05 in absolute -# terms. Written and read back independently, Northing2 would no -# longer be exactly Northing * Northing, and a model fitting both -# would see an inconsistent pair. Deriving keeps them consistent -# with whatever precision the stored columns carry. +# Products of stored columns, not written: the loader recomputes +# them as 01a_data-standardization.R did. Stored separately they +# would drift apart at CSV precision (Northing2 off by ~0.05). plant_site_cols_derived <- c( "Easting2", "Northing2", @@ -311,13 +213,9 @@ plant_site_cols_derived <- c( ) ## 1.5 Name the mammal column blocks ---- -# The mammal files are one frame, `d`, holding design fields, -# species-season columns and covariates. The species span is -# given by the file's own first_sp_col/last_sp_col indices, and -# the rest is split by naming the two non-habitat blocks; what -# is left over is habitat. That is the same split-by-elimination -# the plant-group reader uses, and section 6.6 checks that the -# three blocks account for every column. +# The mammal frame `d`: species from its first_sp_col and +# last_sp_col indices; the two non-habitat blocks are named +# below and habitat is the remainder (checked in 6.6). # Design and identity fields. sites.csv already carries these, # so they are dropped from the covariate tables. @@ -358,32 +256,19 @@ mammal_climate_cols <- c( ) ## 1.6 Name the mammal camera climate ---- -# The SpTable files carry a climate block, but it is not the one -# the mammal models were fitted against. The v2 climate pipeline -# joins this file instead, and at matched deployments the two -# disagree by up to 531 mm of MAP - they are different -# extractions. This file is therefore the authority for the -# mammal climate block, and its columns replace the SpTable's -# rather than sitting beside them. -# -# It is also the only source of CMD and TD for mammals, which -# four of the nine v2 climate models need. +# The v2 mammal climate pipeline joins this file, not the +# SpTable's climate block (they differ by up to 531 mm MAP at +# matched deployments), so its columns replace the SpTable's. +# Also the only mammal source of CMD and TD. mammal_camera_climate_file <- Sys.getenv( "SDM_MAMMAL_CAMERA_CLIMATE", - unset = paste0( - "//ABMI-DATA2/science/sc/AB_data_v2023/sites/processed/", - "climate/abmi-camera-climate_2023.Rdata" + unset = setup_input( + "ab_data_v2023", "abmi-camera-climate_2023.Rdata" ) ) -# What it supplies. Everything else in the file - coordinates, -# natural region, elevation - is already in sites.csv. -# -# pAspen is deliberately not taken from it. The file carries a -# pAspen, but v2's south habitat models fit the SpTable's own, -# a different extraction that differs by up to 0.92 at matched -# deployments; replacing it made every south candidate with -# pAspen disagree with v2. The SpTable's column is kept. +# Not pAspen: v2's south models fit the SpTable's own (up to +# 0.92 different at matched deployments) mammal_camera_climate_cols <- c( "MAT", "MWMT", "MCMT", "TD", "MAP", "MSP", "AHM", "SHM", "DD_0", "DD5", "DD_18", "DD18", "NFFD", "bFFP", "eFFP", @@ -392,39 +277,28 @@ mammal_camera_climate_cols <- c( ) ## 1.7 Name the mammal climate predictions ---- -# The mammal habitat models take a climate offset rather than -# fitting climate themselves: 03_basic-models.R reads this file -# for both regions. It is produced by a separate climate -# pipeline, so it is copied in rather than derived here. +# The climate prediction 03_basic-models.R reads for both +# regions, from a separate pipeline; copied in mammal_climate_pred_file <- Sys.getenv( "SDM_MAMMAL_CLIMATE_PRED", - unset = paste0( - "G:/Shared drives/ABMI Mammals/Results/Habitat Modeling/", - "2024/Climate/Predictions/", + unset = setup_input( + "abmi_mammals", + "Results/Habitat Modeling/2024/Climate/Predictions", "All Species Climate Predictions.csv" ) ) ## 1.8 Name the v2 prediction-matrix lookups ---- -# The v2 modelling scripts do not use the veg.pm and soil.pm -# objects carried in the snapshot. They read these two CSVs from -# the vegetation models project instead, and for veg the two are -# not the same grid. Over the same 43 habitat rows, the CSV adds -# Peatland, Mineral, Upland and CCR1234 - the aggregates that -# four of the twelve vegetation models are predicted onto - and -# drops MarshSwamp, CCPineR1, CCPine234, CCWhiteSpruceR1 and -# CCDecidMixedR1. The 51 columns they share hold identical -# values. The soil matrices are identical either way. -# -# So the CSVs are preferred, with the snapshot objects as the -# fallback if the drive is unreachable. Section 7.4 says which -# was used, because a fallback run cannot fit the vegetation -# models. +# The v2 scripts read these CSVs, not the snapshot's veg.pm and +# soil.pm. The veg CSV adds Peatland, Mineral, Upland and +# CCR1234 (needed by four of the twelve veg models) and drops +# MarshSwamp, CCPineR1, CCPine234, CCWhiteSpruceR1 and +# CCDecidMixedR1; shared columns are identical, as are the soil +# matrices. The snapshot is the fallback (reported in 7.4). v2_lookup_dir <- Sys.getenv( "SDM_V2_LOOKUP", - unset = paste0( - "//ABMI-DATA2/science/sc/ToEmily/VegetationModels/", - "0_data/lookup/prediction-matrix" + unset = setup_input( + "vegetation_models", "0_data/lookup/prediction-matrix" ) ) @@ -437,41 +311,28 @@ v2_lookup_files <- c( # The naming authority for mammal species. wt_species_file <- Sys.getenv( "SDM_WT_SPECIES", - unset = paste0( - "G:/Shared drives/ABMI Mammals/Data/Lookup Tables/", + unset = setup_input( + "abmi_mammals", "Data/Lookup Tables", "WildTrax Species Strings.RData" ) ) ## 1.10 Name the bird data package ---- -# One .Rdata holding every frame the bird models read: `covs`, -# `bird`, `off`, `boot` and `birdlist`. It is the output of the -# stratification stage, which is not in 0_data/v2_scripts/birds/ -# - those start at 06 and take this file as given. -# -# The copy read is the one v2's archived bird results were -# fitted on, kept in this project's remote/birds_data_v2/. The -# BirdModels drive's Data/Archive/2025/Stratified.Rdata is NOT -# it: that file was rebuilt in place on 2026-08-19 with 175 -# fewer surveys per draw and different draws, so no bird draw -# could match v2. With this copy, the harness reproduces v2's -# per-draw bird coefficients to ~1e-10 (checked 2026-10-02). -# -# The BirdModels drive is read-only for this repository. Nothing -# below writes to it. +# `covs`, `bird`, `off`, `boot` and `birdlist`, from a +# stratification stage not in 0_data/v2_scripts/birds/. Use the +# copy v2 was fitted on (remote/birds_data_v2/), NOT the +# BirdModels drive's Archive/2025 file, rebuilt on 2026-08-19 +# with different draws. With this copy the harness reproduces +# v2's per-draw bird coefficients to ~1e-10 (checked +# 2026-10-02). Read-only. bird_data_file <- Sys.getenv( "SDM_BIRD_DATA", - unset = paste0( - "G:/Shared drives/work_abmi/1_projects/active/", - "sdmMethodsDev/remote/birds_data_v2/Stratified.Rdata" - ) + unset = setup_input("birds_data_v2", "Stratified.Rdata") ) ## 1.11 Name the bird covariate blocks ---- -# Taken from the model definitions rather than guessed: every -# name below appears in a formula in 00.ClimateModels.R, -# 00.NorthModels.R or 00.SouthModels.R. Section 6.8 checks that -# the four blocks account for every column of `covs`. +# Every name appears in a formula in 00.ClimateModels.R, +# 00.NorthModels.R or 00.SouthModels.R (checked in 6.8) # The climate stage. EMT is carried because it sits with the # other climate normals in the source, but no formula uses it. @@ -483,10 +344,8 @@ bird_climate_cols <- c( "EMT" ) -# The north landcover stage. The human-footprint, method and -# water terms are shared with the south, so they appear in both -# blocks; each file is then the whole right-hand side of the -# models it feeds. +# The north landcover stage. Footprint, method and water terms +# are shared with the south block. bird_veg_cols <- c( "vegc", "wtAge", @@ -573,17 +432,11 @@ if (length(missing_files) > 0) { dir.create(output_dir, recursive = TRUE, showWarnings = FALSE) # 2. Helper functions ---- -# One reader per source family, plus a name normalizer for the -# mammal crosswalk and a duplicate-key guard shared by both. ## 2.1 load_rdata() ---- #' Load an .RData File into an Isolated Environment #' -#' Reads a saved workspace without touching the global -#' environment, so the six files cannot overwrite each other's -#' objects. -#' #' @param path Character. Path to an .RData file. #' @return An environment holding the file's objects. #' @@ -642,11 +495,8 @@ make_unique_ids <- function(ids, label) { #' Reduce a Species String to a Column-Safe Token #' -#' Collapses a common-name string to CamelCase with no -#' punctuation. -#' Apostrophes are deleted before splitting, so "Richardson's" -#' becomes "Richardsons" and not "Richardson" + "S"; the -#' remaining punctuation and spaces are split points. +#' CamelCase without punctuation; apostrophes are deleted first +#' ("Richardson's" becomes "Richardsons"). #' #' @param x Character. Species strings. #' @return Character. One token per input. @@ -677,11 +527,8 @@ squash_name <- function(x) { #' Resolve Source Species Names against the WildTrax Strings #' -#' Maps each source species name to its WildTrax common-name -#' string, which is the naming authority for mammals. An exact -#' match is tried first, then a punctuation-insensitive one: the -#' North file already uses the WildTrax strings verbatim, while -#' the South file uses their camel-cased form. +#' WildTrax is the mammal naming authority. Exact match first, +#' then punctuation-insensitive (the South file camel-cases). #' #' @param base Character. Source species names, no season suffix. #' @param native_sp Character. WildTrax common-name strings. @@ -777,19 +624,9 @@ normalize_mammal_column <- function(x, native_sp, label) { #' Read One Plant-Group Snapshot File #' -#' Despite the name, "plant group" is a file layout rather than a -#' taxon: this reads the vascular plant, bryophyte, lichen and -#' mite files, which share one site-year-quadrant schema. -#' -#' Splits a plant-group file into its species block and its site -#' block. Species are identified by elimination: every column -#' that is neither an identity column (`plant_id_cols`) nor a -#' site or climate column (`plant_site_cols`) is a species. -#' -#' Two checks keep the rule honest in both directions. Anything -#' wrongly counted as a species fails the 0/1 test, and anything -#' wrongly excluded is caught against the file's own -#' `veg.species.list` and `soil.species.list`. +#' Species are identified by elimination (neither +#' `plant_id_cols` nor `plant_site_cols`), checked both ways: a +#' 0/1 test, and the file's own species lists. #' #' @param path Character. Path to a plant-group .Rdata file. #' @param taxon Character. Taxon slug, used in messages and in @@ -820,8 +657,7 @@ read_plant_taxon <- function(path, taxon) { ) } - # Step 2: Species are what remains once identity and site - # columns are removed, so column order does not matter. + # Step 2: Species by elimination species_cols <- setdiff(cols, c(plant_id_cols, plant_site_cols)) # Step 3: Confirm the block really is presence/absence data. @@ -922,20 +758,10 @@ read_plant_taxon <- function(path, taxon) { #' Read One Plant-Group File's Covariate and Lookup Blocks #' -#' Companion to `read_plant_taxon()`, which takes the response. -#' This takes what the v2 hierarchical models regress against: -#' the climate block, the north vegetation and HF block, the -#' south soil and HF block, the two habitat prediction matrices, -#' and the lists naming which species each model set is fitted -#' for. -#' -#' The habitat blocks are found by difference rather than by -#' position. `climate.data` is identity + species + climate, so -#' whatever `veg.data` holds that `climate.data` does not is the -#' vegetation block, and likewise for soil. That reproduces the -#' positional spans the v2 scripts hard-code - `[403:489]` for -#' bryophytes, `[448:534]`, `[328:414]` and `[1408:1494]` for the -#' other three - without depending on column order staying put. +#' Habitat blocks are found by difference from `climate.data`, +#' reproducing the column spans the v2 scripts hard-code +#' (`[403:489]` for bryophytes, `[448:534]`, `[328:414]` and +#' `[1408:1494]` for the others) without relying on order. #' #' @param path Character. Path to a plant-group .Rdata file. #' @param taxon Character. Taxon slug, used in messages. @@ -965,9 +791,7 @@ read_plant_covariates <- function(path, taxon) { stringsAsFactors = FALSE ) - # Step 2: Take each habitat block by difference from - # climate.data, which leaves the covariates behind and drops - # the identity, species and climate columns both frames share. + # Step 2: Each habitat block, by difference from climate.data take_block <- function(frame) { block_cols <- setdiff(names(frame), clim_cols) @@ -981,10 +805,8 @@ read_plant_covariates <- function(path, taxon) { veg <- take_block(env$veg.data) soil <- take_block(env$soil.data) - # Step 3: The v2 coefficient template is the first 87 columns - # of this block, taken positionally. A snapshot that reordered - # or shortened it would silently rename every vegetation - # coefficient, so the width is checked rather than assumed. + # Step 3: v2 takes the first 87 columns positionally as its + # coefficient template, so check the width if (ncol(veg) - 1 < 87) { stop( taxon, @@ -995,9 +817,7 @@ read_plant_covariates <- function(path, taxon) { ) } - # Step 4: All three frames must describe the same survey units - # in the same order, or the tables cannot be recombined on - # survey_unit_id downstream. + # Step 4: All three frames must list the same units in order if ( !identical(climate$survey_unit_id, veg$survey_unit_id) || !identical(climate$survey_unit_id, soil$survey_unit_id) @@ -1010,17 +830,11 @@ read_plant_covariates <- function(path, taxon) { ) } - # Step 5: Record which species each model set is fitted for. - # These are subsets of the response columns, and the v2 scripts - # read them straight out of the .Rdata, so they have to travel - # with the dataset or the work queue cannot be rebuilt. + # Step 5: Which species each model set is fitted for (the v2 + # work queue) modelled <- union(env$veg.species.list, env$soil.species.list) - # The two lists are separately ordered, and the v2 scripts walk - # each in its own order. A single stacked table would lose that, - # so each species carries its position in whichever lists hold - # it, and the loader sorts on those rather than on the row - # order of this file. + # Each list keeps its own v2 order, as a position per list species_lists <- data.frame( taxon = taxon, species = modelled, @@ -1059,15 +873,9 @@ read_plant_covariates <- function(path, taxon) { #' Read the Mammal Camera Climate #' -#' Climate normals interpolated to each camera deployment. The -#' file is keyed on site and year, but the normals are 30-year -#' averages and so identical across years at one location; v2 -#' strips the year and keeps one row per location, which is -#' reproduced here. -#' -#' The CMU and NWSAR prefixes are project tags on the site name -#' rather than part of the location, and are stripped for the -#' same reason. +#' 30-year normals, so identical across years: as v2, the year +#' and the CMU and NWSAR project prefixes are stripped, leaving +#' one row per location. #' #' @param path Character. Path to abmi-camera-climate_*.Rdata. #' @param columns Character vector of climate columns to keep. @@ -1119,13 +927,8 @@ read_camera_climate <- function(path, columns) { #' Read One Mammal Snapshot File #' -#' Reads the camera tibble `d` and harmonizes its species-season -#' column names. The values are densities. -#' -#' The v2 script 02_process-data-files.R sets a season's -#' densities to NA when that season has fewer than 10 -#' camera-days, so an NA records too little effort to estimate -#' from. +#' Values are densities; NA means fewer than 10 camera-days that +#' season (v2's 02_process-data-files.R). #' #' @param path Character. Path to a mammal .RData file. #' @param region Character. "north" or "south". @@ -1218,12 +1021,8 @@ read_mammal_region <- function(path, region, native_sp) { stringsAsFactors = FALSE ) - # Step 5: Split the covariates off the same frame. The species - # span is already known, and the two non-habitat blocks are - # named in section 1.5, so habitat is what is left. The unit - # ids are reused rather than rebuilt: make_unique_ids() numbers - # duplicate deployments, and a second call would have to - # reproduce that numbering exactly. + # Step 5: Split the covariates off (section 1.5). Reuse the + # unit ids: make_unique_ids() numbered duplicate deployments. take_block <- function(wanted) { present <- intersect(wanted, names(d)) @@ -1237,18 +1036,9 @@ read_mammal_region <- function(path, region, native_sp) { climate <- take_block(mammal_climate_cols) - # Step 5b: Replace the SpTable's climate columns with the - # camera climate, which is what the v2 climate pipeline - # actually fitted against. The two are different extractions - - # they disagree by up to 531 mm of MAP at matched deployments - - # so taking some columns from one and some from the other would - # leave the block internally inconsistent. It is also the only - # source of CMD and TD, which four of the nine v2 climate - # models need. - # - # A deployment the climate file does not reach keeps NA rather - # than a substitute. v2 excludes those sites; a nearest-site - # value would be a different extraction again. + # Step 5b: Replace the climate block wholesale with the camera + # climate (section 1.6). Unmatched deployments keep NA, as v2 + # excludes them. camera_climate <- read_camera_climate( mammal_camera_climate_file, mammal_camera_climate_cols ) @@ -1289,9 +1079,7 @@ read_mammal_region <- function(path, region, native_sp) { ) habitat <- take_block(habitat_cols) - # The four blocks must partition `d` exactly. They are built by - # difference, so an overlap or a gap would quietly move a - # column into the wrong table. + # The four blocks must partition `d` exactly accounted <- length(species_cols) + length(habitat_cols) + length(intersect(mammal_site_cols, names(d))) + length(intersect(mammal_climate_cols, names(d))) @@ -1309,12 +1097,9 @@ read_mammal_region <- function(path, region, native_sp) { ) } - # Step 6: Record the work queue. Mammals model each season - # separately and at two occurrence thresholds - sp_table_* are - # the species with at least 20 detections, which get the full - # habitat model, and sp_table_*_ua those with at least 3, which - # get the use-availability model. One row per species-season, - # with its position in each list it belongs to. + # Step 6: The work queue: per season, sp_table_* (20+ + # detections, full habitat model) and sp_table_*_ua (3+, + # use-availability) season_lists <- list( summer = list( modelled = env$sp_table_summer, @@ -1376,17 +1161,9 @@ read_mammal_region <- function(path, region, native_sp) { #' Read the Bird Data Package #' -#' Splits Stratified.Rdata into the same shape the other taxa -#' use: a response table, covariate blocks, site rows, and the -#' lookups that say what to model. -#' -#' Birds differ from the other taxa in three ways that the output -#' has to carry. Each survey has a QPAD offset per species, so -#' the offsets are a second response-shaped table rather than a -#' covariate. Each survey is separately eligible for the north -#' and south models, so eligibility is two flags rather than one -#' region. And the counts are integer point counts, not the -#' plants' 0/1 or the mammals' densities. +#' Birds add a response-shaped QPAD offset table, eligibility as +#' two flags (north and south; a survey may feed both or +#' neither), and integer point counts. #' #' @param path Character. Path to Stratified.Rdata. #' @return A list with `species`, `offsets`, `sites`, `climate`, @@ -1416,12 +1193,9 @@ read_bird_data <- function(path) { counts <- as.data.frame(env$bird, stringsAsFactors = FALSE) offsets <- as.data.frame(env$off, stringsAsFactors = FALSE) - # Step 1: The three frames are stored row-aligned, but `covs` - # holds surveyid as double and the other two as integer, so - # identical() on them is FALSE for a reason that does not - # matter. Compare on a common type, and refuse to continue if - # the rows ever stop lining up: every table below is built by - # position, not by joining. + # Step 1: The frames must be row-aligned (tables below are + # built by position). Compare surveyid on a common type: `covs` + # holds it as double, the others as integer. key <- as.integer(covs$surveyid) if ( @@ -1441,10 +1215,8 @@ read_bird_data <- function(path) { unit_id <- paste0("bird|", key) - # Step 2: Counts are the response. Offsets are the same shape, - # restricted to the species that have counts: `off` carries 219 - # species and `bird` 127, and an offset with no count column - # cannot be modelled. + # Step 2: Counts, and offsets for species with counts (`off` + # has 219 species, `bird` 127) species_cols <- setdiff(names(counts), "surveyid") absent_offsets <- setdiff(species_cols, names(offsets)) @@ -1473,10 +1245,8 @@ read_bird_data <- function(path) { stringsAsFactors = FALSE ) - # Step 3: Take each covariate block by name. Shared footprint, - # method and water terms appear in both habitat blocks, so the - # blocks are not a partition of the covariate columns and are - # built by intersection rather than by difference. + # Step 3: Blocks by name (intersection): shared terms sit in + # both habitat blocks take_block <- function(wanted) { present <- setdiff( intersect(wanted, names(covs)), @@ -1496,17 +1266,13 @@ read_bird_data <- function(path) { soil <- take_block(bird_soil_cols) design <- take_block(bird_design_cols) - # date_time is POSIXct in local time. fwrite would render it as - # UTC, so a 07:07 Mountain survey would be read back as 13:07 - # and every survey would silently move six hours. Formatting it - # here stores exactly what the source displays. + # Format local time: fwrite would write UTC, shifting surveys + # by six hours design$date_time <- format(design$date_time) - # Step 4: Build the site rows. Birds carry UTM coordinates but - # no latitude, longitude or natural region, so those stay NA - # rather than being reprojected here. `year` is the calendar - # year taken from date_time; the model's recoded year travels - # in the design table. + # Step 4: Site rows. Lat, long and natural region stay NA (not + # reprojected); `year` is calendar year (the recoded model year + # is in the design table). calendar_year <- as.integer(format(covs$date_time, "%Y")) if (!all(covs$year == calendar_year - 1992)) { @@ -1543,27 +1309,21 @@ read_bird_data <- function(path) { stringsAsFactors = FALSE ) - # Step 5: The work queue. birdlist is one row per species and - # region, with the detection counts that decided inclusion. + # Step 5: The work queue (birdlist, per species and region) species_lists <- data.frame( as.data.frame(env$birdlist, stringsAsFactors = FALSE), stringsAsFactors = FALSE ) - # Step 6: Name the bootstrap columns. The source calls them - # "1" to "100", which a CSV header cannot distinguish from a - # row of data: read back, the header becomes a 58036th row and - # the columns are renamed V1 to V100. A word prefix makes the - # header unmistakable and keeps the draw number. + # Step 6: Prefix the bootstrap columns "1" to "100", which a + # CSV reader takes for a data row rather than a header bootstrap <- as.data.frame(env$boot) names(bootstrap) <- sprintf( "boot_%03d", seq_len(ncol(bootstrap)) ) - # Step 7: Record every factor's levels in source order. A CSV - # stores labels, and re-reading gives alphabetical levels; that - # would move the reference level and silently change which - # coefficient a model reports as the intercept. + # Step 7: Factor levels in source order, since a CSV re-read + # would give alphabetical levels and move the reference factor_cols <- names(covs)[vapply(covs, is.factor, logical(1))] factor_levels <- do.call(rbind, lapply( @@ -1626,9 +1386,7 @@ plant_parts <- lapply( names(plant_parts) <- names(plant_files) ## 3.2 Read the plant-group covariates and lookups ---- -# A second pass over the same four files. The response and the -# covariates are kept in separate readers because they answer -# separate questions, and this script runs once per snapshot. +# A second pass over the same four files (see the header TODO) message("Reading plant-group covariates ...") covariate_parts <- lapply( @@ -1643,8 +1401,6 @@ covariate_parts <- lapply( names(covariate_parts) <- names(plant_files) ## 3.3 Read the mammal files ---- -# The WildTrax strings are loaded once and passed to both -# regions, so each resolves its names against the same authority. message("Reading mammal files ...") native_sp <- load_rdata(wt_species_file)$native_sp @@ -1675,12 +1431,9 @@ message("Reading bird data ...") bird_parts <- read_bird_data(bird_data_file) # 4. Harmonize the mammal regions into one taxon table ---- -# North and South are one taxon split across two models, so they -# are stacked. ## 4.1 Confirm the North names reconcile with the South ---- -# A North name that no longer matches means the crosswalk has -# drifted and would split one species across two columns. +# A mismatch would split one species across two columns north_species <- setdiff( names(mammal_parts$north$species), "survey_unit_id" @@ -1737,8 +1490,7 @@ message( ) # 5. Build the master site table ---- -# One row per survey unit across all six sources, plus a flag per -# taxon recording which taxon files cover that unit. +# One row per survey unit, with an in_ flag per taxon ## 5.1 Stack every source's site rows ---- all_sites <- do.call( @@ -1752,8 +1504,7 @@ all_sites <- do.call( rownames(all_sites) <- NULL ## 5.2 Record which taxon files cover each unit ---- -# The four plant taxa share the SiteYearQu key but sample -# different unit sets. The flags make that coverage explicit. +# The plant taxa share keys but sample different units taxon_units <- c( lapply(plant_parts, function(x) x$species$survey_unit_id), list( @@ -1768,8 +1519,6 @@ for (taxon in names(taxon_units)) { } ## 5.3 Collapse to one row per survey unit ---- -# A plant unit contributes an identical site row from each taxon -# file that sampled it. flag_cols <- grep("^in_", names(all_sites), value = TRUE) attribute_cols <- setdiff( names(all_sites), @@ -1796,35 +1545,20 @@ sites <- all_sites[!duplicated(all_sites$survey_unit_id), ] rownames(sites) <- NULL ## 5.4 Build the master covariate table ---- -# One table for every taxon, keyed on survey_unit_id and taxon -# the way sites.csv is keyed on survey_unit_id alone. -# -# Taxon has to be part of the key, because the four plant files -# disagree. On the 7001 quadrats vascular plant and mite share, -# 154 of 162 vegetation columns and 58 of 59 soil columns hold -# different values, while climate agrees to the last digit. -# Bryophyte and lichen match vascular plant exactly. Two -# descriptions of the same quadrat cannot share one row, and -# choosing between them is not this script's call to make. -# -# Columns are the union across taxa. Each taxon fills only the -# blocks it has and leaves the rest empty, so the table is about -# 16 per cent populated - the price of one file instead of -# twenty, paid mostly by the bird rows, which are 86 per cent of -# the rows and share almost no column names with the others. +# Keyed on survey_unit_id and taxon because the plant files +# disagree: on the 7001 quadrats vascular plant and mite share, +# 154 of 162 veg and 58 of 59 soil columns differ (climate +# agrees; bryophyte and lichen match vascular plant). Each file +# is a separate extraction (01a_data-standardization.R); choosing +# between them is not this script's call. Columns are the union +# across taxa (~16% populated; birds are 86% of rows). #' Join One Taxon's Covariate Blocks Side by Side #' -#' Blocks come from one source frame in one row order, so they -#' are combined by position rather than by key. -#' -#' A name carried by more than one block is suffixed with the -#' block it came from. That is not tidiness: in the plant files -#' 42 names sit in both the veg and soil blocks, and 14 of them -#' hold different values, because the footprint terms are -#' computed once under the north vegetation framework and again -#' under the south soil one. Keeping a single `Crop` would feed -#' the south models north numbers. +#' Combined by position (one source frame). Names in more than +#' one block are block-suffixed: of 42 plant names in both veg +#' and soil, 14 differ (footprint computed under each framework), +#' so a single `Crop` would feed the south models north numbers. #' #' @param blocks Named list of data frames, each keyed on #' survey_unit_id. The names become the block suffixes. @@ -1889,10 +1623,9 @@ merge_blocks <- function(blocks, taxon) { return(list(table = out, map = do.call(rbind, map))) } -# The blocks each taxon owns. Mammals are split by region -# because north and south are separate models on overlapping -# deployments; birds add a design block that is neither a -# covariate nor a field sites.csv carries. +# Mammals are split by region (separate models on overlapping +# deployments: 512 location_project values in both); birds add +# a design block covariate_blocks <- c( lapply(covariate_parts, function(x) { list(climate = x$climate, veg = x$veg, soil = x$soil) @@ -1928,8 +1661,6 @@ covariates <- as.data.frame(rbindlist( use.names = TRUE )) -# Where every column came from, so a block can be reconstructed -# exactly and a suffixed name traced back to its v2 spelling. covariate_columns <- do.call( rbind, lapply(merged_covariates, function(x) x$map) @@ -1953,8 +1684,7 @@ message( ) # 6. Validate the assembled tables ---- -# Cheap checks that would catch a layout change in a future -# snapshot before the CSVs are trusted downstream. +# Catch layout changes in a future snapshot species_tables <- c( lapply(plant_parts, function(x) x$species), @@ -1999,11 +1729,8 @@ for (taxon in names(species_tables)) { } ## 6.3 Covariates must cover the response, unit for unit ---- -# A survey unit with a detection but no covariates would be -# dropped silently at model time, so the two are required to -# describe exactly the same set. Mammals contribute two taxon -# values to the covariate table and one response table, so the -# two regions are pooled before comparing. +# Otherwise units drop silently at model time. Mammal regions +# are pooled first (two covariate keys, one response table). covariate_taxa <- list( vascular_plant = "vascular_plant", bryophyte = "bryophyte", @@ -2049,9 +1776,7 @@ if (length(orphan_covariates) > 0) { } ## 6.4 Modelled species must exist as response columns ---- -# veg.species.list and soil.species.list are the work queue. A -# name that is not a column would fail per bootstrap inside a -# tryCatch, so it is caught here instead. +# Otherwise it fails per bootstrap, inside a tryCatch for (taxon in names(covariate_parts)) { declared <- covariate_parts[[taxon]]$species_lists$species available <- setdiff( @@ -2073,8 +1798,7 @@ for (taxon in names(covariate_parts)) { } ## 6.5 The prediction matrices must agree across taxa ---- -# They are written once rather than per taxon, which is only -# sound while all four files carry the same grid. +# Written once, so all four files must carry the same grid for (matrix_name in c("veg_pm", "soil_pm")) { reference <- covariate_parts[[1]][[matrix_name]] @@ -2097,11 +1821,8 @@ for (matrix_name in c("veg_pm", "soil_pm")) { } ## 6.6 Mammal habitat must be cover proportions ---- -# The habitat block is what is left once the design, climate and -# species columns are named, so a new unnamed column would land -# in it silently. The reader has already checked that the blocks -# partition the frame; this checks that nothing non-numeric -# arrived, which is the shape a stray design field would have. +# Habitat is the remainder, so a stray design field would land +# here; check it is all numeric for (region in names(mammal_parts)) { part <- mammal_parts[[region]] @@ -2145,10 +1866,8 @@ for (region in names(mammal_parts)) { } ## 6.8 Bird blocks must account for every covariate column ---- -# The four blocks are taken by name, so a column the source adds -# would be dropped silently rather than landing somewhere wrong. -# Easting and Northing are deliberately excluded: sites.csv -# carries them. +# Blocks are taken by name, so a new source column would drop +# silently. Easting and Northing are in sites.csv. bird_named <- unique(c( bird_climate_cols, bird_veg_cols, @@ -2171,8 +1890,6 @@ if (length(bird_unnamed) > 0) { } ## 6.9 Bird counts and offsets must be model ready ---- -# The models are Poisson with a log offset, so counts must be -# non-negative whole numbers and every count needs an offset. bird_counts <- as.matrix( bird_parts$species[, -1, drop = FALSE] ) @@ -2229,8 +1946,6 @@ if (length(bird_unknown) > 0) { } ## 6.11 Bootstrap draws must resolve to surveys ---- -# The bootstrap matrix holds raw surveyids. They join to the -# design table, so a draw outside it would drop rows at fit time. bird_boot_ids <- unique(unlist(bird_parts$bootstrap)) bird_orphan_draws <- setdiff( bird_boot_ids, @@ -2248,8 +1963,6 @@ if (length(bird_orphan_draws) > 0) { } # 7. Write the CSVs ---- -# One file per taxon plus the site table, all sharing the -# survey_unit_id key. ## 7.1 Write the taxon tables ---- for (taxon in names(species_tables)) { @@ -2280,14 +1993,8 @@ message( ) ## 7.3 Write the master covariate table ---- -# One file beside sites.csv, keyed on survey_unit_id and taxon, -# holding every covariate every taxon's models regress against. -# A model-data frame is rebuilt by joining it to sites.csv and -# the taxon's response table on survey_unit_id, filtering to that -# taxon, and dropping the columns that came back empty. -# -# The per-taxon covariate folder this replaces is removed, so a -# stale copy cannot be picked up by mistake. +# Also removes the old per-taxon covariate folder, so a stale +# copy cannot be read old_covariate_dir <- file.path(output_dir, "covariates") if (dir.exists(old_covariate_dir)) { @@ -2328,8 +2035,7 @@ message( ) ## 7.4 Write the shared lookups ---- -# Section 6.5 has already confirmed the two prediction matrices -# are the same in all four files, so they are written once. +# Written once (checked identical across taxa in 6.5) dir.create(lookup_dir, recursive = TRUE, showWarnings = FALSE) prediction_matrices <- list( @@ -2337,9 +2043,7 @@ prediction_matrices <- list( soil_prediction_matrix = covariate_parts[[1]]$soil_pm ) -# Take the v2 CSVs when they are reachable. They are what the v2 -# scripts read, and the snapshot objects are missing columns the -# vegetation models need. +# The v2 CSVs when reachable (section 1.8) for (matrix_name in names(prediction_matrices)) { if (is.na(v2_lookup_dir) || !nzchar(v2_lookup_dir)) { next @@ -2379,10 +2083,8 @@ for (matrix_name in names(prediction_matrices)) { ) } -# The four aggregate columns the vegetation models predict onto. -# Their absence is what separates the snapshot's veg.pm from the -# CSV the v2 scripts read, so it is reported here rather than -# found later by check_prediction_terms(). +# Report whether the four aggregate columns are present, rather +# than fail later at prediction time veg_aggregates <- c("Peatland", "Mineral", "Upland", "CCR1234") absent_aggregates <- setdiff( veg_aggregates, @@ -2419,9 +2121,7 @@ message( ) ## 7.5 Write the mammal lookups ---- -# The prediction matrices are per region and were checked against -# the CSVs on the ABMI Mammals drive, so they are taken from the -# snapshot rather than copied in. +# From the snapshot (checked identical to the ABMI Mammals CSVs) for (region in names(mammal_parts)) { path <- file.path( lookup_dir, @@ -2440,9 +2140,7 @@ for (region in names(mammal_parts)) { ) } -# Mammals get their own species table rather than rows in -# modelled_species.csv: their work queue is season by occurrence -# threshold, which the plant north/south schema has no room for. +# Own table: season by threshold does not fit the plant schema mammal_modelled <- do.call( rbind, lapply(mammal_parts, function(x) x$species_lists) @@ -2462,11 +2160,8 @@ message( " cols" ) -# The climate offset the mammal habitat models take. It is -# produced by a separate climate pipeline, so it is copied in -# with survey_unit_id added, keyed the way the rest of the -# dataset is. A deployment with no climate prediction keeps the -# NA the v2 join would have produced. +# Copied in with survey_unit_id added; unmatched deployments +# keep NA, as the v2 join gives if (file.exists(mammal_climate_pred_file)) { climate_pred <- as.data.frame( fread(mammal_climate_pred_file), stringsAsFactors = FALSE @@ -2527,10 +2222,7 @@ if (file.exists(mammal_climate_pred_file)) { } ## 7.6 Write the bird offsets ---- -# The QPAD offsets. Response-shaped rather than covariate-shaped: -# one value per survey and species, carrying the detectability -# correction each Poisson model takes as its offset. It is the -# largest table in the dataset by a wide margin. +# One QPAD offset per survey and species; the largest table offsets_path <- file.path(output_dir, "bird_offsets.csv") fwrite(bird_parts$offsets, offsets_path, na = "") @@ -2560,9 +2252,8 @@ message( " cols" ) -# Raw surveyids, not survey_unit_ids: they join to the surveyid -# column carried in covariates.csv for taxon "bird". Keeping them -# raw holds the file to a tenth of the size prefixed keys need. +# Raw surveyids (joined via covariates.csv's surveyid), a tenth +# the size of prefixed keys bird_boot_path <- file.path( lookup_dir, "bird_bootstrap_ids.csv" ) @@ -2576,9 +2267,6 @@ message( " cols" ) -# Without this, re-reading a factor from CSV gives alphabetical -# levels, which moves the reference level and renames every -# coefficient the model reports. bird_levels_path <- file.path( lookup_dir, "bird_factor_levels.csv" ) diff --git a/1_code/_setup/02_validate_test_dataset.R b/1_code/_setup/02_validate_test_dataset.R index c2574a2..64078d2 100644 --- a/1_code/_setup/02_validate_test_dataset.R +++ b/1_code/_setup/02_validate_test_dataset.R @@ -3,6 +3,8 @@ # author: Brendan Casey # created: 2026-09-08 # inputs: +# Read via setup_input() (utils/input_paths.R); the original +# locations are listed, and SDM_* variables can point back. # in 0_data/test_dataset/, written by # _setup/01_harmonize_model_ready_v2.R: # - sites.csv @@ -33,27 +35,15 @@ # outputs: # - none; every result is printed to the console # notes: -# Two passes over the harmonized dataset. Section 2 describes -# it: sizes, coverage, value ranges, and whether the tables -# join the way they are meant to. Section 3 traces every -# written value back to the snapshot it came from: responses in -# 3.3 to 3.5, covariates in 3.6, and lookups in 3.7 to 3.9. -# -# Covariates now live in one master table keyed on -# survey_unit_id and taxon, not in a file per taxon and block. -# Section 3.6 rebuilds each block from it using -# lookup/covariate_columns.csv, which maps every master column -# back to the block and the v2 spelling it came from. That is -# also what catches a column being dropped or renamed, since a -# block is only checked against the map's own account of it. -# -# Covariate values are compared on relative difference, not -# absolute. A CSV carries about 15 significant digits, so UTMY -# at 6.6e6 comes back about 6e-9 away from the source; that is -# a round-trip artefact, not a harmonization error. -# -# Run after 01_harmonize_model_ready_v2.R, and again whenever -# the snapshot or that script changes. +# - Section 2 describes the dataset (sizes, coverage, ranges, +# joins); section 3 traces every written value back to its +# source snapshot. +# - Covariate blocks are rebuilt from the master table via +# lookup/covariate_columns.csv, which also catches dropped or +# renamed columns. +# - Covariates are compared on relative difference: CSV keeps +# ~15 significant digits (UTMY at 6.6e6 drifts ~6e-9). +# - Run after 01, and whenever the snapshot or 01 changes. # --- # 1. Setup ---- @@ -62,6 +52,9 @@ library(data.table) ## 1.2 Configure paths ---- +# The _setup inputs mirrored on ABMI-DATA2 (setup_input()) +source(file.path(getwd(), "1_code/_setup/utils/input_paths.R")) + # data_dir holds the harmonized CSVs; snapshot_dir holds the # read-only sources they were built from. data_dir <- file.path( @@ -71,10 +64,7 @@ data_dir <- file.path( snapshot_dir <- Sys.getenv( "SDM_SNAPSHOT_V2", - unset = paste0( - "//ABMI-DATA2/science/sc/sdmMethodsDev/0_data/", - "data_snapshots/model_ready_v2" - ) + unset = setup_input("model_ready_v2") ) taxa <- c( @@ -90,10 +80,7 @@ taxa <- c( # were fitted on, not the BirdModels drive's rebuilt one. bird_data_file <- Sys.getenv( "SDM_BIRD_DATA", - unset = paste0( - "G:/Shared drives/work_abmi/1_projects/active/", - "sdmMethodsDev/remote/birds_data_v2/Stratified.Rdata" - ) + unset = setup_input("birds_data_v2", "Stratified.Rdata") ) ## 1.3 Read the harmonized CSVs ---- @@ -338,9 +325,9 @@ plant_files <- c( # The mammal climate offset, produced outside this repository. mammal_climate_pred_file <- Sys.getenv( "SDM_MAMMAL_CLIMATE_PRED", - unset = paste0( - "G:/Shared drives/ABMI Mammals/Results/Habitat Modeling/", - "2024/Climate/Predictions/", + unset = setup_input( + "abmi_mammals", + "Results/Habitat Modeling/2024/Climate/Predictions", "All Species Climate Predictions.csv" ) ) @@ -614,10 +601,8 @@ for (rg in names(mammal_files)) { #' Count Disagreements on Relative Difference #' -#' The absolute counterpart, compare_blocks(), is right for the -#' response, whose values are small integers. Covariates span -#' UTM coordinates at 1e6 and cover proportions at 1e-4, so a -#' single absolute tolerance cannot serve both. +#' Covariates span 1e-4 (cover) to 1e6 (UTM), so no single +#' absolute tolerance fits; compare_blocks() suits responses. #' #' @param a,b Data frames or matrices of the same shape. #' @param tol Numeric. Largest relative difference treated as @@ -664,11 +649,9 @@ derived_terms <- list( #' Check One Taxon's Covariate Blocks Against Their Source #' -#' Rebuilds each block from the master table using the column -#' map, then compares it to the frame it came from. The map is -#' what makes this possible: a master column may be suffixed -#' with its block, so the v2 spelling cannot be recovered from -#' the header alone. +#' Rebuilds each block from the master table via the column map +#' (master columns may carry a block suffix) and compares it to +#' its source frame. #' #' @param taxon_label Character. Value in covariates$taxon. #' @param source_frames Named list of source data frames, one @@ -699,9 +682,7 @@ check_covariates <- function(taxon_label, source_frames, return(invisible(NULL)) } - # Step 1: Line the rows up with the source before any value is - # compared, by key where there is one and by position where - # there is not + # Step 1: Align rows with the source, by key or by position n_source <- nrow(source_frames[[1]]) if (is.null(source_ids)) { @@ -915,8 +896,8 @@ for (taxon in names(plant_files)) { ) } -# The prediction matrices are written once for all four taxa, so -# they are checked once, against the last taxon read above. +# Shared by all four taxa, so checked once (against the last +# taxon read above) veg_pm_csv <- as.data.frame(fread( file.path(data_dir, "lookup", "veg_prediction_matrix.csv") )) @@ -962,9 +943,7 @@ note( ) ## 3.6 Mammal covariates and lookups against source ---- -# Mammal covariates come from the same two SpTable files as the -# mammal response, and are written per region because north and -# south are separate models on overlapping deployments. +# From the same two SpTable files as the mammal response mammal_habitat_name <- c(north = "veg", south = "soil") @@ -973,10 +952,9 @@ for (region in names(mammal_files)) { d <- as.data.frame(env$d, stringsAsFactors = FALSE) label <- paste("mammal", region) - # Mammal covariates carry a taxon of mammal_, because - # north and south are separate models on overlapping - # deployments. The habitat block is called veg in the north and - # soil in the south, matching the model each one feeds. + # Keyed mammal_: north and south are separate models on + # overlapping deployments. The habitat block is veg in the north + # and soil in the south. habitat <- mammal_habitat_name[[region]] frames <- list(d, d) names(frames) <- c("climate", habitat) @@ -1141,8 +1119,7 @@ if (file.exists(bird_data_file)) { sprintf("%d cells, max diff %.1g", cmp$n, cmp$max_diff) ) - # Counts feed a Poisson model, so they have to be whole and - # non-negative for the fit to mean anything. + # Poisson counts: whole and non-negative bird_counts <- as.matrix(bird_csv[, ..bird_spp]) note( "bird", @@ -1153,8 +1130,7 @@ if (file.exists(bird_data_file)) { paste("max", max(bird_counts)) ) - # Offsets are response-shaped: one per survey and species, and - # the model cannot run without the pair. + # One offset per survey and species off_csv <- fread(file.path(data_dir, "bird_offsets.csv")) note( @@ -1247,9 +1223,8 @@ if (file.exists(bird_data_file)) { ) ) - # Factor levels have to travel separately: read back from a - # CSV a factor takes alphabetical levels, which moves the - # reference level and renames every coefficient. + # Read back from CSV, factors take alphabetical levels, moving + # the reference level levels_csv <- fread( file.path(data_dir, "lookup", "bird_factor_levels.csv") ) @@ -1282,6 +1257,76 @@ if (file.exists(bird_data_file)) { ) } +## 3.6.4 Harmonized lookups ---- +# _setup/09's outputs against the files checked above, so a +# lookup left stale by a rebuild of 01 or 06 fails +lookup_dir <- file.path(data_dir, "lookup") +harmonized <- c( + "species_queue.csv", "factor_levels.csv", "dataset_manifest.csv", + "bird_north_prediction_matrix.csv", + "bird_south_prediction_matrix.csv" +) +present <- file.exists(file.path(lookup_dir, harmonized)) + +note( + "lookups", "harmonized lookups written", all(present), + if (all(present)) "" else paste0( + "missing ", paste(harmonized[!present], collapse = ", "), + "; run 09_harmonize_lookups.R" + ) +) + +if (all(present)) { + queue <- fread(file.path(lookup_dir, "species_queue.csv"), + na.strings = "") + manifest <- fread(file.path(lookup_dir, "dataset_manifest.csv"), + na.strings = "") + + # Every queued species is a column of its taxon's response file + queued_ok <- vapply(unique(queue$taxon), function(one) { + file <- manifest$response_file[manifest$taxon == one][1] + columns <- names(fread(file.path(data_dir, file), nrows = 0)) + all(queue$species[queue$taxon == one] %in% columns) + }, logical(1)) + + note( + "lookups", "queued species are response columns", + all(queued_ok), + paste(names(queued_ok)[!queued_ok], collapse = ", ") + ) + + # The queue holds what the three source lookups hold + plant_rows <- fread(file.path(lookup_dir, "modelled_species.csv")) + bird_rows <- fread( + file.path(lookup_dir, "bird_modelled_species.csv") + ) + mammal_rows <- fread( + file.path(lookup_dir, "mammal_modelled_species.csv") + ) + expected <- sum(plant_rows$in_veg_models) + + sum(plant_rows$in_soil_models) + nrow(bird_rows) + + sum(mammal_rows$in_models) + sum(mammal_rows$in_ua_models) + + note( + "lookups", "queue rows match the source lookups", + nrow(queue) == expected, + paste0(nrow(queue), " queued, ", expected, " in the sources") + ) + + # Every file the manifest names exists + named <- c( + file.path(data_dir, manifest$response_file), + file.path(data_dir, stats::na.omit(manifest$offset_file)), + file.path(lookup_dir, + paste0(manifest$grid, "_prediction_matrix.csv")) + ) + + note( + "lookups", "manifest files exist", all(file.exists(named)), + paste(basename(named[!file.exists(named)]), collapse = ", ") + ) +} + ## 3.7 Verdict ---- # Every check, then the tally. A FAIL means the CSVs no longer # match the snapshot they claim to come from. diff --git a/1_code/_setup/03_rerun_bryophyte_v2_reference.R b/1_code/_setup/03_rerun_bryophyte_v2_reference.R index cdbd374..580aa08 100644 --- a/1_code/_setup/03_rerun_bryophyte_v2_reference.R +++ b/1_code/_setup/03_rerun_bryophyte_v2_reference.R @@ -3,6 +3,8 @@ # author: Brendan Casey # created: 2026-09-10 # inputs: +# Read via setup_input() (utils/input_paths.R); the original +# locations are listed, and SDM_* variables can point back. # in the v2 project (SDM_V2_PROJECT): # - 0_data/species/processed/bryophyte-model-data.Rdata # - 0_data/bootstrap/bryophyte-bootstrap-ids.Rdata @@ -13,31 +15,15 @@ # in 2_pipeline/v2_reference/: # - bryophyte-species-models.Rdata # notes: -# The published bryophyte reference is unusable. Every species -# and every bootstrap in the copy on ABMI-DATA2 is an error -# object reading `could not find function "model.avg"` - MuMIn -# was not on the library path of the cluster workers that -# produced it. The other three plant taxa came through intact. -# -# This regenerates it by running the frozen v2 functions, -# unmodified, against the v2 project's own data. It reproduces -# what 02a_hierarchical-models-bryophytes.R does rather than -# improving on it: the point is a reference to measure the new -# framework against, so any change would defeat it. -# -# It writes to 2_pipeline/, never to the v2 project. The -# network copy is someone else's artefact and is left alone; -# whoever owns that pipeline can decide whether to replace it. -# -# The run is large - 134 species by 100 bootstraps by 58 -# climate models, then the habitat sets - so `species_n` and -# `boot_n` are here to make a partial reference deliberately. -# A partial one is still useful: the gate compares per species, -# so any species present can be gated. -# -# MuMIn must be installed, and must be loadable by the cluster -# workers, which is exactly what failed before. Section 2.3 -# checks that on a worker rather than trusting it. +# - The published bryophyte reference is unusable: every entry +# is a `could not find function "model.avg"` error (MuMIn was +# missing on the cluster workers). +# - Regenerates the climate stage with the frozen v2 functions, +# unmodified, writing to 2_pipeline/ only. +# - 134 species x 100 draws x 58 models is large; +# `species_n` and `boot_n` build a partial reference, which +# the per-species gate can still use. +# - Section 3.3 checks MuMIn loads on a worker before fitting. # --- # 1. Setup ---- @@ -49,9 +35,12 @@ library(parallel) # bootstrap-level parallelism (version: 4.5.0) ## 1.2 Configure ---- project_root <- normalizePath(getwd(), winslash = "/") +# The _setup inputs mirrored on ABMI-DATA2 (setup_input()) +source(file.path(project_root, "1_code/_setup/utils/input_paths.R")) + v2_project <- Sys.getenv( "SDM_V2_PROJECT", - unset = "//ABMI-DATA2/science/sc/ToEmily/VegetationModels" + unset = setup_input("vegetation_models") ) out_dir <- file.path(project_root, "2_pipeline/v2_reference") @@ -63,16 +52,14 @@ boot_n <- NULL n_clusters <- 14 ## 1.3 Load the frozen v2 functions ---- -# From this repository's snapshot, which is byte-identical to the -# originals and has recorded provenance. +# From the byte-identical snapshot in 0_data/v2_scripts/ source(file.path( project_root, "0_data/v2_scripts/plants/hierarchical-model_functions.R" )) ## 1.4 Set the seed ---- -# v2 seeds nothing. A reference that cannot reproduce itself -# is hard to argue from, so this one is seeded. +# Seeded, unlike v2, so this reference reproduces itself boot_seed <- 20260910L ## 1.5 Load the v2 data ---- @@ -117,9 +104,7 @@ cat( ) # 2. Bootstrap ids ---- -# The stored ids are checked rather than trusted. The bryophyte -# file holds only 3 draws where every other plant taxon holds -# 100, and that is invisible until draw 4 fails hours into a run. +# Checked, not trusted: the bryophyte file holds 3 draws, not 100 ## 2.1 Check what the stored ids carry ---- stored_draws <- min(vapply(bootstrap.ids, ncol, integer(1))) @@ -127,10 +112,8 @@ stored_draws <- min(vapply(bootstrap.ids, ncol, integer(1))) cat("Stored bootstrap draws: ", stored_draws, "\n", sep = "") ## 2.2 Regenerate when they fall short ---- -# A regenerated set is a fresh draw, not the missing part of the -# stored one: v2's bootstrap is unseeded and a partial set cannot -# be extended. The seed is saved beside the ids so this reference -# reproduces itself, which the published one never could. +# A fresh, seeded draw (an unseeded partial set cannot be +# extended); the seed is saved beside the ids if (stored_draws < max(boot.iter)) { cat( "Regenerating: ", max(boot.iter), " draws needed, ", @@ -217,8 +200,7 @@ if (stored_draws < max(boot.iter)) { } # 3. Climate models ---- -# The candidate set and its assembly are v2's: 14 base models, -# a bioclim version of each, and spatial versions of ten of them. +# v2's 58 candidates: 14 base, 14 bioclim, 30 spatial ## 3.1 Define the model sets ---- climate.models <- list( @@ -270,9 +252,7 @@ invisible(clusterEvalQ(core.input, { })) ## 3.3 Confirm MuMIn is loadable on a worker ---- -# The failure that produced the unusable reference was exactly -# this: model.avg missing on the workers, caught only after the -# whole run. Checking here costs nothing. +# The failure behind the unusable published reference worker_has_mumin <- unlist(clusterEvalQ( core.input, exists("model.avg") )) @@ -315,9 +295,7 @@ climate.coef <- foreach(species = species.list) %dopar% names(climate.coef) <- species.list stopCluster(core.input) -# A species is counted on its draws, not all-or-nothing. One bad -# draw in a hundred still leaves a usable distribution, and the -# earlier version of this check called such a species a failure. +# Counted per draw: one bad draw in 100 leaves a usable species usable_species <- sum( vapply(climate.coef, is.numeric, logical(1)) ) @@ -363,9 +341,8 @@ if (usable_species == 0 && failed_draws > 0) { } # 4. Save ---- -# Only the climate object is regenerated here. The habitat sets -# take the climate coefficients as an input, so they can only be -# rebuilt once this one is sound; that is a separate run. +# Climate only; the habitat sets depend on it and are a separate +# run save( climate.coef, file = file.path(out_dir, "bryophyte-species-models.Rdata") diff --git a/1_code/_setup/04_package_bird_coefficients.R b/1_code/_setup/04_package_bird_coefficients.R deleted file mode 100644 index a8382cf..0000000 --- a/1_code/_setup/04_package_bird_coefficients.R +++ /dev/null @@ -1,926 +0,0 @@ -# --- -# title: Package the v2 Bird Coefficients -# author: Elly Knight; adapted by Brendan Casey -# created: 2026-09-13 -# inputs: -# in the BirdModels project (SDM_V2_BIRD_ROOT): -# - Results/ClimateModels/Coefficients//*.csv -# - Results/LandcoverModels/Coefficients///*.csv -# - Data/lookups/Xn-veg-v2024.Rdata -# - Data/lookups/birdlist.csv, birds-v2024.csv -# on the work_abmi shared drive (SDM_BIRD_DATA): -# - 1_projects/active/sdmMethodsDev/remote/birds_data_v2/ -# Stratified.Rdata -# in this repository: -# - 0_data/v2_scripts/birds/00.NorthModels.R -# - 0_data/v2_scripts/birds/00.SouthModels.R -# outputs: -# in 2_pipeline/v2_reference/: -# - Birds2024.RData -# notes: -# - A clone of 0_data/v2_scripts/birds/08.PackageCoefficients.R, -# the v2 step that turns per-draw coefficient csvs into the -# packaged `birds` object. Only the plumbing is changed: -# where it reads from, where it writes to, and how it reads. -# Every line of the coefficient translation is the original's, -# because reproducing it is the entire point. -# -# - It exists because the packaged bird output does not. v2 -# declares Results/Birds2024.RData at line 771 of the -# original, and that file is absent from the drive; so are -# Predictions2024.RData and ModelComparison.csv. Only the -# per-draw csvs from steps 6 and 7 survive. Birds are the one -# taxon with no final result to gate against, and this -# rebuilds it. -# -# - Why it is worth running rather than summarizing the csvs -# directly: this produces coefficients on the cross-taxa -# standardized template, the same WhiteSpruceR and Loamy term -# space the plant tables use, rather than the raw glm factor -# names vegcCrop and soilcLoamy. That is what makes birds -# comparable to the other taxa at all. -# -# - Stratified.Rdata is read from the copy v2's birds were -# fitted on, the same one 01 harmonizes from, not from the -# BirdModels drive. Its Data/Archive/2025/ copy was rebuilt -# in place on 2026-08-19 and no longer matches v2's results. -# -# - Reads are serial and that is deliberate. Measured on this -# mount: 400 consecutive serial reads succeeded with no -# failures, while four parallel readers put Google Drive into -# a refusal state that outlasted repeated retries and lost -# every landcover species. Concurrency trips it, not volume. -# Expect a few hours on a cold cache. -# -# - Read only with respect to the v2 project. The packaged -# object is written to 2_pipeline/, never to the bird drive. -# --- - -# 0. Paths for this repository ---- -# The only block that is not the original script. - -## 0.1 Resolve the bird project ---- -# The path hard-coded in v2 is not mounted here. This is the same -# directory reached through remote/BirdModels.lnk on work_abmi. -project_root <- normalizePath(getwd(), winslash = "/") - -v2_bird_root <- Sys.getenv( - "SDM_V2_BIRD_ROOT", - unset = paste0( - "G:/.shortcut-targets-by-id/", - "17Ymt13eHfKvIiuoMl6x-Kn74Z2uVbbzS/BirdModels" - ) -) - -if (!dir.exists(v2_bird_root)) { - stop( - "Bird project not found:\n ", v2_bird_root, - "\nSet SDM_V2_BIRD_ROOT to the BirdModels directory.", - call. = FALSE - ) -} - -## 0.2 Where the frozen model scripts live ---- -v2_bird_scripts <- file.path( - project_root, "0_data/v2_scripts/birds" -) - -## 0.3 Choose the complete results subtree ---- -# Results/ holds a partial re-run of a single species; the -# complete set is under Results/Archive/. Two runs of an unseeded -# pipeline disagree numerically, so they are never mixed: -# whichever covers more species is used for everything. -bird_results_root <- local({ - options <- c( - file.path(v2_bird_root, "Results"), - file.path(v2_bird_root, "Results", "Archive") - ) - - counts <- vapply(options, function(one) { - leaf <- file.path(one, "LandcoverModels", "Coefficients") - - if (!dir.exists(leaf)) { - return(-1L) - } - - length(list.files(leaf, pattern = "[.]csv$", recursive = TRUE)) - }, integer(1)) - - if (all(counts < 0)) { - stop( - "No bird landcover coefficients under - ", - paste(options, collapse = " - "), - call. = FALSE - ) - } - - options[which.max(counts)] -}) - -## 0.4 Where the packaged object is written ---- -out_dir <- file.path(project_root, "2_pipeline/v2_reference") - -dir.create(out_dir, recursive = TRUE, showWarnings = FALSE) - -out_file <- file.path(out_dir, "Birds2024.RData") - -## 0.5 Locate the stratified modelling data ---- -# The copy v2's birds were fitted on, shared with 01 through the -# same SDM_BIRD_DATA override. -stratified_path <- Sys.getenv( - "SDM_BIRD_DATA", - unset = paste0( - "G:/Shared drives/work_abmi/1_projects/active/", - "sdmMethodsDev/remote/birds_data_v2/Stratified.Rdata" - ) -) - -if (!file.exists(stratified_path)) { - stop( - "Stratified.Rdata not found at\n ", stratified_path, - "\nIt holds the modelling frame the coefficient ", - "translation needs. Set SDM_BIRD_DATA.", - call. = FALSE - ) -} - -cat("Bird project: ", v2_bird_root, "\n", sep = "") -cat("Results: ", bird_results_root, "\n", sep = "") -cat("Stratified: ", stratified_path, "\n", sep = "") -cat("Writing to: ", out_file, "\n\n", sep = "") - -# Everything below this line is 08.PackageCoefficients.R, with -# only its paths repointed. - - -#NOTES################################ - -#The translation of the model coefficients to the coefficients that are standardized across taxa relies on a lookup table. The proportions of the age coefficients in the raw models that contribute to the age stages in the standardized coefficients are area under the curve. - -#PREAMBLE############################ - -#1. Load packages---- -library(tidyverse) #basic data wrangling -library(purrr) #functional programming -library(vegan) #for picking median bootstrap -library(pROC) #AUC - -#2. Set root path for data on google drive---- -root <- v2_bird_root - -#3. Restrict scientific notation---- -options(scipen = 99999) - -#4. Get the list of models run---- -mods <- data.frame(path = list.files(file.path(bird_results_root, "LandcoverModels", "Coefficients"), full.names = TRUE, recursive = TRUE, pattern="*.csv"), - file = list.files(file.path(bird_results_root, "LandcoverModels", "Coefficients"), recursive = TRUE, pattern="*.csv")) |> - separate(file, into=c("region", "f1", "f2", "species", "bootstrap")) |> - mutate(bootstrap = as.numeric(bootstrap)) |> - dplyr::select(-f1, -f2) - -#5. Load model scripts---- -source(file.path(v2_bird_scripts, "00.NorthModels.R")) -source(file.path(v2_bird_scripts, "00.SouthModels.R")) - -#6. Load data---- -load(stratified_path) - -#7. Function to fix interaction term names---- -#make formula term names sorted and predictable, i.e. always A:B instead of B:A -fix_names <- function(x, sep=":") { - unlist(lapply(x, function(z) { - paste(sort(strsplit(z, sep)[[1]]), collapse=sep) - })) -} - -#8. Function to get model terms----- -#Formula from Peter's approach -get_terms <- function(mods, type=c("formula", "list"), intercept=TRUE) { - type <- match.arg(type) - x <- unlist(lapply(unlist(mods), function(z) as.character(z)[3])) - # x <- unname(substr(x, 5, nchar(x))) - x <- gsub(". + ", "", x, fixed=TRUE) - x <- unlist(strsplit(x, "+", fixed=TRUE)) - x <- unlist(strsplit(x, "*", fixed=TRUE)) - if (type == "list") - x <- unlist(strsplit(x, ":", fixed=TRUE)) - x <- sub("^[[:space:]]*(.*?)[[:space:]]*$", "\\1", x, perl=TRUE) - x <- unique(x) - if (type == "formula") { - x <- paste("~", paste(x, collapse=" + ", sep="")) - if (!intercept) - x <- paste(x, "- 1") - x <- as.formula(x) - } - x -} - -#9. Load veg age lookup---- -load(file.path(root, "Data", "lookups", "Xn-veg-v2024.Rdata")) -colnames(age) <- fix_names(colnames(age)) - -#CLIMATE MODEL COEFFICIENTS########## - -#1. Get list of climate models---- -climate <- data.frame(path = list.files(file.path(bird_results_root, "ClimateModels", "Coefficients"), full.names = TRUE, pattern="*.csv", recursive = TRUE), - file = list.files(file.path(bird_results_root, "ClimateModels", "Coefficients"), pattern="*.csv", recursive = TRUE)) |> - separate(file, into=c("f1", "model", "species", "bootstrap", "filetype")) |> - dplyr::select(-model, -filetype, -f1) |> - mutate(bootstrap = as.numeric(bootstrap)) |> - inner_join(mods |> - dplyr::select(species, bootstrap) |> - unique()) - -#2. Get list of species---- -spp <- unique(climate$species) - -#3. Get max # of bootstraps---- -bmax <- max(climate$bootstrap) - -#4. Set the column names---- -coefnames <- c("(Intercept)", "MAP", "FFP", "TD", "CMD", "Easting", "Northing", "I(Easting^2)", "I(Northing^2)", "Easting:Northing") -coefrename <- c("Intercept", "MAP", "FFP", "TD", "CMD", "Easting", "Northing", "Easting2", "Northing2", "EastingNorthing") - -#5. Set up loop----- -for(i in 1:length(spp)){ - - #6. List of models for that species---- - climate.i <- dplyr::filter(climate, species==spp[i], - bootstrap %in% c(1:bmax)) - - #7. Read them in---- - #sort and fix names - coef.list <- list() - se.list <- list() - for(j in 1:nrow(climate.i)){ - coef.list[[j]] <- read.csv(climate.i$path[j]) |> - dplyr::select(-se) |> - pivot_wider(names_from="X", values_from = "coef") |> - dplyr::select(all_of(coefnames)) - colnames(coef.list[[j]]) <- coefrename - - se.list[[j]] <- read.csv(climate.i$path[j]) |> - dplyr::select(-coef) |> - pivot_wider(names_from="X", values_from = "se") |> - dplyr::select(all_of(coefnames)) - colnames(se.list[[j]]) <- coefrename - } - - #8. Make the array if first species---- - if(i==1){ - marginal <- array(0, c(length(spp), length(coefrename), bmax)) - dimnames(marginal) <- list(spp, coefrename, paste0("b", c(1:bmax))) - - marginal_se <- array(0, c(length(spp), length(coefrename), bmax)) - dimnames(marginal_se) <- list(spp, coefrename, paste0("b", c(1:bmax))) - } - - #9. Add species to array---- - marginal[spp[i],,] <- as.matrix(t(do.call(rbind, coef.list))) - marginal_se[spp[i],,] <- as.matrix(t(do.call(rbind, se.list))) - - cat("Finished species", i, "of", length(spp), "\n") - -} - -#LANDCOVER MODEL COEFFICIENTS - NORTH############# - -#1. Get the list of north models---- -north <- mods |> - dplyr::filter(region=="north") - -#2. Get list of species---- -spp <- unique(north$species) - -#3. Get the full list of potential continuous covariates---- -#don't need the categorical ones because they're in every model -names.north <- do.call(rbind, unlist(modelsnorth)) |> - data.frame() |> - separate(X3, into=c("X3", "X4", "X5", "X6", "X7", "X8", "X9", "X10", "X11", "X12", "X13", "X14", "X15", "X16", "X17", "X18", "X19", "X20", "X21", "X22", "X23"), sep = " +") |> - dplyr::select(-X1, -X2, -X3) |> - pivot_longer(cols=X4:X23, names_to="position", values_to="name") |> - dplyr::filter(!is.na(name), name!="+") |> - dplyr::select(-position) |> - unique() |> - mutate(name = fix_names(name)) |> - dplyr::filter(!name %in% c("method", "vegc")) - -#4. Get the full north data---- -covs.n <- dplyr::filter(covs, useNorth==TRUE) - -#5. Get the model matrix---- -Xn <- model.matrix(get_terms(modelsnorth), covs.n) -colnames(Xn) <- fix_names(c("Intercept", colnames(Xn)[-1])) - -#6. Set up loop---- -for(i in 1:length(spp)){ - - #7. List of models for that species---- - north.i <- dplyr::filter(north, species==spp[i]) - - coef.list <- list() - se.list <- list() - for(j in 1:nrow(north.i)){ - - #8. Get the raw coefficients----- - #Add zeros for any covariates that weren't in the best model - coef.j <- read.csv(north.i$path[j]) |> - mutate(name = fix_names(name), - name = ifelse(name=="(Intercept)", "Intercept", name)) |> - full_join(data.frame(name = names.north$name)) |> - mutate(coef = ifelse(is.na(coef), 0, coef), - se = ifelse(is.na(se), 0, se)) |> - suppressMessages() - - raw.j <- coef.j$coef - names(raw.j) <- coef.j$name - - raw_se.j <- coef.j$se - names(raw_se.j) <- coef.j$name - - #9. Translate to standardized & transformed coefficients for the veg coefficients---- - mu <- drop(age %*% raw.j[colnames(age)]) - lam.j <- exp(drop(age %*% raw.j[colnames(age)])) - - mu_se <- drop(age %*% raw_se.j[colnames(age)]) - lam_se.j <- exp(drop(age %*% raw_se.j[colnames(age)])) - - #10. Adjust the linear feature coefficients for competing models---- - #replace others with msoft if it is non-zero - #because mSoft is competed with the others in the model set - hf.j <- coef.j |> - dplyr::select(-se) |> - dplyr::filter(name %in% c("mWell", "mSoft", "mEnSft", "mTrSft", "mSeism")) |> - mutate(coef = exp(coef)) |> - pivot_wider(names_from=name, values_from=coef) |> - mutate(mEnSft = ifelse(mSoft!=0 & mEnSft==0, mSoft, mEnSft), - mTrSft = ifelse(mSoft!=0 & mTrSft==0, mSoft, mTrSft), - mSeism = ifelse(mSoft!=0 & mSeism==0, mSoft, mSeism)) |> - pivot_longer(mWell:mSeism, names_to="name", values_to="coef") |> - data.frame() - - hf_se.j <- coef.j |> - dplyr::select(-coef) |> - dplyr::filter(name %in% c("mWell", "mSoft", "mEnSft", "mTrSft", "mSeism")) |> - mutate(se = exp(se)) |> - pivot_wider(names_from=name, values_from=se) |> - mutate(mEnSft = ifelse(mSoft!=0 & mEnSft==0, mSoft, mEnSft), - mTrSft = ifelse(mSoft!=0 & mTrSft==0, mSoft, mTrSft), - mSeism = ifelse(mSoft!=0 & mSeism==0, mSoft, mSeism)) |> - pivot_longer(mWell:mSeism, names_to="name", values_to="se") |> - data.frame() - - #11. Adjust the linear feature and well coefficients---- - - #Human-modified landcover types that we don't want interfering with adjustment - hfc <- c("Crop", "Industrial", "Mine", "RoughP", "Rural", "TameP", "Urban") - - #Dataframe of variables to adjust - vars <- data.frame(var = c("mWell", "mEnSft", "mTrSft", "mSeism")) - - for(k in 1:nrow(vars)){ - - #Make a mock dataframe for the variable - zero.k <- data.frame(var = 0) - colnames(zero.k) <- vars$var[k] - - #Get the rows of interest from the model matrix - #> 0 proportion of variable, no human-modified landover type, no harvest - rows.k <- covs.n |> - mutate(rowid = row_number()) |> - anti_join(zero.k) |> - dplyr::filter(!vegc %in% hfc & - fcc2==0) |> - suppressMessages() - - Xn.k <- Xn[rows.k$rowid, colnames(age)] - - #Make predictions from those rows using the raw coefficient - lam.k <- exp(Xn.k %*% raw.j[colnames(Xn.k)]) - - #Multiply transformed coefficients by those predictions and take the mean - vars$est[k] <- mean(lam.k * hf.j[hf.j$name==vars$var[k],]$coef) - - #Add some extra tracking information - vars$n[k] <- nrow(rows.k) - vars$original[k] <- exp(raw.j[vars$var[k]]) - - } - - #12. Cap open habitat values---- - #for coef only, not se - - #Get maximum lambda for open habitat types - lam.open <- max(lam.j[c(names(lam.j)[endsWith(names(lam.j), "R")], - "GrassHerb", "Shrub", "GraminoidFen", "Marsh")]) - - #Get maximum lambda for open human footprint types - lam.hf <- max(lam.j[c("Industrial", "Rural", "Urban")]) - - #Overall cap value - lam.max <- max(lam.open, lam.hf) - - #Cap the linear features---- - linear <- vars |> - rowwise() |> - mutate(capped = min(est, lam.max)) |> - ungroup() |> - mutate(cap_open = lam.open, - cap_hf = lam.hf, - cap_total = lam.max, - species = spp[i], - boot = j, - name = c("Wellsites", "EnSoftLin", "TrSoftLin", "EnSeismic")) - - linear.j <- linear$capped - names(linear.j) <- linear$name - - linear_se.j <- hf_se.j[hf_se.j$name %in% linear$var, "se"] - names(linear_se.j) <- linear$name - - #13. Put together---- - lam.out <- c(Climate = exp(raw.j["climate"]), - lam.j[names(lam.j)!="Mine"], - linear.j, - HardLin = 0, - Water = 0, - Bare = 0, - SnowIce = 0, - Mine = 0, - MineV = unname(lam.j["Mine"])) - names(lam.out) <- gsub("Spruce", "WhiteSpruce", names(lam.out)) - names(lam.out) <- gsub("Decid", "Deciduous", names(lam.out)) - names(lam.out) <- gsub("Climate.climate", "Climate", names(lam.out)) - names(lam.out) <- gsub("TreedBog", "BlackSpruce", names(lam.out)) - - se.out <- c(Climate = exp(raw_se.j["climate"]), - lam_se.j[names(lam_se.j)!="Mine"], - linear_se.j, - HardLin = 0, - Water = 0, - Bare = 0, - SnowIce = 0, - Mine = 0, - MineV = unname(lam_se.j["Mine"])) - names(se.out) <- gsub("Spruce", "WhiteSpruce", names(se.out)) - names(se.out) <- gsub("Decid", "Deciduous", names(se.out)) - names(se.out) <- gsub("Climate.climate", "Climate", names(se.out)) - names(se.out) <- gsub("TreedBog", "BlackSpruce", names(se.out)) - - #14. Transform back and cap values---- - lam.final <- log(lam.out) - lam.final[lam.final > 10^4] <- 10^4 - lam.final[lam.final < -10^4] <- -10^4 - - coef.list[[j]] <- lam.final - - se.final <- log(se.out) - se.final[is.infinite(se.final)] <- 0 - - se.list[[j]] <- se.final - - } - - #15. Make the array if first species---- - if(i==1){ - joint.n <- array(0, c(length(spp), length(coef.list[[j]]), nrow(north.i))) - dimnames(joint.n) <- list(spp, names(coef.list[[j]]), paste0("b", c(1:nrow(north.i)))) - - joint_se.n <- array(0, c(length(spp), length(se.list[[j]]), nrow(north.i))) - dimnames(joint_se.n) <- list(spp, names(se.list[[j]]), paste0("b", c(1:nrow(north.i)))) - } - - #16. Add species to array---- - joint.n[spp[i],,] <- as.matrix(do.call(cbind, coef.list)) - joint_se.n[spp[i],,] <- as.matrix(do.call(cbind, se.list)) - - cat("Finished species", i, "of", length(spp), "\n") - -} - -#17. Truncate NESP climate coef because it blows up the relative abundance prediction---- -joint.n["NESP","Climate",] <- joint.n["NESP","Climate",]*0.1 - -#LANDCOVER MODEL COEFFICIENTS - SOUTH############# - -#1. Get the list of north models---- -south <- mods |> - dplyr::filter(region=="south") - -#2. Get list of species---- -spp <- unique(south$species) - -#3. Get the full list of potential continuous covariates---- -#don't need the categorical ones because they're in every model -names.south <- do.call(rbind, unlist(modelssouth)) |> - data.frame() |> - separate(X3, into=c("X3", "X4", "X5", "X6", "X7", "X8", "X9"), sep = " +") |> - dplyr::select(-X1, -X2, -X3) |> - pivot_longer(cols=X4:X9, names_to="position", values_to="name") |> - dplyr::filter(!is.na(name), name!="+") |> - dplyr::select(-position) |> - unique() |> - mutate(name = fix_names(name)) |> - dplyr::filter(!name %in% c("method", "soilc")) - -#4. Get the full south data---- -covs.s <- dplyr::filter(covs, useSouth==TRUE, !is.na(soilc)) - -#5. Get the model matrix---- -#use formula from north section above -Xs <- model.matrix(get_terms(modelssouth), covs.s) -colnames(Xs) <- fix_names(c("Intercept", colnames(Xs)[-1])) - -#6. Set up loop---- -for(i in 1:length(spp)){ - - #7. List of models for that species---- - south.i <- dplyr::filter(south, species==spp[i]) - - coef.list <- list() - se.list <- list() - for(j in 1:nrow(south.i)){ - - #9. Get the raw coefficients----- - #Add zeros for any covariates that weren't in the best model - coef.j <- read.csv(south.i$path[j]) |> - mutate(name = fix_names(name), - name = ifelse(name=="(Intercept)", "Intercept", name)) |> - full_join(data.frame(name = names.south$name)) |> - mutate(coef = ifelse(is.na(coef), 0, coef), - se = ifelse(is.na(se), 0, se)) |> - suppressMessages() - - raw.j <- coef.j$coef - names(raw.j) <- coef.j$name - - raw_se.j <- coef.j$se - names(raw_se.j) <- coef.j$name - - #10. Transform the soil estimates and handle intercept---- - names.soil <- c("soilcBlowout", "soilcClaySub", "soilcCrop", - "soilcIndustrial", "soilcMine", "soilcOther", "soilcRapidDrain", - "soilcRoughP", "soilcRural", "soilcSandyLoam", "soilcSoilUnknown", - "soilcTameP", "soilcThinBreak", "soilcUrban", "soilcWater", "soilcWellsites") - - lam.soil <- exp(c(raw.j[1], raw.j[1] + raw.j[names.soil])) - names(lam.soil) <- levels(covs.s$soilc) - - lam_se.soil <- exp(c(raw_se.j[1], raw_se.j[1] + raw_se.j[names.soil])) - names(lam_se.soil) <- levels(covs.s$soilc) - - #11. Adjust the linear feature and well coefficients---- - - #Human-modified landcover types that we don't want interfering with adjustment - hfc <- c("Crop", "Industrial", "Mine", "RoughP", "Rural", "TameP", "Urban") - - #Dataframe of variables to adjust - vars <- data.frame(var = c("mWell", "mSoft")) - - for(k in 1:nrow(vars)){ - - #Make a mock dataframe for the variable - zero.k <- data.frame(var = 0) - colnames(zero.k) <- vars$var[k] - - #Get the rows of interest from the model matrix - #> 0 proportion of variable, no human-modified landover type, no harvest - rows.k <- covs.s |> - mutate(rowid = row_number()) |> - anti_join(zero.k) |> - dplyr::filter(!vegc %in% hfc) |> - suppressMessages() - - Xs.k <- Xs[rows.k$rowid,] - - #Make predictions from those rows using the raw coefficient - lam.k <- exp(Xs.k %*% raw.j[colnames(Xs)]) - - #Multiply transformed coefficient by those predictions and take the mean - vars$est[k] <- mean(lam.k * exp(raw.j[vars$var[k]])) - - #Add some extra tracking information - vars$n[k] <- nrow(rows.k) - vars$original[k] <- exp(raw.j[vars$var[k]]) - - } - - #12. Cap open habitat values---- - #Get maximum lambda for open habitat types - lam.open <- max(lam.soil[c("Loamy", "Blowout", "ClaySub", "RapidDrain", "SandyLoam", "ThinBreak", "Other")]) - - #Get maximum lambda for open human footprint types - lam.hf <- max(lam.soil[c("Industrial", "Rural", "Urban")]) - - #Overall cap value - lam.max <- max(lam.open, lam.hf) - - #Cap the linear features---- - linear <- vars |> - rowwise() |> - mutate(capped = min(est, lam.max)) |> - ungroup() |> - mutate(cap_open = lam.open, - cap_hf = lam.hf, - cap_total = lam.max, - species = spp[i], - boot = j, - name = c("Wellsites", "EnSoft")) - - linear.j <- linear$capped - names(linear.j) <- linear$name - - #13. Put together---- - lam.out <- c(Climate = exp(raw.j["climate"]), - lam.soil[!names(lam.soil) %in% c("Mine", "Water", "Wellsites")], - pAspen = exp(unname(raw.j["paspen"])), - Wellsites = linear$capped[1], - EnSeismic = linear$capped[2], - EnSoftLin = linear$capped[2], - TrSoftLin = linear$capped[2], - HardLin = 0, - Water = 0, - Mine = 0, - MineV = unname(lam.soil["Mine"])) - names(lam.out) <- gsub("Climate.climate", "Climate", names(lam.out)) - - se.out <- c(Climate = exp(raw_se.j["climate"]), - lam_se.soil[!names(lam_se.soil) %in% c("Mine", "Water", "Wellsites")], - Wellsites = exp(raw_se.j["mWell"]), - EnSeismic = exp(raw_se.j["mSoft"]), - EnSoftLin = exp(raw_se.j["mSoft"]), - TrSoftLin = exp(raw_se.j["mSoft"]), - HardLin = 0, - Water = 0, - Mine = 0, - MineV = unname(lam_se.soil["Mine"])) - names(se.out) <- gsub("Climate.climate", "Climate", names(se.out)) - - #14. Transform back and cap values---- - lam.final <- log(lam.out) - lam.final[lam.final > 10^4] <- 10^4 - lam.final[lam.final < -10^4] <- -10^4 - - coef.list[[j]] <- t(lam.final) - - se.final <- log(se.out) - se.final[is.infinite(se.final)] <- 0 - - se.list[[j]] <- t(se.final) - - } - - #15. Make the array if first species---- - if(i==1){ - joint.s <- array(0, c(length(spp), length(coef.list[[j]]), nrow(south.i))) - dimnames(joint.s) <- list(spp, names(lam.final), paste0("b", c(1:nrow(south.i)))) - - joint_se.s <- array(0, c(length(spp), length(se.list[[j]]), nrow(south.i))) - dimnames(joint_se.s) <- list(spp, names(se.final), paste0("b", c(1:nrow(south.i)))) - } - - #16. Add species to array---- - joint.s[spp[i],,] <- as.matrix(do.call(cbind, coef.list)) - joint_se.s[spp[i],,] <- as.matrix(do.call(cbind, se.list)) - - cat("Finished species", i, "of", length(spp), "\n") - -} - -#17. Truncate NESP climate coef because it blows up the relative abundance prediction---- -joint.n["NESP","Climate",] <- joint.n["NESP","Climate",]*0.1 - -#MEDIAN BOOTSTRAP SELECTION######## - -#1. List of species*region---- -regions <- mods |> - dplyr::select(species, region) |> - unique() - -todo <- regions |> - dplyr::select(species) |> - unique() - -#2. Set up loop---- -boots <- data.frame() -for(i in 1:nrow(todo)){ - - #3. Settings---- - spp.i <- todo$species[i] - regions.i <- regions |> - dplyr::filter(species==spp.i) - region.i <- ifelse(nrow(regions.i)==2, "both", unique(regions.i$region)) - - #4. Get the right matrix---- - if("north" %in% regions.i$region){ - north.i <- t(as.matrix(joint.n[spp.i,,])) - } - if("south" %in% regions.i$region){ - south.i <- t(as.matrix(joint.s[spp.i,,])) - } - - if(region.i=="both"){coefs.i <- cbind(north.i, south.i)} - if(region.i=="north"){coefs.i <- north.i} - if(region.i=="south"){coefs.i <- south.i} - - #5. Run an NMDS---- - set.seed(i) - nmds.i <- metaMDS(coefs.i, k=2, distance="euclidean", trace=0) - - #6. Get the site scores---- - scores.i <- scores(nmds.i, display="sites") - - #7. Calculate centroid---- - centroid.i <- colMeans(scores.i) - - #8. Get distances to centroid---- - dist.i <- data.frame(scores.i) |> - mutate(distance = sqrt((NMDS1-centroid.i[1])^2 + (NMDS2-centroid.i[2])^2)) |> - rownames_to_column() |> - rename(bootstrap = rowname) |> - mutate(bootstrap = as.numeric(str_sub(bootstrap, 2, 3))) - - #9. Get closest bootstrap---- - min.i <- dist.i |> - dplyr::filter(distance==min(distance)) - - boots <- rbind(boots, - min.i |> - mutate(species = spp.i, - region = region.i)) - - cat("Finished nmds", i, "of", nrow(todo), "\n") - - -} - -#MODEL EVALUATION######### - -#1. Peter's AUC functions---- -simple_roc <- function(labels, scores){ - Labels <- labels[order(scores, decreasing=TRUE)] - data.frame( - TPR=cumsum(Labels)/sum(Labels), - FPR=cumsum(!Labels)/sum(!Labels), - Labels=Labels) -} - -simple_auc <- function(ROC) { - ROC$inv_spec <- 1-ROC$FPR - dx <- diff(ROC$inv_spec) - sum(dx * ROC$TPR[-1]) / sum(dx) -} - -#2. Get list of median models---- -loop <- data.frame(path = list.files(file.path(bird_results_root, "LandcoverModels", "Models"), full.names = TRUE, pattern="*.Rdata", recursive = TRUE), - file = list.files(file.path(bird_results_root, "LandcoverModels", "Models"), pattern="*.Rdata", recursive = TRUE)) |> - separate(file, into=c("region", "sp1", "region2", "species", "bootstrap", "filetype")) |> - mutate(bootstrap = as.numeric(bootstrap)) |> - dplyr::select(-filetype, -region2, -sp1) |> - inner_join(rbind(boots |> - dplyr::filter(region %in% c("north", "south")), - boots |> - dplyr::filter(region=="both") |> - mutate(region="north"), - boots |> - dplyr::filter(region=="both") |> - mutate(region="south")), - by=c("species", "region", "bootstrap")) - -#3. Set up loop---- -loop$AUC_binary <- NA -loop$AUC_poisson <- NA -for(i in 1:nrow(loop)){ - - #4. Load the model---- - load.i <-try(load(loop$path[i])) - if(inherits(load.i, "try-error")){ next } - - #5. Make predictions on training data---- - pred.i <- predict(bestmodel, type="response") - - #6. AUC for binary response---- - bestmodel$binary <- ifelse(bestmodel$y > 1, 1, bestmodel$y) - roc.i <- try(roc(response=bestmodel$binary, predictor=pred.i, quiet=TRUE)) - if(inherits(roc.i, "try-error")){loop$AUC_binary[i] <- NA} else {loop$AUC_binary[i] <- auc(roc.i)} - - #7. AUC for count response---- - roc.p <- simple_roc(labels=bestmodel$y, scores=pred.i) - loop$AUC_poisson[i] <- simple_auc(roc.p) - - cat("Finished model ", i, "of", nrow(loop), "\n") - - -} - -#8. Tidy---- -auc.out <- loop -rm(loop) - -#PACKAGING########## - -#1. Change array names from code to common name---- - -#Get lookup table -birdnames <- read.csv(file.path(root, "Data", "lookups", "birdlist.csv")) - -#Match -birdnames.m <- birdnames |> - arrange(code) |> - dplyr::filter(code %in% dimnames(marginal)[[1]]) - -birdnames.n <- birdnames |> - arrange(code) |> - dplyr::filter(code %in% dimnames(joint.n)[[1]]) - -birdnames.s <- birdnames |> - arrange(code) |> - dplyr::filter(code %in% dimnames(joint.s)[[1]]) - -#Replace -dimnames(marginal) <- list(birdnames.m$sppid, dimnames(marginal)[[2]], dimnames(marginal)[[3]]) - -dimnames(marginal_se) <- list(birdnames.m$sppid, dimnames(marginal_se)[[2]], dimnames(marginal)[[3]]) - -dimnames(joint.n) <- list(birdnames.n$sppid, dimnames(joint.n)[[2]], dimnames(marginal)[[3]]) - -dimnames(joint_se.n) <- list(birdnames.n$sppid, dimnames(joint_se.n)[[2]], dimnames(marginal)[[3]]) - -dimnames(joint.s) <- list(birdnames.s$sppid, dimnames(joint.s)[[2]], dimnames(marginal)[[3]]) - -dimnames(joint_se.s) <- list(birdnames.s$sppid, dimnames(joint_se.s)[[2]], dimnames(marginal)[[3]]) - -#2. Bird look up table----- - -#Get the show details -birds2024 <- read.csv(file.path(root, "Data", "lookups", "birds-v2024.csv")) |> - rename(species = common) |> - dplyr::select(species, show) - -#Get the occurrence totals -occurrence <- bird |> - pivot_longer(ALFL:YRWA, names_to="code", values_to="count") |> - dplyr::filter(count > 0) |> - group_by(code) |> - summarize(Occurrences = n()) |> - ungroup() - -#Get the AUC values -auc <- auc.out |> - rename(code = species) |> - group_by(code, region) |> - summarize(AUC = mean(AUC_poisson)) |> - ungroup() |> - pivot_wider(names_from="region", values_from=c("AUC")) - -#Wrangle the median bootstrap -bootuse <- boots |> - dplyr::select(species, bootstrap) |> - rename(code = species) - -#set species to remove -spp.remove <- c("VEER", "LEYE") - -#set different plot truncation quantile species -q990 <- c("NOFL", "BTNW", "BLPW", "CEDW", "CMWA", "COGR", "EAPH", "HOSP", "PUFI", "SOSP", "WBNU", "YHBL", "YRWA") -q999 <- c("BLBW", "MOCH", "WCSP") - -#Make the table -birdtable <- birdnames |> - dplyr::filter(sppid %in% dimnames(joint.n)[[1]] | sppid %in% dimnames(joint.s)[[1]]) |> - left_join(birds2024) |> - left_join(occurrence) |> - left_join(auc) |> - left_join(bootuse) |> - mutate(ModelNorth = ifelse(show %in% c("c", "n"), TRUE, FALSE), - ModelSouth = ifelse(show %in% c("c", "s"), TRUE, FALSE), - LinkHabitat = "log", - LinkSpclim = "log", - Group = "birds", - Nonnative = FALSE, - Use = ifelse(code %in% spp.remove, FALSE, TRUE), - PlotQuantile = case_when(code %in% q990 ~ 0.990, - code %in% q999 ~ 0.999, - !is.na(code) & Use==TRUE ~ 0.995)) |> - rename(SpeciesID = sppid, - ScientificName = scinam, - CommonName = species, - Comments = code) |> - rename(AUCNorth = north, AUCSouth = south, Bootstrap = bootstrap) |> - dplyr::select(SpeciesID, ScientificName, CommonName, ModelNorth, ModelSouth, Occurrences, Nonnative, LinkHabitat, LinkSpclim, AUCNorth, AUCSouth, Bootstrap, Comments, Group, Use, PlotQuantile) - -#3. Put together---- -birds <- list( - north = list(marginal = marginal, marginal_se = marginal_se, joint = joint.n, joint_se = joint_se.n), - south = list(marginal = marginal, marginal_se = marginal_se, joint = joint.s, joint_se = joint_se.s), - species = birdtable -) - -#4. Save---- -save(birds, file = out_file) - -cat("\nWrote ", out_file, "\n", sep = "") - -# End of script ---- diff --git a/1_code/_setup/05_harmonize_v2_results.R b/1_code/_setup/05_harmonize_v2_results.R deleted file mode 100644 index 4745460..0000000 --- a/1_code/_setup/05_harmonize_v2_results.R +++ /dev/null @@ -1,1151 +0,0 @@ -# --- -# title: Harmonize the v2 Results into One Parity Reference -# author: Brendan Casey -# created: 2026-09-13 -# inputs: -# plants, in the VegetationModels project: -# - 3_output/coefficients/COEFS.RData -# mammals, on the ABMI Mammals drive: -# - .../Landcover/North/Coefficient Tables/*.Rdata -# - .../Landcover/South/Coefficient Tables/*.Rdata -# - .../Climate/Coefficients/* Climate Coefficients.RData -# birds, preferred: -# - 2_pipeline/v2_reference/Birds2024.RData, from -# 1_code/_setup/04_package_bird_coefficients.R -# birds, fallback, on the work_abmi shared drive reached -# through remote/BirdModels.lnk: -# - Results/Archive/ClimateModels/Coefficients//*.csv -# - Results/Archive/LandcoverModels/Coefficients// -# /*.csv -# outputs: -# - 0_data/v2_results/v2_results.csv -# - 0_data/v2_results/v2_results_coverage.csv -# notes: -# - One long table of the published v2 results, so that a run -# in 3_output/exp_000_parity_v2/ can be scored against a -# single file rather than against three storage formats on -# two drives. -# -# - The v2 outputs are stored three different ways. Plants are -# 3-D arrays of species by term by bootstrap draw. Mammal -# landcover is a species-by-term matrix with no draws, since -# that stage fits once. Mammal climate is one small data -# frame per species. This flattens all three to the same -# columns, and records `v2_n` so a consumer can tell a -# 100-draw distribution from a single fitted value. -# -# - Plants come from COEFS.RData rather than from the -# per-taxon model files. That is v2's own standardization -# output, and its header says it exists "to allow for the -# checking of model coefficients between versions", which is -# this exact use. It also sidesteps a trap: the published -# bryophyte-species-models.Rdata is entirely error objects, -# while the bryophyte arrays inside COEFS.RData are complete -# and finite. COEFS was built before that file was -# overwritten by a failed re-run. -# -# - Mammal seasons are averaged, not selected. v2 writes one -# row per species in its `.all` tables, holding the mean of -# the summer and winter fits where both exist and the single -# fit where only one does. The harness models seasons -# separately, so a `Moose_Summer` result is NOT comparable -# to the moose row here unless moose is summer-only. The -# season column says "all" to make that visible rather than -# leaving it to be discovered. Comparing properly means -# averaging the harness's two seasons first. -# -# - Plant climate is fitted once on all data, not per region, -# so those rows carry region "all". A run's north and south -# climate results are both scored against them. -# -# - Birds are the slow one, and are cached. Their coefficients -# are stored as one small csv per species per draw, roughly -# 31,000 files, on a Google Drive mount that costs about -# 470 ms per file opened serially. Reading them twelve at a -# time brings a full pass to about half an hour, and the -# result is cached so a re-run is seconds. Delete the cache -# or set refresh_birds to rebuild it. -# -# - The bird path in the v2 scripts, G:/Shared drives/ -# ABMI_RHedley/Projects/BirdModels, is not mounted here. The -# same directory is reached through a shortcut on the -# work_abmi drive, which resolves to a shortcut-targets-by-id -# path. That resolved path is the default below. -# -# - Bird results are taken from Results/Archive/, not from -# Results/. The live directory holds a partial re-run of a -# single species from August 2026; the archive holds the last -# complete set, 133 species for climate and 112 north and 68 -# south for landcover. The two disagree numerically, as two -# runs of an unseeded pipeline must, so they are not mixed. -# The script picks whichever has more species and says which -# it used. -# -# - Read only. Nothing is written to either source drive. -# --- - -# 1. Setup ---- - -## 1.1 Load packages ---- -library(data.table) # fast csv reading and writing (v: 1.16.4) -library(parallel) # bird reads are latency-bound (version: 4.5.0) - -## 1.2 Resolve the source locations ---- -# Overridable, because the drives are mounted differently on -# different machines. -project_root <- normalizePath(getwd(), winslash = "/") - -v2_plant_project <- Sys.getenv( - "SDM_V2_PROJECT", - unset = "//ABMI-DATA2/science/sc/ToEmily/VegetationModels" -) - -v2_mammal_root <- Sys.getenv( - "SDM_V2_MAMMAL_ROOT", - unset = paste0( - "G:/Shared drives/ABMI Mammals/Results/Habitat Modeling/2024" - ) -) - -# The path in the v2 scripts is not mounted. This is the same -# directory reached through remote/BirdModels.lnk on work_abmi. -v2_bird_root <- Sys.getenv( - "SDM_V2_BIRD_ROOT", - unset = paste0( - "G:/.shortcut-targets-by-id/", - "17Ymt13eHfKvIiuoMl6x-Kn74Z2uVbbzS/BirdModels" - ) -) - -# Birds cost about half an hour on a cold cache. FALSE skips -# them and records the taxon as deliberately excluded. -include_birds <- TRUE - -# TRUE ignores the cache and re-reads every file. -refresh_birds <- FALSE - -# 1 reads serially, which is the only setting measured to be -# reliable here: concurrency trips a refusal state in Google -# Drive that outlasts retries. Serial costs about three hours -# for the full set, once, and then it is cached. -bird_workers <- 1L - -# The harmonized reference is analysis-ready data, so it sits in -# 0_data alongside v2_scripts rather than in 2_pipeline. What -# stays in 2_pipeline is genuinely intermediate: the bird read -# cache, and the packaged bird object that 04 writes. -results_dir <- file.path(project_root, "0_data/v2_results") - -out_dir <- file.path(project_root, "2_pipeline/v2_reference") - -for (one in c(results_dir, out_dir)) { - dir.create(one, recursive = TRUE, showWarnings = FALSE) -} - -## 1.3 Name the output columns ---- -# Declared once so every reader returns the same frame and a -# missing source cannot quietly change the schema. -result_columns <- c( - "taxon", "region", "stage", "part", "season", - "species", "species_v2", "term", - "v2_median", "v2_p10", "v2_p90", "v2_se", "v2_n", "source" -) - -# 2. Plants ---- -# COEFS[[group]][[part]] is species x term x draw. - -## 2.1 Name the mapping from v2 groups to data slugs ---- -plant_groups <- c( - Bryophytes = "bryophyte", - Lichens = "lichen", - Mites = "mite", - VascularPlants = "vascular_plant" -) - -# v2 fits vegetation in the north and soil in the south. Climate -# is fitted once on everything. -plant_parts <- list( - vegetation = list(region = "north", stage = "habitat"), - soil = list(region = "south", stage = "habitat"), - climate = list(region = "all", stage = "climate") -) - -## 2.2 read_plant_results() ---- - -#' Flatten the Plant Coefficient Arrays -#' -#' @param path Character. Path to COEFS.RData. -#' @return A data frame in the output schema, or NULL. -#' -#' @example # Example usage of the function -#' # read_plant_results("3_output/coefficients/COEFS.RData") -read_plant_results <- function(path) { - if (!file.exists(path)) { - return(NULL) - } - - env <- new.env() - load(path, envir = env) - - if (is.null(env$COEFS)) { - return(NULL) - } - - rows <- list() - - for (group in names(env$COEFS)) { - slug <- plant_groups[[group]] - - if (is.null(slug)) { - next - } - - for (part in names(plant_parts)) { - values <- env$COEFS[[group]][[part]] - - if (is.null(values) || length(dim(values)) != 3) { - next - } - - # Summarized across draws, which is dimension three. The - # band is what the gate scores against, so it is computed - # here once rather than in every consumer. - summarize <- function(f) { - apply(values, c(1, 2), function(x) { - x <- x[is.finite(x)] - - if (length(x) == 0) { - return(NA_real_) - } - - f(x) - }) - } - - medians <- summarize(stats::median) - p10 <- summarize(function(x) { - stats::quantile(x, 0.1, names = FALSE) - }) - p90 <- summarize(function(x) { - stats::quantile(x, 0.9, names = FALSE) - }) - counts <- apply(values, c(1, 2), function(x) { - sum(is.finite(x)) - }) - - species <- rownames(medians) - terms <- colnames(medians) - - block <- data.frame( - taxon = slug, - region = plant_parts[[part]]$region, - stage = plant_parts[[part]]$stage, - part = "detection", - season = NA_character_, - species = rep(species, times = length(terms)), - species_v2 = rep(species, times = length(terms)), - term = rep(terms, each = length(species)), - v2_median = as.vector(medians), - v2_p10 = as.vector(p10), - v2_p90 = as.vector(p90), - v2_se = NA_real_, - v2_n = as.vector(counts), - source = basename(path), - stringsAsFactors = FALSE - ) - - rows[[paste(group, part)]] <- block[block$v2_n > 0, ] - } - } - - if (length(rows) == 0) { - return(NULL) - } - - do.call(rbind, rows) -} - -# 3. Mammals ---- - -## 3.1 mammal_key() ---- - -#' Reduce a v2 Mammal Name to the Harness Key -#' -#' v2 names species with spaces and punctuation, and the -#' harmonized response squashes them. Reducing both to letters -#' is what the existing comparison already does. -#' -#' @param x Character vector of v2 species names. -#' @return Character vector of keys. -#' -#' @example # Example usage of the function -#' # mammal_key("Elk (wapiti)") # "Elkwapiti" -mammal_key <- function(x) { - gsub("[^A-Za-z]", "", x) -} - -## 3.2 read_mammal_landcover() ---- - -#' Flatten One Mammal Landcover Coefficient Table -#' -#' The three reported quantities are different things, so each -#' is kept as its own `part` rather than merged: presence is a -#' probability, abundance-given-presence is on the response -#' scale, and total is their calibrated product. -#' -#' @param dir_path Character. A Coefficient Tables directory. -#' @param region Character. "north" or "south". -#' @return A data frame in the output schema, or NULL. -#' -#' @example # Example usage of the function -#' # read_mammal_landcover(dir_path, "north") -read_mammal_landcover <- function(dir_path, region) { - if (!dir.exists(dir_path)) { - return(NULL) - } - - files <- sort(list.files( - dir_path, pattern = "[.]Rdata$", full.names = TRUE - )) - - if (length(files) == 0) { - return(NULL) - } - - # The newest table wins. v2 keeps dated copies side by side. - path <- files[length(files)] - - env <- new.env() - load(path, envir = env) - - parts <- list( - presence = c("Coef.pa.all", "Coef.pa.se.all"), - abundance = c("Coef.agp.all", "Coef.agp.se.all"), - total = c("Coef.mean.all", "Coef.mean.se.all") - ) - - rows <- list() - - for (part in names(parts)) { - estimates <- env[[parts[[part]][1]]] - errors <- env[[parts[[part]][2]]] - - if (is.null(estimates)) { - next - } - - species <- rownames(estimates) - terms <- colnames(estimates) - - # The standard errors are a separate matrix and can carry a - # different term set, so they are matched by name. - se_values <- if (is.null(errors)) { - rep(NA_real_, length(estimates)) - } else { - as.vector(errors[ - match(species, rownames(errors)), - match(terms, colnames(errors)), - drop = FALSE - ]) - } - - rows[[part]] <- data.frame( - taxon = "mammal", - region = region, - stage = "habitat", - part = part, - # See the header: this is the mean of the seasons where - # both were fitted, not one of them. - season = "all", - species = rep(mammal_key(species), times = length(terms)), - species_v2 = rep(species, times = length(terms)), - term = rep(terms, each = length(species)), - v2_median = as.vector(estimates), - v2_p10 = NA_real_, - v2_p90 = NA_real_, - v2_se = se_values, - v2_n = 1L, - source = basename(path), - stringsAsFactors = FALSE - ) - } - - if (length(rows) == 0) { - return(NULL) - } - - out <- do.call(rbind, rows) - - out[is.finite(out$v2_median), ] -} - -## 3.3 read_mammal_climate() ---- - -#' Flatten the Mammal Climate Coefficients -#' -#' One file per species, each holding an AICc-averaged -#' coefficient per term. The mammal climate pipeline does not -#' bootstrap, so there is no band. -#' -#' @param dir_path Character. The Climate/Coefficients folder. -#' @return A data frame in the output schema, or NULL. -#' -#' @example # Example usage of the function -#' # read_mammal_climate(dir_path) -read_mammal_climate <- function(dir_path) { - if (!dir.exists(dir_path)) { - return(NULL) - } - - files <- list.files( - dir_path, pattern = "Climate Coefficients[.]RData$", - full.names = TRUE - ) - - if (length(files) == 0) { - return(NULL) - } - - rows <- lapply(files, function(path) { - env <- new.env() - load(path, envir = env) - - if (is.null(env$avg_coef)) { - return(NULL) - } - - name <- sub( - " Climate Coefficients[.]RData$", "", basename(path) - ) - - data.frame( - taxon = "mammal", - region = "all", - stage = "climate", - part = "presence", - season = "all", - species = mammal_key(name), - species_v2 = name, - # v2 names the intercept with parentheses here and without - # them in the plant tables. Normalized to the plant form, - # which is what the comparison already expects. - term = sub( - "^\\(Intercept\\)$", "Intercept", - as.character(env$avg_coef$term) - ), - v2_median = as.numeric(env$avg_coef$coef), - v2_p10 = NA_real_, - v2_p90 = NA_real_, - v2_se = NA_real_, - v2_n = 1L, - source = basename(path), - stringsAsFactors = FALSE - ) - }) - - out <- do.call(rbind, rows) - - if (is.null(out)) { - return(NULL) - } - - out[is.finite(out$v2_median), ] -} - -# 4. Birds ---- -# One csv per species per draw, on a high-latency mount. - -## 4.1 bird_source_dir() ---- - -#' Choose Between the Live and Archived Bird Results -#' -#' The live directory holds a partial re-run; the archive holds -#' the last complete set. Two runs of an unseeded pipeline give -#' different numbers, so they are never mixed: whichever covers -#' more species wins outright. -#' -#' @param root Character. The BirdModels directory. -#' @param leaf Character. Path below Results, for example -#' "ClimateModels/Coefficients". -#' @return A list of dir, label and species count, or NULL. -#' -#' @example # Example usage of the function -#' # bird_source_dir(root, "ClimateModels/Coefficients") -bird_source_dir <- function(root, leaf) { - options <- list( - live = file.path(root, "Results", leaf), - archive = file.path(root, "Results", "Archive", leaf) - ) - - counts <- vapply(options, function(one) { - if (!dir.exists(one)) { - return(-1L) - } - - length(list.dirs(one, recursive = FALSE)) - }, integer(1)) - - if (all(counts < 0)) { - return(NULL) - } - - best <- names(counts)[which.max(counts)] - - list( - dir = options[[best]], - label = best, - species = counts[[best]] - ) -} - -## 4.2 read_bird_csv_set() ---- - -#' Read Every Draw for Every Species Under One Directory -#' -#' Worked one species at a time, with retries, because this -#' mount fails under sustained parallel load. Two things were -#' learned the hard way and are encoded here. -#' -#' A single parallel call over all 13,300 files kills the socket -#' cluster part-way through and loses everything, so the unit of -#' work is one species, about a hundred files. -#' -#' Under load, Google Drive starts refusing to materialize files -#' with "Invalid request code". The failure is transient: every -#' file that failed in one pass read correctly moments later. -#' So a failed read is retried inside the worker after a pause, -#' failed species are retried again at the end, and the caller -#' is told what is still missing rather than being handed a -#' quietly short result. -#' -#' @param dir_path Character. Holds one directory per species. -#' @param term_column Character. Which column names the term. -#' The climate files write rownames into an unnamed first -#' column, which fread calls V1; the landcover files carry a -#' "name" column instead. -#' @param workers Integer. Parallel readers. Kept low on -#' purpose: throughput is not the constraint, the mount is. -#' @param attempts Integer. Passes over the species that failed. -#' @return A data frame of species, term, boot, coef and se, -#' carrying a "failed" attribute naming any species missing. -#' -#' @example # Example usage of the function -#' # read_bird_csv_set(dir_path, "V1", workers = 4) -read_bird_csv_set <- function(dir_path, term_column, workers, - attempts = 3L) { - species_dirs <- list.dirs(dir_path, recursive = FALSE) - - if (length(species_dirs) == 0) { - return(NULL) - } - - cat(" ", length(species_dirs), " species directories\n", - sep = "") - - # workers = 1 means no cluster at all. Measured on this mount: - # 400 consecutive serial reads succeeded with zero failures at - # about 360 ms each, where four parallel readers threw the - # mount into a refusal state that outlasted several retries. - # Concurrency, not volume, is what trips it. - serial <- workers <= 1L - - cluster <- if (serial) NULL else makeCluster(workers) - - on.exit( - if (!is.null(cluster)) try(stopCluster(cluster), silent = TRUE), - add = TRUE - ) - - # Retried inside the worker, because a refusal is transient - # and a short pause is cheaper than losing the species. - read_file <- function(path) { - for (try_n in 1:4) { - x <- tryCatch( - data.table::fread(path), error = function(e) NULL - ) - - if (!is.null(x) && nrow(x) > 0) { - return(as.data.frame(x)) - } - - Sys.sleep(0.25 * try_n) - } - - NULL - } - - if (!serial) { - clusterExport(cluster, "read_file", envir = environment()) - } - - read_species <- function(files) { - if (serial) { - return(lapply(files, read_file)) - } - - parLapply(cluster, files, read_file) - } - - started <- Sys.time() - collected <- list() - pending <- species_dirs - - for (pass in seq_len(attempts)) { - if (length(pending) == 0) { - break - } - - if (pass > 1) { - cat(" retry pass ", pass, " for ", length(pending), - " species\n", sep = "") - - # A refusal state persists well past twenty seconds, which - # is why the first retrying version still failed every - # pass. Two minutes is what it takes to clear. - Sys.sleep(120) - } - - still_pending <- character(0) - - for (index in seq_along(pending)) { - one_dir <- pending[index] - species <- basename(one_dir) - - files <- list.files( - one_dir, pattern = "[.]csv$", full.names = TRUE - ) - - if (length(files) == 0) { - next - } - - parts <- tryCatch( - read_species(files), - error = function(e) NULL - ) - - # A dead cluster is the other failure mode; rebuild it. - if (is.null(parts) && !serial) { - try(stopCluster(cluster), silent = TRUE) - cluster <- makeCluster(workers) - clusterExport( - cluster, "read_file", envir = environment() - ) - - parts <- tryCatch( - read_species(files), - error = function(e) NULL - ) - } - - usable <- !vapply(parts, is.null, logical(1)) - - if (is.null(parts) || !any(usable)) { - still_pending <- c(still_pending, one_dir) - next - } - - draws <- suppressWarnings(as.integer(sub( - "[.]csv$", "", sub(".*_", "", basename(files)) - ))) - - rows <- lapply(which(usable), function(i) { - x <- parts[[i]] - - if (!term_column %in% names(x)) { - return(NULL) - } - - data.frame( - species = species, - term = as.character(x[[term_column]]), - boot = draws[i], - coef = as.numeric(x$coef), - se = as.numeric(x$se), - stringsAsFactors = FALSE - ) - }) - - collected[[species]] <- do.call(rbind, rows) - - # A species read only in part is worth keeping but is not - # complete, so it goes round again. - if (sum(usable) < length(files)) { - still_pending <- c(still_pending, one_dir) - } - - if (index %% 20 == 0 || index == length(pending)) { - cat(" ", index, " of ", length(pending), " (", - format(round(Sys.time() - started, 1)), ")\n", - sep = "") - } - } - - pending <- still_pending - } - - failed <- basename(pending) - - if (length(failed) > 0) { - cat(" incomplete after ", attempts, " passes: ", - length(failed), " species (", - paste(utils::head(failed, 5), collapse = ", "), ")\n", - sep = "") - } - - out <- do.call(rbind, collected) - - if (is.null(out)) { - return(NULL) - } - - out <- out[is.finite(out$coef), ] - attr(out, "failed") <- failed - - out -} - -## 4.3 summarize_bird_draws() ---- - -#' Reduce Bird Draws to the Output Schema -#' -#' @param draws A data frame from read_bird_csv_set(). -#' @param region Character. "north", "south" or "all". -#' @param stage Character. "climate" or "habitat". -#' @param source Character. Which directory it came from. -#' @return A data frame in the output schema, or NULL. -#' -#' @example # Example usage of the function -#' # summarize_bird_draws(draws, "north", "habitat", "archive") -summarize_bird_draws <- function(draws, region, stage, source) { - if (is.null(draws) || nrow(draws) == 0) { - return(NULL) - } - - # Grouped with data.table rather than by pasting species and - # term into one string key. A pasted key needs a separator - # that can appear in neither, and bird terms legitimately - # carry punctuation such as I(Easting^2) and Easting:Northing. - values <- data.table::as.data.table(draws) - - summary_rows <- as.data.frame( - values[ - is.finite(coef), - list( - v2_median = stats::median(coef), - v2_p10 = stats::quantile(coef, 0.1, names = FALSE), - v2_p90 = stats::quantile(coef, 0.9, names = FALSE), - v2_n = .N - ), - by = list(species, term) - ] - ) - - if (nrow(summary_rows) == 0) { - return(NULL) - } - - data.frame( - taxon = "bird", - region = region, - stage = stage, - # Birds model counts, not detection or density. - part = "count", - season = NA_character_, - species = summary_rows$species, - species_v2 = summary_rows$species, - # v2 writes the intercept with parentheses here; normalized - # to the plant form that the comparison expects. - term = sub( - "^[(]Intercept[)]$", "Intercept", summary_rows$term - ), - v2_median = summary_rows$v2_median, - v2_p10 = summary_rows$v2_p10, - v2_p90 = summary_rows$v2_p90, - v2_se = NA_real_, - v2_n = summary_rows$v2_n, - source = source, - stringsAsFactors = FALSE - ) -} - -## 4.4 read_bird_results() ---- - -#' Read and Cache Every Bird Result -#' -#' @param root Character. The BirdModels directory. -#' @param cache_path Character. Where the cached frame lives. -#' @param workers Integer. Parallel readers. -#' @param refresh Logical. TRUE ignores any cache. -#' @return A list of rows and a note. -#' -#' @example # Example usage of the function -#' # read_bird_results(root, cache_path, 12L, FALSE) -read_bird_results <- function(root, cache_path, workers, - refresh = FALSE) { - if (!refresh && file.exists(cache_path)) { - cat("Reading birds from cache:", cache_path, "\n") - - return(list(rows = readRDS(cache_path), note = "from cache")) - } - - if (!dir.exists(root)) { - return(list(rows = NULL, note = "bird root not reachable")) - } - - blocks <- list() - labels <- character(0) - missing <- character(0) - - climate <- bird_source_dir(root, "ClimateModels/Coefficients") - - if (!is.null(climate)) { - cat("Reading bird climate (", climate$label, "):\n", sep = "") - - draws <- read_bird_csv_set(climate$dir, "V1", workers) - - blocks$climate <- summarize_bird_draws( - draws, "all", "climate", paste("bird", climate$label) - ) - - missing <- c(missing, attr(draws, "failed")) - labels <- c(labels, paste0("climate: ", climate$label)) - } - - for (region in c("north", "south")) { - leaf <- file.path("LandcoverModels/Coefficients", region) - landcover <- bird_source_dir(root, leaf) - - if (is.null(landcover)) { - next - } - - cat("Reading bird landcover ", region, " (", - landcover$label, "):\n", sep = "") - - draws <- read_bird_csv_set(landcover$dir, "name", workers) - - blocks[[region]] <- summarize_bird_draws( - draws, region, "habitat", paste("bird", landcover$label) - ) - - missing <- c(missing, attr(draws, "failed")) - labels <- c(labels, paste0(region, ": ", landcover$label)) - } - - kept <- blocks[!vapply(blocks, is.null, logical(1))] - - if (length(kept) == 0) { - return(list(rows = NULL, note = "no bird coefficients found")) - } - - rows <- do.call(rbind, kept) - - note <- paste(labels, collapse = "; ") - - # An incomplete read is never cached. Caching one would make - # the next run silently reuse a short reference, which is the - # worst of both: fast and wrong. - if (length(missing) > 0) { - note <- paste0( - note, "; INCOMPLETE, ", length(missing), - " species unread, not cached" - ) - - cat(" not caching: ", length(missing), - " species could not be read -", sep = "") - - return(list(rows = rows, note = note)) - } - - dir.create( - dirname(cache_path), recursive = TRUE, showWarnings = FALSE - ) - saveRDS(rows, cache_path) - - list(rows = rows, note = note) -} - -## 4.5 read_packaged_birds() ---- - -#' Flatten the Packaged Bird Object -#' -#' Reads the `birds` object that 04_package_bird_coefficients.R -#' writes, which is the v2 packaging step run against the -#' per-draw csvs. -#' -#' Preferred over reading those csvs directly for two reasons. -#' It is one file rather than 31,000 on a mount that throttles. -#' And its coefficients are already on the cross-taxa -#' standardized template, the same WhiteSpruceR and Loamy term -#' space the plant tables use, rather than the raw glm factor -#' names vegcCrop and soilcLoamy, which nothing else shares. -#' -#' The object holds `marginal` for the climate stage and `joint` -#' for the landcover stage, each species by term by draw. The -#' north and south entries carry the same `marginal`, because v2 -#' fits climate once on all data, so it is read from the north -#' entry only and filed as region "all". -#' -#' @param path Character. Path to the packaged Birds RData. -#' @return A data frame in the output schema, or NULL. -#' -#' @example # Example usage of the function -#' # read_packaged_birds("2_pipeline/v2_reference/Birds2024.RData") -read_packaged_birds <- function(path) { - if (!file.exists(path)) { - return(NULL) - } - - env <- new.env() - load(path, envir = env) - - birds <- env$birds - - if (is.null(birds) || is.null(birds$north)) { - return(NULL) - } - - # Summarized across draws, which is the third dimension. - summarize_array <- function(values, region, stage) { - if (is.null(values) || length(dim(values)) != 3) { - return(NULL) - } - - across <- function(f) { - apply(values, c(1, 2), function(x) { - x <- x[is.finite(x)] - - if (length(x) == 0) { - return(NA_real_) - } - - f(x) - }) - } - - medians <- across(stats::median) - p10 <- across(function(x) { - stats::quantile(x, 0.1, names = FALSE) - }) - p90 <- across(function(x) { - stats::quantile(x, 0.9, names = FALSE) - }) - counts <- apply(values, c(1, 2), function(x) { - sum(is.finite(x)) - }) - - species <- rownames(medians) - terms <- colnames(medians) - - out <- data.frame( - taxon = "bird", - region = region, - stage = stage, - # Birds model counts, not detection or density. - part = "count", - season = NA_character_, - species = rep(species, times = length(terms)), - species_v2 = rep(species, times = length(terms)), - term = rep(terms, each = length(species)), - v2_median = as.vector(medians), - v2_p10 = as.vector(p10), - v2_p90 = as.vector(p90), - v2_se = NA_real_, - v2_n = as.vector(counts), - source = basename(path), - stringsAsFactors = FALSE - ) - - out[out$v2_n > 0, ] - } - - rows <- list( - climate = summarize_array( - birds$north$marginal, "all", "climate" - ), - north = summarize_array( - birds$north$joint, "north", "habitat" - ), - south = summarize_array( - birds$south$joint, "south", "habitat" - ) - ) - - kept <- rows[!vapply(rows, is.null, logical(1))] - - if (length(kept) == 0) { - return(NULL) - } - - do.call(rbind, kept) -} - -# 5. Assemble ---- - -blocks <- list() -coverage <- list() - -#' Record What a Source Contributed -#' -#' @param label Character. Taxon or taxon and stage. -#' @param rows A data frame or NULL. -#' @param where Character. The path looked at. -#' @param reason Character. Why it is empty, when it is. -#' @return A one-row data frame. -#' -#' @example # Example usage of the function -#' # note_source("plants", rows, path, NA) -note_source <- function(label, rows, where, reason = NA) { - data.frame( - source_label = label, - reachable = !is.null(rows) && nrow(rows) > 0, - rows = if (is.null(rows)) 0L else nrow(rows), - species = if (is.null(rows)) { - 0L - } else { - length(unique(rows$species)) - }, - looked_in = where, - note = reason, - stringsAsFactors = FALSE - ) -} - -## 5.1 Plants ---- -plant_path <- file.path( - v2_plant_project, "3_output/coefficients/COEFS.RData" -) - -cat("Reading plants from", plant_path, "\n") - -plant_rows <- read_plant_results(plant_path) -blocks$plants <- plant_rows - -coverage$plants <- note_source( - "plants (4 taxa)", plant_rows, plant_path, - if (is.null(plant_rows)) "COEFS.RData not found" else NA -) - -## 5.2 Mammal landcover ---- -for (region in c("north", "south")) { - dir_path <- file.path( - v2_mammal_root, "Landcover", - tools::toTitleCase(region), "Coefficient Tables" - ) - - cat("Reading mammal", region, "from", dir_path, "\n") - - rows <- read_mammal_landcover(dir_path, region) - blocks[[paste0("mammal_", region)]] <- rows - - coverage[[paste0("mammal_", region)]] <- note_source( - paste("mammal landcover", region), rows, dir_path, - if (is.null(rows)) "no coefficient table found" else NA - ) -} - -## 5.3 Mammal climate ---- -climate_dir <- file.path(v2_mammal_root, "Climate/Coefficients") - -cat("Reading mammal climate from", climate_dir, "\n") - -climate_rows <- read_mammal_climate(climate_dir) -blocks$mammal_climate <- climate_rows - -coverage$mammal_climate <- note_source( - "mammal climate", climate_rows, climate_dir, - if (is.null(climate_rows)) "no coefficient files found" else NA -) - -## 5.4 Birds ---- -# The packaged object first. 04_package_bird_coefficients.R -# builds it by running the v2 packaging step against the -# per-draw csvs, which is the only way birds reach the -# cross-taxa standardized term space. Reading the csvs here is -# the fallback for when that has not been run: it works, but it -# yields raw glm factor names that join to nothing else. -bird_packaged <- file.path(out_dir, "Birds2024.RData") -bird_cache <- file.path(out_dir, "cache/bird_v2_results.rds") - -if (!include_birds) { - coverage$bird <- note_source( - "bird", NULL, v2_bird_root, - "skipped; set include_birds to TRUE" - ) -} else if (file.exists(bird_packaged)) { - bird_rows <- read_packaged_birds(bird_packaged) - - blocks$bird <- bird_rows - - coverage$bird <- note_source( - "bird", bird_rows, bird_packaged, - "packaged, standardized term space" - ) -} else { - cat( - "No packaged birds at ", bird_packaged, - "\n falling back to the per-draw csvs; run", - " 04_package_bird_coefficients.R\n for the standardized", - " term space.\n", - sep = "" - ) - - bird_result <- read_bird_results( - v2_bird_root, bird_cache, bird_workers, refresh_birds - ) - - blocks$bird <- bird_result$rows - - coverage$bird <- note_source( - "bird", bird_result$rows, v2_bird_root, - paste("raw csvs;", bird_result$note) - ) -} - -# 6. Write ---- - -kept <- blocks[!vapply(blocks, is.null, logical(1))] -results <- do.call(rbind, kept) - -if (is.null(results) || nrow(results) == 0) { - stop( - "No v2 results were reachable. Check the drive mounts and ", - "the SDM_V2_* environment variables.", - call. = FALSE - ) -} - -rownames(results) <- NULL -results <- results[, result_columns] - -# Ordered so the file is stable between builds and diffable. -results <- results[ - order( - results$taxon, results$region, results$stage, results$part, - results$species, results$term - ), , - drop = FALSE -] - -results_path <- file.path(results_dir, "v2_results.csv") -coverage_path <- file.path( - results_dir, "v2_results_coverage.csv" -) - -fwrite(results, results_path, na = "") -fwrite(do.call(rbind, coverage), coverage_path, na = "") - -cat("\nWrote", results_path, "\n") -cat(nrow(results), "rows,", length(unique(results$species)), - "species,", length(unique(results$taxon)), "taxa\n\n") - -print(as.data.frame(table(results$taxon, results$stage))) - -cat("\nCoverage:\n") -print(do.call(rbind, coverage)[, c( - "source_label", "reachable", "rows", "species" -)]) - -# End of script ---- diff --git a/1_code/_setup/06_harmonize_bird_translation_lookup.R b/1_code/_setup/06_harmonize_bird_translation_lookup.R index 30a9813..16268d7 100644 --- a/1_code/_setup/06_harmonize_bird_translation_lookup.R +++ b/1_code/_setup/06_harmonize_bird_translation_lookup.R @@ -3,33 +3,23 @@ # author: Brendan Casey # created: 2026-09-29 # inputs: +# Read via setup_input() (utils/input_paths.R); the original +# locations are listed, and SDM_* variables can point back. # in the BirdModels project (SDM_V2_BIRD_ROOT): # - Data/lookups/Xn-veg-v2024.Rdata # outputs: # in 0_data/test_dataset/lookup/: # - bird_veg_age_matrix.csv # notes: -# - v2 does not report raw bird landcover coefficients. Its -# 08.PackageCoefficients.R translates them onto the -# standardized cross-taxa habitat types - WhiteSpruceR to -# WhiteSpruce8 and the rest - by multiplying them by the -# `age` matrix in Xn-veg-v2024.Rdata: one row per standardized -# type, one column per raw model term, holding the design -# values of a pure stand of that type, with the age terms -# integrated over each age class. -# - The harness translates bird coefficients the same way, in -# 1_code/modules/birds/standardize.R, so it needs this matrix. -# Copying it here keeps experiments runnable from -# 0_data/test_dataset/ alone. -# - A separate script rather than a section of -# 01_harmonize_model_ready_v2.R, because it reads one small -# file from a different source and 01 takes hours to re-run. -# It writes into test_dataset/lookup/ all the same; re-run it -# whenever 01 rebuilds that folder. -# - The matrix is written as read. Column names are the raw -# glm names v2 used, with interaction terms sorted -# (isCon:wtAge, not wtAge:isCon), which is how v2 matches -# them. +# - v2's 08.PackageCoefficients.R translates raw bird landcover +# coefficients onto the standardized habitat types by the +# `age` matrix (one row per type, one column per raw term: +# a pure stand's design values, age terms integrated over each +# class). modules/birds/standardize.R needs it. +# - Separate from 01 (hours to re-run), but re-run it whenever +# 01 rebuilds lookup/. +# - Written as read: raw glm column names, interactions sorted +# (isCon:wtAge), as v2 matches them. # --- # 1. Setup ---- @@ -40,12 +30,12 @@ library(data.table) # CSV writing (version: 1.16.4) ## 1.2 Resolve paths ---- project_root <- normalizePath(getwd(), winslash = "/") +# The _setup inputs mirrored on ABMI-DATA2 (setup_input()) +source(file.path(project_root, "1_code/_setup/utils/input_paths.R")) + v2_bird_root <- Sys.getenv( "SDM_V2_BIRD_ROOT", - unset = paste0( - "G:/.shortcut-targets-by-id/", - "17Ymt13eHfKvIiuoMl6x-Kn74Z2uVbbzS/BirdModels" - ) + unset = setup_input("bird_models") ) source_path <- file.path( diff --git a/1_code/_setup/07_harmonize_v2_plant_bootstrap_ids.R b/1_code/_setup/07_harmonize_v2_plant_bootstrap_ids.R index 3866d54..66f1fea 100644 --- a/1_code/_setup/07_harmonize_v2_plant_bootstrap_ids.R +++ b/1_code/_setup/07_harmonize_v2_plant_bootstrap_ids.R @@ -3,6 +3,8 @@ # author: Brendan Casey # created: 2026-09-29 # inputs: +# Read via setup_input() (utils/input_paths.R); the original +# locations are listed, and SDM_* variables can point back. # in the v2 plant project (SDM_V2_PROJECT): # - 0_data/bootstrap/lichen-bootstrap-ids.Rdata # - 0_data/bootstrap/mite-bootstrap-ids.Rdata @@ -14,25 +16,17 @@ # in 0_data/test_dataset/lookup/v2_bootstrap_ids//: # - .rds, one per species # notes: -# - v2 draws its plant bootstrap once per species, without a -# seed, and stores the draws. Replaying them is what makes -# parity numerical rather than distributional: the harness -# then fits the rows v2 fitted, draw by draw. The plant specs -# read them with `bootstrap = "v2_ids"`. -# - Each file is v2's matrix for one species, survey units by -# 100 draws, holding SiteYearQu, which is the test dataset's -# survey_unit_id for the plant-group taxa. Draw 1 is the full -# data; the others are drawn with replacement, so ids repeat. -# - Written per species rather than per taxon, because the full -# sets are large - 1.3 GB compressed for vascular plants - and -# a run needs only the species it models. Set SDM_V2_BOOT_TAXA -# and SDM_V2_BOOT_SPECIES (comma-separated) to harmonize a -# subset; unset, every species of every taxon is written. -# - Bryophytes come from the regenerated draws, because the -# published file holds 3 draws. Those are the draws the -# regenerated reference in 2_pipeline/v2_reference/ was fitted -# on, not the ones behind COEFS.RData, so bryophyte parity -# against v2_results.csv stays distributional. +# - v2's stored plant draws, replayed with +# `bootstrap = "v2_ids"` so parity can be numerical. +# - Each file: one species' matrix, units (SiteYearQu = +# survey_unit_id) by 100 draws; draw 1 is the full data. +# - Per species because the full sets are large (1.3 GB +# compressed for vascular plants). SDM_V2_BOOT_TAXA and +# SDM_V2_BOOT_SPECIES (comma-separated) select a subset. +# - Bryophytes use 03's regenerated draws (the published file +# holds 3), which are not those behind COEFS.RData, so +# bryophyte parity against v2_results.csv stays +# distributional. # - Ids not in sites.csv are counted and reported, not dropped. # --- @@ -44,9 +38,12 @@ library(data.table) # reading sites.csv (version: 1.16.4) ## 1.2 Resolve paths ---- project_root <- normalizePath(getwd(), winslash = "/") +# The _setup inputs mirrored on ABMI-DATA2 (setup_input()) +source(file.path(project_root, "1_code/_setup/utils/input_paths.R")) + v2_project <- Sys.getenv( "SDM_V2_PROJECT", - unset = "//ABMI-DATA2/science/sc/ToEmily/VegetationModels" + unset = setup_input("vegetation_models") ) sources <- c( @@ -129,15 +126,17 @@ for (taxon in taxa) { for (species in keep) { draws <- ids[[species]] - # A species whose draws errored in v2 is stored as an error - # object, not a matrix; there is nothing to replay. + # Draws that errored in v2 are stored as error objects if (!is.matrix(draws) || !is.character(draws)) { skipped <- c(skipped, species) next } unmatched <- unmatched + sum(!draws[, 1] %in% sites) - saveRDS(unname(draws), file.path(out_dir, paste0(species, ".rds"))) + saveRDS( + unname(draws), + file.path(out_dir, paste0(species, ".rds")) + ) written <- written + 1L } diff --git a/1_code/_setup/08_harmonize_abmiexplorer_results.R b/1_code/_setup/08_harmonize_abmiexplorer_results.R index 296081f..8ee573e 100644 --- a/1_code/_setup/08_harmonize_abmiexplorer_results.R +++ b/1_code/_setup/08_harmonize_abmiexplorer_results.R @@ -4,6 +4,8 @@ # author: Brendan Casey # created: 2026-10-02 # inputs: +# Read via setup_input() (utils/input_paths.R); the original +# locations are listed, and SDM_* variables can point back. # - data/species-coefs.RData from the ABbiodiversity/ABMIexploreR # GitHub repository, at the commit pinned in section 1.2 # - Data/lookups/birdlist.csv on the BirdModels drive, for the @@ -16,24 +18,17 @@ # - 0_data/v2_results/abmiexplorer/v2_results.csv # - 0_data/v2_results/abmiexplorer/v2_results_coverage.csv # notes: -# - An alternative to 05_harmonize_v2_results.R. Same output -# schema, so 02_compare_to_v2.R reads either; run.R's -# `v2_reference` chooses which. This one needs no network -# drive for any taxon but birds, and those only for names. -# -# - ABMIexploreR is ABMI's published distribution of the v2 -# coefficients: one `species.coefs` list holding, per taxon, -# species x term x draw arrays for Vegetation (north), Soil -# (south) and Climate (province-wide), on the link scale. -# -# - Pinned to a commit, not to HEAD. The data file changes -# between commits: on 2026-07-27 the mammal models added in -# July 2025 (an earlier fit, presence in the north only) were -# replaced by the 2024 models the harness reproduces. An -# unpinned read would move the reference under a finished -# run. Only the data file is -# downloaded; installing the package would pull terra and -# friends for nothing this script uses. +# - Builds the v2 parity reference exp_000 scores against. +# Only birds need a network drive, for names. "05" below is +# the retired by-hand build from the v2 network outputs, which +# this was verified against. +# - ABMIexploreR holds species x term x draw arrays per taxon +# for Vegetation (north), Soil (south) and Climate, on the +# link scale. +# - Pinned to a commit: the data file changes between commits +# (on 2026-07-27 the mammal models were replaced by the 2024 +# models the harness reproduces). Only the data file is +# downloaded, not the package and its dependencies. # # - Checked against 05's reference on 2026-10-02, at the pinned # commit: @@ -59,14 +54,11 @@ # - Amphibians are in the package and have no module yet. # Their rows are written; nothing joins them. # -# - The package relabels some terms against the names the -# harness writes, and mammals on the cross-taxa template -# rather than their native terms. Where the values were -# verified identical, an alias row is added under the -# harness's name and the package's row is kept; see -# term_aliases(). Bird TreedFen +# - The package relabels some terms, and puts mammals on the +# cross-taxa template. Verified-identical terms get an alias +# row under the harness's name (term_aliases()). Bird TreedFen # is one coefficient in the package and nine age classes in -# v2 (r ~ 0.99 with each), so it is not aliased. +# v2 (r ~ 0.99), so it is not aliased. # # - The package pads every species to a full term template # with exact zeros. A cell whose every draw is exactly zero @@ -95,14 +87,15 @@ explorer_file <- "data/species-coefs.RData" ## 1.3 Resolve the other locations ---- project_root <- normalizePath(getwd(), winslash = "/") +# The _setup inputs mirrored on ABMI-DATA2 (setup_input()) +source(file.path(project_root, "1_code/_setup/utils/input_paths.R")) + # The same lookup v2's 08.PackageCoefficients.R uses to turn # bird codes into the names the package carries. bird_lookup_path <- Sys.getenv( "SDM_V2_BIRD_LOOKUP", - unset = paste0( - "G:/.shortcut-targets-by-id/", - "17Ymt13eHfKvIiuoMl6x-Kn74Z2uVbbzS/BirdModels/", - "Data/lookups/birdlist.csv" + unset = setup_input( + "bird_models", "Data/lookups/birdlist.csv" ) ) @@ -144,9 +137,8 @@ if (!file.exists(coefs_path)) { cat("Downloading", url, "\n") - # Written to a temporary name first, so an interrupted - # download never leaves a truncated file where a good one is - # expected. + # Via a temporary name, so an interrupted download leaves no + # truncated file partial <- paste0(coefs_path, ".part") old_timeout <- options(timeout = max(600, getOption("timeout"))) status <- utils::download.file(url, partial, mode = "wb") @@ -384,9 +376,8 @@ flatten_array <- function(values, slug, model, hurdle = NA) { padding <- counts > 0 & is.na(hurdle) & rowSums(draws != 0, na.rm = TRUE) == 0 - # v2 fits mammals once; the package repeats that fit in every - # draw. A repeated fit is one value, not a distribution, so it - # is written as v2_n 1 and scored on iteration 1. + # The package repeats v2's single mammal fit in every draw: + # written as v2_n 1, scored on iteration 1 single_fit <- slug == "mammal" && all(apply(draws, 1, function(x) { length(unique(x[!is.na(x)])) <= 1 @@ -496,8 +487,7 @@ for (taxon in names(species_coefs)) { rows <- add_aliases(rows, slug) blocks[[label]] <- rows - # A bird without a code keeps its package name and cannot - # join, so the count is reported rather than discovered. + # Birds without a code cannot join; report the count unkeyed <- if (slug == "bird" && !is.null(rows)) { sum(!unique(rows$species_v2) %in% names(bird_codes)) } else { diff --git a/1_code/_setup/09_harmonize_lookups.R b/1_code/_setup/09_harmonize_lookups.R new file mode 100644 index 0000000..05baa77 --- /dev/null +++ b/1_code/_setup/09_harmonize_lookups.R @@ -0,0 +1,368 @@ +# --- +# title: Harmonize the Test Dataset Lookups Across Taxa +# author: Brendan Casey +# created: 2026-10-03 +# inputs: +# in 0_data/test_dataset/lookup/ (all written by 01 and 06): +# - modelled_species.csv, mammal_modelled_species.csv, +# bird_modelled_species.csv +# - bird_factor_levels.csv +# - bird_veg_age_matrix.csv +# - veg_, soil_, mammal_north_ and mammal_south_ +# prediction_matrix.csv +# outputs: +# in 0_data/test_dataset/lookup/: +# - species_queue.csv +# - factor_levels.csv +# - dataset_manifest.csv +# - bird_north_prediction_matrix.csv +# - bird_south_prediction_matrix.csv +# notes: +# - Writes one cross-taxon form of each lookup, so no method or +# new taxon needs taxon-specific file logic. Reads only +# 0_data/test_dataset/; run after 01 and 06. +# - species_queue.csv: `order` is the source list's order; +# `tier` separates mammals' full habitat models (20+ +# detections) from use-availability models (3+). +# - dataset_manifest.csv: the only place the harness learns file +# locations, covariate keys, grids and stored-draw layouts. +# - Bird grids: v2 translates bird coefficients with a model +# matrix, not a grid. The north grid rebuilds that matrix in +# covariate space and is checked to reproduce it exactly; the +# south grid is one row per soil class. Terms the matrix lacks +# are zero or the factor reference level, as v2 assumes. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # CSV reading and writing (version: 1.16.4) + +## 1.2 Resolve paths ---- +project_root <- normalizePath(getwd(), winslash = "/") +lookup_dir <- file.path( + project_root, "0_data", "test_dataset", "lookup" +) + +if (!dir.exists(lookup_dir)) { + stop( + "Run this from the repository root; no ", lookup_dir, + call. = FALSE + ) +} + +read_lookup <- function(name) { + path <- file.path(lookup_dir, name) + + if (!file.exists(path)) { + stop( + "Missing ", path, ". Run 01_harmonize_model_ready_v2.R ", + "(and 06 for the bird matrix) first.", + call. = FALSE + ) + } + + as.data.frame(fread(path)) +} + +write_lookup <- function(frame, name) { + fwrite(frame, file.path(lookup_dir, name), na = "") + cat("Wrote ", name, " (", nrow(frame), " rows)\n", sep = "") +} + +# 2. Species queue ---- +# Three source schemas become one + +## 2.1 queue_rows() ---- + +#' Rows of the Unified Species Queue +#' +#' @param species Character vector. +#' @param taxon,region,order,tier,season Values recycled to the +#' number of species. +#' @param source_name Character vector. The name the source +#' pipeline uses, where it differs from the dataset's column +#' name: mammals are "Black BearSummer" in v2's files and +#' "BlackBear_Summer" here. +#' @return A data frame in the queue schema. +#' +#' @example # Example usage of the function +#' # queue_rows(c("ALFL", "AMRO"), "bird", "north", 1:2) +queue_rows <- function( + species, taxon, region, order, tier = "modelled", + season = NA_character_, source_name = species +) { + if (length(species) == 0) { + return(NULL) + } + + data.frame( + taxon = as.character(taxon), + region = as.character(region), + season = as.character(season), + tier = as.character(tier), + species = as.character(species), + order = as.integer(order), + source_name = as.character(source_name), + stringsAsFactors = FALSE + ) +} + +## 2.2 Plant groups ---- +# One row per species, with membership of the north (veg) and +# south (soil) model sets as flags; each flag becomes a region. +plants <- read_lookup("modelled_species.csv") +north_plants <- plants[plants$in_veg_models, ] +south_plants <- plants[plants$in_soil_models, ] + +## 2.3 Mammals ---- +# Region and season are already columns; the two occurrence +# thresholds become tiers. +mammals <- read_lookup("mammal_modelled_species.csv") +full_mammals <- mammals[mammals$in_models, ] +ua_mammals <- mammals[mammals$in_ua_models, ] + +## 2.4 Birds ---- +# One row per species and region; the source order is the order +# within each region. +birds <- read_lookup("bird_modelled_species.csv") + +species_queue <- rbind( + queue_rows( + north_plants$species, north_plants$taxon, "north", + north_plants$veg_order + ), + queue_rows( + south_plants$species, south_plants$taxon, "south", + south_plants$soil_order + ), + queue_rows( + full_mammals$species, "mammal", full_mammals$region, + full_mammals$model_order, tier = "modelled", + season = full_mammals$season, + source_name = full_mammals$species_season + ), + queue_rows( + ua_mammals$species, "mammal", ua_mammals$region, + ua_mammals$ua_order, tier = "ua", season = ua_mammals$season, + source_name = ua_mammals$species_season + ), + queue_rows( + birds$species, "bird", birds$region, + stats::ave(seq_len(nrow(birds)), birds$region, FUN = seq_along) + ) +) + +## 2.5 Check and write ---- +duplicated_rows <- duplicated( + species_queue[, c("taxon", "region", "season", "tier", "species")] +) + +if (any(duplicated_rows)) { + stop( + sum(duplicated_rows), " species appear twice in one queue.", + call. = FALSE + ) +} + +write_lookup(species_queue, "species_queue.csv") + +# 3. Factor levels ---- +factor_files <- list.files( + lookup_dir, pattern = "_factor_levels\\.csv$" +) + +factor_levels <- do.call(rbind, lapply(factor_files, function(file) { + levels_table <- read_lookup(file) + data.frame( + taxon = sub("_factor_levels\\.csv$", "", file), + levels_table[, c("column", "level_order", "level")], + stringsAsFactors = FALSE + ) +})) + +write_lookup(factor_levels, "factor_levels.csv") + +reference_level <- function(taxon, column) { + rows <- factor_levels[ + factor_levels$taxon == taxon & factor_levels$column == column, + ] + rows$level[which.min(rows$level_order)] +} + +levels_of <- function(taxon, column) { + rows <- factor_levels[ + factor_levels$taxon == taxon & factor_levels$column == column, + ] + rows$level[order(rows$level_order)] +} + +# 4. Bird prediction grids ---- + +## 4.1 North, rebuilt from v2's translation matrix ---- +age_table <- read_lookup("bird_veg_age_matrix.csv") +age <- as.matrix(age_table[, -1]) +rownames(age) <- age_table$type + +vegc_levels <- levels_of("bird", "vegc") +dummies <- grep("^vegc", colnames(age), value = TRUE) + +# Each row's class is the dummy set to 1, or the reference level +# where none is +vegc <- vapply(seq_len(nrow(age)), function(i) { + hit <- dummies[age[i, dummies] == 1] + + if (length(hit) > 1) { + stop("Matrix row ", rownames(age)[i], " sets two classes.", + call. = FALSE) + } + + if (length(hit) == 0) vegc_levels[1] else sub("^vegc", "", hit) +}, character(1)) + +# Type flags enter only via age interactions, so are recovered as +# interaction / age where age is non-zero; else left at zero (no +# effect on predictions) +flags <- c("isCon", "isUpCon", "isBogFen", "isMix", "isPine", + "isWSpruce") +age_terms <- c("wtAge", "wtAge2", "wtAge05") + +flag_values <- sapply(flags, function(flag) { + vapply(seq_len(nrow(age)), function(i) { + for (term in age_terms) { + if (age[i, term] != 0) { + return(age[i, paste0(flag, ":", term)] / age[i, term]) + } + } + 0 + }, numeric(1)) +}) + +bird_north <- data.frame( + VegType = rownames(age), + vegc = vegc, + age[, c("wtAge", "wtAge2", "wtAge05", "fcc2"), drop = FALSE], + flag_values, + # Absent from a pure stand of a habitat type + road = 0, mWell = 0, mSoft = 0, mEnSft = 0, mTrSft = 0, + mSeism = 0, pWater_KM = 0, pWater2_KM = 0, + method = reference_level("bird", "method"), + stringsAsFactors = FALSE, + check.names = FALSE, + row.names = NULL +) + +### 4.1.1 Check the rebuild ---- +# Must reproduce v2's matrix column for column +rebuilt <- stats::model.matrix( + ~ vegc + wtAge + wtAge2 + wtAge05 + fcc2 + + (wtAge + wtAge2 + wtAge05):(isCon + isUpCon + isBogFen + + isMix + isPine + isWSpruce), + transform( + bird_north, vegc = factor(vegc, levels = vegc_levels) + ) +) +colnames(rebuilt) <- vapply( + strsplit(sub("^\\(Intercept\\)$", "Intercept", colnames(rebuilt)), + ":", fixed = TRUE), + function(pieces) paste(sort(pieces), collapse = ":"), + character(1) +) + +difference <- max(abs( + rebuilt[, colnames(age), drop = FALSE] - age +)) + +if (difference > 1e-12) { + stop( + "The rebuilt bird north grid does not reproduce ", + "bird_veg_age_matrix.csv (largest difference ", difference, + "). Not written.", + call. = FALSE + ) +} + +cat("Bird north grid reproduces v2's matrix (max diff ", + format(difference), ")\n", sep = "") + +write_lookup(bird_north, "bird_north_prediction_matrix.csv") + +## 4.2 South, one row per soil class ---- +soil_levels <- levels_of("bird", "soilc") + +bird_south <- data.frame( + VegType = soil_levels, + soilc = soil_levels, + paspen = 0, + road = 0, mWell = 0, mSoft = 0, pWater_KM = 0, pWater2_KM = 0, + method = reference_level("bird", "method"), + stringsAsFactors = FALSE +) + +write_lookup(bird_south, "bird_south_prediction_matrix.csv") + +# 5. Dataset manifest ---- +# A new taxon is added here and needs no harness change +plant_taxa <- c("vascular_plant", "bryophyte", "lichen", "mite") + +manifest <- rbind( + do.call(rbind, lapply(plant_taxa, function(taxon) { + data.frame( + taxon = taxon, + region = c("north", "south"), + response_file = paste0(taxon, ".csv"), + offset_file = NA_character_, + covariate_key = taxon, + grid = c("veg", "soil"), + bootstrap_ids = file.path("lookup", "v2_bootstrap_ids", taxon), + bootstrap_layout = "per_species_rds", + stringsAsFactors = FALSE + ) + })), + data.frame( + taxon = "mammal", + region = c("north", "south"), + response_file = "mammal.csv", + offset_file = NA_character_, + covariate_key = c("mammal_north", "mammal_south"), + grid = c("mammal_north", "mammal_south"), + bootstrap_ids = NA_character_, + bootstrap_layout = NA_character_, + stringsAsFactors = FALSE + ), + data.frame( + taxon = "bird", + region = c("north", "south"), + response_file = "bird.csv", + offset_file = "bird_offsets.csv", + covariate_key = "bird", + grid = c("bird_north", "bird_south"), + bootstrap_ids = file.path("lookup", "bird_bootstrap_ids.csv"), + bootstrap_layout = "per_taxon_csv", + stringsAsFactors = FALSE + ) +) + +## 5.1 Check every named file exists ---- +dataset_dir <- dirname(lookup_dir) +named <- c( + file.path(dataset_dir, manifest$response_file), + file.path(dataset_dir, stats::na.omit(manifest$offset_file)), + file.path(lookup_dir, + paste0(manifest$grid, "_prediction_matrix.csv")) +) +absent <- unique(named[!file.exists(named)]) + +if (length(absent) > 0) { + stop( + "The manifest names files that do not exist:\n ", + paste(absent, collapse = "\n "), + call. = FALSE + ) +} + +write_lookup(manifest, "dataset_manifest.csv") + +cat("\nLookups harmonized.\n") + +# End of script ---- diff --git a/1_code/_setup/10_publish_datasets.R b/1_code/_setup/10_publish_datasets.R new file mode 100644 index 0000000..30396f0 --- /dev/null +++ b/1_code/_setup/10_publish_datasets.R @@ -0,0 +1,206 @@ +# --- +# title: Publish the Test Dataset and v2 Results to ABMI-DATA2 +# author: Brendan Casey +# created: 2026-10-05 +# inputs: +# - 0_data/test_dataset/, from _setup/01-09, checked by 02 +# - 0_data/v2_results/abmiexplorer/, from _setup/08 +# outputs: +# in //ABMI-DATA2/science/sdmMethodsDev/0_data/ (SDM_SHARE_ROOT +# overrides it), for each dataset: +# - //, a copy of the local folder +# - //checksums.csv: path, bytes, md5 +# - //publish_record.md: when, by whom, from +# which commit +# notes: +# - Runs read these copies (harness/data_source.R). A version +# is never overwritten: publish a new one and move the pin. +# - Files are copied to .partial/, re-hashed, then +# renamed, so readers never see a half copy. +# - SDM_DATASET_VERSION sets the version (default: today's +# date, YYYY-MM-DD). SDM_PUBLISH picks the datasets, +# comma-separated (default: test_dataset,v2_results). +# - Run 02_validate_test_dataset.R first (not run here: it +# re-reads every source). +# - Run from the repository root. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # checksums writing (version: 1.16.4) + +## 1.2 Resolve paths and settings ---- +project_root <- normalizePath(getwd(), winslash = "/") +resolver <- file.path(project_root, "1_code/harness/data_source.R") + +if (!file.exists(resolver)) { + stop("Run this from the repository root.", call. = FALSE) +} + +# For the share root and each dataset's marker file +source(resolver) + +share_root <- Sys.getenv( + "SDM_SHARE_ROOT", + unset = sdm_published$root +) + +version <- Sys.getenv( + "SDM_DATASET_VERSION", + unset = format(Sys.Date()) +) + +datasets <- trimws(strsplit( + Sys.getenv("SDM_PUBLISH", unset = "test_dataset,v2_results"), + "," +)[[1]]) + +if (!grepl("^[0-9]{4}-[0-9]{2}-[0-9]{2}$", version)) { + stop( + "SDM_DATASET_VERSION must be a date, YYYY-MM-DD.", + call. = FALSE + ) +} + +unknown <- setdiff(datasets, names(sdm_published$versions)) + +if (length(unknown) > 0) { + stop("Unknown dataset(s): ", paste(unknown, collapse = ", "), + call. = FALSE) +} + +## 1.3 The commit the data was built with ---- +git_commit <- tryCatch( + system2("git", c("rev-parse", "HEAD"), stdout = TRUE), + error = function(e) "unknown" +) + +git_dirty <- tryCatch( + length(system2("git", c("status", "--porcelain"), + stdout = TRUE)) > 0, + error = function(e) NA +) + +# 2. publish_dataset() ---- + +#' Copy One Local Dataset to the Share as a New Version +#' +#' @param name Character. A dataset under 0_data/, such as +#' "test_dataset". +#' @param version Character. The version folder to create. +#' @return Character. The published folder, invisibly. +#' +#' @example # Example usage of the function +#' # publish_dataset("v2_results", "2026-10-05") +publish_dataset <- function(name, version) { + # Step 1: The local copy, complete enough to be the dataset + local_dir <- file.path(project_root, "0_data", name) + marker <- sdm_published$marker[[name]] + + if (!file.exists(file.path(local_dir, marker))) { + stop("No ", marker, " in ", local_dir, call. = FALSE) + } + + # Step 2: A new version, never an existing or half-made one + target <- file.path(share_root, name, version) + partial <- paste0(target, ".partial") + + if (dir.exists(target)) { + stop( + "Already published, and never overwritten:\n ", target, + "\nSet SDM_DATASET_VERSION to publish a new version.", + call. = FALSE + ) + } + + if (dir.exists(partial)) { + stop( + "A previous publish did not finish:\n ", partial, + "\nDelete it, then run this again.", + call. = FALSE + ) + } + + # Step 3: The files and their md5s + rel <- list.files(local_dir, recursive = TRUE, all.files = TRUE) + rel <- rel[basename(rel) != ".gitkeep"] + sums <- data.table( + path = rel, + bytes = file.size(file.path(local_dir, rel)), + md5 = unname(tools::md5sum(file.path(local_dir, rel))) + ) + + cat( + name, ": ", nrow(sums), " files, ", + round(sum(sums$bytes) / 1e9, 2), " GB -> ", target, "\n", + sep = "" + ) + + # Step 4: Copy into the partial folder + for (i in seq_len(nrow(sums))) { + to <- file.path(partial, sums$path[i]) + dir.create(dirname(to), recursive = TRUE, showWarnings = FALSE) + + if (!file.copy(file.path(local_dir, sums$path[i]), to, + copy.date = TRUE)) { + stop("Copy failed: ", sums$path[i], call. = FALSE) + } + + cat(sprintf(" [%d/%d] %s\n", i, nrow(sums), sums$path[i])) + } + + # Step 5: Re-hash the share copy before it is made visible + copied <- unname(tools::md5sum(file.path(partial, sums$path))) + bad <- is.na(copied) | copied != sums$md5 + + if (any(bad)) { + stop( + "md5 mismatch on the share; ", partial, " left for ", + "inspection:\n ", paste(sums$path[bad], collapse = "\n "), + call. = FALSE + ) + } + + # Step 6: Checksums and the record, then make it visible + fwrite(sums, file.path(partial, "checksums.csv")) + + writeLines(c( + paste0("# ", name, " ", version), + "", + paste0("- Published: ", format(Sys.time(), "%Y-%m-%d %H:%M")), + paste0("- By: ", Sys.info()[["user"]]), + paste0("- From: ", local_dir), + paste0( + "- Commit: ", git_commit, + if (isTRUE(git_dirty)) " (with uncommitted changes)" else "" + ), + paste0("- Files: ", nrow(sums), ", ", + round(sum(sums$bytes) / 1e6, 1), " MB"), + paste0("- R: ", R.version.string), + "", + "Built by 1_code/_setup/ in the sdmMethodsDev repository and", + "published by 1_code/_setup/10_publish_datasets.R. Never", + "edited after publishing; checksums.csv lists every file." + ), file.path(partial, "publish_record.md")) + + if (!file.rename(partial, target)) { + stop("Could not rename ", partial, " to ", target, call. = FALSE) + } + + invisible(target) +} + +# 3. Publish ---- +published <- vapply( + datasets, publish_dataset, character(1), version = version +) + +cat( + "\nPublished:\n ", paste(published, collapse = "\n "), + "\n\nTo use them, set sdm_published$versions in ", + "1_code/harness/data_source.R to \"", version, "\".\n", + sep = "" +) + +# End of script ---- diff --git a/1_code/_setup/README.md b/1_code/_setup/README.md new file mode 100644 index 0000000..85bcd14 --- /dev/null +++ b/1_code/_setup/README.md @@ -0,0 +1,89 @@ +# _setup + +Builds the frozen test dataset and the v2 reference in `0_data/`, then +publishes them to ABMI-DATA2. Run by hand, by maintainers, once per +source snapshot. No experiment runs these scripts: experiments read the +published copies (see `harness/data_source.R`). + +The scripts read their external inputs from a mirror on +`//ABMI-DATA2/science/sdmMethodsDev/0_data/setup_inputs/`, through +`utils/input_paths.R`, so anyone with access to that share can +rebuild. + +## Run + +From the repository root: + +```r +source("1_code/_setup/run.R") +``` + +`run.R` runs the steps in dependency order, with on/off switches in +section 1.2, and logs timings to `2_pipeline/_setup/run_log.csv`. `00`, +`03` and `10` are off by default. Run order: `01`, `06`, `09`, `02`, +`03`, `07`, `08`, then `10`. There is no `04` or `05`; they were +retired once `08` replaced them. + +## Scripts + +| Script | Reads | Writes | +| --- | --- | --- | +| `00_mirror_setup_inputs.R` | Every external input, at its original location | `setup_inputs/` on ABMI-DATA2, with `inputs_manifest.csv`. Run once; never overwrites | +| `01_harmonize_model_ready_v2.R` | The `model_ready_v2` snapshot, mammal camera climate, WildTrax species lookup, v2 prediction matrices, bird `Stratified.Rdata` | `0_data/test_dataset/`: responses, `sites.csv`, `covariates.csv`, `bird_offsets.csv`, and most of `lookup/` | +| `02_validate_test_dataset.R` | The test dataset and the same sources | Nothing; prints a pass / fail tally. Run after `01`, `06` and `09` | +| `03_rerun_bryophyte_v2_reference.R` | v2 plant project bryophyte data, frozen v2 functions in `0_data/v2_scripts/plants/` | `2_pipeline/v2_reference/`: `bryophyte-species-models.Rdata`, `bryophyte-bootstrap-ids.Rdata`. Takes hours | +| `06_harmonize_bird_translation_lookup.R` | v2's bird `Xn-veg-v2024.Rdata` | `lookup/bird_veg_age_matrix.csv`. Re-run whenever `01` rebuilds `lookup/` | +| `07_harmonize_v2_plant_bootstrap_ids.R` | v2's stored plant draws; `03`'s bryophyte draws | `lookup/v2_bootstrap_ids//.rds`, for `plant_bootstrap = "v2_ids"` | +| `08_harmonize_abmiexplorer_results.R` | ABMIexploreR's `species-coefs.RData` at a pinned commit; bird `birdlist.csv` | `0_data/external/abmiexplorer//`; `0_data/v2_results/abmiexplorer/v2_results.csv` and `v2_results_coverage.csv` | +| `09_harmonize_lookups.R` | `0_data/test_dataset/lookup/` only; offline, seconds | `species_queue.csv`, `factor_levels.csv`, `dataset_manifest.csv`, bird prediction grids | +| `10_publish_datasets.R` | `0_data/test_dataset/`, `0_data/v2_results/` | A new `/` of each on ABMI-DATA2, with `checksums.csv` and `publish_record.md` | +| `run.R` | The scripts above | Runs them in order | +| `utils/input_paths.R` | | `setup_input()`: paths inside the input mirror | + +After `10`, move the pin in `1_code/harness/data_source.R`. A +published version is never overwritten. + +## The test dataset + +`0_data/test_dataset/` holds the data the v2 models were fitted on. It +does not change between experiments. + +| File | Holds | +| --- | --- | +| `sites.csv` | One row per survey unit: identity, design, location, and an `in_` flag per taxon | +| `.csv` | `survey_unit_id` plus one column per species: detections (plants), densities (mammals), counts (birds) | +| `bird_offsets.csv` | QPAD log offsets per survey unit and bird species | +| `covariates.csv` | Every covariate for every taxon, keyed on `survey_unit_id` and a taxon covariate key | +| `lookup/dataset_manifest.csv` | Per taxon and region: files, covariate key, grid and stored draws. The harness reads file locations only from here | +| `lookup/species_queue.csv` | Every taxon's species queue in one schema | +| `lookup/covariate_columns.csv` | Each covariate's block and v2 spelling, per taxon | +| `lookup/factor_levels.csv` | Source order of categorical levels, so contrasts match v2 | +| `lookup/*_prediction_matrix.csv` | Habitat grids: one row per habitat type, a pure stand | +| `lookup/bird_bootstrap_ids.csv`, `lookup/v2_bootstrap_ids/` | v2's stored draws: birds, and plants per species | +| `lookup/mammal_climate_predictions.csv` | v2's precomputed mammal climate prediction | +| `lookup/bird_veg_age_matrix.csv` | v2's bird landcover translation matrix | + +Columns in more than one block are suffixed (`HardLin_veg`, +`HardLin_soil`). Ten derived columns (`TD`, `MAT2`, `Easting2` and +others) are computed on load, not stored. Taxon-specific details are in +[`docs/taxon_quirks.md`](../../docs/taxon_quirks.md). + +## Environment variables + +Each overrides a default path. + +| Variable | Read by | Points at | +| --- | --- | --- | +| `SDM_SETUP_INPUT_ROOT` | `00`–`08` | The input mirror | +| `SDM_SNAPSHOT_V2` | `01`, `02` | The `model_ready_v2` snapshot folder | +| `SDM_MAMMAL_CAMERA_CLIMATE` | `01` | `abmi-camera-climate_2023.Rdata` | +| `SDM_MAMMAL_CLIMATE_PRED` | `01`, `02` | `All Species Climate Predictions.csv` | +| `SDM_V2_LOOKUP` | `01` | The v2 folder holding the plant prediction matrices | +| `SDM_WT_SPECIES` | `01` | `WildTrax Species Strings.RData` | +| `SDM_BIRD_DATA` | `01`, `02` | The bird `Stratified.Rdata` v2 was fitted on | +| `SDM_V2_PROJECT` | `03`, `07` | The v2 plant project (`VegetationModels`) | +| `SDM_V2_BIRD_ROOT` | `06` | The v2 bird project (`BirdModels`) | +| `SDM_V2_BOOT_TAXA`, `SDM_V2_BOOT_SPECIES` | `07` | Comma-separated subsets to copy | +| `SDM_ABMIEXPLORER_SHA` | `08` | The ABMIexploreR commit to download | +| `SDM_V2_BIRD_LOOKUP` | `08` | The v2 bird `birdlist.csv` | +| `SDM_SHARE_ROOT`, `SDM_DATASET_VERSION`, `SDM_PUBLISH` | `10` | Publish root, version folder (default today, `YYYY-MM-DD`), datasets | diff --git a/1_code/_setup/run.R b/1_code/_setup/run.R new file mode 100644 index 0000000..519de94 --- /dev/null +++ b/1_code/_setup/run.R @@ -0,0 +1,165 @@ +# --- +# title: Rebuild the Test Dataset and v2 Reference +# author: Brendan Casey +# created: 2026-10-05 +# inputs: +# - the _setup/ scripts in this folder, and whatever each reads +# (see their headers); most read the input mirror on +# ABMI-DATA2 through utils/input_paths.R +# outputs: +# - 0_data/test_dataset/, from 01, 06 and 09 +# - 0_data/test_dataset/lookup/v2_bootstrap_ids/, from 07 +# - 0_data/v2_results/abmiexplorer/, from 08 +# - 2_pipeline/v2_reference/, from 03 +# - a new published version on ABMI-DATA2, from 10, only if +# switched on +# - 2_pipeline/_setup/run_log.csv: per step, start, end, +# minutes and whether it finished +# notes: +# - Runs the _setup/ scripts in their dependency order, for a +# manual rebuild. Switch steps on or off in section 1.2; the +# defaults rebuild everything local and publish nothing. +# - Off by default: +# - 00 mirrors the inputs onto ABMI-DATA2. Run once, by +# someone who can reach the original drives. +# - 03 re-runs the v2 bryophyte models: hours, and its +# output only changes if its sources do. +# - 10 publishes a new version. Run it deliberately, after +# checking 02's tally, then move the pin in +# 1_code/harness/data_source.R. +# - Each script runs in its own environment; an error stops +# the rebuild and is logged. +# - 02 prints a pass / fail tally rather than stopping; read it +# before using or publishing. +# - Section 3 prints how to point experiments at this rebuild. +# - Run from the repository root. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only; each script loads its own. + +## 1.2 Choose the steps ---- +# In run order: 09 reads 01's and 06's lookups, 02 checks the +# finished dataset, 07 reads 03's bryophyte draws. +steps <- c( + "00_mirror_setup_inputs.R" = FALSE, + "01_harmonize_model_ready_v2.R" = TRUE, + "06_harmonize_bird_translation_lookup.R" = TRUE, + "09_harmonize_lookups.R" = TRUE, + "02_validate_test_dataset.R" = TRUE, + "03_rerun_bryophyte_v2_reference.R" = FALSE, + "07_harmonize_v2_plant_bootstrap_ids.R" = TRUE, + "08_harmonize_abmiexplorer_results.R" = TRUE, + "10_publish_datasets.R" = FALSE +) + +## 1.3 Check the paths ---- +setup_dir <- "1_code/_setup" + +if (!dir.exists(setup_dir)) { + stop( + "Run from the repository root; ", + setup_dir, + " not found.", + call. = FALSE + ) +} + +found <- file.exists(file.path(setup_dir, names(steps))) +missing <- names(steps)[!found] + +if (length(missing) > 0) { + stop( + "Not found in ", + setup_dir, + ":\n ", + paste(missing, collapse = "\n "), + call. = FALSE + ) +} + +log_dir <- "2_pipeline/_setup" +dir.create(log_dir, recursive = TRUE, showWarnings = FALSE) +log_path <- file.path(log_dir, "run_log.csv") + +# 2. Run the steps ---- + +## 2.1 run_step() ---- + +#' Source One _setup Script in an Environment of Its Own +#' +#' @param script Character. File name inside `setup_dir`. +#' @param setup_dir Character. The _setup folder. +#' @param log_path Character. The run log to append to. +#' @return TRUE, invisibly; stops if the script errors. +#' +#' @example # Example usage of the function +#' # run_step("09_harmonize_lookups.R", setup_dir, log_path) +run_step <- function(script, setup_dir, log_path) { + # Step 1: Announce and time the step + message("\n== ", script, " (", format(Sys.time(), "%H:%M"), ")") + started <- Sys.time() + + # Step 2: Source it in isolation, keeping any error + error <- tryCatch( + { + source( + file.path(setup_dir, script), + local = new.env(parent = globalenv()) + ) + NULL + }, + error = function(e) e + ) + + # Step 3: Log the outcome before stopping on an error + ended <- Sys.time() + row <- data.frame( + script = script, + started = format(started, "%Y-%m-%d %H:%M:%S"), + ended = format(ended, "%Y-%m-%d %H:%M:%S"), + minutes = round( + as.numeric(difftime(ended, started, units = "mins")), + 1 + ), + status = if (is.null(error)) "done" else "failed" + ) + + write.table( + row, + log_path, + sep = ",", + row.names = FALSE, + col.names = !file.exists(log_path), + append = file.exists(log_path) + ) + + if (!is.null(error)) { + stop(script, " failed: ", conditionMessage(error), call. = FALSE) + } + + message(" done in ", row$minutes, " min") + invisible(TRUE) +} + +## 2.2 Run the chosen steps in order ---- +to_run <- names(steps)[steps] +message("Rebuilding: ", paste(to_run, collapse = ", ")) + +for (script in to_run) { + run_step(script, setup_dir, log_path) +} + +# 3. Point experiments at the rebuild ---- +# The published copies are what experiments read by default. +message( + "\nRebuild finished. To run an experiment against it:\n", + " Sys.setenv(SDM_TEST_DATASET = \"0_data/test_dataset\")\n", + " Sys.setenv(SDM_V2_RESULTS = \"0_data/v2_results\")\n", + "Run log: ", + log_path +) + +# End of script ---- diff --git a/1_code/_setup/utils/input_paths.R b/1_code/_setup/utils/input_paths.R new file mode 100644 index 0000000..8d42d0e --- /dev/null +++ b/1_code/_setup/utils/input_paths.R @@ -0,0 +1,39 @@ +# --- +# title: Locate the _setup Inputs on ABMI-DATA2 +# author: Brendan Casey +# created: 2026-10-05 +# inputs: none +# outputs: none; defines setup_input_root and setup_input() +# notes: +# - Every external file _setup/ reads is mirrored, unchanged, +# under //ABMI-DATA2/science/sdmMethodsDev/0_data/ +# setup_inputs/ by 00_mirror_setup_inputs.R, so _setup/ can be +# run by anyone with access to that share. The originals stay +# where they were; 00_mirror_setup_inputs.R lists them and +# setup_inputs/inputs_manifest.csv records each copy. +# - The scripts' own SDM_* environment variables still override +# each path, so a source can be read from its original +# location. SDM_SETUP_INPUT_ROOT moves the whole mirror. +# --- + +# 1. The mirror root ---- +setup_input_root <- Sys.getenv( + "SDM_SETUP_INPUT_ROOT", + unset = "//ABMI-DATA2/science/sdmMethodsDev/0_data/setup_inputs" +) + +# 2. setup_input() ---- + +#' Build a Path Inside the _setup Input Mirror +#' +#' @param ... Character. Path components below the mirror root, +#' starting with the source folder (for example "bird_models"). +#' @return Character. The full path. +#' +#' @example # Example usage of the function +#' # setup_input("bird_models", "Data", "lookups", "birdlist.csv") +setup_input <- function(...) { + file.path(setup_input_root, ...) +} + +# End of script ---- diff --git a/1_code/experiments/.gitkeep b/1_code/experiments/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/1_code/experiments/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/1_code/experiments/README.md b/1_code/experiments/README.md new file mode 100644 index 0000000..03dae98 --- /dev/null +++ b/1_code/experiments/README.md @@ -0,0 +1,46 @@ +# experiments + +One folder per methods question, keyed `exp_NNN_short_description`. +Each folder has a `run.R` entry point that calls `experiment_config()` +and `run_experiment()` (see the [harness README](../harness/README.md)), +and a `README.md` stating the question. The same id names the +experiment's folders in `2_pipeline/` and `3_output/`. + +| Folder | Holds | Status | +| --- | --- | --- | +| `_shared/` | `focal_species.R`: `focal_species_sets()`, the named species sets (`full`, `parity_check`, `plants_only`, `one_each`) | | +| `_template/` | The `run.R` and `README.md` that `new_experiment()` fills in | | +| [`exp_000_parity_v2/`](exp_000_parity_v2/README.md) | The v2 parity check and the baseline for every other experiment | Not yet a gate | +| [`exp_001_xgboost/`](exp_001_xgboost/README.md) | Habitat stage fitted with xgboost, every taxon | Test; 5-draw check only | +| [`exp_002_soilgrids/`](exp_002_soilgrids/README.md) | SoilGrids 0–5 cm terms in the climate stage, plants and birds | Test; 5-draw check only | + +exp_000 is the required gate before iterative R&D: every other +experiment is compared against its tables. + +## Adding an experiment + +1. Scaffold it from the repository root: + + ```r + source("1_code/harness/new_experiment.R") + new_experiment("covariate scale") # dry_run = TRUE to preview + ``` + + This takes the next free number across `1_code/experiments/`, + `2_pipeline/` and `3_output/`, copies `_template/` with the id, + title, author and date filled in, and creates + `2_pipeline//logs/` and `3_output//` with `figures/`, + `tables/` and a stub `report.md`. Leave `id` in `run.R` as it is. +2. State the question and design in its `README.md`. +3. Change one thing relative to exp_000: `stage_models`, + `covariate_files` or `specs`. Keep exp_000's species, draws and seed + unless they are the question. +4. Never add columns to `0_data/` or change `_setup/` for an + experiment. Covariates the dataset lacks are written by a script in + the experiment's folder to `2_pipeline//inputs/` and named in + `covariate_files`; `exp_002_soilgrids` is the example. +5. Run it at 5 draws to check it works, then at 100. Compare it with + an exp_000 run at the same number of draws. + +Each lever is walked through in +[`docs/getting_started.md`](../../docs/getting_started.md#test-a-change). diff --git a/1_code/experiments/exp_000_parity_v2/utils/focal_species.R b/1_code/experiments/_shared/focal_species.R similarity index 61% rename from 1_code/experiments/exp_000_parity_v2/utils/focal_species.R rename to 1_code/experiments/_shared/focal_species.R index 58739ad..9f999d5 100644 --- a/1_code/experiments/exp_000_parity_v2/utils/focal_species.R +++ b/1_code/experiments/_shared/focal_species.R @@ -1,30 +1,17 @@ # --- -# title: Focal Species Sets for Experiment 000 +# title: Focal Species Sets, Shared by Every Experiment # author: Brendan Casey # created: 2026-09-11 # inputs: none # outputs: none; returns objects in memory # notes: -# - Holds the species vectors run.R chooses between, so that -# run.R stays a page of decisions rather than a page of -# names. The sets grow; the run script should not. -# - Every set uses the named-vector convention the harness -# expects: names are taxa, values are species, names may -# repeat, and an unnamed entry applies to every taxon. NULL -# means every species the spec's queue declares, which is the -# full parity run. -# - Names are data slugs, not directory names. Soil mites key -# on "mite". See 1_code/modules/soil_mites/spec.R. -# - Species names must match the response columns exactly. The -# loader stops and lists any name it cannot find, so a typo -# fails before modelling starts rather than part-way through. -# - To see what is available for a taxon: -# head(list_species(data_dir, "mite", "north"), 20) -# - A taxon a set does not name is unrestricted, not excluded. -# Narrow run_taxa alongside a partial set; run.R warns if a -# taxon would silently run its full species list. -# - This mirrors model_sets() and get_model_set() in the -# harness, so the two registries read the same way. +# - Named species sets shared by experiments, so a later +# experiment can run exactly exp_000's species. Sets follow +# taxon_values() (names are data slugs, e.g. "mite"); NULL +# means every species. +# - A taxon a set does not name is unrestricted, not excluded; +# narrow `taxa` alongside a partial set. +# - Available species: list_species(data_dir, "mite", "north"). # --- # 1. Setup ---- @@ -46,28 +33,15 @@ focal_species_sets <- function() { list( ## 2.1 full ---- - # Every species each spec declares. The parity run. + # The full parity run full = NULL, ## 2.2 parity_check ---- - # Two per taxon, for a first run that is meant to be read - # rather than merely survived. - # - # These were not picked for being familiar. A rare species - # has a wide bootstrap band, so the v2 value falls inside it - # whatever the harness does and the parity test passes - # without discriminating. These are the species whose bands - # are tight enough for an in-band result to mean something. - # - # Each was chosen on four criteria: present in both regions - # well above the 20-detection bootstrap threshold; - # prevalence between roughly 5 and 45 per cent, so neither - # floor nor ceiling; present in the v2 reference with all - # 100 draws usable; and, across the pair, one balanced - # between north and south so the southern model is genuinely - # exercised. - # - # Birds use v2's stored draws, so their bands are v2's own. + # Two per taxon, chosen so the parity test discriminates: rare + # species have bands wide enough to pass anything. Criteria: + # well above 20 detections in both regions; prevalence ~5-45%; + # all 100 v2 draws usable; and one of each pair balanced + # between north and south. # # Detections north / south, and prevalence north / south: # Ceratodon.purpureus 2140 / 554 44.1 / 24.0 @@ -122,14 +96,8 @@ focal_species_sets <- function() { ), ## 2.3 plants_only ---- - # The four plant-group taxa, for a run that skips the two - # taxa with the slowest stages. - # - # Pair this with a narrowed run_taxa. A taxon that a set does - # not name is unrestricted rather than excluded, so leaving - # mammals and birds in run_taxa would run every species of - # each and make this the longest run in the file. run.R warns - # when that happens. + # Pair with a narrowed `taxa`, or mammals and birds run every + # species (experiment_config() warns) plants_only = c( bryophyte = "Ceratodon.purpureus", bryophyte = "Bryum.All", @@ -142,8 +110,7 @@ focal_species_sets <- function() { ), ## 2.4 one_each ---- - # One species per taxon. The shortest run that still touches - # every module, for checking plumbing after a change. + # Shortest run touching every module, for plumbing checks one_each = c( bryophyte = "Ceratodon.purpureus", lichen = "Physcia.adscendens", @@ -160,9 +127,8 @@ focal_species_sets <- function() { #' Look Up One Focal-Species Set by Name #' -#' An unrecognized value is returned unchanged, so run.R can -#' hand this either a set name or a species vector written -#' inline and get the right thing back either way. +#' An unrecognized value (e.g. an inline species vector) is +#' returned unchanged. #' #' @param name Character. A name from focal_species_sets(), or #' a species vector to pass through, or NULL for all species. diff --git a/1_code/experiments/_template/README.md b/1_code/experiments/_template/README.md new file mode 100644 index 0000000..5ab9715 --- /dev/null +++ b/1_code/experiments/_template/README.md @@ -0,0 +1,40 @@ +# exp_NNN_short_description + +![Status](https://img.shields.io/badge/Status-Planned-lightgrey) +![Languages](https://img.shields.io/badge/Languages-R-blue) + +## Question + +One sentence: what does this experiment ask? + +## Design + +- **What changes relative to exp_000:** the one thing this + experiment varies (a covariate, a model set, an engine, a + selection rule). +- **What stays the same:** taxa, species, draws and seed, unless + they are the question. +- **How it is set:** `stage_models` or `specs` in `run.R`. + +## How to read the result + +Every experiment is compared with the v2 baseline (exp_000): +`run_experiment()` prints the headline per taxon, region and +metric (median difference, and the share of species that did +better), and writes one comparison table to `3_output//tables/`: + +| File | One row per | Read it for | +| --- | --- | --- | +| `comparison_metrics.csv` | species and metric | Which species moved, with both runs' 10-90% bands | + +The metrics are out-of-bag: each draw scored on the survey units it +left out. "Better" is higher AUC, deviance explained and Spearman, +lower RMSE, and a calibration slope closer to 1. + +## Result + +_To be written after a full run._ + +## Assumptions and caveats + +- diff --git a/1_code/experiments/_template/run.R b/1_code/experiments/_template/run.R new file mode 100644 index 0000000..93b2129 --- /dev/null +++ b/1_code/experiments/_template/run.R @@ -0,0 +1,83 @@ +# --- +# title: Run Experiment NNN - [Short Description] +# author: [Your Name] +# created: [YYYY-MM-DD] +# inputs: +# - the published test dataset (harness/data_source.R) +# - 3_output/exp_000_parity_v2/tables/, the v2 baseline +# outputs: +# in 2_pipeline/exp_NNN_short_description/: +# - // result stores, and run_log.csv +# in 3_output/exp_NNN_short_description/: +# - tables/ (summaries, and comparison_metrics.csv against the +# baseline), run_record.md +# notes: +# - Start an experiment with new_experiment("description") +# (harness/new_experiment.R), which numbers it, fills this +# header and creates its 2_pipeline/ and 3_output/ folders. +# Copying by hand instead, set `id` below to the folder's name. +# - State the question in the README. +# - Change one thing relative to exp_000, so a difference in the +# comparison is a difference in that thing. Use the same +# species, draws and seed as exp_000 unless they are the +# question. +# - The usual levers, from least to most invasive: +# - `stage_models`: different candidate formulas for a stage +# (new covariates, a different model set). +# - `covariate_files`: covariates the dataset does not hold, +# from a CSV this experiment's own script writes to +# 2_pipeline//inputs/. Never add them to 0_data/ or +# _setup/, which stay the v2 data. +# - `specs`: a spec with a stage changed - a different engine +# or selection rule. For a new habitat method in every +# taxon, mammals included, one call: +# specs <- lapply(standard_specs(), replace_stage_method, +# engine = "xgboost") +# See exp_001_xgboost and docs/getting_started.md. +# - A new method in 1_code/methods/, then named in a spec; +# see 1_code/methods/README.md. +# --- + +# 1. Setup ---- + +## 1.1 Load the framework ---- +source("1_code/harness/harness.R") +load_framework() + +## 1.2 Configure the run ---- +config <- experiment_config( + id = "exp_NNN_short_description", + + # The runs to fit, from names(standard_specs()) + taxa = c( + "bryophyte", "lichen", "mite", "vascular_plant", + "mammal_summer", "mammal_winter", "bird" + ), + + # The same species exp_000 ran, so the comparison is like for + # like + species = "parity_check", + n_bootstraps = 100, + seed = 20260909, + + # The change this experiment tests. For example, every climate + # candidate with elevation added: + # stage_models = list( + # climate = extend_models("climate_plant_v2_full", "elevation") + # ), + stage_models = NULL, + + workers = 12 +) + +# 2. Run ---- +# Fit, summarize, and compare with the v2 baseline (exp_000); +# every experiment is compared with it. The comparison is +# written to tables/comparison_metrics.csv. Add `steps` for anything +# this experiment needs beyond that. +results <- run_experiment( + config, + translate = list(bird = bird_habitat_translation) +) + +# End of script ---- diff --git a/1_code/experiments/exp_000_parity_v2/.gitkeep b/1_code/experiments/exp_000_parity_v2/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/1_code/experiments/exp_000_parity_v2/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/1_code/experiments/exp_000_parity_v2/01_collect_results.R b/1_code/experiments/exp_000_parity_v2/01_collect_results.R deleted file mode 100644 index 542fc85..0000000 --- a/1_code/experiments/exp_000_parity_v2/01_collect_results.R +++ /dev/null @@ -1,258 +0,0 @@ -# --- -# title: Collect Model Output (Experiment 000) -# author: Brendan Casey -# created: 2026-09-09 -# inputs: -# in pipeline_dir///: -# - coefficients.csv, metrics.csv, meta.json -# 0_data/test_dataset/, for the bird translation -# outputs: -# in out_dir/tables/: -# - coefficient_summary.csv -# - metric_summary.csv -# - coverage.csv -# notes: -# - Reduces the per-draw result stores to per-species summaries, -# which is what the comparison and the report read. The raw -# stores stay in pipeline_dir: 100 draws by every species is -# too large to commit. -# - One row per stage, not only the last, so the climate stage -# can be compared as well as the habitat stage. Stores -# written before stages were recorded carry the final stage -# only, and are labelled with it from meta.json. -# - Bird landcover coefficients are also translated onto v2's -# standardized habitat template, by -# modules/birds/standardize.R, and added as stage "habitat". -# The raw coefficients stay as stage "landcover". -# - Bootstrap draws are summarized by median and the 10th and -# 90th percentiles rather than mean and standard deviation. -# A coefficient distribution over draws is routinely skewed, -# and a rare species can produce an extreme draw that would -# drag a mean somewhere no draw actually is. -# - `boot1` is the iteration-1 value, the full-data fit. It is -# the like-for-like quantity where v2 fits once rather than -# bootstrapping, as the mammal habitat stage does. -# - `run` is the spec's key in run.R and names the store; the -# taxon, season and part come from the store's meta.json, so -# the two mammal season runs share the taxon "mammal". -# - Expects exp_id, data_dir, pipeline_dir and out_dir from -# run.R, and the bird module to be sourced. -# --- - -# 1. Setup ---- - -## 1.1 Load packages ---- -library(jsonlite) # reading meta.json (version: 1.8.9) - -## 1.2 Resolve paths ---- -tables_dir <- file.path(out_dir, "tables") -dir.create(tables_dir, recursive = TRUE, showWarnings = FALSE) - -## 1.3 Find the result stores ---- -# One per run and region, wherever a run wrote one. -store_dirs <- list.dirs(pipeline_dir, recursive = TRUE) -store_dirs <- store_dirs[ - file.exists(file.path(store_dirs, "meta.json")) -] - -if (length(store_dirs) == 0) { - message( - "No result stores in ", pipeline_dir, - "; run section 2 of run.R first." - ) -} - -# 2. Summarize ---- - -## 2.1 Helpers ---- - -#' Summarize a Value over Bootstrap Draws -#' -#' @param x Numeric vector. -#' @param boot Integer vector of the draw each value came from. -#' @return A named numeric vector of n, median, 10th and 90th -#' percentiles, and the iteration-1 value. -#' -#' @example # Example usage of the function -#' # draw_summary(c(0.1, 0.3, 0.2), 1:3) -draw_summary <- function(x, boot) { - boot1 <- x[boot == 1L] - boot1 <- if (length(boot1) == 1 && is.finite(boot1)) { - boot1 - } else { - NA_real_ - } - - x <- x[is.finite(x)] - - if (length(x) == 0) { - return(c(n = 0, median = NA_real_, p10 = NA_real_, - p90 = NA_real_, boot1 = boot1)) - } - - quantiles <- stats::quantile(x, c(0.1, 0.5, 0.9), names = FALSE) - - c(n = length(x), median = quantiles[2], p10 = quantiles[1], - p90 = quantiles[3], boot1 = boot1) -} - -#' Read a Store's meta.json, Tolerating a Missing Field -#' -#' @param dir Character. A result store directory. -#' @return A list with taxon, season, part and final_stage. -#' -#' @example # Example usage of the function -#' # store_meta(store_dirs[1]) -store_meta <- function(dir) { - meta <- jsonlite::fromJSON(file.path(dir, "meta.json")) - - field <- function(name) { - value <- meta[[name]] - - if (is.null(value) || length(value) == 0 || - identical(value, "NA") || all(is.na(value))) { - NA_character_ - } else { - as.character(value[1]) - } - } - - list( - taxon = field("taxon"), - region = field("region"), - season = field("season"), - part = field("part"), - final_stage = if (is.null(meta$stages)) { - NA_character_ - } else { - utils::tail(meta$stages, 1) - } - ) -} - -## 2.2 Walk the stores ---- -coefficient_rows <- list() -metric_rows <- list() -coverage_rows <- list() - -for (dir_one in store_dirs) { - # A store is ///. The run is read - # off the folder; the rest from meta.json, rather than parsed - # out of a path that may not use forward slashes. - meta <- store_meta(dir_one) - run <- basename(dirname(dir_one)) - region <- if (is.na(meta$region)) basename(dir_one) else meta$region - taxon <- if (is.na(meta$taxon)) run else meta$taxon - - labels <- data.frame( - taxon = taxon, run = run, region = region, - season = meta$season, part = meta$part, - stringsAsFactors = FALSE - ) - - coefficients <- read_results(dir_one, "coefficients") - metrics <- read_results(dir_one, "metrics") - - # A store written before stages were recorded holds the final - # stage only. - if (!is.null(coefficients) && !"stage" %in% names(coefficients)) { - coefficients$stage <- meta$final_stage - } - - # Bird landcover, translated onto v2's standardized template - if (identical(taxon, "bird") && !is.null(coefficients)) { - landcover <- coefficients[coefficients$stage == "landcover", ] - - if (nrow(landcover) > 0) { - translated <- standardize_bird_coefficients( - landcover, data_dir, region - ) - coefficients <- rbind( - coefficients, - translated[, names(coefficients)] - ) - } - } - - coverage_rows[[dir_one]] <- cbind( - labels, - data.frame( - species = length(unique(c( - coefficients$species, metrics$species - ))), - draws = length(unique(c(coefficients$boot, metrics$boot))), - stages = if (is.null(coefficients)) { - "" - } else { - paste(unique(coefficients$stage), collapse = ", ") - }, - has_coefficients = !is.null(coefficients), - has_grid = !is.null(read_results(dir_one, "grid_predictions")), - stringsAsFactors = FALSE - ) - ) - - if (!is.null(coefficients)) { - split_key <- interaction( - coefficients$species, coefficients$stage, coefficients$term, - drop = TRUE - ) - - coefficient_rows[[dir_one]] <- do.call(rbind, lapply( - split(coefficients, split_key), - function(block) { - cbind( - labels, - data.frame( - stage = block$stage[1], - species = block$species[1], term = block$term[1], - stringsAsFactors = FALSE - ), - as.data.frame(t(draw_summary(block$estimate, block$boot))) - ) - } - )) - } - - if (!is.null(metrics)) { - split_key <- interaction( - metrics$species, metrics$metric, drop = TRUE - ) - - metric_rows[[dir_one]] <- do.call(rbind, lapply( - split(metrics, split_key), - function(block) { - cbind( - labels, - data.frame( - species = block$species[1], metric = block$metric[1], - stringsAsFactors = FALSE - ), - as.data.frame(t(draw_summary(block$value, block$boot))) - ) - } - )) - } -} - -# 3. Write ---- -coefficient_summary <- do.call(rbind, coefficient_rows) -metric_summary <- do.call(rbind, metric_rows) -coverage <- do.call(rbind, coverage_rows) - -for (one in list( - list(coefficient_summary, "coefficient_summary.csv"), - list(metric_summary, "metric_summary.csv"), - list(coverage, "coverage.csv") -)) { - if (!is.null(one[[1]])) { - rownames(one[[1]]) <- NULL - data.table::fwrite( - one[[1]], file.path(tables_dir, one[[2]]), na = "" - ) - cat("Wrote tables/", one[[2]], ": ", nrow(one[[1]]), - " rows\n", sep = "") - } -} - -# End of script ---- diff --git a/1_code/experiments/exp_000_parity_v2/02_compare_to_v2.R b/1_code/experiments/exp_000_parity_v2/01_compare_to_v2.R similarity index 70% rename from 1_code/experiments/exp_000_parity_v2/02_compare_to_v2.R rename to 1_code/experiments/exp_000_parity_v2/01_compare_to_v2.R index 4e5aece..6195a41 100644 --- a/1_code/experiments/exp_000_parity_v2/02_compare_to_v2.R +++ b/1_code/experiments/exp_000_parity_v2/01_compare_to_v2.R @@ -3,82 +3,54 @@ # author: Brendan Casey # created: 2026-09-09 # inputs: -# in out_dir/tables/: -# - coefficient_summary.csv -# in v2_reference_dir, chosen in run.R: -# - v2_results.csv, built by _setup/08 (ABMIexploreR) or -# _setup/05 (the network drives) +# - `config`, from run.R: out_dir, data_dir, v2_reference_dir, +# n_bootstraps, v2_bootstraps, focal_species +# - `results$coefficient_summary`, from collect_results() +# - in config$v2_reference_dir: +# - v2_results.csv, built by _setup/08 from the pinned +# ABMIexploreR coefficients # outputs: -# in out_dir/tables/: -# - parity_terms.csv -# - parity_summary.csv +# - in out_dir/tables/: parity_terms.csv, parity_summary.csv +# - `results$parity`, `results$parity_summary` and +# `results$parity_notes`, for the later steps # notes: -# - The gate itself. Compares this run's coefficients against -# the published v2 ones, per stage, species and term. -# - Reads only the one v2_results.csv, so it runs with the -# network drives unmounted. +# - The gate: this run's coefficients against v2's, per stage, +# species and term. Reads only v2_results.csv. # - What is joined to what: -# - Climate is fitted once, province-wide, in v2, so its -# reference rows carry region "all". A run's north and -# south climate results are both scored against them. -# The harness fits climate per region, so expect a gap -# until it does not. +# - v2 climate reference rows carry region "all"; a run's +# north and south climate results are both scored against +# them. # - Habitat is joined on region. Plant terms are relabelled # as v2's 04a standardization does; bird terms are the -# translated ones from 01_collect_results.R (stage -# "habitat"), and the raw "landcover" stage is not scored. +# translated ones collect_results() adds (stage "habitat", +# via run.R's `translate`), and the raw "landcover" stage +# is not scored. # - Mammals are joined on the hurdle part, and their two # season runs are averaged first, because v2's `.all` # references average summer and winter. -# - Parity is distributional where v2 bootstraps: the v2 -# median is scored on whether it falls inside this run's -# 10th-to-90th percentile band. Where v2 fits once (v2_n of -# 1: mammal climate and habitat), the like-for-like run value -# is iteration 1, the full-data fit, and the absolute -# difference is the number to read. -# - Terms v2 overwrites with machinery the harness does not -# have, or fixes at a placeholder, are flagged from -# v2_unreachable_terms() and left out of the reachable -# scores. -# - Whatever cannot be compared is left in `notes`, which the -# report prints. -# - Expects exp_id, data_dir, pipeline_dir and out_dir from -# run.R, and the plant-group module to be sourced. +# - Where v2 bootstraps, the v2 median is scored against this +# run's 10-90% band. Where v2 fits once (mammals, v2_n = 1), +# iteration 1 is compared and the absolute difference read. +# - Terms from v2_unreachable_terms() are left out of the +# reachable scores. What cannot be compared goes to +# `results$parity_notes`. +# - Run as a script_step(). # --- # 1. Setup ---- ## 1.1 Resolve paths ---- -tables_dir <- file.path(out_dir, "tables") +tables_dir <- file.path(config$out_dir, "tables") dir.create(tables_dir, recursive = TRUE, showWarnings = FALSE) -# run.R's v2_reference picks the build. Without it, the drive -# build in 0_data/v2_results/, beside test_dataset/. -v2_reference_dir <- get0( - "v2_reference_dir", - ifnotfound = file.path(dirname(data_dir), "v2_results") -) -reference_path <- file.path(v2_reference_dir, "v2_results.csv") - -reference_builder <- if ( - grepl("abmiexplorer", v2_reference_dir) -) { - "1_code/_setup/08_harmonize_abmiexplorer_results.R" -} else { - "1_code/_setup/05_harmonize_v2_results.R" -} +reference_path <- file.path(config$v2_reference_dir, "v2_results.csv") +reference_builder <- config$v2_reference_builder ## 1.2 Read this run and the reference ---- -summary_path <- file.path(tables_dir, "coefficient_summary.csv") +run_coefficients <- results$coefficient_summary -run_coefficients <- if (file.exists(summary_path)) { - as.data.frame(data.table::fread(summary_path)) -} else { - message( - "No coefficient summary at ", summary_path, - "; run 01_collect_results.R first." - ) - NULL +if (is.null(run_coefficients)) { + message("No coefficients were collected; nothing to compare.") } reference <- if (file.exists(reference_path)) { @@ -157,9 +129,8 @@ if (!is.null(run_coefficients)) { !(run_rows$taxon == "bird" & run_rows$stage == "landcover"), ] - # glm names terms as the formula writes them; v2's published - # tables use plain names. The spatial ones are the bird climate - # terms; the plant ones are stored columns already. + # glm names terms as the formula writes them (e.g. the bird + # spatial terms); v2's tables use plain names v2_names <- c( "(Intercept)" = "Intercept", "I(Easting^2)" = "Easting2", @@ -169,11 +140,13 @@ if (!is.null(run_coefficients)) { renamed <- run_rows$term %in% names(v2_names) run_rows$term[renamed] <- v2_names[run_rows$term[renamed]] - # The mammal hurdle writes one stage per v2 table, - # habitat_presence, habitat_abundance and habitat_total; each - # is v2's habitat stage for that part. - hurdle <- grepl("^habitat_(presence|abundance|total)$", run_rows$stage) - run_rows$part[hurdle] <- sub("^habitat_", "", run_rows$stage[hurdle]) + # The hurdle's habitat_ stages map to v2's habitat stage + hurdle <- grepl( + "^habitat_(presence|abundance|total)$", run_rows$stage + ) + run_rows$part[hurdle] <- sub( + "^habitat_", "", run_rows$stage[hurdle] + ) run_rows$stage[hurdle] <- "habitat" # v2 fits climate once across the province. @@ -218,17 +191,14 @@ if (!is.null(run_coefficients)) { } ## 3.3 Average the mammal seasons ---- - # v2's `.all` references are the mean of the summer and winter - # fits where both exist. The season is part of the harness's - # species name, so it is stripped and the runs averaged. + # v2's `.all` references average summer and winter is_mammal <- run_rows$group == "mammal" if (any(is_mammal)) { mammals <- run_rows[is_mammal, ] - # Stores written before meta.json recorded the season and - # part lack them. The season is in the species name, and - # those runs fitted presence, the spec's default. + # Older stores lack season and part in meta.json: take the + # season from the species name; those runs fitted presence no_season <- is.na(mammals$season) | mammals$season == "" mammals$season[no_season] <- tolower( sub("^.*_(Summer|Winter)$", "\\1", mammals$species[no_season]) @@ -248,7 +218,7 @@ if (!is.null(run_coefficients)) { keys <- c("taxon", "group", "region", "ref_region", "stage", "ref_part", "species", "term") - values <- c("n", "median", "p10", "p90", "boot1") + values <- c("n", "median", "p10", "p90", "boot1", "mean", "sd") averaged <- stats::aggregate( mammals[, values], @@ -297,8 +267,7 @@ if (!is.null(run_rows) && !is.null(reference) && ) ## 4.1 The run value to compare ---- - # Where v2 fits once, its value is a full-data fit, and the - # like-for-like run value is iteration 1. + # Where v2 fits once, compare iteration 1 parity$comparison <- ifelse( !is.na(parity$v2_n) & parity$v2_n == 1, "iteration 1", "median" @@ -330,10 +299,8 @@ if (!is.null(run_rows) && !is.null(reference) && parity$reachable <- is.na(parity$unreachable_reason) ## 4.2a Flag negligible terms ---- - # Terms model averaging has shrunk to nothing in both runs. They - # stay in the in-band scores, which the numerical tolerance - # already handles, but not in the band ratio or the Spearman, - # where their noise would decide the row. + # Kept in the in-band scores (the tolerance handles them), not + # in the band ratio or Spearman, where their noise would decide term_scales <- term_scale(reference, c("taxon", "stage")) parity <- merge( parity, term_scales, @@ -351,16 +318,14 @@ if (!is.null(run_rows) && !is.null(reference) && matched, abs(parity$run_value - parity$v2_median), NA_real_ ) - # A term agrees when the v2 median is inside this run's band, - # or when the two are within numerical tolerance. The second - # matters for averaged terms carrying almost no model weight: - # their values run to 1e-20 or 1e-137, and banding numerical - # noise is meaningless. + # In band, or within numerical tolerance (near-zero averaged + # terms run to 1e-20 or 1e-137) parity$in_band <- matched & ( (is.finite(parity$p10) & is.finite(parity$p90) & parity$v2_median >= parity$p10 & parity$v2_median <= parity$p90) | - parity$absolute_difference <= parity_targets()$numerical_tolerance + parity$absolute_difference <= + parity_targets()$numerical_tolerance ) parity$standardized_difference <- ifelse( matched & is.finite(spread) & spread > 0, @@ -429,9 +394,7 @@ if (!is.null(parity) && nrow(parity) > 0) { collapse = "+"), species = length(unique(compared$species)), terms_compared = nrow(compared), - # The draws the store actually holds, which the verdict - # checks: a store overwritten by a shorter run still reads - # the configured draw count. + # Draws actually stored, not configured min_draws = if (nrow(compared) == 0) { NA_integer_ } else { @@ -452,9 +415,8 @@ if (!is.null(parity) && nrow(parity) > 0) { signif(max(reachable$absolute_difference, na.rm = TRUE), 3) }, negligible_terms = sum(reachable$negligible), - # Per species, then the median: the terms span twenty - # orders of magnitude, so a rank correlation, as the v2 - # self-agreement calibration uses. Reported, not gated. + # Per species, then the median (as the v2 calibration). + # Reported, not gated. median_spearman = med(vapply( split(substantive, substantive$species), function(one) { @@ -467,11 +429,8 @@ if (!is.null(parity) && nrow(parity) > 0) { }, numeric(1) )), - # This run's 10-90% band over v2's, per term; the median - # per species, then over species, as the v2 calibration - # takes it. Single terms vary threefold between two v2 - # runs, a species' median does not. NA where v2 has no - # band: the iteration-1 rows. + # Band ratio: median per species, then over species (as + # the calibration). NA for iteration-1 rows. median_band_ratio = med(vapply( split(substantive, substantive$species), function(one) { @@ -488,9 +447,8 @@ if (!is.null(parity) && nrow(parity) > 0) { # The verdict against the proposed targets. A trial run - fewer # draws than v2, or a species subset - is not gated. - gated <- exists("n_bootstraps") && exists("v2_bootstraps") && - exists("focal_species") && n_bootstraps >= v2_bootstraps && - is.null(focal_species) + full_length <- config$n_bootstraps >= config$v2_bootstraps + gated <- full_length && is.null(config$focal_species) verdict_of <- function(gate) { vapply( @@ -508,23 +466,17 @@ if (!is.null(parity) && nrow(parity) > 0) { parity_summary$verdict <- verdict_of(gated) - # A run with v2's full draw count on a species subset is not the - # gate, but its rows can still be read against the targets. Said - # separately, so it is never mistaken for the verdict. - full_length <- exists("n_bootstraps") && exists("v2_bootstraps") && - n_bootstraps >= v2_bootstraps + # Full draws on a species subset: readable, but not the gate parity_summary$indicative_verdict <- if (!gated && full_length) { verdict_of(TRUE) } else { NA_character_ } - # A distributional row read from fewer stored draws than v2 has - # no verdict, whatever the run was configured for: a band from - # five draws is narrower than one from a hundred. + # Fewer stored draws than v2: no verdict short <- grepl("median", parity_summary$comparison) & !is.na(parity_summary$min_draws) & - parity_summary$min_draws < get0("v2_bootstraps", ifnotfound = 0L) + parity_summary$min_draws < config$v2_bootstraps short_note <- paste0( "not gated (store holds ", parity_summary$min_draws, " draws)" ) @@ -542,6 +494,7 @@ if (!is.null(parity) && nrow(parity) > 0) { cat("\nParity against v2, by stage:\n") print(parity_summary) } else { + parity_summary <- NULL cat("\nNo parity comparison produced.\n") } @@ -549,4 +502,9 @@ for (note in notes) { cat(" note: ", note, "\n", sep = "") } +# 6. Hand on ---- +results$parity <- parity +results$parity_summary <- parity_summary +results$parity_notes <- notes + # End of script ---- diff --git a/1_code/experiments/exp_000_parity_v2/04_plot_parity.R b/1_code/experiments/exp_000_parity_v2/02_plot_parity.R similarity index 81% rename from 1_code/experiments/exp_000_parity_v2/04_plot_parity.R rename to 1_code/experiments/exp_000_parity_v2/02_plot_parity.R index e34f094..cec6f34 100644 --- a/1_code/experiments/exp_000_parity_v2/04_plot_parity.R +++ b/1_code/experiments/exp_000_parity_v2/02_plot_parity.R @@ -3,8 +3,9 @@ # author: Brendan Casey # created: 2026-09-30 # inputs: -# in out_dir/tables/: -# - parity_terms.csv, from 02_compare_to_v2.R +# - `config$out_dir`, from run.R +# - `results$parity`, from 01_compare_to_v2.R; or, run on its +# own, out_dir/tables/parity_terms.csv # outputs: # in out_dir/figures/: # - parity_.png, one per taxon @@ -12,38 +13,18 @@ # - parity_bell.png, v2 and the run as bell curves, one facet # each # notes: -# - One figure per taxon. Each panel is a stage and region (and, -# for mammals, a hurdle part) for one species; each row a term. -# v2 and exp_000 are drawn in different colours as a median and -# a 10th-to-90th percentile bar, the same summaries the gate -# compares. -# - Intervals, not densities: v2's reference keeps only the -# median and the 10th and 90th percentiles of its draws, so its -# full distribution cannot be drawn. The run's is summarized -# the same way, so the two are read alike. -# - Terms span 1e-144 to 1e2, so no single axis holds them. Each -# term is put on its own scale, relative to v2: 0 is v2's -# median and one unit is half v2's 10-90% band, the scale the -# gate's standardized difference uses. Where v2 fitted once and -# has no band (the mammal hurdle), one unit is half the run's -# band instead, and v2 is drawn as a point. There the gate -# compares the run's first draw with v2's single fit, so that -# draw is drawn too, as its own colour. -# - Values beyond +/- 4 units are drawn at the panel's edge, so -# one far-off term does not flatten the rest. Terms the gate -# cannot reach, or with no band to scale by, are left out and -# counted in the caption. -# - parity_mean.png averages the same scaled values over every -# species and term, one facet per taxon (or per group; section -# 1.4), one row per stage and region. -# - parity_bell.png draws each row as two curves, v2 and the run, -# each the average of its terms read as normal curves from -# their median and band. A shifted curve is a location -# difference, a wider or narrower one a spread difference. The -# normal shape is a drawing convenience; v2 keeps only three -# summaries of its draws. -# - Expects out_dir from run.R. Can also be run on its own from -# the repository root after 02_compare_to_v2.R. +# - Medians and 10-90% bars, not densities: v2 keeps only those +# three summaries of its draws. +# - Terms span 1e-144 to 1e2, so each is scaled to v2: 0 is v2's +# median, one unit is half v2's band (the gate's standardized +# difference). Where v2 fitted once (mammal hurdle), the unit +# is half the run's band, v2 is a point, and the run's first +# draw (what the gate compares) is shown. +# - Values beyond +/- 4 units are clamped to the edge; +# unreachable or unscalable terms are dropped and counted in +# the caption. +# - Runs as a script_step(), or alone from the repository root +# after 01_compare_to_v2.R. # --- # 1. Setup ---- @@ -53,26 +34,30 @@ library(data.table) # reading and reshaping tables (version: 1.18.0) library(ggplot2) # plotting (version: 4.0.3) ## 1.2 Resolve paths ---- -if (!exists("out_dir")) { - out_dir <- file.path(getwd(), "3_output", "exp_000_parity_v2") +out_dir <- if (exists("config")) { + config$out_dir +} else { + file.path(getwd(), "3_output", "exp_000_parity_v2") } terms_path <- file.path(out_dir, "tables", "parity_terms.csv") figures_dir <- file.path(out_dir, "figures") +dir.create(figures_dir, recursive = TRUE, showWarnings = FALSE) -if (!file.exists(terms_path)) { +## 1.3 Import data ---- +# From the comparison step when it ran; from its table otherwise +parity_terms <- if (exists("results") && !is.null(results$parity)) { + as.data.table(results$parity) +} else if (file.exists(terms_path)) { + fread(terms_path) +} else { stop( - "parity_terms.csv not found; run 02_compare_to_v2.R first:\n ", + "No parity terms; run 01_compare_to_v2.R first:\n ", terms_path, call. = FALSE ) } -dir.create(figures_dir, recursive = TRUE, showWarnings = FALSE) - -## 1.3 Import data ---- -parity_terms <- fread(terms_path) - ## 1.4 Choose the mean figure's facets ---- # "taxon" gives one facet per taxon; "group" pools the four plant # taxa into one facet beside mammals and birds. @@ -255,15 +240,9 @@ for (one_taxon in sort(unique(plot_data$taxon))) { # One figure, one facet per taxon (or group), each row a stage and # region. Input: plot_data. Output: parity_mean.png. # -# The per-term values are averaged over every species and term in -# the row, still in v2 units, so v2 sits at 0 and its bar is its -# mean band. The averages use the clamped values, so a term past -# the axis counts as +/- 4, not as its full distance. -# -# A signed mean shows bias but lets a +2 and a -2 cancel, so each -# row is also labelled with the mean absolute difference of the -# run's median from v2's, and the share of terms whose v2 median -# lies inside the run's band: the gate's first two measures. +# Averages the clamped scaled values over species and terms. A +# signed mean lets +2 and -2 cancel, so rows are also labelled +# with the mean absolute difference and the in-band share. ## 4.1 Average per facet, row and source ---- plot_data[, facet := get(mean_facet)] @@ -367,19 +346,10 @@ cat("Parity figure: ", mean_file, "\n", sep = "") # stage and region, stacked. Input: plottable. Output: # parity_bell.png. # -# Each term's 100 draws are summarized, in both runs, by a median -# and a 10-90% band. Read as a normal curve, the median is its -# centre and the band spans 2 x 1.2816 standard deviations. Each -# row's curve is the average of its terms' curves, in v2 units, -# so v2's sits at 0 and the run's shows where, and how widely, the -# run lands against it: shifted if the medians differ, wider or -# narrower if the bands do. -# -# The normal shape is a drawing convenience, not a claim: v2 keeps -# only the three summaries, so the true shape of its draws is not -# known. Negligible terms are left out, as the gate leaves them -# out. Where v2 fitted once (the mammal hurdle) it has no band and -# is drawn as a line at 0. +# Each term as a normal curve (median centre; the 10-90% band is +# 2 x 1.2816 SD), averaged per row in v2 units. The normal shape +# is a drawing convenience, not a claim. Negligible terms are +# left out; v2 without a band is a line at 0. ## 5.1 One normal curve per term ---- # qnorm(0.9): the 90th percentile is this many standard deviations @@ -390,7 +360,7 @@ bell_terms <- copy(plottable) if (!"negligible" %in% names(bell_terms)) { message( "parity_terms.csv has no `negligible` column; rerun ", - "02_compare_to_v2.R. Every term is drawn." + "01_compare_to_v2.R. Every term is drawn." ) bell_terms[, negligible := FALSE] } diff --git a/1_code/experiments/exp_000_parity_v2/03_build_report.R b/1_code/experiments/exp_000_parity_v2/03_build_report.R index 6dcda77..242ca87 100644 --- a/1_code/experiments/exp_000_parity_v2/03_build_report.R +++ b/1_code/experiments/exp_000_parity_v2/03_build_report.R @@ -3,126 +3,58 @@ # author: Brendan Casey # created: 2026-09-09 # inputs: -# from run.R: -# - specs, run_taxa, focal_species, n_bootstraps, -# v2_bootstraps, boot_seed -# from 02_compare_to_v2.R, when it ran this session: -# - notes, the references it could not reach -# in out_dir/tables/: -# - coverage.csv, metric_summary.csv, parity_summary.csv -# in v2_reference_dir, chosen in run.R: +# - `config`, from run.R: id, specs, taxa, focal_species, +# n_bootstraps, v2_bootstraps, seed, out_dir and the v2 +# reference settings +# - `results`: coverage and metric_summary (collect_results()), +# parity_summary and parity_notes (01_compare_to_v2.R) +# - in config$v2_reference_dir: # - v2_results_coverage.csv # outputs: # in out_dir/: # - report.md # notes: -# - Holds no status of its own. What each spec reproduces of v2 -# is read from the specs' `v2_coverage`, what v2 reference -# exists from v2_results_coverage.csv, and what the gate -# actually reached from the comparison step. Coverage is -# stated once, in the specs, so the report cannot drift from -# them. -# - States what kind of run this was first. A trial run writes -# the same files as a full one, and its parity numbers are -# not a parity read. -# - Written on every run, so a partial run still leaves a -# legible record of what it was. -# - Expects exp_id, data_dir, pipeline_dir, out_dir and the -# run settings above from run.R. +# - Holds no status of its own: coverage comes from the specs' +# `v2_coverage`, references from v2_results_coverage.csv, and +# results from the comparison step. +# - Leads with the kind of run, since a trial writes the same +# files as a full run. Run as a script_step(). # --- # 1. Setup ---- ## 1.1 Resolve paths ---- -tables_dir <- file.path(out_dir, "tables") -report_file <- file.path(out_dir, "report.md") - -# run.R's v2_reference picks the build. Without it, the drive -# build in 0_data/v2_results/, beside test_dataset/. -v2_reference_dir <- get0( - "v2_reference_dir", - ifnotfound = file.path(dirname(data_dir), "v2_results") -) +report_file <- file.path(config$out_dir, "report.md") v2_coverage_path <- file.path( - v2_reference_dir, "v2_results_coverage.csv" + config$v2_reference_dir, "v2_results_coverage.csv" ) - -v2_reference_builder <- if ( - grepl("abmiexplorer", v2_reference_dir) -) { - "1_code/_setup/08_harmonize_abmiexplorer_results.R" -} else { - "1_code/_setup/05_harmonize_v2_results.R" -} - -read_table <- function(path) { - if (!file.exists(path)) { - return(NULL) - } - - as.data.frame(data.table::fread(path)) -} +v2_reference_builder <- config$v2_reference_builder ## 1.2 Read the inputs ---- -coverage <- read_table(file.path(tables_dir, "coverage.csv")) -metric_summary <- read_table( - file.path(tables_dir, "metric_summary.csv") -) -parity_summary <- read_table( - file.path(tables_dir, "parity_summary.csv") -) -v2_coverage <- read_table(v2_coverage_path) - -# The specs that ran, not every spec defined, so the coverage -# table describes this run. -run_specs <- specs[intersect(run_taxa, names(specs))] - -# The comparison step leaves its unreachable references in -# `notes`. Absent when it did not run this session. -gate_notes <- if (exists("notes") && is.list(notes)) { - unlist(notes, use.names = FALSE) +coverage <- results$coverage +metric_summary <- results$metric_summary +parity_summary <- results$parity_summary +v2_coverage <- if (file.exists(v2_coverage_path)) { + as.data.frame(data.table::fread(v2_coverage_path)) } else { - character(0) + NULL } +# The comparison step's unreachable references +gate_notes <- unlist(results$parity_notes, use.names = FALSE) + # 2. Assemble ---- ## 2.1 Helpers ---- - -#' Render a Data Frame as a Markdown Table -#' -#' @param x A data frame, or NULL. -#' @param empty Character. What to say when there is nothing. -#' @return A character vector of markdown lines. -#' -#' @example # Example usage of the function -#' # markdown_table(coverage) +# Tables are rendered by md_table() (harness/run_record.R), so the +# report and the run record format numbers the same way. markdown_table <- function(x, empty = "_Nothing recorded._") { - if (is.null(x) || nrow(x) == 0) { - return(empty) - } - - # A pipe inside a cell would split it into two columns. - cells <- as.data.frame( - lapply(x, function(col) gsub("|", "\\|", col, fixed = TRUE)), - stringsAsFactors = FALSE - ) - - header <- paste("|", paste(names(x), collapse = " | "), "|") - rule <- paste( - "|", paste(rep("---", ncol(x)), collapse = " | "), "|" - ) - - rows <- apply(cells, 1, function(row) { - paste("|", paste(row, collapse = " | "), "|") - }) - - c(header, rule, rows) + if (is.null(x) || nrow(x) == 0) empty else md_table(x, digits = 3L) } ## 2.2 What kind of run this was ---- -is_full_species <- is.null(focal_species) -is_full_draws <- n_bootstraps >= v2_bootstraps +is_full_species <- is.null(config$focal_species) +is_full_draws <- config$n_bootstraps >= config$v2_bootstraps run_kind <- if (is_full_species && is_full_draws) { "**Full run.** Every species, every draw." @@ -130,21 +62,22 @@ run_kind <- if (is_full_species && is_full_draws) { paste0( "**Trial run.** ", if (!is_full_species) { - paste0(length(focal_species), " focal species") + paste0(length(config$focal_species), " focal species") } else { "every species" }, - ", ", n_bootstraps, " of ", v2_bootstraps, " draws. ", + ", ", config$n_bootstraps, " of ", config$v2_bootstraps, + " draws. ", "Its parity numbers show the pipeline ran; they are not a ", "parity read." ) } -seed_text <- if (is.null(boot_seed)) { +seed_text <- if (is.null(config$seed)) { "unseeded, as in v2" } else { paste0( - "`boot_seed = ", boot_seed, "`, one derived seed per species" + "`seed = ", config$seed, "`, one derived seed per species" ) } @@ -164,7 +97,10 @@ reference_lines <- if (is.null(v2_coverage)) { } gate_lines <- if (length(gate_notes) == 0) { - "_Every taxon and region that ran had a reference to compare against._" + paste( + "_Every taxon and region that ran had a reference to", + "compare against._" + ) } else { paste0("- ", gate_notes) } @@ -173,9 +109,7 @@ gate_lines <- if (length(gate_notes) == 0) { fit_lines <- if (is.null(metric_summary)) { "_No metrics recorded._" } else { - # In-sample against held-out, and v2's own validation where - # the spec supplies it. Out-of-bag is the honest read; - # in-sample is what v2 reports. + # Out-of-bag is the honest read; in-sample is what v2 reports shown <- c( insample_auc = "insample_auc", oob_auc = "oob_auc", @@ -199,7 +133,8 @@ fit_lines <- if (is.null(metric_summary)) { for (label in names(shown)) { values <- block$median[block$metric == shown[[label]]] - row[[paste0("median_", label)]] <- if (length(values) == 0) { + column <- paste0("median_", label) + row[[column]] <- if (length(values) == 0) { NA_real_ } else { round(stats::median(values, na.rm = TRUE), 3) @@ -214,7 +149,7 @@ fit_lines <- if (is.null(metric_summary)) { ## 2.5 Write the report ---- lines <- c( - paste0("# ", exp_id), + paste0("# ", config$id), "", paste0("Generated ", format(Sys.time(), "%Y-%m-%d %H:%M:%S")), "", @@ -228,14 +163,14 @@ lines <- c( "stated. Change it there, not here.", "", markdown_table( - spec_coverage(run_specs), + spec_coverage(config$specs), "_No specs ran._" ), "", "## v2 references", "", paste0( - "Scored against `", get0("v2_reference", ifnotfound = "drives"), + "Scored against `", config$v2_reference, "`, built by `", v2_reference_builder, "`. What it holds, ", "from `v2_results_coverage.csv`:" ), @@ -287,11 +222,13 @@ lines <- c( "## Model fit", "", paste( - "Median AUC over species. `insample` scores the units each", - "draw fitted; `oob` the units it left out, the held-out read.", - "`v2val` is v2's validation, scored from the coefficients as v2", - "does (plants only). Iteration 1 is the full data and has no", - "out-of-bag units." + "Median AUC over species, scored on each run's final model:", + "for plants, v2's own prediction from its coefficient tables.", + "`insample` scores the units each draw fitted; `oob` the units", + "it left out, the held-out read. `v2val` is v2's validation,", + "scored from the coefficients as v2 does (plants only), so for", + "plants `v2val_full` equals `insample_auc`. Iteration 1 is the", + "full data and has no out-of-bag units." ), "", fit_lines, @@ -307,5 +244,6 @@ lines <- c( writeLines(lines, report_file) cat("Wrote ", report_file, "\n", sep = "") +results$report_file <- report_file # End of script ---- diff --git a/1_code/experiments/exp_000_parity_v2/README.md b/1_code/experiments/exp_000_parity_v2/README.md index 72aae92..7d4736c 100644 --- a/1_code/experiments/exp_000_parity_v2/README.md +++ b/1_code/experiments/exp_000_parity_v2/README.md @@ -4,251 +4,154 @@ ![Languages](https://img.shields.io/badge/Languages-R-blue) The validation gate. Runs each taxon's v2 spec through the harness and -compares the result against the published v2 output. +compares the result with the published v2 output. Every other +experiment is compared against this one. ## Question -Does the framework, configured as v2, reproduce the v2 results? +Does the framework, configured as v2, reproduce the v2 results? If it +does, a difference in a later experiment is a difference in method, +not in plumbing. -Everything downstream depends on the answer. Later experiments are -compared against this one, so a difference here is a difference in the -plumbing, not in the method being tested. +## How to run + +From the repository root: + +```r +source("1_code/experiments/exp_000_parity_v2/run.R") +``` + +Every decision is in `run.R`'s `experiment_config()` call (section +1.2); section 1.3 points the run at the v2 reference. As committed, +`run.R` runs the `parity_check` species at 5 draws on 12 workers. For +the gate, set `species = NULL` and `n_bootstraps = 100`. +`run_experiment()` fits, summarizes, then runs this experiment's steps: + +| Step | Script | Writes, in `3_output/exp_000_parity_v2/` | +| --- | --- | --- | +| compare | `01_compare_to_v2.R` | `tables/parity_terms.csv`, `tables/parity_summary.csv` | +| plot | `02_plot_parity.R` | `figures/parity_.png`, `parity_mean.png`, `parity_bell.png` | +| report | `03_build_report.R` | `report.md` | + +`utils/parity_targets.R` holds the targets the compare step reads. +`v2_self_agreement.R` is not part of `run.R`: run it once, after +`_setup/03` and `_setup/08`, to write +`tables/v2_self_agreement.csv` and `v2_self_agreement_summary.csv`. + +## Inputs and outputs + +- **Inputs:** the published test dataset, and + `abmiexplorer/v2_results.csv` from the published v2 results (both + via `harness/data_source.R`). +- **Outputs:** the stores in `2_pipeline/exp_000_parity_v2/`, the + summary tables every experiment writes (see the + [harness README](../../harness/README.md#outputs)), and: + +| Output | Holds | +| --- | --- | +| `tables/parity_terms.csv` | Term by term: `coefficient_summary.csv` joined to the v2 reference. Adds `ref_region`, `ref_part`, the v2 summary (`v2_median`, `v2_p10`, `v2_p90`, `v2_n`), `comparison` (`median`, or `iteration 1` where v2 fits once), `reachable`, `in_band`, `absolute_difference` and `standardized_difference` (gap ÷ half this run's band). Gitignored | +| `tables/parity_summary.csv` | The gate: one row per taxon × region × stage, with in-band rates, gaps, `median_band_ratio`, `median_spearman` (not gated), `min_draws`, `verdict` and `indicative_verdict` | +| `figures/` | Each term's run and v2 median and band relative to v2; the per-taxon mean; v2 and the run as distributions | +| `report.md` | Run kind and seed, what each spec reproduces of v2, parity by stage, model fit | ## What it covers -Coverage is stated in one place: each spec's `v2_coverage`, in -`1_code/modules//spec.R` (the four plant-group taxa share -`_shared/plant_group.R`). The report prints it from there on every run, -so read `3_output/exp_000_parity_v2/report.md` for the current -statement. In short: +Each spec's `v2_coverage` is the authoritative statement, and +`report.md` prints it. In short: | Taxon | Climate | Habitat or landcover | Resampling | | --- | --- | --- | --- | -| Plant group (×4) | Reproduced: v2's 58-model set, fitted once province-wide; exact to v2's code | Reproduced: IVW on the grid, stand-age splines, cutblock convergence, footprint pooling and pAspen; exact to v2 on the full-data draw | Reproduced: once per species over the province, or v2's own stored draws | -| Mammals | Reproduced: v2's precomputed climate prediction | Reproduced: the hurdle (presence, abundance, total) on the full habitat set, with splines, calibration and convergence; exact to v2's published tables | v2 fits once; iteration 1 is compared | -| Birds | Reproduced: v2's 25 candidates, province-wide, carried as exp(link + offset); exact to v2's code | Reproduced: weighted staged BIC from `count ~ climate`; exact to v2's code | Reproduced: v2's stored draws | - -If this table and the specs disagree, the specs are right; update this -table. - -## What the gate compares today - -`02_compare_to_v2.R` compares **every stage** against one -`v2_results.csv`, so it runs with the network drives unmounted. -`v2_reference` in `run.R` section 1.3a picks which build of it: - -| `v2_reference` | Built by | Source | Holds | -| -------------- | -------- | ------ | ----- | -| `"abmiexplorer"` (default) | `_setup/08_harmonize_abmiexplorer_results.R` | `data/species-coefs.RData` from [ABMIexploreR](https://github.com/ABbiodiversity/ABMIexploreR), pinned to commit `848eeed` | Only the published species. Every taxon, plus amphibians. | -| `"drives"` | `_setup/05_harmonize_v2_results.R` | `COEFS.RData`, the Mammals drive, `Birds2024.RData` | Every species v2 fitted. | - -The two builds use the same models for every taxon. On 2026-10-02 -every shared species and term matched to about 1e-15, except where -some v2 draws failed. `05` drops failed draws; the package stores 100 finite -draws for every cell. Re-scoring the `parity_check` run against both -builds gave the same in-band rates, standardized differences and -verdicts wherever both had the species. The differences are in -coverage: - -- **Species.** `Bryum.All`, plus `Trhypochthonius.tectorum` in the - north, are not published, so they have no package reference. -- **Bird `TreedFen`.** The package holds one `TreedFen` coefficient - where v2 has nine age classes, so those terms do not join. The - package's value correlates at about 0.99 with each age class. -- **Mammals.** The package's mammal data were replaced on 2026-07-27. - The pinned commit holds the 2024 coefficient tables the harness - reproduces (an earlier July 2025 set was a different fit, presence - in the north only). The package stores them on the link scale, and - `08` back-transforms them (logit for presence, log for abundance - and total). Its 100 draws are one fit repeated, so they are scored - on iteration 1, as against `05`. South `WetlandMargin` and the - placeholder `Bare` are not in the package. Woodland caribou and - deer are not published. - -`08` adds the package's relabelled terms under the harness's names -(`TreedBog*` → `BlackSpruce*` for birds, `TreedFen` → each plant fen -age, `RuralResidential` → `Rural`, mammals' `WhiteSpruce*` → `Spruce*`, -and so on). It adds these only where -the values were checked to be identical. It also drops the template's -all-zero padding cells. - -- **Climate:** scored against v2's single province-wide fit - (`region = "all"`), which the harness now reproduces: one fit per draw, - shared by both regions. -- **Plant habitat:** terms relabelled as v2's `04a` standardization - publishes them (`BlackSpruce` as `TreedBog`, `UrbInd` copied to - `Urban`, `Industrial` and `Rural`, and so on). The unaged stand types do - not join, because v2 publishes only their spline-fitted age classes. -- **Bird habitat:** the landcover coefficients translated onto v2's - standardized template by `modules/birds/standardize.R`, a port of - v2's `08.PackageCoefficients.R`. Checked against v2's own packaged - output: translating v2's raw per-draw coefficients reproduces - `Birds2024.RData` exactly. The raw `landcover` stage is kept in the - summary but not scored. -- **Mammals:** the summer and winter runs are averaged, as v2's `.all` - tables are, and compared at iteration 1, because v2 fits once. The - hurdle writes presence, abundance and total abundance on v2's full - habitat set, each joined to its own v2 table. - -Terms v2 sets with machinery the harness lacks, or fixes at a placeholder, -are listed from the v2 code in `v2_unreachable_terms()` and left out of -the reachable scores. The report lists anything its comparison could not -join, so this is checkable run by run. - -Earlier ad hoc checks put mammal presence at r = 0.985 and abundance at -r = 0.79 against v2 on iteration 1. They were not produced by the -committed gate and cannot be regenerated from it. +| Plant group (×4) | v2's 58-model set, fitted once province-wide; exact to v2's code | IVW on the grid, stand-age splines, cutblock convergence, footprint pooling and pAspen; exact to v2 on the full-data draw | Once per species over the province, or v2's stored draws | +| Mammals | v2's precomputed climate prediction | The hurdle (presence, abundance, total) on the full habitat set, with splines, calibration and convergence; exact to v2's published tables | v2 fits once; iteration 1 is compared | +| Birds | v2's 25 candidates, province-wide, carried as exp(link) without the QPAD offset; exact to v2's code | Weighted staged BIC; exact to v2's code | v2's stored draws | + +If this table and the specs disagree, the specs are right. + +## The v2 reference + +`v2_results.csv` is built by `_setup/08` from ABMIexploreR's published +coefficients (`data/species-coefs.RData`), pinned to commit `848eeed`. +On 2026-10-02 it matched the retired by-hand build from the network +drives to about 1e-15 on every shared species and term. Gaps: + +- `Bryum.All`, and `Trhypochthonius.tectorum` in the north, are not + published. +- Bird `TreedFen` is one coefficient in the package and nine age + classes in v2, so those terms do not join. +- Mammals: the pinned commit holds the 2024 tables the harness + reproduces, on the link scale; `08` back-transforms them. Its draws + are one fit repeated, so they are scored on iteration 1. South + `WetlandMargin`, the placeholder `Bare`, woodland caribou and deer + are not published. + +`08` relabels the package's terms to the harness's names only where +values were checked identical. Plant habitat terms are relabelled as +v2 publishes them; bird landcover is translated onto v2's habitat +template by `modules/birds/standardize.R`; mammal seasons are averaged, +as v2's `.all` tables are. Terms v2 sets by machinery the harness +lacks are listed in `v2_unreachable_terms()` (`01_compare_to_v2.R`) +and left out of the reachable scores. ## Parity targets (proposed) -The targets live in `utils/parity_targets.R`, and `02_compare_to_v2.R` -reads a `verdict` per taxon, region and stage against them. **They are -proposals until agreed**; the report says so. +In `utils/parity_targets.R`. **Proposals until agreed.** -**How closely v2 agrees with itself.** `v2_self_agreement.R` compares two -v2 runs of the same code on the same data: the bryophyte climate stage as -published in `COEFS.RData`, and the rerun `_setup/03` made, 134 species by -19 terms by 100 draws each, scored exactly as the gate scores the harness. +| Comparison | Passes when | +| --- | --- | +| Numerical (iteration 1, where v2 fits once: the mammal hurdle) | Every reachable term within 1e-6 | +| Distributional (plant and bird stages) | At least 90% of reachable terms in band, median standardized difference ≤ 0.25, and median band-width ratio between 0.75 and 1.33 | + +They are set from how closely two v2 runs agree with each other +(`v2_self_agreement.R`: the bryophyte climate stage, ABMIexploreR's +run against `_setup/03`'s rerun). Measured against v2's `COEFS.RData`: | Measure | v2 against itself | | --- | --- | | Terms with one run's median in the other's 10–90% band | 100% | | Median standardized difference | 0.019 | -| Band-width ratio (one run's band over the other's), median per species | 1.005 (5th–95th percentile over species 0.81–1.19) | -| Median per-species Spearman correlation of the medians | 0.998 (10th percentile 0.917) | +| Band-width ratio, median per species | 1.005 (5th–95th percentile 0.81–1.19) | +| Median per-species Spearman of the medians | 0.998 | -The ratio and the Spearman leave out **negligible terms**: those both runs -put below a millionth of the term's typical size, the 90th percentile of its -absolute v2 median over every species of the taxon and stage -(`is_negligible()` in `utils/parity_targets.R`). Model averaging shrinks -unsupported terms to 1e-20 or 1e-140, where band widths and ranks are -noise. Typical size is per term because terms carry units: an `Easting` -coefficient of 1e-6 is real, a `MAP` one of 1e-21 is not. +The band ratio and Spearman skip **negligible terms**, those both runs +put below a millionth of the term's typical size (`is_negligible()`). +Spearman is reported but not gated: it failed mite climate with every +term in band, on the ordering of near-zero terms. -At 100 draws a median is stable relative to the 10–90% band, so a correct -reimplementation should reach close to these. The targets leave a margin -for one run being compared with one run: +Only a full run (every species, 100 draws) is gated. A 100-draw run on +a species subset gets an `indicative_verdict` instead. Neither is +given to a row whose store holds fewer draws than v2 (`min_draws`). -| Comparison | Passes when | -| --- | --- | -| Numerical (iteration 1, where v2 fits once: the mammal hurdle) | Every reachable term within 1e-6 | -| Distributional (v2 bootstraps: plant and bird stages) | At least 90% of reachable terms in band, median standardized difference ≤ 0.25, and median band-width ratio between 0.75 and 1.33 | - -**Why a band-width ratio, and not Spearman.** The in-band and standardized -difference tests both measure against this run's own band, so a harness -noisier than v2 passes them more easily; the ratio fails a band that is too -wide or too narrow, including one read from a store holding too few draws. -Spearman of the medians is still reported (`median_spearman`) but not -gated: it failed mite climate with every term in band, on the ordering of -near-zero terms, and caught nothing the other tests missed. - -Only a full run is gated: every species, 100 draws. A run with 100 draws -on a species subset also gets an `indicative_verdict`, never to be read -as the gate. Either verdict is withheld from a distributional row whose -store holds fewer draws than v2 (`min_draws` in `parity_summary.csv`), -whatever the run was configured for; a store overwritten by a shorter run -would otherwise be scored on five-draw bands. - -**Indicative result (2026-09-30, `parity_check` set, 100 draws).** 18 of -22 rows pass. Mammal habitat matches v2 to ≤1e-14; every plant climate and -habitat row passes, with band-width ratios of 0.88–1.05. Only birds fail: - -- **Birds, all four rows.** The published bird results (December 2025) - were fitted on a `Stratified.Rdata` that was rebuilt in place on - 2026-08-19; the test dataset is built from the rebuilt one. v2's own - code on the test data matches a 2026-08 v2 re-run to 1.5e-12 and the - published files only to 40%, and the harness matches v2's code to - 2.5e-15. See `docs/taxon_quirks.md`. - - **Resolved 2026-10-02.** Two causes, both fixed. The test dataset is - now harmonized from the pre-rebuild `Stratified.Rdata`, kept in - `remote/birds_data_v2/`. The bird spec had also been carrying the - climate prediction with the QPAD offset in it, which v2 does not. - With both fixed, birds reproduce v2's per-draw coefficients to - ~1e-10 and pass. - -Assumptions, stated because they carry the targets: - -- The calibration covers the plant climate stage only; the same targets - are applied to plant habitat and to birds, on the assumption that v2's - self-agreement is similar there. No second v2 run exists to check it. -- The review's strawman asked for a grid-prediction correlation; v2's - reference holds no grid predictions, and the rank correlation of the - coefficient medians that first stood in for it is no longer gated. No - test compares the whole model's predictions. -- The negligible-term fraction (1e-6) and the 90th-percentile typical - size are judgement calls, not calibrated; they only decide which terms - the band ratio and Spearman read. +Assumptions: the calibration covers plant climate only, and is applied +to plant habitat and birds without a second v2 run to check it. No test +compares whole-model predictions. The negligible-term fraction and the +90th-percentile typical size are judgement calls. + +## Status + +**Indicative result, 2026-10-03, `parity_check` set, 100 draws:** all +22 rows pass, 100% of reachable terms in band. Mammal habitat matches +v2 to ≤1e-14; plant band-width ratios are 0.88–1.05. Birds passed after +two fixes on 2026-10-02: the dataset is harmonized from the +pre-rebuild `Stratified.Rdata` in `remote/birds_data_v2/`, and the +carried climate prediction no longer includes the QPAD offset. + +The committed `tables/` and `run_record.md` are from a later 5-draw +trial (2026-10-06) and are not gated. ## Before this gate can be closed -1. **Agree the parity targets above**, or change them in +1. **Agree the parity targets**, or change them in `utils/parity_targets.R`. 2. **Run the full species queues at 100 draws** and commit the report - and record. That run is days of compute (every species of four plant - taxa, mammals and birds); run it on a machine that can be left. -3. **Get references that can be matched exactly.** The harness now - matches v2's own code to numerical precision, but the published lichen, - mite and vascular plant models were fitted on bootstrap draws v2 did - not keep, and the published bird results on a different vintage of - `Stratified.Rdata`. Re-running v2 on the stored draws (as `_setup/03` - did for bryophytes), or finding the bird data those results used, would - make those gates numerical. Until then they stay distributional. - -The review, `docs/reviews/2026-09-28_alignment_review.md`, has the -evidence for each and an ordered plan. - -## Resolved - -- **`CMD` for mammals.** The mammal climate now comes from - `abmi-camera-climate_2023.Rdata` on ABMI-DATA2, the file the v2 - climate pipeline fitted against. `read_camera_climate()` in the - harmonizer joins it as v2 does: strip the year from the site key, - strip the `CMU-` and `NWSAR-` project prefixes, keep one row per - location. It **replaces** the species table's climate block rather - than filling gaps, because the two are different extractions — at - matched deployments they disagree by up to 531 mm of `MAP`. It - matched 4,243 of 4,654 north deployments (91.2%) and all 1,168 - south; unmatched deployments keep `NA`, as v2 excludes them. The - mammal climate stage now fits the full 9-model set, and the climate - covariates all three taxa share went from two to five (`MAP`, `FFP`, - `TD`, `CMD`, `EMT`). -- **A usable bryophyte reference.** The published file failed twice - over: every draw is an error object (`could not find function - "model.avg"`), and the bootstrap-id file holds 3 draws where the - other taxa hold 100. `_setup/03_rerun_bryophyte_v2_reference.R` - re-ran the frozen v2 functions: 134 of 134 species, 100 draws of 19 - climate terms each, no failures. Separately, v2's `COEFS.RData` holds - complete bryophyte arrays, and `v2_results.csv` uses those. -- **The seed.** `boot_seed` in `run.R` seeds every resampled taxon from - one number, with a derived seed per species. v2 is unseeded, so - parity stays distributional, but this side of it now repeats. - -### The plant age splines, scoped - -- Assemble one frame per stand type from units where that type's cover - exceeds 10%, carrying detection, age class, a weight that is cover - times visits, and the site-level IVW prediction as an offset. Grassy - sites join the upland types at age 0.5 and shrubby sites at age 1, - each at half weight. -- Fit `gam(pCount ~ s(sqrt(age), k = 3, m = 2) + offset(p))` per type, - intercept-only where there are 8 or fewer detections, plus two pooled - variants, an all-types model and an intercept-only all-types model — - nine fits. -- Choose among four groupings by summed AIC, corrected for the extra - variance parameters. -- Predict at ages 0.5 and 1–8 with the type's grid effect as offset. -- Converge the cutblock trajectories onto the natural ones for age - classes 2, 3 and 4, then apply the footprint pooling. v2 does it in - that order. - -`select_ivw_grid()` would also need to return the site-level prediction, -which it currently computes only on the grid. + and record. That run is days of compute. +3. **Get references that can be matched exactly.** The published + lichen, mite and vascular plant models were fitted on draws v2 did + not keep. Re-running v2 on the stored draws, as `_setup/03` did for + bryophytes, would make those gates numerical. ## See also -- [`docs/taxon_quirks.md`](../../../docs/taxon_quirks.md) — every - taxon-specific behaviour in the v2 pipelines and the harness's status - against each. -- [`docs/reviews/2026-09-28_alignment_review.md`](../../../docs/reviews/2026-09-28_alignment_review.md) - — the evidence behind the gaps above. +- [`docs/taxon_quirks.md`](../../../docs/taxon_quirks.md): every + taxon-specific v2 behaviour and the harness's status against each. +- [`docs/reviews/2026-09-28_alignment_review.md`](../../../docs/reviews/2026-09-28_alignment_review.md): + the evidence behind the gaps above. diff --git a/1_code/experiments/exp_000_parity_v2/run.R b/1_code/experiments/exp_000_parity_v2/run.R index 198608b..8cb8266 100644 --- a/1_code/experiments/exp_000_parity_v2/run.R +++ b/1_code/experiments/exp_000_parity_v2/run.R @@ -3,529 +3,101 @@ # author: Brendan Casey # created: 2026-09-09 # inputs: -# - 0_data/test_dataset/ -# - 1_code/harness/harness.R -# - 1_code/modules/_shared/plant_group.R -# - 1_code/modules//spec.R -# - utils/focal_species.R +# - the published test dataset (harness/data_source.R) +# - the published v2 results, the ABMIexploreR reference +# (section 1.3; see harness/data_source.R) +# - the framework: 1_code/harness/, methods/, modules/ # outputs: -# in pipeline_dir (2_pipeline/exp_000_parity_v2/ by default): -# - // result stores -# - run_log.csv -# in out_dir (3_output/exp_000_parity_v2/ by default): -# - tables/, figures/, report.md +# in 2_pipeline/exp_000_parity_v2/: +# - // result stores, and run_log.csv +# in 3_output/exp_000_parity_v2/: +# - tables/, figures/, report.md, run_record.md # notes: -# - The validation gate. Runs each taxon's v2 spec through the -# harness and compares the result against the published v2 -# output. -# - The only file to edit. Sections 1.3 to 1.7 are the whole -# configuration: where the run reads and writes, which taxa -# and regions, which focal species, how many draws, and the -# random seed. -# - What each taxon reproduces of v2 is stated once, in its -# spec's `v2_coverage`, and the report prints it from there. -# - Parity here is distributional, not numerical. v2 seeds -# nothing, so v2 does not reproduce against itself; two v2 -# runs give different coefficients. The comparison is -# therefore between bootstrap distributions. See the parity -# ledger in docs/framework_design.md. +# - The validation gate and the baseline for every experiment: +# each taxon's v2 spec against the published v2 output. +# - Section 1.2 is the whole configuration. +# - Parity is mostly distributional (v2 seeds nothing); see the +# parity ledger in docs/framework_design.md. # --- # 1. Setup ---- -## 1.1 Set the working directory ---- -# Every path below is relative to this repository's root. In -# RStudio or Positron, uncomment to set it from this script's -# location. -# setwd(dirname(rstudioapi::getActiveDocumentContext()$path)) -# setwd("../../..") - -## 1.2 Load the harness and the specs ---- -project_root <- normalizePath(getwd(), winslash = "/") - -source(file.path(project_root, "1_code/harness/harness.R")) -load_harness(file.path(project_root, "1_code/harness")) - -# Shared machinery first: the four plant-group modules call it. -source(file.path( - project_root, - "1_code/modules/_shared/plant_group.R" -)) - -for (taxon_dir in c( - "bryophytes", - "lichens", - "soil_mites", - "vascular_plants", - "mammals", - "birds" -)) { - source(file.path( - project_root, - "1_code/modules", - taxon_dir, - "spec.R" - )) -} - -# Translates bird landcover coefficients onto v2's standardized -# habitat terms, so the comparison can join them. -source(file.path( - project_root, - "1_code/modules/birds/standardize.R" -)) - -# v2's mammal hurdle model, which the mammal specs select by -# default. -source(file.path(project_root, "1_code/modules/mammals/hurdle.R")) - -exp_id <- "exp_000_parity_v2" - -exp_code_dir <- file.path( - project_root, - "1_code/experiments", - exp_id -) - -# Named focal-species sets, so run.R chooses rather than lists. -source(file.path(exp_code_dir, "utils/focal_species.R")) - -# The parity targets the comparison reads its verdict against -source(file.path(exp_code_dir, "utils/parity_targets.R")) - -## 1.3 Choose where the run reads and writes ---- -# data_dir is read only. pipeline_dir takes the result stores, -# which are large and gitignored. out_dir takes the committed -# deliverables. Point them anywhere writable. -data_dir <- file.path(project_root, "0_data/test_dataset") +## 1.1 Load the framework ---- +# Run from the repository root +source("1_code/harness/harness.R") +load_framework() + +exp_dir <- "1_code/experiments/exp_000_parity_v2" + +source(file.path(exp_dir, "utils", "parity_targets.R")) + +## 1.2 Configure the run ---- +# See experiment_config() and docs/getting_started.md +config <- experiment_config( + id = "exp_000_parity_v2", + + # Runs, from names(standard_specs()) + taxa = c( + "bryophyte", + "lichen", + "mite", + "vascular_plant", + "mammal_summer", + "mammal_winter", + "bird" + ), -pipeline_dir <- file.path(project_root, "2_pipeline", exp_id) + # A focal_species_sets() name, a vector named by taxon, or NULL + # for every species. Shorten trials by species, not draws, so + # the bands stay honest. + species = "parity_check", -out_dir <- file.path(project_root, "3_output", exp_id) + # 100 is v2's and the only count the gate reads; below ~20 the + # 10-90% bands are unstable. 5 checks the plumbing. + n_bootstraps = 5, -## 1.3a Choose the v2 reference ---- -# Which build of the published v2 coefficients the run is scored -# against. Both write the same schema. -# -# abmiexplorer ABMI's ABMIexploreR package, pinned to a commit; -# 1_code/_setup/08_harmonize_abmiexplorer_results.R. -# The published species only; every taxon. -# drives the v2 project outputs on the network drives; -# 1_code/_setup/05_harmonize_v2_results.R. -# Every species v2 fitted. -v2_reference <- "abmiexplorer" + # Base seed; NULL is unseeded, as v2. Birds replay v2's draws. + seed = 20260909, -v2_reference_dir <- switch( - v2_reference, - abmiexplorer = file.path( - project_root, - "0_data/v2_results/abmiexplorer" - ), - drives = file.path(project_root, "0_data/v2_results"), - stop("v2_reference must be \"abmiexplorer\" or \"drives\".", call. = FALSE) -) + # "spatial_block" draws v2's plant bootstrap afresh; + # "v2_ids" replays v2's stored draws (needs _setup/07). + plant_bootstrap = "spatial_block", -## 1.4 Set the taxa to run ---- -# Each entry names a spec. Comment one out to skip it. -# A key names the run and its folder under pipeline_dir; the -# data slug the spec reads is its own `taxon` field, which is -# what focal_species matches on. Soil mites use the slug "mite". -# -# Mammals run once per season. v2's mammal references average -# the summer and winter fits, so the comparison needs both. -# -# The plant bootstrap. "spatial_block" draws v2's bootstrap -# afresh, so parity is distributional. "v2_ids" replays the draws -# v2 stored, which makes it numerical draw by draw - but only -# against a reference fitted on those draws. The published -# lichen, mite and vascular plant models were not (checked: no -# published draw matches beyond the full-data one), so it pays -# off only for the bryophyte reference _setup/03 regenerated. -# "v2_ids" needs _setup/07 run for every species in the run. -plant_bootstrap <- "spatial_block" + # NULL fits each spec's v2 candidate sets + stage_models = NULL, -specs <- list( - bryophyte = bryophyte_spec(bootstrap = plant_bootstrap), - lichen = lichen_spec(bootstrap = plant_bootstrap), - mite = soil_mite_spec(bootstrap = plant_bootstrap), - vascular_plant = vascular_plant_spec(bootstrap = plant_bootstrap), - mammal_summer = mammal_spec( - season = "summer" - ), - mammal_winter = mammal_spec( - season = "winter" - ), - bird = bird_spec() -) + # Species fitted at once, one R session each + workers = 12, -# Every spec. Drop keys to run fewer. -run_taxa <- c( - "bryophyte", - "lichen", - "mite", - "vascular_plant", - "mammal_summer", - "mammal_winter", - "bird" + # "oob" or "all" write per-unit predictions (GBs for birds) + unit_predictions = "none" ) -## 1.5 Choose the focal species ---- -# Named sets live in utils/focal_species.R, so this stays a -# decision rather than a list of names. See that file for what -# each set is and why its species were chosen. -# -# full every species each spec declares; the parity run -# parity_check two per taxon, chosen for tight bootstrap bands -# plants_only the four plant-group taxa -# one_each one per taxon; the shortest run that touches -# every module -# -# A species vector written inline is also accepted and passes -# straight through, so a one-off does not need a named set: -# -# focal_species <- get_focal_species(c( -# lichen = "Alectoria.sarmentosa", -# mite = "Oppiella.nova" -# )) -# -# Shortening the species list is the right way to shorten a -# trial: it keeps every draw, so the parity bands stay honest. -# Lowering `n_bootstraps` in section 1.6 is cheaper still, but -# it widens those bands and costs you the parity read. -focal_species <- get_focal_species("parity_check") - -# A taxon in run_taxa that the species set does not name runs -# EVERY species, because an absent taxon means "no restriction" -# rather than "none". That is the convention working as intended, -# but it turns a set like plants_only into the longest run in the -# file rather than the shortest, so it is said out loud here. -# Matched on each spec's data slug, not its key, because the two -# mammal keys share the slug "mammal". -run_slugs <- vapply( - specs[intersect(run_taxa, names(specs))], - function(one) one$taxon, - character(1) +## 1.3 The v2 reference ---- +# ABMIexploreR's published coefficients, pinned to a commit and +# harmonized by _setup/08 +config$v2_reference <- "abmiexplorer" +config$v2_reference_dir <- file.path( + v2_results_dir(), "abmiexplorer" ) - -unnamed_taxa <- if (is.null(focal_species)) { - character(0) -} else { - names(run_slugs)[!run_slugs %in% names(focal_species)] -} - -if (length(unnamed_taxa) > 0) { - message( - "Note: ", - paste(unnamed_taxa, collapse = ", "), - " are in run_taxa but not named in focal_species, so every ", - "species will be run for them. Narrow run_taxa to avoid it." - ) -} - -## 1.5b Choose the candidate models ---- -# NULL fits each spec's own v2 candidate set, which is what a -# parity run does. To ask a different question, name a stage and -# hand it a set: either a name from model_sets(), or formulas -# built with extend_models() or models_from_covariates(). -# -# Keys are "taxon.stage", or "stage" to apply to every taxon. -# -# Covariates are not named anywhere. They are read off whichever -# formulas end up being fitted, so changing the models changes -# what is loaded, and a term can never be fitted without its -# column being read. -# -# stage_models <- list( -# # the v2 climate set, plus topography in every candidate -# "climate" = extend_models( -# "climate_plant_v2_full", c("elevation", "slope") -# ), -# # or a set built from covariates alone -# "mite.climate" = models_from_covariates( -# c("MAP", "FFP", "CMD"), form = "ladder" -# ) -# ) -# -# An override is recorded in each run's meta.json, so a run that -# did not fit the v2 sets cannot be mistaken for one that did. -stage_models <- NULL - -## 1.6 Set the run length ---- -# How many bootstrap draws each species gets. Cost is close to -# linear in this number, so 10 draws is roughly a tenth of the -# run, and it is the cheapest way to make a trial affordable. -# -# It is also the one setting that invalidates the parity read. -# The gate scores a term on whether the v2 value falls inside -# this run's 10th-to-90th percentile band, and a band estimated -# from a handful of draws is wide and unstable, so almost -# everything falls inside it and the test stops discriminating. -# Below roughly 20 draws, treat the parity numbers as evidence -# the pipeline ran, not as evidence it agrees with v2. -# -# Shorten the species list instead when you want a cheap run -# whose parity numbers still mean something. See section 1.5. -# -# 100 the v2 value, and the only setting that gates -# 20 a usable band, for a slow-but-real check -# 5 plumbing only -# -# `v2_bootstraps` is v2's draw count, fixed; the run length is -# `n_bootstraps`. A run shorter than v2's, or on a species -# subset, is a trial and is not gated. -v2_bootstraps <- 100L - -n_bootstraps <- 100L - -## 1.6a Validate the run length ---- -# Checked here rather than several hours into a run. -if ( - !is.numeric(n_bootstraps) || - length(n_bootstraps) != 1L || - is.na(n_bootstraps) || - n_bootstraps < 1 || - n_bootstraps != round(n_bootstraps) -) { - stop( - "n_bootstraps must be a single whole number of 1 or more; ", - "got ", - paste(format(n_bootstraps), collapse = ", "), - ".", - call. = FALSE - ) -} - -n_bootstraps <- as.integer(n_bootstraps) - -# Birds resample from stored ids and so have a hard ceiling. -# Every other taxon draws spatial blocks on the fly and has -# none. Counting the columns is cheap; discovering the limit -# from a failed bird run is not. -if ("bird" %in% run_taxa) { - bird_ids_path <- file.path( - data_dir, - "lookup", - "bird_bootstrap_ids.csv" +config$v2_reference_builder <- + "1_code/_setup/08_harmonize_abmiexplorer_results.R" + +# 2. Run ---- +# `fit = FALSE` re-summarizes existing stores without fitting +results <- run_experiment( + config, + fit = TRUE, + # Bird landcover onto v2's habitat template + translate = list(bird = bird_habitat_translation), + steps = list( + compare = script_step(file.path(exp_dir, "01_compare_to_v2.R")), + plot = script_step(file.path(exp_dir, "02_plot_parity.R")), + report = script_step(file.path(exp_dir, "03_build_report.R")) ) - - if (file.exists(bird_ids_path)) { - bird_draws <- length(names( - data.table::fread(bird_ids_path, nrows = 0) - )) - - if (n_bootstraps > bird_draws) { - stop( - "n_bootstraps is ", - n_bootstraps, - " but only ", - bird_draws, - " bird bootstrap draws are stored. Lower ", - "it, or drop \"bird\" from run_taxa.", - call. = FALSE - ) - } - } -} - -if (n_bootstraps < v2_bootstraps) { - message( - "Note: running ", - n_bootstraps, - " of ", - v2_bootstraps, - " v2 bootstrap draws. This is a trial run; its parity ", - "numbers are not a parity read." - ) -} - -iterations <- seq_len(n_bootstraps) - -## 1.6b Set the random seed ---- -# One number governs every random draw in the run. Each species -# resamples under its own seed, derived from this one and its -# name, so species are drawn independently as in v2 and a -# species' draws do not change when the focal set does. -# -# v2 set no seed, so its draws cannot be matched either way; the -# comparison is distributional. Seeding makes this side of it -# repeatable: the same seed and settings give the same stores. -# NULL leaves the draws unseeded, which is what v2 did. -# -# Birds are unaffected. Their draws are the stored v2 ids. -boot_seed <- 20260909L - -## 1.7 Set the stages to run ---- -run_models <- FALSE -run_collect <- FALSE -run_compare <- TRUE -run_plots <- TRUE -run_report <- TRUE - -# Named write_record rather than run_record, which would shadow -# the function it calls. -write_record <- TRUE - -# 2. Fit ---- -# One spec per taxon, each writing a result store per region. - -experiment_start <- Sys.time() -run_log <- NULL - -if (run_models) { - logs <- list() - - for (taxon in run_taxa) { - spec <- specs[[taxon]] - - if (is.null(spec)) { - stop( - "No spec defined for `", - taxon, - "` in section 1.4.", - call. = FALSE - ) - } - - logs[[taxon]] <- run_spec( - spec = spec, - data_dir = data_dir, - run_dir = file.path(pipeline_dir, taxon), - species = focal_species, - iterations = iterations, - stage_models = stage_models, - boot_seed = boot_seed - ) - } - - run_log <- do.call(rbind, logs) - - dir.create(pipeline_dir, recursive = TRUE, showWarnings = FALSE) - data.table::fwrite( - run_log, - file.path(pipeline_dir, "run_log.csv"), - na = "" - ) - - cat("\nRun log:\n") - print(table(run_log$taxon, run_log$status)) -} - -# 3. Summarize ---- -# These read the result stores and write the committed -# deliverables. - -dir.create(out_dir, recursive = TRUE, showWarnings = FALSE) - -## 3.1 Collect model output ---- -if (run_collect) { - source(file.path(exp_code_dir, "01_collect_results.R")) -} - -## 3.2 Compare against the v2.0 reference ---- -if (run_compare) { - source(file.path(exp_code_dir, "02_compare_to_v2.R")) -} - -## 3.3 Plot parity against v2 ---- -# One figure per taxon in figures/, gitignored like the -# per-species tables it is drawn from. -if (run_plots) { - source(file.path(exp_code_dir, "04_plot_parity.R")) -} - -## 3.4 Build the report ---- -if (run_report) { - source(file.path(exp_code_dir, "03_build_report.R")) -} - -## 3.5 Write the run record ---- -# Committed with the report and summary tables. The stores in -# 2_pipeline/ and the per-species tables are gitignored, so when -# the stores are cleared this is what is left to say what -# happened. -# -# The config below is what makes two records comparable, so it -# holds settings rather than paths: an absolute directory -# differs between machines and would diff on every line without -# meaning anything. Volatile values belong in the record's own -# Provenance section, which run_record() writes. -if (write_record) { - stage_engines <- function(field) { - sort(unique(unlist(lapply( - specs[run_taxa], - function(one) { - vapply( - one$stages, - function(st) { - value <- st[[field]] - if (is.function(value)) "custom" else value - }, - character(1) - ) - } - )))) - } - - record_config <- list( - taxa = sort(run_taxa), - focal_species = if (is.null(focal_species)) { - "all" - } else { - sort(paste0(names(focal_species), "=", focal_species)) - }, - species_n = if (is.null(focal_species)) { - NA_integer_ - } else { - length(focal_species) - }, - n_bootstraps = n_bootstraps, - v2_bootstraps = v2_bootstraps, - boot_seed = if (is.null(boot_seed)) "unseeded" else boot_seed, - plant_bootstrap = plant_bootstrap, - stage_models = if (is.null(stage_models)) { - "spec defaults (v2 candidate sets)" - } else { - sort(names(stage_models)) - }, - engines = stage_engines("engine"), - selection = stage_engines("selection"), - data_dir = basename(data_dir) - ) - - # Failures belong in the record. Coverage says what was - # produced; without this, a run that lost half its jobs and - # one that lost none look alike wherever the survivors agree. - if (!is.null(run_log)) { - ok_n <- sum(run_log$status == "ok", na.rm = TRUE) - - record_config$jobs_total <- as.integer(nrow(run_log)) - record_config$jobs_ok <- as.integer(ok_n) - record_config$jobs_not_ok <- as.integer(nrow(run_log) - ok_n) - } - - record_file <- run_record( - pipeline_dir = pipeline_dir, - out_file = file.path(out_dir, "run_record.md"), - exp_id = exp_id, - config = record_config - ) - - cat("Run record: ", record_file, "\n", sep = "") -} - -# 4. Experiment complete ---- -experiment_time <- format(round(Sys.time() - experiment_start, 1)) -cat("\n========================================\n") -cat("EXPERIMENT ", exp_id, " COMPLETE\n", sep = "") -cat("Total run time: ", experiment_time, "\n", sep = "") -cat("========================================\n") -cat("Intermediates are in ", pipeline_dir, "\n", sep = "") -cat("Deliverables are in ", out_dir, "\n", sep = "") -cat( - "Committed: run_record.md, report.md and the summary ", - "tables. Commit them only after a full run.", - "\n", - sep = "" ) +# Commit run_record.md, report.md and the small summary tables +# only after a full run: every species, n_bootstraps = 100. + # End of script ---- diff --git a/1_code/experiments/exp_000_parity_v2/utils/parity_targets.R b/1_code/experiments/exp_000_parity_v2/utils/parity_targets.R index 4263fbf..ff0758a 100644 --- a/1_code/experiments/exp_000_parity_v2/utils/parity_targets.R +++ b/1_code/experiments/exp_000_parity_v2/utils/parity_targets.R @@ -30,11 +30,10 @@ # easily. The ratio closes that: too wide or too narrow a band # fails, including one read from a store holding too few # draws. -# - Spearman of the medians is reported but no longer gated. -# Model averaging shrinks unsupported terms towards zero in -# both runs, and ranking values of 1e-30 against 1e-40 ranks -# noise: it failed the mite climate stage with every term in -# band. It caught nothing the other tests missed. +# - Spearman of the medians is reported, not gated: averaging +# shrinks unsupported terms to ~1e-30 in both runs, so it +# ranks noise (it failed the mite climate stage with every +# term in band). # - The calibration (bryophyte climate, two v2 runs of the same # code): 100% in band, median standardized difference 0.019; # per-species median band ratio 1.00, 5th to 95th percentile @@ -164,7 +163,9 @@ parity_verdict <- function( return("no comparison") } - return(if (max_absolute_difference <= targets$numerical_tolerance) { + return(if ( + max_absolute_difference <= targets$numerical_tolerance + ) { "pass" } else { "fail" diff --git a/1_code/experiments/exp_000_parity_v2/v2_self_agreement.R b/1_code/experiments/exp_000_parity_v2/v2_self_agreement.R index 51359a1..7113bae 100644 --- a/1_code/experiments/exp_000_parity_v2/v2_self_agreement.R +++ b/1_code/experiments/exp_000_parity_v2/v2_self_agreement.R @@ -5,8 +5,10 @@ # inputs: # - 2_pipeline/v2_reference/bryophyte-species-models.Rdata, # from 1_code/_setup/03_rerun_bryophyte_v2_reference.R -# - 0_data/v2_results/v2_results.csv, the bryophyte climate -# rows, which come from v2's COEFS.RData +# - abmiexplorer/v2_results.csv from the published v2 results +# (see 1_code/harness/data_source.R), the bryophyte climate +# rows, from _setup/08. These are v2's original run: they +# match v2's COEFS.RData to ~1e-15 # outputs: # in 3_output/exp_000_parity_v2/tables/: # - v2_self_agreement.csv, per species and term @@ -19,26 +21,26 @@ # closely than v2 agrees with itself would fail a perfect # reimplementation. # - The two runs are the bryophyte climate stage: v2's original -# run, as published in COEFS.RData, and the rerun -# _setup/03 made with the frozen v2 functions. Same code, same -# data, different unseeded draws; 134 species by 19 terms by -# 100 draws each. -# - Scored exactly as 02_compare_to_v2.R scores the harness, in +# run, as ABMIexploreR publishes it, and the rerun _setup/03 +# made with the frozen v2 functions. Same code, same data, +# different unseeded draws; 19 terms by 100 draws for each +# species both hold. Bryum.All is not published, so it +# drops out (133 of the rerun's 134 species). +# - Scored exactly as 01_compare_to_v2.R scores the harness, in # both directions: is one run's median inside the other's # 10th-to-90th percentile band, and how far apart are the # medians relative to half that band. The two directions are # reported separately and averaged. -# - Also the Spearman correlation of the medians across terms, -# per species, because the terms span twenty orders of -# magnitude and a Pearson correlation would read only the -# intercept. Reported, no longer gated. +# - Also the per-species Spearman of the medians (terms span +# twenty orders of magnitude, so not Pearson). Reported, not +# gated. # - And the band-width ratio, the rerun's 10-90% band over the # published run's, as a median per species. Its 5th to 95th # percentile over species sets the gate's band_ratio_range. # - Climate only: the regenerated reference has no habitat # stage. The rate is assumed to carry to the plant habitat # stage; that is an assumption, stated in the README. -# - Run once, from the repository root, after _setup/03 and 05. +# - Run once, from the repository root, after _setup/03 and 08. # Not part of run.R's sequence. # --- @@ -59,8 +61,9 @@ rerun_path <- file.path( project_root, "2_pipeline", "v2_reference", "bryophyte-species-models.Rdata" ) +source(file.path(project_root, "1_code/harness/data_source.R")) results_path <- file.path( - project_root, "0_data", "v2_results", "v2_results.csv" + v2_results_dir(), "abmiexplorer", "v2_results.csv" ) tables_dir <- file.path( project_root, "3_output", "exp_000_parity_v2", "tables" @@ -98,7 +101,13 @@ rerun_summary <- rbindlist(lapply(names(rerun), function(species) { species = species, term = term, rerun_median = stats::median(values), rerun_p10 = stats::quantile(values, 0.1, names = FALSE), - rerun_p90 = stats::quantile(values, 0.9, names = FALSE) + rerun_p90 = stats::quantile(values, 0.9, names = FALSE), + rerun_mean = mean(values), + rerun_sd = if (length(values) < 2) { + NA_real_ + } else { + stats::sd(values) + } ) })) })) diff --git a/1_code/experiments/exp_001_xgboost/README.md b/1_code/experiments/exp_001_xgboost/README.md new file mode 100644 index 0000000..4f5afe9 --- /dev/null +++ b/1_code/experiments/exp_001_xgboost/README.md @@ -0,0 +1,136 @@ +# exp_001_xgboost + +![Status](https://img.shields.io/badge/Status-Test-lightgrey) +![Languages](https://img.shields.io/badge/Languages-R-blue) + +## Question + +Does a boosted regression tree fit the habitat stage better than v2's linear +models, for every taxon: the plant-group taxa, mammals and birds? + +This is a **test experiment**: its first job is to show that a method with +no coefficients runs through the pipeline for every taxon and is compared +with the v2 baseline. Read its numbers as a result only after a run at 100 +draws with tuned settings. + +## Design + +| | exp_000 (baseline) | exp_001 (this) | +| --- | --- | --- | +| Climate stage | v2: GLM candidates, AICc model averaging (mammals: v2's precomputed prediction) | The same | +| Habitat stage (each spec's `habitat_stage`; birds: landcover) | v2: GLM candidates; IVW averaging (plants), staged BIC (birds), or the hurdle with best-by-AICc halves (mammals); v2's post-processing | `xgboost` with the `single` rule, on every covariate v2's habitat candidates use, `Climate` included | +| Climate → habitat | Carried as `Climate` | The same | +| Species, draws, seed | `parity_check`, 20260909 | The same | + +- **What changes:** the habitat stage's engine, selection rule and + candidate set. `run.R` makes the change with one call for every run: + `lapply(standard_specs(), replace_stage_method, engine = "xgboost", + control = xgb_settings)`. +- **Why one formula:** a tree chooses its own variables, splits and + interactions, so it is given every candidate covariate once (as main + effects) rather than v2's competing formulas. +- **Mammals.** The habitat stage is the generic `hurdle` rule. Its + structure stays: presence on every deployment, on v2's lure-scaled + response (`binary:logistic`, which takes a proportion), then abundance + given presence on the deployments with detections (`reg:gamma`, a + log-link Gamma), unweighted and without `Climate`, as in v2. Boosted + trees fit both halves. The hurdle writes `habitat_presence`, + `habitat_abundance` and `habitat_total` (their product) as grid + predictions, the same tables v2 publishes. +- **What is dropped:** + - v2's post-processing of habitat coefficients: the plant stand-age + splines, cutblock convergence, pAspen and footprint pooling, and v2's + plant validation AUCs; + - the mammal one-hot tables, calibration, stand-age splines, Mule Deer + adjustment and cutblock convergence. + + These all read coefficients a tree does not have. The metrics score the + tree's own prediction: presence, for mammals. + +## How to run + +From the repository root: + +```r +source("1_code/experiments/exp_001_xgboost/run.R") +``` + +It reads the published test dataset and exp_000's +`3_output/exp_000_parity_v2/tables/`, and writes stores to +`2_pipeline/exp_001_xgboost/` and summaries to +`3_output/exp_001_xgboost/`. + +`run.R` uses 5 draws to check the pipeline. For a result, set +`n_bootstraps = 100`, and make sure the exp_000 tables it is compared +against are from a 100-draw run of the same species. + +## How to read the result + +`3_output/exp_001_xgboost/tables/`: + +| File | Read it for | +| --- | --- | +| `comparison_metrics.csv` | Per species, both runs' out-of-bag metrics and 10–90% bands | + +Out-of-bag metrics score each draw on the survey units it left out, which +matters here: a tree fits the units it trained on far more closely than a +GLM does, so in-sample metrics would flatter it. + +## Assumptions and caveats + +- **The xgboost settings are untuned.** At most 1000 rounds, learning + rate 0.05, depth 3, subsample 0.5, minimum child weight 1 (`run.R` + section 1.2). The number of rounds is chosen by early stopping on a + seeded 20% holdout of each draw, then the model is refitted on all of + the draw. A fair comparison tunes + these, ideally per taxon. +- **The grid compares like with like only roughly.** The grid holds each + habitat type as a pure stand at `Climate` 0. A GLM extrapolates to that + linearly; a tree holds the value of its nearest split. Where a type is + rare at the survey units, the two can differ for that reason alone. +- **The baseline includes v2's post-processing** (stand-age splines, + footprint pooling, and for mammals calibration and convergence); this + run does not. A difference in the aged, cutblock or footprint habitat + types partly reflects that. +- **Mammal grid predictions are presence**, as in exp_000. Abundance and + total are compared through the `habitat_abundance` and `habitat_total` + coefficient summaries. Note that exp_000's mammal tables are v2's + calibrated tables, so they are not like for like with this run's + uncalibrated grid predictions. +- **Bird counts keep the QPAD offset**, which the engine passes as the + base margin in fitting and prediction. +- **gbm was replaced.** The first version of this experiment + (`exp_001_gbm`, 2026-10-03) used `gbm`, which cannot fit the mammal + hurdle: its bernoulli rejects a 0–1 proportion and it has no Gamma + family. + +## Result + +_Not yet run at 100 draws._ + +**Pipeline check, 2026-10-05: 5 draws, `parity_check` species.** All 160 +species × region × draw jobs ran, mammals included. Against the exp_000 +tables (whose draw counts match): + +| Taxon | Out-of-bag AUC, north (v2 → xgboost) | South (v2 → xgboost) | +| --- | --- | --- | +| Birds | 0.729 → 0.769 | 0.702 → 0.745 | +| Bryophytes | 0.650 → 0.681 | 0.669 → 0.754 | +| Lichens | 0.803 → 0.827 | 0.792 → 0.860 | +| Mammals, summer | 0.704 → 0.727 | 0.660 → 0.684 | +| Mammals, winter | 0.741 → 0.768 | 0.710 → 0.725 | +| Soil mites | 0.716 → 0.739 | 0.774 → 0.811 | +| Vascular plants | 0.828 → 0.869 | 0.846 → 0.905 | + +xgboost's out-of-bag AUC was higher in every taxon, region and season. +Except for birds, and soil mites in the south, its calibration slopes were +further from 1 than v2's. +Mammal slopes fell from 0.65–0.85 to 0.53–0.72, so the untuned trees are +overconfident there. The median rank correlation of the habitat grid +effects with v2's was 0.28–0.73 by taxon and region, but about 0 for +soil mites in the north. Mammals were 0.69 in the north and 0.68 in the +south. + +**Do not read these as the finding.** Five draws give unstable bands, the +settings are untuned, and two species per taxon is a small sample. They +show the comparison works end to end, for every taxon. diff --git a/1_code/experiments/exp_001_xgboost/run.R b/1_code/experiments/exp_001_xgboost/run.R new file mode 100644 index 0000000..c5c3bdb --- /dev/null +++ b/1_code/experiments/exp_001_xgboost/run.R @@ -0,0 +1,62 @@ +# --- +# title: Run Experiment 001 - Boosted Regression Trees for Habitat +# author: Brendan Casey +# created: 2026-10-03 +# inputs: +# - the published test dataset (harness/data_source.R) +# - 3_output/exp_000_parity_v2/tables/, the v2 baseline +# outputs: +# in 2_pipeline/exp_001_xgboost/: +# - // result stores, and run_log.csv +# in 3_output/exp_001_xgboost/: +# - tables/ (summaries, and comparison_metrics.csv against +# exp_000), run_record.md +# notes: +# - Does xgboost fit the habitat stage better than v2's GLMs? +# See README.md. Only each spec's `habitat_stage` changes +# (via replace_stage_method()); v2's climate stage is kept +# and carried as `Climate`. For mammals, xgboost fits both +# hurdle halves. +# - Settings are untuned (see methods/engines/xgboost.R): this +# tests the pipeline, not the method's best. +# --- + +# 1. Setup ---- + +## 1.1 Load the framework ---- +source("1_code/harness/harness.R") +load_framework() + +## 1.2 The change: fit the habitat stage with xgboost ---- +xgb_settings <- list( + nrounds = 1000, # the most rounds; early stopping picks + eta = 0.05, # learning rate + max_depth = 3, # tree complexity + subsample = 0.5, # share of rows each tree sees + min_child_weight = 1 # smallest leaf, in weighted rows +) + +# One call per run: each spec names its own habitat stage +specs <- lapply( + standard_specs(), replace_stage_method, + engine = "xgboost", control = xgb_settings +) + +## 1.3 Configure the run ---- +# Same species and seed as exp_000 +config <- experiment_config( + id = "exp_001_xgboost", + taxa = names(specs), + species = "parity_check", + # 5 checks the pipeline runs; 100 for a result to read + n_bootstraps = 5, + seed = 20260909, + specs = specs, + workers = 12 +) + +# 2. Run ---- +# Fit, summarize, and compare with exp_000 +results <- run_experiment(config) + +# End of script ---- diff --git a/1_code/experiments/exp_002_soilgrids/01_extract_soilgrids_covariates.R b/1_code/experiments/exp_002_soilgrids/01_extract_soilgrids_covariates.R new file mode 100644 index 0000000..ca9e776 --- /dev/null +++ b/1_code/experiments/exp_002_soilgrids/01_extract_soilgrids_covariates.R @@ -0,0 +1,135 @@ +# --- +# title: Extract SoilGrids Covariates for Experiment 002 +# author: Brendan Casey +# created: 2026-10-04 +# inputs: +# - SoilGrids 2.0, ABMI 1 km variant, from the sciSpatialR +# catalogue (geoscientificInformation/soilgrids_250_v2_ab/ +# abmi1km), on //ABMI-DATA2; SDM_SOILGRIDS overrides the path +# - sites.csv from the published test dataset (see +# 1_code/harness/data_source.R), read only +# outputs: +# in 2_pipeline/exp_002_soilgrids/inputs/: +# - soilgrids_0_5cm.csv: survey_unit_id plus ten +# sg_*_0_5cm columns, one row per survey unit +# - soilgrids_0_5cm_source.csv: where the columns came from +# notes: +# - Written to 2_pipeline/, not the frozen test dataset; run.R +# passes the file as `covariate_files`. +# - All ten SoilGrids properties published at 0-5 cm (organic +# carbon stock is 0-30 cm only, so excluded), in the layer's +# units. Gap-filled cells are interpolations (see the layer's +# readme). +# - Each unit takes its 1 km cell's value, located by easting +# and northing in EPSG:3400 (the layer's grid). Checked +# 2026-10-04: plant units match projected lat/long to 1e-8 m; +# 98% of v2's south-only bird units fall in the Grassland and +# Parkland regions. Mammals have no coordinates, so NA. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # reading and writing tables (version: 1.18.0) +library(sciSpatialR) # catalogue access, extraction (version: 0.1.0) +library(sf) # survey unit points (version: 1.1.2) + +## 1.2 Resolve paths ---- +project_root <- normalizePath(getwd(), winslash = "/") +resolver <- file.path(project_root, "1_code/harness/data_source.R") + +if (!file.exists(resolver)) { + stop("Run this from the repository root.", call. = FALSE) +} + +source(resolver) +sites_path <- file.path(test_dataset_dir(), "sites.csv") +out_dir <- file.path( + project_root, "2_pipeline", "exp_002_soilgrids", "inputs" +) + +layer_id <- "geoscientificInformation/soilgrids_250_v2_ab/abmi1km" +layer_path <- Sys.getenv("SDM_SOILGRIDS", unset = "") + +dir.create(out_dir, recursive = TRUE, showWarnings = FALSE) + +# 2. Read the layer's shallowest depth ---- +soil <- if (nzchar(layer_path)) { + terra::rast(layer_path) +} else { + get_layer(layer_id) +} + +shallow <- grep("_0-5cm_mean$", names(soil), value = TRUE) + +if (length(shallow) != 10) { + stop( + "Expected ten 0-5 cm bands in ", layer_id, "; found ", + length(shallow), ".", + call. = FALSE + ) +} + +soil <- soil[[shallow]] + +# Column names the formulas can use: sg_clay_0_5cm +names(soil) <- paste0( + "sg_", sub("_0-5cm_mean$", "", shallow), "_0_5cm" +) +soil_columns <- names(soil) + +# 3. Extract at every survey unit ---- +sites <- fread( + sites_path, + select = c("survey_unit_id", "easting", "northing"), + colClasses = list(character = "survey_unit_id") +) +located <- sites[!is.na(easting) & !is.na(northing)] + +points <- st_as_sf( + located, coords = c("easting", "northing"), crs = 3400 +) + +values <- as.data.table( + extract_points(soil, points, bind = FALSE) +) +values <- values[, soil_columns, with = FALSE] + +# One row per survey unit, NA where a unit has no location +soil_table <- data.table(survey_unit_id = sites$survey_unit_id) +rows <- match(sites$survey_unit_id, located$survey_unit_id) + +for (column in soil_columns) { + set(soil_table, j = column, value = values[[column]][rows]) +} + +## 3.1 Report ---- +cat( + "Survey units:", nrow(sites), "\n", + " located:", nrow(located), "\n", + " without coordinates (soil left NA):", + nrow(sites) - nrow(located), "\n", + " located but outside the layer:", + sum(!stats::complete.cases(values)), "\n" +) +print(summary(values)) + +# 4. Write ---- +fwrite(soil_table, file.path(out_dir, "soilgrids_0_5cm.csv"), + na = "") + +fwrite(data.table( + columns = paste(soil_columns, collapse = " "), + source = layer_id, + doi = paste0( + "10.17027/isric-soilgrids.", + "713396fa-1687-11ea-a7c0-a0481ca9e724" + ), + depth = "0-5 cm", + extracted = format(Sys.Date()), + method = "value of the 1 km cell each survey unit falls in" +), file.path(out_dir, "soilgrids_0_5cm_source.csv")) + +cat("Wrote", nrow(soil_table), "survey units to", out_dir, "\n") + +# End of script ---- diff --git a/1_code/experiments/exp_002_soilgrids/README.md b/1_code/experiments/exp_002_soilgrids/README.md new file mode 100644 index 0000000..8c4d393 --- /dev/null +++ b/1_code/experiments/exp_002_soilgrids/README.md @@ -0,0 +1,131 @@ +# exp_002_soilgrids + +![Status](https://img.shields.io/badge/Status-Test-lightgrey) +![Languages](https://img.shields.io/badge/Languages-R-blue) + +## Question + +Does adding near-surface soil properties from SoilGrids 2.0 improve the +models, for the plant-group taxa and birds? + +This is a **test experiment**: its first job is to show that a covariate set +from the Science Centre's spatial catalogue +([sciSpatialR](https://github.com/ABbiodiversity/sciSpatialR)) can be tested +against the v2 baseline without changing the frozen dataset. Read its numbers as a +result only after a run at 100 draws. + +## The covariates + +SoilGrids 2.0 (Poggio et al. 2021, SOIL 7: 217–240; +doi:10.17027/isric-soilgrids.713396fa-1687-11ea-a7c0-a0481ca9e724), the ABMI +1 km variant in the catalogue (`geoscientificInformation/soilgrids_250_v2_ab/ +abmi1km`), at its shallowest depth, 0–5 cm. + +| Column | Property | Units | In the models | +| --- | --- | --- | --- | +| `sg_bdod_0_5cm` | Bulk density | kg/dm3 | yes | +| `sg_cec_0_5cm` | Cation exchange capacity | cmol(c)/kg | yes | +| `sg_cfvo_0_5cm` | Coarse fragments | vol % | yes | +| `sg_clay_0_5cm` | Clay | % | yes | +| `sg_sand_0_5cm` | Sand | % | yes | +| `sg_silt_0_5cm` | Silt | % | **no**: sand, silt and clay sum to 100%, so one is fixed by the other two | +| `sg_nitrogen_0_5cm` | Nitrogen | g/kg | yes | +| `sg_phh2o_0_5cm` | pH in water | pH | yes | +| `sg_soc_0_5cm` | Soil organic carbon | g/kg | yes | +| `sg_ocd_0_5cm` | Organic carbon density | kg/m3 | yes | + +They are extracted by `01_extract_soilgrids_covariates.R` in this folder, +which takes the value of the 1 km cell each survey unit falls in and writes +`2_pipeline/exp_002_soilgrids/inputs/soilgrids_0_5cm.csv` (one row per survey +unit) with its source in `soilgrids_0_5cm_source.csv`. `run.R` names that file +in `experiment_config(covariate_files = )`, and the harness joins it on +`survey_unit_id` when it loads the data. `0_data/` and `_setup/` are not +touched. Organic carbon stock is published for 0–30 cm only and is not +included. + +## Design + +| | exp_000 (baseline) | exp_002 (this) | +| --- | --- | --- | +| Climate stage | v2's candidates (plants 58, birds 25), model-averaged | The same candidates, each with the nine soil terms added | +| Climate → habitat | Carried as `Climate` | The same; the carried prediction now includes soil | +| Habitat stage | v2 | v2, unchanged | +| Species, draws, seed | `parity_check`, 20260909 | The same | + +- **Why the climate stage.** It is fitted once, province-wide, so one set of + formulas serves every region, and it is v2's broad-scale environmental + stage. Adding soil to the habitat stage instead would need different + formulas per region and a soil value for every row of the habitat grids. +- **Every candidate gets the terms** (`extend_models()`), so the comparison + between candidates is v2's and the only change is the soil. +- **Mammals are not run.** Their v2 climate is a precomputed prediction rather + than a fitted stage, and their deployments carry no coordinates in the + dataset, so they have no soil values. + +## How to run + +From the repository root: + +```r +source("1_code/experiments/exp_002_soilgrids/run.R") +``` + +`run.R` runs the extraction first if the soil file is missing. That step +reads the layer from the `//ABMI-DATA2` share; set `SDM_SOILGRIDS` to a local +copy of the raster to read it from there instead. Delete the file to extract +again. + +It also reads the published test dataset and exp_000's +`3_output/exp_000_parity_v2/tables/`, and writes stores to +`2_pipeline/exp_002_soilgrids/` and summaries to +`3_output/exp_002_soilgrids/`. + +`run.R` uses 5 draws to check the pipeline. For a result, set +`n_bootstraps = 100` and compare against a 100-draw exp_000. + +## How to read the result + +Every experiment is compared with exp_000; see +`3_output/exp_002_soilgrids/tables/comparison_metrics.csv`. + +## Assumptions and caveats + +- **A 1 km cell is not a plot.** Each survey unit takes the mean of its 1 km + cell, aggregated from SoilGrids' 250 m predictions, which are themselves + model predictions, not measurements. Gap-filled cells are interpolated. +- **Some soil terms are correlated**: bulk density with coarse fragments + (r = −0.83) and pH (0.79), sand with clay (−0.80), nitrogen with organic + carbon (0.75). Model averaging spreads weight across them, so individual + soil coefficients are hard to interpret; the fit metrics and grid + predictions are the reliable read. +- **Raw units, unscaled.** As in v2's climate stage; `bayesglm` scales + predictors internally for its priors. +- **Twelve plant-group survey units have no soil value** (just outside the + 1 km grid), so the climate stage fits on those 12 fewer units than + exp_000. No bird unit is missing. +- **More parameters.** Each candidate gains nine terms; the AICc weights + penalize them, but the averaged model is larger than v2's. + +## Result + +_Not yet run at 100 draws._ + +**Pipeline check, 2026-10-04: 5 draws, `parity_check` species.** All 100 +species × region × draw jobs ran. Against exp_000 at the same species, seed +and draws: + +| Taxon | Out-of-bag AUC, north (v2 → soil) | South (v2 → soil) | +| --- | --- | --- | +| Birds | 0.729 → 0.732 | 0.702 → 0.715 | +| Bryophytes | 0.650 → 0.657 | 0.669 → 0.664 | +| Lichens | 0.803 → 0.803 | 0.792 → 0.811 | +| Soil mites | 0.716 → 0.716 | 0.774 → 0.787 | +| Vascular plants | 0.828 → 0.824 | 0.846 → 0.842 | + +The differences are small and mixed in sign, and calibration slopes moved +slightly away from 1 for most taxa. Habitat effects on the grid barely moved +(rank correlation with exp_000 of 0.86–1.00 by species). + +**Do not read these as the finding.** Five draws give unstable bands and two +species per taxon is a small sample. They show the covariates load and the +comparison works end to end. diff --git a/1_code/experiments/exp_002_soilgrids/run.R b/1_code/experiments/exp_002_soilgrids/run.R new file mode 100644 index 0000000..367d145 --- /dev/null +++ b/1_code/experiments/exp_002_soilgrids/run.R @@ -0,0 +1,92 @@ +# --- +# title: Run Experiment 002 - SoilGrids Terms in the Climate Stage +# author: Brendan Casey +# created: 2026-10-04 +# inputs: +# - the published test dataset, the frozen v2 data, unchanged +# - 2_pipeline/exp_002_soilgrids/inputs/soilgrids_0_5cm.csv, the +# soil covariates, written by 01_extract_soilgrids_covariates.R +# in this folder (run here when the file is missing) +# - 3_output/exp_000_parity_v2/tables/, the v2 baseline +# outputs: +# in 2_pipeline/exp_002_soilgrids/: +# - // result stores, and run_log.csv +# in 3_output/exp_002_soilgrids/: +# - tables/ (summaries, and comparison_metrics.csv against +# exp_000), run_record.md +# notes: +# - Does adding 0-5 cm SoilGrids properties to every climate +# candidate improve the models? See README.md. Soil columns +# come from this experiment's file via `covariate_files`. +# - Mammals are not run: their v2 climate is precomputed, and +# their deployments have no coordinates. +# --- + +# 1. Setup ---- + +## 1.1 Load the framework ---- +source("1_code/harness/harness.R") +load_framework() + +## 1.2 The soil covariates ---- +# Extracted once from the sciSpatialR catalogue; needs the +# network share. Delete the file to extract again. +soil_file <- "2_pipeline/exp_002_soilgrids/inputs/soilgrids_0_5cm.csv" + +if (!file.exists(soil_file)) { + source(file.path( + "1_code", "experiments", "exp_002_soilgrids", + "01_extract_soilgrids_covariates.R" + )) +} + +## 1.3 The change: soil terms in every climate candidate ---- +# The 0-5 cm SoilGrids properties. Silt is left out: sand, silt +# and clay sum to 100%. +soil_terms <- c( + "sg_bdod_0_5cm", # bulk density, kg/dm3 + "sg_cec_0_5cm", # cation exchange capacity, cmol(c)/kg + "sg_cfvo_0_5cm", # coarse fragments, vol % + "sg_clay_0_5cm", # clay, % + "sg_sand_0_5cm", # sand, % + "sg_nitrogen_0_5cm", # nitrogen, g/kg + "sg_phh2o_0_5cm", # pH in water + "sg_soc_0_5cm", # soil organic carbon, g/kg + "sg_ocd_0_5cm" # organic carbon density, kg/m3 +) + +plant_taxa <- c("bryophyte", "lichen", "mite", "vascular_plant") + +# Keys are "taxon.stage" +stage_models <- c( + stats::setNames( + rep(list(extend_models("climate_plant_v2_full", soil_terms)), + length(plant_taxa)), + paste0(plant_taxa, ".climate") + ), + list(bird.climate = extend_models("climate_bird_v2", soil_terms)) +) + +## 1.4 Configure the run ---- +# Same species and seed as exp_000 +config <- experiment_config( + id = "exp_002_soilgrids", + taxa = c(plant_taxa, "bird"), + species = "parity_check", + # 5 checks the pipeline runs; 100 for a result to read + n_bootstraps = 5, + seed = 20260909, + stage_models = stage_models, + # Where the soil columns named in stage_models are read from + covariate_files = soil_file, + workers = 12 +) + +# 2. Run ---- +# Fit, summarize, and compare with exp_000 +results <- run_experiment( + config, + translate = list(bird = bird_habitat_translation) +) + +# End of script ---- diff --git a/1_code/harness/.gitkeep b/1_code/harness/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/1_code/harness/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/1_code/harness/README.md b/1_code/harness/README.md new file mode 100644 index 0000000..a75af9d --- /dev/null +++ b/1_code/harness/README.md @@ -0,0 +1,123 @@ +# harness + +The shared pipeline. It names no taxon and no method: taxon modules +supply specs, `methods/` supplies what the specs name, and the harness +runs them. A change here affects every experiment, past and future; +review it as such and check it with [`1_code/tests/`](../tests/). + +Load everything with: + +```r +source("1_code/harness/harness.R") +load_framework() # harness/, methods/, modules/, experiments/_shared/ +``` + +`new_experiment.R` is the exception: it is base R and runs without +`load_framework()`. + +## Files + +| File | Holds | +| --- | --- | +| `harness.R` | `load_framework()`: sources the harness, methods, modules and `experiments/_shared/`, in that order | +| `experiment.R` | `experiment_config()`, `run_experiment()`, `script_step()` | +| `new_experiment.R` | `new_experiment()`: scaffold an experiment's three folders | +| `data_source.R` | The pinned published dataset versions; `test_dataset_dir()`, `v2_results_dir()`, `verify_published()` | +| `data_load.R` | `lookup/dataset_manifest.csv`, loaders, `covariate_files` joins, model frames | +| `cache.R` | Session cache for dataset reads | +| `covariate_sets.R` | Named covariate sets, derived columns, term renaming | +| `model_sets.R` | v2 candidate formulas, spliced verbatim; `extend_models()`, `models_from_covariates()` | +| `spec.R` | `validate_spec()`, `apply_stage_models()`, `replace_stage_method()` | +| `registry.R` | `register_engine()` and the other `register_*()`; `list_methods()` | +| `engines.R` | Engine interface and `check_engine()` | +| `selection.R` | Plumbing for selection rules: `fit_candidates()`, `combined_predictor()`, `selection_result()` | +| `resample.R` | Per-species seeds, `with_seed()`, spatial blocks | +| `species.R` | Species lists and focal-species resolution | +| `predict_grid.R` | Projects a final model onto a habitat grid | +| `eval_metrics.R` | Scores a prediction with every registered metric | +| `run_model.R` | `run_specs()`: prepare each spec, fit species in parallel, merge shards | +| `result.R` | The result store: writes and reads its files | +| `summarise.R` | `collect_results()`: stores to summary tables | +| `compare.R` | `compare_experiments()`: one experiment against another | +| `run_record.R` | `run_record()`: writes `run_record.md` | + +## What `run_experiment()` does + +1. **Configure.** `experiment_config()` checks every decision before + anything is fitted. +2. **Fit.** Every spec is validated and each region's data and grid + are loaded once. Each species is one job; with `workers` above 1, + jobs run in parallel R sessions. Each species draws under its own + seed, so results do not depend on the worker count. A failed + species or draw is logged and the run continues. +3. **Store.** Shards are merged into one store per run and region in + `2_pipeline//`, plus `run_log.csv`. +4. **Summarize.** `collect_results()` writes `3_output//tables/`. +5. **Compare.** Every experiment except the baseline writes + `tables/comparison_metrics.csv` against exp_000. +6. **Steps.** The experiment's own `steps`, in order. +7. **Record.** `run_record.md`. + +`run_experiment(config, fit = FALSE)` re-summarizes existing stores. +Nothing outside `2_pipeline//` and `3_output//` is written. + +## `experiment_config()` arguments + +| Argument | Default | Sets | +| --- | --- | --- | +| `id` | | Folder name under `2_pipeline/` and `3_output/` | +| `taxa` | all | Runs, from `names(standard_specs())` | +| `species` | every species | A `focal_species_sets()` name, or a vector named by taxon | +| `n_bootstraps` | `100` | Draws per species; v2 used 100. 5 checks a run works | +| `seed` | `20260909` | Base seed; `NULL` leaves draws unseeded, as v2 did | +| `stage_models` | v2's | Replacement candidate sets, keyed `stage` or `taxon.stage` | +| `covariate_files` | none | CSVs of extra covariates keyed on `survey_unit_id` | +| `specs` | `standard_specs()` | Changed specs, such as a different engine or rule | +| `plant_bootstrap` | `"spatial_block"` | `"v2_ids"` replays v2's stored plant draws | +| `workers` | `1` | Species fitted at once | +| `unit_predictions` | `"none"` | `"oob"` or `"all"` writes per-unit predictions | +| `baseline` | `"exp_000_parity_v2"` | Experiment to compare with; `NULL` skips | +| `v2_bootstraps` | `100` | Draws a run needs to be gated | +| `project_root`, `data_dir`, `pipeline_dir`, `out_dir` | derived | Path overrides | + +## Outputs + +A *draw* is one bootstrap iteration; draw 1 is the full data. Every +summary reduces draws to `median`, `p10` and `p90` (what comparisons +read), plus `mean`, `sd`, `n` and `boot1` (the draw-1 value). + +### Result stores: `2_pipeline////` + +| File | Holds | +| --- | --- | +| `coefficients.csv` | Every stage's coefficients per species, draw and term | +| `metrics.csv` | Metrics per species and draw | +| `grid_predictions.csv` | The final model's prediction for each habitat type | +| `unit_predictions.csv` | Only with `unit_predictions = "oob"` or `"all"`; several GB for birds | +| `meta.json` | Stages, engines, rules, covariates, resampling, seed, overrides | + +`2_pipeline//run_log.csv` records every species, region and draw: +`ok`, or why not. + +### Summary tables: `3_output//tables/` + +| File | One row per | Notes | +| --- | --- | --- | +| `coverage.csv` | run × region | Species, draws and stages written; `species = 0` means nothing ran (see `run_log.csv`) | +| `metric_summary.csv` | taxon × region × species × metric | Prefixes: `insample_`, `oob_` (held-out; NA for draw 1), `v2val_` and `oob_v2val_` (v2's seven plant validation AUCs) | +| `grid_summary.csv` | taxon × region × species × `grid_unit` | Response scale; pure stand at the stage's constants (`Climate` 0) | +| `coefficient_summary.csv` | run × region × stage × species × term | Gitignored. Term meaning varies by taxon and stage | +| `comparison_metrics.csv` | species × metric | Against exp_000; both sides' summaries, the difference, and whether this run did better. Gitignored | + +Metrics: `auc`, `deviance_explained`, `rmse`, `spearman`, +`calibration_slope`, `prevalence`, `n`. They score each run's final +model, so for plants `insample_auc` equals `v2val_Full`. "Better" is +higher AUC, deviance explained and Spearman, lower RMSE, and a +calibration slope closer to 1. + +## Data access + +Runs read the published dataset versions pinned in `data_source.R` +(`sdm_published$versions`), on `//ABMI-DATA2/science/sdmMethodsDev/0_data/`. +Set `SDM_TEST_DATASET` or `SDM_V2_RESULTS` to read another copy. Each +call checks file presence and size; `verify_published()` checks md5. diff --git a/1_code/harness/cache.R b/1_code/harness/cache.R new file mode 100644 index 0000000..55a303a --- /dev/null +++ b/1_code/harness/cache.R @@ -0,0 +1,72 @@ +# --- +# title: Session Cache for Dataset Reads +# author: Brendan Casey +# created: 2026-10-03 +# inputs: none +# outputs: none; holds read results in memory +# notes: +# - Keeps one copy per file per session of lookups that were +# re-read many times per run. Keyed on path, mtime, size and a +# read tag, so an edited file is re-read, not served stale. +# - Only file reads are cached, never fits or results, so the +# cache cannot change what a run computes. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +## 1.2 The cache ---- +if (!exists(".sdm_cache", inherits = FALSE)) { + .sdm_cache <- new.env(parent = emptyenv()) +} + +# 2. cached_read() ---- + +#' Read a File Once per Session +#' +#' @param path Character. The file to read. +#' @param reader A function of the path returning its contents. +#' @param tag Character. Distinguishes two different reads of the +#' same file, such as one column subset and another. +#' @return Whatever `reader` returns. +#' +#' @example # Example usage of the function +#' # map <- cached_read(path, function(p) as.data.frame(fread(p))) +cached_read <- function(path, reader, tag = "") { + info <- file.info(path) + + if (is.na(info$size)) { + return(reader(path)) + } + + key <- paste( + normalizePath(path, winslash = "/", mustWork = FALSE), + format(info$mtime, "%Y%m%d%H%M%OS3"), info$size, tag, + sep = "|" + ) + + if (!exists(key, envir = .sdm_cache, inherits = FALSE)) { + assign(key, reader(path), envir = .sdm_cache) + } + + get(key, envir = .sdm_cache, inherits = FALSE) +} + +# 3. clear_cache() ---- + +#' Empty the Session Cache +#' +#' @return The number of entries removed, invisibly. +#' +#' @example # Example usage of the function +#' # clear_cache() +clear_cache <- function() { + keys <- ls(.sdm_cache, all.names = TRUE) + rm(list = keys, envir = .sdm_cache) + + invisible(length(keys)) +} + +# End of script ---- diff --git a/1_code/harness/compare.R b/1_code/harness/compare.R new file mode 100644 index 0000000..625ef43 --- /dev/null +++ b/1_code/harness/compare.R @@ -0,0 +1,182 @@ +# --- +# title: Compare an Experiment Against a Baseline +# author: Brendan Casey +# created: 2026-10-03 +# inputs: +# in each experiment's out_dir/tables/: +# - metric_summary.csv (collect_results()) +# outputs: +# in the candidate's out_dir/tables/: +# - comparison_metrics.csv: one row per species and metric +# the per-taxon, region and metric summary is returned (and +# printed by run_experiment()) but not written +# notes: +# - Rows match on taxon, region, season, part, species and +# metric, not run name; differences are in medians over draws. +# - Defaults to out-of-bag metrics: in-sample metrics flatter +# flexible methods. +# - Warns when the two sides hold different draw counts. Judge +# noise from `share_better` and the p10-p90 bands. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # keyed joins (version: 1.16.4) + +# 2. better_direction() ---- + +#' Which Way Is Better for a Metric +#' +#' @param metric Character vector of metric names, with or +#' without an `insample_` or `oob_` prefix. +#' @return Character vector: "higher", "lower", "toward_one" or +#' "none" (prevalence, n: not a measure of fit). +#' +#' @example # Example usage of the function +#' # better_direction(c("oob_auc", "oob_rmse")) +better_direction <- function(metric) { + base <- sub("^(insample_|oob_)", "", metric) + + ifelse( + base %in% c("rmse"), "lower", + ifelse( + base %in% c("calibration_slope"), "toward_one", + ifelse(base %in% c("prevalence", "n"), "none", "higher") + ) + ) +} + +# 3. compare_experiments() ---- + +#' Compare an Experiment's Summaries Against a Baseline's +#' +#' @param candidate_dir Character. The experiment's out_dir. +#' @param baseline_dir Character. The baseline's out_dir, e.g. +#' 3_output/exp_000_parity_v2. +#' @param metrics Character vector of metrics to compare, or NULL +#' for the out-of-bag fit metrics. +#' @param write Logical. Write `metrics` to +#' `candidate_dir/tables/comparison_metrics.csv`. +#' @return A list of data frames: `metrics` (per species) and +#' `summary` (per taxon, region and metric). +#' +#' @example # Example usage of the function +#' # compare_experiments("3_output/exp_001_brt", +#' # "3_output/exp_000_parity_v2") +compare_experiments <- function( + candidate_dir, + baseline_dir, + metrics = NULL, + write = TRUE +) { + read_summary <- function(dir, name) { + path <- file.path(dir, "tables", paste0(name, ".csv")) + + if (!file.exists(path)) { + return(NULL) + } + + rows <- fread(path, na.strings = "") + # Season and part are empty for most taxa; a join needs them + # equal, not missing + for (column in c("season", "part")) { + if (column %in% names(rows)) { + rows[[column]] <- ifelse( + is.na(rows[[column]]), "", as.character(rows[[column]]) + ) + } + } + rows + } + + keys <- c("taxon", "region", "season", "part", "species") + values <- c("n", "median", "p10", "p90") + + # Step 1: Model fit, per species and metric + metrics <- metrics %||% c( + "oob_auc", "oob_deviance_explained", "oob_rmse", + "oob_spearman", "oob_calibration_slope" + ) + + candidate <- read_summary(candidate_dir, "metric_summary") + baseline <- read_summary(baseline_dir, "metric_summary") + + if (is.null(candidate) || is.null(baseline)) { + stop( + "Both experiments need tables/metric_summary.csv; run their ", + "collect step first.", + call. = FALSE + ) + } + + # The mean and sd ride along where both experiments have them; + # a baseline summarized before they were added lacks them + values <- c(values, intersect( + c("mean", "sd"), intersect(names(candidate), names(baseline)) + )) + + pick <- function(rows) { + rows[metric %in% metrics, c(keys, "metric", values), with = FALSE] + } + + joined <- merge( + pick(baseline), pick(candidate), + by = c(keys, "metric"), suffixes = c("_baseline", "_candidate") + ) + + joined[, difference := median_candidate - median_baseline] + joined[, direction := better_direction(metric)] + joined[, better := data.table::fcase( + direction == "higher", median_candidate > median_baseline, + direction == "lower", median_candidate < median_baseline, + direction == "toward_one", + abs(median_candidate - 1) < abs(median_baseline - 1), + default = NA + )] + + # Step 2: Per taxon, region and metric + summary <- joined[, list( + species = .N, + draws_baseline = stats::median(n_baseline, na.rm = TRUE), + draws_candidate = stats::median(n_candidate, na.rm = TRUE), + median_baseline = stats::median(median_baseline, na.rm = TRUE), + median_candidate = stats::median(median_candidate, na.rm = TRUE), + median_difference = stats::median(difference, na.rm = TRUE), + share_better = mean(better, na.rm = TRUE) + ), by = c("taxon", "region", "season", "part", "metric")] + + # A band from 5 draws is not comparable with one from 100, so a + # mismatch is said out loud rather than left in a column + mismatched <- summary[draws_baseline != draws_candidate] + + if (nrow(mismatched) > 0) { + warning( + "The two experiments hold different numbers of draws (", + paste(unique(mismatched$draws_baseline), collapse = ", "), + " in the baseline, ", + paste(unique(mismatched$draws_candidate), collapse = ", "), + " here). Compare like with like before reading the result.", + call. = FALSE + ) + } + + out <- list( + metrics = as.data.frame(joined), + summary = as.data.frame(summary) + ) + + if (write) { + tables_dir <- file.path(candidate_dir, "tables") + dir.create(tables_dir, recursive = TRUE, showWarnings = FALSE) + fwrite( + out$metrics, + file.path(tables_dir, "comparison_metrics.csv"), + na = "" + ) + } + + out +} + +# End of script ---- diff --git a/1_code/harness/covariate_sets.R b/1_code/harness/covariate_sets.R index 198104e..a933b2b 100644 --- a/1_code/harness/covariate_sets.R +++ b/1_code/harness/covariate_sets.R @@ -3,32 +3,17 @@ # author: Brendan Casey # created: 2026-09-09 # inputs: -# in 0_data/test_dataset/lookup/: +# in the test dataset's lookup/ (see data_source.R): # - covariate_columns.csv # outputs: none; returns objects in memory # notes: -# - Turns a named covariate set into the master column names a -# taxon actually has. This is what lets an experiment ask for -# "climate_v2_topography" instead of editing a formula list, -# which is the mechanism behind the include/exclude -# covariate questions this repository exists to ask. -# - covariate_columns.csv maps taxon and block to master column, -# so the same set resolves to different columns per taxon -# where the source data differ, and fails loudly where a -# taxon simply does not have a covariate. -# - Requires covariate_key() from data_load.R, which is the one -# place a taxon's covariate key is derived; source that file -# first. -# - Sets are defined here rather than in a spec because they -# cross taxa: the point of a named set is that the same -# request means the same thing for plants, mammals and birds. -# - A set may name another set, so sets compose. Recursion is -# depth-limited rather than trusted, because a typo that -# makes a set reference itself would otherwise hang. -# - Future improvement - topography and remote-sensing sets are -# declared below but resolve to nothing until a _setup script -# adds those columns to covariates.csv. They are listed so -# the gap is visible rather than discovered mid-experiment. +# - Resolves named, cross-taxa covariate sets to the master +# columns each taxon has (via covariate_columns.csv), failing +# loudly where a taxon lacks one. Sets may name other sets. +# - Needs covariate_key() from data_load.R. +# - The topography and remote_sensing sets are not in the frozen +# dataset and resolve to nothing; they are listed so the gap +# is visible. # --- # 1. Setup ---- @@ -42,10 +27,9 @@ library(data.table) # lookup table reading (version: 1.16.4) #' Named Covariate Bundles #' -#' Each entry is a character vector of master column names, of -#' block names, or of other set names. Blocks are taken from -#' covariate_columns.csv, so `block:veg` means whatever the veg -#' block holds for the taxon being resolved. +#' Entries hold master column names, other set names, or +#' `block:`, which means whatever that block holds for the +#' taxon being resolved. #' #' @return A named list of character vectors. #' @@ -54,38 +38,22 @@ library(data.table) # lookup table reading (version: 1.16.4) #' # covariate_sets()$climate_v2 covariate_sets <- function() { list( - # The whole of a block, as the source file defines it. These - # are what a parity run uses. + # Whole blocks, as parity runs use climate_all = "block:climate", veg_all = "block:veg", soil_all = "block:soil", design_all = "block:design", - # The climate terms the v2 plant model sets draw on. Named - # explicitly rather than as the block, because the block - # carries derived and design columns too. - # - # This is a plant-scale set. Birds carry only five climate - # columns, so asking for it on birds fails; use - # climate_common for an experiment spanning both. + # The climate terms v2 plant model sets draw on (the block + # also holds derived and design columns). Fails on birds; use + # climate_common across taxa. climate_v2 = c( "MAT", "MAP", "MWMT", "MCMT", "TD", "MSP", "AHM", "SHM", "PET", "CMD", "FFP", "EMT", "EXT", "MTD", "PS" ), - # The climate columns every taxon has, and so the widest set - # a cross-taxa experiment can use without the taxa silently - # fitting different models. - # - # Five columns, once the mammal camera climate is joined. - # It was two before that: mammals carried no TD, CMD or EMT, - # because the SpTable climate block is a different and - # narrower extraction than the one their models were fitted - # against. Birds still have no MAT or PET, which is what - # keeps the shared set at five rather than more. - # - # Re-derive with common_covariates() after any dataset - # version change rather than trusting this list. + # Climate columns every taxon has (birds lack MAT and PET). + # Re-derive with common_covariates() after a dataset change. climate_common = c("MAP", "FFP", "TD", "CMD", "EMT"), # The wider set birds and the plant groups share, for an @@ -94,15 +62,12 @@ covariate_sets <- function() { "MAP", "TD", "CMD", "FFP", "EMT" ), - # Topography. Not in the dataset yet; see the header. topography = c("elevation", "slope", "aspect", "TRI", "TPI"), - # Remote sensing. Not in the dataset yet; see the header. remote_sensing = c( "canopy_height", "canopy_cover", "ndvi", "lai" ), - # Compositions, which is the point of naming sets at all. climate_v2_topography = c("climate_v2", "topography"), climate_v2_remote = c("climate_v2", "remote_sensing") ) @@ -114,16 +79,9 @@ covariate_sets <- function() { #' Covariates Computed from Other Stored Columns #' -#' Some taxa lack a covariate that their own model set needs, but -#' hold the columns it is defined from. Deriving it lets that -#' taxon join an experiment it would otherwise be excluded from. -#' -#' Derivation is only sound where the definition is exact. Each -#' entry records how it was checked against a taxon that carries -#' both the inputs and the answer, so a derivation is a verified -#' claim rather than an assumption. A quantity that only -#' correlates with its supposed formula does not belong here - -#' see the note on CMD below. +#' Only exact definitions belong here; each entry's `note` says +#' how it was verified. A quantity that merely correlates with a +#' formula does not (see CMD below). #' #' @return A named list, each entry with `inputs`, `fn` and #' `note`. @@ -133,10 +91,6 @@ covariate_sets <- function() { #' # derived_covariates()$TD$note derived_covariates <- function() { list( - # Continentality: the summer-winter temperature range. - # Checked against the vascular plant table, which carries - # TD, MWMT and MCMT: agrees to 0.1 degrees, which is the - # precision the normals are stored at. TD = list( inputs = c("MWMT", "MCMT"), fn = function(d) d$MWMT - d$MCMT, @@ -146,20 +100,14 @@ derived_covariates <- function() { ) ) - # CMD is deliberately absent. The mammal climate script - # glosses it as "PET - MAP, >= 0", but checked against the - # vascular plant table that formula is off by up to 195 mm - # (r = 0.975). Climatic moisture deficit is a monthly sum, - # not an annual difference, so it cannot be recovered from - # the stored annual columns and has to be sourced. + # CMD is deliberately absent: "PET - MAP, >= 0" (the mammal + # script's gloss) is off by up to 195 mm against + # vascular_plant, because CMD is a monthly sum. , - # The products the harmonizer deliberately does not store, - # because a separately stored square would not equal the - # square of the stored column at CSV precision. The plant - # climate model set fits all four, so a plant run needs them - # derived. Each was checked against the frozen v2 source and - # agrees exactly. + # Products are not stored: a stored square would not equal + # the square of the stored column at CSV precision. The plant + # climate set fits all four. MAPPET = list( inputs = c("MAP", "PET"), fn = function(d) d$MAP * d$PET, @@ -181,11 +129,8 @@ derived_covariates <- function() { note = "MWMT squared. Exact against the v2 source." ), - # The spatial trend terms, likewise. Easting and Northing are - # UTMX and UTMY under another name - identical in the v2 - # source - and sites.csv carries them too, but the model set - # names them, so they are reachable from the covariate table - # alone rather than only through a join. + # Spatial trend terms, named as the v2 model sets name them, + # so they come from the covariate table without a join Easting = list( inputs = "UTMX", fn = function(d) d$UTMX, @@ -280,7 +225,7 @@ apply_derivations <- function( #' Read the Covariate Column Catalogue #' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Keep only this taxon's rows, or NULL #' for all of them. #' @return A data frame of taxon, block, source_column and @@ -302,11 +247,10 @@ covariate_catalogue <- function(data_dir, taxon = NULL) { ) } - catalogue <- as.data.frame(fread(path)) + catalogue <- cached_read(path, function(p) as.data.frame(fread(p))) if (!is.null(taxon)) { - # The catalogue is keyed the way covariates.csv is, so a - # split taxon has one row set per region. + # A taxon split by region has keys like `_` keys <- unique(catalogue$taxon) matching <- keys[ keys == taxon | startsWith(keys, paste0(taxon, "_")) @@ -332,14 +276,12 @@ covariate_catalogue <- function(data_dir, taxon = NULL) { #' List the Covariates One Taxon and Region Have #' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Taxon slug. #' @param region Character. Region name, or NULL. #' @param block Character. Restrict to one block, or NULL. -#' @param include_derived Logical. Count covariates that are not -#' stored but can be computed from stored ones. TRUE by -#' default, because a derived column is as usable as a stored -#' one; FALSE answers what the dataset literally holds. +#' @param include_derived Logical. Also count covariates +#' derivable from stored ones (see derived_covariates()). #' @return A character vector of master column names. #' #' @example # Example usage of the function @@ -351,7 +293,7 @@ available_covariates <- function( block = NULL, include_derived = TRUE ) { - key <- covariate_key(taxon, region) + key <- covariate_key(taxon, region, data_dir) catalogue <- covariate_catalogue(data_dir) rows <- catalogue[catalogue$taxon == key, ] @@ -372,8 +314,7 @@ available_covariates <- function( return(stored) } - # A derived column counts as available wherever its inputs are, - # which is what lets mammals reach TD without it being stored. + # e.g. mammals reach TD without it being stored derived <- derived_covariates() reachable <- names(derived)[vapply( names(derived), @@ -388,12 +329,10 @@ available_covariates <- function( #' Find the Covariates Several Taxa Share #' -#' A cross-taxa experiment can only compare taxa on covariates -#' they all have. Asking for more means each taxon quietly fits a -#' different model, which is the one thing a comparison must not -#' do. +#' Cross-taxa comparisons must use only shared covariates, or +#' each taxon fits a different model. #' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxa Character vector of taxon slugs. #' @param block Character. Restrict to one block, or NULL. #' @return A character vector of master column names present for @@ -424,8 +363,7 @@ common_covariates <- function(data_dir, taxa, block = NULL) { rows <- rows[rows$block %in% block, ] } - # A split taxon must have the column in every region, or it - # is not usable for that taxon as a whole. + # A split taxon must have the column in every region per_key <- split(rows$master_column, rows$taxon) Reduce(intersect, lapply(per_key, unique)) @@ -438,20 +376,13 @@ common_covariates <- function(data_dir, taxa, block = NULL) { #' Map v2 Term Names onto Harmonized Column Names #' -#' The v2 model formulas name columns as the source files did. -#' The harmonizer suffixed any column that appears in more than -#' one block with the block it came from, because the north and -#' south files carry columns of the same name holding different -#' values - `HardLin` means the hard linear cover measured for -#' the vegetation model in one and for the soil model in the -#' other. 84 plant columns and 6 bird columns are affected. -#' -#' Without this mapping a v2 formula asks for `HardLin`, which -#' exists in the wide table for some other taxon and reads back -#' as NA, so the map is what keeps a habitat model from silently -#' losing its footprint terms. +#' The harmonizer suffixed columns found in more than one block +#' with their block, because same-named columns hold different +#' values (e.g. `HardLin` in the veg and soil blocks). 84 plant +#' and 6 bird columns are affected. Unmapped, a v2 formula would +#' read another taxon's column as NA and silently lose terms. #' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Taxon slug. #' @param region Character. Region name, or NULL. #' @param block Character. Restrict to one block, or NULL for @@ -467,7 +398,7 @@ term_map <- function( region = NULL, block = NULL ) { - key <- covariate_key(taxon, region) + key <- covariate_key(taxon, region, data_dir) catalogue <- covariate_catalogue(data_dir) rows <- catalogue[catalogue$taxon == key, ] @@ -488,9 +419,8 @@ term_map <- function( #' Rewrite Source Names to Master Names #' -#' Applied to model formula text and to prediction grid column -#' names, so both speak the dataset's names while the v2 sets and -#' the grids keep the names they were written with. +#' Applied to formula text and prediction grid column names, so +#' v2 sets and grids keep their original names. #' #' @param x Character vector of formula text, or of column names. #' @param map A named vector from term_map(). @@ -503,16 +433,13 @@ apply_term_map <- function(x, map) { return(x) } - # A staged model set is a list of groups, so the rewrite - # recurses rather than assuming a flat vector. + # Staged model sets are lists of groups if (is.list(x)) { return(lapply(x, apply_term_map, map = map)) } # Longest names first, so `SurrHardLin` is not half-rewritten - # by the rule for `HardLin`. Word boundaries alone would not - # settle it, because the shorter name is a suffix of the - # longer one rather than a separate word. + # by the rule for `HardLin` order_by_length <- order(nchar(names(map)), decreasing = TRUE) for (i in order_by_length) { @@ -528,24 +455,18 @@ apply_term_map <- function(x, map) { #' Resolve Covariate Set Names to Master Columns #' -#' Expands set names, block references and bare column names into -#' the master columns a taxon and region actually have. A name -#' that resolves to nothing is an error, not an empty result: -#' silently dropping a covariate would make an experiment look -#' like it tested something it did not. +#' An unavailable covariate is an error by default: dropping it +#' silently would misreport what an experiment tested. #' #' @param requested Character vector of set names, `block:` #' references, or master column names. -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Taxon slug. #' @param region Character. Region name, or NULL. #' @param sets Named list of covariate sets. -#' @param strict Logical. TRUE stops when a requested covariate -#' is unavailable. FALSE drops it with a warning, which is a -#' deliberate research choice - each taxon then fits its own -#' best available set - and is never the default, because a -#' silently different model per taxon would invalidate a -#' cross-taxa comparison. +#' @param strict Logical. FALSE drops unavailable covariates with +#' a warning, so each taxon fits a different set; never use it +#' for cross-taxa comparisons. #' @param max_depth Integer. How far a set may reference another #' set before the nesting is treated as circular. #' @return A character vector of master column names, in the @@ -565,9 +486,8 @@ resolve_covariates <- function( ) { available <- available_covariates(data_dir, taxon, region) - # Step 1: Expand set names and block references until only - # column names are left. Depth-limited, so a set that names - # itself fails rather than hanging. + # Step 1: Expand sets and blocks to columns. Depth-limited, so + # a self-referencing set fails rather than hanging. expand <- function(names_in, depth) { if (depth > max_depth) { stop( @@ -598,10 +518,8 @@ resolve_covariates <- function( columns <- unique(expand(requested, 1L)) - # Step 2: Report what this taxon does not have. Set names are - # reported separately from bare columns, because an empty set - # usually means a covariate has not been added to the dataset - # yet, while an unknown column usually means a typo. + # Step 2: Report what this taxon lacks. Empty sets are named + # separately: they usually mean missing data, not a typo. missing_columns <- setdiff(columns, available) if (length(missing_columns) > 0 && !strict) { diff --git a/1_code/harness/data_load.R b/1_code/harness/data_load.R index 5cbda00..80762a8 100644 --- a/1_code/harness/data_load.R +++ b/1_code/harness/data_load.R @@ -3,39 +3,25 @@ # author: Brendan Casey # created: 2026-09-09 # inputs: -# in 0_data/test_dataset/: +# in the test dataset (see harness/data_source.R): # - sites.csv -# - .csv # - covariates.csv -# - _offsets.csv (birds only) +# - each taxon's response and offset files, as named in +# lookup/dataset_manifest.csv +# - lookup/factor_levels.csv # outputs: none; returns objects in memory # notes: -# - The taxon-agnostic data layer. Every reader takes the taxon -# as an argument and knows nothing else about it; what a taxon -# means is carried by its spec, not by this file. -# - covariates.csv is keyed on survey_unit_id and a covariate -# key, which is the taxon for most taxa but the taxon and -# region for mammals, whose north and south files carry -# different values for the same deployment. covariate_key() -# is the one place that difference is expressed. -# - Reading is column-selective. covariates.csv is 218 MB and -# 457 columns wide; a run wants a handful, so every reader -# takes the columns it needs rather than loading the table. -# - build_model_data() loads once per taxon and region, and -# model_frame() then cuts a per-species frame out of it. A -# bird run is 215,758 units by 172 species, so re-reading per -# species would dominate the run time. -# - covariates.csv is one wide table across every taxon, so a -# column another taxon carries reads back as all NA here. -# load_covariates() checks the per-taxon catalogue rather -# than the file header for that reason. -# - A covariate a taxon does not store but can be computed from -# ones it does is derived on load, through the registry in -# covariate_sets.R. That is how mammals reach TD, which their -# own climate model set needs and their source file lacks. -# Requires covariate_sets.R to be sourced first. -# - Future improvement - a Parquet or duckdb backing store -# would let a bird run select rows as well as columns. +# - Taxon-agnostic: file locations and covariate keys come from +# lookup/dataset_manifest.csv (_setup/09). The key is +# `_` for mammals, whose north and south files +# hold different values for the same deployment. +# - Reads are column-selective (covariates.csv is ~218 MB, 457 +# columns). build_model_data() loads once per taxon and +# region; model_frame() cuts per-species frames from it. +# - Derivable covariates are computed on load (needs +# covariate_sets.R); an experiment's covariate_files are +# joined on load, so 0_data/ is never changed. +# - TODO: a Parquet or duckdb store would allow row selection. # --- # 1. Setup ---- @@ -43,48 +29,105 @@ ## 1.1 Load packages ---- library(data.table) # column-selective CSV reading (version: 1.16.4) -# 2. covariate_key() ---- +# 2. The dataset manifest ---- + +## 2.1 dataset_manifest() ---- + +#' Read Where Each Taxon's Files Are +#' +#' One row per taxon and region. +#' +#' @param data_dir Character. The test dataset folder. +#' @return A data frame. +#' +#' @example # Example usage of the function +#' # dataset_manifest(data_dir) +dataset_manifest <- function(data_dir) { + path <- file.path(data_dir, "lookup", "dataset_manifest.csv") + + if (!file.exists(path)) { + stop( + "Dataset manifest not found: + ", path, + " +Run 1_code/_setup/09_harmonize_lookups.R first.", + call. = FALSE + ) + } + + cached_read(path, function(p) { + as.data.frame(fread(p, na.strings = "", colClasses = "character")) + }) +} + +## 2.2 manifest_entry() ---- + +#' One Taxon's Manifest Row +#' +#' @param data_dir Character. The test dataset folder. +#' @param taxon Character. Taxon slug. +#' @param region Character, or NULL for the taxon's first region, +#' which is enough for anything that does not vary by region. +#' @return A one-row data frame. +#' +#' @example # Example usage of the function +#' # manifest_entry(data_dir, "mammal", "north")$covariate_key +manifest_entry <- function(data_dir, taxon, region = NULL) { + manifest <- dataset_manifest(data_dir) + rows <- manifest[manifest$taxon == taxon, ] + + if (nrow(rows) == 0) { + stop( + "Taxon `", taxon, "` is not in the dataset manifest. Taxa: ", + paste(unique(manifest$taxon), collapse = ", "), + call. = FALSE + ) + } + + if (!is.null(region)) { + rows <- rows[rows$region == region, ] + + if (nrow(rows) == 0) { + stop( + "The manifest has no region `", region, "` for ", taxon, + ".", call. = FALSE + ) + } + } + + rows[1, , drop = FALSE] +} + +## 2.3 covariate_key() ---- #' Name a Taxon's Key in covariates.csv #' -#' Most taxa have one covariate row per survey unit. Mammals have -#' two models fitted on overlapping deployments, north and south, -#' whose covariates disagree, so their rows are keyed by taxon and -#' region together. -#' -#' @param taxon Character. Taxon slug, as in the response file -#' name (e.g. "bryophyte", "mammal", "bird"). -#' @param region Character. Region name, or NULL when the taxon -#' is not split. -#' @param split_taxa Character vector of taxa whose covariates -#' are stored per region. +#' @param taxon Character. Taxon slug. +#' @param region Character, or NULL when the taxon's covariates +#' are not stored per region. +#' @param data_dir Character. The test dataset folder. #' @return Character. The value to match in the `taxon` column of #' covariates.csv. #' #' @example # Example usage of the function -#' # covariate_key("mammal", "north") # "mammal_north" -#' # covariate_key("bryophyte", "north") # "bryophyte" -covariate_key <- function( - taxon, - region = NULL, - split_taxa = c("mammal") -) { - if (is.null(region) || !taxon %in% split_taxa) { +#' # covariate_key("mammal", "north", data_dir) # "mammal_north" +#' # covariate_key("bryophyte", NULL, data_dir) # "bryophyte" +covariate_key <- function(taxon, region = NULL, data_dir) { + if (is.null(region)) { return(taxon) } - paste(taxon, region, sep = "_") + manifest_entry(data_dir, taxon, region)$covariate_key } # 3. load_sites() ---- #' Load the Survey Unit Table #' -#' One row per survey unit across every design. Carries the -#' identity, design and location fields the covariate tables -#' factor out, and an `in_` flag per taxon. +#' One row per survey unit across every design, with an +#' `in_` coverage flag per taxon. #' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Keep only units that taxon covers, or #' NULL for every unit. #' @param columns Character vector of columns to read, or NULL @@ -92,14 +135,13 @@ covariate_key <- function( #' @return A data frame, one row per survey unit. #' #' @example # Example usage of the function -#' # sites <- load_sites("0_data/test_dataset", taxon = "bird") +#' # sites <- load_sites(test_dataset_dir(), taxon = "bird") #' # nrow(sites) load_sites <- function(data_dir, taxon = NULL, columns = NULL) { path <- file.path(data_dir, "sites.csv") check_dataset_file(path) - # Step 1: The coverage flag has to be read even when it is not - # among the requested columns, or the filter cannot be applied + # Step 1: Read the coverage flag even if not requested flag <- if (is.null(taxon)) NULL else paste0("in_", taxon) if (!is.null(columns)) { @@ -135,14 +177,11 @@ load_sites <- function(data_dir, taxon = NULL, columns = NULL) { #' Load One Taxon's Response Table #' -#' `survey_unit_id` plus one column per species. Values are -#' whatever the source recorded - detections for plants, densities -#' for mammals, counts for birds - and are left untransformed -#' here. The spec's response transform is applied in -#' model_frame(). +#' Values are as recorded (plant detections, mammal densities, +#' bird counts); model_frame() applies the spec's transform. #' #' @param taxon Character. Taxon slug. -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param species Character vector of species columns to read, or #' NULL for all of them. #' @return A data frame of survey_unit_id and species columns. @@ -151,15 +190,16 @@ load_sites <- function(data_dir, taxon = NULL, columns = NULL) { #' # y <- load_response("bryophyte", data_dir, #' # species = "Aulacomnium.palustre") load_response <- function(taxon, data_dir, species = NULL) { - path <- file.path(data_dir, paste0(taxon, ".csv")) + path <- file.path( + data_dir, manifest_entry(data_dir, taxon)$response_file + ) check_dataset_file(path) if (is.null(species)) { return(as.data.frame(fread(path))) } - # Step 1: Fail on an unknown species here rather than returning - # a frame that is quietly missing a column + # Step 1: Fail on an unknown species available <- names(fread(path, nrows = 0)) unknown <- setdiff(species, available) @@ -180,14 +220,12 @@ load_response <- function(taxon, data_dir, species = NULL) { #' Load One Taxon's Offsets #' -#' Birds carry a QPAD log-offset per survey unit and species, -#' which makes a count comparable across survey protocols and -#' detectability. No other taxon has one, and NULL is the honest -#' answer for them rather than a column of zeros - a zero offset -#' is a modelling choice, and this is a data reader. +#' Only birds have offsets (QPAD log-offsets per unit and +#' species). Other taxa return NULL, not zeros: a zero offset is +#' a modelling choice. #' #' @param taxon Character. Taxon slug. -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param species Character vector of species columns to read, or #' NULL for all of them. #' @return A data frame of survey_unit_id and species columns, or @@ -196,12 +234,15 @@ load_response <- function(taxon, data_dir, species = NULL) { #' @example # Example usage of the function #' # off <- load_offsets("bird", data_dir, species = "ALFL") load_offsets <- function(taxon, data_dir, species = NULL) { - path <- file.path(data_dir, paste0(taxon, "_offsets.csv")) + offset_file <- manifest_entry(data_dir, taxon)$offset_file - if (!file.exists(path)) { + if (is.na(offset_file)) { return(NULL) } + path <- file.path(data_dir, offset_file) + check_dataset_file(path) + if (is.null(species)) { return(as.data.frame(fread(path))) } @@ -222,11 +263,9 @@ load_offsets <- function(taxon, data_dir, species = NULL) { #' Load Covariates for One Taxon and Region #' -#' Reads the requested columns from covariates.csv and keeps the -#' rows belonging to one covariate key. #' #' @param taxon Character. Taxon slug. -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param columns Character vector of master column names. #' @param region Character. Region name, or NULL when the taxon's #' covariates are not stored per region. @@ -246,21 +285,16 @@ load_covariates <- function( path <- file.path(data_dir, "covariates.csv") check_dataset_file(path) - key <- covariate_key(taxon, region) + key <- covariate_key(taxon, region, data_dir) - # Step 1: Check against what this taxon has, not against the - # file header. covariates.csv is one wide table across every - # taxon, so a column another taxon carries is present in the - # header and reads back as all NA here. Checking the header - # alone would hand back a column of nothing and let the model - # silently drop every row. + # Step 1: Check the taxon's catalogue, not the file header: the + # wide table holds other taxa's columns, which read back as all + # NA here. stored <- available_covariates( taxon, data_dir = data_dir, region = region, include_derived = FALSE ) - # A column this taxon does not store may still be computable - # from ones it does. derivable <- derivable_covariates(columns, stored) to_read <- setdiff(columns, derivable) @@ -282,30 +316,54 @@ load_covariates <- function( ) } - covariates <- fread( - path, - select = unique(c("survey_unit_id", "taxon", to_read)) + # Step 2: Read only columns not already cached for this key; + # stages request overlapping column sets. + info <- file.info(path) + cache_key <- paste( + "covariates", normalizePath(path, winslash = "/"), + format(info$mtime, "%Y%m%d%H%M%OS3"), info$size, key, + sep = "|" ) + held <- if (exists(cache_key, envir = .sdm_cache)) { + get(cache_key, envir = .sdm_cache) + } else { + NULL + } + to_load <- setdiff(to_read, names(held)) - # Step 2: Keep this taxon's rows, then drop the key column so - # the result is survey_unit_id plus covariates - covariates <- covariates[covariates$taxon == key, ] - - if (nrow(covariates) == 0) { - keys <- unique(fread(path, select = "taxon")$taxon) - stop( - "No covariate rows for key `", key, "`. Keys present: ", - paste(keys, collapse = ", "), - call. = FALSE + if (is.null(held) || length(to_load) > 0) { + fresh <- fread( + path, select = unique(c("survey_unit_id", "taxon", to_load)) ) - } - covariates[, taxon := NULL] + fresh <- fresh[fresh$taxon == key, ] + + if (nrow(fresh) == 0) { + keys <- unique(fread(path, select = "taxon")$taxon) + stop( + "No covariate rows for key `", key, "`. Keys present: ", + paste(keys, collapse = ", "), + call. = FALSE + ) + } + + fresh[, taxon := NULL] + fresh <- as.data.frame(fresh) - covariates <- as.data.frame(covariates) + held <- if (is.null(held)) { + fresh + } else { + # Rows always come back in file order, so columns line up + cbind(held, fresh[, to_load, drop = FALSE]) + } + + assign(cache_key, held, envir = .sdm_cache) + } + + covariates <- held[, unique(c("survey_unit_id", to_read)), + drop = FALSE] - # Step 3: Compute whatever was derivable rather than stored, - # then hand back exactly the columns that were asked for + # Step 3: Derive, then return exactly the requested columns if (length(derivable) > 0) { covariates <- apply_derivations(covariates, derivable) } @@ -313,49 +371,159 @@ load_covariates <- function( covariates[, unique(c("survey_unit_id", columns)), drop = FALSE] } +## 6.1 check_covariate_files() ---- + +#' Check the Covariate Files an Experiment Adds +#' +#' Each file must have a unique survey_unit_id key and only +#' columns no other source has. +#' +#' @param files Character vector of CSV paths. +#' @param data_dir Character. The test dataset folder. +#' @return `files`, invisibly; stops on the first problem. +#' +#' @example # Example usage of the function +#' # check_covariate_files("2_pipeline/exp_002/inputs/soil.csv", +#' # data_dir) +check_covariate_files <- function(files, data_dir) { + taken <- names(fread(file.path(data_dir, "covariates.csv"), + nrows = 0)) + + for (path in files) { + if (!file.exists(path)) { + stop( + "Covariate file not found: ", path, "\nRun the ", + "experiment's script that writes it first.", + call. = FALSE + ) + } + + header <- names(fread(path, nrows = 0)) + + if (!"survey_unit_id" %in% header) { + stop("Covariate file ", basename(path), " has no ", + "survey_unit_id column.", call. = FALSE) + } + + ids <- fread(path, select = "survey_unit_id", + colClasses = "character")$survey_unit_id + + if (anyDuplicated(ids) > 0) { + stop("Covariate file ", basename(path), " repeats a ", + "survey_unit_id; it needs one row per unit.", + call. = FALSE) + } + + added <- setdiff(header, "survey_unit_id") + clash <- intersect(added, taken) + + if (length(clash) > 0) { + stop( + "Covariate file ", basename(path), " repeats column(s) ", + "already in covariates.csv or another file: ", + paste(utils::head(clash, 10), collapse = ", "), + ". Give them new names.", + call. = FALSE + ) + } + + taken <- c(taken, added) + } + + invisible(files) +} + +## 6.2 covariate_file_columns() ---- + +#' The Columns a Set of Covariate Files Holds +#' +#' @param files Character vector of CSV paths, or NULL. +#' @return A character vector of column names, without +#' survey_unit_id. +#' +#' @example # Example usage of the function +#' # covariate_file_columns("2_pipeline/exp_002/inputs/soil.csv") +covariate_file_columns <- function(files) { + unique(unlist(lapply(files, function(path) { + setdiff(names(fread(path, nrows = 0)), "survey_unit_id") + }))) +} + +## 6.3 add_covariate_files() ---- + +#' Join Covariates from an Experiment's Own Files +#' +#' Units a file does not list get NA and drop out of models that +#' use the column. +#' +#' @param x A data frame with survey_unit_id, from +#' load_covariates(). +#' @param files Character vector of CSV paths, or NULL. +#' @param columns Character vector of the columns to add. +#' @return `x` with the columns added. +#' +#' @example # Example usage of the function +#' # add_covariate_files(x, "2_pipeline/exp_002/inputs/soil.csv", +#' # "sg_clay_0_5cm") +add_covariate_files <- function(x, files, columns) { + for (path in files) { + table <- cached_read(path, function(p) { + as.data.frame(fread( + p, colClasses = list(character = "survey_unit_id") + )) + }, tag = "covariate_file") + + wanted <- setdiff( + intersect(columns, names(table)), + c("survey_unit_id", names(x)) + ) + + if (length(wanted) == 0) { + next + } + + rows <- match(as.character(x$survey_unit_id), + table$survey_unit_id) + x[wanted] <- table[rows, wanted, drop = FALSE] + } + + x +} + # 7. apply_factor_levels() ---- #' Restore Stored Factor Levels #' -#' Some covariates are categorical - the bird vegetation and soil -#' classes, and the survey method. A CSV reads them back as -#' character, and a model then factors them alphabetically, which -#' silently changes the reference level and so every contrast -#' coefficient. `_factor_levels.csv` records the order the -#' source used, and this restores it. -#' -#' A level present in the data but not in the lookup is an error -#' rather than an extra level: it means the lookup is stale, and -#' quietly admitting it would change the contrasts. +#' CSV reads categoricals back as character, and alphabetical +#' factoring would change the reference level and every contrast. +#' Restores the source order from lookup/factor_levels.csv. An +#' unknown level is an error: the lookup is stale. #' #' @param frame A data frame. -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Taxon slug. #' @return The frame, with the recorded columns as factors. #' #' @example # Example usage of the function #' # apply_factor_levels(frame, data_dir, "bird") apply_factor_levels <- function(frame, data_dir, taxon) { - path <- file.path( - data_dir, "lookup", paste0(taxon, "_factor_levels.csv") + path <- file.path(data_dir, "lookup", "factor_levels.csv") + check_dataset_file(path) + + levels_table <- cached_read( + path, function(p) as.data.frame(fread(p)) ) + levels_table <- levels_table[levels_table$taxon == taxon, ] - if (!file.exists(path)) { + if (nrow(levels_table) == 0) { return(frame) } - levels_table <- as.data.frame(fread(path)) - - # The lookup names columns as the source did. A column the - # harmonizer suffixed because it sits in more than one block - - # the bird `method`, stored as method_veg and method_soil - is - # loaded under its suffixed name, so the lookup is matched - # through covariate_columns.csv as well. Without this those - # columns fall back to alphabetical levels, which moves the - # reference level and every contrast. + # Also match block-suffixed columns (e.g. bird `method` as + # method_veg, method_soil) via covariate_columns.csv map_path <- file.path(data_dir, "lookup", "covariate_columns.csv") column_map <- if (file.exists(map_path)) { - map <- as.data.frame(fread(map_path)) + map <- covariate_catalogue(data_dir) map[map$taxon == taxon, c("source_column", "master_column")] } else { data.frame(source_column = character(0), @@ -398,16 +566,10 @@ apply_factor_levels <- function(frame, data_dir, taxon) { restore_factor <- function(x, ordered_levels, taxon, column) { values <- as.character(x) - # Every harmonized table is written with `na = ""`, so a - # missing factor value reads back as a blank string rather - # than NA. Left alone it survives as an extra level, and the - # guard below then rejects the whole run over a level that - # is not a level at all - which is what a blank `soilc` - # does to 194 of the 91,001 southern bird surveys. - # - # v2 never sees this: it reads the same values as NA and - # glm's default na.action drops those rows. Restoring the - # NA here reproduces that, rather than inventing a rule. + # Tables are written with `na = ""`, so a missing value reads + # back as "" and would fail the level guard (e.g. blank `soilc` + # in 194 southern bird surveys). Restoring NA matches v2, where + # glm drops those rows. values[!nzchar(trimws(values))] <- NA_character_ unknown <- setdiff(stats::na.omit(unique(values)), ordered_levels) @@ -434,16 +596,11 @@ reference level and every contrast.", #' Assemble Everything One Taxon and Region Needs #' -#' Loads the response, covariates, offsets and site fields once, -#' aligned on survey_unit_id, and applies the region filter. The -#' per-species frame is then cut from this by model_frame(). -#' -#' Loading once matters: a bird run is 215,758 survey units by -#' 172 species, and re-reading per species would cost more than -#' the fitting. +#' Loaded once per taxon and region, aligned on survey_unit_id; +#' re-reading per species would cost more than fitting. #' #' @param taxon Character. Taxon slug. -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param species Character vector of species to model. #' @param covariates Character vector of master column names. #' @param region Character. Region name, or NULL. @@ -451,12 +608,14 @@ reference level and every contrast.", #' assembled frame, or NULL. Rows where it is not TRUE are #' dropped (e.g. ~ nr != "Grassland"). #' @param weight_column Character. A covariate to use as model -#' weights. Added to `covariates` automatically, so a caller -#' naming a weight does not also have to remember to load it. +#' weights; loaded automatically. #' @param aliases Named list, alias to source column. Lets a #' model set name a column the dataset stores otherwise. #' @param site_columns Character vector of sites.csv columns to -#' carry, beyond survey_unit_id. +#' carry, beyond survey_unit_id, or NULL for site_columns(). +#' @param covariate_files Character vector of an experiment's own +#' covariate CSVs, or NULL. Columns they hold are read from them +#' rather than from covariates.csv. #' @return A list with `covariates` (one row per unit, including #' the site columns), `response`, `offset` (or NULL), `units`, #' `taxon` and `region`. @@ -476,26 +635,28 @@ build_model_data <- function( region_filter = NULL, weight_column = NULL, aliases = list(), - site_columns = c( - "lat", "long", "easting", "northing", "nr", "nsr", "luf", "year", - "lured", "location", "summer_days", "winter_days" - ) + site_columns = NULL, + covariate_files = NULL ) { - # Step 1: Read each piece, narrowed to what was asked for. The - # weight is a covariate like any other, so it is added here - # rather than left for the caller to remember. + site_columns <- site_columns %||% site_columns() + + # Step 1: Read each piece, narrowed to what was asked for if (!is.null(weight_column)) { covariates <- unique(c(covariates, weight_column)) } - x <- load_covariates(taxon, data_dir, covariates, region) + from_files <- intersect( + covariates, covariate_file_columns(covariate_files) + ) + x <- load_covariates( + taxon, data_dir, setdiff(covariates, from_files), region + ) + x <- add_covariate_files(x, covariate_files, from_files) y <- load_response(taxon, data_dir, species) off <- load_offsets(taxon, data_dir, species) sites <- load_sites(data_dir, taxon, site_columns) - # Step 2: Take the survey units the covariates and the response - # agree on. A unit in one but not the other cannot be modelled, - # and silently dropping it is what a plain join would do. + # Step 2: Keep units present in covariates, response and sites units <- intersect( as.character(x$survey_unit_id), as.character(y$survey_unit_id) ) @@ -525,15 +686,11 @@ build_model_data <- function( frame <- cbind(x, site_rows[, keep, drop = FALSE]) rownames(frame) <- NULL - # Step 3b: Restore any recorded factor levels, before the - # region filter, so a filter may name a categorical column. + # Step 3b: Restore factor levels before the region filter frame <- apply_factor_levels(frame, data_dir, taxon) - # Step 3c: Alias columns a model set names under a different - # name than the dataset stores them under. The mammal formulas - # fit `seas_days`, which is summer_days or winter_days - # depending on which season is being modelled - one name for a - # column whose identity is part of the run's configuration. + # Step 3c: Aliases, e.g. mammal `seas_days` is summer_days or + # winter_days depending on the season modelled for (alias in names(aliases)) { source_column <- aliases[[alias]] @@ -548,9 +705,7 @@ build_model_data <- function( frame[[alias]] <- frame[[source_column]] } - # Step 4: Apply the region filter. It is a spec-supplied - # expression rather than a taxon rule known here, which is what - # keeps this file taxon-agnostic. + # Step 4: Apply the spec's region filter if (!is.null(region_filter)) { keep_rows <- eval( rlang_f_rhs(region_filter), @@ -596,18 +751,15 @@ build_model_data <- function( #' Cut a One-Species Modelling Frame #' -#' Adds the response, offset and weight for one species to the -#' shared covariate frame, under fixed names, so a model formula -#' and a fitting engine never have to know the species. +#' Adds one species' response, offset and weight under fixed +#' names, so formulas and engines never name the species. #' #' @param model_data A list from build_model_data(). #' @param species Character. One species column. -#' @param transform Character or function. "identity" leaves the -#' response alone; "detection" makes it 1 where the value is -#' above zero, which is how the plant scripts turn a cover -#' value into a presence. A function is applied as given, and -#' is passed the frame as a second argument when it takes one - -#' the mammal lure correction needs `lured` and `location`. +#' @param transform Character or function. "identity", or +#' "detection" (value > 0 becomes 1, as in the plant scripts). +#' A function taking two arguments also gets the frame (e.g. +#' the mammal lure correction needs `lured` and `location`). #' @param weight_column Character. A covariate column to use as #' model weights, or NULL. #' @param response_name,offset_name,weight_name Character. Column @@ -638,22 +790,17 @@ model_frame <- function( frame <- model_data$covariates values <- model_data$response[[species]] - # Step 1: Apply the response transform - # The untransformed values are kept alongside, because a stage - # may need a different transform of the same response. The - # mammal climate stage fits presence whichever half of the - # hurdle is being run, so the abundance run still needs the - # presence form to get its climate offset. + # Step 1: Apply the response transform, keeping raw values for + # stages that transform differently (the mammal climate stage + # fits presence even in the abundance run) frame[[paste0(response_name, "_raw")]] <- values frame[[response_name]] <- apply_response_transform( values, transform, frame ) - # Step 2: Attach the offset when the taxon has one. A species - # missing from the offsets file is an error rather than a zero: - # a silent zero offset would make that species' counts look - # like every other species' rates. + # Step 2: Attach the offset. A species missing from the offsets + # file is an error, not a silent zero offset. if (!is.null(model_data$offset)) { if (!species %in% names(model_data$offset)) { stop( @@ -705,7 +852,25 @@ check_dataset_file <- function(path) { invisible(NULL) } -## 10.2 align_to_units() ---- +## 10.2 site_columns() ---- + +#' The Site Fields Every Model Frame Carries +#' +#' Read from sites.csv, so filters and formulas can name them +#' without loading them as covariates. +#' +#' @return A character vector of sites.csv column names. +#' +#' @example # Example usage of the function +#' # site_columns() +site_columns <- function() { + c( + "lat", "long", "easting", "northing", "nr", "nsr", "luf", "year", + "lured", "location", "summer_days", "winter_days" + ) +} + +## 10.3 align_to_units() ---- #' Put a Keyed Frame in a Given Survey Unit Order #' @@ -725,7 +890,7 @@ align_to_units <- function(frame, units) { out } -## 10.3 apply_response_transform() ---- +## 10.4 apply_response_transform() ---- #' Turn a Recorded Value into a Modelling Response #' @@ -742,9 +907,7 @@ apply_response_transform <- function( values, transform, frame = NULL ) { if (is.function(transform)) { - # A transform that declares a second argument gets the - # frame, so a correction can depend on design columns - # rather than on the response alone. + # A two-argument transform also gets the frame if (length(formals(transform)) > 1) { return(transform(values, frame)) } @@ -764,12 +927,11 @@ apply_response_transform <- function( ) } -## 10.4 rlang_f_rhs() ---- +## 10.5 rlang_f_rhs() ---- #' Take the Right-Hand Side of a One-Sided Formula #' -#' Kept as a small local helper rather than a rlang dependency, -#' because the harness is otherwise base R plus data.table. +#' Local helper to avoid an rlang dependency. #' #' @param f A one-sided formula, e.g. ~ nr != "Grassland". #' @return The unevaluated right-hand side. diff --git a/1_code/harness/data_source.R b/1_code/harness/data_source.R new file mode 100644 index 0000000..c3d90ed --- /dev/null +++ b/1_code/harness/data_source.R @@ -0,0 +1,239 @@ +# --- +# title: Locate the Published Datasets +# author: Brendan Casey +# created: 2026-10-05 +# inputs: +# - the published datasets on ABMI-DATA2, each a versioned +# folder with its checksums.csv: +# //ABMI-DATA2/science/sdmMethodsDev/0_data/ +# test_dataset// the frozen test dataset +# v2_results// the v2 parity references +# - SDM_TEST_DATASET and SDM_V2_RESULTS, optional, folders to +# read instead +# outputs: none; returns the folder to read +# notes: +# - Runs read the copies published by +# _setup/10_publish_datasets.R, so only maintainers run +# _setup/. sdm_published$versions pins the version read; +# published versions are never overwritten. +# - The environment variables redirect to a local copy or to +# fresh _setup/ output in 0_data/. +# - Each call checks presence and size against checksums.csv +# (fast); verify_published() does the full md5 check. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # reads checksums.csv (version: 1.16.4) + +## 1.2 The pinned versions ---- +# `marker` is a file every copy holds, published or local, so a +# local folder without checksums.csv can still be recognized. +sdm_published <- list( + root = "//ABMI-DATA2/science/sdmMethodsDev/0_data", + versions = c( + test_dataset = "2026-10-05", + v2_results = "2026-10-05" + ), + env = c( + test_dataset = "SDM_TEST_DATASET", + v2_results = "SDM_V2_RESULTS" + ), + marker = c( + test_dataset = "lookup/dataset_manifest.csv", + v2_results = "abmiexplorer/v2_results.csv" + ) +) + +# 2. published_dir() ---- + +#' Resolve a Published Dataset Folder +#' +#' Precedence: `dir`, then the dataset's environment variable, +#' then the pinned published version. +#' +#' @param name Character. "test_dataset" or "v2_results". +#' @param dir Character or NULL. A folder to use as is. +#' @param check Logical. Check the folder before returning it. +#' @return Character. The folder, with forward slashes. +#' +#' @example # Example usage of the function +#' # published_dir("v2_results") +published_dir <- function(name, dir = NULL, check = TRUE) { + # Step 1: A dataset this file knows + if (!name %in% names(sdm_published$versions)) { + stop( + "Unknown dataset: ", name, ". Known: ", + paste(names(sdm_published$versions), collapse = ", "), + call. = FALSE + ) + } + + # Step 2: The first of argument, environment, pinned version + from_env <- Sys.getenv(sdm_published$env[[name]], unset = "") + + dir <- dir %||% + (if (nzchar(from_env)) from_env else NULL) %||% + file.path( + sdm_published$root, name, sdm_published$versions[[name]] + ) + + # normalizePath() keeps a share's leading backslashes; the run + # record reads better with the forward-slash form + dir <- normalizePath(dir, winslash = "/", mustWork = FALSE) + dir <- sub("^[\\\\/]{2}", "//", dir) + + # Step 3: Fail now, not partway into a run + if (isTRUE(check)) { + check_published(dir, name) + } + + dir +} + +# 3. test_dataset_dir() and v2_results_dir() ---- + +#' Resolve the Test Dataset Folder +#' +#' @param data_dir Character or NULL. A folder to use as is. +#' @param check Logical. Check the folder before returning it. +#' @return Character. The folder. +#' +#' @example # Example usage of the function +#' # data_dir <- test_dataset_dir() +#' # Sys.setenv(SDM_TEST_DATASET = "0_data/test_dataset") +#' # test_dataset_dir() # the local _setup/ output +test_dataset_dir <- function(data_dir = NULL, check = TRUE) { + published_dir("test_dataset", data_dir, check) +} + +#' Resolve the v2 Results Folder +#' +#' The parity reference, from _setup/08, is in abmiexplorer/. +#' Files at the top level of the 2026-10-05 version are the +#' retired network-drive build; nothing reads them. +#' +#' @param dir Character or NULL. A folder to use as is. +#' @param check Logical. Check the folder before returning it. +#' @return Character. The folder. +#' +#' @example # Example usage of the function +#' # file.path(v2_results_dir(), "abmiexplorer", "v2_results.csv") +v2_results_dir <- function(dir = NULL, check = TRUE) { + published_dir("v2_results", dir, check) +} + +# 4. check_published() ---- + +#' Check a Dataset Folder Is Complete +#' +#' Published copies are checked file by file against +#' checksums.csv (presence and size); a local copy without one is +#' only checked for the dataset's marker file. +#' +#' @param dir Character. The folder. +#' @param name Character. The dataset, for its marker file. +#' @return `dir`, invisibly; stops on a problem. +#' +#' @example # Example usage of the function +#' # check_published(test_dataset_dir(check = FALSE), +#' # "test_dataset") +check_published <- function(dir, name) { + # Step 1: The folder + if (!dir.exists(dir)) { + stop( + "Dataset ", name, " not found:\n ", dir, + "\nConnect to the ABMI network (or VPN), or set ", + sdm_published$env[[name]], " to a local copy.", + call. = FALSE + ) + } + + # Step 2: A local copy has no checksums; the marker will do + sums_path <- file.path(dir, "checksums.csv") + + if (!file.exists(sums_path)) { + marker <- sdm_published$marker[[name]] + + if (!file.exists(file.path(dir, marker))) { + stop( + "Not a copy of ", name, " (no ", marker, "):\n ", dir, + call. = FALSE + ) + } + + return(invisible(dir)) + } + + # Step 3: Every published file, at its published size + sums <- fread(sums_path, colClasses = "character") + sizes <- file.size(file.path(dir, sums$path)) + bad <- is.na(sizes) | sizes != as.numeric(sums$bytes) + + if (any(bad)) { + stop( + "Dataset ", name, " incomplete at ", dir, + "; missing or wrong size:\n ", + paste(head(sums$path[bad], 10), collapse = "\n "), + call. = FALSE + ) + } + + invisible(dir) +} + +# 5. verify_published() ---- + +#' Verify Every File in a Published Dataset by md5 +#' +#' Slow over the network (the test dataset is ~0.9 GB); use it +#' after publishing, or when a copy is in doubt. +#' +#' @param dir Character. A published folder. +#' @return Logical. TRUE when every md5 matches, invisibly; +#' stops otherwise. +#' +#' @example # Example usage of the function +#' # verify_published(test_dataset_dir()) +verify_published <- function(dir) { + sums_path <- file.path(dir, "checksums.csv") + + if (!file.exists(sums_path)) { + stop("No checksums.csv in ", dir, call. = FALSE) + } + + sums <- fread(sums_path, colClasses = "character") + found <- unname(tools::md5sum(file.path(dir, sums$path))) + bad <- is.na(found) | found != sums$md5 + + if (any(bad)) { + stop( + "md5 mismatch in ", dir, ":\n ", + paste(sums$path[bad], collapse = "\n "), + call. = FALSE + ) + } + + invisible(TRUE) +} + +# 6. published_label() ---- + +#' Name the Copy a Run Read, for the Run Record +#' +#' @param dir Character. The folder. +#' @return Character. "published ()" for a +#' published copy, "local ()" otherwise. +#' +#' @example # Example usage of the function +#' # published_label(test_dataset_dir()) +published_label <- function(dir) { + if (file.exists(file.path(dir, "checksums.csv"))) { + paste0("published ", basename(dir), " (", dir, ")") + } else { + paste0("local (", dir, ")") + } +} + +# End of script ---- diff --git a/1_code/harness/engines.R b/1_code/harness/engines.R index 11d4244..7bb25bf 100644 --- a/1_code/harness/engines.R +++ b/1_code/harness/engines.R @@ -1,409 +1,352 @@ # --- -# title: Fitting Engines +# title: Engine Interface # author: Brendan Casey # created: 2026-09-09 # inputs: none -# outputs: none; returns objects in memory +# outputs: none; defines functions in memory # notes: -# - A engine is a fitting method behind one interface, so that -# swapping GLM for a boosted tree is a one-word change in an -# experiment rather than a rewrite. Every engine supplies -# fit, coef, predict and ic; an engine that has no -# coefficients returns NULL from coef, and the result writer -# records that rather than failing. -# - Registered here: `glm` and `bayesglm`. `gbm` and `brms` -# come later; the registry is what they slot into. -# - bayesglm and glm sit on different axes and should not be -# read as "frequentist versus Bayesian". bayesglm is -# penalized maximum likelihood - augmented IRLS under weakly -# informative Cauchy priors - giving a point estimate and a -# standard error, with no posterior. It belongs on a -# regularization axis. A Bayesian framework comparison needs -# brms or rstanarm. -# - The plant v2 models use bayesglm because rare species fitted -# against 30-term habitat formulas separate completely, and -# plain glm then returns infinite coefficients. The -# regularization is load-bearing there, not incidental, so -# swapping plants to glm as a control will fail outright for -# the rare end of the species list rather than give a clean -# comparison. -# - predict() takes an `se` flag. Inverse-variance averaging -# weights each candidate by the precision of its prediction, -# so an engine that cannot return a standard error cannot -# use that selection rule. Tree and boosting engines will -# need a different weighting when they arrive. -# - Fitting failures are caught and returned, not raised. A -# candidate model set is expected to contain models that -# cannot be estimated for a given species and bootstrap, and -# the selection rule needs to see which ones failed. +# - The harness's interface to the engines in +# 1_code/methods/engines/, so swapping engines is a one-word +# spec change. The contract is in 1_code/methods/README.md; +# check_engine() tests an engine against it on synthetic data. # --- # 1. Setup ---- ## 1.1 Load packages ---- -# arm is loaded lazily by the bayesglm engine rather than here, -# so a run that never uses it does not need it installed. +# Base R only. -# 2. Engine registry ---- - -## 2.1 engine_registry() ---- - -#' The Available Fitting Engines -#' -#' Each entry supplies four functions: `fit`, `coef`, `predict` -#' and `ic`. Adding an engine means adding an entry, not -#' touching the fitting loop. -#' -#' @return A named list of engine definitions. -#' -#' @example # Example usage of the function -#' # names(engine_registry()) -engine_registry <- function() { - list( - glm = engine_glm(), - bayesglm = engine_bayesglm() - ) -} - -## 2.2 get_engine() ---- +# 2. get_engine() ---- #' Look Up One Engine #' -#' @param name Character. Engine name. -#' @param registry Named list of engine definitions. +#' @param name Character. A registered engine name. #' @return The engine definition. #' #' @example # Example usage of the function #' # get_engine("glm") -get_engine <- function(name, registry = engine_registry()) { - if (!name %in% names(registry)) { - stop( - "Unknown engine `", name, "`. Registered: ", - paste(names(registry), collapse = ", "), - call. = FALSE - ) - } - - registry[[name]] +get_engine <- function(name) { + get_method("engine", name) } -# 3. engine_glm() ---- +# 3. Capabilities ---- -#' Plain Generalized Linear Model +## 3.1 engine_has() ---- + +#' Does an Engine Declare a Capability #' -#' @return An engine definition. +#' @param engine An engine definition. +#' @param capability Character. One of engine_capabilities(). +#' @return Logical. #' #' @example # Example usage of the function -#' # e <- engine_glm() -#' # fit <- e$fit(response ~ MAP, data, family = "binomial") -engine_glm <- function() { - list( - name = "glm", - has_coefficients = TRUE, - fit = function(formula, data, family, weights = NULL, - offset = NULL, control = list()) { - fit_glm_family( - stats::glm, formula, data, family, weights, offset, - control - ) - }, - coef = function(fit) fit_coefficients(fit), - predict = function(fit, newdata, type = "link", se = FALSE) { - fit_predict(fit, newdata, type, se) - }, - ic = function(fit, type = "AICc") information_criterion(fit, type) - ) +#' # engine_has(get_engine("glm"), "se") +engine_has <- function(engine, capability) { + capability %in% engine$capabilities } -# 4. engine_bayesglm() ---- +## 3.2 engine_coef() ---- -#' Penalized GLM with Weakly Informative Priors +#' Coefficients from a Fit, Where the Engine Has Them #' -#' `arm::bayesglm` with its default Cauchy priors, which is what -#' the v2 plant models use. See the header on why this is not a -#' Bayesian framework in the sense a framework comparison means. +#' Reuses coefficients cached on the fit by fit_with(). #' -#' @return An engine definition. +#' @param engine An engine definition. +#' @param fit A fit from engine$fit(). +#' @return A data frame of term, estimate and se, or NULL. #' #' @example # Example usage of the function -#' # e <- engine_bayesglm() -engine_bayesglm <- function() { - list( - name = "bayesglm", - has_coefficients = TRUE, - fit = function(formula, data, family, weights = NULL, - offset = NULL, control = list()) { - if (!requireNamespace("arm", quietly = TRUE)) { - stop( - "The bayesglm engine needs the arm package.", - call. = FALSE - ) - } +#' # engine_coef(engine, fit) +engine_coef <- function(engine, fit) { + if (is.null(fit) || !isTRUE(fit$ok)) { + return(NULL) + } - # maxit defaults to the v2 value, which was raised from the - # arm default because rare species converge slowly - if (is.null(control$maxit)) { - control$maxit <- 250 - } + if (!is.null(fit$coefficients)) { + return(fit$coefficients) + } - fit_glm_family( - arm::bayesglm, formula, data, family, weights, offset, - control - ) - }, - coef = function(fit) fit_coefficients(fit), - predict = function(fit, newdata, type = "link", se = FALSE) { - fit_predict(fit, newdata, type, se) - }, - ic = function(fit, type = "AICc") information_criterion(fit, type) - ) -} + if (!engine_has(engine, "coefficients")) { + return(NULL) + } -# 5. Shared implementations ---- -# glm and bayesglm take the same arguments and return objects of -# the same shape, so the four operations are written once. + engine$coef(fit) +} -## 5.1 fit_glm_family() ---- +## 3.3 engine_converged() ---- -#' Fit a GLM-Family Model, Catching Failure +#' Did a Fit Converge, as Far as the Engine Can Say #' -#' @param fitter Function. stats::glm or arm::bayesglm. -#' @param formula A model formula. -#' @param data A data frame. -#' @param family Character or family object. -#' @param weights Numeric vector or NULL. -#' @param offset Numeric vector or NULL. -#' @param control Named list of extra arguments to the fitter. -#' @return A list with `fit` (or NULL), `ok`, and `message`. +#' @param engine An engine definition. +#' @param fit A fit from engine$fit(). +#' @return Logical. FALSE for a failed fit; TRUE for a fit whose +#' engine cannot report convergence. #' #' @example # Example usage of the function -#' # fit_glm_family(stats::glm, y ~ x, d, "binomial") -fit_glm_family <- function( - fitter, - formula, - data, - family, - weights = NULL, - offset = NULL, - control = list() -) { - args <- c( - list(formula = formula, data = data, family = family), - control - ) - - # Weights and offsets are passed as values rather than named in - # the formula, so the same formula serves a taxon that has them - # and one that does not. - if (!is.null(weights)) { - args$weights <- weights +#' # engine_converged(engine, fit) +engine_converged <- function(engine, fit) { + if (is.null(fit) || !isTRUE(fit$ok)) { + return(FALSE) } - if (!is.null(offset)) { - args$offset <- offset + if (!is.function(engine$converged)) { + return(TRUE) } - # A candidate set is expected to hold models that cannot be - # estimated for a given species and draw. The selection rule - # needs to see which failed, so failure is returned, not raised. - # - # A convergence warning still leaves a usable fit, so it is - # recorded rather than treated as a failure. It is caught with - # withCallingHandlers() and muffled, so the model is fitted - # once; catching it with tryCatch() would abandon the fit and - # force a second one. - warnings_seen <- character(0) - - tryCatch( - { - fit <- withCallingHandlers( - do.call(fitter, args), - warning = function(w) { - warnings_seen <<- c(warnings_seen, conditionMessage(w)) - invokeRestart("muffleWarning") - } - ) - - list( - fit = fit, - ok = TRUE, - message = if (length(warnings_seen) == 0) { - NA_character_ - } else { - paste(unique(warnings_seen), collapse = "; ") - } - ) - }, - error = function(e) { - list(fit = NULL, ok = FALSE, message = conditionMessage(e)) - } - ) + isTRUE(engine$converged(fit)) } -## 5.2 fit_coefficients() ---- +## 3.4 engine_nobs() ---- -#' Take Coefficients and Standard Errors from a Fit +#' Number of Observations a Fit Used #' -#' @param fit A list from fit_glm_family(). -#' @return A data frame of term, estimate and se, or NULL when -#' the fit failed. +#' @param engine An engine definition, or NULL. +#' @param fit A fit from engine$fit(). +#' @return An integer, or NA when it cannot be said. #' #' @example # Example usage of the function -#' # fit_coefficients(fit) -fit_coefficients <- function(fit) { - if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { - return(NULL) +#' # engine_nobs(engine, fit) +engine_nobs <- function(engine, fit) { + if (is.null(fit) || !isTRUE(fit$ok)) { + return(NA_integer_) } - estimates <- stats::coef(fit$fit) - errors <- tryCatch( - stats::coef(summary(fit$fit))[, "Std. Error"], - error = function(e) rep(NA_real_, length(estimates)) - ) + if (!is.null(engine) && is.function(engine$nobs)) { + return(tryCatch( + as.integer(engine$nobs(fit)), + error = function(e) NA_integer_ + )) + } - data.frame( - term = names(estimates), - estimate = as.numeric(estimates), - se = as.numeric(errors[match(names(estimates), names(errors))]), - stringsAsFactors = FALSE + tryCatch( + as.integer(stats::nobs(fit$fit)), + error = function(e) NA_integer_ ) } -## 5.3 fit_predict() ---- +## 3.5 fit_with() ---- -#' Predict from a Fit +#' Fit One Formula Through an Engine +#' +#' The one place a candidate is fitted. Coefficients are computed +#' once and cached on the fit (bayesglm's summary() is slow). #' -#' @param fit A list from fit_glm_family(). -#' @param newdata A data frame to predict onto. -#' @param type Character. "link" or "response". -#' @param se Logical. Return standard errors alongside the fit. -#' Inverse-variance averaging needs them, and an engine that -#' cannot supply them cannot use that selection rule. -#' @return A numeric vector, or when `se` is TRUE a list of `fit` -#' and `se.fit`. NULL when the fit failed. +#' @param engine An engine definition. +#' @param formula A model formula. +#' @param data,family,weights,offset,control As engine$fit takes. +#' @return The engine's fit list, with `coefficients` added when +#' the engine has them. #' #' @example # Example usage of the function -#' # fit_predict(fit, prediction_grid, type = "response") -#' # fit_predict(fit, grid, "link", se = TRUE)$se.fit -fit_predict <- function(fit, newdata, type = "link", se = FALSE) { - if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { - return(NULL) - } - - tryCatch( - { - predicted <- stats::predict( - fit$fit, newdata = newdata, type = type, se.fit = se - ) +#' # fit_with(get_engine("glm"), response ~ MAP, d, "binomial") +fit_with <- function( + engine, formula, data, family, weights = NULL, offset = NULL, + control = list() +) { + fit <- engine$fit( + formula = formula, data = data, family = family, + weights = weights, offset = offset, control = control + ) - if (!se) { - return(as.numeric(predicted)) - } + if (isTRUE(fit$ok) && engine_has(engine, "coefficients")) { + fit$coefficients <- engine$coef(fit) + } - list( - fit = as.numeric(predicted$fit), - se.fit = as.numeric(predicted$se.fit) - ) - }, - error = function(e) NULL - ) + fit } -## 5.4 information_criterion() ---- +# 4. check_engine() ---- -#' Score a Fit for Model Selection +#' Test an Engine Against the Contract #' -#' AICc rather than AIC, because the candidate sets are large -#' relative to the number of survey units a rare species is -#' detected at. +#' Fits synthetic binomial and Poisson data (weights, a formula +#' offset, a factor) and checks each part of the contract the +#' engine declares. #' -#' @param fit A list from fit_glm_family(). -#' @param type Character. "AIC", "AICc" or "BIC". -#' @return A numeric value; Inf when the fit failed, so a failed -#' model never wins a selection. +#' @param engine An engine definition, or a registered name. +#' @param n Integer. Rows of synthetic data. +#' @param seed Integer. Seed for the synthetic data. +#' @param quiet Logical. Suppress the printed table. +#' @return A data frame of check, family, pass and note, +#' invisibly. Every `pass` should be TRUE. #' #' @example # Example usage of the function -#' # information_criterion(fit, "BIC") -information_criterion <- function(fit, type = "AICc") { - if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { - return(Inf) +#' # check_engine("glm") +#' # check_engine(my_new_engine()) +check_engine <- function(engine, n = 400L, seed = 1L, quiet = FALSE) { + if (is.character(engine)) { + engine <- get_engine(engine) + } + + # Step 1: Synthetic data with a known signal + data <- with_seed(seed, { + x1 <- stats::rnorm(n) + x2 <- stats::runif(n) + habitat <- factor(sample(c("A", "B", "C"), n, replace = TRUE)) + effort <- stats::runif(n, 0.5, 2) + eta <- -0.5 + 0.8 * x1 - 0.6 * x2 + (habitat == "B") * 0.7 + + data.frame( + x1 = x1, x2 = x2, habitat = habitat, + offset = log(effort), + weight = stats::runif(n, 0.5, 1.5), + presence = stats::rbinom(n, 1, stats::plogis(eta)), + count = stats::rpois(n, exp(eta + log(effort))) + ) + }) + + checks <- list() + + record <- function(check, family, pass, note = "") { + checks[[length(checks) + 1]] <<- data.frame( + check = check, family = family, pass = isTRUE(pass), + note = note, stringsAsFactors = FALSE + ) } - value <- tryCatch( - switch( - type, - AIC = stats::AIC(fit$fit), - BIC = stats::BIC(fit$fit), - AICc = { - # The parameter count is the log-likelihood's degrees of - # freedom, as MuMIn uses, not the length of the - # coefficient vector, which also counts aliased (NA) - # terms and so over-penalizes a rank-deficient model. - k <- attr(stats::logLik(fit$fit), "df") - n <- stats::nobs(fit$fit) - # The correction is undefined once the parameter count - # reaches the sample size; Inf keeps such a model out of - # the selection rather than returning a negative score. - if (is.na(n) || n - k - 1 <= 0) { - Inf + newdata <- data[1:20, ] + + for (family in c("binomial", "poisson")) { + formula <- if (family == "binomial") { + presence ~ x1 + x2 + habitat + } else { + count ~ x1 + x2 + habitat + offset(offset) + } + + # Step 2: Fitting succeeds, weighted, and reports itself + fit <- tryCatch( + engine$fit( + formula = formula, data = data, family = family, + weights = data$weight, offset = NULL, control = list() + ), + error = function(e) { + list(ok = FALSE, message = conditionMessage(e)) + } + ) + record( + "fit returns list(fit, ok, message) with ok = TRUE", family, + is.list(fit) && isTRUE(fit$ok) && + all(c("fit", "ok", "message") %in% names(fit)), + if (!isTRUE(fit$ok)) format(fit$message) else "" + ) + + if (!isTRUE(fit$ok)) { + next + } + + # Step 3: Predictions on both scales + link <- tryCatch( + engine$predict(fit, newdata, "link"), + error = function(e) NULL + ) + response <- tryCatch( + engine$predict(fit, newdata, "response"), + error = function(e) NULL + ) + record( + "predict returns one finite value per row", family, + is.numeric(link) && length(link) == nrow(newdata) && + all(is.finite(link)) + ) + record( + "response-scale prediction is in range", family, + is.numeric(response) && length(response) == nrow(newdata) && + if (family == "binomial") { + all(response >= 0 & response <= 1) } else { - stats::AIC(fit$fit) + (2 * k * (k + 1)) / (n - k - 1) + all(response >= 0) } - }, - stop( - "Unknown information criterion `", type, "`.", - call. = FALSE + ) + + # Doubling effort on new data must double a Poisson prediction + if (family == "poisson") { + shifted <- newdata + shifted$offset <- shifted$offset + log(2) + doubled <- tryCatch( + engine$predict(fit, shifted, "response"), + error = function(e) NULL ) - ), - error = function(e) Inf - ) + record( + "an offset in the formula is applied to new data", family, + is.numeric(doubled) && + isTRUE(all.equal(doubled, 2 * response, tolerance = 1e-6)) + ) + } - if (is.na(value)) Inf else value -} + # Step 4: The optional capabilities the engine declares + if (engine_has(engine, "coefficients")) { + cf <- tryCatch(engine$coef(fit), error = function(e) NULL) + record( + "coef returns term, estimate and se", family, + is.data.frame(cf) && + all(c("term", "estimate", "se") %in% names(cf)) && + nrow(cf) > 0 + ) + } -## 5.5 fit_converged() ---- + if (engine_has(engine, "ic")) { + scores <- vapply( + c("AIC", "AICc", "BIC"), + function(type) { + tryCatch(engine$ic(fit, type), error = function(e) NA_real_) + }, + numeric(1) + ) + record( + "ic returns a finite AIC, AICc and BIC", family, + all(is.finite(scores)) + ) + } -#' Did a Fit Converge -#' -#' Inverse-variance averaging drops non-converged candidates -#' rather than down-weighting them, which is what the v2 plant -#' code does. -#' -#' @param fit A list from fit_glm_family(). -#' @return Logical. FALSE when the fit failed or did not -#' converge. -#' -#' @example # Example usage of the function -#' # fit_converged(fit) -fit_converged <- function(fit) { - if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { - return(FALSE) + if (engine_has(engine, "se")) { + with_se <- tryCatch( + engine$predict(fit, newdata, "link", se = TRUE), + error = function(e) NULL + ) + record( + "predict(se = TRUE) returns fit and se.fit", family, + is.list(with_se) && + length(with_se$fit) == nrow(newdata) && + length(with_se$se.fit) == nrow(newdata) && + all(with_se$se.fit >= 0) + ) + } } - converged <- fit$fit$converged + # Step 5: Failure is returned, not raised, so a selection rule + # can see which candidates failed + failed <- tryCatch( + engine$fit( + formula = presence ~ no_such_column, data = data, + family = "binomial", weights = NULL, offset = NULL, + control = list() + ), + error = function(e) "raised" + ) + record( + "a failed fit returns ok = FALSE rather than raising", "-", + is.list(failed) && identical(failed$ok, FALSE) + ) - if (is.null(converged)) { - return(TRUE) + if (is.list(failed)) { + record( + "predict on a failed fit returns NULL", "-", + is.null(tryCatch( + engine$predict(failed, newdata, "response"), + error = function(e) "raised" + )) + ) } - isTRUE(converged) -} - -## 5.6 fit_nobs() ---- + result <- do.call(rbind, checks) -#' Number of Observations a Fit Used -#' -#' @param fit A list from fit_glm_family(). -#' @return An integer, or NA when the fit failed. -#' -#' @example # Example usage of the function -#' # fit_nobs(fit) -fit_nobs <- function(fit) { - if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { - return(NA_integer_) + if (!quiet) { + cat("Engine `", engine$name, "`: ", sum(result$pass), " of ", + nrow(result), " checks pass\n", sep = "") + print(result, row.names = FALSE, right = FALSE) } - as.integer(stats::nobs(fit$fit)) + invisible(result) } # End of script ---- diff --git a/1_code/harness/eval_metrics.R b/1_code/harness/eval_metrics.R index c05cd82..100b875 100644 --- a/1_code/harness/eval_metrics.R +++ b/1_code/harness/eval_metrics.R @@ -5,51 +5,40 @@ # inputs: none # outputs: none; returns objects in memory # notes: -# - Metrics computed here are what experiments are compared on, -# so a change affects every past and future experiment and -# should be reviewed on that basis. -# - Every metric takes observed and predicted vectors and -# returns one number, so the set is extended by adding an -# entry to metric_registry() rather than by touching the -# loop. -# - Metrics are computed from predictions, never from a fitted -# object, so a GLM and a boosted tree are scored the same way. -# That is the whole reason predictions rather than -# coefficients are the comparison currency. -# - AUC is computed directly rather than through pROC, so the -# harness does not depend on a package for one number, and so -# ties are handled explicitly. It is the Mann-Whitney form: -# the probability a random detection scores above a random -# non-detection, with ties counting a half. -# - A metric that cannot be computed returns NA rather than -# stopping. A bootstrap where a species was never detected -# has no AUC, and that is a fact about the draw, not an error. +# - Experiments are compared on these metrics, so a change here +# affects every experiment. Metrics live in +# 1_code/methods/metrics/ and score response-scale +# predictions, never a fitted object, so all engines are +# scored alike. +# - An uncomputable metric returns NA rather than stopping (e.g. +# no AUC for a draw with no detections). # --- # 1. Setup ---- ## 1.1 Load packages ---- -# Base R only, deliberately; see the header. +# Base R only. -# 2. metric_registry() ---- +# 2. default_metrics() ---- -#' The Available Metrics +#' The Metrics Scored When a Run Names None #' -#' @return A named list of functions taking `observed` and -#' `predicted` and returning one number. +#' Every registered metric: the seven original ones first, in +#' their original order, so tables stay stable; later additions +#' follow alphabetically. +#' +#' @return A character vector of metric names. #' #' @example # Example usage of the function -#' # names(metric_registry()) -metric_registry <- function() { - list( - auc = metric_auc, - deviance_explained = metric_deviance_explained, - rmse = metric_rmse, - spearman = metric_spearman, - calibration_slope = metric_calibration_slope, - prevalence = metric_prevalence, - n = metric_n +#' # default_metrics() +default_metrics <- function() { + core <- c( + "auc", "deviance_explained", "rmse", "spearman", + "calibration_slope", "prevalence", "n" ) + registered <- registered_names("metric") + + c(intersect(core, registered), sort(setdiff(registered, core))) } # 3. compute_metrics() ---- @@ -60,46 +49,40 @@ metric_registry <- function() { #' @param predicted Numeric vector of predictions, on the #' response scale. #' @param metrics Character vector of metric names, or NULL for -#' every registered metric. +#' default_metrics(). #' @return A data frame of metric and value. #' #' @example # Example usage of the function #' # compute_metrics(obs, pred, metrics = c("auc", "rmse")) compute_metrics <- function(observed, predicted, metrics = NULL) { - registry <- metric_registry() - if (is.null(metrics)) { - metrics <- names(registry) + metrics <- default_metrics() } - unknown <- setdiff(metrics, names(registry)) - - if (length(unknown) > 0) { - stop( - "Unknown metric(s): ", paste(unknown, collapse = ", "), - ". Registered: ", paste(names(registry), collapse = ", "), - call. = FALSE - ) - } + # Looked up first, so a misspelt metric stops the run here + functions <- lapply( + stats::setNames(metrics, metrics), + function(name) get_method("metric", name) + ) - # Step 1: Drop pairs where either side is missing, so every + # Step 1: Drop pairs where either side is non-finite, so every # metric sees the same rows keep <- is.finite(observed) & is.finite(predicted) observed <- observed[keep] predicted <- predicted[keep] - # Step 2: Score, letting an uncomputable metric return NA + # Step 2: Score, letting an uncomputable metric return NA. A + # warning is muffled rather than caught, so the metric is + # computed once. values <- vapply( metrics, function(name) { tryCatch( - as.numeric(registry[[name]](observed, predicted)), - error = function(e) NA_real_, - warning = function(w) { - suppressWarnings( - as.numeric(registry[[name]](observed, predicted)) - ) - } + withCallingHandlers( + as.numeric(functions[[name]](observed, predicted)), + warning = function(w) invokeRestart("muffleWarning") + ), + error = function(e) NA_real_ ) }, numeric(1) @@ -112,169 +95,4 @@ compute_metrics <- function(observed, predicted, metrics = NULL) { ) } -# 4. Metric implementations ---- - -## 4.1 metric_auc() ---- - -#' Area Under the ROC Curve -#' -#' The Mann-Whitney form: the probability that a randomly chosen -#' detection scores above a randomly chosen non-detection, with -#' ties counting a half. Values above zero are treated as -#' detections, so a count or a density scores the same way a -#' presence does. -#' -#' @param observed,predicted Numeric vectors. -#' @return A number between 0 and 1, or NA when the observations -#' are all detections or all non-detections. -#' -#' @example # Example usage of the function -#' # metric_auc(c(0, 0, 1, 1), c(0.1, 0.3, 0.4, 0.9)) -metric_auc <- function(observed, predicted) { - positive <- observed > 0 - - # An AUC needs both classes present. A bootstrap where a - # species was never detected has none, which is a fact about - # the draw rather than a failure. - if (!any(positive) || all(positive)) { - return(NA_real_) - } - - ranks <- rank(predicted, ties.method = "average") - - # Counted as doubles, not integers. sum() of a logical returns - # an integer, and at bird scale - 16,745 detections against - # 199,013 non-detections - their product is 3.3e9, past the - # 32-bit maximum. That overflows to NA silently, so every large - # run would report no AUC at all. - n_pos <- as.numeric(sum(positive)) - n_neg <- as.numeric(sum(!positive)) - - (sum(ranks[positive]) - n_pos * (n_pos + 1) / 2) / - (n_pos * n_neg) -} - -## 4.2 metric_deviance_explained() ---- - -#' Proportion of Null Deviance Explained -#' -#' Poisson deviance for counts, Bernoulli for detections, chosen -#' from whether the observations are all zero or one. -#' -#' @param observed,predicted Numeric vectors. -#' @return A number, or NA when the null deviance is zero. -#' -#' @example # Example usage of the function -#' # metric_deviance_explained(c(0, 1, 2), c(0.2, 0.9, 1.8)) -metric_deviance_explained <- function(observed, predicted) { - binary <- all(observed %in% c(0, 1)) - - deviance <- function(y, mu) { - mu <- pmax(mu, 1e-10) - - if (binary) { - mu <- pmin(mu, 1 - 1e-10) - -2 * sum(y * log(mu) + (1 - y) * log(1 - mu)) - } else { - terms <- ifelse(y > 0, y * log(y / mu), 0) - 2 * sum(terms - (y - mu)) - } - } - - null_deviance <- deviance( - observed, rep(mean(observed), length(observed)) - ) - - if (!is.finite(null_deviance) || null_deviance <= 0) { - return(NA_real_) - } - - 1 - deviance(observed, predicted) / null_deviance -} - -## 4.3 metric_rmse() ---- - -#' Root Mean Squared Error -#' -#' @param observed,predicted Numeric vectors. -#' @return A number. -#' -#' @example # Example usage of the function -#' # metric_rmse(c(0, 1), c(0.1, 0.8)) -metric_rmse <- function(observed, predicted) { - sqrt(mean((observed - predicted)^2)) -} - -## 4.4 metric_spearman() ---- - -#' Rank Correlation of Observed and Predicted -#' -#' Scale-free, so it compares a density model against a count -#' model without either being rescaled. -#' -#' @param observed,predicted Numeric vectors. -#' @return A number between -1 and 1. -#' -#' @example # Example usage of the function -#' # metric_spearman(c(0, 1, 2), c(0.1, 0.5, 0.9)) -metric_spearman <- function(observed, predicted) { - if (stats::sd(observed) == 0 || stats::sd(predicted) == 0) { - return(NA_real_) - } - - stats::cor(observed, predicted, method = "spearman") -} - -## 4.5 metric_calibration_slope() ---- - -#' Calibration Slope -#' -#' The slope of observed on predicted. One is perfect -#' calibration; below one means the predictions are too extreme, -#' which is the usual signature of overfitting and the thing a -#' regularized fit is trying to fix. -#' -#' @param observed,predicted Numeric vectors. -#' @return A number. -#' -#' @example # Example usage of the function -#' # metric_calibration_slope(c(0, 1, 1), c(0.2, 0.6, 0.8)) -metric_calibration_slope <- function(observed, predicted) { - if (stats::sd(predicted) == 0) { - return(NA_real_) - } - - unname(stats::coef(stats::lm(observed ~ predicted))[2]) -} - -## 4.6 metric_prevalence() ---- - -#' Proportion of Survey Units with a Detection -#' -#' Carried alongside the scores because most of them are only -#' interpretable against it: an AUC of 0.9 means something -#' different at 2 per cent prevalence than at 40. -#' -#' @param observed,predicted Numeric vectors. -#' @return A number between 0 and 1. -#' -#' @example # Example usage of the function -#' # metric_prevalence(c(0, 0, 1), c(0, 0, 0)) -metric_prevalence <- function(observed, predicted) { - mean(observed > 0) -} - -## 4.7 metric_n() ---- - -#' Number of Survey Units Scored -#' -#' @param observed,predicted Numeric vectors. -#' @return An integer. -#' -#' @example # Example usage of the function -#' # metric_n(c(0, 1), c(0.2, 0.7)) -metric_n <- function(observed, predicted) { - length(observed) -} - # End of script ---- diff --git a/1_code/harness/experiment.R b/1_code/harness/experiment.R new file mode 100644 index 0000000..fc28df1 --- /dev/null +++ b/1_code/harness/experiment.R @@ -0,0 +1,541 @@ +# --- +# title: Configure and Run an Experiment +# author: Brendan Casey +# created: 2026-10-03 +# inputs: +# the published test dataset (see data_source.R) +# outputs: +# in 2_pipeline//: the result stores and run_log.csv +# in 3_output//: tables/, run_record.md, and whatever the +# experiment's own steps write +# notes: +# - The part every experiment shares, so each run.R is a short +# list of decisions. experiment_config() checks them before +# anything is fitted; run_experiment() fits, summarizes, +# compares with the baseline (exp_000, v2), runs the +# experiment's own steps and writes the run record. +# - Steps receive everything through `config` and `results`. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # run log writing (version: 1.16.4) + +# 2. experiment_config() ---- + +#' State and Check an Experiment's Configuration +#' +#' @param id Character. The experiment identifier, e.g. +#' "exp_001_covariate_scale". Names its 2_pipeline/ and +#' 3_output/ folders. +#' @param taxa Character vector of runs, from +#' names(standard_specs()), or NULL for all of them. +#' @param species A focal-species set name (see +#' focal_species_sets()), a species vector named by taxon, or +#' NULL for every species. +#' @param n_bootstraps Integer. Draws per species. 100 is v2's. +#' @param seed Integer or NULL. The base seed; NULL leaves the +#' draws unseeded, as v2 did. +#' @param stage_models Named list of replacement candidate sets, +#' or NULL for each spec's v2 sets; see apply_stage_models(). +#' @param covariate_files Character vector of CSVs holding +#' covariates the frozen dataset does not, keyed on +#' survey_unit_id, or NULL. Written by the experiment's own +#' scripts, never into 0_data/; their columns can then be named +#' in `stage_models`. +#' @param specs Named list of specs, or NULL for +#' standard_specs(plant_bootstrap). An experiment that changes +#' a spec builds its own and passes it here. +#' @param plant_bootstrap Character. Passed to standard_specs(). +#' @param workers Integer. Species fitted at once. +#' @param unit_predictions Character. "none", "oob" or "all". +#' @param baseline Character. The experiment every result is +#' compared with, by id; NULL for none. Ignored by the baseline +#' itself. +#' @param v2_bootstraps Integer. v2's draw count, which a run must +#' reach to be gated. +#' @param project_root Character. The repository root. +#' @param data_dir,pipeline_dir,out_dir Character, or NULL for +#' test_dataset_dir(), 2_pipeline/ and 3_output/. +#' @return A list of class `experiment_config`. +#' +#' @example # Example usage of the function +#' # config <- experiment_config("exp_001_test", taxa = "lichen", +#' # species = "one_each", +#' # n_bootstraps = 5) +experiment_config <- function( + id, + taxa = NULL, + species = NULL, + n_bootstraps = 100L, + seed = 20260909L, + stage_models = NULL, + covariate_files = NULL, + specs = NULL, + plant_bootstrap = "spatial_block", + workers = 1L, + unit_predictions = "none", + baseline = "exp_000_parity_v2", + v2_bootstraps = 100L, + project_root = getOption("sdm.project_root", "."), + data_dir = NULL, + pipeline_dir = NULL, + out_dir = NULL +) { + # Step 1: The runs + specs <- specs %||% standard_specs(plant_bootstrap) + taxa <- taxa %||% names(specs) + unknown <- setdiff(taxa, names(specs)) + + if (length(unknown) > 0) { + stop( + "Unknown run(s): ", paste(unknown, collapse = ", "), + ". Available: ", paste(names(specs), collapse = ", "), + call. = FALSE + ) + } + + specs <- specs[taxa] + + # Step 2: The draw count, checked now rather than hours in + if (!is.numeric(n_bootstraps) || length(n_bootstraps) != 1L || + is.na(n_bootstraps) || n_bootstraps < 1 || + n_bootstraps != round(n_bootstraps)) { + stop( + "n_bootstraps must be a single whole number of 1 or more.", + call. = FALSE + ) + } + + n_bootstraps <- as.integer(n_bootstraps) + data_dir <- test_dataset_dir(data_dir) + + # Stored draws have a hard ceiling; drawn ones do not + for (key in names(specs)) { + spec <- specs[[key]] + + if (!identical(spec$resample$scheme, "precomputed") || + isTRUE(spec$resample$per_species)) { + next + } + + location <- manifest_entry(data_dir, spec$taxon)$bootstrap_ids + stored <- length(names(fread( + file.path(data_dir, location), nrows = 0 + ))) + + if (n_bootstraps > stored) { + stop( + "n_bootstraps is ", n_bootstraps, " but only ", stored, + " ", spec$taxon, " draws are stored. Lower it, or drop `", + key, "`.", + call. = FALSE + ) + } + } + + if (n_bootstraps < v2_bootstraps) { + message( + "Note: running ", n_bootstraps, " of ", v2_bootstraps, + " v2 draws. This is a trial; its parity numbers are not a ", + "parity read." + ) + } + + # Step 3: Covariates the experiment adds from its own files. + # Absolute paths, so parallel workers find them too. + if (!is.null(covariate_files)) { + covariate_files <- normalizePath( + covariate_files, winslash = "/", mustWork = FALSE + ) + check_covariate_files(covariate_files, data_dir) + + specs <- lapply(specs, function(spec) { + spec$covariate_files <- covariate_files + spec + }) + } + + # Step 4: The species. A taxon the set does not name runs every + # species, so say so: it makes a partial set the longest run. + focal <- get_focal_species(species) + + if (!is.null(focal)) { + slugs <- vapply(specs, `[[`, character(1), "taxon") + unnamed <- names(slugs)[!slugs %in% names(focal)] + + if (length(unnamed) > 0) { + message( + "Note: ", paste(unnamed, collapse = ", "), " are not named ", + "in the species set, so every species will be run for ", + "them. Narrow `taxa` to avoid it." + ) + } + } + + structure( + list( + id = id, + taxa = taxa, + specs = specs, + # The set's name, for the record; "inline" for a vector + species_set = if (is.character(species) && + length(species) == 1L && + is.null(names(species))) { + species + } else { + "inline" + }, + focal_species = focal, + n_bootstraps = n_bootstraps, + v2_bootstraps = as.integer(v2_bootstraps), + seed = seed, + stage_models = stage_models, + covariate_files = covariate_files, + plant_bootstrap = plant_bootstrap, + workers = as.integer(workers), + unit_predictions = unit_predictions, + # The baseline does not compare with itself + baseline = if (identical(baseline, id)) NULL else baseline, + project_root = project_root, + data_dir = data_dir, + pipeline_dir = pipeline_dir %||% + file.path(project_root, "2_pipeline", id), + out_dir = out_dir %||% file.path(project_root, "3_output", id) + ), + class = "experiment_config" + ) +} + +## 2.1 print.experiment_config() ---- + +#' Print an Experiment's Configuration +#' +#' @param x An experiment_config(). +#' @param ... Unused. +#' @return `x`, invisibly. +#' +#' @example # Example usage of the function +#' # print(config) +print.experiment_config <- function(x, ...) { + cat("Experiment ", x$id, "\n", sep = "") + cat(" runs: ", paste(x$taxa, collapse = ", "), "\n", sep = "") + cat(" species: ", if (is.null(x$focal_species)) { + "every species" + } else { + paste0(length(x$focal_species), " focal (", x$species_set, ")") + }, "\n", sep = "") + cat(" draws: ", x$n_bootstraps, " (v2: ", x$v2_bootstraps, + ")\n", sep = "") + cat(" seed: ", x$seed %||% "unseeded", "\n", sep = "") + changes <- experiment_changes(x) + cat(" changes: ", if (length(changes) == 0) { + "none; every spec is v2's" + } else { + paste(changes, collapse = "\n ") + }, "\n", sep = "") + cat(" workers: ", x$workers, "\n", sep = "") + cat(" baseline: ", x$baseline %||% "none (this is the baseline)", + "\n", sep = "") + cat(" writes: ", x$pipeline_dir, "\n ", x$out_dir, + "\n", sep = "") + + invisible(x) +} + +# 3. run_experiment() ---- + +#' Fit, Summarize, Compare with the Baseline, Run the +#' Experiment's Steps, and Record +#' +#' @param config An experiment_config(). +#' @param fit Logical. FALSE skips fitting and re-summarizes the +#' stores already in pipeline_dir. +#' @param steps Named list of functions, run in order after the +#' summary tables are written. Each takes `config` and +#' `results` and returns `results`, which it may add to. +#' @param translate Named list of coefficient translations by +#' taxon, passed to collect_results(). +#' @param record Logical. Write run_record.md. +#' @return The results list, invisibly: the summary tables, the +#' run log, the comparison with the baseline, and whatever the +#' steps added. +#' +#' @example # Example usage of the function +#' # results <- run_experiment(config) +run_experiment <- function( + config, + fit = TRUE, + steps = list(), + translate = list(), + record = TRUE +) { + started <- Sys.time() + print(config) + + # Step 1: Fit + run_log <- NULL + + if (fit) { + run_log <- run_specs( + specs = config$specs, + data_dir = config$data_dir, + pipeline_dir = config$pipeline_dir, + species = config$focal_species, + iterations = seq_len(config$n_bootstraps), + stage_models = config$stage_models, + boot_seed = config$seed, + workers = config$workers, + unit_predictions = config$unit_predictions + ) + + fwrite(run_log, file.path(config$pipeline_dir, "run_log.csv"), + na = "") + cat("\nRun log:\n") + print(table(run_log$taxon, run_log$status)) + } + + # Step 2: Summarize the stores + results <- collect_results( + config$pipeline_dir, config$data_dir, translate = translate + ) + results$run_log <- run_log + write_summaries(results, config$out_dir) + + # Step 3: Compare with the baseline, for every experiment but + # the baseline itself + if (!is.null(config$baseline)) { + cat("\n== compare with ", config$baseline, " ==\n", sep = "") + results <- compare_to_baseline(config, results) + } + + # Step 4: The experiment's own steps, in order + for (name in names(steps)) { + cat("\n== ", name, " ==\n", sep = "") + results <- steps[[name]](config, results) + } + + # Step 5: The committed record of what ran + if (record) { + record_file <- run_record( + pipeline_dir = config$pipeline_dir, + out_file = file.path(config$out_dir, "run_record.md"), + exp_id = config$id, + config = record_config(config, run_log), + collected = results + ) + cat("Run record: ", record_file, "\n", sep = "") + } + + cat( + "\n", config$id, " complete in ", + format(round(Sys.time() - started, 1)), "\n", + "Intermediates: ", config$pipeline_dir, "\n", + "Deliverables: ", config$out_dir, "\n", + sep = "" + ) + + invisible(results) +} + +# 4. record_config() ---- + +#' The Settings a Run Record States +#' +#' Settings, not absolute paths, which differ between machines. +#' +#' @param config An experiment_config(). +#' @param run_log The run log, or NULL when nothing was fitted. +#' @return A named list. +#' +#' @example # Example usage of the function +#' # record_config(config, run_log) +record_config <- function(config, run_log = NULL) { + stage_values <- function(field) { + sort(unique(unlist(lapply(config$specs, function(spec) { + vapply(spec$stages, function(stage) { + value <- stage[[field]] + if (is.function(value)) "custom" else value + }, character(1)) + })))) + } + + out <- list( + taxa = sort(config$taxa), + focal_species = if (is.null(config$focal_species)) { + "all" + } else { + sort(paste0(names(config$focal_species), "=", + config$focal_species)) + }, + species_n = if (is.null(config$focal_species)) { + NA_integer_ + } else { + length(config$focal_species) + }, + n_bootstraps = config$n_bootstraps, + v2_bootstraps = config$v2_bootstraps, + boot_seed = config$seed %||% "unseeded", + plant_bootstrap = config$plant_bootstrap, + stage_models = if (is.null(config$stage_models)) { + "spec defaults (v2 candidate sets)" + } else { + sort(names(config$stage_models)) + }, + covariate_files = if (is.null(config$covariate_files)) { + "none" + } else { + basename(config$covariate_files) + }, + changes_from_v2 = { + changes <- experiment_changes(config) + if (length(changes) == 0) "none" else changes + }, + engines = stage_values("engine"), + selection = stage_values("selection"), + compared_with = config$baseline %||% "none (baseline)", + data_dir = published_label(config$data_dir) + ) + + # Record failures, or a run that lost jobs looks complete + if (!is.null(run_log)) { + ok <- sum(run_log$status == "ok", na.rm = TRUE) + out$jobs_total <- nrow(run_log) + out$jobs_ok <- ok + out$jobs_not_ok <- nrow(run_log) - ok + } + + out +} + +## 4.1 experiment_changes() ---- + +#' What an Experiment Changes from the v2 Specs +#' +#' Changes made through `stage_models`, and through specs changed +#' with replace_stage_method(), which records them. +#' +#' @param config An experiment_config(). +#' @return A character vector, one entry per run and change. +#' +#' @example # Example usage of the function +#' # experiment_changes(config) +experiment_changes <- function(config) { + from_specs <- unlist(lapply(names(config$specs), function(key) { + changed <- config$specs[[key]]$models_overridden + + if (is.null(changed)) NULL else paste0(key, ": ", changed) + })) + + from_models <- if (is.null(config$stage_models)) { + NULL + } else { + paste0("stage_models: ", names(config$stage_models)) + } + + from_files <- if (is.null(config$covariate_files)) { + NULL + } else { + paste0("covariate_files: ", basename(config$covariate_files)) + } + + c(from_specs, from_models, from_files) +} + +# 5. compare_to_baseline() ---- + +#' Compare an Experiment's Summaries with the Baseline's +#' +#' Writes tables/comparison_metrics.csv and prints the headline +#' (see compare_experiments()). A baseline not yet run is +#' reported and skipped, not an error. +#' +#' @param config An experiment_config(). +#' @param results The results so far. +#' @param baseline Character. The baseline experiment's id. +#' @return `results`, with `comparison` added when there was a +#' baseline to compare with. +#' +#' @example # Example usage of the function +#' # results <- compare_to_baseline(config, results) +compare_to_baseline <- function( + config, results, baseline = config$baseline +) { + baseline_dir <- file.path( + config$project_root, "3_output", baseline + ) + baseline_table <- file.path( + baseline_dir, "tables", "metric_summary.csv" + ) + + if (!file.exists(baseline_table)) { + message( + "No comparison: ", baseline, " has no summary tables yet. ", + "Run it first, then re-summarize this experiment with ", + "run_experiment(config, fit = FALSE)." + ) + return(results) + } + + comparison <- compare_experiments( + candidate_dir = config$out_dir, + baseline_dir = baseline_dir + ) + + # The headline, for the console; per-species rows are written + shown <- comparison$summary[ + comparison$summary$metric %in% c( + "oob_auc", "oob_deviance_explained", "oob_calibration_slope" + ), + c("taxon", "region", "metric", "median_baseline", + "median_candidate", "median_difference", "share_better") + ] + numbers <- vapply(shown, is.numeric, logical(1)) + shown[numbers] <- lapply(shown[numbers], round, 3) + + cat( + "Against ", baseline, + " (out-of-bag; per species in comparison_metrics.csv):\n", + sep = "" + ) + print(shown, row.names = FALSE) + + results$comparison <- comparison + results +} + +# 6. script_step() ---- + +#' Turn an Experiment Script into a Step +#' +#' Runs the script in a fresh environment (parent: global) +#' holding `config` and `results`, so it leaves nothing behind, +#' and returns the updated `results`. +#' +#' @param path Character. The script. +#' @return A function(config, results) for run_experiment()'s +#' `steps`. +#' +#' @example # Example usage of the function +#' # steps <- list(report = script_step("03_build_report.R")) +script_step <- function(path) { + force(path) + + if (!file.exists(path)) { + stop("Step script not found: + ", path, call. = FALSE) + } + + function(config, results) { + env <- new.env(parent = globalenv()) + env$config <- config + env$results <- results + + sys.source(path, envir = env, keep.source = FALSE) + + env$results + } +} + +# End of script ---- diff --git a/1_code/harness/harness.R b/1_code/harness/harness.R index 2ea4e26..1531714 100644 --- a/1_code/harness/harness.R +++ b/1_code/harness/harness.R @@ -1,19 +1,19 @@ # --- -# title: Load the Harness +# title: Load the Framework # author: Brendan Casey # created: 2026-09-09 # inputs: -# the other files in 1_code/harness/ -# outputs: none; defines the harness in the calling environment +# - the files in 1_code/harness/ +# - the files in 1_code/methods/ +# - the files in 1_code/modules/ +# outputs: none; defines the framework in the calling environment # notes: -# - One entry point, so an experiment sources this rather than -# nine files in the right order. -# - The order matters. covariate_sets.R calls covariate_key() -# from data_load.R, data_load.R calls the derivation helpers -# from covariate_sets.R, and run_model.R calls most of the -# rest. They are sourced into one environment, so the mutual -# reference between the first two resolves once all are -# loaded; only run_model.R has to come last. +# - Single entry point: experiments source this file and call +# load_framework(). +# - Load order: harness/ (including the method registry), +# methods/ (each file registers itself), modules/, then +# experiments/_shared/. Files or folders starting with "_" +# are shared helpers and load first. # --- # 1. Setup ---- @@ -35,7 +35,14 @@ load_harness <- function( harness_dir = "1_code/harness", envir = parent.frame() ) { + # The registry and cache come first: method files register into + # the one, and loaders read through the other. run_model.R is + # last because it calls most of the rest. files <- c( + "registry.R", + "cache.R", + "data_source.R", + "spec.R", "data_load.R", "covariate_sets.R", "species.R", @@ -48,15 +55,127 @@ load_harness <- function( "result.R", "run_record.R", "run_model.R", - file.path("utils", "step_runner.R") + "summarise.R", + "compare.R", + "experiment.R", + "new_experiment.R" ) - paths <- file.path(harness_dir, files) + source_all(file.path(harness_dir, files), envir) +} + +# 3. load_methods() ---- + +#' Source and Register Every Method +#' +#' @param methods_dir Character. Path to 1_code/methods. +#' @param envir Environment to source into. +#' @return Character vector of the files sourced, invisibly. +#' +#' @example # Example usage of the function +#' # load_methods("1_code/methods") +load_methods <- function( + methods_dir = "1_code/methods", + envir = parent.frame() +) { + kinds <- c("engines", "selection", "resampling", "metrics") + + paths <- unlist(lapply(kinds, function(kind) { + files <- sort(list.files( + file.path(methods_dir, kind), pattern = "\\.R$", + full.names = TRUE + )) + shared <- startsWith(basename(files), "_") + + c(files[shared], files[!shared]) + })) + + source_all(paths, envir) +} + +# 4. load_modules() ---- + +#' Source Every Taxon Module +#' +#' @param modules_dir Character. Path to 1_code/modules. +#' @param envir Environment to source into. +#' @return Character vector of the files sourced, invisibly. +#' +#' @example # Example usage of the function +#' # load_modules("1_code/modules") +load_modules <- function( + modules_dir = "1_code/modules", + envir = parent.frame() +) { + folders <- sort(list.dirs( + modules_dir, recursive = FALSE, full.names = TRUE + )) + shared <- startsWith(basename(folders), "_") + + paths <- unlist(lapply( + c(folders[shared], folders[!shared]), + function(folder) { + sort(list.files(folder, pattern = "\\.R$", full.names = TRUE)) + } + )) + + source_all(paths, envir) +} + +# 5. load_framework() ---- + +#' Load the Harness, Methods, Taxon Modules and Shared Settings +#' +#' What an experiment calls first. Everything is sourced into one +#' environment, the caller's by default. +#' +#' @param project_root Character. The repository root. +#' @param envir Environment to source into. +#' @return The project root, invisibly. +#' +#' @example # Example usage of the function +#' # source("1_code/harness/harness.R") +#' # load_framework() +#' # list_methods() +load_framework <- function( + project_root = ".", + envir = parent.frame() +) { + project_root <- normalizePath(project_root, winslash = "/") + code_dir <- file.path(project_root, "1_code") + + load_harness(file.path(code_dir, "harness"), envir) + load_methods(file.path(code_dir, "methods"), envir) + load_modules(file.path(code_dir, "modules"), envir) + + shared <- sort(list.files( + file.path(code_dir, "experiments", "_shared"), + pattern = "\\.R$", full.names = TRUE + )) + source_all(shared, envir) + + # Parallel workers load the framework from here themselves + options(sdm.project_root = project_root) + + invisible(project_root) +} + +# 6. source_all() ---- + +#' Source Files in Order, Failing Clearly on a Missing One +#' +#' @param paths Character vector of files. +#' @param envir Environment to source into. +#' @return The paths, invisibly. +#' +#' @example # Example usage of the function +#' # source_all(c("a.R", "b.R"), globalenv()) +source_all <- function(paths, envir) { absent <- paths[!file.exists(paths)] if (length(absent) > 0) { stop( - "Harness files not found:\n ", + "Framework files not found:\n ", paste(absent, collapse = "\n "), call. = FALSE ) diff --git a/1_code/harness/model_sets.R b/1_code/harness/model_sets.R index 3f76ee4..db11d9a 100644 --- a/1_code/harness/model_sets.R +++ b/1_code/harness/model_sets.R @@ -5,25 +5,13 @@ # inputs: none # outputs: none; returns objects in memory # notes: -# - The candidate formulas each v2 pipeline chooses among, as -# text, so a spec names a set rather than carrying 40 lines of -# formulas. Text rather than formula objects, because a -# formula captures the environment it was built in, and these -# are evaluated against whatever frame the run assembles. -# - Candidates are updates - `. ~ . + MAP` - applied to a base -# formula by the selection layer. The base supplies the -# response, so one set serves any taxon whose response column -# is named the same way. -# - The climate set is shared by birds and mammals. Their two v2 -# pipelines were written separately by different authors and -# arrived at the same eight combinations of the same four -# variables; the mammal script adds an explicit null, which -# here is the base formula. That is why one set covers both. -# - The plant climate formulas are spliced verbatim from -# 0_data/v2_scripts/plants/, not retyped. -# - Habitat model sets are not here. They are large, taxon -# specific and bound up with a prediction grid, so they belong -# with the taxon specs. +# - v2's candidate formulas, named so a spec can refer to a set. +# Stored as text, not formula objects, so no environment is +# captured. Candidates are updates (`. ~ . + MAP`) applied to a +# base formula that supplies the response. +# - Plant and mammal formulas are spliced verbatim from +# 0_data/v2_scripts/, not retyped; long lines are kept so they +# match v2 line for line. # --- # 1. Setup ---- @@ -41,20 +29,9 @@ #' # names(model_sets()) #' # model_sets()$climate_bird_mammal_v2 model_sets <- function() { - # The mammal climate set, which is also the climate part of the - # bird one. Four variables - frost free period, mean annual - # precipitation, climatic moisture deficit and continentality - - # in eight combinations. - # - # Mammals fit it in full: the camera climate the harmonizer - # joins carries CMD and TD. Birds in v2 also fit each of the - # eight with linear and with quadratic spatial terms, 25 - # candidates in all, which this set does not include; see the - # bird spec's v2_coverage. - # - # The null is a candidate, not just the base: v2 averages over - # `climate.list` including the intercept-only model it starts - # from, so leaving it out would shift every weight. + # Shared by the v2 bird and mammal climate stages. The null is + # a candidate because v2 averages over it too; dropping it + # would shift every weight. climate_bird_mammal_v2 <- c( ". ~ .", ". ~ . + FFP", @@ -67,12 +44,9 @@ model_sets <- function() { ". ~ . + MAP + FFP + TD + CMD" ) - # The bird climate set as 06.ModelClimate.R assembles it: the - # null, the eight climate combinations, each of the eight with - # a linear spatial trend and interaction, and each of those - # again with quadratic terms. 25 candidates, averaged at once. - # The spatial terms are in metres, as v2 fits them, which is - # why their coefficients run to 1e-12. + # As 06.ModelClimate.R assembles it: 25 candidates (null, 8 + # climate, 8 + linear space, 8 + quadratic space). Spatial terms + # are in metres, as in v2, hence coefficients near 1e-12. spatial_linear <- "Easting + Northing + Easting:Northing" spatial_quadratic <- "I(Easting^2) + I(Northing^2)" @@ -85,20 +59,6 @@ model_sets <- function() { ) ) - # The subset of the above without CMD, five of the nine models. - # No spec uses it since the camera climate supplied CMD; kept - # for an experiment on a dataset that lacks CMD. - climate_mammal_available <- c( - ". ~ .", - ". ~ . + FFP", - ". ~ . + MAP", - ". ~ . + TD", - ". ~ . + TD + FFP" - ) - - # The plant climate set, spliced from the v2 source. Wider - # than the bird and mammal one, and includes interaction and - # squared terms the other two do not use. climate_plant_v2 <- c( ". ~ .", ". ~ . + PET", @@ -116,28 +76,11 @@ model_sets <- function() { ". ~ . + TD + FFP + MAT" ) - # Applied on top of a chosen climate model, not instead of it. - bioclim_plant_v2 <- c( - ".~.+ bio9 + bio15" - ) - - # Spatial trend terms, likewise applied on top. - space_plant_v2 <- c( - ".~.+ Easting + Northing", - ".~.+ Easting + Northing + EastingNorthing", - ".~.+ Easting + Northing + Easting2 + Northing2 + EastingNorthing" - ) - - # The plant climate candidate set as v2 actually assembles it. - # v2 does not fit these 14 alone: it appends a bioclim version - # of every one, then spatial versions of ten of them, and - # averages over all 58 at once. Fitting only the base 14 makes - # a different model, and its coefficients are not comparable - # with the published ones. - # - # The ten that take spatial terms are the models carrying - # neither MAT, TD nor PET - v2's `model.update.id`, which - # indexes the base models and their bioclim counterparts. + # v2 averages over 58 candidates, not the 14 above: each with + # and without bioclim, plus spatial versions of the ten + # (five base, five bioclim) without MAT, TD or PET (v2's + # `model.update.id`). The 14 alone give coefficients not + # comparable with the published ones. spatial_targets <- c(1, 3, 5, 6, 7) bioclim_plant <- paste0( @@ -167,14 +110,10 @@ model_sets <- function() { )) ) - # The v2 plant habitat sets, spliced from the source. These - # are fitted with the ivw_grid rule, which predicts each - # candidate onto the prediction matrix and combines those - # predictions by precision - so a "coefficient" here is an - # effect per habitat type, not a regression coefficient. - # - # Protocol appears in the bryophyte and lichen variants - # only; plant_group_spec() adds it, matching v2. + # Fitted with ivw_grid, so a "coefficient" is an effect per + # habitat type, not a regression coefficient. Protocol (in the + # bryophyte and lichen variants only) is added by + # plant_group_spec(). habitat_veg_plant_v2 <- c( ". ~ . + Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + BlackSpruce + TreedFen + TreedSwamp + GraminoidFen + ShrubbyFen + ShrubbyBog + ShrubbySwamp + Marsh + Grass + Shrub + CCWhiteSpruceR + CCWhiteSpruce1 + CCWhiteSpruce234 + CCPineR + CCPine1234 + CCDecidMixedR + CCDecidMixed1 + CCDecidMixed234 + HardLin + EnSeismic + EnSoftLin + TrSoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", ". ~ . + Climate + WhiteSpruce + Pine + Deciduous + Mixedwood + Peatland + Mineral + Grass + Shrub + CCWhiteSpruceR + CCWhiteSpruce1 + CCWhiteSpruce234 + CCPineR + CCPine1234 + CCDecidMixedR + CCDecidMixed1 + CCDecidMixed234 + HardLin + EnSeismic + EnSoftLin + TrSoftLin + UrbInd + Wellsites + Crop + RoughP + TameP", @@ -211,14 +150,10 @@ model_sets <- function() { ". ~ . + Climate + Productive + Nonproductive + Other + Alien + SoftLin + paspen" ) - # The bird landcover sets, as groups. staged_bic walks them in - # order, fitting each group as an update to the previous - # group's winner, so the groups are a claim about the order the - # questions are asked in: habitat type first, then stand age, - # then cutblocks, then contrasts, then survey method and water. - # - # Terms name the source columns; the harness rewrites them to - # the block-suffixed names the dataset uses. See term_map(). + # staged_bic walks the groups in order, each updating the + # previous group's winner, so group order is part of the model. + # Terms are source names; term_map() rewrites them to the + # dataset's block-suffixed names. landcover_bird_north_v2 <- list( Hab = c( ". ~ . + vegc" @@ -276,19 +211,11 @@ model_sets <- function() { ) ) - # The mammal north presence/absence set. Seventeen candidates - # running from fine vegetation and footprint resolution to - # coarse aggregation, selected on AICc. - # - # Each carries a reference category rather than a factor: the - # land cover type left out of the formula, whose effect the - # intercept absorbs. Models 1, 2 and 10 omit Crop, the rest - # omit Alien, and a coefficient table has to name it to be - # read. The two vectors are parallel and must stay so. - # - # This is the presence half of a hurdle model; v2 fits a Gamma - # abundance-given-presence part alongside it and multiplies - # the two. Only the presence half is reproduced here. + # Presence half of the v2 mammal hurdle (see + # modules/mammals/hurdle.R), 17 candidates selected on AICc. + # intercept_mammal_north_pa_v2 names each model's omitted + # reference category, which the intercept absorbs; the two + # vectors are parallel and must stay so. habitat_mammal_north_pa_v2 <- c( ". ~ . + Decid + Mixedwood + Pine + Spruce + TreedBog + TreedFen + TreedSwamp + GrassHerb + Shrub + Marsh + ShrubbySwamp + ShrubbyBogFen + CCDecidR + CCDecid1 + CCDecid2 + CCMixedwoodR + CCMixedwood1 + CCMixedwood2 + CCPineR + CCPine1 + CCSpruceR + CCSpruce1 + CCSpruce2 + EnSoftLin + EnSeismic + TrSoftLin + TameP + RoughP + Well + RurUrbInd + seas_days + Climate", ". ~ . + Decid + Mixedwood + Pine + Spruce + TreedBog + TreedFen + TreedSwamp + GrassHerb + Shrub + Marsh + ShrubbySwamp + ShrubbyBogFen + CCDecidMixed + CCPine + CCSpruce + EnSoftLin + EnSeismic + TrSoftLin + TameP + RoughP + Well + RurUrbInd + seas_days + Climate", @@ -353,7 +280,6 @@ model_sets <- function() { climate_bird_mammal_v2 = climate_bird_mammal_v2, climate_bird_v2 = climate_bird_v2, climate_plant_v2_full = climate_plant_v2_full, - climate_mammal_available = climate_mammal_available, climate_plant_v2 = climate_plant_v2, habitat_mammal_north_pa_v2 = habitat_mammal_north_pa_v2, intercept_mammal_north_pa_v2 = @@ -364,27 +290,21 @@ model_sets <- function() { landcover_bird_north_v2 = landcover_bird_north_v2, landcover_bird_south_v2 = landcover_bird_south_v2, habitat_veg_plant_v2 = habitat_veg_plant_v2, - habitat_soil_plant_v2 = habitat_soil_plant_v2, - bioclim_plant_v2 = bioclim_plant_v2, - space_plant_v2 = space_plant_v2 + habitat_soil_plant_v2 = habitat_soil_plant_v2 ) } # 3. Building model sets ---- -# A stage's `models` may be a name in model_sets() or a character -# vector of formulas given directly, so an experiment defines its -# own candidate set without editing a spec. The covariates a run -# loads are read off whatever formulas it ends up with, so a set -# defined here needs no matching covariate declaration - naming a -# term is what loads it. +# A stage's `models` is a model_sets() name or a vector of +# formulas. Covariates are loaded from the terms the formulas +# name, so no separate declaration is needed. ## 3.1 extend_models() ---- #' Add Terms to Every Candidate in a Set #' -#' The include/exclude comparison: take a v2 candidate set and -#' put the same extra terms in all of it, so the only difference -#' between two runs is those terms. +#' For include/exclude comparisons: two runs then differ only by +#' these terms. #' #' @param models Character vector of formulas, or a name in #' model_sets(). @@ -409,20 +329,13 @@ extend_models <- function(models, terms, sets = model_sets()) { #' Build a Candidate Set from a List of Covariates #' -#' For an experiment asking which covariates matter rather than -#' reproducing a v2 set. `form` decides the shape of the -#' comparison, and the choice is a modelling decision rather than -#' a convenience: +#' `form` is a modelling choice: #' -#' - `single` fits one model holding every covariate. Use when -#' the question is about the whole set, not its members. -#' - `each` fits one model per covariate, so they compete singly. -#' - `ladder` adds them cumulatively in the order given, which -#' asks whether each one earns its place given the ones before -#' it. Order therefore matters and is the user's claim. -#' - `all_subsets` fits every combination. Honest but explosive: -#' ten covariates is 1,023 models per species per draw, so it -#' is capped rather than left to run away. +#' - `single`: one model with every covariate. +#' - `each`: one model per covariate. +#' - `ladder`: cumulative, in the order given (order matters). +#' - `all_subsets`: every combination, capped by `max_subsets` +#' (ten covariates is 1,023 models per species per draw). #' #' @param covariates Character vector of covariate names. #' @param form Character. One of the shapes above. @@ -489,6 +402,36 @@ models_from_covariates <- function( unique(models) } +## 3.3 models_union() ---- + +#' One Formula Holding Every Covariate a Candidate Set Uses +#' +#' For engines that find their own variables, curvature and +#' interactions (e.g. boosted trees). `I(x^2)` and `a:b` +#' contribute x, a and b as main effects. +#' +#' @param models Character vector of formulas, a list of groups of +#' them (a staged set), or a name in model_sets(). +#' @param sets Named list of model sets. +#' @return One formula, as text, in update form (". ~ . + ..."). +#' +#' @example # Example usage of the function +#' # models_union("climate_bird_v2") +models_union <- function(models, sets = model_sets()) { + formulas <- unlist(get_model_set(models, sets), use.names = FALSE) + + terms <- unique(unlist(lapply(formulas, function(one) { + all.vars(stats::as.formula(one)) + }))) + terms <- setdiff(terms, c(".", "response", "offset", "weight")) + + if (length(terms) == 0) { + return(". ~ .") + } + + paste0(". ~ . + ", paste(terms, collapse = " + ")) +} + # 4. get_model_set() ---- #' Look Up One Model Set @@ -501,9 +444,7 @@ models_from_covariates <- function( #' @example # Example usage of the function #' # get_model_set("climate_bird_mammal_v2") get_model_set <- function(name, sets = model_sets()) { - # A stage that declares no set at all - most stages have no - # intercept categories - asks for nothing rather than for a - # set called NULL. + # Most stages declare no intercept-category set if (is.null(name) || length(name) == 0) { return(NULL) } diff --git a/1_code/harness/new_experiment.R b/1_code/harness/new_experiment.R new file mode 100644 index 0000000..4e5752a --- /dev/null +++ b/1_code/harness/new_experiment.R @@ -0,0 +1,341 @@ +# --- +# title: Scaffold a New Experiment +# author: Brendan Casey +# created: 2026-10-06 +# inputs: +# - 1_code/experiments/_template/, the run.R and README.md every +# experiment starts from +# - the exp_NNN_* folders already in 1_code/experiments/, +# 2_pipeline/ and 3_output/, which set the next number +# outputs: +# - 1_code/experiments//: run.R and README.md, from the +# template with the id, title, author and date filled in +# - 2_pipeline// and its logs/, for the gitignored +# intermediates +# - 3_output//: figures/, tables/ and a stub report.md +# notes: +# - Usage: +# source("1_code/harness/new_experiment.R") +# new_experiment("covariate scale") +# - The number is one past the highest exp_NNN in any of the +# three folders, and gaps are not reused, so a new experiment +# never shares stores with a removed one. +# - Base R only, so it runs without load_framework(). Empty +# folders get a .gitkeep so the structure is committed. +# --- + +# 1. Setup ---- +# Base R only; no packages to load. + +# 2. experiment_slug() ---- + +#' Turn an Experiment Description into its Folder-Name Slug +#' +#' @param description Character. A short description, e.g. +#' "Covariate scale". +#' @return Character. The snake_case slug, e.g. "covariate_scale". +#' +#' @example # Example usage of the function +#' # experiment_slug("SoilGrids 0-5 cm") +#' # #> "soilgrids_0_5_cm" +experiment_slug <- function(description) { + if (!is.character(description) || length(description) != 1L || + is.na(description)) { + stop("description must be a single string.", call. = FALSE) + } + + slug <- tolower(trimws(description)) + slug <- gsub("[^a-z0-9]+", "_", slug) + slug <- gsub("^_+|_+$", "", slug) + + if (!nzchar(slug)) { + stop( + "description has no letters or digits to name a folder.", + call. = FALSE + ) + } + + # The number is assigned here, so a user-supplied one would be + # doubled or contradicted + if (grepl("^exp_[0-9]{3}_", slug)) { + stop( + "Give the description without the exp_NNN_ prefix; ", + "the number is assigned for you.", + call. = FALSE + ) + } + + slug +} + +# 3. next_experiment_number() ---- + +#' Find the Next Free Experiment Number +#' +#' @param project_root Character. The repository root. +#' @return Integer. One past the highest exp_NNN folder in +#' 1_code/experiments/, 2_pipeline/ or 3_output/; 0 if there is +#' none. +#' +#' @example # Example usage of the function +#' # next_experiment_number(".") +#' # #> 3 +next_experiment_number <- function(project_root = ".") { + roots <- file.path( + project_root, c("1_code/experiments", "2_pipeline", "3_output") + ) + + folders <- unlist(lapply(roots[dir.exists(roots)], function(root) { + list.dirs(root, recursive = FALSE, full.names = FALSE) + })) + folders <- folders[grepl("^exp_[0-9]{3}_", folders)] + + if (length(folders) == 0) { + return(0L) + } + + number <- max(as.integer(substr(folders, 5, 7))) + 1L + + if (number > 999L) { + stop( + "Experiment numbers are three digits; 999 is taken.", + call. = FALSE + ) + } + + number +} + +# 4. new_experiment() ---- + +#' Scaffold a New Experiment's Folders +#' +#' Creates 1_code/experiments// from the template, +#' 2_pipeline// with logs/, and 3_output// with figures/, +#' tables/ and a stub report.md, where is +#' exp__ and NNN is the next free number. +#' +#' @param description Character. A short description of the +#' question; becomes the id's slug and the run.R title. +#' @param author Character or NULL. For the run.R header; NULL +#' reads `git config user.name`, then the system user. +#' @param project_root Character. The repository root. +#' @param template_dir Character or NULL. The template folder; +#' NULL for 1_code/experiments/_template. +#' @param dry_run Logical. TRUE reports the id and paths without +#' writing anything. +#' @param open Logical. Open the new run.R for editing. +#' @return A list, invisibly: `id`, `number`, `code_dir`, +#' `pipeline_dir` and `out_dir`. +#' +#' @example # Example usage of the function +#' # source("1_code/harness/new_experiment.R") +#' # new_experiment("covariate scale", dry_run = TRUE) +#' # #> exp_003_covariate_scale (dry run; nothing written) +new_experiment <- function( + description, + author = NULL, + project_root = getOption("sdm.project_root", "."), + template_dir = NULL, + dry_run = FALSE, + open = interactive() +) { + # Step 1: The id and where it goes + slug <- experiment_slug(description) + number <- next_experiment_number(project_root) + id <- sprintf("exp_%03d_%s", number, slug) + + template_dir <- template_dir %||% + file.path(project_root, "1_code", "experiments", "_template") + dirs <- c( + code_dir = file.path(project_root, "1_code", "experiments", id), + pipeline_dir = file.path(project_root, "2_pipeline", id), + out_dir = file.path(project_root, "3_output", id) + ) + out <- c(list(id = id, number = number), as.list(dirs)) + + # Step 2: Check everything before writing anything + if (!dir.exists(template_dir)) { + stop("Template folder not found: ", template_dir, call. = FALSE) + } + + taken <- dirs[dir.exists(dirs)] + + if (length(taken) > 0) { + stop( + "Already exists, so ", id, " cannot be created:\n ", + paste(taken, collapse = "\n "), + call. = FALSE + ) + } + + if (dry_run) { + cat(id, " (dry run; nothing written)\n ", + paste(dirs, collapse = "\n "), "\n", sep = "") + return(invisible(out)) + } + + author <- author %||% git_user_name(project_root) + + # Step 3: Remove what this call made if a later step fails, so + # a half-built experiment does not hold the number + created <- character(0) + done <- FALSE + on.exit( + if (!done) unlink(created, recursive = TRUE), + add = TRUE + ) + + for (dir in dirs) { + dir.create(dir, recursive = TRUE) + created <- c(created, dir) + } + + # Step 4: The code, from the template with its placeholders + # filled in + fill_template( + template_dir, dirs[["code_dir"]], + id = id, number = number, slug = slug, author = author + ) + + # Step 5: The pipeline and output structure + for (sub in c( + dirs[["pipeline_dir"]], + file.path(dirs[["pipeline_dir"]], "logs"), + file.path(dirs[["out_dir"]], "figures"), + file.path(dirs[["out_dir"]], "tables") + )) { + dir.create(sub, recursive = TRUE, showWarnings = FALSE) + file.create(file.path(sub, ".gitkeep")) + } + + writeLines( + c( + paste("#", id), + "", + "_Not yet run._ The question and design are in", + paste0("`1_code/experiments/", id, "/README.md`."), + "Write the result here, or replace this file with a report", + "step; `run_record.md` is written beside it on every run." + ), + file.path(dirs[["out_dir"]], "report.md") + ) + + done <- TRUE + + cat( + "Created ", id, "\n ", + paste(dirs, collapse = "\n "), "\n", + "Next:\n", + " 1. State the question and design in its README.md.\n", + " 2. Set the one change in its run.R.\n", + " 3. Run it at 5 draws to check it works, then at 100.\n", + sep = "" + ) + + if (open) { + utils::file.edit(file.path(dirs[["code_dir"]], "run.R")) + } + + invisible(out) +} + +## 4.1 fill_template() ---- + +#' Copy the Template, Filling its Placeholders +#' +#' Also drops the template run.R's note on starting an +#' experiment. +#' +#' @param template_dir Character. The template folder. +#' @param code_dir Character. The new experiment's code folder. +#' @param id,slug,author Character. The experiment's values. +#' @param number Integer. The experiment's number. +#' @return The files written, invisibly. +#' +#' @example # Example usage of the function +#' # fill_template("1_code/experiments/_template", +#' # "1_code/experiments/exp_003_covariate_scale", +#' # id = "exp_003_covariate_scale", number = 3L, +#' # slug = "covariate_scale", author = "B. Casey") +fill_template <- function(template_dir, code_dir, id, number, slug, + author) { + # "covariate_scale" -> "Covariate Scale" + words <- strsplit(slug, "_", fixed = TRUE)[[1]] + title <- paste0( + "Experiment ", sprintf("%03d", number), " - ", + paste0(toupper(substr(words, 1, 1)), substring(words, 2), + collapse = " ") + ) + + placeholders <- c( + "exp_NNN_short_description" = id, + "Experiment NNN - [Short Description]" = title, + "[Your Name]" = author, + "[YYYY-MM-DD]" = format(Sys.Date(), "%Y-%m-%d") + ) + + files <- list.files(template_dir, recursive = TRUE) + + for (file in files) { + text <- readLines(file.path(template_dir, file), warn = FALSE) + + for (key in names(placeholders)) { + text <- gsub(key, placeholders[[key]], text, fixed = TRUE) + } + + # The note bullet and its continuation lines + start <- grep("^# - Start an experiment with new_experiment", + text) + + if (length(start) == 1L) { + end <- start + while (end < length(text) && grepl("^# ", text[end + 1])) { + end <- end + 1L + } + text <- text[-(start:end)] + } + + target <- file.path(code_dir, file) + dir.create(dirname(target), recursive = TRUE, + showWarnings = FALSE) + writeLines(text, target) + } + + invisible(file.path(code_dir, files)) +} + +## 4.2 git_user_name() ---- + +#' The Author for a New Script's Header +#' +#' @param project_root Character. The repository root. +#' @return Character. `git config user.name`, or the system user +#' when git is unavailable or has no name set. +#' +#' @example # Example usage of the function +#' # git_user_name(".") +git_user_name <- function(project_root = ".") { + name <- tryCatch( + suppressWarnings(system2( + "git", c("-C", shQuote(project_root), "config", "user.name"), + stdout = TRUE, stderr = FALSE + )), + error = function(e) character(0) + ) + + if (length(name) == 1L && nzchar(trimws(name))) { + trimws(name) + } else { + Sys.info()[["user"]] + } +} + +## 4.3 %||% ---- +# Defined here only when missing, so the file can be sourced on +# its own; R >= 4.4 and the harness both provide it. +if (!exists("%||%", mode = "function")) { + `%||%` <- function(x, y) if (is.null(x)) y else x +} + +# End of script ---- diff --git a/1_code/harness/predict_grid.R b/1_code/harness/predict_grid.R index 3356612..5b9e5cf 100644 --- a/1_code/harness/predict_grid.R +++ b/1_code/harness/predict_grid.R @@ -3,28 +3,17 @@ # author: Brendan Casey # created: 2026-09-09 # inputs: -# in 0_data/test_dataset/lookup/: +# in the test dataset's lookup/ (see data_source.R): # - _prediction_matrix.csv # outputs: none; returns objects in memory # notes: -# - A prediction grid has one row per habitat type and one -# column per model term, holding the cover composition of a -# stand made entirely of that type. Predicting onto it turns -# a fitted model into an effect per habitat type, which is -# the quantity v2 reports and the Biodiversity Browser shows. -# - This is the comparison currency. Coefficients cannot compare -# a GLM against a boosted tree, because a tree has none, but -# every engine can predict onto these rows. -# - Terms the grid does not carry are set to zero, not dropped. -# A grid row describes a pure stand, so a human footprint term -# is genuinely absent from it rather than unknown, and zero is -# what "none of this here" means. Terms that carry a stage -# forward are the exception and are set to their mean, because -# a habitat effect is read at average climate, not at zero -# climate. -# - A grid whose VegType column is missing is an error rather -# than a silent positional fallback, because the row order of -# a prediction matrix is not something to guess at. +# - A grid row is a pure stand of one habitat type, so +# predicting onto it gives the per-habitat effect v2 reports. +# Every engine can predict (a tree has no coefficients), so +# grids are how engines are compared. Bird grids are rebuilt +# by _setup/09 from v2's coefficient translation matrix. +# - v2's later coefficient adjustments (e.g. plant stand-age +# splines) are in the coefficients, not the grid. # --- # 1. Setup ---- @@ -36,7 +25,7 @@ library(data.table) # lookup table reading (version: 1.16.4) #' Read a Prediction Grid #' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param name Character. Grid name, without the #' `_prediction_matrix.csv` suffix, or NULL for no grid. #' @return A data frame with habitat types as rownames, or NULL. @@ -77,85 +66,79 @@ load_prediction_grid <- function(data_dir, name) { # 3. predict_grid() ---- -#' Predict a Fitted Model onto a Grid +#' Predict a Selection's Final Model onto a Grid #' -#' Builds a prediction frame carrying every term the model needs: -#' the grid's own columns where it has them, zero where it does -#' not, and the mean where a term carries a previous stage. +#' Columns come from the grid, then the stage's constants (v2 +#' holds Climate at zero and Protocol at the new protocol), then +#' zero. Uses the selection's own predictor, so an averaged or +#' combined model is projected as a whole. #' -#' @param selected A selection result, from selection_run(). -#' @param engine An engine definition. +#' @param selected A selection result, carrying `predict`. #' @param grid A prediction grid, from load_prediction_grid(). -#' @param carried Named list of values for terms that carry a -#' previous stage forward, or NULL to use zero. +#' @param constants Named list of values for columns held fixed +#' across the grid. #' @param type Character. "response" or "link". #' @return A named numeric vector, one value per grid row, or #' NULL when the model cannot be predicted. #' #' @example # Example usage of the function -#' # predict_grid(selected, engine, grid, -#' # carried = list(Climate = 0.4)) +#' # predict_grid(selected, grid, constants = list(Climate = 0)) predict_grid <- function( selected, - engine, grid, - carried = NULL, + constants = list(), type = "response" ) { - if (is.null(grid) || is.null(selected$fit)) { + if (is.null(grid) || !is.function(selected$predict)) { return(NULL) } - terms <- model_terms(selected) + newdata <- grid - if (length(terms) == 0) { - return(NULL) + # Step 1: Hold the stage's constants + for (one in names(constants)) { + newdata[[one]] <- constants[[one]] } - # Step 1: Start from the grid, so every term it carries takes - # the composition of a pure stand of that habitat type - newdata <- grid - - # Step 2: Fill in the terms the grid has no column for. Zero is - # what a grid row means by omission - none of that here - and a - # carried stage is the exception, held at its supplied value. - for (one in setdiff(terms, names(newdata))) { - newdata[[one]] <- if (!is.null(carried[[one]])) { - carried[[one]] - } else { - 0 - } + # Step 2: Every column any candidate formula reads, at zero + # where the grid and the constants have none. A grid row is a + # pure stand, so a term it omits is absent rather than unknown. + formulas <- selected$formulas %||% list(selected$formula) + needed <- unique(unlist(lapply( + Filter(function(f) !is.null(f) && !identical(f, NA_character_), + formulas), + formula_variables + ))) + + for (one in setdiff(needed, names(newdata))) { + newdata[[one]] <- 0 } # Step 3: An offset is a property of a survey, not of a habitat # type, so a grid prediction is made without one - if ("offset" %in% names(newdata) == FALSE) { - newdata$offset <- 0 - } + newdata$offset <- 0 - predicted <- engine$predict(selected$fit, newdata, type) + predicted <- tryCatch( + selected$predict(newdata, type), + error = function(e) NULL + ) - if (is.null(predicted)) { + if (is.null(predicted) || length(predicted) != nrow(grid)) { return(NULL) } - stats::setNames(predicted, rownames(grid)) + stats::setNames(as.numeric(predicted), rownames(grid)) } # 4. predict_from_coefficients() ---- #' Predict from a Coefficient Vector Rather Than a Fit #' -#' A stage carried forward is not the best single candidate's -#' prediction: it is the averaged coefficient vector applied to -#' the data. v2 computes the climate term as -#' `plogis(X %*% averaged_coefficients)`, so the habitat models -#' see a probability between 0 and 1, and their coefficient on it -#' is correspondingly large. -#' -#' Carrying a single model's prediction instead would discard the -#' averaging the stage just did, and carrying it on the link -#' scale would change what the next stage's coefficient means. +#' Carries a stage forward as v2 does: the averaged coefficients +#' applied to the data, `plogis(X %*% b)`, so the next stage sees +#' a probability. A single candidate's prediction would discard +#' the averaging; the link scale would change what the next +#' stage's coefficient means. #' #' @param coefficients A data frame of term and estimate. #' @param data A data frame carrying those terms. @@ -227,35 +210,4 @@ predict_from_coefficients <- function( linear } -# 5. model_terms() ---- - -#' Name the Terms a Selected Model Uses -#' -#' Taken from the coefficients where the engine has them, and -#' from the formula otherwise, so an engine without coefficients -#' still reaches a grid prediction. -#' -#' @param selected A selection result. -#' @return A character vector of term names, without the -#' intercept. -#' -#' @example # Example usage of the function -#' # model_terms(selected) -model_terms <- function(selected) { - if (!is.null(selected$coefficients)) { - terms <- selected$coefficients$term - - return(setdiff(terms, c("(Intercept)", "Intercept"))) - } - - if (!is.na(selected$formula)) { - return(setdiff( - all.vars(stats::as.formula(selected$formula)), - c("response", "offset", "weight") - )) - } - - character(0) -} - # End of script ---- diff --git a/1_code/harness/registry.R b/1_code/harness/registry.R new file mode 100644 index 0000000..f8e6c73 --- /dev/null +++ b/1_code/harness/registry.R @@ -0,0 +1,395 @@ +# --- +# title: Method Registry +# author: Brendan Casey +# created: 2026-10-03 +# inputs: none +# outputs: none; holds the registered methods in memory +# notes: +# - Engines, selection rules, resamplers and metrics register +# here by name, so adding a method is one new file in +# 1_code/methods/ with no harness change. Re-sourcing a method +# file replaces its entry. +# - Selection rules declare the engine capabilities they need; +# validate_spec() (spec.R) checks them before any data is read. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +## 1.2 The registry ---- +# Created only if absent, so re-sourcing this file keeps +# registered methods +if (!exists(".sdm_registry", inherits = FALSE)) { + .sdm_registry <- new.env(parent = emptyenv()) +} + +for (.kind in c("engine", "selection", "resampler", "metric")) { + if (!exists(.kind, envir = .sdm_registry, inherits = FALSE)) { + assign(.kind, list(), envir = .sdm_registry) + } +} +rm(.kind) + +# 2. Registering ---- + +## 2.1 method_kinds() ---- + +#' The Kinds of Method the Registry Holds +#' +#' @return A named character vector: kind to a readable label. +#' +#' @example # Example usage of the function +#' # method_kinds() +method_kinds <- function() { + c( + engine = "fitting engine", + selection = "selection rule", + resampler = "resampling scheme", + metric = "metric" + ) +} + +## 2.2 engine_capabilities() ---- + +#' The Capabilities an Engine Can Declare +#' +#' - `coefficients`: the engine returns a table of term, estimate +#' and standard error. Needed by rules that average or read +#' coefficients, and by a stage carried forward through them. +#' - `ic`: the engine scores a fit with AIC, AICc or BIC. Needed +#' by every rule that ranks or weights candidates. +#' - `se`: the engine's predict() returns standard errors. Needed +#' by inverse-variance averaging. +#' - `converged`: the engine reports whether a fit converged. +#' Optional; a fit is assumed converged when it cannot say. +#' +#' @return A character vector. +#' +#' @example # Example usage of the function +#' # engine_capabilities() +engine_capabilities <- function() { + c("coefficients", "ic", "se", "converged") +} + +## 2.3 register_method() ---- + +#' Add One Method to the Registry +#' +#' @param kind Character. One of names(method_kinds()). +#' @param name Character. The name a spec uses. +#' @param value The method: a function, or for an engine a list. +#' @param description Character. One line on what it does. +#' @param requires Character vector of engine capabilities the +#' method needs. Used by selection rules. +#' @return The name, invisibly. +#' +#' @example # Example usage of the function +#' # register_method("metric", "auc", metric_auc, "Area under ROC") +register_method <- function( + kind, + name, + value, + description, + requires = character(0) +) { + if (!kind %in% names(method_kinds())) { + stop( + "Unknown method kind `", kind, "`. Kinds: ", + paste(names(method_kinds()), collapse = ", "), + call. = FALSE + ) + } + + if (!is.character(name) || length(name) != 1L || !nzchar(name)) { + stop("A method name must be one non-empty string.", call. = FALSE) + } + + if (!is.character(description) || length(description) != 1L) { + stop( + "Method `", name, "` needs a one-line description.", + call. = FALSE + ) + } + + unknown <- setdiff(requires, engine_capabilities()) + + if (length(unknown) > 0) { + stop( + "Method `", name, "` requires unknown capabilities: ", + paste(unknown, collapse = ", "), ". Known: ", + paste(engine_capabilities(), collapse = ", "), + call. = FALSE + ) + } + + entries <- get(kind, envir = .sdm_registry) + entries[[name]] <- list( + name = name, + value = value, + description = description, + requires = requires + ) + assign(kind, entries, envir = .sdm_registry) + + invisible(name) +} + +## 2.4 register_engine() ---- + +#' Register a Fitting Engine +#' +#' See 1_code/methods/README.md for the contract. +#' +#' @param engine A list with `name`, `description`, +#' `capabilities`, `fit`, `predict`, and `coef`/`ic` where +#' declared. +#' @return The engine name, invisibly. +#' +#' @example # Example usage of the function +#' # register_engine(engine_glm()) +register_engine <- function(engine) { + required <- c("name", "description", "capabilities", "fit", + "predict") + absent <- setdiff(required, names(engine)) + + if (length(absent) > 0) { + stop( + "An engine needs ", paste(absent, collapse = ", "), ".", + call. = FALSE + ) + } + + if ("coefficients" %in% engine$capabilities && + !is.function(engine$coef)) { + stop( + "Engine `", engine$name, "` declares `coefficients` but has ", + "no coef() function.", + call. = FALSE + ) + } + + if ("ic" %in% engine$capabilities && !is.function(engine$ic)) { + stop( + "Engine `", engine$name, "` declares `ic` but has no ic() ", + "function.", + call. = FALSE + ) + } + + register_method( + "engine", engine$name, engine, engine$description + ) +} + +## 2.5 register_selection() ---- + +#' Register a Selection Rule +#' +#' Rules take `models`, `base`, `data`, `engine`, `family`, +#' `weights`, `offset`, `ic` and `control` and return +#' selection_result(). Further arguments are filled from the +#' stage field of the same name. +#' +#' @param name Character. The name a spec uses. +#' @param fn The rule function. +#' @param description Character. One line. +#' @param requires Character vector of engine capabilities. +#' @return The name, invisibly. +#' +#' @example # Example usage of the function +#' # register_selection("aic_best", select_aic_best, +#' # "Lowest-scoring candidate", "ic") +register_selection <- function(name, fn, description, + requires = character(0)) { + core <- c("models", "base", "data", "engine", "family") + absent <- setdiff(core, names(formals(fn))) + + if (length(absent) > 0) { + stop( + "Selection rule `", name, "` must take ", + paste(absent, collapse = ", "), ".", + call. = FALSE + ) + } + + register_method("selection", name, fn, description, requires) +} + +## 2.6 register_resampler() ---- + +#' Register a Resampling Scheme +#' +#' Resamplers take `frame`, `iterations`, `seed`, `context` and +#' any settings from `spec$resample`, and return one vector of +#' survey unit ids per iteration. Units left out are scored +#' out-of-bag. +#' +#' @param name Character. The name a spec's `resample$scheme` +#' uses. +#' @param fn The resampler function. +#' @param description Character. One line. +#' @return The name, invisibly. +#' +#' @example # Example usage of the function +#' # register_resampler("spatial_block", resampler_spatial_block, +#' # "Bootstrap within spatial blocks") +register_resampler <- function(name, fn, description) { + core <- c("frame", "iterations", "seed", "context") + absent <- setdiff(core, names(formals(fn))) + + if (length(absent) > 0) { + stop( + "Resampler `", name, "` must take ", + paste(absent, collapse = ", "), ".", + call. = FALSE + ) + } + + register_method("resampler", name, fn, description) +} + +## 2.7 register_metric() ---- + +#' Register a Metric +#' +#' Metrics take response-scale `observed` and `predicted` and +#' return one number. +#' +#' @param name Character. The name metrics are reported under. +#' @param fn The metric function. +#' @param description Character. One line. +#' @return The name, invisibly. +#' +#' @example # Example usage of the function +#' # register_metric("rmse", metric_rmse, "Root mean squared error") +register_metric <- function(name, fn, description) { + if (!all(c("observed", "predicted") %in% names(formals(fn)))) { + stop( + "Metric `", name, "` must take `observed` and `predicted`.", + call. = FALSE + ) + } + + register_method("metric", name, fn, description) +} + +## 2.8 unregister_method() ---- + +#' Remove One Method from the Registry +#' +#' For tests and interactive development; a run never needs it. +#' +#' @param kind Character. One of names(method_kinds()). +#' @param name Character. The registered name. +#' @return The name, invisibly. +#' +#' @example # Example usage of the function +#' # unregister_method("metric", "my_test_metric") +unregister_method <- function(kind, name) { + entries <- get(kind, envir = .sdm_registry) + entries[[name]] <- NULL + assign(kind, entries, envir = .sdm_registry) + + invisible(name) +} + +# 3. Looking up ---- + +## 3.1 get_method() ---- + +#' Look Up One Registered Method +#' +#' @param kind Character. One of names(method_kinds()). +#' @param name Character. The registered name. +#' @param entry Logical. Return the whole entry (value, +#' description, requires) rather than the method itself. +#' @return The method, or its entry. +#' +#' @example # Example usage of the function +#' # get_method("selection", "aic_average") +get_method <- function(kind, name, entry = FALSE) { + entries <- get(kind, envir = .sdm_registry) + + if (!is.character(name) || length(name) != 1L || + !name %in% names(entries)) { + stop( + "Unknown ", method_kinds()[[kind]], " `", + paste(format(name), collapse = " "), "`. Registered: ", + if (length(entries) == 0) { + "none - has 1_code/methods/ been loaded?" + } else { + paste(names(entries), collapse = ", ") + }, + call. = FALSE + ) + } + + if (entry) entries[[name]] else entries[[name]]$value +} + +## 3.2 registered_names() ---- + +#' Names Registered for One Kind of Method +#' +#' @param kind Character. One of names(method_kinds()). +#' @return A character vector. +#' +#' @example # Example usage of the function +#' # registered_names("engine") +registered_names <- function(kind) { + names(get(kind, envir = .sdm_registry)) +} + +## 3.3 list_methods() ---- + +#' Show Every Registered Method +#' +#' @param kind Character, or NULL for every kind. +#' @return A data frame of kind, name, requires and description, +#' printed and returned invisibly. +#' +#' @example # Example usage of the function +#' # list_methods() +#' # list_methods("engine") +list_methods <- function(kind = NULL) { + kinds <- if (is.null(kind)) names(method_kinds()) else kind + + rows <- lapply(kinds, function(one) { + entries <- get(one, envir = .sdm_registry) + + if (length(entries) == 0) { + return(NULL) + } + + data.frame( + kind = one, + name = names(entries), + requires = vapply( + entries, + function(e) { + capabilities <- if (one == "engine") { + e$value$capabilities + } else { + e$requires + } + paste(capabilities, collapse = ", ") + }, + character(1) + ), + description = vapply( + entries, `[[`, character(1), "description" + ), + stringsAsFactors = FALSE, + row.names = NULL + ) + }) + + table <- do.call(rbind, rows) + print(table, row.names = FALSE, right = FALSE) + + invisible(table) +} + +# End of script ---- diff --git a/1_code/harness/resample.R b/1_code/harness/resample.R index 7340c9f..4503358 100644 --- a/1_code/harness/resample.R +++ b/1_code/harness/resample.R @@ -1,45 +1,17 @@ # --- -# title: Resampling Schemes +# title: Resampling - Seeds, Blocks and Dispatch # author: Brendan Casey # created: 2026-09-09 -# inputs: -# in 0_data/test_dataset/lookup/: -# - _bootstrap_ids.csv (where one is precomputed) +# inputs: none # outputs: none; returns objects in memory # notes: -# - Produces the survey units one model fit sees. Every scheme -# returns the same thing - a character vector of survey unit -# ids, with replacement where the scheme resamples - so the -# fitting loop does not know which scheme it is running. -# - Three schemes. `precomputed` reads ids the source pipeline -# already generated, which is how birds reach parity. -# `spatial_block` reproduces the plant bootstrap: resample -# sites with replacement within coarse latitude and longitude -# blocks, so a draw keeps the geographic spread rather than -# drifting toward whichever region is best surveyed. -# `spatial_cv` holds out whole blocks, which is the scheme a -# spatial autocorrelation experiment needs and which v2 has -# no equivalent of. -# - Iteration 1 of a bootstrap is the complete data by -# convention, matching v2, so the first fit of every run is -# the full-data fit. -# - v2 sets no seed, so v2 is not reproducible against itself: -# two v2 runs give different coefficients. Strict numerical -# parity is impossible by construction, and a parity check -# can only ever show distributional agreement. -# - `seed` therefore defaults to harness_seed() rather than to -# NULL. Reproducibility is worth more than bit-matching a -# behaviour that does not reproduce itself. Pass NULL to -# restore the v2 behaviour explicitly. See the parity ledger -# in docs/framework_design.md. -# - An experiment sets one base seed. Each species draws from -# its own seed, derived from that base and the species name -# by species_seed(), so species are resampled independently -# as in v2, and a species' draws do not change when others -# are added to or removed from a run. -# - Seeding never leaks. with_seed() restores the caller's -# random state on exit, so resampling does not change what -# any later random call in the session produces. +# - Shared resampling plumbing: seeding, spatial blocks, and +# resample_for_species(), which dispatches to a scheme in +# 1_code/methods/resampling/. Every scheme returns a list of +# survey unit id vectors, one per draw. +# - Seeds default to harness_seed(), not NULL. v2 set no seed, +# so parity with v2 can only be distributional (see the +# parity ledger in docs/framework_design.md). # --- # 1. Setup ---- @@ -53,14 +25,8 @@ library(data.table) # lookup table reading (version: 1.16.4) #' The Default Random Seed #' -#' Every scheme that draws at random seeds from this unless told -#' otherwise, so two runs of the same configuration produce the -#' same result. v2 seeded nothing, so its runs do not reproduce -#' even against themselves; that is the behaviour this default -#' deliberately departs from. -#' -#' The value is recorded in each run's meta.json, so a result can -#' be traced back to the draw that produced it. +#' Used unless a run sets its own seed. Deliberately departs from +#' v2, which seeded nothing and so does not reproduce itself. #' #' @return An integer. #' @@ -74,21 +40,12 @@ harness_seed <- function() { #' Derive One Species' Seed from the Experiment's Base Seed #' -#' Resetting every species to the same seed would give every -#' species the same draws, which v2 did not do and which -#' correlates results across species. Deriving a seed per species -#' keeps one number in charge of the run while giving each -#' species its own stream. -#' -#' The species key is hashed with a polynomial rolling hash, taken -#' modulo a prime below the integer maximum, and added to the -#' base. Every intermediate value stays below 2^53, so the -#' arithmetic is exact in double precision and the result is the -#' same on every platform. -#' -#' Region is deliberately not part of the key. v2 draws once per -#' species across the province, and a species should keep the -#' same draws whichever regions a run includes. +#' A shared seed would give every species identical draws. A +#' per-species seed keeps them independent, as in v2, and stable +#' when other species are added to or dropped from a run. The +#' rolling hash stays below 2^53, so it is exact in double +#' precision on every platform. Region is left out of the key +#' because v2 draws once per species across the province. #' #' @param base Integer or NULL. The experiment's base seed. NULL #' leaves the draw unseeded, which is what v2 did. @@ -122,10 +79,8 @@ species_seed <- function(base, taxon, species) { #' Evaluate Code Under a Seed, Then Restore the Random State #' -#' set.seed() changes the session's random state for everything -#' that follows. Restoring it on exit keeps resampling from -#' changing any later random call, and makes the result depend -#' only on the seed given here. +#' Restoring the caller's state keeps resampling from changing +#' any later random call in the session. #' #' @param seed Integer or NULL. NULL evaluates the code without #' seeding. @@ -143,16 +98,26 @@ with_seed <- function(seed, code) { env <- globalenv() had_state <- exists(".Random.seed", envir = env, inherits = FALSE) old_state <- if (had_state) get(".Random.seed", envir = env) + old_kind <- RNGkind() on.exit({ + # Restore the generator first: setting it reseeds + suppressWarnings(do.call(RNGkind, as.list(old_kind))) + if (had_state) { assign(".Random.seed", old_state, envir = env) - } else if (exists(".Random.seed", envir = env, inherits = FALSE)) { + } else if (exists(".Random.seed", envir = env, + inherits = FALSE)) { rm(".Random.seed", envir = env) } }, add = TRUE) - set.seed(seed) + # Name R's default generator explicitly; a package (e.g. a + # parallel backend) may have switched the session's + set.seed( + seed, kind = "Mersenne-Twister", normal.kind = "Inversion", + sample.kind = "Rejection" + ) code } @@ -161,16 +126,13 @@ with_seed <- function(seed, code) { #' Cut Survey Units into Coarse Spatial Blocks #' -#' Both the bootstrap and the spatial cross-validation scheme -#' need a block label per survey unit. The default cut points are -#' the v2 plant grid. +#' Default cut points are the v2 plant grid. #' #' @param long,lat Numeric vectors of coordinates. #' @param long_breaks,lat_breaks Numeric vectors of cut points. #' @return A character vector of block labels, one per unit. -#' Labels are plain letters rather than interval notation, -#' because a label carrying brackets and commas cannot be used -#' in a regular expression or a file name. +#' Plain letters (or `b1`, `b2`, ... beyond 26), not interval +#' notation, so labels are safe in regexes and file names. #' #' @example # Example usage of the function #' # blocks <- spatial_blocks(frame$long, frame$lat) @@ -199,319 +161,51 @@ spatial_blocks <- function( labels[match(cells, present)] } -# 4. resample_precomputed() ---- - -#' Read Bootstrap Ids the Source Pipeline Generated -#' -#' Two layouts. Birds store one table for the taxon, -#' `lookup/_bootstrap_ids.csv`, one column per draw. The -#' plant-group taxa store v2's draws per species, because v2 -#' drew them per species: -#' `lookup/v2_bootstrap_ids//.rds`, written by -#' 1_code/_setup/07_harmonize_v2_plant_bootstrap_ids.R. -#' -#' Replaying v2's own draws is what makes parity numerical rather -#' than distributional: the harness then fits the same rows v2 -#' fitted, draw by draw. -#' -#' @param data_dir Character. Path to 0_data/test_dataset. -#' @param taxon Character. Taxon slug. -#' @param iterations Integer vector of bootstrap iterations. -#' @param id_column Character. The column in the covariate frame -#' the stored ids refer to, when they are not survey unit ids. -#' @param frame A data frame carrying `survey_unit_id` and -#' `id_column`, used to translate stored ids into survey unit -#' ids. NULL when the stored ids are already unit ids. -#' @param species Character. The species, for per-species ids. -#' @param per_species Logical. Read the per-species layout. -#' @param unique_ids Logical. Keep each unit once per draw. v2's -#' bird scripts select a draw with `%in%`, so a unit drawn twice -#' is fitted once; the plant scripts index rows, so repeats -#' count. -#' @return A list of character vectors, one per iteration. -#' -#' @example # Example usage of the function -#' # ids <- resample_precomputed(data_dir, "bird", 1:3, -#' # id_column = "surveyid", -#' # frame = md$covariates, -#' # unique_ids = TRUE) -#' # ids <- resample_precomputed(data_dir, "lichen", 1:3, -#' # species = "Physcia.adscendens", -#' # per_species = TRUE) -resample_precomputed <- function( - data_dir, - taxon, - iterations, - id_column = NULL, - frame = NULL, - species = NULL, - per_species = FALSE, - unique_ids = FALSE -) { - # Step 1: Read the stored draws, as a list of id vectors - if (per_species) { - path <- file.path( - data_dir, "lookup", "v2_bootstrap_ids", taxon, - paste0(species, ".rds") - ) - - if (is.null(species) || !file.exists(path)) { - stop( - "No stored v2 bootstrap ids for ", taxon, " ", - species, ":\n ", path, - "\nRun 1_code/_setup/07_harmonize_v2_plant_bootstrap_ids.R", - " for this species, or use the spatial_block scheme.", - call. = FALSE - ) - } - - stored <- readRDS(path) - - # v2 stores a species' draws as a matrix, units by draws - if (is.matrix(stored)) { - stored <- lapply(seq_len(ncol(stored)), function(j) stored[, j]) - } - } else { - path <- file.path( - data_dir, "lookup", paste0(taxon, "_bootstrap_ids.csv") - ) - - if (!file.exists(path)) { - stop( - "No precomputed bootstrap ids for ", taxon, ":\n ", path, - "\nUse a different resampling scheme, or generate them.", - call. = FALSE - ) - } - - stored <- as.list(fread(path)) - } - - if (max(iterations) > length(stored)) { - stop( - taxon, " has ", length(stored), " precomputed bootstrap ", - "iterations; ", max(iterations), " were requested.", - call. = FALSE - ) - } - - lapply(iterations, function(i) { - ids <- stored[[i]] - ids <- ids[!is.na(ids)] - - if (unique_ids) { - ids <- unique(ids) - } - - # Step 2: Translate to survey unit ids when the stored ids - # are a different key, which is how the bird ids are stored - if (is.null(id_column)) { - return(as.character(ids)) - } - - if (is.null(frame) || !id_column %in% names(frame)) { - stop( - "Translating precomputed ids needs a frame carrying `", - id_column, "`.", - call. = FALSE - ) - } - - matched <- match(ids, frame[[id_column]]) - as.character(frame$survey_unit_id[matched[!is.na(matched)]]) - }) -} - -# 5. resample_spatial_block() ---- -#' Draw a Spatially Blocked Bootstrap -#' -#' Resamples survey units with replacement within each spatial -#' block, so every block keeps its original number of units and -#' the draw keeps the geographic spread of the survey. -#' -#' Iteration 1 is the complete data, matching v2. -#' -#' @param frame A data frame with `survey_unit_id` and the -#' coordinate columns. -#' @param iterations Integer vector of bootstrap iterations. -#' @param long_column,lat_column Character. Coordinate columns. -#' @param min_detections Integer. Redraw a block set until at -#' least this many detections are present, or 0 to skip the -#' check. -#' @param response Numeric vector, one per row of `frame`, used -#' only for the detection check. -#' @param max_attempts Integer. How many redraws before giving -#' up, so a species that can never meet the threshold fails -#' rather than looping forever. -#' @param seed Integer or NULL. Defaults to harness_seed(), so a -#' run reproduces. NULL leaves the stream unseeded, which is -#' what v2 did. -#' @return A list of character vectors, one per iteration. -#' -#' @example # Example usage of the function -#' # ids <- resample_spatial_block(frame, 1:5, seed = 42) -resample_spatial_block <- function( - frame, - iterations, - long_column = "long", - lat_column = "lat", - min_detections = 0L, - response = NULL, - max_attempts = 100L, - seed = harness_seed() -) { - for (one in c(long_column, lat_column)) { - if (!one %in% names(frame)) { - stop( - "Spatial blocking needs `", one, "` in the frame.", - call. = FALSE - ) - } - } - - units <- as.character(frame$survey_unit_id) - blocks <- spatial_blocks( - frame[[long_column]], frame[[lat_column]] - ) - by_block <- split(seq_along(units), blocks) - - draw_once <- function() { - unlist( - lapply( - by_block, - function(rows) sample(rows, length(rows), replace = TRUE) - ), - use.names = FALSE - ) - } - - # Every draw comes from one stream under `seed`, so a species' - # draws depend only on its seed and the iterations requested. - with_seed(seed, lapply(iterations, function(i) { - # Step 1: Iteration 1 is the complete data, as in v2 - if (i == 1L) { - return(units) - } - - # Step 2: Redraw until the sample carries enough detections - # for a model to be estimable - for (attempt in seq_len(max_attempts)) { - rows <- draw_once() - - if (min_detections <= 0L || is.null(response)) { - return(units[rows]) - } - - if (sum(response[rows] > 0, na.rm = TRUE) >= min_detections) { - return(units[rows]) - } - } +# 4. resample_for_species() ---- - stop( - "No spatially blocked draw reached ", min_detections, - " detections in ", max_attempts, " attempts. The species ", - "is probably too rare to bootstrap at this threshold.", - call. = FALSE - ) - })) -} - -# 6. resample_spatial_cv() ---- - -#' Split Survey Units into Spatial Cross-Validation Folds +#' Draw the Resampling Sets for One Species #' -#' Holds out whole spatial blocks rather than random units, so a -#' held-out score is not inflated by a training unit sitting -#' beside its own test unit. This is the scheme a spatial -#' autocorrelation experiment needs; v2 has no equivalent. +#' Passes the scheme every `spec$resample` setting except +#' `scheme`, `seed` and `scope`, which the harness consumes. #' -#' @param frame A data frame with `survey_unit_id` and the -#' coordinate columns. -#' @param folds Integer. Number of folds. -#' @param long_column,lat_column Character. Coordinate columns. -#' @param seed Integer or NULL. Defaults to harness_seed(). -#' @return A list of lists, each with `train` and `test` -#' character vectors of survey unit ids. +#' @param spec A taxon spec. +#' @param frame The one-species frame, one row per unit that can +#' be drawn. +#' @param iterations Integer vector. +#' @param data_dir Character. +#' @param seed Integer or NULL. This species' seed, from +#' species_seed(). NULL leaves the draw unseeded. +#' @param species Character. Names the stored draws where they +#' are kept per species. +#' @return A list of character vectors of survey unit ids. #' #' @example # Example usage of the function -#' # cv <- resample_spatial_cv(frame, folds = 5, seed = 42) -#' # length(cv[[1]]$train) -resample_spatial_cv <- function( - frame, - folds = 5L, - long_column = "long", - lat_column = "lat", - seed = harness_seed() +#' # resample_for_species(spec, frame, 1:10, data_dir, +#' # seed = 123L, species = "Physcia.adscendens") +resample_for_species <- function( + spec, frame, iterations, data_dir, seed = NULL, species = NULL ) { - units <- as.character(frame$survey_unit_id) - blocks <- spatial_blocks( - frame[[long_column]], frame[[lat_column]] + scheme <- get_method("resampler", spec$resample$scheme) + settings <- spec$resample[ + setdiff(names(spec$resample), c("scheme", "seed", "scope")) + ] + + context <- list( + data_dir = data_dir, + taxon = spec$taxon, + species = species, + spec = spec, + response_name = spec$response_name %||% "response" ) - present <- unique(blocks[!is.na(blocks)]) - - if (length(present) < folds) { - stop( - "Only ", length(present), " spatial block(s) present; ", - folds, " folds were asked for. Use finer block breaks or ", - "fewer folds.", - call. = FALSE - ) - } - # Step 1: Deal whole blocks into folds, so no fold shares a - # block with another - assignment <- with_seed( - seed, sample(rep_len(seq_len(folds), length(present))) - ) - fold_of_block <- assignment[match(blocks, present)] - - lapply(seq_len(folds), function(k) { + do.call(scheme, c( list( - train = units[!is.na(fold_of_block) & fold_of_block != k], - test = units[!is.na(fold_of_block) & fold_of_block == k] - ) - }) -} - -# 7. resample_units() ---- - -#' Produce the Survey Units for Each Model Fit -#' -#' The single entry point the fitting loop calls. Dispatches on -#' the scheme name so a spec names a scheme rather than calling -#' one. -#' -#' @param scheme Character. "precomputed", "spatial_block" or -#' "spatial_cv". -#' @param frame A data frame with `survey_unit_id`. -#' @param iterations Integer vector of iterations, or the number -#' of folds for spatial_cv. -#' @param ... Passed to the scheme's own function. -#' @return For the bootstrap schemes, a list of character vectors -#' of survey unit ids. For spatial_cv, a list of train and test -#' splits. -#' -#' @example # Example usage of the function -#' # resample_units("spatial_block", frame, 1:10, seed = 42) -resample_units <- function(scheme, frame, iterations, ...) { - switch( - scheme, - precomputed = resample_precomputed( - iterations = iterations, frame = frame, ... - ), - spatial_block = resample_spatial_block( - frame = frame, iterations = iterations, ... + frame = frame, iterations = iterations, seed = seed, + context = context ), - spatial_cv = resample_spatial_cv( - frame = frame, folds = length(iterations), ... - ), - stop( - "Unknown resampling scheme `", scheme, "`. Use ", - "\"precomputed\", \"spatial_block\" or \"spatial_cv\".", - call. = FALSE - ) - ) + settings + )) } # End of script ---- diff --git a/1_code/harness/result.R b/1_code/harness/result.R index f126afa..6e44bd0 100644 --- a/1_code/harness/result.R +++ b/1_code/harness/result.R @@ -2,6 +2,7 @@ # title: The Standard Result Object # author: Brendan Casey # created: 2026-09-09 +# inputs: none; writes to the store directory it is given # outputs: # in the store directory: # - grid_predictions.csv @@ -10,27 +11,16 @@ # - metrics.csv # - meta.json # notes: -# - Every run writes the same five files whatever the engine, -# which is what makes a GLM run and a boosted-tree run -# comparable. Comparison code reads this contract and nothing -# else. -# - grid_predictions.csv is the common currency. Coefficients -# cannot compare a GLM to a tree - a tree has no Peatland -# coefficient - but every engine can predict onto the rows of -# a prediction matrix, which is also the quantity v2 reports. -# - coefficients.csv is still written when the engine has -# coefficients, because parity against v2 is checked there. -# Its absence is recorded in meta.json rather than being -# silently empty. -# - Writing is append-per-species rather than one object at the -# end. A bird run is 172 species by 100 bootstraps, and -# holding all of it before the first write would risk losing -# a long run to a failure in its last species. -# - meta.json records the configuration a run cannot be -# reproduced without: engine, selection rule, covariates, -# seed, and the resampling scheme. The seed matters most - v2 -# set none, so a v2-configured run is not reproducible even -# against itself. +# - Every engine writes the same files, so comparison code reads +# only this contract. grid_predictions.csv compares any +# engines; coefficients.csv is written only when the engine +# has coefficients (v2 parity is checked there). +# - Rows are appended per draw, so a failure late in a long run +# loses little. Parallel species write headerless shards; +# merge_result_shards() appends them in queue order, matching +# a serial run byte for byte. +# - unit_predictions.csv is opt-in: it can reach gigabytes +# (birds) and nothing downstream reads it. # --- # 1. Setup ---- @@ -40,39 +30,114 @@ library(data.table) # fast appending writes (version: 1.16.4) # 2. result_store() ---- -#' Open a Result Store +## 2.1 result_columns() ---- + +#' The Columns of Each Result File +#' +#' @return A named list of character vectors. #' -#' Creates the directory and records which files have been -#' started, so each is written with a header once and appended to -#' after that. +#' @example # Example usage of the function +#' # result_columns()$metrics +result_columns <- function() { + list( + grid_predictions = c( + "species", "region", "boot", "grid_unit", "prediction" + ), + unit_predictions = c( + "species", "region", "boot", "survey_unit_id", "prediction", + "observed" + ), + coefficients = c( + "species", "region", "stage", "boot", "term", "estimate", "se" + ), + metrics = c("species", "region", "boot", "metric", "value") + ) +} + +## 2.2 result_store() ---- + +#' Open a Result Store #' #' @param dir Character. Directory to write into. #' @param overwrite Logical. Remove existing result files first. -#' FALSE lets a run add to a store, which is how a run that -#' stopped part way is continued. +#' FALSE continues a run that stopped part way. +#' @param header Logical. Write a header line when a file is +#' started. FALSE for a shard, which is merged under one header. +#' @param unit_predictions Character. "none" writes no unit +#' predictions, "oob" only the units a draw left out, "all" +#' every unit. #' @return A store object. #' #' @example # Example usage of the function #' # store <- result_store("2_pipeline/exp_001/bryophyte") -result_store <- function(dir, overwrite = TRUE) { +result_store <- function( + dir, + overwrite = TRUE, + header = TRUE, + unit_predictions = c("none", "oob", "all") +) { + unit_predictions <- match.arg(unit_predictions) dir.create(dir, recursive = TRUE, showWarnings = FALSE) - files <- c( - "grid_predictions.csv", "unit_predictions.csv", - "coefficients.csv", "metrics.csv" - ) + files <- paste0(names(result_columns()), ".csv") if (overwrite) { - existing <- file.path(dir, files) + existing <- file.path(dir, c(files, "meta.json")) file.remove(existing[file.exists(existing)]) } structure( - list(dir = dir, files = files), + list( + dir = dir, files = files, header = header, + unit_predictions = unit_predictions + ), class = "result_store" ) } +## 2.3 merge_result_shards() ---- + +#' Combine Shard Stores into One Store +#' +#' @param store A result_store() to write into, with a header. +#' @param shard_dirs Character vector of shard directories, in the +#' order their rows should appear. +#' @return NULL, invisibly. +#' +#' @example # Example usage of the function +#' # merge_result_shards(store, file.path(shards, species)) +merge_result_shards <- function(store, shard_dirs) { + columns <- result_columns() + + for (what in names(columns)) { + target <- file.path(store$dir, paste0(what, ".csv")) + parts <- file.path(shard_dirs, paste0(what, ".csv")) + parts <- parts[file.exists(parts)] + + if (length(parts) == 0) { + next + } + + # The header is written by fwrite itself, from an empty table, + # so its line ending matches the rows appended under it + if (!file.exists(target)) { + empty <- stats::setNames( + as.data.frame( + rep(list(character(0)), length(columns[[what]])) + ), + columns[[what]] + ) + fwrite(empty, target) + } + + for (part in parts) { + file.append(target, part) + } + } + + invisible(NULL) +} + # 3. result_append() ---- #' Append Rows to One Result File @@ -92,20 +157,18 @@ result_append <- function(store, what, rows) { } path <- file.path(store$dir, paste0(what, ".csv")) + started <- file.exists(path) - # data.table writes the header only on the first call, so the - # file stays a single valid CSV across appends + # Header on first write only; shards have none fwrite( - rows, path, append = file.exists(path), - col.names = !file.exists(path), na = "" + rows, path, append = started, + col.names = store$header && !started, na = "" ) invisible(NULL) } # 4. Writers ---- -# One per file, each fixing the column set so that a downstream -# comparison can rely on it. ## 4.1 write_grid_predictions() ---- @@ -144,15 +207,30 @@ write_grid_predictions <- function( #' @param boot Integer. #' @param survey_unit_id Character vector. #' @param prediction,observed Numeric vectors. -#' @return NULL, invisibly. +#' @param in_bag Logical vector, TRUE where the unit was in the +#' draw. Used when the store keeps out-of-bag units only. +#' @return NULL, invisibly. Nothing is written when the store's +#' `unit_predictions` is "none". #' #' @example # Example usage of the function #' # write_unit_predictions(store, "ALFL", "north", 1, ids, #' # preds, obs) write_unit_predictions <- function( store, species, region, boot, survey_unit_id, prediction, - observed + observed, in_bag = NULL ) { + if (store$unit_predictions == "none") { + return(invisible(NULL)) + } + + # Only the held-out units, where the run asks for those alone + if (store$unit_predictions == "oob" && !is.null(in_bag)) { + keep <- !in_bag + survey_unit_id <- survey_unit_id[keep] + prediction <- prediction[keep] + observed <- observed[keep] + } + result_append(store, "unit_predictions", data.frame( species = species, region = region, @@ -173,9 +251,8 @@ write_unit_predictions <- function( #' @param boot Integer. #' @param coefficients A data frame of term, estimate and se, or #' NULL for an engine that has none. -#' @param stage Character. The stage the coefficients came from. -#' Every stage is recorded, not only the last, so a staged -#' model can be compared stage by stage. +#' @param stage Character. The stage the coefficients came from; +#' every stage is recorded, not only the last. #' @return NULL, invisibly. #' #' @example # Example usage of the function @@ -232,9 +309,8 @@ write_metrics <- function(store, species, region, boot, metrics) { #' Record What a Run Was #' -#' Everything needed to say what produced the other four files. -#' Written as JSON by hand rather than through jsonlite, so the -#' harness keeps its base-R-plus-data.table dependency. +#' Hand-written JSON, not jsonlite, to keep the harness to base R +#' plus data.table. #' #' @param store A result_store(). #' @param meta A named list of scalars and character vectors. @@ -292,9 +368,8 @@ write_meta <- function(store, meta) { #' Read a Result Store Back #' -#' What comparison scripts use. A missing file returns NULL -#' rather than stopping, because an engine without coefficients -#' legitimately writes none. +#' A missing file returns NULL: an engine without coefficients +#' writes none. #' #' @param dir Character. A store directory. #' @param what Character. Which file, without the extension. diff --git a/1_code/harness/run_model.R b/1_code/harness/run_model.R index 5dbe004..9c1c8de 100644 --- a/1_code/harness/run_model.R +++ b/1_code/harness/run_model.R @@ -1,260 +1,363 @@ # --- -# title: Run a Taxon Spec +# title: Run Taxon Specs # author: Brendan Casey # created: 2026-09-09 # inputs: -# a spec from 1_code/modules//spec.R +# specs from 1_code/modules//spec.R # the harmonized test dataset # outputs: -# in run_dir, one result store per region; see result.R +# in each run directory, one result store per region; see +# result.R # notes: -# - The loop a spec drives: region, then species, then -# resampling iteration, then stage. Everything it does is -# taken from the spec, so this file names no taxon. -# - Stages run in order and each may carry the previous one -# forward. v2's habitat models take the fitted climate -# prediction as a term called `Climate`; here that is -# `carry_as` on a stage, and dropping it is how an experiment -# asks whether staging helps at all. -# - The covariates a run needs are read off the model sets -# rather than declared twice. A spec that adds a term to a -# formula therefore cannot forget to load its column. -# - Predictions are written at survey units always, and onto the -# prediction grid when the spec names one. Grid predictions -# are the comparison currency; see docs/framework_design.md. -# - A species or iteration that fails is recorded and the run -# continues. Across 172 species and 100 iterations something -# always fails, and losing the whole run to it would be -# worse than losing the cell. +# - The spec-driven loop; names no taxon. prepare_spec() loads +# each region once; fit_species() is the unit of (parallel) +# work; merge_spec_shards() joins shards in queue order. +# - Stages run in order; `carry_as` passes a stage's prediction +# forward (v2's `Climate` term). +# - Per-species seeds make parallel and serial runs identical. +# - A failed species or draw is logged and the run continues. # --- # 1. Setup ---- ## 1.1 Load packages ---- -# Sourced alongside the rest of the harness; no direct imports. +# parallel (base R), when `workers` is above 1. -# 2. spec_covariates() ---- +# 2. run_specs() and run_spec() ---- -#' Read the Covariates a Spec's Model Sets Name +## 2.1 run_specs() ---- + +#' Run Several Specs Through One Pool of Workers #' -#' Taken from the formulas rather than declared separately, so a -#' spec cannot name a term it forgot to load. Terms the harness -#' supplies itself - the response, the offset, and any stage -#' carried forward - are excluded. +#' Every species of every spec is one job, so few species per +#' taxon still keeps every worker busy. #' -#' @param spec A taxon spec. -#' @param stage_names Character vector of stages to read, or NULL -#' for all of them. -#' @return A character vector of covariate names. +#' @param specs A named list of taxon specs. The names name the +#' runs, and so the folders under `pipeline_dir`. +#' @param data_dir Character. The test dataset folder. +#' @param pipeline_dir Character. Each spec writes to +#' `pipeline_dir///`. +#' @param species Character vector of focal species, named by +#' taxon (see taxon_values()), or NULL for every species each +#' spec's queue declares. +#' @param regions Character vector of regions to run, or NULL for +#' all each spec defines. +#' @param iterations Integer vector of resampling iterations. +#' @param stage_models Named list of replacement candidate sets, +#' or NULL; see apply_stage_models(). +#' @param metrics Character vector of metric names, or NULL for +#' default_metrics(). +#' @param boot_seed Integer or NULL. The base seed; NULL falls +#' back to `spec$resample$seed`, then to unseeded (as v2). +#' @param workers Integer. Species fitted at once. 1 runs in this +#' session; more start background R sessions. +#' @param unit_predictions Character. "none", "oob" or "all"; see +#' result_store(). +#' @param verbose Logical. Report progress. +#' @return A data frame, one row per spec, species, region and +#' draw, recording what happened. #' #' @example # Example usage of the function -#' # spec_covariates(bryophyte_spec) -spec_covariates <- function(spec, stage_names = NULL) { - stages <- spec$stages - - if (!is.null(stage_names)) { - stages <- stages[vapply( - stages, function(s) s$name %in% stage_names, logical(1) - )] +#' # run_specs(list(lichen = lichen_spec()), data_dir, +#' # "2_pipeline/exp_001", iterations = 1:5, workers = 4) +run_specs <- function( + specs, + data_dir, + pipeline_dir, + species = NULL, + regions = NULL, + iterations = 1:100, + stage_models = NULL, + metrics = NULL, + boot_seed = NULL, + workers = 1L, + unit_predictions = c("none", "oob", "all"), + verbose = TRUE +) { + unit_predictions <- match.arg(unit_predictions) + + if (is.null(names(specs)) || any(!nzchar(names(specs)))) { + stop("`specs` must be a named list; the names name the runs.", + call. = FALSE) } - supplied <- c( - spec$response_name %||% "response", - "offset", "weight", - vapply(spec$stages, function(s) s$carry_as %||% "", character(1)), - # Columns a spec attaches per species rather than reads from - # the dataset: the mammal precomputed Climate. - spec$supplied_columns - ) + # Step 1: Prepare every spec first, so a bad spec fails early + prepared <- lapply(stats::setNames(names(specs), names(specs)), + function(key) { + prepare_spec( + spec = specs[[key]], key = key, data_dir = data_dir, + run_dir = file.path(pipeline_dir, key), species = species, + regions = regions, iterations = iterations, + stage_models = stage_models, metrics = metrics, + boot_seed = boot_seed, unit_predictions = unit_predictions + ) + }) - terms <- unlist(lapply(stages, function(stage) { - # A staged set is a list of groups; flatten before reading - # terms off the formulas. - models <- unlist(get_model_set(stage$models), use.names = FALSE) + # Step 2: One job per spec and species + jobs <- unlist(lapply(prepared, function(p) { + lapply(p$species, function(one) list(key = p$key, species = one)) + }), recursive = FALSE) + + if (verbose) { + cat( + "Fitting ", length(jobs), " species across ", length(specs), + " spec(s), ", length(iterations), " draw(s) each, on ", + workers, " worker(s)\n", + sep = "" + ) + } - unlist(lapply(models, function(one) { - all.vars(stats::as.formula(one)) - })) - })) + logs <- run_jobs(jobs, prepared, workers, verbose) - covariates <- setdiff(unique(terms), c(supplied, ".")) + # Step 3: Join each spec's shards into its region stores + for (p in prepared) { + merge_spec_shards(p) + } - # A stage may need columns no formula names: the plant age - # splines read the aged stand-cover columns. - extra <- unlist(lapply(stages, function(stage) stage$extra_covariates)) + log <- do.call(rbind, logs) + rownames(log) <- NULL - # The weight is a covariate too, and is loaded with the rest - unique(c(covariates, extra, spec$weight_column)) + log } -# 3. apply_stage_models() ---- +## 2.2 run_spec() ---- -#' Swap a Spec's Candidate Sets for an Experiment's Own -#' -#' Lets an experiment define the models a stage fits without -#' editing the taxon spec, which keeps the spec as the record of -#' what v2 did and the experiment as the record of what was -#' changed. +#' Run One Spec #' -#' Entries are named `taxon.stage`, or `stage` to apply to every -#' taxon. Each value is either a name in model_sets() or a -#' character vector of formulas, so an experiment can hand over a -#' set built by extend_models() or models_from_covariates(). +#' run_specs() for a single spec, writing to `run_dir//`. #' -#' \preformatted{ -#' stage_models <- list( -#' "lichen.climate" = extend_models( -#' "climate_plant_v2_full", "elevation" -#' ), -#' "climate" = "climate_common_ladder" -#' ) -#' } +#' @param spec A taxon spec. +#' @param data_dir Character. The test dataset folder. +#' @param run_dir Character. Where the region stores go. +#' @param ... Passed to run_specs(): species, regions, +#' iterations, stage_models, metrics, boot_seed, workers, +#' unit_predictions, verbose. +#' @return As run_specs(). #' -#' Because the covariates a run loads are read off the formulas, -#' swapping a set changes what is loaded too. Nothing else has to -#' be declared. +#' @example # Example usage of the function +#' # run_spec(bryophyte_spec(), data_dir, run_dir, +#' # species = c("Aulacomnium.palustre"), +#' # iterations = 1:5) +run_spec <- function(spec, data_dir, run_dir, ...) { + run_specs( + specs = stats::setNames(list(spec), basename(run_dir)), + data_dir = data_dir, + pipeline_dir = dirname(run_dir), + ... + ) +} + +# 3. Preparing a spec ---- + +## 3.1 prepare_spec() ---- + +#' Check a Spec and Load What Its Regions Need #' #' @param spec A taxon spec. -#' @param stage_models Named list of replacement model sets, or -#' NULL. -#' @return The spec, with matching stages replaced, and a record -#' of what changed in `models_overridden`. +#' @param key Character. The run's name. +#' @param data_dir,run_dir Character. +#' @param species,regions,iterations,stage_models,metrics,boot_seed +#' As in run_specs(). +#' @param unit_predictions Character. +#' @return A list: the spec, its regions (each with its data, +#' grid and species), the province-wide data, the species to +#' fit, and the run settings a worker needs. #' #' @example # Example usage of the function -#' # apply_stage_models(spec, list(climate = my_models)) -apply_stage_models <- function(spec, stage_models) { - if (is.null(stage_models) || length(stage_models) == 0) { - return(spec) - } +#' # p <- prepare_spec(lichen_spec(), "lichen", data_dir, run_dir) +prepare_spec <- function( + spec, key, data_dir, run_dir, species = NULL, regions = NULL, + iterations = 1:100, stage_models = NULL, metrics = NULL, + boot_seed = NULL, unit_predictions = "none" +) { + # Before any read, so covariates follow the replacement sets + spec <- apply_stage_models(spec, stage_models) - changed <- character(0) + validate_spec(spec) - spec$stages <- lapply(spec$stages, function(stage) { - # The taxon-specific key wins over the bare stage name, so a - # run can set a default and then override one taxon. - keys <- c(paste(spec$taxon, stage$name, sep = "."), stage$name) - hit <- keys[keys %in% names(stage_models)][1] + # The experiment's seed wins over the spec's + base_seed <- boot_seed %||% spec$resample$seed - if (is.na(hit)) { - return(stage) - } + # Step 1: The species. Named species are validated against the + # whole taxon, then intersected per region (a species may be + # modelled in one region only). + species <- taxon_values(species, spec$taxon) %||% species - stage$models <- stage_models[[hit]] - changed <<- c(changed, paste0(stage$name, " <- ", hit)) + if (!is.null(species)) { + species <- unname(species) + resolve_species(species, data_dir, spec$taxon, tier = spec$tier) + } - stage - }) + regions <- regions %||% names(spec$regions) + unknown_regions <- setdiff(regions, names(spec$regions)) - # Recorded so meta.json shows a run that did not fit the v2 - # candidate sets, which otherwise looks identical to one that - # did. - spec$models_overridden <- if (length(changed) > 0) { - changed - } else { - NULL + if (length(unknown_regions) > 0) { + stop( + "Spec for ", spec$taxon, " defines no region(s) ", + paste(unknown_regions, collapse = ", "), ". Regions: ", + paste(names(spec$regions), collapse = ", "), + call. = FALSE + ) } - spec + named_regions <- stats::setNames(regions, regions) + region_species <- lapply(named_regions, function(r) { + available <- list_species( + data_dir, spec$taxon, region = r, tier = spec$tier, + season = spec$season + ) + + if (is.null(species)) available else intersect(available, species) + }) + + all_species <- unique(unlist(region_species, use.names = FALSE)) + + # Step 2: Each region's stages, data and grid, loaded once + prepared_regions <- lapply( + regions[lengths(region_species[regions]) > 0], + function(r) { + prepare_region(spec, r, region_species[[r]], data_dir) + } + ) + names(prepared_regions) <- vapply( + prepared_regions, `[[`, character(1), "name" + ) + + list( + key = key, + spec = spec, + data_dir = data_dir, + run_dir = run_dir, + shard_dir = file.path(run_dir, "_shards"), + regions = prepared_regions, + province = province_context(spec, data_dir, all_species), + species = all_species, + iterations = iterations, + base_seed = base_seed, + metrics = metrics, + unit_predictions = unit_predictions + ) } -# 4. run_stage() ---- +## 3.2 prepare_region() ---- -#' Fit One Stage on One Draw +#' Resolve One Region's Stages and Load Its Data and Grid #' -#' @param stage A stage definition from a spec. -#' @param data A data frame for one species and one draw. -#' @param spec The taxon spec. -#' @param offset Numeric vector or NULL. -#' @param weights Numeric vector or NULL. -#' @param grid A prediction grid, or NULL. -#' @param species Character. Passed to a rule that declares it. -#' @return A selection result. +#' A stage with no models takes the region's (veg in the north, +#' soil in the south). Formula terms are rewritten to the +#' dataset's block-suffixed names (see term_map()). +#' +#' @param spec A taxon spec. +#' @param region Character. +#' @param species Character vector of the region's species. +#' @param data_dir Character. +#' @return A list of the region's name, spec (with resolved +#' stages), species, model data, grid, grid name, weight column +#' and covariates. #' #' @example # Example usage of the function -#' # run_stage(spec$stages[[1]], d, spec) -run_stage <- function( - stage, data, spec, offset = NULL, weights = NULL, - grid = NULL, species = NA_character_ -) { - models <- get_model_set(stage$models) +#' # prepare_region(lichen_spec(), "north", "Physcia.adscendens", +#' # data_dir) +prepare_region <- function(spec, region, species, data_dir) { + region_spec <- spec$regions[[region]] + rename <- term_map( + data_dir, spec$taxon, region, block = region_spec$term_block + ) - # A staged rule takes groups of formulas; the others take a - # flat list. Both arrive here as text and become formulas now. - as_formulas <- function(x) { - if (is.list(x)) { - return(lapply(x, as_formulas)) + # Step 1: The stages, with the region's models where a stage + # takes them, and every formula in the dataset's names + spec$stages <- lapply(spec$stages, function(stage) { + if (is.null(stage$models)) { + # Plus what goes with them: reference categories, + # constants, and columns the age splines read + stage$models <- region_spec$habitat_models + stage$intercept_cats <- region_spec$intercept_cats %||% + stage$intercept_cats + stage$constants <- region_spec$constants %||% stage$constants + stage$extra_covariates <- c( + stage$extra_covariates, region_spec$extra_covariates + ) + + # Terms held fixed on the grid can be added per region: v2 + # predicts the south at pAspen = 0. + if (!is.null(region_spec$grid_constants)) { + stage$grid_constants <- c( + stage$grid_constants %||% list(), + region_spec$grid_constants + ) + } } - lapply(x, stats::as.formula) - } + stage$models <- apply_term_map( + get_model_set(stage$models), rename + ) - # The offset goes in the formula, not in an argument. An - # offset supplied as an argument is not carried by predict() - # onto new data: R silently recycles the fitted offset against - # the new rows, which for birds means every prediction outside - # the draw is wrong. v2 writes `offset(offset)` in the formula - # for the same reason. - response <- spec$response_name %||% "response" + stage + }) - # A stage may start every candidate from terms of its own. v2's - # bird landcover selection starts from `count ~ climate`, so - # each group updates a model that already carries the climate - # stage; without it the carried climate would never enter. - base_terms <- c("1", stage$base_terms) + # Step 2: The columns to load + weight_column <- region_spec$weight_column %||% spec$weight_column + covariates <- covariate_request( + spec, spec_covariates(spec), region_spec$filter + ) - base <- stats::as.formula(paste( - response, "~", paste(base_terms, collapse = " + "), - if (is.null(offset)) "" else "+ offset(offset)" - )) + # Step 3: Load once for the region; species are cut from it + model_data <- build_model_data( + taxon = spec$taxon, + data_dir = data_dir, + species = species, + covariates = covariates, + region = region, + region_filter = region_spec$filter, + weight_column = weight_column, + aliases = spec$aliases %||% list(), + covariate_files = spec$covariate_files + ) - selection_run( - rule = stage$selection, - models = as_formulas(models), - base = base, - data = data, - engine = get_engine(stage$engine), - family = stage$family %||% spec$family, - weights = weights, - offset = NULL, - ic = stage$ic %||% "AICc", - control = stage$control %||% list(), - # Used only by rules that declare them; see selection_run(). - grid = grid, - grid_constants = stage$grid_constants %||% list(), - head_terms = stage$head_terms %||% c("Intercept", "Climate"), - intercept_cats = get_model_set(stage$intercept_cats), - constants = stage$constants %||% list(), - slope_terms = stage$slope_terms %||% "Climate", - always_advance = stage$always_advance %||% TRUE, + # Step 4: The grid (the spec's, else the manifest's). Columns + # are renamed to the data's names; row names stay as v2 wrote + # them. + grid_name <- region_spec$grid %||% + manifest_entry(data_dir, spec$taxon, region)$grid + grid <- load_prediction_grid(data_dir, grid_name) + + if (!is.null(grid)) { + names(grid) <- apply_term_map(names(grid), rename) + + # e.g. v2 drops the mammal north's WetlandMargin and Climate + grid <- grid[ + !rownames(grid) %in% region_spec$grid_drop_rows, + !names(grid) %in% region_spec$grid_drop_cols, + drop = FALSE + ] + } + + list( + name = region, + spec = spec, species = species, - stage = stage, - scale = stage$scale %||% "link", - calibrate_against = if (isTRUE(stage$calibrate)) { - data[[spec$response_name %||% "response"]] - } else { - NULL - } + model_data = model_data, + grid = grid, + grid_name = grid_name, + weight_column = weight_column, + covariates = covariates ) } -# 5. run_spec() ---- - -## 5.1 province_context() ---- +## 3.3 province_context() ---- #' Assemble the Province-Wide Data a Spec's Province Scope Needs #' -#' v2 fits the plant and bird climate stages once, on every unit -#' in the draw, and draws the plant bootstrap once per species -#' across the province; only the habitat stage is fitted per -#' region. A spec says so with `scope = "province"` on a stage -#' and on `resample`. This loads the unfiltered data those need, -#' once per taxon. +#' v2 fits plant and bird climate stages, and draws the plant +#' bootstrap, province-wide; only habitat is per region. Specs +#' declare this with `scope = "province"`. Loads the unfiltered +#' data once per spec. #' #' @param spec A taxon spec, with stage models already resolved. #' @param data_dir Character. #' @param species Character vector of the species to model. -#' @return A list of `model_data` and `stages`, or NULL when the -#' spec has no province scope. +#' @return A list of `model_data`, `stages` and `draws`, or NULL +#' when the spec has no province scope. #' #' @example # Example usage of the function #' # province_context(spec, data_dir, c("Physcia.adscendens")) @@ -269,21 +372,13 @@ province_context <- function(spec, data_dir, species) { return(NULL) } - # A province stage is fitted before any region exists, so it - # can take nothing from a region-scope stage. + # Province stages run first, so cannot carry from region stages for (stage in province_stages) { - if (!is.null(stage$carry_from)) { - stop( - "Stage `", stage$name, "` has province scope but carries ", - "from `", stage$carry_from, "`; a province stage must come ", - "first.", call. = FALSE - ) - } - - if (is.null(stage$models)) { + if (!is.null(stage$carry_from) || is.null(stage$models)) { stop( - "Stage `", stage$name, "` has province scope but takes its ", - "models from the region.", call. = FALSE + "Stage `", stage$name, "` has province scope, so it must ", + "come first and name its own models.", + call. = FALSE ) } } @@ -295,27 +390,9 @@ province_context <- function(spec, data_dir, species) { character(0) } - covariates <- setdiff( - covariates, c(names(spec$aliases), unlist(spec$aliases)) - ) - - if (!is.null(spec$resample$id_column)) { - covariates <- unique(c(covariates, spec$resample$id_column)) - } - - # The region filters are evaluated on the province frame for - # v2's redraw rule, so their columns are loaded too. - site_supplied <- c( - "lat", "long", "easting", "northing", "nr", "nsr", "luf", "year", - "lured", "location", - "summer_days", "winter_days", "survey_unit_id" - ) - filter_terms <- unlist(lapply(spec$regions, function(one) { - if (is.null(one$filter)) character(0) else all.vars(one$filter) - })) - covariates <- unique(c( - covariates, setdiff(filter_terms, site_supplied) - )) + # v2's redraw rule evaluates region filters on this frame + filters <- lapply(spec$regions, `[[`, "filter") + covariates <- covariate_request(spec, covariates, filters) list( model_data = build_model_data( @@ -326,457 +403,380 @@ province_context <- function(spec, data_dir, species) { region = NULL, region_filter = NULL, weight_column = NULL, - aliases = spec$aliases %||% list() + aliases = spec$aliases %||% list(), + covariate_files = spec$covariate_files ), stages = province_stages, draws = province_draws ) } -## 5.2 run_spec() ---- +## 3.4 covariate_request() ---- -#' Run a Spec Across Species, Draws and Stages +#' The Columns to Load, Given the Covariates the Formulas Name +#' +#' Adds the stored-draw key and region-filter columns; drops +#' aliases and site fields, which do not come from +#' covariates.csv. #' #' @param spec A taxon spec. -#' @param data_dir Character. Path to 0_data/test_dataset. -#' @param run_dir Character. Where result stores are written; one -#' subdirectory per region. -#' @param species Character vector of focal species, or NULL for -#' every species the spec's queue declares. -#' @param regions Character vector of regions to run, or NULL for -#' all the spec defines. -#' @param iterations Integer vector of resampling iterations. -#' @param stage_models Named list of replacement candidate sets, -#' or NULL to fit the spec's own. Keys are `taxon.stage` or -#' `stage`; see apply_stage_models(). Because covariates are -#' read off the formulas, replacing a set changes what is -#' loaded too. -#' @param metrics Character vector of metric names, or NULL. -#' @param boot_seed Integer or NULL. The experiment's base seed. -#' Each species resamples under its own seed derived from it by -#' species_seed(). NULL falls back to the spec's -#' `resample$seed`, and when that is NULL too the draws are -#' unseeded, as in v2. Ignored by the precomputed scheme, whose -#' draws are stored rather than drawn. -#' @param verbose Logical. Report progress per species. -#' @return A data frame, one row per species, region and -#' iteration, recording what happened. +#' @param covariates Character vector from spec_covariates(). +#' @param filters A one-sided formula, a list of them, or NULL. +#' @return A character vector of covariate names. #' #' @example # Example usage of the function -#' # run_spec(bryophyte_spec, data_dir, run_dir, -#' # species = c("Aulacomnium.palustre"), -#' # iterations = 1:5) -run_spec <- function( - spec, - data_dir, - run_dir, - species = NULL, - regions = NULL, - iterations = 1:100, - stage_models = NULL, - metrics = NULL, - boot_seed = NULL, - verbose = TRUE -) { - # An experiment's own candidate sets replace the spec's before - # anything is read, so the covariates follow from them. - spec <- apply_stage_models(spec, stage_models) +#' # covariate_request(spec, spec_covariates(spec), ~ useNorth == 1) +covariate_request <- function(spec, covariates, filters = NULL) { + if (inherits(filters, "formula")) { + filters <- list(filters) + } - # One base seed governs the run. The experiment's wins over the - # spec's, so a run can seed every taxon from one setting. - base_seed <- boot_seed %||% spec$resample$seed - draws_seeded <- !is.null(base_seed) && - spec$resample$scheme != "precomputed" + filter_terms <- unlist(lapply(filters, function(one) { + if (is.null(one)) character(0) else all.vars(one) + })) - if (is.null(regions)) { - regions <- names(spec$regions) - } + covariates <- setdiff( + covariates, c(names(spec$aliases), unlist(spec$aliases)) + ) - log_rows <- list() + unique(c( + covariates, + spec$resample$id_column, + setdiff(filter_terms, c(site_columns(), "survey_unit_id")) + )) +} - log_outcome <- function(species_name, region, status, note) { - for (boot in iterations) { - log_rows[[length(log_rows) + 1]] <<- data.frame( - taxon = spec$taxon, species = species_name, region = region, - boot = as.integer(boot), status = status, note = note, - stringsAsFactors = FALSE - ) - } - } +# 4. Fitting one species ---- - # A named species is validated once against the whole taxon, - # so a typo stops the run here. Per region it is then - # intersected rather than re-validated: a species modelled in - # the north and not the south is a fact about the species, not - # a mistake by the caller. - # `species` may be a plain vector or one named by taxon; take - # this taxon's entries either way. - species <- taxon_values(species, spec$taxon) %||% species +## 4.1 run_jobs() ---- - if (!is.null(species) && !is.null(names(species))) { - species <- unname(species) +#' Fit Every Job, in This Session or in Parallel +#' +#' In parallel, jobs are handed out as workers free up, so a +#' slow species does not hold up the rest. +#' +#' @param jobs List of `key` and `species` pairs. +#' @param prepared Named list of prepare_spec() results. +#' @param workers Integer. +#' @param verbose Logical. +#' @return A list of log data frames, one per job, in job order. +#' +#' @example # Example usage of the function +#' # run_jobs(jobs, prepared, workers = 4) +run_jobs <- function(jobs, prepared, workers = 1L, verbose = TRUE) { + if (workers <= 1L || length(jobs) <= 1L) { + return(lapply(jobs, function(job) { + log <- fit_species(job$species, prepared[[job$key]]) + if (verbose) cat(".") + log + })) } - if (!is.null(species)) { - resolve_species( - species, data_dir, spec$taxon, tier = spec$tier - ) + project_root <- getOption("sdm.project_root") + + if (is.null(project_root)) { + stop("Parallel runs need load_framework() to have been ", + "called, so workers can load it too.", call. = FALSE) } - # Step 1: The species each region models. A species is - # modelled in one region and not another, so the queue is per - # region rather than per taxon. - region_queue <- lapply(stats::setNames(regions, regions), function(r) { - if (is.null(spec$regions[[r]])) { - stop( - "Spec for ", spec$taxon, " defines no region `", r, - "`. Regions: ", paste(names(spec$regions), collapse = ", "), - call. = FALSE - ) - } + # Step 1: Start the workers and load the framework + cluster <- parallel::makePSOCKcluster(min(workers, length(jobs))) + on.exit(parallel::stopCluster(cluster), add = TRUE) - available <- list_species( - data_dir, spec$taxon, region = r, tier = spec$tier, - season = spec$season - ) + parallel::clusterCall(cluster, load_worker, project_root) - if (is.null(species)) available else intersect(available, species) + # Step 2: Each job carries only its own spec's prepared data + tasks <- lapply(jobs, function(job) { + list(species = job$species, prepared = prepared[[job$key]]) }) - # Step 2: The province-wide data, draws and stage fits, shared - # by every region. Built once per taxon; each species' draws - # and fits are made on first use and reused by later regions. - province <- province_context( - spec, data_dir, unique(unlist(region_queue)) + parallel::clusterApplyLB(cluster, tasks, fit_species_worker) +} + +## 4.2 load_worker() ---- + +#' Load the Framework in a Background Session +#' +#' @param project_root Character. The repository root. +#' @return NULL, invisibly. +#' +#' @example # Example usage of the function +#' # parallel::clusterCall(cluster, load_worker, root) +load_worker <- function(project_root) { + session <- globalenv() + + source( + file.path(project_root, "1_code", "harness", "harness.R"), + local = session ) - province_cache <- new.env() + load <- get("load_framework", envir = session) + load(project_root, envir = session) - province_for_species <- function(one_species) { - if (!is.null(province_cache[[one_species]])) { - return(province_cache[[one_species]]) - } + invisible(NULL) +} - frame <- model_frame( - province$model_data, - species = one_species, - transform = spec$response_transform %||% "identity" +## 4.3 fit_species_worker() ---- + +#' Fit One Job in a Background Session +#' +#' Self-contained, so a worker needs only the job. +#' +#' @param task List of `species` and `prepared`. +#' @return The job's log data frame. +#' +#' @example # Example usage of the function +#' # fit_species_worker(task) +fit_species_worker <- function(task) { + get("fit_species", envir = globalenv())(task$species, task$prepared) +} + +## 4.4 fit_species() ---- + +#' Fit One Species in Every Region of a Spec +#' +#' Writes a shard store per region. Province-wide draws and +#' stage fits are made once and shared by the regions. +#' +#' @param species Character. +#' @param p A prepare_spec() result. +#' @return A data frame, one row per region and draw. +#' +#' @example # Example usage of the function +#' # fit_species("Physcia.adscendens", prepared$lichen) +fit_species <- function(species, p) { + spec <- p$spec + seed <- species_seed(p$base_seed, spec$taxon, species) + logs <- list() + + log_all <- function(region, status, note) { + data.frame( + taxon = spec$taxon, species = species, region = region, + boot = as.integer(p$iterations), status = status, note = note, + stringsAsFactors = FALSE ) + } - draws <- NULL + # Step 1: Province-wide frame and draws, if any. A species + # whose draws cannot be made is logged and skipped. + province <- NULL + province_fits <- list() + + if (!is.null(p$province)) { + province <- list(frame = model_frame( + p$province$model_data, species = species, + transform = spec$response_transform %||% "identity" + )) - if (province$draws) { - draws <- tryCatch( + if (p$province$draws) { + province$draws <- tryCatch( resample_for_species( - spec, province$model_data, frame, iterations, data_dir, - seed = species_seed(base_seed, spec$taxon, one_species), - species = one_species + spec, province$frame, p$iterations, p$data_dir, + seed = seed, species = species ), error = function(e) conditionMessage(e) ) } + } - entry <- list(frame = frame, draws = draws, fits = list()) - province_cache[[one_species]] <- entry + for (region in p$regions) { + if (!species %in% region$species) { + next + } - entry - } + shard <- result_store( + file.path(p$shard_dir, region$name, safe_name(species)), + header = FALSE, unit_predictions = p$unit_predictions + ) - # A province stage is fitted once per species and draw, on the - # draw's province-wide rows, and reused by every region. - province_fits <- function(one_species, entry, boot, draw) { - key <- as.character(boot) + frame <- model_frame( + region$model_data, species = species, + transform = spec$response_transform %||% "identity", + weight_column = region$weight_column + ) - if (!is.null(entry$fits[[key]])) { - return(entry$fits[[key]]) + # Per-species columns (e.g. mammal precomputed climate) + if (is.function(spec$species_frame)) { + frame <- spec$species_frame(frame, species, p$data_dir, spec) } - rows <- match(draw, entry$frame$survey_unit_id) - fitting <- entry$frame[rows[!is.na(rows)], , drop = FALSE] - - fits <- lapply(province$stages, function(stage) { + # Step 2: The draws. Province-wide draws are filtered to the + # region's units when fitted, as v2 subsets them. + draws <- if (!is.null(province$draws)) { + province$draws + } else { tryCatch( - fit_one_stage(stage, fitting, spec, NULL, one_species), - error = function(e) { - list(status = "error", note = conditionMessage(e), - selected = NULL) - } + resample_for_species( + spec, frame, p$iterations, p$data_dir, + seed = seed, species = species + ), + error = function(e) conditionMessage(e) ) - }) - names(fits) <- vapply(province$stages, `[[`, character(1), "name") - - entry$fits[[key]] <- fits - province_cache[[one_species]] <- entry - - fits - } - - for (region in regions) { - region_spec <- spec$regions[[region]] - region_species <- region_queue[[region]] - - if (length(region_species) == 0) { - next } - if (verbose) { - cat( - "\n", spec$taxon, " / ", region, ": ", - length(region_species), " species x ", - length(iterations), " iterations\n", - sep = "" + if (is.character(draws)) { + logs[[region$name]] <- log_all( + region$name, "resample_failed", draws ) + next } - # Step 3: A stage with no models of its own takes the - # region's, which is how one spec fits vegetation types in - # the north and soil types in the south. Covariates are read - # off the resulting formulas, so they differ per region too. - # The v2 formulas name columns as the source files did. Any - # column appearing in more than one block was suffixed by the - # harmonizer, so the names are rewritten here; see - # term_map(). Without it a habitat model asks for HardLin, - # gets a column belonging to another taxon, and reads NA. - rename <- term_map( - data_dir, spec$taxon, region, - block = region_spec$term_block - ) + # Step 3: Every draw, fitting province stages once per draw + outcomes <- lapply(seq_along(p$iterations), function(i) { + boot <- p$iterations[i] + fixed <- NULL - region_spec_stages <- lapply(spec$stages, function(stage) { - # A region that supplies the models may also supply what - # goes with them: the mammal one-hot rule needs each - # candidate's reference category, and the south models - # hold pAspen fixed as well. - if (is.null(stage$models)) { - stage$models <- region_spec$habitat_models - stage$intercept_cats <- region_spec$intercept_cats %||% - stage$intercept_cats - stage$constants <- region_spec$constants %||% stage$constants - stage$extra_covariates <- c( - stage$extra_covariates, region_spec$extra_covariates - ) + if (!is.null(province) && length(p$province$stages) > 0) { + key <- as.character(boot) - # Terms held fixed when predicting onto the grid can be - # added per region: v2 predicts the south at pAspen = 0. - if (!is.null(region_spec$grid_constants)) { - stage$grid_constants <- c( - stage$grid_constants %||% list(), - region_spec$grid_constants + if (is.null(province_fits[[key]])) { + province_fits[[key]] <<- fit_province_stages( + p$province$stages, province$frame, draws[[i]], spec, + species ) } + + fixed <- province_fits[[key]] } - stage$models <- apply_term_map( - get_model_set(stage$models), rename + run_one_draw( + spec = region$spec, frame = frame, draw = draws[[i]], + store = shard, species = species, region = region$name, + boot = boot, grid = region$grid, metrics = p$metrics, + fixed = fixed, + validation_frame = province$frame ) - - stage }) - region_spec_full <- spec - region_spec_full$stages <- region_spec_stages - covariates <- spec_covariates(region_spec_full) - - # A region may weight its models by a column of its own: v2 - # weights the bird landcover models by vegw in the north and - # soilw in the south. - weight_column <- region_spec$weight_column %||% spec$weight_column - - # An alias is made from a column already loaded, so it is - # not itself a covariate to read. - covariates <- setdiff( - covariates, - c(names(spec$aliases), unlist(spec$aliases)) - ) + logs[[region$name]] <- do.call(rbind, outcomes) + } - # A precomputed resampling scheme keys its stored ids on a - # column of its own, so that column is loaded as well. - if (!is.null(spec$resample$id_column)) { - covariates <- unique(c(covariates, spec$resample$id_column)) - } + do.call(rbind, logs) +} - # A region filter that names a covariate has to load it too. - # Birds select their region on useNorth and useSouth, which - # no model formula mentions. Site fields are already loaded - # and are dropped from the request. - if (!is.null(region_spec$filter)) { - filter_terms <- all.vars(region_spec$filter) - site_supplied <- c( - "lat", "long", "easting", "northing", "nr", "nsr", "luf", "year", - "lured", "location", "summer_days", "winter_days", - "survey_unit_id" - ) - covariates <- unique(c( - covariates, setdiff(filter_terms, site_supplied) - )) - } +## 4.5 fit_province_stages() ---- - # Step 4: Load once for the region, then cut per species - model_data <- build_model_data( - taxon = spec$taxon, - data_dir = data_dir, - species = region_species, - covariates = covariates, - region = region, - region_filter = region_spec$filter, - weight_column = weight_column, - aliases = spec$aliases %||% list() +#' Fit the Province-Scope Stages on One Draw +#' +#' @param stages List of province-scope stages. +#' @param frame The species' province-wide frame. +#' @param draw Character vector of survey unit ids. +#' @param spec The taxon spec. +#' @param species Character. +#' @return A named list of fit_one_stage() results. +#' +#' @example # Example usage of the function +#' # fit_province_stages(stages, frame, draws[[1]], spec, sp) +fit_province_stages <- function(stages, frame, draw, spec, species) { + rows <- match(draw, frame$survey_unit_id) + fitting <- frame[rows[!is.na(rows)], , drop = FALSE] + + fits <- lapply(stages, function(stage) { + tryCatch( + fit_one_stage(stage, fitting, spec, NULL, species), + error = function(e) { + list(status = "error", note = conditionMessage(e), + selected = NULL) + } ) + }) - store <- result_store(file.path(run_dir, region)) - - # The grid's columns are renamed to match, but its rownames - - # the habitat type names the coefficients are reported under - # - are left as v2 wrote them. - grid <- load_prediction_grid(data_dir, region_spec$grid) - - if (!is.null(grid)) { - names(grid) <- apply_term_map(names(grid), rename) - - # A region may leave rows or columns of the grid out, as v2 - # drops the mammal north's WetlandMargin and Climate rows. - grid <- grid[ - !rownames(grid) %in% region_spec$grid_drop_rows, - !names(grid) %in% region_spec$grid_drop_cols, - drop = FALSE - ] - } - - for (one_species in region_species) { - frame <- model_frame( - model_data, - species = one_species, - transform = spec$response_transform %||% "identity", - weight_column = weight_column - ) + stats::setNames(fits, vapply(stages, `[[`, character(1), "name")) +} - # A spec may attach per-species columns, and drop rows, - # before fitting: the mammal precomputed climate. - if (is.function(spec$species_frame)) { - frame <- spec$species_frame(frame, one_species, data_dir, spec) - } +# 5. Fitting one draw ---- - # Step 5: The draws. Province-wide ones are shared by every - # region, and filtered to the region's units when fitted, - # which is how v2 subsets its draws. A species whose draws - # cannot be made is logged and skipped, not fatal. - entry <- if (is.null(province)) { - NULL - } else { - province_for_species(one_species) - } - - draws <- if (!is.null(entry) && province$draws) { - entry$draws - } else { - tryCatch( - resample_for_species( - spec, model_data, frame, iterations, data_dir, - seed = species_seed(base_seed, spec$taxon, one_species), - species = one_species - ), - error = function(e) conditionMessage(e) - ) - } +## 5.1 run_stage() ---- - if (is.character(draws)) { - log_outcome(one_species, region, "resample_failed", draws) - next - } +#' Fit One Stage on One Draw +#' +#' @param stage A stage definition from a spec. +#' @param data A data frame for one species and one draw. +#' @param spec The taxon spec. +#' @param offset Numeric vector or NULL. +#' @param weights Numeric vector or NULL. +#' @param grid A prediction grid, or NULL. +#' @param species Character. Passed to a rule that declares it. +#' @return A selection result. +#' +#' @example # Example usage of the function +#' # run_stage(spec$stages[[1]], d, spec) +run_stage <- function( + stage, data, spec, offset = NULL, weights = NULL, + grid = NULL, species = NA_character_ +) { + models <- get_model_set(stage$models) + response <- spec$response_name %||% "response" - for (i in seq_along(iterations)) { - fixed <- if (!is.null(entry) && - length(province$stages) > 0) { - province_fits(one_species, entry, iterations[i], - draws[[i]]) - } else { - NULL - } + # Text to formulas; staged rules take groups + as_formulas <- function(x) { + if (is.list(x)) { + return(lapply(x, as_formulas)) + } - outcome <- run_one_draw( - spec = region_spec_full, frame = frame, - draw = draws[[i]], - units = frame$survey_unit_id, store = store, - species = one_species, region = region, - boot = iterations[i], grid = grid, metrics = metrics, - fixed = fixed, - # The province-wide frame, where there is one, for - # validation that scores climate over every unit - validation_frame = if (is.null(entry)) NULL else entry$frame - ) + lapply(x, stats::as.formula) + } - log_rows[[length(log_rows) + 1]] <- outcome - } + # The offset goes in the formula: an offset argument is not + # applied by predict() to new data (R recycles the fitted one). + # `base_terms` start every candidate from given terms, as v2's + # bird landcover selection starts from `count ~ climate`. + base <- stats::as.formula(paste( + response, "~", paste(c("1", stage$base_terms), collapse = " + "), + if (is.null(offset)) "" else "+ offset(offset)" + )) - if (verbose) { - cat(".") - } - } + # Stage fields the rule declares are passed by name + rule <- if (is.function(stage$selection)) { + stage$selection + } else { + get_method("selection", stage$selection) + } - if (verbose) { - cat("\n") - } + core <- c( + "models", "base", "data", "engine", "family", "weights", + "offset", "ic", "control" + ) + settings <- stage[ + setdiff(intersect(names(stage), names(formals(rule))), core) + ] - write_meta(store, list( - taxon = spec$taxon, - region = region, - # Mammal specs are fitted per season and per hurdle part, - # and the comparison needs both to find the right - # reference. NA for every other taxon. - season = spec$season %||% NA_character_, - part = spec$part %||% NA_character_, - species_n = length(region_species), - iterations = length(iterations), - stages = vapply(spec$stages, function(s) s$name, character(1)), - stage_scope = vapply( - spec$stages, function(s) s$scope %||% "region", character(1) - ), - engines = vapply( - spec$stages, function(s) s$engine, character(1) - ), - selection = vapply( - spec$stages, - function(s) if (is.function(s$selection)) "custom" else s$selection, - character(1) - ), - covariates = covariates, - weight_column = weight_column %||% NA_character_, - resample_scheme = spec$resample$scheme, - resample_scope = spec$resample$scope %||% "region", - # The seed actually used, never a default that was not. - # Per-species seeds are derived from this base; see - # species_seed(). - seed = if (draws_seeded) { - paste0(base_seed, " (per species, via species_seed())") - } else if (spec$resample$scheme == "precomputed") { - "not used (precomputed draws)" - } else { - "unseeded" - }, - grid = region_spec$grid %||% NA_character_, - models_overridden = spec$models_overridden %||% NA_character_, - v2_coverage = if (is.null(spec$v2_coverage)) { - NA_character_ - } else { - paste0( - names(spec$v2_coverage), ": ", - vapply(spec$v2_coverage, `[[`, character(1), "status") - ) - }, - spec_notes = spec$notes %||% NA_character_ - )) + if (!is.null(settings$intercept_cats)) { + settings$intercept_cats <- get_model_set(settings$intercept_cats) } - do.call(rbind, log_rows) + do.call(selection_run, c( + list( + rule = stage$selection, + models = as_formulas(models), + base = base, + data = data, + engine = get_engine(stage$engine), + family = stage$family %||% spec$family, + weights = weights, + offset = NULL, + ic = stage$ic %||% "AICc", + control = stage$control %||% list() + ), + settings, + # Harness-supplied, only to rules that declare them + list( + grid = grid, + species = species, + stage = stage, + calibrate_against = if (isTRUE(stage$calibrate)) { + data[[response]] + } else { + NULL + } + ) + )) } -# 6. run_one_draw() ---- - -## 6.1 fit_one_stage() ---- +## 5.2 fit_one_stage() ---- #' Prepare One Stage's Data and Fit It #' -#' Shared by the per-region loop and by stages fitted once for -#' the whole province, so both prepare a stage the same way. +#' Shared by region-scope and province-scope stages. #' #' @param stage A stage definition. #' @param fitting The draw's rows, with any carried stage added. @@ -797,23 +797,18 @@ fit_one_stage <- function(stage, fitting, spec, grid = NULL, response_name <- spec$response_name %||% "response" stage_data <- fitting - # A stage may read the response differently from the spec's - # default; recomputed from the retained raw values. - if (!is.null(stage$response_transform)) { - raw_name <- paste0(response_name, "_raw") + # Stage-specific response transform, from the raw values + raw_name <- paste0(response_name, "_raw") - if (raw_name %in% names(stage_data)) { - stage_data[[response_name]] <- apply_response_transform( - stage_data[[raw_name]], stage$response_transform, - stage_data - ) - } + if (!is.null(stage$response_transform) && + raw_name %in% names(stage_data)) { + stage_data[[response_name]] <- apply_response_transform( + stage_data[[raw_name]], stage$response_transform, stage_data + ) } - # A stage may fit on a subset: the mammal abundance half is - # estimated from the units where the species was actually - # seen, because abundance given presence is not defined where - # there was none. + # Row subset, e.g. the mammal abundance half fits only units + # where the species was seen if (!is.null(stage$row_filter)) { keep <- stage$row_filter(stage_data) keep[is.na(keep)] <- FALSE @@ -824,8 +819,8 @@ fit_one_stage <- function(stage, fitting, spec, grid = NULL, } } - # v2 skips a plant habitat model outright when the draw, after - # the region filter, holds fewer than 20 detections. + # v2 skips a plant habitat model with fewer than 20 + # detections in the filtered draw if (!is.null(stage$min_detections)) { detections <- sum(stage_data[[response_name]] > 0, na.rm = TRUE) @@ -858,48 +853,43 @@ fit_one_stage <- function(stage, fitting, spec, grid = NULL, return(failed("stage_failed")) } - # A stage may reshape its coefficients after selection with - # steps its spec supplies, applied in order: the plant stand-age - # splines and cutblock convergence. Each takes and returns the - # selection result, and sees the stage's data. They run before - # the footprint pooling below, which is v2's order: the pooling - # borrows from a spline-fitted age class. + # Spec-supplied post-processing, in order (e.g. plant stand-age + # splines, cutblock convergence, footprint pooling). Steps that + # declare `grid`, `species` or `stage` are given them. for (step in stage$post_process) { - selected <- step(selected, stage_data) - } - - # v2 pools a few poorly sampled footprint coefficients with - # types they are assumed to resemble. Applied after the - # averaging, as v2 does, and only where a stage asks. - if (isTRUE(stage$coef_adjust)) { - selected$coefficients <- coef_adjust_plant_veg( - selected$coefficients - ) + context <- list(grid = grid, species = species, stage = stage) + context <- context[names(context) %in% names(formals(step))] + selected <- do.call(step, c(list(selected, stage_data), context)) } list(status = "ok", note = NA_character_, selected = selected) } -## 6.2 carry_values() ---- +## 5.3 carry_values() ---- #' The Value a Stage Hands the Next One #' -#' Computed from the averaged coefficients, not from the best -#' single candidate: the averaging is the point of the stage. +#' From the stage's final (averaged) model, not its best +#' candidate: averaged coefficients applied to the data, as v2 +#' does, or the final model's prediction for engines without +#' coefficients. #' -#' @param coefficients A data frame of term and estimate. +#' @param selected A selection result. #' @param data The rows to compute it for. #' @param stage The stage being carried. #' @return A numeric vector, one per row. #' #' @example # Example usage of the function -#' # carry_values(selected$coefficients, fitting, stage) -carry_values <- function(coefficients, data, stage) { - link <- predict_from_coefficients(coefficients, data, "link") +#' # carry_values(selected, fitting, stage) +carry_values <- function(selected, data, stage) { + link <- if (!is.null(selected$coefficients)) { + predict_from_coefficients(selected$coefficients, data, "link") + } else { + selected$predict(data, "link") + } - # v2's bird climate is predicted from the fitted models, whose - # linear predictor includes the QPAD offset, so the landcover - # model sees the expected count rather than the rate. + # v2's bird climate is carried without the QPAD offset, as + # MuMIn predicts it; a stage can ask for the offset to be added. if (isTRUE(stage$carry_offset) && "offset" %in% names(data)) { link <- link + data$offset } @@ -917,17 +907,15 @@ carry_values <- function(coefficients, data, stage) { ) } -## 6.3 run_one_draw() ---- +## 5.4 run_one_draw() ---- #' Fit Every Stage on One Species and One Draw #' -#' @param spec A taxon spec. -#' @param frame The full one-species frame. +#' @param spec A taxon spec, with the region's stages. +#' @param frame The full one-species frame for the region. #' @param draw Character vector of survey unit ids in the draw. -#' Ids not in `units` are dropped, so a province-wide draw -#' passed to one region keeps only that region's units, with -#' their repeats, as v2 filters its draws. -#' @param units Character vector of the frame's survey unit ids. +#' Ids not in the frame are dropped (repeats kept), as v2 +#' filters province-wide draws per region. #' @param store A result_store(). #' @param species,region Character. #' @param boot Integer. @@ -935,15 +923,15 @@ carry_values <- function(coefficients, data, stage) { #' @param metrics Character vector of metric names, or NULL. #' @param fixed Named list of fit_one_stage() results for stages #' already fitted at province scope, or NULL. -#' @param validation_frame The province-wide frame, or NULL; passed -#' to the spec's `validate` function, which scores climate over -#' every unit. +#' @param validation_frame The province-wide frame, or NULL; +#' passed to the spec's `validate` function. #' @return A one-row data frame recording the outcome. #' #' @example # Example usage of the function -#' # run_one_draw(spec, frame, draw, units, store, sp, "north", 1) +#' # run_one_draw(spec, frame, draw, store, sp, "north", 1, grid, +#' # NULL) run_one_draw <- function( - spec, frame, draw, units, store, species, region, boot, grid, + spec, frame, draw, store, species, region, boot, grid, metrics, fixed = NULL, validation_frame = NULL ) { outcome <- function(status, note = NA_character_) { @@ -954,7 +942,7 @@ run_one_draw <- function( ) } - rows <- match(draw, units) + rows <- match(draw, frame$survey_unit_id) rows <- rows[!is.na(rows)] if (length(rows) == 0) { @@ -964,22 +952,21 @@ run_one_draw <- function( fitting <- frame[rows, , drop = FALSE] response_name <- spec$response_name %||% "response" + # Step 1: Fit the stages in order result <- tryCatch({ carried <- NULL last <- NULL stage_coefficients <- list() for (stage in spec$stages) { - # A stage that carries the previous one forward gets it as - # a column, which is how v2's habitat models take Climate + # Carried stage as a column (v2's `Climate` term) if (!is.null(stage$carry_from) && !is.null(carried)) { - fitting[[stage$carry_from_as %||% "Climate"]] <- - carried$fitting - frame[[stage$carry_from_as %||% "Climate"]] <- carried$full + column <- stage$carry_from_as %||% "Climate" + fitting[[column]] <- carried$fitting + frame[[column]] <- carried$full } - # A province-scope stage was fitted once for the species - # and draw, and is shared by every region. + # Province-scope stages were fitted once per draw fitted <- if (stage$name %in% names(fixed)) { fixed[[stage$name]] } else { @@ -994,34 +981,28 @@ run_one_draw <- function( if (!is.null(stage$carry_as)) { carried <- list( - fitting = carry_values(selected$coefficients, fitting, stage), - full = carry_values(selected$coefficients, frame, stage) + fitting = carry_values(selected, fitting, stage), + full = carry_values(selected, frame, stage) ) } - # Every stage's coefficients are kept, not only the last, - # so each stage can be compared against its own v2 - # reference. They are written only once the whole draw - # succeeds, so a store never holds half a draw. - # A rule with several outputs - the mammal hurdle's - # presence, abundance and total tables - writes each as its - # own stage, `_`. + # Every stage's coefficients are kept, for per-stage v2 + # comparison. Multi-output rules (the mammal hurdle) write + # `_`. if (!is.null(selected$outputs)) { for (output in names(selected$outputs)) { - stage_coefficients[[paste(stage$name, output, sep = "_")]] <- - selected$outputs[[output]] + key <- paste(stage$name, output, sep = "_") + stage_coefficients[[key]] <- selected$outputs[[output]] } } else { stage_coefficients[[stage$name]] <- selected$coefficients } - last <- list( - selected = selected, engine = get_engine(stage$engine), - stage = stage - ) + last <- list(selected = selected, stage = stage) } last$stage_coefficients <- stage_coefficients + last$frame <- frame last }, error = function(e) { @@ -1032,8 +1013,24 @@ run_one_draw <- function( return(result) } - # Step 1: Record every stage's coefficients, then the final - # stage's predictions + frame <- result$frame + + # Step 2: The final model's prediction at every unit (or the + # spec's `final_prediction`, as the plant specs give v2's) + predicted <- tryCatch( + if (is.function(spec$final_prediction)) { + spec$final_prediction(result$stage_coefficients, frame) + } else { + result$selected$predict(frame, "response") + }, + error = function(e) NULL + ) + + if (is.null(predicted) || length(predicted) != nrow(frame)) { + return(outcome("predict_failed", result$stage$name)) + } + + # Written only once the whole draw succeeded for (stage_name in names(result$stage_coefficients)) { write_coefficients( store, species, region, boot, @@ -1042,24 +1039,15 @@ run_one_draw <- function( ) } - predicted <- result$engine$predict( - result$selected$fit, frame, "response" - ) - - if (is.null(predicted)) { - return(outcome("predict_failed", result$stage$name)) - } + in_bag <- frame$survey_unit_id %in% draw write_unit_predictions( store, species, region, boot, frame$survey_unit_id, - predicted, frame[[response_name]] + predicted, frame[[response_name]], in_bag = in_bag ) - # Metrics are scored twice: on the units the draw fitted - # (in-sample) and on the units it left out (out-of-bag), which - # is the held-out read. Iteration 1 is the full data and has no - # out-of-bag units, so its out-of-bag metrics are NA. - in_bag <- frame$survey_unit_id %in% draw + # Step 3: Score in-sample and out-of-bag. A full-data draw + # (bootstrap iteration 1) has no out-of-bag units, so NA. scored <- function(keep, prefix) { out <- compute_metrics( frame[[response_name]][keep], predicted[keep], metrics @@ -1073,8 +1061,7 @@ run_one_draw <- function( scored(!in_bag, "oob_") ) - # A spec may add its own validation: the plant specs add v2's - # seven validation AUCs, in-bag and out-of-bag. + # Spec validation, e.g. the plant specs' v2 validation AUCs if (is.function(spec$validate)) { extra <- tryCatch( spec$validate( @@ -1084,18 +1071,19 @@ run_one_draw <- function( error = function(e) NULL ) - if (!is.null(extra)) { - metric_rows <- rbind(metric_rows, extra) - } + metric_rows <- rbind(metric_rows, extra) } write_metrics(store, species, region, boot, metric_rows) - # Step 2: Project onto the prediction grid, the quantity that - # makes engines comparable + # Step 4: Project the final model onto the prediction grid if (!is.null(grid)) { grid_prediction <- predict_grid( - result$selected, result$engine, grid + result$selected, grid, + constants = c( + result$stage$constants %||% list(), + result$stage$grid_constants %||% list() + ) ) if (!is.null(grid_prediction)) { @@ -1109,191 +1097,119 @@ run_one_draw <- function( outcome("ok") } -# 7. resample_for_species() ---- +# 6. Writing the stores ---- + +## 6.1 merge_spec_shards() ---- -#' Draw the Resampling Sets for One Species +#' Join a Spec's Shards into One Store per Region #' -#' @param spec A taxon spec. -#' @param model_data From build_model_data(). Kept for the -#' signature; the draws are made from `frame`, which carries the -#' same covariates and may have dropped rows. -#' @param frame The one-species frame, one row per unit that -#' can be drawn. -#' @param iterations Integer vector. -#' @param data_dir Character. -#' @param seed Integer or NULL. This species' seed, from -#' species_seed(). Replaces any seed in the spec; NULL leaves -#' the draw unseeded. -#' @param species Character. Names the stored draws where they -#' are kept per species. -#' @return A list of character vectors of survey unit ids. +#' Joins shards in queue order (matching a serial run), writes +#' meta.json, and removes the shards. +#' +#' @param p A prepare_spec() result. +#' @return NULL, invisibly. #' #' @example # Example usage of the function -#' # resample_for_species(spec, md, frame, 1:10, data_dir, -#' # seed = 123L, species = "Physcia.adscendens") -resample_for_species <- function( - spec, model_data, frame, iterations, data_dir, seed = NULL, - species = NULL -) { - scheme <- spec$resample$scheme - - # The spec's seed is a base, not a species seed, so it is - # dropped here and the derived one passed instead. `scope` is - # read by run_spec(), not by a resampler. - args <- spec$resample[ - setdiff(names(spec$resample), c("scheme", "seed", "scope")) - ] - - if (scheme == "precomputed") { - return(do.call(resample_precomputed, c( - list( - data_dir = data_dir, taxon = spec$taxon, - iterations = iterations, frame = frame, - species = species - ), - args - ))) - } - - if (scheme == "spatial_block") { - response <- frame[[spec$response_name %||% "response"]] - - # v2 redraws a sample only when the full data could meet the - # threshold in at least one model region; otherwise it takes - # the first draw. Checked per region, on the full data. - if (!is.null(args$min_detections) && args$min_detections > 0) { - viable <- any(vapply(spec$regions, function(one) { - # A region whose filter names columns this frame lacks - - # another region's, when the draw is not province-wide - - # cannot be judged here, and does not count. - if (!is.null(one$filter) && - !all(all.vars(one$filter) %in% names(frame))) { - return(FALSE) - } - - in_region <- if (is.null(one$filter)) { - rep(TRUE, nrow(frame)) - } else { - keep <- eval(rlang_f_rhs(one$filter), frame) - !is.na(keep) & keep - } - - sum(response[in_region] > 0, na.rm = TRUE) >= - args$min_detections - }, logical(1))) +#' # merge_spec_shards(prepared$lichen) +merge_spec_shards <- function(p) { + for (region in p$regions) { + store <- result_store( + file.path(p$run_dir, region$name), + unit_predictions = p$unit_predictions + ) - if (!viable) { - args$min_detections <- 0L - } - } + merge_result_shards( + store, + file.path(p$shard_dir, region$name, safe_name(region$species)) + ) - return(do.call(resample_spatial_block, c( - # `seed = NULL` is passed through deliberately: it means - # unseeded, not the resampler's harness_seed() default. - list( - frame = frame, iterations = iterations, - response = response, seed = seed - ), - args - ))) + write_meta(store, run_meta(p, region)) } - do.call(resample_units, c( - list( - scheme = scheme, frame = frame, - iterations = iterations, seed = seed - ), - args - )) -} - -# 8. Helpers ---- + unlink(p$shard_dir, recursive = TRUE) -## 8.1 `%||%` ---- - -#' Default for NULL -#' -#' @param a,b Any values. -#' @return `a` unless it is NULL, in which case `b`. -#' -#' @example # Example usage of the function -#' # NULL %||% "default" -`%||%` <- function(a, b) { - if (is.null(a)) b else a + invisible(NULL) } -## 8.2 v2_status() ---- +## 6.2 run_meta() ---- -#' State How Much of One v2 Stage a Spec Reproduces -#' -#' The entries of a spec's `v2_coverage`. The status vocabulary -#' is fixed, so a typo fails when the spec is built rather than -#' reading as a fourth category in the report. +#' What a Region Store Records About Its Run #' -#' @param status Character. "reproduced", "partial" or -#' "not reproduced". -#' @param note Character. What is and is not reproduced, and why. -#' @return A list of `status` and `note`. +#' @param p A prepare_spec() result. +#' @param region A prepare_region() result. +#' @return A named list for write_meta(). #' #' @example # Example usage of the function -#' # v2_status("partial", "No age splines.") -v2_status <- function(status, note) { - allowed <- c("reproduced", "partial", "not reproduced") +#' # run_meta(prepared$lichen, prepared$lichen$regions$north) +run_meta <- function(p, region) { + spec <- p$spec + stages <- region$spec$stages - if (!is.character(status) || length(status) != 1L || - !status %in% allowed) { - stop( - "v2 status must be one of: ", - paste(allowed, collapse = ", "), ".", - call. = FALSE - ) - } - - if (!is.character(note) || length(note) != 1L || - !nzchar(note)) { - stop("A v2 status needs a one-line note.", call. = FALSE) - } - - list(status = status, note = note) + list( + taxon = spec$taxon, + region = region$name, + # Mammals only: needed to find the right v2 reference + season = spec$season %||% NA_character_, + part = if (any(vapply( + stages, function(s) identical(s$selection, "hurdle"), logical(1) + ))) { + "hurdle" + } else { + NA_character_ + }, + species_n = length(region$species), + iterations = length(p$iterations), + stages = vapply(stages, `[[`, character(1), "name"), + stage_scope = vapply( + stages, function(s) s$scope %||% "region", character(1) + ), + engines = vapply(stages, `[[`, character(1), "engine"), + selection = vapply( + stages, + function(s) { + if (is.function(s$selection)) "custom" else s$selection + }, + character(1) + ), + covariates = region$covariates, + weight_column = region$weight_column %||% NA_character_, + resample_scheme = spec$resample$scheme, + resample_scope = spec$resample$scope %||% "region", + # The seed actually used, never a default that was not + seed = if (spec$resample$scheme == "precomputed") { + "not used (precomputed draws)" + } else if (!is.null(p$base_seed)) { + paste0(p$base_seed, " (per species, via species_seed())") + } else { + "unseeded" + }, + grid = region$grid_name %||% NA_character_, + unit_predictions = p$unit_predictions, + models_overridden = spec$models_overridden %||% NA_character_, + v2_coverage = if (is.null(spec$v2_coverage)) { + NA_character_ + } else { + paste0( + names(spec$v2_coverage), ": ", + vapply(spec$v2_coverage, `[[`, character(1), "status") + ) + }, + spec_notes = spec$notes %||% NA_character_ + ) } -## 8.3 spec_coverage() ---- +## 6.3 safe_name() ---- -#' Tabulate What a Set of Specs Reproduces of v2 -#' -#' Flattens each spec's `v2_coverage` into one table, which is -#' what the report prints. Coverage is stated once, in the specs, -#' and read from there. +#' Make a Species Name Safe as a Folder Name #' -#' @param specs A named list of specs. -#' @return A data frame of taxon, stage, status and note. +#' @param x Character vector. +#' @return `x` with anything but letters, digits, dots, hyphens +#' and underscores replaced. #' #' @example # Example usage of the function -#' # spec_coverage(list(lichen = lichen_spec())) -spec_coverage <- function(specs) { - rows <- lapply(names(specs), function(key) { - coverage <- specs[[key]]$v2_coverage - - if (is.null(coverage) || length(coverage) == 0) { - return(data.frame( - taxon = key, stage = NA_character_, - status = "not stated", - note = "The spec has no v2_coverage.", - stringsAsFactors = FALSE - )) - } - - data.frame( - taxon = key, - stage = names(coverage), - status = vapply(coverage, `[[`, character(1), "status"), - note = vapply(coverage, `[[`, character(1), "note"), - stringsAsFactors = FALSE, - row.names = NULL - ) - }) - - do.call(rbind, rows) +#' # safe_name("Black Bear") # "Black_Bear" +safe_name <- function(x) { + gsub("[^A-Za-z0-9._-]", "_", x) } # End of script ---- diff --git a/1_code/harness/run_record.R b/1_code/harness/run_record.R index 909b80d..33d53f7 100644 --- a/1_code/harness/run_record.R +++ b/1_code/harness/run_record.R @@ -8,44 +8,14 @@ # outputs: # - one Markdown or plain-text file per run # notes: -# - The result stores are large and gitignored. This writes the -# one committed artefact of a run: what it was configured to -# do, what it produced, and what it scored. When the stores -# are gone, this is what is left. -# - The format is built to be diffed. Two runs of the same -# configuration should produce byte-identical bodies, so any -# difference in a diff is a real difference in the run. -# Three things follow from that and are not style choices: -# -# 1. Everything that changes between identical runs - the -# timestamp, the host, the elapsed time - is confined to -# one Provenance section at the top, and can be dropped -# with `provenance = FALSE`. Interleaved, it would make -# every diff non-empty and therefore useless. -# 2. Numbers are formatted to a fixed number of decimals. -# Unformatted doubles differ in the last place between -# runs that agree, which reads as a difference. -# 3. Rows and columns are ordered deterministically, and a -# missing value is written as "-" rather than dropped, so -# tables stay aligned and a diff points at the value that -# changed rather than at a row that moved. -# - What the format guarantees is that identical results -# render identically. It cannot make an unseeded pipeline -# reproduce. The plant-group specs pass `seed = NULL` to -# match v2, so two runs of the same configuration draw -# different bootstrap samples and their Metrics sections -# genuinely differ. Birds resample from stored ids and are -# deterministic. So in a diff between two runs, treat the -# Configuration and Coverage sections as exact and the -# Metrics section as exact only for seeded taxa. -# - File sizes are deliberately not recorded. They vary by a -# few bytes between runs that agree, which is noise in a -# document whose purpose is comparison. Row counts carry the -# same information and are stable. -# - "Record" rather than "manifest": a manifest is an inventory -# of contents, and this leads with results. The inventory is -# one section of it. See the repository discussion in -# docs/framework_design.md. +# - The committed record of a run (configuration, coverage, +# metrics); the result stores themselves are gitignored. +# - Built to diff: identical results render byte-identically. +# Volatile fields sit only in the Provenance section +# (`provenance = FALSE` drops it), numbers use fixed decimals, +# rows are ordered deterministically, and missing values are +# "-". An unseeded run (`seed = NULL`) still differs between +# runs. # --- # 1. Setup ---- @@ -73,11 +43,8 @@ fmt_value <- function(x, digits = 4L) { return("-") } - # Decided by type rather than by value. Choosing the number - # of decimals from whether the values happen to be whole - # would let the same column render as "1" in one run and - # "1.0000" in the next, which is a formatting change showing - # up as a result change. + # Format by type, not value, so a column never flips between + # "1" and "1.0000" across runs if (is.integer(x)) { out <- formatC(x, format = "d") out[is.na(x)] <- "-" @@ -104,9 +71,8 @@ fmt_value <- function(x, digits = 4L) { #' Render a Data Frame as a Markdown Table #' -#' @param df A data frame. Zero rows yields a stated absence -#' rather than an empty table, so the section still appears -#' and the diff shows what changed. +#' @param df A data frame. Zero rows renders as "_None._" so the +#' section still appears in a diff. #' @param digits Integer. Decimal places for numeric columns. #' @return A character vector of lines. #' @@ -178,79 +144,7 @@ kv_lines <- function(values, digits = 4L) { ) } -# 3. discover_stores() ---- - -#' Find the Result Stores Under a Pipeline Directory -#' -#' Stores are laid out as ///. A -#' directory holding none of the result files is not a store and -#' is skipped rather than reported empty. -#' -#' @param pipeline_dir Character. The run's pipeline directory. -#' @return A data frame of dir, taxon and region, ordered -#' deterministically. -#' -#' @example # Example usage of the function -#' # discover_stores("2_pipeline/exp_000_parity_v2") -discover_stores <- function(pipeline_dir) { - empty <- data.frame( - dir = character(0), taxon = character(0), - region = character(0), stringsAsFactors = FALSE - ) - - if (!dir.exists(pipeline_dir)) { - return(empty) - } - - known <- c( - "metrics.csv", "coefficients.csv", "grid_predictions.csv", - "unit_predictions.csv", "meta.json" - ) - - candidates <- list.dirs(pipeline_dir, recursive = TRUE) - candidates <- setdiff(candidates, pipeline_dir) - - is_store <- vapply( - candidates, - function(one) any(file.exists(file.path(one, known))), - logical(1) - ) - - candidates <- candidates[is_store] - - if (length(candidates) == 0) { - return(empty) - } - - # Stripped by length rather than by a regex on the path. - # tempdir() returns backslashes on Windows while list.dirs() - # returns forward slashes, so a pattern built from the raw - # path silently fails to match, and every store is then filed - # under a taxon named after the drive. - normalise <- function(x) { - normalizePath(x, winslash = "/", mustWork = FALSE) - } - - root <- sub("/+$", "", normalise(pipeline_dir)) - relative <- substring(normalise(candidates), nchar(root) + 2L) - - parts <- strsplit(relative, "/") - - out <- data.frame( - dir = candidates, - taxon = vapply(parts, function(p) p[1], character(1)), - region = vapply( - parts, - function(p) if (length(p) > 1) p[2] else NA_character_, - character(1) - ), - stringsAsFactors = FALSE - ) - - out[order(out$taxon, out$region), , drop = FALSE] -} - -# 4. run_record() ---- +# 3. run_record() ---- #' Write the Committed Record of One Run #' @@ -263,10 +157,10 @@ discover_stores <- function(pipeline_dir) { #' taxa, species count, draws, engines, selection rules. #' Rendered sorted by name. #' @param provenance Logical. Include the volatile section. -#' FALSE gives a body that is byte-identical between two runs -#' of the same configuration, which is what makes a diff -#' between experiments readable. +#' FALSE gives a body byte-identical across identical runs. #' @param digits Integer. Decimal places for every number. +#' @param collected collect_results() output for this pipeline +#' directory, or NULL to read the stores here. #' @return The path written, invisibly. #' #' @example # Example usage of the function @@ -280,9 +174,12 @@ run_record <- function( exp_id, config = list(), provenance = TRUE, - digits = 4L + digits = 4L, + collected = NULL ) { - stores <- discover_stores(pipeline_dir) + if (is.null(collected)) { + collected <- collect_results(pipeline_dir) + } lines <- c( paste("# Run record:", exp_id), @@ -294,9 +191,7 @@ run_record <- function( "" ) - ## 4.1 Provenance ---- - # Everything that differs between two identical runs lives - # here and nowhere else, so the rest of the file diffs clean. + ## 3.1 Provenance ---- if (provenance) { commit <- tryCatch( system2("git", c("rev-parse", "--short", "HEAD"), @@ -325,49 +220,23 @@ run_record <- function( ) } - ## 4.2 Configuration ---- + ## 3.2 Configuration ---- lines <- c( lines, "## Configuration", "", kv_lines(config, digits), "" ) - ## 4.3 Coverage and artefacts ---- - # The inventory. Row counts rather than file sizes: both say - # what was produced, only one is stable between runs. - coverage <- NULL - metrics_all <- list() - - if (nrow(stores) > 0) { - rows <- lapply(seq_len(nrow(stores)), function(i) { - dir_one <- stores$dir[i] - coefficients <- read_results(dir_one, "coefficients") - metrics <- read_results(dir_one, "metrics") - grid <- read_results(dir_one, "grid_predictions") - - if (!is.null(metrics) && nrow(metrics) > 0) { - metrics$taxon <- stores$taxon[i] - metrics$region <- stores$region[i] - metrics_all[[dir_one]] <<- metrics - } - - data.frame( - taxon = stores$taxon[i], - region = stores$region[i], - species = length(unique(c( - coefficients$species, metrics$species - ))), - draws = length(unique(c(coefficients$boot, metrics$boot))), - coefficient_rows = if (is.null(coefficients)) { - 0L - } else { - nrow(coefficients) - }, - metric_rows = if (is.null(metrics)) 0L else nrow(metrics), - grid_rows = if (is.null(grid)) 0L else nrow(grid), - stringsAsFactors = FALSE - ) - }) - - coverage <- do.call(rbind, rows) + ## 3.3 Coverage and artefacts ---- + # Row counts, not file sizes: sizes vary between identical runs + coverage <- collected$store_rows + + if (!is.null(coverage)) { + names(coverage)[names(coverage) == "run"] <- "taxon" + coverage <- coverage[order(coverage$taxon, coverage$region), ] + coverage$coefficient_rows <- as.integer(coverage$coefficient_rows) + coverage$metric_rows <- as.integer(coverage$metric_rows) + coverage$grid_rows <- as.integer(coverage$grid_rows) + coverage$species <- as.integer(coverage$species) + coverage$draws <- as.integer(coverage$draws) } lines <- c( @@ -375,58 +244,25 @@ run_record <- function( md_table(coverage, digits), "" ) - ## 4.4 Metrics ---- - # One row per taxon, region and metric, ordered so that two - # records line up row for row. + ## 3.4 Metrics ---- + # Pooled over species and draws metric_summary <- NULL - if (length(metrics_all) > 0) { - combined <- do.call(rbind, metrics_all) - - split_key <- interaction( - combined$taxon, combined$region, combined$metric, - drop = TRUE - ) - - metric_summary <- do.call(rbind, lapply( - split(combined, split_key), - function(block) { - values <- block$value[is.finite(block$value)] - - data.frame( - taxon = block$taxon[1], - region = block$region[1], - metric = block$metric[1], - n = length(values), - median = if (length(values)) { - stats::median(values) - } else { - NA_real_ - }, - p10 = if (length(values)) { - stats::quantile(values, 0.1, names = FALSE) - } else { - NA_real_ - }, - p90 = if (length(values)) { - stats::quantile(values, 0.9, names = FALSE) - } else { - NA_real_ - }, - stringsAsFactors = FALSE - ) - } - )) - - rownames(metric_summary) <- NULL - - metric_summary <- metric_summary[ - order( - metric_summary$taxon, metric_summary$region, - metric_summary$metric - ), , - drop = FALSE + if (!is.null(collected$metrics) && nrow(collected$metrics) > 0) { + pooled <- as.data.table(collected$metrics)[ + is.finite(value), + list( + n = .N, + median = stats::median(value), + p10 = stats::quantile(value, 0.1, names = FALSE), + p90 = stats::quantile(value, 0.9, names = FALSE), + mean = mean(value), + sd = if (.N < 2) NA_real_ else stats::sd(value) + ), + by = list(taxon = run, region, metric) ] + setorder(pooled, taxon, region, metric) + metric_summary <- as.data.frame(pooled) } lines <- c( @@ -434,12 +270,9 @@ run_record <- function( md_table(metric_summary, digits), "" ) - ## 4.5 Write ---- - # A .txt record drops the table pipes rather than the tables, - # so the same content survives in a plain-text reader. + ## 3.5 Write ---- if (grepl("[.]txt$", out_file, ignore.case = TRUE)) { - # Dividers go first: once the pipes are gone the line is - # no longer recognisable as a divider. + # Drop dividers before the pipes, while still recognizable lines <- lines[!grepl("^[|] --- ([|] --- )*[|]$", lines)] lines <- gsub("^[|] | [|]$", "", lines) lines <- gsub(" [|] ", " ", lines) diff --git a/1_code/harness/selection.R b/1_code/harness/selection.R index d77255b..a211056 100644 --- a/1_code/harness/selection.R +++ b/1_code/harness/selection.R @@ -1,68 +1,29 @@ # --- -# title: Model Selection Rules +# title: Selection Plumbing # author: Brendan Casey # created: 2026-09-09 # inputs: none -# outputs: none; returns objects in memory +# outputs: none; defines functions in memory # notes: -# - A selection rule turns a candidate model set into one set of -# coefficients. The three taxa use three different rules, and -# each is registered here so a spec names one rather than -# implementing it. -# - `single` fits one formula. `aic_best` keeps the lowest AICc, -# which is what the mammal models do. `aic_average` weights -# candidates by AICc and averages their coefficients, which is -# what the plant models do. `staged_bic` walks groups of -# formulas, keeping the best of each group as the base for the -# next, which is what the bird models do. -# - A rule fits its own candidates rather than being handed -# fitted objects, because staged selection has to refit as it -# goes: each stage updates the previous stage's winner. One -# interface then covers both shapes. -# - Every rule returns the same structure, so the fitting loop -# and the result writer do not know which rule ran. -# - Coefficient averaging is over the terms present, treating a -# term absent from a candidate as zero. That is what "this -# model says the effect is nil" means, and it is how the v2 -# plant code assembles its coefficient template. +# - Shared plumbing for the rules in 1_code/methods/selection/. +# Every rule returns selection_result(), so callers need not +# know which rule ran. +# - Rules fit their own candidates rather than receiving fits, +# because staged selection refits each stage on the previous +# stage's winner. # --- # 1. Setup ---- ## 1.1 Load packages ---- -# Base R only; engines.R supplies the fitting and scoring. +# Base R only. -# 2. Selection registry ---- - -## 2.1 selection_registry() ---- - -#' The Available Selection Rules -#' -#' @return A named list of functions, each taking the arguments -#' selection_run() passes. -#' -#' @example # Example usage of the function -#' # names(selection_registry()) -selection_registry <- function() { - list( - single = select_single, - aic_best = select_aic_best, - aic_average = select_aic_average, - staged_bic = select_staged_bic, - ivw_grid = select_ivw_grid, - aic_best_grid = select_aic_best_grid, - aic_best_onehot = select_aic_best_onehot - ) -} - -## 2.2 selection_run() ---- +# 2. selection_run() ---- #' Run One Selection Rule #' -#' The single entry point the fitting loop calls. -#' -#' @param rule Character, a name in selection_registry(), or a -#' function a spec supplies with the same arguments. +#' @param rule Character, a registered rule name, or a function a +#' spec supplies with the same arguments. #' @param models A list of formulas, or for staged rules a list #' of named groups of formulas. #' @param base A formula the candidates update, e.g. @@ -74,10 +35,11 @@ selection_registry <- function() { #' @param weights,offset Numeric vectors or NULL. #' @param ic Character. "AIC", "AICc" or "BIC". #' @param control Named list passed to the engine. -#' @param ... Passed to the rule, for arguments only some rules -#' take - the prediction grid the ivw_grid rule needs, say. -#' @return A list with `fit`, `coefficients`, `ic_table`, -#' `n_fitted`, `n_failed` and `rule`. +#' @param ... Further arguments; only those the rule declares are +#' passed on. +#' @return A selection result; see selection_result(). `rule` is +#' set to the rule's name, and `predict` to a predictor for the +#' final model when the rule did not supply one. #' #' @example # Example usage of the function #' # selection_run("aic_average", models, response ~ 1, d, @@ -95,33 +57,18 @@ selection_run <- function( control = list(), ... ) { - # A rule is a registered name, or a function a spec supplies - # for a model the registry has no shape for: the v2 mammal - # hurdle is one. - registry <- selection_registry() - + # A function rule covers shapes the registry lacks (e.g. the v2 + # mammal hurdle) if (is.function(rule)) { rule_fn <- rule rule_name <- "custom" } else { - if (!rule %in% names(registry)) { - stop( - "Unknown selection rule `", rule, "`. Registered: ", - paste(names(registry), collapse = ", "), - call. = FALSE - ) - } - - rule_fn <- registry[[rule]] + rule_fn <- get_method("selection", rule) rule_name <- rule } - # Only pass the extra arguments a rule actually declares, so a - # spec can set a grid for every stage without the rules that - # do not use one failing on it. extra <- list(...) - accepted <- names(formals(rule_fn)) - extra <- extra[names(extra) %in% accepted] + extra <- extra[names(extra) %in% names(formals(rule_fn))] out <- do.call(rule_fn, c( list( @@ -134,6 +81,16 @@ selection_run <- function( out$rule <- rule_name + # A rule that combines candidates supplies its own predictor; + # otherwise the final model is the chosen fit. + if (is.null(out$predict) && !is.null(out$fit)) { + chosen <- out$fit + + out$predict <- function(newdata, type = "response") { + engine$predict(chosen, newdata, type) + } + } + out } @@ -148,7 +105,8 @@ selection_run <- function( #' @param data A data frame. #' @param engine An engine definition. #' @param family,weights,offset,ic,control As in selection_run(). -#' @return A list with `fits`, `formulas` and `scores`. +#' @return A list with `fits`, `formulas` and `scores`. Scores are +#' Inf where the engine has no information criterion. #' #' @example # Example usage of the function #' # fit_candidates(models, response ~ 1, d, engine, "binomial") @@ -166,14 +124,15 @@ fit_candidates <- function( formulas <- lapply(models, function(one) resolve_formula(one, base)) fits <- lapply(formulas, function(f) { - engine$fit( - formula = f, data = data, family = family, - weights = weights, offset = offset, control = control - ) + fit_with(engine, f, data, family, weights, offset, control) }) scores <- vapply( - fits, function(f) engine$ic(f, ic), numeric(1) + fits, + function(f) { + if (engine_has(engine, "ic")) engine$ic(f, ic) else Inf + }, + numeric(1) ) list(fits = fits, formulas = formulas, scores = scores) @@ -183,8 +142,8 @@ fit_candidates <- function( #' Apply a Candidate to a Base Formula #' -#' A candidate may be a complete formula or an update such as -#' `. ~ . + MAP`. Both forms appear in the v2 model sets. +#' v2 model sets mix complete formulas and updates such as +#' `. ~ . + MAP`. #' #' @param model A formula. #' @param base A formula to update. @@ -193,8 +152,7 @@ fit_candidates <- function( #' @example # Example usage of the function #' # resolve_formula(. ~ . + MAP, response ~ 1) resolve_formula <- function(model, base) { - # A one-sided candidate, or one whose left side is a dot, is an - # update; anything else is complete in itself. + # One-sided or dot-left candidates are updates left <- if (length(model) == 3) deparse(model[[2]]) else "." if (left == ".") { @@ -204,13 +162,36 @@ resolve_formula <- function(model, base) { model } -## 3.3 candidate_table() ---- +## 3.3 formula_variables() ---- + +#' The Data Columns a Formula Reads +#' +#' Read from the formula, not a fit, so it works for any engine. +#' Excludes response, offset and weight, which a prediction frame +#' never supplies as data. +#' +#' @param formula A formula, or formula text. +#' @return A character vector of column names. +#' +#' @example # Example usage of the function +#' # formula_variables(response ~ MAP + offset(offset)) +formula_variables <- function(formula) { + if (is.character(formula)) { + formula <- stats::as.formula(formula) + } + + setdiff(all.vars(formula), c("response", "offset", "weight")) +} + +## 3.4 candidate_table() ---- #' Summarize a Candidate Set #' #' @param fitted A list from fit_candidates(). #' @param names_in Character vector of candidate names, or NULL. -#' @return A data frame of model, ic, delta, weight, k and ok. +#' @return A data frame of model, formula, ic, delta, weight, k +#' and ok. `k` counts coefficients, NA where the engine has +#' none. #' #' @example # Example usage of the function #' # candidate_table(fitted) @@ -229,7 +210,7 @@ candidate_table <- function(fitted, names_in = NULL) { k <- vapply( fitted$fits, function(f) { - cf <- fit_coefficients(f) + cf <- if (isTRUE(f$ok)) f$coefficients else NULL if (is.null(cf)) NA_integer_ else nrow(cf) }, integer(1) @@ -239,8 +220,7 @@ candidate_table <- function(fitted, names_in = NULL) { best <- suppressWarnings(min(scores, na.rm = TRUE)) delta <- scores - best - # Akaike weights. A failed model scores Inf, so exp(-Inf/2) is - # zero and it carries no weight. + # Akaike weights; a failed model scores Inf and gets weight 0 raw <- exp(-delta / 2) raw[!is.finite(raw)] <- 0 total <- sum(raw) @@ -260,768 +240,91 @@ candidate_table <- function(fitted, names_in = NULL) { ) } -# 4. Selection rules ---- - -## 4.1 select_single() ---- - -#' Fit One Model, No Selection -#' -#' @param models A list holding one formula, or a single formula. -#' @param base,data,engine,family,weights,offset,ic,control As in -#' selection_run(). -#' @return A selection result. -#' -#' @example # Example usage of the function -#' # select_single(list(. ~ . + MAP), response ~ 1, d, engine, -#' # "binomial") -select_single <- function( - models, base, data, engine, family, - weights = NULL, offset = NULL, ic = "AICc", control = list() -) { - if (inherits(models, "formula")) { - models <- list(models) - } - - fitted <- fit_candidates( - models[1], base, data, engine, family, weights, offset, ic, - control - ) - - selection_result( - fit = fitted$fits[[1]], - coefficients = fit_coefficients(fitted$fits[[1]]), - ic_table = candidate_table(fitted), - fitted = fitted - ) -} +# 4. Combining predictions ---- -## 4.2 select_aic_best() ---- +## 4.1 family_linkinv() ---- -#' Keep the Single Best-Scoring Candidate +#' The Inverse Link of a Family #' -#' The mammal rule. -#' -#' @inheritParams select_single -#' @return A selection result. +#' @param family Character (e.g. "binomial") or a family object. +#' @return A function from the link scale to the response scale. #' #' @example # Example usage of the function -#' # select_aic_best(models, response ~ 1, d, engine, "binomial") -select_aic_best <- function( - models, base, data, engine, family, - weights = NULL, offset = NULL, ic = "AICc", control = list() -) { - fitted <- fit_candidates( - models, base, data, engine, family, weights, offset, ic, - control - ) - - table <- candidate_table(fitted) - - if (!any(table$ok)) { - return(selection_result(NULL, NULL, table, fitted)) +#' # family_linkinv("poisson")(0) +family_linkinv <- function(family) { + if (is.character(family)) { + family <- get(family, mode = "function")() } - winner <- which.min(fitted$scores) - - selection_result( - fit = fitted$fits[[winner]], - coefficients = fit_coefficients(fitted$fits[[winner]]), - ic_table = table, - fitted = fitted - ) + family$linkinv } -## 4.3 select_aic_average() ---- +## 4.2 combined_predictor() ---- -#' Average Candidates by Information-Criterion Weight -#' -#' The plant rule. Coefficients are averaged across candidates -#' with Akaike weights, treating a term absent from a candidate -#' as zero - that model's statement that the effect is nil. +#' A Predictor That Averages Several Fits on the Link Scale #' -#' @inheritParams select_single -#' @return A selection result. `fit` is the best single candidate, -#' kept so that predictions onto new data remain possible; -#' `coefficients` are the averaged ones. +#' Returned as `predict` by combining rules, so scoring and grid +#' projection use the combined model, not the best candidate. #' -#' @example # Example usage of the function -#' # select_aic_average(models, response ~ 1, d, engine, -#' # "binomial") -select_aic_average <- function( - models, base, data, engine, family, - weights = NULL, offset = NULL, ic = "AICc", control = list() -) { - fitted <- fit_candidates( - models, base, data, engine, family, weights, offset, ic, - control - ) - - table <- candidate_table(fitted) - - if (!any(table$ok) || sum(table$weight) == 0) { - return(selection_result(NULL, NULL, table, fitted)) - } - - # Step 1: Collect every term any candidate estimated - per_model <- lapply(fitted$fits, fit_coefficients) - terms <- unique(unlist(lapply(per_model, function(x) x$term))) - - # Step 2: Weighted mean per term, absent terms counting as zero - averaged <- do.call(rbind, lapply(terms, function(term) { - estimate <- 0 - variance <- 0 - - for (i in seq_along(per_model)) { - cf <- per_model[[i]] - w <- table$weight[i] - - if (is.null(cf) || w == 0) { - next - } - - # A term the candidate lacks, or estimated as NA because it - # is aliased, counts as zero, as in MuMIn's full average. - row <- match(term, cf$term) - absent <- is.na(row) || is.na(cf$estimate[row]) - value <- if (absent) 0 else cf$estimate[row] - se <- if (absent) 0 else cf$se[row] - - estimate <- estimate + w * value - variance <- variance + w * ifelse(is.na(se), 0, se^2) - } - - data.frame( - term = term, - estimate = estimate, - se = sqrt(variance), - stringsAsFactors = FALSE - ) - })) - - selection_result( - fit = fitted$fits[[which.min(fitted$scores)]], - coefficients = averaged, - ic_table = table, - fitted = fitted - ) -} - -## 4.4 select_staged_bic() ---- - -#' Walk Groups of Candidates, Carrying the Winner Forward -#' -#' The bird rule. `models` is a list of named groups. Each group -#' is fitted as an update to the previous group's winner, and the -#' winner is the smallest model within `threshold` of the best -#' score - a parsimony tie-break, not simply the minimum. -#' -#' @inheritParams select_single -#' @param threshold Numeric. How far above the best score a -#' candidate may sit and still be eligible. -#' @param always_advance Logical. TRUE, v2's rule, takes each -#' group's winner as the next base whatever its score; FALSE -#' takes it only when it improves on the running model. -#' @return A selection result, with `ic_table` carrying a `stage` -#' column. -#' -#' @example # Example usage of the function -#' # select_staged_bic(model_groups, response ~ 1, d, engine, -#' # "poisson", ic = "BIC") -select_staged_bic <- function( - models, base, data, engine, family, - weights = NULL, offset = NULL, ic = "BIC", control = list(), - threshold = 2, always_advance = TRUE -) { - # Step 1: Fit the base model. It is the starting point every - # group updates, and the fallback if no group improves on it. - current <- engine$fit( - formula = base, data = data, family = family, - weights = weights, offset = offset, control = control - ) - current_formula <- base - tables <- list() - - if (!isTRUE(current$ok)) { - return(selection_result( - NULL, NULL, - data.frame( - stage = "base", model = "base", formula = deparse(base), - ic = Inf, delta = NA_real_, weight = 0, k = NA_integer_, - ok = FALSE, stringsAsFactors = FALSE - ), - NULL - )) - } - - group_names <- names(models) - - if (is.null(group_names)) { - group_names <- paste0("stage_", seq_along(models)) - } - - # Step 2: Each group updates the running winner - for (i in seq_along(models)) { - fitted <- fit_candidates( - models[[i]], current_formula, data, engine, family, - weights, offset, ic, control - ) - - table <- candidate_table(fitted) - table$stage <- group_names[i] - tables[[i]] <- table - - eligible <- which(table$ok & is.finite(fitted$scores)) - - if (length(eligible) == 0) { - next - } - - # Step 3: Among candidates within `threshold` of the best, - # take the one with fewest parameters. This is a parsimony - # rule, and it is why the winner is not simply which.min(). - best <- min(fitted$scores[eligible]) - close <- eligible[fitted$scores[eligible] <= best + threshold] - winner <- close[which.min(table$k[close])] - - # Step 4: Carry the group's winner forward. v2 always does, - # even when it scores worse than the model it updated - # (07.ModelLandcover.R, section 14); `always_advance = FALSE` - # keeps the running model instead unless the group improves - # on it. - if (always_advance || - fitted$scores[winner] < engine$ic(current, ic)) { - current <- fitted$fits[[winner]] - current_formula <- fitted$formulas[[winner]] - } - } - - selection_result( - fit = current, - coefficients = fit_coefficients(current), - ic_table = do.call(rbind, tables), - fitted = NULL, - formula = current_formula - ) -} - -## 4.5 select_ivw_grid() ---- - -#' Inverse-Variance Average Predictions onto a Grid -#' -#' The plant habitat rule, and the one place where a "coefficient" -#' is not a regression coefficient. v2 fits each candidate, then -#' predicts it onto the prediction matrix - one row per habitat -#' type - and combines those predictions across candidates -#' weighting each by the precision of its own prediction. The -#' result is an effect per habitat type on the link scale, which -#' is what the v2 coefficient tables hold. -#' -#' Weighting by precision rather than by AICc is a different -#' claim: a candidate that is confident about a habitat type -#' dominates there even if it is a poor model overall, and the -#' weighting is per habitat type rather than per model. -#' -#' Non-converged candidates are dropped, not down-weighted, -#' matching v2. A zero standard error is floored at 1e-4, because -#' a precision weight of 1/0 would take the whole average. -#' -#' @inheritParams select_single -#' @param grid A prediction grid, one row per habitat type. -#' @param grid_constants Named list of values for terms the grid -#' has no column for - v2 predicts at `Climate = 0` and, where -#' Protocol is fitted, at the new protocol. -#' @param head_terms Character vector of leading model -#' coefficients to average separately and prepend, which is how -#' v2 carries Intercept, Climate and Protocol. -#' @return A selection result whose `coefficients` are the -#' averaged per-habitat-type effects. +#' @param fits List of engine fits. +#' @param engine An engine definition. +#' @param family Character or family object. +#' @param weights Numeric vector of fixed weights, one per fit, or +#' NULL to weight each prediction by its own precision +#' (inverse-variance), which needs the `se` capability. +#' @return A function(newdata, type = "response") returning a +#' numeric vector. NULL when nothing can predict, or, under +#' fixed weights, when any weighted fit cannot. #' #' @example # Example usage of the function -#' # select_ivw_grid(models, response ~ 1, d, engine, "binomial", -#' # grid = veg_grid, -#' # grid_constants = list(Climate = 0)) -select_ivw_grid <- function( - models, base, data, engine, family, - weights = NULL, offset = NULL, ic = "AICc", control = list(), - grid = NULL, grid_constants = list(), - head_terms = c("Intercept", "Climate") -) { - if (is.null(grid)) { - stop( - "The ivw_grid rule needs a prediction grid; the spec's ", - "region defines one with `grid`.", - call. = FALSE - ) - } - - fitted <- fit_candidates( - models, base, data, engine, family, weights, offset, ic, - control - ) - - table <- candidate_table(fitted) - converged <- vapply(fitted$fits, fit_converged, logical(1)) - - if (!any(converged)) { - return(selection_result(NULL, NULL, table, fitted)) - } - - # Step 1: Predict every converged candidate onto the grid, at - # the constants v2 holds the non-habitat terms at - newdata <- grid - - for (one in names(grid_constants)) { - newdata[[one]] <- grid_constants[[one]] +#' # predict <- combined_predictor(fits, engine, "binomial", w) +#' # predict(newdata) +combined_predictor <- function(fits, engine, family, weights = NULL) { + linkinv <- family_linkinv(family) + + if (!is.null(weights)) { + keep <- weights > 0 + fits <- fits[keep] + weights <- weights[keep] / sum(weights[keep]) } - predictions <- lapply(seq_along(fitted$fits), function(i) { - if (!converged[i]) { + function(newdata, type = "response") { + if (length(fits) == 0) { return(NULL) } - engine$predict(fitted$fits[[i]], newdata, "link", se = TRUE) - }) - - usable <- !vapply(predictions, is.null, logical(1)) - - if (!any(usable)) { - return(selection_result(NULL, NULL, table, fitted)) - } - - estimate <- do.call( - rbind, lapply(predictions[usable], function(x) x$fit) - ) - error <- do.call( - rbind, lapply(predictions[usable], function(x) x$se.fit) - ) - - # A zero standard error would carry infinite weight - error[error == 0] <- 1e-4 - - grid_coefficients <- ivw_combine(estimate, error) - rownames_out <- rownames(grid) - - # Step 2: The leading coefficients are averaged the same way, - # from the model's own coefficient table rather than from a - # prediction - head_estimate <- do.call(rbind, lapply( - which(converged), function(i) { - cf <- fit_coefficients(fitted$fits[[i]]) - cf$estimate[seq_along(head_terms)] - } - )) - head_error <- do.call(rbind, lapply( - which(converged), function(i) { - cf <- fit_coefficients(fitted$fits[[i]]) - cf$se[seq_along(head_terms)] - } - )) - head_error[head_error == 0 | is.na(head_error)] <- 1e-4 - - head_coefficients <- ivw_combine(head_estimate, head_error) - - coefficients <- data.frame( - term = c(head_terms, rownames_out), - estimate = c(head_coefficients$estimate, - grid_coefficients$estimate), - se = c(head_coefficients$se, grid_coefficients$se), - stringsAsFactors = FALSE - ) - - # Step 3: The site-level prediction, averaged the same way. v2 - # keeps it as `data$prediction` and uses it as the offset of - # its stand-age splines. It averages every fitted candidate, - # converged or not; one that failed outright is left out here, - # where v2's would turn the whole average into NaN. - site <- lapply(which(table$ok), function(i) { - engine$predict(fitted$fits[[i]], data, "link", se = TRUE) - }) - site <- site[!vapply(site, is.null, logical(1))] - - site_prediction <- if (length(site) == 0) { - NULL - } else { - site_fit <- do.call(rbind, lapply(site, `[[`, "fit")) - site_se <- do.call(rbind, lapply(site, `[[`, "se.fit")) - colSums(site_fit / site_se^2) / colSums(1 / site_se^2) - } - - result <- selection_result( - fit = fitted$fits[[which.min(fitted$scores)]], - coefficients = coefficients, - ic_table = table, - fitted = fitted - ) - result$site_prediction <- site_prediction - - # Each fitted candidate's own coefficients, for steps that - # average a term the grid does not carry: v2's pAspen. - result$candidate_coefficients <- lapply( - which(table$ok), function(i) fit_coefficients(fitted$fits[[i]]) - ) - - result -} - -## 4.6 ivw_combine() ---- - -#' Combine Estimates by Inverse-Variance Weight -#' -#' @param estimate Matrix of candidates by columns. -#' @param error Matrix of standard errors, the same shape. -#' @return A list of `estimate` and `se`, one per column. -#' -#' @example # Example usage of the function -#' # ivw_combine(rbind(c(1, 2), c(1.2, 2.1)), -#' # rbind(c(0.1, 0.2), c(0.3, 0.1))) -ivw_combine <- function(estimate, error) { - precision <- 1 / error^2 - - # The variance of the combined estimate is the reciprocal of - # the summed precision, not its mean, so adding candidates - # sharpens rather than dilutes it. - list( - estimate = colSums(estimate * precision) / colSums(precision), - se = sqrt(1 / colSums(precision)) - ) -} - -## 4.7 coef_adjust_plant_veg() ---- - -#' Borrow Strength for Poorly Sampled Footprint Types -#' -#' v2's `coef.adjust`. Some human footprint types are rarely the -#' dominant cover at a survey unit, so their effect is estimated -#' from little data. v2 pools each with a type it is assumed to -#' resemble, by inverse-variance weight: -#' -#' - `HardLin` borrows from `UrbInd`. -#' - The three soft linear types - `EnSoftLin`, `EnSeismic` and -#' `TrSoftLin` - each borrow from a composite of young -#' regenerating stands, weighted by how much those stand types -#' overlap soft linear features in the provincial summary. -#' -#' The weights are v2's, measured from a 1 km summary and fixed -#' since 2020-11-17. They are an assumption about which habitats -#' resemble which, not an estimate, so they are stated here -#' rather than derived. -#' -#' @param coefficients A data frame of term, estimate and se. -#' @param overlap Numeric vector of four proportions, for -#' white spruce, pine, deciduous and black spruce regeneration. -#' @return The coefficients, with the four terms adjusted. -#' -#' @example # Example usage of the function -#' # coef_adjust_plant_veg(selected$coefficients) -coef_adjust_plant_veg <- function( - coefficients, - overlap = c(0.049, 0.0893, 0.434, 0.396) -) { - if (is.null(coefficients)) { - return(NULL) - } - - value <- stats::setNames( - coefficients$estimate, coefficients$term - ) - error <- stats::setNames(coefficients$se, coefficients$term) - - # Pool two estimates by precision. - pool <- function(a, a_se, b, b_se) { - precision <- 1 / a_se^2 + 1 / b_se^2 + if (is.null(weights)) { + predictions <- lapply(fits, function(f) { + engine$predict(f, newdata, "link", se = TRUE) + }) - list( - estimate = (a / a_se^2 + b / b_se^2) / precision, - se = sqrt(1 / precision) - ) - } - - present <- function(...) { - all(c(...) %in% names(value)) && - all(is.finite(value[c(...)])) && - all(is.finite(error[c(...)])) - } - - # Step 1: Hard linear features borrow from urban and industrial - if (present("HardLin", "UrbInd")) { - pooled <- pool( - value["HardLin"], error["HardLin"], - value["UrbInd"], error["UrbInd"] - ) - value["HardLin"] <- pooled$estimate - error["HardLin"] <- pooled$se - } - - # Step 2: Soft linear features borrow from young regeneration - young_types <- c( - "CCWhiteSpruceR", "CCPineR", "CCDeciduousR", "BlackSpruce1" - ) + # Precision weights renormalize, so a candidate that cannot + # predict is dropped, as v2 drops a failed fit + predictions <- predictions[ + !vapply(predictions, is.null, logical(1)) + ] - if (present(young_types)) { - weights <- overlap / sum(overlap) - - young <- sum(weights * value[young_types]) - # The weighted variance carries no between-type component, - # because the weights are fixed rather than estimated. - young_se <- sqrt(sum(weights * error[young_types]^2)) - - for (one in c("EnSoftLin", "EnSeismic", "TrSoftLin")) { - if (!present(one)) { - next + if (length(predictions) == 0) { + return(NULL) } - pooled <- pool(value[one], error[one], young, young_se) - value[one] <- pooled$estimate - error[one] <- pooled$se - } - } - - coefficients$estimate <- unname(value[coefficients$term]) - coefficients$se <- unname(error[coefficients$term]) - - coefficients -} - -## 4.8 select_aic_best_grid() ---- - -#' Best Model, Predicted onto a Prediction Grid -#' -#' The mammal abundance-given-presence rule. Where the presence -#' half reads one habitat type at a time from an identity grid, -#' this one predicts the winning model onto the shared prediction -#' matrix, and reports the result on the link scale. -#' -#' No calibration is applied. v2 calibrates the product of the -#' two halves multiplicatively rather than each half separately, -#' because an additive shift on a log-scale abundance can send a -#' small prediction negative and `log` of that is not a number. -#' -#' @inheritParams select_single -#' @param grid A prediction grid. -#' @param constants Named list of values held fixed across the -#' grid - sampling effort and climate. -#' @param scale Character. "link" reports the linear predictor, -#' which for a log-link Gamma is log abundance; "response" -#' exponentiates it. -#' @return A selection result, one effect per grid row. -#' -#' @example # Example usage of the function -#' # select_aic_best_grid(models, response ~ 1, d, engine, -#' # Gamma(link = "log"), grid = grid) -select_aic_best_grid <- function( - models, base, data, engine, family, - weights = NULL, offset = NULL, ic = "AICc", control = list(), - grid = NULL, constants = list(), scale = "link" -) { - if (is.null(grid)) { - stop( - "The aic_best_grid rule needs a prediction grid.", - call. = FALSE - ) - } - - fitted <- fit_candidates( - models, base, data, engine, family, weights, offset, ic, - control - ) - - table <- candidate_table(fitted) - - if (!any(table$ok)) { - return(selection_result(NULL, NULL, table, fitted)) - } - - best <- fitted$fits[[which.min(fitted$scores)]] - - newdata <- grid - - for (one in names(constants)) { - newdata[[one]] <- constants[[one]] - } - - # A grid row describes a pure stand, so a term the grid has no - # column for is genuinely absent rather than unknown. - terms <- attr(stats::terms(best$fit), "term.labels") - - for (one in setdiff(terms, names(newdata))) { - newdata[[one]] <- 0 - } - - predicted <- engine$predict(best, newdata, scale, se = TRUE) - - if (is.null(predicted)) { - return(selection_result(NULL, NULL, table, fitted)) - } - - selection_result( - fit = best, - coefficients = data.frame( - term = rownames(grid), - estimate = predicted$fit, - se = predicted$se.fit, - stringsAsFactors = FALSE - ), - ic_table = table, - fitted = fitted - ) -} - -## 4.9 select_aic_best_onehot() ---- - -#' Best Model, Read One Habitat Type at a Time -#' -#' The mammal presence rule. Unlike the plant habitat stage there -#' is no fixed prediction matrix: v2 reads each effect by setting -#' that one term to 1 and every other to 0, so the grid is an -#' identity over whichever terms the winning model happens to -#' carry. -#' -#' Three things make the result a probability rather than a -#' coefficient: -#' -#' - Each prediction is passed through the logistic, so an effect -#' is the modelled probability of presence in a pure stand of -#' that type. -#' - `Climate` is a slope, not a habitat type, so it is taken -#' from the fitted coefficient rather than from a one-hot -#' prediction, and is left out of the calibration below. -#' - The whole set is shifted on the logit scale so that mean -#' fitted presence matches mean observed presence. Without it -#' the effects are internally consistent but sit at the wrong -#' level. -#' -#' The reference category - the land cover the winning formula -#' omits - is reported alongside the fitted terms, because a -#' coefficient here means "relative to that type" and the table -#' is unreadable without it. -#' -#' @inheritParams select_single -#' @param intercept_cats Character vector, parallel to `models`, -#' naming each candidate's reference category. -#' @param constants Named list of values every one-hot -#' prediction holds fixed - v2 uses 100 sampling days at zero -#' climate. -#' @param calibrate_against Numeric vector of observed responses -#' to calibrate the level against, or NULL to skip. -#' @param slope_terms Character vector of terms that are slopes -#' rather than habitat types. -#' @return A selection result whose `coefficients` are -#' probabilities of presence per habitat type. -#' -#' @example # Example usage of the function -#' # select_aic_best_onehot(models, response ~ 1, d, engine, -#' # "binomial", intercept_cats = cats) -select_aic_best_onehot <- function( - models, base, data, engine, family, - weights = NULL, offset = NULL, ic = "AICc", control = list(), - intercept_cats = NULL, - constants = list(seas_days = 100, Climate = 0), - calibrate_against = NULL, - slope_terms = "Climate" -) { - fitted <- fit_candidates( - models, base, data, engine, family, weights, offset, ic, - control - ) - - table <- candidate_table(fitted) - - if (!any(table$ok)) { - return(selection_result(NULL, NULL, table, fitted)) - } - - winner <- which.min(fitted$scores) - best <- fitted$fits[[winner]] - - # Step 1: The terms to read are the winning model's own, plus - # the category it leaves out. - terms <- attr(stats::terms(best$fit), "term.labels") - reference <- if (is.null(intercept_cats)) { - NULL - } else { - intercept_cats[winner] - } - - # A slope stays in the reported set even though it is held at - # a constant for the one-hot predictions: its value is taken - # from the fitted coefficient below, and dropping it here would - # make that an append rather than a replacement. Pure design - # constants - sampling effort - are not reported at all. - terms <- setdiff( - c(terms, reference), - setdiff(names(constants), slope_terms) - ) - - if (length(terms) == 0) { - return(selection_result( - best, fit_coefficients(best), table, fitted - )) - } - - # Step 2: One prediction per term, at 100 per cent of that type - onehot <- as.data.frame(diag(length(terms))) - names(onehot) <- terms - - for (one in names(constants)) { - onehot[[one]] <- constants[[one]] - } - - predicted <- engine$predict(best, onehot, "link", se = TRUE) - - if (is.null(predicted)) { - return(selection_result(NULL, NULL, table, fitted)) - } - - estimate <- stats::plogis(predicted$fit) - error <- predicted$se.fit - names(estimate) <- names(error) <- terms - - # Step 3: Shift the level so mean fitted presence matches mean - # observed. Slopes are exempt: the shift is an intercept - # change, and applying it to a slope would be meaningless. - if (!is.null(calibrate_against)) { - on_scale <- engine$predict(best, data, "link") - - if (!is.null(on_scale)) { - shift <- stats::qlogis(mean(calibrate_against)) - - stats::qlogis(mean(stats::plogis(on_scale))) + estimate <- do.call(rbind, lapply(predictions, `[[`, "fit")) + error <- do.call(rbind, lapply(predictions, `[[`, "se.fit")) + link <- colSums(estimate / error^2) / colSums(1 / error^2) + } else { + predictions <- lapply(fits, function(f) { + engine$predict(f, newdata, "link") + }) - if (is.finite(shift)) { - habitat <- setdiff(names(estimate), slope_terms) - estimate[habitat] <- stats::plogis( - stats::qlogis(estimate[habitat]) + shift - ) + if (any(vapply(predictions, is.null, logical(1)))) { + return(NULL) } - } - } - # Step 4: A slope is reported from its own coefficient, not - # from a one-hot prediction of a type that does not exist. - coefficients <- stats::coef(best$fit) - - for (one in intersect(slope_terms, names(coefficients))) { - if (!one %in% names(estimate)) { - next + link <- drop(weights %*% do.call(rbind, predictions)) } - estimate[one] <- stats::plogis(coefficients[[one]]) + if (type == "link") link else linkinv(link) } - - result <- selection_result( - fit = best, - coefficients = data.frame( - term = names(estimate), - estimate = unname(estimate), - se = unname(error), - stringsAsFactors = FALSE - ), - ic_table = table, - fitted = fitted - ) - - result$reference_category <- reference - - result } # 5. selection_result() ---- @@ -1035,6 +338,9 @@ select_aic_best_onehot <- function( #' @param fitted The full candidate set, or NULL. #' @param formula The chosen formula, or NULL to read it from #' `fit`. +#' @param engine The engine, or NULL. Used to count observations. +#' @param predict A function(newdata, type) for the final model, +#' or NULL to predict from `fit` (set by selection_run()). #' @return A list. #' #' @example # Example usage of the function @@ -1044,7 +350,9 @@ selection_result <- function( coefficients, ic_table, fitted = NULL, - formula = NULL + formula = NULL, + engine = NULL, + predict = NULL ) { list( fit = fit, @@ -1053,13 +361,20 @@ selection_result <- function( formula = if (!is.null(formula)) { paste(deparse(formula), collapse = " ") } else if (!is.null(fit) && isTRUE(fit$ok)) { - paste(deparse(stats::formula(fit$fit)), collapse = " ") + tryCatch( + paste(deparse(stats::formula(fit$fit)), collapse = " "), + error = function(e) NA_character_ + ) } else { NA_character_ }, + # Every formula any candidate fitted, so a grid prediction can + # supply each term a combined predictor reads + formulas = if (is.null(fitted)) NULL else fitted$formulas, n_fitted = if (is.null(ic_table)) 0L else sum(ic_table$ok), n_failed = if (is.null(ic_table)) 0L else sum(!ic_table$ok), - nobs = fit_nobs(fit) + nobs = engine_nobs(engine, fit), + predict = predict ) } diff --git a/1_code/harness/spec.R b/1_code/harness/spec.R new file mode 100644 index 0000000..9f4ba9c --- /dev/null +++ b/1_code/harness/spec.R @@ -0,0 +1,639 @@ +# --- +# title: Taxon Specs - Validation and Helpers +# author: Brendan Casey +# created: 2026-10-03 +# inputs: none +# outputs: none; defines functions in memory +# notes: +# - A spec states one taxon's configuration as data; methods it +# names are looked up in the registry. Fields are documented +# in docs/getting_started.md. +# - validate_spec() reports every problem at once, before any +# data is read. Unknown fields are errors, because a misspelt +# field is otherwise silently ignored; arguments of the named +# rule or resampler count as known fields. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. Known fields ---- + +## 2.1 spec_fields() ---- + +#' The Fields a Spec, Region or Stage May Carry +#' +#' Fields the harness itself reads; method-specific settings are +#' known through the method's arguments. +#' +#' @return A named list of character vectors: `spec`, `region`, +#' `stage` and `resample`. +#' +#' @example # Example usage of the function +#' # spec_fields()$stage +spec_fields <- function() { + list( + spec = c( + "taxon", "response_name", "response_transform", "family", + "weight_column", "tier", "season", "aliases", + "regions", "stages", "resample", "v2_coverage", "validate", + "final_prediction", "species_frame", "supplied_columns", + "notes", "models_overridden", "covariate_files", + # The stage a methods experiment replaces by default + "habitat_stage", + # Recorded facts a module keeps for its own reports + "use_protocol", "protocol_is_v2", "climate_source" + ), + region = c( + "filter", "grid", "term_block", "habitat_models", + "grid_constants", "grid_drop_rows", "grid_drop_cols", + "constants", "intercept_cats", "extra_covariates", + "weight_column" + ), + stage = c( + "name", "models", "engine", "selection", "ic", "control", + "family", "scope", "weighted", "base_terms", + "response_transform", "row_filter", "min_detections", + "extra_covariates", "post_process", "calibrate", + "carry_as", "carry_scale", "carry_offset", "carry_from", + "carry_from_as", + # Values held fixed when the final model is projected onto + # the grid; rules that predict onto a grid read them too + "constants", "grid_constants" + ), + resample = c("scheme", "seed", "scope") + ) +} + +# 3. validate_spec() ---- + +#' Check a Spec Before Running It +#' +#' @param spec A taxon spec. +#' @param stop_on_error Logical. Stop with every problem listed, +#' or return the problems. +#' @return The spec, invisibly, when it is valid; otherwise stops, +#' or returns a character vector of problems when +#' `stop_on_error` is FALSE. +#' +#' @example # Example usage of the function +#' # validate_spec(lichen_spec()) +validate_spec <- function(spec, stop_on_error = TRUE) { + problems <- character(0) + problem <- function(...) { + problems <<- c(problems, paste0(...)) + } + fields <- spec_fields() + unknown_fields <- function(x, known, where) { + extra <- setdiff(names(x), known) + + if (length(extra) > 0) { + problem( + where, " has unknown field(s): ", + paste(extra, collapse = ", "), "." + ) + } + } + + # Step 1: The spec itself + if (!is.character(spec$taxon) || length(spec$taxon) != 1L) { + problem("`taxon` must be one data slug, such as \"lichen\".") + } + + label <- paste0("Spec `", spec$taxon %||% "?", "`") + unknown_fields(spec, fields$spec, label) + + # Step 2: Regions + if (!is.list(spec$regions) || length(spec$regions) == 0) { + problem(label, " defines no regions.") + } + + for (region in names(spec$regions)) { + one <- spec$regions[[region]] + unknown_fields( + one, fields$region, paste0("Region `", region, "`") + ) + + if (!is.null(one$filter) && !inherits(one$filter, "formula")) { + problem( + "Region `", region, "`: `filter` must be a one-sided ", + "formula, such as ~ nr != \"Grassland\"." + ) + } + } + + # Step 3: Stages, in order + stage_names <- character(0) + carried <- character(0) + + for (stage in spec$stages) { + name <- stage$name %||% "?" + where <- paste0("Stage `", name, "`") + + if (name %in% stage_names) { + problem(where, " appears twice; stage names must differ.") + } + stage_names <- c(stage_names, name) + + engine <- tryCatch( + get_engine(stage$engine), + error = function(e) { + problem(where, ": ", conditionMessage(e)) + NULL + } + ) + + rule <- if (is.function(stage$selection)) { + stage$selection + } else { + tryCatch( + get_method("selection", stage$selection), + error = function(e) { + problem(where, ": ", conditionMessage(e)) + NULL + } + ) + } + + # A field the rule takes as an argument is a rule setting + known <- c(fields$stage, if (!is.null(rule)) names(formals(rule))) + unknown_fields(stage, known, where) + + # The rule's requirements against the engine's capabilities + if (!is.null(engine) && is.character(stage$selection)) { + needs <- get_method( + "selection", stage$selection, entry = TRUE + )$requires + missing_capabilities <- setdiff(needs, engine$capabilities) + + if (length(missing_capabilities) > 0) { + problem( + where, ": selection rule `", stage$selection, + "` needs ", paste(missing_capabilities, collapse = ", "), + ", which engine `", engine$name, "` does not provide. ", + "Choose a rule that does not, such as `single`." + ) + } + } + + # A composite rule (the hurdle) runs an inner rule per part; + # that rule's requirements are checked against the engine too + if (!is.null(engine) && !is.null(rule) && + "part_selection" %in% names(formals(rule))) { + inner <- stage$part_selection %||% formals(rule)$part_selection + entry <- tryCatch( + get_method("selection", inner, entry = TRUE), + error = function(e) { + problem(where, ": `part_selection`: ", conditionMessage(e)) + NULL + } + ) + missing_capabilities <- setdiff( + entry$requires, engine$capabilities + ) + + if (!is.null(entry) && length(missing_capabilities) > 0) { + problem( + where, ": inner rule `", inner, "` needs ", + paste(missing_capabilities, collapse = ", "), + ", which engine `", engine$name, "` does not provide. ", + "Use `single` as the part_selection." + ) + } + } + + if (!is.null(stage$ic) && + !stage$ic %in% c("AIC", "AICc", "BIC")) { + problem(where, ": `ic` must be \"AIC\", \"AICc\" or \"BIC\".") + } + + # Or the carried column is attached per species via + # `supplied_columns` (mammal precomputed climate) + carried_column <- stage$carry_from_as %||% "Climate" + + if (!is.null(stage$carry_from) && + !stage$carry_from %in% carried && + !carried_column %in% spec$supplied_columns) { + problem( + where, " carries from `", stage$carry_from, "`, which is ", + "not an earlier stage with `carry_as`, and `", + carried_column, "` is not in `supplied_columns`." + ) + } + + if (!is.null(stage$carry_as)) { + carried <- c(carried, name) + } + + # Models: a registered set name, formulas, or NULL to take + # each region's habitat models + if (is.null(stage$models)) { + lacking <- names(spec$regions)[vapply( + spec$regions, function(r) is.null(r$habitat_models), + logical(1) + )] + + if (length(lacking) > 0) { + problem( + where, " takes its models from the region, but ", + paste(lacking, collapse = ", "), " define(s) no ", + "`habitat_models`." + ) + } + } else { + problems <- c(problems, check_model_set(stage$models, where)) + } + } + + # Step 4: Resampling + scheme <- spec$resample$scheme + + resampler <- tryCatch( + get_method("resampler", scheme %||% ""), + error = function(e) { + problem("`resample$scheme`: ", conditionMessage(e)) + NULL + } + ) + + if (!is.null(resampler)) { + unknown_fields( + spec$resample, + c(fields$resample, names(formals(resampler))), + "`resample`" + ) + } + + if (length(problems) == 0) { + return(invisible(spec)) + } + + if (!stop_on_error) { + return(problems) + } + + stop( + label, " has ", length(problems), " problem(s):\n - ", + paste(problems, collapse = "\n - "), + call. = FALSE + ) +} + +## 3.1 check_model_set() ---- + +#' Check a Stage's Models Name Something Real +#' +#' A lone string that is neither a set nor a formula is almost +#' always a misspelt set name. +#' +#' @param models A set name, or formulas as text. +#' @param where Character. Prefix for the message. +#' @return A character vector of problems, empty when fine. +#' +#' @example # Example usage of the function +#' # check_model_set("climate_plant_v2_ful", "Stage `climate`") +check_model_set <- function(models, where) { + if (is.list(models)) { + return(unlist(lapply(models, check_model_set, where = where))) + } + + if (!is.character(models)) { + return(paste0(where, ": `models` must be a set name or text.")) + } + + if (length(models) == 1L && !grepl("~", models, fixed = TRUE) && + !models %in% names(model_sets())) { + return(paste0( + where, ": `", models, "` is not a model set or a formula. ", + "Sets: see names(model_sets())." + )) + } + + character(0) +} + +# 4. spec_covariates() ---- + +#' Read the Covariates a Spec's Model Sets Name +#' +#' Read from the formulas, so a spec cannot name a term it forgot +#' to load. Excludes harness-supplied terms (response, offset, +#' weight, carried stages, `supplied_columns`). +#' +#' @param spec A taxon spec. +#' @param stage_names Character vector of stages to read, or NULL +#' for all of them. +#' @return A character vector of covariate names. +#' +#' @example # Example usage of the function +#' # spec_covariates(bryophyte_spec()) +spec_covariates <- function(spec, stage_names = NULL) { + stages <- spec$stages + + if (!is.null(stage_names)) { + stages <- stages[vapply( + stages, function(s) s$name %in% stage_names, logical(1) + )] + } + + supplied <- c( + spec$response_name %||% "response", + "offset", "weight", + vapply(spec$stages, function(s) s$carry_as %||% "", character(1)), + spec$supplied_columns + ) + + terms <- unlist(lapply(stages, function(stage) { + models <- unlist(get_model_set(stage$models), use.names = FALSE) + + unlist(lapply(models, function(one) { + all.vars(stats::as.formula(one)) + })) + })) + + covariates <- setdiff(unique(terms), c(supplied, ".")) + + # Columns no formula names, e.g. the plant age splines' aged + # stand-cover columns + extra <- unlist(lapply(stages, `[[`, "extra_covariates")) + + unique(c(covariates, extra, spec$weight_column)) +} + +# 5. apply_stage_models() ---- + +#' Swap a Spec's Candidate Sets for an Experiment's Own +#' +#' Keeps the spec as the record of v2 and the experiment as the +#' record of what changed. Entries are named `taxon.stage`, or +#' `stage` for every taxon; values are a model_sets() name or +#' formulas. +#' +#' \preformatted{ +#' stage_models <- list( +#' "lichen.climate" = extend_models( +#' "climate_plant_v2_full", "elevation" +#' ), +#' "climate" = models_from_covariates(c("MAP", "FFP")) +#' ) +#' } +#' +#' @param spec A taxon spec. +#' @param stage_models Named list of replacement model sets, or +#' NULL. +#' @return The spec, with matching stages replaced, and a record +#' of what changed in `models_overridden`. +#' +#' @example # Example usage of the function +#' # apply_stage_models(spec, list(climate = my_models)) +apply_stage_models <- function(spec, stage_models) { + if (is.null(stage_models) || length(stage_models) == 0) { + return(spec) + } + + changed <- character(0) + + for (i in seq_along(spec$stages)) { + stage <- spec$stages[[i]] + + # `taxon.stage` wins over a bare `stage` + keys <- c(paste(spec$taxon, stage$name, sep = "."), stage$name) + hit <- keys[keys %in% names(stage_models)][1] + + if (is.na(hit)) { + next + } + + spec$stages[[i]]$models <- stage_models[[hit]] + changed <- c(changed, paste0(stage$name, " <- ", hit)) + } + + # Recorded in meta.json; otherwise indistinguishable from v2 + spec$models_overridden <- if (length(changed) > 0) { + changed + } else { + NULL + } + + spec +} + +# 6. replace_stage_method() ---- + +#' Fit One Stage of a v2 Spec with a Different Method +#' +#' The rest of the spec stays v2's, so a difference in results is +#' the method's. By default the candidates become one formula of +#' all their covariates (models_union()). A composite rule (e.g. +#' the mammal `hurdle`) keeps its structure and runs the new rule +#' in each part. +#' +#' Removes what reads v2 coefficient tables a new engine may not +#' produce: the stage's `post_process`, fields only the old rule +#' took, and (for the last stage) `final_prediction` and +#' `validate`. The change is recorded in `v2_coverage` and +#' `models_overridden`. +#' +#' @param spec A taxon spec. +#' @param stage Character. The stage's name; the spec's +#' `habitat_stage` by default. +#' @param engine Character. A registered engine. +#' @param selection Character. A registered rule; "single" for an +#' engine with no information criterion. +#' @param control Named list of engine settings. +#' @param union Logical. Replace the candidate set with one formula +#' holding all its covariates. FALSE keeps the candidates, for a +#' rule that compares them. +#' @return The spec. +#' +#' @example # Example usage of the function +#' # spec <- replace_stage_method(lichen_spec(), engine = "xgboost") +replace_stage_method <- function( + spec, stage = spec$habitat_stage, engine, selection = "single", + control = list(), union = TRUE +) { + names_in <- vapply(spec$stages, `[[`, character(1), "name") + + if (is.null(stage)) { + stop( + "Spec `", spec$taxon, "` names no `habitat_stage`; pass ", + "`stage`. Stages: ", paste(names_in, collapse = ", "), + call. = FALSE + ) + } + + index <- match(stage, names_in) + + if (is.na(index)) { + stop( + "Spec `", spec$taxon, "` has no stage `", stage, "`. Stages: ", + paste(names_in, collapse = ", "), + call. = FALSE + ) + } + + get_method("selection", selection) + get_engine(engine) + old <- spec$stages[[index]] + + # A composite rule keeps its place and takes the new rule inside + old_rule <- if (is.function(old$selection)) { + old$selection + } else { + get_method("selection", old$selection) + } + composite <- "part_selection" %in% names(formals(old_rule)) + rule <- if (composite) { + old_rule + } else { + get_method("selection", selection) + } + + # Step 1: The candidate formulas, per region where the region + # supplies them + if (union) { + if (is.null(old$models)) { + for (region in names(spec$regions)) { + spec$regions[[region]]$habitat_models <- models_union( + spec$regions[[region]]$habitat_models + ) + spec$regions[[region]]$intercept_cats <- NULL + } + } else { + old$models <- models_union(old$models) + } + } + + # Step 2: The stage, keeping only what the harness and the new + # rule read + known <- c(spec_fields()$stage, names(formals(rule))) + new <- old[intersect(names(old), known)] + new$engine <- engine + new$control <- control + new$ic <- NULL + new$post_process <- NULL + + if (composite) { + new$part_selection <- selection + } else { + new$selection <- selection + } + + spec$stages[[index]] <- new + + # Step 3: Hooks that read v2 coefficient tables + if (index == length(spec$stages)) { + spec$final_prediction <- NULL + spec$validate <- NULL + } + + # Step 4: Say so + label <- paste0( + engine, " with `", selection, "`", + if (composite) paste0(" inside `", old$selection, "`") else "", + if (union) ", on every covariate the v2 candidates use" else "" + ) + spec$v2_coverage[[stage]] <- v2_status( + "not reproduced", paste0("Replaced: ", label, ".") + ) + spec$models_overridden <- c( + spec$models_overridden, paste0(stage, " <- ", label) + ) + spec$notes <- paste0( + spec$notes %||% "", " Stage `", stage, "` fitted with ", + label, "." + ) + + spec +} + +# 7. v2 coverage ---- + +## 7.1 v2_status() ---- + +#' State How Much of One v2 Stage a Spec Reproduces +#' +#' The status vocabulary is fixed, so a typo fails when the spec +#' is built. +#' +#' @param status Character. "reproduced", "partial" or +#' "not reproduced". +#' @param note Character. What is and is not reproduced, and why. +#' @return A list of `status` and `note`. +#' +#' @example # Example usage of the function +#' # v2_status("partial", "No age splines.") +v2_status <- function(status, note) { + allowed <- c("reproduced", "partial", "not reproduced") + + if (!is.character(status) || length(status) != 1L || + !status %in% allowed) { + stop( + "v2 status must be one of: ", + paste(allowed, collapse = ", "), ".", + call. = FALSE + ) + } + + if (!is.character(note) || length(note) != 1L || + !nzchar(note)) { + stop("A v2 status needs a one-line note.", call. = FALSE) + } + + list(status = status, note = note) +} + +## 7.2 spec_coverage() ---- + +#' Tabulate What a Set of Specs Reproduces of v2 +#' +#' +#' @param specs A named list of specs. +#' @return A data frame of taxon, stage, status and note. +#' +#' @example # Example usage of the function +#' # spec_coverage(list(lichen = lichen_spec())) +spec_coverage <- function(specs) { + rows <- lapply(names(specs), function(key) { + coverage <- specs[[key]]$v2_coverage + + if (is.null(coverage) || length(coverage) == 0) { + return(data.frame( + taxon = key, stage = NA_character_, + status = "not stated", + note = "The spec has no v2_coverage.", + stringsAsFactors = FALSE + )) + } + + data.frame( + taxon = key, + stage = names(coverage), + status = vapply(coverage, `[[`, character(1), "status"), + note = vapply(coverage, `[[`, character(1), "note"), + stringsAsFactors = FALSE, + row.names = NULL + ) + }) + + do.call(rbind, rows) +} + +# 8. Helpers ---- + +## 8.1 `%||%` ---- + +#' Default for NULL +#' +#' @param a,b Any values. +#' @return `a` unless it is NULL, in which case `b`. +#' +#' @example # Example usage of the function +#' # NULL %||% "default" +`%||%` <- function(a, b) { + if (is.null(a)) b else a +} + +# End of script ---- diff --git a/1_code/harness/species.R b/1_code/harness/species.R index 8f25f6d..3d8c898 100644 --- a/1_code/harness/species.R +++ b/1_code/harness/species.R @@ -3,27 +3,15 @@ # author: Brendan Casey # created: 2026-09-09 # inputs: -# in 0_data/test_dataset/lookup/: -# - modelled_species.csv (plant groups) -# - mammal_modelled_species.csv -# - bird_modelled_species.csv +# in the test dataset's lookup/ (see data_source.R): +# - species_queue.csv, from _setup/09_harmonize_lookups.R # outputs: none; returns objects in memory # notes: -# - The three source lookups have three different schemas, one -# per taxon lead. This flattens them to one, so an experiment -# asks for species the same way whatever the taxon. -# - The unified schema is taxon, species, region, tier, order -# and season. `tier` separates a taxon's model classes where -# it has them - mammals fit a full habitat model for species -# with at least 20 detections and a use-availability model -# for those with at least 3 - and is "modelled" elsewhere. -# `season` is NA for everything but mammals. -# - `order` preserves each source list's own ordering, so a -# narrowed run visits species in the order v2 would have. -# - resolve_species() is the entry point an experiment uses to -# pick focal species. Passing NULL takes every modelled -# species; passing a vector takes those, and an unknown name -# stops the run before any fitting rather than after. +# - One queue for every taxon, flattened from the taxon leads' +# lookups by _setup/09. `tier` separates model classes where a +# taxon has them (mammals) and is "modelled" elsewhere; +# `season` is empty except for mammals; `order` keeps v2's +# species order. # --- # 1. Setup ---- @@ -31,132 +19,37 @@ ## 1.1 Load packages ---- library(data.table) # lookup table reading (version: 1.16.4) -# 2. Source lookup declarations ---- +# 2. species_catalogue() ---- -## 2.1 species_lookup_files() ---- - -#' Name Each Taxon's Species Lookup -#' -#' @return A named character vector of file names within -#' 0_data/test_dataset/lookup/. -#' -#' @example # Example usage of the function -#' # species_lookup_files()[["mammal"]] -species_lookup_files <- function() { - c( - plants = "modelled_species.csv", - mammal = "mammal_modelled_species.csv", - bird = "bird_modelled_species.csv" - ) -} - -## 2.2 plant_taxa() ---- - -#' Name the Taxa Held in the Plant-Group Lookup -#' -#' "Plant group" names a file layout rather than a taxonomy: the -#' four share one survey design and one lookup, soil mites -#' included. +#' Read the Species Queue #' -#' @return A character vector of taxon slugs. -#' -#' @example # Example usage of the function -#' # plant_taxa() -plant_taxa <- function() { - c("vascular_plant", "bryophyte", "lichen", "mite") -} - -# 3. species_catalogue() ---- - -#' Flatten Every Species Lookup into One Table -#' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Keep one taxon, or NULL for all. -#' @return A data frame of taxon, species, region, tier, order -#' and season. +#' @return A data frame of taxon, region, season, tier, species, +#' order and source_name. #' #' @example # Example usage of the function #' # species_catalogue(data_dir, taxon = "bryophyte") species_catalogue <- function(data_dir, taxon = NULL) { - lookup_dir <- file.path(data_dir, "lookup") - files <- species_lookup_files() - - read_lookup <- function(name) { - path <- file.path(lookup_dir, files[[name]]) + path <- file.path(data_dir, "lookup", "species_queue.csv") - if (!file.exists(path)) { - stop( - "Species lookup not found:\n ", path, - "\nRun 1_code/_setup/01_harmonize_model_ready_v2.R first.", - call. = FALSE - ) - } - - as.data.frame(fread(path)) - } - - parts <- list() - - # Step 1: Plant groups. One row per species, with membership - # of the north (veg) and south (soil) model sets as flags, so - # each flag becomes a region row. - plants <- read_lookup("plants") - - parts$plants <- rbind( - unified_rows( - plants[plants$in_veg_models, ], - taxon = plants$taxon[plants$in_veg_models], - region = "north", - order = plants$veg_order[plants$in_veg_models] - ), - unified_rows( - plants[plants$in_soil_models, ], - taxon = plants$taxon[plants$in_soil_models], - region = "south", - order = plants$soil_order[plants$in_soil_models] - ) - ) - - # Step 2: Mammals. Region and season are already columns, and - # the two occurrence thresholds become tiers. - mammals <- read_lookup("mammal") - - parts$mammal <- rbind( - unified_rows( - mammals[mammals$in_models, ], - taxon = "mammal", - region = mammals$region[mammals$in_models], - order = mammals$model_order[mammals$in_models], - tier = "modelled", - season = mammals$season[mammals$in_models] - ), - unified_rows( - mammals[mammals$in_ua_models, ], - taxon = "mammal", - region = mammals$region[mammals$in_ua_models], - order = mammals$ua_order[mammals$in_ua_models], - tier = "ua", - season = mammals$season[mammals$in_ua_models] - ) - ) - - # Step 3: Birds. One row per species and region already. - birds <- read_lookup("bird") - - parts$bird <- unified_rows( - birds, - taxon = "bird", - region = birds$region, - order = stats::ave( - seq_len(nrow(birds)), birds$region, FUN = seq_along + if (!file.exists(path)) { + stop( + "Species queue not found:\n ", path, + "\nRun 1_code/_setup/09_harmonize_lookups.R first.", + call. = FALSE ) - ) + } - catalogue <- do.call(rbind, parts) - rownames(catalogue) <- NULL + catalogue <- cached_read(path, function(p) { + queue <- as.data.frame(fread(p, na.strings = "")) + queue$season <- as.character(queue$season) + queue + }) if (!is.null(taxon)) { known <- unique(catalogue$taxon) + if (!taxon %in% known) { stop( "Unknown taxon `", taxon, "`. Taxa in the catalogue: ", @@ -172,20 +65,17 @@ species_catalogue <- function(data_dir, taxon = NULL) { catalogue } -# 4. list_species() ---- +# 3. list_species() ---- #' List the Species Available for a Taxon #' -#' What an experiment reads to decide which focal species to run. -#' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Taxon slug. #' @param region Character. Restrict to one region, or NULL. #' @param tier Character. Restrict to one model tier, or NULL. #' @param season Character. Restrict to one season, or NULL. -#' Only mammals have seasons, and a mammal spec is fitted for -#' one; without this, a summer spec would also queue the -#' `_Winter` species and fit them with summer weights. +#' Mammal specs must pass it, or a summer spec would also queue +#' the `_Winter` species and fit them with summer weights. #' @return A character vector of species names, in source order. #' #' @example # Example usage of the function @@ -219,20 +109,43 @@ list_species <- function( unique(catalogue$species) } +# 4. species_source_name() ---- + +#' The Name a Source Pipeline Used for a Species +#' +#' v2's files sometimes name a species differently from the +#' dataset's column: mammals are "Black BearSummer" there and +#' "BlackBear_Summer" here. +#' +#' @param data_dir Character. The test dataset folder. +#' @param taxon,species Character. +#' @return Character, the source name. +#' +#' @example # Example usage of the function +#' # species_source_name(data_dir, "mammal", "BlackBear_Summer") +species_source_name <- function(data_dir, taxon, species) { + catalogue <- species_catalogue(data_dir, taxon) + name <- unique(catalogue$source_name[catalogue$species == species]) + + if (length(name) == 0) { + stop( + "`", species, "` is not in the ", taxon, " species queue.", + call. = FALSE + ) + } + + name[1] +} + # 5. resolve_species() ---- #' Choose the Focal Species for a Run #' -#' NULL means every species the source lookup declares as -#' modelled. A character vector means those species, and an -#' unknown name stops the run here rather than failing inside a -#' bootstrap hours later. -#' -#' Selection is how a trial run is shortened. Three species is a -#' smoke test; the full list is the parity run. +#' NULL means every modelled species. An unknown name stops the +#' run here rather than hours later inside a bootstrap. #' #' @param species Character vector of species, or NULL for all. -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param taxon Character. Taxon slug. #' @param region Character. Restrict to one region, or NULL. #' @param tier Character. Restrict to one model tier, or NULL. @@ -274,8 +187,7 @@ resolve_species <- function( ) } - # Intersecting rather than reordering keeps the source order, - # so a narrowed run visits species in the sequence v2 would. + # Keeps source (v2) order rather than the caller's intersect(available, species) } @@ -283,10 +195,9 @@ resolve_species <- function( #' Read a Named Vector's Entries for One Taxon #' -#' The convention an experiment uses to select species and -#' covariates: a named character vector whose names are taxa. -#' Names may repeat, so several entries can apply to the same -#' taxon, and an unnamed entry applies to every taxon. +#' Experiments select species and covariates with a character +#' vector named by taxon. Names may repeat; unnamed entries apply +#' to every taxon. #' #' \preformatted{ #' species <- c( @@ -294,23 +205,12 @@ resolve_species <- function( #' lichen = "Bryoria.fremontii", #' mite = "Oppiella.nova" #' ) -#' -#' covariates <- c( -#' "climate_common", # every taxon -#' lichen = "topography" # lichens as well -#' ) #' } #' -#' One convention for both keeps the two selections symmetrical, -#' and lets a single vector configure a run across taxa without a -#' nested list. -#' #' @param x A named character vector, or NULL. #' @param taxon Character. The taxon to read. #' @return A character vector of the entries that apply, or NULL -#' when none do. NULL means "no selection", which callers read -#' as "everything", so an empty result is never an empty -#' selection. +#' when none do. Callers read NULL as "everything". #' #' @example # Example usage of the function #' # taxon_values(c(lichen = "a", mite = "b"), "lichen") @@ -337,49 +237,4 @@ taxon_values <- function(x, taxon) { out } -# 7. Helpers ---- - -## 6.1 unified_rows() ---- - -#' Build Rows in the Unified Schema -#' -#' @param frame A data frame with a `species` column. -#' @param taxon,region,order,tier,season Values recycled to the -#' number of rows in `frame`. -#' @return A data frame in the unified schema, or an empty one -#' when `frame` has no rows. -#' -#' @example # Example usage of the function -#' # unified_rows(birds, "bird", birds$region, seq_len(nrow(birds))) -unified_rows <- function( - frame, - taxon, - region, - order, - tier = "modelled", - season = NA_character_ -) { - if (nrow(frame) == 0) { - return(data.frame( - taxon = character(0), - species = character(0), - region = character(0), - tier = character(0), - order = integer(0), - season = character(0), - stringsAsFactors = FALSE - )) - } - - data.frame( - taxon = as.character(taxon), - species = as.character(frame$species), - region = as.character(region), - tier = as.character(tier), - order = as.integer(order), - season = as.character(season), - stringsAsFactors = FALSE - ) -} - # End of script ---- diff --git a/1_code/harness/summarise.R b/1_code/harness/summarise.R new file mode 100644 index 0000000..f87882e --- /dev/null +++ b/1_code/harness/summarise.R @@ -0,0 +1,358 @@ +# --- +# title: Summarize Result Stores +# author: Brendan Casey +# created: 2026-10-03 +# inputs: +# in pipeline_dir///: +# - coefficients.csv, metrics.csv, grid_predictions.csv, +# meta.json +# outputs: +# collect_results() returns the summaries; write_summaries() +# writes them to out_dir/tables/: +# - coverage.csv, metric_summary.csv, grid_summary.csv +# (small; committed) +# - coefficient_summary.csv (large; gitignored) +# notes: +# - Reduces per-draw result stores to the per-species summaries +# experiments compare. `boot1` is the full-data fit, the +# like-for-like value where a reference fits once (v2 +# mammals). +# - `run` is the store's folder; taxon, season and part come +# from meta.json, so both mammal season runs share "mammal". +# - Taxa whose coefficients need translating (birds, onto v2's +# habitat template) pass a function in `translate`, keeping +# the harness taxon-agnostic. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # grouped summaries (version: 1.16.4) + +# 2. Finding stores ---- + +## 2.1 find_stores() ---- + +#' Find the Result Stores Under a Pipeline Directory +#' +#' A store is a directory holding a meta.json, laid out as +#' `///`. +#' +#' @param pipeline_dir Character. +#' @return A data frame of dir, run, region, taxon, season, part +#' and final_stage, ordered by run and region. +#' +#' @example # Example usage of the function +#' # find_stores("2_pipeline/exp_000_parity_v2") +find_stores <- function(pipeline_dir) { + dirs <- list.dirs(pipeline_dir, recursive = TRUE) + dirs <- dirs[file.exists(file.path(dirs, "meta.json"))] + + if (length(dirs) == 0) { + return(data.frame( + dir = character(0), run = character(0), region = character(0), + taxon = character(0), season = character(0), + part = character(0), final_stage = character(0), + stringsAsFactors = FALSE + )) + } + + rows <- lapply(dirs, function(dir) { + meta <- store_meta(dir) + + # Only the run comes from the path; the rest from meta.json + data.frame( + dir = dir, + run = basename(dirname(dir)), + region = if (is.na(meta$region)) basename(dir) else meta$region, + taxon = if (is.na(meta$taxon)) { + basename(dirname(dir)) + } else { + meta$taxon + }, + season = meta$season, + part = meta$part, + final_stage = meta$final_stage, + stringsAsFactors = FALSE + ) + }) + + stores <- do.call(rbind, rows) + stores[order(stores$run, stores$region), , drop = FALSE] +} + +## 2.2 store_meta() ---- + +#' Read a Store's meta.json, Tolerating a Missing Field +#' +#' @param dir Character. A result store directory. +#' @return A list with taxon, region, season, part and +#' final_stage; NA for what the file does not say. +#' +#' @example # Example usage of the function +#' # store_meta("2_pipeline/exp_000_parity_v2/lichen/north") +store_meta <- function(dir) { + if (!requireNamespace("jsonlite", quietly = TRUE)) { + stop("Reading meta.json needs the jsonlite package.", + call. = FALSE) + } + + meta <- jsonlite::fromJSON(file.path(dir, "meta.json")) + + field <- function(name) { + value <- meta[[name]] + + if (is.null(value) || length(value) == 0 || + identical(value, "NA") || all(is.na(value))) { + NA_character_ + } else { + as.character(value[1]) + } + } + + list( + taxon = field("taxon"), + region = field("region"), + season = field("season"), + part = field("part"), + final_stage = if (is.null(meta$stages)) { + NA_character_ + } else { + utils::tail(meta$stages, 1) + } + ) +} + +# 3. Summarizing draws ---- + +## 3.1 summarise_draws() ---- + +#' Summarize a Value over Bootstrap Draws, per Group +#' +#' @param rows A data frame with `boot` and the value column. +#' @param value Character. The column to summarize. +#' @param keys Character vector of grouping columns. +#' @return A data.table of the keys plus n, median, p10, p90, +#' boot1, mean and sd. n counts the finite values; boot1 is the +#' iteration-1 value where there is exactly one. +#' +#' @example # Example usage of the function +#' # summarise_draws(metrics, "value", c("species", "metric")) +summarise_draws <- function(rows, value, keys) { + rows <- as.data.table(rows) + + # The column is passed through .SDcols rather than named inside + # the call, because a column may itself be called `value` + rows[, draw_summary(.SD[[1L]], boot), + by = keys, .SDcols = value] +} + +## 3.2 draw_summary() ---- + +#' Summarize One Value over Its Draws +#' +#' Comparisons read the median and 10-90% band because draw +#' distributions are often skewed; mean and sd are added for +#' tools that expect them. +#' +#' @param x Numeric vector, one value per draw. +#' @param boot Integer vector of the draw each value came from. +#' @return A list of n, median, p10, p90, boot1, mean and sd. sd +#' is NA with fewer than two values. +#' +#' @example # Example usage of the function +#' # draw_summary(c(0.1, 0.3, 0.2), 1:3) +draw_summary <- function(x, boot) { + first <- x[boot == 1L] + first <- if (length(first) == 1 && is.finite(first)) { + first + } else { + NA_real_ + } + finite <- x[is.finite(x)] + + out <- if (length(finite) == 0) { + list(n = 0, median = NA_real_, p10 = NA_real_, + p90 = NA_real_, boot1 = first) + } else { + quantiles <- stats::quantile( + finite, c(0.1, 0.5, 0.9), names = FALSE + ) + + list(n = as.numeric(length(finite)), median = quantiles[2], + p10 = quantiles[1], p90 = quantiles[3], boot1 = first) + } + + # Appended, so the columns before them keep their positions + out$mean <- if (length(finite) == 0) NA_real_ else mean(finite) + out$sd <- if (length(finite) < 2) NA_real_ else stats::sd(finite) + + out +} + +# 4. collect_results() ---- + +#' Reduce Every Store in a Pipeline Directory to Summaries +#' +#' @param pipeline_dir Character. Where the stores are. +#' @param data_dir Character. Passed to `translate` functions. +#' @param translate Named list of functions, by taxon. Each takes +#' the store's coefficients, `data_dir` and the region, and +#' returns extra coefficient rows - a translation of them - to +#' summarize alongside. +#' @return A list of data frames: `coverage`, `coefficient_summary`, +#' `metric_summary`, `grid_summary`, and for the run record, +#' `store_rows` (row counts per store) and `metrics` (every +#' metric row, labelled with its run and region). +#' +#' @example # Example usage of the function +#' # results <- collect_results("2_pipeline/exp_000_parity_v2", +#' # data_dir, translate = list(bird = bird_habitat_translation)) +collect_results <- function(pipeline_dir, data_dir = NULL, + translate = list()) { + stores <- find_stores(pipeline_dir) + + if (nrow(stores) == 0) { + message("No result stores in ", pipeline_dir, ".") + } + + pieces <- lapply(seq_len(nrow(stores)), function(i) { + store <- stores[i, ] + labels <- store[, c("taxon", "run", "region", "season", "part")] + rownames(labels) <- NULL + + coefficients <- read_results(store$dir, "coefficients") + metrics <- read_results(store$dir, "metrics") + grid <- read_results(store$dir, "grid_predictions") + + counts <- data.frame( + run = store$run, region = store$region, + species = length(unique(c(coefficients$species, + metrics$species))), + draws = length(unique(c(coefficients$boot, metrics$boot))), + coefficient_rows = NROW(coefficients), + metric_rows = NROW(metrics), + grid_rows = NROW(grid), + stringsAsFactors = FALSE + ) + + # A store written before stages were recorded holds the final + # stage only + if (!is.null(coefficients) && !"stage" %in% names(coefficients)) { + coefficients$stage <- store$final_stage + } + + # Coefficients a taxon needs translated before comparing + translator <- translate[[store$taxon]] + + if (is.function(translator) && !is.null(coefficients)) { + extra <- translator(coefficients, data_dir, store$region) + + if (!is.null(extra) && nrow(extra) > 0) { + coefficients <- rbind( + coefficients, extra[, names(coefficients)] + ) + } + } + + summarised <- function(rows, value, keys) { + if (is.null(rows) || nrow(rows) == 0) { + return(NULL) + } + + cbind(labels, summarise_draws(rows, value, keys)) + } + + list( + coverage = cbind(labels, data.frame( + species = counts$species, + draws = counts$draws, + stages = if (is.null(coefficients)) { + "" + } else { + paste(unique(coefficients$stage), collapse = ", ") + }, + has_coefficients = !is.null(coefficients), + has_grid = !is.null(grid), + stringsAsFactors = FALSE + )), + coefficient_summary = summarised( + coefficients, "estimate", c("stage", "species", "term") + ), + metric_summary = summarised( + metrics, "value", c("species", "metric") + ), + grid_summary = summarised( + grid, "prediction", c("species", "grid_unit") + ), + store_rows = counts, + metrics = if (is.null(metrics)) { + NULL + } else { + cbind(run = store$run, metrics) + } + ) + }) + + combine <- function(name) { + parts <- lapply(pieces, `[[`, name) + parts <- parts[!vapply(parts, is.null, logical(1))] + + if (length(parts) == 0) { + return(NULL) + } + + out <- as.data.frame(rbindlist(parts, use.names = TRUE)) + rownames(out) <- NULL + out + } + + stats::setNames( + lapply( + c("coverage", "coefficient_summary", "metric_summary", + "grid_summary", "store_rows", "metrics"), + combine + ), + c("coverage", "coefficient_summary", "metric_summary", + "grid_summary", "store_rows", "metrics") + ) +} + +# 5. write_summaries() ---- + +#' Write the Summary Tables +#' +#' @param results From collect_results(). +#' @param out_dir Character. Tables go in `out_dir/tables/`. +#' @return The paths written, invisibly. +#' +#' @example # Example usage of the function +#' # write_summaries(results, "3_output/exp_000_parity_v2") +write_summaries <- function(results, out_dir) { + tables_dir <- file.path(out_dir, "tables") + dir.create(tables_dir, recursive = TRUE, showWarnings = FALSE) + + tables <- c( + "coverage", "metric_summary", "grid_summary", + "coefficient_summary" + ) + written <- character(0) + + for (name in tables) { + table <- results[[name]] + + if (is.null(table)) { + next + } + + path <- file.path(tables_dir, paste0(name, ".csv")) + fwrite(table, path, na = "") + cat("Wrote tables/", name, ".csv: ", nrow(table), " rows\n", + sep = "") + written <- c(written, path) + } + + invisible(written) +} + +# End of script ---- diff --git a/1_code/harness/utils/step_runner.R b/1_code/harness/utils/step_runner.R deleted file mode 100644 index 1dd6cec..0000000 --- a/1_code/harness/utils/step_runner.R +++ /dev/null @@ -1,109 +0,0 @@ -# --- -# title: Pipeline Step Runner -# author: Brendan Casey -# created: 2026-09-05 -# inputs: none -# outputs: none -# notes: -# - Taxon-agnostic helper for running one numbered module -# script, timing it, and mirroring its output to a log. -# Sourced by the experiments in 1_code/experiments/. -# - Was v2_script_runner.R, which also switched the working -# directory to a v2 project root and restored the caller's -# objects after each script's rm(list = ls()). The module -# scripts do neither now: they take their paths as arguments -# and clear nothing, so both are gone along with the input -# checks, which each script does for itself against the paths -# it was actually given. -# - Scripts are sourced into the global environment on purpose. -# They read their configuration from objects the experiment -# has already set there, and their clusterExport() calls read -# from .GlobalEnv. -# --- - -# 1. run_step() ---- - -#' Run One Pipeline Step -#' -#' Sources a script, reporting its label and the time it took. -#' Mirrors console output to a timestamped log file so an -#' unattended run leaves a record. -#' -#' The script is sourced into the global environment, so it sees -#' the configuration the caller set there - `taxon`, `out_dir` -#' and the rest - and its parallel workers can export from it. -#' -#' @param label Character. A descriptive label for the step, used -#' in the console output and in the log file name -#' (e.g. "02: Hierarchical models (mites)"). -#' @param script Character. Path to the script to run. -#' @param log_dir Character. Directory for log files, or NULL to -#' print to the console only. -#' @return NULL, invisibly. Output is printed to the console and, -#' when `log_dir` is set, copied to a log file. -#' -#' @example -#' # Example usage of the function -#' run_step( -#' label = "02: Hierarchical models (mites)", -#' script = "1_code/modules/lichens/02_hierarchical_models.R", -#' log_dir = log_path -#' ) -#' -#' # Run without writing a log file -#' run_step( -#' label = "02: Hierarchical models (mites)", -#' script = "1_code/modules/lichens/02_hierarchical_models.R", -#' log_dir = NULL -#' ) -run_step <- function(label, script, log_dir = NULL) { - # Step 1: Fail on a missing script before anything is logged - if (!file.exists(script)) { - stop("Script not found: ", script, call. = FALSE) - } - - # Step 2: Announce the step and record the start time - cat("\n--- Running ", label, " ---\n", sep = "") - start <- Sys.time() - - # Step 3: Mirror output to a timestamped log. split = TRUE keeps - # it on the console too, and on.exit() closes the connection - # even if the script errors. - if (!is.null(log_dir)) { - if (!dir.exists(log_dir)) { - dir.create(log_dir, recursive = TRUE) - } - - stamp <- format(Sys.time(), "%Y-%m-%d_%H-%M-%S") - safe_label <- gsub("[^A-Za-z0-9_]+", "_", label) - log_file <- file.path( - log_dir, - paste0(stamp, "_", safe_label, ".log") - ) - - log_con <- file(log_file, open = "wt") - sink(log_con, split = TRUE) - on.exit( - { - sink() - close(log_con) - }, - add = TRUE - ) - - cat("Logging to ", log_file, "\n", sep = "") - } - - # Step 4: Source into the global environment, which is where the - # module scripts read their configuration and where their - # clusterExport() calls look for objects - source(script) - - # Step 5: Report how long the step took - elapsed <- format(round(Sys.time() - start, 1)) - cat("Completed ", label, " in ", elapsed, "\n", sep = "") - - invisible(NULL) -} - -# End of script ---- diff --git a/1_code/methods/README.md b/1_code/methods/README.md new file mode 100644 index 0000000..6c5788c --- /dev/null +++ b/1_code/methods/README.md @@ -0,0 +1,151 @@ +# Methods + +Every method a spec can name lives here, one file each. A file +defines its method and registers it under a name; a spec names it; +the harness looks it up. Adding a method is adding a file. No +harness file changes. + +```r +source("1_code/harness/harness.R") +load_framework() +list_methods() # everything registered, with what it needs +list_methods("engine") # one kind +``` + +| Folder | Kind | Registered with | Named in a spec as | +| --- | --- | --- | --- | +| `engines/` | How a model is fitted | `register_engine()` | `stage$engine` | +| `selection/` | How a candidate set becomes one result | `register_selection()` | `stage$selection` | +| `resampling/` | Which survey units each draw fits | `register_resampler()` | `resample$scheme` | +| `metrics/` | How a prediction is scored | `register_metric()` | scored for every run | + +Files whose names start with `_` are shared helpers and load first +(`engines/_glm_family.R`). `_templates/` holds a copy-and-fill +skeleton for an engine and a selection rule; it is not loaded. + +## What is registered + +| Kind | Name | What it does | Needs | +| --- | --- | --- | --- | +| engine | `glm` | `stats::glm` (v2 mammals, birds) | | +| engine | `bayesglm` | `arm::bayesglm`, weakly informative priors (v2 plants) | | +| engine | `xgboost` | Boosted regression trees, `xgboost::xgb.train`; binomial (proportions too), Poisson, log-link Gamma, Gaussian; no coefficients, information criterion or standard errors, so use the `single` rule | | +| selection | `single` | Fit the first candidate; no ranking | | +| selection | `aic_best` | Keep the lowest-scoring candidate | ic | +| selection | `aic_average` | Average coefficients by Akaike weight (v2 climate) | ic, coefficients | +| selection | `staged_bic` | Forward selection through groups (v2 bird landcover) | ic, coefficients | +| selection | `ivw_grid` | Inverse-variance average onto the habitat grid (v2 plant habitat) | coefficients, se | +| selection | `hurdle` | Presence, then abundance given presence, each through an inner rule (`part_selection`); reports both and their product on the grid (v2 mammals) | those of `part_selection` | +| resampler | `precomputed` | Replay draws stored with the dataset (v2 birds; plant `v2_ids`) | | +| resampler | `spatial_block` | Bootstrap within coarse spatial blocks (v2 plants, mammals) | | +| resampler | `spatial_cv` | Spatial block cross-validation; each draw holds out one fold | | +| metric | `auc`, `deviance_explained`, `rmse`, `spearman`, `calibration_slope`, `prevalence`, `n` | Scored in-sample and out-of-bag for every draw | | + +`validate_spec()` checks a spec against this table before any data +is read. A rule paired with an engine that lacks what it needs, an +unregistered name, or a misspelt field stops the run with a +sentence saying which. + +## The engine contract + +An engine is a list: + +| Element | Required | Contract | +| --- | --- | --- | +| `name`, `description` | yes | The registered name, and one line for `list_methods()` | +| `capabilities` | yes | A subset of `"coefficients"`, `"ic"`, `"se"`, `"converged"` | +| `fit(formula, data, family, weights, offset, control)` | yes | Returns `list(fit, ok, message)`. **Never raises**: a failure returns `ok = FALSE` and the message, because candidate sets hold models that cannot be fitted for every species and draw | +| `predict(fit, newdata, type, se)` | yes | `type` is `"link"` or `"response"`. Returns one number per row, or `NULL` when it cannot. With `se = TRUE` (only if declared), `list(fit, se.fit)`. An offset in the formula, `+ offset(offset)`, must be applied to new data | +| `coef(fit)` | with `coefficients` | A data frame of `term`, `estimate`, `se` | +| `ic(fit, type)` | with `ic` | AIC, AICc or BIC; `Inf` for a failed fit | +| `converged(fit)`, `nobs(fit)` | no | Assumed converged, and observations read with `stats::nobs`, when absent | + +The harness passes the offset in the formula, never as an +argument: an offset argument is not carried onto new data by +`predict()`, so every prediction outside the draw would be wrong. + +`family` arrives as a name (`"binomial"`, `"poisson"`) or a family +object (`Gamma(link = "log")`). An engine with its own vocabulary, +such as `xgboost`'s `objective = "binary:logistic"`, translates it. + +### Testing an engine + +```r +check_engine(my_engine()) # a definition, or a registered name +check_engine("glm") +``` + +`check_engine()` fits small synthetic binomial and Poisson data, +with weights, an offset and a factor, and checks every part of the +contract the engine declares. Every row should read `TRUE` before +the engine meets the dataset. + +### Which rules an engine can use + +An engine without an information criterion or coefficients (a +boosted regression tree, a random forest) can still run any +stage, using the `single` rule: one formula, fitted as given. To +compare it with v2 on a stage v2 fits with model averaging, +replace that stage's engine, selection and models together. The +v2 post-processing steps that read coefficients (the plant +stand-age splines, say) then do not apply, and the stage should +drop them; see +[`docs/getting_started.md`](../../docs/getting_started.md#2-a-different-engine-or-rule-specs). + +`replace_stage_method()` does all of this in one call. Its stage +defaults to the spec's `habitat_stage`, so the same call serves +every taxon: + +```r +specs <- lapply(standard_specs(), replace_stage_method, + engine = "xgboost") +``` + +A **composite** rule (one that declares `part_selection`, as +`hurdle` does) keeps its place, and the new rule becomes the one +each part runs. The mammal hurdle's structure (presence on every +unit, abundance where the species was found) is a property of the +data, so it stays; the new engine fits both halves, and v2's +table-building (`post_process`) is removed. + +A `post_process` step is a function of the selection result and +the stage's data. A step that also declares `grid`, `species` or +`stage` is given them. + +A stage carried into the next one (`carry_as`) is carried from the +stage's final model: its averaged coefficients where it has them, +its prediction otherwise. + +## The other contracts + +**Selection rule.** A function taking `models` (formulas, or named +groups of them), `base`, `data`, `engine`, `family`, `weights`, +`offset`, `ic` and `control`, returning `selection_result()`. Any +further argument is a setting, filled from the stage field of the +same name. Fit candidates with `fit_candidates()`; read +coefficients with `engine_coef()`. A rule that combines candidates +passes `predict` (see `combined_predictor()`) so metrics and grid +predictions use the combined model. Register with the capabilities +it `requires`. + +**Resampler.** A function of `frame`, `iterations`, `seed` and +`context` (data directory, taxon, species, spec), plus any setting +from the spec's `resample` list, returning one character vector of +survey unit ids per draw: the units that draw fits. Units left out +are scored out-of-bag. Draw under `with_seed(seed, ...)` so a +species' draws depend only on its seed. + +**Metric.** A function of `observed` and `predicted`, both on the +response scale, returning one number. Return `NA` rather than +stopping when the metric does not apply to a draw. + +## Checklist for a new method + +- [ ] One file in the right folder, from `_templates/` where there + is one, with the standard header. +- [ ] Registered with a one-line description and, for a rule, what + it `requires`. +- [ ] `check_engine()` passes, for an engine. +- [ ] `Rscript 1_code/tests/run_tests.R` passes. +- [ ] Tried on one species and a few draws (`species = "one_each"`, + `n_bootstraps = 5`) before a full run. diff --git a/1_code/methods/_templates/engine_template.R b/1_code/methods/_templates/engine_template.R new file mode 100644 index 0000000..4e89add --- /dev/null +++ b/1_code/methods/_templates/engine_template.R @@ -0,0 +1,94 @@ +# --- +# title: Engine - [Name] +# author: [Your Name] +# created: [YYYY-MM-DD] +# inputs: none +# outputs: registers the `[name]` engine +# notes: +# - A template. Copy to 1_code/methods/engines/[name].R, fill in +# the four functions, and run check_engine(engine_[name]()) +# until every check passes. Then name it in a spec stage: +# `engine = "[name]"`. +# - The contract, in full, is in 1_code/methods/README.md. In +# short: +# - fit() never raises. It returns list(fit, ok, message), +# with ok = FALSE and the error message when fitting fails, +# because a candidate set is expected to hold models that +# cannot be fitted for every species and draw. +# - predict() returns one number per row of newdata, or NULL +# when it cannot. An offset in the formula +# (`+ offset(offset)`) must be applied to new data. +# - Declare only the capabilities the engine really has. +# Selection rules check them: a rule that ranks candidates +# needs `ic`; one that averages coefficients needs +# `coefficients`; inverse-variance averaging needs `se`. +# With none of them, use the `single` rule. +# - Load any package lazily with requireNamespace(), so a run +# that does not use this engine does not need the package. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# [package], lazily; see the header. + +# 2. engine_[name]() ---- + +#' [One-line title of the method] +#' +#' [What it fits, and any setting it takes through `control`.] +#' +#' @return An engine definition. +#' +#' @example # Example usage of the function +#' # check_engine(engine_[name]()) +engine_template <- function() { + list( + name = "[name]", + description = "[One line for list_methods()]", + + # A subset of engine_capabilities(): "coefficients", "ic", + # "se", "converged" + capabilities = character(0), + + # Step 1: Fit. Never raise; return the failure instead. + fit = function(formula, data, family, weights = NULL, + offset = NULL, control = list()) { + tryCatch( + { + fit <- NULL # [fit the model with formula and data] + list(fit = fit, ok = TRUE, message = NA_character_) + }, + error = function(e) { + list(fit = NULL, ok = FALSE, message = conditionMessage(e)) + } + ) + }, + + # Step 2: Predict. `type` is "link" or "response"; with + # se = TRUE (only when declared), list(fit, se.fit). + predict = function(fit, newdata, type = "link", se = FALSE) { + if (is.null(fit) || !isTRUE(fit$ok)) { + return(NULL) + } + + tryCatch( + NULL, # [predict fit$fit onto newdata, one value per row] + error = function(e) NULL + ) + }, + + # Step 3, when declared: a data frame of term, estimate, se + coef = NULL, + + # Step 4, when declared: a number for type "AIC", "AICc" or + # "BIC"; Inf for a failed fit, so it never wins + ic = NULL + ) +} + +# 3. Register ---- +# Uncomment once check_engine() passes: +# register_engine(engine_template()) + +# End of script ---- diff --git a/1_code/methods/_templates/selection_template.R b/1_code/methods/_templates/selection_template.R new file mode 100644 index 0000000..6aa36c6 --- /dev/null +++ b/1_code/methods/_templates/selection_template.R @@ -0,0 +1,80 @@ +# --- +# title: Selection Rule - [Name] +# author: [Your Name] +# created: [YYYY-MM-DD] +# inputs: none +# outputs: registers the `[name]` rule +# notes: +# - A template. Copy to 1_code/methods/selection/[name].R and +# fill it in. Then name it in a spec stage: +# `selection = "[name]"`. +# - A rule turns a candidate set into one result. It fits its +# own candidates, with fit_candidates(), and returns +# selection_result(). +# - Any argument beyond the standard nine is a setting, filled +# from the stage field of the same name. A stage with +# `my_threshold = 3` passes 3 to a rule declaring +# `my_threshold`. No harness change is needed. +# - Read coefficients with engine_coef(engine, fit), never with +# coef() on the fitted object, so the rule works with any +# engine that declares `coefficients`. +# - When the rule combines candidates, pass `predict` to +# selection_result() - combined_predictor() builds one - so +# metrics and grid predictions use the combined model rather +# than one candidate. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. select_[name]() ---- + +#' [One-line title of the rule] +#' +#' @inheritParams select_single +#' @param [setting] [What the stage field sets.] +#' @return A selection result. +#' +#' @example # Example usage of the function +#' # select_[name](models, response ~ 1, d, get_engine("glm"), +#' # "binomial") +select_template <- function( + models, base, data, engine, family, + weights = NULL, offset = NULL, ic = "AICc", control = list() +) { + # Step 1: Fit every candidate + fitted <- fit_candidates( + models, base, data, engine, family, weights, offset, ic, + control + ) + table <- candidate_table(fitted) + + if (!any(table$ok)) { + return(selection_result(NULL, NULL, table, fitted)) + } + + # Step 2: Choose or combine. [Replace with the rule.] + chosen <- which(table$ok)[1] + + # Step 3: Return the result + selection_result( + fit = fitted$fits[[chosen]], + coefficients = engine_coef(engine, fitted$fits[[chosen]]), + ic_table = table, + fitted = fitted, + engine = engine + ) +} + +# 3. Register ---- +# Name the engine capabilities the rule needs in `requires`. +# Uncomment when ready: +# register_selection( +# "[name]", select_template, +# "[One line for list_methods()]", +# requires = character(0) +# ) + +# End of script ---- diff --git a/1_code/methods/engines/_glm_family.R b/1_code/methods/engines/_glm_family.R new file mode 100644 index 0000000..1f054e1 --- /dev/null +++ b/1_code/methods/engines/_glm_family.R @@ -0,0 +1,256 @@ +# --- +# title: Shared Implementation for GLM-Family Engines +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: none; defines functions in memory +# notes: +# - Shared by glm and bayesglm, which take and return the same +# shapes. The leading underscore makes it load first. +# - Fitting failures are returned, not raised, so selection +# rules can see which candidates failed. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. fit_glm_family() ---- + +#' Fit a GLM-Family Model, Catching Failure +#' +#' @param fitter Function. stats::glm or arm::bayesglm. +#' @param formula A model formula. +#' @param data A data frame. +#' @param family Character or family object. +#' @param weights Numeric vector or NULL. +#' @param offset Numeric vector or NULL. +#' @param control Named list of extra arguments to the fitter. +#' @return A list with `fit` (or NULL), `ok`, and `message`. +#' +#' @example # Example usage of the function +#' # fit_glm_family(stats::glm, y ~ x, d, "binomial") +fit_glm_family <- function( + fitter, + formula, + data, + family, + weights = NULL, + offset = NULL, + control = list() +) { + args <- c( + list(formula = formula, data = data, family = family), + control + ) + + # Passed as values so one formula serves taxa with and without + # weights. The harness passes offset = NULL and puts offsets in + # the formula instead (see run_stage()). + if (!is.null(weights)) { + args$weights <- weights + } + + if (!is.null(offset)) { + args$offset <- offset + } + + # Gamma fits start at the mean response, as v2's mammal + # abundance models do: starting at y diverges for skewed + # densities + family_name <- if (is.character(family)) family else family$family + starts <- c("mustart", "start", "etastart") + + if (identical(family_name, "Gamma") && + !any(starts %in% names(control))) { + y <- data[[all.vars(formula[[2]])[1]]] + args$mustart <- rep(mean(y, na.rm = TRUE), nrow(data)) + } + + # Warnings (e.g. non-convergence) are recorded, not failures; + # muffled rather than caught so the model is fitted once + warnings_seen <- character(0) + + tryCatch( + { + fit <- withCallingHandlers( + do.call(fitter, args), + warning = function(w) { + warnings_seen <<- c(warnings_seen, conditionMessage(w)) + invokeRestart("muffleWarning") + } + ) + + list( + fit = fit, + ok = TRUE, + message = if (length(warnings_seen) == 0) { + NA_character_ + } else { + paste(unique(warnings_seen), collapse = "; ") + } + ) + }, + error = function(e) { + list(fit = NULL, ok = FALSE, message = conditionMessage(e)) + } + ) +} + +# 3. fit_coefficients() ---- + +#' Take Coefficients and Standard Errors from a Fit +#' +#' @param fit A list from fit_glm_family(). +#' @return A data frame of term, estimate and se, or NULL when +#' the fit failed. +#' +#' @example # Example usage of the function +#' # fit_coefficients(fit) +fit_coefficients <- function(fit) { + if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { + return(NULL) + } + + estimates <- stats::coef(fit$fit) + errors <- tryCatch( + stats::coef(summary(fit$fit))[, "Std. Error"], + error = function(e) rep(NA_real_, length(estimates)) + ) + + data.frame( + term = names(estimates), + estimate = as.numeric(estimates), + se = as.numeric(errors[match(names(estimates), names(errors))]), + stringsAsFactors = FALSE + ) +} + +# 4. fit_predict() ---- + +#' Predict from a Fit +#' +#' @param fit A list from fit_glm_family(). +#' @param newdata A data frame to predict onto. +#' @param type Character. "link" or "response". +#' @param se Logical. Return standard errors alongside the fit. +#' @return A numeric vector, or when `se` is TRUE a list of `fit` +#' and `se.fit`. NULL when the fit failed. +#' +#' @example # Example usage of the function +#' # fit_predict(fit, prediction_grid, type = "response") +#' # fit_predict(fit, grid, "link", se = TRUE)$se.fit +fit_predict <- function(fit, newdata, type = "link", se = FALSE) { + if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { + return(NULL) + } + + tryCatch( + { + predicted <- stats::predict( + fit$fit, newdata = newdata, type = type, se.fit = se + ) + + if (!se) { + return(as.numeric(predicted)) + } + + list( + fit = as.numeric(predicted$fit), + se.fit = as.numeric(predicted$se.fit) + ) + }, + error = function(e) NULL + ) +} + +# 5. information_criterion() ---- + +#' Score a Fit for Model Selection +#' +#' AICc by default: candidate sets are large relative to a rare +#' species' detections. +#' +#' @param fit A list from fit_glm_family(). +#' @param type Character. "AIC", "AICc" or "BIC". +#' @return A numeric value; Inf when the fit failed, so a failed +#' model never wins a selection. +#' +#' @example # Example usage of the function +#' # information_criterion(fit, "BIC") +information_criterion <- function(fit, type = "AICc") { + if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { + return(Inf) + } + + value <- tryCatch( + switch( + type, + AIC = stats::AIC(fit$fit), + BIC = stats::BIC(fit$fit), + AICc = { + # k from logLik df, as MuMIn, so aliased (NA) terms are + # not counted + k <- attr(stats::logLik(fit$fit), "df") + n <- stats::nobs(fit$fit) + # Undefined once k + 1 >= n; Inf excludes the model + if (is.na(n) || n - k - 1 <= 0) { + Inf + } else { + stats::AIC(fit$fit) + (2 * k * (k + 1)) / (n - k - 1) + } + }, + stop( + "Unknown information criterion `", type, "`.", + call. = FALSE + ) + ), + error = function(e) Inf + ) + + if (is.na(value)) Inf else value +} + +# 6. fit_converged() ---- + +#' Did a Fit Converge +#' +#' @param fit A list from fit_glm_family(). +#' @return Logical. FALSE when the fit failed or did not +#' converge. +#' +#' @example # Example usage of the function +#' # fit_converged(fit) +fit_converged <- function(fit) { + if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { + return(FALSE) + } + + converged <- fit$fit$converged + + if (is.null(converged)) { + return(TRUE) + } + + isTRUE(converged) +} + +# 7. fit_nobs() ---- + +#' Number of Observations a Fit Used +#' +#' @param fit A list from fit_glm_family(). +#' @return An integer, or NA when the fit failed. +#' +#' @example # Example usage of the function +#' # fit_nobs(fit) +fit_nobs <- function(fit) { + if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit)) { + return(NA_integer_) + } + + as.integer(stats::nobs(fit$fit)) +} + +# End of script ---- diff --git a/1_code/methods/engines/bayesglm.R b/1_code/methods/engines/bayesglm.R new file mode 100644 index 0000000..164507a --- /dev/null +++ b/1_code/methods/engines/bayesglm.R @@ -0,0 +1,74 @@ +# --- +# title: Engine - Penalized GLM with Weakly Informative Priors +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `bayesglm` engine +# notes: +# - arm::bayesglm with default Cauchy priors, as the v2 plant +# models use. It is penalized maximum likelihood (point +# estimate and SE, no posterior), so glm vs bayesglm is a +# regularization contrast, not frequentist vs Bayesian. +# - Rare plant species separate completely under 30-term habitat +# formulas, so plain glm fails for them; the penalty is +# load-bearing. +# - arm is loaded lazily, only when this engine is used. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# arm, lazily; see the header. + +# 2. engine_bayesglm() ---- + +#' Penalized GLM with Weakly Informative Priors +#' +#' @return An engine definition. +#' +#' @example # Example usage of the function +#' # e <- engine_bayesglm() +engine_bayesglm <- function() { + list( + name = "bayesglm", + description = paste( + "Penalized GLM, weakly informative Cauchy priors", + "(arm::bayesglm)" + ), + capabilities = c("coefficients", "ic", "se", "converged"), + fit = function(formula, data, family, weights = NULL, + offset = NULL, control = list()) { + if (!requireNamespace("arm", quietly = TRUE)) { + stop( + "The bayesglm engine needs the arm package.", + call. = FALSE + ) + } + + # maxit defaults to the v2 value, which was raised from the + # arm default because rare species converge slowly + if (is.null(control$maxit)) { + control$maxit <- 250 + } + + fit_glm_family( + arm::bayesglm, formula, data, family, weights, offset, + control + ) + }, + coef = fit_coefficients, + predict = function(fit, newdata, type = "link", se = FALSE) { + fit_predict(fit, newdata, type, se) + }, + ic = function(fit, type = "AICc") { + information_criterion(fit, type) + }, + converged = fit_converged, + nobs = fit_nobs + ) +} + +# 3. Register ---- +register_engine(engine_bayesglm()) + +# End of script ---- diff --git a/1_code/methods/engines/glm.R b/1_code/methods/engines/glm.R new file mode 100644 index 0000000..fe859fc --- /dev/null +++ b/1_code/methods/engines/glm.R @@ -0,0 +1,55 @@ +# --- +# title: Engine - Generalized Linear Model (glm) +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `glm` engine +# notes: +# - stats::glm. The engine the v2 mammal and bird models fit +# with. +# - Shares its implementation with bayesglm; see +# _glm_family.R. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. engine_glm() ---- + +#' Plain Generalized Linear Model +#' +#' @return An engine definition. +#' +#' @example # Example usage of the function +#' # e <- engine_glm() +#' # fit <- e$fit(response ~ MAP, data, family = "binomial") +engine_glm <- function() { + list( + name = "glm", + description = "Generalized linear model (stats::glm)", + capabilities = c("coefficients", "ic", "se", "converged"), + fit = function(formula, data, family, weights = NULL, + offset = NULL, control = list()) { + fit_glm_family( + stats::glm, formula, data, family, weights, offset, + control + ) + }, + coef = fit_coefficients, + predict = function(fit, newdata, type = "link", se = FALSE) { + fit_predict(fit, newdata, type, se) + }, + ic = function(fit, type = "AICc") { + information_criterion(fit, type) + }, + converged = fit_converged, + nobs = fit_nobs + ) +} + +# 3. Register ---- +register_engine(engine_glm()) + +# End of script ---- diff --git a/1_code/methods/engines/xgboost.R b/1_code/methods/engines/xgboost.R new file mode 100644 index 0000000..e3e5cf9 --- /dev/null +++ b/1_code/methods/engines/xgboost.R @@ -0,0 +1,345 @@ +# --- +# title: Engine - Boosted Regression Trees (xgboost) +# author: Brendan Casey +# created: 2026-10-05 +# inputs: none +# outputs: registers the `xgboost` engine +# notes: +# - Use via replace_stage_method() (one models_union() formula, +# `single` rule): it declares no coefficients, IC or SE. +# - Replaces gbm, which cannot fit the mammal hurdle (no 0-1 +# proportion response, no Gamma); xgboost has +# `binary:logistic` with proportions and log-link `reg:gamma`. +# - Only formula variables are used (`I(x^2)`, `a:b` give x, a, +# b). Categoricals are one-hot coded with the fitted levels; +# NAs stay NA. The offset is the `base_margin`, in fitting and +# prediction. +# - Rounds are chosen by early stopping on a seeded holdout +# (`valid_fraction`), then refitted on all rows. Noisy on +# small draws. +# - `control` settings: nrounds, eta, max_depth, subsample, +# colsample_bytree, min_child_weight, early_stopping_rounds, +# valid_fraction, seed. Defaults are untuned and quick (eta +# 0.05 is higher than Elith, Leathwick and Hastie 2008 +# advise); tune before judging the method. +# - One thread per fit; the harness parallelizes over species. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# xgboost (version: 3.2.1), loaded lazily + +# 2. engine_xgboost() ---- + +#' Boosted Regression Trees with xgboost +#' +#' @return An engine definition. +#' +#' @example # Example usage of the function +#' # check_engine(engine_xgboost()) +engine_xgboost <- function() { + list( + name = "xgboost", + description = "Boosted regression trees (xgboost::xgb.train)", + capabilities = character(0), + fit = xgboost_fit, + predict = xgboost_predict + ) +} + +# 3. xgboost_fit() ---- + +#' Fit Boosted Regression Trees, Catching Failure +#' +#' @param formula A model formula. Only its variables and any +#' `offset(offset)` term are used. +#' @param data A data frame. +#' @param family Character ("binomial", "poisson", "gaussian") or +#' a family object (Gamma with a log link). +#' @param weights Numeric vector or NULL. +#' @param offset Unused; the offset comes from the formula. +#' @param control Named list of settings; see the header. +#' @return A list with `fit` (or NULL), `ok` and `message`. +#' +#' @example # Example usage of the function +#' # xgboost_fit(response ~ MAP + FFP, d, "binomial") +xgboost_fit <- function(formula, data, family, weights = NULL, + offset = NULL, control = list()) { + tryCatch( + { + if (!requireNamespace("xgboost", quietly = TRUE)) { + stop("The xgboost engine needs the xgboost package.", + call. = FALSE) + } + + settings <- utils::modifyList( + list( + nrounds = 1000, eta = 0.05, max_depth = 3, + subsample = 0.5, colsample_bytree = 1, + min_child_weight = 1, early_stopping_rounds = 50, + valid_fraction = 0.2, seed = 20260909L + ), + control + ) + + # Step 1: Drop rows missing response, weight or offset (as a + # GLM does); missing predictors are kept + terms <- xgboost_terms(formula) + w_all <- if (is.null(weights)) rep(1, nrow(data)) else weights + keep <- !is.na(data[[terms$response]]) & !is.na(w_all) + + if (terms$offset) { + keep <- keep & !is.na(data$offset) + } + + rows <- data[keep, , drop = FALSE] + design <- xgboost_design(rows, terms$variables) + y <- rows[[terms$response]] + w <- w_all[keep] + margin <- if (terms$offset) rows$offset else NULL + + params <- list( + objective = xgboost_objective(family), + eta = settings$eta, + max_depth = settings$max_depth, + subsample = settings$subsample, + colsample_bytree = settings$colsample_bytree, + min_child_weight = settings$min_child_weight, + nthread = 1 + ) + + matrix_of <- function(index) { + xgboost::xgb.DMatrix( + design$x[index, , drop = FALSE], label = y[index], + weight = w[index], + base_margin = if (is.null(margin)) NULL else margin[index] + ) + } + + # Step 2: Rounds by early stopping, then refit on every row + model <- with_seed(settings$seed, { + n <- length(y) + held <- sample.int( + n, max(1L, floor(n * settings$valid_fraction)) + ) + trial <- xgboost::xgb.train( + params = params, data = matrix_of(-held), + nrounds = settings$nrounds, + evals = list(valid = matrix_of(held)), + early_stopping_rounds = settings$early_stopping_rounds, + verbose = 0 + ) + rounds <- xgboost_best_rounds(trial, settings$nrounds) + + xgboost::xgb.train( + params = params, data = matrix_of(seq_len(n)), + nrounds = rounds, verbose = 0 + ) + }) + + list( + fit = list( + model = model, + variables = terms$variables, + columns = colnames(design$x), + levels = design$levels, + offset = terms$offset, + linkinv = family_linkinv(family) + ), + ok = TRUE, + message = NA_character_ + ) + }, + error = function(e) { + list(fit = NULL, ok = FALSE, message = conditionMessage(e)) + } + ) +} + +# 4. xgboost_predict() ---- + +#' Predict from an xgboost Fit +#' +#' @param fit A list from xgboost_fit(). +#' @param newdata A data frame to predict onto. +#' @param type Character. "link" or "response". +#' @param se Logical. Not supported; NULL is returned when TRUE. +#' @return A numeric vector, or NULL when the fit failed. +#' +#' @example # Example usage of the function +#' # xgboost_predict(fit, newdata, "response") +xgboost_predict <- function(fit, newdata, type = "link", se = FALSE) { + if (is.null(fit) || !isTRUE(fit$ok) || is.null(fit$fit) || se) { + return(NULL) + } + + tryCatch( + { + tree <- fit$fit + design <- xgboost_design(newdata, tree$variables, tree$levels) + + # Without base_margin xgboost uses its own fitted intercept + matrix <- xgboost::xgb.DMatrix( + design$x[, tree$columns, drop = FALSE], + base_margin = if (tree$offset) newdata$offset else NULL + ) + + link <- stats::predict(tree$model, matrix, outputmargin = TRUE) + + if (type == "link") { + as.numeric(link) + } else { + as.numeric(tree$linkinv(link)) + } + }, + error = function(e) NULL + ) +} + +# 5. Helpers ---- + +## 5.1 xgboost_terms() ---- + +#' Read a Formula as the Engine Uses It +#' +#' @param formula A model formula. +#' @return A list of `response`, `variables` and `offset` (TRUE +#' when the formula has an `offset(offset)` term). +#' +#' @example # Example usage of the function +#' # xgboost_terms(count ~ MAP + I(MAP^2) + offset(offset)) +xgboost_terms <- function(formula) { + text <- paste(deparse(formula), collapse = " ") + + list( + response = all.vars(formula[[2]]), + variables = setdiff( + all.vars(formula[[3]]), c("offset", "weight") + ), + offset = grepl("offset(offset)", text, fixed = TRUE) + ) +} + +## 5.2 xgboost_design() ---- + +#' The Numeric Matrix xgboost Takes +#' +#' Text, logical and factor columns become one indicator per +#' fitted level; a missing category is NA in every indicator. +#' +#' @param data A data frame. +#' @param variables Character vector of column names. +#' @param levels Named list of factor levels from the fit, or +#' NULL when fitting. +#' @return A list of `x` (a numeric matrix) and `levels`. +#' +#' @example # Example usage of the function +#' # xgboost_design(d, c("MAP", "Protocol"))$x +xgboost_design <- function(data, variables, levels = NULL) { + absent <- setdiff(variables, names(data)) + + if (length(absent) > 0) { + stop("Columns not in the data: ", paste(absent, collapse = ", "), + call. = FALSE) + } + + fitting <- is.null(levels) + levels <- levels %||% list() + columns <- list() + + for (column in variables) { + value <- data[[column]] + categorical <- column %in% names(levels) || + (fitting && (is.character(value) || is.logical(value) || + is.factor(value))) + + if (!categorical) { + columns[[column]] <- as.numeric(value) + next + } + + if (fitting) { + levels[[column]] <- sort(unique(as.character( + value[!is.na(value)] + ))) + } + + text <- as.character(value) + + for (level in levels[[column]]) { + columns[[paste0(column, "_", level)]] <- ifelse( + is.na(text), NA_real_, as.numeric(text == level) + ) + } + } + + x <- do.call(cbind, columns) + + if (is.null(dim(x))) { + x <- matrix(x, ncol = length(columns)) + } + + colnames(x) <- names(columns) + + list(x = x, levels = levels) +} + +## 5.3 xgboost_objective() ---- + +#' The xgboost Objective for a GLM Family +#' +#' @param family Character or family object. +#' @return Character, an xgboost objective. +#' +#' @example # Example usage of the function +#' # xgboost_objective("binomial") # "binary:logistic" +xgboost_objective <- function(family) { + name <- if (is.character(family)) family else family$family + link <- if (is.character(family)) NA_character_ else family$link + + if (name == "Gamma" && !identical(link, "log")) { + stop("The xgboost engine fits Gamma on the log link only.", + call. = FALSE) + } + + switch( + name, + binomial = "binary:logistic", + poisson = "count:poisson", + Gamma = "reg:gamma", + gaussian = "reg:squarederror", + stop( + "The xgboost engine has no objective for family `", name, + "`. It supports binomial, poisson, Gamma and gaussian.", + call. = FALSE + ) + ) +} + +## 5.4 xgboost_best_rounds() ---- + +#' The Number of Rounds Early Stopping Chose +#' +#' @param model An xgb.Booster trained with early stopping. +#' @param most Integer. The ceiling, used when none was recorded. +#' @return An integer of at least 1. +#' +#' @example # Example usage of the function +#' # xgboost_best_rounds(trial, 1000) +xgboost_best_rounds <- function(model, most) { + best <- suppressWarnings(as.integer( + xgboost::xgb.attr(model, "best_iteration") + )) + + if (length(best) != 1L || is.na(best)) { + best <- most + } + + max(1L, best) +} + +# 6. Register ---- +register_engine(engine_xgboost()) + +# End of script ---- diff --git a/1_code/methods/metrics/auc.R b/1_code/methods/metrics/auc.R new file mode 100644 index 0000000..a7ffb0c --- /dev/null +++ b/1_code/methods/metrics/auc.R @@ -0,0 +1,54 @@ +# --- +# title: Metric - Area Under the ROC Curve +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `auc` metric +# notes: +# - Metric contract: see register_metric() (harness/registry.R). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. metric_auc() ---- + +#' Area Under the ROC Curve +#' +#' Mann-Whitney form (ties count a half). Values above zero are +#' detections, so counts and densities score like presences. +#' +#' @param observed,predicted Numeric vectors. +#' @return A number between 0 and 1, or NA when the observations +#' are all detections or all non-detections. +#' +#' @example # Example usage of the function +#' # metric_auc(c(0, 0, 1, 1), c(0.1, 0.3, 0.4, 0.9)) +metric_auc <- function(observed, predicted) { + positive <- observed > 0 + + # Both classes are needed; a draw without detections has none + if (!any(positive) || all(positive)) { + return(NA_real_) + } + + ranks <- rank(predicted, ties.method = "average") + + # Doubles, not integers: at bird scale the product of the class + # counts (~3.3e9) overflows 32-bit integers to NA + n_pos <- as.numeric(sum(positive)) + n_neg <- as.numeric(sum(!positive)) + + (sum(ranks[positive]) - n_pos * (n_pos + 1) / 2) / + (n_pos * n_neg) +} + +# 3. Register ---- +register_metric( + "auc", metric_auc, + "Area under the ROC curve (Mann-Whitney, ties count a half)" +) + +# End of script ---- diff --git a/1_code/methods/metrics/calibration_slope.R b/1_code/methods/metrics/calibration_slope.R new file mode 100644 index 0000000..fe181dc --- /dev/null +++ b/1_code/methods/metrics/calibration_slope.R @@ -0,0 +1,42 @@ +# --- +# title: Metric - Calibration Slope +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `calibration_slope` metric +# notes: +# - Metric contract: see register_metric() (harness/registry.R). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. metric_calibration_slope() ---- + +#' Calibration Slope +#' +#' Slope of observed on predicted. One is perfect; below one +#' means predictions are too extreme (typical of overfitting). +#' +#' @param observed,predicted Numeric vectors. +#' @return A number. +#' +#' @example # Example usage of the function +#' # metric_calibration_slope(c(0, 1, 1), c(0.2, 0.6, 0.8)) +metric_calibration_slope <- function(observed, predicted) { + if (stats::sd(predicted) == 0) { + return(NA_real_) + } + + unname(stats::coef(stats::lm(observed ~ predicted))[2]) +} + +# 3. Register ---- +register_metric( + "calibration_slope", metric_calibration_slope, + "Slope of observed on predicted; 1 is well calibrated" +) + +# End of script ---- diff --git a/1_code/methods/metrics/deviance_explained.R b/1_code/methods/metrics/deviance_explained.R new file mode 100644 index 0000000..cc12998 --- /dev/null +++ b/1_code/methods/metrics/deviance_explained.R @@ -0,0 +1,60 @@ +# --- +# title: Metric - Deviance Explained +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `deviance_explained` metric +# notes: +# - Metric contract: see register_metric() (harness/registry.R). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. metric_deviance_explained() ---- + +#' Proportion of Null Deviance Explained +#' +#' Poisson deviance for counts, Bernoulli for detections, chosen +#' from whether the observations are all zero or one. +#' +#' @param observed,predicted Numeric vectors. +#' @return A number, or NA when the null deviance is zero. +#' +#' @example # Example usage of the function +#' # metric_deviance_explained(c(0, 1, 2), c(0.2, 0.9, 1.8)) +metric_deviance_explained <- function(observed, predicted) { + binary <- all(observed %in% c(0, 1)) + + deviance <- function(y, mu) { + mu <- pmax(mu, 1e-10) + + if (binary) { + mu <- pmin(mu, 1 - 1e-10) + -2 * sum(y * log(mu) + (1 - y) * log(1 - mu)) + } else { + terms <- ifelse(y > 0, y * log(y / mu), 0) + 2 * sum(terms - (y - mu)) + } + } + + null_deviance <- deviance( + observed, rep(mean(observed), length(observed)) + ) + + if (!is.finite(null_deviance) || null_deviance <= 0) { + return(NA_real_) + } + + 1 - deviance(observed, predicted) / null_deviance +} + +# 3. Register ---- +register_metric( + "deviance_explained", metric_deviance_explained, + "Share of null deviance explained (Bernoulli or Poisson)" +) + +# End of script ---- diff --git a/1_code/methods/metrics/n.R b/1_code/methods/metrics/n.R new file mode 100644 index 0000000..6239db4 --- /dev/null +++ b/1_code/methods/metrics/n.R @@ -0,0 +1,35 @@ +# --- +# title: Metric - Units Scored +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `n` metric +# notes: +# - Metric contract: see register_metric() (harness/registry.R). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. metric_n() ---- + +#' Number of Survey Units Scored +#' +#' @param observed,predicted Numeric vectors. +#' @return An integer. +#' +#' @example # Example usage of the function +#' # metric_n(c(0, 1), c(0.2, 0.7)) +metric_n <- function(observed, predicted) { + length(observed) +} + +# 3. Register ---- +register_metric( + "n", metric_n, + "Number of units scored" +) + +# End of script ---- diff --git a/1_code/methods/metrics/prevalence.R b/1_code/methods/metrics/prevalence.R new file mode 100644 index 0000000..b133506 --- /dev/null +++ b/1_code/methods/metrics/prevalence.R @@ -0,0 +1,38 @@ +# --- +# title: Metric - Prevalence +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `prevalence` metric +# notes: +# - Metric contract: see register_metric() (harness/registry.R). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. metric_prevalence() ---- + +#' Proportion of Survey Units with a Detection +#' +#' Reported because most scores are only interpretable against +#' it. +#' +#' @param observed,predicted Numeric vectors. +#' @return A number between 0 and 1. +#' +#' @example # Example usage of the function +#' # metric_prevalence(c(0, 0, 1), c(0, 0, 0)) +metric_prevalence <- function(observed, predicted) { + mean(observed > 0) +} + +# 3. Register ---- +register_metric( + "prevalence", metric_prevalence, + "Proportion of units with a detection" +) + +# End of script ---- diff --git a/1_code/methods/metrics/rmse.R b/1_code/methods/metrics/rmse.R new file mode 100644 index 0000000..eb1d8d6 --- /dev/null +++ b/1_code/methods/metrics/rmse.R @@ -0,0 +1,35 @@ +# --- +# title: Metric - Root Mean Squared Error +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `rmse` metric +# notes: +# - Metric contract: see register_metric() (harness/registry.R). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. metric_rmse() ---- + +#' Root Mean Squared Error +#' +#' @param observed,predicted Numeric vectors. +#' @return A number. +#' +#' @example # Example usage of the function +#' # metric_rmse(c(0, 1), c(0.1, 0.8)) +metric_rmse <- function(observed, predicted) { + sqrt(mean((observed - predicted)^2)) +} + +# 3. Register ---- +register_metric( + "rmse", metric_rmse, + "Root mean squared error" +) + +# End of script ---- diff --git a/1_code/methods/metrics/spearman.R b/1_code/methods/metrics/spearman.R new file mode 100644 index 0000000..97edab2 --- /dev/null +++ b/1_code/methods/metrics/spearman.R @@ -0,0 +1,41 @@ +# --- +# title: Metric - Spearman Correlation +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `spearman` metric +# notes: +# - Metric contract: see register_metric() (harness/registry.R). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. metric_spearman() ---- + +#' Rank Correlation of Observed and Predicted +#' +#' Scale-free, so density and count models compare directly. +#' +#' @param observed,predicted Numeric vectors. +#' @return A number between -1 and 1. +#' +#' @example # Example usage of the function +#' # metric_spearman(c(0, 1, 2), c(0.1, 0.5, 0.9)) +metric_spearman <- function(observed, predicted) { + if (stats::sd(observed) == 0 || stats::sd(predicted) == 0) { + return(NA_real_) + } + + stats::cor(observed, predicted, method = "spearman") +} + +# 3. Register ---- +register_metric( + "spearman", metric_spearman, + "Rank correlation of observed and predicted" +) + +# End of script ---- diff --git a/1_code/methods/resampling/precomputed.R b/1_code/methods/resampling/precomputed.R new file mode 100644 index 0000000..8f104f2 --- /dev/null +++ b/1_code/methods/resampling/precomputed.R @@ -0,0 +1,190 @@ +# --- +# title: Resampling Scheme - Precomputed Draws +# author: Brendan Casey +# created: 2026-09-09 +# inputs: +# in the test dataset (see harness/data_source.R), where +# lookup/dataset_manifest.csv says (`bootstrap_ids`, +# `bootstrap_layout`): +# - per_taxon_csv: one table, one column per draw (birds) +# - per_species_rds: one file per species (plant groups) +# outputs: registers the `precomputed` scheme +# notes: +# - Replays v2's stored draws, so parity can be numerical rather +# than distributional. Birds: one table keyed on `surveyid`. +# Plant groups: per species, as v2 drew them (written by +# _setup/07_harmonize_v2_plant_bootstrap_ids.R). +# - The seed is ignored. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # lookup table reading (version: 1.16.4) + +# 2. resample_precomputed() ---- + +#' Read Bootstrap Ids the Source Pipeline Generated +#' +#' @param data_dir Character. The test dataset folder. +#' @param taxon Character. Taxon slug. +#' @param iterations Integer vector of bootstrap iterations. +#' @param id_column Character. The column in the covariate frame +#' the stored ids refer to, when they are not survey unit ids. +#' @param frame A data frame carrying `survey_unit_id` and +#' `id_column`, used to translate stored ids into survey unit +#' ids. NULL when the stored ids are already unit ids. +#' @param species Character. The species, for per-species ids. +#' @param per_species Logical. Read the per-species layout. +#' @param unique_ids Logical. Keep each unit once per draw. v2's +#' bird scripts select a draw with `%in%`, so a unit drawn twice +#' is fitted once; the plant scripts index rows, so repeats +#' count. +#' @return A list of character vectors, one per iteration. +#' +#' @example # Example usage of the function +#' # ids <- resample_precomputed(data_dir, "bird", 1:3, +#' # id_column = "surveyid", +#' # frame = frame, unique_ids = TRUE) +resample_precomputed <- function( + data_dir, + taxon, + iterations, + id_column = NULL, + frame = NULL, + species = NULL, + per_species = FALSE, + unique_ids = FALSE +) { + # Step 1: Read the stored draws, as a list of id vectors + stored <- read_stored_draws(data_dir, taxon, species, per_species) + + if (max(iterations) > length(stored)) { + stop( + taxon, " has ", length(stored), " precomputed bootstrap ", + "iterations; ", max(iterations), " were requested.", + call. = FALSE + ) + } + + lapply(iterations, function(i) { + ids <- stored[[i]] + ids <- ids[!is.na(ids)] + + if (unique_ids) { + ids <- unique(ids) + } + + # Step 2: Translate other keys (birds) to survey unit ids + if (is.null(id_column)) { + return(as.character(ids)) + } + + if (is.null(frame) || !id_column %in% names(frame)) { + stop( + "Translating precomputed ids needs a frame carrying `", + id_column, "`.", + call. = FALSE + ) + } + + matched <- match(ids, frame[[id_column]]) + as.character(frame$survey_unit_id[matched[!is.na(matched)]]) + }) +} + +## 2.1 read_stored_draws() ---- + +#' Read One Taxon's or Species' Stored Draws +#' +#' Cached: the bird draw table (~38 MB) is otherwise re-read per +#' species. +#' +#' @param data_dir,taxon,species,per_species As in +#' resample_precomputed(). +#' @return A list of id vectors, one per stored draw. +#' +#' @example # Example usage of the function +#' # length(read_stored_draws(data_dir, "bird", NULL, FALSE)) +read_stored_draws <- function(data_dir, taxon, species, per_species) { + location <- manifest_entry(data_dir, taxon)$bootstrap_ids + + if (is.na(location)) { + stop( + "The dataset manifest records no stored bootstrap draws ", + "for ", taxon, ". Use a scheme that draws them, such as ", + "spatial_block.", + call. = FALSE + ) + } + + if (per_species) { + path <- file.path(data_dir, location, paste0(species, ".rds")) + + if (is.null(species) || !file.exists(path)) { + stop( + "No stored v2 bootstrap ids for ", taxon, " ", + species, ":\n ", path, + "\nRun 1_code/_setup/07_harmonize_v2_plant_bootstrap_ids.R", + " for this species, or use the spatial_block scheme.", + call. = FALSE + ) + } + + stored <- readRDS(path) + + # v2 stores a species' draws as a matrix, units by draws + if (is.matrix(stored)) { + stored <- lapply(seq_len(ncol(stored)), function(j) stored[, j]) + } + + return(stored) + } + + path <- file.path(data_dir, location) + + if (!file.exists(path)) { + stop( + "No precomputed bootstrap ids for ", taxon, ":\n ", path, + "\nUse a different resampling scheme, or generate them.", + call. = FALSE + ) + } + + cached_read(path, function(p) as.list(fread(p))) +} + +# 3. resampler_precomputed() ---- + +#' The Registered Form of the Precomputed Scheme +#' +#' @param frame The one-species frame. +#' @param iterations Integer vector. +#' @param seed Ignored; the draws are stored. +#' @param context List with `data_dir`, `taxon` and `species`. +#' @param id_column,per_species,unique_ids Settings from the +#' spec's `resample` list; see resample_precomputed(). +#' @return A list of character vectors of survey unit ids. +#' +#' @example # Example usage of the function +#' # resampler_precomputed(frame, 1:5, NULL, context, +#' # id_column = "surveyid") +resampler_precomputed <- function( + frame, iterations, seed, context, + id_column = NULL, per_species = FALSE, unique_ids = FALSE +) { + resample_precomputed( + data_dir = context$data_dir, taxon = context$taxon, + iterations = iterations, id_column = id_column, frame = frame, + species = context$species, per_species = per_species, + unique_ids = unique_ids + ) +} + +# 4. Register ---- +register_resampler( + "precomputed", resampler_precomputed, + "Replay draws stored with the dataset (v2 birds; plant v2_ids)" +) + +# End of script ---- diff --git a/1_code/methods/resampling/spatial_block.R b/1_code/methods/resampling/spatial_block.R new file mode 100644 index 0000000..19717e0 --- /dev/null +++ b/1_code/methods/resampling/spatial_block.R @@ -0,0 +1,169 @@ +# --- +# title: Resampling Scheme - Spatially Blocked Bootstrap +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `spatial_block` scheme +# notes: +# - The v2 plant bootstrap: resample units with replacement +# within coarse lat/long blocks, keeping the geographic spread. +# - Iteration 1 is the complete data, as in v2. +# - One seeded stream per species, so draws 1-5 of a 5-draw run +# equal draws 1-5 of a 100-draw run. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. resample_spatial_block() ---- + +#' Draw a Spatially Blocked Bootstrap +#' +#' Each block keeps its original number of units. +#' +#' @param frame A data frame with `survey_unit_id` and the +#' coordinate columns. +#' @param iterations Integer vector of bootstrap iterations. +#' @param long_column,lat_column Character. Coordinate columns. +#' @param min_detections Integer. Redraw a block set until at +#' least this many detections are present, or 0 to skip the +#' check. +#' @param response Numeric vector, one per row of `frame`, used +#' only for the detection check. +#' @param max_attempts Integer. Redraws before failing. +#' @param seed Integer or NULL. NULL leaves the stream unseeded, +#' which is what v2 did. +#' @return A list of character vectors, one per iteration. +#' +#' @example # Example usage of the function +#' # ids <- resample_spatial_block(frame, 1:5, seed = 42) +resample_spatial_block <- function( + frame, + iterations, + long_column = "long", + lat_column = "lat", + min_detections = 0L, + response = NULL, + max_attempts = 100L, + seed = harness_seed() +) { + for (one in c(long_column, lat_column)) { + if (!one %in% names(frame)) { + stop( + "Spatial blocking needs `", one, "` in the frame.", + call. = FALSE + ) + } + } + + units <- as.character(frame$survey_unit_id) + blocks <- spatial_blocks( + frame[[long_column]], frame[[lat_column]] + ) + by_block <- split(seq_along(units), blocks) + + draw_once <- function() { + unlist( + lapply( + by_block, + function(rows) sample(rows, length(rows), replace = TRUE) + ), + use.names = FALSE + ) + } + + with_seed(seed, lapply(iterations, function(i) { + # Step 1: Iteration 1 is the complete data, as in v2 + if (i == 1L) { + return(units) + } + + # Step 2: Redraw until the sample has enough detections + for (attempt in seq_len(max_attempts)) { + rows <- draw_once() + + if (min_detections <= 0L || is.null(response)) { + return(units[rows]) + } + + if (sum(response[rows] > 0, na.rm = TRUE) >= min_detections) { + return(units[rows]) + } + } + + stop( + "No spatially blocked draw reached ", min_detections, + " detections in ", max_attempts, " attempts. The species ", + "is probably too rare to bootstrap at this threshold.", + call. = FALSE + ) + })) +} + +# 3. resampler_spatial_block() ---- + +#' The Registered Form of the Spatially Blocked Bootstrap +#' +#' Adds v2's redraw rule: redraw for detections only if the full +#' data meets the threshold in at least one region; otherwise +#' take the first draw. +#' +#' @param frame The one-species frame. +#' @param iterations Integer vector. +#' @param seed Integer or NULL. This species' seed. +#' @param context List with `spec` and `response_name`. +#' @param min_detections,long_column,lat_column,max_attempts +#' Settings from the spec's `resample` list. +#' @return A list of character vectors of survey unit ids. +#' +#' @example # Example usage of the function +#' # resampler_spatial_block(frame, 1:5, 42L, context, +#' # min_detections = 20L) +resampler_spatial_block <- function( + frame, iterations, seed, context, + min_detections = 0L, long_column = "long", lat_column = "lat", + max_attempts = 100L +) { + response <- frame[[context$response_name]] + + if (min_detections > 0) { + viable <- any(vapply(context$spec$regions, function(region) { + # Skip regions whose filter columns this frame lacks + if (!is.null(region$filter) && + !all(all.vars(region$filter) %in% names(frame))) { + return(FALSE) + } + + in_region <- if (is.null(region$filter)) { + rep(TRUE, nrow(frame)) + } else { + keep <- eval(rlang_f_rhs(region$filter), frame) + !is.na(keep) & keep + } + + sum(response[in_region] > 0, na.rm = TRUE) >= min_detections + }, logical(1))) + + if (!viable) { + min_detections <- 0L + } + } + + # Pass NULL through: unseeded, not the harness_seed() default + resample_spatial_block( + frame = frame, iterations = iterations, + long_column = long_column, lat_column = lat_column, + min_detections = min_detections, response = response, + max_attempts = max_attempts, seed = seed + ) +} + +# 4. Register ---- +register_resampler( + "spatial_block", resampler_spatial_block, + "Bootstrap within coarse spatial blocks (v2 plants, mammals)" +) + +# End of script ---- diff --git a/1_code/methods/resampling/spatial_cv.R b/1_code/methods/resampling/spatial_cv.R new file mode 100644 index 0000000..2d63b3f --- /dev/null +++ b/1_code/methods/resampling/spatial_cv.R @@ -0,0 +1,105 @@ +# --- +# title: Resampling Scheme - Spatial Cross-Validation +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `spatial_cv` scheme +# notes: +# - Holds out whole spatial blocks, so held-out scores are not +# inflated by spatial autocorrelation. No v2 equivalent. +# - One iteration per fold; each draw is the fold's training +# units, so `oob_` metrics score its test units. Set +# `n_bootstraps` to the number of folds. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. resample_spatial_cv() ---- + +#' Split Survey Units into Spatial Cross-Validation Folds +#' +#' @param frame A data frame with `survey_unit_id` and the +#' coordinate columns. +#' @param folds Integer. Number of folds. +#' @param long_column,lat_column Character. Coordinate columns. +#' @param seed Integer or NULL. Defaults to harness_seed(). +#' @return A list of lists, each with `train` and `test` +#' character vectors of survey unit ids. +#' +#' @example # Example usage of the function +#' # cv <- resample_spatial_cv(frame, folds = 5, seed = 42) +#' # length(cv[[1]]$train) +resample_spatial_cv <- function( + frame, + folds = 5L, + long_column = "long", + lat_column = "lat", + seed = harness_seed() +) { + units <- as.character(frame$survey_unit_id) + blocks <- spatial_blocks( + frame[[long_column]], frame[[lat_column]] + ) + present <- unique(blocks[!is.na(blocks)]) + + if (length(present) < folds) { + stop( + "Only ", length(present), " spatial block(s) present; ", + folds, " folds were asked for. Use finer block breaks or ", + "fewer folds.", + call. = FALSE + ) + } + + # Step 1: Deal whole blocks into folds + assignment <- with_seed( + seed, sample(rep_len(seq_len(folds), length(present))) + ) + fold_of_block <- assignment[match(blocks, present)] + + lapply(seq_len(folds), function(k) { + list( + train = units[!is.na(fold_of_block) & fold_of_block != k], + test = units[!is.na(fold_of_block) & fold_of_block == k] + ) + }) +} + +# 3. resampler_spatial_cv() ---- + +#' The Registered Form of Spatial Cross-Validation +#' +#' @param frame The one-species frame. +#' @param iterations Integer vector; its length is the number of +#' folds. +#' @param seed Integer or NULL. +#' @param context Unused. +#' @param long_column,lat_column Settings from the spec's +#' `resample` list. +#' @return A list of character vectors: each fold's training +#' units. +#' +#' @example # Example usage of the function +#' # resampler_spatial_cv(frame, 1:5, 42L, list()) +resampler_spatial_cv <- function( + frame, iterations, seed, context, + long_column = "long", lat_column = "lat" +) { + splits <- resample_spatial_cv( + frame, folds = length(iterations), long_column = long_column, + lat_column = lat_column, seed = seed + ) + + lapply(splits, `[[`, "train") +} + +# 4. Register ---- +register_resampler( + "spatial_cv", resampler_spatial_cv, + "Spatial block cross-validation; each draw holds out one fold" +) + +# End of script ---- diff --git a/1_code/methods/selection/aic_average.R b/1_code/methods/selection/aic_average.R new file mode 100644 index 0000000..503c540 --- /dev/null +++ b/1_code/methods/selection/aic_average.R @@ -0,0 +1,103 @@ +# --- +# title: Selection Rule - Information-Criterion Model Averaging +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `aic_average` rule +# notes: +# - The v2 plant and bird climate rule: Akaike-weighted +# coefficients, absent terms counting as zero (MuMIn's "full" +# average). The predictor averages link predictions, which +# equals applying the averaged coefficients. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. select_aic_average() ---- + +#' Average Candidates by Information-Criterion Weight +#' +#' @inheritParams select_single +#' @return A selection result. `fit` is the best single candidate; +#' `coefficients` are the averaged ones; `predict` is the +#' weighted average of the candidates. +#' +#' @example # Example usage of the function +#' # select_aic_average(models, response ~ 1, d, engine, +#' # "binomial") +select_aic_average <- function( + models, base, data, engine, family, + weights = NULL, offset = NULL, ic = "AICc", control = list() +) { + fitted <- fit_candidates( + models, base, data, engine, family, weights, offset, ic, + control + ) + + table <- candidate_table(fitted) + + if (!any(table$ok) || sum(table$weight) == 0) { + return(selection_result(NULL, NULL, table, fitted)) + } + + # Step 1: Collect every term any candidate estimated + per_model <- lapply(fitted$fits, function(f) engine_coef(engine, f)) + terms <- unique(unlist(lapply(per_model, function(x) x$term))) + + # Step 2: Weighted mean per term, absent terms counting as zero + averaged <- do.call(rbind, lapply(terms, function(term) { + estimate <- 0 + variance <- 0 + + for (i in seq_along(per_model)) { + cf <- per_model[[i]] + w <- table$weight[i] + + if (is.null(cf) || w == 0) { + next + } + + # Absent or aliased (NA) terms count as zero + row <- match(term, cf$term) + absent <- is.na(row) || is.na(cf$estimate[row]) + value <- if (absent) 0 else cf$estimate[row] + se <- if (absent) 0 else cf$se[row] + + estimate <- estimate + w * value + variance <- variance + w * ifelse(is.na(se), 0, se^2) + } + + data.frame( + term = term, + estimate = estimate, + se = sqrt(variance), + stringsAsFactors = FALSE + ) + })) + + selection_result( + fit = fitted$fits[[which.min(fitted$scores)]], + coefficients = averaged, + ic_table = table, + fitted = fitted, + engine = engine, + predict = combined_predictor( + fitted$fits, engine, family, weights = table$weight + ) + ) +} + +# 3. Register ---- +register_selection( + "aic_average", select_aic_average, + paste( + "Average coefficients over candidates by Akaike weight", + "(v2 climate)" + ), + requires = c("ic", "coefficients") +) + +# End of script ---- diff --git a/1_code/methods/selection/aic_best.R b/1_code/methods/selection/aic_best.R new file mode 100644 index 0000000..f24c29d --- /dev/null +++ b/1_code/methods/selection/aic_best.R @@ -0,0 +1,59 @@ +# --- +# title: Selection Rule - Best by Information Criterion +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `aic_best` rule +# notes: +# - Keeps the single candidate with the lowest score (AICc by +# default; the stage's `ic` sets it). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. select_aic_best() ---- + +#' Keep the Single Best-Scoring Candidate +#' +#' @inheritParams select_single +#' @return A selection result. +#' +#' @example # Example usage of the function +#' # select_aic_best(models, response ~ 1, d, engine, "binomial") +select_aic_best <- function( + models, base, data, engine, family, + weights = NULL, offset = NULL, ic = "AICc", control = list() +) { + fitted <- fit_candidates( + models, base, data, engine, family, weights, offset, ic, + control + ) + + table <- candidate_table(fitted) + + if (!any(table$ok)) { + return(selection_result(NULL, NULL, table, fitted)) + } + + winner <- which.min(fitted$scores) + + selection_result( + fit = fitted$fits[[winner]], + coefficients = engine_coef(engine, fitted$fits[[winner]]), + ic_table = table, + fitted = fitted, + engine = engine + ) +} + +# 3. Register ---- +register_selection( + "aic_best", select_aic_best, + "Keep the lowest-scoring candidate (AICc, AIC or BIC)", + requires = "ic" +) + +# End of script ---- diff --git a/1_code/methods/selection/hurdle.R b/1_code/methods/selection/hurdle.R new file mode 100644 index 0000000..d1b7746 --- /dev/null +++ b/1_code/methods/selection/hurdle.R @@ -0,0 +1,244 @@ +# --- +# title: Selection Rule - Hurdle: Presence, Then Abundance Given +# Presence +# author: Brendan Casey +# created: 2026-10-05 +# inputs: none +# outputs: registers the `hurdle` rule +# notes: +# - v2's mammal habitat shape: presence on every unit, abundance +# given presence where detected; their product is total +# abundance. Both halves use the stage's engine via the inner +# `part_selection` rule, so any engine works. +# - Outputs are each half and the product on the grid, written +# as `_presence`, `_abundance` and `_total` +# (modules/mammals/hurdle.R swaps in v2's tables for v2 GLMs). +# The scored final model is the presence half. +# - Abundance candidates are the presence ones minus +# `abundance_drop` (v2: climate effects on abundance are +# minimal) plus a null. With coefficients, a candidate whose +# cover terms sum to all the cover is dropped, as in v2: it is +# not identifiable with an intercept. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. select_hurdle() ---- + +#' Fit a Hurdle Model: Presence, Then Abundance Given Presence +#' +#' @inheritParams select_single +#' @param grid A prediction grid, or NULL; without one there are +#' no outputs, only the presence half's coefficients. +#' @param constants,grid_constants Named lists of values held +#' fixed on the grid: sampling effort, Climate, pAspen. Their +#' names are also the terms that are not cover. +#' @param intercept_cats Character vector, parallel to `models`, +#' naming each candidate's reference category, or NULL. +#' @param part_selection Character. The rule each half runs. +#' @param abundance_family A family for the abundance half. +#' @param abundance_response A function(values, data) computing +#' the abundance response from the raw values, or NULL to take +#' the raw values as they are. +#' @param abundance_drop Character vector of terms the abundance +#' half leaves out. +#' @param abundance_null Character. An extra abundance candidate, +#' or NULL for none. +#' @param abundance_weighted Logical. Weight the abundance half +#' as the presence half is weighted. v2 does not. +#' @return A selection result for the presence half, with `parts` +#' (both halves' selection results), `winner` (the presence +#' candidate chosen), `reference_category` and `outputs`. +#' +#' @example # Example usage of the function +#' # select_hurdle(models, response ~ 1, d, get_engine("glm"), +#' # "binomial", grid = grid) +select_hurdle <- function( + models, base, data, engine, family, + weights = NULL, offset = NULL, ic = "AICc", control = list(), + grid = NULL, constants = list(), grid_constants = list(), + intercept_cats = NULL, + part_selection = "aic_best", + abundance_family = stats::Gamma(link = "log"), + abundance_response = NULL, + abundance_drop = "Climate", + abundance_null = ". ~ . + seas_days", + abundance_weighted = FALSE +) { + response <- all.vars(base[[2]]) + raw <- if ("response_raw" %in% names(data)) { + data$response_raw + } else { + data[[response]] + } + failed <- function(table) { + selection_result(NULL, NULL, table) + } + + # Step 1: Presence, on every unit + presence <- selection_run( + part_selection, models, base, data, engine, family, + weights, offset, ic, control + ) + + if (is.null(presence$fit) || !isTRUE(presence$fit$ok)) { + return(failed(presence$ic_table)) + } + + scores <- presence$ic_table$ic + winner <- if (any(is.finite(scores))) which.min(scores) else 1L + + # Step 2: Abundance given presence, on the units with any + present <- !is.na(raw) & raw > 0 + + if (sum(present) < 2) { + return(failed(presence$ic_table)) + } + + d_p <- data[present, , drop = FALSE] + d_p[[response]] <- if (is.function(abundance_response)) { + abundance_response(raw[present], d_p) + } else { + raw[present] + } + + abundance_models <- hurdle_abundance_models( + presence, models, base, d_p, engine, abundance_drop, + abundance_null, held = names(c(constants, grid_constants)) + ) + + abundance <- selection_run( + part_selection, abundance_models, base, d_p, engine, + abundance_family, + weights = if (abundance_weighted && !is.null(weights)) { + weights[present] + } else { + NULL + }, + offset = NULL, ic = ic, control = control + ) + + if (is.null(abundance$fit) || !isTRUE(abundance$fit$ok)) { + return(failed(presence$ic_table)) + } + + # Step 3: Each half, and their product, on the grid + result <- presence + result$parts <- list(presence = presence, abundance = abundance) + result$winner <- winner + result$reference_category <- if (is.null(intercept_cats)) { + NULL + } else { + intercept_cats[winner] + } + + if (!is.null(grid)) { + held <- c(constants, grid_constants) + on_grid <- lapply(result$parts, predict_grid, grid = grid, + constants = held) + + if (!any(vapply(on_grid, is.null, logical(1)))) { + table_of <- function(values) { + data.frame( + term = names(values), estimate = unname(values), + se = NA_real_, stringsAsFactors = FALSE + ) + } + + result$outputs <- list( + presence = table_of(on_grid$presence), + abundance = table_of(on_grid$abundance), + total = table_of(on_grid$presence * on_grid$abundance) + ) + } + } + + result +} + +# 3. hurdle_abundance_models() ---- + +#' The Abundance Half's Candidates +#' +#' @param presence The presence half's selection result. +#' @param models The presence candidates as given. +#' @param base The base formula. +#' @param d_p The units where the species was present. +#' @param engine An engine definition. +#' @param drop,null As `abundance_drop` and `abundance_null`. +#' @param held Character vector of terms that are not cover. +#' @return A list of formulas. +#' +#' @example # Example usage of the function +#' # hurdle_abundance_models(presence, models, response ~ 1, d_p, +#' # engine, "Climate", NULL, "seas_days") +hurdle_abundance_models <- function(presence, models, base, d_p, + engine, drop, null, held) { + response <- all.vars(base[[2]]) + base_text <- paste(deparse(base[[3]]), collapse = " ") + has_offset <- grepl("offset(offset)", base_text, fixed = TRUE) + + # Only fitted presence candidates get an abundance counterpart, + # as in v2; `single` fits only the first + table <- presence$ic_table + formulas <- lapply(models, resolve_formula, base = base) + + fitted_n <- if (is.null(table)) 1L else nrow(table) + + formulas <- if (fitted_n == length(formulas)) { + formulas[table$ok] + } else { + formulas[seq_len(min(fitted_n, length(formulas)))] + } + + check_cover <- engine_has(engine, "coefficients") + + candidates <- lapply(formulas, function(f) { + terms <- attr(stats::terms(f), "term.labels") + + # Identifiable with an intercept only if some cover is left + # unaccounted for at some unit + if (check_cover) { + cover <- intersect(setdiff(terms, held), names(d_p)) + used <- sum(colSums(d_p[, cover, drop = FALSE])) + + if ((nrow(d_p) - used) <= 0) { + return(NULL) + } + } + + kept <- setdiff(terms, drop) + + stats::as.formula(paste( + response, "~", + paste(c(if (length(kept) == 0) "1" else kept, + if (has_offset) "offset(offset)"), + collapse = " + ") + )) + }) + + candidates <- Filter(Negate(is.null), candidates) + + if (!is.null(null)) { + candidates <- c( + candidates, list(resolve_formula(stats::as.formula(null), base)) + ) + } + + candidates +} + +# 4. Register ---- +register_selection( + "hurdle", select_hurdle, + paste( + "Presence, then abundance given presence, each with an inner", + "rule (v2 mammals)" + ), + requires = character(0) +) + +# End of script ---- diff --git a/1_code/methods/selection/ivw_grid.R b/1_code/methods/selection/ivw_grid.R new file mode 100644 index 0000000..d540db7 --- /dev/null +++ b/1_code/methods/selection/ivw_grid.R @@ -0,0 +1,181 @@ +# --- +# title: Selection Rule - Inverse-Variance Averaging onto a Grid +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `ivw_grid` rule +# notes: +# - The v2 plant habitat rule. Candidates are predicted onto the +# grid and combined per habitat type by inverse variance, so a +# "coefficient" is a link-scale effect per habitat type. +# - Unlike AICc weighting, a candidate confident about one +# habitat type dominates there even if it is poor overall. +# - Non-converged candidates are dropped from the grid average +# (as v2). Zero SEs are floored at 1e-4. +# - `predict()` gives v2's site-level `data$prediction` (offset +# of the stand-age splines): every fitted candidate, converged +# or not, as v2; failed fits are left out (v2 would give NaN). +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. select_ivw_grid() ---- + +#' Inverse-Variance Average Predictions onto a Grid +#' +#' @inheritParams select_single +#' @param grid A prediction grid, one row per habitat type. +#' @param grid_constants Named list of values for terms the grid +#' has no column for - v2 predicts at `Climate = 0` and, where +#' Protocol is fitted, at the new protocol. +#' @param head_terms Character vector of leading model +#' coefficients to average separately and prepend, which is how +#' v2 carries Intercept, Climate and Protocol. +#' @return A selection result whose `coefficients` are the +#' averaged per-habitat-type effects, with +#' `candidate_coefficients` for steps that average a term the +#' grid does not carry. +#' +#' @example # Example usage of the function +#' # select_ivw_grid(models, response ~ 1, d, engine, "binomial", +#' # grid = veg_grid, +#' # grid_constants = list(Climate = 0)) +select_ivw_grid <- function( + models, base, data, engine, family, + weights = NULL, offset = NULL, ic = "AICc", control = list(), + grid = NULL, grid_constants = list(), + head_terms = c("Intercept", "Climate") +) { + if (is.null(grid)) { + stop( + "The ivw_grid rule needs a prediction grid; the spec's ", + "region defines one with `grid`.", + call. = FALSE + ) + } + + fitted <- fit_candidates( + models, base, data, engine, family, weights, offset, ic, + control + ) + + table <- candidate_table(fitted) + converged <- vapply( + fitted$fits, function(f) engine_converged(engine, f), logical(1) + ) + + if (!any(converged)) { + return(selection_result(NULL, NULL, table, fitted)) + } + + # Step 1: Predict every converged candidate onto the grid, at + # the constants v2 holds the non-habitat terms at + newdata <- grid + + for (one in names(grid_constants)) { + newdata[[one]] <- grid_constants[[one]] + } + + predictions <- lapply(seq_along(fitted$fits), function(i) { + if (!converged[i]) { + return(NULL) + } + + engine$predict(fitted$fits[[i]], newdata, "link", se = TRUE) + }) + + usable <- !vapply(predictions, is.null, logical(1)) + + if (!any(usable)) { + return(selection_result(NULL, NULL, table, fitted)) + } + + estimate <- do.call( + rbind, lapply(predictions[usable], function(x) x$fit) + ) + error <- do.call( + rbind, lapply(predictions[usable], function(x) x$se.fit) + ) + + error[error == 0] <- 1e-4 + + grid_coefficients <- ivw_combine(estimate, error) + + # Step 2: Head terms, from each model's coefficient table, read + # by position as v2 does (they lead every formula) + head_estimate <- do.call(rbind, lapply( + which(converged), function(i) { + fitted$fits[[i]]$coefficients$estimate[seq_along(head_terms)] + } + )) + head_error <- do.call(rbind, lapply( + which(converged), function(i) { + fitted$fits[[i]]$coefficients$se[seq_along(head_terms)] + } + )) + head_error[head_error == 0 | is.na(head_error)] <- 1e-4 + + head_coefficients <- ivw_combine(head_estimate, head_error) + + coefficients <- data.frame( + term = c(head_terms, rownames(grid)), + estimate = c(head_coefficients$estimate, + grid_coefficients$estimate), + se = c(head_coefficients$se, grid_coefficients$se), + stringsAsFactors = FALSE + ) + + result <- selection_result( + fit = fitted$fits[[which.min(fitted$scores)]], + coefficients = coefficients, + ic_table = table, + fitted = fitted, + engine = engine, + # Step 3: The final model, averaged per site the same way + predict = combined_predictor( + fitted$fits[table$ok], engine, family + ) + ) + + # For steps averaging a term the grid lacks (v2's pAspen) + result$candidate_coefficients <- lapply( + which(table$ok), function(i) fitted$fits[[i]]$coefficients + ) + + result +} + +# 3. ivw_combine() ---- + +#' Combine Estimates by Inverse-Variance Weight +#' +#' @param estimate Matrix of candidates by columns. +#' @param error Matrix of standard errors, the same shape. +#' @return A list of `estimate` and `se`, one per column. +#' +#' @example # Example usage of the function +#' # ivw_combine(rbind(c(1, 2), c(1.2, 2.1)), +#' # rbind(c(0.1, 0.2), c(0.3, 0.1))) +ivw_combine <- function(estimate, error) { + precision <- 1 / error^2 + + list( + estimate = colSums(estimate * precision) / colSums(precision), + se = sqrt(1 / colSums(precision)) + ) +} + +# 4. Register ---- +register_selection( + "ivw_grid", select_ivw_grid, + paste( + "Inverse-variance average of candidate predictions onto a", + "habitat grid (v2 plant habitat)" + ), + requires = c("coefficients", "se") +) + +# End of script ---- diff --git a/1_code/methods/selection/single.R b/1_code/methods/selection/single.R new file mode 100644 index 0000000..0bd52c3 --- /dev/null +++ b/1_code/methods/selection/single.R @@ -0,0 +1,57 @@ +# --- +# title: Selection Rule - Single Model +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `single` rule +# notes: +# - Fits the first formula only. Needs only fit and predict, so +# it suits engines without an information criterion. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. select_single() ---- + +#' Fit One Model, No Selection +#' +#' @param models A list holding one formula, or a single formula. +#' @param base,data,engine,family,weights,offset,ic,control As in +#' selection_run(). +#' @return A selection result. +#' +#' @example # Example usage of the function +#' # select_single(list(. ~ . + MAP), response ~ 1, d, engine, +#' # "binomial") +select_single <- function( + models, base, data, engine, family, + weights = NULL, offset = NULL, ic = "AICc", control = list() +) { + if (inherits(models, "formula")) { + models <- list(models) + } + + fitted <- fit_candidates( + models[1], base, data, engine, family, weights, offset, ic, + control + ) + + selection_result( + fit = fitted$fits[[1]], + coefficients = engine_coef(engine, fitted$fits[[1]]), + ic_table = candidate_table(fitted), + fitted = fitted, + engine = engine + ) +} + +# 3. Register ---- +register_selection( + "single", select_single, + "Fit the first candidate only; no ranking" +) + +# End of script ---- diff --git a/1_code/methods/selection/staged_bic.R b/1_code/methods/selection/staged_bic.R new file mode 100644 index 0000000..c8964b6 --- /dev/null +++ b/1_code/methods/selection/staged_bic.R @@ -0,0 +1,116 @@ +# --- +# title: Selection Rule - Staged Forward Selection by BIC +# author: Brendan Casey +# created: 2026-09-09 +# inputs: none +# outputs: registers the `staged_bic` rule +# notes: +# - The v2 bird landcover rule. Each group of `models` updates +# the previous group's winner: the smallest model within +# `threshold` of the best score. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. select_staged_bic() ---- + +#' Walk Groups of Candidates, Carrying the Winner Forward +#' +#' @inheritParams select_single +#' @param threshold Numeric. How far above the best score a +#' candidate may sit and still be eligible. +#' @param always_advance Logical. TRUE, v2's rule, takes each +#' group's winner as the next base whatever its score; FALSE +#' takes it only when it improves on the running model. +#' @return A selection result, with `ic_table` carrying a `stage` +#' column. +#' +#' @example # Example usage of the function +#' # select_staged_bic(model_groups, response ~ 1, d, engine, +#' # "poisson", ic = "BIC") +select_staged_bic <- function( + models, base, data, engine, family, + weights = NULL, offset = NULL, ic = "BIC", control = list(), + threshold = 2, always_advance = TRUE +) { + # Step 1: Fit the base model, the fallback if no group improves + current <- fit_with( + engine, base, data, family, weights, offset, control + ) + current_formula <- base + current_score <- engine$ic(current, ic) + tables <- list() + + if (!isTRUE(current$ok)) { + return(selection_result( + NULL, NULL, + data.frame( + stage = "base", model = "base", formula = deparse(base), + ic = Inf, delta = NA_real_, weight = 0, k = NA_integer_, + ok = FALSE, stringsAsFactors = FALSE + ), + NULL + )) + } + + group_names <- names(models) + + if (is.null(group_names)) { + group_names <- paste0("stage_", seq_along(models)) + } + + # Step 2: Each group updates the running winner + for (i in seq_along(models)) { + fitted <- fit_candidates( + models[[i]], current_formula, data, engine, family, + weights, offset, ic, control + ) + + table <- candidate_table(fitted) + table$stage <- group_names[i] + tables[[i]] <- table + + eligible <- which(table$ok & is.finite(fitted$scores)) + + if (length(eligible) == 0) { + next + } + + # Step 3: Fewest parameters within `threshold` of the best + best <- min(fitted$scores[eligible]) + close <- eligible[fitted$scores[eligible] <= best + threshold] + winner <- close[which.min(table$k[close])] + + # Step 4: Carry the winner forward. v2 always does, even when + # it scores worse (07.ModelLandcover.R, section 14) + if (always_advance || fitted$scores[winner] < current_score) { + current <- fitted$fits[[winner]] + current_formula <- fitted$formulas[[winner]] + current_score <- fitted$scores[winner] + } + } + + selection_result( + fit = current, + coefficients = engine_coef(engine, current), + ic_table = do.call(rbind, tables), + fitted = NULL, + formula = current_formula, + engine = engine + ) +} + +# 3. Register ---- +register_selection( + "staged_bic", select_staged_bic, + paste( + "Forward selection through groups of candidates, fewest", + "parameters within `threshold` (v2 birds)" + ), + requires = c("ic", "coefficients") +) + +# End of script ---- diff --git a/1_code/modules/.gitkeep b/1_code/modules/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/1_code/modules/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/1_code/modules/README.md b/1_code/modules/README.md new file mode 100644 index 0000000..45fad3b --- /dev/null +++ b/1_code/modules/README.md @@ -0,0 +1,78 @@ +# modules + +One v2 spec per taxon. A module holds configuration, not modelling +code: it states what v2 does for a taxon as a spec list, and the +[harness](../harness/) runs it. A change here affects every +experiment; review it as such. + +`load_framework()` sources `_shared/` first, then every other folder, +each in file-name order. + +## Files + +| Folder | File | Defines | +| --- | --- | --- | +| `_shared/` | `plant_group.R` | `plant_group_spec()`, the builder for the four plant-group taxa, and their v2 post-processing (`plant_age_splines()`, `plant_cutblock_convergence()`, `plant_paspen()`, `plant_footprint_pooling()`) | +| `_shared/` | `standard_specs.R` | `standard_specs(plant_bootstrap)`: every run in its v2 configuration | +| `bryophytes/` | `spec.R` | `bryophyte_spec(use_protocol, bootstrap)` | +| `lichens/` | `spec.R` | `lichen_spec(use_protocol, bootstrap)` | +| `soil_mites/` | `spec.R` | `soil_mite_spec(use_protocol, bootstrap)` | +| `vascular_plants/` | `spec.R` | `vascular_plant_spec(use_protocol, bootstrap)` | +| `mammals/` | `spec.R` | `mammal_spec(climate_source, season, tier)` | +| `mammals/` | `hurdle.R` | v2's hurdle tables: presence and abundance effects, splines, calibration, convergence | +| `birds/` | `spec.R` | `bird_spec()` | +| `birds/` | `standardize.R` | `bird_habitat_translation()`: landcover coefficients onto v2's standardized habitat types (port of v2's `08.PackageCoefficients.R`) | + +`standard_specs()` returns seven runs: `bryophyte`, `lichen`, `mite`, +`vascular_plant`, `mammal_summer`, `mammal_winter` and `bird`. + +## The module interface + +A taxon module must: + +1. Define a function `_spec(...)` in + `modules//spec.R` that returns a spec list. +2. Return a spec that passes `validate_spec()`. Unknown fields are + errors, and every named engine, rule and resampler must be + registered. +3. Add its run to `standard_specs()` in `_shared/standard_specs.R`. + +Spec fields the harness reads (full list: `spec_fields()` in +`harness/spec.R`): + +| Field | Sets | +| --- | --- | +| `taxon` | Data slug that keys the response file, covariates and lookups | +| `response_name`, `response_transform`, `family`, `weight_column` | How the recorded value becomes a response; error family; weights | +| `regions` | Per region: `filter` (one-sided formula), `grid`, `habitat_models`, constants | +| `stages` | In order: `name`, `models`, `engine`, `selection`, `ic`, `post_process`, and `carry_as` for the stage carried into the next as `Climate` | +| `habitat_stage` | The stage `replace_stage_method()` replaces by default | +| `resample` | `scheme` and its settings | +| `final_prediction` | Optional. The model scored and projected | +| `v2_coverage` | Per stage, what the spec reproduces of v2. The one place coverage is stated; `report.md` prints it | +| `notes` | Configuration facts, such as whether `Protocol` is fitted | + +Fields that a stage's selection rule or resampler takes as arguments +are also accepted, as settings for that method. + +## Adding a taxon module + +1. Put the v2 scripts as received in `0_data/v2_scripts//` and + log the source and commit in its README. +2. Extend `_setup/01` to write the taxon's response, covariates, + lookups, and any grid or stored draws; extend `_setup/02` to trace + them to the source. +3. Add the taxon's species queue and manifest rows to `_setup/09`, + and re-run it. +4. Splice the v2 candidate formulas into `harness/model_sets.R`, + verbatim. +5. Add `spec.R`, modelled on `lichens/spec.R` (plant group), + `birds/spec.R` or `mammals/spec.R`, and add the run to + `standard_specs()`. If v2 does something no registered method + covers, add one to [`methods/`](../methods/README.md). +6. Add the taxon's published v2 results to `_setup/08`, if + ABMIexploreR publishes them. +7. Check that `validate_spec()` passes and + `Rscript 1_code/tests/run_tests.R` runs clean. Record each + taxon-specific behaviour in + [`docs/taxon_quirks.md`](../../docs/taxon_quirks.md). diff --git a/1_code/modules/_shared/plant_group.R b/1_code/modules/_shared/plant_group.R index 222ad76..f7c3fff 100644 --- a/1_code/modules/_shared/plant_group.R +++ b/1_code/modules/_shared/plant_group.R @@ -5,24 +5,12 @@ # inputs: none # outputs: none; returns objects in memory # notes: -# - Bryophytes, lichens, soil mites and vascular plants each -# have their own module directory, because each is free to -# diverge and a taxon-specific quirk should have one obvious -# home. What they still share is the v2 pipeline shape, and -# that lives here so four copies cannot drift apart. -# - "Plant group" names a survey design rather than a taxonomy. -# Soil mites are animals; they are here because v2 fits them -# with the plant scripts, on the plant survey design. -# - Each taxon module states its own facts and calls this -# builder. Nothing in here looks a taxon up in a table, so -# adding a quirk means editing that taxon's file rather than -# adding a branch to a shared one. -# - The underscore prefix marks this as shared machinery rather -# than a taxon, matching _setup/ and _deprecated/. -# - What this spec reproduces of v2 is stated once, in -# `v2_coverage` below. The report is generated from it, so -# change it here when the harness changes, and nowhere else. -# Per-quirk detail is in docs/taxon_quirks.md. +# - The v2 pipeline shape shared by the four plant-group taxa +# (a survey design, not a taxonomy). Taxon quirks live in each +# taxon's module, which calls this builder. +# - `v2_coverage` below is the single statement of what is +# reproduced; the report reads it. Detail is in +# docs/taxon_quirks.md. # --- # 1. Setup ---- @@ -34,27 +22,18 @@ #' Build a v2 Spec for One Plant-Group Taxon #' -#' The v2 pipeline shape shared by bryophytes, lichens, soil -#' mites and vascular plants. Called by each taxon module rather -#' than directly. +#' Called by each taxon module rather than directly. #' -#' @param taxon Character. The data slug, which keys the response -#' file, the species catalogue and the covariate catalogue. -#' Note that soil mites carry the slug "mite". +#' @param taxon Character. The data slug ("mite" for soil mites). #' @param protocol_in_v2 Logical. Whether v2 fits Protocol for -#' this taxon. Recorded so a run can say whether it followed v2 -#' or overrode it. +#' this taxon. #' @param use_protocol Logical, or NULL to take the v2 value. -#' Setting it equalizes a term v2 applies inconsistently, which -#' is what a cross-taxa experiment wants. #' @param reference_note Character, or NULL. One clause on where #' this taxon's v2 reference comes from, appended to the spec #' notes. Bryophytes differ from the other three. -#' @param bootstrap Character. "spatial_block" draws v2's -#' spatially blocked bootstrap afresh; "v2_ids" replays the -#' draws v2 itself made, stored per species by -#' _setup/07_harmonize_v2_plant_bootstrap_ids.R, which makes -#' parity numerical draw by draw. +#' @param bootstrap Character. "spatial_block" draws afresh; +#' "v2_ids" replays v2's stored draws (_setup/07), making +#' parity numerical. #' @return A spec list, as run_spec() consumes. #' #' @example # Example usage of the function @@ -86,11 +65,9 @@ plant_group_spec <- function( use_protocol <- protocol_in_v2 } - # The habitat sets are spliced from the mite script, which - # fits no Protocol. In the bryophyte and lichen scripts the - # same formulas carry the term directly after Climate and - # nowhere else, so inserting it there reproduces them. The - # climate set carries no Protocol in any variant. + # The habitat sets come from the mite script (no Protocol); the + # bryophyte and lichen scripts differ only by Protocol directly + # after Climate. No climate set carries Protocol. with_protocol <- function(set) { models <- get_model_set(set) @@ -107,9 +84,10 @@ plant_group_spec <- function( list( taxon = taxon, response_name = "response", + # The stage a methods experiment replaces by default + habitat_stage = "habitat", - # Plant-group responses are cover or abundance values; v2 - # models presence, so anything above zero is a detection. + # v2 models presence: any value above zero is a detection response_transform = "detection", family = "binomial", weight_column = NULL, @@ -140,17 +118,14 @@ plant_group_spec <- function( selection = "aic_average", ic = "AICc", control = list(maxit = 250), - # Fitted once per draw on every unit, as v2's - # climate_models() is, and shared by both regions. + # Province-wide, as v2's climate_models() scope = "province", - # The habitat stage takes this stage's prediction as a - # term called Climate, which is how v2 composes them. + # Carried into the habitat stage as `Climate`, as in v2 carry_as = "Climate" ), list( name = "habitat", - # Set per region below, because north fits vegetation - # types and south fits soil types. + # Per region below: veg in the north, soil in the south models = NULL, engine = "bayesglm", selection = "ivw_grid", @@ -158,10 +133,7 @@ plant_group_spec <- function( control = list(maxit = 250), carry_from = "climate", carry_from_as = "Climate", - # v2 predicts each candidate onto the grid holding - # Climate at zero, and at the new protocol where Protocol - # is fitted, so the habitat effects are read at an - # average climate under one protocol. + # v2 predicts the grid at Climate = 0 and the new protocol grid_constants = if (use_protocol) { list( Climate = 0, @@ -175,12 +147,12 @@ plant_group_spec <- function( } else { c("Intercept", "Climate") }, - # v2's stand-age splines, then its cutblock convergence, - # both before the footprint pooling (coef_adjust). The - # splines read the aged stand-cover columns, which no - # formula names. Both leave the south unchanged. + # v2's post-averaging steps, in v2's order (pooling + # borrows from a spline-fitted age class). The splines + # read aged stand-cover columns no formula names. post_process = list( - plant_age_splines, plant_cutblock_convergence, plant_paspen + plant_age_splines, plant_cutblock_convergence, + plant_paspen, plant_footprint_pooling ), # The validation also weights each fine cutblock class by # its own effect, and the formulas name only lumped ones. @@ -192,20 +164,13 @@ plant_group_spec <- function( c("R", 1:4), paste0 ))) ), - coef_adjust = TRUE, - # v2 skips the habitat model when the draw, after the - # region filter, holds fewer than 20 detections. min_detections = 20L ) ), - # One draw per species for the whole province, as v2's - # bootstrap_data() makes it, filtered to each region's units - # when the habitat stage is fitted. - # - # No seed here. The experiment sets one base seed, `boot_seed` - # in run.R, and each species derives its own from it. Without - # one the fresh draws are unseeded, as in v2. + # One province-wide draw per species, as v2's + # bootstrap_data(). No seed here: the experiment's seed is + # used. resample = if (bootstrap == "v2_ids") { list( scheme = "precomputed", @@ -220,8 +185,6 @@ plant_group_spec <- function( ) }, - # What this spec reproduces of v2, per stage. The single - # source of truth for coverage: the report reads it. v2_coverage = list( climate = v2_status( "reproduced", @@ -269,11 +232,13 @@ plant_group_spec <- function( # out-of-bag as well; see plant_v2_validation(). validate = plant_v2_validation, + # The final model the harness scores and reports: v2's own + # prediction from the published coefficient tables + final_prediction = plant_v2_full_prediction, + use_protocol = use_protocol, protocol_is_v2 = identical(use_protocol, protocol_in_v2), - # Facts about this taxon's configuration. Coverage is in - # v2_coverage, not here. notes = paste0( "Protocol ", if (use_protocol) "fitted" else "not fitted", if (identical(use_protocol, protocol_in_v2)) { @@ -345,11 +310,10 @@ plant_v2_term_labels <- function(terms) { # 4. v2 habitat machinery ---- # Ports of the steps v2's vegetation_models() applies after the # inverse-variance weighting (hierarchical-model_functions.R), -# in v2's order: the stand-age splines (section 7), then the -# cutblock convergence (7.5). The footprint pooling (8.0) follows -# in the harness, as coef_adjust. Each takes and returns the -# selection result, so the habitat stage runs them as -# `post_process` steps. +# in v2's order: the stand-age splines (section 7), the cutblock +# convergence (7.5), and the footprint pooling (8.0). Each takes +# and returns the selection result, so the habitat stage runs +# them as `post_process` steps. ## 4.1 plant_stand_types() ---- @@ -405,7 +369,7 @@ plant_age_columns <- function() { #' unchanged. #' #' @param selected A selection result from the ivw_grid rule, -#' carrying `site_prediction`. +#' whose `predict()` gives the site-level IVW prediction. #' @param data The stage's data: detection in `response`, and the #' unaged and aged stand-cover columns, `Grass` and `Shrub`. #' @return The selection result with the 45 aged effects added. @@ -422,15 +386,21 @@ plant_age_splines <- function(selected, data) { return(selected) } - if (is.null(selected$site_prediction) || - length(selected$site_prediction) != nrow(data)) { + # Site-level IVW prediction (link scale), computed only where + # aged stand types exist + prediction <- if (is.function(selected$predict)) { + selected$predict(data, "link") + } else { + NULL + } + + if (is.null(prediction) || length(prediction) != nrow(data)) { stop("The age splines need the site-level IVW prediction.", call. = FALSE) } cutoff <- 0.1 count <- data$response - prediction <- selected$site_prediction # Step 1: One frame per stand type stand <- stats::setNames(vector("list", length(types)), types) @@ -581,16 +551,17 @@ plant_age_splines <- function(selected, data) { #' # selected <- plant_cutblock_convergence(selected, stage_data) plant_cutblock_convergence <- function(selected, data = NULL) { coefficients <- selected$coefficients + recovery <- cutblock_recovery_weights() weights <- list( - WhiteSpruce = c(0.50, 0.849, 0.96), - Pine = c(0.50, 0.849, 0.96), - Deciduous = c(0.705, 0.912, 0.97), - Mixedwood = c(0.705, 0.912, 0.97) + WhiteSpruce = recovery$conifer, + Pine = recovery$conifer, + Deciduous = recovery$deciduous, + Mixedwood = recovery$deciduous ) for (type in names(weights)) { - for (k in seq_along(2:4)) { - age <- (2:4)[k] + for (k in 1:3) { + age <- k + 1 w <- weights[[type]][k] cc <- match(paste0("CC", type, age), coefficients$term) natural <- match(paste0(type, age), coefficients$term) @@ -666,7 +637,230 @@ plant_paspen <- function(selected, data = NULL) { selected } -## 4.6 plant_v2_validation() ---- +## 4.6 plant_footprint_pooling() ---- + +#' Pool Poorly Sampled Footprint Effects, as a Post-Process Step +#' +#' v2's section 8.0, applied after the stand-age splines and the +#' cutblock convergence because it borrows from a spline-fitted +#' age class. See coef_adjust_plant_veg() for what is pooled. +#' +#' @param selected A selection result. +#' @param data Unused; the post_process signature. +#' @return The selection result with the four footprint effects +#' pooled. +#' +#' @example # Example usage of the function +#' # selected <- plant_footprint_pooling(selected, stage_data) +plant_footprint_pooling <- function(selected, data = NULL) { + selected$coefficients <- coef_adjust_plant_veg( + selected$coefficients + ) + + selected +} + +## 4.7 coef_adjust_plant_veg() ---- + +#' Borrow Strength for Poorly Sampled Footprint Types +#' +#' v2's `coef.adjust`. Some human footprint types are rarely the +#' dominant cover at a survey unit, so their effect is estimated +#' from little data. v2 pools each with a type it is assumed to +#' resemble, by inverse-variance weight: +#' +#' - `HardLin` borrows from `UrbInd`. +#' - The three soft linear types - `EnSoftLin`, `EnSeismic` and +#' `TrSoftLin` - each borrow from a composite of young +#' regenerating stands, weighted by how much those stand types +#' overlap soft linear features in the provincial summary. +#' +#' The weights are v2's, measured from a 1 km summary and fixed +#' since 2020-11-17. They are an assumption about which habitats +#' resemble which, not an estimate, so they are stated here +#' rather than derived. +#' +#' @param coefficients A data frame of term, estimate and se. +#' @param overlap Numeric vector of four proportions, for +#' white spruce, pine, deciduous and black spruce regeneration. +#' @return The coefficients, with the four terms adjusted. +#' +#' @example # Example usage of the function +#' # coef_adjust_plant_veg(selected$coefficients) +coef_adjust_plant_veg <- function( + coefficients, + overlap = c(0.049, 0.0893, 0.434, 0.396) +) { + if (is.null(coefficients)) { + return(NULL) + } + + value <- stats::setNames( + coefficients$estimate, coefficients$term + ) + error <- stats::setNames(coefficients$se, coefficients$term) + + # Pool two estimates by precision. + pool <- function(a, a_se, b, b_se) { + precision <- 1 / a_se^2 + 1 / b_se^2 + + list( + estimate = (a / a_se^2 + b / b_se^2) / precision, + se = sqrt(1 / precision) + ) + } + + present <- function(...) { + all(c(...) %in% names(value)) && + all(is.finite(value[c(...)])) && + all(is.finite(error[c(...)])) + } + + # Step 1: Hard linear features borrow from urban and industrial + if (present("HardLin", "UrbInd")) { + pooled <- pool( + value["HardLin"], error["HardLin"], + value["UrbInd"], error["UrbInd"] + ) + value["HardLin"] <- pooled$estimate + error["HardLin"] <- pooled$se + } + + # Step 2: Soft linear features borrow from young regeneration + young_types <- c( + "CCWhiteSpruceR", "CCPineR", "CCDeciduousR", "BlackSpruce1" + ) + + if (present(young_types)) { + weights <- overlap / sum(overlap) + + young <- sum(weights * value[young_types]) + # The weighted variance carries no between-type component, + # because the weights are fixed rather than estimated. + young_se <- sqrt(sum(weights * error[young_types]^2)) + + for (one in c("EnSoftLin", "EnSeismic", "TrSoftLin")) { + if (!present(one)) { + next + } + + pooled <- pool(value[one], error[one], young, young_se) + value[one] <- pooled$estimate + error[one] <- pooled$se + } + } + + coefficients$estimate <- unname(value[coefficients$term]) + coefficients$se <- unname(error[coefficients$term]) + + coefficients +} + +## 4.8 plant_v2_parts() ---- + +#' The Pieces of v2's Plant Prediction at a Set of Units +#' +#' v2 predicts a plant from its coefficient tables, not from a +#' fitted model: the climate prediction, and a landcover +#' prediction that weights each habitat type's effect by its +#' cover at the unit. Shared by the validation AUCs and by the +#' final prediction, so the two cannot drift apart. +#' +#' The cover terms are the published effects, less the unaged +#' stand types, which v2 publishes only by age class. A term's +#' column may carry the block suffix the harmonizer gave columns +#' in more than one block (UrbInd_veg, UrbInd_soil). +#' +#' Reproduced as written: the protocol coefficient, a logit, is +#' subtracted from an "Old" survey's landcover prediction on the +#' probability scale. +#' +#' @param stage_coefficients Named list of data frames of term +#' and estimate: `climate` and `habitat`. +#' @param data The units to predict at. +#' @return A list: `climate` (probability), `raw` (landcover, +#' probability scale), `landcover` (`raw` with the protocol +#' adjustment), `slope` (the Climate effect), `aspen`, the cover +#' matrix `x` and the cover terms' `effects` (logit scale). +#' +#' @example # Example usage of the function +#' # parts <- plant_v2_parts(stage_coefficients, region_frame) +plant_v2_parts <- function(stage_coefficients, data) { + climate <- stage_coefficients$climate + habitat <- stage_coefficients$habitat + + estimate <- stats::setNames(habitat$estimate, habitat$term) + head_terms <- c("Intercept", "Climate", "Protocol", "paspen") + + column_of <- function(term) { + candidates <- c(term, paste0(term, c("_veg", "_soil"))) + hit <- candidates[candidates %in% names(data)] + if (length(hit) == 0) NA_character_ else hit[1] + } + + cover <- setdiff( + names(estimate), c(head_terms, plant_stand_types()) + ) + cover <- cover[is.finite(estimate[cover])] + cover_columns <- vapply(cover, column_of, character(1)) + cover <- cover[!is.na(cover_columns)] + cover_columns <- cover_columns[!is.na(cover_columns)] + + x <- as.matrix(data[, cover_columns, drop = FALSE]) + raw <- drop(x %*% stats::plogis(estimate[cover])) + landcover <- raw + + if ("Protocol" %in% names(estimate) && + "Protocol" %in% names(data)) { + old <- as.character(data$Protocol) == "Old" + landcover[old] <- landcover[old] - estimate[["Protocol"]] + } + + list( + climate = stats::plogis( + predict_from_coefficients(climate, data, "link") + ), + raw = raw, + landcover = landcover, + slope = estimate[["Climate"]], + aspen = if ("paspen" %in% names(estimate) && + "paspen" %in% names(data)) { + data$paspen * estimate[["paspen"]] + } else { + 0 + }, + x = x, + effects = estimate[cover] + ) +} + +## 4.9 plant_v2_full_prediction() ---- + +#' v2's Final Plant Prediction: Landcover and Climate Together +#' +#' The prediction v2's "Full" validation AUC scores. The plant +#' specs name it as their `final_prediction`, so the harness's +#' own metrics score the model v2 reports, after its stand-age +#' splines, cutblock convergence and footprint pooling, rather +#' than any single fitted candidate. +#' +#' @param stage_coefficients Named list with `climate` and +#' `habitat` coefficient tables. +#' @param frame The units to predict at. +#' @return A numeric vector of probabilities, one per row. +#' +#' @example # Example usage of the function +#' # plant_v2_full_prediction(stage_coefficients, frame) +plant_v2_full_prediction <- function(stage_coefficients, frame) { + parts <- plant_v2_parts(stage_coefficients, frame) + + stats::plogis( + stats::qlogis(parts$raw) + parts$climate * parts$slope + + parts$aspen + ) +} + +## 4.10 plant_v2_validation() ---- #' v2's Plant Validation AUCs, In-Bag and Out-of-Bag #' @@ -683,10 +877,6 @@ plant_paspen <- function(selected, data = NULL) { #' draw left out, which is the held-out read v2 does not have. #' Iteration 1 is the full data and has no out-of-bag units. #' -#' Reproduced as written: the protocol coefficient, a logit, is -#' subtracted from an "Old" survey's landcover prediction on the -#' probability scale. -#' #' @param stage_coefficients Named list of data frames of term #' and estimate: `climate` and `habitat`. #' @param frame The region's frame. @@ -715,73 +905,42 @@ plant_v2_validation <- function( ) } - climate_prediction <- function(data) { - stats::plogis(predict_from_coefficients(climate, data, "link")) - } - truncate <- function(x) { cap <- stats::quantile(x, 0.99, names = FALSE) ifelse(x >= cap, cap, x) } - estimate <- stats::setNames(habitat$estimate, habitat$term) - head <- c("Intercept", "Climate", "Protocol", "paspen") - - # The cover terms are the published effects, less the unaged - # stand types, which v2 publishes only by age class. A term's - # column may carry the block suffix the harmonizer gave columns - # in more than one block (UrbInd_veg, UrbInd_soil). - column_of <- function(term) { - candidates <- c(term, paste0(term, c("_veg", "_soil"))) - hit <- candidates[candidates %in% names(frame)] - if (length(hit) == 0) NA_character_ else hit[1] - } - - cover <- setdiff(names(estimate), c(head, plant_stand_types())) - cover <- cover[is.finite(estimate[cover])] - cover_columns <- vapply(cover, column_of, character(1)) - cover <- cover[!is.na(cover_columns)] - cover_columns <- cover_columns[!is.na(cover_columns)] - score <- function(in_units) { all_units <- province_frame[ province_frame$survey_unit_id %in% in_units, , drop = FALSE ] - region <- frame[frame$survey_unit_id %in% in_units, , drop = FALSE] + region <- frame[ + frame$survey_unit_id %in% in_units, , drop = FALSE + ] if (nrow(region) == 0 || nrow(all_units) == 0) { return(rep(NA_real_, 7)) } - climate_all <- climate_prediction(all_units) - climate_region <- climate_prediction(region) - cap <- stats::quantile(climate_all, 0.99, names = FALSE) - climate_region_trunc <- ifelse(climate_region >= cap, cap, - climate_region) - - x <- as.matrix(region[, cover_columns, drop = FALSE]) - effect <- stats::plogis(estimate[cover]) - landcover <- drop(x %*% effect) - - if ("Protocol" %in% names(estimate) && "Protocol" %in% names(region)) { - old <- as.character(region$Protocol) == "Old" - landcover[old] <- landcover[old] - estimate[["Protocol"]] - } + parts <- plant_v2_parts(stage_coefficients, region) + climate_all <- stats::plogis( + predict_from_coefficients(climate, all_units, "link") + ) - slope <- estimate[["Climate"]] - aspen <- if ("paspen" %in% names(estimate) && - "paspen" %in% names(region)) { - region$paspen * estimate[["paspen"]] - } else { - 0 - } + # The truncation caps the region's climate at the 99th + # percentile over every unit in the draw + cap <- stats::quantile(climate_all, 0.99, names = FALSE) + climate_region_trunc <- ifelse( + parts$climate >= cap, cap, parts$climate + ) - raw <- drop(x %*% effect) - component <- climate_region * slope + aspen - component_trunc <- climate_region_trunc * slope + aspen + component <- parts$climate * parts$slope + parts$aspen + component_trunc <- climate_region_trunc * parts$slope + + parts$aspen joint <- function(shift) { - rowSums(x * stats::plogis(outer(shift, estimate[cover], `+`))) + shifted <- outer(shift, parts$effects, `+`) + rowSums(parts$x * stats::plogis(shifted)) } observed <- as.integer(region$response > 0) @@ -790,10 +949,13 @@ plant_v2_validation <- function( c( Climate = auc(observed_all, climate_all), Climate_Truncated = auc(observed_all, truncate(climate_all)), - Landcover = auc(observed, landcover), - Full = auc(observed, stats::plogis(stats::qlogis(raw) + component)), + Landcover = auc(observed, parts$landcover), + Full = auc( + observed, stats::plogis(stats::qlogis(parts$raw) + component) + ), Full_Truncated = auc( - observed, stats::plogis(stats::qlogis(raw) + component_trunc) + observed, + stats::plogis(stats::qlogis(parts$raw) + component_trunc) ), Full_Joint = auc(observed, joint(component)), Full_Joint_Truncated = auc(observed, joint(component_trunc)) @@ -813,8 +975,10 @@ plant_v2_validation <- function( score(out_of_bag) } - names_out <- c("Climate", "Climate_Truncated", "Landcover", "Full", - "Full_Truncated", "Full_Joint", "Full_Joint_Truncated") + names_out <- c( + "Climate", "Climate_Truncated", "Landcover", "Full", + "Full_Truncated", "Full_Joint", "Full_Joint_Truncated" + ) data.frame( metric = c(paste0("v2val_", names_out), @@ -824,12 +988,36 @@ plant_v2_validation <- function( ) } -# 5. plant_group_specs() ---- +# 5. cutblock_recovery_weights() ---- + +#' v2's Cutblock Recovery Weights +#' +#' How far a cutblock of age class 2, 3 and 4 has converged on the +#' natural stand of the same age, by stand group. Fixed in v2 and +#' shared by its plant (section 7.5) and mammal (section 6.11) +#' pipelines, so stated once for both modules. +#' +#' The two pipelines' stand-age splines are not shared: their +#' parameters differ (spline basis, detection threshold, model +#' choice), and merging them would put parity at risk. +#' +#' @return A list of `conifer` and `deciduous` weights, one per +#' age class 2 to 4. +#' +#' @example # Example usage of the function +#' # cutblock_recovery_weights()$conifer +cutblock_recovery_weights <- function() { + list( + conifer = c(0.500, 0.849, 0.960), + deciduous = c(0.705, 0.912, 0.970) + ) +} + +# 6. plant_group_specs() ---- #' Every Plant-Group Spec #' -#' A convenience for running all four at once. Each taxon module -#' must already be sourced. +#' Each taxon module must already be sourced. #' #' @param use_protocol Logical, or NULL for the v2 setting per #' taxon. diff --git a/1_code/modules/_shared/standard_specs.R b/1_code/modules/_shared/standard_specs.R new file mode 100644 index 0000000..e1cf499 --- /dev/null +++ b/1_code/modules/_shared/standard_specs.R @@ -0,0 +1,45 @@ +# --- +# title: The v2 Spec of Every Run +# author: Brendan Casey +# created: 2026-10-03 +# inputs: none +# outputs: none; returns objects in memory +# notes: +# - The runs an experiment can name (`taxa = c("lichen", +# "bird")`), each in its v2 configuration. Run names are the +# result folder names. Mammals run once per season (v2's +# references average summer and winter), both with slug +# "mammal". A new taxon module adds its run here. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +# Base R only. + +# 2. standard_specs() ---- + +#' Every Run's Spec, in Its v2 Configuration +#' +#' @param plant_bootstrap Character. "spatial_block" draws v2's +#' plant bootstrap afresh, so parity is distributional; +#' "v2_ids" replays the draws v2 stored (needs +#' _setup/07_harmonize_v2_plant_bootstrap_ids.R run for every +#' species in the run). +#' @return A named list of specs, one per run. +#' +#' @example # Example usage of the function +#' # names(standard_specs()) +standard_specs <- function(plant_bootstrap = "spatial_block") { + list( + bryophyte = bryophyte_spec(bootstrap = plant_bootstrap), + lichen = lichen_spec(bootstrap = plant_bootstrap), + mite = soil_mite_spec(bootstrap = plant_bootstrap), + vascular_plant = vascular_plant_spec(bootstrap = plant_bootstrap), + mammal_summer = mammal_spec(season = "summer"), + mammal_winter = mammal_spec(season = "winter"), + bird = bird_spec() + ) +} + +# End of script ---- diff --git a/1_code/modules/birds/.gitkeep b/1_code/modules/birds/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/1_code/modules/birds/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/1_code/modules/birds/spec.R b/1_code/modules/birds/spec.R index 4937dec..51145c6 100644 --- a/1_code/modules/birds/spec.R +++ b/1_code/modules/birds/spec.R @@ -5,27 +5,13 @@ # inputs: none # outputs: none; returns objects in memory # notes: -# - The v2 configuration for birds, as data. -# - Counts with a QPAD log-offset, which is what makes a count -# comparable across survey protocol and detectability. The -# offset is not optional: without it a bird model is fitting -# effort as much as abundance. -# - Two stages. Climate is model-averaged by AICc. Landcover is -# staged forward selection: groups of formulas, each fitted -# as an update to the previous group's winner, keeping the -# smallest model within 2 BIC of the best. -# - The climate set is v2's 25 candidates (06.ModelClimate.R): -# the null, eight climate combinations, and each of the eight -# with linear and with quadratic spatial terms. The spatial -# terms are the UTM easting and northing sites.csv carries -# for birds, aliased to v2's names. -# - Climate is fitted once per draw on every survey, as v2 -# does, and shared by both regions (`scope = "province"`). -# - Bootstrap ids are precomputed and stored against surveyid -# rather than survey_unit_id, so the draw is translated -# through the design block. -# - What this spec reproduces of v2 is stated once, in -# `v2_coverage`. The report is generated from it. +# - Counts with a QPAD log-offset (required: it separates +# effort and detectability from abundance). +# - Climate: v2's 25 candidates averaged by AICc, province-wide +# per draw. Landcover: staged_bic, smallest model within 2 +# BIC of the best. +# - v2's stored draws are keyed on surveyid and translated to +# survey units. # --- # 1. Setup ---- @@ -45,6 +31,8 @@ bird_spec <- function() { list( taxon = "bird", + # The stage a methods experiment replaces by default + habitat_stage = "landcover", response_name = "response", # Counts are modelled as given; the offset carries effort. @@ -53,14 +41,15 @@ bird_spec <- function() { weight_column = NULL, tier = NULL, - # v2's spatial terms are the survey's UTM easting and northing - # in metres, which sites.csv carries for birds. + # v2's spatial terms: UTM easting and northing, in metres aliases = list(Easting = "easting", Northing = "northing"), regions = list( north = list( filter = ~ useNorth == 1, - grid = NULL, + # One row per habitat type, rebuilt by _setup/09 from + # v2's coefficient translation matrix + grid = "bird_north", term_block = "veg", habitat_models = "landcover_bird_north_v2", # v2 weights the landcover models by region @@ -68,7 +57,9 @@ bird_spec <- function() { ), south = list( filter = ~ useSouth == 1, - grid = NULL, + # One row per habitat type, rebuilt by _setup/09 from + # v2's coefficient translation matrix + grid = "bird_south", term_block = "soil", habitat_models = "landcover_bird_south_v2", weight_column = "soilw" @@ -78,35 +69,27 @@ bird_spec <- function() { stages = list( list( name = "climate", - # v2's 25 candidates, averaged at once by AICc - # (06.ModelClimate.R); only the landcover stage walks - # groups. + # 06.ModelClimate.R models = "climate_bird_v2", engine = "glm", selection = "aic_average", ic = "AICc", - # Fitted once on every survey in the draw, as v2 fits it, - # and shared by both regions. Unweighted, as in v2. + # Province-wide and unweighted, as in v2 scope = "province", weighted = FALSE, - # v2 carries the averaged prediction on the response - # scale WITHOUT the QPAD offset: exp(link), the rate. - # 06.ModelClimate.R asks for predict(type = "link") on a - # MuMIn average, which is built from the averaged - # coefficients and drops the offset. Checked 2026-10-02: - # v2's stored AMRO draw-1 prediction averages 0.118; - # exp(link) on the same surveys gives 0.118, and - # exp(link + offset) gives 0.315. Carrying the offset cut - # the landcover Climate coefficient about threefold - # (AMRO draw 1: 1.60 against v2's 4.91; 4.76 without). + # v2 carries exp(link) WITHOUT the QPAD offset: MuMIn's + # averaged predict(type = "link") drops it. Evidence (AMRO + # draw 1): v2's stored prediction mean 0.118 = exp(link); + # exp(link + offset) = 0.315, and carrying the offset cut + # the landcover Climate coefficient from 4.91 (v2) to + # 1.60. carry_as = "Climate", carry_scale = "exp", carry_offset = FALSE ), list( name = "landcover", - # Set per region: vegetation classes in the north, soil - # classes in the south. + # Per region: veg in the north, soil in the south models = NULL, engine = "glm", selection = "staged_bic", @@ -123,15 +106,12 @@ bird_spec <- function() { resample = list( scheme = "precomputed", id_column = "surveyid", - # v2 selects a draw with %in%, so a survey drawn twice is - # fitted once. + # v2 selects a draw with %in%: repeats are fitted once unique_ids = TRUE, # One draw for the whole province, filtered per region. scope = "province" ), - # What this spec reproduces of v2, per stage. The single - # source of truth for coverage: the report reads it. v2_coverage = list( climate = v2_status( "reproduced", @@ -164,8 +144,6 @@ bird_spec <- function() { ) ), - # Facts about this taxon's configuration. Coverage is in - # v2_coverage, not here. notes = paste0( "QPAD offset inside the formula. The landcover groups' ", "derived terms (wtAge, isCon, fcc2) are all present; ", diff --git a/1_code/modules/birds/standardize.R b/1_code/modules/birds/standardize.R index 7d2b021..772da70 100644 --- a/1_code/modules/birds/standardize.R +++ b/1_code/modules/birds/standardize.R @@ -3,20 +3,17 @@ # author: Brendan Casey; translation logic by Elly Knight (v2) # created: 2026-09-29 # inputs: -# in 0_data/test_dataset/: +# in the test dataset (see harness/data_source.R): # - covariates.csv, the bird rows # - lookup/bird_veg_age_matrix.csv, from # 1_code/_setup/06_harmonize_bird_translation_lookup.R # - lookup/covariate_columns.csv, lookup/bird_factor_levels.csv # outputs: none; returns objects in memory # notes: -# - A port of the translation in v2's 08.PackageCoefficients.R -# (cloned in 1_code/_setup/04_package_bird_coefficients.R). -# v2 publishes bird landcover effects on the standardized -# cross-taxa habitat types - WhiteSpruceR to WhiteSpruce8, -# Loamy, EnSoftLin and so on - not as raw glm coefficients, -# so the harness's raw coefficients have to be translated -# the same way before they can be compared. +# - Port of v2's 08.PackageCoefficients.R translation. v2 +# publishes bird landcover effects on the standardized +# cross-taxa habitat types, not raw glm coefficients, so the +# harness's coefficients are translated the same way. # - North: the raw coefficients are multiplied by v2's `age` # matrix, one row per standardized type. Linear features and # wellsites are re-expressed as the mean predicted abundance @@ -30,8 +27,7 @@ # reads -10,000 rather than -Inf. # - Standard errors are not translated. The gate compares # estimates only. -# - Two v2 quirks are reproduced rather than corrected, because -# the point is to match v2: +# - v2 quirks reproduced for parity: # - The mSoft substitution tests for zero after the # coefficients are exponentiated, so it never fires. # - The NESP north Climate effect is multiplied by 0.1 twice, @@ -79,7 +75,7 @@ bird_sort_interaction <- function(x) { #' block - `road` became `road_veg` and `road_soil` - so model #' terms carry the suffix and v2's lookups do not. #' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param block Character. "veg" for north, "soil" for south. #' @return A data frame of source and master names, one row per #' source column, preferring the given block. @@ -184,7 +180,7 @@ bird_model_terms <- function(set) { #' the covariates under v2's names, the design matrix, and the #' row sets the linear-feature adjustment averages over. #' -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param region Character. "north" or "south". #' @return A list. #' @@ -231,7 +227,9 @@ bird_translation_context <- function(data_dir, region) { # Step 3: The design matrix, with v2's column names design <- stats::model.matrix( - stats::as.formula(paste("~", paste(model_terms, collapse = " + "))), + stats::as.formula( + paste("~", paste(model_terms, collapse = " + ")) + ), stats::model.frame( stats::as.formula( paste("~", paste(model_terms, collapse = " + ")) @@ -335,9 +333,8 @@ standardize_bird_draw <- function(raw, context, species) { # Step 1: Standardized habitat types from the age matrix lam <- exp(drop(age %*% value(colnames(age)))) - # Step 2: Linear features. v2 exponentiates, then tests for - # zero, so the mSoft substitution below never fires; kept - # so the code reads as v2's does. + # Step 2: Linear features. The mSoft substitution never fires + # (v2 tests for zero after exponentiating); kept as in v2. hf <- exp(value(c("mWell", "mSoft", "mEnSft", "mTrSft", "mSeism"))) @@ -436,7 +433,7 @@ standardize_bird_draw <- function(raw, context, species) { #' #' @param coefficients A data frame of species, region, boot, #' term and estimate: the landcover stage, one region. -#' @param data_dir Character. Path to 0_data/test_dataset. +#' @param data_dir Character. The test dataset folder. #' @param region Character. "north" or "south". #' @return A data frame of the same columns, on the standardized #' template, with stage "habitat" and se NA. @@ -486,4 +483,31 @@ standardize_bird_coefficients <- function( out } +# 5. bird_habitat_translation() ---- + +#' Translate a Store's Bird Landcover onto v2's Habitat Template +#' +#' The form collect_results() takes in `translate`: given a +#' store's coefficients, returns the landcover stage translated +#' onto v2's standardized habitat types as extra rows, stage +#' "habitat". The raw "landcover" rows stay as they are. +#' +#' @param coefficients A store's coefficients. +#' @param data_dir Character. The test dataset folder. +#' @param region Character. "north" or "south". +#' @return A data frame of coefficient rows, or NULL. +#' +#' @example # Example usage of the function +#' # collect_results(pipeline_dir, data_dir, +#' # translate = list(bird = bird_habitat_translation)) +bird_habitat_translation <- function(coefficients, data_dir, region) { + landcover <- coefficients[coefficients$stage == "landcover", ] + + if (nrow(landcover) == 0) { + return(NULL) + } + + standardize_bird_coefficients(landcover, data_dir, region) +} + # End of script ---- diff --git a/1_code/modules/bryophytes/spec.R b/1_code/modules/bryophytes/spec.R index c3e4f6c..8183d7a 100644 --- a/1_code/modules/bryophytes/spec.R +++ b/1_code/modules/bryophytes/spec.R @@ -7,23 +7,14 @@ # outputs: none; returns objects in memory # notes: # - Data slug: "bryophyte". Response file bryophyte.csv. -# - v2 fits Protocol for this taxon. Only bryophytes and -# lichens get it; soil mites and vascular plants do not. The -# asymmetry is v2's and is preserved here for parity, with -# `use_protocol` exposed so an experiment can equalize it. -# - The published bryophyte model file is unusable twice over: -# every species and draw in it is an error object reading -# `could not find function "model.avg"`, and the bootstrap-id -# file held 3 draws where the other three taxa hold 100. -# - Two usable references exist instead. v2's own COEFS.RData -# holds complete bryophyte arrays, and _setup/05 takes the -# bryophyte rows of v2_results.csv from it. -# _setup/03_rerun_bryophyte_v2_reference.R also regenerated -# the climate stage into 2_pipeline/v2_reference/. -# - 02_compare_to_v2.R reads neither yet, so bryophytes are not -# compared. Moving the gate onto v2_results.csv fixes that. -# - Anything true of every plant-group taxon belongs in -# _shared/plant_group.R, not here. +# - v2 fits Protocol for this taxon (bryophytes and lichens +# only); `use_protocol` can equalize it across taxa. +# - The published bryophyte model file is unusable (every entry +# is a `could not find function "model.avg"` error; 3 stored +# draws, not 100). The parity reference is ABMIexploreR's +# arrays (_setup/08), matching v2's COEFS.RData to ~1e-15. +# _setup/03 regenerated the climate stage into +# 2_pipeline/v2_reference/ for v2_self_agreement.R. # --- # 1. Setup ---- @@ -53,7 +44,7 @@ bryophyte_spec <- function(use_protocol = NULL, bootstrap = bootstrap, reference_note = paste0( "The published v2 model file is unusable; the v2 ", - "reference is COEFS.RData, via v2_results.csv." + "reference is ABMIexploreR's, via v2_results.csv." ) ) } diff --git a/1_code/modules/lichens/spec.R b/1_code/modules/lichens/spec.R index 590fc01..3eadb1b 100644 --- a/1_code/modules/lichens/spec.R +++ b/1_code/modules/lichens/spec.R @@ -7,14 +7,10 @@ # outputs: none; returns objects in memory # notes: # - Data slug: "lichen". Response file lichen.csv. -# - v2 fits Protocol for this taxon. Only bryophytes and -# lichens get it; soil mites and vascular plants do not. The -# asymmetry is v2's and is preserved here for parity, with -# `use_protocol` exposed so an experiment can equalize it. +# - v2 fits Protocol for this taxon (bryophytes and lichens +# only); `use_protocol` can equalize it across taxa. # - The v2 reference is the published file on ABMI-DATA2 and is # intact, unlike the bryophyte one. -# - Anything true of every plant-group taxon belongs in -# _shared/plant_group.R, not here. # --- # 1. Setup ---- diff --git a/1_code/modules/mammals/.gitkeep b/1_code/modules/mammals/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/1_code/modules/mammals/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/1_code/modules/mammals/hurdle.R b/1_code/modules/mammals/hurdle.R index 0dee071..d52fd4d 100644 --- a/1_code/modules/mammals/hurdle.R +++ b/1_code/modules/mammals/hurdle.R @@ -1,26 +1,20 @@ # --- -# title: The v2 Mammal Hurdle Model +# title: The v2 Mammal Hurdle Tables # author: Brendan Casey; model logic by Marcus Becker and David J. # Huggard (v2) # created: 2026-09-29 # inputs: -# in 0_data/test_dataset/lookup/: +# in the test dataset's lookup/ (see data_source.R): # - mammal_climate_predictions.csv -# - mammal_modelled_species.csv +# - species_queue.csv (each species' v2 name) # outputs: none; returns objects in memory # notes: -# - A port of the per-species, per-season body of v2's -# 03_basic-models.R, north (sections 6.1 to 6.6 and 6.11) and -# south (6.1 to 6.6), for one species, season and region. It -# is registered as the habitat stage's selection rule, so the -# harness runs it once per draw. -# - v2 couples the two halves of the hurdle, so they are fitted -# together here: presence (binomial, best of the candidates by -# AICc), abundance given presence (Gamma on the log scale, over -# the presence candidates that leave some cover unaccounted -# for, plus an effort-only null), their product as total -# abundance, calibrated to the observed mean, and in the north -# stand-age splines and cutblock convergence. +# - Ports how v2 turns the two hurdle GLMs into its published +# tables: 03_basic-models.R after the fits, north sections +# 6.3-6.6 and 6.11, south 6.3-6.6. The hurdle itself is fitted +# by methods/selection/hurdle.R. +# - mammal_v2_habitat_tables() is a post_process step, removed +# by replace_stage_method(); it stops if a half is not a GLM. # - Effects are reported on v2's full habitat set, not the # winning model's categories: each category is spread over # the fine types the prediction matrix assigns to it. @@ -49,11 +43,9 @@ #' Attach v2's Precomputed Climate Prediction to a Mammal Frame #' -#' v2's habitat models do not fit climate; they read a -#' per-species prediction from a separate climate pipeline, and -#' drop the deployments that have none. This does the same, then -#' recomputes the response on the remaining rows, because the lure -#' correction and scaling depend on which rows are in. +#' As v2: drops deployments without a prediction, then recomputes +#' the response, since lure correction and scaling depend on the +#' rows kept. #' #' @param frame A one-species frame from model_frame(). #' @param species Character. The harness species name, e.g. @@ -64,25 +56,25 @@ #' #' @example # Example usage of the function #' # frame <- mammal_precomputed_climate(frame, sp, data_dir, spec) -mammal_precomputed_climate <- function(frame, species, data_dir, spec) { - queue <- data.table::fread( - file.path(data_dir, "lookup", "mammal_modelled_species.csv") - ) - source_name <- unique(queue$species_season[queue$species == species]) - - if (length(source_name) == 0) { - stop("No v2 name for mammal species `", species, "`.", - call. = FALSE) - } - +mammal_precomputed_climate <- function(frame, species, data_dir, + spec) { # The climate file names species as v2 did, without the season - v2_name <- sub("(Summer|Winter)$", "", source_name[1]) + v2_name <- sub( + "(Summer|Winter)$", "", + species_source_name(data_dir, "mammal", species) + ) - climate <- data.table::fread( + # Read whole once per session; every species reads a column + climate <- cached_read( file.path(data_dir, "lookup", "mammal_climate_predictions.csv"), - select = c("survey_unit_id", v2_name) + function(p) as.data.frame(data.table::fread(p)) ) + if (!v2_name %in% names(climate)) { + stop("No precomputed climate for mammal `", v2_name, "`.", + call. = FALSE) + } + frame$Climate <- climate[[v2_name]][ match(frame$survey_unit_id, climate$survey_unit_id) ] @@ -95,7 +87,7 @@ mammal_precomputed_climate <- function(frame, species, data_dir, spec) { frame } -# 3. The hurdle model ---- +# 3. Helpers ---- ## 3.1 mammal_aicc() ---- @@ -132,41 +124,67 @@ mammal_lure_ratio <- function(values, lured, location) { unname(q["Yes"] / q["No"]) } -## 3.3 select_mammal_v2_hurdle() ---- +## 3.3 mammal_held_constants() ---- -#' Fit v2's Mammal Hurdle Model for One Species and Season +#' The Values v2 Holds Fixed for Its One-Hot Predictions #' -#' Registered as the mammal habitat stage's selection rule. See -#' the header for the steps. +#' @param has_aspen Logical. Whether the region fits pAspen (the +#' south). +#' @return A named list. #' -#' @param models List of presence candidate formulas. -#' @param base,engine,family,weights,offset,ic,control As passed -#' to every selection rule; `weights` is the season weight. -#' @param data The draw's rows: `response_raw` is the count, and -#' `lured`, `location`, `seas_days`, `Climate`, the habitat -#' columns and, in the north, the aged stand-cover columns. +#' @example # Example usage of the function +#' # mammal_held_constants(TRUE) +mammal_held_constants <- function(has_aspen) { + constants <- list(seas_days = 100, Climate = 0) + if (has_aspen) constants$pAspen <- 0 + constants +} + +# 4. The v2 tables ---- + +## 4.1 mammal_v2_habitat_tables() ---- + +#' Turn the Fitted Hurdle into v2's Three Habitat Tables +#' +#' Replaces the `hurdle` rule's outputs with v2's presence, +#' abundance and total tables. +#' +#' @param selected The `hurdle` rule's selection result. +#' @param data The stage's data: `response` is the lure-scaled +#' presence, `response_raw` the count, `weight` the season +#' weight, and `lured`, `location`, `seas_days`, `Climate`, the +#' habitat columns and, in the north, the aged stand-cover +#' columns. #' @param grid The prediction matrix, with v2's rows dropped. -#' @param intercept_cats Character vector: each candidate's -#' reference category. #' @param species Character. The harness species name. -#' @param stage The stage definition. Unused; the stand-age steps -#' run where the aged cover columns are present (the north). -#' @return A selection result: `fit` is the presence model, -#' `coefficients` the presence table, and `outputs` the -#' presence, abundance and total tables. +#' @return The selection result, with `coefficients` the presence +#' table and `outputs` v2's three tables. #' #' @example # Example usage of the function -#' # select_mammal_v2_hurdle(models, response ~ 1, d, engine, -#' # "binomial", grid = pm, ...) -select_mammal_v2_hurdle <- function( - models, base, data, engine, family, - weights = NULL, offset = NULL, ic = "AICc", control = list(), - grid = NULL, intercept_cats = NULL, species = NA_character_, - stage = list() -) { +#' # selected <- mammal_v2_habitat_tables(selected, d, pm, +#' # "MuleDeer_Summer") +mammal_v2_habitat_tables <- function(selected, data, grid, + species = NA_character_) { if (!requireNamespace("MuMIn", quietly = TRUE) || !requireNamespace("mgcv", quietly = TRUE)) { - stop("The mammal hurdle model needs MuMIn and mgcv.", + stop("The mammal v2 tables need MuMIn and mgcv.", call. = FALSE) + } + + pa <- selected$parts$presence$fit$fit + agp <- selected$parts$abundance$fit$fit + + # Needs GLM coefficients; other engines should not reach here + if (!inherits(pa, "glm") || !inherits(agp, "glm")) { + stop( + "The mammal v2 tables read GLM fits, but this hurdle was ", + "fitted with another engine. Replace the stage's method ", + "with replace_stage_method(), which removes this step.", + call. = FALSE + ) + } + + if (is.null(grid)) { + stop("The mammal v2 tables need the prediction matrix.", call. = FALSE) } @@ -175,94 +193,105 @@ select_mammal_v2_hurdle <- function( ages <- "SpruceR" %in% names(data) has_aspen <- "pAspen" %in% names(data) count <- data$response_raw - seas_wt <- weights - fail <- function(message) { - stop("mammal hurdle: ", message, call. = FALSE) - } - - # Step 1: Presence (6.1) lure_pa <- mammal_lure_ratio(sign(count), data$lured, data$location) - data$p_count_pa <- sign(count) / ifelse(data$lured == "Yes", lure_pa, 1) - data$p_count_pa <- data$p_count_pa / max(data$p_count_pa) + lure_agp <- mammal_lure_ratio( + count[count > 0], data$lured[count > 0], data$location[count > 0] + ) - formulas <- lapply(models, function(m) { - stats::update(p_count_pa ~ 1, m) - }) + # Step 1: Presence effects, one category at a time (6.3) + presence <- mammal_presence_effects( + pa, data, selected$reference_category, lure_pa, has_aspen + ) - m_pa <- lapply(formulas, function(f) { - try(stats::glm(f, family = "binomial", data = data, - weights = seas_wt), silent = TRUE) - }) - aic_pa <- vapply(m_pa, mammal_aicc, numeric(1)) + # Site-level logits: the age splines' offset + data$p <- stats::predict(pa, newdata = data) + + # Step 2: Abundance effects on the prediction matrix (6.4) + abundance <- mammal_abundance_effects(agp, grid, has_aspen) - if (!any(is.finite(aic_pa))) { - fail("every presence model failed") + # Step 3: Stand-age splines, north only (6.5) + aged <- if (ages) { + mammal_age_splines(data, presence$coef, presence$terms) + } else { + list(p_age = list(), p_age_se = list()) } - best_pa <- which.min(aic_pa) - pa <- m_pa[[best_pa]] - - # Step 2: Abundance given presence (6.2) - d_p <- data[count > 0, , drop = FALSE] - count_p <- count[count > 0] - lure_agp <- mammal_lure_ratio(count_p, d_p$lured, d_p$location) - d_p$p_count_agp <- count_p / - ifelse(d_p$lured == "Yes", lure_agp, 1) - d_p$p_count_agp <- pmin( - d_p$p_count_agp, stats::quantile(d_p$p_count_agp, 0.99) + # Step 4: Spread each category over its fine types (6.5) + spread <- mammal_spread( + presence, abundance$coef, aged, grid, ages ) - agp_start <- rep(mean(d_p$p_count_agp), nrow(d_p)) - m_agp <- list() + # Step 5: v2's Mule Deer adjustment + spread$mean <- mammal_mule_deer(spread$mean, species) - for (i in seq_along(m_pa)) { - if (inherits(m_pa[[i]], "try-error")) { - next - } + # Step 6: Calibrate total abundance to the observed mean (6.6) + spread$mean <- spread$mean * mammal_total_calibration( + pa, agp, data, presence$at_constants, lure_pa, lure_agp, + has_aspen + ) - terms_i <- attr(m_pa[[i]]$terms, "term.labels") - cover <- colSums(d_p[, terms_i, drop = FALSE]) - # pAspen is not cover; v2's south leaves it out of the sum - # (the north has none) - cover <- cover[ - !names(cover) %in% c("Climate", "seas_days", "pAspen") - ] - - if ((nrow(d_p) - sum(cover)) > 0) { - f <- stats::as.formula(paste( - "p_count_agp ~", paste(setdiff(terms_i, "Climate"), - collapse = " + ") - )) - m_agp[[length(m_agp) + 1]] <- try(stats::glm( - f, data = d_p, family = stats::Gamma(link = "log"), - mustart = agp_start - ), silent = TRUE) - } + # Step 7: Cutblock convergence, north only (6.11) + if (ages) { + spread$mean <- mammal_cutblock_convergence(spread$mean) } - m_agp[[length(m_agp) + 1]] <- try(stats::glm( - p_count_agp ~ seas_days, data = d_p, - family = stats::Gamma(link = "log"), mustart = agp_start - ), silent = TRUE) - - aic_agp <- vapply(m_agp, mammal_aicc, numeric(1)) + # v2 adds Bare and Water at zero to every table (7.1) + coef_agp <- abundance$coef + coef_agp_se <- abundance$se + spread$pa[c("Bare", "Water")] <- 0 + spread$pa_se[c("Bare", "Water")] <- 0 + spread$mean[c("Bare", "Water")] <- 0 + coef_agp[c("Bare", "Water")] <- 0 + coef_agp_se[c("Bare", "Water")] <- 0 - if (!any(is.finite(aic_agp))) { - fail("every abundance model failed") + table_of <- function(values, se = NA_real_) { + data.frame( + term = names(values), estimate = unname(values), + se = if (length(se) == length(values)) unname(se) else NA_real_, + stringsAsFactors = FALSE + ) } - agp <- m_agp[[which.min(aic_agp)]] + presence_table <- table_of(spread$pa, spread$pa_se) - # Step 3: Presence effects, one category at a time (6.3) - terms_pa <- c(attr(pa$terms, "term.labels"), - intercept_cats[best_pa]) - held <- c("seas_days", "Climate", if (has_aspen) "pAspen") - constants <- list(seas_days = 100, Climate = 0) - if (has_aspen) constants$pAspen <- 0 + selected$coefficients <- presence_table + selected$outputs <- list( + presence = presence_table, + abundance = table_of(coef_agp, coef_agp_se), + total = table_of(spread$mean) + ) + + selected +} + +## 4.2 mammal_presence_effects() ---- + +#' Presence per Habitat Category, Calibrated (v2 6.3) +#' +#' Every one-hot row is read at seas_days = 100, Climate = 0 and +#' pAspen = 0, including the rows for those terms themselves, as +#' v2's duplicated data-frame columns make it. The level is then +#' shifted on the logit scale so mean fitted presence matches mean +#' observed; Climate is a slope and keeps its own coefficient. +#' +#' @param pa The presence GLM. +#' @param data The stage's data. +#' @param reference Character. The category the winning presence +#' candidate leaves out. +#' @param lure_pa Numeric. The presence lure ratio. +#' @param has_aspen Logical. +#' @return A list of `coef` and `se` (named by term), `terms`, and +#' `at_constants` (the data at v2's held values). +#' +#' @example # Example usage of the function +#' # mammal_presence_effects(pa, d, "Shrub", 1.4, FALSE) +mammal_presence_effects <- function(pa, data, reference, lure_pa, + has_aspen) { + count <- data$response_raw + terms_pa <- c(attr(pa$terms, "term.labels"), reference) + constants <- mammal_held_constants(has_aspen) + held <- names(constants) - # Every one-hot row is read at seas_days = 100, Climate = 0 and - # pAspen = 0, including the rows for those terms themselves, - # as v2's duplicated data-frame columns make it pa_link <- vapply(terms_pa, function(term) { onehot <- stats::setNames( as.list(as.numeric(terms_pa == term)), terms_pa @@ -281,13 +310,6 @@ select_mammal_v2_hurdle <- function( coef_pa[["Climate"]] <- stats::plogis(stats::coef(pa)[["Climate"]]) climate_se <- coef_pa_se[["Climate"]] - # The pAspen slope, added back in the calibration (south) - paspen_pa <- if ("pAspen" %in% names(stats::coef(pa))) { - stats::coef(pa)[["pAspen"]] - } else { - 0 - } - # Calibrate the level: mean fitted presence to mean observed at_constants <- data at_constants$seas_days <- 100 @@ -305,119 +327,170 @@ select_mammal_v2_hurdle <- function( coef_pa[["Climate"]] <- climate_pa coef_pa_se[["Climate"]] <- climate_se - # Site-level logits: the age splines' offset - data$p <- stats::predict(pa, newdata = data) + list( + coef = coef_pa, se = coef_pa_se, terms = terms_pa, + at_constants = at_constants + ) +} + +## 4.3 mammal_abundance_effects() ---- - # Step 4: Abundance effects on the prediction matrix (6.4) - pm <- grid - newdata <- pm +#' Abundance Given Presence on the Prediction Matrix (v2 6.4) +#' +#' @param agp The abundance GLM. +#' @param grid The prediction matrix. +#' @param has_aspen Logical. +#' @return A list of `coef` and `se`, on the response scale, with +#' Climate first at 1 (no effect on abundance). +#' +#' @example # Example usage of the function +#' # mammal_abundance_effects(agp, pm, FALSE) +mammal_abundance_effects <- function(agp, grid, has_aspen) { + constants <- mammal_held_constants(has_aspen) + newdata <- grid for (one in names(constants)) newdata[[one]] <- constants[[one]] agp_pred <- stats::predict(agp, newdata = newdata, se.fit = TRUE) - t_mean_agp <- stats::setNames(as.numeric(agp_pred$fit), rownames(pm)) - t_se_agp <- stats::setNames(as.numeric(agp_pred$se.fit), rownames(pm)) + t_mean_agp <- stats::setNames( + as.numeric(agp_pred$fit), rownames(grid) + ) + t_se_agp <- stats::setNames( + as.numeric(agp_pred$se.fit), rownames(grid) + ) - paspen_agp <- if ("pAspen" %in% names(stats::coef(agp))) { - stats::coef(agp)[["pAspen"]] - } else { - 0 - } + list( + coef = c(Climate = 1, exp(t_mean_agp)), + se = c(Climate = 0, t_se_agp) + ) +} - coef_agp <- c(Climate = 1, exp(t_mean_agp)) - coef_agp_se <- c(Climate = 0, t_se_agp) +## 4.4 mammal_age_splines() ---- - # Step 5: Stand-age splines, north only (6.5) +#' Presence by Stand Age, from v2's Splines (v2 6.5) +#' +#' One frame per stand type from units where an age class covers +#' more than 10%, a binomial GAM on sqrt(age) with the site-level +#' presence logit as offset, used where it beats the +#' intercept-only model by AICc weight. +#' +#' @param data The stage's data, with `p`, the site-level logits. +#' @param coef_pa Named numeric. Calibrated presence per term. +#' @param terms_pa Character. The presence terms. +#' @return A list of `p_age` and `p_age_se`, per term that took a +#' spline: nine values, R and 1 to 8. +#' +#' @example # Example usage of the function +#' # mammal_age_splines(d, coef_pa, names(coef_pa)) +mammal_age_splines <- function(data, coef_pa, terms_pa) { stand_types <- c("Spruce", "Pine", "Decid", "Mixedwood", "TreedBog") + seas_wt <- data$weight p_age <- p_age_se <- list() - - if (ages) { - stand <- stats::setNames(vector("list", 5), stand_types) - - for (i in 0:8) { - label <- if (i == 0) "R" else i - age <- if (i == 0) 0.5 else i - - for (type in stand_types) { - column <- paste0(type, label) - if (!column %in% names(data)) next - rows <- which(data[[column]] > 0.1) - if (length(rows) == 0) next - - stand[[type]] <- rbind(stand[[type]], data.frame( - p_count = data$p_count_pa[rows], age = age, - wt1 = data[[column]][rows] * seas_wt[rows], - p = data$p[rows] - )) - } + stand <- stats::setNames(vector("list", 5), stand_types) + + for (i in 0:8) { + label <- if (i == 0) "R" else i + age <- if (i == 0) 0.5 else i + + for (type in stand_types) { + column <- paste0(type, label) + if (!column %in% names(data)) next + rows <- which(data[[column]] > 0.1) + if (length(rows) == 0) next + + stand[[type]] <- rbind(stand[[type]], data.frame( + p_count = data$response[rows], age = age, + wt1 = data[[column]][rows] * seas_wt[rows], + p = data$p[rows] + )) } + } - frames <- c(stand, list( - UpCon = rbind(stand$Spruce, stand$Pine), - DecidMixed = rbind(stand$Decid, stand$Mixedwood), - TreedAll = do.call(rbind, stand) + frames <- c(stand, list( + UpCon = rbind(stand$Spruce, stand$Pine), + DecidMixed = rbind(stand$Decid, stand$Mixedwood), + TreedAll = do.call(rbind, stand) + )) + + fit_gam <- function(df, spline) { + f <- if (spline) { + p_count ~ s(sqrt(age), k = 4, m = 2) + offset(p) + } else { + p_count ~ 1 + offset(p) + } + suppressWarnings(mgcv::gam( + f, data = df, family = "binomial", weights = df$wt1 )) + } - fit_gam <- function(df, spline) { - f <- if (spline) { - p_count ~ s(sqrt(age), k = 4, m = 2) + offset(p) - } else { - p_count ~ 1 + offset(p) - } - suppressWarnings(mgcv::gam( - f, data = df, family = "binomial", weights = df$wt1 - )) + null_age <- lapply(frames, fit_gam, spline = FALSE) + spline_age <- lapply(seq_along(frames), function(i) { + if (sum(sign(frames[[i]]$p_count)) > 4) { + fit_gam(frames[[i]], TRUE) + } else { + null_age[[i]] } + }) + names(spline_age) <- names(frames) + + # The spline is used where it beats the null by AICc weight + spline_weight <- vapply(names(frames), function(nm) { + q <- c(mammal_aicc(spline_age[[nm]]), mammal_aicc(null_age[[nm]])) + q <- q - min(q) + exp(-0.5 * q[1]) / sum(exp(-0.5 * q)) + }, numeric(1)) + + age_map <- list( + Spruce = "Spruce", Pine = "Pine", Decid = "Decid", + Mixedwood = "Mixedwood", TreedBog = "TreedBog", + TreedWet = "TreedBog", UpCon = "UpCon", + DecidMixed = "DecidMixed", UplandForest = "TreedAll", + TreedAll = "TreedAll" + ) - null_age <- lapply(frames, fit_gam, spline = FALSE) - spline_age <- lapply(seq_along(frames), function(i) { - if (sum(sign(frames[[i]]$p_count)) > 4) { - fit_gam(frames[[i]], TRUE) - } else { - null_age[[i]] - } - }) - names(spline_age) <- names(frames) - - # The spline is used where it beats the null by AICc weight - spline_weight <- vapply(names(frames), function(nm) { - q <- c(mammal_aicc(spline_age[[nm]]), mammal_aicc(null_age[[nm]])) - q <- q - min(q) - exp(-0.5 * q[1]) / sum(exp(-0.5 * q)) - }, numeric(1)) - - age_map <- list( - Spruce = "Spruce", Pine = "Pine", Decid = "Decid", - Mixedwood = "Mixedwood", TreedBog = "TreedBog", - TreedWet = "TreedBog", UpCon = "UpCon", - DecidMixed = "DecidMixed", UplandForest = "TreedAll", - TreedAll = "TreedAll" - ) + smooth <- function(y) { + c(mean(y[1:2]), y[2:4], mean(y[4:6]), mean(y[5:7]), + mean(y[6:8]), mean(y[7:9]), mean(y[8:9])) + } - smooth <- function(y) { - c(mean(y[1:2]), y[2:4], mean(y[4:6]), mean(y[5:7]), - mean(y[6:8]), mean(y[7:9]), mean(y[8:9])) - } + for (term in names(age_map)) { + model_key <- age_map[[term]] + if (!term %in% terms_pa) next + if (spline_weight[[model_key]] <= 0.5) next - for (term in names(age_map)) { - model_key <- age_map[[term]] - if (!term %in% terms_pa) next - if (spline_weight[[model_key]] <= 0.5) next - - predicted <- stats::predict( - spline_age[[model_key]], - newdata = data.frame( - age = c(0.5, 1:8), p = stats::qlogis(coef_pa[[term]]) - ), - se.fit = TRUE - ) - p_age[[term]] <- smooth(stats::plogis(as.numeric(predicted$fit))) - p_age_se[[term]] <- smooth(as.numeric(predicted$se.fit)) - } + predicted <- stats::predict( + spline_age[[model_key]], + newdata = data.frame( + age = c(0.5, 1:8), p = stats::qlogis(coef_pa[[term]]) + ), + se.fit = TRUE + ) + p_age[[term]] <- smooth(stats::plogis(as.numeric(predicted$fit))) + p_age_se[[term]] <- smooth(as.numeric(predicted$se.fit)) } - # Step 6: Spread each category over its fine types (6.5) + list(p_age = p_age, p_age_se = p_age_se) +} + +## 4.5 mammal_spread() ---- + +#' Spread Each Category over Its Fine Types (v2 6.5) +#' +#' @param presence A mammal_presence_effects() result. +#' @param coef_agp Named numeric. Abundance per fine type. +#' @param aged A mammal_age_splines() result. +#' @param grid The prediction matrix. +#' @param ages Logical. Whether the region has stand ages. +#' @return A list of `pa`, `pa_se` and `mean` (presence times +#' abundance), each named by fine type. +#' +#' @example # Example usage of the function +#' # mammal_spread(presence, coef_agp, aged, pm, TRUE) +mammal_spread <- function(presence, coef_agp, aged, grid, ages) { + stand_types <- c("Spruce", "Pine", "Decid", "Mixedwood", "TreedBog") + coef_pa <- presence$coef + coef_pa_se <- presence$se out_pa <- out_pa_se <- out_mean <- numeric(0) - terms_pa1 <- setdiff(terms_pa, c("seas_days", "pAspen")) + terms_pa1 <- setdiff(presence$terms, c("seas_days", "pAspen")) put <- function(names_in, pa_values, pa_se, agp_value) { out_pa[names_in] <<- pa_values @@ -431,17 +504,22 @@ select_mammal_v2_hurdle <- function( next } - if (!term %in% names(pm)) next - fine_types <- rownames(pm)[pm[[term]] == 1] + if (!term %in% names(grid)) next + fine_types <- rownames(grid)[grid[[term]] == 1] for (fine in fine_types) { - agp_value <- if (fine %in% names(coef_agp)) coef_agp[[fine]] else 0 + agp_value <- if (fine %in% names(coef_agp)) { + coef_agp[[fine]] + } else { + 0 + } if (ages && fine %in% stand_types) { age_names <- paste0(fine, c("R", 1:8)) - if (!is.null(p_age[[term]])) { - put(age_names, p_age[[term]], p_age_se[[term]], agp_value) + if (!is.null(aged$p_age[[term]])) { + put(age_names, aged$p_age[[term]], aged$p_age_se[[term]], + agp_value) } else { put(age_names, rep(coef_pa[[term]], 9), rep(coef_pa_se[[term]], 9), agp_value) @@ -452,92 +530,120 @@ select_mammal_v2_hurdle <- function( } } - # v2's Mule Deer adjustment: old pine, and in winter old - # deciduous, stands are pulled toward the 100-119 class + list(pa = out_pa, pa_se = out_pa_se, mean = out_mean) +} + +## 4.6 mammal_mule_deer() ---- + +#' v2's Mule Deer Adjustment to Old Stands +#' +#' Old pine, and in winter old deciduous, stands are pulled +#' toward the 100-119 class. +#' +#' @param out_mean Named numeric. Total abundance per fine type. +#' @param species Character. The harness species name. +#' @return `out_mean`, adjusted for Mule Deer and unchanged +#' otherwise. +#' +#' @example # Example usage of the function +#' # mammal_mule_deer(out_mean, "MuleDeer_Winter") +mammal_mule_deer <- function(out_mean, species) { base_species <- sub("_(Summer|Winter)$", "", species) - if (identical(base_species, "MuleDeer")) { - types <- c("Pine", if (grepl("_Winter$", species)) "Decid") - for (type in types) { - old <- paste0(type, c("7", "8")) - ref <- paste0(type, "6") - if (all(c(old, ref) %in% names(out_mean))) { - out_mean[old] <- (out_mean[old] + out_mean[[ref]]) / 2 - } + if (!identical(base_species, "MuleDeer")) { + return(out_mean) + } + + types <- c("Pine", if (grepl("_Winter$", species)) "Decid") + + for (type in types) { + old <- paste0(type, c("7", "8")) + ref <- paste0(type, "6") + if (all(c(old, ref) %in% names(out_mean))) { + out_mean[old] <- (out_mean[old] + out_mean[[ref]]) / 2 } } - # Step 7: Calibrate total abundance to the observed mean (6.6) + out_mean +} + +## 4.7 mammal_total_calibration() ---- + +#' The Factor That Calibrates Total Abundance (v2 6.6) +#' +#' Mean observed lure-corrected count over mean predicted +#' presence times abundance, both at v2's held values with the +#' pAspen slopes added back. +#' +#' @param pa,agp The presence and abundance GLMs. +#' @param data The stage's data. +#' @param at_constants The data at v2's held values. +#' @param lure_pa,lure_agp Numeric. The two lure ratios. +#' @param has_aspen Logical. +#' @return A number. +#' +#' @example # Example usage of the function +#' # mammal_total_calibration(pa, agp, d, at, 1.4, 1.1, FALSE) +mammal_total_calibration <- function(pa, agp, data, at_constants, + lure_pa, lure_agp, has_aspen) { + slope <- function(model) { + if ("pAspen" %in% names(stats::coef(model))) { + stats::coef(model)[["pAspen"]] + } else { + 0 + } + } + aspen <- if (has_aspen) data$pAspen else 0 + count <- data$response_raw + t_p_pa <- stats::plogis( - stats::predict(pa, newdata = at_constants) + - paspen_pa * (if (has_aspen) data$pAspen else 0) + stats::predict(pa, newdata = at_constants) + slope(pa) * aspen ) t_p_agp <- stats::predict(agp, newdata = at_constants) + - paspen_agp * (if (has_aspen) data$pAspen else 0) + slope(agp) * aspen p_obs_total <- ifelse( data$lured == "Yes", count / (lure_pa * lure_agp), count ) - out_mean <- out_mean * - (mean(p_obs_total) / mean(t_p_pa * exp(t_p_agp))) - # Step 8: Cutblock convergence, north only (6.11) - if (ages) { - convergence <- list( - CCSpruce = list("Spruce", c(0.500, 0.849, 0.960)), - CCPine = list("Pine", c(0.500, 0.849, 0.960)), - CCMixedwood = list("Mixedwood", c(0.705, 0.912, 0.970)), - CCDecid = list("Decid", c(0.705, 0.912, 0.970)) - ) + mean(p_obs_total) / mean(t_p_pa * exp(t_p_agp)) +} - for (cc in names(convergence)) { - for (k in 1:3) { - age <- (2:4)[k] - w <- convergence[[cc]][[2]][k] - cc_name <- paste0(cc, age) - natural <- paste0(convergence[[cc]][[1]], age) +## 4.8 mammal_cutblock_convergence() ---- - if (all(c(cc_name, natural) %in% names(out_mean))) { - out_mean[[cc_name]] <- out_mean[[cc_name]] * (1 - w) + - out_mean[[natural]] * w - } - } - } - } +#' Converge Young Cutblocks onto Natural Stands (v2 6.11) +#' +#' The same recovery weights as the plant pipeline, applied to +#' age classes 2 to 4. +#' +#' @param out_mean Named numeric. Total abundance per fine type. +#' @return `out_mean`, converged. +#' +#' @example # Example usage of the function +#' # mammal_cutblock_convergence(out_mean) +mammal_cutblock_convergence <- function(out_mean) { + recovery <- cutblock_recovery_weights() + convergence <- list( + CCSpruce = list("Spruce", recovery$conifer), + CCPine = list("Pine", recovery$conifer), + CCMixedwood = list("Mixedwood", recovery$deciduous), + CCDecid = list("Decid", recovery$deciduous) + ) - # v2 adds Bare and Water at zero to every table (7.1) - out_pa[c("Bare", "Water")] <- 0 - out_pa_se[c("Bare", "Water")] <- 0 - out_mean[c("Bare", "Water")] <- 0 - coef_agp[c("Bare", "Water")] <- 0 - coef_agp_se[c("Bare", "Water")] <- 0 + for (cc in names(convergence)) { + for (k in 1:3) { + age <- k + 1 + w <- convergence[[cc]][[2]][k] + cc_name <- paste0(cc, age) + natural <- paste0(convergence[[cc]][[1]], age) - table_of <- function(values, se = NA_real_) { - data.frame( - term = names(values), estimate = unname(values), - se = if (length(se) == length(values)) unname(se) else NA_real_, - stringsAsFactors = FALSE - ) + if (all(c(cc_name, natural) %in% names(out_mean))) { + out_mean[[cc_name]] <- out_mean[[cc_name]] * (1 - w) + + out_mean[[natural]] * w + } + } } - presence <- table_of(out_pa, out_pa_se) - abundance <- table_of(coef_agp, coef_agp_se) - - result <- selection_result( - fit = list(fit = pa, ok = TRUE, message = NA_character_), - coefficients = presence, - ic_table = data.frame( - model = paste0("pa_", seq_along(aic_pa)), ic = aic_pa, - ok = is.finite(aic_pa), stringsAsFactors = FALSE - ) - ) - - result$outputs <- list( - presence = presence, - abundance = abundance, - total = table_of(out_mean) - ) - - result + out_mean } # End of script ---- diff --git a/1_code/modules/mammals/spec.R b/1_code/modules/mammals/spec.R index 0d51da6..fd62ed4 100644 --- a/1_code/modules/mammals/spec.R +++ b/1_code/modules/mammals/spec.R @@ -5,28 +5,16 @@ # inputs: none # outputs: none; returns objects in memory # notes: -# - The v2 configuration for mammals, as data. -# - v2 does not fit climate in this pipeline. It reads a -# prediction from mammal_climate_predictions.csv, produced by -# 1_code/2_habitat-modeling/climate/ in ABbiodiversity/ -# MammalModels: nine binomial GLMs over FFP, MAP, CMD and TD, -# averaged by AICc weight. That pipeline is adapted here as a -# fitted stage, so a bioclimatic question can be asked of -# mammals at all. `climate_source` selects which, but only -# "fitted" is implemented: nothing reads the prediction file -# yet. -# - The climate block, CMD and TD included, comes from -# abmi-camera-climate_2023.Rdata, the file v2 fitted against, -# joined by the harmonizer. The fitted stage runs the full -# nine-model v2 set. -# - What this spec reproduces of v2 is stated once, in -# `v2_coverage`. The report is generated from it. -# - North and south are separate models on overlapping -# deployments, and their covariates disagree, so the two are -# separate covariate keys rather than a filter on one table. -# - Weights are season days. The season a species is modelled -# in is part of its name, so the weight column follows the -# species rather than the taxon. +# - v2 reads mammal climate from mammal_climate_predictions.csv +# (ABbiodiversity/MammalModels, 1_code/2_habitat-modeling/ +# climate/: nine binomial GLMs over FFP, MAP, CMD, TD, AICc +# averaged). `climate_source = "fitted"` instead fits that set +# as a stage, on the climate from +# abmi-camera-climate_2023.Rdata that v2 used. +# - North and south are separate covariate keys: overlapping +# deployments with disagreeing covariates. +# - Weights are season days, so the weight column follows the +# season in the species name. # --- # 1. Setup ---- @@ -38,20 +26,11 @@ #' Standardise Detections to the No-Lure Scale #' -#' A lured camera detects more, so a raw detection is not -#' comparable across lured and unlured deployments. v2 estimates -#' the lure effect from the numbered ABMI grid sites alone - -#' those are the ones deployed in matched lured and unlured pairs, -#' so the ratio between them is a lure effect rather than a -#' difference in where cameras were put - and divides the lured -#' detections by it. -#' -#' The result is then scaled to a maximum of one, which is what -#' makes it a proportion a binomial model can take. -#' -#' A ratio that cannot be estimated - no lured or no unlured grid -#' sites for this species and season - leaves the detections -#' uncorrected rather than dividing by zero or by NA. +#' As v2: lured detections are divided by the lure ratio +#' estimated from the numbered ABMI grid sites (matched lured and +#' unlured pairs), then scaled to a maximum of one for a binomial +#' model. If the ratio cannot be estimated, detections are left +#' uncorrected. #' #' @param values Numeric vector of counts. #' @param frame The assembled frame, carrying `lured` and @@ -63,8 +42,7 @@ mammal_pa_response <- function(values, frame) { detected <- sign(values) - # Numbered locations are the ABMI grid; other projects are - # not part of the lure comparison. + # Numbered locations are the ABMI grid grid_site <- grepl("^[[:digit:]]+", as.character(frame$location)) lured <- as.character(frame$lured) == "Yes" @@ -95,17 +73,9 @@ mammal_pa_response <- function(values, frame) { #' Standardise Abundance to the No-Lure Scale #' -#' The abundance half of the hurdle. Lure is corrected as for -#' presence, but from the ratio of mean counts rather than mean -#' detections, because the question is how many rather than -#' whether. -#' -#' The result is winsorized at the 99th percentile of the -#' positive counts. A handful of density estimates are genuinely -#' enormous, and a Gamma model fitted on the log scale gives them -#' leverage out of all proportion to how much they say about a -#' habitat type. Capping is a judgement that those points are -#' real but uninformative, not that they are wrong. +#' Lure-corrected from the ratio of mean counts, then winsorized +#' at the 99th percentile of positive counts: a few enormous +#' densities would otherwise dominate the log-link Gamma fit. #' #' @param values Numeric vector of counts. #' @param frame The assembled frame, carrying `lured` and @@ -149,6 +119,10 @@ mammal_agp_response <- function(values, frame) { #' Build the v2 Spec for Mammals #' +#' The habitat stage is the generic `hurdle` rule with v2's GLMs +#' (best by AICc), turned into v2's tables by +#' mammal_v2_habitat_tables() (hurdle.R). +#' #' @param climate_source Character. "precomputed", the v2 #' default, reads each species' climate prediction from #' mammal_climate_predictions.csv and drops the deployments @@ -159,46 +133,18 @@ mammal_agp_response <- function(values, frame) { #' @param tier Character. "modelled" is the full habitat model, #' fitted for species with at least 20 detections; "ua" is the #' use-availability model, at least 3. -#' @param part Character. "hurdle", v2's model, fits both halves -#' together (hurdle.R) and reports presence, abundance and -#' total abundance on v2's full habitat set, with the stand-age -#' splines and cutblock convergence. "presence" and -#' "abundance" fit one half alone with the harness's generic -#' rules and report the winning model's own categories. #' @return A spec list, as run_spec() consumes. #' #' @example # Example usage of the function #' # spec <- mammal_spec() -#' # spec <- mammal_spec(climate_source = "fitted", -#' # part = "presence") +#' # spec <- mammal_spec(climate_source = "fitted") mammal_spec <- function( climate_source = c("precomputed", "fitted"), season = c("summer", "winter"), - tier = "modelled", - part = c("hurdle", "presence", "abundance") + tier = "modelled" ) { climate_source <- match.arg(climate_source) season <- match.arg(season) - part <- match.arg(part) - hurdle <- part == "hurdle" - - # The abundance half fits the same candidates with Climate - # dropped - v2's note is that climate effects on abundance - # given presence are minimal - plus a null carrying only - # sampling effort. - habitat_models <- function(set) { - models <- get_model_set(set) - - # The hurdle rule derives its abundance candidates itself - if (part %in% c("presence", "hurdle")) { - return(models) - } - - c( - gsub(" \\+ Climate", "", models), - ". ~ . + seas_days" - ) - } stages <- list() @@ -206,17 +152,13 @@ mammal_spec <- function( stages <- list( list( name = "climate", - # The full v2 set. CMD and TD come from the camera - # climate file, which the harmonizer now joins. + # The full v2 set (CMD and TD from the camera climate file) models = "climate_bird_mammal_v2", engine = "glm", selection = "aic_average", ic = "AICc", carry_as = "Climate", - # Climate is modelled on presence whichever half of the - # hurdle is being fitted: the abundance half takes the - # same climate offset, and fitting a binomial on counts - # would fail outright. + # Fitted on presence; the abundance half reuses it response_transform = mammal_pa_response ) ) @@ -225,15 +167,11 @@ mammal_spec <- function( list( taxon = "mammal", response_name = "response", + # The stage a methods experiment replaces by default + habitat_stage = "habitat", - # v2 models presence rather than the recorded density. The - # hurdle rule recomputes both halves' responses from the raw - # count itself. - response_transform = if (part == "abundance") { - mammal_agp_response - } else { - mammal_pa_response - }, + # Presence; the abundance half reads the raw count + response_transform = mammal_pa_response, # v2 reads climate from a separate pipeline's prediction species_frame = if (climate_source == "precomputed") { @@ -249,8 +187,7 @@ mammal_spec <- function( family = "binomial", weight_column = paste0("wt_", season), - # The formulas fit `seas_days`; which column that is depends - # on the season being modelled. + # `seas_days` is the season's days column aliases = list(seas_days = paste0(season, "_days")), tier = tier, season = season, @@ -271,17 +208,13 @@ mammal_spec <- function( grid_drop_rows = c("WetlandMargin", "Climate"), grid_drop_cols = "WetlandMargin", term_block = "veg", - habitat_models = habitat_models("habitat_mammal_north_pa_v2"), + habitat_models = "habitat_mammal_north_pa_v2", intercept_cats = "intercept_mammal_north_pa_v2", - # The stand-age splines read the aged cover columns - extra_covariates = if (hurdle) { - as.vector(t(outer( - c("Spruce", "Pine", "Decid", "Mixedwood", "TreedBog"), - c("R", 1:8), paste0 - ))) - } else { - NULL - } + # The v2 stand-age splines read the aged cover columns + extra_covariates = as.vector(t(outer( + c("Spruce", "Pine", "Decid", "Mixedwood", "TreedBog"), + c("R", 1:8), paste0 + ))) ), south = list( # The south drops all-water deployments instead @@ -294,88 +227,54 @@ mammal_spec <- function( grid_drop_rows = "Climate", term_block = "soil", # v2's 30 south candidates, from south-models/00_models.R - habitat_models = habitat_models("habitat_mammal_south_pa_v2"), + habitat_models = "habitat_mammal_south_pa_v2", intercept_cats = "intercept_mammal_south_pa_v2", - # Half the south candidates carry pAspen. Like sampling - # effort, it is held fixed for the one-hot predictions - # rather than read as a habitat type. + # pAspen (in half the candidates) is held fixed on the + # grid and is not cover constants = list(seas_days = 100, Climate = 0, pAspen = 0) ) ), stages = c(stages, list( - if (hurdle) { - list( - name = "habitat", - models = NULL, - engine = "glm", - # Both halves of v2's hurdle together; see hurdle.R. - # Writes stages habitat_presence, habitat_abundance and - # habitat_total. - selection = select_mammal_v2_hurdle, - ic = "AICc", - carry_from = "climate", - carry_from_as = "Climate" - ) - } else if (part == "presence") { - list( - name = "habitat", - models = NULL, - engine = "glm", - selection = "aic_best_onehot", - ic = "AICc", - carry_from = "climate", - carry_from_as = "Climate", - # Each candidate leaves one land cover out; the rule - # reports it, because every other effect is relative - # to it. Set per region, with the region's models. - constants = list(seas_days = 100, Climate = 0), - slope_terms = "Climate", - # v2 shifts the whole set on the logit scale so mean - # fitted presence matches mean observed. Without it - # the effects are internally consistent but sit at - # the wrong level. - calibrate = TRUE - ) - } else { - list( - name = "habitat", - models = NULL, - engine = "glm", - selection = "aic_best_grid", - ic = "AICc", - # Gamma on the log scale. mustart is set to the mean - # rather than left at the default of y itself, which - # spans orders of magnitude for a right-skewed density - # and sends the fitting algorithm off. - family = stats::Gamma(link = "log"), - control = list(), - # Abundance given presence is undefined where there - # was no presence. - row_filter = function(d) d$response > 0, - constants = list(seas_days = 100, Climate = 0), - # Coef.agp.all is stored on the response scale - - # abundance, not log abundance - despite the source - # comment describing it as raw log-scale predictions. - # Its published range runs 0 to 10.4. - scale = "response" - ) - } + list( + name = "habitat", + models = NULL, + engine = "glm", + # Each half best by AICc; see methods/selection/hurdle.R + selection = "hurdle", + part_selection = "aic_best", + ic = "AICc", + carry_from = "climate", + carry_from_as = "Climate", + # The presence response, recomputed on each draw because + # the lure ratio and scaling depend on which units are in + response_transform = mammal_pa_response, + # Abundance given presence: the lure-corrected count, + # winsorized, on a log-link Gamma, unweighted, without + # Climate, plus an effort-only null - all as v2 + abundance_response = mammal_agp_response, + abundance_family = stats::Gamma(link = "log"), + abundance_drop = "Climate", + abundance_null = ". ~ . + seas_days", + abundance_weighted = FALSE, + # Each habitat type is read at v2's 100 sampling days and + # zero climate + grid_constants = list(seas_days = 100, Climate = 0), + # v2's own tables from the two GLMs: the full habitat + # set, calibration, stand-age splines, Mule Deer and + # cutblock convergence. Removed by replace_stage_method(). + post_process = list(mammal_v2_habitat_tables) + ) )), resample = list( scheme = "spatial_block", min_detections = 20L - # No seed here. The experiment sets one base seed, - # `boot_seed` in run.R, and each species derives its own - # from it. Without one the draws are unseeded, as in v2. + # No seed here: the experiment's seed is used ), climate_source = climate_source, - part = part, - # What this spec reproduces of v2, per stage. The single - # source of truth for coverage: the report reads it. v2_coverage = list( climate = if (climate_source == "precomputed") { v2_status( @@ -395,25 +294,15 @@ mammal_spec <- function( ) ) }, - habitat = if (hurdle) { - v2_status( - "reproduced", - paste( - "v2's hurdle: presence, abundance and total abundance", - "on the full habitat set, with the stand-age splines,", - "calibration and cutblock convergence. Matches v2's", - "published tables to 2e-14, north and south." - ) - ) - } else { - v2_status( - "partial", - paste( - "One half of the hurdle alone, reporting the winning", - "model's own categories; use part = \"hurdle\" for v2." - ) + habitat = v2_status( + "reproduced", + paste( + "v2's hurdle: presence, abundance and total abundance", + "on the full habitat set, with the stand-age splines,", + "calibration and cutblock convergence. Matches v2's", + "published tables to 2e-14, north and south." ) - }, + ), resampling = v2_status( "partial", paste( @@ -431,11 +320,13 @@ mammal_spec <- function( ) ), - # Facts about this run's configuration. Coverage is in - # v2_coverage, not here. notes = paste0( - "Climate fitted with the full 9-model v2 set. Season: ", - season, ". Hurdle part: ", part, ". Tier: ", tier, "." + if (climate_source == "fitted") { + "Climate fitted with the full 9-model v2 set." + } else { + "Climate read from v2's precomputed predictions." + }, + " Season: ", season, ". Tier: ", tier, "." ) ) } diff --git a/1_code/modules/soil_mites/spec.R b/1_code/modules/soil_mites/spec.R index e6661cb..910bece 100644 --- a/1_code/modules/soil_mites/spec.R +++ b/1_code/modules/soil_mites/spec.R @@ -6,24 +6,12 @@ # - 1_code/modules/_shared/plant_group.R # outputs: none; returns objects in memory # notes: -# - Data slug: "mite", not "soil_mite". The directory is named -# for the taxon; the slug is the key the response file, the -# species catalogue and the covariate catalogue all use, and -# renaming it would mean renaming harmonized data. Use "mite" -# wherever a taxon is passed to the harness, including in -# focal_species and run_taxa. -# - Soil mites are animals. They sit with the plant group -# because v2 fits them with the plant scripts, on the plant -# survey design, and the harness follows v2. -# - v2 does not fit Protocol for this taxon, passing the flag -# as FALSE. Only bryophytes and lichens get it. The asymmetry -# is v2's and is preserved here for parity, with -# `use_protocol` exposed so an experiment can equalize it. -# - The habitat model sets the shared builder uses are spliced -# from this taxon's v2 script, which is why they carry no -# Protocol term until one is inserted. -# - Anything true of every plant-group taxon belongs in -# _shared/plant_group.R, not here. +# - Data slug: "mite", not "soil_mite"; it keys the harmonized +# data, so pass "mite" wherever a taxon is named. +# - Animals, but fitted with the plant scripts on the plant +# survey design, as in v2. +# - v2 does not fit Protocol for this taxon (bryophytes and +# lichens only); `use_protocol` can equalize it across taxa. # --- # 1. Setup ---- diff --git a/1_code/modules/vascular_plants/spec.R b/1_code/modules/vascular_plants/spec.R index 46a9ad2..6f4dcd8 100644 --- a/1_code/modules/vascular_plants/spec.R +++ b/1_code/modules/vascular_plants/spec.R @@ -9,15 +9,9 @@ # - Data slug: "vascular_plant". Response file # vascular_plant.csv. v2 names this taxon with a hyphen in # its file names, which the parity gate translates. -# - The largest plant-group taxon by species count, so it is -# usually the slowest of the four to run and the one worth -# restricting with focal_species during a smoke test. -# - v2 does not fit Protocol for this taxon, passing the flag -# as FALSE. Only bryophytes and lichens get it. The asymmetry -# is v2's and is preserved here for parity, with -# `use_protocol` exposed so an experiment can equalize it. -# - Anything true of every plant-group taxon belongs in -# _shared/plant_group.R, not here. +# - The most species of the four, so the slowest to run. +# - v2 does not fit Protocol for this taxon (bryophytes and +# lichens only); `use_protocol` can equalize it across taxa. # --- # 1. Setup ---- diff --git a/1_code/tests/README.md b/1_code/tests/README.md new file mode 100644 index 0000000..74a4a3e --- /dev/null +++ b/1_code/tests/README.md @@ -0,0 +1,26 @@ +# tests + +Checks for any change to `harness/`, `methods/` or `modules/`. Run both +from the repository root. + +| File | Does | +| --- | --- | +| `run_tests.R` | Loads the framework and runs `testthat/`. Takes seconds. Dataset checks are skipped when the published dataset cannot be reached | +| `compare_stores.R` | Compares two sets of result stores, matched on keys, to a tolerance (default 1e-10) | +| `testthat/test-dataset.R` | Manifest, species queue, covariate keys, bird grid, covariate cache, `covariate_files` joins | +| `testthat/test-methods.R` | Every engine passes `check_engine()`; registry lookups; metrics | +| `testthat/test-new_experiment.R` | Slugs, numbering and scaffolding of `new_experiment()` | +| `testthat/test-results.R` | Shard merging, seeds, spatial CV folds, baseline comparison, draw summaries | +| `testthat/test-specs.R` | Every standard spec validates; `replace_stage_method()`; hurdle; error reporting; `stage_models` | + +```sh +Rscript 1_code/tests/run_tests.R +Rscript 1_code/tests/compare_stores.R [max_boot] [file,file] +``` + +The tests do not replace the parity check. For a change meant to leave +results alone, run the `parity_check` species at 5 draws to a scratch +folder and compare its stores with exp_000's, as in +[`docs/getting_started.md`](../../docs/getting_started.md#check-that-nothing-broke). +Draws are seeded per species and made in order, so draws 1–5 match +those of a 100-draw run. diff --git a/1_code/tests/compare_stores.R b/1_code/tests/compare_stores.R new file mode 100644 index 0000000..31c0806 --- /dev/null +++ b/1_code/tests/compare_stores.R @@ -0,0 +1,203 @@ +# --- +# title: Compare Two Sets of Result Stores +# author: Brendan Casey +# created: 2026-10-03 +# inputs: +# two directories laid out as // result stores, as +# run_spec() writes them +# outputs: none; returns a data frame and prints a summary +# notes: +# - The regression check for any change to the harness, methods +# or modules: a refactor that is meant to leave results alone +# should leave every stored coefficient within `tolerance`. +# - Rows are matched on their keys, not their position, so a +# change in write order is not mistaken for a change in value. +# - `max_boot` restricts both sides to the first draws. Draws +# are seeded per species and drawn in order, so draws 1-5 of a +# 5-draw run are the same rows as draws 1-5 of a 100-draw run. +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(data.table) # keyed joins on large tables (version: 1.16.4) + +# 2. compare_stores() ---- + +#' Compare the Values Two Sets of Stores Hold +#' +#' @param old_dir,new_dir Character. Directories holding +#' `//` stores. +#' @param files Character vector of store files to compare, +#' without the extension. +#' @param tolerance Numeric. Largest absolute difference that +#' still counts as a match. +#' @param max_boot Integer or NULL. Compare draws up to this one. +#' @param runs Character vector of run folders, or NULL for every +#' run present in both. +#' @return A data frame, one row per run, region and file, with +#' row counts on each side, rows only on one side, the largest +#' absolute difference and whether it matches. +#' +#' @example # Example usage of the function +#' # compare_stores("2_pipeline/_baseline/stores", +#' # "2_pipeline/exp_000_parity_v2", max_boot = 5) +compare_stores <- function( + old_dir, + new_dir, + files = c("coefficients", "metrics", "grid_predictions"), + tolerance = 1e-10, + max_boot = NULL, + runs = NULL +) { + store_dirs <- function(root) { + found <- list.files( + root, pattern = "^meta\\.json$", recursive = TRUE + ) + dirname(found) + } + + shared <- intersect(store_dirs(old_dir), store_dirs(new_dir)) + + if (!is.null(runs)) { + shared <- shared[dirname(shared) %in% runs] + } + + if (length(shared) == 0) { + stop("No store is present in both directories.", call. = FALSE) + } + + # Value columns per file; everything else is a key + value_columns <- list( + coefficients = c("estimate", "se"), + metrics = "value", + grid_predictions = "prediction", + unit_predictions = c("prediction", "observed") + ) + + read_one <- function(root, store, what) { + path <- file.path(root, store, paste0(what, ".csv")) + + if (!file.exists(path)) { + return(NULL) + } + + rows <- fread(path) + + if (!is.null(max_boot) && "boot" %in% names(rows)) { + rows <- rows[boot <= max_boot] + } + + rows + } + + out <- list() + + for (store in shared) { + for (what in files) { + old <- read_one(old_dir, store, what) + new <- read_one(new_dir, store, what) + + if (is.null(old) && is.null(new)) { + next + } + + values <- value_columns[[what]] + keys <- setdiff( + intersect(names(old) %||% names(new), names(new) %||% + names(old)), + values + ) + + row <- data.frame( + store = store, file = what, + old_rows = if (is.null(old)) 0L else nrow(old), + new_rows = if (is.null(new)) 0L else nrow(new), + only_old = NA_integer_, only_new = NA_integer_, + max_abs_difference = NA_real_, + stringsAsFactors = FALSE + ) + + if (!is.null(old) && !is.null(new)) { + # A key can repeat within a draw (a term written twice is a + # bug the comparison should surface), so rows are numbered + # within their key before joining. + old[, .dup := seq_len(.N), by = keys] + new[, .dup := seq_len(.N), by = keys] + joined <- merge( + old, new, by = c(keys, ".dup"), all = TRUE, + suffixes = c(".old", ".new") + ) + + first <- paste0(values[1], c(".old", ".new")) + row$only_old <- sum(is.na(joined[[first[2]]]) & + !is.na(joined[[first[1]]])) + row$only_new <- sum(is.na(joined[[first[1]]]) & + !is.na(joined[[first[2]]])) + + differences <- unlist(lapply(values, function(v) { + a <- joined[[paste0(v, ".old")]] + b <- joined[[paste0(v, ".new")]] + both <- !is.na(a) & !is.na(b) + # NA on one side only is a mismatch, NA on both is not + c(abs(a[both] - b[both]), + if (any(xor(is.na(a), is.na(b)))) Inf) + })) + + row$max_abs_difference <- if (length(differences) == 0) { + 0 + } else { + max(differences) + } + } + + out[[length(out) + 1]] <- row + } + } + + result <- do.call(rbind, out) + result$match <- result$old_rows == result$new_rows & + result$only_old %in% 0L & result$only_new %in% 0L & + result$max_abs_difference <= tolerance + + result +} + +`%||%` <- function(a, b) if (is.null(a)) b else a + +# 3. Command line ---- +# Rscript 1_code/tests/compare_stores.R +# [max_boot] [files] +if (sys.nframe() == 0L) { + args <- commandArgs(trailingOnly = TRUE) + + if (length(args) < 2) { + stop( + "Usage: Rscript compare_stores.R ", + "[max_boot] [file,file]", + call. = FALSE + ) + } + + result <- compare_stores( + args[1], args[2], + max_boot = if (length(args) >= 3 && args[3] != "all") { + as.integer(args[3]) + } else { + NULL + }, + files = if (length(args) >= 4) { + strsplit(args[4], ",")[[1]] + } else { + c("coefficients", "metrics", "grid_predictions") + } + ) + + print(result, row.names = FALSE) + cat( + "\n", sum(result$match), " of ", nrow(result), + " store files match.\n", sep = "" + ) +} + +# End of script ---- diff --git a/1_code/tests/run_tests.R b/1_code/tests/run_tests.R new file mode 100644 index 0000000..290cd11 --- /dev/null +++ b/1_code/tests/run_tests.R @@ -0,0 +1,45 @@ +# --- +# title: Run the Framework's Tests +# author: Brendan Casey +# created: 2026-10-03 +# inputs: +# - 1_code/tests/testthat/test-*.R +# - the published test dataset (see harness/data_source.R), +# for the dataset checks; skipped when it cannot be reached +# outputs: none; prints a pass / fail summary +# notes: +# - Fast checks of the contracts the framework relies on: every +# engine meets the engine contract, every v2 spec validates, +# the harmonized lookups are consistent, shards merge in +# order, and seeds reproduce. They take seconds, so run them +# after any change to harness/, methods/ or modules/. +# - They do not replace the parity check. A change meant to +# leave results alone is confirmed by running the parity +# species at 5 draws and comparing the stores with +# 1_code/tests/compare_stores.R; see docs/getting_started.md. +# - Run from the repository root: +# Rscript 1_code/tests/run_tests.R +# --- + +# 1. Setup ---- + +## 1.1 Load packages ---- +library(testthat) # test runner (version: 3.3.2) + +## 1.2 Load the framework ---- +source("1_code/harness/harness.R") +load_framework() + +# Unchecked, so an unreachable share skips the dataset checks +# rather than stopping the rest +data_dir <- test_dataset_dir(check = FALSE) + +# 2. Run ---- +results <- testthat::test_dir( + "1_code/tests/testthat", + env = environment(), + reporter = "summary", + stop_on_failure = TRUE +) + +# End of script ---- diff --git a/1_code/tests/testthat/test-dataset.R b/1_code/tests/testthat/test-dataset.R new file mode 100644 index 0000000..597b45a --- /dev/null +++ b/1_code/tests/testthat/test-dataset.R @@ -0,0 +1,125 @@ +# --- +# title: Tests - Dataset Lookups +# author: Brendan Casey +# created: 2026-10-03 +# inputs: the published test dataset (`data_dir`) +# outputs: none; testthat results +# notes: +# - The harmonized lookups agree with each other and with the files +# they describe. Skipped when the dataset is not present. +# - Run through 1_code/tests/run_tests.R, which loads the +# framework and sets `data_dir`. +# --- + +skip_if_not(dir.exists(data_dir), "No test dataset") + +test_that("the manifest names files that exist", { + manifest <- dataset_manifest(data_dir) + + expect_true(all(file.exists( + file.path(data_dir, manifest$response_file) + ))) + expect_true(all(file.exists(file.path( + data_dir, "lookup", + paste0(manifest$grid, "_prediction_matrix.csv") + )))) +}) + +test_that("every taxon in the queue is in the manifest", { + expect_setequal( + unique(species_catalogue(data_dir)$taxon), + unique(dataset_manifest(data_dir)$taxon) + ) +}) + +test_that("the species queue matches the source lookups", { + # The counts the three source schemas give, checked against + # the one queue _setup/09 wrote from them + plants <- data.table::fread( + file.path(data_dir, "lookup", "modelled_species.csv") + ) + expect_equal( + length(list_species(data_dir, "lichen", "north")), + sum(plants$taxon == "lichen" & plants$in_veg_models) + ) + + birds <- data.table::fread( + file.path(data_dir, "lookup", "bird_modelled_species.csv") + ) + expect_equal( + list_species(data_dir, "bird", "south"), + birds$species[birds$region == "south"] + ) + + expect_equal( + species_source_name(data_dir, "mammal", "BlackBear_Summer"), + "Black BearSummer" + ) +}) + +test_that("covariate keys come from the manifest", { + expect_equal(covariate_key("mammal", "north", data_dir), + "mammal_north") + expect_equal(covariate_key("lichen", "south", data_dir), "lichen") +}) + +test_that("the bird north grid reproduces v2's translation matrix", { + grid <- load_prediction_grid(data_dir, "bird_north") + age <- as.data.frame(data.table::fread( + file.path(data_dir, "lookup", "bird_veg_age_matrix.csv") + )) + + expect_equal(rownames(grid), age$type) + expect_equal(grid$wtAge, age$wtAge) + expect_equal( + grid$isCon * grid$wtAge, age[["isCon:wtAge"]], tolerance = 1e-12 + ) +}) + +test_that("the covariate cache returns what a fresh read does", { + clear_cache() + first <- load_covariates("lichen", data_dir, c("MAP", "FFP")) + wider <- load_covariates("lichen", data_dir, c("MAP", "CMD")) + clear_cache() + fresh <- load_covariates("lichen", data_dir, c("MAP", "CMD")) + + expect_identical(wider, fresh) + expect_identical(first$MAP, fresh$MAP) +}) + +test_that("an experiment's covariate file is checked and joined", { + # A stand-in dataset, so the check runs without the 218 MB file + fake_dir <- withr::local_tempdir() + data.table::fwrite( + data.frame(survey_unit_id = c("a", "b"), taxon = "x", MAT = 1:2), + file.path(fake_dir, "covariates.csv") + ) + good <- file.path(fake_dir, "good.csv") + data.table::fwrite( + data.frame(survey_unit_id = c("b", "a"), new_1 = c(20, 10)), + good + ) + clash <- file.path(fake_dir, "clash.csv") + data.table::fwrite( + data.frame(survey_unit_id = "a", MAT = 5), clash + ) + repeated <- file.path(fake_dir, "repeated.csv") + data.table::fwrite( + data.frame(survey_unit_id = c("a", "a"), new_2 = 1:2), repeated + ) + + expect_silent(check_covariate_files(good, fake_dir)) + expect_error(check_covariate_files(clash, fake_dir), "already in") + expect_error(check_covariate_files(repeated, fake_dir), "repeats") + expect_error( + check_covariate_files(file.path(fake_dir, "none.csv"), fake_dir), + "not found" + ) + + # Joined on the id, NA for a unit the file does not list + x <- data.frame(survey_unit_id = c("a", "b", "c")) + joined <- add_covariate_files(x, good, "new_1") + expect_equal(joined$new_1, c(10, 20, NA)) +}) + +# End of script ---- diff --git a/1_code/tests/testthat/test-methods.R b/1_code/tests/testthat/test-methods.R new file mode 100644 index 0000000..bcf86e7 --- /dev/null +++ b/1_code/tests/testthat/test-methods.R @@ -0,0 +1,102 @@ +# --- +# title: Tests - Method Contracts and Registry +# author: Brendan Casey +# created: 2026-10-03 +# inputs: none; the loaded framework +# outputs: none; testthat results +# notes: +# - Every registered method meets its contract, and the registry +# behaves as a spec relies on it to. +# - Run through 1_code/tests/run_tests.R, which loads the +# framework and sets `data_dir`. +# --- + +test_that("every registered engine passes check_engine()", { + for (name in registered_names("engine")) { + checks <- check_engine(name, quiet = TRUE) + expect_true( + all(checks$pass), + info = paste(name, ":", paste( + checks$check[!checks$pass], collapse = "; " + )) + ) + } +}) + +test_that("the v2 methods are all registered", { + expect_true(all( + c("glm", "bayesglm", "xgboost") %in% registered_names("engine") + )) + expect_true(all(c( + "single", "aic_best", "aic_average", "staged_bic", "ivw_grid", + "hurdle" + ) %in% registered_names("selection"))) + expect_true(all(c("precomputed", "spatial_block", "spatial_cv") + %in% registered_names("resampler"))) +}) + +test_that("an unknown method fails with the registered names", { + expect_error(get_engine("no_such_engine"), "Registered: .*glm") + expect_error(get_method("selection", "best"), "aic_best") +}) + +test_that("a registered method can be looked up and removed", { + register_metric( + "test_mean_prediction", + function(observed, predicted) mean(predicted), + "Test only" + ) + on.exit(unregister_method("metric", "test_mean_prediction")) + + expect_equal( + get_method("metric", "test_mean_prediction")(0, c(1, 3)), 2 + ) + expect_true("test_mean_prediction" %in% default_metrics()) +}) + +test_that("compute_metrics keeps its order and scores known cases", { + scores <- compute_metrics(c(0, 0, 1, 1), c(0.1, 0.2, 0.8, 0.9)) + + expect_equal( + scores$metric[1:7], + c("auc", "deviance_explained", "rmse", "spearman", + "calibration_slope", "prevalence", "n") + ) + expect_equal(scores$value[scores$metric == "auc"], 1) + expect_equal(scores$value[scores$metric == "prevalence"], 0.5) + # A draw with one class has no AUC, and that is not an error + expect_true(is.na( + compute_metrics(c(0, 0), c(0.1, 0.2), "auc")$value + )) +}) + +test_that("an engine without coefficients can run the single rule", { + # A stand-in for a boosted tree: predictions only + glm <- get_engine("glm") + register_engine(list( + name = "test_predict_only", + description = "Test only", + capabilities = character(0), + fit = glm$fit, + predict = glm$predict + )) + on.exit(unregister_method("engine", "test_predict_only")) + + engine <- get_engine("test_predict_only") + data <- data.frame(response = rep(c(0, 1), 50), x = 1:100) + selected <- selection_run( + "single", list(stats::as.formula(". ~ . + x")), response ~ 1, + data, engine, "binomial" + ) + + expect_null(selected$coefficients) + expect_length(selected$predict(data, "response"), 100) + + # And a rule that needs coefficients is refused before fitting + spec <- lichen_spec() + spec$stages[[1]]$engine <- "test_predict_only" + problems <- validate_spec(spec, stop_on_error = FALSE) + expect_true(any(grepl("aic_average.*needs", problems))) +}) + +# End of script ---- diff --git a/1_code/tests/testthat/test-new_experiment.R b/1_code/tests/testthat/test-new_experiment.R new file mode 100644 index 0000000..f7f92c7 --- /dev/null +++ b/1_code/tests/testthat/test-new_experiment.R @@ -0,0 +1,139 @@ +# --- +# title: Tests - Experiment Scaffolding +# author: Brendan Casey +# created: 2026-10-06 +# inputs: none; throwaway repositories under tempfile() +# outputs: none; testthat results +# notes: +# - new_experiment() numbers an experiment from every folder that +# could hold one, and scaffolds all three of its folders or none. +# Each test builds a throwaway repository, so the real one is +# never touched. +# - Run through 1_code/tests/run_tests.R, which loads the +# framework and sets `data_dir`. +# --- + +# A minimal repository: the real template, and an exp_ folder in +# each of the given roots +fake_repo <- function(existing = list()) { + root <- tempfile("repo_") + template <- file.path(root, "1_code", "experiments", "_template") + dir.create(template, recursive = TRUE) + # testthat runs from the tests folder; the framework records + # the repository root + source_template <- file.path( + getOption("sdm.project_root", "."), "1_code", "experiments", + "_template" + ) + file.copy(list.files(source_template, full.names = TRUE), template) + + for (where in names(existing)) { + dir.create(file.path(root, where, existing[[where]]), + recursive = TRUE) + } + + root +} + +test_that("a description becomes a snake_case slug", { + expect_equal(experiment_slug("Covariate scale"), "covariate_scale") + expect_equal(experiment_slug(" SoilGrids 0-5 cm! "), + "soilgrids_0_5_cm") + expect_error(experiment_slug(""), "no letters or digits") + expect_error(experiment_slug("---"), "no letters or digits") + expect_error(experiment_slug(NA_character_), "single string") + expect_error(experiment_slug(c("a", "b")), "single string") + expect_error(experiment_slug("exp_004_scale"), "exp_NNN_ prefix") +}) + +test_that("the number is one past the highest in any folder", { + expect_equal(next_experiment_number(fake_repo()), 0L) + + # A pipeline-only folder, as a removed experiment leaves, still + # holds its number + root <- fake_repo(list( + "1_code/experiments" = "exp_001_xgboost", + "2_pipeline" = "exp_004_stale", + "3_output" = "exp_002_soilgrids", + "2_pipeline/_setup" = "exp_009_nested" + )) + expect_equal(next_experiment_number(root), 5L) +}) + +test_that("new_experiment() scaffolds all three folders", { + root <- fake_repo(list("2_pipeline" = "exp_002_old")) + out <- expect_output( + new_experiment("Covariate scale", author = "Test Author", + project_root = root, open = FALSE), + "Created exp_003_covariate_scale" + ) + + expect_equal(out$id, "exp_003_covariate_scale") + + code <- file.path(root, "1_code/experiments", out$id) + expected <- c( + file.path(code, c("run.R", "README.md")), + file.path(root, "2_pipeline", out$id, + c(".gitkeep", "logs/.gitkeep")), + file.path(root, "3_output", out$id, + c("figures/.gitkeep", "tables/.gitkeep", "report.md")) + ) + expect_true(all(file.exists(expected))) + + run <- readLines(file.path(code, "run.R")) + readme <- readLines(file.path(code, "README.md")) + text <- c(run, readme) + + expect_false(any(grepl("exp_NNN|\\[Your Name\\]|\\[YYYY-MM-DD\\]", + text))) + expect_true(any(grepl( + 'id = "exp_003_covariate_scale"', run, fixed = TRUE + ))) + expect_true(any(grepl("Experiment 003 - Covariate Scale", run, + fixed = TRUE))) + expect_true(any(grepl("author: Test Author", run, fixed = TRUE))) + expect_false(any(grepl("new_experiment", run, fixed = TRUE))) + expect_equal(readme[1], "# exp_003_covariate_scale") + expect_silent(parse(file.path(code, "run.R"))) +}) + +test_that("each call takes a new number; a failure leaves none", { + root <- fake_repo(list("2_pipeline" = "exp_000_x")) + + # The same description twice: the second gets the next number + scaffold <- function() { + expect_output( + new_experiment("scale", author = "a", project_root = root, + open = FALSE), + "Created" + ) + } + scaffold() + second <- scaffold() + expect_equal(second$id, "exp_002_scale") + + # A missing template stops the call before any folder is made + expect_error( + new_experiment("other", author = "a", project_root = root, + template_dir = file.path(root, "missing"), + open = FALSE), + "Template folder not found" + ) + expect_equal(next_experiment_number(root), 3L) +}) + +test_that("a dry run writes nothing", { + root <- fake_repo() + before <- list.files(root, recursive = TRUE, include.dirs = TRUE) + + expect_output( + new_experiment("scale", project_root = root, dry_run = TRUE), + "dry run" + ) + + expect_equal( + list.files(root, recursive = TRUE, include.dirs = TRUE), before + ) +}) + +# End of script ---- diff --git a/1_code/tests/testthat/test-results.R b/1_code/tests/testthat/test-results.R new file mode 100644 index 0000000..1658aef --- /dev/null +++ b/1_code/tests/testthat/test-results.R @@ -0,0 +1,193 @@ +# --- +# title: Tests - Result Stores, Shards and Seeds +# author: Brendan Casey +# created: 2026-10-03 +# inputs: none; temporary stores under tempdir() +# outputs: none; testthat results +# notes: +# - Result stores, shards and seeds behave as a parallel run relies +# on them to. +# - Run through 1_code/tests/run_tests.R, which loads the +# framework and sets `data_dir`. +# --- + +test_that("merged shards equal a store written in one pass", { + root <- file.path(tempdir(), "shard_test") + unlink(root, recursive = TRUE) + on.exit(unlink(root, recursive = TRUE)) + + rows <- function(species) { + data.frame(term = c("a", "b"), estimate = c(0.1, 0.2), + se = c(0.01, 0.02)) + } + + # Serial: one store, appended in order + serial <- result_store(file.path(root, "serial")) + for (sp in c("sp1", "sp2")) { + for (boot in 1:2) { + write_coefficients(serial, sp, "north", boot, rows(sp), "s") + } + } + + # Parallel: a header-less shard per species, merged after + shards <- file.path(root, "shards", c("sp1", "sp2")) + for (i in 1:2) { + shard <- result_store(shards[i], header = FALSE) + for (boot in 1:2) { + write_coefficients( + shard, c("sp1", "sp2")[i], "north", boot, rows(), "s" + ) + } + } + merged <- result_store(file.path(root, "merged")) + merge_result_shards(merged, shards) + + expect_identical( + readBin(file.path(root, "serial", "coefficients.csv"), + "raw", 1e6), + readBin(file.path(root, "merged", "coefficients.csv"), + "raw", 1e6) + ) +}) + +test_that("unit predictions are written only when asked", { + root <- file.path(tempdir(), "unit_test") + unlink(root, recursive = TRUE) + on.exit(unlink(root, recursive = TRUE)) + + for (mode in c("none", "oob", "all")) { + store <- result_store(file.path(root, mode), + unit_predictions = mode) + write_unit_predictions( + store, "sp", "north", 1, c("u1", "u2", "u3"), + c(0.1, 0.2, 0.3), c(0, 1, 0), + in_bag = c(TRUE, FALSE, TRUE) + ) + } + + expect_null(read_results(file.path(root, "none"), + "unit_predictions")) + expect_equal( + read_results( + file.path(root, "oob"), "unit_predictions" + )$survey_unit_id, + "u2" + ) + expect_equal( + nrow(read_results(file.path(root, "all"), "unit_predictions")), 3 + ) +}) + +test_that("a seed gives the same draws whatever the generator", { + default <- with_seed(42, sample(100, 5)) + + old <- RNGkind() + RNGkind("L'Ecuyer-CMRG") + switched <- with_seed(42, sample(100, 5)) + after <- RNGkind()[1] + do.call(RNGkind, as.list(old)) + + expect_identical(default, switched) + # And the caller's generator is left as it was + expect_equal(after, "L'Ecuyer-CMRG") +}) + +test_that("species seeds do not depend on the other species", { + expect_identical( + species_seed(20260909L, "lichen", "Physcia.adscendens"), + species_seed(20260909L, "lichen", "Physcia.adscendens") + ) + expect_false(identical( + species_seed(20260909L, "lichen", "Physcia.adscendens"), + species_seed(20260909L, "lichen", "Cladonia.chlorophaea") + )) +}) + +test_that("spatial cross-validation holds out whole folds", { + frame <- data.frame( + survey_unit_id = paste0("u", 1:400), + long = stats::runif(400, -120, -110), + lat = stats::runif(400, 49, 60) + ) + draws <- resampler_spatial_cv( + frame, 1:3, seed = 1L, context = list() + ) + + expect_length(draws, 3) + # Every unit is held out by exactly one fold + held_out <- lapply( + draws, function(d) setdiff(frame$survey_unit_id, d) + ) + expect_equal(sum(lengths(held_out)), 400) +}) + +test_that("an experiment is compared with its baseline by key", { + root <- file.path(tempdir(), "compare_test") + unlink(root, recursive = TRUE) + on.exit(unlink(root, recursive = TRUE)) + + write_summary <- function(dir, auc, draws) { + dir.create(file.path(dir, "tables"), recursive = TRUE) + data.table::fwrite( + data.frame( + taxon = "lichen", run = c("lichen", "lichen"), + region = "north", season = NA, part = NA, + species = c("sp1", "sp2"), metric = "oob_auc", + n = draws, median = auc, p10 = auc - 0.05, + p90 = auc + 0.05, boot1 = NA + ), + file.path(dir, "tables", "metric_summary.csv") + ) + } + + write_summary(file.path(root, "base"), c(0.70, 0.80), 100) + write_summary(file.path(root, "same"), c(0.75, 0.78), 100) + write_summary(file.path(root, "short"), c(0.75, 0.78), 5) + + out <- compare_experiments( + file.path(root, "same"), file.path(root, "base") + ) + expect_equal(out$summary$share_better, 0.5) + expect_equal(out$summary$draws_baseline, 100) + expect_true(file.exists( + file.path(root, "same", "tables", "comparison_metrics.csv") + )) + expect_false(file.exists( + file.path(root, "same", "tables", "comparison_summary.csv") + )) + + # A trial against a full run is warned about + expect_warning( + compare_experiments( + file.path(root, "short"), file.path(root, "base") + ), + "different numbers of draws" + ) +}) + +test_that("the baseline does not compare with itself", { + expect_null(experiment_config("exp_000_parity_v2")$baseline) + expect_equal( + experiment_config("exp_009_test")$baseline, "exp_000_parity_v2" + ) +}) + +test_that("draw summaries carry the mean and standard deviation", { + rows <- data.frame( + species = "sp", metric = "oob_auc", boot = 1:4, + value = c(0.6, 0.7, 0.8, NA) + ) + out <- summarise_draws(rows, "value", c("species", "metric")) + + expect_equal(out$n, 3) + expect_equal(out$mean, 0.7) + expect_equal(out$sd, 0.1) + # Appended, so earlier columns keep their positions + expect_equal( + names(out), + c("species", "metric", "n", "median", "p10", "p90", "boot1", + "mean", "sd") + ) +}) + +# End of script ---- diff --git a/1_code/tests/testthat/test-specs.R b/1_code/tests/testthat/test-specs.R new file mode 100644 index 0000000..f3e12c0 --- /dev/null +++ b/1_code/tests/testthat/test-specs.R @@ -0,0 +1,130 @@ +# --- +# title: Tests - Taxon Spec Validation +# author: Brendan Casey +# created: 2026-10-03 +# inputs: none; the loaded framework +# outputs: none; testthat results +# notes: +# - The v2 specs are valid, and validate_spec() catches the mistakes +# a user is likely to make. +# - Run through 1_code/tests/run_tests.R, which loads the +# framework and sets `data_dir`. +# --- + +test_that("every standard spec validates", { + for (key in names(standard_specs())) { + expect_silent(validate_spec(standard_specs()[[key]])) + } + expect_silent(validate_spec( + mammal_spec(climate_source = "fitted", season = "summer") + )) +}) + +test_that("one call swaps the habitat method of every run", { + specs <- lapply( + standard_specs(), replace_stage_method, engine = "xgboost" + ) + + for (key in names(specs)) { + expect_silent(validate_spec(specs[[key]])) + } + + # The mammal hurdle keeps its structure; both halves take the + # new engine, and v2's GLM table-building is removed with them + habitat <- specs$mammal_summer$stages[[1]] + expect_equal(habitat$selection, "hurdle") + expect_equal(habitat$part_selection, "single") + expect_equal(habitat$engine, "xgboost") + expect_null(habitat$post_process) + expect_equal(specs$bird$stages[[2]]$engine, "xgboost") +}) + +test_that("a hurdle's inner rule is checked against its engine", { + spec <- replace_stage_method( + mammal_spec(), engine = "xgboost", selection = "aic_best" + ) + + expect_error(validate_spec(spec), "inner rule `aic_best` needs") +}) + +test_that("the hurdle fits both halves with any engine", { + set.seed(1) + n <- 300 + data <- data.frame( + x = stats::runif(n), cover = stats::rbinom(n, 1, 0.5), + seas_days = 100 + ) + data$response_raw <- stats::rbinom(n, 1, 0.4) * + stats::rgamma(n, 2, 1) + data$response <- sign(data$response_raw) + grid <- data.frame(cover = c(0, 1), row.names = c("A", "B")) + + for (engine in c("glm", "xgboost")) { + selected <- selection_run( + "hurdle", list(stats::as.formula(". ~ . + x + cover")), + response ~ 1, data, get_engine(engine), "binomial", + grid = grid, + constants = list(x = 0.5), + # seas_days is constant here, so the effort-only null + # candidate would be rank-deficient + abundance_null = NULL, + part_selection = if (engine == "glm") "aic_best" else "single" + ) + + expect_named( + selected$outputs, c("presence", "abundance", "total") + ) + expect_equal( + selected$outputs$total$estimate, + selected$outputs$presence$estimate * + selected$outputs$abundance$estimate + ) + expect_length(selected$predict(data, "response"), n) + } +}) + +test_that("a misspelt field is reported, not ignored", { + spec <- bird_spec() + spec$stages[[2]]$selction <- "staged_bic" + + expect_error(validate_spec(spec), "unknown field.*selction") +}) + +test_that("a misspelt model set is reported", { + spec <- lichen_spec() + spec$stages[[1]]$models <- "climate_plant_v2_ful" + + expect_error(validate_spec(spec), "not a model set") +}) + +test_that("every problem is reported at once", { + spec <- bird_spec() + spec$stages[[1]]$engine <- "no_such_engine" + spec$stages[[1]]$ic <- "DIC" + spec$resample$scheme <- "no_such_scheme" + + problems <- validate_spec(spec, stop_on_error = FALSE) + expect_length(problems, 3) +}) + +test_that("a rule's settings come from stage fields by name", { + # staged_bic's `threshold` is a setting no harness code names + spec <- bird_spec() + spec$stages[[2]]$threshold <- 4 + + expect_silent(validate_spec(spec)) +}) + +test_that("stage_models replaces a set and is recorded", { + spec <- apply_stage_models( + lichen_spec(), + list(climate = models_from_covariates(c("MAP", "FFP"))) + ) + + expect_equal(spec$models_overridden, "climate <- climate") + expect_setequal( + spec_covariates(spec, "climate"), c("MAP", "FFP") + ) +}) + +# End of script ---- diff --git a/3_output/exp_000_parity_v2/.gitkeep b/3_output/exp_000_parity_v2/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/3_output/exp_000_parity_v2/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/3_output/exp_000_parity_v2/figures/.gitkeep b/3_output/exp_000_parity_v2/figures/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/3_output/exp_000_parity_v2/figures/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/3_output/exp_000_parity_v2/report.md b/3_output/exp_000_parity_v2/report.md index 86b90ce..8e123ac 100644 --- a/3_output/exp_000_parity_v2/report.md +++ b/3_output/exp_000_parity_v2/report.md @@ -1,10 +1,10 @@ # exp_000_parity_v2 -Generated 2026-10-03 05:16:40 +Generated 2026-10-06 05:25:57 -**Trial run.** 14 focal species, 100 of 100 draws. Its parity numbers show the pipeline ran; they are not a parity read. +**Trial run.** 14 focal species, 5 of 100 draws. Its parity numbers show the pipeline ran; they are not a parity read. -Draws: `boot_seed = 20260909`, one derived seed per species. +Draws: `seed = 20260909`, one derived seed per species. ## What each spec reproduces of v2 @@ -47,27 +47,27 @@ Scored against `abmiexplorer`, built by `1_code/_setup/08_harmonize_abmiexplorer | source_label | reachable | rows | species | note | | --- | --- | --- | --- | --- | -| amphibian north habitat | TRUE | 364 | 4 | NA | -| amphibian south habitat | FALSE | 0 | 0 | NA | -| amphibian all climate | TRUE | 44 | 4 | NA | -| bird north habitat | TRUE | 10609 | 103 | NA | -| bird south habitat | TRUE | 1747 | 65 | NA | -| bird all climate | TRUE | 1240 | 124 | NA | -| bryophyte north habitat | TRUE | 12154 | 118 | NA | -| bryophyte south habitat | TRUE | 673 | 25 | NA | -| bryophyte all climate | TRUE | 2242 | 118 | NA | -| lichen north habitat | TRUE | 13596 | 132 | NA | -| lichen south habitat | TRUE | 1206 | 45 | NA | -| lichen all climate | TRUE | 2755 | 145 | NA | -| mammal north habitat | TRUE | 5760 | 15 | NA | -| mammal south habitat | TRUE | 1079 | 13 | NA | -| mammal all climate | TRUE | 105 | 21 | NA | -| mite north habitat | TRUE | 9785 | 95 | NA | -| mite south habitat | TRUE | 755 | 28 | NA | -| mite all climate | TRUE | 2052 | 108 | NA | -| vascular_plant north habitat | TRUE | 32754 | 318 | NA | -| vascular_plant south habitat | TRUE | 6257 | 233 | NA | -| vascular_plant all climate | TRUE | 8246 | 434 | NA | +| amphibian north habitat | TRUE | 364 | 4 | - | +| amphibian south habitat | FALSE | 0 | 0 | - | +| amphibian all climate | TRUE | 44 | 4 | - | +| bird north habitat | TRUE | 10609 | 103 | - | +| bird south habitat | TRUE | 1747 | 65 | - | +| bird all climate | TRUE | 1240 | 124 | - | +| bryophyte north habitat | TRUE | 12154 | 118 | - | +| bryophyte south habitat | TRUE | 673 | 25 | - | +| bryophyte all climate | TRUE | 2242 | 118 | - | +| lichen north habitat | TRUE | 13596 | 132 | - | +| lichen south habitat | TRUE | 1206 | 45 | - | +| lichen all climate | TRUE | 2755 | 145 | - | +| mammal north habitat | TRUE | 5760 | 15 | - | +| mammal south habitat | TRUE | 1079 | 13 | - | +| mammal all climate | TRUE | 105 | 21 | - | +| mite north habitat | TRUE | 9785 | 95 | - | +| mite south habitat | TRUE | 755 | 28 | - | +| mite all climate | TRUE | 2052 | 108 | - | +| vascular_plant north habitat | TRUE | 32754 | 318 | - | +| vascular_plant south habitat | TRUE | 6257 | 233 | - | +| vascular_plant all climate | TRUE | 8246 | 434 | - | What this run's comparison could not reach: @@ -77,20 +77,20 @@ _Every taxon and region that ran had a reference to compare against._ | taxon | run | region | season | part | species | draws | stages | has_coefficients | has_grid | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| bird | bird | north | | | 2 | 100 | climate, landcover, habitat | TRUE | FALSE | -| bird | bird | south | | | 2 | 100 | climate, landcover, habitat | TRUE | FALSE | -| bryophyte | bryophyte | north | | | 2 | 100 | climate, habitat | TRUE | FALSE | -| bryophyte | bryophyte | south | | | 2 | 100 | climate, habitat | TRUE | FALSE | -| lichen | lichen | north | | | 2 | 100 | climate, habitat | TRUE | FALSE | -| lichen | lichen | south | | | 2 | 100 | climate, habitat | TRUE | FALSE | -| mammal | mammal_summer | north | summer | hurdle | 2 | 100 | habitat_presence, habitat_abundance, habitat_total | TRUE | FALSE | -| mammal | mammal_summer | south | summer | hurdle | 2 | 100 | habitat_presence, habitat_abundance, habitat_total | TRUE | FALSE | -| mammal | mammal_winter | north | winter | hurdle | 2 | 100 | habitat_presence, habitat_abundance, habitat_total | TRUE | FALSE | -| mammal | mammal_winter | south | winter | hurdle | 2 | 100 | habitat_presence, habitat_abundance, habitat_total | TRUE | FALSE | -| mite | mite | north | | | 2 | 100 | climate, habitat | TRUE | TRUE | -| mite | mite | south | | | 2 | 100 | climate, habitat | TRUE | TRUE | -| vascular_plant | vascular_plant | north | | | 2 | 100 | climate, habitat | TRUE | TRUE | -| vascular_plant | vascular_plant | south | | | 2 | 100 | climate, habitat | TRUE | TRUE | +| bird | bird | north | - | - | 2 | 5 | climate, landcover, habitat | TRUE | TRUE | +| bird | bird | south | - | - | 2 | 5 | climate, landcover, habitat | TRUE | TRUE | +| bryophyte | bryophyte | north | - | - | 2 | 5 | climate, habitat | TRUE | TRUE | +| bryophyte | bryophyte | south | - | - | 2 | 5 | climate, habitat | TRUE | TRUE | +| lichen | lichen | north | - | - | 2 | 5 | climate, habitat | TRUE | TRUE | +| lichen | lichen | south | - | - | 2 | 5 | climate, habitat | TRUE | TRUE | +| mammal | mammal_summer | north | summer | hurdle | 2 | 5 | habitat_presence, habitat_abundance, habitat_total | TRUE | TRUE | +| mammal | mammal_summer | south | summer | hurdle | 2 | 5 | habitat_presence, habitat_abundance, habitat_total | TRUE | TRUE | +| mammal | mammal_winter | north | winter | hurdle | 2 | 5 | habitat_presence, habitat_abundance, habitat_total | TRUE | TRUE | +| mammal | mammal_winter | south | winter | hurdle | 2 | 5 | habitat_presence, habitat_abundance, habitat_total | TRUE | TRUE | +| mite | mite | north | - | - | 2 | 5 | climate, habitat | TRUE | TRUE | +| mite | mite | south | - | - | 2 | 5 | climate, habitat | TRUE | TRUE | +| vascular_plant | vascular_plant | north | - | - | 2 | 5 | climate, habitat | TRUE | TRUE | +| vascular_plant | vascular_plant | south | - | - | 2 | 5 | climate, habitat | TRUE | TRUE | ## Parity against v2, by stage @@ -102,28 +102,28 @@ fixes at a placeholder are left out of the reachable scores. | taxon | region | stage | comparison | species | terms_compared | min_draws | in_band_pct | reachable_terms | reachable_in_band_pct | median_standardized_difference | median_absolute_difference | max_absolute_difference | negligible_terms | median_spearman | median_band_ratio | verdict | indicative_verdict | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| bird | north | climate | median | 2 | 20 | 100 | 100 | 20 | 100 | 0 | 0 | 2.2e-11 | 0 | 1 | 1 | not gated (trial run) | pass | -| bryophyte | north | climate | median | 1 | 19 | 100 | 100 | 19 | 100 | 0.035 | 0 | 1.4 | 6 | 1 | 1.062 | not gated (trial run) | pass | -| lichen | north | climate | median | 2 | 38 | 100 | 100 | 38 | 100 | 0 | 0 | 0.644 | 17 | 0.993 | 0.879 | not gated (trial run) | pass | -| mite | north | climate | median | 2 | 38 | 100 | 100 | 38 | 100 | 0 | 0 | 1.95 | 20 | 0.905 | 0.953 | not gated (trial run) | pass | -| vascular_plant | north | climate | median | 2 | 38 | 100 | 100 | 38 | 100 | 0 | 0 | 0.266 | 24 | 1 | 0.97 | not gated (trial run) | pass | -| bird | south | climate | median | 2 | 20 | 100 | 100 | 20 | 100 | 0 | 0 | 2.2e-11 | 0 | 1 | 1 | not gated (trial run) | pass | -| bryophyte | south | climate | median | 1 | 19 | 100 | 100 | 19 | 100 | 0.035 | 0 | 1.4 | 6 | 1 | 1.062 | not gated (trial run) | pass | -| lichen | south | climate | median | 2 | 38 | 100 | 100 | 38 | 100 | 0 | 0 | 0.644 | 17 | 0.993 | 0.879 | not gated (trial run) | pass | -| mite | south | climate | median | 2 | 38 | 100 | 100 | 38 | 100 | 0 | 0 | 1.95 | 20 | 0.905 | 0.953 | not gated (trial run) | pass | -| vascular_plant | south | climate | median | 2 | 38 | 100 | 100 | 38 | 100 | 0 | 0 | 0.266 | 24 | 1 | 0.97 | not gated (trial run) | pass | -| bird | north | habitat | median | 2 | 178 | 100 | 100 | 168 | 100 | 0 | 0 | 2.71e-11 | 0 | 1 | 1 | not gated (trial run) | pass | -| bryophyte | north | habitat | median | 1 | 94 | 100 | 100 | 94 | 100 | 0.072 | 0.023 | 0.0808 | 0 | 0.997 | 0.911 | not gated (trial run) | pass | -| lichen | north | habitat | median | 2 | 188 | 100 | 100 | 188 | 100 | 0.091 | 0.024 | 0.118 | 0 | 0.999 | 0.988 | not gated (trial run) | pass | -| mammal | north | habitat | iteration 1 | 2 | 448 | 65 | 100 | 436 | 100 | 0 | 0 | 9.77e-15 | 0 | 1 | NA | not gated (trial run) | pass | -| mite | north | habitat | median | 1 | 94 | 100 | 100 | 94 | 100 | 0.14 | 0.035 | 0.148 | 0 | 0.999 | 0.993 | not gated (trial run) | pass | -| vascular_plant | north | habitat | median | 2 | 188 | 100 | 100 | 188 | 100 | 0.108 | 0.022 | 0.125 | 0 | 0.999 | 0.95 | not gated (trial run) | pass | -| bird | south | habitat | median | 2 | 46 | 100 | 100 | 40 | 100 | 0 | 0 | 5.61e-11 | 0 | 1 | 1 | not gated (trial run) | pass | -| bryophyte | south | habitat | median | 1 | 19 | 100 | 100 | 19 | 100 | 0.047 | 0.021 | 0.0601 | 0 | 0.993 | 1.042 | not gated (trial run) | pass | -| lichen | south | habitat | median | 2 | 40 | 100 | 100 | 40 | 100 | 0.088 | 0.029 | 0.278 | 1 | 0.994 | 0.961 | not gated (trial run) | pass | -| mammal | south | habitat | iteration 1 | 2 | 126 | 99 | 100 | 120 | 100 | 0 | 0 | 5.33e-15 | 0 | 1 | NA | not gated (trial run) | pass | -| mite | south | habitat | median | 2 | 40 | 100 | 100 | 40 | 100 | 0.068 | 0.024 | 0.464 | 1 | 0.991 | 0.975 | not gated (trial run) | pass | -| vascular_plant | south | habitat | median | 2 | 40 | 100 | 100 | 40 | 100 | 0.074 | 0.027 | 0.184 | 0 | 0.997 | 1.021 | not gated (trial run) | pass | +| bird | north | climate | median | 2 | 20 | 5 | 90.000 | 20 | 90.000 | 0.351 | 0.000 | 0.651 | 0 | 1.000 | 0.694 | not gated (store holds 5 draws) | - | +| bryophyte | north | climate | median | 1 | 19 | 5 | 94.700 | 19 | 94.700 | 0.671 | 0.000 | 1.810 | 5 | 0.991 | 0.214 | not gated (store holds 5 draws) | - | +| lichen | north | climate | median | 2 | 38 | 5 | 89.500 | 38 | 89.500 | 0.003 | 0.000 | 1.990 | 17 | 0.991 | 0.301 | not gated (store holds 5 draws) | - | +| mite | north | climate | median | 2 | 38 | 5 | 94.700 | 38 | 94.700 | 0.082 | 0.000 | 3.790 | 20 | 0.973 | 0.656 | not gated (store holds 5 draws) | - | +| vascular_plant | north | climate | median | 2 | 38 | 5 | 97.400 | 38 | 97.400 | 0.000 | 0.000 | 0.120 | 24 | 0.982 | 0.602 | not gated (store holds 5 draws) | - | +| bird | south | climate | median | 2 | 20 | 5 | 90.000 | 20 | 90.000 | 0.351 | 0.000 | 0.651 | 0 | 1.000 | 0.694 | not gated (store holds 5 draws) | - | +| bryophyte | south | climate | median | 1 | 19 | 5 | 94.700 | 19 | 94.700 | 0.671 | 0.000 | 1.810 | 5 | 0.991 | 0.214 | not gated (store holds 5 draws) | - | +| lichen | south | climate | median | 2 | 38 | 5 | 89.500 | 38 | 89.500 | 0.003 | 0.000 | 1.990 | 17 | 0.991 | 0.301 | not gated (store holds 5 draws) | - | +| mite | south | climate | median | 2 | 38 | 5 | 94.700 | 38 | 94.700 | 0.082 | 0.000 | 3.790 | 20 | 0.973 | 0.656 | not gated (store holds 5 draws) | - | +| vascular_plant | south | climate | median | 2 | 38 | 5 | 97.400 | 38 | 97.400 | 0.000 | 0.000 | 0.120 | 24 | 0.982 | 0.602 | not gated (store holds 5 draws) | - | +| bird | north | habitat | median | 2 | 178 | 5 | 92.700 | 168 | 92.300 | 0.569 | 0.023 | 2.820 | 0 | 0.989 | 0.699 | not gated (store holds 5 draws) | - | +| bryophyte | north | habitat | median | 1 | 94 | 5 | 98.900 | 94 | 98.900 | 0.360 | 0.062 | 0.256 | 0 | 0.987 | 0.502 | not gated (store holds 5 draws) | - | +| lichen | north | habitat | median | 2 | 188 | 5 | 94.700 | 188 | 94.700 | 0.251 | 0.038 | 0.289 | 0 | 0.996 | 0.642 | not gated (store holds 5 draws) | - | +| mammal | north | habitat | iteration 1 | 2 | 448 | 4 | 100.000 | 436 | 100.000 | 0.000 | 0.000 | 0.000 | 0 | 1.000 | - | not gated (trial run) | - | +| mite | north | habitat | median | 1 | 94 | 5 | 76.600 | 94 | 76.600 | 0.579 | 0.078 | 0.673 | 0 | 0.993 | 0.642 | not gated (store holds 5 draws) | - | +| vascular_plant | north | habitat | median | 2 | 188 | 5 | 91.500 | 188 | 91.500 | 0.303 | 0.034 | 0.326 | 0 | 0.997 | 0.596 | not gated (store holds 5 draws) | - | +| bird | south | habitat | median | 2 | 46 | 5 | 80.400 | 40 | 77.500 | 0.334 | 0.021 | 0.121 | 0 | 0.995 | 0.508 | not gated (store holds 5 draws) | - | +| bryophyte | south | habitat | median | 1 | 19 | 5 | 100.000 | 19 | 100.000 | 0.399 | 0.066 | 0.267 | 0 | 0.991 | 0.633 | not gated (store holds 5 draws) | - | +| lichen | south | habitat | median | 2 | 40 | 5 | 87.500 | 40 | 87.500 | 0.337 | 0.048 | 0.655 | 1 | 0.985 | 0.554 | not gated (store holds 5 draws) | - | +| mammal | south | habitat | iteration 1 | 2 | 126 | 5 | 100.000 | 120 | 100.000 | 0.000 | 0.000 | 0.000 | 0 | 1.000 | - | not gated (trial run) | - | +| mite | south | habitat | median | 2 | 40 | 5 | 95.000 | 40 | 95.000 | 0.175 | 0.030 | 0.754 | 1 | 0.992 | 0.581 | not gated (store holds 5 draws) | - | +| vascular_plant | south | habitat | median | 2 | 40 | 5 | 75.000 | 40 | 75.000 | 0.585 | 0.070 | 0.552 | 0 | 0.977 | 0.566 | not gated (store holds 5 draws) | - | Where v2 bootstraps, parity is **distributional**: v2 seeds no random draw, so two v2 runs differ, and each term is scored on @@ -134,22 +134,22 @@ is compared directly; read `median_absolute_difference`. ## Model fit -Median AUC over species. `insample` scores the units each draw fitted; `oob` the units it left out, the held-out read. `v2val` is v2's validation, scored from the coefficients as v2 does (plants only). Iteration 1 is the full data and has no out-of-bag units. +Median AUC over species, scored on each run's final model: for plants, v2's own prediction from its coefficient tables. `insample` scores the units each draw fitted; `oob` the units it left out, the held-out read. `v2val` is v2's validation, scored from the coefficients as v2 does (plants only), so for plants `v2val_full` equals `insample_auc`. Iteration 1 is the full data and has no out-of-bag units. | taxon | region | species | median_insample_auc | median_oob_auc | median_v2val_full | median_oob_v2val_full | | --- | --- | --- | --- | --- | --- | --- | -| bird | north | 2 | 0.751 | 0.73 | NA | NA | -| bryophyte | north | 2 | 0.666 | 0.646 | 0.668 | 0.652 | -| lichen | north | 2 | 0.807 | 0.798 | 0.808 | 0.802 | -| mammal | north | 4 | 0.751 | 0.73 | NA | NA | -| mite | north | 2 | 0.747 | 0.722 | 0.735 | 0.718 | -| vascular_plant | north | 2 | 0.836 | 0.829 | 0.833 | 0.829 | -| bird | south | 2 | 0.706 | 0.702 | NA | NA | -| bryophyte | south | 2 | 0.705 | 0.686 | 0.676 | 0.664 | -| lichen | south | 2 | 0.836 | 0.816 | 0.802 | 0.792 | -| mammal | south | 4 | 0.739 | 0.716 | NA | NA | -| mite | south | 2 | 0.809 | 0.774 | 0.796 | 0.781 | -| vascular_plant | south | 2 | 0.862 | 0.851 | 0.85 | 0.844 | +| bird | north | 2 | 0.752 | 0.729 | - | - | +| bryophyte | north | 2 | 0.668 | 0.650 | 0.668 | 0.650 | +| lichen | north | 2 | 0.808 | 0.803 | 0.808 | 0.803 | +| mammal | north | 4 | 0.746 | 0.731 | - | - | +| mite | north | 2 | 0.733 | 0.716 | 0.733 | 0.716 | +| vascular_plant | north | 2 | 0.833 | 0.828 | 0.833 | 0.828 | +| bird | south | 2 | 0.704 | 0.702 | - | - | +| bryophyte | south | 2 | 0.671 | 0.669 | 0.671 | 0.669 | +| lichen | south | 2 | 0.802 | 0.792 | 0.802 | 0.792 | +| mammal | south | 4 | 0.753 | 0.686 | - | - | +| mite | south | 2 | 0.799 | 0.774 | 0.799 | 0.774 | +| vascular_plant | south | 2 | 0.848 | 0.846 | 0.848 | 0.846 | ## What is still needed diff --git a/3_output/exp_000_parity_v2/run_record.md b/3_output/exp_000_parity_v2/run_record.md index c7ca08c..17f6105 100644 --- a/3_output/exp_000_parity_v2/run_record.md +++ b/3_output/exp_000_parity_v2/run_record.md @@ -6,20 +6,26 @@ Written by `run_record()`. The result stores this describes are gitignored; this _Volatile. Excluded from cross-run comparison._ -- **git_commit**: c6852d4 +- **git_commit**: a45a17d - **pipeline_dir**: D:/local_projects/active/sdmMethodsDev/2_pipeline/exp_000_parity_v2 - **r_version**: 4.5.0 -- **written_at**: 2026-10-03 11:16:41 +- **written_at**: 2026-10-06 11:25:57 ## Configuration -- **boot_seed**: 20260909 -- **data_dir**: test_dataset +- **boot_seed**: 20260909.0000 +- **changes_from_v2**: none +- **compared_with**: none (baseline) +- **covariate_files**: none +- **data_dir**: published 2026-10-05 (//ABMI-DATA2/science/sdmMethodsDev/0_data/test_dataset/2026-10-05) - **engines**: bayesglm, glm - **focal_species**: bird=AMRO, bird=YEWA, bryophyte=Bryum.All, bryophyte=Ceratodon.purpureus, lichen=Cladonia.chlorophaea, lichen=Physcia.adscendens, mammal=Coyote_Summer, mammal=Coyote_Winter, mammal=Moose_Summer, mammal=Moose_Winter, mite=Ceratozetes.gracilis, mite=Trhypochthonius.tectorum, vascular_plant=Galium.boreale, vascular_plant=Vicia.americana -- **n_bootstraps**: 100 +- **jobs_not_ok**: 0 +- **jobs_ok**: 140 +- **jobs_total**: 140 +- **n_bootstraps**: 5 - **plant_bootstrap**: spatial_block -- **selection**: aic_average, custom, ivw_grid, staged_bic +- **selection**: aic_average, hurdle, ivw_grid, staged_bic - **species_n**: 14 - **stage_models**: spec defaults (v2 candidate sets) - **taxa**: bird, bryophyte, lichen, mammal_summer, mammal_winter, mite, vascular_plant @@ -29,331 +35,331 @@ _Volatile. Excluded from cross-run comparison._ | taxon | region | species | draws | coefficient_rows | metric_rows | grid_rows | | --- | --- | --- | --- | --- | --- | --- | -| bird | north | 2 | 100 | 9380 | 2800 | 0 | -| bird | south | 2 | 100 | 6986 | 2800 | 0 | -| bryophyte | north | 2 | 100 | 22000 | 5600 | 0 | -| bryophyte | south | 2 | 100 | 7800 | 5600 | 0 | -| lichen | north | 2 | 100 | 22000 | 5600 | 0 | -| lichen | south | 2 | 100 | 7800 | 5600 | 0 | -| mammal_summer | north | 2 | 100 | 43450 | 2800 | 0 | -| mammal_summer | south | 2 | 100 | 13731 | 2786 | 0 | -| mammal_winter | north | 2 | 100 | 44008 | 2800 | 0 | -| mammal_winter | south | 2 | 100 | 13800 | 2800 | 0 | -| mite | north | 2 | 100 | 21800 | 5600 | 8600 | -| mite | south | 2 | 100 | 7600 | 5600 | 3200 | -| vascular_plant | north | 2 | 100 | 21800 | 5600 | 8600 | -| vascular_plant | south | 2 | 100 | 7600 | 5600 | 3200 | +| bird | north | 2 | 5 | 467 | 140 | 890 | +| bird | south | 2 | 5 | 350 | 140 | 170 | +| bryophyte | north | 2 | 5 | 1100 | 280 | 430 | +| bryophyte | south | 2 | 5 | 390 | 280 | 160 | +| lichen | north | 2 | 5 | 1100 | 280 | 430 | +| lichen | south | 2 | 5 | 390 | 280 | 160 | +| mammal_summer | north | 2 | 5 | 2204 | 140 | 450 | +| mammal_summer | south | 2 | 5 | 690 | 140 | 200 | +| mammal_winter | north | 2 | 5 | 2222 | 140 | 450 | +| mammal_winter | south | 2 | 5 | 690 | 140 | 200 | +| mite | north | 2 | 5 | 1090 | 280 | 430 | +| mite | south | 2 | 5 | 380 | 280 | 160 | +| vascular_plant | north | 2 | 5 | 1090 | 280 | 430 | +| vascular_plant | south | 2 | 5 | 380 | 280 | 160 | ## Metrics -| taxon | region | metric | n | median | p10 | p90 | -| --- | --- | --- | --- | --- | --- | --- | -| bird | north | insample_auc | 200 | 0.7500 | 0.7218 | 0.7798 | -| bird | north | insample_calibration_slope | 200 | 0.7722 | 0.6665 | 0.8113 | -| bird | north | insample_deviance_explained | 200 | 0.1986 | 0.1403 | 0.2614 | -| bird | north | insample_n | 200 | 46751.0000 | 46746.9000 | 46756.1000 | -| bird | north | insample_prevalence | 200 | 0.1847 | 0.1183 | 0.2520 | -| bird | north | insample_rmse | 200 | 0.6168 | 0.5471 | 0.6665 | -| bird | north | insample_spearman | 200 | 0.3268 | 0.3129 | 0.3440 | -| bird | north | oob_auc | 200 | 0.7300 | 0.7283 | 0.7318 | -| bird | north | oob_calibration_slope | 200 | 0.7890 | 0.6073 | 0.8883 | -| bird | north | oob_deviance_explained | 200 | 0.1637 | 0.1508 | 0.1808 | -| bird | north | oob_n | 200 | 124873.0000 | 124867.9000 | 124877.1000 | -| bird | north | oob_prevalence | 200 | 0.1874 | 0.1341 | 0.2415 | -| bird | north | oob_rmse | 200 | 0.6356 | 0.5854 | 0.6588 | -| bird | north | oob_spearman | 200 | 0.3108 | 0.2728 | 0.3502 | -| bird | south | insample_auc | 200 | 0.7048 | 0.6906 | 0.7212 | -| bird | south | insample_calibration_slope | 200 | 0.8737 | 0.8625 | 0.9017 | -| bird | south | insample_deviance_explained | 200 | 0.1578 | 0.0908 | 0.2349 | -| bird | south | insample_n | 200 | 23638.0000 | 23635.0000 | 23641.0000 | -| bird | south | insample_prevalence | 200 | 0.2151 | 0.1714 | 0.2596 | -| bird | south | insample_rmse | 200 | 0.6840 | 0.6223 | 0.7470 | -| bird | south | insample_spearman | 200 | 0.2942 | 0.2895 | 0.2994 | -| bird | south | oob_auc | 200 | 0.7012 | 0.6882 | 0.7164 | -| bird | south | oob_calibration_slope | 200 | 0.8829 | 0.8479 | 0.9478 | -| bird | south | oob_deviance_explained | 200 | 0.1475 | 0.0919 | 0.2042 | -| bird | south | oob_n | 200 | 67169.0000 | 67166.0000 | 67172.0000 | -| bird | south | oob_prevalence | 200 | 0.2175 | 0.1751 | 0.2606 | -| bird | south | oob_rmse | 200 | 0.6870 | 0.6369 | 0.7370 | -| bird | south | oob_spearman | 200 | 0.2911 | 0.2888 | 0.2936 | -| bryophyte | north | insample_auc | 200 | 0.6685 | 0.6364 | 0.6965 | -| bryophyte | north | insample_calibration_slope | 200 | 0.9694 | 0.9335 | 1.0068 | -| bryophyte | north | insample_deviance_explained | 200 | 0.0680 | 0.0445 | 0.0890 | -| bryophyte | north | insample_n | 200 | 3069.0000 | 3035.0000 | 3101.0000 | -| bryophyte | north | insample_prevalence | 200 | 0.4195 | 0.3957 | 0.4466 | -| bryophyte | north | insample_rmse | 200 | 0.4715 | 0.4664 | 0.4759 | -| bryophyte | north | insample_spearman | 200 | 0.2878 | 0.2314 | 0.3383 | -| bryophyte | north | oob_auc | 198 | 0.6436 | 0.6102 | 0.6808 | -| bryophyte | north | oob_calibration_slope | 198 | 0.8419 | 0.6967 | 0.9782 | -| bryophyte | north | oob_deviance_explained | 198 | 0.0432 | 0.0209 | 0.0745 | -| bryophyte | north | oob_n | 200 | 1783.0000 | 1751.0000 | 1817.0000 | -| bryophyte | north | oob_prevalence | 198 | 0.4186 | 0.3933 | 0.4485 | -| bryophyte | north | oob_rmse | 198 | 0.4773 | 0.4707 | 0.4833 | -| bryophyte | north | oob_spearman | 198 | 0.2451 | 0.1874 | 0.3103 | -| bryophyte | north | oob_v2val_Climate | 198 | 0.6045 | 0.5460 | 0.6535 | -| bryophyte | north | oob_v2val_Climate_Truncated | 198 | 0.6045 | 0.5459 | 0.6535 | -| bryophyte | north | oob_v2val_Full | 198 | 0.6547 | 0.6113 | 0.6909 | -| bryophyte | north | oob_v2val_Full_Joint | 198 | 0.6525 | 0.6101 | 0.6885 | -| bryophyte | north | oob_v2val_Full_Joint_Truncated | 198 | 0.6534 | 0.6104 | 0.6886 | -| bryophyte | north | oob_v2val_Full_Truncated | 198 | 0.6555 | 0.6115 | 0.6910 | -| bryophyte | north | oob_v2val_Landcover | 198 | 0.5970 | 0.5707 | 0.6262 | -| bryophyte | north | v2val_Climate | 200 | 0.6069 | 0.5588 | 0.6582 | -| bryophyte | north | v2val_Climate_Truncated | 200 | 0.6069 | 0.5588 | 0.6582 | -| bryophyte | north | v2val_Full | 200 | 0.6680 | 0.6329 | 0.7027 | -| bryophyte | north | v2val_Full_Joint | 200 | 0.6660 | 0.6316 | 0.6994 | -| bryophyte | north | v2val_Full_Joint_Truncated | 200 | 0.6662 | 0.6326 | 0.6996 | -| bryophyte | north | v2val_Full_Truncated | 200 | 0.6682 | 0.6341 | 0.7028 | -| bryophyte | north | v2val_Landcover | 200 | 0.6103 | 0.5865 | 0.6342 | -| bryophyte | south | insample_auc | 200 | 0.7094 | 0.6571 | 0.7537 | -| bryophyte | south | insample_calibration_slope | 200 | 0.9613 | 0.9174 | 1.0075 | -| bryophyte | south | insample_deviance_explained | 200 | 0.0987 | 0.0544 | 0.1444 | -| bryophyte | south | insample_n | 200 | 1461.5000 | 1440.9000 | 1484.0000 | -| bryophyte | south | insample_prevalence | 200 | 0.3038 | 0.2345 | 0.3768 | -| bryophyte | south | insample_rmse | 200 | 0.4321 | 0.3912 | 0.4661 | -| bryophyte | south | insample_spearman | 200 | 0.3290 | 0.2631 | 0.3747 | -| bryophyte | south | oob_auc | 198 | 0.6867 | 0.6283 | 0.7410 | -| bryophyte | south | oob_calibration_slope | 198 | 0.8419 | 0.6738 | 0.9682 | -| bryophyte | south | oob_deviance_explained | 198 | 0.0611 | 0.0230 | 0.1224 | -| bryophyte | south | oob_n | 200 | 848.5000 | 826.0000 | 869.1000 | -| bryophyte | south | oob_prevalence | 198 | 0.3062 | 0.2295 | 0.3842 | -| bryophyte | south | oob_rmse | 198 | 0.4434 | 0.3959 | 0.4758 | -| bryophyte | south | oob_spearman | 198 | 0.2901 | 0.2154 | 0.3532 | -| bryophyte | south | oob_v2val_Climate | 198 | 0.6045 | 0.5460 | 0.6535 | -| bryophyte | south | oob_v2val_Climate_Truncated | 198 | 0.6045 | 0.5459 | 0.6535 | -| bryophyte | south | oob_v2val_Full | 198 | 0.6605 | 0.6050 | 0.7223 | -| bryophyte | south | oob_v2val_Full_Joint | 198 | 0.6613 | 0.6046 | 0.7229 | -| bryophyte | south | oob_v2val_Full_Joint_Truncated | 198 | 0.6613 | 0.6048 | 0.7229 | -| bryophyte | south | oob_v2val_Full_Truncated | 198 | 0.6605 | 0.6050 | 0.7223 | -| bryophyte | south | oob_v2val_Landcover | 198 | 0.6058 | 0.5744 | 0.6394 | -| bryophyte | south | v2val_Climate | 200 | 0.6069 | 0.5588 | 0.6582 | -| bryophyte | south | v2val_Climate_Truncated | 200 | 0.6069 | 0.5588 | 0.6582 | -| bryophyte | south | v2val_Full | 200 | 0.6758 | 0.6249 | 0.7251 | -| bryophyte | south | v2val_Full_Joint | 200 | 0.6771 | 0.6255 | 0.7263 | -| bryophyte | south | v2val_Full_Joint_Truncated | 200 | 0.6771 | 0.6256 | 0.7263 | -| bryophyte | south | v2val_Full_Truncated | 200 | 0.6759 | 0.6249 | 0.7251 | -| bryophyte | south | v2val_Landcover | 200 | 0.6182 | 0.5948 | 0.6421 | -| lichen | north | insample_auc | 200 | 0.8071 | 0.7657 | 0.8473 | -| lichen | north | insample_calibration_slope | 200 | 0.9991 | 0.9802 | 1.0132 | -| lichen | north | insample_deviance_explained | 200 | 0.2375 | 0.1849 | 0.2980 | -| lichen | north | insample_n | 200 | 3674.5000 | 3639.0000 | 3709.0000 | -| lichen | north | insample_prevalence | 200 | 0.3844 | 0.3787 | 0.3905 | -| lichen | north | insample_rmse | 200 | 0.4111 | 0.3869 | 0.4329 | -| lichen | north | insample_spearman | 200 | 0.5160 | 0.4480 | 0.5856 | -| lichen | north | oob_auc | 198 | 0.8015 | 0.7542 | 0.8424 | -| lichen | north | oob_calibration_slope | 198 | 0.9725 | 0.9251 | 1.0142 | -| lichen | north | oob_deviance_explained | 198 | 0.2240 | 0.1638 | 0.2887 | -| lichen | north | oob_n | 200 | 2136.5000 | 2102.0000 | 2172.0000 | -| lichen | north | oob_prevalence | 198 | 0.3841 | 0.3745 | 0.3945 | -| lichen | north | oob_rmse | 198 | 0.4147 | 0.3893 | 0.4382 | -| lichen | north | oob_spearman | 198 | 0.5063 | 0.4286 | 0.5763 | -| lichen | north | oob_v2val_Climate | 198 | 0.7084 | 0.6845 | 0.7263 | -| lichen | north | oob_v2val_Climate_Truncated | 198 | 0.7084 | 0.6845 | 0.7263 | -| lichen | north | oob_v2val_Full | 198 | 0.8039 | 0.7636 | 0.8410 | -| lichen | north | oob_v2val_Full_Joint | 198 | 0.8020 | 0.7631 | 0.8384 | -| lichen | north | oob_v2val_Full_Joint_Truncated | 198 | 0.8021 | 0.7631 | 0.8386 | -| lichen | north | oob_v2val_Full_Truncated | 198 | 0.8040 | 0.7638 | 0.8411 | -| lichen | north | oob_v2val_Landcover | 198 | 0.7459 | 0.6978 | 0.8174 | -| lichen | north | v2val_Climate | 200 | 0.7072 | 0.6898 | 0.7278 | -| lichen | north | v2val_Climate_Truncated | 200 | 0.7072 | 0.6898 | 0.7278 | -| lichen | north | v2val_Full | 200 | 0.8091 | 0.7720 | 0.8423 | -| lichen | north | v2val_Full_Joint | 200 | 0.8077 | 0.7715 | 0.8400 | -| lichen | north | v2val_Full_Joint_Truncated | 200 | 0.8077 | 0.7716 | 0.8402 | -| lichen | north | v2val_Full_Truncated | 200 | 0.8092 | 0.7718 | 0.8425 | -| lichen | north | v2val_Landcover | 200 | 0.7453 | 0.7048 | 0.8159 | -| lichen | south | insample_auc | 200 | 0.8375 | 0.8219 | 0.8500 | -| lichen | south | insample_calibration_slope | 200 | 0.9835 | 0.9416 | 1.0174 | -| lichen | south | insample_deviance_explained | 200 | 0.2523 | 0.2053 | 0.2864 | -| lichen | south | insample_n | 200 | 1482.5000 | 1459.0000 | 1509.0000 | -| lichen | south | insample_prevalence | 200 | 0.1501 | 0.0887 | 0.2202 | -| lichen | south | insample_rmse | 200 | 0.3089 | 0.2665 | 0.3531 | -| lichen | south | insample_spearman | 200 | 0.4132 | 0.3181 | 0.4977 | -| lichen | south | oob_auc | 198 | 0.8170 | 0.7870 | 0.8399 | -| lichen | south | oob_calibration_slope | 198 | 0.9037 | 0.7274 | 1.0417 | -| lichen | south | oob_deviance_explained | 198 | 0.2035 | 0.1335 | 0.2674 | -| lichen | south | oob_n | 200 | 863.5000 | 837.0000 | 887.0000 | -| lichen | south | oob_prevalence | 198 | 0.1503 | 0.0880 | 0.2276 | -| lichen | south | oob_rmse | 198 | 0.3184 | 0.2720 | 0.3643 | -| lichen | south | oob_spearman | 198 | 0.3848 | 0.2846 | 0.4861 | -| lichen | south | oob_v2val_Climate | 198 | 0.7084 | 0.6845 | 0.7263 | -| lichen | south | oob_v2val_Climate_Truncated | 198 | 0.7084 | 0.6845 | 0.7263 | -| lichen | south | oob_v2val_Full | 198 | 0.7929 | 0.7520 | 0.8241 | -| lichen | south | oob_v2val_Full_Joint | 198 | 0.7932 | 0.7527 | 0.8219 | -| lichen | south | oob_v2val_Full_Joint_Truncated | 198 | 0.7932 | 0.7527 | 0.8219 | -| lichen | south | oob_v2val_Full_Truncated | 198 | 0.7929 | 0.7520 | 0.8241 | -| lichen | south | oob_v2val_Landcover | 198 | 0.7499 | 0.7160 | 0.7745 | -| lichen | south | v2val_Climate | 200 | 0.7072 | 0.6898 | 0.7278 | -| lichen | south | v2val_Climate_Truncated | 200 | 0.7072 | 0.6898 | 0.7278 | -| lichen | south | v2val_Full | 200 | 0.8072 | 0.7724 | 0.8287 | -| lichen | south | v2val_Full_Joint | 200 | 0.8057 | 0.7733 | 0.8261 | -| lichen | south | v2val_Full_Joint_Truncated | 200 | 0.8057 | 0.7733 | 0.8261 | -| lichen | south | v2val_Full_Truncated | 200 | 0.8072 | 0.7724 | 0.8287 | -| lichen | south | v2val_Landcover | 200 | 0.7520 | 0.7339 | 0.7705 | -| mammal_summer | north | insample_auc | 200 | 0.7195 | 0.6076 | 0.8295 | -| mammal_summer | north | insample_calibration_slope | 200 | 0.9415 | 0.7950 | 0.9994 | -| mammal_summer | north | insample_deviance_explained | 200 | 0.1457 | 0.0267 | 0.2897 | -| mammal_summer | north | insample_n | 200 | 2563.5000 | 2538.0000 | 2587.0000 | -| mammal_summer | north | insample_prevalence | 200 | 0.2733 | 0.2209 | 0.3269 | -| mammal_summer | north | insample_rmse | 200 | 0.3850 | 0.3230 | 0.4431 | -| mammal_summer | north | insample_spearman | 200 | 0.3181 | 0.1570 | 0.4710 | -| mammal_summer | north | oob_auc | 198 | 0.7008 | 0.5803 | 0.8199 | -| mammal_summer | north | oob_calibration_slope | 198 | 0.8458 | 0.5527 | 0.9867 | -| mammal_summer | north | oob_deviance_explained | 198 | 0.0563 | 0.0013 | 0.2662 | -| mammal_summer | north | oob_n | 200 | 1489.5000 | 1466.0000 | 1515.0000 | -| mammal_summer | north | oob_prevalence | 198 | 0.2735 | 0.2193 | 0.3287 | -| mammal_summer | north | oob_rmse | 198 | 0.3917 | 0.3259 | 0.4480 | -| mammal_summer | north | oob_spearman | 198 | 0.2890 | 0.1134 | 0.4545 | -| mammal_summer | south | insample_auc | 199 | 0.7546 | 0.6196 | 0.7995 | -| mammal_summer | south | insample_calibration_slope | 199 | 0.9613 | 0.8460 | 1.0524 | -| mammal_summer | south | insample_deviance_explained | 199 | 0.1647 | 0.0214 | 0.2324 | -| mammal_summer | south | insample_n | 199 | 674.0000 | 662.0000 | 689.0000 | -| mammal_summer | south | insample_prevalence | 199 | 0.2069 | 0.1750 | 0.6994 | -| mammal_summer | south | insample_rmse | 199 | 0.3563 | 0.3391 | 0.4250 | -| mammal_summer | south | insample_spearman | 199 | 0.3342 | 0.1592 | 0.3998 | -| mammal_summer | south | oob_auc | 197 | 0.7154 | 0.5675 | 0.7903 | -| mammal_summer | south | oob_calibration_slope | 197 | 0.7382 | 0.3358 | 1.0724 | -| mammal_summer | south | oob_deviance_explained | 197 | 0.0147 | -0.0349 | 0.2068 | -| mammal_summer | south | oob_n | 199 | 391.0000 | 376.0000 | 403.0000 | -| mammal_summer | south | oob_prevalence | 197 | 0.2300 | 0.1685 | 0.7076 | -| mammal_summer | south | oob_rmse | 197 | 0.3897 | 0.3411 | 0.4384 | -| mammal_summer | south | oob_spearman | 197 | 0.2707 | 0.0727 | 0.3889 | -| mammal_winter | north | insample_auc | 200 | 0.7665 | 0.6675 | 0.8446 | -| mammal_winter | north | insample_calibration_slope | 200 | 0.9541 | 0.8712 | 1.0110 | -| mammal_winter | north | insample_deviance_explained | 200 | 0.1900 | 0.0702 | 0.3056 | -| mammal_winter | north | insample_n | 200 | 2609.5000 | 2585.0000 | 2640.1000 | -| mammal_winter | north | insample_prevalence | 200 | 0.1764 | 0.1455 | 0.2061 | -| mammal_winter | north | insample_rmse | 200 | 0.3161 | 0.2973 | 0.3353 | -| mammal_winter | north | insample_spearman | 200 | 0.3518 | 0.2111 | 0.4734 | -| mammal_winter | north | oob_auc | 198 | 0.7540 | 0.6364 | 0.8417 | -| mammal_winter | north | oob_calibration_slope | 198 | 0.8826 | 0.6191 | 1.0325 | -| mammal_winter | north | oob_deviance_explained | 198 | 0.0839 | 0.0080 | 0.2921 | -| mammal_winter | north | oob_n | 200 | 1517.5000 | 1486.9000 | 1542.0000 | -| mammal_winter | north | oob_prevalence | 198 | 0.1742 | 0.1426 | 0.2060 | -| mammal_winter | north | oob_rmse | 198 | 0.3199 | 0.2972 | 0.3413 | -| mammal_winter | north | oob_spearman | 198 | 0.3263 | 0.1702 | 0.4645 | -| mammal_winter | south | insample_auc | 200 | 0.7315 | 0.6779 | 0.8331 | -| mammal_winter | south | insample_calibration_slope | 200 | 0.9493 | 0.8167 | 1.0904 | -| mammal_winter | south | insample_deviance_explained | 200 | 0.1130 | 0.0574 | 0.2652 | -| mammal_winter | south | insample_n | 200 | 695.0000 | 682.0000 | 709.0000 | -| mammal_winter | south | insample_prevalence | 200 | 0.3351 | 0.0822 | 0.6144 | -| mammal_winter | south | insample_rmse | 200 | 0.3196 | 0.2155 | 0.4099 | -| mammal_winter | south | insample_spearman | 200 | 0.3081 | 0.2774 | 0.3331 | -| mammal_winter | south | oob_auc | 198 | 0.7123 | 0.6520 | 0.8075 | -| mammal_winter | south | oob_calibration_slope | 198 | 0.7412 | 0.4709 | 1.0531 | -| mammal_winter | south | oob_deviance_explained | 198 | 0.0619 | 0.0061 | 0.1945 | -| mammal_winter | south | oob_n | 200 | 403.0000 | 389.0000 | 416.0000 | -| mammal_winter | south | oob_prevalence | 198 | 0.3423 | 0.0803 | 0.6235 | -| mammal_winter | south | oob_rmse | 198 | 0.3325 | 0.2222 | 0.4211 | -| mammal_winter | south | oob_spearman | 198 | 0.2687 | 0.2234 | 0.3096 | -| mite | north | insample_auc | 200 | 0.7521 | 0.6888 | 0.8020 | -| mite | north | insample_calibration_slope | 200 | 0.9863 | 0.9206 | 1.0220 | -| mite | north | insample_deviance_explained | 200 | 0.1291 | 0.0625 | 0.1988 | -| mite | north | insample_n | 200 | 3611.0000 | 3575.8000 | 3648.1000 | -| mite | north | insample_prevalence | 200 | 0.1491 | 0.0931 | 0.2122 | -| mite | north | insample_rmse | 200 | 0.3243 | 0.2837 | 0.3643 | -| mite | north | insample_spearman | 200 | 0.3111 | 0.1924 | 0.4248 | -| mite | north | oob_auc | 198 | 0.7259 | 0.6442 | 0.7940 | -| mite | north | oob_calibration_slope | 198 | 0.9046 | 0.6725 | 1.0204 | -| mite | north | oob_deviance_explained | 198 | 0.0943 | 0.0131 | 0.1858 | -| mite | north | oob_n | 200 | 2100.0000 | 2062.9000 | 2135.2000 | -| mite | north | oob_prevalence | 198 | 0.1509 | 0.0922 | 0.2141 | -| mite | north | oob_rmse | 198 | 0.3327 | 0.2860 | 0.3702 | -| mite | north | oob_spearman | 198 | 0.2796 | 0.1476 | 0.4139 | -| mite | north | oob_v2val_Climate | 198 | 0.6902 | 0.6270 | 0.7420 | -| mite | north | oob_v2val_Climate_Truncated | 198 | 0.6902 | 0.6270 | 0.7420 | -| mite | north | oob_v2val_Full | 198 | 0.7251 | 0.6397 | 0.7915 | -| mite | north | oob_v2val_Full_Joint | 198 | 0.7243 | 0.6393 | 0.7914 | -| mite | north | oob_v2val_Full_Joint_Truncated | 198 | 0.7243 | 0.6394 | 0.7914 | -| mite | north | oob_v2val_Full_Truncated | 198 | 0.7251 | 0.6398 | 0.7913 | -| mite | north | oob_v2val_Landcover | 198 | 0.6958 | 0.5964 | 0.7674 | -| mite | north | v2val_Climate | 200 | 0.6938 | 0.6397 | 0.7436 | -| mite | north | v2val_Climate_Truncated | 200 | 0.6938 | 0.6396 | 0.7436 | -| mite | north | v2val_Full | 200 | 0.7415 | 0.6711 | 0.7964 | -| mite | north | v2val_Full_Joint | 200 | 0.7411 | 0.6708 | 0.7961 | -| mite | north | v2val_Full_Joint_Truncated | 200 | 0.7411 | 0.6708 | 0.7961 | -| mite | north | v2val_Full_Truncated | 200 | 0.7414 | 0.6710 | 0.7964 | -| mite | north | v2val_Landcover | 200 | 0.7117 | 0.6394 | 0.7723 | -| mite | south | insample_auc | 200 | 0.8147 | 0.7113 | 0.9040 | -| mite | south | insample_calibration_slope | 200 | 0.9928 | 0.9285 | 1.0911 | -| mite | south | insample_deviance_explained | 200 | 0.2006 | 0.0826 | 0.3433 | -| mite | south | insample_n | 200 | 1492.5000 | 1471.0000 | 1516.1000 | -| mite | south | insample_prevalence | 200 | 0.0681 | 0.0529 | 0.0865 | -| mite | south | insample_rmse | 200 | 0.2315 | 0.1931 | 0.2731 | -| mite | south | insample_spearman | 200 | 0.2713 | 0.2030 | 0.3223 | -| mite | south | oob_auc | 198 | 0.7550 | 0.6586 | 0.8840 | -| mite | south | oob_calibration_slope | 198 | 0.8287 | 0.5660 | 1.0766 | -| mite | south | oob_deviance_explained | 198 | 0.0874 | -0.0174 | 0.2927 | -| mite | south | oob_n | 200 | 870.5000 | 846.9000 | 892.0000 | -| mite | south | oob_prevalence | 198 | 0.0680 | 0.0520 | 0.0901 | -| mite | south | oob_rmse | 198 | 0.2352 | 0.2013 | 0.2827 | -| mite | south | oob_spearman | 198 | 0.2294 | 0.1535 | 0.3140 | -| mite | south | oob_v2val_Climate | 198 | 0.6902 | 0.6270 | 0.7420 | -| mite | south | oob_v2val_Climate_Truncated | 198 | 0.6902 | 0.6270 | 0.7420 | -| mite | south | oob_v2val_Full | 198 | 0.7758 | 0.6618 | 0.8935 | -| mite | south | oob_v2val_Full_Joint | 198 | 0.7761 | 0.6619 | 0.8938 | -| mite | south | oob_v2val_Full_Joint_Truncated | 198 | 0.7761 | 0.6619 | 0.8938 | -| mite | south | oob_v2val_Full_Truncated | 198 | 0.7758 | 0.6618 | 0.8935 | -| mite | south | oob_v2val_Landcover | 198 | 0.6739 | 0.6215 | 0.7295 | -| mite | south | v2val_Climate | 200 | 0.6938 | 0.6397 | 0.7436 | -| mite | south | v2val_Climate_Truncated | 200 | 0.6938 | 0.6396 | 0.7436 | -| mite | south | v2val_Full | 200 | 0.7987 | 0.6949 | 0.8911 | -| mite | south | v2val_Full_Joint | 200 | 0.7990 | 0.6953 | 0.8919 | -| mite | south | v2val_Full_Joint_Truncated | 200 | 0.7990 | 0.6953 | 0.8919 | -| mite | south | v2val_Full_Truncated | 200 | 0.7987 | 0.6949 | 0.8911 | -| mite | south | v2val_Landcover | 200 | 0.6921 | 0.6525 | 0.7356 | -| vascular_plant | north | insample_auc | 200 | 0.8356 | 0.8299 | 0.8404 | -| vascular_plant | north | insample_calibration_slope | 200 | 0.9970 | 0.9837 | 1.0096 | -| vascular_plant | north | insample_deviance_explained | 200 | 0.2724 | 0.2607 | 0.2811 | -| vascular_plant | north | insample_n | 200 | 4350.0000 | 4321.9000 | 4383.0000 | -| vascular_plant | north | insample_prevalence | 200 | 0.3962 | 0.3735 | 0.4214 | -| vascular_plant | north | insample_rmse | 200 | 0.4002 | 0.3947 | 0.4053 | -| vascular_plant | north | insample_spearman | 200 | 0.5681 | 0.5585 | 0.5769 | -| vascular_plant | north | oob_auc | 198 | 0.8287 | 0.8197 | 0.8384 | -| vascular_plant | north | oob_calibration_slope | 198 | 0.9723 | 0.9325 | 1.0229 | -| vascular_plant | north | oob_deviance_explained | 198 | 0.2570 | 0.2396 | 0.2763 | -| vascular_plant | north | oob_n | 200 | 2524.0000 | 2491.0000 | 2552.1000 | -| vascular_plant | north | oob_prevalence | 198 | 0.3937 | 0.3713 | 0.4251 | -| vascular_plant | north | oob_rmse | 198 | 0.4041 | 0.3966 | 0.4130 | -| vascular_plant | north | oob_spearman | 198 | 0.5568 | 0.5420 | 0.5734 | -| vascular_plant | north | oob_v2val_Climate | 198 | 0.6889 | 0.6788 | 0.6980 | -| vascular_plant | north | oob_v2val_Climate_Truncated | 198 | 0.6890 | 0.6788 | 0.6980 | -| vascular_plant | north | oob_v2val_Full | 198 | 0.8282 | 0.8186 | 0.8382 | -| vascular_plant | north | oob_v2val_Full_Joint | 198 | 0.8270 | 0.8155 | 0.8373 | -| vascular_plant | north | oob_v2val_Full_Joint_Truncated | 198 | 0.8272 | 0.8157 | 0.8374 | -| vascular_plant | north | oob_v2val_Full_Truncated | 198 | 0.8282 | 0.8187 | 0.8384 | -| vascular_plant | north | oob_v2val_Landcover | 198 | 0.8033 | 0.7927 | 0.8137 | -| vascular_plant | north | v2val_Climate | 200 | 0.6903 | 0.6846 | 0.6956 | -| vascular_plant | north | v2val_Climate_Truncated | 200 | 0.6903 | 0.6846 | 0.6956 | -| vascular_plant | north | v2val_Full | 200 | 0.8328 | 0.8275 | 0.8378 | -| vascular_plant | north | v2val_Full_Joint | 200 | 0.8314 | 0.8249 | 0.8370 | -| vascular_plant | north | v2val_Full_Joint_Truncated | 200 | 0.8315 | 0.8250 | 0.8370 | -| vascular_plant | north | v2val_Full_Truncated | 200 | 0.8329 | 0.8276 | 0.8377 | -| vascular_plant | north | v2val_Landcover | 200 | 0.8077 | 0.8018 | 0.8141 | -| vascular_plant | south | insample_auc | 200 | 0.8620 | 0.8175 | 0.9066 | -| vascular_plant | south | insample_calibration_slope | 200 | 1.0058 | 0.9832 | 1.0274 | -| vascular_plant | south | insample_deviance_explained | 200 | 0.3234 | 0.2372 | 0.4314 | -| vascular_plant | south | insample_n | 200 | 1674.0000 | 1652.0000 | 1698.1000 | -| vascular_plant | south | insample_prevalence | 200 | 0.2431 | 0.2173 | 0.2692 | -| vascular_plant | south | insample_rmse | 200 | 0.3415 | 0.3018 | 0.3767 | -| vascular_plant | south | insample_spearman | 200 | 0.5349 | 0.4834 | 0.5862 | -| vascular_plant | south | oob_auc | 198 | 0.8543 | 0.8031 | 0.9038 | -| vascular_plant | south | oob_calibration_slope | 198 | 0.9737 | 0.8996 | 1.0404 | -| vascular_plant | south | oob_deviance_explained | 198 | 0.2934 | 0.2049 | 0.4160 | -| vascular_plant | south | oob_n | 200 | 974.0000 | 949.9000 | 996.0000 | -| vascular_plant | south | oob_prevalence | 198 | 0.2453 | 0.2136 | 0.2763 | -| vascular_plant | south | oob_rmse | 198 | 0.3498 | 0.3057 | 0.3852 | -| vascular_plant | south | oob_spearman | 198 | 0.5226 | 0.4606 | 0.5816 | -| vascular_plant | south | oob_v2val_Climate | 198 | 0.6889 | 0.6788 | 0.6980 | -| vascular_plant | south | oob_v2val_Climate_Truncated | 198 | 0.6890 | 0.6788 | 0.6980 | -| vascular_plant | south | oob_v2val_Full | 198 | 0.8408 | 0.7938 | 0.8962 | -| vascular_plant | south | oob_v2val_Full_Joint | 198 | 0.8371 | 0.7904 | 0.8903 | -| vascular_plant | south | oob_v2val_Full_Joint_Truncated | 198 | 0.8371 | 0.7904 | 0.8903 | -| vascular_plant | south | oob_v2val_Full_Truncated | 198 | 0.8408 | 0.7938 | 0.8962 | -| vascular_plant | south | oob_v2val_Landcover | 198 | 0.7754 | 0.7521 | 0.8003 | -| vascular_plant | south | v2val_Climate | 200 | 0.6903 | 0.6846 | 0.6956 | -| vascular_plant | south | v2val_Climate_Truncated | 200 | 0.6903 | 0.6846 | 0.6956 | -| vascular_plant | south | v2val_Full | 200 | 0.8496 | 0.8025 | 0.8978 | -| vascular_plant | south | v2val_Full_Joint | 200 | 0.8441 | 0.8002 | 0.8908 | -| vascular_plant | south | v2val_Full_Joint_Truncated | 200 | 0.8441 | 0.8001 | 0.8908 | -| vascular_plant | south | v2val_Full_Truncated | 200 | 0.8496 | 0.8025 | 0.8978 | -| vascular_plant | south | v2val_Landcover | 200 | 0.7822 | 0.7595 | 0.8038 | +| taxon | region | metric | n | median | p10 | p90 | mean | sd | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| bird | north | insample_auc | 10 | 0.7522 | 0.7235 | 0.7796 | 0.7516 | 0.0288 | +| bird | north | insample_calibration_slope | 10 | 0.7559 | 0.6982 | 0.8102 | 0.7542 | 0.0543 | +| bird | north | insample_deviance_explained | 10 | 0.1996 | 0.1409 | 0.2605 | 0.2005 | 0.0613 | +| bird | north | insample_n | 10 | 46747.0000 | 46744.0000 | 46750.0000 | 46747.6000 | 2.3664 | +| bird | north | insample_prevalence | 10 | 0.1851 | 0.1178 | 0.2520 | 0.1850 | 0.0702 | +| bird | north | insample_rmse | 10 | 0.6070 | 0.5467 | 0.6643 | 0.6056 | 0.0609 | +| bird | north | insample_spearman | 10 | 0.3277 | 0.3148 | 0.3446 | 0.3293 | 0.0149 | +| bird | north | oob_auc | 10 | 0.7299 | 0.7284 | 0.7308 | 0.7296 | 0.0010 | +| bird | north | oob_calibration_slope | 10 | 0.7875 | 0.6454 | 0.8810 | 0.7730 | 0.1073 | +| bird | north | oob_deviance_explained | 10 | 0.1647 | 0.1509 | 0.1821 | 0.1658 | 0.0150 | +| bird | north | oob_n | 10 | 124877.0000 | 124874.0000 | 124880.0000 | 124876.4000 | 2.3664 | +| bird | north | oob_prevalence | 10 | 0.1877 | 0.1343 | 0.2411 | 0.1877 | 0.0561 | +| bird | north | oob_rmse | 10 | 0.6250 | 0.5831 | 0.6583 | 0.6230 | 0.0372 | +| bird | north | oob_spearman | 10 | 0.3117 | 0.2736 | 0.3483 | 0.3110 | 0.0389 | +| bird | south | insample_auc | 10 | 0.7052 | 0.6892 | 0.7215 | 0.7054 | 0.0153 | +| bird | south | insample_calibration_slope | 10 | 0.8775 | 0.8649 | 0.8901 | 0.8777 | 0.0119 | +| bird | south | insample_deviance_explained | 10 | 0.1561 | 0.0911 | 0.2337 | 0.1609 | 0.0728 | +| bird | south | insample_n | 10 | 23639.0000 | 23636.0000 | 23640.0000 | 23638.4000 | 1.4298 | +| bird | south | insample_prevalence | 10 | 0.2142 | 0.1711 | 0.2590 | 0.2149 | 0.0459 | +| bird | south | insample_rmse | 10 | 0.6840 | 0.6232 | 0.7452 | 0.6839 | 0.0623 | +| bird | south | insample_spearman | 10 | 0.2943 | 0.2880 | 0.2978 | 0.2933 | 0.0046 | +| bird | south | oob_auc | 10 | 0.7013 | 0.6892 | 0.7156 | 0.7019 | 0.0134 | +| bird | south | oob_calibration_slope | 10 | 0.9220 | 0.8622 | 0.9549 | 0.9086 | 0.0414 | +| bird | south | oob_deviance_explained | 10 | 0.1483 | 0.0969 | 0.2066 | 0.1498 | 0.0555 | +| bird | south | oob_n | 10 | 67168.0000 | 67167.0000 | 67171.0000 | 67168.6000 | 1.4298 | +| bird | south | oob_prevalence | 10 | 0.2178 | 0.1757 | 0.2603 | 0.2180 | 0.0444 | +| bird | south | oob_rmse | 10 | 0.6869 | 0.6369 | 0.7359 | 0.6867 | 0.0513 | +| bird | south | oob_spearman | 10 | 0.2916 | 0.2888 | 0.2927 | 0.2913 | 0.0017 | +| bryophyte | north | insample_auc | 10 | 0.6642 | 0.6314 | 0.7085 | 0.6676 | 0.0347 | +| bryophyte | north | insample_calibration_slope | 10 | 0.8499 | 0.7585 | 0.8973 | 0.8326 | 0.0644 | +| bryophyte | north | insample_deviance_explained | 10 | -0.1896 | -0.2902 | -0.0937 | -0.1928 | 0.0883 | +| bryophyte | north | insample_n | 10 | 3091.0000 | 3041.1000 | 4852.0000 | 3429.3000 | 750.2771 | +| bryophyte | north | insample_prevalence | 10 | 0.4175 | 0.3976 | 0.4423 | 0.4194 | 0.0223 | +| bryophyte | north | insample_rmse | 10 | 0.4736 | 0.4634 | 0.4802 | 0.4727 | 0.0076 | +| bryophyte | north | insample_spearman | 10 | 0.2805 | 0.2228 | 0.3578 | 0.2866 | 0.0614 | +| bryophyte | north | oob_auc | 8 | 0.6562 | 0.6254 | 0.6808 | 0.6537 | 0.0253 | +| bryophyte | north | oob_calibration_slope | 8 | 0.7707 | 0.6959 | 0.8841 | 0.7784 | 0.0797 | +| bryophyte | north | oob_deviance_explained | 8 | -0.1858 | -0.2157 | -0.1326 | -0.1818 | 0.0371 | +| bryophyte | north | oob_n | 10 | 1761.0000 | 0.0000 | 1810.9000 | 1422.7000 | 750.2771 | +| bryophyte | north | oob_prevalence | 8 | 0.4201 | 0.4007 | 0.4503 | 0.4233 | 0.0233 | +| bryophyte | north | oob_rmse | 8 | 0.4801 | 0.4711 | 0.4829 | 0.4781 | 0.0053 | +| bryophyte | north | oob_spearman | 8 | 0.2675 | 0.2128 | 0.3105 | 0.2630 | 0.0449 | +| bryophyte | north | oob_v2val_Climate | 8 | 0.6045 | 0.5487 | 0.6482 | 0.5986 | 0.0463 | +| bryophyte | north | oob_v2val_Climate_Truncated | 8 | 0.6045 | 0.5486 | 0.6482 | 0.5986 | 0.0463 | +| bryophyte | north | oob_v2val_Full | 8 | 0.6562 | 0.6254 | 0.6808 | 0.6537 | 0.0253 | +| bryophyte | north | oob_v2val_Full_Joint | 8 | 0.6538 | 0.6242 | 0.6786 | 0.6519 | 0.0247 | +| bryophyte | north | oob_v2val_Full_Joint_Truncated | 8 | 0.6546 | 0.6248 | 0.6788 | 0.6524 | 0.0246 | +| bryophyte | north | oob_v2val_Full_Truncated | 8 | 0.6571 | 0.6259 | 0.6811 | 0.6542 | 0.0252 | +| bryophyte | north | oob_v2val_Landcover | 8 | 0.5971 | 0.5798 | 0.6148 | 0.5974 | 0.0150 | +| bryophyte | north | v2val_Climate | 10 | 0.6069 | 0.5555 | 0.6643 | 0.6086 | 0.0518 | +| bryophyte | north | v2val_Climate_Truncated | 10 | 0.6069 | 0.5555 | 0.6643 | 0.6086 | 0.0518 | +| bryophyte | north | v2val_Full | 10 | 0.6642 | 0.6314 | 0.7085 | 0.6676 | 0.0347 | +| bryophyte | north | v2val_Full_Joint | 10 | 0.6615 | 0.6293 | 0.7047 | 0.6648 | 0.0336 | +| bryophyte | north | v2val_Full_Joint_Truncated | 10 | 0.6621 | 0.6299 | 0.7052 | 0.6652 | 0.0334 | +| bryophyte | north | v2val_Full_Truncated | 10 | 0.6647 | 0.6320 | 0.7089 | 0.6679 | 0.0346 | +| bryophyte | north | v2val_Landcover | 10 | 0.6082 | 0.5829 | 0.6350 | 0.6089 | 0.0226 | +| bryophyte | south | insample_auc | 10 | 0.6741 | 0.6221 | 0.7221 | 0.6723 | 0.0498 | +| bryophyte | south | insample_calibration_slope | 10 | 0.8833 | 0.8604 | 0.9293 | 0.8836 | 0.0487 | +| bryophyte | south | insample_deviance_explained | 10 | -0.0076 | -0.1062 | 0.0587 | -0.0180 | 0.0817 | +| bryophyte | south | insample_n | 10 | 1454.5000 | 1440.8000 | 2310.0000 | 1623.0000 | 362.3789 | +| bryophyte | south | insample_prevalence | 10 | 0.3052 | 0.2352 | 0.3713 | 0.3043 | 0.0687 | +| bryophyte | south | insample_rmse | 10 | 0.4407 | 0.3994 | 0.4751 | 0.4384 | 0.0377 | +| bryophyte | south | insample_spearman | 10 | 0.2678 | 0.2042 | 0.3271 | 0.2668 | 0.0610 | +| bryophyte | south | oob_auc | 8 | 0.6737 | 0.6156 | 0.7162 | 0.6682 | 0.0497 | +| bryophyte | south | oob_calibration_slope | 8 | 0.8626 | 0.7944 | 0.9233 | 0.8595 | 0.0569 | +| bryophyte | south | oob_deviance_explained | 8 | 0.0177 | -0.0430 | 0.0521 | 0.0106 | 0.0409 | +| bryophyte | south | oob_n | 10 | 855.5000 | 0.0000 | 869.2000 | 687.0000 | 362.3789 | +| bryophyte | south | oob_prevalence | 8 | 0.3139 | 0.2297 | 0.3751 | 0.3071 | 0.0716 | +| bryophyte | south | oob_rmse | 8 | 0.4447 | 0.3999 | 0.4766 | 0.4408 | 0.0371 | +| bryophyte | south | oob_spearman | 8 | 0.2693 | 0.1933 | 0.3245 | 0.2610 | 0.0620 | +| bryophyte | south | oob_v2val_Climate | 8 | 0.6045 | 0.5487 | 0.6482 | 0.5986 | 0.0463 | +| bryophyte | south | oob_v2val_Climate_Truncated | 8 | 0.6045 | 0.5486 | 0.6482 | 0.5986 | 0.0463 | +| bryophyte | south | oob_v2val_Full | 8 | 0.6737 | 0.6156 | 0.7162 | 0.6682 | 0.0497 | +| bryophyte | south | oob_v2val_Full_Joint | 8 | 0.6754 | 0.6148 | 0.7178 | 0.6689 | 0.0510 | +| bryophyte | south | oob_v2val_Full_Joint_Truncated | 8 | 0.6754 | 0.6148 | 0.7178 | 0.6689 | 0.0510 | +| bryophyte | south | oob_v2val_Full_Truncated | 8 | 0.6737 | 0.6156 | 0.7162 | 0.6682 | 0.0497 | +| bryophyte | south | oob_v2val_Landcover | 8 | 0.6029 | 0.5970 | 0.6332 | 0.6102 | 0.0172 | +| bryophyte | south | v2val_Climate | 10 | 0.6069 | 0.5555 | 0.6643 | 0.6086 | 0.0518 | +| bryophyte | south | v2val_Climate_Truncated | 10 | 0.6069 | 0.5555 | 0.6643 | 0.6086 | 0.0518 | +| bryophyte | south | v2val_Full | 10 | 0.6741 | 0.6221 | 0.7221 | 0.6723 | 0.0498 | +| bryophyte | south | v2val_Full_Joint | 10 | 0.6743 | 0.6214 | 0.7227 | 0.6726 | 0.0505 | +| bryophyte | south | v2val_Full_Joint_Truncated | 10 | 0.6743 | 0.6215 | 0.7227 | 0.6726 | 0.0505 | +| bryophyte | south | v2val_Full_Truncated | 10 | 0.6741 | 0.6221 | 0.7221 | 0.6723 | 0.0498 | +| bryophyte | south | v2val_Landcover | 10 | 0.6137 | 0.5905 | 0.6386 | 0.6139 | 0.0223 | +| lichen | north | insample_auc | 10 | 0.8074 | 0.7781 | 0.8388 | 0.8083 | 0.0311 | +| lichen | north | insample_calibration_slope | 10 | 1.0001 | 0.9745 | 1.0269 | 0.9999 | 0.0221 | +| lichen | north | insample_deviance_explained | 10 | 0.1360 | 0.1091 | 0.1649 | 0.1405 | 0.0261 | +| lichen | north | insample_n | 10 | 3692.0000 | 3654.0000 | 5811.0000 | 4109.1000 | 897.3877 | +| lichen | north | insample_prevalence | 10 | 0.3839 | 0.3815 | 0.3891 | 0.3840 | 0.0037 | +| lichen | north | insample_rmse | 10 | 0.4112 | 0.3929 | 0.4298 | 0.4111 | 0.0183 | +| lichen | north | insample_spearman | 10 | 0.5176 | 0.4685 | 0.5707 | 0.5194 | 0.0520 | +| lichen | north | oob_auc | 8 | 0.7997 | 0.7622 | 0.8453 | 0.8016 | 0.0403 | +| lichen | north | oob_calibration_slope | 8 | 0.9794 | 0.9485 | 1.0076 | 0.9784 | 0.0272 | +| lichen | north | oob_deviance_explained | 8 | 0.1271 | 0.0891 | 0.1748 | 0.1334 | 0.0469 | +| lichen | north | oob_n | 10 | 2119.0000 | 0.0000 | 2157.0000 | 1701.9000 | 897.3877 | +| lichen | north | oob_prevalence | 8 | 0.3864 | 0.3761 | 0.3921 | 0.3853 | 0.0072 | +| lichen | north | oob_rmse | 8 | 0.4171 | 0.3889 | 0.4363 | 0.4142 | 0.0224 | +| lichen | north | oob_spearman | 8 | 0.5059 | 0.4420 | 0.5824 | 0.5084 | 0.0681 | +| lichen | north | oob_v2val_Climate | 8 | 0.7041 | 0.6878 | 0.7158 | 0.7028 | 0.0127 | +| lichen | north | oob_v2val_Climate_Truncated | 8 | 0.7041 | 0.6878 | 0.7158 | 0.7028 | 0.0127 | +| lichen | north | oob_v2val_Full | 8 | 0.7997 | 0.7622 | 0.8453 | 0.8016 | 0.0403 | +| lichen | north | oob_v2val_Full_Joint | 8 | 0.7969 | 0.7616 | 0.8417 | 0.7996 | 0.0394 | +| lichen | north | oob_v2val_Full_Joint_Truncated | 8 | 0.7969 | 0.7615 | 0.8418 | 0.7996 | 0.0396 | +| lichen | north | oob_v2val_Full_Truncated | 8 | 0.7997 | 0.7624 | 0.8453 | 0.8016 | 0.0404 | +| lichen | north | oob_v2val_Landcover | 8 | 0.7481 | 0.6957 | 0.8186 | 0.7528 | 0.0579 | +| lichen | north | v2val_Climate | 10 | 0.7093 | 0.6916 | 0.7289 | 0.7101 | 0.0183 | +| lichen | north | v2val_Climate_Truncated | 10 | 0.7093 | 0.6916 | 0.7290 | 0.7101 | 0.0183 | +| lichen | north | v2val_Full | 10 | 0.8074 | 0.7781 | 0.8388 | 0.8083 | 0.0311 | +| lichen | north | v2val_Full_Joint | 10 | 0.8057 | 0.7771 | 0.8370 | 0.8071 | 0.0308 | +| lichen | north | v2val_Full_Joint_Truncated | 10 | 0.8057 | 0.7770 | 0.8372 | 0.8071 | 0.0309 | +| lichen | north | v2val_Full_Truncated | 10 | 0.8074 | 0.7780 | 0.8390 | 0.8083 | 0.0312 | +| lichen | north | v2val_Landcover | 10 | 0.7424 | 0.7115 | 0.8121 | 0.7566 | 0.0468 | +| lichen | south | insample_auc | 10 | 0.8021 | 0.7789 | 0.8223 | 0.8009 | 0.0226 | +| lichen | south | insample_calibration_slope | 10 | 0.9035 | 0.8740 | 1.0439 | 0.9436 | 0.0783 | +| lichen | south | insample_deviance_explained | 10 | 0.1705 | 0.0924 | 0.2412 | 0.1684 | 0.0753 | +| lichen | south | insample_n | 10 | 1484.5000 | 1478.5000 | 2346.0000 | 1657.8000 | 362.9370 | +| lichen | south | insample_prevalence | 10 | 0.1511 | 0.0868 | 0.2167 | 0.1525 | 0.0659 | +| lichen | south | insample_rmse | 10 | 0.3163 | 0.2688 | 0.3582 | 0.3146 | 0.0440 | +| lichen | south | insample_spearman | 10 | 0.3667 | 0.2735 | 0.4597 | 0.3676 | 0.0948 | +| lichen | south | oob_auc | 8 | 0.7976 | 0.7632 | 0.8185 | 0.7926 | 0.0270 | +| lichen | south | oob_calibration_slope | 8 | 0.9438 | 0.7927 | 1.1880 | 0.9710 | 0.1829 | +| lichen | south | oob_deviance_explained | 8 | 0.1218 | -0.0135 | 0.2395 | 0.1153 | 0.1246 | +| lichen | south | oob_n | 10 | 861.5000 | 0.0000 | 867.5000 | 688.2000 | 362.9370 | +| lichen | south | oob_prevalence | 8 | 0.1578 | 0.0943 | 0.2207 | 0.1582 | 0.0637 | +| lichen | south | oob_rmse | 8 | 0.3261 | 0.2806 | 0.3630 | 0.3232 | 0.0402 | +| lichen | south | oob_spearman | 8 | 0.3688 | 0.2665 | 0.4581 | 0.3651 | 0.0946 | +| lichen | south | oob_v2val_Climate | 8 | 0.7041 | 0.6878 | 0.7158 | 0.7028 | 0.0127 | +| lichen | south | oob_v2val_Climate_Truncated | 8 | 0.7041 | 0.6878 | 0.7158 | 0.7028 | 0.0127 | +| lichen | south | oob_v2val_Full | 8 | 0.7976 | 0.7632 | 0.8185 | 0.7926 | 0.0270 | +| lichen | south | oob_v2val_Full_Joint | 8 | 0.7958 | 0.7656 | 0.8153 | 0.7914 | 0.0240 | +| lichen | south | oob_v2val_Full_Joint_Truncated | 8 | 0.7958 | 0.7656 | 0.8153 | 0.7914 | 0.0240 | +| lichen | south | oob_v2val_Full_Truncated | 8 | 0.7976 | 0.7632 | 0.8185 | 0.7926 | 0.0270 | +| lichen | south | oob_v2val_Landcover | 8 | 0.7535 | 0.7464 | 0.7630 | 0.7537 | 0.0086 | +| lichen | south | v2val_Climate | 10 | 0.7093 | 0.6916 | 0.7289 | 0.7101 | 0.0183 | +| lichen | south | v2val_Climate_Truncated | 10 | 0.7093 | 0.6916 | 0.7290 | 0.7101 | 0.0183 | +| lichen | south | v2val_Full | 10 | 0.8021 | 0.7789 | 0.8223 | 0.8009 | 0.0226 | +| lichen | south | v2val_Full_Joint | 10 | 0.8018 | 0.7799 | 0.8189 | 0.8001 | 0.0202 | +| lichen | south | v2val_Full_Joint_Truncated | 10 | 0.8018 | 0.7799 | 0.8189 | 0.8001 | 0.0202 | +| lichen | south | v2val_Full_Truncated | 10 | 0.8021 | 0.7789 | 0.8223 | 0.8009 | 0.0226 | +| lichen | south | v2val_Landcover | 10 | 0.7535 | 0.7435 | 0.7653 | 0.7532 | 0.0119 | +| mammal_summer | north | insample_auc | 10 | 0.7207 | 0.6085 | 0.8216 | 0.7173 | 0.1099 | +| mammal_summer | north | insample_calibration_slope | 10 | 0.9417 | 0.8161 | 1.0198 | 0.9320 | 0.0845 | +| mammal_summer | north | insample_deviance_explained | 10 | 0.1542 | 0.0268 | 0.2763 | 0.1529 | 0.1296 | +| mammal_summer | north | insample_n | 10 | 2587.5000 | 2542.1000 | 4053.0000 | 2865.2000 | 626.3850 | +| mammal_summer | north | insample_prevalence | 10 | 0.2736 | 0.2243 | 0.3206 | 0.2726 | 0.0496 | +| mammal_summer | north | insample_rmse | 10 | 0.3836 | 0.3234 | 0.4410 | 0.3831 | 0.0608 | +| mammal_summer | north | insample_spearman | 10 | 0.3160 | 0.1627 | 0.4610 | 0.3131 | 0.1542 | +| mammal_summer | north | oob_auc | 8 | 0.7007 | 0.5879 | 0.8116 | 0.7020 | 0.1114 | +| mammal_summer | north | oob_calibration_slope | 8 | 0.8971 | 0.7046 | 0.9590 | 0.8583 | 0.1139 | +| mammal_summer | north | oob_deviance_explained | 8 | 0.1194 | 0.0155 | 0.2466 | 0.1282 | 0.1142 | +| mammal_summer | north | oob_n | 10 | 1465.5000 | 0.0000 | 1510.9000 | 1187.8000 | 626.3850 | +| mammal_summer | north | oob_prevalence | 8 | 0.2774 | 0.2208 | 0.3261 | 0.2746 | 0.0539 | +| mammal_summer | north | oob_rmse | 8 | 0.3919 | 0.3312 | 0.4445 | 0.3895 | 0.0585 | +| mammal_summer | north | oob_spearman | 8 | 0.2911 | 0.1268 | 0.4415 | 0.2892 | 0.1555 | +| mammal_summer | south | insample_auc | 10 | 0.7054 | 0.6336 | 0.8030 | 0.7149 | 0.0833 | +| mammal_summer | south | insample_calibration_slope | 10 | 0.9832 | 0.8710 | 1.0087 | 0.9605 | 0.0585 | +| mammal_summer | south | insample_deviance_explained | 10 | 0.1016 | 0.0257 | 0.2383 | 0.1233 | 0.1035 | +| mammal_summer | south | insample_n | 10 | 676.5000 | 670.8000 | 1065.0000 | 753.7000 | 164.1402 | +| mammal_summer | south | insample_prevalence | 10 | 0.4451 | 0.1814 | 0.7098 | 0.4419 | 0.2704 | +| mammal_summer | south | insample_rmse | 10 | 0.3814 | 0.3405 | 0.4191 | 0.3811 | 0.0380 | +| mammal_summer | south | insample_spearman | 10 | 0.2798 | 0.1705 | 0.4148 | 0.2864 | 0.1148 | +| mammal_summer | south | oob_auc | 8 | 0.6568 | 0.5888 | 0.7557 | 0.6672 | 0.0845 | +| mammal_summer | south | oob_calibration_slope | 8 | 0.6589 | 0.4354 | 1.0000 | 0.7200 | 0.2938 | +| mammal_summer | south | oob_deviance_explained | 8 | 0.0059 | -0.0790 | 0.1356 | 0.0219 | 0.1214 | +| mammal_summer | south | oob_n | 10 | 388.5000 | 0.0000 | 394.2000 | 311.3000 | 164.1402 | +| mammal_summer | south | oob_prevalence | 8 | 0.4195 | 0.1726 | 0.6930 | 0.4251 | 0.2678 | +| mammal_summer | south | oob_rmse | 8 | 0.3986 | 0.3422 | 0.4425 | 0.3948 | 0.0479 | +| mammal_summer | south | oob_spearman | 8 | 0.2181 | 0.1135 | 0.3243 | 0.2204 | 0.1080 | +| mammal_winter | north | insample_auc | 10 | 0.7608 | 0.6687 | 0.8445 | 0.7578 | 0.0888 | +| mammal_winter | north | insample_calibration_slope | 10 | 0.9604 | 0.9077 | 1.0052 | 0.9518 | 0.0647 | +| mammal_winter | north | insample_deviance_explained | 10 | 0.1877 | 0.0712 | 0.3014 | 0.1868 | 0.1161 | +| mammal_winter | north | insample_n | 10 | 2624.5000 | 2596.8000 | 4127.0000 | 2921.0000 | 635.9504 | +| mammal_winter | north | insample_prevalence | 10 | 0.1752 | 0.1467 | 0.2040 | 0.1753 | 0.0287 | +| mammal_winter | north | insample_rmse | 10 | 0.3149 | 0.2994 | 0.3346 | 0.3161 | 0.0165 | +| mammal_winter | north | insample_spearman | 10 | 0.3463 | 0.2134 | 0.4714 | 0.3427 | 0.1314 | +| mammal_winter | north | oob_auc | 8 | 0.7383 | 0.6458 | 0.8353 | 0.7401 | 0.0953 | +| mammal_winter | north | oob_calibration_slope | 8 | 0.8540 | 0.6612 | 0.9533 | 0.8259 | 0.1149 | +| mammal_winter | north | oob_deviance_explained | 8 | 0.1614 | 0.0413 | 0.2853 | 0.1646 | 0.1234 | +| mammal_winter | north | oob_n | 10 | 1502.5000 | 0.0000 | 1530.2000 | 1206.0000 | 635.9504 | +| mammal_winter | north | oob_prevalence | 8 | 0.1756 | 0.1450 | 0.2025 | 0.1746 | 0.0269 | +| mammal_winter | north | oob_rmse | 8 | 0.3180 | 0.3015 | 0.3423 | 0.3198 | 0.0192 | +| mammal_winter | north | oob_spearman | 8 | 0.3198 | 0.1800 | 0.4528 | 0.3184 | 0.1382 | +| mammal_winter | south | insample_auc | 10 | 0.7642 | 0.6906 | 0.8292 | 0.7651 | 0.0684 | +| mammal_winter | south | insample_calibration_slope | 10 | 0.9542 | 0.8588 | 1.0389 | 0.9614 | 0.0769 | +| mammal_winter | south | insample_deviance_explained | 10 | 0.1574 | 0.0675 | 0.2501 | 0.1637 | 0.0946 | +| mammal_winter | south | insample_n | 10 | 694.0000 | 684.8000 | 1098.0000 | 773.7000 | 171.0842 | +| mammal_winter | south | insample_prevalence | 10 | 0.3450 | 0.0865 | 0.6052 | 0.3446 | 0.2704 | +| mammal_winter | south | insample_rmse | 10 | 0.3145 | 0.2215 | 0.4071 | 0.3127 | 0.0960 | +| mammal_winter | south | insample_spearman | 10 | 0.3224 | 0.2964 | 0.3373 | 0.3206 | 0.0167 | +| mammal_winter | south | oob_auc | 8 | 0.7095 | 0.6257 | 0.7915 | 0.7074 | 0.0854 | +| mammal_winter | south | oob_calibration_slope | 8 | 0.7307 | 0.4495 | 0.8353 | 0.6770 | 0.1665 | +| mammal_winter | south | oob_deviance_explained | 8 | 0.0389 | 0.0035 | 0.1901 | 0.0790 | 0.0856 | +| mammal_winter | south | oob_n | 10 | 404.0000 | 0.0000 | 413.2000 | 324.3000 | 171.0842 | +| mammal_winter | south | oob_prevalence | 8 | 0.3501 | 0.0777 | 0.6225 | 0.3492 | 0.2818 | +| mammal_winter | south | oob_rmse | 8 | 0.3337 | 0.2202 | 0.4271 | 0.3264 | 0.1027 | +| mammal_winter | south | oob_spearman | 8 | 0.2450 | 0.1833 | 0.2817 | 0.2399 | 0.0463 | +| mite | north | insample_auc | 10 | 0.7336 | 0.6697 | 0.7962 | 0.7334 | 0.0621 | +| mite | north | insample_calibration_slope | 10 | 1.0028 | 0.9276 | 1.0341 | 0.9867 | 0.0488 | +| mite | north | insample_deviance_explained | 10 | 0.0502 | -0.0427 | 0.1395 | 0.0479 | 0.0867 | +| mite | north | insample_n | 10 | 3630.0000 | 3598.5000 | 5711.0000 | 4041.0000 | 880.4163 | +| mite | north | insample_prevalence | 10 | 0.1545 | 0.0943 | 0.2113 | 0.1531 | 0.0602 | +| mite | north | insample_rmse | 10 | 0.3279 | 0.2873 | 0.3658 | 0.3272 | 0.0404 | +| mite | north | insample_spearman | 10 | 0.2958 | 0.1729 | 0.4186 | 0.2953 | 0.1234 | +| mite | north | oob_auc | 8 | 0.7189 | 0.6461 | 0.7894 | 0.7187 | 0.0708 | +| mite | north | oob_calibration_slope | 8 | 0.9472 | 0.7462 | 1.0295 | 0.9187 | 0.1294 | +| mite | north | oob_deviance_explained | 8 | 0.0324 | -0.0828 | 0.1165 | 0.0211 | 0.1006 | +| mite | north | oob_n | 10 | 2081.0000 | 0.0000 | 2112.5000 | 1670.0000 | 880.4163 | +| mite | north | oob_prevalence | 8 | 0.1509 | 0.0925 | 0.2048 | 0.1495 | 0.0588 | +| mite | north | oob_rmse | 8 | 0.3263 | 0.2853 | 0.3680 | 0.3268 | 0.0416 | +| mite | north | oob_spearman | 8 | 0.2764 | 0.1475 | 0.4035 | 0.2761 | 0.1298 | +| mite | north | oob_v2val_Climate | 8 | 0.6835 | 0.6308 | 0.7383 | 0.6832 | 0.0539 | +| mite | north | oob_v2val_Climate_Truncated | 8 | 0.6834 | 0.6308 | 0.7383 | 0.6832 | 0.0539 | +| mite | north | oob_v2val_Full | 8 | 0.7189 | 0.6461 | 0.7894 | 0.7187 | 0.0708 | +| mite | north | oob_v2val_Full_Joint | 8 | 0.7191 | 0.6447 | 0.7888 | 0.7183 | 0.0712 | +| mite | north | oob_v2val_Full_Joint_Truncated | 8 | 0.7192 | 0.6448 | 0.7887 | 0.7183 | 0.0712 | +| mite | north | oob_v2val_Full_Truncated | 8 | 0.7190 | 0.6462 | 0.7893 | 0.7188 | 0.0708 | +| mite | north | oob_v2val_Landcover | 8 | 0.6850 | 0.6172 | 0.7620 | 0.6890 | 0.0750 | +| mite | north | v2val_Climate | 10 | 0.6950 | 0.6449 | 0.7401 | 0.6928 | 0.0480 | +| mite | north | v2val_Climate_Truncated | 10 | 0.6950 | 0.6449 | 0.7401 | 0.6928 | 0.0480 | +| mite | north | v2val_Full | 10 | 0.7336 | 0.6697 | 0.7962 | 0.7334 | 0.0621 | +| mite | north | v2val_Full_Joint | 10 | 0.7336 | 0.6692 | 0.7962 | 0.7334 | 0.0622 | +| mite | north | v2val_Full_Joint_Truncated | 10 | 0.7336 | 0.6692 | 0.7962 | 0.7334 | 0.0621 | +| mite | north | v2val_Full_Truncated | 10 | 0.7336 | 0.6697 | 0.7961 | 0.7334 | 0.0620 | +| mite | north | v2val_Landcover | 10 | 0.7057 | 0.6335 | 0.7708 | 0.7020 | 0.0683 | +| mite | south | insample_auc | 10 | 0.7997 | 0.7044 | 0.8840 | 0.7968 | 0.0893 | +| mite | south | insample_calibration_slope | 10 | 0.7882 | 0.6097 | 0.9768 | 0.7906 | 0.1787 | +| mite | south | insample_deviance_explained | 10 | 0.0405 | -0.0053 | 0.1740 | 0.0764 | 0.0831 | +| mite | south | insample_n | 10 | 1493.0000 | 1480.9000 | 2363.0000 | 1665.8000 | 367.6444 | +| mite | south | insample_prevalence | 10 | 0.0721 | 0.0556 | 0.0841 | 0.0703 | 0.0144 | +| mite | south | insample_rmse | 10 | 0.2482 | 0.2139 | 0.2709 | 0.2446 | 0.0273 | +| mite | south | insample_spearman | 10 | 0.2519 | 0.1954 | 0.3151 | 0.2550 | 0.0557 | +| mite | south | oob_auc | 8 | 0.7769 | 0.6582 | 0.8839 | 0.7745 | 0.1136 | +| mite | south | oob_calibration_slope | 8 | 0.6139 | 0.5353 | 0.8012 | 0.6444 | 0.1288 | +| mite | south | oob_deviance_explained | 8 | 0.0535 | 0.0095 | 0.2026 | 0.0819 | 0.0851 | +| mite | south | oob_n | 10 | 870.0000 | 0.0000 | 882.1000 | 697.2000 | 367.6444 | +| mite | south | oob_prevalence | 8 | 0.0719 | 0.0489 | 0.0868 | 0.0689 | 0.0174 | +| mite | south | oob_rmse | 8 | 0.2447 | 0.2147 | 0.2788 | 0.2456 | 0.0307 | +| mite | south | oob_spearman | 8 | 0.2233 | 0.1492 | 0.3189 | 0.2298 | 0.0764 | +| mite | south | oob_v2val_Climate | 8 | 0.6835 | 0.6308 | 0.7383 | 0.6832 | 0.0539 | +| mite | south | oob_v2val_Climate_Truncated | 8 | 0.6834 | 0.6308 | 0.7383 | 0.6832 | 0.0539 | +| mite | south | oob_v2val_Full | 8 | 0.7769 | 0.6582 | 0.8839 | 0.7745 | 0.1136 | +| mite | south | oob_v2val_Full_Joint | 8 | 0.7766 | 0.6587 | 0.8846 | 0.7751 | 0.1137 | +| mite | south | oob_v2val_Full_Joint_Truncated | 8 | 0.7766 | 0.6587 | 0.8846 | 0.7751 | 0.1137 | +| mite | south | oob_v2val_Full_Truncated | 8 | 0.7769 | 0.6582 | 0.8839 | 0.7745 | 0.1136 | +| mite | south | oob_v2val_Landcover | 8 | 0.6792 | 0.6125 | 0.7124 | 0.6651 | 0.0460 | +| mite | south | v2val_Climate | 10 | 0.6950 | 0.6449 | 0.7401 | 0.6928 | 0.0480 | +| mite | south | v2val_Climate_Truncated | 10 | 0.6950 | 0.6449 | 0.7401 | 0.6928 | 0.0480 | +| mite | south | v2val_Full | 10 | 0.7997 | 0.7044 | 0.8840 | 0.7968 | 0.0893 | +| mite | south | v2val_Full_Joint | 10 | 0.8014 | 0.7048 | 0.8843 | 0.7975 | 0.0894 | +| mite | south | v2val_Full_Joint_Truncated | 10 | 0.8014 | 0.7048 | 0.8843 | 0.7975 | 0.0894 | +| mite | south | v2val_Full_Truncated | 10 | 0.7997 | 0.7044 | 0.8840 | 0.7968 | 0.0893 | +| mite | south | v2val_Landcover | 10 | 0.7027 | 0.6602 | 0.7282 | 0.6970 | 0.0287 | +| vascular_plant | north | insample_auc | 10 | 0.8331 | 0.8298 | 0.8373 | 0.8332 | 0.0032 | +| vascular_plant | north | insample_calibration_slope | 10 | 0.9949 | 0.9743 | 1.0061 | 0.9918 | 0.0148 | +| vascular_plant | north | insample_deviance_explained | 10 | 0.1616 | 0.1396 | 0.1816 | 0.1602 | 0.0191 | +| vascular_plant | north | insample_n | 10 | 4347.5000 | 4314.6000 | 6874.0000 | 4846.9000 | 1068.6395 | +| vascular_plant | north | insample_prevalence | 10 | 0.3951 | 0.3770 | 0.4211 | 0.3974 | 0.0209 | +| vascular_plant | north | insample_rmse | 10 | 0.4022 | 0.3958 | 0.4067 | 0.4017 | 0.0048 | +| vascular_plant | north | insample_spearman | 10 | 0.5651 | 0.5611 | 0.5668 | 0.5644 | 0.0038 | +| vascular_plant | north | oob_auc | 8 | 0.8283 | 0.8227 | 0.8329 | 0.8284 | 0.0052 | +| vascular_plant | north | oob_calibration_slope | 8 | 0.9755 | 0.9539 | 1.0021 | 0.9776 | 0.0210 | +| vascular_plant | north | oob_deviance_explained | 8 | 0.1803 | 0.1354 | 0.2064 | 0.1750 | 0.0321 | +| vascular_plant | north | oob_n | 10 | 2526.5000 | 0.0000 | 2559.4000 | 2027.1000 | 1068.6395 | +| vascular_plant | north | oob_prevalence | 8 | 0.3990 | 0.3725 | 0.4275 | 0.3991 | 0.0248 | +| vascular_plant | north | oob_rmse | 8 | 0.4052 | 0.3988 | 0.4110 | 0.4048 | 0.0061 | +| vascular_plant | north | oob_spearman | 8 | 0.5545 | 0.5499 | 0.5651 | 0.5564 | 0.0065 | +| vascular_plant | north | oob_v2val_Climate | 8 | 0.6916 | 0.6808 | 0.6996 | 0.6902 | 0.0106 | +| vascular_plant | north | oob_v2val_Climate_Truncated | 8 | 0.6916 | 0.6808 | 0.6996 | 0.6902 | 0.0106 | +| vascular_plant | north | oob_v2val_Full | 8 | 0.8283 | 0.8227 | 0.8329 | 0.8284 | 0.0052 | +| vascular_plant | north | oob_v2val_Full_Joint | 8 | 0.8258 | 0.8182 | 0.8325 | 0.8258 | 0.0065 | +| vascular_plant | north | oob_v2val_Full_Joint_Truncated | 8 | 0.8258 | 0.8183 | 0.8323 | 0.8258 | 0.0064 | +| vascular_plant | north | oob_v2val_Full_Truncated | 8 | 0.8285 | 0.8228 | 0.8328 | 0.8284 | 0.0051 | +| vascular_plant | north | oob_v2val_Landcover | 8 | 0.8009 | 0.7954 | 0.8101 | 0.8024 | 0.0067 | +| vascular_plant | north | v2val_Climate | 10 | 0.6886 | 0.6855 | 0.6950 | 0.6902 | 0.0048 | +| vascular_plant | north | v2val_Climate_Truncated | 10 | 0.6886 | 0.6855 | 0.6949 | 0.6902 | 0.0048 | +| vascular_plant | north | v2val_Full | 10 | 0.8331 | 0.8298 | 0.8373 | 0.8332 | 0.0032 | +| vascular_plant | north | v2val_Full_Joint | 10 | 0.8318 | 0.8275 | 0.8367 | 0.8320 | 0.0039 | +| vascular_plant | north | v2val_Full_Joint_Truncated | 10 | 0.8319 | 0.8277 | 0.8366 | 0.8321 | 0.0038 | +| vascular_plant | north | v2val_Full_Truncated | 10 | 0.8330 | 0.8299 | 0.8373 | 0.8332 | 0.0031 | +| vascular_plant | north | v2val_Landcover | 10 | 0.8087 | 0.8049 | 0.8130 | 0.8092 | 0.0040 | +| vascular_plant | south | insample_auc | 10 | 0.8460 | 0.8042 | 0.8967 | 0.8490 | 0.0462 | +| vascular_plant | south | insample_calibration_slope | 10 | 0.9608 | 0.9208 | 0.9855 | 0.9580 | 0.0297 | +| vascular_plant | south | insample_deviance_explained | 10 | 0.2841 | 0.1600 | 0.3822 | 0.2752 | 0.1074 | +| vascular_plant | south | insample_n | 10 | 1688.5000 | 1654.9000 | 2648.0000 | 1869.6000 | 410.6067 | +| vascular_plant | south | insample_prevalence | 10 | 0.2438 | 0.2212 | 0.2654 | 0.2437 | 0.0227 | +| vascular_plant | south | insample_rmse | 10 | 0.3477 | 0.3070 | 0.3827 | 0.3458 | 0.0374 | +| vascular_plant | south | insample_spearman | 10 | 0.5180 | 0.4628 | 0.5647 | 0.5163 | 0.0529 | +| vascular_plant | south | oob_auc | 8 | 0.8412 | 0.8038 | 0.8909 | 0.8455 | 0.0409 | +| vascular_plant | south | oob_calibration_slope | 8 | 0.9408 | 0.9073 | 0.9701 | 0.9401 | 0.0277 | +| vascular_plant | south | oob_deviance_explained | 8 | 0.2459 | 0.1728 | 0.3387 | 0.2532 | 0.0850 | +| vascular_plant | south | oob_n | 10 | 959.5000 | 0.0000 | 993.1000 | 778.4000 | 410.6067 | +| vascular_plant | south | oob_prevalence | 8 | 0.2418 | 0.2131 | 0.2706 | 0.2421 | 0.0259 | +| vascular_plant | south | oob_rmse | 8 | 0.3513 | 0.3168 | 0.3827 | 0.3502 | 0.0336 | +| vascular_plant | south | oob_spearman | 8 | 0.5033 | 0.4643 | 0.5589 | 0.5099 | 0.0446 | +| vascular_plant | south | oob_v2val_Climate | 8 | 0.6916 | 0.6808 | 0.6996 | 0.6902 | 0.0106 | +| vascular_plant | south | oob_v2val_Climate_Truncated | 8 | 0.6916 | 0.6808 | 0.6996 | 0.6902 | 0.0106 | +| vascular_plant | south | oob_v2val_Full | 8 | 0.8412 | 0.8038 | 0.8909 | 0.8455 | 0.0409 | +| vascular_plant | south | oob_v2val_Full_Joint | 8 | 0.8369 | 0.8023 | 0.8809 | 0.8406 | 0.0373 | +| vascular_plant | south | oob_v2val_Full_Joint_Truncated | 8 | 0.8369 | 0.8023 | 0.8809 | 0.8406 | 0.0373 | +| vascular_plant | south | oob_v2val_Full_Truncated | 8 | 0.8412 | 0.8038 | 0.8908 | 0.8455 | 0.0409 | +| vascular_plant | south | oob_v2val_Landcover | 8 | 0.7762 | 0.7677 | 0.7961 | 0.7793 | 0.0125 | +| vascular_plant | south | v2val_Climate | 10 | 0.6886 | 0.6855 | 0.6950 | 0.6902 | 0.0048 | +| vascular_plant | south | v2val_Climate_Truncated | 10 | 0.6886 | 0.6855 | 0.6949 | 0.6902 | 0.0048 | +| vascular_plant | south | v2val_Full | 10 | 0.8460 | 0.8042 | 0.8967 | 0.8490 | 0.0462 | +| vascular_plant | south | v2val_Full_Joint | 10 | 0.8411 | 0.8013 | 0.8920 | 0.8446 | 0.0448 | +| vascular_plant | south | v2val_Full_Joint_Truncated | 10 | 0.8411 | 0.8013 | 0.8920 | 0.8446 | 0.0448 | +| vascular_plant | south | v2val_Full_Truncated | 10 | 0.8460 | 0.8042 | 0.8967 | 0.8490 | 0.0462 | +| vascular_plant | south | v2val_Landcover | 10 | 0.7783 | 0.7609 | 0.8028 | 0.7802 | 0.0202 | diff --git a/3_output/exp_000_parity_v2/tables/.gitkeep b/3_output/exp_000_parity_v2/tables/.gitkeep deleted file mode 100644 index 8b13789..0000000 --- a/3_output/exp_000_parity_v2/tables/.gitkeep +++ /dev/null @@ -1 +0,0 @@ - diff --git a/3_output/exp_000_parity_v2/tables/coverage.csv b/3_output/exp_000_parity_v2/tables/coverage.csv index 6e729eb..cc2de87 100644 --- a/3_output/exp_000_parity_v2/tables/coverage.csv +++ b/3_output/exp_000_parity_v2/tables/coverage.csv @@ -1,15 +1,15 @@ taxon,run,region,season,part,species,draws,stages,has_coefficients,has_grid -bird,bird,north,,,2,100,"climate, landcover, habitat",TRUE,FALSE -bird,bird,south,,,2,100,"climate, landcover, habitat",TRUE,FALSE -bryophyte,bryophyte,north,,,2,100,"climate, habitat",TRUE,FALSE -bryophyte,bryophyte,south,,,2,100,"climate, habitat",TRUE,FALSE -lichen,lichen,north,,,2,100,"climate, habitat",TRUE,FALSE -lichen,lichen,south,,,2,100,"climate, habitat",TRUE,FALSE -mammal,mammal_summer,north,summer,hurdle,2,100,"habitat_presence, habitat_abundance, habitat_total",TRUE,FALSE -mammal,mammal_summer,south,summer,hurdle,2,100,"habitat_presence, habitat_abundance, habitat_total",TRUE,FALSE -mammal,mammal_winter,north,winter,hurdle,2,100,"habitat_presence, habitat_abundance, habitat_total",TRUE,FALSE -mammal,mammal_winter,south,winter,hurdle,2,100,"habitat_presence, habitat_abundance, habitat_total",TRUE,FALSE -mite,mite,north,,,2,100,"climate, habitat",TRUE,TRUE -mite,mite,south,,,2,100,"climate, habitat",TRUE,TRUE -vascular_plant,vascular_plant,north,,,2,100,"climate, habitat",TRUE,TRUE -vascular_plant,vascular_plant,south,,,2,100,"climate, habitat",TRUE,TRUE +bird,bird,north,,,2,5,"climate, landcover, habitat",TRUE,TRUE +bird,bird,south,,,2,5,"climate, landcover, habitat",TRUE,TRUE +bryophyte,bryophyte,north,,,2,5,"climate, habitat",TRUE,TRUE +bryophyte,bryophyte,south,,,2,5,"climate, habitat",TRUE,TRUE +lichen,lichen,north,,,2,5,"climate, habitat",TRUE,TRUE +lichen,lichen,south,,,2,5,"climate, habitat",TRUE,TRUE +mammal,mammal_summer,north,summer,hurdle,2,5,"habitat_presence, habitat_abundance, habitat_total",TRUE,TRUE +mammal,mammal_summer,south,summer,hurdle,2,5,"habitat_presence, habitat_abundance, habitat_total",TRUE,TRUE +mammal,mammal_winter,north,winter,hurdle,2,5,"habitat_presence, habitat_abundance, habitat_total",TRUE,TRUE +mammal,mammal_winter,south,winter,hurdle,2,5,"habitat_presence, habitat_abundance, habitat_total",TRUE,TRUE +mite,mite,north,,,2,5,"climate, habitat",TRUE,TRUE +mite,mite,south,,,2,5,"climate, habitat",TRUE,TRUE +vascular_plant,vascular_plant,north,,,2,5,"climate, habitat",TRUE,TRUE +vascular_plant,vascular_plant,south,,,2,5,"climate, habitat",TRUE,TRUE diff --git a/3_output/exp_000_parity_v2/tables/grid_summary.csv b/3_output/exp_000_parity_v2/tables/grid_summary.csv new file mode 100644 index 0000000..2c15e38 --- /dev/null +++ b/3_output/exp_000_parity_v2/tables/grid_summary.csv @@ -0,0 +1,945 @@ +taxon,run,region,season,part,species,grid_unit,n,median,p10,p90,boot1,mean,sd +bird,bird,north,,,AMRO,SpruceR,5,0.0460214970922903,0.0454744611146199,0.0478061585686823,0.0460214970922903,0.0464553702378501,0.00127197102591019 +bird,bird,north,,,AMRO,Spruce1,5,0.0336956807487701,0.0322810241650681,0.0345043254597905,0.0348507381455713,0.0334501194856929,0.00111752698926678 +bird,bird,north,,,AMRO,Spruce2,5,0.029967501134538,0.028689533256257,0.0309030506053303,0.0314691851923315,0.0298058122210272,0.00115338472906637 +bird,bird,north,,,AMRO,Spruce3,5,0.0258465915567713,0.0248847311078244,0.0269161638569105,0.0275986396962082,0.0258442666230429,0.00110362371041956 +bird,bird,north,,,AMRO,Spruce4,5,0.0234032880077339,0.0227504665773472,0.0245509472534085,0.0252157131909561,0.0235452704143634,0.00100329299043297 +bird,bird,north,,,AMRO,Spruce5,5,0.021731186122784,0.0213608104410011,0.0229344203019209,0.0235320177392087,0.0220072737052839,0.000897599585942329 +bird,bird,north,,,AMRO,Spruce6,5,0.0205987834958812,0.0203164427428931,0.0217537403151274,0.0222602046395292,0.0209098073999222,0.000807535567378126 +bird,bird,north,,,AMRO,Spruce7,5,0.0200852925184205,0.0194255140428591,0.0208590897859755,0.0212606901038818,0.0201004990260241,0.000749493164941381 +bird,bird,north,,,AMRO,Spruce8,5,0.0197353820173931,0.0187353929181975,0.020184041325968,0.0204546292796539,0.0194949645525674,0.000737409567340656 +bird,bird,north,,,AMRO,PineR,5,0.055663430449507,0.0543192030149498,0.0562118535015652,0.0535435973650151,0.0554222551033113,0.00109941804600945 +bird,bird,north,,,AMRO,Pine1,5,0.0400367096048969,0.0393476184649974,0.0404105539729648,0.0405470054004968,0.0398971042060573,0.000533927985762279 +bird,bird,north,,,AMRO,Pine2,5,0.0356069417161638,0.0348465927331193,0.0362895221196775,0.0366127459514038,0.0355486479216501,0.000734768131510243 +bird,bird,north,,,AMRO,Pine3,5,0.0307105378920888,0.0301269732104456,0.0316527175268143,0.0321095693334898,0.0308232640754777,0.000810293506935437 +bird,bird,north,,,AMRO,Pine4,5,0.0278758021291746,0.027484805318442,0.0288506206173034,0.0293371593640406,0.02808218595386,0.000767241565317905 +bird,bird,north,,,AMRO,Pine5,5,0.0261879900411373,0.0256371542993188,0.0269648161579907,0.0273782680404298,0.0262492756503036,0.000708980982835126 +bird,bird,north,,,AMRO,Pine6,5,0.0247842187753593,0.0243295516540588,0.0256818169038513,0.0258985802241854,0.0249420574021771,0.000679984539350903 +bird,bird,north,,,AMRO,Pine7,5,0.0237257385224398,0.0233340026010986,0.0247312486059623,0.0247356975011427,0.0239786994144854,0.000706043794957849 +bird,bird,north,,,AMRO,Pine8,5,0.0229103307685888,0.022572936039687,0.0241270556938794,0.0237978880218547,0.0232585155386345,0.000793690250372404 +bird,bird,north,,,AMRO,DecidR,5,0.0229559115592863,0.0221306044429141,0.0235352610263104,0.0237579890560399,0.0228382610280493,0.000702614250549723 +bird,bird,north,,,AMRO,Decid1,5,0.030553448874834,0.0298317579364032,0.031292026146399,0.0315584625182446,0.0305372824644636,0.000731937077174177 +bird,bird,north,,,AMRO,Decid2,5,0.0321828552741194,0.0315537397610501,0.0329318625383302,0.0331998686079368,0.0322057305060789,0.000695310231250941 +bird,bird,north,,,AMRO,Decid3,5,0.0324092194941181,0.031901614565924,0.0331074352588107,0.0333598077618143,0.0324673474226511,0.000609213373089749 +bird,bird,north,,,AMRO,Decid4,5,0.0309081262409317,0.0304692552537095,0.0315217987870488,0.0317500403920757,0.0309728473938434,0.000540903174694158 +bird,bird,north,,,AMRO,Decid5,5,0.0286739687567556,0.0282544017321468,0.0291788293287055,0.0293809707313715,0.0287147521596427,0.000501576553529028 +bird,bird,north,,,AMRO,Decid6,5,0.026263369067971,0.0256592232040337,0.0265292756315766,0.0267049789894919,0.0261396737871554,0.000487038563065855 +bird,bird,north,,,AMRO,Decid7,5,0.0235829355542547,0.022996829053316,0.0238783254234698,0.0239651842606415,0.0234890258373333,0.000486787939794446 +bird,bird,north,,,AMRO,Decid8,5,0.0209712010709133,0.020405623716483,0.0212840394420598,0.0212996862068994,0.0209009094923812,0.000490950705287533 +bird,bird,north,,,AMRO,MixedwoodR,5,0.0216440418420787,0.0214366764666218,0.0229103918922939,0.0235968351556159,0.0220089762264377,0.000908464817530186 +bird,bird,north,,,AMRO,Mixedwood1,5,0.0291224189433723,0.0286974308645267,0.0305147340021009,0.0313443968700871,0.0294293794567081,0.0010999235629169 +bird,bird,north,,,AMRO,Mixedwood2,5,0.0306755089393133,0.0302969705278782,0.0321556893179631,0.0329746690631811,0.0310373758199453,0.00111497918591993 +bird,bird,north,,,AMRO,Mixedwood3,5,0.0308912709528129,0.030589685080101,0.0323492807237952,0.0331335233264806,0.0312892785989983,0.00106272343336921 +bird,bird,north,,,AMRO,Mixedwood4,5,0.0295710415430225,0.0291898679692672,0.0307877153702325,0.0315346752432942,0.0298485867463704,0.000975648708630527 +bird,bird,north,,,AMRO,Mixedwood5,5,0.0274075295438599,0.0270457330870442,0.0285447239533767,0.0291816753271841,0.0276719693316288,0.000884980899063123 +bird,bird,north,,,AMRO,Mixedwood6,5,0.0248744913880928,0.024598727956784,0.0260231600770452,0.0265238352270822,0.0251899056928364,0.000802290447826292 +bird,bird,north,,,AMRO,Mixedwood7,5,0.0223638470242142,0.0220520253761532,0.0234214558496831,0.0238026249249641,0.0226350829214757,0.000731084140709001 +bird,bird,north,,,AMRO,Mixedwood8,5,0.0199126764372763,0.0195738570800957,0.0208756562752855,0.0211552073327846,0.0201406033213722,0.000670966355515117 +bird,bird,north,,,AMRO,TreedBogR,5,0.0331850952367972,0.0319507285402324,0.0343298174584785,0.0320053956170601,0.0331519445129201,0.00118442927359984 +bird,bird,north,,,AMRO,TreedBog1,5,0.0441859758905737,0.04277191207391,0.0459514038332503,0.0425137445589404,0.0443337761762291,0.00156198810108254 +bird,bird,north,,,AMRO,TreedBog2,5,0.0465280988683981,0.0450759279854423,0.0485442094151598,0.0447249523823355,0.0467597382128024,0.00170719065750671 +bird,bird,north,,,AMRO,TreedBog3,5,0.0468127655397305,0.0453501172953296,0.0491179865619394,0.0449404132061633,0.0471460379457872,0.00186526187035362 +bird,bird,north,,,AMRO,TreedBog4,5,0.0445965703925972,0.0431609344939285,0.0470359967670578,0.0427718272455254,0.0449814750753084,0.00192768335450474 +bird,bird,north,,,AMRO,TreedBog5,5,0.0413013718566653,0.0399132456880923,0.043776679654854,0.0395803529352897,0.0417070190418488,0.00193042065727443 +bird,bird,north,,,AMRO,TreedBog6,5,0.0375650211610718,0.03623802297114,0.0400079459227759,0.0359754108602334,0.0379711836795942,0.0018921239404454 +bird,bird,north,,,AMRO,TreedBog7,5,0.0337311435879773,0.032474706589339,0.0360937692651888,0.0322845170729035,0.0341246159398247,0.00182499536957324 +bird,bird,north,,,AMRO,TreedBog8,5,0.0299955276096956,0.0288159490396316,0.0322449753172705,0.028693711490609,0.0303679835711985,0.00173785947843209 +bird,bird,north,,,AMRO,TreedFenR,5,0.0422645144129336,0.0411206711160944,0.0444612609388868,0.0416408069894085,0.0426593947598897,0.00168457584655721 +bird,bird,north,,,AMRO,TreedFen1,5,0.0567852320289917,0.0546993980136261,0.0595487401346133,0.0553127557852217,0.057053309735497,0.00238325844479936 +bird,bird,north,,,AMRO,TreedFen2,5,0.0601053333210814,0.057576793915605,0.062801721494816,0.0581896606449257,0.0601748343893455,0.0025620770336273 +bird,bird,north,,,AMRO,TreedFen3,5,0.061004684620066,0.0578987833513519,0.0632661157627814,0.0584699872088007,0.0606684764795377,0.00263876467136345 +bird,bird,north,,,AMRO,TreedFen4,5,0.0585750472403725,0.0551364039392565,0.060286084605513,0.0556485357726957,0.0578783628836603,0.002555165865925 +bird,bird,north,,,AMRO,TreedFen5,5,0.0546495586388256,0.0510462396551786,0.0558163205477266,0.0514962494721526,0.053660035636928,0.00240507475761092 +bird,bird,north,,,AMRO,TreedFen6,5,0.0499841351763957,0.0464156762019834,0.0507663812147309,0.0468060185201128,0.0488485601936305,0.0022298132966526 +bird,bird,north,,,AMRO,TreedFen7,5,0.0447008747250632,0.0416684838734553,0.0457440880993633,0.0420039595905649,0.0438953797644894,0.00204991323315053 +bird,bird,north,,,AMRO,TreedFen8,5,0.0395694346049872,0.0370458190070238,0.0408183339762998,0.0373321210050387,0.0390587727531739,0.00187548466068124 +bird,bird,north,,,AMRO,GrassHerb,5,0.0391157074651561,0.0374665801039995,0.0407064521459671,0.0414109112421211,0.0390437971413913,0.0016402212390184 +bird,bird,north,,,AMRO,Shrub,5,0.0242992189479925,0.0228825094225992,0.0246776245457063,0.0249051907368317,0.0238911541881936,0.000924348597764569 +bird,bird,north,,,AMRO,GraminoidFen,5,0.0240253006082896,0.0229416430615678,0.0249398229795977,0.0240253006082896,0.023983448309286,0.000993958517209816 +bird,bird,north,,,AMRO,Marsh,5,0.0190409138857816,0.0176773805870109,0.0216818753326244,0.0201855790618525,0.0195090503185658,0.00208856238176789 +bird,bird,north,,,AMRO,RoughP,5,0.0284487010336342,0.0273540658384352,0.029100651467195,0.0283163935552878,0.028312104577748,0.000976790108762144 +bird,bird,north,,,AMRO,TameP,5,0.0337677662581705,0.0332118392222873,0.036508679760649,0.037370361410168,0.0345916414986334,0.00176937305868641 +bird,bird,north,,,AMRO,Crop,5,0.0303304446509559,0.0292521806779319,0.0312352000249702,0.0316675460758044,0.0302312594816356,0.00100540739345621 +bird,bird,north,,,AMRO,Industrial,5,0.0347264320174887,0.033312971177683,0.0355682981118906,0.0357679771660033,0.0344866786749583,0.00111665066023152 +bird,bird,north,,,AMRO,Mine,5,0.0436153300361437,0.0381990210765919,0.0459245387062479,0.0436153300361437,0.0423961825295189,0.00389106674125516 +bird,bird,north,,,AMRO,Rural,5,0.0396443515023387,0.038005707047867,0.0414176232218955,0.041757773007241,0.0396782863611264,0.0016777934772785 +bird,bird,north,,,AMRO,Urban,5,0.0403436023988292,0.0374075928778925,0.0427692941093427,0.0377072326980573,0.040094022635699,0.00264783886461812 +bird,bird,north,,,AMRO,CCSpruceR,5,0.0464207068040547,0.0457292776866576,0.0488055453828144,0.0464207068040547,0.0470442907734406,0.00167197396263071 +bird,bird,north,,,AMRO,CCSpruce1,5,0.0334382471003622,0.0328903186279197,0.0349792099606696,0.0351307392341287,0.0338342587671066,0.00104802117766549 +bird,bird,north,,,AMRO,CCSpruce2,5,0.0297844962219075,0.0291681742313691,0.0312403869295451,0.0316709611104685,0.0300774692961451,0.00106585636584408 +bird,bird,north,,,AMRO,CCSpruce3,5,0.0257839800291967,0.0250483982402682,0.0270314998715563,0.0276685704867883,0.0259369345988329,0.00106777390956264 +bird,bird,north,,,AMRO,CCSpruce4,5,0.0233883758956751,0.0227898099129418,0.0245785085975324,0.0252324884758929,0.0235674959495998,0.000996700398796307 +bird,bird,north,,,AMRO,CCPineR,5,0.0556588717866301,0.0544149295122647,0.058052743467746,0.0540080569202541,0.0561277724704085,0.00180666291382705 +bird,bird,north,,,AMRO,CCPine1,5,0.0402344808099467,0.0399196288909847,0.0408216202332076,0.0408727719768734,0.040357160451343,0.00046025801408057 +bird,bird,north,,,AMRO,CCPine2,5,0.0358025164520706,0.0354021820907293,0.0364721654805386,0.0368475016460527,0.0358739771129343,0.000590299549843217 +bird,bird,north,,,AMRO,CCPine3,5,0.0306877354469581,0.0303682798458408,0.0317012367664505,0.0321909301394351,0.0309342419940114,0.000755692302324986 +bird,bird,north,,,AMRO,CCPine4,5,0.0279566387258189,0.0275081026095794,0.0288622599803365,0.0293566765279551,0.0281088057786237,0.000758937295851063 +bird,bird,north,,,AMRO,CCDecidR,5,0.0235157538328627,0.0220205659375402,0.0240398063704617,0.0239640757885976,0.0231324711414825,0.00101843877053024 +bird,bird,north,,,AMRO,CCDecid1,5,0.0311469864460262,0.0297314532115381,0.0318112029859532,0.031776862556441,0.0308482456975221,0.00104616123097925 +bird,bird,north,,,AMRO,CCDecid2,5,0.0325949076990241,0.0314791572519253,0.0332838607667157,0.0333516171167642,0.0324211482728123,0.000897113117886519 +bird,bird,north,,,AMRO,CCDecid3,5,0.0325265668622635,0.0318777585482393,0.033207441139438,0.0334030551273305,0.0325285168793225,0.00065734364091383 +bird,bird,north,,,AMRO,CCDecid4,5,0.0309528425918292,0.0304601698476958,0.0315598462965716,0.0317664981841116,0.0309961301025662,0.000557779201706145 +bird,bird,north,,,AMRO,CCMixedwoodR,5,0.0224136935821678,0.0213872957159068,0.0232466535727542,0.0238015239718475,0.0222860528997309,0.000988289616189019 +bird,bird,north,,,AMRO,CCMixedwood1,5,0.0296185552493973,0.0288303211553137,0.0308120520947235,0.0315613154690116,0.0297222065212846,0.00110977637889517 +bird,bird,north,,,AMRO,CCMixedwood2,5,0.0308728597959445,0.0305223049649659,0.0323025377803166,0.033125388239769,0.0312402265656467,0.00108688467332021 +bird,bird,north,,,AMRO,CCMixedwood3,5,0.0310031221297793,0.0306928887240712,0.0323595803262164,0.033176477338817,0.0313468792258484,0.00104120784061213 +bird,bird,north,,,AMRO,CCMixedwood4,5,0.0295708962308159,0.0292462349147995,0.0307916306804173,0.0315510213997294,0.0298705111701574,0.00096556171349458 +bird,bird,north,,,AMRO,ShrubbySwamp,5,0.0280532806155646,0.0272120492548132,0.0304723043995906,0.0293807705242182,0.028642432626409,0.00168768533141114 +bird,bird,north,,,AMRO,ShrubbyBog,5,0.0312066411236273,2.39623255235884e-06,0.0912997718305207,0.0987592495876081,0.0420163327688555,0.0456017647742937 +bird,bird,north,,,AMRO,ShrubbyFen,5,0.0199096452985301,0.0173227067986502,0.0204962269519268,0.0179711671460881,0.0191153200531081,0.00163113398716313 +bird,bird,north,,,AMRO,TreedSwamp,5,0.0150538387752528,0.0144236036440235,0.0158664078264896,0.0147780567655432,0.0151226098591078,0.000732979915113703 +bird,bird,north,,,YEWA,SpruceR,5,0.0258559745060982,0.0234403345263018,0.0260111209530433,0.0244391479557886,0.0250156649200479,0.00141587922713189 +bird,bird,north,,,YEWA,Spruce1,5,0.0228379925381185,0.0217902474136419,0.0235046389346874,0.0217244417914461,0.0226588624294516,0.000867039120790799 +bird,bird,north,,,YEWA,Spruce2,5,0.0202131466194631,0.0195276592826221,0.0212015763249332,0.0191951189565087,0.0203300867521231,0.000872220968316701 +bird,bird,north,,,YEWA,Spruce3,5,0.0152260266705766,0.014857155524532,0.0172302414914592,0.0147168214368909,0.015884604119312,0.00122343184525584 +bird,bird,north,,,YEWA,Spruce4,5,0.0111004447615773,0.011013191632923,0.0132562242110949,0.0110142694942302,0.0119062108855284,0.0011988279728585 +bird,bird,north,,,YEWA,Spruce5,5,0.00804666353131241,0.00782404584193233,0.00956383307230264,0.00804666353131241,0.00855088083029275,0.000905640135543961 +bird,bird,north,,,YEWA,Spruce6,5,0.00573844799961777,0.00536607662283719,0.00647092405422643,0.00573844799961777,0.00588369887690215,0.000555796810229676 +bird,bird,north,,,YEWA,Spruce7,5,0.0039947676530843,0.0035660316198102,0.00416241768282985,0.0039947676530843,0.00388279318439441,0.00030409785039464 +bird,bird,north,,,YEWA,Spruce8,5,0.00235082179047476,0.00227361425460549,0.00270649382245452,0.00271460762119826,0.00246262309860069,0.000223214918491482 +bird,bird,north,,,YEWA,PineR,5,0.0205785305776538,0.016318274681477,0.0214352987031151,0.0205785305776538,0.019217883621867,0.00262425459458988 +bird,bird,north,,,YEWA,Pine1,5,0.0182926626777856,0.0153708906689175,0.01895474719957,0.0182926626777856,0.0173660023342792,0.00184714292249953 +bird,bird,north,,,YEWA,Pine2,5,0.0161628933669374,0.0141454379072083,0.0166341234332659,0.0161628933669374,0.0155489855935594,0.00126842607226756 +bird,bird,north,,,YEWA,Pine3,5,0.0123056049633671,0.0114779279888745,0.0125556597430834,0.0123920261251661,0.0121106264069309,0.000641272645106542 +bird,bird,north,,,YEWA,Pine4,5,0.00925626382247502,0.00858692715044315,0.00939389903902125,0.00927436103695466,0.00906124080178095,0.000445903613405282 +bird,bird,north,,,YEWA,Pine5,5,0.00657162382380723,0.00614692336606859,0.00680475100874576,0.0067755435593238,0.00650526021516012,0.000331932489863419 +bird,bird,north,,,YEWA,Pine6,5,0.00453218358829318,0.00422748241429677,0.00472061781142898,0.00483195354609082,0.00448089947179597,0.000253288198947572 +bird,bird,north,,,YEWA,Pine7,5,0.00285709675383384,0.00277596123213307,0.00323273918784264,0.00336371989925068,0.00296438042813188,0.000247212233446868 +bird,bird,north,,,YEWA,Pine8,5,0.00182760895381318,0.00166130029618671,0.00216184116705074,0.00228578492344407,0.00188735457061119,0.000259401049664766 +bird,bird,north,,,YEWA,DecidR,5,0.067934626437015,0.0674576672463633,0.0729195663394245,0.067934626437015,0.0695161849099952,0.00337617618393999 +bird,bird,north,,,YEWA,Decid1,5,0.0765302527003605,0.0753508333805385,0.0809224163873259,0.0757433785005077,0.0775653404553756,0.0034849797906858 +bird,bird,north,,,YEWA,Decid2,5,0.0831933705848618,0.0821163970415955,0.0879817188793552,0.0824551590396893,0.084437018083834,0.00354655002105156 +bird,bird,north,,,YEWA,Decid3,5,0.0909542914829032,0.0906172695168214,0.09667461567637,0.0909542914829032,0.0929168749649886,0.00348284090180142 +bird,bird,north,,,YEWA,Decid4,5,0.0911816141264914,0.0902820435873628,0.0960411655989214,0.0911816141264914,0.0926283291485939,0.00310481576821379 +bird,bird,north,,,YEWA,Decid5,5,0.0830749476256945,0.0815385432720113,0.0862567937273009,0.0830749476256945,0.0836509205612147,0.00245302779264785 +bird,bird,north,,,YEWA,Decid6,5,0.0684511157991116,0.0668156484857424,0.0701683129834214,0.0687878278508456,0.0684357600162802,0.00175553578959361 +bird,bird,north,,,YEWA,Decid7,5,0.0502439884002606,0.0495409659411423,0.052147813318469,0.0517644774974931,0.0507231692666107,0.00130741350672055 +bird,bird,north,,,YEWA,Decid8,5,0.0339005594888284,0.0329378261237809,0.0352547173481443,0.0354022441617264,0.0340630233668366,0.00114017049560887 +bird,bird,north,,,YEWA,MixedwoodR,5,0.0321181710498639,0.0319864486524918,0.0343492759912242,0.0321181710498639,0.0329514089784471,0.00125801887647939 +bird,bird,north,,,YEWA,Mixedwood1,5,0.0361186343792291,0.0357543657081755,0.0381193002267473,0.0358099972600774,0.0367664871038007,0.00122372290033736 +bird,bird,north,,,YEWA,Mixedwood2,5,0.0396578845880583,0.0389647638764836,0.0412876201913861,0.0389831966535626,0.0400235347483558,0.00116873050229461 +bird,bird,north,,,YEWA,Mixedwood3,5,0.044161071804344,0.0430114778126448,0.0450311260127699,0.0430014212895615,0.0440431761917261,0.00100387518506072 +bird,bird,north,,,YEWA,Mixedwood4,5,0.0442351820713005,0.0430824404362304,0.0445653556762868,0.0431088950173672,0.0439072599866513,0.000761025409505987 +bird,bird,north,,,YEWA,Mixedwood5,5,0.0398578791525603,0.0391443456983723,0.0400558611174636,0.0392762206512514,0.0396532399203009,0.000461594422361296 +bird,bird,north,,,YEWA,Mixedwood6,5,0.0324703166414877,0.032179685405207,0.0326999878711223,0.0325215480961012,0.0324424353965295,0.000268202474568964 +bird,bird,north,,,YEWA,Mixedwood7,5,0.0239356684881771,0.0236920166090268,0.0244336977342141,0.0244732389028849,0.0240472848311377,0.00037626223519239 +bird,bird,north,,,YEWA,Mixedwood8,5,0.0161253113583632,0.0156693072501918,0.0166180780337648,0.0167374929865786,0.0161502413826242,0.000485932289158991 +bird,bird,north,,,YEWA,TreedBogR,5,0.011556676565964,0.0106150734211072,0.015167132081919,0.0131547525960213,0.0125010321176789,0.00246609868223356 +bird,bird,north,,,YEWA,TreedBog1,5,0.0130294668017344,0.0118161055388771,0.0168577424412391,0.0146668268778186,0.0139449259066216,0.00270635025396768 +bird,bird,north,,,YEWA,TreedBog2,5,0.0143062189240542,0.0128370960433458,0.0182880974231299,0.0159664853451186,0.0151767961731582,0.00290241602990773 +bird,bird,north,,,YEWA,TreedBog3,5,0.0159307025000537,0.0140854433394799,0.0200138051382188,0.0176122437813552,0.0166943895130734,0.00311323961347103 +bird,bird,north,,,YEWA,TreedBog4,5,0.0159574371006158,0.0140225317629725,0.0198790018140412,0.0176562621751068,0.0166376260902294,0.00304031786082071 +bird,bird,north,,,YEWA,TreedBog5,5,0.0143783651328877,0.0126656720514962,0.0179214155743405,0.0160865002173325,0.0150221957500339,0.00270292011233113 +bird,bird,north,,,YEWA,TreedBog6,5,0.0116539735513248,0.0103793735442064,0.0146647056806911,0.0133199651555387,0.012288662885541,0.00219011143934681 +bird,bird,north,,,YEWA,TreedBog7,5,0.00849682553839672,0.00771705937769765,0.0108920493747369,0.0100235907733028,0.00910814502782403,0.00162151072877657 +bird,bird,north,,,YEWA,TreedBog8,5,0.00557259559451445,0.00520557300073712,0.00734335343414964,0.00685523403478533,0.00611717440175423,0.00110087226572051 +bird,bird,north,,,YEWA,TreedFenR,5,0.0236371088398181,0.0172768808475891,0.024924502499633,0.0236371088398181,0.0216232860603613,0.00391878121541672 +bird,bird,north,,,YEWA,TreedFen1,5,0.0263540785518553,0.0192542019396591,0.0279298914574923,0.0263540785518553,0.0241380888094778,0.00443627285396668 +bird,bird,north,,,YEWA,TreedFen2,5,0.0286893690426434,0.0209405142100181,0.0305054446689458,0.0286893690426434,0.0262867689668403,0.00489124468060123 +bird,bird,north,,,YEWA,TreedFen3,5,0.031404364802713,0.0230203258078457,0.0337911333203274,0.0316465490426032,0.0289430156828602,0.00548345748227703 +bird,bird,north,,,YEWA,TreedFen4,5,0.0310353005156193,0.0229524913747421,0.0338582352701967,0.0317256434654338,0.0288609514455203,0.00550888542351665 +bird,bird,north,,,YEWA,TreedFen5,5,0.0278154609778999,0.0207556385456937,0.0306353026947788,0.0289050176894887,0.0260631606546629,0.0049565302639343 +bird,bird,north,,,YEWA,TreedFen6,5,0.0226089781361948,0.0170225823525394,0.0250329038492433,0.023933970922363,0.0213158555743729,0.00399734472991934 +bird,bird,north,,,YEWA,TreedFen7,5,0.0166663297897427,0.0126617741481907,0.0184756251524069,0.0180108827091141,0.0157894768182317,0.00289715122347528 +bird,bird,north,,,YEWA,TreedFen8,5,0.0111420056428385,0.00854164778652357,0.0123193300297175,0.0123178229176012,0.010594291818947,0.0018963603193835 +bird,bird,north,,,YEWA,GrassHerb,5,0.0932809108920782,0.0913238743042866,0.099354153490354,0.0901967196519561,0.0947526791900974,0.00452534589281357 +bird,bird,north,,,YEWA,Shrub,5,0.0137692356170479,0.0119314278078182,0.0157329585961985,0.0157895871550544,0.0138175228868595,0.00189032792293432 +bird,bird,north,,,YEWA,GraminoidFen,5,0.0568382240230587,0.0469653962284414,0.0581610479717628,0.0584309966782358,0.0538402009564441,0.00660625447641818 +bird,bird,north,,,YEWA,Marsh,5,0.0523104434118613,0.0513226604208148,0.0592605534505783,0.0508708598171353,0.0546191488316924,0.00413758571332043 +bird,bird,north,,,YEWA,RoughP,5,0.0682370509878256,0.0670669867920174,0.0695536106096402,0.0675367024470256,0.0682490773036854,0.0013325208003094 +bird,bird,north,,,YEWA,TameP,5,0.0388719938949029,0.0351357025693022,0.0407515647841364,0.0388719938949029,0.0381829277637048,0.00280038505345587 +bird,bird,north,,,YEWA,Crop,5,0.0480473182692122,0.0467260154201003,0.0494996772635692,0.0480608449912437,0.048048644626923,0.00153038688777236 +bird,bird,north,,,YEWA,Industrial,5,0.0209926161493617,0.0179403892660471,0.0274121855791778,0.0163774906175213,0.0224244654410295,0.00477087310459995 +bird,bird,north,,,YEWA,Mine,5,0.0298281089999606,0.0252803812327327,0.0438712258060967,0.0425264275216284,0.0335602920110228,0.00942421656959815 +bird,bird,north,,,YEWA,Rural,5,0.0520106523964021,0.0509948165571747,0.0580204328798446,0.0511767629721787,0.0538135079742067,0.00385508094702323 +bird,bird,north,,,YEWA,Urban,5,0.0189358775335448,0.0180723788869236,0.0232307511964699,0.0175580460951392,0.0200793068448389,0.00338352975115308 +bird,bird,north,,,YEWA,CCSpruceR,5,0.0152672175746062,0.0139145872911392,0.0165869364759518,0.0140004857871316,0.0152198272082254,0.00132815720387295 +bird,bird,north,,,YEWA,CCSpruce1,5,0.0145361447342197,0.0133048327331401,0.0151662007660585,0.0129654763620514,0.0142937917361502,0.000959125878049657 +bird,bird,north,,,YEWA,CCSpruce2,5,0.0142897376611622,0.0132388070120826,0.0146721348538594,0.0127102186233497,0.0140660338253786,0.000805593935662693 +bird,bird,north,,,YEWA,CCSpruce3,5,0.0134450799502097,0.0127354661170899,0.0148322925225234,0.0125004237260447,0.0137249541584959,0.00104203775139054 +bird,bird,north,,,YEWA,CCSpruce4,5,0.0107434349248054,0.0105759890760241,0.0127445041685453,0.0105517888223615,0.0114569858337227,0.00114367697811895 +bird,bird,north,,,YEWA,CCPineR,5,0.0117888489972604,0.00978058439715013,0.0136122385160046,0.0117888489972604,0.0117069884728364,0.00190028909974178 +bird,bird,north,,,YEWA,CCPine1,5,0.0109173385362286,0.00950427775443443,0.0124464416038799,0.0109173385362286,0.0109676663715755,0.0014493957862129 +bird,bird,north,,,YEWA,CCPine2,5,0.0107024034987813,0.00963763502050784,0.0118767346358323,0.0107024034987813,0.0107680430445057,0.00110689821889728 +bird,bird,north,,,YEWA,CCPine3,5,0.0105322296684723,0.00985879892133643,0.0109469939924601,0.0105257496024575,0.0104678436776119,0.000624606314885391 +bird,bird,north,,,YEWA,CCPine4,5,0.00888493777781158,0.00827021172815224,0.00903877043393079,0.00888493777781158,0.00872016122181434,0.000433694848090059 +bird,bird,north,,,YEWA,CCDecidR,5,0.0408740518525815,0.0395742880103873,0.0463855635812126,0.0389177958906805,0.0423301854605452,0.00406186997061815 +bird,bird,north,,,YEWA,CCDecid1,5,0.0504993452950062,0.0491803622371971,0.0566278245259164,0.048545974417128,0.0521870283212628,0.00434829871604341 +bird,bird,north,,,YEWA,CCDecid2,5,0.062995265035987,0.0619691334562304,0.0695286705182462,0.0614429350398245,0.0649617151096108,0.00438105204093069 +bird,bird,north,,,YEWA,CCDecid3,5,0.0839595481698585,0.0837446499537815,0.090415392433214,0.0836626760316502,0.0862405157433662,0.00384879137472909 +bird,bird,north,,,YEWA,CCDecid4,5,0.0881841836019783,0.0875140301840755,0.0935026282391147,0.0881841836019783,0.089905070222935,0.00326484969143502 +bird,bird,north,,,YEWA,CCMixedwoodR,5,0.0198089465088669,0.0188166896946456,0.0215401269632987,0.0183995775182405,0.0200471562894921,0.00152526660104984 +bird,bird,north,,,YEWA,CCMixedwood1,5,0.0246050062604319,0.0233792653816228,0.0262550008779679,0.0229515931990476,0.0247196289741307,0.0015772935132775 +bird,bird,north,,,YEWA,CCMixedwood2,5,0.0307677459222674,0.0294152736977039,0.0322590575421635,0.0290490255252145,0.0307779939565969,0.00150046114776195 +bird,bird,north,,,YEWA,CCMixedwood3,5,0.0410887724516163,0.0397071014363877,0.0419786408808556,0.0395540872189119,0.0408732849664851,0.00113007823491948 +bird,bird,north,,,YEWA,CCMixedwood4,5,0.0429775093725811,0.0417350634696179,0.0433309104754257,0.0416917681213261,0.0426142220574469,0.000811468917178181 +bird,bird,north,,,YEWA,ShrubbySwamp,5,0.084170088903374,0.0814218719690635,0.0880371848217152,0.079822188726824,0.0845953556361207,0.00359783935045764 +bird,bird,north,,,YEWA,ShrubbyBog,5,4.8908918803986e-06,4.23856436295343e-06,5.39123857787991e-06,4.42578200846098e-06,4.83402759311359e-06,5.66960969792273e-07 +bird,bird,north,,,YEWA,ShrubbyFen,5,0.0414566214404942,0.039261597045695,0.0505385114750219,0.0484352778746871,0.0440925167844869,0.00577045904748459 +bird,bird,north,,,YEWA,TreedSwamp,5,0.0406512816816615,0.0399137539513559,0.0426021810257371,0.0406996342472028,0.041037209932847,0.00163945654589851 +bird,bird,south,,,AMRO,Loamy,5,0.0241321618221621,0.0236693559161623,0.0244417594641063,0.0239569869686121,0.0240845194350461,0.000397588872294506 +bird,bird,south,,,AMRO,Blowout,5,0.00525978394452474,0.00483082627794943,0.00554114840096604,0.00559111949874543,0.00521832850528163,0.000380997984164263 +bird,bird,south,,,AMRO,ClaySub,5,0.0224545103948504,0.0209236101394104,0.0249199215728223,0.0250347637306298,0.0228949983648061,0.00202623689619071 +bird,bird,south,,,AMRO,Crop,5,0.0230932157889498,0.0226956842202502,0.0238595151902264,0.0236839952497224,0.0232465903153888,0.000582704673265551 +bird,bird,south,,,AMRO,Industrial,5,0.0262354418047225,0.0234169654211887,0.029992373999167,0.0233617397737627,0.0264579449889971,0.00338847243340584 +bird,bird,south,,,AMRO,Mine,5,0.0159069410465305,0.014010684448404,0.025109840836752,0.0159069410465305,0.018698641258821,0.00572004799070757 +bird,bird,south,,,AMRO,Other,5,0.016098468661524,0.0128083644871958,0.0205965606097986,0.0210202102474962,0.016661932215088,0.00387486651255373 +bird,bird,south,,,AMRO,RapidDrain,5,0.0236194782771747,0.0230076960669836,0.0255994765544292,0.0251675270889975,0.0241724133059695,0.00131670630516629 +bird,bird,south,,,AMRO,RoughP,5,0.0117188270413036,0.0109276575293055,0.0122383602743591,0.0112036711001846,0.0116002215755914,0.000670782916246255 +bird,bird,south,,,AMRO,Rural,5,0.0437070809393601,0.0424258544652182,0.0441423406644261,0.0437070809393601,0.0434272383100641,0.00106086054088164 +bird,bird,south,,,AMRO,SandyLoam,5,0.0147924310215059,0.0145440646614329,0.0149852796963371,0.0148469628367301,0.0147628355850776,0.000223475299912133 +bird,bird,south,,,AMRO,SoilUnknown,5,0.0243829782616065,0.0238722939341933,0.0257510776949805,0.0235542385568422,0.0247241157220053,0.00096914474918751 +bird,bird,south,,,AMRO,TameP,5,0.0270019486472823,0.0254866831491566,0.0280561852466744,0.0286155273878474,0.0268110439620644,0.00133276654620004 +bird,bird,south,,,AMRO,ThinBreak,5,0.0338003999398653,0.0330400812092674,0.037911798723991,0.0338003999398653,0.0350955166553413,0.00251362313805057 +bird,bird,south,,,AMRO,Urban,5,0.0508184548862111,0.0491287965989474,0.0516212401204611,0.0483719535027645,0.0505268596480749,0.00133294289276666 +bird,bird,south,,,AMRO,Water,5,0.0162706932878444,0.0146410988852416,0.0316387080129026,0.0364343359767869,0.0213464232510502,0.00935524776744551 +bird,bird,south,,,AMRO,Wellsites,5,0.0389513908707596,0.0124040818853545,0.0420540796368127,0.0437526708429192,0.0306430201674771,0.0177335069810587 +bird,bird,south,,,YEWA,Loamy,5,0.0889862342739654,0.0833142720796077,0.0912893187301941,0.0911235422512483,0.0878843046841073,0.00434504301981026 +bird,bird,south,,,YEWA,Blowout,5,0.0401628947157703,0.0373442373283019,0.0409157196574691,0.0401628947157703,0.0393477535478598,0.00184103064636671 +bird,bird,south,,,YEWA,ClaySub,5,0.0776026172652354,0.072738347688647,0.0806537209465533,0.0776026172652354,0.0770737187243197,0.00428126144830476 +bird,bird,south,,,YEWA,Crop,5,0.0477258909723985,0.0445439418592085,0.0483983633635829,0.0485578693440802,0.0468618523426874,0.00225396025244564 +bird,bird,south,,,YEWA,Industrial,5,0.0527276712827915,0.0407707589259449,0.0732258847421466,0.0332758320869208,0.0564078240303074,0.0167976738050071 +bird,bird,south,,,YEWA,Mine,5,0.0971806065268231,0.0846776257647363,0.121933813451961,0.112500894420852,0.101522798600799,0.0187712933702914 +bird,bird,south,,,YEWA,Other,5,0.0875251951229225,0.0772673767356427,0.0958802062242052,0.100975326566864,0.0863792254951111,0.0097153991353414 +bird,bird,south,,,YEWA,RapidDrain,5,0.0665179523278429,0.0604144987507378,0.0698664068896251,0.0626153705338298,0.0654553589739474,0.00472953584424184 +bird,bird,south,,,YEWA,RoughP,5,0.0573815111243077,0.0533876378460484,0.0600900298992678,0.0564654939180325,0.0569589790219802,0.00361546417207247 +bird,bird,south,,,YEWA,Rural,5,0.0661716927633792,0.0628530852788089,0.0689878709663489,0.0657781939712168,0.0660096425642764,0.00343389323428838 +bird,bird,south,,,YEWA,SandyLoam,5,0.0511277264137194,0.0440509842319305,0.0548605397562365,0.0475053886435979,0.0500154530025099,0.0055363202275427 +bird,bird,south,,,YEWA,SoilUnknown,5,0.0997155153997096,0.0970882713180495,0.100977766167759,0.0979710879919298,0.0991489817590435,0.00200077514407797 +bird,bird,south,,,YEWA,TameP,5,0.0457686292658743,0.0435985886063968,0.0531975721116954,0.050747465664211,0.0477197523221636,0.00492698526781333 +bird,bird,south,,,YEWA,ThinBreak,5,0.183371746527312,0.178987377035422,0.188606353214059,0.179236197079509,0.183711346298015,0.00477581737073191 +bird,bird,south,,,YEWA,Urban,5,0.0507331439696001,0.0448023038764347,0.0520602501639247,0.0507331439696001,0.0488964675676067,0.00370427005030944 +bird,bird,south,,,YEWA,Water,5,5.79878735375159e-07,4.76822525465649e-07,1.2010658345155e-06,6.60804504718257e-07,7.52565880644025e-07,4.58515195214571e-07 +bird,bird,south,,,YEWA,Wellsites,5,6.90003938618652e-06,5.25255285811954e-06,0.365617585043016,0.413267228722246,0.141485598674952,0.19825432154381 +bryophyte,bryophyte,north,,,Bryum.All,WhiteSpruce,5,0.104623666278372,0.0874722861544613,0.112679942833954,0.105666560031913,0.101095355834302,0.0132153348867197 +bryophyte,bryophyte,north,,,Bryum.All,Pine,5,0.0674475602586271,0.0606847013691161,0.0749869714210322,0.0672116807219031,0.0678337227606607,0.00792832957785061 +bryophyte,bryophyte,north,,,Bryum.All,Deciduous,5,0.0919819625968023,0.0817645391225394,0.0978941606981865,0.0919819625968023,0.0907672040562639,0.008858552895604 +bryophyte,bryophyte,north,,,Bryum.All,Mixedwood,5,0.0937191187701653,0.0766558041922239,0.101378049564084,0.0937191187701653,0.0906024818416207,0.0130996076667672 +bryophyte,bryophyte,north,,,Bryum.All,BlackSpruce,5,0.0821737878210885,0.0796472718492078,0.0914478284992744,0.082297728208448,0.0843434814639106,0.00751140230506868 +bryophyte,bryophyte,north,,,Bryum.All,TreedFen,5,0.103825175281253,0.09066068154714,0.115404378478857,0.103825175281253,0.102851672188754,0.0127091628628293 +bryophyte,bryophyte,north,,,Bryum.All,TreedSwamp,5,0.14838302487508,0.134285947262339,0.15433966365031,0.14838302487508,0.145026963697524,0.0100379018142146 +bryophyte,bryophyte,north,,,Bryum.All,GraminoidFen,5,0.102608545171503,0.091652995844401,0.112334612258834,0.102608545171503,0.101863685320028,0.0108083654561642 +bryophyte,bryophyte,north,,,Bryum.All,ShrubbyFen,5,0.108214861633538,0.0993466210779264,0.120681656680431,0.108214861633538,0.109555249188518,0.0105175724757792 +bryophyte,bryophyte,north,,,Bryum.All,ShrubbyBog,5,0.0822115868613973,0.0802689106984959,0.0930553295605826,0.0837594118723497,0.0852163915453797,0.00798883798126357 +bryophyte,bryophyte,north,,,Bryum.All,ShrubbySwamp,5,0.142659930075182,0.128769149552852,0.158781358177345,0.147623294355338,0.143670605582959,0.0164498267712743 +bryophyte,bryophyte,north,,,Bryum.All,Marsh,5,0.103008117733597,0.0928698823634094,0.112264034639262,0.105256814676515,0.102812793714352,0.0108157208529732 +bryophyte,bryophyte,north,,,Bryum.All,Shrub,5,0.205603841480222,0.174880422823466,0.253531593550174,0.19679926045177,0.213170463378129,0.0395744190804504 +bryophyte,bryophyte,north,,,Bryum.All,Grass,5,0.205731498249484,0.170688321636038,0.244995847117187,0.195483946860916,0.207623905262506,0.0385480440688637 +bryophyte,bryophyte,north,,,Bryum.All,CCWhiteSpruceR,5,0.165475582798583,0.15518207397193,0.199998766587222,0.169554083366261,0.173608751875238,0.0270490393438733 +bryophyte,bryophyte,north,,,Bryum.All,CCWhiteSpruce1,5,0.177961608354587,0.154126217921708,0.209058286085648,0.177961608354587,0.18107397234559,0.0302138318102788 +bryophyte,bryophyte,north,,,Bryum.All,CCWhiteSpruce2,5,0.202884480777997,0.171266875481849,0.231723154322466,0.202884480777997,0.200890380741,0.0305589414199888 +bryophyte,bryophyte,north,,,Bryum.All,CCWhiteSpruce3,5,0.202884480777997,0.171266875481849,0.231723154322466,0.202884480777997,0.200890380741,0.0305589414199888 +bryophyte,bryophyte,north,,,Bryum.All,CCWhiteSpruce4,5,0.202884480777997,0.171266875481849,0.231723154322466,0.202884480777997,0.200890380741,0.0305589414199888 +bryophyte,bryophyte,north,,,Bryum.All,CCPineR,5,0.16022882422794,0.145827136293942,0.180254773430031,0.16022882422794,0.16121093845615,0.0194156293916364 +bryophyte,bryophyte,north,,,Bryum.All,CCPine1,5,0.164291720401128,0.133145499888549,0.203845059962508,0.164291720401128,0.16608604674609,0.0373996968191405 +bryophyte,bryophyte,north,,,Bryum.All,CCPine2,5,0.166830898209846,0.13451769307428,0.206946727968588,0.166830898209846,0.168100580656605,0.0385257305816659 +bryophyte,bryophyte,north,,,Bryum.All,CCPine3,5,0.166830898209846,0.13451769307428,0.206946727968588,0.166830898209846,0.168100580656605,0.0385257305816659 +bryophyte,bryophyte,north,,,Bryum.All,CCPine4,5,0.166830898209846,0.13451769307428,0.206946727968588,0.166830898209846,0.168100580656605,0.0385257305816659 +bryophyte,bryophyte,north,,,Bryum.All,CCDeciduousR,5,0.161533823124016,0.1404659727257,0.174401252222805,0.161533823124016,0.158553527581378,0.0169028615434247 +bryophyte,bryophyte,north,,,Bryum.All,CCDeciduous1,5,0.174108402828508,0.161888476473846,0.17922980187219,0.17800935952687,0.171879211863207,0.00993958041925984 +bryophyte,bryophyte,north,,,Bryum.All,CCDeciduous2,5,0.150834982909368,0.122713525430442,0.157034030570364,0.150834982909368,0.142500719768878,0.0176710184164726 +bryophyte,bryophyte,north,,,Bryum.All,CCDeciduous3,5,0.150834982909368,0.122713525430442,0.157034030570364,0.150834982909368,0.142500719768878,0.0176710184164726 +bryophyte,bryophyte,north,,,Bryum.All,CCDeciduous4,5,0.150834982909368,0.122713525430442,0.157034030570364,0.150834982909368,0.142500719768878,0.0176710184164726 +bryophyte,bryophyte,north,,,Bryum.All,CCMixedwoodR,5,0.161533823124016,0.1404659727257,0.174401252222805,0.161533823124016,0.158553527581378,0.0169028615434247 +bryophyte,bryophyte,north,,,Bryum.All,CCMixedwood1,5,0.174108402828508,0.161888476473846,0.17922980187219,0.17800935952687,0.171879211863207,0.00993958041925984 +bryophyte,bryophyte,north,,,Bryum.All,CCMixedwood2,5,0.150834982909368,0.122713525430442,0.157034030570364,0.150834982909368,0.142500719768878,0.0176710184164726 +bryophyte,bryophyte,north,,,Bryum.All,CCMixedwood3,5,0.150834982909368,0.122713525430442,0.157034030570364,0.150834982909368,0.142500719768878,0.0176710184164726 +bryophyte,bryophyte,north,,,Bryum.All,CCMixedwood4,5,0.150834982909368,0.122713525430442,0.157034030570364,0.150834982909368,0.142500719768878,0.0176710184164726 +bryophyte,bryophyte,north,,,Bryum.All,EnSeismic,5,0.143247261154779,0.132144241170755,0.15978556612201,0.143247261154779,0.145235354857167,0.0137587578338995 +bryophyte,bryophyte,north,,,Bryum.All,EnSoftLin,5,0.181976104710237,0.176334441522147,0.207421144310894,0.181976104710237,0.188523522297283,0.0195432641471242 +bryophyte,bryophyte,north,,,Bryum.All,TrSoftLin,5,0.172100168424541,0.147841165917588,0.187901203255329,0.169203477599876,0.169132193136031,0.0230076012908074 +bryophyte,bryophyte,north,,,Bryum.All,HardLin,5,0.119094247542269,0.0902286867109537,0.153196513874268,0.124691301578451,0.121155863573338,0.035675361881301 +bryophyte,bryophyte,north,,,Bryum.All,UrbInd,5,0.15870235487759,0.131129493678025,0.174263860237473,0.15870235487759,0.154671253635583,0.0218939433100647 +bryophyte,bryophyte,north,,,Bryum.All,Wellsites,5,0.133875416816634,0.127095175106388,0.159927124091914,0.133875416816634,0.14193011214681,0.0168562821513424 +bryophyte,bryophyte,north,,,Bryum.All,Crop,5,0.0820223721088346,0.0684673797777877,0.0983546690511286,0.078395085879973,0.0834195722128434,0.0152344649418885 +bryophyte,bryophyte,north,,,Bryum.All,TameP,5,0.14146232517738,0.121223613545195,0.153442524325088,0.14146232517738,0.138962795983375,0.0169647163928601 +bryophyte,bryophyte,north,,,Bryum.All,RoughP,5,0.144150717399513,0.133810027719149,0.158938339577548,0.143669438263278,0.14578740024442,0.0141397169788724 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,WhiteSpruce,5,0.170617726106481,0.150531559521949,0.174419709411094,0.176935668903147,0.164807860658135,0.0139893488460334 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,Pine,5,0.155784234979198,0.149699740009459,0.170954712542573,0.172791155095638,0.159310643176365,0.0106361302435088 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,Deciduous,5,0.157244630521525,0.142941372795819,0.163468234285499,0.162052893569508,0.154697151179962,0.0115856768365642 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,Mixedwood,5,0.158627676901957,0.13860286258159,0.171247492283289,0.15788444654738,0.15647078515815,0.018649527031244 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,BlackSpruce,5,0.0677921007733422,0.0521429659792482,0.0729564235762428,0.0685353715648879,0.0638599149238077,0.0110949308248088 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,TreedFen,5,0.0704119831070097,0.059745149134307,0.0811325128636527,0.0794290899103082,0.0706134625134515,0.0105585327678486 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,TreedSwamp,5,0.112752005656074,0.0984937696101871,0.137854437407414,0.122922619803698,0.116412196602916,0.0203688771304333 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,GraminoidFen,5,0.071246619098096,0.0646705299404144,0.0824857475812227,0.0833874009572022,0.0733295931607397,0.00895134876938999 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,ShrubbyFen,5,0.0704520736989616,0.0644476326851635,0.0829604405865539,0.0826619615495143,0.0733794195671157,0.00945907870284914 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,ShrubbyBog,5,0.0669469626238629,0.0541167658533186,0.0737851895171726,0.0696158682654045,0.0648674857861799,0.0105595763340405 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,ShrubbySwamp,5,0.115517060865014,0.0928903325785583,0.132534115088148,0.121979163778046,0.112959172899961,0.0198288996671609 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,Marsh,5,0.113514973804854,0.096254727615824,0.137347333930943,0.121750013890524,0.115138525017619,0.0213109634132099 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,Shrub,5,0.258030208654547,0.21433734112913,0.288008334755188,0.275203776452487,0.251704525155945,0.0367095721590051 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,Grass,5,0.271067495374639,0.240923289977324,0.31781564474553,0.292406492273734,0.276440725276057,0.0390993119735511 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCWhiteSpruceR,5,0.338256695749658,0.31824880432862,0.364273497154808,0.345415843034206,0.340062112951671,0.0241555437358403 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCWhiteSpruce1,5,0.310222319426016,0.28440620861304,0.33208710700024,0.310222319426016,0.308604286297291,0.0258019292550109 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCWhiteSpruce2,5,0.344912345599046,0.314582284578097,0.372632043273075,0.35184667493467,0.343098817236955,0.0297346606357341 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCWhiteSpruce3,5,0.344912345599046,0.314582284578097,0.372632043273075,0.35184667493467,0.343098817236955,0.0297346606357341 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCWhiteSpruce4,5,0.344912345599046,0.314582284578097,0.372632043273075,0.35184667493467,0.343098817236955,0.0297346606357341 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCPineR,5,0.332237563055896,0.30230266389427,0.354086616306992,0.332237563055896,0.327969786265574,0.0265319724200263 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCPine1,5,0.345687365666162,0.317841589858981,0.373089123054865,0.354005283762296,0.344278366892055,0.0283247255662619 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCPine2,5,0.348225414237899,0.321582172515133,0.377622132855574,0.359033033466831,0.348154712660996,0.0285763459461223 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCPine3,5,0.348225414237899,0.321582172515133,0.377622132855574,0.359033033466831,0.348154712660996,0.0285763459461223 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCPine4,5,0.348225414237899,0.321582172515133,0.377622132855574,0.359033033466831,0.348154712660996,0.0285763459461223 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCDeciduousR,5,0.318830335897459,0.27955994009676,0.332129458277748,0.329698896772244,0.308702802560019,0.0264775701221862 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCDeciduous1,5,0.311890371027914,0.291615240620109,0.319691638401899,0.321691125624866,0.307838810847009,0.0161966294285432 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCDeciduous2,5,0.30432063546174,0.283360736827309,0.339489268918774,0.311263402327306,0.309378509074765,0.0314758135403339 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCDeciduous3,5,0.30432063546174,0.283360736827309,0.339489268918774,0.311263402327306,0.309378509074765,0.0314758135403339 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCDeciduous4,5,0.30432063546174,0.283360736827309,0.339489268918774,0.311263402327306,0.309378509074765,0.0314758135403339 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCMixedwoodR,5,0.318830335897459,0.27955994009676,0.332129458277748,0.329698896772244,0.308702802560019,0.0264775701221862 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCMixedwood1,5,0.311890371027914,0.291615240620109,0.319691638401899,0.321691125624866,0.307838810847009,0.0161966294285432 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCMixedwood2,5,0.30432063546174,0.283360736827309,0.339489268918774,0.311263402327306,0.309378509074765,0.0314758135403339 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCMixedwood3,5,0.30432063546174,0.283360736827309,0.339489268918774,0.311263402327306,0.309378509074765,0.0314758135403339 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,CCMixedwood4,5,0.30432063546174,0.283360736827309,0.339489268918774,0.311263402327306,0.309378509074765,0.0314758135403339 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,EnSeismic,5,0.159710015581722,0.116043191735016,0.17072117947456,0.165838564517478,0.148662018229779,0.0324110247868619 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,EnSoftLin,5,0.159418027877254,0.124794622673536,0.167171385865762,0.168995137454078,0.15011162020904,0.0243734811900978 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,TrSoftLin,5,0.171669259495879,0.127684819372006,0.18790477785131,0.180401709818175,0.162342674332897,0.0340630905757622 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,HardLin,5,0.10993941717652,0.0990227818218094,0.119734614597133,0.113196196589015,0.109200436852334,0.0104589500726513 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,UrbInd,5,0.125398730461756,0.108350462054011,0.129809259709619,0.129273397383927,0.120827281896463,0.0113320859784318 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,Wellsites,5,0.120865154448231,0.106933037026666,0.135907579517197,0.135555384778656,0.121358221869125,0.0143857836418377 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,Crop,5,0.074461797530342,0.0663055752886782,0.0784646756931369,0.080469463459423,0.0727333613705306,0.00612570365671694 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,TameP,5,0.139209058513911,0.127041125175049,0.146790768834801,0.148781802724197,0.137618984018511,0.0100770984650779 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,RoughP,5,0.141247542624157,0.127091822586834,0.155254205987659,0.153373422859104,0.141796780024815,0.0145812899277074 +bryophyte,bryophyte,south,,,Bryum.All,Loamy,5,0.128371275228788,0.126267345563482,0.192554766969062,0.156609764097912,0.150844569501336,0.0388926494419214 +bryophyte,bryophyte,south,,,Bryum.All,SandyLoam,5,0.163687813793554,0.157864329641462,0.257888717822438,0.209805018041589,0.196019214469412,0.0568093152706092 +bryophyte,bryophyte,south,,,Bryum.All,ClaySub,5,0.138847380253595,0.129440293252389,0.192920064856309,0.163480694825708,0.154866293409777,0.0350918992412223 +bryophyte,bryophyte,south,,,Bryum.All,RapidDrain,5,0.151000064270357,0.12841178366917,0.214806584231797,0.179839142499951,0.165207990926408,0.0458845605396699 +bryophyte,bryophyte,south,,,Bryum.All,Blowout,5,0.177847228366661,0.13169779331486,0.213345983888902,0.18962303788414,0.172170997481874,0.0412393742258946 +bryophyte,bryophyte,south,,,Bryum.All,ThinBreak,5,0.108774649500815,0.100351516195694,0.149118127661729,0.13142432263142,0.120624872013578,0.0258432353890413 +bryophyte,bryophyte,south,,,Bryum.All,Other,5,0.153008911192339,0.143619231829843,0.22761954763385,0.192944265564736,0.176977047927853,0.0459440004598543 +bryophyte,bryophyte,south,,,Bryum.All,EnSeismic,5,0.334606425412252,0.156177390749094,0.434765493190856,0.334606425412252,0.302593815200546,0.140891000271527 +bryophyte,bryophyte,south,,,Bryum.All,EnSoftLin,5,0.329548544110659,0.161421900535046,0.471022939170799,0.329548544110659,0.317093878942589,0.158105322154712 +bryophyte,bryophyte,south,,,Bryum.All,TrSoftLin,5,0.375563737003068,0.183612394029062,0.460581289024939,0.383815980210159,0.332867119897228,0.143170601701847 +bryophyte,bryophyte,south,,,Bryum.All,HardLin,5,0.1468328366873,0.0883371659600431,0.208225904838994,0.1468328366873,0.147701185603674,0.0590188717154669 +bryophyte,bryophyte,south,,,Bryum.All,UrbInd,5,0.104701482537215,0.103446912111058,0.180846515565828,0.15376441179003,0.132924773411517,0.0427257127054497 +bryophyte,bryophyte,south,,,Bryum.All,Wellsites,5,0.103827170399858,0.0947594363214667,0.177418037109713,0.140420183947776,0.127451513800266,0.0457368790572256 +bryophyte,bryophyte,south,,,Bryum.All,Crop,5,0.0834326498037117,0.0762103903441286,0.118437315338461,0.100381454106891,0.0935683426293577,0.0228646836791101 +bryophyte,bryophyte,south,,,Bryum.All,TameP,5,0.124085918128196,0.113948945983715,0.179967952926487,0.164423435504234,0.141507431179504,0.0342621971358284 +bryophyte,bryophyte,south,,,Bryum.All,RoughP,5,0.102704627973536,0.0996345261643043,0.149622110445284,0.13419258459678,0.119382463691813,0.0269065385329828 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,Loamy,5,0.0641849871927666,0.050457745645129,0.07270277108154,0.0641849871927666,0.062283620197523,0.0112971952892291 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,SandyLoam,5,0.0742797240604397,0.0646788146747532,0.0864582537717612,0.0742797240604397,0.0755252597804641,0.0107846710275309 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,ClaySub,5,0.0592505109132571,0.0488344242492647,0.0699996689330231,0.0592505109132571,0.0597454510981899,0.010682085375127 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,RapidDrain,5,0.102551453445416,0.0986747245418018,0.114617392978353,0.102551453445416,0.105621369756077,0.00812414366804172 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,Blowout,5,0.108577749315386,0.0975839841162081,0.119450204289074,0.108577749315386,0.109033282223042,0.0116238846665256 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,ThinBreak,5,0.0863196169586629,0.0821949327472845,0.0998445672388945,0.0863196169586629,0.0901856350845793,0.0089444795540914 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,Other,5,0.0345770287640794,0.0255035080368273,0.0522013822338746,0.0345770287640794,0.037612312759654,0.0151375482648069 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,EnSeismic,5,0.21574628197629,0.142477035309188,0.248955821906755,0.16811461642476,0.20078218558815,0.05348224298659 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,EnSoftLin,5,0.211754576394213,0.13866240930989,0.222190780941109,0.169876045775971,0.188478407991055,0.0449378897052303 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,TrSoftLin,5,0.194018984551885,0.147159530414209,0.240287969097993,0.160797502761418,0.194008212581741,0.0456931192595006 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,HardLin,5,0.0304162076535258,0.029166064034397,0.0432718766595132,0.0318703430228878,0.0343665218104774,0.00930525115482798 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,UrbInd,5,0.0356740467063795,0.0286632065765225,0.0429956144747444,0.0356740467063795,0.0357589664307443,0.00723711747496788 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,Wellsites,5,0.0429922761944743,0.0335372326498226,0.0467533160557549,0.0401699600814057,0.0411525817485948,0.00727575405039167 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,Crop,5,0.0228689540457825,0.0221752232262121,0.0277192165390448,0.0251044943216744,0.0243976201715565,0.00308215429653747 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,TameP,5,0.0603942872656395,0.0505159739370433,0.0666385260267113,0.054988635814872,0.0588718286082372,0.00839424264952957 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,RoughP,5,0.0576120452574267,0.0486453851644709,0.0647244005298698,0.0547537316462016,0.0571023350335811,0.00830015570016307 +lichen,lichen,north,,,Cladonia.chlorophaea,WhiteSpruce,5,0.299712823406177,0.267794399850091,0.31752477391247,0.299712823406177,0.294332455268539,0.0254319271895433 +lichen,lichen,north,,,Cladonia.chlorophaea,Pine,5,0.211658158470293,0.187362325848506,0.221031041496282,0.213983673760142,0.205564667451797,0.0169792480382245 +lichen,lichen,north,,,Cladonia.chlorophaea,Deciduous,5,0.242873835985486,0.212311835312585,0.251351915041242,0.242873835985486,0.234339735870853,0.0204421025800494 +lichen,lichen,north,,,Cladonia.chlorophaea,Mixedwood,5,0.236140666947542,0.203901410662992,0.241600545650375,0.236140666947542,0.225630360660801,0.0195391792025033 +lichen,lichen,north,,,Cladonia.chlorophaea,BlackSpruce,5,0.225302562827919,0.196261190714303,0.244967963651107,0.227843083435444,0.22067099355486,0.0249924693619386 +lichen,lichen,north,,,Cladonia.chlorophaea,TreedFen,5,0.224783844707001,0.197704546932338,0.235642483863854,0.224783844707001,0.218150987211939,0.0190646078283693 +lichen,lichen,north,,,Cladonia.chlorophaea,TreedSwamp,5,0.242591050953153,0.214272220682311,0.263632169592523,0.252640022245541,0.239209971917443,0.024497807226558 +lichen,lichen,north,,,Cladonia.chlorophaea,GraminoidFen,5,0.202508662292495,0.17836174844921,0.21210857272356,0.202508662292495,0.19688618246433,0.0169331647912046 +lichen,lichen,north,,,Cladonia.chlorophaea,ShrubbyFen,5,0.199216086052812,0.177644856358741,0.212793282034292,0.199216086052812,0.196320054390399,0.0174207915910404 +lichen,lichen,north,,,Cladonia.chlorophaea,ShrubbyBog,5,0.221625062963166,0.192800896978028,0.237791260930442,0.22188456568578,0.215779368688039,0.0231976975585119 +lichen,lichen,north,,,Cladonia.chlorophaea,ShrubbySwamp,5,0.200597494570593,0.181101882111374,0.22301802236821,0.214356168657179,0.201391766660484,0.0207072653283275 +lichen,lichen,north,,,Cladonia.chlorophaea,Marsh,5,0.175479744974187,0.145546102447274,0.1832220543811,0.175479744974187,0.166610979908685,0.019157106802069 +lichen,lichen,north,,,Cladonia.chlorophaea,Shrub,5,0.106371701235334,0.0787744447962615,0.122122293070246,0.12074279741616,0.10224974598974,0.0217390056603546 +lichen,lichen,north,,,Cladonia.chlorophaea,Grass,5,0.096028783987272,0.0852156012169443,0.115683635060321,0.119014260443607,0.0998143813618244,0.0154318059448801 +lichen,lichen,north,,,Cladonia.chlorophaea,CCWhiteSpruceR,5,0.169230436345503,0.139669943547809,0.195621086516419,0.18615405031743,0.168442254933479,0.0281649616426591 +lichen,lichen,north,,,Cladonia.chlorophaea,CCWhiteSpruce1,5,0.181781734947402,0.144657786769576,0.210046678174516,0.20367100535893,0.17894644898064,0.0325932819474839 +lichen,lichen,north,,,Cladonia.chlorophaea,CCWhiteSpruce2,5,0.208024972338578,0.167844483274645,0.235168093350801,0.225668903881609,0.203677228485813,0.0342607412322384 +lichen,lichen,north,,,Cladonia.chlorophaea,CCWhiteSpruce3,5,0.208024972338578,0.167844483274645,0.235168093350801,0.225668903881609,0.203677228485813,0.0342607412322384 +lichen,lichen,north,,,Cladonia.chlorophaea,CCWhiteSpruce4,5,0.208024972338578,0.167844483274645,0.235168093350801,0.225668903881609,0.203677228485813,0.0342607412322384 +lichen,lichen,north,,,Cladonia.chlorophaea,CCPineR,5,0.167651692032762,0.133400179473255,0.18613840582969,0.177981748325327,0.16153475071322,0.0262767055968338 +lichen,lichen,north,,,Cladonia.chlorophaea,CCPine1,5,0.205931254374661,0.156797989826113,0.214389722872389,0.215966217533035,0.190474350731153,0.0300029266310009 +lichen,lichen,north,,,Cladonia.chlorophaea,CCPine2,5,0.20940040025485,0.161084160390568,0.217535050260925,0.219134448870029,0.194244531984155,0.0295998827337324 +lichen,lichen,north,,,Cladonia.chlorophaea,CCPine3,5,0.20940040025485,0.161084160390568,0.217535050260925,0.219134448870029,0.194244531984155,0.0295998827337324 +lichen,lichen,north,,,Cladonia.chlorophaea,CCPine4,5,0.20940040025485,0.161084160390568,0.217535050260925,0.219134448870029,0.194244531984155,0.0295998827337324 +lichen,lichen,north,,,Cladonia.chlorophaea,CCDeciduousR,5,0.182662436599129,0.143290947030515,0.19168320742142,0.185594844139235,0.170905137694288,0.0249018018786431 +lichen,lichen,north,,,Cladonia.chlorophaea,CCDeciduous1,5,0.169020934098229,0.153496766364474,0.186476102510212,0.18593824743331,0.170474553252928,0.0168133393763722 +lichen,lichen,north,,,Cladonia.chlorophaea,CCDeciduous2,5,0.192198726287736,0.158139093430204,0.206351929165891,0.20739871277065,0.184869780779868,0.0243239274357252 +lichen,lichen,north,,,Cladonia.chlorophaea,CCDeciduous3,5,0.192198726287736,0.158139093430204,0.206351929165891,0.20739871277065,0.184869780779868,0.0243239274357252 +lichen,lichen,north,,,Cladonia.chlorophaea,CCDeciduous4,5,0.192198726287736,0.158139093430204,0.206351929165891,0.20739871277065,0.184869780779868,0.0243239274357252 +lichen,lichen,north,,,Cladonia.chlorophaea,CCMixedwoodR,5,0.182662436599129,0.143290947030515,0.19168320742142,0.185594844139235,0.170905137694288,0.0249018018786431 +lichen,lichen,north,,,Cladonia.chlorophaea,CCMixedwood1,5,0.169020934098229,0.153496766364474,0.186476102510212,0.18593824743331,0.170474553252928,0.0168133393763722 +lichen,lichen,north,,,Cladonia.chlorophaea,CCMixedwood2,5,0.192198726287736,0.158139093430204,0.206351929165891,0.20739871277065,0.184869780779868,0.0243239274357252 +lichen,lichen,north,,,Cladonia.chlorophaea,CCMixedwood3,5,0.192198726287736,0.158139093430204,0.206351929165891,0.20739871277065,0.184869780779868,0.0243239274357252 +lichen,lichen,north,,,Cladonia.chlorophaea,CCMixedwood4,5,0.192198726287736,0.158139093430204,0.206351929165891,0.20739871277065,0.184869780779868,0.0243239274357252 +lichen,lichen,north,,,Cladonia.chlorophaea,EnSeismic,5,0.0677503172110941,0.0471429245544719,0.0732894752325876,0.0738978700051816,0.0617492589794445,0.0133508758867419 +lichen,lichen,north,,,Cladonia.chlorophaea,EnSoftLin,5,0.0560689010826879,0.0412420864698087,0.0607043262160254,0.0624980816232269,0.0520148802546486,0.00982245734083476 +lichen,lichen,north,,,Cladonia.chlorophaea,TrSoftLin,5,0.0605465982137621,0.0416835448522874,0.0681202259279406,0.0691216097124846,0.0563835551646629,0.0133846793873165 +lichen,lichen,north,,,Cladonia.chlorophaea,HardLin,5,0.0163800997555834,0.0142046646641618,0.0207897341200193,0.0172490121407761,0.0170729876025659,0.00365003569007476 +lichen,lichen,north,,,Cladonia.chlorophaea,UrbInd,5,0.026932901747997,0.0217317496788856,0.032534963821666,0.0293523293874362,0.0268883344802531,0.00545997483483957 +lichen,lichen,north,,,Cladonia.chlorophaea,Wellsites,5,0.0269864370031369,0.0206711523672041,0.0318822891198792,0.0309948356921504,0.0263704156910059,0.00555421916938355 +lichen,lichen,north,,,Cladonia.chlorophaea,Crop,5,0.00896331783563511,0.0086673108662899,0.0116267325606197,0.00981647353265912,0.00980133894395803,0.00175930642910351 +lichen,lichen,north,,,Cladonia.chlorophaea,TameP,5,0.0284182757721163,0.0236344354897084,0.0319552752867162,0.0276152084684389,0.0280727667342162,0.0044863473416081 +lichen,lichen,north,,,Cladonia.chlorophaea,RoughP,5,0.025017860881902,0.0188312694768778,0.026815348770033,0.0254030986816399,0.0234837747077663,0.00461769635994761 +lichen,lichen,north,,,Physcia.adscendens,WhiteSpruce,5,0.0674088713205342,0.0614470981390403,0.0709653557850045,0.0640273150290537,0.0664616657481686,0.00483973536953268 +lichen,lichen,north,,,Physcia.adscendens,Pine,5,0.0472827411302775,0.0444708053145572,0.0510027667554357,0.0463066694973039,0.0475721599857571,0.00352533170238091 +lichen,lichen,north,,,Physcia.adscendens,Deciduous,5,0.171848485649294,0.166777961008327,0.183050642925004,0.171848485649294,0.173855238772586,0.00914403110161088 +lichen,lichen,north,,,Physcia.adscendens,Mixedwood,5,0.103902979074589,0.0988920160049394,0.118672559907655,0.103902979074589,0.107234040275857,0.0114771871495709 +lichen,lichen,north,,,Physcia.adscendens,BlackSpruce,5,0.0116692571413722,0.0109018461718084,0.0148387626612113,0.0111517688016057,0.0125461466167073,0.00209093161445967 +lichen,lichen,north,,,Physcia.adscendens,TreedFen,5,0.0134516061910062,0.0123980063480553,0.0165551828482426,0.0127565152125194,0.0141541095221522,0.00229453976133663 +lichen,lichen,north,,,Physcia.adscendens,TreedSwamp,5,0.0684768936890089,0.0623100924562356,0.0815946560942004,0.0684768936890089,0.0705801981762149,0.0108957147150476 +lichen,lichen,north,,,Physcia.adscendens,GraminoidFen,5,0.0143523346004894,0.0138707572661006,0.0201350817078948,0.0143523346004894,0.0163626382844971,0.00335206751763776 +lichen,lichen,north,,,Physcia.adscendens,ShrubbyFen,5,0.0135292337016339,0.0132145942878114,0.0184646324742449,0.0134564657796432,0.0152732316995139,0.0028508926162948 +lichen,lichen,north,,,Physcia.adscendens,ShrubbyBog,5,0.0122579132788025,0.0117121025240539,0.0159189671771411,0.0119692218653384,0.0134175491402069,0.00225699190399091 +lichen,lichen,north,,,Physcia.adscendens,ShrubbySwamp,5,0.0642372109431116,0.0574653507523936,0.0782239746399321,0.0642372109431116,0.0669036310187336,0.0104713536929674 +lichen,lichen,north,,,Physcia.adscendens,Marsh,5,0.0421183317862372,0.0354669594231994,0.0538859562288842,0.0421183317862372,0.0442954195638172,0.00917399885026702 +lichen,lichen,north,,,Physcia.adscendens,Shrub,5,0.148981950749374,0.139298464646428,0.195805779656963,0.159263060245612,0.161673968525425,0.0337376300319071 +lichen,lichen,north,,,Physcia.adscendens,Grass,5,0.155922328843144,0.144176169953379,0.217726117790744,0.165761675145329,0.172790434462016,0.0454060373755947 +lichen,lichen,north,,,Physcia.adscendens,CCWhiteSpruceR,5,0.0524933497432348,0.0448033430759125,0.0658373565662838,0.0524933497432348,0.0544000314789238,0.0109830467809342 +lichen,lichen,north,,,Physcia.adscendens,CCWhiteSpruce1,5,0.0676142549226719,0.0595386229110912,0.0842373851191382,0.0676142549226719,0.0707090727129221,0.0133590871961955 +lichen,lichen,north,,,Physcia.adscendens,CCWhiteSpruce2,5,0.075962081883682,0.0661774643260701,0.0924254985770442,0.075962081883682,0.0784452753755624,0.013398142459152 +lichen,lichen,north,,,Physcia.adscendens,CCWhiteSpruce3,5,0.075962081883682,0.0661774643260701,0.0924254985770442,0.075962081883682,0.0784452753755624,0.013398142459152 +lichen,lichen,north,,,Physcia.adscendens,CCWhiteSpruce4,5,0.075962081883682,0.0661774643260701,0.0924254985770442,0.075962081883682,0.0784452753755624,0.013398142459152 +lichen,lichen,north,,,Physcia.adscendens,CCPineR,5,0.0530687182992354,0.0447898946528459,0.0673364978932711,0.0530687182992354,0.0551521174496051,0.0115755412404529 +lichen,lichen,north,,,Physcia.adscendens,CCPine1,5,0.054822170261719,0.0468044875054749,0.0681089706028397,0.054822170261719,0.0567280326295669,0.0111281917828034 +lichen,lichen,north,,,Physcia.adscendens,CCPine2,5,0.0575642554040823,0.0490603503195198,0.0707082507546723,0.0575642554040823,0.0592950171173939,0.0111107786936451 +lichen,lichen,north,,,Physcia.adscendens,CCPine3,5,0.0575642554040823,0.0490603503195198,0.0707082507546723,0.0575642554040823,0.0592950171173939,0.0111107786936451 +lichen,lichen,north,,,Physcia.adscendens,CCPine4,5,0.0575642554040823,0.0490603503195198,0.0707082507546723,0.0575642554040823,0.0592950171173939,0.0111107786936451 +lichen,lichen,north,,,Physcia.adscendens,CCDeciduousR,5,0.0827606661851487,0.0719443499282712,0.0987851185715972,0.0827606661851487,0.08424359648793,0.013891073963227 +lichen,lichen,north,,,Physcia.adscendens,CCDeciduous1,5,0.100618826509217,0.0877886957545074,0.125713872307938,0.100618826509217,0.105057955345838,0.0192301298426666 +lichen,lichen,north,,,Physcia.adscendens,CCDeciduous2,5,0.084212382915902,0.073575752550813,0.103101354854498,0.084212382915902,0.0874790260715549,0.01475504906051 +lichen,lichen,north,,,Physcia.adscendens,CCDeciduous3,5,0.084212382915902,0.073575752550813,0.103101354854498,0.084212382915902,0.0874790260715549,0.01475504906051 +lichen,lichen,north,,,Physcia.adscendens,CCDeciduous4,5,0.084212382915902,0.073575752550813,0.103101354854498,0.084212382915902,0.0874790260715549,0.01475504906051 +lichen,lichen,north,,,Physcia.adscendens,CCMixedwoodR,5,0.0827606661851487,0.0719443499282712,0.0987851185715972,0.0827606661851487,0.08424359648793,0.013891073963227 +lichen,lichen,north,,,Physcia.adscendens,CCMixedwood1,5,0.100618826509217,0.0877886957545074,0.125713872307938,0.100618826509217,0.105057955345838,0.0192301298426666 +lichen,lichen,north,,,Physcia.adscendens,CCMixedwood2,5,0.084212382915902,0.073575752550813,0.103101354854498,0.084212382915902,0.0874790260715549,0.01475504906051 +lichen,lichen,north,,,Physcia.adscendens,CCMixedwood3,5,0.084212382915902,0.073575752550813,0.103101354854498,0.084212382915902,0.0874790260715549,0.01475504906051 +lichen,lichen,north,,,Physcia.adscendens,CCMixedwood4,5,0.084212382915902,0.073575752550813,0.103101354854498,0.084212382915902,0.0874790260715549,0.01475504906051 +lichen,lichen,north,,,Physcia.adscendens,EnSeismic,5,0.0256366775276155,0.0220396461224062,0.0258667757531508,0.0258176949589414,0.0244432729669476,0.00230206038606853 +lichen,lichen,north,,,Physcia.adscendens,EnSoftLin,5,0.0273345543422601,0.0238543237522917,0.0285016462750445,0.0273345543422601,0.0265845186308551,0.00281690803091184 +lichen,lichen,north,,,Physcia.adscendens,TrSoftLin,5,0.0237240874831371,0.0199061804072359,0.0246224989022288,0.0242538337610053,0.022644867925346,0.00248798442401649 +lichen,lichen,north,,,Physcia.adscendens,HardLin,5,0.019409667521594,0.0160175308478019,0.0222563530629276,0.0164220788059039,0.0190357313695079,0.00329744230865101 +lichen,lichen,north,,,Physcia.adscendens,UrbInd,5,0.0269462495385606,0.0228419440845845,0.0343747203469902,0.0269462495385606,0.0279577105129281,0.00665341387285884 +lichen,lichen,north,,,Physcia.adscendens,Wellsites,5,0.0189042713169681,0.0175846129876694,0.0279662325594366,0.0189042713169681,0.0215065656131665,0.00650008155356583 +lichen,lichen,north,,,Physcia.adscendens,Crop,5,0.0061978772206181,0.00485501825215725,0.00819431107983461,0.0061978772206181,0.00646734484973649,0.00175972170076663 +lichen,lichen,north,,,Physcia.adscendens,TameP,5,0.0217469866099191,0.0205259688525677,0.0269346862339065,0.0217469866099191,0.0231110724644053,0.00372969845233129 +lichen,lichen,north,,,Physcia.adscendens,RoughP,5,0.039996203554655,0.0376640264085737,0.0479047806634932,0.039996203554655,0.0418609321868831,0.00636136742819892 +lichen,lichen,south,,,Cladonia.chlorophaea,Loamy,5,0.118687300046851,0.104702736998946,0.12654114918338,0.118687300046851,0.117003347991407,0.0123251487872755 +lichen,lichen,south,,,Cladonia.chlorophaea,SandyLoam,5,0.119258055085243,0.104379389676667,0.12444806981178,0.122725360583689,0.116076296806775,0.0115812144030893 +lichen,lichen,south,,,Cladonia.chlorophaea,ClaySub,5,0.115647388648516,0.100225827670201,0.120420274007026,0.11648008810727,0.112104149326241,0.0125379204899668 +lichen,lichen,south,,,Cladonia.chlorophaea,RapidDrain,5,0.134401942825707,0.130919679999458,0.142386868085905,0.134401942825707,0.135822231258499,0.00674887938113901 +lichen,lichen,south,,,Cladonia.chlorophaea,Blowout,5,0.151003729060033,0.142629562152347,0.160102906059062,0.143701037126792,0.151258127765863,0.00859040607681507 +lichen,lichen,south,,,Cladonia.chlorophaea,ThinBreak,5,0.137503969297793,0.129839880347417,0.148096403740304,0.137503969297793,0.138488768710106,0.00920385085107052 +lichen,lichen,south,,,Cladonia.chlorophaea,Other,5,0.011592343091593,0.0046173071411171,0.0231715222749202,0.011592343091593,0.0131772510513215,0.0108696999060124 +lichen,lichen,south,,,Cladonia.chlorophaea,EnSeismic,5,0.0962488205728889,0.0753549922631135,0.252780357762154,0.0746380066039094,0.142260934924217,0.1103764472823 +lichen,lichen,south,,,Cladonia.chlorophaea,EnSoftLin,5,0.100106376262895,0.0629831331203498,0.25663036318437,0.0593528864488744,0.140489678275968,0.112766217720885 +lichen,lichen,south,,,Cladonia.chlorophaea,TrSoftLin,5,0.104765725082673,0.0622678835132489,0.285864107347038,0.104765725082673,0.152752793742919,0.12644472051553 +lichen,lichen,south,,,Cladonia.chlorophaea,HardLin,5,0.0224141883326856,0.0219404643210603,0.0240466025090097,0.0224141883326856,0.0228389695045971,0.00110409171662926 +lichen,lichen,south,,,Cladonia.chlorophaea,UrbInd,5,0.0316560982778314,0.0261042267475289,0.0343242569398769,0.0287494220624193,0.0306598280995116,0.0042085259092217 +lichen,lichen,south,,,Cladonia.chlorophaea,Wellsites,5,0.0355928387293783,0.0297480296972117,0.0456614857314296,0.0355928387293783,0.0370490799107639,0.00881310026886407 +lichen,lichen,south,,,Cladonia.chlorophaea,Crop,5,0.0115918691627171,0.0109797156544521,0.0127817951554016,0.0108908943675099,0.0118111694039124,0.000895132212105925 +lichen,lichen,south,,,Cladonia.chlorophaea,TameP,5,0.0528035523685076,0.0519157543218056,0.0592890736068868,0.0527988783208814,0.0548657123745536,0.00411763620408946 +lichen,lichen,south,,,Cladonia.chlorophaea,RoughP,5,0.0704617752927988,0.0683113023707642,0.0754893494676391,0.0704617752927988,0.0715386169482165,0.00392836590938631 +lichen,lichen,south,,,Physcia.adscendens,Loamy,5,0.165058834401252,0.151196844376258,0.179127371014315,0.169700398944145,0.164609273022419,0.0142193641222609 +lichen,lichen,south,,,Physcia.adscendens,SandyLoam,5,0.178172116249331,0.161926494649089,0.189941302784499,0.181481384725004,0.176731233762671,0.0154464567930835 +lichen,lichen,south,,,Physcia.adscendens,ClaySub,5,0.155520095046812,0.142809679634744,0.167690632468226,0.162178734164762,0.154965775121554,0.0123850335584002 +lichen,lichen,south,,,Physcia.adscendens,RapidDrain,5,0.133157166351654,0.123712598889612,0.139250440689584,0.138113163461797,0.132146689844523,0.00806228414697056 +lichen,lichen,south,,,Physcia.adscendens,Blowout,5,0.0780059275319079,0.0605615716646166,0.0817059444700948,0.0791009289771177,0.0731408926112592,0.0121622767884476 +lichen,lichen,south,,,Physcia.adscendens,ThinBreak,5,0.103023942244417,0.0819954494155731,0.103983652295498,0.103023942244417,0.0955490857463932,0.0122219699134275 +lichen,lichen,south,,,Physcia.adscendens,Other,5,0.108836563387276,0.0821453221454944,0.133452128995239,0.127814796703126,0.109107706996853,0.0269474171371931 +lichen,lichen,south,,,Physcia.adscendens,EnSeismic,5,0.140820975204423,0.129482190661842,0.19424153380181,0.134144710345317,0.156786420793645,0.0335710178960414 +lichen,lichen,south,,,Physcia.adscendens,EnSoftLin,5,0.106052188933668,0.0814400391043974,0.130572037732656,0.0868856909303751,0.104827215208426,0.0262751436146954 +lichen,lichen,south,,,Physcia.adscendens,TrSoftLin,5,0.157835534383738,0.120487024187209,0.209774497540774,0.157835534383738,0.163288539797896,0.0441400473178623 +lichen,lichen,south,,,Physcia.adscendens,HardLin,5,0.0165350535429243,0.0120621070012098,0.0195286140965431,0.0165350535429243,0.0160352349984333,0.00373898650121779 +lichen,lichen,south,,,Physcia.adscendens,UrbInd,5,0.0389262598086267,0.0294939748318359,0.0530990959477553,0.0431506202004028,0.0404214534693983,0.0124003014049701 +lichen,lichen,south,,,Physcia.adscendens,Wellsites,5,0.0361212212999052,0.0282999511156899,0.0508786677112775,0.0413073764753538,0.0385851768216347,0.0119892304602698 +lichen,lichen,south,,,Physcia.adscendens,Crop,5,0.00923975055880215,0.00688935614723793,0.00990739362805418,0.00923975055880215,0.00866135825638311,0.00174961488852405 +lichen,lichen,south,,,Physcia.adscendens,TameP,5,0.026713869603247,0.0180795051503958,0.0297402158652557,0.026713869603247,0.0245784040412905,0.00588581925861034 +lichen,lichen,south,,,Physcia.adscendens,RoughP,5,0.039320400519205,0.0281759149172245,0.0418780712232037,0.039320400519205,0.0359534518990116,0.00708818168961686 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Decid,5,0.047640807803491,0.0421034563656317,0.0521300438758573,0.0487558878166635,0.0473450096826365,0.0056501316974934 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Mixedwood,5,0.0279054540414128,0.0181658689236166,0.0367559849225421,0.0339606541972421,0.0278096939902147,0.00951815620753908 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Pine,5,0.0268788058190438,0.0209609632124494,0.0305097583077334,0.0258942184497686,0.0262889918843959,0.00524934568929363 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Spruce,5,0.0176721263875699,0.0146608207690292,0.0193257868464236,0.0155269078679997,0.0171502536690568,0.00231397502995575 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,TreedBog,5,0.0312070092273878,0.0296707284210888,0.0333575322543523,0.0299192022319042,0.0313555861600363,0.00196530518299601 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,TreedFen,5,0.031652259109271,0.0295795976387661,0.0337216938968473,0.0299192022319042,0.0315963400302195,0.00207244069441546 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,TreedSwamp,5,0.0344943810177399,0.0306124249828509,0.0489375718055997,0.0299192022319042,0.0386608055204566,0.00930458029009551 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,GrassHerb,5,0.0653279741583057,0.061888271218626,0.089856562855897,0.0701423452426464,0.0725277655971948,0.0173644289990184 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Shrub,5,0.0611030881192147,0.0442445083456052,0.0682165968089101,0.0701423452426464,0.0581276698433782,0.0144904584849003 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,GraminoidFen,5,0.0329008222005278,0.0311059034360441,0.0791720523993423,0.0320501190239091,0.0501938242980929,0.0256718232770762 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Marsh,5,0.0329008222005278,0.0311059034360441,0.0791720523993423,0.0320501190239091,0.0501938242980929,0.0256718232770762 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,ShrubbyBog,5,0.0304764263774674,0.0254605104823409,0.0325605409298803,0.0320501190239091,0.0294402995205514,0.00368131508014908 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,ShrubbyFen,5,0.0304764263774674,0.0254605104823409,0.0325605409298803,0.0320501190239091,0.0294402995205514,0.00368131508014908 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,ShrubbySwamp,5,0.0304764263774674,0.0207171959827077,0.0325605409298803,0.0320501190239091,0.027783252167868,0.00665855467631068 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCDecidR,5,0.0519264751930211,0.0421909761363827,0.0610656929689814,0.0664247062431653,0.0513344183248704,0.00990292477458262 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCDecid1,5,0.0664247062431653,0.0454313130756783,0.161801935827499,0.0664247062431653,0.0956407900305149,0.0594616376704109 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCDecid2,5,0.0403674064209935,0.0226811921693289,0.0610656929689814,0.0664247062431653,0.0413647407355767,0.0189529178903395 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCDecid3,5,0.0664247062431653,0.0454313130756783,0.13984577572166,0.0664247062431653,0.0874581435990877,0.047869158265578 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCDecid4,5,0.0664247062431653,0.0454313130756783,0.13984577572166,0.0664247062431653,0.0874581435990877,0.047869158265578 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCMixedwoodR,5,0.0403674064209935,1.86696880287542e-07,0.0610656929689814,0.0664247062431653,0.0319639391176817,0.0305989508139031 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCMixedwood1,5,0.0664247062431653,0.0454313130756783,0.0985082759740731,0.0664247062431653,0.0708925437744992,0.0261644585445364 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCMixedwood2,5,0.0664247062431653,0.0454313130756783,0.0927460852883475,0.0664247062431653,0.0679067027845017,0.0243051352751106 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCMixedwood3,5,0.0664247062431653,0.0454313130756783,0.13984577572166,0.0664247062431653,0.0874581435990877,0.047869158265578 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCMixedwood4,5,0.0664247062431653,0.0454313130756783,0.13984577572166,0.0664247062431653,0.0874581435990877,0.047869158265578 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCPineR,5,0.0329977774300808,0.0275981212995825,0.055494150911045,0.0418434478038768,0.0389994170914637,0.0154321958967688 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCPine1,5,0.0286931152246994,0.00895589257187608,0.0383051796543584,0.0418434478038768,0.0246424860352798,0.0146575340381232 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCPine2,5,0.0418434478038768,0.030414980106852,0.13984577572166,0.0418434478038768,0.0762011545464462,0.0574167919857827 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCPine3,5,0.0418434478038768,0.030414980106852,0.13984577572166,0.0418434478038768,0.0762011545464462,0.0574167919857827 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCPine4,5,0.0418434478038768,0.030414980106852,0.13984577572166,0.0418434478038768,0.0762011545464462,0.0574167919857827 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCSpruceR,5,0.0418434478038768,0.030414980106852,0.305015280094132,0.0418434478038768,0.134680455516499,0.154520697103704 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCSpruce1,5,0.0329977774300808,0.0114772753535638,0.0426296284343427,0.0418434478038768,0.0293376276172234,0.0174770304250845 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCSpruce2,5,0.0329977774300808,0.0196456208529556,0.0493468226750582,0.0418434478038768,0.0342994741972592,0.0151650436180593 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCSpruce3,5,0.0418434478038768,0.030414980106852,0.13984577572166,0.0418434478038768,0.0762011545464462,0.0574167919857827 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,CCSpruce4,5,0.0418434478038768,0.030414980106852,0.13984577572166,0.0418434478038768,0.0762011545464462,0.0574167919857827 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,HardLin,5,0.134939746303501,0.11313165192637,0.138252211851646,0.140140987080189,0.128430781048037,0.0150773054438261 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,EnSoftLin,5,0.135928985585703,0.114033932348076,0.146463088964367,0.108633567141907,0.131593724319033,0.0163096665223475 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,EnSeismic,5,0.12213448015733,0.0890416535123073,0.143994279540726,0.108633567141907,0.11836959804192,0.0281935358437126 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,TrSoftLin,5,0.108633567141907,0.0713041482970402,0.138717152503764,0.108633567141907,0.105920961385117,0.0342224601999073 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Urban,5,0.134939746303501,0.11313165192637,0.138252211851646,0.140140987080189,0.128430781048037,0.0150773054438261 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Rural,5,0.134939746303501,0.11313165192637,0.138252211851646,0.140140987080189,0.128430781048037,0.0150773054438261 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Industrial,5,0.134939746303501,0.11313165192637,0.138252211851646,0.140140987080189,0.128430781048037,0.0150773054438261 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Well,5,0.129304108911142,0.0508739875681123,0.138060490769514,0.140140987080189,0.101626632226259,0.0457159602942333 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,Crop,5,0.134939746303501,0.130856632631654,0.142628003268766,0.140140987080189,0.136371254913681,0.0058948625509627 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,TameP,5,0.140140987080189,0.131558363868086,0.166944782085177,0.140140987080189,0.146328175511539,0.0200400340628098 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,RoughP,5,0.134939746303501,0.107211805137829,0.197177402914023,0.140140987080189,0.146414025010739,0.0530459646613359 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Decid,5,0.257670301669833,0.25268511782041,0.299216536351889,0.250263196314734,0.272180704204058,0.0242350996923886 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Mixedwood,5,0.275371757751266,0.246931841338979,0.300679741105016,0.267778296796379,0.274736706733593,0.0279617066651882 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Pine,5,0.191627552662267,0.185639377125035,0.22504720435644,0.191627552662267,0.201501274262095,0.0214876480685103 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Spruce,5,0.227199239938309,0.197719069725711,0.261529215102722,0.211937623492895,0.229168587007038,0.0318091845104133 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,TreedBog,5,0.148793830904151,0.132497404514779,0.155371350064858,0.13349759896445,0.144807122602355,0.0114762101186987 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,TreedFen,5,0.144475982710738,0.132978225672994,0.154438886014026,0.140460621172556,0.144313480947399,0.0109807508607257 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,TreedSwamp,5,0.249048354321576,0.223475542712116,0.289138875493625,0.219333664647888,0.254605490569608,0.0323842455145578 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,GrassHerb,5,0.2480296169969,0.237318154647019,0.315522864674229,0.243208536186666,0.268568532907938,0.0437249717988789 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Shrub,5,0.23362253489809,0.21634137243735,0.257032219262653,0.23362253489809,0.236470744324128,0.0201811658199 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,GraminoidFen,5,0.272335942420037,0.244138225439184,0.296606395166859,0.272335942420037,0.270211095150555,0.026687514195625 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Marsh,5,0.272335942420037,0.244138225439184,0.296606395166859,0.272335942420037,0.270211095150555,0.026687514195625 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,ShrubbyBog,5,0.129805148819312,0.116205825982704,0.158770458351882,0.123758665516073,0.134605575728791,0.0251125137099972 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,ShrubbyFen,5,0.129805148819312,0.116205825982704,0.158770458351882,0.123758665516073,0.134605575728791,0.0251125137099972 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,ShrubbySwamp,5,0.188745324781336,0.167005561530848,0.253626121411953,0.188745324781336,0.202711431947935,0.0494613150959488 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCDecidR,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCDecid1,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCDecid2,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCDecid3,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCDecid4,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCMixedwoodR,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCMixedwood1,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCMixedwood2,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCMixedwood3,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCMixedwood4,5,0.231790845755958,0.199250708118932,0.239359596218743,0.204735734181689,0.222063537492967,0.0204896232573866 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCPineR,5,0.334283211887097,0.223646034396249,0.350195977658891,0.252728055432474,0.297760629606335,0.0659734111545076 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCPine1,5,0.334283211887097,0.223646034396249,0.350195977658891,0.252728055432474,0.297760629606335,0.0659734111545076 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCPine2,5,0.334283211887097,0.223646034396249,0.350195977658891,0.252728055432474,0.297760629606335,0.0659734111545076 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCPine3,5,0.334283211887097,0.223646034396249,0.350195977658891,0.252728055432474,0.297760629606335,0.0659734111545076 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCPine4,5,0.334283211887097,0.223646034396249,0.350195977658891,0.252728055432474,0.297760629606335,0.0659734111545076 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCSpruceR,5,0.321916199827704,0.226786073940564,0.3362304790634,0.251426565070253,0.290385479825988,0.0569310858978242 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCSpruce1,5,0.321916199827704,0.226786073940564,0.3362304790634,0.251426565070253,0.290385479825988,0.0569310858978242 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCSpruce2,5,0.321916199827704,0.226786073940564,0.3362304790634,0.251426565070253,0.290385479825988,0.0569310858978242 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCSpruce3,5,0.321916199827704,0.226786073940564,0.3362304790634,0.251426565070253,0.290385479825988,0.0569310858978242 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,CCSpruce4,5,0.321916199827704,0.226786073940564,0.3362304790634,0.251426565070253,0.290385479825988,0.0569310858978242 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,HardLin,5,0.19071774622534,0.0974007399519308,0.240733568496156,0.111539574120126,0.17127771039873,0.0732372269463743 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,EnSoftLin,5,0.248284317219045,0.226169685664436,0.301392052670499,0.245765939186683,0.25939698564774,0.0420508019366517 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,EnSeismic,5,0.426961347747829,0.389435322188954,0.481650459050364,0.40014515763882,0.432415278327126,0.047044930339176 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,TrSoftLin,5,0.17814172514827,0.12158426489911,0.222472294628215,0.191095088735685,0.172686214591363,0.0514005542419291 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Urban,5,0.19071774622534,0.0974007399519308,0.240733568496156,0.111539574120126,0.17127771039873,0.0732372269463743 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Rural,5,0.19071774622534,0.0974007399519308,0.240733568496156,0.111539574120126,0.17127771039873,0.0732372269463743 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Industrial,5,0.19071774622534,0.0974007399519308,0.240733568496156,0.111539574120126,0.17127771039873,0.0732372269463743 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Well,5,0.204111777253966,0.157588378902068,0.3180994853061,0.135502134019886,0.226718306143626,0.0934254943677214 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,Crop,5,0.177273110573243,0.159339604167137,0.20698726239642,0.177273110573243,0.181419649055733,0.0243804143233458 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,TameP,5,0.106013821908483,0.0894430978024326,0.17346343225447,0.0952839991630323,0.12502933843141,0.0436710630773195 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,RoughP,5,0.274174686267773,0.223732051403491,0.374273016748219,0.274174686267773,0.292241558777143,0.0847262045748597 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,ClaySub,5,0.555220139484688,0.394498331723014,0.702736975528346,0.555220139484688,0.548583153087831,0.151762015935046 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Other,5,0.555220139484688,0.394498331723014,0.702736975528346,0.555220139484688,0.548583153087831,0.151762015935046 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,RapidDrain,5,0.634195131080269,0.466725133776188,0.702736975528346,0.555220139484688,0.598286530503394,0.123050914931351 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Loamy,5,0.585624995059312,0.517090683974723,0.733450970235701,0.585624995059312,0.612539176846356,0.112558481100448 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,SandyLoam,5,0.779017339675273,0.680988596683654,0.866425950439531,0.779017339675273,0.77884451338414,0.0956809767276209 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,ThinBreak,5,0.555220139484688,0.448231043418837,0.702736975528346,0.555220139484688,0.573244487005602,0.126605629946665 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Blowout,5,0.555220139484688,0.448231043418837,0.702736975528346,0.555220139484688,0.573244487005602,0.126605629946665 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Urban,5,0.583445114492641,0.499588697752921,0.825386051737734,0.487723056831294,0.639664312948093,0.170888464211465 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Rural,5,0.583445114492641,0.499588697752921,0.825386051737734,0.487723056831294,0.639664312948093,0.170888464211465 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Industrial,5,0.583445114492641,0.499588697752921,0.825386051737734,0.487723056831294,0.639664312948093,0.170888464211465 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Well,5,0.517387159135361,0.485351712822258,0.654799112289952,0.487723056831294,0.554938918286071,0.0915711112104892 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Mine,5,0.583445114492641,0.499588697752921,0.825386051737734,0.487723056831294,0.639664312948093,0.170888464211465 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,EnSoftLin,5,0.551988691379397,0.472907672172372,0.746855800537036,0.551988691379397,0.596628214316857,0.137830596967816 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,EnSeismic,5,0.551988691379397,0.472907672172372,0.746855800537036,0.551988691379397,0.596628214316857,0.137830596967816 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,HardLin,5,0.583445114492641,0.499588697752921,0.825386051737734,0.487723056831294,0.639664312948093,0.170888464211465 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,TrSoftLin,5,0.551988691379397,0.472907672172372,0.661295206321867,0.551988691379397,0.564149522780887,0.093422616108171 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,Crop,5,0.664605920258043,0.543810341278273,0.777406068435428,0.664605920258043,0.66361926030854,0.115409799954802 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,RoughP,5,0.661922813941234,0.543810341278273,0.734870844221513,0.664605920258043,0.641815026938221,0.0992280794953468 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,TameP,5,0.664605920258043,0.543810341278273,0.762991086100113,0.664605920258043,0.656411769140883,0.107913421931019 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,WetlandMargin,5,0.585444465824865,0.239308585750385,0.749205253519565,0.585444465824865,0.514520714062245,0.25399488681222 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,ClaySub,5,0.0667209228725318,0.0471871616217136,0.0745121978337484,0.0784267337005647,0.0619677799657342,0.0138015043715517 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Other,5,0.0667209228725318,0.0471871616217136,0.0745121978337484,0.0784267337005647,0.0619677799657342,0.0138015043715517 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,RapidDrain,5,0.0784267337005647,0.0580184128835627,0.182475942831045,0.0784267337005647,0.108634134915957,0.0683649493619883 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Loamy,5,0.0565859324870483,0.0491432799330989,0.0744882681216602,0.0784267337005647,0.0605808564697218,0.0127502513381831 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,SandyLoam,5,0.0522167395575833,0.022088416299373,0.106394166195476,0.0784267337005647,0.0614464088165364,0.0437090748528919 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,ThinBreak,5,0.0667209228725318,0.0526910929547497,0.0819102188921822,0.0784267337005647,0.0669999123068879,0.0144139003416867 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Blowout,5,0.0667209228725318,0.0526910929547497,0.0819102188921822,0.0784267337005647,0.0669999123068879,0.0144139003416867 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Urban,5,0.00922653651061864,0.0027634925310201,0.030904020696116,0.00922653651061864,0.0144698682404388,0.0153586078670068 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Rural,5,0.00922653651061864,0.0027634925310201,0.030904020696116,0.00922653651061864,0.0144698682404388,0.0153586078670068 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Industrial,5,0.00922653651061864,0.0027634925310201,0.030904020696116,0.00922653651061864,0.0144698682404388,0.0153586078670068 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Well,5,0.00242981545308046,4.85369257366382e-09,0.00684152516554301,0.00922653651061864,0.0029840741768291,0.00378031859912425 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Mine,5,0.00922653651061864,0.0027634925310201,0.030904020696116,0.00922653651061864,0.0144698682404388,0.0153586078670068 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,EnSoftLin,5,0.036098691627639,0.0108694320397718,0.0881199281160513,0.036098691627639,0.0452302386297233,0.0427459018693586 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,EnSeismic,5,0.036098691627639,0.0108694320397718,0.0881199281160513,0.036098691627639,0.0452302386297233,0.0427459018693586 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,HardLin,5,0.00922653651061864,0.0027634925310201,0.030904020696116,0.00922653651061864,0.0144698682404388,0.0153586078670068 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,TrSoftLin,5,0.0271735638239476,0.0226676570357414,0.0432617606269774,0.036098691627639,0.0315033862556764,0.0107685423453701 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,Crop,5,0.0206055204843312,0.0177111458771107,0.0251768860663094,0.0270020288489522,0.0211820007883143,0.00385182168133274 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,RoughP,5,0.0225176634330294,0.0183397065659335,0.0271753939444167,0.0270020288489522,0.0228490694789415,0.00442606378043097 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,TameP,5,0.0168291639536684,0.00689293851916878,0.0244434255031038,0.0270020288489522,0.0157357123464475,0.00870155989086529 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,WetlandMargin,5,1.94731766462466e-08,9.60713813825307e-09,2.26961479274612e-08,2.33837497296015e-08,1.70089835588826e-08,6.72415947275868e-09 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Decid,5,0.0449203614522765,0.0399272055174853,0.0461207026143072,0.0449203614522765,0.0433845385662547,0.00315950589820464 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Mixedwood,5,0.0273241474240622,0.0148672640830864,0.029705093739567,0.0273241474240622,0.0234559291752824,0.00765746375489848 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Pine,5,0.0196006162097023,0.0177223221151389,0.0208573336024009,0.0176556438960936,0.0193416828929529,0.00155313864498343 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Spruce,5,0.0133118475600801,0.00928641896189441,0.0154669185911262,0.0137961282417973,0.0125673305270286,0.00317121265527406 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,TreedBog,5,0.0269986743089763,0.0246332515524093,0.0296528479842816,0.0286355651253948,0.0271316160284842,0.00251156778735683 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,TreedFen,5,0.0346703571209486,0.0300665281749436,0.0351451063464642,0.0316720599445966,0.0330965907211805,0.00271068571353499 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,TreedSwamp,5,0.0562955719457925,0.0435834392821119,0.07751039884971,0.0562955719457925,0.0596535487656531,0.0168206952691527 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,GrassHerb,5,0.0804381287263335,0.077265490603449,0.0897442881448977,0.0783755759784833,0.0824259622430937,0.00737264781225501 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Shrub,5,0.0681831360493072,0.0495067618820586,0.0834804792057293,0.0693588214886089,0.0659469604326635,0.0178542302535926 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,GraminoidFen,5,0.08298929154192,0.0746709031965066,0.0861134405713966,0.0834692933781883,0.0811320484264769,0.00639156303460706 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Marsh,5,0.08298929154192,0.0746709031965066,0.0861134405713966,0.0834692933781883,0.0811320484264769,0.00639156303460706 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,ShrubbyBog,5,0.0137861868727856,0.0133312667653021,0.0163982311535578,0.0174941754511015,0.0145534097592103,0.00174334914133735 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,ShrubbyFen,5,0.0137861868727856,0.0133312667653021,0.0163982311535578,0.0174941754511015,0.0145534097592103,0.00174334914133735 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,ShrubbySwamp,5,0.023346797778904,0.015694365834426,0.0247290058987953,0.0195830589922788,0.0210911900079341,0.00493525892739733 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCDecidR,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCDecid1,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCDecid2,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCDecid3,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCDecid4,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCMixedwoodR,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCMixedwood1,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCMixedwood2,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCMixedwood3,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCMixedwood4,5,0.0634232741924905,0.0524183155350296,0.0816623550282871,0.0634232741924905,0.0661907389690236,0.0145258686389515 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCPineR,5,0.0429017164110987,0.0334830377890367,0.05791632388432,0.0429017164110987,0.0456388904072951,0.0126462818641536 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCPine1,5,0.0429017164110987,0.0334830377890367,0.05791632388432,0.0429017164110987,0.0456388904072951,0.0126462818641536 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCPine2,5,0.0429017164110987,0.0334830377890367,0.05791632388432,0.0429017164110987,0.0456388904072951,0.0126462818641536 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCPine3,5,0.0429017164110987,0.0334830377890367,0.05791632388432,0.0429017164110987,0.0456388904072951,0.0126462818641536 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCPine4,5,0.0429017164110987,0.0334830377890367,0.05791632388432,0.0429017164110987,0.0456388904072951,0.0126462818641536 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCSpruceR,5,0.024139993434008,0.0150053643826653,0.0287361913785556,0.0309648973086927,0.0221568969593286,0.00691592629008353 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCSpruce1,5,0.024139993434008,0.0150053643826653,0.0287361913785556,0.0309648973086927,0.0221568969593286,0.00691592629008353 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCSpruce2,5,0.024139993434008,0.0150053643826653,0.0287361913785556,0.0309648973086927,0.0221568969593286,0.00691592629008353 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCSpruce3,5,0.024139993434008,0.0150053643826653,0.0287361913785556,0.0309648973086927,0.0221568969593286,0.00691592629008353 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,CCSpruce4,5,0.024139993434008,0.0150053643826653,0.0287361913785556,0.0309648973086927,0.0221568969593286,0.00691592629008353 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,HardLin,5,0.121863990133419,0.119864788586073,0.129947053494084,0.118992631310027,0.124132350649919,0.00533782210285643 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,EnSoftLin,5,0.108883674200791,0.0983056772774538,0.132874774675726,0.0980871465772408,0.113902365396764,0.0173390198715597 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,EnSeismic,5,0.0980871465772408,0.0436913018520788,0.104783593851584,0.0980871465772408,0.0799588266598827,0.0326087744287224 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,TrSoftLin,5,0.0980871465772408,0.0718039852980543,0.104783593851584,0.0980871465772408,0.0908443583884561,0.0180214456135981 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Urban,5,0.121863990133419,0.119864788586073,0.129947053494084,0.118992631310027,0.124132350649919,0.00533782210285643 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Rural,5,0.121863990133419,0.119864788586073,0.129947053494084,0.118992631310027,0.124132350649919,0.00533782210285643 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Industrial,5,0.121863990133419,0.119864788586073,0.129947053494084,0.118992631310027,0.124132350649919,0.00533782210285643 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Well,5,0.121863990133419,0.119864788586073,0.129947053494084,0.118992631310027,0.124132350649919,0.00533782210285643 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,Crop,5,0.121863990133419,0.119864788586073,0.129947053494084,0.118992631310027,0.124132350649919,0.00533782210285643 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,TameP,5,0.121863990133419,0.119864788586073,0.129947053494084,0.118992631310027,0.124132350649919,0.00533782210285643 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,RoughP,5,0.121863990133419,0.119864788586073,0.129947053494084,0.118992631310027,0.124132350649919,0.00533782210285643 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Decid,5,0.111664402452981,0.0925676785255877,0.12838442931739,0.111664402452981,0.110119895193492,0.0190661663439829 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Mixedwood,5,0.0666129636526026,0.0517060528658316,0.0780092522284266,0.0655577834788831,0.0661193316569303,0.0145003800102491 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Pine,5,0.0387812701455966,0.0349164779226351,0.0431552649817136,0.0434917161194809,0.0391574742770485,0.00424769857635412 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Spruce,5,0.0695392350566296,0.0500711157624263,0.0819420036005477,0.0695392350566296,0.0666263862635545,0.0157200412672632 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,TreedBog,5,0.026015783845678,0.0246467881759166,0.0391195468077491,0.0324223167296191,0.0303153574549933,0.00808061452875623 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,TreedFen,5,0.0324223167296191,0.0231448273336468,0.0416424211240961,0.0324223167296191,0.0323574475758651,0.00908161962000169 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,TreedSwamp,5,0.12705806311569,0.0777590339638333,0.135275094643934,0.138924945537604,0.111707211905002,0.0308139140813987 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,GrassHerb,5,0.0538649801932773,0.0494114363460887,0.0771016483764267,0.0674988002326724,0.0610238337471224,0.0146762162898288 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Shrub,5,0.0845204524428601,0.0813230199104712,0.110177551407743,0.0845204524428601,0.0919694455707154,0.0196217660132652 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,GraminoidFen,5,0.0664848446869057,0.0548562317765635,0.0823057352013198,0.0715579377130805,0.0681187967555788,0.0148350802392212 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Marsh,5,0.0664848446869057,0.0548562317765635,0.0823057352013198,0.0715579377130805,0.0681187967555788,0.0148350802392212 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,ShrubbyBog,5,0.0664848446869057,0.0400219913426505,0.0823057352013198,0.0715579377130805,0.0618909308970976,0.0211868093289378 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,ShrubbyFen,5,0.0664848446869057,0.0400219913426505,0.0823057352013198,0.0715579377130805,0.0618909308970976,0.0211868093289378 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,ShrubbySwamp,5,0.0894709335268126,0.0685140818973756,0.11559222101601,0.0715579377130805,0.091532280448262,0.0231972179734856 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCDecidR,5,0.0979659072936202,0.056000550317278,0.119035445956366,0.0987996864776002,0.0899407938396021,0.0339449857842051 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCDecid1,5,0.0979659072936202,0.0754740008765856,0.119035445956366,0.0987996864776002,0.0964319440260379,0.0232198918994744 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCDecid2,5,0.0987996864776002,0.0879270736378564,0.143484471733302,0.0987996864776002,0.112263243057184,0.0284752655903563 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCDecid3,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCDecid4,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCMixedwoodR,5,0.0979659072936202,0.0324939517016264,0.119035445956366,0.0987996864776002,0.0821052609677182,0.0495340856552004 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCMixedwood1,5,0.0987996864776002,0.0879270736378564,0.190271057342052,0.0987996864776002,0.127858771593434,0.0594059091938825 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCMixedwood2,5,0.0979659072936202,0.0408003100761594,0.119035445956366,0.0987996864776002,0.0848740470925625,0.043859188592961 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCMixedwood3,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCMixedwood4,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCPineR,5,0.0979659072936202,0.06146626030886,0.119035445956366,0.0987996864776002,0.0917626971701294,0.0306210063404481 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCPine1,5,0.0979659072936202,0.0742813046037392,0.119035445956366,0.0987996864776002,0.0960343786017558,0.0237613346782708 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCPine2,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCPine3,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCPine4,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCSpruceR,5,0.0987996864776002,0.0879270736378564,0.133811680469912,0.0987996864776002,0.109038979302721,0.0235003797651685 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCSpruce1,5,0.0979659072936202,0.0842318615581226,0.119035445956366,0.0987996864776002,0.0998507882016794,0.0196380282283496 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCSpruce2,5,0.0979659072936202,0.0553515374833462,0.119035445956366,0.0987996864776002,0.0897244562282914,0.0343504130227519 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCSpruce3,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,CCSpruce4,5,0.0979659072936202,0.0621286619393974,0.119035445956366,0.0987996864776002,0.0919834977136418,0.0302306364918027 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,HardLin,5,0.0655704593026337,0.0386964513077742,0.0806664925562941,0.0681539858129174,0.061490055174022,0.0235635493086013 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,EnSoftLin,5,0.0499491621686334,0.0432606306115374,0.0917851939975131,0.0630037477010183,0.0622091308002257,0.0284117016778473 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,EnSeismic,5,0.0630037477010183,0.0470136277181822,0.0956609574711948,0.0630037477010183,0.0683350991735864,0.0261979131095086 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,TrSoftLin,5,0.0499491621686334,0.0381918030077569,0.0917851939975131,0.0630037477010183,0.0605195215989655,0.03010971736645 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Urban,5,0.0655704593026337,0.0386964513077742,0.0806664925562941,0.0681539858129174,0.061490055174022,0.0235635493086013 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Rural,5,0.0655704593026337,0.0386964513077742,0.0806664925562941,0.0681539858129174,0.061490055174022,0.0235635493086013 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Industrial,5,0.0655704593026337,0.0386964513077742,0.0806664925562941,0.0681539858129174,0.061490055174022,0.0235635493086013 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Well,5,0.0656654516139695,0.0626306304625554,0.0806664925562941,0.0681539858129174,0.0698137610034471,0.0110678450805123 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,Crop,5,0.0655704593026337,0.0539031526201264,0.0806664925562941,0.0681539858129174,0.0665589556114727,0.0144644680111463 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,TameP,5,0.0655704593026337,0.0400397821358317,0.0806664925562941,0.0681539858129174,0.0619378321167078,0.0226787995328322 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,RoughP,5,0.0681539858129174,0.0626306304625554,0.157698264126943,0.0681539858129174,0.097379003560495,0.0602979045619669 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,ClaySub,5,0.691634949610229,0.550848790356094,0.802465195536948,0.714170375192759,0.68019609479513,0.13479154309237 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Other,5,0.691634949610229,0.550848790356094,0.802465195536948,0.714170375192759,0.68019609479513,0.13479154309237 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,RapidDrain,5,0.710928475172056,0.569824590326749,0.802465195536948,0.714170375192759,0.693542699892822,0.133909499622018 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Loamy,5,0.614208250714799,0.535537936114048,0.793086839692097,0.714170375192759,0.65392920928007,0.134177594939503 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,SandyLoam,5,0.710928475172056,0.569824590326749,0.851537864173555,0.714170375192759,0.709900256105024,0.161650897739812 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,ThinBreak,5,0.714170375192759,0.493395564734683,0.821258910605355,0.714170375192759,0.665373424635133,0.164378669365967 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Blowout,5,0.714170375192759,0.493395564734683,0.821258910605355,0.714170375192759,0.665373424635133,0.164378669365967 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Urban,5,0.757773549137013,0.657116840415395,0.839728338581547,0.757773549137013,0.745609599201415,0.0963118964374143 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Rural,5,0.757773549137013,0.657116840415395,0.839728338581547,0.757773549137013,0.745609599201415,0.0963118964374143 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Industrial,5,0.757773549137013,0.657116840415395,0.839728338581547,0.757773549137013,0.745609599201415,0.0963118964374143 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Well,5,0.665294102611545,0.500844990155389,0.838316421030783,0.757773549137013,0.673408549679142,0.180300241875787 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Mine,5,0.757773549137013,0.657116840415395,0.839728338581547,0.757773549137013,0.745609599201415,0.0963118964374143 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,EnSoftLin,5,0.652008305064925,0.550739776228706,0.78332637135667,0.628634628150965,0.663407830088547,0.123697662896884 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,EnSeismic,5,0.652008305064925,0.550739776228706,0.78332637135667,0.628634628150965,0.663407830088547,0.123697662896884 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,HardLin,5,0.757773549137013,0.657116840415395,0.839728338581547,0.757773549137013,0.745609599201415,0.0963118964374143 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,TrSoftLin,5,0.628634628150965,0.425072256873216,0.765678825360405,0.628634628150965,0.599365165042208,0.174842335297327 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,Crop,5,0.753168326869958,0.644219978099744,0.838316421030783,0.757773549137013,0.738775057178943,0.101908824049686 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,RoughP,5,0.753168326869958,0.5812621652318,0.838316421030783,0.757773549137013,0.717789119556295,0.133449191398277 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,TameP,5,0.753168326869958,0.660198982643474,0.838316421030783,0.757773549137013,0.745523667086661,0.0945203457137365 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,WetlandMargin,5,0.870880735779777,0.824509912771411,0.934646436936643,0.870880735779777,0.878612329229634,0.0544004215859729 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,ClaySub,5,0.0108355773911345,0.00400015925223842,0.0145163537839974,0.0141945654676911,0.00964996452335385,0.00522727711737457 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Other,5,0.0108355773911345,0.00400015925223842,0.0145163537839974,0.0141945654676911,0.00964996452335385,0.00522727711737457 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,RapidDrain,5,0.0141945654676911,0.00412645388529901,0.0149565220342368,0.0141945654676911,0.0106852617341246,0.0058358125599737 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Loamy,5,0.0141945654676911,0.00459311158353375,0.0160836483068433,0.0141945654676911,0.0112135235259748,0.00595036378174178 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,SandyLoam,5,0.0141945654676911,0.0101099232735522,0.0149565220342368,0.0141945654676911,0.0131064936286486,0.00313916311192032 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,ThinBreak,5,0.0147308793282016,0.0124614000782512,0.0267202462630917,0.0141945654676911,0.0185713582518418,0.00732893490342272 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Blowout,5,0.0147308793282016,0.0124614000782512,0.0267202462630917,0.0141945654676911,0.0185713582518418,0.00732893490342272 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Urban,5,9.37290365828135e-10,5.80885222938139e-10,1.02858504274146e-09,9.67698723675452e-10,8.39452871070888e-10,2.39628364395568e-10 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Rural,5,9.37290365828135e-10,5.80885222938139e-10,1.02858504274146e-09,9.67698723675452e-10,8.39452871070888e-10,2.39628364395568e-10 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Industrial,5,9.37290365828135e-10,5.80885222938139e-10,1.02858504274146e-09,9.67698723675452e-10,8.39452871070888e-10,2.39628364395568e-10 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Well,5,8.90552084803996e-10,3.87060694623438e-10,1.24813630833937e-09,8.90552084803996e-10,8.29769567113585e-10,4.42230997130351e-10 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Mine,5,9.37290365828135e-10,5.80885222938139e-10,1.02858504274146e-09,9.67698723675452e-10,8.39452871070888e-10,2.39628364395568e-10 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,EnSoftLin,5,0.0367985674989132,0.018521560297194,0.0629261097189651,0.0367985674989132,0.0409860878361041,0.0232841750821887 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,EnSeismic,5,0.0367985674989132,0.018521560297194,0.0629261097189651,0.0367985674989132,0.0409860878361041,0.0232841750821887 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,HardLin,5,9.37290365828135e-10,5.80885222938139e-10,1.02858504274146e-09,9.67698723675452e-10,8.39452871070888e-10,2.39628364395568e-10 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,TrSoftLin,5,0.0022392793878417,0.00098112936391626,0.00374456156057241,0.00374705166332369,0.002382448206564,0.00138529642908284 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,Crop,5,0.0051325112730342,0.00259608383885104,0.00661901820530916,0.00559204853976238,0.00475794570322879,0.00214570634790766 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,RoughP,5,0.00988095151082199,0.00435679506693851,0.0155976420996964,0.00988095151082199,0.0098898655353174,0.00559256086775056 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,TameP,5,0.00292203063138844,0.000742013980820096,0.00419414527275806,0.00325978967361389,0.00253230027213867,0.00173829261991583 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,WetlandMargin,5,2.01430416447776e-09,1.84918028139387e-09,3.24837282140699e-09,2.06303834717273e-09,2.3671775646272e-09,9.39547051285219e-10 +mite,mite,north,,,Ceratozetes.gracilis,WhiteSpruce,5,0.083324839926525,0.0773341259190752,0.0945808748551744,0.0783644708180673,0.0853402065330667,0.00855276292906888 +mite,mite,north,,,Ceratozetes.gracilis,Pine,5,0.058491231242114,0.0555475742526604,0.0633659637812255,0.0561087495328157,0.0592708435583154,0.00387279881709013 +mite,mite,north,,,Ceratozetes.gracilis,Deciduous,5,0.126688146057812,0.122367096566167,0.134062754480131,0.120032888034155,0.127932927536251,0.00601740175453024 +mite,mite,north,,,Ceratozetes.gracilis,Mixedwood,5,0.0990912857393305,0.0936387390038836,0.106178442160669,0.0922465109400969,0.099754338890101,0.0061787047771312 +mite,mite,north,,,Ceratozetes.gracilis,BlackSpruce,5,0.0210071881552033,0.0194431031984639,0.0243987448926687,0.0201312092719344,0.0216292215405341,0.00266784711827234 +mite,mite,north,,,Ceratozetes.gracilis,TreedFen,5,0.026580724324828,0.024242440270075,0.0332422257164936,0.0260589020761079,0.0282071028227065,0.00473764455368065 +mite,mite,north,,,Ceratozetes.gracilis,TreedSwamp,5,0.0749606880704819,0.0680207443603623,0.0931006713769451,0.0749606880704819,0.0790188543800901,0.0129650555502465 +mite,mite,north,,,Ceratozetes.gracilis,GraminoidFen,5,0.0274581642389803,0.0235853563324741,0.0322028810746652,0.0256468478624544,0.0277164575380178,0.00451584646869259 +mite,mite,north,,,Ceratozetes.gracilis,ShrubbyFen,5,0.0270291842209052,0.0233274790353779,0.0319634992742681,0.0254940252637942,0.0274147093852006,0.00460949355442051 +mite,mite,north,,,Ceratozetes.gracilis,ShrubbyBog,5,0.0221218552654225,0.0203526759367032,0.0258708245808148,0.0212207018552364,0.0228041297102978,0.00295575949415321 +mite,mite,north,,,Ceratozetes.gracilis,ShrubbySwamp,5,0.0787981143345215,0.0700959541686417,0.0912831982274847,0.0787981143345215,0.0801664992667429,0.0104554025659284 +mite,mite,north,,,Ceratozetes.gracilis,Marsh,5,0.0614924047415441,0.0503717579633235,0.0745768553412364,0.0549398690582114,0.0617365809631295,0.0133002434093838 +mite,mite,north,,,Ceratozetes.gracilis,Shrub,5,0.0690326282206128,0.0637085121852232,0.0855845655242005,0.0690326282206128,0.0728644764878167,0.0133405493919686 +mite,mite,north,,,Ceratozetes.gracilis,Grass,5,0.0717095541732357,0.0665149686947679,0.0985453710875707,0.0717095541732357,0.0801620849184831,0.0164795867929258 +mite,mite,north,,,Ceratozetes.gracilis,CCWhiteSpruceR,5,0.107672074614269,0.0972477109913009,0.118541349512236,0.104199864858387,0.107929352562419,0.0109657793464986 +mite,mite,north,,,Ceratozetes.gracilis,CCWhiteSpruce1,5,0.123393977309761,0.112425193396225,0.139387320231618,0.123393977309761,0.125728485214081,0.0140274122599666 +mite,mite,north,,,Ceratozetes.gracilis,CCWhiteSpruce2,5,0.12437831599326,0.111359493905728,0.132705413454322,0.121062001485335,0.123126831939747,0.0114011890353867 +mite,mite,north,,,Ceratozetes.gracilis,CCWhiteSpruce3,5,0.12437831599326,0.111359493905728,0.132705413454322,0.121062001485335,0.123126831939747,0.0114011890353867 +mite,mite,north,,,Ceratozetes.gracilis,CCWhiteSpruce4,5,0.12437831599326,0.111359493905728,0.132705413454322,0.121062001485335,0.123126831939747,0.0114011890353867 +mite,mite,north,,,Ceratozetes.gracilis,CCPineR,5,0.113773520489961,0.101859256408014,0.122211261862114,0.108797637100679,0.112569868029736,0.0104271811989201 +mite,mite,north,,,Ceratozetes.gracilis,CCPine1,5,0.116106855588166,0.100441204830365,0.123976852017442,0.108122243613567,0.113246745405846,0.0120392008920083 +mite,mite,north,,,Ceratozetes.gracilis,CCPine2,5,0.11817066960193,0.100316838150487,0.122679046365626,0.107706392899407,0.113074011687537,0.0117087853519938 +mite,mite,north,,,Ceratozetes.gracilis,CCPine3,5,0.11817066960193,0.100316838150487,0.122679046365626,0.107706392899407,0.113074011687537,0.0117087853519938 +mite,mite,north,,,Ceratozetes.gracilis,CCPine4,5,0.11817066960193,0.100316838150487,0.122679046365626,0.107706392899407,0.113074011687537,0.0117087853519938 +mite,mite,north,,,Ceratozetes.gracilis,CCDeciduousR,5,0.144899037202292,0.133976042079909,0.148502963566285,0.134759239890019,0.14193514316751,0.00733510776173594 +mite,mite,north,,,Ceratozetes.gracilis,CCDeciduous1,5,0.173136461097876,0.161422569448153,0.176484401334677,0.159272382892999,0.169983786435827,0.00768804108229138 +mite,mite,north,,,Ceratozetes.gracilis,CCDeciduous2,5,0.134339082107598,0.127640012945417,0.1355719650486,0.12851467723662,0.132140904595935,0.00407933337080277 +mite,mite,north,,,Ceratozetes.gracilis,CCDeciduous3,5,0.134339082107598,0.127640012945417,0.1355719650486,0.12851467723662,0.132140904595935,0.00407933337080277 +mite,mite,north,,,Ceratozetes.gracilis,CCDeciduous4,5,0.134339082107598,0.127640012945417,0.1355719650486,0.12851467723662,0.132140904595935,0.00407933337080277 +mite,mite,north,,,Ceratozetes.gracilis,CCMixedwoodR,5,0.144899037202292,0.133976042079909,0.148502963566285,0.134759239890019,0.14193514316751,0.00733510776173594 +mite,mite,north,,,Ceratozetes.gracilis,CCMixedwood1,5,0.173136461097876,0.161422569448153,0.176484401334677,0.159272382892999,0.169983786435827,0.00768804108229138 +mite,mite,north,,,Ceratozetes.gracilis,CCMixedwood2,5,0.134339082107598,0.127640012945417,0.1355719650486,0.12851467723662,0.132140904595935,0.00407933337080277 +mite,mite,north,,,Ceratozetes.gracilis,CCMixedwood3,5,0.134339082107598,0.127640012945417,0.1355719650486,0.12851467723662,0.132140904595935,0.00407933337080277 +mite,mite,north,,,Ceratozetes.gracilis,CCMixedwood4,5,0.134339082107598,0.127640012945417,0.1355719650486,0.12851467723662,0.132140904595935,0.00407933337080277 +mite,mite,north,,,Ceratozetes.gracilis,EnSeismic,5,0.0802806120357541,0.0732029866316614,0.108944026446484,0.0802806120357541,0.0889151563754509,0.0183077040123399 +mite,mite,north,,,Ceratozetes.gracilis,EnSoftLin,5,0.0877847291238477,0.0716619099530992,0.114953790660522,0.0797199425033713,0.0919192069743567,0.0218745202790536 +mite,mite,north,,,Ceratozetes.gracilis,TrSoftLin,5,0.0824456293556031,0.0689700330268259,0.109575945632792,0.0742439958268106,0.087725018747832,0.0202270595680831 +mite,mite,north,,,Ceratozetes.gracilis,HardLin,5,0.0268643924591711,0.0257899412772818,0.0304421627930103,0.0268643924591711,0.0278383582584405,0.0023610121346177 +mite,mite,north,,,Ceratozetes.gracilis,UrbInd,5,0.0514318415097621,0.0380292662597745,0.0597468130162695,0.0442950175704556,0.0497575520724983,0.0109608497443732 +mite,mite,north,,,Ceratozetes.gracilis,Wellsites,5,0.035050889042714,0.0311049144974151,0.0455477284585666,0.0328506872802035,0.0375357285513769,0.00732507152064465 +mite,mite,north,,,Ceratozetes.gracilis,Crop,5,0.0215366173604663,0.0212023183077577,0.027823845073315,0.0215366173604663,0.0237248493509573,0.00377213954135615 +mite,mite,north,,,Ceratozetes.gracilis,TameP,5,0.0350884778505926,0.0296620798510974,0.0366792367119695,0.0304807640689111,0.0335100259921961,0.00357425249741613 +mite,mite,north,,,Ceratozetes.gracilis,RoughP,5,0.0516992449980882,0.0487998406748331,0.0647307503735416,0.0516284245607066,0.0557447503800291,0.00818009586326218 +mite,mite,north,,,Trhypochthonius.tectorum,WhiteSpruce,5,0.0556561525490625,0.0510717677245686,0.0590786392356535,0.0510483651362441,0.0550945831947592,0.00398683118385017 +mite,mite,north,,,Trhypochthonius.tectorum,Pine,5,0.0594751004150133,0.0567146072015788,0.07263415395067,0.0594751004150133,0.0635901086572095,0.00817455694019514 +mite,mite,north,,,Trhypochthonius.tectorum,Deciduous,5,0.0535252270853639,0.0476609061082985,0.0610014811112496,0.0514776313631932,0.0542589538223274,0.00671799963208163 +mite,mite,north,,,Trhypochthonius.tectorum,Mixedwood,5,0.0563420244590419,0.0494241961076028,0.0610897108501066,0.0518923527010723,0.0554261228003333,0.00588626866974516 +mite,mite,north,,,Trhypochthonius.tectorum,BlackSpruce,5,0.0479250894638649,0.0425692505329721,0.053061593643466,0.0479250894638649,0.047801363444515,0.00523509405567491 +mite,mite,north,,,Trhypochthonius.tectorum,TreedFen,5,0.0408841194527538,0.0377784135790944,0.0459994038000058,0.0408841194527538,0.0415437643159498,0.00430137829845262 +mite,mite,north,,,Trhypochthonius.tectorum,TreedSwamp,5,0.0337675075759103,0.0315015742262157,0.0379398823268421,0.033440734757267,0.034577386415437,0.00328149143833793 +mite,mite,north,,,Trhypochthonius.tectorum,GraminoidFen,5,0.0407667314478653,0.0375105292015872,0.0453644606141766,0.0407667314478653,0.0412580654715362,0.00397523017374455 +mite,mite,north,,,Trhypochthonius.tectorum,ShrubbyFen,5,0.040625299041178,0.0380204987264493,0.0451305060870398,0.040625299041178,0.0413277749642994,0.00367908136762045 +mite,mite,north,,,Trhypochthonius.tectorum,ShrubbyBog,5,0.0468903066670746,0.0421032711447172,0.0516624938386451,0.0468903066670746,0.0468445466693048,0.00485242361640853 +mite,mite,north,,,Trhypochthonius.tectorum,ShrubbySwamp,5,0.0292316557779894,0.0280267417712238,0.0321894546760997,0.0286964469315836,0.0299491838822735,0.00208326484249253 +mite,mite,north,,,Trhypochthonius.tectorum,Marsh,5,0.0364031182916975,0.03246314137157,0.0387931260174801,0.0341610990357597,0.0358147412526288,0.00318818300839268 +mite,mite,north,,,Trhypochthonius.tectorum,Shrub,5,0.025223191055215,0.020275577906772,0.0300665662767551,0.0282145223452054,0.0251788966856238,0.00483753272341203 +mite,mite,north,,,Trhypochthonius.tectorum,Grass,5,0.0390477719581986,0.0270406150928457,0.0418778229533938,0.0390477719581986,0.0356863738057074,0.00785398705488737 +mite,mite,north,,,Trhypochthonius.tectorum,CCWhiteSpruceR,5,0.0467252252362432,0.0453107339176826,0.0515629452365753,0.0467252252362432,0.0479127175322202,0.00344675695471763 +mite,mite,north,,,Trhypochthonius.tectorum,CCWhiteSpruce1,5,0.0439503346126213,0.0425636157475749,0.0475181466638955,0.0423468795516491,0.0446340028664967,0.0029027760022658 +mite,mite,north,,,Trhypochthonius.tectorum,CCWhiteSpruce2,5,0.0443841971990541,0.0430576800412776,0.0470857489800855,0.043016752346317,0.0447823320518329,0.00228950512145567 +mite,mite,north,,,Trhypochthonius.tectorum,CCWhiteSpruce3,5,0.0443841971990541,0.0430576800412776,0.0470857489800855,0.043016752346317,0.0447823320518329,0.00228950512145567 +mite,mite,north,,,Trhypochthonius.tectorum,CCWhiteSpruce4,5,0.0443841971990541,0.0430576800412776,0.0470857489800855,0.043016752346317,0.0447823320518329,0.00228950512145567 +mite,mite,north,,,Trhypochthonius.tectorum,CCPineR,5,0.0526683392262,0.0494908781950819,0.0603392406871664,0.0518112255096306,0.0542593278877188,0.0059740696863024 +mite,mite,north,,,Trhypochthonius.tectorum,CCPine1,5,0.0468991106936687,0.0446094454502591,0.0499294662723947,0.0447974822354535,0.0471891769898958,0.00263347402531829 +mite,mite,north,,,Trhypochthonius.tectorum,CCPine2,5,0.0470893805229334,0.0447496174892123,0.0504045194737155,0.0449930704020798,0.0474865986654937,0.00280934882782067 +mite,mite,north,,,Trhypochthonius.tectorum,CCPine3,5,0.0470893805229334,0.0447496174892123,0.0504045194737155,0.0449930704020798,0.0474865986654937,0.00280934882782067 +mite,mite,north,,,Trhypochthonius.tectorum,CCPine4,5,0.0470893805229334,0.0447496174892123,0.0504045194737155,0.0449930704020798,0.0474865986654937,0.00280934882782067 +mite,mite,north,,,Trhypochthonius.tectorum,CCDeciduousR,5,0.0514484824942495,0.0469845195905157,0.052906753720658,0.0451521693584135,0.050407563825017,0.00321163341168778 +mite,mite,north,,,Trhypochthonius.tectorum,CCDeciduous1,5,0.0404798408803705,0.0375927192417578,0.0440082227599156,0.0368681428962799,0.0407762801539658,0.00315994871488567 +mite,mite,north,,,Trhypochthonius.tectorum,CCDeciduous2,5,0.0425578952838125,0.0405173081997961,0.0458365008762514,0.039821927948205,0.0429957451821641,0.00274287276636226 +mite,mite,north,,,Trhypochthonius.tectorum,CCDeciduous3,5,0.0425578952838125,0.0405173081997961,0.0458365008762514,0.039821927948205,0.0429957451821641,0.00274287276636226 +mite,mite,north,,,Trhypochthonius.tectorum,CCDeciduous4,5,0.0425578952838125,0.0405173081997961,0.0458365008762514,0.039821927948205,0.0429957451821641,0.00274287276636226 +mite,mite,north,,,Trhypochthonius.tectorum,CCMixedwoodR,5,0.0514484824942495,0.0469845195905157,0.052906753720658,0.0451521693584135,0.050407563825017,0.00321163341168778 +mite,mite,north,,,Trhypochthonius.tectorum,CCMixedwood1,5,0.0404798408803705,0.0375927192417578,0.0440082227599156,0.0368681428962799,0.0407762801539658,0.00315994871488567 +mite,mite,north,,,Trhypochthonius.tectorum,CCMixedwood2,5,0.0425578952838125,0.0405173081997961,0.0458365008762514,0.039821927948205,0.0429957451821641,0.00274287276636226 +mite,mite,north,,,Trhypochthonius.tectorum,CCMixedwood3,5,0.0425578952838125,0.0405173081997961,0.0458365008762514,0.039821927948205,0.0429957451821641,0.00274287276636226 +mite,mite,north,,,Trhypochthonius.tectorum,CCMixedwood4,5,0.0425578952838125,0.0405173081997961,0.0458365008762514,0.039821927948205,0.0429957451821641,0.00274287276636226 +mite,mite,north,,,Trhypochthonius.tectorum,EnSeismic,5,0.0522085190169312,0.0252569549779169,0.0665538824210485,0.0544387764396551,0.0480639407088111,0.0242917296488739 +mite,mite,north,,,Trhypochthonius.tectorum,EnSoftLin,5,0.0357984916598279,0.0190297380136641,0.0645302317518731,0.0402530079505476,0.0399474391546608,0.0257997073771445 +mite,mite,north,,,Trhypochthonius.tectorum,TrSoftLin,5,0.0330350845777108,0.0153319420477188,0.0512108000402237,0.0363880404889003,0.0331889308199334,0.0196952425672342 +mite,mite,north,,,Trhypochthonius.tectorum,HardLin,5,0.0243958652679851,0.0239704518035738,0.0279997624131438,0.0243958652679851,0.0256743635254181,0.00210475588272506 +mite,mite,north,,,Trhypochthonius.tectorum,UrbInd,5,0.0191105703106928,0.0167840749551519,0.0272407405128375,0.0191105703106928,0.0212349933310677,0.00546676273320779 +mite,mite,north,,,Trhypochthonius.tectorum,Wellsites,5,0.023351034935016,0.021158057868554,0.03223617749299,0.023351034935016,0.0259815706015033,0.00569106451947394 +mite,mite,north,,,Trhypochthonius.tectorum,Crop,5,0.015382693809727,0.0151069200337959,0.016892523752058,0.0152304430995811,0.0158155351878053,0.00103439159129019 +mite,mite,north,,,Trhypochthonius.tectorum,TameP,5,0.0258915433874115,0.0222945289246262,0.0264680931726066,0.024941743213498,0.0248462640710173,0.00249172002079151 +mite,mite,north,,,Trhypochthonius.tectorum,RoughP,5,0.0328180516103049,0.0322239355913061,0.0387542589134784,0.0375661766298358,0.0348949895183175,0.00342630396737567 +mite,mite,south,,,Ceratozetes.gracilis,Loamy,5,0.0218423245463604,0.0173672203176517,0.0230295231243844,0.0218423245463604,0.020747214981276,0.00349328573027265 +mite,mite,south,,,Ceratozetes.gracilis,SandyLoam,5,0.0188962100537126,0.0162686755399601,0.0200379292106758,0.0185871549772747,0.0184312571346189,0.00217691199563813 +mite,mite,south,,,Ceratozetes.gracilis,ClaySub,5,0.0190796631779778,0.0156078918819073,0.0210172713505711,0.0190796631779778,0.018604719484685,0.00298067726778378 +mite,mite,south,,,Ceratozetes.gracilis,RapidDrain,5,0.00969791273666386,0.00942196936388417,0.0112386251640146,0.0110451655878437,0.010200805407554,0.000934771929061484 +mite,mite,south,,,Ceratozetes.gracilis,Blowout,5,0.0106020081675796,0.00863605999566094,0.0123463313279677,0.0108663421303504,0.0105398831320641,0.00211371743591744 +mite,mite,south,,,Ceratozetes.gracilis,ThinBreak,5,0.00878234749614848,0.00798269496694178,0.0107794669342902,0.00996372938352331,0.00924540029110673,0.00143456006902455 +mite,mite,south,,,Ceratozetes.gracilis,Other,5,0.0042686164608738,0.00063572314444851,0.00731890391266918,0.0042686164608738,0.00407391061994681,0.00329148805533948 +mite,mite,south,,,Ceratozetes.gracilis,EnSeismic,5,0.0128342993205689,0.00852724915739246,0.0633937793424426,0.0128342993205689,0.0301196175594858,0.0296767966386737 +mite,mite,south,,,Ceratozetes.gracilis,EnSoftLin,5,0.0100953485593809,0.00859879515579537,0.0518084711684424,0.0100953485593809,0.0252270354093779,0.0237484297671399 +mite,mite,south,,,Ceratozetes.gracilis,TrSoftLin,5,0.00945493320863686,0.00758629346095882,0.0565822367112004,0.00945493320863686,0.0263834449674863,0.0270499023163353 +mite,mite,south,,,Ceratozetes.gracilis,HardLin,5,0.0012818592428196,0.00122480067982412,0.00171071495716722,0.00176154072489266,0.00142873179782241,0.000252311336064879 +mite,mite,south,,,Ceratozetes.gracilis,UrbInd,5,0.00425941571595646,0.00295877473501209,0.00595057748445942,0.00425941571595646,0.00440787198450415,0.00153151605374925 +mite,mite,south,,,Ceratozetes.gracilis,Wellsites,5,0.00295291600474109,0.0021973086508854,0.00411443751717638,0.00295291600474109,0.00310781362870248,0.000980789777054813 +mite,mite,south,,,Ceratozetes.gracilis,Crop,5,0.00116588959329944,0.00104101704994527,0.00133284725072961,0.00116588959329944,0.00117831505621224,0.000167208502276051 +mite,mite,south,,,Ceratozetes.gracilis,TameP,5,0.0024464095231432,0.00214847243566944,0.00305858273470107,0.0023921447017999,0.00257807014953875,0.000460747234171761 +mite,mite,south,,,Ceratozetes.gracilis,RoughP,5,0.0023839015453601,0.00176469662219973,0.00260842275480983,0.00218127936434137,0.00224033670141993,0.000466899834329463 +mite,mite,south,,,Trhypochthonius.tectorum,Loamy,5,0.0499778174612043,0.0477160469108668,0.0521891286113536,0.0467852744100426,0.0500400973814781,0.00226406307817558 +mite,mite,south,,,Trhypochthonius.tectorum,SandyLoam,5,0.0427437044233035,0.0388352222189595,0.0446878887454315,0.0397630370091323,0.0419370284087006,0.00292844495366087 +mite,mite,south,,,Trhypochthonius.tectorum,ClaySub,5,0.0448532047061975,0.0434266302673582,0.0480060341777117,0.0429142869672478,0.0454925438914735,0.00235630079412914 +mite,mite,south,,,Trhypochthonius.tectorum,RapidDrain,5,0.0431738225185761,0.036600607606764,0.0449491670506471,0.0431738225185761,0.0415360440333195,0.0048551707088825 +mite,mite,south,,,Trhypochthonius.tectorum,Blowout,5,0.0612889561023144,0.0491998328469309,0.0746776432481998,0.0612889561023144,0.0622915416103894,0.013487238751435 +mite,mite,south,,,Trhypochthonius.tectorum,ThinBreak,5,0.0472879860660164,0.037920270451541,0.0523574915662019,0.0472879860660164,0.0458133113776339,0.00776225914513275 +mite,mite,south,,,Trhypochthonius.tectorum,Other,5,0.0366771544480599,0.0228268207831538,0.0785517999620398,0.0478734329425522,0.0465964414370044,0.0315527855316975 +mite,mite,south,,,Trhypochthonius.tectorum,EnSeismic,5,0.005718045734362,0.000756250016234319,0.00887832731960815,0.00888461717648373,0.00506648764138872,0.00403707236060029 +mite,mite,south,,,Trhypochthonius.tectorum,EnSoftLin,5,0.00350908933263554,0.000310987183741067,0.00741715216602833,0.00723292387135889,0.0038111174289397,0.00351580714176629 +mite,mite,south,,,Trhypochthonius.tectorum,TrSoftLin,5,0.00296547128398102,0.000271010557576361,0.00513825945876903,0.00426948908637977,0.00272565881367204,0.0023988556977813 +mite,mite,south,,,Trhypochthonius.tectorum,HardLin,5,0.0352820429856263,0.0249011898739522,0.0517606271764086,0.0281443034455901,0.0370353971906418,0.0145923801432061 +mite,mite,south,,,Trhypochthonius.tectorum,UrbInd,5,0.0197564607327612,0.0161346626233351,0.0233198049521763,0.0197564607327612,0.0196905028450629,0.00383354217797874 +mite,mite,south,,,Trhypochthonius.tectorum,Wellsites,5,0.0218127811081771,0.0174582402547921,0.0273373033988181,0.0219652759405907,0.0220697701960811,0.00550369496809574 +mite,mite,south,,,Trhypochthonius.tectorum,Crop,5,0.0174842052478879,0.0146182157223179,0.0189255858640754,0.0177171723605232,0.0168444896471002,0.00219178225580625 +mite,mite,south,,,Trhypochthonius.tectorum,TameP,5,0.0334374326549006,0.0271748172003896,0.0414968532162421,0.0334374326549006,0.0341507630943171,0.00734564079431658 +mite,mite,south,,,Trhypochthonius.tectorum,RoughP,5,0.0330917206876311,0.0281248674259134,0.0375615466118284,0.0330917206876311,0.0330472057370728,0.00502984292008219 +vascular_plant,vascular_plant,north,,,Galium.boreale,WhiteSpruce,5,0.136563367302205,0.119462518901444,0.14703521115999,0.128707314176715,0.134438167919802,0.0140790776061126 +vascular_plant,vascular_plant,north,,,Galium.boreale,Pine,5,0.116853425497967,0.107796441903968,0.124081762452193,0.108967866096832,0.115733912410419,0.00857502539967135 +vascular_plant,vascular_plant,north,,,Galium.boreale,Deciduous,5,0.25887511408307,0.239156447325757,0.263482889907708,0.243876909457892,0.252938955637841,0.0124338313554241 +vascular_plant,vascular_plant,north,,,Galium.boreale,Mixedwood,5,0.193041971414329,0.179396957180547,0.201982078860589,0.180205162477876,0.190818610539326,0.0113637588317406 +vascular_plant,vascular_plant,north,,,Galium.boreale,BlackSpruce,5,0.0197313184843524,0.0186615095379396,0.0216346630739543,0.0190889513235295,0.019992432400775,0.00162296400530185 +vascular_plant,vascular_plant,north,,,Galium.boreale,TreedFen,5,0.0190279322211574,0.0171394699429018,0.0204803027951359,0.0184868712710714,0.0188628976679575,0.00180798674115567 +vascular_plant,vascular_plant,north,,,Galium.boreale,TreedSwamp,5,0.0537157382799687,0.0494547113304226,0.0594834887289069,0.0502065849301497,0.0541128640771137,0.00517780601239197 +vascular_plant,vascular_plant,north,,,Galium.boreale,GraminoidFen,5,0.0207928426181494,0.0196811431002025,0.0226503378120183,0.0207928426181494,0.0210542335971933,0.0015438256424816 +vascular_plant,vascular_plant,north,,,Galium.boreale,ShrubbyFen,5,0.0201391073392683,0.0190935576196826,0.0221089799761179,0.0201391073392683,0.0204418135172878,0.00167612468006209 +vascular_plant,vascular_plant,north,,,Galium.boreale,ShrubbyBog,5,0.0199495272991526,0.0188220126503618,0.0218233356490728,0.0193388406568705,0.0201794222265195,0.00164466018474895 +vascular_plant,vascular_plant,north,,,Galium.boreale,ShrubbySwamp,5,0.045688982509881,0.042171496476318,0.0571441122739526,0.045688982509881,0.0487114824451972,0.00788260483962487 +vascular_plant,vascular_plant,north,,,Galium.boreale,Marsh,5,0.041347185000327,0.0369944475953885,0.0480790679841081,0.041347185000327,0.0422373156592065,0.00549952228800232 +vascular_plant,vascular_plant,north,,,Galium.boreale,Shrub,5,0.189304761120091,0.175516236345066,0.241103793907795,0.189304761120091,0.204712579589746,0.033439991322378 +vascular_plant,vascular_plant,north,,,Galium.boreale,Grass,5,0.214959292683282,0.180869477747652,0.249359372444126,0.191616359545112,0.215525986480311,0.0336975831668747 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCWhiteSpruceR,5,0.154137545244941,0.132581336610888,0.172927485618722,0.143043678232451,0.153186792800721,0.0202008044704786 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCWhiteSpruce1,5,0.163315085825216,0.141002273364349,0.18905935770425,0.158059058434557,0.165296954029114,0.0246109466321505 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCWhiteSpruce2,5,0.160436563203294,0.14069222929203,0.183967841364971,0.148743886710094,0.161992104476718,0.021365400368824 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCWhiteSpruce3,5,0.160436563203294,0.14069222929203,0.183967841364971,0.148743886710094,0.161992104476718,0.021365400368824 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCWhiteSpruce4,5,0.160436563203294,0.14069222929203,0.183967841364971,0.148743886710094,0.161992104476718,0.021365400368824 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCPineR,5,0.143905674843488,0.117440734273828,0.157542491898074,0.126516675922557,0.13912009366266,0.0200221797521959 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCPine1,5,0.135255973929374,0.114079157254064,0.142460383178462,0.123115076680262,0.129792039810823,0.0148449152061657 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCPine2,5,0.134477552857653,0.11341349299191,0.141401321460665,0.12088426096238,0.128884997779635,0.0144383340887116 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCPine3,5,0.134477552857653,0.11341349299191,0.141401321460665,0.12088426096238,0.128884997779635,0.0144383340887116 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCPine4,5,0.134477552857653,0.11341349299191,0.141401321460665,0.12088426096238,0.128884997779635,0.0144383340887116 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCDeciduousR,5,0.208696634917917,0.175761239502858,0.219246203164378,0.186398225011703,0.200187968090386,0.022106120959872 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCDeciduous1,5,0.256400751129383,0.223605802344556,0.280481606679469,0.232409451432954,0.253034900240866,0.0279823632616457 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCDeciduous2,5,0.196550402723055,0.177363867788528,0.223430883577534,0.189116187021577,0.199874753985579,0.0230468937687924 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCDeciduous3,5,0.196550402723055,0.177363867788528,0.223430883577534,0.189116187021577,0.199874753985579,0.0230468937687924 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCDeciduous4,5,0.196550402723055,0.177363867788528,0.223430883577534,0.189116187021577,0.199874753985579,0.0230468937687924 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCMixedwoodR,5,0.208696634917917,0.175761239502858,0.219246203164378,0.186398225011703,0.200187968090386,0.022106120959872 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCMixedwood1,5,0.256400751129383,0.223605802344556,0.280481606679469,0.232409451432954,0.253034900240866,0.0279823632616457 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCMixedwood2,5,0.196550402723055,0.177363867788528,0.223430883577534,0.189116187021577,0.199874753985579,0.0230468937687924 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCMixedwood3,5,0.196550402723055,0.177363867788528,0.223430883577534,0.189116187021577,0.199874753985579,0.0230468937687924 +vascular_plant,vascular_plant,north,,,Galium.boreale,CCMixedwood4,5,0.196550402723055,0.177363867788528,0.223430883577534,0.189116187021577,0.199874753985579,0.0230468937687924 +vascular_plant,vascular_plant,north,,,Galium.boreale,EnSeismic,5,0.0675357940232228,0.0536641840810974,0.0782448772200584,0.0666893133709304,0.067002736116644,0.0134984598966857 +vascular_plant,vascular_plant,north,,,Galium.boreale,EnSoftLin,5,0.0666450469830431,0.0514688065567067,0.0769360972499699,0.0632085789748528,0.0651110464552629,0.0136601040156264 +vascular_plant,vascular_plant,north,,,Galium.boreale,TrSoftLin,5,0.0681813236346331,0.0554454641142448,0.0796107459686051,0.0681813236346331,0.0680202870748694,0.0125432834976175 +vascular_plant,vascular_plant,north,,,Galium.boreale,HardLin,5,0.032772954964339,0.0254952937216265,0.0427628821147054,0.0304014203144609,0.0337542163171502,0.00906837104577133 +vascular_plant,vascular_plant,north,,,Galium.boreale,UrbInd,5,0.032891069156295,0.0290596861735981,0.0426826279515852,0.0323859521239614,0.0350214724592512,0.00762900448359572 +vascular_plant,vascular_plant,north,,,Galium.boreale,Wellsites,5,0.0313114303488669,0.0258224080114288,0.0445788771357667,0.0313114303488669,0.03407323779985,0.0104681365081302 +vascular_plant,vascular_plant,north,,,Galium.boreale,Crop,5,0.0114819275038514,0.00962690543049514,0.0155602283722968,0.0114819275038514,0.0123694186188488,0.00300450571509112 +vascular_plant,vascular_plant,north,,,Galium.boreale,TameP,5,0.0336485279117461,0.0256580113642758,0.0375637064974261,0.0283837215953415,0.0320336236319229,0.00600032864643425 +vascular_plant,vascular_plant,north,,,Galium.boreale,RoughP,5,0.0579560450257088,0.045965364331808,0.0773913737984615,0.0532646872909717,0.0606155261777185,0.0164348500854686 +vascular_plant,vascular_plant,north,,,Vicia.americana,WhiteSpruce,5,0.0717211911881468,0.0649940227788211,0.0757806047387571,0.0717211911881468,0.0707611702697563,0.00560071424718983 +vascular_plant,vascular_plant,north,,,Vicia.americana,Pine,5,0.0717443897992296,0.0637388171549185,0.0792558735029598,0.0717443897992296,0.0716467451517877,0.00773413521691436 +vascular_plant,vascular_plant,north,,,Vicia.americana,Deciduous,5,0.174528296976248,0.164090538301261,0.183795585911188,0.174528296976248,0.173876959799232,0.0107120597059939 +vascular_plant,vascular_plant,north,,,Vicia.americana,Mixedwood,5,0.12738407433249,0.118327966346565,0.12914316163785,0.123903841395087,0.124825781574101,0.00609464551627694 +vascular_plant,vascular_plant,north,,,Vicia.americana,BlackSpruce,5,0.0108602686526196,0.00947682296052838,0.0111197345835797,0.0110635169948055,0.0104434018004056,0.000868359893238048 +vascular_plant,vascular_plant,north,,,Vicia.americana,TreedFen,5,0.0104889884001634,0.00918775668655501,0.0114259715545775,0.0111781283427943,0.0104110091403938,0.00125060799360988 +vascular_plant,vascular_plant,north,,,Vicia.americana,TreedSwamp,5,0.0474586407623415,0.0434617027113533,0.0573459768855838,0.0505384149026876,0.0495081405966026,0.00757780069669055 +vascular_plant,vascular_plant,north,,,Vicia.americana,GraminoidFen,5,0.012380186783363,0.0106055968142056,0.0129729819618124,0.0128602111397556,0.0119986050213911,0.00138377521512343 +vascular_plant,vascular_plant,north,,,Vicia.americana,ShrubbyFen,5,0.0122885356189932,0.0107844229705085,0.0129771366500186,0.0127117113440904,0.0120371989894358,0.001277638072515 +vascular_plant,vascular_plant,north,,,Vicia.americana,ShrubbyBog,5,0.0111679104948312,0.00984870753108598,0.01144241264329,0.0114521553910913,0.0107985890482843,0.000879046887316585 +vascular_plant,vascular_plant,north,,,Vicia.americana,ShrubbySwamp,5,0.0442619565088811,0.039395611852956,0.0514550187832082,0.0467803608934761,0.045111534045516,0.00637251444497983 +vascular_plant,vascular_plant,north,,,Vicia.americana,Marsh,5,0.0395269377197916,0.0379010981822006,0.0480740857526984,0.0426749909362641,0.0419761840756894,0.00575798827386886 +vascular_plant,vascular_plant,north,,,Vicia.americana,Shrub,5,0.124413496826489,0.11630879041929,0.148831794626625,0.129561071424494,0.129997351609217,0.0187159161627647 +vascular_plant,vascular_plant,north,,,Vicia.americana,Grass,5,0.140899561631048,0.132647046765579,0.149765706126522,0.140899561631048,0.141003833991578,0.00975138040532848 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCWhiteSpruceR,5,0.106333838059199,0.10563563998252,0.1113390336656,0.113324612808195,0.107878609308115,0.00324098638860528 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCWhiteSpruce1,5,0.132673132079881,0.131014835763159,0.139245186106205,0.140904763899433,0.134539838012183,0.00426782271680661 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCWhiteSpruce2,5,0.122518637692463,0.118113328719225,0.134306251834235,0.129378591587472,0.125179320953689,0.0083140378258789 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCWhiteSpruce3,5,0.122518637692463,0.118113328719225,0.134306251834235,0.129378591587472,0.125179320953689,0.0083140378258789 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCWhiteSpruce4,5,0.122518637692463,0.118113328719225,0.134306251834235,0.129378591587472,0.125179320953689,0.0083140378258789 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCPineR,5,0.108086061678854,0.105116454412193,0.111847166658453,0.113106276644051,0.108442371914164,0.00352518043702876 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCPine1,5,0.1122188310588,0.109732769391169,0.12258810268514,0.121735348725543,0.115445950654567,0.00657248417948193 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCPine2,5,0.109183815724588,0.106720994046657,0.120537973445125,0.118682344001148,0.112706633163643,0.00705791950102736 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCPine3,5,0.109183815724588,0.106720994046657,0.120537973445125,0.118682344001148,0.112706633163643,0.00705791950102736 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCPine4,5,0.109183815724588,0.106720994046657,0.120537973445125,0.118682344001148,0.112706633163643,0.00705791950102736 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCDeciduousR,5,0.145906249437239,0.140931022371577,0.15495341081461,0.151623580611421,0.147413609979929,0.00700112483338789 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCDeciduous1,5,0.180166516850587,0.168695392463167,0.188786472181105,0.183212683986236,0.178869118903198,0.010081669607544 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCDeciduous2,5,0.156982604814944,0.145251669372505,0.15939063660912,0.156982604814944,0.153284371506591,0.00726664394733218 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCDeciduous3,5,0.156982604814944,0.145251669372505,0.15939063660912,0.156982604814944,0.153284371506591,0.00726664394733218 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCDeciduous4,5,0.156982604814944,0.145251669372505,0.15939063660912,0.156982604814944,0.153284371506591,0.00726664394733218 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCMixedwoodR,5,0.145906249437239,0.140931022371577,0.15495341081461,0.151623580611421,0.147413609979929,0.00700112483338789 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCMixedwood1,5,0.180166516850587,0.168695392463167,0.188786472181105,0.183212683986236,0.178869118903198,0.010081669607544 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCMixedwood2,5,0.156982604814944,0.145251669372505,0.15939063660912,0.156982604814944,0.153284371506591,0.00726664394733218 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCMixedwood3,5,0.156982604814944,0.145251669372505,0.15939063660912,0.156982604814944,0.153284371506591,0.00726664394733218 +vascular_plant,vascular_plant,north,,,Vicia.americana,CCMixedwood4,5,0.156982604814944,0.145251669372505,0.15939063660912,0.156982604814944,0.153284371506591,0.00726664394733218 +vascular_plant,vascular_plant,north,,,Vicia.americana,EnSeismic,5,0.216656015711896,0.168139707348619,0.254376282862556,0.216656015711896,0.213098910528715,0.0452535455491439 +vascular_plant,vascular_plant,north,,,Vicia.americana,EnSoftLin,5,0.240353254005159,0.182193409855272,0.287354412958843,0.240353254005159,0.235885270025637,0.0525043948865044 +vascular_plant,vascular_plant,north,,,Vicia.americana,TrSoftLin,5,0.185413846451138,0.143618351581936,0.21612482061455,0.185413846451138,0.182132121492675,0.03663912522577 +vascular_plant,vascular_plant,north,,,Vicia.americana,HardLin,5,0.0510142064754805,0.0476995560327749,0.0534135994986585,0.0494242045429297,0.0506844765474707,0.00289619110670257 +vascular_plant,vascular_plant,north,,,Vicia.americana,UrbInd,5,0.060270971424988,0.0527176972963101,0.0669155114807268,0.0570164101210413,0.0598116387590278,0.00755833843006598 +vascular_plant,vascular_plant,north,,,Vicia.americana,Wellsites,5,0.0495816235777512,0.045261299115898,0.0675981711099802,0.0583800066861835,0.0547080768995733,0.0120228518497499 +vascular_plant,vascular_plant,north,,,Vicia.americana,Crop,5,0.0198021833360757,0.0187130492427477,0.0215978803800823,0.020499648487122,0.0200793479234006,0.00155071434603584 +vascular_plant,vascular_plant,north,,,Vicia.americana,TameP,5,0.0435685547091363,0.0402056228771878,0.0533172713621298,0.0435685547091363,0.0457946494684893,0.00709067313037913 +vascular_plant,vascular_plant,north,,,Vicia.americana,RoughP,5,0.0428632794956036,0.0372440489603054,0.0534178816037923,0.0428632794956036,0.0444803287325393,0.00836089560838824 +vascular_plant,vascular_plant,south,,,Galium.boreale,Loamy,5,0.128653205010673,0.114611972972335,0.130784674892184,0.129566548831882,0.123883607184517,0.00838623611422754 +vascular_plant,vascular_plant,south,,,Galium.boreale,SandyLoam,5,0.0836110810472021,0.0746816948024742,0.0894208141830331,0.0846341790254643,0.0821153995584725,0.00751049498598007 +vascular_plant,vascular_plant,south,,,Galium.boreale,ClaySub,5,0.101500964583409,0.0940741502041846,0.108970943759895,0.106582920663178,0.101462761523999,0.00731517240310769 +vascular_plant,vascular_plant,south,,,Galium.boreale,RapidDrain,5,0.0388994289485226,0.0346245997860596,0.0529303847658474,0.0448318785090138,0.0424505639887575,0.00987053097134808 +vascular_plant,vascular_plant,south,,,Galium.boreale,Blowout,5,0.0506231282331206,0.0397681707601058,0.0574808564556098,0.0506231282331206,0.0488032044086913,0.00890397864448909 +vascular_plant,vascular_plant,south,,,Galium.boreale,ThinBreak,5,0.0479802424154928,0.0420075163241584,0.0517461800601525,0.051022562219615,0.0472278000794808,0.00488658793646957 +vascular_plant,vascular_plant,south,,,Galium.boreale,Other,5,0.0106061234511555,0.00720081589340935,0.0217479373094132,0.0177197087652051,0.0136291572357968,0.00750330686270611 +vascular_plant,vascular_plant,south,,,Galium.boreale,EnSeismic,5,0.0472125133205467,0.0295924639744709,0.110125197726374,0.0287456404184259,0.0636144790463627,0.0421297958204094 +vascular_plant,vascular_plant,south,,,Galium.boreale,EnSoftLin,5,0.0457108677001631,0.0342927417032321,0.133999583987908,0.0329811808897455,0.0738711315094049,0.0544940604720431 +vascular_plant,vascular_plant,south,,,Galium.boreale,TrSoftLin,5,0.0254268182100485,0.0116038912895437,0.0640062980763588,0.0143164876649795,0.0337396409014619,0.0288740328272036 +vascular_plant,vascular_plant,south,,,Galium.boreale,HardLin,5,0.0032621931802875,0.00285851385150083,0.00701707647763527,0.00538608376568717,0.00451424153984983,0.00226371892322936 +vascular_plant,vascular_plant,south,,,Galium.boreale,UrbInd,5,0.00523304185300579,0.00516680867336443,0.00698867393616868,0.00519980939835304,0.00583554776095676,0.00110498090685346 +vascular_plant,vascular_plant,south,,,Galium.boreale,Wellsites,5,0.00835585771334364,0.00699275850468526,0.00923375703992824,0.00769427442003421,0.0081731242777985,0.00114681019974826 +vascular_plant,vascular_plant,south,,,Galium.boreale,Crop,5,0.00170386065696764,0.00159546521345581,0.00280933357726183,0.00225132683231577,0.00206801467544986,0.00067868464225799 +vascular_plant,vascular_plant,south,,,Galium.boreale,TameP,5,0.00370735476367147,0.00294261667373477,0.00558659216344689,0.00460921237816997,0.00410416049108617,0.00137496323633074 +vascular_plant,vascular_plant,south,,,Galium.boreale,RoughP,5,0.00542654337673869,0.00450815448744484,0.00938121797095186,0.00706698519527935,0.00652782936700246,0.00267894412351735 +vascular_plant,vascular_plant,south,,,Vicia.americana,Loamy,5,0.280824609747584,0.267644583576923,0.289709776497706,0.273848614363572,0.279520051194194,0.0111565423059792 +vascular_plant,vascular_plant,south,,,Vicia.americana,SandyLoam,5,0.243407150088342,0.234204092076879,0.250601947319584,0.239431294773893,0.242922321341985,0.00832583255140513 +vascular_plant,vascular_plant,south,,,Vicia.americana,ClaySub,5,0.29707235668037,0.282488985824439,0.305118529233993,0.291486932950625,0.294780505613319,0.0118392065552878 +vascular_plant,vascular_plant,south,,,Vicia.americana,RapidDrain,5,0.149586849572318,0.132500335286096,0.175603735182626,0.149586849572318,0.152479085699985,0.0216357444325811 +vascular_plant,vascular_plant,south,,,Vicia.americana,Blowout,5,0.106950650814662,0.0907039605371348,0.121336311234738,0.103869081093715,0.106769891695465,0.0160956019903277 +vascular_plant,vascular_plant,south,,,Vicia.americana,ThinBreak,5,0.165570611241508,0.152106899610986,0.186190099937036,0.165570611241508,0.167794270266068,0.0186214443757594 +vascular_plant,vascular_plant,south,,,Vicia.americana,Other,5,0.102651600983928,0.0624149706900878,0.11235472915199,0.102651600983928,0.0922572898504469,0.0286236891835733 +vascular_plant,vascular_plant,south,,,Vicia.americana,EnSeismic,5,0.0987770601076429,0.0590846473435492,0.125153182170927,0.142317281363727,0.0937633038233823,0.0369262953861683 +vascular_plant,vascular_plant,south,,,Vicia.americana,EnSoftLin,5,0.124641821593195,0.0681868022186765,0.173426846411391,0.184942717247886,0.121843755999216,0.0520061572176103 +vascular_plant,vascular_plant,south,,,Vicia.americana,TrSoftLin,5,0.0632469144327385,0.0350786082680709,0.083335221572322,0.0865561569046063,0.0607365434283008,0.0244516505628151 +vascular_plant,vascular_plant,south,,,Vicia.americana,HardLin,5,0.034768741467538,0.0317254788925045,0.0411641044703837,0.0319213954295495,0.0358461107697019,0.00503353926558842 +vascular_plant,vascular_plant,south,,,Vicia.americana,UrbInd,5,0.0317815932567955,0.0271544908564896,0.0356957214790492,0.0250464912812154,0.0316935547923611,0.00440557843195001 +vascular_plant,vascular_plant,south,,,Vicia.americana,Wellsites,5,0.0341400292126126,0.0279864934648464,0.0423488240839157,0.0262553863998158,0.034854112138194,0.00723959324487174 +vascular_plant,vascular_plant,south,,,Vicia.americana,Crop,5,0.0239011853373428,0.0220577047645298,0.0264567603480559,0.0226673334870154,0.0241526345182289,0.00219200225599506 +vascular_plant,vascular_plant,south,,,Vicia.americana,TameP,5,0.0312372001895896,0.0292297510784975,0.0351771559180196,0.0306301444015498,0.0319120315749738,0.00324668537783287 +vascular_plant,vascular_plant,south,,,Vicia.americana,RoughP,5,0.0374123655632768,0.0359902090243626,0.0432584803569596,0.0374123655632768,0.0388877931026711,0.00439952763870878 diff --git a/3_output/exp_000_parity_v2/tables/metric_summary.csv b/3_output/exp_000_parity_v2/tables/metric_summary.csv index 06daa4c..546c2e7 100644 --- a/3_output/exp_000_parity_v2/tables/metric_summary.csv +++ b/3_output/exp_000_parity_v2/tables/metric_summary.csv @@ -1,617 +1,617 @@ -taxon,run,region,season,part,species,metric,n,median,p10,p90,boot1 -bird,bird,north,,,AMRO,insample_auc,100,0.722941291958026,0.720972868858002,0.725025884727729,0.721701661835893 -bird,bird,north,,,YEWA,insample_auc,100,0.778186953289465,0.775689125540707,0.78064354760127,0.779841092447633 -bird,bird,north,,,AMRO,insample_calibration_slope,100,0.80439129216793,0.794598673414551,0.813263646692552,0.80440117768677 -bird,bird,north,,,YEWA,insample_calibration_slope,100,0.68667256345246,0.65127717317625,0.716419429614271,0.716264166431109 -bird,bird,north,,,AMRO,insample_deviance_explained,100,0.141892228737756,0.139516850461149,0.144440700348185,0.140587647159979 -bird,bird,north,,,YEWA,insample_deviance_explained,100,0.25902142141057,0.254928440560819,0.26248816234098,0.259141214360432 -bird,bird,north,,,AMRO,insample_n,100,46751,46746.9,46756.1,46747 -bird,bird,north,,,YEWA,insample_n,100,46751,46746.9,46756.1,46747 -bird,bird,north,,,AMRO,insample_prevalence,100,0.251013357719858,0.249469515535677,0.252466035822628,0.250861017819325 -bird,bird,north,,,YEWA,insample_prevalence,100,0.119004833940844,0.118139055614803,0.119814101354467,0.118296361263824 -bird,bird,north,,,AMRO,insample_rmse,100,0.663529568645484,0.659762946701523,0.667800183229973,0.664239276180889 -bird,bird,north,,,YEWA,insample_rmse,100,0.551892225105279,0.544039433841324,0.560242541149428,0.547158394049375 -bird,bird,north,,,AMRO,insample_spearman,100,0.341332033991246,0.337958860752305,0.344541559969754,0.339249301212572 -bird,bird,north,,,YEWA,insample_spearman,100,0.315129544166726,0.312069108270516,0.318157185925632,0.316093683296469 -bird,bird,north,,,AMRO,oob_auc,100,0.731083020235213,0.729728694406019,0.731997269207461,0.730766733254682 -bird,bird,north,,,YEWA,oob_auc,100,0.728919378934302,0.727561350140747,0.730408468931927,0.730809600284142 -bird,bird,north,,,AMRO,oob_calibration_slope,100,0.87705334102173,0.859046659953532,0.892405134838974,0.881696624797909 -bird,bird,north,,,YEWA,oob_calibration_slope,100,0.64415917079899,0.561997241434001,0.704587672561171,0.718728425403305 -bird,bird,north,,,AMRO,oob_deviance_explained,100,0.152102139021978,0.150344464224191,0.153768435297582,0.152754178062373 -bird,bird,north,,,YEWA,oob_deviance_explained,100,0.179057138830537,0.175210549900854,0.181998524533593,0.183210182399369 -bird,bird,north,,,AMRO,oob_n,100,124873,124867.9,124877.1,124877 -bird,bird,north,,,YEWA,oob_n,100,124873,124867.9,124877.1,124877 -bird,bird,north,,,AMRO,oob_prevalence,100,0.241188364632105,0.240643779705197,0.241766126792771,0.241245385459292 -bird,bird,north,,,YEWA,oob_prevalence,100,0.134323099251165,0.134021369904622,0.134647493509128,0.13458843502006 -bird,bird,north,,,AMRO,oob_rmse,100,0.65796553610531,0.656284605026556,0.659354479236559,0.657851304309638 -bird,bird,north,,,YEWA,oob_rmse,100,0.591993296525432,0.582825632432554,0.603629490711002,0.581496589737255 -bird,bird,north,,,AMRO,oob_spearman,100,0.349000795818394,0.346958699831137,0.350450615002725,0.348509330988823 -bird,bird,north,,,YEWA,oob_spearman,100,0.273840417323665,0.272323641876612,0.27551058043422,0.276382265893519 -bird,bird,south,,,AMRO,insample_auc,100,0.692241105195695,0.689277087417165,0.695254459353952,0.689537506786954 -bird,bird,south,,,YEWA,insample_auc,100,0.71931881761428,0.715530606727552,0.722950865983364,0.721262730392428 -bird,bird,south,,,AMRO,insample_calibration_slope,100,0.891418274449085,0.875752221574822,0.907045276999385,0.885403476414633 -bird,bird,south,,,YEWA,insample_calibration_slope,100,0.866682086841971,0.860362489808398,0.873353081017801,0.860619291916303 -bird,bird,south,,,AMRO,insample_deviance_explained,100,0.0922921788656006,0.0899257972801956,0.0959833319452778,0.0905093060629211 -bird,bird,south,,,YEWA,insample_deviance_explained,100,0.23129680597227,0.226310074573946,0.235915323956573,0.229114411717017 -bird,bird,south,,,AMRO,insample_n,100,23638,23635,23641,23640 -bird,bird,south,,,YEWA,insample_n,100,23638,23635,23641,23640 -bird,bird,south,,,AMRO,insample_prevalence,100,0.257858442272708,0.255935560656765,0.259915379733446,0.25655668358714 -bird,bird,south,,,YEWA,insample_prevalence,100,0.172480169299808,0.171161682117149,0.173982762791556,0.171742808798646 -bird,bird,south,,,AMRO,insample_rmse,100,0.742969543166091,0.737049963427855,0.748369656814777,0.746104469516561 -bird,bird,south,,,YEWA,insample_rmse,100,0.626430819488391,0.620643891953174,0.63211456133757,0.629305991109963 -bird,bird,south,,,AMRO,insample_spearman,100,0.292644243026994,0.288035146020249,0.296748860514725,0.287696121760142 -bird,bird,south,,,YEWA,insample_spearman,100,0.29573200936581,0.290701723813243,0.300697611022088,0.297439751075024 -bird,bird,south,,,AMRO,oob_auc,100,0.689119265153378,0.68765134183379,0.690180541905298,0.689591934626615 -bird,bird,south,,,YEWA,oob_auc,100,0.715167591737123,0.71286824374569,0.716852012704592,0.716561203894474 -bird,bird,south,,,AMRO,oob_calibration_slope,100,0.926837573631262,0.897339361283655,0.960759916528504,0.953623540744284 -bird,bird,south,,,YEWA,oob_calibration_slope,100,0.857604141687727,0.843443165747965,0.875941675070625,0.871741212363925 -bird,bird,south,,,AMRO,oob_deviance_explained,100,0.0970655203501608,0.0907384885890809,0.0989923781381314,0.0984328670162571 -bird,bird,south,,,YEWA,oob_deviance_explained,100,0.200176382933312,0.196725767352594,0.206420626511906,0.208119807948008 -bird,bird,south,,,AMRO,oob_n,100,67169,67166,67172,67167 -bird,bird,south,,,YEWA,oob_n,100,67169,67166,67172,67167 -bird,bird,south,,,AMRO,oob_prevalence,100,0.26025011165699,0.25952629071637,0.260926785729864,0.260708383581223 -bird,bird,south,,,YEWA,oob_prevalence,100,0.17548143131411,0.17495262154107,0.175945404105742,0.175741063319785 -bird,bird,south,,,AMRO,oob_rmse,100,0.735441343525441,0.733487115915094,0.737659872237311,0.734395013830007 -bird,bird,south,,,YEWA,oob_rmse,100,0.638166771124373,0.636266196981271,0.640144594296377,0.636772974666958 -bird,bird,south,,,AMRO,oob_spearman,100,0.290675252027123,0.288615904926869,0.292588132435043,0.291376773818019 -bird,bird,south,,,YEWA,oob_spearman,100,0.291825737345894,0.288897993957802,0.294289916161369,0.294073915420202 -bryophyte,bryophyte,north,,,Bryum.All,insample_auc,100,0.642318444539308,0.633921086630186,0.651884158043418,0.640680298981408 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_auc,100,0.689919413502033,0.681548570015025,0.69857719393943,0.687411866264163 -bryophyte,bryophyte,north,,,Bryum.All,insample_calibration_slope,100,0.955325651891997,0.921784876414541,0.994289263502767,0.994238258040669 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_calibration_slope,100,0.982104855078527,0.956074728361977,1.01367711616798,1.00727116584583 -bryophyte,bryophyte,north,,,Bryum.All,insample_deviance_explained,100,0.0475926557100003,0.0429069337919808,0.053599259247204,0.0473841241891654 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_deviance_explained,100,0.0838806697287246,0.0766425254757005,0.0918192109321069,0.0799213091698396 -bryophyte,bryophyte,north,,,Bryum.All,insample_n,100,3072,3037,3101.1,4852 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_n,100,3065,3034.8,3098.1,4852 -bryophyte,bryophyte,north,,,Bryum.All,insample_prevalence,100,0.399349043162305,0.393690846676296,0.405271234894856,0.400041220115416 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_prevalence,100,0.441423922595881,0.434027652325691,0.448036090018821,0.441055234954658 -bryophyte,bryophyte,north,,,Bryum.All,insample_rmse,100,0.474452320597645,0.472245590342395,0.476430126142195,0.474777494875911 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_rmse,100,0.468134439224328,0.465591168622141,0.470778976868538,0.469374241782048 -bryophyte,bryophyte,north,,,Bryum.All,insample_spearman,100,0.241880966381948,0.227489701074908,0.257061610015512,0.238746523656073 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_spearman,100,0.326942501811839,0.312732952582419,0.341695266915511,0.322343360373914 -bryophyte,bryophyte,north,,,Bryum.All,oob_auc,99,0.62017084478022,0.603856982933903,0.633969290458781, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_auc,99,0.672728837382749,0.659997052859653,0.686897759352686, -bryophyte,bryophyte,north,,,Bryum.All,oob_calibration_slope,99,0.779099220956986,0.66468412261174,0.917938092422145, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_calibration_slope,99,0.892217996848082,0.799264221532109,0.998536704936227, -bryophyte,bryophyte,north,,,Bryum.All,oob_deviance_explained,99,0.028633074039904,0.0171610604129409,0.0392434042960176, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_deviance_explained,99,0.0647350614834209,0.0515925692157936,0.0785472562171148, -bryophyte,bryophyte,north,,,Bryum.All,oob_n,100,1780,1750.9,1815,0 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_n,100,1787,1753.9,1817.2,0 -bryophyte,bryophyte,north,,,Bryum.All,oob_prevalence,99,0.401248581157775,0.390844052298067,0.411007337376053, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_prevalence,99,0.440387906446092,0.428865082471199,0.453160994868052, -bryophyte,bryophyte,north,,,Bryum.All,oob_rmse,99,0.480513128479088,0.476615100187253,0.484954013021498, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_rmse,99,0.47387539884342,0.469269889512006,0.478829523309486, -bryophyte,bryophyte,north,,,Bryum.All,oob_spearman,99,0.204292949982475,0.175952087053534,0.227155326132657, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_spearman,99,0.297224506192767,0.274722444418865,0.321893706579838, -bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Climate,99,0.552916481913235,0.540226254499829,0.567241677635273, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Climate,99,0.646423397353534,0.635163324971908,0.658730585161026, -bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Climate_Truncated,99,0.552925779182699,0.540226609429324,0.567255933484287, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Climate_Truncated,99,0.646430101039444,0.635179589567065,0.658754818805508, -bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Full,99,0.622515284380487,0.606966046632109,0.637941781112142, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Full,99,0.680703511329967,0.668408531167835,0.695661297003528, -bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Full_Joint,99,0.620799473715077,0.605648460579975,0.636990572220664, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Full_Joint,99,0.677589700394554,0.664504947148201,0.693024309187494, -bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Full_Joint_Truncated,99,0.621996503907548,0.606257564993021,0.637854921013833, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Full_Joint_Truncated,99,0.677695162032219,0.664672623978323,0.693261314402482, -bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Full_Truncated,99,0.623158079831194,0.607536063147181,0.638923145728668, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Full_Truncated,99,0.681193862566742,0.668549373133085,0.695968955838593, -bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Landcover,99,0.578478654378855,0.563340215848909,0.595416571811786, -bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Landcover,99,0.616438734578608,0.601823118907835,0.630341259705069, -bryophyte,bryophyte,north,,,Bryum.All,v2val_Climate,100,0.564449154820466,0.554012171366573,0.570488080028572,0.563268568361301 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Climate,100,0.653628249222615,0.646582072273794,0.660645471865441,0.652571065182321 -bryophyte,bryophyte,north,,,Bryum.All,v2val_Climate_Truncated,100,0.564434908994878,0.553999088676995,0.57047860242542,0.563263005953167 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Climate_Truncated,100,0.653631181256494,0.646567181729957,0.660656834468788,0.652577122144193 -bryophyte,bryophyte,north,,,Bryum.All,v2val_Full,100,0.639513842289103,0.630384469487024,0.648716294484894,0.638127846001886 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Full,100,0.697312704147698,0.689101809484888,0.706500159400666,0.694675016541229 -bryophyte,bryophyte,north,,,Bryum.All,v2val_Full_Joint,100,0.638157016545815,0.628828304666193,0.647302051967581,0.63656986211763 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Full_Joint,100,0.693548409915653,0.685028144012786,0.703341076889968,0.69133661745651 -bryophyte,bryophyte,north,,,Bryum.All,v2val_Full_Joint_Truncated,100,0.63900274875512,0.628781781025239,0.648591404080496,0.637500263262641 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Full_Joint_Truncated,100,0.693647343658095,0.685257034827932,0.703395393409226,0.69158473589171 -bryophyte,bryophyte,north,,,Bryum.All,v2val_Full_Truncated,100,0.63997798915985,0.63067647736737,0.650065196096129,0.639020549706553 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Full_Truncated,100,0.697442938749871,0.689332998208995,0.706642286722351,0.694914692057453 -bryophyte,bryophyte,north,,,Bryum.All,v2val_Landcover,100,0.591982008865627,0.583299565240258,0.602597071514015,0.59031873097319 -bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Landcover,100,0.628211583853316,0.619312449475028,0.637212297876016,0.626159436771152 -bryophyte,bryophyte,south,,,Bryum.All,insample_auc,100,0.665806112785856,0.653994723957733,0.676094367539239,0.664235544228618 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_auc,100,0.744712111382323,0.736661862126833,0.756533309970613,0.744223004366669 -bryophyte,bryophyte,south,,,Bryum.All,insample_calibration_slope,100,0.963895419689307,0.909117948595872,1.01600769727917,0.994227632577823 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_calibration_slope,100,0.958632660333046,0.922837353048757,1.0032102926837,0.984379677252375 -bryophyte,bryophyte,south,,,Bryum.All,insample_deviance_explained,100,0.060647418310011,0.0519172518787997,0.0685880593571607,0.0582058232553434 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_deviance_explained,100,0.134087667715628,0.126054104059364,0.147661710838995,0.133761560466983 -bryophyte,bryophyte,south,,,Bryum.All,insample_n,100,1461,1441,1483.1,2310 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_n,100,1462,1439.9,1485.5,2310 -bryophyte,bryophyte,south,,,Bryum.All,insample_prevalence,100,0.37019021854078,0.360595240622412,0.379679386889781,0.370562770562771 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_prevalence,100,0.23977643361205,0.230206220419202,0.24915207746165,0.23982683982684 -bryophyte,bryophyte,south,,,Bryum.All,insample_rmse,100,0.463823679787936,0.460173033764873,0.4672287398531,0.464783812560398 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_rmse,100,0.395176481470325,0.389074942964999,0.400757094795644,0.395821890229672 -bryophyte,bryophyte,south,,,Bryum.All,insample_spearman,100,0.276125986422118,0.257199859142186,0.294584764044912,0.274767526720997 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_spearman,100,0.362228960302291,0.350062231973397,0.379817377754883,0.361229177029678 -bryophyte,bryophyte,south,,,Bryum.All,oob_auc,99,0.641746613175185,0.621549873118513,0.661902477499436, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_auc,99,0.729595755352832,0.707659975120298,0.746574925136452, -bryophyte,bryophyte,south,,,Bryum.All,oob_calibration_slope,99,0.799638025660274,0.64705136717253,0.946234221472978, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_calibration_slope,99,0.871686323040223,0.772018187900515,1.01323238589723, -bryophyte,bryophyte,south,,,Bryum.All,oob_deviance_explained,99,0.0358645687227639,0.0186324634218165,0.050802278597151, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_deviance_explained,99,0.110691969317517,0.0859500240961407,0.128368392600255, -bryophyte,bryophyte,south,,,Bryum.All,oob_n,100,849,826.9,869,0 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_n,100,848,824.5,870.1,0 -bryophyte,bryophyte,south,,,Bryum.All,oob_prevalence,99,0.371203599550056,0.355314877874469,0.387708127053306, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_prevalence,99,0.24,0.22333963288802,0.255865292031658, -bryophyte,bryophyte,south,,,Bryum.All,oob_rmse,99,0.471159298712749,0.464486718714448,0.478079898398807, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_rmse,99,0.401778979201565,0.392569891777703,0.412481107813987, -bryophyte,bryophyte,south,,,Bryum.All,oob_spearman,99,0.239111146291676,0.202887915299472,0.272642778009816, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_spearman,99,0.337919402029935,0.305482988407238,0.364596471722889, -bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Climate,99,0.552916481913235,0.540226254499829,0.567241677635273, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Climate,99,0.646423397353534,0.635163324971908,0.658730585161026, -bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Climate_Truncated,99,0.552925779182699,0.540226609429324,0.567255933484287, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Climate_Truncated,99,0.646430101039444,0.635179589567065,0.658754818805508, -bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Full,99,0.619011406844107,0.598240308408525,0.638566231844498, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Full,99,0.70961611965701,0.685563349905508,0.730088189467665, -bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Full_Joint,99,0.618268164367979,0.598732723008374,0.639880025973981, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Full_Joint,99,0.710644087069526,0.688159303290908,0.730668435774751, -bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Full_Joint_Truncated,99,0.618268164367979,0.598668784662836,0.639894643627021, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Full_Joint_Truncated,99,0.710644087069526,0.688159303290908,0.730668435774751, -bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Full_Truncated,99,0.619011406844107,0.598240019402878,0.63877582946581, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Full_Truncated,99,0.70961611965701,0.685563349905508,0.730088189467665, -bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Landcover,99,0.591609023285703,0.5684817431571,0.613703223028674, -bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Landcover,99,0.624022482893451,0.601386928063719,0.645259704463817, -bryophyte,bryophyte,south,,,Bryum.All,v2val_Climate,100,0.564449154820466,0.554012171366573,0.570488080028572,0.563268568361301 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Climate,100,0.653628249222615,0.646582072273794,0.660645471865441,0.652571065182321 -bryophyte,bryophyte,south,,,Bryum.All,v2val_Climate_Truncated,100,0.564434908994878,0.553999088676995,0.57047860242542,0.563263005953167 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Climate_Truncated,100,0.653631181256494,0.646567181729957,0.660656834468788,0.652577122144193 -bryophyte,bryophyte,south,,,Bryum.All,v2val_Full,100,0.633431695701558,0.621072760304728,0.644071139511728,0.631477859980203 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Full,100,0.717588984872617,0.707940136120907,0.729990413520888,0.717485896729522 -bryophyte,bryophyte,south,,,Bryum.All,v2val_Full_Joint,100,0.633675239217769,0.621111812013989,0.644079463996898,0.630846343956086 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Full_Joint,100,0.719214027389327,0.70812487919115,0.732006695871702,0.718564200718732 -bryophyte,bryophyte,south,,,Bryum.All,v2val_Full_Joint_Truncated,100,0.633706694184747,0.621120911195063,0.644107658819543,0.630860002699611 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Full_Joint_Truncated,100,0.719214027389327,0.70812487919115,0.732006695871702,0.718564200718732 -bryophyte,bryophyte,south,,,Bryum.All,v2val_Full_Truncated,100,0.633501295313286,0.621085499158231,0.644081810530497,0.631535708776305 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Full_Truncated,100,0.717588984872617,0.707940136120907,0.729990413520888,0.717485896729522 -bryophyte,bryophyte,south,,,Bryum.All,v2val_Landcover,100,0.604108905134766,0.590811249395407,0.614816041548742,0.604777025029246 -bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Landcover,100,0.632138458386798,0.620856342049895,0.64588084751364,0.630393575816386 -lichen,lichen,north,,,Cladonia.chlorophaea,insample_auc,100,0.769904413128653,0.764063950185549,0.774994859530566,0.769639340981023 -lichen,lichen,north,,,Physcia.adscendens,insample_auc,100,0.843858472001524,0.839627119885778,0.849457973616683,0.843598873256081 -lichen,lichen,north,,,Cladonia.chlorophaea,insample_calibration_slope,100,0.99506569309268,0.975357328535013,1.00788600252369,1.00264901740429 -lichen,lichen,north,,,Physcia.adscendens,insample_calibration_slope,100,1.0023434293955,0.992290677679192,1.0144227094458,1.0054450276181 -lichen,lichen,north,,,Cladonia.chlorophaea,insample_deviance_explained,100,0.189091223352288,0.181672074359276,0.19571373866611,0.188532571921241 -lichen,lichen,north,,,Physcia.adscendens,insample_deviance_explained,100,0.29252733374457,0.28393309923068,0.30281084569531,0.291544396783705 -lichen,lichen,north,,,Cladonia.chlorophaea,insample_n,100,3676.5,3643.5,3708.1,5811 -lichen,lichen,north,,,Physcia.adscendens,insample_n,100,3672,3638.7,3709,5811 -lichen,lichen,north,,,Cladonia.chlorophaea,insample_prevalence,100,0.384437187559081,0.380547590781089,0.390534077268186,0.384787472035794 -lichen,lichen,north,,,Physcia.adscendens,insample_prevalence,100,0.384489533269728,0.377838704868109,0.390009815932809,0.384099122354156 -lichen,lichen,north,,,Cladonia.chlorophaea,insample_rmse,100,0.431560366260102,0.429640350605955,0.433524488609716,0.431642242867297 -lichen,lichen,north,,,Physcia.adscendens,insample_rmse,100,0.38926940084163,0.386109979689022,0.392548447467983,0.38952477075237 -lichen,lichen,north,,,Cladonia.chlorophaea,insample_spearman,100,0.454700594463317,0.444291297886564,0.464217347731443,0.454461373310586 -lichen,lichen,north,,,Physcia.adscendens,insample_spearman,100,0.579034123424778,0.572457031426613,0.588107541175059,0.57892123989803 -lichen,lichen,north,,,Cladonia.chlorophaea,oob_auc,99,0.758701523142145,0.751167957205167,0.77037258371695, -lichen,lichen,north,,,Physcia.adscendens,oob_auc,99,0.836309233023938,0.826949725892923,0.844910528306798, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_calibration_slope,99,0.954267136908917,0.906613172943602,1.01012439935484, -lichen,lichen,north,,,Physcia.adscendens,oob_calibration_slope,99,0.983916249515356,0.942689284867629,1.01579271399548, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_deviance_explained,99,0.171702415001063,0.160902175054887,0.185575056097344, -lichen,lichen,north,,,Physcia.adscendens,oob_deviance_explained,99,0.27675887709098,0.255216215590799,0.292828895189064, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_n,100,2134.5,2102.9,2167.5,0 -lichen,lichen,north,,,Physcia.adscendens,oob_n,100,2139,2102,2172.3,0 -lichen,lichen,north,,,Cladonia.chlorophaea,oob_prevalence,99,0.38539741219963,0.374872643000928,0.392180977534085, -lichen,lichen,north,,,Physcia.adscendens,oob_prevalence,99,0.383356708742403,0.373863819973773,0.394833611223034, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_rmse,99,0.435804948437572,0.431963537434734,0.43930758296454, -lichen,lichen,north,,,Physcia.adscendens,oob_rmse,99,0.393757557408559,0.387508441417089,0.40047991779162, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_spearman,99,0.436467740943861,0.424128371825953,0.456602087912647, -lichen,lichen,north,,,Physcia.adscendens,oob_spearman,99,0.567387904302673,0.549537429654247,0.581886454163815, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Climate,99,0.720918773373224,0.712650560435026,0.728579673956133, -lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Climate,99,0.689144635427072,0.680873381791638,0.698761438702953, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Climate_Truncated,99,0.720930740463725,0.712658541338133,0.728614575624258, -lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Climate_Truncated,99,0.689133844867517,0.680847879279384,0.698755518234618, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Full,99,0.769142363003573,0.760363798954931,0.781551514547456, -lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Full,99,0.834569321101047,0.825318195008198,0.843996747262564, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Full_Joint,99,0.768152331423773,0.759812672712765,0.780189734262456, -lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Full_Joint,99,0.833778617163388,0.821731526307177,0.841857565835814, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Full_Joint_Truncated,99,0.768049343702181,0.759794968553464,0.780225490484061, -lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Full_Joint_Truncated,99,0.833901297402281,0.821793123470401,0.841904293589348, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Full_Truncated,99,0.769375783962666,0.760288655670733,0.781582641288071, -lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Full_Truncated,99,0.834936696506996,0.825421706977217,0.844055449907669, -lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Landcover,99,0.707003638756629,0.694851661808356,0.720077223146866, -lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Landcover,99,0.806874249599937,0.78887752964357,0.820974567104598, -lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Climate,100,0.7235281228999,0.717441862095723,0.729839575280893,0.723004804073768 -lichen,lichen,north,,,Physcia.adscendens,v2val_Climate,100,0.693903155154763,0.68873689780885,0.698042527967169,0.693422907498 -lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Climate_Truncated,100,0.723538229483672,0.717445780953459,0.729849398789947,0.72301652132686 -lichen,lichen,north,,,Physcia.adscendens,v2val_Climate_Truncated,100,0.693897896800193,0.688717298999803,0.698042748005106,0.693419511127206 -lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Full,100,0.776586031004036,0.769782653235173,0.781235452865446,0.776659932196605 -lichen,lichen,north,,,Physcia.adscendens,v2val_Full,100,0.838564851497925,0.833625144582723,0.845039678547128,0.838268220834197 -lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Full_Joint,100,0.775847920853584,0.769071476627846,0.780273084489337,0.77578324180292 -lichen,lichen,north,,,Physcia.adscendens,v2val_Full_Joint,100,0.836617198875312,0.832065573684458,0.842958635790047,0.836357683360022 -lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Full_Joint_Truncated,100,0.775630433031142,0.768905297903972,0.78063749666941,0.775630371417491 -lichen,lichen,north,,,Physcia.adscendens,v2val_Full_Joint_Truncated,100,0.836790613008269,0.832110390227703,0.843010056321518,0.836525866238843 -lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Full_Truncated,100,0.776419861818452,0.769668394570003,0.781498799112205,0.77651231594881 -lichen,lichen,north,,,Physcia.adscendens,v2val_Full_Truncated,100,0.838676134749758,0.833749257931369,0.845128664248426,0.838439032548488 -lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Landcover,100,0.709593137989017,0.701801434200798,0.71574025781388,0.710689730662897 -lichen,lichen,north,,,Physcia.adscendens,v2val_Landcover,100,0.80961766255631,0.788306510386797,0.821112870692013,0.812994033294577 -lichen,lichen,south,,,Cladonia.chlorophaea,insample_auc,100,0.830525429515063,0.81604844430429,0.841787762025551,0.828408463820101 -lichen,lichen,south,,,Physcia.adscendens,insample_auc,100,0.841765750148897,0.835332420212302,0.851818483932403,0.841173848599416 -lichen,lichen,south,,,Cladonia.chlorophaea,insample_calibration_slope,100,0.982972841611774,0.933529016241715,1.02605515147802,1.01467814581042 -lichen,lichen,south,,,Physcia.adscendens,insample_calibration_slope,100,0.98414680389702,0.949866080740196,1.00735121035812,0.989153221348672 -lichen,lichen,south,,,Cladonia.chlorophaea,insample_deviance_explained,100,0.217596986009839,0.199272844258197,0.233746882017517,0.217566602485613 -lichen,lichen,south,,,Physcia.adscendens,insample_deviance_explained,100,0.274920311445747,0.263973025543899,0.295736386939481,0.274902459396964 -lichen,lichen,south,,,Cladonia.chlorophaea,insample_n,100,1483.5,1457,1509.1,2346 -lichen,lichen,south,,,Physcia.adscendens,insample_n,100,1480.5,1463,1508.1,2346 -lichen,lichen,south,,,Cladonia.chlorophaea,insample_prevalence,100,0.093062874358488,0.0871251477911935,0.0985096364127046,0.092924126172208 -lichen,lichen,south,,,Physcia.adscendens,insample_prevalence,100,0.214722124199446,0.205436679722394,0.223097894761383,0.215686274509804 -lichen,lichen,south,,,Cladonia.chlorophaea,insample_rmse,100,0.27087268791403,0.26414104023987,0.277530954067709,0.271312770803834 -lichen,lichen,south,,,Physcia.adscendens,insample_rmse,100,0.349408213370138,0.341231568382864,0.355156444882723,0.349756738076147 -lichen,lichen,south,,,Cladonia.chlorophaea,insample_spearman,100,0.331663035664903,0.312174742385052,0.347856252239412,0.330286991594626 -lichen,lichen,south,,,Physcia.adscendens,insample_spearman,100,0.486225737344946,0.475186496857954,0.501475433694051,0.48609712754117 -lichen,lichen,south,,,Cladonia.chlorophaea,oob_auc,99,0.799970484061393,0.775333716499924,0.821361725248609, -lichen,lichen,south,,,Physcia.adscendens,oob_auc,99,0.832216295609153,0.812918356886962,0.84383834193707, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_calibration_slope,99,0.83226582711674,0.687252862288971,1.04162320779769, -lichen,lichen,south,,,Physcia.adscendens,oob_calibration_slope,99,0.933315477460017,0.84750745834188,1.03678504002113, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_deviance_explained,99,0.168367974657616,0.118832513303203,0.197439279118203, -lichen,lichen,south,,,Physcia.adscendens,oob_deviance_explained,99,0.250569721574222,0.213912803567623,0.275038264461662, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_n,100,862.5,836.9,889,0 -lichen,lichen,south,,,Physcia.adscendens,oob_n,100,865.5,837.9,883,0 -lichen,lichen,south,,,Cladonia.chlorophaea,oob_prevalence,99,0.0926553672316384,0.0833333333333333,0.102941033138401, -lichen,lichen,south,,,Physcia.adscendens,oob_prevalence,99,0.217582417582418,0.203154716242863,0.232991265333397, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_rmse,99,0.276805383506645,0.265439259208283,0.289284420169747, -lichen,lichen,south,,,Physcia.adscendens,oob_rmse,99,0.35779053373906,0.346088831243775,0.368235479879014, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_spearman,99,0.304103065146685,0.271871702578512,0.32936871363484, -lichen,lichen,south,,,Physcia.adscendens,oob_spearman,99,0.473517262633272,0.443141256672266,0.491064816140324, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Climate,99,0.720918773373224,0.712650560435026,0.728579673956133, -lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Climate,99,0.689144635427072,0.680873381791638,0.698761438702953, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Climate_Truncated,99,0.720930740463725,0.712658541338133,0.728614575624258, -lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Climate_Truncated,99,0.689133844867517,0.680847879279384,0.698755518234618, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Full,99,0.769192913385827,0.740816145064209,0.792152407634443, -lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Full,99,0.814451917130489,0.796076806462607,0.831527942532319, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Full_Joint,99,0.769436743032249,0.744011243283277,0.793514941384641, -lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Full_Joint,99,0.81242151354924,0.791924831211599,0.828123603004887, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Full_Joint_Truncated,99,0.769436743032249,0.744011243283277,0.793514941384641, -lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Full_Joint_Truncated,99,0.81242151354924,0.791924831211599,0.828123603004887, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Full_Truncated,99,0.769192913385827,0.740816145064209,0.792152407634443, -lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Full_Truncated,99,0.814451917130489,0.796076806462607,0.831534146727906, -lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Landcover,99,0.738834202254804,0.70857488379101,0.767503416567657, -lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Landcover,99,0.759559552987862,0.73362889344165,0.776333033950638, -lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Climate,100,0.7235281228999,0.717441862095723,0.729839575280893,0.723004804073768 -lichen,lichen,south,,,Physcia.adscendens,v2val_Climate,100,0.693903155154763,0.68873689780885,0.698042527967169,0.693422907498 -lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Climate_Truncated,100,0.723538229483672,0.717445780953459,0.729849398789947,0.72301652132686 -lichen,lichen,south,,,Physcia.adscendens,v2val_Climate_Truncated,100,0.693897896800193,0.688717298999803,0.698042748005106,0.693419511127206 -lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Full,100,0.781684042308046,0.765640384141264,0.797179703714229,0.782940004828585 -lichen,lichen,south,,,Physcia.adscendens,v2val_Full,100,0.821831844343548,0.812065145466866,0.832981098960484,0.821573724007562 -lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Full_Joint,100,0.782929579158528,0.767421869497933,0.797927045706971,0.783724649927571 -lichen,lichen,south,,,Physcia.adscendens,v2val_Full_Joint,100,0.818461282378356,0.808982334731472,0.829948045116519,0.818413816807012 -lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Full_Joint_Truncated,100,0.782929579158528,0.767421869497933,0.797927045706971,0.783724649927571 -lichen,lichen,south,,,Physcia.adscendens,v2val_Full_Joint_Truncated,100,0.818465382979777,0.808982334731472,0.829948045116519,0.818413816807012 -lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Full_Truncated,100,0.781684042308046,0.765640384141264,0.797179703714229,0.782940004828585 -lichen,lichen,south,,,Physcia.adscendens,v2val_Full_Truncated,100,0.821831844343548,0.812065145466866,0.832981098960484,0.821573724007562 -lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Landcover,100,0.746462990180373,0.72844374826332,0.760216371643953,0.74708991515486 -lichen,lichen,south,,,Physcia.adscendens,v2val_Landcover,100,0.757941670475573,0.744902995002577,0.775307203498065,0.755514263619179 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_auc,100,0.823632636160737,0.814781230564576,0.831408128980217,0.820911148938545 -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_auc,100,0.61475601886175,0.603679345102737,0.627353730564579,0.614800455189874 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_calibration_slope,100,0.974719951819278,0.940060183795489,1.01331111594891,0.987210797047211 -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_calibration_slope,100,0.847565289100029,0.777129508768977,0.945716349335048,0.901041907949805 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_deviance_explained,100,0.279791843738347,0.261697566054768,0.292547470573807,0.272167552780268 -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_deviance_explained,100,0.032556283657145,0.0243296542272488,0.039523255842162,0.0327130199155846 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_n,100,2564.5,2538.9,2588.1,4053 -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_n,100,2562.5,2537.8,2585.2,4053 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_prevalence,100,0.225366262673778,0.220070866141732,0.230634088644114,0.22526523562793 -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_prevalence,100,0.321224265570394,0.314927013046706,0.329608505982353,0.321243523316062 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_rmse,100,0.325947657311819,0.321499471172476,0.331112157543476,0.327783381237672 -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_rmse,100,0.440780068007451,0.437909796174194,0.443705369273563,0.440494520447441 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_spearman,100,0.462005079284434,0.44910358192168,0.473619794940904,0.457622037926128 -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_spearman,100,0.169333701727882,0.148930684100538,0.188146271759874,0.166867112583273 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_auc,99,0.808896640826873,0.79100198897739,0.827548123693069, -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_auc,99,0.59059424536098,0.573979407712817,0.607160496358349, -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_calibration_slope,99,0.93389114249429,0.850341206695487,1.01749958453456, -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_calibration_slope,99,0.643049641195582,0.517287601824814,0.821946111710213, -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_deviance_explained,99,0.243486713148957,0.193079357602948,0.27747427555585, -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_deviance_explained,99,0.0121132026875843,-0.00570566280736071,0.0241511023697404, -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_n,100,1488.5,1464.9,1514.1,0 -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_n,100,1490.5,1467.8,1515.2,0 -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_prevalence,99,0.224915824915825,0.215935838563886,0.234336783397599, -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_prevalence,99,0.321309285237141,0.306674120479106,0.332295711996362, -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_rmse,99,0.331823647228338,0.323085219723366,0.340284753308317, -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_rmse,99,0.445175541404113,0.440470485230338,0.448886295412126, -mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_spearman,99,0.4394124176263,0.417905127967147,0.465739312197084, -mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_spearman,99,0.130783814760799,0.101891242189749,0.155800550653179, -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_auc,99,0.635001427076396,0.606104262619653,0.657438718250968,0.634240369366988 -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_auc,100,0.786511166542812,0.769504152294664,0.804093618520575,0.786049924840508 -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_calibration_slope,99,0.913890378466184,0.820802523135607,1.01427035726621,1.00531761653027 -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_calibration_slope,100,0.984182683497037,0.904891334217879,1.06778036362132,0.986324497785332 -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_deviance_explained,99,0.026440006660178,0.0180120188346803,0.0396931836083979,0.0264112493368064 -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_deviance_explained,100,0.21040174777469,0.184564736775399,0.242108898186897,0.205216892616348 -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_n,99,675,663,685.4,1065 -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_n,100,674,660.9,691.1,1065 -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_prevalence,99,0.690265486725664,0.677825953587333,0.705306871830739,0.691079812206573 -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_prevalence,100,0.182355333743735,0.168834754036547,0.195212578714749,0.182159624413146 -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_rmse,99,0.421280362710698,0.414044874992987,0.426531332439636,0.421286846656193 -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_rmse,100,0.343613620192013,0.334680149069424,0.351940369457748,0.345153854253612 -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_spearman,99,0.180460025029286,0.146567810345527,0.214594440680258,0.175369074792115 -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_spearman,100,0.38143205353404,0.353940396364084,0.410801187004518,0.381089371361329 -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_auc,98,0.591021454838246,0.551176084882234,0.62061474225386, -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_auc,99,0.766688006828852,0.728742672765759,0.797692233844295, -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_calibration_slope,98,0.602514054060098,0.232115218957736,0.865596324101407, -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_calibration_slope,99,0.887224345060846,0.665784227384331,1.20908224079875, -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_deviance_explained,98,-0.000836625554455406,-0.0331073359910842,0.015317940018266, -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_deviance_explained,99,0.160410765184501,-0.0354375437473713,0.21724314670011, -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_n,99,390,379.6,402,0 -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_n,100,391,373.9,404.1,0 -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_prevalence,98,0.69250645994832,0.665551467442457,0.71439588688946, -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_prevalence,99,0.181578947368421,0.160028119078184,0.205617057719676, -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_rmse,98,0.429820967393949,0.42171561998268,0.442398381954269, -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_rmse,99,0.350823352193687,0.337300969232687,0.368279364518069, -mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_spearman,98,0.110037755841753,0.0541619409153014,0.160705561767885, -mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_spearman,99,0.356733670694178,0.298765552548449,0.401384960970594, -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_auc,100,0.840215938467687,0.83092115117171,0.849787411243392,0.839440907842931 -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_auc,100,0.67821310892138,0.663270440192927,0.695381764999782,0.668767750495832 -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_calibration_slope,100,0.972671886418681,0.93201508456542,1.02152533613636,0.982050704865843 -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_calibration_slope,100,0.917861855484528,0.84216777747185,0.991483856313351,0.94812085347111 -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_deviance_explained,100,0.295983987195951,0.279264150591629,0.313078108968483,0.294409607036222 -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_deviance_explained,100,0.0786332530404745,0.0665566425621889,0.0944247372062849,0.0713753243188524 -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_n,100,2609,2585,2637.1,4127 -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_n,100,2610,2586.7,2642.1,4127 -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_prevalence,100,0.201888432073211,0.197431539212855,0.208159731764965,0.201356917858008 -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_prevalence,100,0.149990410433449,0.143887902486318,0.155025771607528,0.148776350860189 -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_rmse,100,0.300488510882096,0.296257654339334,0.306287986982839,0.30069616131153 -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_rmse,100,0.332560139164442,0.327709275223946,0.336999200640023,0.332910274284271 -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_spearman,100,0.464310589597897,0.45159625601271,0.478558931582411,0.463240577826878 -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_spearman,100,0.223831736114593,0.205313082589125,0.244481495615383,0.211955551417089 -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_auc,99,0.83021391101824,0.813671018298189,0.846159487988212, -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_auc,99,0.650339977044136,0.626283007371369,0.674110216571647, -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_calibration_slope,99,0.943994670763352,0.852757151203258,1.04492756918031, -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_calibration_slope,99,0.765031662278474,0.564319012355523,0.978273019865697, -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_deviance_explained,99,0.273408193489382,0.229352594121753,0.302275051480294, -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_deviance_explained,99,0.0437348310014123,-0.0244103779855468,0.070888637861963, -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_n,100,1518,1489.9,1542,0 -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_n,100,1517,1484.9,1540.3,0 -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_prevalence,99,0.200259235255995,0.189775542999326,0.208169005020511, -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_prevalence,99,0.146603900470746,0.138208405767461,0.157249189724539, -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_rmse,99,0.304113428131756,0.295100478568192,0.311362926507058, -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_rmse,99,0.335213172770143,0.328331589790136,0.342856337727437, -mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_spearman,99,0.448324145179786,0.427614830976871,0.47442464641456, -mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_spearman,99,0.189218625360862,0.156934768956714,0.217963256910742, -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_auc,100,0.691441153027097,0.672240387183235,0.706212764410456,0.689082366331686 -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_auc,100,0.820968430916565,0.801436993738574,0.84112134496232,0.819927852628077 -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_calibration_slope,100,0.992627010851258,0.920219152025468,1.09520751465362,1.03306130883064 -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_calibration_slope,100,0.883711120332416,0.748929633856669,1.03639244285071,0.926166715024209 -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_deviance_explained,100,0.067567099805677,0.0560696673854207,0.0822942032238073,0.0670547674954939 -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_deviance_explained,100,0.239038869889767,0.21013927948378,0.277280840586616,0.234870911391229 -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_n,100,695,682,708.1,1098 -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_n,100,696,681,709.1,1098 -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_prevalence,100,0.603312544835565,0.587842923920052,0.618115052537349,0.604735883424408 -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_prevalence,100,0.0873239436619718,0.0790283819028552,0.0939931596468854,0.0874316939890711 -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_rmse,100,0.40682238205156,0.401865860344406,0.410771399612378,0.406888363248781 -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_rmse,100,0.222369754770973,0.210112146550844,0.231243986964309,0.223331046901821 -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_spearman,100,0.299224586077027,0.269355636443703,0.329699476853509,0.296933239332004 -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_spearman,100,0.312958724627694,0.294454461225426,0.33498073496289,0.313170610278462 -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_auc,99,0.664455665867854,0.639651542814446,0.690741654078452, -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_auc,99,0.783524426719841,0.742644818301232,0.814561651279978, -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_calibration_slope,99,0.817106846468162,0.635134217999263,1.06374115196959, -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_calibration_slope,99,0.631374792805361,0.431072836936144,0.94521686736471, -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_deviance_explained,99,0.0446136452786412,0.00619180902025967,0.0633330011439923, -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_deviance_explained,99,0.154109596533938,-0.020676255891081,0.207600467554891, -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_n,100,403,389.9,416,0 -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_n,100,402,388.9,417,0 -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_prevalence,99,0.6075,0.580323143750253,0.635349582154713, -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_prevalence,99,0.0876288659793814,0.0760837438423646,0.101576468326444, -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_rmse,99,0.413228278046299,0.406653224893685,0.424163637351331, -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_rmse,99,0.232970422089708,0.214829244725729,0.249813879153275, -mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_spearman,99,0.261506753321649,0.206422225163459,0.30115717511071, -mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_spearman,99,0.278122302398705,0.235207424325587,0.312315920243565, -mite,mite,north,,,Ceratozetes.gracilis,insample_auc,100,0.79764960251103,0.790265112142925,0.803014153893967,0.796797763564616 -mite,mite,north,,,Trhypochthonius.tectorum,insample_auc,100,0.695668269661138,0.683311469089793,0.708716981054314,0.693478045026932 -mite,mite,north,,,Ceratozetes.gracilis,insample_calibration_slope,100,0.998797614762679,0.973003562776025,1.02348834676389,1.01060441588758 -mite,mite,north,,,Trhypochthonius.tectorum,insample_calibration_slope,100,0.953819502076839,0.906510887653655,1.02084762209438,0.99979630802942 -mite,mite,north,,,Ceratozetes.gracilis,insample_deviance_explained,100,0.193008737804738,0.181739448301126,0.201661801567595,0.192285651521356 -mite,mite,north,,,Trhypochthonius.tectorum,insample_deviance_explained,100,0.0674511356385436,0.0598971412561373,0.0762925190230293,0.0657438604252465 -mite,mite,north,,,Ceratozetes.gracilis,insample_n,100,3611.5,3576.4,3648.1,5711 -mite,mite,north,,,Trhypochthonius.tectorum,insample_n,100,3611,3575.8,3647.3,5711 -mite,mite,north,,,Ceratozetes.gracilis,insample_prevalence,100,0.208199871528185,0.202330059472997,0.21373147884126,0.208369812642269 -mite,mite,north,,,Trhypochthonius.tectorum,insample_prevalence,100,0.0955235096257067,0.0915774802199236,0.0988495023046604,0.0956049728593941 -mite,mite,north,,,Ceratozetes.gracilis,insample_rmse,100,0.362341821380525,0.357614975130255,0.365783840587183,0.362271480910673 -mite,mite,north,,,Trhypochthonius.tectorum,insample_rmse,100,0.287055116874456,0.282074628131594,0.291589162021195,0.287472688930716 -mite,mite,north,,,Ceratozetes.gracilis,insample_spearman,100,0.418676629729472,0.407119399739678,0.427276208006511,0.417570583307426 -mite,mite,north,,,Trhypochthonius.tectorum,insample_spearman,100,0.199311545833827,0.186622637832808,0.211624444539445,0.197079610929558 -mite,mite,north,,,Ceratozetes.gracilis,oob_auc,99,0.786867707648098,0.774187910909352,0.796514509877961, -mite,mite,north,,,Trhypochthonius.tectorum,oob_auc,99,0.656996441402617,0.636882624100218,0.67929424581142, -mite,mite,north,,,Ceratozetes.gracilis,oob_calibration_slope,99,0.96183303495678,0.883323795220391,1.02731965461652, -mite,mite,north,,,Trhypochthonius.tectorum,oob_calibration_slope,99,0.772962411920347,0.625887214843551,0.971044514733484, -mite,mite,north,,,Ceratozetes.gracilis,oob_deviance_explained,99,0.174074875598022,0.153589237127838,0.189989626088515, -mite,mite,north,,,Trhypochthonius.tectorum,oob_deviance_explained,99,0.0286565486886089,0.00771368081782448,0.0495098764287371, -mite,mite,north,,,Ceratozetes.gracilis,oob_n,100,2099.5,2062.9,2134.6,0 -mite,mite,north,,,Trhypochthonius.tectorum,oob_n,100,2100,2063.7,2135.2,0 -mite,mite,north,,,Ceratozetes.gracilis,oob_prevalence,99,0.208786017949929,0.199496646948345,0.218887953819956, -mite,mite,north,,,Trhypochthonius.tectorum,oob_prevalence,99,0.095769409576941,0.0898898158460644,0.102527927842943, -mite,mite,north,,,Ceratozetes.gracilis,oob_rmse,99,0.365594372577947,0.360562265939922,0.372213007268495, -mite,mite,north,,,Trhypochthonius.tectorum,oob_rmse,99,0.290903335956598,0.28306112008192,0.298263526752633, -mite,mite,north,,,Ceratozetes.gracilis,oob_spearman,99,0.402588005893801,0.38436112841948,0.421432163723525, -mite,mite,north,,,Trhypochthonius.tectorum,oob_spearman,99,0.160614554026014,0.139028864717418,0.181462351625277, -mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Climate,99,0.734956389373006,0.72425367499707,0.745909153373562, -mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Climate,99,0.638342287752377,0.618187226042641,0.657504024919964, -mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Climate_Truncated,99,0.734924300615503,0.724212897038428,0.74588498630922, -mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Climate_Truncated,99,0.638365821429941,0.618184154764979,0.657498616560185, -mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Full,99,0.784719038881003,0.773178754620244,0.795479334633468, -mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Full,99,0.651718898385565,0.636655558271588,0.67451733769942, -mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Full_Joint,99,0.78489164605236,0.773137004390567,0.795379252566855, -mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Full_Joint,99,0.650619319414049,0.636142187917328,0.674471465378823, -mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Full_Joint_Truncated,99,0.784845263863121,0.773124471326533,0.795223511559166, -mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Full_Joint_Truncated,99,0.650697674418605,0.636256531006593,0.674578866782116, -mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Full_Truncated,99,0.784702420108714,0.773175774381691,0.795336105984979, -mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Full_Truncated,99,0.651754510921178,0.636777941613843,0.674583194603189, -mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Landcover,99,0.758805093399336,0.746485924819688,0.771387018903544, -mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Landcover,99,0.612701645627281,0.590187575597952,0.631325206986045, -mite,mite,north,,,Ceratozetes.gracilis,v2val_Climate,100,0.737438837044945,0.73108563444224,0.745220440466574,0.738382414749567 -mite,mite,north,,,Trhypochthonius.tectorum,v2val_Climate,100,0.646399889389764,0.635927059979148,0.656681007348258,0.646091585413555 -mite,mite,north,,,Ceratozetes.gracilis,v2val_Climate_Truncated,100,0.737426078766032,0.731081895217831,0.745228984913048,0.738392463572007 -mite,mite,north,,,Trhypochthonius.tectorum,v2val_Climate_Truncated,100,0.646412518420192,0.635932057855205,0.656705753288746,0.646095557008706 -mite,mite,north,,,Ceratozetes.gracilis,v2val_Full,100,0.792129407675089,0.784999351238186,0.798007653851505,0.791183069113511 -mite,mite,north,,,Trhypochthonius.tectorum,v2val_Full,100,0.678069892111616,0.668115642289897,0.691570135316744,0.669778092188547 -mite,mite,north,,,Ceratozetes.gracilis,v2val_Full_Joint,100,0.792319179894978,0.785107596237888,0.798106897210031,0.791225262500488 -mite,mite,north,,,Trhypochthonius.tectorum,v2val_Full_Joint,100,0.677875105270547,0.667971879477692,0.691152263754938,0.66926215120794 -mite,mite,north,,,Ceratozetes.gracilis,v2val_Full_Joint_Truncated,100,0.792270646461233,0.784996783953025,0.798085250808367,0.7911589984368 -mite,mite,north,,,Trhypochthonius.tectorum,v2val_Full_Joint_Truncated,100,0.677978875377472,0.668016715774074,0.691209732839322,0.669242116386356 -mite,mite,north,,,Ceratozetes.gracilis,v2val_Full_Truncated,100,0.792086330384434,0.784902829890835,0.797978154154108,0.791105466738786 -mite,mite,north,,,Trhypochthonius.tectorum,v2val_Full_Truncated,100,0.67819925700999,0.668189835369413,0.691638395354139,0.669767986128102 -mite,mite,north,,,Ceratozetes.gracilis,v2val_Landcover,100,0.766528013047557,0.758672408700658,0.774519100120648,0.766722893536977 -mite,mite,north,,,Trhypochthonius.tectorum,v2val_Landcover,100,0.647813214614084,0.635162138711342,0.660710676806236,0.637987440117159 -mite,mite,south,,,Ceratozetes.gracilis,insample_auc,100,0.892707050359217,0.880490910244845,0.908212907848489,0.890997673556088 -mite,mite,south,,,Trhypochthonius.tectorum,insample_auc,100,0.724867191947384,0.70852165349298,0.745317546849153,0.723973639682964 -mite,mite,south,,,Ceratozetes.gracilis,insample_calibration_slope,100,1.00502080599402,0.952455083247924,1.09112875870501,1.03680964341723 -mite,mite,south,,,Trhypochthonius.tectorum,insample_calibration_slope,100,0.981215565874312,0.893626222637007,1.09078831127912,1.03282742419112 -mite,mite,south,,,Ceratozetes.gracilis,insample_deviance_explained,100,0.325497099511249,0.297912532981948,0.361127487557951,0.320494419788214 -mite,mite,south,,,Trhypochthonius.tectorum,insample_deviance_explained,100,0.0934320512138448,0.0779654055574169,0.112971811878278,0.0913494137264966 -mite,mite,south,,,Ceratozetes.gracilis,insample_n,100,1494,1472,1519,2363 -mite,mite,south,,,Trhypochthonius.tectorum,insample_n,100,1491,1464.9,1513.1,2363 -mite,mite,south,,,Ceratozetes.gracilis,insample_prevalence,100,0.0560369998468874,0.0512751399543853,0.0605680889804702,0.0562843842573 -mite,mite,south,,,Trhypochthonius.tectorum,insample_prevalence,100,0.0834614879480407,0.0777956447270506,0.0898914722165834,0.0833685992382565 -mite,mite,south,,,Ceratozetes.gracilis,insample_rmse,100,0.199761264937152,0.190846959832617,0.207839536498707,0.200524923589364 -mite,mite,south,,,Trhypochthonius.tectorum,insample_rmse,100,0.268245651768535,0.260181738514282,0.275872677318287,0.267980602329676 -mite,mite,south,,,Ceratozetes.gracilis,insample_spearman,100,0.312194294877045,0.298401375089775,0.326491820416125,0.312161907288093 -mite,mite,south,,,Trhypochthonius.tectorum,insample_spearman,100,0.216574263979227,0.19736592523133,0.235039604757477,0.214479910507598 -mite,mite,south,,,Ceratozetes.gracilis,oob_auc,99,0.866366090050301,0.832423247232638,0.89832578771871, -mite,mite,south,,,Trhypochthonius.tectorum,oob_auc,99,0.681894702096055,0.644529496094018,0.718523652572189, -mite,mite,south,,,Ceratozetes.gracilis,oob_calibration_slope,99,0.870142167861266,0.718742962870896,1.16328663725671, -mite,mite,south,,,Trhypochthonius.tectorum,oob_calibration_slope,99,0.724900423720063,0.47529962795477,1.0104925890134, -mite,mite,south,,,Ceratozetes.gracilis,oob_deviance_explained,99,0.263266494848867,0.170366258386125,0.308407819151291, -mite,mite,south,,,Trhypochthonius.tectorum,oob_deviance_explained,99,0.0302321727646686,-0.036349004988869,0.0683437026947676, -mite,mite,south,,,Ceratozetes.gracilis,oob_n,100,869,844,891,0 -mite,mite,south,,,Trhypochthonius.tectorum,oob_n,100,872,849.9,898.1,0 -mite,mite,south,,,Ceratozetes.gracilis,oob_prevalence,99,0.0567536889897843,0.0492060040243461,0.0647286476351338, -mite,mite,south,,,Trhypochthonius.tectorum,oob_prevalence,99,0.0830489192263936,0.0720677621944575,0.0927861264477562, -mite,mite,south,,,Ceratozetes.gracilis,oob_rmse,99,0.20783366391651,0.195621348914586,0.223198277996607, -mite,mite,south,,,Trhypochthonius.tectorum,oob_rmse,99,0.272200342412914,0.258672733203831,0.285569347368102, -mite,mite,south,,,Ceratozetes.gracilis,oob_spearman,99,0.293951803574799,0.261879056437077,0.321029993144448, -mite,mite,south,,,Trhypochthonius.tectorum,oob_spearman,99,0.175614116696667,0.138478936612199,0.208838587852502, -mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Climate,99,0.734956389373006,0.72425367499707,0.745909153373562, -mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Climate,99,0.638342287752377,0.618187226042641,0.657504024919964, -mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Climate_Truncated,99,0.734924300615503,0.724212897038428,0.74588498630922, -mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Climate_Truncated,99,0.638365821429941,0.618184154764979,0.657498616560185, -mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Full,99,0.876754158004158,0.836903325941546,0.898009878626587, -mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Full,99,0.6853880850931,0.650846584321518,0.723794885344973, -mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Full_Joint,99,0.876374929418408,0.836856202096876,0.898594288869518, -mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Full_Joint,99,0.685599471717562,0.651414735716439,0.724097221093466, -mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Full_Joint_Truncated,99,0.876374929418408,0.836856202096876,0.89863479109714, -mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Full_Joint_Truncated,99,0.685599471717562,0.651414735716439,0.724082791079036, -mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Full_Truncated,99,0.876754158004158,0.836903325941546,0.89813801872643, -mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Full_Truncated,99,0.6853880850931,0.650846584321518,0.723780455330544, -mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Landcover,99,0.709904009034444,0.670283600525517,0.7409295732199, -mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Landcover,99,0.64709784411277,0.613057015806844,0.677195295537004, -mite,mite,south,,,Ceratozetes.gracilis,v2val_Climate,100,0.737438837044945,0.73108563444224,0.745220440466574,0.738382414749567 -mite,mite,south,,,Trhypochthonius.tectorum,v2val_Climate,100,0.646399889389764,0.635927059979148,0.656681007348258,0.646091585413555 -mite,mite,south,,,Ceratozetes.gracilis,v2val_Climate_Truncated,100,0.737426078766032,0.731081895217831,0.745228984913048,0.738392463572007 -mite,mite,south,,,Trhypochthonius.tectorum,v2val_Climate_Truncated,100,0.646412518420192,0.635932057855205,0.656705753288746,0.646095557008706 -mite,mite,south,,,Ceratozetes.gracilis,v2val_Full,100,0.882042472336732,0.867054900665937,0.899986632313541,0.880842240129472 -mite,mite,south,,,Trhypochthonius.tectorum,v2val_Full,100,0.709232811452144,0.690803634887013,0.726193259599038,0.705313075635924 -mite,mite,south,,,Ceratozetes.gracilis,v2val_Full_Joint,100,0.88218958764066,0.867694458860852,0.899921384780916,0.880966991469706 -mite,mite,south,,,Trhypochthonius.tectorum,v2val_Full_Joint,100,0.709509901712321,0.691147459247223,0.726823632228287,0.705786473932628 -mite,mite,south,,,Ceratozetes.gracilis,v2val_Full_Joint_Truncated,100,0.882197733815623,0.867697066513443,0.899923812672535,0.880966991469706 -mite,mite,south,,,Trhypochthonius.tectorum,v2val_Full_Joint_Truncated,100,0.709509901712321,0.691165548972259,0.726823632228287,0.705786473932628 -mite,mite,south,,,Ceratozetes.gracilis,v2val_Full_Truncated,100,0.882042472336732,0.867050421871751,0.899986632313541,0.880842240129472 -mite,mite,south,,,Trhypochthonius.tectorum,v2val_Full_Truncated,100,0.709232811452144,0.690805644856461,0.726193259599038,0.705313075635924 -mite,mite,south,,,Ceratozetes.gracilis,v2val_Landcover,100,0.722459055126778,0.706537970365812,0.746090508309774,0.719697225125594 -mite,mite,south,,,Trhypochthonius.tectorum,v2val_Landcover,100,0.664748292002699,0.648949581753739,0.681661010034853,0.660231262098608 -vascular_plant,vascular_plant,north,,,Galium.boreale,insample_auc,100,0.834468988277657,0.828928336349903,0.838915518684986,0.833320286593388 -vascular_plant,vascular_plant,north,,,Vicia.americana,insample_auc,100,0.836601185549824,0.831382391528133,0.840900880382826,0.836493020513452 -vascular_plant,vascular_plant,north,,,Galium.boreale,insample_calibration_slope,100,0.996687593783804,0.985621938666354,1.00809480601565,1.00255532835023 -vascular_plant,vascular_plant,north,,,Vicia.americana,insample_calibration_slope,100,0.997485731515118,0.983694544181411,1.01070470921699,1.0015303112941 -vascular_plant,vascular_plant,north,,,Galium.boreale,insample_deviance_explained,100,0.270038134398835,0.258860358224687,0.277684812114698,0.267292470908523 -vascular_plant,vascular_plant,north,,,Vicia.americana,insample_deviance_explained,100,0.274272213778602,0.264271361845172,0.28249828148454,0.274178640552758 -vascular_plant,vascular_plant,north,,,Galium.boreale,insample_n,100,4347,4323.8,4387.3,6874 -vascular_plant,vascular_plant,north,,,Vicia.americana,insample_n,100,4351.5,4320.7,4383,6874 -vascular_plant,vascular_plant,north,,,Galium.boreale,insample_prevalence,100,0.417040220010148,0.411257497538122,0.423685039429209,0.418242653476869 -vascular_plant,vascular_plant,north,,,Vicia.americana,insample_prevalence,100,0.377306878145649,0.372421470873602,0.383305291434879,0.377654931626418 -vascular_plant,vascular_plant,north,,,Galium.boreale,insample_rmse,100,0.402807779521981,0.400119105003523,0.406220795953825,0.403734289920767 -vascular_plant,vascular_plant,north,,,Vicia.americana,insample_rmse,100,0.396543835095377,0.393279626670222,0.400410954684666,0.396623189931934 -vascular_plant,vascular_plant,north,,,Galium.boreale,insample_spearman,100,0.571838349507647,0.560911569893269,0.57919529778825,0.569557439251032 -vascular_plant,vascular_plant,north,,,Vicia.americana,insample_spearman,100,0.565450887890109,0.556980305750447,0.573253743615028,0.565106061015985 -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_auc,99,0.826467105666453,0.818134882684323,0.836065242715415, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_auc,99,0.831547875351781,0.82281859588538,0.840212743351642, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_calibration_slope,99,0.966009815108812,0.929602025875279,1.02111854359086, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_calibration_slope,99,0.977657631263549,0.939047491110731,1.02921485678769, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_deviance_explained,99,0.251408916844086,0.235763656796932,0.272400515048929, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_deviance_explained,99,0.264305366197852,0.245361740606673,0.279169764617225, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_n,100,2527,2486.7,2550.2,0 -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_n,100,2522.5,2491,2553.3,0 -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_prevalence,99,0.42032967032967,0.408683029891007,0.430237615488184, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_prevalence,99,0.378367511057499,0.367908590365397,0.386595299016258, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_rmse,99,0.408346436964315,0.402215206238207,0.413653651292851, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_rmse,99,0.399648634867756,0.393721677322426,0.406070570088633, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_spearman,99,0.55690592378909,0.543877653886252,0.574760225509608, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_spearman,99,0.556770739143624,0.541586098282066,0.570253107766034, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Climate,99,0.691331058165982,0.68123155690267,0.699550146575686, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Climate,99,0.68632433751116,0.677583119454865,0.696836659504796, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Climate_Truncated,99,0.691338305732185,0.681225313457141,0.699561362431822, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Climate_Truncated,99,0.686313307565072,0.677589110480343,0.696832207844605, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Full,99,0.825698367814741,0.8151054765314,0.834113856944233, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Full,99,0.831908202854914,0.823072908701431,0.839274892124523, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Full_Joint,99,0.822773343144525,0.812613222290088,0.831727369850404, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Full_Joint,99,0.831226336599178,0.822851131341598,0.83913038872951, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Full_Joint_Truncated,99,0.823115028001722,0.812677850152807,0.831916422372746, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Full_Joint_Truncated,99,0.831315346461213,0.822816857019605,0.839004986966977, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Full_Truncated,99,0.82605648894629,0.81517852582592,0.834212154167315, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Full_Truncated,99,0.831845257508434,0.823068897237684,0.839217924213412, -vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Landcover,99,0.799340331576827,0.791358391790512,0.809202566122227, -vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Landcover,99,0.80682764061935,0.799080161359292,0.816069162479425, -vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Climate,100,0.691863415791796,0.686489408923301,0.696719641146645,0.691923775211569 -vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Climate,100,0.688696490409687,0.683131845162456,0.693889144615178,0.688388374288585 -vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Climate_Truncated,100,0.691859795870376,0.686499004324016,0.696720903409118,0.691928871889894 -vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Climate_Truncated,100,0.68869360113843,0.683133652338722,0.693889124375781,0.688377053340582 -vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Full,100,0.831134546695233,0.826453555719213,0.836685470451462,0.830322798090827 -vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Full,100,0.834462365229321,0.829555965644719,0.838247068153718,0.834425701496386 -vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Full_Joint,100,0.828855917801472,0.823553013652392,0.83406033197307,0.82780564706394 -vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Full_Joint,100,0.834056473759319,0.828597426961828,0.837786033350782,0.833795663987679 -vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Full_Joint_Truncated,100,0.828971835122009,0.823795676153817,0.834143980306853,0.827924459375713 -vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Full_Joint_Truncated,100,0.834022959691282,0.828654047414811,0.837750473486802,0.833746590035665 -vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Full_Truncated,100,0.8312443254568,0.826661177425835,0.836774505270543,0.830433260489035 -vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Full_Truncated,100,0.834403943416699,0.829592504956551,0.838224259430919,0.834353666337466 -vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Landcover,100,0.80581950873551,0.801033316272367,0.811556580125044,0.804887352272851 -vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Landcover,100,0.810187285280975,0.804248133867056,0.815353122716748,0.809938249660894 -vascular_plant,vascular_plant,south,,,Galium.boreale,insample_auc,100,0.901035156137018,0.893563651374437,0.91031768915141,0.900337246549105 -vascular_plant,vascular_plant,south,,,Vicia.americana,insample_auc,100,0.823127372215157,0.814110129779612,0.829186986563772,0.821021560574949 -vascular_plant,vascular_plant,south,,,Galium.boreale,insample_calibration_slope,100,1.01031904559746,0.99173321586267,1.02882378259652,1.0165389783207 -vascular_plant,vascular_plant,south,,,Vicia.americana,insample_calibration_slope,100,1.0005676324237,0.975719044216311,1.02197748582502,1.00704511924372 -vascular_plant,vascular_plant,south,,,Galium.boreale,insample_deviance_explained,100,0.418065739289151,0.398117525823418,0.441782950315863,0.416204794122737 -vascular_plant,vascular_plant,south,,,Vicia.americana,insample_deviance_explained,100,0.244223301177304,0.228971217143153,0.257532605798324,0.241174697140535 -vascular_plant,vascular_plant,south,,,Galium.boreale,insample_n,100,1673.5,1649.8,1699,2648 -vascular_plant,vascular_plant,south,,,Vicia.americana,insample_n,100,1674,1653.9,1694.3,2648 -vascular_plant,vascular_plant,south,,,Galium.boreale,insample_prevalence,100,0.222355073854372,0.213781211105625,0.230552475073514,0.222054380664653 -vascular_plant,vascular_plant,south,,,Vicia.americana,insample_prevalence,100,0.264335131120709,0.2545896459474,0.271697484166548,0.264350453172205 -vascular_plant,vascular_plant,south,,,Galium.boreale,insample_rmse,100,0.305845546065687,0.299100662189831,0.313303408267496,0.306524157374337 -vascular_plant,vascular_plant,south,,,Vicia.americana,insample_rmse,100,0.37329886994026,0.36809978882764,0.378671627245352,0.374125283110615 -vascular_plant,vascular_plant,south,,,Galium.boreale,insample_spearman,100,0.579702201309579,0.563715539561319,0.591925892607868,0.576396653446468 -vascular_plant,vascular_plant,south,,,Vicia.americana,insample_spearman,100,0.492415963836901,0.478110160826574,0.503525927797863,0.490400472492442 -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_auc,99,0.892130345980071,0.875664360061375,0.90622002473392, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_auc,99,0.810120710671945,0.798108645425018,0.824817132427904, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_calibration_slope,99,0.985772407752148,0.921597320021943,1.03687803556838, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_calibration_slope,99,0.960276556523553,0.894277734258451,1.04390870829479, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_deviance_explained,99,0.391892468625026,0.34789014703091,0.428784802284979, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_deviance_explained,99,0.220097431051985,0.196095548881429,0.246442169331092, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_n,100,974.5,949,998.2,0 -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_n,100,974,953.7,994.1,0 -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_prevalence,99,0.221428571428571,0.207598481204393,0.237028340402618, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_prevalence,99,0.26440329218107,0.251305848210322,0.281129701184139, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_rmse,99,0.313481223183848,0.30107791062169,0.32484275529861, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_rmse,99,0.379044237085407,0.370203730169587,0.387481062834015, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_spearman,99,0.562971474999272,0.542747647731082,0.589490245195104, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_spearman,99,0.474975972054997,0.452667957114156,0.499158420181428, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Climate,99,0.691331058165982,0.68123155690267,0.699550146575686, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Climate,99,0.68632433751116,0.677583119454865,0.696836659504796, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Climate_Truncated,99,0.691338305732185,0.681225313457141,0.699561362431822, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Climate_Truncated,99,0.686313307565072,0.677589110480343,0.696832207844605, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Full,99,0.886224245054594,0.869117901860841,0.900597865515415, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Full,99,0.802421159411642,0.790284494061973,0.816722022145616, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Full_Joint,99,0.879593365215479,0.861830386153107,0.895754987854818, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Full_Joint,99,0.799441687344913,0.787028453317583,0.814549757529911, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Full_Joint_Truncated,99,0.879593365215479,0.86182583656257,0.895721611731583, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Full_Joint_Truncated,99,0.799412018556479,0.787026434186315,0.814551885415856, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Full_Truncated,99,0.886258508840057,0.869093352344466,0.90056962418037, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Full_Truncated,99,0.802421159411642,0.790291050006029,0.816723086088588, -vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Landcover,99,0.791207477106083,0.77381064055505,0.807775099233706, -vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Landcover,99,0.763763509218055,0.748548282413935,0.778515474422235, -vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Climate,100,0.691863415791796,0.686489408923301,0.696719641146645,0.691923775211569 -vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Climate,100,0.688696490409687,0.683131845162456,0.693889144615178,0.688388374288585 -vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Climate_Truncated,100,0.691859795870376,0.686499004324016,0.696720903409118,0.691928871889894 -vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Climate_Truncated,100,0.68869360113843,0.683133652338722,0.693889124375781,0.688377053340582 -vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Full,100,0.890558388031281,0.882171718281364,0.900674587667653,0.890081979393699 -vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Full,100,0.809819891801173,0.800947962462695,0.816880517210972,0.808012980346143 -vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Full_Joint,100,0.883627775630597,0.874438730295859,0.894315052003888,0.88338493164256 -vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Full_Joint,100,0.807080547858112,0.797845801552239,0.813867490124536,0.804804561454972 -vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Full_Joint_Truncated,100,0.883627775630597,0.87442136391673,0.894317713023663,0.88338493164256 -vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Full_Joint_Truncated,100,0.807067965171889,0.797841230192768,0.81385074593849,0.804805294807862 -vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Full_Truncated,100,0.890574871795092,0.882170390818819,0.900674379527283,0.890081979393699 -vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Full_Truncated,100,0.809800062836453,0.800947962462695,0.816879321482204,0.808015180404811 -vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Landcover,100,0.797226829924464,0.785166290798729,0.807521076761336,0.795978634172116 -vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Landcover,100,0.766748996494141,0.75614009820194,0.776155586594122,0.76658587562335 +taxon,run,region,season,part,species,metric,n,median,p10,p90,boot1,mean,sd +bird,bird,north,,,AMRO,insample_auc,5,0.72487649207557,0.722489112963132,0.72576145123834,0.721701661835893,0.724313916200942,0.00170956352292163 +bird,bird,north,,,AMRO,insample_deviance_explained,5,0.142215080261758,0.140707166105981,0.144363142127984,0.140587647159979,0.142358220452553,0.00199530922315655 +bird,bird,north,,,AMRO,insample_rmse,5,0.663619993960517,0.661624845419777,0.664719637388853,0.664239276180889,0.663325410909454,0.00165768430943888 +bird,bird,north,,,AMRO,insample_spearman,5,0.344389309730045,0.340479678130316,0.345510088540595,0.339249301212572,0.343324801699925,0.00266479191004938 +bird,bird,north,,,AMRO,insample_calibration_slope,5,0.80440117768677,0.799100431621395,0.810804150827482,0.80440117768677,0.805142929671317,0.00621763816210709 +bird,bird,north,,,AMRO,insample_prevalence,5,0.251486631016043,0.251004373826971,0.252433263182893,0.250861017819325,0.251649276245993,0.000745639638422784 +bird,bird,north,,,AMRO,insample_n,5,46747,46745.2,46750,46747,46747.6,2.50998007960223 +bird,bird,north,,,AMRO,oob_auc,5,0.730373137296335,0.729897207197475,0.730684827423547,0.730766733254682,0.730315896538717,0.000399387356506946 +bird,bird,north,,,AMRO,oob_deviance_explained,5,0.151664799728976,0.150795650809459,0.152520757577584,0.152754178062373,0.151645303910227,0.000851367053496551 +bird,bird,north,,,AMRO,oob_rmse,5,0.658090400254619,0.657748490410106,0.658646272283023,0.657851304309638,0.658152604633444,0.000488187377741323 +bird,bird,north,,,AMRO,oob_spearman,5,0.348000943732488,0.347176086570868,0.348405542162744,0.348509330988823,0.34783832412029,0.000612352651071532 +bird,bird,north,,,AMRO,oob_calibration_slope,5,0.878884682385081,0.857978242744613,0.881378788597934,0.881696624797909,0.871660692760926,0.0122350948521806 +bird,bird,north,,,AMRO,oob_prevalence,5,0.241010939026539,0.240656718532255,0.241191818078906,0.241245385459292,0.240950250724789,0.000279167052473422 +bird,bird,north,,,AMRO,oob_n,5,124877,124874,124878.8,124877,124876.4,2.50998007960223 +bird,bird,north,,,YEWA,insample_auc,5,0.778635633427003,0.778333348933502,0.779726988023268,0.779841092447633,0.778948847840918,0.000705556804087617 +bird,bird,north,,,YEWA,insample_deviance_explained,5,0.259141214360432,0.25548753680516,0.260976441286628,0.259141214360432,0.258564177416315,0.00302380870747948 +bird,bird,north,,,YEWA,insample_rmse,5,0.547361440801496,0.544308917361954,0.551849139043791,0.547158394049375,0.547972539003203,0.00399109094044317 +bird,bird,north,,,YEWA,insample_spearman,5,0.315565876581618,0.314186701701599,0.31608982965666,0.316093683296469,0.315273234600625,0.00100534988197753 +bird,bird,north,,,YEWA,insample_calibration_slope,5,0.705708332778131,0.69321277112837,0.712055437422832,0.716264166431109,0.703225661526711,0.009932019727973 +bird,bird,north,,,YEWA,insample_prevalence,5,0.118296361263824,0.117706951871658,0.119214980837802,0.118296361263824,0.118418937588893,0.00074585019883292 +bird,bird,north,,,YEWA,insample_n,5,46747,46745.2,46750,46747,46747.6,2.50998007960223 +bird,bird,north,,,YEWA,oob_auc,5,0.728595786591222,0.728230136724689,0.73005150345856,0.730809600284142,0.728967653050156,0.00107004022194788 +bird,bird,north,,,YEWA,oob_deviance_explained,5,0.179475572495229,0.177360125449593,0.182723444677448,0.183210182399369,0.179952283630345,0.00265860832017606 +bird,bird,north,,,YEWA,oob_rmse,5,0.591197627682823,0.582204245542092,0.591914132572009,0.581496589737255,0.587923953982158,0.00511357694956418 +bird,bird,north,,,YEWA,oob_spearman,5,0.273873998143111,0.273042129222169,0.27544277961002,0.276382265893519,0.274116908523614,0.00138126305304161 +bird,bird,north,,,YEWA,oob_calibration_slope,5,0.66953950556298,0.63693641889123,0.713887951481982,0.718728425403305,0.674429581507813,0.0378468170998992 +bird,bird,north,,,YEWA,oob_prevalence,5,0.13458843502006,0.134244406707603,0.134809487963868,0.13458843502006,0.134542639141448,0.000279456693593184 +bird,bird,north,,,YEWA,oob_n,5,124877,124874,124878.8,124877,124876.4,2.50998007960223 +bird,bird,south,,,AMRO,insample_auc,5,0.689537506786954,0.689085767534014,0.694179854134546,0.689537506786954,0.69116429025569,0.00267686382030874 +bird,bird,south,,,AMRO,insample_deviance_explained,5,0.0914919911243302,0.0907862366206112,0.0936622473347538,0.0905093060629211,0.0920142540132908,0.0015677422769352 +bird,bird,south,,,AMRO,insample_rmse,5,0.742475038662062,0.740051841768597,0.745694348908307,0.746104469516561,0.742888160715582,0.00289570319718642 +bird,bird,south,,,AMRO,insample_spearman,5,0.289206389172517,0.287837179517966,0.295789973779885,0.287696121760142,0.291229007576228,0.00408372016605238 +bird,bird,south,,,AMRO,insample_calibration_slope,5,0.885482053955006,0.883345065042026,0.893982715783319,0.885403476414633,0.887857917212169,0.00576920241512031 +bird,bird,south,,,AMRO,insample_prevalence,5,0.258598079444985,0.257039179794386,0.259660332963851,0.25655668358714,0.258393144664086,0.00135079625989253 +bird,bird,south,,,AMRO,insample_n,5,23639,23636.8,23639.6,23640,23638.4,1.51657508881031 +bird,bird,south,,,AMRO,oob_auc,5,0.689591934626615,0.688348632550972,0.689843166919568,0.689591934626615,0.68924802394965,0.000920081348058652 +bird,bird,south,,,AMRO,oob_deviance_explained,5,0.0984328670162571,0.094117282107193,0.0994994384196623,0.0984328670162571,0.0973395768527953,0.00316116064937106 +bird,bird,south,,,AMRO,oob_rmse,5,0.735225643291462,0.734322072987703,0.736655125441161,0.734395013830007,0.735378999246935,0.00121712350913242 +bird,bird,south,,,AMRO,oob_spearman,5,0.291547044066434,0.289653619948351,0.29213277997847,0.291376773818019,0.291100972713629,0.00151818475240607 +bird,bird,south,,,AMRO,oob_calibration_slope,5,0.934404203818623,0.926891186190493,0.961180760134325,0.953623540744284,0.941985087798582,0.0175025538271432 +bird,bird,south,,,AMRO,oob_prevalence,5,0.259989876131491,0.259616009340861,0.260538534890505,0.260708383581223,0.260061995930024,0.000475417680104008 +bird,bird,south,,,AMRO,oob_n,5,67168,67167.4,67170.2,67167,67168.6,1.51657508881031 +bird,bird,south,,,YEWA,insample_auc,5,0.719332020865892,0.7167959223474,0.722607418769604,0.721262730392428,0.719662412560823,0.00297152545331512 +bird,bird,south,,,YEWA,insample_deviance_explained,5,0.231210217495144,0.222251823593889,0.235961745747393,0.229114411717017,0.229807382164686,0.00749454908006547 +bird,bird,south,,,YEWA,insample_rmse,5,0.625043487437273,0.62080865697937,0.628590391800858,0.629305991109963,0.624889605300925,0.00399863862852713 +bird,bird,south,,,YEWA,insample_spearman,5,0.295095530582618,0.291167713950288,0.299659209105424,0.297439751075024,0.295393951865559,0.00438378006368663 +bird,bird,south,,,YEWA,insample_calibration_slope,5,0.866874540634912,0.862503477222317,0.872567336726769,0.860619291916303,0.867538534232098,0.00505022303747937 +bird,bird,south,,,YEWA,insample_prevalence,5,0.171376597004823,0.170895553957443,0.17183640258529,0.171742808798646,0.171382167859753,0.000475861876686407 +bird,bird,south,,,YEWA,insample_n,5,23639,23636.8,23639.6,23640,23638.4,1.51657508881031 +bird,bird,south,,,YEWA,oob_auc,5,0.714680925082376,0.712798268254752,0.716148435308362,0.716561203894474,0.714502329965285,0.00165165381687826 +bird,bird,south,,,YEWA,oob_deviance_explained,5,0.20248138732713,0.197183679478114,0.207430346720716,0.208119807948008,0.202298579885998,0.00504819057507152 +bird,bird,south,,,YEWA,oob_rmse,5,0.638667289278492,0.636815197956095,0.639153235243619,0.636772974666958,0.63809587657418,0.00120027991161497 +bird,bird,south,,,YEWA,oob_spearman,5,0.291734550104421,0.289334731699639,0.29340482100249,0.294073915420202,0.291424590009334,0.00203454648872441 +bird,bird,south,,,YEWA,oob_calibration_slope,5,0.871741212363925,0.853830810138639,0.901368353942083,0.871741212363925,0.875139424030135,0.0275139438874958 +bird,bird,south,,,YEWA,oob_prevalence,5,0.175869820899522,0.175707986864887,0.176039185326346,0.175741063319785,0.175867891893267,0.000167508533766851 +bird,bird,south,,,YEWA,oob_n,5,67168,67167.4,67170.2,67167,67168.6,1.51657508881031 +bryophyte,bryophyte,north,,,Bryum.All,insample_auc,5,0.638127846001886,0.629068513424739,0.639705775242135,0.638127846001886,0.635308902250384,0.00556090012471537 +bryophyte,bryophyte,north,,,Bryum.All,insample_deviance_explained,5,-0.275950193493276,-0.30093504049497,-0.246978357823554,-0.24424818155784,-0.273766521408161,0.0268076084815421 +bryophyte,bryophyte,north,,,Bryum.All,insample_rmse,5,0.479527636307096,0.479331369604887,0.480302514101111,0.479248548520264,0.479762898472393,0.000494745588454859 +bryophyte,bryophyte,north,,,Bryum.All,insample_spearman,5,0.234421074075063,0.218639785436767,0.237098983905807,0.234421074075063,0.229509973229964,0.00968700796838076 +bryophyte,bryophyte,north,,,Bryum.All,insample_calibration_slope,5,0.76525545060837,0.744233658530692,0.8107675025251,0.800400269549026,0.775500149433843,0.0336341853608396 +bryophyte,bryophyte,north,,,Bryum.All,insample_prevalence,5,0.398261429491307,0.395579807718977,0.401857219116663,0.400041220115416,0.3986710498744,0.00332558122320578 +bryophyte,bryophyte,north,,,Bryum.All,insample_n,5,3097,3072.4,4153.6,4852,3440.8,789.041633882522 +bryophyte,bryophyte,north,,,Bryum.All,oob_auc,4,0.630809068622639,0.623078012227396,0.646295585466171,,0.6335788759256,0.0130102647026339 +bryophyte,bryophyte,north,,,Bryum.All,oob_deviance_explained,4,-0.19138651532205,-0.229639096179351,-0.14780622710667,,-0.189483762694165,0.0443897167597502 +bryophyte,bryophyte,north,,,Bryum.All,oob_rmse,4,0.480264599990569,0.480001687050415,0.482221755757651,,0.480869686714472,0.00145214204934777 +bryophyte,bryophyte,north,,,Bryum.All,oob_spearman,4,0.222299987465914,0.20864983396156,0.249132606040734,,0.227008010705366,0.022676062398745 +bryophyte,bryophyte,north,,,Bryum.All,oob_calibration_slope,4,0.767411113061744,0.685944635693733,0.863863918531363,,0.772763373098032,0.0990836541433614 +bryophyte,bryophyte,north,,,Bryum.All,oob_prevalence,4,0.403358051486104,0.397360409603543,0.408630979157295,,0.403099224982043,0.006528394280969 +bryophyte,bryophyte,north,,,Bryum.All,oob_n,5,1755,698.4,1779.6,0,1411.2,789.041633882522 +bryophyte,bryophyte,north,,,Bryum.All,v2val_Climate,5,0.562296054365257,0.554173006352827,0.563975593348497,0.563268568361301,0.559777731714894,0.00502167306258926 +bryophyte,bryophyte,north,,,Bryum.All,v2val_Climate_Truncated,5,0.562295197193571,0.554169893407269,0.56396889522421,0.563263005953167,0.559774417476873,0.00502075379775713 +bryophyte,bryophyte,north,,,Bryum.All,v2val_Landcover,5,0.59031873097319,0.582725624858305,0.592734774936281,0.59031873097319,0.58822375369532,0.00508433205604652 +bryophyte,bryophyte,north,,,Bryum.All,v2val_Full,5,0.638127846001886,0.629068513424739,0.639705775242135,0.638127846001886,0.635308902250384,0.00556090012471537 +bryophyte,bryophyte,north,,,Bryum.All,v2val_Full_Truncated,5,0.639020549706553,0.629556667762114,0.64021053224959,0.639020549706553,0.635852413234428,0.00561081241692955 +bryophyte,bryophyte,north,,,Bryum.All,v2val_Full_Joint,5,0.63656986211763,0.627565538171321,0.638033581053569,0.63656986211763,0.633648909724541,0.00542015487462954 +bryophyte,bryophyte,north,,,Bryum.All,v2val_Full_Joint_Truncated,5,0.637367602039122,0.62809310218947,0.638655312558534,0.637500263262641,0.634239708554392,0.00549431056719841 +bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Climate,4,0.557120566175802,0.540969693146743,0.575367468760527,,0.557869148159969,0.0192626917262308 +bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Climate_Truncated,4,0.557104758803387,0.540966602037286,0.575374474456983,,0.557866029834635,0.0192692966023613 +bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Landcover,4,0.586912742882926,0.576860469268986,0.598756809203001,,0.587552668849403,0.0117834395761671 +bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Full,4,0.630809068622639,0.623078012227396,0.646295585466171,,0.6335788759256,0.0130102647026339 +bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Full_Truncated,4,0.63137585065822,0.623530655084542,0.647529866197263,,0.634343286360136,0.0135026415297771 +bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Full_Joint,4,0.62971870560941,0.622041714357572,0.644532558641158,,0.63226758481652,0.012530612942673 +bryophyte,bryophyte,north,,,Bryum.All,oob_v2val_Full_Joint_Truncated,4,0.630321350629701,0.622479940089637,0.645772715128109,,0.63303919132911,0.0130277082370542 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_auc,5,0.698174658798715,0.69094551695802,0.709058701464829,0.694675016541229,0.699837511037708,0.00901778644380229 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_deviance_explained,5,-0.114510476579041,-0.131813022554523,-0.0904615002285642,-0.127187969172517,-0.111764640500469,0.0203388736323564 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_rmse,5,0.466695436776973,0.462912051047972,0.467727184931025,0.467432435043113,0.465613065181239,0.00244205972530663 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_spearman,5,0.341399187547095,0.328255177404971,0.359282419780438,0.334844748701068,0.343616320027039,0.0154028239772293 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_calibration_slope,5,0.886684142091591,0.88294240979435,0.897629158250886,0.897907532572089,0.889629148428203,0.00742841787636394 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_prevalence,5,0.441055234954658,0.434155026365629,0.445516968426104,0.441055234954658,0.440040906513639,0.00592042164227121 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,insample_n,5,3068,3031.6,4152,4852,3417.8,802.273768734838 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_auc,4,0.669185680513321,0.66194940480315,0.689277603994979,,0.673776983288852,0.0156568896063866 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_deviance_explained,4,-0.185591758024389,-0.197381970216131,-0.141841988806489,,-0.174177630515047,0.0329785238818983 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_rmse,4,0.475105452566444,0.469359003725674,0.481375013261819,,0.475292278228802,0.00645345889371808 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_spearman,4,0.29101450640103,0.278786640187495,0.325621827918843,,0.299007169009701,0.0269716818569824 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_calibration_slope,4,0.772810531065414,0.725727448039512,0.851082802104523,,0.783949526784417,0.0700828360330596 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_prevalence,4,0.443665764352131,0.432608384046217,0.454192825112108,,0.443476364514296,0.0116999631210214 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_n,5,1784,700,1820.4,0,1434.2,802.273768734838 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Climate,5,0.656006784390431,0.650667856695395,0.664851582602822,0.652571065182321,0.657491388780721,0.00702864165145057 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Climate_Truncated,5,0.656014130093929,0.650669976200325,0.664873979724118,0.652577122144193,0.657501706850359,0.00703792811898215 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Landcover,5,0.626159436771152,0.623847896625279,0.63744965808967,0.626159436771152,0.629605667370405,0.00700538360157201 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Full,5,0.698174658798715,0.69094551695802,0.709058701464829,0.694675016541229,0.699837511037708,0.00901778644380229 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Full_Truncated,5,0.698175100984084,0.691211313926766,0.709267355194508,0.694914692057453,0.700043394374941,0.00901659546042277 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Full_Joint,5,0.693839915631032,0.687220562229904,0.705393695319871,0.69133661745651,0.696035361241937,0.00910536620980844 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,v2val_Full_Joint_Truncated,5,0.693823112587028,0.687469531659753,0.705583368208495,0.69158473589171,0.69622715639523,0.00910978022700139 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Climate,4,0.636250911438379,0.627380368267631,0.653685141830713,,0.639309371160374,0.0145611916830304 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Climate_Truncated,4,0.636263529886685,0.627402598788168,0.653701328647776,,0.639326696909033,0.0145585625333492 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Landcover,4,0.608585612621007,0.595998860291286,0.617214693566341,,0.607172158555155,0.0115275207748297 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Full,4,0.669185680513321,0.66194940480315,0.689277603994979,,0.673776983288852,0.0156568896063866 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Full_Truncated,4,0.669430040893281,0.6621007168852,0.689600433702903,,0.674016136893831,0.0157430215314559 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Full_Joint,4,0.667551944528101,0.659697874596654,0.686791205149657,,0.671618084060283,0.0153861204532186 +bryophyte,bryophyte,north,,,Ceratodon.purpureus,oob_v2val_Full_Joint_Truncated,4,0.667760268462804,0.659841527855415,0.687057304097503,,0.671823945258272,0.015448661895241 +bryophyte,bryophyte,south,,,Bryum.All,insample_auc,5,0.624781080501705,0.619098305791523,0.631250290557445,0.631477859980203,0.62531135077244,0.00613408721709493 +bryophyte,bryophyte,south,,,Bryum.All,insample_deviance_explained,5,-0.0882074375042718,-0.116317967526412,-0.0745997780131311,-0.0659569204939741,-0.0940636853986955,0.0217357006072699 +bryophyte,bryophyte,south,,,Bryum.All,insample_rmse,5,0.473712635656491,0.473250036655967,0.475178356015742,0.473182584522988,0.474116438717835,0.000979445089225269 +bryophyte,bryophyte,south,,,Bryum.All,insample_spearman,5,0.208905797682641,0.198735688372861,0.219564280666345,0.21996433153061,0.209499332030303,0.0105641202170489 +bryophyte,bryophyte,south,,,Bryum.All,insample_calibration_slope,5,0.874557636620987,0.806197667935386,0.910286420946533,0.929039182089329,0.863965312292241,0.0611476778254253 +bryophyte,bryophyte,south,,,Bryum.All,insample_prevalence,5,0.370562770562771,0.365528360602252,0.371569237798783,0.370562770562771,0.369235286484943,0.00385091464779508 +bryophyte,bryophyte,south,,,Bryum.All,insample_n,5,1444,1429.8,1968,2310,1614.6,388.936113005722 +bryophyte,bryophyte,south,,,Bryum.All,oob_auc,4,0.622587939645232,0.609906666129414,0.635174456157602,,0.622554097858286,0.0139528811025767 +bryophyte,bryophyte,south,,,Bryum.All,oob_deviance_explained,4,-0.0294046792827352,-0.0439442359246929,0.0123272085791315,,-0.019693132418482,0.0322171478788032 +bryophyte,bryophyte,south,,,Bryum.All,oob_rmse,4,0.475290386543187,0.473040753420391,0.476838977026333,,0.475040014171884,0.00206603954639396 +bryophyte,bryophyte,south,,,Bryum.All,oob_spearman,4,0.204907419101645,0.18388423679321,0.22723046467613,,0.205371655982377,0.0239816446700714 +bryophyte,bryophyte,south,,,Bryum.All,oob_calibration_slope,4,0.862631630882613,0.796586982841539,0.926986747501851,,0.862028226803386,0.0714420365714851 +bryophyte,bryophyte,south,,,Bryum.All,oob_prevalence,4,0.3703593055487,0.368749240306309,0.380219211352906,,0.37330567717792,0.0072755089177001 +bryophyte,bryophyte,south,,,Bryum.All,oob_n,5,866,342,880.2,0,695.4,388.936113005722 +bryophyte,bryophyte,south,,,Bryum.All,v2val_Climate,5,0.562296054365257,0.554173006352827,0.563975593348497,0.563268568361301,0.559777731714894,0.00502167306258926 +bryophyte,bryophyte,south,,,Bryum.All,v2val_Climate_Truncated,5,0.562295197193571,0.554169893407269,0.56396889522421,0.563263005953167,0.559774417476873,0.00502075379775713 +bryophyte,bryophyte,south,,,Bryum.All,v2val_Landcover,5,0.590977483143704,0.588222769171523,0.60141327621516,0.604777025029246,0.593896071204394,0.0070332679741888 +bryophyte,bryophyte,south,,,Bryum.All,v2val_Full,5,0.624781080501705,0.619098305791523,0.631250290557445,0.631477859980203,0.62531135077244,0.00613408721709493 +bryophyte,bryophyte,south,,,Bryum.All,v2val_Full_Truncated,5,0.624781080501705,0.619154420181747,0.631292646881434,0.631535708776305,0.625345448851566,0.00612033629471389 +bryophyte,bryophyte,south,,,Bryum.All,v2val_Full_Joint,5,0.624358409137791,0.618605858115063,0.631039389378428,0.630846343956086,0.624944360355056,0.00625821729858158 +bryophyte,bryophyte,south,,,Bryum.All,v2val_Full_Joint_Truncated,5,0.624356386786767,0.618642020722096,0.631057597953052,0.630860002699611,0.624962990194972,0.0062460677616445 +bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Climate,4,0.557120566175802,0.540969693146743,0.575367468760527,,0.557869148159969,0.0192626917262308 +bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Climate_Truncated,4,0.557104758803387,0.540966602037286,0.575374474456983,,0.557866029834635,0.0192692966023613 +bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Landcover,4,0.597272483474034,0.596765466015466,0.601974542549405,,0.598770712622892,0.00348361543505928 +bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Full,4,0.622587939645232,0.609906666129414,0.635174456157602,,0.622554097858286,0.0139528811025767 +bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Full_Truncated,4,0.622567363513545,0.609906666129414,0.63516211047859,,0.622543809792443,0.0139501445980818 +bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Full_Joint,4,0.622331585916656,0.607992717674767,0.636786215054005,,0.622372929093606,0.0160574239141607 +bryophyte,bryophyte,south,,,Bryum.All,oob_v2val_Full_Joint_Truncated,4,0.622313949232352,0.607992717674767,0.636775633043423,,0.622364110751454,0.0160556605380964 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_auc,5,0.717485896729522,0.71685899652647,0.72287874285206,0.717485896729522,0.719337658395294,0.00314832137641283 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_deviance_explained,5,0.0555144394517471,0.0525022042487002,0.0659817133460382,0.0552076565672067,0.0580917971366114,0.00804978167070715 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_rmse,5,0.403372277591519,0.398339551678763,0.406855811762638,0.403372277591519,0.402708178674735,0.00422999542451598 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_spearman,5,0.324238098784219,0.318040132102534,0.330331200221374,0.321683515420121,0.324176941097839,0.00635489788234709 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_calibration_slope,5,0.900294144555213,0.878293846695361,0.929217371465325,0.926240194199798,0.903277573722839,0.025082790791702 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_prevalence,5,0.23982683982684,0.231460603487586,0.246697056850865,0.23982683982684,0.239405392468179,0.00765174673579537 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,insample_n,5,1462,1452.8,1977.6,2310,1631.4,379.498089586759 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_auc,4,0.715093054248096,0.709938966718536,0.716833687112666,,0.713873963296314,0.00407985613655125 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_deviance_explained,4,0.0485196427393902,0.0213884985438529,0.0541796492701325,,0.0408513792876777,0.0205022125506785 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_rmse,4,0.406296961312814,0.398165216308586,0.415127438523745,,0.406546508529494,0.00911762395306138 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_spearman,4,0.316460819822169,0.305912040269855,0.327620546786264,,0.316679015326377,0.0116401179241884 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_calibration_slope,4,0.856225115735854,0.811357325680967,0.90322965680456,,0.856988241097932,0.0492572682113721 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_prevalence,4,0.238999572635154,0.227667581639544,0.255610993801512,,0.240885083410421,0.0151422609821477 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_n,5,848,332.4,857.2,0,678.6,379.498089586759 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Climate,5,0.656006784390431,0.650667856695395,0.664851582602822,0.652571065182321,0.657491388780721,0.00702864165145057 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Climate_Truncated,5,0.656014130093929,0.650669976200325,0.664873979724118,0.652577122144193,0.657501706850359,0.00703792811898215 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Landcover,5,0.633036828963795,0.625748811394207,0.642238687874322,0.630393575816386,0.633819706982903,0.00839614169240918 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Full,5,0.717485896729522,0.71685899652647,0.72287874285206,0.717485896729522,0.719337658395294,0.00314832137641283 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Full_Truncated,5,0.717485896729522,0.71685899652647,0.72287874285206,0.717485896729522,0.719337658395294,0.00314832137641283 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Full_Joint,5,0.718564200718732,0.717751679375772,0.723867311169212,0.718564200718732,0.720297293047133,0.00318115267699769 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,v2val_Full_Joint_Truncated,5,0.718564200718732,0.717751679375772,0.723867311169212,0.718564200718732,0.720297293047133,0.00318115267699769 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Climate,4,0.636250911438379,0.627380368267631,0.653685141830713,,0.639309371160374,0.0145611916830304 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Climate_Truncated,4,0.636263529886685,0.627402598788168,0.653701328647776,,0.639326696909033,0.0145585625333492 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Landcover,4,0.621073947742974,0.605174465777582,0.638800057775284,,0.621726314909731,0.0180551389037002 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Full,4,0.715093054248096,0.709938966718536,0.716833687112666,,0.713873963296314,0.00407985613655125 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Full_Truncated,4,0.715093054248096,0.709938966718536,0.716833687112666,,0.713873963296314,0.00407985613655125 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Full_Joint,4,0.716849054867145,0.711184101825957,0.718706027458805,,0.715489061849456,0.00457687182026539 +bryophyte,bryophyte,south,,,Ceratodon.purpureus,oob_v2val_Full_Joint_Truncated,4,0.716849054867145,0.711184101825957,0.718706027458805,,0.715489061849456,0.00457687182026539 +lichen,lichen,north,,,Cladonia.chlorophaea,insample_auc,5,0.778362159718991,0.777313519482956,0.781078019062442,0.776659932196605,0.778941977854504,0.00220139213343037 +lichen,lichen,north,,,Cladonia.chlorophaea,insample_deviance_explained,5,0.129922817133758,0.116058360191463,0.137037124714767,0.126180029301491,0.127497727648612,0.0115563906422122 +lichen,lichen,north,,,Cladonia.chlorophaea,insample_rmse,5,0.428984342825858,0.426484514201266,0.429776953204885,0.429796613403064,0.428367541663588,0.00170602773901413 +lichen,lichen,north,,,Cladonia.chlorophaea,insample_spearman,5,0.46931020347218,0.467301179157511,0.473793771369763,0.466302441462258,0.470210732962937,0.00348588836105702 +lichen,lichen,north,,,Cladonia.chlorophaea,insample_calibration_slope,5,1.02280943622485,1.00638380806286,1.02708521369247,1.02280943622485,1.0184707448443,0.011026980822553 +lichen,lichen,north,,,Cladonia.chlorophaea,insample_prevalence,5,0.384787472035794,0.382719077877937,0.387875670039505,0.384787472035794,0.385092893007771,0.00278105998275642 +lichen,lichen,north,,,Cladonia.chlorophaea,insample_n,5,3685,3644,4976.6,5811,4102.6,955.610956404331 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_auc,4,0.766320433722221,0.757378470612268,0.770532908104046,,0.764631330605033,0.00773409788726129 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_deviance_explained,4,0.113996779288933,0.0998702886563739,0.140537458619322,,0.118430418109587,0.0237257998755231 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_rmse,4,0.434233197376139,0.431409497762948,0.437875343465515,,0.434525499689062,0.00353785187767737 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_spearman,4,0.448965855284841,0.434402780928462,0.454722953452224,,0.445820863788771,0.0120096288277829 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_calibration_slope,4,0.967661500539704,0.947481883037204,0.982662848959669,,0.965812118724513,0.0194935179999311 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_prevalence,4,0.385212683681362,0.378550429000763,0.388624647224836,,0.384051865418103,0.00565743785692542 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_n,5,2126,834.4,2167,0,1708.4,955.610956404331 +lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Climate,5,0.726648362079003,0.7237588448296,0.731162951758353,0.723004804073768,0.72719557447664,0.00380494512788104 +lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Climate_Truncated,5,0.726660385143037,0.723768652905815,0.731181039693866,0.72301652132686,0.727208895973783,0.00380873317415131 +lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Landcover,5,0.714897867010021,0.711070873547497,0.715616075360572,0.710689730662897,0.713682108226793,0.00234024727927697 +lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Full,5,0.778362159718991,0.777313519482956,0.781078019062442,0.776659932196605,0.778941977854504,0.00220139213343037 +lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Full_Truncated,5,0.778213392638019,0.777186308510683,0.780963739171022,0.77651231594881,0.778814568724427,0.00222483325194008 +lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Full_Joint,5,0.777290152345703,0.776349528728965,0.780295682938712,0.77578324180292,0.778055337809396,0.00214296304404187 +lichen,lichen,north,,,Cladonia.chlorophaea,v2val_Full_Joint_Truncated,5,0.777190658438047,0.776223771512686,0.780168440645134,0.775630371417491,0.777933926824515,0.00215290328592244 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Climate,4,0.714982827935826,0.71142147790233,0.716134560839275,,0.714122250389381,0.00278710207176498 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Climate_Truncated,4,0.714986555031238,0.711429083580888,0.716141935437511,,0.714128665372639,0.00278474056829603 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Landcover,4,0.6993486440522,0.695238223478961,0.703833711587922,,0.699482446538801,0.00471231540415982 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Full,4,0.766320433722221,0.757378470612268,0.770532908104046,,0.764631330605033,0.00773409788726129 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Full_Truncated,4,0.766446335783094,0.757266971247998,0.770241672495975,,0.764523468703731,0.00779476503948412 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Full_Joint,4,0.76558237025298,0.756446722492137,0.76890475840522,,0.76350620610705,0.00763281653645435 +lichen,lichen,north,,,Cladonia.chlorophaea,oob_v2val_Full_Joint_Truncated,4,0.765720545345162,0.756181429801128,0.768751167860589,,0.763396083549231,0.00772657199336835 +lichen,lichen,north,,,Physcia.adscendens,insample_auc,5,0.837335108574662,0.833892427673138,0.841684373333987,0.838268220834197,0.837631185433223,0.00421142443545695 +lichen,lichen,north,,,Physcia.adscendens,insample_deviance_explained,5,0.153309875432323,0.124539848626552,0.181355576712871,0.153309875432323,0.153462981135222,0.0313003485289416 +lichen,lichen,north,,,Physcia.adscendens,insample_rmse,5,0.394413849121186,0.390679733700768,0.396338542998067,0.393389135816662,0.393852028605293,0.00305493252353979 +lichen,lichen,north,,,Physcia.adscendens,insample_spearman,5,0.569716833320736,0.562212442609498,0.574320845837935,0.569949863531718,0.568531717754906,0.00642768440020838 +lichen,lichen,north,,,Physcia.adscendens,insample_calibration_slope,5,0.977543215514836,0.973882604264117,0.992189734009598,0.974581235728712,0.981283244985178,0.01051079416882 +lichen,lichen,north,,,Physcia.adscendens,insample_prevalence,5,0.382961350027218,0.378754207234193,0.387054085275807,0.384099122354156,0.382969715660275,0.00456567123234541 +lichen,lichen,north,,,Physcia.adscendens,insample_n,5,3699,3673.4,4975,5811,4115.6,947.965083745177 +lichen,lichen,north,,,Physcia.adscendens,oob_auc,4,0.839292658298271,0.829533400959301,0.846965637654569,,0.838547559018745,0.00938564979201381 +lichen,lichen,north,,,Physcia.adscendens,oob_deviance_explained,4,0.143454749419506,0.0949908751245048,0.205499371215224,,0.148305016384047,0.0631136911709378 +lichen,lichen,north,,,Physcia.adscendens,oob_rmse,4,0.391817546344859,0.388424442525194,0.400850115043751,,0.393831640944583,0.00707925330540285 +lichen,lichen,north,,,Physcia.adscendens,oob_spearman,4,0.570820101540557,0.556966290991812,0.585222167374373,,0.571015906999511,0.0152763365469035 +lichen,lichen,north,,,Physcia.adscendens,oob_calibration_slope,4,0.993551581711484,0.96245857697318,1.01756614536709,,0.991023567039093,0.0304614734345619 +lichen,lichen,north,,,Physcia.adscendens,oob_prevalence,4,0.386432660247333,0.378648004672367,0.394703317951312,,0.386606232436266,0.00923888826136022 +lichen,lichen,north,,,Physcia.adscendens,oob_n,5,2112,836,2137.6,0,1695.4,947.965083745177 +lichen,lichen,north,,,Physcia.adscendens,v2val_Climate,5,0.693422907498,0.690915989050979,0.694883881152127,0.693422907498,0.692986505923955,0.00201451745990401 +lichen,lichen,north,,,Physcia.adscendens,v2val_Climate_Truncated,5,0.693419511127206,0.690911780963104,0.694883929491704,0.693419511127206,0.692985163486467,0.00201463625805256 +lichen,lichen,north,,,Physcia.adscendens,v2val_Landcover,5,0.801861109542011,0.782213209132931,0.812598365772438,0.812994033294577,0.799562929201212,0.0178185824794553 +lichen,lichen,north,,,Physcia.adscendens,v2val_Full,5,0.837335108574662,0.833892427673138,0.841684373333987,0.838268220834197,0.837631185433223,0.00421142443545695 +lichen,lichen,north,,,Physcia.adscendens,v2val_Full_Truncated,5,0.837482703696643,0.83402227575171,0.841815848139725,0.838439032548488,0.837773708478258,0.00421550438320122 +lichen,lichen,north,,,Physcia.adscendens,v2val_Full_Joint,5,0.836047248740013,0.832067905608026,0.840315800616736,0.836357683360022,0.836103552465423,0.0046074652071796 +lichen,lichen,north,,,Physcia.adscendens,v2val_Full_Joint_Truncated,5,0.836286222674123,0.832185134167773,0.840452132671447,0.836525866238843,0.836258764262809,0.00461711990867249 +lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Climate,4,0.690146520368093,0.687623451889863,0.696399093335788,,0.691478486257188,0.00495021261044504 +lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Climate_Truncated,4,0.690133662354544,0.687619313393814,0.696403232441228,,0.69147481275667,0.00495756138170841 +lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Landcover,4,0.80689396202876,0.792826777899119,0.818710192371022,,0.806090049961839,0.0145903596426596 +lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Full,4,0.839292658298271,0.829533400959301,0.846965637654569,,0.838547559018745,0.00938564979201381 +lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Full_Truncated,4,0.839479011781648,0.829796481228334,0.8470008753914,,0.83870734501609,0.00926667222581543 +lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Full_Joint,4,0.835740019351969,0.826152734502598,0.845016117951229,,0.835628881405501,0.0101826777472231 +lichen,lichen,north,,,Physcia.adscendens,oob_v2val_Full_Joint_Truncated,4,0.835910659316576,0.826439317818255,0.845028920277397,,0.835784559124612,0.0100393680766585 +lichen,lichen,south,,,Cladonia.chlorophaea,insample_auc,5,0.782940004828585,0.771430317517624,0.78704102003786,0.782940004828585,0.780384815223684,0.00892062786427006 +lichen,lichen,south,,,Cladonia.chlorophaea,insample_deviance_explained,5,0.101560194956707,0.0833055588905147,0.10855690687591,0.0760150539678254,0.0977267101316104,0.0134864921912352 +lichen,lichen,south,,,Cladonia.chlorophaea,insample_rmse,5,0.273630488580498,0.266940670870867,0.27923536011912,0.27615265636656,0.273137946045151,0.00613430562265861 +lichen,lichen,south,,,Cladonia.chlorophaea,insample_spearman,5,0.278345072756788,0.271410866729923,0.283855802016148,0.284558850730728,0.277868922002142,0.00614111968498575 +lichen,lichen,south,,,Cladonia.chlorophaea,insample_calibration_slope,5,1.00445803190767,0.923032164793458,1.06063235332037,1.00400781901235,0.998413735524807,0.0779356859153725 +lichen,lichen,south,,,Cladonia.chlorophaea,insample_prevalence,5,0.0907849829351536,0.0854880550520455,0.0942546707117982,0.092924126172208,0.0900691492139895,0.00433337917377339 +lichen,lichen,south,,,Cladonia.chlorophaea,insample_n,5,1482,1471.8,2004.4,2346,1653.4,387.296785424305 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_auc,4,0.767777065619901,0.759098616020395,0.777508870753697,,0.768153264025005,0.0103158231014254 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_deviance_explained,4,0.00020738830490835,-0.0197727594970929,0.0186135944535137,,-0.0003547337141616,0.0206046152183967 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_rmse,4,0.285215861553978,0.278859149933553,0.293735548818788,,0.285988352855544,0.00803695663998821 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_spearman,4,0.280771471067578,0.258586974156161,0.293911115946987,,0.277541166770433,0.0193124585097445 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_calibration_slope,4,1.11452074384603,0.83884295294175,1.23312243580024,,1.05842213707815,0.227871215589685 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_prevalence,4,0.0997452652709462,0.0913286406440493,0.106312613843352,,0.0990848095371992,0.00808662138066247 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_n,5,864,341.6,874.2,0,692.6,387.296785424305 +lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Climate,5,0.726648362079003,0.7237588448296,0.731162951758353,0.723004804073768,0.72719557447664,0.00380494512788104 +lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Climate_Truncated,5,0.726660385143037,0.723768652905815,0.731181039693866,0.72301652132686,0.727208895973783,0.00380873317415131 +lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Landcover,5,0.746232134390029,0.738477213352093,0.757183243479499,0.74708991515486,0.747240463783824,0.0106065443436887 +lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Full,5,0.782940004828585,0.771430317517624,0.787041020037861,0.782940004828585,0.780384815223684,0.0089206278642703 +lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Full_Truncated,5,0.782940004828585,0.771430317517624,0.787041020037861,0.782940004828585,0.780384815223684,0.0089206278642703 +lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Full_Joint,5,0.783724649927571,0.773780455672375,0.788033764122229,0.783724649927571,0.781776344950047,0.00786739974674226 +lichen,lichen,south,,,Cladonia.chlorophaea,v2val_Full_Joint_Truncated,5,0.783724649927571,0.773780455672375,0.788033764122229,0.783724649927571,0.781776344950047,0.00786739974674226 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Climate,4,0.714982827935826,0.71142147790233,0.716134560839275,,0.714122250389381,0.00278710207176498 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Climate_Truncated,4,0.714986555031238,0.711429083580888,0.716141935437511,,0.714128665372639,0.00278474056829603 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Landcover,4,0.75082184394749,0.74168570185442,0.752461597648907,,0.748144562379042,0.00674874952586361 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Full,4,0.767777065619901,0.759098616020395,0.777508870753697,,0.768153264025005,0.0103158231014254 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Full_Truncated,4,0.767777065619901,0.759098616020395,0.777508870753697,,0.768153264025005,0.0103158231014254 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Full_Joint,4,0.76868800182822,0.762466250385714,0.777619034446911,,0.769655602248286,0.00865338800241536 +lichen,lichen,south,,,Cladonia.chlorophaea,oob_v2val_Full_Joint_Truncated,4,0.76868800182822,0.762466250385714,0.777619034446911,,0.769655602248286,0.00865338800241536 +lichen,lichen,south,,,Physcia.adscendens,insample_auc,5,0.821353781719635,0.817185593102722,0.825672849056938,0.821573724007562,0.82133199677784,0.00470181458246041 +lichen,lichen,south,,,Physcia.adscendens,insample_deviance_explained,5,0.239704709286469,0.231908892003898,0.24739316046634,0.239704709286469,0.239170938481984,0.00836987471623949 +lichen,lichen,south,,,Physcia.adscendens,insample_rmse,5,0.356794925429378,0.352929674110169,0.358725757363671,0.356794925429378,0.356142732606163,0.00305057472651177 +lichen,lichen,south,,,Physcia.adscendens,insample_spearman,5,0.458171869563314,0.449023619570466,0.466231041170605,0.458171869563314,0.457232139057486,0.00919412871966326 +lichen,lichen,south,,,Physcia.adscendens,insample_calibration_slope,5,0.881651302841094,0.877029058458768,0.905002744726106,0.895884570463108,0.888780956639862,0.014710326211831 +lichen,lichen,south,,,Physcia.adscendens,insample_prevalence,5,0.215686274509804,0.210370852380551,0.218705697782215,0.215686274509804,0.214933253293698,0.00478783760132252 +lichen,lichen,south,,,Physcia.adscendens,insample_n,5,1487,1480.8,2014,2346,1662.2,382.531305908418 +lichen,lichen,south,,,Physcia.adscendens,oob_auc,4,0.816688066173278,0.814577556447952,0.819582119521717,,0.816967903181533,0.00271138269936014 +lichen,lichen,south,,,Physcia.adscendens,oob_deviance_explained,4,0.229581557918766,0.221042427704859,0.242028578202743,,0.230977232943791,0.0116106526749679 +lichen,lichen,south,,,Physcia.adscendens,oob_rmse,4,0.361221111344793,0.357219734510275,0.363119037857434,,0.360469879086979,0.00328089666967988 +lichen,lichen,south,,,Physcia.adscendens,oob_spearman,4,0.454435749483605,0.444604153459308,0.459399541619317,,0.452697248094824,0.00824469525388058 +lichen,lichen,south,,,Physcia.adscendens,oob_calibration_slope,4,0.869323881706107,0.820424126275093,0.95799881977588,,0.883529304077092,0.0756154292842296 +lichen,lichen,south,,,Physcia.adscendens,oob_prevalence,4,0.21549434057534,0.209833885194357,0.226186106799465,,0.217291237305033,0.00944381193189753 +lichen,lichen,south,,,Physcia.adscendens,oob_n,5,859,332,865.2,0,683.8,382.531305908418 +lichen,lichen,south,,,Physcia.adscendens,v2val_Climate,5,0.693422907498,0.690915989050979,0.694883881152127,0.693422907498,0.692986505923955,0.00201451745990401 +lichen,lichen,south,,,Physcia.adscendens,v2val_Climate_Truncated,5,0.693419511127206,0.690911780963104,0.694883929491704,0.693419511127206,0.692985163486467,0.00201463625805256 +lichen,lichen,south,,,Physcia.adscendens,v2val_Landcover,5,0.755367662346663,0.753354388319629,0.769154488554709,0.755514263619179,0.75923726755233,0.0106880634487565 +lichen,lichen,south,,,Physcia.adscendens,v2val_Full,5,0.821353781719635,0.817185593102722,0.825672849056938,0.821573724007562,0.82133199677784,0.00470181458246041 +lichen,lichen,south,,,Physcia.adscendens,v2val_Full_Truncated,5,0.821369179600887,0.817185593102722,0.825672849056938,0.821573724007562,0.82133507635409,0.00470183746079766 +lichen,lichen,south,,,Physcia.adscendens,v2val_Full_Joint,5,0.818351194875585,0.815473629919288,0.821595518961059,0.818413816807012,0.818450596628366,0.0035138580973253 +lichen,lichen,south,,,Physcia.adscendens,v2val_Full_Joint_Truncated,5,0.818366592756837,0.815473629919288,0.821595518961059,0.818413816807012,0.818453676204616,0.0035137559475432 +lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Climate,4,0.690146520368093,0.687623451889863,0.696399093335788,,0.691478486257188,0.00495021261044504 +lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Climate_Truncated,4,0.690133662354544,0.687619313393814,0.696403232441228,,0.69147481275667,0.00495756138170841 +lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Landcover,4,0.757543713583416,0.754237750244239,0.765899299116814,,0.759347150081352,0.00659093377352209 +lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Full,4,0.816688066173278,0.814577556447952,0.819582119521717,,0.816967903181533,0.00271138269936014 +lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Full_Truncated,4,0.816692374438254,0.814577556447952,0.819584704480702,,0.81697005731402,0.00271219354015405 +lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Full_Joint,4,0.812861687935889,0.810647101871693,0.815758867203804,,0.813105471222931,0.00283145974958857 +lichen,lichen,south,,,Physcia.adscendens,oob_v2val_Full_Joint_Truncated,4,0.81287030446584,0.810647101871693,0.815764037121775,,0.813109779487907,0.00283515421967582 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_auc,5,0.820367038624296,0.819088803830954,0.825096132690793,0.820911148938545,0.821548232901657,0.00368021243181869 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_deviance_explained,5,0.274643755760762,0.273047064346357,0.2795535093301,0.272167552780268,0.275794916286106,0.0037829114549962 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_rmse,5,0.325649814939506,0.323044576138151,0.32774821127909,0.327783381237672,0.325475899728406,0.0023268778087488 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_spearman,5,0.457622037926128,0.455146980332479,0.464800704476141,0.457622037926128,0.459282943376137,0.0052329898486681 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_calibration_slope,5,0.987210797047211,0.964756720342875,1.04226829149403,0.987210797047211,0.999628097376395,0.040457896858658 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_prevalence,5,0.22526523562793,0.222685585610422,0.228865718677616,0.22526523562793,0.225662811559798,0.00324773944134521 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,insample_n,5,2588,2554.4,3469,4053,2869.4,661.914873680899 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_auc,4,0.807437018082126,0.795187683185118,0.814600712078279,,0.805620717760392,0.0109848697690175 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_deviance_explained,4,0.234952851608951,0.215144910318834,0.252509532861174,,0.234148830166846,0.0201759809192477 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_rmse,4,0.33540874228767,0.32943601838744,0.340042074806677,,0.334930388222947,0.00571392258320504 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_spearman,4,0.436123893428745,0.419582521233002,0.446016838076314,,0.433749455018683,0.0150488838521891 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_calibration_slope,4,0.947080559125973,0.940176391586403,0.972191240627968,,0.953582885541124,0.0195519425313773 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_prevalence,4,0.224511499051979,0.218291264544164,0.230417458254175,,0.224399257871401,0.00661456979352361 +mammal,mammal_summer,north,summer,hurdle,Coyote_Summer,oob_n,5,1465,584,1498.6,0,1183.6,661.914873680899 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_auc,5,0.61119205862304,0.608247846213005,0.619766590249117,0.614800455189874,0.613133257123311,0.00617079565055122 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_deviance_explained,5,0.0293312456190202,0.0254868824061227,0.034855941334284,0.0327130199155846,0.0299655255688071,0.00466042949610858 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_rmse,5,0.440494520447441,0.439655702708701,0.441992443932362,0.440494520447441,0.440710638869574,0.00129009568326754 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_spearman,5,0.166867112583273,0.161253581705846,0.173443277843013,0.166867112583273,0.166942767059625,0.00668917082484309 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_calibration_slope,5,0.871613828265815,0.807561232834562,0.918699876937025,0.901041907949805,0.864331745671229,0.05473797753352 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_prevalence,5,0.320047355958958,0.317847829087179,0.320941159211816,0.321243523316062,0.319582666739872,0.00164288413830535 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,insample_n,5,2587,2537.6,3467,4053,2861,666.806193732482 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_auc,4,0.599789507623449,0.585096398809124,0.610287133771428,,0.598291120956897,0.0139243777141284 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_deviance_explained,4,0.0222358139724005,0.0154302383695453,0.0289963709245666,,0.0222197358828687,0.00780005134828563 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_rmse,4,0.444244510956,0.443337295940116,0.444727707984992,,0.444093075960681,0.000801980504115758 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_spearman,4,0.1428985967646,0.125110680243685,0.165466638254086,,0.144605784253376,0.0220824267179884 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_calibration_slope,4,0.750011624893702,0.699600524966396,0.836834785057386,,0.76301593212098,0.0752804461935625 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_prevalence,4,0.324076824388238,0.322733281306541,0.3276244709432,,0.324864004200118,0.00280085582083958 +mammal,mammal_summer,north,summer,hurdle,Moose_Summer,oob_n,5,1466,586,1515.4,0,1192,666.806193732482 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_auc,5,0.634240369366988,0.632910641847936,0.642195650517714,0.634240369366988,0.636730093444244,0.00492494577892738 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_deviance_explained,5,0.0264112493368064,0.0244020509882967,0.0303865560800429,0.0264112493368064,0.0271321926805094,0.00319183688778419 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_rmse,5,0.418769573067301,0.41254704078318,0.420323940326511,0.421286846656193,0.41688100357794,0.00397229497435235 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_spearman,5,0.175369074792115,0.169472066216479,0.19311238624043,0.175369074792115,0.17910418059275,0.0143000493783131 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_calibration_slope,5,0.975536082659011,0.869669271029773,0.999133512851468,1.00531761653027,0.942116568564117,0.0667346863953477 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_prevalence,5,0.691079812206573,0.689952548788955,0.71053806886065,0.691079812206573,0.698355803841802,0.0110051698108056 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,insample_n,5,675,672.2,911.8,1065,753.4,174.236333753899 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_auc,4,0.590185162579443,0.588561095758765,0.594993617451278,,0.591322444026285,0.00370894024345057 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_deviance_explained,4,-0.00660912212895907,-0.0246222554112858,0.00596211189481146,,-0.00855265757844343,0.0170007059445371 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_rmse,4,0.438945021715404,0.435169754735052,0.442994297582763,,0.439042882032192,0.00430413105762848 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_spearman,4,0.119269331592725,0.112670153678564,0.141855035472014,,0.1249788051517,0.0171203610940835 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_calibration_slope,4,0.529521999315492,0.405879483641657,0.748020467389396,,0.56339912517266,0.188100087615398 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_prevalence,4,0.675933879994794,0.656439043984736,0.693034260713775,,0.675078717390838,0.0207397429628256 +mammal,mammal_summer,south,summer,hurdle,Coyote_Summer,oob_n,5,390,153.2,392.8,0,311.6,174.236333753899 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_auc,5,0.800563803516706,0.774332077165384,0.807116397983381,0.786049924840508,0.793142123263336,0.0172774284548912 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_deviance_explained,5,0.235315369629889,0.184794016646685,0.243734500249067,0.205216892616348,0.219402980661022,0.0313035508744373 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_rmse,5,0.345153854253612,0.340273324511787,0.350387335024819,0.345153854253612,0.345250928073386,0.00498539064156107 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_spearman,5,0.399822439479778,0.365817333025682,0.416474286805188,0.381089371361329,0.393764975108055,0.025777771060145 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_calibration_slope,5,0.983951125311349,0.934535204259224,1.0182141754068,0.986324497785332,0.978961877909731,0.0489672106534818 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_prevalence,5,0.182159624413146,0.181333087877553,0.193172072927236,0.182159624413146,0.185526823886116,0.00827314119136472 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,insample_n,5,678,670.6,913,1065,754,173.956891211587 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_auc,4,0.729574202549047,0.71815250764054,0.778542353806383,,0.742983651245057,0.0363678903345032 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_deviance_explained,4,0.0957319747373082,-0.112729476086203,0.182551981226243,,0.0522886017606739,0.177839885253858 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_rmse,4,0.349900330806755,0.341019339169151,0.360550613582775,,0.350532220499047,0.0106010756691699 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_spearman,4,0.291859756320935,0.287348724179463,0.363426726063271,,0.315808305464101,0.0514234239419856 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_calibration_slope,4,0.774041963702232,0.646822148840561,1.18815164727979,,0.876502631100762,0.317190446694699 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_prevalence,4,0.182520739834082,0.160534290271133,0.183610188261351,,0.175057525142768,0.01572313523028 +mammal,mammal_summer,south,summer,hurdle,Moose_Summer,oob_n,5,387,152,394.4,0,311,173.956891211587 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_auc,5,0.843771320712376,0.837298705573725,0.845386776229526,0.839440907842931,0.841900799199367,0.00416458415403682 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_deviance_explained,5,0.296908480586636,0.290887810793796,0.302200604755114,0.294409607036222,0.296791618459388,0.00570554859270136 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_rmse,5,0.300280860452662,0.299352741208206,0.302368697081708,0.30069616131153,0.300636486451009,0.00169518301694859 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_spearman,5,0.466818565812599,0.461436555661139,0.473138781426199,0.463240577826878,0.467175270468735,0.00576420559514205 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_calibration_slope,5,0.972690836668142,0.921935358308296,0.979611212169833,0.982050704865843,0.955256551495782,0.0299777079445712 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_prevalence,5,0.201509433962264,0.199761756423448,0.205640478898507,0.201356917858008,0.202439531288226,0.00307554695366447 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,insample_n,5,2612,2595.8,3536.2,4127,2916.2,677.217616427689 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_auc,4,0.827458353428436,0.817805717079038,0.840574988117616,,0.828695495692644,0.0124179160962348 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_deviance_explained,4,0.276689902789578,0.265842733106712,0.294949072324388,,0.279337045593844,0.0167940142463136 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_rmse,4,0.303789329751735,0.298771516378234,0.304833455528629,,0.302370155610089,0.00382820840385957 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_spearman,4,0.444765026946551,0.435147313816253,0.460529325956992,,0.446960236189459,0.0143757349540166 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_calibration_slope,4,0.912370954577014,0.861212539179959,0.954228059409819,,0.909049057946925,0.0520155611998991 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_prevalence,4,0.199234448913847,0.193549889929862,0.204483398915383,,0.199078874277258,0.00592784350128194 +mammal,mammal_winter,north,winter,hurdle,Coyote_Winter,oob_n,5,1515,590.8,1531.2,0,1210.8,677.217616427689 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_auc,5,0.672085273881372,0.668633720672182,0.680693747031501,0.668767750495832,0.673650345261601,0.00705894383852787 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_deviance_explained,5,0.0758678220182751,0.0704991263862156,0.0841775407251298,0.0713753243188524,0.0768329848204126,0.00689725835100292 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_rmse,5,0.332910274284271,0.327210641524145,0.335100999244875,0.332910274284271,0.331554332789458,0.00395381643788123 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_spearman,5,0.215829774013176,0.212589282113879,0.226422981371282,0.211955551417089,0.218242903591179,0.00816350970165605 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_calibration_slope,5,0.94812085347111,0.856889525073144,1.03389671800408,0.94812085347111,0.948321951639006,0.092126092648448 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_prevalence,5,0.148776350860189,0.144579171619019,0.151265484707482,0.148776350860189,0.148217744909364,0.00345089736498891 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,insample_n,5,2637,2606,3537,4127,2925.8,671.783223964398 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_auc,4,0.654372882277188,0.640403021566792,0.6605591763494,,0.651593037049265,0.0112866225169797 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_deviance_explained,4,0.0514858356828411,0.0379579660013158,0.0604573235872615,,0.0498585564767322,0.0123265017677637 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_rmse,4,0.337444976131703,0.331449906564068,0.342571110976009,,0.337134642301943,0.00618013722586392 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_spearman,4,0.19099043338874,0.174162150130626,0.204597549588522,,0.189840016582193,0.0164077747297465 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_calibration_slope,4,0.732075381978751,0.657958194547818,0.836247242841193,,0.742809193918576,0.0983856367690785 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_prevalence,4,0.148630312373211,0.144212802079742,0.157027721824677,,0.150051704929638,0.00706694376971498 +mammal,mammal_winter,north,winter,hurdle,Moose_Winter,oob_n,5,1490,590,1521,0,1201.2,671.783223964398 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_auc,5,0.706112866420923,0.689756836780519,0.712422673247893,0.689082366331686,0.701766116171277,0.0115819259664459 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_deviance_explained,5,0.077948761211899,0.0672394512735814,0.0862779229583741,0.0670547674954939,0.0767109599030692,0.00953073159442438 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_rmse,5,0.401891914348534,0.400056693343447,0.408313222434912,0.406888363248781,0.403660884327179,0.00418719958207863 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_spearman,5,0.328404695790939,0.293718951070702,0.340256897739946,0.296933239332004,0.319246035124502,0.0234446146251979 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_calibration_slope,5,1.02265042079081,0.949079646474498,1.06820491938664,1.03306130883064,1.01079025774546,0.0605675993019255 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_prevalence,5,0.602816901408451,0.593809006916747,0.607790307126989,0.604735883424408,0.60104129788425,0.00700249933764362 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,insample_n,5,696,686.6,942.8,1098,775.8,180.377936566532 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_auc,4,0.642530046696975,0.604052771943084,0.646988682070108,,0.630380532632419,0.0285027427030458 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_deviance_explained,4,0.0177640077575169,-0.00914291472953676,0.0380006855045052,,0.0153817774932079,0.0262728517098914 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_rmse,4,0.423675312133193,0.412365569779127,0.4289659133495,,0.421525618869708,0.00945427379070984 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_spearman,4,0.215041364313525,0.166273559830134,0.234657571247567,,0.204630079474472,0.0385961235199444 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_calibration_slope,4,0.666986903656138,0.480343171377765,0.83150584600084,,0.659085192965541,0.188467730825247 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_prevalence,4,0.614967084834182,0.599715606114469,0.623570700713301,,0.612592848105398,0.0130873792975551 +mammal,mammal_winter,south,winter,hurdle,Coyote_Winter,oob_n,5,402,155.2,411.4,0,322.2,180.377936566532 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_auc,5,0.823493333333333,0.815129597967925,0.847125543751909,0.819927852628077,0.828483048702628,0.0191754521582077 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_deviance_explained,5,0.240858392609539,0.229287932861753,0.282840102014879,0.234870911391229,0.250773020824705,0.0333046711345811 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_rmse,5,0.223331046901821,0.21469247206876,0.227275887816135,0.223331046901821,0.221804231165123,0.00746931749125918 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_spearman,5,0.319693927847967,0.314626273281237,0.331123877059782,0.313170610278462,0.321987168463127,0.00854028427675327 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_calibration_slope,5,0.926166715024209,0.8521293240645,0.969181499315011,0.926166715024209,0.911949295173001,0.0594253375701516 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_prevalence,5,0.0875912408759124,0.0815913826315997,0.0946898807160816,0.0874316939890711,0.0882209860082516,0.00699025063520411 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,insample_n,5,693,685.8,936.8,1098,771.6,182.509725768245 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_auc,4,0.77744050520223,0.771065167348639,0.803486958443855,,0.784465903555099,0.0194119950156149 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_deviance_explained,4,0.17108179189974,0.0694866534919466,0.192858665045726,,0.142575268591952,0.0750549614358755 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_rmse,4,0.226318832277226,0.216290927259541,0.250116289221731,,0.231236529393947,0.0196223161851492 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_spearman,4,0.278455188286047,0.262015370235656,0.285433310292326,,0.275076011127436,0.0140713668010713 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_calibration_slope,4,0.751176575138291,0.529405518004174,0.815471311990914,,0.694935032180614,0.168343487942426 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_prevalence,4,0.0837295205104187,0.0742110480309994,0.0991029746633899,,0.0858205853938301,0.0138195684717859 +mammal,mammal_winter,south,winter,hurdle,Moose_Winter,oob_n,5,405,161.2,412.2,0,326.4,182.509725768245 +mite,mite,north,,,Ceratozetes.gracilis,insample_auc,5,0.793023395932593,0.785116760768433,0.797585561245424,0.791183069113511,0.791977110922841,0.00673340419157571 +mite,mite,north,,,Ceratozetes.gracilis,insample_deviance_explained,5,0.134692532005047,0.108608328871156,0.141811919550427,0.125527507164404,0.128052047876132,0.0184307041865782 +mite,mite,north,,,Ceratozetes.gracilis,insample_rmse,5,0.364641744759417,0.36299256091057,0.368618183340602,0.364308079016483,0.365438034871869,0.00324688670297244 +mite,mite,north,,,Ceratozetes.gracilis,insample_spearman,5,0.41429348998048,0.402422002613553,0.419681351800533,0.409677886636093,0.412099312835312,0.00911372223585555 +mite,mite,north,,,Ceratozetes.gracilis,insample_calibration_slope,5,1.0224894029886,1.00755092646514,1.03915812721702,1.0224894029886,1.02287359767688,0.0156175294996684 +mite,mite,north,,,Ceratozetes.gracilis,insample_prevalence,5,0.211169769380383,0.208505864577966,0.211369634473166,0.208369812642269,0.210190443530451,0.00151474200965215 +mite,mite,north,,,Ceratozetes.gracilis,insample_n,5,3651,3606.6,4890.2,5711,4047.6,930.188045504779 +mite,mite,north,,,Ceratozetes.gracilis,oob_auc,4,0.779665972511523,0.773198985489411,0.798395168992114,,0.784045333032408,0.0147724589078592 +mite,mite,north,,,Ceratozetes.gracilis,oob_deviance_explained,4,0.0923986988857812,0.0862667569754759,0.144304465844404,,0.108746493545894,0.037849846139005 +mite,mite,north,,,Ceratozetes.gracilis,oob_rmse,4,0.36680003014665,0.360411853868392,0.368997514633471,,0.365303354506851,0.00518143947194182 +mite,mite,north,,,Ceratozetes.gracilis,oob_spearman,4,0.389977816116695,0.383012434237582,0.415943717902066,,0.396763716083216,0.0196903404412637 +mite,mite,north,,,Ceratozetes.gracilis,oob_calibration_slope,4,0.956222238793897,0.921365026616042,1.05973298874715,,0.980741359427966,0.0799378045179831 +mite,mite,north,,,Ceratozetes.gracilis,oob_prevalence,4,0.203407429558746,0.203105380508127,0.206516438658429,,0.204409915290554,0.0022495255846381 +mite,mite,north,,,Ceratozetes.gracilis,oob_n,5,2060,820.8,2104.4,0,1663.4,930.188045504779 +mite,mite,north,,,Ceratozetes.gracilis,v2val_Climate,5,0.738382414749567,0.734358129806975,0.742005285315151,0.738382414749567,0.738111987253943,0.00384260421914782 +mite,mite,north,,,Ceratozetes.gracilis,v2val_Climate_Truncated,5,0.738392463572007,0.734342356311168,0.742026178178654,0.738392463572007,0.738117914905037,0.00385983439540224 +mite,mite,north,,,Ceratozetes.gracilis,v2val_Landcover,5,0.766722893536977,0.759062628411847,0.772422596192916,0.766722893536977,0.766125561315905,0.00683173671440066 +mite,mite,north,,,Ceratozetes.gracilis,v2val_Full,5,0.793023395932593,0.785116760768433,0.797585561245424,0.791183069113511,0.791977110922841,0.00673340419157571 +mite,mite,north,,,Ceratozetes.gracilis,v2val_Full_Truncated,5,0.793016443892401,0.785027851595894,0.797500866905145,0.791105466738786,0.791908988560901,0.00674045451455018 +mite,mite,north,,,Ceratozetes.gracilis,v2val_Full_Joint,5,0.793386060695946,0.785055839040043,0.797509252679056,0.791225262500488,0.792011165992849,0.00676871644676792 +mite,mite,north,,,Ceratozetes.gracilis,v2val_Full_Joint_Truncated,5,0.793386524165292,0.784963217939285,0.797444970586129,0.7911589984368,0.791952343573282,0.00679046380342073 +mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Climate,4,0.731027419469463,0.725410991997312,0.741834431700471,,0.732881199740483,0.00906851156052711 +mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Climate_Truncated,4,0.731045119285728,0.72540173801145,0.741842884282973,,0.732885970615359,0.00907305785634944 +mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Landcover,4,0.752661723212296,0.748458309671417,0.773156655512682,,0.758480122769263,0.0155945518677711 +mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Full,4,0.779665972511523,0.773198985489411,0.798395168992114,,0.784045333032408,0.0147724589078592 +mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Full_Truncated,4,0.77962848748914,0.773216243029163,0.798382692848602,,0.784036330667528,0.014780937706049 +mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Full_Joint,4,0.779066438961531,0.77354347637541,0.798343558984922,,0.78397863804627,0.0148677753889034 +mite,mite,north,,,Ceratozetes.gracilis,oob_v2val_Full_Joint_Truncated,4,0.779044896212606,0.773564180389716,0.798356007315929,,0.783984323098475,0.01488666359881 +mite,mite,north,,,Trhypochthonius.tectorum,insample_auc,5,0.672907942808574,0.669208813819547,0.682343647003947,0.669778092188547,0.674857259308144,0.0070191421865739 +mite,mite,north,,,Trhypochthonius.tectorum,insample_deviance_explained,5,-0.0324180258517079,-0.0527646417815135,-0.0101623689114898,-0.0407269697080057,-0.0321707940218346,0.0231090933708154 +mite,mite,north,,,Trhypochthonius.tectorum,insample_rmse,5,0.288562681524747,0.286522184662904,0.291945228269945,0.288562681524747,0.288981155568958,0.00296084278549151 +mite,mite,north,,,Trhypochthonius.tectorum,insample_spearman,5,0.174619892294846,0.172564649955749,0.187939106716607,0.1729413278484,0.178541440689235,0.00902094001424548 +mite,mite,north,,,Trhypochthonius.tectorum,insample_calibration_slope,5,0.938290627841302,0.909627160028179,0.994839719200875,0.931192865479756,0.95045611024787,0.0429430392043578 +mite,mite,north,,,Trhypochthonius.tectorum,insample_prevalence,5,0.0956049728593941,0.0940001763114542,0.0987931854359719,0.0956049728593941,0.0960754727002272,0.00270898671716411 +mite,mite,north,,,Trhypochthonius.tectorum,insample_n,5,3624,3599.2,4881,5711,4034.4,937.384819591186 +mite,mite,north,,,Trhypochthonius.tectorum,oob_auc,4,0.651421835190664,0.64513479278205,0.663121005511504,,0.653354738016459,0.00989688245354325 +mite,mite,north,,,Trhypochthonius.tectorum,oob_deviance_explained,4,-0.0625612551961812,-0.104386756333539,-0.0317884651562399,,-0.066508652016687,0.0413281120013629 +mite,mite,north,,,Trhypochthonius.tectorum,oob_rmse,4,0.289482213990474,0.281791769789717,0.293681830417276,,0.28823548978549,0.00666038076943173 +mite,mite,north,,,Trhypochthonius.tectorum,oob_spearman,4,0.153802350804888,0.145130738202924,0.167192976536939,,0.155487712637062,0.0120299047392454 +mite,mite,north,,,Trhypochthonius.tectorum,oob_calibration_slope,4,0.866440051991562,0.715195080326882,0.99057532202289,,0.856758015693936,0.149800198792233 +mite,mite,north,,,Trhypochthonius.tectorum,oob_prevalence,4,0.0963264503316864,0.0893459374618542,0.0984905896614327,,0.0946063169245129,0.00535078906761863 +mite,mite,north,,,Trhypochthonius.tectorum,oob_n,5,2087,830,2111.8,0,1676.6,937.384819591186 +mite,mite,north,,,Trhypochthonius.tectorum,v2val_Climate,5,0.646091585413555,0.643946189711462,0.652343816700896,0.646091585413555,0.647435005669506,0.00501081050104212 +mite,mite,north,,,Trhypochthonius.tectorum,v2val_Climate_Truncated,5,0.646095557008706,0.643948358640746,0.65233771826661,0.646095557008706,0.647436539502018,0.00500120525659027 +mite,mite,north,,,Trhypochthonius.tectorum,v2val_Landcover,5,0.637987440117159,0.627044747234771,0.648796943002273,0.637987440117159,0.637774648879966,0.0120277548025983 +mite,mite,north,,,Trhypochthonius.tectorum,v2val_Full,5,0.672907942808574,0.669208813819547,0.682343647003947,0.669778092188547,0.674857259308144,0.0070191421865739 +mite,mite,north,,,Trhypochthonius.tectorum,v2val_Full_Truncated,5,0.672968462835808,0.669234243838635,0.682370652186889,0.669767986128102,0.67489223986993,0.00700566517575936 +mite,mite,north,,,Trhypochthonius.tectorum,v2val_Full_Joint,5,0.673039662867848,0.668775078579256,0.682456515305188,0.66926215120794,0.674743399528349,0.00725984991743693 +mite,mite,north,,,Trhypochthonius.tectorum,v2val_Full_Joint_Truncated,5,0.673102852896284,0.66877285388055,0.682481051437855,0.669242116386356,0.674767969249559,0.0072566620089462 +mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Climate,4,0.637448091750764,0.62319195321542,0.64083947116042,,0.633567820634447,0.0109633746747622 +mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Climate_Truncated,4,0.637447116712891,0.623191511787853,0.640852747518307,,0.633572125955883,0.0109617111390789 +mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Landcover,4,0.619482659509824,0.616517647587969,0.622452812919573,,0.6194844957555,0.00319182447037644 +mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Full,4,0.651421835190664,0.64513479278205,0.663121005511504,,0.653354738016459,0.00989688245354325 +mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Full_Truncated,4,0.651598760851872,0.645145346470754,0.66328863933151,,0.653468926601344,0.00996350123797742 +mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Full_Joint,4,0.650909465977038,0.643507260329979,0.663048883832917,,0.652601327480188,0.0106588967286394 +mite,mite,north,,,Trhypochthonius.tectorum,oob_v2val_Full_Joint_Truncated,4,0.65107837071969,0.643490423905209,0.66321170510179,,0.652701723422411,0.0107382013149323 +mite,mite,south,,,Ceratozetes.gracilis,insample_auc,5,0.883740175681923,0.875951595662976,0.884218489485534,0.880842240129472,0.881127727799286,0.00491960965959818 +mite,mite,south,,,Ceratozetes.gracilis,insample_deviance_explained,5,0.169227984866749,0.102917912757133,0.174897444771582,0.169227984866749,0.148837676671015,0.0479777276568308 +mite,mite,south,,,Ceratozetes.gracilis,insample_rmse,5,0.223410917578014,0.207894263041054,0.228617168412238,0.215105469199925,0.219622699857429,0.0107678160117743 +mite,mite,south,,,Ceratozetes.gracilis,insample_spearman,5,0.304054710362117,0.292604158893579,0.321411532274628,0.304054710362117,0.306352068526036,0.0145523795554875 +mite,mite,south,,,Ceratozetes.gracilis,insample_calibration_slope,5,0.646501812554423,0.571714943118018,0.663896293625869,0.652084939529121,0.625798242702713,0.0510970524920574 +mite,mite,south,,,Ceratozetes.gracilis,insample_prevalence,5,0.056970509383378,0.0521775638545351,0.0622115928022082,0.0562843842573,0.0572190090351997,0.00535467847864499 +mite,mite,south,,,Ceratozetes.gracilis,insample_n,5,1494,1479.4,2024.6,2363,1667.4,389.193139713433 +mite,mite,south,,,Ceratozetes.gracilis,oob_auc,4,0.877556363090999,0.870858501486971,0.891246533596325,,0.880053616270034,0.0119804912573777 +mite,mite,south,,,Ceratozetes.gracilis,oob_deviance_explained,4,0.149139947228324,0.081863654137057,0.211545016984438,,0.147400224608626,0.0705190613624988 +mite,mite,south,,,Ceratozetes.gracilis,oob_rmse,4,0.217549055983636,0.21158190570823,0.223701160718158,,0.217615111147606,0.00687714178519329 +mite,mite,south,,,Ceratozetes.gracilis,oob_spearman,4,0.297912570168464,0.269110530526911,0.325939210959734,,0.297635642007648,0.0304788257744718 +mite,mite,south,,,Ceratozetes.gracilis,oob_calibration_slope,4,0.588033894967732,0.50519914194533,0.720419749435623,,0.60573071691255,0.11934507589046 +mite,mite,south,,,Ceratozetes.gracilis,oob_prevalence,4,0.0533392883804503,0.0452752248602848,0.0645232647455575,,0.0544535429679295,0.0107814199736608 +mite,mite,south,,,Ceratozetes.gracilis,oob_n,5,869,338.4,883.6,0,695.6,389.193139713433 +mite,mite,south,,,Ceratozetes.gracilis,v2val_Climate,5,0.738382414749567,0.734358129806975,0.742005285315151,0.738382414749567,0.738111987253943,0.00384260421914782 +mite,mite,south,,,Ceratozetes.gracilis,v2val_Climate_Truncated,5,0.738392463572007,0.734342356311168,0.742026178178654,0.738392463572007,0.738117914905037,0.00385983439540224 +mite,mite,south,,,Ceratozetes.gracilis,v2val_Landcover,5,0.722020131552721,0.71607655717679,0.728899344516339,0.719697225125594,0.722583698825812,0.00643427629050986 +mite,mite,south,,,Ceratozetes.gracilis,v2val_Full,5,0.883740175681923,0.875951595662976,0.884218489485534,0.880842240129472,0.881127727799286,0.00491960965959818 +mite,mite,south,,,Ceratozetes.gracilis,v2val_Full_Truncated,5,0.883740175681923,0.875951595662976,0.884218489485534,0.880842240129472,0.881127727799286,0.00491960965959818 +mite,mite,south,,,Ceratozetes.gracilis,v2val_Full_Joint,5,0.883869625520111,0.877386169471364,0.884867266298163,0.880966991469706,0.88186690451822,0.00416238011000395 +mite,mite,south,,,Ceratozetes.gracilis,v2val_Full_Joint_Truncated,5,0.883869625520111,0.877386169471364,0.884867266298163,0.880966991469706,0.88186690451822,0.00416238011000395 +mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Climate,4,0.731027419469463,0.725410991997312,0.741834431700471,,0.732881199740483,0.00906851156052711 +mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Climate_Truncated,4,0.731045119285728,0.72540173801145,0.741842884282973,,0.732885970615359,0.00907305785634944 +mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Landcover,4,0.704830138651714,0.697578522816295,0.713165675388293,,0.705217253259271,0.00876901015163448 +mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Full,4,0.877556363090999,0.870858501486971,0.891246533596325,,0.880053616270034,0.0119804912573777 +mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Full_Truncated,4,0.877556363090999,0.870858501486971,0.891281973045494,,0.880066273216166,0.0120040417679418 +mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Full_Joint,4,0.87838605738981,0.870791888918913,0.892459363692493,,0.880700035186876,0.0128370905847578 +mite,mite,south,,,Ceratozetes.gracilis,oob_v2val_Full_Joint_Truncated,4,0.87838605738981,0.870791888918913,0.892494803141663,,0.880712692133008,0.012860350145155 +mite,mite,south,,,Trhypochthonius.tectorum,insample_auc,5,0.714545787058589,0.69969622945697,0.724109648011322,0.705313075635924,0.712537633834069,0.0121603195766141 +mite,mite,south,,,Trhypochthonius.tectorum,insample_deviance_explained,5,-2.20551740843788e-05,-0.00644243886806564,0.0163984110082346,0.01492775208855,0.00397390748081414,0.0114643898167834 +mite,mite,south,,,Trhypochthonius.tectorum,insample_rmse,5,0.269858367078513,0.266406986696229,0.272808873150662,0.270464873051431,0.269513931566532,0.00330133552867803 +mite,mite,south,,,Trhypochthonius.tectorum,insample_spearman,5,0.210611679646456,0.189085620295154,0.213764285932634,0.196610759018781,0.203586560421865,0.0130220945300142 +mite,mite,south,,,Trhypochthonius.tectorum,insample_calibration_slope,5,0.964366086336983,0.916309849664698,0.989356334697425,0.964366086336983,0.955309012900657,0.0368054765299321 +mite,mite,south,,,Trhypochthonius.tectorum,insample_prevalence,5,0.0833685992382565,0.0807473443243282,0.0863035095821866,0.0833685992382565,0.083401335320589,0.00299510185415199 +mite,mite,south,,,Trhypochthonius.tectorum,insample_n,5,1489,1484.4,2017.4,2363,1664.2,390.68875079787 +mite,mite,south,,,Trhypochthonius.tectorum,oob_auc,4,0.669707753515078,0.654014549095664,0.683226886999099,,0.66893129960958,0.0157013541907933 +mite,mite,south,,,Trhypochthonius.tectorum,oob_deviance_explained,4,0.0238861350837344,-0.00501344344448882,0.0319775842734965,,0.0164546603199983,0.0222551548342608 +mite,mite,south,,,Trhypochthonius.tectorum,oob_rmse,4,0.275344693909917,0.2660800590245,0.279902566485838,,0.273663707370811,0.00770431969533571 +mite,mite,south,,,Trhypochthonius.tectorum,oob_spearman,4,0.16338068295489,0.142721241028181,0.180002438567577,,0.161938652128453,0.0200563721519488 +mite,mite,south,,,Trhypochthonius.tectorum,oob_calibration_slope,4,0.640530845430618,0.577798461860074,0.822553046730012,,0.68313435176235,0.143050298074835 +mite,mite,south,,,Trhypochthonius.tectorum,oob_prevalence,4,0.0842873358196854,0.0775201528145626,0.0881856815952926,,0.0832627510948584,0.00595703320150957 +mite,mite,south,,,Trhypochthonius.tectorum,oob_n,5,874,345.6,878.6,0,698.8,390.68875079787 +mite,mite,south,,,Trhypochthonius.tectorum,v2val_Climate,5,0.646091585413555,0.643946189711462,0.652343816700896,0.646091585413555,0.647435005669506,0.00501081050104212 +mite,mite,south,,,Trhypochthonius.tectorum,v2val_Climate_Truncated,5,0.646095557008706,0.643948358640746,0.65233771826661,0.646095557008706,0.647436539502018,0.00500120525659027 +mite,mite,south,,,Trhypochthonius.tectorum,v2val_Landcover,5,0.671864124042916,0.659898184775652,0.684374697179488,0.660231262098608,0.671375041215641,0.0130121969567772 +mite,mite,south,,,Trhypochthonius.tectorum,v2val_Full,5,0.714545787058589,0.69969622945697,0.724109648011322,0.705313075635924,0.712537633834069,0.0121603195766141 +mite,mite,south,,,Trhypochthonius.tectorum,v2val_Full_Truncated,5,0.714545787058589,0.69969622945697,0.724109648011322,0.705313075635924,0.712537633834069,0.0121603195766141 +mite,mite,south,,,Trhypochthonius.tectorum,v2val_Full_Joint,5,0.714906009620713,0.700143210072346,0.72490193539616,0.705786473932628,0.713089166909465,0.012325737475379 +mite,mite,south,,,Trhypochthonius.tectorum,v2val_Full_Joint_Truncated,5,0.714906009620713,0.700143210072346,0.72490193539616,0.705786473932628,0.713089166909465,0.012325737475379 +mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Climate,4,0.637448091750764,0.62319195321542,0.64083947116042,,0.633567820634447,0.0109633746747622 +mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Climate_Truncated,4,0.637447116712891,0.623191511787853,0.640852747518307,,0.633572125955883,0.0109617111390789 +mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Landcover,4,0.613895073569526,0.611766319340838,0.647057552774624,,0.624978546775387,0.0240120090979382 +mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Full,4,0.669707753515078,0.654014549095664,0.683226886999099,,0.66893129960958,0.0157013541907933 +mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Full_Truncated,4,0.669707753515078,0.654014549095664,0.683226886999099,,0.66893129960958,0.0157013541907933 +mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Full_Joint,4,0.670252865183779,0.654581402849557,0.683544630202293,,0.669402973285312,0.0155825833517288 +mite,mite,south,,,Trhypochthonius.tectorum,oob_v2val_Full_Joint_Truncated,4,0.670252865183779,0.654581402849557,0.683544630202293,,0.669402973285312,0.0155825833517288 +vascular_plant,vascular_plant,north,,,Galium.boreale,insample_auc,5,0.830322798090827,0.829752014715758,0.833521610487142,0.830322798090827,0.831256480687861,0.00210224952478856 +vascular_plant,vascular_plant,north,,,Galium.boreale,insample_deviance_explained,5,0.169049735433044,0.156535489819263,0.180380243811949,0.172184312829234,0.16845440325201,0.01233664255535 +vascular_plant,vascular_plant,north,,,Galium.boreale,insample_rmse,5,0.406331801158912,0.404646760463192,0.406711203597052,0.406706951792596,0.405904427434075,0.00126412825140784 +vascular_plant,vascular_plant,north,,,Galium.boreale,insample_spearman,5,0.564448272444098,0.564022722895439,0.568538539772008,0.564448272444098,0.565752949420103,0.002625462296498 +vascular_plant,vascular_plant,north,,,Galium.boreale,insample_calibration_slope,5,0.979847210350883,0.969126982244339,0.991954887525608,0.979847210350883,0.980520168665477,0.0114519899974685 +vascular_plant,vascular_plant,north,,,Galium.boreale,insample_prevalence,5,0.418242653476869,0.411391418988775,0.421189975169674,0.418242653476869,0.416870814782784,0.00501851999449775 +vascular_plant,vascular_plant,north,,,Galium.boreale,insample_n,5,4366,4320.2,5874.4,6874,4851.6,1130.83522230252 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_auc,4,0.825970385744679,0.820899166013025,0.829858683383589,,0.825547913568699,0.00483682169731863 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_deviance_explained,4,0.180338934815369,0.153363367618616,0.193673751289778,,0.175467238128817,0.0239800671027935 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_rmse,4,0.409024585589369,0.407545829323972,0.412796643199926,,0.409843621784069,0.00304952623195646 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_spearman,4,0.556036446141893,0.549781718602555,0.564239538039467,,0.556732290555549,0.00779665231289283 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_calibration_slope,4,0.961989811631539,0.953282864779601,0.97698898933143,,0.964237036934379,0.0130331463165358 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_prevalence,4,0.419241311741466,0.413108455574493,0.430771977433407,,0.421169100857526,0.00978800188867454 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_n,5,2508,999.6,2553.8,0,2022.4,1130.83522230252 +vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Climate,5,0.688882537608652,0.686220497467121,0.693315342224415,0.691923775211569,0.689606019273988,0.00355153250685828 +vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Climate_Truncated,5,0.688853782361867,0.686211230349888,0.693311107621612,0.691928871889894,0.689595671925047,0.00355473297057612 +vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Landcover,5,0.806658535413912,0.804749026795703,0.81034804923884,0.804887352272851,0.807185726647177,0.00308448997445622 +vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Full,5,0.830322798090827,0.829752014715758,0.833521610487142,0.830322798090827,0.831256480687861,0.00210224952478856 +vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Full_Truncated,5,0.830433260489035,0.829860825639713,0.833584571804586,0.830433260489035,0.83134635595991,0.00208467504714064 +vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Full_Joint,5,0.82780564706394,0.827522559630011,0.831849313017971,0.82780564706394,0.829181386915621,0.00243189954325233 +vascular_plant,vascular_plant,north,,,Galium.boreale,v2val_Full_Joint_Truncated,5,0.827924459375713,0.827636541171172,0.831943133083281,0.827924459375713,0.829288457143518,0.0024223441955198 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Climate,4,0.69847723797285,0.691752649872066,0.699715919161363,,0.696517985504182,0.00490694367651728 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Climate_Truncated,4,0.698467309951771,0.691745994349179,0.699715497383917,,0.696512621319469,0.0049068149193327 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Landcover,4,0.798538423391199,0.793813753185317,0.801044488836154,,0.797746064548011,0.0039851186104012 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Full,4,0.825970385744679,0.820899166013025,0.829858683383589,,0.825547913568699,0.00483682169731863 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Full_Truncated,4,0.826084368160094,0.821025133464921,0.829955751376353,,0.825660135489053,0.00481955559769384 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Full_Joint,4,0.822407455245542,0.816503165651276,0.825556259025818,,0.821423353169117,0.00501007923907512 +vascular_plant,vascular_plant,north,,,Galium.boreale,oob_v2val_Full_Joint_Truncated,4,0.822548387109745,0.816621603632805,0.825661726608092,,0.821543585688819,0.00501141542481554 +vascular_plant,vascular_plant,north,,,Vicia.americana,insample_auc,5,0.836315133959189,0.831991084674704,0.837528607951877,0.834425701496386,0.835217103556636,0.00298930683727749 +vascular_plant,vascular_plant,north,,,Vicia.americana,insample_deviance_explained,5,0.154953219466334,0.129409063699085,0.172774430677382,0.160280734393984,0.151845840183807,0.0222292641700649 +vascular_plant,vascular_plant,north,,,Vicia.americana,insample_rmse,5,0.397835953480652,0.39520050591856,0.39953977885822,0.397892542228527,0.397399168674453,0.00225088132355584 +vascular_plant,vascular_plant,north,,,Vicia.americana,insample_spearman,5,0.565718307196976,0.558075214292711,0.566158340164243,0.561646901356684,0.563051440283389,0.00452676813056749 +vascular_plant,vascular_plant,north,,,Vicia.americana,insample_calibration_slope,5,1.00378490707536,0.996670213069317,1.00931321625889,1.00546871573532,1.00306243102853,0.00640638209817766 +vascular_plant,vascular_plant,north,,,Vicia.americana,insample_prevalence,5,0.377654931626418,0.37492184315556,0.38072781584232,0.377654931626418,0.377957278345016,0.00308047552556824 +vascular_plant,vascular_plant,north,,,Vicia.americana,insample_n,5,4329,4315.8,5874,6874,4842.2,1136.06104589498 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_auc,4,0.829938952959948,0.828125419425383,0.835304874971869,,0.831207663130432,0.00415809140956838 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_deviance_explained,4,0.177626825486026,0.135153586377154,0.21154877519812,,0.174572793558605,0.0427317407176602 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_rmse,4,0.400006137187779,0.39713029702198,0.40231573999719,,0.399803909560497,0.0030220247869821 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_spearman,4,0.554463344935879,0.551184680243305,0.562068414950895,,0.556008489693894,0.00617058608931537 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_calibration_slope,4,0.988524116774644,0.97389136931948,1.00993315311349,,0.990944219947388,0.0195740205696745 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_prevalence,4,0.375247544204322,0.372430611979277,0.382966913772923,,0.376998414684163,0.00599903919350225 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_n,5,2545,1000,2558.2,0,2031.8,1136.06104589498 +vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Climate,5,0.688388374288585,0.686629323150551,0.697141250977293,0.688388374288585,0.69087203080297,0.00615433602729216 +vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Climate_Truncated,5,0.688377053340582,0.686621028690138,0.697132635984666,0.688377053340582,0.690862704228311,0.00615461616566359 +vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Landcover,5,0.812599770966484,0.806923106797722,0.814411627611935,0.809938249660894,0.811141578894183,0.00400693028240299 +vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Full,5,0.836315133959189,0.831991084674704,0.837528607951877,0.834425701496386,0.835217103556636,0.00298930683727749 +vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Full_Truncated,5,0.836225603913632,0.831944005617998,0.837453646171789,0.834353666337466,0.835150674964538,0.0029708276538074 +vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Full_Joint,5,0.836242738307649,0.831840647476923,0.837075962254889,0.833795663987679,0.834913342412997,0.00279108797567405 +vascular_plant,vascular_plant,north,,,Vicia.americana,v2val_Full_Joint_Truncated,5,0.836167732781523,0.831822036390158,0.836995692851111,0.833746590035665,0.834858253988271,0.00275653331501612 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Climate,4,0.687058088763642,0.673058403439112,0.691946304395709,,0.683803992444905,0.0115178096712181 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Climate_Truncated,4,0.687038471404578,0.673051662415016,0.691940326726772,,0.683793845094804,0.0115133321710675 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Landcover,4,0.807847376592108,0.801759141298583,0.811699920923116,,0.807048915534066,0.00558077270332568 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Full,4,0.829938952959948,0.828125419425383,0.835304874971869,,0.831207663130432,0.00415809140956838 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Full_Truncated,4,0.829888329553378,0.828175765215782,0.835181080251946,,0.831166967539439,0.00409066773151742 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Full_Joint,4,0.828971535109331,0.826291764196716,0.83479597749998,,0.830094632065772,0.00475621189002634 +vascular_plant,vascular_plant,north,,,Vicia.americana,oob_v2val_Full_Joint_Truncated,4,0.828915617285276,0.826336395712055,0.834689140179746,,0.830056439185722,0.00469352433889885 +vascular_plant,vascular_plant,south,,,Galium.boreale,insample_auc,5,0.890081979393699,0.886120541286867,0.900531693288428,0.890081979393699,0.8925816929913,0.0074671519015807 +vascular_plant,vascular_plant,south,,,Galium.boreale,insample_deviance_explained,5,0.376511843709018,0.359648245967408,0.39273006526476,0.36183949121173,0.375558492410363,0.0170745413457127 +vascular_plant,vascular_plant,south,,,Galium.boreale,insample_rmse,5,0.308360864442848,0.306833389069796,0.315704381717812,0.312233283277407,0.31046772667452,0.00475923236811526 +vascular_plant,vascular_plant,south,,,Galium.boreale,insample_spearman,5,0.561632279820523,0.56034731464871,0.575734647134824,0.561632279820523,0.565885670838601,0.0104854350954813 +vascular_plant,vascular_plant,south,,,Galium.boreale,insample_calibration_slope,5,0.980281716931255,0.970651842047086,0.998280590750127,0.982887664309838,0.982912614118871,0.0154729818019227 +vascular_plant,vascular_plant,south,,,Galium.boreale,insample_prevalence,5,0.222088835534214,0.217190546592102,0.228625506806496,0.222054380664653,0.222881775063893,0.00607679419012681 +vascular_plant,vascular_plant,south,,,Galium.boreale,insample_n,5,1692,1666,2267.2,2648,1873.6,433.131388841769 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_auc,4,0.884065848019888,0.870449780280702,0.894068384737495,,0.882775301226467,0.0128406847119162 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_deviance_explained,4,0.334099694985671,0.299899048318077,0.344909076796622,,0.325745671822584,0.0279649666452476 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_rmse,4,0.322280229393816,0.311448708149722,0.324570568538386,,0.319229807215414,0.0080667999016699 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_spearman,4,0.558011118128369,0.532461517776284,0.559103143514103,,0.549276269926101,0.0182973871982467 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_calibration_slope,4,0.947730194323529,0.912734464703454,0.971381900858514,,0.943678757507426,0.0317919329981155 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_prevalence,4,0.218666030567013,0.210018355611084,0.23207994955219,,0.220368260577459,0.0121728164316861 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_n,5,956,380.8,982,0,774.4,433.131388841769 +vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Climate,5,0.688882537608652,0.686220497467121,0.693315342224415,0.691923775211569,0.689606019273988,0.00355153250685828 +vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Climate_Truncated,5,0.688853782361867,0.686211230349888,0.693311107621612,0.691928871889894,0.689595671925047,0.00355473297057612 +vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Landcover,5,0.795978634172116,0.790420342908561,0.808241109353126,0.795978634172116,0.79826909072766,0.00925670982213445 +vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Full,5,0.890081979393699,0.886120541286867,0.900531693288428,0.890081979393699,0.8925816929913,0.0074671519015807 +vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Full_Truncated,5,0.890081979393699,0.886107519131847,0.900531693288428,0.890081979393699,0.892577213245043,0.00747321026543204 +vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Full_Joint,5,0.885428136469803,0.879749497228007,0.894203850953663,0.88338493164256,0.886730028942607,0.00724717960067507 +vascular_plant,vascular_plant,south,,,Galium.boreale,v2val_Full_Joint_Truncated,5,0.885423965632299,0.87972024605796,0.894203850953663,0.88338493164256,0.886719444385091,0.00726319634118945 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Climate,4,0.69847723797285,0.691752649872066,0.699715919161363,,0.696517985504182,0.00490694367651728 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Climate_Truncated,4,0.698467309951771,0.691745994349179,0.699715497383917,,0.696512621319469,0.0049068149193327 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Landcover,4,0.785174421252252,0.769559195023864,0.799812878397794,,0.784825575151236,0.016225503840661 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Full,4,0.884065848019888,0.870449780280702,0.894068384737495,,0.882775301226467,0.0128406847119162 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Full_Truncated,4,0.884061344921756,0.870449780280702,0.894063477626334,,0.882772262087301,0.0128382393423392 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Full_Joint,4,0.873729773605152,0.862934967954651,0.88650495339372,,0.874437050083033,0.0129679425986572 +vascular_plant,vascular_plant,south,,,Galium.boreale,oob_v2val_Full_Joint_Truncated,4,0.873725270507021,0.862934967954651,0.886486814768876,,0.874429285403265,0.0129581962884508 +vascular_plant,vascular_plant,south,,,Vicia.americana,insample_auc,5,0.805930865573238,0.802524114144756,0.807930971919586,0.808012980346143,0.805492289441972,0.00279684596223116 +vascular_plant,vascular_plant,south,,,Vicia.americana,insample_deviance_explained,5,0.166618669672263,0.15835736048416,0.197983766968233,0.179903957587885,0.174785331031235,0.0215605448558036 +vascular_plant,vascular_plant,south,,,Vicia.americana,insample_rmse,5,0.381878187765555,0.378497935845721,0.383275373980955,0.380313032587985,0.381159180017538,0.00249488775218777 +vascular_plant,vascular_plant,south,,,Vicia.americana,insample_spearman,5,0.465238638302507,0.460583361067045,0.473614426763325,0.470529100145484,0.466694650560087,0.00654368164273618 +vascular_plant,vascular_plant,south,,,Vicia.americana,insample_calibration_slope,5,0.933688472158773,0.920475046205582,0.947214542702693,0.933688472158773,0.933156328591101,0.0138732265313946 +vascular_plant,vascular_plant,south,,,Vicia.americana,insample_prevalence,5,0.264350453172205,0.25911663216011,0.270295509723619,0.264350453172205,0.264443478235907,0.00620048832939598 +vascular_plant,vascular_plant,south,,,Vicia.americana,insample_n,5,1685,1649.4,2266.4,2648,1865.6,437.838212128635 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_auc,4,0.807298000134658,0.803023991535991,0.814183808910683,,0.808230785912286,0.0060785023634324 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_deviance_explained,4,0.201855986026952,0.138430007183509,0.206184030963771,,0.180749581060301,0.0455406997595895 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_rmse,4,0.37987633843542,0.378039277925496,0.38526074563498,,0.381143247967433,0.00422553209428043 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_spearman,4,0.471049244528984,0.458945127185501,0.481459447588745,,0.47044427514194,0.0124124270312768 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_calibration_slope,4,0.934673431866213,0.911850976434992,0.962640657947191,,0.936510849955412,0.0273611637238912 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_prevalence,4,0.266148052343275,0.252246321440467,0.273763588267456,,0.263902982708051,0.0122396324927964 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_n,5,963,381.6,998.6,0,782.4,437.838212128635 +vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Climate,5,0.688388374288585,0.686629323150551,0.697141250977293,0.688388374288585,0.69087203080297,0.00615433602729216 +vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Climate_Truncated,5,0.688377053340582,0.686621028690138,0.697132635984666,0.688377053340582,0.690862704228311,0.00615461616566359 +vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Landcover,5,0.761379244969105,0.759537679717784,0.765001614229777,0.76658587562335,0.762041460585166,0.00296099903341381 +vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Full,5,0.805930865573238,0.802524114144756,0.807930971919586,0.808012980346143,0.805492289441972,0.00279684596223116 +vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Full_Truncated,5,0.805930865573238,0.802524057680271,0.807932999886811,0.808015180404811,0.805493243742915,0.0027981951952029 +vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Full_Joint,5,0.802576290466739,0.799474271873651,0.804841725613092,0.804804561454972,0.802383816798592,0.00280990996220882 +vascular_plant,vascular_plant,south,,,Vicia.americana,v2val_Full_Joint_Truncated,5,0.802576290466739,0.799476478074435,0.804843080852285,0.804805294807862,0.802385231980135,0.00280955135311763 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Climate,4,0.687058088763642,0.673058403439112,0.691946304395709,,0.683803992444905,0.0115178096712181 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Climate_Truncated,4,0.687038471404578,0.673051662415016,0.691940326726772,,0.683793845094804,0.0115133321710675 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Landcover,4,0.774223898160634,0.769748999911181,0.777668369854279,,0.773855888676417,0.00434521003225348 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Full,4,0.807298000134658,0.803023991535991,0.814183808910683,,0.808230785912286,0.0060785023634324 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Full_Truncated,4,0.807292347351227,0.803023991535991,0.814176792476235,,0.808226664961859,0.00607512490581962 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Full_Joint,4,0.805961431482898,0.801581678357063,0.812743710082227,,0.80681947629486,0.00606828197945081 +vascular_plant,vascular_plant,south,,,Vicia.americana,oob_v2val_Full_Joint_Truncated,4,0.805961431482898,0.801581678357063,0.812736460553446,,0.806816887177438,0.00606401787708264 diff --git a/3_output/exp_000_parity_v2/tables/parity_summary.csv b/3_output/exp_000_parity_v2/tables/parity_summary.csv index f85e0b6..bc9bbba 100644 --- a/3_output/exp_000_parity_v2/tables/parity_summary.csv +++ b/3_output/exp_000_parity_v2/tables/parity_summary.csv @@ -1,23 +1,23 @@ taxon,region,stage,comparison,species,terms_compared,min_draws,in_band_pct,reachable_terms,reachable_in_band_pct,median_standardized_difference,median_absolute_difference,max_absolute_difference,negligible_terms,median_spearman,median_band_ratio,verdict,indicative_verdict -bird,north,climate,median,2,20,100,100,20,100,0,0,2.2e-11,0,1,1,not gated (trial run),pass -bryophyte,north,climate,median,1,19,100,100,19,100,0.035,0,1.4,6,1,1.062,not gated (trial run),pass -lichen,north,climate,median,2,38,100,100,38,100,0,0,0.644,17,0.993,0.879,not gated (trial run),pass -mite,north,climate,median,2,38,100,100,38,100,0,0,1.95,20,0.905,0.953,not gated (trial run),pass -vascular_plant,north,climate,median,2,38,100,100,38,100,0,0,0.266,24,1,0.97,not gated (trial run),pass -bird,south,climate,median,2,20,100,100,20,100,0,0,2.2e-11,0,1,1,not gated (trial run),pass -bryophyte,south,climate,median,1,19,100,100,19,100,0.035,0,1.4,6,1,1.062,not gated (trial run),pass -lichen,south,climate,median,2,38,100,100,38,100,0,0,0.644,17,0.993,0.879,not gated (trial run),pass -mite,south,climate,median,2,38,100,100,38,100,0,0,1.95,20,0.905,0.953,not gated (trial run),pass -vascular_plant,south,climate,median,2,38,100,100,38,100,0,0,0.266,24,1,0.97,not gated (trial run),pass -bird,north,habitat,median,2,178,100,100,168,100,0,0,2.71e-11,0,1,1,not gated (trial run),pass -bryophyte,north,habitat,median,1,94,100,100,94,100,0.072,0.023,0.0808,0,0.997,0.911,not gated (trial run),pass -lichen,north,habitat,median,2,188,100,100,188,100,0.091,0.024,0.118,0,0.999,0.988,not gated (trial run),pass -mammal,north,habitat,iteration 1,2,448,65,100,436,100,0,0,9.77e-15,0,1,,not gated (trial run),pass -mite,north,habitat,median,1,94,100,100,94,100,0.14,0.035,0.148,0,0.999,0.993,not gated (trial run),pass -vascular_plant,north,habitat,median,2,188,100,100,188,100,0.108,0.022,0.125,0,0.999,0.95,not gated (trial run),pass -bird,south,habitat,median,2,46,100,100,40,100,0,0,5.61e-11,0,1,1,not gated (trial run),pass -bryophyte,south,habitat,median,1,19,100,100,19,100,0.047,0.021,0.0601,0,0.993,1.042,not gated (trial run),pass -lichen,south,habitat,median,2,40,100,100,40,100,0.088,0.029,0.278,1,0.994,0.961,not gated (trial run),pass -mammal,south,habitat,iteration 1,2,126,99,100,120,100,0,0,5.33e-15,0,1,,not gated (trial run),pass -mite,south,habitat,median,2,40,100,100,40,100,0.068,0.024,0.464,1,0.991,0.975,not gated (trial run),pass -vascular_plant,south,habitat,median,2,40,100,100,40,100,0.074,0.027,0.184,0,0.997,1.021,not gated (trial run),pass +bird,north,climate,median,2,20,5,90,20,90,0.351,0,0.651,0,1,0.694,not gated (store holds 5 draws), +bryophyte,north,climate,median,1,19,5,94.7,19,94.7,0.671,0,1.81,5,0.991,0.214,not gated (store holds 5 draws), +lichen,north,climate,median,2,38,5,89.5,38,89.5,0.003,0,1.99,17,0.991,0.301,not gated (store holds 5 draws), +mite,north,climate,median,2,38,5,94.7,38,94.7,0.082,0,3.79,20,0.973,0.656,not gated (store holds 5 draws), +vascular_plant,north,climate,median,2,38,5,97.4,38,97.4,0,0,0.12,24,0.982,0.602,not gated (store holds 5 draws), +bird,south,climate,median,2,20,5,90,20,90,0.351,0,0.651,0,1,0.694,not gated (store holds 5 draws), +bryophyte,south,climate,median,1,19,5,94.7,19,94.7,0.671,0,1.81,5,0.991,0.214,not gated (store holds 5 draws), +lichen,south,climate,median,2,38,5,89.5,38,89.5,0.003,0,1.99,17,0.991,0.301,not gated (store holds 5 draws), +mite,south,climate,median,2,38,5,94.7,38,94.7,0.082,0,3.79,20,0.973,0.656,not gated (store holds 5 draws), +vascular_plant,south,climate,median,2,38,5,97.4,38,97.4,0,0,0.12,24,0.982,0.602,not gated (store holds 5 draws), +bird,north,habitat,median,2,178,5,92.7,168,92.3,0.569,0.023,2.82,0,0.989,0.699,not gated (store holds 5 draws), +bryophyte,north,habitat,median,1,94,5,98.9,94,98.9,0.36,0.062,0.256,0,0.987,0.502,not gated (store holds 5 draws), +lichen,north,habitat,median,2,188,5,94.7,188,94.7,0.251,0.038,0.289,0,0.996,0.642,not gated (store holds 5 draws), +mammal,north,habitat,iteration 1,2,448,4,100,436,100,0,0,9.77e-15,0,1,,not gated (trial run), +mite,north,habitat,median,1,94,5,76.6,94,76.6,0.579,0.078,0.673,0,0.993,0.642,not gated (store holds 5 draws), +vascular_plant,north,habitat,median,2,188,5,91.5,188,91.5,0.303,0.034,0.326,0,0.997,0.596,not gated (store holds 5 draws), +bird,south,habitat,median,2,46,5,80.4,40,77.5,0.334,0.021,0.121,0,0.995,0.508,not gated (store holds 5 draws), +bryophyte,south,habitat,median,1,19,5,100,19,100,0.399,0.066,0.267,0,0.991,0.633,not gated (store holds 5 draws), +lichen,south,habitat,median,2,40,5,87.5,40,87.5,0.337,0.048,0.655,1,0.985,0.554,not gated (store holds 5 draws), +mammal,south,habitat,iteration 1,2,126,5,100,120,100,0,0,5.33e-15,0,1,,not gated (trial run), +mite,south,habitat,median,2,40,5,95,40,95,0.175,0.03,0.754,1,0.992,0.581,not gated (store holds 5 draws), +vascular_plant,south,habitat,median,2,40,5,75,40,75,0.585,0.07,0.552,0,0.977,0.566,not gated (store holds 5 draws), diff --git a/3_output/exp_000_parity_v2/tables/v2_self_agreement_summary.csv b/3_output/exp_000_parity_v2/tables/v2_self_agreement_summary.csv deleted file mode 100644 index 6d46fa1..0000000 --- a/3_output/exp_000_parity_v2/tables/v2_self_agreement_summary.csv +++ /dev/null @@ -1,2 +0,0 @@ -stage,taxon,species,terms,in_band_pct_published_in_rerun,in_band_pct_rerun_in_published,in_band_pct,median_standardized_difference,median_spearman,p10_spearman -climate,bryophyte,134,2546,100,100,100,0.019,0.998,0.887 diff --git a/README.md b/README.md index 394956b..dc4722d 100644 --- a/README.md +++ b/README.md @@ -10,854 +10,276 @@ > [!IMPORTANT] > This repository is developed by and for the Science Centre at the Alberta Biodiversity Monitoring Institute (ABMI). It is intended for internal use. -> - -Shared pipeline and static cross-taxa test dataset for methods R&D on ABMI -species models. - -## Contents - -- [Overview](#overview) -- [Status](#status) -- [Pipeline flow](#pipeline-flow) -- [Directory structure](#directory-structure) - - [`0_data/`](#0_data) - - [`1_code/`](#1_code) - - [`2_pipeline/`](#2_pipeline) - - [`3_output/`](#3_output) - - [`docs/`](#docs) -- [Parity gate](#parity-gate) -- [Known gaps](#known-gaps) -- [Naming conventions](#naming-conventions) -- [Adding an experiment](#adding-an-experiment) -- [Contributing a taxon module](#contributing-a-taxon-module) -- [Setup](#setup) -- [Related resources](#related-resources) -- [Contact](#contact) - -## Overview - -A shared environment for testing methods across taxa. It holds a static -cross-taxa test dataset and a companion R&D pipeline: a taxon-agnostic -harness, and one taxon spec per taxon that states that taxon's v2 -configuration as data. - -Because the dataset and pipeline are fixed, R&D questions can be run once -across all taxa rather than reimplemented per taxon, and results stay -comparable over time. - -This repository is for methods research and development. It is not for the -final species models and does not generate reporting products. +> + +The ABMI Science Centre's shared R&D repository for Species Models 3.0. +It holds a frozen cross-taxa test dataset, one modelling pipeline that +runs every taxon's v2.0 model as configuration, and a v2 parity check +(`exp_000`) that every later experiment is compared against. A methods +question is asked once and answered for every taxon. It is for +Science Centre modellers testing methods; it does not produce the +final species models or reporting products. ## Status | Component | State | | --- | --- | -| Test dataset | Built from the `model_ready_v2` snapshot and the bird `Stratified.Rdata`. Responses, covariates and lookups for the four plant-group taxa, mammals and birds. Validated by `_setup/02` | -| v2 reference | Harmonized into `0_data/v2_results/v2_results.csv` by `_setup/05`. Bryophyte model reference regenerated by `_setup/03`; bird coefficients re-packaged by `_setup/04` | -| Harness | Built: loading, resampling, fitting engines, selection rules, metrics, result stores, run records | -| Taxon specs | Every stage of every taxon runs, and every one is partial against v2. The detail is each spec's `v2_coverage`, printed in `report.md` | -| v2.0 parity check (`exp_000`) | Runs end to end. Not yet able to gate; see [Known gaps](#known-gaps) | +| Test dataset | Built for bryophytes, lichens, soil mites, vascular plants, mammals and birds. Validated by `_setup/02`. Published to ABMI-DATA2, version 2026-10-05 | +| v2 reference | ABMIexploreR's published coefficients, harmonized by `_setup/08` | +| Pipeline | Built: plug-in methods, parallel species, final-model metrics and habitat grids for every taxon, experiment runner, contract tests | +| Taxon specs | Every stage of every taxon runs. Each spec's `v2_coverage` states what it reproduces; `report.md` prints it | +| exp_000 (v2 parity) | A 100-draw run on the `parity_check` species passed all 22 taxon × region × stage rows on 2026-10-03 (see the [exp_000 README](1_code/experiments/exp_000_parity_v2/README.md)). The committed tables are from a later 5-draw trial. Not yet a gate: see [Parity gate](#parity-gate) | -## Pipeline flow +## Repository structure -Four parts, each with one job. +``` +sdmMethodsDev/ +├── 0_data/ # gitignored except v2_scripts/ and *.R +│ ├── test_dataset/ # _setup/ output; runs read the published copy +│ ├── v2_results/ # the v2 parity reference, from _setup/08 +│ ├── external/ # downloaded sources (ABMIexploreR), from _setup/08 +│ ├── covariates/ # reserved; empty +│ └── v2_scripts/ # v2 scripts as received; reference only +├── 1_code/ +│ ├── harness/ # the shared pipeline; names no taxon or method +│ ├── methods/ # engines, selection rules, resampling, metrics +│ ├── modules/ # one v2 spec per taxon; _shared/ for common code +│ ├── experiments/ # one folder per question; _template/, _shared/ +│ ├── _setup/ # 00-10: build, check and publish the dataset +│ ├── tests/ # contract tests and the store comparison +│ └── _scratch.R # dated workspace for exploration +├── 2_pipeline/ # result stores and intermediates; gitignored +├── 3_output// # per-experiment outputs; summaries committed +└── docs/ # getting started, design, taxon quirks, reviews +``` +What each folder is for, and what an edit there affects, is in +[`docs/getting_started.md`](docs/getting_started.md#where-things-are). + +## How it works + +The v2.0 pipelines share one sequence: pick survey units, fit candidate +models, choose among or average them, predict and score. Here each +taxon's contents of those steps are data, in a **spec** +(`1_code/modules//spec.R`). One **harness** runs any spec, and +looks up each **method** (engine, selection rule, resampling scheme, +metric) by the name the spec gives. + +```mermaid +%% ---------------------------- +%% Taxon as configuration, not codebase +%% ---------------------------- +%%{init: {"themeVariables": {"edgeLabelBackground": "#ffffff"}}}%% +flowchart TD + + subgraph fw["framework: one pipeline, taxon as configuration"] + direction TB + PARAM["response · family · offset · regions · stages · resampling"] + PARAM --> SP["plant-group specs
bryophytes · lichens · mites · vascular plants"] + PARAM --> SM["mammal specs
summer · winter"] + PARAM --> SB["bird spec"] + SP --> H["1_code/harness/"] + SM --> H + SB --> H + MT["1_code/methods/
engines · selection rules · resampling · metrics"] + MT -.->|"looked up by name"| H + H -->|"the same result files
for every taxon and method"| R["result stores"] + end + + %% ---------------------------- + %% Styles + %% ---------------------------- + + %% Containers + style fw fill:none,stroke:#2D415B,stroke-width:1px + + %% Configuration + style PARAM fill:#ffffff,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + style SP fill:#A3B4C7,stroke:#2D415B,stroke-width:1px + style SM fill:#A3B4C7,stroke:#2D415B,stroke-width:1px + style SB fill:#A3B4C7,stroke:#2D415B,stroke-width:1px + style MT fill:#A3B4C7,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + + %% Shared pipeline + style H fill:#A4A88780,stroke:#C8A02C,stroke-width:4px + style R fill:#E8A396,stroke:#2D415B,stroke-width:4px ``` -0_data/test_dataset/ (a) the frozen dataset - │ responses, covariates, lookups; read only - │ read by - ▼ -1_code/harness/ (b) the harness - ▲ load, resample, fit, select, score, store; - │ configured by names no taxon - │ -1_code/modules//spec.R (c) taxon specs - each taxon's v2 configuration, as data - -1_code/experiments/exp_NNN_*/run.R (d) experiments - choose specs, species, draws and seed; - call the harness; summarize the results + +### Everything is compared with one v2 baseline + +```mermaid +%% ---------------------------- +%% Everything comparable to one v2 baseline +%% ---------------------------- +%%{init: {"themeVariables": {"edgeLabelBackground": "#ffffff"}}}%% +flowchart TD + + subgraph fw["framework: everything comparable to one v2 baseline"] + direction TB + D["0_data/test_dataset/"] --> M["1_code/modules/[taxon]/
v2 specs"] + V2["0_data/v2_results/
published v2.0"] + M --> E0["exp_000_parity_v2"] + M --> E1["exp_001"] + M --> E2["exp_002"] + V2 -->|"parity check"| E0 + E0 --> O0["3_output/exp_000_parity_v2/"] + E1 --> O1["3_output/exp_001/"] + E2 --> O2["3_output/exp_002/"] + O0 --> C["compare_experiments()"] + O1 --> C + O2 --> C + end + + %% ---------------------------- + %% Styles + %% ---------------------------- + + %% Container + style fw fill:none,stroke:#2D415B,stroke-width:1px + + %% Highlighted borders + style D fill:#A4A88780,stroke:#B8860B,stroke-width:4px + style M fill:#A4A88780,stroke:#B8860B,stroke-width:4px + style V2 fill:#ffffff,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + style E0 fill:#A3B4C7,stroke:#B8860B,stroke-width:4px + style E1 fill:#A3B4C7,stroke:#2D415B,stroke-width:1px + style E2 fill:#A3B4C7,stroke:#2D415B,stroke-width:1px + style O0 fill:#ffffff,stroke:#B8860B,stroke-width:4px,stroke-dasharray: 5 5 + style O1 fill:#ffffff,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + style O2 fill:#ffffff,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + style C fill:#E8A396,stroke:#2D415B,stroke-width:4px ``` -An experiment run does three things: +exp_000 runs every taxon's v2 spec and checks it against the published +v2.0 output. Every later experiment changes one thing and is compared +with exp_000, so a difference is a difference in method rather than in +plumbing. Because the dataset is frozen, results stay comparable. -1. **Configures.** `run.R` is the only file to edit. It sets where the run - reads and writes (`data_dir`, `pipeline_dir`, `out_dir`), which taxa run, - the focal species, any replacement candidate models, the number of - bootstrap draws and the random seed. -2. **Fits.** For each taxon, `run_spec()` loops region → species → draw → - stage, doing what the spec says, and writes one result store per region - to `pipeline_dir///`. A failed species or draw is logged - and the run continues. -3. **Summarizes.** The experiment's numbered scripts read the stores and - write the deliverables to `out_dir`. +The design, the spec and result contracts, and the parity ledger are in +[`docs/framework_design.md`](docs/framework_design.md). Every +taxon-specific v2 behaviour is in +[`docs/taxon_quirks.md`](docs/taxon_quirks.md). -Nothing outside `pipeline_dir` and `out_dir` is written during a run, and -`0_data/` is never written to outside `1_code/_setup/`. +## Getting started -### What a taxon spec holds +New to the repository? Read +[`docs/getting_started.md`](docs/getting_started.md) first: vocabulary, +running the baseline, reading the outputs, and testing a change. -A spec is a list returned by a function such as `lichen_spec()`. It holds no -modelling code; everything it names is looked up in a harness registry. +### Prerequisites -| Field | What it sets | -| --- | --- | -| `taxon` | The data slug that keys the response file, covariates and lookups | -| `response_transform`, `family`, `weight_column` | How the recorded value becomes a response, and the error family and weights | -| `regions` | Per region: the row filter, the prediction grid, and the habitat candidate models | -| `stages` | In order: candidate models, engine, selection rule, information criterion, and which stage's prediction is carried into the next as `Climate` | -| `resample` | The resampling scheme and its settings | -| `v2_coverage` | Per stage, whether the spec reproduces v2 (`reproduced`, `partial`, `not reproduced`) and what differs. **The one place coverage is stated**; the report prints it | -| `notes` | Configuration facts, such as whether `Protocol` is fitted | - -| Registry | File | Entries | -| --- | --- | --- | -| Engines | `harness/engines.R` | `glm`, `bayesglm` | -| Selection rules | `harness/selection.R` | `single`, `aic_best`, `aic_average`, `staged_bic`, `ivw_grid`, `aic_best_grid`, `aic_best_onehot` | -| Resampling schemes | `harness/resample.R` | `precomputed`, `spatial_block`, `spatial_cv` | -| Candidate model sets | `harness/model_sets.R` | The v2 formulas, spliced verbatim from `0_data/v2_scripts/` | -| Metrics | `harness/eval_metrics.R` | `auc`, `deviance_explained`, `rmse`, `spearman`, `calibration_slope`, `prevalence`, `n` | - -The covariates a run loads are read off the formulas it fits, so a term can -never be fitted without its column being loaded. - -### Where the modelling code came from - -`0_data/v2_scripts//` holds the v2 scripts as received from the taxon -leads. They are kept unmodified as the reference the harness is checked -against, and are **not run** by any experiment. The v2 method is restated in -this repository rather than copied: - -- **The configuration moved into specs.** What differed between v2 scripts — - response, family, candidate models, selection rule, region filters — is a - field in a spec. What they shared is harness code, written once. -- **Formulas are spliced, not retyped.** The candidate sets in - `model_sets.R` are copied from the v2 source. The 58-model plant climate - set is rebuilt the way v2 builds it: 14 base models, a bioclim variant of - each, and three spatial variants of ten of them. -- **Paths are arguments.** v2 hard-coded paths relative to each v2 project - root. Every path is now set by the calling experiment. -- **Inputs come from the test dataset.** v2 loaded per-taxon `.Rdata` files; - the harness reads the harmonized CSVs. -- **Run length and seed are configurable.** v2 hard-coded 100 bootstrap - draws and set no seed. Both are settings in `run.R`. - -Every taxon-specific behaviour in the v2 pipelines, and whether the harness -reproduces it, is listed in [`docs/taxon_quirks.md`](docs/taxon_quirks.md). - -## Directory structure +R 4.4 or later, and these packages: -``` -sdmMethodsDev/ -├── README.md -│ -├── 0_data/ # read-only inputs; written only by 1_code/_setup/ -│ ├── covariates/ # reserved for the covariate catalogue; empty -│ ├── test_dataset/ # frozen: built by 1_code/_setup/01_ -│ │ ├── sites.csv # one row per survey unit, all sources -│ │ ├── vascular_plant.csv # survey_unit_id + species columns -│ │ ├── bryophyte.csv -│ │ ├── lichen.csv -│ │ ├── mite.csv -│ │ ├── mammal.csv -│ │ ├── bird.csv # counts -│ │ ├── bird_offsets.csv # QPAD log offsets -│ │ ├── covariates.csv # every covariate, keyed on survey_unit_id + taxon -│ │ └── lookup/ -│ │ ├── covariate_columns.csv # block and v2 name of each covariate -│ │ ├── veg_prediction_matrix.csv # plant north habitat grid -│ │ ├── soil_prediction_matrix.csv # plant south habitat grid -│ │ ├── modelled_species.csv # plant-group work queue -│ │ ├── mammal_north_prediction_matrix.csv -│ │ ├── mammal_south_prediction_matrix.csv -│ │ ├── mammal_modelled_species.csv -│ │ ├── mammal_climate_predictions.csv -│ │ ├── bird_modelled_species.csv -│ │ ├── bird_bootstrap_ids.csv # the 100 stored v2 draws -│ │ ├── bird_factor_levels.csv -│ │ ├── bird_veg_age_matrix.csv # v2 bird translation, from _setup/06_ -│ │ └── v2_bootstrap_ids// # v2's stored plant draws, from _setup/07_ -│ ├── v2_results/ # the v2 outputs, harmonized: built by _setup/05_ -│ │ ├── v2_results.csv -│ │ └── v2_results_coverage.csv -│ └── v2_scripts/ # as-received snapshots; reference only -│ ├── README.md # sources, commits, snapshot date -│ ├── birds/ -│ ├── mammals/ -│ └── plants/ # all four plant-group taxa -│ -├── 1_code/ -│ ├── _scratch.R # temporary workspace -│ ├── _setup/ # one-off; run by hand, not per experiment -│ │ ├── 01_harmonize_model_ready_v2.R -│ │ ├── 02_validate_test_dataset.R -│ │ ├── 03_rerun_bryophyte_v2_reference.R -│ │ ├── 04_package_bird_coefficients.R -│ │ ├── 05_harmonize_v2_results.R -│ │ ├── 06_harmonize_bird_translation_lookup.R -│ │ └── 07_harmonize_v2_plant_bootstrap_ids.R -│ ├── _deprecated/ # superseded code; not sourced -│ ├── harness/ # shared, taxon-agnostic -│ │ ├── harness.R # load_harness(): sources the rest in order -│ │ ├── data_load.R -│ │ ├── covariate_sets.R -│ │ ├── species.R -│ │ ├── model_sets.R -│ │ ├── resample.R -│ │ ├── engines.R -│ │ ├── selection.R -│ │ ├── eval_metrics.R -│ │ ├── predict_grid.R -│ │ ├── result.R -│ │ ├── run_record.R -│ │ ├── run_model.R # run_spec(): the fitting loop -│ │ └── utils/ -│ │ └── step_runner.R -│ ├── modules/ # one spec per taxon -│ │ ├── _shared/ -│ │ │ └── plant_group.R # the v2 shape the four plant-group taxa share -│ │ ├── bryophytes/spec.R -│ │ ├── lichens/spec.R -│ │ ├── soil_mites/spec.R -│ │ ├── vascular_plants/spec.R -│ │ ├── mammals/ -│ │ │ ├── spec.R -│ │ │ └── hurdle.R # v2's hurdle model, both halves together -│ │ └── birds/ -│ │ ├── spec.R -│ │ └── standardize.R # raw coefficients → v2's standardized terms -│ └── experiments/ -│ └── exp_000_parity_v2/ -│ ├── README.md # question, design, coverage -│ ├── run.R # the only file to edit -│ ├── utils/ -│ │ ├── focal_species.R # named species sets -│ │ └── parity_targets.R # what pass means -│ ├── 01_collect_results.R -│ ├── 02_compare_to_v2.R -│ ├── 03_build_report.R -│ ├── 04_plot_parity.R # one parity figure per taxon -│ └── v2_self_agreement.R # v2 against itself; run once -│ -├── 2_pipeline/ # intermediates; gitignored -│ ├── exp_000_parity_v2/ -│ │ ├── run_log.csv # one row per species, region and draw -│ │ └── // # result stores -│ ├── v2_reference/ # rebuilt v2 references, from _setup/03_ and 04_ -│ └── logs/ # _setup logs -│ -├── 3_output/ # deliverables; summaries committed -│ └── exp_000_parity_v2/ -│ ├── figures/ # parity_, _mean, _bell .png; gitignored -│ ├── tables/ -│ │ ├── parity_summary.csv # committed -│ │ ├── coverage.csv # committed -│ │ ├── metric_summary.csv # committed -│ │ ├── v2_self_agreement_summary.csv # committed -│ │ ├── coefficient_summary.csv # gitignored; rebuilt -│ │ └── parity_terms.csv # gitignored; rebuilt -│ ├── report.md # committed -│ └── run_record.md # committed -│ -└── docs/ - ├── framework_design.md # the harness design and parity ledger - ├── taxon_quirks.md # every taxon-specific v2 behaviour - ├── reviews/ # code reviews - └── images/ +```r +install.packages(c( + "data.table", "arm", "mgcv", "MuMIn", "pROC", # harness, methods + "jsonlite", "ggplot2", # summaries, figures + "xgboost", # xgboost engine + "testthat", "withr" # 1_code/tests/ +)) ``` -### `0_data/` - -Read-only during experiments. The one exception is `1_code/_setup/`, which -builds `test_dataset/` and `v2_results/` by hand, once per source snapshot; -nothing in the harness, modules or experiments writes here. The data is -gitignored; `v2_scripts/` is the tracked part of this folder. - -`test_dataset/` holds the frozen cross-taxa dataset. Its contents do not -change between experiments; an amendment is a rebuild by `_setup/01`, -checked by `_setup/02`. - -Every response table is `survey_unit_id` followed by one column per species, -and every identity, design and location field is factored out into -`sites.csv`. Join any response table to `sites.csv` on `survey_unit_id`. -`sites.csv` carries an `in_` flag per taxon. - -`covariates.csv` holds everything the models regress against, for every -taxon, in one wide table keyed on `survey_unit_id` and `taxon`. The key -includes the taxon because the sources disagree: bryophyte, lichen and -vascular plant share identical values on shared survey units, but mite does -not, which points at a different spatial support or HF vintage in the mite -source. Mammals are keyed per region (`mammal_north`, `mammal_south`), -because north and south are separate models on overlapping deployments — -512 `location_project` values appear in both — and their covariates differ. -The harness reads only the columns a run needs. - -`lookup/covariate_columns.csv` records, per taxon, which block each column -came from (climate, veg, soil, design) and its v2 spelling. A column present -in more than one block was suffixed by the harmonizer; the harness renames -model terms to match, so v2 formulas run unchanged. - -Ten columns are deliberately **not** stored, because they are products of -stored columns: `TD`, `MAPPET`, `MAT2`, `CMDMAT`, `MWMT2`, `Easting`, -`Northing`, `Easting2`, `Northing2` and `EastingNorthing`. The harness -computes them on load (`harness/covariate_sets.R`). A CSV carries about 15 -significant digits, so a separately stored `Northing2` would not equal -`Northing` squared, and a model fitting both would see an inconsistent pair. - -Mammal climate comes from `abmi-camera-climate_2023.Rdata`, the file the v2 -mammal climate pipeline fitted against. It replaces the `SpTable` climate -block rather than filling gaps in it, because the two are different -extractions. It matched 4,243 of 4,654 north deployments and all 1,168 -south; unmatched deployments carry `NA`, as v2 excludes them. - -`lookup/` holds the habitat prediction matrices, the work queues — which -species each model set is fitted for — and the stored bird draws. - -- Mammals have their own `mammal_modelled_species.csv`, because their queue - is season by occurrence threshold: `sp_table_*` are the species with at - least 20 detections, which get the full habitat model, and - `sp_table_*_ua` those with at least 3, which get the use-availability - model. -- `mammal_climate_predictions.csv` is the climate offset the v2 mammal - habitat models read from a separate climate pipeline. 509 of 5,822 - deployments carry `NA`, as the v2 join produced. The harness does not - read it yet: the mammal spec fits climate itself. -- The mammal prediction matrices were checked against the ABMI Mammals - drive copies and are identical. -- `bird_bootstrap_ids.csv` holds v2's 100 stored draws, keyed on - `surveyid`, so bird runs resample exactly as v2 did. -- `bird_factor_levels.csv` restores the source order of categorical levels, - so reference levels, and every contrast, match v2. - -`v2_results/` holds the published v2 results as one long table, so a run can -be scored against a single file rather than three storage formats on two -drives. Columns are `taxon`, `region`, `stage`, `part`, `season`, `species`, -`species_v2`, `term`, `v2_median`, `v2_p10`, `v2_p90`, `v2_se`, `v2_n` and -`source`. Plant climate is fitted once province-wide in v2, so those rows -carry `region = "all"`. `v2_results_coverage.csv` records which source was -reached for each part. `exp_000` does not read this table yet (see -[Known gaps](#known-gaps)). - -`v2_scripts/` holds the modelling scripts as received from the taxon leads, -unmodified, in one subfolder per source repository. The four plant-group -taxa share `plants/` because v2 fits them with one set of scripts. All three -are cloned snapshots taken on 2026-09-08; their sources and commits are -recorded in [`0_data/v2_scripts/README.md`](0_data/v2_scripts/README.md), -which is also where a refreshed snapshot gets logged. - -`covariates/` is reserved for a versioned catalogue of covariate layers — -metadata and server paths only, with rasters on the ABMI server. It is not -built yet. - -### `1_code/` - -`harness/` holds the shared, taxon-agnostic pipeline. `harness.R` defines -`load_harness()`, which sources the other files in the order they need. -Changes here affect every experiment, past and future, and should be -reviewed accordingly. - -| File | Does | -| --- | --- | -| `data_load.R` | Reads the test dataset column-selectively and assembles one frame per taxon and region | -| `covariate_sets.R` | Named covariate bundles, derived covariates, and term renaming | -| `species.R` | Work queues and focal-species resolution | -| `model_sets.R` | The v2 candidate formulas | -| `resample.R` | Bootstrap and cross-validation schemes, and per-species seeds | -| `engines.R` | Fitting engines behind one interface | -| `selection.R` | Turning a candidate set into one set of coefficients | -| `eval_metrics.R` | Metrics computed from predictions | -| `predict_grid.R` | Projecting a fit onto a habitat prediction grid | -| `result.R` | Writing and reading result stores | -| `run_record.R` | The committed run record | -| `run_model.R` | `run_spec()`, the loop a spec drives | -| `utils/step_runner.R` | `run_step()`: times a sourced script and logs its output | - -`modules/` holds one spec per taxon, in a folder named for the taxon. A spec -is configuration only; experiment-specific logic belongs in the experiment. -The four plant-group taxa share their v2 shape through -`_shared/plant_group.R`, and each taxon's folder states only what is its -own, such as whether v2 fits `Protocol`. Soil mites use the data slug -`mite`, not `soil_mite`, wherever a taxon is named in `run.R`. - -`experiments/` holds one folder per R&D question. Each contains a -`README.md` stating the question and design, a `run.R` entry point, and -numbered scripts that turn the result stores into deliverables. Experiments -call the harness and specs; they do not reimplement them. - -`run.R` is the only file that has to be edited to change a run: - -| Section | Sets | -| --- | --- | -| 1.3 | `data_dir`, `pipeline_dir`, `out_dir` | -| 1.4 | The specs, and `run_taxa` | -| 1.5 | The focal species, by name from `utils/focal_species.R` or inline | -| 1.5b | `stage_models`: replacement candidate sets, or `NULL` for v2's | -| 1.6 | `n_bootstraps` | -| 1.6b | `boot_seed`; `NULL` leaves the draws unseeded, as v2 did | -| 1.4 (`plant_bootstrap`) | `"spatial_block"` draws v2's bootstrap afresh; `"v2_ids"` replays v2's stored draws | -| 1.7 | Which steps run: fit, collect, compare, report, record | - -Output locations are the experiment's to choose. `pipeline_dir` takes the -intermediates and `out_dir` the deliverables; both can point anywhere -writable — a scratch disk for a long run, say — and nothing else changes. - -`_setup/` holds one-off scripts that prepare the repository itself. None -runs as part of an experiment: they are run by hand, in order, when the -dataset is first built or a source is replaced. The leading underscore keeps -them sorted clear of the run-time code. - -| Script | Reads | Writes | -| --- | --- | --- | -| `01_harmonize_model_ready_v2.R` | The `model_ready_v2` snapshot, the mammal camera climate, the WildTrax species lookup, the v2 prediction matrices, the bird `Stratified.Rdata` | `0_data/test_dataset/` | -| `02_validate_test_dataset.R` | The test dataset and the same sources | Nothing; prints a pass / fail tally | -| `03_rerun_bryophyte_v2_reference.R` | The v2 plant project's bryophyte data, and the frozen v2 functions | `2_pipeline/v2_reference/bryophyte-species-models.Rdata` | -| `04_package_bird_coefficients.R` | The per-draw v2 bird coefficient CSVs | `2_pipeline/v2_reference/Birds2024.RData` | -| `05_harmonize_v2_results.R` | v2 plant `COEFS.RData`, the mammal coefficient tables, `Birds2024.RData` | `0_data/v2_results/` | -| `06_harmonize_bird_translation_lookup.R` | v2's bird `Xn-veg-v2024.Rdata` | `0_data/test_dataset/lookup/bird_veg_age_matrix.csv` | -| `07_harmonize_v2_plant_bootstrap_ids.R` | v2's stored plant bootstrap draws | `0_data/test_dataset/lookup/v2_bootstrap_ids/` | - -- **`01`** writes every response, covariate and lookup table described - under [`0_data/`](#0_data), so experiments run from - `0_data/test_dataset/` alone. Mammal species names are resolved against - `WildTrax Species Strings.RData`, so the north and south files reach the - same column name; a species the lookup lacks keeps its source spelling - and is reported (as of the 2024 snapshot, `Raccoon` alone). -- **`02`** is run after `01`, and again whenever the snapshot or the - harmonizer changes. Its first pass describes the tables (sizes, coverage, - value ranges, joins). Its second traces every written value back to its - source, restating the rules rather than calling the harmonizer's - functions, so a bug there shows up rather than cancelling itself out. It - writes nothing, so it is safe to re-run. -- **`03`** exists because the published bryophyte reference is unusable: - every draw is an error object, and its bootstrap-id file holds 3 draws - instead of 100. It re-runs the frozen v2 functions unmodified and writes to - `2_pipeline/`, never to the v2 project. It covers the climate stage only. -- **`04`** rebuilds the packaged bird output, which is absent from the bird - drive. It is a clone of v2's `08.PackageCoefficients.R` with only the paths - changed, and puts bird coefficients on the same standardized term names - as the plant tables. Reads are serial, because parallel reads trip Google - Drive's rate limiting. -- **`05`** flattens the three v2 storage formats into - `0_data/v2_results/v2_results.csv`. Plants come from `COEFS.RData`, v2's - own standardization output, which holds complete bryophyte arrays. Mammal - results average the two seasons, as v2's `.all` tables do. Note that - `Birds2024.RData` labels draws 2 to 100 in file-listing order (`b2` is draw - 10, `b3` draw 100), a v2 packaging quirk: medians and bands are unaffected, - but a draw-by-draw comparison has to remap them. -- **`06`** copies the `age` matrix v2 uses to translate bird landcover - coefficients onto the standardized habitat types, so - `modules/birds/standardize.R` can do the same without the bird drive. Re-run - it whenever `01` rebuilds `lookup/`. -- **`07`** copies v2's stored plant bootstrap draws, one file per species, so a - run with `plant_bootstrap = "v2_ids"` fits exactly the rows v2 fitted. The - full sets are large (1.3 GB compressed for vascular plants), so - `SDM_V2_BOOT_TAXA` and `SDM_V2_BOOT_SPECIES` restrict it to what a run needs. - Bryophytes come from `03`'s regenerated draws. - -`_deprecated/` holds code superseded by the harness: the earlier -plant-only modules, placeholder harness files and the first version of -`exp_000`. Nothing sources it; its -[`README.md`](1_code/_deprecated/README.md) records what replaced each file. - -`_scratch.R` is a dated workspace for exploration. Work that proves durable -is promoted into a proper script and removed from it. - -### `2_pipeline/` - -Intermediate and cached files. Gitignored. - -Each experiment writes one result store per taxon and region to -`///`: - -| File | Holds | -| --- | --- | -| `coefficients.csv` | Final-stage coefficients per species, draw and term | -| `metrics.csv` | Metrics per species and draw | -| `unit_predictions.csv` | Predictions at every survey unit | -| `grid_predictions.csv` | Predictions onto the habitat grid, where the spec names one | -| `meta.json` | What produced the store: stages, engines, selection rules, covariates, resampling scheme, the seed actually used, any model overrides, and the spec's notes | +`_setup/03` also needs `foreach`, `AICcmodavg`, `binom`, `mapproj` and +`RcmdrMisc` (v2's functions). `exp_002_soilgrids` also needs `sf`, +`terra` and [sciSpatialR](https://github.com/ABbiodiversity/sciSpatialR). -`/run_log.csv` records the outcome of every species, region and -draw — `ok`, or why not. The stores are too large to commit: 100 draws by -every species by every term. +Runs need read access to `//ABMI-DATA2/science/sdmMethodsDev/` (ABMI +network or VPN). They read the published test dataset and v2 results +from there, at the versions pinned in +[`1_code/harness/data_source.R`](1_code/harness/data_source.R); nothing +in `_setup/` has to be run. To read another copy, such as a local one, +set `SDM_TEST_DATASET` and `SDM_V2_RESULTS` to its folders. -`v2_reference/` holds v2 references rebuilt by `_setup/03` and `04`, which -`_setup/05` reads. `logs/` holds the logs of the long `_setup` runs. +### Run the v2 baseline (exp_000) -### `3_output/` +From the repository root: -Deliverables, organized by experiment. Only derived artefacts belong here: -summary tables, figures, and the report. +```r +source("1_code/harness/harness.R") +load_framework() # the harness, every method, every taxon module +list_methods() # engines, selection rules, resampling, metrics +names(standard_specs()) # the runs an experiment can choose -Only the small summaries are committed, so experiments can be compared across -time from the repository alone without large files changing on every run: +# Check species, n_bootstraps and workers in run.R first +source("1_code/experiments/exp_000_parity_v2/run.R") +``` -| Committed | Gitignored | -| --- | --- | -| `run_record.md`, `report.md` | `figures/` | -| `tables/parity_summary.csv` | `tables/coefficient_summary.csv` | -| `tables/coverage.csv` | `tables/parity_terms.csv` | -| `tables/metric_summary.csv` | | +### Check that a change broke nothing -The gitignored tables hold per-species, per-term detail. They grow with the -species list — a full plant run is tens of thousands of rows — and are rebuilt -from the `2_pipeline/` stores by re-running `run.R` with `run_models <- FALSE`. -Keep the stores for as long as that detail may be needed. +```sh +Rscript 1_code/tests/run_tests.R +``` -**Commit only runs that count.** A trial run writes to the same files as a -full run, so committing after a smoke test would record trial numbers as the -experiment's result. Commit `3_output/` after a run with the full species -list and `n_bootstraps` equal to `v2_bootstraps`, and check that -`run_record.md` describes that run. +Then compare stores at 5 draws with `1_code/tests/compare_stores.R`, as +in [`docs/getting_started.md`](docs/getting_started.md#check-that-nothing-broke). -Output folders are named by experiment (`3_output//`) rather than by -version (`3_output/v0.1/`) as the Science Centre code standards suggest. Each -experiment is its own record, and a rerun replaces it; its history is in git. +## Outputs: `2_pipeline/` and `3_output/` -#### Output key +Each run writes result stores to `2_pipeline////` +(`coefficients.csv`, `metrics.csv`, `grid_predictions.csv`, +`meta.json`) and `2_pipeline//run_log.csv`. It then writes +summary tables and `run_record.md` to `3_output//`. Every +experiment except exp_000 also writes `tables/comparison_metrics.csv` +against exp_000. The file and column definitions are in +[`1_code/harness/`](1_code/harness/). -A run writes its outputs in this order: +Committed per experiment, by `.gitignore`: `run_record.md`, `report.md`, +and `tables/` `coverage.csv`, `metric_summary.csv`, `grid_summary.csv`, +`parity_summary.csv` and `v2_self_agreement_summary.csv`. Everything else +is rebuilt from the stores with `run_experiment(config, fit = FALSE)`. -``` -2_pipeline stores ─► 01_collect ─► coverage · metric_summary · coefficient_summary - 02_compare ─► parity_terms · parity_summary - 04_plot ─► figures/parity_.png · parity_mean.png · parity_bell.png - 03_report ─► report.md - run.R 3.5 ─► run_record.md -``` +**Commit only runs that count:** every species at `n_bootstraps = 100`. +Check that `run_record.md` says so. -A *draw* is one bootstrap iteration. Every summary reduces the draws to a -median and a 10th–90th percentile band (`p10`, `p90`) rather than a mean and -standard deviation, because the spread over draws is often skewed. +## Parity gate -**`run_record.md`** — which run the other files came from. Read it first: if -its configuration is not a full run (full species list, `n_bootstraps` equal -to `v2_bootstraps`), the tables describe a trial. +exp_000 runs every taxon's v2 configuration and compares it with the +published v2 coefficients. v2 sets no seed, so parity is distributional: +a term passes when the v2 median falls inside the run's 10–90% band. +Mammals, which v2 fits once, are compared on the full-data fit. The +proposed targets are in +`1_code/experiments/exp_000_parity_v2/utils/parity_targets.R`. -| Section | Holds | -| --- | --- | -| Provenance | Git commit, R version, time written, pipeline folder. Volatile; ignored when comparing records | -| Configuration | Taxa, focal species, `n_bootstraps`, `boot_seed`, candidate model sets, engines, selection rules, jobs ok / not ok | -| Coverage | Rows written per taxon and region | -| Metrics | A short metric summary | +Before the gate is closed: -**`tables/coverage.csv`** — what ran. One row per run × region. +- Agree the parity targets, per taxon, or change them. +- Run the full species queues at 100 draws and commit the report and + record. Only a full run is gated. -| Column | Meaning | -| --- | --- | -| `taxon` | Data slug, from the store's `meta.json` | -| `run` | The spec's key in `run.R`, which names the store; the two mammal runs are `mammal_summer` and `mammal_winter` | -| `region` | Model region (`north`, `south`) | -| `season`, `part` | Mammals only: season, and hurdle part (`presence`, `abundance`) | -| `species` | Species that produced output | -| `draws` | Draws that produced output | -| `stages` | Stages with stored coefficients | -| `has_coefficients` | Whether coefficients were stored | -| `has_grid` | Whether predictions onto the habitat prediction grid were stored | - -A row with `species = 0` means that taxon and region produced nothing; the -reason is in `2_pipeline//run_log.csv`. - -**`tables/metric_summary.csv`** — how well the models fit. One row per -taxon × region × species × metric, with `n`, `median`, `p10` and `p90` over -draws. Each harness metric comes twice, prefixed by where it is scored: - -| Prefix | Scored on | -| --- | --- | -| `insample_` | The units the draw fitted | -| `oob_` | The units the draw left out: the held-out read. NA for iteration 1, the full data | -| `v2val_` | v2's own validation (plants): seven AUCs scored from the coefficients on the draw's units, as v2 does | -| `oob_v2val_` | The same seven, on the units the draw left out | +Known gaps: published lichen, mite and vascular plant models were +fitted on draws v2 did not keep, so parity against them stays +distributional; and no parity target has been agreed. Each spec's +`v2_coverage` is the authoritative statement per stage. Details are in +the [exp_000 README](1_code/experiments/exp_000_parity_v2/README.md). -| Metric | Meaning | -| --- | --- | -| `auc` | Separation of detections from non-detections; 0.5 is chance, 1 is perfect | -| `deviance_explained` | Share of the null deviance the model explains | -| `rmse` | Typical prediction error | -| `spearman` | Rank agreement between observed and predicted | -| `calibration_slope` | 1 is well calibrated; below 1, predictions are over-confident | -| `prevalence` | Observed detection rate | -| `n` | Survey units scored | -| `Climate`, `Climate_Truncated`, `Landcover`, `Full`, `Full_Truncated`, `Full_Joint`, `Full_Joint_Truncated` | v2's validation AUCs: climate alone (and capped at its 99th percentile), landcover alone, the two combined, and combined per habitat type | - -The harness metrics come from the final stage's best single candidate model; -the v2 validation AUCs from the coefficients, as v2 reports them. - -**`tables/coefficient_summary.csv`** (gitignored) — what the models -estimated. One row per run × region × stage × species × term, with the -labels from `coverage.csv`, `n`, `median`, `p10` and `p90` over draws, and -`boot1`, the iteration-1 (full-data) value. Every stage is stored, and a term -means something different by taxon and stage: - -| Taxon and stage | `term` holds | -| --- | --- | -| Any, `climate` | Model-averaged climate coefficients, on the link scale | -| Plant group, `habitat` | Effect of each habitat type on the logit scale, plus `Intercept`, `Climate` and, for bryophytes and lichens, `Protocol` | -| Mammals, `habitat` | Probability of presence per habitat type in the winning model | -| Birds, `landcover` | Raw `glm` coefficients (e.g. `vegcCrop`) | -| Birds, `habitat` | The same, translated onto v2's standardized habitat types by `modules/birds/standardize.R`, log scale | - -**`tables/parity_terms.csv`** (gitignored) — the term-by-term comparison -with v2: `coefficient_summary.csv` joined to `0_data/v2_results/v2_results.csv` -on taxon, region, stage, part, species and term. Plant habitat terms are -relabelled as v2 publishes them; mammal seasons are averaged first. Use it to -find which terms or species disagree. - -| Column | Meaning | -| --- | --- | -| `ref_region`, `ref_part` | The reference rows joined to: climate joins region `all`, because v2 fits it province-wide | -| `seasons` | Mammals: the seasons averaged (`summer+winter`) | -| `median`, `p10`, `p90`, `boot1`, `n` | This run's summary over draws | -| `v2_median`, `v2_p10`, `v2_p90`, `v2_n`, `source` | The v2 summary and where it came from; `v2_n = 1` means v2 fitted once | -| `comparison`, `run_value` | `median`, or `iteration 1` where v2 fitted once, and the run value used | -| `unreachable_reason`, `reachable` | Why a term cannot match yet (age spline, cutblock convergence, pooled with a spline term, v2 placeholder), from `v2_unreachable_terms()` in `02_compare_to_v2.R` | -| `in_band` | `TRUE` where the v2 median falls inside this run's `p10`–`p90` band | -| `absolute_difference` | \|run value − v2 median\| | -| `standardized_difference` | The absolute difference ÷ half this run's band width. Below 1 is close; far above 1 is a systematic difference | - -**`tables/parity_summary.csv`** — the gate result. One row per -taxon × region × stage. - -| Column | Meaning | -| --- | --- | -| `comparison` | `median` or `iteration 1` | -| `species` | Species with at least one joined term | -| `terms_compared` | Terms joined to a v2 value | -| `min_draws` | Fewest draws the store holds for any compared term; a `median` row below v2's 100 gets no verdict | -| `in_band_pct` | Percentage of compared terms in band | -| `reachable_terms` | Compared terms v2 does not set by missing machinery or a placeholder | -| `reachable_in_band_pct` | Percentage of reachable terms in band | -| `median_standardized_difference` | Typical size of the gap, over reachable terms | -| `median_absolute_difference` | Typical absolute gap, over reachable terms | -| `max_absolute_difference` | Largest absolute gap, over reachable terms; the numerical test for `iteration 1` rows | -| `negligible_terms` | Reachable terms both runs shrink to nothing (below a millionth of the term's typical size); left out of the two columns below | -| `median_spearman` | Median, over species, of the rank correlation between run and v2 medians. Reported, not gated | -| `median_band_ratio` | Median, over species, of the median of this run's 10–90% band width ÷ v2's. Near 1 when the run is as variable as v2; gated between 0.75 and 1.33 | -| `verdict` | `pass`, `fail`, `no comparison`, `not gated (trial run)`, or `not gated (store holds N draws)`, against the targets in `utils/parity_targets.R` | -| `indicative_verdict` | For a 100-draw run on a species subset: the same test, **not** the gate | - -The targets are proposed from `v2_self_agreement.R`, which measures how -closely two v2 runs agree with each other; see the experiment README. A trial -run — fewer draws than v2's 100, or a species subset — is not gated: with few -draws the run's 10–90% band is estimated from a handful of values, and is -typically narrower than v2's 100-draw band, so `standardized_difference` is -inflated and the in-band tests are unreliable in both directions. - -**`tables/v2_self_agreement_summary.csv`** — how closely v2 agrees with itself -on the bryophyte climate stage, the evidence the distributional targets are -set from. Written once by `v2_self_agreement.R`, not by `run.R`. - -**`figures/parity_.png`** — one figure per taxon, drawn from -`parity_terms.csv` by `04_plot_parity.R`. Panels are stage × region (and -hurdle part, for mammals) by species; each row is a term. v2 and exp_000 are -drawn in different colours as a median and a 10th–90th percentile bar, which -is all v2's reference keeps of its draws. Terms span 1e-144 to 1e2, so each -is drawn relative to v2: 0 is v2's median and one unit is half v2's band, -the scale of `standardized_difference`. Where v2 fitted once (mammals), the -unit is half the run's band, v2 is a point, and the run's iteration 1 — what -the gate compares — is drawn in a third colour. Unreachable terms are left -out and counted in the caption. - -**`figures/parity_mean.png`** — the same scaled values averaged over every -species and term: one facet per taxon, one row per stage and region (and -hurdle part). Each row is labelled with `|d|`, the mean absolute difference of -the run's median from v2's, and the share of terms in band. Setting -`mean_facet <- "group"` in `04_plot_parity.R` pools the four plant taxa into -one facet beside mammals and birds. Averages use values clamped at ±4, so one -far-off term cannot dominate a row. - -**`figures/parity_bell.png`** — v2 and exp_000 as distributions. One -facet per taxon (or group, as above) and one pair of curves per stage and -region, v2 in blue and exp_000 in orange. Each term's median and 10–90% -band is read as a normal curve, in v2 units, and a row's curve is the -average over its terms, negligible ones left out. A shifted orange curve is -a difference in location; a wider or narrower one, a difference in spread -(what `median_band_ratio` measures). The normal shape is a drawing -convenience: v2 keeps only three summaries of its draws. Where v2 fitted -once (mammals) it is a blue line at 0. - -**`report.md`** — a readable summary, generated rather than written: - -| Section | Comes from | -| --- | --- | -| Run kind and seed | `run.R`: full or trial run, and `boot_seed` | -| What each spec reproduces of v2 | Each spec's `v2_coverage` | -| v2 references | `0_data/v2_results/v2_results_coverage.csv`, and what this run's comparison could not reach | -| What ran | `tables/coverage.csv` | -| Parity against v2, by stage | `tables/parity_summary.csv` | -| Model fit | Median AUC from `tables/metric_summary.csv` | - -### `docs/` +## Extending the repository -| File | Holds | +| To | See | | --- | --- | -| [`framework_design.md`](docs/framework_design.md) | How the three v2 pipelines became one harness: the spec and result contracts, covariate choice, the parity ledger, and the build order | -| [`taxon_quirks.md`](docs/taxon_quirks.md) | Every taxon-specific behaviour in the v2 pipelines, and whether the harness reproduces it | -| `reviews/` | Dated code reviews and their remediation plans | -| `images/` | Logos used in this README | +| Add an experiment | `new_experiment("short description")` in `1_code/harness/new_experiment.R`; then [`1_code/experiments/`](1_code/experiments/) and [`docs/getting_started.md`](docs/getting_started.md#test-a-change) | +| Add an engine, selection rule, resampling scheme or metric | [`1_code/methods/README.md`](1_code/methods/README.md) | +| Add a taxon module | [`1_code/modules/`](1_code/modules/) | +| Rebuild or republish the dataset | [`1_code/_setup/`](1_code/_setup/); only maintainers need this | -## Parity gate +## Conventions -`exp_000` is the validation gate. It runs the v2.0 configuration of every -taxon through this repository's harness and compares the results against the -v2.0 outputs. Later experiments are compared against `exp_000`, so a -difference there is a difference in method rather than in plumbing. - -Two things are needed before the gate can be closed: - -- The harness has to reproduce v2 where it currently does not; see - [Known gaps](#known-gaps). -- The minimum acceptable parity target, which has not been defined and will - need to be agreed per taxon. - -The v2 bootstrap resamples without a fixed seed, so two v2 runs of the same -code do not produce identical coefficients, and the comparison is between -distributions rather than numbers. `boot_seed` in `run.R` makes this side -repeatable: each species draws from its own seed, derived from `boot_seed` -and the species name. Setting it to `NULL` reproduces v2's unseeded -behaviour. Birds are unaffected, because their draws are v2's stored ones. - -## Known gaps - -Per stage and taxon, each spec's `v2_coverage` is the authoritative -statement, and `report.md` prints it. The full list, with evidence and a -remediation plan, is in -[`docs/reviews/2026-09-28_alignment_review.md`](docs/reviews/2026-09-28_alignment_review.md). -The main ones: - -- **Some published v2 results cannot be reproduced exactly.** The harness - matches v2's own code to numerical precision (plants and birds, climate - and habitat). But the published lichen models were fitted on bootstrap - draws v2 did not keep, so parity against those is approximate. Birds - reproduce exactly, but only from the `Stratified.Rdata` in - `remote/birds_data_v2/`; the BirdModels drive's copy was rebuilt - after v2 and no longer matches. - See [`docs/taxon_quirks.md`](docs/taxon_quirks.md). -- **No parity target has been agreed.** - -The v2 machinery is now all in the harness: the plant age splines, cutblock -convergence and validation (`modules/_shared/plant_group.R`), and the mammal -hurdle with its splines, calibration and convergence -(`modules/mammals/hurdle.R`). Each matches v2 numerically where the -reference was fitted on data the test dataset holds. - -## Naming conventions +Code follows [code_standards](https://github.com/bgcasey/code_standards): +`snake_case` names, a standard header and numbered sections in every +script, ≤70-character lines, and tidyverse style. | Item | Convention | Example | | --- | --- | --- | -| Experiments | `exp_NNN_short_description` | `exp_001_covariate_scale` | +| Experiments | `exp_NNN_short_description`, the same in `1_code/experiments/`, `2_pipeline/` and `3_output/` | `exp_001_xgboost` | +| Sequenced scripts | `NN_` run order; entry points are `run.R` | `08_harmonize_abmiexplorer_results.R` | | Taxon slugs | As the data files name them | `vascular_plant`, `mite`, `bird` | | Module folders | Plural taxon name | `vascular_plants/`, `soil_mites/` | -| Setup scripts | `NN_` run order, in `1_code/_setup/` | `05_harmonize_v2_results.R` | - -An experiment identifier is used verbatim in `1_code/experiments/`, -`2_pipeline/`, and `3_output/`. `run.R` derives all three paths from -`exp_id` rather than typing the identifier per stage. - -## Adding an experiment - -1. Copy `1_code/experiments/exp_000_parity_v2/` to - `exp_00N_short_description/`. `run.R` creates the matching - `2_pipeline/` and `3_output/` folders on its first run. -2. Set `exp_id` in `run.R` to the new identifier. Every other path derives - from it. -3. State the question and design in the experiment's `README.md`. -4. Configure the run in `run.R`: the taxa (1.4), the focal species (1.5, - adding a named set to `utils/focal_species.R` if it will be reused), any - replacement candidate models (1.5b), the draws (1.6) and the seed (1.6b). -5. Write the numbered scripts. They read the result stores in - `2_pipeline//` with `read_results()` and write to - `3_output//`. - -Experiment-specific logic stays in the experiment. To change the models a -stage fits, use `stage_models` rather than editing a spec: the spec is the -record of what v2 did, and `meta.json` records the override. If a change is -needed in `harness/` or `modules/`, it affects every experiment and should -be reviewed on that basis. - -## Contributing a taxon module - -1. Put the v2 scripts as received in `0_data/v2_scripts//`, - unmodified, and log the source and commit in its `README.md`. They are - the reference, and are never edited and never run. -2. Extend `1_code/_setup/01_harmonize_model_ready_v2.R` to write the taxon's - response table, its rows in `covariates.csv` and `covariate_columns.csv`, - its species queue, and any prediction grid or stored bootstrap draws. - Extend `02_validate_test_dataset.R` to trace them back to the source. -3. Splice the v2 candidate formulas into `harness/model_sets.R`, verbatim. -4. Add `1_code/modules//spec.R`, a function returning the spec. Model - it on `lichens/spec.R` for a plant-group taxon, or on `birds/spec.R` or - `mammals/spec.R` for a standalone pipeline. If v2 does something no - registry entry covers, add an engine, selection rule or resampling scheme - to the harness, and review it as a harness change. -5. Add the taxon's published v2 results to `_setup/05_harmonize_v2_results.R`, - so `v2_results.csv` carries a reference for it. -6. Source the spec in section 1.2 of `run.R` and add it to `specs` in 1.4. -7. Check the result against the originals rather than trusting it. Record - every taxon-specific behaviour, and whether the spec reproduces it, in - `docs/taxon_quirks.md`, and anything the spec does not reproduce in its - `notes`. - -## Setup - -R 4.4 or later, plus the packages the harness and the `_setup` scripts use: - -```r -install.packages(c( - "data.table", "foreach", "AICcmodavg", "arm", "binom", - "mapproj", "mgcv", "MuMIn", "pROC", "RcmdrMisc" -)) -``` - -Build the test dataset once per snapshot, from the repository root: - -```r -source("1_code/_setup/01_harmonize_model_ready_v2.R") -source("1_code/_setup/02_validate_test_dataset.R") -``` - -Build the v2 reference once, after that. `03` and `04` are long — hours — -and only need re-running if their sources change: - -```r -source("1_code/_setup/03_rerun_bryophyte_v2_reference.R") -source("1_code/_setup/04_package_bird_coefficients.R") -source("1_code/_setup/05_harmonize_v2_results.R") -``` - -The `_setup` scripts read their sources over the network. Set these when a -source is not in its default location: - -| Variable | Read by | Points at | -| --- | --- | --- | -| `SDM_SNAPSHOT_V2` | `01`, `02` | The `model_ready_v2` snapshot folder | -| `SDM_MAMMAL_CAMERA_CLIMATE` | `01` | `abmi-camera-climate_2023.Rdata` | -| `SDM_MAMMAL_CLIMATE_PRED` | `01`, `02` | `All Species Climate Predictions.csv` | -| `SDM_V2_LOOKUP` | `01` | The v2 folder holding the two plant prediction-matrix CSVs | -| `SDM_WT_SPECIES` | `01` | `WildTrax Species Strings.RData` | -| `SDM_BIRD_DATA` | `01`, `02`, `04` | The bird `Stratified.Rdata` v2 was fitted on; defaults to `work_abmi/1_projects/active/sdmMethodsDev/remote/birds_data_v2/` | -| `SDM_V2_PROJECT` | `03`, `05` | The v2 plant project (`VegetationModels`) | -| `SDM_V2_BIRD_ROOT` | `04`, `05` | The v2 bird project (`BirdModels`) | -| `SDM_V2_MAMMAL_ROOT` | `05` | The v2 mammal habitat-modelling results for 2024 | - -Then run an experiment, also from the repository root: - -```r -source("1_code/experiments/exp_000_parity_v2/run.R") -``` - -Building the dataset needs the network sources. Fitting an experiment does -not. `exp_000`'s comparison step still reads v2 from the network drives, -through `SDM_V2_PLANT_OUTPUT` and `SDM_V2_MAMMAL_OUTPUT`, until it is moved -onto `0_data/v2_results/`. ## Related resources -- [code_standards](https://github.com/ABbiodiversity/code_standards) — Science - Centre coding conventions -- [sciCentRverse](https://github.com/ABbiodiversity/sciCentRverse) — Science - Centre R functions +- [sciCentRverse](https://github.com/ABbiodiversity/sciCentRverse): + Science Centre R functions. +- [sciSpatialR](https://github.com/ABbiodiversity/sciSpatialR): + Science Centre spatial catalogue. ## Contact diff --git a/cap b/cap deleted file mode 100644 index 83ae1b5..0000000 --- a/cap +++ /dev/null @@ -1,2 +0,0 @@ -numeric integer same values in order: TRUE -bird rows with any count see birdlist cols: species region det.mean det.min diff --git a/code_audit_2026-10-06.md b/code_audit_2026-10-06.md new file mode 100644 index 0000000..729849d --- /dev/null +++ b/code_audit_2026-10-06.md @@ -0,0 +1,244 @@ +# Code audit: sdmMethodsDev, 2026-10-06 + +Phase 1: read-only static scan of branch `improve_code_efficiency`. +Nothing was executed: only `parse()`, token and `codetools` analysis, +and path checks. All 83 R files outside `0_data/v2_scripts/` and all +50 legacy v2 scripts were covered. ⚠ marks a file in exp_000 (the +parity gate); those are not edited without separate approval. + + +## Summary + +- All 83 R files outside `v2_scripts/` parse cleanly. +- Every `source()` target and every `.R` path string resolves. +- `codetools` finds no function that is called but never defined, + and no package that is used but never loaded. The remaining + "global variable" notes are data.table column names or + package-level environments (`.sdm_cache`, `.sdm_registry`). +- No pipes of either kind (`|>` or `%>%`) are used anywhere. This + matches the repo's base-R/data.table, no-pipe convention, so + nothing is flagged. +- Sequenced naming: `_setup/` runs 00–03 and 06–10. The 04/05 gap is + deliberate history, since 08 says it replaced the old 04/05. There + are no prefix collisions and no `MM_NN_` staging in use. +- Errors: **0 high, 1 medium, 7 low**, plus 4 informational items + that need no change. +- Deprecated candidates: **1** (`1_code/_scratch.R`, low + confidence). It is not eligible for deletion under the rules. + +## Errors table + +| # | file | line | issue | severity | proposed fix | +|---|------|------|-------|----------|--------------| +| E1 | `1_code/_setup/run.R` | 49–59 | Step defaults contradict the header (lines 21–29: "defaults rebuild everything local and publish nothing"; 00, 03 and 10 off). In the code every step is `FALSE` except `10_publish_datasets.R = TRUE`, so sourcing `run.R` publishes whatever is in `0_data/` as today's version, with no rebuild and no 02 tally. 10 refuses to overwrite an existing version, which limits the damage. | medium | Set the defaults to match the header: 01, 06, 09, 02, 07 and 08 `TRUE`; 00, 03 and 10 `FALSE`. (Alternative: only set 10 to `FALSE`. See J1.) | +| E2 | `1_code/experiments/exp_000_parity_v2/02_plot_parity.R` ⚠ exp_000 | 399 | A runtime `message()` tells the user to "rerun `02_compare_to_v2.R`", but the script is `01_compare_to_v2.R`. | low | Change the string to `01_compare_to_v2.R`. Message text only; no change to outputs. | +| E3 | `1_code/experiments/exp_000_parity_v2/v2_self_agreement.R` ⚠ exp_000 | 29 | Header comment says "Scored exactly as `02_compare_to_v2.R`"; the file is `01_`. | low | Fix the comment. | +| E4 | `1_code/_setup/01_harmonize_model_ready_v2.R` | 2384 | Comment cites `check_prediction_terms()`, which no longer exists anywhere in the repo. | low | Reword to "found later, at prediction time" without naming a function (the successor is not certain). | +| E5 | `1_code/harness/result.R` | 1–10 | The header has no `inputs:` field (template §5.1). | low | Add `# inputs: none; writes to the store directory it is given`. | +| E6 | `1_code/_setup/07_harmonize_v2_plant_bootstrap_ids.R:147`, `harness/spec.R:590`, `modules/birds/standardize.R:234`, `experiments/exp_000_parity_v2/utils/parity_targets.R:167` ⚠ exp_000 | — | Single lines over the 70-character limit. | low | Re-wrap each line. Line breaks only; no change to the code's logic. | +| E7 | `1_code/tests/testthat/test-*.R` (5 files) | 1 | No standard header and no `# End of script ----`. | low | See J5. Testthat files commonly omit headers, so this needs a decision. | +| E8 | `1_code/_scratch.R` | 30–33 | Code sits *after* `# End of script ----`. It holds a hard-coded UNC path to the old `science\sc\sdmMethodsDev\...` share and a line over 70 characters. | low | See J6 and the deprecated table. | +| I1 | `0_data/v2_scripts/mammals/north-models/archive/04 Basic Models Figures Maps.R` | 1465 | Parse error: unexpected `}`. | info | **No change** (`v2_scripts/` is off-limits). Recorded for taxon leads. | +| I2 | `1_code/harness/model_sets.R` | 154–200 (57 lines) | Lines over 70 characters. These are v2 habitat formula strings, spliced verbatim. | info | No change. Splitting them hurts the line-by-line match with v2. | +| I3 | `1_code/harness/model_sets.R` | 156 | `TreedFen + + NonTreedFen`. The doubled `+` is copied from v2 (`02a/b/c/d_hierarchical-models-*.R:102`) and is a harmless unary plus. | info | No change, to keep fidelity with v2. | +| I4 | `%||%` defined in `harness/spec.R:683`, `harness/new_experiment.R:345` (guarded), `_setup/07…:82`, `tests/compare_stores.R:166` | — | Duplicate definitions. All four are identical, and base R ≥ 4.4 also provides `%||%`. The other duplicated names (`combine`, `failed`, `table_of`, `take_block`, `rows`) are local to their functions, so they never collide. | info | No change. | + +## Deprecated candidates + +| file | reason | evidence it is unused | confidence | +|------|--------|-----------------------|------------| +| `1_code/_scratch.R` | `scratch` naming. Its only code is a stray 2026-09-09 `load()` of an old-share file, placed after the end marker. | No `source()` or path reference in any code. **However**, `README.md:593` documents it as the "dated workspace for exploration", and it follows the personal scratch-file convention. | **low**: documented and intentional, so do not delete. A better option is to clear the stray snippet (J6). | + +Nothing else qualifies: +- Every other file is loaded by `load_framework()` (harness/, + methods/ kind folders, modules/, experiments/_shared/), is an + entry point (`run.R`, `run_tests.R`, `compare_stores.R`, + `v2_self_agreement.R`, `new_experiment.R`), or is called from one. +- `methods/_templates/` and `experiments/_template/` are scaffolding + referenced from READMEs and `new_experiment()`. +- Every file in `3_output/` has a generating script. + +## Items needing human judgement + +- **J1. `_setup/run.R` defaults (E1).** Should the defaults restore + the header's stated behaviour (rebuild everything local), or just + turn 10 off? The current state looks like a leftover from the + 2026-10-05 publish. +- **J2. Committed exp_000 summaries are from a 5-draw plumbing run.** + `3_output/exp_000_parity_v2/run_record.md` records + `n_bootstraps: 5` and 14 parity_check species, and every verdict + reads "not gated (store holds 5 draws)". But `run.R:125–126` says + to commit these only after a full run (all species, 100 draws). Is + this a known interim state? exp_001 and exp_002 compare against + this baseline. +- **J3. `v2_self_agreement_summary.csv` is missing.** + `parity_targets.R` lists it as an input, and `.gitignore` + whitelists it, but it is not in `3_output/exp_000_parity_v2/tables/`. + - *What the script does.* v2 seeds none of its bootstraps, so two + v2 runs of the same code on the same data give different + coefficients. `v2_self_agreement.R` measures how far apart two + such runs are. The first is v2's original bryophyte climate + stage, as published in ABMIexploreR (from `_setup/08`). The + second is `_setup/03`'s re-run of the frozen v2 functions, in + `2_pipeline/v2_reference/`. It scores them against each other + with the gate's own statistics, in both directions: + - median-in-band (each run's median inside the other's + 10–90% band); + - standardized median difference; + - per-species band-width ratio; + - Spearman of the medians (reported, not gated). + It writes `v2_self_agreement.csv` (per term) and a one-row + summary. + - *Why it exists.* It sets the ceiling for the gate: a + reimplementation cannot be expected to agree with v2 more + closely than v2 agrees with itself. The results (100% in band, + standardized difference 0.019, band ratio 0.78–1.21) are the + stated evidence for the thresholds hard-coded in + `parity_targets()`: 90%, 0.25, and 0.75–1.33. + - *Is it necessary?* No experiment or test needs it at run time. + It is not in `run.R`'s sequence, and nothing reads its CSV. It + is the provenance for the gate's thresholds, which + `parity_targets()` still labels "proposed, not yet agreed". + Whoever agrees them, or later questions a pass/fail, needs that + evidence. Keep the script. + - *Limits.* It covers only the bryophyte climate stage; applying + the same thresholds to the habitat and bird stages is an + assumption. Re-running it needs `_setup/03` (hours) and the + gitignored `2_pipeline/v2_reference/`. `_setup/07`'s `v2_ids` + option also depends on 03. + - *Where it runs.* By hand, from the repo root: + `source("1_code/experiments/exp_000_parity_v2/v2_self_agreement.R")`. + Its header says "Run once … after _setup/03 and 08. Not part of + run.R's sequence". It is not one of `run.R`'s `script_step()`s + (compare, plot, report). + - *Should it join the exp_000 pipeline?* No. Its folder is right + (it calibrates exp_000's gate), but it should not be a + `run.R` step: + 1. It reads no harness result. Its output changes only when + the v2 reference changes (`_setup/03` or `08`), not when + exp_000 runs, so every run would recompute an identical + table. + 2. Its key input, `2_pipeline/v2_reference/…Rdata`, is local + and gitignored, and exists only on the machine that ran 03. + Everything else in exp_000 reads the published datasets, so + anyone can run it. As a step, this script would stop + exp_000 for every other user. + 3. Moving it to `_setup/` breaks the rule that `_setup` is + fixed and that experiments own their scripts. + A cleaner long-term option, if wanted, is outside this audit. + `_setup/10` could publish 03's re-run inside `v2_results/`, and + the script could read it via `v2_results_dir()`. It could then + become an optional step that anyone can run. + - *Recommendation.* No code change. Either run it once and commit + the summary CSV, which `.gitignore` already allows (this needs + your go-ahead, because it reads data and writes outputs), or + accept that the numbers recorded in `parity_targets.R:38–47` + are the record and drop the CSV from that header's `inputs:`. + - *Confidence: High* on what it does. *Moderate* on "necessary": + that depends on whether the team treats the thresholds as + settled. +- **J4. Absolute local path in a committed file.** + `run_record.md` records `pipeline_dir: + D:/local_projects/active/...`. It sits in the "volatile" section, + so it is harmless, but the path is machine-specific. +- **J5. Test-file conventions (E7 and naming).** `test-*.R` uses a + hyphen, which is the testthat convention. `test_*.R` also works, + but renaming has little value. Should the 5 test files get + standard headers? +- **J6. `_scratch.R` (E8).** Empty the 2026-09-09 snippet (keeping + the header), leave it as is, or delete the file (it is documented + in the README)? +- **J7. Out-of-scope leftovers I will not touch.** + - `2_pipeline/exp_001_gbm/` is an orphan store from a renamed + experiment. + - `0_data/test_dataset_backup_2026-10-02_aug2026_birds/` is a + backup dataset folder. + - `3_output/exp_001_xgboost/` and `exp_002_soilgrids/` are empty + folders. + - None of these is tracked by git; you may want to clear them by + hand. +- **J8. "the project file sets it"** (`exp_000/run.R:34`). No + `.Rproj` is tracked in the repo. This is a documentation and setup + question for the separate docs prompt. + +## Appendix A: repository map + +The table lists each file with its purpose, from its header title, +and its main reads and writes. Key reads: TD is the published test +dataset via `data_source.R`; V2R is the published v2 results; SI is +the `setup_inputs` mirror via `_setup/utils/input_paths.R`. + +| file | purpose | reads → writes | +|------|---------|----------------| +| `_setup/run.R` | Entry point: runs the _setup steps in isolated environments | _setup scripts → `2_pipeline/_setup/run_log.csv` | +| `_setup/00_mirror_setup_inputs.R` | Mirror the original inputs (old shares, G: drives) onto ABMI-DATA2 | originals → SI | +| `_setup/01_harmonize_model_ready_v2.R` | v2 snapshot → harmonized CSVs | SI → `0_data/test_dataset/` | +| `_setup/02_validate_test_dataset.R` | Pass/fail tally of the dataset | `0_data/test_dataset/`, SI → console | +| `_setup/03_rerun_bryophyte_v2_reference.R` | Re-run the v2 bryophyte models (sources `v2_scripts/plants/*functions.R`) | SI → `2_pipeline/v2_reference/` | +| `_setup/06_…bird_translation_lookup.R` | Bird coefficient translation lookup | SI → `test_dataset/lookup/` | +| `_setup/07_…plant_bootstrap_ids.R` | v2's stored plant bootstrap ids | SI, v2_reference → `lookup/v2_bootstrap_ids/` | +| `_setup/08_…abmiexplorer_results.R` | ABMIexploreR coefficients at a pinned commit | GitHub → `0_data/v2_results/abmiexplorer/` | +| `_setup/09_harmonize_lookups.R` | Cross-taxa lookups | 01/06 outputs → `lookup/` | +| `_setup/10_publish_datasets.R` | Publish a versioned copy with checksums | `0_data/` → ABMI-DATA2 | +| `_setup/utils/input_paths.R` | `setup_input()` resolver | — | +| `harness/harness.R` | `load_framework()`: sources harness, then methods, then modules, then experiments/_shared | all of those | +| `harness/{registry,cache,data_source,spec,data_load,covariate_sets,species,model_sets,resample,engines,selection,eval_metrics,predict_grid,result,run_record,run_model,summarise,compare,experiment,new_experiment}.R` | Taxon-agnostic pipeline (loaded in this order) | TD/V2R → `2_pipeline//`, `3_output//` | +| `methods/engines/{_glm_family,glm,bayesglm,xgboost}.R` | Engine plug-ins | — | +| `methods/selection/{aic_average,aic_best,hurdle,ivw_grid,single,staged_bic}.R` | Selection plug-ins | — | +| `methods/resampling/{precomputed,spatial_block,spatial_cv}.R` | Resampling plug-ins (precomputed reads `lookup/v2_bootstrap_ids/`) | TD | +| `methods/metrics/{auc,calibration_slope,deviance_explained,n,prevalence,rmse,spearman}.R` | Metric plug-ins | — | +| `methods/_templates/*.R` | Skeletons; not loaded | — | +| `modules/_shared/{plant_group,standard_specs}.R` | Plant-group spec builder; the v2 spec list | — | +| `modules//spec.R` (+ `birds/standardize.R`, `mammals/hurdle.R`) | Per-taxon v2 specs and helpers | TD lookups | +| `experiments/_shared/focal_species.R` | Named focal-species sets | — | +| `experiments/_template/run.R` | Scaffold copied by `new_experiment()` | — | +| `experiments/exp_000_parity_v2/run.R` | Entry: parity gate; runs 01–03 as `script_step`s | TD, V2R → `2_pipeline/exp_000…`, `3_output/exp_000…` | +| `exp_000/01_compare_to_v2.R` | Coefficient parity vs v2 | stores, V2R → `parity_summary.csv`, `parity_terms.csv`, `coefficient_summary.csv` | +| `exp_000/02_plot_parity.R` | Parity figures | tables → `figures/*.png` | +| `exp_000/03_build_report.R` | `report.md` | tables → `report.md` | +| `exp_000/utils/parity_targets.R` | Gate thresholds | — | +| `exp_000/v2_self_agreement.R` | Stand-alone: v2 vs v2 calibration | `2_pipeline/v2_reference/`, V2R → `v2_self_agreement*.csv` | +| `experiments/exp_001_xgboost/run.R` | Entry: xgboost habitat stage | TD → `2_pipeline/exp_001_xgboost/`, `3_output/…` | +| `experiments/exp_002_soilgrids/run.R` | Entry: soil terms in the climate stage; sources 01 if the input is missing | soil CSV, TD → stores and outputs | +| `exp_002/01_extract_soilgrids_covariates.R` | sciSpatialR/sf extraction | TD sites, SoilGrids → `2_pipeline/exp_002…/inputs/soilgrids_0_5cm.csv` | +| `tests/run_tests.R` | Entry: testthat over `tests/testthat/` | TD (skipped if unreachable) | +| `tests/testthat/test-{dataset,methods,new_experiment,results,specs}.R` | Contract tests | TD | +| `tests/compare_stores.R` | Stand-alone/CLI: compare two store trees | `2_pipeline` stores | +| `_scratch.R` | Personal scratch | old-share `.Rdata` | + +## Appendix B: dependency map + +- **Entry points**: + - `_setup/run.R` + - each `experiments/*/run.R` + - `tests/run_tests.R` + - `tests/compare_stores.R` (CLI) + - `exp_000/v2_self_agreement.R` + - `harness/new_experiment.R` (also stand-alone) + - `_scratch.R` +- **Sourced by `load_framework()`**: all of `harness/`, the + `methods/{engines,selection,resampling,metrics}/` folders (files + starting with `_` first), `modules/*/` (`_shared` first), and + `experiments/_shared/`. +- **Also re-sourced**: parallel workers source it again via + `run_model.R:555` (`load_worker`). exp_000's 01–03 run through + `experiment.R:556` (`sys.source`). +- **Other `source()` edges**: + - `_setup/{01,02,03,06,07,08}` → `_setup/utils/input_paths.R` + - `00` → `input_paths.R` (via `helper`) + - `10` → `harness/data_source.R` (via `resolver`) + - `03` → `0_data/v2_scripts/plants/{hierarchical-model_functions,bootstrapping_functions}.R` + - `v2_self_agreement.R` → `parity_targets.R` and `data_source.R` + - `exp_002/run.R` → `exp_002/01_…R` + - `exp_002/01` → `data_source.R` +- **Packages**: + - Attached with `library()`: data.table, ggplot2 (exp_000/02), + testthat, foreach and parallel (03), sciSpatialR and sf + (exp_002/01). + - Attached on cluster workers in 03 only: AICcmodavg, arm, binom, + mapproj, mgcv, MuMIn, pROC, RcmdrMisc. + - Called with `::` and guarded by `requireNamespace()`: arm, + xgboost, MuMIn, mgcv, pROC, jsonlite, terra, withr, stats, + utils, tools, parallel. + - All the `::` packages are installed on this machine. + diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 0000000..47c274c --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1,2 @@ +/.quarto/ +/_site/ diff --git a/docs/_quarto.yml b/docs/_quarto.yml new file mode 100644 index 0000000..34b918a --- /dev/null +++ b/docs/_quarto.yml @@ -0,0 +1,37 @@ +# Docs site for sdmMethodsDev, published to GitHub Pages by +# .github/workflows/docs.yaml on every push to main that touches +# docs/. Preview locally from this folder: quarto preview +# Only the pages listed under `render` are built; the other +# docs stay repository-only. +project: + type: website + output-dir: _site + render: + - index.md + - getting_started.md + +website: + title: "sdmMethodsDev" + site-url: "https://abbiodiversity.github.io/sdmMethodsDev/" + repo-url: "https://github.com/ABbiodiversity/sdmMethodsDev" + navbar: + logo: images/abmi_logo.png + left: + - href: index.md + text: Home + - href: getting_started.md + text: Getting started + right: + - icon: github + href: https://github.com/ABbiodiversity/sdmMethodsDev + +format: + html: + theme: cosmo + toc: true + toc-depth: 3 + code-copy: true + +# Renders ```mermaid blocks as diagrams, as GitHub does +filters: + - mermaid.lua diff --git a/docs/framework_design.md b/docs/framework_design.md index 7b78cb0..4cb6726 100644 --- a/docs/framework_design.md +++ b/docs/framework_design.md @@ -1,6 +1,6 @@ # Modelling framework design -![Status](https://img.shields.io/badge/Status-Steps%201--3%20built-yellow) +![Status](https://img.shields.io/badge/Status-Steps%201--5%20built-yellow) How the three taxon pipelines become one configurable framework, so that a methods question can be asked once and answered across taxa. @@ -72,49 +72,92 @@ requires a separate pipeline. ``` 1_code/ ├── harness/ # taxon-agnostic; the framework -│ ├── data_load.R # readers + build_model_data + model_frame +│ ├── harness.R # load_framework(): everything below, in order +│ ├── registry.R # register_*(), get_method(), list_methods() +│ ├── spec.R # validate_spec() and spec helpers +│ ├── data_load.R # dataset manifest, readers, model frames │ ├── covariate_sets.R # named bundles → the column catalogue -│ ├── species.R # one species catalogue across three schemas -│ ├── resample.R # precomputed | spatial_block | spatial_cv -│ ├── engines.R # registry: glm, bayesglm -│ ├── selection.R # single, aic_best, aic_average, staged_bic -│ ├── eval_metrics.R # scores computed from predictions only -│ ├── result.R # the five-file result store -│ └── utils/step_runner.R # timing and logging +│ ├── species.R # the one species queue +│ ├── resample.R # seeds, spatial blocks, scheme dispatch +│ ├── engines.R # engine interface, check_engine() +│ ├── selection.R # candidate fitting, combined predictors +│ ├── eval_metrics.R # compute_metrics() +│ ├── predict_grid.R # final model onto a habitat grid +│ ├── result.R # the result store, and shards +│ ├── run_model.R # run_specs(): prepare, fit (parallel), merge +│ ├── summarise.R # collect_results() +│ ├── compare.R # compare_experiments() +│ └── experiment.R # experiment_config(), run_experiment() +├── methods/ # one file per method, each self-registering +│ ├── engines/ # glm, bayesglm, xgboost +│ ├── selection/ # single, aic_best, aic_average, staged_bic, +│ │ # ivw_grid, hurdle +│ ├── resampling/ # precomputed, spatial_block, spatial_cv +│ └── metrics/ # auc, deviance_explained, rmse, … ├── modules/ -│ ├── _shared/plant_group.R # the four plant-group taxa share this -│ └── /spec.R # that taxon's v2 configuration (step 4) -├── experiments/exp_NNN_*/ -│ └── run.R # taxon, species, covariates, engine -└── _deprecated/ # superseded; see its README +│ ├── _shared/ # plant_group.R; standard_specs.R +│ └── /spec.R # that taxon's v2 configuration +├── experiments/ +│ ├── _shared/ # focal-species sets +│ ├── _template/ # the starting point for a new one +│ └── exp_NNN_*/run.R # configuration, then run_experiment() +└── tests/ # contract tests; store comparison ``` -The harness knows nothing about any taxon. A module contributes only a -spec. An experiment overrides parts of a spec. +The harness knows nothing about any taxon, and nothing about any +particular method: it looks methods up by the name a spec gives. A +module contributes only a spec. An experiment changes parts of a +spec, or names different methods in it. There are six taxon modules: `bryophytes`, `lichens`, `soil_mites`, -`vascular_plants`, `mammals` and `birds`. The first four were one -`plants` module until they were split, so that a taxon-specific -quirk has one obvious home and the four can diverge. What they still -share is the v2 pipeline shape, which lives in -`_shared/plant_group.R` so four copies cannot drift apart. - -Directory names are plural and readable; **taxon slugs are the data -keys and are not**. Soil mites live in `soil_mites/` and key on -`mite`, which is what `run_taxa`, `focal_species` and every harness -call expect. - -Files are consolidated rather than one function per file: `engines.R` -holds the registry and both engines, and splits when `gbm` and `brms` -arrive. Two pieces named in the original sketch are deliberately not -built yet, because both depend on decisions a spec makes: - -- `predict_grid.R` — projecting a fitted model onto a prediction matrix. - v2 does this inside its habitat-model functions, and how the terms map - onto grid rows is spec-dependent. `write_grid_predictions()` is ready - for it; nothing produces grid predictions yet. -- `run_model.R` — the species-by-bootstrap loop. It is the glue a spec - drives, so it lands with the first spec. +`vascular_plants`, `mammals` and `birds`. The first four share the v2 +pipeline shape in `_shared/plant_group.R`, so four copies cannot +drift apart. Directory names are plural and readable; **taxon slugs +are the data keys and are not**. Soil mites live in `soil_mites/` and +key on `mite`. + +### Methods are plug-ins + +An engine, selection rule, resampling scheme or metric is one file in +`1_code/methods/` that registers itself. Each kind has a contract, +stated in `1_code/methods/README.md`, and a rule states the engine +capabilities it needs (`coefficients`, `ic`, `se`). `validate_spec()` +checks a spec against the registry before any data is read, so an +engine paired with a rule it cannot serve - a boosted tree with +inverse-variance averaging, say - stops with a sentence rather than +silently returning nothing hours into a run. `check_engine()` tests a +new engine against the contract on synthetic data. + +The rules talk to an engine only through its interface: coefficients +come from `engine$coef()`, computed once per fit, and every rule's +result carries a `predict()` for its final model. That is what lets +an engine without coefficients run any stage with the `single` rule, +be carried into a later stage, and be projected onto the grid. + +### The data is described, not assumed + +`0_data/test_dataset/lookup/dataset_manifest.csv`, written by +`_setup/09`, names each taxon's response, offset and grid files, its +key in `covariates.csv`, and its stored draws. The species queue and +factor levels are one table each for every taxon. The harness reads +file locations from the manifest and nowhere else, so a new taxon is +manifest rows plus its files. + +The dataset folder itself is resolved once, by `test_dataset_dir()` in +`harness/data_source.R`: the version published to +`//ABMI-DATA2/science/sdmMethodsDev/0_data/test_dataset/` that the file +pins, unless `SDM_TEST_DATASET` names another copy. A published version +is never overwritten and carries a `checksums.csv`, so every run reads +the same bytes, and the run record says which version it read. + +### Runs are parallel and reproducible + +Each species of each spec is one job. Jobs run on a cluster of R +sessions, each writing a shard of its own, and the shards are joined +in queue order, so a parallel run writes byte for byte what a serial +one does. Draws are seeded per species from one base seed, under R's +default generator named explicitly, so a run's results do not depend +on the number of workers or the order jobs finish in. ## Contract 1: the taxon spec @@ -165,23 +208,27 @@ run_experiment( species = c("Aulacomnium.palustre", "Sphagnum.fuscum"), overrides = list( stages = list(climate = list(models = "climate_plus_topography")), - engine = "gbm")) + engine = "xgboost")) ``` ## Contract 2: the result object -Every run writes the same five files, whatever the engine. This is what +Every run writes the same files, whatever the engine. This is what makes a GLM run and a BRT run comparable. | File | Columns | Notes | | --- | --- | --- | | `grid_predictions.csv` | species, region, boot, grid_unit, prediction | **the common currency** | -| `unit_predictions.csv` | species, region, boot, survey_unit_id, prediction, observed | held-out scoring | +| `unit_predictions.csv` | species, region, boot, survey_unit_id, prediction, observed | optional (`unit_predictions = "oob"` or `"all"`); metrics are scored in memory | | `coefficients.csv` | species, region, boot, term, estimate, se | absent for engines without coefficients | | `metrics.csv` | species, region, boot, metric, value | AUC, deviance, calibration | | `meta.json` | engine, selection, covariate set, spec hash, seed, timing | provenance | -`grid_predictions` is the linchpin. Coefficients cannot compare a GLM to +`grid_predictions` is the linchpin. It is the stage's **final** model +projected onto the grid: the averaged or combined one where a rule +combines candidates, not the best single candidate. Every taxon has a +grid; the bird grids are v2's coefficient translation matrix rebuilt +in covariate space by `_setup/09`. Coefficients cannot compare a GLM to a boosted tree — a tree has no `Peatland` coefficient — but every engine can predict onto the 43 rows of the vegetation prediction matrix or the 16 of the soil matrix. Comparing there keeps the comparison on the @@ -223,7 +270,9 @@ That means a spec cannot name a term it forgot to load, and it means selecting covariates on their own would do nothing: loading `elevation` while every candidate still fits `MAP + FFP` changes only the I/O. -So **the model set is the lever**, and an experiment sets it directly: +So **the model set is the lever**, and an experiment sets it directly +(the extra terms here must be ones the dataset or a covariate file +holds; see below): ```r stage_models <- list( @@ -271,6 +320,27 @@ catalogue, with a dataset version bump, so the frozen dataset stays the single source of truth and `02_validate_test_dataset.R` keeps covering everything. +### Covariates the dataset does not hold + +`0_data/test_dataset/` and the `_setup/` scripts that build it are +the v2 data and stay fixed. An experiment that tests a new covariate +brings it itself: + +1. A script in the experiment's folder (e.g. + `exp_002_soilgrids/01_extract_soilgrids_covariates.R`) extracts + it and writes a CSV to `2_pipeline//inputs/`: one row per + `survey_unit_id`, one column per covariate, `NA` where a unit has + no value. +2. `run.R` names the file in + `experiment_config(covariate_files = )` and the new columns in + `stage_models`. +3. The harness joins the file on `survey_unit_id` when it loads each + region. `experiment_config()` checks the file first: unique ids, + and no column that `covariates.csv` already has. + +A covariate that later becomes part of the standard data goes through +a new dataset build, not through an experiment. + ## Parity ledger Parity is strict by default. This section records every place it is @@ -389,39 +459,43 @@ It sits on a regularization axis. A Bayesian framework comparison needs ## Build order -1. **Built.** Harness data layer and covariate sets — - `data_load.R`, `covariate_sets.R`, `species.R`. -2. **Built.** Result object and its writers — `result.R`. -3. **Built.** Engines `glm` and `bayesglm` (`engines.R`); selection - `single`, `aic_best`, `aic_average`, `staged_bic` (`selection.R`); - resample `precomputed`, `spatial_block`, `spatial_cv` - (`resample.R`); metrics (`eval_metrics.R`). -4. **Built.** Specs for all six taxa, in - `1_code/modules//spec.R`, plus `run_model.R` (the - species-by-draw-by-stage loop) and `predict_grid.R`. +1. **Built.** Harness data layer and covariate sets. +2. **Built.** Result object and its writers. +3. **Built.** Engines `glm` and `bayesglm`; selection `single`, + `aic_best`, `aic_average`, `staged_bic`; resample `precomputed`, + `spatial_block`, `spatial_cv`; metrics. +4. **Built.** Specs for all six taxa, the fitting loop and grid + prediction. 5. **Built.** `exp_000_parity_v2` runs each spec and compares - against the published v2 output, on both the climate and the - habitat stage, for all three taxa. Plants reach **90.9% of - terms in band** among those the harness fits; the mammal - presence stage correlates **0.98** with `Coef.pa.all`. Birds - run but have no reachable reference. Coverage and the - remaining gaps are in the experiment's README and report. - - Five selection rules now cover the three pipelines' - different ideas of what a coefficient is: `aic_average` - (plants and birds, climate), `ivw_grid` (plants, habitat - - precision-weighted predictions onto a prediction matrix), - `staged_bic` (birds, landcover), and `aic_best_onehot` - (mammals - one-hot predictions on the probability scale with - a calibration shift). -6. Engines `gbm` and `brms`; the topography and remote-sensing covariate - extension; exp_001 onward. - -Steps 1–3 are taxon-agnostic and were built and exercised before any -taxon spec exists. All four taxon-and-region combinations — -bryophyte north, mammal north, mammal south, bird north — run through -one interface, covering the three response types, the offset path and -the weighted path. + against the published v2 output, on every stage, for all three + taxa. Five selection rules cover the three pipelines' different + ideas of what a coefficient is: `aic_average`, `ivw_grid`, + `staged_bic`, and the mammal `hurdle` (2026-10-05: an + engine-agnostic rule, replacing `aic_best_grid`, + `aic_best_onehot` and the custom hurdle function; v2's + table-building is now a mammal `post_process` step). +5a. **Built (2026-10-03).** The plug-in method layer + (`1_code/methods/`, the registry, `validate_spec()`, + `check_engine()`); the dataset manifest and harmonized lookups + (`_setup/09`); parallel species with byte-identical shards; + metrics and grids from each stage's final model, and grids for + every taxon; `experiment_config()` and `run_experiment()`; the + experiment template and `compare_experiments()`; contract tests. + Checked against the previous code at 5 draws for the 14 parity + species: every stage's coefficients identical. +6. **Started (2026-10-03).** A boosted-tree engine and + `replace_stage_method()`; `exp_001_xgboost`, a test + experiment fitting the habitat stage with boosted regression + trees, every taxon including mammals. (The first engine, `gbm`, + was replaced by `xgboost` on 2026-10-05: gbm cannot fit the + mammal hurdle's 0-1 presence response or its Gamma + abundance.) `covariate_files` lets an experiment add covariates the + frozen dataset does not hold, from its own files; + `exp_002_soilgrids` adds SoilGrids at 0–5 cm from the + sciSpatialR catalogue to the climate stage. Still to come: + `mgcv` GAMs and `brms` as method files, the topography and + remote-sensing covariates, and + spatial prediction onto the 1 km grid. ### What steps 1-3 settled diff --git a/docs/getting_started.md b/docs/getting_started.md new file mode 100644 index 0000000..3790ca7 --- /dev/null +++ b/docs/getting_started.md @@ -0,0 +1,740 @@ +# Getting started with sdmMethodsDev + +This guide takes you from a fresh clone to a run of the v2 +baseline, and then to your first experiment or module change. It +assumes you know R, species distribution models, and spatial +analysis. It does not assume you know this repository or how the +v2.0 models were built. Work through it in order; it takes under +an hour, plus run time. + +Run every code block from the repository root. Lines starting +`#>` are the output the block printed when this guide was written. + +- [What this repository is for](#what-this-repository-is-for) +- [Setup](#setup) +- [How the pipeline fits together](#how-the-pipeline-fits-together) +- [Run the baseline (exp_000)](#run-the-baseline-exp_000) +- [Test a change](#test-a-change) +- [Add or update a taxon module](#add-or-update-a-taxon-module) +- [Check that nothing broke](#check-that-nothing-broke) +- [Working conventions](#working-conventions) +- [Where to get help](#where-to-get-help) + +## What this repository is for + +`sdmMethodsDev` is the Science Centre's shared R&D repository for +Species Models 3.0. It holds a frozen cross-taxa test dataset, one +pipeline that runs every taxon's v2.0 model as configuration, and +a v2 parity check (`exp_000_parity_v2`) that every later +experiment is compared against. A methods question is asked once +and answered for every taxon. + +It is for testing methods. It does not produce the final species +models or reporting products. The status, structure, and design +summary are in the [root README][readme]. + +### Words used throughout + +| Word | Means | +| --- | --- | +| **Spec** | A taxon's v2 configuration as a list: response, family, regions, stages, resampling. Built by functions such as `lichen_spec()` in `1_code/modules//spec.R`. Holds no modelling code. | +| **Run** | One spec in an experiment, named as its output folder: `lichen`, `mammal_summer`. `standard_specs()` lists them. | +| **Stage** | One model fitted in order. Plants and birds fit `climate`, then `habitat` (or `landcover`). | +| **Carry** | Passing one stage's prediction into the next as a covariate called `Climate`. This is how v2 combines its climate and habitat models. | +| **Candidate set** | The formulas a stage chooses among. The v2 sets are in `1_code/harness/model_sets.R`. | +| **Method** | A named, registered piece: an engine (`glm`, `bayesglm`, `xgboost`), a selection rule (`aic_average`, `staged_bic`, `ivw_grid`), a resampling scheme, or a metric. | +| **Draw** | One bootstrap resample of the survey units. v2 uses 100. Draw 1 is the full data. | +| **Store** | The results for one run and region: `coefficients.csv`, `metrics.csv`, `grid_predictions.csv`, `meta.json`. | +| **Grid** | A prediction matrix with one row per habitat type, each row a pure stand of that type. Predicting onto it lets methods without coefficients be compared. | +| **Out-of-bag (oob)** | Scored on the survey units a draw left out. `insample` scores the units the draw fitted. | +| **Parity** | Whether a run reproduces v2. Distributional, not numerical: v2 sets no seed, so two v2 runs differ. | + +## Setup + +### Prerequisites + +You need R 4.4 or later. Open the repository in RStudio or +Positron so the working directory is the repository root. Every +path in this guide, and every script, assumes that. + +The repository has no `renv.lock` or `DESCRIPTION`, so packages +are installed by hand. These are the packages the +[root README][readme-prereq] lists: + +```r +install.packages(c( + "data.table", "arm", "mgcv", "MuMIn", "pROC", # harness, methods + "jsonlite", "ggplot2", # summaries, figures + "xgboost", # xgboost engine + "testthat", "withr" # 1_code/tests/ +)) +``` + +`exp_002_soilgrids` also needs `sf`, `terra`, and +[sciSpatialR](https://github.com/ABbiodiversity/sciSpatialR). +Maintainers rebuilding the dataset with `_setup/03` also need +`foreach`, `AICcmodavg`, `binom`, `mapproj`, and `RcmdrMisc`. + +> **TODO:** confirm with Brendan Casey whether package versions +> will be pinned (for example with `renv`). Until then, the version +> comments in each script's `library()` calls are the only record. + +### Getting the data + +You do not build the data. `0_data/` is gitignored, and runs do +not read it. Runs read two published, versioned datasets from the +ABMI network share: + +- **Test dataset:** + `//ABMI-DATA2/science/sdmMethodsDev/0_data/test_dataset//` +- **v2 results** (the parity reference): + `//ABMI-DATA2/science/sdmMethodsDev/0_data/v2_results//` + +The version read is pinned in `sdm_published$versions` in +[`1_code/harness/data_source.R`][data-source]. A published version +is never overwritten. Each folder carries a `checksums.csv`, and +the test dataset's file index is `lookup/dataset_manifest.csv`. +The harness reads file locations only from that manifest. + +So the one requirement is **read access to the share**, on the +ABMI network or VPN. + +> **TODO:** confirm with Brendan Casey how new staff request read +> access to `//ABMI-DATA2/science/sdmMethodsDev/` and the VPN. + +Off the network, copy the two published folders to your machine +and point the harness at them. The variables can also go in your +user `.Renviron`: + +```r +Sys.setenv( + SDM_TEST_DATASET = "D:/data/sdmMethodsDev/test_dataset/2026-10-05", + SDM_V2_RESULTS = "D:/data/sdmMethodsDev/v2_results/2026-10-05" +) +``` + +`1_code/_setup/` builds and publishes these datasets. Only +maintainers run it; see the [`_setup` README][setup] if that is +you. + +### Check that the setup works + +First, load the framework. This sources the harness, every +method, every taxon module, and the shared species sets. It reads +no data. + +```r +source("1_code/harness/harness.R") +load_framework() +``` + +Then confirm the data can be reached. `test_dataset_dir()` and +`v2_results_dir()` stop with a clear message if the share is +unreachable or a file is missing or the wrong size. + +```r +data_dir <- test_dataset_dir() +v2_dir <- v2_results_dir() + +file.exists(c( + file.path(data_dir, "lookup", "dataset_manifest.csv"), + file.path(v2_dir, "abmiexplorer", "v2_results.csv") +)) +#> [1] TRUE TRUE +``` + +`verify_published(test_dataset_dir())` does a full md5 check. It +is slow over the network (about 0.9 GB), so use it only when a +copy is in doubt. + +Last, run the contract tests from a terminal at the repository +root. They take seconds, and skip the dataset checks if the share +cannot be reached. + +```sh +Rscript 1_code/tests/run_tests.R +``` + +If the tests pass and both files exist, you are ready to run. + +## How the pipeline fits together + +The v2.0 pipelines share one sequence: pick survey units, fit +candidate models, choose among or average them, then predict and +score. What differs between taxa is the contents of each step. +Here, those contents are data in a **spec**, and one **harness** +runs any spec. + +```mermaid +%% ---------------------------- +%% How one experiment runs +%% ---------------------------- +%%{init: {"themeVariables": {"edgeLabelBackground": "#ffffff"}}}%% +flowchart TD + + subgraph fw["framework: one experiment, from spec to comparison"] + direction TB + M["1_code/modules/[taxon]/
v2 spec per taxon"] --> SS["standard_specs()"] + SS --> C["1_code/experiments/exp_NNN/run.R
experiment_config()"] + C --> H["1_code/harness/
run_experiment()"] + MT["1_code/methods/
engines · selection rules · resampling · metrics"] + MT -.->|"looked up by name"| H + D["published test dataset
harness/data_source.R"] --> H + H --> P["2_pipeline/[id]/
result stores · run_log.csv"] + P --> O["3_output/[id]/
tables · run_record.md"] + O --> CMP["compare_experiments()
against exp_000"] + end + + %% ---------------------------- + %% Styles + %% ---------------------------- + + %% Container + style fw fill:none,stroke:#2D415B,stroke-width:1px + + %% Configuration + style M fill:#A3B4C7,stroke:#2D415B,stroke-width:1px + style SS fill:#A3B4C7,stroke:#2D415B,stroke-width:1px + style C fill:#A3B4C7,stroke:#2D415B,stroke-width:1px + style MT fill:#A3B4C7,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + + %% Inputs and outputs + style D fill:#ffffff,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + style P fill:#ffffff,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + style O fill:#ffffff,stroke:#2D415B,stroke-width:1px,stroke-dasharray: 5 5 + + %% Shared pipeline + style H fill:#A4A88780,stroke:#C8A02C,stroke-width:4px + + %% Comparison + style CMP fill:#E8A396,stroke:#2D415B,stroke-width:4px +``` + +The runs the standard specs offer, and the methods registered: + +```r +names(standard_specs()) +#> [1] "bryophyte" "lichen" "mite" "vascular_plant" +#> [5] "mammal_summer" "mammal_winter" "bird" + +list_methods() # kind, name, requires, description +``` + +`run_experiment()` does seven things in order: + +1. **Configure.** `experiment_config()` checks every decision + before anything is fitted. +2. **Fit.** Each spec is validated, and each species is one job. + With `workers` above 1, jobs run in parallel R sessions. Each + species draws under its own seed, so results do not depend on + the worker count. A failed species or draw is logged and the + run continues. +3. **Store.** One store per run and region in `2_pipeline//`, + plus `run_log.csv`. +4. **Summarize.** `collect_results()` writes + `3_output//tables/`. +5. **Compare.** Every experiment except exp_000 writes + `tables/comparison_metrics.csv` against exp_000. +6. **Steps.** The experiment's own `steps`, in order. +7. **Record.** `run_record.md`. + +The full argument list and output file definitions are in the +[harness README][harness]. + +### Where things are + +| Folder | Holds | Edit it to | +| --- | --- | --- | +| `1_code/harness/` | The shared pipeline: configure, fit, store, summarize, compare. Names no taxon and no method | Change how every experiment runs (review as such) | +| `1_code/methods/` | Engines, selection rules, resampling schemes, metrics; each file registers one method | Add a method | +| `1_code/modules//` | Each taxon's v2 spec; `_shared/` holds the plant-group builder and `standard_specs()` | Record what v2 does for a taxon | +| `1_code/experiments/exp_NNN_*/` | One folder per question; `run.R` holds the decisions | Ask a question | +| `1_code/experiments/_shared/` | Named focal-species sets | Add a reusable species set | +| `1_code/tests/` | Contract tests and the store comparison | Add a check | +| `1_code/_setup/` | One-off scripts that build and publish the dataset | Rebuild or extend the dataset (maintainers) | +| `0_data/test_dataset/` | The dataset as `_setup/` builds it; runs read the published copy | Nothing; rebuilt by `_setup/` | +| `2_pipeline//` | Result stores and intermediates; gitignored | Nothing; written by runs | +| `3_output//` | Summary tables, figures, report, run record | Nothing by hand; small summaries are committed | +| `docs/` | This guide, design, taxon quirks, reviews | | + +## Run the baseline (exp_000) + +`exp_000_parity_v2` runs each taxon's v2 spec and compares the +result with the published v2 coefficients. It is the baseline for +every other experiment. It is intended as the gate before +iterative R&D, but it is **not yet a gate**: the parity targets +are proposals, and no full run has been committed. See the +[exp_000 README][exp000] for what must happen first. + +### Step 1: Check the settings + +Open [`1_code/experiments/exp_000_parity_v2/run.R`][exp000-run]. +Section 1.2, the `experiment_config()` call, is the whole +configuration. Check three settings: + +| Setting | Quick check | Full run | +| --- | --- | --- | +| `species` | `"one_each"` (one species per taxon) or `"parity_check"` (two per taxon; the committed value) | `NULL`, every species | +| `n_bootstraps` | `5` (the committed value) | `100` | +| `workers` | 4 | Up to the cores you can spare (committed: 12) | + +The named species sets are: + +```r +names(focal_species_sets()) +#> [1] "full" "parity_check" "plants_only" "one_each" +``` + +Shorten a trial by cutting species, not draws. A 10-90% band read +from five draws is unstable. On a 24-core machine, the 14 +`parity_check` species at 5 draws took about 6 minutes on one +worker and about 1.6 minutes on 12. A full run of every species +at 100 draws is days of compute. + +Parallel workers load the framework from disk when the run +starts, so do not edit files in `1_code/` while one is starting. + +### Step 2: Run it + +```r +source("1_code/experiments/exp_000_parity_v2/run.R") +``` + +`experiment_config()` prints what it will do, and notes if you are +running fewer than 100 draws. After fitting, `run_experiment()` +runs exp_000's three steps: `01_compare_to_v2.R`, +`02_plot_parity.R`, and `03_build_report.R`. + +To rebuild the tables, figures, and report from existing stores +without refitting, set `fit = FALSE` in the `run_experiment()` call +in section 2 of `run.R`. + +### Step 3: Find the outputs + +| Where | What | +| --- | --- | +| `2_pipeline/exp_000_parity_v2///` | Result stores: `coefficients.csv`, `metrics.csv`, `grid_predictions.csv`, `meta.json` | +| `2_pipeline/exp_000_parity_v2/run_log.csv` | Every species, region, and draw: `ok`, or why not | +| `3_output/exp_000_parity_v2/run_record.md` | What ran: species, draws, seed, data versions | +| `3_output/exp_000_parity_v2/report.md` | What each spec reproduces of v2, parity by stage, and model fit | +| `3_output/exp_000_parity_v2/tables/parity_summary.csv` | The gate table: one row per taxon, region, and stage | +| `3_output/exp_000_parity_v2/tables/parity_terms.csv` | Term-by-term comparison (gitignored) | +| `3_output/exp_000_parity_v2/tables/metric_summary.csv` | Fit metrics per species; read the `oob_` rows | +| `3_output/exp_000_parity_v2/tables/grid_summary.csv` | Predicted effect of each habitat type, per species | +| `3_output/exp_000_parity_v2/figures/` | Parity plots per taxon | + +Read `run_record.md` first. If it does not say every species at +100 draws, the numbers describe a trial. + +### Step 4: Tell whether parity passed + +Open `tables/parity_summary.csv`: + +```r +library(data.table) + +parity <- fread( + "3_output/exp_000_parity_v2/tables/parity_summary.csv" +) +parity[, .( + taxon, region, stage, comparison, min_draws, + reachable_in_band_pct, verdict, indicative_verdict +)] +``` + +Read the `verdict` and `indicative_verdict` columns: + +| Value | Means | +| --- | --- | +| `pass` / `fail` | Against the targets in `utils/parity_targets.R` | +| `not gated (trial run)` | Fewer draws than v2, or a species subset; no verdict given | +| `not gated (store holds N draws)` | The store holds fewer draws than v2 | +| `no comparison` | No reachable terms to compare | + +Only a full run (`species = NULL`, `n_bootstraps = 100`) fills +`verdict`. A 100-draw run on a species subset fills +`indicative_verdict` instead. The proposed targets: + +- **Distributional** (plant and bird stages): at least 90% of + reachable terms with the v2 median inside the run's 10-90% band, + median standardized difference at most 0.25, and median + band-width ratio between 0.75 and 1.33. +- **Numerical** (mammals, fitted once by v2): every reachable term + within 1e-6 on iteration 1. + +Commit `run_record.md`, `report.md`, and the summary tables only +after a full run. `.gitignore` already limits what can be +committed from `3_output/`. + +## Test a change + +An experiment asks one question by changing one thing relative to +exp_000. Everything else (taxa, species, draws, seed) stays the +same unless it is the question. + +### Scaffold an experiment + +`new_experiment()` is base R and runs without `load_framework()`. +Preview first; a dry run writes nothing: + +```r +source("1_code/harness/new_experiment.R") +new_experiment("covariate scale", dry_run = TRUE) +#> exp_003_covariate_scale (dry run; nothing written) +#> ./1_code/experiments/exp_003_covariate_scale +#> ./2_pipeline/exp_003_covariate_scale +#> ./3_output/exp_003_covariate_scale +``` + +Then create it: + +```r +new_experiment("covariate scale") +``` + +It takes the next free number across `1_code/experiments/`, +`2_pipeline/`, and `3_output/`, and creates: + +- `1_code/experiments//`: `run.R` and `README.md` from + `_template/`, with id, title, author, and date filled in. +- `2_pipeline//logs/`: for intermediates (gitignored). +- `3_output//`: `figures/`, `tables/`, and a stub `report.md`. + +Leave `id` in `run.R` as it is. Write the question and design in +the experiment's `README.md` before you run anything. + +`run.R` loads the framework, calls `experiment_config()`, then +`run_experiment()`. Change one thing, using one of three levers, +from least to most invasive. + +### 1. Different candidate formulas: `stage_models` + +The covariates a run loads are read off its formulas, so this is +also how you try new covariates. Keys are `stage` (every taxon in +the run) or `taxon.stage` (one taxon). + +```r +config <- experiment_config( + id = "exp_003_covariate_scale", + taxa = c("bryophyte", "lichen", "mite", "vascular_plant"), + species = "parity_check", + n_bootstraps = 5, + stage_models = list( + # every v2 plant climate candidate, plus elevation + climate = extend_models("climate_plant_v2_full", "elevation"), + # mites only: a ladder built from three covariates + mite.climate = models_from_covariates( + c("MAP", "FFP", "CMD"), + form = "ladder" + ) + ), + workers = 4 +) +``` + +A covariate the dataset does not hold comes from the experiment. +Write a script in the experiment's folder that saves a CSV to +`2_pipeline//inputs/`, one row per `survey_unit_id`, and name +it in `covariate_files`. Never add columns to `0_data/` or change +`_setup/`: those hold the v2 data every experiment is compared +against. [`exp_002_soilgrids`][exp002] is the worked example. + +### 2. A different engine or rule: `specs` + +Build the standard specs, change a stage, and pass them in. +`replace_stage_method()` swaps each spec's habitat stage, and +drops v2 post-processing that needs coefficients: + +```r +specs <- lapply( + standard_specs(), replace_stage_method, + engine = "xgboost" +) + +config <- experiment_config( + id = "exp_003_covariate_scale", + specs = specs, + species = "parity_check", + n_bootstraps = 5 +) +``` + +To change one stage by hand instead, edit the spec list directly, +for example `specs$bird$stages[[2]]$selection <- "single"`. + +`validate_spec()` checks the combination before anything runs. A +rule that needs what the engine cannot give, such as standard +errors for inverse-variance averaging, stops the run with a +sentence that says so. [`exp_001_xgboost`][exp001] is the worked +example. + +### 3. A new method + +Copy the matching template from `1_code/methods/_templates/` into +the right folder of `1_code/methods/`, fill it in, and register +it. Check an engine with `check_engine()`, confirm it appears in +`list_methods()`, then name it in a spec as in lever 2: + +```r +check_engine("glm") # or check_engine(engine_mine()) +list_methods("engine") +``` + +The engine, selection, resampler, and metric contracts, and a +checklist, are in the [methods README][methods]. + +### Run it and read the comparison + +Run it at 5 draws to check it works, then at 100. Compare it with +an exp_000 run at the same number of draws. + +```r +source("1_code/experiments/exp_003_covariate_scale/run.R") +``` + +Outputs go to `2_pipeline//` and `3_output//`, laid out as +for exp_000. `run_experiment()` prints a headline per taxon, +region, and metric (the median difference from exp_000, and the +share of species that did better). It writes +`tables/comparison_metrics.csv`: one row per species and metric, +with both runs' 10-90% bands. Set `baseline = NULL` in +`experiment_config()` to skip the comparison. + +"Better" means higher AUC, deviance explained, and Spearman; lower +RMSE; and a calibration slope closer to 1. A difference is worth +reading only if it is consistent across species and large relative +to the bands. + +## Add or update a taxon module + +A module is configuration, not modelling code. It states what v2 +does for a taxon as a spec list, and the harness runs it. + +### The interface + +From the [modules README][modules] and `1_code/harness/spec.R`, a +module must: + +1. Define `_spec(...)` in + `1_code/modules//spec.R`, returning a spec list. +2. Return a spec that passes `validate_spec()`. Unknown fields are + errors, and every engine, rule, and resampler it names must be + registered. +3. Add its run to `standard_specs()` in + `1_code/modules/_shared/standard_specs.R`. + +`load_framework()` sources `_shared/` first, then every other +module folder, each in file-name order. The fields a spec may use +are defined by `spec_fields()`, one vector per level: + +```r +names(spec_fields()) +#> [1] "spec" "region" "stage" "resample" + +spec_fields()$stage +#> [1] "name" "models" "engine" +#> [4] "selection" "ic" "control" +#> [7] "family" "scope" "weighted" +#> [10] "base_terms" "response_transform" "row_filter" +#> [13] "min_detections" "extra_covariates" "post_process" +#> [16] "calibrate" "carry_as" "carry_scale" +#> [19] "carry_offset" "carry_from" "carry_from_as" +#> [22] "constants" "grid_constants" +``` + +A stage may also hold any argument its selection rule takes, as a +setting for that rule (for example `always_advance` for +`staged_bic`). Check an existing spec, and what it reproduces: + +```r +validate_spec(lichen_spec()) +spec_coverage(list(bird = bird_spec())) +``` + +### A minimal skeleton + +Modelled on `bird_spec()`. Replace every `<...>`. The model set +names must exist in `model_sets()` (`1_code/harness/model_sets.R`), +and the grid name must exist in the dataset manifest. + +```r +#' Build the v2 Spec for +#' +#' @return A spec list, as run_spec() consumes. +#' +#' @example # Example usage of the function +#' # validate_spec(_spec()) +_spec <- function() { + list( + taxon = "", + habitat_stage = "habitat", + response_name = "response", + response_transform = "identity", + family = "binomial", + weight_column = NULL, + + # One-sided formulas on the site columns, as in bird_spec() + regions = list( + north = list( + filter = ~ useNorth == 1, + grid = "", + habitat_models = "" + ), + south = list( + filter = ~ useSouth == 1, + grid = "", + habitat_models = "" + ) + ), + + stages = list( + list( + name = "climate", + models = "", + engine = "glm", + selection = "aic_average", + ic = "AICc", + carry_as = "Climate" + ), + list( + name = "habitat", + models = NULL, # per region: habitat_models + engine = "glm", + selection = "aic_best", + ic = "AICc", + carry_from = "climate", + carry_from_as = "Climate" + ) + ), + + resample = list(scheme = "spatial_block"), + + v2_coverage = list( + climate = v2_status("partial", ""), + habitat = v2_status("partial", "") + ), + + notes = "" + ) +} +``` + +Then add it to `standard_specs()`: + +```r +# in 1_code/modules/_shared/standard_specs.R +standard_specs <- function(plant_bootstrap = "spatial_block") { + list( + # ... existing runs ... + = _spec() + ) +} +``` + +A new taxon also needs data. The full sequence is in the +[modules README][modules-add]: v2 scripts into +`0_data/v2_scripts/`, `_setup/01`, `02`, `09`, and `08` extended, +candidate formulas spliced into `model_sets.R`, and each +taxon-specific behaviour recorded in +[`docs/taxon_quirks.md`][quirks]. Those `_setup/` steps are +maintainer work. + +## Check that nothing broke + +A change to `harness/`, `methods/`, or `modules/` affects every +experiment, past and future. Before relying on it, take two steps. + +**1. Run the tests.** These take seconds: + +```sh +Rscript 1_code/tests/run_tests.R +``` + +**2. Confirm the v2 results are unchanged.** Draws are seeded per +species and made in order, so draws 1-5 of a 5-draw run match +draws 1-5 of a 100-draw run. Run the `parity_check` species at 5 +draws to a scratch folder (set `pipeline_dir` and `out_dir` in +`experiment_config()`), then compare: + +```sh +Rscript 1_code/tests/compare_stores.R \ + 2_pipeline/exp_000_parity_v2 2_pipeline/ 5 +``` + +Every coefficient file should match to 1e-10. A change meant to +alter results should alter only what it says it does. More on both +tools is in the [tests README][tests]. + +## Working conventions + +- **Naming:** `snake_case` everywhere. Experiments are + `exp_NNN_short_description`, the same in `1_code/experiments/`, + `2_pipeline/`, and `3_output/`. Sequenced scripts take an `NN_` + prefix; entry points are `run.R`. Module folders use the plural + taxon name (`soil_mites/`); specs use the data slug (`mite`). + See the [root README conventions][readme-conv] and + [code_standards: project structure][cs-structure]. +- **Scripts and style:** every script has the standard header and + numbered `----` sections; tidyverse style, `%>%`, lines of 70 + characters or fewer. See [code_standards: templates][cs-templates] + and [code style][cs-style]. +- **Code review:** review against the five areas in + [code_standards: code review][cs-review]. Treat any change to + `harness/`, `methods/`, or `modules/` as a change to every + experiment: include passing tests and a `compare_stores.R` + result. +- **Commits from `3_output/`:** commit only runs that count, every + species at 100 draws, and check `run_record.md` says so. +- **Branching and pull requests:** + + > **TODO:** confirm with Brendan Casey. Neither this repository + > nor `code_standards` documents a branching or pull request + > workflow. + +## Where to get help + +- **Repository and data access:** Brendan Casey, + brendan.casey@ualberta.ca (contact in the + [root README][readme-contact]). +- **Taxon-specific v2 questions:** + + > **TODO:** confirm with Brendan Casey who owns each taxon + > (bryophytes, lichens, soil mites, vascular plants, mammals, + > birds). + +- **Design and contracts:** [`docs/framework_design.md`][design]. +- **Every taxon-specific v2 behaviour:** + [`docs/taxon_quirks.md`][quirks]. +- **Folder detail:** the READMEs in [`harness/`][harness], + [`methods/`][methods], [`modules/`][modules], + [`experiments/`][experiments], [`tests/`][tests], and + [`_setup/`][setup]. +- **Related tools:** + [sciCentRverse](https://github.com/ABbiodiversity/sciCentRverse) + and [sciSpatialR](https://github.com/ABbiodiversity/sciSpatialR). + + + +[readme]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/README.md +[readme-prereq]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/README.md#prerequisites +[readme-conv]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/README.md#conventions +[readme-contact]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/README.md#contact +[data-source]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/harness/data_source.R +[harness]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/harness/README.md +[methods]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/methods/README.md +[modules]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/modules/README.md +[modules-add]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/modules/README.md#adding-a-taxon-module +[experiments]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/experiments/README.md +[exp000]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/experiments/exp_000_parity_v2/README.md +[exp000-run]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/experiments/exp_000_parity_v2/run.R +[exp001]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/experiments/exp_001_xgboost/README.md +[exp002]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/experiments/exp_002_soilgrids/README.md +[tests]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/tests/README.md +[setup]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/1_code/_setup/README.md +[design]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/docs/framework_design.md +[quirks]: https://github.com/ABbiodiversity/sdmMethodsDev/blob/main/docs/taxon_quirks.md +[cs-structure]: https://github.com/bgcasey/code_standards#6-project-directory-structure +[cs-templates]: https://github.com/bgcasey/code_standards#1-script-templates +[cs-style]: https://github.com/bgcasey/code_standards#2-code-style +[cs-review]: https://github.com/bgcasey/code_standards#7-code-review diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..01746a5 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,15 @@ +# sdmMethodsDev + +The ABMI Science Centre's shared R&D repository for Species Models +3.0: a frozen cross-taxa test dataset, one pipeline that runs every +taxon's v2.0 model as configuration, and a v2 parity check that +every later experiment is compared against. + +This site is for Science Centre modellers. It is intended for +internal use. + +- **New here?** Start with [Getting started](getting_started.md): + setup, running the v2 baseline, starting an experiment, and + adding a taxon module. +- **The code and the full documentation** are in the + [repository](https://github.com/ABbiodiversity/sdmMethodsDev). diff --git a/docs/mermaid.lua b/docs/mermaid.lua new file mode 100644 index 0000000..7955a03 --- /dev/null +++ b/docs/mermaid.lua @@ -0,0 +1,41 @@ +-- --- +-- title: Render Mermaid Code Blocks on the Docs Site +-- author: Brendan Casey +-- created: 2026-10-07 +-- inputs: ```mermaid fenced code blocks in the site's pages +-- outputs: 
 blocks, and the mermaid.js
+--   script, in the HTML
+-- notes:
+--   - GitHub renders ```mermaid blocks; Quarto only renders its
+--     own {mermaid} cells. This filter lets one Markdown source
+--     show the diagram in both places.
+-- ---
+
+local function escape_html(text)
+  return (text:gsub("&", "&"):gsub("<", "<"):gsub(">", ">"))
+end
+
+local used = false
+
+function CodeBlock(block)
+  if not block.classes:includes("mermaid") then
+    return nil
+  end
+
+  used = true
+  return pandoc.RawBlock(
+    "html",
+    '
' .. escape_html(block.text) .. "
"
+  )
+end
+
+function Pandoc(doc)
+  if used then
+    quarto.doc.include_text("after-body", [[
+]])
+  end
+  return doc
+end
diff --git a/docs/taxon_quirks.md b/docs/taxon_quirks.md
index 84da41f..fec697f 100644
--- a/docs/taxon_quirks.md
+++ b/docs/taxon_quirks.md
@@ -91,7 +91,7 @@ prediction for plants, and a calibrated probability for mammals.
 | Factor levels | `vegc` 22, `soilc` 17, `method` 4, `block` 60. Stored order matters, since a level absent from one bootstrap sample must not renumber the rest | OK. Until 2026-09-29 `method` was loaded as `method_veg` and `method_soil` and missed the lookup, so it fell back to alphabetical order with `1SPM` as reference instead of `PC`; `apply_factor_levels()` now follows the suffixes |
 | Column suffixes | Six columns arrive suffixed `_veg` or `_soil` and are rewritten by `term_map()` | OK |
 | No coordinates | 0 of 215,758 rows in `sites.csv` carry latitude or longitude | GAP blocks spatial resampling and every spatial term |
-| Reference output | The packaged `Birds2024.RData` is absent from the drive. `_setup/04` rebuilds it from the per-draw CSVs, on the standardized term template, and `_setup/05` carries it into `v2_results.csv` | OK rebuilt and compared. `modules/birds/standardize.R` ports the translation onto the standardized template and reproduces the packaged values exactly. Note the package labels draws 2 to 100 in file-listing order (`b2` is draw 10) |
+| Reference output | The packaged `Birds2024.RData` is absent from the drive's `Results/`; only the per-draw CSVs and an archived copy survive. The parity reference is ABMIexploreR's packaged bird coefficients (`_setup/08`), which match it to ~1e-15 | OK compared. `modules/birds/standardize.R` ports the translation onto the standardized template and reproduces the packaged values exactly. Note the package labels draws 2 to 100 in file-listing order (`b2` is draw 10) |
 
 **The published bird results come from a different data vintage.**
 Run on the current `Stratified.Rdata`, v2's own climate code gives
@@ -108,7 +108,7 @@ check:
 
 | Date | On the bird drive |
 | --- | --- |
-| 2025-12-12 / 15 | Per-draw climate and landcover csvs in `Results/Archive/`, the reference `_setup/04` packages |
+| 2025-12-12 / 15 | Per-draw climate and landcover csvs in `Results/Archive/`, packaged by v2's `08.PackageCoefficients.R` |
 | 2026-01-16 | `Results/Archive/Birds2024.RData`, packaged from them |
 | 2026-02 to 03 | New WildTrax pulls; `Data/Archive/2025/Wrangled.Rdata` rebuilt |
 | 2026-08-19 | `Data/Archive/2025/Stratified.Rdata` rebuilt, in place |
@@ -122,7 +122,7 @@ only to 40%. The published results were fitted on a
 **Resolved 2026-10-02.** A copy of the pre-rebuild file was
 recovered and is kept at
 `work_abmi/1_projects/active/sdmMethodsDev/remote/birds_data_v2/Stratified.Rdata`,
-which `SDM_BIRD_DATA` now defaults to in `_setup/01`, `02` and `04`.
+which `SDM_BIRD_DATA` now defaults to in `_setup/01` and `02`.
 Its draws hold 58,210 surveys (the rebuilt file's hold 58,035), and
 v2's own climate code run on it reproduces the published AMRO draw-1
 coefficients to 2e-15. With the test dataset re-harmonized from it,
@@ -154,7 +154,7 @@ AUC would have been silently invalid. Fixed with `as.numeric()`.
 
 | Quirk | What v2 does | Status |
 | --- | --- | --- |
-| Hurdle structure | Binomial presence times Gamma abundance given presence, on a log link. Their product is total abundance | OK `mammal_spec(part = "hurdle")`, both halves together (hurdle.R) |
+| Hurdle structure | Binomial presence times Gamma abundance given presence, on a log link. Their product is total abundance | OK, the generic `hurdle` rule fits both halves through the stage's engine; v2's table-building is the `mammal_v2_habitat_tables()` post-process step (hurdle.R) |
 | Lure correction | Estimated **only** from numbered ABMI grid sites, which are the ones with matched lured and unlured deployments. Applied as `sign(Count) / lure_ratio`, then rescaled to a maximum of 1 | OK |
 | Abundance winsorizing | Capped at the 99th percentile of abundance given presence, before fitting | OK |
 | Season | Separate summer and winter models, with `wt_summer` and `wt_winter` passed as `glm` weights | OK |
@@ -237,9 +237,10 @@ plant taxa hold 100. Both sit upstream of the modelling, so the file
 cannot be salvaged, only regenerated.
 `1_code/_setup/03_rerun_bryophyte_v2_reference.R` does that for the
 climate stage, and has been run: 134 of 134 species, 100 draws each,
-no failures. Separately, v2's `COEFS.RData` holds complete bryophyte
-arrays, and `_setup/05` builds the bryophyte rows of `v2_results.csv`
-from it. The gate reads neither yet, so bryophytes are not compared.
+no failures. Separately, ABMIexploreR publishes complete bryophyte
+arrays (matching v2's `COEFS.RData` to ~1e-15), and `_setup/08` builds
+the bryophyte rows of `v2_results.csv` from them, so the gate compares
+bryophytes.
 
 ## Shared across all three