Repository navigation
Development - #3
Merged
Merged
Conversation
…grity and consistency
…y handling and NA preservation
…pdate README to reflect current project status, known gaps, and setup instructions.
Adding BirdModels scripts to github repo.
- Removed empty .gitkeep file from plants module. - Added new spec.R file for soil mites, defining the v2 specification for the taxon. - Introduced a new spec.R file for vascular plants, detailing the v2 specification. - Updated README.md to reflect changes in module structure, referencing lichens as a model for new taxa. - Created framework_design.md to outline the architecture and purpose of the modelling framework. - Added taxon_quirks.md to document specific behaviours and quirks for each taxon in the modelling pipelines.
- Implemented the run_record function to create a Markdown or plain-text file summarizing the results of a pipeline run. - Introduced helper functions for formatting values, rendering tables, and discovering result stores. - The output includes provenance information, configuration details, coverage statistics, and metrics for each taxon and region. - Ensured consistent formatting for comparison across runs, with fixed decimal places and ordered output. - Created initial run record for experiment exp_000_parity_v2.
- Introduced a base random seed (`boot_seed`) for consistent random draws across species in `run.R`. - Updated `resample.R` to derive species-specific seeds from the base seed, ensuring independent resampling. - Modified `run_model.R` to utilize the new seeding mechanism, allowing for reproducible results. - Adjusted plant and mammal specifications to remove individual seeds, relying on the experiment's base seed. - Improved documentation to clarify the new seeding approach and its implications for reproducibility. - Fixed issues with the output record to accurately reflect the seed used during the run.
…rics for improved analysis
- Created report.md detailing the coverage and model fit for various taxa. - Added coverage.csv summarizing the presence of coefficients and grid availability for taxa across regions. - Introduced metric_summary.csv with detailed metrics for each species, including AUC, calibration slope, and RMSE. - Added parity_summary.csv to summarize the comparison of terms across taxa and regions, including in-band percentages and standardized differences.
- Introduced `v2_coverage` in taxon specifications to clearly state what each spec reproduces of v2, including climate, habitat, and resampling stages. - Added `v2_status()` function to standardize the status reporting for v2 coverage. - Updated `spec_coverage()` to tabulate coverage across all specs for reporting. - Enhanced comments and documentation throughout the codebase to clarify the purpose and functionality of various components, particularly regarding v2 reproduction and model fitting. - Adjusted README and framework design documentation to reflect changes in v2 coverage reporting and clarify known gaps in the current implementation.
- Updated the alignment review with detailed status updates for each action in Phases 1 to 4, indicating completion and specific improvements made. - Enhanced the taxon quirks documentation to reflect the current status of various quirks related to climate selection, model fitting, and data handling for both birds and mammals. - Clarified the handling of climate candidate sets, carrying climate forward, and the BIC rule, ensuring consistency with v2's methodology. - Addressed gaps in the documentation regarding bootstrap processes, climate extraction, and the handling of south habitat models. - Noted discrepancies in published plant models and clarified the need for re-running v2 on stored draws for accurate parity.
- Updated `parity_summary.csv` to include additional metrics and comparisons for various taxa and regions. - Added `04_plot_parity.R` script to generate detailed parity figures for each taxon, including mean parity visualizations. - Modified `README.md` to document the new plotting functionality and clarify output file structures. - Revised alignment review documentation to reflect the current status of parity checks and targets. - Expanded taxon quirks documentation with confirmed data lineage for bird results. - Introduced a new `cap` file for numeric integer checks and bird row specifications.
- Disable model running and collection in run.R for trial runs. - Update parity_targets.R to include band-width ratio and negligible term handling. - Modify v2_self_agreement.R to calculate and report band-width ratios and negligible terms. - Enhance report.md to reflect new metrics and their implications. - Revise parity_summary.csv to include negligible terms and median band ratios. - Update README.md to clarify output files and metrics. - Adjust alignment review documentation to reflect changes in target agreements and evaluation criteria.
- Updated git commit hash in run_record.md - Modified written_at timestamp in run_record.md - Adjusted metric values for bird species in metric_summary.csv - Revised parity summary metrics for bird and bryophyte species in parity_summary.csv - Enhanced habitat comparison metrics for bird and bryophyte species in parity_summary.csv
…e/birds_data_v2 and clarify documentation regarding data sources and harmonization.
- Removed obsolete 'cap' file. - Updated framework design documentation to reflect changes in the pipeline structure and method organization. - Added a comprehensive 'getting started' guide for running the v2 baseline, reading outputs, and testing changes. - Enhanced clarity on the roles of specs, stages, engines, and selection rules in the modeling process. - Improved organization of methods and modules for better maintainability and usability.
- Modified the parity_summary.csv to reflect updated metrics and draw limits for various taxa across climate and habitat stages. - Adjusted the documentation in getting_started.md to point to the correct section for output definitions.
- Updated README.md to include new engine `gbm` and details on `replace_stage_method()` and `extend_models()`. - Revised flowchart and steps in README.md to clarify experiment comparisons and outputs. - Added section in framework_design.md on handling new covariates not present in the dataset, detailing extraction and integration processes. - Expanded getting_started.md with instructions for adding covariates from experiments and automatic comparison with exp_000.
- Replaced `gbm` with `xgboost` in the engines section of README.md and framework_design.md. - Updated selection rules to include `hurdle` in README.md and framework_design.md. - Modified experiment configuration in framework_design.md to use `xgboost`. - Clarified hurdle structure in taxon_quirks.md to reflect the new generic `hurdle` rule. - Adjusted experiment names and descriptions to align with the new engine and its capabilities.
- Updated comments and documentation across multiple scripts to clarify that the test dataset and v2 results are read from the published copies on ABMI-DATA2. - Introduced a new `data_source.R` file to centralize the logic for resolving dataset paths and checking their integrity. - Modified functions to utilize `test_dataset_dir()` for retrieving the dataset folder, ensuring consistency and clarity in data access. - Enhanced README and documentation to reflect changes in dataset handling and clarify the workflow for accessing datasets.
- Deleted the script `05_harmonize_v2_results.R` which was responsible for harmonizing the v2 results into a single parity reference. - This script included functions for reading and processing plant, mammal, and bird coefficient data, as well as assembling the final results into a CSV format. - The removal is part of a refactor to streamline the data processing pipeline and eliminate legacy code that is no longer needed.
… and update references to ABMIexploreR results. Add run script for rebuilding test dataset and v2 reference with logging.
- Changed references from `comparison_*.csv` to `comparison_metrics.csv` in README.md to clarify the output files generated during experiments. - Enhanced descriptions of the comparison metrics in README.md, including details about the data contained in `comparison_metrics.csv`. - Updated the getting_started.md to reflect the changes in output files and provide clearer instructions on how to interpret the comparison results.
… across multiple scripts
…ty and consistency - Updated comments in `single.R` to clarify the purpose of the `single` rule. - Revised notes in `staged_bic.R` to enhance understanding of model selection process. cleaned up wrong or stale comments, condensed narrative explanations, dropped changelog notes, and removed duplicate comments repeated for the same shared logic across files, keeping the one at the definition.
…les, and tests - Introduced README.md files in the following directories: - `1_code/_setup/`: Details the process for building the frozen test dataset and v2 reference, including script descriptions and environment variables. - `1_code/experiments/`: Outlines the structure for experiments, including how to add new experiments and the required folder organization. - `1_code/harness/`: Describes the shared pipeline, its functions, and the experiment configuration process. - `1_code/modules/`: Provides guidelines for taxon modules, including the module interface and how to add new modules. - `1_code/tests/`: Lists available tests and their purposes, along with instructions for running them. These changes enhance the clarity and usability of the codebase for current and future contributors.
- Created .github/workflows/docs.yaml to automate building and deploying the Quarto documentation site to GitHub Pages on pushes to the main branch. - Added docs/.gitignore to exclude Quarto build artifacts from version control. - Introduced docs/_quarto.yml to configure the Quarto project, specifying the website structure and rendering settings. - Created index.md as the homepage for the documentation site, providing an overview and links to getting started. - Added mermaid.lua filter to render Mermaid diagrams in Markdown, enabling compatibility with both GitHub and Quarto rendering.
Improve code efficiency
Collaborator
Author
|
Getting the R&D playground usable and able to reach parity with the V2 model results. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.