Repository navigation
Description
Phase 5: merging, evidence sources, provenance, and PTM integration
Summary
Phase 5 of the refactor toward a generic network-backend API
(tracked in MSstats-ai/todos/active/TODO-MSBio-20261001_refactor_api.md).
It adds the tools for working with a network after get_network()
returns it: combining networks, filtering by evidence source, recording
where a network came from, and interpreting PTM networks with a global
profiling experiment. Phase 5 shipped as four PRs, all merged to devel.
Scope
- 5a.
merge_networks()and the shared backend contract test helper (Added merge_networks() and a shared backend contract test helper #130,619c4fc, 2026-10-09) - 5b. Exported INDRA evidence-source constants (Added exported INDRA evidence-source lists #131,
a05d90d, 2026-10-09) - 5c. Network provenance and
save_network()/load_network()(Added network provenance and save_network() / load_network() #132,4daf101, 2026-10-09) - 5d. PTM integration:
add_entity_values(),filter_by_ptm_site(), new PTM vignette (fix(ptm): Fix PTM filtering and add function to add global profiling results to network #134,bdd0d80, 2026-10-14)
Technical Details
5a. merge_networks() (#130)
merge_networks(...) combines networks from one or more get_network()
calls (different backends, query types, or cutoffs) into one list that
passes validate_network().
- Edges are deduplicated on
source+target+backend_database+
statement_id. CoGEx lists an undirected Complex in both directions
under one hash, and this key keeps both rows. When an edge appears in
several networks, theirquery_typevalues are combined (;-joined, in
order of first appearance) and the copy with the largerevidence_count
is kept (the first one on ties; anNAcount loses to any count). A
message is shown when the copies differ inevidence_countor
confidence. The same relation from two backends stays two rows.
Columns that only one network has are filled withNA. - Nodes are keyed on
id+site, sincenodes$idrepeats for PTM
site rows.node_rolevalues are combined (;-joined), and
validate_network()now splitsnode_roleon;.included_in_query
isany()across the networks, because it describes the query that was
sent.has_measured_sitesis set per proteinidafter merging. - Without
entities,measured,logFC, andadj.pvaluemust agree
for each node key. If they don't, the merge stops, naming the nodes and
the columns that differ. Withentities, those columns are
recomputed from the table; nodes not in it becomemeasured = FALSE
withNAstatistics. regulatorsandprovenanceelements are row-bound with column fill;
provenanceis not deduplicated, so two retrievals stay two rows. Other
list elements are dropped with a message. Each input is validated
first, so errors name the network (network 2, or its argument name).- Test helper
tests/testthat/helper-contract.R:
expect_backend_contract(backend, entities, ...)checks the shape of
backend_capabilities(). For each query type it lists, the helper runs
get_network(), validates the result, and checksbackend_database,
query_type, and that the interactions are within the capabilities. It
also checks thatmerge_networks(network, network)gives the network
back. It runs against the INDRA mock intest-backend-indra.R, so every
future backend gets the same checks.
5b. INDRA evidence-source constants (#131)
- Exported
INDRA_DATABASE_SOURCES(22:acsn,bel,bel_lc,
biogrid,cbn,conib,creeds,crog,ctd,dgi,drugbank,
hprd,minerva,omnipath,pc,pe,psp,signor,tas,
trrust,ubibrowser,vhn) andINDRA_TEXT_MINED_SOURCES(11:
eidos,geneways,gnbr,isi,medscan,reach,rlimsp,
semrep,sparser,tees,trips), documented on one help page,
?indra_evidence_sources.
Usage:get_network(..., evidence_sources = INDRA_DATABASE_SOURCES). - The lists come from
indra.util.statement_presentation(INDRA 1.23.0),
the module CoGEx's own app uses. These are INDRA's internal names
(psp,pc,bel_lc,pe,vhn). A live CoGEx check on 1926
statements confirmed these are the names insource_counts. The
SOURCE_INFOnames (phosphosite,biopax, ...) would have silently
filtered out PhosphoSitePlus, Pathway Commons, and BEL. - An unknown source name warns rather than errors, because INDRA adds
sources between releases. For a known alias, the warning names the
internal name to use ("phosphosite"→"psp").
getSubnetworkFromIndra(sources_filter =)warns too. backend_capabilities(indra_backend())$evidence_sourcesis now
list(database =, text_mined =).
5c. Provenance, save_network(), load_network() (#132)
- Every
get_network()result hasnetwork$provenance: a data.frame with
one row per call and the columnsbackend_database,query_type,
retrieved_at(UTC, taken when the response arrives),
backend_version,backend_url,organism,parameters(JSON of the
shared and backend arguments), andpackage_version. It joins to edges
onbackend_database+query_type. INDRA CoGEx has no version
endpoint, so itsbackend_versionisNAandretrieved_atis the
only record of when the network was retrieved. - It's a list element rather than an edge column because a timestamp
describes a retrieval, not a statement. As an edge column,
merge_networks()would have to pick one of two timestamps per edge. save_network(network, file)validates the network and writes an.rds
file. It warns whenprovenanceis missing (a sign the list was rebuilt
by hand).load_network(file)reads the file and validates it, so a
file saved under an older contract gives a clear error.filter_by_curation()keepsprovenance.deleteEdgeFromNetwork()and
filterSubnetworkByContext()take the tables separately and can't keep
it; that limitation was accepted when the network was kept a plain list.
5d. PTM integration with a global profiling experiment (#134)
MSstatsPTM's groupComparisonPTM() returns site changes adjusted for
protein abundance (ADJUSTED.Model) and the global protein changes
(PROTEIN.Model). Showing both on one network used to need one stacked
input table, which also put every significant global protein into the
query. 5d separates the two steps: build the network from the PTM table,
then copy the global values onto its nodes.
add_entity_values(network, entities, overwrite = FALSE) copies an
entity table's logFC / adj.pvalue onto matching nodes. It sends no
query.
entitiesmust come fromconvert_ids(); at least one row has to be
grounded.- Nodes are matched on
id+sitefirst. Nodes left unmatched are then
matched on grounding (namespace+entity_id). The grounding step
covers latent nodes, which have the backend's name as theirid: a
kinase added withinclude_entitiescomes back from INDRA asMAPK1,
while the global table hasP28482. A node that matches several rows by
grounding (isoforms) gets no values, with a message. - Matched rows get
measured = TRUE. A protein that has only site rows
gets a new protein-level row (site = NA) holding the global values, so
cytoscapeNetwork()colors the protein node with no display change. - Nodes that already have values cause an error unless
overwrite = TRUE.
Rows that match no node are not added. Edges,regulators, and
provenanceare unchanged. The source of the values is not recorded.
filter_by_ptm_site(network) takes a network and returns a network,
like filter_by_curation(); it is not a get_network() argument. It
keeps an edge when the edge's site is one of the measured sites of its
target. Compared with the internal .filterByPtmSite() that
getSubnetworkFromIndra() uses:
- Edges are checked one at a time, not pooled per
source+target+
interaction. Before, an edge for an unmeasured site survived whenever
another statement for the same pair hit a measured site. Checking per
edge also makes the result independent of whether this filter or
filter_by_curation()runs first. - Node sites are split on
_and;; before, only on_. - A node is dropped only when it had edges and lost all of them, so
isolated nodes stay, andprovenanceis kept. - Edges with
site = NAare dropped. That includes every non-modification
statement and every STRING edge, which is documented. The function stops
when no edges remain. getSubnetworkFromIndra(filter_by_ptm_site = TRUE)deliberately keeps
the old helper until Phase 7, so existing scripts keep their results.
Other changes in #134:
- UniProt FASTA headers.
prepare_entities()/convert_ids()accept
them (sp|P32004|L1CAM_HUMAN, plus a site suffix for PTMs), as FragPipe
writes them.idandparent_idkeep the header; the conversion uses
the accession or mnemonic (UNIPROT_HEADER_PATTERN,
.parse_uniprot_headers(),.get_grounding_inputs()). - Bug fix:
GlobalProtein = "missing". MSstatsPTM writes"missing"
for sites whose protein has no global result.prepare_entities()took
it as the parent, leaving those rows ungrounded (1143 of 3113 CPTAC
sites).annotateProteinInfoFromIndra()had the same bug (3875 of
16356 IFNG rows). Both now parse the parent from the site ID. - Bug fix: sites on several group members.
.stripPtmSite()on a
protein group dropped every member after the first site
("P13747_S148;P23132_T20"→"P13747"). The new
.stripPtmSiteFromGroups()strips each member and joins the unique
parents, both for the"missing"fallback and when there is no
GlobalProteincolumn. - Vignette.
vignettes/PTM-Analysis.Rmdis rewritten as "PTM Analysis
with Global Profiling", on CPTAC CCRCC (tumor vs normal, TMT, FragPipe;
Li et al., Cancer Cell 2023, doi:10.1016/j.ccell.2023.06.009). It covers
236 sites at adj.p < 0.05. The sites-only subnetwork has 30 edges, 1
afterfilter_by_ptm_site(). With ten common kinases added, it has 599
edges, 19 after filtering.add_entity_values()then colors PML, LMNA,
BRD4, IQGAP1, CDK18, and the kinases MAPK1, MAPK3, AKT1, and PRKACA. The
INDRA chunks don't run at build time; their results load from
inst/extdata/cptac-ccrcc-network.rds, made by
inst/script/cptac-ccrcc-network.R.garrido-2024.csvwas removed.
Affected Files
R/network-postprocess.R:merge_networks(),add_entity_values(),
filter_by_ptm_site(), and helpersR/network-io.R(new): provenance,save_network(),load_network()R/backend-indra.R,R/AllGenerics.R: evidence-source constants,
provenance inget_network(), capabilities, FASTA-header grounding
inputsR/contract.R:node_rolesplit on;R/entities.R: FASTA headers,GlobalProtein = "missing"fixR/annotateProteinInfoFromIndra.R:"missing"fix,
.stripPtmSiteFromGroups()R/getSubnetworkFromIndra.R: source-name warningvignettes/PTM-Analysis.Rmd,inst/extdata/cptac-ccrcc-*.csv/.rds,
inst/script/cptac-ccrcc-network.R- Tests:
helper-contract.R,test-merge-networks.R,
test-network-io.R,test-ptm-integration.R,test-backend-indra.R,
test-entities.R,test-annotateProteinInfoFromIndra.R NAMESPACE,NEWS.md,man/for each new export
Activity
[-]Network-backend API refactor[/-][+]Network-backend API refactor: Phase 5[/+]