foundrHarmony Data Pipeline & Normalization Methodology

Data Pipeline & Normalization Methodology

This document details the underlying data models, transformation algorithms, filtering criteria, and multi-study consolidation mechanisms in foundrHarmony.

---

1. Tidy Long-Format Data Schema

All data processing in foundrHarmony standardizes heterogeneous raw input matrices (wide CSVs, multi-tab Excel sheets, time-series traces) into a clean, canonical long-format data frame schema:

Column	Type	Description	Example Values
dataset	character	Data source identifier	"Physio", "PlaMet", "LivRna", "IsletCalc"
strain	character	Collaborative Cross founder strain (1 of 8)	"AJ", "B6", "129", "NOD", "NZO", "CAST", "PWK", "WSB"
sex	character	Biological sex	"M", "F"
animal	character	Unique animal or subject identifier	"1001", "1002", "F012"
condition	character	Experimental condition or diet treatment	"HC_LF" (High Carb Low Fat), "HF_LC" (High Fat Low Carb)
trait	character	Phenotypic measurement name	"Glucose_0min", "AUC_Glucose", "Insulin"
value	numeric	Quantified measurement value	112.5, 0.0482, 14.2

---

2. Normal Quantile Rank Transformation

To ensure statistical robustness and comparability across disparate measurement scales (e.g. mRNA counts vs. metabolite peak areas vs. weight in grams), foundrHarmony transforms raw trait values using normal quantile rank scores via nqrank().

Mathematical Definition

For a vector of $n$ observation values $X = (x_1, x_2, \dots, x_n)$ for a given trait:

1. Compute sample ranks $r_i = \text{rank}(x_i)$, handling ties by average rank.
2. Calculate uniform rank probabilities $u_i = \frac{r_i - 0.5}{n}$.
3. Apply the inverse standard normal cumulative distribution function (probit function $\Phi^{-1}$):

$$q_i = \Phi^{-1}\left( \frac{r_i - 0.5}{n} \right)$$

The resulting normalized values $q_i \sim \mathcal{N}(0, 1)$ follow a standard normal distribution.

Group-wise Execution (R/normalscores.R)

normalscores() applies nqrank() across all traits in a long data frame:

# Internal group-wise quantile transformation
traitData <- object %>%
  dplyr::group_by(trait) %>%
  dplyr::mutate(value = nqrank(value, jitter = jitter)) %>%
  dplyr::ungroup()

---

3. Data Processing & Quality Control Filtering in harmonize()

The orchestration runner harmonize() applies strict statistical quality filters before serializing output files:

flowchart TD
    step1["1. Raw Data Ingestion via userHarmony()"]
    step2["2. Group-wise Normalization via normalscores()"]
    step3["3. Linear Model Fitting via foundr::strainstats()"]
    step4["4. Drop Traits with Missing ANOVA p-values"]
    step5["5. Drop Traits Missing Strain Coverage (< minstrains)"]
    step6["6. Variance Partitioning via foundr::partition()"]
    step7["7. Serialize Outputs (Stats.rds, Data.rds, Signal.rds)"]

    step1 --> step2
    step2 --> step3
    step3 --> step4
    step4 --> step5
    step5 --> step6
    step6 --> step7

    classDef proc fill:#1f77b4,stroke:#333,stroke-width:2px,color:#fff
    classDef filter fill:#ff7f0e,stroke:#333,stroke-width:2px,color:#fff
    classDef out fill:#2ca02c,stroke:#333,stroke-width:2px,color:#fff

    class step1,step2,step3,step6 proc
    class step4,step5 filter
    class step7 out

Trait Filtering Rules

1. ANOVA Convergence Filter: Drops traits where linear model fitting fails, resulting in is.na(p.value) in strainstats() output.
2. Founder Strain Representation Filter: Identifies total unique founder strains per trait and drops any trait represented in fewer than minstrains (default $N_{\text{strains}} - 2$).

---

4. Multi-Study Consolidation & traitObject S3 Class

When multiple dataset files (Physio, PlaMet, LivRna, etc.) have been harmonized, bind_traits() combines them into a unified traitObject structure:

traitObject <- list(
  Data   = traitData,   # Concatenated long-format measurements
  Signal = traitSignal, # Concatenated fitted strain-condition signals
  Stats  = traitStats   # Concatenated ANOVA statistics and cell means
)
class(traitObject) <- c("traitObject", class(traitObject))

This S3 traitObject serves as the primary input into the foundrShiny exploratory web dashboard.