---
output:
  html_document
bibliography: ref.bib
---

# Correction diagnostics 

```{r setup, echo=FALSE, results="asis"}
library(rebook)
chapterPreamble()
```

## Motivation

Ideally, batch correction would remove the differences between batches while preserving the heterogeneity within batches.
In the corrected data, cells of the same type should be intermingled and indistinguishable even if they come from different batches, while cells of different types should remain well-separated.
Unfortunately, we rarely have prior knowledge of the underlying types of the cells, 
making it difficult to unambiguously determine whether differences between batches represent geniune biology or incomplete correction.
Indeed, it could be said that all correction methods are at least somewhat incorrect (Section \@ref(sacrificing-differences)), though that not preclude them from being useful.

In this chapter, we will describe a few diagnostics that, when combined with biological context, can be used to identify potential problems with the correction.
We will recycle the `mnn.out`, `tab.mnn` and `clusters.mnn` objects that were produced in Section \@ref(mnn-correction).
For the sake of brevity, we will not reproduce the relevant code - see Chapter \@ref(integrating-datasets) for more details.

```{r, results='hide', echo=FALSE}
extractCached("batch-correction.Rmd", chunk="mnn-clustering", 
    objects=c("mnn.out", "tab.mnn", "clusters.mnn"))
```

```{r}
mnn.out
tab.mnn
```

## Mixing between batches

The simplest way to quantify the degree of mixing across batches is to test each cluster for imbalances in the contribution from each batch [@buttner2019test].
This is done by applying Pearson's chi-squared test to each row of `tab.mnn` where the expected proportions under the null hypothesis proportional to the total number of cells per batch.
Low $p$-values indicate that there are significant imbalances
In practice, this strategy is most suited to experiments where the batches are technical replicates with identical population composition; 
it is usually too stringent for batches with more biological variation, where proportions can genuinely vary even in the absence of any batch effect. 

```{r}
library(batchelor)
p.values <- clusterAbundanceTest(tab.mnn)
p.values
```

```{r, echo=FALSE}
stopifnot(median(p.values) <= 0.01)
```

```{r, eval=FALSE, echo=FALSE}
# Proving that this is also the case with the 
# original kBET implementation, as of 4c9dafa.
library(kBET)
a <- matrix(rnorm(10000), ncol=10)
b <- matrix(rnorm(10000, 10), ncol=10)
ID <- rep(c(1,2,1,2), c(900, 100, 100, 900))
stats <- kBET(rbind(a, b), ID)
stats$summary[,3] # all-zero p-values.
```

We favor a more qualitative approach where we compute the variance in the log-normalized abundances across batches for each cluster.
A highly variable cluster has large relative differences in cell abundance across batches;
this may be an indicator for incomplete batch correction, e.g., if the same cell type in two batches was not combined into a single cluster in the corrected data.
We can then focus our attention on these clusters to determine whether they might pose a problem for downstream interpretation.
Of course, a large variance can also be caused by genuinely batch-specific populations,
so some prior knowledge about the biological context is necessary to distinguish between these two possibilities.
For the PBMC dataset, none of the most variable clusters are overtly batch-specific, consistent with the fact that our batches are effectively replicates.

```{r}
rv <- clusterAbundanceVar(tab.mnn)

# Also printing the percentage of cells in each cluster in each batch:
percent <- t(t(tab.mnn)/colSums(tab.mnn)) * 100 
df <- DataFrame(Batch=unclass(percent), var=rv)
df[order(df$var, decreasing=TRUE),]
```

## Preserving biological heterogeneity

Another useful diagnostic check is to compare the pre-correction clustering of each batch to the clustering of the same cells in the corrected data.
Accurate data integration should preserve population structure within each batch as there is no batch effect to remove between cells in the same batch.
This check complements the previously mentioned diagnostics that only focus on the removal of differences between batches.
Specifically, it protects us against scenarios where the correction method simply aggregates all cells together, which would achieve perfect mixing but also discard the biological heterogeneity of interest.
To illustrate, we will use clustering results from the analysis of each batch of the PBMC dataset:

```{r, results='asis', echo=FALSE}
extractCached("tenx-filtered-pbmc3k-4k-8k.Rmd", 
    chunk="clustering", objects=c("all.sce", "all.dec"))
```

```{r}
pbmc3k <- all.sce$pbmc3k
table(colLabels(pbmc3k))
pbmc4k <- all.sce$pbmc4k
table(colLabels(pbmc4k))
```

Ideally, we should see a many-to-1 mapping where the post-correction clustering is nested inside the pre-correction clustering.
This indicates that any within-batch structure was preserved after correction while acknowledging that greater resolution is possible with more cells.
We quantify this mapping using the `nestedClusters()` function from the `r Biocpkg("bluster")` package,
which identifies the nesting of post-correction clusters within the pre-correction clusters.
Well-nested clusters have high `max` values, indicating that most of their cells are derived from a single pre-correction cluster.

```{r}
library(bluster)
tab3k <- nestedClusters(ref=paste("before", colLabels(pbmc3k)),
    alt=paste("after", clusters.mnn[mnn.out$batch==1]))
tab3k$alt.mapping
```

We can visualize this mapping for the PBMC dataset in Figure \@ref(fig:heat-after-mnn).
Ideally, each row should have a single dominant entry close to unity.
Horizontal stripes are more concerning as these indicate that multiple pre-correction clusters were merged together,
though the exact level of concern will depend on whether specific clusters of interest are gained or lost.
In practice, more discrepancies can be expected even when the correction is perfect, due to the existence of closely related clusters that were arbitrarily separated in the within-batch clustering.

```{r heat-after-mnn, fig.asp=1.8, fig.cap="Comparison between the clusterings obtained before (columns) and after MNN correction (rows). One heatmap is generated for each of the PBMC 3K and 4K datasets, where each entry is colored according to the proportion of cells distributed along each row (i.e., the row sums equal unity)."}
library(pheatmap)

# For the first batch:
heat3k <- pheatmap(tab3k$proportions, cluster_row=FALSE, cluster_col=FALSE,
                   main="PBMC 3K comparison", silent=TRUE)

# For the second batch:
tab4k <- nestedClusters(ref=paste("before", colLabels(pbmc4k)),
                        alt=paste("after", clusters.mnn[mnn.out$batch==2]))
heat4k <- pheatmap(tab4k$proportions, cluster_row=FALSE, cluster_col=FALSE,
                   main="PBMC 4K comparison", silent=TRUE)

gridExtra::grid.arrange(heat3k[[4]], heat4k[[4]])
```

We use the adjusted Rand index (`r link("comparing-different-clusterings", "OSCA.advanced")`)
to quantify the agreement between the clusterings before and after batch correction. 
Recall that larger indices are more desirable as this indicates that within-batch heterogeneity is preserved,
though this must be balanced against the ability of each method to actually perform batch correction.

```{r}
library(bluster)
ri3k <- pairwiseRand(clusters.mnn[mnn.out$batch==1], colLabels(pbmc3k), mode="index")
ri3k
ri4k <- pairwiseRand(clusters.mnn[mnn.out$batch==2], colLabels(pbmc4k), mode="index")
ri4k
```

```{r, echo=FALSE}
# Checking that it works.
stopifnot(ri3k > 0.64)
stopifnot(ri4k > 0.7)
```

We can also break down the ARI into per-cluster ratios for more detailed diagnostics (Figure \@ref(fig:rand-after-mnn)).
For example, we could see low ratios off the diagonal if distinct clusters in the within-batch clustering were incorrectly aggregated in the merged clustering.
Conversely, we might see low ratios on the diagonal if the correction inflated or introduced spurious heterogeneity inside a within-batch cluster.

```{r rand-after-mnn, fig.asp=1.8, fig.cap="ARI-derived ratios for the within-batch clusters after comparison to the merged clusters obtained after MNN correction. One heatmap is generated for each of the PBMC 3K and 4K datasets."}
# For the first batch.
tab <- pairwiseRand(colLabels(pbmc3k), clusters.mnn[mnn.out$batch==1])
heat3k <- pheatmap(tab, cluster_row=FALSE, cluster_col=FALSE,
    col=rev(viridis::magma(100)), main="PBMC 3K probabilities", silent=TRUE)

# For the second batch.
tab <- pairwiseRand(colLabels(pbmc4k), clusters.mnn[mnn.out$batch==2])
heat4k <- pheatmap(tab, cluster_row=FALSE, cluster_col=FALSE,
    col=rev(viridis::magma(100)), main="PBMC 4K probabilities", silent=TRUE)

gridExtra::grid.arrange(heat3k[[4]], heat4k[[4]])
```

## MNN-specific diagnostics

For `fastMNN()`, one useful diagnostic is the proportion of variance within each batch that is lost during MNN correction.
Specifically, this refers to the within-batch variance that is removed during orthogonalization with respect to the average correction vector at each merge step. 
This is returned via the `lost.var` field in the metadata of `mnn.out`, which contains a matrix of the variance lost in each batch (column) at each merge step (row).

```{r}
metadata(mnn.out)$merge.info$lost.var
```

Large proportions of lost variance (>10%) suggest that correction is removing genuine biological heterogeneity.
This would occur due to violations of the assumption of orthogonality between the batch effect and the biological subspace [@haghverdi2018batch].
In this case, the proportion of lost variance is small, indicating that non-orthogonality is not a major concern.

Another MNN-related diagnostic involves examining the variance in the differences in expression between MNN pairs.
A small variance indicates that the correction had little effect - either there was no batch effect, or any batch effect was simply a constant shift across all cells.
On the other hand, a large variance indicates that the correction was highly non-linear, most likely involving subpopulation-specific batch effects.
This computation is achieved using the `mnnDeltaVariance()` function on the MNN pairings produced by `fastMNN()`.

```{r}
library(batchelor)
common <- rownames(mnn.out)
vars <- mnnDeltaVariance(pbmc3k[common,], pbmc4k[common,], 
   pairs=metadata(mnn.out)$merge.info$pairs)
vars[order(vars$adjusted, decreasing=TRUE),]
```

Such genes with large variances are particularly interesting as they exhibit complex differences between batches that may reflect real biology. 
For example, in Figure \@ref(fig:mnn-delta-var-pbmc), the _KLRB1_-positive clusters in the second batch lack any counterpart in the first batch, despite the two batches being replicates.
This may represent some kind of batch-specific state in two otherwise identical populations, though whether this is biological or technical in nature is open for interpretation.

```{r mnn-delta-var-pbmc, fig.wide=TRUE, fig.cap="Distribution of the expression of the gene with the largest variance of MNN pair differences in each batch of the the PBMC dataset."}
library(scater)
top <- rownames(vars)[order(vars$adjusted, decreasing=TRUE)[1]]
gridExtra::grid.arrange(
    plotExpression(pbmc3k, x="label", features=top) + ggtitle("3k"),
    plotExpression(pbmc4k, x="label", features=top) + ggtitle("4k"),
    ncol=2
)
```

```{r, echo=FALSE}
# Sanity check.
stopifnot(top == "ENSG00000111796")
prop.3k <- vapply(split(logcounts(pbmc3k)[top,]!=0, colLabels(pbmc3k)), mean, 0)
prop.4k <- vapply(split(logcounts(pbmc4k)[top,]!=0, colLabels(pbmc4k)), mean, 0)
stopifnot(!any(prop.3k > 0.3))
stopifnot(any(prop.4k > 0.5))
```

## Session Info {-}

```{r sessionInfo, echo=FALSE, results='asis'}
prettySessionInfo()
```
