# Correction for multiple testing

```{r, results="asis", echo=FALSE}
library(rebook)
chapterPreamble()
```

## Overview

The false discovery rate (FDR) is usually the most appropriate measure of error for high-throughput experiments. 
Control of the FDR can be provided by applying the Benjamini-Hochberg (BH) method [@benjamini1995] to a set of $p$-values. 
This is less conservative than the alternatives (e.g., Bonferroni) yet still provides some measure of error control. 
The most obvious approach is to apply the BH method to the set of $p$-values across all windows. 
This will control the FDR across the set of putative DB windows.

However, the FDR across all detected windows is not necessarily the most relevant error rate. 
Interpretation of ChIP-seq experiments is more concerned with regions of the genome in which (differential) protein binding is found, rather than the individual windows.
In other words, the FDR across all detected DB regions is usually desired. 
This is not equivalent to that across all DB windows as each region will often consist of multiple overlapping windows.
Control of one will not guarantee control of the other [@lun2014].

To illustrate this difference, consider an analysis where the FDR across all window positions is controlled at 10\%. 
In the results, there are 18 adjacent window positions in one region and 2 windows in a separate region.
The first set of windows is a truly DB region whereas the second set is a false positive. 
A window-based interpretation of the FDR is correct as only 2 of the 20 window positions are false positives.
However, a region-based interpretation results in an actual FDR of 50%.

<!--
The BH method is particularly popular as it is simple to apply and robust to
correlations [@reiner2003,kim2008].  Simes' is also pretty robust to
correlations [@samuel1996,sarkar1997], in the same respect as the FDR.
Say you control the FDR within a cluster using the BH method, so
E(FDR)<=0.05. Now, the probability of all false positives (i.e. FDR=1) must
be under 0.05 as well. So, if the BH method works, so does Simes' method.
-->

To avoid misinterpretation of the FDR, `r Biocpkg("csaw")` provides a number of strategies to obtain region-level results.
This involves defining the regions of interest - possibly from the windows themselves - 
and converting per-window statistics into a $p$-value for each region.
Application of the BH method to the per-region $p$-values will then control the relevant FDR across regions.
These strategies are demonstrated below using the NF-YA data.

## Grouping windows into regions

### Quick and dirty clustering {#sec:cluster}

The `mergeWindows()` function provides a simple single-linkage algorithm to cluster windows into regions.
Windows that are less than `tol` apart are considered to be adjacent and are grouped into the same cluster.
The chosen `tol` represents the minimum distance at which two binding events are treated as separate sites.
Large values (500 - 1000 bp) reduce redundancy and favor a region-based interpretation of the results, 
while smaller values (< 200 bp) allow resolution of individual binding sites.

```{r, echo=FALSE, results="asis"}
extractCached("nfya-stock.Rmd", "modelling", c("filtered.data", "param"))
```

```{r}
library(csaw)
merged <- mergeWindows(filtered.data, tol=1000L)
merged$regions
```

If many adjacent windows are present, very large clusters may be formed that are difficult to interpret. 
We perform a simple check below to determine whether most clusters are of an acceptable size. 
Huge clusters indicate that more aggressive filtering from Chapter \@ref(chap-filter) is required.  
This mitigates chaining effects by reducing the density of windows in the genome.

<!--
Note that several large clusters may still be present due to high coverage within long tandem repeat loci.  
In general, chaining isn't as bad as single-linkage on the reads themselves, 
because windows that survive weak filtering should have reasonably high read counts. 
-->

```{r}
summary(width(merged$regions))
```

Alternatively, chaining can be limited by setting `max.width` to restrict the size of the merged intervals. 
Clusters substantially larger than `max.width` are split into several smaller subclusters of roughly equal size.
The chosen value should be small enough so as to separate DB regions from unchanged neighbors, 
yet large enough to avoid misinterpretation of the FDR.
Any value from 2000 to 10000 bp is recommended. 
This paramater can also interpreted as the maximum distance at which two binding sites are considered part of the same event.

```{r}
merged.max <- mergeWindows(filtered.data, tol=1000L, max.width=5000L)
summary(width(merged.max$regions))
```

### Using external information

Another approach is to group together windows that overlap with a pre-specified region of interest.
The most obvious source of pre-specified regions is that of annotated features such as promoters or gene bodies.
Alternatively, called peaks can be used provided that sufficient care has been taken to avoid loss of error control from data snooping [@lun2014].
Regardless of how they are specified, each region of interest corresponds to a group that contains all overlapping windows,  
as identified by the `findOverlaps` function from the `r Biocpkg("GenomicRanges")` package. 

```{r}
library(TxDb.Mmusculus.UCSC.mm10.knownGene)
broads <- genes(TxDb.Mmusculus.UCSC.mm10.knownGene)
broads <- resize(broads, width(broads)+3000, fix="end")

olap <- findOverlaps(broads, rowRanges(filtered.data))
olap
```

At this point, one might imagine that it would be simpler to just collect and analyze counts over the pre-specified regions. 
This is a valid strategy but will yield different results. 
Consider a promoter containing two separate sites that are identically DB in opposite directions. 
Counting reads across the promoter will give equal counts for each condition so changes within the promoter will not be detected. 
Similarly, imprecise peak boundaries can lead to loss of detection power due to "contamination" by reads in background regions.
Window-based methods may be more robust as each interval of the promoter/peak region is examined separately [@lun2014], 
avoiding potential problems with peak-calling errors and incorrect/incomplete annotation.

## Obtaining per-region $p$-value

### Combining window-level $p$-values

We compute a combined $p$-value for each region based on the $p$-values of the constituent windows [@simes1986].
This tests the joint null hypothesis for each region, i.e., that no enrichment is observed across any of its windows.
Any DB within the region will reject the joint null and yield a low $p$-value for the entire region.
The combined $p$-values are then adjusted using the BH method to control the region-level FDR.

```{r}
tabcom <- combineTests(merged$ids, rowData(filtered.data))
is.sig.region <- tabcom$FDR <= 0.05
summary(is.sig.region)
```

Summarizing the direction of DB for each cluster requires some care as the direction of DB can differ between constituent windows.
The `num.up.tests` and `num.down.tests` fields contain the number of windows that change in each direction, 
and can be used to gauge whether binding increases or decreases across the cluster.
A complex DB event may be present if both `num.up.tests` and `num.down.tests` are non-zero 
(i.e., opposing changes within the region) or if the total number of windows is much larger than either number 
(e.g., interval of constant binding adjacent to the DB interval).

Alternatively, the `direction` field specifies which DB direction contributes to the combined $p$-value.
If `"up"`, the combined $p$-value for this cluster is driven by $p$-values of windows with positive log-fold changes.
If `"down"`, the combined $p$-value is driven by windows with negative log-fold changes.
If `"mixed"`, windows with both positive and negative log-fold changes are involved.
This allows the dominant DB in significant clusters to be quickly summarized, as shown below.

```{r}
table(tabcom$direction[is.sig.region])
```

For pre-specified regions, the `combineOverlaps()` function will combine the $p$-values for all windows in each region. 
This is a wrapper around `combineTests()` for `Hits` objects.
It returns a single combined $p$-value (and its BH-adjusted value) for each region. 
Regions that do not overlap any windows have values of `NA` in all fields for the corresponding rows.

```{r}
tabbroad <- combineOverlaps(olap, rowData(filtered.data))
head(tabbroad[!is.na(tabbroad$PValue),])
is.sig.gene <- tabcom$FDR <= 0.05
table(tabbroad$direction[is.sig.gene])
```

### Based on the most significant window {#sec:mostsig}

Another approach is to use the single window with the strongest DB as a representative of the entire region.
This is useful when a log-fold change is required for each cluster, e.g., for plotting.
(In contrast, taking the average log-fold change across all windows in a region will understate the magnitude of DB,
especially if the region includes some non-DB background intervals of the genome.)
Identification of the most significant (i.e., "best") window is performed using the `getBestTest()` function.
This reports the index of the window with the lowest $p$-value in each cluster as well as the associated statistics.

```{r}
tab.best <- getBestTest(merged$ids, rowData(filtered.data))
head(tab.best)
```

A Bonferroni correction is applied to the $p$-value of the best window in each region, 
based on the number of constituent windows in that region.
This is necessary to account for the implicit multiple testing across all windows in each region.
The corrected $p$-value is reported as `PValue` in `tab.best`,
and can be used for correction across regions using the BH method to control the region-level FDR.

In addition, it is often useful to report the start location of the best window within each cluster.
This allows users to easily identify a relevant DB subinterval in large regions.
For example, the sequence of the DB subinterval can be extracted for motif discovery.

```{r}
tabcom$rep.start <- start(rowRanges(filtered.data))[tab.best$rep.test]
head(tabcom[,c("rep.logFC", "rep.start")])
```

The same approach can be applied to the overlaps between windows and pre-specified regions, 
using the `getBestOverlaps()` wrapper function.
This is demonstrated below for the broad gene body example.
As with `combineOverlaps()`, regions with no windows are assigned `NA` in the output table, 
but these are removed here to show some actual results.

```{r}
tab.best.broad <- getBestOverlaps(olap, rowData(filtered.data))
tabbroad$rep.start <- start(rowRanges(filtered.data))[tab.best.broad$rep.test]
head(tabbroad[!is.na(tabbroad$PValue),c("rep.logFC", "rep.start")])
```

### Wrapper functions

For convenience, the steps of merging windows and computing statistics are implemented in a single wrapper function.
This simply calls `mergeWindows()` followed by `combineTests()` and `getBestTest()`.

```{r}
merge.res <- mergeResults(filtered.data, rowData(filtered.data), tol=100,
    merge.args=list(max.width=5000))
names(merge.res)
```

An equivalent wrapper function is also available for handling overlaps to pre-specified regions.
This simply calls `findOverlaps()` followed by `combineOverlaps()` and `getBestOverlaps()`.

```{r}
broad.res <- overlapResults(filtered.data, regions=broads,
    tab=rowData(filtered.data))
names(broad.res)
```

## Squeezing out more detection power

### Integrating across multiple window sizes {#sec:bin-integrate}

Repeating the analysis with different window sizes may uncover new DB events at different resolutions.
Multiple sets of DB results are integrated by clustering adjacent windows together (even if they differ in size) and combining $p$-values within each of the resulting clusters.
The example below uses the H3 acetylation data from Chapter \@ref(chap-norm).
Some filtering is performed to avoid excessive chaining in this demonstration.
Corresponding tables of DB results should also be obtained -- for brevity, mock results are used here.

```{r}
library(chipseqDBData)
ac.files <- H3K9acData()$Path
ac.small <- windowCounts(ac.files, width=150L, spacing=100L, 
    filter=25, param=param)
ac.large <- windowCounts(ac.files, width=1000L, spacing=500L, 
    filter=35, param=param)

# TODO: actually do the analysis here.
# In the meantime, mocking up results for demonstration purposes.
ns <- nrow(ac.small)
mock.small <- data.frame(logFC=rnorm(ns), logCPM=0, PValue=runif(ns)) 
nl <- nrow(ac.large)
mock.large <- data.frame(logFC=rnorm(nl), logCPM=0, PValue=runif(nl)) 
```

The `mergeResultsList()` function merges windows of all sizes into a single set of regions, 
and computes a combined $p$-value from the associated $p$-values for each region.
Equal contributions from each window size are enforced by setting `equiweight=TRUE`, 
which uses a weighted version of Simes' method [@benjamini1997].
The weight assigned to each window is inversely proportional to the number of windows of that size in the same cluster.
This avoids the situation where, if a cluster contains many small windows, 
the DB results for the analysis with the small window size contribute most to the combined $p$-value.
This is not ideal when results from all window sizes are of equal interest.

```{r}
cons.res <- mergeResultsList(list(ac.small, ac.large), 
    tab.list=list(mock.small, mock.large), 
    equiweight=TRUE, tol=1000)
cons.res$regions
cons.res$combined
```

Similarly, the `overlapResultsList()` function is used to merge windows of varying size that overlap pre-specified regions.

```{r}
cons.broad <- overlapResultsList(list(ac.small, ac.large),
    tab.list=list(mock.small, mock.large), 
    equiweight=TRUE, region=broads)
cons.broad$regions
cons.res$combined
```

In this manner, DB results from multiple window widths can be gathered together and reported as a single set of regions.
Consolidation is most useful for histone marks and other analyses involving diffuse regions of enrichment.
For such studies, the ideal window size is not known or may not even exist, 
e.g., if the widths of the enriched regions or DB subintervals are variable.

### Weighting windows on abundance

Windows that are more likely to be DB can be upweighted to improve detection power.
For example, in TF ChIP-seq data, the window of highest abundance within each enriched region probably contains the binding site.
It is reasonable to assume that this window will also have the strongest DB.
To improve power, the weight assigned to the most abundant window is increased relative to that of other windows in the same cluster.
This means that the $p$-value of this window will have a greater influence on the final combined $p$-value.

Weights are computed in a manner to minimize conservativeness relative to the optimal unweighted approaches in each possible scenario. 
If the strongest DB event is at the most abundant window, the weighted approach will yield a combined $p$-value that is no larger than twice the $p$-value of the most abundant window.
(Here, the optimal approach would be to use the $p$-value of the most abundance window directly as a proxy for the $p$-value of the cluster.)
If the strongest DB event is _not_ at the most abundant window, the weighted approach will yield a combined $p$-value that is no larger than twice the combined $p$-value without wweighting (which is optimal as all windows have equal probabilities of containing the strongest DB).
All windows have non-zero weights, which ensures that any DB events in the other windows will still be considered when the $p$-values are combined.

The application of this weighting scheme is demonstrated in the example below.
First, the `getBestTest} function with \Rcode{by.pval=FALSE()` is used to identify the most abundant window in each cluster.
Window-specific weights are then computed using the `upweightSummits} function, and supplied to \Rcode{combineTests()` to use in computing combined $p$-values.

```{r}
tab.ave <- getBestTest(merged$id, rowData(filtered.data), by.pval=FALSE)
weights <- upweightSummit(merged$id, tab.ave$rep.test)
head(weights)
tabcom.w <- combineTests(merged$id, rowData(filtered.data), weight=weights)
head(tabcom.w)
```

The weighting approach can also be applied to the clusters from the broad gene body example.
This is done by replacing the call to `getBestTest} with one to \Rfunction{getBestOverlaps()`, as before.
Similarly, `upweightSummit} can be replaced with \Rfunction{summitOverlaps()`.
These wrappers are designed to minimize book-keeping problems when one window overlaps multiple regions.

```{r}
broad.best <- getBestOverlaps(olap, rowData(filtered.data), by.pval=FALSE)
head(broad.best[!is.na(broad.best$PValue),])
broad.weights <- summitOverlaps(olap, region.best=broad.best$rep.test)
tabbroad.w <- combineOverlaps(olap, rowData(filtered.data), o.weight=broad.weights) 
```

### Filtering after testing but before correction

Most of the filters in Chapter~\@ref(chap-filter) are applied before the statistical analysis.
However, some of the approaches may be too aggressive, e.g., filtering to retain only local maxima or based on pre-defined regions.
In such cases, it may be preferable to initially apply one of the other, milder filters.
This ensures that sufficient windows are retained for stable normalization and/or EB shrinkage.
The aggressive filters can then be applied after the window-level statistics have been calculated, but before clustering into regions and calculation of cluster-level statistics.
This is still beneficial as it removes irrelevant windows that would increase the severity of the BH correction.
It may also reduce chaining effects during clustering.

## FDR control in difficult situations

### Clustering only on DB windows for diffuse marks

The clustering procedures described above rely on independent filtering to remove irrelevant windows.
This ensures that the regions of interest are reasonably narrow and can be easily interpreted, 
    which is typically the case for most protein targets, e.g., TFs, narrow histone marks.
However, enriched regions may be very large for more diffuse marks.
Such regions may be difficult to interpret when only the DB subinterval is of interest.
To overcome this, a post-hoc analysis can be performed whereby only significant windows are used for clustering.

```{r}
postclust <- clusterWindows(rowRanges(filtered.data), rowData(filtered.data),
                            target=0.05, tol=100, max.width=1000)
postclust$FDR
postclust$region
```

This will define and cluster significant windows in a manner that controls the cluster-level FDR at 5%.
The clustering step itself is performed using `mergeWindows()` with the specified parameters.
Each cluster consists entirely of DB windows and can be directly interpreted as a DB region or a DB subinterval of a larger enriched region.
This reduces the pressure on abundance filtering to obtain well-separated regions prior to clustering, e.g., for diffuse marks or in data sets with weak IP signal.
That said, users should be aware that calculation of the cluster-level FDR is not entirely rigorous.
As such, independent clustering and FDR control via Simes' method should be considered as the default for routine analyses.

### Using the empirical FDR for noisy data

Some analyses involve comparisons of ChIP samples to negative controls.
In such cases, any region exhibiting enrichment in the negative control over the ChIP samples must be a false positive.
The number of significant regions that change in the "wrong" direction can be used as an estimate of the number of false positives at any given $p$-value threshold.
Division by the number of discoveries changing in the "right" direction yields an estimate of the FDR, i.e., the empirical FDR [@zhang2008].
This strategy is implemented in the `empiricalFDR()` function, which controls the empirical FDR across clusters based on their combined $p$-values.
Its use is demonstrated below, though the output is not meaningful in this situation as genuine changes in binding can be present in both directions.

```{r}
empres <- empiricalFDR(merged$id, rowData(filtered.data))
```

The empirical FDR is useful for analyses of noisy data with high levels of non-specific binding.
This is because the estimate of the number of false positives adapts to the observed number of regions exhibiting enrichment in the negative controls.
In contrast, the standard BH method in `combineTests()` relies on proper type I error control during hypothesis testing.
As non-specific binding events tend to be condition-specific, they are indistinguishable from DB events and assigned low $p$-values, resulting in loss of FDR control.
Thus, for noisy data, use of the empirical FDR may be more appropriate to control the proportion of "experimental" false positives.
However, calculation of the empirical FDR is not as statistically rigorous as that of the BH method, so users are advised to only apply it when necessary.

### Detecting complex DB

Complex DB events involve changes to the shape of the binding profile, not just a scaling increase/decrease to binding intensity.
Such regions may contain multiple sites that change in binding strength in opposite directions, 
or peaks that change in width or position between conditions.
This often manifests as DB in opposite directions in different subintervals of a region.
Some of these events can be identified using the `mixedTests()` function.

```{r}
tab.mixed <- mixedTests(merged$ids, rowData(filtered.data))
tab.mixed
```

`mixedTests()` converts the $p$-value for each window into two one-sided $p$-values.
The one-sided $p$-values in each direction are combined using Simes' method,
and the two one-sided combined $p$-values are themselves combined using an intersection-union test [@berger1996bioequivalence].
The resulting $p$-value is only low if a region contains strong DB in both directions.

`combineTests()` also computes some statistics for informal detection of complex DB.
For example, the `num.up.tests` and `num.down.tests` fields can be used to identify regions with changes in both directions.
The `direction` field will also label some regions as `"mixed"`, though this is not comprehensive.
Indeed, regions labelled as `"up"` or `"down"` in the `direction` field may also correspond to complex DB events,
but will not be labelled as `"mixed"` if the significance calculations are dominated by windows changing in only one direction.

### Enforcing a minimal number of DB windows

On occasion, we may be interested in genomic regions that contain at least a minimal number or proportion of DB windows.
This is motivated by the desire to avoid detecting DB regions where only a small subinterval exhibits a change,
instead favoring more systematic changes throughout the region that are easier to interpret.
We can identify these regions using the `minimalTests()` function.

```{r}
tab.min <- minimalTests(merged$ids, rowData(filtered.data),
    min.sig.n=3, min.sig.prop=0.5)
tab.min
```

`minimalTests()` applies a Holm-Bonferroni correction to all windows in the same cluster and picks the $x$^th^-smallest adjusted $p$-value (where $x$ is defined from `min.sig.n` and `min.sig.prop`).
This tests the joint null hypothesis that the per-window null hypothesis is false for fewer than $x$ windows in the cluster.
If the $x$^th^-smallest $p$-value is low, this provides strong evidence against the joint null for that cluster.

As an aside, this function also has some utility outside of ChIP-seq contexts.
For example, we might want to obtain a single $p$-value for a gene set based on the presence of a minimal percentage of differentially expressed genes.
Alternatively, we may be interested in ranking genes in loss-of-function screens based on a minimal number of shRNA/CRISPR guides that exhibit a significant effect.
These problems are equivalent to that of identifying a genomic region with a minimal number of DB windows.

## Session information {-}

```{r, echo=FALSE, results='asis'}
rebook::prettySessionInfo()
```
