---
title: "Introduction"
author: Constantin Ahlmann-Eltze
date: "`r format(Sys.time(), '%d %B, %Y')`"
format:
  html:
    embed-resources: true
    page-layout: full
    toc: true
highlight-style: tango
vignette: >
  %\VignetteIndexEntry{Introduction}
  %\VignetteEngine{quarto::html}}
  %\VignetteEncoding{UTF-8}
lightbox: true
---

<!-- 'Stangle' analogon: knitr::purl("Introduction.qmd")
-->

```{r}
#| label: setup_alt_IS_THIS_REALLY_NEEDED
#| echo: false
#| eval: false
if(! exists("..options_set") || isFALSE(..options_set)){
  knitr::opts_chunk$set(
    collapse = TRUE,
    comment = "#>",
    dpi = 40,
    fig_retina = 1,
    dev = "jpeg"
    # dev.args = list(quality = 20)
  )
  ..options_set <- TRUE
}
```
```{r initialize}
#| label: init
#| echo: false
#| cache: false
knitr::opts_chunk$set(cache = TRUE, autodep = TRUE)
options(width = 100)
```


# Latent Embedding Multivariate Regression (LEMUR)

<!-- badges: start -->

<!-- badges: end -->

![](man/figures/lemur-art.jpg){fig-align="center"}


The goal of [`lemur`](https://www.biorxiv.org/content/10.1101/2023.03.06.531268)
is to simplify the analysis of multi-condition
single-cell data. If you have collected a single-cell RNA-seq dataset
with more than one condition, `lemur` predicts for each cell and gene
what the expression data would be in all of the other
conditions. Furthermore, `lemur` finds neighborhoods of cells that show
consistent differential expression. The results are statistically
validated using a pseudo-bulk differential expression test on hold-out
data using
[glmGamPoi](https://bioconductor.org/packages/glmGamPoi/)
or [edgeR](https://bioconductor.org/packages/edgeR/).

`lemur` implements a novel framework to disentangle the effects of known covariates, latent cell states, and their interactions. At the core, is a combination of matrix factorization and regression analysis implemented as geodesic regression on Grassmann manifolds. We call this _latent embedding multivariate regression_. For more details see our [preprint](https://www.biorxiv.org/content/10.1101/2023.03.06.531268).

![Schematic of the matrix decomposition at the core of LEMUR](man/figures/equation_schematic.png)

## Installation

You can install `lemur` directly from Bioconductor. Just paste the following snippet into your R console:

```{r}
#| label: install1
#| eval: false
if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

BiocManager::install("lemur")
```

Alternatively, you can install the package from Github using `devtools`:

```{r}
#| label: install2
#| eval: false
devtools::install_github("const-ae/lemur")
```

## A note to users

We continue working on improvements to this package. 
We are delighted if you decide to try out the package.
Please use the [Bioconductor forum](https://support.bioconductor.org/) or 
open an issue in the [GitHub repository](https://github.com/const-ae/lemur/) if you think you found a bug, have an idea for a
cool feature, or have any questions about how LEMUR works. 

## Overview

A basic lemur workflow is as easy as the following.

```{r} 
#| label: preparation
#| echo: false
library("lemur")
set.seed(42)
data("glioblastoma_example_data")
sce <- glioblastoma_example_data[1:50, sample.int(5000, size = 500)]
```

```{r}
#| label: quick_start
#| message: false
#| warning: false
# ... sce is a SingleCellExperiment object with your data 
fit <- lemur(sce, design = ~ patient_id + condition, n_embedding = 15)
fit <- align_harmony(fit)   # This step is optional
fit <- test_de(fit, contrast = cond(condition = "ctrl") - cond(condition = "panobinostat"))
nei <- find_de_neighborhoods(fit, group_by = vars(patient_id, condition))
```

We will now go through these steps one by one.


# A worked through example

We demonstrate `lemur` using data by [Zhao et al. (2021)](https://doi.org/10.1186/s13073-021-00894-y). The data consist of tumor biopsies from five glioblastomas, each of which was treated with the drug panobinostat and with a control. Accordingly, we look at ten samples in a paired experimental design.

We start by loading required packages.

```{r}
#| label: load_packages
#| message: false
#| warning: false
library("tidyverse")
library("SingleCellExperiment")
library("lemur")
set.seed(42)
```

The `lemur` package ships with a reduced-size version of the glioblastoma data, which we use in the following.

```{r}
#| label: load_data
data("glioblastoma_example_data", package = "lemur")
glioblastoma_example_data
```

As is, the data separated by the known covariates `patient_id` and `condition`.

```{r}
#| label: fig-raw_umap
#| fig-cap: "UMAP of the example dataset `glioblastoma_example_data`, prior to any between-condition alignment."
#| fig-width: 6
#| fig-height: 4
#| out-width: "80%"
orig_umap <- uwot::umap(as.matrix(t(logcounts(glioblastoma_example_data))))

as_tibble(colData(glioblastoma_example_data)) |>
  mutate(umap = orig_umap) |>
  ggplot(aes(x = umap[,1], y = umap[,2])) +
    geom_point(aes(color = patient_id, shape = condition), size = 0.5) +
    labs(title = "UMAP of logcounts") + coord_fixed()
```

We fit the LEMUR model by calling the function `lemur`. We provide the experimental design using a formula. The elements of the formula can refer to columns of the `colData` of the `SingleCellExperiment` object. 

We also set the number of latent dimensions, `n_embedding`, which has a similar interpretation as the number of dimensions in PCA. 
The `test_fraction` argument sets the fraction of cells which are exclusively used to test for differential expression and not for inferring the LEMUR parameters. It balances the sensitivity to detect subtle patterns in the latent space against the power to detect differentially expressed genes. <!--FIXME Do we need to bother users with this parameter here? Just use a reasonable default, and later have a section on "variations", like in the [DESeq2 vignette](https://www.bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#variations-to-the-standard-workflow) where such things are mentioned?-->

```{r}
#| label: fit_lemur
#| message: false
fit <- lemur(glioblastoma_example_data, design = ~ patient_id + condition, 
             n_embedding = 15, test_fraction = 0.5)
fit
```

The `lemur` function returns an object of class `r class(fit)`, which extends the `SingleCellExperiment` class. It supports subsetting and all the usual accessor methods (e.g., `nrow`, `assay`, `colData`, `rowData`). In addition, `lemur` overloads the `$` operator to allow easy access to additional fields produced by the LEMUR model. For example, the low-dimensional embedding can be accessed using `fit$embedding`:

```{r}
#| label: fitembedding
fit$embedding |> str()
```

Optionally, we can further align corresponding cells using manually annotated cell types (`align_by_grouping`) or an automated alignment procedure (e.g., `align_harmony`). This ensures that corresponding cells are close to each other in the `fit$embedding`.

```{r}
#| label: align_lemur
fit <- align_harmony(fit)
```

<!--FIXME "Select cells that are considered close with 'harmony'" -- should it not be 'Selecting' to avoid confusion with imperative verb form? Is the message even needed? -->

@fig-lemur_umap shows a UMAP of `fit$embedding`. This is similar to working on the integrated PCA space in a traditional single-cell analysis.

```{r}
#| label: fig-lemur_umap
#| fig-cap: "UMAPs of `fit$embedding`. The points are shown separately for the two conditions, but reside in the same latent space."
#| fig-width: 7
#| fig-height: 3
#| out-width: "100%"
umap <- uwot::umap(t(fit$embedding))

as_tibble(fit$colData) |>
  mutate(umap = umap) |>
  ggplot(aes(x = umap[,1], y = umap[,2])) +
    geom_point(aes(color = patient_id), size = 0.5) +
    facet_wrap(vars(condition)) + coord_fixed()
```

Next, let's predict the effect of the panobinostat treatment for **each** cell and each gene -- even for the cells that were observed in the control condition. The `test_de` function takes a `lemur_fit` object and returns the object with a new slot (in `SummarizedExperiment` parlance: `assay`) called `DE`. This slot contains the predicted logarithmic fold changes between the two conditions specified in `contrast`. Note that `lemur` implements a special notation for contrasts. Instead of providing a contrast vector or design matrix column names, you provide for each *condition* the levels, and `lemur` automatically forms the contrast vector. This is intended to make the notation more readable.

```{r}
#| label: lemur_test_de
fit <- test_de(fit, contrast = cond(condition = "panobinostat") - cond(condition = "ctrl"))
```

We can pick any gene, say GAP43, which in our data is represented by its Ensembl gene ID ENSG00000172020, and show its differential expression pattern on the UMAP plot:

```{r}
#| label: fig-umap_de
#| fig-subcap: 
#|   - "Superimposed on the same UMAP plot as in @fig-lemur_umap"
#|   - "The histogram shows that the values are negative for most cells."
#| fig-cap: "Differential expression (log fold changes) of GAP43"
#| fig-width: 6
#| fig-height: 4
#| out-width: "100%"
#| layout: "[65, 35]"
df <- tibble(umap = umap) |>
  mutate(de = assay(fit, "DE")["ENSG00000172020", ])
 
ggplot(df, aes(x = umap[,1], y = umap[,2])) +
    geom_point(aes(color = de)) +
    scale_color_gradient2(low = "#FFD800", high= "#0056B9") + coord_fixed()

ggplot(df, aes(x = de)) + geom_histogram(bins = 100)
```

More systematically, we can now search through all the genes, and use their expression values (`assay(fit, "DE")`) to search for cell neighborhoods (sets of cells that are close together in latent space) that show consistent differential expression. The function `find_de_neighborhoods` validates the results of such a search with a pseudobulked diferential expression test. For that, it uses the test data (`fit$test_data`) that was put aside in the first call to `lemur()`. In addition, `find_de_neighborhoods` assesses if the difference between the conditions is significantly larger for the cells inside the neighborhood than the cells outside the neighborhood (see columns starting with `did`, short for difference-in-difference).

The `group_by` argument determines how the pseudobulk samples are formed. It specifies the columns in the `fit$colData` that are used to define a sample and is inspired by the `group_by` function in `dplyr`. Typically, you provide the covariates that were used for the experimental design plus the sample id (in this case `patient_id`).

```{r}
#| label: fig-de_neighborhoods
#| message: false 
neighborhoods <- find_de_neighborhoods(fit, group_by = vars(patient_id, condition))

as_tibble(neighborhoods) |>
  left_join(as_tibble(rowData(fit)[,1:2]), by = c("name" = "gene_id")) |>
  relocate(symbol, .before = "name") |>
  arrange(pval) |>
  head(5)
```
<!-- FIXME can the dplyr gymnastics be offloaded from the user? Why is it a data.frame if anyway you use tibbles all round? -->

To continue, we investigate one gene for which the neighborhood shows a significant differential expression pattern:  here we choose a _CXCL8_ (also known as interleukin 8), an important inflammation signalling molecule. We see that it is upregulated by panobinostat in a subset of cells (blue). We chose this gene because it (1) had a significant change between panobinostat and negative control condition (`adj_pval` column) and (2) showed much larger differential expression for the cells inside the neighborhood than for the cells outside (`did_lfc` column).

```{r}
#| label: fig-umap_de2
#| fig-cap: "Differential expression of CXCL8 superimposed on the same UMAP plot as in @fig-lemur_umap."
#| fig-width: 6
#| fig-height: 4
#| out-width: "80%"
sel_gene <- "ENSG00000169429" # is CXCL8

p <- tibble(umap = umap) |>
  mutate(de = assay(fit, "DE")[sel_gene,]) |>
  ggplot(aes(x = umap[,1], y = umap[,2])) +
    geom_point(aes(color = de)) +
    scale_color_gradient2(low = "#FFD800", high= "#0056B9") +
    coord_fixed()
p
```

Next, we are going to try something ambitious: in the LEMUR model, the cells in a neighborhood are separated from the rest of the cells by a $(k-1)$-dimensional hyperplane in the $k$-dimensional latent space ($k$ being the same as `n_embedding` from above, i.e., $k=$ `r fit$n_embedding`). We can try to approximate this separation as a line in the two-dimensional UMAP plot.

To this end, we create a helper dataframe and use the `geom_density2d` function from `ggplot2`. To avoid the cutting of the boundary to the extremes of the cell coordinates, add `lims` to the plot with an appropriately large limit.

```{r}
#| label: fig-umap_de3
#| fig-cap: "Same as @fig-umap_de2, with an attempt to draw a neighborhood boundary."
#| fig-width: 6
#| fig-height: 4
#| out-width: "80%"
neighborhood_coordinates <- neighborhoods |>
  dplyr::filter(name == sel_gene) |>
  unnest(c(neighborhood)) |>
  dplyr::rename(cell_id = neighborhood) |>
  left_join(tibble(cell_id = rownames(umap), umap), by = "cell_id") |>
  dplyr::select(name, cell_id, umap)

p + geom_density2d(data = neighborhood_coordinates, breaks = seq(0.2, 0.6, by = 0.1), 
                   contour_var = "ndensity", color = "#808080") 
```

To summarize our results, we can make a volcano plot of the differential expression results to better understand the expression differences across all genes.

```{r}
#| label: fig-volcano_plot
#| fig-cap: "Volcano plot, each point corresponds to one neighborhood."
#| fig-width: 6
#| fig-height: 4
#| out-width: "50%"
neighborhoods |>
  drop_na() |>
  ggplot(aes(x = lfc, y = -log10(pval))) +
    geom_point(aes(col  = adj_pval < 0.1)) 
```

<!--FIXME What is the point of the following plot? That there is some dependence on size (too small is not good) but not too much? I am not sure it's that interesting for first-time users, maybe relegate this to a supplement?-->

```{r}
#| label: fig-Neighborhood_size_vs_significance
#| fig-cap: "Neighborhood size vs neighborhood significance."
#| fig-width: 6
#| fig-height: 4
#| out-width: "50%"
neighborhoods |>
  drop_na() |>
  ggplot(aes(x = n_cells, y = -log10(pval))) +
    geom_point(aes(color  = adj_pval < 0.1)) 
```

## Using cell type annotation

The analyses up to here were conducted without using any cell type information. Often, such additional cell type information is available or can be obtained from the data by other means. For instance, here, we can distinguish the tumor cells from non-malignment other cell, using the fact that the tumor cells had a deletion of Chromosome 10 and a duplication of Chromosome 7. We build a simple classifier to distinguish the cells accordingly. (This is just to illustrate the process; for a real analysis, we would use more sophisticated methods.)

```{r}
#| label: fig-tumor_cell_annotation1
#| fig-cap: "A simple gating strategy to find tumor cells"
#| fig-width: 6
#| fig-height: 4
#| out-width: "60%"
tumor_label_df <- tibble(cell_id = colnames(fit),
       chr7_total_expr  = colMeans(logcounts(fit)[rowData(fit)$chromosome == "7",]),
       chr10_total_expr = colMeans(logcounts(fit)[rowData(fit)$chromosome == "10",])) |>
  mutate(is_tumor = chr7_total_expr > 0.8 & chr10_total_expr < 2.5)

ggplot(tumor_label_df, aes(x = chr10_total_expr, y = chr7_total_expr)) +
    geom_point(aes(color = is_tumor), size = 0.5) +
    geom_hline(yintercept = 0.8) +
    geom_vline(xintercept = 2.5) 
```
```{r}
#| label: fig-tumor_cell_annotation2
#| fig-cap: "The tumor cells are enriched in parts of the big (left) blob."
#| fig-width: 6
#| fig-height: 4
#| out-width: "60%"
tibble(umap = umap) |>
  mutate(is_tumor = tumor_label_df$is_tumor) |>
  ggplot(aes(x = umap[,1], y = umap[,2])) +
    geom_point(aes(color = is_tumor), size = 0.5) +
    facet_wrap(vars(is_tumor)) + coord_fixed()
```

We use this cell annotation to focus our neighborhood finding within the tumor cells, to find tumor subpopulations.

```{r}
#| label: tumor_de_neighborhood
#| paged.print: false
tumor_fit <- fit[, tumor_label_df$is_tumor]
tum_nei <- find_de_neighborhoods(tumor_fit, group_by = vars(patient_id, condition), verbose = FALSE)

as_tibble(tum_nei) |>
  left_join(as_tibble(rowData(fit)[,1:2]), by = c("name" = "gene_id")) |>
  dplyr::relocate(symbol, .before = "name") |>
  filter(adj_pval < 0.1) |>
  arrange(did_pval)  |>
  dplyr::select(symbol, name, neighborhood, n_cells, adj_pval, lfc, did_pval, did_lfc) |>
  print(n = 10)
```

Focusing on RPS11, we see that panobinostat mostly has no effect on its expression, except for a subpopulation of tumor cells where RPS11 was originally upregulated and panobinostat downregulates the expression. A small caveat: this analysis is conducted on a subset of all cells and should be interpreted carefully. Yet, this section demonstrates how `lemur` can be used to find tumor subpopulations which show differential responses to treatments.

```{r}
#| label: fig-tumor_de_neighborhood_plot
#| paged.print: false
#| fig-cap: ""
#| fig-width: 6
#| fig-height: 4
#| out-width: "80%"
sel_gene <- "ENSG00000142534" # is RPS11

as_tibble(colData(fit)) |>
  mutate(expr = assay(fit, "logcounts")[sel_gene,]) |>
  mutate(is_tumor = tumor_label_df$is_tumor) |>
  mutate(in_neighborhood = id %in% filter(tum_nei, name == sel_gene)$neighborhood[[1]]) |>
  ggplot(aes(x = condition, y = expr)) +
    geom_jitter(size = 0.3, stroke = 0) +
    geom_point(data = . %>% summarize(expr = mean(expr), .by = c(condition, patient_id, is_tumor, in_neighborhood)),
               aes(color = patient_id), size = 2) +
    stat_summary(fun.data = mean_se, geom = "crossbar", color = "red") +
    facet_wrap(vars(is_tumor, in_neighborhood), labeller = label_both) 

```


# FAQ

##### I have already integrated my data using Harmony / MNN / Seurat. Can I call `lemur` directly with the aligned data?

No. You need to call `lemur` with the unaligned data so that it can learn how much the expression of each gene changes between conditions.

##### Can I call lemur with [sctransformed](https://github.com/satijalab/sctransform) instead of log-transformed data?

Yes. You can call lemur with any variance stabilized count matrix. Based on a [previous project](https://www.biorxiv.org/content/10.1101/2021.06.24.449781v4), I recommend to use log-transformation, but other methods will work just fine.

##### My data appears less integrated after calling `lemur()` than before. What is happening?!

This is a known issue and can be caused if the data has large compositional shifts (for example, if one cell type disappears). The problem is that the initial linear regression step, which centers the conditions relative to each other, overcorrects and introduces a consistent shift in the latent space. You can either use `align_by_grouping` / `align_harmony` to correct for this effect or manually fix the regression coefficient to zero:

```{r}
#| label: fix_linear_coef
#| message: false
#| warning: false
fit <- lemur(sce, design = ~ patient_id + condition, n_embedding = 15, linear_coefficient_estimator = "zero")
```

##### The conditions still separate if I plot the data using UMAP / tSNE. Even after calling `align_harmony` / `align_neighbors`. What should I do?

You can try to increase `n_embedding`. If this still does not help, there is little use in inferring differential expression neighborhoods. But as I haven't encountered such a dataset yet, I would like to try it out myself. If you can share the data publicly, please open an issue.

##### How do I make `lemur` faster?

Several parameters influence the duration to fit the LEMUR model and find differentially expressed neighborhoods:

-   Make sure that your data is stored in memory (not a `DelayedArray`) either as a sparse dgCMatrix or dense matrix.
-   A larger `test_fraction` means fewer cells are used to fit the model (and more cells are used for the DE test), which speeds up many steps.
-   A smaller `n_embedding` reduces the latent dimensions of the fit, which makes the model less flexible, but speeds up the `lemur()` call.
-   Providing a pre-calculated set of matching cells and calling `align_grouping` is faster than `align_harmony`.
-   Setting `selection_procedure = "contrast"` in `find_de_neighborhoods` often produces better neighborhoods, but is a lot slower than `selection_procedure = "zscore"`.
-   Setting `size_factor_method = "ratio"` in `find_de_neighborhoods` makes the DE more powerful, but is a lot slower than `size_factor_method = "normed_sum"`.

# Session Info

```{r}
#| label: session_info
sessionInfo()
```