---
title: "3 Introduction of BenchmarkInsights class"
output: BiocStyle::html_document
vignette: >
  %\VignetteIndexEntry{3 Introduction of BenchmarkInsights class}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  markdown: 
    wrap: 72
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE)
```

```{r load-package}
library(BenchHub)
library(readr)
library(dplyr)
library(stringr)
```

# Motivation

BenchHub—an R ecosystem to make benchmarking easier. It organizes
evaluation metrics, gold standards (Supporting Evidence), and even
provides built-in visualization tools to help interpret results. With
BenchHub, researchers can quickly compare new methods, gain insights,
and actually trust their benchmarking studies. In this vignette, we are
going to introduce BenchmarkInsights class.

In this vignette, we use a subset of results generated under the
SpatialSimBench framework as a case study. SpatialSimBench benchmarks
spatial transcriptomics simulation methods across ten spatially resolved
transcriptomics datasets using multiple evaluation metrics. The subset
considered here includes simulations from scDesign2, scDesign3 (across
three distributions), SPARsim, splatter, SRTsim, symsim, and ZINB-WaVE,
evaluated across tasks covering data properties, spatial downstream
analyses, and scalability. These results are used to illustrate
downstream analysis and interpretation.

# Creating BenchmarkInsights class

`BenchmarkInsights` objects can be created using the corresponding
constructor. For example, if you have a benchmark result formatted in
dataframe, you can create a `BenchmarkInsights` object as follows. The
dataframe includes fixed name columns: `datasetID`, `method`,
`evidence`, `metric`, and `result`. Here I will use the benchmark result
from SpatialSimBench to create a new object.

```{r load-result}
result_path <- system.file("extdata", "spatialsimbench_result.csv", package = "BenchHub")
spatialsimbench_result <- read_csv(result_path)
glimpse(spatialsimbench_result)
```

If you use `trio$evaluation()`, the output will be automatically
formatted as the required dataframe. However, if you use your own
benchmark evaluation results, you ensure they adhere to the expected
format.

```{r create-bmi}
bmi <- BenchmarkInsights$new(spatialsimbench_result)
bmi
```

If you have additional evaluation result, you can use
`addevalSummary()`. Here is the example:

```{r addeval}
add_result <- data.frame(
  datasetID = rep("BREAST", 9),
  method = c(
    "scDesign2", "scDesign3_gau", "scDesign3_nb", "scDesign3_poi",
    "SPARsim", "splatter", "SRTsim", "symsim", "zinbwave"
  ),
  evidence = rep("svg", 9),
  metric = rep("recall", 9),
  result = c(
    0.921940928, 0.957805907, 0.964135021, 0.989451477, 0.774261603,
    0.890295359, 0.985232068, 0.067510549, 0.888185654
  ),
  stringsAsFactors = FALSE
)

bmi$addevalSummary(add_result)
```

If you add additional metadata of method, you can use `addMetadata()`.
Here is the example:

```{r addmeta}
metadata_srtsim <- data.frame(
  method = "SRTsim",
  year = 2023,
  packageVersion = "0.99.6",
  parameterSetting = "default",
  spatialInfoReq = "No",
  DOI = "10.1186/s13059-023-02879-z",
  stringsAsFactors = FALSE
)

bmi$addMetadata(metadata_srtsim)
```

# Visualization

## Available plot

-   `getHeatmap()`: Creates a heatmap from the stored evaluation
    summary by averaging results across datasets. You can also provide
    a custom `evalResult` dataframe if needed.

-   `getCorplot(input_type)`: Creates a correlation plot based on the
    stored evaluation summary. You can also provide a custom
    `evalResult` dataframe if needed.

-   `getBoxplot(metricVariable, evidenceVariable)`: Creates a boxplot
    based on the stored evaluation summary. You can also provide a
    custom `evalResult` dataframe if needed.

-   `getForestplot(input_group, input_model)`: Create a forest plot
    using linear models based on the comparison between groups in the
    stored evaluation summary. You can also provide a custom
    `evalResult` dataframe if needed.

-   `getScatterplot(variables)`: a scatter plot for the same evidence,
    with two method metrics, using the stored evaluation summary by
    default.

-   `getLineplot(metricVariable, order)`: Creates a line plot for the
    given x and y variables, with an optional grouping and fixed x
    order, using the stored evaluation summary by default.

## Interpretation benchmark result

### Case Study: What is the overview of summary?

To get a high-level view of method performance, we use a heatmap to
summarize evaluation results across datasets. This helps identify
overall trends, making it easier to compare methods and performance
differences.

```{r heatmap, fig.width = 10, fig.height = 8, fig.cap = "Fig.1 Heatmap of benchmarking performance across simulation methods and evaluation metrics. Each row corresponds to a simulation method and each column to an evaluation criterion. The size and colour of the circles represent the normalised metric values (scaled to [0, 1])."}
bmi$getHeatmap()
```

This funkyheatmap summarises the comparative performance of spatial
transcriptomics simulation methods across a diverse set of evaluation
tasks. Methods such as SRTsim and scDesign3 (NB/Poisson variants) show
relatively stable performance across multiple criteria, whereas
scDesign3_gau and Symsim exhibit consistently lower rankings across many
tasks. ZINB-WaVE performs less well on several spatial and downstream
evaluations, while most other methods demonstrate moderate performance
in at least some categories. Importantly, this case study highlights
that multiple viable choices exist, and method selection should be
guided by the intended downstream application and evaluation priorities.

### Case Study: What is the correlation between evidence/metric/method?

To understand the relationships between different evaluation factors, we
use a correlation plot to examine how evidence, metrics, and methods are
interrelated. This helps identify patterns, redundancies, or
dependencies among evaluation components.

```{r corplot, fig.cap = "Fig.2 Correlation plot illustrating relationships among evaluation metrics across different methods. Correlation coefficients are computed based on benchmarking results, with colour intensity indicating the strength and direction of association."}
bmi$getCorplot(input_type = "method")
```

Splatter and SRTsim show moderate similarity, which is consistent with
their shared generative framework: both methods estimate global
distributional properties from real data and simulate counts using
related hierarchical models. The scDesign3 negative binomial and Poisson
variants exhibit strong correlations with each other, indicating that
these distributions are better able to capture key data features,
whereas the Gaussian variant shows much weaker correspondence. In
contrast, symsim displays relatively low correlations with many other
methods, suggesting that it captures different characteristics of the
underlying datasets and follows a distinct simulation strategy.

To further investigate the relationship between two specific metrics, we
use a scatter plot. This visualization helps assess how well two metrics
align or diverge across different methods, providing insights into
trade-offs and performance consistency.

```{r scatterplot, fig.cap = "Fig.3 Scatter plot illustrating the relationship between recall and precision across methods. Each point represents a method, with positions reflecting its performance on the two metrics."}
bmi$getScatterplot(variables = c("recall", "precision"))
```

scDesign3 variants and SRTsim are positioned toward the top right of the
plot, indicating strong and well-balanced performance in terms of both
precision and recall. In contrast, splatter and symsim exhibit
relatively lower values for both metrics, suggesting weaker performance
for this precision–recall combination.

### Case Study: What is the time and memory trend?

To evaluate the scalability of different methods, we use a line plot to
visualize trends in computational time and memory usage across different
conditions. This helps identify how methods perform as data complexity
increases, revealing potential efficiency trade-offs.

```{r lineplot, fig.cap = "Fig.4 Line plot illustrating memory usage across increasing data complexity for different methods. Each line represents a method, showing how memory consumption scales under varying conditions."}
bmi$getLineplot(metricVariable = "memory")
```

Most methods exhibit only modest increases in memory usage as dataset
size grows and remain relatively efficient overall. In contrast,
zinbwave shows the highest and steepest increase in memory consumption,
indicating substantially greater memory requirements for large datasets.
Methods such as splatter, SRTsim, and SPARsim maintain consistently low
memory usage across dataset sizes, suggesting strong scalability and
suitability for large-scale applications.

### Case Study: Which metric is most effective on the method?

To assess which metrics have the strongest influence on method
performance, we use a forest plot to visualize the relationship between
metrics and methods. This allows us to quantify and compare the impact
of different metrics, helping to identify the most critical evaluation
factors.

```{r forestplot, fig.cap = "Fig.5 Forest plot of regression coefficients estimating the influence of evaluation metrics on method performance. The vertical line at zero indicates no difference relative to the reference method; larger absolute coefficients indicate stronger metric-specific discrimination between methods."}
bmi$getForestplot(input_group = "metric", input_model = "method")
```

In the forest plot, the x-axis represents regression coefficients
obtained from linear models fitted to each evaluation metric, with the
vertical line at zero indicating no difference relative to the reference
method. There is no universally “good” or “bad” coefficient value, as
both the direction and magnitude of effects depend on the definition and
interpretation of each evaluation metric.

This forest plot summarises method-specific effects across different
evaluation metrics using regression coefficients. Metrics such as
KDEstat exhibit large coefficient magnitudes across methods, indicating
strong discriminative power between simulation approaches. In contrast,
metrics including cosine similarity, Mantel statistic, precision, and
recall show smaller coefficient differences, suggesting more
conservative behaviour across methods. Method families such as scDesign3
variants tend to behave similarly across many metrics, although
differences emerge for specific metrics, highlighting that
distributional assumptions continue to influence performance.

### Case Study: How does method variability differ across datasets for a specific metric?

To examine the consistency of each method across different datasets for
a given metric, we use a boxplot. This visualization helps assess the
variability of method performance, highlighting robustness or
instability when applied to different datasets.

```{r boxplot, fig.cap = "Fig.6 Boxplot showing the distribution of KDEstat values across simulation methods under the selected evidence setting. Differences in spread reflect variability and stability of method performance across datasets."}
bmi$getBoxplot(metricVariable = "KDEstat", evidenceVariable = "scaledVar")
```

Regarding the interpretation of values, there is no single universally
“good” value for this boxplot, as the scale and direction depend on the
definition of the metric. Instead, the plot is intended to compare the
distribution, variability, and stability of methods under the same
metric and evidence setting.

This boxplot compares the distribution of KDEstat values across
simulation methods under the selected evidence setting. scDesign2 and
SRTsim show relatively small interquartile ranges, indicating more
stable behaviour across datasets. In contrast, splatter and symsim
exhibit larger variability, suggesting greater sensitivity to
dataset-specific characteristics. ZINB-WaVE tends to produce lower and
more concentrated values, reflecting more consistent but comparatively
conservative behaviour under this metric. Rather than identifying a
single “good” value, the plot highlights differences in variability and
robustness across methods.

## Cheatsheet

|              Question              | Code                                                  |
|:------------------------:|:---------------------------------------------|
|          Summary Overview          | `getHeatmap()`                                        |
|        Correlation Analysis        | `getCorplot(input_type)`                              |
|  Scalability Trend (Time/ Memory)  | `getLineplot(metricVariable, order)`                  |
|   Metric-Model Impact (Modeling)   | `getForestplot(input_group, input_model)`             |
| Method Variability Across Datasets | `getBoxplot(metricVariable, evidenceVariable)`        |
|        Metric Relationship         | `getScatterplot(variables)`                           |

# Session Info

```{r session-info}
sessionInfo()
```