# Permutation-Based Testing

By default, CaDrA performs both forward and backward search algorithm to look for a subset of features whose union is maximally associated with an outcome of interest, based on (currently) one of four scoring functions (Kolmogorov-Smirnov, Conditional Mutual Information, Wilcoxon, and custom-defined). To test whether the strength of the association between the set of features and the observed input scores (e.g., pathway activity, drug sensitivity, etc.) is greater than it would be expected by chance, CaDrA supports permutation-based significance testing. Importantly, the permutation test iterates over the entire search procedure (e.g., if `top_N = 7`, each permutation iteration will consist of running the search over the top 7 features).

# Load packages

    library(CaDrA)

# Load required datasets

1. A `binary features matrix` also known as `Feature Set` (such as somatic mutations, copy number alterations, chromosomal translocations, etc.) The 1/0 row vectors indicate the presence/absence of ‘omics’ features in the samples. The `Feature Set` can be a matrix or an object of class SummarizedExperiment from SummarizedExperiment package)
2. A vector of continuous scores (or `Input Scores`) representing a functional response of interest (such as protein expression, pathway activity, etc.)

    # Load pre-simulated feature set
    # See ?sim_FS for more information
    data(sim_FS)

    # Load pre-computed input-score
    # See ?sim_Scores for more information
    data(sim_Scores)

# Find a subset of features that maximally associated with a given outcome of interest

Here we are using Kolmogorow-Smirnow (KS) scoring method to search for best features

    candidate_search_res <- CaDrA::candidate_search(
      FS = sim_FS,
      input_score = sim_Scores,
      method = "ks_pval",          # Use Kolmogorow-Smirnow scoring function
      method_alternative = "less", # Use one-sided hypothesis testing
      weights = NULL,              # If weights is provided, perform a weighted-KS test
      search_method = "both",      # Apply both forward and backward search
      top_N = 7,                   # Number of top features to kick start the search
      max_size = 10,               # Allow at most 10 features in meta-feature matrix
      best_score_only = FALSE      # Return all results from the search
    )

# Visualize best meta-features result

    # Extract the best meta-feature result
    topn_best_meta <- CaDrA::topn_best(topn_list = candidate_search_res)

    # Visualize meta-feature result
    CaDrA::meta_plot(topn_best_list = topn_best_meta)

# Perform permutation-based testing

    # Set seed for permutation-based testing
    set.seed(123)

    perm_res <- CaDrA::CaDrA(
      FS = sim_FS,
      input_score = sim_Scores,
      method = "ks_pval",             # Use Kolmogorow-Smirnow scoring function
      method_alternative = "less",    # Use one-sided hypothesis testing
      weights = NULL,                 # If weights is provided, perform a weighted-KS test
      search_method = "both",         # Apply both forward and backward search
      top_N = 7,                      # Repeat the search with the top N features
      max_size = 10,                  # Allow at most 10 features in the meta-feature matrix
      n_perm = 100,                   # Number of permutations to perform
      perm_alternative = "one.sided", # One-sided permutation-based p-value alternative type
      plot = FALSE,                   # We will plot later
      ncores = 2                      # Number of cores to perform parallelization
    )

# Visualize permutation result

    # Visualize permutation results
    permutation_plot(perm_res = perm_res)

# SessionInfo

    sessionInfo()
    R version 4.5.2 (2025-10-31)
    Platform: x86_64-pc-linux-gnu
    Running under: Ubuntu 24.04.4 LTS

    Matrix products: default
    BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
    LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0

    locale:
     [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
     [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C
     [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
     [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
     [9] LC_ADDRESS=C               LC_TELEPHONE=C
    [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

    time zone: Etc/UTC
    tzcode source: system (glibc)

    attached base packages:
    [1] stats     graphics  grDevices utils     datasets  methods   base

    other attached packages:
    [1] CaDrA_1.9.0    testthat_3.3.2 devtools_2.4.6 usethis_3.2.1  rmarkdown_2.30

    loaded via a namespace (and not attached):
     [1] farver_2.1.2                R.utils_2.13.0
     [3] S7_0.2.1                    bitops_1.0-9
     [5] fastmap_1.2.0               digest_0.6.39
     [7] lifecycle_1.0.5             ellipsis_0.3.2
     [9] magrittr_2.0.4              compiler_4.5.2
    [11] rlang_1.1.7                 sass_0.4.10
    [13] tools_4.5.2                 yaml_2.3.12
    [15] knitr_1.51                  labeling_0.4.3
    [17] S4Arrays_1.11.1             pkgbuild_1.4.8
    [19] DelayedArray_0.37.0         plyr_1.8.9
    [21] RColorBrewer_1.1-3          pkgload_1.5.0
    [23] abind_1.4-8                 KernSmooth_2.23-26
    [25] R.cache_0.17.0              withr_3.0.2
    [27] purrr_1.2.1                 BiocGenerics_0.57.0
    [29] sys_3.4.3                   desc_1.4.3
    [31] R.oo_1.27.1                 grid_4.5.2
    [33] stats4_4.5.2                caTools_1.18.3
    [35] ggplot2_4.0.2               scales_1.4.0
    [37] gtools_3.9.5                iterators_1.0.14
    [39] MASS_7.3-65                 SummarizedExperiment_1.41.1
    [41] cli_3.6.5                   ppcor_1.1
    [43] generics_0.1.4              remotes_2.5.0
    [45] otel_0.2.0                  rstudioapi_0.18.0
    [47] reshape2_1.4.5              sessioninfo_1.2.3
    [49] cachem_1.1.0                stringr_1.6.0
    [51] parallel_4.5.2              XVector_0.51.0
    [53] matrixStats_1.5.0           vctrs_0.7.1
    [55] misc3d_0.9-2                Matrix_1.7-4
    [57] jsonlite_2.0.0              IRanges_2.45.0
    [59] S4Vectors_0.49.0            maketools_1.3.2
    [61] foreach_1.5.2               jquerylib_0.1.4
    [63] glue_1.8.0                  codetools_0.2-20
    [65] stringi_1.8.7               gtable_0.3.6
    [67] GenomicRanges_1.63.1        knnmi_1.0
    [69] brio_1.1.5                  htmltools_0.5.9
    [71] Seqinfo_1.1.0               gplots_3.3.0
    [73] R6_2.6.1                    tcltk_4.5.2
    [75] doParallel_1.0.17           rprojroot_2.1.1
    [77] evaluate_1.0.5              Biobase_2.71.0
    [79] lattice_0.22-9              R.methodsS3_1.8.2
    [81] memoise_2.0.1               bslib_0.10.0
    [83] Rcpp_1.1.1                  SparseArray_1.11.10
    [85] xfun_0.56                   fs_1.6.6
    [87] MatrixGenerics_1.23.0       buildtools_1.0.0