---
title: "Finding and downloading data"
author: "Sean Davis"
format:
  html:
    toc: true
vignette: >
  %\VignetteIndexEntry{Finding and downloading data}
  %\VignetteEngine{quarto::html}
  %\VignetteEncoding{UTF-8}
---

Most GEOquery examples assume you already have an accession. This article covers
the steps before and around that: **finding** studies programmatically,
**controlling** what `getGEO()` fetches, **caching** downloads for reproducible
work, and reaching **private** records under review.


``` r
library(GEOquery)
```

## Finding studies from R

If you do not yet have an accession, `searchGEO()` queries the NCBI GEO DataSets
database with the standard Entrez syntax and returns one row per hit:


``` r
res <- searchGEO("asthma[Title] AND GSE[ETYP]")
nrow(res)
#> [1] 387
res[1:5, c("Series Accession", "Title", "Organism")]
#>   Series Accession
#> 1        GSE301357
#> 2        GSE325572
#> 3        GSE334345
#> 4        GSE332995
#> 5        GSE297032
#>                                                                                                    Title
#> 1                       The airway transcriptome in type2 cytokine biomarker-high and -low severe asthma
#> 2         Transcriptomic analyses identify TMPRSS4 as a shared biomarker in asthma and atopic dermatitis
#> 3 Exploratory investigation of changes in blood transcriptome following mepolizumab treatment for asthma
#> 4        Shuangpi Formula Treats Neutrophilic Asthma via Circadian-Mediated Anti-Inflammatory Mechanisms
#> 5                                                       Atopic asthma lowers nasal mucosal sphingolipids
#>            Organism
#> 1      Homo sapiens
#> 2      Homo sapiens
#> 3      Homo sapiens
#> 4 Rattus norvegicus
#> 5      Homo sapiens
```

The query uses Entrez field tags — `[Title]`, `[Organism]`, `[Description]`,
`[ETYP]` (entity type, e.g. `GSE`), and many more — combined with `AND` / `OR`.
To see the fields available for searching:


``` r
head(searchFieldsGEO(), 12)
#>    Name     FullName  Description TermCount IsDate IsNumerical SingleToken
#> 1   ALL   All Fields All term....  48154253      N           N           N
#> 2   UID          UID Unique n....         0      N           Y           Y
#> 3  FILT       Filter Limits t....        71      N           N           Y
#> 4  ORGN     Organism exploded....     78347      N           N           Y
#> 5  ACCN GEO Acce.... accessio....  20700173      N           N           Y
#> 6  TITL        Title Words in....  10584350      N           N           Y
#> 7  DESC  Description Text fro....  11474236      N           N           Y
#> 8  SFIL Suppleme.... Suppleme....       256      N           N           Y
#> 9  ETYP   Entry Type Entry ty....         4      N           N           Y
#> 10 STYP  Sample Type  Sample type         9      N           N           Y
#> 11 VTYP Sample V.... type of ....         7      N           N           Y
#> 12 PTYP Platform.... Platform....        17      N           N           Y
#>    Hierarchy IsHidden
#> 1          N        N
#> 2          N        Y
#> 3          N        N
#> 4          Y        N
#> 5          N        N
#> 6          N        N
#> 7          N        N
#> 8          N        N
#> 9          N        N
#> 10         N        N
#> 11         N        N
#> 12         N        N
```

Once a hit looks promising, hand its accession straight to `getGEO()`:


``` r
gse <- getGEO(res[["Series Accession"]][1])
```

To open a record in your web browser instead — handy for eyeballing a study
before committing to a download — use `browseGEOAccession()`:


``` r
browseGEOAccession("GSE2553")
```

## Controlling what `getGEO()` fetches

`getGEO()` has a few knobs worth knowing:


``` r
# Default: fast Series Matrix path -> list of SummarizedExperiment
getGEO("GSE2553")

# Legacy container instead of SummarizedExperiment
getGEO("GSE2553", returnType = "ExpressionSet")

# Skip the platform (GPL) download when you don't need feature annotation;
# faster, and avoids fetching a large platform table
getGEO("GSE2553", getGPL = FALSE)

# Full SOFT record (a GSE S4 object) instead of the Series Matrix
getGEO("GSE2553", GSEMatrix = FALSE)

# Store the downloaded file somewhere stable (see Caching, below)
getGEO("GSE2553", destdir = "~/geo-cache")

# Override character encoding for the rare non-UTF-8 record
getGEO("GSE2553", encoding = "Latin-1")
```

The `encoding` override is also available globally with
`options(GEOquery.encoding = "Latin-1")` for a whole session.

## Supplementary files

Processed expression tables are only part of a study; raw and processed
supplementary files (RNA-seq counts, single-cell matrices, and more) are listed
and downloaded separately. Always look before you leap:


``` r
getGEOSuppFiles("GSE63137", fetch_files = FALSE)   # inspect: a data.frame of files
#>                                                               fname
#> 1                   GSE63137_ATAC-seq_PV_neurons_HOMER_peaks.bed.gz
#> 2                  GSE63137_ATAC-seq_VIP_neurons_HOMER_peaks.bed.gz
#> 3           GSE63137_ATAC-seq_excitatory_neurons_HOMER_peaks.bed.gz
#> 4   GSE63137_ChIP-seq_H3K27ac_excitatory_neurons_SICER_peaks.bed.gz
#> 5  GSE63137_ChIP-seq_H3K27me3_excitatory_neurons_SICER_peaks.bed.gz
#> 6   GSE63137_ChIP-seq_H3K4me1_excitatory_neurons_SICER_peaks.bed.gz
#> 7   GSE63137_ChIP-seq_H3K4me3_excitatory_neurons_SICER_peaks.bed.gz
#> 8                         GSE63137_MethylC-seq_DMRs_methylpy.txt.gz
#> 9                  GSE63137_MethylC-seq_PV_neurons_UMRs_LMRs.txt.gz
#> 10                GSE63137_MethylC-seq_VIP_neurons_UMRs_LMRs.txt.gz
#> 11         GSE63137_MethylC-seq_excitatory_neurons_UMRs_LMRs.txt.gz
#> 12                                                 GSE63137_RAW.tar
#>                                                                                                                                 url
#> 1                   https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_ATAC-seq_PV_neurons_HOMER_peaks.bed.gz
#> 2                  https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_ATAC-seq_VIP_neurons_HOMER_peaks.bed.gz
#> 3           https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_ATAC-seq_excitatory_neurons_HOMER_peaks.bed.gz
#> 4   https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_ChIP-seq_H3K27ac_excitatory_neurons_SICER_peaks.bed.gz
#> 5  https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_ChIP-seq_H3K27me3_excitatory_neurons_SICER_peaks.bed.gz
#> 6   https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_ChIP-seq_H3K4me1_excitatory_neurons_SICER_peaks.bed.gz
#> 7   https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_ChIP-seq_H3K4me3_excitatory_neurons_SICER_peaks.bed.gz
#> 8                         https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_MethylC-seq_DMRs_methylpy.txt.gz
#> 9                  https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_MethylC-seq_PV_neurons_UMRs_LMRs.txt.gz
#> 10                https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_MethylC-seq_VIP_neurons_UMRs_LMRs.txt.gz
#> 11         https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_MethylC-seq_excitatory_neurons_UMRs_LMRs.txt.gz
#> 12                                                 https://ftp.ncbi.nlm.nih.gov/geo/series/GSE63nnn/GSE63137/suppl/GSE63137_RAW.tar
```


``` r
getGEOSuppFiles("GSE63137")                        # download all into ./GSE63137/
```

## Caching and reproducibility

By default, GEOquery downloads to a **temporary** directory
(`destdir = tempdir()`), which R clears at the end of the session — so the same
`getGEO()` call re-downloads every session. For interactive work and
reproducible pipelines you usually want downloads to persist. Two options:

**Point `destdir` at a stable directory** on any call that downloads:


``` r
getGEO("GSE2553", destdir = "~/geo-cache")
getGEOSuppFiles("GSE63137", baseDir = "~/geo-cache")
```

**Or turn on the built-in persistent cache**, backed by `BiocFileCache`. Once
enabled, repeated downloads of the same URL are served from disk across sessions:


``` r
options(GEOquery.cache = TRUE)                 # enable for the session
options(GEOquery.cache.path = "~/geo-cache")   # optional: choose the location

geoCache()        # the BiocFileCache object backing the cache
clearGEOCache()   # wipe it when you want a clean re-download
```

For results you intend to publish, record what you used: pin a stable `destdir`
or the cache, and capture `sessionInfo()` (GEOquery and dependency versions)
alongside your output. GEO records can change over time, so a saved object plus a
version record is the difference between reproducible and "worked on my machine."

## Private and embargoed records

Records still under peer review are not published to GEO's public FTP tree, so a
normal `getGEO()` cannot see them. If you have a **reviewer access token** (the
"Reviewer access" link on the private GSE's page), pass it with `token`:


``` r
gse <- getGEO("GSE123456", token = "yourreviewertoken")
```

Because private records bypass the FTP tree, a token forces the SOFT
(`acc.cgi`) download path. For a GSE that means you get a `GSE` S4 object (as with
`GSEMatrix = FALSE`) rather than a `SummarizedExperiment`.

## Where to go next

- [Understanding GEO data formats](geo-data-formats.html) — what SOFT vs. Series
  Matrix means for the object you get back.
- [RNA-seq quantifications from GEO](rnaseq.html) and
  [Single-cell data from GEO](single-cell.html) — dedicated paths for
  sequencing-era supplementary data.
- [From GEO to downstream analysis](downstream-analysis.html) — where the object
  goes next.
