GDS (Genomic Data Structure) is a high-performance file format for storing and accessing large-scale genomic data, implemented by the gdsfmt package. It supports hierarchical data organization with efficient random access and data compression.
The gdscloud package extends gdsfmt to provide transparent read-only access to GDS files stored on cloud storage services. It uses HTTP Range requests via libcurl with an efficient LRU block cache to minimize network overhead.
Supported backends:
http:// or https:// URL (public or authenticated)s3://bucket/key; S3-compatible services
(MinIO, Ceph RGW, Cloudflare R2, Wasabi, Backblaze B2, …) via a custom endpointgs://bucket/keyaz://container/blobif (!requireNamespace("BiocManager", quietly=TRUE))
install.packages("BiocManager")
BiocManager::install("gdscloud")gdscloud recognizes the following URL schemes:
library(gdscloud)
## Loading required package: gdsfmt
gdsCloudSchemes()
## http
## "HTTP"
## https
## "HTTPS"
## s3
## "Amazon S3 (and S3-compatible services)"
## gs
## "Google Cloud Storage"
## az
## "Azure Blob Storage"Once gdscloud is loaded, openfn.gds() from the gdsfmt package automatically recognizes cloud URLs and opens them transparently:
library(gdscloud)
# Open a GDS file from S3 — transparent via openfn.gds()
gds <- openfn.gds("s3://gds-stat/download/hapmap/hapmap_r23a.gds")
gds
## File: s3://gds-stat/download/hapmap/hapmap_r23a.gds (86.5M)
## + [ ] *
## |--+ description [ ] *
## |--+ sample.id { Str8 270 LZMA_ra(17.3%), 381B } *
## |--+ variant.id { Int32 4098136 LZMA_ra(3.19%), 511.4K } *
## |--+ position { Int32 4098136 LZMA_ra(48.2%), 7.5M } *
## |--+ chromosome { Str8 4098136 LZMA_ra(0.02%), 1.7K } *
## |--+ allele { Str8 4098136 LZMA_ra(12.8%), 2.0M } *
## |--+ genotype [ ] *
## | |--+ data { Bit2 2x270x4098136 LZMA_ra(12.4%), 65.4M } *
## | |--+ extra.index { Int32 3x0 LZMA_ra, 18B } *
## | \--+ extra { Int16 0 LZMA_ra, 18B }
## |--+ phase [ ]
## | |--+ data { Bit1 270x4098136 LZMA_ra(0.01%), 19.8K } *
## | |--+ extra.index { Int32 3x0 LZMA_ra, 18B } *
## | \--+ extra { Bit1 0 LZMA_ra, 18B }
## |--+ annotation [ ]
## | |--+ id { Str8 4098136 LZMA_ra(27.7%), 11.1M } *
## | |--+ qual { Float32 4098136 LZMA_ra(0.02%), 2.5K } *
## | |--+ filter { Int32,factor 4098136 LZMA_ra(0.02%), 2.5K } *
## | |--+ info [ ]
## | \--+ format [ ]
## \--+ sample.annotation [ ]
## |--+ family { Str8 270 LZMA_ra(17.3%), 381B } *
## |--+ father { Str8 270 LZMA_ra(28.2%), 261B } *
## |--+ mother { Str8 270 LZMA_ra(28.2%), 261B } *
## |--+ sex { Str8 270 LZMA_ra(27.0%), 153B } *
## \--+ phenotype { Int32 270 LZMA_ra(8.70%), 101B } *
table(read.gdsn(index.gdsn(gds, "chromosome")))
##
## 1 10 11 12 13 14 15 16 17 18 19
## 318558 216535 209679 201179 161696 126523 109664 112428 91821 123089 58432
## 2 20 21 22 3 4 5 6 7 8 9
## 333056 122926 53454 58111 263547 252385 254297 278119 220384 222010 188661
## MT X XY Y
## 218 120679 362 323
summary(read.gdsn(index.gdsn(gds, "position")))
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 410 32182751 68995421 78255683 114256766 247195690
closefn.gds(gds)Alternatively, call gdsCloudOpen() directly with a cloud URL (see ?gdsCloudOpen); the S3, GCS, Azure, and HTTP examples below use it explicitly.
Each cloud backend requires credentials. Credentials can be set either via environment variables (recommended for non-interactive use) or via R functions (convenient for interactive sessions). R-configured credentials take priority over environment variables.
The secrets shown below are placeholders. Avoid typing real credentials directly into R scripts or on a shell command line, since both are saved to the R (.Rhistory) and shell histories. To keep secrets out of your history, store them in an OS credential store with the keyring package and retrieve them at run time:
# one-time, interactive: store the secret in the OS keychain
keyring::key_set("gdscloud", "aws_secret_access_key")
# later, in a script: retrieve without exposing it in history
gdsCloudConfigS3(
aws_access_key_id = keyring::key_get("gdscloud", "aws_access_key_id"),
aws_secret_access_key = keyring::key_get("gdscloud", "aws_secret_access_key"),
region = "us-east-1"
)For public URLs (e.g., files hosted on a web server with Range request support), no credentials are needed:
gds <- gdsCloudOpen("https://example.com/path/to/file.gds")
closefn.gds(gds)For authenticated HTTP endpoints, set a Bearer token:
Environment variable:
export GDSCLOUD_HTTP_TOKEN=your_bearer_tokenR configuration:
gdsCloudConfigHTTP(bearer_token = "your_bearer_token")
# URL-specific token for a private server
gdsCloudConfigHTTP(
bearer_token = "token_for_private_server",
url = "https://private.example.com/"
)Then open:
gds <- gdsCloudOpen("https://private.example.com/data/file.gds")
closefn.gds(gds)Environment variables:
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export AWS_DEFAULT_REGION=us-east-1
# Optional for temporary credentials:
export AWS_SESSION_TOKEN=your_tokenR configuration:
gdsCloudConfigS3(
aws_access_key_id = "your_key",
aws_secret_access_key = "your_secret",
region = "us-east-1"
)Then open:
gds <- gdsCloudOpen("s3://my-bucket/path/to/file.gds")
# ... work with the file ...
closefn.gds(gds)Any service implementing the S3 API (MinIO, Ceph RGW, Cloudflare R2, Wasabi,
Backblaze B2, DigitalOcean Spaces, …) can be used by pointing the S3
backend at its endpoint. Path-style addressing (https://host/bucket/key)
is the default for a custom endpoint; region is still part of the
signature and should be set to what the service documents:
# a MinIO or Ceph instance on the local network
gdsCloudConfigS3(
aws_access_key_id = "minioadmin",
aws_secret_access_key = "minioadmin",
endpoint = "http://minio.lab.internal:9000"
)
# Cloudflare R2, scoped to one bucket
gdsCloudConfigS3(
aws_access_key_id = "R2_KEY",
aws_secret_access_key = "R2_SECRET",
region = "auto",
endpoint = "https://ACCOUNT_ID.r2.cloudflarestorage.com",
url = "s3://my-r2-bucket/"
)The endpoint can also be given through the AWS_ENDPOINT_URL_S3 or
AWS_ENDPOINT_URL environment variable, as understood by the AWS CLI and
SDKs. Then open s3://bucket/key as usual:
gds <- gdsCloudOpen("s3://genomics/hapmap.gds")
closefn.gds(gds)Environment variable:
export GCS_ACCESS_TOKEN=your_oauth2_tokenR configuration:
gdsCloudConfigGCS(access_token = "your_oauth2_token")Then open:
gds <- gdsCloudOpen("gs://my-bucket/path/to/file.gds")
closefn.gds(gds)Environment variables:
export AZURE_STORAGE_ACCOUNT=your_account
export AZURE_STORAGE_KEY=your_key
# Or use a SAS token instead:
export AZURE_STORAGE_SAS_TOKEN=your_sas_tokenR configuration:
gdsCloudConfigAzure(
account_name = "mystorageaccount",
account_key = "base64encodedkey=="
)
# Or with SAS token:
gdsCloudConfigAzure(
account_name = "mystorageaccount",
sas_token = "sv=2021-06-08&ss=b&srt=co&sp=r..."
)An OAuth2 access token from Microsoft Entra ID (for example from
az account get-access-token --resource https://storage.azure.com/, or a
managed identity) can be used instead of a key or SAS token, and sovereign
clouds or local emulators are reached through endpoint_suffix or a full
endpoint:
gdsCloudConfigAzure(
account_name = "mystorageaccount",
access_token = "eyJ0eXAiOiJKV1QiLCJhbGci...",
endpoint_suffix = "blob.core.chinacloudapi.cn" # Azure China
)
# Azurite emulator
gdsCloudConfigAzure(
account_name = "devstoreaccount1",
account_key = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==",
endpoint = "http://127.0.0.1:10000/devstoreaccount1"
)Then open:
gds <- gdsCloudOpen("az://my-container/path/to/file.gds")
closefn.gds(gds)Google OAuth2 access tokens and Microsoft Entra ID tokens expire after
about an hour. Instead of a fixed string, the token arguments
(access_token for GCS and Azure, bearer_token for HTTP) accept a
function that returns the token; it is called each time a file is opened,
so a fresh token is fetched automatically. This is also the way to reuse
the login machinery of dedicated packages:
# Google Cloud Storage via gargle, or via the gcloud CLI
gdsCloudConfigGCS(access_token = function() {
scope <- "https://www.googleapis.com/auth/devstorage.read_only"
gargle::token_fetch(scopes = scope)$credentials$access_token
})
gdsCloudConfigGCS(access_token = function()
system2("gcloud", c("auth", "print-access-token"), stdout = TRUE))
# Azure via AzureAuth
gdsCloudConfigAzure(account_name = "mystorageaccount", access_token = function()
AzureAuth::get_azure_token("https://storage.azure.com/",
tenant = "mytenant", app = "myapp")$credentials$access_token)AWS temporary credentials (STS, SSO, instance roles) are three values that
expire together, so for S3 a single credentials function returns the
whole set; the AWS CLI’s export-credentials command resolves the complete
AWS credential chain:
gdsCloudConfigS3(credentials = function() {
out <- system2("aws", c("configure", "export-credentials",
"--format", "process"), stdout = TRUE)
j <- jsonlite::fromJSON(paste(out, collapse = ""))
list(aws_access_key_id = j$AccessKeyId,
aws_secret_access_key = j$SecretAccessKey,
session_token = j$SessionToken)
})A function returning NULL or "" means no token is available and the
next fallback is used. Provider functions are exported to parallel workers
by gdsCloudExportCredentials() like any other credential. Static keys
(AWS access keys, Azure account keys, SAS tokens) are given as strings.
Sometimes different URLs need different credentials — for example, two S3 buckets owned by different accounts, or a mix of public and private resources. Each gdsCloudConfig*() function accepts an optional url argument that associates the supplied credentials with a URL prefix rather than the global defaults:
# Different keys for two S3 buckets
gdsCloudConfigS3(
aws_access_key_id = "KEY_A",
aws_secret_access_key = "SECRET_A",
url = "s3://bucket-a/"
)
gdsCloudConfigS3(
aws_access_key_id = "KEY_B",
aws_secret_access_key = "SECRET_B",
url = "s3://bucket-b/"
)
# You can also scope credentials to a sub-prefix within a bucket
gdsCloudConfigS3(
aws_access_key_id = "KEY_SHARED",
aws_secret_access_key = "SECRET_SHARED",
url = "s3://bucket-a/shared/"
)
# Remove a previously registered URL-specific entry
gdsCloudConfigS3(url = "s3://bucket-a/")When opening a URL, credentials are resolved in the following order (the first non-empty value wins for each field):
gdsCloudConfig*() with url=NULL;The URL scheme passed via url= must match the function — http:// or https:// for gdsCloudConfigHTTP(), s3:// for gdsCloudConfigS3(), gs:// for gdsCloudConfigGCS(), and az:// for gdsCloudConfigAzure(). Registered prefixes are normalized by appending a trailing / if missing, so "s3://bucket" and "s3://bucket/" behave identically.
Cloud access uses an LRU block cache (1 MB blocks) to minimize HTTP requests. The default cache size is 64 MB per stream.
# Set the default cache size for new streams (in MB)
gdsCloudCacheSize(128)
# Clear all cached data
gdsCloudCacheClear()
# Display cache statistics
gdsCloudCacheInfo()
## gdscloud cache settings:
## Default cache size: 128 MB
## Block size: 1 MB
## Open cloud streams: 0
## Global cache hits: 0
## Global cache misses: 0
## Retried requests: 0
# List all open cloud streams
gdsCloudList()
## [1] url file_size cache_blocks cache_hits cache_misses
## <0 rows> (or 0-length row.names)Increasing cache size is beneficial when working with large files that require many random seeks (e.g., subsetting genotype matrices by sample and variant).
Since gdscloud works transparently through openfn.gds(), packages built on gdsfmt, such as SeqArray, can open cloud-hosted files directly.
Note: SeqArray >= v1.53.1 is recommended for full cloud support. This version allows seqParallel() to automatically load cloud-related packages on worker processes, so parallel operations on cloud-hosted GDS files work seamlessly.
library(SeqArray)
library(gdscloud)
# Open a SeqArray GDS file from S3
gds <- seqOpen("s3://gds-stat/download/1000g/2022/1kGP_high_coverage_Illumina.allchr.filtered.SNV_INDEL_SV_phased_panel.gds")
gds
## File: s3://gds-stat/download/1000g/2022/1kGP_high_coverage_Illumina.allchr.filtered.SNV_INDEL_SV_phased_panel.gds (2.4G)
## + [ ] *
## |--+ description [ ] *
## |--+ sample.id { Str8 3202 LZMA_ra(6.61%), 1.7K } *
## |--+ variant.id { Int32 73554796 LZMA_ra(2.36%), 6.6M } *
## |--+ position { Int32 73554796 LZMA_ra(27.9%), 78.4M } *
## |--+ chromosome { Str8 73554796 LZMA_ra(0.01%), 25.3K } *
## |--+ allele { Str8 73554796 LZMA_ra(15.8%), 51.4M } *
## |--+ genotype [ ] *
## | |--+ data { Bit2 2x3202x73554796 LZMA_ra(1.77%), 1.9G } *
## | |--+ extra.index { Int32 3x0 LZMA_ra, 18B } *
## | \--+ extra { Int16 0 LZMA_ra, 18B }
## |--+ phase [ ]
## | |--+ data { Bit1 3202x73554796 LZMA_ra(0.01%), 4.1M } *
## | |--+ extra.index { Int32 3x0 LZMA_ra, 18B } *
## | \--+ extra { Bit1 0 LZMA_ra, 18B }
## |--+ annotation [ ]
## | |--+ id { Str8 73554796 LZMA_ra(17.2%), 186.3M } *
## | |--+ qual { Float32 73554796 LZMA_ra(0.01%), 42.0K } *
## | |--+ filter { Int32,factor 73554796 LZMA_ra(0.01%), 42.0K } *
## | |--+ info [ ]
## | | |--+ AF { Float32 73554796 LZMA_ra(23.6%), 66.3M } *
## | | |--+ AC { Int32 73554796 LZMA_ra(22.3%), 62.5M } *
## | | |--+ CM { Float32 73554796 LZMA_ra(6.08%), 17.1M } *
## | | |--+ AN { Int32 73554796 LZMA_ra(0.01%), 42.0K } *
## | | \--+ SVTYPE { Str8 73554796 LZMA_ra(0.33%), 240.0K } *
## | \--+ format [ ]
## \--+ sample.annotation [ ]
seqSummary(gds)
## File: s3://gds-stat/download/1000g/2022/1kGP_high_coverage_Illumina.allchr.filtered.SNV_INDEL_SV_phased_panel.gds
## Format Version: v1.0
## Reference: unknown
## Ploidy: 2
## Number of samples: 3,202
## Number of variants: 73,554,796
## Chromosomes:
## chr1 : 5759060, chr2 : 6088598, chr3 : 4983185, chr4 : 4875465, chr5 : 4536819, chr6 : 4315217
## chr7 : 4137254, chr8 : 3886222, chr9 : 3165513, chr10: 3495473, chr11: 3423341, chr12: 3332788
## chr13: 2509179, chr14: 2290400, chr15: 2109285, chr16: 2362361, chr17: 2073624, chr18: 1963845
## chr19: 1670692, chr20: 1644384, chr21: 1002753, chr22: 1066557, chrX : 2862781
## ...
seqClose(gds)No code changes are needed in downstream packages; loading gdscloud is sufficient to enable cloud URL support.
sessionInfo()
## R version 4.6.1 (2026-06-24)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.4 LTS
##
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.24-bioc/R/lib/libRblas.so
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0 LAPACK version 3.12.0
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_GB LC_COLLATE=C
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## time zone: America/New_York
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] gdscloud_0.99.5 gdsfmt_1.49.8 BiocStyle_2.41.0
##
## loaded via a namespace (and not attached):
## [1] digest_0.6.39 R6_2.6.1 bookdown_0.48
## [4] fastmap_1.2.0 xfun_0.61 cachem_1.1.0
## [7] knitr_1.52 htmltools_0.5.9 rmarkdown_2.32
## [10] lifecycle_1.0.5 cli_3.6.6 sass_0.4.10
## [13] jquerylib_0.1.4 compiler_4.6.1 tools_4.6.1
## [16] evaluate_1.0.5 bslib_0.12.0 yaml_2.3.12
## [19] otel_0.2.0 BiocManager_1.30.27 crayon_1.5.3
## [22] jsonlite_2.0.0 rlang_1.3.0