Package: MsBackendParquet
Authors: Ossama Edbali [aut, cre] (ORCID:
https://orcid.org/0000-0003-0132-8668),
Ralf Johannes Maria Weber [aut] (ORCID:
https://orcid.org/0000-0002-8796-4771),
BioFAIR UK [fnd] (BioFAIR Pathfinder Project)
Compiled: Wed Oct 14 19:04:24 2026
The MsBackendParquet package provides a backend for
Spectra objects that stores mass spectrometry (MS) data
in an on-disk dataset built on
Apache Parquet.
Datasets are written with the
Apache Arrow R bindings and queried
with DuckDB, keeping the spectra metadata
together with the mz/intensity peak values in a columnar format that
supports compression, dictionary encoding and Hive-style partitioning.
A dataset is a directory identified by an mzStack.json
manifest, holding one or more runs. A run is one source of spectra
and owns a contiguous block of the dataset’s spectrum ids.
A run is either
native (converted from a raw MS data file, with peaks stored alongside
the metadata) or mzpeak, an index over external
HUPO-PSI mzPeak archives that are
read where they are and never modified (see
Querying collections of mzPeak archives below). The manifest records
which kind each run is, so the two never have to be told apart by
inspecting the directory.
To keep access fast, the (small) non-peak spectra variables are held in a bounded in-memory index while the peak data is read from disk on demand. As a result, metadata filters cost about as much as with an in-memory backend, while the bulk peak storage stays out of core.
The backend extends MsBackendCached from Spectra so it
benefits from caching of user-defined spectra variables while keeping
the on-disk dataset read-only.
MsBackendMzR from Spectra keeps the spectra metadata in
memory and reads peaks from the raw mzML files on demand: nothing has to be
converted, but every peak access parses the files again.
MsBackendSql stores spectra in an SQL database, which suits
data shared from a database server. SpectraStash saves and
restores whole Spectra objects in interoperable formats such as HDF5. It
is for exchanging an object, not for querying it on disk.
MsBackendParquet converts once, like MsBackendSql, and then queries a
database engine, DuckDB. But its storage is plain Parquet files that need
no server and that other tools, such as Python, DuckDB or Spark, read
directly. It can also index mzPeak archives where they are, without
converting them.
The package is installed from Bioconductor:
if (!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("MsBackendParquet")Reading raw mzML files needs the mzR package, which
BiocManager::install("MsBackendParquet", dependencies = TRUE) installs
too. Datasets made from mzPeak archives or from in-memory data do not
need it.
The examples below use a DDA data file of a pesticide mix, provided by the MsDataHub package. The file is downloaded once and cached locally.
library(Spectra)
library(MsBackendParquet)
library(MsDataHub)
fl <- MsDataHub::PestMix1_DDA.mzML()MsDataHub returns the file under its cache name, and the id of a run is
taken from the name of the file it was imported from. Copying the file to
a named path first therefore gives the dataset readable run ids:
fl_named <- file.path(tempdir(), "PestMix1_DDA.mzML")
invisible(file.copy(fl, fl_named, overwrite = TRUE))mzMLToParquet() converts raw MS data files into a Parquet dataset. It
validates the input files, writes the dataset and returns an initialised
backend, ready to be wrapped in a Spectra object:
path <- tempfile()
be <- mzMLToParquet(fl_named, path = path, verbose = FALSE)
#> Importing data ...
#> duckdb keeps downloaded extensions and secrets in a temporary directory:
#> ℹ /tmp/RtmpwgYFK3/duckdb
#> This is removed when the R session ends.
#> • Extensions are re-downloaded each session.
#> • Secrets are lost.
#> ℹ Run duckdb(shared_home = TRUE) (or create ~/.duckdb) to keep them (suitable for most users).
#> ℹ Run duckdb(shared_home = FALSE) to accept the temporary directory (and silence this message).
#> ℹ See ?duckdb_storage for details and alternatives.
sps <- Spectra(be)
sps
#> MSn data (Spectra) with 7602 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 1 NA 1
#> 3 1 NA 1
#> 4 1 NA 1
#> 5 1 NA 1
#> ... ... ... ...
#> 7598 1 NA 1
#> 7599 1 NA 1
#> 7600 1 NA 1
#> 7601 1 NA 1
#> 7602 1 NA 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c505d89ba29Each input file becomes its own run, written to
<path>/spectra/run_id=<id>/, so a filter that selects whole files
prunes at the directory level:
list.files(file.path(path, "spectra"))
#> [1] "run_id=PestMix1_DDA"mzMLToParquet() imports MS data via MsBackendMzR() and writes the
spectra metadata together with the mz/intensity values as Parquet
list<double> columns.
A Spectra object that is already loaded, with any backend, can instead be
moved onto Parquet with setBackend(). The destination is given as path:
sps_mem <- Spectra(fl_named, source = MsBackendMzR())
sps_pq <- setBackend(sps_mem, MsBackendParquet(), path = tempfile())
sps_pq
#> MSn data (Spectra) with 7602 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 1 NA 1
#> 3 1 NA 1
#> 4 1 NA 1
#> 5 1 NA 1
#> ... ... ... ...
#> 7598 1 NA 1
#> 7599 1 NA 1
#> 7600 1 NA 1
#> 7601 1 NA 1
#> 7602 1 NA 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c5034a24b07
#> Processing:
#> Switch backend from MsBackendMzR to MsBackendParquet [Wed Oct 14 19:04:40 2026]spectraData(), peaksData(), mz(), intensity() and related
methods work the same way as for any other MsBackend. The spectra
variables of the dataset are:
spectraVariables(sps)
#> [1] "msLevel" "rtime"
#> [3] "acquisitionNum" "scanIndex"
#> [5] "dataStorage" "dataOrigin"
#> [7] "centroided" "smoothed"
#> [9] "polarity" "precScanNum"
#> [11] "precursorMz" "precursorIntensity"
#> [13] "precursorCharge" "collisionEnergy"
#> [15] "isolationWindowLowerMz" "isolationWindowTargetMz"
#> [17] "isolationWindowUpperMz" "spectrum_id_"
#> [19] "run_id" "spectrum_index"
#> [21] "spectrumId" "totIonCurrent"
#> [23] "basePeakMz" "basePeakIntensity"
#> [25] "lowMz" "highMz"
#> [27] "peaksCount" "injectionTime"
#> [29] "filterString" "scanWindowLowerLimit"
#> [31] "scanWindowUpperLimit" "instrumentConfigurationId"
#> [33] "spectrumType" "basePeakMZ_1"
#> [35] "electronBeamEnergy" "ionisationEnergy"
#> [37] "lowMZ_1" "highMZ_1"
#> [39] "mergedScan" "mergedResultScanNum"
#> [41] "mergedResultStartScanNum" "mergedResultEndScanNum"
#> [43] "ionMobilityDriftTime"Values are returned as usual, with the peak data read from disk on demand:
spectraData(sps, c("msLevel", "rtime", "precursorMz"))
#> DataFrame with 7602 rows and 3 columns
#> msLevel rtime precursorMz
#> <integer> <numeric> <numeric>
#> 1 1 0.231 NA
#> 2 1 0.351 NA
#> 3 1 0.471 NA
#> 4 1 0.591 NA
#> 5 1 0.711 NA
#> ... ... ... ...
#> 7598 1 899.491 NA
#> 7599 1 899.613 NA
#> 7600 1 899.747 NA
#> 7601 1 899.872 NA
#> 7602 1 899.993 NA
head(peaksData(sps)[[1L]])
#> mz intensity
#> [1,] 50.01579 0.03076322
#> [2,] 52.01793 0.16344352
#> [3,] 53.00306 0.38972694
#> [4,] 53.01328 1.38231599
#> [5,] 53.84987 1.00965071
#> [6,] 53.93832 0.03967578
head(tic(sps))
#> [1] 168989 167674 193191 146445 145577 156807Filter methods are translated into DuckDB SQL predicates that are pushed down to the Parquet scan. DuckDB prunes whole row groups using their min/max statistics, and skips whole partitions for Hive-partitioned datasets, so a range or set filter reads only the data it needs. When the filter column is held in the in-memory metadata index (see The in-memory metadata cache below), the predicate is evaluated in memory instead.
Selecting spectra by MS level and by retention time, which is given in seconds:
filterMsLevel(sps, 2L)
#> MSn data (Spectra) with 2975 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 137.9639 1
#> 2 2 56.9419 1
#> 3 2 89.9449 1
#> 4 2 207.0294 1
#> 5 2 55.0083 1
#> ... ... ... ...
#> 2971 2 56.0495 1
#> 2972 2 67.0289 1
#> 2973 2 142.9611 1
#> 2974 2 53.0129 1
#> 2975 2 91.5022 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c505d89ba29
#> Processing:
#> Filter: select MS level(s) 2 [Wed Oct 14 19:04:40 2026]
filterRt(sps, c(200, 300))
#> MSn data (Spectra) with 717 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 1 NA 1
#> 3 1 NA 1
#> 4 1 NA 1
#> 5 1 NA 1
#> ... ... ... ...
#> 713 1 NA 1
#> 714 1 NA 1
#> 715 2 320.107 1
#> 716 1 NA 1
#> 717 2 320.107 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c505d89ba29
#> Processing:
#> Filter: select retention time [200..300] on MS level(s) [Wed Oct 14 19:04:40 2026]Filters return a Spectra object and can be chained. Each call narrows
the predicate rather than reading data, so the combination is answered
by a single scan:
sps |>
filterRt(c(200, 300)) |>
filterMsLevel(2L) |>
length()
#> [1] 216MS2 spectra can be selected by precursor m/z, either over a range or against a set of values with a mass tolerance in ppm:
filterPrecursorMzRange(sps, c(278, 279))
#> MSn data (Spectra) with 9 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 278.094 1
#> 2 2 278.091 1
#> 3 2 278.095 1
#> 4 2 278.092 1
#> 5 2 278.091 1
#> 6 2 278.092 1
#> 7 2 278.091 1
#> 8 2 278.091 1
#> 9 2 278.092 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c505d89ba29
#> Processing:
#> Filter: select spectra with a precursor m/z within [278, 279] [Wed Oct 14 19:04:40 2026]
ms2 <- filterPrecursorMzValues(sps, 278.093, ppm = 30)
ms2
#> MSn data (Spectra) with 9 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 278.094 1
#> 2 2 278.091 1
#> 3 2 278.095 1
#> 4 2 278.092 1
#> 5 2 278.091 1
#> 6 2 278.092 1
#> 7 2 278.091 1
#> 8 2 278.091 1
#> 9 2 278.092 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c505d89ba29
#> Processing:
#> Filter: select spectra with precursor m/z matching 278.093 [Wed Oct 14 19:04:40 2026]
range(rtime(ms2))
#> [1] 268.587 387.849filterPrecursorMzValues() accepts several values at once, which become
one OR-ed predicate and therefore one scan rather than one scan per
value:
filterPrecursorMzValues(sps, c(278.093, 124.086), tolerance = 0.01)
#> MSn data (Spectra) with 63 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 124.086 1
#> 2 2 278.094 1
#> 3 2 278.091 1
#> 4 2 278.095 1
#> 5 2 124.086 1
#> ... ... ... ...
#> 59 2 124.086 1
#> 60 2 124.086 1
#> 61 2 124.086 1
#> 62 2 124.085 1
#> 63 2 124.087 1
#> ... 42 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c505d89ba29
#> Processing:
#> Filter: select spectra with precursor m/z matching 278.093, 124.086 [Wed Oct 14 19:04:41 2026]None of the filters above read peak data. They are answered from the spectra variables, and the resulting predicate is carried into any later peak access.
Peak data is read when it is asked for, and only for the spectra that survived the filters. The nine MS2 spectra selected above share one precursor, so their fragments can be compared directly:
lengths(ms2)
#> [1] 25 14 12 4 8 4 1 18 6
pk <- peaksData(ms2)[[which.max(lengths(ms2))]]
head(pk[order(-pk[, "intensity"]), ])
#> mz intensity
#> [1,] 164.0541 1.1601784
#> [2,] 278.0952 1.0027263
#> [3,] 161.1078 0.4363745
#> [4,] 120.0820 0.2798612
#> [5,] 130.0637 0.1656372
#> [6,] 149.1082 0.1439338containsMz() from Spectra reports which of those
spectra contain a peak at a given m/z:
which(containsMz(ms2, mz = 164.054, tolerance = 0.01))
#> [1] 1 2 3Because the peaks are stored columnwise, DuckDB reads only the mz and
intensity columns, and only the rows belonging to the selected
spectra.
createMsBackendParquetDataset() writes a dataset without opening it
and exposes the layout options: the reader backend, chunk size,
compression codec, Parquet row-group size, and Hive partitioning.
path_part <- tempfile()
createMsBackendParquetDataset(path = path_part, x = fl,
partitioning = "msLevel",
compression = "zstd")
#> Importing data ...
be_part <- backendInitialize(MsBackendParquet(), path = path_part)
length(be_part)
#> [1] 7602partitioning adds a further Hive level below the run.
The partition directories carry mzPeak column names (see How the
dataset interfaces with mzPeak’s model below):
list.files(list.files(file.path(path_part, "spectra"), full.names = TRUE))
#> [1] "ms_level=1" "ms_level=2"The same function writes an existing DataFrame of spectra data when it
is passed as data instead of x.
mzPeak is the HUPO-PSI successor to mzML. An mzPeak archive holds one MS run as a folder (or an uncompressed ZIP) of Parquet files plus a JSON manifest naming them. That structure is well suited to distributing and archiving individual runs, but inefficient for queries spanning thousands of runs. Answering such a query would require opening and parsing each archive and its manifest.
MsBackendParquet bridges the two. createMzPeakDataset() registers a
set of archives as one dataset. The package ships two small archives for
this purpose:
archives <- system.file("extdata", c("QC01.mzpeak", "QC02.mzpeak"),
package = "MsBackendParquet")
qc_path <- tempfile()
createMzPeakDataset(archives[1L], path = qc_path, verbose = FALSE)Further archives join the same dataset later. Only the index grows:
addMzPeakArchives(qc_path, archives[2L], verbose = FALSE)
be_qc <- backendInitialize(MsBackendParquet(), path = qc_path)
sps_qc <- Spectra(be_qc)
sps_qc
#> MSn data (Spectra) with 10 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> 5 1 NA 1
#> 6 2 405 1
#> 7 1 NA 1
#> 8 2 401 1
#> 9 1 NA 1
#> 10 2 403 1
#> ... 36 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c5054c0676a
#> mzPeak archives: 2 run(s), signal read as: auto
table(basename(dataOrigin(sps_qc)))
#>
#> QC01.mzpeak QC02.mzpeak
#> 6 4With two runs in the dataset, filterDataOrigin() selects one of them:
filterDataOrigin(sps_qc, grep("QC02", dataOrigin(sps_qc), value = TRUE)[1L])
#> MSn data (Spectra) with 4 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> ... 36 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c5054c0676a
#> mzPeak archives: 2 run(s), signal read as: auto
#> Processing:
#> Filter: select data origin(s) /tmp/RtmpwgYFK3/file3c1c5054c0676a/runs/QC02.mzpeak [Wed Oct 14 19:04:44 2026]The archives are neither modified nor copied. Ingest reads each
archive’s manifest, its spectrum metadata table and its scan, precursor
and selected-ion side tables, and writes a small derived index next to
them, around 3% of the size of the archives. It never reads a byte of
signal data, so the cost scales with the number of spectra rather than
the size of the data. Peaks are then read from the archives themselves,
on demand. Pass link = "copy" to place a copy of each archive inside
the dataset directory instead of referencing it in place.
Because the archives stay exactly as they were, they remain readable by every other mzPeak tool, and exporting a run back out of the dataset is a directory copy rather than a conversion.
Several aspects of the data are translated or transformed on input, so it is worth being precise about what changes.
time in minutes;
Spectra expects rtime in seconds. The conversion, and every other
column rename and re-encoding (scan_polarity ±1, spectrum_representation
as a CURIE, the isolation window as a target plus offsets, and so on),
happens in a single SQL view. The same view serves a natively converted
dataset: its spectra/ files are written with mzPeak’s column names and
encodings too, so both kinds are directly comparable and either can be
exported back to an archive mechanically. spectrum_index is always the
dataset’s unique 0-based key; a spectrum’s position within its source file
(scanIndex, which restarts per file) is kept in scan_index. The package
adds only a handful of reserved columns (spectrum_id_, acquisition_num_,
the n_* counts).n_scans,
n_selected_ions and n_precursors. The archive still holds the full
detail, but a spectrum with several scans is summarised in the index.representation, which defaults
to "auto":be_centroid <- backendInitialize(MsBackendParquet(), path = qc_path,
representation = "centroid")
be_centroid
#> MsBackendParquet with 10 spectra
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> 5 1 NA 1
#> 6 2 405 1
#> 7 1 NA 1
#> 8 2 401 1
#> 9 1 NA 1
#> 10 2 403 1
#> ... 36 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c5054c0676a
#> mzPeak archives: 2 run(s), signal read as: centroidNA.Archives that use mzPeak’s chunked signal layout are rejected with a clear message; only the point layout is supported so far.
Filters on spectra variables work as they do for any other dataset. Finding spectra that contain a peak at a given m/z is different: it has to look at the signal, and mzPeak stores signal ordered by spectrum, which means a filter on m/z cannot skip anything, because every block of rows spans the whole mass range.
filterContainsMz() runs that search inside the storage engine rather
than pulling peaks into R:
hits <- filterContainsMz(be_qc, c(201, 10101), ppm = 20)
length(hits)
#> [1] 2
basename(dataOrigin(hits))
#> [1] "QC01.mzpeak" "QC02.mzpeak"filterContainsMz() works on the backend, and the result is wrapped in a
Spectra object like any other backend:
Spectra(hits)
#> MSn data (Spectra) with 2 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 2 401 1
#> 2 1 NA 1
#> ... 36 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c5054c0676a
#> mzPeak archives: 2 run(s), signal read as: autoFor datasets where this is a common query, build an m/z-ordered copy of the signal. Sorting is what makes the block statistics useful, so the same search then reads a handful of blocks instead of everything:
buildProjection(qc_path, verbose = FALSE)
length(filterContainsMz(be_qc, 201, ppm = 20))
#> [1] 1On a 5-million-point test dataset this made filterContainsMz() around
eight times faster. A projection is a cache: it is a derived copy of the
signal and so costs roughly as much disk again, it can be dropped with
dropProjection(), and results are identical with or without it. Only
the performance changes. Build it per run, for the runs you query often.
dropProjection(qc_path, verbose = FALSE)An experiment records factors about the sample a run came from, such as
subject, timepoint, treatment group or injection order, that are not in
the raw data. runData() stores them in the dataset, one row per run,
keyed by run_id.
runData(be_qc) <- data.frame(run_id = c("QC01", "QC02"),
subject = c("S1", "S2"),
timepoint = c(0L, 6L))
runVariables(be_qc)
#> [1] "subject" "timepoint"The columns then behave like any other spectra variable:
spectraData(be_qc, c("rtime", "subject", "timepoint"))
#> DataFrame with 10 rows and 3 columns
#> rtime subject timepoint
#> <numeric> <character> <integer>
#> 1 1 S1 0
#> 2 2 S1 0
#> 3 3 S1 0
#> 4 4 S1 0
#> 5 5 S1 0
#> 6 6 S1 0
#> 7 1 S2 6
#> 8 2 S2 6
#> 9 3 S2 6
#> 10 4 S2 6and filterSampleData() selects on them:
Spectra(filterSampleData(be_qc, timepoint == 6))
#> MSn data (Spectra) with 4 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> ... 38 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c5054c0676a
#> mzPeak archives: 2 run(s), signal read as: autoThe sample metadata is held once per run, not copied onto every spectrum.
Each run owns a
contiguous block of spectrum_id_, so timepoint == 6 is exactly a
range of ids, and filterSampleData() resolves it against a table of a
few thousand rows rather than reading a column of a few million. The
resulting predicate is a range on the column the files are ordered by,
which is the case Parquet’s row-group statistics prune best, and it is
carried into any later peaksData().
Keeping it out of the per-spectrum data also makes it correctable. Sample metadata is the one thing that reliably changes after ingest. Rewriting it touches a few kilobytes and leaves the signal, and any projections built over it, untouched.
Runs with no row get NA, so metadata covering only some runs is fine. To
go further, and link samples to spectra in an experiment-wide object, pass
the same table to MsExperiment and link the two on
run_id:
library(MsExperiment)
samples <- S4Vectors::DataFrame(run_id = c("QC01", "QC02"),
subject = c("S1", "S2"),
timepoint = c(0L, 6L))
mse <- MsExperiment(sampleData = samples, spectra = Spectra(be_qc))
mse <- linkSampleData(mse, with = "sampleData.run_id = spectra.run_id")
mse
#> Object of class MsExperiment
#> Spectra: MS1 (5) MS2 (5)
#> Experiment data: 2 sample(s)
#> Sample data links:
#> - spectra: 2 sample(s) to 10 element(s).
## The spectra of the second sample.
spectra(mse[2])
#> MSn data (Spectra) with 4 spectra in a MsBackendParquet backend:
#> msLevel precursorMz polarity
#> <integer> <numeric> <integer>
#> 1 1 NA 1
#> 2 2 401 1
#> 3 1 NA 1
#> 4 2 403 1
#> ... 38 more variables/columns.
#> Use 'spectraVariables' to list all of them.
#> Dataset: /tmp/RtmpwgYFK3/file3c1c5054c0676a
#> mzPeak archives: 2 run(s), signal read as: autoThe non-peak spectra variables are typically small compared to the peak
data, so MsBackendParquet keeps them in a bounded, read-through
in-memory index. The cache is held at the package level, keyed by
dataset path and shared by every backend over that dataset, so it is
independent of the individual, serialisable backend objects. It is
warmed on backendInitialize() and dropped automatically whenever the
dataset on disk is (re-)written. It is entirely separate from the
user-modified variables tracked by MsBackendCached, which always take
precedence.
The cache is controlled through three options, shown here with their defaults:
## Enable ("auto", the default) or disable ("off") the metadata cache.
options(MsBackendParquet.cacheMetadata = "auto")
## Columns held by default (the ones the optimised filters need).
options(MsBackendParquet.cacheColumns =
c("msLevel", "rtime", "precursorMz", "dataOrigin"))
## Budget in cells (rows x columns); 5e7 is roughly 400 MB of doubles.
## Requests over budget quietly fall back to reading from disk.
options(MsBackendParquet.cacheMaxCells = 5e7)Two further options tune the underlying DuckDB session.
MsBackendParquet.threads sets how many threads DuckDB uses, and Arrow
too. When it is unset, both use at most two threads, and one when the
registered BiocParallel backend is serial, so the package
does not take every core of a shared machine. DuckDB reads the option when
the package first opens its connection, and Arrow when the package is
loaded, so set it before either.
MsBackendParquet.memoryLimit maps to DuckDB’s memory_limit, and is
left at DuckDB’s default when unset.
For very large mzML files, the default conversion path goes through a
Spectra object built with MsBackendMzR(), which materialises a
whole chunk of peaks in memory before writing it to Parquet. The
engine = "mzr" option bypasses Spectra entirely and streams
batches of spectra directly from mzR::openMSfile() into Apache
Arrow’s incremental ParquetFileWriter:
be_stream <- mzMLToParquet(fl, path = tempfile(),
engine = "mzr", batch_size = 1000L,
verbose = FALSE)
length(be_stream)
#> [1] 7602Memory use stays bounded by batch_size * mean(peaks per spectrum),
regardless of total file size. The streaming engine processes files
serially and requires the mzR package.
Storing the peaks as columnar list<double> values lets DuckDB skip
whole row groups when a query touches only part of the data, while
projection and predicate push-down keep both the columns and rows read
to a minimum. Combined with the in-memory metadata index, this makes
metadata filters roughly as fast as an in-memory backend while the peak
storage stays on disk.
Across a suite of seven queries over 380,100 spectra (50 mzML files),
the Parquet backend is the fastest of the on-disk Spectra backends on
5 of 7 queries (medians in ms, lower is better):
| query | MsBackendParquet | MsBackendMzR | MsBackendHdf5Peaks | MsBackendOfflineSql |
|---|---|---|---|---|
filterRt(), wide range |
12.24 | 15.44 | 24.05 | 380.19 |
filterRt(), narrow range |
11.23 | 5.33 | 14.76 | 348.17 |
filterPrecursorMzValues(), ppm |
14.09 | 21.66 | 28.66 | 336.33 |
filterRt() and filterPrecursorMzValues() |
12.56 | 17.25 | 27.75 | 389.22 |
peaksData() of MS1 spectra |
314.82 | 1510 | 1440 | 919.04 |
tic() of MS1 spectra |
506.79 | 1390 | 1490 | 890.82 |
spectraData() of MS1 spectra |
26.09 | 17.27 | 22.30 | 718.91 |
The benchmark scripts live in
inst/benchmarks/;
see inst/benchmarks/performance.Rmd for the full numbers, the
configuration sweeps over the tunable options, and the design
considerations behind them.
Because MsBackendParquet objects only contain a file system path,
they are fully serialisable and can be sent to parallel workers
without re-opening connections:
rds <- tempfile(fileext = ".rds")
saveRDS(be, rds)
be2 <- readRDS(rds)
identical(spectraNames(be), spectraNames(be2))
#> [1] TRUEsessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 24.04.5 LTS
#>
#> Matrix products: default
#> BLAS: /home/biocbuild/bbs-3.24-bioc/R/lib/libRblas.so
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0 LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_GB LC_COLLATE=C
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: America/New_York
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] MsExperiment_1.15.0 ProtGenerics_1.45.0 MsDataHub_1.13.3
#> [4] MsBackendParquet_0.99.8 Spectra_1.23.5 BiocParallel_1.47.0
#> [7] S4Vectors_0.51.10 BiocGenerics_0.59.12 generics_0.1.4
#> [10] BiocStyle_2.41.0
#>
#> loaded via a namespace (and not attached):
#> [1] DBI_1.3.0 httr2_1.3.0
#> [3] rlang_1.3.0 magrittr_2.0.5
#> [5] clue_0.3-68 otel_0.2.0
#> [7] matrixStats_1.5.0 compiler_4.6.1
#> [9] RSQLite_3.53.3 png_0.1-9
#> [11] vctrs_0.7.3 reshape2_1.4.5
#> [13] stringr_1.6.0 pkgconfig_2.0.3
#> [15] MetaboCoreUtils_1.21.1 crayon_1.5.3
#> [17] fastmap_1.2.0 dbplyr_2.6.0
#> [19] XVector_0.53.0 rmarkdown_2.32
#> [21] purrr_1.2.2 bit_4.6.0
#> [23] xfun_0.61 MultiAssayExperiment_1.39.1
#> [25] cachem_1.1.0 jsonlite_2.0.0
#> [27] blob_1.3.0 DelayedArray_0.39.8
#> [29] parallel_4.6.1 cluster_2.1.8.3
#> [31] R6_2.6.1 bslib_0.12.0
#> [33] stringi_1.8.9 GenomicRanges_1.65.4
#> [35] jquerylib_0.1.4 Rcpp_1.1.2
#> [37] Seqinfo_1.3.2 bookdown_0.48
#> [39] assertthat_0.2.1 SummarizedExperiment_1.43.0
#> [41] knitr_1.52 IRanges_2.47.5
#> [43] BiocBaseUtils_1.15.1 Matrix_1.7-6
#> [45] igraph_2.3.4 tidyselect_1.2.1
#> [47] abind_1.4-8 yaml_2.3.12
#> [49] codetools_0.2-20 curl_8.1.0
#> [51] plyr_1.8.9 lattice_0.23-1
#> [53] tibble_3.3.1 Biobase_2.73.2
#> [55] withr_3.0.3 KEGGREST_1.53.6
#> [57] evaluate_1.0.5 BiocFileCache_3.3.0
#> [59] ExperimentHub_3.3.2 Biostrings_2.81.9
#> [61] pillar_1.11.1 BiocManager_1.30.27
#> [63] filelock_1.0.3 MatrixGenerics_1.25.0
#> [65] ncdf4_1.24 BiocVersion_3.24.0
#> [67] glue_1.8.1 lazyeval_0.2.3
#> [69] tools_4.6.1 AnnotationHub_4.3.2
#> [71] data.table_1.18.6.1 QFeatures_1.23.2
#> [73] mzR_2.47.1 fs_2.1.0
#> [75] grid_4.6.1 tidyr_1.3.2
#> [77] MsCoreUtils_1.25.4 AnnotationDbi_1.75.2
#> [79] duckdb_1.5.6 cli_3.6.6
#> [81] rappdirs_0.3.4 S4Arrays_1.13.2
#> [83] arrow_25.0.1 dplyr_1.2.1
#> [85] AnnotationFilter_1.37.0 sass_0.4.10
#> [87] digest_0.6.39 SparseArray_1.13.4
#> [89] memoise_2.0.1 htmltools_0.5.9
#> [91] lifecycle_1.0.5 httr_1.4.9
#> [93] bit64_4.8.6 MASS_7.3-66