Unit Testing
Testing Base Julia
Julia is under rapid development and has an extensive test suite to verify functionality across multiple platforms. If you build Julia from source, you can run this test suite with make test. In a binary install, you can run the test suite using Base.runtests().
Base.runtests — FunctionBase.runtests(tests=["all"]; ncores=ceil(Int, Sys.CPU_THREADS / 2),
exit_on_error=false, revise=false, [seed])Run the Julia unit tests listed in tests, which can be either a string or an array of strings, using ncores processors. If exit_on_error is false, when one test fails, all remaining tests in other files will still be run; they are otherwise discarded, when exit_on_error == true. If revise is true, the Revise package is used to load any modifications to Base or to the standard libraries before running the tests. If a seed is provided via the keyword argument, it is used to seed the global RNG in the context where the tests are run; otherwise the seed is chosen randomly.
Basic Unit Tests
The Test module provides simple unit testing functionality. Unit testing is a way to see if your code is correct by checking that the results are what you expect. It can be helpful to ensure your code still works after you make changes, and can be used when developing as a way of specifying the behaviors your code should have when complete. You may also want to look at the documentation for adding tests to your Julia Package.
Simple unit testing can be performed with the @test and @test_throws macros:
Test.@test — Macro@test ex
@test f(args...) key=val ...
@test ex broken=true
@test ex skip=trueTest that the expression ex evaluates to true. If executed inside a @testset, return a Pass Result if it does, a Fail Result if it is false, and an Error Result if it could not be evaluated. If executed outside a @testset, throw an exception instead of returning Fail or Error.
Examples
julia> @test true
Test Passed
julia> @test [1, 2] + [2, 1] == [3, 3]
Test PassedThe @test f(args...) key=val... form is equivalent to writing @test f(args..., key=val...) which can be useful when the expression is a call using infix syntax such as approximate comparisons:
julia> @test π ≈ 3.14 atol=0.01
Test PassedThis is equivalent to the uglier test @test ≈(π, 3.14, atol=0.01). It is an error to supply more than one expression unless the first is a call expression and the rest are assignments (k=v).
You can use any key for the key=val arguments, except for broken and skip, which have special meanings in the context of @test:
broken=condindicates a test that should pass but currently consistently fails whencond==true. Tests that the expressionexevaluates tofalseor causes an exception. Returns aBrokenResultif it does, or anErrorResultif the expression evaluates totrue. Regular@test exis evaluated whencond==false.skip=condmarks a test that should not be executed but should be included in test summary reporting asBroken, whencond==true. This can be useful for tests that intermittently fail, or tests of not-yet-implemented functionality. Regular@test exis evaluated whencond==false.
Examples
julia> @test 2 + 2 ≈ 6 atol=1 broken=true
Test Broken
Expression: ≈(2 + 2, 6, atol = 1)
julia> @test 2 + 2 ≈ 5 atol=1 broken=false
Test Passed
julia> @test 2 + 2 == 5 skip=true
Test Broken
Skipped: 2 + 2 == 5
julia> @test 2 + 2 == 4 skip=false
Test PassedThe broken and skip keyword arguments require at least Julia 1.7.
Test.@test_throws — Macro@test_throws exception exprTests that the expression expr throws exception. The exception may specify either a type, a string, regular expression, or list of strings occurring in the displayed error message, a matching function, or a value (which will be tested for equality by comparing fields). Note that @test_throws does not support a trailing keyword form.
The ability to specify anything other than a type or a value as exception requires Julia v1.8 or later.
Examples
julia> @test_throws BoundsError [1, 2, 3][4]
Test Passed
Thrown: BoundsError
julia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]
Test Passed
Thrown: DimensionMismatch
julia> @test_throws "Try sqrt(Complex" sqrt(-1)
Test Passed
Message: "DomainError with -1.0:\nsqrt was called with a negative real argument but will only return a complex result if called with a complex argument. Try sqrt(Complex(x))."In the final example, instead of matching a single string it could alternatively have been performed with:
["Try", "Complex"](a list of strings)r"Try sqrt\([Cc]omplex"(a regular expression)str -> occursin("complex", str)(a matching function)
For example, suppose we want to check our new function foo(x) works as expected:
julia> using Test
julia> foo(x) = length(x)^2
foo (generic function with 1 method)If the condition is true, a Pass is returned:
julia> @test foo("bar") == 9
Test Passed
julia> @test foo("fizz") >= 10
Test PassedIf the condition is false, then a Fail is returned and an exception is thrown:
julia> @test foo("f") == 20
Test Failed at none:1
Expression: foo("f") == 20
Evaluated: 1 == 20
ERROR: There was an error during testingIf the condition could not be evaluated because an exception was thrown, which occurs in this case because length is not defined for symbols, an Error object is returned and an exception is thrown:
julia> @test foo(:cat) == 1
Error During Test
Test threw an exception of type MethodError
Expression: foo(:cat) == 1
MethodError: no method matching length(::Symbol)
The function `length` exists, but no method is defined for this combination of argument types.
Closest candidates are:
length(::SimpleVector) at essentials.jl:256
length(::Base.MethodList) at reflection.jl:521
length(::MethodTable) at reflection.jl:597
...
Stacktrace:
[...]
ERROR: There was an error during testingIf we expect that evaluating an expression should throw an exception, then we can use @test_throws to check that this occurs:
julia> @test_throws MethodError foo(:cat)
Test Passed
Thrown: MethodErrorWorking with Test Sets
Typically a large number of tests are used to make sure functions work correctly over a range of inputs. In the event a test fails, the default behavior is to throw an exception immediately. However, it is normally preferable to run the rest of the tests first to get a better picture of how many errors there are in the code being tested.
The @testset will create a local scope of its own when running the tests in it.
The @testset macro can be used to group tests into sets. All the tests in a test set will be run, and at the end of the test set a summary will be printed. If any of the tests failed, or could not be evaluated due to an error, the test set will then throw a TestSetException.
Test.@testset — Macro@testset [CustomTestSet] [options...] ["description"] begin test_ex end
@testset [CustomTestSet] [options...] ["description $v"] for v in itr test_ex end
@testset [CustomTestSet] [options...] ["description $v, $w"] for v in itrv, w in itrw test_ex end
@testset [CustomTestSet] [options...] ["description"] test_func()
@testset let v = v, w = w; test_ex; endWith begin/end or function call
When @testset is used, with begin/end or a single function call, the macro starts a new test set in which to evaluate the given expression.
If no custom testset type is given it defaults to creating a DefaultTestSet. DefaultTestSet records all the results and, if there are any Fails or Errors, throws an exception at the end of the top-level (non-nested) test set, along with a summary of the test results.
Any custom testset type (subtype of AbstractTestSet) can be given and it will also be used for any nested @testset invocations. The given options are only applied to the test set where they are given. The default test set type accepts three boolean options:
verbose: iftrue, the result summary of the nested testsets is shown even when they all pass (the default isfalse).showtiming: iftrue, the duration of each displayed testset is shown (the default istrue).failfast: iftrue, any test failure or error will cause the testset and any child testsets to return immediately (the default isfalse). This can also be set globally via the env varJULIA_TEST_FAILFAST.
@testset test_func() requires at least Julia 1.8.
failfast requires at least Julia 1.9.
The description string accepts interpolation from the loop indices. If no description is provided, one is constructed based on the variables. If a function call is provided, its name will be used. Explicit description strings override this behavior.
By default the @testset macro will return the testset object itself, though this behavior can be customized in other testset types. If a for loop is used then the macro collects and returns a list of the return values of the finish method, which by default will return a list of the testset objects used in each iteration.
Before the execution of the body of a @testset, there is an implicit call to Random.seed!(seed) where seed is the current seed of the global RNG. Moreover, after the execution of the body, the state of the global RNG is restored to what it was before the @testset. This is meant to ease reproducibility in case of failure, and to allow seamless re-arrangements of @testsets regardless of their side-effect on the global RNG state.
Examples
julia> @testset "trigonometric identities" begin
θ = 2/3*π
@test sin(-θ) ≈ -sin(θ)
@test cos(-θ) ≈ cos(θ)
@test sin(2θ) ≈ 2*sin(θ)*cos(θ)
@test cos(2θ) ≈ cos(θ)^2 - sin(θ)^2
end;
Test Summary: | Pass Total Time
trigonometric identities | 4 4 0.2s@testset for
When @testset for is used, the macro starts a new test for each iteration of the provided loop. The semantics of each test set are otherwise identical to that of that begin/end case (as if used for each loop iteration).
@testset let
When @testset let is used, the macro starts a transparent test set with the given object added as a context object to any failing test contained therein. This is useful when performing a set of related tests on one larger object and it is desirable to print this larger object when any of the individual tests fail. Transparent test sets do not introduce additional levels of nesting in the test set hierarchy and are passed through directly to the parent test set (with the context object appended to any failing tests.)
@testset let requires at least Julia 1.9.
Multiple let assignments are supported since Julia 1.10.
Examples
julia> @testset let logi = log(im)
@test imag(logi) == π/2
@test !iszero(real(logi))
end
Test Failed at none:3
Expression: !(iszero(real(logi)))
Context: logi = 0.0 + 1.5707963267948966im
ERROR: There was an error during testing
julia> @testset let logi = log(im), op = !iszero
@test imag(logi) == π/2
@test op(real(logi))
end
Test Failed at none:3
Expression: op(real(logi))
Context: logi = 0.0 + 1.5707963267948966im
op = !iszero
ERROR: There was an error during testingTest.TestSetException — TypeTestSetExceptionThrown when a test set finishes and not all tests passed.
We can put our tests for the foo(x) function in a test set:
julia> @testset "Foo Tests" begin
@test foo("a") == 1
@test foo("ab") == 4
@test foo("abc") == 9
end;
Test Summary: | Pass Total Time
Foo Tests | 3 3 0.0sTest sets can also be nested:
julia> @testset "Foo Tests" begin
@testset "Animals" begin
@test foo("cat") == 9
@test foo("dog") == foo("cat")
end
@testset "Arrays $i" for i in 1:3
@test foo(zeros(i)) == i^2
@test foo(fill(1.0, i)) == i^2
end
end;
Test Summary: | Pass Total Time
Foo Tests | 8 8 0.0sAs well as call functions:
julia> f(x) = @test isone(x)
f (generic function with 1 method)
julia> @testset f(1);
Test Summary: | Pass Total Time
f | 1 1 0.0sThis can be used to allow for factorization of test sets, making it easier to run individual test sets by running the associated functions instead. Note that in the case of functions, the test set will be given the name of the called function. In the event that a nested test set has no failures, as happened here, it will be hidden in the summary, unless the verbose=true option is passed:
julia> @testset verbose = true "Foo Tests" begin
@testset "Animals" begin
@test foo("cat") == 9
@test foo("dog") == foo("cat")
end
@testset "Arrays $i" for i in 1:3
@test foo(zeros(i)) == i^2
@test foo(fill(1.0, i)) == i^2
end
end;
Test Summary: | Pass Total Time
Foo Tests | 8 8 0.0s
Animals | 2 2 0.0s
Arrays 1 | 2 2 0.0s
Arrays 2 | 2 2 0.0s
Arrays 3 | 2 2 0.0sIf we do have a test failure, only the details for the failed test sets will be shown:
julia> @testset "Foo Tests" begin
@testset "Animals" begin
@testset "Felines" begin
@test foo("cat") == 9
end
@testset "Canines" begin
@test foo("dog") == 9
end
end
@testset "Arrays" begin
@test foo(zeros(2)) == 4
@test foo(fill(1.0, 4)) == 15
end
end
Arrays: Test Failed
Expression: foo(fill(1.0, 4)) == 15
Evaluated: 16 == 15
[...]
Test Summary: | Pass Fail Total Time
Foo Tests | 3 1 4 0.0s
Animals | 2 2 0.0s
Arrays | 1 1 2 0.0s
ERROR: Some tests did not pass: 3 passed, 1 failed, 0 errored, 0 broken.Testing Log Statements
One can use the @test_logs macro to test log statements, or use a TestLogger.
Test.@test_logs — Macro@test_logs [log_patterns...] [keywords] expressionCollect a list of log records generated by expression using collect_test_logs, check that they match the sequence log_patterns, and return the value of expression. The keywords provide some simple filtering of log records: the min_level keyword controls the minimum log level which will be collected for the test, the match_mode keyword defines how matching will be performed (the default :all checks that all logs and patterns match pairwise; use :any to check that the pattern matches at least once somewhere in the sequence.)
The most useful log pattern is a simple tuple of the form (level,message). A different number of tuple elements may be used to match other log metadata, corresponding to the arguments to passed to AbstractLogger via the handle_message function: (level,message,module,group,id,file,line). Elements which are present will be matched pairwise with the log record fields using == by default, with the special cases that Symbols may be used for the standard log levels, and Regexs in the pattern will match string or Symbol fields using occursin.
Examples
Consider a function which logs a warning, and several debug messages:
function foo(n)
@info "Doing foo with n=$n"
for i=1:n
@debug "Iteration $i"
end
42
endWe can test the info message using
@test_logs (:info,"Doing foo with n=2") foo(2)If we also wanted to test the debug messages, these need to be enabled with the min_level keyword:
using Logging
@test_logs (:info,"Doing foo with n=2") (:debug,"Iteration 1") (:debug,"Iteration 2") min_level=Logging.Debug foo(2)If you want to test that some particular messages are generated while ignoring the rest, you can set the keyword match_mode=:any:
using Logging
@test_logs (:info,) (:debug,"Iteration 42") min_level=Logging.Debug match_mode=:any foo(100)The macro may be chained with @test to also test the returned value:
@test (@test_logs (:info,"Doing foo with n=2") foo(2)) == 42If you want to test for the absence of warnings, you can omit specifying log patterns and set the min_level accordingly:
# test that the expression logs no messages when the logger level is warn:
@test_logs min_level=Logging.Warn @info("Some information") # passes
@test_logs min_level=Logging.Warn @warn("Some information") # failsIf you want to test the absence of warnings (or error messages) in stderr which are not generated by @warn, see @test_nowarn.
Test.TestLogger — TypeTestLogger(; min_level=Info, catch_exceptions=false)Create a TestLogger which captures logged messages in its logs::Vector{LogRecord} field.
Set min_level to control the LogLevel, catch_exceptions for whether or not exceptions thrown as part of log event generation should be caught, and respect_maxlog for whether or not to follow the convention of logging messages with maxlog=n for some integer n at most n times.
See also: LogRecord.
Examples
julia> using Test, Logging
julia> f() = @info "Hi" number=5;
julia> test_logger = TestLogger();
julia> with_logger(test_logger) do
f()
@info "Bye!"
end
julia> @test test_logger.logs[1].message == "Hi"
Test Passed
julia> @test test_logger.logs[1].kwargs[:number] == 5
Test Passed
julia> @test test_logger.logs[2].message == "Bye!"
Test PassedTest.LogRecord — TypeLogRecordStores the results of a single log event. Fields:
level: theLogLevelof the log messagemessage: the textual content of the log message_module: the module of the log eventgroup: the logging group (by default, the name of the file containing the log event)id: the ID of the log eventfile: the file containing the log eventline: the line within the file of the log eventkwargs: any keyword arguments passed to the log event
Other Test Macros
As calculations on floating-point values can be imprecise, you can perform approximate equality checks using either @test a ≈ b (where ≈, typed via tab completion of \approx, is the isapprox function) or use isapprox directly.
julia> @test 1 ≈ 0.999999999
Test Passed
julia> @test 1 ≈ 0.999999
Test Failed at none:1
Expression: 1 ≈ 0.999999
Evaluated: 1 ≈ 0.999999
ERROR: There was an error during testingYou can specify relative and absolute tolerances by setting the rtol and atol keyword arguments of isapprox, respectively, after the ≈ comparison:
julia> @test 1 ≈ 0.999999 rtol=1e-5
Test PassedNote that this is not a specific feature of the ≈ but rather a general feature of the @test macro: @test a <op> b key=val is transformed by the macro into @test op(a, b, key=val). It is, however, particularly useful for ≈ tests.
Test.@inferred — Macro@inferred [AllowedType] f(x)Tests that the call expression f(x) returns a value of the same type inferred by the compiler. It is useful to check for type stability.
f(x) can be any call expression. Returns the result of f(x) if the types match, and an Error Result if it finds different types.
Optionally, AllowedType relaxes the test, by making it pass when either the type of f(x) matches the inferred type modulo AllowedType, or when the return type is a subtype of AllowedType. This is useful when testing type stability of functions returning a small union such as Union{Nothing, T} or Union{Missing, T}.
julia> f(a) = a > 1 ? 1 : 1.0
f (generic function with 1 method)
julia> typeof(f(2))
Int64
julia> @code_warntype f(2)
MethodInstance for f(::Int64)
from f(a) @ Main none:1
Arguments
#self#::Core.Const(f)
a::Int64
Body::UNION{FLOAT64, INT64}
1 ─ %1 = (a > 1)::Bool
└── goto #3 if not %1
2 ─ return 1
3 ─ return 1.0
julia> @inferred f(2)
ERROR: return type Int64 does not match inferred return type Union{Float64, Int64}
[...]
julia> @inferred max(1, 2)
2
julia> g(a) = a < 10 ? missing : 1.0
g (generic function with 1 method)
julia> @inferred g(20)
ERROR: return type Float64 does not match inferred return type Union{Missing, Float64}
[...]
julia> @inferred Missing g(20)
1.0
julia> h(a) = a < 10 ? missing : f(a)
h (generic function with 1 method)
julia> @inferred Missing h(20)
ERROR: return type Int64 does not match inferred return type Union{Missing, Float64, Int64}
[...]Test.@test_deprecated — Macro@test_deprecated [pattern] expressionWhen --depwarn=yes, test that expression emits a deprecation warning and return the value of expression. The log message string will be matched against pattern which defaults to r"deprecated"i.
When --depwarn=no, simply return the result of executing expression. When --depwarn=error, check that an ErrorException is thrown.
Examples
# Deprecated in julia 0.7
@test_deprecated num2hex(1)
# The returned value can be tested by chaining with @test:
@test (@test_deprecated num2hex(1)) == "0000000000000001"Test.@test_warn — Macro@test_warn msg exprTest whether evaluating expr results in stderr output that contains the msg string or matches the msg regular expression. If msg is a boolean function, tests whether msg(output) returns true. If msg is a tuple or array, checks that the error output contains/matches each item in msg. Returns the result of evaluating expr.
See also @test_nowarn to check for the absence of error output.
Note: Warnings generated by @warn cannot be tested with this macro. Use @test_logs instead.
Test.@test_nowarn — Macro@test_nowarn exprTest whether evaluating expr results in empty stderr output (no warnings or other messages). Returns the result of evaluating expr.
Note: The absence of warnings generated by @warn cannot be tested with this macro. Use @test_logs instead.
Broken Tests
If a test fails consistently it can be changed to use the @test_broken macro. This will denote the test as Broken if the test continues to fail and alerts the user via an Error if the test succeeds.
Test.@test_broken — Macro@test_broken ex
@test_broken f(args...) key=val ...Indicates a test that should pass but currently consistently fails. Tests that the expression ex evaluates to false or causes an exception. Returns a Broken Result if it does, or an Error Result if the expression evaluates to true. This is equivalent to @test ex broken=true.
The @test_broken f(args...) key=val... form works as for the @test macro.
Examples
julia> @test_broken 1 == 2
Test Broken
Expression: 1 == 2
julia> @test_broken 1 == 2 atol=0.1
Test Broken
Expression: ==(1, 2, atol = 0.1)@test_skip is also available to skip a test without evaluation, but counting the skipped test in the test set reporting. The test will not run but gives a Broken Result.
Test.@test_skip — Macro@test_skip ex
@test_skip f(args...) key=val ...Marks a test that should not be executed but should be included in test summary reporting as Broken. This can be useful for tests that intermittently fail, or tests of not-yet-implemented functionality. This is equivalent to @test ex skip=true.
The @test_skip f(args...) key=val... form works as for the @test macro.
Examples
julia> @test_skip 1 == 2
Test Broken
Skipped: 1 == 2
julia> @test_skip 1 == 2 atol=0.1
Test Broken
Skipped: ==(1, 2, atol = 0.1)Test result types
Test.Result — TypeTest.ResultAll tests produce a result object. This object may or may not be stored, depending on whether the test is part of a test set.
Test.Pass — TypeTest.Pass <: Test.ResultThe test condition was true, i.e. the expression evaluated to true or the correct exception was thrown.
Test.Fail — TypeTest.Fail <: Test.ResultThe test condition was false, i.e. the expression evaluated to false or the correct exception was not thrown.
Test.Error — TypeTest.Error <: Test.ResultThe test condition couldn't be evaluated due to an exception, or it evaluated to something other than a Bool. In the case of @test_broken it is used to indicate that an unexpected Pass Result occurred.
Test.Broken — TypeTest.Broken <: Test.ResultThe test condition is the expected (failed) result of a broken test, or was explicitly skipped with @test_skip.
Creating Custom AbstractTestSet Types
Packages can create their own AbstractTestSet subtypes by implementing the record and finish methods. The subtype should have a one-argument constructor taking a description string, with any options passed in as keyword arguments.
Test.record — Functionrecord(ts::AbstractTestSet, res::Result)Record a result to a testset. This function is called by the @testset infrastructure each time a contained @test macro completes, and is given the test result (which could be an Error). This will also be called with an Error if an exception is thrown inside the test block but outside of a @test context.
Test.finish — Functionfinish(ts::AbstractTestSet)Do any final processing necessary for the given testset. This is called by the @testset infrastructure after a test block executes.
Custom AbstractTestSet subtypes should call record on their parent (if there is one) to add themselves to the tree of test results. This might be implemented as:
if get_testset_depth() != 0
# Attach this test set to the parent test set
parent_ts = get_testset()
record(parent_ts, self)
return self
endTest takes responsibility for maintaining a stack of nested testsets as they are executed, but any result accumulation is the responsibility of the AbstractTestSet subtype. You can access this stack with the get_testset and get_testset_depth methods. Note that these functions are not exported.
Test.get_testset — Functionget_testset()Retrieve the active test set from the task's local storage. If no test set is active, use the fallback default test set.
Test.get_testset_depth — Functionget_testset_depth()Return the number of active test sets, not including the default test set
Test also makes sure that nested @testset invocations use the same AbstractTestSet subtype as their parent unless it is set explicitly. It does not propagate any properties of the testset. Option inheritance behavior can be implemented by packages using the stack infrastructure that Test provides.
Defining a basic AbstractTestSet subtype might look like:
import Test: Test, record, finish
using Test: AbstractTestSet, Result, Pass, Fail, Error
using Test: get_testset_depth, get_testset
struct CustomTestSet <: Test.AbstractTestSet
description::AbstractString
foo::Int
results::Vector
# constructor takes a description string and options keyword arguments
CustomTestSet(desc; foo=1) = new(desc, foo, [])
end
record(ts::CustomTestSet, child::AbstractTestSet) = push!(ts.results, child)
record(ts::CustomTestSet, res::Result) = push!(ts.results, res)
function finish(ts::CustomTestSet)
# just record if we're not the top-level parent
if get_testset_depth() > 0
record(get_testset(), ts)
return ts
end
# so the results are printed if we are at the top level
Test.print_test_results(ts)
return ts
endAnd using that testset looks like:
@testset CustomTestSet foo=4 "custom testset inner 2" begin
# this testset should inherit the type, but not the argument.
@testset "custom testset inner" begin
@test true
end
endIn order to use a custom testset and have the recorded results printed as part of any outer default testset, also define Test.get_test_counts. This might look like so:
using Test: AbstractTestSet, Pass, Fail, Error, Broken, get_test_counts, TestCounts, format_duration
function Test.get_test_counts(ts::CustomTestSet)
passes, fails, errors, broken = 0, 0, 0, 0
# cumulative results
c_passes, c_fails, c_errors, c_broken = 0, 0, 0, 0
for t in ts.results
# count up results
isa(t, Pass) && (passes += 1)
isa(t, Fail) && (fails += 1)
isa(t, Error) && (errors += 1)
isa(t, Broken) && (broken += 1)
# handle children
if isa(t, AbstractTestSet)
tc = get_test_counts(t)::TestCounts
c_passes += tc.passes + tc.cumulative_passes
c_fails += tc.fails + tc.cumulative_fails
c_errors += tc.errors + tc.cumulative_errors
c_broken += tc.broken + tc.cumulative_broken
end
end
# get a duration, if we have one
duration = format_duration(ts)
return TestCounts(true, passes, fails, errors, broken, c_passes, c_fails, c_errors, c_broken, duration)
endTest.TestCounts — TypeTestCountsHolds the state for recursively gathering the results of a test set for display purposes.
Fields:
customized: Whether the functionget_test_countswas customized for theAbstractTestSetthis counts object is for. If a custom method was defined, always passtrueto the constructor.passes: The number of passing@testinvocations.fails: The number of failing@testinvocations.errors: The number of erroring@testinvocations.broken: The number of broken@testinvocations.passes: The cumulative number of passing@testinvocations.fails: The cumulative number of failing@testinvocations.errors: The cumulative number of erroring@testinvocations.broken: The cumulative number of broken@testinvocations.duration: The total duration theAbstractTestSetin question ran for, as a formattedString.
Test.get_test_counts — Function" gettestcounts(::AbstractTestSet) -> TestCounts
Recursive function that counts the number of test results of each type directly in the testset, and totals across the child testsets.
Custom AbstractTestSet should implement this function to get their totals counted & displayed with DefaultTestSet as well.
If this is not implemented for a custom TestSet, the printing falls back to reporting x for failures and ?s for the duration.
Test.format_duration — Functionformat_duration(::AbstractTestSet)Return a formatted string for printing the duration the testset ran for.
If not defined, falls back to "?s".
Test.print_test_results — Functionprint_test_results(ts::AbstractTestSet, depth_pad=0)Print the results of an AbstractTestSet as a formatted table.
depth_pad refers to how much padding should be added in front of all output.
Called inside of Test.finish, if the finished testset is the topmost testset.
Test utilities
Test.GenericArray — TypeThe GenericArray can be used to test generic array APIs that program to the AbstractArray interface, in order to ensure that functions can work with array types besides the standard Array type.
Test.GenericDict — TypeThe GenericDict can be used to test generic dict APIs that program to the AbstractDict interface, in order to ensure that functions can work with associative types besides the standard Dict type.
Test.GenericOrder — TypeThe GenericOrder can be used to test APIs for their support of generic ordered types.
Test.GenericSet — TypeThe GenericSet can be used to test generic set APIs that program to the AbstractSet interface, in order to ensure that functions can work with set types besides the standard Set and BitSet types.
Test.GenericString — TypeThe GenericString can be used to test generic string APIs that program to the AbstractString interface, in order to ensure that functions can work with string types besides the standard String type.
Test.detect_ambiguities — Functiondetect_ambiguities(mod1, mod2...; recursive=false,
ambiguous_bottom=false,
allowed_undefineds=nothing)Return a vector of (Method,Method) pairs of ambiguous methods defined in the specified modules. Use recursive=true to test in all submodules.
ambiguous_bottom controls whether ambiguities triggered only by Union{} type parameters are included; in most cases you probably want to set this to false. See Base.isambiguous.
See Test.detect_unbound_args for an explanation of allowed_undefineds.
allowed_undefineds requires at least Julia 1.8.
Test.detect_unbound_args — Functiondetect_unbound_args(mod1, mod2...; recursive=false, allowed_undefineds=nothing)Return a vector of Methods which may have unbound type parameters. Use recursive=true to test in all submodules.
By default, any undefined symbols trigger a warning. This warning can be suppressed by supplying a collection of GlobalRefs for which the warning can be skipped. For example, setting
allowed_undefineds = Set([GlobalRef(Base, :active_repl),
GlobalRef(Base, :active_repl_backend)])would suppress warnings about Base.active_repl and Base.active_repl_backend.
allowed_undefineds requires at least Julia 1.8.
Workflow for Testing Packages
Using the tools available to us in the previous sections, here is a potential workflow of creating a package and adding tests to it.
Generating an Example Package
For this workflow, we will create a package called Example:
pkg> generate Example
shell> cd Example
shell> mkdir test
pkg> activate .Creating Sample Functions
The number one requirement for testing a package is to have functionality to test. For that, we will add some simple functions to Example that we can test. Add the following to src/Example.jl:
module Example
function greet()
"Hello world!"
end
function simple_add(a, b)
a + b
end
function type_multiply(a::Float64, b::Float64)
a * b
end
export greet, simple_add, type_multiply
endCreating a Test Environment
From within the root of the Example package, navigate to the test directory, activate a new environment there, and add the Test package to the environment:
shell> cd test
pkg> activate .
(test) pkg> add TestTesting Our Package
Now, we are ready to add tests to Example. It is standard practice to create a file within the test directory called runtests.jl which contains the test sets we want to run. Go ahead and create that file within the test directory and add the following code to it:
using Example
using Test
@testset "Example tests" begin
@testset "Math tests" begin
include("math_tests.jl")
end
@testset "Greeting tests" begin
include("greeting_tests.jl")
end
endWe will need to create those two included files, math_tests.jl and greeting_tests.jl, and add some tests to them.
Note: Notice how we did not have to specify add
Exampleinto thetestenvironment'sProject.toml. This is a benefit of Julia's testing system that you could read about more here.
Writing Tests for math_tests.jl
Using our knowledge of Test.jl, here are some example tests we could add to math_tests.jl:
@testset "Testset 1" begin
@test 2 == simple_add(1, 1)
@test 3.5 == simple_add(1, 2.5)
@test_throws MethodError simple_add(1, "A")
@test_throws MethodError simple_add(1, 2, 3)
end
@testset "Testset 2" begin
@test 1.0 == type_multiply(1.0, 1.0)
@test isa(type_multiply(2.0, 2.0), Float64)
@test_throws MethodError type_multiply(1, 2.5)
endWriting Tests for greeting_tests.jl
Using our knowledge of Test.jl, here are some example tests we could add to greeting_tests.jl:
@testset "Testset 3" begin
@test "Hello world!" == greet()
@test_throws MethodError greet("Antonia")
endTesting Our Package
Now that we have added our tests and our runtests.jl script in test, we can test our Example package by going back to the root of the Example package environment and reactivating the Example environment:
shell> cd ..
pkg> activate .From there, we can finally run our test suite as follows:
(Example) pkg> test
Testing Example
Status `/tmp/jl_Yngpvy/Project.toml`
[fa318bd2] Example v0.1.0 `/home/src/Projects/tmp/errata/Example`
[8dfed614] Test `@stdlib/Test`
Status `/tmp/jl_Yngpvy/Manifest.toml`
[fa318bd2] Example v0.1.0 `/home/src/Projects/tmp/errata/Example`
[2a0f44e3] Base64 `@stdlib/Base64`
[b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
[56ddb016] Logging `@stdlib/Logging`
[d6f4376e] Markdown `@stdlib/Markdown`
[9a3f8284] Random `@stdlib/Random`
[ea8e919c] SHA `@stdlib/SHA`
[9e88b42a] Serialization `@stdlib/Serialization`
[8dfed614] Test `@stdlib/Test`
Testing Running tests...
Test Summary: | Pass Total
Example tests | 9 9
Testing Example tests passedAnd if all went correctly, you should see a similar output as above. Using Test.jl, more complicated tests can be added for packages but this should ideally point developers in the direction of how to get started with testing their own created packages.
Code Coverage
Code coverage tracking during tests can be enabled using the pkg> test --coverage flag (or at a lower level using the --code-coverage julia arg). This is on by default in the julia-runtest GitHub action.
To evaluate coverage either manually inspect the .cov files that are generated beside the source files locally, or in CI use the julia-processcoverage GitHub action.
Since Julia 1.11, coverage is not collected during the package precompilation phase.