diff --git a/_quarto.yml b/_quarto.yml index 9319e5739..af4e2ab63 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -90,6 +90,7 @@ website: - usage/varnamedtuple/index.qmd - usage/vectorisation/index.qmd - usage/external-samplers/index.qmd + - usage/debugging/index.qmd - usage/troubleshooting/index.qmd - section: "Tutorials" @@ -202,6 +203,7 @@ seasonal-time-series: tutorials/bayesian-time-series-analysis usage-automatic-differentiation: usage/automatic-differentiation usage-custom-distribution: usage/custom-distribution +usage-debugging: usage/debugging usage-dynamichmc: usage/dynamichmc usage-external-samplers: usage/external-samplers usage-external-likelihoods: usage/external-likelihoods diff --git a/contributing/start-contributing/index.qmd b/contributing/start-contributing/index.qmd index d10bcc224..574e47e08 100644 --- a/contributing/start-contributing/index.qmd +++ b/contributing/start-contributing/index.qmd @@ -15,7 +15,7 @@ If you're not sure where to begin, feel free to ask on any issue, or come say he - **Bug reports and edge cases:** If you run into something that doesn't work, please [open an issue](https://github.com/TuringLang/Turing.jl/issues)! If you aren't sure which repository to post on, don't worry: just post it on the main Turing.jl repo and we can triage it. We're especially keen on finding cases where automatic differentiation fails. - If you find a model that doesn't work with AD, please report it, and we can either help to report it upstream, and/or add it to [the ADTests website](https://turinglang.org/ADTests/). + If you find a model that doesn't work with AD, please report it, and we can help to report it upstream. - **Tutorials and example models:** If you have an interesting model that you think others could learn from, we'd love to hear about it! You don't have to write a full docs page immediately; just open an issue and we'd be very happy to chat about it. diff --git a/core-functionality/index.qmd b/core-functionality/index.qmd index 641103540..b8577fbe1 100755 --- a/core-functionality/index.qmd +++ b/core-functionality/index.qmd @@ -587,7 +587,7 @@ The default AD backend is [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff Three other backends are also supported: [Mooncake](https://github.com/chalk-lab/Mooncake.jl), [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl), and [Enzyme](https://github.com/EnzymeAD/Enzyme.jl). These require the user to explicitly load them, for example `import Mooncake` next to `using Turing`. -For more information on Turing's automatic differentiation backend, please see the [Automatic Differentiation]({{}}) article as well as the [ADTests website](https://turinglang.org/ADTests/), where a number of AD backends (not just those above) are tested against Turing.jl. +For more information on Turing's automatic differentiation backend, please see the [Automatic Differentiation]({{}}) article. #### Progress Logging diff --git a/faq/index.qmd b/faq/index.qmd index 145edbf61..23f756212 100644 --- a/faq/index.qmd +++ b/faq/index.qmd @@ -82,7 +82,7 @@ model = setthreadsafe(f(y), true) - **Observe statements**: Generally safe to use in threaded loops - **Assume statements** (sampling statements): Often crash unpredictably or produce incorrect results -- **AD backend compatibility**: Many AD backends don't support threading. Check the [multithreaded column in ADTests](https://turinglang.org/ADTests/) for compatibility +- **AD backend compatibility**: Many AD backends don't support threading. See the [Threadsafe Evaluation]({{< meta usage-threadsafe-evaluation >}}) page for which backends work reliably ## How do I check the type stability of my Turing model? @@ -95,6 +95,7 @@ Type stability is crucial for performance. Check out: For debugging both statistical and syntactical issues: +- [Debugging Models]({{< meta usage-debugging >}}) - tools for checking model validity, type stability, and gradients - [Troubleshooting Guide]({{< meta usage-troubleshooting >}}) - common errors and their solutions - For more advanced debugging, DynamicPPL provides [the `DynamicPPL.DebugUtils` module](https://turinglang.org/DynamicPPL.jl/stable/api/#Debugging-Utilities) for inspecting model internals @@ -139,7 +140,6 @@ The choice of AD backend can significantly impact performance. See: - [Automatic Differentiation Guide]({{< meta usage-automatic-differentiation >}}) - comprehensive comparison of ForwardDiff, Mooncake, ReverseDiff, and other backends - [Performance Tips]({{< meta usage-performance-tips >}}#choose-your-ad-backend) - quick guide on choosing backends -- [AD Backend Benchmarks](https://turinglang.org/ADTests/) - performance comparisons across various models ## I changed one line of my model and now it's so much slower; why? diff --git a/usage/automatic-differentiation/index.qmd b/usage/automatic-differentiation/index.qmd index 754e28a35..428b1981d 100755 --- a/usage/automatic-differentiation/index.qmd +++ b/usage/automatic-differentiation/index.qmd @@ -65,14 +65,6 @@ AbstractPPL's interface provides a `prepare` step that performs a one-time setup Turing will automatically perform this preparation for you when calling functions such as `sample` or `vi`, so you do not need to worry about this step. ::: -### ADTests - -Before describing how to choose the best AD backend for your model, we should mention that we also publish a table of benchmarks for various models and AD backends in [the ADTests website](https://turinglang.org/ADTests/). -These models aim to capture a variety of different features of Turing.jl and Julia in general, so that you can see which AD backends may be compatible with your model. -Benchmarks are also included, although it should be noted that many of the models in ADTests are small and thus the timings may not be representative of larger, real-life models. - -If you have suggestions for other models to include, please do let us know by [creating an issue on GitHub](https://github.com/TuringLang/ADTests/issues/new)! - ### The Best AD Backend for Your Model Given the number of possible backends, how do you choose the best one for your model? diff --git a/usage/debugging/index.qmd b/usage/debugging/index.qmd new file mode 100644 index 000000000..a9fe0fb03 --- /dev/null +++ b/usage/debugging/index.qmd @@ -0,0 +1,89 @@ +--- +title: Debugging Models +engine: julia +--- + +```{julia} +#| echo: false +#| output: false +using Pkg; +Pkg.instantiate(); +``` + +This page describes the tools available for checking that a Turing model is defined correctly. +It complements the [Troubleshooting]({{}}) page, which explains specific error messages: the tools here help you find problems before (or without) hitting an error. + +```{julia} +using Turing +``` + +## Checking model validity with `check_model` + +`DynamicPPL.check_model` evaluates a model once and checks it for common problems. +It returns `true` if the check passes, and `false` (with warnings explaining the problem) otherwise. + +Specifically, it checks for: + +- the same variable, or overlapping variables (such as `x` and `x[1]`), appearing on the left-hand side of more than one tilde statement; +- `NaN` values on the left-hand side of observe statements; +- with the keyword argument `fail_if_discrete=true`, the presence of discrete random variables, which gradient-based samplers such as HMC and NUTS cannot handle. + +It is equally important to know what it does not catch. +The model is evaluated a single time, with values drawn from the prior, so problems that only occur for some parameter values (for example, in a branch that this particular evaluation does not take) can be missed. +It also checks only the structure of the model, not its statistical soundness: it cannot tell you that a prior is unreasonable or that parameters are not identifiable. +Type instability and incorrect gradients are separate concerns, covered by the checks in the following sections. + +For example, this model observes a `NaN` value, which usually indicates a bug in the data preparation: + +```{julia} +using DynamicPPL: check_model + +@model function gaussian(y) + x ~ Normal() + y ~ Normal(x) +end + +check_model(gaussian(NaN)) +``` + +whereas the same model with valid data passes: + +```{julia} +check_model(gaussian(1.5)) +``` + +Turing runs this check automatically at the start of `sample` (and the mode estimation functions). +For gradient-based samplers, this includes the discrete-variable check. +If the check reports a false positive for your model, you can disable it by passing `check_model=false` to `sample`. + +The keyword argument `error_on_failure=true` makes `check_model` throw an error instead of returning `false`, which is useful in scripts and tests. + +## Checking type stability + +Type instability is one of the most common reasons for a model being slower than expected. +`DynamicPPL.DebugUtils.model_warntype(model)` works like `@code_warntype`, but for an entire model: types that the compiler cannot infer are marked in red in the output. +The [Performance Tips]({{}}#ensure-that-types-in-your-model-can-be-inferred) page has a worked example, along with advice on how to fix the instabilities it finds. +There is also `DynamicPPL.DebugUtils.model_typed`, which returns the typed representation instead of printing it. + +## Checking gradients + +If a model samples poorly with a gradient-based sampler such as NUTS, it is worth checking whether the AD backend computes correct gradients for it. +`DynamicPPL.TestUtils.AD.run_ad` evaluates the gradient of a model with a given backend and, by default, tests it against a reference backend: + +```{julia} +using ADTypes: AutoReverseDiff +using DynamicPPL.TestUtils.AD: run_ad +using ReverseDiff + +run_ad(gaussian(1.5), AutoReverseDiff()) +``` + +If the gradient is incorrect, this errors. +For debugging `NaN` or infinite gradients, see the [Troubleshooting]({{}}) page. + +## Further help + +This page covers the most commonly useful tools. +DynamicPPL provides a few more, such as `DynamicPPL.has_static_constraints`; the full list is in [the DynamicPPL API documentation](https://turinglang.org/DynamicPPL.jl/stable/api/#Debugging-Utilities). + +If these tools do not surface the problem, feel free to [open an issue](https://github.com/TuringLang/Turing.jl/issues), ideally with the smallest model that reproduces it. diff --git a/usage/performance-tips/index.qmd b/usage/performance-tips/index.qmd index 854cac645..b486b2b75 100755 --- a/usage/performance-tips/index.qmd +++ b/usage/performance-tips/index.qmd @@ -15,6 +15,52 @@ Pkg.instantiate(); This section briefly summarises a few common techniques to ensure good performance when using Turing. We refer to [the Julia documentation](https://docs.julialang.org/en/v1/manual/performance-tips/index.html) for general techniques to ensure good performance of Julia programs. +## Benchmark and profile your model + +Before changing your model for performance reasons, it is worth measuring where the time actually goes. +When sampling with a gradient-based method such as NUTS, almost all of the time is spent evaluating the log density of the model and its gradient. +You can time both with `DynamicPPL.TestUtils.AD.run_ad`: + +```{julia} +using Turing +using ADTypes: AutoReverseDiff +using DynamicPPL.TestUtils.AD: run_ad +using ReverseDiff + +@model function bmodel(x) + m ~ Normal() + s ~ truncated(Normal(); lower=0) + return x ~ MvNormal(fill(m, length(x)), s^2 * I) +end + +result = run_ad(bmodel(randn(100)), AutoReverseDiff(); benchmark=true) +result.primal_time, result.grad_time +``` + +Both times are median seconds per evaluation. +If the gradient takes many times longer than the log density itself, the AD backend is the thing to change: see [Choose your AD backend](#choose-your-ad-backend) below, and the [Automatic Differentiation]({{}}) page for how to compare several backends on your model. +If the log density itself is slow, the model is the thing to change, and the remaining sections on this page apply. + +To find out where the time goes within a single evaluation, you can use [Julia's built-in profiler](https://docs.julialang.org/en/v1/manual/profile/) on the model's log density: + +```{julia} +#| eval: false +using DynamicPPL, LogDensityProblems, Profile + +model = bmodel(randn(100)) +ldf = LogDensityFunction(model, getlogjoint_internal, LinkAll(); adtype=AutoReverseDiff()) +x = rand(ldf) + +Profile.@profile for _ in 1:10_000 + LogDensityProblems.logdensity_and_gradient(ldf, x) +end +Profile.print(; mincount=10) +``` + +This profiles the model evaluation and the gradient computation together, which is what a sampler actually runs. +We do not run the profiler on this page because its output is long and machine-specific: `Profile.print` shows a tree of function calls, each annotated with the number of samples in which it appeared, and the hot spots are the lines with the largest counts (`mincount=10` hides rarely-seen calls). +Graphical viewers such as `@profview` in VS Code, [ProfileView.jl](https://github.com/timholy/ProfileView.jl), or [PProf.jl](https://github.com/JuliaPerf/PProf.jl) make the output much easier to read than `Profile.print`. + ## Use multivariate distributions It is generally preferable to use multivariate distributions if possible. diff --git a/usage/threadsafe-evaluation/index.qmd b/usage/threadsafe-evaluation/index.qmd index c45e1f2cc..18fb66d72 100755 --- a/usage/threadsafe-evaluation/index.qmd +++ b/usage/threadsafe-evaluation/index.qmd @@ -242,8 +242,6 @@ Note that for Enzyme, you should use a relatively recent version (at least v0.13 In contrast, ReverseDiff sometimes gives right results, but quite often gives incorrect gradients. Mooncake [currently does not support multithreading at all](https://github.com/chalk-lab/Mooncake.jl/issues/570). -For more details you can take a look at [the `threaded_...` models on ADTests](https://turinglang.org/ADTests/). - ## Under the hood :::{.callout-note}