Skip to content

fix sugar expression lifetimes, aliased assignment, and operand lengths - #1513

Open
kevinushey wants to merge 11 commits into
masterfrom
bugfix/sugar-soundness
Open

kevinushey wants to merge 11 commits into
masterfrom
bugfix/sugar-soundness

Conversation

@kevinushey

@kevinushey kevinushey commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1512.

This fixes the ways sugar expressions could silently give wrong results (or crash) described in #1512. It's split into commits by problem; each commit's message has the details.

1. Stored expressions no longer dangle

Sugar nodes held every operand by const&, including temporary sub-expressions, so an expression kept past the statement that built it referred to destroyed objects. auto e = x + x * 2.0; return e; returned garbage, and auto e = -(x * 2.0); segfaulted R.

Nodes now hold their operands through a new traits::sugar_operand<T>: objects that own an R object (vectors, matrices and classes derived from them) by reference, and everything else (nested expressions, views such as MatrixColumn, scalars in rep()) by value. auto e = x * 2.0 + y; is now safe. An expression built on a temporary vector, e.g. auto e = clone(x) + 1.0;, can still dangle; that is documented in the sugar vignette.

2. Aliased in-place assignment is evaluated correctly

Assigning a same-length expression to a vector, or any expression to a range, row or column, writes into existing storage while the expression is still reading. Expressions that read other positions of that storage gave wrong results: x = rev(x) gave 5 4 3 4 5, and the shift x[Range(1, n - 1)] = head(x, n - 1) gave 1 1 1 1 1.

Assignment still happens in place (so attributes are kept and the caller's object is still modified, as before), but only expressions known to be safe are written directly:

  • Nodes whose element i depends only on element i of their operands declare them with typedef traits::elementwise_operands<...> rcpp_elementwise. That covers arithmetic, comparisons, !, &, |, math functions, ifelse(), pmin()/pmax(), is_na() and friends, and the d/p/q functions.
  • Views declare what kind of view they are (rcpp_view). Whether reading one in place is safe depends on the target:
    • a whole vector can read any view;
    • a column can read other columns;
    • a row can read other rows;
    • a range can't read any view.
  • Anything else (e.g. rev(), head(), sapply(), whose function may read the target) is evaluated into a temporary first.

3. Operand lengths are checked

Operations on two or more vectors took their length from one operand, so a shorter operand was read past its end (with only the bounds warning) and a longer one was silently truncated. These nodes now check their operands' lengths once, when they are constructed, and signal an error on a mismatch, matching the vignette's documented requirement that operands have the same size. Vectors assigned into a range, row or column are checked the same way. Several of these constructors already carried FIXME/TODO notes asking for exactly this check.

This is the user-visible behavior change to watch for in reverse dependencies: code that relied on the silent truncation (a longer right-hand operand) now errors.

4. !, && and || on single logical results compile, and handle NA like R

None of !all(x), all(x) && any(y), all(x) || any(y), all(x) && true or all(x) || true compiled on master:

  • The combinators called the non-const get() through const references.
  • The &&/|| overloads for two results declared their second parameter with the first one's template arguments, so they could never be deduced (leaving an ambiguity between the bool overloads).
  • Or_SingleLogicalResult_bool named the And class as its CRTP base.

The combinators now hold their operands by value like other sugar nodes, so auto b = !(all(x) && any(y)); is also safe. Since this code had never run, its logic had a bug too: the general &&/|| treated an NA on the left as final, so NA && FALSE gave NA and NA || TRUE gave NA. They now follow R's three-valued logic.

Performance

Same functions compiled against master vs this branch (R 4.6.1, arm64 macOS); results are identical.

code master this PR
x + y (1e7) 24.8 ms 24.9 ms
x * y + 2.0 * x - y (1e7) 52.2 ms 50.2 ms
ifelse(x > 0.5, x, -x) (1e7) 86.1 ms 82.2 ms
x = x * c + d in place, 10 times (1e7) 61.6 ms 41.2 ms
z = x * y + x, n = 10, 1e6 times 29.0 ms 29.6 ms
m(_, j) = m(_, j) * 2.0 for each column (1000 x 1000) 0.20 ms 0.20 ms

Two things were needed to keep these at parity:

  • Target-aware view rule. Without the per-target rule for views, the column update above went through a temporary and was ~4.75x slower.
  • Out-of-line error path. With the length check's error path inlined, the compiler stopped propagating the constant multiplier into that loop, and it vectorized half as wide.

Expressions that aren't elementwise (e.g. x = rev(x)) now pay for a temporary when assigned in place, which is what makes them correct.

Other changes

  • SugarBlock_3_VVV declared its third operand with the second operand's type; fixed.
  • MatrixBase gains a const get_ref(), matching VectorBase.
  • The sugar vignette gains a section on operand lengths, storing expressions and assignment, and its Sapply example uses sugar_operand. I edited only vignettes/rmd/Rcpp-sugar.Rmd; the pre-built PDF hasn't been regenerated.

Testing

New tests are in inst/tinytest/cpp/sugar_expressions.cpp, run from test_sugar.R. They cover:

  • stored expressions of each kind of node, after overwriting the stack;
  • each aliasing case above, including row/column cross-reads that must go through a temporary;
  • which expressions are written in place for each kind of target;
  • length mismatches in both directions;
  • !, && and || on single logical results, against R's own operators for every combination of TRUE, FALSE and NA (including the specializations for operands known not to be NA, and the bool overloads).

On master, the lifetime tests return wrong values or crash R, and the aliasing tests fail. The full suite passes locally with RunAllRcppTests=yes (1732 tests, 0 failures). All test sources also compile with -std=c++11, and introduce no new warnings under -Wall -Wextra -Wconversion other than two -Wsign-conversion notes (clang-only; gcc's -Wconversion doesn't enable those for C++) where rowSums()/diag() now call Matrix::operator()(size_t, size_t) directly.

Sugar nodes held every operand by const reference, so an expression kept past the full-expression that built it (e.g. in an auto variable) referred to destroyed temporaries. Nodes now hold nested expressions and views by value, and only objects that own an R object (vectors, matrices) by reference.

Part of #1512.
Assigning a same-length sugar expression to a vector, or any expression to a range, matrix row or matrix column, writes into existing storage while the expression is still being read. When the expression read from that storage at other positions (e.g. `x = rev(x)`, `x[Range(1, n - 1)] = head(x, n - 1)`), the result was silently wrong.

Elementwise nodes now declare themselves via a nested `rcpp_elementwise` type; those are still written in place, and everything else is evaluated into a temporary first.

Part of #1512.
Sugar operations on two or more vectors took their length from one operand and never compared it with the others, so a shorter operand was read past its end and a longer one was silently truncated. Likewise for vectors assigned into a range, matrix row or matrix column. These now signal an error, matching the documented requirement that operands have the same size.

Part of #1512.
Evaluating every expression that reads a view into a temporary made updates like `m(_, j) = m(_, j) * 2` several times slower. Whether reading a view in place is safe depends on where the result is written: a column can read other columns of the same matrix (they coincide or are disjoint), a row other rows, and a whole vector any view, but a range can't read another range. Sugar expressions now list their elementwise operands, views declare what kind of view they are, and each write site checks the expression against its own kind of target.

Part of #1512.
With the error path inlined, the compiler could no longer propagate constants into the loops following these checks (e.g. `m(_, j) = m(_, j) * 2.0` lost its constant multiplier and vectorized half as wide).
None of `!all(x)`, `all(x) && any(y)`, `all(x) || any(y)`, `all(x) && true` or `all(x) || true` compiled: the combinators called the non-const get() through const references, the && and || overloads for two results declared their second parameter with the first one's template arguments (so they could never be deduced), and Or_SingleLogicalResult_bool named the And class as its CRTP base.

The combinators now hold their operands by value, like other sugar expressions, and the general && and || follow R's three-valued logic (NA && FALSE is FALSE, NA || TRUE is TRUE).
Also mark Range as elementwise, include sugar_operand.h from
is_elementwise.h, and drop stale FIXME comments in ifelse.h.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sugar expressions give wrong results under aliasing, length mismatch, and captured temporaries

1 participant