Cosmos: single-property projection no longer implicitly filters undefined

[Copilot]

The Cosmos provider translated single-property anonymous/dto/scalar projections of a nested property as SELECT VALUE prop, which causes Cosmos to silently drop documents where the path is undefined (e.g. an optional navigation is null). Multi-property projections instead throw Nullable object must have a value for undefined non-nullable values. Behavior should be consistent based on projection syntax, not vary with the number of selected properties.

// Single property — previously filtered the null-Associate document out entirely
context.Entities.Select(x => new { x.Associate.NestedAssociate.Id });

// Multiple properties — threw "Nullable object must have a value"
context.Entities.Select(x => new { x.Associate.NestedAssociate.Id, x.Associate.NestedAssociate.String });

After this change both forms behave the same: undefined surfaces consistently (throws for non-nullable, yields null for nullable).

Changes

• SelectExpression.ApplyProjection(bool clientProjection = false): when the projection is a single nested-scalar VALUE projection (a ScalarAccessExpression whose Object is an ObjectAccessExpression), demote it to an object projection — emitting SELECT c["a"]["b"] instead of SELECT VALUE c["a"]["b"] — so the document is retained and undefined reaches the shaper.
• CosmosQueryTranslationPostprocessor: passes clientProjection: true so the demotion applies only to the top-level projection. Subqueries and collection projections keep VALUE semantics, which their shaping depends on.
• Top-level scalars accessed directly off the root (x.Name, where Object is an ObjectReferenceExpression) are intentionally left as VALUE projections — they cannot be undefined-by-nesting.
• The optimal SELECT VALUE projection is preserved — and the now-redundant guards are removed — when the predicate already guarantees the projected path cannot be undefined. The optimization is applied when the single nested-scalar projection is created (in ApplyProjection): a new TryMakeNestedScalarProjectionDefined helper walks the top-level AND-conjuncts of the Where predicate and, when the projected path is guaranteed defined, keeps the VALUE projection and rewrites the predicate. Pure definedness guards (IS_DEFINED(path) or path != null) on the scalar or any ancestor object along its path are dropped, since the VALUE projection already filters those rows; value comparisons that merely imply definedness (e.g. ...["Country"] = "USA") and any unrelated conjuncts are kept, with the projection still marked scalar for consistency. Navigation null-checks (x.Associate.NestedAssociate != null) compare a StructuralTypeProjectionExpression whose .Object is the ObjectAccessExpression, so the helper unwraps it. As a result, guarded queries like Where(x => x.Associate != null && x.Associate.NestedAssociate != null).Select(x => x.Associate.NestedAssociate.Id) and the EF.Functions.IsDefined equivalent emit SELECT VALUE c["Associate"]["NestedAssociate"]["Id"] with the redundant WHERE removed.

Tests

Updated Cosmos baselines/overrides where the generated SQL drops VALUE and where the new behavior throws instead of filtering: ComplexProperties/OwnedNavigations projection suites, OwnedQuery, JsonQuery, ComplexType, and ComplexTypeToJsonProperty. The issue's single-member anonymous case is covered by Non_nullable_property_through_optional_navigation, which now throws InvalidOperationException matching the relational baseline.

Added Select_nested_scalar_guarded_by_navigation_predicate_uses_VALUE and Select_nested_scalar_guarded_by_IsDefined_uses_VALUE to the OwnedNavigations and ComplexProperties projection suites, asserting that guarded queries keep the optimal SELECT VALUE projection with the redundant definedness guards dropped from the WHERE. The Navigation_rewrite_on_owned_reference_projecting_scalar baseline (guarded by a = "USA" value comparison) keeps its SELECT VALUE projection and its WHERE unchanged.

[AndriySvyryd]

@copilot Instead of checking this on the translated SqlExpression this should be done before, when a Projection of the matching shape is created you try to apply this optimization and if the query matches, remove the filter predicates and mark the projection as scalar (meaning that it cannot be undefined, so it can be translated to VALUE ...)

[Copilot]

To make sure I build the right thing in one pass, here's the plan I'd follow — please confirm or correct:

1. Location — move the logic out of ApplyProjection (postprocessor) into the translation phase where the single nested-scalar projection mapping is created (CosmosProjectionBindingExpressionVisitor / TranslateSelect), so the matching-shape detection happens when the projection is built rather than by re-walking the final Predicate.
2. Optimization when the shape matches — when the projection is a single scalar nested in an embedded object (ScalarAccessExpression whose Object is an ObjectAccessExpression) and the predicate guarantees that path is defined: drop the now-redundant definedness-guard conjuncts (IS_DEFINED(path) and ancestor != null along the path chain) from the predicate, and mark the projection scalar so it stays SELECT VALUE ....
3. Default — otherwise the matching-shape projection becomes an object projection (SELECT c[...]) so undefined surfaces consistently (the main fix).

Two things I want to confirm:

• For guards that imply definedness but aren't pure null/IS_DEFINED checks and so can't be removed (e.g. ...["Country"] = "USA" in Navigation_rewrite_on_owned_reference_projecting_scalar): keep the predicate as-is but still mark the projection scalar (VALUE)? That preserves the current optimal baseline.
• "remove the filter predicates" — drop only the guard conjunct(s) on the projected path/ancestors and keep any unrelated conjuncts, correct?

If that matches your intent I'll implement it; the two new tests' baselines will lose their WHERE (replaced by VALUE-only filtering).

[AndriySvyryd]

@copilot For guards that imply definedness mark the projection scalar for consistency even though the result from the query would be the same either way.
Right, drop only the guard conjunct(s) on the projected path/ancestors and keep any unrelated conjuncts. Don't drop the guard conjunct(s) if they affect the unrelated conjuncts.

[Copilot]

Done in the latest commit. The optimization is now applied when the single nested-scalar projection is created (SelectExpression.ApplyProjection): TryMakeNestedScalarProjectionDefined walks the top-level AND-conjuncts and, when the projected path is guaranteed defined, drops the redundant definedness guards (IS_DEFINED(path) / path != null on the scalar or any ancestor object) and keeps the SELECT VALUE projection. Value comparisons that merely imply definedness (e.g. ...["Country"] = "USA") and any unrelated conjuncts are retained; otherwise the projection is demoted to an object projection. The two guarded-projection baselines now have their WHERE dropped accordingly.