fix(index): a Text field on a numeric column no longer makes the contentlet unindexable (#37272)

CLAUDE.md

@@ -127,6 +127,7 @@ How it's built + upgrade re-apply notes: [.specify/CUSTOMIZATIONS.md](.specify/C
 - [Security Patterns](docs/backend/SECURITY_BACKEND.md) — Input validation, auth, SQL/XSS prevention, secure logging
 - [Search API Migration](docs/backend/SEARCH_API_MIGRATION.md) — ES → OpenSearch: deprecated `ContentletAPI` search methods, plugin migration guide
 - [Telemetry Implementation](docs/backend/TELEMETRY_IMPLEMENTATION.md) — CDI-based metrics system, creating new metrics, `/v1/usage` endpoints
+- [Index Field Emission](docs/backend/INDEX_FIELD_EMISSION.md) — how `loadFields`/`toMap` build the index document; the `_dotraw` zero-padding sort invariant
 - [Jandex Metadata Scanning](docs/backend/JANDEX_METADATA_SCANNING.md) — Fast class/annotation metadata lookup, prefer over reflection
 - **ES → OpenSearch Migration** — infra migration from ElasticSearch to OpenSearch, phased dual-write/read rollout
   - [Migration Design](docs/backend/OPENSEARCH_MIGRATION.md) — Architecture, phased rollout, configuration

docs/backend/INDEX_FIELD_EMISSION.md

@@ -0,0 +1,105 @@
+# Index Field Emission — how a contentlet becomes an index document
+
+How `ESMappingAPIImpl` turns a contentlet's fields into the document written to Elasticsearch and
+OpenSearch, and the one invariant in that path that is easy to break and impossible to see from
+any single file.
+
+Applies to both engines: the document is rendered **once** and shared.
+
+## The two keys every field emits
+
+`loadFields` writes each field under two names (`ESMappingAPIImpl.java`, in the per-field loop):
+
+| Local variable | Key written | What happens to it |
+|---|---|---|
+| `keyName` | `<contenttype>.<field>` | survives into the final document |
+| `keyNameText` | `<contenttype>.<field>_text` | **renamed** to `_dotraw`, then dropped |
+
+The rename is not in `loadFields` — it is in `toMap`, in the post-processing loop that runs
+*after* `loadFields` returns. That loop:
+
+1. lowercases every key,
+2. renames `_text` → `_dotraw`,
+3. drops the `_text` key unless `CREATE_TEXT_INDEX_FIELD_FOR_NON_TEXT_FIELDS=true` (default false).
+
+**So `_dotraw` is the only survivor of the pair**, and a test that calls `loadFields` in isolation
+cannot observe it at all.
+
+## The invariant: `_dotraw` on a numeric field is zero-padded, or absent
+
+`*_dotraw` is mapped `keyword` (`es-content-mapping.json`, `template_1`), so it sorts
+**lexicographically**. And every sort in dotCMS resolves through it — `addBuilderSort` appends
+`_dotraw` to whatever field name the caller passes (`ContentFactoryIndexOperationsES`, and the
+OpenSearch counterpart).
+
+That is why numeric fields are padded to a fixed width by the `DecimalFormat` in `loadFields`:
+
+```
+"0000000000000000000.000000000000000000"     19 integer digits, 18 decimals
+54  →  "0000000000000000054.000000000000000000"
+```
+
+Fixed-width zero-padding is the **only** reason ordering by a numeric field is numeric rather than
+alphabetical. Break it and results silently reorder — no error, no exception, nothing in the log:
+
+- `"N/A"` sorts after every padded digit string, because `'N'` (0x4E) > `'9'` (0x39).
+- an unpadded `54` sorts before `9`.
+
+**Rule: for a field on a numeric storage column, `<field>_dotraw` is either the padded form or the
+key is not written at all. Never raw text, never an unpadded number.**
+
+### The trap when omitting a field
+
+`toMap`'s derivation loop has a fallback: when `<field>_text` is missing it synthesizes
+`<field>_dotraw` **from the bare key's value, unpadded**. So omitting only `_text` while still
+writing the numeric key produces exactly the malformed `_dotraw` this invariant forbids.
+
+**Omitting a numeric field means omitting both keys together.** They are one atomic change.
+
+## Storage column decides the branch, not the field type
+
+`loadFields` selects its serialization branch from `field.getFieldContentlet()` — the storage
+column — not from the declared field type. That matters because dotCMS allows a `TextField` to be
+backed by a numeric column, and its own built-in types do it:
+`htmlpageasset.sortOrder` is an `ImmutableTextField` with `DataTypes.INTEGER`
+(`PageContentType`), and `FieldFactoryImpl` accepts it because `TextField.acceptedDataTypes()`
+includes `INTEGER` and `FLOAT`.
+
+Consequence: **the value in a numeric-column field may be a `Number` or a `String`.** Handing a
+`String` straight to `DecimalFormat.format` throws `IllegalArgumentException`, and since the
+per-field catch rethrows, the whole contentlet is lost from the index — from *both* engines at
+once, in every migration phase, because the mapping is computed before the provider fan-out
+(`ContentletIndexAPIImpl.mapContentletForProcessor`: *"Compute mapping once; reuse across all
+providers"*). This was issue #37272.
+
+`loadNumericField` handles it: a `Number` takes the identical pre-existing path, a `String` is
+converted best-effort against the column's own type (`NumberUtil.toLongOrEmpty` /
+`toFloatOrEmpty`), and a value that cannot be represented numerically omits both keys and logs a
+WARN naming the field and content type.
+
+### Generated mapping by column
+
+| `field_contentlet` | Mapping | Emitted class |
+|---|---|---|
+| `integer%` | `long` | `Long` |
+| `float%` | `double` | **`Float`** |
+| `bool%` | `boolean` | `Boolean` |
+| `*_dotraw` | `keyword`, `ignore_above: 8191` | `String` |
+
+Mapping type and emitted class differ by design (`double` vs `Float`). It is pre-existing and
+harmless — do not "normalize" it, since changing an emitted class changes the document for every
+correctly-stored value in every installation.
+
+## Testing this path
+
+- **Assert against `toMap`, not `loadFields`.** `_dotraw` does not exist yet when `loadFields`
+  returns, so the assertions that matter are unreachable from there.
+- **Assert the key is absent**, not that it differs from the raw text. `assertNotEquals("n/a", …)`
+  passes against the unpadded-number bug described above.
+- **Field defaults bite.** `FieldDataGen` defaults `defaultValue` to `"testDefaultValue<millis>"`,
+  a non-numeric String; combined with the generator's `IndexPolicy.FORCE`, the save indexes
+  synchronously and a numeric-column field will fail during test setup. Give it a numeric default.
+- **A `unique` field is implicitly required** and must carry a value at save time; a field-level
+  default is not enough.
+- **`CheckboxField` does not accept `DataTypes.BOOL`** — only Hidden, Radio and Select do.
+- Working example: `dotcms-integration/src/test/java/com/dotcms/content/elasticsearch/business/ESMappingAPINumericFieldTest.java`

dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESMappingAPIImpl.java

@@ -65,6 +65,7 @@
 import com.dotmarketing.util.Config;
 import com.dotmarketing.util.InodeUtils;
 import com.dotmarketing.util.Logger;
+import com.dotmarketing.util.NumberUtil;
 import com.dotmarketing.util.PaginatedArrayList;
 import com.dotmarketing.util.ThreadSafeSimpleDateFormat;
 import com.dotmarketing.util.UtilMethods;
@@ -1114,8 +1115,8 @@ else if (field.getFieldType().equals(ESMappingConstants.FIELD_ELASTIC_TYPE_DATE)
 							.startsWith(ESMappingConstants.FIELD_ELASTIC_TYPE_FLOAT) || field
 							.getFieldContentlet()
 							.startsWith(ESMappingConstants.FIELD_ELASTIC_TYPE_INTEGER)) {
-						contentletMap.put(keyName, valueObj);
-						contentletMap.put(keyNameText, numFormatter.format(valueObj));
+						loadNumericField(contentletMap, field, structure, valueObj, keyName,
+								keyNameText, numFormatter);
 					} else {
 					    if (valueObj instanceof Date){
                             try {
@@ -1147,6 +1148,66 @@ else if (field.getFieldType().equals(ESMappingConstants.FIELD_ELASTIC_TYPE_DATE)
 		}
 	}
 
+	/**
+	 * Emits a field whose <em>storage column</em> is numeric ({@code integerN} / {@code floatN}).
+	 *
+	 * <p>The column, not the field type, decides this branch — and dotCMS allows a
+	 * {@link com.dotcms.contenttype.model.field.TextField} to be backed by a numeric column, as
+	 * its own built-in types do ({@code htmlpageasset.sortOrder}). So the value in hand may be a
+	 * {@link Number} or a numeric {@link String}, and handing a String straight to
+	 * {@code DecimalFormat.format} threw, aborting the whole document (issue #37272).</p>
+	 *
+	 * <p>A value that is already a {@link Number} takes exactly the path it always took — same
+	 * object, same formatter call — so documents for correctly-stored content do not move. A
+	 * String is converted best-effort against the column's own type, so a stored {@code "54"}
+	 * produces a document identical to a natively-stored {@code 54}.</p>
+	 *
+	 * <p>When the value cannot be represented numerically, <strong>both</strong> keys are
+	 * omitted. That pairing is not a preference:</p>
+	 * <ul>
+	 *   <li>The numeric key is skipped rather than defaulted to {@code 0}, because a {@code 0} is
+	 *       indistinguishable from a genuine {@code 0} in range queries, sorts and aggregations.</li>
+	 *   <li>{@code keyNameText} must go with it. {@code toMap}'s post-processing derives
+	 *       {@code <field>_dotraw} from {@code <field>_text}, and falls back to the bare key when
+	 *       that is absent — writing the numeric key alone would therefore produce an
+	 *       <em>unpadded</em> {@code _dotraw}. Since {@code _dotraw} is a {@code keyword} and every
+	 *       sort resolves through it, the fixed-width zero padding is the only reason ordering by
+	 *       a numeric field is numeric rather than lexicographic; an unpadded entry silently
+	 *       reorders results.</li>
+	 * </ul>
+	 *
+	 * @param valueObj the raw value, a {@link Number} or {@link String}
+	 */
+	private void loadNumericField(final Map<String, Object> contentletMap, final Field field,
+			final Structure structure, final Object valueObj, final String keyName,
+			final String keyNameText, final DecimalFormat numFormatter) {
+
+		if (valueObj instanceof Number) {
+			contentletMap.put(keyName, valueObj);
+			contentletMap.put(keyNameText, numFormatter.format(valueObj));
+			return;
+		}
+
+		final boolean floatColumn = field.getFieldContentlet()
+				.startsWith(ESMappingConstants.FIELD_ELASTIC_TYPE_FLOAT);
+		final Optional<? extends Number> converted = floatColumn
+				? NumberUtil.toFloatOrEmpty(valueObj)
+				: NumberUtil.toLongOrEmpty(valueObj);
+
+		if (converted.isEmpty()) {
+			Logger.warn(ESMappingAPIImpl.class, String.format(
+					"Field '%s' of Content Type '%s' is stored in the numeric column '%s' but its "
+							+ "value is not numeric; the field is omitted from the index document "
+							+ "for this contentlet. The rest of the contentlet is indexed.",
+					field.getVelocityVarName(), structure.getVelocityVarName(),
+					field.getFieldContentlet()));
+			return;
+		}
+
+		contentletMap.put(keyName, converted.get());
+		contentletMap.put(keyNameText, numFormatter.format(converted.get()));
+	}
+
 	private boolean loadTagsField(final Contentlet contentlet,
 								  final Map<String, Object> contentletMap,
 								  final Structure structure,

dotCMS/src/main/java/com/dotmarketing/util/NumberUtil.java

@@ -1,6 +1,7 @@
 package com.dotmarketing.util;
 
 import java.text.DecimalFormat;
+import java.util.Optional;
 import java.util.function.Supplier;
 
 public class NumberUtil {
@@ -78,4 +79,77 @@ public static int asInt(final Object value) {
 		return value != null ? Integer.parseInt(value.toString()) : 0;
 	} // asInt.
 
+	/**
+	 * Best-effort coercion to a {@link Long} of a value that may already be a {@link Number} or a
+	 * numeric {@link String}.
+	 *
+	 * <p>Unlike {@link #toLong(String, Supplier)} this reports failure instead of substituting a
+	 * default: an empty result means "this value is not a long", which lets the caller omit the
+	 * value entirely rather than fabricate one. Parsing is strict — a value carrying a fractional
+	 * part is <em>not</em> silently truncated.</p>
+	 *
+	 * @param value the value to coerce; may be null
+	 * @return the coerced value, or empty when it cannot be represented as a long
+	 */
+	public static Optional<Long> toLongOrEmpty(final Object value) {
+
+		if (value instanceof Number) {
+			final Number number = (Number) value;
+			// Reject a value carrying a fractional part rather than truncating it silently.
+			// Also rejects NaN and the infinities, whose longValue() is a meaningless clamp.
+			if (number.longValue() != number.doubleValue()) {
+				return Optional.empty();
+			}
+			return Optional.of(number.longValue());
+		}
+		if (!(value instanceof String)) {
+			return Optional.empty();
+		}
+		final String candidate = ((String) value).trim();
+		if (!UtilMethods.isSet(candidate)) {
+			return Optional.empty();
+		}
+		try {
+			return Optional.of(Long.parseLong(candidate));
+		} catch (final NumberFormatException e) {
+			// Not a long: non-numeric text, a fractional value, or beyond Long's range.
+			return Optional.empty();
+		}
+	} // toLongOrEmpty.
+
+	/**
+	 * Best-effort coercion to a {@link Float} of a value that may already be a {@link Number} or a
+	 * numeric {@link String}. Same reporting semantics as {@link #toLongOrEmpty(Object)}.
+	 *
+	 * @param value the value to coerce; may be null
+	 * @return the coerced value, or empty when it cannot be represented as a finite float
+	 */
+	public static Optional<Float> toFloatOrEmpty(final Object value) {
+
+		if (value instanceof Number) {
+			return finiteOrEmpty(((Number) value).floatValue());
+		}
+		if (!(value instanceof String)) {
+			return Optional.empty();
+		}
+		final String candidate = ((String) value).trim();
+		if (!UtilMethods.isSet(candidate)) {
+			return Optional.empty();
+		}
+		try {
+			return finiteOrEmpty(Float.parseFloat(candidate));
+		} catch (final NumberFormatException e) {
+			return Optional.empty();
+		}
+	} // toFloatOrEmpty.
+
+	/**
+	 * Guards against the non-finite values {@link Float#parseFloat} accepts. "Infinity" and "NaN"
+	 * parse successfully but cannot be serialized to JSON, so letting them through would push the
+	 * failure downstream into the index write instead of reporting it here.
+	 */
+	private static Optional<Float> finiteOrEmpty(final float candidate) {
+		return Float.isFinite(candidate) ? Optional.of(candidate) : Optional.empty();
+	} // finiteOrEmpty.
+
 } // E:O:F:NumberUtil