Map DateTime to google.protobuf.Timestamp and document temporal string formats - #1306
Merged
Conversation
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
July 9, 2026 16:00
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 9, 2026 17:29
5a2e67d to
1537869
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
2 times, most recently
from
July 10, 2026 15:44
721fe97 to
d516c31
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 10, 2026 15:52
d516c31 to
9e7369b
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 11, 2026 13:36
9e7369b to
9e2b508
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 11, 2026 13:54
9e2b508 to
06e1531
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 11, 2026 14:00
06e1531 to
ee20f41
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 15, 2026 23:52
508c016 to
4a6aa37
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 16, 2026 14:21
4a6aa37 to
1aa384b
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 17, 2026 15:05
1aa384b to
b94fd0e
Compare
jwils
added a commit
that referenced
this pull request
Jul 20, 2026
## Why - continue the pluggable ingestion serializer proposal after pulling JSON Schema into its own gem - revive the earlier protobuf prototype from #1056 on top of the new serializer extension points ## What - fill in the `elasticgraph-proto_ingestion` extension gem with core generation: `schema_artifacts:dump` emits a `proto3` `schema.proto` covering the schema's indexed types - map built-in ElasticGraph scalars to proto types, with `t.protobuf type:` for custom scalars (resolved via `type_ref.with_reverted_override` so built-ins renamed with `type_name_overrides` keep working) - generate messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid - keep `schema.proto` on public GraphQL field names; validate proto package names - hold extension state on a `ProtoIngestionState` container behind a single `proto_ingestion_state` reader (matching #1281) Field and enum value numbers are assigned sequentially in definition order in this PR; the stacked follow-up adds the `proto_field_numbers.yaml` sidecar that keeps them wire-stable across schema evolution. ## Stacked follow-ups 1. this PR — core `schema.proto` generation 2. wire-stable field/enum value numbers via a `proto_field_numbers.yaml` sidecar 3. `syntax: :proto2` support and custom file-level `headers:` 4. #1286 — enum value sourcing from existing proto enums + external proto type references ## Verification - `script/run_gem_specs elasticgraph-proto_ingestion` (100% line + branch coverage at this commit) - `script/type_check`, `script/lint`, `script/spellcheck` - `script/quick_build` green at the stack head (whose tree is identical to the previously reviewed single-PR revision) ## References - #1059 - #1056 - #1079 ## Update — 2026-07-10 - The current stack is #1080 → #1304 → #1306 → #1305 → #1286. - Proto extension state now uses a mutable Struct, and keyword package-name segments are validated without being rewritten. - Lists of lists now raise an actionable schema error instead of generating wrapper messages; this supersedes the earlier wrapping note above. --- ## Update — 2026-07-15 - Interface and union messages now wrap concrete subtype messages in a `oneof`, matching the JSON Schema `oneOf` representation. - Concrete subtype messages omit the redundant `__typename` discriminator. - Proto type rendering is now stateless: the generator selects the reachable type graph up front, then each extended type renders itself without mutating shared traversal state. - No-block extension coverage now completes the definitions so the fixture remains valid under CI GraphQL-schema validation. --- ## Update — 2026-07-19 - Replaced keyword suffixing with fully qualified local message and enum references, preserving source type and field names while disambiguating contextual protobuf words and built-in scalar names. - Removed the now-unnecessary keyword collision tracking and verified a generated schema containing contextual names with `protoc` 35.1.
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
2 times, most recently
from
July 31, 2026 03:45
1a80948 to
e08778f
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
July 31, 2026 13:31
e08778f to
b4fe4d2
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
August 3, 2026 15:52
b4fe4d2 to
c70dec9
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
August 3, 2026 15:57
c70dec9 to
a35049d
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
August 6, 2026 15:54
a35049d to
d64ce6d
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
3 times, most recently
from
August 8, 2026 17:59
debba00 to
c77e713
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
August 8, 2026 18:57
c77e713 to
809cc18
Compare
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
2 times, most recently
from
August 15, 2026 23:54
e1b71bd to
9736919
Compare
…g formats `DateTime` fields now use the well-known `google.protobuf.Timestamp` type (with `google/protobuf/timestamp.proto` imported automatically) instead of `string`. A Timestamp cannot be malformed the way a string can, and proto consumers get language-native timestamp types. Note that a Timestamp is a UTC instant, so a publisher's original UTC offset is not preserved; `t.protobuf type: "string"` remains available to override. To support this, `t.protobuf` gains two options usable by any scalar: - `import:` maps a scalar to an externally defined proto type, emitting the needed `import` statement in `schema.proto`. - `comment:` documents the expected format on each generated field, used by the built-in `string`-typed temporal scalars (`Date`, `LocalTime`, `TimeZone`) whose proto type is wider than the ElasticGraph type (e.g. `// ISO 8601 date`). Values are still validated at ingestion time, just as with JSON ingestion.
jwils
force-pushed
the
joshuaw/proto-timestamp
branch
from
August 18, 2026 13:43
9736919 to
2cdb436
Compare
myronmarston
approved these changes
Aug 18, 2026
myronmarston
left a comment
myronmarston
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. I thought I approved-with-feedback last time but I guess I didn't approve. Approving this time. Left some more comments since I thought of some things while re-reviewing but feel free to ignore, address in a followup, or address here before you merge (no need for me to re-review this regardless).
Addresses review feedback on #1306. Proto compilers attach a field's leading comments to the code they generate for that field, while a trailing comment on the field line is usually discarded. Render the format comment above the field instead, below the field's own doc comment and separated from it by a blank `//` line. This also removes the reason the comment had to be a single line, so a multi-line `field_comment:` is now allowed. Rename the option to `field_comment:`, since a scalar type has no proto representation of its own to comment on. The comment applies to each field of that type. Reword the built-in temporal comments to read well above the field (e.g. `Must be formatted as an ISO 8601 date`). Also: - Document that each `protobuf` call replaces the full configuration, so an override that omits `import:` or `field_comment:` clears the value a prior call set. Cover both directions with specs. - Give enum and object types a `protobuf_import` method that returns `nil`, so rendering imports no longer needs to grep for scalar types and non-scalar types can start requiring an import without a change there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`script/validate_readme_snippets` had no validator for the `protobuf` snippet type, so the fallback validator failed the build as soon as `elasticgraph-proto_ingestion/README.md` gained a ```protobuf block. Register a `ProtobufSnippetValidator` that displays the snippet and reports it as unvalidated, the same way `text` snippets are handled. Most protobuf snippets in our READMEs are excerpts of a generated `schema.proto` rather than complete proto files, so `protoc` cannot compile them on their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jwils
added a commit
that referenced
this pull request
Aug 19, 2026
## Why Some consumers need the generated messages to reference proto2 types — `protoc` forbids a proto3 message from referencing a proto2 enum — and language-specific file options (like `java_package`) shouldn't require baking any convention into the gem. ## What `proto_schema_artifacts` gains two options: - `syntax: :proto2` emits a proto2 file instead of the default proto3, labeling every field `optional` or `repeated` - `headers:` injects file-level lines (such as `option` declarations) verbatim as a contiguous section after the `package` declaration ## Risk Assessment Low — only affects the unreleased `elasticgraph-proto_ingestion` extension; both options are opt-in. ## References - Stacked on #1304 (which is stacked on #1080) ## Update — 2026-07-10 This PR is stacked directly on #1306 (→ #1304 → #1080).
jwils
added a commit
that referenced
this pull request
Aug 19, 2026
`schema_spec.rb` still held 34 examples covering every proto schema generation concern, and "schema" is a broad enough label that it would keep growing. Move each cohesive group out of it: - `scalar_proto_types_spec.rb` (added by #1306) gains the 4 remaining scalar examples, so every scalar proto type example now lives together - `stable_numbers_spec.rb` — field and enum value numbers staying stable as the schema evolves - `external_proto_enums_spec.rb` — `external_proto_enum`, both sourcing values and referencing an external enum type - `abstract_types_spec.rb` — `oneof` wrappers for interface and union types - `proto_syntax_and_headers_spec.rb` — the `syntax:` and `header_lines:` options `schema_spec.rb` keeps the 6 examples that cover schema generation as a whole. Every example moved verbatim, so this changes no behavior and adds no coverage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Mapping
DateTimetostringloses the type safety proto offers — a string can be malformed in ways agoogle.protobuf.Timestampcannot, and proto consumers get language-native timestamp types. For the remainingstring-typed temporal scalars, the proto type is much wider than the ElasticGraph type, so the expected format deserves documentation at the point of use (per review discussion).What
DateTimetogoogle.protobuf.Timestamp, importinggoogle/protobuf/timestamp.protoautomatically. ATimestampis a UTC instant, so a publisher's original UTC offset is not preserved;t.protobuf type: "string"remains available as an override.t.protobufgains two options usable by any scalar:import:maps a scalar to an externally defined proto type, emitting the neededimportstatementfield_comment:documents the expected format above each generated fieldstring-typed temporal scalars (Date,LocalTime,TimeZone) usefield_comment:to document their formats (e.g.// Must be formatted as an ISO 8601 date, e.g. "2024-11-25".). Values are still validated at ingestion time, exactly as with JSON ingestion.protobufcall replaces the full protobuf configuration, so an override that omitsimport:orfield_comment:clears the value a prior call set.Risk Assessment
Low — only affects the unreleased
elasticgraph-proto_ingestionextension from #1080.References
Update — 2026-07-10
Nested-list wrapper comment propagation was removed after #1080 changed to reject lists of lists. Format comments continue to apply to supported scalar and single-list fields.
Update — 2026-08-18
Addressed review feedback:
comment:is nowfield_comment:, and renders as//lines above the field rather than trailing the field line. Proto compilers attach leading comments to the code they generate for a field; a trailing comment is usually discarded. This also removed the reason the comment had to be a single line, so a multi-linefield_comment:is now allowed.Must be formatted as an ISO 8601 date, e.g. "2024-11-25".).protobufcall replaces the full configuration" behavior is now documented and covered in both directions (override that drops the import, and override that drops the field comment).protobuf_importwithnil, so import rendering no longer greps for scalar types and a non-scalar type can start requiring an import without a change there.schema_spec.rb) is handled in a follow-up PR at the end of this stack.