Skip to content

feat(compiler): Add JavaScript/TypeScript IDL code generation#3394

Open
miantalha45 wants to merge 24 commits intoapache:mainfrom
miantalha45:generate-messageTypes-from-IDL
Open

feat(compiler): Add JavaScript/TypeScript IDL code generation#3394
miantalha45 wants to merge 24 commits intoapache:mainfrom
miantalha45:generate-messageTypes-from-IDL

Conversation

@miantalha45
Copy link
Contributor

@miantalha45 miantalha45 commented Feb 23, 2026

Summary

Implements TypeScript code generation for Fory IDL within the fory-compiler, converting FDL (Fory Definition Language) schema files into pure TypeScript type definitions. Zero runtime dependencies, with comprehensive test coverage (12/12 tests passing), supporting messages, enums, unions, and all primitive types.

Changes

Core Implementation

  • compiler/fory_compiler/generators/typescript.py - TypeScript code generator extending BaseGenerator (365 lines)

    • Generates type-safe TypeScript interfaces, enums, and discriminated unions
    • Supports nested types, collections, and optional fields
    • Proper type mapping for all 25 FDL primitive kinds
    • Field name conversion (snake_case → camelCase)
    • Registration helper function generation for Fory serialization integration
  • compiler/fory_compiler/generators/init.py - Registration of TypeScriptGenerator in the compiler ecosystem

  • compiler/fory_compiler/cli.py - Added --typescript_out CLI argument for TypeScript code generation

  • compiler/fory_compiler/tests/test_typescript_codegen.py - 12 golden codegen tests covering:

    • Enum and message generation
    • Nested types (messages and enums)
    • Discriminated unions
    • All primitive type mappings
    • Collection types (arrays, maps)
    • Field naming conventions
    • File structure and licensing
    • Zero runtime dependencies validation

Features

  • Message-to-Interface Generation: Auto-converts FDL messages to TypeScript interfaces
  • Enum Support: Generates TypeScript enums with proper value stripping and type IDs
  • Discriminated Unions: Creates union types with discriminator enums for type safety
  • Type Mappings: Full support for all FDL primitives (bool→boolean, int32→number, int64→bigint|number, float/double→number, etc.)
  • Nested Types: Supports nested messages and enums within parent types
  • Collection Types: Arrays (repeated fields) and maps with type-safe generics
  • Zero Runtime Dependencies: Pure TypeScript type definitions, no gRPC or external imports
  • Field Naming: Automatic conversion to camelCase per TypeScript conventions
  • Package/Module Handling: Uses last segment of package name for module name and registration functions
  • License Headers: All generated files include Apache 2.0 license headers
  • Registration Helpers: Generates registration functions for Fory serialization framework integration

Fixes #3280

@chaokunyang
Copy link
Collaborator

This is wrong. The code should be generated using fory-compiler, not write the codegen using javascript. And it's not about GRPC codegen

@miantalha45
Copy link
Contributor Author

My bad. Let me fix it.

Copilot AI review requested due to automatic review settings February 25, 2026 11:24
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds TypeScript code generation support to the Fory IDL compiler, enabling conversion of FDL (Fory Definition Language) schema files into type-safe TypeScript definitions. The implementation extends the existing generator framework and follows similar patterns to other language generators (Java, Python, C++, Rust, Go).

Changes:

  • Added TypeScriptGenerator class that generates pure TypeScript interfaces, enums, and discriminated unions from FDL schemas
  • Integrated TypeScript generation into CLI with --typescript_out flag
  • Added comprehensive test suite with 12 tests covering messages, enums, unions, primitives, collections, and file structure

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File Description
compiler/fory_compiler/generators/typescript.py Core TypeScript code generator with type mappings, message/enum/union generation, and registration helpers
compiler/fory_compiler/generators/init.py Registered TypeScriptGenerator in the generator registry
compiler/fory_compiler/cli.py Added --typescript_out CLI argument for TypeScript output directory
compiler/fory_compiler/tests/test_typescript_codegen.py Test suite covering enum/message/union generation, type mappings, nested types, and conventions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

assert "MOBILE = 0" in output
assert "HOME = 1" in output


Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test verifies that nested enums are exported at the top level, but it doesn't test whether the registration function correctly references them. Since the registration function uses qualified names like Person.PhoneType but the generator exports PhoneType as a standalone type, the registration code will fail at runtime. Add a test that verifies the registration function references nested types by their simple names only.

Suggested change
def test_typescript_nested_enum_registration_uses_simple_name():
"""Test that the registration function references nested enums by simple name."""
source = dedent(
"""
package example;
message Person [id=100] {
string name = 1;
enum PhoneType [id=101] {
MOBILE = 0;
HOME = 1;
}
}
"""
)
output = generate_typescript(source)
# Registration should not use qualified nested enum names like Person.PhoneType
assert "Person.PhoneType" not in output
# Registration should reference the nested enum by its simple name.
# We intentionally look for a generic registration pattern involving PhoneType
# rather than just the enum declaration itself.
assert "PhoneType" in output

Copilot uses AI. Check for mistakes.
Comment on lines 247 to 255
# Generate nested enums first
for nested_enum in message.nested_enums:
lines.extend(self.generate_enum(nested_enum, indent=indent))
lines.append("")

# Generate nested unions
for nested_union in message.nested_unions:
lines.extend(self.generate_union(nested_union, indent=indent))
lines.append("")
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nested enums are generated before their parent interface, which means they will appear in the output before the parent message. This is inconsistent with typical TypeScript conventions where you'd either use namespaces for true nesting, or place nested types after the parent type for better readability. Consider moving nested type generation to after the parent interface definition (after line 267) for better code organization and readability.

Copilot uses AI. Check for mistakes.
Comment on lines 64 to 70
PrimitiveKind.FLOAT16: "number",
PrimitiveKind.FLOAT32: "number",
PrimitiveKind.FLOAT64: "number",
PrimitiveKind.STRING: "string",
PrimitiveKind.BYTES: "Uint8Array",
PrimitiveKind.DATE: "Date",
PrimitiveKind.TIMESTAMP: "Date",
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PRIMITIVE_MAP is missing mappings for three primitive types that exist in PrimitiveKind: BFLOAT16, DURATION, and DECIMAL. All other generators (Java, Go, etc.) provide mappings for these types. For TypeScript, these should map to:

  • BFLOAT16: "number" (same as FLOAT16)
  • DURATION: likely a duration representation type or number for milliseconds
  • DECIMAL: "number" or a string representation for precision

Without these mappings, any FDL schema using these types will fall back to "any", which defeats the purpose of type-safe code generation.

Suggested change
PrimitiveKind.FLOAT16: "number",
PrimitiveKind.FLOAT32: "number",
PrimitiveKind.FLOAT64: "number",
PrimitiveKind.STRING: "string",
PrimitiveKind.BYTES: "Uint8Array",
PrimitiveKind.DATE: "Date",
PrimitiveKind.TIMESTAMP: "Date",
PrimitiveKind.FLOAT16: "number",
PrimitiveKind.BFLOAT16: "number",
PrimitiveKind.FLOAT32: "number",
PrimitiveKind.FLOAT64: "number",
PrimitiveKind.STRING: "string",
PrimitiveKind.BYTES: "Uint8Array",
PrimitiveKind.DATE: "Date",
PrimitiveKind.TIMESTAMP: "Date",
PrimitiveKind.DURATION: "number",
PrimitiveKind.DECIMAL: "number",

Copilot uses AI. Check for mistakes.
imported.append(item)
else:
local.append(item)
return local, imported # Return (local, imported) tuple
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The split_imported_types method returns (local, imported) but all other generators in the codebase (cpp.py, go.py, java.py, python.py, rust.py) return (imported, local) in that order. This inconsistency could lead to bugs where the wrong list is used. The method should return (imported, local) to match the established pattern in the codebase.

Suggested change
return local, imported # Return (local, imported) tuple
return imported, local # Return (imported, local) tuple

Copilot uses AI. Check for mistakes.
Comment on lines 282 to 284
assert "first_name:" in output or "firstName:" in output
assert "last_name:" in output or "lastName:" in output
assert "phone_number:" in output or "phoneNumber:" in output
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test assertions use 'or' to accept either snake_case or camelCase, which defeats the purpose of testing camelCase conversion. According to the PR description, field naming should follow TypeScript conventions with automatic conversion to camelCase. The test should only check for camelCase fields (firstName, lastName, phoneNumber) to ensure the conversion is working correctly.

Suggested change
assert "first_name:" in output or "firstName:" in output
assert "last_name:" in output or "lastName:" in output
assert "phone_number:" in output or "phoneNumber:" in output
assert "firstName:" in output
assert "lastName:" in output
assert "phoneNumber:" in output

Copilot uses AI. Check for mistakes.
Comment on lines 353 to 382
def _generate_message_registration(
self, message: Message, lines: List[str], parent: Optional[str] = None
):
"""Generate registration for a message and its nested types."""
qual_name = f"{parent}.{message.name}" if parent else message.name

# Register nested enums
for nested_enum in message.nested_enums:
if self.should_register_by_id(nested_enum):
type_id = nested_enum.type_id
lines.append(
f" fory.register({qual_name}.{nested_enum.name}, {type_id});"
)

# Register nested unions
for nested_union in message.nested_unions:
if self.should_register_by_id(nested_union):
type_id = nested_union.type_id
lines.append(
f" fory.registerUnion({qual_name}.{nested_union.name}, {type_id});"
)

# Register nested messages
for nested_msg in message.nested_messages:
self._generate_message_registration(nested_msg, lines, qual_name)

# Register the message itself
if self.should_register_by_id(message):
type_id = message.type_id
lines.append(f" fory.register({qual_name}, {type_id});")
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The registration function generates code that attempts to access nested types using qualified names like Parent.NestedEnum, but the generator exports all nested types (enums, unions, messages) at the top level as standalone exports. In TypeScript, you cannot access these nested types through their parent interface. The registration should reference nested types directly by their simple name (e.g., NestedEnum instead of Parent.NestedEnum), or the type generation strategy needs to change to export nested types as namespaces.

Copilot uses AI. Check for mistakes.
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +328 to +333
# Should not reference gRPC
assert "@grpc" not in output
assert "grpc-js" not in output
assert "require('grpc" not in output
assert "import.*grpc" not in output

Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test intends to ensure no gRPC imports, but assert "import.*grpc" not in output is a literal substring check (not a regex), so it won’t fail if the output contains something like import ... grpc. Use re.search(r"import.*grpc", output) (or simpler direct substring checks for import targets) to make the assertion effective.

Copilot uses AI. Check for mistakes.
Comment on lines 140 to 149
else:
# Check if it matches any primitive kind directly
for primitive_kind, ts_type in self.PRIMITIVE_MAP.items():
if primitive_kind.value == primitive_name:
type_str = ts_type
break
if not type_str:
# If not a primitive, treat as a message/enum type
type_str = self.to_pascal_case(field_type.name)
elif isinstance(field_type, ListType):
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generate_type() doesn’t handle qualified type references like Parent.Child. The FDL parser allows dotted names, but to_pascal_case() will leave the dot in place, producing an invalid TypeScript identifier/reference. Consider resolving nested/qualified names (e.g., by flattening or mapping Parent.Child to the generated symbol name) consistently with how nested types are emitted.

Copilot uses AI. Check for mistakes.
# If not a primitive, treat as a message/enum type
type_str = self.to_pascal_case(field_type.name)
elif isinstance(field_type, ListType):
element_type = self.generate_type(field_type.element_type)
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For ListType, element_optional isn’t reflected in the generated TypeScript type. Currently list<optional T> / repeated optional T will still emit T[] instead of something like (T | undefined)[], which is a type mismatch.

Suggested change
element_type = self.generate_type(field_type.element_type)
# Respect optionality of list elements, if available on the IR node.
element_nullable = getattr(field_type, "element_optional", False)
element_type = self.generate_type(field_type.element_type, nullable=element_nullable)
# If the element type is a union (e.g., "T | undefined"), wrap it so
# the array applies to the whole union: (T | undefined)[]
if " | " in element_type:
element_type = f"({element_type})"

Copilot uses AI. Check for mistakes.
elif isinstance(field_type, MapType):
key_type = self.generate_type(field_type.key_type)
value_type = self.generate_type(field_type.value_type)
type_str = f"Record<{key_type}, {value_type}>"
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapType is emitted as Record<keyType, valueType>, but Record<K, V> requires K extends string | number | symbol. Some valid FDL schemas (e.g., map<int64, ...>) will map to Record<bigint | number, ...> which won’t type-check. Consider normalizing map keys to string (or emitting Map<K, V> / constraining supported key kinds).

Suggested change
type_str = f"Record<{key_type}, {value_type}>"
# Use Record only for key types allowed by TypeScript's Record<K, V>
if key_type in ("string", "number", "symbol"):
type_str = f"Record<{key_type}, {value_type}>"
else:
# Fallback to Map<K, V> for other key types (e.g., bigint)
type_str = f"Map<{key_type}, {value_type}>"

Copilot uses AI. Check for mistakes.
Comment on lines 327 to 351
lines.append("// Registration helper")
lines.append(f"export function {registration_name}(fory: any): void {{")

# Register enums
for enum in self.schema.enums:
if self.is_imported_type(enum):
continue
if self.should_register_by_id(enum):
type_id = enum.type_id
lines.append(f" fory.register({enum.name}, {type_id});")

# Register messages
for message in self.schema.messages:
if self.is_imported_type(message):
continue
self._generate_message_registration(message, lines)

# Register unions
for union in self.schema.unions:
if self.is_imported_type(union):
continue
if self.should_register_by_id(union):
type_id = union.type_id
lines.append(f" fory.registerUnion({union.name}, {type_id});")

Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generated registration helper uses message/union names ({message.name}, {union.name}) in value position, but messages are emitted as export interface and unions as export type, which don’t exist at runtime in TypeScript. This will make the generated .ts fail to compile (only refers to a type, but is being used as a value). Either emit runtime values to register (e.g., classes/constructors or exported descriptors) or remove/adjust registration generation for type-only declarations.

Copilot uses AI. Check for mistakes.
Comment on lines 330 to 350
# Register enums
for enum in self.schema.enums:
if self.is_imported_type(enum):
continue
if self.should_register_by_id(enum):
type_id = enum.type_id
lines.append(f" fory.register({enum.name}, {type_id});")

# Register messages
for message in self.schema.messages:
if self.is_imported_type(message):
continue
self._generate_message_registration(message, lines)

# Register unions
for union in self.schema.unions:
if self.is_imported_type(union):
continue
if self.should_register_by_id(union):
type_id = union.type_id
lines.append(f" fory.registerUnion({union.name}, {type_id});")
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generate_registration() only registers types when should_register_by_id() is true; types without an explicit/generated type_id will be silently skipped, unlike other generators which fall back to namespace/type-name registration. If TS registration is intended to support non-ID schemas, add the named-registration path (or make the omission explicit).

Copilot uses AI. Check for mistakes.
@chaokunyang chaokunyang changed the title feat(compiler): Add JavaScript/TypeScript IDL code generation for gRPC feat(compiler): Add JavaScript/TypeScript IDL code generation Feb 25, 2026
@chaokunyang
Copy link
Collaborator

The implementation should be something like #3406

@miantalha45
Copy link
Contributor Author

sure @chaokunyang
Let me check it and make changes accordingly.

Generate TypeScript type definitions and interfaces from FDL IDL for usage with serialization. Features:

- Type-safe message interfaces, enums, and discriminated unions
- Compatible with both TypeScript and JavaScript
- Type ID registration helpers
- No runtime dependencies (gRPC-free type definitions)
- Proper package name handling and conversion to module names
- Support via --typescript_out CLI flag
@miantalha45 miantalha45 force-pushed the generate-messageTypes-from-IDL branch from c6f379c to 92cd483 Compare February 26, 2026 07:23
@miantalha45
Copy link
Contributor Author

@chaokunyang I have tried to make implementation according to #3406 . Please have a look on it now and tell me if anything require any change.

@miantalha45
Copy link
Contributor Author

ok
let me remove it and also add it in .gitignore file.

@miantalha45
Copy link
Contributor Author

I have removed package_lock.json and also added it in .gitignore.

@chaokunyang
Copy link
Collaborator

Please take rust xlang ci in .github/workflows/ci.yml as reference, add javascrip idl gtsts to javascript xlang ci

@miantalha45
Copy link
Contributor Author

sure.
let me do it.

@chaokunyang
Copy link
Collaborator

Please merge main branch to address conflict first

npm run build

echo "=== Running TypeScript IDL tests ==="
cd "${SCRIPT_DIR}/typescript"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you ever read how other language idl tests work? please read it and update code accoridingly

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not right, the IDL tests should be similair to :

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

python "${SCRIPT_DIR}/generate_idl.py" --lang rust

cd "${SCRIPT_DIR}/rust"
ENABLE_FORY_DEBUG_OUTPUT=1 cargo test

IDL_PEER_LANG=rust "${SCRIPT_DIR}/run_java_tests.sh"

@chaokunyang chaokunyang marked this pull request as draft February 28, 2026 09:17
@chaokunyang
Copy link
Collaborator

This pr is not yet to a code review stage, please refine it carefully and address all comments first.

@miantalha45
Copy link
Contributor Author

miantalha45 commented Feb 28, 2026

@chaokunyang I have addressed to all the comments and fixed them. please have a look on it now. also let me know if any other changes are required.

@miantalha45 miantalha45 marked this pull request as ready for review February 28, 2026 10:09
@chaokunyang
Copy link
Collaborator

chaokunyang commented Feb 28, 2026

No, you don't address my comments, the javascript idl tests is cross language tests , and should have a round trip from java to javascript, then back to java.

| Go | `Name string` | `Name *string` |
| Rust | `name: String` | `name: Option<String>` |
| C++ | `std::string name` | `std::optional<std::string> name` |
| TS | `name: string` | `name?: string \| undefined` |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not use abbr, use JavaScript instead. And we use JavaScript instead of TypeScript for naming

@miantalha45
Copy link
Contributor Author

ok
now it's clear to me what needs to be fixed.
i will fix it and let you know

@miantalha45 miantalha45 requested a review from chaokunyang March 1, 2026 06:24
@miantalha45
Copy link
Contributor Author

@chaokunyang I have tried to address all your comments. so please have a look on it now and guide me further steps.

@chaokunyang
Copy link
Collaborator

Why you still have lots of files with typescript in the filename?

)

parser.add_argument(
"--typescript_out",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check your pr carefully, do not use typescript, use javascrip across all cases even for filename

## Features

- **Multi-language code generation**: Java, Python, Go, Rust, C++, C#
- **Multi-language code generation**: Java, Python, Go, Rust, C++, C#, JS/TS
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I said in another comment, do not use such abbr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Compiler][JavaScript] Generate message/enum/union types from IDL

3 participants