Added cardinal semiotic class for te ITN - #451
Open
jaya-TN wants to merge 1 commit into
Open
Conversation
jaya-TN
marked this pull request as ready for review
July 7, 2026 10:51
jaya-TN
changed the title
folivoramanh
requested changes
Jul 21, 2026
folivoramanh
left a comment
folivoramanh
left a comment
Collaborator
There was a problem hiding this comment.
- add sparrowhawk file and run sh test (we need to make sure both pytest and sparrowhawk pass)
- add te to jenkins file
- try to optimize cardinal logic (you can look up other languages eg. Hindi, Spanish, Portuguese, etc... for both TN and ITN) to optimize graph logic, too much looping could lead to slow runtime
- do te ITN support english number (eg. 123456789?) if yes, can you add test cases for both te number and en number?
jaya-TN
force-pushed
the
feat-te-itn-cardinal-v2
branch
from
July 29, 2026 14:01
33c98ab to
f5bb3de
Compare
Add the Telugu (te) cardinal semiotic class for ITN: taggers, verbalizers, data files, graph_utils and utils. Uses the Indian numbering system (hundred/thousand/lakh/crore) with a fixed-width place-value graph and native Telugu digit output; English digits are passed through. The number lexicon is data-driven via TSV files. Adds pytest cases and a Sparrowhawk test, and registers te in the Jenkinsfile and pynini_export. Signed-off-by: jaya-TN <seruganti@nvidia.com>
jaya-TN
force-pushed
the
feat-te-itn-cardinal-v2
branch
from
July 29, 2026 14:06
f5bb3de to
ddc54ad
Compare
folivoramanh
requested changes
Jul 29, 2026
Collaborator
There was a problem hiding this comment.
why first and second line got only 1 value while third row got 2?
| super().__init__(name="cardinal", kind="classify") | ||
| self.input_case = input_case | ||
|
|
||
| graph_zero = pynini.string_file(get_abs_path("data/numbers/zero.tsv")).invert() |
Collaborator
There was a problem hiding this comment.
do not abuse .invert(), try to standardize the data
| two_te_digits = NEMO_TE_DIGIT + NEMO_TE_DIGIT | ||
| graph_digit |= (graph_people | graph_special) @ NEMO_TE_DIGIT | ||
|
|
||
| self.graph_zero = graph_zero |
Collaborator
There was a problem hiding this comment.
only set self.variable when it's reusable on other semiotic class
| self.graph_two_digit = graph_two_digit | ||
| two_digit_or_zeros = graph_two_digit | pynutil.insert("౦౦") | ||
|
|
||
| graph_case_suffix = pynini.string_file(get_abs_path("data/numbers/case_suffix.tsv")) |
Collaborator
There was a problem hiding this comment.
move all pynini.string_file call to begin of init function for easier checking file call
| @@ -0,0 +1,142 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. | |||
Collaborator
There was a problem hiding this comment.
change all 2 first lines to
"# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#"
| @pytest.mark.run_only_on('CPU') | ||
| @pytest.mark.unit | ||
| def test_denorm(self, test_input, expected): | ||
| tagged = pynini.shortestpath(test_input @ self.tagger).string() |
Collaborator
There was a problem hiding this comment.
read hindi test and use the same code, do not use pynini.shortestpath here
| args = parse_args() | ||
|
|
||
| if args.language in ['ru', 'es_en', 'hi_en', 'mr'] and args.grammars == 'tn_grammars': | ||
| if args.language in ['ru', 'es_en', 'hi_en', 'mr', 'te'] and args.grammars == 'tn_grammars': |
Collaborator
There was a problem hiding this comment.
we also have PIC for te TN, right? if yes, please remove this change
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
This PR adds support for the Telugu Inverse Text Normalization (ITN) Cardinal semiotic class.
Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files?Copyright 2015 and onwards Google, Inc.. See an example here.try import: ... except: ...) if not already done.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.