Skip to content

Asn1 oer uper contrib - #5050

Open
polybassa wants to merge 46 commits into
secdev:masterfrom
polybassa:asn1-oer-uper-contrib
Open

Asn1 oer uper contrib#5050
polybassa wants to merge 46 commits into
secdev:masterfrom
polybassa:asn1-oer-uper-contrib

Conversation

@polybassa

Copy link
Copy Markdown
Contributor

No description provided.

@polybassa
polybassa force-pushed the asn1-oer-uper-contrib branch from e7bc1d3 to 4cdc2de Compare August 7, 2026 18:45
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.52890% with 82 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.10%. Comparing base (c0686b3) to head (37fc5bf).

Files with missing lines Patch % Lines
scapy/asn1/oer.py 95.28% 18 Missing ⚠️
scapy/asn1/uper.py 96.98% 18 Missing ⚠️
scapy/asn1fields.py 91.37% 17 Missing ⚠️
scapy/asn1/compound_ber.py 91.50% 9 Missing ⚠️
scapy/asn1/constraints.py 89.77% 9 Missing ⚠️
scapy/asn1/compound_uper.py 97.94% 3 Missing ⚠️
scapy/asn1/asn1.py 95.34% 2 Missing ⚠️
scapy/asn1/context.py 97.61% 2 Missing ⚠️
scapy/asn1/oid.py 90.00% 2 Missing ⚠️
scapy/asn1/compound.py 93.75% 1 Missing ⚠️
... and 1 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5050      +/-   ##
==========================================
+ Coverage   80.81%   81.10%   +0.29%     
==========================================
  Files         390      401      +11     
  Lines       96947    98621    +1674     
==========================================
+ Hits        78344    79991    +1647     
- Misses      18603    18630      +27     
Files with missing lines Coverage Δ
scapy/asn1/ber.py 96.98% <100.00%> (+0.87%) ⬆️
scapy/asn1/intutil.py 100.00% <100.00%> (ø)
scapy/asn1/tag.py 100.00% <100.00%> (ø)
scapy/asn1packet.py 92.85% <100.00%> (+2.85%) ⬆️
scapy/asn1/compound.py 93.75% <93.75%> (ø)
scapy/asn1/compound_oer.py 99.00% <99.00%> (ø)
scapy/asn1/asn1.py 88.46% <95.34%> (+1.62%) ⬆️
scapy/asn1/context.py 97.61% <97.61%> (ø)
scapy/asn1/oid.py 90.00% <90.00%> (ø)
scapy/asn1/compound_uper.py 97.94% <97.94%> (ø)
... and 5 more

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread scapy/asn1/ber.py Outdated
@polybassa
polybassa marked this pull request as ready for review August 11, 2026 19:03
@polybassa
polybassa requested a lite review from Copilot August 11, 2026 19:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds ASN.1 OER and UPER (registered on ASN1_Codecs.OER and ASN1_Codecs.PER) plus a new “field hooks” mechanism to let codecs override compound-field behavior (tagging, SEQUENCE/CHOICE/SEQUENCE OF) while keeping BER as the default behavior.

Changes:

  • Introduces ASN1Codec.register_field_hooks() / field_hook() and updates ASN.1 fields to consult codec-specific hooks.
  • Adds new contrib codecs: scapy.contrib.oer (OER) and scapy.contrib.uper (UPER/PER).
  • Expands/creates UTS coverage for cross-codec build/dissect and OER vectors/fuzzing.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
test/scapy/layers/ber.uts Updates tests for the new BER tagging hooks + adds BER build/dissect test coverage.
test/scapy/layers/asn1.uts Adds cross-codec (BER/OER/PER) build/dissect tests and codec-opts/default-component checks.
test/contrib/oer.uts New OER-focused test suite (vectors, fuzzing, interop, conformance checks).
scapy/contrib/uper.py New UPER implementation and PER field hooks for bitstream-oriented encoding/decoding.
scapy/contrib/oer.py New OER implementation and OER field hooks for preamble/CHOICE-tag behavior.
scapy/asn1fields.py Adds codec_opts plumbing and consults codec field hooks for tagging/compound-field operations.
scapy/asn1/ber.py Registers BER field hooks (tagging) via the new hook mechanism.
scapy/asn1/asn1.py Adds codec-level field hook registration + safe default for _field_hooks.
.config/codespell_ignore.txt Adds OER/UPER-related ignore words.
Suppressed comments (1)

test/scapy/layers/asn1.uts:402

  • Duplicate helper function: _roundtrip is defined twice back-to-back here. One of them should be removed to avoid confusion and reduce noise in the test file.
def _roundtrip(cls, pkt):
    # type: (type, ASN1_Packet) -> ASN1_Packet
    return cls(raw(pkt))

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/scapy/layers/ber.uts
Comment thread test/scapy/layers/ber.uts
Comment thread test/scapy/layers/asn1.uts
Comment thread test/scapy/layers/asn1.uts Outdated
Comment thread test/contrib/oer.uts
Comment thread scapy/asn1/ber.py Outdated

@gpotter2 gpotter2 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very hard to review. I think it needs some cleanup phase where it reduces the number of functions that are used only once.

I also think that eventually it makes more sense to include all of this in scapy/asn1 directly

Comment thread scapy/contrib/uper.py Outdated
Comment thread scapy/contrib/uper.py Outdated
Comment thread scapy/contrib/oer.py Outdated
Comment thread scapy/contrib/oer.py Outdated
Comment thread scapy/asn1fields.py Outdated
Comment thread scapy/asn1fields.py Outdated
Comment thread scapy/asn1fields.py Outdated
Comment thread scapy/contrib/oer.py
Comment on lines +120 to +129
def _OER_check_len(name, s, number_of_bytes, offset=0):
# type: (str, bytes, int, int) -> None
"""Raise unless s carries number_of_bytes octets past its first offset."""
available = len(s) - offset
if available < number_of_bytes:
raise OER_Decoding_Error(
"%s: Got %i bytes while expecting %i" %
(name, available, number_of_bytes),
remaining=s
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Useful as a standalone function? I don't like this pattern of calling a function that might throw an error, it's not clear from the parent code

Comment thread scapy/contrib/oer.py
Comment on lines +164 to +208
def OER_signed_integer_enc(i):
# type: (int) -> bytes
# X.696 10.4: the shortest two's complement encoding. A negative value
# needs one bit less than its magnitude suggests, as -2**(8n-1) still
# fits in n octets, hence the increment before measuring.
magnitude = i + 1 if i < 0 else i
number_of_bytes = (magnitude.bit_length() + 8) // 8
value = i & ((1 << (8 * number_of_bytes)) - 1)
return OER_len_enc(number_of_bytes) + value.to_bytes(number_of_bytes, "big")


def OER_signed_integer_dec(s):
# type: (bytes) -> Tuple[int, bytes]
number_of_bytes, s = OER_len_dec(s)
_OER_check_len("OER_signed_integer_dec", s, number_of_bytes)
if number_of_bytes == 0:
raise OER_Decoding_Error(
"OER_signed_integer_dec: got an empty length determinant",
remaining=s
)
value = int.from_bytes(s[:number_of_bytes], "big")
number_of_bits = 8 * number_of_bytes
if value & (1 << (number_of_bits - 1)):
value -= (1 << number_of_bits) - 1
value -= 1
return value, s[number_of_bytes:]


def OER_unsigned_integer_enc(i):
# type: (int) -> bytes
if i < 0:
raise OER_Encoding_Error(
"OER_unsigned_integer_enc: %i is negative" % i
)
number_of_bits = max(i.bit_length(), 1)
number_of_bytes = (number_of_bits + 7) // 8
return OER_len_enc(number_of_bytes) + i.to_bytes(number_of_bytes, "big")


def OER_unsigned_integer_dec(s):
# type: (bytes) -> Tuple[int, bytes]
number_of_bytes, s = OER_len_dec(s)
_OER_check_len("OER_unsigned_integer_dec", s, number_of_bytes)
value = int.from_bytes(s[:number_of_bytes], "big")
return value, s[number_of_bytes:]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have all of those for BER? They're part of the classes directly, I think it makes more sense

Comment thread scapy/contrib/uper.py Outdated
Comment thread scapy/contrib/uper.py
Comment thread scapy/contrib/uper.py Outdated
Comment thread scapy/contrib/uper.py Outdated
Comment thread scapy/contrib/uper.py
@gpotter2

gpotter2 commented Sep 1, 2026

Copy link
Copy Markdown
Member

Very sorry @polybassa but this still looks like slop for the most part.

This probably means we should try to add more coding guidances to the AI

polybassa pushed a commit to polybassa/scapy-1 that referenced this pull request Sep 2, 2026
Restore extension points and correct OID, integer bounds, PER padding,
X.509 extract_packet, and BER DEFAULT handling; add regression tests.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
polybassa pushed a commit to polybassa/scapy-1 that referenced this pull request Sep 2, 2026
Restore extension points and correct OID, integer bounds, PER padding,
X.509 extract_packet, and BER DEFAULT handling; add regression tests.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
@polybassa
polybassa force-pushed the asn1-oer-uper-contrib branch from 3aba643 to 07160bb Compare September 2, 2026 05:06
polybassa pushed a commit to polybassa/scapy-1 that referenced this pull request Sep 2, 2026
Restore extension points and correct OID, integer bounds, PER padding,
X.509 extract_packet, and BER DEFAULT handling; add regression tests.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
polybassa pushed a commit to polybassa/scapy-1 that referenced this pull request Sep 2, 2026
Fix UPER trailing-octet handling, OER integer constraints, X.509
underlayer extraction, known-multiplier string typing, and restore
codec tagging/_codec_kwargs contracts with regression tests.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
@polybassa
polybassa force-pushed the asn1-oer-uper-contrib branch from 071ffa8 to 2f31055 Compare September 2, 2026 07:45
polybassa pushed a commit to polybassa/scapy-1 that referenced this pull request Sep 2, 2026
Add semi-constrained INTEGER and extensible SIZE paths, honor OER size
constraints and named enum encode values, and drop leftover dead helpers.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
@polybassa
polybassa force-pushed the asn1-oer-uper-contrib branch from 584768f to faaa99b Compare September 2, 2026 13:09
polybassa pushed a commit to polybassa/scapy-1 that referenced this pull request Sep 2, 2026
Move compound wire rules onto encoder/decoder methods, reject empty OER
SEQUENCEs with mandatory fields, shrink BER churn, and canonicalize UPER
constraint kwargs.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
polybassa pushed a commit to polybassa/scapy-1 that referenced this pull request Sep 2, 2026
Keep the HEAD^1..HEAD^2 restriction on this branch so secdev#5050 CI does not
false-fail on master tips until the standalone CI PR lands.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Nils Weiss added 7 commits September 3, 2026 19:43
Move OER/UPER codec implementations to scapy.contrib and wire
asn1fields for OER/PER using the pluggable tagging/kwargs hooks.

AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
AI-Assisted: yes (Cursor)
Nils Weiss and others added 29 commits September 3, 2026 19:43
Finish the OER/UPER architectural cleanup by reading constraints from
fields directly, routing build/dissect through encode_to/decode_from,
and sharing SEQUENCE/CHOICE/SEQUENCE OF logic in compound.py. Contrib
modules become thin re-exports.

Co-authored-by: Cursor <cursoragent@cursor.com>

AI-Assisted: yes (Cursor Agent)
Unify compound child dispatch on encode_to/decode_from, add OER encoder
contexts and PER bit-stream helpers, and read field constraints directly in
primitive codecs instead of round-tripping through codec_kwargs.

Co-authored-by: Cursor <cursoragent@cursor.com>

AI-Assisted: yes (Cursor Agent)
Remove EncodingParams, codec_kwargs, and field codec_opts/_codec_kwargs;
codecs resolve size_len and constraints via field= directly.

Co-authored-by: Cursor <cursoragent@cursor.com>

AI-Assisted: yes (Cursor Agent)
Restore extension points and correct OID, integer bounds, PER padding,
X.509 extract_packet, and BER DEFAULT handling; add regression tests.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Remove unused imports, tighten typing for observed tags and codec
returns, and satisfy line-length checks on BER override annotations.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Fix UPER trailing-octet handling, OER integer constraints, X.509
underlayer extraction, known-multiplier string typing, and restore
codec tagging/_codec_kwargs contracts with regression tests.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Rebase onto master picked up secdev#5062; replace remaining orb() uses with
direct byte indexing so the contrib codecs import cleanly.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Rewrite the intentional typo kwargs in oer.uts so codespell stays
quiet, and flatten the oer_int_wire_params docstring for Sphinx -W.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
ASN1F_PACKET.any2i and UPER nestings set parent only, which broke
EncryptedData.encrypt in KerberosSSP; set underlayer and parent together.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Import codecs from scapy.asn1 and use minimum/maximum/extensible/unsigned in tests.

Co-authored-by: Cursor <cursoragent@cursor.com>

AI-Assisted: yes (Cursor Agent)
Drop unused compound helper re-exports, constraint aliases, and stale smoke tests.

Co-authored-by: Cursor <cursoragent@cursor.com>

AI-Assisted: yes (Cursor Agent)
Write OER SEQUENCE children into one encoder, resolve UPER bounds once, and
centralize two's-complement octet math.

Co-authored-by: Cursor <cursoragent@cursor.com>

AI-Assisted: yes (Cursor Agent)
Add semi-constrained INTEGER and extensible SIZE paths, honor OER size
constraints and named enum encode values, and drop leftover dead helpers.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
actions/checkout builds a merge of the PR into the base branch, so
rev-list from HEAD was also validating base-branch tips.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Move compound wire rules onto encoder/decoder methods, reject empty OER
SEQUENCEs with mandatory fields, shrink BER churn, and canonicalize UPER
constraint kwargs.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the HEAD^1..HEAD^2 restriction on this branch so secdev#5050 CI does not
false-fail on master tips until the standalone CI PR lands.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Cast codec.enc results after neutral get_codec typing, and drop obsolete
attr-defined ignores.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Bind compound encode/decode hooks directly on BER/OER/UPER contexts,
separate field-layer contexts from raw UPER bit streams, and inline
trivial constraint getters and single-use compound forwarders.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Drop _codec_kwargs and ASN1Codec.new_encoder/new_decoder wrappers, stream
BER SEQUENCE children through a nested context, and avoid SEQUENCE OF
fragment list slices.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep OPTIONAL child walking in compound.py and move BER/OER/UPER
SEQUENCE, CHOICE, SEQUENCE OF, and PACKET implementations into
compound_{ber,oer,uper}.py.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Drop dead decoder offset/chunk counters, use int.from_bytes/to_bytes for
OER length determinants, and make UPER read_bit avoid redundant arithmetic.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Fold CHOICE/PACKET second-layer helpers into the bound codec hooks, drop
sequence_encode_children and UPER set_remainder, and keep only the shared
OPTIONAL decode walk.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Store UPER input as a byte buffer with a bit cursor instead of one giant
integer, drop per-field kwargs copies, look up CHOICE by tag, and join
OER SEQUENCE OF payloads once.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Large OCTET STRING reads at a non-byte offset use one bulk
int.from_bytes/shift instead of growing an integer per source byte;
delegate whole-octet read_bits to that path and drop duplicate bounds
checks on the small-field integer reader.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Reject SEQUENCE/CHOICE/SEQUENCE OF as SEQUENCE OF field elements with
an explicit error, keep ASN1F_PACKET (used by Kerberos) via UPER context
hooks, and drop misleading compound encode_into/dissect_from_decoder
wrappers so only primitives use the raw-bit API.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace chunked giant-int finalization with a bytearray plus pending-bit
accumulator, speed up Decoder.remaining(), and reject compound SEQUENCE OF
elements only on the UPER path so BER/OER keep master construction behavior.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the per-byte _peek_bits_int loop, keep append_bits on the byte
path except for a trailing partial octet, bulk-shift unaligned
append_bytes, and trim redundant SEQUENCE OF PACKET checks.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Cache ENUMERATED PER value order, avoid fragment bytes copies with
memoryview, replace OER_Exception with OER_Encoding_Error, collapse
redundant ASN1_Error handlers, and restore check_commits.sh so the CI
fix stays out of the ASN.1 PR.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the broken _uper_enum_values cache (sorted the swapped local map),
create memoryviews only for >=16K fragmented payloads, slim the UPER enum
helper, and avoid copying textual BIT STRING padding.

AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
@polybassa
polybassa force-pushed the asn1-oer-uper-contrib branch from c3368cb to 37fc5bf Compare September 3, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants