PyPCAPKit Enhancement Proposals

[JarryShaw]

As PyPCAPKit reaches its 16k lines of code and 800th commit, I figure it would be a better idea to record the project enhancement proposals here in the discussion thread. The proposals and/or notes will be documented and maintained here.

Pull requests for the existing proposals and any new ideas are highly welcomed and encouraged. Should you have any questions, please leave a note either in this thread or under the Q&A category discussions.

Wish you enjoy PyPCAPKit!!!

[JarryShaw]

More Protocols, More!!!

As you may have noticed, there are some protocol-named files under the NotImplemented folders. These protocols are what I planned to implement but not yet done. Namely, grouped by each TCP/IP layer and ordered by protocol name alphabetically,

• Link Layer: DSL, EAPOL, FDDI, ISDN, PPP
• Internet Layer: ECN, ESP, ICMP, ICMPv6, IGMP, NDP, Shim6
• Transport Layer: DCCP, QUIC, RSVP, SCTP
• Application Layer: BGP, DHCP, DHCPv6, DNS, IMAP, LDAP, MQTT, NNTP, NTP, ONC/RPC, POP, RIP, RTP, SIP, SMTP, SNMP, SSH, Telnet, TLS/SSL, XMPP

Specifically, I have attempted to implement ESP several years ago, and I abandoned the implementation in the NotImplemented folder due to some design flaws within PyPCAPKit at that time. But now, the protocol should be able to implement quite smoothly.

More over, MH (pcapkit.protocols.internet.mh.MH) requires some help to implement all the message data types, you can find more information in the specific file.

Also, for the existing protocols, I am looking for a helping hand to implement the construction features, as defined in the Protocol.make method. You can find some reference from the PCAP's Frame header class. See comment below.

[JarryShaw]

After merging the test/rc/schema branch, all protocols are now by nature designed and implemented with support of both parsing and construction - through the specified Schema classes. Therefore, when creating new protocols, it is only required to create the schema definition first, and then create the Protocol.read and Protocol.make functions accordingly.

[JarryShaw]

Three more have landed since that reply, and the two singled out in the original post are both done.

SCTP (#379) — a first-class transport protocol per :rfc:9260: the common header, all thirteen chunk types the RFC defines, chunk parameters, error causes, and CRC32c verification. Registered-but-unimplemented chunk types, parameters and error causes fall through to the generic handlers rather than failing the extraction.

One way it differs from its siblings, worth knowing: the next layer is dispatched on the DATA chunk's payload protocol identifier through register_sctp(), not on port numbers — so register_apptype() deliberately does not fan out to it.

ESP (#378) — the one this post described as abandoned in NotImplemented for years. It parses without keys, and decrypts when a Security Association is supplied through the new protocol-keyed pcapkit.corekit.context channel. Still wanted there is wider algorithm coverage: ChaCha20-Poly1305 (:rfc:7634), AES-CCM (:rfc:4309), AES-XCBC integrity (:rfc:3566) and Extended Sequence Numbers are not implemented — only 5 encryption and 5 integrity algorithms are actually applied.

Mobility Header — the FMIPv6 fast-handover messages and options landed in #383, so "requires some help to implement all the message data types" is now narrower rather than open-ended. What remains: 10 of 24 message data types, 51 of 71 options, and 3 of 4 CGA extensions. Construction was also wholly broken for MH, HOPOPT and IPv6-Opts until #398 — HOPOPT(next=…, options=[…]) raised ProtocolError for any option set, including none — so building these headers now actually works.

The NotImplemented protocol list in the original post is otherwise unchanged and still accurate, with two corrections: NDP's stub is under link/, not internet/, and QUIC has no stub in the tree at all.

If you are looking for somewhere to start, #251 is the most useful guide — it walks through the schema, data model, skeleton and registration for adding a protocol end to end.

[JarryShaw]

Two protocol-shaped gaps that are not on the list in this topic, plus two corrections to my reply above. All of it verified against the code rather than against the docs; the Help Wanted page now carries the detail and is the canonical list.

DTLS

There is no dtls.py stub anywhere in the tree — only TLS/SSL has one — so DTLS never appeared on the list above. It has earned its own entry, because the registries have moved ahead of it and a growing part of SCTP's surface now names a protocol that does not exist:

• the DTLS chunk, type 65 of Chunk, and the DTLS Key Management chunk parameter, 0x8006 — both fall through to the generic handlers, so they parse as opaque;
• four of its error causes, 100 to 103;
• seven payload protocol identifiers: 47 (Diameter over DTLS/SCTP), 66 to 69 (NGAP, XnAP, F1AP and E1AP, each over DTLS over SCTP) and 4242 (DTLS chunk key management).

That last group is the one that bites. NGAP over DTLS over SCTP is PPID 66, and it is blocked on DTLS rather than on NGAP. Even with an NGAP dissector registered on 66, the bytes arriving are a DTLS record rather than an NGAP PDU, so they can only degrade to Raw. A record-layer dissector is enough to unblock it — content type, version, epoch, sequence number, length, and the fragment offset and length DTLS adds to handshake messages. Decryption is a separate question and is not needed to make the records legible, in the same way ESP parses without keys. DTLS over UDP is the other half of the same work and wants the same dissector. Also noted in #251, where someone tracking NGAP would look for it.

Registered, but not dissected

A different shape of gap from the empty stubs, and invisible until someone hits it. The pcapkit.const.reg enumerations are complete, but only a small part of each is bound to a dissector; everything else resolves to Raw, so a capture parses without complaint and yields nothing useful.

• 3 of the 219 LinkType values — ETHERNET, IPV4, IPV6, declared identically in Frame and PCAPNG
• 15 of the 151 TransType values
• 6 of the 160 EtherType values — ARP, RARP, IPv4, IPv6, IPX and the customer VLAN tag
• 2 port numbers out of 8182 AppType members — TCP 21 to FTP, and port 80 to HTTP on both TCP and UDP
• none of the 75 SCTP payload protocol identifiers: SCTP.__proto__ ships empty, so every DATA chunk payload is Raw until something calls register_sctp

Most of that needs a dissector written and is covered by the stub list above. A handful needs only a table entry, because the dissector already exists:

• OSPF and L2TP are implemented but reachable from no registry at all — TransType 89 and 115 are both unbound, and each class deliberately makes __index__ raise, so registering them means settling that question first;
• FTP_DATA is implemented and exported, but TCP port 20 is unbound;
• HTTP is bound on port 80 only, not on 8080 or 8443;
• the service VLAN tag identifier (S-Tag), 0x88A8, is unbound, though VLAN already parses that shape and the customer tag 0x8100 is bound to it;
• LinkType NULL, LOOP and RAW carry bare IPv4 or IPv6, both of which pcapkit dissects — NULL and LOOP need their four-octet address family word skipped first, RAW needs a version sniff.

Beyond those, the gaps most likely to be met in a real capture are ICMP (1), ICMPv6 (58) and IGMP (2) on the internet layer, all three of which have stubs; and LINUX_SLL / LINUX_SLL2 at the link layer, which every tcpdump -i any capture produces and which have no stub.

Two corrections to the progress reply above

ESP. "Only five encryption and five integrity algorithms are actually applied" is right, but two of those ten are the no-ops ENCR_NULL and NONE, so the real coverage is AES-CBC and AES-GCM at all three tag lengths, plus HMAC-SHA1-96 and the three RFC 4868 HMAC-SHA2 truncations. The "not implemented" list was also incomplete — it should include AES-CTR (RFC 3686), AES-CMAC-96 (RFC 4494), the AES-GMAC family (RFC 4543) with its ENCR_NULL_AUTH_AES_GMAC counterpart, and the Camellia, implicit-IV (RFC 8750) and MGM (RFC 9227) families. Two structural notes for anyone picking this up: IntegritySuite records the digest as the name of a hashlib constructor, so AES-XCBC and AES-CMAC need the record widened rather than a row added; and load_cryptography imports only the ciphers submodules, so the AEAD and CMAC primitives have to be added there before a suite can reach them.

Mobility Header. The three counts (10 of 24 messages, 51 of 71 options, 3 of 4 CGA extensions) all check out. But the # TODO markers do not sit at the dispatch tables as claimed — there are six of them and they sit at the end of each handler block, while the tables are elsewhere in the file, so both places need editing for each addition. Read and construction are symmetric throughout (37 handlers each way), so every gap is a gap in both directions. And less of it is groundwork than the numbers suggest: mh.py already imports 32 const/mh sub-registry enums without using them, so the enumerations the missing options need — binding revocation types and triggers, handoff indicators, ANI sub-options, flow identification and flow binding sub-options, LMA-controlled MAG parameters, DNS update status, traffic selector formats, QoS attributes — are already generated in full. What is missing is the handlers, not the enumerations.

One small correction to the option grouping while I am here: NEMO's Mobile Network Prefix (option 6) is implemented, and DNS-UPDATE-TYPE (17), Vendor Specific (19) and Service Selection (20) are missing but belong to none of the named blocks.

[JarryShaw]

PyPCAPKit now has built-in support for PCAP-NG file format starting from version 1.0.0.

[JarryShaw]

After reviewing the source code of Scapy, DPKT, etc., and the protocol specifications, this approach seems to be unrealistic - it is difficult to determine the length of some fields as they may depend on the values of preceding fields. Thus, I decided to discard this proposal.

[JarryShaw]

Done. Logging is now integrated throughout, in #384.

Every module logs through its own child logger under a hierarchy rooted at pcapkit, obtained from get_logger(__name__), so a record carries the name of the code that emitted it and any subtree can be addressed on its own:

import logging
logging.getLogger('pcapkit.foundation.registry').setLevel(logging.WARNING)   # quieten bookkeeping
logging.getLogger('pcapkit.foundation.extraction').setLevel(logging.DEBUG)   # follow extraction

There is also a public configuration API — configure(), reset(), get_logger() and ensure_output().

One behaviour change worth knowing before upgrading. Importing pcapkit no longer configures logging. It used to attach a StreamHandler on sys.stderr and force the level at import time, which hijacked the logging setup of every application that imported it; only a NullHandler is attached now, as is conventional for a library. So messages that used to appear on stderr no longer do. One line restores the old behaviour:

from pcapkit.utilities.logging import configure
configure(logging.INFO, stream=sys.stderr)

Registration bookkeeping also moved from INFO to DEBUG — a library announcing its own registry entries is not news to its consumer. Extractor(verbose=True) and the CLI's -v are unaffected and still print a line per frame to stdout.

At DEBUG the library explains what it did with a file — which input was opened, which engine was requested and which actually ran, the format identified from the magic number, whether reassembly and flow tracing were enabled, how many frames were read — without descending into per-field parsing, so enabling it on a million-packet capture does not produce a million records.

[JarryShaw]

Both candidates are done, and a third engine came out of the work.

pypcap and pypcapfile landed in #386, each with a matching pcapkit.toolkit adapter, selected with engine='pypcap' and engine='pypcapfile'. As the thread anticipated, no Extractor handler methods were needed — an engine now subclasses Engine and implements run() and read_frame(), and subclassing registers it.

A third engine, pcap_ct (#405), because upstream pypcap turned out to be unusable on a current interpreter. Measured with libpcap present and found: it builds on 3.10 and 3.11 and fails on 3.12 and 3.14, since its 1.3.0 sdist ships a pcap.c pre-generated by Cython 0.29.32 that no longer compiles against the 3.12+ C API. pcap-ct reimplements the same API over ctypes and works across the whole supported range.

It is a separate engine rather than a second backend, because the two are mutually exclusive: both distributions own the import name pcap, and with both installed pcap-ct wins the import while upstream's extension module is shadowed and unreachable. Each engine detects which backend it actually got and says so, rather than one engine= string silently meaning two different implementations.

Speed is not free, and the gaps are announced rather than silent. pypcap and pcap_ct perform no protocol dissection at all, so they disable reassembly and flow tracing; pypcapfile has no IPv6 decoder, so IPv6 reassembly is unavailable while IPv4 and TCP still work. Each gap raises a warning or an exception instead of quietly returning nothing.

Every engine also answers a preflight check, unsupported_reason() (#396, extended in #405), so asking for one that cannot run gives a single warning naming the real cause — a Python version, a missing tshark, a missing libpcap, the wrong pcap distribution — and a clean fall back to pcapkit's own parser, rather than an error from inside the third-party package.

Two things worth knowing if you plan to use them:

• Python 3.11 is the last version on which every engine runs, and even there pypcap and pcap_ct cannot coexist — so no single environment ever has all seven at once.
• pypcap and pypcapfile are gated below 3.12 in the extras, and pypcap is deliberately not in the all extra: it is an sdist-only C extension, so including it would demand a compiler and the libpcap development files from everyone running pip install pypcapkit[all].

[JarryShaw]

Test Cases

PyPCAPKit still does not have a systematic testing suite to be bundled with it. The only test cases I have worked out are those in the /tests folder - mostly functional tests. As PyPCAPKit is growing bigger and bigger, a comprehensive test suite is coming much more of demand for a more reliable development process.

[JarryShaw]

Largely done. There is now a systematic suite under tests/ — 91 modules matching test_*.py — running in CI against Python 3.10 through 3.14, plus an allowed-to-fail 3.15 leg (#340, #376, #393).

It is split by what a test needs, not by what it covers, and the split is enforced rather than documented:

• the unit tier may read only captures committed to the repository;
• the fixture-dependent tier — tests/integration/ plus every *_runtime.py and *_regression.py — may also read generated sample captures.

Those samples are not tracked in git, so examples/generators/make_samples.py (make samples) rebuilds them from source. A guard in tests/_tiers.py (#393) refuses a unit-tier module that reads a generated capture, deciding on the calling module's tier rather than on whether the file happens to exist — so it fails on the machine that made the mistake instead of on the next fresh clone. That trap had produced local-pass/CI-fail three separate times before the guard existed.

Committedness is asked of git rather than hardcoded, which mattered immediately: examples/captures/ has six tracked files, not the two captures one would assume.

What remains wanted is coverage, not infrastructure — the unimplemented protocols and registered-but-unhandled type codes have no tests because they have no implementation.

One correction to the original ask, since it said the suite should be bundled with the distribution. It is in the sdist (all 91 modules) and deliberately not in the wheel. That is not an oversight: the suite could not run from an installed package anyway, because the generated captures are not shipped and tests/_tiers.py resolves paths from a repository root an installed package does not have. Anyone wanting to run the tests wants the repository, which is where they are.

[JarryShaw]

Reassembly Beyond IP and TCP

A new topic rather than a reply, because nothing in this thread covers reassembly or flow tracing — the existing topics are protocols, PCAP-NG, performance, logging, engines and tests. Detail on the Help Wanted page, which is the canonical list and is kept in step with the code.

pcapkit.foundation.reassembly covers three protocols and no more. IPv4 and IPv6 share the RFC 791 procedure, and TCP uses the RFC 815 hole-descriptor algorithm, which does handle out-of-order and overlapping segments. SCTP has nothing — a user message split across DATA chunks is never put back together, and sctp appears nowhere in pcapkit/foundation/ at all.

The dissector is not the problem. DATAChunk already exposes everything a reassembler needs — tsn, stream_id, stream_seq, ppid, data, and flags.B / flags.E for the beginning and ending fragment bits — and nothing reads those two bits outside the construction path. So each fragment is dispatched on its own PPID as though it were a whole PDU, and a registered upper layer is handed half a message. Two related holes compound it:

• only the first DATA chunk of a bundle is dispatched at all;
• I-DATA (RFC 8260, chunk type 64) is not dissected, so the MID and FSN fields that exist precisely to make interleaved reassembly tractable are never parsed — and an I-DATA-only packet yields no payload whatsoever. FORWARD TSN (192) and I-FORWARD-TSN (194) are unhandled too, so partial-reliability stream advancement is invisible.

Writing the reassembler is the smaller half of the job. It would go at pcapkit/foundation/reassembly/sctp.py, subclass Reassembly and implement two methods, reassembly() and submit() — those are the only abstract ones. But registering it is inert as things stand: Extractor.register_reassembly accepts any protocol name, while Extractor only ever instantiates the three it names literally, and ReassemblyManager is a fixed-field Info container with no room for a fourth. So adding SCTP also means touching that manager and its read-side twin, the construction branch in Extractor, one adapter per engine in pcapkit.toolkit, the call site in each of those engines, and interface.core.reassemble.

Generalising that wiring so a registered reassembler is actually reached looks worth doing on its own account — it would make the fourth protocol cheaper than the third rather than dearer, and the same change unblocks the next item.

Two smaller things in the same subsystem:

• Flow tracing is TCP only, blocked on the same generalisation — TraceFlowManager holds a single field, so UDP, SCTP and IP conversation tracing have nowhere to go. The TCP tracer itself closes a flow on FIN but never on RST, which is not in its packet model at all, and treats each direction of a connection as a separate flow.
• Nothing ever times a partial datagram out. RFC 791 gives IP reassembly 15 seconds and RFC 8200 gives IPv6 60; neither is implemented, and neither can be until the buffer models carry a timestamp. A buffer is released only when its datagram completes or its flow is torn down, and every in-flight IP datagram identifier holds a fixed 72 KiB of preallocated space — so a lossy capture, or one with spoofed identifiers, grows the buffer monotonically.

Worth knowing before benchmarking against them: reassembly is not merely slower on some engines but unavailable. pyshark, pypcap and pcap_ct disable it entirely, and pypcapfile disables the IPv6 half of it.