send() and sendp() not working on PPPoE interface

[cinit]

Brief description

send() and sendp() sets invalid protocol when using AF_PACKET on PPPoE interfaces, causing packets to be dropped by PPP driver (skb->protocol = 0).

Scapy version

2.7.0.dev46

Python version

Python 3.13.5, using IPython 8.35.0

Operating system

Linux 6.12.74+deb13+1-amd64

Additional environment information

When using send(with ScopedIP) and sendp() on PPPoE (ppp0) interface, Scapy sends packets via AF_PACKET without correctly setting the L3 protocol (sll_protocol). As a result, packet_snd assigns skb->protocol = 0 and causes packets to be dropped by PPP driver that rely on skb->protocol (in this case, ppp_start_xmit).

How to reproduce

Create a PPPoE link interface. Make sure it's type 512.

cat /sys/class/net/ppp0/type 
512

Send an IPv6 ICMPv6 echo request by send() and sendp().

>>> pkt = IPv6(dst="ff02::1%ppp0")/ICMPv6EchoRequest()
>>> pkt.show()
###[ IPv6 ]###
  version   = 6
  tc        = 0
  fl        = 0
  plen      = None
  nh        = ICMPv6
  hlim      = 64
  src       = fe80::be24:11c6:ebd1:9865
  dst       = ScopedIP('ff02::1', scope='ppp0')
###[ ICMPv6 Echo Request ]###
     type      = Echo Request
     code      = 0
     cksum     = None
     id        = 0x0
     seq       = 0x0
     data      = b''

>>> sendp(pkt, iface="ppp0")
.
Sent 1 packets.
>>> send(pkt)
.
Sent 1 packets.
>>> 

I also tried pkt = CookedLinux(proto=0x86DD) / IPv6(src="fe80::1", dst="ff02::2") / ICMPv6ND_RS() and pkt = PPP() / IPv6(src="fe80::1", dst="ff02::2") / ICMPv6ND_RS(), none of them worked.

Actual result

Look at tcpdump output, just nothing there.

~# tcpdump -n -i ppp0 'icmp6'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ppp0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes

Look at ifconfig ppp0, there 2 more dropped TX packets.
Before:

~# ifconfig ppp0
ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1492
        inet 100.64.10.201  netmask 255.255.255.255  destination 100.64.0.1
        inet6 fe80::be24:11c6:ebd1:9865  prefixlen 128  scopeid 0x20<link>
        ppp  txqueuelen 3  (Point-to-Point Protocol)
        RX packets 4752172  bytes 3692861302 (3.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4389765  bytes 935034757 (891.7 MiB)
        TX errors 0  dropped 388 overruns 0  carrier 0  collisions 0

After:

~# ifconfig ppp0
ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1492
        inet 100.64.10.201  netmask 255.255.255.255  destination 100.64.0.1
        inet6 fe80::be24:11c6:ebd1:9865  prefixlen 128  scopeid 0x20<link>
        ppp  txqueuelen 3  (Point-to-Point Protocol)
        RX packets 4752398  bytes 3692927445 (3.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4390064  bytes 935094677 (891.7 MiB)
        TX errors 0  dropped 390 overruns 0  carrier 0  collisions 0

Run dropwatch -l kas start + scapy while True:sendp(pkt, iface="ppp0") and it shows the following:

...
1 drops at tcp_validate_incoming+1a3 (0xffffffffa21125e3) [software]
1 drops at __netif_receive_skb_core.constprop.0+144 (0xffffffffa2034584) [software]
1 drops at __netif_receive_skb_core.constprop.0+144 (0xffffffffa2034584) [software]
1 drops at __init_scratch_end+1c80a73c (0xffffffffc100a73c) [software]
12 drops at __init_scratch_end+1c80c9b0 (0xffffffffc100c9b0) [software]
10 drops at __init_scratch_end+1c80c9b0 (0xffffffffc100c9b0) [software]
...

And trace it:

bpftrace -e 'tracepoint:skb:kfree_skb /args->location == 0xffffffffc100c9b0/ { print(kstack); }'
Attaching 1 probe...

        sk_skb_reason_drop+148
        ppp_start_xmit+112
        dev_hard_start_xmit+100
        sch_direct_xmit+159
        __dev_queue_xmit+2540
        packet_sendmsg+4285
        __sys_sendto+479
        __x64_sys_sendto+36
        do_syscall_64+130
        entry_SYSCALL_64_after_hwframe+118


        sk_skb_reason_drop+148
        ppp_start_xmit+112
        dev_hard_start_xmit+100
        sch_direct_xmit+159
        __dev_queue_xmit+2540
        packet_sendmsg+4285
        __sys_sendto+479
        __x64_sys_sendto+36
        do_syscall_64+130
        entry_SYSCALL_64_after_hwframe+118

It seems that ppp_start_xmit+112 is dropping the TX packet.

Trace skb->protocol:

# bpftrace -e '
tracepoint:skb:kfree_skb
{
    printf("protocol=0x%x\n", args->protocol);
}'

protocol=0x800
protocol=0x800
protocol=0x800
protocol=0x800
protocol=0x800
protocol=0x800
protocol=0x0
protocol=0x0
protocol=0x800
protocol=0x0
protocol=0x0
protocol=0x800
protocol=0x0

There are lots of protocol=0x0 when I run scapy while True:sendp(pkt, iface="ppp0")

Packets sent via sendp() on ppp0:

• Are transmitted via AF_PACKET
• End up with skb->protocol = 0
• Are dropped in kernel:

ppp_start_xmit()
  -> ethertype_to_npindex(ntohs(skb->protocol)) -> -1
  -> drop

No packets are visible on the wire.

Expected result

The ping request should at lease appear on tcpdump output, and the ifconfig TX drop counter should not change when I send()/sendp().

Expected something like this (with my simple AF_PACKET test program):

# tcpdump -n -i ppp0 'icmp6'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ppp0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
13:14:02.186611 IP6 fe80::be24:11c6:ebd1:9865 > ff02::1: ICMP6, echo request, id 16494, seq 1, length 16
13:14:02.188890 IP6 fe80::[REDACTED] > fe80::be24:11c6:ebd1:9865: ICMP6, echo reply, id 16494, seq 1, length 16
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

Related resources

packet_snd: https://elixir.bootlin.com/linux/v6.19.11/source/net/packet/af_packet.c#L2939
packet_snd is where skb->protocol assigned.

ppp_start_xmit: https://elixir.bootlin.com/linux/v6.19.11/source/drivers/net/ppp/ppp_generic.c#L1451
ppp_start_xmit is where packets are droped.

My test AF_PACKET program (written by GPT for test purpose only, compiles on debian 13 x86_64 GCC):

// ppp_ping_demo.c
#define _GNU_SOURCE
#include <arpa/inet.h>
#include <errno.h>
#include <linux/if_packet.h>
#include <net/if.h>
#include <netinet/icmp6.h>
#include <netinet/ip6.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <time.h>
#include <unistd.h>
#include <linux/if_ether.h>

static uint16_t csum16(const void *data, size_t len) {
    const uint8_t *p = (const uint8_t *)data;
    uint32_t sum = 0;
    while (len > 1) {
        sum += (p[0] << 8) | p[1];
        p += 2;
        len -= 2;
    }
    if (len) sum += p[0] << 8;
    while (sum >> 16) sum = (sum & 0xFFFF) + (sum >> 16);
    return (uint16_t)(~sum);
}

static uint16_t icmp6_checksum(const struct in6_addr *src,
                               const struct in6_addr *dst,
                               const void *icmp, size_t icmp_len) {
    // Pseudo-header
    struct {
        struct in6_addr src;
        struct in6_addr dst;
        uint32_t len;
        uint8_t zero[3];
        uint8_t next;
    } ph;

    memset(&ph, 0, sizeof(ph));
    ph.src = *src;
    ph.dst = *dst;
    ph.len = htonl((uint32_t)icmp_len);
    ph.next = IPPROTO_ICMPV6;

    // sum pseudo + icmp
    uint32_t sum = 0;
    sum += (~csum16(&ph, sizeof(ph)) & 0xFFFF);
    sum += (~csum16(icmp, icmp_len) & 0xFFFF);

    while (sum >> 16) sum = (sum & 0xFFFF) + (sum >> 16);
    return (uint16_t)(~sum);
}

static void usage(const char *p) {
    fprintf(stderr, "Usage: %s <ifname> <src-ll> <dst-addr>\n", p);
    fprintf(stderr, "Example: %s ppp0 fe80::1 ff02::1\n", p);
}

int main(int argc, char **argv) {
    if (argc != 4) {
        usage(argv[0]);
        return 1;
    }

    const char *ifname = argv[1];
    const char *src_str = argv[2];
    const char *dst_str = argv[3];

    int ifindex = if_nametoindex(ifname);
    if (!ifindex) {
        perror("if_nametoindex");
        return 1;
    }

    // fill in address (supports link local)
    struct in6_addr src, dst;
    if (inet_pton(AF_INET6, src_str, &src) != 1) {
        fprintf(stderr, "invalid src\n");
        return 1;
    }

    char dst_buf[256];
    memset(dst_buf, 0, sizeof(dst_buf));
    // for link-local, add %iface
    if (strchr(dst_str, '%') == NULL &&
        strncmp(dst_str, "fe80:", 5) == 0) {
        snprintf(dst_buf, sizeof(dst_buf), "%s%%%s", dst_str, ifname);
    } else {
        snprintf(dst_buf, sizeof(dst_buf), "%s", dst_str);
    }

    struct sockaddr_in6 dst_in6;
    memset(&dst_in6, 0, sizeof(dst_in6));
    dst_in6.sin6_family = AF_INET6;
    if (inet_pton(AF_INET6, dst_buf, &dst) != 1) {
        // inet_pton not supports %scope
        char tmp[256];
        strncpy(tmp, dst_buf, sizeof(tmp)-1);
        char *pct = strchr(tmp, '%');
        if (pct) *pct = '\0';
        if (inet_pton(AF_INET6, tmp, &dst) != 1) {
            fprintf(stderr, "invalid dst\n");
            return 1;
        }
    }

    // PF_PACKET + SOCK_DGRAM (set skb->protocol here)
    int fd = socket(AF_PACKET, SOCK_DGRAM, htons(ETH_P_IPV6));
    if (fd < 0) {
        perror("socket(AF_PACKET)");
        return 1;
    }

    // set dest ll address
    struct sockaddr_ll sll;
    memset(&sll, 0, sizeof(sll));
    sll.sll_family   = AF_PACKET;
    sll.sll_ifindex  = ifindex;
    sll.sll_protocol = htons(ETH_P_IPV6);
    // PPP has no L2 addr, so no sll_addr

    // construct IPv6 + ICMPv6 Echo Request
    uint8_t buf[1500];
    memset(buf, 0, sizeof(buf));

    struct ip6_hdr *ip6 = (struct ip6_hdr *)buf;
    struct icmp6_hdr *icmp = (struct icmp6_hdr *)(buf + sizeof(struct ip6_hdr));

    const char payload[] = "ppp-demo";
    size_t payload_len = sizeof(payload) - 1;

    // IPv6 header
    ip6->ip6_flow = htonl((6 << 28) | 0); // version + tc/flow
    ip6->ip6_plen = htons(sizeof(struct icmp6_hdr) + payload_len);
    ip6->ip6_nxt  = IPPROTO_ICMPV6;
    ip6->ip6_hlim = 255; // for ND/LL. hlim should be 255
    ip6->ip6_src  = src;
    ip6->ip6_dst  = dst;

    // ICMPv6 Echo
    icmp->icmp6_type = ICMP6_ECHO_REQUEST;
    icmp->icmp6_code = 0;
    icmp->icmp6_id   = htons((uint16_t)getpid());
    icmp->icmp6_seq  = htons(1);

    uint8_t *data = (uint8_t *)(icmp + 1);
    memcpy(data, payload, payload_len);

    // checksum
    icmp->icmp6_cksum = 0;
    icmp->icmp6_cksum = htons(icmp6_checksum(&src, &dst, icmp,
                                  sizeof(struct icmp6_hdr) + payload_len));

    size_t pkt_len = sizeof(struct ip6_hdr) + sizeof(struct icmp6_hdr) + payload_len;

    // sendp
    ssize_t n = sendto(fd, buf, pkt_len, 0,
                       (struct sockaddr *)&sll, sizeof(sll));
    if (n < 0) {
        perror("sendto");
        close(fd);
        return 1;
    }

    printf("sent %zd bytes on %s (proto=0x86dd)\n", n, ifname);
    close(fd);
    return 0;
}

[cinit]

Here's a workaround. socket.socket(socket.AF_PACKET, socket.SOCK_DGRAM) + s.bind((iface, socket.ETHERTYPE_IPV6)) + s.send(raw(pkt[IPv6])) can be used to send IPv6 packets on PPP link.

    def _native_send_ipv6(self, pkt: Packet, iface: str, is_multicast: bool) -> None:
        # make sure it's IPv6 packet
        if not pkt.haslayer(IPv6):
            LOG.warning("attempted to send non-IPv6 packet: %s", pkt.summary())
            return
        # 1. chek interface link type, raw IP or Ethernet
        ifarphdr = self._arphdr_type(iface)
        is_ethernet = ifarphdr == ArphrdType.ARPHRD_ETHER
        # 2. if raw IP, sendp as is; if Ethernet, add an Ethernet header with proper src and dst MAC addresses
        ret = None
        if is_ethernet:
            # ... skipped here
            ret = sendp(ether, iface=iface, verbose=False)
        elif ifarphdr == ArphrdType.ARPHRD_PPP:
            # PPP workaround, scapy has bug transmitting AF_PACKET on PPP link
            # We need to use scapy to build the L3 IPv4 packet
            # and manually send with AF_PACKET sll_proto=ETH_P_IPV6(0x86DD)
            l3_payload = raw(pkt[IPv6])
            # The addresses are represented by the tuple (ifname, proto[, pkttype[, hatype[, addr]]])
            sll = (iface, socket.ETHERTYPE_IPV6)
            with socket.socket(socket.AF_PACKET, socket.SOCK_DGRAM) as s:
                s.bind(sll)
                s.send(l3_payload)
        # elif other cases ...

    def _arphdr_type(self, ifname: str) -> Optional[ArphrdType]:
        try:
            with open(f"/sys/class/net/{ifname}/type", "r", encoding="utf-8") as f:
                t = int(f.read().strip())
                if t == ArphrdType.ARPHRD_ETHER.value:
                    return ArphrdType.ARPHRD_ETHER
                elif t == ArphrdType.ARPHRD_PPP.value:
                    return ArphrdType.ARPHRD_PPP
                elif t == ArphrdType.ARPHRD_VOID.value:
                    return ArphrdType.ARPHRD_VOID
                else:
                    raise ValueError(f"unsupported ARPHRD type {t} for interface {ifname}")
        except OSError as e:
            # unexpected
            LOG.error("failed to read ARPHRD type for interface %s: %s", ifname, e)
            raise e