IPA Issue - Unable to learn MAC address of ethernet client

Problem

• When testing the downlink throughput using iperf, we found that traffic can not go through the IPA.
• After checking the ip4_rt and hdr in the kernel debugfs, there is no client information inside them.

Debug Messages Analysis

• In order to find out the reason, we enable the log message of ipacm. From the log we can observe that some errors occurred when calling the IOCTL(IPA_IOC_QUERY_INTF).
• In the IOCTL handler of IPA driver, it will return the error -1 if it can not find the registered interface.

ipacm log

IPACM_IfaceManager.cpp:362 create_iface_instance() Creating ODU interface in router mode
IPACM_Iface.cpp:85 IPACM_Iface() dev_name: eth0 virtual_iface: false phy_dev_name:  
IPACM_Iface.cpp:566 query_iface_property() iface query-property 
IPACM_Iface.cpp:582 query_iface_property() iface name eth0
IPACM_Iface.cpp:96 IPACM_Iface()  create iface-index(25) constructor
IPACM_Lan.cpp:139 IPACM_Lan() Mem-setting iface_ul_firewall of size 268
IPACM_Lan.cpp:188 IPACM_Lan()  IPACM->IPACM_Lan(25) constructor: Tx:0 Rx:0 
IPACM_Lan.cpp:244 IPACM_Lan() Device type 2
IPACM_Lan.cpp:308 IPACM_Lan() Need to add 0 IPv4 and 0 IPv6 routing rules for eth bridge for each client.
IPACM_Iface.cpp:1185 delete_iface() netdev (eth0):ipa_index (25) instance close 
IPACM_EvtDispatcher.cpp:135 ProcessEvt()  Find matched registered events IPA_USB_LINK_UP_EVENT
IPACM_EvtDispatcher.cpp:140 ProcessEvt()  Finished process events IPA_USB_LINK_UP_EVENT
IPACM_EvtDispatcher.cpp:144 ProcessEvt() free the event:IPA_USB_LINK_UP_EVENT data: 0xa38011c0
IPACM_CmdQueue.cpp:186 Process() Waiting for Message
IPACM_Netlink.cpp:635 ipa_nl_decode_nlmsg() Received msg:16 from netlink
IPACM_Netlink.cpp:648 ipa_nl_decode_nlmsg() Got RTM_NEWLINK with below values
IPACM_Netlink.cpp:649 ipa_nl_decode_nlmsg() RTM_NEWLINK, ifi_change:0
IPACM_Netlink.cpp:650 ipa_nl_decode_nlmsg() RTM_NEWLINK, ifi_flags:69699
IPACM_Netlink.cpp:651 ipa_nl_decode_nlmsg() RTM_NEWLINK, ifi_index:11
IPACM_Netlink.cpp:652 ipa_nl_decode_nlmsg() RTM_NEWLINK, family:7
IPACM_Netlink.cpp:1654 ipa_get_if_name() Interface index 11
IPACM_Netlink.cpp:1664 ipa_get_if_name() interface name eth0
IPACM_Netlink.cpp:724 ipa_nl_decode_nlmsg() Got a usb link_up event (Interface eth0, 11) 
IPACM_Netlink.cpp:731 ipa_nlioctl IPA_IOC_QUERY_INTF failed: Operation not permitted

IOSS (IPA Offload Sub-System)

• The new framwork IOSS is used to manage 3rd ethernet driver to attach to IPA, and we can check the related log from ipc_logging mechanism.
• In the IOSS log, it shows that the qcom,ioss property can not be matched when parsing the device tree, so the IPA registration was skipped.

ioss log

# cat /sys/kernel/debug/ipc_logging/ioss/log
[     6.131883872/         0xb600bd8]  ioss_log_init:74 cfg:IOSS version 0x1000000, API version 6
[     6.132167726/         0xb60212e]  ioss_parse_dt:32 dbg:DDR bandwidth limit set to 5000
[     6.140466997/         0xb628f94]  ioss_probe:109 cfg:Root device probe completed
[     7.040227625/         0xc6a29a0]  __ioss_pci_register_driver:346 cfg:Registering PCI driver qps615_ioss
[    12.150249619/        0x12433d46]  __ioss_pci_register_driver:346 cfg:Registering PCI driver r8125_ioss
[    22.514692828/        0x1e1fb283]  ioss_ipa_ready_notif:43 cfg:IPA is ready
[    22.514704078/        0x1e1fb35b]  ioss_pci_start:94 cfg:Starting PCI
[    22.514709026/        0x1e1fb3ba]  __pcie_walk_add_device:78 dbg:PCIe walk add device: 0000:00:00.0, driver: pci
[    22.514712776/        0x1e1fb402]  __pcie_walk_add_device:78 dbg:PCIe walk add device: 0000:01:00.0, driver: r8125
[    22.514715328/        0x1e1fb433]  pci_bus_notification:59 dbg:PCIe bus notif: 5 device: 0000:01:00.0, driver: r8125
[    22.514722776/        0x1e1fb4c2]  ioss_pci_add_device:18 cfg:Adding device 0000:01:00.0
[    22.514733036/        0x1e1fb587]  ioss_of_parse:191 dbg:(0000:01:00.0) DT prop qcom,ioss not found, skipping.
[    22.514745589/        0x1e1fb678]  ioss_bus_register_idev:430 ERR:(0000:01:00.0) Failed to parse devicetree
[    22.514755328/        0x1e1fb733]  ioss_pci_add_device:28 ERR:(0000:01:00.0) Failed to register device with ioss bus

Solution

• When the ethernet device is detected by PCI bus, it will notify the IOSS subsystem to process the registration, but since the device does not have the IPA related property, the registration will fail.
• The device tree for ethernet device (RTL8125) is sdxlemur-mtp-mbb-pcie-rc-ep.dtsi, including this DTSI file in the sdxlemur-v2-mtp-cpe.dts makes it have the qcom,ioss property.

sdxlemur-mtp-mbb-pcie-rc-ep.dtsi

...
    realtek,rtl8125@pcie0_rp {
        reg = <0 0 0 0 0>;

        compatible = "qcom,ioss-v2-device";
        qcom,ioss = <&ioss>;
        qcom,ioss_instance = <0>;
        qcom,ioss_channels =
            <&ioss_r8125_eth0_rx>,
            <&ioss_r8125_eth0_tx>;

        /* ipa shares this group with rtl */
        qcom,iommu-group = <&ipa_eth_group1>;

        #address-cells = <1>;
        #size-cells = <1>;

        pci-ids =
            "10ec:8125",
            "10ec:3000";
    };
...