---
# System prepended metadata

title: '[Streaming] 使用GStreamer Python API解析RTP與RTCP封包'
tags: [RTCP, rtcp, RTSP, Streaming, streams, rtp, GStreamer, RTP]

---

# 使用GStreamer Python API解析RTP與RTCP封包

---
## Streaming 系列筆記
- [[Streaming] GStreamer簡介與筆記](https://hackmd.io/@YungHuiHsu/ryhRTZpt3)
- [[Streaming] 使用GStreamer Python API解析RTP與RTCP封包](https://hackmd.io/@YungHuiHsu/ryEMq3BT3)
- [How to read and process multiple camera streams simultaneously in dataloader。如何使用pytorch dataloader迭代同步讀取和處理多個視訊(相機/錄影)串流](https://hackmd.io/@YungHuiHsu/BJr5SyYQn)

更多deepstream的實戰應用:
- [Accelerate multi-streaming cameras with DeepStream and deploy custom (YOLO) models<br>使用DeepStream加速多串流攝影機並部署客製(YOLO)模型](https://hackmd.io/@YungHuiHsu/rJKx-tv4h)
- [Use Deepstream python API to extract the model output tensor and customize model post-processing (e.g., YOLO-Pose)<br>使用Deepstream python API提取模型輸出張量並定製模型后處理（如：YOLO-Pose）](https://hackmd.io/@YungHuiHsu/rk41ISKY2)

---
## 如何使用GStreamer元件解析RTP與RTCP內的資料

主要依賴gstreamer中的[rtpbin](https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpbin.html?gi-language=python#rtpbin)元素實踐

![](https://hackmd.io/_uploads/Hk2oPP8b6.png =1200x)

![](https://hackmd.io/_uploads/rJjgCELWa.png  =400x)

RTP bin結合了`rtpsession`、`rtpssrcdemux`、`rtpjitterbuffer`和`rtpptdemux`的功能，將它們集成為一個元素。它允許多個RTP會話，這些會話將使用RTCP SR封包進行同步。

 

`rtpbin`配置了一些請求端口(request pads)，這些端口定義了要啟動的功能，類似於`rtpsession`元素。
- 使用`rtpbin`進行**RTP**接收與發送
    - 接收RTP封包
        - 需請求一個`recv_rtp_sink_%u`端口
            - 必須在端口名稱中指定會話號碼。
        - 在`recv_rtp_sink_%u`端口上接收的數據將在`rtpsession`管理器中進行處理，經過驗證後轉發到`rtpssrcdemux`元素。
        - 每個RTP流都是基於SSRC進行解多路復用的，然後發送到`rtpjitterbuffer`。在從抖動緩衝區釋放數據包後，它們將被轉發到`rtpptdemux`元素。
        - `rtpptdemux`元素將根據有效載荷類型解多路復用數據包，並將會話號碼、SSRC和載荷類型分別作為端口名稱，在`rtpbin`上創建一個唯一的pad `recv_rtp_src_%u_%u_%u`。
    - 發送RTP封包
        - 需請求(request)一個`send_rtp_sink_%u`端口
        - 它將自動創建一個`send_rtp_src_%u`端口。如果未提供會話號碼，則將返回最低可用會話的端口。
        - 會話管理器將修改RTP封包中的SSRC為其自己的SSRC，然後在更新其內部狀態後將封包轉發到`send_rtp_src_%`u端口。
 
-  使用`rtpbin`進行**RTCP**接收與發送
    - 接收RTCP封包
        需請求(request)一個`recv_rtcp_sink_%u`端口。必須在端口名稱中指定會話號碼。
    - 發送RTCP封包
        如果希望會話管理器生成並發送RTCP封包，需請求一個帶有會話號碼的`send_rtcp_src_%u`端口。在此端口上推送的封包包含應發送給會話中所有參與者的SR/RR RTCP報告。


使用`get-internal-session`屬性來訪問`rtpbin`的內部統計信息。此動作信號提供對RTPSession對象的訪問，RTPSession對象進一步提供了獲取內部源和其他源的動作信號。

### Gstreamer範例Pipeline
 
 - 範例1：單純使用`rtpbin`接收RTP packets
     接收來自端口5000的RTP數據，並發送到`rtpbin`中的session 0
     ```bash=
     gst-launch-1.0 udpsrc port=5000 caps="application/x-rtp, ..." ! .recv_rtp_sink_0 \
         rtpbin ! rtptheoradepay ! theoradec ! xvimagesink
     ```

 
- 範例2： 視訊、音訊分別接收並發送RTP與RTCP packets 
   - RTP packets管道
       - 將來自v4l2src的H263video編碼並打包，從audiotestsrc生成AMR audio並打包。
       - video
           - 發送到rtpbin中的session 0，然後至`udpsink`的port 5000
       - audio
           - 發送到rtpbin中的session 1，然後至`udpsink`的port 5002

   - RTCP packets管道
       - video
           - session 0的RTCP packets會被發送到port 5000+1，在port 5005上接收
       - audio
           - session 1的RTCP packets會被發送到port 5002+1。在port 5007上接收
           - 在port 5007上接收
       由於發送者的RTCP數據包應該盡快發送並且不參與在port 5005上接收，因此在`udpsink`上配置`sync=false`和`async=false`

    ```bash=
     gst-launch-1.0 rtpbin name=rtpbin \
             v4l2src ! videoconvert ! ffenc_h263 ! rtph263ppay ! rtpbin.send_rtp_sink_0 \
                       rtpbin.send_rtp_src_0 ! udpsink port=5000                            \
                       rtpbin.send_rtcp_src_0 ! udpsink port=5001 sync=false async=false    \
                       udpsrc port=5005 ! rtpbin.recv_rtcp_sink_0                           \
             audiotestsrc ! amrnbenc ! rtpamrpay ! rtpbin.send_rtp_sink_1                   \
                       rtpbin.send_rtp_src_1 ! udpsink port=5002                            \
                       rtpbin.send_rtcp_src_1 ! udpsink port=5003 sync=false async=false    \
                       udpsrc port=5007 ! rtpbin.recv_rtcp_sink_1
    ```
- 範例3： 視訊、音訊分別接收並發送RTP與RTCP packets並撥放
   - RTP packets管道
       - video
           - 使用`udpsrc`接收port 5000上的H263視頻，通過`rtpbin`的`recv_rtp_sink_0`發送到session 0，解包、解碼並顯示影片
       - audio
           - 接收port 5002上的AMR音檔，通過`rtpbin`的`recv_rtp_sink_1`發送到session 1，解包、解碼並播放
   - RTCP packets管道
        這些RTCP 封包的資料將經過以下管道用於會話管理和同步
       - video
           - 通過`rtpbin`的`recv_rtcp_sink_0`以port 5000+1接收session 0的RTCP封包
           - 再發送session 0的RTCP報告到port 5005
       - audio
           - 通過`rtpbin`的`recv_rtcp_sink_0`以port 5002+1接收session 1的RTCP封包
           - 再發送session 1的RTCP報告到port 5007
       
   
    ```bash=
     gst-launch-1.0 -v rtpbin name=rtpbin                                          \
         udpsrc caps="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H263-1998" \
                 port=5000 ! rtpbin.recv_rtp_sink_0                                \
             rtpbin. ! rtph263pdepay ! ffdec_h263 ! xvimagesink                    \
          udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0                               \
          rtpbin.send_rtcp_src_0 ! udpsink port=5005 sync=false async=false        \
         udpsrc caps="application/x-rtp,media=(string)audio,clock-rate=(int)8000,encoding-name=(string)AMR,encoding-params=(string)1,octet-align=(string)1" \
                 port=5002 ! rtpbin.recv_rtp_sink_1                                \
             rtpbin. ! rtpamrdepay ! amrnbdec ! alsasink                           \
          udpsrc port=5003 ! rtpbin.recv_rtcp_sink_1                               \
          rtpbin.send_rtcp_src_1 ! udpsink port=5007 sync=false async=false
    ```

---
## 對Deepstream 使用`uridecodebin` 進行客製修改
- 對應Deepstream中使用`uridecodebin`元素作為接收rtsp stream的工具，因此必須根據使用的插件內容來發送訊號、訪問RTCP SR、並捕獲、解析RTCP 封包
    - 當輸入來源為"rtsp://..."時 ，`uridecodebin`內部完整的的element組成
    - 當資料來源是rtsp時，`decodebin`內部會調用`rtspsrc`來解析

    ![](https://hackmd.io/_uploads/SJtYz_6T3.png =1200x)


    ```mermaid 
    flowchart LR

        0[decodebin] --> 1[rtph264depay] --> 2[h264parse] --> 3[capsfilter] --> 4[nvv4l2decoder]

    ```
- 範例中的Video Decode Pipeline
    - `rtspsrc` 負責連接 RTSP 服務器並讀取數據。`rtspsrc`就像一個實時信號源，因此只會在播放狀態下生成數據。

    ![](https://hackmd.io/_uploads/ryWwp4UWT.png =800x)


    - `rtspsrc` 在內部使用`rtpbin`實例化一個 "RTP 會話管理器"(RTP session manager)元件，負責與 rtsp server雙向對話，取得RTCP訊息



- `decodebin`與`rtspsrc`
    | 特性       | decodebin                                                                 | rtspsrc                                                                                       |
    |------------|---------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|
    | 定義       | - `decodebin` 是一個自動解碼器元件 (element)                               | - `rtspsrc` 是一個用於接收 RTSP 流的元件 (element)                                             |
    | 功能       | - 自動檢測和使用適當的解碼器來解碼多媒體數據                               | - 處理 RTSP 協議，從 RTSP 伺服器接收數據                                                       |
    |            | - 可以處理音頻和視頻數據                                                   | - 可以接收 RTP 和 RTCP 數據                                                                    |
    | 用途       | - 用於動態解碼多媒體數據，不需要事先知道數據的格式                         | - 用於從 RTSP 伺服器接收多媒體流                                                               |
    | 連接性     | - 通常連接到數據源 (例如 `filesrc` 或 `rtspsrc`) 來接收未解碼的多媒體數據   | - 通常連接到解碼器 (例如 `decodebin`) 或其他處理 RTP 數據的元件，以獲取解碼後的多媒體數據       |
    | 特殊功能   | - 當它接收到數據時，它會動態地選擇和更改解碼器                             | - 支持 RTSP 的各種功能，如認證、選擇媒體流、控制播放速度等                                      |

### 實作
- [完整sample code](https://gist.github.com/YunghuiHsu/e5222cf09815833ce9db95a598dd9014#file-rtsp_processor-py)

#### 實作-解析RTCPBuffer
- `def on_receiving_rtcp_callback`
    - `rtcp_packet.sr_get_sender_info()`裡面包含了:
        -  ssrc、ntptime、rtptime、packet_count、octet_count等資料
    - 記得要用GstRtp.RTCPBuffer.unmap(rtcp_buffer)釋放記憶體
    ```python=!
        # Process RTCP Packets  ---------------- 
        def on_receiving_rtcp_callback(session, buffer: Gst.Buffer):
            """
            Fetch the NTP and RTP reference Timestamp in the RTCP Sender Report
            Map RTCP and RTP packet data to calculate latency.
            """
            rtcp_buffer = GstRtp.RTCPBuffer()
            if not GstRtp.RTCPBuffer.map(buffer, Gst.MapFlags.READ, rtcp_buffer):
                logging.warning("\t\t[on_receiving_rtcp_callback] : Unable to map RTCP buffer")
                return Gst.PadProbeReturn.PASS
            try:
                rtcp_packet = GstRtp.RTCPPacket()
                if not rtcp_buffer.get_first_packet(rtcp_packet):
                    logging.warning("\t\t[on_receiving_rtcp_callback] : Unable to get the first RTCP packet")
                    return Gst.PadProbeReturn.PASS
                while rtcp_packet:
                    if rtcp_packet.get_type() == GstRtp.RTCPType.SR:
                        rtcp_data = *rtcp_packet.sr_get_sender_info()
                    if not rtcp_packet.move_to_next():
                        break
            finally:  # Add finally block to ensure buffer is unmapped
                GstRtp.RTCPBuffer.unmap(rtcp_buffer)

            return Gst.PadProbeReturn.PASS
    ```
#### 實作-解析RTPBuffer
- `def rtp_depay_sink_pad_buffer_probe`
    - 這個探針函式要靜態插在第一個接收RTP的元件前，名稱通常是`rtpdepay`
    - 記得同樣要在最後面用.unmap()釋放記憶體

    ```python=!
    # Process RTP Packets and Calculate RTSP Latency ----------------------------- 

    def rtp_depay_sink_pad_buffer_probe(
        self, pad: Gst.Pad, info: Gst.PadProbeInfo, u_data: Any
    ) -> Gst.PadProbeReturn:
        """
        Handle the sink pad buffer probe for RTP buffer in depay.

        Parameters:
        - pad (Gst.Pad): The pad to fetch the buffer from.
        - info (Gst.PadProbeInfo): Information about the buffer.
        """
        buffer = info.get_buffer()
        res, rtp_buffer = GstRtp.RTPBuffer.map(buffer, Gst.MapFlags.READ)
        try:
            rtp_data = [
                rtp_buffer.get_timestamp(),
                rtp_buffer.get_ssrc(),
                rtp_buffer.get_payload_type(),
                rtp_buffer.get_marker(),
                rtp_buffer.get_seq()
            ]
        finally:
            rtp_buffer.unmap()
        return Gst.PadProbeReturn.OK
    ```


### 如何在Gstreamer `uridecodebin` 中使用
Sample code modified from [NVIDIA-AI-IOT/deepstream_python_apps/deepstream_test1_rtsp_in_rtsp_out.py](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/apps/deepstream-rtsp-in-rtsp-out/deepstream_test1_rtsp_in_rtsp_out.py)

由於`uridecodebin`內部會自動建立接收rtsp等相關的元件，且內部會使用高階API`rtspsrc`，因此需要建立一個manager的callback函式，讓`rtspsrc`內部`rtpbin`的連接到RTPsession物件，透過`rtspsrc.connect("new manager", new_manager_callback)`這樣的方式連接
 

- `def new_manager_callback`
對應前面所提：`rtspsrc` 在內部使用rtpbin實例化一個 “RTP 會話管理器”(RTP session manager)元件，負責與 rtsp server雙向對話，取得RTCP訊息
    - 先確認`rtpsrc`中的元件為`rtpbin`
    - 為了使用`rtpbin`來作為RTCP接收器，使用`get_request_pad`動態建立`recv_rtcp_sink_%u`接口
    - 確認後發送"get-internal-session"訊號，這個操作可以訪問rtpbin內部的RTPSession object以取得rtpbin內部統計資料，透過連接RTPSession也可以進一步檢索內部Buffer及其他資料源
        > [GStreamer/rtpbin](https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpbin.html?gi-language=c)
        > - use `rtpbin` as an RTCP receiver, request a `recv_rtcp_sink_%u` pad. The session number must be specified in the pad name.
        >- Access to the internal statistics of `rtpbin` is provided with the `get-internal-session` property. This action signal gives access to the RTPSession object which further provides action signals to retrieve the internal source and other sources.
        
    
    ```python!
    def new_manager_callback (rtspsrc, manager, index, *args):
        '''
        Manage RTP sessions for accessing RTCP packets
        '''
        element_name = manager.get_factory().get_name()
        # print(f"\t[new_manager_callback] : List element_name '{element_name}-{index}' in Decodebin ")
        if element_name != "rtpbin":
            # print(f"\t[new_manager_callback] : Manager is of type {element_name}, not rtspsrc. ")
            return

        # Establish RTP session in src pad for fetching RTCP packets
        sinkpad = manager.get_request_pad(f"recv_rtcp_sink_{index}") 
        session = manager.emit("get-internal-session", 0)
        session.connect("on-receiving-rtcp", on_receiving_rtcp_callback)

    ```

- 上面的manager的相關操作等價於下面的操作方式:
    ![](https://hackmd.io/_uploads/B1Qq-uU-T.png =400x)

    ```python=!
    # Receive RTCP Packets -------------------------         
    # Configure rtpbin to handle the RTP/RTCP session
    recv_rtcp_sink_pad = Gst.Element.get_request_pad(rtpbin, f"recv_rtcp_sink_{index}")
    session = rtpbin.emit("get-internal-session", 0)
    session.connect("on-receiving-rtcp", on_receiving_rtcp_callback)
    ```


- `def decodebin_child_added`
    - 找出`uridecodebin`中`rtspsrc`元件，準備連接建立"RTP session manager"以接收RTCP資料
        ```
        [decodebin_child_added] : Element type of 'source-00': 'rtspsrc'
        [decodebin_child_added] : Successfully connected 'on-receiving-rtcp' with 'source-00'.
        ```
    - 找出`uridecodebin`中，名稱有"depay"的元件，插入解析RTP Buffer的探針函示(`rtp_depay_sink_pad_buffer_probe`)以獲取資料
    
    ```python!
    def decodebin_child_added(child_proxy, Object, name, user_data, index):
        print("Decodebin child added:", name, "\n")
        if name.find("decodebin") != -1:
            Object.connect("child-added", new_manager_callback, user_data)

        if "source" in name:
            element_type = Object.get_factory().get_name()
            if element_type == "rtspsrc":
                try:
                    # Use functools.partial to freeze the 'index' parameter for the callback
                    # Object.connect("new-manager", new_manager_callback)
                    Object.connect("new-manager", partial(new_manager_callback, index=index))
                    
        # Add Probe for *depay, which is used to extract video from RTP packets.
        # uridecodebin is a high-level element that dynamically creates its internal pipeline, including elements like rtph264depay,
        # based on the media it's given. This means that the rtph264depay element may not exist until the pipeline is set to the PAUSED
        # or PLAYING state and has detected that the incoming stream is H.264 encoded.
        if ("depay") in name:
            element_type = Object.get_factory().get_name()
            depay_sinkpad = Object.get_static_pad("sink")
            if depay_sinkpad:
                depay_sinkpad.add_probe(
                    Gst.PadProbeType.BUFFER,
                    rtp_depay_sink_pad_buffer_probe,
                    0,
                )
    ```

- `def create_source_bin`
    - 使用`partial(decodebin_child_added, index=index)`連結`decodebin_child_added`函式，方便傳遞`index`參數進入、增加可讀性
    ```python=!            
    def create_source_bin(index, uri):
        bin_name = "source-bin-%02d" % index
        nbin = Gst.Bin.new(bin_name)
        ...
        uri_decode_bin.connect("child-added", partial(decodebin_child_added, index=index), nbin)

        ...
        return nbin
    ```
 



---


## Reference
### Blog and Forum
#### [2022/10。qorela.com。Absolute Timestamp of Arbitrary Frame in RTSP Stream with GStreamer](https://qorela.com/en/blog-detail/absolute-timestamp-of-arbitrary-frame-in-rtsp-stream-with-gstreamer)

:::spoiler

##### 1. 使用`rtspsrc`元素來接收rtsp的uri，並以此發送&quot;new-manager&quot;訊號(emit signal)給`rtpbin`

*   首先，需要捕獲並連接 rtpbin 元素的創建信號，以便訪問它。為此，我們從第 5 行的管道中提取了 rtspsrc 元素。
*   在創建管理器後，該元素會產生一個 &quot;new-manager &quot;信號
*   對於 rtpbin 元素，回調函數與該信號相連，如第 6 行所示。我們將使用在該函數中獲得的會話信息訪問 RTCP SR。
*   訪問 RTP 數據包中的時間值
*   在 rtph264depay 元素的 sink pad 上添加一個探針
*   第 8 行中的 rtph264depay 元素和第 9 行中該元素的 sink pad 已被提取
*   通過在第 10 行獲得的pad(即`rtph264depay`的輸入sink pad)上添加探針來連接該功能

```python=3
pipeline = Gst.parse_launch("rtspsrc name = rtspsrc locationmrtsp://some.server/url | rtph264depay nane=depay ! appsink nane=sink") 

rtspsrc = pipeline.get_by_name('rtspsrc') 
rtspsrc.connect("new-manager" ‚ new_manager_callback) 

depay = pipeline.get_by_name('depay') 
sinkpad = depay.get_static_pad('sink') 
probeID = sinkpad.add_probe(Gst.PadProbeType.BUFFER, calculate_tinestamp) 
```

##### 2. `rtpbin`收到訊號，透過`def new_manager_callbacK`函式獲得的對話訊息訪問&quot;RTCP SR&quot;

*   完成上述設置後，當創建 `rtpbin` 元素時，它將作為管理器參數傳遞給我們的回調函數
*   使用會話捕獲傳入的 RTCP 數據包。如第 14 行所述，為了到達會話，應請求該會話的 pad。
*   如第 15 行所述，應發出內部會話。我們應通過一個新的回調函數連接到 &quot;on-receiving-rtcp &quot;信號。
*   這樣，每次有新的 RTCP 數據包到達時，都會調用該回調函數

```python=14
def new_manager_callback ( rtspsrc, manager): 
    sinkpad = manager.get_request_pad("recv_rtp_sink_0") 
    session = manager.emit("get-internal-session", 0)
    if session:
        session.connect("on-receiving-rtcp", on_receiving_rtcp_callback)
    else:
        print("Failed to get RTPSession from the manager")
```

##### 3. 建立`def on_receiving_rtcp_callback`，使用`GstRtp.RTCPBuffer`捕獲、解析RTCP 數據封包

*   `def on_receiving_rtcp_callback` 函數將傳遞一個 Gst.Buffer 類型的緩沖區
*   該緩沖區必須映射到第 21 行中的 RTCP 緩沖區，這樣才能訪問 RTCP 數據包。
*   除了發送方報告外，RTCP 還可以接收其他數據包。因此，應從接收到的數據包中提取帶有發送方報告的數據包。
*   我通過 RTCP 緩沖區獲取第一個數據包，如第 23 行中的 get\_first\_packet。隨後的數據包也可通過第 30 行獲取。與第 27 行一樣，可以通過數據包訪問發送方報告本身。
*   到達 GStreamer 緩沖區的發送方報告示例為&quot;(ssrc=1644170567, ntptime=16645962074316441600, rtptime=2847690, packet\_count=475, octet\_count=623890)&quot;。應保存此報告中的 NTP 和 RTP 時間值，以便日後使用。

```python=19
def on_receiving_rtcp_callback (session, buffer: Gst.Buffer): 
  rtcp_buffer = GstRtp.RTCPBuffer() 
  res = GstRtp.RTCPBuffer.map (buffer, Gst.MapFlags.READ, rtcp_buffer) 
  rtcp_packet = GstRtp.RTCPPacket() 
  packet = rtcp_buffer.get_first _packet ( rtcp_packet) 

  while True: 
    if rtcp_packet.get_type() == GstRtp.RTCPType.SR : 
      si = rtcp_packet.sr_get_sender_info() 
      ntp_time = si[1] 
      rtp_time = si[2] 
    if rtcp_packet.move_to_next () ==False: 
      break
```

##### 4. `def calculate_timestamp`

`FRAME_TS= LAST_NTP + (RTP — LAST_RTP) / 90000`

Where

*   RTP is the frame timestamp received via RTP header
*   LAST\_NTP is NTP clock time received via RTCP Sender Report
*   LAST\_RTP is a reference RTP time received via RTCP Sender Report
*   90000 is required because the RTP timestamps increment with 90 kHz per RTSP specification. That means the RTP timestamp increments by 90000 every second. This value may vary for the different clock rates.

```python=34
def calculate_timestamp(pad, info): 
    res, rtpBuff = GstRtp.RTPBuffer.map(info.get_buffer(), Gst.MapFlags.READ) 
    tv = timevalue() 
    last_rtcp_ntp_time = c_uint64(ntp_timestamp) 
    LibNTP.ntp2tv(byref(last_rtcp_ntp_time), byref(tv)) 
    rtp_diff = float(rtpBuff.get_timestamp() - rtp_timestamp) / 90000.0 
    timestamp = float(tv.tv_sec) + float(tv.tv_usec) / 1000000.0 + rtp_diff 
    return Gst.PadProbeReturn.OK
```

:::


#### [2020。Sending and Receiving custom RTCP packets/events with GStreamer](https://stackoverflow.com/questions/62918542/sending-and-receiving-custom-rtcp-packets-events-with-gstreamer#:~:text=To%20handle%20RTCP%20you%20will%20need%20to%20include,%22on-receiving-rtcp%22%20signal%20%28or%20to%20the%20more%20specialized%20ones%29%3A)
> 1.Get the RTPSession object from the RtpBin
> 
> ```cpp=!
> g_signal_emit_by_name (rtpbin, "get-internal-session", id, &session);
> ```
> 
> 2.Attach to the "on-receiving-rtcp" signal (or to the more specialized ones):
> ```cpp=!
> g_signal_connect_after (session, "on-receiving-rtcp",
>     G_CALLBACK (on_rtcp_callback), my_callback_data);
> ```
> 3.Look for the RTCP message you want
> ```cpp=!
> gst_rtcp_buffer_map (buffer, GST_MAP_READ, &rtcp_buffer));
> new_packet = gst_rtcp_buffer_get_first_packet (&rtcp_buffer, &rtcp_packet);
> 
> while (gst_rtcp_packet_move_to_next (&rtcp_packet) {
>   type = gst_rtcp_packet_get_type (&rtcp_packet);
>   switch (type) {
>   ...
>   }
> }
> ```


### [2020。Antonio Ospite。collabora.com。High bitrate video streaming with GStreamer's RTP elements](https://www.collabora.com/news-and-blog/blog/2020/08/20/paving-the-way-high-bitrate-video-streaming-gstreamer-rtp-elements/)

![](https://hackmd.io/_uploads/r13AikHWT.png =800x)


### Document

#### [gstreamer/documentation/additional/RTP and RTSP support](https://gstreamer.freedesktop.org/documentation/additional/rtp.html?gi-language=c)

Most of GStreamer's key RTP components live in gst-plugins-good:

* The `rtpmanager` plugin contains elements like rtpbin and rtpjitterbuffer
* The `rtp` plugin contains RTP payloading and depayloading elements for many different codecs and container formats

- `rtpbin` is the high-level RTP component and supports sending and receiving, just sending or just receiving data, with and without RTCP support. This is the bin that does it all: it adapts dynamically to your needs based on the requested pads; it also contains an rtpjitterbuffer.
#### [gstreamer/rtpmanager/rtpbin](https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpbin.html?gi-language=python#rtpbin)

> To use rtpbin as an RTP receiver, request a ==`recv_rtp_sink_%u` pad==. The session number must be specified in the pad name. Data received on the `recv_rtp_sink_%u` pad will be processed in the rtpsession manager and after being validated forwarded on rtpssrcdemux element. ... 
> Access to the internal statistics of rtpbin is provided with the ==get-internal-session== property. This action signal gives access to the RTPSession object which further provides action signals to retrieve the internal source and other sources.

- get-internal-session 
    ```cpp=!
    g_signal_emit_by_name (rtpbin, "get-internal-session", id, &ret);
    ```

#### [gstreamer/rtsp/rtspsrc](https://gstreamer.freedesktop.org/documentation/rtsp/rtspsrc.html?gi-language=python#rtspsrc)
> rtspsrc will internally instantiate an ==RTP session manager element== that will handle the RTCP messages to and from the server, jitter removal, packet reordering along with providing a clock for the pipeline. This feature is implemented using the ==`gstrtpbin`== element.

> To also use rtpbin as an RTCP receiver, request a ==`recv_rtcp_sink_%u` pad==. The session number must be specified in the pad name.
- new-manager
    ```cpp=!
    new_manager_callback (GstElement * rtspsrc,
                          GstElement * manager,
                          gpointer udata)
    ```

#### [gstreamer/rtpmanager/RTPSession](https://gstreamer.freedesktop.org/documentation/rtpmanager/RTPSession.html?gi-language=python)

- on-receiving-rtcp 
    This signal is emitted when receiving an RTCP packet before it is handled by the session. It can be used to extract custom information from RTCP packets.
    ```python!
    def on_receiving_rtcp_callback (session, buffer, udata):
        #python callback for the 'on-receiving-rtcp' signal
    ```

#### [gstreamer/rtplib/GstRTPBuffer](https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=python)
- `GstRtp.RTPBuffer.get_timestamp` 
    Get the timestamp of the RTP packet in buffer.
    ```python=
    def GstRtp.RTPBuffer.get_timestamp (self):
        #python wrapper for 'gst_rtp_buffer_get_timestamp'
    ```

- `GstRtp.RTPBuffer.get_marker`
    Check if the marker bit is set on the RTP packet in buffer.
    ```python= 
    def GstRtp.RTPBuffer.get_marker (self):
        #python wrapper for 'gst_rtp_buffer_get_marker'
    ```
 
- `GstRtp.RTPBuffer.get_ssrc`
    Get the SSRC of the RTP packet in buffer.
    ```python!
    def GstRtp.RTPBuffer.get_ssrc (self):
        #python wrapper for 'gst_rtp_buffer_get_ssrc'
    ```



#### [gstreamer/rtplib/GstRTCPBuffer](https://gstreamer.freedesktop.org/documentation/rtplib/gstrtcpbuffer.html?gi-language=python )

GstRTCPBuffer 的c++原始程式碼

> The GstRTPCBuffer helper functions makes it easy to parse and create regular Gst.Buffer objects that contain compound RTCP packets. These buffers are typically of 'application/x-rtcp' Gst.Caps.
> 
> An RTCP buffer consists of 1 or more GstRtp.RTCPPacket structures that you can retrieve with ==GstRtp.RTCPBuffer.get_first_packet==. GstRtp.RTCPPacket acts as a pointer into the RTCP buffer; you can move to the next packet with GstRtp.RTCPPacket.move_to_next.

- [source code](https://github.com/Kurento/gst-plugins-base/blob/master/gst-libs/gst/rtp/gstrtcpbuffer.c)
- GstRtp.RTCPBuffer.get_first_packet 
    Initialize a new GstRtp.RTCPPacket pointer that points to the first packet in rtcp.
    ```python=
    def GstRtp.RTCPBuffer.get_first_packet (self, packet):
        #python wrapper for 'gst_rtcp_buffer_get_first_packet'
    ```

- GstRtp.RTCPBuffer.map
    The resulting RTCP buffer state is stored in rtcp. 
    ```python!
    def GstRtp.RTCPBuffer.map (buffer, flags, rtcp):
        #python wrapper for 'gst_rtcp_buffer_map'
    Open buffer for reading or writing, depending on flags.
    ```

- [sr_get_sender_info](https://gstreamer.freedesktop.org/documentation/rtplib/gstrtcpbuffer.html?gi-language=python#gst_rtcp_buffer_get_first_packet)
    ```python!
    def GstRtp.RTCPPacket.sr_get_sender_info (self):
        #python wrapper for 'gst_rtcp_packet_sr_get_sender_info'
    ```

    ```cpp!
    gst_rtcp_packet_sr_get_sender_info (GstRTCPPacket * packet,
                                        guint32 * ssrc,
                                        guint64 * ntptime,
                                        guint32 * rtptime,
                                        guint32 * packet_count,
                                        guint32 * octet_count)
    ```
    - source code : [gst-libs/gst/rtp/gstrtcpbuffer.c/gst_rtcp_packet_sr_get_sender_info](https://github.com/Kurento/gst-plugins-base/blob/master/gst-libs/gst/rtp/gstrtcpbuffer.c#L697)
        
 
        ```cpp=697
        /**
         * gst_rtcp_packet_sr_get_sender_info:
         * @packet: a valid SR #GstRTCPPacket
         * @ssrc: result SSRC
         * @ntptime: result NTP time
         * @rtptime: result RTP time
         * @packet_count: result packet count
         * @octet_count: result octet count
         *
         * Parse the SR sender info and store the values.
         */
        void
        gst_rtcp_packet_sr_get_sender_info (GstRTCPPacket * packet, guint32 * ssrc,
            guint64 * ntptime, guint32 * rtptime, guint32 * packet_count,
            guint32 * octet_count)
        {
          guint8 *data;

          g_return_if_fail (packet != NULL);
          g_return_if_fail (packet->type == GST_RTCP_TYPE_SR);
          g_return_if_fail (packet->rtcp != NULL);
          g_return_if_fail (packet->rtcp->map.flags & GST_MAP_READ);

          data = packet->rtcp->map.data;

          /* skip header */
          data += packet->offset + 4;
          if (ssrc)
            *ssrc = GST_READ_UINT32_BE (data);
          data += 4;
          if (ntptime)
            *ntptime = GST_READ_UINT64_BE (data);
          data += 8;
          if (rtptime)
            *rtptime = GST_READ_UINT32_BE (data);
          data += 4;
          if (packet_count)
            *packet_count = GST_READ_UINT32_BE (data);
          data += 4;
          if (octet_count)
            *octet_count = GST_READ_UINT32_BE (data);
        }
        ```
        
        
 
 
 


