# libwebrtc Congestion Control and BWE

*NOTE:* Using libwebrtc M77 branch.

## Resources

* [Quality with Real-Time OTT media? It’s all about the feedback!](http://webrtcbydralex.com/index.php/2019/07/30/quality-with-real-time-ott-media-its-all-about-the-feedback/) (by Dr Alex)
* [WebRTC Congestion Control Algorithm - Introduction to GCC](http://www.programmersought.com/article/5754684436/)
* [WebRTC based on GCC congestion control analysis](http://www.programmersought.com/article/26011549416/)
* [Bandwidth Estimation in WebRTC (and the new Sender Side BWE)](http://www.rtcbits.com/2017/01/bandwidth-estimation-in-webrtc-and-new.html) (by Gustavo)
* [libwebrtc Congestion Controller and Bandwidth Estimator (Notes)](https://gist.github.com/ibc/77ed0355ea617a47823cf70d13a065dd)

## Modules and Classes

Here the identified libwebrtc modules and classes must be exposed, with a brief description about their purpose and exposed API, and whether they are "public" classes or internally used ones by others.

### Classes implementing `RtcpBandwidthObserver`

In `modules/rtp_rtcp/include/rtp_rtcp_defines.h`:

class RtcpBandwidthObserver {
 public:
  // REMB or TMMBR
  virtual void OnReceivedEstimatedBitrate(uint32_t bitrate) = 0;

  virtual void OnReceivedRtcpReceiverReport(
      const ReportBlockList& report_blocks,
      int64_t rtt,
      int64_t now_ms) = 0;

  virtual ~RtcpBandwidthObserver() {}
};

In `audio/channel_send.cc`:

class VoERtcpObserver : public RtcpBandwidthObserver

In `call/rtp_transport_controller_send.h` (*NOTE:* this is the one that `RTCPReceiver` uses):

class RtpTransportControllerSend final
  : public RtpTransportControllerSendInterface,
    public RtcpBandwidthObserver,
    public TransportFeedbackObserver {

In `call/rtp_transport_controller_send.h`:

std::unique_ptr<NetworkControllerInterface> controller_

### Classes implementing `TransportFeedbackObserver`

In `modules/rtp_rtcp/include/rtp_rtcp_defines.h`:

class TransportFeedbackObserver {
 public:
  TransportFeedbackObserver() {}
  virtual ~TransportFeedbackObserver() {}

  virtual void OnAddPacket(const RtpPacketSendInfo& packet_info) = 0;
  virtual void OnTransportFeedback(const rtcp::TransportFeedback& feedback) = 0;
};

In `audio/channel_send.cc`:

class TransportFeedbackProxy : public TransportFeedbackObserver {
 public:
  TransportFeedbackProxy() : feedback_observer_(nullptr) {
    pacer_thread_.Detach();
    network_thread_.Detach();
  }

In `call/rtp_transport_controller_send.h`:

class RtpTransportControllerSend final
    : public RtpTransportControllerSendInterface,
      public RtcpBandwidthObserver,
      public TransportFeedbackObserver,
      public NetworkStateEstimateObserver {

### Classes implementing `NetworkControllerInterface`

In `modules/congestion_controller/goog_cc/goog_cc_network_control.h`:

class GoogCcNetworkController : public NetworkControllerInterface {
 public:
  GoogCcNetworkController(NetworkControllerConfig config,
                          GoogCcConfig goog_cc_config);

(similar classes in `pcc` and `bbr` congestion control implementations).

### `PacketRouter` class

In `src/modules/pacing/packet_router.h`.

It inherits from `PacedSender::PacketSender` (in `src/modules/pacing/paced_sender.h`).

class PacedSender : public Pacer {
 public:
  class PacketSender {
   public:
    // Note: packets sent as a result of a callback should not pass by this
    // module again.
    // Called when it's time to send a queued packet.
    // Returns false if packet cannot be sent.
    virtual bool TimeToSendPacket(uint32_t ssrc,
                                  uint16_t sequence_number,
                                  int64_t capture_time_ms,
                                  bool retransmission,
                                  const PacedPacketInfo& cluster_info) = 0;
    // Called when it's a good time to send a padding data.
    // Returns the number of bytes sent.
    virtual size_t TimeToSendPadding(size_t bytes,
                                     const PacedPacketInfo& cluster_info) = 0;

## Flows

### Incoming RTCP REMB flow

In `modules/rtp_rtcp/source/rtcp_receiver.hpp`:

RtcpBandwidthObserver* const rtcp_bandwidth_observer_

In `modules/rtp_rtcp/source/rtcp_receiver.cpp`:

RTCPReceiver::IncomingPacket(
  const uint8_t* packet, size_t packet_size)
// =>
RTCPReceiver::TriggerCallbacksFromRtcpPacket(
  packet_information)
// =>
// REMB or TMMBR RTCP packet.
rtcp_bandwidth_observer_->OnReceivedEstimatedBitrate(
  packet_information.receiver_estimated_max_bitrate_bps);

In `call/rtp_transport_controller_send.cc` (the implementation of `RtcpBandwidthObserver` in use):

void RtpTransportControllerSend::OnReceivedEstimatedBitrate(uint32_t bitrate)
// =>
if (controller_)
  PostUpdates(controller_->OnRemoteBitrateReport(msg));

In `modules/congestion_controller/goog_cc/goog_cc_network_control.cc` (the implementation of `NetworkControllerInterface`) in use:

NetworkControlUpdate GoogCcNetworkController::OnRemoteBitrateReport(msg)
// =>
bandwidth_estimation_->UpdateReceiverEstimate(
  msg.receive_time, msg.bandwidth);

In `modules/bitrate_controller/send_side_bandwidth_estimation.cc`:

void SendSideBandwidthEstimation::UpdateReceiverEstimate(
  Timestamp at_time, DataRate bandwidth)
// =>
// Cap |bitrate| to [min_bitrate_configured_, max_bitrate_configured_] and
// set |current_bitrate_| to the capped value and updates the event log.
CapBitrateToThresholds(at_time, current_bitrate_);

### Incoming RTCP TransportFeedback flow

In `modules/rtp_rtcp/source/rtcp_receiver.h`:

TransportFeedbackObserver* const;

In `modules/rtp_rtcp/source/rtcp_receiver.cc`:

transport_feedback_observer_->OnTransportFeedback(
  *packet_information.transport_feedback);

In `call/rtp_transport_controller_send.cc`:

void RtpTransportControllerSend::OnTransportFeedback(
    const rtcp::TransportFeedback& feedback) {
  RTC_DCHECK_RUNS_SERIALIZED(&worker_race_);

  absl::optional<TransportPacketsFeedback> feedback_msg =
      transport_feedback_adapter_.ProcessTransportFeedback(
          feedback, Timestamp::ms(clock_->TimeInMilliseconds()));
  if (feedback_msg) {
    task_queue_.PostTask([this, feedback_msg]() {
      RTC_DCHECK_RUN_ON(&task_queue_);
      if (controller_)
        PostUpdates(controller_->OnTransportPacketsFeedback(*feedback_msg));
    });
  }
  pacer_.UpdateOutstandingData(
      transport_feedback_adapter_.GetOutstandingData().bytes());
}

In `congestion_controller/rtp/transport_feedback_adapter.cc`:

absl::optional<TransportPacketsFeedback>
TransportFeedbackAdapter::ProcessTransportFeedback(
    const rtcp::TransportFeedback& feedback,
    Timestamp feedback_receive_time) {
  DataSize prior_in_flight = GetOutstandingData();

// (much more...)

### Ougoing RTP Packet flow

**NOTE:** Somewhere on RTP packet sent process it arrives here.

In `modules/rtp_rtcp/source/rtp_sender.cc`:

bool RTPSender::PrepareAndSendPacket(std::unique_ptr<RtpPacketToSend> packet,
                                     bool send_over_rtx,
                                     bool is_retransmit,
                                     const PacedPacketInfo& pacing_info)
// (more)
packet_to_send->SetExtension<AbsoluteSendTime>(
      AbsoluteSendTime::MsTo24Bits(now_ms));
// =>
AddPacketToTransportFeedback(
  options.packet_id, *packet_to_send, pacing_info);

void RTPSender::AddPacketToTransportFeedback(
    uint16_t packet_id,
    const RtpPacketToSend& packet,
    const PacedPacketInfo& pacing_info) {
// =>
RtpPacketSendInfo packet_info;
packet_info.ssrc = SSRC();
packet_info.transport_sequence_number = packet_id;
packet_info.has_rtp_sequence_number = true;
packet_info.rtp_sequence_number = packet.SequenceNumber();
packet_info.length = packet_size;
packet_info.pacing_info = pacing_info;
transport_feedback_observer_->OnAddPacket(packet_info);

In `call/rtp_transport_controller_send.cc`:

void RtpTransportControllerSend::OnAddPacket(
    const RtpPacketSendInfo& packet_info) {
  transport_feedback_adapter_.AddPacket(
      packet_info,
      send_side_bwe_with_overhead_ ? transport_overhead_bytes_per_packet_.load()
                                   : 0,
      Timestamp::ms(clock_->TimeInMilliseconds()));
}

// [...]

void RtpTransportControllerSend::OnSentPacket(
    const rtc::SentPacket& sent_packet) {
  absl::optional<SentPacket> packet_msg =
      transport_feedback_adapter_.ProcessSentPacket(sent_packet);
  
  if (packet_msg) {
    task_queue_.PostTask([this, packet_msg]() {
      RTC_DCHECK_RUN_ON(&task_queue_);
      if (controller_)
        PostUpdates(controller_->OnSentPacket(*packet_msg));
    });
  }
  
  pacer_.UpdateOutstandingData(
      transport_feedback_adapter_.GetOutstandingData().bytes());
}

In `modules/congestion_controller/rtp/transport_feedback_adapter.cc`:

void TransportFeedbackAdapter::AddPacket(const RtpPacketSendInfo& packet_info,
                                         size_t overhead_bytes,
                                         Timestamp creation_time) {
  {
    rtc::CritScope cs(&lock_);
    PacketFeedback packet_feedback(
        creation_time.ms(), packet_info.transport_sequence_number,
        packet_info.length + overhead_bytes, local_net_id_, remote_net_id_,
        packet_info.pacing_info);
    if (packet_info.has_rtp_sequence_number) {
      packet_feedback.ssrc = packet_info.ssrc;
      packet_feedback.rtp_sequence_number = packet_info.rtp_sequence_number;
    }
    send_time_history_.RemoveOld(creation_time.ms());
    send_time_history_.AddNewPacket(std::move(packet_feedback));
  }

  {
    rtc::CritScope cs(&observers_lock_);
    for (auto* observer : observers_) {
      observer->OnPacketAdded(packet_info.ssrc,
                              packet_info.transport_sequence_number);
    }
  }
}

In `call/rtp_video_sender.h`:

void OnPacketAdded(uint32_t ssrc, uint16_t seq_num) override {}

*NOTE:* So it does nothing.

## Getting the new target bitrate

Whenever the `controller_` is feed via any of its public methods, which all return a `NetworkControlUpdate`object , the following method is called:

RtpTransportControllerSend::PostUpdates(NetworkControlUpdate update)

Such method calls the private method:

void RtpTransportControllerSend::UpdateControlState() 

Which:

// Calculates and retrieves the target transfer reate!!
// Generates the log: RTC_LOG(LS_INFO) << "Bitrate estimate state changed, BWE: "
                       << ToString(log_target_rate) << ".";

absl::optional<TargetTransferRate> update = control_handler_->GetUpdate();

// Notifies the observer about the new trasnfer rate!!
observer_->OnTargetTransferRate(*update);

`observer_` is `Call`:

void Call::OnTargetTransferRate(TargetTransferRate msg) {
  // TODO(bugs.webrtc.org/9719)
  // Call::OnTargetTransferRate requires that on target transfer rate is invoked
  // from the worker queue (because bitrate_allocator_ requires it). Media
  // transport does not guarantee the callback on the worker queue.
  // When the threading model for MediaTransportInterface is update, reconsider
  // changing this implementation.
  if (!transport_send_ptr_->GetWorkerQueue()->IsCurrent()) {
    transport_send_ptr_->GetWorkerQueue()->PostTask(
        [this, msg] { this->OnTargetTransferRate(msg); });
    return;
  }

  uint32_t target_bitrate_bps = msg.target_rate.bps();
  int loss_ratio_255 = msg.network_estimate.loss_rate_ratio * 255;
  uint8_t fraction_loss =
      rtc::dchecked_cast<uint8_t>(rtc::SafeClamp(loss_ratio_255, 0, 255));
  int64_t rtt_ms = msg.network_estimate.round_trip_time.ms();
  int64_t probing_interval_ms = msg.network_estimate.bwe_period.ms();
  uint32_t bandwidth_bps = msg.network_estimate.bandwidth.bps();
  {
    rtc::CritScope cs(&last_bandwidth_bps_crit_);
    last_bandwidth_bps_ = bandwidth_bps;
  }
  // For controlling the rate of feedback messages.
  receive_side_cc_.OnBitrateChanged(target_bitrate_bps);
  bitrate_allocator_->OnNetworkChanged(target_bitrate_bps, bandwidth_bps,
                                       fraction_loss, rtt_ms,
                                       probing_interval_ms);

  // Ignore updates if bitrate is zero (the aggregate network state is down).
  if (target_bitrate_bps == 0) {
    rtc::CritScope lock(&bitrate_crit_);
    estimated_send_bitrate_kbps_counter_.ProcessAndPause();
    pacer_bitrate_kbps_counter_.ProcessAndPause();
    return;
  }

  bool sending_video;
  {
    ReadLockScoped read_lock(*send_crit_);
    sending_video = !video_send_streams_.empty();
  }

  rtc::CritScope lock(&bitrate_crit_);
  if (!sending_video) {
    // Do not update the stats if we are not sending video.
    estimated_send_bitrate_kbps_counter_.ProcessAndPause();
    pacer_bitrate_kbps_counter_.ProcessAndPause();
    return;
  }
  estimated_send_bitrate_kbps_counter_.Add(target_bitrate_bps / 1000);
  // Pacer bitrate may be higher than bitrate estimate if enforcing min bitrate.
  uint32_t pacer_bitrate_bps =
      std::max(target_bitrate_bps, min_allocated_send_bitrate_bps_);
  pacer_bitrate_kbps_counter_.Add(pacer_bitrate_bps / 1000);
}

The following method distributes the bandwdith within its `bitrate_observers_configs_` and

void BitrateAllocator::OnNetworkChanged(uint32_t target_bitrate_bps,
                                        uint32_t link_capacity_bps,
                                        uint8_t fraction_loss,
                                        int64_t rtt,
                                        int64_t bwe_period_ms) {
...
    // Calculate a new allocation and update all observers.

    ObserverAllocation allocation = AllocateBitrates(last_target_bps_);
    ObserverAllocation bandwidth_allocation =
        AllocateBitrates(last_link_capacity_bps_);
    for (auto& config : bitrate_observer_configs_) {
      uint32_t allocated_bitrate = allocation[config.observer];
      uint32_t bandwidth = bandwidth_allocation[config.observer];
      BitrateAllocationUpdate update;
      update.target_bitrate = DataRate::bps(allocated_bitrate);
      update.link_capacity = DataRate::bps(bandwidth);
      update.packet_loss_ratio = last_fraction_loss_ / 256.0;
      update.round_trip_time = TimeDelta::ms(last_rtt_);
      update.bwe_period = TimeDelta::ms(last_bwe_period_ms_);
      uint32_t protection_bitrate = config.observer->OnBitrateUpdated(update);
      config.allocated_bitrate_bps = allocated_bitrate;
      if (allocated_bitrate > 0)
        config.media_ratio = MediaRatio(allocated_bitrate, protection_bitrate);
    }

...
}

## Implementation at mediasoup

*TODO:* Let's see.

It seems that the `WebRtcTransport` class should have an instance of `RtpTransportControllerSend` that is defined/declared at:

* `call/rtp_transport_controller_send.h`
* `call/rtp_transport_controller_send.cc`

We must also check the interface definition:

* `call/rtp_transport_controller_send_interface.h`

The user of `RtpTransportControllerSendInterface` are:

* `call/call.h`
* `call/call.cc`
* `video/video_send_stream.h`
* `video/video_send_stream_impl.h`
* `video/video_send_stream_impl.cc`
* `audio/channel_send.h`
* `audio/channel_send.cc`
* `audio/audio_send_stream.h`
* `audio/audio_send_stream.cc`
* `call/rtp_video_sender.h`
* `call/rtp_video_sender.cc`

It seems that, instead of directly calling `TimeUntilNextProcess()` in modules (most above classes inherit from `Module`), libwebrtc uses `modules/utility/source/process_thread_impl.cc`:

int64_t GetNextCallbackTime(Module* module, int64_t time_now) {
  int64_t interval = module->TimeUntilNextProcess();
  if (interval < 0) {
    // Falling behind, we should call the callback now.
    return time_now;
  }
  return time_now + interval;
}

(Check also the `ProcessThreadImpl::Process()` method).

## Files related to client-side BWE

* `PacketFeedback` struct:
* Defined in `modules/rtp_rtcp/include/rtp_rtcp_defines.h`.
* Holds info about sent packets and their corresponding receive feedbacks.
* It has these members:

  // Time corresponding to when this object was created.
  int64_t creation_time_ms;
  // Time corresponding to when the packet was received. Timestamped with the
  // receiver's clock. For unreceived packet, the sentinel value kNotReceived
  // is used.
  int64_t arrival_time_ms;
  // Time corresponding to when the packet was sent, timestamped with the
  // sender's clock.
  int64_t send_time_ms;
  // Packet identifier, incremented with 1 for every packet generated by the
  // sender.
  uint16_t sequence_number;
  // Session unique packet identifier, incremented with 1 for every packet
  // generated by the sender.
  int64_t long_sequence_number;
  // Size of the packet excluding RTP headers.
  size_t payload_size;
  // Size of preceeding packets that are not part of feedback.
  size_t unacknowledged_data;
  // The network route ids that this packet is associated with.
  uint16_t local_net_id;
  uint16_t remote_net_id;
  // Pacing information about this packet.
  PacedPacketInfo pacing_info;

  // The SSRC and RTP sequence number of the packet this feedback refers to.
  absl::optional<uint32_t> ssrc;
  uint16_t rtp_sequence_number;

* `PacedPacketInfo` struct:
* Defined in `api/transport/network_types.h`.
* It has these members:

  int send_bitrate_bps = -1;
  int probe_cluster_id = kNotAProbe;
  int probe_cluster_min_probes = -1;
  int probe_cluster_min_bytes = -1;

* `PacketResult` struct:
* Defined in `api/transport/network_types.h`.
* It has these members:

  SentPacket sent_packet;
  Timestamp receive_time = Timestamp::PlusInfinity();