libwebrtc Congestion Control and BWE

NOTE: Using libwebrtc M77 branch.

Resources

• Quality with Real-Time OTT media? It’s all about the feedback! (by Dr Alex)
• WebRTC Congestion Control Algorithm - Introduction to GCC
• WebRTC based on GCC congestion control analysis
• Bandwidth Estimation in WebRTC (and the new Sender Side BWE) (by Gustavo)
• libwebrtc Congestion Controller and Bandwidth Estimator (Notes)

Modules and Classes

Here the identified libwebrtc modules and classes must be exposed, with a brief description about their purpose and exposed API, and whether they are "public" classes or internally used ones by others.

Classes implementing RtcpBandwidthObserver

In modules/rtp_rtcp/include/rtp_rtcp_defines.h:

class RtcpBandwidthObserver {
 public:
  // REMB or TMMBR
  virtual void OnReceivedEstimatedBitrate(uint32_t bitrate) = 0;

  virtual void OnReceivedRtcpReceiverReport(
      const ReportBlockList& report_blocks,
      int64_t rtt,
      int64_t now_ms) = 0;

  virtual ~RtcpBandwidthObserver() {}
};

In audio/channel_send.cc:

class VoERtcpObserver : public RtcpBandwidthObserver

In call/rtp_transport_controller_send.h (NOTE: this is the one that RTCPReceiver uses):

class RtpTransportControllerSend final
  : public RtpTransportControllerSendInterface,
    public RtcpBandwidthObserver,
    public TransportFeedbackObserver {

In call/rtp_transport_controller_send.h:

std::unique_ptr<NetworkControllerInterface> controller_

Classes implementing TransportFeedbackObserver

In modules/rtp_rtcp/include/rtp_rtcp_defines.h:

class TransportFeedbackObserver {
 public:
  TransportFeedbackObserver() {}
  virtual ~TransportFeedbackObserver() {}

  virtual void OnAddPacket(const RtpPacketSendInfo& packet_info) = 0;
  virtual void OnTransportFeedback(const rtcp::TransportFeedback& feedback) = 0;
};

In audio/channel_send.cc:

class TransportFeedbackProxy : public TransportFeedbackObserver {
 public:
  TransportFeedbackProxy() : feedback_observer_(nullptr) {
    pacer_thread_.Detach();
    network_thread_.Detach();
  }

In call/rtp_transport_controller_send.h:

class RtpTransportControllerSend final
    : public RtpTransportControllerSendInterface,
      public RtcpBandwidthObserver,
      public TransportFeedbackObserver,
      public NetworkStateEstimateObserver {

Classes implementing NetworkControllerInterface

In modules/congestion_controller/goog_cc/goog_cc_network_control.h:

class GoogCcNetworkController : public NetworkControllerInterface {
 public:
  GoogCcNetworkController(NetworkControllerConfig config,
                          GoogCcConfig goog_cc_config);

(similar classes in pcc and bbr congestion control implementations).

PacketRouter class

In src/modules/pacing/packet_router.h.

It inherits from PacedSender::PacketSender (in src/modules/pacing/paced_sender.h).

class PacedSender : public Pacer {
 public:
  class PacketSender {
   public:
    // Note: packets sent as a result of a callback should not pass by this
    // module again.
    // Called when it's time to send a queued packet.
    // Returns false if packet cannot be sent.
    virtual bool TimeToSendPacket(uint32_t ssrc,
                                  uint16_t sequence_number,
                                  int64_t capture_time_ms,
                                  bool retransmission,
                                  const PacedPacketInfo& cluster_info) = 0;
    // Called when it's a good time to send a padding data.
    // Returns the number of bytes sent.
    virtual size_t TimeToSendPadding(size_t bytes,
                                     const PacedPacketInfo& cluster_info) = 0;

Flows

Incoming RTCP REMB flow

In modules/rtp_rtcp/source/rtcp_receiver.hpp:

RtcpBandwidthObserver* const rtcp_bandwidth_observer_

In modules/rtp_rtcp/source/rtcp_receiver.cpp:

RTCPReceiver::IncomingPacket(
  const uint8_t* packet, size_t packet_size)
// =>
RTCPReceiver::TriggerCallbacksFromRtcpPacket(
  packet_information)
// =>
// REMB or TMMBR RTCP packet.
rtcp_bandwidth_observer_->OnReceivedEstimatedBitrate(
  packet_information.receiver_estimated_max_bitrate_bps);

In call/rtp_transport_controller_send.cc (the implementation of RtcpBandwidthObserver in use):

void RtpTransportControllerSend::OnReceivedEstimatedBitrate(uint32_t bitrate)
// =>
if (controller_)
  PostUpdates(controller_->OnRemoteBitrateReport(msg));

In modules/congestion_controller/goog_cc/goog_cc_network_control.cc (the implementation of NetworkControllerInterface) in use:

NetworkControlUpdate GoogCcNetworkController::OnRemoteBitrateReport(msg)
// =>
bandwidth_estimation_->UpdateReceiverEstimate(
  msg.receive_time, msg.bandwidth);

In modules/bitrate_controller/send_side_bandwidth_estimation.cc:

void SendSideBandwidthEstimation::UpdateReceiverEstimate(
  Timestamp at_time, DataRate bandwidth)
// =>
// Cap |bitrate| to [min_bitrate_configured_, max_bitrate_configured_] and
// set |current_bitrate_| to the capped value and updates the event log.
CapBitrateToThresholds(at_time, current_bitrate_);

Incoming RTCP TransportFeedback flow

In modules/rtp_rtcp/source/rtcp_receiver.h:

TransportFeedbackObserver* const;

In modules/rtp_rtcp/source/rtcp_receiver.cc:

transport_feedback_observer_->OnTransportFeedback(
  *packet_information.transport_feedback);

In call/rtp_transport_controller_send.cc:

void RtpTransportControllerSend::OnTransportFeedback(
    const rtcp::TransportFeedback& feedback) {
  RTC_DCHECK_RUNS_SERIALIZED(&worker_race_);

  absl::optional<TransportPacketsFeedback> feedback_msg =
      transport_feedback_adapter_.ProcessTransportFeedback(
          feedback, Timestamp::ms(clock_->TimeInMilliseconds()));
  if (feedback_msg) {
    task_queue_.PostTask([this, feedback_msg]() {
      RTC_DCHECK_RUN_ON(&task_queue_);
      if (controller_)
        PostUpdates(controller_->OnTransportPacketsFeedback(*feedback_msg));
    });
  }
  pacer_.UpdateOutstandingData(
      transport_feedback_adapter_.GetOutstandingData().bytes());
}

In congestion_controller/rtp/transport_feedback_adapter.cc:

absl::optional<TransportPacketsFeedback>
TransportFeedbackAdapter::ProcessTransportFeedback(
    const rtcp::TransportFeedback& feedback,
    Timestamp feedback_receive_time) {
  DataSize prior_in_flight = GetOutstandingData();

// (much more...)

Ougoing RTP Packet flow

NOTE: Somewhere on RTP packet sent process it arrives here.

In modules/rtp_rtcp/source/rtp_sender.cc:

bool RTPSender::PrepareAndSendPacket(std::unique_ptr<RtpPacketToSend> packet,
                                     bool send_over_rtx,
                                     bool is_retransmit,
                                     const PacedPacketInfo& pacing_info)
// (more)
packet_to_send->SetExtension<AbsoluteSendTime>(
      AbsoluteSendTime::MsTo24Bits(now_ms));
// =>
AddPacketToTransportFeedback(
  options.packet_id, *packet_to_send, pacing_info);

void RTPSender::AddPacketToTransportFeedback(
    uint16_t packet_id,
    const RtpPacketToSend& packet,
    const PacedPacketInfo& pacing_info) {
// =>
RtpPacketSendInfo packet_info;
packet_info.ssrc = SSRC();
packet_info.transport_sequence_number = packet_id;
packet_info.has_rtp_sequence_number = true;
packet_info.rtp_sequence_number = packet.SequenceNumber();
packet_info.length = packet_size;
packet_info.pacing_info = pacing_info;
transport_feedback_observer_->OnAddPacket(packet_info);

In call/rtp_transport_controller_send.cc:

void RtpTransportControllerSend::OnAddPacket(
    const RtpPacketSendInfo& packet_info) {
  transport_feedback_adapter_.AddPacket(
      packet_info,
      send_side_bwe_with_overhead_ ? transport_overhead_bytes_per_packet_.load()
                                   : 0,
      Timestamp::ms(clock_->TimeInMilliseconds()));
}

// [...]

void RtpTransportControllerSend::OnSentPacket(
    const rtc::SentPacket& sent_packet) {
  absl::optional<SentPacket> packet_msg =
      transport_feedback_adapter_.ProcessSentPacket(sent_packet);
  
  if (packet_msg) {
    task_queue_.PostTask([this, packet_msg]() {
      RTC_DCHECK_RUN_ON(&task_queue_);
      if (controller_)
        PostUpdates(controller_->OnSentPacket(*packet_msg));
    });
  }
  
  pacer_.UpdateOutstandingData(
      transport_feedback_adapter_.GetOutstandingData().bytes());
}

In modules/congestion_controller/rtp/transport_feedback_adapter.cc:

void TransportFeedbackAdapter::AddPacket(const RtpPacketSendInfo& packet_info,
                                         size_t overhead_bytes,
                                         Timestamp creation_time) {
  {
    rtc::CritScope cs(&lock_);
    PacketFeedback packet_feedback(
        creation_time.ms(), packet_info.transport_sequence_number,
        packet_info.length + overhead_bytes, local_net_id_, remote_net_id_,
        packet_info.pacing_info);
    if (packet_info.has_rtp_sequence_number) {
      packet_feedback.ssrc = packet_info.ssrc;
      packet_feedback.rtp_sequence_number = packet_info.rtp_sequence_number;
    }
    send_time_history_.RemoveOld(creation_time.ms());
    send_time_history_.AddNewPacket(std::move(packet_feedback));
  }

  {
    rtc::CritScope cs(&observers_lock_);
    for (auto* observer : observers_) {
      observer->OnPacketAdded(packet_info.ssrc,
                              packet_info.transport_sequence_number);
    }
  }
}

In call/rtp_video_sender.h:

void OnPacketAdded(uint32_t ssrc, uint16_t seq_num) override {}

NOTE: So it does nothing.

Getting the new target bitrate

Whenever the controller_ is feed via any of its public methods, which all return a NetworkControlUpdateobject , the following method is called:

RtpTransportControllerSend::PostUpdates(NetworkControlUpdate update)

Such method calls the private method:

void RtpTransportControllerSend::UpdateControlState() 

Which:

// Calculates and retrieves the target transfer reate!!
// Generates the log: RTC_LOG(LS_INFO) << "Bitrate estimate state changed, BWE: "
                       << ToString(log_target_rate) << ".";

absl::optional<TargetTransferRate> update = control_handler_->GetUpdate();

// Notifies the observer about the new trasnfer rate!!
observer_->OnTargetTransferRate(*update);

observer_ is Call:

void Call::OnTargetTransferRate(TargetTransferRate msg) {
  // TODO(bugs.webrtc.org/9719)
  // Call::OnTargetTransferRate requires that on target transfer rate is invoked
  // from the worker queue (because bitrate_allocator_ requires it). Media
  // transport does not guarantee the callback on the worker queue.
  // When the threading model for MediaTransportInterface is update, reconsider
  // changing this implementation.
  if (!transport_send_ptr_->GetWorkerQueue()->IsCurrent()) {
    transport_send_ptr_->GetWorkerQueue()->PostTask(
        [this, msg] { this->OnTargetTransferRate(msg); });
    return;
  }

  uint32_t target_bitrate_bps = msg.target_rate.bps();
  int loss_ratio_255 = msg.network_estimate.loss_rate_ratio * 255;
  uint8_t fraction_loss =
      rtc::dchecked_cast<uint8_t>(rtc::SafeClamp(loss_ratio_255, 0, 255));
  int64_t rtt_ms = msg.network_estimate.round_trip_time.ms();
  int64_t probing_interval_ms = msg.network_estimate.bwe_period.ms();
  uint32_t bandwidth_bps = msg.network_estimate.bandwidth.bps();
  {
    rtc::CritScope cs(&last_bandwidth_bps_crit_);
    last_bandwidth_bps_ = bandwidth_bps;
  }
  // For controlling the rate of feedback messages.
  receive_side_cc_.OnBitrateChanged(target_bitrate_bps);
  bitrate_allocator_->OnNetworkChanged(target_bitrate_bps, bandwidth_bps,
                                       fraction_loss, rtt_ms,
                                       probing_interval_ms);

  // Ignore updates if bitrate is zero (the aggregate network state is down).
  if (target_bitrate_bps == 0) {
    rtc::CritScope lock(&bitrate_crit_);
    estimated_send_bitrate_kbps_counter_.ProcessAndPause();
    pacer_bitrate_kbps_counter_.ProcessAndPause();
    return;
  }

  bool sending_video;
  {
    ReadLockScoped read_lock(*send_crit_);
    sending_video = !video_send_streams_.empty();
  }

  rtc::CritScope lock(&bitrate_crit_);
  if (!sending_video) {
    // Do not update the stats if we are not sending video.
    estimated_send_bitrate_kbps_counter_.ProcessAndPause();
    pacer_bitrate_kbps_counter_.ProcessAndPause();
    return;
  }
  estimated_send_bitrate_kbps_counter_.Add(target_bitrate_bps / 1000);
  // Pacer bitrate may be higher than bitrate estimate if enforcing min bitrate.
  uint32_t pacer_bitrate_bps =
      std::max(target_bitrate_bps, min_allocated_send_bitrate_bps_);
  pacer_bitrate_kbps_counter_.Add(pacer_bitrate_bps / 1000);
}

The following method distributes the bandwdith within its bitrate_observers_configs_ and

void BitrateAllocator::OnNetworkChanged(uint32_t target_bitrate_bps,
                                        uint32_t link_capacity_bps,
                                        uint8_t fraction_loss,
                                        int64_t rtt,
                                        int64_t bwe_period_ms) {
...
    // Calculate a new allocation and update all observers.

    ObserverAllocation allocation = AllocateBitrates(last_target_bps_);
    ObserverAllocation bandwidth_allocation =
        AllocateBitrates(last_link_capacity_bps_);
    for (auto& config : bitrate_observer_configs_) {
      uint32_t allocated_bitrate = allocation[config.observer];
      uint32_t bandwidth = bandwidth_allocation[config.observer];
      BitrateAllocationUpdate update;
      update.target_bitrate = DataRate::bps(allocated_bitrate);
      update.link_capacity = DataRate::bps(bandwidth);
      update.packet_loss_ratio = last_fraction_loss_ / 256.0;
      update.round_trip_time = TimeDelta::ms(last_rtt_);
      update.bwe_period = TimeDelta::ms(last_bwe_period_ms_);
      uint32_t protection_bitrate = config.observer->OnBitrateUpdated(update);
      config.allocated_bitrate_bps = allocated_bitrate;
      if (allocated_bitrate > 0)
        config.media_ratio = MediaRatio(allocated_bitrate, protection_bitrate);
    }

...
}

Implementation at mediasoup

TODO: Let's see.

It seems that the WebRtcTransport class should have an instance of RtpTransportControllerSend that is defined/declared at:

• call/rtp_transport_controller_send.h
• call/rtp_transport_controller_send.cc

We must also check the interface definition:

• call/rtp_transport_controller_send_interface.h

The user of RtpTransportControllerSendInterface are:

• call/call.h
• call/call.cc
• video/video_send_stream.h
• video/video_send_stream_impl.h
• video/video_send_stream_impl.cc
• audio/channel_send.h
• audio/channel_send.cc
• audio/audio_send_stream.h
• audio/audio_send_stream.cc
• call/rtp_video_sender.h
• call/rtp_video_sender.cc

It seems that, instead of directly calling TimeUntilNextProcess() in modules (most above classes inherit from Module), libwebrtc uses modules/utility/source/process_thread_impl.cc:

int64_t GetNextCallbackTime(Module* module, int64_t time_now) {
  int64_t interval = module->TimeUntilNextProcess();
  if (interval < 0) {
    // Falling behind, we should call the callback now.
    return time_now;
  }
  return time_now + interval;
}

(Check also the ProcessThreadImpl::Process() method).

Files related to client-side BWE

• PacketFeedback struct:
  • Defined in modules/rtp_rtcp/include/rtp_rtcp_defines.h.
  • Holds info about sent packets and their corresponding receive feedbacks.
  • It has these members:

  // Time corresponding to when this object was created.
  int64_t creation_time_ms;
  // Time corresponding to when the packet was received. Timestamped with the
  // receiver's clock. For unreceived packet, the sentinel value kNotReceived
  // is used.
  int64_t arrival_time_ms;
  // Time corresponding to when the packet was sent, timestamped with the
  // sender's clock.
  int64_t send_time_ms;
  // Packet identifier, incremented with 1 for every packet generated by the
  // sender.
  uint16_t sequence_number;
  // Session unique packet identifier, incremented with 1 for every packet
  // generated by the sender.
  int64_t long_sequence_number;
  // Size of the packet excluding RTP headers.
  size_t payload_size;
  // Size of preceeding packets that are not part of feedback.
  size_t unacknowledged_data;
  // The network route ids that this packet is associated with.
  uint16_t local_net_id;
  uint16_t remote_net_id;
  // Pacing information about this packet.
  PacedPacketInfo pacing_info;

  // The SSRC and RTP sequence number of the packet this feedback refers to.
  absl::optional<uint32_t> ssrc;
  uint16_t rtp_sequence_number;

• PacedPacketInfo struct:
  • Defined in api/transport/network_types.h.
  • It has these members:

  int send_bitrate_bps = -1;
  int probe_cluster_id = kNotAProbe;
  int probe_cluster_min_probes = -1;
  int probe_cluster_min_bytes = -1;

• PacketResult struct:
  • Defined in api/transport/network_types.h.
  • It has these members:

  SentPacket sent_packet;
  Timestamp receive_time = Timestamp::PlusInfinity();