Findings on SIP communication on the BTicino C300X

Goal of the document

The goal of this document is to describe the process of the communication between components using SIP calls on the BTicino (C100X and) C300X.

Furthermore we define the steps that we need to take to gradually build out a system that allows us to communicate with other clients (HomeBridge, HASS, OpenHAB)

Disclaimer

I am not a hacker and have no experience in it. I love finding out how things are set up.

I don't know much, but I do know much about nothing.

If you break your system due to change you make, don't blame me.

Contributors

  • Thanks to R who was kind enough to join forces and work on this
  • Thanks to Neo who was also candidate to work on this

Terminology

This explains some terms that are used in either this document or log files.

  • SIP: Session Initiation Protocol
  • SIPs: SIP Secure = SIP extended with TLS
  • flexisip: The SIP server
  • awsm: The answering machine
  • ops: openserver used for (?)
  • intercom: We use this in this document to reference the C100X/C300X video unit.

Steps

I am trying to gradually build a path to a system where we can call our intercom to peak and listen to the video and camera,
as well as a system where we can receive an incoming call from the unit (more or less how our Door Entry app works).

How we are gradually building this up, is also how the system is going to be integrated in a home automation platform.

The idea is to install transcoding of audio/video on "our" system and not hack it into the intercom.
So the SIP handling is installed additionally on "our" system.

This makes a loosely coupled system where everything audio/video related is handled by SIP communication.

The focus is on video/audio and call handling. Opening of doors/locks is NOT our scope (this is already partially achieved and being handled).

Step goals:

  1. Make a call to the intercom: using a SIP client we attempt to achieve a call to the intercom

    1. Starting with SIP (unsecured)
    2. Enhancing this further with SIPs

  2. Receive a call from the intercom: Receive a call on the SIP client when we "call home" or press the bell button

    1. Starting with SIP
    2. Enhancing this further with SIPs

  3. Install a SIP server on a seperate host (e.g. where you run your Home automation platform), make the intercom talk to it
    Using either something like a proxy flexisip service or installing a SIP client which will be used to connect it into our home automation platform.

  4. Make a SIP call to the intercom from your home automation platform
    Using "our" server either streams the video and audio to "our" channels (using ffmpeg, vlc, …)

  5. Using this SIP service integrate it into a platform (our SIP server or client triggers something in our automation platform)
    It sends you a notification or just opens the door automatically for example.

    Current research for 1i

We assume you have succesfully flashed the adapted firmware from: https://github.com/fquinto/bticinoClasse300x to continue with this and have access via SSH.

I have decided to use Linphone (for desktop) to use as a client, you can install it on your desktop.
Fetch it at https://new.linphone.org/technical-corner/linphone?qt-technical_corner=2#qt-technical_corner

The first attempt to communicate on SIP level was to get access to the network ports (5060 SIP, 5061 SIPs).

You need to make flexisip listen on an outward facing interface (wlan0):

Mount read/write

$ mount -oremount,rw /

open /etc/init.d/flexisipsh and change:

case "$1" in
  start)
	start-stop-daemon --start --quiet --exec $DAEMON -- $DAEMON_ARGS --transports "sips:$2:5061;maddr=$2;require-peer-certificate=1 sip:127.0.0.1;maddr=127.0.0.1 sip:192.168.0.XX;maddr=192.168.0.XX"
	;;

If you use a fixed IP for you unit (e.g DHCP reservations), you can put the fixed IP.

You can also let the script handle it (uses your current IP if dynamic):

start-stop-daemon --start --quiet --exec $DAEMON -- $DAEMON_ARGS --transports "sips:$2:5061;maddr=$2;require-peer-certificate=1 sip:127.0.0.1;maddr=127.0.0.1 sip:$2;maddr=$2"

TODO: We might need to change the main.py script from https://github.com/fquinto/bticinoClasse300x to ask to automatically expose daemon ports to the LAN

Next, We need to change the firewall. The default value is to DROP when it isn't matched in the firewall rules.

For simplicity we just ACCEPT on the three chains.

NOTE: Don't have it assigned a public IP … or you will be hacked.

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

TODO: We might need to change the main.py script from https://github.com/fquinto/bticinoClasse300x to ask to automatically drop the firewall rules

YOU NEED TO DO THIS EVERYTIME YOU REBOOT YOUR DEVICE (to be fixed later to automatically flush them (?))

Next we want more logging!

In /home/bticino/cfg/flexisip.conf change 3 lines (make sure you create a backup of this file somewhere if you are scared).

Set log-level and syslog-level to debug

In trusted-hosts add the IP address of your Desktop client (where you installed Linphone client).
This makes sure we don't need to bother with the initial authentication of username/password (problem for later?).

[global]
...
log-level=debug
syslog-level=debug

[module::Authentication]
enabled=true
auth-domains=c300x.bs.iotleg.com
db-implementation=file
datasource=/etc/flexisip/users/users.db.txt
trusted-hosts=127.0.0.1 192.168.0.XX
hashed-passwords=true
reject-wrong-client-certificates=true

Next, reboot by typing reboot

Verify it is listening and the transport listens on your IP:

~# ps aux|grep flexis
bticino    741  0.0  0.3   9732  1988 ?        SNs  Oct28   0:00 /usr/bin/flexisip --daemon --syslog --pidfile /var/run/flexisip.pid --p12-passphrase-file /var/tmp/bt_answering_machine.fifo --transports sips:192.168.0.XX:5061;maddr=192.168.0.XX;require-peer-certificate=1 sip:127.0.0.1;maddr=127.0.0.1  sip:192.168.0.XX;maddr=192.168.0.XX
bticino    742  0.1  1.6  45684  8408 ?        SNl  Oct28   1:44 /usr/bin/flexisip --daemon --syslog --pidfile /var/run/flexisip.pid --p12-passphrase-file /var/tmp/bt_answering_machine.fifo --transports sips:192.168.0.XX:5061;maddr=192.168.0.XX;require-peer-certificate=1 sip:127.0.0.1;maddr=127.0.0.1  sip:192.168.0.XX;maddr=192.168.0.XX

Next we need to use the account name that our Door Entry app uses:

$ cat /etc/flexisip/users/users.db.txt

username-hotmail.com-93C0CF187E366B254@XXXXXXX.bs.iotleg.com md5:00411460f7c92d2124a67ea0f4cb5f85;
....

Check Syncing of users when this file happens to be empty.

You will find several entries that start with the email address you used in your Door Entry app, I used that one to test.

Interesting to see is that the MD5 does not match the MD5 of my password in the Door Entry app (I just did a quick check, so they might be mangled by them?).

Reversing them with some online tools does not yield much results (yet … might be fun to look into this at some point).

Next, In the Linphone client, click on Account Assistant and Use a SIP account, use your account login.

Put username-hotmail.com-FEIOFJEIOFJEOIFJ in username.
XXXXXXX.bs.iotleg.com in domain.

Replace with your values of course… from /etc/flexisip/users/users.db.txt

No need to enter password.

Transport TCP.

Next, create a new contact in the Linphone client, give it a name and enter sip:c300x@XXXXXXXX.bs.iotleg.com in SIP account field.

Next, we need to make sure we can reach XXXXXXXX.bs.iotleg.com. So add this hostname to the hosts file (guides are online).
Hint: edit /etc/hosts on Mac/Linux … windows … check on google ;-)

Make it resolve to the IP of your intercom unit.

Restart the Linphone client (sometimes it needs to reload the hostname from the hosts file you just added).

Next, tail the log on the intercom.

# tail -f /var/log/log_rotation.log

Start a video call to the c300x contact on Linphone client and check the log for all the useful info in there. I won't copy/paste much. It has sensitive data.

You should at least find:

flexisip - - -  nta: received INVITE sip:c300x@XXXXXXX.bs.iotleg.com SIP/2.0 (CSeq 20)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
flexisip - - -  nta: INVITE (20) to message callback                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
flexisip - - -  New SipEvent 0x16031b4 - msg 0x15fdf30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
flexisip - - -  Receiving new Request SIP message INVITE from sip:username-hotmail.com-OIEFJEIOFJEIOFJEOIFJEFOJ@XXXXXXX.bs.iotleg.com :#012INVITE sip:c300x@XXXXXXX.bs.iotleg.com SIP/2.0#015#012Via: SIP/2.0/TCP 192.168.XX.XX:57366;branch=z9hG4bK.m79GjLeBw;rport=57366#015#012From: <sip:username-hotmail.com-OIEFJEIOFJEIOFJEOIFJEFOJ@XXXXXXX.bs.iotleg.com>;tag=uK~Uw7yNa#015#012To: sip:c300x@XXXXXXX.bs.iotleg.com#015#012CSeq: 20 INVITE#015#012Call-ID: m1FJTLgmTM#015#012Max-Forwards: 70#015#012Supported: replaces, outbound, gruu#015#012Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE, SUBSCRIBE, INFO, PRACK, UPDATE#015#012Content-Type: application/sdp#015#012Content-Length: 847#015#012Contact: <sip:username-hotmail.com-OIEFJEIOFJEIOFJEOIFJEFOJ@XXXXXXX.bs.iotleg.com;gr=urn:uuid:122beb47-575d-0013-9522-44f54cf1bc85>#015#012User-Agent: Linphone Desktop/4.4.1 LinphoneCore/5.1.19-1-g6cdd0918e#015#012#015#012v=0#015#012o=username-hotmail.com-OIEFJEIOFJEIOFJEOIFJEFOJ 2552 1319 IN IP4 192.168.XX.XX#015#012s=Talk#015#012c=IN IP4 192.168.XX.XX#015#012t=0 0#015#012a=rtcp-xr:rcvr-rtt=all:10000 stat-summary=loss,dup,jitt,TTL voip-metrics#015#012m=audio 7078 RTP/AVP 96 97 98 0 8 18 99 100 101#015#012a=rtpmap:96 opus/48000/2#015#012a=fmtp:96 useinbandfec=1#015#012a=rtpmap:97 speex/16000#015#012a=fmtp:97 vbr=on#015#012a=rtpmap:98 speex/8000#015#012a=fmtp:98 vbr=on#015#012a=fmtp:18 annexb=yes#015#012a=rtpmap:99 telephone-event/48000#015#012a=rtpmap:100 telephone-event/16000#015#012a=rtpmap:101 telephone-event/8000#015#012a=rtcp-fb:* trr-int 1000#015#012a=rtcp-fb:* ccm tmmbr#015#012m=video 9078 RTP/AVP 96 97#015#012a=rtpmap:96 VP8/90000#015#012a=rtpmap:97 H264/90000#015#012a=fmtp:97 profile-level-id=42801F#015#012a=rtcp-fb:* trr-int 1000#015#012a=rtcp-fb:* ccm tmmbr#015#012a=rtcp-fb:96 nack pli#015#012a=rtcp-fb:96 nack sli#015#012a=rtcp-fb:96 ack rpsi#015#012a=rtcp-fb:96 ccm fir#015#012a=rtcp-fb:97 nack pli#015#012a=rtcp-fb:97 ccm fir#015
...

Step: 1#011Found contact sip:c300x@2068164.bs.iotleg.com -> sip:c300x@127.0.0.1:55788;transport=tcp;regid=fze5fze5f usedAsRoute:0
Step: 1#011Returning collected records 1      

flexisip - - -  Listener created for sipUri = sip:c300x@XXXXXXXX.bs.iotleg.com
flexisip - - -  Subscribe topic = c300x@XXXXXXXX.bs.iotleg.com
flexisip - - -  Inject Request SIP message:#012INVITE sip:c300x@127.0.0.1:55788;transport=tcp ... rest ommitted
...
flexisip - - -  Record route added.
flexisip - - -  Sending Request SIP message to sip:c300x@127.0.0.1:55788;transport=tcp#012INVITE sip:c300x@127.0.0.1:55788;transport=tcp
...
flexisip - - -  Message is sent through an outgoing transaction.
flexisip - - -  nta: received 100 Trying for INVITE (20)
flexisip - - -  nta: received 488 Not acceptable here for INVITE (20)
flexisip - - -  nta: 488 Not acceptable here is going to a transaction

This is where the Linphone client hangs up with a message Remote party cannot accept the call.
We probably don't provide all the needed options for SIP … or it is requiring TLS in someway maybe.

A normal call doesn't return a 488 but a 200 and continues like this where the aswm also kicks in.

flexisip - - -  nta: sent 200 Ok for INVITE (20) 
 flexisip - - -  Looking at fork contexts with key c300x@XXXXXXX.bs.iotleg.com
 flexisip - - -  Remove fork c300x@XXXXXXX.bs.iotleg.com from store
 flexisip - - -  Unsubscribe topic = c300x@XXXXXXX.bs.iotleg.com
 flexisip - - -  Destroy ForkCallContext 0xc43cd4
 <aswm> - - -  SIP: Media established

After some time on Google I came across this post:
https://stackoverflow.com/questions/15852013/sip-2-0-488-not-acceptable-here-error

And indeed comparing the normal call and the attempted called revealed in the SIP message:

... TTL voip-metrics#015#012a=DEVADDR:20#015#012m=audio 46160 RTP/SAVP 96 97 98 0 8 99 ....
and
... TTL voip-metrics#015#012m=audio 7078 RTP/AVP ...

Since everyone was talking about encryption I went to look for the Encryption option in the Linphone settings.

Under calls and chat change from none to SRTP under Encryption.

Call your contact again. Your intercom will ring. You can pickup with the green phone.

This was an intercom to Client connection. Not what I was expecting, but at least we managed to setup a call.

Other findings

Syncing of users

At a moment I had a Forbidden error when I tried connecting from the Door Entry App to the intercom.

It seemed like my users were gone in /etc/flexisip/users/users.db.txt.

When tracing bt_answering_machine I came across an interesting line:

$ strace -p 650 2>&1 | egrep -v "gettime|poll"

stat64("/var/tmp/user_list.txt", 0x7ea109f0) = -1 ENOENT (No such file or directory)

Every once in a while this file gets polled.

I placed an empty file there at some point to test what would happen.

I think this file was then copied to `/etc/flexisip/users/users.db.txt and erased my users.

We need to verify this.

Anyhow, I wasn't sure how to reset this. I deleted the Door Entry app and reinstalled it.

You then get a popup saying that it will sync and be available shortly.

This is a lovely time to check out the /var/log/log_rotation.log file to see some interesting stuff happening.

You will see how the SIP accounts/records/routes get created.

flexisip - - -  Syncing password file
flexisip - - -  Opening file /etc/flexisip/users/users.db.txt
flexisip - - -  Syncing done

flexisip - - -  Found registrar domain: XXXXXXXX.bs.iotleg.com
flexisip - - -  Reading static records file
flexisip - - -  Creating AOR alluser@XXXXXXXX.bs.iotleg.com association
flexisip - - -  Trying to insert new contact sip:....omitted....
... more accounts being added ....

Receive a "ring door" in the linphone client:

In order for the door call to be received also by the linphone client it is necessary to add the registered user in the following files (respecting the already existing input format):

/etc/flexisip/users/users.db.txt
/etc/flexisip/users/route.conf
/etc/flexisip/users/route_int.conf

Config a Baresip client in Raspberry (comparable to the desktop client linphone)

the steps will then be divided into 2 main parts:

  1. will be to set up c300x to accept the raspberry client
  2. install and configure the Baresip client in raspberry

So…
1° part

  1. a) create a new user in /etc/flexisip/users/users.db.txt
  2. b) add raspberry ip address in /home/bticino/cfg/flexisip.conf separed with a blank space –> trusted-hosts=
  3. c) add user created in 1)a) into /etc/flexisip/users/route.conf and /etc/flexisip/users/route_int.conf
  4. d) reboot c300X

2° part on raspberry (this is based on the document https://github.com/spbroot/sipdoorbell rep)
2) a) Adding the ALSA Loopback Sound Device:

sudo su
echo 'snd-aloop' >> /etc/modules
exit

NOTE

On Ubuntu server you might need to install:

$ sudo apt install linux-modules-extra-$(uname -r) alsa-base
sudo reboot

To have the ALSA config

  1. b) Making changes to the ALSA configuration
    Create or edit the /etc/asound.conf file
    # output device
    pcm.loop_out {
      type dmix
      ipc_key 100001
      slave.pcm "hw:Loopback,0,0"
    }

    # input device
    pcm.loop_in {
      type dsnoop
      ipc_key 100002
      slave.pcm "hw:Loopback,1,1"
    }

    # plug device
    pcm.sipdoorbell_return {
      type plug
      slave.pcm 'loop_out'
      hint {
        show on
        description 'sipdoorbell return channel'
      }
    }

    # plug device
    pcm.sipdoorbell_main {
      type plug
      slave.pcm loop_in
      hint {
        show on
        description 'sipdoorbell main channel'
      }
    }

!!! if you get homebridge error when open doorbell device on iphone/ipad/AppleTV:

[error] cannot open audio device sipdoorbell_main (No such file or directory)
[error] sipdoorbell_main: Input/output error

or

[error] cannot open audio device sipdoorbell_return (No such file or directory)
[error] sipdoorbell_return: Input/output error

put this code at the end /usr/share/alsa/alsa.conf instead /etc/asound.conf

Taken from: https://github.com/spbroot/sipdoorbell

  1. c) Installing Baresip

sudo apt install baresip

Run baresip to create a configuration file, after launch press 'q' to exit
Making changes to the baresip configuration file: file /home/pi/.baresip/config

Remove the comment from the following lines and make changes

sip_listen              0.0.0.0:5060

audio_player            alsa,hw:Loopback,0,1
audio_source            alsa,hw:Loopback,1,0
audio_alert             alsa,hw:Loopback,0,1

ausrc_srate             48000
auplay_srate            48000
ausrc_channels          2
auplay_channels         2

module                  opus.so
module                  srtp.so
module                  httpd.so

http_listen             0.0.0.0:8000
  1. d) Add a SIP account to the file /home/pi/.baresip/accounts

<sip:*USERCREATEDIN1-a*@XXXXXXX.bs.iotleg.com;transport=tcp>;outbound="sip:*IPINTERCOM*;transport=tcp";auth_user=*FIRT_PART_USER_CREATED_IN_1a*;auth_pass=NOPASSNEED;mediaenc=srtp;audio_codecs=opus/48000/2;audio_source=alsa,default

  1. e) Reboot Raspberry
  2. f) run Baresip:
baresip

Try to made a call with /dial command

Processes

Process Function More info
BtClass The frontend GUI on the intercom Started from: /home/bticino/binBtClass_qws and loading it's UI from /home/bticino/bin/gui
bt_answering_machine Handles recordings of incoming calls
openserver OpenWebNet implementation https://developer.legrand.com/documentation/open-web-net-for-myhome/ listening on port 20000 see cat /var/tmp/conf.xml for the password under pwd_open - also handling the Authentication of Door Entry app at load (see Axiom client in logs)
bt_vct Receives openwebnet commands on port 30006, already used to unlock door

Commands

  • Open and close lock echo *8*19*20## |nc 0 30006; sleep 1; echo *8*20*20##|nc 0 30006 (also sometimes 21 instead of 20)

Viewing firmware differences

This step of notes explains the commands you can use to compare the different firmware versions.

You will need to understand how to extract the firmware and mount it beforehand.

The instructions are for executed on a Linux host. You might need to be root to mount.
You also need git to be installed to make life easy.

root@server:/home/user/download# mount /home/user/1_7_17/btweb_only.ext4 /media/1_7_17
root@server:/home/user/download# mount /home/user/1_7_19/btweb_only.ext4 /media/1_7_19 && cd /media
root@server:/media# ls
1_7_17  1_7_19
root@server:/media# mkdir tmpdir
root@server:/media# cp -r 1_7_17/* tmpdir/ && cd tmpdir
root@server:/media/tmpdir# git init && git add * && git commit -m Initial
root@server:/media/tmpdir# cp -r ../1_7_19/* .
root@server:/media/tmpdir# git diff
Select a repo