9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice.

Free e-book

[Link]
[Link] SIP Tester IP Networks Testing Softswitch

Measuring
Measuringquality
quality
CallXML Misc Contact us

VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
The book is designed for people who want to learn how to fix issues with VoIP - SIP, RTP, RTCP, UDP, TCP, IP protocol stack. It includes small tasks for homework to check your new learne
skills by practice.
Please contact us if you want us to extend or clarify any part of the book.

Warning: the e-book is protected by copyright law, making copies is not allowed

The e-book is dynamically updated with new content.

Subscribe to changes in the book

email:   
Subscribe
 

Contents

What is VoIP. VoIP business types, market players.

What is VoIP call

Encapsulation and structure of packets

Protocols used for VoIP

Network protocol analyser tool "Wireshark"

IP protocol (IP)

Transport protocols: User Datagram Protocol (UDP)

Transport protocols: Transmission Control Protocol (TCP)

SIP protocol
SIP response codes

SIP trace

Types of SIP packets

SIP headers

Registration

Digest authentication

SIP extensions
Real-time Transport Protocol (RTP)

RTP streams in Wireshark

Types of VoIP issues

VoIP quality indicators

VoIP software bugs

VoIP troubleshooting methods

Real VoIP troubleshooting cases

What is VoIP. VoIP business types, market players.

VoIP stands for "voice over IP protocol". Today it is used not only for voice: it transfers video data,
IM (instant messaging, text chat)/presence, other data like SS7 protocol (SIGTRAN). Majo
benefit of the VoIP is low price of data transfer: internet is very cheap.
Another benefit is low price of both client-side and server-side hardware needed for voice/video calls: VoIP clien
software can run on cheap Android or Raspberry Pi devices, on already existing Windows laptops and PCs;
VoIP server software can run on Linux or Windows servers; no special hardware
needed (if we compare VoIP with SS7/TDM).
The VoIP has helped to reduce price of international calls dramatically, prices still go down (it is good for customers).
Major drawback of the VoI
is easiness of fraud and untransparency in market: fraud in IP network is much easier than fraud in PSTN network.

Here is list of popular VoIP business types, VoIP market players. The VoIP market is complex and some business types are confidential, so this list is not full.

OTT (over the top) app. developers provide apps for free voice/video calls over internet; Skype, WhatsApp, Viber, etc. The OTT apps allow free calls between people having sam
application installed on 2 remote devices.
They also allow making calls to real phone numbers for some payment (in this way generating VoIP retail traffic), and in case of internationa
OTT bypass allow making VoIP calls to real phone number which is linked to customer's account,
via internet and for lower price (compared to termination prices of MNOs).
Mobile network operators (MNOs) / tier-1 operators offer official ("white") VoIP routes where calls are terminated to real phones over GSM networks. Such VoIP routes are direc
they deliver Caller ID, and they have great quality and high price.
VoIP retail - B2C business when customers make mostly long-distance (international) calls to real phone numbers at cheaper price (compared to dialing from mobile phones).
Suc
business aggregate voice calls from many people (typically immigrants) and send the voice traffic to VoIP termination providers.
Wholesale VoIP termination providers (tier-2, tier-3 operators) aggregate VoIP traffic and resell VoIP routes.
Having many interconnections,
wholesalers provide A-Z VoIP routes
VoIP routes to all countries in the world.
Also, the wholesalers are able to redirect calls to multiple redundant routes: if one route is down, call gets redirected to another one which
up.
Some wholesalers specialize on particular country or region and particular VoIP traffic type.
Grey VoIP termination providers (are considered by MNOs as fraudsters) provide a way to send international (long-distance) voice calls to real phone numbers via GSM gateway
(SIM boxes), reducing price by using local tariffs.
Together with OTT termination providers, the GSM termination providers play important role in the market, reducing prices for call
Although the GSM termination is not considered as "legal" in destination countries,
we see that it is done by many people including corrupt employees, right inside MNO.
VoIP marketplaces where wholesalers and direct providers meet and interconnect: various websites and global telecom industry events/conferences
Fraud management and revenue assurance (FM&RA) companies help MNOs to fight against grey termination - block SIM cards that are being used for grey international traffi
This business does not only include blocking SIM cards, there are many other ways of revenue leakage; it is very complex and very confidential. FM&RA use many interconnections t
send test calls to block SIM cards which are used by grey termination providers
Call centers receive calls on real phone number(s) and transfer the calls to agents.
The call centers may also generate outgoing calls in case of telemarketing, this type of VoIP traff
(CC) is unloved by VoIP termination providers.
Incoming calls go to call centers via PSTN-SIP gateway or via DID providers
DID (direct inward dialing) providers route calls from real phone number to call center or IP PBX via VoIP protocol
IP PBX / UC (unified communications) platforms connect employees of a company with free voice/video calls; also connect employees using real phone number(s). The IP PB
systems generate retail VoIP traffic and send it to VoIP providers
Software/hardware developers - the complex VoIP market needs complex software and hardware for every component of VoIP system, also test tools and services. Some companie
like Speedflow and China Skyline also do VoIP wholesale business
Looped generated traffic providers send VoIP calls via a chain of VoIP wholesalers for various confidential purposes. The generated calls usually get disconnected if generated traff
gets terminated by some other provider or IVR, or if generation side detects mismatch in billing.
IPRN (international premium rate number) providers are same as DID providers except that the calls are used for payments from caller to service provider. Unlike a normal ca
premium rate calls are more expensive, and part of the call charge is paid to the service provider,
in this way enabling businesses to be funded via the calls. Adult chat lines (phon
sex) and tech support are a very common use of premium-rate numbers. Other services include directory enquiries, weather forecasts, competitions and voting (especially relating t
TV shows).
Various fraudsters and hackers - see types of VoIP fraud and attacks here

For advanced readers - diagram of international VoIP market is here.

What is VoIP call

[Link]/VoIP/VoipTroubleshootingBook/[Link] 1/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
VoIP call is a series of IP packets with data inside the packets, sent between caller and callee over IP network.
Structure (format) of the data inside the packets and way of communicatio
between caller and callee are standartized by protocols. In 2018 most typically used protocols for VoIP are IP, UDP, SIP and RTP.
Here is diagram of VoIP call using the above-listed protocols

Arrows mean transmitted packets; "SIP/UDP/IP" means SIP-protocol data encapsulated into UDP and IP packet, in other words SIP packet transmitted by means of UDP and IP protoco
"RTP/UDP/IP" bold bidirectional arrow means multiple audio packets with voice data transmitted between caller and callee in two ways. The voice audio data is encoded by code
encapsulated into RTP and transmitted by means of UDP and IP protocol.

Encapsulation and structure of packets

Encapsulation in IP packets is a way how the data inside packets is organised: one upper-level protocol is nested inside another lower-level protocol.

The next picture displays captured packet (sniffed, recorded packet data) by wireshark software tool.

[Link]/VoIP/VoipTroubleshootingBook/[Link] 2/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book

Protocols used for VoIP

Network protocol analyser software tool "Wireshark"

Before you continue with the protocols, please download and install wireshark - free and open source software tool.
It is necessary for VoIP troubleshooting and understanding netwo
protocols.
The wireshark is able to record (sniff, capture) packets from network adapter, save them to file (.pcap, .pcapng), parse captured packets and display them in user-friendly way.

also includes tools to parse, display, listen VoIP calls, RTP media streams.
To analyse a VoIP issue with Wireshark one should

Select network adapter(s), start capture of VoIP packets. Reproduce the issue. Stop capture. The recorded packets could be exported/imported via .pcap/.pcapng files.
Apply a display filter. Examples: "sip", "sip || rtp", "http", "tcp", "udp"
Look into captured packets and see the sequence, timestamp and contents of packets
Analyse captured VoIP call(s) (Menu - Telephony - VoIP calls)

[Link]/VoIP/VoipTroubleshootingBook/[Link] 3/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
Analyse captured RTP streams (Menu - Telephony - RTP - RTP Streams)

Task: please do all above things yourself: record any network activity (e.g. you can visit HTTP or HTTPS website and record it in wireshark).
You can use sample pcap file with SIP and RTP
packets, single VoIP call.

The VoIP troubleshooting is complex; it requires deep understanding of protocols.

IP protocol (IP)

The Internet protocol (IP) is designed to send data packets across IP networks from source host to destination host. The source and destination hosts both have IP address.
There are 2 mos
popular versions of the IP protocol: version 4 and version 6. In 2018 most widely used version is still version 4; in the IP v.4 the address includes 32 bits and is written as 4 numbers (byte
from 0 to 255 separated by dot, example: [Link].
In IP v.6 the address includes 128 bits, written as 8 groups of 4 hexadecimal digits with the groups being separated by colons, exampl
2018:0db8:0000:0042:0000:8a2e:0370:5678.
Here is a task to check your new skills:

Please download this pcap file,

open it in wireshark and see what is source IP address and what is destination IP address of the single packet inside the file.

If you are unable to do it, please contact us and we will clarify this chapter.
As you can see in wireshark, IP protocol data in the packet (also called IP header) has many fields, most importan
are "Source", "Destination". The "Differentiated Services Field" (includes DSCP bits) is used to prioritise VoIP traffic in IP routers.

Tasks with this file:

See what are source and destination IP addresses of the first packet in file (SIP INVITE packet)
See what is "Differentiated Services Field" of the first INVITE packet
See what are IP addresses of RTP packets
See what is "Differentiated Services Field" of the all RTP packets. Is there any pattern?
See what is average time period (in milliseconds) between consequent RTP packets going from one source IP address to another
See what is size of RTP packet, what is size of largest and smallest SIP packet
See what is size of IP header

Transport protocols: User Datagram Protocol (UDP)

The UDP is designed to send data over IP protocol and UDP header contains fields "Source port" and "Destination port". The port is a 16-bit integer having values from 0 to 65535.
Multip
applications running on source and destination host at the same time use different port numbers; the port number identifies application which is linked (bound) to the port.
Standard po

[Link]/VoIP/VoipTroubleshootingBook/[Link] 4/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
number for SIP protocol is 5060; for RTP protocol port numbers are assigned dynamically.
The UDP protocol has no means to control delivery of packets from source to destination, so th
packets could be lost in IP network. Small packet loss (below 2%) is normally handled by SIP and RTP.

Tasks with this file:

See what is source and destination UDP port number for both SIP and RTP packets in the file

Transport protocols: Transmission Control Protocol (TCP)

In some cases VoIP calls go over Transmission Control Protocol (TCP). In other words same SIP packet data can be encapsulated into either TCP or UDP packets.
The TCP protocol heade
contains source and destination port numbers as UDP, it also contains fields used to control guaranteed delivery of packets.
TCP protocol provides guaranteed delivery of packets in sam
order from source to destination, it retransmits lost packets unlike UDP protocol.
On the other hand TCP adds extra delay for setting up connection, increasing Post-dial delay (PDD).
Beside
the guaranteed delivery of data, use of TCP instead of UDP for VoIP means

Better passing through IP routers (NAT, firewalls): TCP is used for HTTP and HTTPS web browsing and it works everywhere
Better security: source IP address in TCP can not be spoofed
SIPS: encryption of SIP data using TLS method (similar to HTTPS encryption)
Stateful processing of IP packets, no possibility of lightweight stateless SIP proxy.

Task: open this file to see which TCP ports are used by caller and called endpoints.

Note: wireshark does not automatically recognize TCP packets in this file as SIP. Please select TCP packet, click "Decode As" - enter "SIP" in table field "Current" - OK

The TCP protocol introduces concepts of client, server and connection. Client initializes TCP connection to server using TCP SYN packet; the connection is ended with TCP FIN packet. Defau
server-side TCP port number for SIP is 5060; client-side TCP port number is selected randomly.

Session Initiation Protocol (SIP)

The SIP is most widely used protocol for VoIP calls. It defines specific format of messages (SIP packets) and their sequence to make calls over IP networks.
The SIP messages are carried b
means of UDP, TCP, or TLS/TCP (it is called SIPS).
A typical sequence of SIP messages during a VoIP call (SIP call flow diagram) is displayed in beginning of the book, here we show it on
more time:

INVITE packet (request from endpoint A to endpoint B) means initiation of call: "A wants to make new call to B"
100 Trying packet (response from B to A to INVITE) means confirmation of INVITE packet delivery: "B received the INVITE". 100 means response code
180 Ringing packet (response from B to A to INVITE) means indication of ringing state: "Phone B started to make rings". At ths time caller (A) starts to hear ringback tone. 180 mean
response code
200 OK packet (response from B to A to INVITE) means indication of answer (connection, start of billing): "Phone B answered to the call". 200 means response code
ACK packet (from B to A) means confirmation of 200 OK packet delivery: "A received the 200 OK"
BYE packet (request from A to B) means termination of call: "Phone A ends the call"
200 OK packet (response from B to A to BYE) means confirmation of BYE packet delivery: "B received the BYE"

Task: open this file in wireshark, go to "Telephony - VoIP calls" in menu and see SIP call flow diagram for the recorded VoIP call.

Other possible scenarios of VoIP calls:

[Link]/VoIP/VoipTroubleshootingBook/[Link] 5/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
Call is ended by B after answer

Difference from previous flow is initiator of BYE packet

Task: open this file in wireshark, see which party (A or B, caller or callee) ended the call with BYE.

Call is ended by A before answer

Caller party (endpoint A) sends SIP packet "CANCEL" before connection - means "A ends the call before answer"
Called party (endpoint B) responds to CANCEL with "200 OK" - means "B confirms receiving of CANCEL"
Called party (endpoint B) responds to initial INVITE with "487 Request Terminated" - means "B confirms ending of call". 487 means response code
Caller party (endpoint A) sends ACK to 487 packet - means "A confirms receiving of 487"
Call is rejected by B (ended before answer)

Called party (endpoint B) responds to initial INVITE with "486 Busy Here" - means "B rejects the call with status = busy". 486 means response code
Caller party (endpoint A) sends ACK to 486 packet - means "A confirms receiving of 486"

[Link]/VoIP/VoipTroubleshootingBook/[Link] 6/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
Normal call flow (same as on first diagram) but with audio ringtone coming from B (can be ringback tone or music)

Called party (endpoint B) responds to INVITE with "183 Session Progress" which means initiation of audio channel between A and B
RTP media packets (audio signal) are sent between endpoints after "183", before "200 OK" (answer signal), it is called early media. Note that it is bidirectional audio stream
but it does not mean that SIP servers always pass early media from A to B. It depends on every particular softswitch. From our experience we see that most softswitches real
pass early media from caller to called party in early media, before start of billing. It is safe for caller-side software because caller does not pay for early media RTP traffic.

Tasks:
Does this file contain early media (RTP packets before 200 OK)?
Does this file contain early media?

Caller and called parties are also called SIP endpoints, A and B, user agents, UAC (User Agent Client) and UAS (User Agent Server). Full technical documentation is available in RFC326
standard document published by IETF.
Also, you can find many articles and tutorials published on our website.

SIP response codes

In this book we publish only most needed information about SIP, here is list of most popular response codes. Full list of SIP response codes is in RFC3261,

100 Trying - confirmation about receiving INVITE request

180 Ringing - indication of ringing phone and ringback tone
183 Session Progress - set up of early media (sound coming from destination with custom ringback tone before answer, during call setup)
200 OK - answer of the call, connection between A and B endpoints
302 Moved Temporarily - redirection of call to another destiantion
400 Bad Request - indication of incorrect format (malformed request packet). Usually the malformed packets are sent by hackers. Normally the 400 response is received if endpoin
have different understanding of SIP packet format and syntax (e.g. if made by different vendors)
401 Unauthorized - it is actually request of another INVITE with encrypted user/password credentials
403 Forbidden - means that the caller has no right to make call (invalid user name or password, invalid source IP address)
404 Not Found - usually means wrong number format or call to non-existent extension in PBX
401 Proxy Authentication Required - it is actually request of another INVITE with encrypted user/password credentials. Same as 401, but sent by SIP servers
408 Request Timeout - means that UAC did not receive any response from UAS after series of INVITE retransmissions. Usually happens when destination server is down, whe
sending call to wrong IP address and port, when INVITE is blocked by firewall
486 Busy Here - destination is busy (usually busy by another call)
487 Request Terminated - usually follows after CANCEL SIP packet, when call is cancelled, ended before answer
488 Not Acceptable Here - usually means incompatible codecs between UAC and UAS
500 Internal Server Error - some internal error happened in the UAS, usually indicates a bug in VoIP server, or malformed INVITE request
503 Service Not Available - usually means temporarily overload: UAS can not accept more calls at this time
603 Decline - means generic declining for various reasons

Task: see SIP response code in this file

SIP trace

SIP trace is a series of captured/recorded SIP packets, SIP packet data represented as text. The SIP trace is necessary element for VoIP troubleshooting.
When something goes wrong wit
your phone system, the first thing to look at is SIP trace. SIP trace and pcap recordings provide evidence of VoIP issues.

SIP messages contain request/status line (red), headers (blue) and body (green) (format of body is defined by protocol SDP).
Here is sample SIP trace:

SIP messages for Call-ID 524c3ea2fdd1401ab4a90c940b073359

17:42:22.442 [+0,00ms] [TX] INVITE to [Link]:6000

INVITE sip:123456@[Link] SIP/2.0

Via: SIP/2.0/UDP [Link]:5090;rport;branch=z9hG4bKPjefcc1928b9d44bf68b202f5674f15b9c

Max-Forwards: 70

From: sip:unknown@[Link];tag=d03ca0a253134ecc82c17a89e4b31e45

To: sip:123456@[Link]

Contact: <sip:unknown@[Link]:5090>

Call-ID: 524c3ea2fdd1401ab4a90c940b073359

CSeq: 17970 INVITE

Allow: INFO, PRACK, SUBSCRIBE, NOTIFY, REFER, INVITE, ACK, BYE, CANCEL, UPDATE

You can see INVITE request sent by UAC, multiple responses sent by UAS (100 Trying, 183 Session Progress, 503 Service Unavailable).
SIP Request Line INVITE sip:123456@[Link]
SIP/2.0 means request type = "INVITE" (making a VoIP call), SIP protocol version = "2.0" (the only one version used, as we see in 2018), and destination SIP URI

[Link]/VoIP/VoipTroubleshootingBook/[Link] 7/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
"sip:123456@[Link]".
The URI in SIP is simular to URL in HTTP, it specifies target for the request. "sip:" or "sips:" here define protocol type, "123456" is called number (telephon
number of B endpoint).
In next chapters we will describe the SIP trace in detail.

Types of SIP packets

SIP Requests also known as SIP methods, SIP request methods. Most popular SIP request types:

INVITE - most widely used SIP request, means request to make new VoIP call from UAC to UAS. When the INVITE is sent after connection (after "200 OK response") it usually mean
change of audio codec, putting call on hold, or sending/receiving fax
ACK - confirmation that response was delivered to UAS
BYE - ends current VoIP call after answer, sent by UAS or UAC
CANCEL - ends a call before answer, sent by UAC
REGISTER - sent from SIP phone user-agent to SIP server to register the phone to specific extension (i.e. internal phone number)
REFER - starts call transfer procedure: e.g. A makes call, B answers, understands that call needs to be passed to C and sends request to B: "please transfer to C"
We have listed only most popular SIP request types, see others in RFC3261

SIP Responses

The SIP response packets are is sent back to requesting endpoint and contain SIP response code and optional error details in SIP headers:

SIP/2.0 500 Internal Server Error

Via: SIP/2.0/UDP [Link]:5081;branch=z9hG4bK-10854-1-0

From: sipp <sip:111@[Link]:5081>;tag=10854SIPpTag001

To: sut <sip:222@[Link]:5060>;tag=38404-23F2

Date: Tue, 05 Feb 2013 05:49:37 GMT

Call-ID: 1-10854@[Link]

CSeq: 1 INVITE

Allow-Events: telephone-event

Warning: 399 [Link] "Transcoder Not Configured"

Server: Cisco-SIPGateway/IOS-15.3.20130202.123643.

Reason: Q.850;cause=16

Content-Length: 0
In this example "Warning" header gives details about the error and "Reason" header gives information about PSTN release code.

If a packet is lost somewhere between the endpoints A or B (UDP protocol does not gaurantee delivery of packets), the SIP packets are retransmitted.

Task: see types of SIP packets (requests, responses), and number of retransmitted packets in file #1, file #2, file #3.
See what is destination IP address where packets are retransmitted
to, do you think it is correct or not?

SIP headers

A header in SIP packet has a simple format: [name]: [value]. SIP headers are separated by line breaks.
Most widely used SIP headers are

From header indicates identity of caller, usually contains Caller ID (ANI, CLI, A number). Also contains unique "tag" set by UAC
To header identifies recipient of the call, usually contains Called ID (called number, B number). Also contains unique "tag" set by UAS
Via header saves information about proxy servers - server IP address and port - to pass response from UAS back to UAC via same set of proxy servers
Record-Route header is optionally inserted by proxy servers to route subsequent SIP requests for the same SIP call
Call-ID header identifies SIP call along with "From" and "To" headers
Contact header specifies IP address and port to be used by peer SIP endpoints for subsequent SIP requests for the same SIP call
CSeq header contains a decimal number which increases by 1 with every subsequent SIP request within the same SIP call, with the exception of CANCEL and ACK requests which us
the CSeq number of corresponding INVITE request
Server header indicates model, software/firmware version of UAS
User-Agent header indicates model, software/firmware version of UAC
Content-Length header as you will read later, INVITE and "200 OK" packets usually contain encapsulated SDP protocol data with information about RTP audio streams.
This heade
specifies size of encapsulated SDP (or other) data
Content-Type header specifies type of encapsulated data. Equals to "application/sdp" in case of encapsulated SDP
Allow header lists set of SIP requests supported by user agent which generated the SIP message
Supported header specifies set of supported SIP extensions by user agent which generated the SIP message
Require header specifies set of required SIP extensions by user agent which generated the SIP message
Session-Expires header - part of RFC 4028 (Session Timers) SIP extension - specifies max. interval to refresh SIP call with keep-alive packet (the RFC3261 standard does not defin
way to abort hang calls)
Expires header specifies lifetime of corresponding SIP request
WWW-Authenticate, Authorization headers are used for digest authentication

There are many more headers; adding custom headers into requests and responses is a way to extend SIP protocol.

Tasks:

See SIP headers in this file: what software is used for UAC and UAS? What is SIP Call ID?
See how to fix Contact header sent by UAS in this file

Registration

Registration in SIP means passing information about user agent to server, it is implemented with REGISTER.
It means "Hello SIP server, here are my contact details for extension number 45
IP address [Link], SIP port 5060".
Without registration SIP phones must know IP addresses of every destination phone; with registration the IP addresses are stored by SIP server, also know
as SIP registrar server.
Her is example SIP trace for the registration process:

SIP messages for Call-ID 64b10daa51dd4c5a9d6038a43163b22d

14:32:00.599 [+0,00ms] [TX] REGISTER to [Link]:6000

REGISTER sip:[Link] SIP/2.0

Via: SIP/2.0/UDP [Link]:5090;rport;branch=z9hG4bKPj609d6772ba754dcab9cba92bcb34be69

Max-Forwards: 70

From: <sip:100@[Link]>;tag=aef6297c37914f69832efc2c44c980aa

To: <sip:100@[Link]>

Call-ID: 64b10daa51dd4c5a9d6038a43163b22d

CSeq: 22151 REGISTER

Allow: INFO, PRACK, SUBSCRIBE, NOTIFY, REFER, INVITE, ACK, BYE, CANCEL, UPDATE

Contact: <sip:100@[Link]:5090>

As you can see in the SIP trace, first REGISTER request sent by UAC [Link] to UAS [Link] means resitration of extension 100 at SIP server [Link] on SIP port 6000.
Th
UAC sends its IP address in Contact header: Contact: <sip:100@[Link]:5090>.
"Expires" SIP header equals to 3600 means that the registration entry should be valid for 3600 second

[Link]/VoIP/VoipTroubleshootingBook/[Link] 8/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
-
in 60 minutes the registrar server should clear the registration entry if not received new REGISTER requests for this extension.
REGISTER request with "Expires: 0" means unregistration.

Digest authentication

As you can see in previous SIP trace (with REGISTER), first request is rejected with status "401 Unauthorized" and WWW-Authenticate header, it is called digest challenge. The WWW
Authenticate SIP header contains nonce and opaque values which are used by UAC along with user name and password to send second INVITE request.
The second INVITE request contain
Authorization header with digest response, it is used by UAS to verify user name and password.
If user name and password are correct, UAS sends "200 OK" response, otherwise it sends "40
Unauthorized" response.

SIP extensions

Extensions of SIP protocol are defined by additional RFC standard documents (beyond RFC3261),
you can find them here.
Most important SIP extensions are

100rel extension defines PRACK method to provide reliable delivery of provisional SIP responses.
It is fully described in RFC3262.
The PRACK packets are sent in response to 18x provisional responses via chain of SIP proxy servers from UAC to UAS:

Here is example SIP call flow for PRACK:

Without the PRACK, 18x SIP responses could be lost in IP network between UAS and UAC, it means no ringback tone played to caller.
With PRACK, 18x SIP response is retransmitte
by UAS if it does not receive PRACK from UAC.
timer extension defines a keep-alive mechanism for SIP calls, it is described in RFC4068 (Session Timers). The keep-alive packets are sent as re-INVITE or UPDATE SIP packets betwee
UAC and UAS; if the keep-alive packet is not received within session interval,
UAC or UAS considers call as hang (dead) and drops the call.
Without the session timer extension ther
could appear hang calls, existing on one SIP UA and already destroyed on another SIP UA. Such situation can happen if internet connection drops between UAC and UAS during en
of call, and BYE packet gets lost.

Real-time Transport Protocol (RTP)

The RTP transfers audio or video stream between user agents over UDP protocol.
RTP media stream goes from source IP address and UDP port to destination IP address and UDP port.

most cases RTP streams in VoIP are bidirectional.

Task: open this file in wireshark, see what is source and destination IP addresses and UDP ports for RTP streams. How many RTP streams do you see in the file?

RTP is defined by RFC3550 IETF standard document; its usage for VoIP is described in RFC3551. RTP packets (encapsulated in UDP) contain following important fields:

SSRC (synchronization source identifier) - a randomly selected 32-bit identifier for source of RTP stream,
usually also unique for many VoIP calls and does not change during call (SSR
may be changed only in rare cases, for example when call is put on hold). The SSRC is used to detect RTP collisions (when multiple RTP streams are sent to same UDP port)
Sequence number - 16-bit integer number which is incrememented by 1 with every subsequent transmitted RTP packet.
Is used in jitter buffer on receiver side to detect an
compensate lost and delayed packets.
Initial value for the sequence number is usually assigned randomly.
Timestamp - 32-bit field, usually randonly initialized like sequence number and incremented with every transmitted RTP frame by number of transmitted samples.
For example fo
audio RTP packets with sample rate 8kHz and containing 20ms audio in every RTP packets, RTP timestamp gets incremented by
8000 / 1000 * 20 = 160 samples.
For a MPEG video RT
packet with sample rate 90000Hz and 30FPS it would be 90000/30 = 3000 samples per frame.
For codecs with variable bit rate (VBR) it is more complex.
Note: Timestamp an
Sequence number values may jump, and VoIP software must be able to handle the jumps.
Payload - encapsulated voice/video data, or DTMF digit.
The media stream is splitted into small frames of 10..50 milliseconds and transmitted over RTP packets. The duration of th
frame is called packet time, it is configurable in VoIP software.
The audio/video data is encoded with a codec. Type of the codec is specified by payload type and SDP.
Payload type - 7-bit integer (0-127), specifies format of encapsulated payload: encoded audio/video or DTMF digit. RTP streams are used to transfer DTMF digits (when caller presse
a key on phone during call), it is defined by RFC2833.
Range of payload types (0..127) is divided into static payload types (0..95) and dynamic payload types (96..127),
for mor
information please refer to RFC3551.
Most common static payload types / audio codecs are:
0 / G.711 mU-law (also known as PCMU) and
8 / G.711 A-law (also known as PCMA) - stateless audio codec with simplest implementation.
There is one-to-one mapping between uncompressed 16-bit PCM samples an
compressed 8-bit G.711 samples; the mapping is different for mu-law and a-law.
A-law is used in EU; mu-law is used in US.
The G.711 codec is used in TDM networks.
Samp
frequency is 8 kHz (8000 audio samples per second).
Here word stateless means that encoding/decoding of an audio sample does not use information about previous
encoded/decoded audio samples.
Bandwidth of G.711 codec is 64kbit/s.
18 / G.729 - stateful 8kbit/s audio codec, royalty-free since 2017 (patent has expired). G.729 has been extended with annexes G.729A and G.729B:
G.729A has a medium complexity, and is compatible with original G729. It provides a slightly lower voice quality
G.729B extends G.729 with silence suppression, and is not compatible with the previous versions.
The G729B introduces CNG (comfort noise generator) packets wit
smaller size, in this way reducing bandwidth when there is no sound
G.729AB extends G729A with silence suppression, and is only compatible with G729B
DTMF digits and fax transmission cannot be transported reliably with G729.
3 / GSM - GSM codec, stateful. Is rarely used
4 / G.723 - G.723 codec, stateful. Is rarely used

[Link]/VoIP/VoipTroubleshootingBook/[Link] 9/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
Other codecs and DTMF digits are transmitted with dynamic payload types. The dynamic payload types are used in conjunction with SDP, where actual name of codec is specified
SDP - example #1

INVITE sip:123456@[Link];user=phone SIP/2.0 SIP request line: method = "INVITE" (make call)

Via: SIP/2.0/UDP [Link]:5061;rport;branch=z9hG4bK-3767867992-3776019369-1982645923-1415686538 various SIP headers

From: <sip:srct_tel_num@[Link]:5061;user=phone>;tag=910826072-3776019369-1982645923-1415686538

To: <sip:123456@[Link];user=phone>

Call-ID: 581a4a72a97b11e1a3c62c768aa96154@[Link]

CSeq: 1 INVITE

Contact: <sip:srct_tel_num@[Link]:5061;user=phone>

Content-Type: application/sdp SIP header indicating type of encapsulated content: SDP = Session Description Protocol (what kind of data is carried inside the SIP INVITE packet?)

Allow: ACK, BYE, CANCEL, INFO, INVITE, OPTIONS, REFER, REGISTER, UPDATE

Max-Forwards: 70

User-Agent: StarTrinity v.4.4.0-19a

Content-Length: 714 size of encapsulated content

Marker bit - could be is set to 1 with every start of new RTP stream fragment (for example when RTP stream is interrupted by DTMF digit).
The marker bit is used as a flag to re
initialize jitter buffer.

Task: open this file in wireshark, see RTP codec, packet time in samples, packet time in milliseconds, SSRC identifiers.

RTP streams in Wireshark

The Wireshark tool provides great functionality to analyse RTP streams in recorded VoIP traffic.
You can view RTP streams in Wireshark using menu Telephony - RTP - RTP Streams.

Task: open this file, see

How many VoIP calls captured in the file, what are dialed destination numbers, what are caller ID's
How many RTP streams captured in the file
What are source and destination IP addresses and UDP port numbers

Multiple RTP streams are selectable with a mouse (left button click).
Button "prepare filter" allows you to view only selected RTP streams in main window area.
It is useful when you hav
many VoIP calls recorded and need to debug only one call:

[Link]/VoIP/VoipTroubleshootingBook/[Link] 10/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
The filtered RTP packets could be analysed to check their payload type, timestamp, sequence field, the payload itself as hex bytes (see screenshot).

Tasks with same file: see

Is there any pattern in marker bit?

What is size of DTMF payload, size of G.711 payload?
How many RTP packets transmit every single DTMF digit? What is duration of DTMF digits (in milliseconds and samples)?
Which party sends DTMF: UAC or UAS?

[Link]/VoIP/VoipTroubleshootingBook/[Link] 11/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
The selected RTP stream(s) could be analysed (button "Analyse") to see some great details (see screenshot)

and listened via sound card:

If you need to link RTP streams to VoIP calls, it is possible with Wireshark's VoIP call flow diagram: menu item Telephony - VoIP calls - (select a call) - Flow sequence:

[Link]/VoIP/VoipTroubleshootingBook/[Link] 12/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book

Clicking on RTP stream in the call flow selects corresponding RTP packet inside main window area.

Task with same file: what DTMF sequence(s) are played within the recorded VoIP calls? All same or different?

todo: what is delta and skew? What are normal values? What if skew is too high: desync of RTP clock. in TDM circuits there was a special wire for this 8khz synchronization. and in voip - no
and it is a potential issue, not covered by RTP and SIP.

Encrypted RTP: SRTP (unfinished)

The content is not ready yet.

ZRTP

RTCP protocol (unfinished)

The content is not ready yet

Session Description Protocol (SDP) (unfinished)

See SDP example #1

Types of VoIP issues (unfinished)

Types of VoIP issues by symptom

Failed calls
Calls dropped after connection
no response to 200 OK - firewall on UAC side [sample sip trace]
200 OK sent to wrong destination. UAC behind NAT
failed re-invite or UPDATE (ping) request (done by session timer)
Audio quality issues
No audio
One-way audio
Bad audio quality
High round-trip delay

Types of VoIP issues by root cause

Firewall misconfiguration
Router or server may require reboot to apply new firewall settings
Bugs in VoIP software

VoIP quality indicators (unfinished)

Multiple-calls quality indicators (unfinished)

Indicators applicable for VoIP route, VoIP traffic, VoIP provider, VoIP originator.

Answer seizure ratio (ASR). Averaging interval

Average call duration (ACD). Averaging interval
Post-dial delay (PDD)

Single-call quality indicators: audio quality (unfinished)

[Link]/VoIP/VoipTroubleshootingBook/[Link] 13/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
RTP max delta
RTP delta
RTP jitter
RTP packet loss
RTP packet loss burst rate
RTP packet loss in jitter buffer caused by jitter
RTP packet loss in jitter buffer caused by RTP clock skew
G.107 MOS
PESQ MOS
Confidence score (0-100%)

VoIP troubleshooting methods (unfinished)

Understanding components of the VoIP system and interaction between them

Short-time experiments
Making test call or registering phone to the server and sniffing traffic with wireshark
Running VoIP readiness test - we have free demo VoIP readiness test with VoIP Status
Long-time experiments
Continuous active monitoring (with test calls) - we have a software product VoIP Status
Test audio quality by verifying audio sent from an IVR number via your VoIP route
Test internet connection with continuous web-based HTTP tests: continuous bandwidth test and continuous online latency test

VoIP software bugs (draft)

Bug is a symptom of abnormal, unexpected behaviour of the VoIP software. For example:

Rejected VoIP call in case when the call should not be rejected
Bad performance: high RTP jitter and high post-dial and answer delays
VoIP software crash causing dropping of all current calls: major error or memory leak. The crash could be permanent (when one has to restart the VoIP software manually)
o
temporary, when the software gets recovered by itself automatically
drop of current call after some duration
invalid audio: silence, one-way, invalid IVR
lost CDR records or corrupt CDR files
invalid billing calculation

.....reproducing the bug is the key.

Ways to debug the VoIP software (is also related to other software, in general):

Code review, making experiment in developer's mind

Reproduce the bug in debugger, within ide integrated development environment (IDE), put breakpoint, see values of variables and code flow path. Pros: easiest way to see wha
happens in the code, using the IDE.
Cons: is not possible when the bug happens only on customer's site, remotely, and remote debugging is not possible.
In many cases th
reproducing of customer's environment on developer side is possible, but complex.
.. run software in IDE but access customer's environment from developer's machine via internet
Reproduce the bug on customer's side, analyse logs
For regression bugs: trace at which version bug appears. go back in history of versions to see the difference

when the bug is reproduced, we have 2 code flows: working and non-working.
find difference - minimize the difference (iterations)
- Root causes of bugs:
reasons:
regression bug
change o
interface between modules ,
refactor of module
types. bug or issue?

Real VoIP troubleshooting cases

The real cases are reported with permission of our customers and readers of the book.

Case #1: Mitel 6873i SIP phone registration at 3CX IP PBX

The issue has been reported by Telin, cloud IP PBX service provider, reseller of 3CX:
We have new Mitel 6873 handsets (SIP phones) deployed at a client's location and there appears to be an issue with the phones connect back to a 3CX server at the same time when anothe
softphone connects to the same extension. It creates a contact name mismatch and mitel phone unregisters. We have been working with mitel for a couple weeks and they are pointing the
finger at 3CX and 3CX pointing finger back at them.
Troubleshooting steps:

Clarify the situation with Telin. What happens in terms of SIP protocol
At the same time both Mitel phone and 3CX softphone send REGISTER to 3CX server to same extension. Mitel phone sends via UDP. 3CX softphone sends via tunnel.
The Mitel phone receives 200 OK response to the REGISTER, this response contains multiple "Contact" headers.
The Mitel phone displays an error related to registration, and calls do not work. When 3CX softphone is not registered, everything is fine.
Response from Mitel:
Mitel phone analyses contact header which is received from server in response to register. It does not accept the syntax.

Incorrect:

Contact: <sip:003@[Link]:50195;transport=TCP;rinstance=1-xs7ou4dgsbv7tg5.ntvyho6kmku6kv-a;ob;inst="D7FCE2">;reg-id=1;+[Link]="<urn:uuid:00000000-0000-0000-0000
00008340d0fa>";expires=74

Correct:

Contact: <sip:003@[Link]:50195;transport=TCP>;rinstance=1-xs7ou4dgsbv7tg5.ntvyho6kmku6kv-a;ob;inst="D7FCE2";reg-id=1;+[Link]="<urn:uuid:00000000-0000-0000-0000
00008340d0fa>";expires=74

The problem seems to be in incorrect contact header in the 200 OK for the softphones. Below is one of the sipphone contact headers in NotWorking trace. In this contact header the > sig
is added in an incorrect place which takes some of the feature parameters like rinstance inside the sip: part. I have shown the incorrect and correct way (based on how Mitel phon
processes the contact header).
Clarify the situation ourselves by making the experiment one more time, sniffing working and non-working REGISTER cases.
Working case: one mitel phone registers and receives "200 OK" response to REGISTER with following contact header:

Contact: "Mitel Phone"<sip:701@[Link]:5060>;+[Link]="<urn:uuid:00000000-0000-1000-8000-00085D5C0866>";expires=120

Not working case: one 3CX softphone is registered, at the same time one more (mitel) phone send REGISTER with contact header

Contact: "Mitel Phone" <sip:701@[Link]:5060>;+[Link]="<urn:uuid:00000000-0000-1000-8000-00085D5C0866>";expires=0

and receives "200 OK" response to REGISTER with following 3 contact headers:

Contact: "Mitel Phone"<sip:701@[Link]:5060>;+[Link]="<urn:uuid:00000000-0000-1000-8000-00085D5C0866>";expires=120

Contact: "Mitel Phone"<sip:701@[Link]:5060;rinstance=1-hkg2btlqkdv6ydfaixt62uwrre1zz4sd;inst="b98a8bbf">;expires=62

Contact: <sip:701@[Link]:5488;rinstance=f27cf6aeccdab97f>;expires=55

We see that the response is not correct. It muct contain only 2 contact headers, not 3, since only 2 phones are registered to one extension
At this point we know that we have a multi-component system and an issue related to one or multiple components, or to combination of components. It is helpful to understan
which component causes the issue. So we make a new experiement with a new system containing different components: 3CX PBX, Mitel phone and Polycom phone: we replace 3C
softphone by Polycom phone to see what happens. This is what we observe:
In this combination it works. All phones registered successfully
Contact headers in "200 OK" response to REGISTER:

[Link]/VoIP/VoipTroubleshootingBook/[Link] 14/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book

Contact: "Mitel Phone"<sip:701@[Link]:5060>;+[Link]="<urn:uuid:00000000-0000-1000-8000-00085D5C0866>";expires=120

Contact: <sip:701@[Link]:5488;rinstance=f27cf6aeccdab97f>;expires=72

We see that the 3CX PBX server responded correctly with 2 contact headers
One more experiment with Polycom phone and Mitel phone. Polycom sends contact header
Contact: <sip:007@[Link]>;methods="INVITE, ACK, BYE, CANCEL, OPTIONS, INFO, MESSAGE, SUBSCRIBE, NOTIFY, PRACK, UPDATE, REFER"
and receives

Contact: <sip:007@[Link]:5060>;methods="INVITE, ACK, BYE, CANCEL, OPTIONS, INFO, MESSAGE, SUBSCRIBE, NOTIFY, PRACK, UPDATE, REFER";expires=120

Contact: <sip:007@[Link]:5488;rinstance=8d481b0693d20404>;expires=106

- this is also correct, 2 contact headers in response, and it works in this combination too.
Based on the experiments we make conclusion and suggest possible ways towards solution
3CX PBX is wrong in sending 3 Contact headers. Further action: submit bug report to 3CX team including SIP trace
Mitel phone could have a configurable setting whether to verify Contact headers in response to REGISTER or not. Further actions: try to downgrade to old firmware versio
which does not verify contact headers in response, or request a new version with the configurable setting

Submit my case

Chapters to be written in future

components of voip and telephone systems

jitter buffer. impact of JB: higher RTT. JB affects only RX RTP stream, not TX
codec: encoder, decoder
RTP clock. RTP clock skew. Synchronization in PSTN
PLC (for G.711)
SIP endpoints (User Agents)
pc phones
mobile dialers
hardware phones
VoIP-PSTN gateways: SIP-PRI, SIP-SIGTRAN-SS7, SIP-GSM, SIP-bluetooth
IP Network, other traffic running along with VoIP traffic. Congestions causing packet loss. Duplicate packets
IP routers, IP switches
Network Address Translation (NAT). port triggering. port forwarding
SIP going through NAT. Symmetric SIP behaviour on SIP servers
RTP going through NAT. Symmetric RTP behaviour on SIP servers
SIP ALG (application-level gateway)
DSCP, prioritization, VoIP QoS, IP SLA
firewalls
VPN and other tunnels
Synchronous Bandwidth Optimizers (SBOs)
Internet Service Providers. Static IP, dynamic IP. NAT, firewall
Hosting providers. Amazon, Azure, NAT, firewall
SIP servers
Routing-billing engine
Proxy servers
Softswitches
Session Border Controllers (SBCs)
Heterogenous SIP implementations
SIP attacks
Fraudulent VoIP traffic
IP PBX servers
voip route providers, sip trunks. direct routes. PDD, ASR, ACD. FAS, international bypass
DID providers. 1-800 lines. Packet loss and audio quality issues. Call center audio quality monitoring
Call generators and receivers. Looped generated traffic.
Public Switched Telephone Network (PSTN).
TDM. E1, PRI, SS7 trunks.
Routing. OPC, DPC.
Vulnerabilities and fraud scenarios
network impacts. jitter, packet loss. Burst rate, G.107 e-model. RTT. report via rtcp within 1 hop. non-visibility after sbc - need of full audio path testing.
cases of troubleshooting. 1 works in one case, doesnt work in another case. comparison method.
intermittent bugs.
software tools and services for VoIP tests.
active testing, passive monitoring - startrinity, sipp, gl
continuous monitoring, one-time tests - voipmonitor, startrinity
test automation. unit tests, unit test frameworks.
voip routes testing. [Link], chekmyroutes. cli delivery
penetration test tools
IVR audio verification / 2-way audio verification test tools
test cases by object of testing
ip network. nat. bandwidth, dscp
softswitch, pbx,cucm
gateway sip-pstn
call center
PSTN public switched telephone network
conference servers
ivr servers. ivr testing audio verification
voip routes. cli delivery. uptime. fas, overbilling. voip-OTT routes
test cases by symptom
always happens or intermittent issue?
ways to increase probability of intermittent issue. stress testing. complexity of test cases. difference between test case and real voip calls.
audio issues: chopped sound, one way audio, no audio. nat issues. jitter, loss, dscp flag
call failure:408, 503,other codes
server memory leak
server intermittent crashes
server sip and rtp stack software upgrade, stability test
test scenarios (scripts)
unit tests with IVR audio verification
regular dialing: same number, params from csv
monitoring, email notifications
SIP Security, SIP Trunking
list of RFC documents related to SIP. general overview for every RFC. RFC3261, RFC 5359. SIP REFER, blind transfer

[Link]/VoIP/VoipTroubleshootingBook/[Link] 15/16
9/30/21, 10:49 AM VoIP. How it works in detail. Troubleshooting, fraud cases. Theory and practice. Free e-book
VoIP fraud types
FAS: types: IVR, LDFAS, etc. generation, detection. 200OK-ACK delay
International bypass: OTT, GSM. Blending. CLI delivery testing. CDR profiling, filtering
Sending dialer traffic
Brute force attack
Man-in-the-middle attacks
Sending malformed SIP packets.
DOS attacks (REGISTER, INVITE). DDOS attacks.
UDP flooding: SIP, RTP. TCP SYN flood attack. Role of SBC's
Weak/default passwords - brute force attacks
IP spoofing
Spam over IP telephone
Toll fraud
Vishing - Phishing over VoIP
Missed calls with CLI = IPRN
Traffic to IPRN while route provider is not aware of IPRN ranges
Fraud tools
SIPVicious
[Link]

Copyright 2011-2021 [Link] | Blog

| Contact lead developer via LinkedIn | TeamViewer link

[Link]/VoIP/VoipTroubleshootingBook/[Link] 16/16