Deep Packet Inspection (DPI) - what is it?

Ruslan Rakhmetov, Security Vision

Network security continues to be a critical aspect of cybersecurity, along with data and account protection, on-prem and cloud infrastructure security, and cyber incident response. From exploiting web vulnerabilities to unauthorized remote management of compromised infrastructures, multiple security solutions identify and prevent remote cyberattacks. To ensure network security and control access to web resources, deep traffic analysis technology is also used - Deep Packet Inspection (DPI), which we will talk about today.

First, you should briefly tell the background of the emergence of the very concept of in-depth analysis of network traffic. The first firewalls (firewalls) appeared in the late 1980s - these were the simplest packet filters without monitoring the state of network connections (stateless firewalls), which considered each network connection separately and worked at the network (third) level of the OSI model: for example, they could prohibit incoming traffic from a specific IP address. Further, in the early 1990s, stateful firewalls appeared. They could already see the relationships between network connections and worked at the transport (fourth) level of the OSI model: for example, they could allow an incoming connection from a specific IP address only to a specific TCP port and only within an already established network session. Further in the 2000s, combined network security systems began to appear, such as UTM solutions (Unified Threat Management, unified cyber threat management), which combined stateful firewall, network intrusion detection and prevention tool (IDS/IPS), VPN gateway, antivirus, content filtering tool. They were replaced by NGFW (Next-Generation Firewalls, next-generation firewalls), which, in addition to UTM functionality, were able to control application network activity at the application (seventh) level of the OSI model, filter malicious URLs, integrate with sandboxes to check suspicious objects, check encrypted traffic (so-called SSL/TLS inspection), DDoS protection.

The idea of ​ ​ deep analysis of the operation of web applications, i.e. emphasis on controlling traffic of the application (seventh) layer of the OSI model, is the basis of DPI technology - Deep Packet Inspection, literally - "deep inspection of (network) packets." Initially, the concept of deep traffic analysis appeared along with the emergence of stateful firewalls, but at that time computing power did not allow such analysis to be carried out on the fly. However, by the mid-2000s, demand for such technology had formed, and hardware became more affordable. Now DPI technology is used to solve the following problems:

1) Traffic prioritization: critical business applications should receive network priority, and users should be provided with uninterrupted voice and video communication, while individual resource-intensive network applications (for example, peer-to-peer networks or streaming services) should not slow down the network. ISPs and companies use DPI on their networks to completely block or limit the speed of certain file-sharing protocols, such as BitTorrent.

2) Formation of special tariff offers for various groups of users: mobile Internet providers often offer their customers unlimited access to popular resources (social networks, instant messengers) at some tariffs. DPI is also used to detect and disable charging for this type of traffic.

3) Ad targeting: ISPs can enter into lucrative contracts with marketing agencies, giving them data about the interests and preferences of their users. Such analytics is easiest to carry out by studying Internet applications and web services used by the provider's clients using DPI technologies.

4) Cybersecurity and providing controlled Internet access in the workplace: DPI technology helps block access to malicious and phishing sites, restrict Internet connections of potentially dangerous or unwanted software (for example, BitTorrent or Dropbox), ensure employee productivity by blocking social networks and instant messengers in the corporate network, identify potential data leaks and abnormal network activity of hosts.

5) Ensuring compliance with legal requirements regarding access to Internet sites: in a number of countries (for example, in China or the UAE), the Internet is significantly limited by the requirements of local legislation, therefore, Internet providers in these countries are required to block access to various web resources, social networks, instant messengers, online games, streaming services, etc. - this is also implemented through the use of DPI technology. 

In Russian legislation, access to Internet resources is regulated, among other things, by the norms of the following Federal Laws:

   -   No. 149-FZ "On Information, Information Technologies and Information Protection" dated 27.07.2006;

   -   No. 139-FZ "On Amendments to the Federal Law" On the Protection of Children from Information Harmful to Their Health and Development "and Certain Legislative Acts of the Russian Federation of 28.07.2012 (Law on the Creation of a Unified Register of Prohibited Sites);

   -   No. 90-FZ "On Amendments to the Federal Law" On Communications "and the Federal Law" On Information, Information Technologies and Information Protection "of 01.05.2019 (Law on Sovereign Runet).

Solutions that implement DPI technology, as a rule, are high-performance software and hardware complexes that integrate with the main network equipment. The principle of operation of DPI devices is as follows:

1)   The DPI device is installed either in passive mode (through the SPAN port or optical splitter), or in active mode ("in line," between the BRAS router and the border router). Accordingly, in passive mode, the DPI system will only be able to detect an attempt to access the Internet resource after the fact and block it by sending a message to the client such as "HTTP 302" (with a redirect link to the blocking message) or the "TCP Reset" packet allegedly on behalf of the site requested by him. In active mode, prohibited traffic is detected and blocked on the fly.

2)   The DPI solution follows the headers of the received traffic sample: the source and destination IP addresses are determined, the used transport/session/representative/application layer protocols are identified, and some significant header fields are analyzed. Some web applications that do not use obfuscation (a method of hiding the presence in traffic) can be easily identified at this stage.

3)   A detailed analysis of the package at the application level is performed using one of the following techniques:

   -   Content search: a DPI solution accesses machine-readable data in a network packet and tries to find characteristic features (sequences of characters) in them using regular expressions;

   -   Signature analysis: by analogy with IDS/IPS systems, a DPI device can use signatures to identify known malicious or prohibited traffic - this is fast, but not always effective against new versions of various protocols and new malware samples;

   -   Heuristic analysis: based on data on previous connections and characteristics of various network protocols, information on the size and structure of the packet and its contents, the DPI device can make an assumption about the network activity of a particular web application, the malware or legitimacy of traffic;

   -   Behavioral and statistical analysis: based on the accumulated information about the characteristic patterns of traffic types and trends, it is concluded that the traffic belongs to a particular web application/service.

4)   If the traffic is encrypted (and this is found in the vast majority of cases), then without decryption, machine learning methods can be used for analysis, including decision trees, convolutional and recurrent neural networks.

5)   As a result of in-depth traffic analysis, the DPI decision makes a verdict: block or allow traffic, apply a speed limit, send a notification to the administrator or SIEM system about a possible violation or potential malicious activity. In addition, DPI devices can have a fault tolerance (secure unlocking) mechanism: if the web request cannot be analyzed, and the requested web resource has not been identified as malicious or Internet traffic has not been categorized as suspicious, then access is provided. However, in some companies, DPI solutions can be configured in the "block everything that is not explicitly allowed/could not be determined" mode.

In addition to the described process, the following features are characteristic of DPI systems:

1)   Access to prohibited and potentially malicious resources can be blocked based on the analysis of a DNS request - such a request is transmitted unencrypted, so the DPI device can intercept it and independently generate a response to it by issuing the IP address of the stub site, for example. DNS traffic analysis was widely used to identify malware - it was enough to create a list of malicious domains and identify attempts to DNS resolve their names from infected hosts. However, the spread of technologies for protecting DNS traffic (DNS-over-HTTPS, DoH/DNS-over-TLS, DoT/DNS-over-QUIC, DoQ/DNSSEC) provided not only increased user privacy, but also deprived information security specialists of the opportunity to check DNS traffic.

2)   Before the widespread introduction of the secure HTTPS protocol, DPI solutions had the ability to analyze open HTTP traffic and apply security policies (limit speed or block access). With the development of HTTPS encryption, DPI solutions began to focus either on the SNI field (Server Name Indication - the requested name of the Internet host), or on the properties of the TLS certificate, which is transmitted to the client by the server to establish a secure connection. Both the SNI field and the TLS server certificate contain in clear text the domain name to which the user is trying to access - and DPI can limit the speed of this network connection or block access to this resource. However, the described methods of deep traffic analysis are hindered by the systematic implementation of methods for ensuring the privacy of Internet users - in particular, ensuring the encryption of the SNI field by expanding eSNI (encrypted SNI, the encrypted name of the requested Internet host), using the TLS extension ECH (Encrypted Client Hello), as well as the gradual introduction of the TSI protocol LS version 1.3, in which the server's TLS certificate is already transmitted in encrypted form.

3)   In corporate infrastructures, users can be installed a digital certificate issued by an internal certification authority - it can be used to decrypt user HTTPS traffic through an authorized MitM inspection. However, the actively implemented Certificate Transparency technology allows you to identify such substitutions, which can make such a TLS inspection impossible.

4)   Downgrade techniques can also be used in corporate infrastructures to legitimately lower the    level of encryption of connections (for example, from TLS 1.3 to TLS 1.2) to ensure legitimate TLS inspection. "SSL/HTTPS stripping" techniques can also be used, which, however, will be ineffective when the HSTS policy is configured on the web server and when the user's Internet browser is in HTTPS-only mode.

5)   If no more advanced traffic control methods work, the DPI solution can restrict network traffic on the IP address and a specific port (i.e. work in stateful firewall mode), however, in this case, legitimate web resources that are on the same IP.

Modern commercial DPI solutions allow you to thoroughly analyze Internet traffic at all levels of the OSI model, recognize web traffic of up to 10 thousand different applications and protocols, allow you to add your own rules and signatures, conduct network analytics, and integrate with IDS/IPS, NTA, NDR, SIEM, SOAR class solutions. In addition, there are Open Source solutions for DPI analysis: the nDPI project allows you to implement deep traffic inspection and along the way get acquainted with the characteristic features of various network protocols that it allows you to determine.