Analysis: A history of advanced evasion techniques

Hackers have tried to circumvent intrusion prevention systems (IPS) since the early days of its invention.

Until recently only a handful of well-researched evasion techniques were known. Most security systems are able to deal with these techniques effectively and can deflect them.

With the discovery of Advanced Evasion Techniques (AETs) in late 2010, however, it became apparent that IT security providers had underestimated these methods of attack.

Researchers did not begin to focus a lot of attention on evasions until the late 1990s.

In 1998, Timothy N. Newsham and Thomas H. Ptacek proposed different techniques for circumventing security systems in their research paper 'Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection'.

Evasion techniques use the way in which intrusion detection systems (IDS) and IPS systems work to trick their security mechanisms. Thus, an end host can receive and interpret data packets that the IDS erroneously misunderstands and therefore does not provide a warning.

Hackers can also exploit this requirement for launching an attack with evasions. In this case, important information that diverts the IDS's attention away from itself is hidden in the packets. This makes it possible to sneak entire data sessions in packets right past the IDS.

Using the packets, the attacker forces the IDS to view a different data stream than the end host. The security system therefore does not see the very information that would indicate an attack.

For example, a hacker sends an HTTP query in multiple data packets, which the IDS erroneously cannot interpret correctly. As a result, the attacks can virtually hide essential portions of the data stream from the IDS.

The TCP/IP protocol suite, which is used in most networks on the internet, plays a special role when it comes to attacks involving evasion techniques. The suite is based on the RFC 791 IP standard of 1981, and specifies that a system must have a conservative sending behaviour and a liberal receiving behaviour.

In other words, it may send only well-formed data packets, but must accept any data packet it can interpret. This means that even if data packets have different forms, the receiving host must still interpret all of them in the same way. Originally, this liberal approach was intended to ensure interoperability between systems.

However, the tolerant interpretation method smooths the way for evasion techniques to enter the network, as different operating systems and applications behave in different ways when receiving packets.

The target system may then detect or assume something entirely different than the security system. Also, the network itself may alter the traffic between the detection system and the host.

For example, if an IPS system cannot store enough data fragments before creating the signatures, or if it is unable to determine the possible re-sequencing, it will not have the original context of the data packet. As a result, the system will rewrite the data stream before passing it on to the target host.

What that means is that the IPS has a different understanding of the protocol state than the end host has. We refer to the change in context between the IPS and the target system as 'state de-synchronisation'. Evasions can therefore make malware look like normal, secure data packets, which only turn out to be attacks when they are interpreted by the end host.

IP fragmentation is a known evasion technique that was described by Newsham and Ptacek. According to the RFC 791 rule, IP fragmentation is intended to ensure the actual interoperability between systems and smooth handling of different network topologies (Postel, 1981).

With IP fragmentation evasions, on the other hand, the attacker disguises her malicious code using out-of-order data fragments, for instance. In another variant, it floods the IPS with a mass of fragments.

The authors caution: “An IDS that does not properly handle out-of-order fragments is vulnerable; an attacker can intentionally scramble her fragment streams to elude the IDS” (Newsham & Ptacek, 1998).

IPS devices check the entire data traffic and permit it to enter the network only if no threats were detected. If malware attempts to penetrate the network, these devices will either alert the administrator (in the case of an IDS) or drop the connection (in the case of an IPS).

Devices that inspect the data traffic use different techniques. The most common ones are protocol analysis and signature recognition. In doing so, the security system compares the data traffic with known patterns of attack by malware programs or evasions.

The number of known exploits and vulnerabilities is large and continues to grow steadily. However, the inspection functions of IPS products also continue to develop rapidly. When a new IP threat becomes known, administrators can usually implement appropriate detection methods in the devices within just a few days, sometimes even within hours.

However, an IPS vulnerability lies in the fact that the "received fragments must be stored until the stream of fragments can be reassembled into an entire IP datagram" (Newsham & Ptacek, 1998). IDS and IPS architectures therefore have to compensate for all the possible ways that the target system can potentially re-assemble the fragments, and the IPS itself has to cover all possibilities.

Other known evasion techniques that are based, for example, on IP and TCP options as well as TCP sequences exist in addition to IP fragmentation. Most security systems can also deal effectively with these variants. However, Advanced Evasion Techniques (AETs) are now presenting them with entirely new challenges.

They combine new methods of disguise with already known evasion techniques and so are able to circumvent virtually any network security solution. With 2180 possible combinations, security systems are unable to cope.

The second part of this series will be published next week and will demonstrate how AETs differ from known evasion techniques and which networks are especially at risk.

This article originally appeared at scmagazineuk.com