Download Detection of the security mechanism` violations and other

PROFESSIONAL SECURITY SYSTEMS
Detection of the security mechanism' violations
and other security breaches
by Mariusz Stawowski
email: [email protected]
The article was published in IT-FAQ (http://www.it-faq.pl)
Detecting and responding to incidents related to security breaches and in particularly
situations where security mechanisms are violated, are topics, which are often forgotten
when planning security systems. In reality there is no protection, which can protect your
system in one hundred percent. Therefore, we always have to take into account possibility
that an intruder breaks into our network or a worm infects our site. Breaking into our system
can be performed, for instance, using zero-day exploit (exploit which have not yet been
announced) or from inside the network, bypassing intrusion prevention systems (IPS). Once
an intruder installs a Trojan in our network, he or she has an easy access to protected
systems, and is very difficult to notice. A typical Trojan applications allow an intruder to
record a user's keystrokes (key loggers), capture screen contents, and to perform many
things aiming to irritate user, such as ejecting CD-ROM, displaying messages or playing
irritating sounds. Sample Trojan application named Luzak written by polish programmer is
shown in Figure 1.
Figure 1) Trojan allows an intruder to get an unauthorized access to data and computer
resources
Detection of a Trojan horse installed in an internal network is not easy. Such
applications disguise themselves in operating systems and for communication with an
intruder use common network protocols, (for example HTTP/HTTPS, SMTP). Trojans can
also cheat personal firewalls pretending to be trusted programs, such as web browsers. Very
common in Poland are Trojans communicating via Gadu-Gadu Instant Messaging system.
For instance, Trojan's G@du-Ghost activity in network looks like normal Gadu-Gadu chat.
Trojan G@du-Ghost logs in Gadu-Gadu server and therefore is available for an intruder in
the Internet. Commands to G@du-Ghost are sent through normal Gadu-Gadu client (for
example ‘keylog on’ command turns on recording user's keystrokes, ‘cftp’ command opens a
ftp connection from the users computer).
CLICO Ltd., Al. 3-go Maja 7, 30-063 Kraków; Tel: 12 6325166; 12 2927525; Fax: 12 6323698;
E-mail: [email protected], [email protected].; Ftp.clico.pl.; http://www.clico.pl
Detection of the security mechanism' violations and other security breaches
A basic technique used for detection of Trojans available in network IPS systems are
signatures (Figure 2). This method is effective for such common Trojans like SubSeven,
BackOrifice, NetBus or polish Prosiak and Konik. However in the Internet there are
thousands Trojans including many modifications of other Trojans. For example, before
mentioned Luzak is a polish clone of NetBus. Network communication of Luzak differs from
NetBus and therefore this Trojan can not be detected using NetBus signatures. For this the
other detection techniques are required, for instance methods based on heuristic analysis
and monitoring of modifications occurred in the network.
Figure 2) Signatures are basic method of detection Trojans. This method is usually available
in IPS systems
Vast majority of IPS and intrusion detection systems (IDS) available in the market
have little features allowing detection of security breaches and in particular situations where
protections were broken. Usually their functionality in this respect is limited to recognizing
known network attacks (exploit, DoS) and discrepancies in network communication when
compared to RFC standards. As of yet the progress has been made only in one protection
system — Juniper NetScreen-Intrusion Detection and Prevention (IDP). IDP system has
Security Profiler mechanism implemented, which constantly records and updates information
about computers and applications installed in the protected network. This is performed by
transparent network traffic analysis.
IDP sensors analyze network traffic and store user, computer and application specific
data related to security management (e.g. MAC and IP addresses, NetBIOS names, network
adapters types, types and versions of client and server applications, applications commands,
and so on). With Security Profiler a security administrator can quickly determine changes
which have occurred in the network during specific period of time (e.g. she or he can identify
new applications, open ports and new computers) and thanks to this detect Trojan
applications installed and unauthorized intruders' computers connected to private network,
and even employees breaking IT system use policy rules. The effects of connecting Luzak
Trojan application recorded by the IDP Security Profiler have been shown in figures 3 and 4.
© 2004 CLICO LTD . ALL RIGHTS RESERVED
2
Detection of the security mechanism' violations and other security breaches
Figure 3. Notification to administrator of detection a new open port on the server
Figure 4. Trojan detection based on analysis of changes in the protected network
© 2004 CLICO LTD . ALL RIGHTS RESERVED
3
Detection of the security mechanism' violations and other security breaches
The other method of detection connections with Trojans implemented in IDP systems
is a heuristic analysis of network communication. Aside from detection of Trojans using
signatures and network changes analysis (Security Profiler), in the IDP a dedicated
mechanism called Backdoor Detection has been implemented for this purpose. It works by
detecting interactive sessions, typical for connections with Trojan application. This method is
effective for situations where an intruder establishes connection with a Trojan horse installed
earlier, and this communication is monitored by IDP protections. This can happen when
Trojan has been installed in an internal network by user (whether intentionally or not) or
exploit attack, which has occurred using unknown security hole or from inside the network,
bypassing IPS protection (Figure 5).
Figure 5. Detection and blocking an interactive connection with Trojan
© 2004 CLICO LTD . ALL RIGHTS RESERVED
4
Detection of the security mechanism' violations and other security breaches
Backdoor Detection mechanism is effective for majority exploits available in the
Internet. These applications run shellcode on the server attacked, which an intruder uses to
get access to the operating system management console. Usually a connection with the
server is not encrypted. Only using this connection, an intruder can copy and install
dedicated, usually difficult to detect Trojan application (like Setiri). Sample exploit attack on
Windows 2000 SP4 server has been shown in Figure 5. The attack is an implementation of
the security hole detected by polish group called Last Stage of Delirium Research Group.
Then using Trojan an intruder establishes connection to the server from an external network.
After few seconds of intruders' work with Trojan, the IDP system blocks the session and
displays alarm of detecting unauthorized application on the server (Figure 6).
Figure 6. Detection of connection with Trojan using heuristic analysis of network traffic
Without dedicated tools a security administrator has practically very little means of
detection break-ins (i.e. attacks resulting in unauthorized control over the system) and
explaining security breaches. The analysis of events logged by IDP/IDS systems is difficult
because Internet worms generate many attacks and therefore well-thought-of intruders'
attacks can go unspotted. The analysis of several thousands events recorded in an IPS/IDS
log requires that a vendor of a security solution delivers relevant tools. The tools for viewing
and sorting logs can help very little for this purpose. On the other hand, generating reports
from historical logs usually takes long time, especially when an IPS/IDS system does not
have its own database and instead uses an SQL server.
In case of detection Trojan on a computer, a security administrator should have
possibility of performing a quick check, if the attacks have been performed from the server
captured and to determine resources which have been attacked (i.e. if an intruder attempted
to get control over other system and if she or he succeeded). Sample tool for supporting
administrators in detecting and explaining security breaches is Log Investigator, available in
© 2004 CLICO LTD . ALL RIGHTS RESERVED
5
Detection of the security mechanism' violations and other security breaches
before mentioned IDP security system. With this tool a security administrator can display on
the screen an interactive table with the list of intruders and systems attacked (axis Y and X).
In the Log Investigator's table fields there is a number of attacks detected (Figure 7). Data in
the table apply to the period of time indicated by the administrator (for instance last two
hours). With right mouse button click an administrator can read attacks performed by an
intruder, services which have been attacked and time of attacks as well as other related data
(such as protection system reaction). In order to further explain events, an administrator can
analyze sessions related to attacks logged (i.e. 20 packets before the attack an 10 packets
after). For this in the IDP system there are relevant tools available (Packet Viewer).
Administrator can also perform this using other tools (e.g. Ethereal).
Figure 7. Detecting and explaining security breaches based on correlated information from
IPS/IDS logs
Working out of effective tools for detecting and explaining security mechanisms
violations is a big challenge for security solutions vendors. Using for this purpose
conventional detection techniques like signature analysis and analysis of matching network
communication with the RFC standards is not enough. One hundred percent protections do
not exist and is not likely that they will ever be available. Therefore a security system should
be prepared for situations of security violations and allow for quick detection and explanation
of their results.
© 2004 CLICO LTD . ALL RIGHTS RESERVED
6