Download A Data Mining Perspective

Criminal Network Analysis and Visualization: A Data Mining
Perspective
(Forthcoming article accepted for publication in Communications of the ACM)
By Jennifer Xu and Hsinchun Chen
Criminal Network Analysis
After the tragic events of September 11, 2001, academics have been called on for possible
contributions in uncovering terrorist networks to enhance public safety and national security.
Both the public and the Pentagon have realized that knowledge of the structure of terrorist
networks and how those networks operate is one of the key factors in winning the so-called
“netwar”. In this new war against terrorists probably the most critical weapons that our
intelligence and law enforcement agencies should be armed with are reliable data and
sophisticated techniques that help discover useful knowledge from the data.
Study of terrorist networks falls into the larger category of criminal network analysis, which
is often applied to investigations of organized crimes (e.g., terrorism, narcotics trafficking, fraud,
gang-related crimes, armed robbery, etc.). Unlike other types of crimes such as homicide and sex
offenses that often are committed by single or a few offenders, organized crimes are carried out
by multiple, collaborating offenders. These offenders may form groups and teams and play
different roles. In a narcotics network, for instance, different groups may be responsible for
handling the drug supply, distribution, sales, smuggling, and money laundering. In each group,
there may be a leader who issues commands and provides steering mechanisms to the group, as
well as gatekeepers who ensure that information and drugs flow effectively to and from other
groups. Criminal network analysis therefore requires the ability to integrate information from
1
multiple crime incidents or even multiple sources and discover regular patterns about the
structure, organization, operation, and information flow in criminal networks.
To untangle and disrupt criminal networks, both reliable data and sophisticated techniques
are indispensable. However, intelligence and law enforcement agencies often are faced with the
dilemma of having too much data, which in effect makes too little value. On one hand, they have
large volumes of “raw data” collected from multiple sources: phone records, bank accounts and
transactions, vehicle sales and registration records, surveillance reports, etc. [9, 10]. On the other
hand, they lack sophisticated network analysis tools and techniques to utilize the data effectively
and efficiently.
Currently, criminal network analysis is primarily a manual process that
consumes much human time and efforts, thus has limited applicability to crime investigation.
The objective of this paper is to provide a data mining perspective for criminal network analysis.
In following sections, we discuss the challenges in data processing, review existing network
analysis and visualization tools, and recommend the Social Network Analysis (SNA) approaches.
Although SNA is not traditionally considered as a data mining technique, it is especially suitable
for mining large volumes of association data to discover hidden structural patterns in criminal
networks [9, 10]. We also report some data mining projects for criminal network analysis in the
COPLINK research, which is the NIJ- and NSF- funded research for management of law
enforcement knowledge [3].
Challenges in Data Processing
Like data mining applications in many other domains, mining law enforcement data involves
many challenges. First, incomplete, incorrect, or inconsistent data can create problems. Second,
the special characteristics of criminal networks cause difficulties that are not common in other
data mining applications.
2

Incompleteness [10]. Criminal networks are covert networks that operate in secrecy and
stealth [8]. Criminals may minimize interactions to avoid attracting police attention and
their interactions are hidden behind various illicit activities. Thus, data about criminals
and their interactions and associations are inevitably incomplete, causing missing nodes
and links in networks.

Incorrectness. Incorrect data regarding criminals’ identities, physical characteristics, and
addresses may result either from unintentional data entry errors or from intentional
deception by criminals. Many criminals lie about their identity information when caught
and investigated.

Inconsistency. Information about a criminal who has multiple police contacts may be
entered into law enforcement databases multiple times. These records are not necessarily
consistent. Multiple data records could make a single criminal to appear to be different
individuals. When seemingly different individuals are included in a network under study,
misleading information may result.
Problems specific to criminal network analysis lie in data transformation, fuzzy boundaries,
and network dynamics:

Data transformation. Network analysis requires that data be presented in a specific
format, in which network members are represented by nodes, and their associations or
interactions are represented by links. However, information about criminal associations
usually is not explicit in raw data. The task of extracting criminal associations from raw
data and transforming them to the required format can be fairly labor-intensive and timeconsuming.
3

Fuzzy boundaries [10]. Boundaries of criminal networks are likely to be ambiguous. It
can be quite difficult for an analyst to decide whom to include and whom to exclude from
a network under study.

Network dynamics [10]. Criminal networks are not static, but are subject to changes over
time. New data and even new methods of data collection may be required to capture the
dynamics of criminal networks.
Some techniques have been developed to address these problems. For example, to improve
data correctness and consistency, many heuristics are employed in the FinCEN system at the U.S.
Department of the Treasury to disambiguate and consolidate financial transactions into uniquely
identified individuals in the system [5]. Other approaches like the concept space method [3] can
transform crime incident data into a networked format [12].
Criminal Network Analysis and Visualization Tools
Klerks [7] categorized existing criminal network analysis approaches and tools into three
generations.
First generation: manual approach
Representative of the first generation is the Anacpapa Chart [6]. With this approach, an
analyst must first construct an association matrix by identifying criminal associations from raw
data. A link chart for visualization purposes can then be drawn based on the association matrix.
For example, to map the terrorist network containing the 19 hijackers in the September 11
attacks, Krebs [8] gathered data about the relationships among the hijackers from publicly
released information reported in several major newspapers. He then manually constructed an
association matrix to integrate these relations [8] and drew a network representation to analyze
the structural properties of the network (Figure 1).
4
Figure 1: The terrorist network containing the 19 hijackers on September 11, 2001 (Source: Business 2.0
Magazine, December 2001).
Although such a manual approach for criminal network analysis is helpful in crime
investigation, when data sets are very large it becomes an extremely ineffective and inefficient
method.
Second generation: graphic-based approach
Second-generation tools can automatically produce graphical representations of criminal
networks. Most existing network analysis tools belong to this generation. Among them Analyst’s
Notebook [7], Netmap [5], and Watson [1] are the most popular. For example, Analyst’s
5
Notebook can automatically generate a link chart based on relational data from a spreadsheet or
text file (Figure 2A).
Recently, two second-generation network analysis approaches have been developed by the
COPLINK research and its co-developer, the Knowledge Computing Corporation. The first
approach employs a hyperbolic tree metaphor to visualize crime relationships [3]. It is especially
helpful for visualizing a large amount of relationship data because it simultaneously handles both
focus and context (Figure 2B). The second approach uses a spring embedder algorithm [4] to
adjust positions of nodes automatically to prevent a network display from being too cluttered. In
such a view, icons represent different types of entities. A filtering function allows a user to select
only those entity types of interest. (Figure 2C-E).
Although second-generation tools are capable of using various methods to visualize criminal
networks, their sophistication level remains modest because they produce only graphical
representations of criminal networks without much analytical functionality. They still rely on
analysts to study the graphs with awareness to find structural properties of the network.
6
A
C
B
D
E
Figure 2. Second-generation criminal network analysis and visualization tools. (A) Analyst’s Notebook
(Source: i2, Inc.). The system can automatically arrange network nodes and allows one to drag nodes
around for easier interpretation. (B) The hyperbolic tree view of relations among multiple criminal
entities (Source: COPLINK). (C) The initial layout of a criminal network produced by the network view
(Source: Knowledge Computing Corporation). (D) The network layout is adjusted automatically. The
system moves the nodes having the largest number of links to the center of the display. A user can
actually see movements of the nodes while their positions are adjusted and can fix the display at any time
during the adjustment. (E) A user may choose only the entity type that is of interest (e.g., person) and
view textual explanations (e.g., person name, address, and relations).
7
Third generation: structural analysis approach
The third generation approach is expected to provide more advanced analytical functionality
to assist crime investigation. Sophisticated structural analysis tools are needed to go from merely
drawing networks to mining large volumes of data to discover useful knowledge about the
structure and organization of criminal networks.
Data Mining Perspective
Intelligence and law enforcement agencies often are interested in finding structural properties of
criminal networks [9]:

What subgroups exist in the network?

How do these subgroups interact with each other?

What is the overall structure of the network?

What are the roles (central/peripheral) network members play?
Clear understanding of these structural properties of a criminal network may help analysts
target critical network members for removal or surveillance, and locate network vulnerabilities
where disruptive actions can be effective. Appropriate network analysis techniques, therefore,
are needed to mine criminal networks and gain insight into these problems.
Social Network Analysis
Because Social Network Analysis (SNA) techniques are designed to discover patterns of
interaction between social actors in social networks [11], they are especially appropriate for
studying criminal networks [9, 10].
Specifically, SNA is capable of detecting subgroups, discovering their patterns of interaction,
identifying central individuals, and uncovering network organization and structure.
8
Subgroup detection
A criminal network often can be partitioned into subgroups consisting of individuals who
closely interact with each other. Given a network, traditional data mining techniques such as
cluster analysis may be employed to detect underlying groupings that are not otherwise apparent
in the data. Hierarchical clustering methods have been proposed to partition a network into
subgroups [11]. Cliques whose members are fully or almost fully connected can also be detected
based on clustering results.
Discovery of patterns of interaction
Patterns of interaction between subgroups can be discovered using an SNA approach called
blockmodeling [11]. This approach was originally designed to interpret and validate theories of
social structures. When used in criminal network analysis, it can reveal patterns of betweengroup interactions and associations and can help reveal the overall structure of criminal networks
under study. Given a partitioned network, blockmodel analysis determines the presence or
absence of an association between a pair of subgroups based on a link density measure. In a
network with undirected links, for example, the link density between two subgroups i and j can
be calculated by d ij 
mij
ni n j
, where mij is the actual number of links between subgroups i and j; ni
and nj represent the number of nodes within subgroups i and j, respectively. When the density of
the links between the two subgroups is greater than a predefined threshold value, a betweengroup association is present, indicating that the two subgroups interact with each other constantly
and thus have a strong association. By this means, blockmodeling summarizes individual
interaction details into interactions between groups so that the overall structure of the network
becomes more prominent.
9
Centrality
Centrality deals with the roles of individuals in a network. Several centrality measures, such
as degree, betweenness, and closeness can suggest the importance of a node in a network [11].
The degree of a particular node is its number of links; its betweenness is the number of geodesics
(shortest paths between any two nodes) passing through it; and its closeness is the sum of all the
geodesics between the particular node and every other node in the network. An individual’s
having a high degree, for instance, may imply his leadership; whereas an individual with a high
betweenness may be a gatekeeper in the network. Baker and Faulkner [2] employed these three
measures, especially degree, to find the central individuals in a price-fixing conspiracy network
in the electrical equipment industry. Krebs [8] found that in the network consisting of the 19
hijackers, Mohamed Atta scored the highest on degree and closeness, but not on betweenness.
Implications
Effective use of SNA techniques to mine criminal network data can have important
implications for crime investigations. For example, clustering with blockmodeling can help show
the hidden structure of a criminal network. The knowledge gained may aid law enforcement
agencies fighting crime proactively, e.g., allocating an appropriate amount of police effort to
prevent a crime’s taking place, or ensuring a police presence when the crime is carried out [9].
Sometimes, new structures discovered may even modify investigators’ conventional views of
certain crimes. For instance, Klerks [7] has found that the stereotype of organized crime’s
consisting of stable and hierarchical organizations is being replaced by an image of more fluid
and flattened networks. Traditional police strategies targeting leaders of a hierarchical criminal
organization, as represented by the Italian Mafia, may have become less effective in fighting
organized crimes today. The work by Krebs also demonstrates that the network consisting of the
10
19 hijackers in the September 11 attacks is fairly flat and dispersed [8]. The advantage of such a
structure is an increase in the network’s resilience and an emphasis on minimizing damage
should some network members be captured or compromised.
SNA may also help address the challenges of data processing. Blockmodeling, for example,
can easily detect “structural holes” [7] in which the link density is lower than a threshold density
value. According to McAndrew [9], structural holes may indicate incomplete or missing data
thereby drawing analysts’ attention to further data collection and improvement.
Data Mining Projects for Criminal Network Analysis and Visualization: The
COPLINK Research Testbed
Several data mining projects in the COPLINK research have begun to employ these SNA
techniques for criminal network analysis. The goal has been to provide law enforcement and
intelligence agencies with third-generation network analysis techniques that not only produce
graphical representations of criminal networks but also provide structural analysis functionality
to facilitate crime investigations. Prior to these data mining activities, several methods were
employed to address the challenges of data processing. For inconsistency and incorrectness
problems, we used the record linkage algorithm to relate multiple database records that actually
refer to a single individual. For data transformation, we used the concept space approach [3] to
extracting criminal associations from crime incident data.
The first stage of our network analysis development was intended to automatically identify
the strongest association paths, or geodesics, between two or more network members using
shortest-path algorithms. In practice, such a task often entails crime analysts to manually explore
links and try to find association paths that might be useful for generating investigative leads. In
the user study our domain expert evaluated the paths identified automatically and those identified
11
manually. He considered the former to be useful around 70% of time and the latter to be useful
around only 30% of time.
Extending this attempt, a more sophisticated system for mining criminal network data has
been developed. In addition to the visualization functionality, the system is intended to help
detect subgroups in a network, discover interaction patterns between groups, and identify central
members in a network.
Based on the crime incident data provided by the Tucson Police Department, several
networks consisting of criminals who were involved in different types of crimes have been
created and analyzed. Several domain experts validated the results of analysis (e.g., subgroups,
leaders, gatekeepers, etc.). Moreover, interesting patterns of interactions between criminal
groups and network structures were revealed in the networks. For example, a network of
criminals dealing with narcotic drugs and a network of gang members were found to have
different structures, which became evident after cluster and blockmodel analysis (Figure 3). It
appears that the chain-structure network (e.g., the narcotics network), for instance, may be
disrupted by removing any part of the chain. Removing the central members from a star-structure
network (e.g., the gang network) might cause fatal damage to it.
To evaluate the system’s performance we conducted a laboratory experiment involving 30
(student) subjects who performed 14 investigative tasks under two experimental conditions: (a)
structural analysis plus visualization (characteristics of third-generation tools), and (b)
visualization only (characteristics of second-generation tools). These 14 tasks were divided into
two types: identifying interaction patterns between subgroups and identifying central members
within a given subgroup. Our main performance metrics were effectiveness (defined as the total
number of correct answers a subject generated for a given type of tasks) and efficiency (defined
12
as the average time a subject spent to complete a given type of tasks). The results showed the
average time spent on interaction pattern identification tasks under condition (a) (7.13 seconds)
was significantly shorter than that under condition (b) (12.10 seconds; t = 6.92, p < 0.001). The
difference in efficiency was also significant for central member identification tasks (6.24 seconds
vs. 26.93 seconds; t = 10.66, p < 0.001). Such a gain in efficiency has important implications
because time is of the essence for law enforcement and intelligence agencies seeking to prevent
or respond to terrorist attacks or other serious crimes. No significant improvement in
effectiveness was present (t = 1.80, p > 0.05), probably due to the small sizes and relatively
simple structures of the testing networks [12].
Concluding Remarks
It is believed that reliable data and sophisticated analytical techniques are critical for law
enforcement and intelligence agencies to understand and possibly disrupt terrorist or criminal
networks. Using automated social network analysis and visualization techniques to reveal
various structures and interactions within a network is a promising step forward. The continued
advancement of criminal network analysis techniques will enable us finally to win the new
“netwar”.
13
Figure 3. The interaction patterns (and overall structures of networks) discovered from two criminal
networks. (A) The network consisted of 60 criminals dealing with narcotic drugs. It is difficult to detect
subgroups and interaction patterns from this original network manually. (B) A chain structure becomes
apparent using clustering and blockmodeling (see the red mark). Circles represent groups, which are
labeled by their leaders’ name, and straight lines represent between-group relationships. (C) The system
could also show the inner structure of a selected group, identify its central members (leaders by degree,
gatekeepers by betweenness, and outliers by closeness), and presents the centrality rankings of the
members in a table in a separate window. (D) The network consisting of 57 gang members. (E) The star
structure found in the gang network. (F) The details of a selected group in the gang network.
14
References
1. Anderson, T., Arbetter, L., Benawides, A., and Longmore-Etheridge, A. Security works.
Security Management, 38, No.17, 17-20 (1994).
2. Baker, W. E. and Faulkner, R. R. The social organization of conspiracy: Illegal networks in
the heavy electrical equipment industry. American Sociological Review, 58, No. 12, 837-860
(1993).
3. Chen, H., Zeng, D., Atabakhsh, H., Wyzga, W., and Schroeder, J. COPLINK: Managing law
enforcement data and knowledge, Communications of the ACM, 46(1), 28-34 (2003).
4. Eades, P. A heuristic for graph drawing. Congressus Numerantium, 42, 149-160 (1984).
5. Goldberg, H. G., and Senator, T. E. Restructuring databases for knowledge discovery by
consolidation and link formation. In Proceedings of 1998 AAAI Fall Symposium on Artificial
Intelligence and Link Analysis, AAAI Press (1998).
6. Harper,W. R., and Harris, D. H. The application of link analysis to police intelligence.
Human Factors, 17, No. 2, 157-164 (1975).
7. Klerks, P. The network paradigm applied to criminal organizations: theoretical nitpicking or
a relevant doctrine for investigators? Recent developments in the Netherlands. Connections,
24, No. 3, 53-65 (2001).
8. Krebs, V. E. Mapping networks of terrorist cells. Connections, 24, No. 3, 43-52 (2001).
9. McAndrew, D. The structural analysis of criminal networks. In D. Canter and L. Alison
(Eds.), The Social Psychology of Crime: Groups, Teams, and Networks, Offender Profiling
Series, III. Aldershot: Dartmouth (1999).
10. Sparrow, M. K. The application of network analysis to criminal intelligence: An assessment
of the prospects. Social Networks, 13, 251-274 (1991).
15
11. Wasserman, S., and Faust, K. Social Network Analysis: Methods and Applications
(Cambridge: Cambridge University Press, 1994).
12. Xu, J., and Chen, H. CrimeNet explorer: A framework for criminal network knowledge
discovery. Technical Report, January 2003.
Jennifer Xu ([email protected]) is a doctoral candidate in Management Information
Systems (MIS) at the University of Arizona, Tucson, AZ.
Hsinchun Chen ([email protected]) is McClelland Professor of Management
Information Systems and head of Artificial Intelligence Lab at the Management Information
Systems (MIS) Department at the University of Arizona, Tucson, AZ.
16