Download darkhadoop - TutorialsPoint

DARKHADOOP
RISHABH SHARMA
CDAC-ACTS
Email: [email protected]
ABSTRACT
With the emergence of network globalization and advent of Internet being the major tool for
international information exchange and platform for the future. Security has always been the
most talked about topics. Network administrators have often tried their best by improving their
network security, however with rapid surface of new expolits, the best way of ensuring that the
system is secure is to take the entire system to a new level of complexity.To protect the critical
data we need to store it in a complex infrastructure named Dark Net.To process our big chuck of
log a efficient distributed approach is required which is being fullfilld by Hadoop.
INTRODUCTION
When system & network admin talk about plugging all the holes and securing there network of
denial-of-service attacks, one of the least talk about but one of the most effective network
security technique is sinkholing.
A sinkhole is defined as a method in which we redirect specific IP network traffic for different
security reason for analysis,diversion of attacks and detection of anamolus activities. However
for the network administrators, sinkholes are generally deployed to provide valuable intelligence
regarding the security threats there networks are facing.
Two kinds of sinkholes implementations:
1. DARKNETS
2. HONEYNETS
Ok what is a sinkhole again..??
Say a malicious hacker has a target of 192.168.2.113, as shown in the figure above. The target is
part of a network block 192.168.2.0/254. When the hacker unleashes his DDOS attack, he is not
only disrupting the business operations of the target organization, but also crippling the network
besides increasing their cost$ because of the increasing bandwidth utilization. If the organization
complains to the ISP, what they would then do is initiate a blackhole type sinkhole, by injecting
a specific route for the target. They make the next hop the discard interface on their edge router.
This is also known as null() or the “bit bucket”. What this effectively did was redirect the
offensive traffic towards the ISP’s sinkhole instead of allowing it to flow downstream to the
original target. Unfortunately, the downside is that the IP being attacked cannot communicate
with the internet until the sinkhole is removed. A workaround could be that the target gets a
different IP, but then you have to think about stuff like DNS TTL expiry and so on….
WHAT IS DARKNET :
A Darknet is a private file sharing network where users only connect to people they trust.
Typically such networks are small, often with fewer than 10 users each. It is a portion of routed,
allocated IP space in which no responsive services reside. So again, why the name Darknet
because there is nothing “lit up” inside these networks. So how are they used in the so called
entrapment? Well, no legitimate packets should ever enter a darknet. So if one does appear, it is
either because of misconfiguration or malware scanning for vulnerable devices. Thus, because of
this a darknet is a powerful security tool because a security administrator can spot scanning
without the need for complicated analysis gear and best of all, without any false positives. At the
same time, a darknet could be complemented with the inclusion of detectors, packet sniffers and
IDS systems.
DARKNET CAN BE USED AS A FUTURE IDS (INTRUSION DETECTION SYSTEM) &
IPS (INTRUSION PREVENTION SYSTEM).
WHAT IS HADOOP :
Apache Hadoop is an open-source software framework for storage and large-scale processing of
data-sets on clusters of commodity hardware. Hadoop is an Apache-toplevel project being built
and used by a global community of contributors and users. It is licensed under the Apache
license 2.0.
The Apache Hadoop framework is composed of the following modules:




Hadoop Common – contains libraries and utilities needed by other Hadoop modules.
Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on
commodity machines, providing very high aggregate bandwidth across the cluster.
Hadoop YARN – a resource-management platform responsible for managing compute
resources in clusters and using them for scheduling of users' applications.
Hadoop MapReduce – a programming model for large scale data processing.
MY PROJECT ON DARKHADOOP:
The project is on DARKHADOOP. Which is a combination of DARKNET & HADOOP.
We use the DARKNET concept to protect webserver from attackers.we use it as a IDS &
IPS.
We thinking that this project is useful in future to make such a complex security
INFRASTRUCTURE that help to protect the webserver.
EXPLAINATION OF DARKHADOOP:
In my project we take a scenario that daily thousands of attack done to the google server.so to
protect and to collect the attacker information we use darknet.
We take three machines.we use one machine as a router and in router we use sniffer script
that monitor all the packet travel from the router and in this script it is decide that the attack
is done or not.in secound machine we use three virtual machine which is darknet in my
project and third machine is for hadoop that use map reduce algorithm and send top ten
attacks to the administrator watsapp number.
Rough sketch of a project only difference is the google.com server is not at that place where
it is shown.it is in other machine.
Let say if attacker attack on google webserver.then all traffic goes from router where sniffer
script monitor all packets,and in sniffer script the different attacks is defined if any condition
true then the IP of attacker block by IP Tables for 1 day and send the IP and attack name to
the darknet.
Darknet is made by three virtual machines as shown in figure.lets assume that this three
machines in diffrent country or locations and between them the IP is circulated in 1 min gap
as shown in figure we takw a IP pool of 22,23,24,25 one extra ip. We say that IP & attack
name send to darknet.in darknet some data copied to one virtual machine some data to
other.there is no particular place no one know where data is saved.and we know last part of
hacker to remove footprints.let say attackes find the IP where log stored and let say attacker
also connect to that but because of 1 min time they disconnect and if they again scan of that
IP it can find that IP in different location.so it is complicated to remove footprints.we can
make it more complicated by adding more machines in darknet and port forwarding.
Now after data store in darknet it can send to hadoop for the map reducing because the data is
in large amount.we use hadoop to find top 10 attack who can attack more and that top ten list
is send to administrator we use RASPBEERY PI to send top ten attack to the administrator
watsap .
CONCLUSION
The main objective of the project is to apply the various technologies learnt during the course of
instruction. The application has been created to detect various attacks and sending log to hadoop
using darknet. It can be used for network protection of an organization, and further development
can also be done on this concept. So even though something useful was created in this project,
there is still potential to expand what we’ve done to capture more data and automate the task of
extracting the data into a user friendly format. There is also a lot of potential to taking another
step to fully integrate the data and to somehow automate the process of extracting information
about specific events.
THIS IS FUTURE SECURITY INFRASTRUCTURE USED TO PREVENT ATTACKS.
WE ONLY GET TEN DAYS TO MAKE THIS PROJECT .WE HAVE MORE IDEAS TO MAKE
IT MORE COMPLEX AND SECURE. IN MY PROJECT WE ARE FOUR MEMBERS (RISHABH
SHARMA , RAUSHAN RAJ , MOHIT VYAD,KRISHAN GOPAL).
WE GET A+ IN THIS PROJECT.IN THE CLASS OUR PROJECT IS ON THE TOP LIST.