Survey
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project
DGMA 2008 GRID ACCULTURATION Zaharin Yusoff (Prof. Dr.) President, Multimedia University Eureka Building, USM, Penang 21st October 2008 A University Grid Acculturation Where to begin? • Parallel computing • Supercomputing • ….. • Distributed computing • High-performance computing • …. • Grid computing • Utility computing • Cloud computing (inc. Virtualisation) • ……. Grid Acculturation – where to begin? http://linux.sys-con.com/node/587717.... By: Michael Seehan Jul. 25, 2008 10:15 AM We all know that the term “Cloud Computing” is relatively new to the Technology buzz. But just how new is it? For starters, I ran a quick comparison of “Cloud Computing,” “Grid Computing” and “Utility Computing”. The term Grid Computing has been around for a while (even before Google Trends tracking shows it). But as you can see from the graphic above, it is trending downwards. Utility Computing has pretty much remained below the radar in comparison. But, the newcomer Cloud Computing, which made its full entrance into this trend analysis around 2007 is rapidly gaining momentum. 2008 seems to be a pivotal time where it surpassed Grid Computing (and continues to grow). Grid Acculturation – where to begin? http://hothardware.com/News/Cloud_Computing_The_Future_Takes_Nebulous_Shape/ Monday, June 30, 2008 – by Dave Altavilla …. In the final analysis, there's no question that Cloud Computing, Grid Computing, Utility Computing or whatever else you'd like to call it, is definitely the wave of the future for many applications and usage models. Granted, the average power user or enthusiast will likely still have a powerful desktop or notebook system for many years to come…. http://blogs.wsj.com/biztech/2008/09/25/larry-ellisons-brilliant-anti-cloud-computing-rant/ September 25, 2008, 7:53 pm – Larry Ellison’s Brilliant Anti-Cloud Computing Rant “The interesting thing about cloud computing is that we’ve redefined cloud computing to include everything that we already do. I can’t think of anything that isn’t cloud computing with all of these announcements. The computer industry is the only industry that is more fashion-driven than women’s fashion. Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s insane. When is this idiocy going to stop? …” Table of Contents Introduction • Generalities (… naive…) • At the School of Computer Sciences, USM Some Attempts at National Initiatives • Centre for Computational Sciences • 8th Malaysia Plan – 20012005 • 9th Malaysia Plan – 20062010 Some Pertinent Questions • Some Input & Questions • Grid Acculturation Introduction Generalities Some Terminologies http://en.wikipedia.org/wiki/Grid_computing Parallel computing is a form of computation in which many instructions are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently (‘in parallel’). There are several different forms of parallel computing: bit-level parallelism, instructionlevel parallelism, data parallelism, and task parallelism ... ‘Distributed’ or ‘grid’ computing in general is a special type of parallel computing which relies on complete computers (with onboard CPU, storage, power supply, network interface, etc.) connected to a network (private, public or the Internet) by a conventional network interface, such as ethernet. This is in contrast to the traditional notion of a supercomputer, which has many processors connected by a local high-speed computerbus … What distinguishes grid computing from typical cluster computing systems is that grids tend to be more loosely coupled, heterogeneous, and geographically dispersed. Also, while a computing grid may be dedicated to a specialized application, it is often constructed with the aid of general purpose grid software libraries and middleware. Grid Computing – Historical Perspective The term Grid Computing originated in the early 1990s as a metaphor for making computer power as easy to access as an electric power grid in Ian Foster and Carl Kesselmans seminal work, "The Grid: Blueprint for a new computing infrastructure". In Malaysia, grid computing also evolved from parallel computing and distributed computing in the early nineties. Along with these came application domains that need high performance computing, such as computational sciences (e.g. crystallography) and bioinformatics … GRID COMPUTING Using the resources of many computers in a network at the same time, to solve a single problem (http://www.netnw.net.uk/Jargon_Explained/jargon.htm) …. is a form of distributed computing whereby a "super and virtual computer" is composed of a cluster of networked, loosely-coupled computers, acting in concert to perform very large tasks. http://en.wikipedia.org/wiki/Grid_computing The technology has been applied to computationally-intensive scientific, mathematical, and academic problems through volunteer computing, and it is used in commercial enterprises for such diverse applications as drug discovery, economic forecasting, seismic analysis, and back-office data processing in support of e-commerce and web-services. At the School of Computer Sciences, USM School of Computer Sciences, USM 1993: Parallel & Distributed Processing Research Group Parallel Computing (1992 1994) • Parallel constructs, • Parallelisation of sequential programs Distributed Computing (1993 1996) 1980s (UM, UKM, UTM, USM, …) Parallel computing Computational Science …… • Distributed databases • Distributed processing Grid Computing (1997 ..) • Parallel numerical algorithms for message-passing • • • • • • architectures Cost Effective High Performance PC Cluster with Virtual Shared Memory Meta-computing Environment for Computational Sciences e-Science Grid (back-end engine and grid infrastructure) Knowledge Grid Seeding Bioinformatics …. Grid Intrusion & Detection System - AS Resource Allocation - CHY, FH & GCS Resource Monitoring - CHY, FH & GCS Fault Tolerance (algorithm level & grid level - RA, CHY Parallel Iterative & direct solvers - RKS, NMA Fast Cryptography Protocols - AS Parallelization, Dependency, Aliases - RKS, GCS Distributed shared Memory (replication & consistency) - RKS & MAO Parallelising Access to Large Databases (matching, indexing R&D&(C) Workshop 28-30July 2003 and clustering) – NAR, RA, ZZ E-Science Grid Framework Applications e-Science Portal Replica Management Service User Replica Catalog File Transfer Job Manager Intrusion Detection Service Bill Scheduler Specific plug-in Mobile Agent Resources Account Manager Resource Usage Tracking Agent Directory Service 5/25/2017 R&D&(C) Workshop 28-30July 2003 14 Overall Architecture User Access Visualization Iterative Solver Agent e-Sciences Grid Portal (Dynamic Information Services) Processed Data Resource Allocation Resource Monitoring) Authorized Prediction Resource Monitor Event Publication Information Dispatch Agent Mobile Agents Facility Invitation/Correction Resources Directory Service Event Publication Information Platform Type, Operating System, CPU, Memory, Network, File System, Job Status … Knowledge Grid Current Scenario at USM Math Tsunami Modelling Group School of Computer Sciences, USM MBBS CLUSTER OTHER RESEARCH IN USM Digital Content at School of Arts Geographical Information System at School of Humanities stealth School of Computer Sciences, USM CEDEC Aurora CEDEC Some Attempts at National Initiatives Proposal for a National Centre for Computational Sciences National Centre for Computational Sciences 2000: …. Quality Hotel Inter-university & interdisciplinary Collaboration • About 80 researchers (UM, USM, UKM, UTM, …..) • Computer Science, Chemistry, Biology, … • …. Ministry of Science • KSU • Science Advisor • …. 8th Malaysia Plan (ICT Sector) – 20012005 RM8 – LAYERS DELIVERY SERVICES PROCESSORS KB SERVICES SERVICES PROCESSORS PROCESSORS KB KB INFRASTRUCTURE PROCESSORS KB RM8 – SERVICES (inc. Processors & KBs) ESTABLISHMENT OFSERVICES ENHANCEMENT OF MSC FLAGSHIP APPLICATIONS SCIENCE, ENGINEERING & TECHNOLOGY (SET) SERVICES SERVICE STANDARDS & PROTOCOLS SET SERVICES SMART E-BUSINESS • Architecture & E-HEALTHCARE EDUCATION Software Infrastructure • Business • Knowledge & Data • Wellness Maintenance • System Software Engineering Acquisition • Healthcare & Tools • Enablers for • Computer-Assisted Practitioner Portal • Applications E-Business Education & Training • Healthcare Enterprise • Sophisticated • Teaching-Learning Modelling Processors Materials • Socio• Business Process Economic • Networking of Educators Engineering Studies • Student-Educator Consultation • Personalised • HCI Tools for Information on • Personalised Lifetime Education Plan E-Business • Online Education and Open Learning Education 9th Malaysia Plan (Grid Computing) – 20062010 Grid Computing Domains APPLICATIONS SERVICE MANAGEMENT •Managing Users & Applications •Managing lower level technical components •Utility grid management modules •…. RESOURCE MONITORING •Detecting faults •Managing faults •…. GRID DATA WAREHOUSE •Grid Database •Data Replication •…. GRID SECURITY •Component security •Data security •…. •Compute-Intensive •Data Intensive •On-Demand •Collaborative •… RESOURCE ALLOCATION •Service Aggregation •Resource Aggregation •Scheduler (load balancing) •… TOOLS & ENABLERS •Tools for specific applications •Tools for grid construction •Data warehousing •Grid Algorithms •Mobile Agents & Software agents •Grid Protocols •… GRID INFOSTRUCTURE GRID INFRASTRUCTURE (incl. Networks) Grid Computing Architecture SET Services compute intensive (bioinformatics, pharmaceuticals, …) Middleware Engineering (e.g. aggregator) ….. Knowledge Grid (e.g. e-Science) National Grid Utilities (c.f. TNB, Jaring/TMNet ) Knowledge Dissemination Support Computational Resources …… Users Grid Resources Software Industry data intensive (financial, administrative, …) Human Data ☺Scientists ☺Engineers ☺Researchers Hardware e.g. High Performance Cluster and etc. Data Source Molecular Health Care Database Information etc Grid Computing Architecture Individual User Company Commercialisation or Service to Public A Cluster Centre is inevitable Satellite Center others Government Main Center for Grid Computing Research Cluster Satellite Center USM MIMOS Satellite Center Satellite Satellite Satellite Center Center UTM Center UKM UPM UM Grid Computing Projects Technology Development SERVICE MANAGEMENT •Tokens & Metering – 2008 •Negotiator (Agent, AI, etc.) – 2008 – 2009 •Search & Optimization Data Set – 2007 •Service Resource Discovery/Retrieval – 2007 – 2008 •Service Resource Management – 2005 - 2006 •Accounting/Billing/Service Level Agreement – 2005- 2006 •Generic Gateway (Portal) – 2006 – 2007 •Grid Human Computer Interface – 2008 – 2009 •Policy Service Management – 2008 – 2009 •Provisioning (license management, etc) – 2008 – 2009 GRID SECURITY •Intrusion & Prevention Detection – 2005 – 2006 •Fast Cryptography – 2006 - 2007 •Data Security – 2006 - 2007 •Identification & Authentication – 2005 -2006 •Authorisation & Policy – 2006 - 2007 GRID INFOSTRUCTURE RESOURCE ALLOCATION •Resource Aggregation – 2006 – 2007 •Service Aggregation – 2007 - 2008 •Scheduler, Meta-Scheduler, Load Balancer – 2006 - 2007 •Resource Reservation – 2007 - 2008 •Trader/Broker – 2007 - 2008 •National Data Centres – 2005 •Grid Database – 2006 - 2007 Data Replication – 2006 – 2007 •Grid Storage – 2006 - 2007 •Transaction Management – 2007 - 2008 •Dist. Backup & Recovery – 2005 - 2006 •Parallel Access to Databases – 2007 - 2008 •Content-Based Info Retrieval – 2007 - 2008 •Parallel Data Mining – 2008 - 2009 •Knowledge Engineering – 2008 - 2009 RESOURCE MONITORING TOOLS & ENABLERS •Fault Management/Tolerance – 2007 – 2008 •Discovery Protocol – 2008 - 2009 •Grid Monitoring Kit – 2006 - 2007 •Grid Sensor – 2005 – 2006 GRID INFRASTRUCTURE (incl. Networks) •High Speed Grid (MYREN, IPv6) – 2005 •Mobile/Wireless Grid – 2006 – 2007 •Distributed Shared Memory – 2006 – 2007 •Parallel Dependencies, Aliases – 2006 – 2007 •Parallel Iterative &Direct Solvers – 2005 – 2006 •Mobile Agents & Software agents – 2006 – 2007 •Grid Protocols – 2009 – 2010 •Interconnection of Clusters – 2006 – 2007 •Algorithm Analysis – 2006 - 2007 •Search Algorithms (Drug Design) – 2006 – 2007 •Grid S’ware Dev Lib (Numerical, Graphics) – 2006 - 2007 •Connectivity/Comms (Master/Slave, P2P) – 2005 - 2006 •Cluster Node Management – 2006 – 2007 •Grid Simulator – 2005 – 2006 Roadmap – Technology Development Generic Grid Portal 2010 2009 2008 2007 • ADVANCE DISTRIBUTED KNOWLEDGE BASE • COLLABORATIVE MULTIPLE INTELLIGENT SCHEDULER • SCHEDULER WITH MULTI-DIMENSION PREDICTION • AGGREGRATED MACHINE’S BEHAVIOR PREDICTOR • HIGH TRANSPARENCY DISTRIBUTED DATABASE • HETEROGENOUS DISTRIBUTED DATABASE • ADVANCE INTRUSION DETECTOR • FAST CRYPTOGRAPHY 2006 Middleware Engineering Toolkits • CRYPTOGRAPHY • INTRUSION DETECTOR • CLUSTER MACHINE USAGE CLASSIFIER • SINGLE MACHINE‘S BEHAVIOR PREDICTOR • SCHEDULER WITH SINGLE DIMENSION PREDICTION • COMPUTE POWER MARKET ii • MULTI-MACHINE USAGE CLASSIFIER • HETEROGENOUS MULTI ALGORITHM SCHEDULER National Grid Utility • NON-DOMAIN SPECIFIC INTELLIGENT NEGOTIATOR • OPTIMIZED SERVICE MATCHER WITH CONTENT AWARENESS • DOMAIN ADMINISTRATOR • Global Grid • KNOWLEDGE FILTER • SERVICE MATCHER • COMPUTATIONAL ECONOMY SCHEDULER • Bioinformatics Grid • SERVICE BROWSER 2008 • Financial Grid • POLICY ENFORCER • GRID VISUALISATOR • GRID SERVICE TEMPLATE • HIGH THROUGHPUT SCHEDULER 2009 • MULTI-PLATFORM ONTOLOGIES AND DESCRIPTOR • HETEROGENOUS MONITORING SYSTEM • HOMOGENEOUS MONITORING SYSTEM • National Grid • INTELLIGENT COORDINATOR AND COLLABORATOR • DOMAIN SPECIFIC MULTIISSUE INTELLIGENT NEGOTIATOR. • ONE STEP AHEAD PREDICTOR 2010 • Knowledge Grid • MULTIPLE STEP AHEAD PREDICTOR • MULTI-CRITERIA SCHEDULER • SINGLE MACHINE USAGE CLASSIFIER SET / Industry Applications 2007 • DATASET FILTER & OPTIMISER • TOKEN MANAGER • COMMUNICATION PROTOCOL • DOMAIN SPECIFIC NEGOTIATOR • Campus Grid 2006 Grid Computing Projects Applications Development • • • • • • • Life Science Grid – Bioinformatics – Biotechnology – Medical Grid (e.g. Virtual Anatomy, Virtual Surgery) – Pharmaceuticals (e.g. Genetically Modified Gamat / Tongkat Ali) – Agriculture Grid – Environment Computational Science Grid – Physics (e.g. Nuclear Applications) – Biology – Chemistry (e.g. Liquid Crystals, Molecular Dynamics) – Mathematics (e.g. Modeling) Computational Engineering Grid – Volumetric Rendering Social Science Grid – Culture, Heritage & Civilisation Grid Commercial Grid – Financial (e.g. Forecasting, Banking) – Multimedia – Oil & Gas Education – E-Learning – Language Disaster Mitigation TARGETS •Clustering •Campus Grid •National Grid •Global Grid •Grid Services Provider •National Grid Utility Grid Computing Projects Capacity Development Years Human Resources Development 2005 Formation of Core Teams - Management Team - Technical Team - Marketing Team - Content Team 2007 Developing Curriculum for Grid Computing 2006 Developing Programmes for Post Graduate 2006 Strategic distribution of projects (hands on) 2006 Training/Certification - Capacity Building of Users - Training of Trainers Years Policy & Governance Years 2005-2006 Central Governance Body 2005-2006 Formulation of Policies - Grid - Users - Application - Security - Operation - Database - Collaboration - IP - Funding - Metering/Billing 2006 Cluster Centre Years 2005 2005 2005 2006 Type of Programmes 2005 a) Programme to Inculcate Culture of Sharing & Collaboration b) Awareness Programme 2007 - seminar & Workshop (Twice a year) - Conference 2008 - website - Campaign/Roadshow 2007-2008 2005 2006 2006 2006 Marketing - Market Survey - Market Research - Market Creation Programme - International Benchmarking 2006 2006-2007 Competency Centre 2006 Awareness & Market Creation Industrial Participation - Technology - Industry End Users - Service Providers Advisory Panel 2007 International Collaboration 2006 2006 2007 2007 2007 2007 Priority Application (Inter Agenda) - BioInformatics - Agriculture - BioMedical - BioTechnology - Modelling - Culture & Heritage Towards National Utility Grid National Agenda Help Desk Inspectorate (License) Grid Service Providers 2008-2009 Application Service Providers 2006-2007 Grid of Grids 2007-2008 Computing Power Transmitting (metering) 2009 National Grid 2010 MyGrid 2010 International Grid Some Pertinent Questions Some Input & Questions Main Points Grid computing is much more than the deployment of hardware and software resulting in a higher performing network. It also includes a culture of sharing, of content as well as computational resources Another point to look at is whether or not we are asking the appropriate questions of the domain. The goals should not only be of the operational type (such as on efficiency and performance), but also of the functional type. Can there be: a universal grid operating system, a grid computing language, formal criteria for usability, and grid computing as a utility. Such questions (or goals) would not only lead to the corresponding R&D aspirations but will also open up discussions on very pertinent issues that need to be resolved before any implementation. Some Input (1/2) There are a number of R&D areas and questions asked in grid computing: 1) Traditionally, many researchers conduct investigation on resource management, namely on the issues of scalability, heterogeneity, efficiency, availability and transparency. For high availability and adaptability, IBM would term these as autonomic computing (for selfhealing or for auto-configuration). For transparency, the term cloud computing is used when viewing resources as services. 2) Grid is viewed as a body or brawn, while an agent is viewed as a brain. Researchers attempt to meet the brain with the brawn, and many are talking about multi-agent systems on the grid. Some Input (2/2) 3) Grid computing can be viewed as a super virtual computer, and researchers explore further on virtualisation techniques such as VMware, Virtual Organisation Management, etc. 4) In terms of application domains: Grid combines with pervasive computing to integrate sensor networks, mobile devices, etc. Many grid researchers collaborate with application domain experts to jointly develop grid applications such as data grid, computational grid, medical grid, e-science grid, eco-grid, rendering farm, financial grid and etc. SOME PERTINENT QUESTIONS (1/2) Some questions asked many years ago are still valid…: 1) There should be a universal grid 'Operating System' that makes the underlying infrastructure completely transparent – such an infrastructure should be heterogeneous in nature, in terms of hardware and OS, and independent of geographical and logical domains….. (c.f. Globus..)… 2) There should be a 'grid computing language' that rides on the said OS with all the appropriate data structures and programme constructs – such a language should be independent of the infrastructure beneath the OS, but its compiler/interpreter should be intelligent enough to take full advantage of the configurations available. SOME PERTINENT QUESTIONS (2/2) Some questions asked many years ago are still valid…: …. 3) There should be clear and formal measures/criteria to determine whether a given problem/application is best implemented in a grid environment or otherwise ... (c.f. coarse/fine grain size/ granularity, total cost of ownership, …. But why not something simpler? e.g. 3phase power..) 4) Grid computing should be made a public utility (as in electricity, water, etc.) -- and with this should come the means for provision of its services, metering and payment like any other public utility. Grid Acculturation Attempts at National Initiatives There have been many attempts at making grid computing a national initiative, where some failed while some succeeded to a certain extent but have arguably not met the original goals and intentions. Perhaps one of the reasons for this limited success is the lack of understanding of the different roles of the players within and those related to the domain …. Grid Acculturation Grid Acculturation – Acculturate Who? OVERALL • Speak the same language • Win-win-win situation Researchers (& Students) • • • • • Fundamentals (incl. abstraction, …) Synergy (related domains, critical mass, specialise, …) Incrementality (core, processors, .. , applications, …) Heterogeneity (multiple platforms, applications, …) Educate (others & themselves … security, support, …) Industry • • • • Less confusion (tone down the hype, …) Longer term perspectives (patience wins, ..) Fundamentals (e.g. platform dependence kills, …) Business model Decision Makers (Government) • No Big Bang theory (need to make informed decisions, ..) • Technology is not cheap (but no need for Father Christmas, ..) • We do not have to be technology consumers THANK YOU TERIMA KASIH MERCI ARIGATO GRAZZIE SHUKRIYA GRACIAS XIE-XIE NI SPASIBA KAMSIAH / MMKOI DANKE JABAI INAU MANGE TAK NGGO BUTE KABU NAN DHRI KOP KUN KAH Back-Up Slides Computer Networks APPLICATIONS SERVICE MANAGEMENT •Multimedia Conferencing •Distributed Systems (e.g. Digital Libraries) •… •Tokens & Charging •Negotiator •…. NETWORK MONITORING •Intelligent Network Monitoring •Fault Tolerance •Down-time Management •…. COMMUNICATION MANAGEMENT •Resource Aggregation •Services Aggregation •… NETWORK SECURITY •Intrusion Detection •Prevention Systems •Cryptography •…. TOOLS & ENABLERS •Network Operating Systems •Compression/Decompression •Streaming •.. COMMUNICATION PROTOCOLS •Wired Protocols (e.g. Fast Ethernet) •Wireless Protocols (e.g. Satellite) •Emerging Protocols (IPV6) •.. SECURITY SPECIFIC APPLICATION ORIENTED •Secure game-play •e-Voting •… CONFIDENTIALITY TRUST ABUSE Enterprise •Digital Signature •Public key infrastructure •…. • Enterprise level security • Agent-Server Security • Radius/Kerberos • Honeypot/Honeynet • Man-in-the-Middle (MIM) • Dos/DDoS • Virus/Worm, Spam • Drone Armies Applications • Biometrics • Smart Card • One time password • Database security • Web-based Application Security • SSL, SSH • Buffer Overflow • Format String • Client-side (XST,XSS) • SQL Injection • Phising •Authentication •Non-repudiation •Integrity •Tripwire • Cryptography (inc. encryption, braid) • steganography • Parallelising crypto operations • Video/Image security Data OS (incl. Drivers & Registeries, H/W Interfaces) •Network security •Mobile IPv6 security •Tunneling •…. • IPSec • VPN • Firewall • Intrusion Prevention • Trusted OS PROTECTION Physical Network ANALYSIS • Forensics • Enterprise Audit • Enterprise PenTest • Appl. Forensics • Appl. Audit • Appl. Pentest • Packet Spoofing • Cryptanalysis • Brute Force • ISN Predictions • Cache Poisoning • Data Forensics • Log/Alert Analysis • False Positive Reduction • Rootkit • Trojan Horse • OS Fingerprinting • Sniffing • Hijacking • Re-routing • OS Forensics • OS PenTest • Intrusion Detection BIOINFORMATICS Wet Lab Experimentation (DNA/Genome Sequencing) SEQUENCE ANALYSIS DNA / Genome String of Nucleic Acids (A,T,C,G) •Sequence search •Verification •Cleansing •‘Parsing •Classification •…. Amino Acids (V,S,W, .. – 20) LITERATURE SEARCH •Meaning-based •Literature Manager •…. Proteins / Peptides Junk DNA / UNKNOWN GENES (NEW !!) STRUCTURAL ANALYSIS 1&2D3D TRANSFn •…. •…. Protein/Peptide Database Virtual Experimentations Junk / New !! Database Virtual Experimentations Protein-Based Applications DNA-Based Applications Dissemination •Modelling •Visualisation •Matching •Comparisons •Simulation •Transformation/ Mutation •…. Input from MMU There is a lot of work on applications for the GRID -- medical, education, etc. Here in MMU, Dr. Ho Sin Ban and his ROs are looking into some of that. Nithiapidary is looking into increasing the efficiency of programs that have many small-scale jobs by grouping them together. Nathar Shah is beginning a study on how to make writing GRID programs less problematic by using Aspect-Oriented programming. There is also research at other universities into making GRID easier to set up -- making installers, security issues, how to promote participation and prevent cheating, etc. Sin Ban, Nithia, and Nathar, do you have anything to add beyond what I wrote above?