Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
732A54 Big Data Analytics 6hp http://www.ida.liu.se/~732A54 Teachers 2 Examiner: Patrick Lambrix (B:2B:474) Lectures: Patrick Lambrix, Christoph Kessler, Jose Pena, Valentina Ivanova, Johan Falkenjack Labs: Zlatan Dragisic, Valentina Ivanova Director of studies: Patrick Lambrix Course literature Articles (on web/handout) Lab descriptions (on web) 3 Data and Data Storage 4 Data and Data Storage Database / Data source One (of several) ways to store data in electronic format Used in everyday life: bank, hotel reservations, library search, shopping 5 Databases / Data sourcces Database management system (DBMS): a collection of programs to create and maintain a database Database system = database + DBMS 6 Databases / Data sources Information Queries Model Database system Database management system Processing of queries/updates Access to stored data Physical database 7 Answer What information is stored? 8 Model the information - Entity-Relationship model (ER) - Unified Modeling Language (UML) What information is stored? ER entities and attributes entity types key attributes relationships cardinality constraints 9 EER: sub-types 1 tgctacccgc gcccgggctt ctggggtgtt ccccaaccac ggcccagccc tgccacaccc 61 cccgcccccg gcctccgcag ctcggcatgg gcgcgggggt gctcgtcctg ggcgcctccg 121 agcccggtaa cctgtcgtcg gccgcaccgc tccccgacgg cgcggccacc gcggcgcggc 181 tgctggtgcc cgcgtcgccg cccgcctcgt tgctgcctcc cgccagcgaa agccccgagc 241 cgctgtctca gcagtggaca gcgggcatgg gtctgctgat ggcgctcatc gtgctgctca 301 tcgtggcggg caatgtgctg gtgatcgtgg ccatcgccaa gacgccgcgg ctgcagacgc 361 tcaccaacct cttcatcatg tccctggcca gcgccgacct ggtcatgggg ctgctggtgg 421 tgccgttcgg ggccaccatc gtggtgtggg gccgctggga gtacggctcc ttcttctgcg 481 agctgtggac ctcagtggac gtgctgtgcg tgacggccag catcgagacc ctgtgtgtca 541 ttgccctgga ccgctacctc gccatcacct cgcccttccg ctaccagagc ctgctgacgc 601 gcgcgcgggc gcggggcctc gtgtgcaccg tgtgggccat ctcggccctg gtgtccttcc 661 tgcccatcct catgcactgg tggcgggcgg agagcgacga ggcgcgccgc tgctacaacg 721 accccaagtg ctgcgacttc gtcaccaacc gggcctacgc catcgcctcg tccgtagtct 781 ccttctacgt gcccctgtgc atcatggcct tcgtgtacct gcgggtgttc cgcgaggccc 841 agaagcaggt gaagaagatc gacagctgcg agcgccgttt cctcggcggc ccagcgcggc 901 cgccctcgcc ctcgccctcg cccgtccccg cgcccgcgcc gccgcccgga cccccgcgcc 961 ccgccgccgc cgccgccacc gccccgctgg ccaacgggcg tgcgggtaag cggcggccct 1021 cgcgcctcgt ggccctacgc gagcagaagg cgctcaagac gctgggcatc atcatgggcg 1081 tcttcacgct ctgctggctg cccttcttcc tggccaacgt ggtgaaggcc ttccaccgcg 1141 agctggtgcc cgaccgcctc ttcgtcttct tcaactggct gggctacgcc aactcggcct 1201 tcaaccccat catctactgc cgcagccccg acttccgcaa ggccttccag ggactgctct 1261 gctgcgcgcg cagggctgcc cgccggcgcc acgcgaccca cggagaccgg ccgcgcgcct 1321 cgggctgtct ggcccggccc ggacccccgc catcgcccgg ggccgcctcg gacgacgacg 1381 acgacgatgt cgtcggggcc acgccgcccg cgcgcctgct ggagccctgg gccggctgca 1441 acggcggggc ggcggcggac agcgactcga gcctggacga gccgtgccgc cccggcttcg 1501 cctcggaatc caaggtgtag ggcccggcgc ggggcgcgga ctccgggcac ggcttcccag 1561 gggaacgagg agatctgtgt ttacttaaga ccgatagcag gtgaactcga agcccacaat 1621 cctcgtctga atcatccgag gcaaagagaa aagccacgga ccgttgcaca aaaaggaaag 1681 tttgggaagg gatgggagag tggcttgctg atgttccttg ttg 10 DEFINITION Homo sapiens adrenergic, beta-1-, receptor ACCESSION NM_000684 SOURCE ORGANISM human REFERENCE 1 AUTHORS Frielle, Collins, Daniel, Caron, Lefkowitz, Kobilka TITLE Cloning of the cDNA for the human beta 1-adrenergic receptor REFERENCE 2 AUTHORS Frielle, Kobilka, Lefkowitz, Caron TITLE Human beta 1- and beta 2-adrenergic receptors: structurally and functionally related receptors derived from distinct genes 11 Entity-relationship protein-id source PROTEIN accession definition m Reference n title article-id ARTICLE author 12 Databases / Data sources Information Queries Model Database system Database management system Processing of queries/updates Access to stored data Physical database 13 Answer How is the information stored? (high level) How is the information accessed? (user level) Text (IR) Semi-structured data Data models (DB) Rules + Facts (KB) 14 structure precision IR - formal characterization Information retrieval model: (D,Q,F,R) D is a set of document representations Q is a set of queries F is a framework for modeling document representations, queries and their relationships R associates a real number to documentquery-pairs (ranking) 15 IR - Boolean model adrenergic cloning receptor Doc1 yes yes no --> Doc2 no yes no --> (0 1 0) (1 1 0) Q1: cloning and (adrenergic or receptor) --> (1 1 0) or (1 1 1) or (0 1 1) Q2: cloning and not adrenergic --> (0 1 0) or (0 1 1) 16 Result: Doc1 Result: Doc2 IR - Vector model (simplified) Doc1 (1,1,0) cloning Doc2 (0,1,0) Q (1,1,1) adrenergic sim(d,q) = d . q |d| x |q| receptor 17 Semi-structured data NM_000684 ACCESSION Protein DB ”Homo sapiens adrenergic, beta-1-, receptor” human SOURCE DEFINITION PROTEIN REFERENCE REFERENCE AUTHOR TITLE AUTHOR AUTHOR ”Cloning of …” Frielle Collins AUTHOR AUTHOR Daniel AUTHOR AUTHOR Caron Lefkowitz 18 AUTHOR AUTHOR AUTHOR Kobilka TITLE ”Human beta-1 …” Semi-structured data - Queries select source from PROTEINDB.protein P where P.accession = ”NM_000684”; 19 Relational databases PROTEIN REFERENCE PROTEIN-ID 1 ACCESSION DEFINITION SOURCE PROTEIN-ID ARTICLE-ID NM_000684 Homo sapiens adrenergic, beta-1-, receptor human 1 1 1 2 ARTICLE-AUTHOR ARTICLE-ID 1 1 1 1 1 1 2 2 2 2 20 ARTICLE-TITLE AUTHOR Frielle Collins Daniel Caron Lefkowitz Kobilka Frielle Kobilka Lefkowitz Caron ARTICLE-ID TITLE 1 Cloning of the cDNA for the human beta 1-adrenergic receptor 2 Human beta 1- and beta 2adrenergic receptors: structurally and functionally related receptors derived from distinct genes Relational databases - SQL select source from protein where accession = NM_000684; PROTEIN PROTEIN-ID 1 21 ACCESSION DEFINITION SOURCE NM_000684 Homo sapiens adrenergic, beta-1-, receptor human Evolution of Database Technology 1960s: 1970s: Relational data model, relational DBMS implementation 1980s: Advanced data models (extended-relational, OO, deductive, etc.) Application-oriented DBMS (spatial, temporal, multimedia, etc.) 1990s: Data collection, database creation, IMS and network DBMS Data mining, data warehousing, multimedia databases, and Web databases 2000s Stream data management and mining Data mining and its applications Web technology (XML, data integration) and global information systems NoSQL databases 22 Knowledge bases (F) source(NM_000684, Human) (R) source(P?,Human) => source(P?,Mammal) (R) source(P?,Mammal) => source(P?,Vertebrate) Q: ?- source(NM_000684, Vertebrate) A: yes Q: ?- source(x?, Mammal) A: x? = NM_000684 23 Interested in more? 732A57 Database Technology (relational databases) TDDD43 Advanced data models and databases (IR, semi-structured data, DB, KB) 24 732A47 Text mining (includes IR) Analytics Analytics 26 Discovery, interpretation and communication of meaningful patterns in data Analytics - IBM What is happening? Descriptive Discovery and explanation Why did it happen? Diagnostic Reporting, analysis, content analytics What could happen? Predictive Predictive analytics and modeling What action should I take? Prescriptive Decision management What did I learn, what is best? Cognitive Analytics - Oracle Classification Regression Clustering Attribute importance Anomaly detection Feature extraction and creation Market basket analysis Why Analytics? The Explosive Growth of Data Data collection and data availability Automated data collection tools, database systems, Web, computerized society Major sources of abundant data Business: Web, e-commerce, transactions, stocks, … Science: Remote sensing, bioinformatics, scientific simulation, … Society and everyone: news, digital cameras, YouTube We are drowning in data, but starving for knowledge! 29 Ex.: Market Analysis and Management Where does the data come from?—Credit card transactions, loyalty cards, discount coupons, customer complaint calls, plus (public) lifestyle studies Target marketing Find clusters of “model” customers who share the same characteristics: interest, income level, spending habits, etc. Determine customer purchasing patterns over time Cross-market analysis—Find associations/co-relations between product sales, & predict based on such association Customer profiling—What types of customers buy what products (clustering or classification) Customer requirement analysis Identify the best products for different groups of customers Predict what factors will attract new customers Provision of summary information Multidimensional summary reports Statistical summary information (data central tendency and variation) 30 Ex.: Fraud Detection & Mining Unusual Patterns Approaches: Clustering & model construction for frauds, outlier analysis Applications: Health care, retail, credit card service, telecomm. Auto insurance: ring of collisions Money laundering: suspicious monetary transactions Medical insurance Professional patients, ring of doctors, and ring of references Unnecessary or correlated screening tests Telecommunications: phone-call fraud Phone call model: destination of the call, duration, time of day or week. Analyze patterns that deviate from an expected norm Anti-terrorism 31 Knowledge Discovery (KDD) Process Pattern evaluation and presentation Data Mining Task-relevant Data Data Warehouse Selection and transformation Data Cleaning Data Integration Databases 32 Data Mining: Classification Schemes General functionality Descriptive Predictive data mining data mining 33 Data Mining – what kinds of patterns? Concept/class description: Characterization: summarizing the data of the class under study in general terms E.g. Characteristics of customers spending more than 10000 sek per year Discrimination: comparing target class with other (contrasting) classes E.g. Compare the characteristics of products that had a sales increase to products that had a sales decrease last year 34 Data Mining – what kinds of patterns? Frequent patterns, association, correlations Frequent itemset Frequent sequential pattern Frequent structured pattern E.g. buy(X, “Diaper”) buy(X, “Beer”) [support=0.5%, confidence=75%] confidence: if X buys a diaper, then there is 75% chance that X buys beer support: of all transactions under consideration 0.5% showed that diaper and beer were bought together E.g. Age(X, ”20..29”) and income(X, ”20k..29k”) buys(X, ”cd-player”) [support=2%, confidence=60%] 35 Data Mining – what kinds of patterns? Classification and prediction Construct models (functions) that describe and distinguish classes or concepts for future prediction. The derived model is based on analyzing training data – data whose class labels are known. E.g., classify countries based on (climate), or classify cars based on (gas mileage) Predict some unknown or missing numerical values 36 Data Mining – what kinds of patterns? Cluster analysis Class label is unknown: Group data to form new classes, e.g., cluster customers to find target groups for marketing Maximizing intra-class similarity & minimizing interclass similarity Outlier analysis Outlier: Data object that does not comply with the general behavior of the data Noise or exception? Useful in fraud detection, rare events analysis Trend and evolution analysis Trend and deviation 37 Interested in more? 732A95 Introduction to machine learning 732A61 Data mining – clustering and association analysis 38 Big Data 39 Big Data 40 So large data that it becomes difficult to process it using a ’traditional’ system Big Data – 3Vs Volume 41 size of the data Volume - examples Facebook processes 500 TB per day Walmart handles 1 million customer transaction per hour Airbus generates 640 TB in one fligth (10 TB per 30 minutes) 72 hours of video uploaded to youtube every minute SMS, e-mail, internet, social media https://y2socialcomputing.files.wordpress.com/2012/06/ social-media-visual-last-blog-post-what-happens-in-an-internet-minute-infographic.jpg Big Data – 3Vs Volume size of the data Variety type and nature of the data 44 text, semi-structured data, databases, knowledge bases Linking Open Data cloud diagram 2014, by Max Schmachtenberg, Christian Bizer, Anja Jentzsch and Richard Cyganiak. http://lod-cloud.net/ Linked open data of US government Format (# Datasets) HTML (27005) XML (24077) PDF (19628) CSV (10058) JSON (8948) RDF (6153) JPG (5419) WMS (5019) Excel (3389) WFS (2781) http://catalog.data.gov/ Big Data – 3Vs Volume Variety type and nature of the data Velocity 47 size of the data speed of generation and processing of data Velocity - examples Traffic data Financial market Social networks http://www.ibmbigdatahub.com/infographic/four-vs-big-data Big Data – other Vs Variability Veracity 50 quality of the data Value inconsistency of the data … useful analysis results BDA system architecture Specialized services for domain A Specialized services for domain B Big Data Services Layer Knowledge Management Layer Data Storage and Management Layer BigMecs BDA system architecture Large amounts of data, distributed environment Unstructured and semi-structured data Not necessarily a schema Heterogeneous Streams Varying quality Data Storage and Management Layer Data Storage and management – this course Data storage: NoSQL databases OLTP vs OLAP Horizontal scalability Consistency, availability, partition tolerance Data management Hadoop Data management systems BDA system architecture Semantic technologies Integration Knowledge acquisition Knowledge Management Layer Knowledge management – this course Not a focus topic in this course For semantic and integration approaches see TDDD43 BDA system architecture Analytics services for Big Data Big Data Services Layer Big Data Services – this course Big data versions of analytics/data mining algorithms Databases Parallel programming Machine learning 2016: course for SDM year 1 and year 2 some review/introduction lectures 61 Course overview Review Databases (lectures + labs) Python (lectures + labs) Databases for Big Data (lectures + lab) Parallel algorithms for processing Big Data (lectures + lab + exercise session) Machine Learning for Big Data (lectures + lab) Visit to National Supercomputer Centre 62 Info Results reported in connection to exams Info about handing in labs on web; strong recommendation to hand in as soon as possible Sign up for labs via web (in pairs) 63 Info Relational database labs require special database account make sure you are registered for the course BDA labs require special access to NSC resources fill out forms 64 Info Lab deadlines: Final deadlines in connection to the exams; no reporting between exams HARD DEADLINE: March exam (No guarantee NSC resources available after April.) 65 Examination Written exam Labs 66 What if I already took …? What if I also take…? TDDD37/732A57 Database technology RDB labs 1-2 in one of the courses, results registered for both 732A47 Text mining Python labs in one of the courses, results registered for both Changes w.r.t. last year 68 New course My own interest and research Modeling of data Ontologies Ontology engineering Ontology alignment (Winner Anatomy track OAEI 2008 / Organizer OAEI tracks since 2013) Ontology debugging (Founder and organizer WoDOOM/CoDeS since 2012) Ontologies and databases for Big Data Former work: knowledge representation, data integration, knowledge-based information retrieval, object-centered databases 69 http://www.ida.liu.se/~patla00/research.shtml https://www.youtube.com/watch?v=LrNlZ7-SMPk 70