* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Where Government Science Policy Matters: Two National Initiatives
Survey
Document related concepts
Personal knowledge base wikipedia , lookup
Technological singularity wikipedia , lookup
Human-Computer Interaction Institute wikipedia , lookup
Ecological interface design wikipedia , lookup
Embodied cognitive science wikipedia , lookup
Philosophy of artificial intelligence wikipedia , lookup
Ethics of artificial intelligence wikipedia , lookup
Intelligence explosion wikipedia , lookup
Knowledge representation and reasoning wikipedia , lookup
History of artificial intelligence wikipedia , lookup
Existential risk from artificial general intelligence wikipedia , lookup
Transcript
Where Government Science Policy Matters: Two National Initiatives in Fundamental CS & AI John C. Mallery ([email protected]) Artificial Intelligence Laboratory Massachusetts Institute of Technology DANGEROUS IDEAS SEMINAR, March 19, 2003 (Revised April 8, 2003) © John C. Mallery, 2003. All rights reserved. Overview 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. Introduction: Two Research Initiatives Tom Kalil: How to Roll Your Own National Initiative in Science or Technology Ultrastable Computing Initiative Problem: Computer Science in Crisis Long-range Risks to Economic & National Security 21st Century Programming Languages Self-adaptive Software Semantic Integration Introspective Software Development Introspective Development Environment Introspective Computing Introspective Computing Architecture Rationalized Bootstrap Language Introspective Language Level M.I.T. Artificial Intelligence Laboratory 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 2 Natural Language Understanding Initiative Problem: NLU in Crisis Natural Language Research Context NLU Research Objectives Impact of Successes in NLU Substantial Recent Progress in Natural Language Processing Missing Ingredients: Semantics & Reference Prospects for Filling Gaps in Semantics & Reference Machine-parsable Knowledge Sources Q & A Based Testing & Evaluation Performance Metrics Initial NLU Program Structure Conclusions Appendix: How to Shape US Science Policy: Selections From a Talk at MIT AI 4/30/2017 John C. Mallery Introduction: Two Research Initiatives Reinventing computing as ultrastable computing (5-10 years) Self-adaptation & ultra-high productivity based on introspection Pre-competitive basic/applied research Tremendous economic impact Market forces reinforcing the COTS status quo Industry cannot do it now High risk & moderate time horizons Wide scope Orders of magnitude reduction in costs of software development, maintenance, & evolution Superior reliability, flexibility, security Breaking the natural language barrier (10-20 years) Acquiring useable knowledge from vast quantities of human text Pre-competitive basic research Small, fragmented research community Need to converge on cumulative paradigms Very high risk with long time horizons Tremendous economic impact Wide scope High-productivity human utilization of text bases Critical masses of machine useable knowledge M.I.T. Artificial Intelligence Laboratory 3 4/30/2017 John C. Mallery Tom Kalil: How to Roll Your Own National Initiative in Science or Technology Need a well-developed idea! Degree of Difficulty What’s the goal? Why is achieving this goal important? What are the mechanisms for reaching that goal? Why is government involvement required? Who else will help? How will success be measured? M.I.T. Artificial Intelligence Laboratory 4 Level at which decision can be made (new DARPA start or legislation) Support or opposition from interest groups, agencies, Congress, public Level of resources required ($, time, political capital) Complexity of implementation Precedence Level of partisanship Level of risk or uncertainty about outcomes Incremental or dramatic change 4/30/2017 John C. Mallery Ultrastable Computing Initiative What’s the goal? Government funding for research & development Market forces reinforcing COTS (1960s computing core) Large-scale research High development risk Complexity of implementation Precedence Level of risk or uncertainty about outcomes 5 bipartisan Research & development risks are moderate Wide-adoption risk is high Incremental or dramatic change M.I.T. Artificial Intelligence Laboratory Early computers Supercomputers (parallel) Level of partisanship Working systems will be disseminated for use by the research community Companies will begin selling to government & private sectors Moderate How will success be measured? $1 billion over 5-10 years Some political capital Research universities & other research institutions Some related industry efforts & spinouts Potential for international participation Support from research community, agencies Support from industry, if viewed as a precompetitive R&D subsidy Opposition from industry, if viewed as creating new subsidized competitors or exposing fraudulent or negligent practices Support from Congress & public if strategic importance of computer leadership understood Level of resources Who else will help? Why is government involvement required? Economy Science & Engineering Education National Security Congress because new budget required Support or opposition from interest groups, agencies, Congress, public What are the mechanisms for reaching that goal? Tremendous impacts Decision Level New hardware/software platforms Human centric peripherals Why is achieving this goal important? Making computers more cooperative Revolutionary change 4/30/2017 John C. Mallery Problem: Computer Science in Crisis Rise of COTS hardware/software has narrowed the gene pool 1960s language & OS concepts provide the playing field Market forces (scale, monopoly, inertia) reinforce narrowing hardware & OS choices Appearance of progress Massification of computers has pulled elite centers down towards the commercial mean Critical applications in government dumbed down to fit COTS Developers, researchers, & students pulled to COTS relevance Increasingly difficult for researchers & funders to conceptualize outside the COTS box Changing assumptions not reflected in dominant architectures Tremendous penetration of computers throughout society `New’ languages introduce 1970s concepts (dynamism, GC) to the masses in the 1990s Computation is now almost free Humans are now the extremely expensive factor Symbolic computing dominates numerical computing Evolution is the norm Security & reliability are critical Computation is now at the center of the economy & national defense Consequences Modern pyramids are being built inside the computer Software costs are extremely high (due to labor intensity) Agility & adaptation are relatively low (due to primitive design foundations) Security, brittleness, & reliability are inherent problems Economic & national security goals are at risk Market pressures reinforce lock-in & make transition to alternate architectures difficult M.I.T. Artificial Intelligence Laboratory 6 4/30/2017 John C. Mallery Long-range Risks to Economic & National Security Loss of software industries High-wage US economy cannot compete with Low-wage countries Unless Large productivity increases reduce the labor component Proximity to markets & head offices dominates National security restricts location of activity Scale economies offset R&D costs Anti-competitive market forces dominate M.I.T. Artificial Intelligence Laboratory 7 Fast time to market `Inside the opponent’s decision cycle’ New technologies become feasible due to faster computation, development, debugging More intensive utilization of labor Technologies unknown to competitors Scale economies offset R&D costs Faster computing Faster software development, certification, deployment, revision & evolution Lower costs of ownership due reliable operation & high-productivity user interfaces Superior organizational structures for software engineering & exploitation Speed Kills [Adversaries] Individuals, companies & countries More is achieved per time unit by: High labor costs mean US cannot produce software-intensive systems to match foreign competitors in ‘bang for buck’ Unless Time is a finite resource for Competitive disadvantage in defense systems Failure to utilize time efficiently Higher productivity research & education Productive resources freed for new economic activities Lower unit costs Wider product dissemination 4/30/2017 John C. Mallery 21st Century Programming Languages Design goals: Handle greater complexity Provide higher certainty Support faster development & smoother evolution Build in self-adaptation Deliver superior performance Key enablers: Design for understanding (human & computer) Semantic integration of code base Introspective descriptions Built-in problem solvers in support of self-adaptation M.I.T. Artificial Intelligence Laboratory 8 4/30/2017 John C. Mallery Self-adaptive software Automatic programming revisited? Simplifying assumption: sequencing operators rather than automatically programming operators Combinatorics & dataflow more manageable Analogous to quasi-custom mass production More at self-reconfiguring software Goals are explicitly represented Operators are explicitly annotated with meta-data: Means-ends analysis is used to maintain or achieve domain objectives Human programmers provide code, descriptions, and constraints Programming is indirect Function they perform Inputs, outputs, side effects Performance characteristics High-level assembly is flexibly controlled by goals and functional metadata descriptions But, programmers still stand behind the code Next step down the abstraction road The domain descriptive vocabulary is now annotated for (semi-) automatic assembly M.I.T. Artificial Intelligence Laboratory 9 4/30/2017 John C. Mallery Semantic Integration of the Code Base Self-adaptation requires knowledge representation to store goals, states, operator metadata, etc Once knowledge representation becomes a standard facility of the runtime & development environments, why not utilize it fully? Key idea: all knowledge about programs is stored in a uniform encoding Eliminates `transaction costs’ of accessing divergent data structures Enables full introspective access to the software-development & runtime environments Integration across the domain, the code base, the compiler, and the runtime environment enables further abstractive activities The development environment can further support the programmer, thus reducing cognitive load & increasing effective complexity The compiler can break abstractions and specialize functions based on domain and runtime knowledge M.I.T. Artificial Intelligence Laboratory 10 4/30/2017 John C. Mallery Introspective Software Development Design Rationales Basic Methodology Problem Solver Composes Operators to Realize Goals Humans Sequence Goals Humans Provide Search Control Hints Programmers Define New Operators Primitives Descriptions Objectives Runtime Adaptivity Monitor Program Execution Evaluate Performance Diagnose, Replan, Recompose Execute Replacement Program Raise Abstraction Level Increase Productivity Rapid Adaptation Encapsulated Computation Meta-Data Systematically Captured Machine AccessibleRepresentation Required for Programs to Work Basis for Adaptation to Changing Assumptions Enhance Flexibility Ultra Rapid Prototyping Fully Functional Procedural Description Dataflow Description Linkage of Procedural & Declarative Description M.I.T. Artificial Intelligence Laboratory 11 4/30/2017 John C. Mallery Introspective Development Environment Lisp Machine Recapitulation Design Rationale Capture Tracked Consequences Represented 12 Novice to Expert Intelligent Tutoring Based on Full Representation of Code Base Dependency Tracking for code & Data System Versioning Based on Code Assumptions M.I.T. Artificial Intelligence Laboratory Structured to Speed Humans to Relevant Documentation Multiple Models of Systems and Components Adaptation in Response Drives Problem-solving Semantic Operation on Code Goal-based Documentation Multimedia Assumptions Teaches All Aspects of the System Extended As Programs Are Written 4/30/2017 John C. Mallery Introspective Computing Rationalized language substrate Heterogeneous hardware Reconfigurable logic Symbolic computing Self-adaptive software Introspectively self-aware Goal-directed programming & adaptation Design rationale capture Foundational security Transactional & versioned memory Persistent Large-scale M.I.T. Artificial Intelligence Laboratory Functional justifications recorded Supports Hardware-grounded Expectation-based auditing Adaptive response to breaches Knowledge bases Parallel ready operations & data representations Simplified primitives & data flow Higher level abstraction for analysis & synthesis Ultra-rapid programming, testing, deployment & modification Self-adaptive software Smooth transition between software & hardware implementations Collaboration & organizational design 13 Group design & decision-making Information logistics Reactive planning 4/30/2017 John C. Mallery Introspective Computing Architecture Domain Applications Embedded Domain Languages Introspective Programming Language Multiple Representations Bootstrap Programming Language Operating System Hardware External Systems Introspective Operating System M.I.T. Artificial Intelligence Laboratory Introspective Peripherals 14 4/30/2017 John C. Mallery Rationalized Bootstrap Language Higher-level Abstractions Units for Quantities Futures Inter-process Communication With Built-in Constructs Networked Communication With Built-in Constructs Mobile Continuations Perceptual Loops Modularity Institutionalized Uniform Interface Protocol Interfaces Are Self-Descriptive 15 Event-based High-level Process Programming Constructs Built-in Introspection Security Program Analysis/Verification M.I.T. Artificial Intelligence Laboratory Computation Communication Power Consumption Process Orientation Data Flow First-Class Rationalized to Support: Test Vectors Associated With Operators, Data, Modules Automatic Self-Diagnosis Computing Costs First-class Sharing Explicit Transparent Networking Value Meta-Data Diagnostics First-class Parallel-ready Iteration Higher-Level Mathematics Resources First-Class Represented, Indexed Automatic Invocation 4/30/2017 John C. Mallery Introspective Language Level Problem-Solver/Planner Language Support Operator Description Framework Data Flow Description Framework Normative Constraints on Action Requirements Drive Action 16 Executable External Descriptions Goal-based Programming Bootstrapping Introspection Priority-based Resource Allocation Understand Programs Automatic Code Rationalization Performance Feedback Algorithms Library Building Expectations for Runtime Alerting When Expectations Fail Adaptive Repair M.I.T. Artificial Intelligence Laboratory Program Description Framework Active Self-monitoring Minimize Size & Complexity of Pre-introspective Cold Load Security Issues 4/30/2017 John C. Mallery Natural Language Understanding Initiative What’s the goal? NL systems will be developed that: Bipartisan Level of risk or uncertainty about outcomes Route documents more accurately Acquire useable knowledge from text Support education & research Translate semantically across languages DARPA funding for AI NASA `Man on the Moon’ Level of partisanship How will success be measured? High Precedence Research universities & other research institutions Some related industry efforts & spinouts Potential for international participation $4 billion over 10-20 years Considerable political capital Complexity of implementation High research risk Large-scale, long duration research effort Need to develop the research community Support from research community, agencies, industry Support from Congress & public if the tremendous payoff understood Level of resources Government funding for research & development Who else will help? Congress because new budget required Support or opposition from interest groups, agencies, Congress, public Economy Science & Engineering Education National Security Why is government involvement required? What are the mechanisms for reaching that goal? Incremental products towards the goal Building up the research infrastructure Tremendous impacts Decision Level Why is achieving this goal important? Making computers understand human language Research risks are high Incremental or dramatic change Revolutionary change Progressive research paradigms M.I.T. Artificial Intelligence Laboratory 17 4/30/2017 John C. Mallery Problem: NLU in Crisis Considerable progress was made in natural language understanding up to 1987, but After 1987 `AI winter,’ funded NL systems focused on: Speech (based on Markov processes) Extraction of information from text, too often using ad hoc methods Statistical techniques for parsing (learning from annotated corpora) Some domain-specific applications Result: But, systems erected too much ad hoc machinery to achieve `depth understanding’ on small texts Principled foundations for broad shallow understanding, let alone deep understanding, were not all there NL was too often constrained to interfacing to arbitrary databases, making semantic perception & generation intractable Very little work on acquiring semantic representations from text or `understanding’ those texts Many `schools of thought’ in computational linguistics with no clearly dominant paradigm, together with some atheoretic regression Yet, considerable progress in broad-coverage syntactic parsing based on large grammars coded by hand or induced from corpora Consequence: US poorly positioned to exploit (or generate) major advances in automatic creation of knowledge representation from unrestricted text based on broad coverage shallow understanding US risks technological surprise from well-funded European efforts M.I.T. Artificial Intelligence Laboratory 18 4/30/2017 John C. Mallery Natural Language Research Context Challenge of large-scale NL research in Europe $100 million Verbmobil Project in Germany Hiatus in NLU research in US post 1987 Pent up opportunities for significant advances Need to build foundation for large-scale NL systems DARPA BAA-02-21 (Cognitive systems) inspired hope Opportunity for basic research (once again) Relevant areas of interest Computational Perception Cognitive Architectures & Integrated Cognitive Agents Representation & Reasoning Learning Underlying Foundations But, the funding is Ramping up slowly Suitable for relatively small efforts M.I.T. Artificial Intelligence Laboratory 19 4/30/2017 John C. Mallery NLU Research Objectives Platform for Natural Language Understanding & Cognitive Simulation Full-cycle Language Understanding & Synthesis Modular, Extensible, Scalable Syntactic Parsing & Generation LF Construction & Base Structure Synthesis Referentially-integrated Surface Semantic Representation Deliberative Reference (deduction, induction, abduction, analogy, metaphor) Teleological Processing (Goal-directed text generation) Exploration Surface Semantic Representations Constructivist Perceptual Models Reversible Parsing & Generation Analogical & Metaphorical Reasoning Multi-lingual Understanding & Translation Cognitive Architectures for Linguistically-grounded Intelligence M.I.T. Artificial Intelligence Laboratory 20 4/30/2017 John C. Mallery Impact of Successes in NLU Semantically grounded information retrieval & document routing Constraint of grammatical relations Knowledge acquisition from text Bootstrapping machine parsable text Knowledge reuse (encoded as text) Enables cumulative research on ultra-large knowledge bases Increased precision Superior felicity of returns Correct level of generality Large-scale inference & Learning Debugging ultra complex computations Some applications: Knowledge-based retrieval, routing, and analysis of information Generation of user-specific reports & texts Superior organizational collaboration Support for education, science & technology development Large symbolic political models Value systems Organizational decision processes M.I.T. Artificial Intelligence Laboratory 21 4/30/2017 John C. Mallery Substantial Recent Progress in Natural Language Processing Large deep grammars have been developed (e.G. LKB, LFG) Wide coverage (approx. 40-60% for unrestricted syntax) Acceptable efficiency Declarative formalisms have been explored Linguistic communities with shared development of grammar resources Reversible parsing & generation on the horizon Lexical semantics, word-sense disambiguation Empirical methods applied to very large text sets Useful methods Part-of--speech disambiguation Lexical components Managing unknown words Heavy emphasis on English German, Japanese, French, Dutch, Chinese, Arabic, etc General approach is bottom-up Statistical approaches are increasing robustness Corpora-based acquisition of shallow grammar as aspects of learning parsers Target languages Focus on careful syntactic analysis Wide syntactic coverage & applicability But some good work on discourse analysis Semantic integration can improve coverage Interleaving of shallow & deep parsing is coming M.I.T. Artificial Intelligence Laboratory 22 4/30/2017 John C. Mallery Missing Ingredients: Semantics & Reference Bottom-up semantic representation Compatible with bottom-up syntax High scalability Characterization of the linguistic & referential model implemented by a system Platform for NL understanding & synthesis Categories Propositions Constructivist model of reference & semantic perception Semantic corpora for system training, calibration, and testing Bootstrap models for incrementally building up background knowledge necessary to understand texts in specific domains Metaphors & other tropes Belief & counter-factual contexts Inconsistencies Shallow understanding Efficient algorithms Parallel hardware implementation Large hardware capacity (memory & performance) Robustness under error & ambiguity Capable of handling vagaries of surface language Enables higher-level research Enables semantically-grounded applications Decompositional semantics discredited Discrimination network models of perception cognitively implausible (and impractical) Retrieval -- higher precision, felicity, and appropriate specificity Acquisition of correct knowledge from machine parsable text Generation of documents Effective decomposition of reference Selection of deliberation strategies in reference Supporting inference techniques M.I.T. Artificial Intelligence Laboratory 23 4/30/2017 John C. Mallery Prospects for Filling Gaps in Semantics & Reference MIT has been working on ternary relation representation technology for 30 years Winston Thesis Winston & Katz Analogy System Katz START system Mallery RELATUS system Bottom-up semantics matches bottom-up syntax well Surface semantic representation Easy mappings in and out of syntax New constructivist perceptual model Uniform model of inference & relational learning based on graph rewriting Deduction, induction, analogy, planning Other inferences New hardware architectures can help scaling up Graph representation of lexical items and grammatical relations Constraint-based model of reference Effective decomposition of reference\ Fast graph operations Ultra-large graphs Parallel computation New introspective programming languages & development environments can help cope with program & data complexity NLU systems are extremely complex & difficult to debug M.I.T. Artificial Intelligence Laboratory 24 4/30/2017 John C. Mallery Machine-Parsable Knowledge Sources New model of how human knowledge is transferred to computers: Knowledge is organized into: Machine-parsable texts Associated machine parsable background knowledge (Taxonomic knowledge and axioms) “Machine-parsable texts” refers to texts that adhere to a set of coding rules defining the syntactic, semantic, and inferential coverage of a natural language system Explicit characterization of the machine parsing model enables generation of texts conforming to a given model (by more capable systems) Natural language texts plus specialized mathematical formalisms NL as communications language between NL systems Ability to downgrade a text for a less-capable system Textual knowledge encodings can be reused because they are free of epistemic constraints and formalisms specific to individual systems Window of machine parsability will only expand over time Previously coded knowledge will remain accessible Large-scale investments in knowledge encoding become viable M.I.T. Artificial Intelligence Laboratory 25 4/30/2017 John C. Mallery Q & A Based Testing & Evaluation Correct operation & system coverage is determined by answering a series of questions based on associated background text Like reading comprehension tests except these are designed to test syntactic, semantic and inferential processing Comparison across systems is enabled because all inputs, questions, and answers are in natural language Q&A-based test vectors enable rapid detection of bugs introduced as systems are developed Q&A-based benchmarking can support measurement of progress in natural language understanding research (and thereby provide a basis for allocating funding) M.I.T. Artificial Intelligence Laboratory 26 4/30/2017 John C. Mallery Performance Metrics Problem: local maxima Switching to new promising approaches may produce performance drops before systems achieve higher capabilities Rigid performance testing can force researchers to adhere to the current `best’ approach, especially when lower performers are defunded Yet, no performance testing obscures progress Solution: Measure research progress by both performance metrics & introduction of new approaches Reward performance to achieve incremental progress Reward new approaches to achieve revolutionary progress Assumption of reasonable expectations for superior performance Critical analysis of approaches Explain successes & limitations Seek out bases for superior, subsuming approaches M.I.T. Artificial Intelligence Laboratory 27 4/30/2017 John C. Mallery Initial NLU Program Structure NLU theory & system development Constructivist model of semantic perception LKB/HPSG parsing & generation High-coverage statistical parsing (e.g. Collins Parser) Strongly theoretical parsing & generation (e.g., Chomskian) Meaning-postulates (equivalent surface structures) Word-sense disambiguation Discourse context Space/time context Document deixis Deliberative reference M.I.T. Artificial Intelligence Laboratory Intelligent tutoring Lexicon Development 28 Learning taxonomies from text Learning word-sense disambiguators Learning meaning postulates Evaluation One-step inferences in sentential Reference Framework for rapid development of knowledge bases derived from text Supporting infrastructure Language-derived `expert systems’ Background `common-sense’ knowledge Engineering design documentation Military doctrine Knowledge-based question answering Semantic analysis & referential integration Document routing Document retrieval Sentence-level indexing Acquisition of useable knowledge from machine parsable text Integrating several major existing parsing/generation efforts Intelligent information access Sharable knowledge representation substrate Constraint-based construction of LF Building scalable surface-semantic representations Applications Characterization of models linguistic coverage Development of test texts & associated questions 4/30/2017 John C. Mallery Conclusions This talk has described two national research initiatives in CS & AI Ultrastable computing initiative ($1 billion over 5-10 years) Self-adaptation & ultra-high productivity based on introspection Pre-competitive basic/applied research Market forces reinforcing the COTS status quo Industry cannot do it now High risk & moderate time horizons Tremendous economic impact Wide scope Orders of magnitude reduction in costs of software development, maintenance, & evolution Superior reliability, flexibility, security Natural language understanding initiative ($4 billion over 10-20 years) Acquiring useable knowledge from vast quantities of human text Pre-competitive basic research These two critical technologies of the future, but they will never come to into existence unless government begins working to make them a reality today The potential payoffs are so great that they clearly justify the relatively modest funding levels within the overall US budget Failure to pursue these initiatives in a timely fashion can result in a diminished US competitive position, both economically and militarily, as we enter further into the 21st century and other technically-sophisticated powers arise. Small, fragmented research community Need to converge on cumulative paradigms Very high risk with long time horizons Tremendous economic impact Wide scope High-productivity human utilization of text bases Critical masses of machine useable knowledge M.I.T. Artificial Intelligence Laboratory 29 4/30/2017 John C. Mallery Appendix: How to Shape US Science Policy: Selections from a Talk at MIT AI Thomas Kalil ([email protected]) University of California at Berkeley May 1, 2001 Shaping S&T policy The good news … Ideas matter Growing awareness of importance of S&T People from outside can have an impact Many public servants are hard-working, want to do the “right thing” S&T relatively non-partisan The bad news … Interest groups can trump “public interest” High-levels of cluelessness Lower incentives for politicians to worry about long term Growing imbalance in research funding (NIH vs. everything else) Complacency about U.S. competitiveness M.I.T. Artificial Intelligence Laboratory 31 4/30/2017 John C. Mallery Importance of agenda-setting Need a well-developed idea! What’s the goal? Why is achieving this goal important? What are the mechanisms for reaching that goal? Why is government involvement required? Who else will help? How will success be measured? M.I.T. Artificial Intelligence Laboratory 32 4/30/2017 John C. Mallery Degree of difficulty Level at which decision can be made (new DARPA start vs. legislation) Support or opposition from interest groups, agencies, Congress, public Level of resources required ($, time, political capital) Complexity of implementation Precedence Level of partisanship Level of risk or uncertainty about outcomes Incremental or dramatic change M.I.T. Artificial Intelligence Laboratory 33 4/30/2017 John C. Mallery How to get things done A compelling idea Authority and credibility of proponent(s) Breadth, power and “intensity of preference” of coalition Positive media coverage Well-funded “inside” and “outside”gov’t relations and PR strategy Let others take credit Third party validators -- not seen as self-serving Persistence Ability to communicate w. multiple audiences Willingness to compromise to diffuse opposition or broaden coalition Friends in high & low places M.I.T. Artificial Intelligence Laboratory 34 4/30/2017 John C. Mallery Existing mechanisms for input National level (PCAST, PITAC, NAS) Agency-specific (DARPA ISAT, NSF/CISE) IPAs University and scientific society gov’t relations Ad hoc M.I.T. Artificial Intelligence Laboratory 35 4/30/2017 John C. Mallery Kalil’s suggestions for S&T community Take public policy & communicating with public more seriously Develop mechanism for aggregation & synthesis of good ideas – include ideas from “lunatic fringe” Make agenda-setting part of workshop & conferences Willingness to engage in “demand pull” as well as “technology push” issues M.I.T. Artificial Intelligence Laboratory 36 4/30/2017 John C. Mallery