* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Proceedings of the Workshop “Formalizing Mechanisms for Artificial
Survey
Document related concepts
Neural modeling fields wikipedia , lookup
Agent-based model in biology wikipedia , lookup
Mathematical model wikipedia , lookup
Ethics of artificial intelligence wikipedia , lookup
Agent-based model wikipedia , lookup
Intelligence explosion wikipedia , lookup
Existential risk from artificial general intelligence wikipedia , lookup
Philosophy of artificial intelligence wikipedia , lookup
Ecological interface design wikipedia , lookup
Knowledge representation and reasoning wikipedia , lookup
History of artificial intelligence wikipedia , lookup
Transcript
Ahmed M. H. Abdel-Fattah & Kai-Uwe Kühnberger (eds.) Proceedings of the Workshop “Formalizing Mechanisms for Artificial General Intelligence and Cognition (Formal MAGiC)” PICS Publications of the Institute of Cognitive Science Volume 1-2013 ISSN: 1610-5389 Series title: PICS Publications of the Institute of Cognitive Science Volume: 1-2013 Place of publication: Osnabrück, Germany Date: July 2013 Editors: Kai-Uwe Kühnberger Peter König Sven Walter Cover design: Thorsten Hinrichs ! Institute of Cognitive Science Ahmed M. H. Abdel-Fattah Kai-Uwe Kühnberger (Eds.) Formalizing Mechanisms for Artificial General Intelligence and Cognition (Formal MAGiC) 1st International Workshop, FormalMAGiC @ AGI 2013, Beijing, China, July 31, 2013 Proceedings Volume Editors Ahmed M. H. Abdel-Fattah Institute of Cognitive Science University of Osnabrück Kai-Uwe Kühnberger Institute of Cognitive Science University of Osnabrück This volume contains the proceedings of the workshop “Formalizing Mechanisms for Artificial General Intelligence and Cognition (FormalMAGiC)” at AGI-2013. The final version of this volume will appear online in the “Publication Series of the Institute of Cognitive Science” (PICS, ISSN 1610-5389). PICS can be accessed at: http://ikw.uni-osnabrueck.de/de/ikw/pics 1st International Workshop, FormalMAGiC @ AGI 2013, Beijing, China, July 31, 2013 Program Committee Committee Co-Chairs ! Ahmed M. H. Abdel-Fattah, University of Osnabrück ! Kai-Uwe Kühnberger, University of Osnabrück Committee Members ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Joscha Bach, Humboldt University of Berlin Tarek R. Besold, University of Osnabrück Stefano Borgo, ISTC-CNR Selmer Bringsjord, RPI Ross Gayler, Senior R&D Consultant Melbourne Markus Guhe, University of Edinburgh Helmar Gust, University of Osnabrück Jerry R. Hobbs, USC/ISI Haythem O. Ismail, Cairo University Arne Jönsson, Linköping University Ulf Krumnack, University of Osnabrück Oliver Kutz, University of Bremen Mark Lee, University of Birmingham Maricarmen Martínez Baldares, University of the Andes Bogota Ekaterina Ovchinnikova, USC/ISI Ute Schmid, University of Bamberg Kristinn Thórisson, Reykjavik University & Icelandic Institute for Intelligent Machines Pei Wang, Temple University Philadelphia Table of Contents Keynotes: ! Zhongzhi Shi: “Computational Model of Memory in CAM” ! Stuart C. Shapiro: “Specifying Modalities in the MGLAIR Architecture” Accepted papers: ! Daniele Porello, Roberta Ferrario, Cinzia Giorgetta1: “Ontological modeling of emotion-based decisions” ! Naoya Arakawa: “Information Binding with Dynamic Associative Representations” ! Kristinn Thórisson: “Reductio ad Absurdum: Oversimplification in Computer Science and its Devastating Effect on Artificial Intelligence Research” ! Naveen Sundar Govindarajulu, Selmer Bringsjord, John Licato: “On Deep Computational Formalization of Natural Language” ! Ulf Krumnack, Ahmed Abdel-Fattah, and Kai-Uwe Kühnberger: “Formal Magic for Analogies” ! Pei Wang: “Formal Models in AGI Research” ! Xiaolong Wan: “On Special Theory of Relativity of Function – an Interpretation to `the Failure of Equivalent Substitution Principle’ ” Computational Model of Memory in CAM Zhongzhi Shi Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences [email protected] Abstract The goal of artificial general intelligence (AGI) is the development and demonstration of systems that exhibit the broad range of general intelligence found in humans. The key issue underlying the achievement of AGI is the effective modeling of the mind, which may be considered the focus of an interdisciplinary subject of “intelligence science“. First I outline an attempt at a comprehensive mind model, the Consciousness And Memory model (CAM). Then I talk about the computational model of memory in CAM, including working memory, semantic memory, episodic memory and procedural memory. All of these models are described by dynamic description logic (DDL), which is a formal logic with the capability for description and reasoning regarding dynamic application domains characterized by actions. The directions for further researches on CAM will be pointed out and discussed. Bio-Sketch: Zhongzhi Shi is a professor at the Institute of Computing Technology, Chinese Academy of Sciences, leading the Intelligence Science Laboratory. His research interests include intelligence science, cognitive science, machine learning, multi-agent systems, semantic Web and service computing. Professor Shi has published 14 monographs, 15 books and more than 450 research papers in journals and conferences. He has won a 2nd-Grade National Award at Science and Technology Progress of China in 2002, two 2nd-Grade Awards at Science and Technology Progress of the Chinese Academy of Sciences in 1998 and 2001, respectively. He is a fellow of CCF and CAAI, senior member of IEEE, member of AAAI and ACM, Chair for the WG 12.2 of IFIP. He serves as Editor in Chief of Series on Intelligence Science. Acknowledgement: This work is supported by National Basic Research Priorities Programme (No. 2013CB329502), National Science Foundation of China (No. 61035003 , 60933004), National High-tech R&D Program of China (No.2012AA011003). Specifying Modalities in the MGLAIR Architecture Jonathan P. Bona and Stuart C. Shapiro Department of Computer Science and Engineering University at Buffalo, The State University of New York {jpbona,shapiro}@buffalo.edu Abstract. The MGLAIR cognitive agent architecture includes a general model of modality and support for concurrent multimodal perception and action. An MGLAIR agent has as part of its implementation multiple modalities, each defined by a set of properties that govern its use and its integration with reasoning and acting. This paper presents MGLAIR’s model of modality and key mechanisms supporting their use as parts of computational cognitive agents. Keywords: multi-modality, cognitive architectures, agent architectures, embodied agents 1 Introduction The MGLAIR (Multimodal Grounded Layered Architecture with Integrated Reasoning) cognitive agent architecture extends the GLAIR architecture [1] to include a model of concurrent multimodal perception and action. Of central importance to MGLAIR is its treatment of afferent and efferent modalities as instantiable objects that are part of agent implementations. The architecture specifies how modalities are defined and managed, what properties they posses, and how their use is integrated with the rest of the system. Agents using these modalities deal independently with sense data and acts that correspond to distinct capabilities. MGLAIR is divided into three major layers, illustrated in Figure 1. The Knowledge Layer (KL) and its subsystems perform conscious reasoning, planning, and acting. A gradation of abstractions across the layers of the architecture terminates in symbolic knowledge at the KL. An MGLAIR agent becomes consciously aware of percepts when they are added to its KL. The SensoriActuator Layer (SAL) is embodiment-specific and includes low-level controls for the agent’s sensori-motor capabilities. The Perceptuo-Motor Layer (PML) connects the mind (KL) to the body (SAL), grounding conscious symbolic representations through perceptual structures. The PML is further stratified into sub-layers. The highest PML sub-layer, comprised of PMLa and PMLs, grounds KL symbols for actions and percepts in subconscious actions and perceptual structures respectively. The lowest PML sub-layer, the PMLc, directly abstracts 2 Specifying Modalities in the MGLAIR Architecture the sensors and effectors at the SAL into the basic behavioral repertoire of the robot body. The middle PML layer, the PMLb, handles translation and communication between the PMLa and the PMLc. The inward-pointing and outwardpointing arrows spanning the layers in Figure 1 represent afferent and afferent modalities respectively. Fig. 1. MGLAIR MGLAIR’s KL is implemented in SNePS, a logic-based Knowledge Representation and Reasoning system [2] [3]. SNeRE (the SNePS Rational Engine), the SNePS subsystem that handles planning and acting [4], is a key component of MGLAIR. SNeRE connects agents’ logic-based reasoning with acting. The plans formed and actions taken by an agent at any time depend in part on the agent’s beliefs (including beliefs about the world based on its perceptions) at that time. 2 Modality A modality corresponds to a single afferent or efferent capability of an agent: a limited resource capable of implementing only a limited number of related activities simultaneously. Each agent’s embodiment determines which modalities are available to it. An MGLAIR modality possesses a directional data channel that connects the mind with the body. The modality itself handles the transmission and transformation of data in the channel. Within an afferent modality raw sensory data originating at the SAL is passed up to the PML and converted to perceptual structures that are aligned with and used as the basis for conscious symbolic percepts at the KL. Any action an agent consciously performs is available as Specifying Modalities in the MGLAIR Architecture 3 an abstract symbolic representation in the knowledge layer, corresponding to low-level motor control commands in the SAL, which it is connected to through alignment with the intermediate PML structures. The flow of distinct types of sensory and motor impulses between the agent’s mind and body occur independently, each in its own modality. MGLAIR’s PML structures constitute unconscious multi-modal representations. Though they are inaccessible to the agent for conscious reflection, they play a crucial role in its cognition. 2.1 Modality Properties An MGLAIR modality specification is a 9-tuple of modality properties, which determine its behavior: � name, type, pred, chan, access, focus, conflict, desc, rels �: These are: a unique name for the modality; a type, (subtype of afferent or efferent); KL predicates used for percepts or acts in the modality; the modality’s data channel ; a flag granting/denying the agent conscious access to the modality; a specification for modality focus (see §3.5); a conflict handler for when multiple acts or multiple sensations try to use the modality simultaneously (see §3.2, §3.4); a description; and relations to other modalities. 3 3.1 Key Modality Structures and Mechanisms Perceptual Functions Each afferent modality has a function specific to it that is applied to perceptual structures in its PMLs to convert them into symbolic knowledge and assert them to the KL. The nature of the perceptual function depends on the type of perceptual structures it handles, and on the nature of the sensor and modality. For instance, a perceptual function for a visual modality might take as input a structure representing different visual features (shapes, colors, etc), and produce KL terms and propositions classifying and identifying objects in the field of vision. All perceptual functions take as input a timestamped PML structure from the modality’s perceptual buffer and produce as output KL terms that represent the percept using predicates associated with the modality. Each modality’s perceptual function also links the resulting terms to the modality in which they originated by embedding a reference to the modality within the SNePS data structure that represents each term. 3.2 Perceptual Buffers Modality buffers for each perceptual modality in the PMLs queue up perceptual structures to be processed and consciously perceived by the agent. Perceptual buffers may have a fixed capacity that limits the number of elements the buffer can hold. Otherwise, buffers with unlimited capacity must have an expiration interval, an amount of time after which data in the buffer expires and is discarded without being perceived by the agent. 4 Specifying Modalities in the MGLAIR Architecture Modality buffers serve to smooth out perception and minimize the loss of sensory data that the agent would otherwise fail to perceive because of temporary disparities between the speed with which the sensor generates data and the time it takes to process and perceive that data. By adjusting a buffer’s size or expiration interval, and by specifying how full buffers are handled, an agent designer may achieve a range of possible effects suitable for different types of agents and modalities. Timestamped sensory structures assembled at the PMLb are added to the modality buffer as they are created. If the buffer is full when a structure is ready to be added, for instance because the buffer has a fixed capacity and the expiration interval is too long, then an attempt to add a new structure to the buffer will be handled either by blocking and discarding the new data rather than adding it to the buffer - in which case it will never be perceived by the agent - or by making space for the new data by deleting, unperceived, the oldest unprocessed structure in the buffer even if it has not expired. Each modality buffer has a buffer management process that repeatedly removes the oldest non-expired structure from the buffer and applies the modality’s main perceptual function to it. These processes are affected by changes in the modality’s focus, discussed more in §3.5. 3.3 Act impulses in efferent modalities At the knowledge layer, SNeRE connects the agent’s reasoning and acting capabilities through the management of policies and plans. An example policy (stated in English from the agent’s perspective) is, “Whenever there is an obstacle close in front of me, I should move back, then turn, then resume operating.” An example plan is, “To pick up an object, I first open my grasping effector, then position it over the object, then I lower it, then I grasp with it, then I raise it.” Each act plan consists of a complex act, which is being defined by the act plan, and a sequence of other acts that comprise the complex act. These may be themselves complex acts or primitive acts. All such plans must eventually bottom out in primitive acts – acts that the agent cannot introspect on or further divide into their composite parts, but which it may simply perform. Each efferent modality contains within its PML an action buffer that stores act impulses resulting from conscious actions the agent has performed, but that have not yet been executed at the lower levels (i.e. they have not yet caused the relevant effectors to have their effects on the world). Plans for complex acts may be comprised of acts that use different modalities. For instance, depending on the agent’s embodiment, moving a grasper and using it may be independent of each other, and therefore use separate modalities. Since these separate modalities operate independently, actions may occur in them simultaneously. For instance, moving the grasper arm to a position and opening the grasper, if they use separate motors, could very well be performed at the same time without effecting each other’s operation. Specifying Modalities in the MGLAIR Architecture 3.4 5 Efferent Modality Buffers Like percept buffers, action buffers are located in the PMLb. Act impulses are added to the buffer as a result of primitive acts that are performed at the PMLa, and are removed and processed at the PMLc, where they are further decomposed into low-level commands suitable for use by the SAL. For instance, a primitive action for a grasping effector might be to move the effector some number of units in a particular direction. When the PML function attached to this action is called, it places a structure representing this impulse and its parameters into the modality’s action buffer. As soon as the modality is available (immediately, if it was not already carrying out some action when the move action was performed), the structure is removed from the buffer and processed at the PMLc, where it is converted into commands the SAL can execute (e.g. apply a certain amount of voltage to a particular motor for a duration). When an efferent modality’s act buffer is empty, an act impulse added to the buffer as a result of the agent’s consciously performing an act will be immediately removed and sent to the SAL for execution. When an action is performed using an efferent modality whose buffer is not empty, the default behavior is to add the act impulse to the buffer, from which it is be removed and executed when the modality is available. For example, a speech modality might be configured to buffer parts of utterances and realize them in order as the relevant resource becomes available. Another option is for the new impulse to clear the buffer of any existing act impulses. For instance, an agent with a locomotive modality might be configured to discard outstanding impulses and navigate to the location it has most recently selected. The manner in which such conflicts are resolved is specified as part of the modality specification. Action buffers in efferent modalities have similar properties to the percept buffers in afferent modalities: their capacities are configurable as are their expiration intervals. One difference is that MGLAIR’s model of modality focus currently applies only to perception and does not account for focusing on actions in efferent modalities. 3.5 Regulating Modality Focus MGLAIR’s model of modality focus allows agents to selectively dedicate more or less of their resources to the task of perceiving within a particular afferent modality. An agent designer may specify a default focus level for a modality and either permit or forbid the agent to adjust its focus levels on its own modalities – including the possibility of ignoring a modality altogether. Altering a modality’s focus increases or decreases the frequency with which its internal processes run. This allows an agent to prioritize processing of percepts from a particular modality relative to the others depending on its current task and priorities. Because each agent has only a limited amount of computational resources it is possible for a modality to interfere with the operation of others by monopolizing those resources, even though each modality is operating independently with its own processes and structures. This happens when a modality has to work harder 6 Specifying Modalities in the MGLAIR Architecture than others to convert its sensory data into perceptual knowledge, or when one modality is producing much more, or much richer, sensory data than others. Without a mechanism to selectively allocate its focus to the modality that is more relevant to its situation and task, agents would expend most of their processing power on more demanding modalities, even when they are less urgent or less relevant to the agent’s task than some less demanding modalities. Altering a modality’s focus adjusts the priority of the modality’s buffer management and perceptual function processes. For an agent with just two modalities with nearly all of the same properties connected to two identical sensors in the same environment, but with different levels of focus, the modality with the lower focus will remove and process PML structures from its buffer less frequently, and will therefore be more likely to reach its capacity or to have elements reach their expiration times before they are processed and perceived. That is, an agent that is not focused on some modality may fail to perceive some things that the modality is capable of sensing. 4 Example Modality As an example, consider an agent embodied in a small robot with - among other sensors - a unidirectional ultrasonic range finder that the agent uses to avoid bumping into obstacles as it moves around its environment. The agent may be equipped with other sensors and effectors as well – a camera based visual modality, motorized wheels, a grasping effector. To make use of this sensor, the agent must have a way of representing the information that the sensor conveys and of becoming aware of (perceiving) it. It becomes aware of percepts via an afferent modality that connects the sensor to the agent’s mind, with mechanisms to convert raw sensory data into the agent’s perceptual representation. The range finder operates by sending out sound waves and then detecting how long it takes for the echo to bounce off of the nearest object and return to the sensor. At the lowest level - the SAL - the data produced by this sensor is in the form of voltage levels generated by a piezoelectric receiver. These values are converted into an integer value between 0 and 255, which corresponds to the distance to the nearest object in front of the sensor. To use this sense to avoid banging into walls and other obstacles, it may suffice for the agent to perceive that the next thing in front of it is very close, very far, or in-between, rather than a precise value. This is achieved by having the PML convert values like 34, 148, etc, into structures at the granularity we want the agent to perceive: close, medium, and so on. These are removed from the modality’s sensory buffer and processed by the modality’s perceptual function, which produces symbolic representations that are then asserted at the KL, e.g. the term DistanceIs(far), which represents the proposition that the distance to the nearest object in front of the agent is far. These percepts (beliefs) affect the agents behavior if it holds policies like “when the nearest thing in front of me is very close, stop any forward motion and turn before resuming it.” Figure 2 shows the layers and examples of information present at each. Specifying Modalities in the MGLAIR Architecture Fig. 2. View of a single afferent modality for an ultrasonic sensor 5 7 1 Conclusions The MGLAIR architecture provides a model of afferent and efferent modalities for computational cognitive agents. MGLAIR agents that instantiate these modality objects can use them to sense and act - and make inferences and plans about percepts and actions - concurrently in different modalities. By dividing agents’ capabilities into modular modalities, each with its own properties governing its use, MGLAIR allows agents to sense and act simultaneously using different resources with minimal interference, and to consciously decide which resources to focus on for particular tasks. The specific properties that determine how each modality functions, are defined as part of a modality specification that is shared among the layers of the architecture. References 1. Shapiro, S.C., Bona, J.P.: The GLAIR Cognitive Architecture. International Journal of Machine Consciousness 2(2) (2010) 307–332 2. Shapiro, S.C., Rapaport, W.J.: The SNePS family. Computers & Mathematics with Applications 23(2–5) (1992) 243 – 275 3. Shapiro, S.C., The SNePS Implementation Group: SNePS 2.7 User’s Manual. Department of Computer Science and Engineering, University at Buffalo, The State University of New York, Buffalo, NY. (2007) Available as http://www.cse. buffalo.edu/sneps/Manuals/manual27.pdf. 4. Kumar, D.: A unified model of acting and inference. In Nunamaker, Jr., J.F., Sprague, Jr., R.H., eds.: Proceedings of the Twenty-Sixth Hawaii International Conference on System Sciences. Volume 3. IEEE Computer Society Press, Los Alamitos, CA (1993) 483–492 1 c Sensor diagram �Parallax Inc., available under the Creative Commons AttributionShare Alike 3.0 license at http://learn.parallax.com/KickStart/28015 Ontological modeling of emotion-based decisions Daniele Porello1 , Roberta Ferrario1 , Cinzia Giorgetta1 Institute of Cognitive Sciences and Technologies, CNR {daniele.porello, roberta.ferrario, cinzia.giorgetta}@loa.istc.cnr.it Abstract. In this paper, we discuss a number of elements for developing an ontological model for representing decision making of human agents. In particular, our aim is to connect the results in cognitive science that show how emotions affect decisions with agent systems and knowledge representation. We focus in particular on the case of regret. 1 Introduction Modeling real agents’ decisions in real-world scenarios is a challenging problem for multiagent systems, normative systems and knowledge representation studies. When designing formal systems that are intended to provide tools to assist the organization of human agents’ interaction and decision making (e.g. socio-technical systems, organizations, law, [2]), it is important to account for what a real agent would do in order to evaluate the system or to design efficient prescriptions. For example, imagine a security officer in an airport (a typical case of socio-technical system, in which processes are carried out partly by humans and partly by artificial agents), who has to decide whether to check a customers’ belonging or not. This is a decision to be taken under uncertainty and risk conditions, as the officer has to judge, without much information, whether the customer could be a suspect. Moreover, we can associate payoffs to the officer’s decision: the officer’s utility increases if (s)he checks a customer that turns out to be a suspect, it may decrease in case (s)he looses time in checking regular customers, it may decrease significantly in case (s)he misses a dangerous suspect by not checking him/her. When designing tools to aid or to evaluate decisions in such a scenario, it is important to understand and to represent how real agents face this kind of situations. Decision theory and expected utility theory have been widely studied in economics and they provide prescriptions that represent what an abstract rational agent would do in risky situations. Real agents, however, often exhibit behavior that significantly diverge from expected utility theory prescriptions [3]. Since the works of Kahneman and Tverski [9], models that are able to represent cognitively biased decisions under uncertainty and risk have been developed. Moreover, several results in the decision making field and in behavioral game theory show how emotions affect decisions under uncertainty and risk (e.g. [11],[7],[5],[4],[12], [8]). For the sake of example, we shall focus on investigations in neuroeconomics that show how emotions like disappointment or regret play an important role in the way in which agents evaluate risk. The distinction between disappointment and regret is significant in order to model in a principled way expectations of what agents would do !2 in a given scenario. Both emotions are reactions to an unsatisfactory outcome and both arise from counterfactual thinking. In particular, following the analysis in [14], regret is based on behaviour-focused counterfactuals, whereas disappointment is based on situation-focused counterfactuals. The difference being that regret entails that the agent directly chose a course of actions with an unsatisfactory outcome and a better choice would have been available, whereas disappointment entails that the agent bears no direct responsibility for the choice and its outcome. Or, better, both depend on what the agent believes about his/her responsibility with respect to the negative outcome [17]. Results in neuroeconomics show for example that regret leads to riskier choices [8], whereas, in the case of disappointment, there is no relevant influence on the level of risk of the subsequent choices. In our airport example, suppose the officer checks nearly every passenger and the procedure takes so long that the flight must be delayed. Suppose also that the airline officially complains with the security company. Disappointment may be caused in case an external decision, like a supervisor’s order, has forced the officer to check all passengers. By contrast, regret may be a consequence of the officer’s decision of checking all passengers. The results that we have mentioned state that it is likely that, after (s)he has felt regret, the officer’s behavior on his/her successive choices is going to be riskier, for instance leading him/her to lower security standards. In case of disappointment, his/her behavior does not vary considerably with respect to subsequent choices. The aim of this paper is to introduce a number of important elements in order to develop a model that is capable of representing and accounting for the effects of emotions in decision making. Moreover, the model has to interface such information with the normative specifications of the system the agent is living and acting in. In order to achieve our goal, we start developing an ontological analysis of the elements that constitute agents’ decisions. The motivations behind our choice to adopt an ontological approach are manifold. First of all, we are interested in applications in socio-technical systems, and in particular in designed systems where multiple artificial agents interact with humans. A well-founded ontological model should, on the one hand, allow humans to better understand the system they are living and acting in and, on the other hand, once embedded in an artificial agent, should enable the latter to automatically reason about the system, and to exchange information with other (human and artificial) agents in a sound and interoperable way. In particular, including the representation of emotions and of their influence on decisions into the ontological model, should provide the artificial agents with means for interpreting, evaluating, and possibly foreseeing humans’ decisions. We shall use the approach to ontology provided by DOLCE [10] as it is capable of specifying some of the essential elements that we need for our analysis; in particular, we will focus on the notions of agent (inspired by the belief-desire-intentions (BDI) model), of course of actions, decision, outcome, and notions describing the emotions that may affect the decision. The remainder of this paper is organized as follows. In Section 2, we discuss the elements of the ontological model we are interested in developing and we present our discussion of emotion-based decisions. In Section 3, some final remarks are drawn. 3 ! 2 Ontological analysis: DOLCE We present some features of DOLCE [10], the ground ontology, in order to place the elements of our analysis within the general context of a foundational ontology1 . The ontology partitions the objects of discourse into the following basic categories: endurants (intuitively, objects) ED, perdurants (intuitively, events) PD, individual qualities Q, and abstracts ABS. Individual qualities are entities that agents can perceive or measure or evaluate that inhere to a particular. For example, ”the color of my car”. They are partitioned into quality kinds Qi , such as the color, the weight, the length. Examples of abstracts are provided by the categories of spaces. Spaces intuitively represent values for quality kinds. The relationship between quality kinds and spaces is modeled by the location relation loc(qx , sx ) that means, for example, that the redness of John’s car is located at a certain point in the space of colors. Figure 1 shows the categories we are interested in. We restrict our presentation to the case of endurants, namely we focus on the objects of emotions or decisions. 3 ED PED NPED Physical End. Nonphysical End. ... PO Physical object NAPO APO Nonagent. p. o. Agentive p. o. NPO D Nonphy. obj. Descriptions sD Sdesc. cD ... MOB SOB Mental obj. Social obj. PLN PRC COJ NASO ASO Plans Percept Computed obj. Nonagentive s. o. Agentive s. o. ... ... COB COI SAG Social Agent SC Society ... PEL Primary feeling ... COD Comp. Comp. Comp. belief desire intention COF Complex feeling ... ... REGO DISO disappointment Regret obj. obj. ... Fig. 1. A fragment of DOLCE Fig. 1. An excerpt of the DOLCE taxonomy focused on mental objects situations by means of S-Descriptions (descriptions of situations). Plans are thus a subcategory of S-Descriptions, they are the way in which agents think about situations. 1 We present the descriptive features DOLCE For implementability issues, see [10]. Plans are complex objects as theyofmay be . composed of a number of tasks, they have preconditions, post-conditions, and so on [1]. An agent may think about several plans (i.e. several courses of action). Following the BDI model [10], we represent an agent that is willing to bring about a particular plan p by means of the relation has BDI on which holds between an agentive object i and a plan p when the agent has the relevant beliefs, desires, intentions to bring a bout the plan: has BDI on(i, p).2 In order to evaluate the consequences of a plan for an agent i, we consider the situation that is the outcome of the plan: Out(s, p). We introduce a preference relation on situations P ref (s, s� , i) meaning that the agent i prefers situation s to situation s� . Moreover, we introduce a preference relation on plans: an agent i prefers the plan p to plan p� , pref (p� , p, i), iff the outcome of p is preferred to the outcome of p� by i. In our model, a decision that an agent takes is a choice between plans. According to expected utility theory, we denote ui (s) the utility for agent i of situation s, and the expected utility of s as uei (s) = u(s) · π(s), where π(s) is the probability of s to be the case. In order to represent uncertainty, we introduce a quality space QL of situations that represents the 2 Agentive objects allow for distinguishing decisions taken by agentive physical objects, e.g. a person, and agentive social objects, e.g. a person in the role of an officer. !4 2.1 Plans and decisions We follow the analysis of plans and norms provided by [1], that introduces a new basic category of situations SIT. Situations may be viewed as complex perdurants and they may be also considered a type of (possibly partial) events. Ho let to queste cose 100 anni fa e non ricordo null; davvero le situazioni possono essere sia endurant che perdurant? Ti credo che a Nicola non sono mai piaciute! Devo guardarmi il paper Agents refer to or think about situations by means of S-Descriptions (descriptions of situations). Plans are thus a subcategory of S-Descriptions, they are the way in which agents think about situations. Plans are complex objects as they may be composed of a number of tasks, they have preconditions, post-conditions, and so on [1]. An agent may think about several plans (i.e. several courses of action). Following the BDI model [15], we represent an agent that is willing to bring about a particular plan p by means of the relation has BDI on which holds between an agentive object i and a plan p when the agent has the relevant beliefs, desires, intentions to bring about the plan: has BDI on(i, p).2 In order to evaluate the consequences of a plan for an agent i, we consider the situation that is the outcome of the plan: Out(s, p). We introduce a preference relation on situations P ref (s, s� , i) meaning that the agent i prefers situation s to situation s� . Moreover, we introduce a preference relation on plans: an agent i prefers the plan p to plan p� , pref (p� , p, i), iff the outcome of p is preferred to the outcome of p� by i. In our model, a decision that an agent takes is a choice between plans. According to expected utility theory, we denote ui (s) the utility for agent i of situation s, and the expected utility of s as uei (s) = u(s) · π(s), where π(s) is the probability of s to be the case. Riscirtto un po’ In order to represent uncertainty in our ontological model, we introduce a quality kind QL for the likelihood of situations, that is used to express the extent to which the agent views the situation as likely. Moreover, we assume a quality space SL to represent probability values. By modeling the likelihood of a situation as an individual quality, we are assuming that human agents can measure or evaluate the likelihood of situations and this is compatible with the psychological literature, see for example [9]. We define the location relation loc(qs , ls , i) that associates the likelihood quality qs of the situation s with the probability ls according to agent i’s view. The relationship with decision theory is the following: we can view situations as the space of events and the quality as the mapping of values of a probability distribution that i defines on events. Note that, since a plan describes a complex situation, the probability of the execution of the plan has to be computed taking into account conditional dependencies of sub-situations that occur in the plan. We are not going to go into the details of computing the probability and we abstractly represent the likelihood of a plan by means of the likelihood of its outcome: Out(p, s) ∧ loc(qs , ls , i). According to expected utility theory, the abstract rational agent would choose the plan p that maximizes the expected utility has BDI on(i, p) ↔ ¬∃p� pref (p, p� , i), where i’s preferences are defined as: pref (p, p� , i) iff the expected utility of p is greater than the expected utility of p� . However, as our discussion is centered on human agents, there may be several reasons that prevent it to be the case (besides not knowing what the best plan is), as recent studies 2 Agentive objects allow for distinguishing decisions taken by agentive physical objects, e.g. a person, and agentive social objects, e.g. a person in the role of an officer. !5 in behavioral decision [18] making have already pointed out. As we have seen, in the context of risky choices, regret is one of these reasons. 2.2 Representing emotion-based decisions under uncertainty An ontological analysis of the BDI model and of emotions has been developed in [13] and [6]. In particular, feelings like disappointment and regret can be categorized as particular complex feelings3 , which are in turn particular types of complex percepts. Complex feelings depend on primary feelings (PEL) as well as on beliefs. Our analysis of the distinction between disappointment and regret is then the following. For reasons of space, we omit the subtleties of the analysis in [13] and we define regret and disappointment referred to a situation that is the outcome of a plan. We introduce a dependence relation of a mental object x on a situation s, Dep(x, s). At this level of analysis, the relation Dep is very abstract and it may be open to several interpretations: a situation may cause disappointment because of some of the objects that participate in the situation (e.g. the outcome of the plan is an asset for which the agent gets a very low payoff) or because of some property of the situation itself (e.g. the outcome of the plan is an event that entails waiting in line in a long queue). Regret is caused by the negative outcome of a plan that is chosen by agent i in spite of the existence of an alternative plan p� that the agent might have chosen with a better outcome4 . Disappointment is caused by a plan whose outcome is dominated by another situation, but i is not responsible. REGO (x, i) ≡ AP O(i) ∧ P LN (p) ∧ Dep(x, s) ∧ Out(p, s) ∧ has BDI on(i, p) ∧ ∃p� pref (p, p� , i) DISO (x, i) (1) ≡ AP O(i) ∧ P LN (p) ∧ Dep(x, s) ∧ Out(p, s) ∧ ∃p� pref (p, p� , i) (2) The condition AP O(i) specifies that the emotions apply to agentive physical objects and the condition P LN (p) specifies that both emotions are dependent on the outcome of a plan. The distinction shows that, in case of regret, the actualization of the outcome is caused by the agent’s choice, whereas disappointment is caused by the negative outcome deriving from a course of actions triggered by someone else’s choice, namely by another plan. Note that the definition of disappointment is narrower than its intuitive meaning, as it makes the disappointing situation depend on the course of action described by a plan, i.e. decided by someone else. However, our definitions avoid unintuitive cases such as being disappointed by any possible disfavored event. A more precise definition can be formulated by introducing hope or expectations [13]. We can now present our preliminary analysis of how regret that is caused by a particular choice 3 4 Disappointment and regret can be more precisely defined as cognitive emotions [16]; here we use the locution “complex feelings” following [13]. The responsibility of a choice of a plan is defined by has BDI on. A closer examination shall single out the role of intentions in taking responsibility. We leave this point to future work. !6 of p may affect subsequent choices of plans. We define it as a reason to choose a plan, namely we restate the definition of has BDI on. We present it in a semi-formal way in order to avoid introducing the relevant elements of temporal analysis and of expected utility theory. Definition 1. An agent i has BDI on a plan p at time t iff either p is its most preferred implementable plan; or p is dominated by p�� , p is more risky than p�� , p has a higher utility value, and there is a precedent time t� , such that the agent has chosen p� and p� has caused regret on i. 3 Conclusion We have presented a number of elements that are important to develop an ontological analysis of the role of emotions in decision making. In particular, we have sketched how to integrate in DOLCE decisions as choices between plans, emotions, and risk attitudes. Future work shall provide a tighter connection between the cognitive and the normative modules of DOLCE and the analysis of decisions under uncertainty. There are several possible extensions of the present work, which is centered on the influence of regret on risk-seeking behavior, for what concerns a single agent. However, in environments populated by more agents, as for instance multiagent systems, understanding and modeling how beliefs, emotions and expectations mutually influence each other becomes particularly important. In such systems agents, while planning, should try to foresee which would be the preferences of the other agents and their propensity to risk, given also the previous history. It is worth noting that, in the specific case of regret, when other agents are involved, the agent who was in charge of the choice may feel regret for him/herself and guilt with respect to the others, who in their turn can feel disappointment for the negative outcome. Furthermore, we could also think about the emotional reactions following collective decisions. In the more specific case of socio-technical systems, decisions can also depend on the performance of technical devices, so not directly from agents, but there may be agents who are responsible of the functioning of such devices (sometimes their designer may be at the same time a participant of the system) and may eventually feel regret. It is exactly in such complex scenarios that the use of the ontological approach is especially useful. Finally, so far we have focused just on a specific emotion, regret (and marginally disappointment), and a specific attitude, propensity to risk, but cognitive science studies have dealt with other emotions, like fear, sadness, happiness, anger, guilt etc. and also with other mental attitudes, like for instance propensity to cooperation, that can affect each other and then influence (individual and/or social) decisions. There are then many possible extensions to this preliminary model that, once complete, could be also implemented in artificial agents, thanks to the use of ontologies. Acknowledgements Daniele Porello, Roberta Ferrario and Cinzia Giorgetta are supported by the V IS C O S O project, which the present paper is part of. V IS C O S O is financed by the Autonomous Province of Trento through the “Team 2011” funding programme. Bibliography [1] G. Boella, L. Lesmo, and R. Damiano. On the ontological status of plans and norms. Artif. Intell. Law, 12(4):317–357, 2004. [2] G. Boella and L. V. D. Torre. Introduction to normative multiagent systems. Computational and Mathematical Organization Theory, 12:71–79, 2006. [3] C. Camerer. Behavioral Game Theory: Experiments in Strategic Interaction. The Roundtable Series in Behavioral Economics. Princeton University Press, 2003. [4] H. Chua, R. Gonzalez, S. Taylor, R. Welsh, and I. Liberzon. Decision-related loss: regret and disappointment. Neuroimage, 47:2031–2040, 2009. [5] G. Coricelli, H. Critchley, M. Joffily, J. O’Doherty, A. Sirigu, and R. Dolan. Regret and its avoidance: A neuroimaging study of choice behaviour. Nature Neuroscience, 8:1255–1262, 2005. [6] R. Ferrario and A. Oltramari. Towards a computational ontology of mind. [7] W. Gehring and A. Willoughby. The medial frontal cortex and the rapid processing of monetary gains and losses. Science, 295:2279–2282, 2002. [8] C. Giorgetta, A. Grecucci, N. Bonini, G. Coricelli, G. Demarchi, C. Braun, and A. G. Sanfey. Waves of regret: A meg study of emotion and decision-making. Neuropsychologia, 2012. [9] D. Kahneman and A. Tversky. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263–91, March 1979. [10] C. Masolo, S. Borgo, A. Gangemi, N. Guarino, and A. Oltramari. Wonderweb deliverable d18. Technical report, CNR, 2003. [11] B. Mellers, Schwartz, and I. A., Ritov. Emotion-based choice. Journal of Experimental Psychology: General, 128:332–345, 1999. [12] A. Nicolle, D. Bach, J. Driver, and R. Dolan. A role for the striatum in regretrelated choice repetition. Journal of Cognitive Neuroscience, 23:1–12, 2010. [13] A. Oltramari. Hybridism in cognitive science and technology. Foundational and implementational issues. PhD Thesis, Università degli Studi di Trento. Scienze della Cognizione e della Formazione, 2006. [14] W. W. van Dijk, M. Zeelenberg, and J. van der Pligt. Blessed are those who expect nothing: Lowering expectations as a way of avoiding disappointment. Journal of Economic Psychology, 24(4):505–516, 2003. [15] M. Woolridge. Introduction to Multiagent Systems. John Wiley & Sons, Inc., New York, NY, USA, 2008. [16] M. Zeelenberg. Anticipated regret, expected feedback and behavioral decisionmaking. Journal of Behavioral Decision Making, 12:93–106, 1999. [17] M. Zeelenberg, W. van Dijk, A. Manstead, and J. van der Pligt. The experience of regret and disappointment. Cognition and Emotion, 12:221–230, 1998. [18] M. Zeelenberg, W. van Dijk, A. Manstead, and J. van der Pligt. On bad decisions and disconfirmed expectancies: The psychology of regret and disappointment. Cognition and Emotion, 14:521–541, 2000. Information Binding with Dynamic Associative Representations ARAKAWA, Naoya Imaging Science and Engineering Laboratory, Tokyo Institute of Technology Abstract. This paper proposes an explanation of human cognitive functions in terms of association. In this proposal, representation is not static but is achieved by dynamic retrieval of patterns by means of association. With this dynamic representation, information binding in cognitive functions such as physical world recognition, planning and language understanding is illustrated. Keywords: association, generativity, dynamic representation, the binding problem 1 Introduction This paper addresses the issue of representation in human cognitive functions in terms of association and attempts to show the benefit of seeing cognitive representation as dynamic associative processes with examples of perception, planning and language understanding. In this paper, association designates the process of obtaining a pattern from another through learning. As discussed in [4], patterns here can be sub-symbolic vector representations formed by nonsupervised learning. The reason why association is taken to be the basis for explaining cognitive functions here is two-fold: 1) cognitive functions are often accounted for with sub-symbolic information processing and association is considered to be its abstraction; 2) while sub-symbolic information processing is formalized with models such as neural networks and Bayesian models, its technical details could obscure the essence of over-all functions in certain discussions. Having said that association is an abstraction of sub-symbolic information processing, the following are reasons why sub-symbolic information processing is important for explaining cognitive functions. First, for models of brain functions such as neural networks are normally assumed to be sub-symbolic, their incorporation would be essential for biologically natural or realistic cognitive modeling. Second, while learning is an essential cognitive function, most modern-day learning models are sub-symbolic. Finally, sub-symbolic information processing would evade thorny issues in cognitive science such as the issue of classical categories brought forth by cognitive linguists such as Lakoff [7] and the frame problem. Associative memory !2 could evade the issue of classical categories, since sub-symbolic associative models (e.g., neural models) can represent prototypical and fuzzy categories. Certain frame problems for intelligent agents to find relevant information could also be evaded, as sub-symbolic associative memory could retrieve most relevant information (association) for given patterns first (by means of a competitive process among‘ neurons, ’for example) (see [5] for a debate). However, there are numerous problems with using association with cognitive modeling. First of all, it is not clear if association alone is enough for explaining dynamic cognitive functions such as planning or language understanding, as association is nothing more than mapping between patterns. Secondly, as it was argued in the computationalism vs. connectionism debate (see [2]), it is not clear how connectionist/associationist models account for generativity of, say, linguistic structure. Moreover, associationism also bears the so-called binding problem for explaining the integration of various pieces of information (see [12] and 2.1 below). Given the aforementioned pros and cons, my motivation is to explain human cognitive functions as an associative processes by addressing the problems raised above. 2 The Proposal The basic idea of the proposal is that associative representation is better conceived dynamically as traversal of association paths, if we are to address the issues of information binding and generativity. Note that the issues are not problematic in the case of classical symbolic models (e.g., [9]) or symbolic/subsymbolic hybrid cognitive architectures (e.g., [1] [10]), in which the composition/binding of information can be done symbolically by means of variable binding. However, when we regard the human brain as a sub-symbolic associative machine, the apparent lack of binding mechanism becomes problematic for explaining human cognitive functions. Therefore, a (sub-) motivation here is to explain cognitive functions with an associative mechanism without bringing in variable binding. While the motivation is mainly explanatory, the proposal may entertain technological merit: a cognitive system may be constructed with subsymbolic learning mechanisms without generating symbolic rules for the sake of simplicity, although whether such a system has a practical advantage would depend on the case. In the sections to follow, the proposed method will be explained more in detail by using concrete examples. First, physical world (scene) recognition is discussed as the basic case of the binding problem. Then, discussions of planning and language understanding will follow. 2.1 Physical world recognition Recognizing the physical environment involves recognizing objects, their locations and features such as motion. When a cognitive system has to perceive multiple objects at once, the binding problem occurs: the system has to extract !3 features from each entity and ascribe them to the proper item. The problem is acute with parallel information processing presumably performed in our brain; how can pieces of information represented in parallel be integrated in a coherent representation? For example, seeing a blue circle and a red rectangle, how does a brain integrate the representations of blue, red, circle and rectangle as those of a blue circle and a red rectangle in a particular relative location? In the current binding problem literature, synchronicity is often mentioned as a possible solution to the problem ([12][3], for example). However, synchronicity alone does not solve the problem. In the first place, it does not answer the question of how different objects are represented in the brain. Secondly, considering that visual scenes are perceived with saccades, there remains a suspicion that visual perception is rather dynamic than a synchronic process. An associative system could cope with the (perceptual) binding problem in the following manner. Suppose that the system obtains a feature pattern of a physical object as well as a pattern representing the coordinates1 of the object and that it obtains the information of an object at a time. In this situation, the system shifts ‘ attention ’ among objects to recognize Fig. 1 Association among figure repremore than one object2 . The associa- sentations tive retrieval of object representation Each figure represents feature patterns for could be driven by cues representing the figure. The numbers represent cue patorientations (such as 15 degrees left) terns for relative degrees and distances. and relative distances (Fig. 1). In this scheme, a scene (containing more than one object) may not be simultaneously represented, but object representations may be retrieved one-by-one by means of association in short-term memory or working memory and bound together as a coherent scene representation. What assures the information binding here is not synchronicity but the potential or mechanism to bring all relevant representations together. 2.2 Planning A planning mechanism by means of association could be conceived in a rather straightforward way as described by the following (pseudo) algorithm: 1 2 Information on the location of physical objects can be obtained via integrating various modalities such as vision, touch, vestibular sense and the sense of eye rotation. The idea is in line with the Feature Integration Theory [14] but formulated in a more abstract level. 4 ! if the pattern representing the current situation is not desirable according to certain criteria, then associate the current situation pattern (CSP) with an action pattern (APT), which represents either a directly executable action or a plan. if CSP and APT are not associated with (do not predict) a pattern representing a satisfactory situation, then associate another APT with CSP by suppressing the current APT. end if APT represents a learned plan (PL), then LP: associate APT with a sequence of APTs (APT1, APT2,…). if the situation pattern associated with (predicted from) CSP and a sequence consisting of APTi (i=1,2,…) is evaluated as unsatisfactory, then abort and suppress PL. end if APTi is a learned plan, then retrieve a sub-sequence for APTi recursively (⇒ LP:). end if all the sub-sequences consist of directly executable actions and the patterns of situations associated (predicted) with the entire APTs and CSP are satisfactory, then the entire action (a new plan) is ready to be executed. end end While such an algorithm is a variation of classical planning algorithms, some notes can be given for that to be realized associatively. In planning, the output sequence could have any length and plan hierarchy could have any depth. In other words, a plan is a cognitive representation having a combinatory/generative nature. A combinatory representation can be represented Fig. 2 Planning by means of association with an associative network, Ovals are patterns representing the indicated which actually is a dynamic content. Blue solid lines represent associative process of recalling (associrelations. ating) patterns. In case of a plan, the entire plan can be represented by retrieving its parts one by one by means of association. A representation of an action in a plan may be retrieved 5 ! from a representation of the preceding action with a cue pattern representing the temporal relation succeeding. Here, note that in the sample algorithm above, a plan is formed (information is bound) with association but without having recourse to rules with variables. A plan hierarchy (Fig. 2) would be represented by associating a pattern representing a learned plan with patterns representing actions or sub-plans3 . Constructing hierarchical plans requires back-tracking, which, in turn, requires a mechanism that suppresses an unwanted pattern and retrieves another relevant pattern (by means of association). 2.3 Language understanding Understanding language involves syntactic parsing and associating syntactic structures with semantic representations. As a sentence in a human language can be indefinitely long, the corresponding associative syntactic/semantic representation can be indefinitely large. Again, such representation has the combinatory/generative nature. While an associative network may not be able to keep simultaneous activation of patterns for a large representation, the entire representation could be retrieved piece by piece. As for semantic representation, computational models known as semantic networks (e.g., [11]) are considered to be associative. A node of a semantic network represents an individual or type of physical objects, events, situations, time or location.4 Semantic networks normally have labeled edges to represent relations between nodes. A labeled node may be realized in an associative network as a patFig. 3 A regular phrase structure tern representing a node and a cue Blue solid lines can be associative relations, pattern representing the type of where ovals represent patterns for syntactic an edge or relation. For example, categories. the edge or relation representing bigger-than may be realized as association from a pattern representing an individual and a (cue) pattern representing bigger-than to the representation of another individual. With regard to parsing, the syntax of human language has recursive structure, i.e., a structure may be embedded in another structure. Such recursive 3 4 [13] can be an exemplary work in this line. A node in an associative network may accrue meaning by means of non-supervised learning so that they can be interpreted as a node of a ‘ semantic ’ network. In fact, a semantic network alone does not give the semantics to its components. See discussion in [6]. !6 structure would be constructed by a mechanism similar to the one discussed in the section of planning above, where the construction of sentential structure would be the goal. The system could traverse a syntactic tree by associating daughter constituents with the parent node and associating one node to its sibling nodes with cue patterns representing such as left and right (Fig. 3). Finally, mapping from syntactic structure to semantic structure must be taken into consideration. A pattern representing a syntactic constituent such as a clause, verb and noun may be associated with a pattern representing situation, event and object respectively. Here, syntactic relations may be associated with Fig. 4 Syntactic and Semantic Patterns patterns representing semantic relations. For example, an English verb phrase consisting of a verb and a following noun (having the accusative relation) would be associated with the representation of an event, an object and the theme relation between them (Fig. 4). 3 Conclusion The associative representations proposed herein have shared characteristics. In the three cases, representation is not static but is achieved by dynamically retrieving patterns by association. The associative retrieval is accompanied by cue patterns such as those indicating spatial and temporal directions and semantic/syntactic relations. The proposed representations also cope with the issues of dynamicity and generativity and the binding problem. As for dynamicity, planning and language understanding discussed above are dynamic and the representations proposed here are all dynamic. As for generativity, representations in planning and language understanding are generative and those in physical world recognition can also be generative, as a scene is composed of many objects. The issue of binding without variable was addressed in physical world recognition as well as in language understanding and planning, as these processes must also bind pieces of information together into coherent representations. The proposal here apparently requires empirical studies. In particular, if the illustrated mechanisms are to serve as certain functions, association should be properly controlled. So, while the author plans to implement experimental systems for corroborating the proposal, the issue of the (executive) control [8] shall be seriously addressed. !7 Acknowledgements I would like to express my deep gratitude to the comments from the referees of this paper and also from Quentin Quarles, without which the paper did not form the current shape. References [1] ACT-R: http://act-r.psy.cmu.edu [2] Fodor, J.A., Pylyshyn, Z.W.: Connectionism and cognitive architecture: A critical analysis. Cognition 28(1), 3–71 (1988) [3] Fuster, J.M.: Cortex and mind: Unifying cognition. Oxford University Press (2005) [4] Gärdenfors, P.: Conceptual spaces: The geometry of thought. MIT press (2004) [5] Haselager, W., Van Rappard, J.: Connectionism, systematicity, and the frame problem. Minds and Machines 8(2), 161–179 (1998) [6] Johnson-Laird, P.N., Herrmann, D.J., Chaffin, R.: Only connections: A critique of semantic networks. Psychological Bulletin 96(2), 292–315 (1984) [7] Lakoff, G.: Women, fire, and dangerous things: What categories reveal about the mind. University Of Chicago Press (1990) [8] Miyake, A., Shah, P.: Models of working memory: Mechanisms of active maintenance and executive control. Cambridge University Press (1999) [9] Newell, A.: Unified theories of cognition. Harvard University Press (1994) [10] OpenCog: http://wiki.opencog.org/w/The Open Cognition Project [11] Quillian, M.R.: Semantic memory. In: Minsky, M. (ed.) Semantic information processing, pp. 227–259. MIT Press (1968) [12] Revonsuo, A.: Binding and the phenomenal unity of consciousness. Consciousness and cognition 8(2), 173–185 (1999) [13] Subagdja, B., Tan, A.H.: A self-organizing neural network architecture for intentional planning agents. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2. pp. 1081–1088. International Foundation for Autonomous Agents and Multiagent Systems (2009) [14] Treisman, A.M., Gelade, G.: A feature-integration theory of attention. Cognitive psychology 12(1), 97–136 (1980) Reductio ad Absurdum: On Oversimplification in Computer Science and its Pernicious Effect on Artificial Intelligence Research Kristinn R. Thórisson1,2 1 Center for Analysis and Design of Intelligent Agents / School of Computer Science, Reykjavik University, Venus, Menntavegur 1, Reykjavik, Iceland 2 Icelandic Institute for Intelligent Machines, 2.h. Uranus, Menntavegur 1, Reykjavik, Iceland [email protected] Abstract. The Turing Machine model of computation captures only one of its fundamental tenets – the manipulation of symbols. Through this simplification it has relegated two important aspects of computation – time and energy – to the sidelines of computer science. This is unfortunate, because time and energy harbor the largest challenges to life as we know it, and are therefore key reasons why intelligence exists. As a result, time and energy must be an integral part of any serious analysis and theory of mind and thought. Following Turing's tradition, in a misguided effort to strengthen computer science as a science, an overemphasis on mathematical formalization continued as an accepted approach, eventually becoming the norm. The side effects include artificial intelligence research largely losing its focus, and a significant slowdown in progress towards understanding intelligence as a phenomenon. In this position paper I briefly present the arguments behind these claims. 1 Introduction A common conception is that the field of computer science provides an obvious and close to ideal foundation for research in artificial intelligence (AI). Unfortunately some fundamental derailments prevent computer science – as practiced today – from providing the perfectly fertile ground necessary for the two to have the happy marriage everybody is hoping for. Here we will look at two major such derailments. 2 Derailment #1: The Turing Machine Alan Turing's characterization of computation – as the sequential reading and writing of symbols by a simple device we now know as a Turing Machine (Turing 1948) – is generally considered to be a cornerstone of computer science. Turing's influential paper On Computing Machinery and Intelligence (Turing 1950) took notable steps towards considering intelligence as a computational system. The foundation of what came to be called artificial intelligence – the quest for making machines capable of what we commonly refer to as "thought" and "intelligent action" – was laid a few years after his papers were written. The main inspiration for the field came of course from nature – this is where we still find the best (some might say only) examples of intelligence. The original idea of artificial intelligence, exploration of which already started with cybernetics (cf. Heylighen & Josly 2001), was to apply the tools of modern science and engineering to the creation of generally intelligent machines that could be assigned any task: that could wash dishes and skyscraper windows to writing research reports and discovering new laws of nature; that could invent new things and solve difficult problems requiring imagination and creativity. Before we go further on this historical path, let's look at two natural phenomena that play a large role in science and engineering: time and energy. Time and energy are directly relevant to the nature of intelligence on two levels. First, because every computation must take place in a medium, and every medium requires some amount of time and energy to act, there are limits on the number of computations that can be produced a given timeframe. This level of detail is important to AI because intelligence must be judged in the context of the world – including the computing medium – in which it occurs: If the mind of an intelligent agent cannot support a sufficient computation speed for it to act and adapt appropriately in its environment, we would hardly say that the agent is "dumb" because it would be physically incapable of acting intelligently. The physical properties of environments present time and energy constraints; the "hardware" of a thinking agent must meet some minimum specification to support thought at sufficient speeds for survival. Unless we study the role time and energy play at this level of the cognitive computing medium we are neither likely to understand the origins of intelligence nor its operating principles. At the cognitive level time must in fact occupy part of the content of any intelligent mind: Every real-world intelligent agent must be able to understand and think about time, because everything they do happens in time. The situation is similar with respect to energy at this level (although in the case of humans it used to be more relevant the past than it is now, as foraging and farming occupied more time in our ancestors' minds than ours). In either case, a key role of intelligence from moment to moment remains in large part to help us handle the ticking of a real-world clock by thinking about time: To make better use of time, to be able to meet deadlines and understand the implications of missing them, to shorten the path from a present state to a new state, to speed up decision time by using past experiences and decision aids, and so on. Having unbounded time means that any problem can be solved by a complete search of all possibilities and outcomes. But if this is the case, intelligence is essentially not needed: Disregarding time renders intelligence essentially irrelevant. And so the very subject of our study has been removed. Therein lies the rub: Unlike the paths taken (so far) in some of the subdomains of computer science, the field of AI is fundamentally dependent on time and energy – these are two of its main raison d'être – and therefore must be an integral part of its theoretical foundation. Fast forward to the present. The field we know as 'computer science' has been going strong for decades. But it gives time and energy short shrift as subjects of importance. To be sure, progress continues on these topics, e.g. in distributed systems theory and concurrency theory, among others. But it is a far cry from what is needed, and does not change historical facts: Few if any programming languages exist where time is a first-class citizen. Programming tools and theories that can deal properly with time are sorely lacking, and few if any real methods exist to build systems for realtime performance without resorting to hardware construction. Good support for the design and implementation of energy-constrained and temporally-dependent systems (read: all software systems) is largely relegated to the field of "embedded systems" (cf. Sifakis 2011) – a field that limits its focus to systems vastly simpler than any intelligent system and most biological process found in nature, thus bringing little additional value to AI. As a result, much of the work in computer science practitioners – operating systems, databases, programming tools, desktop applications, mathematics – are rendered irrelevant to a serious study of intelligence. What caused this path to be taken, over the numerous others possibilities suggested by cybernetics, psychology, engineering, or neurology? Finding an explanation takes us back to Turing's simplified model of computation: When he proposed his definition of computation Turing branched off from computer engineering through a dirty trick: His model of computation is completely mute on the aspects of time and energy. Yet mind exists in living bodies because time is a complicating factor in a world where energy is scarce. These are not some take-it-or-leave-it variables that we are free to include or exclude in our scientific models, these are inseparable aspects of reality. As an incremental improvement on past treatments, some might counter, Turing's ideas were an acceptable next step, in a similar way that Newton's contributions in physics were before Einstein (they were not as thoroughly temporally grounded). But if time and energy are not needed in our theories of computation we are saying that they are irrelevant in the study of computation, implying that it does not matter whether the computations we are studying take no time or infinite time: The two extremes would be equivalent. Such reductio ad absurdum, in the literal meaning of the phrase, might possibly be true in some fields of computer science – as they happen to have evolved so far – but it certainly is not true for AI. If thinking is computation we have in this case rendered time irrelevant to the study of thought. Which is obviously wrong. An oversimplification such as this would hardly have been tolerated in engineering, which builds its foundations on physics. Physicists take pride in making their theories actually match reality; would a theory that ignores significant parts of reality have been made a cornerstone of the field? Would the theory of relativity have received the attention it did had Einstein not grounded it with a reference to the speed of light? Somehow E = m is not so impressive. The situation in computer science is even worse, in fact, because with Turing's oversimplification – assuming infinite time and energy – nothing in Einstein's equation would remain. 4 Derailment #2: Premature Formalization The inventors of the flying machine did not sit around and wait for the theory of aerodynamics to mature. Had the Wright brothers waited for the "right mathematics", or focused on some isolated part of the problem simply because the available mathematics could address it, they would certainly not be listed in history as the pioneers of aviation. Numerous other major discoveries and inventions – electricity, wireless communications, genetics – tell a similar story, providing equally strong examples of how scientific progress is made without any requirement for strict formal description or analysis. In addition to relegating time and energy to a status of little importance in AI, rubbing shoulders with computer science for virtually all of its 60-year existence has brought with it a general disinterest in natural phenomena and a pernicious obsession with formalization. Some say this shows that AI suffers from physics envy – envy of the beauty and simplicity found in many physics equations – and the hope of finding something equivalent for intelligence. I would call it a propensity for premature formalization. One manifestation of this is researchers limiting themselves to questions that have a clear hope of being addressed with today's mathematics – putting the tools in the driver's seat. Defining research topics in that way – by exclusion, through the limitations of current tools – is a sure way to lose touch with the important aspects of an unexplained natural phenomenon. Mathematical formalization does not work without clear definitions of terms. Definition requires specifics. Such specification, should the mathematics invented to date not be good for expressing the full breadth of the phenomena to be defined (which for complex systems is invariably the case), can only be achieved through simplification of the concepts involved. There is nothing wrong with simplification in and of itself – it is after all a principle of science. But it matters how such simplification is done. Complex systems implement intricate causal chains, with multiple negative and positive feedback loops, at many levels of detail. Such systems are highly sensitive to changes in topology. Early simplifications are highly likely to leave out key aspects of the phenomena to be defined. The effects can be highly unpredictable; the act will likely result in devastating oversimplification. General intelligence is capable of learning new tasks and adapting to novel environments. The field of AI has, for the most part, lost its ambition towards this general part of the intelligence spectrum, and focused instead on the making of specialized machines that only slightly push the boundaries of what traditional computer science tackles every day. Part of the explanation is an over-reliance on Turing's model of computation, to the exclusion of alternatives, and a trust in the power of formalization that borders on the irrational. As concepts get simplified to fit available tools, their correspondence with the real world is reduced, and the value of subsequent work is diminished. In the quest for a stronger scientific foundation for computer science, by threading research through the narrow eye of formalization, exactly the opposite of what was intended has been achieved: The field has been made less scientific. 5 What Must Be Done In science the questions are in the driver seat: A good question comes first, everything else follows. Letting the tools decide which research questions to pursue is not the right way to do science. We should study more deeply the many principles of cognition that are difficult to express in today's formalisms, system architectures implementing multiple feedback loops at many levels of detail, for instance; only this way can we simultaneously address the self-organizing hierarchical complexity and networked nature of intelligent systems. Temporal latency is of course of central importance in feedback loops and information dissemination in a large system. All this calls for greater levels of system understanding than achieved to date (cf. Sifakis 2011), and an understanding of how time and energy affect operational semantics. The very nature of AI – and especially artificial general intelligence (AGI) – calls for a study of systems. But systems theory is immature (cf. Lee 2006) and computer science textbooks typically give system architecture short shrift. The rift between computer science and artificial intelligence is not a problem in principle – computer science could easily encompass the numerous key subjects typically shunned in AI today, such as non-axiomatic reasoning, existential autonomy, fault-tolerance, graceful degradation, automatic prioritization of tasks and goals, and deep handling of time, to name some basic ones. Creativity, insight and intuition, curiosity, perceptual sophistication, and inventiveness are examples of more exotic, but no less important, candidates that are currently being ignored. Studying these with current formalisms is a sure bet on slow or no progress. We don't primarily need formalizations of cognitive functions per se, first and foremost we need more powerful tools: New formalisms that don't leave out key aspects of the real world; methods that can address its dynamic complexity head-on, and be used for representing, analyzing, and ultimately understanding, the operation of large complex systems in toto. Acknowledgments. Thanks to Eric Nivel, Pei Wang, Hrafn Th. Thórisson, Helgi Páll Helgason, Luca Aceto, Jacky Mallett, and the anonymous reviewers for comments on the paper. This work has been supported in part by the EU-funded project HUMANOBS: Humanoids That Learn Socio-Communicative Skills by Observation, contract no. FP7-STReP-231453 (www.humanobs.org), by a Centres of Excellence project grant (www.iiim.is) from the Icelandic Council for Science and Technology Policy, and by grants from Rannís, Iceland. References Heylighen, F. & C. Joslyn (2001). Cybernetics and Second-Order Cybernetics. Encyclopedia of Physical Science & Technology, 3rd ed. New York: Academic Press. Lee, E. E. (2006). Cyber-Physical Systems - Are Computing Foundations Adequate? Position Paper for NSF Workshop On Cyber-Physical Systems: Research Motivation, Techniques and Roadmap. Sifakis, J. (2011). A Vision for Computer Science – the System Perspective. Cent. Eur. J. Comp. Sci., 1:1, 108-116. Turing, A. (1948). Intelligent Machinery. Reprinted in C. R. Evans and A. D. J. Robertson (eds.), Cybernetics: Key Papers, Baltimore: University Park Press, 1968. Turing, A. M. (1950). Computing Machinery and Intelligence. Mind, 59:236, 433460. On Deep Computational Formalization of Natural Language Naveen Sundar Govindarajulu is a PhD candidate in Computer Science at RPI.! ! His thesis work is on designing and implementing the first uncomputable games and has been partly funded by the International Fulbright Science and Technology Award. Naveen holds a dual degree in Physics and Electronics & Electrical engineering from BITS-Pilani, obtained with help provided by the GE Foundation's Scholar-Leader award. His prior experience includes research at Hewlett Packard Labs, the Tata Institute of Fundamental Research, and the Indian Space Research Organization’s Mission Control Center. Dr. Selmer Bringsjord is Professor of Logic, Computer Science, Cognitive Science, Management & Technology, and is the Chair of the Department of Cognitive Science at RPI. Prof. Bringsjord is also the Director of the Rensselaer AI & Reasoning (RAIR) Lab. Bringsjord’s focus is on logicbased AI, from both an engineering and foundations point of view. He is the author of numerous papers and books, and routinely lectures and demos across the globe. John Licato! is a computer science Ph.D. student at Rensselaer Polytechnic Institute, working with the Rensselaer AI and Reasoning lab. His research interests include Analogico-Deductive Reasoning (ADR) which combines analogical and deductive reasoning techniques to solve problems, the computational modeling and philosophical basis of analogy, robotics, and the modeling of higher cognition. On Deep Computational Formalization of Natural Language Naveen Sundar Govindarajulu1 • Selmer Bringsjord2 • John Licato3 Rensselaer AI & Reasoning (RAIR) Lab Cognitive Science2 , Computer Science1,2,3 & Lally School of Management & Technology2 Rensselaer Polytechnic Institute (RPI), Troy NY 12180 USA. {govinn,selmer,licatj}@rpi.edu Keywords: Deontic Cognitive Event Calculus, formalization of representation and reasoning, Montague semantics, Discourse Representation Theory. 1 Introduction Current AI and NLP operate on fragments of natural language and within specific application domains. Even the most successful AI/NLP system in existence today, IBM’s Watson, is highly limited when it comes to simple language processing just beyond its ken. In order to paralyze (at least the Jeopardy!-winning version of) Watson, one has only to ask it questions that have never been asked before, or questions to which answers have never been recorded before. Such queries are easy to formulate; for example: – “If I have 4 foos and 5 bars, and if foos are not the same as bars, how many foos will I have if I get 3 bazes which just happen to be foos?”; or – “What was IBM’s Sharpe ratio in the last 60 days of trading?”1 We contend that one of the major reasons for this lack of generality in AI/NLP systems is the absence of a wide-ranging formalization of natural language that is both fully formal and rigorously computational. We herein elaborate on the requirements that such a theory should meet, and very briefly evaluate the two most prominent projects in formal semantics: the Montagovian approach and Kamp’s DRT approach (laid out under our rubric). We then encapsulate our own approach, which is rooted in formal computational logic; this approach fares markedly better than either the Montagovian or the DRT tack. 2 Requirements for a Computational Formalization of Natural Language Formalizing language can quickly become a philosophically troubled project, but we are humble, in that we want to formalize language just to the extent that it allows us to 1 The approach presented in this paper will be used in a collaboration with IBM, and with RPI’s Jim Hendler, in order to enable subsequent versions of Watson to answer such questions on the strength of robust problem-solving. do certain meaningful things computationally. The requirements that drive our efforts are spelled out below. All language formalization approaches treat language as an isolated phenomenon separate from cognitive processes that use language. From a purely scientific perspective, this view, while convenient, is incomplete. The existing general approach in formal semantics (or even pragmatics) is that sentences (or linguistic phenomena) are considered in isolation, and various formal structures are posited for their meaning. Pragmatics tries to rectify this, but there is no convincing unified framework, for example like model theory for formal semantics, for formal pragmatics. Our requirements below stem from the observation that any account of language needs to include a full account of how and where it is used. Let us denote the set of all natural-language sentences and expressions in some language by ‘L ,’ and the set of all formal expressions that we can computationally handle by ‘F .’ Though some will be philosophically inclined to reject the idea, it is not unreasonable to hold that objects in F represent meaning, or, a bit more precisely, are what sentences mean. On this foundation, any formalization of language would naturally attempt to define F and give us a “meaning mapping” µ such that ∀s ∈ L ∃m ∈ F µ(s, m). With this background, we can distill our requirements into the following pair. Formalization of Extensional and Intensional Representation The formalization should specify both F and some general class of µ. The class F should be syntactically rich enough to handle not just extensional sentences of natural language such as “All apples are red.” but also challenging intensional sentences such as “Jack believes that Jane believes that all apples are red.” (Roughly put, extensional concepts pertain to those of the physical kind and intensional concepts to cognitive concepts.) The requirement to formalize both extensional and intensional sentences rules out simple extensional languages such as those at the heart of the Semantic Web (e.g., description logics, covered e.g. in Baader, Calvanese & McGuinness 2007), or those that are the target of simple domain-specific semantic parsing techniques (Kwiatkowski, Zettlemoyer, Goldwater & Steedman 2010). For a more extensive defense of the position that extensional languages and logics are not adequate to capture intensional concepts, please consult (Bringsjord & Govindarajulu 2012), in which we look at three different ways of formalizing knowledge within first-order logic. All three approaches fail by either introducing an inconsistency or by enabling unsound inferences to be drawn. Formalization of Reasoning The formalization should also have scope for including all the different kinds of reasoning processes that can be carried out with the aid of natural language. Such wide-ranging reasoning of course goes beyond just simple classical deduction. In general, given a set ΓL of sentences in natural language from which we can deduce/produce/infer another set of sentences Γ�L via some reasoning process θ, we should have formalized θ computationally as Θ (or at least be able to). That is, we should have ΓF →Θ Γ�F . 3 Current Approaches to Formalizing of Natural Language Current approaches to formalizing language can be broadly divided into the two aforementioned camps: the Montagovian approach based on Montague’s work, and the Discourse Representation Theory (DRT) framework. DRT can be considered an offshoot of the Montagovian approach and aims to incorporate pragmatics fully into its own account. 3.1 Montague’s Framework Based on the simple ontology in our requirements, we can say that this approach tries to give a formal account of the possible meanings, the set F , of different natural-language expressions and how they relate to the syntax of natural-language expressions, L , with a general theory for µ. The main shortcoming of this approach is that it fails to give a unified account, either informally and formally, of the various kinds of cognitive processes that language takes part in. Another shortcoming of this approach is that there seems to be a general absence of proof-theoretic methods. Formal Montagovian semantics leans heavily on model-theoretic methods. While model-theoretic semantics has found success in mathematical logic, the absence of proof-theoretic methods makes it hard to build computational methods.2 From a purely extensional standpoint, modeltheoretic semantics would seem to suffice. However, using model theory to account for cognitive concepts such as knowledge becomes more problematic, a fact long reflected in the invention of intensional logics in order to model knowledge, belief, obligation, and so on (a survey of such logicist modeling is provided in Bringsjord 2008). A good overview of the general approach and a brief history of Montague’s approach can be found in (Montague 1974, Dowty, Wall & Peters 1981).3 3.2 Discourse Representation Theory DRT (Kamp & Reyle 1993) arose to address one of the main perceived shortfalls of Montague’s approach: assigning meaning to linguistic expressions in a sentence taking into account other sentences. While DRT has been successful in modeling quite a bit of pragmatics, it lacks a unified formal framework like that of Montagovian formal semantics. The lack of this framework renders it incomplete for our purposes. Another shortcoming of DRT is that there is no support for intensional concepts and tense. There are some theories within the DRT family that try to mould intensionalities into extensional predicates. We show in detail in (Bringsjord & Govindarajulu 2012) how this moulding can be problematic in general. 4 A More Rigorous Framework We suggest an approach based on a formal logic with an explicit proof theory rooted in computational methods. We implement one incarnation of this approach via the Deontic Cognitive Event Calculus (DC EC ∗ ). DC EC ∗ is a quantified modal logic that builds upon on the first-order Event Calculus (EC). EC has been used quite successfully in modelling a wide range of phenomena, from those that are purely physical to narratives expressed in natural-language stories (Mueller 2006).4 2 3 4 See (Francez & Dyckhoff 2010) for more discussion on the computational advantages of a proof-theoretic semantics for natural language. Janssen’s article (Janssen 2012) on Montague’s approach is laconic and lucid, but readers who want more can consult the original sources. A nice overview of EC is provided in the well-known “AIMA” textbook (Russell & Norvig 2009). EC is also a natural platform to capture natural-language semantics, especially that of tense; see (Van Lambalgen & Hamm 2005). EC has a shortcoming: it is fully extensional and hence, as explained above, has no support for capturing intensional concepts such as knowledge and belief without introducing unsoundness or inconsistencies. For example, consider the possibility of modeling changing beliefs with fluents. We can posit a “belief” fluent belief(a, f) which says whether an agent a believes another fluent f. This approach quickly leads to serious problems, as one can substitute co-referring terms into the belief term, which leads to either unsoundness or an inconsistency (see Figure 3 in (Bringsjord & Govindarajulu 2012)). One can try to overcome this using more complex schemes of belief encoding in FOL, but they all seem to fail. A more detailed discussion of such schemes and how they fail can be found in the analysis in (Bringsjord & Govindarajulu 2012). 5 More Detailed Discussion 5.1 Formalization of Representation DC EC ∗ (deontic cognitive event calculus) is a multi-sorted quantified modal logic (for coverage of multi-sorted logic, see Manzano 1996), that has a formal, recursively defined syntax, and a proof calculus.5 DC EC ∗ syntax includes a system of sorts S, a signature f , a grammar for terms t, and a grammar for sentences φ. The proof calculus is based on natural deduction (Jaśkowski 1934), and includes all the introduction and elimination rules for first-order logic, as well as rules for the intensional operators. The formal semantics for DC EC ∗ is still under development; a semantic account of the wide array of cognitive and epistemic constructs found in the logic is no simple task — especially because of two self-imposed constraints: resisting fallback to the standard ammunition of possible-worlds semantics (which for reasons beyond the scope of the present paper we find manifestly implausible as a technique for formalizing the meaning of epistemic operators), and resisting the piggybacking of deontic operators on pre-established logics not expressly created and refined for the purpose of doing justice to moral reasoning in the human realm. For an introduction, see (Arkoudas & Bringsjord 2009). 5.2 Formalization of Reasoning DC EC ∗ supports three different modes of reasoning. We briefly touch upon them, thereby illustrating that DC EC ∗ (or, in fairness, any similar calculus) can be used to model a diverse array of reasoning processes. 5 The syntax of the language of DC EC ∗ and the rules of inference for its proof calculus are described in detail in the formal specification available at http://www.cs.rpi.edu/ ˜govinn/dcec.pdf. For prior work describing the system, please refer to (Arkoudas & Bringsjord 2009, Bringsjord & Govindarajulu 2013). We are grateful to the referees for their comments and observations on a partial version of this specification, which we had included in a prior version of the present paper. 5.3 Deductive Reasoning DC EC ∗ includes a deductive proof calculus Θd . This calculus enables one to reason not only about purely extensional concepts like events, fluents, and moments, which is reasoning enabled by the pure Event Calculus, but in addition enables reasoning about intensional concepts, such as the beliefs, knowledge, desires, intentions, communications, etc. of agents. This calculus has for instance been used to analyze the false-belief task computationally. While it is possible in principle to have a monolithic proof finder to answer whether Γ �Θd γ, proof search in this calculus can be broken down into cognitively plausible reusable procedures called λµ-methods. For a set of methods relevant to the false-belief task, see (Arkoudas & Bringsjord 2009). This deductive calculus has also been used to solve the generalized wise-men puzzle (for any number of wise men); see (Arkoudas & Bringsjord 2004). 5.4 De Se Statements We have proposed certain syntactic constructs in (Bringsjord & Govindarajulu 2013) by which one can: 1. distinguish de se statements from de dicto and de re statements; and 2. differentiate first- and third-person de se beliefs. The former can be achieved by introducing an operator ∗ which can let us know if agents are referring to themselves in an irreducible manner. This construct, stemming from Castañeda’s work (e.g., Castañeda 1999), is insufficient alone, and most work in formal semantics of self-reference stops with this mechanism. This mechanism fails when we want to distinguish between third-person de se statements and first-person de se statements. What is needed is some way of recording in the reasoning process Θ a symbol for the agent doing the reasoning. This syntactic construct goes beyond what can be talked about in classical formal semantics. Table 1 illustrates representations for different de se statements in DC EC ∗ .6 5.5 Extensional Analogical Reasoning Another focus of some of the present authors is Analogico-Deductive Reasoning (ADR) (Licato, Bringsjord & Hummel 2012), a hybrid of analogical and hypothetico-deductive reasoning that attempts to automate the generation (through analogical mapping and inference) of hypotheses, which are then subjected to deductive reasoning techniques. The use of ADR has been modeled in psychological experiments (Licato et al. 2012, Bringsjord & Licato 2012); and most recently, in Licato et al. (2013), an ADR system, Modifiable Engine for Tree-based Analogical Reasoning (META-R), was applied to the domain of theorem- and proof-discovering in cutting-edge mathematical logic. Because the surface syntax of the extensional subset of DC EC ∗ can be represented using tree structures, it lends itself nicely to the algorithms implemented by META-R. 6 We capture the indexicals {agent, time, place, . . .}, relevant for any inference, by this this notation: Γ �{agent,time,place,...} φ. For example, irreducible first-person inferences can be partially written as Γ �I φ. NL Sentence Table 1: De Se Belief in DC EC ∗ DC EC ∗ Jack believes that the per- B(jack, now, ∃x : Agent(named(x, “Jack”) ∧ rich(x))) son named “Jack” is rich. Jack believes of the person ∃x : Agent(named(x, “Jack”) ∧ B(jack, now, rich(x))) named “Jack” that he is rich. Jack believes that he himB(jack, now, rich(jack∗))) self is rich. I believe that I myself am �I B(I, now, rich(I∗))) rich. 6 Type De Dicto De Re Third-person De Se First-person De Se Conclusion & Next Steps Our future work falls into two categories: work aimed at increasing the expressive power of DC EC ∗ , and work aimed at capturing more cognitive processes. 6.1 Representation: Scoped Terms and Natural-Language Connectives The DC EC ∗ syntax presented in this paper lacks a way to represent general noun phrases and quantifiers. All the connectives in the current syntax derive from mathematical logic; for example, the conjunctions in “Jack fell down and Jill came tumbling after” and “x ≥ 3 and x ≤ 5” do not have the same meaning. Future work will incorporate such constructs and refine the proof theory to accommodate more cleanly these new elements. 6.2 Reasoning: Planning Integrated with Intensional Concepts Another thread of research is to focus on designing a planning framework (for an agent) that integrates all the intensional operators with planning in an event calculus-based planning formalism. This work is well underway, and some demonstrations should soon be possible. References Arkoudas, K. & Bringsjord, S. (2004), Metareasoning for Multi-agent Epistemic Logics, in ‘Proceedings of the Fifth International Conference on Computational Logic In Multi-Agent Systems (CLIMA 2004)’, Lisbon, Portugal, pp. 50–65. URL: http://kryten.mm.rpi.edu/arkoudas.bringsjord.clima.crc.pdf Arkoudas, K. & Bringsjord, S. (2009), ‘Propositional Attitudes and Causation’, International Journal of Software and Informatics 3(1), 47–65. URL: http://kryten.mm.rpi.edu/PRICAI w sequentcalc 041709.pdf Baader, F., Calvanese, D. & McGuinness, D., eds (2007), The Description Logic Handbook: Theory, Implementation (Second Edition), Cambridge University Press, Cambridge, UK. Bringsjord, S. (2008), Declarative/Logic-Based Cognitive Modeling, in R. Sun, ed., ‘The Handbook of Computational Psychology’, Cambridge University Press, Cambridge, UK, pp. 127– 169. URL: http://kryten.mm.rpi.edu/sb lccm ab-toc 031607.pdf Bringsjord, S. & Govindarajulu, N. (2013), Toward a Modern Geography of Minds, Machines, and Math, in V. C. Müller, ed., ‘Philosophy and Theory of Artificial Intelligence’, Vol. 5 of Studies in Applied Philosophy, Epistemology and Rational Ethics, Springer Berlin Heidelberg, pp. 151–165. URL: http://www.springerlink.com/content/hg712w4l23523xw5 Bringsjord, S. & Govindarajulu, N. S. (2012), ‘Given the web, what is intelligence, really?’, Metaphilosophy 43(4), 464–479. URL: http://dx.doi.org/10.1111/j.1467-9973.2012.01760.x Bringsjord, S. & Licato, J. (2012), Psychometric Artificial General Intelligence: The PiagetMacGyver Room, in P. Wang & B. Goertzel, eds, ‘Theoretical Foundations of Artificial General Intelligence’, Atlantis Press. URL: http://kryten.mm.rpi.edu/Bringsjord Licato PAGI 071512.pdf Castañeda, H.-N. (1999), ‘He’: A Study in the Logic of Self-Consciousness, in J. G. Hart & T. Kapitan, eds, ‘The Phenomeno-Logic of the I: Essays on Self-Consciousness’, Indiana University Press, 601 North Morton Street, Bloomington, Indiana 4704-3797 USA. Dowty, D., Wall, R. & Peters, S. (1981), Introduction to Montague Semantics, D. Reidel, Dordrecht, The Netherlands. Francez, N. & Dyckhoff, R. (2010), ‘Proof-theoretic Semantics for a Natural Language Fragment’, Linguistics and philosophy 33(6), 447–477. Janssen, T. M. V. (2012), Montague semantics, in E. N. Zalta, ed., ‘The Stanford Encyclopedia of Philosophy’, winter 2012 edn. Jaśkowski, S. (1934), ‘On the Rules of Suppositions in Formal Logic’, Studia Logica 1, 5–32. Kamp, H. & Reyle, U. (1993), From Discourse to Logic: Introduction to Model-theoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory, 1 edn, Springer. Kwiatkowski, T., Zettlemoyer, L., Goldwater, S. & Steedman, M. (2010), Inducing Probabilistic CCG Grammars from Logical form with Higher-order Unification, in ‘Proceedings of the 2010 conference on empirical methods in natural language processing’, Association for Computational Linguistics, pp. 1223–1233. Licato, J., Bringsjord, S. & Hummel, J. E. (2012), Exploring the Role of Analogico-Deductive Reasoning in the Balance-Beam Task, in ‘Rethinking Cognitive Development: Proceedings of the 42nd Annual Meeting of the Jean Piaget Society’, Toronto, Canada. URL: https://docs.google.com/open?id=0B1S661sacQp6NDJ0YzVXajJMWVU Licato, J., Govindarajulu, N. S., Bringsjord, S., Pomeranz, M. & Gittelson, L. (2013), ‘Analogicodeductive Generation of Gödel’s First Incompleteness Theorem from the Liar Paradox’, Proceedings of the 23rd Annual International Joint Conference on Artificial Intelligence (IJCAI– 13) . Manzano, M. (1996), Extensions of First Order Logic, Cambridge University Press, Cambridge, UK. Montague, R. (1974), Formal Philosophy: Selected Papers of Richard Montague, Yale University Press, New Haven, CT. Mueller, E. (2006), Commonsense Reasoning, Morgan Kaufmann, San Francisco, CA. Russell, S. & Norvig, P. (2009), Artificial Intelligence: A Modern Approach, Prentice Hall, Upper Saddle River, NJ. Third edition. Van Lambalgen, M. & Hamm, F. (2005), The Proper Treatment of Events, Vol. 6, Blackwell Publishing. Formal Magic for Analogies Ulf Krumnack, Ahmed Abdel-Fattah, and Kai-Uwe Kühnberger Institute of Cognitive Science, University of Osnabrück, Germany Abstract. During the last decades, a number of different approaches to model analogies and analogical reasoning have been proposed, that apply different knowledge representation and mapping strategies. Nevertheless, analogies still seem to be hard to grasp from a formal perspective, with no known treatment in the literature of their formal semantics. In this paper we present a universal framework that allows to analyze the syntax and the semantics of analogies in a logical setting without committing ourselves to a specific type of logic. We then apply these ideas by considering an analogy model that is based on classical first-order logic. 1 Introduction There is no uncontroversial theory of the semantics of analogies (although several models for ‘computing’ analogies have been proposed). Even worse, for all established frameworks even an endeavor to find a semantics of the underlying computations cannot be found. As a consequence of these deficiencies, the desired generalization capabilities of appropriate AI systems are currently far from being reachable. From an AI perspective, it would be desirable to have at least a model that could be combined with standard mechanisms of knowledge representation and reasoning. In order to bridge the gap between algorithmic approaches for analogical reasoning and the denotational semantics underlying these algorithms, the usage of analogy mechanisms for AI systems is proposed, in this paper, in an abstract way. Semantic issues of analogical relations are investigated and a model theory of analogical transfers is specified. 2 Modeling Analogies: The Setting Our approach aims at achieving an abstract syntactic and semantic representation of modeling analogies in an arbitrary logical framework. The approach is loosely discussed in this section and formalized in the next. By an arbitrary logical framework we refer to a general framework that represents knowledge using a logic language (e.g. predicate first-order logic). There seems to be a generally non-controversial core interpretation of analogies in the literature: Analogical relations can be established between a well-known domain, the source, and a formerly unknown domain, the target, without taking much input data (examples) into account. It is rather the case that a conceptualization of the source domain is sufficient to generate knowledge about the target domain, which can be achieved by associating attributes and relations of the source and target domains. New “conceptual entities” can be productively introduced in the target domain by the projection (or the transfer) of attributes and relations from the source to the target, which allow the performance of reasoning processes to take place in the target domain. That is, analogical relations between the entities in the input domains result from analogical transfer, which identifies their common (structural) aspects, and exports some of these aspects from the source to the target, in order for the treatment of the target (as being similar to the source) to consistently come about. The common aspects of the input domains are identified, and made explicit, in a generalized domain that captures the common parts of the input domains. This generalized domain can be thought of as being the mutual “generalization” sub-domain of both input domains. Figure 1 depicts this overall idea using S and T as source and target domains, respectively, where m represents an analogical relation between them, and the generalization, G, represents the common parts of S and T . Generalization (G) abstraction Source (S) analogical transfer Target (T ) m Fig. 1. An overall view of creating analogies. It is worth noting that the amount of “coverage” in analogy-making plays a role, since it basically reflects the limit to which parts of the domains may or may not be included in the generalization. Moreover, not only some parts of the domains may be irrelevant to each other, of course, but also the domains can be pragmatically irrelevant for a human reasoner. These issues are not discussed in this paper in further detail. Notice that the abstraction process results in a generalization of source and target. The inverse operation, namely to construct the input domains from the generalization can be modeled by substitutions and therefore result in specializations of the generalization. 3 A Formal Framework We will present our ideas using the theory of institutions by briefly recalling the central notions (the interested reader is referred to [1, 2]). An institution formalizes the intuitive notion of a logical system into a mathematical object. A particular system is constituted by a signature Σ, which gives rise to a syntax, formalized as a set of sentences Sen(Σ), and a semantics, formalized by a category of models Mod(Σ). Sentences and models are related by satisfaction |=Σ expressing which sentences hold in which models. A simple example of an institution can be given in well-known terms of first-order logic (FOL). In this case, the Σ-sentences Sen(Σ) corresponds to the set of all FOL formulas that can be built using symbols from a signature Σ. For each signature Σ the collection Mod(Σ) of all Σ-models corresponds in FOL to the collection of all possible interpretations of symbols from Σ. The Σ-models and Σ-sentences are related by the relation of Σ-satisfaction, which corresponds to the classical model theoretic satisfaction relation in FOL. A change in notation should not alter truth of formulae. This is formalized by assuming functoriality of the mappings Sen and Mod in the following way: given two signatures Σ1 and Σ2 , and a signature morphism, i.e. a structure preserving mapping between signatures, σ : Σ1 → Σ2 , then this will induce a mapping Sen(σ) : Sen(Σ1 ) → Sen(Σ2 ) on the syntactic level and a contravariant mapping Mod(σ) : Mod(Σ2 ) → Mod(Σ1 ) on the semantic level, such that satisfaction is preserved: Sen(σ)(ϕ) ∈ Sen(Σ2 ) ϕ ∈ Sen(Σ1 ) |=Σ1 Mod(σ)(m) ∈ Mod(Σ1 ) iff |=Σ2 (1) m ∈ Mod(Σ2 ) For our exposition, the notion of a signature morphism does not suffice, so we will make use of a more general concept.1 Definition 1. For any signature Σ of an institution, and any signature morphisms χ1 : Σ → Σ1 and χ2 : Σ → Σ2 , a general Σ-substitution ψχ1 :χ2 , depicted by Σ1 !! ψ " Σ2 !! # """ !! " χ1 !!! "" χ2 "" Σ consists of a pair #Sen(ψ), Mod(ψ)$ , where – Sen(ψ) : Sen(Σ1 ) → Sen(Σ2 ) is a function – Mod(ψ) : Mod(Σ2 ) → Mod(Σ1 ) is a functor such that both of them preserve Σ, i.e. the following diagrams commute: Mod(ψ) Sen(ψ) $ Sen(Σ2 ) Mod(Σ1 ) ( Mod(Σ2 ) Sen(Σ1 ) %## &$ %% %% && ## $ & $ % & ## %% & $$ & # $ % & Mod(χ1 ) % Sen(χ1 ) # $$ Sen(χ2 ) ' )&& Mod(χ2 ) Sen(Σ) Mod(Σ) and such that the following satisfaction condition holds: Mod(ψ)(m2 ) |= ρ1 if and only if m2 |= Sen(ψ)(ρ1 ) for each Σ2 -model m2 and each Σ1 -sentence ρ1 . General Σ-substitutions extend the idea of a signature morphism. Although, in general there does not need to be a mapping on the level of signatures between Σ1 und Σ2 , most general Σ-substitution considered in practice are induced by some form of signature mapping. Every signature morphism can be seen as general Σ-substitution, and many other mappings, like classical first-order substitutions, second-order substitutions (for FOL), and derived signature morphisms give rise to a general Σ-substitution. We now turn to modeling analogies in this framework. Analogy-making can be broadly characterized by the result of finding an analogical relation linking salient ‘conceptual entities’ of two (structured) domains to each other. The first of of these domains is designated as the “source” domain (the one which is standardly considered to be the richer domain including more available and accessible knowledge) and the other as the “target”. Analogical reasoning is then seen as the ability to treat the target as being similar, in one aspect or another, to the source, depending on their shared commonalities in relational structure or appearance. Once such an analogical mapping is discovered it can give rise the to a transfer of knowledge between the two domains. The basic idea in our approach is to view analogy-making as an “generalization process”: the association of corresponding ‘conceptual entities’ in the input domains gives 1 Based on [1, section 5.3] rise to a generalization capturing the commonalities. In other words, the two given input domains, source and target, have a common core that is given by specific instantiations for the generalization as depicted in figure 1. We assume that a source domain S and a target domain T are given using a logical formalism, i.e. within a common institution I. The signatures for these domains are denoted by ΣS and ΣT respectively. We will further allow the use of common symbols from a background signature Σ. To spell out the above-mentioned scheme of analogy by generalization, we introduce a further signature ΣG that provides the generalized symbols: Definition 2. Given two signatures ΣS and ΣT over a common signature Σ, a generalization גis defined to be a triple #ΣG , σ, τ $, consisting of a signature ΣG , and general Σ-substitutions σ and τ as indicated in the following diagram: Σ* G σ ΣS ( + τ , Σ $ ΣT As a Σ-substitution is defined as a pair of mappings on sentence and model level, every generalization gives rise to the following diagrams: Sen(ΣG ) * %% %%Sen(τ ) && & %% & & %% & & %. -&& $ Sen(ΣT ) Sen(ΣS ) ( Sen(Σ) Sen(σ) Mod(ΣG ) 1(( '0 ((Mod(τ ) (( ' ' (( ' ( '' / $ Mod(Σ) ( Mod(ΣS ) Mod(ΣT ) Mod(σ)''' Furthermore, for every ΣG -sentence ρ, every ΣS -model mS and every ΣT -model mT , the following satisfiability conditions hold: ρ Sen(σ)(ρ) ΣS |= / mS iff ΣG |= Sen(τ )(ρ) ρ and / Mod(σ)(mS ) iff / Mod(τ )(mT ) ΣG |= ΣT |= / mT In this setting, we can introduce an analogical relation on the level of sentences as well as on the level of models. ℵ ℵ Definition 3. An analogy ℵ is defined as a pair #∼Sen , ∼Mod $ of relations such that for every pair of sentences s ∈ Sen(ΣS ), t ∈ Sen(ΣT ), and every pair of models ℵ ℵ mS ∈ Mod(ΣS ), mT ∈ Mod(ΣT ) with s ∼Sen t and mS ∼Mod mT it holds mS |= s iff mT |= t A direct consequence from this definition is Fact 1 Every generalization גgives rise to an analogy ℵ in the following way: – On the level of sentences, for every s ∈ Sen(ΣS ) and t ∈ Sen(ΣT ) the relation ℵ s ∼Sen t holds iff there exists a g ∈ Sen(ΣG ) such that Sen(σ)(g) = s and Sen(τ )(g) = t. – On the semantic level, for every mS ∈ Mod(ΣS ) and mT ∈ Mod(ΣT ) the ℵ relation mS ∼Mod mT holds iff Mod(σ)(mS ) = Mod(τ )(mT ). Based on these notions, we now consider the domain theories, i.e. sets of sentences ThS ⊆ Sen(ΣS ) and ThT ⊆ Sen(ΣT ), used to model the source and target domain respectively. For a given generalization גwe say that a set of ΣG -sentences Th is a ג-generalization of ThS and ThT , if Sen(σ)(Th) ⊆ ThS and Sen(τ )(Th) ⊆ ThT . Fact 2 For every generalization גand every pair of theories ThS , ThT there exists a maximal (with respect to set inclusion) ג-generalization ThG . We call that maximal ThG the ℵ-generalization of ThS and ThT for the analogy ℵ induced by ג. The sentences Sen(σ)(ThG ) and Sen(σ)(ThG ) are the parts of the domains that are covered by the analogy. Coverage comprises the idea of a “degree of association” of sub-theories between theories ThS and ThT and plays an important role in current approaches for analogy-making. Fact 3 For every sentence s ∈ Sen(ΣS ) that is covered by an anlogy ℵ, there exists a ℵ sentence t ∈ Sen(ΣT ) such that s ∼Sen t and vice versa. 4 An Example Section 3 proposes a general framework applicable to arbitrary analogy-making approaches which are based on a broad range of underlying logical systems. As long as common expressions in the source and target domain are associated via the analogy ℵ, a generalization for both domains is computed, and source and target can be recovered from the generalization using general Σ-substitutions, an analogy ℵ can be formally described on the syntactic and semantic level. Heuristic-Driven Theory Projection (HDTP) [3] is an example of an analogy engine that instantiates this general formal framework described in Section 3 specifically for first-order logic (FOL). HDTP provides an explicit generalization of two domains specified as theories in (many-sorted) FOL as a by-product of establishing an analogy.2 HDTP proceeds in two phases: in the mapping phase, the source and target domains are compared to find structural commonalities, and a generalized description is created, which subsumes the matching parts of both domains. In the transfer phase, unmatched knowledge in the source domain can be mapped to the target domain to establish new hypotheses. The overall idea of establishing an analogy between two domains in the HDTP framework is nicely covered in Figure 1. For our current purposes, we will only consider the mapping mechanism in more detail. The mapping is achieved via a generalization process, in which pairs of formulas from the source and target domain are anti-unified resulting in a generalized theory that reflects common aspects of the two domains. Formulas that are generalized to the same element in the generalized theory are considered to be analogically related. The generalized theory can be projected into the original domains by substitutions which are computed during anti-unification. We say that a domain formula is covered by the analogy, if it is within the image of this projection, otherwise it is uncovered. In analogy making, the analogical relation is used in the transfer phase to translate additional uncovered knowledge from the source to the target domain. 2 To improve readability we omit the sortal specifications of terms in this paper. Technically, HDTP is based on restricted higher-order anti-unification [3] which is defined on four basic types of substitutions.3 In order to allow a mild form of higherorder anti-unification, we extend classical FOL terms by introducing variables that can take arguments: for every natural number n we assume an infinite set Vn of variables with arity n Here we explicitly allow the case n = 0 with V0 being the set of FOL variables. In this setting, a term is either a first-order or a higher-order term, i.e. an expression of the form F (t1 , . . . , tn ) with F ∈ Vn and terms t1 , . . . , tn . Definition 4. We define the following set of basic substitutions: ! 1. A renaming ρF,F replaces a variable F ∈ Vn by another variable F " ∈ Vn of the same argument structure: ρF,F ! F (t1 , . . . , tn ) −−−→ F " (t1 , . . . , tn ). 2. A fixation φF c replaces a variable F ∈ Vn by a function symbol f of the same argument structure: φF f F (t1 , . . . , tn ) −−→ f (t1 , . . . , tn ). ! 3. An argument insertion ιF,F G,i with 0 ≤ i ≤ n, F ∈ Vn , G ∈ Vk with k ≤ n − i, and F " ∈ Vn−k+1 is defined by ιF,F G,i ! F (t1 , . . . , tn ) −−−→ F " (t1 , . . . , ti , G(ti+1 , . . . , ti+k ), ti+k+1 , . . . , tn ). ! 4. A permutation παF,F with F, F " ∈ Vn and bijective α : {1, . . . , n} → {1, . . . , n} rearranges the arguments of a term: π F,F ! α F (t1 , . . . , tn ) −− −→ F " (tα(1) , . . . , tα(n) ). For generalizing complex terms, we can successively apply several substitutions: To receive a non-ambiguous set of substitutions we apply the basic substitutions in the order renaming, argument insertion, permutation, and finally fixation. We will call any composition of basic substitutions a (higher-order) substitution and write t → t" , if there exists a sequence of basic substitutions that transforms t into t" . We will call t" an (higher-order) instance of t, and t an (higher-order) anti-instance of t" . HDTP can now be interpreted using the concepts described in Section 3. The underlying institution I is the institution FOL, i.e. the category of signatures Sign is the category of logical first-order signatures, the collection of Σ-sentences is the class of all first-order formulas, and the collection Mod(Σ) of all Σ-models is the collection of all possible interpretations of symbols from Σ. The basic substitutions from Definition 4 give rise to mappings on the syntactic and semantic level.4 On the syntactic level, a basic substitution replaces function and predicate symbols in the way described above, inducing a function on the set of FOL 3 4 Restricted higher-order anti-unification resembles to a certain extent strategies proposed in the context of ontology repair plans [4]. In fact, the basic substitutions are special cases of second-order substitutions in first-order logic, which can also be described as derived signature morphisms, cf. [1, 101]. formulas. For example, given a FOL signature Σ = #Φ, Π$ with function symbols Φ ! and predicate symbols Π, a renaming ρF,F induces a function ! Sen(ρF,F ) : Sen(Φ ∪ {F }, Π) → Sen(Φ ∪ {F " }, Π) On the semantic level, we get a mapping in the opposite direction: given a model for #Φ ∪ {F " }, Π$, we can interpret #Φ ∪ {F }, Π$-formulas, by first translating them via ! ! ρF,F and using the given model. Hence ρF,F induces a mapping of models ! Mod(ρF,F ) : Mod(Φ ∪ {F " }, Π) → Mod(Φ ∪ {F }, Π). It is easy to see, that the satisfiability condition is fulfilled: ) $ Sen(ρF,F ! )(s) s |=$Φ∪{F ! },Π % |="Φ∪{F },Π$ / ! Mod(ρF,F )(m) ( ) / m The same holds for the other basic substitutions as can be easily verified. Hence every HDTP substitution t → t" , as a composition of basic substitutions resulting in an anti-instance t of t" , gives rise to a general substitution in the sense of Definition 1. Therefore the operations HDTP performs in computing an analogical relation between a given source and target domain fit into the framework presented in Section 3. Furthermore, a syntactic and semantic interpretation of the HDTP computation is provided by the presented approach. This allows to adopt the notions of analogical relation and coverage introduced there. 5 Conclusion Establishing an analogy between two input domains by an abstraction process does not only agree with the utilization of cognitive capabilities, but can also be given a sensible interpretation on the semantic level, and implemented in logic-based AI systems. As the paper shows, the generalized theory and the analogical relation (established on a purely syntactic basis) can be connected to model theoretic relations on the semantic level in a coherent manner. We think that the analysis of the model theoretic semantics of analogies will be helpful for developing and improving computational models for (or based on) analogical reasoning. A better understanding of this type of representation can help in modeling certain types of creativity as well as understanding and explaining a broad range of cognitive capacities of humans, as recently discussed in e.g. [5]. On the other hand, several interesting open questions remain. First, it seems to be straight forward to apply our approach to a logic based systems like Heuristic-Driven Theory Projection [3]. But the framework should be applied to other symbolic analogy models, such as, for example, SME [6]. Second, several approaches in the field of theorem proving and ontology repair systems include substitutions resulting in a change of the underlying signature of the respective theory [4]. Last but not least, the present paper sketches only the specification of the syntax and semantics of certain non-classical forms of reasoning in a FOL setting. A thorough examination of extensions of reasoning with respect to alternative logical systems (like higher-order logic, description logic, model logic, equational logic) remains a topic for the future. References 1. Diaconescu, R.: Institution-independent Model Theory. Studies in Universal Logic. Birkhäuser, Basel (2008) 2. Goguen, J.A., Burstall, R.M.: Institutions: abstract model theory for specification and programming. Journal of the ACM 39(1) (January 1992) 95–146 3. Schwering, A., Krumnack, U., Kühnberger, K.U., Gust, H.: Syntactic principles of HeuristicDriven Theory Projection. Special Issue on Analogies - Integrating Cognitive Abilities. In: Journal of Cognitive Systems Research 10(3) (2009) 251–269 4. McNeill, F., Bundy, A.: Dynamic, Automatic, First-Order Ontology Repair by Diagnosis of Failed Plan Execution. International Journal of Semantic Web and Information Systems 3 (2007) 1–35 5. Abdel-Fattah, A., Besold, T., Kühnberger, K.U.: Creativity, cognitive mechanisms, and logic. In Bach, J., Goertzel, B., Iklé, M., eds.: Proc. of the 5th Conference on AGI, Oxford. Volume 7716 of Lecture Notes in Computer Science. (2012) 1–10 6. Falkenhainer, B., Forbus, K., Gentner, D.: The Structure-Mapping Engine: Algorithm and Example. Artificial Intelligence 41 (1989) 1–63 Authors: Ulf Krumnack Ahmed Abdel-Fattah Kai-Uwe Kühnberger Artificial Intelligence Research Group, Institute of Cognitive Science, University of Osnabrück, Albrechtstr. 28, 49076 Osnabrück, Germany. Formal Models in AGI Research Pei Wang Temple University, Philadelphia PA 19122, USA http://www.cis.temple.edu/∼pwang/ Abstract. Formal models are necessary for AGI systems, though it does not mean that any formal model is suitable. This position paper argues that the dominating formal models in the field, namely logical models and computational models, can be misleading. What AGI really needs are formal models that are based on realistic assumptions on the capacity of the system and the nature of its working environment. 1 The Power and Limit of Formal Models The need for formal models for AGI research is not a novel topic. For example, AGI-09 had a workshop titled “Toward a Serious Computational Science of Intelligence” [1]. In [2], I proposed the opinion that a complete A(G)I work should consist of (1) a theory of intelligence, expressed in a natural language, (2) a formal model of the theory, expressed in a symbolic language, and (3) a computer implementation of the model, expressed in a programming language. Though the necessity of (1) and (3) are obvious, there is a large number of AGI projects without a clearly specified formal model. Such projects are often described and carried out according to the common practice of software engineering. If an AGI system is eventually built as a computer system with software and hardware, why bother to have a formal model as an intermediate step between the conceptual design and the physical implementation? As I argued in [3], formalization improves a theoretical model by disambiguating (though not completely) its notions and statements. In particular for AGI, a formal model tends to be domain independent, with its notions applicable to various domains by giving the symbols different interpretations. Though it is possible to skip formalization, such a practice often mixes the conceptual issues and the implementational issues, thus increasing the complexity of a system’s design and development. However, to overemphasize the importance of formalization for AGI may lead to the other extreme, that is, to evaluate a formal model for its own sake, without considering its empirical justification as a model of intelligence, or its feasibility of being implemented in a computer system. Though the rigor and elegance of a model are highly desired, they are still secondary when compared with the correctness and applicability of the fundamental assumptions of the model. A mathematical theory may have many nice properties and may solve many practical problems in various fields, but this does not necessarily mean that it will be equally useful for AGI. Actually it is my conjecture that a major 2 ! reason for the lack of rapid progress in this field is the dominance of the wrong formal models, in particular, those based on mathematical logic, the theory of computation, and probability theory. In this paper, I summarize my arguments against certain logical models and computational models in AGI. 2 Logical Models and AGI As I argued in [4, 2], mathematical logic was established to provide a logical foundation for mathematics, by formalizing the valid inference patterns in theorem proving. However, “theorem proving” is very different from commonsense reasoning, and this conclusion has been reached by many logicians and AI researchers. Consequently, various non-classical logics and reasoning models have been proposed, by revising or extending traditional mathematical logic [5, 6]. Even so, the following fundamental assumptions in classical logic are still often taken for granted: Correspondence theory of truth: The truth-value of a statement indicates the extent to which the statement corresponds to an objective fact. Validity as truth-preserving: An inference rule is valid if and only if it derives true conclusions from true premises. My own AGI project NARS is a reasoning system that rejects both of the above assumptions. Instead, they are replaced by two new assumptions: Empirical theory of truth: The truth-value of a statement indicates the extent to which the statement agrees with the system’s experience. Validity as evidence-preserving: An inference rule is valid if and only if its conclusion is supported by the evidence provided by its premises. Based on the above assumptions, as well as the assumption that an intelligent system should be adaptive and can work with insufficient knowledge and resources, NARS is designed, which implements a formal logic [7, 2, 8]. NARS fundamentally differs from mathematical logic, since it is designed to work in realistic situations, while the latter is for idealized situations. NARS consistently handles many issues addressed in non-classical logics: Uncertainty: NARS represents several types of uncertainty, including randomness, fuzziness, ignorance, inconsistency, etc., altogether as the effects of various forms of negative or future evidence. Ampliativity: Beside deduction, NARS also carries out various types of nondeductive inference, such as induction, abduction, analogy, and other types of inference that produce “ampliative” conclusions. Openness: NARS is always open to new evidence, which may challenge the previous beliefs of the system, and therefore lead to belief revisions and conceptual changes. Relevance: The inference rules not only demand truth-value relationships between the premises and the conclusions, but also semantic relationships, that is, their contents must be related. !3 Both in logic and in AI, the above issues are usually addressed separately, and a new logic is typically built by extending or revising a single aspect of classical logic, while leaving the other aspects unchanged [6, 9, 10]. NARS takes a different approach, by treating the issues as coming from a common root, that is, the assumption on the insufficiency of knowledge and resources [4, 2]. 3 Computational Models and AGI Since an AGI will eventually be implemented in a computer system, it is often taken for granted that all processes in the system should be designed and analyzed according to the theory of computation. Concretely, it means the problem the system needs to solve will be defined as a computation, and its solution as an algorithm that can be implemented in a computer [11]. I have argued previously that such a conceptual framework is not suitable for AI at the problem-solving level [7, 2]. Like mathematical logic, the theory of computation also came from the study of problem solving in mathematics, where the “problems” are abstracted from their empirical originals, and the “solutions” are expected to be conclusively correct (i.e., cannot be refuted or revised later), context independent (i.e., having nothing to do with where and when the problem appears), and expense irrelevant (i.e., having nothing to do with how much time has been spent on producing it). Therefore, problem solving in mathematics can be considered as “time-free” and repeatable. When dealing with abstract problems, such an attitude is justifiable and even preferred — mathematical solutions should be universally applicable to different places in different times. However, the problem-solving processes in intelligence and cognition are different, where neither the problems nor the solutions are time-free or accurately repeatable. In practical situations, most problems are directly or indirected related to predictions of future events, and therefore have time requirements attached. In other words, “solving time” is part of the problem, and a “solution” coming too late will not qualify as a solution at all. On the other hand, the solutions for a problem usually depend on the system’s history and the current context. This dependency comes from the adaptive nature of the system and the real-time requirement. By definition, in an adaptive system the occurrences of the same problem get different solutions, which are supposed to be better and better in quality. Therefore, each occurrence of a problem is unique, if the system’s state is taken into consideration as a factor. For an adaptive system, usually in its lifetime its internal states never repeat, and nor does its external environment. Consequently, its problem-solving processes cannot be accurately repeatable. Though it is still possible to focus on the relative stable aspects of an intelligent system, so as to specify its stimulus–response relationship as a function, in the sense that the same (immediate) input always lead to the same (immediate) output. However, such a treatment excludes some of the prominent features of intelligence, such as its adaptivity, originality, and flexibility. 4 ! For instance, from the very beginning of AI, “learning” has been recognized by many researchers as a central aspect of intelligence. However, the mainstream “machine learning” research has been carried out in the framework of computation: The objective of learning is to get a function. At the object level, though during the learning process the same problem instance gets multiple solutions with improving quality, it is usually expected that the problem-solution relation will eventually converge to a function, The learning process follows an algorithm. At the meta-level, each type of learning is usually defined as a computation, and follows an algorithm, with the training data as input, and the learned function as output. The human learning process does not fit into this framework, because it is usually open-ended, and does not necessarily converge into a stable function that maps problems into solutions. Even after intensive training in a certain domain, an expert can still keep the flexibility and adaptivity when solving problems. Also, human learning processes do not follow fixed procedures, because such processes are usually not accurately predictable or repeatable. The above conclusions do not mean that intelligence has nothing to do with computation. At a certain level of description, the activities of an intelligent system can be analyzed as consisting of many “basic steps”, each of them is repeatable and can be specified as a computation following a fixed algorithm. It is just that a problem-solving process typically consists of many such steps, and its composition depends on many factors that are usually not repeated during the system’s life cycle. Such a formal model is provided in NARS [7, 2, 8]. As a reasoning system, NARS solves each problem by a sequence of inference steps, where each step is a simple computation, but since the steps are linked together at run time according to many ever-changing factors, the problem-solving process cannot be considered as a computation, since it is not repeatable. Here I want to argue that “intelligence” should be formalized differently from “computation”. As far as time is concerned, the differences are: The time-dependency of problem. In computation, a problem-solving process can take an arbitrarily long time, as long as it is finite. In intelligence. a problem-solving process is always under a time pressure, though the time requirement is not necessarily represented as a hard deadline. In general, the utility value of a solution decreases over time, and a solution may lose most of its utility if it is found too late. When time-pressure changes, the problem is also more or less changed. The time-dependency of solution. In computation, the correctness of a solution has nothing to do with when the problem appears, but in intelligence, it does. Whether a solution is reasonable should be judged not only according to the problem, but also the available knowledge and resources at the moment. A reasonable solution obtained by a student in an emergency may not be reasonable when provided by an expert after a long deliberation. 5 ! To implement such a model, NARS is designed in the following way: – Each inference task has a priority-value attached to reflect the (relatively defined) urgency for it to be processed. Similarly, each belief and concept in the system has a priority-value attached to reflect its importance at the moment. All these values are adjusted by the system according to its experience. – The selection of inference rules is data-driven, decided by the task and belief winning the system’s attention at the moment. Since the selection of task and belief is context-sensitive, so is the overall inference process. In this way, it is possible to implement a non-computable process using computable steps [2]. 4 Summary AGI research needs formal models of intelligence and cognition. Since all the existing models were designed for other purposes, they should not be directly applied without fundamental revision. Here the key issue is not in the specific features of a model, but in the basic assumptions behind it. More effort should be put into the developing of new formal models that satisfy the requirements of AGI, though the task is difficult and the result will not be perfect soon. References 1. Bringsjord, S., Sundar G, N.: Toward a serious computational science of intelligence (2009) Call for Papers for an AGI 2010 Workshop. 2. Wang, P.: Rigid Flexibility: The Logic of Intelligence. Springer, Dordrecht (2006) 3. Wang, P.: Theories of artificial intelligence – meta-theoretical considerations. In Wang, P., Goertzel, B., eds.: Theoretical Foundations of Artificial General Intelligence. Atlantis Press, Paris (2012) 305–323 4. Wang, P.: Cognitive logic versus mathematical logic. In: Lecture notes of the Third International Seminar on Logic and Cognition, Guangzhou (2004) Full text available online. 5. McCarthy, J.: Artificial intelligence, logic and formalizing common sense. In Thomason, R.H., ed.: Philosophical Logic and Artificial Intelligence. Kluwer, Dordrecht (1989) 161–190 6. Haack, S.: Deviant Logic, Fuzzy Logic: Beyond the Formalism. University of Chicago Press, Chicago Press (1996) 7. Wang, P.: Non-Axiomatic Reasoning System: Exploring the Essence of Intelligence. PhD thesis, Indiana University (1995) 8. Wang, P.: Non-Axiomatic Logic: A Model of Intelligent Reasoning. World Scientific, Singapore (2013). 9. Gabbay, D.M.: Logic for Artificial Intelligence and Information Technology. College Publications, London (2007) 10. Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach. 3rd edn. Prentice Hall, Upper Saddle River, New Jersey (2010) 11. Marr, D.: Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman & Co., San Francisco (1982) !"# $%&'()*# +,&-./# -0# 1&*)2(3(2/# -0# 45"'2(-"6)"# 7"2&.%.&2)2(-"# 2-# 82,&# 4)(*5.&# -0#9:5(3)*&"2#$5;<2(252(-"#=.("'(%*&>?# #$%&'&()!*%(! ! @;<2.)'2A#+,!-./!0/'%-$1$-,!&2!23(4-$&( 56/4$%'!7./&0,!&2!8/'%-$1$-,!&2!93(4-$&(!:5789;! ! $<!&(',! 3</23'!2&0!-./!-03-.=23(4-$&(!2&0>/?!@,!23(4-$&(!@$(%0,!4&((/4-$1/<!$(!4'%<<$4!60&6&<$-$&(%'!'&)$4! /A3$1%'/(-',! -0%(<2&0>/?! &(/! -&! &(/! @,! -B&=1%'3/?! (&(=-03-.=23(4-$&(! 2&0>/?! @,! (&(=23(4-$&(! 3(%0,! &6/0%-&0<C! D&0/&1/0E! @,! -./! 5789E! F-./! 2%$'30/! &2! -./! /A3$1%'/(-! <3@<-$-3-$&(! 60$(4$6'/G! B.$4.!%66/%0<!$(!6.$'&<&6.$4%'!'&)$4E!%(%',-$4!6.$'&<&6., %(?!$(2&0>%-$&(!<4$/(4/!-30(<!-&!@/!%! (%-30/!&2!4'%<<$4%'!'&)$4C! ! B&/#C-.D<E!HA3$1%'/(4/!53@<-$-3-$&(E!I&(=703-.=J%'3/?=93(4-$&(E!703-.=93(4-$&(E!5789C! ! FA# # 7"2.-D5'2(-"# 7./! >&?/0(! >%-./>%-$4%'! '&)$4! .%<! @/4&>/! %! 4&>>&(! -./&0/-$4%'! 2&3(?%-$&(! @/-B//(! $(2&0>%-$&(!<4$/(4/!%(?!>&?/0(!%(%',-$4!6.$'&<&6.,C!K&B/1/0E!-./!4'%<<$4%'!6%0-!&2!>&?/0(!'&)$4! <//><!(&-!1/0,!B/''!-&!?/%'!B$-.!F-./!2%$'30/!&2!-./!/A3$1%'/(-!<3@<-$-3-$&(!60$(4$6'/G!B.$4.!&2-/(! %66/%0<! $(! (%-30%'! '%()3%)/E! -0%?$-$&(%'! 6.$'&<&6.,E! A3%(-3>! -./&0,! %(?! 4&>63-/0! '%()3%)/C! 9&0! /L%>6'/M! ! FN'-.&3).!O!)/-!%!.$).!<%'%0,!@3-!O!%>!(&-!1/0,!0$4.C!O!6%,!%!-%L!%(?!?&(%-/!>34.!>&(/,C! O-!$<!&@'$)%-&0,!-.%-!O!6%,!%!-%LC!O-!$<!(&-!&@'$)%-&0,!-.%-!O!?&(%-/!>34.!>&(/,CG! 7./!'&)$4%'!</>%(-$4<!$(!-./!'%<-!2/B!</(-/(4/<!%0/M!FO!6%,!%!-%LG!$<!-03/P!FO!?&(%-/!>34.!>&(/,G!$<! -03/C!FO-!$<!&@'$)%-&0,!-.%-!O!6%,!%!-%LG!$<!-03/P!FO-!$<!&@'$)%-&0,!-.%-!O!?&(%-/!>34.!>&(/,G!$<!2%'</C! 7./0/2&0/E!-./0/!$<!%!2&0>%'!60&@'/>!QM!F-./!<%>/!-03-.=1%'3/!&2!-./!-B&!?$0/4-',!4&00/<6&(?$()! 4'%3</<!$(!-B&!4&>6'/L!</(-/(4/<!B.$4.!.%1/!?$22/0/(-!-03-.=1%'3/!/'</!$(!B.$4.!-./,!%0/!&(',!R&$(! @,!-./!<%>/!4&((/4-$1/GC! ! 534.! 60&@'/>! Q! %'<&! .%?! '&()! 63SS'/?! -./! 2&3(?/0! &2! >&?/0(! '&)$4! 90/)/C! 90/)/! %(?! .$<! 2&''&B/0<!-.&3).-!-.%-!B/!.%?!-&!3</!$(-/(<$&(%'!'&)$4!-&!<&'1/!$-C!7./!>&?/0(!>&?%'!'&)$4 %<!&(/! &2!-./!>&<-!-,6$4%'!%(?!>%-30/!T$(?!&2!$(-/(<$&(%'!'&)$4<!$(!(&(=4'%<<$4%'!'&)$4< B.$4.!.%<!.%?!(&-! &(',! %! '%0)/! %(?! 4&>6'/L! <,<-/><! $(! 2&3(?%-$&(E! @3-! %'<&! B$?/',! %66'$/?! $(! 4&>63-/0! <4$/(4/E! >&?/0(! %(%',-$4! 6.$'&<&6.,E! -./! $(-/060/-%-$&(! &2! A3%(-3>! >/4.%($4<! %(?! /1/(! >%(,! <&4$%'! <4$/(4/<!%(?!.3>%($-$/<C! ! U3$(/V<! %0)3>/(-! WXY! '$T/! -./! 60&@'/> %@&1/! A3/<-$&($()! -./! 1%'$?$-,! &2! /%0',! >&?/0(! >&?%'! '&)$4E! @3-! $-! <-$>3'%-/?! -./! ?/1/'&6>/(-! &2! <,<-/>! -./&0$/<! &2! >&?%'! '&)$4E! <34.! %<! -./! %??$-$&(! &2! -./! (/4/<<$-%-$&(! 03'/! $(! <,(-%L! %(?! -./! $(1/(-$&(! &2! 6&<<$@'/! B&0'?! </>%(-$4<C! K&B/1/0E! -./! 6&<<$@'/! B&0'?! </>%(-$4<! ?&/<! (&-! >%T/! %(,! >&?%'! 2&0>3'%! 4&00/<6&(?! -&! %! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Z ! #$%&'&()!*%(! [/6%0->/(-!&2!\.$'&<&6.,E!K3%S.&()!]($1/0<$-,!&2!54$/(4/!%(?!7/4.(&'&),! \C^C!+&L!X_``aXE!*3.%(E!K3@/$E!\C!8C!&2!b.$(%C! ! ! H=>%$' ! .B%(L'c"d_C4&>! !"! 2$0<-=&0?/0! 2&0>3'%! &(/! @,! &(/E! %(?! 1$4/! 1/0<%W"YC! D&0/&1/0E! <&>/! )/(/0%'$S/?! >&?%'! '&)$4! %<! ?/&(-$4! '&)$4! 60&?34/<! '&)$4%'! 6%0%?&L/<! B.$4.! %0/! ?$22$43'-! -&! <&'1/C! D%(,! <4.&'%0<! $(! $(2&0>%-$&(!<4$/(4/!%(?!%0-$2$4$%'!$(-/''$)/(4/!B&00,!%@&3-!-./!?$<3($-,!@/-B//(!>&?%'!'&)$4!%(?! 4'%<<$4%'!'&)$4C!N-!2$0<-E!F(/4/<<%0$',G!$(!>&?%'!'&)$4!$<!%!(&(=-03-.=23(4-$&(!&6/0%-/ @3-!4'%<<$4%'! '&)$4!$<!-03-.=23(4-$&(!'&)$4!&(',C! ! 7.$<!%0-$4'/!$<!%'&()!%!-&6=?&B(!%660&%4.E!20&>!@/.$(?!-./!4'%<<$4%'!60&6&<$-$&(!'&)$4!2&0>!&2! 3($1/0<%'$-,! %(?! $-<! 60&6/0! 4&>6'$%(4/! &2! (%-30%'! '%()3%)/E! <//T$()! -./! 0/'%-$&(<.$6! @/-B//(! %! 3($2&0>! >/-.&?! &0! 4&>60/./(<$1/! -./&0,! -&! /L6'&0/! -./! (&(=4'%<<$4%'! '&)$4! %(?! 4'%<<$4%'! '&)$4C! 5$(4/! 4'%<<$4%'! 60&6&<$-$&(%'! '&)$4! $<! -./! >&<-! >%-30/! %(?! -./! >&<-! <$>6'/! >&?/0(! '&)$4E! %(?! >&?%'! &6/0%-&0! $<! %! 3(%0,! &6/0%-&0E! <&! B/! 4&(<$?/0! -./! 0/'%-$&(<! @/-B//(! -03-.=1%'3/?! 23(4-$&(! 4&((/4-$1/! $(! 4'%<<$4%'! 60&6&<$-$&(%'! '&)$4! %(?! (&(=-03-.=1%'3/?! 23(4-$&(! &6/0%-/! $(! >&?%'! 60&6&<$-$&(%'!'&)$4!B$-.!-B&!-03-.=1%'3/?C!7./!-./&0,!%@&3-!-./!0/'%-$&(<.$6!$<!56/4$%'!7./&0,!&2! 8/'%-$1$-,!&2!93(4-$&(!:5789;C! 7./0/2&0/E!4&>6'/-/',!3($2$/?!-./!(%-30/!&2!>&?%'!'&)$4!%(?!&2!4'%<<$4%'!'&)$4!0/1/%'/?!@,!&(/! %0-$4'/!$<!(&-!0/%'$<-$4E!@3-!20&>!-./!F-./!2%$'30/!&2!/A3$1%'/(-!<3@<-$-3-$&(!60$(4$6'/fE!B/!4%(!230-./0! 4'%0$2,!>&?%'!60&6&<$-$&(%'!'&)$4!B$-.!-B&!-03-.=1%'3/?!60/2/00$()!%<!%(!$?$&>!-&!%!F20%)>/(-G!&2! 4'%<<$4%'!60&6&<$-$&(%'!'&)$4C! ! GA# # !3&.3(&H#-0#$+14# N!2/B!6%6/0<!%0/!?/?$4%-/?!-&!<,<-/>%-$4%'',!$(1/<-$)%-/!$(-&!-./!(%-30/!&2!-./!(&(=-03-.=23(4-$&(<! &0! &2! -./! (&(=-03-.=23(4-$&(%'! 4&((/4-$1/<! WgE! aE! eYC! H1/(! -.&3).! -./,! .%?! <-3?$/?! $(! ?/-%$'! -./! ?$22/0/(4/<! @/-B//(! (&(=-03-.=23(4-$&(! %(?! -03-.=23(4-$&(! &0! ?$22/0/(4/<! @/-B//(! (&(=-03-.=23(4-$&(%'!4&((/4-$1/!%(?!-03-.=23(4-$&(%'!4&((/4-$1/!W_E!dE!hY (&!&(/!0/4&)($S/<!-./! /A3$1%'/(-!-0%(<2&0>%-$&(!@/-B//(!-./>E!(&0!?&!>&0/!0/2/0!-&!)$1/!<34.!-0%(<2&0>%-$&(!%!-$).-',! 4'/%0!?/2$($-$&(C! ! GAF#I-"J$2.('2*/#K&0("&D#I-"J+.52,J45"'2(-"# N!23(4-$&(!$<!%!4&00/<6&(?$()!0/'%-$&(!&2!-B&!1%0$%@'/<!:&0!&2!-B&!</-<!&0!&2!-B&!2&0>3'%<;C!9&0!%! 1%0$%@'/!L!%(?!%!?/6/(?/(-!1%0$%@'/!,!@&-.!$(!-./$0!1%'3/?=?&>%$(E!$2!L!.%<!%!1%'3/!4&00/<6&(?$()! ,! .%1$()! &(/! %(?! &(',! &(/! 1%'3/E! -./(! -.$<! ,! $<! %! 23(4-$&(! &2! LC! N(&-./0! B%,E! B./(! ,! $<! (&-! %! 23(4-$&(!&2!LE!,!$<!%!23(4-$&(!&2!i,!%-!'/<-C!7.%-!$<!-./!0/'%-$1$-,!&2!(&(=23(4-$&(C!703-.=23(4-$&(!$<! -./! 23(4-$&(! $(! B.$4.! -./! ?&>%$(! &2! -./! 1%0$%@'/<! %(?! -./! ?/6/(?/(-! 1%0$%@'/! %0/! %''! -./! -03-.=1%'3/?=?&>%$(! &(',C! I&(=-03-.=23(4-$&(! $<! (&-! -./! 23(4-$&(! $(! B.$4.! -./! ?&>%$(<! &2! -./! 1%0$%@'/<! %(?! &2! -./! ?/6/(?/(-! 1%0$%@'/! %0/! %''! -./! -03-.=1%'3/?=?&>%$(! &(',C! O-! $(4'3?/<! -./! (&(=-03-.=1%'3/?! 23(4-$&(! %(?! -./! -03-.=1%'3/! &2! (&(=23(4-$&(C! 7.%-! $<! -&! <%,E! B/! <-3?,! -./! (&(=-03-.=23(4-$&(!$<!%4-3%'',! -B&=-03-.=1%'3/?!(&(=23(4-$&(C!K&B/1/0E!-&!.%1/!-./!6.$'&<&6.$4%'! '&)$4!4&00/<6&(?$()!-&!-./!-/L-E!B/!<-$''!4%''!$-!(&(=-03-.=1%'3/?!23(4-$&(C! j/(/0%'',E!B/!-.$(T!%(,!1%0$%@'/<!6!%(?!6"!B.$4.!%0/!$(?/6/(?/(-!&2!/%4.!&-./0!%0/!-B&!@%<$4! -03-.!23(4-$&(<C!K&B/1/0E!B./(!6!$<!-03/E!6"!$<!(&-!&(',!%!-03-.=1%'3/!/$-./0!-03/!&0!2%'</E!@3-!-B&! -03-.=1%'3/<!-03/!%(?!2%'</!4&00/<6&(?/?C!5&!6"!$<!(&-!%!-03-.!23(4-$&(!&2!6C!7.$<!$<!(&-!-&!<%,!B./(! 6!$<!-03/!-./(!6"!$<!@&-.!-03/!%(?!2%'</!%-!-./!<%>/!-$>/E!@3-!-&!<%,!-.%-!6!%(?!6"!.%<!%!0/'%-$&(!&2! (&(=-03-.!23(4-$&(!B.$4.!4%(!@/!.&'?!@&-.!B./(!6!$<!-03/!-./(!6"$<!-03/!%(?!B./(!6!$<!-03/!-./(!6"! $<!2%'</C!7.$<!<$-3%-$&(!$<!>34.!'$T/!-./!)/(/0%'!>%-./>%-$4%'!0/'%-$&(!&2!-B&!1%0$%@'/<!Lk,eC!7./! ! !e! 0/'%-$&(!Lk,e!$<!.&'?!@&-.!B./(!Lk"!-./(!,k"!%(?!B./(!Lk"!-./(!,k="C!^2!4&30</E!B./(!B/!-.$(T! %@&3-!6"!$<!%!(&(=-03-.!23(4-$&(!&2!6E!B/!&(',!3</!-./!2&0>%'!(%-30/!&2!>&?/0(!>%-./>%-$4%'!'&)$4! %(?!$-!$<!$(?/6/(?/(-!&(!B.%-!$<!-./!/L%4-!>/%($()!&2!-03/C! ! GAG#@#4-.L)*#K&0("(2(-"#-0#I-"J+.52,J45"'2(-"# l$T/! >&?%'! '&)$4E! 5789! &(',! %??<! %! <,>@&'! -&! '%()3%)/! &2! 4'%<<$4%'! 60&6&<$-$&(%'! '&)$4! <,<-/>! b\P!@3-!3('$T/!>&?%'!'&)$4E!-./!<,>@&'!5789!&(',!%??/?!4%(!@/!/(-$0/',!?/2$(/?!@,!-./!<,>@&'<! $(!b\C!7./0/2&0/E!-./!(/B!<,<-/>!5b\!$<!%4-3%'',!<-$''!b\C! O(!b\ @%<$4!<,>@&'<!%0/ ! "=" 6 6" 6eE!m 6(m ( nIoC! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! "=e ! "=_ ! :!E! ! ; ! O(!5b\E!@%<$4!<,>@&'!%??/?!$< ! "=X ! K ! K&0("(2(-"#FE!+%<$4!3(%0,!&6/0%-/!K! K6!k?/2!6"C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :";! M-.-**)./FJFE!&-./0!"g!3(%0,!&6/0%-&0<!4%(!@/!0/430<$1/',!?/2$($-/!@,!</-!n E!K!o!$(!7%@'/!"C! ! 7%@'/!"M!! 703-.!7%@'/!&2!+%<$4!](%0,!^6/0%-&0<! 6! 6 "! [ "! [ e! [ _! [ X! [ g! [ d! [ a! [ h! [ p! [! [! [ "`! ""! "e! [ "_! [ "X! [ "g! [ "d! "! "! "! "! "! "! `! "! "! "! `! `! `! "! `! `! `! `! "! `! "! "! "! `! "! "! `! `! "! "! `! `! "! `! `! `! `! "! "! "! `! "! "! `! "! `! "! `! "! `! `! "! `! `! `! `! "! `! "! "! "! `! `! "! `! "! "! `! `! `! "! `! 6 6"k! ! 6 k! Ka6 K! k! K6! 6! d 6! 6 6 6! K! 6! 6 6 ! 6 ! K! 6! 6 ! K! 6! K! 6! 6 6! K! 6! ! 6! K! 6 ! K! 6! K! 6! 6 \! 6 K! 6! 6 ! K! 6! K! 6! 6! K! K! K! K! K! K! K! "` "" "e "_ "X "g "d 6! 6! 6! 6! 6! ! 6! 6! K! 6! 6 K! K! 6 ! 6 ! ! ! K" 6! Ke 6! K_ 6! KX 6! ! M-.-**)./FJGE!KK K6k?/2!6(E!( !! (! ! ! Kg 6! Kd 6! Ka 6! Kh 6! Kp 6! nIoC! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! e !! !_! M-.-**)./FJNE!-./!</-!&2!d!<,>@&'<!n P! ! KP! ! 6P! ! :!E! ! ;!o!4%(!?/2$(/!%(,!60&6&<$-$&(!%(?!%(,! 2&0>3'%! $(! 5b\E! <$(4/! /1/0,! 1%0$%@'/! 4%(! @/! ?/2$(/?! @,! K! %(?! 6E!%(?! %(,! 2&0>3'%! $<! 2&0>/?! @,! 1%0$%@'/<!$(!b\!%(?!</-!n E!o!%(?! ! :!E! ! ;C! ! ! ! ! 9&0!4&(1/($/(4/E!B/!3</!</1/0%'!%3L$'$%0,!<,>@&'<!$(!5b\M! M-.-**)./FJN ! "6E! e6E!m%<!)/(/0%'!(&(=-03-.=23(4-$&(<!&2!6E!%(?!/%4.!&2!-./>!4%(!@/! /A3$1%'/(-!-0%(<2&0>/?!$(-&!%!4&>6&</?!-03-.=1%'3/?!23(4-$&(E!%(?!1$</!1$<%C! ! "NE! "+E! eNE! e+E! m%<! )/(/0%'! (&(=-03-.=23(4-$&(<! &2! %(,! 2&0>3'%! E! %(?! /%4.! &2! -./>!4%(!@/!/A3$1%'/(-!-0%(<2&0>/?!@&-.!$(-&!%!4&>6&</?!-03-.=1%'3/?!23(4-$&(!%(?!$(-&!%!)/(/0%'! (&(=-03-.=23(4-$&(!&2!6C!9&0!/L%>6'/E!K!$N!k?$:NE!N";E!K!$+!k?$:+E!+";C! 6! %<! %! (&(=-03-.=23(4-$&(! &2! 6! B.$4.! $<! /A3$1%'/(-! -&! %! -03-.! 23(4-$&(! %0@$-0%0,! </'/4-/?! 20&>!-./!%''!</-!&2!-03-.!23(4-$&(<!$(!b\C! N!%<!%(,!)/(/0%'!(&(=-03-.=23(4-$&(!&2!2&0>3'%!N!$(! 5b\C! */!0/)%0?!-./!-03-.=1%'3/?!23(4-$&(<!%<!%!<6/4$%'!T$(?!&2!(&(=-03-.=1%'3/?!23(4-$&(<C! ! K&0("(2(-"#GE!5/>%(-$4!0/60/</(-%-$&(! 578\! >%$(-%$(<! -./! </>%(-$4<! &2! b\! 3(4.%()/?P! &(',! -./! 0&B<! &2! -./! -03-.! -%@'/! %0/! /A3$1%'/(-! -&! 0/B0$-/! %<! %! <$()'/! 0&B! -03-.! 2&0> $(! B.$4.! FqG! 0/60/</(-<! -./! 2$0<-! ?$4.&-&>,! @/-B//(!-03/!%(?!(&-!-03/!:&0!2%'</;E!Fq!qG0/60/</(-<!-./!</4&(?!?$4.&-&>,E!%(?!<&!&(C! ! 9&0!/L%>6'/E!$2!-./!-03-.=1%'3/!&2!6!%<<$)(/?!-&!@/!F"q`GE!-./(!-./!-03-.=1%'3/!&2!K!6!%<<$)(/?! -&!@/!F"qq`q"qq`GE!%(?!-./!-03-.=1%'3/!&2!K!K!6!%<<$)(/?!-&!@/!F"qqq`qq"qqq`q"qqq`qq"qqq`GC! M-.-**)./#GJFE#I/)%-$&(! 7./!(/)%-$&(!&2!%(,!2&0>3'%!$<!(/)%-$&(!&2!/%4.!</6%0%-/!-03-.=1%'3/C! 9&0! /L%>6'/E! -./! </>%(-$4! 2&0>3'%! &2! N! $<M! `q"E! <&! -./! (/)%-$&(! &2! N! $<M! :`q";k `q "k"q`C! ! M-.-**)./F#GJGE#[$<R3(4-$&(! 9&0!%(,!-B&!2&0>3'%<!N!%(?!+E!N +!$<!-.%-!-./!?$<R3(4-$&(!2&0!/1/0,!6%0-$-$&(!@/-B//(!N +C! HC)C $2!Nk`q"qq`E!+k"qq`q" -./(!N +!k`q"qq` "qq`q"k!:` "qq`;q:"qq` ";k!::` ";qq:` `;;q::" ";qq!:` ";;k!"qq`q"qq"C! ! ! ! ! ! ! GAN#9O%*)")2(-"# # */! %0/! 2%>$'$%0! B$-.! -./! 4/0-%$(-,! &2! F-./! 3(4/0-%$(-,G! $(! A3%(-3>! >/4.%($4<C! U3%(-3>! >/4.%($4<! 4%((&-! 60/4$</',! 2&0/<//! -./! 60/?$4-$1/! 0/<3'-<! &2! <$()'/! A3%(-3>! >/%<30/>/(-! @3-! $-! 4%(!2&0/<//!-./!60&@%@$'$-,C! K&B/1/0E!-./!60&@%@$'$-,!&2!-./!0/<3'-<!&2!%!<$()'/! >/%<30/>/(-!-.%-! A3%(-3>!>/4.%($4<!4%(!2&0/<//!$<!4&>6'/-/',!?/2$($-/C!5-$''!-%T/!K!6!%<!%(!/L%>6'/C!9$0<-!&2!%''E! B./(! 6! $<! %! ?/2$($-/! -03-.! 1%'3/! `E! -./! -03/! 1%'3/! &2! K! 6! $<! (&-! %! ?/2$($-/! /$-./0! "&0! `! @3-! .%<! 3(4/0-%$(-,!&2!-03-.!1%'3/!"q`!:B/!4%''!$-!4%(!@/!-03/!4%(!@/!2%'</!@3-!(&-!@/!@&-.!-03/!%(?!2%'</!%-! -./!<%>/!-$>/;P!@3-!-./(E!-.%-!-03-.!1%'3/!&2!K!6!$<!"q`!$<!4&>6'/-/',!?/2$($-/C! \/0.%6<!<&>/!&(/!>%,!-.$(T!-.%-!-&!?/2$(/!-./!</-!&2!(&(=-03-.=1%'3/?!23(4-$&(<!4&>6'/-/',E! B/!.%1/!-&!3</!/1/0,!-03-.=23(4-$&(!2&0>/?!@,!&(/!&0!</1/0%'!1%0$%@'/<!%(?!4'%<<$4%'!4&((/4-$1/<! -&!4&00/<6&(?!&(/!@,!&(/!-&!%!3($A3/',!(&(=-03-.=1%'3/?!23(4-$&(C!9&0!/L%>6'/E! 6"e6!k!?/2!6 6" 6ee6!k!?/2!6 6e mC!I/1/0-./'/<<E!&@1$&3<',E!-.$<!?/2$($-$&(!$<!(&-!2$--$()!B$-.!(%-30%'! '%()3%)/C!9&0!/L%>6'/E!/1/(!$2!B/!@/'$/1/!-./!F&@'$)%-&0,G!$<!%!(&(!-03-.=23(4-$&(%'!4&((/4-$1/E! B/!(/1/0!-.$(T!-./0/!%0/!-B&!?$22/0/(-!4&((/4-$1/<!$(!FO-!$<!&@'$)%-&0,!-.%-!O!6%,!%!-%LG!%(?!FO-!$<! ! X! ! &@'$)%-&0,!-.%-!O!?&(%-/!>&(/,CG!7./!0/4300/(4/!&2!F&@'$)%-&0,G!$<!R3<-!-./!<%>/r! NA# # 7"2&.%.&2)2(-"<#2-#2,)(*5.&#-0#9:5(3)*&"2#$5;<2(252(-"#=.("'(%*&# # 5b\!$<!R3<-!%(&-./0!2&0>!&2!b\E!@3-!$-!B$''!2&0>!2/B!$>6&0-%(-!?/0$1/?!03'/<!-.%-!%0/!(&-!/%<,!-&! ?/-/4-!$(!-./!&0$)$(%'!b\!<,<-/>C! NAF#7L%-.2)"2#=.-%&.2(&<#-0#M*)<<(')*#=.-%-<(2(-")*#P-Q('#1&3&)*&D#R/#$+1=# # 15*&#FE!?$22/0/(-$%-$()!&6/0%-&0!B$-.!<36/06&<$-$&(!03'/! HC)C!KK6kK6"k!6e 6"kK6 ! 15*&#GE#I&(=4&>6'/-/',!<3@<-$-3-%@$'$-,!03'/! ! 9&0!%(,!(&(=-03-.=23(4-$&( +!0/6'%4/ NE!$(!)/(/0%'E!%44&0?$()!-&!-./!s=e!0%-./0!-.%(!-./! s="M! s="M!$2! s=e N!k?!:N N" -./(! +k?!:+E! ! N" C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :_;! ! $2 N!k?!:N N" -./(! +k?!:+ +" C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :X;! O(!2%4-E!B./( $<!0/?34/?!-&!@/!%!-03-.=1%'3/?!23(4-$&(!&6/0%-&0E!-./(!s=e!$<!0/?34/?!-&!s="P! O2!-./!'%<-!1%0$%@'/<!<.&B(!$(!N!%(?!$(!+!%0/!-./!<%>/E!-./(!s=e!$<!0/?34/?!-&!s="C! NAG#+,)(*5.&#-0#9:5(3)*&"2#$5;<2(252(-"#=.("'(%*&#("#'*)<<(')*#3(&H# N44&0?$()!-&!5789E!(&!>%--/0!B.%-!6"!/L%4-',!$<!$(!%!(&(=-03-.=1%'3/?!23(4-$&(!&2!6!-.%-!$<!K6k! 6"M!$2!6k"E!-./(!K!6k"q`P!$2!6k`E!-./(!K!6k"q`C!5&E!B/!4%(!)/-!%!0/430<$1/!?/2$($-$&(!2&0!K!K!6M!$2! K!6k"E!-./(!K!K!6k!K!:K!6;!k"q`P!$2!K!6k`E!-./(!K!K!6k!K!:K6;!k"q`C!O(!%??$-$&(E!B/!)/-!-.$<! ?/2$($-$&(M!$2!6k"E!-./(!K!K!6!k"qq`q"qq`P!$2!6k`E!-./(!KK!6!k"qq`q"qq`C! */! &0?/0! -B&! 2&0>3'%<E!Nk! 6E!+k!K6C!^@1$&3<',E!N! %(?! +! %0/! -B&! ?$22/0/(-! 2&0>3'%<C! +3-! B./(! 6k"! %(?! Nk6k"P! +kK6kKNk"q`E! K+k! KK6k"qq`q"qq`C! 7.$<! <.&B<! -.%-! -./! 0/<3'-! 4%(! @/! <%-$<2%4-&0,M!B./(!Nk+k"E!K!Nk" K!+k`C!7&!%44&>>&?%-/!-./!0/<3'-E!%44&0?$()!-&!5789E!K!K! 6!$(!<,(-%L!.%<!-&!@/!6eE!@3-!4%(!(&-!@/!6"C! ! 7&! <3>! 36E! B/! )/-! F%! 2%$'30/! &2! /A3$1%'/(-! <3@<-$-3-$&(! 60$(4$6'/G! R3<-! $(! 4'%<<$4%'! '&)$4! <,<-/>!5b\!&(',M!B./(!6k"E!6"k"!%(?!6ek`P!Nk6k!6"k+k"E!@3-!K!Nk!K!6k!6"k"E!K!+k!6ek!`C! 7.%-!$<!-&!<%,E!-./0/!$<!%!)0&36!&2!-03-.!%<<$)(>/(-E!N!%(?!+!%0/!/A3$1%'/(-E!@3-!B./(!B/!3</!+!-&! <3@<-$-3-/!N!&2!K!NE!-./(!K!N!%(?!K!+!%0/!(&-!/A3$1%'/(-C! ! ^2! 4&30</E! F%! 2%$'30/! &2! /A3$1%'/(-! <3@<-$-3-$&(! 60$(4$6'/G! ./0/! $<! -03',! F(&(=4'%<<$4%'GE! 2&0! FK! G! $<! (&-! 4'%<<$4%'! '&)$4! 4&((/4-$1/! @3-! %! (&(=-03-.=1%'3/?! 23(4-$&(%'! &6/0%-/E! @3-! B/! (//?(V-! $(-/(<$&(%'! '&)$4! &0! %(,! &-./0! (&(=4'%<<$4%'! '&)$4! -&! $(-/060/-! $-C! 7.%-! &(4/! 0/)%0?/?! %<! 63SS'$()! F(&(=4'%<<$4%'G! 4.%0%4-/0$<-$4<! 60/</(-/?! $(!(%-30%'! '%()3%)/!&0! $(! <4$/(-$2$4! '%()3%)/! -30(! &3-! -&! @/!F4'%<<$4%'G!&(/<!$(!2%4-C! NAN#7"2&.%.&2)2(-"#0-.#%.-;*&L#S# NANAF#!3&.3(&H#-0#L-D)*#%.-%-<(2(-"#*-Q('# # 7&!>%T/!-.$()<!<$>6'/!%(?!60/4$</E!B/!4&(<$?/0!%-&>$4!>&?%'!60&6&<$-$&(!&(',!$(!-.$<!%0-$4'/C!^2! 4&30</E!F G$<!&6/0%-/!F(/4/<<%0,GE!@3-!20&>!630/!2&0>%'!1$/BE!F G!$<!%!<,>@&'!4%''/?!F@&LG!&(',C! */! (//?(V-! -&! T(&B! -./! ?/-%$'! >/%(<! &2! -./! F G! $(! (%-30/! '%()3%)/! %(?! -./! '&()! .$<-&0,! &2! ! !g! F(/4/<<%0,G! $(! 6.$'&<&6.,! &2! '&)$4C! O-! $<! B/''=T(&B(! -.%-! 6! $<! %! @$1%'/(-! 3(%0,! (&(=-03-.=23(4-$&(!&2!6E!5&!B/!4%(!3</!-.$<!/A3$1%'/(-!-0%(<2&0>%-$&(!-&!4&(<$?/0!/1/0,! 6!$(!%(,! >&?%'!<4./>%C!7%T/!7!<4./>%! 6t6!2&0!%(!/L%>6'/C!*/!4%(!2$(?!-.%-M! 9$0<-',E! -./! </>%(-$4! 0/'%-$&(! @/-B//(! K"_6! %(?! 6! $<! -./! <%>/! %<! -./! </>%(-$4! 0/'%-$&(! @/-B//(! 6!%(?!6E!%(?!%44&0?$()!-&!5789E!%(,!@$1%'/(-!3(%0,!(&(=-03-.=23(4-$&(!>%,!%'B%,<!@/! /A3$1%'/(-!-0%(<2&0>/?!-&!%!-03-.=1%'3/?!23(4-$&(E!<$(4/!K"_6k!6 6"E!B/!4%(!3</ "_6k!6 6! -&!0/60/</(-! 6!$(!7!<4./>%C! ! 5/4&(?',E! F 6G! $(! 7! <4./>%! B.$4.! %4-3%'',! 0/60/</(-<! %! </-! &2! -./&0/><! &2! 4'%<<$4%'! 60&6&<$-$&(%'! '&)$4! 0/60/</(-<! %! </-! &2! :@3-! (&-! %;! -03-.=23(4-$&(<! $(! B.$4.! /%4.! >%T/! 7! <4./>%! 1%'$?!$<!$$2!/%4.!$<!-.$<!2&0>M! ! 6!k?/2!6 6C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :g;! ! ! ! ! 7.$0?',E!%44&0?$()!-&!F6&<<$@'/!B&0'?!</>%(-$4< -.%-! 6!$<!-03/!$(!%4-3%'!B&0'?!$(!7!<,<-/>! 0/60/</(-<! 6! $<! -03/! $(! %''! &2! %44/<<$@'/! 6&<<$@'/! B&0'?! B.$4.! %0/! 0/2'/4-$1/! -&! %4-3%'',! B&0'?C! 9&0-3(%-/',E!%(,!-03-.=23(4-$&(!2&0>/?!%<!6 6E!<34.!%<!6E!6 6"E!6 6"E!6 6E!6 : 6 6e!;E!/-4E!B.$4.!'/-<!7!<4./>%!1%'$?!$<!%(?!&(',!$<!-./!-03-.=23(4-$&(!$(!b\!B.$4.!$<!8/2'/4-$1/!-&!6! $(!-./!</(</!&2!-./!</-!-./&0,!WpYE!%(?!%(,!-03-.=23(4-$&(!B.$4.!'/-<![!<4./>%!1%'$?!$<!%(?!&(',!$<! 5/0$%'$-,!-&!6!$(!-./!</(</!&2!-./!</-!-./&0, %(,!-03-.=23(4-$&(!B.$4.!'/-<!X!<4./>%!1%'$?!$<!%(?! &(',!$<!H34'$?/%(!-&!6!$(!-./!</(</!&2!-./!</-!-./&0, <&!&(!%(,!&-./0!<4./>%C! ! 9&30-.',E! .&B! %@&3-! F 6! $<! -03/G! $(! %! 6&<<$@'/! B&0'?! &-./0! -.%(! $(! -./! %4-3%'! B&0'?u! 9&0>%'',E!(&!>%--/0!.&B!4&>6'/L!&2!-./!$((/0!<-034-30/!&2! 6E!B./(!B/!-.$(T!%@&3-!-./!</>%(-$4! 0/'%-$&(!@/-B//(! 6!%(?!6E!-./,!.%1/!%(?!&(',!.%1/!-./</!_!6&<<$@$'$-$/<M!"E"P!`E"P!`E`C!7.%-!$<!-&! <%,E!$2! 6!$<!-03/E!-./!2&0>%'!</>%(-$4<!$<!&(',!-./!<%>/!%<!-03/E!(&!>%--/0!-.%- 6!$<!-03/!$(!-.$<! 6&<<$@'/!B&0'?!&0!$(!%(,!&-./0!&(/P!$2! 6!$<!2%'</E!-./!2&0>%'!</>%(-$4<!$<!&(',!-./!<%>/!%<!2%'</E! (&!>%--/0!-.%- 6!$<!2%'</!$(!-.%-!6&<<$@'/!B&0'?!&0!$(!%(,!&-./0!&(/C! ! 9$2-.',E!$(!b\E! 6k?/2!6 6E! 6!$<!/A3$1%'/(-!-&!%!-03-.!23(4-$&(!%0@$-0%0,!</'/4-/?!20&>! -./!</-!&2!-03-.!23(4-$&(<!$(!b\C!O(!(%-30%'',!'%()3%)/E! 6!$<!F%(,B%,E!6GE!&0!>&0/!4&(1$(4/?!F$-! $<!%'B%,<!2%'</!-.%-!%(,!60/>$</!$(2/0<!(/)%-$&(!&2!6GC! ! 5$L-.',E!B./(!6!$<!-&!?/2$($-/!-03-.!1%'3/!"E!-./!-03/!1%'3/!&2! 6!$<!(&-!?/2$($-/!/$-./0!"&0!`! @3-!.%<!3(4/0-%$(-,!&2!-03-.!1%'3/!"q`!:B/!4%''!$-!4%(!@/!-03/!4%(!@/!2%'</!@3-!(&-!@/!@&-.!-03/!%(?! 2%'</!%-!-./!<%>/!-$>/;E!@3-!-.%-!-03-.!1%'3/!&2! 6!$<!"q`!$<!4&>6'/-/',!?/2$($-/C!D&0/&1/0E!B./(! -./!6!$<!%!?/2$($-/!-03-.!1%'3/!`E!%'-.&3).!-.%-!-./!</>%(-$4<!&2!-./! 6!B.$4.!$<!3(4/0-%$(',!-03-.! 1%'3/!"q`!$<!4&>6'/-/',!?/2$($-/E!-.%-!-./!'&)$4%'!<,(-%L!&2! 6!3(?/-/0>$(/?E!2&0!B/!<-$''!4%(!(&-! 0/)%0?! 6!%<!-./!4&(R3(4-$&(!@/-B//(!6!%(?!%(&-./0!?/2$($-/!(&(=-03-.!23(4-$&(!&2!6C!K&B/1/0E! -.$<! $<! -&-%'! 4/0-%$(-,! -.%-! %-! -.$<! -$>/E! 6! $(! '&)$4%'! <,(-%L! 4%(! @/! ?/2$(/?! %<! %! 4&(R3(4-$&(! @/-B//(!1%0$%@'/!6!%(?!<&>/!(&(=-03-.!23(4-$&(!&2!6!$(!b\P!-.%-!$<!-&!<%,E!-.$<!$<!-&-%'!4/0-%$(-,!-.%-E! '&)$4%'! <,(-%L! &2! 6! $(! 7! <4./>%! 4%(! &(',! @/! ?/2$(/?! %<! %! (&(=-03-.! 23(4-$&(! &2! 6! B.$4.! $<! /A3$1%'/(-!-&!%(,!-03-.!23(4-$&(!B.$4.!$<!8/2'/4-$1/!-&!6!$(!-./!</(</!&2!-./!</-!-./&0,!C!7.$<!$<!-&-%'! 4/0-%$(-,!-.%-E! 6!4%(!(&-!@/!?/2$(/?!%<!%!4&(R3(4-$&(!@/-B//(!6!%(?!<&>/!&(/!(&(=-03-.!23(4-$&(! &2!6!$(!\b!%(?!%<!%(&-./0!4&(R3(4-$&(!@/-B//(!6!%(?!%(,!&-./0!(&(=-03-.!23(4-$&(!&2!6!$(!b\!%-!-./! <%>/!-$>/C! ! N-!'%<-E!.&B!%@&3-!-./!F(/4/<<$-%-$&(!03'/G!$<u! ! ^2!4&30</E!$-!$<!1/0,!4&>6'/Lr!+3-!-./0/!$<!(&! 3<%)/!&2!F(/4/<<$-%-$&(!03'/G!./0/C!9&0!B/!&(',!4&(<$?/0!%-&>$4!>&?%'!60&6&<$-$&(!(&BE!-.%-!$<!-&! <%,!Nk!6!4%(!(&-!@/!/1/0!-03/E!%(?!F(/4/<<$-%-$&(!03'/G!$<!F$2 NE!-./( NGC! ! ! d! ! NANAG#@#R.(&0#("2.-D5'2(-"#-0#2,&#K&-"2('#*-Q('# 5$(4/!$-!.%<!%'>&<-!$(/1$-%@',!@//(!?/&(-$4!6%0%?&L!B./(!?/&(-$4!&6/0%-&0 &@'$)%-&0, &0!F&3).-! -&G!$<!?$0/4-',!$(-/060/-/?!$(-&!>&?%'!&6/0%-/ (/4/<<%0, $(![!<,<-/>!&2!>&?%'!60&6&<$-$&(%'!'&)$4E! B/! -0,! -&! $(-/060/-! ?/&(-$4! &6/0%-&0! %<! <36/06&<$-$&(! &2! (&(=-03-.=1%'3/?! 23(4-$&(<C! */! -0,! -&! ?/2$(/!%-&>$4!?/&(-$4!60&6&<$-$&(<!'$T/!-.$<M! [/&(-$4!'&)$4!$(4'3?/<!%(?!&(',!$(4'3?/<!'&)$4!&2!>&0%'$-,!%(?!'&)$4!&2!&@'$)%-$&(!W"`YC! @A#P-Q('#-0#L-.)*(2/# O-!$<!6/0>$<<$&(!-.%-!6G!$<!F$-!$<!6&<<$@'/!-.%-!6!$<!>&0%'GM! \6!k?/2! D!6C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :d;! ! FO-!&3).-!-&!@/!-.%-!6G!$<!F$-!$<!(&-!6&<<$@'/!-.%-!6!$<!(&-!>&0%'GM! 6!k?/2 ^@1$&3<',E! 6 \ 6!k D!6C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :a;! D!: 6!;C! 6t!D!6E!@3- 6!t\6!$<!(&-!1%'$?C! RA#P-Q('#-0#-;*(Q)2(-"# O-!$<!6/0>$<<$&(!-.%-!6G!$<!F$-!$<!6&<<$@'/!-.%-!6!$<!(&0>%-$1/GM! \6!k?/2! I!6C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :h;! ! FO-!$<!&@'$)%-&0,!-.%-!6G!$<!F$-!$<!(&-!6&<<$@'/!-.%-!(&!6!$<!(&0>%-$1/GM! 6!k?/2 \ 6C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :p;! ^@1$&3<',E! 6k!9: 6;!k I!: 6!; I!6k I!6C!9!$<!2&0@$??/(C 6t\6E!@3- 6! tI!6!$<!(&-!1%'$?C! 6!t6!$<!(&-!1%'$?C!6t\6!$<!(&-!1%'$?C! O(! @&-.! :";! %(?! :e;E! F G$<! F6&<<$@$'$-,G! $(! <&>/! <,<-/>! &2! >&?%'! 60&6&<$-$&(%'! '&)$4! 20&>! 5789E! I! :6;! $<! K6E! <&! ?&! D! :6;C! 7.%-! $<! -&! <%,E! $2! \! B%<! R3<-! F GE! -./(! -./! '&)$4! <,<-/>! &2! &@'$)%-$&(!0%-./0!-.%(!-./!'&)$4!<,<-/>!&2!>&0%'$-,!B&3'?!@/!0/)%0?/?!%<![!<,<-/>C!^2!4&30</E!$(! '&)$4!<,<-/>!&2!&@'$)%-$&(E!-.&3).! 6!$<!(/4/<<%0,!\E!\6!$<!(&-!6&<<$@',!6!@3-!6&<<$@',!-.%-!6!$<! (&0>%-$1/C! ! D&0/&1/0 $(!(%-30/!<$-3%-$&( F-.%-!6!$<!(&0>%-$1/G!%(?!F-.%-!(&!6!$<!(&0>%-$1/G!4%(!(&-!@/! @&-.!2%'</C!7.%-!$<!-&!<%,E!B./(!2&0>/0!I!6!$<!"E!-./(!-./!'%-/0!I!: 6!;!$<!"q`P!*./(!2&0>/0!$<!`E! -./(!-./!'%-/0!$<!"E!<34.!-.%-! ! ! B./(!I!6k!K6E!I!: 6!;k!KXK6C! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! :"`;! NANAN#@#=.&*(L(")./#7"2&.%.&2)2(-"#0-.#%.-;*&L#S# */! &0?/0! -.&</! 60&6&<$-$&(<! %-! -./! @/)$(($()! &2! -.$<! %0-$4'/M! FO! 6%,! %! -%LG! %<! 6E! FO! ?&(%-/! >34.! >&(/,G!%<!6TC!FTG!0/60/</(-<!-.%-!-./0/!%0/!T!-$>/V<!<36/06&<$-$&(!&2!FKGC!FO-!$<!&@'$)%-&0,!-.%-!O! 6%,! %! -%LG! %<! 6E! FO-! $<! &@'$)%-&0,! -.%-! O! ?&(%-/! >34.! >&(/,G! %< 6TC! T eE! T IE! $(! -.$<! <$-3%-$&(E!<34.!-.%-C! "M! 6k!9: 6;!k I!: 6!;!k! I!: 6!;P! e I!6k!K6k!6"P! _ 6k I!: 6!;k KXK6!k KX6"!k : 6"! K6";k :6" 6e;C! ! a! ! X 6Tk! I!: 6T!; ! g I!6Tk!K6Tk6Tv"P! d 6Tk I!: 6T!;k KXK6T!k! :6Tv" 6!Tve;C! O-!$<!/%<,!-&!<//!-.%-!B./(!6k6Tk"E! 6!%(?! 6T!4%(!@/!?$22/0/(-!-03-.!1%'3/<E!2&0!6 6" 6e 6T 6Tv"!%(?!6Tve!%0/!$(?/6/(?/(-!&2!/%4.!&-./0E!(&!>%--/0!.&B!F G!$<!$(![!<,<-/>!&0!$(!7!<,<-/>C! TA#M-"'*5<(-"<# 7./! 4&0/! &2! 5789! -./&0,! $<! -./! ?$<4&1/0,! %(?! 0$)&0&3<! 4'%<<$4%'! ?/2$($-$&(! &(! -./! /A3$1%'/(-! -0%(<2&0>%-$&(!@/-B//(!-03-.!23(4-$&(!%(?!(&(=-03-.!23(4-$&(!$(!-./!<,<-/>!&2!60&6&<$-$&(%'!'&)$4! -B&=1%'3/?C!9$0<-',E!B/!0/%'$S/!-./!0/'%-$1$-,!&2!-03-.=1%'3/?!23(4-$&(=(/<<C!5/4&(?',E!B/!3</!&(',! <$L!<,>@&'<!n!KE!6P! E! P!:E! ! ;!o!-&!?/2$(/!%''!-03-.=23(4-$&(<!$(!B.$4.!/%4.!4%(!@/!/A3$1%'/(-!-&! %! (&(=-03-.! 23(4-$&(! $(! b\C! 7.$0?',E! B/! 2$(?! &3-! -.%-! F-./! 2%$'30/! &2! -./! /A3$1%'/(-! <3@<-$-3-$&(! 60$(4$6'/G! 4%(! @/! %3-&>%-$4%'',! )/(/0%-/?! 20&>! 5b\! $(!4'%<<$4%'! 1$/BC! 9&30-.',E! B/! )/-! -.%-! %(,! <4./>%!$(!60&6&<$-$&(%'!>&?%'!'&)$4!>3<-!@/!%!)0&36!&2!-./&0/><!$(!\bC!9$2-.',E!/1/(!-.&3).!B/! .%1/! (&-! @//(! -.&0&3).',! 0/?34/?! F6&<<$@'/! B&0'?! </>%(-$4<GE! B/! T(&B! -.%-! >%(,! <$()'/! (&(=-03-.!23(4-$&(!&6/0%-&0<!$(!(%-30/!'%()3%)/!-30(!-&!@/!<&>/!<36/06&<$-$&(!&2!</1/0%'!(&(=-03-.! 23(4-$&(! &6/0%-&0<! $(! 5b\C!7./</! (&-! &(',! <.&B! -.%-! -./! (%-30%'! '%()3%)/! <,<-/>! $<! %4-3%'',! -./! F<%1/G! %(?! F<%2/-,G @3-! %'<&! <.&B! %! @0$).-! B%,! 2&0! >%(,! <4$/(-$<-<! $(! $(2&0>%-$&(! <4$/(4/<! &0! 4&>63-/0! -/4.(&'&)$/<! B.&! %0/! B&00,$()! %@&3-! -./! ?$<3($-/! &2! (&(=4'%<<$4%'! '&)$4! %(?! 4'%<<$4%'! '&)$4C! ! ! ! ! D&0/&1/0E! %44&0?$()! -&! 5789! $(! 60&6&<$-$&(%'! '&)$4E! %(,! -03-.! 23(4-$&(! 4%(! @/! /A3$1%'/(-',! -0%(<2&0>/?! -&! %! (&(=-03-.! 23(4-$&(! ?/2$($-/',E! @3-! B./-./0! %(,! (&(=-03-.! 23(4-$&(! 4%(! @/! /A3$1%'/(-',!-0%(<2&0>/?!-&!%!-03-.!23(4-$&(!?/2$($-/',E!$-!$<!<-$''!%(!&6/(!A3/<-$&(C! @'U"-H*&DQ&L&"2<A#7.$<!B&0T!$<!<366&0-/?!@,!-./!I%-$&(%'!5&4$%'!54$/(4/!9&3(?%-$&(!&2!b.$(%! :I&<Ce``aSL4__;! %(?! -./! K3>%($-$/<! 8/</%04.! 93(?<! $(! K3%S.&()! ]($1/0<$-,! &2! 54$/(4/! %(?! 7/4.(&'&),!:e`"e8*w[`h;C! 1&0&.&"'&<# "C!+'%4T@30(E!\CE!J%(!+/(-./>E!xCE!*&'-/0E!9CM!K%(?@&&T!&2!D&?%'!l&)$4E!H'</1$/0!:e``a;! eC!K$''E! [C!xCE! D4'/&?E!5C! sCM! ^(!703-.=93(4-$&(%'$-,E! 8/1$/B! &2! 5,>@&'$4! l&)$4! _! :X;E! deh=d_e! :e`"`;! _C!D%04&<E!xCM!*.%-!$<!%!(&(=-03-.=23(4-$&(%'!'&)$4u!5-3?$%!l&)$4%!pe!:e;E!e"g=eX`!:e``p;! XC! U3$(/E! *CJCM! 7./! \0&@'/>! &2! O(-/060/-$()! D&?%'! l&)$4E! -./! x&30(%'! &2! 5,>@&'$4! l&)$4! "eE! X_=Xh!:"pXa;! gC!U3$(/E!*C!JCM!D/-.&?<!&2!l&)$4E!66C!"="`C!K%01%0?!]($1/0<$-,!\0/<<E!K%01%0?!:"phe;! dC!8/<4./0E!ICM!7./!b&(4/6-!&2!8%(?&>(/<<E!7./&0$%!ea!:";E!"="":"pd";! aC!54.($/?/0E!+CM!703-.=23(4-$&(%'$-,E!8/1$/B!&2!5,>@&'$4!l&)$4!"!:";E!dX=ae!:e``h;! hC! 5-/6./(E! 8CM! b&(?$-$&(%'<! %0/! (&-! 703-.=93(4-$&(%'M! N(! N0)3>/(-! 20&>! \/$04/E! N(%',<$<! geE! g="e!:"ppe;! pC!*%(E!#ClCE!l$E!9CyCE!7$%(E!#CM!7./!5/4&(?!HL6'&0%-$&(!-&!](%0,!^6/0%-&0<M![/&(-$4!l&)$4!%(?! [/&(-$4! \%0%?&LE! -./! x&30(%'! &2! N(.3$! ]($1/0<$-,! :5&4$%'! 54$/(4/! /?$-E! $(! b.$(/</;! "h:_;E! _h=Xa!:e`"e;! "`C!*%(E!#C'CM!7./!7.$0?!HL6'&0%-$&(!-&!](%0,!^6/0%-&0<M!&(!D&?/0(!D&?%'!l&)$4!20&>!5789E! ! h! ! -./! x&30(%'! &2! K3%S.&()! ]($1/0<$-,! &2! 54$/(4/! %(?! 7/4.(&'&),C! :5&4$%'! 54$/(4/! /?$-E! $(! b.$(/</;!ed:_;E!__=_p!:e`"e;! ! ! p! !