Survey
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project
Conditions for Selection and Conceptualization in Diagrams and Sentences Rossano Barone ([email protected]) Peter C-H. Cheng ([email protected]) Representation & Cognition Group, Department of Informatics, University of Sussex, Brighton, BN1 9RN, UK. Abstract This paper considers several ways in which diagrams minimally designate the selection and conceptualization of information compared to sentential forms of external representation (ER). We group these factors as different ways that an ER designates cognitive engagement (DCE) and propose that this concept is important in distinguishing and understanding differences between different classes of ER. The article outlines components of a conceptual framework we are currently developing for characterizing expressions of information in diagrams and other classes of ER. The framework is employed to better understand the nature of DCE in diagrams and its semantic conditions. Our account of DCE is then used to shed light on three important characteristics of diagrammatic ERs: (1) the ill-defined nature of the information they express, (2) their rich potential for providing expressions and (3) their efficacy for tasks that are ill-defined in certain ways. Keywords: Externally supported cognition, diagrams, semantics, conceptualization. Introduction In describing particular states of affairs certain conditions present in sentences of a natural language designate both the selection of particular objects and relations as well as the ways selected elements should be conceptualized. By designation we mean roughly to specify or prescribe. For example, consider the expression node-27 has been connected to the semantic network. To start with the expression designates that the cognitive system should construct an interpretation involving particular types of tokens (i.e., node, network) and relations (e.g., connected to). The syntactic properties of sentences also provide a condition for designating the roles of the objects in the connected to relation so that the cognitive system interprets the network as a kind of landmark and node-27 as the manipulated object. The fact that the syntactic structure designates the interpretation can be shown by changing the arrangement of words e.g. the semantic network has been connected to node-27 changes the landmark interpretation in a way that seems inconsistent with how such events are normally conceptualized (e.g. Talmy 1978). The sentence also specifies how objects should be construed. Consider expressions referring to the same situation but replacing node-27 with the bird node or the penguin node. Each sentence refers to the same token node but designates a conceptualization of the node at a different level of abstraction (e.g. Langacker 1991). The reader familiar with cognitive linguistics will recognize these devices in natural language. Yet these different ways of des- ignating what a cognitive system should do in interpreting expressions are not just characteristics of natural language as one will also find them in other forms of external representation (ER) that have a sentential character such as mathematical and programming notations. For example, the electricity formula I = V/R designates particular classes of operations (e.g., /) that should be performed between entities and roles that should be assigned to objects (e.g., V – numerator, R – denominator). In this article we will outline ways in which sentential and diagrammatic ERs vary in the extent that they designate the kinds of phenomena discussed. To be clear, the kinds of designation associated with classes of ERs will include, amongst others; the selection of tokens, types of objects and relations; operations to be performed over objects; the roles to be assigned to objects in relations; ways of mentally simulating objects; and the level of abstraction that objects should be conceptualized. We classify these and other cases that we will discuss in this article as particular ways that an ER designates cognitive engagement (DCE). Our choice of term is abstract to accommodate the range of cognitive activities that are designated in interpreting expressions of an ER. The extent of DCE associated with classes of ER will be lean when it contains conditions that designate only a few forms of cognitive engagement and rich when it can designate many forms of engagement. We chose natural language as a case to introduce DCE because of the rich forms of DCE it offers. Our major concern however is with understanding cognitive and semantic characteristics of diagrammatic ERs, which as we will show normally offer very lean forms of DCE. We argue that the notion of DCE offers an important perspective that can be applied to different classes of ERs and for which ERs vary in different ways. We propose that an explicit characterization of DCE is related to and can be used to shed understanding on three important properties of diagrams: (1) the ill-defined nature of what they express; (2) their rich potential for providing expressions; and (3) their efficacy for tasks that are ill-defined in certain ways. The next issue concerns the difficultly in determining what expressions can actually be attributed to a diagram. A common observation is that what expressions can be attributed to a diagram by a particular user is in part dependent on what knowledge they possess about the diagram and the domain being represented. Expressions of information in diagrams are implicit in certain ways. 1705 The rich potential for providing expressions attributed to diagrams concerns the observation that diagrammatic ERs often reveal a range of expressions that were not entertained during the diagrams construction. There are many factors behind this property that we cannot fully address here. Our particular concern is with the observation that diagrams can afford a range of different ways of relating and conceptualizing information often without any form of prior instruction. What is critical is the role of conceptual knowledge possessed by the user in determining expressions in a diagram. Many researchers have appealed to this property of diagrams. For example, Cheng considers how diagrams can be designed to simultaneously express interdependent classes of representing information at different level of abstraction and from alternative perspectives (e.g., Cheng 2002). Shimojima (1999) considers particular cases of higher-order expressions in diagrams that he classifies as derivative meaning and proposes a formal semantics account implicating chains of logical entailments between the ER and represented states of affairs. The derivative meaning discussed by Shimojima may depend on the user applying uninstructed conceptual knowledge in interpreting these classes of expressions. One of course can also consider the domain of information visualization which is concerned to a large part with exploiting the conceptual knowledge of a user in deriving higher-order expressions in diagrams. This cluster of issues is also related to particular ways in which diagrams are efficient for certain kinds of ill-defined cognitive tasks. The potential of diagrams to systematically represent different perspectives is considered by some important in their efficacy (e.g., Cheng, 2002; Cheng & Barone, 2007; Vicente, 1996). Information visualizations also appear to support tasks that are ill-defined in various ways (e.g. discovering trends). Whilst this is an important benefit of diagrams, in our view, it has not been given a clear characterization from a cognitive and semantic perspective. Our analysis suggests that the ill-defined nature and rich potential for expressions afforded by a diagram overlap because their characterization depends on recognizing the intervention of the cognitive system in determining the existence of expressions in a diagram in a way that is qualitatively different to many sentential systems. Hence our basic argument is this, when DCE is lean certain kinds of interventions with conceptual knowledge are needed by the cognitive system to make expressions determinate and it is this that is partly responsible for a diagrams’ rich potential for expressions and their ill-defined nature. An implication of this argument is that the efficacy of diagrams for certain kinds of ill-defined tasks can be seen to depend on the lean amount of DCE offered by diagrams. In the next section we will introduce components of our conceptual framework that we will use to characterize relevant properties of representing information and diagrammatic expressions before describing the notion of an abstraction hierarchy of cognitive correspondence that can be attributed to a diagram. We will then consider several cases of lean DCE in diagrams before examining what we believe to be a fundamental semantic condition for the lean DCE typical of diagrams. Following this we will consider how our characterization of lean DCE in diagrams can be used to shed light on factors responsible for the ill-defined status of what expressions they possess and their rich potential for providing expressions. We will end by considering the role of lean DCE in the efficacy of diagrams for ill-defined problems. Conceptual Framework Representing Information An ER represents information about some knowledge domain or state of affairs that has been interpreted into units, often in accord with the format of some general class of ER (e.g., sentence, table, etc). In considering the representation of information in diagrams it is useful to distinguish between information that is referentially specific from information that is purely schematic. We define a unit of information as referentially specific if the interpretation of the types of entities it identifies are assigned definite token identity. In turn, we define token identity as temporally cognitive representation of the reference or existence per se of a token independent of any perceptual or conceptual ascription that may be applied to it. One candidate account of token identity is the theory of visual indexes (e.g., Pylyshyn 2002) although the theory proposes that such indexes are confined to online visual processing. We also consider token identity to be implicit in other accounts of internal representation of particular states of affairs such as mental models and imagery. Referentially specific information results from an interpretation of real, hypothetical or imaginary states of affairs. Examples include: concrete physical relations between individuals, e.g., connected-to (resistor-A, resistor-B); abstract relations between particular individuals, e.g., loves (Bill, Mary); arbitrary information ascribed to particular individuals, e.g., loan duration (psy24657) = short; abstract physical properties of individuals, e.g., voltage (battery) = 12 Volts. What is common in these cases is that the information specifies particular tokens rather than just classes of them. We define an interpreted unit of information as purely schematic information if conceptualized entities are not assigned particular token identities. Such information includes: (1) relations about classes of things, e.g., all canaries are birds; (2) abstract physical laws, e.g., I = V/R; (3) arbitrary rules, e.g., if the request of a book consistently exceeds its threshold then put it on short loan; (4) regularities, e.g., old books tend to be requested less. Purely schematic information only identifies types or classes of entities rather than particular tokens. Diagrammatic Expressions For clarity we will distinguish between diagrammatic and semantic propositions. Propositions are diagrammatic when they are conceptually interpreted in the domain of the dia- 1706 gram whereas propositions are semantic when they are conceptually interpreted in the represented domain. For example deriving that the R2 box is wider than the R1 box in fig 2 is a diagrammatic proposition. Interpreting that this means the R2 resistor has a greater current than the R1 resistor is a semantic proposition. An important property of diagrammatic systems is that the ER tokens are taken to stand in for the tokens being represented (e.g., Barwise & Etchemendy 1995; Stenning et al., 1995). We call this kind of token representation in diagrams direct token reference. From a semantic perspective we say that an ER affords direct token reference whenever the token identity of its representing tokens stand in for the token identity of represented tokens. From a cognitive perspective direct token reference occurs whenever a cognitive system takes the token identity of an ER token to be the target or argument of semantic propositions. For example, if we arbitrarily name the token identity of R1 and R2 in the AVOW diagram as in Fig. 2 as t1 and t2, then we can say that in direct token reference not only are these tokens the argument of diagrammatic propositions – e.g., wider-than (t2, t1) – the same token identities are also the arguments of semantic propositions – e.g., greater-current-than (t2, t1). The circuit diagram has a battery connect to two resistors, R1 and R2, in parallel, which are connect to a third resistor, R3. Rectangles in the AVOW diagram represents resistors and the topology of the circuit is represented by the spatial layout of the rectangles. The height, width, area and slope of the diagonal line in each rectangle respectively represent the voltage, current, power and resistance of its resistor. The geometry and spatial layout of the diagrams encodes the basic laws of electrical circuits (Cheng, 2002). Fig. 2. Circuit diagram (A) and an AVOW diagrams (B) In the last section we stated that expressions afforded by diagrams are implicit in certain ways. This sense of implicitness is based in part on our characterization of the representation of relations in diagrams. We classify relations in diagrams as operational relations because representing relations are literally opportunities for a cognitive system to execute some cognitive routine that computes the relation. For example, R2 has a greater current than R1 is an operational relation in an AVOW diagram because it involves executing a cognitive routine that compares the width of the token rectangles that stand in for R1 and R2. Operational relations depend on direct token reference because the token identities of the ER tokens are the subjects of the relation. Fig. 1. Schematization of semantic mappings in (A) direct token reference and (B) indirect token reference Sentential systems on the other hand are intended to designate an indirect mode of token reference. For example, given that PSY679 is the name of a token book then the expression Title (PSY679) = Cognitive Psychology involves indirectly referring to that token by name. Expressing another attribution Publication year (PSY679) = 1972 involves expressing PSY679 again. This is permitted because the token identity of the symbol PSY679 is not meant to stand in for the token identity of the particular book it identifies. Another way to refer to a token indirectly is by attribute descriptions, such as when the librarian refers to the book as “The red psychology book with the torn sleeve”. In systems intended to afford indirect token reference the ER tokens are not intended to be the arguments of semantic propositions. An important implication of direct token reference is that attributes of ER tokens normally represent attributes of represented tokens and types of ER tokens, often classified in terms of their representing attributes, normally represent types of representing tokens. In sentential classes of ERs the type of ER tokens (i.e. symbols) rather than their attributes identify semantic information (see Fig. 1). Fig. 3. A semantic network (A) and a production rule in a cognitive modeling language (B) A common property of operational relations is that no symbol exists that represents the identity of the type of relation although there are many exceptions. For example, semantic networks use symbols such as arcs and text to represent the identity of a relation; e.g., Fig. 3. Even though such relations use symbols they are still operational relations. The critical issue to note is that the difference between operational relations that are symbolized in diagrams and symbols that identify relations in sentential systems is that the diagrammatic cases afford direct token reference whereas the sentential cases are normally indirect. This allows the ER token relations to be the arguments of semantic propositions. For example, one can compare the arcs of different nodes. 1707 Cognitive Correspondence Hierarchy DCE in Diagrams We propose that understanding differences in lean DCE in diagrams, and the related issues just mentioned, is informed by appreciating the abstraction levels of cognitive correspondence that can be attributed to diagrams, see Fig 4. Fig. 4. Diagram correspondence abstraction hierarchy Level 1 correspondence relates to the physical markings that make up a diagram. These are low-level visual representations that precede the assignment of token identity. Level 2 correspondence relates to what we term the referential information structure (RIS) of a diagram. The RIS is the space of information states that exist in a diagram that are conditions for the affordance of particular expressions. The cognitive correspondence of a RIS is the representation of references between semantically interpretable token identities and visual and spatial attributes. As the RIS only includes the sub-set of representing token identities and types of attributes it is more specific than the notion of an image or depiction per se. The RIS has representing status because its token identities and attribute bindings stand in for those of a represented state of affairs. Level 3 correspondences are referentially specific expressions afforded by a diagram that result from cognitive routines following selections of the RIS. The corresponding cognitive representations are referentially specific propositions that refer to token identities in the RIS (e.g., greaterresistance-than (t2, t1)). This level depends on the intervention of the cognitive system to independently designate ways of interpreting pre-conceptual elements of the RIS. Level 4 correspondences are the purely schematic expressions afforded by a diagram derived from one or more referential expressions. For example, deriving referentially specific propositions such as - the resistance of R1 (i.e. diagonal of a box) increases as a result of increasing the voltage of R1 (i.e. height of box) may lead to the abstraction of a purely schematic proposition that - if the resistance of a resistor is increased then its voltage will also be increased. Selection In general, diagrams tend to differ from sentential ERs because they do not designate particular selection strategies associated with how information should be acquired by a cognitive system. Sentences, paragraphs, lines of code, mathematical expressions are ordered in ways that represent an intended designation of the order that represented information should be selected and comprehended to support an intended interpretation. While it is true that diagrams represent ordinal structures and such structures may influence the order in which expressions are selected these are better interpreted as incidental rather than intended DCE of an ER system. At a more local level one can also consider ways that diagrams typically do not designate which particular tokens should be selected and the kinds of relations that should be derived. For instance, in Fig. 2b any resistor may be related to any of the other resistor, and because resistors have multiple types of attribute there are many different ways in which a resistor can be related. The selected tokens and relations are abstractions taken from the referential information in Fig. 2 that identify specific kinds of DCE. Conceptualization Relating entities in a diagram involves, in particular, assigning specific conceptual roles to entities in the relations. The same representing entities may be assigned different roles for the same and different types of relations in which they participate. For example, in recognizing the consequence of changing the value of the voltage of a resistor R1 (i.e. height of AVOW box) on its resistance (i.e. diagonal of AVOW box) in an AVOW diagram the cognitive system must strategically assign the role of independent variable to the height of the AVOW box and the role of dependent variable to its diagonal. In deriving the implication of changing the value of the resistance with respect to the voltage these roles are reversed. This example shows that the roles that are assigned to the representation of a specific object in a derived expression are left to the choice of the interpreter rather than being designated by the ER. In sentential systems roles are often designated by the syntax and semantics of relational symbols expressed in sentences. For example, as considered in the introduction, arithmetic expressions (e.g., I = V/R) typically assign specific roles to the elements (e.g. I – dependent variable) just as natural language expressions such as Bill loves Mary designate lover and beloved roles to Bill and Mary respectively. Another way in which diagrams do not designate the conceptualization of expressions is in terms of the level of abstraction or specificity with which representing elements can be selected. We distinguish between two ways that a cognitive system can select the level of abstraction to conceptualize objects represented by a diagram - inherent or projected abstractions. 1708 Inherent abstractions correspond to cases when there are one or more combinations of represented attributes of a type of object that systematically identify the abstract conceptual information that characterize the classification. For example, consider an augmented AVOW diagram in which the type of shading of the resistor distinguishes between whether it is a bulb or a heater. This augmentation allows a user to classify a token AVOW box abstractly as a resistor or more specifically as a bulb or heater. The former classification is identified by the form per se (i.e., rectangles), whereas the latter classification is identified by a combination of form and shading. Opportunities for inherent abstraction can also arise through emergent properties in diagrams. For example, in a normal AVOW diagram the overall shape of the rectangles for the resistors in Fig. 2b identifies whether the resistor is more specifically a conductor (R3, tall and thin) or an insulator (R1, wide and short) (Cheng, 2002). Projected abstractions of objects in diagrams merely require that a cognitive system knows more than one level that types of represented objects can be conceptualized at. In projected abstractions these different levels of abstraction are not identified by specific combinations of representing attributes. For example, a user may conceptualize a token resistor more abstractly as an electronic component given that they know the AVOW rectangles represent resistors. In projected abstraction the level of abstraction that can be assigned must be more abstract than the classification denoted by the represented object. Hence, on the basis of knowing only that R1 is a resistor per se one can refer to it as an electric component but cannot validly refer to it as bulb. The designation of the level of abstraction is often specified by types of symbols in natural language as well as mathematical notations and programming languages. For example the words bulb, insulator and resistor are symbols that can identify the same type of token at different levels of abstraction. Certain notational structures may also permit conceptualizations at different level of abstraction because they also afford direct token reference. For example the production rule as shown in Fig. 3B may be construed as a visual strategy production or more specifically as a shifting attention production depending on the interpretation task. A further important distinction to be made concerns cases where the same representing tokens can be classified through different ontological classes. We use the term ontological class to refer to the abstract and ubiquitous ways of conceptualizing selected entities that are afforded by the cognitive system (e.g., as objects, configurations, properties, relations, states, processes, paths, points, etc.,). Different tasks may invoke particular ontological conceptualizations. For example, the arcs in the semantic network of Fig. 3A will be conceptualized as: (1) objects when being enumerated; (2) as relations in establishing whether two nodes are connected; (3) as paths in the course of tracing the direction of activation between nodes. In natural language some of these ontological classes are normally designated by a combination of properties of sentences including the choice of word class. We have shown that different types of DCE present in sentential systems are lean in diagrammatic forms of ER. In the case of diagrams these designations need to be independently selected by a cognitive system, such as the tokens and the relations it should select, how it should conceptualize them and the order in which it should make these selections. We propose however that it is problematic to characterize DCE as being completely absent in diagrams because they are often designed, manipulated or selected for the purpose of positively biasing particular ways of interpreting information by making certain classes of expressions more salient or easier to apprehend. Lean DCE and Direct Token Reference Parts of our framework were introduced to better understand the lean DCE that we attribute to diagrammatic ERs. Our analysis suggests that the existence of lean DCE in diagrams is fundamentally dependent on direct token reference. To comprehend this one needs to show the dependencies that link direct token reference to DCE. This can be done by appealing to parts of the cognitive correspondence hierarchy. The RIS contains the space of bindings between representing token identities and perceptual attributes and exists only because diagrams afford direct token reference. The RIS represents conditions for different kinds of operational relations and ways of conceptualizing the tokens it contains. Diagrams are not direct specifications of expressions because expressions identify one or more propositions. Rather diagrams are specifications of an RIS that satisfy certain semantic propositions. The crucial fact is that expressions are still indeterminate at the RIS level because they require cognitive intervention that designates the particular ways of selecting and conceptualizing its elements and thus forming semantic propositions. Hence lean DCE occurs because diagrams are representations of information at the RIS level and the RIS level exists because of direct token reference. Sentential systems do not have an RIS level because (with some exceptions) they are intended to afford an indirect mode of token reference. Ill-defined Status of Expressions As diagrams do not designate the DCE needed to specify particular expressions what expressions can be attributed to a diagram is ill-defined. We can again appeal to the cognitive correspondence hierarchy to characterize this. The hierarchy implies that what constitutes an expression in a diagram involves a collaboration between elements of the RIS and the cognitive system’s own designations of the kind of cognitive engagement that are typically designated by sentential ERs. The ill-defined nature of what expressions can be attributed to diagrams shows their conceptual underspecificity. This is not surprising since the RIS represents referential information by token identities and visual spatial properties that are pre-conceptual. This status of the RIS explains, in part, why what expressions can be attributed to a diagram is ill-defined. 1709 Rich Potential for Providing Expressions The lean DCE in diagrams has an important role in understanding their rich potential for providing expressions. Firstly, because there is little or no DCE on the selection of tokens and relations this has a combinatorial consequence on the space of possible expressions. That is, any token can be related to any other token sharing the same representing attributes, given certain semantic conditions hold. Moreover operational relations can be combined to form more complex higher-order expressions. Note also that the existence of semantically meaningful emergent tokens in diagrams formed by groupings and sub-partitions of ER tokens depends on the capacity to semantically relate the emergent token to their constituent ER tokens. The lack of DCE on the conceptualization of objects is responsible for extending the range of expressions that can be afforded. As the RIS is pre-conceptual there may be a number of different ways of conceptualizing its elements. To appreciate this, consider again how the different ways of conceptualizing arcs in graphs are specific to different kinds of operational relations. There are two other important issues that are required for a more complete account of the rich potential for providing expression in diagrams that we do not address here. The first of these concerns how a cognitive system constructs valid semantic interpretations from more primitive ER mappings. A second of these concerns how the particular visual and spatial properties of diagrammatic formats may make target expressions easier to apprehend. Clearly lean DCE alone does not guarantee that expressions implicit in a diagram can be apprehended by a cognitive system so is best viewed as a contributory factor responsible for a diagrams rich potential for providing expressions. Cognitive Efficacy The lean DCE in diagrams allows problem solving and reasoning to be more flexible, in certain ways, even though they lack the expressive flexibility of sentential ERs. This is because users can select expressions on an as-needed basis from the rich pool of expressions that they are able to apprehend. The different ways of conceptually interpreting information afforded by a diagram can support learning (e.g. Cheng 2002). Our analysis of the relationship between ER and task suggest that the lean DCE is particularly useful in tasks that are ill-defined in various ways. These include cases where the problem would be objectively classified as ill-defined such as combinatorial constraint satisfaction (e.g., Cheng & Barone, 2007), design and discovery problems. They also include cases where the problem is subjectively ill-defined such as when users lack knowledge of how to solve a problem; such as naive students leaning the physics of electric circuits (Cheng 2002). This is because in all these cases what specific referential information should be selected and what conceptualization applied is not known to the user beforehand. Hence, the combination of lean DCE and the capacity of appropriate diagrammatic formats to facilitate the recognition of expressions both permits and supports a user in finding this out. However this is just one of several factors behind the efficacy of diagrams for particular kinds of tasks. Conclusion The article has distinguished between different kinds of DCE in ERs. We have shown that (1) DCE is lean in diagrams and rich in ERs that have a more sentential character; (2) the lean DCE in diagrams is dependent on direct token reference and is related to (3) their rich potential for providing expressions and (4) the ill-defined status of what expressions a diagram affords. These relations can be understood by appealing to the idea of the cognitive correspondence hierarchy. The issue of DCE encompasses issues of conceptualization in diagrams that has not been given much attention from a cognitive and semantic perspective and provides a new dimension for distinguishing between diagrams and sentential classes of ERs. References Cheng, P. C.-H. (2002). Electrifying diagrams for learning: principles for effective representational systems. Cognitive Science, 26(6), 685-736. Cheng, P.C.-H. & Barone, R., (2007). Representing Complex Problems: A Representational Epistemic Approach. In Jonassen. D. (Eds.), Learning to solve complex scientific problems. Lawrence Erlbaum Assoc., Mahwah NJ. Langacker, R, W. (1991). Concept, Image, and Symbol: The Cognitive Basis of Grammar. Mouton de Gruyter: Berlin & New York. Pylyshyn, Z. (2002). Visual Indexes, Preconceptual Objects, and Situated Vision. Cognition, 2002, 80(1/2). Shimojima, A. (1999). Derivative meaning in graphical representations. Proceedings of 1999 IEEE Symposium on Visual Languages (pp.212-219). Stenning, K., Inder, R. & Neilson, I. (1995). Applying semantic concepts to analyzing media and modalities. In Glasgow, J., Narayanan, N. H., & Chandrasekaran, B., (Eds.): Diagrammatic Reasoning: Cognitive and Computational Perspectives. AAAI Press: Menlo Park, CA. Barwise, J. & Etchemendy, J. (1995). Heterogeneous Logic. In Glasgow, J., Narayanan, N. H., & Chandrasekaran, B., (Eds.): Diagrammatic Reasoning: Cognitive and Computational Perspectives. AAAI Press: Menlo Park, CA. Talmy, L. (1978). Figure and Ground in Complex Sentences. In J. Greenberg (Eds.): Universals of Human Language. Stanford: Stanford University Press. Vincente, K. J. (1996). Improving dynamic decision making in complex systems through ecological interface design: A research overview. Systems Dynamics Review, 12(4), 251-279. 1710