Download Decoding visual consciousness from human

TICS-762; No of Pages 9
Opinion
Decoding visual consciousness from
human brain signals
John-Dylan Haynes1,2,3
1
Bernstein Center for Computational Neuroscience, Charité – Universitätsmedizin Berlin, 10115, Germany
Graduate School of Mind and Brain, Humboldt-Universität zu Berlin, 10099, Germany
3
Max-Planck-Institute for Human Cognitive and Brain Sciences, Leipzig, 04103, Germany
2
Despite many years of research on the neural correlates
of consciousness (NCCs), it is still unclear how the
detailed contents of consciousness are represented in
the human brain. It is often assumed that specific contents of consciousness are encoded in dedicated core
NCCs – one for each different aspect of conscious experience. Now, the approach of multivariate decoding provides a novel framework for studying the relationship
between consciousness and content-selective processing in more detail. This approach makes it possible
to assess how conscious experience is encoded in the
brain and how the encoding of sensory information is
affected when it enters awareness.
Core neural correlates of consciousness
One distinguishing feature of visual consciousness (see
Glossary) is its vivid, experiential quality. Take, for
example, the famous sketch by Ernst Mach in his Analyse
der Empfindungen [1] (Figure 1a). It shows Mach’s attempt
to capture the first-person perspective of his visual experiences (specifically, the distribution of light and dark
regions in his visual field) while he is looking out into
his study. The traces of his eyebrow, nose and moustache
enhance the first-person feeling for the observer, as if one
were Mach himself, looking out of his left eye. This image
immediately makes it clear that phenomenal consciousness is composed of several complex, structured contents
that range from the fine-grained patterns of light and dark
to meaningful individual objects.
Several theoretical approaches have been proposed to
unravel how the detailed contents of consciousness are
realized in the human brain. Typically, these proposals
entail a distinction between two types of conditions for
consciousness [2–4]: first, ‘enabling’ or ‘background’ conditions, such as subcortical neural mechanisms of wakefulness, that are necessary to make an individual awake and
conscious of anything (these conditions are unspecific in
the sense that they are required for a broad range or even
all conscious experiences to occur); and second, ‘contentspecific’ conditions that are necessary for a specific
category of conscious experiences (such as brightness,
colour or motion sensations). These specific conditions
are thought to constitute a ‘core neural correlate of consciousness (NCC)’, a minimal set of neurons that shows a
tight mapping or ‘direct correlation’ [2] with a category of
Corresponding author: Haynes, J.-D. ([email protected]).
experiences; it is conceivable that each different category
has a different core NCC. Here, the view is adopted that on
the one hand, a core NCC is necessary for a specific conscious experience (Box 1), and on the other hand, its
activity patterns exhibit a consistent mapping to specific
experiences that can be viewed as an ‘encoding’ of the
experience in question.
A key criterion for a core NCC is that it shows a
maximally ‘tight’ and ‘direct’ correlation with the specific
contents it realizes [2,3]. A frequent example is the correlation of motion experiences with processing in the human
motion area MT/V5 [4]. Motion perception activates MT/V5
[5], the stimulation of MT/V5 influences motion judgements [6] and creates motion hallucinations [7], and the
removal of MT/V5 yields motion blindness (also known as
akinetopsia) [8]. This plausibly suggests that MT/V5 is the
core NCC for conscious motion perception [4]. However, a
tight or direct correlation seems to imply more than showing a dependence of motion perception on activity in MT/
V5. Direct correlation refers to the ability of different states
of a core NCC to explain individual-specific subtle differences between various percepts. This requires showing the
link at a more fine-grained, content-based level, as will be
outlined below.
Glossary
Binocular rivalry: when conflicting stimuli are presented to the two eyes,
conscious perception can alternate spontaneously between the input to the left
and to the right eye [21,54,58].
Consciousness: the term ‘consciousness’ is used in different ways. This article
follows the schema employed in Ref. [78]. In one sense, consciousness refers
to the state of being awake and responsive as opposed to being asleep, in a
coma or under anaesthesia. In a second sense, consciousness refers to the
moment-to-moment contents of a person’s experience, which is the focus of
the current article. ‘Awareness’ is used interchangeably with the second sense
of ‘consciousness’ to denote conscious experience.
Global workspace theory (GWS): a theory that postulates the neural process
underlying conscious experience is a global distribution of information
throughout the brain [74,75].
Lower bound of information: an estimate of the minimal amount of
information encoded in a population signal regarding a specific content of
consciousness.
Microconsciousness: the theory that perceptual experience depends only on
representation in specific sensory brain regions [59].
Multivariate decoding: the attempt to infer something about a person’s mind
(such as the contents of their conscious experience) from distributed patterns
of their brain activity. Multivariate (as opposed to univariate) indicates the
simultaneous measurement of activity from many locations or neurons in the
brain [10,11].
Neural correlate of consciousness (NCC): neural processes that correlate with
many different contents of consciousness are considered to reflect ‘enabling’
or ‘background’ NCCs, such as wakefulness. Neural processes that encode
specific contents of consciousness are termed ‘core’ NCCs [2–5].
1364-6613/$ – see front matter ! 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.tics.2009.02.004 Available online xxxxxx
1
TICS-762; No of Pages 9
Opinion
Trends in Cognitive Sciences Vol.xxx No.x
Figure 1. Encoding of the contents of consciousness in a core NCC. (a) This classic sketch by Ernst Mach shows his first-person experience while he is looking out into his
study. Experiences can vary along several dimensions (shades of brightness, orientations, textures and so on) and hierarchical levels (simple features, intermediate shapes
and complex objects). (b) Each particular class of experiences is presumably encoded in a specific core NCC. For intensities, such NCCs presumably reflect univariate
encoding by the level of activity in certain neurons (e.g. contrast in the spike rate of cells in V1). A different coding scheme is multivariate, in which each sensation is
encoded by a different pattern of activity in a population of neurons. Such multivariate coding schemes can be either sparse, meaning that each different sensation is
encoded by a single, specialized cell (a ‘cardinal’ or ‘grandmother’ cell), or distributed where the entire population of cells participates in encoding of each sensation [24]. (c)
The mapping relationship observed between sensations and states of the core NCC has to fulfil certain requirements. It has to assign one neural state to each sensation
(totality) and assign a different neural state to each sensation (injectivity). A violation of injectivity whereby two different sensations are mapped to the same state of the
core NCC would mean that the sensation could not be decoded from the neural state in a lossless fashion. Please note that the mapping need not be single valued, meaning
that a conscious sensation can be mapped to two different neural states and still be decodable. This allows the possibility of multiple realization of a sensation by different
states of the core NCC (as when, for example, different microscopic constellations of spikes lead to the same average spike rate).
Criteria for mapping conscious experiences onto neural
states
A different view is to think of a core NCC as a neural carrier
for each particular category of experiences (e.g. brightness,
colour and motion) in which these experiences are encoded.
‘Encoding’, here, simply means that there is a stable
mapping between states of the core NCC and conscious
experiences. It does not imply that there is a level at which
the signals encoded in a neural carrier need to be ‘read out’
or ‘interpreted’ by some later level in the system. The
reason for replacing ‘correlation’ with ‘encoding’ is that
encoding provides a more generic and more powerful
framework for identifying core NCCs. This provides a
natural link to recent developments in experimental
neuroscience based on multivariate decoding and pattern
recognition [9–24]. This promising approach could yield a
much tighter link between hypothetical NCCs and conscious experiences.
Box 1. Necessary conditions of conscious experiences
The question of which neural processes are necessary for awareness
is elusive and will, thus, be briefly discussed here. Under normal
conditions, an intact primary visual cortex seems to be a necessary
condition for conscious experience because it is the main entry point
for visual information into the cortical visual system, and damage to
V1 leads to complete blindness in the corresponding region of the
visual field [81] (but for an exception, see Ref. [82]). This could give
the impression that activity in V1 is necessary for any kind of visual
experience. However, V1 can be bypassed and vivid object experiences can be induced by directly stimulating the object-processing
regions of the temporal lobe [83]. Hence, primary visual cortex is not
strictly necessary for conscious experiences of objects and, thus,
cannot be the core NCC for experiences regarding objects. This
shows that it can be difficult to infer necessity from brain lesion data.
To make this clearer, it can help to distinguish between strict and
weak necessity. What is meant by strict necessity of a core NCC of an
experience is that every possible way of achieving a particular
experience requires activity in that core NCC (or, more formally: a
strictly necessary condition for an experience is a condition that is a
necessary part of every set of sufficient conditions for creating this
experience; see, for example, Ref. [84]). A weakly necessary
condition for a specific experience is required only for some ways
2
of achieving that experience. Therefore, activity in V1 is weakly
necessary because it is a necessary part of some causal chains,
including those that normally lead to object experiences whereby
activity passes through V1. Unfortunately, many lesion studies do
not enable one to distinguish whether a loss of awareness of a
particular feature following a lesion to an area [8] means that this
area is necessary in a strict sense or necessary in a weak sense for
awareness of that feature (see, for example, Ref. [9] for an exception
to Ref. [8]). This definition has to be distinguished from the notions
of specific and unspecific conditions (equivalent to the ‘core neural
basis’ and ‘total neural basis’ of conscious experiences in Refs [2–4]).
Specific conditions are those that are required for a particular
content of consciousness but not for others (such as activity in the
fusiform colour area). Unspecific conditions are those that are
required for many or even all contents of consciousness (such as
activity in the brainstem). In the terminology of this paper, the ‘core
neural basis’ is a strictly necessary condition that allows for lossless
decoding of a specific type of experience. Please note that some
authors also allow for the realization of a specific experience by
multiple neural systems, in which case the necessity used here is
dropped in favour of a minimal set of sufficient conditions (for
details, see Refs [2,3]).
TICS-762; No of Pages 9
Opinion
Trends in Cognitive Sciences
Vol.xxx No.x
Figure 2. Encoding of intensity by signals in early visual cortex. (a) The perceived intensity of simple visual features is, presumably, encoded in the firing rate of cells in
early visual cortex. The solid line shows a saturating model function that relates the perceived magnitude of contrast to the physical contrast of a stimulus. The monotonic
relationship between physical and perceptual contrast is maintained in the responses of cells in primary visual cortex (dotted line, fMRI signals from V1 [26]; dashed line,
average single cell in V1 [79]; solid line, model derived from behavioural psychophysics [80]). Thus, it is possible to decode the perceived intensity from signals in this NCC.
The fMRI signal that reflects signals from the entire population of cells in V1 has the advantage of also reflecting the shape of the perceived contrast function and, thus, it
can also explain magnitude relations between stimuli. (b) An important requirement for a core NCC is that it encodes a certain dimension of experience under various
conditions (’invariance’). For example, when a central grating is surrounded by a larger region of the same orientation, the perceived intensity of the central grating is
perceived to be reduced compared to when it is embedded in an orthogonal grating (right two gratings, effect enhanced for clarity). This enables one to test whether signals
in V1 really indicate the same contrast level under different conditions. The magnetoencephalography (MEG) responses from primary visual cortex were different for stimuli
of the same physical contrast that seemed to be different (two stimuli on the right), but they were the same when the physical contrast of the gratings was adjusted to make
them seem to match (bottom left stimulus shows orthogonal matching stimulus M with reduced centre contrast, and bottom right shows parallel standard stimulus S).
Thus, signals in V1 indeed reflect the perceived contrast of stimuli [30]. Importantly, it has been observed repeatedly that intensity encoding in V2 and V3 also closely
matched that in V1 [26,28] and, thus, there are currently several candidate populations that could encode perceived intensity.
A core NCC has to have the representational accuracy to
encode a class of sensations. For this, it needs to fulfil
specific mapping criteria (Figure 1b,c). These guarantee
that each different experience can always be explained by a
different neural state. Take as an example a hypothetical
encoding of brightness and contrast experiences by activity
in early visual cortex [25–30]. To be able to explain all
brightness sensations by activity, say, in V1, this area
needs to adopt a different activity pattern for each distinguishable level of brightness a person can experience. If
V1 does not take on a different state for each experience, it
does not have the representational accuracy to fully encode
the different experiences and, thus, cannot be the core NCC
for conscious brightness perception. Thus, the mapping has
to assign one different and distinguishable neural state to
each different sensation (‘totality’) and it has to be invertible, in the sense that no two sensations can be mapped to
the same neural state (‘injectivity’; Figure 1c). The
criterion of totality, for example, is violated in the case
of MT/V5 because damage to MT/V5 mainly affects fast
(but not slow) motion percepts [31]. Thus, MT/V5 cannot be
the only core NCC in which all motion experiences are
encoded. An example of a total mapping is illustrated in
Figure 2a, in which each contrast experience is associated
with a different state of a neural carrier in V1. The second
criterion of invertibility of a mapping can be assessed by
testing whether a particular dimension of conscious experience can be decoded in a lossless way from a parameter of
neural activity [10,11,30] (Figure 1c). Importantly, the new
approach of multivariate pattern recognition of brain signals enables one to test directly for such decodability
[10,11]. This considerably extends previous studies that
have investigated rudimentary content selectivity [32–35],
such as the encoding of stimulus presence versus absence
[34].
Representational units
A class of experiences can be mapped either to a single
property of neural activity (such as the activity level of
single cells in early visual cortex [27]; Figure 1b, Figure 2)
or to a property defined across large populations of cells in
an area (as is the case for object perception, whereby cells
specialized for different object features are distributed
across a larger brain region, yielding a distributed representation; Figure 1b, Figure 3a) [36,37]. For this reason, the
decoding approach needs to take the spatial response
pattern across the entire population of neurons into
account to reveal the information encoded in a brain
region. It is even conceivable that a core NCC might be
a parameter defined across multiple populations of
neurons (e.g. a correlation pattern between two neural
ensembles [38]). It could also turn out to be a property
of a subpopulation of neurons in a region. It is possible that
the activity of some neurons in an area might show a
constant mapping to an experience but the remaining
neurons fail to do so. Take, for example, the responses of
primary visual cortex to unperceived stimulus features
[20,33,39–41] – for example, that subjects cannot tell the
eye of origin of a monocular stimulus [40]. This unconscious
stimulus feature could be encoded in monocular subpopulations of primary visual cortex, whereas other conscious
stimulus features (such as brightness) could be encoded in
3
TICS-762; No of Pages 9
Opinion
Trends in Cognitive Sciences Vol.xxx No.x
Figure 3. Distributed encoding of percept-based information. (a) In monkeys, complex object features are encoded in a columnar fashion in the inferior temporal cortex.
This indicates a distributed, multivariate coding scheme for objects [37]. (b) Distributed spatial pattern responses obtained from the human object-processing region lateral
occipital complex (LOC) using fMRI reflect the perceived similarity between different objects [47] and, thus, go beyond simple encoding by also explaining the topology of
perceived relationships between different objects. This requires a relationship-preserving or ‘homeomorphic’ mapping between a set of experiences and the states of their
core NCC. (c) Pattern-based decoding of information related to low-level features: the orientation of visual stimuli is encoded in a columnar pattern in primary visual cortex
(from Ref. [19], reprinted with modifications as in Ref. [10] with permission). The left figure shows an optical imaging map of V1, in which each colour indicates a local
predominance of cells of one particular orientation. The spatial resolution of fMRI (black grid) is not sufficient to resolve individual orientation columns. However, when
subjects view gratings of different orientation, a reliable patterning of the fMRI responses from V1 is observed, such that different voxels respond best to different
orientations (second from right) [19]. This can be explained as a result of small fluctuations in the density of cells specialized for different orientations in different fMRI
voxels (second from left). Using multivariate decoding techniques, the orientation-related information encoded in these spatial patterns can be read out, making it possible
to reconstruct the orientation of stimuli, despite the lack of spatial resolution of fMRI signals (right; for details, see Ref. [19]).
binocular subpopulations. Also, it has to be ensured that
the observed sources of decoded information are not
physiologically epiphenomenal (e.g. subthreshold activity).
In early visual areas, conscious versus unconscious perception is mainly reflected in blood-oxygenation-level dependent functional magnetic resonance imaging (BOLD fMRI)
signals and local field potentials, particularly at lower
frequencies, but, interestingly, not in spiking activity
[42,43]. This shows that great care is needed when interpreting neural signals [44] and their potential implications
for the NCC.
Invariance across conditions
An important further criterion is that the decoding needs
to hold up under several different experimental conditions.
For example, brightness and contrast can be probed under
various contextual conditions (lighting, context, masking,
and so on) [27,29,30]. Different physical stimuli leading
to the same brightness sensations would need to be
4
invariantly mapped to the same states of the core NCC
under all conditions [30] (Figure 2b).
Lossless decoding versus correlation
The ideal mapping criteria employed here are very strict
and go beyond the simple correlation often measured
between consciousness and brain activity. Most studies
report only partial, rather than perfect, correlation (see
Refs [32,33] for examples). This requires only that a part of
the variance in neural signals is explained by consciousness. Such a lax criterion yields too many neural populations that correlate with consciousness and, thus, could
all potentially be the core NCC where a specific content is
encoded. By contrast, a lossless decoding of conscious
perception from a corresponding brain signal would
require a perfect correlation. Another problem is that
correlation assumes a linear mapping between brain states
and sensations, which is not necessarily required for perfect decoding (i.e. an increase in brightness must not
TICS-762; No of Pages 9
Opinion
necessarily be encoded in a linear or even monotonic
increase in the spike rate of cells). Instead, there could
be a more complicated or even arbitrary mapping that can
be assessed using information-theoretic measures, such as
nonlinear multivariate decoding and mutual information
[10–13]. These measures can be used to reveal a constant
mapping of one variable to another without requiring that
the mapping be systematic.
Encoding of similarity
In an extension of the previous point, however, a systematic mapping between states of the core NCC and conscious
experiences [2,45] would have additional explanatory
power. It could explain how similarities or relationships
between different sensations are encoded by similar
relationships between states of the core NCC, as in the
case of perceptual spaces [14,15,46,47] (Figure 3b). Some
studies even show that it might be possible to provide a
complete model for the encoding of a particular category of
experiences in a core NCC [16,17].
Physical versus perceived features
In most cases, our knowledge about a person’s sensations is
based on psychophysical judgements about external
physical stimuli [48]. However, it is important to clarify
whether a particular brain area has information about the
physical stimulus features (e.g. the light energy reflected
by an object) or about a person’s conscious experiences (e.g.
the perceived brightness of the surface). The maximum
information about physical stimuli should be reconstructable from the retina, where graded receptor potentials
closely follow physical properties of incoming signals. However, the perceived properties of objects are often different
from the physical properties, as is the case in contextual
interactions in brightness and colour perception
[27,29,30,49]. For example, the encoding of chromatic signals in the retina and in V1 does not match the subject’s
conscious colour perception that exhibits colour constancy
across different illumination conditions [49]. Thus, activity
patterns in the retina and V1 cannot explain colour perception because there is no constant mapping between
states in these areas and individual colour percepts (see
Ref. [50] for a similar argument). In contrast, a region in
the temporal cortex of macaques contains cells that closely
match the four non-reducible ‘unique hues’ of colour perception [51]. However, even if V1 does not encode colour
sensations, it could still encode other, simple features of
conscious experience, such as brightness or contrast sensations.
Lossless decoding and bridge laws
As mentioned previously, the quality with which a neural
signal encodes a particular conscious experience can be
probed by attempting to decode the contents of consciousness from that neural signal. As also mentioned previously, the term ‘decoding’ (as used here) refers to
brute-force statistical techniques for testing whether a
systematic mapping holds between states of a core NCC
and a category of experiences. This is achieved by testing
whether a classifier can learn to assign labels (i.e. sensations) to brain states correctly [10,11]. However, it does
Trends in Cognitive Sciences
Vol.xxx No.x
not imply that brain signals are ‘interpreted’ at a semantic level or that a homunculus is required to decode the
signals. Ideally, for a given class of sensations (e.g. brightness sensations), one would find only one particular
parameter of neural activity (e.g. spike rate in V1) that
allows for a lossless decoding. Then, sensations of this
particular type could only be explained by this activity,
and every time a person has a brightness sensation, one
could argue that this state is directly encoded in the spike
rate of V1 (however, see Ref. [2] for potential exceptions).
Thus, the mapping can be used to establish ‘rules of
correspondence’ or ‘bridge laws’ that enable one to link
statements about the core NCC to statements about
conscious percepts [52]. Currently, however, the lossless-decoding criterion has to be seen as an empirical
ideal because of limitations in spatial and temporal
resolution of non-invasive brain-imaging techniques
and limitations in coverage of cell populations, brain
regions and brain-activity parameters in many electrophysiological studies (but see Refs [42,43]). However, to
an approximation, the brain parameter currently
enabling the highest decoding accuracy can play the part
of a candidate NCC for a specific sensation. Such decoding-based approaches can now also be performed using
distributed ensembles of fMRI voxels [9–17,19–23]
(Figure 3c), thus providing, for the first time, a noninvasive way of assessing the information encoded in
various cortical areas. However, it is important to realize
that such methods can only reveal a lower bound of
information encoded in a brain region.
The link between encoding and consciousness
The ideal criteria outlined earlier provide an important
contribution to the search for core NCCs (see supplementary material online for a full list of criteria and
corresponding studies). They enable one to test whether
a specific candidate NCC fulfils the necessary requirements to encode a specific class of sensations. Importantly,
however, the criteria also allow one to address the question of what happens to a core NCC when the corresponding feature it encodes enters consciousness. At first, it
might seem obvious that when a conscious percept fades in
and out of consciousness (say, during binocular rivalry),
the content would need to fade in and out of the corresponding core NCC (Figure 4a, second from left). In line
with this, several invasive electrophysiological studies in
humans and monkeys have shown that content-selective
processes in the brain are modulated by consciousness of
the corresponding contents [53–56], possibly even in an
all-or-nothing manner [57]. For example, it has been
demonstrated repeatedly that during binocular rivalry
and flash suppression, content-selective cells in higher
visual areas modulate their activity when their preferred
object fades in and out of consciousness [53,54,58]. This is
consistent with the microconsciousness theory that postulates that all that is required for representation of a
content in consciousness is that it is encoded in a specialized perceptual processing module [59]. However, there
are also several observations that complicate this
straightforward equation of encoding and consciousness,
as outlined below. These only become clear when one takes
5
TICS-762; No of Pages 9
Opinion
Trends in Cognitive Sciences Vol.xxx No.x
Figure 4. Encoding of information and conscious perception. (a) Different models can be used to explain why a stimulus fails to reach consciousness. When encoding of a
stimulus and access or read out are both intact, a stimulus is consciously seen. A loss of conscious perception could be due to a failure of encoding of the stimulus in a core
NCC, a failure of access or a failure of re-representation of a stimulus in a global workspace. (b) Attention. It is possible to decode which of two superimposed visual line
images a person is currently covertly attending to from pattern signals in visual areas V1 to V4 [19]. (c) Consciousness. Even when line stimuli (T) are rendered invisible by
rapidly exchanging them with a mask (M), their orientation can still be recovered from signals in primary visual cortex, indicating that V1 is not the core NCC of orientation
sensations [20]. Interestingly, there is no evidence for information related to invisible orientation stimuli in higher visual areas beyond V1, indicating that V2 and V3 match
conscious perception and, thus, one of these areas might contain the core NCCs of orientation sensations. (d) Left: the information about visual object stimuli that can be
decoded from population signals in monkey temporal cortex is strongly reduced when a stimulus is rendered invisible by backward masking. Right: the information about a
stimulus feature encoded in a brain region can be estimated by comparing the correlation between the responses to the repeated presentation of the same image (‘same
category’) to the correlation between different images [9,23]. This can be used to compare the information encoded when objects are seen, as opposed to unseen. When
human subjects view complex shapes, the information that can be encoded in V1 does not depend on whether they have successfully seen it or not (‘correct’ versus
‘incorrect’), whereas the information encoded in object-processing LOC does. This has two important implications for the relationship between information and awareness:
first, conscious object perception might fail even though the simple features providing input to the object-processing network are fully intact; second, the information
encoded in LOC closely matches trial-by-trial fluctuations in conscious perception.
a more fine-grained, content-based view of the encoding of
visual information in hypothetical core NCCs.
First, it needs to be assessed whether a hypothesized
core NCC that modulates with awareness has the representational accuracy to fully encode all experiences of the
type in question. For example, when a face enters awareness during binocular rivalry, the activity in the fusiform
face area (FFA) is increased [58], which could be taken to
6
indicate that encoding in the core NCC for faces is decisive
for regulating whether a face is seen or not. However, it has
been debated whether the face-related information
encoded in FFA is sufficient to explain conscious perception
of face identity, in contrast to a region more anterior in the
temporal cortex [22,60,61]. Simply monitoring activity in
specialized processing regions (such as FFA and parahippocampal place area [PPA]) allows one to track contents
TICS-762; No of Pages 9
Opinion
only in a coarse way and does not reveal whether such a
content-selective region contains the core NCC encoding
the fine-grained details of conscious experience (such as
the faces of specific individuals).
Second, consciousness is a multi-level phenomenon that
spans from simple sensations of a distribution of brightness in the visual field to complex object perception. The
fact that signals in higher but not lower visual areas match
conscious perception of objects during rivalry has long been
taken as evidence that consciousness occurs late in the
visual system [2,33]. However, this neglects the multi-level
nature of consciousness that includes not only the highlevel object category but also the fine-grained spatial layout of shades of brightness and colours of which an object is
composed (Figure 1). Importantly, high-level object-processing regions do not encode information about these lowlevel features because they respond invariantly when the
same object is defined by different features [62,63]. This
invariance of high-level responses to low-level features is
an important computational achievement of higher-order
regions, but it implies that the information about simple
contents of consciousness such as brightness and contrast
is lost so that they have to be encoded in other neural
populations, presumably in earlier brain regions.
Third, it has been shown that cells in MT can signal the
conscious perception of their preferred direction of motion
under one experimental condition but fail to do so under a
different experimental condition [55]. This means a simple
equation between encoding in content-selective cells and
consciousness of a corresponding feature cannot be true.
Furthermore, content-specific brain regions that are modulated by consciousness can also be activated by unattended
and even unconscious, invisible contents [64–66]. This
would indicate that the encoding of signals in a contentselective region does not always lead to conscious experience of the corresponding content.
Fourth, the encoding of sensory signals can be differentially affected by visibility in early and high-level visual
areas. Using multivariate decoding of fMRI signals
(Figure 3c), it has now been shown that V1 continues to
encode information about the orientation of simple stimuli
when the stimulus is unattended or even made invisible by
masking [19,20] (Figure 4b,c). If primary visual cortex
encodes information about orientation stimuli that fail
to reach consciousness, the states of primary visual cortex
do not provide the mapping needed to explain our simple
sensations, as has been debated previously [32,33,39–41].
The findings regarding objects are different. Reliable information about conscious object percepts is encoded at the
cortical site of high-level object recognition [9,14–
16,23,47,67] (Figure 4d). However, when an object is rendered invisible by masking or rivalry, the information
encoded about its identity can be strongly diminished
[67,68]. Similarly, trial-by-trial fluctuations in the visibility of objects are reflected in the information encoded
in the human object-processing area [24] (Figure 4d). Thus,
consciousness in the case of high-level object perception
seems to strongly disrupt the encoding of objects in the
corresponding core NCC, whereas in early visual areas, the
information about low-level visual features can remain
intact even though a stimulus fails to be seen. This
Trends in Cognitive Sciences
Vol.xxx No.x
indicates that different levels of information can be differentially affected by changes in the level of visibility. Information about the simple constituent features of objects in
early visual areas might remain unaffected by changes in
their visibility, even when the neural encoding of the
objects themselves is lost.
Taken together, a closer look at the detailed encoding
reveals that the representation in modality-specific
regions is not sufficient to explain why perceptual information enters consciousness. This indicates that
additional processes are required that regulate which
contents gain access to consciousness [4,69] (Figure 4a,
second from right). These mechanisms are believed to be
closely linked to attention [70] but clearly involve more
than attentional selection [71–73]. A hypothetical mechanism that regulates access to consciousness is described
by the global workspace theory (GWS). The GWS postulates that encoding of information in content-specific and
modality-specific brain regions is not sufficient and that
information has to be globally distributed throughout the
brain to reach consciousness [74,75]. In line with this,
when a visual stimulus crosses the threshold to consciousness, fMRI signals are increased not only in content-selective visual brain areas but also in specific prefrontal
brain regions (such as dorsolateral prefrontal cortex and
medial prefrontal cortex) [76,77]. However, to test the
hypothesis of global distribution of sensory information,
one would need to investigate the degree to which the
processes in prefrontal cortex (PFC) indeed receive such
information when it reaches consciousness. Instead, the
activity in PFC could reflect purely unspecific processes
without content-specific encoding.
For many processes that correlate with consciousness, it
remains unclear the degree to which they are content
specific. This makes it difficult to test the global workspace
theory. This is a particular problem for fMRI studies, in
which detailed information on content specificity of neural
processes is only rarely available, particularly for supramodal cortical areas. As outlined above, the recent
advances in decoding-based analyses of brain signals
now enable one to directly probe the representational
accuracy of brain signals recorded from different areas
[10,11] and monitor how the encoding of information
changes under varying levels of conscious access to visual
information. This could help reveal whether the global
distribution of information occurs in a graded manner,
as hypothesized previously [35]. It would be particularly
informative to attempt to decode visual information from
activity in prefrontal brain regions and assess whether
more information is available in PFC when a stimulus
reaches consciousness. Importantly, such studies could
also help distinguish between different readings of the
GWS. In one variant, the core NCC is in modality-specific
cortex and the global distribution of information is an
‘additional process’ that regulates whether a sensory
representation enters awareness. In a different variant,
one might assume that information is completely (rather
than partially) re-represented in PFC when it reaches
consciousness. This would indicate that the core NCC that
fully spans a perceptual space for the currently relevant
perceptual features is located in PFC, rather than in
7
TICS-762; No of Pages 9
Opinion
modality-specific areas. These and similar issues require a
decoding-based approach that assesses the degree to which
the processes involved in conscious perception encode and
re-distribute sensory information.
Taken together, multivariate decoding provides a novel
and powerful framework for identifying core NCCs and for
investigating the link between neural encoding and consciousness. Further work is needed to re-examine how the
decoding of information can inform models of consciousness. The key questions that now need to be addressed
from an information-theoretic perspective are: Which
neural core NCCs encode specific dimensions of conscious
experience most veridically? To what degree do different
manipulations of awareness (masking, rivalry and attention) affect the encoding of conscious experiences? The
GWS needs to be further tested by assessing whether
the neural activity found in prefrontal cortex when stimuli
reach awareness contains sensory information. This could
reveal a global distribution or possibly even a re-representation of information. Thus, studying the neural encoding of contents will be a key stage in unravelling the neural
mechanisms of consciousness.
Acknowledgements
The author thanks Michael Pauen, Jochen Braun and Frederique de
Vignemont for their valuable comments on the manuscript. This work
was funded by the Max Planck Society, the German Research Foundation
and the Bernstein Computational Neuroscience Program of the German
Federal Ministry of Education and Research.
Appendix A. Supplementary data
Supplementary material associated with this article can be
found at doi:10.1016/j.tics.2009.02.004.
References
1 Mach, E. (1886) Beiträge zur Analyse der Empfindungen. Fischer
2 Chalmers, D. (2000) What is a neural correlate of consciousness? In
Neural Correlates of Consciousness: Conceptual and Empirical
Questions (Metzinger, T., ed.), pp. 17–40, MIT
3 Koch, C. (2004) The Quest for Consciousness: A Neurobiological
Approach. Roberts
4 Block, N. (2007) Consciousness, accessibility and the mesh between
psychology and neuroscience. Behav. Brain Sci. 30, 481–548
5 Tootell, R.B. et al. (1995) Functional analysis of human MT and related
visual cortical areas using magnetic resonance imaging. J. Neurosci.
15, 3215–3230
6 Salzman, C.D. et al. (1992) Microstimulation in visual area MT:
effects on direction discrimination performance. J. Neurosci. 12,
2331–2355
7 Pascual-Leone, A. and Walsh, V. (2001) Fast backprojections from the
motion to the primary visual area necessary for visual awareness.
Science 292, 510–512
8 Zeki, S. (1991) Cerebral akinetopsia (visual motion blindness). A
review. Brain 114, 811–824
9 Haxby, J.V. et al. (2001) Distributed and overlapping representations of
faces and objects in ventral temporal cortex. Science 293, 2425–2430
10 Haynes, J.D. and Rees, G. (2006) Decoding mental states from brain
activity in humans. Nat. Rev. Neurosci. 7, 523–534
11 Norman, K.A. et al. (2006) Beyond mind-reading: multi-voxel pattern
analysis of fMRI data. Trends Cogn. Sci. 10, 424–430
12 Muller, K.R. et al. (2001) An introduction to kernel-based learning
algorithms. IEEE Trans. Neural Netw. 12, 181–201
13 Kriegeskorte, N. et al. (2006) Information-based functional brain
mapping. Proc. Natl. Acad. Sci. U. S. A. 103, 3863–3868
14 O’Toole, A.J. et al. (2005) Partially distributed representations of
objects and faces in ventral temporal cortex. J. Cogn. Neurosci. 17,
580–590
8
Trends in Cognitive Sciences Vol.xxx No.x
15 Haushofer, J. et al. (2008) Multivariate patterns in object-selective
cortex dissociate perceptual and physical shape similarity. PLoS Biol.
6, e187
16 Kay, K.N. et al. (2008) Identifying natural images from human brain
activity. Nature 452, 352–355
17 Thirion, B. et al. (2006) Inverse retinotopy: inferring the visual content
of images from brain activation patterns. Neuroimage 33, 1104–1116
18 Quiroga, R.Q. et al. (2005) Invariant visual representation by single
neurons in the human brain. Nature 435, 1102–1107
19 Kamitani, Y. and Tong, F. (2005) Decoding the visual and subjective
contents of the human brain. Nat. Neurosci. 8, 679–685
20 Haynes, J.D. and Rees, G. (2005) Predicting the orientation of invisible
stimuli from activity in human primary visual cortex. Nat. Neurosci. 8,
686–691
21 Haynes, J.D. and Rees, G. (2005) Predicting the stream of
consciousness from activity in human visual cortex. Curr. Biol. 15,
1301–1307
22 Kriegeskorte, N. et al. (2007) Individual faces elicit distinct response
patterns in human anterior temporal cortex. Proc. Natl. Acad. Sci. U. S.
A. 104, 20600–20605
23 Williams, M.A. et al. (2007) Only some spatial patterns of fMRI
response are read out in task performance. Nat. Neurosci. 10, 685–686
24 Quiroga, R.Q. et al. (2008) Sparse but not ‘grandmother-cell’ coding in
the medial temporal lobe. Trends Cogn. Sci. 12, 87–91
25 Ress, D. and Heeger, D.J. (2003) Neuronal correlates of perception in
early visual cortex. Nat. Neurosci. 6, 414–420
26 Boynton, G.M. et al. (1999) Neuronal basis of contrast discrimination.
Vision Res. 39, 257–269
27 Rossi, A.F. et al. (1996) The representation of brightness in primary
visual cortex. Science 273, 1104–1107
28 Haynes, J.D. et al. (2004) Responses of human visual cortex to uniform
surfaces. Proc. Natl. Acad. Sci. U. S. A. 101, 4286–4291
29 Cornelissen, F.W. et al. (2006) No functional magnetic resonance
imaging evidence for brightness and color filling-in in early human
visual cortex. J. Neurosci. 26, 3634–3641
30 Haynes, J.D. et al. (2003) Neuromagnetic correlates of perceived
contrast in primary visual cortex. J. Neurophysiol. 89, 2655–2666
31 Zihl, J. et al. (1983) Selective disturbance of movement vision after
bilateral brain damage. Brain 106, 313–340
32 Tong, F. (2003) Primary visual cortex and visual awareness. Nat. Rev.
Neurosci. 4, 219–229
33 Rees, G. et al. (2002) Neural correlates of consciousness in humans.
Nat. Rev. Neurosci. 3, 261–270
34 Sergent, C. et al. (2005) Timing of the brain events underlying access to
consciousness during the attentional blink. Nat. Neurosci. 8, 1391–
1400
35 Kouider, S. et al. (2007) Cerebral bases of subliminal and supraliminal
priming during reading. Cereb. Cortex 17, 2019–2029
36 Bonhoeffer, T. and Grinvald, A. (1991) Iso-orientation domains in cat
visual cortex are arranged in pinwheel-like patterns. Nature 353, 429–
431
37 Fujita, I. et al. (1992) Columns for visual features of objects in monkey
inferotemporal cortex. Nature 360, 343–346
38 Engel, A.K. and Singer, W. (2001) Temporal binding and the neural
correlates of sensory awareness. Trends Cogn. Sci. 5, 16–25
39 Gur, M. and Snodderly, D.M. (1997) A dissociation between
brain activity and perception: chromatically opponent cortical
neurons signal chromatic flicker that is not perceived. Vision Res.
37, 377–382
40 Blake, R. and Cormack, R. (1979) On utrocular discrimination. Percept.
Psychophys. 26, 53–68
41 He, S. et al. (1996) Attentional resolution and the locus of visual
awareness. Nature 383, 334–337
42 Maier, A. et al. (2008) Divergence of fMRI and neural signals in V1
during perceptual suppression in the awake monkey. Nat. Neurosci. 11,
1193–1200
43 Wilke, M. et al. (2006) Local field potential reflects perceptual
suppression in monkey visual cortex. Proc. Natl. Acad. Sci. U. S. A.
103, 17507–17512
44 Logothetis, N.K. (2008) What we can do and what we cannot do with
fMRI. Nature 453, 869–878
45 Müller, G.E. (1896) Zur Psychophysik der Gesichtsempfindungen.
Zeitschrift für Psychologie und Physiologie der Sinnesorgane 10, 1–82
TICS-762; No of Pages 9
Opinion
46 Palmer, S.E. (1999) Color, consciousness and the isomorphism
constraint. Behav. Brain Sci. 22, 923–943
47 Edelman, S. et al. (1998) Toward direct visualization of the internal
shape representation space by fMRI. Psychobiology 26, 309–321
48 Fechner, G.T. (1860) Elemente der Psychophysik. Breitkopf und Härtel
49 Conway, B.R. and Livingstone, M.S. (2006) Spatial and temporal
properties of cone signals in alert macaque primary visual cortex. J.
Neurosci. 26, 10826–10846
50 Crick, F. and Koch, C. (1995) Are we aware of neural activity in primary
visual cortex? Nature 375, 121–123
51 Stoughton, C.M. and Conway, B.R. (2008) Neural basis for unique
hues. Curr. Biol. 18, R698–R699
52 Nagel, E. (1979) The Structure of Science. Hackett
53 Kreiman, G. et al. (2002) Single-neuron correlates of subjective vision
in the human medial temporal lobe. Proc. Natl. Acad. Sci. U. S. A. 99,
8378–8383
54 Leopold, D.A. and Logothetis, N.K. (1999) Multistable phenomena:
changing views in perception. Trends Cogn. Sci. 3, 254–264
55 Maier, A. et al. (2007) Context-dependent modulation of single neurons
in primate visual cortex. Proc. Natl. Acad. Sci. U. S. A. 104, 5620–5625
56 Reddy, L. et al. (2006) A single-neuron correlate of change detection
and change blindness in the human medial temporal lobe. Curr. Biol.
16, 2066–2072
57 Quiroga, R.Q. et al. (2008) Human single-neuron responses at the
threshold of conscious recognition. Proc. Natl. Acad. Sci. U. S. A.
105, 3599–3604
58 Tong, F. et al. (1998) Binocular rivalry and visual awareness in human
extrastriate cortex. Neuron 21, 753–759
59 Zeki, S. and Bartels, A. (1999) Towards a theory of visual
consciousness. Conscious. Cogn. 8, 225–259
60 Leopold, D.A. et al. (2006) Norm-based face encoding by single neurons
in the monkey inferotemporal cortex. Nature 442, 572–575
61 Tsao, D.Y. et al. (2006) A cortical region consisting entirely of faceselective cells. Science 311, 670–674
62 Sáry, G. et al. (1993) Cue-invariant shape selectivity of macaque
inferior temporal neurons. Science 260, 995–997
63 Kourtzi, Z. and Kanwisher, N. (2001) Representation of perceived
object shape by the human lateral occipital complex. Science 293,
1506–1509
64 Rees, G. et al. (2002) Neural correlates of conscious and unconscious
vision in parietal extinction. Neurocase 8, 387–393
65 Moutoussis, K. and Zeki, S. (2002) The relationship between cortical
activation and perception investigated with invisible stimuli. Proc.
Natl. Acad. Sci. U. S. A. 99, 9527–9532
Trends in Cognitive Sciences
Vol.xxx No.x
66 Fang, F. and He, S. (2005) Cortical responses to invisible objects in
the human dorsal and ventral pathways. Nat. Neurosci. 8, 1380–
1385
67 Rolls, E.T. et al. (1999) The neurophysiology of backward visual
masking: information analysis. J. Cogn. Neurosci. 11, 300–311
68 Sterzer, P. et al. (2008) Fine-scale activity patterns in high-level visual
areas encode the category of invisible objects. J. Vis. 8, 1–12
69 Block, N. (1995) On a confusion about a function of consciousness.
Behav. Brain Sci. 18, 227–287
70 Simons, D.J. and Rensink, R.A. (2005) Change blindness: past, present,
and future. Trends Cogn. Sci. 9, 16–20
71 Braun, J. and Julesz, B. (1998) Withdrawing attention at little or
no cost: detection and discrimination tasks. Percept. Psychophys. 60,
1–23
72 Lamme, V.A.F. (2003) Why visual attention and awareness are
different. Trends Cogn. Sci. 7, 12–18
73 Koch, C. and Tsuchiya, N. (2007) Attention and consciousness: two
distinct brain processes. Trends Cogn. Sci. 11, 16–22
74 Baars, B.J. (1997) In the Theater of Consciousness: The Workspace of
Mind. Oxford University Press
75 Dehaene, S. and Naccache, L. (2001) Towards a cognitive neuroscience
of consciousness: basic evidence and a workspace framework.
Cognition 79, 1–37
76 Dehaene, S. et al. (2001) Cerebral mechanisms of word masking and
unconscious repetition priming. Nat. Neurosci. 4, 752–758
77 Haynes, J.D. et al. (2005) Visibility reflects dynamic changes of
effective connectivity between V1 and fusiform cortex. Neuron 46,
811–821
78 Zeman, A. (2007) Consciousness. In The Blackwell Companion to
Consciousness (Velmans, M. and Schneider, S., eds), pp. 703–711,
Blackwell
79 Albrecht, D.G. and Hamilton, D.B. (1982) Striate cortex of monkey and
cat: contrast response function. J. Neurophysiol. 48, 217–237
80 Legge, G.E. and Foley, J.M. (1980) Contrast masking in human vision.
J. Opt. Soc. Am. 70, 1458–1471
81 Holmes, G. (1918) Disturbances of vision by cerebral lesions. Br. J.
Ophthalmol. 2, 353–384
82 Zeki, S. and Ffytche, D.H. (1998) The Riddoch syndrome: insights into
the neurobiology of conscious vision. Brain 121, 25–45
83 Lee, H.W. et al. (2000) Mapping of functional organization in
human visual cortex: electrical cortical stimulation. Neurology 54,
849–854
84 Mackie, J. (1974) The Cement of the Universe. A Study on Causation.
Clarendon
9