Active Perception
Self-motion, Performative Action, and the Episodic
Brain
George Kampis
Fujitsu Chair of Complex Systems
Japan Advanced Institute of Science and Technology
Tatsunokuchi, Ishikawa
Talk Prepared for the
Tamagawa University Brain Science Reserch Center
(held at Tamagawa University March 14, 13:00, 2003.)
Brief Personal History
M.Sc. electrical engineering,
specialization in technical physics for microelectronics
1981-82 young res. fellow, Central
Res.Inst for Physics, Hung.Acad.Sci., software development for "silicon
compiler"
1983-84 assistant, Deparment of Ethology,
ELTE Budapest, experimental design and statistics
1984-87 PhD student, evolution theory,
cellular automata, computer models in biology (e.g. protein nets)
PhD, biology
1988-94 senior res. fellow, Department
of Ethology, meaning and origin of dynamical models in biology and cognition
1990-91 A.v.Humboldt-Fellow,
Universtiy of Tubingen, Germany (O.E. Rossler)
1991-93 Research Fellow,
University of Tubingen, DFG Project "New Ideas in Complexity"
1994
visiting professor, Dept.og Logic and Methodology, U. Basque Country,
Spain, A. Moreno
D.Sc., philosophy
1995- chairman
of dept, History and Philosophy of Science, causality, general philosophy
of science, Darwin, mental models
Recent Works (2002-2003)
complex systems:
G. Kampis (2002): "Complexity as a Cue to the Mind" (commentary
on Tsuda)
Behavioral and Brain Sciences,
submitted, http://www.jaist.ac.jp/~g-kampis/BBS/Complexity_as_a_Cue_to_the_Mind.html
G. Kampis (2002): "Chaotic Itinerancy from a Multi-Disciplinary Perspective"
Chaos and Complexity Letters,
submitted, http://www.jaist.ac.jp/~g-kampis/Chaos/CI.pdf
G. Kampis (2003): “Complexity and the Mind”, invited talk at GLCA
Meeeting (Kalamazoo College, MI), 2003. Feb. 21.
abstract: http://www.jaist.ac.jp/~g-kampis/USA/Four_Talks.html,
full text: http://www.jaist.ac.jp/~g-kampis/USA/Complexity_and_Mind.html
causality:
G. Kampis (2002): “Causal Intentionality”, lecture at: Intentionality:
Past and Future (Miskolc, Hungary), 2002 June 21-23.
abstract: http://hps.elte.hu/intentionality/Abstracts.htm#Kampis,
full text: http://www.jaist.ac.jp/~g-kampis/Intentionality/Causal_Intentionality.html
G. Kampis (2003): "Causal Depth and the Modal View of Causality",
IUHPS/DLMPS World Conference, Oviedo, Spain, August 8-13 2003.
abstract: http://www.jaist.ac.jp/~g-kampis/Oviedo/CausalDepth.html
evolution:
G. Kampis (2002): "Towards an Evolutionary Technology", Nikkei Science
(Japanese translation to be submitted in 2003)
http://www.jaist.ac.jp/~g-kampis/EvoTech/Towards.html
G. Kampis (2002): "A Causal Model of Evolution", Proc. 4th
Asia-Pacific Conference on Simulated Evolution and Learning (SEAL '02), pp.
836-839.
http://www.jaist.ac.jp/~g-kampis/SEAL02/A_Causal_Model_of_Evolution.htm
G. Kampis (2003): “A Causal Model of Evolution”, invited seminar
lecture, Center for the Study of Complex Systems, U. Michigan, Ann Arbor,
. Feb. 20.,
abstract: http://www.pscs.umich.edu/calendar/abstracts-w03.html#Kampis,
full text: http://www.jaist.ac.jp/~g-kampis/USA/Causal_Model.html
active perception:
G. Kampis (2002): “Dynamic Models of the Mind”, lecture at: 7th
Tamagawa Dynamic Brain Forum (Visegrad, Hung.), 2002 September 8-11.
http://www.jaist.ac.jp/~g-kampis/DBF/Dynamic_Models_of_the_Mind_abstract.htm,
full text: http://www.jaist.ac.jp/~g-kampis/DBF/Dynamic_Models_of_the_Mind.html
G. Kampis: “Knowledge Without Knowing”, lecture at: World Congress of Sociology
(Brisbane, AU), 2002 July 7-15.
abstract: http://www.jaist.ac.jp/~g-kampis/Brisbane/Knowledge_Without_Knowing.htm
G. Kampis – L. Gulyas (2003): “Causal Structures in Embodied Systems",
ERCIM News (invited paper, in print)
http://www.jaist.ac.jp/~g-kampis/ERCIM/Kampis-Gulyas.html
general theory of cognition:
Graduate course, title: “A Methodology of Human Knowledge” (2002
Fall Semester, in JAIST; to be published in Japanese
in 2003-3004)
Syllabus: http://www.jaist.ac.jp/~g-kampis/Course/Syllabus.html,
Lecture notes available in fulltext: http://www.jaist.ac.jp/~g-kampis/Human_Knowledge.html
Other Research Topics
- animal intentionality research with respect to agency detection (especially
in dog and cat)
- evolutionary linguistics from a sociolinguistic perspective (i.e. distributed,
socially structured and heterogeneous; dialects)
- mental models
- Wittgenstein's theory of "family resemblance" in various contexts
Abstract of Talk
Cognitive science and cognitive neuroscience increasingly recognize
the significance of interactions with the environment. This puts perception
and action in the focus of cognition. However, the integration of the two
aspects is missing, or is at best partial. In lack of that, current approaches
to cognition and perception tend to be limited to a "traveling eye" framework.
Based on an ethological model of behavior, and on considerations from causality,
I develop a model of active perception. I discuss active perception's features,
and its requirements for the mind and the brain.
Active perception will be understood here as a process where various self-actions
and the percepts which are directly or indirectly evoked by them form a single,
complex spatio-temporal unity, called an episode. I characterize episodes
from various points of view: as combinations of multiple modalities,
as causal structures, and as basis of narratives. Finally, I discuss implications
for dynamical brain theory, and for issues like interpretation through restricted
perception, such as viewing or speech audition.
Active Perception
a hypothesis,
mostly about the mind, but also puts strong requirements for brain
theory
relevant:
"Brain for the Mind" thesis (Kampis 2002)
the brain is studied because of the mind (learning, memory, language, perception,
etc are all known from the 1st person + psychology)
consequence: concepts about the mind influence (or determine) concepts
about the brain
in order to get to the brain, we have to know what is in the mind
What is cognition? Kinds of Minds
reasoning, symbol manipulation - machine,
"robot" metaphor
motion control and associative learning
- insect metaphor
coordination and detection of action and manipulation - vertebrate,
with a limbic system
(mental models and their secondary or "symbolic" use - higher
organism, with neocortex)
The Origin of the Idea of Cognition in Percepcion
Gestalt phenomena (G. Kanizsa, W.Koehler, etc.; Kuhn's paradigms, or "seeing-as")
(illustrations from public domain)
such "higher perception" is active, but in a limited sense.
it is not just a reception of input, but requires (re- or pre-)processing
Perception in Cognitive Science (Early History)
classically, treated from the point of view of the result, which is a representation
(which produces symbols)
accordingly, "perception" is coding
(from Kampis 1991)
(from Ars Magna Lulli)
(but what interprets a symbol? how does it arise? is it symbolic in itself?
a road to Wittgenstein, to pictures, and animations)
Perception in Cognitive Science (More Recent)
(Rosenblatt's Perceptron; backpropagation network)
perception is modular, based on preselected (coded) input, with learning-based
categorization
e.g. "rat's brain controlled robot":
perception is in input encoding
The Role of the Body in Cognition
(1) general considerations
(2) "active perception" today
(1) Embodiment
Varela Thompson and Rosch 1991
Brooks 1989, Steels 1989
van Gelder and Port 1995
Sporns, O. (2002) "Embodied Cognition".In: MIT Handbook of Brain Theory
and Neural Networks, M. Arbib, Ed., MIT Press, Cambridge, MA.
(S. Karmiloff-Smith, E. Bates, G. Westerman, M. Sheutz, etc.)
Traditionally, cognition has been considered as a ``mental processes'' only
domain. Recently, however, there is a growing conscensus that cognition should
be ``embodied'', i.e. it emerges from physical interaction with the world
through a body with given perceptual and motoric abilities. Terms like ``emergence'',
``enaction'', ``grounding'', and ``situatedness'' are often used (quote
from Berthouze and Tijsseling 2001).
importance of body and real-world embeddednes; relation to phenomenology
(M. Heidegger and M. Merleau-Ponty)
anti-representationism ("emerges from interaction")
dynamical models of mind/world - brain/world coupled systems
(2) "Active Perception"
locomotion
moving sensory system (e.g. approach) and provoked change of input
resolution of ambiguity such as geometric distance (triangulation, "optical
flow", recent results on insects such as bees)
Ballard, D.H. (1991): Animate Vision, Artificial Intelligence 48,
57-86.
Bajcsy, R. (1988): Active perception, Proc. IEEE 76, 996-1005.
recent combination with embodiment:
no distinction between input, action, motor response, efferent output (limited
on sensors)
perfect symmetry between organism and environment (Jarvilehto, A.Clark)
sensing not a direct source of knowledge: it happens
when a given element of the environment gets involved in the
sensorimotor loop
model: coupled dynamic recognizers/optimizers (O.E. Rossler, T.Ikegami,
J. Tani)
(Illustrations from J. Tani's papers)
this is a paradigm of attention, stimulus selection, stimulus change by
locomotion; sensitization
active vision mostly means foveation, saccadic integration, visual field
stabilization etc.
typically even theory speaks of efferent as opposed to afferent action
if combined with motion ("eploration learning", Tani), updating of visual
field according to position: how multiple positions help geometry (depth etc)
(illustration from public domain)
limited to Gibsonian motion-for-vision, vision-for motion (after J.J.
Gibson) --> insects ("Gibsonian animals")
M.V. Srinivasan (1998) Insects as Gibsonian Animals. Ecol. Psychol.
10, 251-270 (1998).
exceptions: S. Nolfi & D. Marocco: active touch in object categorization
(2002)
"The [..] principle of direct perception states that perception and
action are inextricably linked. This means that perception is used to
guide action,
and this action provides additional cues to be processed by the perceptual
system, and this in turn provides more guidance for the ongoing movement (Allard,
2001).
visual paradigm: most important but very limited in isolation, biased by
interest in consciousness
a visual image is a conscious experience of internal brain/mind states,
but the latter can be more complex
this visual paradigm is essentially passive, cumulative, as seen from the
very notion of direct perception: the travelling eye
features (summary): never change or manipulate the environment
no attempt to go multi-modal (transcend vision paradigm)
brain requirements: "insect cortex" (haha)
Question
how to turn embodiment and "active perception" into a vertebrate model of
perception and action?
discover the limbic system and its biological function
How Does an Organism Behave? An Ethological Model
the center of behavior is self-activity, and the detection of similar (pro)activity
("agency") in others
perception in the ecological-ethological setting means a maintenance of
the contact with relevant stimuli
this is a complex concept when viewed from the perspective of self-activity:
full content of exploration is not just attention and/or locomotion (turning,
approach), these are special cases (suggested by vision paradigm)
no complete symmetry exists between organism and environment (also stressed
by W. Freeman)
the symmetry is broken by proactive causality (i.e. self-action with manipulative
effects)
e.g. visually guided approaching of food, touch and smell, taste and eat
(difficult to tell where perception ends; perception-as-food involves the
whole cycle up to the beginning of consumption)
(hunting illustration goes here)
(Golden Eagle in Ishikawa, K.Sudo)
approach occurs in a very complex way, involves scaring-away attempts and
other "tests",
manipulation includes hiding behind the sun, etc - in order to indentify
the target from its
similarly complex reactions
(both illustrations copyright and courtesy Prof. V. Csanyi)
literally speaking, this is a very different illustration but shows the
essential ingredients of motor-perception cycle
in other words, perception is action, action is manipulation
pure locomotion is a (not-so-special) case here: provides kinesthetic feedback,
based on the applied force etc, just as in any explicit manipulation
Action-Perception Complex
involves
interaction between sensory modalities - kinesthetic,
tactile, visual, (acoustic, olfactory, taste)
genuinely new changes induced in the environment by actions
of the organism
in a coupled dynamics, effectively nothing changes, there is only a temporal
unfolding of information present
by contrast, causal action changes things by manipulation, this is a key
component in active perception
(e.g. as a result, imitation never does the same thing, but achieves
the same result)
(cat by Michael Wolf)
How to grasp the above features?
Active Perception
multi-modal
performative
(model based, in higher animals)
can be grasped: in the concepts of episode, mechanism, and narrative; requires
episodic brain to support
Performative Action is Causal; Causality Involves Depth
performative action in perception is exemplified
by the combination of locomotion-based
approaching, reaching, touching (and perhaps tasting), with evoked responses
(1) interaction of different sensory modalities (e.g. vision with tactile
and force-based or haptic) is very different from single mode perception!
one essence of active perception is this full utilization of causality:
that one causal relation implies many ("depth")
a realized action involves several causal interactions, which need to be
integrated in perception
utilzation of the potential of the body in a complex way
sets prerequisites for the brain in terms of representation
(organisms are always engaged in complex interactions; illustration
from R. Burton)
(2) another prerequisite is self-action possibility ("agency" = the ability
to initiate action); of fundamental importance
for instance, in humans:
environmental feedback obtained through proactive motor actions proves to
be crucial in development (Thelen & Smith 1994)
prewired dispositions exist towards the detection of agency and teleological
behavior Gergely & Watson (1999)
(picture copyright Gy. Gergely and I. Kiraly)
Rational imitation in preverbal infants. Gergely, G., Bekkering, H.,&
Kiraly, I., Nature, 2002, 415
consequently, in summary, perceptive units must have an action-response
structure with reference to a "goal"
Episode
key concept of the talk, it captures the above
an episode is a temporally extended, meaningful event, composed of actions
and other causally significant events
(relates to Arthur Koestler and creativity
studies; illustration from Ch.J.Nyiri)
An episode is not an animation, but animations can represent episodes.
An animation is an episode viewed "from outside".
The Causal Structure of an Episode
how do we know, what constitutes an episode? why not any set of events within
the given time interval?
the notion of "causal significance"
(illustration after I. Tsuda)
Mechanisms as Efficient Ways of Generating Episodes
mechanisms are friendly systems with a robust causal structure; machines
and action-driven systems constitute mechanisms
mechanisms, at the same time are familiar explanation structures
(Illustration: "The Incredible Machine" by Jeff Tunnell)
--> nature of actors and actions, nature of causal systems that support
episodes, as well-defined composite events
Other Special Properties of Mechanisms
The principle of positivity
only positive cases; compare this use of if.... then with the definition
of the same connective in logic
The princple of unity
a mechanism is not an inference from A, but rather an inference scheme A
--> B,
where A and B together constitute the mechanism
The principle of opacity
mechanisms are intrasparent
Narratives as Causal Structures
I will claim that there is a direct link between the episodic organization
of perception and the concept of narrative.
Narrative: a linear ordering of facts, actions and events typical when telling
a story.
Narrative theory in cognitive science: the theory that a story-like, narrative
organization is fundamental to human cognition
in various forms and at various
levels.
The narrative
teory of consciousness (Dennett)
The narrative
theory of personal identity (McIntyre)
Narratives
as sources of psychological coherence through memory in particular
Narratives
as sources of coherence about the world.
Features of a narrative:
unique
single line, or thread of events (or singular causation)
temporal
the linear order is a temporal order
coherent
narrative coherence is a linkage between the events that occur in the narrative
it is usually assumed that a narrative is a construction, "fiction", a product
of the "literary mind" (Mark Turner)
yet a narrative is not an arbitrary recollection ("just a story") but a mechanism,
a reliable causal structure evoked by actions.
(illustration from "Introduction
to Narratology" at Purdue U.)
"The old writers strived to unfold the simple and clear thread of rationalism
from a strange, chaotic material of life; in their optic,
action is born by a cause which we can grasp by reason, and the act gives
rise to a new act. Adventure is nothing but a transparent
causal chain of such deeds."
Milan Kundera: The Art of the Novel
<Recalling Wittgenstein's vocabulary, Ricoeur suggests that, "if narrating
is a unique 'language-game,' and if a language game 'is part of an activity
or a form of life,' then we must ask to which form of life narrative discourse
as a whole is bound" (Ricoeur 1981: 274). Ricoeur says that any narrative
is endowed with an episodic dimension, the dimension of time, which is expressed
in the succession of events; and a non-chronological dimension, which constructs
"meaningful totalities out of scattered events." An essential aspect, "the
art of narrating, as well as the corresponding art of following a story,
... require that we are able to extract a configuration from a succession"
(ibid:278). Narrative may be considered as the outcome of an act of interpretation
which gives meaning to a sequence of actions. >
http://research.umbc.edu/eol/magrini/mag-narr.html
in a significant sense, both are wrong.
episodes are "self-simplifiying" because of their mechanism structure, narratives
capture episodes in a faithful way
Episodes, Mechanisms and Narratives for Active Perception (Summary)
I suggest that we tell stories because we perceive stories
we perceive stories because perception itself equals with the active, performative
making of a story.
Back to Perception:
(1) perception occurs in an integrated set of events of actions and effects
connected in a reliable way
ie perception is possible because the action-environment
complex supports it!
Gibson' s "affordances" in a more dynamic sense: opportunity
to perceive
(2) events occur in specific roles (such as mover and moved etc).
(3) percepts must be complex reconstructions based on the double causal structure
of an episode: depth and narrative
reverse: to have percepts requires
full episodes (participation, immersion - the meaning of "being there")
a model of perception must be a system that supports these features
Consequences to be Discussed
(1) Restricted Perception - How can it Work at all?
(2) Requirements for Brain Theory
(1) Restricted Perception
for instance, the case of viewing (e.g. of pictures)
an important problem, "passive" viewing is usually considered a typical
form of perception (e.g. viewing this lecture file)
occurs when an action-perception cycle is blocked (as in "dont touch", or
in viewing a postcard)
the model of active perception suggests that perception in such a case works
by analogy:
a secondary use of memories from active perception (evoked e.g. by spreading
activation)
searches for narrative structure, as if viewing person is a part (i.e. active)
successful because in most cases there IS a narrative structure (ie. a reduced
causal structure) in a picture.
(ie. active perception gives double support for narrative picture theory:
from ethology and - we will see - from the brain)
completely static pictures (landscape still life etc) interpreted as de-activated
mechanisms (e.g. "Toons" illustration)
in terms of the potential narrative episodes they support
(e.g. street is a path)
a borderline case: pure patterns.
perhaps patterns are remembered by simulated motor behavior? (of drawing
etc) --> motor theory of speech understanding
Speech Audition
can be a similar case to passive vision
requires the mental models of higher organisms
activates mental simulation by associative linkage to similar episodes
(2) Requisites for Brain Modeling
finally, how to represent this in the brain
the limbic brain (rediscovery urged by Nunez & Freeman ed 1999: Reclaiming
Cognition: The Primacy of Action, Intention and Emotion)
active perception stresses particular features of the limbic system and
implies mathematical consequences
Episodic Memory
Tulving's original idea (1972): all sematic memories are in fact episodic
connectionists & neurodynamics people do not believe (e.g. McClelland)
the importance of hippocampal system
episodic memory as understood here is not
autonoetic recollection (E. Tulving)
movie ('conjunctive code' or 'auto-associative
memory' as assumed by most researchers)
exception L. Shastri (2002): episodic memory trace (and cortico-hippocampal
connections)
"events are inherently relational", "small number of relations"
relational representations in hippocampus, embodied mental simulation (reconstruction)
in cortex
compatible with active perception model: role-entity bindings and entity
- episode bindings in episodes
a key difference: language-like symbolic traces, activating semantic memory
components
active perception requires dealing with causal depth on entities, not just
names
where do entitites come from? also must be produced by the brain
(stronger statetement: mental models in the brain, Kampis 2002)
Multiple Dynamical Systems
the brain is probably not an ordinary dynamical system
represent (reproduce) episodes as causal structures
each episode must be a separate dynamical system
Freeman
chaos + resetting mechanism
Tsuda
chaotic itinerancies
active perception segment dynamics
Support for Multiple Dyamics
the importance of molecules in brain dynamics (P. Thagard, J. Bickle)
introduce extra-dynamical variables
motor control and molecular level in tandem is example for multiple dynamics
multiple dynamics is the same idea as causality: that one thing (one dynamics)
produces another
active perception is having a causal body and a causal brain in a causal
environment
Questions
philogenetic history of active perception, with or without mental models,
cortex
(ethological examples come from birds, mammals - reptile
brain?)
hippocampus - thalamus - cortex in episodic memory (in the present sense)
role of various parts of limbic system in active perception (in the present
sense, cf Freeman 1999)
etc.
Conclusions
(1) impact for cognition
dominated by a "text in the head" view (cold and dry)
ethological cognition surrounded by (warm, wet)
cf. my book "The Active Nature" (Typotex, Budapest, 2003
- a collection of related papers)
(2) impact for brain theory
multiple modality sensory models (thalamus?)
episodic brain with causal (not-only-dynamical) internal structure
(3) impact for philosophy, philosophy of science
much speak about the "observed world", the "world of five senses" etc
seriously misleading
the world as we know it is the field of actions