Download YBurak-Sloan-Swartz-Annual-2009

Image Stabilization by
Bayesian Dynamics
Yoram Burak
Sloan-Swartz annual meeting, July 2009
What does neural activity represent?
In Bayesian models: probabilities
Accumulated evidence in area LIP
Shadlen and Newsome (2001)
Direction of motion: single, static variable
What does neural activity represent?
In Bayesian models: probabilities
Accumulated evidence in area LIP
Shadlen and Newsome (2001)
Direction of motion: single, static variable
What about multi-dimensional, dynamic quantities?
Foveal vision and fixational drift
Foveal vision and fixational drift
Fixational drift is large in the fovea:
- between micro-saccades ~20 receptive fields
- between spikes (100 Hz) ~2-4 receptive fields !
By Xaq from: X. Pitkow
Image
Pitkow
cone separation:
0.5 arcmin
Foveal vision and fixational drift
Fixational drift is large in the fovea:
- between micro-saccades ~20 receptive fields
- between spikes (100 Hz) ~2-4 receptive fields !
By Xaq from: X. Pitkow
Image
Pitkow
cone separation:
0.5 arcmin
Downstream areas require knowledge
of trajectory to interpret spikes
Joint decoding of image and position
Bayesian:
Discrimination task:
vs.
N x 2 probabilities
X. Pitkow et al, Plos Biology (2007)
# positions
Joint decoding of image and position
Bayesian:
Discrimination task:
vs.
N x 2 probabilities
X. Pitkow et al, Plos Biology (2007)
# positions
Unconstrained image
30 x 30
binary
pixels
N x 2900 probabilities
Joint decoding of image and position
Bayesian:
Discrimination task:
vs.
N x 2 probabilities
X. Pitkow et al, Plos Biology (2007)
# positions
Unconstrained image
30 x 30
binary
pixels
N x 2900 probabilities
Can the brain apply a Bayesian approach to
this problem?
Can the brain apply a Bayesian approach to
this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
Can the brain apply a Bayesian approach to
this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
Decoding strategy
Factorized representation:
Discards information about correlations
Decoding strategy
Factorized representation:
Discards information about correlations
Update dynamics:
evidence,
diffusion
Exact if trajectory is known.
minimize
DKL
Decoding strategy
Factorized representation:
Discards information about correlations
Update dynamics:
evidence,
diffusion
minimize
DKL
Exact if trajectory is known.
Retinal encoding model:
evidence - Poisson spiking (rate λ1 for on pixels, λ0 for off)
diffusion - Random walk (diffusion coefficient D)
Decoding strategy
Factorized representation:
Discards information about correlations
Neural Implementation -
Two populations: where
, what
For 30 x 30 pixels: N × 2900 → N + 900 quantities.
Update rules
Update of what neurons:
Ganglion cells
multiplicative gating
nonlinearity
What
Where
Update rules
Update of what neurons:
Ganglion cells
multiplicative gating
What
nonlinearity
Where
Update of where neurons:
Ganglion cells
multiplicative gating
Where
What
+ diffusion
Demo
m x m binary pixels
image
retina
2d diffusion (D)
Poisson spikes:
100 Hz (on), 10 Hz (off)
Decoder
Demo
Can the brain apply a Bayesian approach to
this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
Performance
Performance degrades with larger D
accuracy
Convergence time [s]
(and smaller λ)
D
D
Performance
Faster and more accurate for larger images
accuracy
Convergence time [s]
m = 5, 10, 30, 50, 100
D
D
Demo
Performance
Faster and more accurate for larger images
accuracy
Convergence time [s]
m = 5, 10, 30, 50, 100
D
D
Performance
Faster and more accurate for larger images
accuracy
Convergence time [s]
m = 5, 10, 30, 50, 100
D
D
Performance
Faster and more accurate for larger images
accuracy
Convergence time [s]
m = 5, 10, 30, 50, 100
D
D
Performance
accuracy
Convergence time [s]
scales with linear image size m
m x m pixels
D/m
D/m
Performance
m x m pixels
D*
D/m
D/m
Analytical scaling:
Convergence time [s]
accuracy
scales with linear image size m
Performance
Performance improves with image size.
Success for images 10 x 10 or larger
Prediction for psychophysics:
Degradation in high acuity tasks when visual scene contains little
background detail.
Temporal response of Ganglion cells
f(t)
Common view: fixational motion important to activate cells, due to biphasic
response
50 ms
t
Temporal response makes decoding much more difficult.
Non-Markovian:
Need history
Temporal response of Ganglion cells
What
Where
“filtered
trajectory”
accuracy
Ganglion
Convergence time [s]
Approach: Choose decoder that is Bayes optimal if the
trajectory is known.
D
D
history dependent decoder /
naive decoder
Temporal response of Ganglion cells
Is fixational motion beneficial?
Convergence time [s]
Known trajectory , perfect inhibitory balance
D
Optimal D - order of magnitude smaller than biological value
Can the brain apply a Bayesian approach to
this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
Network architecture
Each ganglion cell innervates multiple what & where cells
(spread: ~10 arcmin)
Reciprocal, multiplicative gating
Ganglion
What
Where
Activity:
What neurons
Slow dynamics, evidence accumulation
Where neurons Fewer. Highly dynamic activity
Tonic, sparse in retinal stabilization conditions.
Activity:
What neurons
Slow dynamics, evidence accumulation
Where neurons Fewer. Highly dynamic activity
Tonic, sparse in retinal stabilization conditions.
Where in the brain?
Monocular
If so, suggests LGN or V1
LGN?
Modulatory inputs to relay cells (gating?)
V1?
Lateral connectivity in where network,
Increase in number of neurons.
Summary
Strategy for stabilization of foveal vision
Factorized Bayesian approach to multi-dimensional inference
Summary
Strategy for stabilization of foveal vision
Factorized Bayesian approach to multi-dimensional inference
Explicit representation of stabilized image
“What” and “where” populations
Summary
Strategy for stabilization of foveal vision
Factorized Bayesian approach to multi-dimensional inference
Explicit representation of stabilized image
“What” and “where” populations
Good performance at 1 arcmin resolution
Problem is easier for large images, for coarser reconstruction
Summary
Strategy for stabilization of foveal vision
Factorized Bayesian approach to multi-dimensional inference
Explicit representation of stabilized image
“What” and “where” populations
Good performance at 1 arcmin resolution
Problem is easier for large images, for coarser reconstruction
Network architecture:
Many-to-one inputs from retina, multiplicative gating (what/where)
Acknowledgments
Uri Rokni
Haim Sompolinsky
Markus Meister
Special thanks - the Swartz foundation