Download A physics approach to classical and quantum machine learning

A physics approach to
classical and quantum machine learning
Alexey Melnikov
Institute for Theoretical Physics, University of Innsbruck
Institute for Quantum Optics and Quantum Information
Supervisor: Hans J. Briegel
Co-supervisors: Justus Piater and Gerhard Kirchmair
Jointly with Adi Makmal, Vedran Dunjko and Nicolai Friis
MIP Seminar
April 15, 2015
Alexey Melnikov
A physics approach to machine learning
Interplay between
quantum information theory and concepts from AI
Quantum physics
Artificial intelligence (AI)
Quantum computing
Quantum error correction
Quantum walks
PS
model
Intelligent agent
Machine learning
PS - projective simulation
Alexey Melnikov
A physics approach to machine learning
Outline
◦ Introduction
– artificial intelligence (AI) and its applications
– projective simulation (PS) model, a physical approach to AI
◦ Standard (classical) PS agent
– benchmarking (grid-world and mountain-car problems)
– generalization within PS Model
◦ Quantum PS agent
– implementation of a quantum agent
– superconductiong transmon qubits
Alexey Melnikov
A physics approach to machine learning
Artificial intelligence (AI) and intelligent agents
AI is the study of agents that receive percepts from the environment and
perform actions.* Any AI program is called intelligent agent.
Environment
Intelligent agent
percepts
actions
* S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall,
2009).
Alexey Melnikov
A physics approach to machine learning
AI in robotics
A robotic agent might have microphones, cameras, touch sensors and various
motors for actuators.*
Environment
Robot
microphones
cameras, touch
Applications:
• robotics
• finance
• games
• Google
motors, voice
• QEC
• ...
* S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall,
2009).
Alexey Melnikov
A physics approach to machine learning
AI in finance
A trading agent perceives market rates, news and trades in stock market.
A robotic agent
Applications:
Stock market
Trading agent
rates, news
• robotics
• finance
• games
trades
• Google
• QEC
• ...
* S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall,
2009).
Alexey Melnikov
A physics approach to machine learning
AI in games
A game agent plays with you.
A robotic agent
You
Game agent
your moves
Applications:
• robotics
• finance
• games
it’s own moves
• Google
• QEC
• ...
* S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall,
2009).
Alexey Melnikov
A physics approach to machine learning
AI on the web
Search engine interacts with a user.
Google
User
Google
query
Applications:
• robotics
• finance
• games
web page
• Google
• QEC
• ...
* S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall,
2009).
Alexey Melnikov
A physics approach to machine learning
AI in quantum error correction (QEC)
AI can be useful for quantum physics. A QEC agent gets data from syndrome
measurements and performs error correction.*
Quantum register
QEC agent
syndrome data
Applications:
• robotics
• finance
• games
apply unitaries
• Google
• QEC
• ...
* J. Combes, et al., arXiv:1405.5656 (2014).
Alexey Melnikov
A physics approach to machine learning
Projective simulation (PS) agent
• PS model is a novel physical approach to AI
• PS agent process information stochastically in a directed, weighted
network of clips (units of memory)
• No computations, simple adjustment rules
• Natural candidate for quantization, using methods of quantum walks
Clip network
...
PS agent
...
percepts
p41
percept clip
Clip 1
Clip 4
action clip
p13
Clip 3
actions
input
p12
p23
p32
Clip 2
p35
Clip 6
p56
Clip 5
output
H. J. Briegel and G. De las Cuevas, Scientic reports 2 (2012).
Alexey Melnikov
A physics approach to machine learning
Projective simulation (PS) model
Each edge connects some clip ci with a clip cj and has a time-dependent weight
h(t) (ci , cj ). The h-values represent the unnormalized strength of the edges, and
determine the hopping probabilities from clip ci to clip cj according to
h(t) (ci , cj )
.
p (t) (cj |ci ) = P (t)
k h (ci , ck )
h-values are updated according to
h(t+1) (ci , cj ) = h(t) (ci , cj ) − γ(h(t) (ci , cj ) − 1) + g (t) (ci , cj )λ,
where 0 ≤ γ ≤ 1 is a damping parameter and λ is a non-negative reward given by
the environment.
Each time an edge is visited, the corresponding g -value is set to 1, following which
it is decreased after each time step with a rate η:
g (t+1) (ci , cj ) = g (t) (ci , cj )(1 − η).
J. Mautner, A. Makmal, D. Manzano, M. Tiersch, and H. J. Briegel, New Generation Computing
33 (2015)
Alexey Melnikov
A physics approach to machine learning
Grid-world task
• The agent always starts from the (1,3)
cell
• It can choose among four actions: left,
right, up or down
• If the agent decides to go to a square labeled as “wall” or to go beyond the grid,
then no movement is performed but the
time step is counted
The grid-world task: The goal of the
game is to find the “star”.
• Reward of λ = 1 is received only after
reaching the goal
• A performance of an agent in this task is
evaluated by the number of steps it makes
before reaching the goal at each trial
R. S. Sutton, Proc. of the 7th International Conference on Machine Learning (1990)
Alexey Melnikov
A physics approach to machine learning
PS network construction
x =1
y =1
x =1
y =2
...
x =6
y =9
⇑
⇒
⇓
hij
gij
⇐
Alexey Melnikov
A physics approach to machine learning
PS network construction
x =1
y =1
x =1
y =2
...
x =6
y =9
⇑
⇒
⇓
hij
gij
⇐
Alexey Melnikov
A physics approach to machine learning
PS network construction
x =1
y =1
x =1
y =2
...
x =6
y =9
⇑
⇒
⇓
hij
gij
⇐
Alexey Melnikov
A physics approach to machine learning
PS in the grid-world task. Learning curves
Η=0.03
average number of steps
24
Η=0.12
22
Η=0.15
20
18
16
14
0
50
100
150
200
trials
The learning curves of the PS agent in the grid-world task, with different η values.
A trade-off is observed between the best performance and the number of trials
required to reach it.
Model
# of steps to goal after 100 trials
Parameters
PS†
PI∗
15.4
14
λ = 1, η = 0.12, γ = 0
β = 0.1, γ = 0.9, α = 1000
Performance of the PS model in comparison with the PI model
†
A. A. Melnikov, A. Makmal, and H. J. Briegel, Artificial Intelligence Research 3 (2014)
* R. S. Sutton, Proc. of the 7th International Conference on Machine Learning (1990)
Alexey Melnikov
A physics approach to machine learning
Mountain car problem
• The agent always starts with a
random position and velocity:
x ∈ [−1.2, 0.5], v ∈ [−0.7, 0.7]
• It can choose among 3 actions:
forward thrust (to the right), no
thrust, and reverse thrust (to the
left)
The goal is to find the “star” at x = 0.5
• The next state is defined by the equations
vnew
xnew
= vold + 0.001 ∗ Action − 0.0025 cos(3xold )
= xold + vold
• Reward of λ = 1 is received only after reaching the goal
• A performance of an agent in this task is evaluated by the number of steps it
makes before reaching the goal at each trial
S. P. Singh and R. S. Sutton, Machine learning 22, 123 (1996).
Alexey Melnikov
A physics approach to machine learning
PS network construction
[x0 , x1 ],
[v0 , v1 ]
(x1 , x2 ],
[v0 , v1 ]
(x19 , x20 ],
(v19 , v20 ]
...
hij
gij
=
−
Alexey Melnikov
+
A physics approach to machine learning
PS in the mountain car task. Learning curves
450
500
pHtL Hc j Èci L by Eq. 2 HsoftmaxL, Η=0.02
400
æ
300
200
400
à
10
15
20
æ
æ
à
æ
à
æ
300
à
æ
æ
æ
à
æ
æ
æ
æ
à
æ
æ
200
æ
æ
æ
æ
à
à
à
150
à
æ
à
250
à
à
à
à
à
5
æ
æ
pHtL Hc j Èci L by Eq. 2 HsoftmaxL
à
æ
350
100
0.00
100
0
pHtL Hc j Èci L by Eq. 1
æ
average number of steps
average number of steps
pHtL Hc j Èci L by Eq. 1, Η=0.02
à
à
à
0.02
à
à
à
à
à
à
0.04
0.06
0.08
0.10
Η parameter
trials
(a) PS learning curves are shown for optimal values of η = 0.02 (for 20 trials).
(b) The dependence of the PS performance
on the η parameter is shown after 20 trials.
Model
# of steps to goal after 100 trials
Parameters
PS†
SARSA∗
223/trial
450/trial
λ = 1, η = 0.02, γ = 0
5 grids, each of 9 by 9 input space
Performance of the PS model in comparison with the SARSA algorithm
†
A. A. Melnikov, A. Makmal, and H. J. Briegel, Artificial Intelligence Research 3 (2014)
* S. P. Singh and R. S. Sutton, Machine learning 22, 123 (1996)
Alexey Melnikov
A physics approach to machine learning
Generalization. Motivation
There are many tasks in which percepts are composed of several elements. Even
if two percept clips are different they may contain some common set of elements.
This common set of elements should be taken into account in order to share the
experience between different inputs.
Useful generalization *:
⇐
⇒
⇒
• An ability for categorization (recognizing
that all red signals have a common property, which we can refer to as redness)
⇐
• An ability to classify
+
−
• Relevant
learned
While driving the agent sees a traffic
light with an arrow sign and should
choose among two actions: continue
driving (+) or stop a car (−).
generalizations
should
be
• Correct actions should be associated with
relevant generalized properties
• The generalization mechanism should be
flexible
* A. A. Melnikov, A. Makmal, and H. J. Briegel, arXiv:1504.02247 (2015).
Alexey Melnikov
A physics approach to machine learning
Mechanism of generalization
⇐
⇒
⇒
⇒
⇐
⇐
⇐
⇒
⇒
⇒
⇐
#
−
⇒
⇒
⇒
⇐
−
+
(a) t ≤ 1000
(b) 1000 < t ≤ 2000
⇐
⇐
#
+
(a) (1 ≤ t ≤ 1000), the agent is
rewarded for stopping at red
light and for driving at green
light
#
+
⇐
⇐
⇒
⇒
⇒
⇐
⇐
#
−
(c) 2000 < t ≤ 3000
+
−
(d) 3000 < t ≤ 4000
Alexey Melnikov
(b) (1000 < t ≤ 2000), the agent
is rewarded for doing the
opposite
(c) (2000 < t ≤ 3000), the agent
should only follow the arrows
(d) (3000 < t ≤ 4000), the
environment rewards the
agent whenever it chooses to
drive
A physics approach to machine learning
Mechanism of generalization
(a) (1 ≤ t ≤ 1000), the agent is
rewarded for stopping at red
light and for driving at green
light
1.0
efficiency Et
0.8
(a)
(b)
(c)
(d)
0.6
(b) (1000 < t ≤ 2000), the agent
is rewarded for doing the
opposite
0.4
0.2
0.0
0
1000
2000
3000
time step
The performance of the
PS agent with generalization
4000
(c) (2000 < t ≤ 3000), the agent
should only follow the arrows
(d) (3000 < t ≤ 4000), the
environment rewards the
agent whenever it chooses to
drive
A. A. Melnikov, A. Makmal, and H. J. Briegel, arXiv:1504.02247 (2015).
Alexey Melnikov
A physics approach to machine learning
Quantum PS agent
• PS model is a novel physical approach to AI
• PS agent process information stochastically in a directed, weighted
network of clips (units of memory)
• No computations, simple adjustment rules
• Natural candidate for quantization, using methods of quantum walks
Quantum clip network
classical
percepts
...
quantum
PS agent
...
p41
percept clip
Clip 1
classical
actions
Clip 4
action clip
p13
Clip 3
classical
input
p12
p23
p32
Clip 2
p35
Clip 6
p56
Clip 5
classical
output
*G. D. Paparo, V. Dunjko, A. Makmal, M. A. Martin-Delgado, and H. J. Briegel, Phys. Rev. X
4, 031002 (2014).
Alexey Melnikov
A physics approach to machine learning
Quantum PS agent
Classical random walk on a network with N clips is characterised by a transition
T
matrix P, where each clip is a vector ci = [0, . . . , 1, 0, . . . , 0] with unity on the
i-th position
N
X
P ci =
pij cj ,
j=1
In the quantum case each clip is a state |ci i. However a single unitary cannot
encode the P matrix.
We use the set of N unitaries for a
quantum walk
Ui |0i =
N
X
√
pij |cj i .
j=1
Two-qubit probability unitaries for PS
network with 4 clips.
V. Dunjko, N. Friis, and H. J. Briegel, New J. Phys. 17 (2015)
Alexey Melnikov
A physics approach to machine learning
Nested coherent controlization
Three-qubit probability unitaries for PS
network with 8 clips.
Alexey Melnikov
No-go theorem.
Additional degrees of freedom are
needed.
A physics approach to machine learning
Transmon qubits
An aluminum transmon qubit with the dipole antenna is mounted at the center of
the cavity.
For one qubit, the system is described by the Hamiltonian
2
2
H/~ = ωr a† a + ωq b † b − χqr /2 a† ab † b − χrr /2 a† a − χqq /2 b † b ,
where a and b are the dressed mode operators of the resonator and the qubit,
respectively, ωr and ωq are their frequencies, χqr is the coupling between them and
χrr , χqq are the anharmonicities.
H. Paik, et al., Phys. Rev. Lett. 107 (2011).
B. Vlastakis, et al., Science 342 (2013).
Alexey Melnikov
A physics approach to machine learning
Coherent controlization using transmon qubits
We use the cavity as a additional degree of freedom to implement the coherent
controlization.
a
b
c
The resonance frequency of the cavity depends on the state of the qubits.
For two superconducting qubits, we may hence label these ω00 , ω01 , ω10 , ω11 , corresponding to the two-qubit states |00i, |01i, |10i, and |11i, respectively.
Alexey Melnikov
A physics approach to machine learning
Coherent controlization using transmon qubits
6
4
2
0
2
4
6
6
4
2
0
2
4
6
1
0
1
6
4
2 0
2
4
6
6
4
2 0
2
4
6
Alexey Melnikov
6
4
2 0
2
4
6
6
4
2 0
2
4
6
6
4
A physics approach to machine learning
2 0
2
4
6
Conclusion
◦ Standard (classical) PS agent
– is a competitive AI model (grid-world and mountain-car problems)
– generalization mechanism improves the model
– has potentially many applications
◦ Quantum PS agent
– quantization, using known methods of quantum walks
– implementation using superconductiong qubits
Thank you for your attention!
Alexey Melnikov
A physics approach to machine learning
Conclusion
◦ Standard (classical) PS agent
– is a competitive AI model (grid-world and mountain-car problems)
– generalization mechanism improves the model
– has potentially many applications
◦ Quantum PS agent
– quantization, using known methods of quantum walks
– implementation using superconductiong qubits
Thank you for your attention!
Alexey Melnikov
A physics approach to machine learning