Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Pattern recognition wikipedia , lookup
Person of Interest (TV series) wikipedia , lookup
Existential risk from artificial general intelligence wikipedia , lookup
Reinforcement learning wikipedia , lookup
Concept learning wikipedia , lookup
Philosophy of artificial intelligence wikipedia , lookup
Embodied cognitive science wikipedia , lookup
History of artificial intelligence wikipedia , lookup
A physics approach to classical and quantum machine learning Alexey Melnikov Institute for Theoretical Physics, University of Innsbruck Institute for Quantum Optics and Quantum Information Supervisor: Hans J. Briegel Co-supervisors: Justus Piater and Gerhard Kirchmair Jointly with Adi Makmal, Vedran Dunjko and Nicolai Friis MIP Seminar April 15, 2015 Alexey Melnikov A physics approach to machine learning Interplay between quantum information theory and concepts from AI Quantum physics Artificial intelligence (AI) Quantum computing Quantum error correction Quantum walks PS model Intelligent agent Machine learning PS - projective simulation Alexey Melnikov A physics approach to machine learning Outline ◦ Introduction – artificial intelligence (AI) and its applications – projective simulation (PS) model, a physical approach to AI ◦ Standard (classical) PS agent – benchmarking (grid-world and mountain-car problems) – generalization within PS Model ◦ Quantum PS agent – implementation of a quantum agent – superconductiong transmon qubits Alexey Melnikov A physics approach to machine learning Artificial intelligence (AI) and intelligent agents AI is the study of agents that receive percepts from the environment and perform actions.* Any AI program is called intelligent agent. Environment Intelligent agent percepts actions * S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall, 2009). Alexey Melnikov A physics approach to machine learning AI in robotics A robotic agent might have microphones, cameras, touch sensors and various motors for actuators.* Environment Robot microphones cameras, touch Applications: • robotics • finance • games • Google motors, voice • QEC • ... * S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall, 2009). Alexey Melnikov A physics approach to machine learning AI in finance A trading agent perceives market rates, news and trades in stock market. A robotic agent Applications: Stock market Trading agent rates, news • robotics • finance • games trades • Google • QEC • ... * S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall, 2009). Alexey Melnikov A physics approach to machine learning AI in games A game agent plays with you. A robotic agent You Game agent your moves Applications: • robotics • finance • games it’s own moves • Google • QEC • ... * S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall, 2009). Alexey Melnikov A physics approach to machine learning AI on the web Search engine interacts with a user. Google User Google query Applications: • robotics • finance • games web page • Google • QEC • ... * S. Russell and P. Norvig. Artificial intelligence: A Modern Approach, 3rd edition (Prentice Hall, 2009). Alexey Melnikov A physics approach to machine learning AI in quantum error correction (QEC) AI can be useful for quantum physics. A QEC agent gets data from syndrome measurements and performs error correction.* Quantum register QEC agent syndrome data Applications: • robotics • finance • games apply unitaries • Google • QEC • ... * J. Combes, et al., arXiv:1405.5656 (2014). Alexey Melnikov A physics approach to machine learning Projective simulation (PS) agent • PS model is a novel physical approach to AI • PS agent process information stochastically in a directed, weighted network of clips (units of memory) • No computations, simple adjustment rules • Natural candidate for quantization, using methods of quantum walks Clip network ... PS agent ... percepts p41 percept clip Clip 1 Clip 4 action clip p13 Clip 3 actions input p12 p23 p32 Clip 2 p35 Clip 6 p56 Clip 5 output H. J. Briegel and G. De las Cuevas, Scientic reports 2 (2012). Alexey Melnikov A physics approach to machine learning Projective simulation (PS) model Each edge connects some clip ci with a clip cj and has a time-dependent weight h(t) (ci , cj ). The h-values represent the unnormalized strength of the edges, and determine the hopping probabilities from clip ci to clip cj according to h(t) (ci , cj ) . p (t) (cj |ci ) = P (t) k h (ci , ck ) h-values are updated according to h(t+1) (ci , cj ) = h(t) (ci , cj ) − γ(h(t) (ci , cj ) − 1) + g (t) (ci , cj )λ, where 0 ≤ γ ≤ 1 is a damping parameter and λ is a non-negative reward given by the environment. Each time an edge is visited, the corresponding g -value is set to 1, following which it is decreased after each time step with a rate η: g (t+1) (ci , cj ) = g (t) (ci , cj )(1 − η). J. Mautner, A. Makmal, D. Manzano, M. Tiersch, and H. J. Briegel, New Generation Computing 33 (2015) Alexey Melnikov A physics approach to machine learning Grid-world task • The agent always starts from the (1,3) cell • It can choose among four actions: left, right, up or down • If the agent decides to go to a square labeled as “wall” or to go beyond the grid, then no movement is performed but the time step is counted The grid-world task: The goal of the game is to find the “star”. • Reward of λ = 1 is received only after reaching the goal • A performance of an agent in this task is evaluated by the number of steps it makes before reaching the goal at each trial R. S. Sutton, Proc. of the 7th International Conference on Machine Learning (1990) Alexey Melnikov A physics approach to machine learning PS network construction x =1 y =1 x =1 y =2 ... x =6 y =9 ⇑ ⇒ ⇓ hij gij ⇐ Alexey Melnikov A physics approach to machine learning PS network construction x =1 y =1 x =1 y =2 ... x =6 y =9 ⇑ ⇒ ⇓ hij gij ⇐ Alexey Melnikov A physics approach to machine learning PS network construction x =1 y =1 x =1 y =2 ... x =6 y =9 ⇑ ⇒ ⇓ hij gij ⇐ Alexey Melnikov A physics approach to machine learning PS in the grid-world task. Learning curves Η=0.03 average number of steps 24 Η=0.12 22 Η=0.15 20 18 16 14 0 50 100 150 200 trials The learning curves of the PS agent in the grid-world task, with different η values. A trade-off is observed between the best performance and the number of trials required to reach it. Model # of steps to goal after 100 trials Parameters PS† PI∗ 15.4 14 λ = 1, η = 0.12, γ = 0 β = 0.1, γ = 0.9, α = 1000 Performance of the PS model in comparison with the PI model † A. A. Melnikov, A. Makmal, and H. J. Briegel, Artificial Intelligence Research 3 (2014) * R. S. Sutton, Proc. of the 7th International Conference on Machine Learning (1990) Alexey Melnikov A physics approach to machine learning Mountain car problem • The agent always starts with a random position and velocity: x ∈ [−1.2, 0.5], v ∈ [−0.7, 0.7] • It can choose among 3 actions: forward thrust (to the right), no thrust, and reverse thrust (to the left) The goal is to find the “star” at x = 0.5 • The next state is defined by the equations vnew xnew = vold + 0.001 ∗ Action − 0.0025 cos(3xold ) = xold + vold • Reward of λ = 1 is received only after reaching the goal • A performance of an agent in this task is evaluated by the number of steps it makes before reaching the goal at each trial S. P. Singh and R. S. Sutton, Machine learning 22, 123 (1996). Alexey Melnikov A physics approach to machine learning PS network construction [x0 , x1 ], [v0 , v1 ] (x1 , x2 ], [v0 , v1 ] (x19 , x20 ], (v19 , v20 ] ... hij gij = − Alexey Melnikov + A physics approach to machine learning PS in the mountain car task. Learning curves 450 500 pHtL Hc j Èci L by Eq. 2 HsoftmaxL, Η=0.02 400 æ 300 200 400 à 10 15 20 æ æ à æ à æ 300 à æ æ æ à æ æ æ æ à æ æ 200 æ æ æ æ à à à 150 à æ à 250 à à à à à 5 æ æ pHtL Hc j Èci L by Eq. 2 HsoftmaxL à æ 350 100 0.00 100 0 pHtL Hc j Èci L by Eq. 1 æ average number of steps average number of steps pHtL Hc j Èci L by Eq. 1, Η=0.02 à à à 0.02 à à à à à à 0.04 0.06 0.08 0.10 Η parameter trials (a) PS learning curves are shown for optimal values of η = 0.02 (for 20 trials). (b) The dependence of the PS performance on the η parameter is shown after 20 trials. Model # of steps to goal after 100 trials Parameters PS† SARSA∗ 223/trial 450/trial λ = 1, η = 0.02, γ = 0 5 grids, each of 9 by 9 input space Performance of the PS model in comparison with the SARSA algorithm † A. A. Melnikov, A. Makmal, and H. J. Briegel, Artificial Intelligence Research 3 (2014) * S. P. Singh and R. S. Sutton, Machine learning 22, 123 (1996) Alexey Melnikov A physics approach to machine learning Generalization. Motivation There are many tasks in which percepts are composed of several elements. Even if two percept clips are different they may contain some common set of elements. This common set of elements should be taken into account in order to share the experience between different inputs. Useful generalization *: ⇐ ⇒ ⇒ • An ability for categorization (recognizing that all red signals have a common property, which we can refer to as redness) ⇐ • An ability to classify + − • Relevant learned While driving the agent sees a traffic light with an arrow sign and should choose among two actions: continue driving (+) or stop a car (−). generalizations should be • Correct actions should be associated with relevant generalized properties • The generalization mechanism should be flexible * A. A. Melnikov, A. Makmal, and H. J. Briegel, arXiv:1504.02247 (2015). Alexey Melnikov A physics approach to machine learning Mechanism of generalization ⇐ ⇒ ⇒ ⇒ ⇐ ⇐ ⇐ ⇒ ⇒ ⇒ ⇐ # − ⇒ ⇒ ⇒ ⇐ − + (a) t ≤ 1000 (b) 1000 < t ≤ 2000 ⇐ ⇐ # + (a) (1 ≤ t ≤ 1000), the agent is rewarded for stopping at red light and for driving at green light # + ⇐ ⇐ ⇒ ⇒ ⇒ ⇐ ⇐ # − (c) 2000 < t ≤ 3000 + − (d) 3000 < t ≤ 4000 Alexey Melnikov (b) (1000 < t ≤ 2000), the agent is rewarded for doing the opposite (c) (2000 < t ≤ 3000), the agent should only follow the arrows (d) (3000 < t ≤ 4000), the environment rewards the agent whenever it chooses to drive A physics approach to machine learning Mechanism of generalization (a) (1 ≤ t ≤ 1000), the agent is rewarded for stopping at red light and for driving at green light 1.0 efficiency Et 0.8 (a) (b) (c) (d) 0.6 (b) (1000 < t ≤ 2000), the agent is rewarded for doing the opposite 0.4 0.2 0.0 0 1000 2000 3000 time step The performance of the PS agent with generalization 4000 (c) (2000 < t ≤ 3000), the agent should only follow the arrows (d) (3000 < t ≤ 4000), the environment rewards the agent whenever it chooses to drive A. A. Melnikov, A. Makmal, and H. J. Briegel, arXiv:1504.02247 (2015). Alexey Melnikov A physics approach to machine learning Quantum PS agent • PS model is a novel physical approach to AI • PS agent process information stochastically in a directed, weighted network of clips (units of memory) • No computations, simple adjustment rules • Natural candidate for quantization, using methods of quantum walks Quantum clip network classical percepts ... quantum PS agent ... p41 percept clip Clip 1 classical actions Clip 4 action clip p13 Clip 3 classical input p12 p23 p32 Clip 2 p35 Clip 6 p56 Clip 5 classical output *G. D. Paparo, V. Dunjko, A. Makmal, M. A. Martin-Delgado, and H. J. Briegel, Phys. Rev. X 4, 031002 (2014). Alexey Melnikov A physics approach to machine learning Quantum PS agent Classical random walk on a network with N clips is characterised by a transition T matrix P, where each clip is a vector ci = [0, . . . , 1, 0, . . . , 0] with unity on the i-th position N X P ci = pij cj , j=1 In the quantum case each clip is a state |ci i. However a single unitary cannot encode the P matrix. We use the set of N unitaries for a quantum walk Ui |0i = N X √ pij |cj i . j=1 Two-qubit probability unitaries for PS network with 4 clips. V. Dunjko, N. Friis, and H. J. Briegel, New J. Phys. 17 (2015) Alexey Melnikov A physics approach to machine learning Nested coherent controlization Three-qubit probability unitaries for PS network with 8 clips. Alexey Melnikov No-go theorem. Additional degrees of freedom are needed. A physics approach to machine learning Transmon qubits An aluminum transmon qubit with the dipole antenna is mounted at the center of the cavity. For one qubit, the system is described by the Hamiltonian 2 2 H/~ = ωr a† a + ωq b † b − χqr /2 a† ab † b − χrr /2 a† a − χqq /2 b † b , where a and b are the dressed mode operators of the resonator and the qubit, respectively, ωr and ωq are their frequencies, χqr is the coupling between them and χrr , χqq are the anharmonicities. H. Paik, et al., Phys. Rev. Lett. 107 (2011). B. Vlastakis, et al., Science 342 (2013). Alexey Melnikov A physics approach to machine learning Coherent controlization using transmon qubits We use the cavity as a additional degree of freedom to implement the coherent controlization. a b c The resonance frequency of the cavity depends on the state of the qubits. For two superconducting qubits, we may hence label these ω00 , ω01 , ω10 , ω11 , corresponding to the two-qubit states |00i, |01i, |10i, and |11i, respectively. Alexey Melnikov A physics approach to machine learning Coherent controlization using transmon qubits 6 4 2 0 2 4 6 6 4 2 0 2 4 6 1 0 1 6 4 2 0 2 4 6 6 4 2 0 2 4 6 Alexey Melnikov 6 4 2 0 2 4 6 6 4 2 0 2 4 6 6 4 A physics approach to machine learning 2 0 2 4 6 Conclusion ◦ Standard (classical) PS agent – is a competitive AI model (grid-world and mountain-car problems) – generalization mechanism improves the model – has potentially many applications ◦ Quantum PS agent – quantization, using known methods of quantum walks – implementation using superconductiong qubits Thank you for your attention! Alexey Melnikov A physics approach to machine learning Conclusion ◦ Standard (classical) PS agent – is a competitive AI model (grid-world and mountain-car problems) – generalization mechanism improves the model – has potentially many applications ◦ Quantum PS agent – quantization, using known methods of quantum walks – implementation using superconductiong qubits Thank you for your attention! Alexey Melnikov A physics approach to machine learning