* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Neural Network Algorithms-Quantum-Glia
Survey
Document related concepts
Quantum electrodynamics wikipedia , lookup
Probability amplitude wikipedia , lookup
Symmetry in quantum mechanics wikipedia , lookup
Quantum fiction wikipedia , lookup
Many-worlds interpretation wikipedia , lookup
EPR paradox wikipedia , lookup
History of quantum field theory wikipedia , lookup
Interpretations of quantum mechanics wikipedia , lookup
Quantum teleportation wikipedia , lookup
Canonical quantization wikipedia , lookup
Quantum computing wikipedia , lookup
Quantum key distribution wikipedia , lookup
Quantum group wikipedia , lookup
Quantum state wikipedia , lookup
Hidden variable theory wikipedia , lookup
Transcript
Neural Network Algorithms Review, Quantum and Glial Directions Martin Dimkovski CSE 6111 Presentation York University March 31st, 2011 Presentation Goals Hope to increase awareness of NNs’ potential (<10 min) Very general toolkit, applicable to many problems Overview: Algorithm models Computational power Complexity, limitations Interesting research directions (<5 min) (<5 min) March 32st, 2011 Quantum Glia Neural Network Algorithms 2 Presentation Motivation Can’t explain Fight common over-simplification 0.5 ubiquitous (brains, computers, society, space-time fabric), accessible & good inspiration On a serious note…: Backpropagation-only/mostly view Practical applications and solid theoretical foundations exist for presented alternatives March 32st, 2011 Neural Network Algorithms 3 Network Graphs Unidirectional: simple, feed-forward + Bidirectional: recurrent, interactive 0.5 Backpropagation is here Network dynamics comes up ++ lateral: resonance, competition, pattern completion March 32st, 2011 Neural Network Algorithms 4 Learning Approaches Supervised vs Unsupervised Error-driven 1.5 Straightforward Mature Adaptive basis functions For well defined tasks input>output, functions Hebbian-Style (ex: backprop) (more on next slide) More sophisticated, more bio-inspired, self-organizing But not as mature (still weak like error-driven in 1960s) Order and Chaos Combinations (superior) March 32st, 2011 Neural Network Algorithms 5 On Hebbian-Style Network dynamics, more complex graphs Build internal models of environ. - Identify principal features Constraint satisfaction Find energy function minima Attractors = memorized patterns 1.5 Fire together – wire together Compete Resonate Deal with corrupt and partial memory March 32st, 2011 Neural Network Algorithms 6 More Modeling Aspects Complex (superior) vs real valued Temporal dynamics Signal coding: Static vs dynamic weights (in between training) Stay the same for any input/output condition Superior: Adjust to input/output condition Excitation - inhibition modeling 1.5 Discreet, analog, pulse averaging, or Superior: detailed pulse/spike pattern modeling Superior: not on the same weight March 32st, 2011 Neural Network Algorithms 7 Use of Probability Bayesian: Solves a BIG problem Over-fitting Solves it because it samples from whole posterior and does not depend on a single set of weights To get a feeling, compare: MAP = argmax P( | data) E[] = P( | data) d The benefit of noise 1.5 When noise and peculiarities become more attended-to than the general features of interest Problem especially with error-driven, like backprop. To avoid local minima/maxima (ketchup) March 32st, 2011 Neural Network Algorithms 8 Research Directions 1.0 NNs have come a long way Yet, still far below known upper bounds For precision, performance, and usability What better place to turn for help, than back to our original inspiration? In green are my personal speculations March 32st, 2011 Neural Network Algorithms 9 Cues to Quantum Classically unexplained brain features Simultaneous synchronized stimulations in distant regions for same stimuli 1.5 Perception unification – global attractor states Speculation brain has ingredients for macroscopic quantum state highly structured in phase and amplitude High metabolic energy; extreme dielectric prop. Microtubules, superconducting waves, gap junctions (anaesthesia) Interest in just plain quantum computing power March 32st, 2011 Neural Network Algorithms 10 A (Qu)bit of Quantum Basics If left alone – a linear ‘combination’ of basis states (in coherence) Each |i is a single reality for us classical beings Into one of the basis states, as per probabilities Entanglement 1.5 |ci|2 giving the probability If ‘touched’ by anything – decoheres (ex: |0 or |1) But in quantum world, they all exist at once | = ci|i Instantaneous sync link between remote qubits March 32st, 2011 Neural Network Algorithms 11 How can Quantum Translate for Artificial NNs? 1. Run existing NN algorithms on quantum computers 2. Could we simulate the quantum ‘spooky’ effects in new NN algorithms? 0.5 …getting there, but will take a long time Extra slide in appendix March 32st, 2011 Using our classical computers Neural Network Algorithms 12 Simulating Quantum Effects Maybe the brain uses certain quantum features for evolutionary reasons. Could we program/simulate?: Synchronization and unification of distant physically unconnected neurons? Coherence and decoherence of macroscopic quantum states 1.0 even though we would have to use many more bits Interference and quantum functions in discretized approximations? March 32st, 2011 Neural Network Algorithms 13 Old View on Glia Myelinate for insulation only Clean-up and recycle neurotransmitters Feed and heal neurons …But, Einstein’s brain 0.5 Double the glia March 32st, 2011 Neural Network Algorithms 14 Recent Findings Glia-Neuron and Glia-Glia Information Processing Listen to all neurotransmitters, and uses them to communicate with both glia and neurons Control synapse formation and operations Connect neurons which have no synapses between them, and correlate them Run separate network in parallel to NNs Control speed of neuron’s output (axon) Most regulated genes during REM are in glia (integration/consolidation) 2.0 As many as 100,000 synapses per glia It’s a whole new brain out there… And there’s more: March 32st, 2011 Neural Network Algorithms 15 Glia Quantum Correlates Brain-wide calcium broadcast network Glia Quantum Correlates 1.0 Connect through gap junctions Calcium messaging affects neural circuits Drive global broadcast waves Calcium stores related to microtubules Gap junctions as hypothesized Quantum aspects might play a big function in glia networks March 32st, 2011 Neural Network Algorithms 16 Glia as Biological Bayesian? Accumulated effect from previous inputs (old posterior) serves as baseline for new input (new prior). Posterior (t ) Norm _ Const (t ) * Posterior (t 1) * Data(t ) 2.0 Glia excitations last second to minutes, compared to ms for neurons, and it span much wider This could produce something alike cumulative data likelihood during the period t of glia excitation Glia could then adjust/sample weights for neurons as per latest posterior (weight factors coupled to posterior) Would need a mechanism for normalization to 1 March 32st, 2011 Neural Network Algorithms 17 Conclusion Existing NN algorithms offer a rich toolkit for computing 1.0 Much more beyond plain backpropagation Take advantage of combinations and complex graphs Use as many of the superior modeling aspects as affordable Use probability theory Glia networks and interactions with neurons can be modeled in new algorithms Might be possible to simulate quantum effects for more enhancements March 32st, 2011 Neural Network Algorithms 18 The End Questions? March 32st, 2011 Neural Network Algorithms 19 References O’Reilly, Y. Munakata, “Computational Explorations in Cognitive Neuroscience, The MIT Press, 2000 V. Ivancevic, T. Ivancevic, “Quantum Neural Computation”, Springer, 2010 U. Ramacher, C. v.d. Malsburg, “On the Construction of Artificial Brains”, Springer, 2010 (Ed.) A. Volterra, P. Magistretti, P. Haydon, “The Tripartite Synapse”, Oxford University Press 2002 R. M. Neal, “Bayesian Learning for Neural Networks”, Springer, 1996 R. D. Fields, “The Other Brain”, Simon & Schuster 2009 S. Gupta, R. Zia, “Quantum Neural Networks”, Journal of Computer and Systems Sciences 63, 355383, 2001 A. A. Ezhov, D. Ventura, “Quantum Neural Networks”, Future Directions for Intelligent Systems and Information Science. Physica-Verlang, 2000 J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities”, Proc. Natl. Acad. Sci. USA Vol 79, pp. 2554-2558, April 1982 March 32st, 2011 Neural Network Algorithms 20 Additional Slides Existing NNs on Quantum Computers Quantum Computing 1. 2. 3. 4. N-qubit register can contain all 2N values at once You can have a quantum ‘circuit’ ‘computing’ on all of them at once But when you ‘touch it’, you will get one value only. Goal – how to touch it, to get the value you want, with high probability March 32st, 2011 Neural Network Algorithms 22