Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download PPT
Document related concepts
Transcript
ΗΜΥ 653 Ενσωματωμένα Συστήματα και Συστήματα Πραγματικού Χρόνου Εαρινό Εξάμηνο 2017 ΔΙΑΛΕΞΕΙΣ 5 - 6: Intelligent Embedded Systems - S/W and Real-Time O/S ΧΑΡΗΣ ΘΕΟΧΑΡΙΔΗΣ Επίκουρος Καθηγητής ΗΜΜΥ ([email protected]) Slides adopted from Prof. Peter Marwedel Additional Ack: Chris Gregg,UVA Ηλεκτρονικος Εγκέφαλος Κεντρικό Νευρικό Σύστημα Υλικό: Επεξεργαστές και Κυκλώματα Λογισμικό: Αλγόριθμοι Έλεγχου, Ανάλυσης και Επεξεργασίας Δεδομένων... Τεχνητή Νοημοσύνη Μνήμη ΗΜΥ653 Δ05-6 Embedded Systems Software.2 © Θεοχαρίδης, ΗΜΥ, 2017 «να καταλαβαίνει και να αντιλαμβάνεται το περιβάλλον» Πώς αντιλαμβανόμαστε εμείς το περιβάλλον μας? Ποια η διαφορά το «αντιλαμβάνομαι» και του «καταλαβαίνω»? Ακούω την ομιλία αυτή ΑΝΤΙΛΑΜΒΑΝΟΜΑΙ ότι κάποιος μιλά! Αυτή η ομιλία όμως δεν μου αρέσει! ΚΑΤΑΛΑΒΑΙΝΩ τί λέει και ΔΕΝ με αφορά! ΗΜΥ653 Δ05-6 Embedded Systems Software.3 © Θεοχαρίδης, ΗΜΥ, 2017 Δεδομένα και Αίσθηση! Σήμερα μπορούμε να μεταφέρουμε σε ένα υπολογιστή απεριόριστα δεδομένα • Ήχος, εικόνα, μετρήσεις αισθητήρων, κλπ. Ενσύρματα αλλά και Ασύρματα, με ΜΕΓΑΛΗ ΤΑΧΥΤΗΤΑ! ΠΩΣ ΑΝΤΙΛΑΜΒΑΝΕΤΑΙ ΚΑΙ ΠΩΣ ΚΑΤΑΝΟΕΙ ΤΑ ΔΕΔΟΜΕΝΑ ΕΝΑΣ ΥΠΟΛΟΓΙΣΤΗΣ ΟΜΩΣ; ΗΜΥ653 Δ05-6 Embedded Systems Software.4 © Θεοχαρίδης, ΗΜΥ, 2017 Intelligence! (Ευφυία!) ΗΜΥ653 Δ05-6 Embedded Systems Software.5 © Θεοχαρίδης, ΗΜΥ, 2017 Ο Εγκέφαλος (ηλεκτρονικός) σήμερα! ΗΜΥ653 Δ05-6 Embedded Systems Software.6 © Θεοχαρίδης, ΗΜΥ, 2017 Στην πράξη… Συλλογή, κωδικοποίηση και αποθήκευση δεδομένων Αποθήκευση και αναγνώριση προτύπων στα πλαίσια «εκπαίδευσης» – Τι σημαίνουν δηλαδή τα δεδομένα; Ποια δεδομένα είναι χρήσιμα; Πώς μπορώ να «εκπαιδεύσω» ένα ηλεκτρονικό σύστημα; ΗΜΥ653 Δ05-6 Embedded Systems Software.7 © Θεοχαρίδης, ΗΜΥ, 2017 ΠΡΑΓΜΑΤΙΚΟΣ ΧΡΟΝΟΣ ΑΝΤΑΠΟΚΡΙΣΗΣ ΗΜΥ653 Δ05-6 Embedded Systems Software.8 © Θεοχαρίδης, ΗΜΥ, 2017 ΑΠΟΦΑΣΕΙΣ! Τι κάνουμε με τα δεδομένα? Τι κάνει ένας άνθρωπος όταν δεί στον δρόμο ένα φορτηγό να τρέχει προς τα πάνω του? ΑΝΤΙΔΡΑΣΗ! ΗΜΥ653 Δ05-6 Embedded Systems Software.9 © Θεοχαρίδης, ΗΜΥ, 2017 We want to design some truly intelligent robot ΗΜΥ653 Δ05-6 Embedded Systems Software.10 © Θεοχαρίδης, ΗΜΥ, 2017 What is AI? Discipline that systematizes and automates intellectual tasks to create machines that: Act like humans Act rationally Think like humans Think rationally More formal and mathematical ΗΜΥ653 Δ05-6 Embedded Systems Software.11 © Θεοχαρίδης, ΗΜΥ, 2017 Some Important Achievements of AI Logic reasoning (data bases) Search and game playing Applied in reasoning interactive robot helper Applied in robot motion planning Knowledge-based systems Applied in robots that use knowledge like internet robots Bayesian networks (diagnosis) Machine learning and data mining Planning and military logistics Autonomous robots ΗΜΥ653 Δ05-6 Embedded Systems Software.12 © Θεοχαρίδης, ΗΜΥ, 2017 MAIN BRANCHES OF AI APPLICABLE TO ROBOTICS Artificial Intelligence Neural Nets ΗΜΥ653 Δ05-6 Embedded Systems Software.13 Fuzzy Logic Genetic Algorithms © Θεοχαρίδης, ΗΜΥ, 2017 What we’ll be doing Uncertain knowledge and reasoning Probability, Bayes rule Machine learning Decision trees, computationally learning theory, reinforcement learning ini W j ,i a j Wi a j j ΗΜΥ653 Δ05-6 Embedded Systems Software.14 © Θεοχαρίδης, ΗΜΥ, 2017 A Generalized Model of Learning Correct outputs Training System Output Input Learning Element Knowledge Base Performance Element Feedback Element Training system is used to create learning pairs (input, output) to train our robot Starts with some knowledge. The performance element participates in performance. The feedback element provides a comparison of actual output vs correct output which becomes input to the learning element. It analyzes the differences and updates the knowledge base. ΗΜΥ653 Δ05-6 Embedded Systems Software.15 © Θεοχαρίδης, ΗΜΥ, 2017 Correct outputs Training System Output Input Learning Element Knowledge Base Performance Element Feedback Element Starts with some knowledge. The performance element participates in performance. The feedback element provides a comparison of actual output vs correct output which becomes input to the learning element. It analyzes the differences and updates the knowledge base. ΗΜΥ653 Δ05-6 Embedded Systems Software.16 © Θεοχαρίδης, ΗΜΥ, 2017 Introduction to Pattern Recognition The applications of Pattern Recognition can be found everywhere. Examples include disease categorization, prediction of survival rates for patients of specific disease, fingerprint verification, face recognition, iris discrimination, chromosome shape discrimination, optical character recognition, texture discrimination, speech recognition, and etc. The design of a pattern recognition system should consider the application domain. A universally best pattern recognition system has never existed. ΗΜΥ653 Δ05-6 Embedded Systems Software.17 © Θεοχαρίδης, ΗΜΥ, 2017 A Pattern Recognition Paradigm ΗΜΥ653 Δ05-6 Embedded Systems Software.18 © Θεοχαρίδης, ΗΜΥ, 2017 Texture Discrimination ΗΜΥ653 Δ05-6 Embedded Systems Software.19 © Θεοχαρίδης, ΗΜΥ, 2017 Shape Discrimination ΗΜΥ653 Δ05-6 Embedded Systems Software.20 © Θεοχαρίδης, ΗΜΥ, 2017 Are They From the Same Person? ΗΜΥ653 Δ05-6 Embedded Systems Software.21 © Θεοχαρίδης, ΗΜΥ, 2017 What is pattern recognition? “The assignment of a physical object or event to one of several prespecified categeries” -- Duda & Hart A pattern is an object, process or event that can be given a name. A pattern class (or category) is a set of patterns sharing common attributes and usually originating from the same source. During recognition (or classification) given objects are assigned to prescribed classes. A classifier is a machine which performs classification. ΗΜΥ653 Δ05-6 Embedded Systems Software.22 © Θεοχαρίδης, ΗΜΥ, 2017 Examples of applications • Handwritten: sorting letters by postal code, input device for PDA‘s. • Optical Character Recognition (OCR) • Biometrics • Diagnostic systems • Military applications • Printed texts: reading machines for blind people, digitalization of text documents. • Face recognition, verification, retrieval. • Finger prints recognition. • Speech recognition. • Medical diagnosis: X-Ray, EKG analysis. • Machine diagnostics, waster detection. • Automated Target Recognition (ATR). • Image segmentation and analysis (recognition from aerial or satelite photographs). ΗΜΥ653 Δ05-6 Embedded Systems Software.23 © Θεοχαρίδης, ΗΜΥ, 2017 Examples of applications ΗΜΥ653 Δ05-6 Embedded Systems Software.24 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.25 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.26 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.27 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.28 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.29 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.30 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.31 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.32 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.33 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.34 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.35 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.36 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.37 © Θεοχαρίδης, ΗΜΥ, 2017 Overfitting and underfitting Problem: how rich class of classifications q(x;θ) to use. underfitting good fit overfitting Problem of generalization: a small emprical risk Remp does not imply small true expected risk R. ΗΜΥ653 Δ05-6 Embedded Systems Software.38 © Θεοχαρίδης, ΗΜΥ, 2017 Basic concepts Patter n x1 x 2 x xn y Hidden state Feature vector xX - A vector of observations (measurements). X . - x is a point in feature space y Y - Cannot be directly measured. - Patterns with equal hidden state belong to the same class. Task - To design a classifer (decision rule) q : X Y which decides about a hidden state based on an onbservation. ΗΜΥ653 Δ05-6 Embedded Systems Software.39 © Θεοχαρίδης, ΗΜΥ, 2017 Example height Task: horse jockey recognition. weight x1 x x 2 The set of hidden stateY is {H , J } The feature space X is 2 Training examples Linear classifier: {( x1 , y1 ), , (xl , yl )} yH x2 H if (w x) b 0 q(x) J if (w x) b 0 yJ ΗΜΥ653 Δ05-6 Embedded Systems Software.40 ( w x) b 0 x 1 © Θεοχαρίδης, ΗΜΥ, 2017 Components of PR system Pattern Sensors and preprocessin g Teacher Feature extraction Classifier Class assignment Learning algorithm • Sensors and preprocessing. • A feature extraction aims to create discriminative features good for classification. • A classifier. • A teacher provides information about hidden state -- supervised learning. • A learning algorithm sets PR from training examples. ΗΜΥ653 Δ05-6 Embedded Systems Software.41 © Θεοχαρίδης, ΗΜΥ, 2017 Feature extraction Task: to extract features which are good for classification. Good features:• Objects from the same class have similar feature values. • Objects from different classes have different values. “Good” features ΗΜΥ653 Δ05-6 Embedded Systems Software.42 “Bad” features © Θεοχαρίδης, ΗΜΥ, 2017 Feature extraction methods Feature extraction m1 m 2 mk φ1 φ2 φn x1 x 2 xn Feature selection m1 m 2 m3 mk x1 x 2 xn φ(θ) Problem can be expressed as optimization of parameters of featrure extractor . Supervised methods: objective function is a criterion of separability (discriminability) of labeled examples, e.g., linear discriminat analysis (LDA). Unsupervised methods: lower dimesional representation which preserves important characteristics of input data is sought for, e.g., principal component ΗΜΥ653 Δ05-6 Embedded Systems Software.43 © Θεοχαρίδης, ΗΜΥ, 2017 analysis (PCA). Classifier A classifier partitions feature space X into class-labeled regions such that X X 1 X 2 X |Y | and X 1 X 2 X |Y | {0} X1 X1 X3 X2 X1 X2 X3 The classification consists of determining to which region a feature vector x belongs to. Borders between decision boundaries are called decision regions. ΗΜΥ653 Δ05-6 Embedded Systems Software.44 © Θεοχαρίδης, ΗΜΥ, 2017 Representation of classifier A classifier is typically represented as a set of discriminant functions f i (x) : X , i 1,, | Y | The classifier assigns a feature vector x to the i-the class if f i (x) f j (x) j i f1 (x) x Feature vector ΗΜΥ653 Δ05-6 Embedded Systems Software.45 f 2 (x) max y Class identifier f|Y | (x) Discriminant function © Θεοχαρίδης, ΗΜΥ, 2017 Review Fig.1 Basic components of a pattern recognition system ΗΜΥ653 Δ05-6 Embedded Systems Software.46 © Θεοχαρίδης, ΗΜΥ, 2017 Steps Data acquisition and sensing Pre-processing Removal of noise in data. Isolation of patterns of interest from the background. Feature extraction Finding a new representation in terms of features. (Better for further processing) ΗΜΥ653 Δ05-6 Embedded Systems Software.47 © Θεοχαρίδης, ΗΜΥ, 2017 Steps Model learning and estimation Learning a mapping between features and pattern groups. Classification Using learned models to assign a pattern to a predefined category Post-processing Evaluation of confidence in decisions. Exploitation of context to improve performances. ΗΜΥ653 Δ05-6 Embedded Systems Software.48 © Θεοχαρίδης, ΗΜΥ, 2017 Table 1 : Examples of pattern recognition applications ΗΜΥ653 Δ05-6 Embedded Systems Software.49 © Θεοχαρίδης, ΗΜΥ, 2017 Bayes Statistical Classifiers Consideration Randomness of patterns Decision criterion Pattern x is labeled as class wi if W W L p(x / w )P(w ) < L p(x / w )P(w ) ki k=1 k k qj q q q=1 Lij : Misclassification loss function p(x/wi) : P.d.f. of a particular pattern x comes from class wi P(wi) : Probability of occurrence of class wi ΗΜΥ653 Δ05-6 Embedded Systems Software.50 © Θεοχαρίδης, ΗΜΥ, 2017 Bayes Statistical Classifiers Decision criterion : Given Lij is symmetrical function Posterior probability decision rule p(x / wi)P(wi) > p(x / wj)P(wj) dj(x)= p(x / wj)P(wj)= P(wj / x) dj(x) : decision functions Pattern x classifies to class j if dj(x) yields the largest value ΗΜΥ653 Δ05-6 Embedded Systems Software.51 © Θεοχαρίδης, ΗΜΥ, 2017 Classification in Statistical PR • A class is a set of objects having some important properties in common • A feature extractor is a program that inputs the data (image) and extracts features that can be used in classification. • A classifier is a program that inputs the feature vector and assigns it to one of a set of designated classes or to the “reject” class. With what kinds of classes do you work? ΗΜΥ653 Δ05-6 Embedded Systems Software.52 © Θεοχαρίδης, ΗΜΥ, 2017 Feature Vector Representation X=[x1, x2, … , xn], each xj a real number xj may be an object measurement xj may be count of object parts Example: object rep. [#holes, #strokes, moments, …] ΗΜΥ653 Δ05-6 Embedded Systems Software.53 © Θεοχαρίδης, ΗΜΥ, 2017 Possible features for char rec. ΗΜΥ653 Δ05-6 Embedded Systems Software.54 © Θεοχαρίδης, ΗΜΥ, 2017 Some Terminology Classes: set of m known categories of objects (a) might have a known description for each (b) might have a set of samples for each Reject Class: a generic class for objects not in any of the designated known classes Classifier: Assigns object to a class based on features ΗΜΥ653 Δ05-6 Embedded Systems Software.55 © Θεοχαρίδης, ΗΜΥ, 2017 Discriminant functions Functions f(x, K) perform some computation on feature vector x Knowledge K from training or programming is used Final stage determines class ΗΜΥ653 Δ05-6 Embedded Systems Software.56 © Θεοχαρίδης, ΗΜΥ, 2017 Classification using nearest class mean Compute the Euclidean distance between feature vector X and the mean of each class. Choose closest class, if close enough (reject otherwise) ΗΜΥ653 Δ05-6 Embedded Systems Software.57 © Θεοχαρίδης, ΗΜΥ, 2017 Nearest mean might yield poor results with complex structure ΗΜΥ653 Δ05-6 Embedded Systems Software.58 Class 2 has two modes; where is its mean? But if modes are detected, two subclass mean vectors can be used © Θεοχαρίδης, ΗΜΥ, 2017 Classifiers • Decision Tree Classifiers • Artificial Neural Net Classifiers • Bayesian Classifiers and Bayesian Networks (Graphical Models) • Support Vector Machines ΗΜΥ653 Δ05-6 Embedded Systems Software.59 © Θεοχαρίδης, ΗΜΥ, 2017 Human Brain 60 ΗΜΥ653 Δ05-6 Embedded Systems Software.60 © Θεοχαρίδης, ΗΜΥ, 2017 Biological Neural Networks Several neurons are connected to one another to form a neural network or a layer of a neural network. 61 ΗΜΥ653 Δ05-6 Embedded Systems Software.61 © Θεοχαρίδης, ΗΜΥ, 2017 Introduction – Neural Nets What are Neural Networks? Neural networks are a paradigm of programming computers. They are exceptionally good at performing pattern recognition and other tasks that are very difficult to program using conventional techniques. Programs that employ neural nets are also capable of learning on their own and adapting to changing conditions. ΗΜΥ653 Δ05-6 Embedded Systems Software.62 © Θεοχαρίδης, ΗΜΥ, 2017 Background An Artificial Neural Network (ANN) is an information processing paradigm that is inspired by the biological nervous systems, such as the human brain’s information processing mechanism. The key element of this paradigm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements (neurons) working in unison to solve specific problems. NNs, like people, learn by example. An NN is configured for a specific application, such as pattern recognition or data classification, through a learning process. Learning in biological systems involves adjustments to the synaptic connections that exist between the neurons. This is true of NNs as well. ΗΜΥ653 Δ05-6 Embedded Systems Software.63 © Θεοχαρίδης, ΗΜΥ, 2017 Artificial Neural Networks Activation function: Threshold logic function (Perceptron) Sigmoidal function (MLFFNN) Soft-max function (MLFFNN for classification) Recti-linear function (CNN) Gaussian function (RBFNN for regression) Spiking function (Pulsed NN) Structure of network: Feedforward neural networks Feedback neural networks Feedforward and feedback neural networks Learning method: Supervised learning Unsupervised learning Competitive learning Pattern analysis task: ΗΜΥ653 Δ05-6 Embedded Systems Software.64 64 © Θεοχαρίδης, ΗΜΥ, 2017 How the Human Brain learns In the human brain, a typical neuron collects signals from others through a host of fine structures called dendrites. The neuron sends out spikes of electrical activity through a long, thin stand known as an axon, which splits into thousands of branches. At the end of each branch, a structure called a synapse converts the activity from the axon into electrical effects that inhibit or excite activity in the connected neurons. ΗΜΥ653 Δ05-6 Embedded Systems Software.65 © Θεοχαρίδης, ΗΜΥ, 2017 Neural Networks Biological Biological approach to AI Developed in 1943 Comprised of one or more layers of neurons Several types, we’ll focus on feed-forward networks Artificial Neurons http://faculty.washington.edu/chudler/color/pic1an.gif http://research.yale.edu/ysm/images/78.2/articles-neural-neuron.jpg ΗΜΥ653 Δ05-6 Embedded Systems Software.66 © Θεοχαρίδης, ΗΜΥ, 2017 A Neuron Receives n-inputs Multiplies each input by its weight Applies activation function to the sum of results Outputs result http://www-cse.uta.edu/~cook/ai1/lectures/figures/neuron.jpg ΗΜΥ653 Δ05-6 Embedded Systems Software.67 © Θεοχαρίδης, ΗΜΥ, 2017 A Neuron Model When a neuron receives excitatory input that is sufficiently large compared with its inhibitory input, it sends a spike of electrical activity down its axon. Learning occurs by changing the effectiveness of the synapses so that the influence of one neuron on another changes. ΗΜΥ653 Δ05-6 Embedded Systems Software.68 © Θεοχαρίδης, ΗΜΥ, 2017 Activation Functions Controls when unit is “active” or “inactive” Threshold function outputs 1 when input is positive and 0 otherwise i.e. Sigmoid function = 1 / (1 + e-x) Hyperbolic Tangent…etc. ΗΜΥ653 Δ05-6 Embedded Systems Software.69 © Θεοχαρίδης, ΗΜΥ, 2017 Neural Network Layers Each layer receives its inputs from the previous layer and forwards its outputs to the next layer http://smig.usgs.gov/SMIG/features_0902/tualatin_ann.fig3.gif ΗΜΥ653 Δ05-6 Embedded Systems Software.70 © Θεοχαρίδης, ΗΜΥ, 2017 Multilayer Feedforward Neural Network Architecture of an MLFFNN Input layer: Linear neurons Hidden layers (1 or 2): Sigmoidal neurons Output layer: - Linear neurons (for function approximation task) - Sigmoidal neurons (for pattern classification task) Input Layer x1 1 Hidden Layer Output Layer 1 1 s1o s2o x2 2 2 2 . . . . . . . . . . . . xi i j k sko K o sK . . . xd . . . d ΗΜΥ653 Δ05-6 Embedded Systems Software.71 . . . J . . . 71 © Θεοχαρίδης, ΗΜΥ, 2017 Pattern Recognition An important application of neural networks is pattern recognition. Pattern recognition can be implemented by using a feed-forward neural network that has been trained accordingly. During training, the network is trained to associate outputs with input patterns. When the network is used, it identifies the input pattern and tries to output the associated output pattern. The power of neural networks comes to life when a pattern that has no output associated with it, is given as an input. In this case, the network gives the output that corresponds to a taught input pattern that is least different from the given pattern. ΗΜΥ653 Δ05-6 Embedded Systems Software.72 © Θεοχαρίδης, ΗΜΥ, 2017 Pattern Recognition (cont.) Suppose a network is trained to recognize the patterns T and H. The associated patterns are all black and all white respectively as shown above. ΗΜΥ653 Δ05-6 Embedded Systems Software.73 © Θεοχαρίδης, ΗΜΥ, 2017 Pattern Recognition (cont.) Since the input pattern looks more like a ‘T’, when the network classifies it, it sees the input closely resembling ‘T’ and outputs the pattern that represents a ‘T’. ΗΜΥ653 Δ05-6 Embedded Systems Software.74 © Θεοχαρίδης, ΗΜΥ, 2017 Pattern Recognition (cont.) The input pattern here closely resembles ‘H’ with a slight difference. The network in this case classifies it as an ‘H’ and outputs the pattern representing an ‘H’. ΗΜΥ653 Δ05-6 Embedded Systems Software.75 © Θεοχαρίδης, ΗΜΥ, 2017 Pattern Recognition (cont.) Here the top row is 2 errors away from a ‘T’ and 3 errors away from an H. So the top output is a black. The middle row is 1 error away from both T and H, so the output is random. The bottom row is 1 error away from T and 2 away from H. Therefore the output is black. Since the input resembles a ‘T’ more than an ‘H’ the output of the network is in favor of a ‘T’. ΗΜΥ653 Δ05-6 Embedded Systems Software.76 © Θεοχαρίδης, ΗΜΥ, 2017 Learning by Back-Propagation: Illustration ARTIFICIAL NEURAL NETWORKS Colin Fahey's Guide (Book CD) ΗΜΥ653 Δ05-6 Embedded Systems Software.77 © Θεοχαρίδης, ΗΜΥ, 2017 Computational Complexity Could lead to a very large number of calculations Input Units Influence Map Layer 1 Hidden Units Output Units Influence Map Layer 2 ΗΜΥ653 Δ05-6 Embedded Systems Software.78 © Θεοχαρίδης, ΗΜΥ, 2017 Standard computers Referred to as Von Neumann machines Follows explicit instructions Sample program if (time < noon) print “Good morning” else print “Good afternoon” No learning No autonomous intelligence ΗΜΥ653 Δ05-6 Embedded Systems Software.79 © Θεοχαρίδης, ΗΜΥ, 2017 Neural networks Modeled Does Is off the human brain not follow explicit instructions trained instead of programmed Key papers McCulloch, W. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 7:115 - 133. Rosenblatt, Frank. (1958) The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review, 65:386-408. ΗΜΥ653 Δ05-6 Embedded Systems Software.80 © Θεοχαρίδης, ΗΜΥ, 2017 Sources for lecture comp.ai.neural-networks FAQ Gurney, Kevin. An Introduction to Neural Networks, 1996. 81 ΗΜΥ653 Δ05-6 Embedded Systems Software.81 © Θεοχαρίδης, ΗΜΥ, 2017 Neuron drawing 82 ΗΜΥ653 Δ05-6 Embedded Systems Software.82 © Θεοχαρίδης, ΗΜΥ, 2017 Neuron behavior Signals travel between neurons through electrical pulses Within neurons, communication is through chemical neurotransmitters If the inputs to a neuron are greater than its threshold, the neuron fires, sending an electrical pulse to other neurons This is a simplification. ΗΜΥ653 Δ05-6 Embedded Systems Software.83 © Θεοχαρίδης, ΗΜΥ, 2017 Perceptron (artificial neuron) inputs weights threshold xa output + >10? xb ΗΜΥ653 Δ05-6 Embedded Systems Software.84 © Θεοχαρίδης, ΗΜΥ, 2017 Training Inputs and outputs are 0 (no) or 1 (yes) Initially, weights are random Provide training input Compare output of neural network to desired output If same, reinforce patterns If different, adjust weights ΗΜΥ653 Δ05-6 Embedded Systems Software.85 © Θεοχαρίδης, ΗΜΥ, 2017 Example If both inputs are 1, output should be 1. inputs weights threshold x2 output + >10? x3 ΗΜΥ653 Δ05-6 Embedded Systems Software.86 © Θεοχαρίδης, ΗΜΥ, 2017 Example (1,1) If both inputs are 1, output should be 1. inputs 1 weights threshold x2 output + 1 >10? x3 ΗΜΥ653 Δ05-6 Embedded Systems Software.87 © Θεοχαρίδης, ΗΜΥ, 2017 Example (1,1) If both inputs are 1, output should be 1. inputs 1 weights x2 threshold 2 output + 1 x3 ΗΜΥ653 Δ05-6 Embedded Systems Software.88 >10? 3 © Θεοχαρίδης, ΗΜΥ, 2017 Example (1,1) If both inputs are 1, output should be 1. inputs 1 weights x2 threshold 2 + 1 x3 ΗΜΥ653 Δ05-6 Embedded Systems Software.89 5 output >10? 3 © Θεοχαρίδης, ΗΜΥ, 2017 Example (1,1) If both inputs are 1, output should be 1. inputs 1 weights x2 threshold 2 + 1 x3 ΗΜΥ653 Δ05-6 Embedded Systems Software.90 5 output >10? 0 3 © Θεοχαρίδης, ΗΜΥ, 2017 Example (1,1) If both inputs are 1, output should be 1. inputs 1 weights x2 threshold 2 + 1 x3 ΗΜΥ653 Δ05-6 Embedded Systems Software.91 5 output >10? 0 3 © Θεοχαρίδης, ΗΜΥ, 2017 Example (1,1) If both inputs are 1, output should be 1. inputs 1 weights x2 threshold 2 + 1 x3 ΗΜΥ653 Δ05-6 Embedded Systems Software.92 5 output >10? 0 3 © Θεοχαρίδης, ΗΜΥ, 2017 Example (1,1) If both inputs are 1, output should be 1. inputs 1 weights threshold x2 output + 1 >10? x3 Repeat for all inputs until weights stop changing. ΗΜΥ653 Δ05-6 Embedded Systems Software.93 © Θεοχαρίδης, ΗΜΥ, 2017 Function Learning Formulation Goal function f Training set: (x(i), f(x(i))), i = 1,…,n Inductive inference: find a function h that fits the points well f(x) x ΗΜΥ653 Δ05-6 Embedded Systems Software.94 © Θεοχαρίδης, ΗΜΥ, 2017 Different types of Neural Networks Feed-forward networks Feed-forward NNs allow signals to travel one way only; from input to output. There is no feedback (loops) i.e. the output of any layer does not affect that same layer. Feed-forward NNs tend to be straight forward networks that associate inputs with outputs. They are extensively used in pattern recognition. This type of organization is also referred to as bottom-up or topdown. ΗΜΥ653 Δ05-6 Embedded Systems Software.95 © Θεοχαρίδης, ΗΜΥ, 2017 Continued Feedback networks Feedback networks can have signals traveling in both directions by introducing loops in the network. Feedback networks are dynamic; their 'state' is changing continuously until they reach an equilibrium point. They remain at the equilibrium point until the input changes and a new equilibrium needs to be found. Feedback architectures are also referred to as interactive or recurrent, although the latter term is often used to denote feedback connections in single-layer organizations. ΗΜΥ653 Δ05-6 Embedded Systems Software.96 © Θεοχαρίδης, ΗΜΥ, 2017 Diagram of an NN Fig: A simple Neural Network ΗΜΥ653 Δ05-6 Embedded Systems Software.97 © Θεοχαρίδης, ΗΜΥ, 2017 Network Layers Input Layer - The activity of the input units represents the raw information that is fed into the network. Hidden Layer - The activity of each hidden unit is determined by the activities of the input units and the weights on the connections between the input and the hidden units. Output Layer - The behavior of the output units depends on the activity of the hidden units and the weights between the hidden and output units. ΗΜΥ653 Δ05-6 Embedded Systems Software.98 © Θεοχαρίδης, ΗΜΥ, 2017 Continued This simple type of network is interesting because the hidden units are free to construct their own representations of the input. The weights between the input and hidden units determine when each hidden unit is active, and so by modifying these weights, a hidden unit can choose what it represents. ΗΜΥ653 Δ05-6 Embedded Systems Software.99 © Θεοχαρίδης, ΗΜΥ, 2017 Network Structure The number of layers and of neurons depend on the specific task. In practice this issue is solved by trial and error. Two types of adaptive algorithms can be used: start from a large network and successively remove some neurons and links until network performance degrades. begin with a small network and introduce new neurons until performance is satisfactory. ΗΜΥ653 Δ05-6 Embedded Systems Software.100 © Θεοχαρίδης, ΗΜΥ, 2017 Network Parameters How are the weights initialized? How many hidden layers and how many neurons? How many examples in the training set? ΗΜΥ653 Δ05-6 Embedded Systems Software.101 © Θεοχαρίδης, ΗΜΥ, 2017 Weights In general, initial weights are randomly chosen, with typical values between -1.0 and 1.0 or -0.5 and 0.5. There are two types of NNs. The first type is known as Fixed Networks – where the weights are fixed Adaptive Networks – where the weights are changed to reduce prediction error. ΗΜΥ653 Δ05-6 Embedded Systems Software.102 © Θεοχαρίδης, ΗΜΥ, 2017 Neurone vs. Node ΗΜΥ653 Δ05-6 Embedded Systems Software.103 © Θεοχαρίδης, ΗΜΥ, 2017 Structure of a node: Squashing function limits node output: ΗΜΥ653 Δ05-6 Embedded Systems Software.104 © Θεοχαρίδης, ΗΜΥ, 2017 Feed-forward nets Information flow is unidirectional Data is presented to Input layer Passed on to Hidden Layer Passed on to Output layer Information is distributed Information processing is parallel Internal representation (interpretation) of data ΗΜΥ653 Δ05-6 Embedded Systems Software.105 © Θεοχαρίδης, ΗΜΥ, 2017 Feeding data through the net: (1 0.25) + (0.5 (-1.5)) = 0.25 + (-0.75) = - 0.5 Squashing: ΗΜΥ653 Δ05-6 Embedded Systems Software.106 1 0.3775 0.5 1 e © Θεοχαρίδης, ΗΜΥ, 2017 Data is presented to the network in the form of activations in the input layer Examples Data usually requires preprocessing Pixel intensity (for pictures) Molecule concentrations (for artificial nose) Share prices (for stock market prediction) Analogous to senses in biology How to represent more abstract data, e.g. a name? Choose a pattern, e.g. - 0-0-1 for “Chris” - 0-1-0 for “Becky” ΗΜΥ653 Δ05-6 Embedded Systems Software.107 © Θεοχαρίδης, ΗΜΥ, 2017 Weight settings determine the behaviour of a network How can we find the right weights? ΗΜΥ653 Δ05-6 Embedded Systems Software.108 © Θεοχαρίδης, ΗΜΥ, 2017 Training the Network - Learning Backpropagation Requires training set (input / output pairs) Starts with small random weights Error is used to adjust weights (supervised learning) Gradient descent on error landscape ΗΜΥ653 Δ05-6 Embedded Systems Software.109 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.110 © Θεοχαρίδης, ΗΜΥ, 2017 Advantages Downsides It works! Relatively fast Requires a training set Can be slow Probably not biologically realistic Alternatives to Backpropagation Hebbian learning - Not successful in feed-forward nets Reinforcement learning - Only limited success Artificial evolution - More general, but can be even slower than backprop ΗΜΥ653 Δ05-6 Embedded Systems Software.111 © Θεοχαρίδης, ΗΜΥ, 2017 Example: Voice Recognition Task: Learn to discriminate between two different voices saying “Hello” Data Sources - Steve Simpson - David Raubenheimer Format - Frequency distribution (60 bins) - Analogy: cochlea ΗΜΥ653 Δ05-6 Embedded Systems Software.112 © Θεοχαρίδης, ΗΜΥ, 2017 Network architecture Feed forward network - 60 input (one for each frequency bin) - 6 hidden - 2 output (0-1 for “Steve”, 1-0 for “David”) ΗΜΥ653 Δ05-6 Embedded Systems Software.113 © Θεοχαρίδης, ΗΜΥ, 2017 Presenting the data Steve David ΗΜΥ653 Δ05-6 Embedded Systems Software.114 © Θεοχαρίδης, ΗΜΥ, 2017 Presenting the data (untrained network) Steve 0.43 0.26 David 0.73 0.55 ΗΜΥ653 Δ05-6 Embedded Systems Software.115 © Θεοχαρίδης, ΗΜΥ, 2017 Calculate error Steve 0.43 – 0 = 0.43 0.26 –1 = 0.74 0.73 – 1 = 0.27 0.55 – 0 = 0.55 David ΗΜΥ653 Δ05-6 Embedded Systems Software.116 © Θεοχαρίδης, ΗΜΥ, 2017 Backprop error and adjust weights Steve 0.43 – 0 = 0.43 0.26 – 1 = 0.74 1.17 David 0.73 – 1 = 0.27 0.55 – 0 = 0.55 0.82 ΗΜΥ653 Δ05-6 Embedded Systems Software.117 © Θεοχαρίδης, ΗΜΥ, 2017 Repeat process (sweep) for all training pairs Present data Calculate error Backpropagate error Adjust weights Repeat process multiple times ΗΜΥ653 Δ05-6 Embedded Systems Software.118 © Θεοχαρίδης, ΗΜΥ, 2017 Presenting the data (trained network) Steve 0.01 0.99 David 0.99 0.01 ΗΜΥ653 Δ05-6 Embedded Systems Software.119 © Θεοχαρίδης, ΗΜΥ, 2017 Results – Voice Recognition Performance of trained network - Discrimination accuracy between known “Hello”s – 100% - Discrimination accuracy between new “Hello”’s – 100% ΗΜΥ653 Δ05-6 Embedded Systems Software.120 © Θεοχαρίδης, ΗΜΥ, 2017 Size of Training Data Rule of thumb: the number of training examples should be at least five to ten times the number of weights of the network. Other rule: |W| N (1 - a) ΗΜΥ653 Δ05-6 Embedded Systems Software.121 |W|= number of weights a = expected accuracy on test set © Θεοχαρίδης, ΗΜΥ, 2017 Training Basics The most basic method of training a neural network is trial and error. If the network isn't behaving the way it should, change the weighting of a random link by a random amount. If the accuracy of the network declines, undo the change and make a different one. It takes time, but the trial and error method does produce results. ΗΜΥ653 Δ05-6 Embedded Systems Software.122 © Θεοχαρίδης, ΗΜΥ, 2017 Training: Backprop algorithm The Backprop algorithm searches for weight values that minimize the total error of the network over the set of training examples (training set). Backprop consists of the repeated application of the following two passes: Forward pass: in this step the network is activated on one example and the error of (each neuron of) the output layer is computed. Backward pass: in this step the network error is used for updating the weights. Starting at the output layer, the error is propagated backwards through the network, layer by layer. This is done by recursively computing the local gradient of each neuron. ΗΜΥ653 Δ05-6 Embedded Systems Software.123 © Θεοχαρίδης, ΗΜΥ, 2017 Back Propagation Learning Methodology Back-propagation training algorithm Network activation Forward Step Error propagation Backward Step Backprop adjusts the weights of the NN in order to minimize the network total mean squared error. ΗΜΥ653 Δ05-6 Embedded Systems Software.124 © Θεοχαρίδης, ΗΜΥ, 2017 The Learning Process (cont.) Every neural network possesses knowledge which is contained in the values of the connection weights. Modifying the knowledge stored in the network as a function of experience implies a learning rule for changing the values of the weights. ΗΜΥ653 Δ05-6 Embedded Systems Software.125 © Θεοχαρίδης, ΗΜΥ, 2017 The Learning Process (cont.) Recall: Adaptive networks are NNs that allow the change of weights in its connections. The learning methods can be classified in two categories: Supervised Learning Unsupervised Learning ΗΜΥ653 Δ05-6 Embedded Systems Software.126 © Θεοχαρίδης, ΗΜΥ, 2017 Supervised Learning Supervised learning which incorporates an external teacher, so that each output unit is told what its desired response to input signals ought to be. An important issue concerning supervised learning is the problem of error convergence, ie the minimization of error between the desired and computed unit values. The aim is to determine a set of weights which minimizes the error. One well-known method, which is common to many learning paradigms is the least mean square (LMS) convergence. ΗΜΥ653 Δ05-6 Embedded Systems Software.127 © Θεοχαρίδης, ΗΜΥ, 2017 Supervised Learning In this sort of learning, the human teacher’s experience is used to tell the NN which outputs are correct and which are not. This does not mean that a human teacher needs to be present at all times, only the correct classifications gathered from the human teacher on a domain needs to be present. The network then learns from its error, that is, it changes its weight to reduce its prediction error. ΗΜΥ653 Δ05-6 Embedded Systems Software.128 © Θεοχαρίδης, ΗΜΥ, 2017 Unsupervised Learning Unsupervised learning uses no external teacher and is based upon only local information. It is also referred to as self-organization, in the sense that it self-organizes data presented to the network and detects their emergent collective properties. The network is then used to construct clusters of similar patterns. This is particularly useful is domains were a instances are checked to match previous scenarios. For example, detecting credit card fraud. ΗΜΥ653 Δ05-6 Embedded Systems Software.129 © Θεοχαρίδης, ΗΜΥ, 2017 Neural Network in Use Since neural networks are best at identifying patterns or trends in data, they are well suited for prediction or forecasting needs including: sales forecasting industrial process control customer research data validation risk management ANN are also used in the following specific paradigms: recognition of speakers in communications; diagnosis of hepatitis; undersea mine detection; texture analysis; threedimensional object recognition; hand-written word recognition; and facial recognition. ΗΜΥ653 Δ05-6 Embedded Systems Software.130 © Θεοχαρίδης, ΗΜΥ, 2017 Multilayer Feedforward Neural Network Architecture of an MLFFNN Input layer: Linear neurons Hidden layers (1 or 2): Sigmoidal neurons Output layer: - Linear neurons (for function approximation task) - Sigmoidal neurons (for pattern classification task) Input Layer x1 1 Hidden Layer Output Layer 1 1 s1o s2o x2 2 2 2 . . . . . . . . . . . . xi i j k sko K o sK . . . xd . . . d ΗΜΥ653 Δ05-6 Embedded Systems Software.131 . . . J . . . 131 © Θεοχαρίδης, ΗΜΥ, 2017 ANNs – The basics ANNs incorporate the two fundamental components of biological neural nets: 1. Neurones (nodes) 2. Synapses (weights) ΗΜΥ653 Δ05-6 Embedded Systems Software.132 © Θεοχαρίδης, ΗΜΥ, 2017 Perceptron (The goal function f is a boolean one) + + x1 + + - - - + xi wi x2 S g y w 1 x1 + w 2 x2 = 0 x1 - xn y = g(Si=1,…,n wi xi) ΗΜΥ653 Δ05-6 Embedded Systems Software.133 © Θεοχαρίδης, ΗΜΥ, 2017 Perceptron (The goal function f is a boolean one) + + x0 xi wi ? - S g y + + - + - xn y = g(Si=1,…,n wi xi) ΗΜΥ653 Δ05-6 Embedded Systems Software.134 © Θεοχαρίδης, ΗΜΥ, 2017 Unit (Neuron) x1 xi wi S g y xn y = g(Si=1,…,n wi xi) g(u) = 1/[1 + exp(-au)] ΗΜΥ653 Δ05-6 Embedded Systems Software.135 © Θεοχαρίδης, ΗΜΥ, 2017 Neural Network Network of interconnected neurons x1 xi w x1 i xi w i xn S g y S g y xn Acyclic (feed-forward) vs. recurrent networks ΗΜΥ653 Δ05-6 Embedded Systems Software.136 © Θεοχαρίδης, ΗΜΥ, 2017 Two-Layer Feed-Forward Neural Network w1j Inputs ΗΜΥ653 Δ05-6 Embedded Systems Software.137 w2k Hidden layer Output layer © Θεοχαρίδης, ΗΜΥ, 2017 Comments and Issues on ANN How to choose the size and structure of networks? • If network is too large, risk of over-fitting (data caching) • If network is too small, representation may not be rich enough Role of representation: e.g., learn the concept of an odd number Incremental learning ΗΜΥ653 Δ05-6 Embedded Systems Software.138 © Θεοχαρίδης, ΗΜΥ, 2017 Artificial Neural Networks Neural networks are composed of: Layers of nodes (artificial neurons) Weights connecting the layers of nodes Different types of neural networks: Radial basis function networks Multi-layer feed forward networks Recurrent networks (feedback) Learning algorithms: Modify the weights connecting the nodes ΗΜΥ653 Δ05-6 Embedded Systems Software.139 © Θεοχαρίδης, ΗΜΥ, 2017 Convolutional Neural Networks ΗΜΥ653 Δ05-6 Embedded Systems Software.140 © Θεοχαρίδης, ΗΜΥ, 2017 Convolution Neural Networks Convolutional neural network (CNN) is a special type of multilayer feedforward neural network (MLFFNN) that is well suited for pattern classification. Development motivated. A CNN is an MLFFNN designed specifically to recognize 2-dimensional shapes with a high degree of invariance to translation, scaling, skewing and other forms of distortion. of CNN is neuro-biologically S. Haykin, Neural Networks and Learning Machines, Prentice-Hall of India, 2011 ΗΜΥ653 Δ05-6 Embedded Systems Software.141 © Θεοχαρίδης, ΗΜΥ, 2017 Structure of a CNN Feature extraction: Each neuron takes inputs from a local receptive field in the previous layer, thereby forcing it to extract local features. Feature mapping: Each convolutional layer of a CNN is composed of multiple feature maps. Each feature map is in the form of a plane within which the individual neurons are constrained to share the same set of synaptic weights. Effects of this constraint are the following: Shift invariance, forced into the operation of a feature map through the use of convolution, followed by a sigmoid function. Reduction in number of free parameters, accomplished through the use of weight sharing. Subsampling: Each convolutional layer is followed by a subsampling layer that performs local averaging and subsampling, whereby the resolution of feature map is reduced. Weights of connections to nodes in the convolutional layers are learned through training. ΗΜΥ653 Δ05-6 Embedded Systems Software.142 © Θεοχαρίδης, ΗΜΥ, 2017 LeNet5: CNN for Handwritten Character Recognition Input 32x32 6 Feature Maps 28x28 Convolution 6 Feature Maps 14x14 16 16 Feature Maps Feature Maps 10x10 5x5 Convolution Output 26 Subsampling Subsampling Input: 32x32 pixel image of a character centered and normalized in size Overlapping receptive field for a neuron in a convolutional layer with sigmoidal neurons: – For a node in the 1st hidden layer: 5x5 units in the input layer – For a node in the 3rd hidden layer: 5x5 units in 3, 4 or 6 feature maps in the 2nd hidden layer Non-overlapping receptive field for a neuron in a feature map of a subsampling layer (2nd and 4th hidden layers) with linear neurons: 2x2 units in the corresponding feature map in the previous layer Weight sharing: All the nodes in a feature map in a convolutional layer have the same synaptic weights (~278000 connections, but only ~1700 weight parameters) Output layer: 26 nodes with one node for each character. Each node in the output layer is connected to the nodes in all the feature maps in the 4th hidden layer. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of IEEE, vol.86, no.11, pp.2278-2324, November 1998. ΗΜΥ653 Δ05-6 Embedded Systems Software.143 © Θεοχαρίδης, ΗΜΥ, 2017 ROC Curves (Receiver operating characteristics) Plots trade-off between false positives and false negatives for different values of a threshold Figure from “Statistical color models with application to skin detection,” M.J. Jones and J. Rehg, Proc. Computer Vision and Pattern Recognition, 1999 copyright 1999, IEEE ΗΜΥ653 Δ05-6 Embedded Systems Software.144 © Θεοχαρίδης, ΗΜΥ, 2017 In what problems ANNs are good? Noisy data from sensors overhead camera poor for orientation Some sensors don’t work well Can’t add new sensors (rules of the game) Other parts of the system constrain available information (eg. radio communications takes 25ms, frame processing takes 15ms, etc) ΗΜΥ653 Δ05-6 Embedded Systems Software.145 © Θεοχαρίδης, ΗΜΥ, 2017 Face recognition 146 Steve Lawrence, C. Lee Giles, A.C. Tsoi and A.D. Back. Face Recognition: A Convolutional Neural Network Approach. IEEE Transactions on Neural Networks, Special Issue on Neural Networks and Pattern Recognition, Volume 8, Number 1, pp. 98-113, 1997. © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.146 Famous Applications NASA’s intelligent flight control system (F15As and C19s – NOT space shuttle) Handwritten number recognition (postcodes on snail mail) Robotic arm control Number plate recognition on moving cars Detection of cancerous cells in smear tests ΗΜΥ653 Δ05-6 Embedded Systems Software.147 © Θεοχαρίδης, ΗΜΥ, 2017 Example: Application of NN to Motion Planning (Climbing Robot) ΗΜΥ653 Δ05-6 Embedded Systems Software.148 © Θεοχαρίδης, ΗΜΥ, 2017 Transition one-step planning initial 4-hold stance ... 3-hold stance ... ... ... 4-hold stance ΗΜΥ653 Δ05-6 Embedded Systems Software.149 one-step planning contact / zero force breaking © Θεοχαρίδης, ΗΜΥ, 2017 Idea: Learn Feasibility Create a large database of labeled transitions Train a NN classifier Q : transition {feasible, not feasible) Learning is possible because: Shape of a feasible space is mostly determined by the equilibrium condition that depends on relatively few parameters ΗΜΥ653 Δ05-6 Embedded Systems Software.150 © Θεοχαρίδης, ΗΜΥ, 2017 Creation of Database Sample transitions at random (by picking 4 holds at random within robot’s limb span) Label each transition – feasible or infeasible – by sampling with high time limit over 95% infeasible transitions Re-sample around feasible transitions 35-65% feasible transitions ~1 day of computation to create a database of 100,000 labeled transitions ΗΜΥ653 Δ05-6 Embedded Systems Software.151 © Θεοχαρίδης, ΗΜΥ, 2017 Training of a NN Classifier 3-layer NN, with 9 input units, 100 hidden units, and 1 output unit Training on 50,000 examples (~3 days of computation) Validation on the remaining 50,000 examples ~78% accuracy (ε = 0.22) 0.003ms average running time ΗΜΥ653 Δ05-6 Embedded Systems Software.152 © Θεοχαρίδης, ΗΜΥ, 2017 Other (Linear) Classifiers a x denotes +1 w x + b>0 f yest f(x,w,b) = sign(w x + b) denotes -1 How would you classify this data? w x + b<0 ΗΜΥ653 Δ05-6 Embedded Systems Software.153 © Θεοχαρίδης, ΗΜΥ, 2017 Linear Classifiers a x denotes +1 f yest f(x,w,b) = sign(w x + b) denotes -1 How would you classify this data? ΗΜΥ653 Δ05-6 Embedded Systems Software.154 © Θεοχαρίδης, ΗΜΥ, 2017 a Linear Classifiers x denotes +1 f yest f(x,w,b) = sign(w x + b) denotes -1 How would you classify this data? ΗΜΥ653 Δ05-6 Embedded Systems Software.155 © Θεοχαρίδης, ΗΜΥ, 2017 Linear Classifiers a x denotes +1 f yest f(x,w,b) = sign(w x + b) denotes -1 Any of these would be fine.. ..but which is best? ΗΜΥ653 Δ05-6 Embedded Systems Software.156 © Θεοχαρίδης, ΗΜΥ, 2017 a Linear Classifiers x denotes +1 f yest f(x,w,b) = sign(w x + b) denotes -1 How would you classify this data? Misclassified to +1 class ΗΜΥ653 Δ05-6 Embedded Systems Software.157 © Θεοχαρίδης, ΗΜΥ, 2017 a Classifier Margin x denotes +1 denotes -1 ΗΜΥ653 Δ05-6 Embedded Systems Software.158 f yest f(x,w,b) = sign(w x + b) Define the margin of a linear classifier as the width that the boundary could be increased by before hitting a datapoint. © Θεοχαρίδης, ΗΜΥ, 2017 a Maximum Margin x denotes +1 denotes -1 f 1. Maximizing the margin is good 2. Implies that only support vectors f(x,w,b) = sign(w x + b)are important; other training examples are ignorable. The maximum margin linear 3. Empirically it works very very well. classifier is the linear classifier with the, um, maximum margin. Support Vectors are those datapoints that the margin pushes up against Linear SVM ΗΜΥ653 Δ05-6 Embedded Systems Software.159 yest This is the simplest kind of SVM (Called an LSVM) © Θεοχαρίδης, ΗΜΥ, 2017 Support Vector Machines SVMs pick best separating hyperplane according to some criterion e.g. maximum margin Training process is an optimisation Training set is effectively reduced to a relatively small number of support vectors ΗΜΥ653 Δ05-6 Embedded Systems Software.160 © Θεοχαρίδης, ΗΜΥ, 2017 SVM applications SVMs were originally proposed by Boser, Guyon and Vapnik in 1992 and gained increasing popularity in late 1990s. SVMs are currently among the best performers for a number of classification tasks ranging from text to genomic data. SVM techniques have been extended to a number of tasks such as regression [Vapnik et al. ’97], principal component analysis [Schölkopf et al. ’99], etc. Most popular optimization algorithms for SVMs are SMO [Platt ’99] and SVMlight [Joachims’ 99], both use decomposition to hill-climb over a subset of αi’s at a time. Tuning SVMs remains a black art: selecting a specific kernel and parameters is usually done in a try-and-see manner. ΗΜΥ653 Δ05-6 Embedded Systems Software.161 © Θεοχαρίδης, ΗΜΥ, 2017 Feature Spaces We may separate data by mapping to a higherdimensional feature space The feature space may even have an infinite number of dimensions! We need not explicitly construct the new feature space ΗΜΥ653 Δ05-6 Embedded Systems Software.162 © Θεοχαρίδης, ΗΜΥ, 2017 Kernels We may use Kernel functions to implicitly map to a new feature space Kernel fn: K x1 , x2 R Kernel must be equivalent to an inner product in some feature space ΗΜΥ653 Δ05-6 Embedded Systems Software.163 © Θεοχαρίδης, ΗΜΥ, 2017 Example Kernels xz Linear: P x z Polynomial: Gaussian: ΗΜΥ653 Δ05-6 Embedded Systems Software.164 exp x z / 2 2 © Θεοχαρίδης, ΗΜΥ, 2017 Perceptron Revisited: Linear Separators Binary classification can be viewed as the task of separating classes in feature space: wTx + b = 0 wTx + b > 0 wTx + b < 0 f(x) = sign(wTx + b) ΗΜΥ653 Δ05-6 Embedded Systems Software.165 © Θεοχαρίδης, ΗΜΥ, 2017 Which of the linear separators is optimal? ΗΜΥ653 Δ05-6 Embedded Systems Software.166 © Θεοχαρίδης, ΗΜΥ, 2017 Best Linear Separator? ΗΜΥ653 Δ05-6 Embedded Systems Software.167 © Θεοχαρίδης, ΗΜΥ, 2017 Best Linear Separator? ΗΜΥ653 Δ05-6 Embedded Systems Software.168 © Θεοχαρίδης, ΗΜΥ, 2017 Best Linear Separator? ΗΜΥ653 Δ05-6 Embedded Systems Software.169 © Θεοχαρίδης, ΗΜΥ, 2017 Best Linear Separator? ΗΜΥ653 Δ05-6 Embedded Systems Software.170 © Θεοχαρίδης, ΗΜΥ, 2017 Find Closest Points in Convex Hulls d c ΗΜΥ653 Δ05-6 Embedded Systems Software.171 © Θεοχαρίδης, ΗΜΥ, 2017 Plane Bisect Closest Points wT x + b =0 w=d-c d c ΗΜΥ653 Δ05-6 Embedded Systems Software.172 © Θεοχαρίδης, ΗΜΥ, 2017 Classification Margin wT x b separator ris w Distance from example data to the Data closest to the hyperplane are support vectors. Margin ρ of the separator is the width of separation between ρ classes. r ΗΜΥ653 Δ05-6 Embedded Systems Software.173 © Θεοχαρίδης, ΗΜΥ, 2017 Maximum Margin Classification Maximizing the margin is good according to intuition and theory. Implies that only support vectors are important; other training examples are ignorable. ΗΜΥ653 Δ05-6 Embedded Systems Software.174 © Θεοχαρίδης, ΗΜΥ, 2017 Margins and Complexity Skinny margin is more flexible thus more complex. ΗΜΥ653 Δ05-6 Embedded Systems Software.175 © Θεοχαρίδης, ΗΜΥ, 2017 Margins and Complexity Fat margin is less complex. ΗΜΥ653 Δ05-6 Embedded Systems Software.176 © Θεοχαρίδης, ΗΜΥ, 2017 Nonlinear SVM - Overview SVM locates a separating hyperplane in the feature space and classify points in that space It does not need to represent the space explicitly, simply by defining a kernel function The kernel function plays the role of the dot product in the feature space. ΗΜΥ653 Δ05-6 Embedded Systems Software.177 © Θεοχαρίδης, ΗΜΥ, 2017 Properties of SVM Flexibility in choosing a similarity function Sparseness of solution when dealing with large data sets - only support vectors are used to specify the separating hyperplane Ability to handle large feature spaces - complexity does not depend on the dimensionality of the feature space Overfitting can be controlled by soft margin approach Nice math property: a simple convex optimization problem which is guaranteed to converge to a single global solution Feature Selection ΗΜΥ653 Δ05-6 Embedded Systems Software.178 © Θεοχαρίδης, ΗΜΥ, 2017 SVM Applications SVM has been used successfully in many real-world problems - text (and hypertext) categorization - image classification - bioinformatics (Protein classification, Cancer classification) - hand-written character recognition ΗΜΥ653 Δ05-6 Embedded Systems Software.179 © Θεοχαρίδης, ΗΜΥ, 2017 Additional Resources An excellent tutorial on VC-dimension and Support Vector Machines: C.J.C. Burges. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2):955974, 1998. The VC/SRM/SVM Bible: Statistical Learning Theory by Vladimir Vapnik, WileyInterscience; 1998 http://www.kernel-machines.org/ ΗΜΥ653 Δ05-6 Embedded Systems Software.180 © Θεοχαρίδης, ΗΜΥ, 2017 Application Knowledge Embedded System Hardware 2: Specification 3: ES-hardware 4: system software (RTOS, middleware, …) ΗΜΥ653 Δ05-6 Embedded Systems Software.181 Design repository 6: Application mapping Design 8: Test 7: Optimization 5: Validation & Evaluation (energy, cost, performance, …) © Θεοχαρίδης, ΗΜΥ, 2017 Overview Application Space OPERATING SYSTEM O/S Tasks Architectural Space PROCESSOR ΗΜΥ653 Δ05-6 Embedded Systems Software.182 CUSTOM H/W I/O MEMORY © Θεοχαρίδης, ΗΜΥ, 2017 Ιs programming for an embedded system different? Speed Size Correctness Portability Real-Time Other or not Hardware Considerations ΗΜΥ653 Δ05-6 Embedded Systems Software.183 © Θεοχαρίδης, ΗΜΥ, 2017 How does one actually program an embedded system? Has this changed over the years? Has there been significant progress? How does Ford program their Fuel Injection System computer? How does GE program their Microwave Ovens (and was this different 30 years ago?) How does Nikon program their digital SLR cameras? How does Apple/RIM/Microsoft program their iPhone/Blackberry/SmartPhone? How does a hobbyist program a Pic microprocessor? How does NASA program (and reprogram) the Mars Rover? How did James Gosling at Sun want his set-top boxes programmed? (How was the original 128K Macintosh programmed?) ΗΜΥ653 Δ05-6 Embedded Systems Software.184 © Θεοχαρίδης, ΗΜΥ, 2017 Programming Languages Machine Code Fixed Rom, Ram, Firmware Assembly Code Higher Level Languages Compiled C, nesC, C++, Ada, Forth, etc. Interpreted (?) Perl, Python, Javascript Java ΗΜΥ653 Δ05-6 Embedded Systems Software.185 Markup HTML, XML All Eventually End up as Machine Code © Θεοχαρίδης, ΗΜΥ, 2017 So you’ve chosen C. Let’s Go! (Not so fast.) What does a programmer need to know about programming for an embedded system? • S/he’d better know about the hardware. • Purpose • How data flows (to include getting the program onto the system, I/O), how to interface with sensors, actuators, etc. • Whether there is an operating system, and how it runs programs • Limitations: memory, speed, upgradability (firmware?) • How are hardware errors handled? (think Mars Rover) • Plan on debugging hardware issues… ΗΜΥ653 Δ05-6 Embedded Systems Software.186 © Θεοχαρίδης, ΗΜΥ, 2017 So you’ve chosen C. Let’s Go! (Not so fast.) What does a programmer need to know about programming for an embedded system? • S/he’d better know about the software tools related to programming for the specific hardware. • Is there a compiler/linker/assembler? (hopefully!) • How is the memory addressed, what bit-level knowledge of the hardware is necessary, how does one access pins, etc.? • How will debugging be managed? • How will testing be done? • What licensing needs to be organized (this can be tricky!) • Does the software need to be portable? (using C is probably going to help you—see the second point above). ΗΜΥ653 Δ05-6 Embedded Systems Software.187 © Θεοχαρίδης, ΗΜΥ, 2017 The Embedded Software Development Process Barr, M. & Massa, A. Oram, A. (ed.) Programming Embedded Systems in C and C++, 2nd Edition. O'Reilly & Associates, Inc., 2006 , p.55 ΗΜΥ653 Δ05-6 Embedded Systems Software.188 © Θεοχαρίδης, ΗΜΥ, 2017 The Tools Compiler: Translates human readable code into assembly language or opcodes for a particular processor (or possibly into machine-independent opcodes a la Java). Produces an object file. Assembler: Translates assembly language into opcodes (it is really a compiler, too). Also produces an object file. Linker: Organizes the object files, necessary libraries, and other data and produces a relocatable file. Locator: Takes the relocatable file and information about the memory of the system and produces an executable. (By the way: gcc takes care of all of these functions at once) ΗΜΥ653 Δ05-6 Embedded Systems Software.189 © Θεοχαρίδης, ΗΜΥ, 2017 The Tools: Embedded System Specifics All of the tools run on the host computer, not the embedded computer. Compiler: Has to know about the specific hardware (except in very trivial cases). Should be able to optimize for size. Assembler: Produces “startup code”; not inserted automatically as in general purpose computers (i.e., the programmer needs to compile it independently). Linker: Needs the correct libraries (open source c libraries, such as newlib, are available). Locator: Needs programmer input for information about memory. Bottom Line: There can be a lot of extra work for the programmer, although certain systems (e.g. Pic programming) tools can automate most of it. ΗΜΥ653 Δ05-6 Embedded Systems Software.190 © Θεοχαρίδης, ΗΜΥ, 2017 Moving the program onto the embedded system Remember, the program is written (and possibly run in an emulator) on a host computer, but it still needs to get onto the embedded system. Methods: Build/burn the program into the hardware (firmware or other flash memory) Bootloader: a bootloader resides on the embedded system and facilitates loading programs onto the system. Debug Monitor: The debug monitor is a more robust program on an embedded system that helps with debugging and other chores, and can include a bootloader as well. ΗΜΥ653 Δ05-6 Embedded Systems Software.191 © Θεοχαρίδης, ΗΜΥ, 2017 Debugging Debugging embedded systems can be facilitated with a Debug Monitor, or through a remote debugger on the host computer. A serial link is normally set up, and the debugger acts more or less like a general purpose debugger. Emulators can be used to test the system without utilizing the actual hardware (but this has many caveats, and nothing beats testing on the real system). Software Simulators allow the programmer to debug completely on the host system, which can be quicker and can allow faster code turnaround. When it comes down to it, an oscilloscope and a multimeter can be your best friend for debugging. ΗΜΥ653 Δ05-6 Embedded Systems Software.192 © Θεοχαρίδης, ΗΜΥ, 2017 Reuse of standard software components Knowledge from previous designs to be made available in the form of intellectual property (IP, for SW & HW). Operating systems Middleware …. ΗΜΥ653 Δ05-6 Embedded Systems Software.193 © Θεοχαρίδης, ΗΜΥ, 2017 Embedded operating systems - Requirement: Configurability - Configurability No single RTOS will fit all needs, no overhead for unused functions tolerated configurability needed. simplest form: remove unused functions (by linker ?). Conditional compilation (using #if and #ifdef commands). Dynamic data might be replaced by static data. Advanced compile-time evaluation useful. Object-orientation could lead to a derivation subclasses. ΗΜΥ653 Δ05-6 Embedded Systems Software.194 © Θεοχαρίδης, ΗΜΥ, 2017 Verification of derived OS? Verification a potential problem of systems with a large number of derived OSs: Each derived OS must be tested thoroughly; potential problem for eCos (open source RTOS from Red Hat), including 100 to 200 configuration points [Takada, 01]. ΗΜΥ653 Δ05-6 Embedded Systems Software.195 © Θεοχαρίδης, ΗΜΥ, 2017 Requirement: Disc and network handled by tasks Disc & network handled by tasks instead of integrated drivers. Relatively slow discs & networks can be handled by tasks. Many ES without disc, a keyboard, a screen or a mouse. Effectively no device that needs to be supported by all versions of the OS, except maybe the system timer. Embedded OS Standard OS kernel ΗΜΥ653 Δ05-6 Embedded Systems Software.196 © Θεοχαρίδης, ΗΜΥ, 2017 - Requirement: Protection is optional Protection mechanisms not always necessary: ES typically designed for a single purpose, untested programs rarely loaded, SW considered reliable. (However, protection mechanisms may be needed for safety and security reasons). Privileged I/O instructions not necessary and tasks can do their own I/O. Example: Let switch be the address of some switch Simply use load register,switch instead of OS call. ΗΜΥ653 Δ05-6 Embedded Systems Software.197 © Θεοχαρίδης, ΗΜΥ, 2017 - Requirement: Interrupts not restricted to OS Interrupts can be employed by any process For standard OS: serious source of unreliability. Since embedded programs can be considered to be tested, since protection is not necessary and since efficient control over a variety of devices is required, it is possible to let interrupts directly start or stop tasks (by storing the tasks start address in the interrupt table). More efficient than going through OS services. Reduced composability: if a task is connected to an interrupt, it may be difficult to add another task which also needs to be started by an event. ΗΜΥ653 Δ05-6 Embedded Systems Software.198 © Θεοχαρίδης, ΗΜΥ, 2017 - Requirement: Real-time capability Many embedded systems are real-time (RT) systems and, hence, the OS used in these systems must be real-time operating systems (RTOSes). ΗΜΥ653 Δ05-6 Embedded Systems Software.199 © Θεοχαρίδης, ΗΜΥ, 2017 - Real-time OS (1) Def.: (A) real-time operating system is an operating system that supports the construction of real-time systems The following are the three key requirements 1. The timing behavior of the OS must be predictable. services of the OS: Upper bound on the execution time! RTOSs must be deterministic: unlike standard Java, short times during which interrupts are disabled, contiguous files to avoid unpredictable head movements. [Takada, 2001] ΗΜΥ653 Δ05-6 Embedded Systems Software.200 © Θεοχαρίδης, ΗΜΥ, 2017 - Real-time OS (2) 2. OS must manage the timing and scheduling OS possibly has to be aware of task deadlines; (unless scheduling is done off-line). OS must provide precise time services with high resolution. [Takada, 2001] ΗΜΥ653 Δ05-6 Embedded Systems Software.201 © Θεοχαρίδης, ΗΜΥ, 2017 Time services Time plays a central role in “real-time” systems. Actual time is described by real numbers. Two discrete standards are used in real-time equipment: International atomic time TAI (french: temps atomic internationale) Free of any artificial artifacts. Universal Time Coordinated (UTC) UTC is defined by astronomical standards UTC and TAI identical on Jan. 1st, 1958. 30 seconds had to be added since then. Not without problems: New Year may start twice per night. ΗΜΥ653 Δ05-6 Embedded Systems Software.202 © Θεοχαρίδης, ΗΜΥ, 2017 Internal synchronization 1. 2. 3. Synchronization with one master clock Typically used in startup-phases Distributed synchronization: Collect information from neighbors Compute correction value Set correction value. Precision of step 1 depends on how information is collected: Application level: ~500 µs to 5 ms Operation system kernel: 10 µs to 100 µs Communication hardware: < 10 µs ΗΜΥ653 Δ05-6 Embedded Systems Software.203 © Θεοχαρίδης, ΗΜΥ, 2017 External synchronization External synchronization guarantees consistency with actual physical time. Trend is to use GPS for ext. synchronization GPS offers TAI and UTC time information. Resolution is about 100 ns. GPS mouse © Dell ΗΜΥ653 Δ05-6 Embedded Systems Software.204 © Θεοχαρίδης, ΗΜΥ, 2017 Problems with external synchronization Problematic from the perspective of fault tolerance: Erroneous values are copied to all stations. Consequence: Accepting only small changes to local time. Many time formats too restricted; e.g.: NTP protocol includes only years up to 2036 For time services and global synchronization of clocks synchronization see Kopetz, 1997. ΗΜΥ653 Δ05-6 Embedded Systems Software.205 © Θεοχαρίδης, ΗΜΥ, 2017 - Real-time OS (3) 3. The OS must be fast Practically important. [Takada, 2001] ΗΜΥ653 Δ05-6 Embedded Systems Software.206 © Θεοχαρίδης, ΗΜΥ, 2017 RTOS-Kernels Distinction between real-time kernels and modified kernels of standard OSes. Distinction between general RTOSes and RTOSes for specific domains, standard APIs (e.g. POSIX RT-Extension of Unix, ITRON, OSEK) or proprietary APIs. ΗΜΥ653 Δ05-6 Embedded Systems Software.207 © Θεοχαρίδης, ΗΜΥ, 2017 Functionality of RTOS-Kernels Includes processor management, memory management, resource management and timer management; task management (resume, wait etc), inter-task communication and synchronization. ΗΜΥ653 Δ05-6 Embedded Systems Software.208 © Θεοχαρίδης, ΗΜΥ, 2017 Requirements for RTOS Predictability of timing The timing behavior of the OS must be predictable For all services of the OS, there is an upper bound on the execution time Scheduling policy must be deterministic The period during which interrupts are disabled must be short (to avoid unpredictable delays in the processing of critical events) 209 ΗΜΥ653 Δ05-6 Embedded Systems Software.209 © Θεοχαρίδης, ΗΜΥ, 2017 Requirements for RTOS OS should manage timing and scheduling OS possibly has to be aware of task deadlines; (unless scheduling is done off-line). Frequently, the OS should provide precise time services with high resolution. - Important if internal processing of the embedded system is linked to an absolute time in the physical environment Speed: The OS must be fast 210 ΗΜΥ653 Δ05-6 Embedded Systems Software.210 © Θεοχαρίδης, ΗΜΥ, 2017 Functionality of RTOS Kernel Processor management Memory management Timer management Task management (resume, wait etc) Inter-task communication and synchronization resource management 211 ΗΜΥ653 Δ05-6 Embedded Systems Software.211 © Θεοχαρίδης, ΗΜΥ, 2017 Why Use an RTOS? Can use drivers that are available with an RTOS Can focus on developing application code, not on creating or maintaining a scheduling system Multi-thread support with synchronization Portability of application code to other CPUs Resource handling by RTOS Add new features without affecting higher priority functions Support for upper layer protocols such as: TCP/IP, USB, Flash Systems, Web Servers, CAN protocols, Embedded GUI, SSL, SNMP 212 ΗΜΥ653 Δ05-6 Embedded Systems Software.212 © Θεοχαρίδης, ΗΜΥ, 2017 Classification of RTOS RT kernels vs modified kernels of standard OS Fast proprietary kernels: may be inadequate for complex systems, because they are designed to be fast rather than to be predictable in every respect, e.g., QNX, PDOS, VCOS, VTRX32, VxWORKS RT extensions to standard OS: RT-kernel runs all RT-tasks and standard-OS executed as one task on it General RTOS vs RTOS for specific domains Standard APIs vs proprietary APIs e.g. POSIX RT-Extension of Unix, ITRON, OSEK) 213 Source: R. Gupta, UCSD ΗΜΥ653 Δ05-6 Embedded Systems Software.213 © Θεοχαρίδης, ΗΜΥ, 2017 Classes of RTOSes according to R. Gupta 1. Fast proprietary kernels Fast proprietary kernels For complex systems, these kernels are inadequate, because they are designed to be fast, rather than to be predictable in every respect [R. Gupta, UCI/UCSD] Examples include QNX, PDOS, VCOS, VTRX32, VxWORKS. ΗΜΥ653 Δ05-6 Embedded Systems Software.214 © Θεοχαρίδης, ΗΜΥ, 2017 Classes of RTOSes according to R. Gupta 2. Real-time extensions to standard OSs Real-time extensions to standard OSes: Attempt to exploit comfortable main stream OSes. RT-kernel running all RT-tasks. Standard-OS executed as one task. + Crash of standard-OS does not affect RT-tasks; - RT-tasks cannot use Standard-OS services; less comfortable than expected ΗΜΥ653 Δ05-6 Embedded Systems Software.215 © Θεοχαρίδης, ΗΜΥ, 2017 Example: RT-Linux Init Bash Mozilla scheduler Linux-Kernel RT-tasks cannot use standard OS calls. Commercially available from fsmlabs (www.fsmlabs.com) RT-Task RT-Task driver interrupts RT-Linux RT-Scheduler I/O interrupts interrupts Hardware ΗΜΥ653 Δ05-6 Embedded Systems Software.216 © Θεοχαρίδης, ΗΜΥ, 2017 Example: Posix 1.b RT-extensions to Linux Standard scheduler can be replaced by POSIX scheduler implementing priorities for RT tasks RT-Task RT-Task Init Bash POSIX 1.b scheduler Linux-Kernel Mozilla Special RT-calls and standard OS calls available. Easy programming, no guarantee for meeting deadline driver I/O, interrupts Hardware ΗΜΥ653 Δ05-6 Embedded Systems Software.217 © Θεοχαρίδης, ΗΜΥ, 2017 Evaluation (Rajesh Gupta) According to Gupta, trying to use a version of a standard OS: not the correct approach because too many basic and inappropriate underlying assumptions still exist such as optimizing for the average case (rather than the worst case), ... ignoring most if not all semantic information, and independent CPU scheduling and resource allocation. Dependences between tasks not frequent for most applications of std. OSs & therefore frequently ignored. Situation different for ES since dependences between tasks are quite common. ΗΜΥ653 Δ05-6 Embedded Systems Software.218 © Θεοχαρίδης, ΗΜΥ, 2017 Classes of RTOSes according to R. Gupta 3. Research systems trying to avoid limitations Research systems trying to avoid limitations. Include MARS, Spring, MARUTI, Arts, Hartos, DARK, and Melody Research issues [Takada, 2001]: low overhead memory protection, temporal protection of computing resources RTOSes for on-chip multiprocessors support for continuous media quality of service (QoS) control. Competition between Market traditional vendors (e.g. Wind River Systems) and Embedded Windows XP and Windows CE ΗΜΥ653 Δ05-6 Embedded Systems Software.219 © Θεοχαρίδης, ΗΜΥ, 2017 Virtual machines Emulate several processors on a single real processor Running - As Single process (Java virtual machine) - On bare hardware - Allows several operating systems to be executed on top - Very good shielding between applications Temporal behavior ΗΜΥ653 Δ05-6 Embedded Systems Software.220 © Θεοχαρίδης, ΗΜΥ, 2017 Resource access protocols Critical sections: sections of code at which exclusive access to some resource must be guaranteed. Can be guaranteed with semaphores S or “mutexes”. Task 1 P(S) V(S) P(S) checks semaphore to see if resource is available and if yes, sets S to “used“. Uninterruptible operations! If no, calling task has to wait. Task 2 Mutually exclusive access to resource guarded by S ΗΜΥ653 Δ05-6 Embedded Systems Software.221 P(S) V(S): sets S to “unused“ and starts sleeping task (if any). V(S) © Θεοχαρίδης, ΗΜΥ, 2017 Blocking due to mutual exclusion Priority T1 assumed to be > than priority of T2. If T2 requests exclusive access first (at t0), T1 has to wait until T2 releases the resource (time t3), thus inverting the priority: In this example: blocking is bounded by length of critical section of T2. ΗΜΥ653 Δ05-6 Embedded Systems Software.222 © Θεοχαρίδης, ΗΜΥ, 2017 Blocking with >2 tasks can exceed the length of any critical section Priority of T1 > priority of T2 > priority of T3. T2 preempts T3: T2 can prevent T3 from releasing the resource. normal execution critical section Priority inversion! ΗΜΥ653 Δ05-6 Embedded Systems Software.223 © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.224 © Θεοχαρίδης, ΗΜΥ, 2017 The MARS Pathfinder problem (1) “But a few days into the mission, not long after Pathfinder started gathering meteorological data, the spacecraft began experiencing total system resets, each resulting in losses of data. The press reported these failures in terms such as "software glitches" and "the computer was trying to do too many things at once".” … http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html ΗΜΥ653 Δ05-6 Embedded Systems Software.225 © Θεοχαρίδης, ΗΜΥ, 2017 The MARS Pathfinder problem (2) “VxWorks (the OS used) provides preemptive priority scheduling of threads. Tasks on the Pathfinder spacecraft were executed as threads with priorities that were assigned in the usual manner reflecting the relative urgency of these tasks.” “Pathfinder contained an "information bus", which you can think of as a shared memory area used for passing information between different components of the spacecraft.” A bus management task ran frequently with high priority to move certain kinds of data in and out of the information bus. Access to the bus was synchronized with mutual exclusion locks (mutexes).” http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html ΗΜΥ653 Δ05-6 Embedded Systems Software.226 © Θεοχαρίδης, ΗΜΥ, 2017 The MARS Pathfinder problem (3) The meteorological data gathering task ran as an infrequent, low priority thread, … When publishing its data, it would acquire a mutex, do writes to the bus, and release the mutex. .. The spacecraft also contained a communications task that ran with medium priority.” High priority: retrieval of data from shared memory Medium priority: communications task Low priority: thread collecting meteorological data http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html ΗΜΥ653 Δ05-6 Embedded Systems Software.227 © Θεοχαρίδης, ΗΜΥ, 2017 The MARS Pathfinder problem (4) “Most of the time this combination worked fine. However, very infrequently it was possible for an interrupt to occur that caused the (medium priority) communications task to be scheduled during the short interval while the (high priority) information bus thread was blocked waiting for the (low priority) meteorological data thread. In this case, the long-running communications task, having higher priority than the meteorological task, would prevent it from running, consequently preventing the blocked information bus task from running. After some time had passed, a watchdog timer would go off, notice that the data bus task had not been executed for some time, conclude that something had gone drastically wrong, and initiate a total system reset. This scenario is a classic case of priority inversion.” http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html ΗΜΥ653 Δ05-6 Embedded Systems Software.228 © Θεοχαρίδης, ΗΜΥ, 2017 Coping with priority inversion: the priority inheritance protocol Tasks are scheduled according to their active priorities. Tasks with the same priorities are scheduled FCFS. If task T1 executes P(S) & exclusive access granted to T2: T1 will become blocked. If priority(T2) < priority(T1): T2 inherits the priority of T1. T2 resumes. Rule: tasks inherit the highest priority of tasks blocked by it. When T2 executes V(S), its priority is decreased to the highest priority of the tasks blocked by it. If no other task blocked by T2: priority(T2):= original value. Highest priority task so far blocked on S is resumed. Transitive: if T2 blocks T1 and T1 blocks T0, then T2 inherits the priority of T0. ΗΜΥ653 Δ05-6 Embedded Systems Software.229 © Θεοχαρίδης, ΗΜΥ, 2017 Example How would priority inheritance affect our example with 3 tasks? T3 inherits the priority of T1 and T3 resumes. V(S) ΗΜΥ653 Δ05-6 Embedded Systems Software.230 © Θεοχαρίδης, ΗΜΥ, 2017 P(a) V(a) P(b) P(a) P(b) ΗΜΥ653 Δ05-6 Embedded Systems Software.231 V(b) V(b) V(a) [P/V added@unido] © Θεοχαρίδης, ΗΜΥ, 2017 V(a) P(a) P(a) P(b) P(b) ΗΜΥ653 Δ05-6 Embedded Systems Software.232 V(b) V(a) V(b) [P/V added@unido] © Θεοχαρίδης, ΗΜΥ, 2017 ΗΜΥ653 Δ05-6 Embedded Systems Software.233 © Θεοχαρίδης, ΗΜΥ, 2017 Priority Inversion Problem Pathfinder mission on Mars in 1997 Used VxWorks, an RTOS kernel, from WindRiver Software problems caused the total system resets of the Pathfinder spacecraft in mission - Watchdog timer goes off, informing that something has gone dramatically wrong and initiating the system reset 234 ΗΜΥ653 Δ05-6 Embedded Systems Software.234 © Θεοχαρίδης, ΗΜΥ, 2017 Priority Inversion Problem VxWorks provides preemptive priority scheduling of threads Tasks on the Pathfinder spacecraft were executed as threads with priorities that were assigned in the usual manner reflecting the relative urgency of these tasks. Task 1 tries to get the semaphore Task 1 preempts Task3 Task 1 gets the semaphore and execute Priority Inversion Task 1 (highest priority) Task 2 (medium priority) Task 2 preempts task 3 Task 3 (lowest priority) Task 3 is resumed Task 3 gets semaphore Task 3 is resumed Time Task 3 releases the semaphore 235 ΗΜΥ653 Δ05-6 Embedded Systems Software.235 © Θεοχαρίδης, ΗΜΥ, 2017 Priority Inheritance A chain of processes could all be accessing resources that the highpriority process needs All these processes inherit the high priority until they are done with the resource When they are finished, their priority reverts to its original value Task 1 tries to get the semaphore (Priority of Task 3 is raised to Task 1’s) Task 1 preempts Task3 Priority Inversion Task 1 completes Task 1 (highest priority) Task 2 (medium priority) Task 3 (lowest priority) Task 3 gets semaphore Task 3 is resumed with the highest priority Time Task 3 releases the semaphore 236 ΗΜΥ653 Δ05-6 Embedded Systems Software.236 © Θεοχαρίδης, ΗΜΥ, 2017 Priority inversion on Mars Priority inheritance also solved the Mars Pathfinder problem: the VxWorks operating system used in the pathfinder implements a flag for the calls to mutex primitives. This flag allows priority inheritance to be set to “on”. When the software was shipped, it was set to “off”. The problem on Mars was corrected by using the debugging facilities of VxWorks to change the flag to “on”, while the Pathfinder was already on the Mars [Jones, 1997]. ΗΜΥ653 Δ05-6 Embedded Systems Software.237 © Θεοχαρίδης, ΗΜΥ, 2017 Remarks on priority inheritance protocol Possible large number of tasks with high priority. Possible deadlocks. Ongoing debate about problems with the protocol: Victor Yodaiken: Against Priority Inheritance, Sept. 2004, http://www.fsmlabs.com/resources/white_papers/priority-inheritance/ Finds application in ADA: During rendez-vous, task priority is set to the maximum. Protocol for fixed set of tasks: priority ceiling protocol. ΗΜΥ653 Δ05-6 Embedded Systems Software.238 © Θεοχαρίδης, ΗΜΥ, 2017 Summary General requirements for embedded operating systems - Configurability - I/O - Interrupts General properties of real-time operating systems - Predictability - Time services, - Synchronization - Classes of RTOSs, - Device driver embedding Priority inversion - The problem - Priority inheritance ΗΜΥ653 Δ05-6 Embedded Systems Software.239 © Θεοχαρίδης, ΗΜΥ, 2017 Complexity trends in OS OS functionality drives the complexity Small controllers sensors Home appliances Mobile phones PDAs Game Machines Router 240 ΗΜΥ653 Δ05-6 Embedded Systems Software.240 © Θεοχαρίδης, ΗΜΥ, 2017 Requirements of EOS Memory Resident: size is important consideration Data structure optimized Kernel optimized and usually in assembly language Support of signaling & interrupts Real-time scheduling tight-coupled scheduler and interrupts Power Management capabilities Power aware schedule Control of non-processor resources 241 ΗΜΥ653 Δ05-6 Embedded Systems Software.241 © Θεοχαρίδης, ΗΜΥ, 2017 Embedded OS Design approach Traditional OS: Monolithic or Distribute Embedded: Layered is the key (Constantine D. P, UIUC 2000) Basic Loader Power Management Interrupt & signalling Real-Time Scheduler Memory Management Networking support Custom device support 242 ΗΜΥ653 Δ05-6 Embedded Systems Software.242 © Θεοχαρίδης, ΗΜΥ, 2017 Power Management by OS Static Approach: Rely on pre-set parameters Example: switch off the power to devices that are not in use for a while (pre-calculated number of cycles). Used in laptops now. Dynamic Approach: Based on dynamic condition of workloads and per specification of power optimization guidelines Example: Restrict multitasking progressively, reduce context switching, even avoid cache/memory access, etc.. 243 ΗΜΥ653 Δ05-6 Embedded Systems Software.243 © Θεοχαρίδης, ΗΜΥ, 2017 H/W-S/W Architecture A significant part of the problem is deciding which parts should be in s/w on programmable processors, and which in specialized h/w Today: Ad hoc approaches based on earlier experience with similar products, & on manual design H/W-S/W partitioning decided at the beginning, and then designs proceed separately ΗΜΥ653 Δ05-6 Embedded Systems Software.244 © Θεοχαρίδης, ΗΜΥ, 2017 Embedded System Design CAD tools take care of h/w fairly well Although a productivity gap emerging But, S/W is a different story… HLLs such as C help, but can’t cope with complexity and performance constraints Holy Grail for Tools People: H/W-like synthesis & verification from a behavior description of the whole system at a high level of abstraction using formal computation models ΗΜΥ653 Δ05-6 Embedded Systems Software.245 © Θεοχαρίδης, ΗΜΥ, 2017 Embedded System Design from a Design Technology Perspective Intertwined subtasks Specification/modeling H/W ASIC Processor Analog I/O Memory & S/W partitioning Scheduling & resource allocations H/W & S/W implementation Verification DSP Code ΗΜΥ653 Δ05-6 Embedded Systems Software.246 & debugging Crucial is the co-design and joint optimization of hardware and software © Θεοχαρίδης, ΗΜΥ, 2017 Embedded System Design Flow Environ -ment ASIC Analog I/O Processor Memory DSP Code ΗΜΥ653 Δ05-6 Embedded Systems Software.247 Modeling the system to be designed, and experimenting with algorithms involved; Refining (or “partitioning”) the function to be implemented into smaller, interacting pieces; HW-SW partitioning: Allocating elements in the refined model to either (1) HW units, or (2) SW running on custom hardware or a suitable programmable processor. Scheduling the times at which the functions are executed. This is important when several modules in the partition share a single hardware unit. Mapping (Implementing) a functional description into (1) software that runs on a processor or (2) a collection of custom, semicustom, or commodity HW. © Θεοχαρίδης, ΗΜΥ, 2017 Microsoft Mobility Platforms Pocket PC Smart Personal Objects • One-way network • Information consumption Windows CE • Information consumption • View and some data entry Smartphone • Integrated PDA with • Information phone consumption • Interoperability with • Primarily data Office, Exchange viewing • Integrated phone and SQL Server • .NET Compact with PDA Framework • Interoperability • ASP.NET mobile with Exchange controls • .NET Compact Framework • ASP.NET mobile controls Tablet PC Notebook PC • Keyboard and mouse input methods • Keyboard centric at the desk, pen & keyboard away from the desk • Keyboard, mouse plus pen, ink, and speech input methods • Full .NET framework preinstalled • Full .NET framework available • Pen, ink, handwriting and speech recognition API’s • Complex document authoring, editing and reading • Keyboard centric at the desk Windows Mobile ΗΜΥ653 Δ05-6 Embedded Systems Software.248 • Complex document authoring, editing and active reading • Note taking and ink annotating Increased Functionality Windows XP/XPE © Θεοχαρίδης, ΗΜΥ, 2017 Windows Embedded Family ΗΜΥ653 Δ05-6 Embedded Systems Software.249 © Θεοχαρίδης, ΗΜΥ, 2017 Embedded Linux in various devices: ΗΜΥ653 Δ05-6 Embedded Systems Software.250 NASA personal assistant © Θεοχαρίδης, ΗΜΥ, 2017 Thread vs. Process http://www.lynuxworks.com/products/posix/processes.php3 ΗΜΥ653 Δ05-6 Embedded Systems Software.251 © Θεοχαρίδης, ΗΜΥ, 2017 Thread vs. Process http://www.lynuxworks.com/products/posix/processes.php3 ΗΜΥ653 Δ05-6 Embedded Systems Software.252 © Θεοχαρίδης, ΗΜΥ, 2017 Thread vs. Process http://www.lynuxworks.com/products/posix/processes.php3 ΗΜΥ653 Δ05-6 Embedded Systems Software.253 © Θεοχαρίδης, ΗΜΥ, 2017 Developing Process From system design t e s t Linux OS select OS Porting and improvement Driver and Application software development ΗΜΥ653 Δ05-6 Embedded Systems Software.254 Rehat,bluecat,RT Linux,Monta Vista Linux,RTAI,… http://linux.org http://www.gn u.org… Tekram,HP,Intel, … © Θεοχαρίδης, ΗΜΥ, 2017 Windows embedded VS Cont. Linux Embedded Both on strong uptake curve! Oses targeted in current and next embedded projects,2002,data from EDC ΗΜΥ653 Δ05-6 Embedded Systems Software.255 © Θεοχαρίδης, ΗΜΥ, 2017 Application Knowledge Embedded System Software 2: Specification 3: ES-hardware 4: system software (RTOS, middleware, …) ΗΜΥ653 Δ05-6 Embedded Systems Software.256 Design repository 6: Application mapping Design 8: Test 7: Optimization 5: Validation & Evaluation (energy, cost, performance, …) © Θεοχαρίδης, ΗΜΥ, 2017 Mapping of Applications to Platforms © Renesas, Thiele ΗΜΥ653 Δ05-6 Embedded Systems Software.257 © Θεοχαρίδης, ΗΜΥ, 2017 Distinction between mapping problems ΗΜΥ653 Δ05-6 Embedded Systems Software.258 © Θεοχαρίδης, ΗΜΥ, 2017 Problem Description Given A set of applications Scenarios on how these applications will be used A set of candidate architectures comprising - (Possibly heterogeneous) processors - (Possibly heterogeneous) communication architectures - Possible scheduling policies Find A mapping of applications to processors Appropriate scheduling techniques (if not fixed) A target architecture (if DSE is included) Objectives Keeping deadlines and/or maximizing performance Minimizing cost, energy consumption ΗΜΥ653 Δ05-6 Embedded Systems Software.259 © Θεοχαρίδης, ΗΜΥ, 2017 Related Work Mapping to EXUs in automotive design Scheduling theory: Provides insight for the mapping task start times Hardware/software partitioning: Can be applied if it supports multiple processors High performance computing (HPC) Automatic parallelization, but only for - single applications, - fixed architectures, - no support for scheduling, - memory and communication model usually different High-level synthesis Provides useful terms like scheduling, allocation, assignment Optimization theory ΗΜΥ653 Δ05-6 Embedded Systems Software.260 © Θεοχαρίδης, ΗΜΥ, 2017 Scope of mapping algorithms Useful terms from hardware synthesis: Resource Allocation Decision concerning type and number of available resources Resource Assignment Mapping: Task (Hardware) Resource xx to yy binding: Describes a mapping from behavioral to structural domain, e.g. task to processor binding, variable to memory binding Scheduling Mapping: Tasks Task start times Sometimes, resource assignment is considered being included in scheduling. ΗΜΥ653 Δ05-6 Embedded Systems Software.261 © Θεοχαρίδης, ΗΜΥ, 2017 Classes of mapping algorithms (considered in this course) Classical scheduling algorithms Mostly for independent tasks & ignoring communication, mostly for mono- and homogeneous multiprocessors Hardware/software partitioning Dependent tasks, heterogeneous systems, focus on resource assignment Dependent tasks as considered in architectural synthesis Initially designed in different context, but applicable Design space exploration using genetic algorithms Heterogeneous systems, incl. communication modeling ΗΜΥ653 Δ05-6 Embedded Systems Software.262 © Θεοχαρίδης, ΗΜΥ, 2017 Real-time scheduling Assume that we are given a task graph G=(V,E). Def.: A schedule s of G is a mapping VT of a set of tasks V to start times from domain T. G=(V,E) V1 V2 V3 V4 s T t Typically, schedules have to respect a number of constraints, incl. resource constraints, dependency constraints, deadlines. Scheduling = finding such a mapping. ΗΜΥ653 Δ05-6 Embedded Systems Software.263 © Θεοχαρίδης, ΗΜΥ, 2017 Hard and soft deadlines Def.: A time-constraint (deadline) is called hard if not meeting that constraint could result in a catastrophe [Kopetz, 1997]. All other time constraints are called soft. We will focus on hard deadlines. ΗΜΥ653 Δ05-6 Embedded Systems Software.264 © Θεοχαρίδης, ΗΜΥ, 2017 Periodic and aperiodic tasks Def.: Tasks which must be executed once every p units of time are called periodic tasks. p is called their period. Each execution of a periodic task is called a job. All other tasks are called aperiodic. Def.: Tasks requesting the processor at unpredictable times are called sporadic, if there is a minimum separation between the times at which they request the processor. ΗΜΥ653 Δ05-6 Embedded Systems Software.265 © Θεοχαρίδης, ΗΜΥ, 2017 Preemptive and non-preemptive scheduling Non-preemptive schedulers: Tasks are executed until they are done. Response time for external events may be quite long. Preemptive schedulers: To be used if - some tasks have long execution times or - if the response time for external events to be short. ΗΜΥ653 Δ05-6 Embedded Systems Software.266 © Θεοχαρίδης, ΗΜΥ, 2017 Centralized and distributed scheduling Centralized and distributed scheduling: Multiprocessor scheduling either locally on 1 or on several processors. Mono- and multi-processor scheduling: - Simple scheduling algorithms handle single processors, - more complex algorithms handle multiple processors. – algorithms for homogeneous multi-processor systems – algorithms for heterogeneous multi-processor systems (includes HW accelerators as special case). ΗΜΥ653 Δ05-6 Embedded Systems Software.267 © Θεοχαρίδης, ΗΜΥ, 2017 Dynamic/online scheduling Dynamic/online scheduling: Processor allocation decisions (scheduling) at run-time; based on the information about the tasks arrived so far. ΗΜΥ653 Δ05-6 Embedded Systems Software.268 © Θεοχαρίδης, ΗΜΥ, 2017 Static/offline scheduling Static/offline scheduling: Scheduling taking a priori knowledge about arrival times, execution times, and deadlines into account. Dispatcher allocates processor when interrupted by timer. Timer controlled by a table generated at design time. ΗΜΥ653 Δ05-6 Embedded Systems Software.269 © Θεοχαρίδης, ΗΜΥ, 2017 Time-triggered systems (1) In an entirely time-triggered system, the temporal control structure of all tasks is established a priori by off-line support-tools. This temporal control structure is encoded in a Task-Descriptor List (TDL) that contains the cyclic schedule for all activities of the node. This schedule considers the required precedence and mutual exclusion relationships among the tasks such that an explicit coordination of the tasks by the operating system at run time is not necessary. .. The dispatcher is activated by the synchronized clock tick. It looks at the TDL, and then performs the action that has been planned for this instant [Kopetz]. ΗΜΥ653 Δ05-6 Embedded Systems Software.270 © Θεοχαρίδης, ΗΜΥ, 2017 Time-triggered systems (2) … pre-run-time scheduling is often the only practical means of providing predictability in a complex system. [Xu, Parnas]. It can be easily checked if timing constraints are met. The disadvantage is that the response to sporadic events may be poor. ΗΜΥ653 Δ05-6 Embedded Systems Software.271 © Θεοχαρίδης, ΗΜΥ, 2017 Schedulability Set of tasks is schedulable under a set of constraints, if a schedule exists for that set of tasks & constraints. Exact tests are NP-hard in many situations. Sufficient tests: sufficient conditions for schedule checked. (Hopefully) small probability of not guaranteeing a schedule even though one exists. Necessary tests: checking necessary conditions. Used to show no schedule exists. There may be cases in which no schedule exists & we cannot prove it. ΗΜΥ653 Δ05-6 Embedded Systems Software.272 sufficient schedulable necessary © Θεοχαρίδης, ΗΜΥ, 2017 Cost functions Cost function: Different algorithms aim at minimizing different functions. Def.: Maximum lateness = maxall tasks (completion time – deadline) Is <0 if all tasks complete before deadline. T1 T2 t ΗΜΥ653 Δ05-6 Embedded Systems Software.273 © Θεοχαρίδης, ΗΜΥ, 2017 Aperiodic scheduling - Scheduling with no precedence constraints Let {Ti } be a set of tasks. Let: ci be the execution time of Ti , di be the deadline interval, that is, the time between Ti becoming available and the time until which Ti has to finish execution. ℓi be the laxity or slack, defined as ℓi = di - ci fi be the finishing time. i di ci ℓi t ΗΜΥ653 Δ05-6 Embedded Systems Software.274 © Θεοχαρίδης, ΗΜΥ, 2017 Uniprocessor with equal arrival times Preemption is useless. Earliest Due Date (EDD): Execute task with earliest due date (deadline) first. fi fi fi EDD requires all tasks to be sorted by their (absolute) deadlines. Hence, its complexity is O(n log(n)). ΗΜΥ653 Δ05-6 Embedded Systems Software.275 © Θεοχαρίδης, ΗΜΥ, 2017 Optimality of EDD EDD is optimal, since it follows Jackson's rule: Given a set of n independent tasks, any algorithm that executes the tasks in order of non-decreasing (absolute) deadlines is optimal with respect to minimizing the maximum lateness. Proof (See Buttazzo, 2002): Let be a schedule produced by any algorithm A If A EDD Ta, Tb, da ≤ db, Tb immediately precedes Ta in . Let ' be the schedule obtained by exchanging Ta and Tb. ΗΜΥ653 Δ05-6 Embedded Systems Software.276 © Θεοχαρίδης, ΗΜΥ, 2017 Exchanging Ta and Tb cannot increase lateness Max. lateness for Ta and Tb in is Lmax(a,b)=fa-da Max. lateness for Ta and Tb in ' is L'max(a,b)=max(L'a,L'b) Two possible cases 1. L'a ≥ L'b: L'max(a,b) = f'a – da < fa – da = Lmax(a,b) since Ta starts earlier in schedule '. 2. L'a ≤ L'b: L'max(a,b) = f'b – db = fa – db ≤ fa – da = Lmax(a,b) since fa=f'b and da ≤ db L'max(a,b) ≤ Lmax(a,b) Tb ' Ta Ta Tb fa=f'b ΗΜΥ653 Δ05-6 Embedded Systems Software.277 © Θεοχαρίδης, ΗΜΥ, 2017 EDD is optimal Any schedule with lateness L can be transformed into an EDD schedule n with lateness Ln ≤ L, which is the minimum lateness. EDD is optimal (q.e.d.) ΗΜΥ653 Δ05-6 Embedded Systems Software.278 © Θεοχαρίδης, ΗΜΥ, 2017 Earliest Deadline First (EDF) - Horn’s Theorem - Different arrival times: Preemption potentially reduces lateness. Theorem [Horn74]: Given a set of n independent tasks with arbitrary arrival times, any algorithm that at any instant executes the task with the earliest absolute deadline among all the ready tasks is optimal with respect to minimizing the maximum lateness. ΗΜΥ653 Δ05-6 Embedded Systems Software.279 © Θεοχαρίδης, ΗΜΥ, 2017 Earliest Deadline First (EDF) - Algorithm Earliest deadline first (EDF) algorithm: Each time a new ready task arrives: It is inserted into a queue of ready tasks, sorted by their absolute deadlines. Task at head of queue is executed. If a newly arrived task is inserted at the head of the queue, the currently executing task is preempted. Straightforward approach with sorted lists (full comparison with existing tasks for each arriving task) requires run-time O(n2); (less with binary search or bucket arrays). Sorted queue Executing task ΗΜΥ653 Δ05-6 Embedded Systems Software.280 © Θεοχαρίδης, ΗΜΥ, 2017 Earliest Deadline First (EDF) - Example - Earlier deadline preemption Later deadline no preemption ΗΜΥ653 Δ05-6 Embedded Systems Software.281 © Θεοχαρίδης, ΗΜΥ, 2017 Optimality of EDF To be shown: EDF minimizes maximum lateness. Proof (Buttazzo, 2002): Let be a schedule produced by generic schedule A Let EDF: schedule produced by EDF Preemption allowed: tasks executed in disjoint time intervals divided into time slices of 1 time unit each Time slices denoted by [t, t+1) Let (t): task executing in [t, t+1) Let E(t): task which, at time t, has the earliest deadline Let tE(t): time (t) at which the next slice of task E(t) begins its execution in the current schedule ΗΜΥ653 Δ05-6 Embedded Systems Software.282 © Θεοχαρίδης, ΗΜΥ, 2017 Optimality of EDF (2) If EDF, then there exists time t: (t) E(t) Idea: swapping (t) and E(t) cannot increase max. lateness. t=4; (t)=4; E(t)=2; tE=6; (tE)=2 T1 t T2 t T3 t T4 0 2 4 6 8 10 12 14 16 (t)=2; (tE)=4 T1 t t T2 t T3 t T4 0 2 4 6 8 10 12 14 16 t If (t) starts at t=0 and D=maxi{di } then EDF can be obtained from by at most D transpositions. ΗΜΥ653 Δ05-6 Embedded Systems Software.283 [Buttazzo, © Θεοχαρίδης,2002] ΗΜΥ, 2017 Optimality of EDF (3) Algorithm interchange: { for (t=0 to D-1) { if ((t) E(t)) { (tE) = (t); (t) = E(t); }}} Using the same argument as in the proof of Jackson’s algorithm, it is easy to show that swapping cannot increase maximum lateness; hence EDF is optimal. Does interchange preserve schedulability? 1. task E(t) moved ahead: meeting deadline in new schedule if meeting deadline in 2. task (t) delayed: if (t) is feasible, then (tE+1) ≤ dE, where dE is the earliest deadline. Since dE ≤ di for any i, we have tE+1 ≤ di, which guarantees schedulability of the delayed task. [Buttazzo, 2002] ΗΜΥ653 Δ05-6 Embedded Systems Software.284 q.e.d. © Θεοχαρίδης, ΗΜΥ, 2017 Least laxity (LL), Least Slack Time First (LST) Priorities = decreasing function of the laxity (the less laxity, the higher the priority); dynamically changing priority; preemptive. ℓ ℓ ℓ ℓ ℓ ℓ ℓ ΗΜΥ653 Δ05-6 Embedded Systems Software.285 ℓ ℓ ℓ ℓ ℓ © Θεοχαρίδης, ΗΜΥ, 2017 Properties Not sufficient to call scheduler & re-compute laxity just at task arrival times. Overhead for calls of the scheduler. Many context switches. Detects missed deadlines early. LL is also an optimal scheduling for mono-processor systems. Dynamic priorities cannot be used with a fixed prio OS. LL scheduling requires the knowledge of the execution time. ΗΜΥ653 Δ05-6 Embedded Systems Software.286 © Θεοχαρίδης, ΗΜΥ, 2017 Scheduling without preemption (1) Lemma: If preemption is not allowed, optimal schedules may have to leave the processor idle at certain times. Proof: Suppose: optimal schedulers never leave processor idle. ΗΜΥ653 Δ05-6 Embedded Systems Software.287 © Θεοχαρίδης, ΗΜΥ, 2017 Scheduling without preemption (2) T1: periodic, c1 = 2, p1 = 4, d1 = 4 T2: occasionally available at times 4*n+1, c2= 1, d2= 1 T1 has to start at t=0 deadline missed, but schedule is possible (start T2 first) scheduler is not optimal contradiction! q.e.d. ΗΜΥ653 Δ05-6 Embedded Systems Software.288 © Θεοχαρίδης, ΗΜΥ, 2017 Scheduling without preemption Preemption not allowed: optimal schedules may leave processor idle to finish tasks with early deadlines arriving late. Knowledge about the future is needed for optimal scheduling algorithms No online algorithm can decide whether or not to keep idle. EDF is optimal among all scheduling algorithms not keeping the processor idle at certain times. If arrival times are known a priori, the scheduling problem becomes NP-hard in general. B&B typically used. ΗΜΥ653 Δ05-6 Embedded Systems Software.289 © Θεοχαρίδης, ΗΜΥ, 2017 Scheduling with precedence constraints Task graph and possible schedule: ΗΜΥ653 Δ05-6 Embedded Systems Software.290 © Θεοχαρίδης, ΗΜΥ, 2017 Simultaneous Arrival Times: The Latest Deadline First (LDF) Algorithm LDF [Lawler, 1973]: reads the task graph and among the tasks with no successors inserts the one with the latest deadline into a queue. It then repeats this process, putting tasks whose successor have all been selected into the queue. At run-time, the tasks are executed in the generated total order. LDF is non-preemptive and is optimal for mono-processors. If no local deadlines exist, LDF performs just a topological sort. ΗΜΥ653 Δ05-6 Embedded Systems Software.291 © Θεοχαρίδης, ΗΜΥ, 2017 Asynchronous Arrival Times: Modified EDF Algorithm This case can be handled with a modified EDF algorithm. The key idea is to transform the problem from a given set of dependent tasks into a set of independent tasks with different timing parameters [Chetto90]. This algorithm is optimal for mono-processor systems. If preemption is not allowed, the heuristic algorithm developed by Stankovic and Ramamritham can be used. ΗΜΥ653 Δ05-6 Embedded Systems Software.292 © Θεοχαρίδης, ΗΜΥ, 2017 © L. Thiele, 2006 ΗΜΥ653 Δ05-6 Embedded Systems Software.293 © Θεοχαρίδης, ΗΜΥ, 2017 Summary Definition mapping terms Resource allocation, assignment, binding, scheduling Hard vs. soft deadlines Static vs. dynamic TT-OS Schedulability Classical scheduling Aperiodic tasks - No precedences – Simultaneous (EDD) & Asynchronous Arrival Times (EDF, LL) - Precedences – Simultaneous Arrival Times ( LDF) – Asynchronous Arrival Times ( mEDF) ΗΜΥ653 Δ05-6 Embedded Systems Software.294 © Θεοχαρίδης, ΗΜΥ, 2017
