1 / 31

Introduction to Neural Networks

Introduction to Neural Networks. John Paxton Montana State University Summer 2003. Chapter 2: Simple Neural Networks for Pattern Classification. x 0. 1. w 0. w 0 is the bias f(y in ) = 1 if y in >= 0 f(y in ) = 0 otherwise ARCHITECTURE. y. w 1. x 1. w n. x n. Representations.

marlee
Download Presentation

Introduction to Neural Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Introduction to Neural Networks John Paxton Montana State University Summer 2003

  2. Chapter 2: Simple Neural Networks for Pattern Classification x0 1 w0 w0 is the bias f(yin) = 1 if yin >= 0 f(yin) = 0 otherwise ARCHITECTURE y w1 x1 wn xn

  3. Representations • Binary: 0 no, 1 yes • Bipolar: -1 no, 0 unknown, 1 yes • Bipolar is superior

  4. Interpreting the Weights • w0 = -1, w1 = 1, w2 = 1 • 0 = -1 + x1 + x2 or x2 = 1 – x1 YES x1 NO x2 decision boundary

  5. Modelling a Simple Problem • Should I attend this lecture? • x1 = it’s hot • x2 = it’s raining x0 2.5 y -2 x1 1 x2

  6. Linear Separability 1 0 1 1 0 1 0 0 1 0 1 0 AND OR XOR

  7. Hebb’s Rule • 1949. Increase the weight between two neurons that are both “on”. • 1988. Increase the weight between two neurons that are both “off”. • wi(new) = wi(old) + xi*y

  8. Algorithm 1. set wi = 0 for 0 <= i <= n 2. for each training vector 3. set xi = si for all input units 4. set y = t 5. wi(new) = wi(old) + xi*y

  9. Example: 2 input AND

  10. Training Procedure

  11. Result Interpretation • -2 + 2x1 + 2x2 = 0 OR • x2 = -x1 + 1 • This training procedure is order dependent and not guaranteed.

  12. Pattern Recognition Exercise • #.# .#..#. #.##.# .#. “X” “O”

  13. Pattern Recognition Exercise • Architecture? • Weights? • Are the original patterns classified correctly? • Are the original patterns with 1 piece of wrong data classified correctly? • Are the original patterns with 1 piece of missing data classified correctly?

  14. Perceptrons (1958) • Very important early neural network • Guaranteed training procedure under certain circumstances x0 1 w0 y w1 x1 wn xn

  15. Activation Function • f(yin) = 1 if yin > q f(yin) = 0 if -q <= yin <= q f(yin) = -1 otherwise • Graph interpretation 1 -1

  16. Learning Rule • wi(new) = wi(old) + a*t*xi if error • a is the learning rate • Typically, 0 < a <= 1

  17. Algorithm 1. set wi = 0 for 0 <= i <= n (can be random) 2. for each training exemplar do 3. xi = si 4. yin = S xi*wi 5. y = f(yin) 6. wi(new) = wi(old) + a*t*xi if error 7. if stopping condition not reached, go to 2

  18. Example: AND concept • bipolar inputs • bipolar target • q = 0 • a = 1

  19. Epoch 1

  20. Exercise • Continue the above example until the learning algorithm is finished.

  21. Perceptron Learning Rule Convergence Theorem • If a weight vector exists that correctly classifies all of the training examples, then the perceptron learning rule will converge to some weight vector that gives the correct response for all training patterns. This will happen in a finite number of steps.

  22. Exercise • Show perceptron weights for the 2-of-3 concept

  23. Adaline (Widrow, Huff 1960) • Adaptive Linear Network • Learning rule minimizes the mean squared error • Learns on all examples, not just ones with errors

  24. Architecture x0 1 w0 y w1 x1 wn xn

  25. Training Algorithm 1. set wi (small random values typical) 2. set a (0.1 typical) 3. for each training exemplar do 4. xi = si 5. yin = S xi*wi 6. wi(new) = wi(old) + a*(t – yin)*xi 7. go to 3 if largest weight change big enough

  26. Activation Function • f(yin) = 1 if yin >= 0 • f(yin) = -1 otherwise

  27. Delta Rule • squared error E = (t – yin)2 • minimize error E’ = -2(t – yin)xi = a(t – yin)xi

  28. Example: AND concept • bipolar inputs • bipolar targets • w0 = -0.5, w1 = 0.5, w2 = 0.5 • minimizes E

  29. Exercise • Demonstrate that you understand the Adaline training procedure.

  30. Madaline • Many adaptive linear neurons 1 1 y x1 z1 xm zk

  31. Madaline • MRI (1960) – only learns weights from input layer to hidden layer • MRII (1987) – learns all weights

More Related