1 / 43

Neural Network Architectures

Neural Network Architectures. Aydın Ulaş. 02 December 2004. ulasmehm @boun.edu.tr. Outline Of Presentation. Introduction Neural Networks Neural Network Architectures Conclusions. Introduction. Some numbers… The human brain contains about 10 billion nerve cells (neurons)

JasminFlorian
Download Presentation

Neural Network Architectures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Neural Network Architectures Aydın Ulaş 02December 2004 ulasmehm@boun.edu.tr

  2. Outline Of Presentation • Introduction • Neural Networks • Neural Network Architectures • Conclusions

  3. Introduction • Some numbers… • The human brain contains about 10 billion nerve cells (neurons) • Each neuron is connected to the others through 10000 synapses • Brain as a computational unit • It can learn, reorganize from experience • It adapts to the environment • It is robust and fault tolerant • Fast computations with too much individual computational units

  4. Introduction • Taking the nature as a model. Consider the neuron as a PE • A neuron has • Input (dendrites) • Output (the axon) • The information circulates from the dendrites to the axon via the cell body • Axon connects to dendrites via synapses • Strength of synapses change • Synapses may be excitatory or inhibitory

  5. Perceptron (Artificial Neuron) • Definition : Non linear, parameterized function with restricted output range

  6. Activation Functions Linear Sigmoid Hyperbolic tangent

  7. Neural Networks • A mathematical model to solve engineering problems • Group of highly connected neurons to realize compositions of non linear functions • Tasks • Classification • Clustering • Regression • According to input flow • Feed forward Neural Networks • Recurrent Neural Networks

  8. Feed Forward Neural Networks • The information is propagated from the inputs to the outputs • Time has no role (Acyclic, no feedbacks from outputs to inputs)

  9. Recurrent Networks • Arbitrary topologies • Can model systems with internal states (dynamic ones) • Delays can be modeled • More difficult to train • Problematic performance • Stable Outputs may be more difficult to evaluate • Unexpected behavior (oscillation, chaos, …) x1 x2

  10. Learning • The procedure that consists in estimating the parameters of neurons (setting up the weights) so that the whole network can perform a specific task. • 2 types of learning • Supervised learning • Unsupervised learning • The Learning process (supervised) • Present the network a number of inputs and their corresponding outputs (Training) • See how closely the actual outputs match the desired ones • Modify the parameters to better approximate the desired outputs • Several passes over the data

  11. Supervised Learning • The real outputs of the model for the given inputs is known in advance. The networks task is to approximate those outputs. • A “Supervisor” provides examples and teach the neural network how to fulfill a certain task

  12. Unsupervised learning • Group typical input data according to some function. • Data clustering • No need of a supervisor • Network itself finds the correlations between the data • Examples: • Kohonen feature maps (SOM)

  13. Properties of Neural Networks • Supervised networks are universal approximators (Non recurrent networks) • Can act as • Linear Approximator (Linear Perceptron) • Nonlinear Approximator (Multi Layer Perceptron)

  14. Other Properties • Adaptivity • Adapt weights to the environment easily • Ability to generalize • May provide against lack of data • Fault tolerance • Not too much degradation of performances if damaged  The information is distributed within the entire net.

  15. An Example Regression

  16. Example Classification • Handwritten digit recognition • 16x16 bitmap representation • Converted to 1x256 bit vector • 7500 points on training set • 3500 points on test set 0000000001100000 0000000110100000 0000000100000000 0000001000000000 0000010000000000 0000100000000000 0000100000000000 0000100000000000 0000100000000000 0001000111110000 0001011000011000 0001100000001000 0001100000001000 0001000000001000 0000100000010000 0000011111110000

  17. Training • Try to minimize an error or cost function • Backpropogation algorithm • Gradient Descent • Learn the weights of the network • Update the weights according to the error function

  18. Applications • Handwritten Digit Recognition • Face recognition • Time series prediction • Process identification • Process control • Optical character recognition • Etc…

  19. Neural Networks • Neural networks are statistical tools • Adjust non linear functions to accomplish a task • Need of multiple and representative examples but fewer than in other methods • Neural networks can model static (FF) and dynamic (RNN) tasks • NN’s are good classifiers BUT • Good representations of data have to be formulated • Training vectors must be statistically representative of the entire input space • The use of NN needs a good comprehension of the problem

  20. Implementation of Neural Networks • Generic architectures (PC’s etc) • Specific Neuro-Hardware • Dedicated circuits

  21. Generic architectures • Conventional microprocessors Intel Pentium, Power PC, etc … • Advantages • High performances (clock frequency, etc) • Cheap • Software environment available (NN tools, etc) • Drawbacks • Too generic, not optimized for very fast neural computations

  22. Classification of Hardware • NN Hardware • Neurochips • Special Purpose • General Purpose (Ni1000, L - Neuro) • NeuroComputers • Special Purpose (CNAPS, Synapse) • General Purpose

  23. Specific Neuro-hardware circuits • Commercial chips CNAPS, Synapse, etc. • Advantages • Closer to the neural applications • High performances in terms of speed • Drawbacks • Not optimized to specific applications • Availability • Development tools

  24. CNAPS • SIMD • One instruction sequencing and control unit • Processor nodes (PN) • Single dimensional array (only right or left nodes)

  25. CNAPS 1064

  26. CNAPS

  27. Dedicated circuits • A system where the functionality is buried in the hardware. • For specific applications only not changeable • Advantages • Optimized for a specific application • Higher performances than the other systems • Drawbacks • High development costs in terms of time and money

  28. What type of hardware to be used in dedicated circuits ? • Custom circuits • ASIC (Application-Specific Integrated Circuit) • Necessity to have good knowledge of the hardware design • Fixed architecture, hardly changeable • Often expensive • Programmable logic • Valuable to implement real time systems • Flexibility • Low development costs • Lower performances compared to ASIC (Frequency, etc.)

  29. Programmable logic • Field Programmable Gate Arrays (FPGAs) • Matrix of logic cells • Programmable interconnection • Additional features (internal memories + embedded resources like multipliers, etc.) • Reconfigurability • We can change the configurations as many times as desired

  30. Real Time Systems • Execution of applications with time constraints. • Hard real-time systems • Digital fly-by-wire control system of an aircraft:No lateness is accepted. The lives of people depend on the correct working of the control system of the aircraft. • Soft real-time systems • Vending machine:Accept lower performance for lateness, it is not catastrophic when deadlines are not met. It will take longer to handle one client with the vending machine.

  31. Real Time Systems • ms scale real time system • Connectionist retina for image processing • Artificial Retina: combining an image sensor with a parallel architecture • µs scale real time system • Level 1 trigger in a HEP experiment

  32. Eye CAN Connectionist Retina • Integration of a neural network in an artificial retina • Screen • Matrix of Active Pixel sensors • CAN • 8 bits ADC converter 256 levels of grey • Processing Architecture • Parallel system where neural networks are implemented Processing Architecture

  33. Maharadja Processing Architecture Command bus Micro-controller • Micro-controller • Generic architecture executing sequential cost with low power consumption • Memory • 256 Kbytes shared between processor, PE’s, input • Store the network parameters • UNE (Unit Neural SIMD • Completely pipelined • 16 bit internal data bus) • Processors to compute the neurons outputs • Command bus manages all different operators in UNE • Input/Output module • Data acquisition and storage of intermediate results M M M M UNE-0 UNE-1 UNE-2 UNE-3 Sequencer Instruction Bus Input/Output unit

  34. Level 1 trigger in a HEP experiment • High Energy Physics (Particle Physics) • Neural networks have provided interesting results as triggers in HEP. • Level 2 : H1 experiment 10 – 20 µs • Level 1 : Dirac experiment 2 µs • Particle Recognition • High timing constraints (in terms of latency and data throughput)

  35. Neural Network architecture Electrons, tau, hadrons, jets 4 64 …….. …….. 128 Execution time : ~500 ns with data arriving every BC=25ns Weights coded in 16 bits States coded in 8 bits

  36. Very Fast Architecture • 256 PE’s • Matrix of n*m matrix elements • Control unit • I/O module • TanH are stored in LUTs • 1 matrix row computes a neuron • The results is back-propagated to calculate the output layer PE PE PE PE ACC TanH PE PE PE PE ACC TanH PE PE PE PE ACC TanH PE PE PE PE ACC TanH Control unit I/O module

  37. PE architecture Data in Data out Multiplier Accumulator Input data 8 + X 16 Weights mem Addr gen Control Module cmd bus

  38. Neuro-hardware today • Generic Real time applications • Microprocessors technology (PCs, computers, i.e. software) is sufficient to implement most of neural applications in real-time (ms or sometimes µs scale) • This solution is cheap • Very easy to manage • Constrained Real time applications • It still remains specific applications where powerful computations are needed e.g. particle physics • It still remains applications where other constraints have to be taken into consideration (Consumption, proximity of sensors, mixed integration, etc.)

  39. Clustering • Idea : Combine performances of different processors to perform massive parallel computations High speed connection

  40. Clustering • Advantages • Take advantage of the implicit parallelism of neural networks • Utilization of systems already available (university, Labs, offices, etc.) • High performances : Faster training of a neural net • Very cheap compare to dedicated hardware

  41. Clustering • Drawbacks • Communications load : Need of very fast links between computers • Software environment for parallel processing • Not possible for embedded applications

  42. Hardware Implementations • Most real-time applications do not need dedicated hardware implementation • Conventional architectures are generally appropriate • Clustering of generic architectures to combine performances • Some specific applications require other solutions • Strong Timing constraints • Technology permits to utilize FPGAs • Flexibility • Massive parallelism possible • Other constraints (consumption, etc.) • Custom or programmable circuits

  43. Questions?

More Related