Volodymyr STEPASHKO Prof., Dr. Sci., Head of Department

1. International Research and Training Centre for Information Technologies and Systemsof the National Academy of Sciences of Ukraine Volodymyr STEPASHKO Prof., Dr. Sci., Head of Department for Information Technologies of Inductive Modeling Kyiv, March 2013 Inductive Modeling from historical perspective

2. Terminology evolution: heuristic self-organizationof models (1970s) inductive methodof model building (1980s) inductive learningalgorithms for modeling (1992) inductive modeling(1998) MGUA: Method of Group Using of Arguments (direct translation of original name of the method) GMDH: Group Method of Data Handling (standard international name starting from the very first translation in USA) 2

3. Structure of Knowledge in the Inductive Modeling Area 3

4. GMDH-Based Information Technology A S T R I D for modeling complex processes from data

5. Attempt to define IM: what is it? • IM generally is aprocess of inductive transition from data to models under uncertainty conditions: • limited size of data set: small sample of noisy data • limited volume of a priori information: - unknown character and level of noise - inexact composition of relevant arguments (inputs) - unknown structure of relationships in an object IM is the MGUA/GMDH based technique for model self-organization IM is a technology for noise-immunity modeling Deduction Induction MODEL DATA THEORY Two opposite (but supplemental) approaches to model building

6. IM destination: what is this for? There is a wide experience of IM using for the following problems to be solved: • Forecasting of complex processes • Structural and parametric identification • Data compression (via optimal approximation) • Classification and pattern recognition (supervised learning) • Data clustering (unsupervised learning) • Machine learning • Data Mining • Knowledge Discovery

7. IM explanation: main tasks General problem definition Given: data set of n observations after m input x1,x2,…xm and one output y variables Find: model y=f(x1,x2,…xm ,θ)with minimum variance of prediction error, θis unknown vector of model parameters GMDH Task:f *= arg minΦC (f ) C (f ) is a model quality criterion Φisaset of models, f Φ Illustration: choose an optimal subset of monomials out of the member set of Kolmogorov-Gabor polynomial: 7

8. Two interdependent tasks Discrete optimization task Φ– set of model structures С– criterion of a model quality Structure of a model: Estimation of parameters: Continuous optimization task Q – quality criterion for parameters estimation 8

9. Main stages of the modeling process 9

10. Main components of a method of inductive modeling Class of models Generator of model structures Method of inductive modeling Method of parameter estimation Criterion of model selection

11. Basic Principles of GMDH as an Inductive Method 1. Generation of variants of the gradually complicated structures of models 2. Successive selection of the best variants using the freedom of choice of decisions 3. External criteria (based on the sample division) for the best model selection Data-driven inductive modelling process 11

12. External Selection Criteria Given data set: W = (X y), X [n x m], y [n x 1] Division into two subsets A and B : Parameter estimation by LSM for a model y =X : W=AB Regularity criterion: Unbiasednesscriterion: 12

13. General classification of GMDH algorithms GMDH algorithms Sorting-out Iterative 1.Combinatorial (Exhaustive Search) COMBI 2.Multistage (Directed Search) MULTI 3.Multilayered Iterative MIA 4.Relaxation Iterative RIA

14. Main types of model structure generators • COMBI GMDH = COMBInatorial algorithm • (sorting-out type) • All possible combinations are considered: • yν = Xνθν,ν=1,2,3,…,2m • Structural binary vector:d =(d1,d2,…,dm), dj ={0;1} • Example • Level 1: yi = αi xii = 1,2,…,m • Level 2: yk = αkixi +αkj xj , i,j =1,2,…,m ,k=1…Сm2 • Level 3: Сm3 model structures of 3 arguments etc. • Total number of models: pm = sСms = 2m

15. Main generator types 2.MULTI GMDH = MULTIstage (combinatorial- selective) algorithm (sorting-out type) It selects only the most significant models and retrieves the exhaustive search result In any stage s, the argument subset of a best model of previous stage is supplemented by anyone argument absent in this model by turn: ysk = (X is-1 | xj )θs , s,j =1,2,…,m ; i=1,2,…,Fs-1;k = 1,2,…,(m – s)Fs-1 Fs is the freedom of choice in a stage s , usually Fs≤ m The whole number of generated models ~m 3 (polynomial rise) vs. 2m(exponential rise) in COMBI 15

16. Main generator types 3. MIA GMDH = Multilayered Iterative Algorithm (basic, classical, iterative type) Layer 1: yh = φh (xi ,xj ) , h = 1,2,…Cm2 Layer 2: fk = φk (yi ,yj ), k = 1,2,…CF2 Layer 3: gk = φk (fi , fj ), k = 1,2,…CF2 etc. Typical variants of the partial description φ : (a)ykr+1=α0 +α1kyir +α2kyjr , yi0=xi , yj0=xj (b) ykr+1=α1kyir+α2kyjr+α3kyiryjr (c) ykr+1=α1kyir+α2kyjr +α3kyiryjr +α4k(yir)2+α5k(yjr)2 r = 1,2,…..; i,j =1,2…F ; k = 1,2,…CF2 F is the freedom of choice, usually F =m 16

17. Classical Multilayered Iterative Algorithm MIA GMDH 1st layer 2nd layer 17

18. Main generator types 4. RIA GMDH = Relaxational Iterative Algorithm (non-classical, iterative type) Any iteration (layer): fk = φk (yi ,xj ), k = 1,2,…CF2 It considers pairs (yi ,xj) instead of (yi ,yj) in MIA Typical variants of the partial description φ: (a) ykr+1=α0+α1kyir+α2kxj , yi0=xi (b) ykr+1= α1kyir +α2kxj +α3kyirxj (c) ykr+1= α1kyir +α2kxj +α3kyirxj +α4k(yir)2+α5k(xj)2 r= 1,2,…; i=1,2,…,F ; j=1,2,…,m ; k=1,2,…,mF F is the freedom of choice, usually F =m 18

19. Basic Theoretical Results • Main concept: Self-organizing evolution of the model ofoptimal complexityunder uncertainty conditions • Main result: Complexity of the optimum forecasting model depends onthe level of uncertaintyin the data: the higher it is, the simpler (more robust) there must be the optimum model • Main conclusion: GMDH is the method for construction of models with minimum varianceof forecasting error

20. Illustration to GMDH theory Coordinates: complexitys criterionC (s) Parameter: noise variance σ2 Reduction of optimal complexity sowhen σ2grows Here true model contains: 3 relevant + 2 redundant arguments

21. IM compared to ANN MIA GMDH as Polynomial Neural Network (PNN) Illustration for inductive (forward) process of model construction 21

22. Optimal structure of the tunedGMDH net Trained GMDH network (after backward tuning) Argument x2 appears to be redundant

23. IM from CI perspective • GMDH-based algorithms have typical features • of Computational Intelligence tools: • evolutionary-type computations • network-like structures • data-driven learning • nature-inspired procedures – BICA ! • Main advantages of IM algorithms: • automaticevolving of model structure and parameters • self-organizing net structures (node and layer numbers) • fast learning (locally optimized nodes) • Equally, IM may be attributed to Soft Computing means: • inductiveinference procedures • precedence-based reasoning • fuzzy GMDH realizations

24. Some real-world applications of IM • Modelling of economical processes: - prediction of tax revenues and inflation - system prediction of power indicators • Modelling of ecological processes: - activity of microorganisms in soil and green algae in water under influence of heavy metals - irrigation of trees by processed wastewaters • Simulation in medicine: - self-monitoring of diabetes - prediction of drugs effectiveness • Integral evaluation of the state of complex multidimensional systems - economic safety - investment activity - ecological state of water reservoirs • Technology of informational support of operative management decisions

25. Illustration example 1: Comparison of prediction quality of the real inflation process (USA, 1999) using regr. analysis (LSM) and GMDH LSMmodel structure (static model): Y = a1X1 + a2X2 + a3X3+ a4X4+a5X5+ a6X6 GMDHmodel structure (static model): Y = a1X1 + a3X3+ a6X6 Y – inflation rate X1 – personal savings; X4 – personal consumption; X2 – total unemployed number;X5 – personal incomes; X3 – interest rates; X6 – GDP Note: learning interval = 15 months (training+checking) prediction points (3 months) indicated by arrows 25

26. Illustration example 2:Ukraine budget revenues Y – budget revenues X1 – cash income of population X2 – cash disbursements and savings of population X3 – consumer price index(%) X4 – price index in light industry X5 – GDPindex X6 – total retail turnover X7 – total employment X8 – light industry employment X9 – wage (nominal) X10 – wage index (real, %) X11 – money supply X12 – US$ official course X13 – account payablebetveen enterprises X14 – expenditure of the consolidated budget Dependence of prediction efficiency for tax revenues on the statistical sample volume is investigated.

27. 1995 1996 1997 1998 1999 Training Prediction Fig.1. Prediction quality (14 trainingpoints)

28. 1995 1996 1997 1998 1999 Training Prediction Fig.2. Prediction quality (18trainingpoints)

29. 1995 1996 1997 1998 1999 Training Prediction Fig.3. Prediction quality (29trainingpoints)

30. Main developed IM tools • Information Technology ASTRID (IRTC, Kyiv) www.mgua.irtc.org.ua • KnowledgeMiner (Frank Lemke, Berlin) www.knowledgeminer.net • FAKE GAME (Pavel Kordik et al., Prague) http://ncg.felk.cvut.cz • GMDH_Shell (Oleksiy Koshulko, Kyiv) http://www.gmdhshell.com

31. Main centers of IM research • IRTC ITS NANU, Kyiv, Ukraine • NTUU “KPI”, Kyiv, Ukraine • KNURE, Kharkiv, Ukraine • KnowledgeMiner, Berlin, Germany • CTU in Prague, Czech • Sichuan University, Chengdu, China

32. IM development prospects The most promising directions: 1. Theoretical investigations Study of GMDH performance for cases of differentnoise properties and modelclasses, new external criteria, etc. 2. Integration of IM, NN and CI best developments Algorithms with heterogenic and fuzzy neurons, immune networks, GA and nature-inspired estimators etc. 3. Paralleling Constructing computational schemes oriented to multi-core and cluster computer architectures

33. The most promising directions : 4. Preprocessing Constructing the optimal combination of preprocessing procedures 5. Ensembling Modelling and forecasting based on weighted averaging of a groupof the best models 6. Intellectual interface Knowledge-based interactive modes with strong support and control of user’s activity, default modes, etc. 7. Case studies New applications to real-world tasks of different nature 33

34. MainWeb sources on GMDH Basic home page:info, books, articles, reviews, software http://www.gmdh.net Professional research group site: ITIM Department developments, staff, info, Proceedings of ICIM/IWIMs http://www.mgua.irtc.org.ua Research&training group at CTU in Prague: Developments, staff, ICIM/IWIM home pages http://cig.felk.cvut.cz Business site:info,software, applications http://www.knowledgeminer.net Open-source site:info, parallel algorithms http://opengmdh.org 34

35. THANK YOU ! Volodymyr STEPASHKO Address: Prof. Volodymyr Stepashko, International Centre for ITS of NANUAkademikGlushkovProspekt 40 Kyiv, MSP, 03680, UkrainePhone: +38 (044) 526-30-28 Fax: +38 (044) 526-15-70 E-mail:stepashko@irtc.org.uaWeb: www.mgua.irtc.org.ua