Engineering Design Optimization Course

1. TTÜ, Department of Machinery Lead researcher Jüri Majak E-mail: juri.majak@ttu.ee MER9020

2. Structure of the course Introduction • Mathematical models and tools for building these models. Use of experimental data based models. Engineering statistics • Engineering statistics and design. Statistical Design of Experiments. Use of statistical models in design. • Artificial neural network models. • Accuracy of the model. Statistical tests of Hypotheses. Analysis of Variance. Risk, reliability, and Safety • Basics of Optimisation methods and search of solutions • Types of Optimization Problem • Multicriteria optimal decision theory. Pareto optimality, use of multicriteria decisiontheory in engineering design. • Traditional gradient based optimization. Lagrange multipliers. Examples of use the classical optimization method in engineering. • Mathematical programming methods and their use for engineering design, process planning and manufacturing resource planning.Direct and Dual Optimization tasks. Sensitivity analysis. • Evolutionary algorithms • Design of products and processes. Applications, samples

3. Experimental study, computer simulations. Protsess monitoring, statistical analysis of experimental data, model evaluation etc Design of Experiment (DOE) Simulation, analysis Meta modeling Development of response surface, its evaluation, validation Design optimization:decision making, linear and nonlinear programming Engineering design optimization Basic activities and relations

4. Homework topics • Descriptive statistics: statistical estimates, realtion between two test series(correlation) , variance. • Response modeling • Regressionanalysis,model validation. • ANN based models • Design optimization. Engineering problems with one and multiple optimality criteria.

5. Overall Goal in Selecting Methods The overall goal in selecting basic research method(s) in engineering design is to get the most useful information to key decision makers in the most cost-effective and realistic fashion. Consider the following questions:1. What information is needed to make current decisions about a product or technology?2. How much information can be collected and analyzed., using experiments ,questionnaires, surveys and checklists?3. How accurate will the information be?4. Will the methods get all of the needed information?5. What additional methods should and could be used if additional information is needed?6. Will the information appear as credible to decision makers, e.g., to engineers or top management? 7. How can the information be analyzed?

6. Definition of Engineering tasks Simulations – used for analysis of the object with given structure, an aim is to evaluate the behaviour of the objects depending on given values of input parameters Analysis – The parameters set and strcuture of the object are given, an aim is to analyse the behaviour of the object caused by changes of the parameter values Diagnostics – inverse to simulation, an aim is to determine parameters providing prescribed behaviour Synthesis – inverse to analysis, an aim is to determine the strcuture and the values of input parameters based on required output values

7. Basic Guidelines to Problem Solving and Decision Making 1. Define the problemIf the problem still seems overwhelming, break it down by repeating steps Prioritize the problems 2. Look at potential causes for the problem 3. Identify alternatives for approaches to resolve the problemAt this point, it's useful to keep others involved. 4. Select method, tool ,technique etc to solve the problem 5. Plan the implementation of the best alternative solution ( action plan) 6. Monitor implementation of the plan 7. Verify if the problem has been resolved or not

8. Response Surface Methodology (RSM) There is a difference between data and information. To extract information from data you have to make assumptions about the system that generated the data. Using these assumptions and physical theory you may be able to develop a mathematical model of the system. Generally, even rigorously formulated models have some unknown parameters.. Identifying of those unknown constants and fitting an appropriate response surface model from experimental data requires knowledge of Design of Experiments, regression modelling techniques, and optimization methods. The response surface equations give the response in terms of the several independent variables of the problem. If the response is plotted as a function of etc., we obtain a response surface Response surface methodology (RSM) has two objectives: 1. To determine with one experiment where to move in the next experiment so as to continually seek out the optimal point on the response surface. 2. To determine the equation of the response surface near the optimal point. Response surface methodology (RSM) uses a two-step procedure aimed at rapid movement from the current position into the region of the optimum. This is followed by the characterization of the response surface in the vicinity of the optimum by a mathematical model. The basic tools used in RSM are two-level factorial designs and the method of least squares (regression) model and its simpler polynomial forms

9. The Purpose of Modelling 1. To make an idea concrete. This is done by representing it mathematically, pictorially or symbolically. 2. To reveal possible relationships between ideas. Relationships of hierarchy, support, dependence, cause, effect, etc. can be revealed by constructing a model. We have to be careful, then, how much we let our models control our thinking. 3. To simplify the complex design problem to make it manageable or understandable. Almost all models are simplifications because reality is so complex. 4. The main purpose of modelling, which often includes all of the above three purposes, is to present a problem in a way that allows us to understand it and solve it.. Types of Models A. Visual. Draw a picture of it. If the problem is or contains something physical, draw a picture of the real thing--the door, road, machine, bathroom, etc. If the problem is not physical, draw a symbolic picture of it, either with lines and boxes or by representing aspects of the problem as different items--like cars and roads representing information transfer in a company. B. Physical. The physical model takes the advantages of a visual model one step further by producing a three dimensional visual model. C. Mathematical. Many problems are best solved mathematically.

10. Complexity theory for engineering design Three types of complexity: • Description complexity • Numerical complexity • Understanding (regognation) complexity • Numerical complexity theory is part of the theory of computation dealing with the resources required during computation to solve a given problem. The most common resources are time (how many steps does it take to solve a problem) and space (how much memory does it take to solve a problem). Other resources can also be considered, such as how many parallel processors are needed to solve a problem in parallel. Complexity theory differs from computability theory, which deals with whether a problem can be solved at all, regardless of the resources required.

11. Complexity classes(reference-functions Big O notation) : • Logaritmiccomplexity, O(log n) • Linear complexity, O(n) • Polynomial complexity, O( nq ) • Eksponentialcomplexity • Factorialcomplexity • Double-eksponentialcomplexity. Algorithmic complexity is concerned about how fast or slow par- ticular algorithm performs. We define complexity as a numerical function T(n) - time versus the input size n. We want to define time taken by an algorithm without depending on the implementation details.

12. Complexity classes http://www.cs.cmu.edu/~adamchik/15-121/lectures/Algorithmic%20Complexity/complexity.html Asymptotic Notations The goal of computational complexity is to classify algorithms according to their performances. We will represent the time function T(n) using the "big-O" notation to express an algorithm runtime complexity. For example, the following statement T(n) = O(n2) says that an algorithm has a quadratic time complexity. Definition of "big Oh" For any monotonic functions f(n) and g(n) from the positive integers to the positive integers, we say that f(n) = O(g(n)) when there exist constants c > 0 and n0 > 0 such that f(n) ≤ c * g(n), for all n ≥ n0 Intuitively, this means that function f(n) does not grow faster than g(n), or that function g(n) is an upper bound for f(n), for all sufficiently large n→∞

13. Complexity classes Exercise. Let us prove n2+ 2 n + 1 = O(n2). We must find such c and n0that n2+ 2 n + 1 ≤ c*n2. Let n0=1, then for n ≥ 1 1 + 2 n + n2≤ n + 2 n + n2≤ n2+ 2 n2+ n2= 4 n2 Therefore, c = 4.

14. Complexity classes Constant Time: O(1) An algorithm is said to run in constant time if it requires the same amount of time regardless of the input size. Example:array: accessing any element Linear Time: O(n)An algorithm is said to run in linear time if its time execution is directly proportional to the input size, i.e. time grows linearly as input size increases. Examples:array: linear search, traversing, find minimumi:=1p:=1for i:=1 to n Ex: Find complexity of the algorithm p:=p*i i:=i+1 f(n)=4*n+2 O(n)endfor

15. Complexity classes Logarithmic Time: O(log n) An algorithm is said to run in logarithmic time if its time execution is proportional to the logarithm of the input size. Example:binary search Quadratic Time: O(n2)An algorithm is said to run in quadratic time if its time execution is proportional to the square of the input size. Examples:bubble sort, selection sort, insertion sort for i:=1 to n for j:=1 to n A(i,j):=x Ex: Find complexity of the algorithm endfor endfor

16. Complexity classes Ex: Find complexity of the algorithm s:=0 for i:=1 to n for j:=1 to i s:=s+j*(i-j+1) endfor endfor

17. Complexity classes

18. Complexity for Recursive algorithms Initial problem with data capacity n will be divided into b subproblems with equal capacity. Only a (a<b) subproblems is solved (others not needed). (1) Theorem: Assuming a>=1 ja b>1 are constants, f(n) function and T(n) defined for non-negative n by formula (1). Then: a) T(n) is if f(n) is (e – positive constant) , b) T(n) is if f(n) is c) T(n) is if f(n) is (e – positive constant , and af(n/b)<=cf(n) Ex1: apply theorem to binary search Ex2: apply theorem for problem where a=2, b=4, f(n)=n2+2n+9 Ex3: apply theorem for problem where a=2, b=4, f(n)=3 Ex4: Find number of opertions for a=2, b=4, f(n)=2, n=2000.

19. Ex2: apply theorem for problem where a=2, b=4, f(n)=n2+2n+9 Complexity of f(n) is O(n2+2n+9) = O(n2) Case c) constant e=1.5 Asymptotic estimate from case c):

20. Ex3: apply theorem for problem where a=2, b=4, f(n)=3 Complexity of f(n) is O(1) Case a) constant e=0.5 Asymptotic estimate from case a):

21. Descriptive Statistics (Excel,...)

22. Descriptive Statistics Find the mean, median, mode, and range for the following list of values: 13, 18, 13, 14, 13, 16, 14, 21, 13 The mean is the usual average, so: (13 + 18 + 13 + 14 + 13 + 16 + 14 + 21 + 13) ÷ 9 = 15 Note that the mean isn't a value from the original list. This is a common result. You should not assume that your mean will be one of your original numbers. The median is the middle value, so I'll have to rewrite the list in order: 13, 13, 13, 13, 14, 14, 16, 18, 21 There are nine numbers in the list, so the middle one will be the (9 + 1) ÷ 2 = 10 ÷ 2 = 5th number: 13, 13, 13, 13, 14, 14, 16, 18, 21 So the median is 14.    The mode is the number that is repeated more often than any other, so 13 is the mode. The largest value in the list is 21, and the smallest is 13, so the range is 21 – 13 = 8. mean:15median:14mode:13range: 8

23. Descriptive Statistics Standard deviation: Standard error: The variance of a random variable X is its second central moment, the expected value of the squared deviation from the mean μ = E[X] The variance is quadrat of standard deviation Sample variance In probability theory and statistics, the variance is used as a measure of how far a set of numbers are spread out from each other. It is one of several descriptors of a probability distribution, describing how far the numbers lie from the mean (expected value).

24. Descriptive Statistics Kurtosis In probability theory and statistics, kurtosis (from the Greek word κυρτός, kyrtos or kurtos, meaning bulging) is a measure of the "peakedness" of the probability distribution of a real-valued random variable, although some sources are insistent that heavy tails, and not peakedness, is what is really being measured by kurtosis. Higher kurtosis means more of the variance is the result of infrequent extreme deviations, as opposed to frequent modestly sized deviations. Kurtosis is a measure of how outlier-prone a distribution is. The kurtosis of the normal distribution is 3. Distributions that are more outlier-prone than the normal distribution have kurtosis greater than 3; distributions that are less outlier-prone have kurtosis less than 3. The kurtosis of a distribution is defined as

25. Descriptive Statistics Skewness In probability theory and statistics, skewness is a measure of the asymmetry of the probability distribution of a real-valued random variable. The skewness value can be positive or negative, or even undefined. Qualitatively, a negative skew indicates that the tail on the left side of the probability density function is longer than the right side and the bulk of the values (possibly including the median) lie to the right of the mean. A positive skew indicates that the tail on the right side is longer than the left side and the bulk of the values lie to the left of the mean. A zero value indicates that the values are relatively evenly distributed on both sides of the mean, typically but not necessarily implying a symmetric distribution. The skewness of a random variable X is the third standardized moment, denoted γ1 and defined as

26. Correlation.

27. Correlation Correlation is a statistical measure that indicates the extent to which two or more variables fluctuate together. A positive correlation indicates the extent to which those variables increase or decrease in parallel; a negative correlation indicates the extent to which one variable increases as the other decreases. Samples: http://www.mathsisfun.com/data/correlation.html

28. The local ice cream shop keeps track of how much ice cream they sell versus the temperature on that day, here are their figures for the last 12 days: Correlation

29. Correlation

30. Correlation

31. Example of correlation using Excel Data analysis

32. Estimation of the variance of data series using F-test http://en.wikipedia.org/wiki/F-test_of_equality_of_variances This F-test is known to be extremely sensitive to non-normality

33. Estimation of the variance of data series using F-test 0,172124 - Generally, if this value is less than 0.05 you assume that the variances are NOT equal.

34. Estimation of the variance of data series using single factor ANOVA http://www.miracosta.edu/Home/rmorrissette/Chapter13.htm

35. Homework1: Statistical evaluation of test dataDescriptive Statistics • Calculate basic descriptive statistics for two series of test data (data series selected by yourself). • Estimate relations between these two data series (correlation, covariation) • Estimate the variance of data series using F-test. • If then zero hypothesis is valid and the variance of two data series is arbitrary. FT istable values taken with given confidence interval (95% or 99%, etc.).  If Fobserved > Fcritical, we conclude with 95% confidence that the null hypothesis is false. • Null hypothesis Ho: all sample means arising from different factors are equal • Alternative hypothesis Ha: the sample means are not all equal

36. Homework1- all work done should be explainedmeaning of parameters, etc. 1.Explain results of descriptive statistics 2. Explain relations between data series 3. Explain variance of data series

37. MATLAB Statistics Descriptive StatisticsData summariesStatistical VisualizationData patterns and trendsProbability DistributionsModeling data frequencyHypothesis TestsInferences from dataAnalysis of VarianceModeling data varianceRegression AnalysisContinuous data modelsMultivariate MethodsVisualization and reductionCluster AnalysisIdentifying data categoriesModel AssessmentIdentifying data categoriesClassificationCategorical data modelsHidden Markov ModelsStochastic data modelsDesign of ExperimentsSystematic data collectionStatistical Process ControlProduction monitoring

38. Linear regression analysisThe earliest form of regression was the method of least squares, which was published by Legendre in 1805, and by Gauss in 1809.

39. Regression models Regression models involve the following variables: • The unknown parameters, denoted as β, which may represent a scalar or a vector. • The independent variables, X. • The dependent variable, Y. A regression model relates Y to a function of X and β. The approximation is usually formalized as E(Y | X) = f(X, β). To carry out regression analysis, the form of the function f must be specified. Sometimes the form of this function is based on knowledge about the relationship between Y and X that does not rely on the data. If no such knowledge is available, a flexible or convenient form for f is chosen.

40. Regression analysis(just any data)

41. Regression analysis(particular model)

42. Results samplehttp://www.excel-easy.com/examples/regression.html

43. Linearization

44. MATLAB regression analysis Regression Plots, Linear Regression, Nonlinear Regression, Regression Trees, Ensemble Methods

45. REGRESSION ANALYSIS. Matlab Simple linear regression % data xx=[2.38 2.44 2.70 2.98 3.32 3.12 2.14 2.86 3.5 3.2 2.78 2.7 2.36 2.42 2.62 2.8 2.92 3.04 3.26 2.30] yy=[51.11 50.63 51.82 52.97 54.47 53.33 49.90 51.99 55.81 52.93 52.87 52.36 51.38 50.87 51.02 51.29 52.73 52.81 53.59 49.77] % data sort [x,ind]=sort(xx); y=yy(ind); % linear regression [c,s]=polyfit(x,y,1); % structure s contains fileds R,df, normr…. [Y,delta]=polyconf(c,x,s,0.05); % plot plot(x,Y,'k-',x,Y-delta,'k--',x,Y+delta,'k--',x,y,'ks',[x;x],[Y;y],'k-') xlabel('x (input)'); ylabel('y (response)');

46. REGRESSION ANALYSIS. Matlab , Multiple linear regression , Example1: linear model for cubic polynomial in one independent variable Let: Linear model: , Example2: linear model for quadratic polynomial in two independent variable Let: , , , Linear model:

47. Example x1=[7.3 8.7 8.8 8.1 9.0 8.7 9.3 7.6 10.0 8.4 9.3 7.7 9.8 7.3 8.5 9.5 7.4 7.8 7.8 10.3 7.8 7.1 7.7 7.4 7.3 7.6]’ x2=[0.0 0.0 0.7 4.0 0.5 1.5 2.1 5.1 0.0 3.7 3.6 2.8 4.2 2.5 2.0 2.5 2.8 2.8 3.0 1.7 3.3 3.9 4.3 6.0 2.0 7.8]’ x3=[0.0 0.3 1.0 0.2 1.0 2.8 1.0 3.4 0.3 4.1 2.0 7.1 2.0 6.8 6.6 5.0 7.8 7.7 8.0 4.2 8.5 6.6 9.5 10.9 5.2 20.7]’ y=[0.222 0.395 0.422 0.437 0.428 0.467 0.444 0.378 0.494 0.456 0.452 0.112 0.432 0.101 0.232 0.306 0.0923 0.116 0.0764 0.439 0.0944 0.117 0.0726 0.0412 0.251 0.00002] X=[ones(length(y),1),x1,x2,x3,x1.*x2,x1.*x3,x2.*x3, x1.^2, x2.^2, x3.^2] % Regression [b,bcl,er,ercl,stat]=regress(y,X,0.05) disp(['R2=' num2str(stat(1))]) disp(['F0=' num2str(stat(2))]) disp(['p-value=' num2str(stat(3))]) rcoplot(er,ercl) In cases where the distribution of errors is asymmetric or prone to outliers, the computed statistics with function regress() become unreliable. Latter case the function robustfit() can be preferred. % if the distribution of errors is asymmetric or prone to outliers X2=X(:,2:9) robustbeta = robustfit(X2,y)

48. Design of experiment The aim in general: to extract as much as possible information from a limited set of experimental study or computer simulations. to maximize the information content of the measurements/simulations in the context of their utilization for estimating the model parameters. Particularly: the selection of the points where the response should be evaluated Can be applied for model design with: Experimental data Simulation data Can be applied for model fitting Objective functions Constraint functions Why DOE DOE allows the simultaneous investigation of the effect of a set of variables on a response in a cost effective manner. DOE is superior to the traditional one-variable-at-a-time method, which fails to consider possible interaction between the factors.

49. DOE techniques selection • DOE is introduced for describing real life problems: • 1920-s by R.Fisher, agricultural experiments, • G.Box 1950-s, for modeling chemical experiments • Nowadays various engineering applications, production planning, etc. • A huge number of DOE methods are available in literature and selection of the most suitable method is not always the simplest task. Preparative activities needed: • formulation of the problem to be modelled by DOE, • selection of the response variable(s), • choise of factors (design variables), • determining ranges for design variables. • If this preliminar analysis is successfully done, then the selection of the suitable DOE method is simpler. The selection of the levels of factors is also often classified as preliminar work. • In the following two DOE methods are selected and discussed in more detail: • the Taguchi methods, • allows to obtain preliminary robust design with small number of experiments and it is the most often applied at early stages of process development or used as initial design • full factorial design • resource consuming, but leads to more accurate results

50. DOE techniques selection Note, that the Taguchi design can be obtained from full factorial design by omitting certain design points. Also there are several approaches, based on full factorial design. For example central composite design can be obtained from 2N full factorial design by including additional centre and axial points. An alternate well known DOE techniques can be outlined as D-optimality criterion, Latin hypercube, Van Keulen scheme, etc. The D-optimality criterion is based on maximization of the determinant t , where stand for the matrix of the design variables. Application of the D optimality criterion yields minimum of the maximum variance of predicted responses (the errors of the model parameters are minimized). The Latin hypercube design maximizes the minimum distance between design points, but requires even spacing of the levels of each factor [12]. The van Keulen’s scheme is useful in cases where the model building is to be repeated within an iterative scheme, since it adds points to an existing plan.