SAE 599 - Modeling and Simulation for Systems Architecting and Engineering

1. SAE 599 - Modeling and Simulation for Systems Architecting and Engineering Dr. Raymond Madachy November 21, 2007

2. Outline Test problem review Analysis of simulation output Confidence intervals Homework and readings

3. Review � Combined Discrete and Continuous Simulation Types of interactions between discretely changing and continuously changing variables A discrete event causes a discrete change in the value of a continuous state variable A discrete event causes a relationship governing a continuous state variable to change at a particular time A continuous state variable achieving a threshold value causes a discrete event to occur or to be scheduled

4. Planetary Rover Combined Simulation Review

5. Analysis of Simulation Output  Simulation models convert stochastic inputs and system components into statistical data output hence, simulation is another sampling method and the output is subject to statistical analysis Variable estimates are subject to sampling choice and sample size determining proper sample size may require some knowledge of parameters to estimate conclusions should not be based on results of single simulation run System Classification for Output Analysis Non-terminating have no end to operations for practical time horizon e.g. phone system, emergency room, traffic, even factories when daily start condition is same as end of previous day usually, but not always have steady state(s) may wish to study transient or steady state conditions treat as terminating system to study transient state

6. Analysis of Simulation Output (cont.) System Classification for Output Analysis Terminating systems usually start from no-action or empty state and end with either of same; termination occurs after time delta or event occurrence e.g. time lapse termination: bank hours, quarterly inventory event termination: device failure, bid process, war may or may not reach steady state if steady state exists, system may be treated as non-terminating for certain purposes Several runs are necessary if event-controlled terminating systems don't reach steady state use different random number allocation per replication Data Dependency Independency of sample data allows use of classical statistical analysis However, most discrete event simulation models lack independency queueing process usually leads to autocorrelation may choose to select samples far apart in the system

7. Analysis of Simulation Output (cont.) Independent Replications Addresses problem of autocorrelation. Statistical techniques covered so far assume samples are independent and identically distributed (IID). perform several short runs (replications) from time=0 use different random number seeds per run (or possibly different initial conditions) replications are then independent of each other each replication mean is an independent observation Compute mean, variance and confidence interval Batch Means Method only applies to steady state analysis alleviates problem of handling transitional periods during independent replications divide single run into multiple intervals (batches) shorter intervals impose stronger dependencies, so larger batch sizes and fewer batches are recommended batch mean values serve as independent samples compute each mean and the grand mean

8. Confidence Intervals Estimate accuracy expressed as a confidence interval (CI) CI interpretation: if one constructs a very large number of 1-a CIs each based on n observations, in the long run 1- a of the intervals contain the parameter value. a is also called the significance level, or the probability of rejecting a hypothesis when it's true. for given confidence level, a small confidence interval is preferable for given confidence interval, a higher confidence level is preferable in practice, choose a confidence level sample size affects confidence level and interval smaller confidence intervals with larger sample size estimation of mean due to central limit theorem, distribution of sample mean is normal Z is normally distributed Compute 1-a CI. Use normal distribution tables if >30 samples, or student t-distribution if < 30 samples. Other measures for confidence intervals estimation of proportion estimation of difference between means

9. Confidence Intervals (cont.) Confidence interval equation: where X is the sample mean, m is the population mean and s is the standard deviation Below is a normal distribution with a mean of one. The true mean falls in the confidence interval with a probability of (1-a).