Automating estimation of warm-up length

1. The AutoSimOA Project A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation. http://www.wbs.ac.uk/go/autosimoa Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School Simulation Workshop - April 2008

2. Research Aim • To create an automated system for dealing with the problem ofinitial bias, for implementation into simulation software. • Target audience: non- (statistically) expert simulation users.

3. The Initial Bias Problem • Model may not start in a “typical” state. • Can cause initial bias in the output. • Many methods proposed for dealing with initial bias: e.g. Initial steady state conditions; run model for ‘long’ time… • This project uses: Deletion of the initial transient data by specifying a warm-up period (or truncation point).

4. Question is: How do you estimate the length of the warm-up period required?

5. Methods fall into 5 main types : • Graphical Methods. • Heuristic Approaches. • Statistical Methods. • Initialisation Bias Tests. • Hybrid Methods.

6. Literature search – 42 methods Summary of methods and literature references on project web site: http://www.wbs.ac.uk/go/autosimoa

7. Short-listing warm-up methods for automation using literature Short-listing Criteria • Accuracy & robustness • Simplicity • Ease of automation • Generality • Number of parameters to estimate • Computer running time

8. Short-listing results: reasons for rejection of methods

9. Short-listing results: 6 Methods taken forward to testing • Statistical methods: • Goodness of Fit (GoF) test • Algorithm for a static data set (ASD) • Algorithm for a Dynamic data set (ADD) • Heuristics: • MSER-5 • Kimbler’s Double Exponential Smoothing • Euclidean Distance Method (ED)

10. Testing Procedure Test short-listed methods using: • Artificial data – controllable & comparable • initial bias functions • steady statefunctions • Set of performance criteria.

11. 1. Artificial Data Sets Initial bias functions - 3 Criteria: i) Length – proportion of data length. • Severity – maximum bias value is a function of the difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data. • Shape and Orientation – 7 shapes:

12. Mean Shift: • Linear: • Quadratic: • Exponential: • Oscillating (decreasing):

13. Steady state functions - 3 Criteria: i)Constant steady state variance ii) Error Terms: Normal or Exponential distribution iii) Auto-Correlation: No AutoCorrelation; AR(1); AR(2); AR(4); MA(2); ARMA(5,5). Add Initial Bias to Steady state: Superpostion: Bias Fn, a(t), added onto end of steady state function: e.g.

14. 2. Performance Criteria • Closeness of estimated truncation point (Lsol) to true truncation point (L). • Coverage of true mean • ½ width of 95% CI for average truncated mean. • Bias and absolute bias in estimated mean. • Number of failures of method.

15. Test Results • Rejections: • ASD & ADD required a prohibitively large number of replications • GoF & Kimbler’s method consistently severely underestimated truncation point. • ED failed to give any result on majority of occasions • MSER-5 most accurate and robust method.

16. MSER-5 test statistic Output data (batched) MSER-5 Method

17. MSER-5 Results MSER5 result eg.xls Does the true mean fall into the 95% CI for the estimated mean?

18. Summary / Future Work • 42 warm-up methods • Short-listing and testing • MSER-5 most promising method for automation • Creation of heuristic framework around MSER-5 method for implementation into simulation software.

19. Thank you for listening. ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School SW08

20. ii) SEVERITY OF BIAS FUNCTION Set maximum value of bias fn, a(t), so that max |a(t)|t≤L = M×Q Q = difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data. M = relative maximum bias – user set: 1, 2, 5 M ≥ 1 → bias significantly separate from steady state data → easier to detect. M ≤ 1 → bias absorbed into steady state data variance → harder to detect.