Measuring and Explaining Differences in Wireless Simulation Models

1. Measuring and Explaining Differences in Wireless Simulation Models Dheeraj Reddy, George F. Riley, Yang Chen, Bryan Larish Georgia Institute of Technology

2. Outline • Introduction • Simulation Experiments • Parameters • Topology • Description • Results and Discussion • Conclusions

3. Introduction • Discrete-Event Simulations • Why wireless simulations ? • GTNetS, ns-2 and GloMoSim • IEEE 802.11 (MAC and PHY) • Motivated by Cavin et. al (POMC 2002) • Same specification but different results • Why is this important ?

4. Introduction • Quantifiable differences • Goals • Not to point correct/incorrectness • Why wireless simulations will often provide differing results • Precautions when drawing conclusions from simulations

5. Simulation Experiments • Two experiments • Concentration at MAC layer only • Ideal Behavior Verification under no contention • Behavior during contention resolution

6. Simulation Experiments IEEE 802.11 parameters used in simulations

7. Simulation Experiments • Experimental Methodology • Simple forwarding protocol • Node n forwards packet to Node (n+1)%100 • A round is finished when node 0 receives its packet from node 99 • Experiment finishes at the end of 100 rounds

8. Simulation Experiments Simulation Network Topology

9. Simulation Experiments • First Experiment • Node 0 originates a packet • Experiment finishes at the end of 100 rounds • Deterministic results ? • There is contention even when a single packet is making rounds • 500 us forwarding delay

10. Simulation Experiments • Implementation artifact resulting in contention (all three simulators)

11. Results and Discussions • Pristine Simulators • GloMoSim sends control frames as well as data frames at the same rate • ns-2 uses ARP • ns-2 and GloMoSim ignore LLC/SNAP layer • Additional random delays in ns-2 Experiment 1 using default Simulation Parameters

12. Results and Discussions • Theoretical Analysis • DR=BR=11Mbps • Deterministic results

13. Results and Discussions • Set control frame rate as well as data frame rate to 11 Mbps • Ignore 1st round in ns-2 • Adjust payload size in ns-2 and GloMoSim to account for missing headers • Extra random delays removed in ns-2 • Closely matches with theoretical analysis

14. Simulation Experiments • Second Experiment • Exercise the contention resolution mechanisms • All nodes create and send a packet to their neighbor at time picked from [0,10ms) • 100 packets contending for medium • Significant packet loss • Widely varying results

15. Results and Discussions • Code Inspection and Testing • Causes for differing behavior • Sample the contention window before/after incrementing when initiating a backoff • Contention window increment while sending back-to-back packets • Detecting a busy medium (VCS vs. PCS) • Interface between MAC and higher layers

16. Results and Discussions • 100 simulation runs • GTNetS and GloMoSim have some overlap • ns-2 takes longest to finish its rounds • Later rounds finish significantly faster • Data ignored when packet originated by node 0 is dropped

17. Results and Discussions • Backoff behavior (Contention Window Sampling) • Average number of rounds completed per run/100

18. Results and Discussions • Complicated Scenarios still have considerable variations • Analyses/Corrections needed to verify if identified variations help bridge the differences • Undetermined variations ? • Identify sources of differences and effects of them in a big picture

19. Conclusions • Simple Experiments eliminating effects of mobility, path-loss and modulation choices. • Simple scenarios have been fixed to provide identical results. • Complicated scenarios with channel contention still have considerable variation

20. Questions ?