Distributed Systems Concepts and Design Chapter 11: Time and Global States

1. Distributed Systems Concepts and Design Chapter 11: Time and Global States Steve Wallis, Raymond Ho, Bruce Hammer

2. 11.1: Introduction Time and Global States Time is a quantity we need to measure accurately Synchronization of Clocks Development Of Algorithms Relative Order Of Events Notion of Physical Time – this results from the need to know the order in which a pair of events or whether they occur simultaneously Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 2

3. 11.2: Clocks, events and Process States Model of interaction between the processes within a distributed system History of process Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 3

4. 11.2: Clocks, events and Process States Clocks Clocks are electronic devices Operating System reads the node’s hardware clock, scales it and adds an offset to produce a software clock In general, clock is not completely accurate Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 4

5. 11.2: Clocks, events and Process States Clock skew and clock drift Computer clocks tend not to be in perfect agreement Clock skew - Instantaneous difference between the readings of any two clocks is called their skew Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 5

6. 11.2: Clocks, events and Process States Clock drift – computer crystal-based clocks count time at different rates, and so diverge Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 6

7. 11.2: Clocks, events and Process States Coordinated Universal Time Synchronize computer clocks to external source of highly accurate time Such as atomic clock with drift rate is about one part in 1013 Output from these atomic clocks known as International Atomic Time Seconds and years and other time units are rooted in astronomical time Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 7

8. 11.2: Clocks, events and Process States Coordinated Universal Time (UTC) International standard for timekeeping Based on atomic time UTC signals are synchronized and broadcast regularly from land-based radio stations and satellites to many parts of the world Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 8

9. 11.3: Synchronizing physical clocks External synchronization - For process accountancy purposes, necessary to synchronize the processes’ clocks with an authoritative, external source of time Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 9

10. 11.3: Synchronizing physical clocks Internal synchronization – measure of the interval between two events occurring at different computers by appealing to their local clocks Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 10

11. 11.3: Synchronizing physical clocks Clocks not necessary synchronized internally and externally Common to define a hardware clock H to be correct if it drift rate falls with a known bound p>0, mean the error in measuring the interval between real times t and t’ (t’ >t) is bound Monotonicity – is the condition that a clock C only ever advances Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 11

12. 11.3: Synchronizing physical clocks Faulty – A clock that does not keep to whatever correctness condition apply Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 12

13. 11.3: Synchronizing physical clocks Synchronization in a synchronous system Synchronous System Has minimum transmission time min Also has maximum transmission time max So transmission time be u = (max - min) Optimum bound on clock skew when synchronizing N clocks is u(1 – 1/N) Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 13

14. 11.3: Synchronizing physical clocks Asynchronous System Found in most distributed systems Message delays are not bounded in their effect No upper bound max on message transmission delays T trans = min + x, where x>0, x is not known in a particular case, although a distribution of values may be measurable for a particular installation Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 14

15. 11.3: Synchronizing physical clocks Cristian’s method for synchronizing clocks Probabilistic: Algorithm achieves synchronization only if he observed round-trip times between client and server are sufficiently short compared with the required accuracy Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 15

16. 11.3: Synchronizing physical clocks Discussion of Cristian’s algorithm Cristian method suffers from the problem associated with all services implemented by a single server but the server might fail and thus render synchronization impossible temporarily Cristian suggested that time should be provided by a group of synchronized time servers and use only the first reply obtained Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 16

17. 11.3: Synchronizing physical clocks The Berkeley algorithm Algorithm from Gusella and Zatti for internal synchronization Developed for collections of computers running Berkeley UNIX A coordinator computer to act as the master This computer periodically polls the other computers whose clocks are to be synchronized, called slaves Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 17

18. 11.3: Synchronizing physical clocks Master estimates the local clock times by observing the round-trip times sent from each slave computer Master sends the amount by which each individual slaves clock requires adjustment The algorithm eliminates readings from faulty clocks Should the master fail, then another can be elected to take over and function as it predecessor Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 18

19. 11.3: Synchronizing physical clocks The Network Time Protocol (NTP) NTP [Mills 1995] defines an architecture for a time service and a protocol to distribute time information over the Internet Provide a service enabling clients across the Internet to be synchronized accurately to UTC. NTP employs statistical techniques for the filtering of timing data and discriminates between the quality of timing data from servers Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 19

20. 11.3: Synchronizing physical clocks To provide a reliable service that can survive lengthy losses of connectivity To enable clients to resynchronize sufficiently frequently to offset the rates of drift found in most computers To provide protection against interference with the time service, whether malicious or accidetal Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 20

21. 11.3: Synchronizing physical clocks NTP service is provided by a network of servers located across the Internet Primary servers are connected directly to a time source such as a radio clock receiving UTC Secondary servers are synchronized wit primary servers The servers are connected in a logical hierarchy called a synchronization subnet Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 21

22. 11.3: Synchronizing physical clocks Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 22

23. 11.3: Synchronizing physical clocks The subnet levels are called strata Primary servers occupy stratum 1 at the root Secondary servers are stratum 2, and so on The lowest-level (leaf) serves execute in users’ workstations The synchronization subnet can reconfigure as servers become unreachable or failures occur Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 23

24. 11.3: Synchronizing physical clocks NTP servers synchronize with one another in one of three modes: Multicast mode – intended for use on a high-speed LAN, can achieve only relatively low accuracies Procedure-call mode – one server accepts requests from other computers and reply with its timestamp, it is suitable for higher accuracies than multicast Symmetric mode – intended for use by the servers that supply time information in LANs and by the higher levels (lower strata) of synchronization subnet to achieve highest accuracies Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 24

25. 11.3: Synchronizing physical clocks In all modes, messages are delivers unreliably, using the standard UDP Internet transport protocol In procedure-call and symmetric modes, processes exchange pairs of messages Each message bears timestamps of recent message events Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 25

26. 11.3: Synchronizing physical clocks Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 26

27. 11.3: Synchronizing physical clocks NTP calculates an offset for each pair of messages sent between two servers Which is an estimate of the actual offset between the two clocks, a delay, which is the total transmission time for the two messages If the true offset of the clock at B relative to that at A is o, and if the actual transmission times for m and m’ are t ad t’ respectively then we have: Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 27

28. 11.3: Synchronizing physical clocks Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 28

29. 11.3: Synchronizing physical clocks In general, NTP server engages in message exchanges with several of its peers with data filtering NTP applies a peer-selection algorithm to examines the values obtained from exchanges with each of several peers. The output may cause a server to change the peer that it primarily uses for synchronization Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 29

30. 11.3: Synchronizing physical clocks NTP employs a phase lock loop model, which modifies the local clock’s update frequency in accordance with observations of its drift rate Steve Wallis, Raymond Ho, Bruce Hammer Bruce Hammer, Steve Wallis, Raymond Ho 30

31. 11.4: Logical Time and Logical Clocks Ordering Of Events For a single process, events are ordered uniquely by times on the local clock Clocks cannot be synchronized perfectly across a distributed system, so we cannot use general time to find out the order of a pair of events occurring within it. In the ordering of events, there are 2 obvious points If 2 events occurred at the same process pi (i=1,2,..,N) then they occurred in the order pi observes them. This is denoted by i Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 31

32. 11.4: Logical Time and Logical Clocks Ordering Of Events Whenever a message is sent between processes , the sending event always occurs before the receiving event. Happened Before This partial ordering was named the ‘happened before’ relation by Lamport Also known as casual ordering or potential casual ordering The ‘happened before’ relation is denoted by  Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 32

33. 11.4: Logical Time and Logical Clocks Happened Before (contd) Definitions of HB For any message m, send(m)  receive(m) , where send(m) is the sending event and receive(m) the receiving event If e, e` and e`` are 3 events where ee`, and e`e`` then ee`` HB does not imply actual causality, If event e1 happens before event e2 it does not imply e1 caused e2 If 2 events a and b are not related i.e they occur in different processes and there is no chain of messages between them, so neither ab or ba is true, then they are concurrent and denoted as a||b Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 33

34. 11.4: Logical Time and Logical Clocks Happened Before (contd) Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 34

35. 11.4: Logical Time and Logical Clocks Logical Clocks Happened Before ordering can be captured numerically by a mechanism called a logical clock Invented by Lamport A software counter with no relationship to any physical clock Each process pi keeps its own Logical Clock Li This is used to apply the Lamport Timestamp to the event Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 35

36. 11.4: Logical Time and Logical Clocks Logical Clocks (contd) The Lamport Timestamp is denoted by Li(e) To capture the happened before relation , processes update their Logical Clocks and transmit the values as follows: LC1: Li is incremented by 1 before each event is issued at process pi : Li = Li + 1 LC2: when process pi sends message m, it appends to the message the value t = Li On receiving message (m,t) process pj computes Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 36

37. 11.4: Logical Time and Logical Clocks Logical Clocks (contd) Lj = max (Lj, t) and applies LC1 before time stamping the event receive(m). For the happened before relation it then follows for 2 events a, b that a b => L(a) < L(b) Note : converse if not true : if L(b) > L(a) then it does not follow ab (they may be concurrent e.g a || b) Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 37

38. 11.4: Logical Time and Logical Clocks Logical Clocks Lamport Timestamps Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 38

39. 11.4: Logical Time and Logical Clocks Totally Ordered Logical Clocks Some pairs of distinct events, generated by different processes, have numerically equivalent Lamport Timestamps A total order of events – one for which all pairs of distinct events are ordered – can be created by taking into account the identifiers of the processes at which events occur. If e is the event at process pi, with local Timestamp Ti, then global logical timestamp is denoted as (Ti, i) Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 39

40. 11.4: Logical Time and Logical Clocks Vector Clocks Vector Clocks were developed by Mattern and Fidge to overcome the shortcoming of Lamport clocks That is if L(e`) > L(e) we cannot conclude that ee` A Vector Clocks for a system of N process is an array of N integers Each process pi keeps its own Vector Clock Vi , and like Lamport timestamps processes piggy back Vector timestamps to messages they send to other processes. Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 40

41. 11.4: Logical Time and Logical Clocks Vector Clocks (contd) Vector Clocks are updated as follows : VC1 Vi[j]=0 for i, j=1,2,…,N VC2: Vi is incremented by 1 before each event is issued at process pi : Vi [i] = Vi [i] +1 VC3: process pi includes the value t= Vi in every message it sends When a process pi receives a Vector timestamp in a message it sets Vi [j]=max (Vi [j],t[j]) for j=1,2,…,N Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 41

42. 11.4: Logical Time and Logical Clocks Vector Clocks Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 42

43. 11.4: Logical Time and Logical Clocks Vector Clocks (contd) Comparing Vector Timestamps : V = V′ if V[j]=V′ [j] for j=1,2,…,N V ≤ V′ if V[j] ≤ V′[j] for j=1,2,…,N V < V′ if V ≤ V′ ⋀ V ≠ V′ As in Logical Clocks, it follows that : For 2 events e, e′ that e e′ => V(e) < V(e′ ) But now the converse IS true If V(e) < V(e′ ) then e e′ Steve Wallis, Raymond Ho, Bruce Hammer Steve Wallis, Raymond Ho, Bruce Hammer 43

44. 11.5: Global States • Using Vector Timestamps • Reconstruct the State of a system thru logical sequencing of events across multiple processes • Find out the state of a process • Whether a particular property of the system or a process is true or has been true Steve Wallis, Raymond Ho, Bruce Hammer

45. 11.5: Global States • Examples • Distributed Garbage Collection – • Are there any references to an object anywhere in the distributed system? If no, reuse memory • Deadlock Detection • Are two processes just waiting for each other to send? • Distributed Termination Detection • Has a process terminated or is it merely waiting for a response? • Similar to Deadlock Steve Wallis, Raymond Ho, Bruce Hammer

46. 0 1 2 3 e e e e 1 1 1 1 p 1 m m 1 2 Physical p 2 time 0 1 2 e e e 2 2 2 Inconsistent cut Consistent cut Figure 11.9Cuts Steve Wallis, Raymond Ho, Bruce Hammer

47. 11.5: Global States • Slicing a distributed system to determine the system’s state or a process’s state • The vector times, as noted above, sequence the system’s messages and processes • Cut 1 is inconsistent because the receipt of the message by p2 is included but the sending by p1 is not Steve Wallis, Raymond Ho, Bruce Hammer

48. 11.5: Global States • Linearization or Consistent Run • An ordering of all the events in a cut or global history that is consistent with happened-before • Determine what state is reachable from sequence of events • Global State Predicate • Function that maps from the set of global states of processing in the system to true or false • Termination, deadlock, garbage Steve Wallis, Raymond Ho, Bruce Hammer

49. 11.5: Global States • Chandy and Lamport’s ‘snapshot’ algorithm • Goal to record a snapshot of channels and processes at any point in time • Uses a designated collector process • Each process records its state and its messages • Uses marker messages – initiates the snapshot • Marker receiving rule – obligates a process that has not recorded its state to do so • Marker sending rule – obligates a process to send a marker after they have recorded their state and before they send messages Steve Wallis, Raymond Ho, Bruce Hammer

50. actual execution e ,e ,... 0 1 recording S recording S init final begins ends S snap ' ' ' ' ' ,... pre-snap: e ,e ,...e post-snap: e ,e 0 1 R-1 R R+1 11.5: Global States (contd)Reachability between states in the snapshot algorithm • Sinit – the global state before the first process recorded its state • Ssnap – the constantly-changing global state • Sfinal – the global state when the snapshot terminates • Upper linearization is Sys; Lower linearization is Sys’ • Establishes the reachability relationship Steve Wallis, Raymond Ho, Bruce Hammer