input

2. input output MAC* (*) monitoring and control

3. Herding FPGAs or: distributed monitor and control harroverkouter Joint Institute for VLBI in Europe

7. 16x 10 Gbps ethernet 16x 10 Gbps ethernet

8. input output MAC*

9. UDP/IPv4 file server file server file server output MAC*

10. UDP/IPv4 UDP/IPv4 file server file server file server capture MAC*

11. UDP/IPv4 UDP/IPv4 file server file server file server capture UDP/IPv4 unb_control

12. UDP/IPv4 UDP/IPv4 file server file server file server capture UDP/IPv4 unb_control correlator_control

13. UDP/IPv4 UDP/IPv4 file server file server file server capture UDP/IPv4 unb_control correlator_control

14. UDP/IPv4 UDP/IPv4 Mark5ABC file server capture capture ...... ≤ 32 ≤ 4/Board ... UDP/IPv4 unb_control correlator_control

15. correlator_control • distributed ⇒ much communication • multiple UniBoards • up to 32 input data servers • multiple capture nodes • heterogeneous

16. Which language/system? • MarkIV correlator control s/w • archaic C++ (predates STL!) • home grown message passing • id. for distributed processing • Other C/C++ implementations • MPI library • Boost library • LOFAR control s/w • ...

17. Which language/system? • JAVA • XMLRPC or RMI for remote execution • no distributed multiprocessing • Python • multiprocessing module (threads, not machines) • message passing module

21. Write the control system from scratch in another language!

22. http://www.erlang.org • functional programming language • high level • compiles to byte code (cf. Python, JAVA) • developed by Ericsson AB since 1987 • designed for monitoring/control • designed for concurrency • designed to build distributed systems • open sourced in 1998

23. Functional goodies • immutable data • functions can return functions • functions can take functions as arguments • pattern matching + guard expressions • no for() or while() loops • less Lines Of Code • you have to think about your code

24. Functional goodies • immutable data • functions can return functions • functions can take functions as arguments • pattern matching + guard expressions • no for() or while() loops • less Lines Of Code • you have to think about your code code explodes!

25. Functional goodies • immutable data • functions can return functions • functions can take functions as arguments • pattern matching + guard expressions • no for() or while() loops • less Lines Of Code • you have to think about your code head explodes!

26. Pid = spawn(‘node@host’, Function, [Arg1, Arg2, ...]) • built-in execute function on remote node! • creates a new, concurrent, Erlang process • “almost, but not quite, ...” (no unix process) • lightweight – 106 processes/node easily • returned process identifier “Pid” is built-in type • encodes which process running where

27. Pid!Message, receive Message -> do_something() end • communication exclusively via messages • syntax elements: • operator! for sending • keywordreceive accepts incoming messages

28. V = 42 LE = <<V:32-unsigned-little>> BE = <<V:9-unsigned-big>> • binaries are a built-in type • arbitrary sizes of bit fields • syntax elements: • keywords<< and >> for creation • keyword: for specification of field width &cet.

29. {_,Code,_} = code:get_object_code(Module) rpc:call(‘node@host’, code, load_binary, [Code]) executescode:load_binary(Code) remotely, transfers the Code blob via the network

30. host A init() job0() job1() host Y host X writerN() reader0_job0() readerZ() writer1_job1() host Z process23() process42()

31. host A init() job0() job1() host Y host X writerN() reader0_job0() readerZ() writer1_job1() host Z process23() process42()

32. worker supervisor

33. {error, Reason} {error, Reason} {error, Reason} {error, Reason} worker supervisor

35. ccs.jive.nl fileserv0.jive.nl unb_ctl.jive.nl capture.jive.nl

36. ccs.jive.nl UNB = spawn(‘unb_ctl.jive.nl’, unb_control, [...]) unb_control setup_board() -> fpga:read(version), fpga:write(Config).

37. ccs.jive.nl UNB = spawn(‘unb_ctl.jive.nl’, unb_control, [...]) CAP = spawn(‘capture.jive.nl’, capturer, [...]) unb_control capturer receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl

38. ccs.jive.nl UNB = spawn(‘unb_ctl.jive.nl’, unb_control, [...]) CAP = spawn(‘capture.jive.nl’, capturer, [...]) SEND = spawn(‘fileserv0.jive.nl’, sender, [...]) unb_control sender capturer spawn(file_reader,..) receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl

39. ccs.jive.nl UNB = spawn(‘unb_ctl.jive.nl’, unb_control, [...]) CAP = spawn(‘capture.jive.nl’, capturer, [...]) SEND = spawn(‘fileserv0.jive.nl’, sender, [...]) unb_control sender capturer spawn(file_reader,..) receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl file_reader FD = open_file(Infile)

40. ccs.jive.nl UNB = spawn(‘unb_ctl.jive.nl’, unb_control, [...]) CAP = spawn(‘capture.jive.nl’, capturer, [...]) SEND = spawn(‘fileserv0.jive.nl’, sender, [...]) unb_control sender capturer receive {do,UTSecond} -> ...; end receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl file_reader receive {do,UTSecond} -> Frame = file:read(FD), udp:write(Frame) end

41. ccs.jive.nl CAP ! {start, OutfileName} unb_control sender capturer receive {do,UTSecond} -> ...; end receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl file_reader do_capture receive {do,UTSecond} -> Frame = file:read(FD), udp:write(Frame) end FD = file:open(Out)

42. ccs.jive.nl unb_control sender capturer receive {do,UTSecond} -> ...; end receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl file_reader do_capture receive {do,UTSecond} -> Frame = file:read(FD), udp:write(Frame) end receive {udp,Data} -> file:write(FD,Data) end

43. ccs.jive.nl case UNB ! fifo_level of ... unb_control sender capturer receive {do,UTSecond} -> ...; end receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl file_reader do_capture receive {do,UTSecond} -> Frame = file:read(FD), udp:write(Frame) end receive {udp,Data} -> file:write(FD,Data) end

44. ccs.jive.nl case UNB ! fifo_level of empty-> SND ! {do, 1426550400}; unb_control sender capturer receive {do,UTSecond} -> ...; end receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl file_reader do_capture receive {do,UTSecond} -> Frame = file:read(FD), udp:write(Frame) end receive {udp,Data} -> file:write(FD,Data) end

45. ccs.jive.nl case UNB ! fifo_level of empty-> SND ! {do, 1426550400}, UNB ! {do, 1426550400} unb_control sender capturer receive {do,UTSecond} -> ...; end receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl file_reader do_capture receive {do,UTSecond} -> Frame = file:read(FD), udp:write(Frame) end receive {udp,Data} -> file:write(FD,Data) end

46. Works like a charm! file_sender and do_capture pure Erlang • trivial to implement  • too slow  Erlang can interact with external (UNIX) processes • implement file_sender/do_capture in C

47. ccs.jive.nl case UNB ! fifo_level of empty-> SND ! {do, 1426550400}; unb_control sender capturer receive {do,UTSecond} -> ...; end receive {start,F} ->; stop -> ...; end receive fifo_level -> ...; {do,UTSecond} -> ...; end fileserv0.jive.nl cpp_sender cpp_capture fd = open(“..”, O_RDONLY) while( true ) { if( new_second ) { read(fd, buf, frame_sz); write(sok, buf, frame_sz); } fd = open(“.”, O_WRONLY) while( true ) { read(sok, buf, N); write(fd, buf, N); }

49. Everything you read on the internet about Erlang is true ...