1 / 43

Parallel and Distributed Programming

Parallel and Distributed Programming. Kashif Bilal. Parallel and Distributed Programming. . Generally We have two types of Distributed and parallel Systems. Multiprocessors ( Shared memory System) Having single memory accessible to every processor Multicomputers ( Message Passing System)

conroy
Download Presentation

Parallel and Distributed Programming

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Parallel and Distributed Programming Kashif Bilal

  2. Parallel and Distributed Programming. • Generally We have two types of Distributed and parallel Systems. • Multiprocessors ( Shared memory System) • Having single memory accessible to every processor • Multicomputers ( Message Passing System) • Every Processor has its own local memory.

  3. Parallel Programming Architectures • Generally we have two basic architectures for parallel programming • Supervisor Worker Structure (Master Slave model) • Hierarchy Structure (Tree Model )

  4. Supervisor Worker Model • There is only one level of hierarchy in this structure • one supervisor and many workers. • Supervisor • It normally interacts with the user • activates the workers • assigns work to the workers • collects results from the workers. • Workers perform calculations and send result back to supervisor

  5. Hierarchy Structure • The hierarchy structure allows the workers to create new levels of workers. • The top-level supervisor is the initiating task, which creates a set of workers at the second level. • These workers may create other sets of workers.

  6. Process ….. • A program (task) in execution. • A process is a set of executable instructions (program) which runs on a processor. • Process is basic entity to achieve parallelism in both multiprocessors and multicomputers.

  7. Distributes System Classification • John Flynn classified computers in four categories. • SISD – Single Instruction stream/Single Data stream • SIMD – Single Instruction stream/Multiple Data stream OR SPMD – Single Program/Multiple Data stream • MISD – Multiple Instruction stream/Single Data stream (No real application) • MIMD – Multiple Instruction stream/Multiple Data stream.

  8. Message Passing • The method by which data from one processor’s memory is copied to the memory of another processor. • In distributed memory systems, data is generally sent as packets of information over a network from one processor to another.

  9. Thumb Recognition Example • Suppose we have a database of 10 lack thumb impressions and their related data. • A user comes, system takes ones impression and searches the database for ones information. • Suppose one database match take 1/100 seconds. • To search the complete database, system will take approx. 10000 seconds. i.e. 2.7 hours.

  10. Algorithm for Thumb Recognition Main() { Picture =Capture the thumb impression to be matched Details = Match( picture ) } Match(Picture) { Pick record from database one by one and match it with Picture If (matched) return Details of that record }

  11. Distributed Algorithm… Main() { Receive impression, start record number and end record number from master. Details = Match( picture, start_no, end_no ) Send details to Master or supervisor code. } Match( Picture , start_no, end_no) { Pick record from database one by one and match it with Picture from record no start_no till end_no. If (matched) return Details of that record }

  12. Problem… • How to receive data from supervisor to workers. • How send details back to supervisor process. • How supervisor allocate and communicate with workers. • Etc……

  13. Possible Solutions • Make a new programming language for programming in distributed and parallel. • Change existing languages. • Build libraries and API’s( Set of functions) to perform all tasks related to Distributed and parallel programming like remote spawning to task, communication, synchronization etc.

  14. PVM and MPI • PVM and MPI are two famous libraries used for parallel and distributed programming. • PVM is a bit older than MPI. • MPI is now considered as standard for building parallel and distributed programs. • Both libraries provide different functions to perform different tasks required in parallel programs.

  15. PVM (Parallel Virtual Machine) • Started as a research project in 1989 • The Parallel Virtual Machine (PVM) was originally developed at Oak Ridge National Laboratory and the University of Tennessee. • The PVM offers a powerful set of process control and dynamic resource management functions.

  16. PVM….. • It provides programmers with a library of routines for • Initiation and termination of tasks. • Synchronization. • Alteration of the virtual machine configuration. • Facilitates message passing via a number of simple constructs. • Interoperability among different heterogeneous computers. • Programs written for some architecture can be copied to another architecture, compiled and executed without modification.

  17. PVM… • PVM has two components: • A library of PVM routines • A daemon that should reside on all the hosts in the virtual machine. • The pvmd serves as a message router and controller. It provides a point of contact, authentication, process control, and fault detection. • One task (process) normally instantiated on one machine (Supervisor), and other tasks as instantiated automatically by supervisor.

  18. Task Creation… • A task in PVM can be started manually or can be spawned from another task. • The initiating task is always activated manually by simply running its executable code on one of the hosts. • Other PVM tasks can be created dynamically from within other tasks. • The function pvm_spawn() is used for dynamic task creation. • The task • that calls the function pvm_spawn() is referred to as the parent and the newly created • tasks are called children.

  19. To Create a child, you must specify.. • The machine on which the child will be started • A path to the executable file on the specified machine • The number of copies of the child to be created • An array of arguments to the child tasks

  20. Pvm_spawn()… • Num=pvm_spawn (Child, Arguments, Flag, Where, How Many, &Tids) • This function has six parameters and returns the actual number of the successfully created tasks in the variable num. • Child: Name of task (process) to be executed. • Arguments : Same as command line arguments. • Flag : A flag value decides what machine will run the spawned task.

  21. Flag values • PvmTaskDefault • 0 PVM can choose any machine to start task • PvmTaskHost • 1 where specifies a particular host • PvmTaskArch • 2 where specifies a type of architecture • Where : Depends on value of flag. • How Many : Number of tasks to be spawned. • Tids: Array to store Tid’s of spawned tasks. • Return : Total number of tasks created.

  22. n1 = pvm_spawn(“simple_pvm”, 0, 1, “mynode11”, 1, &tid1) • Will create 1 tasks and run executables named simple_pvm on node mynode. • numt = pvm_spawn( “simple_pvm”, 0, PvmTaskArch, “LINUX", 5, &tids); • Will create 5 tasks running simple_pvm onspecific architecture i.e. Linux. • res = pvm_spawn(“simple_pvm”, NULL, PvmTaskDefault, "", 5, children); • Will ask pvm to choose nodes itself.

  23. Task Id’s • All PVM tasks are identified by an integer task identifier • When a task is created it is assigned a unique identifier (TID) • Task identifiers can be used to identify senders and receivers during communication. • It can also be used to assign functions to different tasks based on their TIDs

  24. Retrieval Of Tid • Task’s TID  pvm_mytid() Mytid = pvm_mytid; • Child’s TID  pvm_spawn() pvm_spawn(…,…,…,…,…, &tid); • Parent’s TID  pvm_parent() my_parent_tid = pvm_parent(); • Daemon’s TID  pvm_tidtohost() daemon_tid = pvm_tidtohost(id);

  25. Communication among Tasks • Communication among PVM tasks is performed using the message passing approach, • Achieved using a library of routines and a daemon. • User program communicates with the PVM daemon • Daemon determines the destination of each message. • Communication is generally asynchronous.

  26. Sending Task Receiving Task User application Library User application Library 5 1 4 8 6 2 3 7 Daemon Daemon

  27. How to send a message • Sending a message requires 3 steps. • A send buffer must be initialized. • The message is packed into the buffer. • The completed message is sent to its destination(s). • Receiving of message requires 2 steps • The message is received. • The received items are unpacked.

  28. Message Sending… • Buffer Creation (before packing) Bufid = pvm_initsend(encoding_option) Bufid = pvm_mkbuf(encoding_option) Encoding option Meaning 0 XDR 1 No encoding 2 Leave data in place

  29. Message Sending… • Data Packing pvm_pk*() • pvm_pkstr() – one argument pvm_pkstr(“This is my data”); • Others – three arguments • Pointer to the first item • Number of items to be packed • Stride pvm_pkint(my_array, n, 1);

  30. Message Sending… • Point to point (one receiver) info = pvm_send(tid, tag) • broadcast (multiple receivers) info = pvm_mcast(tids, n, tag) info = pvm_bcast(group_name, tag) • Pack and Send (one step) info = pvm_psend(tid, tag, my_array, length, data type) • The call returns integer status code info. A negative value of info indicates an error.

  31. Receiving a Message • PVM supports three types of message receiving functions: blocking, nonblocking, and timeout. • Blocking bufid = pvm_recv (tid, tag) -1  wild card in either tid or tag • Nonblocking bufid = pvm_nrecv (tid, tag) bufid = 0 (no message was received) • Timeout bufid = pvm_trecv (tid, tag, timeout) bufid = 0 (no message was received)

  32. Data Unpacking pvm_upk*() • pvm_upkstr() – one argument pvm_upkstr(string); • Others – three arguments • Pointer to the first item • Number of items to be unpacked • Stride pvm_upkint(my_array, n, 1);

  33. Data Unpacking • pvm_upk*() • pvm_upkstr() – one argument pvm_upkstr(string); • Others – three arguments • Pointer to the first item • Number of items to be unpacked • Stride pvm_upkint(my_array, n, 1); • Receiving and unpacking in single step • Info = pvm_precv(tid, tag, my_array, len, datatype, &src,&atag, &alen)

  34. Work Assignment (different programs) info1 = pvm_spawn(“worker1”, 0, 1, “lpc01”, 1, &tid1) info2 = pvm_spawn(“worker2”, 0, 1, “lpc02”, 1, &tid2) info3 = pvm_spawn(“worker3”, 0, 1, “lpc03”, 1, &tid3) info4 = pvm_spawn(“worker4”, 0, 1, “lpc04”, 1, &tid4)

  35. Distributed Algorithm… Main() { Receive impression, start record number and end record number from master. Details = Match( picture, start_no, end_no ) Send details to Master or supervisor code. } Match( Picture , start_no, end_no) { Pick record from database one by one and match it with Picture from record no start_no till end_no. If (matched) return Details of that record }

  36. Distributed Algorithm using PVM • Master Main() {Input thumb Impression… Int arr[2]; Int start=1; Pvm_spawn(“slave”,0,0,””,100,children); Pvm_init_send(XDR); for(i=0;i<100;i++) { arr[0]=start; arr[1]=start+999; pvm_pkint(arr,2,1); pvm_send(children[i],) Start+=1000; Pvm_send(picture); } Pvm_recv(-1,-1); }

  37. Distributed Algorithm using PVM • Slave Main() { Int arr[2]; Pvm_recv(pvm_parent(),1); Pvm_upkint(arr,2,1); Pvm_recv(pvm_parent(),-1); Match(picture,arr[0],arr[1]); } Match( Picture , start_no, end_no) { Pick record from database one by one and match it with Picture from record no start_no till end_no. If (matched) return Details of that record// pvm_send(pvm_parent,2); }

  38. Distributed Algorithm using PVM • Master and slave both in single program Main() {int tid = pvm_parent(); if(tid=PvmNoParent) { Write supervisor code Here..} Else { Write worker code here.. } }

  39. PVM Master Program #include <stdio.h> #include <pvm3.h> int main() { int myTID; int x = 12; int children[10]; int res; myTID = pvm_mytid(); printf("Master: TID is %d\n", myTID); res = pvm_spawn("slave", NULL, PvmTaskDefault, "", 1, children); if (res<1) { printf("Master: pvm_spawn error\n"); pvm_exit(); exit(1); }

  40. pvm_initsend(PvmDataDefault); pvm_pkint(&x, 1, 1); pvm_send(children[0], 1); pvm_recv(-1, -1); pvm_upkint(&x, 1, 1); printf("Master has received x=%d\n", x); pvm_exit(); return 0; }

  41. How to Compile and Execute? • gcc -I /opt/pvm3/include myprogram.c -L /opt/pvm3/lib/LINUX/ -lpvm3 -o myprogramexe • Illustration: • gcc = C Compiler • -I = Include • opt/pvm3/include = Path (include files) • myprogram = Name of source code file • -L = search path info as well as the pm3 lib • -o = output file

More Related