1 / 55

Cluster Computing at IQSS

Cluster Computing at IQSS. Alex Storer, Research Technology Consultant. What is the RCE?. R esearch C omputing E nvironment For research in the social sciences Get an account! support@help.hmdc.harvard.edu. How do I access it?. How do I access it?. How do I access it?.

zinnia
Download Presentation

Cluster Computing at IQSS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cluster Computing at IQSS Alex Storer, Research Technology Consultant

  2. What is the RCE? • Research Computing Environment • For research in the social sciences • Get an account! support@help.hmdc.harvard.edu

  3. How do I access it?

  4. How do I access it?

  5. How do I access it?

  6. How do I access it?

  7. What are your needs? Many Processes Gigantic Process

  8. Gigantic Process RCE Powered

  9. Gigantic Process

  10. RCE Powered

  11. Many Processes Input Output

  12. Many Processes Input 1 Output 1 Input 2 Output 2 Input 3 Output 3 Input 4 Output 4 Input 5 Output 5

  13. Many Processes

  14. Example: Simulating • You have a model that takes 30 minutes to run and computes a result • You want to establish confidence intervals for this number by running it many times

  15. Example .submit file

  16. Example .submit file What command do I run?

  17. Example .submit file What arguments do I give to the command?

  18. Example .submit file What input do I give to the command?

  19. Example .submit file Where do I save the outputs?

  20. Example .submit file How many times do I run this?

  21. Example .submit file Unix Standard Input/Output /usr/bin/R --no-save --vanilla < simulate.R > out.1/usr/bin/R --no-save --vanilla < simulate.R > out.2. . . /usr/bin/R --no-save --vanilla < simulate.R > out.10

  22. How to submit a file condor_submitsimulate.submit

  23. Your out.0 file It's just everything that R writes to the screen from the script!

  24. Example: Simulating • Instead of using the out.$(Process) structure, you can save the data in your script • You cannot expect the Processes to complete in order! • You shouldn't write to the same file until all processes are complete • Instead of calling a script, use a function

  25. Example: Simulating (with a function) • procid is an input • We tell the function what to save and where to save it

  26. Example: submitting a function in batch Execute this command in R. Specifically, run the simfunction.R file which we defined on the previous slide.

  27. Example: submitting a function in batch Execute this command in R. Specifically, call the function sim.function with the input as $(Process).sim.function(0)sim.function(1) …

  28. Example: submitting a function in batch We are no longer using the standard input and standard output, so we can leave these blank.

  29. MATLAB Example • An example where we need to do the same thing to a number of data files and write out the results • Call a function which knows how to map the process ID to the data to load

  30. MATLAB function

  31. MATLAB function We will pass $(process) as the function input

  32. MATLAB function Try to load: data_0.mat data_1.mat etc.

  33. MATLAB function Compute the relevant result.

  34. MATLAB function Save the results as: result_0.mat result_1.mat etc.

  35. .submit File Example • The Arguments section is the most important, let's look at each piece individually

  36. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" Start the arguments with double quotes (")

  37. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" -nodisplaytells Matlab to not pop up the GUI

  38. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" -singleCompThread tells Matlab to use only one core (this is what condor expects)

  39. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" -r tells Matlab to execute whatever commands come next.

  40. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" Put the commands to run in single quotes.

  41. .submit File Example • Do NOT try to write your entire Matlab script in the submit file! • Some arguments must be executed before calling your script, however…

  42. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" The commands to Matlab will go on inside the single quotes They must be on a single line!

  43. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" setenv(''HOME'',''nfs/home/A/astorer''); cd(''/nfs/home/A/astorer/Work/outreach/matlab''); mytest($(PROCESS))

  44. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" setenv(''HOME'',''nfs/home/A/astorer''); cd(''/nfs/home/A/astorer/Work/outreach/matlab''); mytest($(PROCESS)) setenv is required for Matlab to load your local preferences. You must use two single quotesinstead of one single quote. Remember to set your own home directory, e.g. nfs/home/J/jdoe

  45. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" setenv(''HOME'',''nfs/home/A/astorer''); cd(''/nfs/home/A/astorer/Work/outreach/matlab''); mytest($(PROCESS)) Change to the directory that contains the script you want to run.

  46. .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" setenv(''HOME'',''nfs/home/A/astorer''); cd(''/nfs/home/A/astorer/Work/outreach/matlab''); mytest($(PROCESS)) Finally, run the function on the $(PROCESS) variable.

  47. Example submission • Because our function reads in data, we generate the data ahead of time • This is what is in our directory before submitting • (ls lists directory contents) • Notice that we count starting from 0!

  48. Example submission • Use condor_submit to submit the file • Depending on the job, this may take some time to complete!

  49. Example submission • Use condor_q <username> to check the status of your jobs • Use condor_rm <username> to clear your jobs.

  50. Example submission • Use condor_q <username> to check the status of your jobs • When this returns with no result, your jobs are complete.

More Related