Command Line Variations Selected Readings in Chapters 3, 5, 6

1. Command Line Variations Selected Readings in Chapters 3, 5, 6 • Project 1 kickoff • Streams & Redirection: Sections 5.2, 5.3 • Pipes and Tees: Sections 5.4, 5.5 • Filenames & Wildcards: Section 3.1 • Job Control: Section 5.10 • Filter Commands: details in Chapter 6 • cat, tr, head, tail, cut, paste, sort, • wc (word count)

2. Figure 5-4 Standard Streams

3. Standard I/O • Standard Output (stdout) labelled (1) above • default place to which programs write • Standard Input (stdin) labelled (0) above • default place from which programs read • Standard Error (stderr) labelled (2) above • default place where errors are reported • To demonstrate --cat • Echoes everything you typed in with an <enter> • Quits when you press Ctrl-d at a new line -- (EOF)

4. Figure 5-6 Redirecting Standard Output

5. Redirecting Standard Output • cat file1 file2 > file3 • concatenates file1 and file2 into file3 • file3 is created if not there • Any pre-existing file3 is clobbered if there • cat file1 file2 >> file3 • file3 is created if not there • file3 is appended to if it is there • cat > file3 • file3 is created from whatever user provides from standard input

6. Figure 5-7 Print who Output

7. Redirecting Standard Error • In bash (for shell scripts), standard error is redirected using2> • cat myfile > yourfile 2> yourerrorfile • A better example that generates errors: ls –R / recursively list all files from root you will get many screens of output • Redirect output into a file: ls –R / > allfiles save output from above you will get many “permission denied errors”more allfiles but you can look at listing in file now • Redirect errors to a second file: ls –R / > allfiles 2> errors errors go into file errors • If you don’t care about errors ls –R / > allfiles 2> /dev/null /dev/null is a ‘bit bucket’ where bits go to die!

8. Figure 5-5 Redirecting Standard Input

9. Redirecting Standard Input • terra[1] $ cat < oldfile > newfile • A more useful example: • terra[2] $ tr string1 string2 • Read from standard input. • Character n of string1 translated to character n of string2. • Results written to standard output. • Example of use: terra[3] $ tr aeoiu eoiua terra[4] $ tr a-z A-Z < file1 > file2

10. /dev/null • /dev/null • A virtual file that is always empty. • Copy things to here and they disappear. • cp myfile /dev/null • mv myfile /dev/null • Copy from here and get an empty file. • cp /dev/null myfile • Redirect error messages to this file • cat myfile > yourfile 2> /dev/null • Basically, all error messages are discarded.

11. Figure 6-1 Concept Of A Filter

12. Basic Filters • Filters are programs that: • Read stdin. • Modify it. • Write the results to stdout. • Filters typically do not need user input. • Example: • tr (translate): • Read stdin • Echo to stdout, translating some specified characters • Many filters can also take file names as operands for input, instead of using stdin.

13. Filtering using tr (translate command) • Using tr to translate what you type into upper case tr [a-z] [A-Z] (enter, type, enter etc. ctrl-D to quit) • Using tr to translate a file using input redirection tr [a-z] [A-Z] < foods (now foods will appear UC) tr [a-z] [A-Z] foods (also OK, tr accepts filename args) • Using tr to translate a file using input redirection and store results using output redirection tr [a-z] [A-Z] < foods > foodsCAP tr [a-z] [A-Z] foods > foodsCAP (also OK, see above) (now foodsCAP will hold UC output from foods)

14. More filters: grep, wc, sort (see Ch6) • grep patternstr • Read stdin and write lines with patternstr to stdout terra[1] $ grep "unix is easy" < myfile1 > myfile2 terra[1] $ grep "unix is easy" myfile1 > myfile2 (also OK) • Write all lines of myfile1 containing phrase unix is easy to myfile2 • wc • Count the number of chars/words/lines on stdin • Write the resulting statistics to stdout • Example: (wc < mars.txt OR wc mars.txt ) • sort • Sort all the input lines in alphabetical order and write to the standard output.

15. Figure 5-8 Piping Output to Next Command

16. Pipes • The pipe: • Connects stdout of one program with stdin of another • General form:command1 | command2 • stdout of command1 used as stdin for command2 • Example: terra[1] $ cat foods | grep apples | wc -l • A pipe works the same as (but more efficient than) terra[2] $ cat foods > tmp1 terra[3] $ grep apples < tmp1 > tmp2 terra[4] $ wc -l < tmp2 • Actually cat is redundant: grep apples foods | wc -l

17. Redirecting and Pipes (2) • Note: The name of a command always comes first on the line. • There may be a tendency to say: terra[1] $ foods > grep apples | wc -l • This is WRONG!!! • Your shell will go looking for a program named foods • To do it correctly, many alternatives! terra[1] $ cat foods | grep apples | wc -l terra[2] $ grep apples < foods | wc -l terra[3] $ grep apples foods | wc -l terra[4] $ grep -c apples foods

18. The tee Command (don’t worry about this) • tee - replicate the standard output • cat readme.txt | tee myfile stdin tee stdout myfile

19. Unix Filename Rules • Almost any character is valid in a file name • all the punctuation and digits • the one exception is the / (slash) character • the following are not encouraged • ? * [ ] “ ” ’ ( ) & : ; ! • the following are not encouraged as the first character • - ~ • control characters are also allowed, but are not encouraged • UPPER and lower case letters are different • A.txt and a.txt are different files

20. Unix Filename Extensions • No enforced extensions • The following are all legal Unix file names • a • a. • .a • … • a.b.c • Remember files beginning with dot are hidden • ls cannot see them, use ls -a • . and .. are reserved for current and parent directories

21. Unix Filename Extensions • Even though Unix doesn't enforce extensions, • “.” and an extension are still used for clarity • .jpg for JPEG images • .tex for LaTeX files • .sh for shell scripts • .txt for text files • .mp3 for MP3’s • some applications mayenforce their own extensions • Compilers look for these extensions by default • .c means a C program file • .C or .cpp or .cc for C++ program files • .h for C or C++ header files • .o means an object file

22. Unix Executable Files • Executable files usually have no extensions • cannot execute file a.exe by just typing a • telling executable files from data files can be difficult • “file” command Use: file filename Result: print the type of the file Example: terra[1] $ file ~/.bashrc .bashrc: executable bash-shell script • Filenames and pathnames have limits on lengths • 1024 characters typically • these are pretty long (much better than MS-DOS days and the 8.3 filenames)

23. Creates a file with name -i Fixing Filename Mistakes • It is very easy to get the wrong stuff into filenames • Say you accidentally typed terra[3] $ cp myfile -i • What if you type terra[4] $ rm -i • The shell thinks -i is an option, not a file • Getting rid of these files can be painful • There is an easy way to fix this... • You simply type terra[5] $ rm -- -i • Many commands use “--” to say there are no more options

24. Filename Wildcarding (See section 3.1) • Wildcarding is the use of “special” characters to represent or match a sequence of other characters • a short sequence of characters can match a long one • a sequence may also match a large number of sequences • Often use wildcard characters to match filenames • filename substitution – generally known as “globbing” • Wildcard characters * matches a sequence of zero or more characters • Example: a*.c* matches abc.c, abra.cpp, ?matches any single character • Example: a?.c matches ab.c, ax.c, but not abc.c [...]matches any one character between the braces • Example: b[aei]t matches bat, bet, or bit, not baet

25. Filename Wildcarding (2) • Wildcard sequences can be combined terra[6] $ mv a*.[ch] cfiles/ • mv all files beginning with a and ending with .c or .h into the directory cfiles terra[7] $ ls [abc]*.? • list files whose name begins with a, b, or c and ends with . (dot) followed by a single character • Wildcards do not cross "/" boundaries • Example: csnow*c does not match csnow/codec • Wildcards are expanded by the shell, and not by the program • Programmers of commands do not worry about searching the directory tree for matching file names • The program just sees the list of files matched

26. cat all files whose names begin with .c Filename Wildcarding (3) • Matching the dot • A dot ( . ) at • the beginning of a filename, or • immediately following a / must be matched explicitly. • Similar to the character / • Example: terra[8] $ cat .c* • As mentioned earlier, [....] matches any one of the characters enclosed • Within “[...]”, a pair of characters separated by “-” matches any character lexically between the two • Example: terra[9] $ ls [a-z]* lists all files beginning with a character between ASCII ‘a’ and ASCII ‘z’

27. Processes and Job Control

28. Figure 5-14 Job States

29. Foreground and Background (1) • Unix is a multi-tasking operating system • some of these tasks are being done by other users logged in • some are being done by you in the background • e.g. watching for incoming mail • When you run a task (a Unix command, like ls or vi) it executes in the foreground of your shell • it has the “control" of your screen and keyboard

30. Foreground and Background (2) • If you still want to use your shell command line • terra[1] $ a_heavy_task & • [1] 13607 • terra[2] $ • When you put a task in background • task keeps running, but you continue to work at the shell in the foreground • if any output is done, it appears on your screen immediately (can be confusing) • if input is required, process prints a message and stops • when it is done, a message will be printed

31. Example ls –R / (again) • Kick off a long command ls –R / > listall • Suspend, Ctrl-Z • Send to background bg [1] ls –R / >listall& • Oops, still get errors! Now you have to kill a background job: kill %1 (or kill -9 %1) [1] terminated • Restart it the right way, in background, with err redirect: ls –R / > listall 2> /dev/null &

32. Foreground and Background (3) • Explicit background processes are needed less often with windowing systems • Just go to another window and run the command • But explicit background processes are used often ... • A command needs a long time, you do not want to close that window by accident • Run a job at the background and logout • netscape& will open a new window, but leave the current shell window still available to use

33. A Simple Script • We use the following shell script to illustrate job control • Edit a file make_noise terra[1] $ cat > make_noise #!/bin/sh while true do date sleep 1 done ^D terra[2] $ chmod u+x make_noise • make_noise then is a shell script repeats to print the time for every second, until you terminate it using Ctrl-c.

34. Job Control – Suspending Jobs • csh, tcsh, and bash allow you to manage the running of different processes • Suspending jobs • the Ctrl-z special character stops the job terra[1] $ make_noise Fri May 16 14:14:43 EDT 2003 …… ^Z Suspended terra[2] $ vi readme ^Z

35. Job Control - Monitoring Jobs • The "jobs" command shows which of your jobs are running and/or stopped. terra[3] $ jobs [1] + Suspended make_noise [2] + Suspended vi readme • Here there are two suspended processes, the make_noise and a vi process.

36. Job Control – Resuming Jobs • Putting jobs back into the foreground: • Use the "fg" command to move a job into the foreground. terra[4] $ fg %2 • Puts job number 2 into the foreground. • Works with either a background or stopped job. • Putting jobs into the background: terra[5] $ bg %1

37. Job Control – Killing Jobs • Jobs can also be killed • Use the Unix "kill" command terra[6] $ kill %1 or if it won't die ... terra[7] $kill –9 %1 • Jobs can be stopped and continued terra[8] $ a_heavy_task & terra[9] $ stop %1 terra[10] $ bg %1

38. Using ps (1) • Jobs are really just a special case of Unix processes • ps can list your current processes terra[11] $ ps PID TT S TIME COMMAND 2312 pts/0 T 0:00 vi 2296 pts/0 R 0:00 tcsh 2313 pts/0 R 0:00 ps

39. Using ps (2) • The ps command takes a number of options • -l gives you a long listing of what is going on • -u loginid tells you about loginid's processes • use man ps to see more options • kill pid kills the process pid • TERM signal will be sent to the process pid • kill -9 or kill -KILL will send the KILL signal • Use man –s 5 kill to find out more signals