Introduction to Unix (CA263) File Processing (continued)

1. Introduction to Unix (CA263)File Processing (continued) By Tariq Ibn Aziz

2. Objectives • Use the pipe operator to redirect the output of one command to another command • Use the cut, paste, tr and grep command to search for a specified pattern in a file • Use the uniq command to remove duplicate lines from a file • Use the comm and diff commands to compare two files • Use manipulation and transformation commands, which include sed, tr, and pr • Design a new file-processing application by creating, testing, and running shell scripts Guide to UNIX Using Linux, Third Edition

3. Advancing YourFile-Processing Skills • Selection commands focus on extracting specific information from files Guide to UNIX Using Linux, Third Edition

4. Advancing YourFile Processing Skills (continued) Guide to UNIX Using Linux, Third Edition

5. Using the Selection Commands • Using the Pipe Operator – The pipe operator (|) redirects the output of one command to the input of another • The pipe operator can connect several commands on the same command line Guide to UNIX Using Linux, Third Edition

6. Cut Command • To extract various column or fields of data file or the output of the command. • Cut –cchars file • cut –c5- data • It will extract characters 5 through the end of the line of data and write the results to standard output. Guide to UNIX Using Linux, Third Edition

7. Cut Example $ who root console Feb 24 08:54 taziz tty02 Feb 24 12:55 dawn tty08 Feb 24 09:15 amin tty10 Feb 24 15:35 $ who | cut –c1-8 root taziz dawn amin Guide to UNIX Using Linux, Third Edition

8. Cut Example $ who root console Feb 24 08:54 taziz tty02 Feb 24 12:55 dawn tty08 Feb 24 09:15 amin tty10 Feb 24 15:35 $ who | cut –c1-8 | sort amin dawn root taziz Guide to UNIX Using Linux, Third Edition

9. Cut Example $ who root console Feb 24 08:54 taziz tty02 Feb 24 12:55 dawn tty08 Feb 24 09:15 amin tty10 Feb 24 15:35 $ who | cut –c10-16 console tty02 tty08 tty10 Guide to UNIX Using Linux, Third Edition

10. Cut Example $ who root console Feb 24 08:54 taziz tty02 Feb 24 12:55 dawn tty08 Feb 24 09:15 amin tty10 Feb 24 15:35 $ who | cut –c1-8,18- root Feb 24 08:54 taziz Feb 24 12:55 dawn Feb 24 09:15 amin Feb 24 15:35 Guide to UNIX Using Linux, Third Edition

11. Cut CommandThe –d and –f Option • The -d and –f option are used with cut when you have data that is delimited by a particular character. The format of the cut command is as follow: cut –ddchar –ffields file • Where dchar is the character that delimits each field of data, and fields specifies the field to be extracted from file. Field numbers start at 1. Guide to UNIX Using Linux, Third Edition

12. Cut CommandThe –d and –f Option cut –ddchar –ffields file $cat phonebook tariq:905-3456:11 Driscoll Dr kalim:205-3456:12 Driscoll Dr imran:304-3456:13 Driscoll Dr hasam:203-3456:14 Driscoll Dr $cut –d: –f1phonebook tariq kalim imran Hasam $ Guide to UNIX Using Linux, Third Edition

13. Cut Command • Field are separated by tabs. $cat phonebook Alice Chebba 596-2015 Barbara Swingle 205-9257 Jeff Goldberg 295-3378 Liz Stachiw 775-2298 $cut –c1-15phonebook Alice Chebba 59 Barbara Swingle Jeff Goldberg 2 Liz Stachiw 775 $ Guide to UNIX Using Linux, Third Edition

14. Cut Command • Field are separated by tabs, you should use the –f option to cut $cat phonebook Alice Chebba 596-2015 Barbara Swingle 205-9257 Jeff Goldberg 295-3378 Liz Stachiw 775-2298 $cut –f1phonebook Alice Chebba Barbara Swingle Jeff Goldberg Liz Stachiw $ Guide to UNIX Using Linux, Third Edition

15. Its opposite of cut command, cut break lines apart, and paste command put lines together. $ cat names Tubs Emanuel Lucy Ralph $ cat numbers (307)542-5356 (212)954-3456 (212)MH6-9959 (212)BE0-7741 $ paste names numbers Tubs (307)542-5356 Emanuel (212)954-3456 Lucy (212)MH6-9959 Ralph (212)BE0-7741 Paste Command Guide to UNIX Using Linux, Third Edition

16. Paste Command The –d option • Its opposite of cut command, cut break lines apart, and paste command put lines together. $ paste –d'+‘ names numbers Tubs+(307)542-5356 Emanuel+(212)954-3456 Lucy+(212)MH6-9959 Ralph+(212)BE0-7741 Guide to UNIX Using Linux, Third Edition

17. Paste Command The –s option • The –s option tells paste to paste together lines separated by tab from the same file, not from alternate files. $ paste –s names Tubs Emanuel Lucy Ralph Fred $ paste –d' ' –s - Tubs Emanuel Lucy Ralph Fred Guide to UNIX Using Linux, Third Edition

18. tr command • The tr filter is used to translate characters from standard output. The general form of the command is: trfrom-char to-char $ cat intro The UNIX operating system was pioneered by Ken $ tr e x < intro Thx UNIX opxrating system was pionxxrxd by Kxn Guide to UNIX Using Linux, Third Edition

19. tr command • You can translate colon into tab character to produce more readable output, by simply tacking tr command to the end of the pipeline. $ cut –d: -f1,6 /etc/passwd root:/ cron:/ bin:/ uucp:/usr/spool/uucp asg:/ $ cut –d: -f1,6 /etc/passwd |tr : ' ' root / cron / bin / uucp /usr/spool/uucp asg / Guide to UNIX Using Linux, Third Edition

20. Octal Values of some ASCII characters Character Octal value Bell 7 Backspace 10 Tab 11 NewLine 12 Linefeed 12 Formfeed 14 Carriage Return 15 Escape 33 $ date |tr ' ' '\12' Sun Mar 10 19:13:46 EST 1985 tr command Guide to UNIX Using Linux, Third Edition

21. tr command $ cat intro The UNIX operating system was pioneered by Ken $tr '[a-z]' '[A-Z]' < intro THE UNIX OPERATING SYSTEM WAS PIONEERED BY KEN Guide to UNIX Using Linux, Third Edition

22. tr commandThe –s Option • You can use the –s option to tr to “squeeze” out multiple occurances of characters will be replaced by a single character. $ cat lotsofspaces This is an example of a file that contains a lot of blank spaces. $ tr -s' ' ' ' < lotsofspaces This is an example of a file that contains a lot of blank spaces. Guide to UNIX Using Linux, Third Edition

23. tr commandThe –d Option • tr can also be used to delete single characters from a stream of input. $ tr –d ' ' < intro TheUNIXoperatingsystemwaspioneeredbyKen Guide to UNIX Using Linux, Third Edition

24. Using the grep Command • Used to search for a specific pattern in a file, such as a word or phrase • grep’s options and wildcard support allow for powerful search operations • You can increase grep’s usefulness by combining with other commands, such as head or tail command Syntax: grep [-options] [pattern] [filename] Useful options includes -i ignore case -l lists only file names -c counts the number of line instead of showing them -r searches through files under all subdirectories Guide to UNIX Using Linux, Third Edition

25. grep Example $ grep shell ed.cmd files, and is independent of the shell. to the shell, just type in a q. Guide to UNIX Using Linux, Third Edition

26. grep example $ cat phone_book Alice Chebba 596-2015 Barbara Swingle 598-9257 Jeff Goldberg 295-3378 Liz Stachiw 775-2298 Susan Goldberg 338-7776 Tony Iannino 386-1295 $ $ grep Susan phone_book Susan Goldberg 338-7776 Guide to UNIX Using Linux, Third Edition

27. grep –v options Example • Print all lines that don’t contain Unix $ cat intro The Unix operating system was pioneered by Ken Thompson. Main goal of Unix was to create Unix environment for efficient program development $ $ grep –v 'Unix' intro environment for efficient program development Guide to UNIX Using Linux, Third Edition

28. grep –i option Example • Print all lines by ignoring upper and lower case letters $ cat intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development $ $ grep –i 'Unix' intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX Guide to UNIX Using Linux, Third Edition

29. grep [tT] option Example • grep search for either upper or lower case letters T $ cat intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development $ $ grep '[tT]he intro The unix operating system was pioneered by Ken Guide to UNIX Using Linux, Third Edition

30. grep -i option Example • grep with –l just gives you a list of file that contains specified pattern $ cat intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development $ $ grep –l 'Main goal' * intro $ Guide to UNIX Using Linux, Third Edition

31. grep -n option Example • grep with –n gives you output of specified pattern with relative line number $ cat intro The unix operating system was pioneered by Ken Thompson. Main goal of UNIX was to create UNIX environment for efficient program development $ $ grep –n 'environment' intro 3:environment for efficient program development $ Guide to UNIX Using Linux, Third Edition

32. Using the uniq Command • Uniq command removes duplicate lines from a file • Compares only consecutive lines, therefore uniq requires sorted input • uniq has an option that allows you to generate output that contains a copy of each line that has a duplicate Syntax: uniq [-option] [file1 > file2] Useful options includes -u outputs only the lines of the source file that are not duplicated -d outputs only the copy of each lines that has a duplicate, and does not show unique line -i ignores case -c starts each line by showing the number of each instance Guide to UNIX Using Linux, Third Edition

33. $ cat parts muffler muffler shocks alternator battery battery radiator radiator coil spark plugs spark plugs coil $ uniq parts >inventory $ more inventory muffler shocks alternator battery radiator coil spark plugs coil $ Why coil is listed twice? Uniq Command Example Guide to UNIX Using Linux, Third Edition

34. $ cat parts muffler muffler shocks alternator battery battery radiator radiator coil spark plugs spark plugs coil $ uniq –u parts >inventory $ more inventory shocks alternator coil coil $ Why coil is listed twice? Ans: Two occurrences of coil are not together Uniq –u Command Example Guide to UNIX Using Linux, Third Edition

35. Using the comm Command • Used to identify duplicate lines in sorted files • Unlike uniq, it does not remove duplicates, and it works with two files rather than one • It compares lines common to file1 and file2, and produces three column output • Column one contains lines found only in file1 • Column two contains lines found only in file2 • Column three contains lines found in both files Guide to UNIX Using Linux, Third Edition

36. Using the diff Command • Attempts to determine the minimal changes needed to convert file1 to file2 • The output displays the line(s) that differ • Codes in the output indicate that in order for the files to match, specific lines must be added or deleted Guide to UNIX Using Linux, Third Edition

37. Using the pr Command toFormat Your Output • pr prints specified files on the standard output in paginated form • By default, pr formats the specified files into single-column pages of 66 lines • Each page has a five-line header containing the file name, its latest modification date, and current page, and a five-line trailer consisting of blank lines Guide to UNIX Using Linux, Third Edition

38. Using a Shell Script toImplement the Application • Shell scripts should contain: • The commands to execute • Comments to identify and explain the script so that users or programmers other than the author can understand how it works • Use the pound (#) character to mark comments in a script file Guide to UNIX Using Linux, Third Edition

39. Running a Shell Script • You can run a shell script in virtually any shell that you have on your system • The Bash shell accepts more variations in command structures that other shells • Run the script by typing sh followed by the name of the script, or make the script executable and type ./ prior to the script name Guide to UNIX Using Linux, Third Edition

40. Putting it All Together toProduce the Report • An effective way to develop applications is to combine many small scripts in a larger script file • Have the last script added to the larger script print a report indicating script functions and results Guide to UNIX Using Linux, Third Edition