and now: Unix Shell Quoting

and now:Unix Shell Quoting ( Based on http://www.grymoire.com/Unix/Quote.html )

Why is quoting a problem? • Some programs (like echo) might need special symbols in their inputs. • But UNIX shells also uses special symbols • The meanings of these symbols are not always the same between the shell and the command. • More to the point: even if the meanings were the same, the real problem is that you don’t want the UNIX shell to replace these special symbols before sending the input to grep (because grep is looking for those symbols.)

Resolving the conflicting meanings of special symbols • When you want to use a regular expression in a shell command, will the shell: • Do something special with the character? • Or passed it unchanged to the program? • The "$" character is a good example: • It could be the beginning of a shell variable name • It could be part of a regular expression. • If you need a regular expression, you must know: 1. Are any symbols in the expression also shell symbols? 2. The right way to quote such symbols, so as to pass them to the command without modifying by the shell

OK, so how do we quote them? • We have three special shell symbols to do choose from: Note: The ` symbol is different that the ' symbol. The ` symbol is not used for quoting.

The \ You can prevent the shell from interpreting a character by placing a backslash ("\") in front of it. Here is a script to delete files with an asterisk in their names: echo This script removes all files that echo contain an asterisk in the name. echo echo Are you sure you want to remove these files\? rm -i *\** This “\” was necessary because the “?” is also a shell symbol. Without the “\”, the program would look for all files that match the pattern "files?." If you had “filesA” and “filesB” then you would have (wrongly) gotten: Are you sure you want to remove these filesA filesB

The \ You can prevent the shell from interpreting a character by placing a backslash ("\") in front of it. Here is a script to delete files with an asterisk in their names: echo This script removes all files that echo contain an asterisk in the name. echo echo Are you sure you want to remove these files\? rm -i *\** This “\” was necessary because the “?” is also a shell symbol. Without the “\”, the program would look for all files that match the pattern "files?”. If you had “filesA” and “filesB” then you would have (wrongly) gotten: Are you sure you want to remove these filesA filesB

The \ The backslash is the "strongest" method of quotation. It works when every other method fails. If you want to place text on two or more lines for readability use the backslash as the last character on the line: % echo This could be \ a very \ long line\! But it was not This could be a very long line! But it was not % The “\” escapes (or quotes) the end of line character, so that it no longer has a special meaning. The other “\” escapes the exclamation point

The ' To quote several character at once, you can use backslashes: % echo a\ \ \ \ \ \ \ b This is ugly but works. It is easier to use pairs of quotation marks to indicate the start and end of the characters to be quoted: % echo 'a b' Inside the single quotes, you can use almost all shell symbols: % echo 'What the *heck* is a $ doing here???'What the *heck* is a $ doing here??? The single quotes should be used when you want the text left alone. (The "!" sometimes still needs a backslash before it.)

The " Sometimes you want a weaker type of quoting: one that leaves symbols like "*" or "?" alone, but still expands variables and does command substitution: % echo "Is your path $PATH?"Is your path /usr/local/bin:/usr/bin:…?% echo "Your current directory is `pwd`"Your current directory is /home/Cse Once you learn the difference between single quotes and double quotes, you will have mastered a very useful skill. It's not hard: The single quotes are stronger than double quotes. And the backslash is the strongest of all.

Quotes Within Quotes Having two types of quotes (three if you count the backslash) might seem confusing, but it provides you with the flexibility to express what you want to. You can put either type of quotes inside of the other. If you want to quote single quotes, use double quotes around it. To quote double quotes, use single quotes: % echo "Isn't it easy to get a single quote\?"Isn't it easy to get a single quote?% echo 'And I replied, "Double quotes are easy too."' And I replied, "Double quotes are easy too."%

Including identical quotes within quotes One problem people have is including the same quotes within quotes. Many expect the following to work: echo "The word for today is \"TGIF\"" ← What is the output?echo 'Don\'t quote me'← What is the output? People are confused by this, because we think of strings in programming languages like C. These quotes are different. • They just turn substitution on and off. • They do not indicate the starting and ending of a string. Consider: echo 'a'b'c' ← What is the output? This is broken up into three units. The first and last are quoted, and the middle is not. After quoting and substitution occurs, the three units are combined. The middle can be a variable, for instance: echo 'a'$HOME'b'

Including identical quotes within quotes One problem people have is including the same quotes within quotes. Many expect the following to work: echo "The word for today is \"TGIF\""echo 'Don\'t quote me' People are confused by this, because we think of strings in programming languages like C. These quotes are different. • They just turn substitution on and off. • They do not indicate the starting and ending of a string. Consider: echo 'a'b'c' ← What is the output? This is broken up into three units. The first and last are quoted, and the middle is not. After quoting and substitution occurs, the three units are combined. The middle can be a variable, for instance: echo 'a'$HOME'b'

Including identical quotes within quotes One problem people have is including the same quotes within quotes. Many expect the following to work: echo "The word for today is \"TGIF\"" echo 'Don\'t quote me' People are confused by this, because we think of strings in programming languages like C. These quotes are different. • They just turn substitution on and off. • They do not indicate the starting and ending of a string. Consider: echo 'a'b'c' This is broken up into three units. The first and last are quoted, and the middle is not. After quoting and substitution occurs, the three units are combined. The middle can be a variable, for instance: echo 'a'$PATH'b'

Including identical quotes within quotes Don't put quotes within the same quotes, instead combine or concatenate several units into one argument. Let me rephrase that. If you want to include a single quote in an argument that starts with a single quote, you must turn off the mechanism started by the single quote, and use a different quoting method: echo ' "Don" " ' t" Remember, the backslash is the strongest of all quoting mechanisms. You can quote anything with the backslash. This example quotes all three quote characters: % echo \'\"\\ '"\ You can always use the backslash to quote a character. However, within the single quote mechanism, "\'" does not "quote the quote." The proper way to do this is: % echo 'Don'\' 't do that'Don't do that

Including identical quotes within quotes Don't put quotes within the same quotes, instead combine or concatenate several units into one argument. Let me rephrase that. If you want to include a single quote in an argument that starts with a single quote, you must turn off the mechanism started by the single quote, and use a different quoting method: Remember, the backslash is the strongest of all quoting mechanisms. You can quote anything with the backslash. This example quotes all three quote characters: % echo \'\"\\ '"\ You can always use the backslash to quote a character. However, within the single quote mechanism, "\'" does not "quote the quote." The proper way to do this is: % echo 'Don'\' 't do that'Don't do that

Including identical quotes within quotes Don't put quotes within the same quotes, instead combine or concatenate several units into one argument. Let me rephrase that. If you want to include a single quote in an argument that starts with a single quote, you must turn off the mechanism started by the single quote, and use a different quoting method: echo ' "Don'" ' t" Remember, the backslash is the strongest of all quoting mechanisms. You can quote anything with the backslash. This example quotes all three quote characters: % echo \'\"\\ '"\ You can always use the backslash to quote a character. However, within the single quote mechanism, "\'" does not "quote the quote." The proper way to do this is: % echo 'Don'\' 't do that'Don't do that

Including identical quotes within quotes Don't put quotes within the same quotes, instead combine or concatenate several units into one argument. Let me rephrase that. If you want to include a single quote in an argument that starts with a single quote, you must turn off the mechanism started by the single quote, and use a different quoting method: echo ' "Don' " ' t" Remember, the backslash is the strongest of all quoting mechanisms. You can quote anything with the backslash. This example quotes all three quote characters: % echo \'\"\\ '"\ You can always use the backslash to quote a character. However, within the single quote mechanism, "\'" does not "quote the quote." The proper way to do this is: % echo 'Don'\' 't do that'Don't do that

Including identical quotes within quotes Just remember to match the quotes together when you mentally parse a shell script. This also works with double quotes: % echo "The quote for today is "\"TGIF\"The quote for today is "TGIF" Or, if you want to also turn off substitution for "TGIF:" % echo "The quote for today is "\""TGIF"\"The quote for today is "TGIF" (In this case, the answer comes out the same, because there are no special characters for substitution, anyway.)

Finding out if your quotes are wrong You may become confused about when to use the backslash and when not to. Q: So, how can you find out if you’re quoting correctly? A: By adding an "echo" before the command so that you can see: %echo fgrep 'He said, "She said, '"'Hello!'"\" file fgrep He said, "She said,'Hello!'" file % By putting the echo in the front, we don’t do the fgrep. Instead we are printing what the arguments to the fgrep would have actually been. If you are debugging a shell script, and you want to see what your script is doing, you can duplicate lines and insert an "echo" in front of the copies.

Finding out if your quotes are wrong You may become confused about when to use the backslash and when not to. Q: So, how can you find out if you’re quoting correctly? A: By adding an "echo" before the command so that you can see how it ends up: %echo fgrep 'He said, "She said, '"'Hello!'"\" file fgrep He said, "She said,'Hello!'" file % By putting the echo in the front, we don’t do the fgrep. Instead we are printing what the arguments to the fgrep would have actually been. If you are debugging a shell script, and you want to see what your script is doing, you can duplicate lines and insert an "echo" in front of the copies.

Checking without using an echo But, the quoting rules do vary (just a little bit) between shells, as the following slides will explain… set is a command that you use to define a variable. The tcsh shell has 2 variables that, when set, will help you follow the trail of variable and symbol expansions: set verbosewill echo every line of your script before the variables have been evaluated. set echowill display each line after the variables and meta-characters have been substituted. If you wish to turn the variables off, use unset instead of set

Comparing C-shell & bash? • Sometimes you’ll hear me say something about bash shell. • Like: “This thing here is C-shell syntax.” • Or: “bash does quotes better than C-shell.” • Well, I think that might make you curious: • Like: “Why does he mentioning bash?” • Answer – because is a more popular shell. • Or: “Well, then, why doesn’t he teach bash?” • Answer – because its syntax is uglier. • Or: “Just what is the difference between shells?” • Answer – I’ll show you some examples now…

Comparing C-shell & bash? • Sometimes you’ll hear me say something about bash shell. • Like: “This thing here is C-shell syntax.” • Or: “bash does quotes better than C-shell.” • Well, I think that might make you curious: • Like: “Why does he keep mentioning bash?” • Answer – Because bash is a more popular shell. • Or: “Well, then, why doesn’t he teach bash?” • Answer – because its syntax is uglier. • Or: “Just what is the difference between shells?” • Answer – I’ll show you some examples now…

Comparing C-shell & bash? • Sometimes you’ll hear me say something about bash shell. • Like: “This thing here is C-shell syntax.” • Or: “bash does quotes better than C-shell.” • Well, I think that might make you curious: • Like: “Why does he keep mentioning bash?” • Answer – Because bash is a more popular shell. • Or: “Well, then, why doesn’t he teach bash?” • Answer – Because its syntax is uglier. • Or: “Just what is the difference between shells?” • Answer – I’ll show you some examples now…

Family relationships among shells

Flavors of Unix Shells There are 2 main flavors of Unix Shells: Bourne (Standard Shell): sh, ksh, bash, zsh Faster Has a more consistent behavior C shell : csh, tcsh Easier to learn at first Has features that make it good working at the command prompt But, as you get more advanced, you begin to encounter weird features

Flavors of Unix Shells There are 2 main flavors of Unix Shells: Bourne (Standard Shell): sh, ksh, bash, zsh Faster Has a more consistent behavior C shell : csh, tcsh Easier to learn at first Has features that make it good working at the command prompt But, as you get more advanced, you begin to encounter weird features…

An example of one of csh’s weird features (in comparison to bash) In csh, this is an error: grep "$1$" helloworld.c Variable name must contain alphanumeric characters Instead you need: grep"$1"'$' Interestingly (and illogically) the $ inside of a "" is sometimes OK: grep"Allice$1"

An example of one of csh’s weird features (in comparison to bash) In csh, this is an error: grep "$1$" helloworld.c Variable name must contain alphanumeric characters Instead you need: grep"$1"'$' Interestingly (and illogically) the $ inside of a "" is sometimes OK: grep"Allice$1" This $ has special meaning to the grep command. (We will learn grep later today.)

An example of one of csh’s weird features (in comparison to bash) In csh, this is an error: grep "$1$" helloworld.c Variable name must contain alphanumeric characters The error message occurs because *this*is considered as a variable with no name. Instead you need: grep"$1"'$' Interestingly (and illogically) the $ inside of a "" is sometimes OK: grep"Allice$1"

In csh, this is an error: grep "$1$" helloworld.c Variable name must contain alphanumeric characters The error message occurs because *this*is considered as a variable with no name. Instead you need: grep"$1"'$' helloworld.c Interestingly (and illogically) the $ inside of a "" is sometimes OK: grep"Allice$1" An example of one of csh’s weird features (in comparison to bash)

In csh, this is an error: grep "$1$" helloworld.c Variable name must contain alphanumeric characters The error message occurs because *this* is considered as a variable with no name. Instead you need: grep"$1"'$' helloworld.c Interestingly (and illogically) the $ outside of the "" is also OK, if nothing comes after it: grep"Hello"$ helloworld.c An example of one of csh’s weird features (in comparison to bash) The illogicality here is that the "$" has a less-strict quoting than the $ does (in this case).

Searching for something in a filethe greps • The grep family is a collection of three related programs for finding patterns in files. Their names are grep, fgrep, and egrep. • The name grep has its origin in the phrase “Get Regular Expression and Print” • fgrep = “fixed string grep”, only searches for strings • grep = a regular-expression matcher • egrep = “extended grep” (  after the midterm )

Searching for something in a filefgrep Format: % fgrep [options] 'search string' filenames Interpretation: In the name fgrep, the f stands for "Fixed string." The fgrep program finds all the lines in a file that contain a certain fixed string. So, for example, I could find all occurrences of CA in the files in the current working directory simply by typing this command: % fgrep CA *

Whenfgrep is not enough Key limitations of fgrep you cannot use it to get approximate matches you cannot use it to get matches of more complicated patterns that cannot be described by just giving a fixed string. Sometimes you are not sure about the string you want for example, you might know only that the word you are seeking begins with z and ends with -ic, and had the sequence gm in it somewhere. What you need, then, is something more than fgrep You need a program that can understand a language in which you can say things like "begins with z and ends with -ic or -ics and had gm in it somewhere." You need grep, a searching program for regular expressions Reg xpressionoa 44

Whenfgrep is not enough Key limitations of fgrep you cannot use it to get approximate matches you cannot use it to get matches of more complicated patterns that cannot be described by just giving a fixed string. Sometimes you are not sure about the string you want for example, you might know only that the word you are seeking begins with z and ends with -ic, and had the sequence gm in it somewhere. What you need, then, is something more than fgrep You need a program that can understand a language in which you can say things like "begins with z and ends with -ic or -ics and had gm in it somewhere." You need grep, a searching program forregular expressions Reg xpressionoa 47

Whenfgrep is not enough key limitations of fgrep you cannot use it to get approximate matches you cannot use it to get matches of more complicated patterns that cannot be described by just giving a fixed string. Sometimes you are not sure about the string you want. for example, you might know only that the word you are seeking begins with z and ends with -ic, and had the sequence gm in it somewhere. What you need, then, is something more than fgrep You need a program that can understand a language in which you can say things like "begins with z and ends with -ic or -ics and had gm in it somewhere." You need grep, a searching program for regular expressions Reg. expressions are keyboard-based pattern specifications

The following slides are based upon:http://www.grymoire.com/Unix/Regular.html

Regular Expressions ^ (caret, as the first symbol in the regular expression) matches the expression only if it is at the start of a line, as in: ^A $ (dollar, as the last symbol in the regular expression) matches expression at the end of a line, as in: A$ \ (backslash) turns off any special meaning of the next character, as in: \^ [ ] (brackets) matches to any one of the enclosed characters, as in: [aeiou] • (hyphen, inside brackets) match to a range, as in: [0-9] ^ (caret, as the first symbol inside a bracket) matches any one character except those enclosed in the [ ], as in: [^0-9] . (period) matches to any one character, as in: ^.$ * (asterisk) matches to zero or more of the precedingcharacter or expression, as in: ^[^a-z]*$

and now: Unix Shell Quoting