19,614
社区成员
发帖
与我相关
我的任务
分享分享国外Perl One-Liners、awk、sed经典，转自Peteris Krumins' blog
What are Perl One-Liners?
Perl one-liners are small and awesome Perl programs that fit in a single line of code and they do one thing really well. These things include changing line spacing, numbering lines, doing calculations, converting and substituting text, deleting and printing certain lines, parsing logs, editing files in-place, doing statistics, carrying out system administration tasks, updating a bunch of files at once, and many more.
Example 1: Replace a string in multiple files at once
perl -p -i.bak -e 's/Config/config/g' conf1 conf2 conf3
Suppose you have 3 configuration files, and you discover that you made a mistake and need to replace all occurrences of Config with config. This one-liner does just that. It executes the s/Config/config/g that replaces all occurrences of Config with config on all lines. And since you're smart about it, you always do -i.bak to make backup files in case something goes wrong.
I explain the -i, -p, and -e arguments in the e-book in great detail.
Example 2: Generate a random 8 character password
perl -le 'print map { ("a".."z")[rand 26] } 1..8'
This one-liner generates and prints a random 8 character password. It uses the list range operator .. operator to produce all strings from "a" to "z", which is the alphabet. Then a random letter is chosen by rand 26 and this operation is repeated 8 times.
Example 3: URL-escape a string
perl -MURI::Escape -lne 'print uri_escape($string)'
Here we use the URI::Escape module from CPAN. It exports the uri_escape function that does URL-escaping.
You can install this module from CPAN by running perl -MCPAN -e'install URI::Escape' on the command line.
I have this one-liner as an alias actually for both URL-escaping and unescaping URL-escaping as it's such a common thing to do:
urlescape () { perl -MURI::Escape -lne 'print uri_escape($_)' <<< "$1" }
urlunescape () { perl -MURI::Escape -lne 'print uri_unescape($_)' <<< "$1"; }
Then I can do this in the shell:
$ urlescape "http://www.catonmat.net"
http%3A%2F%2Fwww.catonmat.net
$ urlunescape http%3A%2F%2Fwww.catonmat.net
http://www.catonmat.net
Very useful!
Example 4: Print all lines from line 17 to line 30
perl -ne 'print if 17..30'
Here we use the binary flip-flop operator .. that becomes true when the input line number is 17, stays true while the line number is less than or equal to 30, and then becomes false. Combining the flip-flop operator with print if makes it print only lines 17-30.
Example 5: Remove all consecutive blank lines, leaving just one
perl -00pe0
I included this one-liner here in the examples just to show you how funny and obscure one-liners can get. This one-liner deletes all repeated blank lines from the input or from the given file. It does it by enabling the paragraph slurp mode through -00 command line argument, which reads the input paragraph-by-paragraph, rather than line-by-line, and prints the paragraphs. This way any number of blank lines between the paragraphs get ignored.
What is awk?
Awk is this awesome, little program that's present on nearly ever Unix machine. It's designed to carry out various text processing tasks easily, such as numbering lines, replacing certain words, deleting and printing certain lines.
Let's take a look at several examples.
Example 1: Print the second column from a file
awk '{ print $2 }'
That's all there is to it. Awk automatically splits each line into columns and puts each column in variables $1, $2, $3, etc. This one-liner prints just the 2nd column, which is in variable $2.
You can also specify the symbol or word that you wish to split on with the -F command line switch. This switch is explained in more details in the e-book and in the last example below.
Example 2: Number lines in a file
awk '{ print NR ": " $0 }' file
The whole line itself goes into variable $0. This one-liner prints it but prepends the NR special variable and a colon ": " before it. The special variable NR always contains the current line number.
There are many other special variables and they're all explained in the e-book and summarized in the appendix.
Example 3: Count the number of words in a file
awk '{ total = total + NF } END { print total }'
Here another special variable is used. It's the NF that stands for number of fields, or number of columns, or number of words in the current line. This one-liner then just sums the total number of words up and prints them before quitting in the END block.
Example 4: Print only lines shorter than 64 characters
awk 'length < 64'
This one-liner uses the length function to determine the length of the current line. If the current line is less than 64 characters in length, then length < 64 evaluates to true that instructs awk to print the line.
Finally, let's take a look at an example that compares an Awk program with an equivalent C program. Suppose you want to print the list of all users on your system. With Awk it's as simple as this one-liner:
awk -F: '{ print $1 }' /etc/passwd
This one-liner says, "Take each line from /etc/passwd, split it on the colon and print the first field of each line." Very straightforward and easy to write once you know Awk!
Suppose you didn't know Awk. Then you'd have to write it in some other language, such as C. Compare the example above with the example in C language:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MAX_LINE_LEN 1024
int main () {
char line [MAX_LINE_LEN];
FILE *in = fopen ("/etc/passwd", "r");
if (!in) exit (EXIT_FAILURE);
while (fgets(line, MAX_LINE_LEN, in) != NULL) {
char *sep = strchr(line , ':');
if (!sep) exit (EXIT_FAILURE);
*sep = '\0';
printf("%s\n", line);
}
fclose(in);
return EXIT_SUCCESS ;
}
This is much longer and you have to compile the program first, only then you can run it. If you make any mistakes, you have to recompile it again. That's why one-liners are called one-liners! They are short, easy to write and they do one thing really well. I am pretty sure you're starting to see how mastering Awk and one-liners can make you much more efficient when working in the shell and with text files in general.
What's sed?
Sed is the superman of UNIX stream editing. It's a small utility that's present on every UNIX system and it transforms one stream of text into another. Let's take a look at several practical examples that sed can carry out easily.
Example 1: Replace "lamb" with "goat" on every line
sed 's/lamb/goat/'This one-liner uses the famous s/.../.../ command. The s command substitutes the text in the first part of the command with the text in the second part. In this one-liner it replaces "lamb" with "goat".
A very detailed explanation of how sed reads the lines, how it executes the commands and how the printing happens is presented in the freely available introduction chapter. Please take a look.
Example 2: Replace only the second occurrence of "lamb" with "goat" on every line
sed 's/lamb/goat/2'Sed is the only tool that I know that takes a numeric argument to the s command. The numeric argument, in this case 2, specifies which occurrence of the text to replace. In this example only the 2nd occurrence of "lamb" gets replaced with "goat".
Example 3: Number the lines in a file
sed = file | sed 'N; s/\n/: /'This one-liner is actually two one-liners. The first one uses the = command that inserts a line containing the line number before every original line in the file. Then this output gets piped to the second sed command that joins two adjacent lines with the N command. When joining lines with the N command, a newline character \n is placed between them. Therefore it uses the s command to replace this newline \n with a colon followed by a space ": ".
So for example, if the file contains lines:
hello world
good job
sunny day
Then after running the one-liner, the result is going to be:
1: hello world
2: good job
3: sunny day
Example 4: Delete every 2nd line
sed 'n;d'This one-liner uses the n command that prints the current line (actually the current pattern space, see the introduction chapter for in-depth explanation), deletes it, and reads the next line. Then sed executes the d command that deletes the current line without printing. This way the 1st line gets printed, the 2nd line gets deleted, then the 3rd line gets printed again, then the 4th gets deleted, etc.
Example 5: ROT 13 encode every line
sed '
y/abcdefghijklmnopqrstuvwxyz/nopqrstuvwxyzabcdefghijklm/
y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/NOPQRSTUVWXYZABCDEFGHIJKLM/
'
Here the y/set1/set2/ command is used. The y command substitutes elements in the set1 with the corresponding elements in the set2. The first y command replaces all lowercase letters with their 13-char-shifted counterparts, and the second y command does the same for the uppercase letters. So for example, character a gets replaced by n, b gets replaced by o, character Z gets replaced by M, etc.
Sed is actually very powerful. It's as powerful as a Turing machine, meaning you can write any computer program in it. Check out these programs written in sed. Run them as sed -f file.sed:
转自Peteris Krumins' blog
Perl one-liners are small and awesome Perl programs that fit in a single line of code and they do one thing really well. These things include changing line spacing, numbering lines, doing calculations, converting and substituting text, deleting and printing certain lines, parsing logs, editing files in-place, doing statistics, carrying out system administration tasks, updating a bunch of files at once, and many more.
Example 1: Replace a string in multiple files at once
perl -p -i.bak -e 's/Config/config/g' conf1 conf2 conf3
Suppose you have 3 configuration files, and you discover that you made a mistake and need to replace all occurrences of Config with config. This one-liner does just that. It executes the s/Config/config/g that replaces all occurrences of Config with config on all lines. And since you're smart about it, you always do -i.bak to make backup files in case something goes wrong.
I explain the -i, -p, and -e arguments in the e-book in great detail.
Example 2: Generate a random 8 character password
perl -le 'print map { ("a".."z")[rand 26] } 1..8'
This one-liner generates and prints a random 8 character password. It uses the list range operator .. operator to produce all strings from "a" to "z", which is the alphabet. Then a random letter is chosen by rand 26 and this operation is repeated 8 times.
Example 3: URL-escape a string
perl -MURI::Escape -lne 'print uri_escape($string)'
Here we use the URI::Escape module from CPAN. It exports the uri_escape function that does URL-escaping.
You can install this module from CPAN by running perl -MCPAN -e'install URI::Escape' on the command line.
I have this one-liner as an alias actually for both URL-escaping and unescaping URL-escaping as it's such a common thing to do:
urlescape () { perl -MURI::Escape -lne 'print uri_escape($_)' <<< "$1" }
urlunescape () { perl -MURI::Escape -lne 'print uri_unescape($_)' <<< "$1"; }
Then I can do this in the shell:
$ urlescape "http://www.catonmat.net"
http%3A%2F%2Fwww.catonmat.net
$ urlunescape http%3A%2F%2Fwww.catonmat.net
http://www.catonmat.net
Very useful!
Example 4: Print all lines from line 17 to line 30
perl -ne 'print if 17..30'
Here we use the binary flip-flop operator .. that becomes true when the input line number is 17, stays true while the line number is less than or equal to 30, and then becomes false. Combining the flip-flop operator with print if makes it print only lines 17-30.
Example 5: Remove all consecutive blank lines, leaving just one
perl -00pe0
I included this one-liner here in the examples just to show you how funny and obscure one-liners can get. This one-liner deletes all repeated blank lines from the input or from the given file. It does it by enabling the paragraph slurp mode through -00 command line argument, which reads the input paragraph-by-paragraph, rather than line-by-line, and prints the paragraphs. This way any number of blank lines between the paragraphs get ignored.
What is awk?
Awk is this awesome, little program that's present on nearly ever Unix machine. It's designed to carry out various text processing tasks easily, such as numbering lines, replacing certain words, deleting and printing certain lines.
Let's take a look at several examples.
Example 1: Print the second column from a file
awk '{ print $2 }'
That's all there is to it. Awk automatically splits each line into columns and puts each column in variables $1, $2, $3, etc. This one-liner prints just the 2nd column, which is in variable $2.
You can also specify the symbol or word that you wish to split on with the -F command line switch. This switch is explained in more details in the e-book and in the last example below.
Example 2: Number lines in a file
awk '{ print NR ": " $0 }' file
The whole line itself goes into variable $0. This one-liner prints it but prepends the NR special variable and a colon ": " before it. The special variable NR always contains the current line number.
There are many other special variables and they're all explained in the e-book and summarized in the appendix.
Example 3: Count the number of words in a file
awk '{ total = total + NF } END { print total }'
Here another special variable is used. It's the NF that stands for number of fields, or number of columns, or number of words in the current line. This one-liner then just sums the total number of words up and prints them before quitting in the END block.
Example 4: Print only lines shorter than 64 characters
awk 'length < 64'
This one-liner uses the length function to determine the length of the current line. If the current line is less than 64 characters in length, then length < 64 evaluates to true that instructs awk to print the line.
Finally, let's take a look at an example that compares an Awk program with an equivalent C program. Suppose you want to print the list of all users on your system. With Awk it's as simple as this one-liner:
awk -F: '{ print $1 }' /etc/passwd
This one-liner says, "Take each line from /etc/passwd, split it on the colon and print the first field of each line." Very straightforward and easy to write once you know Awk!
Suppose you didn't know Awk. Then you'd have to write it in some other language, such as C. Compare the example above with the example in C language:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MAX_LINE_LEN 1024
int main () {
char line [MAX_LINE_LEN];
FILE *in = fopen ("/etc/passwd", "r");
if (!in) exit (EXIT_FAILURE);
while (fgets(line, MAX_LINE_LEN, in) != NULL) {
char *sep = strchr(line , ':');
if (!sep) exit (EXIT_FAILURE);
*sep = '\0';
printf("%s\n", line);
}
fclose(in);
return EXIT_SUCCESS ;
}
This is much longer and you have to compile the program first, only then you can run it. If you make any mistakes, you have to recompile it again. That's why one-liners are called one-liners! They are short, easy to write and they do one thing really well. I am pretty sure you're starting to see how mastering Awk and one-liners can make you much more efficient when working in the shell and with text files in general.
What's sed?
Sed is the superman of UNIX stream editing. It's a small utility that's present on every UNIX system and it transforms one stream of text into another. Let's take a look at several practical examples that sed can carry out easily.
Example 1: Replace "lamb" with "goat" on every line
sed 's/lamb/goat/'This one-liner uses the famous s/.../.../ command. The s command substitutes the text in the first part of the command with the text in the second part. In this one-liner it replaces "lamb" with "goat".
A very detailed explanation of how sed reads the lines, how it executes the commands and how the printing happens is presented in the freely available introduction chapter. Please take a look.
Example 2: Replace only the second occurrence of "lamb" with "goat" on every line
sed 's/lamb/goat/2'Sed is the only tool that I know that takes a numeric argument to the s command. The numeric argument, in this case 2, specifies which occurrence of the text to replace. In this example only the 2nd occurrence of "lamb" gets replaced with "goat".
Example 3: Number the lines in a file
sed = file | sed 'N; s/\n/: /'This one-liner is actually two one-liners. The first one uses the = command that inserts a line containing the line number before every original line in the file. Then this output gets piped to the second sed command that joins two adjacent lines with the N command. When joining lines with the N command, a newline character \n is placed between them. Therefore it uses the s command to replace this newline \n with a colon followed by a space ": ".
So for example, if the file contains lines:
hello world
good job
sunny day
Then after running the one-liner, the result is going to be:
1: hello world
2: good job
3: sunny day
Example 4: Delete every 2nd line
sed 'n;d'This one-liner uses the n command that prints the current line (actually the current pattern space, see the introduction chapter for in-depth explanation), deletes it, and reads the next line. Then sed executes the d command that deletes the current line without printing. This way the 1st line gets printed, the 2nd line gets deleted, then the 3rd line gets printed again, then the 4th gets deleted, etc.
Example 5: ROT 13 encode every line
sed '
y/abcdefghijklmnopqrstuvwxyz/nopqrstuvwxyzabcdefghijklm/
y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/NOPQRSTUVWXYZABCDEFGHIJKLM/
'
Here the y/set1/set2/ command is used. The y command substitutes elements in the set1 with the corresponding elements in the set2. The first y command replaces all lowercase letters with their 13-char-shifted counterparts, and the second y command does the same for the uppercase letters. So for example, character a gets replaced by n, b gets replaced by o, character Z gets replaced by M, etc.
Sed is actually very powerful. It's as powerful as a Turing machine, meaning you can write any computer program in it. Check out these programs written in sed. Run them as sed -f file.sed:
转自Peteris Krumins' blog
...全文
请发表友善的回复…
发表回复
姜小白- 2013-10-24
- 打赏
- 举报
顶一下 学习下 