One of the Linux command line tools I had initially under-estimated is netcat or just nc. By default, netcat creates a TCP socket either in listening mode (server socket) or a socket that is used in order to connect to a server (client mode). Actually, netcat does not care whether the socket is meant to be a server or a client. All it does is to take the data from stdin and transfer it to the other end across the network.
The simplest example of its usage is to create a server-client chat system. Although this is a very primitive way to chat, it shows how netcat works. In the following examples it is assumed that the machine that creates the listening socket (server) has the 192.168.0.1 IP address. So, create the chat server on this machine and set it to listen to 3333 TCP port:
$ nc -l 3333
On the other end, connect to the server with the following:
$ nc 192.168.0.1 3333
In this case, the keyboard acts as the stdin. Anything you type in the server machine’s terminal is transfered to the client machine and vice-versa.
Transfering Files
In the very same way it can be used to transfer files between two computers. You can create a server that serves the file with the following:
$ cat backup.iso | nc -l 3333
Receive backup.iso on the client machine with the following:
$ nc 192.168.0.1 3333 > backup.iso
As you may have noticed, netcat does not show any info about the progress of the data transfer. This is inconvenient when dealing with large files. In such cases, a pipe-monitoring utility like pv can be used to show a progress indicator. For example, the following shows the total amount of data that has been transfered in real-time on the server side:
$ cat backup.iso | pv -b | nc -l 3333
Of course, the same can be implemented on the client side by piping netcat’s output through pv:
$ nc 192.168.0.1 3333 | pv -b > backup.iso
Other Examples
Netcat is extremely useful for creating a partition image and sending it to a remote machine on-the-fly:
$ dd if=/dev/hdb5 | gzip -9 | nc -l 3333
On the remote machine, connect to the server and receive the partition image with the following command:
$ nc 192.168.0.1 3333 | pv -b > myhdb5partition.img.gz
This might not be as classy as the partition backups using partimage, but it is efficient.
Another useful thing is to compress the critical files on the server machine with tar and have them pulled by a remote machine:
$ tar -czf - /etc/ | nc -l 3333
As you can see, there is a dash in the tar options instead of a filename. This is because tar’s output needs to be passed to netcat.
On the remote machine, the backup is pulled in the same way as before:
$ nc 192.168.0.1 3333 | pv -b > mybackup.tar.gz
Security
It is obvious that using netcat in the way described above, the data travels in the clear across the network. This is acceptable in case of a local network, but, in case of transfers across the internet, then it would be a wise choice to do it through an SSH tunnel.
Using an SSH tunnel has two advantages:
1. The data is transfered inside an encrypted tunnel, so it is well-protected.
2. You do not need to keep any open ports in the firewall configuration of the machine that will act as the server, as the connections will take place through SSH.
You pipe the file to a listening socket on the server machine in the same way as before. It is assumed that an SSH server runs on this machine too.
$ cat backup.iso | nc -l 3333
On the client machine connect to the listening socket through an SSH tunnel:
$ ssh -f -L 23333:127.0.0.1:3333 me@192.168.0.1 sleep 10; \
nc 127.0.0.1 23333 | pv -b > backup.iso
This way of creating and using the SSH tunnel has the advantage that the tunnel is automagically closed after file transfer finishes. For more information and explanation about it please read my article about auto-closing SSH tunnels.
Telnet-like Usage
Netcat can be used in order to talk to servers like telnet does. For example, in order to get the definition of the word “server” from the “WordNet” database at the dict.org dictionary server, I’d do:
$ nc dict.org 2628
220 ..............some WELCOME.....
DEFINE wn server
150 1 definitions retrieved
151 "server" wn "WordNet (r) 2.0"
server
n 1: a person whose occupation is to serve at table (as in a
restaurant) [syn: {waiter}]
2: (court games) the player who serves to start a point
3: (computer science) a computer that provides client stations
with access to files and printers as shared resources to a
computer network [syn: {host}]
4: utensil used in serving food or drink
.
250 ok [d/m/c = 1/0/18; 0.000r 0.000u 0.000s]
QUIT
221 bye [d/m/c = 0/0/0; 16.000r 0.000u 0.000s]
Works as a Port Scanner too
A useful command line flag is -z. When it is used, netcat does not initiate a connection to the server, but just informs about the open port it has found. Also, instead of a single port, it can accept a port-range to scan. For example:
$ nc -z 192.168.0.1 80-90
Connection to 192.168.0.1 80 port [tcp/http] succeeded!
In this example, netcat scanned the 80-90 range of ports and reported that port 80 is open on the remote machine.
The man page contains some more interesting examples, so take the time to read it.
Notes
All the above examples have been performed on Fedora 5/6. Netcat syntax may vary slightly among Linux distributions, so read the man page carefully.
Netcat provides a primitive way to transfer data between two networked computers. I wouldn’t say it’s an absolutely necessary tool in the everyday use, but there are times that this primitive functionality is very useful.
Monday, September 27, 2010
Thursday, September 2, 2010
Vi Cheat Sheet
Modes
Vi has two modes insertion mode and command mode. The editor begins in command mode, where the cursor movement and text deletion and pasting occur. Insertion mode begins upon entering an insertion or change command. [ESC] returns the editor to command mode (where you can quit, for example by typing :q!). Most commands execute as soon as you type them except for "colon" commands which execute when you press the ruturn key.
Quitting
:x Exit, saving changes
:q Exit as long as there have been no changes
ZZ Exit and save changes if any have been made
:q! Exit and ignore any changes
Inserting Text
i Insert before cursor
I Insert before line
a Append after cursor
A Append after line
o Open a new line after current line
O Open a new line before current line
r Replace one character
R Replace many characters
Motion
h Move left
j Move down
k Move up
l Move right
w Move to next word
W Move to next blank delimited word
b Move to the beginning of the word
B Move to the beginning of blank delimted word
e Move to the end of the word
E Move to the end of Blank delimited word
( Move a sentence back
) Move a sentence forward
{ Move a paragraph back
} Move a paragraph forward
0 Move to the begining of the line
$ Move to the end of the line
1G Move to the first line of the file
G Move to the last line of the file
nG Move to nth line of the file
:n Move to nth line of the file
fc Move forward to c
Fc Move back to c
H Move to top of screen
M Move to middle of screen
L Move to botton of screen
% Move to associated ( ), { }, [ ]
Deleting Text
Almost all deletion commands are performed by typing d followed by a motion. For example, dw deletes a word. A few other deletes are:
x Delete character to the right of cursor
X Delete character to the left of cursor
D Delete to the end of the line
dd Delete current line
:d Delete current line
Yanking Text
Like deletion, almost all yank commands are performed by typing y followed by a motion. For example, y$ yanks to the end of the line. Two other yank commands are:
yy Yank the current line
:y Yank the current line
Changing text
The change command is a deletion command that leaves the editor in insert mode. It is performed by typing c followed by a motion. For wxample cw changes a word. A few other change commands are:
C Change to the end of the line
cc Change the whole line
Putting text
p Put after the position or after the line
P Put before the poition or before the line
Buffers
Named buffers may be specified before any deletion, change, yank or put command. The general prefix has the form "c where c is any lowercase character. for example, "adw deletes a word into buffer a. It may thereafter be put back into text with an appropriate "ap.
Markers
Named markers may be set on any line in a file. Any lower case letter may be a marker name. Markers may also be used as limits for ranges.
mc Set marker c on this line
`c Go to beginning of marker c line.
'c Go to first non-blank character of marker c line.
Search for strings
/string Search forward for string
?string Search back for string
n Search for next instance of string
N Search for previous instance of string
Replace
The search and replace function is accomplished with the :s command. It is commonly used in combination with ranges or the :g command (below).
:s/pattern/string/flags Replace pattern with string according to flags.
g Flag - Replace all occurences of pattern
c Flag - Confirm replaces.
& Repeat last :s command
Regular Expressions
. (dot) Any single character except newline
* zero or more occurances of any character
[...] Any single character specified in the set
[^...] Any single character not specified in the set
^ Anchor - beginning of the line
$ Anchor - end of line
\< Anchor - begining of word
\> Anchor - end of word
\(...\) Grouping - usually used to group conditions
\n Contents of nth grouping
[...] - Set Examples [A-Z] The SET from Capital A to Capital Z
[a-z] The SET from lowercase a to lowercase z
[0-9] The SET from 0 to 9 (All numerals)
[./=+] The SET containing . (dot), / (slash), =, and +
[-A-F] The SET from Capital A to Capital F and the dash (dashes must be specified first)
[0-9 A-Z] The SET containing all capital letters and digits and a space
[A-Z][a-zA-Z] In the first position, the SET from Capital A to Capital Z
In the second character position, the SET containing all letters
Regular Expression Examples /Hello/ Matches if the line contains the value Hello
/^TEST$/ Matches if the line contains TEST by itself
/^[a-zA-Z]/ Matches if the line starts with any letter
/^[a-z].*/ Matches if the first character of the line is a-z and there is at least one more of any character following it
/2134$/ Matches if line ends with 2134
/\(21|35\)/ Matches is the line contains 21 or 35
Note the use of ( ) with the pipe symbol to specify the 'or' condition
/[0-9]*/ Matches if there are zero or more numbers in the line
/^[^#]/ Matches if the first character is not a # in the line
Notes:
1. Regular expressions are case sensitive
2. Regular expressions are to be used where pattern is specified
Counts
Nearly every command may be preceded by a number that specifies how many times it is to be performed. For example, 5dw will delete 5 words and 3fe will move the cursor forward to the 3rd occurence of the letter e. Even insertions may be repeated conveniently with thismethod, say to insert the same line 100 times.
Ranges
Ranges may precede most "colon" commands and cause them to be executed on a line or lines. For example :3,7d would delete lines 3-7. Ranges are commonly combined with the :s command to perform a replacement on several lines, as with :.,$s/pattern/string/g to make a replacement from the current line to the end of the file.
:n,m Range - Lines n-m
:. Range - Current line
:$ Range - Last line
:'c Range - Marker c
:% Range - All lines in file
:g/pattern/ Range - All lines that contain pattern
Files
:w file Write to file
:r file Read file in after line
:n Go to next file
:p Go to previos file
:e file Edit file
!!program Replace line with output from program
Other
~ Toggle upp and lower case
J Join lines
. Repeat last text-changing command
u Undo last change
U Undo all changes to line
Vi has two modes insertion mode and command mode. The editor begins in command mode, where the cursor movement and text deletion and pasting occur. Insertion mode begins upon entering an insertion or change command. [ESC] returns the editor to command mode (where you can quit, for example by typing :q!). Most commands execute as soon as you type them except for "colon" commands which execute when you press the ruturn key.
Quitting
:x Exit, saving changes
:q Exit as long as there have been no changes
ZZ Exit and save changes if any have been made
:q! Exit and ignore any changes
Inserting Text
i Insert before cursor
I Insert before line
a Append after cursor
A Append after line
o Open a new line after current line
O Open a new line before current line
r Replace one character
R Replace many characters
Motion
h Move left
j Move down
k Move up
l Move right
w Move to next word
W Move to next blank delimited word
b Move to the beginning of the word
B Move to the beginning of blank delimted word
e Move to the end of the word
E Move to the end of Blank delimited word
( Move a sentence back
) Move a sentence forward
{ Move a paragraph back
} Move a paragraph forward
0 Move to the begining of the line
$ Move to the end of the line
1G Move to the first line of the file
G Move to the last line of the file
nG Move to nth line of the file
:n Move to nth line of the file
fc Move forward to c
Fc Move back to c
H Move to top of screen
M Move to middle of screen
L Move to botton of screen
% Move to associated ( ), { }, [ ]
Deleting Text
Almost all deletion commands are performed by typing d followed by a motion. For example, dw deletes a word. A few other deletes are:
x Delete character to the right of cursor
X Delete character to the left of cursor
D Delete to the end of the line
dd Delete current line
:d Delete current line
Yanking Text
Like deletion, almost all yank commands are performed by typing y followed by a motion. For example, y$ yanks to the end of the line. Two other yank commands are:
yy Yank the current line
:y Yank the current line
Changing text
The change command is a deletion command that leaves the editor in insert mode. It is performed by typing c followed by a motion. For wxample cw changes a word. A few other change commands are:
C Change to the end of the line
cc Change the whole line
Putting text
p Put after the position or after the line
P Put before the poition or before the line
Buffers
Named buffers may be specified before any deletion, change, yank or put command. The general prefix has the form "c where c is any lowercase character. for example, "adw deletes a word into buffer a. It may thereafter be put back into text with an appropriate "ap.
Markers
Named markers may be set on any line in a file. Any lower case letter may be a marker name. Markers may also be used as limits for ranges.
mc Set marker c on this line
`c Go to beginning of marker c line.
'c Go to first non-blank character of marker c line.
Search for strings
/string Search forward for string
?string Search back for string
n Search for next instance of string
N Search for previous instance of string
Replace
The search and replace function is accomplished with the :s command. It is commonly used in combination with ranges or the :g command (below).
:s/pattern/string/flags Replace pattern with string according to flags.
g Flag - Replace all occurences of pattern
c Flag - Confirm replaces.
& Repeat last :s command
Regular Expressions
. (dot) Any single character except newline
* zero or more occurances of any character
[...] Any single character specified in the set
[^...] Any single character not specified in the set
^ Anchor - beginning of the line
$ Anchor - end of line
\< Anchor - begining of word
\> Anchor - end of word
\(...\) Grouping - usually used to group conditions
\n Contents of nth grouping
[...] - Set Examples [A-Z] The SET from Capital A to Capital Z
[a-z] The SET from lowercase a to lowercase z
[0-9] The SET from 0 to 9 (All numerals)
[./=+] The SET containing . (dot), / (slash), =, and +
[-A-F] The SET from Capital A to Capital F and the dash (dashes must be specified first)
[0-9 A-Z] The SET containing all capital letters and digits and a space
[A-Z][a-zA-Z] In the first position, the SET from Capital A to Capital Z
In the second character position, the SET containing all letters
Regular Expression Examples /Hello/ Matches if the line contains the value Hello
/^TEST$/ Matches if the line contains TEST by itself
/^[a-zA-Z]/ Matches if the line starts with any letter
/^[a-z].*/ Matches if the first character of the line is a-z and there is at least one more of any character following it
/2134$/ Matches if line ends with 2134
/\(21|35\)/ Matches is the line contains 21 or 35
Note the use of ( ) with the pipe symbol to specify the 'or' condition
/[0-9]*/ Matches if there are zero or more numbers in the line
/^[^#]/ Matches if the first character is not a # in the line
Notes:
1. Regular expressions are case sensitive
2. Regular expressions are to be used where pattern is specified
Counts
Nearly every command may be preceded by a number that specifies how many times it is to be performed. For example, 5dw will delete 5 words and 3fe will move the cursor forward to the 3rd occurence of the letter e. Even insertions may be repeated conveniently with thismethod, say to insert the same line 100 times.
Ranges
Ranges may precede most "colon" commands and cause them to be executed on a line or lines. For example :3,7d would delete lines 3-7. Ranges are commonly combined with the :s command to perform a replacement on several lines, as with :.,$s/pattern/string/g to make a replacement from the current line to the end of the file.
:n,m Range - Lines n-m
:. Range - Current line
:$ Range - Last line
:'c Range - Marker c
:% Range - All lines in file
:g/pattern/ Range - All lines that contain pattern
Files
:w file Write to file
:r file Read file in after line
:n Go to next file
:p Go to previos file
:e file Edit file
!!program Replace line with output from program
Other
~ Toggle upp and lower case
J Join lines
. Repeat last text-changing command
u Undo last change
U Undo all changes to line
Subscribe to:
Posts (Atom)