Filters and commands in Unix with examples

Basic Filters and commands in unix systems

• Commands in unix OS are case-sensitive i.e 'CLEAR' is different from 'clear' command. (if you try to run the CLEAR in terminal (it shows "command not found" message), 'clear' command  will clear the terminal space)

1. awk command : Pattern scanning and text processing language. Finds patterns and Replaces patterns or text, validate, index and database sort. 

This UNIX command utility scans a file by lines and also split the each line into columns based on default delimiter. Then match input fields/pattern to lines and also performs the specified actions on the matching line.

basic Builtin-variables:
NR : current record number in the total input stream.
NF : number of fields in the current record.

example:
~$ ls -l                            //list current directory files and folders
~$ ls -l  > testfile5                          //save the list current directory files and folders to the testfile5
~$ awk '{print $1,$6}' testfile5      //select the column 1 and 6 from the testfile5 and display

2. cat command:
The cat ("concatenate" ) is filter command of operating systems like Linux/Unix. cat utility command used to create one or more files, view content of file, concatenate different files and can redirect the output to files or terminal.

example:
~$ cat > testfile1      // to create file, exit terminal and save the file content after pressing CTRL+D.
~$ cat testfile            //command to display the content of the file to terminal 
~$ cat testfile3 >> testfile // Append the content of testfile3 to the end of testfile2.

3. comm command:

• comm UNIX command compare two sorted files line by line and display in (terminal) standard output. 
• Suppose there are two file, file1 contains the list of the student names opted for subject DBMS and file2 contains the list of the student names opted for subject OS. To find the student names who are common in both files then, comm command will help you to achieve result for above query.
• To run this comm command content of the files to compare should be sorted. 

example:
~$ cat testfile6                         //displays the content of testfile6
~$ cat testfile7                         //displays the content of testfile7
~$ comm testfile6 testfile7     
In output of the above command,
First column shows distinct lines in testfile6
Second column shows distinct lines in testfile7
Third column shows lines common in both files

4. head command:

The head command, as the name suggesting that it display the first N lines of given input file content. By default, the value of N is 10 lines. While no option is mentioned then, by default, it display first 10 lines of the file content.

example:
~$ awk 'END { print NR}' testfile4      //prints the count of lines in testfile4
~$ head  testfile4                                      //by default(with no options), prints first 10 lines in testfile4
~$ head  testfile4 | awk 'END { print NR} '//pipeline with above command, prints the count of lines
~$ head -n 4 testfile4                             //prints first 4 lines in testfile4

5. paste command:

• It is used to merge files in horizontal/parallel manner consisting of lines from each specified file to output, separated by tab (delimiter).

•  Write lines consisting of the sequentially corresponding lines from each FILE, separated by TABs, to standard output.

example:
~$ paste -d "|"  testfile1 testfile2     //merge parallel and  delimiter- "|"
In output, the first column consist of content of testfile1 and second column represent the content of testfile2 separated by delimiter "|" .

~$ paste -d "|#" testfile1 testfile2 testfile9 //merge parallel and delimiter-"|", "#" used as in image

~$ paste -s testfile1 testfile2 testfile9  //merge horizontally and  default delimiter- TAB

6. sed command:

• Sed stands for stream editor. A sed command is used to perform basic text modifications on an input stream (a file or input from a pipeline). 

• sed command works by making only one pass over the input(s), and is consequently more efficient. It used in the function on file like, searc, find, replace, insert or delete. most commonly used for substitution.

•  By using sed command in UNIX you can edit files without opening it. This way it's much quicker way then to open that file in Editor and then modifying it.

• SED command supports can perform complex pattern matching as it supports regular expression.

example:

~$sed 's/Jul/Oct/2' testfile8
In output, the 2^nd occurrence of phrase "Jul" in each line will be replaced by the phrase "Oct" for content of testfile8. If the second occurrence of word  "Jul" found in line then, it replaced by "Oct", otherwise, it command for match in the next line. 

~$sed 's/Jul/Oct/g' testfile8
In output, the all occurrence of phrase "Jul" in each line will be replaced by the phrase "Oct" for content of testfile8.

~$sed  's/pradip/Pradip/2g'  testfile8   //second occurrence of "pradip" replaced to "PRADIP"
In output, starting from the 2^nd occurrence each of phrase "Jul" in each line will be replaced by the phrase "Oct" for content of testfile8.

 7 . sort command:

• Write sorted concatenation of all FILE(s) to standard output. 
• sort command can also be used to sort numerically. (if file content is numerical), sort in reverse order, sort alphabetically, sort by month and can also used to remove duplicates.
• By default, if no options specified then, command sorts assuming that the file contents are ASCII, the entire line is taken as sort key and Default field separator will be a Blank space.

example:
~$ cat testfile2   //The file display content of testfile2
~$ sort testfile2   //The file lines are sorted alphabetically. 
~$ sort -r testfile2  //The file lines are sorted alphabetically in reverse order.

8 . tail command:

• The tail command, as the name refers that it display the last N lines of given input content. 

• By default, the value of N is 10 lines. While no option is mentioned in command then it display last 10 lines of the file content.

example:

 ~$cat testfile4           //displays content of testfile4
 ~$tail -5 testfile4   //displays last five lines of testfile4 (total lines are 20)

 9 . tr command:

• The tr ("translate" ) is filter command of operating systems like Linux/Unix. 
• It is a command line utility to translate, squeeze, and/or delete characters from standard input, writing to standard output. 
• It supports transformations like uppercase to lowercase, find and replace character . It can be used to support more complex transformations while used with UNIX pipes. 

example: 
~$cat testfile3 | tr “[a-z]” “[A-Z]”

In output, it replaces the all LOWERCASE letters to UPPERCASE alphabets with range specified and display translated content of the testfile3.
~$cat testfile3 | tr “[a-w]” “[A-W]”

In output, it replaces the all LOWERCASE letters to UPPERCASE alphabets with range specified and display translated content of the testfile3. If character is not in range then it will not be traslated.

 10 . uniq command:

• The uniq command utility will filter adjacent matching lines from INPUT (or standard input), helps to find the adjacent duplicate lines and can be used to delete the duplicate lines and writes to file/OUTPUT (or standard output).
• If no options provided then, matching lines are merged to the first occurrence.

example:

~$cat testfile2     //displays the testfile2

~$uniq testfile2   //in output, repeated lines are removed

Filters and commands: sed command in Unix systems with examples

SED command in UNIX is stands for stream editor and it can perform lot’s of function on file like, searching, find and replace, insertion or deletion. Though most common use of SED command in UNIX is for substitution or for find and replace. By using SED you can edit files even without opening it, which is much quicker way to find and replace something in file, than first opening that file in VI Editor and then changing it.

Filters and commands: join command in Unix systems with examples

join command is the tool for it. join command is used to join the two files based on a key field present in both the files. The input file can be separated by white space or any delimiter.

Filters and commands: tr command in Unix systems with examples

The tr command in UNIX is a command line utility for translating or deleting characters. It supports a range of transformations including uppercase to lowercase, squeezing repeating characters, deleting specific characters and basic find and replace.

Filters and commands: uniq command in Unix systems with examples

The uniq command in Linux is a command line utility that reports or filters out the repeated lines in a file.

Filters and commands: paste command in Unix systems with examples

It is used to join files horizontally (parallel merging) by outputting lines consisting of lines from each file specified, separated by tab as delimiter, to the standard output.

 

Filters and commands: sort command in Unix systems with examples

SORT command is used to sort a file, arranging the records in a particular order. By default, the sort command sorts file assuming the contents are ASCII. Using options in sort command, it can also be used to sort numerically. 

Filters and commands: cut command in Unix systems with examples

The cut command in UNIX is a command for cutting out the sections from each line of files and writing the result to standard output. It can be used to cut parts of a line by byte position, character and field. Basically the cut command slices a line and extracts the text. It is necessary to specify option with command otherwise it gives error. 
 

Filters and commands: Tail command in Unix systems with examples

The tail command, as the name refers that it display the last N lines of given input content. By default, the value of N is 10 lines. While no option is mentioned in command then it display last 10 lines of the file content.

~$ man tail   (on Linux ubuntu 16.10)
 

~$cat testfile4      // display content of testfile4

~$ tail -5 testfile4   // displays last five lines of the content of the file

~$ awk 'END { print NR }' testfile4   // no of lines in testfile4

~$ tail -n +14 testfile4   // displays last 7 lines starting from the line No. 14

~$ tail -c -5 testfile4   // displays last five bytes of the file

~$ tail -c +100 testfile4   // displays last 23 bytes starting from the byte No. 100

~$tail -f testfile3      // display live content of testfile3 (logfile)

In case, if user want to see log content of log-file of some currently running process. For example, Android studio log file is generated and can be seen while App is debugging.

The head command, as the name refers that it display the first N lines of given input content. By default, the value of N is 10 lines. While no option is mentioned then it display first 10 lines of the file content.

 

Computer network security, Question-23, GATE 2016 (set-2)

Anarkali digitally signs a message and sends it to Salim. Verification of the signature by Salim requires

(A) Anarkali’s public key.
(B) Salim’s public key.
(C) Salim’s private key.
(D) Anarkali’s private key.

Computer network security, Question 68, GATE IT 2008

The total number of keys required for a set of n individuals to be able to communicate with each other using secret key and public key crypto-systems, respectively are:

(A) n(n-1) and 2n
(B) 2n and ((n(n - 1))/2)
(C) ((n(n - 1))/2) and 2n
(D) ((n(n - 1))/2) and n

Filters and commands in Unix temp temp

Commonly used filter commands 

    awk
    cat
    comm
    cut
    expand
    compress
    fold
    grep
    head
    less
    more
    nl
    perl
    paste
    pr
    sed
    sh
    sort
    split
    strings
    tail
    tac
    tee
    tr
    uniq
    wc
    zcat

 

awk command:

Pattern scanning and text processing language. Finds patterns and Replaces patterns or text, validate, index and database sort

~$ man awk

Built In Variables:

Awk’s built-in variables include the field variables—$1, $2, $3, and so on ($0 is the entire line) — that break a line of text into individual words or pieces called fields.
NR: NR command keeps a current count of the number of input records. Remember that records are usually lines. Awk command performs the pattern/action statements once for each record in a file.
NF: NF command keeps a count of the number of fields within the current input record.
FS: FS command contains the field separator character which is used to divide fields on the input line. The default is “white space”, meaning space and tab characters. FS can be reassigned to another character (typically in BEGIN) to change the field separator.
RS: RS command stores the current record separator character. Since, by default, an input line is the input record, the default record separator character is a newline.
OFS: OFS command stores the output field separator, which separates the fields when Awk prints them. The default is a blank space. Whenever print has several parameters separated with commas, it will print the value of OFS in between each parameter.
ORS: ORS command stores the output record separator, which separates the output lines when Awk prints them. The default is a newline character. print automatically outputs the contents of ORS at the end of whatever it is given to print.

Here, we listed the root directory files detail and saved the same to the tesfile5
Here, awk command retries the field values of the column 1 and 6.

Below will prints only line which are non-empty. First and last line of testfile5 are empty. So, empty line only printed while displaying the  file using cat command.

Below image show some of the functions implemented using the awk command

 

cat command:

The cat ("concatenate") is filter command of operating systems like Linux/Unix. cat command used to create one or more files, view content of file, concatenate different files and can redirect the output to files or terminal.

In this post article, we are going to find out the very basic use  of the cat command and options related to the command with examples in Linux.

Syntax of the command

cat [OPTION] [FILE]...

(1) cat command to Create file and display File content in terminal

~$ cat > testfile1      //command to create file testfile1
~$ cat testfile            //command to display the content of the file to terminal

System creates file opens the blank file. then, user can write and press CNTRL+D to exit the file and the written content will be saved to file test3.

(2) Content Display for Multiple Files in terminal

~$ cat testfile1 testfile2

 Shows content of the testfile1 and at the end content testfile2


(3) Appending Standard Output with Redirection Operator 

~$ cat testfile3 >> testfile2

 Append the content of testfile3 to the end of testfile2. As we can see above the in the content display of filetest2.


(4) Redirection Operator Uses Standard Output to overwrite the file content. ~$ cat testfile3 > testfile4

 content of testfile4 will be overwritten by the testfile3 file content.

 (4) Redirection Operator Uses Standard Output to write Multiple Files content to a file. 

~$ cat testfile1 testfile2 > testfile4 

It's multiple file contents redirecting to a Single File
Conent of testfile1 and testfile3 overwritten to the testfile4

(5) Redirection Operator Uses Standard input to display content in terminal.
~$ cat < testfile4

Here the content of the file testfile4 will be used as input to the terminal. So, terminal will display the testfile4 content.

Comm Command:
comm compare two sorted files line by line and write to standard output; the lines that are common and the lines that are unique.

  -1 : hide first column (first column shows distinct lines in testfile6 )
  -2 : hide second column (second column shows distinct lines in testfile7 )
  -3 : hide third column (second column shows lines common in both files).

Filters and commands in Unix

The cat ("concatenate") is filter command of operating systems like Linux/Unix. cat command used to create one or more files, view content of file, concatenate different files and can redirect the output to files or terminal. In article of this post, we are going to find out the very basic use cat command and options related to the command with examples in Linux.
Syntax of the command

cat [OPTION] [FILE]...

(1) Display File content in terminal
# cat /etc/test1
Shows content of the test1 file
(2) Content Display for Multiple Files in terminal
# cat test1 test2
Shows content of the test1 and test2 file
(3) cat command to Create file
# cat > test3
create file and open the blank file then, user can write and press CNTRL+D to exit the file and the written content will be saved to file test3.
(4) Redirection Operator Uses Standard Output to overwrite the file content.
# cat test3 > test4
test4 file content will be overwritten by the test file content.
(5) Redirection Operator Uses Standard input to display content in terminal.
# cat < test5
This is content of test5 file.
Here the content of the file test5 will be used as input to the terminal. So, terminal will display the file content.
(6) Multiple Files content Redirecting to a Single File
# cat test test1 test2 > test3
content $ cat sample.txt 1,2,3,4,5,6,7,8,9
$ cat sample.txt | cut -d \; -f 2 > output.txt
$ cat output.txt
2

semaphore

A semaphore is variable with integer value that is updated through 2 standard operations which are atomic :

(1) P: wait () operation
and
(2) V: signal () operation.
Lets say Semaphore variable is Sm

wait(Sm) {
while Sm<= 0 ;    //No-operation, keep in while loop if condition true, next line will not execute
Sm - - ;       //executed if while loop is not satisfied
}

signal(Sm) { 
 Sm + +;
 }

As both functions are atomic whole wait() and signal() function must be executed without interruption. This means the while condition check and the Sm - - operation will be executed without break or interruption. Variable Sm can only be accessed by only one function either wait() or signal() at a time.