Linux and Unix Commands, Shell Scripting Concepts, Operating System, for Beginners and Experts - The Essential Linux and Unix Handbook
Linux and Unix Commands, Shell Scripting Concepts, Operating System, for Beginners and Experts - The Essential Linux and Unix Handbook
Introduction
Linux and Unix are powerful operating systems widely used in various industries, including web development, networking, and system administration. With the increasing demand for skilled Linux/Unix professionals, it's essential to have a comprehensive understanding of the commands, concepts, and scripting techniques that make up these operating systems. In this blog post, we'll take you on a journey to master Linux and Unix operating systems covering all the essential commands, concepts, and shell scripting techniques. Whether you're a beginner or an experienced user, this guide will help you improve your skills and become proficient in using Linux and Unix.
Unix/Linux Basics
Unix/Linux is a multiuser operating system, which means that multiple users can access the system simultaneously. It's also a multitasking operating system, which means that multiple tasks can be executed concurrently. We can conclude it as below:
- Multiuser and Multitasking Operating System
- Previously it was providing CLI but now we have GUI as well
- Fast and Secure
- Interactive
Unix/Linux has two primary interfaces:
- Command-Line Interface (CLI): The CLI is a text-based interface where users can interact with the system using commands.
- Graphical User Interface (GUI): The GUI is a graphical interface where users can interact with the system using visual elements like windows, icons, and menus.
Different Unix and Linux Versions
There are several Unix and Linux versions available, including:
- Unix: Solaris, AIX, HPUX
- Linux: Ubuntu, Fedora, Red Hat
Why Learn Unix/Linux?
Learning Unix/Linux is essential for anyone working in the IT industry, particularly in the fields of development, testing, and system administration. Unix/Linux is widely used in the industry, and knowledge of its basics is a prerequisite for many jobs.
A software tester should also have knowledge of Unix/Linux because they also need it for software testing for the below-stated reasons:
- In most companies, Application servers are placed on Unix/ Linux machines so many times we need to access these server machines to perform different activities.
- Install/Uninstall Software/Application
- View log files
- Take a backup of logs or other files
- Clean-up space or memory management
- Verify CPU/Memory status
- Start/Stop processes, Kill processes
Accessing Unix/Linux Machines
To access a Unix/Linux machine, you can use a tool like Putty. Here's how:
- Open Putty and enter the IP address of the machine.
- Enter the port number (default is 22).
- Enter your username and password.
- Click "Open" to connect to the machine.
Online Linux Environment
If you don't have access to a Unix/Linux machine, you can use an online Linux environment here. This is a web-based Linux environment that allows you to practice Unix/Linux commands without installing anything on your local machine.
Directory Structure in Unix
In Unix, the directory structure is hierarchical, with the root directory at the top. Here's an example of how to create a directory:
- mkdir mydirectory
(ex. mkdir directory_name)
To view the files and directories in the current directory, use the ls command:
Command used : ls (List)
- ls -l: long list
- Ls -lrt: long list, reverse, time (sort the files and directories as per time in increasing order)
 |
To get inside a directory, use the cd (change directory) command:
Permissions
In Unix, permissions are used to control access to files and directories. There are three types of permissions:
- User: The owner of the file or directory.
- Group: The group that the file or directory belongs to.
- Other: Everyone else.
- Chmod:
Permissions in Unix/Linux are not in the form of Characters, rather are represented as a three-digit octal number.
- Read : 4 : r--
- Write : 2 : -w-
- Execute : 1 : --x
Examples:
- Read + Write + Execute = 4+2+1 =7
- Read + Write = 6
- Write + Execute = 3
[root@localhost ~]# chmod 777 testdir
To change permissions, use the 'chmod' command:
- chmod 755 myfile
This sets the permissions to read, write, and execute for the owner, read and execute for the group, and read and execute for others.
- What is the default permission whenever a directory is created :
drwxr-xr-x
drwxr-xr-x
- Create multiple Directories:
- Permission to a specific user:
🞇 User : rwx
🞇 Group : rwx
🞇 User : rwx
🞇 Group : rwx
File Management
Unix provides several commands for managing files, including:
- cp: Copy a file.
- mv: Move or rename a file.
- rm: Remove a file or directory.
- cat: View the contents of a file.
Text Editors
Unix provides several text editors, including VI (Visual) Editor. Here's how to open any file in VI Editor:
vi file_name
VI Editor has two modes:
- Insert Mode: Type 'i' to enter into insert mode.
- Command Mode: Press the 'Esc' button to enter into command mode.
- To Save the file and quit ==> ':wq' (in command mode).
- To Quit the file without saving it ==> ':q!' (in command mode).
- To Quit the file ==> ':q' (in command mode).
Cursor Movement in VI Editor :
🞇 Left : h
🞇 Right : l
🞇 Down : j
🞇 Up : k
Note: You can also use arrow keys in some Linux/unix versions to move the cursor
🞇 To Beginning of file : 1G or G1
🞇 To the bottom line of the file : L
🞇 To end of any word : e
Delete a character in VI :
🞇 X : The character just before the cursor will be deleted.
🞇 x : The character under the cursor will be deleted.
Delete a line in VI:
🞇 dd : deletes the entire line where the cursor is present at that moment.
🞇 2dd : deletes 2 lines starting from the line where the cursor is present at that moment.
Note: You can use any number before 'dd' to delete as many lines as you want.
Examples - For deleting 15 lines: 15dd
For deleting 8 lines: 8dd
Insert character or string at a particular position in VI:
🞇 a : insert any character or string towards the right of the cursor.
🞇 A : insert any character or string at the end of the line where the cursor is present.
How to create an empty file in Linux:
Use command: 'touch'
For example- touch myfile, touch file1 file2 file3
What is the default permission whenever a file is created:
-rw-r--r--
Create Multiple Files using VI:
- vi file1 file2 file3 file4
- :n ==> Move to the next file
- :n! ==> Move to the next file after discarding unsaved changes in the current file
- :wn ==> Move to the next file after saving changes done in the first file
- :rew ==> Move to the first file in the list of files and edit it
- :xa ==> Save all files at once and exit
- :set autowrite (or) :set aw ==> write to the current file automatically when switching to another file
View any file:
🞇 Command: 'cat'
Create a new file using CAT:
🞇 Command: cat > filename
Enter your data
Ctrl+c
Ctrl+c
To view the content of any file:
Use the command: cat filename
Copy content of one file to another file using CAT in override mode:
🞇 Command: cat filename1 > filename2
Note: '>' refers to override mode
Copy content of one file to another file using CAT in Append mode:
🞇 Command: cat filename1 >> filename2
Note: '>> refers append mode
Merge multiple files to create a new file using CAT:
🞇 Command: cat filename1 filename2 > filename3
Display content of the file using CAT in numeric order:
🞇 Command :
- Ascending order by serial no: cat -n filename
- Descending order by serial no: tac –n filename
Remove a file or directory using rm commands:
🞇 rm -i : will ask the user before deleting the file or directory
🞇 rm -r : will recursively delete the file
🞇 rm -rf : will recursively and forcefully delete the file or directory without asking the user
Remove empty directory using rm commands:
🞇 rmdir directoryname
Note: Here directory to be deleted should not contain any data (files or subdirectories)
Head Command
The head command is used to display the first few lines of a file. By default, it shows the first 10 lines, but you can specify a different number of lines as an option.
Syntax:
head [options] [file_name]
Options:
-n specifies the number of lines to display (ex: -n 5 to show the first 5 lines)
Tail Command
The tail command is used to display the last few lines of a file. By default, it shows the last 10 lines, but you can specify a different number of lines as an option.
Syntax:
tail [options] [file_name]
Options:
-n specifies the number of lines to display (ex: -n 5 to show the last 5 lines)
-f follows the file as it grows, displaying new lines as they are added (useful for monitoring log files)
Grep: Global regular expression print
Syntax:
grep -<options><string your want to search> <file or location where you want to search>
Grep options :
- Grep -i : Ignore case
- Grep -I : list only file names containing matching lines
- Grep -w : match exact word
- Grep -n : Precede each matching line with its line number
- Grep -e ‘string1|string2|string3’ : Multiple string search
- Grep -v : this will give the lines where given pattern is not found
Grep vs Fgrep:
- Grep: Uses regular expression for pattern match.
- Fgrep: Also called File grep, does not use regular expression, instead uses exact string.
Grep vs Egrep:
- Grep: Uses regular expression for pattern match.
- Egrep: Also called Extended grep.
Top Command
The top command is used to display real-time information about running processes on a Linux system. It provides a dynamic view of the system's processes, including CPU usage, memory usage, and other details.
Options:
-n numeric output, no DNS lookups
-p specify a port number
-w set the timeout value
Example: traceroute google.com
Df Command
The df command is used to display information about disk space usage on a Linux system. It shows the total, used, and available space for each mounted file system.
Options:
-h human-readable output
-i display inode information
-T display file system type
Ping Command
The ping command is used to test network connectivity to a destination host. It sends ICMP echo request packets to the host and displays the response.
Options:
-c specify the number of packets to send
-i specify the interval between packets
-s specify the packet size
Ps Command
The ps command is used to display information about running processes on a Linux system. It provides a snapshot of the current processes, including their process ID, parent process ID, CPU usage, memory usage, and other details.
Options:
-a show all processes, including those not attached to a terminal
-e show all processes, including those not attached to a terminal, and including environment variables
-f full format, showing more detailed information
-p specify a process ID to display information about
-u display information about processes owned by a specific user
Kill -9 Command
The kill -9 command is used to forcefully terminate a running process on a Linux system. It sends a SIGKILL signal to the process, which cannot be caught or ignored by the process. This command is often used to terminate a process that is not responding or is causing issues.
kill -9 PID
Examples: Terminate a process with a specific PID: (You can get PID Process ID from Top command, it is similar thing kind of we do start task manager and end running process in windows)
kill -9 1234
This will send a SIGKILL signal to the process with PID 1234, forcing it to terminate immediately.
Note: Be careful when using kill -9, as it can cause data loss or corruption if the process is in the middle of writing to a file or performing some other critical operation.
Why use kill -9 instead of just kill?
The kill command without any options sends a SIGTERM signal to the process, which can be caught and handled by the process. This allows the process to perform any necessary cleanup or shutdown procedures before terminating. However, if the process is not responding or is ignoring the SIGTERM signal, kill -9 can be used to forcefully terminate the process.
When to use kill -9?
Use kill -9 when:
- A process is not responding or is frozen.
- A process is consuming excessive system resources.
- A process is causing system instability or crashes.
- You need to terminate a process immediately, without allowing it to perform any cleanup or shutdown procedures.
File comparison commands:
- Cmp :
ex. Cmp file1 file2
Cmp -n : compare specific number of lines.
Cmp -b : show exact data that is different.
2. Comm :
Compare 2 sorted files line by line
Output of compare :
Column 1 : Content in file1
Column 2 : Content in file2
Column 3 : Common content of both files
3. Diff :
diff file1 file2
Options :
🞇 -a Treat all files as text
🞇 -b Ignore changes in the amount of whitespace
🞇 -B Ignore changes whose lines are all blank
🞇 -d Try hard to find a smaller set of changes
🞇 -i Ignore case differences
🞇 -L Use LABEL instead of the filename in the unified header
🞇 -N Treat absent files as empty
🞇 -q Output only whether files differ
🞇 -r Recurse
🞇 -S Start with FILE when comparing directories
🞇 -T Make tabs line up by prefixing a tab when necessary
🞇 -s Report when two files are the same
🞇 -t Expand tabs to spaces in output
🞇 -U Output LINES lines of context
🞇 -w Ignore all whitespace
SED: Stream Editor
Command syntax : sed options [address]commands file(s)
It doesn’t make changes in the original file, rather just displays the change on the console.
Output :
Reads 1st line of data and processes it, writes the output to standard output
Reads 2nd line of data and processes it, writes the output to standard output
Reads 3rd line of data and processes it, writes the output to standard output
::
::
Reads the last line of data and processes it, writes the output to standard output
SED: Stream Editor Substitution
Command syntax : sed ‘s/searchstring/replacementstring/flags’ file
SED: Stream Editor Substitution : Addressing
Addressing: sed options [address]commands file
Line Addressing:
Examples:
- sed ‘2s/str1/str2/’ file.txt
- sed ‘2,5s/str1/str2/’ file.txt
- sed ‘3,$s/str1/str2/’ file.txt
- sed ‘$s/sr1/str2/’ file.txt
Context Addressing:
Examples:
- sed ‘/pattern/s/str1/str2/’ file.txt
- sed ‘/pattern1/, /pattern2/s/str1/str2/’ file.txt {Working like OR Operator of SQL}
- sed ‘3,/pattern/s/str1/str2/’ file.txt
- sed ‘/pattern/,9s/str1/str2’ file.txt
- sed ‘/pattern/,+4s/str1/str2’ file.txt
SED: Negation
Denoted by exclamation (!) :
If a substitution is supposed to be done in records except for the selected one then the negation is used.
Below are some examples:
- sed ‘2!s/str1/str2/’ file.txt
- sed ‘2,5!s/str1/str2/’ file.txt
- sed ‘3,$!s/str1/str2/’ file.txt
- sed ‘$!s/sr1/str2/’ file.txt
- sed ‘/pattern/s/str1/str2/’ file.txt
- sed ‘/pattern1/, /pattern2/s/str1/str2/’ file.txt
- sed ‘3,/pattern/s/str1/str2/’ file.txt
- sed ‘/pattern/,9s/str1/str2’ file.txt
- sed ‘/pattern/,+4s/str1/str2’ file.txt
SED: Deletion
Denoted by exclamation (d) : Used to delete specified lines
Examples:
- sed ‘d’ file.txt
- sed ‘2,5d’ file.txt
- sed ‘3,$d’ file.txt
- sed ‘$d’ file.txt
- sed ‘/pattern/d’ file.txt
- sed ‘/pattern1/, /pattern2/d’ file.txt
- sed ‘3,/pattern/d’ file.txt
- sed ‘/pattern/,9d’ file.txt
- sed ‘/pattern/,+4d’ file.txt
- sed ‘5d’ file.txt
- sed ‘5!d’ file.txt
- sed ‘/^$/d’ file.txt
- sed ‘/^[ \t]*$/d’ file.txt
SED: Change
Change : (c) command
sed ‘[address]c\
Changed Line1\
Changed Line2\
Changed Line3’ file.txt
SED: Inserting and Appending Lines
- Inserting: (i) command
sed ‘[address]i\
Inserting new line1\
Inserting new line2\
Inserting new line3’ file.txt
- Appending: (a) command
sed’[address]a\
Appending new line1\
Appending new line2\
Appending new line3’ file.txt
SED: Printing
Printing: (p) option
Used to print the lines :
- sed '2p' file.txt
Print excluding the non addressed lines or input lines: (-n ) is used to suppress the automatic printing
- sed -n '2p' file.txt
- sed -n '/pattern/p' file.txt
- sed –n ‘/patern/,$p’ file.txt
SED: Multiple Commands
Using multiple edit option : (e) option
Using file option : (f) option
Method 1: sed –e ‘1,4s/text1/text2/; 3,5s/text3/text4/; 1,2d’ file.txt
command1 command2 command3
Method 2:
sed –e ‘
>1,4s/text1/text2/
>3,5s/text3/text4/
>1,2d’ file.txt
Method 3: sed –e ‘1,4s/text1/text2/’ -e ‘3,5s/text3/text4/’ -e ‘1,2d’ file.txt
Method 4:
- sed -f commandfile.sed file.txt
- sed -f commandfile.sed file1.txt file2.txt
SED: Printing Line Numbers
: (=) option
sed ‘=‘ file.txt
SED: Transforming Characters
: (y) option
sed ‘y/str1/str2’ file.txt
ex. sed ‘y/:${}/;#[]/’ data.txt
Note: It is Global and not limited to an occurrence.
List: (l) option- Used to find out tabs, spaces, etc available in a file.
Quit: (q) option
Write: (w) option: Write or copy selected records from one file to another
AWK:
- Aho , Weinberger, and Kernigham
- Text processing tool
- This is most widely used to extract fields from large files and create formatted reports.
- Can incorporate concepts of programming language like using variables and constructs.
- Reads one line at a time from the input file or input stream and applies commands on it.
- Do not make any changes to the input file.
- Syntax : awk options ‘selection_criteria {command}’ file(s)
- Default action is print.
AWK: Print
It is the default action of AWK.
Syntax:
- awk ‘selection_criteria {print}’ file
- awk ‘selection_criteria’ file
- awk ‘{print}’ file
Field Built In Variables:
$1 First field of the input data file or stream.
$2 Second field of the input data file or stream.
$3 Third field of the input data file or stream.
……………………
……………………
……………………
……………………
$NF Last field of the input data file or stream.
Note: '$NF' represents the total no of fields or the last record of input file or stream.
Shell Scripting
- Unix/Linux Shell Scripting can be defined as a collection of Unix/Linux commands used along with other programming concepts like printing, taking user inputs, use of variables, decision loops, etc to perform a Job.
- The extension of a shell script is (.sh)
- The permission of a shell script file should have execute permission mandatorily and Read, Write, and Execute preferably.
- Since Shell script uses Unix/Linux commands which are themselves interactive in nature, Shell scripts are also interactive in Nature
Conclusion
In conclusion, this essential handbook has equipped you with the fundamental knowledge of Linux and Unix commands, shell scripting concepts, and operating system principles. Whether you're a beginner or an expert, this guide has provided you with a solid foundation to master Linux and Unix. Apply these concepts, practice regularly, and take your skills to the next level. We hope you like it, Please do comment with your suggestions and feedback.
Happy learning!
