Linux commands to make you a CLI Wizard

What is Piping?

An introduction to pipes and named pipes in Linux

In Linux, the pipe command lets you sends the output of one command to another. Piping, as the term suggests, can redirect the standard output, input, or error of one process to another for further processing.

https://opensource.com/article/18/8/introduction-pipes-linux

grep command

Useful Options

1. e <pattern> or -regexp=Patterns - You can pass a certain regular expression to be used while searching for strings in text. If this option is used multiple times or is combined with the f (-file) option, it then searches for all patterns given.

2. f FILE or -file=FILE - Instead of passing the patterns as options in a command, you can store them all in a single file (separated by lines) and then pass the filename with f option to search for all the patterns given.

3. v or -invert-match - As the name suggests, this option selects everything which does not matches the given pattern.

4. c or -count - Print the number of lines that match the given patterns. This suppresses the normal output and only prints the count.

5. C Num or -context NUM - Print adjoining NUM lines when this option is passed.

6. m NUM, -max-count=NUM - Stop reading a file after NUM matching lines.

grep(1) - Linux manual page

GREP(1) User Commands GREP(1) grep - print lines that match patterns grep searches for PATTERNS in each FILE. PATTERNS is one or more patterns separated by newline characters, andgrep prints each line that matches a pattern. TypicallyPATTERNS should be quoted whengrep is used in a shell command.

https://man7.org/linux/man-pages/man1/grep.1.html

Examples

• Basic: Checking for a particular process. Let's say if you wish to check for the Kafka process running on a particular machine.

ps -ef | grep kafka

• Intermediate: Find groups of n digits, but no more than n.

# Using Extended regular expressions (by -E flag)
grep -E '(^|[^0-9])[0-9]{4}($|[^0-9])' file

• Advanced: Match Valid Email Addresses

$ grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" data.txt

find command

Useful Options

1. exec <expression> \; - This option is used for adding subsequent operations that needed to be performed on the search result files/directories. All following expressions to exec option are taken for consideration until a ; is encountered. The string {} is replaced by the current file name being processed. Make sure you escape (\) the ; to prevent your shell from interpreting it as a new line.

2. regex <expression> - I don't think I need to explain this much. But one important thing to note is that this regex works on the whole file path and not just on the filename. By default, the regular expressions understood by find are Emacs Regular Expressions. But this can be changed by regextype option.

3. name <pattern> - Now if you want to apply patterns only on the filename, you can use this option. You can see the Intermediate example below. Because the leading directories are removed, the file names considered for a match with name will never include a slash, so name a/b will never match anything.

4. empty - This is a test option and tests if the file is empty and is either a regular file or a directory.

5. maxdepth N - Helps configure how much the find command should descend in the directory tree to find the files/directories from the starting point. There is also an opposite of this - mindepth N which starts searching from the given minimum tree depth.

6. type c - Search for a particular type of entity which could be a regular file (f), directory (d), a character (c), a symbolic link (l), socket (s), etc.

find(1) - Linux manual page

FIND(1) General Commands Manual FIND(1) find - search for files in a directory hierarchy find [-H] [-L] [-P] [-D debugopts] [-Olevel] [starting-point...] [expression] This manual page documents the GNU version offind.

https://man7.org/linux/man-pages/man1/find.1.html

Examples

• Basic: Get all hidden files in the home directory

find /home -name ".*"

• Intermediate: Finding and deleting all files except with particular extensions

find . -type f ! -name '*.yaml' -delete

• Advanced: Clean out temporary files

find . \( ‑name a.out ‑o ‑name '∗.o' ‑o ‑name 'core' \) ‑exec rm {} \;

sed command

Useful Options

sed(1) - Linux manual page

Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.

https://man7.org/linux/man-pages/man1/sed.1.html

sed, a stream editor

This file documents version 4.8 of GNU sed, a stream editor. Copyright © 1998-2020 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.

https://www.gnu.org/software/sed/manual/sed.html

1. e script - add the script to the commands to be executed

2. f script-file - add the contents of a script file to the commands to be executed

3. i SUFFIX - edit files in place (makes backup if SUFFIX supplied). The older file is saved with the suffix added to the filename.

Script Synopsis

1. /g - replace globally

2. /3g replace from the third occurrence to globally

Examples

• Basic: Consider a file containing text - Apples are good for health. Replace Apples with Mangoes.

sed 's/Apples/Mangoes/g' file.txt

• Intermediate: Enclose the first letter of every word in a sentence inside round brackets.

echo "Apples are good for health" | sed 's/\(\b[A-Z]\)/\(\1\)/g'

# Output
(A)pples (a)re (g)ood (f)or (h)ealth

• Advanced: Removing all the commented & empty lines from a file.

sed -e 's/#.*//;/^$/d' testfile.txt

awk command

Useful Options

awk(1p) - Linux manual page

This manual page is part of the POSIX Programmer's Manual. The Linux implementation of this interface may differ (consult the corresponding Linux manual page for details of Linux behavior), or the interface may not be implemented on Linux.

https://man7.org/linux/man-pages/man1/awk.1p.html

1. F - Helps define the input field separator. Default is space.

2. f - If you don't wish to pass the whole awk program script from CLI, you can also use an external file to store the awk program.

Examples

• Basic: Print the first word of every line in a file

awk '{print $1}' myfile

• Intermediate: Let's say if you wish to remove inactive replica sets in a Kubernetes cluster.

$ kubectl get all | grep replicaset.apps
NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/limosyn-com-ghost-6d47f4c9b9   1         1         1       7d9h
replicaset.apps/limosyn-com-mysql-dffbd7cdb    1         1         1       7d9h
replicaset.apps/nuxt-frontend-546dff5b99       1         1         1       7d2h
replicaset.apps/nuxt-frontend-67f5659d95       0         0         0       7d8h


# Delete the 4th replica
$ kubectl delete $(kubectl get all | grep replicaset.apps | awk '{if ($2 + $3 + $4 == 0) print $1}')

• Advanced: Print the length of the longest line present in the file.

awk '{ if (length($0) > max) max = length($0) } END { print max }' sample.txt

comm command

1. Non-matching items of the first file

2. Non-matching items of the second file

3. Matching items of both files

Useful Options

1. 1/-2/-3 - Does not display column 1/2/3 respectively.

2. i - Case insensitive comparison of lines

3. -check-order - Check the order of the input, even if all input lines are pairable.

Examples

$ cat words.txt
Apple
Banana
Orange
India
US
Canada

$ cat countries.txt
India
US
Canada

$ comm -23 < (sort words.txt | uniq) < (sort countries.txt | uniq)
Apple
Banana
Orange

Last Words

Be Lazy, do crazy