Shell

Echo

-n do not output the trailing newline

-e enable interpretation of backslash escapes (example: \ prints )




nicky $: ps fxo pid,cmd
 PID CMD
  7997 emacs --daemon
  8117  \_ /usr/bin/python /usr/bin/ipython -c ?import sys, site?site.addsitedir('.')?import anaconda_mode?anaconda_mode.main(sys.argv[1:])?

to show up an obviousness between an empty variable and an unset variable:

$ set -u
echo $test
zsh: test: parameter not set

All variables will be automatically exported to child processes

$ set -a

Do not executed script commands but show them

$ set -n

Raise an exception if using of non set variable

#! /bin/sh
set -u (set -o nounset)
 
if [ -z "$string" ]; then
    echo "empty string"
fi

Follow the program

#! /bin/sh
set -x (set -o xtrace)

tee

plumbig tool. Put that tool into the pipeline to get an outing while sending an extra copy in any given files. Ideal to debug.

$ ps aux | tee /dev/tty | grep postgres > postgres_process.log

bc – An arbitrary precision calculator language

$ bc -l
25 * a(1)
quit

#! /bin/sh
total=$(echo "45.6 * 2.3" | bc -l)
echo $total

Internal variables

The $? parameter give us the return code of last tool executed:

$ mkdir /folder
$ $?

Then you can compare if it is equals to 0 …

Another solution would be to use {}

{ psql -h localhost -U user1 db -c "insert ...." 2 &> sql_trace.log; } 
|| 
{ echo >&2 "Can not insert to the database"; exit 2; }

Create a directory and go inside directly:

$ mkdir my_folder && $_

Process indentifier of current shell (main shell)

$ echo $$

Process indentifier of current father shell: PPID ( Parent PID)

$ echo $PPID

Process indentifier of the last command executed in background

$ echo $!

User identifier (used by the kernel to check every authorization needed)

$ echo $UID

$ echo $EUID

Use UID to know the user who is using the script and EUID to know which authorizations were used.

List all arguments given to a script:

$ echo "$@"

IFS (Internal Field Separator) : Define the default operator used to cut words got on a line
By default: space, tab and carrier return

Note: That variable is never imported to child shells
[[https://www.admin-linux.fr/ifs-le-separateur-standard-du-shell/]]

extraction of sub string

$ variable="totofjweofjwefjw"
$ echo ${variable:5:3}
fjw
 
$ echo ${variable:1}
otofjweofjwefjw

Operator # and ##

${var#Pattern}
${var##Pattern}

${var#Pattern} Remove from $var the shortest part of $Pattern that matches the front end of $var.

${var##Pattern} Remove from $var the longest part of $Pattern that matches the front end of $var.

Examples for #:

$ querty="QWERTYUIOPASDFGHJKLMNBVCXZ"
echo ${querty#QWER}
TYUIOPASDFGHJKLMNBVCXZ
$ echo ${querty#*}
QWERTYUIOPASDFGHJKLMNBVCXZ
echo ${querty#*[URI]} # removing from beginning until the first letter found
TYUIOPASDFGHJKLMNBVCXZ

* -> any string (may be empty)

? -> any character

\* -> *

\\ -> \

\? -> ?

[A-Z][0-9] ^ or ! -> The matching excludes any character that belongs to the expression.

Parameter substitution (lentgh, substitution …):

http://www.tldp.org/LDP/abs/html/parameter-substitution.html#PSUB2

Get the file extension:

$ schema_file=schema.sql
$ echo "${filename##*.}"
sql

Get the length of variable:

$ schema_file="schema_blablalala"
$ echo "${#schema_file}"
17

Exceptions:

For an array, use

${#array[*]}
${#array[@]}
give the number of elements in the array.

Input/Output with cat

 $ cat << EOF - Choose a function to start 1 - Create 2 - List 0 - Quit EOF 
while [ $"user_input" != 0 ]; 
do    echo -n "Please choose: "    
read    case "$user_input" in        
1) echo "kkkk";;        
2) ...        
0) ...        
*) echo "Not a valid choice"     
# [yY]*) echo "lala";     
esac done

Comparison

^ Tool ^ Description ^
| -d | is a directory |
| -f | is a file |
| -p | is a pipe |
| -S | is a socket |
| -h or -L | is a symbolink link |
| -n | string not null |
| -z | string null |
| -w | can write to the file |
| -x | executable |
| -s | File size > 0 |
| -t | TTY |
| file1 -nt file2 | last modification of file1 is more recent |
| file1 -ot file2 | last modification of file1 is older |
| -e | File exists |

getopts

http://wiki.bash-hackers.org/howto/getopts_tutorial

Signal

^ Signal ^ Description ^
| SIGUSR1 | Signal use by personal application |
| SIGUSR2 | Signal use by personal application |
| SIGHUP | Reload configuration (if not handle by the application, can kill the application: see nohup) |
| SIGTERM | KILL (nice) |
| SIGINT | KILL |
| SIGQUIT | KILL |
| SIGTERM | KILL (nasty)|

SIGTERM, SIGINT, SIGQUIT, SIGKILL => from nice to nasty

Use the command kill to send a signal

kill -TERM $PID
kill -HUP $PID

Note: PID -1 means all processus

Capturing signals

 
function cleanup {
  # clean resources
  rm -r tmp_file
}
 
trap cleanup EXIT SIGHUP SIGINT SIGTERM

Tools

^ Tool ^ Description ^
| flock | Lock file |
| mkfifo | Pipe with file [[https://linux.die.net/man/3/mkfifo]] |
| file | file is a standard Unix program for recognizing the type of data contained in a computer file. (wikipedia)|
| fmt | The fmt command in Unix is used to format natural language text for humans to read (wikipedia) |
| seq | generate a seq [0-10] |

https://en.wikipedia.org/wiki/List_of_Unix_commands

Various

Copy some folder somewhere else

$ cp /home/nicky/{work,document} backups/

select-do

select variable in words
do
   case $answer in
       Word1 )
          ****
          *;;
       word2 )
          ;;
   esac
done

Source command

$ source local_file.sh

Note: Do not need to be executable and can update the current shell environment

Transmit arguments received from parent process to the child process

$0 "$@" &

This is usefull when you want to parallelize a huge migration. See also $! and wait (wait without arguments waits until all child processes are done)

compound statements

When you use the () to execute a command, it will be executed in a child process and will not update any stuff to the father process.

Input/output process

Redirect stderr to stdout

$ grep lala myfile > file 2>&1

Note: 2>&1 means we copy the configuration from stdout to stderr (can use &>file but not common).

Print an message to stderr

$ echo "lalal" >&2

Note: >&2 means we copy the configuration from stderr to stdout

Buffer

^ Buffer ^ Description ^
| stderr | No buffer |
| stdout (TTY) | line-buffered |
| stdout (FILE) | raw-bloc-buffered |

Buffer: Stderr does not have a buffer. So when you redirect stderr to stdout, the synchronization is not perfect.

Log rotate

Dailyfile

This small tool copies the content of its standard input into daily switching files.

https://github.com/cpb-/dailyfile

Regular expression (Grep, Sed, AWK)

https://github.com/oyoun/Command-line-text-processing/

https://www.rexegg.com/

https://regexr.com/

Grep

$ grep -F "toto" file

or

$ cat file | grep -F "toto"

^ Option ^ Description ^
| -F | Not a regular expression but a simple string |
| -E | Extented Regular expression (by default simple) |
| -i | case insensitive |
| -v | Print lines that do not contain the regular expression |

Symbols:

^ Meta-character ^ Description ^
| . | Any character |
| * | 0, 1 or many occurences of the previous element |
| ? | 0 or 1 of the previous element |
| + | 1 or many occurences of the previous element |
| vertical bar | alternative |
| [a-z] | range |
| circumflex accent | Beginning of the string |
| $ | End of the string |
| \ | Print a special character |
| {n,m} | At least n occurences to m of the previous element |
| {n,} | At least n occurences of the previous element |
| {0, m} | Max m occurences but can be 0, 1 .. of the previous element |
| {n} | Exact occurences of the previous element |
| ( ) | capturing group |
| \n | Back reference to n th element |
| [] | character class |

SED

^ Option ^ Description ^
| -s | substitution |
| -d | delete |
| -p | print |

First line

$ sed -ne '1p' < file

Last line

$ sed -ne '/^$/p' < file

Print lines that are not empty

$ sed -ne '/^$/!p' < file

Print lines that match the pattern

$ sed -ne '/pattern_to_find/p' < file

Delete lines that match a pattern

$ sed -e '/pattern_to_find/d' < file

opposite:

$ sed -e '/pattern_to_find/!d' < file

Substitution

^ Meta-character ^ Description ^
| g | replace every pattern found |
| p | print the line that was updated |
| w | send the result to a file |
| i | substitute the pattern between n,m |

$ echo "windows linux linux " | sed -e 's/windows/linux/g'
$ echo "windows linux linux " | sed -e 's/windows/linux/1'

Note: only replace the first patter found

&:

 $ echo "tata" | sed -e "s/tata/&<\/title>/g" 
</pre></p>

AWK

https://connect.ed-diamond.com/GNU-Linux-Magazine/GLMF-131/AWK-le-langage-script-de-reference-pour-le-traitement-de-fichiers

http://www.grymoire.com/Unix/Awk.html

http://www.funix.org/fr/unix/awk.htm

Publier pour ne pas oublier