tldr effective-shell

Executables are programs your system can use; your shell just calls out to them.

Executables are just files with the 'executable' bit set. If I execute the cat command, the shell will search for an executable named cat in my $PATH. If it finds it, it will run the program.

$PATH is the standard environment variable used to define where the shell should search for programs.

The shell will start with the earlier locations and move to the later ones. This allows local flavours of tools to be installed for users, which will take precedence over general versions of tools.

Executables don't have to be compiled program code, they can be scripts. If a file starts with #! (the 'shebang'), then the system will try to run the contents of the file with the program specified in the shebang.

Builtins are very shell-specific and usually control the shell itself

"Built-Ins" 的指令是和 shell 强相关的，换言之，某个命令，可能 bash 有，但 zsh 就没有。

之所以需要有 "Built-Ins" 命令，是因为内建的会执行得更快，而且作用的范围更广。

Some commands are a builtin so that they can function in a sensible manner. For example, cd command changes the current directory - if we executed it as a process, it would change only the directory for the cd process itself, not the shell, making it much less useful.

Echo is builtin because the shell can run much more quickly by not actually running a program if it has its own built in implementation.

Functions are powerful ways to write logic but will normally be shell-specific.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

  wsl_proxy_on() {
      local port=10810
      export windows_host=`cat /etc/resolv.conf | grep nameserver | cut -d' ' -f 2`
      export ALL_PROXY=socks5://$windows_host:$port
      export HTTP_PROXY=$ALL_PROXY
      export http_proxy=$ALL_PROXY
      export HTTPS_PROXY=$ALL_PROXY
      export https_proxy=$ALL_PROXY
      echo -e "proxy on"
  }

  wsl_proxy_off() {
      unset ALL_PROXY HTTPS_PROXY https_proxy HTTP_PROXY http_proxy
      echo -e "proxy off"
  }

Aliases are conveniences for human operators, but only in the context of an interactive shell.

1
2
3
4
5
6

  # ---------------------------------------------------------------- #
  # alias
  # ---------------------------------------------------------------- #
  alias wpon=wsl_proxy_on
  alias pbcopy='clip.exe'
  alias pbpaste="powershell.exe -command 'Get-Clipboard' | tr -d '\r' | head -n -1"

• manutal sections: man 1 intro

  Section 1

  Executable programs or shell commands

  Section 2

  System calls (functions provided by the kernel)

  Section 3

  Library calls (functions within program libraries)

  Section 4

  Special files (usually found in /dev)

  Section 5

  File formats and conventions (e.g. /etc/passwd)

  Section 6

  Games

  Section 7

  Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)

  Section 8

  System administration commands (usually only for root)

  Section 9

  Kernel routines (Non standard)

• man -k : 模糊搜索

pwd

Print Working Directory

ls

List Directory Contents

cd

Change Directory

.

This folder

..

The parent folder

~

Home (cd without any parameters)

-

Go back to the last location you moved to

1
2
3
4
5
6
7

  alias ..="cd .."
  alias ...="cd ../.."
  alias ....="cd ../../.."
  alias .....="cd ../../../.."
  alias ......="cd ../../../../.."
  alias .......="cd ../../../../../.."
  alias ........="cd ../../../../../../.."

One thing we might want to do is quickly move from one location to another, then go back again.

pushd

'pushes' a new working directory onto a stack - moving you there.

popd

'pops' the working directory off the top of the stack

dirs

查看 poshd 和 popd 操作后的堆栈情况

• ls

• rm

• rmdir

• cp

• mv

• mkdir

• tree

• file

• find

  您当然可以使用 alias 设置别名来简化上述操作，但 shell 的哲学之一便是 寻找（更好用的）替代方案。

  记住，shell 最好的特性就是您只是在调用程序，因此您只要找到合适的替 代程序即可（甚至自己编写）。

  例如，fd 就是一个更简单、更快速、更友好的程序，它可以用来作为 find 的 替代品。它有很多不错的默认设置，例如输出着色、默认支持正则匹配、支持 unicode 并且我认为它的语法更符合直觉。以模式 PATTERN 搜索的语法是 fd PATTERN 。 – Shell 工具和脚本

Ctrl key is called the Control Key is that it is used to send control sequences to the computer.

Ctrl-C

Terminates the current program

Ctrl-V

Verbatim Insert, it tells the shell to write out the next keystroke you give it.

熟悉的 cv 是 shell 中的控制指令，没法完成粘贴复制，换成 Ctrl + Shift + C 和 Ctrl + Shift + V

• pbpaste | pbcopy : 可以移除格式

• pbpaste | sort | uniq | tr ' ' '_' | sed 's/$/@simpsons.com/' | pbcopy : 快速转换 excel 表格的内容

• pbpaste | grep 评选中 | grep 前端 | sort -g | cut -f 1,4 : 快速查看评选的改进项

grep

g/re/p , This command ran on all lines (g, for global), applied a regular expression (re, for regular expression) and then printed (p for print) the results.

• 结合多个管道过滤内容

• -v 排除 grep 的内容

head / tails

head is used to extract part of the top of a file and tail is used to extract part of the end of a file.

• head ~/effective-shell/data/top100.csv

• head -n 3 ~/effective-shell/data/top100.csv

• tail $HISTFILE

• tail -f $HISTFILE

• head ~/effective-shell/data/top100.csv | tail -n +2 去掉表头，从第二行开始输出

tr (translate characters)

Perform a simple substitution of characters.

• head -n 1 ~/effective-shell/data/top100.csv | tr ',' '\n'

• head -n 1 ~/effective-shell/data/top100.csv | tr ',' '\n' | tr -d '"'

• echo "Welcome to the shell" | tr 'shell' 'machine'

• echo "Use your inside voice..." | tr '[[:lower:]]' '[[:upper:]]'

cut

The cut command splits a line of text, using a given delimiter.

• cut -d',' -f 3 ~/effective-shell/data/top100.csv | head

• echo "2020-11-29T12:50:52.762Z: info - Request: GET /svg/menu.svg" | cut -c 12-19

• echo "2020-11-29T12:50:52.762Z: info - Request: GET /svg/menu.svg" | cut -c 27-

rev

Reverse the given input.

• pwd | rev | cut -d'/' -f 1 | rev

sort and uniq

The uniq command removes duplicate lines from a stream of text.

• cut -c 27- ~/effective-shell/logs/web-server-logs.txt | grep error | sort | uniq

less

Open a file for interactive reading, allowing scrolling and search.

Xargs

The xargs [build and execute commands] command takes input, uses the input to create commands, then executes the commands. I tend to remember it as "Execute with Arguments" as the name xargs sounds a little odd!

By default xargs take the input, joins each line together with a space and then passes it to the echo command.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

  # 将找到的文件通过 xargs 拼接后，传给 rm 执行
  touch file{1..100}.txt
  # fail
  find . -empty | rm
  # success
  find . -empty | xargs rm

  touch "chapter "{1,2,3}.md
  find . -type f
  # ./chapter 1.md
  # ./chapter 2.md
  # ./chapter 3.md

  # 上面的文件名有空格
  # -print0 给每一个 item 追加一个特殊 'null' 字符
  # -0 告诉 xargs 每个元素是用特殊的 'null' 字符分隔的（而不是空格）
  # 这么做，可以避免一些空格，tab，引号造成的问题，建议总是带上这两个选项
  find . -type f -print0 | xargs -0 -t rm

  touch file{1..5}
  # 告诉 xargs 最多用多少行去执行命令
  find . -type f | xargs -L 3 echo
  ./file1 ./file2 ./file3
  ./file4 ./file5

  # 默认 xargs 将参数放到最后
  # 有时想把参数放在别的地方，可以使用 -I 选项
  # -I {} 表示将参数把 {} 当作 placeholder，然后可以在后面任何地方多次去用
  find . -name "*.txt" -print0 | xargs -0 -t -I {} cp {} ~/backups
  cp ./file2.txt /home/dwmkerr/backups
  cp ./file3.txt /home/dwmkerr/backups
  cp ./file1.txt /home/dwmkerr/backups

  # -p 在执行命令时进行询问
  kubectl get pods -o name | xargs -L 1 -p kubectl delete

  # -d (delimiter) 告诉 xargs ，输入是通过什么分隔符，分割成多个参数的
  echo $PATH | xargs -d ':' -p -L 1 ls

• regex101

My general advice for regular expressions is start simple and add complexity only if you need it.

We can build regular expressions using an 'iterative' process, starting with the basics, then adding more features as we need them.

Let's take validating an email address as an example. The way I would build a regular expression to validate an email address would be to use the following steps:

• Create a small list of valid email address

• Add some items to the list which look 'kind of' valid but are not quite right

• Build a regular expression which matches the correct email address

• Refine the expression to eliminate the invalid addresses

In most cases this will be sufficient.

I would advise that you keep expressions simple if possible - if they are getting too complex then break up your input or break up the processing into smaller chunks of work!

Remember that a regular expression does not have to be the only way you validate input. You might use a regular expression to do a quick check on a form on a website to make sure that an email address has at least the correct structure, but you might then use a more sophisticated check later on (such as sending the user an activation email) to actually confirm that the address actually belongs to the user.

为什么不是 2>1 ? 这样实际是将 stderr 重定向到文件 1 中，而不是重定向到 stdout 。

如果想重定向到 stdout ，就需要使用 &1, 表达 stdout 的 file descriptor。

2 >&1, 2> &1 也是不对的，不能有空格, 两者都会被当作命令执行。

对于前者， 2 会被当作命令执行，可以通过 type 2 看看对应的是什么命令；

对于后者，&1 也会被当作命令解析，此时 & 无法解析对应的命令，就会报错。

1
2
3
4
5
6
7

  # Show the most commonly used commands.
  tail ~/.bash_history -n 1000 \
      | sort \
      | uniq -c \
      | sed 's/^ *//' \
      | sort -n -r \
      | head -n 10

Be careful when you split lines up - the continuation character must be the last character on the line. If you add something after it (such as a comment) then the command will fail.

1
2
3

  chmod +x ~/scripts/common.sh

  ~/scripts/common.v1.sh

但这种方式由于没有指定执行脚本的 shell 程序，如果你用的是 Bash，那就是 用 Bash 执行，如果用的是 zsh，那就是 zsh 执行。

shebangs 指定的程序，需要通过完整路径指向程序的可执行文件，而如果指向的程序不存在，就会出错。

你可以 type 命令找到某个程序的路径，但会有些麻烦。

此时就可以利用 env (set environment and execute command) ，它会去执行命令，并从 $PATH 上找到命令所在的路径。

1
2
3

  #!/usr/bin/env bash

  echo "Hello from Bash"

Using a shebang to specify the exact command to run, and then using the env command to allow the $PATH to be searched is generally the safest and most portable way to specify how a shell script should run.

Arrays in Bash start at index zero. Arrays in the Z-Shell start at index one - this can cause confusion and mistakes in scripts so it is something you might have to consider if you are writing scripts that can be used by either shell.

It's important to use curly braces around your array expressions.

1
2
3
4

  days=("Monday" "Tuesday" "Wednesday" "Thursday" "Friday" "Saturday" "Sunday")

  echo "The first day is: ${days[0]}"
  echo "The last day is: ${days[6]}"

1
2
3
4
5
6
7
8
9

  # Create an associative array called 'book'.
  declare -A book

  # Set some values on the array.
  book[title]="Effective Shell"
  book[author]="Dave Kerr"

  # Show one of the values.
  echo "Book details: ${book[title]} - ${book[author]}"

If you find yourself using associative arrays, I expect that there is a good chance you are trying to do something that is more complex than is suitable for a shell script.

There is often a lot of confusion about a specific topic in the shell - when should you surround a variable in quotes?

This might sound like a purely stylistic question, but surrounding a variable in quotes can dramatically change how your script works.

Quoting Tips:

• Use double quotes most of the time - they will handle variables and sub-shells for you and not do weird things like word splitting

• Use single quotes for literal values

• Use no quotes if you want to expand wildcards

1
2

  message='   ~~ Save $$$ on with ** "this deal" ** ! ~~   '
  echo "$message"

1
2
3
4
5
6

  deal="Buy one get one free"
  message="Deal is '$deal' - save \$"
  echo "$message"

  # `` 内的也在一个 sub-shell 执行，但应该避免使用，统一使用 $() 的形式
  echo "The date is `date`"

The read (read from standard input) command can be used to read a line of text from standard input. When the text is read it is put into a variable, allowing it to be used in our scripts.

The read command reads a line of text from standard input and stores the result in a variable called REPLY. We can then use this variable to use the text that was read.

In general you should provide a variable name for read - it will make your script a little easier to understand. Not every user will know that the $REPLY variable is the default location, so they might find it confusing if you don't provide a variable name. By specifying a variable name explicitly we make our script easier to follow.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

  # 默认存在 $REPLY
  echo "What is your name?"
  read
  echo "Hello, $REPLY"

  # 指定存值的变量
  echo "What is your name?"
  read name
  echo "Hello, ${name}"

  # prompt (bash)
  read -p "Please enter your name: " name
  echo "Hello, $name"

  # prompt (zsh)
  read "?Please enter your name: "
  echo "Hello, $REPLY"

  # The -s (silent) flag can be used to hide the input as it is being written.
  read -s -p "Enter a new password: " password
  masked_password=$(echo "$password" | sed 's/./*/g')
  echo ""
  echo "Your password is: $masked_password"

  # Limiting the Input
  # Use the -n flag with the value 1 to specify that we want to read a single character only.
  read -n 1 -p "Continue? (y/n): " yesorno
  echo ""
  echo "You typed: ${yesorno}"

Here 文档, 一种输入多行字符串的方法。

格式： $((expression))

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

  case <expression> in
      pattern1)
          <pattern1-commands>
          ;;
      pattern2 | pattern3)
          <pattern2and3-commands>
          ;;
      ,*)
          <default-commands>
          ;;
  esac

以 case 开头，以 esac 结束（反转了词序）。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

  read -p "Yes or no: " response
  case "${response}" in
      y | Y | yes | ok)
          echo "You have confirmed"
          ;;
      n | N | no)
          echo "You have denied"
          ;;
      ,*)
          echo "'${response}' is not a valid response"
          ;;
  esac

  read -p "Yes or no: " response
  case "${response}" in
      [yY]*)
          echo "You have (probably) confirmed"
          ;;
      [nN]*)
          echo "You have (probably) denied"
          ;;
      ,*)
          echo "'${response}' is not a valid response"
      ;;
  esac

The while loop is a loop that executes commands until a certain condition is met.

基本结构：

1
2
3
4
5
6

  while <test-commands>
  do
      <conditional-command 1>
      <conditional-command 2>
      <conditional-command n>
  done

例子：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

  # Create an empty array of random numbers.
  random_numbers=()

  # As long as the length of the array is less than five, continue to loop.
  while [ ${#random_numbers[@]} -lt 5 ]
  do
      # Get a random number, ask the user if they want to add it to the array.
      random_number=$RANDOM
      read -p "Add $random_number to the list? (y/n): " choice

      # If the user chose 'y' add the random number to the array.
      if [ "$choice" = "y" ]; then random_numbers+=($random_number); fi
  done

  # Show the contents of the array.
  echo "Random Numbers: ${random_numbers[@]}"

1
2
3

  while read line; do
      echo "Read: $line"
  done < ~/effective-shell/data/top100.csv

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

  while true
  do
      echo "1) Move forwards"
      echo "2) Move backwards"
      echo "3) Turn Left"
      echo "4) Turn Right"
      echo "5) Explore"
      echo "0) Quit"

      read -p "What will you do: " choice
      if [ $choice -eq 0 ]; then
          exit
      fi
      # The rest of the game logic would go here!
      # ...
  done

The until loop operates just like the while loop, except that it runs until the test commands return success.

As long as the test commands do not return success, the loop will run the conditional commands. After the conditional commands have been run, the loop goes 'back to the start' and evaluates the test commands again.

In general I would recommend using while loops rather than until loops. While loops are going to be more familiar to readers as they exist in many programming languages - until loops are a little more rare. And you can easily turn any until loop into a while loop by simply inverting the test commands you are running.

1
2
3
4
5
6

  until <test-commands>
  do
      <conditional-command 1>
      <conditional-command 2>
      <conditional-command n>
  done

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

  # until loop
  # Create an empty random number string - we're going to build it up in the loop.
  random_number=""

  # Keep on looping until the random number is at least 15 characters long.
  until [ "${#random_number}" -ge 15 ]
  do
      random_number+=$RANDOM
  done
  echo "Random Number: ${random_number}"

  # while loop
  random_number=""
  while [ "${#random_number}" -lt 15 ]
  do
      random_number+=$RANDOM
  done
  echo "Random Number: ${random_number}"

You can use the set (set option) command to set an option in the shell. There is an option that tells the shell to exit when a command fails.

The 'set' command allows you to turn on and turn off shell options. The 'e' option means 'exit if any command exits with a non-zero status'.

1
2
3
4
5
6

  #!/usr/bin/env sh

  # Exit if any command fails.
  set -e

  # ...

One thing to be aware of is that the set -e option only affects the final command of a pipeline.

To ensure that the shell terminates if a command in a pipeline fails we must set the pipefail option: set -o pipefail

1

  grep '[:space:]*#' ~/effective-shell/scripts/common.sh | tr 'a-z' 'A-Z'

You can use the trap (trap signals and events) command to specify a set of commands to run when the shell receives signals, or at certain points such as when the script exits or a function returns.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

  # Create a temporary folder for the effective shell download.
  source="https://effective-shell.com/downloads/effective-shell-samples.tar.gz"
  tmp_dir=$(mktemp -d 2>/dev/null || mktemp -d -t 'effective-shell')
  tmp_tar="${tmp_dir}/effective-shell.tar.gz"

  # Define a cleanup function that we will call when the script exits or if
  # it is aborted.
  cleanup () {
      if [ -e "${tmp_tar}" ]; then rm "$tmp_tar}"; fi
      if [ -d "${tmp_dir}" ]; then rm -rf "${tmp_dir}"; fi
  }

  # Cleanup on interrupt or terminate signals and on exit.
  trap "cleanup" INT TERM EXIT

  # Download the samples.
  curl --fail --compressed -q -s "${source}" -o "${tmp_tar}"

  # Extract the samples.
  tar -xzf "${tmp_tar}" -C "${tmp_dir}"

You can use the getopts (parse option arguments) command to process the arguments for a script or function.

处理 -h, -e 等选项

The select compound command prints a menu and allows the user to make a selection. It is not part of the Posix standard, but is available in Bash and most Bash-like shells.

1
2
3
4
5

  select fruit in Apple Banana Cherry Durian
  do
      echo "You chose: $fruit"
      echo "This is item number: $REPLY"
  done

You will often see a nice little trick that allows you to change the current directory for a specific command, without affecting the current directory for the shell.

The brackets around the statements mean that these commands are run in a sub-shell. Because they run in a sub-shell, they change the directory in the sub-shell only, not the current shell. This means we don't need to change back to the previous directory after the commands have completed.

1

  (mkdir -p ~/new-project; cd ~/new-project; touch README.md)