tldr effective-shell

Executables

"Built-Ins"

Functions

wsl_proxy_on() {
    local port=10810
    export windows_host=`cat /etc/resolv.conf | grep nameserver | cut -d' ' -f 2`
    export ALL_PROXY=socks5://$windows_host:$port
    export HTTP_PROXY=$ALL_PROXY
    export http_proxy=$ALL_PROXY
    export HTTPS_PROXY=$ALL_PROXY
    export https_proxy=$ALL_PROXY
    echo -e "proxy on"
}

wsl_proxy_off() {
    unset ALL_PROXY HTTPS_PROXY https_proxy HTTP_PROXY http_proxy
    echo -e "proxy off"
}

Aliases

# ---------------------------------------------------------------- #
# alias
# ---------------------------------------------------------------- #
alias wpon=wsl_proxy_on
alias pbcopy='clip.exe'
alias pbpaste="powershell.exe -command 'Get-Clipboard' | tr -d '\r' | head -n -1"

man

tldr

# install
# use npm
npm install -g tldr

# use python
pip3 install tldr

# use Homebrew
brew install tldr

# how to use
tldr git

cht.sh

curl cheat.sh/tar
curl cht.sh/curl
curl https://cheat.sh/rsync
curl https://cht.sh/tr

# keyword
curl cht.sh/~snapshot

# language name space
curl cht.sh/go/Pointers
curl cht.sh/scala/Functions
curl cht.sh/python/lambda

curl cht.sh/go/reverse+a+list
curl cht.sh/python/random+list+elements
curl cht.sh/js/parse+json
curl cht.sh/lua/merge+tables
curl cht.sh/clojure/variadic+function

# ask question
curl cht.sh/"How do I copy a folder in bash?"

# pick another answer
curl cht.sh/python/random+string
curl cht.sh/python/random+string/1
curl cht.sh/python/random+string/2

导航

pwd

Print Working Directory

ls

List Directory Contents

cd

Change Directory

.

This folder

..

The parent folder

~

Home (cd without any parameters)

-

Go back to the last location you moved to

alias ..="cd .."
alias ...="cd ../.."
alias ....="cd ../../.."
alias .....="cd ../../../.."
alias ......="cd ../../../../.."
alias .......="cd ../../../../../.."
alias ........="cd ../../../../../../.."

One thing we might want to do is quickly move from one location to another, then go back again.

pushd

'pushes' a new working directory onto a stack - moving you there.

popd

'pops' the working directory off the top of the stack

dirs

查看 poshd 和 popd 操作后的堆栈情况

文件 CRUD

剪贴板体操运动员 (Clipboard Gymnast)

文本操作

# 将找到的文件通过 xargs 拼接后，传给 rm 执行
touch file{1..100}.txt
# fail
find . -empty | rm
# success
find . -empty | xargs rm

touch "chapter "{1,2,3}.md
find . -type f
# ./chapter 1.md
# ./chapter 2.md
# ./chapter 3.md

# 上面的文件名有空格
# -print0 给每一个 item 追加一个特殊 'null' 字符
# -0 告诉 xargs 每个元素是用特殊的 'null' 字符分隔的（而不是空格）
# 这么做，可以避免一些空格，tab，引号造成的问题，建议总是带上这两个选项
find . -type f -print0 | xargs -0 -t rm

touch file{1..5}
# 告诉 xargs 最多用多少行去执行命令
find . -type f | xargs -L 3 echo
./file1 ./file2 ./file3
./file4 ./file5

# 默认 xargs 将参数放到最后
# 有时想把参数放在别的地方，可以使用 -I 选项
# -I {} 表示将参数把 {} 当作 placeholder，然后可以在后面任何地方多次去用
find . -name "*.txt" -print0 | xargs -0 -t -I {} cp {} ~/backups
cp ./file2.txt /home/dwmkerr/backups
cp ./file3.txt /home/dwmkerr/backups
cp ./file1.txt /home/dwmkerr/backups

# -p 在执行命令时进行询问
kubectl get pods -o name | xargs -L 1 -p kubectl delete

# -d (delimiter) 告诉 xargs ，输入是通过什么分隔符，分割成多个参数的
echo $PATH | xargs -d ':' -p -L 1 ls

正则表达式

2>&1

• 2>&1 的位置
  

  Bash (and most bash-like shells) process redirections from left to right, and when we redirect we duplicate the source.

  如果想将所有的输出 (包括 stderr) 重定向到一个文件，以下顺序得到的结果是不同的：

  • ls /usr/bin /nothing 2>&1 > all-output.txt

    2>&1

    duplicate file descriptor 2 (stderr) and write it to 1 - which is currently the terminal!

    > all-output.txt

    duplicate file descriptor 1 (stdout) and write it to a file called all-output.txt

  • ls /usr/bin /nothing > all-output.txt 2>&1
    • Redirect stdout to the file all-output.txt
    • Now redirect stderr to stdout - which by this point has already been redirected to a file

命令过长时如何换行

# Show the most commonly used commands.
tail ~/.bash_history -n 1000 \
    | sort \
    | uniq -c \
    | sed 's/^ *//' \
    | sort -n -r \
    | head -n 10

通过 shell 程序执行

bash ~/scripts/common.sh

sh ~/scripts/common.sh

让脚本可执行，通过脚本的路径执行

chmod +x ~/scripts/common.sh

~/scripts/common.v1.sh

env

#!/usr/bin/env bash

echo "Hello from Bash"

赋值和引用

# 通过 `=` 赋值变量，注意没有空格
password="somethingsecret"

# $(...) execute a set of commands in a 'sub shell'
masked_password=$(echo "$password" | sed 's/./*/g')

echo "Setting password '${masked_password}'..."

# 显示引用变量 ${variable}

# wrong，会找 USER_backup 变量，但找不到
echo "Creating backup folder at: '$USER_backup'"
mkdir $USER_backup

# correct
echo "Creating backup folder at: '${USER}_backup'"
mkdir "${USER}_backup"

数组

days=("Monday" "Tuesday" "Wednesday" "Thursday" "Friday" "Saturday" "Sunday")

echo "The first day is: ${days[0]}"
echo "The last day is: ${days[6]}"

对象 (Associative Arrays)

# Create an associative array called 'book'.
declare -A book

# Set some values on the array.
book[title]="Effective Shell"
book[author]="Dave Kerr"

# Show one of the values.
echo "Book details: ${book[title]} - ${book[author]}"

关于引号

• Single Quotes - Literal Values
  

  Single quotes should be used when you want to put special characters into a variable, or call a command that includes whitespace or special characters.

  message='   ~~ Save $$$ on with ** "this deal" ** ! ~~   '
  echo "$message"
  

• Double Quotes - Parameter Expansion
  

  Double quotes work in a very similar way to single quotes except that they allow you to use parameter expansion with the $ dollar symbol and escaping with the \ symbol.

  deal="Buy one get one free"
  message="Deal is '$deal' - save \$"
  echo "$message"
  
  # `` 内的也在一个 sub-shell 执行，但应该避免使用，统一使用 $() 的形式
  echo "The date is `date`"
  

• No Qoutes - Shell Expansion
  

  If you don't include quotes around a variable or value, then the shell will perform a series of operations called Shell Expansion.

  Brace expansion

  touch file{1,2,3} is expanded to touch file1 file2 file3

  Tilde expansion

  cd ~ is expanded to cd /home/dwmkerr

  Parameter and variable expansion

  echo $SHELL is expanded to echo /usr/bin/sh (note that this expansion also occurs with double quotes)

  Command substitution

  echo $(date) is expanded to echo the results of the date command (this also occurs with double quotes)

  Arithmetic expansion

  square=$((4 * 4)) has the value 4 * 4 evaluated mathematically (we see this at the end of this chapter)

  Word splitting

  see Loops and working with Files and Folders

  Pathname expansion

  ls *.txt is expanded to all filename that match the wildcard pattern *.txt

• Shell Parameter Expansion
  

  Shell Parameter Expansion is the process by which the shell evaluates a variable that follows the $ dollar symbol.

  But there are a number of special features we can use when expanding parameters. There are many options available and you can find them all by running man bash and searching for the text EXPANSION.

  I would avoid these techniques if possible as they are fairly specific to Bash and likely will be confusing to readers.

  It is generally enough to know that if you see special symbols inside a ${variable} expression then the writer is performing some kind of string manipulation.

  • Length: ${#var}
  • Set Default Value: ${var:-default}
  • Substring: ${var:start:count}
  • Make Uppercase: ${var^^}
  • Make Lowercase: ${var,,}
  • …

The Read Command

# 默认存在 $REPLY
echo "What is your name?"
read
echo "Hello, $REPLY"

# 指定存值的变量
echo "What is your name?"
read name
echo "Hello, ${name}"

# prompt (bash)
read -p "Please enter your name: " name
echo "Hello, $name"

# prompt (zsh)
read "?Please enter your name: "
echo "Hello, $REPLY"

# The -s (silent) flag can be used to hide the input as it is being written.
read -s -p "Enter a new password: " password
masked_password=$(echo "$password" | sed 's/./*/g')
echo ""
echo "Your password is: $masked_password"

# Limiting the Input
# Use the -n flag with the value 1 to specify that we want to read a single character only.
read -n 1 -p "Continue? (y/n): " yesorno
echo ""
echo "You typed: ${yesorno}"

Mathematics

The Test Command

if ! test -d ~/backups
then
    echo "Creating backups folder"
    mkdir ~/backups
fi

# 简写
if ! [ -d ~/backups ]
then
    echo "Creating backups folder"
    mkdir ~/backups
fi

if [ -x /usr/local/bin/common ]; then
    echo "The 'common' command has been installed and is executable."
elif [ -e /usr/local/bin/common ]; then
    echo "The 'common' command has been installed and is not executable."
else
    echo "The 'common' command has not been installed."
fi

# && 'and' || 'or'
if [ $year -ge 1980 ] && [ $year -lt 1990 ]; then
    echo "$year is in the 1980s"
fi

# -a 'and' -o 'or'
if [ $year -ge 1980 ] && [ $year -lt 1990 ]; then
    echo "$year is in the 1980s"
fi

# Chaining
# Run command1, if it succeeds run command2.
command1 && command2

# Run command1, if it does not succeed run command2.
command1 || command2

Case Statements

case <expression> in
    pattern1)
        <pattern1-commands>
        ;;
    pattern2 | pattern3)
        <pattern2and3-commands>
        ;;
    *)
        <default-commands>
        ;;
esac

read -p "Yes or no: " response
case "${response}" in
    y | Y | yes | ok)
        echo "You have confirmed"
        ;;
    n | N | no)
        echo "You have denied"
        ;;
    *)
        echo "'${response}' is not a valid response"
        ;;
esac

read -p "Yes or no: " response
case "${response}" in
    [yY]*)
        echo "You have (probably) confirmed"
        ;;
    [nN]*)
        echo "You have (probably) denied"
        ;;
    *)
        echo "'${response}' is not a valid response"
    ;;
esac

The For Loop

for <name> in <words>
do
    <conditional-command 1>
    <conditional-command 2>
    <conditional-command n>
done

for item in ./*
do
    echo "Found: $item"
done

• For Loops - Arrays
  

  days=("Monday" "Tuesday" "Wednesday" "Thursday" "Friday" "Saturday" "Sunday")
  for day in ${days[@]}
  do
      echo -n "$day, "
  done
  echo "happy days!"
  

• For Loops - Words
  

  sentence="What can the harvest hope for, if not for the care of the Reaper Man?"
  for word in $sentence
  do
      echo "$word"
  done
  

  The reason is that the shell is a text based environment and the designers have taken this into account. Most of the time when we are running shell commands in a terminal we are running commands that simply output text. If we want to be able to use the output of these commands in constructs like loops, the shell has to decide how to split the output up.

• For Loops - Files with Wildcards
  

  for script in ~/effective-shell/scripts/*.sh
  do
      echo "Found script: $script"
  done
  

  By default, if the shell doesn't find anything with a wildcard pattern it does not expand it. This is very confusing.

  By default, if a shell 'glob' (a pattern that includes a wildcard) does not match any files, the shell simply leaves the pattern as-is.

  • nullglob (return null for unmatched globs)
    

    shopt -s nullglob
    for script in ~/bad-shell/scripts/*.sh
    do
        echo "Found: $script"
    done
    

  • test
    

    for script in ~/bad-shell/scripts/*.sh
    do
        # If the file / folder doesn't exist, skip it.
        if ! [ -e "$script" ]; then continue; fi
        echo "Found: $script"
    done
    

• For Loops - Files with Find
  

  If the files that you are trying to loop through are too complex to match with a shell pattern, you can use the find command to search for files, then loop through the results.

  # Create a symlink to 'effective-shell' that has a space in it...
  ln -s ~/effective shell ~/effective\ shell
  
  # Find all symlinks and print each one.
  links=$(find ~ -type l)
  for link in $links
  do
      echo "Found Link: $link"
  done
  

  在 For Loops - Files with Wildcards 中看到，shell 会按照空格分割文本， 此时 find 找到的文件如果带有空格，也会被分割，导致文件名不对。

  一种解决办法时临时改变 shell 使用的分割符，由于 find 找回来的文件都是 以换行符分割的，因此，可以将分割符临时从空格设置为换行符。

  # Save the current value of IFS - so we can restore it later. Split on newlines.
  old_ifs=$IFS
  # We have to use the complex looking 'ANSI C Quoting' syntax to set $IFS to a newline
  IFS=$'\n'
  
  # Find all symlinks and print each one.
  links=$(find ~ -type l)
  for link in $links
  do
      echo "Found Link: $link"
  done
  
  # Restore the original value of IFS.
  IFS=$old_ifs
  

  The $IFS variable is the 'internal field separator' variable. It is what the shell uses to decide what characters should be used to split up text into words. By default, this variable includes the space character, the tab character and the newline character.

  I believe that in this case it is probably best to not use a shell script. There is no solution that is particularly clean or simple. In this case I think you might be better off using a programming language.

• For Loops - Looping over Sequences
  

  Another common way to use a for loop is with brace expansion. Brace expansion we have already seen a number of times so far - we can use it to generate a sequence of values.

  touch {coffee,tea,milkshake}-menu.txt
  
  # loop through a sequence of values or a range of numbers with 'increment'
  for i in {0..25..5}
  do
      echo "Loop ${i}"
  done
  

The While Loop

while <test-commands>
do
    <conditional-command 1>
    <conditional-command 2>
    <conditional-command n>
done

# Create an empty array of random numbers.
random_numbers=()

# As long as the length of the array is less than five, continue to loop.
while [ ${#random_numbers[@]} -lt 5 ]
do
    # Get a random number, ask the user if they want to add it to the array.
    random_number=$RANDOM
    read -p "Add $random_number to the list? (y/n): " choice

    # If the user chose 'y' add the random number to the array.
    if [ "$choice" = "y" ]; then random_numbers+=($random_number); fi
done

# Show the contents of the array.
echo "Random Numbers: ${random_numbers[@]}"

• While Loops - Looping through the lines in a file
  

  while read line; do
      echo "Read: $line"
  done < ~/effective-shell/data/top100.csv
  

  存在问题，避免使用。

• While Loops - The Infinite Loop
  

  There are times that you may want to loop forever. For example you might be writing a script that reads an option from the user, processes it, and then starts again.

  while true
  do
      echo "1) Move forwards"
      echo "2) Move backwards"
      echo "3) Turn Left"
      echo "4) Turn Right"
      echo "5) Explore"
      echo "0) Quit"
  
      read -p "What will you do: " choice
      if [ $choice -eq 0 ]; then
          exit
      fi
      # The rest of the game logic would go here!
      # ...
  done
  

The Until Loop

until <test-commands>
do
    <conditional-command 1>
    <conditional-command 2>
    <conditional-command n>
done

# until loop
# Create an empty random number string - we're going to build it up in the loop.
random_number=""

# Keep on looping until the random number is at least 15 characters long.
until [ "${#random_number}" -ge 15 ]
do
    random_number+=$RANDOM
done
echo "Random Number: ${random_number}"

# while loop
random_number=""
while [ "${#random_number}" -lt 15 ]
do
    random_number+=$RANDOM
done
echo "Random Number: ${random_number}"

Continue and Break

echo "For each folder, choose y/n to show contents, or c to cancel."
for file in ~/*
do
    # If the file is not a directory, or it cannot be searched, skip it.
    if ! [ -d "$file" ] || ! [ -x "$file" ]; then continue; fi

    # Ask the user if they want to see the contents.
    read -p "Show: $file? [y/n/c]: " choice

    # If the user chose 'c' for cancel, break.
    if [ "$choice" = "c" ]; then break; fi

    # If the user choice 'y' to show contents, list them.
    if [ "$choice" = "y" ]; then ls "$file"; fi
done

变量

# Set some variables.
title="My Cool Script"
version="1.2"
succeeded=0

# Create a function that writes a message and changes a variable.
title() {
    # Note that we can read variables...
    title_message="${title} - version ${version}"
    echo "${title_message}"

    # ...and set them as well.
    succeeded=1
}

# Show the value of 'succeeded' before and after the function call.
echo "Succeeded: ${succeeded}"
title
echo "Succeeded: ${succeeded}"
echo "Title Message: ${title_message}"

• 作用域
  

  If you come from a programming background you might find it odd that you can create a variable in a function and use it outside of the function. This is a feature known as dynamic scoping. Many common programming languages like Python, JavaScript, C, Java and others use an alternative mechanism called lexical scoping.

  Lexical scoping is a feature that ensures that you can only use a variable from within the 'scope' that it is defined. This can reduce errors - because it means that if you define a variable in a function you don't accidentally 'overwrite' the value of another variable that is used elsewhere.

  You can use the local keyword to define a variable that is only available in the 'local' scope, i.e. the function that it is defined in. This allows you to use lexical scoping and can reduce the risk of errors.

  run_loop() {
      local count=0
      for i in {1..10}; do
          # Update our counter.
          count=$((count + 1))
      done
      echo "Count is: ${count}"
  }
  

  In general, you should use 'local' variables inside functions. This can help to avoid problems where calling a function can have an unintended side effects:

  # 比较用 local 和不用的区别
  # Set a count variable somewhere in our script...
  count=3
  
  # Call our 'run_loop' function.
  run_loop
  
  # Write out the value of 'count'.
  echo "The 'count' variable is: ${count}"
  

传参

sum() {
    local value1=$1
    local value2=$2
    local result=$((value1 + value2))
    echo "The sum of ${value1} and ${value2} is ${result}"
}

# Create a function that calculates the sum of two numbers.
sum() {
    echo "The sum of $1 and $2 is $(($1 + $2))"
}

# usage
# sum 3 6
# sum 10 33

• Parameter Shifting
  

  # Show the top 'n' values of a set.
  show_top() {
      # Grab the number of values to show, then shift.
      local n=$1
      shift
  
      # Get the set of values to show. Notice that we start in position 1 now.
      local values=${@:1:n}
      echo "Top ${n} values: ${values}"
  }
  

返回值

• 通过设置变量值
  

  is_even() {
      local number=$1
  
      # A number is even if when we divide it by 2 there is no remainder.
      # Set 'result' to 1 if the parameter is even and 0 otherwise.
      if [ $((number % 2)) -eq 0 ]; then
          result=1
      else
          result=0
      fi
  }
  

  $ number=33
  $ is_even $number
  $ echo "Result is: $result"
  Result is: 0
  

  In general, this method of returning values from a function should be avoided. It overwrites the value of a global variable and that can be confusing for the operator.

  A more common way to return a value from a function is to write its result to stdout

• 输出到 stdout
  

  lowercase() {
      local params="$@"
      # Translate all uppercase characters to lowercase characters.
      echo "$params" | tr '[:upper:]' '[:lower:]'
  }
  

  $ result=$(lowercase "Don't SHOUT!")
  $ echo "$result"
  don't shout!
  

  If you have a programming background it might seem very strange that you write results in a function by writing to stdout. Remember - the shell is a text based interface to the computer system. The majority of commands that we have seen so far that provide output write their output to the screen. This is what ls does, what find does, what cat does and so on. When we echo a result from a function, we are really just following the Unix standard of writing the results of a program to the screen.

  Remember - shell functions are designed to behave in a similar way to shell commands. They write their output to stdout.

  Although it might feel a bit clunky, writing the results of a command to stdout is a tried and tested method of returning results.

  但是，如果脚本中有很多次输出，最终的结果可能不是我们期待的。

  command_exists() {
      if type "$1"; then
          echo "1"
      else
          echo "0"
      fi
  }
  
  # result=$(command_exists "touch")
  # echo "Result is: ${result}"
  

  解决办法就是移除调不需要的输出，输出到 /dev/null

  command_exists() {
      if type "$1" >> /dev/null; then
          echo "1"
      else
          echo "0"
      fi
  }
  

• Returning Status Codes
  

  The return (return from shell function) command causes a function to exit with a given status code.

  This is something that often causes confusion in shell scripts. The reason is that in most programming languages, you would use a 'return' statement to return the result of a function. But in the shell, when we return, we set the status code of the function.

  What is a status code? When a command runs, we expect it to return a status code of 'zero' to indicate success. Any non-zero status code is used to specify an error code.

  Remember - only use the 'return' command to set a status code. Many shells will only allow values from 0-255 to be set, and most users will expect that a command should return zero for success and that any non-zero value is an error code. If you need to provide output for a command that is not just a status code, you should write it to stdout or if you must, set the value of a global variable.

  command_exists() {
      if type "$1" >> /dev/null; then
          return 0
      else
          return 1
      fi
  }
  

  if command_exists "common"; then
      echo "The 'common' command is installed on your system"
  else
      echo "The 'common' command is not installed on your system"
  fi
  

出错后退出

#!/usr/bin/env sh

# Exit if any command fails.
set -e

# ...

grep '[:space:]*#' ~/effective-shell/scripts/common.sh | tr 'a-z' 'A-Z'

Checking for Existing Variables or Functions

Unsetting Values

# Remove the 'is_even' function from the shell session.
unset -f is_even

Traps

# Create a temporary folder for the effective shell download.
source="https://effective-shell.com/downloads/effective-shell-samples.tar.gz"
tmp_dir=$(mktemp -d 2>/dev/null || mktemp -d -t 'effective-shell')
tmp_tar="${tmp_dir}/effective-shell.tar.gz"

# Define a cleanup function that we will call when the script exits or if
# it is aborted.
cleanup () {
    if [ -e "${tmp_tar}" ]; then rm "$tmp_tar}"; fi
    if [ -d "${tmp_dir}" ]; then rm -rf "${tmp_dir}"; fi
}

# Cleanup on interrupt or terminate signals and on exit.
trap "cleanup" INT TERM EXIT

# Download the samples.
curl --fail --compressed -q -s "${source}" -o "${tmp_tar}"

# Extract the samples.
tar -xzf "${tmp_tar}" -C "${tmp_dir}"

Handling Options

Using 'Select' to Show a Menu

select fruit in Apple Banana Cherry Durian
do
    echo "You chose: $fruit"
    echo "This is item number: $REPLY"
done

Running Commands in Subshells

(mkdir -p ~/new-project; cd ~/new-project; touch README.md)