Joshua's Cheatsheets - Bash / Shell - Cheatsheet
Light
help

Bash / Shell - Cheatsheet

Resources

What & Link Type
SS64 Bash Reference Docs
man7: Linux man Pages Docs
Wooledge / GreyCat: Bash Reference Sheet Cheatsheet
ExplainShell.com
- breaks down any given command and explains what it does)
Interactive Tool

Current directory:

echo $PWD

Including the "Hash-Bang" / "She-Bang"

#!/bin/bash
  • ^ Should go at the top of sh files.

Commenting

# My comment

Logic / flow

Test

Before using advanced branching logic, you should know that the shell has a built-in test check - basically coerces the output of an expression to a boolean that can be used in logic flows. Simply encase the expression/condition in brackets:

[ check-this-condition ]

or double brackets, as the newer version

If / Else

Great guides

As long as the thing you want to do if the conditional is true always exits with exit code 0 (success), this works as simple one-liner if/else:

conditional_test && echo "it worked" || echo "it failed"

Helpful S/O's for understanding the above: 1, 2

Double Pipe vs Double Ampersand, and Other Flow Operators

Quick reference:

  • && = Only execute right side if left side succeeds

    • Examples:

      • false && echo "this text will NOT show"
      • true && echo "this text will show"
  • || = Only execute right side if left side FAILS (non-zero exit code)

    • Essentially the inverse of &&
    • Examples:

      • false || echo "this text will show"
      • bad_commmand || echo "this text will show"
      • true || echo "this text will NOT show"
  • & = asynchronously runs both commands on either side, regardless of success of either, in detached (forked) processes

    • Warning: This can be a hackish way to do things
    • Definitely do not use this if the second command is dependent on the output of the first
    • Examples:

      • slow_command_to_execute & echo "this will appear before the left side is done!"
      • true & echo "this text will show"
      • false & echo "this text will also show"
  • ; = Execute both side, sequentially, regardless of success of either

    • Examples:

      • true; echo "this text will show"
      • bad_command; echo "this text still shows"
  • | = Not for logic flow, used for redirection

Grep

  • In general, if you are a RegEx power user, you will probably find sed much preferable. Or awk.

    • grep can actually be a bit of a pain when trying to do things like use capture groups (1, 2)
  • Cheatsheets:

  • (Common) Flags:

    Flag Description
    -E Extended regex
    -o Only output the matching part of the line
    -p Treat as perl style regular exp
    -i Ignore case
    -e Pass explicit patterns, multiple allowed

sed

  • Cheatsheets

  • Common flags

    Flag Description
    -n Silent, suppress printing of pattern space
    -r
    (or -E on some systems)
    Use extended regexp - I always prefer
  • Syntax

    • print output

      • echo $'hello world\nLine Two\nGoodbye' | sed -E -n '/Line.+/p'

        • Prints "Line Two"
    • substitute

      • echo $'hello world\nLine Two\nGoodbye' | sed -E 's/Line.+/my substitution/'

        • Prints:

          • hello world
            my substitution
            Goodbye
    • Print only a specific capture group

      • This is actually a little complicated. Basically, you have to substitute the entire input with the back-reference of the capture.

        • sed -E -n 's/.*(My_Search).*/\1/p
      • In action:

        • echo $'hello world\nLine Two\nGoodbye' | sed -E -n 's/.*^Line (.+)$.*/\1/p'

          • Prints:

            • "Two"

Warning: sed on your system might have limitations - for example, be warned that if you can't use the Perl (-p) mode, you will need to use [0-9] instead of /d, for digits.

Capturing and Executing Output

If you simply want to "capture" the value output by a script and store it as a variable, you can use substitution. See "Storing into a variable".

If you want to execute the output of a command as a new command / script, you can use the (dangerous) eval command, plus substitution: eval $( MY_COMMAND ).

Here is a full example:

(echo echo \"in test.txt\") > test.txt
eval $( cat test.txt )
# "in test.txt"

Piping and redirection

  • Piping VS Redirection

    • Simple answer:

      • Piping: Pass output to another command, program, etc.
      • Redirect: Pass output to file or stream
  • Pipe

    • |
    • echo 'hello world' | grep -o 'hello'

      • Prints hello
  • Redirection

    • >
    • echo "hello" > output.txt

Problems with piping

Piping, in general, is taking the stdout of one process to the stdin of another. If the process you are trying to pipe to is expecting arguments and doesn't care about stdin, or ignores it, piping won't work as you want it to.

The best solution for this is usually to use xargs, which reads stdin and converts the input into arguments which are passed to the command of your choice.

Or, you can use substitution to capture the result of the first part of the pipe and reuse it in the second.

See this S/O answer for details.

If the input you are passing contains special characters or spaces (such as spaces in a filename), take extra care to handle it. For example, see if the thing generating the input can escape it and null terminate the fields (e.g. git-diff --name-only-z), and then you can use the -0 or --null option with xargs to tell it to expect null terminated fields.

??? - 2>&1

You see 2>&1 all over the place in bash scripts, because it is very useful. Essentially, it forces errors (stderr) to be piped to whatever value stdout is set to.

This has a few really handy and common uses:

  1. See both the output and the errors in the console at the same time

    • Often errors are routed to stderr and not shown in the console.
  2. Suppress errors

    • Since this forces errors to stdout, this has the side effect of suppressing them from their normal destination

      • However, they are still going to show up in stdout obviously. If you really want to suppress them entirely, use 2> /dev/null, which essentially sends them to oblivion
  3. Send both output and errors to file

    • If you redirect to a file before using 2>&1, then both outputs gets sent to the file.

      • ls file-does-not-exist.txt > output.txt 2>&1

        • output.txt will now contain "ls: cannot access 'file-does-not-exist.txt': No such file or directory"

On a more technical level, Unix has descriptors that are kind of like IDs. 2 is the descriptor/id for stderr, and 1 is the id for stdout. In the context of redirection, using & + ID (&{descriptorId}) means copy the descriptor given by the ID. This is important for several reasons - one of which is that 2>1 could be interpreted as "send output of 2 to a file named 1", whereas 2>&1 ensures that it is interpreted as "send output of 2 to descriptor with ID=1".

So... kinda...

  • 2>&1

    • can be broken down into:
  • stderr>&stdout

    • ->
  • stderr>value_of_stdout

    • ->
  • stdout = stderr + stdout

Suppress errors

Make sure to see above section about how descriptors work with redirection, but a handy thing to remember is:

# Pretend command 'foobar' is likely to throw errors that we want to completely suppress
foobar 2>/dev/null

This sends error output to /dev/null, which basically discards all input.

Additional reading

Using variables

Simple:

VARIABLE_NAME=VARIABLE_VALUE

Example:

MYPATH="C:\Users\Joshua"
cd $MYPATH
echo "I'm in Joshua's folder!"

Storing into a variable

How to store the result of a command into a variable:

  • There are two methods:

    • Command/process substitution (easy to understand)

      VARIABLE_NAME=$(command)
      • However, this doesn't always work with complex multi-step operations
    • read command (complicated) - works with redirection / piping

      echo "hello" | read VARIABLE_NAME

Environment Variables

List all env values

printenv

Set an environment variable - current process and sub-processes

export VARIABLE_NAME=VARIABLE_VALUE

Set an environment variable - permanently

In order for an environment variable to be persisted across sessions and processes, it needs to get saved and exported from a config file.

This is often done by manually editing /etc/environment:

    1. Launch editor: sudo -H gedit /etc/environment
    1. Append key-value pair: VAR_NAME="VAR_VAL"
    1. Save

The difference between setting a variable with export vs without, is similar to the difference in MS Windows, for using setx vs just set -> export persists the value.

Global path

echo $PATH

Triggering / running a SH file

  • Make sure it is "runnable" - that it has the execute permission

    • chmod +x /scriptfolder/scriptfile.sh
  • Then call it:

    • /scriptfolder/scriptfile.sh

If you are having issues running from Windows...

  • MAKE SURE LINE ENDINGS ARE \n and not \r\n

Also, make sure you specify directory, as in ./myscript.sh, not myscript.sh, even if you are currently in the same directory as script.

Keeping file open after execution

Add this to very end of file:

exec $SHELL

Note: this will interfere with scripts handing back control to other scripts; ie resuming flow between multiple scripts.

Strings

Special characters (newline, etc)

You need to prefix with $ before string to use special characters.

Example:

  • echo 'hello\ngoodbye'

    • Prints:

      • "hello\ngoodbye"
  • echo $'hello\ngoodbye'

    • Prints:

      • "hello
        goodbye"

You can also use printf for linebreaks: printf '\n\n'

Joining Strings

You can simply put strings together in variable assignment, like this:

FOO="Test"
BAR=$FOO"ing"
echo $BAR

echoes:

testing

How to generate keys and certs

  • SSH Key: Using ssh-keygen (available on most Unix based OS's, included Android)

    • You can run it with no arguments and it will prompt for file location to save to

      • ssh-keygen
    • Or, pass arguments, like -t for algorithm type, and -f for filename, -c for comment

      • ssh-keygen -t rsa -C "your_email@example.com"

Create new files, or update existing file timestamps

  • Touch without any flags will create a file if it does not exist, and if it does already exist, update timestamp of "last accessed" to now

    • touch "myfolder/myfile.txt"
  • If you want touch to only touch existing and never create new files, use -c

    • touch -c "myfile.txt"
  • Specifically update last accessed stamp of file

    • touch -a "myfolder/myfile.txt"
  • specifically update "Last MODIFIED" stamp of file

    • touch -m "myfolder/myfile.txt"
  • You can also use wildcard matching

    • touch -m *.txt
  • and combine flags

    • touch -m -a *.txt

Verify Files

You can verify that a file exists with test -f {filepath}. Handy guide here.

Deleting

  • Delete everything in a directory you are CURRENTLY in:

    • Best:

      • find -mindepth 1 -delete
    • UNSAFE!!!

      • rm -rf *
    • Better, since it prompts first:

      • rm -ri *
  • Delete everything in a different directory (slightly safer than above)

    • rm -rf path/to/folder
  • Delete based on pattern

    • find . -name '*.js' -delete

LS and File Listing

📄 LS - SS64

LS Cheatsheet

How to... Cmd
Show all files ls -a
Show filesizes (human readable) ls -lh
Show filesize (MB) ls --block-size=MB
Show details (long) ls -l (or, more commonly, ls -al)
Sort by last modified ls -t

⚡ -> Nice combo: ls -alht --color (or, easier to remember ls -halt --color). All files, with details, human readable filesizes, color-coded, and sorted by last modified.

ls - show all files, including hidden files and directories (like .git)

ls -a

Show progress bar / auto-update / keep console updated:

Great SO Q&A

Find executable paths

If you are looking for the bash equivalent of Window's "where" command, to find how a binary is exposed, try using which. E.g. which node.

You can use the ln command (ss64) to create symbolic links.

# Works for both files and directories
ln -s {realTargetPath} {symbolicFileName}

# If you need to update an existing symlink, you need to use "force"
ln -sf {realTargetPath} {symbolicFileName}

In general, it is best / easiest to always use absolute paths for the targets.

You can use ls -la to list all files, including symlinks.

If you just want to see resolved symlinks, you can use grep - ls -la | grep "\->"


Networking

cURL

  • Good cheatsheets

  • Show headers only curl -I http://example.com
  • Search for something

    • You can't just pipe directly to grep or sed, because curl sends progress info stderr, so use --silent flag:

      • curl --silent https://joshuatz.com | sed -E -n 's/.*<title>(.+)<\/title>.*/\1/p'

        • Prints: Joshua Tzucker&#039;s Site

Networking - How do I...

  • Resolve DNS hostname to IP

    • getent hosts HOST_NAME | awk '{ print $1 }'
    • Credit goes to this S/O
  • Download a file and save it locally with bash?

    • You can use wget or cURL (S/O):

      • wget -O {New_Filename_Or_Path} {URL}
      • curl -o {New_Filename_Or_Path} {URL}
    • If you want to just use the name of the file as-is, you can drop -O with wget

Archives

How do I...


Handy Commands for Exploring a New OS

Command What?
uname -a Display system OS info
apt list --installed List installed packages
crontab -l or less /etc/crontab View crontab entries
Markdown Source Last Updated:
Sun Oct 04 2020 19:25:10 GMT+0000 (Coordinated Universal Time)
Markdown Source Created:
Mon Aug 19 2019 17:06:24 GMT+0000 (Coordinated Universal Time)
© 2020 Joshua Tzucker, Built with Gatsby