Reference
Every command and tool used in the tasks: what it does, the options you actually need, examples and pitfalls — instead of man.
Files, permissions, users
ACL: getfacl and setfaclPermissions for individual users and groups beyond owner/group/other rwx; the mask and default ACLsbasename, dirname, realpath, readlinkTaking paths apart — file name, directory, absolute path, where a link pointscapabilities: getcap and setcapGive a program a single root privilege without setuid — setcap cap_net_bind_service=ep, getcap -r, setcap -rchagePassword and account expiry — chage -l, -M, -m, -W, -I, -E, -d 0 and the /etc/shadow fieldschmodFile and directory permissions — numeric (644) and symbolic (u+x), plus setuid, setgid and the sticky bitchownChange a file's owner and group; chgrp — group onlycpCopy files and directories — with permissions, owners, timestamps and linksddByte-level block copying — files of a given size, cutting out bytes, disk imagesfindFind files by name, type, size, modification time and permissionsid, groups, whoamiWho am I and which groups am I in — UID, GID, the group list of the current process or any userlnHard and symbolic links to files and directorieslsList directory contents — permissions, owners, sizes, dates, sortingmkdirCreating directories — with parents and the right permissions in one gomkfs, tune2fs, dumpe2fsCreate ext4 (including in an image file), set the label, reserve and inode count, inspect filesystem parametersmktempSafely create temporary files and directories with unique namesmountAttaching file systems — mount, umount, -o options, /etc/fstab, findmnt, loop devices and bindmvMove and rename files and directoriesrmDeleting files and directories — no trash, no undosplitCut a file into pieces by lines, bytes or number of chunks and put it back togetherstatAll file metadata — permissions, owner, size, inode, timestamps — in the format you needsuSwitch user in the shell — su, su -, su -c, su -s; how it differs from sudo -isudoRunning commands as root or another user; rules in /etc/sudoers and visudotouchCreate an empty file or change a file's access and modification timestruncate and fallocateShrink or grow a file to the size you need, create a sparse or preallocated fileumaskDefault permission mask — which bits are removed from new files and directoriesuseradd and usermodCreating and changing users and groups — useradd, usermod, groupadd, passwd, chage, userdelwhich, type, command -vWhat exactly runs for a command name — a file from PATH, a builtin, an alias or a functionxargsTurns lines from standard input into command arguments
Text processing
Regular expressionsBRE, ERE and PCRE in practice — classes, anchors, quantifiers, groups, backreferences, greedinessawkColumn-based text processing — selecting fields, filters, counts and sumscat and tacPrint a file, join several files, number lines, show invisible characterscolumnAlign text into a table, set headers, reorder and hide columns, output JSONcommCompare two sorted lists — what's only in the first, only in the second, in bothcutCut columns by a delimiter or characters by position out of every linedatamashColumn statistics from the command line — sums, means, medians, groupingdiff, cmp and patchCompare files and directories line by line or byte by byte, make a patch and apply itgrepSearch for lines matching a pattern in files and command outputhead and tailFirst and last lines of a file, slicing by line numbers and following a growing log with tail -fjoinJoin two files on a common field, like JOIN in SQL — inner, outer, "unmatched"nlNumber lines — all, non-empty only or by pattern, with your own number formatpasteMerge files line by line into columns, or join all lines of a file into one with a delimiterrevReverse each line character by character; the "last field" trick and reversing field ordersedStream editor — regex substitution, deleting and inserting lines, editing files in placeseqA sequence of numbers — with a step, counting down, fractions, leading zeros and a custom formatshufShuffle lines, pick random lines or numbers, generate a sample with repeatssortSorting lines — alphabetically, by numbers, sizes, versions and by individual columnsteeSplit a stream — show it on screen and write it to a file at the same time, including under sudotrCharacter-by-character replacement and deletion — case, newlines, extra spacesuniqCollapse adjacent identical lines, count repeats, find duplicateswcCount lines, words, bytes and characters in files or command outputxxd and odView a file's bytes in hex, octal and character form; rebuild a file from a hex dump
Bash
Arithmetic in bashInteger math — $(( )), (( )), let, expr, number bases, and fractions via awkBash functionsDeclaring functions, arguments $1 $@ $#, local, return and returning output, source and export -fConditions and loopsif, [[ ]], test, case, for/while/until loops, reading a file line by line, break and continueEnvironment variables: env, export, printenvA process's environment — export, env -i and env VAR=…, printenv, where to set variables and which files bash readsExit codes$?, && and || chains, set -e/-u/-o pipefail, exit and trap — how a script finds out about errorsRedirection and pipelinesWhere a command's output goes — >, >>, 2>&1, &>, here-doc, the | pipeline and pipefailVariables and expansionsBash variables, default values, string trimming, $(…), $((…)), quoting and arraysecho and printfPrinting text from a script — echo and its traps, and printf with the %s %d %f %x %q %b formatsgetoptsParsing a script's short options (-v, -o file) — the option string, OPTARG, OPTIND, silent mode, getopt vs getoptsreadRead a line from stdin or a file into variables — options -r -p -s -t -n -a -d, IFS and looping over linesset and shoptbash modes — set -euo pipefail, -x for debugging, noclobber, set -- and shopt options for globsshellcheckStatic analyzer for shell scripts — finds mistakes with quoting, conditions and portability
Processes and system
/procVirtual FS with kernel data — a process's arguments, environment and open files, memory, loadBackground processes: &, nohup, setsid, disown, jobs, waitRunning in the background, bash job control, and how to make a process survive closing the terminalapt and dpkgPackages on Debian and Ubuntu — install, remove, search, versions, which package owns a filecron and crontabRunning commands on a schedule — crontab format, crontab -e/-l, system files and environment trapsdateCurrent time in any format, date parsing and arithmetic (-d), Unix time and time zones (TZ=)df and duHow much space is free on filesystems (df) and how much directories and files take up (du)flockFile locks from the shell — keep two copies of a script or cron job from running at the same timekillSend a signal to a process — terminate it gracefully, kill it, reload its config, pause itlogrotateLog rotation — rename, compress, delete old copies and let the service start a new filenice and reniceProcess priority for the scheduler — values from -20 to 19, start with nice -n, change with renice -p, who may raise itpgrep and pkillFind processes by name or command line and send them a signal without ps | greppsProcess list — PID, parent, owner, memory, CPU, state and full command lineservice and systemctlStarting, stopping and checking services — systemctl on a regular server, service and init scripts in a containersupervisor: supervisord and supervisorctlThe supervisor process manager — the [program:…] section, auto-restart, logs and supervisorctl commandstimeout and sleepLimit how long a command runs (timeout -s -k, code 124) and pause (sleep with units s, m, h, d)tmuxTerminal multiplexer — sessions that survive a dropped connection, windows and panes in one terminaltopProcesses and load in real time — keys, batch mode -b -n, columns, load averageuname, hostname, uptime, freeSystem information — kernel and architecture, host name and IPs, uptime and load, free memorywho, w, lastWho is on the system now and what they're doing, login and reboot history, failed login attempts
Archives and copying
gzipCompressing individual files — gzip, gunzip, zcat, zgrep; a brief look at xz and bzip2rsyncSync directories locally and over SSH — copies only changes, can delete extras and exclude filestarPacking directories into an archive and unpacking — .tar, .tar.gz, .tar.xz, listing and selective extraction
Network and web
ab (Apache Bench)Load-test an HTTP server — request count, concurrency, keep-alive and how to read the reportcurlHTTP requests from the terminal — headers, status codes, POST with a body, redirects, HTTPS, timingdigDNS queries — A, AAAA, MX, TXT, NS, reverse lookup, tracing; how dig differs from getent hostsgetentUser, group, host and port entries as the system sees them through NSShtpasswdA login and password file for HTTP Basic auth — creating, bcrypt, deleting, verifying, nginx auth_basicipNetwork interfaces, IP addresses, routes and the ARP table — ip addr, ip link, ip route, ip neighnginxHow an nginx config is structured, how server and location are chosen, root vs alias, proxy_pass, testing and reloadingopensslCertificates and keys — inspect, create, sign with your own CA, check a TLS server and a key↔certificate pairpingCheck host reachability over ICMP — packet loss, latency, MTU, exit codes for scriptsssWhich ports are listening, which connections are open and which process owns themsshKey-based login, ssh-keygen, permissions on ~/.ssh and authorized_keys, ~/.ssh/config, tunnels, copying with scpsshdThe SSH server — key sshd_config options, checking the config with sshd -t / -T, reloading without losing accesswgetDownloading files over HTTP/HTTPS — resuming, link checking, recursive download, exit codes
Data formats
Checksumssha256sum, md5sum and friends — compute a hash, verify files against a list, why MD5 is not for securitybase64Base64 encoding and decoding — -d, -w0, the URL-safe variant, common newline mistakesgpgGnuPG — keys, encryption for a recipient and with a password, signatures, export and import, use in scriptsiconvConverting text between UTF-8, CP1252, ISO-8859-15 and other encodings; transliteration and how to detect an encodingjqParsing and transforming JSON — selecting fields, select, map, strings, dates, output to CSV/TSVpython3 one-linerspython3 -c and -m — web server, JSON validation, venv, parsing CSV and JSON right on the command linexmllintChecking, formatting, schema validation and XPath queries for XML — --format, --xpath, string(), count()xmlstarletXPath queries on XML, editing elements and attributes, checking and formatting — all from the command lineyqReading and editing YAML jq-style — selection, select, in-place editing, merging files, converting to JSON
Databases
PostgreSQL utilities: createdb, pg_dump, pg_restore, pg_isready, pg_ctlclusterCreate a database, take and restore a dump, check that the server is ready, start and stop a clusterpsqlPostgreSQL command-line client — connecting, meta-commands \l \dt \d \du, queries from scripts, roles and privilegesredis-cliRedis client — commands for strings, lists, hashes and sets, TTL, finding keys with SCAN, INFO and CONFIGsqlite3SQLite command-line client — a database in a single file, SQL queries and dot commands
Git and builds
Kubernetes
Kubernetes: ConfigMap, Secret, variables and volumesPod settings and data — ConfigMap and Secret, env and envFrom, configMap/secret/emptyDir/PVC/hostPath volumes, subPathKubernetes: Pod, Deployment, StatefulSet, DaemonSet, Job, CronJobWorkload manifests — containers, init and sidecar containers, replicas, rollout strategies, jobsKubernetes: Service, Ingress, NetworkPolicyNetworking in Kubernetes — Service types, ports, Ingress with paths and TLS, NetworkPolicy and default-denyKubernetes: ServiceAccount, Role, RoleBindingKubernetes permissions — ServiceAccount, Role and ClusterRole, apiGroups/resources/verbs rules, bindings, pod tokenKubernetes: probes, resources and HPALiveness, readiness and startup probes, requests and limits, CPU and memory units, QoS classes, HorizontalPodAutoscalerkubeconformValidating Kubernetes manifests against JSON schemas without a cluster — strict, summary, CRDs, offline schemaskubectlThe Kubernetes client — generating manifests with --dry-run=client, apply/get/describe, namespaces and labels
DevOps tools
GitHub ActionsGitHub Actions workflows — on, jobs, runs-on, steps uses/run, needs, matrix, expressions, workflow_call, actionlintGitLab CI (.gitlab-ci.yml)GitLab CI pipelines — stages, jobs, script, rules, needs, artifacts, cache, extends, include, variablesactionlintGitHub Actions workflow linter — syntax, ${{ }} expressions, needs, runner labels, shellcheck for runansibleAnsible playbooks — structure, the user/file/copy/template/lineinfile modules, loops, conditions, idempotency, --checkdocker compose (file)The compose.yaml format — services, images, ports, variables, volumes, networks, depends_on and healthcheckterraformInfrastructure as code — init/plan/apply/destroy, providers, resources, variables, outputs and stateyamlschemaOffline validation of .gitlab-ci.yml and docker-compose.yml against the official JSON schemas — an IzzyLab sandbox tool