Skip to content

linux-awk-tutorial.md

Latest commit

 

History

History
229 lines (169 loc) · 8.38 KB

linux-awk-tutorial.md

File metadata and controls

229 lines (169 loc) · 8.38 KB
Raw
title Linux Command Tutorial: awk
date 2026-09-12 00:00:00 +0000
categories
Technology
tags
Linux
GNU GAWK
awk
Linux Command Tutorial
draft false
slug linux-awk-tutorial
description Authoritative reference tutorial for awk (GNU GAWK), detailing pattern-action architecture, field separators, associative arrays, built-in variables, and POSIX portability.
upstream_suite gnu-gawk
upstream_version GNU GAWK 5.3.0
posix_standard POSIX.1-2024
research_date 2026-09-12

The Linux Command Tutorial series provides rigorous, upstream-verified references for essential system commands across Linux distributions and UNIX-like environments. Each article focuses on a single executable, combining exhaustive option documentation, verified real-world examples, security boundaries, and best practices directly derived from official source documentation and POSIX standards.

1. Introduction

Upstream: GNU GAWK 5.3.0 | POSIX: POSIX.1-2024 (with GNU extensions) | Safety Tier: safe-read-only | Scope: text-processing

awk is a Turing-complete, pattern-directed scanning and processing language. It parses input streams into records (lines) and fields (columns), executing action blocks on records matching specified patterns or conditional expressions.

  • Upstream Project & Provenance: The standard Linux implementation is GNU Awk (GAWK), maintained by the Free Software Foundation.
  • Portability & Standards Baseline: Standardized in IEEE Std 1003.1-2024 (POSIX.1-2024). GAWK adds multi-dimensional arrays, regular expression record separators (RS), coprocesses (|&), and TCP/IP networking sockets.
  • Target Research Implementation: Audited against GNU GAWK 5.3.0 (awk(1), gawk(1)).
  • Applicability & Lifecycle: The premier text-processing tool for tabular data, log parsing, calculations, and complex column-oriented transformations.

2. Syntax and Command Model

2.1 Canonical Synopsis

awk [options] -f progfile [--] file ...
awk [options] [--] 'program' file ...

2.2 Pattern-Action Structure

Every AWK program consists of a sequence of pattern-action statements:

pattern { action }
  • If pattern is omitted, action executes for every record.
  • If { action } is omitted, matching records are printed verbatim (print $0).
  • Special Blocks:
    • BEGIN { ... }: Executes once before any input files are read.
    • END { ... }: Executes once after all input streams have reached EOF.

2.3 Records and Fields Model

  • $0: The entire current record (line).
  • $1, $2, ... $NF: The individual fields in the record.
  • Field Separator (FS): Controls how fields are split (default is any sequence of spaces or tabs).
  • Record Separator (RS): Controls how records are separated (default is newline \n).

3. Options

3.1 Primary Flags

Flag Description POSIX Defined Upstream Note
-F fs Set input field separator (FS) to regex or character fs. Yes Standard column separator
-v var=val Assign value to variable var before program execution. Yes Parameter injection
-f progfile Read the AWK program source from progfile. Yes Modular scripts
-posix Enforce strict IEEE POSIX compliance; disable GNU extensions. No Portability testing
-i file Include AWK library file (GAWK extension). No Code reuse

4. Basic Usage

4.1 Quick-Reference Cheatsheet Card

Operation Command Notes
Print specific column awk '{print $1}' file.txt Prints 1st column separated by whitespace
Custom delimiter awk -F: '{print $1, $7}' /etc/passwd Uses : as field separator
Filter by condition awk '$3 > 100 {print $1, $3}' data.txt Prints when 3rd field exceeds 100
Match regex on field awk '$1 ~ /^192\.168/ {print $1}' access.log Matches IP address prefix
Sum a column awk '{sum += $1} END {print sum}' numbers.txt Accumulates total and prints at EOF
Count rows matching pattern awk '/ERROR/ {count++} END {print count}' app.log Pattern occurrence counter
Print line numbers with content awk '{print NR, $0}' file.txt Prefixes current record number

4.2 Printing Specific Columns

Printing username and shell from /etc/passwd:

awk -F: '{print $1, $7}' /etc/passwd | head -n 3
root /bin/bash
daemon /usr/sbin/nologin
bin /usr/sbin/nologin

4.3 Pattern Filtering

Printing processes consuming more than 10% CPU:

ps aux | awk '$3 > 10.0 {print $1, $2, $3, $11}'

5. Practical Operations

5.1 Computing Column Totals and Averages

Calculating the total and average memory consumed by Nginx worker processes:

ps -C nginx -o rss= | awk '{sum += $1; count++} END {printf "Total: %.2f MB | Avg: %.2f MB\n", sum/1024, (sum/count)/1024}'
Total: 48.50 MB | Avg: 12.13 MB

5.2 Aggregating Frequencies with Associative Arrays

Counting HTTP status codes from a web server access log:

awk '{status[$9]++} END {for (code in status) printf "%-5s : %d\n", code, status[code]}' access.log
200   : 14820
301   : 412
404   : 95
500   : 12
  • Technical Analysis: status[$9]++ dynamically allocates keys in the associative hash array status and increments counters in a single pass.

5.3 Formatting Formatted Output via printf

awk -F: 'BEGIN {printf "%-15s %-10s\n", "USER", "UID"} {printf "%-15s %-10d\n", $1, $3}' /etc/passwd | head -n 4
USER            UID       
root            0         
daemon          1         
bin             2

6. Advanced Usage

6.1 Multi-Line Records via Custom RS and FS

Parsing stanza-based files (like /etc/network/interfaces or paragraphs) separated by blank lines:

awk 'BEGIN {RS=""; FS="\n"} {print "Paragraph:", NR, "Lines:", NF}' document.txt
  • Setting RS="" configures paragraph mode where blank lines separate records, and newlines separate fields.

7. Exit Status, Environment, and Configuration

7.1 Built-in Variables Table

Variable Description Default
NR Total number of input records read so far across all files. Dynamically updated
FNR Input record number in the current file. Resets per file
NF Number of fields in the current input record. Dynamically updated
FS Input field separator regular expression. " " (whitespace)
OFS Output field separator string. " " (space)
ORS Output record separator string. "\n" (newline)

7.2 Exit Status Codes

Exit Code Meaning
0 Success.
>0 Program syntax error, unreadable input file, or explicit exit N call.

8. Safety, Security, and Portability

8.1 Passing External Shell Variables

Warning

Direct shell variable interpolation into AWK scripts (awk "{print \"$USER_INPUT\"}" file) creates critical shell injection vulnerabilities. Always use -v target="$USER_INPUT" to pass variables safely into the AWK runtime environment.

  • Vulnerable Injection Pattern: 
    awk "{print \"$USER_INPUT\"}" file
  • Safe Parameter Injection: 
    awk -v target="$USER_INPUT" '$1 == target {print $2}' file

9. Best Practices

  1. Use -F Instead of Complex Substrings:
    • Guidance: Set explicit delimiters (awk -F, or awk -F:) rather than using substr().
    • Authoritative Justification: GNU documentation notes that native regex field splitting is significantly faster and handles variable column widths cleanly.
  2. Inject Shell Parameters with -v:
    • Guidance: Always pass outside variables via awk -v name="$val".
    • Authoritative Justification: Prevents shell escaping errors and code injection.
  3. Use END Blocks for Aggregate Math:
    • Guidance: Accumulate variables during line cycles and format results in the END block.
    • Authoritative Justification: Standardized by POSIX and eliminates temporary files.

References

  1. GNU Awk (GAWK) User's Guide: https://www.gnu.org/software/gawk/manual/gawk.html
  2. POSIX.1-2024 awk Specification: The Open Group Base Specifications Issue 8. https://pubs.opengroup.org/onlinepubs/9799919799/utilities/awk.html