Learn › Bash Redirection and File Descriptors › Here strings, substitutions, and newline fidelity

Chapter 9. Strings, substitutions and exact bytes

Strings, substitutions and exact bytes. Try the examples, inspect their output, and explain the result. Practice feedback does not award XP or mastery.

Choose a transport for bytes

A saved report contains two final newline bytes. After a script stores it in a variable and sends it through a here string, one newline has vanished. The displayed text looks almost identical, but its checksum changes.

Byte fidelity means preserving the exact byte sequence, including invisible separators and the absence of a final newline. Decide whether the next program needs exact bytes or only a text value before choosing how to feed it.

Use Bash 5.2 or later, od, and cmp in your private directory. Process substitution examples require a supported system with usable FIFO or /dev/fd facilities. A process substitution supplies a filename connected to a running command; it is not an ordinary temporary file with all ordinary-file properties.

In your managed Bash practice terminal

Run camp-redirection 09 to enter this chapter's private directory. While this lab machine is running, the same command returns to it after reconnecting; camp-redirection 09 --fresh creates a new one without deleting earlier files. Run the code blocks there and use cd to leave.

Make the newline loss visible

od displays byte values. Its -An -tx1 options omit offsets and show one-byte hexadecimal values. Hexadecimal 41 represents the letter A; 0a represents a newline.

Predict how many 0a bytes will appear for the file and for the variable. Compare both displays to check what command substitution removed.

printf 'A\n\n' > original.txt
printf 'file:\n'
od -An -tx1 original.txt
value=$(cat original.txt)
printf 'variable:\n'
printf '%s' "$value" | od -An -tx1

Expected output

file:
 41 0a 0a
variable:
 41

Command substitution, written $(...), removes trailing newline bytes from its captured output. It retains embedded newlines. Double quotes prevent later word splitting and filename expansion; they do not disable the capture rule.

Measure what a here string supplies

A Bash here string, <<<, supplies a word's expanded value followed by one newline. This often suits a line-oriented reader. It is not an exact-byte replacement for a file or pipe.

Compare both hex outputs; a here string adds a newline even when its value is empty.

value=A
od -An -tx1 <<< "$value"
printf 'empty here string:\n'
od -An -tx1 <<< ''

Expected output

 41 0a
empty here string:
 0a

Combining command substitution and a here string can therefore remove several final newlines and add one back. It does not preserve the original count. Use an ordinary file argument or input redirect when exact file bytes matter.

Keep binary bytes out of shell variables

A NUL byte has hexadecimal value 00. Bash variables cannot serve as arbitrary binary buffers. Avoid capturing binary output into a variable; transport it through descriptors or keep it in a file.

Check that cmp succeeds before reading the hex dump; both files should retain the same embedded NUL byte.

printf 'A\0B\n' > binary.dat
cat < binary.dat > copied.dat
cmp binary.dat copied.dat && od -An -tx1 copied.dat

Expected output

 41 00 42 0a

cmp succeeds only when both files contain the same bytes. Its silence is useful evidence here because && runs the byte display only after the comparison succeeds. The printf builtin creates the NUL directly from its format string.

Separate the consumer's result from its producer

Bash process substitution, <(...), can avoid command substitution's newline stripping. Its producer runs asynchronously, however. A consumer's success does not automatically include that producer's success.

(
exec {input}< <(printf 'row\n'; exit 6)
producer=$!
cat <&"$input"
consumer_status=$?
exec {input}<&-
if wait "$producer"; then producer_status=0; else producer_status=$?; fi
printf 'consumer=%s producer=%s\n' "$consumer_status" "$producer_status"
)

The output is row, followed by consumer=0 producer=6. The redirect-only exec opens an allocated descriptor in the surrounding subshell. Save $! immediately there; another asynchronous operation can replace it. Waiting from an unrelated child shell cannot collect this subshell's child status. The descriptor closes when the subshell ends, and the managed parent retains its original descriptors.

Owner	Descriptor or status	Meaning
Parent shell	Allocated input descriptor	Process-substitution read end
Producer process	1	Matching write end
Consumer	0	Copy of the parent's read end
Parent shell	Saved producer PID	Separate completion result for wait

Close output before waiting for its reader

The companion Bash syntax >(...) gives a writer access to a consumer's input. Close the writing descriptor before waiting, so the consumer can reach end of input.

(
exec {sink}> >(cat > received.txt)
receiver=$!
printf 'delivered\n' >&"$sink"
exec {sink}>&-
wait "$receiver"
cat received.txt
)

This prints delivered. The surrounding subshell owns both the allocated descriptor and the reader process; it closes the writer and waits before ending. A fixed sleep would guess at scheduling instead of establishing completion. Chapter 10 explains why leaving the writing descriptor open can stall that wait.

Preserve a changed payload

Try this before reading the solution: create a payload containing a NUL and two final newlines. Copy it using a file argument, a pipeline, and process substitution. Compare every copy with the original without storing its contents in a variable.

(
  printf 'Z\0Q\n\n' > payload.dat
  cat payload.dat > by-file.dat
  cat payload.dat | cat > by-pipe.dat
  exec {source}< <(cat payload.dat)
  producer=$!
  cat <&"$source" > by-substitution.dat
  consumer_status=$?
  exec {source}<&-
  if wait "$producer"; then
    producer_status=0
  else
    producer_status=$?
  fi
  if (( producer_status != 0 )); then
    printf 'producer failed: %s\n' "$producer_status" >&2
    exit "$producer_status"
  fi
  if (( consumer_status != 0 )); then
    printf 'consumer failed: %s\n' "$consumer_status" >&2
    exit "$consumer_status"
  fi
  cmp payload.dat by-file.dat && cmp payload.dat by-pipe.dat && cmp payload.dat by-substitution.dat && printf 'all copies match\n'
)

The worked result is all copies match. The subshell contains deliberate failure exits, and each important status is saved before inspection replaces it. All three transports can preserve these bytes. Process substitution adds lifecycle work; use it for an actual filename-interface need. A consumer that seeks backwards may require a regular file.

Ready to practice

Repeat with no final newline, then an empty file. Explain why the file-based method needs no newline repair. Also make the process-substitution producer fail and verify that your script records its separate status before accepting the result.

Now return to Chapters 5 and 6 after these intervening lessons. Make a new later-value.sh that contains a literal variable reference, alongside a payload with two trailing newline bytes. This scaffold creates the two files without deciding the value the script will read:

cat > later-value.sh <<'SCRIPT'
printf 'value=%s\n' "$value"
cat payload.txt
SCRIPT
printf 'A\n\n' > payload.txt

Before executing the script, predict which text was fixed when the file was generated and which value will be read later. Run it twice with different value assignments and save each output. Inspect later-value.sh as text and use od -An -tx1 on the payload and saved outputs. Explain why both outputs retain the payload's two final newline bytes. Record whether you reopened the earlier heredoc explanation or used a hint.

References: Bash redirections, Bash process substitution.

Practice Chapter 9. Strings, substitutions and exact bytes in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.