Learn › Bash Redirection and File Descriptors › Incident lab: a trustworthy report exporter
Incident: repair a report exporter. Try the examples, inspect their output, and explain the result. Practice feedback does not award XP or mastery.
A report exporter copies transformed data into a CSV file. The source fails halfway through. The exporter still reports success, its CSV contains a diagnostic, and the last good report has disappeared.
This final incident brings together routing, stage status, validation, temporary files, and publication. Work in a private disposable directory with Bash, Python 3, and GNU coreutils. The course folder is available through LC_COURSE. Treat its supplied files as reference material: copy the starter into your private directory before editing.
The producer is a local simulation, not a network service. Its CSV is text with one row per line and fields separated by commas. Input rows contain a name and count. It uppercases names and adds a header. A row beginning ERROR simulates a source interruption with status 23. Rows written before that interruption remain observable.
In your managed Bash practice terminal
Run camp-redirection 15 to enter this chapter's private directory. While this lab machine is running, the same command returns to it after reconnecting; camp-redirection 15 --fresh creates a new one without deleting earlier files. Run the code blocks there and use cd to leave.
Your edited export-report.sh is saved in this directory. Returning to it does not reset the file. The first reproduction below runs only in a clean chapter directory; if you have already done it, keep your work and use camp-redirection 15 --fresh for another broken-starter experiment. The block also refuses to replace existing experiment files.
The starter sends the producer's stderr into stdout, pipes that mixture through tee, and prints a success message afterward. GNU tee writes a copy to the destination while also forwarding bytes on stdout.
Before running the supplied starter, predict which bytes could enter the saved report and whether the final success message proves that the producer completed. This block makes an editable working copy and a separate broken-starter copy. It checks for existing files first so a resumed learner repair is not replaced.
existing=no
for path in export-report.sh broken-starter.sh interrupted.csv report.csv observed.txt observed-status.txt; do
if [[ -e $path || -L $path ]]; then existing=yes; break; fi
done
if [[ $existing == yes ]]; then
printf 'Keep this directory; use camp-redirection 15 --fresh for a new broken-starter experiment.\n' >&2
else
cp "$LC_COURSE/practice/exporter/starter.sh" export-report.sh
cp "$LC_COURSE/practice/exporter/starter.sh" broken-starter.sh
printf 'elm,4\nERROR,23\n' > interrupted.csv
printf 'last good report\n' > report.csv
PYTHONUNBUFFERED=1 bash ./broken-starter.sh interrupted.csv report.csv > observed.txt
status=$?
printf 'status=%s\n' "$status" > observed-status.txt
fiOn a clean run, the wrapper's stdout goes into observed.txt; the assignment captures its status immediately. The guard treats an existing path or symbolic link as prior work. PYTHONUNBUFFERED makes the producer's write order reproducible for this demonstration. On a resumed run, the message tells you how to repeat without changing any existing files.
Try in the practice terminal: capture the broken exporter
cat observed-status.txt observed.txt; printf 'saved file:\n'; cat report.csvExpected result
status=0
name,count
ELM,4
producer: source interrupted
Published report
saved file:
name,count
ELM,4
producer: source interruptedThree observations are separate evidence: status is zero, diagnostic text has entered the data stream, and the old file is gone. Fixing one does not automatically fix the other two.
| Stage | Descriptor or status consequence | Defect |
|---|---|---|
Producer with 2>&1 | Its stderr duplicates its stdout pipe destination | Diagnostics enter CSV |
tee opens report.csv | Existing report is truncated before source completion | Last good artifact is exposed |
| Pipeline completes | Default pipeline status follows tee | Producer status 23 can disappear |
| Final printf succeeds | Wrapper's final status becomes zero | Even a retained pipeline failure can be overwritten |
Adding pipefail addresses the default pipeline status, but the final successful print can still hide it. Keeping the status does not recover a file already truncated. Splitting stderr keeps the data clean, but partial data can still replace a complete report.
The repair needs a deliberate sequence, not one more operator appended to the broken line.
On success, both stdout and the destination must contain the same valid CSV. Stdout contains no progress messages. On a source interruption, return 23. On invalid output, return 65. On preparation, stdout, or publication failure, return 74.
The supplied validator requires a name,count header, at least one data row, uppercase identifier names, and nonnegative integer counts. A successful producer can still produce an invalid row. Source status and data validity answer different questions.
Every failure before publication must leave an existing destination byte-identical. Cleanup must remove only the temporary file created by this invocation. Use a trusted private directory; reject an existing destination symlink or directory. These checks do not secure an attacker-controlled directory against races.
Edit your copied export-report.sh. Start by separating data from diagnostics. Send producer stdout to a uniquely created temporary file beside the destination; leave stderr as stderr. Capture the producer's nonzero status in the failure branch before another command replaces it.
Next, invoke the supplied practice/exporter/validate.py on the completed temporary file. Stop with 65 if validation fails. Attach cleanup as soon as temporary creation succeeds. Remember the incoming status inside the exit trap before removing the temporary name.
Finally, check delivery of the accepted CSV on stdout, then rename the temporary file onto the destination. Check both operations. A GNU mv -T destination is one pathname, which avoids silently moving a report into an existing directory.
This is the independent repair task. Attempt it before opening practice/exporter/solution.sh. Reading that file is a recovery option when you need to compare the complete control flow, not evidence that your version works.
The public checker runs your script in separate temporary directories with a deadline. It tests changed names and counts, spaced paths, an identical source and destination, partial and immediate source failures, missing input, invalid data, and unavailable destinations. Linux additionally checks a destination symlink and /dev/full as stdout.
Run the checker against the export-report.sh you edited. The unchanged starter must fail. Treat a passing check as evidence for the tested cases, then inspect the changed payload and failure evidence yourself.
Check your repair
python3 "$LC_COURSE/practice/exporter/check.py" ./export-report.shExpected result when your repair passes all supplied checks
PASS success with spaces
PASS changed data
PASS same source and destination
PASS partial source failure
PASS immediate source failure
PASS missing source
PASS invalid negative count
PASS invalid row shape
PASS empty source
PASS destination directory
PASS missing destination parent
PASS destination symlink
PASS stdout write failure
13/13 checks passedThe checker accepts behavior, not a memorized script. Hardcoding ELM cannot pass changed inputs. Converting status 23 to success cannot pass the interruption cases. A failure case also checks the previous report, so merely returning a nonzero status is insufficient.
The solution avoids a producer pipeline entirely. It checks the producer, the validator, stdout copying, and the final rename separately. The temporary file stays private under umask 077. Its exit trap retains the failure status while removing an unaccepted temporary file.
If your own check does not pass, compare it with the supplied reference. This next command checks solution.sh, not your edited file:
python3 "$LC_COURSE/practice/exporter/check.py" "$LC_COURSE/practice/exporter/solution.sh"The supplied reference passes 13 of 13 checks on the tested Linux environment. Its result cannot establish that your export-report.sh is repaired.
printf 'oak,9\nERROR,23\n' > failed-source.csv
printf 'last good report\n' > preserved.csv
bash "$LC_COURSE/practice/exporter/solution.sh" failed-source.csv preserved.csv > rejected-output.txt 2> failure.txt
status=$?
printf 'status=%s\n' "$status" > failure-status.txtThis run captures both streams and the status so you can inspect them without mixing their meanings.
Try in the practice terminal: verify that an interruption preserves the old report
cat failure-status.txt failure.txt preserved.csv; printf 'stdout bytes='; wc -c < rejected-output.txtExpected result
status=23
producer: source interrupted
last good report
stdout bytes=0After a failed attempt, keep the old report available, inspect the stderr diagnostic, fix the source or destination problem, and rerun. Do not rename a rejected temporary file manually to make the check appear successful.
Stdout and a filesystem pathname cannot be committed together as one transaction by this script. The solution sends validated CSV before rename so a stdout failure preserves the old file. A later rename failure can therefore follow delivery of CSV. Callers must inspect status; receiving bytes alone is not a publication receipt.
The solution offers atomic replacement in a trusted directory. It does not provide crash durability, concurrent-publisher ordering, protection against hostile directory mutation, or guaranteed cleanup after SIGKILL. Those are separate design requirements, not promises established by thirteen checks.
Run the checker against your own file, then change a source payload and repeat the interrupted-source experiment. Before opening the supplied solution, write down your prediction and the earlier Chapter 1, 8, 11 or 12 mechanism you used. Submit a descriptor trace before and after your repair. Explain the first failed stage, the returned status, which bytes each stream contains, and why the previous report survives. Record which hints or worked solutions you used.
Return on another day without first opening the worked solutions. These changed cases test whether you can transfer the mechanisms; a completed reading or checker run alone is not independent evidence:
1. Start with an old report whose bytes end in two newlines and save a byte-for-byte baseline. Make the source fail after writing a row. Predict the old report and stdout bytes, then use cmp against the baseline and inspect status and diagnostics. This returns to Chapter 9's exact-byte problem. A successful CSV conversion need not preserve arbitrary input bytes; a rejected attempt must preserve the old destination. 2. Return to Chapter 13 with two new record names, each written in three fragments under one stable lock. Make one writer fail before writing any fragment. Collect both writer statuses and inspect the log for the successful whole record without assuming which writer ran first. Explain why locking each fragment separately is insufficient. 3. Put three new input rows through a local job loop whose child could drain the remaining input, then feed the completed rows to your repaired exporter. Make required stdout delivery fail with /dev/full. Predict which child owns input, the exporter's status and whether the previous report survives. Repair the child input boundary and the exporter separately, then verify both. No remote host is needed.
For each return, keep the prediction, command or file evidence, explanation and hint use together. If a check fails, revise the causal explanation before retrying.
A complete answer combines working behavior with a recovery explanation. The course ends with a method you can reuse: identify ownership, preserve failure meaning, validate the result, and publish only when the operation's contract permits it.
Practice Chapter 15. Incident: repair a report exporter in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.