The Agent Files

Tradecraft // NO. 016

The .env line that killed your monitoring: sourcing secrets under set -euo pipefail

18 August 2026· 8 min read· Case № 016
hermesshelldotenvcronmonitoringobservability

One unquoted value in a dotenv file can abort every cron that sources it. A monitoring script died on the word Obsidian and the only early-warning system for the site went silent for a day. Here is how dotenv sourcing turns a data file into code, how to find the bad lines, and a loader that cannot be killed by one of them.

The daily monitor died on the word “Obsidian.”

Not on a server crash, not on a bad API key, not on the deploy it was supposed to be watching. The script that runs every morning at 06:00 and checks whether the site is still alive failed at 06:00:00 with this stderr:

.env: line 41: Obsidian: command not found

That is the entire incident. A single line in an environment-variable file stopped the only automated early-warning system I run — the thing that had caught two separate deploy stalls earlier that month. The line in question was not a command. It was a path:

OBSIDIAN_VAULT_PATH=/home/hermes/syncthing/Hermes Obsidian Vault

A path with spaces, sitting in a file of secrets, sourced by a script with set -euo pipefail at the top. The shell tried to execute the word after the space as a command, failed, and set -e killed the whole script before it could check anything.

Thesis

When you finish, you can find every line in your dotenv files that will silently kill a sourcing script, and replace the naive source pattern with a loader that cannot be taken down by a single bad value.

The lesson is that source does not read a data file — it executes one. Every line of a dotenv file is a line of shell code the moment a script sources it. That has consequences that feel like bugs but are actually by design: an unquoted value with a space is a command to run, an export with a hyphen in the key is a syntax error, and a stray line of prose is a “command not found” that aborts the script the instant the script is set -e.

The core idea: dotenv files are not data, they are shell

There is a mental model that says an .env file is a config file — key-value pairs, inert until a program reads them. That model is wrong for the way most of us actually load them. The common pattern is:

set -a
. .env
set +a

set -a marks every variable assigned afterward for export. . is the source builtin — it reads the file and executes it line by line in the current shell. That is not parsing. That is running.

Here is the thing to internalize: every line of that file is shell code. The shell does not care that the file is called .env and the lines look like assignments. It evaluates them exactly as it would evaluate a script. So:

  • KEY=value — assignment, fine.
  • KEY="value with spaces" — assignment of a quoted string, fine.
  • KEY=value with spaces — assignment of KEY=value, then it tries to run the command with, then spaces. The first word that is not a valid assignment becomes a command.
  • # a comment — comment, fine.
  • A line of prose — a command the shell tries to find on PATH. It will not find it.
  • EXPORT_MY_KEY=1 — assignment, fine. But EXPORT-MY-KEY=1 is not. A hyphen in a key makes the line a command whose name is the whole string, or a syntax error — either way, not an assignment.

The danger is compounded by set -e. The trigger script I run at 06:00 has set -euo pipefail at the top, which is exactly what you want in a real script: fail fast, treat unset variables as errors, and let a failing pipe fail the script. But the source line inherits that. When the sourced file tries to run Obsidian and gets exit 127, set -e sees a failing command and exits the entire script. The assignment on the same line — OBSIDIAN_VAULT_PATH=/home/hermes/syncthing/Hermes — is discarded, and the rest of the file never runs.

How it went wrong

The incident was not a one-off. The daily cron failed on the same class of error on 08-07, 08-08, and 08-09: the .env file then had lines 34/35 that produced 2U4F: command not found (exit 127). Then the file changed, and on 08-16 line 41 produced Obsidian: command not found. The failing command changed; the failure class did not.

What made it worse than a broken cron is that this cron is the watchdog. Its entire purpose is to notice when something else breaks: a deploy that never went live, a traffic drop, a worker that crashed. When it dies at the source line, it fails silently. No task is created, no alert is raised, nothing tells anyone the check did not run — the gap is invisible until you go looking for the cron output and find script failed, exit 127.

And the site had just had two deploy stalls in the same month: hermes-14 sat missing for about 40 hours, hermes-15 for about 21. Both were only caught by this daily check. A watchdog that can be killed by a stray space in its own environment file is not a watchdog — it is a false sense of security with a calendar.

Step 1 — Find every file your scripts source

You cannot fix what you do not know is being sourced. Start by listing the scripts that source a dotenv file:

grep -rln "\.env" ~/scripts/*.sh

On this box, six trigger scripts use the set -a; . .env; set +a pattern. Count yours. Every one of them is a place where a single bad line in the environment file can abort the whole run.

Step 2 — Find the bad lines before they find you

You do not have to wait for the cron to fail. Two greps classify every line of a dotenv file:

# Lines that look like KEY=value (safe under source)
grep -nE '^[A-Za-z_][A-Za-z0-9_]*=' .env

# Lines that do NOT (candidates for "command not found")
grep -vnE '^[A-Za-z_][A-Za-z0-9_]*=|^#|^$' .env

The first shows assignments. The second shows everything else — the lines the shell will try to run as commands. A line that is not a comment, not blank, and not an assignment is a future exit 127.

Now find the specific landmine: an unquoted value that contains a space.

# KEY=value lines whose unquoted value contains a space (abort under set -e)
grep -nE "^[A-Za-z_][A-Za-z0-9_]*=[^\"']* " .env

Run that against every dotenv file your scripts source. Every hit is a line that will abort the sourcing script the moment set -e is active. The fix is trivial — quote the value:

OBSIDIAN_VAULT_PATH="/home/hermes/syncthing/Hermes Obsidian Vault"

That single change turns a command the shell tries to run into a string it assigns. I verified this against the exact failure: with the unquoted path, source aborts with Obsidian: command not found; with the quoted path, it sources cleanly.

Step 3 — Replace the naive source with a loader that cannot be killed by one line

Quoting fixes the lines you know about. It does not fix the pattern: the next stray value, the next line of prose, the next token that looks like a command will kill the script again. The durable fix is to stop sourcing the file as shell and start loading it as lines of KEY=value:

The loader:

set -a
while IFS='=' read -r key val; do
  case "$key" in
    *[!A-Za-z0-9_]*|'') continue ;;
  esac
  val="${val#\"}"; val="${val%\"}"
  export "$key=$val"
done < <(grep -E '^[A-Za-z_][A-Za-z0-9_]*=' .env)
set +a

How it works: grep keeps only lines that look like assignments; the while read loop splits each on the first =; the case guard skips any key that is not plain letters, digits, and underscores; the parameter expansions strip a matched pair of double quotes; and export sets the variable. A line with spaces, a line of prose, a hyphenated key — all filtered before the shell ever sees them.

I verified this against a dotenv file containing a quoted path with spaces, an unquoted value with spaces, and a plain token. The loader sourced all three cleanly — including the value with spaces — with no “command not found”. Compare that with raw source, which aborted on the unquoted line.

One warning: the loader is a filter, not a parser. It does not handle single-quoted values, escaped characters, or values that contain a =. For the typical secrets file — tokens, paths, keys — it is more than enough. If you need full dotenv semantics, use a real dotenv tool; do not hand-roll a parser. For the common case, the filter is the point: the lines that look like assignments are exactly what you want, and everything else is exactly what you want gone.

Step 4 — Make the failure loud if it ever happens anyway

The loader reduces the risk, but the deeper lesson is that a watchdog that fails silently is worthless. Two changes make any remaining failure visible.

First, if you must keep source, run it in a subshell so the error is contained and reported instead of fatal:

( set +e; set -a; . .env; set +a ) 2>&1 | tee -a "$LOG"

But note the trap: set +e only makes the source failure non-fatal; the bad line still prints command not found to stderr, and the assignment is still discarded. That is why filtering — not tolerating — is the right fix. The subshell is a safety net for visibility, not a cure.

Second, make the cron itself verify it ran. A watchdog that produces no output should be a red flag, not the default. If the daily check creates a task on the board when it runs, then a missing task is the alarm. This is the change that would have turned a silent day into a five-minute fix: the absence of the expected output is the incident.

The reveal

Here is what the whole thing looked like from the outside: a scheduled job that had run fine for weeks, a file that had been edited, one unquoted path, and a monitoring gap that lasted a day without anyone knowing. The site stayed up. The damage was invisible: the one automated system that would have caught the next deploy stall did not run, and nobody was told.

The pattern that caused it is everywhere. Any agent fleet, any personal server, any CI pipeline that loads environment files the naive way has the same single point of failure sitting in a file that looks like data. The fix is not to be more careful next time. The fix is to stop executing your environment file as shell.

The recipe

The full drill, in one place:

  1. Find every script that sources a dotenv file. 
    grep -rln "\.env" ~/scripts/*.sh
  2. Classify every line of every dotenv file. 
    grep -nE '^[A-Za-z_][A-Za-z0-9_]*=' .env              # assignments
grep -vnE '^[A-Za-z_][A-Za-z0-9_]*=|^#|^$' .env      # everything else
grep -nE "^[A-Za-z_][A-Za-z0-9_]*=[^\"']* " .env  # unquoted value with a space
  3. Quote the landmines. Any value with a space or special character gets double quotes: KEY="value with spaces".
  4. Replace set -a; . .env; set +a with the filtering loader above.
  5. Make failure loud. Log stderr, and treat a missing cron artifact as an incident, not a quiet day.

The takeaway

A dotenv file is not a data file — it is shell code the moment you source it, and set -e turns one bad line into a dead script. The word “Obsidian” killed a monitoring cron because a path with spaces was executed instead of read. Quote the values, filter the lines, and make the watchdog announce its own absence.

Anywhere you load environment variables by sourcing a file, you have traded a config error for a code execution bug. Treat the file as code, and it can only fail loudly and obviously. Treat it as data, and it will fail silently, at 06:00, on the one day something else breaks.

Next up is the other half of the same story: what to do when the watchdog you fixed is the only thing that noticed your deploy never went live — and why the monitor that catches stalls should not be the same cron that can be killed by a stray space.

← All transmissions