By Tom —

The Case of the Vanishing Status Bar

A YouTube video, a realtime signal, and five wrong turns before the actual bug confessed.

I spent a night migrating my personal dotfiles to NixOS — sway, waybar, the works, ported into home-manager as a fully reproducible flake. Almost everything came up clean on the first real reboot. One thing didn't: every time I reloaded the sway config, the status bar would vanish and never come back. This is the log of chasing that down, including the four theories that felt right and weren't.

The symptom

Fresh boot: waybar shows up fine. Press $mod+Shift+C to reload the sway config — an entirely routine action, done a dozen times a session — and the bar disappears. Not "flickers and comes back." Gone, for good, until the next full login.

Starting it back up by hand always worked. That detail turned out to matter more than anything else in this story.

Suspect no. 1 — the name that wasn't there

(Already fixed, earlier in the same investigation)

The wrapper script that launches waybar used to open with killall waybar before starting a new one — standard "don't run two of these" hygiene. Except nixpkgs wraps the real binary for GTK theming, so the actual process shows up in /proc as .waybar-wrapped, not waybar. killall matches on the exact name. It matched nothing, ever, on every single reload — which meant every reload just piled another waybar on top of the last one.

Fix: track the PID directly in a file instead of matching by name. That solved the duplicate bars problem cleanly. It did not solve today's problem, which is bars not existing at all.

Suspect no. 2 — a number that meant something else

(Ruled out)

The PID-tracking fix had its own hole: once the tracked process actually exits, the kernel is free to hand that same PID to something completely unrelated. kill -0 $OLD_PID would then report "yes, alive" forever — just not about waybar. The wait loop would spin against a process that was never going to die from a signal meant for something else.

Fix: before trusting a stored PID, confirm /proc/$PID/exe still actually resolves to a waybar binary. Reasonable bug, real fix — but the bar was still disappearing after this shipped.

Suspect no. 3 — the lock that wouldn't let go

(Ruled out — made it worse first)

Next theory: overlapping reloads racing each other against the same tracking file. Reasonable enough to add a lock. Except the launch line backgrounded waybar with a trailing & — and a backgrounded process inherits every open file descriptor from its parent, lock included. The lock wasn't held for "the few milliseconds this script needs to hand off to a new bar." It was held for waybar's entire lifetime, by waybar itself.

Every subsequent reload hung forever waiting for a lock the currently-running bar was quietly sitting on. This was a strictly worse bug than the one it was meant to fix, and I get full credit for writing it.

Fix: close that file descriptor explicitly (9>&-) on the line that launches waybar, so only the wrapper script — never the long-running bar — ever holds the lock.

Suspect no. 4 — the laptop lid

(Ruled out)

Reload also re-runs a script that disables the internal display if it thinks the lid is shut — useful with an external monitor, catastrophic if it misreads the lid sensor with no monitor attached. Checked /proc/acpi/button/lid/LID/state mid-incident: reported open, correctly, every time. Not this.

The evidence that actually mattered

Logging the wrapper script's own actions showed it running fine — lock acquired, old PID handled, new waybar launched, PID recorded. Whatever was going wrong was happening to the process that got launched, not in the logic that launched it. So I stopped reading scripts and started watching processes, sampling the entire tree ten times a second across a live reload:

  PID   PPID  PGID  STAT  ARGS
99633      1  99605   R    waybar -c .../config.jsonc -s .../style.css

One sample later, that PID was simply gone. No trace in coredumpctl — meaning nothing crashed, no segfault, no signal that leaves a corpse behind. A process that vanishes in under a hundred milliseconds, cleanly, with zero output, looks like exactly one thing: it was killed by a signal it never got the chance to handle.

So — what's sending it a signal, milliseconds after it starts, on every single reload?

$ grep -rn "waybar" ~/.config/sway/ ~/.config/waybar/
recorder.sh:        pkill -RTMIN+8 waybar
99-autostart-applications.conf:  pkill -RTMIN+5 waybar
config.jsonc:        "on-click": "...; pkill -RTMIN+9 waybar"

Every single one of these is a custom realtime signal, not a plain kill — waybar uses them internally to know "go refresh this specific module." Harmless, by design. Unless the process on the receiving end hasn't finished telling the kernel "I'll handle that one myself" yet. A realtime signal with no handler registered doesn't get ignored. Its default action is to terminate the process — silently, no output, no core dump. Exactly what we were staring at.

Confirmed

One of the other autostart lines runs a small watcher: playerctl --follow, piped into a loop that fires pkill -RTMIN+5 waybar whenever the currently-playing media changes — so the media widget updates live. That watcher also restarts on every reload. And --follow doesn't wait for a change to report one; it announces the current status the instant it starts.

So on every reload, two things raced out of the gate at once: a fresh waybar booting up, and a media watcher restarting and immediately shouting "the song is: this" at it — via a signal waybar hadn't yet told the kernel it would catch. If nothing was playing, the watcher had nothing to announce, no signal went out, and the race never triggered. If something was playing —

$ playerctl -a status
Playing
$ playerctl -a metadata
firefox … Relaxing Celtic Fantasy Music for Study & Relaxation …

— the bar died before it finished being born. Every time I'd tested this myself over SSH, nothing was playing anything. Every time the bug actually happened, a video was running in the background. The bug wasn't in the reload. It was in the soundtrack.

The actual fix

The watcher never needed to restart on reload in the first place — it doesn't read any config, there's nothing for a reload to give it. So: stop unconditionally killing and re-launching it every time, and only start one if one isn't already running.

[ -x "$(command -v playerctl)" ] && ! pgrep -f "playerctl -a metadata" >/dev/null \
  && playerctl -a metadata --follow | while read line; do pkill -RTMIN+5 waybar; done

No restart, no fresh --follow burst, no signal fired at the exact moment a new bar is mid-handshake with the kernel. The race doesn't get a starting gun anymore.

What this was actually worth

  1. An unhandled realtime signal's default action is to terminate, not ignore. If you send custom signals to a process you don't fully control the startup timing of, there is a window where that signal kills it outright — no crash log, no core dump, nothing to grep for.
  2. exec_always means all of them, together, every time. A reload isn't "restart the bar" — it's every autostart line firing at once, and any two of them can race if their timing ever overlaps.
  3. "Works every time I test it, fails every time you do" is itself a clue. The variable wasn't the machine, the config, or the timing of the keypress — it was ambient state neither of us thought to compare: whether a video happened to be playing.
  4. A backgrounded process inherits open file descriptors by default. A lock you thought you scoped to "this script's critical section" can end up scoped to "this script's child's entire lifetime" instead — check what a trailing & actually hands down before trusting it.
  5. When a process disappears leaving no crash log and no core dump, stop looking for a bug in logic and start looking for something arriving from outside it.

Filed from lftl · sway + home-manager · resolved after five leads, one wrong fix, and a Celtic fantasy playlist.