60-Second Linux Analysis, Supercharged with Nix and LLMs

Piotr Grabowski

tl;dr:
Analyze health of your server based on the famous “60-second Linux Performance Analysis” but:

  • One command to run it all
  • Fast
  • Just works – no sudo, no Docker, no installation of system-wide packages
  • An easy-to-understand LLM summary at the end – no need to read walls of command outputs

Try it:

# export OPENAI_API_KEY=...
curl -fsSL https://gradient.engineer/60-second-linux.sh | sh

Background

In 2015, Brendan Gregg’s Netflix Tech Blog post introduced an excellent “60-second” playbook for quick Linux server diagnostics:

  1. uptime
  2. dmesg | tail
  3. vmstat 1
  4. mpstat -P ALL 1
  5. pidstat 1
  6. iostat -xz 1
  7. free -m
  8. sar -n DEV 1
  9. sar -n TCP,ETCP 1
  10. top

This set of commands enables you to quickly identify CPU, memory, disk, and network bottlenecks and determine which areas require further investigation.

What’s the challenge? Not every Linux distribution includes these tools by default (e.g., iostat and mpstat might be missing on minimal installations). During an outage, installing missing packages wastes precious time – especially when dealing with firewalls, immutable filesystems, outdated tool versions, or slow connections, as highlighted in Gregg’s more recent “Linux Crisis Tools” post.

We’re introducing open-source utility – gradient-engineer: it automatically downloads all necessary Linux tools to a temporary directory, executes the “60-second” playbook, and optionally summarizes results with an LLM. Our tool leverages Nix package manager to create a portable “toolbox” of Linux programs – without requiring root access, Docker, or permanent installations (everything is cleaned up when finished).

First challenge: Tools aren’t always there

A common scenario: you connect to a remote server via SSH, want to perform basic troubleshooting, and immediately encounter:

$ iostat
bash: iostat: command not found

Every time I SSH to a newly created server, I’m frustrated that I can’t use my favorite tools (like ripgrep, htop, jq, tcpdump). It turns out LLMs can get frustrated too – here’s a snippet from Gemini 2.5 Flash reasoning while debugging a network problem:

The inability to run tcpdump and iftop cripples my real-time packet inspection.

As mentioned in Gregg’s Linux Crisis Tools post, installing tools mid-crisis can be problematic: timed-out package repositories, permission issues, or having only outdated tools in your distribution.

Our utility solves this problem by downloading tools on-demand without altering your system: a portable toolbox of necessary Linux utilities ready for the job. [1]

[1] Even though our utility greatly simplifies running troubleshooting tools, there are still some valid reasons to have them installed beforehand: slow network, slow or full disk might hinder our utility from downloading the tools.

Solution: Nix-powered Toolbox

Traditional Linux package managers (apt, yum) require root access, fail on distribution mismatches, and leave system clutter. Docker is heavy and requires root privileges. Manually compiling static builds is difficult for most software. We wanted something much more lightweight, in the spirit of modern package managers like NPM, pip, or Cargo.

So that is why we picked Nix.

Nix is a package manager that treats software and dependencies as pure code definitions, powered by its own functional language. Builds are reproducible and isolated. What’s particularly great for us is that Nix-built programs are independent of your Linux distribution and self-contained: they only use libraries/dependencies from other Nix packages, not from the system package manager. Moreover, Nix doesn’t rely on containers, VMs, or emulation – Nix-built programs are standard Linux executables.

In practice, this means you can copy the Nix root directory (/nix), move it to another computer, and run programs without issues. Since we copy the toolbox to a temporary directory rather than the /nix root directory, we use PRoot to redirect paths to the correct folder.

The Nix ecosystem includes nixpkgs – a vast repository of Nix-packaged programs (over 120,000 software packages), making it easy to extend our toolbox with additional tools in the future. It’s also straightforward to modify build instructions for programs defined in nixpkgs, allowing us to easily build custom versions of any packages.

AI-powered insights: From walls of text to actionable summary

Raw outputs from all “60-second” playbook commands are useful, but interpreting them can be challenging (what do us, sy, id, wa, st, avgqu-sz mean in command outputs?). Fortunately, this is where LLMs shine:

  • Even less-powerful LLMs are great for summariztion tasks
  • The full command output is included in the LLM prompt, reducing hallucination risk
  • Since we use standard Linux tools, LLMs are familiar with their command-line interfaces and outputs
  • LLM gives you a short overview, possible interpretations and next step suggestions - you’re still fully in control.

What’s even more powerful is that we can go beyond just running 10 commands sequentially: LLMs can analyze output from dozens of tools in seconds. With the “toolbox,” LLMs can easily use powerful troubleshooting tools.

Demo: Putting it to the test

We ran our utility on a server with overloaded disk with the following command:

# export OPENAI_API_KEY=...
curl -fsSL https://gradient.engineer/60-second-linux.sh | sh

In less than a minute, we received this report:

The system shows severe I/O bottleneck with CPU spending excessive time waiting for disk operations, primarily caused by a write-intensive process.

asciicast

Feedback and future ideas

This is an early prototype, and we’re excited to see how we can improve upon the initial idea. The repository is open-source at github.com/QuesmaOrg/gradient-engineer.

We believe there’s tremendous power in giving LLMs access to a vast toolbox with Nix. We could develop sophisticated playbooks with hundreds of commands: LLMs could analyze outputs of powerful programs such as BPF tools (like bpftrace), profilers, and Kubernetes to troubleshoot complex problems.

Another idea is to offer the toolbox as a Model Context Protocol (MCP) server. Instead of “command not found” errors, LLMs could confidently execute comprehensive workflows knowing bpftrace, perf, tcpdump, and hundreds of tools are ready – with no installation needed!

Finally, LLMs could suggest commands rather than rely on fixed playbooks. With the toolbox, LLMs know that all tools are available and don’t get distracted with installing missing packages. For safety, the utility could ask for permission before each command and implement sandboxing.

We’d love your feedback. Try it out and star the project on GitHub if you find it useful!

Related Articles

Continue exploring similar topics