Five bad mornings, step by step

Each of these is a real failure, and each is a sequence of tabs and keys in siltide. Nothing here is theory: the steps are what to press and what to conclude from each screen.

guide

A device is held but doing nothing.

The most expensive state a fleet can be in: the scheduler counts the device as taken, the bill counts it as running, and nothing is using it.

guide

A card is slower than its neighbors.

Identical devices in one box should do identical work. When one does not, the reason is usually written down somewhere siltide already reads.

guide

A device fell off the bus.

An Xid, an ECC storm or a device that simply stopped answering. The question is always the same: what did it do before it went.

guide

Memory is full and utilization is zero.

A device with its memory taken and nothing computing is usually a process that has allocated and then gone to sleep, or one that died without releasing.