“Press Acknowledge All and Reset All, they said. Everything will be fine, they said.”
An alarm system’s job is to tell an operator the one thing they need to know right now. Its failure mode is telling them four thousand things, at which point it has become noise and they will stop looking at it — which is worse than having no alarm system, because everyone believes they’re being watched over.
Alarm state is already in the records. Nothing extra is needed to generate it:
record(ai, "SR-C05-VA-IP-03:Pressure") {
field(HIGH, "1e-8") field(HSV, "MINOR")
field(HIHI, "1e-7") field(HHSV, "MAJOR")
field(HYST, "1e-9") # anti-chatter
}
Every record computes SEVR and STAT on every process. The alarm system is a set of services that subscribe to those fields, apply a hierarchy and a workflow, annunciate, and record what happened. See core concepts.
The current standard. Three cooperating services plus Phoebus UI applications, coordinated through Apache Kafka.
| Documentation | Alarm system UI |
| Source | github.com/ControlSystemStudio/phoebus |
| This guide | Install the alarm system |
flowchart LR
IOC["IOCs<br/>SEVR / STAT"]
AS["Alarm Server<br/>state machine,<br/>latching, delays"]
K["Kafka<br/>config + state topics"]
UI["Phoebus<br/>Alarm Tree / Table / Panel"]
AL["Alarm Logger<br/>→ Elasticsearch"]
ACL["Alarm Config Logger<br/>→ git"]
EM["Email / SMS<br/>annunciation"]
IOC -->|"CA/PVA monitors"| AS
AS <--> K
K <--> UI
K --> AL
K --> ACL
AS --> EM
The engine. Monitors a configured PV list, maintains each alarm’s state machine, and applies the logic that makes an alarm system usable rather than merely noisy:
Consumes the Kafka topics and writes every alarm transition and every acknowledgement into Elasticsearch. That gives you searchable alarm history: how often does this trip, at what time of day, and who acknowledged it?
Which is what you need for the only reliable way to improve an alarm system: measure which alarms fire most, and fix or delete them.
Records every configuration change into a git repository. Answers “who added this alarm, when, and why was that threshold changed?” — a question that comes up during every incident review.
It looks like heavy infrastructure for annunciating alarms. The payoff:
The cost is a Kafka cluster to operate. For a facility this is worth it. For a test stand it is not, and record-level alarm limits displayed on a Phoebus screen are entirely sufficient.
| System | Status |
|---|---|
| ALH (Alarm Handler) | The original Motif alarm GUI. Still running at some facilities. Configuration files can inform a migration. |
| BEAST | The CS-Studio-era alarm system, RDB-backed. Direct ancestor of the Phoebus system; migration tooling exists. |
| Grafana alerting | Fine for infrastructure. Not an operational alarm system: no severity model, no latching, no acknowledgement workflow, no relationship to record fields. Don’t let it become the machine’s alarm system by accident. |
| Facility-specific | Several labs run their own, usually with local annunciation and paging integrations. |
The hardest part, and it isn’t software. The relevant industrial standards are IEC 62682 and ISA-18.2 (“Management of Alarm Systems for the Process Industries”), and while a synchrotron isn’t a refinery, the human factors are identical.
Every alarm must require a specific operator action. If the answer to “what should the operator do?” is “nothing” or “tell someone tomorrow”, it is not an alarm. Make it a status indicator, a report, or a maintenance ticket.
Target rate: an operator can handle roughly one alarm every few minutes, sustained. Published guidance is around ten per hour average, with bursts tolerable. A system generating hundreds per hour is generating zero, because it is being ignored.
Design for the flood. A beam dump sets thousands of records into MAJOR within a second. If your display shows all of them, the operator cannot find the cause. Mitigations:
Write the guidance text. Every alarm gets: what it means, what the likely causes are, what to do, and who to call. Empty guidance fields are the difference between an alarm system and a list of red rectangles.
Review the alarm log monthly. Rank alarms by frequency. The top ten are your problem list — each is either a real recurring fault worth fixing, a threshold that’s wrong, or an alarm that shouldn’t exist. This single practice does more for alarm quality than any amount of configuration.
Set thresholds where the record is, not where the screen is. A screen-side threshold is invisible to the alarm system, the archiver, and everyone not looking at that screen.
Use HYST and MDEL. A value oscillating across a threshold generates an alarm storm from a single physical condition. Hysteresis costs one field.
!!! danger “Alarms are not interlocks” An alarm tells a human something. An interlock acts. They have different requirements, different reliability targets, and belong in different systems. Anything that must act to protect people or equipment belongs in certified PLC or hard-wired logic; EPICS may alarm on it and display it, and must not be relied upon to perform it.
Getting a human’s attention:
| Channel | Notes |
|---|---|
| Visual — Phoebus alarm panel, always-on control-room display | The baseline. |
| Audible — Phoebus has an annunciator | Effective, and universally muted within a week if the rate is too high. Rate discipline is the prerequisite, not the volume. |
| Email — built-in automated action | Good for engineers, useless for anything urgent. |
| SMS / paging / on-call | Facility-specific integration, usually a script triggered by an automated action or a Kafka consumer. Route to a rota, not an individual. |
| Chat integration (Slack/Teams/Matrix) | Increasingly common via a Kafka consumer. Convenient; not a substitute for a rota with an escalation path. |
The Helios alarm plan builds a complete hierarchy for a synchrotron: the tree structure, which subsystems get delays, how a beam-dump flood is kept legible, and the on-call routing.