← All posts

Group the alarms, then read the cause

The syslog was already in Splunk. An agent groups it into episodes, then Splunk returns the chassis line the alert list never had.

A network operations team had Splunk full of protocol syslog, and an alert list that never fired. Reading those messages one by one was not a job a person could finish. I connected an agent to that Splunk through the Model Context Protocol so the on-call question could be “what happened”, and the answer could be a cause.

Splunk is doing the storage and the grouping. The agent is only allowed to ask.

The alert list was the obvious first ask. Splunk keeps alerts as their own objects, separate from the indexed messages. I counted the definitions. Dozens of them were enabled. None of them had fired. The messages that mattered were ordinary protocol syslog, and there were thousands of them. An empty alert list is a true Splunk answer. The incident is still in the index, and a search is how you reach it. The search runs inside the app that owns the field extractions, so device, signature, and interface come back as fields.

Splunk already classifies the lines worth keeping. Event types mark an interface going down, an adjacency dropping, a session timing out, and they leave out the lines that repeat all day. A saved search then groups what remains. On one device, a new episode starts when the quiet gap is longer than five minutes. Across devices, a five-minute window counts when more than one device speaks. Splunk writes those episodes into a summary index on a schedule. You read the summary. You do not re-scan the raw syslog to see that an episode exists.

The cause is a second search, bounded by the episode’s start and end, and this one includes the hardware lines the episode filter left out. Splunk returns the row. The agent reads it.

Three episodes were enough to name three causes.

The first was one router whose episode held every protocol at once, and lasted about half a minute. Other routers lost the same neighbor in that same second. The follow-up search returned the shelf manager reloading every card: both control processors, the fabric, the line cards. The interfaces had already gone down twenty seconds earlier. The optic alarms in that minute were clears.

The second was a core router. The reload line said a person had started it. About fifteen minutes later, a window across many routers was mostly traffic-engineering policy interfaces going down and coming back. The network was settling after the control processor returned.

The third never left one router. One interface went down and up on a fixed interval of a few seconds, for the better part of an hour. No neighbor in the same Splunk index recorded the other end. No adjacency fell inside the episode. The interface was flapping on its own.

The order is dull, and it is the whole method. Ask Splunk which episode is large. Ask which line is first. Then ask, still in Splunk, what the episode filter threw away. The thrown-away line was the reload.

I wrote that order down as a skill, so the next session starts from the summary index and the excluded line. Splunk remains the system that runs the search. The skill is who may ask, and which question comes next.

If the syslog is already in Splunk, start with the episode, and let Splunk return the line it excluded.