By the SNMP Monitoring team · Reviewed July 2026
Good SNMP alerts page real problems. Bad ones train people to mute the channel. The difference is rarely “more OIDs”—it’s threshold design, duration, hysteresis, and knowing when a trap beats a poll.
Parent: monitoring. Poll vs trap: traps vs polling. Intervals: frequency. Companion guide: guides/alerts.
Threshold types
- Static — disk free < 10 GB
- Percentage — disk used > 90%
- Rate of change — interface errors/sec, octets/sec (never alert raw Counter totals)
- Baseline / anomaly — deviation from normal (needs history)
Counters must be rates. Absolute Counter32 “over 4 billion” is not an alert, it’s a wrap waiting to happen.
Sustained vs spike (hysteresis)
Single-sample spikes after backups are not incidents. Use sustained breaches (N consecutive polls or M minutes).
Hysteresis: separate trigger and clear levels so the alert doesn’t flap.
Numbered pattern:
- Trigger critical at 95% used for 10 minutes
- Clear only when back below 85% for 10 minutes
- Warn band in between if you want noise-controlled heads-up
- Document the numbers in the runbook
Poll-based vs trap-based alerts
| Style | Good for | Weakness |
|---|---|---|
| Poll + threshold | CPU, disk, load, bandwidth rates | Latency up to poll interval |
| Trap / inform | linkDown, hardware events | Can be lost; needs receiver |
Most estates use both. Details: traps vs polling.
Severity & escalation
Example mapping (tune per service):
| Metric | Warning | Critical |
|---|---|---|
| Disk used | ≥ 80% sustained | ≥ 95% sustained |
| CPU avg | ≥ 85% sustained | ≥ 95% + load high |
| Agent timeout | 2 failed polls | 5 failed + ICMP down |
Escalate critical to on-call; warn to ticket/Slack. Don’t page humans for warns all night.
Reducing alert fatigue
- Dedup — one incident per host/metric storm
- Group — related disks/interfaces together
- Maintenance windows — silence during patch
- Delete alerts nobody actioned in 90 days
- Fix chronic warns (capacity or bad threshold)
Mute culture is a symptom. Treat it.
Example: a disk-space alert
snmpget -v2c -c <RO_string> <host> 1.3.6.1.2.1.25.2.3.1.5.31 1.3.6.1.2.1.25.2.3.1.6.31 1.3.6.1.2.1.25.2.3.1.4.31
HOST-RESOURCES-MIB::hrStorageSize.31 = INTEGER: 26214400
HOST-RESOURCES-MIB::hrStorageUsed.31 = INTEGER: 24000000
HOST-RESOURCES-MIB::hrStorageAllocationUnits.31 = INTEGER: 4096 Bytes
Compute used % (used/size×100). If ≥ 95% for 3 polls → critical page with mount name from descr. Full how-to: disk space.
External alerting independence
When the host is the outage, you still need a path that pages. ostr.io can alert from external SNMPv2c checks (double durability: frequent + infrequent) and deliver email and SMS so notifications aren’t trapped on a dead box. Setup: ostr.io, external.
One product CTA; principles above apply to any NMS.
Runbook quality
Every critical alert needs: what failed, which OID/host, what to check first, when to escalate. “Disk warning” without mount is incomplete.
Common anti-patterns
- Alert on raw counters
- No hysteresis
- Same threshold for jump host and DB
- Page on every trap type enabled by default
- No maintenance windows
Glossary: glossary.
Writing the alert title first
Before thresholds, draft the page text:
[CRIT] {host} /var free {free_pct}% ({free_gb} GB) for {duration} — snmp hrStorage
If you can’t fill those fields from the check, the check is unfinished. On-call time is more expensive than another hour of NMS macros.
Testing without production pain
- Lower thresholds in staging until they fire.
- Confirm clear/hysteresis behavior.
- Restore production thresholds.
- Use maintenance windows for real fill tests on lab volumes.
Never test critical disk alerts by filling prod. Yes, people still try.
Ownership and review
Every alert rule needs an owner team. Quarterly: delete orphans, retune chronic warns, confirm runbooks still match OIDs after OS upgrades. Unowned alerts become noise; owned alerts become product quality.
Multi-condition examples worth copying
- CPU high and load/cores high → page
- CPU high alone on batch host during window → ticket only
- SNMP fail and ICMP fail → host/network
- SNMP fail and ICMP ok → agent/creds
Boolean logic beats a dozen independent pages for the same incident.
Trap storms
After power events, trap receivers melt. Rate-limit, buffer, and suppress flapping link traps on known bad ports. Polling thresholds still catch sustained damage when traps were dropped—another reason to keep both paths.
Severity inflation
If everything is critical, nothing is. Reserve pages for customer impact or data-loss risk. Capacity warnings become tickets with due dates. Informational traps go to a log stream. Review page/ack ratios monthly: if people ack without action, the rule is wrong or the ownership is.
Connecting frequency to alerts
Alert duration must be longer than one poll interval or you page on single bad samples. If polls are 5 minutes and you want 15 minutes of evidence, require 3 samples. Write that as for: 15m equivalent in whatever NMS syntax you use. See frequency.
Example CPU rule (shape, not gospel)
- Input: average hrProcessorLoad
- Warn ≥ 85% for 15m
- Crit ≥ 95% for 10m and load/cores ≥ 1.2
- Clear < 75% for 15m
- Silence during known batch window
Copy the shape; retune numbers per service class (CPU how-to).
After the page
The alert system’s job ends when a human has context. ChatOps that posts graph permalinks and last snmpget values beats “host bad, plz fix.” Invest in enrichment once; every future incident is cheaper.
Trap vs threshold cheatsheet
- linkDown trap → immediate network incident candidate
- disk 95% poll → capacity incident with lead time
- coldStart trap → correlate with uptime OID
- authFailure trap → security pipeline, not NOC CPU channel
Routing the wrong class of alert to the wrong team is fatigue by misdirection. Keep network traps on network; capacity polls on platform; security traps on security.
Closing the loop
After major incidents, ask: did an SNMP alert fire? Early enough? Actionable? If no, fix the rule or admit the gap (e.g. needed synthetics). Continuous improvement beats a static threshold wiki from 2019.
Also re-read disk, CPU, and availability when tuning—those pages own the metric semantics; this page owns how hard you page on them.
Frequently asked questions
How do I set SNMP alert thresholds?
Pick metric type (level vs rate), set warn/crit with duration, add hysteresis, test with real values from snmpget.
What is hysteresis in alerting?
Separate trigger and clear thresholds (and/or times) so alerts don’t flap around a single line.
Trap or poll for alerts?
Both. Polls for metrics; traps for discrete events.
How do I reduce alert fatigue?
Dedup, windows, delete useless alerts, fix chronic capacity, stop paging on warns.
Key takeaways
- Rates for counters; duration for levels.
- Hysteresis kills flaps.
- Traps + polls.
- Severity mapped deliberately.
- Off-box pages: ostr.io email/SMS pattern when you need independence.