Deduplication: Events into Alerts

Once your workflows finish processing events, Moogsoft deduplicates the events into alerts. When it adds a new event to an alert, Moogsoft updates the alert with the latest information from the event. Thus you can think of events as hard-coded snapshots of an issue, while alerts get updated with each new event.

This topic describes how Moogsoft removes duplicate events from the data ingestion stream.

Deduplication and noise reduction

A busy service with multiple monitors can generate a flood of metrics, anomalies, and events. One issue might trigger a large number of repeat and duplicate events. Moogsoft analyzes every new piece of data — What is this? When did it happen? What is its severity? How often has it happened before? — and aggregates events for the same issue into alerts. Whenever it adds a new event, Moogsoft updates the alert fields — event count, last event time, severity — so the alert always contains the latest information about the underlying issue. This process removes the duplicate, repeat, and obsolete noise from the data stream.

How deduplication works

Moogsoft converts each new event notification and metric anomaly into a generic JSON object that describes a specific performance-related event: what happened, when it happened, where it happened, and so on.

Moogsoft then adds each new event to an alert. An event describes one specific event; an alert describes a set of events that all relate to the same issue. For example, an alert might be "High CPU load on server 23." and consist of the following events:

  1. server 23, 12:00: CPU load = 72%

  2. server 23, 12:01: CPU load = 80%

  3. server 23, 12:02: CPU load = 67%

Moogsoft adds new events to alerts as follows:

  1. A new JSON event object arrives at the ingestion endpoint.

  2. Moogsoft generates a dedupe key for the event based on the source, service, and check fields in the event.

  3. Moogsoft compares the new event with each open alert using the dedupe key.

    • If the dedupe key matches an open alert, Moogsoft increments the alert's event count field and updates the severity field based on the new event.

    • If the dedupe key does not match any open alert, Moogsoft creates a new alert and adds the event to it.

See also Event deduplication: how-to and best practices