Alert Rules Guide

Alerts Overview

Each model has associated metrics, on which alert rules can be created and retrieved. By default, certain metrics are provided, but you may create custom metrics for a model for creating custom alerts.

The dedicated alerts endpoint is /models/{model_id}/alert_rules.

Create Alerts

  • metric_id - UUID of the metric to use to create an alert rule.

  • name - A name for the alert rule, a default will be generated if this is not supplied.

  • bound - Either "upper" or "lower"

  • threshold - Threshold of alert rule

  • severity - AlertRuleSeverity of the alert which gets triggered when the metric violates the threshold of the alert rule.

  • lookback_period - For streaming models only. The lookback time or “window length” in minutes to use when calculating the alert rule metric. For example, a lookback period of 5 minutes for an alert rule on average prediction will calculate average prediction for the past 5 minutes in a rolling window format. This will default to 5 minutes.

  • subsequent_alert_wait_time - For streaming models only. This is the time in minutes to wait after the most recent triggered alert before checking if the same alert is triggered. This defaults to 0.

Query Request:

{
    "metric_id": "<metric_id> [string]",
    "bound": "[upper|lower]",
    "threshold": "<threshold_value> [int|float]",
    "severity": "[warning|critical]",
    "name [Optional]": "<alert_name> [string]",
    "lookback_period [Optional]": "<lookback_period> [int|float]",
    "subsequent_alert_wait_time [Optional]": "<lookback_period> [int|float]"
}

If using the SDK, you may call the create_alert_rule method from your Warrior model.

back to top

Get Alerts

Getting alerts returns a paginated list of rules for the model, and the GET endpoint includes query parameters, as /models/{model_id}/alert_rules?page_size={page_size}&page={page}

If using the SDK, you may call the get_alert_rules method from your Warrior model.

back to top

Update Alerts

With the alert ID handy, you can send a PATCH request to /models/{model_id}/alert_rules/{alert_rule_id}.

  • bound - Either "upper" or "lower"

  • threshold - Threshold of alert rule

  • severity - AlertRuleSeverity of the alert which gets triggered when the metric violates the threshold of the alert rule.

  • lookback_period - The lookback time or “window length” in minutes to use when calculating the alert rule metric. For example, a lookback period of 5 minutes for an alert rule on average prediction will calculate average prediction for the past 5 minutes in a rolling window format. This will default to 5 minutes

  • subsequent_alert_wait_time - If metric continues to pass threshold this is the time in minutes to wait before triggering another alert. This defaults to 0. This does not need to be set for batch alerts.

  • enabled - Whether or not an alert rule is active. Defaults to true.

Query Request:

{
    "bound [Optional]": "[upper|lower]",
    "threshold [Optional]": "<threshold_value> [int|float]",
    "severity [Optional]": "[warning|critical]",
    "name [Optional]": "<alert_name> [string]",
    "lookback_period [Optional]": "<lookback_period> [int|float]",
    "subsequent_alert_wait_time [Optional]": "<lookback_period> [int|float]",
    "enabled [Optional]": "<bool> [bool]"
}

back to top

Alert Notification Configuration

A specific alert notification configuration determines if and where an email notification should be sent when an alert on a specific model is triggered.

Create Alert Notification Configuration

To create an alert notification configuration, you can send a POST request to /alert_notification_configurations.

  • user_id - UUID of the user this alert notification configuration belongs to. Only specify if type is "email".

  • model_id - UUID of the model this alert notification configuration belongs to.

  • type - Type of notification to send, currently the only supported values are "email", "PagerDuty", and "ServiceNow".

  • destination - The email address to send the notification to.

  • If type is "email", destination must match the email address associated with the above specified user id’s Warrior account.

  • If type is "PagerDuty", destination should be the PagerDuty provided integration email address. Please see the PagerDuty Email Integration Guide for details on where this can be found in the PagerDuty UI.

  • If type is "ServiceNow", destination should be the email address configured for integration with your ServiceNow instance. Please see the ServiceNow Email Actions Guide for details on how this can be configured in ServiceNow.

  • enabled - Whether or not the notification configuration is enabled. Defaults to true.

Query Request:

{
    "user_id" : "<user_id> [string]" ,
    "model_id" : "<model_id> [string]",
    "type" : "[email]",
    "destination" : "<email> [string]",
    "enabled" : "<bool> [bool]"
}

Users can create configurations for themselves or must have at least a role of “Administrator” to create configurations that are not associated with their user id.

back to top

Get Alert Notification Configuration

The GET endpoint returns a paginated list of alert notification configurations and includes query parameters, as /alert_notification_configurations?model_id={model_id}&user_id{user_id}&page_size={page_size}&page={page}

With a specific configuration ID handy, you can also send a GET request to /alert_notification_configurations/{configuration_id}.

back to top

Update Alert Notification Configuration

With the specific configuration ID handy, you can send a PATCH request to /alert_notification_configurations/{configuration_id}.

The fields supported for update are:

  • enabled - Whether or not the notification configuration is enabled. If enabled, the destination address will be auto-notified when an alert is triggered.

Query Request:

{
    "enabled" : "<bool> [bool]"
}

back to top

Metrics Overview

To create a custom alert, you must first create a custom metric associated with your Warrior model.

The dedicated metrics endpoint is /models/{model_id}/metrics.

  1. The query should perform some sort of aggregate function. See the API Query Guide for descriptions and query examples.

  2. The query should only return a single value. For example, the query should not return time-series data or scores for multiple attributes.

  3. The query should not include timestamp filters (for streaming models) or batch_id filters (for batch models). We will dynamically add these filters to your query.

Create Metrics

When creating a custom alert metric, we will do some validation on the backend to try to prevent invalid queries, but we also suggest hitting our query endpoint (/models/{model_id}/inferences/query) or data drift endpoint (/models/{model_id}/inferences/data_drift) before saving your new metric to test your query.

  • name - Name of the metric to create.

  • query - Query which makes up the metric.

  • endpoint - Endpoint determining class of metric. If not data drift query, then /models/{model_id}/inference/query, else /models/{model_id}/inference/data_drift.

Query Request:

{
    "name": "<metric_name> [string]",
    "query": "<query> [dict]",
    "endpoint": "<endpoint> [string]"
}

If using the SDK, you may call the create_metric method from your Warrior model.

back to top

Get Metrics

If you have the metric ID handy, you can send a GET request to /models/{model_id}/metrics/{metric_id}.

Otherwise, you can send a GET request to /models/{model_id}/metrics/ with certain filters.

  • expand - If set to "type" will return the type with each metric.

  • default - If set to True will return only metrics that are automatically created by default for your model.

  • type - MetricType to filter metric query with.

  • metric_name - Metric name filter to use in metric search.

  • attribute_name - Attribute name filter to use in metric search.

Query Request:

{
    "expand [Optional]": "type",
    "default [Optional]": "<bool> [bool]",
    "type [Optional]": "[model_output_metric|model_input_data_metric|model_performance_metric|model_data_drift_metric]",
    "metric_name [Optional]": "<metric_name> [string]",
    "attribute_name [Optional]": "<attribute> [string]"
}

If using the SDK, you may call the get_metrics method from your Warrior model.

back to top

Default Metrics and Alerts

The following metrics are created by default for every model, on a per feature basis where applicable.

Default data drift alerts for PSI are automatically created for every feature once there is reference and inference data sent to Warrior. These alerts are created with dynamic threshold values specific to your reference dataset from the data_drift endpoint with "metric": "Thresholds".

Aggregation Functions

  • avg

  • count

  • min

  • max

Model Evaluation Functions

  • Regression

    • rmse

    • mae

    • rSquared

  • Classification

    • accuracyRate

    • balancedAccuracyRate

    • f1

    • falseNegativeRate

    • falsePostiveRate

    • precision

    • recall

    • sensitivity

    • specificity

    • truePositiveRate

    • trueNegativeRate

  • Multiclass Classification

    • accuracyRateMulticlass

    • falseNegativeRateMulticlass

    • falsePositiveRateMulticlass

    • truePositiveRateMulticlass

    • trueNegativeRateMulticlass

Data Drift Metrics

  • PSI

  • KLDivergence

  • JSDivergence

  • HellingerDistance

  • HypothesisTest

  • NLPDataDrift (if NLP model)

Metric Examples

The following metric examples give instances of how to customize the vanilla default metrics.

Average Prediction

Average prediction for class_a for inferences whose city is equal to “Boston”.

{
    "name": "avg pred for Boston for class_a less.50",
    "query": {
        "select": [
            {
                "function": "avg",
                "alias": "avg",
                "parameters": {
                    "property": "class_a"
                }
            }
        ],
        "filter": [
            {
                "property": "city",
                "comparator": "eq",
                "value": "Boston"
            }
        ]
    },
    "endpoint": "/models/{model_id}/inference/query"
}

back to top

Inference Count

Inference count is less than 1 where the city equals “Boston”.

{
    "name": "Boston Inference Count",
    "query": {
        "select": [
            {
                "function": "count",
                "alias": "count"
            }
        ],
        "filter": [
            {
                "property": "city",
                "comparator": "eq",
                "value": "Boston"
            }
        ]
    },
    "endpoint": "/models/{model_id}/inference/query"
}

back to top

RMSE

RMSE for class_a when attribute temperature is greater than or equal to 32.

{
    "name": "RMSE when temp > 32",
    "query": {
        "select": [
            {
                "function": "rmse",
                "alias": "rmse",
                "parameters": {
                    "ground_truth_property": "class_a_ground_truth",
                    "predicted_property": "class_a"
                }
            }
        ],
        "filter": [
            {
                "property": "temperature",
                "comparator": "gte",
                "value": 32
            }
        ]
    },
    "endpoint": "/models/{model_id}/inference/query"
}

back to top

Data Drift

PSI data drift for temperature in the city of Boston. Note that the endpoint changes in the request body!

{
    "name": "PSI Data Drift Reference Set in Boston",
    "query": {
        "properties": [
            "temperature"
        ],
        "num_bins": 20,
        "base": {
            "source": "reference"
        },
        "target": {
            "source": "inference",
            "filter": [
                {
                    "property": "city",
                    "comparator": "eq",
                    "value": "Boston"
                }
            ]
        }
    },
    "endpoint": "/models/{model_id}/inference/data_drift"
}

back to top

Confusion Matrix Variants

True Positive Rate for class_a when the age is between 30 and 50.

{
    "name":"True Positive Rate for 30-50 yr olds",
    "query":{
        "select":[
            {
                "function":"truePositiveRate",
                "parameters":{
                    "threshold":0.5,
                    "ground_truth_property":"class_a",
                    "predicted_property":"ground_truth_a"
                }
            }
        ],
        "filter":[
            {
                "property":"AGE",
                "comparator":"gte",
                "value":30
            },
            {
                "property":"AGE",
                "comparator":"lt",
                "value":50
            }
        ]
    },
    "endpoint": "/models/{model_id}/inference/query"
}

back to top

Accuracy Rate

Accuracy rate for class_a when the age is between 30 and 50.

{
    "name":"Accuracy Rate for 30-50 yr olds",
    "query":{
        "select":[
            {
                "function":"accuracyRate",
                "parameters":{
                    "threshold":0.5,
                    "ground_truth_property":"ground_truth_a",
                    "predicted_property":"class_a"
                }
            }
        ],
        "filter":[
            {
                "property":"AGE",
                "comparator":"gte",
                "value":30
            },
            {
                "property":"AGE",
                "comparator":"lt",
                "value":50
            }
        ]
    },
    "endpoint": "/models/{model_id}/inference/query"
}

back to top