Here are a few things that any organization deploying NLP models into production should be doing to ensure that those models continue to perform as expected.

Ensure consistency in data extraction pipelines and monitor for data drift.

Interact with feature scores and observing the impact on model outputs.

Identify the most important features impacting predictions of your NLP models.

Monitoring

Compare the similarity of new input documents to the documents used to train your NLP models

Bias Analytics

Detect biases in your NLP models by uncovering differences in accuracy and other performance metrics across different subgroups

Explainability

Identify the specific words within a document that contributed the most to a given prediction

Thanks to Warrior, we know that our preventative care models are fair and that we can catch any potential issues before they impact our members.

– Heather Carroll Cox, Chief Data & Analytics Officer at Humana