JavaScript is not available.

Jul 19, 2022

🔥 A new blog! An introductory overview of the different facets of ML system monitoring. When one mentions “ML monitoring,” this can mean many things. Are you tracking service latency? Model accuracy? Data quality? We made a comprehensive overview:

Monitoring ML systems in production. Which metrics should you track?

When one mentions "ML monitoring," this can mean many things. Are you tracking service latency? Model accuracy? Data quality? This blog organizes everything one can look at in a single framework.

Feb 4

📌In case you missed it Which metrics to track when monitoring ML systems in production? We suggested a framework that consists of 4 components: 🏥 Software health ✅ Data quality 📈 ML model quality 💰 Business KPI More on ML monitoring facets 👇

Monitoring ML systems in production. Which metrics should you track?

When one mentions "ML monitoring," this can mean many things. Are you tracking service latency? Model accuracy? Data quality? This blog organizes everything one can look at in a single framework.

Feb 3

Great to see so many folk—old friends and new—at #PyData #Edinburgh last night. Thanks to Emeli Dral from

for talking on Data Drift. Slides available at github.com/pydataedinburg

Machine Learning for Snapchat Ad Ranking

Feb 3

A Friday ML use case📕 How Snapchat aims to deliver the right ad to the right user at the right time: an overview of its ad ranking system and end-to-end ML development 👇

eng.snap.com

Data Drift Monitoring with Evidently - Fuzzy Labs

Sam Rhynas

@lilacpatsy

Feb 2

And we’re off! First #Pydata #Edinburgh in three years! Welcome back everyone, and big thanks to wood Mackenzie and

🙈 Is it okay if I pretend data drift doesn't exist? A great overview of why one should care about data drift and how to monitor it in production using open-source MLOps tools 📈 Thanks, Oscar Wong and

@FuzzyLabsAI

, for mentioning Evidently!

fuzzylabs.ai

Feb 1

Language models are changing AI. Yet we lack evaluation standards for them. The article suggests a holistic approach: - Evaluating over a large # of scenarios - Multi-metric measurement - Standardised evaluation Findings from evaluating 30 major LMs: crfm.stanford.edu/2022/11/17/hel

Emeli Dral

@EmeliDral

Feb 1

📚How can you evaluate data drift on text data? We’ve just released the latest functionality in the

open-source library that does just that. Here are the two approaches that we implemented. 👇🏼 (1/4)

“Can I use Evidently to monitor text data?” Yes, you can! With the latest release, you can use Evidently to monitor and evaluate text data quality and drift for your NLP models in production 🔡 Read the release blog 👇 evidentlyai.com/blog/evidently

GIF

Elena Samuylova

@elenasamuylova

Jan 30

🚀Super happy to announce the latest

release: text data support! It has been one of the most requested features: “How can I use Evidently to monitor and evaluate text data drift?” Now, you can do that! Our open-source library now supports raw text inputs. (1/3)

Show this thread

Data Drift: Uncovering Changes in Your ML Model's Input Data, Thu, Feb 2, 2023, 6:15 PM | Meetup

Jan 30

In case you missed it:

@EmeliDral

will be talking about Data Drift at PyData Edinburgh this Thursday eveing, live in Edinburgh. Free signup at: meetup.com/pydata-edinbur Plus lightning talks. Beer. Pizza. Craic.

@WoodMackenzie

@CathcartRecruit

#python #data #edinburgh #ml #ai

meetup.com

Welcome back to PyData Edinburgh! We're delighted to say we will be starting in person events again on the anniversary of our 4th birthday, at one of traditional venues, W

Monitoring Machine Learning APIs

Matthew Lungren MD MPH

@mattlungrenMD

Jan 29

Replying to

@suchisaria

Such an important area - post deployment model monitoring in healthcare - still under-explored especially for the complex medical imaging AI domain...(but we are working hard on it!😅) In general am a big fan of the work by

in this space!

Breaking down ML APIs monitoring 👇

@duarteocarmo

suggests three main areas: ❌ Logs & Error monitoring 📉 Drift/Degradation monitoring 🏢 Business monitoring/reporting More about implementing each monitoring area:

duarteocarmo.com

Opening some doors to the machine learning black blox. An article about monitoring Machine Learning APIs

Jan 28

📌In case you missed it How can you analyze past drifts in the data you train the model on? A tutorial to detect, evaluate, and visualize historical data drift 👇

How to detect, evaluate and visualize historical drifts in the data

You can look at historical drift in data to understand how your data changes and choose the monitoring thresholds. Here is an example with Evidently, Plotly, Mlflow, and some Python code.

Jan 27

✅ / ❌ DOs and DON'Ts of shipping a model to production for the first time One of our favorite questions from yesterday's Ask-Me-Anything on scaling ML teams and MLOps with

! Hop on our Discord for more discussions with Neal 👇 discord.com/invite/g8QEjzx

How DoorDash Upgraded a Heuristic with ML to Save Thousands of Canceled Orders

Jan 27

A Friday ML use case📕 How Doordash predicts the operational status of a store: they started from a simple heuristic and upgraded to an ML model with a dynamic loss function 👇

doordash.engineering

Learn how DoorDash engineering utilized ML models to o accurately track Merchants' operational status and ability to full fill orders.

Jan 26

Reminder: We're back, in person. meetup.com/pydata-edinbur Join us to hear Emeli Dral, of

, discuss Data Drift: Uncovering Changes in Your ML Model's Input Data. Lightning talks & refereshments, too. At

@WoodMackenzie

Edinburgh. 18:30, Thurs 2 Feb.

@CathcartRecruit

Data drift. (Two normal distributions, with the same area, one shifted to the right about 1 standard deviation, with a lower peak.)

read image description

ALT

Jan 26

We're very much looking forward to the talk! There are still free places available, just remember to sign up through meetup link below...

Quote Tweet

Jan 24

Anyone in Edinburgh next week? Our CTO, @EmeliDral, will be talking about data drift at @PyDataEdinburgh on February 2nd. She'll discuss drift detection methods and how to incorporate it in the production ML workflow

Stop by if you're nearby

meetup.com/PyData-Edinbur

🚀 AMA with

from Monzo is starting in a few minutes! Jump to our Discord to ask anything about building ML platforms and high-performance ML teams, scaling MLOps, and more! 👇 discord.com/invite/g8QEjzx

Automatic retraining for machine learning models - Building Nubank

Jan 26

5️⃣ Take costs into account when deciding how often to retrain. Model retraining does not come for free 💵 Compare associated system and labor costs with the expected benefits of retraining a model. More tips by the Nubank team 👇 building.nubank.com.br/automatic-retr 6/6 🧵

building.nubank.com.br

In this post we'll go over some of the lessons we learned over the last years retraining models in an automatic fashion at Nubank.

4️⃣ Do unit-testing (sort of) for ML models. How can we guarantee that a newly retrained model is better for every single example scored? We cannot 🤷‍♀️ But we can run synthetic examples to verify that retrained model gets the expected outcome. 5/6

3️⃣ Use fair comparison benchmarks. After retraining a model, compare it against the current model in production using: - Feature importances - Out-of-sample performance - Stability of predictions, etc. Whatever the metric, calculate on an unbiased holdout dataset 👩‍🔬 4/6

2️⃣ Don’t forget about other teams using the model. Whenever you retrain a model, you may need to adjust how its predictions are used. One way to make adjustments easier is to use calibrated probability predictions ⚖️ 3/6

1️⃣ There are prerequisites for automatic retraining. It will be impossible without these 3 MLOps components: ✅ CI/CD pipeline for testing and deployment 📈 ML monitoring to check the current model state 🔢 Solid data pipeline to generate train/test datasets 2/6

Sooner or later, every ML model decays. The world changes, and its static picture (aka train set) becomes obsolete. Often the easiest way to update a model is to retrain it on newer data. Nubank retrains ML models automatically. Here are 5 tips by

: 1/6 🧵

In case you missed it, we have an AMA session with

tomorrow! Join us to chat about: 🔥 Building high-performance ML teams 🚀 Scaling MLOps ✅ Moving from one model to many 🛠 Monzo’s ML stack 📦 modelstore Register here 👇 lu.ma/k6en6l67

Jan 24

Anyone in Edinburgh next week? Our CTO,

@EmeliDral

, will be talking about data drift at

Data Drift: Uncovering Changes in Your ML Model's Input Data, Thu, Feb 2, 2023, 6:15 PM | Meetup

on February 2nd. She'll discuss drift detection methods and how to incorporate it in the production ML workflow 🌊 Stop by if you're nearby 👇

meetup.com

Welcome back to PyData Edinburgh! We're delighted to say we will be starting in person events again on the anniversary of our 4th birthday, at one of traditional venues, W

Jan 23

🦸‍♂️ Want to know how top engineering teams do ML? We selected 50 company tech blogs that write about applied ML. Get ready for a lot of reading! See the list 👇 evidentlyai.com/blog/engineeri

Jan 23

💬 We’re kicking off the 2023 AMA season! Our new guest is

! Neal is the Staff ML Engineer and former Associate Director of ML at Monzo. Join us to chat about building high-performance ML teams, scaling MLOps, and more! Register here👇 lu.ma/k6en6l67

Jan 21

📌In case you missed it: Want to know how popular drift detection methods perform on large datasets? 📈Add artificial drift 👩‍🔬Compare different tests 📊Visualize and explore Check out the blog to repeat the experiment on your data👇 evidentlyai.com/blog/data-drif

Jan 18

PyData Edinburgh, is finally back for an in-person event! Emeli Dral from Evidently AI will discuss data drift and how their open source tools can help. Read more and register at meetup.com/PyData-Edinbur Thanks

(refreshments) &

(venue).

Want to join the next AMA? 💬 Our new guest is

from Monzo. Join us to chat about building ML platforms and high-performance ML teams, scaling MLOps, and more! Join our Discord to ask Neal a question discord.com/invite/g8QEjzx Register here lu.ma/k6en6l67

The last (but not least) AMA in 2022 was with Ben Wilson from Databricks 🤖 "I don't care about the tools. You shouldn't either." Ben on approaching any ML project, AutoML, deploying models into the wild, and how to learn about ML deployment 👇 evidentlyai.com/blog/ama-ben-w 10/11

AMA with Ben Wilson: planning ML projects, AutoML, and ML deployment at scale

In this blog, we recap the Ask-Me-Anything session with Ben Wilson. We chatted about AutoML use cases, deploying ML models to production, and how one can learn about ML engineering.

With Rick Lamers from

@orchestofficial

, we discussed data orchestration 🎵 We chatted about how orchestration tools evolved, their place within the MLOps landscape, and the future of data pipelines. Read the full discussion 👇 evidentlyai.com/blog/ama-rick- 9/11

AMA with Rick Lamers: orchestration tools and data pipelines

In this blog, we recap the Ask-Me-Anything session with Rick Lamers, where we chatted about the evolution of orchestration tools, their place within the MLOps landscape, the future of data pipelines,...

We asked

about ML and MLOps at a reasonable scale 👩‍💻 "You do not need to be Big Tech to do meaningful ML work." Read the recap about reasonable scale ML, recsys, and MLOps anti-patterns 👇 evidentlyai.com/blog/ama-jacop 8/11

AMA with Jacopo Tagliabue: reasonable scale ML and testing RecSys

In this blog, we recap the Ask-Me-Anything session with Jacopo Tagliabue, where we chatted about ML at a reasonable scale, testing RecSys, MLOps anti-patterns, what’s hot in DataOps, fundamentals in...

With

and Sean Sheng from

, we tried to answer why ML deployment is hard 🤯 We discussed beginner deployment mistakes, how to avoid them, and the challenges of building an open-source product. More on the topic 👇 evidentlyai.com/blog/ama-bozha 7/11

AMA with Bo Yu and Sean Sheng: why ML deployment is hard

In this blog, we recap the Ask-Me-Anything session with Bozhao Yu and Sean Sheng, where we chatted about why deploying a model is hard, beginner mistakes and how to avoid them, the challenges of...

We spoke with

from

about engineering bottlenecks in data science 👾 "If anything, we have too many tools!" Read the recap about AutoML, ML tools, and how to simplify repetitive and mundane data engineering tasks 👇 evidentlyai.com/blog/ama-doris 6/11

AMA with Doris Xin: AutoML, modern data stack, and MLOps tools

In this blog, we recap Ask-Me-Anything session with Doris Xin, that covered the roles of Data Scientists and Data Engineers in an ML cycle, automation, MLOps tooling, bridging the gap between...

We sat with

from

to talk synthetic data 🔢 We covered beginner mistakes in data generation, the data-centric approach, and how well companies are doing in getting there. More on the topic 👇 evidentlyai.com/blog/ama-fabia 5/11

AMA with Fabiana Clemente: synthetic data, data-centric approach, and rookie mistakes to avoid

We recap Ask-Me-Anything session with Fabiana Clemente, which covered synthetic data, its quality, beginner mistakes in data generation, the data-centric approach, and how well companies are doing in...

We chatted with Matt Squire from

@FuzzyLabsAI

on MLOps pain points. ❌ Here are 3 of those: - Tools are heavy-weight and laborious to run - Monitoring is hard - It's no good in great tools if nobody can use them More discussions in recap blog👇 evidentlyai.com/blog/ama-matt- 4/11

AMA with Matt Squire: what makes a good open-source tool great, and the future of MLOps

In this blog we recap Ask-Me-Anything session with Matt Squire, that covered MLOps maturity and future, how MLOps fits in data-centric AI, and why open-source wins.

We spoke with

from

about the next big thing in MLOps 👩‍🚀 "The MLOps stack will come from the open-source movement." More about tools, the future of real-time ML, and building an open-source startup 👇 evidentlyai.com/blog/ama-hamza 3/11