Are any of these landmines hiding in your real-world ML initiative?

87% of ML projects fail today!

These numbers should be taken with a grain of salt. Irrespective of the actual number, it does reflect reality — I have seen a significant percentage of ML-based projects never get into production!

GIF via giphy
Experiences divided into 6 phases of any ML project. Depending on your role, feel free to read the respective sections in this blog (Image by author)

This is a…


These mistakes are easy to overlook but costly to redeem

two stacks of colored coffee mugs with the names of different cities on them
two stacks of colored coffee mugs with the names of different cities on them
Photo by Frank Vessia on Unsplash

ML model training is the most time-consuming and resource-expensive part of the overall model-building journey. Training by definition is iterative, but somewhere during the iterations, mistakes seep into the mix. In this article, I share the ten deadly sins during ML model training — these are the most common as well as the easiest to overlook.

Ten Deadly Sins of ML Model Training

1. Blindly increasing the number of epochs when the model is not converging

During model training, there are scenarios when the loss-epoch graph keeps bouncing around and does not seem to converge irrespective of the number of epochs. There is no silver bullet as there are multiple root causes to investigate — bad training examples, missing truths…


Understanding under-the-hood details of modern NoSQL systems

What distinguishes a good data engineer from a great one? Having an understanding of distributed systems concepts can help in making the right Big Data technology choices as well as write better data apps and pipelines.

To illustrate, imagine buying a car. You will typically have a few different car models and then compare the price/performance i.e., engine, transmission, braking, etc. …


Blog series on DataOps for effective AI/ML

GIF by giphy

Let’s start with a real-world example from one of my past ML projects: We were building a customer churn model. “We urgently need an additional feature related to sentiment analysis of the customer support calls.” Creating the data pipeline to extract this dataset took about 4 months! Preparing, building, and scaling the Spark MLlib code took about 1.5-2 months! Later we realized that “an additional feature related to the time spent by the customer in accomplishing certain tasks in our app would further improve the model accuracy” — another 5 months gone in the data pipeline! …


Weekly curated articles on applying AI without the hype

Why read this newsletter?

The Four Waves of AI — Understanding the AI task you are trying to solve as a combination of four capabilities: Perceiving, Learning, Abstracting, Reasoning

AI in Action

This week we are focussing on articles related to : Customer service management, Next product to Buy, Churn reduction, Pricing, and…


In addition to skills, IQ, EQ, look for Data IQ (DQ)

GIF by giphy

With the rapid technology evolution, the roles of data engineers, data analysts, data scientists, ML engineers, AI product developers, will be significantly different in the next to 3-5 years!

How do you future proof your Data + AI/ML team hires? Will your current team be able to learn the skills for technology that does not exist today?

Consider the role of a data engineer. Two decades ago, the role was focussed around data warehouse schema designs, schema-on-write governance, generating weekly business reports. This is the time when data was considered part of the IT expense.

Over the…


Weekly curated articles on applying AI without the hype

Source

AI in Action

  • Using re-enforcement learning is answer the question: Which of the many ads is more likely to appeal to a certain viewer? Finding the right…


GIF Source

AI in Action

  • AI robo-advisors in Finance: AI is helping human financial advisors by relieving them from having to perform mundane portfolio monitoring and administrative tasks that currently take…


21 Playbooks to help navigate the common leadership scenarios

Whether you are a software engineer looking to grow into a Tech Lead/ Manager role or a data engineer/analyst/scientist aspiring to move into a data leadership role, there are a common set of behaviors that are required.

GIF via giphy

Google’s Project Oxygen uncovered 10 must-have behaviors that effective managers must exhibit. As an Individual Contributor (IC), given your super hectic work schedule, finding time to develop these skills can take weeks or months!

Source: https://rework.withgoogle.com/guides/managers-identify-what-makes-a-great-manager/steps/learn-about-googles-manager-research/

I have gone through the journey from an IC to VP of Engineering and Chief Data Officer. …

Sandeep Uttamchandani

Democratize Data + AI/ML — real-world battle scars to help w/ your journey. Product builder(Engg VP) & Data/ML leader (CDO). O’Reilly Author|DataForHumanity.org

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store