Machine Learning: Avoiding Data-Driven Disasters

February 23, 2018

Machine Learning_part2

Machine learning is a powerful tool with the potential to damage your business if not used correctly. Instances of AI models producing unforeseen and sometimes disastrous effects regularly make the headlines – even Uber came under fire when its algorithms overcharged people fleeing hurricanes. To capitalize on the advantages of machine learning, companies need to know how they can avert their own data-driven disasters. Read the second part of this two-part series on machine learning.

Mistakes in configuring machine-learning models will generate multiple unexpected and undesired effects. If an invalid target value is used to optimize prices, it may boost profits in the short term but is likely to result in a significant decline in customer loyalty as time goes on.  Another risk is that the model will fit too closely to the dataset that was used to train it, causing major flaws in its predictions when the system is applied to further data. This is known as “overfitting”. Conversely, not receiving sufficient training is also a problem for the model, since this lays unstable foundations for its predictions. The implementation team needs to take care to eliminate any inaccuracies from the configuration and train the model to meet the exact aims of the business goals.

Low-quality data leads to faulty conclusions
Machine-learning systems are guaranteed to fail without a good source of clean data. There are several ways a dataset can be sub-optimal, for example, if the model is trained using historical data, the machine may simply repeat mistakes made in the past. The dataset may also lack certain features the machine needs to reach its conclusion. In this case, additional input will have to be found elsewhere. All data issues must be resolved before the machine-learning project can begin and, ideally, both internal and external data sources will be harmonized to maximize the reliability of the recommendations.

Out-of-sync models generate untrustworthy recommendations
When the two parts that make up machine-learning systems aren’t synchronized, serious errors can emerge. Ideally, a machine-learning system consists of a predictive model (to predict a variable based on the learned model) and a decision engine (to provide recommendations based on the prediction and a set of decision rules). Both components need to be well aligned, as without proper synchronization, decisions will be made based on incorrect predictions or the decision model won’t reflect rules of the business. To avoid a potentially disastrous lack of coordination, it is essential to have competent data scientists on the team working closely with business users to design the machine’s set of rules. The involvement of business users in the design process ensures predictions can be successfully translated into practical recommendations.

Poorly chosen projects restrict profit potential
Managers usually recognize the importance of data and appreciate insight-driven decisions. However, they are often unsure about how to use the data at hand. It is even harder to quantify how much value investments in data-driven projects might generate. Due this uncertainty, managers invest in projects that have limited impact on their bottom lines or in areas where machine learning makes no sense at all. To properly quantify the potential and feasibility of data-driven projects, implementation teams need to define suitable measurement criteria and conduct comprehensive analyses, incorporating relevant benchmarks and past project experience into their assessment. Managers will then be able to pinpoint exactly where to focus their efforts to maximize the impact of their investments.

Set up monitoring and risk-mitigation systems before implementation starts
Before implementing your new pricing system, evaluate whether machine learning actually makes sense for your company and identify the precise areas that will deliver the most value. Careful planning will help ensure your new automated pricing approach won’t harm your business. Follow these four steps to avoid the major pitfalls in implementation before going live with your new machine-learning pricing system:

  • Identify and improve your data sources and rethink your approach to data management
  • Use business knowledge to define pricing KPIs that cover more than just profit and set up an effective monitoring system
  • Implement a traffic-light system with escalation rules that signal whether to recalibrate the machine-learning model or carry out a human “overwrite”
  • Define risk mitigation strategies in case machine-learning pricing decisions produce negative effects on a large scale

    Machine Learning: Cutting-edge Tech Makes the Difference - read part 1 of the series here