Rules Based Engine vs Machine Learning Monitoring Systems

Rules-based engine vs machine learning systems for risk, fraud, and operations monitoring and decision making.

As a company matures, a number of opportunities for both rules-based and machine learning based monitoring systems can present themselves. For financial technology and platform companies, risk and fraud monitoring will often be a focal point for the application of these systems. Both these systems have its strengths and weaknesses and many companies will find that starting out with a rules-based engine and layering on machine learning as a complementary system as user behavior data size increases over time is the way to go. Let's dive into the pros and cons of each.

Rules-Based Systems

Rules based systems can be very effective in addressing fraudulent behavior. Companies have been employing such rules based systems for decades. Some examples of rules-based alerts:

  • Alert on any transaction that is very large e.g. over $10,000
  • Alert on any user whose transaction velocity is unusually high e.g. more than four transactions in the last hour
  • Alert on any transaction that has occurred within a list of suspicious zip code areas

These alerts can be quite good at identifying and mitigating fraudulent behavior while not raising false alarms for good users, as long as the behavior being monitored is highly correlated with what actually results in fraud.

Pros of rules-based systems

  • Rules-based systems tend to be more easily understood by human beings. If a user is confused by why a certain transaction of theirs was blocked, it's easier for a customer support associate to explain why a certain decision was made under a rules-based system.
  • Rules-based systems can take advantage of knowledge shared by risk and fraud experts over years of experience and capture industry best practices.
  • In certain cases, rules-based systems can help you achieve 100% accuracy. For example, if your company follows a regulation to report all transactions > $10,000, a rules-based system can help you achieve that goal with 100% accuracy, whereas a machine learning system can run the risk of mis-classifying data.
  • Rules can be iterated upon and improved over time. If a rule is fairly effective but can use some tweaking in a few specific areas, a rules-based system will give you the flexibility to express where tweaks need to be made.
  • A rules-based system is also easier to enact from the start. Machine learning systems typically require many training data points in order to bootstrap a decisioning system whereas rules-based systems can be enacted from day 1.
  • Finally, rules-based systems tend to be cheaper and faster to stand up as they don't require complex data pipelines and expensive machine learning engineers to implement.

Cons of rules-based systems

  • Rules and thresholds need to be manually tuned over time.
  • Rules-based systems are often reactive and cannot capture certain complex, interdependent, hidden behavioral patterns present in large systems.
  • Fraudsters may attempt to reverse engineer rules.
  • If the fraud analysts enacting the rules are biased in a certain way, outcomes may end up reflecting that bias

Rules-based systems are generally quicker and easier to standup. If your user data is stored in a database like Postgres or MySQL, or warehouse like Redshift, Snowflake, or BigQuery, or captured by APIs like Mixpanel or Segment, you can hook up your data to a tool like LogicLoop and your analysts can start writing SQL-based rules in minutes. Rules-based alerts are often expressed via a combination of conditional logic e.g. "if-and-or-else", thresholds e.g. ">= X", and aggregates e.g "count, sum, max." Complex iterations of these types of rules may be difficult to express with simple drag-and-drop filters, but can be easily captured using a language like SQL with LogicLoop. Moreover, SQL comes with a powerful toolkit of functions that enable you to express even more sophisticated logic.

Machine Learning Systems

In the recent years, machine learning systems have grown in popularity to complement rules-based systems. For big complex systems with large volumes of data, machine learning algorithms can be a powerful tool to find non-liner, non-intuitive patterns. In order to train a ML model, you need to first have large volumes of past data whose outcomes have already been determined. The goal of the ML model is often anomaly detection (discover / alerts on abnormal behavior) or classification (determining which data points should go into one category vs another e.g. fraudsters vs good users or spam vs not-spam). Some popular machine learning algorithms include:

  • Logistic regression
  • Decision trees
  • Random forests
  • Neural networks

Pros of machine learning systems

  • ML systems can pick up on complex features and multi-degree causal relationships that may not be discoverable to a human writing rules-based alerts.
  • ML systems can automatically tune thresholds and improve accuracy over time. There's less manual work required.
  • ML models can scale to massive volumes of data.
  • Sophisticated fraudsters who are aware of the most commonly enacted rules-based alerts may behave in complex ways that allow them to dodge rules-based checks. However, since each ML model is different, a fraudster cannot reverse engineer a ML based detection model.

While machine learning models are very powerful, they are not a panacea for everything and they have their limitations as well.

Cons of machine learning systems

  • Since fraudulent behavior typically only represents a small percentage of the total user behavior on a platform, datasets need to be extremely large to provide enough signal to the model. Some machine learning algorithms do not perform well when the ratio of positive to negative outcomes is too imbalanced.
  • ML models are hard to explain and it's sometimes difficult to understand why one user was flagged as suspicious whereas another was not. ML models can also feel like a black box, which makes it difficult to understand whether it's working or not. If you're outsourcing your machine learning to a third party vendor, analyze their offerings carefully as transparency into to what's actually effective is not easy to come by.
  • Combating fraud is a cat-and-mouse game. Fraudsters are constantly changing their behaviors. As fraudulent behavior changes, models also need to be retrained. Complex machine learning models can take hours or even days to retrain whereas rules can be changed in as little as 15 minutes with tools like LogicLoop.

Marrying the Best of Both Worlds

As a company matures, oftentimes the most robust solutions require a mixture of both types of systems. One of the most common ways to use the strengths of both machine learning models and rules-based systems is to feed the output of the machine learning model into one of the inputs of the rules-based alerts.

With tools like LogicLoop, you can combine data from your machine learning models with additional data about user behavior, blocklists and more to build a powerful rules engine to combat fraud.

Conclusion

By utilizing both machine learning and rules, you can capture fraudulent user behavior that is both human intuitive and complex. Rules are powerful when you know the logic and you need to be precise. Machine learning is powerful when you want to be proactive and predictive when approaching a large, complex data set. When it comes to fraud detection, it's clear that there's a use case for both of these methodologies and that both are here to stay.

Get started with a free trial

Improve your business operations today
Start Now
No credit card required
Cancel anytime