A rise in technologies like AI and robotic process automation (RPA) has increased demand for process mining tools. And more recently, an enterprise spurt in technology investment to respond to the pandemic has accelerated that demand.

Process mining enables companies to automate and streamline operations, by identifying best practices and then disseminating them across an organization. This reduces waste, allocates physical and human resources more efficiently, and enables faster responses to internal and environmental changes. Gartner estimated the market for dedicated process mining tools grew between from $ 110 million in 2018 to $ 320 million in 2019.

“The challenges of 2020 have shifted perspectives back toward efficiency for the sake of business continuity,” said Gustavo Gomez, CEO of Bizagi, a business process automation tools provider.

Many organizations know that the behavior of their staff and customers has changed, but how exactly? Process mining is about seeing the way the processes really work and interact with other processes. Many companies have an understanding of processes that don’t match reality, and most managers don’t really know how a process actually works in practice, said Gomez. They may have a definition, but in practice, businesses often have many gaps where the process is applied differently. “Because process mining uses real data, you find out what the real process is, not the imaginary process is in somebody’s head,” said Gomez.

What process mining is, and how it’s different from RPA

RPA and low-code tools are starting to use process mining to model existing workflows to create a template for new automations. RPA stands for “robotic process automation” and is actually a different step that sometimes comes after process mining. Process mining involves mining data logs from applications like ERP and CRM to assemble an accurate model of how a business process, like order to cash (OTC), works. These models are sometimes used to automatically generate RPA programs, which are often described as software robots or “bots” for short.

These modern tools are sometimes complemented by tools for business process modeling and business process management, which are sometimes both confusingly called BPM. Legacy BPM tools providers are starting to incorporate process mining capabilities as the market heats up. One of the key advantages of legacy BPM tools providers is their experience in identifying and remediating bad processes.

Many enterprises are finding it difficult to scale beyond a few software robots or bots because they are automating a bad process that cannot scale. “Most businesses are automating processes through RPA and hyperautomation without first fully understanding their data and processes,” explained Gero Decker, CEO of Signavio, an SAP spinoff focused on business transformation.

As enterprises pursue increased efficiencies, there is debate on whether it makes more sense to automate what exists, or to fix it first. Automating a bad process may make it faster, but it also may suffer from chokepoints caused by integration with legacy systems or approval processes. Process mining can help a company to fix a bad process first.

Chris Nicholson, CEO of Pathmind, a company applying AI to industrial operations, argues, “The main challenge to overcome before applying process automation is to standardize the current processes performed by people. If they are not standardized, there can be no automation.”

With process mining, companies can see whether or not their current processes are standardized, so they know which problem they have to solve first: standardization, or automation. If the processes are standardized, process mining can clarify what those processes are, to enable automation with RPA. Process mining helps RPA projects discover and validate the processes that are actually being carried out in a company.

Decker expects 2021 to be a year where there is a re-evaluation of whether RPA projects were architected effectively, and whether the underlying business processes were thoroughly understood enough to automate effectively at scale. By more closely associating RPA with process mining and process management, RPA will stand a better chance for success — and organizations will not adopt automation for automation’s sake, and instead focus on ROI and higher success rates.

Task mining connects the dots

Traditional process mining is being enhanced with something called task mining to connect the dots. Task mining uses computer vision apps running on the desktop to record how a user accomplishes a task. This complements process mining with more data about steps that application logs might miss.

Task mining helps solve two of the key limitations with process mining, said Harel Tayeb, CEO of Kryon, an RPA tools provider. First, to extract knowledge from event logs, process mining technology requires integration with the underlying systems. Although the vendors have streamlined this process, it still requires some time and resources. Second, the level of insights generated with process mining is still at a process or transaction level, so it fails to capture low-level user behavior.

Sem Sergunin, director of product strategy at Automation Anywhere, another RPA provider, said, “Traditional process mining cannot capture user interactions with systems and is blind to an application, which doesn’t record logs, such as Microsoft Office, or terminal and virtual desktop environments.”

Task mining operates at the level of the user desktop, which ensures compatibility with any app with a graphical user interface. Many task mining tools can also capture the steps involved in a process across multiple users involved in a process. This aids in process discovery, making visible the invisible connections and patterns of individual actions in the real world.

Another benefit is that the artifacts can serve as a better starting point for coding RPA bots, which also operate through a GUI interface. Automation Anywhere’s Sergunin said, “Before, the ultimate purpose of process mining was process reengineering and optimization.  Now, process discovery captures manual user activities, which can also be transformed into automation opportunities for RPA.”

Seeing the invisible

Process mining makes the invisible workflows between organizational nodes visible by analyzing application logs to automatically monitor ongoing processes and create process maps. It’s like seeing an invisible man by throwing a handful of sand on him.

The fundamental ideas behind process mining date back to the time of Henry Ford to answer questions about how to build a better task, and how to build a process that weaves multiple tasks together. Task optimization grew out of the scientific management techniques of Frederick Taylor at the dawn of the 20th century. Taylor carefully observed and documented the step-by-step processes that workers used to shovel coal, move pig iron, and build machines. These ideas limped along on the backs of manual efforts by highly specialized and well paid process experts.

Improvements in AI and the rise of integrated cloud services is now making it feasible to automate and scale process mining in a way that speeds the ability to internally sense what is going on within the enterprise. From an evolutionary perspective, process mining represents the development of a new sensory system for organizations and confers a huge advantage for survival in the ecology of enterprises. In the ecology of companies battling for market share, a company with even a primitive capability of seeing invisible work flows better than the competition has a huge advantage over companies that cannot see. As Erasmus stated in 1500 AD: “In the land of the blind, the one-eyed man is king.”

“Many years ago, when operations had paper files that were passed along, the backlog of the work was easily visible,” said Erlin Kakkanad, Founder and CEO of Engineer Creative Thinking, a process improvement consultancy. “With automation of workflow, work has become invisible in the physical space, and leaders have to ensure they are tracking efficiency online.”

Process mining is also helping to break down the silos of data about how different parts of the organization work. “We’ll see enterprises break up internal data silos using process mining for an aggregated point of view and full transparency into end-to-end processes, making operations measurable and delivering necessary data and insights to make business decisions at all levels,” said Jon Weiss, senior vice-president of emerging technology at Software AG. “We’ll see companies leaning on process management to react quickly to all types of new circumstances — in the market, technology, or even regulatory changes.”

The leading players

Both Software AG and the SAP spinoff Signavio have found some early success using process mining to help businesses respond to COVID-19. Weifu High Technology Group, a leading engine component manufacturer in China, was having a hard time identifying the cause of new bottlenecks across multiple applications after COVID-19 restrictions. Weifu implemented Software AG’s ARIS to track, integrate, and manage processes and regulations in one place. Yang Guifeng, CIO of Weifu High Technology Group, said this has allowed them to identify performance problems and significantly reduce communication costs within the enterprise.

A global fruit producer had to ramp up a new training program to address new social distancing requirements for its nearly 1,300 employees. They turned to Signavio for tools to identify how changes impacted customers and to analyze the minimum staff requirements to execute the core process and to ensure backups were identified for crucial staff positions. The tools also helped ensure staff could find centralized data drive documentation on how best to perform new tasks. It also helped management identify and invest in training for areas that have the greatest impact on customer service and long-term loyalty.

Last week, enterprise software giant SAP announced it would acquire Signavio, to double down on processes organizations can automate. This brings one of the big software giants directly into this market, which has until now been filled with relatively small players. Celonis is one of the largest pure play process mining vendors, which raised $ 290 million in late 2019 at a $ 2.5 billion valuation.  Other pure-play process mining vendors include:

• Apromore
• Cognitive Technology
• Everflow
• Fluxicon
• FortressIQ
• INTEGRIS
• Lana Labs
• Logpickr
• Mehrwerk AG
• Minit
• Process Analytics Factory
• Puzzle Data
• QPR Software
• StereoLOGIC

Major RPA vendors are starting to invest in building out their process mining capabilities. UIPath developed some of its own internal tools before buying Process Gold for its process mining capabilities in 2019. It also acquired StepShot for its task mining tools. Automation Anywhere has developed its own process mining and task mining capabilities internally. Meanwhile, Blue Prism has announced a technology alliance program with Celonis to improve integration of their offerings.

Other RPA vendors like ABBYY and Kryon are similarly building up their own process mining capabilities.  A number of digital transformation tools providers also offer process mining capabilities as part of their portfolio of offerings including Bizagi, Signavio (SAP), and Software AG (as mentioned earlier).

Down the road, expect to see the major SaaS providers develop their own embedded process mining capabilities. These could provide better integration that comes with the context gleaned from the apps.  Karel van der Poel, senior vice-president at NowX, ServiceNow’s future technologies divisions, said he believes leading platform providers will benefit from an embedded model where process mining technologies are natively being used in the core business applications and not seen as a separate toolset or afterthought.

This has already occurred with BI and AI tooling. ServiceNow and all the other big enterprise software vendors already are providing BI and AI tooling built into their tools. Poel expects leading enterprise software vendors to provide process mining technologies natively through OEM partnerships, acquisitions, or organic build. These internal tools will benefit from familiarity with their own audit logs and best practices for their application. For example, ServiceNow’s Service Graph can capture the context of the event and its overall impact on the process and the business.

Process mining in security, and beyond

Security experts are starting to turn to process mining to provide a new layer for security at the application level. Security tools have traditionally focused on monitoring activity at the infrastructure level. But as hackers grow more ingenious and subtle, process mining could help to identify anomalies that are ever more subtle. “Process mining will become one of the major methods in the coming years for dealing with more complex threats,” said Andreas Grant, founder of Networks Hardware. It’s well suited for seeing threats that manifest themselves gradually over time.

Leading edge companies are starting to use process mining to capture data about physical activities in warehouses and factories or the ways humans build things. These tools often provide integration into the manufacturing execution systems (MES) or warehouse management systems (WMS) used in the factories or facilities.

For example, startup Drishti has developed a set of apps that automate time and motion analysis studies of humans. It can remind human operators when they miss a step and can also be used to capture best practices that can be shared with new operators. It can also identify the root cause of quality issues after the fact. Some early adopters have found significant reductions in training time. Other vendors diving into this space include Hitachi Vantara with its Lumada Video Insights and Tulip Interfaces.

“As companies evolve, knowledge about processes tends to disappear,” said Vadim Tabakman, director of technical evangelism at Nintex, a process management and workflow automation tools provider. He sees process mining being critical in the understanding of how processes work, in order to build out process maps and documentation, so that process knowledge is stored and maintained.

A few hiccups

Despite the advancement, process experts see a few hiccups that need to be ironed out. According to Creative Thinking’s Kakkanad, “The technology is great for processes that are 100% automated and run off the server, but that’s not the reality. Business processes use a slew of applications to provide customer service.”

She also found there are process aspects such as compliance, legal, quality, and value add versus non-value add tasks where the traditional mapping done by a practitioner with Lean and Six Sigma expertise can provide greater value than looking at the process purely from timing. For example, a process could be doing steps that are non-value-adding, and process mining tools may lack the context to detect them.

Another concern is that process mining tools sometimes recommend automating band-aid processes to solve a problem that has long since faded. In these cases, the enterprise will end up spending money to automate a waste process. This is where traditional process mapping work can help to avoid these costly mistakes.

Some believe that the advances in AI, process discovery, and hyperautomation will help fill the gaps. “The most significant indication that process mining technology is maturing has been the introduction of complementary tools and capabilities,” said Dave Borowski, director of IT strategy and business process outsourcing at West Monroe, a business technologies consulting firm.

As powerful as process mining is, it inherently has certain limitations outside enterprise systems in its ability to collect data at an individual user level, and in measuring metrics such as productivity and intra-process efficiency. Consequently, process mining has evolved beyond pure process mining into task mining and other process discovery or intelligence tools, which address these deficiencies.

The combination of these capabilities into process intelligence tools are just starting to be adopted. But Borowski notes that they can create visibility into everything that happens across an entire process cycle, at the system and user level. This greatly expands the types of use cases and value of the technology, as it creates an opportunity to measure and assess elements such as individual and team productivity and procedural compliance, objectively identify and justify opportunities for optimization and automation, and even create reference documentation and other training artifacts.

Keeping humans in the loop

In the long run, to achieve the greatest gains, any successful process mining effort will have to get buy-in from the humans involved. Like other forms of automation, process mining has the potential to threaten jobs, increase the number of challenging tasks people do in a day, or reduce connection with co-workers and customers. “The biggest challenge in technology adoption in enterprise are the humans involved,” said Pathmind’s Nicholson.

There are also concerns that intrusive monitoring can raise new privacy concerns, which could pose significant challenges with new regulations like the European Union’s GDPR. As promising and valuable as these tools can be, they also introduce unique challenges due to the extent and nature of the data they collect, said West Monroe’s Borowski. He recommends companies engage InfoSec, policy, privacy, and risk management teams in the initial stages to further their input into the deployment approach.

He also found that some clients struggle with the “big brother” implications of deploying process discovery tools to collect data at the user and task level. The decision about whether or not to deploy these tools, especially for continuous process monitoring, often depends on the culture of the company, the type of data collected, and consent requirements for the populations impacted.

Rather than just looking at process mining as something to impose on workers, companies may see the biggest gains by finding ways to include and reward employees as part of the adoption. After all, thousands of eyes in the field may see some opportunities that a few experts in the office may miss. Process mining promises an opportunity to not only scale efficiency, but to also scale learning and adaptability across the organization as well.