‘If you can’t describe what you are doing as a process, you don’t know what you’re doing.’ This quote by W. Edwards Deming of TQM fame is taking on new meaning in the context of data-led auditing.

The massive volumes of data available and the power of data analytics technologies are fundamentally changing audit and assurance. But the availability of data as a new source of information is not only a blessing for auditors, it is also an arduous operational challenge for audit organisations. What are the tools that can help us navigate audit-relevant data and overcome these challenges?

Process mining in audit

An increasing number of public audit institutions have started developing their own analytical capabilities and data-centric methods to enhance audits, including process mining. Process mining is designed to discover, monitor and improve real processes (i.e., not assumed processes) by extracting knowledge from event logs readily available in today’s information systems [1]. Process mining is a very different approach to evidence collection and analysis, as it does not focus on the ‘content,’ such as the value of the transactions and its aggregations, but rather on the path of transactions and the transactional processes themselves. It is thus a powerful tool for tests of controls, such as those for segregation of duties. It offers actionable ways to dive into data flows without getting lost. It provides auditors with a tested and mature technology for extracting useful information about business processes, enabling them to navigate throughout the huge quantity of data stored and displayed in financial and management information systems.

Modern technology gives us the ‘possibility for reviewing entire populations to assist the auditor to test for items that are outside the norm’ [2], but processing the results of such testing in the form of reports or long lists of exceptions is a tedious task. Process mining also offers an easy-to-understand visual representation of process conformance, with interactive features, such as the ability to filter and zoom in on exceptions and transactions of interest.

Understanding complexity

While process mining offers many benefits, using the methodology as an auditor is, however, not without challenges. The European Court of Auditors started experimenting with process mining two years ago, under the auspices of the ECALab, our in-house technology incubator. We have carried out a number of pilots with promising results, and they highlighted a key issue: we, as external auditors, are not data producers; we have to use the data that we obtain from the auditees for process mining. Unlike internal auditors, who audit an organisation from within, have full access to all systems and only have to put in place the necessary interfaces between the audited data and the process mining solution once, an external public auditor might have to carry out data extraction and pre-processing for each auditee. As organisational processes tend to be quite complex in real life, raw process data often results in overly complicated, so-called ‘spaghetti’ process maps.

A key lesson we learned for overcoming this challenge is to focus on collaboration with the “owners” of a process early on. The data analyst must work together with a domain expert (preferably an expert auditor) to further pre-process the data and/or simplify the process map to allow for meaningful analysis (Figure 2).

In addition, we work in multi-disciplinary teams, data analysts and expert auditors are closely cooperating to ensure that we make the best out of the data received from external auditees. As the guardian of the EU’s finances, we follow the amounts paid from the EU budget up to the final beneficiaries, for example, a young farmer in a Member State applying for EU subsidies. However, national public administrations often use custom-developed and/or legacy information systems that may not be able to export data in a ‘process mining friendly’ format. Working with a range of experts across disciplines is key to our approach for process mining, and addressing underlying technical and data quality issues to support analysis.

Process mining has the potential to challenge the traditional working methods of public auditors. It offers an interactive, visual approach to examine the entire population instead of sampling, and allows us to create more added value with our audits and provide better assurance to the public. Even though there are ‘no magic recipes’ ready for use, the currently available process mining solutions already allow public auditors to integrate this technology into their audit work.

[1] Aalst, Wil van der, 2016, Process Mining – data science in action, Springer, 2016, 2nd ed.

[2] ACCA, 2019, Machine Learning: more science than fiction