As emerging technologies continue to develop, organizations across industries are increasingly implementing generative artificial intelligence (AI) in areas such as auditing, banking, education, finance, public health, and more. For example, in August 2023, the University of Hong Kong (China) integrated generative AI into its teaching and learning environments, marking the beginning of a new era in education.1 On the enterprise side, in May 2024, PricewaterhouseCoopers (PwC) made a notable announcement outlining a strategic partnership agreement between its US and UK firms and OpenAI.2
As the landscape evolves, it is foreseeable that a growing number of enterprises offering certification and IS services will leverage the power of generative AI for conducting IS audits. While this is likely during the initial phase of integrating ChatGPT into professional services, it presents both benefits and potential risk that demand the attention of IS auditors. There is value in examining the opportunities and challenges associated with incorporating generative AI into IS audit engagements, with the goal of providing valuable insights into its implications.
As the landscape evolves, it is foreseeable that a growing number of enterprises offering certification and IS services will leverage the power of generative AI for conducting IS audits.Opportunities From the IS Audit Perspective
Generative AI has transformed IS audit engagements, improving efficiency and effectiveness significantly. By addressing challenges such as compliance, standard interpretation, and ensuring the reliability of information provided by entities, generative AI has become a cornerstone of innovation in the IS audit industry.
Efficiency and Effectiveness
The emergence of generative AI has undeniably brought forth a paradigm shift in the realm of IS audit engagements, primarily in terms of efficiency and effectiveness.3 Generative AI surpasses mere tool status; it is a transformative force that has revolutionized the planning, execution, and completion stages of IS audits. Therefore, the integration of generative AI has the potential to unlock unprecedented levels of efficiency, streamlining auditing processes and enhancing overall effectiveness in delivering audit outcomes.4
The successful utilization of generative AI in IS audit engagements hinges on comprehensive training aligned with performance standards, reporting standards, and other guidelines, such as International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) 42001 and the EU Artificial Intelligence (AI) Act. When appropriately trained, generative AI can effectively expedite the completion of IS audit engagements. This efficiency stems from the ability of generative AI to overcome the limitations associated with long working hours, which can lead to decreased efficiency and productivity.
Research indicates that cognitive abilities, such as concentration and decision making, are significantly impacted after an individual works more than 55 hours in a week.5 By leveraging generative AI, organizations can mitigate these challenges and maintain optimal levels of efficiency throughout the IS audit process. Furthermore, research shows that the current generation of workers prioritizes work-life balance more than previous generations.6
Through the incorporation of generative AI into IS auditing, the challenges posed by decreased cognitive abilities resulting from long working hours can be effectively mitigated. Generative AI’s efficiency and productivity remain unaffected by extended work periods, enabling the delivery of IS audit deliverables with consistent quality. For example, generative AI could be employed to analyze reports generated by the accounting system, ensuring both accuracy and completeness. This utilization would help streamline substantive testing procedures, which are traditionally resource intensive, within the audit process. In addition, generative AI can also predict business trends and analyze enterprise performance based on reports collected from the accounting and enterprise resource planning (ERP) systems. Consequently, generative AI can significantly enhance the efficiency and effectiveness of IS audits, ensuring that optimal outcomes are achieved.
Compliance
In the 21st century, the emergence of technological advancements is occurring at a pace faster than previously envisioned. Technologies such as AI, edge computing, digital twins, and more are reshaping many industries.7 Consequently, the development of related standards, regulations, and certifications is crucial for effectively governing and regulating these disruptive technologies and ensuring their responsible implementation by qualified professionals. For example, the Council of the European Union approved the EU AI Act on 21 May 2024, marking a significant milestone in the governance of AI use.8 The AI Act includes requirements for data governance, technical documentation, recordkeeping, technical robustness, transparency, human oversight, and cybersecurity.9 Such an act is influential as it serves as a potential blueprint for other jurisdictions to implement AI legislation in the future.
Because regulatory standards are often lengthy and complex, it is impractical to expect IS auditors to recall every minute detail within them. Typically, IS auditors possess in-depth familiarity with the standards that are specifically relevant to their daily responsibilities. However, they may be less acquainted with sections of regulations that fall outside their routine work scope, which could result in an elevated detection risk. Consequently, when conducting IS audits in areas where auditors lack familiarity or prior experience, there is a risk that the testing may not adequately address the assertions and risk outlined in the standards.
Nevertheless, with the aid of a well-trained generative AI model, an IS auditor can effectively mitigate this risk by incorporating compliance testing into the IT environment, even in IT environments where the auditor has little familiarity. It is foreseeable that generative AI will soon offer suggestions on compliance testing templates tailored to specific industries and IT environments. Furthermore, generative AI could provide concise summaries of key points from relevant standards, enabling IS auditors to prioritize areas that require heightened attention.
Standard Interpretation
Judicial interpretation refers to the process by which judges interpret legislation and regulations. It is important to note that judges may have varying interpretations, potentially resulting in differences not only between countries but also within the same jurisdiction.10 The concept of judicial interpretation can also be applied to IS audit, specifically in the context of standard interpretation. Standard interpretation involves IS auditors basing their interpretations of standards on their individual experiences, which can vary. Consequently, misunderstandings can arise, leading to different IS audit conclusions for the same IT environment.
Such issues can be effectively addressed by gaining a comprehensive understanding of the standard’s background, including the reasons behind its establishment, the events that influenced its development, and the risk it aims to mitigate.
Accounting for historical context is crucial when interpreting standards due to the rapid pace of technological advancements compared to the rate at which standards evolve. There may exist a gap between the latest technology and the corresponding standard. Therefore, it is advisable to interpret standards with reference to the time of their establishment rather than solely relying on current circumstances. To ensure accurate interpretation, IS auditors can seek verification from leading authorities.
Generative AI can contribute to the IS audit field by enabling the consistent delivery of findings that reflect compliance with the standards, thereby assisting IS auditees in meeting IT environment concerns more effectively.How does generative AI address the issue of IS audit interpretation? If generative AI is trained using appropriate standards and related background knowledge, it can interpret the standards based on their original background and objectives. This capability would assist IS auditors in avoiding misunderstandings and facilitate more effective IS audits aligned with the standards. Generative AI can contribute to the IS audit field by enabling the consistent delivery of findings that reflect compliance with the standards, thereby assisting IS auditees in meeting IT environment concerns more effectively.
Reliability of Information Provided by Entity
Information provided by entity (IPE)—that is, any forms or documentation produced or provided by the entity under audit—serves as audit evidence.11 Evaluating the completeness and accuracy of IPE is essential, as it profoundly impacts the reliability and integrity of the IS audit conclusions.12 If the data being retrieved is inaccurate, it compromises both the reporting and the execution of controls. Incorrect data can lead to inaccurate results.13
The IS auditor should focus on testing the report logic instead of solely observing the report generation process. Report logic encompasses the code, algorithm, and steps involved in extracting data from its source and transforming it into a human-readable format.14 Testing the report logic helps prevent auditees from altering data sources, thereby ensuring the completeness and accuracy of reported information.
For instance, the riskiest type of query is the ad hoc query, which falls outside the scope of IT general controls. An ad hoc query is any nonstandard query created to produce information on an as-needed basis,15 stemming from the lack of predefined parameters.16 To address the ad hoc query risk, IS auditors are obliged to allocate a significant amount of time to the examination of nonstandard queries.
Nevertheless, IS auditors are not limited to reviewing codes for a single report; instead, they may need to review codes for hundreds of report generations, which can be a time-consuming task. To streamline this process, generative AI assistance would be invaluable. Generative AI could verify the compliance of the code or algorithm with IS audit requirements in terms of completeness and accuracy for the reports. Furthermore, generative AI would have the ability to identify and highlight questionable queries, allowing IS auditors to conduct further assessments. As a result, IS auditors could allocate their attention to other high-risk areas with greater focus and effectiveness.
Challenges From the IS Audit Perspective
While generative AI brings unprecedented benefits to numerous industries, it also presents challenges. It is crucial to address issues such as professional judgment, interactions with auditees, concerns related to hallucination and overreliance, and cybersecurity and privacy when implementing generative AI in IS audits.
Professional Judgment
Professional judgment is crucial for a successful IS audit engagement; however, its definition can be challenging. According to the literature, professional judgment encompasses an amalgamation of relevant training, knowledge, and experience in the field of IS audit and ethical standards.17 It enables an auditor to make informed decisions regarding appropriate procedures within the context of an engagement.18
Many accounting and auditing bodies worldwide, including the Hong Kong Institute of Certified Public Accountants, the Association of Chartered Certified Accountants, and ISACA®, mandate that candidates seeking professional membership possess several years of experience and pass examinations. These requirements underscore the need for candidates to acquire professional expertise to support their professional judgment.
For example, ISACA, in accordance with the Certified Information Systems Auditor® (CISA®) requirements, necessitates a minimum of five years of professional experience in information systems auditing, control, assurance or security for candidates applying for the CISA designation.19 The enhancement of professional judgment holds significant importance in the context of IS audit engagements.
However, the question arises as to how to evaluate the professional judgment of generative AI in the context of IS audit. With the advancement of customized generative AI systems, it is foreseeable that generative AI will not be limited to providing simple “Yes” or “No” answers in IS audit engagements. Instead, generative AI will encompass various engagement sections that demand judgment. Different industries necessitate specific professional judgment in their IS audit engagements. While generative AI can certainly obtain extensive knowledge of IS audit through appropriate training, it will be challenging for it to reproduce the professional judgment of an experienced auditor.
Interactions With Auditees
Another challenge for generative AI lies in its interactions with auditees during the IS audit process. From the planning stage to the execution and completion stages, there is a substantial level of interaction between IS auditors and auditees.20 For instance, IS auditors often initiate a kick-off meeting with auditees to understand IT processes, infrastructure, and personnel. They also conduct closing meetings with those charged with governance to discuss the audit findings. Effective communication skills are crucial for IS auditors, as they need to actively listen to people and engage in discussions regarding the daily work performed. These exchanges constitute a significant portion of an IS auditor’s responsibilities.21
Furthermore, IS auditors require conflict management skills beyond mere communication skills. Auditees sometimes regret not providing supporting documents to IS auditors, for example, or complain that the auditors requested too many supporting documents. Some typical complaints from auditees include:
- “Why would the IS auditors request these documents again when we already provided them last year?”
- “We did not have any update on the IT environment this year. Why were there no findings last year, but there are significant findings this year?”
- “We did not provide these documents last year. How did the auditor conduct the IS audit last year without these documents?”
There is no doubt that such complaints would directly affect the relationship between auditors and auditees and may have a negative impact on the efficiency of the IS audit engagement. More important, if a positive relationship between auditor and auditee cannot be maintained, such an engagement may not recur in the next year. Hence, maintaining good relationships between auditors and auditees is key for engagement.
To effectively apply generative AI in the field of IS audit, it is crucial to possess effective conflict management skills. Without such skills, the use of generative AI in conducting IS audits, delivering findings to auditees, and engaging in other interactions with auditees may introduce challenges. It is imperative to consider how conflict management skills can be embedded into generative AI, as this would be beneficial for the IS audit industry in the long run. It is recommended that conflict management skills and communication skills be provided during the training of customized generative AI. However, the challenge lies in how to qualify and quantify conflict management skills and communication skills effectively. Engaging in co-auditing with human auditors can be one of the most effective ways to tackle these issues, particularly when the implementation of generative AI within IS audits is not yet fully developed.
Hallucination and Overreliance
Another challenge is ensuring the integrity of generative AI while providing IS audit services. IS auditors need to pay close attention to generative AI’s integrity, as any shortcomings may have a negative influence on IS audit engagements. Many people who have implemented generative AI in their workplaces assume that it is highly reliable and will not provide false information or unfair analyses. However, it is worth noting that generative AI models, such as ChatGPT, have previously provided bogus cases as legal references.22
For instance, according to published news accounts, a New York (USA) judge fined lawyers and law firms in 2023 over an unprecedented instance in which ChatGPT was blamed for the submission of fictitious legal research.23 A Canadian lawyer was also fined for citing fictitious cases created by an AI chatbot.24 Such events demonstrate the significant impact of hallucination on professional fields. (Hallucination refers to generative AI responses that are characterized by fabrication or inaccuracy.25) This phenomenon results from training a generative AI model on insufficient or flawed information.26 Professionals who rely on hallucination in their work will inevitably introduce adverse consequences, potentially resulting in financial penalties. In particularly grave situations, there could be legal repercussions, including imprisonment.
To address hallucinations in generative AI, several preventive approaches can be employed:27
- Use high-quality training data. Since generative AI relies on data to perform specific tasks, enterprises must prioritize data quality when developing their customized generative AI models and ensure that the AI is trained on diverse and well-structured data.28
- Define the purpose the generative AI model will serve. Enterprises should clearly define the specific purpose, responsibilities, and limitations of the generative AI model. This approach will enhance the generative AI’s efficiency by reducing the generation of irrelevant information and preventing the occurrence of hallucinatory results.29
- Continuously test and modify the model. Enterprises must continuously test the generative AI model to align results with expectations. Moreover, emphasis should be placed on reviewing and updating the training data to prevent its obsolescence, which can also address the issues of generative AI’s standard interpretation. Such processes directly and significantly enhance performance.30
Research shows that excessive reliance on generative AI can also become problematic.31 Allowing all tasks to be performed solely by generative AI without human review and critical thinking could result in misleading IS audit conclusions.32 Hence, based on news reports and studies, it is evident that heavy reliance on generative AI can have a detrimental impact on professionalism.
It is recommended that when implementing a well-trained generative AI solution, professionals should thoroughly review any research it generates with a sense of professional skepticism. The human professional should assume ultimate responsibility for the deliverables of the IS audit engagement and sole liability in the event of any misunderstandings regarding the IS audit conclusions, even with the assistance of generative AI. In cases that involve conflict between the judgment of generative AI and that of the IS auditors, the IS audit conclusion must align with the conclusions reached by the human IS auditor. Generative AI serves as a tool to support IS auditors and must always be relegated to the role of preparer, while human professionals must fulfill the role of reviewer.
Cybersecurity and Privacy
Privacy is also considered a challenge when implementing generative AI for IS audit.33 In general, generative AI learns a vast amount of information from unstructured data without supervision, which may also include personally identifiable information (PII).34 Using PII for training purposes without the related person’s permission may result in legal action.
IS auditors may be wondering how a customized generative AI model can be trained appropriately. One route to take is using reports from SAP, a widely used ERP platform that centralizes data for cross-departmental access and sharing. The auditor would collect various types of SAP reports from clients across industries to ensure that the customized generative AI system can summarize common patterns and address IS audit risk.
It is important to note that before providing such SAP reports to the customized generative AI model for training purposes, IS auditors must comply with national or local legislation, i.e., obtain clients’ consent, anonymize and de-identify the information, and ensure secure data storage and transfer.35
Moreover, when training customized generative AI, it is crucial to protect the sensitive information provided by clients, as hackers may exploit it for illegal purposes, such as selling it on the dark web. Hence, a detailed incident response plan is recommended.
Conclusion
There are both opportunities and challenges involved in incorporating generative AI into IS audit engagements. In addition to the opportunities mentioned, IS auditors should prioritize addressing potential risk associated with generative AI, including privacy, cybersecurity, and ethical concerns.36 Furthermore, enterprises should make efforts to avoid succumbing to herd instinct bias. It is crucial for enterprises to conduct thorough cost-benefit analyses before implementing generative AI in their IS audit engagements. Now is the time to consider how to maximize the opportunities associated with generative AI and minimize the associated costs. If the risk is under control, generative AI can be a powerful tool for the IS audit industry.
Endnotes
1 The University of Hong Kong, “HKU Introduces New Policy to Fully Integrate GenAI in Teaching and Learning,” Yahoo! Finance, 3 August 2023
2 Atkinson, J.; Umang, P.; “PwC Is Accelerating Adoption of AI With ChatGPT Enterprise in US and UK and With Clients,” PwC, 29 May 2024
3 Boussour, L.; “The Productivity Potential of GenAI,” EY Parthenon, 22 January, 2024
4 Boussour, “The Productivity Potential”
5 Virtanen, M.; Singh-Manoux, A.; et al.; “Long Working Hours and Cognitive Function: The Whitehall II Study,” American Journal of Epidemiology, vol. 169, iss. 5, 2009
6 Hermanson, D.; Houston, R.; et al.; “The Work Environment in Large Audit Firms: Current Perceptions and Possible Improvements,” Current Issues in Auditing, vol. 10, iss. 2, 2016; Smith, K.; “Work-Life Balance Perspectives of Marketing Professionals in Generation Y,” Services Marketing Quarterly, vol. 31, iss. 4, 2010
7 Nin, C.; “How Does the Synergy Between AI, 5G, Edge and Digital Twins Enable Better Decision Making?,” RCR Wireless News, 21 November 2023
8 Osborne Clarke, “When Will Businesses Have to Comply With the EU’s AI Act?,” 16 July 2024
9 Hickman, T.; Lorenz, S.; et al.; “Long Awaited EU AI Act Becomes Law after Publication in the EU’s Official Journal,” White & Case LLP, 16 July 2024
10 Brooks, E.; “What Is Judicial Interpretation: Definition, Methods,” Liberties, 22 November 2022
11 Vicente, V.; “IPE Best Practices for Audits and Controls,” AuditBoard, 28 June 2023
12 Vicente, “IPE Best Practices”
13 Childress, L.; “What’s Information Produced by the Entity (IPE) and How Does It Impact Financial Reporting Controls?,” Armanino LLP, 11 February 2021
14 Childress, “What’s Information”
15 Childress, “What’s Information”
16 Pattamatta, R.; “What Is an Ad Hoc Query? Benefits, Types, & Examples,” Databrain, 8 July 2024
17 Oana, A.; Maria, A.; “Professional Judgement. The Key to a Successful Audit,” SEA-Practical Application of Science, vol. 2, iss. 3, 2014
18 Oana and Maria, “Professional Judgement”
19 ISACA®, “Certification Application: What Are the Requirements to Become CISA Certified?”
20 Carlisle, M.; Hamilton, E.; “The Role of Communication Mode in Auditor-Client Interactions: Insights From Staff Auditors,” Current Issues in Auditing, vol. 15, iss. 1, 2020
21 David, T.; “6 Tips to Build Stronger Relationships With Auditees,” AuditBoard, 8 June 2021
22 Neumeister, L.; “Lawyers Submitted Bogus Case Law Created by ChatGPT. A Judge Fined Them $5,000,” AP, 22 June 2023
23 Armstrong, K.; “ChatGPT: US Lawyer Admits Using AI for Case Research,” BBC, 27 May 2023
24 Cecco, L.; “Canada Lawyer Under Fire for Submitting Fake Cases Created by AI Chatbot,” The Guardian, 29 February 2024
25 Deakin University, “GenAI Limitations”
26 Deakin, “GenAI Limitations”
27 IBM,“What Are AI Hallucinations?,” 1 September 2023
28 IBM, “What Are AI Hallucinations?”
29 IBM, “What Are AI Hallucinations?”
30 IBM, “What Are AI Hallucinations?”
31 Singirikonda, M.; “AI and Automation in Cybersecurity: Future Skilling for Efficient Defense,” ISACA® Journal, 1 May 2024
32 Singirikonda, “AI and Automation”
33 Bible, W.; “Generative AI in Accounting: Opportunities and Risks to Assess Today,” Deloitte, 18 December 2023
34 Chung, A.L.; “Hong Kong: The Privacy and Ethical Risks of Generative AI Cannot Be Ignored,” Office of the Privacy Commissioner for Personal Data, China
35 Baig, A.; “Navigating Generative AI Privacy: Challenges & Safeguarding Tips,” Securiti, 7 March 2024
36 Bible, W.; “Generative AI in Accounting: Opportunities and Risks to Assess Today,” Deloitte, 18 December 2023
RUI FENG ISAAC LEE | CCSK, FMVA
Is an associate from BDO Hong Kong who specializes in information systems audit.