In April 2020, I penned a column titled “Qualifying the Skills Gap” which arose from ISACA’s findings in the State of Cybersecurity 2020, Part 1: Global Update on Workforce Efforts and Resources report and discussed the need to dissect the skills gap. Part 1 of this report and my ensuing column were well received, especially among those involved in cybersecurity apprenticeship programs. One in particular—CyberUp1—contacted me because the ISACA report findings were beneficial to its purpose and positioning and led to a 2-part podcast2 that was reportedly popular among listeners. Those podcasts and continuing dialog solidify the real challenge plaguing the security industry—a pipeline shortage.
The shortage of human resources will not diminish until we adequately ramp up digital literacy activities earlier in the academic trajectory and diminish the digital divide that affects so many across the globe. We must also minimize human-enabled barriers that presently exist within talent management systems.
Accumulating experience takes time, and many trying to enter the field struggle to find employment after completing university, college or reskilling programs, which is unfathomable given the large imbalance of supply and demand. I am not advocating that all jobs be stripped of experience requirements, but hiring managers should reevaluate all job postings to avoid unnecessarily restricting applicants who could, arguably, perform the job. I further recommend judicious use of required and preferred knowledge, skills and experience statements. To mitigate the pipeline shortage, the cybersecurity industry must provide means of accelerating the attainment of experience. Enter apprenticeships and performance testing.
Apprenticeships are relatively new to cybersecurity and, here in the United States, they differ from traditional apprenticeship programs and are called Industry-Recognized Apprenticeship Programs (IRAPs).3 According to the US Department of Labor, “IRAPs are developed or delivered by entities such as trade and industry groups, corporations, nonprofits, educational institutions, unions and joint labor-management organizations.”4 IT-related occupations do not have the benefit of an overarching authoritative body, which distinguishes them from longstanding construction industry apprenticeship programs. In this regard, IRAPs may perpetuate existing problems of varying degrees of knowledge and inconsistent competency levels. In other words, cybersecurity IRAPs may produce different outputs if one favors penetration testing and red teaming while another focuses on defensive analyst work. This situation is not novel and occurred within the US Federal Reskilling program, which taught 2 cohorts using 2 different training providers with distinct curriculum. Industry reporting suggests graduates had to wait for the creation of tailored federal jobs to find work in their trained field.
Performance tests, on the other hand, exist today and provide a viable attestation of minimum competency. According to the Performance Testing Council (PTC), a performance test is “an assessment which includes a demonstration that a person can do what you want them to be able to do.”5 For those unfamiliar with performance testing, an excellent example is taking a driving test to demonstrate one can operate a car within the guidelines of the rules of the road before a driver’s license is issued. With this understanding, a performance test can fulfill at least portions of experience requirements.
Performance tests are not new. They can be found in academia, construction, healthcare and other industries—to include IT. Performance tests range from low (pencil and paper) to high realism (real-world conditions). Academic writings often refer to these as performance assessments which “can be made more authentic by presenting performance tasks more like those in the real world.”6 Fortunately, technology has greatly enabled the testing industry’s ability to increase authenticity and mitigate errors involving observation and judgement.
Performance testing is not appropriate for everything and, therefore, will not fully replace traditional cognitive tests. In fact, multiple-choice exam items offer many benefits when constructed properly. However, multiple-choice items cannot measure some types of problem solving and reading ability is a success factor.7 Most of us know individuals who are great test takers but cannot perform the work. Performance tests help address these issues.
The IT industry is plagued by misinterpretations of performance tests which are not apparent in other career fields. A notable example is confusion between performance-based tests (PBT) and performance tests (PT). People may contest they are the same but, according to the PTC, they are not. The example PTC often uses involves baking a cake whereby testing a candidate on their knowledge about cake making is a performance-based test, which also includes drag and drop items. Conversely, if you wanted to know whether a person could source the needed ingredients and tools, measure ingredients, safely operate equipment and produce a tasty dessert, you need a performance test. Performance tests increase exam fidelity and face validity, which is the degree to which the assessment appears effective in its stated aims.8
Conclusion
The shortage of cybersecurity professionals will remain for the foreseeable future. To survive, organizations must think differently, act differently and challenge the status quo. The COVID-19 pandemic disrupted global markets and some economists believe we will see widespread job loss as enterprises reopen. While tragic, the pandemic serves as a powerful reminder that the only constant in this world is change. Only time will tell exactly how many jobs disappear or organizations close their doors but, in the meantime, we must find ways to employ those interested in any aspect of information security. Human networks remain an important factor in filling open roles, but those just starting out may not “know” someone who can help get their foot in the door. Apprenticeships help build experience, however, we must strive to standardize programs and outcomes. Today, performance tests serve to validate that individuals can do certain things—something once limited to hands-on work experience.
Jonathan Brandt, CISM, CCISO, CFR, CISSP, CSA+, PMP
Is a senior information security practice manager in ISACA’s Knowledge and Research department. In this role, he contributes thought leadership by generating ideas and deliverables relevant to ISACA’s constituents. He serves ISACA® departments as a subject matter expert on information security projects and leads author management teams whenever external resources are necessary. Brandt is a highly accomplished US Navy veteran with more than 25 years of experience spanning multidisciplinary security, cyberoperations and technical workforce development. Prior to joining ISACA, Brandt was a project manager for classified critical infrastructure projects across the globe.
Endnotes
1 CyberUp is a nonprofit that evolved from the Midwest Cyber Center in the St. Louis, Missouri, USA, region. CyberUp is led by Executive Director Tony Bryan, who also serves as chair of the US National Initiative for Cybersecurity Education (NICE) Apprenticeship Sub-Working Group.
2 CyberUp, “Cy Saves the Day Podcast Part 1 w/Jon Brandt,” and “Cy Saves the Day Podcast Part 2 w/Jon Brandt”
3 Apprenticeship.gov, Industry-Recognized Apprenticeship Program
4 Ibid.
5 Performance Testing Council
6 Waugh, C. K.; N. E. Gronlund; Assessment of Student Achievement, 10th Edition, Pearson, USA, 2012
7 Ibid., p. 93
8 Lexico.com, “Face validity”