|Effective date:||June 14, 2022|
|Prepared by:||Office of Audit and Evaluation, National Research Council Canada|
|Related document:||Summary Report – Evaluation of the NRC's Industrial Research Assistance Program|
|Alternate format:||Evaluation of the NRC's Industrial Research Assistance Program (PDF, 1.1 MB)|
Abbreviations and acronyms
- Accelerated growth service
- Accelerated review process
- Compound annual growth rate
- Client Engagement Advisor
- Canadian International Innovation Program
- Contribution to organization
- Equity, diversity and inclusion
- Fiscal year
- GBA Plus
- Gender-based analysis plus
- Innovation Assistance Program
- Information and communication technology
- IRAP innovation portal
- Industrial Research Assistance Program
- Innovative Solutions Canada
- Industrial Technology Advisor
- Large value contribution
- National Research Council Canada
- Office of Audit and Evaluation
- Other government department
- Partial benefit cost analysis
- Program Delivery Advisory Committee
- Public service engagement survey
- Research and Development
- Regional Contribution Agreement Officer
- Strategic Innovation Fund
- Senior leadership team
- Small and medium-sized enterprise
- Technology Access Centre
The evaluation of the National Research Council's Industrial Research Assistance Program (NRC IRAP) covered the period of FY 2017-18 to FY 2021-22. It assessed NRC IRAP's reach, effectiveness, efficiency and relevance. Data collection and analysis was conducted during FY 2021-22. The evaluation was carried out in accordance with the NRC's approved evaluation plan, Treasury Board Secretariat's Policy on Results (2016) and the requirements of the Financial Administration Act. The program was last evaluated in FY 2016-17.
A comprehensive mixed-methods approach was used in the evaluation, including data and document reviews, internal and external interviews, case studies, a survey, focus group discussions, journey mapping, an international comparison study and a partial benefit cost analysis (PBCA). This report provides an overview of the key findings, conclusions, and recommendations.
The evaluation was led by the NRC's Office of Audit and Evaluation (OAE). It applied a mixed-methods approach, incorporating both qualitative and quantitative data from several lines of evidence. This allowed for the triangulation of data across diverse lines of evidence and ensured that consequent evaluation findings were robust and reliable. A gender-based analysis plus lens was also applied in order, to assess how diverse groups of people may be experiencing or differentially impacted by the program.
The evaluation covered the period from FY 2017-18 to FY 2021-22. The evaluation considered all NRC IRAP activities and expenditures, except for its collaborative programs with Global Affairs Canada (i.e. CanExport and Canadian International Innovation Program (CIIP)) and the Innovation Assistance Program (IAP). IRAP International, though included in the evaluation to some extent, was not a primary focus.
- Reach: To what extent have the NRC IRAP's client engagement efforts been effective at reaching intended (i.e. high potential and innovative) clients?
- Effectiveness: To what extent has the NRC IRAP been able to contribute to SME growth through innovation and sustainable economic growth within Canada?
- Efficiency: To what extent has the NRC IRAP been designed and delivered in an effective and efficient manner?
- Relevance: To what extent is the NRC IRAP responsive to Government of Canada priorities and the need for innovation support among Canadian SMEs?
Lines of evidence that support the evaluation include:
- data review (5 years+ of administrative and performance data)
- document review
- Industrial Technology Advisor (ITA) survey (n=125 responses; response rate=41%)
- internal and external interviews (n=86 interviewees)
- partial benefit cost analysis (17 firms)
- ITA focus groups (5 focus groups, n=36 participants)
- contributions to organizations (CTO) case studies (4 organizations)
- international comparison study (4 comparator programs)
- journey mapping of unfunded clients (2 unfunded firms)
For more detailed information on the evaluation methodology, including limitations, refer to Appendix A .
NRC IRAP was established in 1947 and has a mandate to stimulate wealth creation in Canada through innovation. The program offers a suite of innovation support services, including business and technical advisory services, financial assistance and linkages to national and international research and development (R&D) opportunities. In so doing, it aims to accelerate growth among Canadian small and medium-sized enterprises (SMEs) in various science and technology fields.
NRC IRAP Overview
Source(s): Document review and data review
NRC IRAP is an innovation assistance program designed to help Canadian SMEs grow through innovation and technology
Established in 1947, the program is one of four flagship platforms through which the federal government has streamlined the delivery of innovation programming across Canada since 2018. The other three platforms are the Strategic Innovation Fund (SIF), Canadian Trade Commissioner Service, and Regional Development Agencies.
Stimulate wealth creation in Canada through innovation.
Accelerate the growth of small and medium-sized enterprises (SMEs) by providing them with a comprehensive suite of innovative services and with funding.
Be the most impactful program of its kind in the world, where Canadian firms go first to transform their ideas into commercial success.
- help small and medium-sized businesses in Canada develop and commercialize technologies
- collaborate with regional and national organizations on initiatives that support the development and commercialization of technologies by small and medium-sized businesses
NRC IRAP helps firms build their innovation capacities and take their ideas to market
The program supports SMEs through advisory services, financial assistance and fostering linkages to appropriate business and research and development (R&D) expertise in Canada and beyond. NRC IRAP clients currently stem from all sectors of the Canadian economy.
29% of clients come to the program from the information and communication technology (ICT) or digital economy sector, representing the highest concentration of clients in any sector.
NRC IRAP is delivered to clients by five regional teams. Distribution of clients who received advisory services and/or funding between 2018 and 2022:
- 10% Atlantic & Nunavut
- 19% Quebec
- 32% Ontario
- 19% Prairies and Northwest Territories
- 18% Pacific and Yukon
Source(s): Document review and data review
The Vice-President (VP) of NRC IRAP reports to the NRC's President and has overall managerial accountability for the program. The NRC IRAP VP is responsible for making program-wide decisions, informed by advice from members of the senior leadership team (SLT). In addition to the NRC IRAP VP, the SLT is comprised of seven Directors General (DGs) and the executive director of the International Innovation Office. A Board of Advisors helps ensure the program continues to offer the support Canadian businesses need to innovate and grow. They offer perspectives and advice to guide the program's strategic direction and management. Advisory Board members are appointed by the NRC IRAP VP and represent business and innovation stakeholders from across Canada.
Figure 1. NRC IRAP Organizational Structure (2021-2022)
Figure 1 – Text version
This illustrates the National Research Council's Industrial Research Assistance Program's (NRC IRAP) structure in a three-tiered hierarchy.
At the top tier is the President of the National Research Council of Canada.
The second tier is the Vice-President of NRC IRAP and the NRC IRAP Advisory Board composed of businesses and innovation stakeholders, which advises the Vice-President of NRC IRAP.
The final tier is the Direction General: Division Services, Director General: Client Enablement, Five Director Generals: Regional Directors General, and the Executive Director: International Innovation Office overseeing NRC IRAP International. NRC IRAP International was not a primary focus of the evaluation.
Director General, Division Services
Oversees the planning and development of operational policies, tools, and procedures.
Director General, Client Enablement
Ensures that NRC IRAP's client base is sufficiently supported by program delivery and connected to appropriate resources.
Regional Directors General
Responsible for the implementation of the program in their respective locales. Industrial Technology Advisors (ITAs) in each region are the primary holders of relationships with clients and partners. NRC IRAP has a total of 267 ITAs located across the country.
NRC IRAP InternationalFootnote*
Connects Canadian SMEs with the funding, advisory, export and innovation services they need to access new markets and global value chains.
Innovation support services
Source(s): Document review, interviews
NRC IRAP contributes funding to eligible firms and organizations through three primary streams. Regular IRAP is the main funding stream supporting clients with awards between $50,000 and $1M. The accelerated review process (ARP) supports small projects below $50,000. Larger projects (i.e. $1M up to $10M) are supported through large value contributions (LVC). These together make up core IRAP.
White label services
NRC IRAP supports other government departments (OGDs) and non-government organizations (NGOs) that champion innovation among Canadian SMEs. The exact suite of services is still being evolved by the program. However, for the moment, it primarily involves conducting business and technical assessments of funding proposals submitted by SMEs and providing advice to support the development of new and existing innovation programs. Importantly, clients are charged a service fee to ensure the partial or full recovery of costs associated with service delivery.
The main mechanism through which NRC IRAP engages SMEs is through advisory services. Skilled ITAs manage relationships with firms and provide business and technical advice as well as critical referrals to R&D and business/innovation expertise. All firms that engage with the program receive advisory services but not all are subsequently funded.
International co-innovation support
NRC IRAP funds collaborative industrial research and innovation projects, with high potential for commercialization, between Canadian SMEs and international counterparts and/or partners. This is achieved through NRC IRAP's partnerships with innovation networks, primarily in Europe, Asia and (to a lesser extent) South America.
Minimum eligibility for NRC IRAP support
Firms must be incorporated, operating in Canada, have 500 or fewer employees and have the objective to grow through innovation.
Linkages to Canadian expertise
NRC IRAP enhances linkages between SMEs and R&D services and expertise. Through programs such as Certificate (on hiatus), NRC Researcher Engagement and Contributions to Organizations (CTO) eligible firms are provided access to unique research facilities as well as to skilled experts.
NRC IRAP provides support through numerous other temporary and ongoing initiatives. The most relevant for the present evaluation include:
- the Youth Employment Program (YEP) through which SMEs receive funding to hire qualified post-secondary graduates as interns
- the global talent stream and dedicated service channel through which firms are supported to access qualified international talent
- the Accelerated Growth Service (AGS) through which SMEs are supported to access key government services needed to expand their businesses
Source(s): Document review and data review
The federal government continued to increase its investments in NRC IRAP over the evaluation period. Budget 2021 proposed $500 million over five years, starting in FY 2021-22, to expand NRC IRAP support to more Canadian businesses. This was in addition to the $700M already committed as part of Budget 2018. As a result, NRC IRAP distributed at least $431 million more in contribution funding to Canadian SMEs and organizations in the present evaluation period than it did previously.
NRC IRAP disbursed more than $1.3 billion to clients between 2018 and 2022.
At the time of reporting, the total financial support provided by the program over the evaluation period was more than $1.3 billion. However, the actual total is expected to exceed $1.8 billion once disbursements slated for the third and fourth quarters of FY 2021-22 are tallied. Importantly, this figure does not include about $450 million in contribution funding that NRC IRAP distributed to clients through other non-core programming delivered during the same timeframe (e.g. through the Innovation Assistance Program).
Figure 2. Total expenditures increased year over year and are expected to almost double in FY 2021-22, but operating expenditures did not increase in the same proportion as Grants and Contributions (Gs&Cs).
Note: Based on budgeted amounts – reporting on 2021-22 actuals had not been completed.
Figure 2 – Text version
A horizontal line chart representing the NRC IRAP total expenditures by fiscal year, broken into Grants and Contributions, and operating expenditures between the years FY 2017-18 and FY 2021-22. Over the 5 year span, total expenditures increased from $277M to $686M (+148%), with grants and contributions increasing from $216M to $582M (+169%), and operating expenditures increasing from $61M to $104M (+71%).
Figure 3. Costs recovered through white label services remained relatively constant throughout the evaluation period
Note: Based on budgeted amounts – reporting on 2021-22 actuals had not been completed.
Figure 3 – Text version
A horizontal line chart with a single line representing actual costs recovered through white label services for the NRC IRAP. The costs recovered remained relatively constant over the 5 year span, going from $2.1M in FY 2017-18 to $1.5M in FY 2021-22. FY 2021-22 is based on budgeted amounts because reporting on actuals had not been completed at the time of the evaluation.
Source(s): Document and data review
NRC IRAP staff are classified into three main categories: management, administrative, and technical
There are a total of 454 full-time staff presently employed by the program. The majority (~70%) of the workforce occupy technical roles, while about 20% are in administrative positions and less than 10% are in management roles. This overall distribution remained relatively stable throughout the evaluation period. Technical positions are classified as Research Council Officers (RCO) and include ITAs and Client Engagement Advisors (CEAs) among other policy/program advisory roles. Administrative or corporate positions include Regional Contribution Agreement Officers (RCAOs) and other administrative support staff. Management includes the VP, DGs and Directors.
The NRC IRAP workforce grew; however, the number of full-time staff fell below program expectations
In response to program delivery expansions proposed as part of Budget 2018, the NRC IRAP committed to increasing its workforce by 99 full-time-equivalents (FTEs), including 51 ITAs. The program planned to have 50% of these staff in place by FY 2018-19 while the balance were to be engaged by the end of FY 2019-20. During the evaluation period, the overall number of FTEs increased from 382 at the beginning of FY 2017-18 to 454 at the end of FY 2020-21 (latest available data). However, this meant that only 72 of the targeted 99 full-time positions were filled. ITAs made up only 19 of the 72 new FTE positions compared to the target of 51.
Figure 4. Overall NRC IRAP's workforce representation has not yet met labour market availability for staff from historically underrepresented groups
Note: Based on March 2021 data (latest available). Data on labour market availability derived from the 2016 Census and 2017 Canadian Survey on Disability.
Figure 4 – Text version
An overlapping horizontal bar chart showcasing workforce representation compared to labour market availability from historically underrepresented groups for the NRC IRAP. The groups represented include Women, Visible minorities, Persons with disabilities, and Indigenous persons. The graph indicates that the NRC IRAP has not yet met labour market availability for staff from the aforementioned groups (Women: -9%; Visible minorities: -5%; Persons with disabilities: -7%), with the exception being Indigenous peoples who account for 2% labour market availability.
Figure 5. Women occupy fewer management and scientific and technical roles (RCO); however, the overall number of women in these roles increased slightly over the evaluation period.
Note: The data is based on active indeterminate employees and employees with terms of three months or more. It does not include students, employees on secondment from other organizations or visiting workers.
Figure 5 – Text version
A horizontal line chart representing the change in the amount of women at the NRC IRAP who occupy management, and scientific and technical roles. The figure indicates that women occupy fewer positions than their counterparts, although the overall number of women in these roles increased from 30% (management) and 22% (scientific and technical) in FY 2017-18 to 41% (management) and 27% (scientific and technical) in FY 2021-22.
Source(s): Document review, interviews and data review
NRC IRAP and its clients spent two years, during the five year evaluation period, navigating the COVID-19 pandemic
As with most of the world, NRC IRAP staff, clients and partners were operating within a context that was challenging and highly uncertain. SMEs have been particularly vulnerable to the impacts of the pandemic, as they have been affected by issues such as staff cutbacks, supply chain and market disruptions, cancelled contracts and risk aversion from investors. Program staff played pivotal roles in the implementation of several federal government pandemic-related temporary programs and services, and provided over 3,000 pandemic specific advisory services to SMEs within the first year alone. In addition, the program delivered the (now closed) emergency wage subsidy Innovation Assistance Program (IAP) and the conduct of technical assessments for Innovative Solutions Canada (ISC) COVID-19 Testing and Challenge streams. These activities added to staff workloads and affected regular program delivery.
The pandemic-induced switch to a remote work environment brought with it some challenges
For SME clients, challenges included difficulties accessing innovation testing facilities and expertise, work stoppages particularly on international collaboration projects and reduced availability of funding for R&D in most sectors. For staff, adverse impacts included challenges with engaging new clients (See figure 18 in Appendix B) and monitoring project activities. At the onset of the pandemic, the program was reportedly able to leverage social capital that had been established through years of relationship building to execute these tasks. However, staff indicate that, as the pandemic progresses, this critical social capital is being rapidly erased by unprecedented levels of attrition within key stakeholder organizations.
The program is in the early stages of its GBA Plus integration journey
The program has been making deliberate strides towards the full integration of GBA Plus considerations into its operations for several years. During the evaluation period, its efforts have been primarily in alignment with the federal Women Entrepreneurship Strategy outlined in Budget 2018. The universal goal of the strategy is to improve the participation of underrepresented groups, including women entrepreneurs, in the innovation economy. In addition to implementing an inclusiveness strategy to support achievement of this goal, NRC IRAP committed to collecting more robust client equity, diversity and inclusion (EDI) data, and establishing a baseline for the participation of women and other underrepresented groups in the program by the end of FY 2019-20.
Client EDI data collection activities were launched recently and are still ongoing
NRC IRAP launched client-related EDI data collection activities in FY 2020-21. Data were actively being collected during the conduct of the evaluation and at the time of reporting were available for 47%, or 3,688 out of 7,915 clients that were funded in the evaluation period. The program did not have data available for unfunded clients. This is due to the fact that the current phase of data collection is focused on gathering data through the program portal to which, for the most part, only presently funded clients have access. Furthermore, EDI data were being provided by clients on a voluntary basis. Consequently, the evaluation was only able to partially assess program performance on GBA Plus based on available information.
NRC IRAP is reaching high-potential and innovative clients as intended. Client selection processes meet the needs of the program. The introduction of Client Engagement Advisors (CEAs) and the continued implementation of the portfolio model positively impacted the quality of clients reached during the evaluation period. The program placed more youth, and specifically more female youth, than ever before on client projects. The reach of business and technical assessment services provided by ITAs to government partners also continued to expand, as did the volume of assessments conducted. Overall, NRC IRAP reached more SME clients than it did in the previous evaluation period. Strategies to identify and reach clients from historically underrepresented groups are still in the process of being operationalized. The program is making strides towards collecting more robust EDI data on funded clients; however, the data are currently not available for the majority of unfunded clients. As a result, gender-based analysis of the program's reach could not be fully assessed as part of the evaluation.
Reach among intended clients increased
During the evaluation period, the program continued to reach an increasing number of innovative and high-potential SME clients. The overall reach (i.e. unique firms receiving advisory services and/or funding) of NRC IRAP increased. Annual targets for total reach were met in every single fiscal year (note that FY 2021-22 data were not completely recorded at the time of reporting).
Source(s): Document and data review
The total program reach increased between the two evaluation periods
The total reach, as measured by the total number of unique firms receiving support from NRC IRAP (advisory services and/or funding), increased by about 20% from 18,732 to 22,458 between the two evaluation periods.
The sector distribution of clients were similar between the evaluation periods of FY 2018-2022 and FY 2013-2017.
|Main sectors||% 2018-2022||% point change from 2013-2017|
|ICT (Digital Technology)||29%||+6%|
|Manufacturing and Materials||10%||-2%|
|Agriculture and Food||8%||0%|
|Construction and Related Products/Services||7%||-1%|
|Energy and Resources||7%||0%|
|Health and Life Sciences||7%||0%|
Annual targets for total reach were achieved
In FY 2017-18, the program set an annual reach target of 8,000 SMEs. This was revised to 10,000 SMEs per year as part of program commitments following Budget 2021 and a target achievement date of March 31, 2022 was established. The program exceeded annual targets except for FY 2021-22.
Figure 6. NRC IRAP exceeded its total client reach targets, except for its newly set target in FY 2021-22
Clients who received advisory services (Including IAP support) and funding.
Source(s): Departmental Results Reports (FY 2017-18 to FY 2020-21), unpublished data for FY 2021-22
Figure 6 – Text version
An overlapping vertical bar chart representing total reach targets of clients who received advisory services (Including IAP support) and funding from FY 2017-18 to FY 2021-22. The total client reach targets were exceeded from FY 2017-18 to FY 2020-21, but fell short by 922 its newly set target in FY 2021-22.
More clients received financial support
The number of funded clients increased slightly since the last evaluation. Annual reach remained relatively stable over the current evaluation period but increased in FY 2021-22. At the time of analysis, the program target for FY 2021-22 had been met.
Source(s): Document and data review
The number of funded firms increased slightly since the last evaluation
The vast majority of clients during the evaluation period received funding for regular IRAP, Youth and ARP projects. Only 40 clients in the evaluation period (compared to the targeted 44 per year) had an LVC project. However, it should be noted that NRC IRAP made a decision to delay the ramp up of the LVC program in FY 2020-21 due to the pandemic.
Since the previous evaluation, the total number of unique funded clients increased by 4%, from 7,456 (during the FY 2013-2017 period) to 7,717 (during the FY 2018-2022 period). In addition, the average annual number funded firms increased by 2% between periods, from 3,295 firms per year on to 3,362 firms per year.
The number of unique funded firms refers to the number of individual clients (with unique identification numbers) that are funded within a given timeframe. If a client were to be funded for a project duration of more than a year, the client would be counted as an individual client in more than one year. However, that client would only be counted once in an assessment of data for the entire evaluation period.
The FY 2021-22 program target for funded clients was met
An annual reach target of 3,500 funded firms was established in FY 2020-21 for achievement by the end of FY 2021-22. At the time of analysis, the program had exceeded its target, having recorded a reach of 3,657 funded firms. Prior to this, a target was only set for overall reach (i.e. unfunded/advisory service only and funded clients).
Figure 7. At the time of reporting, NRC IRAP slightly exceeded its newly set target for funded firms for FY 2021-22
Source(s): Departmental Results Reports (FY 2017-18 to FY 2020-21), unpublished data for FY 2021-22
Figure 7 – Text version
A vertical bar chart representing the number of funded firms by the NRC IRAP from FY 2017-18 to FY 2021-22. Yearly targets were not set for FY 2017-18 to FY 2020-21. A target of 3,500 funded firms was set for FY 2021-22, which the NRC IRAP exceeded by 157 funded firms.
Improved reach of underrepresented client groups
Early insights from program client EDI data indicate that NRC IRAP funded a higher proportion of businesses owned by diverse Canadians than general SME ownership trends. Some suggested improvements related to the implementation of the program's EDI strategy emerged as part of the evaluation.
Source(s): Document and data review
NRC IRAP funded proportionally more SMEs owned by individuals from historically underrepresented groups than were operating in the Canadian economy
NRC IRAP data indicate that, among the clients funded during the evaluation period, 28% are owned or led by women, another approximately 23% are owned or led by individuals identifying as Visible minorities, about 2% by Indigenous peoples and about 1% by Persons with disabilities.
Available national data (i.e. 2017 Statistics Canada information) on business ownership specifically, indicate that NRC IRAP's client population has a higher proportional representation of underrepresented groups than the general SME population in Canada (See Figure 9).
Staff are uncertain about the implementation of the EDI strategies
More than half of NRC IRAP staff, who were consulted as part of the evaluation, expressed a desire to continue to improve program outreach among underrepresented groups. However, there was a genuine lack of awareness about how they could operationalize the program inclusiveness strategy in the execution of their duties. Staff were nonetheless able to propose some recommendations for consideration by management:
- expand the number and diversity of ITAs in order to improve the diversity of reach – present workforce representation for women and Visible minorities is below labour market availability (See Figure 17 in Appendix B);
- collect client EDI data more routinely; and
- design and market more programs specifically targeted to underrepresented groups to improve appeal/awareness of the program.
Figure 8. The proportion of business owners from underrepresented groups among NRC IRAP clients is higher than that among all Canadian SMEs
Note: Latest available data on SMEs ownership trends in Canada (Statistics Canada).
Figure 8 – Text version
An overlapping horizontal bar chart representing the proportion of business owners that are from underrepresented groups among NRC IRAP clients. The proportion of business owners from underrepresented groups among NRC IRAP clients was found to be higher than that among all Canadian SMEs: Women (+2.0%), Visible minorities (+5.3%), Indigenous peoples (+0.2%), and Persons with disabilities (+0.3%).
NRC IRAP recorded more unfunded clients
There was an increase in the number of documented unfunded clients between the two evaluation periods. A spike observed in the number of unfunded clients receiving advisory services in FY 2020-21 was due to a surge in SME requests for support after the launch of the Innovation Assistance Program (IAP).
Source(s): Data review, interviews
The number of unfunded firms recorded by the program increased between the two evaluation periods
Unfunded clients are typically SMEs that are not yet ready to implement an NRC IRAP-funded project but have received, or are in the process of receiving advisory services from ITAs or other third-party advisory service providers engaged by the program.
Since the last evaluation, the documented number of unique unfunded firms that had received advisory services increased by 21%, from 11,276 to 13,699.
Firms and types of advice are concentrated in specific areas
The majority of unfunded firms were from the Ontario, Quebec and Pacific regions (where the majority of potential clients are located) and from the ICT (digital economy), manufacturing & materials, and agriculture & food sectors. Technical advice was the top service provided followed by technical information and technical partnership. Business information was the top type of business advice followed by business referrals and business partnership advice.
Figure 9. Despite the spike in FY 2020-21, the annual number of unfunded clients remained relatively stable during the evaluation period
Note: The high number in FY 2020-21 is attributable to a surge in SME requests for support from the program after the launch of the IAP, a COVID response program.
Figure 9 – Text version
A horizontal line chart representing the annual number of unfunded NRC IRAP clients from FY 2017-18 to FY 2021-22. The annual number of unfunded clients remained relatively stable during the 5 year span, with the exception of a spike of 6,294 unfunded clients in FY 2020-21.
Youth reach increased
NRC IRAP provided more youth, and specifically more female youth, than ever before, with job placements within SME clients' projects. The program also reached more youth who identify as Visible minorities, Indigenous peoples and Persons with disabilities.
Source(s): Data and document review
Youth reach almost doubled between the two evaluation periods
Reflecting the increase in available funding, the number of youth participating in the YEP increased from 4,878 in the previous evaluation period to about 7,232, representing about a 50% increase in the number of unique youth participating in the program.
Most youth participants came from Ontario, Quebec and Pacific. The vast majority of youth placements were in the ICT sector, followed by manufacturing and materials, and energy and resources.
The program is reaching more female youth
In response to federal government commitments to support the career advancement of women in STEM, NRC IRAP increased the proportion of female youth engaged in the program over the last decade. The growth in number of female youth reached between FY 2013-14 (earliest available data) and FY 2021-22 was about 262% while that of male youth was about 57%.
The proportion of youth in the program who are female increased compared to the previous evaluation period (38% in FY 2018-2022 vs 32% in FY 2014-2017). During the evaluation period, there was also a steady increase in the proportion of female youth annually engaged by the program from 31% in FY 2017-18 to 54% in FY 2021-22.
Figure 10. More youth from underrepresented groups were reached during 2018-2021 compared to 2014-2017
Note: EDI data were being provided on a voluntary basis.
Figure 10 – Text version
A horizontal segmented line chart representing the amount of youth from underrepresented groups who took part in the Youth Employment Program, part of the NRC IRAP. The proportion of youth were compared from the two 4-year periods, 2014-2017 and 2018-2021. The figure shows that there were increases in every underrepresented youth group, with the largest increases in proportion coming from Women (+6%) and Visible minorities (other than Indigenous peoples]) (+5%).
More other government departments were supported
During the evaluation period, the program continued to serve an increasing number of other government department (OGD) clients through white label services.
Source(s): Document review
NRC IRAP has supported more other government departments
The number of partner programs that have been seeking NRC IRAP support for technical assessments has been increasing gradually year over year. To date, a total of 30+ partner programs have benefited from NRC IRAP support. These have included programs run by OGDs, provincial governments, regional development agencies and non-profits. Since the last evaluation period, the number of active other government department clients has increased from 8 in FY 2017-18 to 21 in FY 2021-22.
The number of technical assessments conducted has also increased
The number of partners seeking support has increased, as has the volume of assessments being conducted. Between FY 2012-13 and FY 2021-22, more than 9,700 assessments were conducted. The average annual number of assessments completed more than doubled in the current evaluation period (i.e. 1,402 per year) compared to the previous evaluation period (i.e. 547 per year).
In FY 2020-21 the number of assessments spiked to 2,730 in response to an increase in demand for support from OGDs running pandemic related programs. This was about 1,300 more than the highest number of assessments the program ever previously recorded (i.e. 1,394 in FY 2017-18).
Figure 11. The number of technical assessments completed for OGDs increased between 2013-2017 to 2018-2021
Note: FY 2022 was not completed at the time of analysis.
Figure 11 – Text version
A horizontal line chart representing the amount of technical assessments performed by the NRC IRAP for other government departments (OGDs) from FY 2012-13 to FY 2021-22. There have been significantly more technical assessments performed during 2018-21 period than there was from 2013-17 period, with the largest increase being from FY 2019-20 (769) to FY 2020-21 (2,730). FY 2021-22 was not completed at the time of analysis with 966 assessments completed for 21 different active OGD partners.
Improved quality of program reach
The NRC IRAP client selection processes meet the needs of the program. The introduction of the CEA role was found to have made many positive contributions to the quality of program reach. The portfolio model has helped improve the program's focus on high potential firms.
Source(s): Interviews, data and document review
Client engagement advisors (CEAs) have had a positive impact on the quality of clients entering the program
Overall perceptions of CEAs are positive. CEAs are noted to have helped improve the program's reach by contributing to focusing resources on high potential firms. CEAs seek out and screen SME clients before passing leads on to ITAs. In doing so, they reportedly play an instrumental role in easing ITA workloads, allowing ITAs to continue working on delivering the program to existing clients while simultaneously engaging with additional innovative and high-potential clients.
The portfolio model contributes to the expansion of program reach to higher growth clients
The majority of ITAs indicated that the portfolio model contributes to focusing resources on high-potential firms. Once client leads have been referred to ITAs, they, along with their regional directors, work together to identify high-potential portfolio clients, a process which can involve multiple years of relationship building, advising, funding and meeting milestones. The selection process under the portfolio model is more refined and interviewees state that it works well to identify high-potential clients. This is also evidenced by the 96% increase in the proportion of firms who were considered to be high-potential clients over the course of the two evaluation periods (i.e. 15% in FY 2012-13 to >40% in FY 2021-22).
The portfolio model
The previous evaluation recommended that NRC IRAP develop a targeted strategy to identify new, high-potential clients. In response, the program made several adaptations including but not limited to continuing to scale up the implementation of its portfolio management approach.
First introduced in FY 2014, the portfolio model provides a framework for ITAs to focus the majority of their time and allotted financial resources on SME clients who are deemed through a discretionary process to be high-potential.
Efforts to diversify program reach improved
The program supported efforts to improve the diversity of its reach over time. The SONAR4d pilot and contributions to organizations (CTOs) were perceived to be successful and to show some promise of supporting the program to continue to diversify its reach. However, some opportunities to further improve exist.
Source(s): Document review, data review, case studies and interviewees
The SONAR4d pilot yielded positive results for the diversification of program reach
During the evaluation period, NRC IRAP piloted a new platform, known as SONAR4d, that combines data from multiple organizational sources into one interface to help ITAs and CEAs better identify, research and select SME clients. Though still limited in terms of deployment in Quebec and Ontario, the SONAR4d pilot was perceived to be successful at diversifying program reach. The platform has reduced the time and effort needed to identify potential clients of all profiles. It therefore has the potential to be another valuable pipeline of new client leads in an ITA's toolbox. There are currently plans in place to expand the use of the platform to other regions. However, some staff thought that the feasibility of these plans may be hindered by the considerable variations in provincial rules governing the sharing of data between government agencies, something essential to the success of SONAR4d.
The portfolio model may not be compatible with efforts to expand program reach to new clients
Although the majority of ITAs indicated that the portfolio model contributed to focusing resources on high-potential firms, some found the program's approach to implementing the model problematic. This is because, in their view, the approach ascribes higher value to proven high impact firms over potential high impact firms, thus inadvertently nudging them towards making more conservative client selections. Some ITA perceptions that their annual performance reviews were linked to the performance of their portfolio clients were thought by some in senior management to exacerbate this issue. Program data appeared to support the prevalence of these perceptions, as the number of new funded clients fluctuated during the evaluation period (see figure 18 in Appendix B).
Contributions to Organizations (CTO) Case Study
Efforts to increase the diversity of program reach
The Canadian Executive Service Organization's (CESO) mission is to empower people and drive economic development through sharing expertise. NRC IRAP first funded CESO in 2020 for a pilot project to support Indigenous businesses and communities. Participating SMEs received 50 hours of mentorship mainly for guidance on how to develop their innovative products with coaching on developing strategic, operational and HR plans, and other early stage needs. CESO stated that NRC IRAP support is:
- integral as it allows the organization to continue offering its mentorship services
- a critical success factor for Indigenous SMEs as it helps improve general capacities and accelerates the development of their innovation projects
CESO support is expected to increase the proportion of Indigenous businesses supported by NRC IRAP and to ultimately lead to significant impacts for jobs, revenue and wealth creation within Indigenous communities.
"We felt so good about going this path with CESO which ended with tangible documents. We have no certainty about the future success of our business, but we feel at least that we have the right approach and tools to go forward"
Challenges impacting program reach
NRC IRAP's overall performance as it relates to reach was assessed to be good, as demonstrated by the evidence presented thus far. Nonetheless, staff reported areas for improvement and provided suggestions for the program to maintain or improve upon results achieved.
Source(s): Interviews, data review, document review
Staff involved in outreach activities feel under-resourced
Staff perceived that their abilities to effectively reach more clients were impacted by a lack of sufficient resources (human, technical and financial). In urban centers, the sheer volume of clients seeking support make it difficult for small teams to properly execute their duties. On the other hand, staff in less-densely populated areas reportedly have to develop outreach strategies in order to reach new and diverse clients. However, they cited a lack resources to address multiple government priorities, and a potential opportunity for the program to review resource allocations to ensure the right balance is being achieved based on geography and the types of outreach services required.
Staff lack some information required to make decisions
The program made great strides over the evaluation period to improve performance monitoring of reach data. However, there was still some opportunity for further improvements, particularly related to how performance monitoring data are communicated to staff with decision-making authority. For example, although some initial reach targets were determined, they were not systematically included in monthly reports, annual datasheets and operational plans used by regional staff to plan their outreach activities. In addition, although data quality has been improving, a large proportion of unfunded client sector data are still unknown (e.g. about 30% are tagged as "N/A" or "Other"). This limits the program's ability to accurately track the sector distribution of its reach of unfunded clients and thus make better informed decisions about resource allocations.
Some aspects of client selection processes hinder reach expansion
As previously mentioned, NRC IRAP made many improvements to the client selection process over the course of the evaluation period. However, the selection process, for large value contribution (LVC) clients in particular, was still perceived by most interviewees to be onerous and a hindrance to the expansion of reach among that segment of clients. Although senior staff indicate that the process is presently not any more bureaucratic than similar government programs, there may be an opportunity to further review and refine project development and approval processes.
Interviewees additionally reported that regular program rules around client eligibility can hinder expanded reach within Indigenous communities. This is because many Indigenous SMEs tend to use a different legal structure from that which government typically deems as eligible for funding. Even though this was viewed as a barrier that was beyond the entire control of the program, interviewees still thought there was an opportunity for the program to push to evolve the rules.
NRC IRAP has been successful in supporting innovation and sustainable economic growth within Canada. The program supported SME clients to consistently achieve the majority of their project objectives and register marked growth post program engagement. The resulting growth has facilitated the expansion of supply chains, supported the creation of jobs and the generation of new capital within and beyond Canada. NRC IRAP clients also made contributions to Canadian efforts to combat the COVID-19 pandemic. The program contributed to this by providing advisory support and funding to assist clients in the development of new innovations, scaling up production of existing products and the adaptation of facilities to meet pandemic-related needs. The program has stimulated growth in the economy and supported the creation of wealth in Canada.
Increased SME access to an ecosystem of support
SME clients reported that engaging with the program helped de-risk their innovation projects, making their innovations more attractive to other key players in the ecosystem of support. This, in turn, facilitated client access to critical external financing, strategic advice/connections and testing facilities that supported improved outcomes.
Source(s): Document review, case studies and interviewees
NRC IRAP facilitated increased access of SMEs to an ecosystem of support
In addition to providing support through its regular program, NRC IRAP expanded its service offerings to include large value contributions (LVC) and more international opportunities. SMEs were also supported by ITAs to access external funding (private, OGD and international), make connections with strategic partners and test their innovations. This support was made available through NRC research centres, contributions to organizations (CTO), the accelerated growth service (AGS), IRAP International and other relationships that the program had established with key actors in the Canadian and global innovation support ecosystem.
The complementarity of NRC IRAP service offerings with externally available support makes the program more effective
NRC IRAP was perceived by clients to be instrumental in supporting the timely achievement of their respective innovation objectives. Importantly, working with the program de-risked SME clients, making them more attractive to other funders and innovation service providers. Service offerings were reportedly complementary to available external support, which together generally provided the robust and all-encompassing support that firms needed to be successful.
Contributions to Organizations (CTO) Case Study
NRC IRAP supported SMEs to access a network of innovation expertise and technology testing facilities
Tech-Access Canada has a network of 60 Technology Access Centres (TACs) across Canada and has been funded by NRC IRAP since 2016. The funding gives SMEs 20 hours of interactive visits which could include access to equipment, facilities, expertise or advice. The visits focus on getting to know the company, scoping their challenges, demystifying the innovation ecosystem, validating a technical solution they are exploring, giving them objective advice or making suggestions for future activities.
The result of a visit is often a plan that describes next steps which might be another project with a TAC or something the SME does on their own. For SMEs that are at a very early stage, these visits are low risk with a lot of potential. They help them move towards their goals and ultimately create a wider pool of SMEs who might be ready for direct IRAP support.
Enhanced client innovation capacity and productivity
NRC IRAP contributes to enhanced innovation capacity and productivity among Canadian businesses. The full spectrum of support provided by the program consistently allows SME clients to achieve the majority of their objectives and puts them on a better path towards growth, commercialization, increased revenue and profits.
Source(s): Interviews, data and document review
More R&D clients than targeted derive positive benefits by working with NRC IRAP
The proportion of surveyed clients who reported an increase in jobs, sales, R&D expenditures or other positive benefits as a result of services received increased slightly. According to data reported by NRC IRAP, in the FY 2017-18 through to FY 2020-21 NRC Departmental Results Reports, on average, 89% of all R&D clients reported some positive benefits of working with the program. This is compared to an average of 82% reported between FY 2013-14 (earliest available data) and FY 2016-17, and the NRC target of 86% for the current evaluation period.
Clients value the linkages the program provides to NRC experts and facilities
Clients indicate that NRC researcher interactions and the certificate program, in addition to NRC IRAP funding and advisory services, were critical contributors to improvements in SMEs' innovation capacities and the commercialization of their ideas. The certificate program was a pilot and the program did not obtain funds beyond the pilot phase. However, the closure of the certificate program was a source of disappointment for clients and NRC researchers alike. It was perceived to have offered great value to all SMEs, but particularly to early stage firms, on their journeys toward increased innovation capacity, productivity and profitability.
Figure 12. The proportion of NRC IRAP funded clients who reported benefits from the program increased from 2013-2017 to 2018-2021
Note: Based on data categories "Agree and/or strongly agree" of clients who reported benefits from the program.
Source(s): Post project reports
Figure 12 – Text version
A figure representing the change in proportion of benefits reported by clients funded by the NRC IRAP between two five periods: 2013-2017 and 2018-2021. The figure shows that there was a significant increase in all impact categories, which include: Increased technical capability (+40%), Increased competitiveness (+39%), Enhanced ability to conduct R&D (+39%), Increased productivity (+30%), Improved network (+29%) and Increased business capability (+28%).
Increased business growth
NRC IRAP accelerated growth among innovative SMEs during the evaluation period. According to data submitted to NRC IRAP by previously funded clients, participation in the program enabled supported firms to not only increase their annual revenues, but also the size of their overall workforce. Program business growth targets were exceeded.
Source(s): Data and document review
The program supported SMEs to grow; certain profiles of firms grew at a faster rate than others
NRC IRAP clients exceeded compound annual growth rate (CAGR) targets for both revenue and employment for the evaluation period. Portfolio clients registered higher growth than non-portfolio clients (i.e. 36% versus 25% for revenue growth and 15% versus 11% for employment growth). Although there was no difference in revenue growth rates, NRC IRAP International program clients grew slightly more than regular program clients in terms of employment (i.e. 17% versus 14%).
Firms in the health and life sciences sector grew the most in both revenue (65% CAGR) and employment (22% CAGR). LVC clients also registered high growth rates for both revenue (45%) and employment (21%). However, the LVC clients sample size was small (~25).
Figure 13. Compound annual growth rate (CAGR) of firms in revenue and employment exceeded targets for the evaluation period
Note: Revenue n=1,660; Employment n=1,760; CAGR of >500% were excluded).
Figure 13 – Text version
An overlapping horizontal bar chart representing the compound annual growth rate of firms in revenue and employment of NRC IRAP clients for the evaluation period. The figure shows that the compound annual growth rate target for both revenue and employment were exceeded for the evaluation period by 12% and 3% respectively.
Early insights from available EDI data on funded clients indicated that firms owned or led by individuals from under-represented groups grew at a similar or better rate to that of the overall client population
At the time of reporting, EDI data were available for about half (47%) of firms funded in the evaluation period. Despite this, it was possible for NRC IRAP to provide some early insights into program impacts on growth rates of firms led/owned by individuals from historically under-represented groups. It was found that FY 2020-21 revenue and employment CAGRs for firms led or owned by women (i.e. 36% and 35% respectively) and Visible minorities (i.e. 49% and 30% respectively) were greater than those of the general NRC IRAP client population in the same year (32% and 20% respectively). Due to the relatively small absolute number firms owned and/or led by individuals identifying as Indigenous peoples and/or Persons with disabilities, initial CAGR calculations do not offer the same statistical significance and so are not presented.
Enhanced creation of wealth through innovation
The program has made contributions to the creation of wealth in Canada. NRC IRAP-supported clients have been able to grow rapidly and in so doing, support the development of new supply chains that feed further innovation, the creation jobs and the generation of capital within the Canadian economy. The impacts of the program are perceived by stakeholders to be largely sustainable.
Source(s): Interviews, data and document review
The program is making strong contributions to the sustainable creation of wealth in Canada
Stakeholders perceive the program to be making strong contributions to the sustainable creation of wealth in Canada. NRC IRAP is viewed as an accelerator that helps catalyze the growth of young companies. In so doing, the program supports the creation and expansion of new supply chains that feed SME innovation and facilitate wider benefits for other actors operating within the Canadian economy.
The majority of NRC IRAP funded SME clients are profitable following project completion
NRC IRAP clients are required to provide the program with snapshot data (including profit and loss information) for five years after the completion of their funded project. Each fiscal year since FY 2014-15, a larger proportion of NRC IRAP funded SME clients reported making a profit. During the evaluation period, the percentage of clients reporting a profit each fiscal year increased from 48% in FY 2017-18 to 64% in FY 2020-21 (latest available data).
NRC IRAP supported about 62,000 jobs during the evaluation period
Program funding was used to support 62,076 high quality jobs within client firms during the evaluation period. This included job placements for 7,232 youth. At the time of reporting, 5,823 of the youth placements had been completed and 4,121 (i.e. 71%) of youth continued to be employed post program engagement. Overall, the program supported 9,512 more jobs than in the previous evaluation period when a total of 52,564 jobs were recorded.
Figure 14. NRC IRAP exceeded its annual jobs supported targets throughout the evaluation period
Figure 14 – Text version
A vertical overlapping bar chart representing the amount of jobs supported as well as the target amount of jobs supported by the NRC IRAP from FY 2017-18 to FY 2021-22. The figure shows that the targets were exceeded each year. In FY 2021-22, 16,160 jobs were supported exceeding the target (9,539 jobs) by 6,621 jobs.
Impacts on the Canadian economy are positive
Results from the Partial benefit cost analysis (PBCA) indicate that the NRC IRAP continued to have net positive impacts on the Canadian economy. The net benefits produced by the program far exceed the program costs.
Source(s): PBCA study, interviews, data review, survey, focus group discussions
The net benefit of the program to the Canadian economy is greater than the program costs
PBCA net present values and benefit/cost ratios have increased from the FY 2016-17 PBCA (even if 2016 dollars are inflated to 2021).The results of the FY 2021-22 PBCA show that even under the most conservative assumptions and highest discount rate, the net benefits produced by the NRC IRAP in the Canadian economy are 2.8 times greater than the program costs. This is a slight improvement over the FY 2016-17 PBCA results that showed net benefits were about 2.5 times greater than program costs.
- FY 2021-22 PBCA net present value: $16 billion
- FY 2021-22 PBCA benefit-cost ratio: 5.34 : 1
Under more "reasonable" assumptions (i.e. taking the average value of the lower and upper bounds, at a 5% discount rate) the net benefit of NRC IRAP to the Canadian economy is calculated to be 5.3 times or at least $16 billion greater than the program's costs. This is higher, in contrast to the findings of the FY 2016-17 PBCA that showed that net benefits of the program to the economy were 4.9 times or at least $10 billion higher than the program's cost.
Current NRC IRAP clients are more high-growth and innovative than they were in FY 2016-17
The positive results demonstrate that the strength of NRC IRAP client selection processes, funding decisions, advisory services and general support continue to be critically important to the viability and success of clients and the program as a whole. In fact, IRAP is working with more high-growth, innovative firms since FY 2016-17:
- three firms in FY 2016-17 with lower bound net benefits >$1B; in FY 2021-22 there are five
- two firms in FY 2016-17 with lower bound net benefits <$1B but >$500M; in FY 2021-22 there are seven
- funding and ongoing ITA advice is considered critical to getting to market faster, de-risking research, increasing innovation capacity, and keeping businesses in Canada
PBCA study considerations
The FY 2021-22 PBCA was based on an assessment of 17 NRC IRAP SME clients that were determined to be "big winners" of the program for the present evaluation period. Importantly, this included at least two "unicorn" companies that NRC IRAP had supported from the beginning. This is the same methodology to the one that was applied during the previous evaluation (i.e. FY 2016-17 PBCA).
Benefits to the Canadian economy improved
Source(s): PBCA study
FY 2021-22 PBCA Results
Presented in 2021 dollars
Lower bound ($millions)
Upper bound ($millions)
|Discount rate||2%||5%||7%||Average of 5% discounts||2%||5%||7%|
|Net present value (benefits-minus-costs)||10,537||8,451||7,239||16,142||28,847||23,833||21,040|
FY 2016-17 PBCA Results
Presented in 2016 dollars
Lower bound ($millions)
Upper bound ($millions)
|Discount rate||2%||5%||7%||Average of 5% discounts||2%||5%||7%|
|Net present value (benefits-minus-costs)||6,618||5,245||4,425||10,153||18,529||15,061||13,107|
Cost/benefit analysis: case study examples
Source(s): PBCA study
Kraken Robotics is a marine technology company engaged in the design, development and marketing of advanced sonar and laser sensors and systems and subsea power solutions for Unmanned Underwater Vehicles for military and commercial applications. NRC IRAP supported the firm from a pre-revenue start up to becoming a publicly listed company.
Arctic Spas manufactures hot tubs engineered for harsh winter climates. Their products are manufactured in Canada and sold worldwide. They have developed a niche due to their products' energy efficiency, as well as advanced technological applications. NRC IRAP projects have resulted in practical applications, increased employment and sales.
Questrade is an independent, online investment brokerage that offers Canadian citizens an alternative to trading and investing with big banks. Trading for a variety of asset classes can be done via a desktop or mobile app. NRC IRAP funding has supported the development of technology. The most recent project, starting in 2020, has helped with the development of an online mortgage platform. The firm was also part of the previous PBCA study in FY 2016-17.
Pavemetrics specializes in 3D vision systems for the automated inspection of transportation infrastructure. Their systems are used in 45 countries around the world to automatically inspect roads, runways, railways and tunnels. IRAP projects have helped the company in its product research and development.
Innovative Automation designs and develops automated assembly machines for the automotive industry, consumer goods and medical devices. IRAP funding contributed to product development in addition to services provided, increasing firm revenues and creating jobs. A patent application was filed for its RoboTape automatic tape system.
Semios helps growers use technology to sustainably produce more food. Through their farming platform, they provide crop management tools for specialty crops. Using a wireless network, machine learning and big data analytics, they help growers manage pests and disease, and weather-related risks in real-time. Semios recently acquired Agworld, Altrac and Centricity to further its commitment to simplifying the growers' experience. IRAP projects have supported research and development.
Advisory services are as impactful as financial support
Source(s): Advisory services client journey mapping
Building relationships leads to multiple impacts
The subject of this journey mapping exercise has been in business for over 9 years and sells an environmentally friendly culvert system. With the aid of an ITA, the firm made a connection with the local college and completed a collaborative applied research project.
The project validated the firm's technology and filled an information gap on impacts to fish habitat and behaviours at stream crossings. As a result, sales increased with installations made across Canada and the United States. A demonstration project organized with the City of Lethbridge showcased the technology's environmental benefits and cost-saving features. The city was able to purchase the culvert with considerable savings and install it with no streambed disruption. In addition to preserving the habitat quality for fish, the research project also kick-started the growth of the applied research program at the college, which now ranks on Canada's Top 50 research colleges, and has led to an ongoing relationship between the college and the City of Lethbridge.
The firm maintained their relationship with the ITA who introduced them to at least five other organizations, and continued to provide advice and referrals for business and technical information. One referral led to a distribution agreement in Ontario and Quebec, and the culvert system was sold for use in a wind farm, in an agricultural application, and in the forest industry.
- ITA connections
- technology validated
- increased commercial success
- preserved habitat quality
- ITA referrals
- continued growth
Clients are supporting the fight against COVID-19
NRC IRAP supported SME clients made contributions to Canadian efforts to combat the COVID-19 pandemic. The program provided advisory support and funding to assist clients in the development of new innovations, scaling-up production of existing products and the adaptation of facilities to meet pandemic-related needs.
Source(s): Document review and interviews
NRC IRAP is supporting the development of COVID-19 vaccines and therapeutics
The program has provided advisory services and more than $37 million (and plans to provide an additional $113 million over the next 3 years) in funding to support 13 clients working towards research and development of made-in-Canada vaccines and therapeutics for the COVID-19 pandemic.
When the R&D phase projects are completed, NRC IRAP will invite the companies who received funding to submit proposals to undertake a second round of projects to advance the most promising candidates to the next phase of clinical development.
NRC IRAP funded clients used their facilities and technologies to combat the pandemic
During the pandemic, NRC IRAP actively supported and assisted firms in the development of innovations needed in the fight against COVID-19.
As a result, multiple clients supported by the program were able to contribute to national efforts to combat the pandemic by producing personal protective equipment (PPE), disinfection and sanitization solutions and medical innovations for improved diagnostics, testing and treatment of COVID-19. In several cases, NRC IRAP clients scaled up production of existing products and/or adapted them to meet pandemic-related needs.
Advisory services helped clients fight against COVID-19
Source(s): Advisory services client journey mapping
Canada-made personal protective equipment (PPE)
The subject of this advisory services journey mapping exercise is a start-up that wanted to contribute to Canada's pandemic response. The firm's journey with the program started when they were referred to NRC IRAP by a contact. Once connected to the program, the firm worked with NRC IRAP to navigate regulatory hurdles in the development of medical devices.
With the advice and help from two ITAs, the firm achieved an ISO 13485 certification for quality management of medical devices. Upon receiving further advice from an ITA on how to navigate regulatory requirements, Health Canada approved the firm's product: the first N95 respirator manufactured in Canada. This was followed by approval from the National Institute of Occupational Health and Safety (NIOHS), which allowed the firm to expand their market into the United States. The firm continues to work with IRAP and is exploring further development and expansion of their products. They credit ITA advice for speeding up the commercialization process.
- ITA advice
- ISO certification
- product approved
- expanding market
- positive future outlook
Opportunities to maximize outcomes exist
The program was assessed to be performing well on objectives related to improving client innovation capacity, growth, productivity and profitability. To maintain momentum and encourage continuous improvement, the evaluation identified ways for NRC IRAP to further maximize results.
Source(s): Case studies, interviews, international comparison study
Strengthen connections within the ecosystem to more efficiently bridge the gap between R&D and commercialization
Clients and staff perceived the program to be more geared towards supporting firms in the early stages of their R&D journey. Some clients indicated that there was an opportunity for the program to improve support (including access to a network of supports) for SMEs seeking to bridge the gap between the prototyping and commercialization phases of their innovation processes. For example, it was suggested that the program encourage ITA engagement with broad sector partners (e.g. accelerator and incubator organizations) to optimize development of SMEs. In their view, this would help ensure their long-term success.
Although the program is already well-connected within the Canadian and international innovation support ecosystem, some additional lessons could be learned from international comparator programs which rely on their respective ecosystems to bridge this gap. Business Finland leverages services through their active participation in the Team Finland network advisory services to provide clients with advice on commercialization (e.g. concrete market entry). Following the provision of R&D financial support and information, Vinnova connects their clients with Business Sweden, whose strengths are to connect Swedish companies with international collaborators or investors, especially companies at the commercialization phase.
Engage more organizations to support service delivery
Stakeholders suggested that the NRC IRAP increase its outreach to CTO clients to ensure SME clients can be holistically supported to innovate/evolve their innovations. Case studies confirmed that CTOs supported by NRC IRAP are offering an important suite of services that are filling a gap for many SME clients by, for example, assisting SMEs with small projects, helping scope or solve a technical or business challenge and/or making suggestions for next steps. Many SMEs do not have the scientific expertise nor the larger scale processing equipment or pilot scale infrastructure needed to advance.
Supported CTOs included in the case studies deliver a wide range of services, from technical (e.g. access to technical expertise, mentorship, testing/trials, proofs of concept and access to equipment) to business supports (e.g. business planning, strategic planning, human resources management, work organization), and are typically oversubscribed.
To improve NRC IRAP's impact, organizations could be engaged to specifically focus on SMEs in sectors that require more support or that strongly support Canada's economic priorities (e.g. the environment), and better program promotion with underrepresented groups.
NRC IRAP has evolved over the past 75 years into a key player in the Canadian ecosystem for innovation support. The program is designed and delivered in a manner that is consistent with international comparator programs, and generally meets the needs of intended clients. The vast majority of clients are satisfied with their interactions with the program and with the services that are provided. However, some adjustments could be made to improve overall efficiency and to ensure that the program meets the specific needs of diverse groups.
NRC IRAP is effectively designed to meet client needs
NRC IRAP is flexible, agile and well-designed to meet the needs of intended clients. The effectiveness of the design is demonstrated by the program's ability to rapidly respond to evolving government priorities, and introduce new services and programs as client needs arise. It was able to continue to serve the needs of its clients during the COVID-19 pandemic while simultaneously launching and supporting new temporary pandemic-related programs.
Source(s): Interviews and document reviews, international comparison study
NRC IRAP is flexible, agile and capable of rapidly responding to the changing needs of clients
Staff indicate that the design of the program is fit-for-purpose (i.e. capable of meeting objectives and service standards). In their view, NRC IRAP was agile and generally responsive to the changing needs of clients throughout the evaluation period. This was particularly the case during the pandemic when the program was able to stand-up a fully operational temporary emergency wage subsidy program within a week of its announcement. In addition, the program conducted technical assessments for the ISC COVID-19 Testing and Challenge temporary program streams.
Subject expert teams that were set up in response to the pandemic were an important program change, and served the needs of clients well. These teams allowed the program to more efficiently distribute information on pandemic-related opportunities to clients, and to effectively reserve funds for projects that were deemed to be urgent in the context of the pandemic, all while simultaneously delivering the regular program.
Over the last five years, NRC IRAP effectively designed several new sub-programs in response to client needs, as identified in federal budget announcements. These included two new international programs, a Youth Green program, the AGS, the Certificate program, LVC and the emergency IAP. The program also provided OGDs with administrative support, business and technical assessment services and support in the design and launch of new programming (e.g. ISED with SIF, ISC Challenges and Superclusters, CanExport). It continues to be a key platform for the federal government to launch business innovation services and programs.
Client-centric adjustments were made to the design of the program
The NRC IRAP project development process is generally perceived by staff and clients to be effective and efficient (at least compared to other government processes). Improvements made to the core program over the evaluation period include:
- the introduction of new sub-programs (as previously mentioned) to respond to clients' needs and Government priorities;
- the establishment of the client enablement team to meet evolving program delivery requirements, including EDI and GBA Plus commitments;
- an ever-expanding and better-evolved suite of white label services;
- a streamlined project review and approval process (including a revised RCAO checklist); and
- refinements to the IRAP Innovation Portal (IIP) to, among other objectives, allow clients to have a better experience when developing and submitting project proposals.
NRC IRAP, like similar international programs, relies on in-house and third-party expertise to provide advice to funded and unfunded clients. This allows the programs to reach more clients and provide more specialized advisory services.
Clients are satisfied with program delivery
The NRC IRAP has generally been delivered in an efficient and effective manner. The vast majority of clients are satisfied with their interactions with the program and with the services that are provided.
Source(s): Interviews, data and document reviews
Clients are satisfied with NRC IRAP service delivery
Clients are satisfied with services they receive from NRC IRAP. Most clients were satisfied with the knowledge and professionalism of staff, and the overall level of support rendered.
- 91% of client firms had positive sentiments about the program
- 88% of CTO clients had positive sentiments about the services they received
Clients identifying as belonging to a historically under-represented group (i.e. women, Visible minorities, Indigenous peoples and Persons with disabilities) are just as satisfied (~92%) with their interactions with the program, and would recommend it to their professional networks.
"They have filled an important need for us. I am an engineer and neither I nor my partner are business people."
Technical assessment services are delivered in an effective manner
A review of white label services (specifically, the technical assessments) found that OGD clients perceive delivery to be in accordance with service standards outlined in their respective memoranda of understanding.
NRC IRAP was adept at communicating any significant changes in agreed upon service standards. OGDs reported being highly satisfied (90-95% of the time) with the overall technical assessment process and general quality of service received from NRC IRAP.
Reasons provided for high levels of satisfaction included the:
- NRC IRAP Partnership and Assessment team's professionalism
- effectiveness of communication
- flexibility of services offered
- program's commitment to honoring service standards
- unmatched combination of technical and business expertise possessed by the ITAs
Some services are not well integrated into program
The level of integration of various service offerings into the core IRAP varies. Overall levels of awareness of some funding streams and non-core program elements is relatively low. Although progress has been made during the evaluation period to better integrate program elements such as the Large Value Contributions (LVC) and NRC IRAP International, there is opportunity for improvement.
Source(s): Interviews, international comparison study
ARP and Youth components are perceived to be more integrated than LVC and International programs
The integration of various components of the program into core IRAP has been improving over time. The ARP and Youth funding streams were considered to be the best integrated within the core program. The roll-out of the LVC funding stream has been progressing well and its integration has improved, although challenges in ensuring eligible clients' progression from regular IRAP to LVC projects persist.
IRAP International was viewed as the most challenging sub-program to integrate because of the fundamental differences in the delivery mechanisms (i.e. an ITA discretionary delivery model vs. a traditional call for proposals). Some ITAs within the regular program do not have positive views on the international project development process, and this hampers willingness to participate and integration with core programming.
The program introduced a three-year rotation program for regular ITAs to serve as International ITAs in an effort to increase buy-in for the program. Other solutions to improve integration are still needed but may be limited by the large variety of international partners and distinct processes that would need to be considered.
Stakeholders' levels of awareness of NRC IRAP services varies
External stakeholders had mixed levels of awareness of various components of the program (based on a small sample size ~30). Awareness of main funding streams (ARP, Youth, IRAP), advisory services, Certificate and researcher interaction programs and the ISC challenges were generally high among all stakeholders. Fewer stakeholders were familiar with IRAP International, LVC and talent acquisition programs i.e. Dedicated Service Channel; Global Talent Stream.
Similarly to NRC IRAP, international comparator programs have made efforts to offer a full range of support to SME clients (e.g. financial management and action plan advice, internationalization support), and these are well integrated into their core programs. Comparator programs routinely work with other organizations/service providers to ensure client access to the full spectrum of business support services. In general, support from outside organizations tends to centre on internationalization. Like NRC IRAP, some comparator organizations face challenges in fully integrating specialized support elements of their programs.
Staff job satisfaction is high but some HR issues persist
Overall, staff are satisfied with their jobs and are proud to be affiliated with the NRC IRAP. However, some segments of staff are reportedly stretched thin by a consistent mismatch between workloads and available human resources. This situation was exacerbated during the pandemic when the NRC IRAP staff implemented or supported multiple temporary programs and served an unprecedented number of clients all while continuing to deliver the regular program.
Source(s): Interviews, data and document reviews
NRC IRAP staff are generally satisfied with their jobs
Responses to the public service engagement survey (PSES) during the evaluation period indicate that staff at NRC IRAP are at least as satisfied or more satisfied with their jobs as staff in the broader NRC. According to the latest available information from FY 2020-21, about 90% indicated that they liked their job overall. No PSES was conducted in FY 2021-22.
Some segments of staff are overextended
Although data show improvements since FY 2017-18, the FY 2020-21 PSES results still showed NRC IRAP staff were stressed by heavy workloads (51%) and a lack of enough resources on their team (38%). Similarly, the vast majority of internal interviewees, consulted as part of the evaluation, stated that staff are stretched too thin. RCAO, CEA and ITA positions were viewed to be the most understaffed. Staff with these positions were perceived to be particularly vulnerable to burnout as the program continues to grow.
Due to an increasing number of sub-programs and perceived complexities in processes, many staff reported a capacity to address only immediate needs, with little or no time to address clients' strategic needs. They described being overwhelmed by pandemic-related tasks that disproportionately impacted their abilities to efficiently deliver the regular program. Although staff viewed, with a sense of pride, the delivery of IAP as an important way for them to contribute to the combat of the pandemic, they also cited it as a significant contributor to their elevated stress levels during the short timeframe under which it had to be delivered.
HR challenges make the program delivery model difficult to effectively scale-up
As with many other government units, the program consistently faces challenges in the recruitment, hiring and retention of staff and, in particular, qualified ITAs. Recruitment and hiring are impacted by long hiring cycles (~150 days) and salary budget constraints (- viewed by some as below market value). Meanwhile, retention is impacted by significant numbers of retirements. Qualified ITAs are often highly skilled technical professionals in the later stage of their careers.
Figure 15. More than a third of the NRC IRAP RCO-level staff will be eligible to retire by 2027, 10% more than the NRC overall
Note: Projections based on data as of 28 February 2022.
Figure 15 – Text version
A projected line chart of the Industrial Research Assistance Program Research Council Officer (RCO) level staff's eligibility to retire by 2027. The figure compares NRC IRAP RCO staff with the NRC staff that will have the same eligibility to retire by 2027, finding that 37% of NRC IRAP RCO level staff will be eligible to retire by 2027, compared to the NRC average of 27%.
Program delivery tools are being improved
Program management and implementation tools generally serve the needs of the program well. However, there are some opportunities for improvement to ensure the overall effectiveness and efficiency of all tools employed. NRC IRAP has begun to make some of these changes.
Source(s): Interviews and document review
Performance measurement tools and approaches are consistent with international good practice
NRC IRAP's performance measurement framework is for the most part aligned with international comparator programs. Indicators tracked (employment, R&D investments, growth) are similar. Like NRC IRAP, comparator programs align performance indicators with United Nations Sustainable Development Goals to assess sustainability of impacts on the wider economy. Periodic follow-ups with clients to assess durability of business are also conducted. NRC IRAP is currently collecting EDI data in accordance with its Inclusiveness Strategy.
However, there is future opportunity for the program to revisit its FY 2017-18 plans to systematically collect voluntary EDI data at the onset of their engagements with clients thus increasing the likelihood that data are available for more clients. Similar practices have been successfully adopted by international comparator programs.
Tools are more consistently adapted
Although the consistent and timely adaptation of program delivery tools has improved over time, there continue to be opportunities for more adjustment. For instance, the vast majority of staff interviewed stated that, despite efforts made to regularly update the field manual (a critical delivery tool), it still did not adequately reflect the numerous evolutions that the program has undergone.
Digital tools being employed are adequate for program needs; however, staff view some as inefficient
Most digital tools being utilized for program delivery are perceived to be adequate for the purpose for which they are employed. SONAR is the exception; the majority of staff interviewed stated that the tool is not ideal for all program delivery and impact assessment needs. SONAR continues to be the primary data repository for field staff, and persistent issues with data extraction and quality make it a challenge to work with for administrative staff. SONAR improvements are underway; however, communications to staff regarding the benefits of the revamped SONAR 360 system could be improved.
SAP Business Objects reportedly works well for NRC IRAP's core analytics team and has enabled the group to produce regular performance reports to support management decisions. However, the tool is also perceived by some staff to be an obstacle to good performance monitoring and reporting. The tool is reportedly clunky, not user-friendly and requires significant training to master. Power BI has been adopted as a replacement tool by the analytics team, but has not as yet been rolled out throughout the program.
Overall, the primary challenge with digital tools stems from the perception among staff that some tools currently in use across the program are not well-suited for the tasks to which they are applied. This can make performance reporting at different levels of the program (e.g. within regional and partnership and assessment teams) a laborious undertaking as data sometimes needs to be extracted from different systems in order to produce a simple data report.
Opportunities exist to improve program efficiency
As noted, the program is generally effectively designed and delivered. However, some areas for improvement exist.
Source(s): Document review, interviews, survey, focus group discussions
Communication with staff on their involvement in program design
Many improvements made to the program were supported by the Program Delivery Advisory Committee (PDAC). Senior staff indicate that the PDAC has been revamped to include a wider cross-section of representatives based on their geographic location, program experience and subject matter expertise. However, the vast majority of staff consulted were not aware of the existence of the PDAC. Furthermore, staff stated that there was an opportunity for their operational knowledge of the program to be better leveraged by senior management in design discussions. This suggests a need to better communicate to staff about the role of the PDAC in supporting program improvements, and how they might be able to contribute to related decision-making processes.
Program delivery tools
As previously noted, program delivery tools generally serve their intended purpose. However, consultations with staff also revealed some inefficiencies. This is specifically the case for tools that support program operations and performance reporting. Staff also identified a need for management to ensure that they have access to required tools and resources to allow the program to continue to effectively serve clients even as pandemic-related restrictions on travel and physical distancing persist. There is therefore an opportunity for the program to review operational tools such as the program field manual and digital platforms and ensure they are enhanced to meet the needs of users.
Implementation of the portfolio approach
Although the portfolio model was found to be effective at achieving its objective of expanding program reach among higher growth clients, the manner with which it was implemented introduced other issues. Staff consequently had mixed views about whether newly refined program delivery mechanisms (i.e. related to the portfolio model) better served the needs of clients. On the one hand, ITAs reported that portfolios allow them to focus resources on high potential firms, customize the time they spend with high-potential clients and find ways to push companies along their growth journey. Some though, were critical of the approach, categorizing it as bureaucratic, adding an administrative burden every few months and restricting their ability to focus on other categories of clients such as those belonging to EDI groups or aligned with other government priorities such as climate change. Only 46% of ITAs surveyed stated that implementing the portfolio approach helped increase the efficiency of their work.
Project development and approval processes
Although staff and clients perceive project development and approval processes to be generally effective, they indicate that opportunities to create more efficiencies exist. Potential improvements include:
- shortening the length of the processes (particularly for elements of the International program that are within NRC IRAP control)
- ensuring consistency in updating staff guidance (particularly for ITAs) in the field manual and operational policy documents
- further refining the contents of the RCAO checklist to improve clarity and applicability for all staff who need to refer to it
Human resource challenges
The program is actively working to address the human resource issues identified earlier by exploring more creative ways to recruit and retain qualified technical staff. These ways include:
- leveraging professional platforms like LinkedIn to proactively engage potential candidates, including those from diverse backgrounds
- enhancing the level of flexibility in retirement options for technical staff. For example, some retired ITAs are offered the opportunity to support OGD technical assessments on a contractual basis
However, there is still room to explore more creative options. One such option which may warrant consideration is engaging technical staff from within the NRC to support existing ITAs in the execution of some of their duties. Some NRC staff, particularly researchers and NRC IRAP CEAs, expressed interest in taking on tasks traditionally assigned to the ITA role. In their view, they have the industry experience and technical expertise required to be successful. Leveraging the support of other NRC staff would help NRC IRAP reduce the workloads of ITAs and allow them to focus their efforts on addressing clients' strategic needs.
Another potential solution is to review decision-making processes around the allocation of tasks to staff. Interviewees suggested that such a review begin with the premise that the program will always be slightly understaffed, and this should factor into any plans to take on large projects. The ultimate aim would be to ensure a better match between the volume of tasks assigned and the available resources.
White label services
The idea of white label services was still evolving as the evaluation was being conducted. As a result, not all staff were aware of what (beyond the conduct of technical assessments) the services entail. With respect to technical assessments, client perceptions were mostly positive, as mentioned earlier.
However, there are still some opportunities for improvement in efficiency (including data collection, ITA matching to and training on assessments, appropriateness of tools, etc.). Inefficiencies were exacerbated by a lack of formal standard operating procedures to guide the technical assessment process. This made it challenging, but not entirely impossible for ITAs to provide consistent assessments on OGD program client bids.
There is an opportunity for NRC IRAP to play a more active role in ensuring that some formal standard operating procedures are put in place to enhance OGD clients' individual processes. Clients interviewed were open to receiving this sort of guidance from the program.
NRC IRAP is overall well aligned with Government of Canada priorities focused on supporting innovation. The program specifically aligns with the Government's Inclusive Innovation Agenda and the six areas identified for action. During the evaluation period, NRC IRAP proved to be a key player in Canada's innovation ecosystem, and remained uniquely positioned to support government efforts to achieve its vision of building Canada as a global centre for innovation. Program progress towards full alignment with Government priorities around GBA Plus has been positive. The program continues to evolve from data collection to the actual use of data for program management. As such, there are opportunities for continuous refinement over the next few years. There is a shared perception among clients and stakeholders that NRC IRAP is unique among Government innovation support programs in the combination and quality of services that it offers. The program has been increasingly more relevant as it fills a significant gap in the innovation support ecosystem for small and medium-sized enterprises as well as other government departments.
Entrepreneurship and science excellence supported
During the evaluation period, the program supported government efforts to foster innovation focused entrepreneurship among Canadians and enhance global science excellence. This was achieved through the delivery of the regular program and connections to innovation expertise and facilities made to SMEs.
Source(s): Document reviews, interviews
NRC IRAP supported Government efforts to foster an entrepreneurial and creative society
The program aligns with government priorities aimed at fostering economic growth through SME innovation. NRC IRAP ITAs are unique in the Canadian innovation support ecosystem in their ability to provide both technical and business advisory services to program clients. The long term ITA-client relationship was perceived to be integral to encouraging and supporting Canadian entrepreneurs, allowing them to adopt and maintain a mindset that fosters creativity, risk-taking and ambition.
NRC IRAP supported government efforts to enhance global science excellence
The program facilitated SME client access to world class scientific expertise and infrastructure. Through the Contribution to Organizations (CTOs), researcher interaction and certificate programs, NRC IRAP provided clients an opportunity to experiment, explore and strengthen their basic and applied research capabilities. In this way, the program promoted pathways to support the transformation of new ideas into practical solutions Canada could leverage to support economic growth.
Inclusive Innovation Agenda
In Budget 2016, the Government defined a new vision to build Canada as a global centre for innovation. To achieve this vision, the Government launched an Inclusive Innovation Agenda that identified six areas for action:
- fostering an entrepreneurial and creative society
- supporting global science excellence
- building world-leading clusters of innovation
- growing companies and accelerating clean growth
- competing in a digital world
- increasing the ease of doing business with Canada
Business innovation and competition supported
Over the evaluation period, NRC IRAP aligned its operational plans with government efforts to increase innovation clusters, the ease of doing business with Canada and SME competition in digital spaces.
Source(s): Document review
NRC IRAP supported Government efforts to build world-leading clusters of innovation
In alignment with the Innovation Agenda's call for more innovation-focused partnerships among businesses, research institutes and governments, NRC IRAP partnered with other government departments (OGDs) to create the Accelerated Growth Service (AGS). Through the AGS, the program was able to identify more than 200 high potential firms and support their growth. In addition, NRC IRAP has entered into memoranda of understanding with OGDs and not-for-profit organizations to support their respective SME innovation programs by contributing to program design and conducting technical assessments.
NRC IRAP supported Government efforts to increase the ease of doing business with Canada
Through its core and international programs, NRC IRAP provided an avenue for SME clients to co-innovate with foreign partners, commercialize their ideas and gain access to global markets/contribute to global value chains. As an example, during the evaluation period, NRC IRAP continued to invest in the Eureka program (a network of 40+ partners' economies, primarily in Europe) and the Canada International Innovation Program (CIIP) in Brazil, India, China and South Korea. The program additionally engaged with two new programs in Europe (i.e. Eureka Eurostars and 2+2 Germany) to support market oriented R&D collaboration. Consequently, the number of unique NRC IRAP clients engaging in the international program increased by 210% from 123 in the previous evaluation period to 381 in the current period. The number of international projects between the two periods also increased substantially (by 292%) from 158 to 620.
NRC IRAP supported Government efforts to improve Canada's ability to compete in a digital world
During the evaluation period, the program supported Canadian SME clients through advisory services and funding to leverage the benefits of current and emerging digital technologies for their products. This helped firms produce more innovative products, offer customers more choice and increase their competitive edge both nationally and internationally.
This support builds upon the 2011-2014 NRC IRAP delivery of the Digital Technology Adoption Pilot Program (DTAPP). The objective of DTAPP was to accelerate the adoption of digital technologies in SMEs operating in Canada so they can boost their productivity and create economic growth and opportunity.
Business innovation and competition supported
Through the delivery of its regular program, NRC IRAP supported government efforts to enhance growth among Canadian companies. In addition, the program accelerated clean growth by making funding contributions towards clean technology projects. NRC's sustainable development strategy clean technology targets were met and exceeded.
Source(s): Document and data review
NRC IRAP supported Government efforts to grow companies
During the evaluation period, NRC IRAP continued to focus on supporting SMEs to start-up and scale-up in alignment with its mandate and the Government's Innovation Agenda. The program ensured that clients were supported with funding and advice through the regular IRAP, Youth employment program (YEP), Accelerated Review Process (ARP) and Large Value Contributions (LVC) newly introduced to encourage transformational growth.
NRC IRAP supported Government efforts to accelerate clean growth
NRC IRAP also contributed to the acceleration of clean growth by performing well against clean technology targets set out in the NRC Departmental Sustainable Development Strategy. The program spent between $15M and $73M per year on 194 to 412 clean tech projects, compared to the targeted $15M for 125 projects per fiscal year.
Out of 14,499 projects funded in the evaluation period, 1,579 or 11% were clean technology projects, and roughly 10% of clients had a clean technology project. The top categories were emissions control, clean energy and smart grid.
Figure 16. NRC IRAP exceeded its targets on total number of clean tech projects and contributions between FY 2017-18 and FY 2020-21
Project types included: NRC IRAP, Youth, ARP, CIIP and LVC
Targets: NRC's Departmental Sustainable Development Strategy 2020 to 2023
Figure 16 – Text version
Two vertical bar charts representing the number of clean tech projects and contributions made by the NRC IRAP from FY 2017-18 to FY 2020-21. The NRC IRAP exceeded its targets on total number of both clean tech projects and financial contributions to clean tech projects during the 5 year period.
Increased relevance and alignment with GBA Plus
During the evaluation period, NRC IRAP proved to be a key and increasingly relevant player in Canada's innovation ecosystem. The program remained uniquely positioned to support government efforts to achieve success in all six areas of focus identified in the Innovation Agenda. Program progress towards full alignment with the Government's GBA Plus priority is positive, although it could be better communicated to staff.
Source(s): Document review and interviews
NRC IRAP became an increasingly relevant player in the Canadian ecosystem for innovation support
As previously mentioned, in FY 2017-18 the program became one of four flagship platforms through which the federal government is streamlining the delivery of innovation programming across Canada. During the evaluation period, the Government dedicated increasingly more financial resources to support NRC IRAP programs.
In Budget 2021, the federal government committed an additional $500M to the program to be rolled out over the next five years in equal installments of $100M per annum. In addition, over the course of the last five years, an increasing number of OGD partners have recognized value in and have sought out the expertise of NRC IRAP staff. The program provided OGD programs with back-end administrative support (e.g. Can Export) as well as business and technical assessments. As a result, the program was able to increase costs recovered each fiscal year.
GBA Plus commitments
In the years leading up to the evaluation period, the government renewed its commitment to supporting the full implementation of GBA Plus across federal departments and agencies. This included a mandate to ensure that the development of policies, programs and legislation is sensitive to the different impacts that decisions can have on diverse groups.
NRC IRAP made progress towards alignment with GBA Plus priority
NRC IRAP has taken steps towards greater equity, diversity, and inclusion (EDI) among staff and clients served. The establishment of the Client Enablement team in FY 2019-20 has been one of the most critical steps. The program has implemented some initiatives to increase staff diversity, ensuring that it at least meets labour market availability, and to begin the process of systematically collecting critical EDI data on clients.
However, EDI information is not yet available on the majority of unfunded clients and about half of the funded clients during the present evaluation period.
Importantly, most staff interviewed (including senior NRC IRAP officials) were not aware of the EDI client-related initiatives that were being integrated or considered for integration into programming.
There is, therefore, an opportunity for the program to better communicate about the progress made in monitoring client EDI information (as well as actual EDI data) and how the GBA Plus lens will be integrated in decision-making and the development of programming.
The program meets a need in the ecosystem
NRC IRAP is unique among government innovation support programs and the combination of services offered meet a need in the ecosystem of innovation support for SME and OGD clients. Services offered by funded organizations are considered to be valuable and, in some cases, crucial to the enhancement of SMEs' innovation capacity.
Source(s): Interviews, case studies
NRC IRAP is unique among government innovation support programs and meets a need in the ecosystem for SME and OGD clients
Stakeholders consulted indicated that NRC IRAP meets a need in the ecosystem for innovation support. The program is considered to be unique because it operates at the national level, is sector agnostic, and offers advisory services combined with direct funding. Unlike other programs, NRC IRAP targets SMEs of all sizes and sectors, and offers non-repayable contribution funding rather than repayable loans. The program also has access to a team of highly technical- and business-savvy ITAs that are able to maintain long-term relationships with SMEs and provide them with invaluable advice.
The program also meets a need in the ecosystem for innovation support services required by other government departments. It does this by continuing to conduct technical assessments and by contributing to new program design. OGDs state that they would have a difficult time finding an alternative service provider if NRC IRAP no longer provided technical assessments. Affiliation with ITAs, who are respected within industry, is seen to add a level of rigour/credibility to partners' processes that can stand up to intense public scrutiny around the disbursement of large-value awards for innovation.
Services offered by organizations are perceived as valuable by SME clients
NRC IRAP contributions to organizations (CTOs) facilitate access to an important suite of services that fill a gap for many SME clients. Importantly, CTO services are complementary to the support provided by ITAs. CTOs assist SMEs with small projects, providing them with highly specialised support, helping scope or solve a technical or business challenge, and/or making suggestions for next steps.
Many SMEs do not have the scientific expertise on staff nor the larger-scale processing equipment or pilot-scale infrastructure needed to advance their innovations. For them, the relationship with organizations is critical.
"We went to market sooner and were able to hire people thanks to the project."
Recommendations and supporting rationale
NRC IRAP should explore alternative solutions to tackle current human resource challenges, including options for improving surge capacities, reviewing hiring and promotion practices and better aligning work planning with available resources. Issues particularly affecting staff in the ITA, CEA and RCAO roles should be prioritized.
The program did not meet its hiring target for ITAs (i.e. its primary workforce) during the reporting period. Hiring has been and will continue to be a challenge due to the hard-to-fill nature of the ITA position that typically requires highly experienced technically savvy and late-career industry professionals. This constitutes a significant risk for the program, as HR data show that almost 40% of technical staff, of which ITAs make up the majority, will be eligible for retirement by 2027. This would be 10% more than the NRC overall.
Furthermore, the latest PSES results show that NRC IRAP staff were stressed by heavy workloads (51%) and not enough resources on their teams (38%). The vast majority of internal interviewees, consulted as part of the evaluation, also stated that staff are stretched too thin. The imbalance in staff workload was particularly amplified during the implementation of COVID-19 related temporary programs (e.g. the IAP). RCAO, CEA and ITA positions are viewed to be the most understaffed and vulnerable to burnout.
NRC IRAP should further refine project development and approval processes and tools, particularly those related to LVC and IRAP International, in order to enhance overall service delivery and the integration of these components in regular programming.
Since the last evaluation, the program took concrete steps to streamline and refine project development and approval processes. Staff with authority on the subject indicated that processes were presently more efficient but required some further improvements to become optimal. Opportunities were identified to make further adjustments to the RCAO checklist to ensure its applicability to all (non-RCAO) staff that need to refer to it and to reduce the level of bureaucracy embedded specifically in LVC and IRAP INT project development and approval processes.
Program data show that the originally established target for LVC projects (i.e. 44 per year) was not met, as only 40 projects in total were funded over the evaluation period. Consequently, not all committed funds were spent. The NRC IRAP management decision to slow implementation of LVC during the first year of the pandemic was a contributing factor. However, stringent processes around project development and approvals as well as the program's lack of success in meeting related hiring goals for additional ITAs are perceived to be the primary cause of this situation. With respect to IRAP INT, inefficiencies related to process have deterred ITAs from pursuing international opportunities for their clients, an issue that was later confirmed by IRAP INT staff consulted.
GBA Plus integration
NRC IRAP should ensure that its existing inclusiveness strategy and initiatives are better communicated to and operationalized by staff in order to ensure enhanced integration of GBA Plus considerations into all aspects of the program from planning to execution and reporting.
Additionally, client EDI data collection need to continue to be improved, and clear EDI key performance indicators (KPIs) established to guide staff in their outreach activities. The effect(s) of program design and delivery (e.g. portfolio approach, reach, eligibility criteria) on clients of different backgrounds should also be monitored using a GBA Plus lens, and the findings integrated into program management decisions.
The program recently made progress in the collection of client EDI data and several initiatives are currently underway to meet evolving program delivery needs related to GBA Plus commitments. However, there is a need to leverage this momentum to better communicate and integrate GBA Plus considerations into program management and operations.
Staff reported a genuine lack of awareness about ongoing EDI initiatives and how they might be able to operationalize a program's inclusiveness strategy in the execution of their duties. In addition, they do not currently have clear information on program EDI objectives and/or KPIs that they should be working towards. Staff are critical of the portfolio approach, categorizing it as bureaucratic, administratively burdensome and restricting to their ability to focus on other categories of clients including those belonging to underrepresented groups. Interviewees additionally reported that rules around client eligibility can hinder expanded reach within Indigenous communities. This is because many Indigenous SMEs tend to use a different legal structure from that which government typically deems as eligible for funding. There is a risk that these issues could negatively impact client access to and public perceptions of the program.
In order for GBA Plus to be fully integrated into the program, staff need to be supported to understand the hidden barriers faced by diverse clients seeking access to NRC IRAP services. International comparator programs are establishing direct relationships with underrepresented groups. In so doing, they are obtaining guidance related to program design and support in the integration of feedback loops in overall program management processes. This has permitted the development of programs specifically targeted at underrepresented groups. For instance, Innovate UK, an international comparator program, supported in-depth research, outlining barriers, challenges, opportunities and support needs for Visible minority groups and Persons with disabilities to participate in business innovation. Since then, the program has continuously engaged with special interest groups to better understand barriers and support evidence-based solutions in increasing their reach. Given that NRC IRAP has identified a need to expand its reach beyond previously engaged clients, the program may benefit from adopting a similar evidence-based approach to identifying barriers limiting the greater participation of underrepresented groups in its program. Addressing ongoing EDI issues and sustaining solutions in the long-term may also necessitate that NRC IRAP embed regular dialogues with underrepresented groups in program management processes.
NRC IRAP should strengthen its performance measurement system and further use data analytics to inform program management processes by:
- developing enhanced and formally documented performance measurement guidance accessible to management and staff that clearly defines/outlines standard operating procedures for performance monitoring and measurement
- using an advanced and integrated interactive software platform accessible to management and staff that allow for specific data analytics and effective reporting in the context of progress towards or on key performance indicator (KPI) targets
- developing specific indicators to monitor the effectiveness of program design and delivery (e.g. selection of high-potential portfolio firms versus actual performance against KPIs such as business growth and jobs supported)
The program has existing processes and tools to govern performance measurement. However, these are not always formally documented or clearly outlined in a single repository. This poses a risk to the program in the event of employee attrition or departures. There is specifically a potential for corporate knowledge loss on mechanisms for the development, update and deployment of performance monitoring and measurement tools, the frequency of data collection, data quality assurance, data analysis processes and procedures for internal and external results reporting.
In addition to introducing improved reporting tools (annual datasheets and monthly management reports), the program initiated the development of some integrated data spreadsheets (Excel) to annually compile data on KPIs. More recently, the program also developed an Excel-based management toolkit to monitor progress towards KPI targets. However, there is an opportunity to leverage the new management toolkit to streamline data compilation efforts as well as the number of tools being used to communicate results. While the program has already taken steps to adopt more advanced integrated software (e.g. Power BI), there is an additional opportunity to make use of interactive features of this software to convert elements of the management toolkit into a dashboard. This would make performance data more accessible (to the extent that is appropriate) to operational and management staff based on their varying levels of need for the information (e.g. operations, decision-making and/or reporting).
Additionally, KPIs and targets exist for some core program components but not others. Results reporting, particularly to senior management, is not always framed in the context of objectives and targets that were formally set by the program or envisioned for specific program components. It is good performance monitoring practice to track all indicators in the context of a target and, in cases where none is available, such rationale should be clearly documented within tracking tools. Consequently, there are opportunities to adopt more indicators (for internal tracking purposes) and further leverage data available for existing KPIs to test critical program assumptions (e.g. effectiveness of the portfolio model at reaching and supporting high-potential firms) and inform decision-making.
Management Response and Action Plan
|Recommendation||Risk‑level||Management response||Measure of achievements||Proposed person(s) responsible||Expected date of completion|
|1. NRC IRAP should explore alternative solutions to tackle current human resource challenges, including options for improving surge capacities, reviewing hiring and promotion practices and better aligning work planning with available resources. Issues particularly affecting staff in the ITA, CEA and RCAO roles should be prioritized.||High||
Background: In order to address the challenges of recruiting ITA candidates, since 2019 IRAP has been engaged in direct and targeted recruiting of HQP, with 2 staff members devoted to this activity. In FY 2021, that initiative resulted in 18 hires, including 11 in EDI categories. IRAP has also added incremental staffing in multiple areas, including regional RCAOs to address administrative workload. IRAP has also increased HR support by adding additional Human Resources Generalists and Hiring Specialists. Lastly, in an effort to identify efficiencies and maximize efforts, IRAP launched the Design for Growth organizational review initiative.
Action 1: Engage in a comprehensive review of ITA duties, exploring methods of augmenting ITA access to specialized expertise, and refocusing core duties to those which directly leverage ITA key competencies.
Action 2: Complete "Design for Growth" project and its comprehensive analysis of centralized, national and corporate office functions, with the goal of eliminating duplication of efforts, streamlining processes, and introducing automation where practical.
Action 3: Collaborate with Human Resources Branch on an analysis of HR resource requirements needed to shorten the hiring cycle and optimize onboarding and orientation, and shorten the overall timelines to employee operational effectiveness.
Action 3: Analysis completed and appropriate incremental resources allocated
Action 1: DG Division Services
Action 2: DG Division Services, DG Client Enablement, ED International Innovation Office, IRAP VPO
Action 3: DG Division Services
Action 1: September 2024
Action 2: June 2024 (and ongoing)
Action 3: October 2024
2. NRC IRAP should further refine project development and approval processes and tools, particularly those related to LVC and IRAP International, in order to enhance overall service delivery and the integration of these components in regular programming.
Background: IRAP has already made improvements to refine LVC processes and enhance its support for international projects. IRAP will continue to strengthen the coordination with International and LVC, increase awareness of these programs by the field staff, further streamline processes in place, and optimize project opportunities and funding allocations. It should be noted that LVC and international programs target specific client personas and are delivered by a subset of specialized ITAs. Therefore, the focus is on program positioning and strengthened program coordination rather than full integration into regular programming.
Action 1: Closely monitor expanded and newly-implemented eligibility thresholds for LVC projects to ensure increased program uptake.
Action 2: Adapt program staffing to include more half-time LVC advisors as a means to increase LVC scouting capabilities and to stabilize staffing levels.
Action 3: Continue the development of enabling tools and the refinement of processes to improve workflow management for the development and approval of international projects.
Action 1: Development and rollout of revised LVC content in the IRAP Field Manual and communications that supports field staff in the process of working with firms for potential LVC funding.
Action 2: Training to improve understanding and communication of client financial statements, ownership and corporate structures and business objectives developed and delivered to field staff.
Action 3: New tools are developed and implemented, and processes are refined for the development and approval of international projects.
|DG Client Enablement, ED International Innovation Office, DG Division Services||
Action 1: March 2024
Action 2: September 2023
Action 3: March 2024
|3. NRC IRAP should ensure that its existing inclusiveness strategy and initiatives are better communicated to and operationalized by staff in order to ensure enhanced integration of GBA Plus considerations into all aspects of the program from planning to execution and reporting. Additionally, client EDI data collection need to continue to be improved and clear EDI KPIs established to guide staff in their outreach activities. The effect(s) of program design and delivery (e.g. portfolio approach, reach, eligibility criteria) on clients should also be monitored using a GBA Plus lens and integrated into program management.||Moderate||
Background: NRC IRAP presented a program-wide EDI strategy to the IRAP Senior Leadership Team in April 2022. In alignment with the approved strategy, NRC IRAP will be launching a number of actions that address the findings of the evaluation.
Action 1: Foster an inclusive, accessible and anti-racist culture through the sharing of a lexicon tool to promote inclusive language; launch of a speaker series and facilitated conversations; piloting Snippets training (scenario-based EDI content) and piloting client awareness training for field staff.
Action 2: Address barriers in policies and systems by conducting accessibility and process reviews, incorporating EDI lens in due diligence considerations and putting unconscious bias training into practice.
Action 3: Enable sustained and measurable progress by identifying and monitoring key performance indicators to support field staff in understanding their role in guiding clients on their EDI journey.
Action 4: Assist ITAs in better understanding how portfolio decision-making is strengthened through diversity.
Action 1: Communications plan design is completed, including effective use of NRC IRAP Town Halls and other team gatherings to increase staff awareness and understanding of NRC IRAP's EDI Strategy.
Action 2: Change management plan developed with supporting tools for staff engagement, training and ongoing communications to increase awareness across staff.
Action 3: Completion of pilot of learning and development materials and resources, including speaker series, information toolkits, Snippets training and case studies.
Action 4: Evidence of increasing diversity in IRAP's portfolio of funded clients as demonstrated through IRAP's annual year-end performance reporting.
|DG Client Enablement||
Action 1: March 2023
Action 2: March 2023
Action 3: September 2023
Action 4: March 2024
|4. NRC IRAP should improve its performance measurement system and further use data analytics to inform program management processes by:
Background: Building on its existing solid foundation for data collection, analysis and reporting, IRAP is in the process of implementing a number of actions to further enhance its performance measurement practices.
Action 1: Develop a management toolkit, including a revised logic model and a comprehensive list of meaningful performance indicators. The toolkit will also include the information needed to facilitate reporting (e.g. detailed definitions, methodologies, and targets where applicable).
Action 2: Adopt a Power BI reporting platform, integrating the reports currently available in SAP Business Objects, and roll out to management and required staff within IRAP. Power BI reporting will provide an opportunity for rich data visualization to monitor key trends and performance against targets where applicable.
Action 3: Develop additional performance indicators to monitor the effectiveness of program design and delivery, including a set of comparative performance measures for Canadian SMEs in collaboration with Statistics Canada. These new indicators will provide insights on IRAP's contribution to enhance clients' capacity and productivity.
Action 1: IRAP Management toolkit is completed, including the revised logic model, the list of KPIs and the required supporting information (e.g. definitions, methodologies and targets, where applicable). Material is presented to leadership team and is made available to staff.
Action 2: Key performance measurement reports currently available in SAP BO are developed in Power BI and made available to management and staff, and training is delivered as needed.
Action 3: New performance indicators, including comparative metrics with Statistic Canada are developed and included in reporting.
|DG Division Services||
Action 1: March 2023
Action 2: September 2023
Action 3: March 2024
Appendix A – Methods
Internal and external documents were reviewed to provide context and to complement other lines of evidence in assessing relevance and performance. Internal documents included program area documents and strategic business documents. External documents included previous evaluative works, published and grey literature, and Government of Canada publications.
The data review covered IRAP administrative data on clients and projects, characteristics information on YOUTH program participants, snapshot information on clients, post-project assessments, post-project report data, data on advisory services, and other data regarding budget, human resources and key performance indicators.
A total of 308 ITAs were invited to compete an online survey which included questions related to the value and impact of certain tools on the ITA's work, as well as common barriers to their success.
125 ITAs responded for a response rate of 41%.
Importantly, ITAs were asked to identify potential candidate projects and firms for the PBCA study. They were asked to identify IRAP projects for which Canada has obtained:
- very high economic impacts—and ideally huge impacts
- impacts which can be quantified in dollars ("dollarized")
- impacts that are attributable to IRAP support, at least in part
143 firms, most of which had multiple IRAP projects, were identified through the ITA survey.
Partial benefit cost analysis (PBCA)
A PBCA is an approach to quantifying and comparing the costs of a program with the benefits it generates. With a PBCA, all of the program costs are accounted for, but only some of the benefits are quantified. As a result, the findings of the PBCA can indicate that the benefit-cost ratio is at least as large as estimated, and likely larger. Under this approach, the gross and net benefits of what are believed to be the "highest impact" projects are quantified. These are compared to the total costs of the program thus returning a lower bound estimate of value generated and a benefit-cost ratio.
The study identified 31 high impact firms through indications in the survey sent to ITAs, taking into account the number of projects completed, total value of projects and revenue and profit from FY 2011 to 2021.
In addition to internet searches, 22 firms were interviewed, and 17 were included in the study (9 firms declined or did not respond, and 5 firms dropped due to inability to clearly align with NRC IRAP contributions).
Journey mapping was conducted with two advisory services-only firms or unfunded firms. This client journey map consists of a diagram that illustrates the process clients go through in engaging with NRC IRAP and the key steps or milestones produced as a result.
Conversations were scheduled with founders that resulted in charts depicting the company's trajectory from their start to their contact with NRC IRAP to where they stand presently. ITAs responsible of those firms were also consulted.
International comparison study
From a preliminary list of NRC IRAP's international counterparts, four institutions were selected based on the structure of programs, services offered, and their financial/resource capacity: Business Finland (Finland); Innovate UK (UK); Vinnova (Sweden); and RVO.nl – Netherlands Enterprise Agency (The Netherlands).
Representatives from all four international comparators were interviewed. An interview guide was developed based on elements that could not be found in NRC IRAP's benchmark report or in the public domain (such as websites). Four interviews with five (n=5) representatives were conducted to fill in information gaps and to validate information that was collected through the document and website review.
CTO case studies
The case studies focused on documenting and understanding the outcomes that SMEs experience as a result of assistance from funded organizations (CTOs), specifically the impacts of this assistance on innovation and commercialization.
The case studies were conducted with four companies (from Ontario, Quebec and Atlantic regions). Interviews were conducted with the responsible ITAs for each company/project, as well as representatives from the companies.
Interviews and focus groups
A total of 64 people were interviewed between July 7th, 2021 and October 28th, 2021. Interviewees included IRAP staff, external stakeholders, and IRAP clients. Interviews conducted in the context of CTO case studies, journey mapping and PBCA mapping are not included in this total.
Five focus groups of approximately two hours were held with ITAs from each of the five IRAP regions. Participants were selected from those who volunteered and those who were nominated by the regional Directors General, totaling 36 participants. A list of discussion topics, as well as a note from the relevant NRC IRAP regional Directors General explaining the purpose of the exercise, was provided in advance. The topics were mainly related to the need for IRAP, the impacts of IRAP advisory and funding for SMEs, and ways in which program delivery and administration could be improved.
|Planning and evaluation interviewees||32||10||12||10||10||12||86|
Limitations and mitigation strategies
Although the evaluation encountered some challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. Details on limitations and their associated mitigation strategies are described below.
Assumptions underlying the PBCA results
Partial benefit cost analysis relies on a limited number of cases and assumptions about the economic value of benefits which could result in some inaccurate conclusions.
Mitigation: These assumptions have been taken into account in the sensitivity tests (upper and lower bounds; three discount rates). Also, where there are methodological uncertainties (e.g. with regard to attribution to IRAP), the analysis has been based on conservative assumptions.
Limitations of administrative and performance data
While NRC IRAP provided a large quantity of data to the OAE team, extensive data cleaning was required before analysis could be conducted. The extent of this cleaning exceeded the capacities of the digital systems available for the evaluation and thus impacted the ability of OAE to accurately perform some of the data analysis.
Mitigation: OAE and IRAP worked together to fully assess NRC IRAP's impact through creating a shared understanding of program data strengths and limitations. The program's analytics team additionally provided the evaluation team with some pre-analyzed data.
Sample size and bias with interviews
Relative to the selected populations, the number of interviews may not be representative of the total numbers of each group. Also, interviewees who are or were clients of NRC IRAP may be biased to respond either positively or negatively depending on whether they received funding or advice only.
Mitigation: Internal interviewees were carefully selected to obtain a comprehensive and balanced sampling across all five program regions.
Furthermore, 34 more interviews, for a total of 64, were conducted during the evaluation fieldwork than initially planned (n=30) in the evaluation methodology report.
The primary strategy to mitigate potential bias from client interviews was to ensure multiple lines of evidence addressing the same evaluation questions, interviews with funded clients in the context of the PBCA study (n=22), and the journey Mapping (n=2), CTO case studies (n=4) as well as administrative and survey client and projects data collected by the program .The primary strategy to mitigate potential bias from client interviews was to ensure multiple lines of evidence addressing the same evaluation questions, interviews with funded clients in the context of the PBCA study (n=22), and the journey Mapping (n=2), CTO case studies (n=4) as well as administrative and survey client and projects data collected by the program.
Furthermore, the interviews were used to obtain a range of informed and in-depth opinions rather than a consensus.
Appendix B – Supplementary data on program reach
NRC IRAP reach
|Total firms supported||2013||2014||2015||2016||2017||2018||2019||2020||2021||2022||2013-2017||2018-2022||Difference|
|# unique firms receiving advisory services and/or funding||4,025||7,687||6,682||7,712||7,471||8,288||8,159||8,034||9,379||9,078||18,732||22,458||+20%|
NRC IRAP funded firms
|Total firms funded||2013||2014||2015||2016||2017||2018||2019||2020||2021||2022||2013-2017||2018-2022||Difference|
|# unique firms receiving funding||3,270||3,044||3,320||3,434||3,407||3,255||3,541||3,302||3,055||3,657||7,456||7,717||+4%|
NRC IRAP advisory services
|Total firms who received advisory services||2013||2014||2015||2016||2017||2018||2019||2020||2021||2022||2013-2017||2018-2022||Difference|
|# unique firms receiving advisory services||2,020||6,606||5,425||6,669||7,319||7,343||7,399||7,228||8,679||8,410||17,370||20,840||+20%|
|Funded firms who received advisory services||707||1,763||2,063||2,386||3,191||2,420||1,712||2,506||2,385||2,989||6,095||7,141||+17%|
|Unfunded firms who received advisory services||1,313||4,843||3,362||4,283||4,128||4,923||5,687||4,722||6,294||5,421||11,276||13,699||+21%|
Figure 17. Professional (Mainly ITAs) workforce representation for women and Visible minorities groups did not meet labour market availability during the evaluation period and the gaps were slightly reduced
Source(s): Data review
Note: Analysis of data for Indigenous peoples and Persons with disabilities were not possible due to too few records.
Figure 17 – Text version
Two vertical overlapping bar charts illustrating professional workforce representation statistics for women and Visible minority groups compared to labour market availability statistics during the evaluation period. The first bar chart represents women, where the gap between workforce representation and labour market availability decreased over the 5 year evaluation period. The second bar chart represents Visible minorities, where the gap between workforce representation and labour market availability also decreased over the 5 year span. Analysis of data for Indigenous peoples and Persons with disabilities were not possible during this evaluation period due to too few records.
Figure 18. The annual number of new funded clients peaked in FY 2018-19 and was relatively low in FY 2019-20 and FY 2020-21. Although FY 2021-22 is not complete at the time of analysis, it is likely to be similar to the FY 2017-18 level. The number of new funded clients has fluctuated during the evaluation period. However, downward trend in new clients was reported by interviewees.
Source(s): Data review
Note: The definition of new funded clients is as follows: new funded clients in FY18 = clients never funded between FY13 and FY17, new funded clients in FY19 = clients never funded between FY13 and FY18, new funded clients in FY20 = clients never funded between FY13 and FY19, etc. Overall, a total of 4,575 new clients were funded in 2018-22. Complete dataset for FY 2021-22 was not available at the time of this analysis.
Figure 18 – Text version
A vertical bar chart representing the annual number of new funded clients from FY 2017-18 to FY 2020-21. The amount of new funded clients peaked in FY 2018-19, then decreased in both FY 2019-20 and FY 2020-21. FY 2021-22 is projected to be similar to FY 2018-19, although FY 2021-22 was not complete at the time of analysis.