Evaluation of the NRC Centre for Surface Transportation Technology

Executive Summary

This report presents the results of the 2012-13 evaluation of surface transportation activities at the National Research Council (NRC). The NRC Centre for Surface Transportation Technology (CSTT) provided rail and ground vehicle engineering, testing and technology development services to Canadian and international public and private sector clients. In 2011-12, as part of NRC's corporate transformation, the CSTT became the Surface Transportation (ST) Portfolio (effective April 1, 2012). The organization operated as the ST Portfolio for one year and, effective April 1, 2013, the ST Portfolio merged with the NRC Automotive Portfolio – now named, the Automotive and Surface Transportation (AST) Portfolio.

The evaluation, launched in 2012, focused primarily on the period of 2007-08 to 2011-12. However, where appropriate and available, more recent information relating to the organization's activities and operations as the ST Portfolio (for fiscal year 2012-13) is also included. The announcement to merge Surface Transportation and Automotive into the AST Portfolio occurred following the data collection phase of this evaluation and has, consequently, not been reflected in the retrospective assessment of the CSTT and ST Portfolio. The organizational change has however been considered in the recommendations stemming from the evaluation.

For simplicity, and since there was little change in activities and operations between when the organization was called the CSTT and when it operated as the ST Portfolio, retrospective information in this report relating to operations up to and including 2012-13 refers to the organization as the CSTT. Forward-looking information (relating to the period beyond April 2013) refers to the organization as a component of the AST Portfolio.

The evaluation study was designed to address the core evaluation issues required by the Treasury Board Policy on Evaluation: relevance (i.e., continued need, alignment with government priorities and roles) and performance (i.e., achievement of intended outcomes, as well as resource utilization). It was led by an independent evaluation team within NRC's Office of Audit and Evaluation. The methodology developed for this evaluation made use of multiple lines of evidence and complementary research methods as a means to enhance the reliability and validity of the information and data collected. The specific methods used in the study include a document review, a review of administrative and performance data, key informant interviews, and a series of ten case studies of CSTT projects. The key findings, recommendations and management response are summarized below.

Findings – Relevance

AST Portfolio functions and activities inherited from the CSTT continue to meet an important need within the transportation sector. Historical data reveals a consistent and, in the case of the RVMS/Ground Vehicles division, growing demand for surface transportation related services. Clients reported that CSTT activities were in line with their needs and that, in most cases, these needs could not have been met as fully or adequately in the absence of the CSTT. Moving forward, the AST Portfolio appears to have clearly taken sector needs into consideration in preparing the business cases for the proposed Fleet Forward 2020 and Rail NRC Programs. Both NRC Programs, if approved, intend to maintain testing/engineering solutions to address short term client needs, and also identify and execute upon technology opportunities relevant to the sector needs.

As a Technology Centre, the CSTT's alignment with NRC priorities was less clear, as this was not an explicit requirement at the time. Rather, while the CSTT received very little direction from NRC in terms of priorities, it focused primarily on financial sustainability and relevance in niche markets. As a result, the work of the CSTT was largely opportunistic and the organization more or less operated as a high end engineering consulting group. Although a lot of this work could be seen as somewhat aligning to NRC's priorities, strategic alignment was not a primary consideration. Externally financed client project work often took precedence over technology development activities. Still, over the course of the evaluation period, efforts were made to progress from primarily testing and skilled technical services, which historically constituted the bulk of the work at the CSTT, toward "higher value" projects.

Moving forward, the Portfolio's objectives and future activities (as described in the business cases for the proposed Rail and Fleet Forward 2020 NRC Programs) align well with NRC's new strategy, and also with Government of Canada priorities. One area where the alignment with NRC's strategy has not been as apparent is with respect to work with international clients, which made up almost two-thirds of Rail division revenues. For the CSTT, work for foreign clients helped to establish a strong international reputation in the area of transportation technologies and has led to significant benefits for NRC and for its Canadian clients. Still, it appears that there have been some cases where the benefits to Canada derived from these projects were not always apparent.

CSTT activities were generally in line with the mandate of NRC, and were therefore consistent with federal roles and responsibilities. Interviewees specifically referenced the role played by the CSTT in terms of providing access to specialized facilities and supporting the work of other government departments, including support to policy/regulatory decision-making, a role that is not normally appropriate for the private sector. However, there were isolated cases where work conducted by the CSTT may have been more appropriate for the private sector, such as the production or manufacturing of previously designed and tested prototypes.

Findings – Performance

The CSTT was successful in generating sufficient revenue and managing its business to operate in a financially sustainable manner (as per commitments outlined in the original agreement between the CSTT and NRC); in fact, the CSTT generated financial surpluses in each of the five years reviewed as part of this evaluation. Most of the revenue generated by the CSTT came from its project work for clients, although it also generated additional revenue through licensing of some development work. Revenue generated by the CSTT increased over the past five years, primarily due to additional revenue generated by the RVMS (Ground Vehicles) group, through its work for DND.

Many factors enabled the CSTT to generate revenue and to operate in a fiscally independent manner within NRC. These include: the use of business practices and measures to support operations; a strong client focus; the positive reputation of NRC; the CSTT's ability to charge market rates to its clients; and the historically high engagement of CSTT personnel. Project management practices, in particular, were viewed as a real strength of the organization, although these were not implemented consistently across the CSTT. It is unknown to what extent the merge with the Automotive Portfolio will have an impact on project management practices.

As some factors enabled the CSTT's revenue generation, other factors impeded its ability to generate revenue. In particular, the CSTT's focus on fewer clients made it vulnerable to the priorities and needs of these clients. The evaluation found that opportunities exist for the new AST Portfolio to diversify its client base in areas related to surface transportation.

The evaluation assessed client satisfaction through a variety of methods, all of which demonstrated that overall, clients had a very positive view of the CSTT and of the services received. In many cases, the needs of clients could not have been met as easily in the absence of the CSTT, as the organization had unique qualities or delivered certain efficiencies (examples provided include the CSTT's access to unique facilities and expertise, its ability to deliver "turnkey" or "full-circle" solutions, and its reputation for neutrality and quality). CSTT projects are also seen to have achieved clear benefits for clients and contributed to the achievement of higher-level outcomes, in terms of environmental, safety and economic impacts.

Work performed by the CSTT has, in some cases, led to unintended impacts. Examples of positive unintended impacts include the development of unique expertise for Portfolio staff, unexpectedly strong relationships with key public sector clients, and positive media coverage. Examples of negative unintended impacts include the isolation of CSTT staff from the rest of NRC and perceived competition with private sector firms.

Findings – Resource Utilization

Without direct NRC A-base investment, CSTT operated efficiently and economically, taking steps to minimize overhead costs, and these practices carried over into the operations of the ST Portfolio. However, the organization's inability to access working capital from year to year necessitated a considerable focus on cost reduction measures, sometimes to the detriment of its effectiveness. For example, interviewees perceived the CSTT as historically being very frugal when it came to investing in facilities. As a result, the Portfolio has reached the point where some of its main facilities (most notably the 30-meter long Environmental Chamber and the Vehicle Dynamics Facility) are now thirty years old and in need of major upgrades.

More recently, internal and external interviewees highlighted some challenges associated with increased accountability requirements and the gradual loss of flexibility in Portfolio operations. It should be noted that these changes have been implemented across all NRC portfolios. However, based on interviewee opinions, the challenges appear to be more apparent for the ST Portfolio, given that they represent a marked change from previous operations as a technology centre.

There was little or no duplication with other Portfolios and the CSTT at times cooperated with other areas of NRC where the work has been seen as being complementary. Still, the evaluation revealed that there are opportunities for greater synergy with other portfolios, particularly in the area of automotive research. This should be addressed, given the recent merger of the ST and Automotive portfolios under a single management structure, Automotive and Surface Transportation.

Conclusions

Overall, the findings of the evaluation of the CSTT show that the activities undertaken by the Centre (which have now been transferred to the AST Portfolio) represented good value-for-money for NRC and for Canada. As NRC continues to implement the final stages of its new strategy, including the recent merger of the ST and Automotive portfolios, it will be important to exercise caution in order to ensure that the strong client focus, efficient management practices, and reputation developed at the former CSTT are maintained and further enhanced.

Acronyms And Abbreviations

Acronyms And Abbreviations

aIWS	Autonomous Instrumented Wheelsets
APT	Advanced Products and Technologies
ASPM	Administrative Services and Property Management
AST	Automotive and Surface Transportation
CPR	Canadian Pacific Railway
CSTT	Centre for Surface Transportation Technology
DND	Department of National Defence
DRDC	Defence Research and Development Canada
FTE	Full-Time Equivalent
GDP	Gross Domestic Product
GHG	Greenhouse Gas
HQP	Highly Qualified Personnel
IAR	Institute for Aerospace Research
IP	Intellectual Property
LETE	Land Engineering Test Establishment
LSV	Low Speed Vehicles
MOU	Memorandum of Understanding
MSE	Methodological and Subject Matter Expert
MTO	Ministry of Transportation of Ontario
NRC	National Research Council
OAE	Office of Audit and Evaluation
OEM	Original Equipment Manufacturer
OGD	Other Government Department
QETE	Quality Engineering Test Establishment
R&D	Research and Development
RCMP	Royal Canadian Mounted Police
RTO	Research and technology organization
RVMS	Road Vehicles and Military Systems
SEC	Senior Executive Committee
ST	Surface Transportation
SWOT	Strengths, Weaknesses, Opportunities and Threats
TFP	Total Factor Productivity
TPOF	Technical and Protective Operations Facility
TTCI	Transportation Technology Centre Inc.

1. Introduction

This report presents the results of the 2012-13 evaluation of surface transportation activities at the National Research Council (NRC). NRC surface transportation activities underwent a number of changes over the course of the evaluation. The NRC Centre for Surface Transportation Technology (CSTT) provided rail and ground vehicle engineering, testing and technology development services to Canadian and international public and private sector clients. In 2011-12, as part of NRC's corporate transformation, the CSTT became the Surface Transportation (ST) Portfolio (effective April 1, 2012). The organization operated as the ST Portfolio for one year and, effective April 1, 2013, the ST Portfolio merged with the NRC Automotive Portfolio – now named, the Automotive and Surface Transportation (AST) Portfolio.

The announcement to merge Surface Transportation and Automotive into the AST Portfolio occurred following the data collection phase of this evaluation and has, consequently, not been reflected in the retrospective assessment of the CSTT and ST Portfolio. The organizational change has however been considered in the recommendations stemming from the evaluation. For simplicity, and since there was little change in activities and operations between when the organization was called the CSTT and when it operated as the ST Portfolio, retrospective information in this report relating to operations up to and including 2012-13 refers to the organization as the CSTT. Forward-looking information (relating to the period beyond April 2013) refers to the organization as a component of the AST Portfolio.

The evaluation of the CSTT was led by an independent evaluation team from the NRC Office of Audit and Evaluation (OAE). The work of the evaluation team was supported by Methodological and Subject Matter Experts (MSEs) who provided advice related to the evaluation framework, approach, interpretation of findings and recommendations. These experts did not act in a decision-making fashion, but rather played an advisory role in ensuring a high-quality and useful evaluation product.

Following the evaluation overview presented below, Section 2 of this report provides a profile of the Portfolio. Sections 3, 4 and 5 present the evaluation study's findings organized by broad evaluation question (relevance, performance, resource utilization), along with associated recommendations, embedded. Section 6 presents the general conclusions from the study. Section 7 of the report lays out management's response to these recommendations and the actions that will be taken as a result of the evaluation.

1.1 Evaluation Overview

The evaluation assessed the value-for-money of the CSTT, including relevance, performance, and resource utilization. The evaluation focused primarily on the period of 2007-2008 to 2011-2012. Where appropriate and available, the evaluation is also supplemented with more recent information relating to the organization's activities and operations as the ST Portfolio (2012-13) and with prior information dating back to 2002-03.

1.1.1 Evaluation Rationale

This evaluation of the CSTT was carried out in accordance with NRC's approved evaluation plan. The primary decision factor, which prompted the CSTT to be evaluated at this time, is the fact that there has been no previous evaluation of the Centre. As per the Treasury Board's Policy on Evaluation (2009), all programs benefiting from any level of direct program spending must now be evaluated every five years.

1.1.2 Evaluation Design and Methodology

The approach and level of effort for the evaluation was commensurate with the low level of risk facing the Centre, as identified during an assessment conducted as part of the planning phase. As such, an objectives-oriented approach was taken, using a non-experimental, largely descriptive evaluation design. The specific evaluation questions, outlined at the start of each section, are based on consultations with program management and a review of key documents during the planning stage. The evaluation issues identified are also aligned to the requirements of the 2009 Treasury Board Policy on Evaluation. The complete evaluation matrix, which identifies the lines of evidence employed to respond to each question, was developed by the evaluation team in consultation with Methodological and Subject Matter Experts, and is presented in Appendix A: Evaluation Matrix.

The evaluation methodology integrated the use of multiple lines of evidence and complementary research methods as a means to enhance the reliability and validity of the information and data to be collected. The specific methods used in the study include:

• Internal and external document review;
• Administrative and performance data review;
• Key informant interviews (internal interviewees: n = 20, external interviewees n = 42, from 23 different public and private, client and non-client organizations); and
• A series of 10 case studies, based upon reviews of project documentation and both internal and external interviews (these interviews are included in the count above).

For key informant interviews, the following scale is used in the text of the report to indicate the relative weight of the responses for each of the respondent groups.

• "All/almost all" – findings reflect the views and opinions of 90% or more of the respondents commenting on that particular issue;
• "Many/most" – findings reflect the views and opinions of at least 50% but less than 90% of the respondents commenting on that particular issue;
• "Some/several" – findings reflect the views and opinions of at least 25% but less than 50% of the respondents commenting on that particular issue; and,
• "A few" – findings reflect the views and opinions of at least two respondents but less than 25% of the respondents commenting on that particular issue.

A more detailed description of the study methodology, limitations and challenges is provided in Appendix B: Methodology.

2. Program Profile

2.1 Brief History of the Surface Transportation Portfolio/CSTT

NRC created the CSTT in 1993 as a funded institute, by joining existing business units focused on road and rail engineering, testing and technology with others that worked in the areas of pavement and climatic engineering. In June 1995, following a comprehensive review of its activities and programs in response to the federal government's Program Review, NRC announced that the CSTT was to become one of six cost-recoverable technology centres operating within the organization. As of 1998-99, the CSTT began generating sufficient revenue to cover its expenses (as per commitments outlined in the original agreement with NRC) and be considered cost-neutral.^{Footnote 1}

In 2011-12 NRC underwent a corporate transformation, whereby its mosaic of research institutes, programs, cluster initiatives and technology centres were restructured into thirteen streamlined portfolios that focus on key industrial sectors and key priorities facing Canada. As part of this transformation (effective April 1, 2012), the CSTT was renamed the Surface Transportation Portfolio, and was declared to be "in transition" while it developed and sought approval for its new NRC Programs^{Footnote 2} It is important to note that during this time, there was little change in the activities and operations of the organization.

Most recently, as of April 1, 2013, the ST Portfolio has been merged with the NRC Automotive Portfolio under a single management structure, in order to enhance the management of resources and capabilities. This new Portfolio is now called the Automotive and Surface Transportation Portfolio.

2.2 Portfolio Objectives

From 1995 to 2012, the primary objective of the CSTT was to generate sufficient revenue to sustain itself as a cost-recoverable technology centre within NRC, while retaining responsibility for maintaining and offering unique large-scale vehicle testing infrastructure for Canada. Although CSTT planning documents describe high-level goals for the organization, some in the area of technology development, these were internally driven, and the CSTT was not expected to meet any technology development objectives specified by NRC.

With the organization's transition to an NRC portfolio, clear objectives outlining its contribution to the overall NRC strategy have been identified in its Strategic Plan. It is unknown at this time how the objectives of the Portfolio will change as a result of its recent merger with the Automotive Portfolio. At the time of the evaluation, the goal of the ST Portfolio was "to vigorously develop and deliver on vehicle mobility technologies that attract and retain world-class Canadian public and private sector paying clients."^{Footnote 3} This was intended to "reduce the incidence of derailments, improve fuel efficiency, reduce environmental impact/accidents, improve capacity, reduce maintenance costs and improve operational capability in the rail and ground vehicle transportation systems under all environmental conditions."^{Footnote 4}

The following operational goals were identified to support the ST Portfolio's objectives:^{Footnote 5}

1. Execute NRC Programs to maximize NRC investment and beneficial impacts in key surface transportation market segments;
2. Build a sustainable and relevant technology development capability;
3. Sustain and diversify the technical services business and outreach in niche segments of the surface transportation market; and
4. Build a new employee culture around innovation and risk-taking.

As these goals and objectives were announced in 2012, and the evaluation assessed the past performance of the program (particularly during the time it operated as the CSTT), the organization's performance is measured against its objectives over the five years of the 2007-2012 evaluation period (i.e., financial sustainability and ensuring value to clients). However, the more recent goals and objectives are considered when assessing the relevance of the Portfolio activities, moving forward.

2.3 Portfolio's Suite of Programs

In the early years of the period being evaluated, the CSTT was organized into three business divisions: Road Vehicles and Military Systems (RVMS), Rail, and Climatic Engineering. Separate from these three business divisions, the CSTT also had a corporate research and development group, Advanced Products and Technologies (APT), which supported/collaborated with the three business divisions, but was focused on longer-term opportunities with the potential to earn licensing income rather than revenue. With the CSTT's transformation into an NRC Portfolio, the organization was restructured to support NRC's new program-based structure and the result was two consolidated groups: Rail and Ground Vehicles (the Climatic Engineering division was integrated into the RVMS division, which was then renamed Ground Vehicles). The APT group was expanded and renamed Advanced Mobility Technologies.

Figure 1 shows the proportion of revenue derived from each CSTT business division, illustrating the relative size of each.

Figure 1 : Proportion of CSTT Revenue by Business Group

Long description of Figure 1: Proportion of CSTT Revenue by Business Group

 

Revenues	RVMS	Rail	Climatic	Other	Total
2007-08	0.78208893	0.176305657	0.025398896	0.016206517	1
2008-09	0.771679091	0.187840037	0.029405447	0.011075424	1
2009-10	0.847187535	0.106644657	0.046167808	0	1
2010-11	0.791307793	0.156927809	0.051764398	0	1
2011-12	0.830370106	0.136489673	0.033140222	0	1

Source: CSTT Financial Statements (2007-08 to 2011-12)

Going forward, the AST Portfolio is developing two NRC Programs (the Fleet Forward 2020 NRC Program and the Rail NRC Program), which align with historical CSTT activities. At the time of writing, these NRC Programs are still in development. Both business cases for these NRC Programs have been presented to NRC's Senior Executive Committee (SEC) and the Portfolio is now proceeding to the preparation of program business plans.

The proposed Fleet Forward 2020 NRC Program is divided into three distinct yet complementary sub-sector business thrusts (military, trucking and mining, and public vehicles) and has identified three technology opportunities along which it will develop and deploy new technologies: Vehicle Intelligence (with a focus on Vehicle Health Management), Power Management, and Enhanced Aerodynamic Performance.^{Footnote 6}

The proposed Rail NRC Program focuses on generating efficiencies through reduction of the rail industry's operating costs. To do so, the following technology thrusts have been identified: Autonomous Instrumented Wheelsets, Data Analytics, and Sensor Technologies.^{Footnote 7}

2.4 Resources

The CSTT operated as a technology centre within NRC, whereby it was driven primarily by external revenue. Although the Centre did not receive direct A-base funding, it did benefit from corporate services paid for by NRC. Figure 2 illustrates the annual revenue generated by the CSTT between 2002-03 and 2011-12, as well as the size of its workforce, represented in full-time equivalents (FTEs). While the organization experienced a 119% increase in revenue between 2002-03 and 2011-12, the number of staff only increased by 13% over the same period.

Figure 2: CSTT Resources

Long description of Figure 2: CSTT Resources

 

Year	Revenue	FTEs
2002-03	10,774,000	108.7
2003-04	13,639,000	108.5
2004-05	14,995,000	96.2
2005-06	16,341,000	98.6
2006-07	16,217,000	106.1
2007-08	18,650,000	109.2
2008-09	18,649,000	116.5
2009-10	23,385,000	121.1
2010-11	21,715,000	121.7
2011-12	23,549,200	122.9

Source: NRC Utilization of Resources Reports (2002-03 to 2011-12)

The CSTT was based in Ottawa. Its staff occupied:

1. A 45 acre campus at 2320 Lester Road, on the east side of the Ottawa International Airport (AST Portfolio staff still occupy this facility).
2. Another Ottawa facility (located at the corner of Hunt Club and Woodroffe roads) although personnel were moved back to the main site on Lester Road in the Fall of 2012 and the facility was closed down.

In addition to its human resources, the CSTT maintained unique vehicle testing facilities to serve its clients' needs (these have all been transferred to the AST Portfolio). Some of the facilities include:

• Large scale climatic test facility
• Large scale vibration test facility
• Wheel bearing and brake facility
• Rail impact hill facility
• Large scale squeeze frame facility
• Indoor vehicle tilt table facility
• Large scale outdoor vehicle tilt table facility

The CSTT also sometimes used (and the AST Portfolio still uses) other government facilities, where appropriate, to meet its clients' needs. The most notable example is the Royal Canadian Mounted Police's (RCMP) Technical and Protective Operations Facility (TPOF), which includes a 400 acre parcel of land that has numerous calibrated road and off-road vehicle courses used for ground vehicle testing. The use of the facility is governed under a Memorandum of Understanding (MOU) between NRC and the RCMP.

When tracking expenditures, the CSTT differentiated between direct costs (i.e. labour and materials directly associated with a project), indirect costs (i.e., marketing, business development and facility costs incurred at the business division level), and overhead costs that are not attributable to a particular CSTT business division. These costs, as well as the CSTT financial surplus (included as part of NRC corporate accounts at year-end), are presented in Figure 3.

Figure 3: CSTT Expenditures and Financial Surplus (in millions)

Long description of Figure 3: CSTT Expenditures and Financial Surplus (in millions)

 

	2007-08	2008-09	2009-10	2010-11	2011-12
Direct Costs	$ 11,392,267	$ 10,614,245	$ 12,845,260	$ 12,112,340	$ 13,498,300
Indirect Costs	$ 4,237,836	$ 4,210,993	$ 4,455,956	$ 4,955,468	$ 5,354,207
CSTT Overhead	$ 2,235,341	$ 2,404,107	$ 4,477,632	$ 4,256,486	$ 3,715,453
Surplus	$ 491,338	$ 1,202,274	$ 989,202	$ 319,346	$ 927,085

	2007-08	2008-09	2009-10	2010-11	2011-12
Direct Costs	$ 11.4	$ 10.6	$ 12.8	$ 12.1	$ 13.5
Indirect Costs	$ 4.2	$ 4.2	$ 4.5	$ 5.0	$ 5.4
CSTT Overhead	$ 2.2	$ 2.4	$ 4.5	$ 4.3	$ 3.7
Surplus	$ 0.5	$ 1.2	$ 1.0	$ 0.3	$ 0.9

Source: CSTT Financial Statements (2007-08 to 2011-12)

2.5 Client Profile

The CSTT served both public and private sector clients, within Canada as well as internationally. As shown in Figure 4, the majority of the CSTT's revenue in 2011-12 came from Canadian public sector sources (84% overall in 2011-12), with 69% coming directly from the Department of National Defence (DND). A number of clients were also DND suppliers who used CSTT facilities to support their own work with DND (e.g., one firm, whose primary client is DND, worked with the CSTT to test the products produced for DND). As a result, 69% is an underestimate of the overall CSTT activity conducted for DND's benefit.

In 2011-12, the CSTT's top 10 clients represented 92% of the organization's revenue for the year and included:

• 4 Canadian federal departments and agencies,
• 1 Canadian firm, and
• 5 U.S. organizations (4 private sector firms and 1 public sector organization).

Figure 4  : CSTT Sources of Revenue (2011-12)

Long description of Figure 4  : CSTT Sources of Revenue (2011-12)

 

	Rev Proportion	Revenue
DND	69%	16,298,282
Transport Canada	9%	2,214,203
Other Government (federal and provincial, excluding DND and Transport)	4%	826,291
Municipal / Transit Authorities (Canada)	2%	399,636
Industry - Canada	7%	1,608,543
International (includes international transit authorities)	9%	2,148,090
International government		✓
Other		✓
Total		23,495,045

2011-12 Data:

Rail

OGD(Federal and provincial)	867059	0.270378763
Municipal / Transit Authorities	259288	0.0808549
Industry - Domestic	45074	0.014055621
International	2035410	0.634710716

RVMS

OGD (Federal and provincial)	17869697	0.91594459
Municipal / Transit Authorities	140348	0.007193798
Industry - Domestic	1494723	0.07661481
Industry - International	0	0
Other (NRC Internal)	4815	0.000246802

Climatic

OGD (Federal and provincial)	597205	0.766993608
Municipal / Transit Authorities	0	0
Industry - Domestic	68746	0.088290859
Industry - International	112680	0.144715533
Other (NRC Internal)	0	0

Source: CSTT Financial Statements (2011-12)

The Rail and RVMS/Ground Vehicles business divisions generally serve distinct client groups (with the exception of Transport Canada and some transit authorities). The Rail division's primary clients were international industry clients (63.5% of its revenue in 2011-12), Transport Canada (27%) and transit authorities (8%). The RVMS/Ground Vehicles division's work is primarily for DND (81% of its revenue in 2011-12), other federal government departments (11%) and Canadian industry (8%). Both divisions have developed long-standing relationships with key clients, in some cases dating back more than 20 years. The impact of this is discussed more in Section 4.2 Client Satisfaction and Client Benefits.

3. Findings: Relevance

The relevance of the CSTT was examined through three evaluation issues: the continued need for the program (section 3.1); its alignment with the priorities of the federal government (section 3.2); and its alignment with the roles and responsibilities of the federal government and of NRC (section 3.3). It is important to note that because the assessment of relevance is primarily forward-looking, this section of the report refers mostly to the AST Portfolio (as opposed to the CSTT).

3.1 Continued Need

3.1.1 Importance of the Canadian Transportation Industry to the Canadian Economy

Key Finding 1: External documentation demonstrates the importance of the transportation industry to the Canadian economy.

External documentation reviewed as part of the evaluation confirms that transport is a significant force within the Canadian economy and plays a key role in sustaining Canada's productivity and competitiveness. In 2011, the transportation services sector represented 4.2% of Canada's gross domestic product (GDP), or $53 billion. Truck transportation represented the largest segment of transportation services and accounted for 31% of the sector's share of GDP, while the rail segment represented 11%. Overall, for 2011, growth experienced in the transportation sector's GDP surpassed the Canadian average. On a year-over-year basis, the sector's GDP growth increased by 3%, while that for all industries increased by 1.9%.^{Footnote 8}

The 2012-2013 ST Portfolio Operating Plan identifies six segments of the Canadian transportation market which are to be addressed by the new AST Portfolio: trucking; rail; emergency services; urban transit and intercity buses; defence/military ground vehicles; and, mining ground vehicles. Each of these segments has experienced strong to moderate growth in recent decades, which is forecast to continue.

The efficiency of the transportation sector has a major impact on just how effectively Canadian-based firms can compete, for example, in terms of shipping goods or sourcing components cost-effectively. The performance of the transportation system is also vital to enabling mobility in a timely, cost-effective, and environmentally sustainable manner.^{Footnote 9}

Since the 1980s, productivity gains in Canada's transportation sector have outpaced those of Canada's business sector as a whole. A report prepared by the Conference Board of Canada^{Footnote 10} shows that from 1981 to 2006, total factor productivity (TFP)^{Footnote 11} grew by 3.6% per year for rail freight and 1.8% per year for trucking (available data to 2003 only). By comparison, TFP growth in Canada's overall business sector was only 0.2% per year during the same period. This underlines the important contribution of these modes to productivity in the overall business sector. By contrast, public transit has historically been characterized by poor productivity performance (productivity in public transit declined by 1.2% per year between 1986 and 2006), despite a recent upswing in ridership and activity across the country.^{Footnote 12} The report attributes this in large part to the public mandate of the sector and to challenges associated with increased population dispersion, although lower rates of technology adoption are also believed to be a factor.^{Footnote 13}

3.1.2 Needs Facing the Surface Transportation Sector

The main industry players in each of NRC's addressable surface transportation related market segments are fleet owners/operators, original equipment manufacturers (OEMs), systems integrators, industry associations and regulatory agencies. The major needs facing the surface transportation industry have been well articulated in the business cases for the Fleet Forward 2020 and Rail NRC Programs, put forward by the AST Portfolio. These are also consistent with views provided by internal and external interviewees, and with information drawn from the document review.

To varying degrees, all Canadian vehicle fleet operators in rail, trucking, transit, emergency services, military, and mining are seeking ways to lower costs and improve operations, from a safety, efficiency and reliability standpoint (which, for industry clients, will ultimately result in enhanced competitiveness). This includes:

• Reducing fleet operating costs related to fuel consumption;
• Reducing fleet operating costs related to unplanned repairs and maintenance, and increasing fleet availability;
• Reducing accidents and rail derailments; and
• Increasing automation (e.g. track inspections).

At the same time, fleet operators and OEMs are being required to meet higher environmental, safety and technical standards and tighter regulations. Indeed, in tackling greenhouse gas (GHG) emissions, the Government of Canada's initial focus has been directed at regulating the transportation sector, the largest source of Canadian emissions.^{Footnote 14} For example, Environment Canada's Heavy-Duty Vehicle and Engine Greenhouse Gas Emission Regulations seek to reduce GHG emissions from heavy duty vehicles (including semi-trucks, garbage trucks and buses) by up to 23% in 2018.^{Footnote 15} On the rail side, meeting track safety regulations is also very costly for rail operators due to the frequent (daily) and labour-intensive monitoring and inspections required.

Meanwhile, federal and provincial transport ministries set regulatory requirements and are evaluators of advanced technologies that are proposed to be introduced into their respective jurisdictions. Government regulators at both levels therefore require impartial, independent and credible science-based inputs into policy and regulatory decision-making.

The Canadian Army owns and operates about 9,000 land vehicles. In the past, DND operated the Land Engineering Test Establishment (LETE), which provided "one-stop shopping" for Canadian Army design, development and testing. Despite the close of LETE in 1994, DND continues to require full engineering solutions to technical problems encountered on its vehicles, from the concept and design phase, to prototype building and demonstration.

In recent years, these needs have mainly been driven by DND's mission in Afghanistan, where work revolved around ensuring that vehicles were fitted to optimize the safety and security of deployed forces, and often required very quick turnaround times. Now that the mission in Afghanistan has ended, the needs facing DND are less reactionary and immediate, so the focus of DND has shifted to longer-term issues (examples highlighted by interviewees include adjustments that would improve vehicle effectiveness or would extend asset life, maximizing return on investment).

Still on the military side, original equipment manufacturers and other DND suppliers require testing services in order to pre-qualify their equipment for use by DND.

3.1.3 Alignment with Client Needs

Key Finding 2: AST Portfolio surface transportation related activities and proposed programs are generally in line with client needs.

All federal and provincial government clients interviewed felt that AST Portfolio surface transportation related activities and research competencies are aligned with the needs and priorities of their organization and that the CSTT has been able to respond well to these needs and priorities in the past. DND interviewees were particularly adamant about this, and cited the CSTT's work during the Canadian Forces' mission in Afghanistan (2005 through 2012) as a prime example of the organization's responsiveness to their needs.

Of the remaining active clients^{Footnote 16} and non- or inactive clients^{Footnote 17} who discussed this issue, about half (four out of seven clients and two out of four non/inactive clients) felt that CSTT/Portfolio activities are aligned with their needs, while the rest felt that their needs are partially met and noted that there were areas not served by the Centre. All clients interviewed were also asked to rate, out of five, the availability of scientific/technical competencies to meet their organization's needs. Responses to this question were very positive, with an average, from all clients, of 4.4 out of five.

Some clients and non/inactive clients (seven out of 22) noted the importance of regular communications and outreach from the CSTT/Portfolio, in order to gauge future client needs and inform clients and potential clients about its competencies and available solutions. While this already happens to a certain extent with some of the Portfolio's key clients (DND and Transport Canada, in particular), the general consensus was that there are opportunities for more communication and outreach.

Moving forward, the most recent Portfolio Operating Plan recognized that "the need to realize economic efficiencies in the Canadian transportation system, while also responding to the patchwork of road and rail safety regulations in Canada, is the opportunity for a technology agenda". Accordingly, the AST Portfolio appears to have clearly taken these sector needs into consideration in preparing the business cases for the proposed Fleet Forward 2020 and Rail NRC Programs. Both NRC Programs propose to maintain testing/engineering solutions, on a full cost recovery basis, to address short term client needs. As well, the NRC Program business cases identify technology opportunities to be explored, which are very much aligned with surface transportation sector needs. On the Fleet Forward 2020 side:

• Vehicle Intelligence, with a focus on vehicle health management, will contribute to reduced repair and maintenance costs and improved vehicle availability; and
• Power Management and Enhanced Aerodynamic Performance are both expected to lead to fuel savings and GHG reductions.

As for the Rail NRC Program, the three proposed rail technology thrusts (Autonomous Instrumented Wheelsets, Data Analytics and Sensor Technologies) all contribute to more cost-effective preventative maintenance practices.

3.1.4 Ability to Meet Client Needs in the Absence of the Portfolio

Key Finding 3: Most client needs could not be met as fully or adequately in the absence of the AST Portfolio.

Most internal and external stakeholders noted that there are organizations that provide similar services or have comparable facilities to those offered by the AST Portfolio. The Transportation Technology Centre Inc., or TTCI, in Pueblo, Colorado, was frequently noted by interviewees on the rail side. However, generally, the Portfolio has unique qualities (e.g. superior technical expertise in certain areas, high standards in terms of client confidentiality) or delivers certain efficiencies (due to proximity or ease of contracting, for example). Of the ten case studies that were conducted as part of the evaluation, two clients indicated that other organizations could likely have met their needs, although it is unknown whether there would have been impacts to the quality or efficiency of the project.

In the case of DND specifically, although some of the department's technology needs can be met by the private sector, interviewees repeatedly stressed that the AST Portfolio has a unique ability to deliver "turnkey" or "full-circle" solutions. Many also noted the fact that the MOU-based relationship between NRC and DND is useful, particularly in cases where it is difficult to establish a clear contractual relationship (e.g. for the development of prototypes, or for specialized engineering services, where the scope of the project is not defined at the outset) or when quick turnaround times are required. DND also has some in-house capabilities, either housed with Defence Research and Development Canada (DRDC), DND's Quality Engineering Test Establishment (QETE), or the Department's 202 Workshop Depot,^{Footnote 18} which address some of the Department's research and technology needs. However, the AST Portfolio generally provides services and expertise unique from those offered through these business units. In rare cases where there may be some overlap, particularly with 202 Workshop Depot, DND interviewees indicated that given the limited capacity of the unit, working with the AST Portfolio is much more likely to produce timely results.

For its part, Transport Canada also has some in-house capabilities, although typically when they go to NRC, it is because the required expertise or facilities do not exist in-house. Transport Canada clients also noted that it would be difficult to contract their work to the private sector, since they have limited capacity and would not have the same access to large-scale testing facilities (e.g., the NRC wind tunnel). In addition, although there are research laboratories in the U.S. that could provide comparable services, the physical distance and difficulty of transacting with U.S. entities would be challenging. As one interviewee pointed out, "In my opinion, NRC is the only game in town. […] We would be in pretty big trouble without them."

Similar to DND, Transport Canada clients also agreed that an important benefit of the AST Portfolio is that it acts as a full-service deliverer. Portfolio staff have the technical expertise to develop a research plan, execute it, and write up or present the information in a professional way to Transport Canada stakeholders. Finally, Transport Canada representatives (as well as other public sector clients) also stressed the importance of having access to a credible and objective third-party to support evidence-based regulatory work, and noted that NRC's expertise is recognized in Canada and internationally, in academic, government and industry circles.

3.1.5 Demand for NRC Surface Transportation Related Services

Key Finding 4: Historical data reveals a consistent and, in some cases, growing demand for NRC surface transportation related services.

The need for NRC surface transportation related services can be shown, in part, by the steady growth in CSTT gross sales revenues observed between 2002-03 and 2011-12 (see Figure 5). However, it is important to note that much of this growth (81%) is attributable to revenues from DND (for work with the RVMS/Ground Vehicles group), which increased by over $10 million (or 165%) during this period. In contrast, demand for Rail services has remained relatively stable.

Figure 5  : CSTT Gross Sales Revenue

Long description of Figure 5  : CSTT Gross Sales Revenue

 

2001/02	10,529,592
2002/03	11,306,035
2003/04	13,674,057
2004/05	15,050,518
2005/06	16,714,649
2006/07	16,460,852
2007/08	18,568,244
2008/09	19,149,740
2009/10	23,383,243
2010/11	21,776,329
2011/12	23,821,357	2.262(growth)

Source: NRC Finance Branch / SIGMA

3.2 Alignment with Government and NRC Priorities

3.2.1 Alignment with NRC Priorities

Key Finding 5: As a Technology Centre, the CSTT's alignment with NRC priorities was less clear, as this was not an explicit requirement at the time; however the AST Portfolio's new structure and proposed NRC Programs are clearly aligned with NRC's new strategy as a Research and Technology Organization.

Between 1996 and 2011, the CSTT existed as a market-based technology centre with two primary areas of focus: financial sustainability and relevance in niche markets.^{Footnote 19} The creation of the CSTT (along with five other technology centres) was motivated primarily by the imperative for NRC to meet its budget reduction targets, while minimizing the disruption to its operations and human resources. The only formal performance measures revolved around financial indicators (earned revenues, financial margins and break-even), although the NRC Office for Technology Centres^{Footnote 20} also outlined a series of other indicators, which are reflective of good business practices (e.g. cost savings to NRC, responsiveness to client expectations, the introduction and use of new management tools).^{Footnote 21}

Former CSTT staff confirmed that, as a technology centre, the CSTT received very little direction from NRC, in terms of priorities. As one interviewee stated, "NRC priorities were for the CSTT to survive and not cost money". As a result, the work of the CSTT was largely opportunistic and market/revenue-driven. Although a lot of this work could be seen as somewhat aligning to NRC's priorities, if a client was willing to pay, strategic alignment with NRC goals was not a primary consideration. As noted in the CSTT Business Plan for 2009-10 to 2011-12, the CSTT frequently had difficulty being seen as relevant to the NRC vision of "from discovery to innovation" because of its need to assure its own financial viability.^{Footnote 22} This was again confirmed in the internal interviews, where interviewees stated that, during that time, paid project work took precedence over technology development activities. It is important to note that such comments made by Portfolio staff and in CSTT documents reflect definitions adopted by the CSTT at that time, and present a fairly narrow view of what constitutes research and technology development (prototyping, engineering work, and testing solutions fall under these broad categories). As well, CSTT business plans from 2008 and 2009 do discuss the intent to progress from primarily testing and skilled technical services, which historically constituted the bulk of the work at the CSTT, toward "higher value" projects. Efforts in this area include investments in internal research and

development (R&D) conducted by the Advanced Products and Technologies group, ranging from $266,149 in 2007-08 to $739,872 in 2011-12. In addition, the RVMS and Rail divisions ramped up their own spending on internal R&D over this period. CSTT licensing income averaged $218,355 per year over the five-year period, representing approximately 1.1% of total expenditures, and was derived from two technologies, both licensed to U.S. companies (it is important to note that for the period under consideration, no A-base investments were made in the CSTT with the intent of generating licensing income). By comparison, IP and royalties represented approximately 1.2% of NRC expenditures (not counting grants and contribution expenditures) in 2012-13.^{Footnote 23}

Since the Portfolio has been brought back into the fold of NRC's core activities (April 2012), mechanisms have been put in place to ensure that, moving forward, surface transportation related projects align to NRC priorities. The most significant of these mechanisms is a business planning process whereby the work of the AST Portfolio, like that of all other NRC portfolios, must fall within the scope of programs, which are to be approved by NRC's Senior Executive Committee (SEC) following a rigorous stage-gate process. At the time of writing, the Portfolio's proposed NRC Programs for Rail and Fleet Forward 2020 are still in development. The business cases for these programs have been presented to NRC's SEC and the Portfolio is now proceeding to the preparation of NRC Program business plans. Although this is only the first stage in the program development process, it demonstrates the alignment of these programs with NRC's new strategic direction and stated goals. The proposed NRC Programs also address four of NRC's six stated critical issues for Canada:

• Economic Development, e.g. increasing the competitiveness of Canadian organizations (both public and private), through increased operational efficiency.
• Climate Change, e.g. contributions to the reduction of GHG emissions through improved transportation fuel efficiency.
• Security, e.g. enhancing DND capabilities and readiness, and providing for safer transportation through support to transportation-related public policies and through the design of safer vehicles.
• Communities, e.g. working with transit and passenger railway operators, as well as emergency services organizations, to improve their fleets, thereby improving the availability, reliability, and cost-effectiveness of their services.

Key Finding 6: Although the CSTT's work for foreign clients has proven to be a good source of knowledge, intellectual property and partnership for NRC, almost two thirds of its rail work was international and the benefits to Canada derived from these projects are not apparent in all cases.

One area where the alignment with NRC's strategy is not as apparent is with respect to international clients. The CSTT, and particularly the Rail division, undertook a significant amount of international work (in 2011-12, 63.5% of Rail revenues were derived from international clients). This is largely explained by the fact that Canada operates within a North American rail system; tracks and rail cars are shared and there must be harmony in rail regulations on both sides of the border.

It must be recognized that work for foreign clients helps to establish a strong international reputation in this domain and can be a good source of knowledge, intellectual property and partnership for NRC. Some interviewees identified benefits that can be, and have been, gained from international work conducted by the CSTT. One of these benefits is that work with U.S. and other foreign clients allowed the Centre to generate revenues, as well as be exposed to new knowledge and technology, which it can later leverage to the benefit of Canadian clients. Another benefit is that the CSTT, and now the AST Portfolio, has been able to establish arrangements with both Canadian and international stakeholders, in order to pool resources and expertise in an effort to solve problems that are not unique to Canada. One example of this is an arrangement with Transport Canada, the Federal Railroad Administration in the U.S., Canadian National Railway, and Canadian Pacific Railway, which allowed for cooperative research aimed at improving the safety and productivity of North American rail services by studying friction, rolling contact fatigue, and wear in rail and wheel.

It should also be noted that an NRC International Framework was approved by NRC's Senior Executive Committee in 2012. This framework recognizes the critical importance of international engagement to NRC's success and provides some direction for decision making with respect to international strategic alliances and partnerships.^{Footnote 24}

Regardless of alignment to NRC priorities, it must be recognized that, as a cost-recoverable organization, one of the main impetuses for the CSTT to engage in international work was the opportunity to generate additional revenue, while at the same time establishing its reputation as a world-leading centre for transportation technologies.^{Footnote 25} Although much of the international work conducted has resulted in benefits to Canada, interviewees identified isolated cases where the benefits to Canada derived from these projects were not apparent and others where the benefits to Canada appear to have been arrived at as a result of happenstance, rather than strategic intent.

3.2.2 Alignment with Government of Canada Priorities

Key Finding 7: Current AST Portfolio surface transportation related objectives and activities align well with Government of Canada priorities.

Evaluative evidence gathered through interviews, documentation, and case studies confirm that, in addition to supporting the mandates of its federal clients (e.g. DND, Transport Canada, Canadian Space Agency, and RCMP), the surface transportation related activities and objectives of the AST Portfolio specifically align to the following federal priorities:

• National Security: The AST Portfolio's work with DND enhances the department's capabilities and readiness, enabling more effective deployments of military personnel and assets. This is very much in line with the federal government's Canada First Defense Strategy (announced in 2008), which outlines a comprehensive, long-term plan to provide the Canadian Forces with the people, equipment and support they need to carry out their core missions in Canada and elsewhere in North America and abroad.^{Footnote 26} The Portfolio's work in support of National Security is illustrated in one of the evaluation case studies; that is, the development of Tactical Lightweight Radio Rebroadcasting systems for DND.
• Environment: AST Portfolio technology thrusts in the areas of Power Management and Enhanced Aerodynamic Performance are both expected to lead to fuel savings, thereby supporting Government of Canada commitments under the Copenhagen Accord, with respect to the reduction of GHG emissions.^{Footnote 27} The potential for such benefits is illustrated in one of the evaluation case studies; that is, the development of an effective and advanced idle reduction solution for the Ottawa Police Service that could later be deployed to other fleets in Canada.
• Economic Development: As noted above, much of the work conducted by the AST Portfolio (both for Rail and Fleet Forward 2020) is focused on finding ways to lower costs and improve transportation industry operations, from a safety, efficiency and reliability standpoint (ultimately resulting in enhanced competitiveness). This work is well aligned with the Government of Canada's National Policy Framework for Strategic Gateways and Trade Corridors, which was released in July 2007. The Canadian government recognizes the importance of maintaining a stable supply chain network within Canada, especially given the uncertain impact the Panama Canal expansion may present to Canadian shippers and carriers. The Framework was developed to improve the capacity and efficiency of the country's transportation system (which includes the railway network) to support international trade, thereby advancing the competitiveness of the Canadian economy.^{Footnote 28} The text box below presents information drawn from one of the evaluation case studies, which illustrates how the work of the CSTT has supported this federal priority.

3.3 Role of Government

Key Finding 8: The Federal Government, and more specifically NRC, is the most appropriate delivery agent for many of the surface transportation related activities undertaken by the AST Portfolio; however, there are isolated cases where work conducted by the CSTT may have been more appropriate for the private sector.

AST Portfolio surface transportation related activities are generally in line with the mandate of NRC, as it relates to the following responsibilities, included in the NRC Act: "undertaking, assisting or promoting scientific and industrial research in fields of importance to Canada", and "providing vital scientific and technological services to the research and industrial communities". The two operating models proposed as part of the business cases for the Rail and Fleet Forward 2020 NRC Programs align well with this mandate. The first, collaboration, involves a risk-sharing, strategic technology approach involving multi-party collaborations, in order to develop and deploy technology capabilities to meet Canadian market needs, while fee-for-service refers to testing and applied engineering solutions provided to clients on a cost-recovery basis.

Almost all interviewees agreed that the work of the Portfolio is generally consistent with federal roles and responsibilities, particularly in relation to supporting the work of other government departments (including support to policy/regulatory decision-making, a role that is not appropriate to be conducted by the private sector) and providing access to specialized facilities. The evaluation case studies serve to illustrate the value-added of NRC. Two of these studies (along with a number of external interviews) highlighted cases where clients highly valued the ability to present information that was supported or developed by the CSTT to their stakeholders. The reason for this value is the perceived objectivity and credibility that NRC and the Portfolio have within the industry. Others drew attention to the fact that the Portfolio offers unique facilities and expertise, in specific areas, that are not readily available elsewhere.

However, a few internal interviewees felt that some of the fee-for-service work may be less appropriate for government, and several external interviewees raised concerns about the potential of the Portfolio encroaching on the work of the private sector.^{Footnote 29} This concern was also made apparent in one of the evaluation case studies, where a project that began as prototyping of a specific piece of equipment (this included building 10 initial units), led to follow on work for the production of 25 additional units, which may have been more appropriate for the private sector. It should be noted that NRC currently does not have a corporate policy or philosophy that defines what constitutes R&D and/or prototyping work, and at what point such work falls crosses into the area of production or manufacturing (and thereby falls outside the mandate of NRC).

4. Findings: Performance

The evaluation assessed the past performance of the CSTT against its objectives over the evaluation period, i.e., financial sustainability (section 4.1) and ensuring value to clients (section 4.2). In addition, section 4.3 examines whether any unintended outcomes have resulted from NRC's surface transportation activities.

4.1 Revenue Generation and Cost Recovery Operation

Key Finding 9: The CSTT generated high revenues and financial surpluses in each of the past five years.

The CSTT successfully generated sufficient revenue and managed its business to operate on a cost-recovery basis (as per commitments outlined in the original agreement with NRC); in fact, the CSTT generated financial surpluses (included as part of NRC corporate accounts at year-end) in each of the past five years. In 2011-12, the CSTT generated about 14% of NRC's total external revenue, second only to the former NRC Institute for Aerospace Research (IAR, which has since been renamed the Aerospace Portfolio), which generated about 20% of NRC's external revenue. However, in making this comparison, one must keep in mind two important considerations. Firstly, at the time, IAR had approximately 380 FTEs, compared to the CSTT's 123 FTEs. It is also important to take into account the fact that the CSTT charged full cost rates to its public sector clients, since it received no direct NRC A-base.

Most of the revenue generated by the CSTT came from its project work for clients, although it also generated additional revenue through licensing of some development work (approximately $218,355 annually over the past five years). As the Portfolio plans to undertake more development work in the future, it is expected that licensing revenue will continue to increase and re-investment into future research and development programs will be enhanced. The Portfolio is currently pursuing the industry commercialization of a technology and the associated licensing opportunities from one of the projects included as a case study: the development of an effective and advanced idle-reduction solution for the Ottawa Police Services.

Revenue generated by the CSTT increased over the past five years, primarily due to additional revenue generated by the RVMS (Ground Vehicles) group through its work for DND (see Figure 6, on the following page). While overall revenues increased, the number of distinct clients decreased from 62 in 2007-08 to only 37 in 2011-12. The organization therefore appears to have focused on more significant relationships with fewer clients, as opposed to undertaking a larger number of small projects with distinct clients. As discussed in Section 2.5 Client Profile, the CSTT's largest client, by a significant margin, was DND, representing 69% of revenues in 2011-12. Other public sector clients made up an additional 15% of revenues, with the remainder coming from Canadian industry (7%) and international clients, both public and private (9%).

Figure 6: CSTT Revenue

Long description of Figure 6: CSTT Revenue

 

Earned Revenue	2007-08	2008-09	2009-10	2010-11	2011-12
RVMS	14,356,636	14,223,295	19,288,809	17,126,781	19,509,583
Rail	3,236,405	3,462,196	2,428,091	3,396,489	3,206,831

Source: CSTT Financial Statements (2007-08 to 2011-12)

As the CSTT operated in many ways as a business, it used (and the AST Portfolio still uses) business measures to track its progress throughout the year and to ensure that it is on track to generate sufficient income to cover its costs. Some of the key measures include contribution margins, the calculation of financial surpluses, and tracking of both direct and indirect costs. Contribution margins (the difference between the organization's revenues and its direct costs) are useful in computing a business' break-even point, and therefore forecasting profitability. The Portfolio calculates specific contribution margins for every project, and project managers are held to these. For the CSTT overall the target project contribution margin has historically been 45%, in order to sufficiently cover overhead expenses and generate a small surplus to be set aside to offset potential downturns in revenue. In actual fact, margins have increased slightly over the past five years, growing from 38% in 2007-08 to 43% in 2011-12. The average annual contribution margin for the CSTT from 2007-08 to 2011-12 was 42%, with the RVMS group averaging 41% and Rail averaging 51% over the same period (lower contribution margins for RVMS are partly explained by the fact that rates charged to DND have historically been lower, given the relatively low level of overhead – e.g. marketing and sales – required to generate business with DND, compared to business with other clients, and particularly with new clients). In comparison, the NRC Canadian Hydraulics Centre (which also operated as a technology centre up until 2012, and thus presented its financial data in the same way as the CSTT), reported an average annual contribution margin of 50.5% between 2008-09 and 2011-12.^{Footnote 30}

The CSTT realized positive financial surpluses in each of the past five years, supporting the organization's ability to remain cost-neutral. Fluctuations from year to year appear to be largely due to increased costs (including the transfer of expenditures related to employee benefits from NRC to the CSTT in 2009-10), as the gross revenue generally continued to increase in years when the financial surplus declined (see Table 1, on the following page).

Table 1: CSTT Financial Surplus and Gross Revenue (2007-08 to 2011-12)

Year	2007-08	2008-09	2009-10	2010-11	2011-12	TOTAL
Financial Surplus	$491,338	$1,202,274	$989,202	$319,346	$927,085	$3,929,246
Gross Revenue	$18,356,782	$18,431,619	$22,768,051	$21,643,640	$23,495,045	$104,695,137

Source: CSTT Financial Statements (2007-08 to 2011-12)

4.1.1 Factors Enabling CSTT Revenue Generation

Project Management

Key Finding 10: Internal interviewees and NRC documents identified project management as having been a key strength of the CSTT, although it was not implemented consistently across the Centre.

Many factors enabled the CSTT to generate revenue over the past five years and to operate as a cost-recoverable centre within NRC. The factor noted most often by interviewees was the CSTT's use of business practices and measures to support its operations. This includes a focus on the measures, noted in the previous section, to ensure that the organization is on track to meet its revenue targets, as well as the strong project management practices applied to individual projects. Project management was raised by many internal interviewees (9 of 13) as being a best practice for the CSTT. This was noted particularly for the Ground Vehicles/RVMS group, given that they have historically been involved in a large number of projects with complex work breakdown structures and project requirements that often evolve rapidly. These have necessitated a greater attention to project management processes, as opposed to Rail projects, which were smaller in number and tended to have fairly sequential work breakdown structures. Particular strengths of the CSTT's project management practices include:

• Skilled project managers (some projects are managed by dedicated project managers, while in other cases technical staff have the required skills and prefer to take on this role themselves);
• Strong financial tracking and financial systems, which lead to a high level of cost awareness;
• Resource planning and load-leveling; and
• Material flows and planning, as well as inventory control.

A report prepared by the NRC Investment Plan and Management of Projects Office in April 2010, regarding the Organizational Project Management Capacity Assessment for the CSTT, also noted that "they have an excellent core project management capacity" and that "clearly, project management is a big part of CSTT work practices". The report went so far as to suggest that "eventually, the NRC project management working group^{Footnote 31} would benefit from having a project manager from CSTT".^{Footnote 32} In fact, the CSTT's expertise in project management has been shared, to some extent, with other areas of NRC through the use of staff interchanges or assignments.

Still, some internal interviewees pointed out inconsistencies, in terms of how project management practices were implemented across the Centre, and reported that these inconsistencies occasionally affected client service and communication. In addition, one interviewee described some project management practices as being a hindrance, due to the level of bureaucracy involved and the confusion associated with the role project managers are expected to play on projects.

Internal interviewees also noted that a certain level of duplication exists as project managers currently enter project information into the SAP Project System module, in line with corporate processes and reporting requirements, but also continue to maintain separate systems in Microsoft Project, since, in their view, these allow for a more precise tracking of external project costs.

Key Finding 11: In addition to strong project management practices, many other factors enabled the CSTT to generate value for clients and associated revenues including: a strong client focus; the positive reputation of NRC; its ability to charge full market rates to its public sector clients; and the historically high engagement of CSTT personnel.

Client Focus

Beyond the project management practices implemented within the CSTT, a variety of other factors were noted as enabling the generation of high revenues. The strong client focus of the organization, in terms of the culture, personnel, and practices in place, was noted as a critical factor. The high level of client satisfaction is discussed in more detail in the following section; however, internal interviewees noted that the message within the organization had historically been to focus on serving client needs and ensure that they were satisfied, above almost anything else. This high client focus is also evidenced by the strong relationships that have been developed with many key and long-standing clients. Internal and client interviewees did mention, however, that in more recent years, the ability to attend to client needs has been diminished somewhat, as more processes and accountability measures (some government-wide) have been put in place (e.g., travel expenditure restrictions, centralized project management processes, more strictly regulated procurement processes).

NRC Reputation and Credibility

The NRC reputation and the level of credibility associated with the NRC name have also supported the CSTT's ability to generate revenue. Clients have sought the services of the CSTT in cases where it was important for the project results to be recognized as coming from a neutral, third party. Several instances of this were identified through the evaluation, including in two of the case studies, which illustrate how results have been used to inform policy and regulatory development, or how the involvement of NRC helped to ensure that findings were accepted by potentially unreceptive stakeholders. One case study, summarized below, describes how NRC's reputation and credibility were noted as important assets in the CSTT's conduct of the project.

Case Study Excerpt: Study for the Ontario Ministry of Transportation

The Ministry of Transportation of Ontario (MTO) contracted the CSTT to undertake a study on the safe integration of electric low speed vehicles (LSVs) into mixed traffic, despite the fact that the Road Safety Policy Office of MTO had internal resources who could also conduct research to support policy development. As the subject had garnered both public and political attention, MTO opted to have the work conducted by the CSTT, due to both the expertise of its staff as well as the credibility associated with a third-party NRC study. The study did not support the political agenda at the time; however, it was accepted by stakeholders and subsequently used to redesign the next stage of the pilot program. The report is still publicly available on the MTO website.

Ability to Charge Market Rates

As a technology center, the CSTT (at the time) received no direct NRC A-base. It had therefore been granted authority to charge market rates to its non-governmental clients and to recover full-costs from OGD clients.^{Footnote 33} As a result, OGD rates charged by the CSTT have been estimated by Finance Branch as being approximately 30-40% higher than those charged by the rest of NRC. Given that over 80% of the CSTT's work was for public sector clients, this higher charge-out rate has been a significant factor enabling the organization to generate high revenues.

ST Portfolio Staff Engagement and Motivation

Another significant factor that enabled the CSTT to generate strong revenues and remain fiscally independent was its staff. This factor was identified through a variety of evaluation methods. Internal and external interviewees commented that CSTT staff were very motivated to succeed as a revenue-generating centre, and driven to meet their targets. This is not a new development as it was also raised in 2000 in a University of Ottawa review of NRC's Technology Centres, which included the CSTT. As noted in the review:^{Footnote 34}

[…] for the most part employees enjoy a sense of accomplishment and many have developed new skills in critical areas such as project management and client services. A spirit of "entrepreneurship" predominates.

More recently, the high level of satisfaction of CSTT staff was apparent in the 2011 Public Service Employee Survey, illustrated through the selected results in Table 2. CSTT staff were much more positive than both NRC and the Public Service overall in many areas, including both their work and work conditions.

Table 2: Selected Results of 2011 Public Service Employee Survey

Question	Proportion of Positive Answers (Strongly Agree or Somewhat Agree)
	CSTT	NRC	Public Service of Canada
I get a sense of satisfaction from my work.	91%	81%	76%
In my department or agency, I have the opportunities I need to implement ideas on how to improve my work.	84%	65%	61%
I have opportunities to provide input into decisions that affect my work.	85%	70%	68%
I get the training I need to do my job.	92%	63%	69%
I have support at work to provide a high level of service.	91%	74%	75%
I would recommend my department or agency as a great place to work.	85%	61%	64%
I am satisfied with my department or agency.	78%	58%	65%

Source: 2011 Public Service Employee Survey

4.1.2 Factors Hindering CSTT Fiscal Stability

Key Finding 12: The CSTT's focus on fewer clients made it vulnerable to the priorities and needs of these clients. Opportunities exist for the AST Portfolio to diversify its client base in the area of surface transportation.

As some factors enabled the CSTT's revenue generation and fiscal stability, other factors also impeded its ability to generate revenue. Some of this is due to the organization's reliance on a limited number of clients, whose needs can sometimes be quite volatile. For example, the cyclical nature of the rail industry was raised repeatedly by interviewees (specifically that the industry operates according to approximately a 7-year cycle and that, at several points during this cycle, firms can be reluctant to invest in R&D). This was identified as a contributing factor to the decline in rail revenue in 2009-10 (from $3.4M the previous year to $2.4M), although it then recovered in 2010-11.

As a large proportion of the CSTT's activities were for public sector clients, its business was also greatly affected by other government priorities. One notable example is the CSTT's work for DND, which increased to 78% of the CSTT's revenue in 2009-10, in large part due to DND's operations in Afghanistan, but then dropped from $18.2M to $14.5M in 2010-11. It is unknown at this time whether DND priorities will result in future declines in revenue. However, as some DND interviewees stated, while the combat mission in Afghanistan has ended, the Department is now seeking to "catch up" on much of the work that had previously been deferred, as higher priority projects in support of the mission took precedence.

Non- (or potential) clients who were interviewed as part of the evaluation, particularly those outside of Ontario, noted that they were not aware of many of the services offered by the CSTT. However, after being informed about these services over the course of the interview, most could foresee working with the Portfolio in the future, if they had a better understanding of its service offerings.

4.2 Client Satisfaction and Client Benefits

4.2.1 Client Satisfaction

Key Finding 13: Clients have been very satisfied with the services provided by the CSTT.

The evaluation assessed client satisfaction through a variety of methods, all of which demonstrated that overall, clients have been very satisfied with the services they have received from the CSTT. Client interviewees were asked to rate their level of satisfaction with the CSTT against a set of criteria; the results are presented in Table 3. Clients generally had a very positive view of the CSTT and of the services received, and any negative issues raised were relatively minor. Some of the issues raised included:

• Staff being spread thinly;
• Some examples of individual delays (not overall timeliness); and
• Facilities seen as "functional, but not state-of-the-art."

Several client interviewees noted the importance of security, confidentiality and discretion, both in terms of national security as well as the protection of IP when they are accessing services and sharing sensitive information. They commented that this has been an issue in some of their interactions with other organizations, but that this was a definite strength of the CSTT and that staff was very conscientious about protecting client information.

Table 3  : Interviewee Client Satisfaction

Criteria	Interviewee Rating*
Ability of CSTT staff to understand your organization's needs	4.5 (n=19)
Availability of scientific / technical competencies to meet your organization's needs	4.4 (n=19)
Availability of CSTT staff to meet your organization's needs	3.7 (n=19)
Ability of CSTT staff to solve real world problems	3.9 (n=19)
Timeliness of services, deliverables	3.7 (n=19)
Access to world-class infrastructure and equipment	3.9 (n=16)
Quality of data / results	4.4 (n=19)
Confidentiality and security of information	4.6 (n=17)
Overall quality of services	4.2 (n=19)

*1=not at all satisfied, 5=very satisfied

The CSTT did not collect client satisfaction data in a systematic way, although there have been one-off efforts to gauge client satisfaction over the years. Some feedback from a limited number of clients (12) was collected in June 2003, by the Business Development Officer at the time. Comments from this exercise echo those raised in the interviews and case studies: clients were satisfied with many aspects of the work conducted by the CSTT, including the access to unique facilities and specialized technical expertise, the value of the NRC brand and reputation, and the fact that staff were knowledgeable and easy to work with. Some also commented on the age of some of the facilities ("some are older, but get the job done"), and others noted that there appeared to be a lot of overhead. Since 2003, some post-project feedback forms were implemented on a project by project basis; however, only a small handful of completed forms were returned to the CSTT.^{Footnote 35}

Repeat business was noted by internal interviewees as being a strong indicator of client satisfaction. Of the CSTT's 269 distinct clients from 2002-03 to 2011-12, at least half were repeat clients (that is, they undertook more than one project with the CSTT during this time). As there are other reasons beyond dissatisfaction for clients not returning to the CSTT for other services (e.g., initial access was due to a unique need, reduced spending in R&D), this indicator does not imply that clients who only accessed services from the CSTT once were dissatisfied; rather, that those who did return were satisfied with the services. The proportion of repeat clients has been identified as a performance indicator for NRC. It will be tracked in the future and used to assess NRC's performance.

4.2.2 Client Benefits

Key Finding 14: CSTT projects are seen to have achieved clear benefits for clients, and contributed to the achievement of higher-level outcomes.

While the objectives of the CSTT primarily revolved around fiscal stability, its work also responded to client needs and produced client benefits. This is not surprising, as clients, particularly those in the private sector, would not pay for these services if they did not experience clear benefits as a result. All ten of the CSTT projects examined as part of the case studies successfully met their objectives, and resulted in various benefits for both public and private sector clients. The CSTT's work for private sector clients contributed to product development and improvement, resulting in increased capabilities, better operation, increased revenues and reduced operating costs for these clients. Work for public sector clients has supported OGD priorities and mandates (e.g., DND, Canadian Space Agency), and several clients used the information provided by the CSTT to support evidence-based decision-making and policy development. The case studies also illustrate how CSTT projects have contributed to higher-level impacts for Canada. These include environmental, safety, and economic impacts (examples of these impacts, as well as others raised through the interviews and document review, are presented below).

Contributions to Higher-Level Impacts made by CSTT Projects

Environmental Impacts: reduced fuel consumption and emissions (anti-idling projects, NRC-designed side guards)

Safety Impacts: improved safety on roads and rail from work done for MTO and Transport Canada, increased safety of deployed forces stemming from work for DND

Economic Impacts: increased revenues due to improved products, support to Canadian trade routes through work to prevent outages along key rail lines

Work performed by the CSTT has been viewed as being of value to many of its clients; so much so, that some public sector clients have given or shared awards with CSTT staff, in recognition of their work. For example, in April 2008, DND presented CSTT staff with a Value for Canada award, in recognition of their work to support operations in Afghanistan. In another case, the Canadian Space Agency's Exploration Ground Infrastructure group was presented with an Exceptional Achievement Award and chose to share it with CSTT staff as a result of their contribution to the project.

Key Finding 15: There has been added value stemming from the CSTT's involvement in client projects, and, in many cases, client needs could not have been met as easily in the absence of the CSTT.

For the projects examined as part of the case studies, most clients highlighted specific reasons why the CSTT was best suited to undertake the work and, in some cases, was the only option available. Of the ten projects examined through the case studies:

• Two (2) would not have been completed in the absence of the CSTT;
• Six (6) could likely have been done through other means, but with an impact on the project's quality and/or efficiency; and
• Two (2) could have been done by other organizations (e.g., TTCI, private sector), and it is unknown whether there would have been a negative impact on the project.

Internal and external stakeholders noted that there are organizations that provide similar services or have comparable facilities, but that the CSTT had unique qualities or delivered certain efficiencies. These included the Centre's unique cold-weather facilities and expertise and their ability to deliver "turnkey" or "full-circle" solutions, as opposed to clients having to search out different options for each component of the project (e.g., design, prototyping, testing, manufacturing). As noted earlier, NRC's reputation for neutrality and the quality of the work completed were critical factors for two cases, and helped present the project deliverables as credible sources of information to the involved parties.

Some organizations with in-house capabilities chose to come to the CSTT because: a) they did not have the precise expertise and capabilities; b) there were reasons why an outside perspective was more appropriate; or c) they did not have the capacity to undertake the work. All of these reasons were illustrated through one or more of the case studies.

Other factors that were noted regarding the relative benefits of the CSTT include: its location, which is viewed as convenient for many of its larger clients (a significant factor when considering the costs of transporting large vehicles); its ability to coordinate and secure access to other government facilities and competencies, both at NRC and within other government departments (e.g., climatic chamber at Environment Canada, the RCMP testing facilities at TPOF); and the ease of contracting, both in setting up contracts between NRC and the client, as well as NRC setting up contracts with other third parties for subcontracted work. The value of DND being able to access NRC services more quickly than many of their private sector options was noted as being a critical factor by many DND interviewees. These interviewees stated that delays in some of their projects would have increased the risk of injury and/or death to deployed personnel.

There were examples noted in the case studies and interviews of projects, or components of projects, undertaken by the CSTT, which could have been completed by other organizations, including the private sector. In these cases, there were often reasons why it was logistically easier for the client to get the work done through the CSTT, as opposed to a private sector option; however, as discussed earlier in Section 3.3 Role of Government, this work may not be appropriate for the Portfolio to undertake on a regular basis.

4.3 Unintended Impacts

4.3.1 Unintended Positive Impacts

Work performed by the CSTT has, in some cases, led to additional impacts, over and above those originally intended for the projects. One such impact is that personnel have developed unique expertise through project work, which has enabled them to respond to needs in specific domains (e.g., cold weather testing of fuel cells), and which they can now market to new clients. Similarly, the CSTT has been able to find additional applications for deliverables beyond those included in the original project scope. Some of these have been additional applications for the original client, and in other cases, CSTT staff modified deliverables for use by different clients and in different environments (e.g., DND components which were redesigned on a smaller scale to fit into complex systems have subsequently been used in other DND projects requiring similar components).

The CSTT also developed strong relationships with key clients (e.g., DND, Transport Canada), and the Portfolio is now viewed by these clients as being a "trusted technology partner". These relationships have enabled the Portfolio to become specialized in the unique needs of its key clients and allow the Portfolio to support them in a variety of ways beyond simple project work (for example, involvement in longer term projects, and participation on advisory committees for the other departments). Interviewees from Transport Canada said that they had recently discussed what the impact on their organization would be if they could no longer access the services of the Portfolio. They determined that it would take the Department about three years to build a similar relationship with another service provider, and that this would have significant impacts on the department's programs, including the 5-year ecoTECHNOLOGY program that the Portfolio is currently supporting.

The CSTT's involvement in high-priority projects, including its work for the Ottawa Police Services (anti-idling systems) and for the Ontario Ministry of Transportation (examining the safety considerations of integrating electric low-speed vehicles into traffic) resulted in positive media coverage for the client, the CSTT, and NRC overall. Internal interviewees highlighted the importance of demonstrating to the public the relevance of NRC, through its involvement in current issues that are in the public eye.

Finally, interviewees suggested that increased awareness of rail transportation sector needs, generated partly through the work of the CSTT, may have been a factor that led to the creation of the Canadian Rail Research Laboratory, a partnership between the Government of Canada, Alberta Innovates, the Association of American Railroads, Canadian Pacific Railway, Canadian National and the University of Alberta, which aims to conduct specialized research and educate qualified engineers to meet the future needs of the Canadian rail sector.

4.3.2 Unintended Negative Impacts

There have also been negative impacts as a result of the CSTT's operations. One such impact is the feeling of isolation experienced by staff, due to their geographic and, in the past, organizational, separation from NRC.

A few internal and external interviewees also noted that perceived competition with private sector firms, particularly within the context of the CSTT's work for DND, may have damaged relationships with some of its private sector clients who also perform work for DND.

5. Findings: Resource Utilization

5.1 Economy and Operational Efficiency

Key Finding 16: As a technology centre, the CSTT operated efficiently and economically, taking steps to minimize overhead costs, and these practices carried over into the operations of the new Portfolio.

Internal and external interviewees generally share the view that CSTT operations have historically been very efficient. Many noted that as a technology centre, the CSTT was very "business-minded" (highly focused on contribution margins, profitability, keeping overhead low, etc.). More than half of the clients interviewed supported this view by stating that the CSTT was at least as efficient as a private sector service provider, if not more.

Internal interviewees stated that within the CSTT, overhead/administrative costs were avoided to the extent possible. Over the past five years, the ratio of indirect and corporate costs as a proportion of total expenditures (i.e. overhead) has fluctuated between 36% (2007-08) and 43% (2010-11). By comparison, the same ratio for the NRC Canadian Hydraulics Centre (which also operated as a cost-recoverable technology centre up until 2011-12) averaged 48% between 2008-09 and 2011-12.^{Footnote 36}

Similarly, the CSTT has historically been mindful of staff utilization (in terms of hours billable to projects) and individual utilization rates are a performance target that every employee is heavily encouraged to meet. Historical data for the CSTT reveals a declining trend in the proportion of net externally billable hours over the past five years. For research staff in the RVMS (Ground Vehicles) division, the ratio of net externally billable hours fell from an average of 79.9% in 2007-08 to 64.9% in 2011-12. For research staff in the Rail division, this ratio went from 70.8% to 44.2% over the same period. Portfolio representatives explained that this decline was in part due to increased resources applied to CSTT internally funded technology development projects (for example, Rail division staff worked to refine a rough prototype of autonomous instrumented wheelsets, while R&D work on the Ground Vehicles side was largely focused on the development of idle-reduction technology). Consequently, the reporting of such projects had the impact of reducing overall externally funded utilization for many employees. Another factor noted, which contributed to a substantial portion of the dip in utilization for Rail division staff in particular, was the recession that occurred in 2008. Although business with OGD clients remained relatively steady, business with Rail corporate clients over the following years declined dramatically.

Figure 7: Average Net Percent Externally Billable (Research Staff Only)

Long description of Figure 7: Average Net Percent Externally Billable (Research Staff Only)

 

	2007-08	2008-09	2009-10	2010-11	2011-12
Rail	70.79%	59.42%	36.22%	38.67%	44.16%
RVMS	79.87%	76.14%	78.20%	64.90%	64.92%
Total	76.34%	71.53%	61.87%	52.95%	53.23%

Source: CSTT Staff Utilization Data (2007-08 to 2011-12)

Key Finding 17: Portfolio office/lab space is near full capacity.

AST Portfolio staff in Ottawa operates out of three buildings classified as office/lab space by NRC's Administrative Services and Property Management (ASPM) Branch: U-87, U-88 and U-89. Office space efficiency, as calculated by ASPM, for U-87 and U-88 is 101% and 126%, respectively. ASPM's target for "utilized" space is 75% - 95%.

Prior to Fall 2012, part of the RVMS/Ground Vehicles group occupied space at a separate location in Ottawa's west end (they had originally located there because of space issues and storage needs). In Fall 2012, management decided to move all staff back to U-89. U-89 therefore had an office space efficiency of 83% prior to the repatriation of staff from the west end location (Fall 2012) and 99% after the move.

The near-maximization of office/lab space is not common for NRC, as the NRC average for office/lab space efficiency is 68%. As well, only eight of NRC's 56 office/lab buildings across Canada have space efficiency ratios over 100% (this includes two of the three AST Portfolio buildings identified above).

To date, the near-maximization of space was not raised by interviewees as being a significant issue. Staff noted that a lot of work is carried out "on the shop floor", and so the Portfolio makes use of drop stations/hotelling, where possible, in order to operate within space constraints. Representatives from ASPM Branch also do not consider these buildings to be a concern from a safety and occupancy perspective. However, it is unknown at this time where the Portfolio would house the additional staff that is presumed to be needed to support the expected future growth proposed as part of the business cases for the Rail and Fleet Forward 2020 NRC Programs.

Key Finding 18: While the CSTT operated efficiently and attempted to maximize economy in the past, its inability to access working capital from year to year necessitated that the organization place a considerable focus on cost reduction measures, sometimes to the detriment of its effectiveness.

Some internal and external interviewees noted that, while the CSTT operated efficiently in the past, it may also have been "too lean" in some respects. This is likely as a reaction to the fact that the organization historically did not have access to working capital early in the year, and therefore could not afford to take any risks beyond what it knew was available as cash on hand from current year business operations. One impact of this, noted earlier, was the CSTT's primary focus on paid project work, and accordingly, more limited attention to strategically relevant technology development projects.

Almost half of internal interviewees believed that there was little or no duplication of effort within the CSTT. Although this is generally perceived as an indicator of efficiency, two interviewees pointed out the downsides associated with the lack of duplication and reliance on single individuals for specific expertise. As one interviewee noted, "If someone takes a holiday, gets sick, or goes on language training, then we're stuck and delays occur." A similar sentiment was also echoed by one of the clients interviewed: "[Portfolio staff] have a lot of projects on the go and sometimes they get spread very thinly. It can be hard to access the guy you need to help you, because he's working on another project when the testing is taking place."

Limited administrative staff within the Centre was also raised by many interviewees as an example of the CSTT's attention to cost control. However, some pointed out that this may occasionally result in inefficiencies, as more administrative work is taken on by senior level staff.

Finally, internal interviewees also perceived the CSTT as having been quite frugal when it came to investing in facilities. They noted that as a cost-recoverable organization, overhead costs were frequently throttled back, particularly in the early part of the fiscal year, until good visibility of the revenue picture could be assured. This, coupled with corporate controls on capital spending, meant that spending on facilities was limited to essential repairs and/or minor improvements, which could be paid for using current year surpluses. As a result, the Portfolio has now reached the point where some of its main facilities (most notably the 30-meter long Environmental Chamber and the Vehicle Dynamics Facility) are now thirty years old and in need of major upgrades. Internal interviewees provided examples of cases where the current state of facilities (or the lack of certain functionality) has hindered work in the past, or been prohibitive to some contracts. A couple of clients also commented that Portfolio facilities are "functional, but not state-of-the-art". Under the NRC Investment Planning Framework, the Portfolio has recently prepared business cases to support major capital investments in its Environmental Chamber and Vehicle Dynamics Facility.

Overall, the technology centre model "pushed" the CSTT to more or less resemble a high end consulting group, where value for clients and fiscal stability were equally weighted. For a research and technology organization (RTO) this mechanism is weak due to the failure to invest and co-invest in the future technologies and facilities that Canada requires – without this renewal mechanism the life of the Centre in the longer term was questionable. The move to the program-based Portfolio resource structure addresses this future stability challenge.

Key Finding 19: Portfolio staff, as well as some clients, highlighted some challenges associated with increased accountability requirements and the gradual loss of flexibility in operations.

As a technology centre, the CSTT operated as a profit-motivated business within a public sector environment. In order for this model to succeed, it became important to minimize the constraints inherent in being part of NRC and the public service.^{Footnote 37} An operational framework for technology centres, established in March 1996, defines certain flexibilities and operating freedoms, with respect to the management of financial and human resources, the management of real property, and the type of work they could engage in.^{Footnote 38} Moreover, the CSTT operated with some measure of autonomy from NRC, provided it met established financial performance targets. As a result, interviewees felt that the organization operated with a high degree of efficiency and nimbleness, which allowed it to be very responsive to client needs.

Internal interviewees, as well as some clients (4 of 14), highlighted challenges that have been experienced in maintaining this same level of client responsiveness since the CSTT was brought back into the fold of NRC (April 2012). Overall, interviewees perceive Portfolio operations as having become increasingly process-oriented and lacking the sense of urgency that had developed as a technology centre.

It should be noted that the changes noted by interviewees are mainly accountability-driven, that associated challenges are largely transitory, and that these exist for all NRC portfolios. However, it appears, based on interviewee opinions, that these challenges and the perceived loss of efficiency as a result have been more apparent for the Portfolio, given that they represent a marked change from previous operations as a technology centre.

5.2 Interaction with Other Portfolios

Key Finding 20: There are opportunities for greater synergy with other NRC portfolios.

All internal interviewees felt that there was little or no duplication with other NRC portfolios. As well, the CSTT has cooperated with other areas of NRC, where the work has been seen as being complementary (examples include work with aerospace, fuel cells, information and communication technologies, and construction groups). One interviewee estimated that they interact with other portfolios for about five to eight projects a year. However, most internal interviewees stated that there could potentially be a greater level of collaboration. Some of the factors identified as having inhibited further collaboration in the past include:

• Limited communication / information sharing tools among portfolios;
• Inconsistent financial mechanisms among portfolios; and
• The nature of the work undertaken by the CSTT (i.e. other areas of NRC tended to have longer-term projects, while the CSTT had to be more responsive and quick to turn around).

One specific area where internal interviewees reported a greater potential for cooperation is with the NRC Automotive Portfolio, since some complementarities existed with respect to lightweight materials for ground vehicles (e.g. buses, trucks and rail cars). Given the recent (April 1, 2013) announcement that the ST and Automotive portfolios would merge under a single management structure, this development should partially address the above key finding. NRC's adoption of a portfolio-program interface also more easily allows for projects to be sourced from anywhere within and external to NRC to support the outcomes. This program mechanism was partly created to prevent siloing of projects within institutes and the overlap and inefficiencies which ensued.

6. General Conclusions

This evaluation demonstrates that the CSTT met, and the AST Portfolio continues to meet, an important need within the transportation sector. Its relevance is evidenced, in large part, by its ability to remain cost-neutral for almost 15 years and by the strong relationships it has developed with some of its key clients. The role played by the CSTT has been particularly critical to other government departments, such as DND and Transport Canada. These departments depend on NRC services in order to carry out their mandates, and interviewees from both of these departments have stressed how detrimental it would be to their organizations if they could no longer access these services in the manner that they are currently delivered.

As a technology centre, the CSTT operated fairly autonomously from NRC. Staff were highly motivated by the cost-recovery model, and driven to meet their targets. The performance of the organization was dependent on its ability to satisfy client needs, and this therefore became a primary element of the culture and practices in place within the organization. This desire to meet client needs, in addition to the CSTT's own need to ensure financial sustainability, occasionally led the CSTT to take on work that has been viewed by some as being beyond the mandate of NRC (e.g. encroaching on the work of the private sector). It also meant that cost-recoverable work took precedence over strategic technology development and other collaborative research.

As a technology centre, the CSTT was also granted certain operating flexibilities, which led to a high degree of nimbleness, and allowed the organization to be very responsive to client needs. Since the organization's transition to an NRC Portfolio, staff and clients noted challenges associated with maintaining the same level of nimbleness within increased accountability requirements.

Moving forward, the Rail and Fleet Forward 2020 NRC Programs proposed by the Portfolio appear to position it to continue to be relevant in the coming years, while ensuring alignment to NRC's new strategy and benefit to Canada. As NRC continues to implement the final stages of its transformation, including the recent merger of the ST and Automotive portfolios, it will be important to exercise caution in order to ensure that the strong client focus, efficient management practices, and reputation developed at the former CSTT are maintained and further enhanced.

7. Management Response

Recommendation	Response and Planned Action(s)	Proposed Person(s) Responsibilities	Timelines	Measure(s) of Achievement
Recommendation 1: Given the magnitude and importance of the international component of work within the Rail division, the AST Portfolio should implement measures, in line with the NRC International Framework, to ensure that international engagement is strategically targeted and potential benefits to Canada are considered at project outset.	Recommendation accepted. AST Portfolio project selection and execution processes will ensure that benefits to Canada are sufficiently ensured and clearly defined. Advice / support from NRC's International Relations Office will be sought as necessary. As part of the execution of the Rail program, an approval process will be developed to ensure alignment with overall program objectives and expected benefits as defined in the business plan. A post-project evaluation process will also be developed to validate the outcomes of individual projects, in alignment with expected program benefits.	R&D Director and Rail Vehicle and Track Optimization Program Leader	Implemented by April 1, 2014.	Continuous monitoring by Program Board of project outcome alignment, with records of decisions and appropriate project level sign-off by accountable authority
Recommendation 2: A)The AST Portfolio should establish procedures defining what work does and does not fit within its mandate (for example, what would constitute production or manufacturing work), and clearly communicate this position to its key clients and staff. B) NRC senior management should also consider the need for corporate guidelines to this effect.	A) Recommendation accepted. The AST Portfolio will establish the suggested internal procedures to make sure it operates within NRC's mandate and will communicate to the leadership team both the process and clear criteria to judge project fit with programs' value propositions. B) Recommendation accepted. The Vice President, Engineering and Business Management Support, will discuss with Senior Management and action accordingly.	Portfolio Business Advisor and Client Relationship Leaders, Program Leaders and General Manager VP Engineering and Business Management Support	Implemented by April 1, 2014, then on-going monitoring. April 1, 2014	Monitoring of project outcome alignment by Program Boards, with records of decisions and appropriate project level sign-off by accountable authority Decision on whether guidelines are necessary and if so, method of communication
Recommendation 3: The AST Portfolio should take steps to harmonize project management practices across the Portfolio.	Recommendation accepted. The AST Portfolio will take the necessary steps to further harmonize and strengthen project (and Program) management practices across the Portfolio. The AST Portfolio will work with NRC Program and Project Services (PPS) to identify the appropriate approach for this. It will build on the strengths and program management practices that have been demonstrated in the past and transform them to portfolio-wide practices.	Team Leader, Project Management Support Office	July 29, 2013 March 31, 2014	Establishment of AST Project Management Support Office (PMSO) Establishment of a Project Management Framework for AST, with tools and templates
Recommendation 4: Given that the CSTT's ability to generate revenue has been dependent on a limited set of clients, and given future plans to further pursue technology development activities higher on the value chain, the AST Portfolio should increase targeted outreach and communication to potential clients.	Recommendation accepted. The AST Portfolio is aware of the risks related to a limited number of clients and the need to diversify its client base. The programs' marketing and client engagement strategies comprise both the attraction of new clients and the maintenance of existing client project interface and support. AST Portfolio management will ensure that the programs' strategies are effectively deployed and client relationship management is enhanced.	Portfolio Business Advisor and Client Relationship Leaders, Program Leaders and General Manager	December 2013 June 2014 Ongoing	Programs are approved Programs' marketing and client engagement strategies are effectively deployed Client portfolio reviewed
Recommendation 5: The AST Portfolio should fully review its major facilities through the development of a Portfolio-wide asset recapitalization plan. This plan should clearly identify and prioritize capital asset needs and should include a strategy for providing for those needs over the long term.	Recommendation accepted. The AST Portfolio will develop an asset recapitalization plan.	Director of Operations	September 2014 Ongoing	Asset Recapitalization Plan completed Plan updated annually

Appendix A: Evaluation Matrix

Questions	Methods
	Document Review (Internal and External)	Administrative and Business Performance Data Analysis	Stakeholder Interviews	Case Studies
Relevance
R1. Continued Need for the Program
1. To what extent do NRC surface transportation related activities continue to address a demonstrable need?	✓	✓	✓	✓
R2. Alignment with Government Priorities
2. Do NRC surface transportation related objectives and activities align with NRC and Government of Canada priorities?	✓		✓	✓
R3. Alignment with Federal Roles and Responsibilities
3. To what extent are NRC surface transportation related activities and expected outcomes consistent with federal roles and responsibilities?	✓		✓	✓
Performance (Effectiveness, Efficiency and Economy)
P1. Achievement of Expected Outcomes
4. To what extent was the CSTT successful in: Generating revenues through the provision of services to rail, heavy vehicle and other clients? Operating on a cost-recovery basis?	✓	✓	✓	✓
5. To what extent have NRC surface transportation related services been successful in meeting client needs?	✓	✓	✓	✓
6. Have there been any unintended (positive or negative) outcomes as a result of NRC's surface transportation activities?	✓		✓	✓
P2. Demonstration of Economy and Efficiency
7. Was the CSTT able to deliver its services in the most economical manner?	✓	✓	✓
8. As a cost-recovery program, to what extent was the CSTT managed efficiently in order to optimize its performance?	✓	✓	✓	✓

Appendix B: Methodology

The evaluation of the NRC Centre for Surface Transportation Technology (CSTT) was conducted to assess the value for money of the program (i.e., relevance and performance, including resource utilization), and focuses mainly on the period of 2007-08 to 2011-12. However, more recent information relating to the organization's activities and operations as the ST Portfolio (for fiscal year 2012-13) and the AST Portfolio (2013 to present) was also included. Where appropriate and available, the evaluation is also supplemented with prior information dating back to 2002-03.

The selection of methods was based upon the most efficient means of addressing the evaluation issues in a rigorous way, while taking into account cost, time and resource constraints, as well as other considerations, such as evaluation scope, evaluation budget, and minimizing response burden. The evaluation approach and level of effort was commensurate with the program risk, which was assessed as low during an assessment conducted as part of the planning phase. As such, an objectives-oriented approach was taken, using a non-experimental, largely descriptive evaluation design.

In order to maximize the possibility of generating useful, valid and relevant evaluation findings, mixed methods were used for this evaluation, allowing for triangulation (i.e., convergence of results across lines of evidence) and complementarity (i.e., developing better understanding by exploring different facets of a complex issue). Both qualitative and quantitative methods were used, and include:

• Internal and external document review;
• Administrative and performance data review;
• Key informant interviews (internal and external); and
• Project case studies, including reviews of project documentation and both internal and external interviews.

A discussion of the approach used for each of these methods as well as their limitations is provided in the following paragraphs.

Internal and External Document Review

Internal and external documents were reviewed, synthesized and integrated into the evaluation to provide context and history, and to complement other lines of evidence in assessing relevance and performance. Internal documents reviewed included strategic and business plans (both as the CSTT and more recent plans for its operation as the ST Portfolio), performance reports, presentations, special studies, intranet articles (i.e., NRC's Zone), and others such as corporate files (meeting minutes, correspondences, etc.)
In addition, a wide range of external documentation was reviewed by the evaluation team, including but not limited to, documents produced by other government departments and central agencies, the Canadian Rail Association, and other key stakeholders. A selected list of the documents reviewedcan be found in

Appendix C: Selection of Documents Reviewed.

External documents to be reviewed were identified by a variety of sources, including: Portfolio personnel, NRC Finance Branch, the evaluation's Methodology and Subject Matter Experts (MSEs), other stakeholders, online searches, as well as NRC's Knowledge Management Branch who identified and procured relevant market/sector information.

Administrative and Performance Data Review

Administrative and performance data for 2007-08 to 2011-12 were reviewed to provide information on program outputs and contribute to the analysis of resource utilization (e.g., staff utilization rates, productivity ratios and office space efficiency ratios) and achievement of expected outcomes (i.e., financial and client data, which helped to assess the extent to which the CSTT has been able to generate revenue and operate on a cost-recovery basis). Where available and appropriate, administrative and financial data for 2002-03 to 2006-07 were also included. Administrative and performance data were provided by Portfolio staff, as well as by corporate branches including NRC's Administrative Services and Property Management Branch, Strategic and Operational Planning Branch and Finance Branch. Comparative financial data was also obtained from staff from the former Canadian Hydraulics Centre.

Key Informant Interviews

Conducting interviews with key informants is an essential element of an evaluation methodology. The information gathered through the qualitative, semi-structured interview process was based on personal experiences, opinions and expert knowledge. This information plays an important role in contextualizing performance data and other statistics.

Interviews were conducted either in-person or by telephone. Each interview lasted between 1 and 2 hours and was conducted using an interview guide. Interview guides provided the common questions to be asked of each interviewee thus ensuring that the same issues were addressed by all relevant interviewees. All interviewees received the interview guide in advance of the interview. In some cases, interviewees elected to provide their comments in writing to the evaluation team, either following the interview or in a few cases, instead of participating in an interview. The majority of interviews were individual; however, a few group interviews were conducted where interviewees felt this would be more efficient and provide a richer discussion (e.g., DND, Transport Canada, and group leaders within the ST Portfolio).

A total of 62 stakeholders were consulted through the evaluation, including 20 internal stakeholders (current and former NRC management and CSTT staff), and 40 external stakeholders (see Table 4). External stakeholders included active clients (defined as those who had had a project with the CSTT in the past two years), non-clients (those who the CSTT/Portfolio had identified as potential clients, but had not yet worked with on a project), and inactive clients (those who have accessed CSTT services in the past, but not in the past two years). A list of the organizations consulted is included in Appendix D: Organizations Consulted. Interviewees were selected in consultation with Methodological and Subject Matter Experts and Portfolio management, as well as through a review of the CSTT's revenue sources. To encourage interviewee participation, the General Manager of the Portfolio sent a memo to all potential interviewees informing them of the evaluation and advising that they may be contacted to participate as interviewees or to contribute to a case study.

Table 4 : Stakeholders Interviewed

Interviewee Category	Number of Individuals Interviewed
Internal stakeholders	20
External Stakeholders
Active clients	37
Non-clients	2
Inactive clients	3
Total	62

In order to minimize burden on clients, participants who were asked to contribute to a case study interview were not asked to be interviewed as key informants, rather, a few key questions (primarily related to relevance) were asked in addition to the questions to support the case study development and were then analyzed along with the responses of the key informants. For the Portfolio's larger clients who were included in case studies (e.g., DND, Transport Canada), the case study interviews were conducted and representatives from other parts of the organization (who had also engaged in work with the CSTT) were interviewed separately.

Case Studies

A series of 10 case studies were developed to illustrate the types of projects and activities that had been conducted by the CSTT, to identify whether client needs had been met through these projects, and to identify any unintended impacts, both positive and negative.
The projects included were not randomly selected, but rather were chosen to ensure variation against the following criteria, listed below in priority order:

1. Distribution across the CSTT's programs and key facilities;
2. Distribution between private and public sector clients;
3. Inclusion of the CSTT's primary clients, based on gross revenue.

To determine the most appropriate number of cases to be included in each of the above categories, the total gross revenue across each category was considered. The projects selected for study are included in Table 5, on the following page.

The original intent of the case studies had been to also select projects across various project types (i.e., Type 1 (research and intellectual property), Type 2 (prototyping, systems integration, engineering, and value audits) and Type 3 (testing and skilled technical work). Although CSTT business plans discussed the proportion of their activities in each of these categories, it was found that very few projects are strictly one type and the organization does not typically track their projects in these categories. As such, at the time of project selection, the evaluation team reviewed the proposed projects to ensure that there would be some with components of each type of work. The majority of the projects selected were primarily Type 2 projects, where most of the CSTT's work is expected to have occurred, although projects with Type 1 and 3 components were also included.

Table 5  : CSTT Clients Included in Case Studies

	Public Sector	Private Sector
Rail	Transport Canada	Trinity Industries
	Toronto Transit Commission	Canadian Pacific Railway
Ground Vehicles	Department of National Defence	DEW Engineering and Development Ltd.*
	Canadian Space Agency
	City of Ottawa - Ottawa Police Services	New Flyer Industries
	Ontario Ministry of Transportation

* This project included a significant climatic test chamber component.

** Blue cells represent projects with one of the CSTT's top 10 clients over the past five years.

The case studies were developed through reviews of project documentation, available external documentation (e.g., newspaper articles discussing the project or its impacts), and interviews with both internal Portfolio project staff and external client representatives. Semi-structured interview guides were developed and tailored for each client type (i.e., international, public and private). The case studies were developed using a common template and drafts were shared with each project interviewee for factual validation prior to their completion.

Limitations and Mitigating Strategies

• Changing organizational context: NRC is currently undertaking significant changes, including the implementation of a new strategy and the re-organization of its structure. Although the evaluation focused mainly on the past performance of the CSTT (primarily from 2007-08 to 2011-12), the current situation and future implications for the Portfolio must also be considered. This resulted in challenges for the evaluation team. In particular, the discussion around relevance was limited by the fact that proposed NRC Programs for Rail and Fleet Forward 2020, which will form the basis for Portfolio priorities going forward, are still in development. The draft versions of NRC Program documents were reviewed for the purposes of the evaluation; however, limited details pertaining to these NRC Programs could be shared publicly prior to their approval by NRC's Senior Executive Committee. As well, the merger of the Automotive and ST Portfolios was announced on April 1st, 2013. This announcement occurred following the data collection phase of the evaluation and has, consequently, not been reflected in the retrospective assessment of the Portfolio. The change has however been considered in the recommendations stemming from the evaluation. Overall, in order to minimize potential challenges, the evaluation team consulted extensively with methodological and subject matters, with the Portfolio General Manager, and with the VP Office, Engineering, to ensure that findings and proposed recommendations are relevant within the changing organizational context.
• Potential response bias of client interviewees: Client interviewees (as well as inactive and non-clients) were limited to those who could be contacted, using the most up-to-date contact information obtained from the Portfolio, and who agreed to be interviewed. This could bias interviewees towards those who have a more positive view of the program, and therefore a greater vested interest in the results of the evaluation. In order to mitigate this potential limitation, clients were contacted repeatedly, at different moments throughout the day. Those who were hesitant to participate were provided with thorough information about the evaluation, its importance (and the value of their participation in ensuring valid results), and about the confidentiality of responses. In the end, clients interviewed as part of the evaluation represented organizations that were the source of 87% of the CSTT's revenue over the past two years.
• Case studies were not identified randomly: As the Portfolio proposed the list of projects from which the case study projects were selected, these projects are not representative of all CSTT projects. As such, they are not intended to be representative of all client activity. To ensure the Portfolio did not select their strongest cases, the evaluation team made the final selection and also asked for additional projects to be proposed to include greater distribution across the criteria discussed in the previous section.

Appendix C: Selection of Documents Reviewed

Conference Board of Canada. (2009). The Productivity Performance of Canada's Transportation Sector: Market Forces and Governance Matter. Ottawa, Ontario: Mario Iacobacci and Joseph Schulman.

Department of Finance Canada. (2012). Jobs, Growth and Long-term Prosperity: Economic Action Plan 2012. (Cat. No.: F1-23/3-2012E; ISBN: 978-0-660-20189-4). Ottawa, Ontario.

Environment Canada. (2013). Backgrounder: Canada Continues to Align Greenhouse Gas Emissions Measures with the United States. http://www.ec.gc.ca/default.asp?lang=En&n=56D4043B-1&news=0EAD8AB1-52CF-44B3-951E-E832B933B684

Environment Canada. (2012). Canada's Emissions Trends 2012 (Cat. No. En84-83/2012E-PDF; ISBN 978-1-100-21063-6). Ottawa, Ontario. http://www.ec.gc.ca/Publications/253AE6E6-5E73-4AFC-81B7-9CF440D5D2C5%5C793-Canada's-Emissions-Trends-2012_e_01.pdf

Environment Canada. (2012). The 2012 Progress Report of the Federal Sustainable Development Strategy. (Cat. No. En1-46/2012E-PDF; ISSN: 1925-8402). Ottawa, Ontario. http://www.ec.gc.ca/dd-sd/23E4714E-B774-4CC5-9337-F87B01556727/FSDS-Progress-Report-2012-E-Feb15_WEBv4.pdf

National Research Council Canada. (2008). Centre for Surface Transportation Technology (CSTT) Business Plan: FY09-10 to FY11-12. Ottawa, Ontario.

National Research Council Canada. (2013). Fleet Forward 2020 NRC Program Business Case. Ottawa, Ontario.

National Research Council Canada. (2012). Here's what Paul Treboutat, General Manager, Surface Transportation, had to say. NRC Reach article dated June 13, 2012.

National Research Council Canada. (2012). Internal Audit of the Financial Management Control Framework - Revenue. Ottawa, Ontario.

National Research Council Canada. (1999). Memorandum to Senior Executive Committee: Technology Centres financial framework.

National Research Council Canada. (1996). Operational Framework: Centre for Surface Transportation Technology.

National Research Council Canada. (2012). Operational Plan for FY 2012-13: "Positioning NRC for Success as a Research and Technology Organization". Ottawa, Ontario.

National Research Council Canada. (2013). Rail NRC Program Business Case. Ottawa, Ontario.

National Research Council Canada. (2006). Some differences between NRC institutes and technology centres.

National Research Council Canada. (2012). Surface Transportation group recognized for out-of-this-world research support. NRC Reach article dated April 5, 2012.

National Research Council Canada. (2012). Surface Transportation Operating Plan 2013-2015. Ottawa, Ontario.

National Research Council Canada. (2012). Surface Transportation Portfolio Strategic Plan 2012-2017. Ottawa, Ontario.

NRC Investment Plan and Management of Projects Office. (2010). Organizational Project Management Capacity Assessment (OPMCA): Report for CSTT.

Railway Association of Canada. (2011). Rail 2030: Preparing Today for Tomorrow's Challenges. Calgary, Alberta: Robert McKinstry.

Transport Canada. (2012). Transportation in Canada 2011: Comprehensive Review (Cat. No. T1-23A/2011E-PDF; ISSN 1482-1311). Ottawa, Ontario.

Treasury Board of Canada Secretariat. (2012). 2011 Public Service Employee Survey Organizational Results: Centre for Surface Transportation Technology (CSTT). http://www.tbs-sct.gc.ca/pses-saff/2011/results-resultats/bd-pm/50/160/org-eng.aspx.

University of Ottawa. (2000). Five Year Review: NRC's Technology Centres. Ottawa, Ontario: Estelle Vincent-Fleurs, under the supervision of Professor Jérôme Doutriaux.

Appendix D: Organizations Consulted

Canadian Public Sector Clients

• BC Transit*
• Canadian Space Agency*
• Department of National Defence (DND)*
• Ontario Ministry of Transportation (MTO)*
• Ottawa Police Services*
• Royal Canadian Mounted Police (RCMP)
• Toronto Transit Commission (TTC)*
• Transport Canada*

Canadian Private Sector Clients

• Canadian National Railroad
• Canadian Pacific Railway*
• DEW Engineering*
• General Dynamics Land Systems
• New Flyer Industries*
• Railway Association of Canada
• Other private consultants*

Canadian Potential and Inactive Clients

• Bombardier Transportation
• Calgary Transit
• National Steel Car Ltd.
• VIA Rail Canada

International Clients

• Federal Railroad Administration (US)
• Trinity Industries*
• TTX Company

*Denotes clients who were included in a case study