Evaluation of the Strategic Investments in Northern Economic Development Program

Final Report
(Conducted by AANDC on behalf of CanNor)
Date: December 2013
Project Number: 1570-7/12037

PDF Version (386 Kb, 67 Pages)

Table of contents

List of Acronyms

AANDC
Aboriginal Affairs and Northern Development Canada
CanNor
Canadian Northern Economic Development Agency
EPMRB
Evaluation, Performance Measurement and Review Branch
G&C
Grants and Contributions
O&M
Operations and Maintenance
SINED
Strategic Investments in Northern Economic Development

Executive Summary

Introduction

This report presents the findings of the evaluation of the Strategic Investments in Northern Economic Development (SINED), which is administered by the Canadian Northern Development Agency (CanNor). This summative evaluation was conducted on behalf of CanNor by the Evaluation, Performance Measurement and Review Branch (EPMRB) of the Department of Aboriginal Affairs and Northern Development (AANDC). The study covered the ongoing implementation of SINED from 2007-08 to 2011-12. The methodologies used to gather the evidence included: literature review, document and file reviews, key informant interviews and case studies.

SINED is a suite of programs that have provided funding for economic development in the North since May 30, 2005. A formative evaluation covering over two years (2005-06 to 2007-08) of SINED activity under AANDC was completed in September 2008. This follow up evaluation is in compliance with the 2009 Treasury Board Policy on Evaluation. The objectives are to:

  • provide a neutral and independent analysis of SINED’s ongoing relevance (continued need, alignment with government priorities, and alignment with government roles and responsibilities) and performance (effectiveness, efficiency and economy);
  • inform the decision-making for program renewal in 2013-14; and
  • provide recommendations for SINED's improvement.

Evaluation Findings

Relevance - Continued need

Primary arguments for the continuation of SINED include the need to address the enormous challenges to economic development in the North, a shortage of funding, the need for further capacity development and diversification, and the importance of a northern approach in the North. Canada's territories have significant natural resource deposits in oil and gas, metals, diamonds and minerals, however, there is a need for economic diversification to protect the economy from the boom-bust cycles of the resource sector. Natural and cultural attractions have contributed to growth in tourism and cultural industries. Renewable energy and fisheries have also advanced.

However, the development of all sectors has been challenged by a lack of transportation and communication infrastructure, skills and housing shortages and high energy costs. The public sector remains the largest employer, private sector investment is difficult to attract and funding from the territorial government is limited. Within this context, the North needs programs like SINED, which integrates northern perspectives and priorities through the Territorial Investment Plans to address the barriers to economic development and contribute to a foundation for economic development to take place.

Relevance - Alignment with federal priorities

There is a very strong alignment between SINED objectives and federal priorities. From 2005 when SINED was created up to now, the North has featured predominantly in federal announcements and investments. Also, Canada's Northern Strategy was introduced in 2007 and since that time, the Government of Canada has committed to the streamlining of major project approvals, housing construction, Aboriginal skills training, adult basic education, mapping of the Arctic seabed, construction of a world-class Arctic research station, completion of the Dempster highway and more. The 2008 and 2011 Speeches from the Throne strengthened the government's commitment to the North and the 2009 SINED renewal complements other federal initiatives (e.g. 2009 Economic Action Plan) in the North.

Relevance - Alignment with federal roles and responsibilities

SINED's role in economic development is embedded in the federal government's jurisdiction in the North, which is clearly mandated in legislation namely, the Canadian Constitution and the Department of Indian Affairs and Northern Development Act. Other entities (e.g. territorial governments) are involved in the economic development of the territories; however, the evaluation found that their roles and activities are complimentary to that of SINED. Along the same lines, while some "province-like" responsibilities for governance and administration have been successfully transferred to the territorial governments, control over land and resource management is in varying stages of devolution from AANDC to the territories with Nunavut only now beginning to negotiate a devolution plan. In the end, economic development in the North remains a shared responsibility between the federal government, territorial governments and self-governing First Nations. Given the continued reliance of the territorial governments on federal transfers, federal involvement in northern economic development will likely continue for some time.

Performance – Effectiveness/success

The evaluation was unable to make any conclusions on the overall impact of SINED investments due to the lack of processes or systems to monitor and aggregate results over time and difficulties accessing project reports through the First Nations and Inuit Transfer Payment System. A Performance Measurement Strategy created for SINED in 2009 was never implemented. In the absence of a formal coordinated approach to monitoring progress, performance information has been collected at the project level and documented in project reports. Performance reporting thus involves the manual review and roll-up of individual reports.

Some positive outcomes and useful indicators for investments in energy, renewable energy, mining, geoscience, fisheries, tourism, information communications technology/broadband and cultural industries were documented through the case studies.

Performance – Efficiency and economy

SINED staff was identified by recipients as a strength due to their dedication, knowledge, and support. However, SINED has experienced difficulties attracting and retaining staff. This has resulted in increased workloads, impacted planning, performance measurement, monitoring and reporting. In turn, these difficulties have led to a reduction in the support that staff provides to project proponents.

Changes to the project approvals process have contributed to delays in project and funding approvals. These delays have disrupted recipient organizations' planning processes, increased project costs and impacted the timely delivery of quality projects. In addition, delays in funding contribute to financial hardship for recipient organizations.

The transition of SINED from Aboriginal Affairs and Northern Development Canada to CanNor has also impacted the effective and efficient delivery of SINED programs. SINED was transferred to CanNor just as the new agency was getting established. Shortly thereafter, SINED staff was tasked with delivering additional programming introduced through the federal government's Economic Action Plans. These actions resulted in increased workloads and impacted SINED delivery. The evaluation also found that the transition has resulted in a "disconnec" between management and operations.

Another key finding is the many similarities between the four SINED component programs. Not only do these components have similar eligible recipients but they also fund similar projects thus calling into question the need for four separate component programs.

Other Issues

An analysis of Canadian and international approaches to economic development in northern and remote regions concluded that SINED is consistent with international best practices for short-term policies to improve economic development. However, without significant, coordinated, long-term investments in the North, SINED is unlikely to result in sustained change in economic conditions. To this end, the literature emphasized the importance of sustained long-term investment in leadership, institutions, education, skills development, infrastructure, entrepreneurship and healthy, vibrant communities.

Conclusions

SINED is consistent with federal government priorities and roles and responsibilities and responds to a continued need for foundational investments supporting economic development in the North. SINED funding complements what is available through territorial governments and other federal departments and should continue given the limed resources and difficulties attracting private-sector investment.

While some quantifiable measures of success were documented for particular sectors, without a process to properly monitor the outcomes of SINED investments it is not possible to assess the impacts of these investments. SINED needs a performance measurement strategy, which identifies outputs, outcomes and indicators and which outlines a process for collecting performance information. Upon implementation, such performance information can be used for ongoing management, reporting on results and the early identification of problems.

The evaluation identified inefficiencies in the project and funding approvals, program design and tracking tools. In addition to negatively impacting internal operations, inefficiencies in these areas have also negatively impacted recipient organizations and the cost, quality and timeliness of projects. SINED should apply lessons learned and best practices from regions and from Regional Development Agencies to address issues impacting efficiency and economy.

The disruption caused by the transition of SINED to CanNor, the additional workload and the disconnection between SINED management and operations must be recognized and addressed immediately. A staffing strategy is needed to identify responsibilities and appropriate resource levels for the effective delivery of SINED and to attract and retain staff. The strategy should target individuals with the right skills (e.g., new business graduates), address barriers to moving north (e.g., housing) and provide incentives for moving north (e.g., training).

Finally, there is a great need for foundational investments in the North to support economic development. SINED provides essential support for capacity building in existing sectors and economic diversification, and has adapted well to northern economic development strategies. However, the need in the North is such that it is greater than one economic development program can possibly fulfill. In order to achieve real change, SINED must be part of a long-term, sustained, coordinated approach to northern development.

Recommendations

It is recommended that CanNor:

  1. Introduce processes, systems and resources to properly monitor and manage performance (effectiveness, efficiency and economy) and report on the impact of SINED investments.
  2. Review the approach (program design) to achieving outcomes based on lessons learned and best practices. Consider collapsing the four programs into one or better distinguishing between them.
  3. Introduce processes/procedures to improve approval time and delivery of funding.
  4. Update policies and procedures manual.
  5. Develop a Human Resources Plan / Recruitment Strategy to attract and retain staff and ensure CanNor has needed resources (with relevant experience) to meet program delivery demands.

Management Response / Action Plan

Provided by the Canadian Northern Economic development Agency

Recommendations Response and Planned Actions Responsible Manager (Title / Sector) Planned
Implementation and
Completion Dates
1. Introduce processes, systems and resources to properly monitor and manage performance (effectiveness, efficiency and economy) and report on the impact of SINED investments. Agreed.

CanNor has systematically updated its performance measures since its creation in 2009. Its departmental Performance Measurement Framework has been updated with changes complementary to its MRRS elements and is now fully operational, including elements that report on program (including SINED) performance. With new measures in place for Northern Adult Basic Education Program, updates made to the Aboriginal Economic Development program suite, and SINED's update completed in line with its renewal, all of CanNor's performance measures are current.
Vice-President Policy By December 31, 2013
To complement work on measures, CanNor's Operations group is implementing mechanisms to gather the necessary data via internal spreadsheets. DG Operations By March 31, 2014
Further, work is underway to update CanNor's program forms and processes, which will include clear and simple procedures to gather performance data. DG Operations By March 31, 2014
The Agency is consulting with other Regional Development Agencies and Shared Service Canada to seek long-term solutions. Director, Corporate Services and DG Operations Ongoing
2. Review the approach (program design) to achieving outcomes based on lessons learned and best practices. Consider collapsing the four programs into one or better distinguishing between them. Agreed

CanNor is reviewing program design, taking into account input from the SINED evaluation and from stakeholders.
Vice-President Policy By June 30, 2014
3. Introduce processes/procedures to improve approval time and delivery of funding. Agreed

CanNor is reviewing business processes and will introduce changes to enhance timeliness of support.
DG Operations By March 31, 2014
4. Update policies and procedures manual. Agreed.

CanNor has begun developing a program officer tool kit that will include an updated business process manual, program manuals and guides, and project risk assessment templates.

CanNor will provide copies of the manual and training on the policies and processes to all staff involved in the Grants and Contributions business process. The training will be an ongoing process to ensure new staff are trained and existing staff are made aware of updates.
DG Operations By March 31, 2014
5. Develop a Human Resources Plan / Recruitment Strategy to attract and retain staff and ensure CanNor has needed resources (with relevant experience) to meet program delivery demands. Agreed.

The Agency is in the process of renewing its HR plan and finalizing the Inuit Employment Plan (IEP) with target completion date by the first quarter of 2014. CanNor commits to review these plans on an annual basis.
Director Corporate Services By June 30, 2014

The Management Response / Action Plan for the Evaluation of the Strategic Investments in Northern Economic Development (SINED) Program were approved by the Policy Committee on December 19, 2013.

1. Introduction

1.1 Overview

Strategic Investments in Northern Economic Development (SINED) is a suite of programs that has provided funding for economic development in the North since May 30, 2005. A formative evaluation was completed in September 2008 covering over two years of activity between 2005-06 and 2007-08. SINED was renewed for an additional five years commencing April 2009 and is currently funded until March 31, 2014. The objective of this summative SINED evaluation is to provide CanNor senior management with credible and neutral information on the ongoing relevance and performance of SINED in order to inform the decision for program renewal in 2013-14. A further objective is to provide recommendations for program improvement.

In 2008, when the formative evaluation was conducted by the Evaluation, Performance Measurement and Review Branch (EPMRB) of the Department of Aboriginal Affairs and Northern Development Canada (AANDC), SINED was delivered through AANDC's Northern Affairs Organization. Northern Affairs Organization orchestrated the program renewal and initial delivery in 2009 pending the establishment of the Canadian Northern Economic Development Agency (CanNor), a new agency responsible for economic development in the North. CanNor was established, and became responsible for SINED, in August 2009 in accordance with paragraph 2(a) of the Public Service Rearrangement and Transfer of Duties Act. EPMRB is undertaking this summative evaluation of SINED on behalf of CanNor.

1.2 Program Profile

1.2.1. Program Context

Canada's North faces unprecedented opportunities: it has impressive economic assets, principally in the form of identified but untapped natural resources. However, on the opposite side of the equation lies a daunting list of challenges to economic development. Beyond the capital cities, the North is composed of remote communities separated by huge distances. The transportation and communications infrastructure is underdeveloped and energy costs are high. Arctic conditions, including permafrost, snow and ice cover, extreme weather, prolonged darkness in winter and light in summer and limited local and regional capacity are a part of life. A large public sector remains the primary employer in the territories. Attracting private sector investment to the North is a challenge and industries struggle to overcome many barriers to business. Opportunities and risks, or barriers specific to key sectors in the North are summarized below:

In 2010-11, tourism contributed over $110 million to the economy in the Northwest Territories. Tourists can take advantage of various activities like sport fishing, hunting, outdoor adventures, aurora viewings, and Aboriginal cultural events across the territories. Long-term trends indicate that the sector has opportunity to grow in adventure and eco-tourism and is expected to benefit from a growing desire among travellers for authentic experiences. Threats to the sector include limited airline capacity, particularly in Nunavut and the Northwest Territories, competition from other tourist destinations, limited connectivity, a lack of capital investments and a slow national and international economic recovery.

The North has a distinct and proud history of Aboriginal culture and traditions and one of the highest forms of Aboriginal cultural expression is through arts and crafts. Cultural industries (such as film, carving, printmaking, weaving, jewellery, clothing) across the North are recognized and respected internationally and have gained an increase in international exposure in recent years, particularly as a result of cultural displays and events at the 2010 Olympics in Vancouver. In Nunavut, the cultural industries employ 20 percent of the local workforceFootnote 1 and contribute a total of $33.4 millionannually to the Nunavut economy, including $22.9 million in direct impact, and $10.5 millionin related spinoffsFootnote 2. The cultural sector of the Northwest Territories, including artists, consists of 750 workers, almost three percent of the territorial workforce (2011-12)Footnote 3. New markets in Europe have been steadily growing and there is potential for a Chinese market, but the majority of the market remains in Canada.

Challenges to cultural industries include economic conditions/downturns, barriers to accessing new markets, difficulties accessing materials such as seal skins or carving stone (some quarries can only be accessed once a year), linguistic profiles of artists, a lack of a distinct brand image for the various products and competition from high-paying jobs in other sectors. Obstacles identified by the film industry include lack of tax credits, of the kind offered by many jurisdictions, and limited availability of technicians and equipment.

Fisheries has been identified as an important sector for both the Northwest Territories and Nunavut; however, Nunavut has seen the most promising growth in recent years. In 2010, it was estimated that Nunavut's fisheries contributed between $12 and $14 million annually to the territorial economy ($7.5 to $9.5 million as income)Footnote 4. In 2010, the Standing Senate Committee on Fisheries and Oceans identified an increased harvest in Arctic char as a goal for future development of the industryFootnote 5. Arctic char are a highly priced delicacy, marketed mainly fresh and frozen as whole-dressed fish and steaksFootnote 6. The most important fish species in the Northwest Territories are in Great Slave Lake and include whitefish, pickerel, lake trout, northern pike, and inconnuFootnote 7.

The greatest threat to the fisheries industry is the availability of supply of fish, so the focus of much of the industry is maintaining a stable industry rather than increasing productionFootnote 8. In Nunavut, the fisheries industry has also been hampered by the lack of a deep-water port to offload catches. Although Nunavut has Canada's largest coastline, it is the only coastal jurisdiction that does not have a deep-water port, which can accommodate large fishing vessels and the processing of all their catches. As a result, the offloading of catches from large fishing vessels is time consuming and catches that cannot be processed in Pangnirtung are being shipped internationally to Greenland for processing and to other markets for packaging. In turn, these countries label this Canadian product as their own. Investments in infrastructure like port facilities and harbours to support fishing vessels (even of modest size), as well as processing plants and cold-storage operations, are needed for the sector to thrive in Nunavut. In the Northwest Territories, there has been a drop in fish harvests due to challenges with the fish population, however, a recovery strategy is in place to address these challengesFootnote 9.

The United States Geological Survey estimates that the North contains up to 13 percent of the world's undiscovered oil and 30 percent of undiscovered natural gas reservesFootnote 10. However, the oil and gas industry has been impacted by declining international prices and delays to the Mackenzie Gas Pipeline Project. Since peaking in 2001, the Northwest Territories' fuel industry has been in decline with no new wells expected to begin production over the next few yearsFootnote 11. Additional challenges to the development of the energy sector include the fragility of northern ecosystems, a shortage of skilled labour, a lack of communication and transportation infrastructure and a lack of private sector incentives and investment.

Likewise, there is great renewable energy potential in the North. In the Northwest Territories, there are multiple larger-scale hydro opportunities, creating a foundation for a power grid, which would reduce rates and enhance reliability and diversity to the economy. In the Yukon, hydro and wind comprise 17 percent of energy useFootnote 12 and the supply of fuel wood has been valued at approximately $4 million a year in employment and import substitutionFootnote 13. While some studies have been conducted to better understand the Yukon's wind, geothermal and solar energy potential, it is largely unknown and undevelopedFootnote 14. Yet, threats such as limited local capacity and funds for renewable energy development in the North and poor economies of scale pose particular challenges to the sector's growth.

Moreover, the potential of mining in the North also looks bright. Many northern regions have a strong history in mining and are highly developed, while others offer underexplored regions with significant potential for resourcesFootnote 15. It is estimated that northern diamond mining accounts for 8.87 percent of total global diamond production, making Canada the third-largest global diamond producer by valueFootnote 16. The global economy is recovering and the long-term global demand for commodities is increasing, making mines in remote northern areas potentially viableFootnote 17. Several new mines are slated to open before the end of the decade in Nunavut, Yukon, and the Northwest Territories and at least five mines in the Yukon are expected to be producing during the mid and later part of the decadeFootnote 18. In addition, the overall northern metal and non-metallic mineral output is expected to grow by 91 percent from 2011 to 2020, with a compound annual growth rate of 7.5 percentFootnote 19.

Devolution will likely contribute to increased mining activity. Since 2005, the Yukon has been described as one of the world's best mining jurisdictions because of its very sta­ble political climate, good transportation and energy infra­structure, and demonstrably pro-mining government. As a result, the Fraser Insti­tute's annual mining survey regularly places post-devolution Yukon near the top of its Policy Potential Index for best mining jurisdic­tions within Canada and around the worldFootnote 20. Until devolution is finalized in the Northwest Territories and Nunavut, regulatory uncertainty will continue to be a threat to mining in those territories. Additional threats include limited capacity and resources of First Nations and Inuit to engage in consultation and negotiation, recruitment of labour to work on remote construction and mining sites, poor road infrastructure, lack of an energy strategy for large-scale mining, serious shortages of housingFootnote 21, a fragile natural environment, distance to market and climate change.

Within this context, an economic development program, such as SINED, can address many of the barriers identified above and help establish the conditions for future economic growth and stability.

1.2.2 History of SINED

SINED emerged as the federal government's primary response to economic development in the North in March 2004 when the federal budget committed $90 million over a period of five-years for northern economic development. In the spring of 2005, SINED was launched by AANDC. It included three complementary programs: a new Targeted Investment Program was created with incremental funds. The existing Innovation and Knowledge fund and Northwest Territories-specific Partnership and Advisory Forums fund, both A-based programs, were renewed and integrated into the set.

In 2009, SINED was renewed as part of Canada's Economic Action Plan, whichallocated a new $90 million in funding for another five years (April 1, 2009 to March 30, 2014) over and above the ongoing funds, which totaled almost $6 million over the same period. Following review and stakeholder input, the program was renewed with essentially the same design, but with enhanced flexibility to realign funds between program elements, and (consistent with the announcement that it would form the core activity of a new regional economic development agency for the North) increased emphasis on leadership and helping build partnerships. The Partnership Advisory Forums was extended to all territories to help promote multi-stakeholder discussions, and a new Pan-Territorial Fund was established to support projects impacting more than one territory.

1.2.3 SINED Components

Targeted Investment Program

In 2009, Targeted Investment Program was allocated $67 million over five years to be divided equally among the territories. An analysis of the distribution of Targeted Investment Program funding between 2007-08 and 2011-12 revealed a fairly equal allocation of Targeted Investment Program funding across the three territories. Northwest Territories accounted for 36 percent of expenditures and Nunavut and the Yukon each spent 32 percent of Targeted Investment Program funding.

A single project cannot exceed $2 million in annual funding and the amount payable to any one recipient cannot exceed $6 million. Targeted Investment Program projects must align with Territorial Investment Plans (which were developed in advance of the 2009 renewal based on progress in the first round of SINED); economic realities in each territory; and, input from Territorial and Aboriginal governments, federal departments, the private sector and other stakeholders. Investment Plans are reviewed and updated as necessary.

The Investment Plan in each territory may identify a mix of specific activities, recipients and objectives; more broadly defined activities with specific objectives and outputs; and broad developmental areas with activities and outputs to be identified through general calls for proposals. See Section 1.2.8 for more information on Territorial Investment Plans.

Targeted Investment Program projects must also align with the following four thematic areas:

  1. Building the knowledge base in key economic sectors to help set the stage for further economic opportunities. Eligible projects include research projects, strategic plans, feasibility studies and other related projectsFootnote 22.
  2. Enhancing the economic infrastructure base through the development of multi-user physical infrastructure critical to economic growth. Eligible projects include feasibility or planning studies for public or multi-user infrastructure; public/multi-user telecommunications investments that fill gaps not covered by other federal funding; and, small-scale investments in new or existing infrastructure with direct linkages to enhanced economic activityFootnote 23.
  3. Capacity development of organizations, associations, small and medium enterprises and individuals to help them take advantage of economic opportunities. Eligible projects include research projects, development of materials, curriculum, standards which support capacity development, and capacity development initiativesFootnote 24.
  4. Economic diversification within regions and sectors with a focus on the development of new sectors, products, and markets. Eligible activities include, but are not limited to, the development of sectoral strategies that can be then used to strategically guide investments; support for the creation of technology clusters; support to cultural industries; and, new product development and marketingFootnote 25.

As shown in Graph 1, the majority (43 percent) of Targeted Investment Program funding between 2007-08 and 2011-12 was under the "building the knowledge base" theme. 30 percent of projects were classified as "capacity development" and 16 percent were directed towards "economic diversification". Projects for "enhancing the economic infrastructure base" represented 11 percent of total SINED projects during that time.

Graph 1: Targeted Investment Program Expenditures by Theme 2007-08 to 2011-12
Targeted Investment Program Expenditures by Theme 2007-08 to 2011-12
Source: FNITP – List of SINED projects funded between 2007-08 and 2011-12
Text description of Graph 1

Graph 1 depicts the Targeted Investment Program Expenditures from 2007-08 to 2011-12, broken down by theme. The majority (43 percent) of Targeted Investment Program funding was under the "building the knowledge base" theme. 30 percent of projects were classified as "capacity development" and 16 percent were directed towards "economic diversification". Projects for "enhancing the economic infrastructure base" represented 11 percent of total SINED projects during that time.

Ineligible projects/expenditures include: core funding; ongoing operations and maintenance of organizations; major capital investments in infrastructure (e.g., laboratories); research of knowledge-based activities that are primarily a resource management function; construction activities related to large engineering works (e.g., roads, highways, bridges); investments in new or existing infrastructure which is either owned or operated by a single for-profit enterprise that is not for public use; direct skills training of individuals of the sort normally offered through learning institutions or apprenticeship programs; and wage subsidies for persons receiving training through course work or on the jobFootnote 26.

Eligible recipientsinclude organizations, associations or social enterprises whose mandate includes work with respect to northern economic development; individuals and small and medium-sized enterprises; other levels of government; and other non-federal entities, public or private, that have an interest/involvement in economic development in the NorthFootnote 27. An analysis of recipients for Targeted Investment Program expenditures between 2007-08 and 2011-12 reveals that the territorial government is the primary recipient in all three territories - mainly for geoscience. Between 2007-08 and 2011-12, the Government of Nunavut received $15 millionFootnote 28, the Government of Northwest Territories received $10.7 millionFootnote 29 and the Government of Yukon received $8 millionFootnote 30.

Graph 2: Recipients of Targeted Investment Program funding by territory between 2007-08 and 2011-12
Targeted Investment Program Expenditures by Theme 2007-08 to 2011-12
Source: FNITP – List of SINED projects funded between 2007-08 and 2011-12
Text description of Graph 2

Graph 2 depicts the recipients of Targeted Investment Program funding between 2007-08 and 2011-12, broken down by the following regions: Nunavut, Northwest Territories, and Yukon. The graph shows that the Northwest Territories and Yukon regions support industry associations more frequently than Nunavut. The Yukon supports Aboriginal organizations significantly more than the Northwest Territories and Nunavut. Also, the Nunavut region has the most dealings with the territorial government and non-government organizations.

Graph 2 above shows that the Northwest Territories and Yukon regions support industry associations more frequently than Nunavut. The Yukon supports Aboriginal organizations significantly more than the Northwest Territories and Nunavut. Nunavut region has the most dealings with the territorial government and non-government organizations.

Innovation and Knowledge Fund

In 2009, Innovation and Knowledge was allocated $7.5 million over five years (divided equally among the three territories, it equals $500,000 per territory per year) to help territorial residents participate in and benefit from the following:

  1. Innovation - new or adaptations to products, services, or processes
  2. Knowledge - production/dissemination, access to, increased understanding and ability to use
  3. Knowledge-based economy – defined by the Organization for Economic Cooperation and Development as a greater dependence on knowledge, information and high skills levels by business and the public sector

The intention is to help Northerners seize opportunities in the new economy, and further science and technology. A single project cannot exceed $95,000 in total.

Innovation and Knowledge can invest in a full array of small or emerging economic development opportunities; its projects do not need to align with Territorial Investment Plans. CanNor does have discretion to set local Innovation and Knowledge priorities and may choose to link with Investment Plan targets. Like Targeted Investment Program, research and consultation are eligible for funding under Innovation and Knowledge, however, Innovation and Knowledge also supports innovation, building capacity of individuals, and building capacity of organizationsFootnote 31.

Similar to Targeted Investment Program, ineligible expenditures include core funding, major capital items, fees for conventional skills training and wage subsidies. However, loan fund top-upsFootnote 32 are not eligible for funding under Innovation and Knowledge. Eligible recipients are the same for Targeted Investment Program and Innovation and Knowledge.

Partnership Advisory Forums

The Partnership Advisory Forums provides $1.5 million over five years (or $100,000 per territory per year) to increase dialogue on northern economic development issues and help increase capacity in delivering projects in the North. The funding is also used to increase knowledge, skills and understanding related to the roles and responsibilities of the various players involved in economic development in the territories and supporting stakeholders at the local, regional, territorial or pan-territorial levels. Contribution to any single project cannot exceed $95,000.

The Partnership Advisory Forums projects are not included in individual Territorial Investment Plans. The Partnership Advisory Forums priorities for investments are informed by the advice of key economic players and other consultation and co-ordination activities. Eligible activities/expenditures are identical to Innovation and Knowledge, except Partnership Advisory Forums will provide core funding and will not fund individual capacity. Ineligible expenditures are the same as Innovation and Knowledge except as mentioned aboveFootnote 33.

Pan-Territorial Fund

The Pan-Territorial Fund, under the renewed 2009 SINED, is for projects that involve multiple territories or a territory and one or more provinces. $5 million over four years was provided only for project elements related to benefits within the territories. Other funding partners such as federal Regional Development Agencies were encouraged to fund project elements in proportion to benefits to their areas. Due to its pan-territorial scope, the fund is not included in individual Territorial Investment Plans.

Eligible projects must meet the eligibility criteria under Targeted Investment Program, Innovation and Knowledge and Partnership Advisory Forums; must involve at least two territories; and demonstrate pan-territorial benefits. Ineligible projects include construction activity (as opposed to planning activity) related to large engineering works such as roads, highways, airports, railroads, ports, wharfs, dams, commercial buildings and similar infrastructureFootnote 34.

1.2.4 SINED Investments

As shown in Graph 3 below, between 2007-08 and 2011-12, 87 percent of SINED investments were made through Targeted Investment Program, eight percent through Innovation and Knowledge, four percent through Pan-Territorial Fund and one percent through Partnership Advisory Forums.

Graph 3: Distribution of SINED funding by program between 2007-08 and 2011-12
Distribution of SINED funding by program  between 2007-08 and 2011-12
Source: FNITP – List of SINED projects funded between 2007-08 and 2011-12
Text description of Graph 3

Graph 3 depicts the distribution of SINED funding between 2007-08 and 2011-12, broken down by program. The vast majority of SINED investments, at 87 percent, were made through Targeted Investment Program. Eight percent of investments were distributed through Innovation and Knowledge, four percent through Pan-Territorial Fund, and one percent through Partnership Advisory Forums.

For the second round of SINED, 84 percent of the $90 million was expected to be distributed through Targeted Investment Program, eight percent through Innovation and Knowledge, six percent through Pan-Territorial Fund and two percent through Partnership Advisory Forums. Graph 3 shows that the actual investments are within range of planned expenditures.

1.2.5 Objectives and Expected Outcomes

In June 2009, while SINED was still with AANDC, a Performance Measurement Strategy was created to measure, manage and report on the performance of SINED investments. The following outcomes were taken from the Performance Measurement Strategy:

Immediate Outcomes
  • Expanded publicly accessible information: Access to and use of knowledge products developed by SINED-funded activities. For example, the production of publicly accessible geoscience information creates opportunities for investments in resource exploration and extraction.
  • Expanded multi-user infrastructure: Access to and use of northern economic infrastructure created by SINED-funded activities. For example, businesses accessing telecommunications infrastructure such as broadband connections in order to market their products.
  • Increased access to capital: Access to business funding as a result of SINED investments. One example is SINED funding provided to Community Futures Development Corporations, which in turn provide capital to the private sector.
  • Increased ability of Northerners to respond to economic development opportunities: Access to and use of capacity-building products and services developed by SINED-funded activities. For example, Northerners accessing training services developed by project proponents.
  • Increased partnerships with northern governments and organizations: Cooperation and collaboration amongst organizations that play a role in northern economic development.
Intermediate Outcomes
  • Increased private sector investment and expanded business and employment opportunities: Growth in investments related to SINED-funded activities, and an increased private sector role in the territorial economies.
  • Increased sustainability of principal economic drivers: Growth in resource-based industries, especially mining and oil and gas, and tourism. This includes increased capacity to handle downturns in global commodities markets.
  • Increased economic development and diversification: Growth in small and medium-sized enterprises, and in emerging northern industries such as culture and fisheries.
Long-term Outcomes
  • Strong, stable territorial economies for the benefit of Northerners and all Canadians: Economically diverse, self-reliant territorial economies with high levels of participation by Northerners.

1.2.6 Program Management, Key Stakeholders and Beneficiaries

Transition of SINED from AANDC to CanNor

In the January 2009 federal Budget, the Government of Canada tabled an Economic Action Plan, which committed $50 million over five years towards a new Regional Development Agency. On August 18, 2009, the Prime Minister announced the creation of CanNor as the Regional Development Agency for Canada's North, with headquarters in Iqaluit and regional offices in Nunavut, Northwest Territories and the Yukon, and a policy/coordination office in Ottawa.

The creation of CanNor was accompanied by the announcement that SINED, and all other AANDC territorial economic development programming, would be transferred from AANDC to CanNor. Some rental space was secured in 2010 to provide room for headquarter positions, notably the Vice President of Operations and other staff. As well, the AANDC economic development unit in Iqaluit was transitioned to CanNor offices in the middle of fiscal year 2009-10. The other three regional offices had similar moves. A new building was constructed in Iqaluit and, in June 2012, almost three years after the announcement of the transfer of responsibilities, CanNor headquarters staff moved into the new space.

While the new agency benefitted from the transfer of AANDC staff experienced with SINED, CanNor had to fill new positions on the corporate side. In addition, a number of AANDC staff did not want to relocate from AANDC Headquarters in Ottawa to Iqaluit. As a result, new staff had to be hired. According to SINED staff, from the beginning, CanNor has been challenged with filling vacant positions. CanNor management has experienced frequent turnovers. For example, since 2009, there have been four deputy ministers and six chief financial officers.

After the transition to CanNor, regions continued to deliver SINED as outlined in the program manuals, however, in November 2010, limited or reduced sub-delegation of Ministerial authority was introduced. Government policy entrusts ministers and deputy heads with the responsibility to delegate financial and operational authority to managers thereby enabling them to more efficiently administer programs that fall within their jurisdiction. The delegation of signing authorities provides managers with some power to authorize expenditures and introduces efficiencies.

In the first round of SINED, the regional directors could approve projects up to $175,000 and regional directors general could approve projects less than $250,000. Projects between $250,000 and $499,000 were approved by the Assistant Deputy Minister and projects over $500,000 were approved by the Minister. In the second round of SINED (following November 2010), all SINED projects went to the Minister for approval.

In fiscal year 2011-12, a total of 96 projects were funded. Sixty-seven projects were under $250,000, 20 projects were between $250,000 and $499,000 and nine projects were over $500,000. The project with the lowest dollar value was for $3,150 and the highest value project was $1.55 million. While the evaluation was unable to determine what percentage of these projects were actually approved by the Minister (as some could have been multi-year projects and approved several years prior), it does illustrate the volume of projects per year. It also shows that if sub-delegation of ministerial authority was still in place, only nine percent of projects in 2011-12 would have required ministerial approval.

A Memorandum of Understanding between AANDC and CanNor, in operation since August 18, 2009, compensated AANDC for continued support for corporate services in human resources, information technology, financial services, administration, contracting, transfer payment management, audit and evaluation, financial systems, communications, executive services and material management supportFootnote 35. Human resources, financial services and information technology have been since transferred to shared Government of Canada services. At the time of writing this report (August 2013), the parties have been operating without a Memorandum of Understanding for five months. The evaluation is aware that the Memorandum of Understanding is being reviewed for renewal, and in the meantime, AANDC will continue to provide corporate services until other arrangements are made.

In 2009, the same year CanNor was launched and SINED was renewed, CanNor was also tasked with delivering both the Community Adjustment Fund and the Recreational Infrastructure Fund, which were announced as part of Canada's Economic Action Plan. These two-year stimulus programs, designed to address short-term economic needs of Canadian communities impacted by the global recession, were delivered through the Regional Development Agencies and provided $32.8 million in total contributions funding for the territories. With three incremental staff for delivery, CanNor Operations staff also drew on experienced SINED staff to ensure timely implementation.

Shortly after the Community Adjustment Fund expired in 2011, CanNor (including SINED) was tasked with delivering the Northern Adult Basic Education Program, a five-year program totalling $27.0 million for the three territories with only one additional delivery officer. This was followed by the Community Infrastructure Improvement Fund, which was announced in the 2012 Economic Action Plan. The Community Infrastructure Improvement Fund provides the three territories with $6.4 million over two years for repairs and improvements to existing community infrastructure that is accessible to the public. The program expires on March 31, 2014.

The evaluation also notes that in 2012, CanNor was impacted by the federal Deficit Reduction Action Plan. In 2009-10, CanNor had 106 employeesFootnote 36 but this quickly fell to 75 in 2010-11 and has hovered at this level ever since. SINED had 22 employees in 2009 and the number of employees peaked at 31 in March 2011. By September 2011, the number of SINED employees was reduced to 25 and has, for the most part, stayed at this level. Most change has been at the CO-01 and CO-02 levels and PM 4, 5 and 6 levels.Footnote 37

Governance

CanNor headquarters in Iqaluit is responsible for the overall management of SINED delivery while the regional offices develop the Territorial Investment Plans and take care of day-to-day management of SINED, including issuing calls for proposals, completing project assessments, developing recommendations for approval, project tracking, monitoring and reporting.

CanNor's policy office in Ottawa is responsible for program design, policy development and drafting of Memoranda to Cabinet and Treasury Board Submissions, as well as annual planning and reporting to Parliament through the Report on Plans and Priorities and the Departmental Performance Report. The Vice President and President are also based in Ottawa. The Minister of CanNor, who is also the Minister of Environment and Minister for the Arctic Council, is responsible for project approvals.

Key Stakeholders and Beneficiaries

A number of diverse stakeholders are involved in SINED projects, including territorial governments, industry associations, Aboriginal organizations, other federal departments, municipalities, non-governmental organizations, post-secondary institutions and the private sector. Stakeholders are engaged in discussion to help ensure the Investment Plans fit the needs for economic growth in the specific territory.

The primary beneficiaries of SINED are organizations engaged in economic development in Canada's three territories and businesses in targeted sectors such as mining and tourism.

1.2.7 SINED Budget Allocations for 2007-08 to 2011-12

Table 1: SINED Planned/Forecast spending from 2007-08 to 2011-12
2007-08 2008-09 2009-10 2010-11 2011-12 TOTAL
Operations and Maintenance
Salary 913728 913728 913728 2741184
Other 182746 182746 182746 548238
TOTAL O&M 1,096,474 1,096,474 1096474 3,289,422
G&C
Targeted Investment Program 18,000,000 18,000,000 6,357,000 15,107,000 15,107,000 72,571,000
Innovation and Knowledge 1,078,000 1,078,000 1,500,000 1,500,000 1,500,000 6,656,000
Partnership Advisory Forums 79,000 79,000 300,000 300,000 300,000 1,058,000
Pan-Territorial Fund 0 0 0 1,250,000 1,250,000 2,500,000
TOTAL G&C 19,157,000 19,157,000 8,157,000 18,157,000 18,157,000 82,785,000
TOTAL SINED 9,253,474 19,253,474 19,253,474 86,074,422
Source: G&C for 2007-08 to 2008-09 from TB Submission. O&M and G&C for 2009-10 to 2011-12 provided by CanNor
Table 2: SINED Actual spending from 2007-08 to 2011-12
2007-08 2008-09 2009-10 2010-11 2011-12 TOTAL
Operations and Maintenance
Salary 0
Other 0
TOTAL O&M 0
G&C
Targeted Investment Program 18,367,202 21,080,202 6,239,807 14,832,306 13,692,034 74,211,551
Innovation and Knowledge 963,771 967,542 1,573,834 1,514,635 1,291,315 6,311,097
Partnership Advisory Forums 77,715 65,000 279,633 222,778 50,000 695,126
Pan-Territorial Fund 0 0 0 1,078,888 2,599,000 3,677,888
TOTAL G&C 19,408,688 22,112,744 8,093,274 17,648,607 17,632,349 848,95,662
TOTAL SINED
Source: Actual G&C expenditures from FNITP

A comparison of Tables 1 and 2 above reveals that actual Grants and Contributions expenditures are close to planned/forecast spending, but are slightly below expected levels. Since EPMRB did not have access to all planned/forecast and actual spending for Operations and Maintenance, the evaluation was unable to comment on changes in these areas.

1.2.8 Design and Delivery

Territorial Investment Plans

In anticipation of renewal of funding after the first round of SINED, discussions were held to consider what had been achieved against the Territorial Investment Plans, what remained to be done and any changes in the economic environment. Upon renewal of funding in 2009, new five-year Territorial Investment Plans were developed in consultation with key economic players in each territory. In this way, individual Plans were tailored to the needs of each of the three territories, which are at different stages in their development.

The process for developing the Plans is determined by each region. For the second round of SINED, the Nunavut regional office brought major stakeholders together in a workshop setting to develop the Plan, while the Northwest Territories and Yukon held smaller meetings with key players. There are also differences in how the Plans are structured. Nunavut and Yukon offices aligned their Plans with sectors, but the Northwest Territories structured their Plan along the Targeted Investment Program themes. Once complete, Plans are approved by the Minister.

There are oversight processes/structures available to revisit Plans on an annual basis, through an advisory group or bi-lateral meetings, to ensure Plans remain aligned with current realities. The extent to which these oversight processes/structures have been used varies between territories.

Call for Proposals

SINED project proposals are solicited through targeted or general calls for proposals, however, the approach varies between SINED components. Targeted Investment Program projects are typically the result of a targeted call for proposals (i.e., proposals are sought from specific organizations). General calls for proposals, where multiple organizations are invited to submit a proposal, may be used as well, but are limited to a specific investment streams or a portion of the total annual budget. Innovation and Knowledge uses targeted or general calls for proposals, which can vary per year and by territory. Due to the very limited funding involved, no call for proposals is done for Partnership Advisory Forums.

Preparation of the Statement of Intent

While not mandatory, applicants are encouraged to submit a Statement of Intent for each funding request to the CanNor office in the territory where the project is to take place. This allows CanNor to conduct a quick review to determine if the project is eligible and can be considered for funding. Alternatively, project proponents may discuss an idea directly with an Economic Development Officer, however, these verbal discussions provide no certainty that the final project description will meet the necessary criteria. After reviewing the Statement of Intent, the Economic Development Officer will advise the applicant on the potential fit with SINED and whether to proceed with an application. An application form, with its more detailed documentation requirements, is usually not offered before this point.

Preparation of the Application for Funding

The application form for funding is not available on-line, however, an electronic template is shared with applicants. SINED does not have on-line forms because it was reasoned when SINED was first established that it would be best for proponents to contact the regional office for assistance in completing the forms. In addition, because of connectivity challenges, not all proponents would have on-line access. There is typically much back and forth between the Economic Development Officer before all details in the application form are covered. Upon completion of the application form, the applicant submits it to the CanNor regional office for consideration.

Review/Assessment of Application for Funding

Applications for funding are reviewed by Economic Development Officers to ensure they are eligible, complete and of satisfactory quality. Special attention is paid to partner(s) and leveraging opportunities, project objectives, expected results, and the applicant’s ability to carry out the project. The applicant’s history with CanNor is examined as well as their finances. Independent third parties are used if expertise is not available in the regional office. The Economic Development Officer prepares a Project Assessment Report and forwards it to an independent third party (usually the manager in the region) for quality review. The Project Assessment Report’s purpose is to ensure that the project has been planned well and has the potential of achieving expected results. Additional information may be requested from the applicant until the Economic Development Officer and Manager have all the information necessary to complete the assessment and make a recommendation.

Approval of Application for Funding

When a project is recommended for funding, a Project Synopsis Report and a Minister’s Decision Note are drafted by the Economic Development Officer and the Regional Manager and forwarded to the Regional Director for signature. This recommendation must then be approved by the Director General of Operations, the Deputy Minister and then the Minister.

At any time, the package can go back one or more steps in the process for editing, rewriting or clarification. Questions about the project often go back down the project approval chain to the region.Footnote 38

On approval of a project, the regional office is notified and when a copy of the Project Synopsis Report and Decision Note are received, the regional office proceeds to create the Contribution Agreement.

Preparation of Contribution Agreement

The Contribution Agreement is created in the First Nations and Inuit Transfer Payment SystemFootnote 39 by the CanNor regional office. The Regional Director must sign the Section 32 budget approval and then the package is submitted to AANDC for quality assurance in accordance with the Memorandum of Understanding between the departments. When signed off by AANDC, it is sent to the recipient for signature signifying acceptance of the offer for funding along with associated conditions and reporting requirements.

Monitoring and Reporting

Recipients are required to submit project status reports based on financial and program management reporting requirements. The timing of reports is determined by the Economic Development Officer based on the project workplan, scale, risk, amount of funding as well as the First Nations and Inuit Transfer Payment System requirements. Project status reporting requirements are discussed between the Economic Development Officer and the recipient either before or when the contribution agreement is being finalized. For single-year projects, the minimum requirement is a Project Status Report due 90 days after the project has started and an end-of-project report due 90 days after the project is completed. Multi-year projects are required to submit a Project Status Report on March 31 of each year to assess future cash flow and progress of the project workplan.

2. Evaluation Methodology

2.1 Evaluation Scope and Timing

The evaluation examined the relevance and performance of SINED (i.e. the Targeted Investment Program, the Innovation and Knowledge Fund, the Partnership and Advisory Forums, and the Pan-Territorial Fund between 2007-08 and 2011-12).

Terms of Reference for the evaluation were approved by the Vice President of the Policy, Planning, Communications, and Northern Projects Management Office branch of the Canadian Northern Economic Development Agency on February 25, 2013. The Terms of Reference were also brought to AANDC's Evaluation, Performance Measurement and Review Committee for information purposes in February 2013. Data collection and site visits to each of the territories were conducted between March and July 2013. Preliminary findings were shared with the Working GroupFootnote 40 on July 30, 2013 and a draft final report was circulated to the Working Group on September 6, 2013.

2.2 Evaluation Issues and Questions

In line with Treasury Board requirements, the evaluation looked at relevance and performance. See Evaluation Matrix in Annex A for evaluation questions and proposed methodologies to answer questions.

2.3 Evaluation Methodology

2.3.1 Data Sources

The evaluation's findings and conclusions are based on the analysis and triangulation of the following lines of evidence:

Literature Review

A literature review of academic and government reports was conducted to examine economic development strategies employed in other circumpolar countries and remote regions and determine which strategies yielded the most promising results. SINED was compared to these recognised best practices.

The literature review also looked at government and industry studies, reports and statistical analysis relating to trends in economic opportunities and challenges in the North. This analysis informed the ongoing need for SINED programming. Current federal initiatives and strategies relating to economic development in the North were also looked at to test SINED alignment with Government of Canada priorities.

Document and File Review

CanNor provided the evaluators with a list of projects funded through SINED between 2007-08 and 2011-12. The list contained project descriptions and projects were classified by recipient, year, project amount, sector (for Targeted Investment Program projects), region and theme (for Targeted Investment Program projects). This list of projects was used to determine the distribution of funding by region and theme. It was also used to better understand which sectors were supported, identify the major recipients, and quantify investments in capacity.

The quantification of investments in capacity is linked to the assessment of the immediate outcome "Increased ability of Northerners to respond to economic development opportunities". The first step to understanding the impact of access to and use of capacity-building products and services is to calculate investments in this area.

This document review also looked at internal SINED documents such as planning and program reports, program manuals, templates, foundational documents, the performance measurement strategy, recipient reports, previous reviews, evaluations, and audits. These documents contributed to the development of the program profile and the assessment of its performance.

Key Informant Interviews

A total of 46 interviews were conducted: 15 were with SINED staff and 31 with recipients. Seven SINED staff members interviewed were from senior management (Director and above) and the remaining were Economic Development Officers and Managers. Recipients selected for interview represented the largest projects (in terms of dollar value) that were funded in each sector. However, some recipients were selected for interview based on the perceived success of their projects.

The interviews were based on standardized, largely open-ended questions that addressed evaluation issues pertaining to program design and delivery and performance. The majority of interviews were conducted in person during site visits, however, some interviews were conducted by phone to accommodate schedules.

Interview notes were recorded and analyzed manually to identify patterns in responses. The final report presents these patterns in the findings, so no individual may be identified.

Case Studies

A total of 12 case studies were conducted. Seven sector case studies were completed to explore the opportunities and threats facing major sectors in the North, review the types of projects supported by SINED in each sector, identify major recipients and document the impacts of SINED investments.

The sector case studies were selected based on the analysis of the list of SINED projects and input from the Working Group that was created for the purposes of this evaluation. The sectors that received the most funding were selected along with some smaller emerging sectors such as fisheries and renewable/alternative energy, which have responded particularly well to SINED investments. The seven sector case studies were: Energy, Alternative Energy, Cultural Industries, Fisheries, Geoscience, Mining and Tourism

The sector case studies incorporated interviews with recipients, a review of program documentation and literature relating to the sector in the North, project products and reports.

Five other case studies were completed. These took a closer look at the impacts of capacity investments, Innovation and Knowledge, Partnership Advisory Forums, Pan-Territorial Fund and broadband/information communication technologies on SINED outcomes. The case studies consisted largely of document reviews, which looked at the types of projects funded and their impacts, however, interviews were also conducted with recipients working on broadband.

Survey

As part of the geoscience case study, a survey was administered to users of geoscience data produced by the Nunavut geoscience office to further supplement information found in project files. Five organizations were surveyed - two representing mining/exploration companies, one Aboriginal organization and two municipalities. The survey was drafted with the assistance of the Nunavut Geological Society which also provided names and contact information for survey recipients. Surveys were e-mailed and reviewed by EPMRB.

2.3.2 Considerations, Strengths and Limitations

Key Considerations

The design and delivery of SINED, lessons learned and best practices would have been greatly informed by a comparative analysis with programs in other Regional Development Agencies. This approach was proposed as an additional line of evidence for this evaluation, but was removed because it was felt that this information could be gathered through an existing Regional Development Agency Working Group. Discussions of the Regional Development Agency Working Group were not documented as part of this evaluation.

Attribution/Contribution

The evaluation acknowledges that other factors may have contributed to SINED's expected results/outcomes making it difficult to isolate SINED's specific influence/impact. As a result, each line of inquiry was designed to best overcome the issue as much as possible. Analysis has been confined to program participants to limit the effects of external variables. Also, immediate impacts have been sought out with the knowledge that the expected results/outcomes of many investments, especially in capacity building, may not be seen for many years.

Strengths

The strengths of this evaluation include the use of multiple lines of evidence, as indicated in the evaluation matrix, in order to triangulate findings and increase reliability. With key informant interviews, a significant sample size incorporated the perspectives of all key SINED stakeholder groups. The methodology for the evaluation, including the data collection instruments, was developed in consultation with CanNor officials and the evaluation Working Group.

Limitations, Analytical Challenges and Mitigation Approaches

The preference in all areas of inquiry is to have quantitative data and documented information complemented where appropriate by the informed views of stakeholders. Several limitations encountered by EPMRB evaluators include differences in how client data and information are reported across jurisdictions and over time, reliance on qualitative data, and differences in delivery across the three territories. Because no formal system is in place to collect performance information, EPMRB had limited access to quantitative data on SINED activities and outcomes. So, SINED's effectiveness, such as improved performance due to enhanced behaviors, was a challenge to ascertain. Other methodological and analytical challenges are listed below by line of evidence.

Document/File Review

(a) Difficulty isolating "sectors"

The analysis of the list of funded projects presented the greatest challenge for the document and file review. The list of projects provided by CanNor was generated through the First Nations and Inuit Transfer Payment System. While the financial information proved solid, EPMRB noted inconsistencies in the assignment of sectors across regions. The sector classifications came from the Territorial Investment Plans, which are unique in each territory and are not subject to a common definition in SINED program documentation.

For example, one territory had "geoscience and mining" as one sector while another territory tracked geoscience projects separately from mining projects. Similarly, one territory included broadband projects under two sectors, referring to one as "broadband" and the other as "economic infrastructure," while another territory included broadband projects under "information communication technologies". In one case, fisheries projects were included under commercial/traditional harvesting while another territory listed fisheries as a separate category.

EPMRB looked at the entire list of sectors and decided to track the following 10 sectors which appeared most consistently:

  1. economic infrastructure - split into broadband, information communication technologies, energy, renewable energy and other
  2. commercial/traditional harvesting - includes projects related to sealing; fur production sale and marketing; muskox; caribou, and; other sport hunting and/or sport fishing
  3. cultural industries - including festivals, film, music, culture, arts and crafts
  4. fisheries
  5. forestry
  6. geoscience
  7. mining
  8. small and medium-sized enterprises
  9. tourism
  10. trade – which includes economic development strategies, trade shows and conferences not specific to a particular industry
(b) Difficulty determining "capacity" projects

SINED has made significant foundational investments in the emerging economies in the territories to set the stage for future economic development and to respond to the immediate outcome "Increased ability of Northerners to respond to economic development opportunities". In order to understand the impact of capacity investments, EPMRB first had to categorize and quantify capacity projects. SINED program documentation does not contain a definition of capacity. The only guidance came from a list of eligible projects that could be funded under the capacity theme of Targeted Investment Program, which included research projects, development of materials, curriculum, standards which support capacity development and capacity development initiativesFootnote 41.

Broadly defined, capacity projects could include not only the building of individual capacity through skills training, but also organizational capacity through the production of information products that increase the capacity for decision making. Public Works and Government Services Canada's definition of capacity "consists of four basic elements: creating an enabling environment with appropriate policy and legal frameworks; institutional strengthening and development, including local community participation; human resources development; awareness building and education at all levels of society".

Internationally, the Organization of Economic Cooperation and Development defines capacity development as "the process by which individuals, groups and organizations, institutions and countries develop, enhance and organize their systems, resources and knowledge; all reflected in their abilities, individually and collectively, to perform functions, solve problems and achieve objectives." These definitions clearly support a broad definition of capacity and, if applied to SINED, it could be easily argued that projects under the theme of "building the knowledge base" could also be considered capacity as research projects under this theme generate new information and analysis that contribute greatly to the capacity of organizations to make decisions.

In order to come up with a definition of capacity, EPMRB looked to the list of projects considered by SINED staff to be capacity related. In the list of projects, SINED staff labeled capacity projects under the sector classification and/or under the Targeted Investment Program theme. The types of projects that were labeled this way included the following:

  1. Training/skills development – including development of training tools/materials, training programs/curriculum, e-training, skills inventories, training plans/strategies, training software, training assets and facilities
  2. Conferences/workshops/summits/forums/symposiums
  3. Organizational development – including consultations, expert knowledge, project management, forms development, e-business products/solutions, organizational reviews and restructuring, Information Technology upgrades, partnership development agreements/negotiations, website development, partnership and business development programs
  4. Research projects – including feasibility studies, pilot projects, surveys, and mapping
  5. Strategic planning – including business plans, management planning, and economic development planning
  6. Loan top-ups

Based on the above, it would appear that SINED staff also took a broad definition of capacity, however, limitations in the classification of projects contributed to a lower number of capacity projects being identified. To explain, when selecting a sector for the projects, SINED staff had to choose between the actual sector (i.e. energy, fisheries, mining) and capacity, so a project could belong to a sector or to capacity, but not to both. In fact, projects frequently belonged to two sectors i.e. both capacity and mining but SINED staff most often selected the actual sector. As a result, capacity projects are underrepresented in CanNor tracking. Similar observations were made regarding the assignment of themes where projects could easily touch on multiple themes.

Ultimately, EPMRB decided to apply a broad definition of capacity to the list of projects. Projects were identified with a particular sector and if they were similar to the types of projects labeled by SINED staff as capacity, they were also labeled as a capacity project. All of the capacity projects were then further broken down by type of capacity projectFootnote 42. The impact of this re-classification process was a significant number of projects being classified as capacity, as shown in Section 4.

Key Informant Interviews

The weakness of all key informant interviews is their reliance on the opinions and perceptions of individuals, which are subject to bias. To mitigate this risk, the evaluation does not reference individual interviews, but reports on trends or themes in responses to questions. In addition, interviewees were asked for supporting evidence for statements and key informant interviews are cross-referenced with other evaluation lines of evidence. Validating findings through other primary and secondary research mitigates this limitation.

Case Studies

The biggest challenge for the case studies was the lack of consistent performance data. Project outcomes were agreed upon with proponents on a project-by-project basis and documented in project reports. However, due to the non-implementation of the performance measurement strategy and the overwhelming variety of projects, there is little consistency in the measurement of outcomes. The case studies sought to overcome this challenge by looking at impacts of major projects in supported sectors, identifying potential indicators for success, and then seeking out evidence that demonstrated progress. As a result, impacts documented through the case studies are project-specific and cannot be rolled up to describe the impact of investments on a territory or sector, nor can they be linked to expected outcomes. Instead, they provide anecdotal evidence of successful outcomes.

2.4 Roles, Responsibilities and Quality Assurance

EPMRB was responsible for the overall management of the SINED evaluation and for ensuring the project proceeded as scheduled. EPMRB planned the evaluation, organized the Working Group and engaged in data collection and analysis. Professor Ken Coates was hired to conduct the literature review portion of the evaluation, which was overseen by EPMRB. Further, EPMRB managed the engagement of stakeholders and other partners, translated the final report and reviewed and approved all deliverables.

The evaluation Working Group comprised of CanNor representatives from headquarters, the National Capital Region and the regional office in each territory was established as the main advisory body throughout the evaluation. This group was responsible for reviewing and providing insights on the methodology report, preliminary findings deck and the draft evaluation report. Additionally, a structured internal peer review and response process was undertaken on the evaluation methodology, including the research instruments, and the draft evaluation report.

CanNor was responsible for preparing the Management Response and Action Plan, which addresses evaluation recommendations. CanNor will be responsible for publishing the final report and the Management Response and Action Plan on the internet.

The role of the Senior Management Committee at CanNor, consisting of the Deputy Minister, Assistant Deputy Minister, regional directors, Director General of Operations, and Director General of Policy and Planning, included reviewing and approving key documents, for example, the Terms of Reference, final report and Management Response and Action Plan. The same documentation, as appropriate, was then submitted to the Evaluation, Performance Measurement and Review Committee at AANDC for information purposes only.

3. Evaluation Findings - Relevance

A program continues to be relevant if there is a clear demonstrable need for it, if it aligns with federal priorities and is consistent with federal roles and responsibilities. The continued relevance of SINED was explored primarily through the literature review, and was supplemented by the case studies and key informant interviews. The literature review examined roles, responsibilities and priorities of the Government of Canada in northern economic development. Case studies looked at continued need by examining the threats and opportunities facing key sectors in the North. Key informants were asked about the importance of SINED, duplication with other departments and jurisdictions, and possible priorities of a potential next generation of SINED.

Based on the results of the lines of enquiry above, SINED continues to be relevant. Clearly, the federal government has a role to play in the North and recent public statements, announcements and action by the Government of Canada, show that northern economic development is a priority. Continued need for SINED is demonstrated by the enormous potential for economic growth in the North; however, it would appear that the need for investments exceeds SINED's scope and resources, signaling a need for a more coordinated approach to federal investments in the North. These findings are summarized below.

3.1 Continuing Need

Finding: Primary arguments for the continuation of SINED include the need to address the enormous challenges to economic development in the North, a shortage of funding, the need for further capacity development and economic diversification, and the importance of a northern approach to economic development in the North.

As outlined in Section 1.2.1, the tremendous opportunities for economic growth in Canada's north are eclipsed by formidable challenges to development. Both recipients and the literature review identified the need for significant investment in transportation and communication infrastructure, energy, housing and capacity in the North, however, they also revealed limited funding available from territorial governments, other federal departments and the private sector.

Sixty-one percent of recipients argued for the continuation of SINED due to limited funding for northern economic development from other sources. Only small pots of money are available through the territorial governments, which are largely dependent on federal transfers because they do not have the tax or industrial base that southern provinces do. As a result, the territorial governments can only allocate minimal resources for economic development. Furthermore, existing agreements between the federal government and territories place restrictions on the ability of territorial governments to take on debt, so they have limited capacity to borrow for much needed infrastructure. Private sector investment is equally difficult to secure because of high capital costs and uncertain returns on investment.

Some funding is available through other federal departments. For example, Industry Canada has an infrastructure fund for broadband and Natural Resources Canada has the Geo-mapping for Energy and Minerals program. However, these funds are very targeted and not all aspects of a proposed project are eligible for funding. As a result, SINED is often used to fill the gaps even for projects that would normally fall within the jurisdiction of another federal department. Sixty-seven percent of CanNor officials interviewed indicated that SINED does not duplicate other programs, but complements them.

Fifty-eight percent of recipients indicated that SINED contributed greatly to capacity development in the North and argued for continued support for the development of both human and information capacity. Human capacity issues were highlighted as a challenge in a number of the sector case studies and as shown in Table 4, a significant number of SINED projects were designed to address capacity issues in key sectors.

For example, in the area of mining, companies operating in the Northwest Territories couldn't find skilled workers as the industry increased, so SINED supported mine training to increase local employment in the industry instead of relying solely on fly-in workers. Importantly, SINED's support of mine training complemented the role of Human Resources and Skills Development Canada, now Employment and Social Development Canada. Human Resources and Skills Development Canada funded training and SINED funded organizations to deliver the training, curriculum development and the purchasing of equipment, none of which could be funded by Human Resources and Skills Development Canada.

Experts in skills development interviewed suggested that increased human capacity benefits industry, contributes to community well-being, and benefits individuals through household income. The experts asserted that people who are employed are better role models for their kids, have increased self esteem, feel like they have something to offer the community and have a purpose, all of which helps to address social issues.

SINED support for information capacity was also seen by recipients as addressing a fundamental need in the North. As a new frontier for development, there is still much to learn about northern resources and economic opportunities. Research into fish stocks, geoscience, broadband, skills inventories, and feasibility projects are some examples of information capacity projects supported by SINED. Recipients argue these projects help define opportunities, reduce risk, guide investment decisions and contribute to the diversification of the economy.

The need for economic diversification is critical in the North to protect against the boom-bust cycle of the mining industry. For instance, in Nunavut, SINED funding was used to conduct fish stock assessments and assist the arts and crafts industry. In the Yukon, SINED money was used to explore cold climate technologies and in the Northwest Territories, to support renewable energy in forestry. Recipients appreciated that SINED recognized the importance of economic diversification and invested in projects outside the mining and hydrocarbon sectors. SINED also invested in some projects, which were deemed more "risky" by other funders because they involved emerging technologies, new approaches or non-traditional industries.

The final argument advanced for the continuation of SINED by recipients and SINED staff, and echoed in the literature review, was the need for policy interventions like SINED that are sensitive to territorial circumstances. Before SINED was created, the North and southern Ontario were the only regions in Canada where the federal government did not have a regional development agency. Sixteen percent of recipients and 43 percent of SINED staff spoke to the importance of having a northern presence. With offices in each of the territories and headquarters in Iqaluit, CanNor has an established presence in the North.

According to interviewees, the engagement process for establishing the Territorial Investment Plans is a strength of SINED and should continue. Along the same lines, the literature review described the importance of being respectful of territorial institutions, particularly Aboriginal organizations, to gain buy-in and local support. It noted that the most successful policy initiatives capitalize on local knowledge and involve regional leaders and organizations in the priority setting and decision-making procedures.

3.2 Alignment with federal priorities

Finding: There is a very strong alignment between SINED objectives and federal priorities. Recent announcements and commitments made by the Government of Canada confirm that SINED's objectives remain key priorities for the federal government.

Northern economic development has been gaining prominence since the creation of SINED in 2005. Federal announcements and investments since that time clearly demonstrate that the North is a priority for the Government of Canada. In the October 2007 Speech from the Throne, the federal government committed to the creation of a northern strategy. Canada's Northern Strategy, announced in 2009, communicated the Government of Canada's vision for the North and committed to action in four priority areas: 1) exercising Canada's Arctic sovereignty; 2) protecting Canada's environmental heritage; 3) promoting social and economic development; and 4) improving and devolving northern governance. Northern economic development has been mentioned consistently as a top priority in Speeches from the Throne since 2007, showing its importance to the federal government.

In 2007, the Government of Canada made a number of commitments to advance economic development in the North. It announced streamlining of major project approvals, Aboriginal skills training and better housing in the North for First Nations and Inuit. As part of asserting sovereignty in the Arctic, the federal government announced it would complete a comprehensive mapping of Canada's Arctic seabed. It also committed to build a world-class Arctic research station that would be on the cutting edge of Arctic issues, including environmental science and resource development.

In the 2008 Speech from the Throne, the federal government continued to focus on the advancement of Northern economic development through the reduction of regulatory and other barriers to pipeline extensions and the creation of a new northern economic development agency. In 2009, the federal government renewed SINED and transferred it to CanNor. In addition, as part of the 2009 Economic Action Plan, funding was made available for further northern economic investments through Community Adjustment Fund a national program delivered through Regional Development Agencies. CanNor delivered Community Adjustment Fund in the North.

The 2011 Speech from the Throne announced, "Our Government has made Canada's north a cornerstone of its agenda". It then outlined commitments to complete the Dempster Highway by linking Inuvik to Tuktoyaktuk; invest in First Nations land management; improve access to clean water; deploy clean energy technology in Aboriginal and northern communities; and expand adult basic education programming in the territories. Clearly, northern economic development remains a key priority for the federal government.

3.3 Alignment with federal roles and responsibilities

Finding: The federal government's jurisdiction is clearly mandated in legislation. Although some "province-like" responsibilities are being devolved to territorial governments, the lack of a tax base in the North translates to continued reliance on federal transfers in the short and medium term and continued federal responsibilities.

Federal jurisdiction for Canada's north is outlined in a number of pieces of legislation. The Canadian Constitution and statutes enacted in the late 1960s and early 1970s, including the Department of Indian Affairs and Northern Development Act, give responsibility for the North to the federal government. Responsibilities are further defined in statutes enacting modern treaties north of 60°, such as the Nunavut Land Claims Agreement Act and self-government agreements, such as the Yukon First Nations Self-Government Act.

While some "province-like" responsibilities for governance and administration have been successfully transferred to the territorial governments, control over land and resource management is in varying stages of devolution from AANDC to the territories. In 2003, land and resource management responsibilities were successfully devolved in the Yukon. In the Northwest Territories, a final agreement on the devolution of similar responsibilities has been signed, while Nunavut is beginning to negotiate a devolution plan.

In the end, economic development remains a shared responsibility with the federal government, territorial governments and self-governing First Nations all playing a role. In addition, as noted earlier, until the territories have an established tax base, they will also continue to rely on federal transfers after devolution. In addition, the Government of Canada's economic development activities in the North are a continuation of a long history of regional economic development. The literature review noted that since Confederation, the federal government has been preoccupied with providing a reasonable measure of equality of economic opportunity across the country and noted a national commitment to fairness and promoting business and job creation. Throughout Canada's history, the federal government has worked to stabilize regional economies and promote growth and poverty reduction in poorer areas, so it is reasonable to expect this will continue.

4. Evaluation Findings – Performance - Effectiveness / Success

Programs are proven effective or successful when it can be clearly shown that they have met or exceeded stated objectives or outcomes. Ideally, program effectiveness is determined by looking at stated outcomes and data collected against those outcomes. In the absence of the systematic collection of performance information against identified outcomes, as in the case of SINED, other methodologies are required. For this evaluation, effectiveness was examined through key informant interviews with SINED staff and case studies. The case studies involved interviews with major recipients in all three territories and a review of project reports and other literature identified by project proponents as providing evidence of success.

While the evaluation was able to determine how SINED funding was distributed, it was not able to conclude on the impact of those expenditures on expected outcomes because performance information collected related to specific projects and could not be rolled up to the outcomes identified in the performance measurement strategy. Case studies on the impact of Innovation and Knowledge, Partnership Advisory Forums, Pan-Territorial Fund and capacity investments were not successful due to difficulties accessing final reports, but the other case studies did reveal some positive outcomes. A breakdown of SINED expenditures, the state of performance measurement at CanNor and case study findings are documented below.

Table 3: Total SINED Investments by Sector and Funding Stream 2007-08 to 2011-12
Targeted Investment Program Innovation and Knowledge Partnership Advisory Forums Pan-Territorial Fund TOTAL % of total funding
Economic infrastructure
Broadband $1,132,800 $315,000 $0 $0 $1,447,800 2%
Information communication technologies $1,538,393 $372,772 $25,000 $0 $1,936,165 2%
Energy $1,524,740 $115,000 $113,915 $0 $1,753,655 2%
Renewable/Alternative Energy $2,591,547 $206,600 $0 $0 $2,798,147 3%
Other $1,301,918 $662,318 $25,000 $0 $1,989,236 2%
Commercial/Traditional harvesting $397,450 $126,103 $0 $0 $523,553 1%
Cultural industries $4,275,521 $442,030 $0 $0 $4,717,551 6%
Fisheries $1,891,031 $109,315 $0 $0 $2,000,346 2%
Forestry $495,610 $126,974 $0 $0 $622,584 1%
Geoscience $23,632,000 $0 $0 $0 $23,632,000 28%
Mining $5,258,500 $493,244 $5,000 $0 $5,756,744 7%
Small and Medium Enterprises $7,676,006 $342,944 $0 $0 $8,018,950 9%
Tourism $13,713,095 $715,306 $118,000 $1,598,500 $16,144,901 19%
Trade $1,522,345 $907,981 $215,245 $0 $2,645,571 3%
Other sectors $7,260,595 $1,375,510 $192,966 $2,079,388 $10,908,459 13%
TOTAL $74,211,551 $6,311,097 $695,126 $3,677,888 $84,895,662 100%

Eighty-seven percent of SINED investments were directed towards 10 areas indentified in the Territorial Investment Plans: economic infrastructure, commercial/traditional harvesting, cultural industries, fisheries, forestry, geoscience, mining, small and medium enterprises, tourism and trade. Targeted Investment Program investments were mainly in geoscience, tourism and small and medium-sized enterprises. Innovation and Knowledge investments touched on all areas except geoscience. Trade and tourism were favored by Innovation and Knowledge, in addition to other sectors. Partnership Advisory Forums invested mainly in trade, other sectors and energy. Finally, Pan-Territorial Fund invested primarily in other sectors and tourism. Overall, geoscience received 28 percent of total SINED funding, tourism 19 percent and other sectors received 13 percent.

Finding: Performance data collected relates to specific project outcomes and not to outcomes identified in the Performance Measurement Strategy. Processes and systems are not in place to collect and report on performance information or results.

Key informant interviews with SINED staff revealed that there is no formal performance measurement system in place even though a performance measurement strategy was developed following the 2008 evaluation and the Auditor General of Canada recommended in 2010 that CanNor "clarify the objectives of its economic development programs in the Northwest Territories and develop a strategic approach that includes … clear and coordinated objectives, effective performance measurement and consistent reporting on results"Footnote 43. While SINED staff recognised the importance of performance measurement, several noted that priority was given to getting funds out and projects underway. A lack of human resources dedicated to performance measurement and a lack of systems to store and manipulate performance information also hampered work in this area.

In the absence of any agreement or requirement to report on specific indicators related to SINED outcomes, Economic Development Officers identified performance measures on a project-by-project basis. This has resulted in significant variation in the type and quality of indicators used. The general process for collecting performance information is as follows: Performance indicators are agreed upon at the start of a project and written into the Contribution Agreement. Project proponents report on performance indicators in project reports which are submitted to CanNor and, following verification that reporting requirements have been met, stored in the document management system called Comprehensive Integrated Document Management (CIDM). Links to the reports are made in First Nations and Inuit Transfer Payment System, the financial management system used to track transfer payments. It is at the next level, the aggregation of individual project data into program monitoring and tracking results, the process falters. There is no documentation, regular roll-up, or analysis of performance measures contained in project reports.

While the First Nations and Inuit Transfer Payment System is effective for tracking funding, the system is inadequate for performance monitoring and reporting. SINED performance reporting is done manually, on an ad-hoc basis, usually in response to a ministerial request. For example, to answer an inquiry about the impacts of tourism investments in the Northwest Territories, an officer must conduct a manual search for relevant reports in First Nations and Inuit Transfer Payment System and then analyze them. As SINED gained more visibility through the creation of CanNor, information requests have grown, but the ability of staff to respond to increased requests for information has been hampered by this system, and human resource limitations.

It is worth mentioning at this point that EPMRB was unable to conclude on the effectiveness of Partnership Advisory Forums, Innovation and Knowledge and Pan-Territorial Fund due to difficulties accessing documents in First Nations and Inuit Transfer Payment System. EPMRB's attempt to find documents through First Nations and Inuit Transfer Payment System proved futile as it appeared that these had not been uploaded. SINED staff did not report similar difficulties accessing project reports.

Finding: Case studies on energy, renewable energy, geoscience, mining, fisheries, tourism and broadband revealed some SINED projects that have achieved some positive outcomes.

Case studies identified indicators of success for projects in key sectors, which are documented below:

a) Impact of SINED investments in energy and renewable energy

Between 2007-08 and 2011-12, SINED invested $4.6 million in energy and renewable energy. The case study covered capacity/knowledge building projects amounting to $2.1 million, which represents 47 percent of the total SINED investment in energy and renewable energy. The projects reviewed in the case study were in the Northwest Territories and Yukon and included studies, conferences, guides, public engagement and workshops on natural gas, pipelines, geothermal from abandoned mines sites, biomass, wind energy and waste to energy. The project proponents examined in the case study were Yukon Energy, Alaska Highway Aboriginal Pipeline Coalition, Department of Environment and Natural Resources of the Government of Northwest Territories and the City of Yellowknife.

Energy projects examined by EPMRB evaluators showed that though these had generated new knowledge, the impact of this new knowledge on the energy sector was not documented and, in addition, the extent to which this knowledge was made available to, and accessed by, the public is also not known. For example, the Eagle Plain Case Study determined and compared costs of different options for electricity generation from natural gas; however, no information is provided about the usage or impact of this new knowledge on the energy sector.

On the other hand, some renewable energy projects that were assessed led to concrete steps to advance biomass energy. For example, EPMRB found that studies in the Northwest Territories led to the identification of a biomass industry, attracted private sector partners and led to the development of Forest Management Agreements with First Nations for harvesting rightsFootnote 44. Also, the Fort Smith Métis Council used the findings of a feasibility study to convert the heating systems in four buildings to bio-fuel, which has resulted in estimated savings of $17,000 per yearFootnote 45.

The case studies on energy and renewable energy highlight the fact that most northern communitiesFootnote 46 as well as mining developments rely upon diesel generation for electricity. The North contains potentially vast hydro resources, which implies not only an economic opportunity, but also an expedient way to reduce greenhouse gas emissions. Other renewable energy sources (e.g., wind, biomass, geothermal) all have potential to reduce the North's reliance upon imported diesel. To some extent, SINED funds have increased attention on these areas.

b) Impact of SINED investments in mining

Between 2007-08 and 2011-12, SINED invested $5.8 million in mining. The case study covered capacity/knowledge-building projects amounting to $2.5 million, which represents 43 percent of the total SINED investment in mining. The projects reviewed in the case study were in the Northwest Territories and Yukon and were focused on training development, facilities, "asset acquisition" and additions to simulators. Recipients were the Mine Training Societies in the Northwest Territories and Yukon.

The impacts of the above investments documented in the case study are as follows: In the Northwest Territories, $226,000 of SINED funding allowed for the training of 22 participants and purchase of training equipment (which appears to have been donated to communities). Further results, such as the number of additional trainees and employment afterwards, were not provided.

In contrast, performance measures collected by the Yukon Mine Training Association showed a direct impact of SINED investments on employment. The Yukon Mine Training Association trained 852 applicants and 457 graduates obtained jobs between April 2008 and March 2011Footnote 47. In 2010-11, almost 200 trainees in the Yukon were placed in careersFootnote 48.

c) Impact of SINED investments in geoscience

Between 2007-08 and 2011-12, SINED invested $23.6 million in geoscience. The case study covered capacity/knowledge building projects amounting to $20.3 million, which represents 86 percent of the total SINED investment in geoscience. The projects reviewed in the case study were in all three territories and included geophysics (i.e., magnetic property of rocks), geochemical (i.e., analysis of soil samples for mineral composition), and geological maps and geo-technical reports (i.e., for sand, gravel, permafrost). The project proponents examined in the case study were the Canada Nunavut Geoscience Office, the Department of Economic Development and Transportation of the Government of Nunavut, Industry Tourism and Investment from the Government of Northwest Territories and the Yukon Geological Survey.

The case study showed that SINED funding has been instrumental in increasing the availability of geoscience information for the territories' mineral industry and territorial, municipal and community planners. The Canada Nunavut Geoscience Office calculated the extent to which geoscience was made available to the public by looking at the amount of information available and the number of downloads. In Nunavut, 4 terabitsFootnote 49 of information is available online and approximately 8,000 reports have been downloaded. Similarly, a data mining project undertaken by the Yukon Geological Survey resulted in the scanning of 4,364 documents comprising 2.29 GB of data, which is new information available for use.

Even though SINED geoscience investments are meant to contribute to the immediate outcome "expanded publically-accessible information", the ease of access to geoscience information by the public was not monitored by SINED. In response, the EPMRB survey of geoscience users in Nunavut asked respondents to comment on their experiences identifying and accessing geoscience data through the Canada Nunavut Geoscience Office. The majority of respondents indicated that the information they were looking for was easy to access and the ease of obtaining copies of reports ranged from "easy" to "very easy". Similar information was not available for Yukon and Northwest Territories.

Geoscience information is used by private sector companies and has contributed to leveraging private sector investments in the territories. For instance, preliminary results of the "Yukon Geological Survey" Client Questionnaire for 2012-2013 reveal that geoscience information has resulted in staking of claims, informed investment decisions, helped companies identify new areas of interest and reduced risk. All respondents reported that they "are either planning to, or already have, integrated the data into their corporate datasets". Further, press releases by companies (Arcturus Ventures Inc. and Commander Resources Ltd.) exploring in the Yukon state that "SINED-funded publications provided data that positively influenced exploration".

Similarly, the two exploration companies in Nunavut surveyed by EPMRB agreed that the geoscience information reduced risk, the amount of time required to advance exploration and reduced company costs by identifying areas not viable for exploration. Both companies indicated they have invested as a result of Nunavut geoscience information.

The case study found that geoscience information has been used by federal, territorial, municipal governments and First Nations to inform land use planning. Interviews with territorial government representatives in Nunavut revealed that geoscience data has been used to plan the location of a new airport and deep water harbour. Interviews with territorial government representatives in the Northwest Territories showed that geoscience information has been used in the planning of the route for the extension of the Dempster highway to Tuktoyaktuk. The survey conducted by EPMRB of geoscience users in Nunavut revealed that geoscience information has also been used by Aboriginal organizations to make decisions regarding cultural/heritage sites and has reduced the costs of infrastructure planning and land use planning in communities.

The geoscience case study also shows new discoveries. According to the Canada Nunavut Geoscience Office, there have been discoveries of 11 new potential sites for soapstone and nine undeveloped deposits contain substantial resources of high-quality stone. These discoveries of raw material for high-value-added fine art carvings have been well received by local artists and communities. In the Yukon, a geoscience study led to the identification of a source of aggregate half the distance of the existing site. The local community estimated a potential savings of $34,000 in hauling costsFootnote 50.

As shown above, there are many indicators that SINED investments in geoscience have resulted in positive impacts, but since these impacts are measured or reported on differently in each territory and for each project, the evaluation was unable to conclude on the overall impact of all investments.

d) Impact of SINED investments in fisheries

Between 2007-08 and 2011-12, SINED invested $2 million in fisheries. The case study covered capacity/knowledge-building projects amounting to $1.9 million, which represents 95 percent of the total SINED investment in fisheries. The projects reviewed in the case study were in Nunavut and included research primarily relating to fish stock assessments, however, some funding was also provided for the testing of fishing vessels and a fisheries symposium. The project proponents examined in the case study were Department of Economic Development and Transportation and the Department of the Environment from the Government of Nunavut.

The fisheries case study found that in Nunavut, the annual offshore surveys, supported by SINED, contributed to a 90 percent increase in the turbot quota. The total landed value of turbot in Nunavut increased from $35.4 million in 2001 to $78.8 million in 2012Footnote 51. Stock assessments conducted for shrimp and arctic char also contributed to increases in catches. The total landed value of shrimp in Nunavut increased from $12.1 million in 2001 to $32.6 million in 2012Footnote 52.

The increases in quotas have contributed to growth and private sector investment in the fisheries sector. The case study found there are now four 100 percent Inuit-owned companies, which hold 100 percent of Nunavut's share of offshore shrimp and turbot allocations. When the industry began, these companies did not own a single vessel, however, in 2013 they have two offshore factory freezer trawlers and three large fixed gear vesselsFootnote 53.

e) Impact of SINED investments in tourism

Between 2007-08 and 2011-12, SINED invested $16.1 million in tourism. The case study covered projects amounting to $7.1 million, which represents 44 percent of the total SINED investment in tourism. The projects reviewed in the case study were in all three territories and included marketing projects, strategy development, feasibility studies, tourism infrastructure, product development, visitor exit surveys and research. The major recipients in each territory were examined in the case study and included the Department of Economic Development and Transportation from the Government of Nunavut, Nunavut Tourism, Northwest Territories Tourism Association, Department of Industry Tourism and Investment from the Government of Northwest Territories, Carcross/Tagish Development Corporation, Department of Economic Development, Department of Tourism from the Government of Yukon and Tr'ondëk Hwëch'in First Nation.

Sixty-eight percent of the projects looked at in the case study were marketing projects so, naturally, the case study sought to document the impact of SINED investments in this area. Although many measures for marketing campaigns were identified in the case studies, these measures were not uniformly applied to all marketing activities across the three territories. This can be seen in the impacts documented by each of the territories:

  1. The Northwest Territories analysis of marketing activities looked at the impact of individual marketing campaigns. For example, during the 2009 "Spectacular Northwest Territories" marketing campaign, the number of Twitter/Facebook followers increased from 4,000 to 19,000Footnote 54. In addition, over 50 percent of visitors recalled having seen "Spectacular Northwest Territories" advertisements. In 2008, 53 percent recalled the marketing campaign for that year compared to 32 percent who recalled the 2007 advertisements. The 2010 Globe and Mail campaign resulted in over 1,200 entries into a contest associated with the campaign, 500 information requests and 191 print guide packages being sent outFootnote 55. Finally, a pavilion at the 2010 Olympics which was funded by SINED had between 6,000 and 8,000 visitors daily. Additionally, it was featured in publications, newspapers and the nightly news. It was calculated that the resulting exposure was the equivalent of purchasing more than $150 million in advertizingFootnote 56.
  2. Alternatively, the Yukon reported on the overall impact of tourism initiatives. Outcomes recorded for 2010-11 indicate that the Yukon implemented marketing initiatives with a consumer reach of over 23.6 million; hosted 81 media journalists generating the equivalent of over $31.6 million in advertising; generated 230,505 requests for tourism information; and had 241,879 unique visitors to the tourism websiteFootnote 57.
  3. Nunavut Tourism took a similar approach to the Yukon and linked the success of their marketing campaigns to the following outcomes: 1) Increased requests for tourism materials such as travel planner, CD-ROM, maps, DVD and other promotional materials; 2) Increased media exposure through travel articles and television shows; 3) Increased bookings for market-ready operators; 4) New promotional materials; and 5) Number of images in photo library and hours of high quality video available for tourism promotionFootnote 58.

As shown above, the case study revealed many useful measures for documenting the success of marketing activities. Unfortunately, the lack of consistency between territories does not allow for a comparison of marketing activities nor an analysis of overall impacts of these investments.

In addition to marketing, SINED invested in tourism infrastructure and products such as interpretation centres, signage, beautification and aerial parks with a number of different partners. The impacts of investments to increase the number of visitors to Carcross/Tagish First Nation in the Yukon are particularly well documented. SINED provided funding for the establishment of a carving program, which led to the training of new carvers and the creation of 10 totem poles to attract visitors to the community. SINED also invested in the development of 35 km of bike trails, which employs 13 youth. The Mountain Hero Trail, which is part of this network, has received national and international attention as well as an "Epic Rides" designation from the International Mountain Bike AssociationFootnote 59. The case study documented an increase in employment and mentoring of over 20 First Nation and non-Aboriginal youth over past seven yearsFootnote 60 and an increase in visitors to the Carcross Tagish First Nation community, from 150 before the projects, to 2,000 in 2011.

SINED projects also attracted private sector investment. $9,000 was leveraged from hotels, companies, and First Air for the annual tourism conference in the Northwest Territories. SINED investments in tourism infrastructure and products at Mt. Sima Alpine Adventure Park, combined with the leveraged funds, led to an increase in season pass sales from $30,000 to $90,000 and the involvement of new sponsors (Royal Bank and Air North). In addition, nine tour operators confirmed packages to bring groups to the attractionFootnote 61.

Not so successful were investments made in the Great River Journey project in the Yukon. Between 2004-05 and 2008-09, SINED invested $3.1 million in this project. Two early independent studies of the project were very positive. However, when the project proponents indicated they needed an additional $1.2 million in operating funding to stay afloat in 2010, AANDC commissioned another third-party review to assess performance to date, the business model and long-term viability of the project. The assessment concluded that the business model was flawed, and even if the best-case scenarios for demand came to pass, the venture would operate at a loss. In response to this assessment, there have been no further investments in this project and according to the Yukon News, the eco-tourism company is bankruptFootnote 62.

Finally, the impact of SINED investments in tourism can be measured indirectly through visitor surveys. The 2001-02 Northwest Territories visitor survey recorded 53,917 visitors that spent $103.8 million. The 2010-11 visitor survey recorded 69,830 visitors that spent $110 million. The 2009 Northwest Territories visitor survey notes that 63 percent of Northwest Territories visitors can be considered Northwest Territories "promoters" of tourism, based on their willingness to recommend the Northwest Territories to others (up from 54 percent in 2008). While the visitor surveys are excellent tools for assessing the health of the tourism industry, their contents are inconsistent between territories and vary in year and frequency, so cannot provide a bigger picture of the impact of SINED investments in tourism.

f) Impact of SINED investments in information communication technologies and broadband

Between 2007-08 and 2011-12, SINED invested $3.4 million in information communication technologies and broadband. The case study covered projects amounting to $1.2 million, which represents 35 percent of the total SINED investment in information communication technologies and broadband. The projects reviewed in the case study were in all three territories and included training, infrastructure planning, feasibility studies and investments in information communication technologies/broadband infrastructure. The recipients examined in the case study included the Nunavut Broadband Development Corporation, the Department of Finance from the Government of Northwest Territories, Northern Native Broadcast Corporation and Katlodeeche First Nation in the Northwest Territories.

In 2010, SINED funded the Arctic Communications Infrastructure Assessment Report, which provides arguments and recommendations for the development of a reliable communications network in the North to establish sovereignty, service communities and support emergency response services. The final report is accessible through a companion website and, according to key informant interviews, has been referenced in Canadian Radio-television Telecommunications Commission hearings, newspapers and magazine articles. It has also been used as a source document for other organizations to support the prioritization of communications projects and has guided other agreements like the development of a Mackenzie Valley fibre link. In addition, the ten recommendations that came out of the report guide the Northern Communications and Information Systems Working Group, which is looking at how to address the recommendations.

The Nunavut Broadband Development Corporation received SINED support for two studies in 2011: one looked into quantifying the social and economic benefits of high-speed internet infrastructure to build a case for continued investment. The second was a study of the feasibility and cost of fibre-optic connectivity in Nunavut. Both studies had findings and recommendations, but the extent to which the studies have been made available to, and accessed by, the public is not known. Information on the extent to which recommendations have been implemented was not provided.

In contrast, the impact of infrastructure investments in information communication technologies are well documented by the Katlodeeche First Nation in the Northwest Territories. With SINED funding, Katlodeeche installed a mixed network of fiber-optic cable and Wi-Fi to provide internet access to the band office, day care, group home and community members. This investment has resulted in cost savings of $35,000 per year in licensing fees for the band office. They have also saved $5,000 per year in external technical services because they hired and trained someone from the community to provide technical support. This technician has held the position for four years. The network also allowed them to introduce video-conferencing, which has reduced travel expenses. Prior to the availability of video conferencing, Chief and Council were spending approximately $135,000 on travel. In 2011, travel expenses were $37,000 and, in 2012, they were under $18,000Footnote 63.

The infrastructure project has benefitted Katlodeeche First Nation in other ways. The new network provides a secondary communications system if phone lines go down, as happened after a 2007 fire. It improved safety in the community and also allowed for the delivery of better education through online education programs. The project has also opened up some business/income earning opportunities. The band office is currently looking into investing in 12 km of fibre-optic line to connect their network to Northwest Tel. This would provide better download speeds, but they could also potentially lease their infrastructure back to telecommunication companies wishing to move north.

g) Impact of SINED investments in cultural industries

Between 2007-08 and 2011-12, SINED invested $4.7 million in cultural industries. The case study covered projects amounting to $2.4 million, which represents 51 percent of the total SINED investment in cultural industries. The projects reviewed in the case study were in Nunavut and included marketing initiatives, strategy implementation, training and planning. The recipients examined in the case study included the Department of Economic Development and Transportation from the Government of Nunavut, Nunavut Film and the Nunavut Arts and Crafts Association.

SINED investments in the film industry in Nunavut supported training, global marketing and producer tours, however, beyond anecdotal evidence of success, information on the impact of these investments could not be found. The key informant interviews also revealed that many social benefits of the film industry are impossible to measure such as the pride film makers have in their work and the hope and self esteem that comes with it.

The impact of SINED investment in the arts and crafts industry was also not well documented. While key informants identified many successes, these were not captured in available reports. Outcomes identified in the interviews included the construction of studio space and the showcasing of Nunavut art in Paris, which led to the purchase of $20,000 worth of art by a gallery. An art auction that led to the $44,000 in sales where artists got 100 percent of their profits could not be confirmed, nor could the leveraging of funds.

It was mentioned that with $200,000 of SINED funding, the Nunavut Arts and Crafts Association was able to leverage an additional $600,000 from other sources, including the Government of Nunavut, the Department of Canadian Heritage and Inuit organizations. However, leveraged funds were not documented in available reports. Essentially, year-end reports focused on activities and not on the impact of those activities.

On a positive note, under the period covered by the evaluation, SINED funds have contributed to upgrades to the Nunavut Arts and Crafts Association's website, which has resulted in an increase in new users. For example, in 2011-12, the website received 2,657 visits of which 74 percent were new visits. Most visits were from Canada (2,058), followed by the United States of America (160), France (136), Germany (90), United Kingdom (74), Bangladesh (13), and India (11)Footnote 64. The evaluation found that for a territory that is so undeveloped, these numbers are significant and encouraging.

h) Impact of SINED investments in capacity

If a broad definition of capacity, as outlined in Section 2.3.2, is consistently applied to the entire list of SINED projects, the total SINED investment in capacity was calculated at 79 percent. In contrast, CanNor classified 30 percent of projects under the "capacity development" theme.

Table 4: Total SINED Capacity Investments by Sector and Funding Stream 2007-08 to 2011-12
Targeted Investment Program Innovation and Knowledge Partnership Advisory Forums Pan-Territorial Fund TOTAL % of total sector investments in capacity
Economic infrastructure
Broadband $1,000,000 $0 $0 $0 $1,000,000 69%
Information communication technologies $1,170,153 $148,720 $25,000 $0 $1,343,873 69%
Energy $829,365 $65,000 $53,915 $0 $948,280 54%
Alternative energy/ Renewable energy $2,001,547 $206,600 $0 $0 $2,208,147 79%
Other economic infrastructure $1,301,918 312,318 $25,000 $0 $1,639,236 82%
Other sector $7,260,595 $1,367,260 $40,000 $2,079,388 $10,747,243 100%
Commercial/Traditional harvesting $54,000 $101,103 $0 $0 $155,103 30%
Cultural industries $2,417,890 $318,030 $0 $0 $2,735,920 58%
Fisheries $1,891,031 $109,315 $0 $0 $2,000,346 100%
Forestry $495,610 $126,974 $0 $0 $622,584 100%
Geoscience $23,632,000 $0 $0 $0 $23,632,000 100%
Mining $4,555,450 $493,244 $5,000 $0 $5,053,694 88%
Small and medium-sized enterprise (loan top-up) $7,676,006 $342,944 $0 $0 $8,018,950 100%
Tourism $4,936,714 $446,854 $93,000 $0 $5,476,568 34%
Trade $479,145 $631,145 $171,467 $0 $1,281,757 48%
Total Capacity $59,701,424 $4,669,507 $413,382 $2,079,388 $66,863,701
Total SINED funding $74,211,551 $6,311,097 $695,126 $3,677,888 $84,895,662
% of total capacity 80.45% 73.99% 59.47% 56.54% 78.76%

As shown above, SINED capacity investments totalled $66.9 million from 2007-08 to 2011-12. Eighty percent of Targeted Investment Program, seventy-four percent of Innovation and Knowledge, fifty-nine percent of Partnership Advisory Forums and fifty-seven percent of Pan-Territorial Fund investments have been directed towards capacity building. One hundred percent of SINED investments in fisheries, forestry, geoscience and small and medium-sized enterprises have been for capacity building. The breakdown of the types of capacity investments are as follows:

Table 5: Types of Capacity Investments by Funding Stream 2007-08 to 2011-12
Targeted Investment Program Innovation and Knowledge Partnership Advisory Forums Pan-Territorial Fund TOTAL
# of projects value ($) # of projects value ($) # of projects value ($) # of projects value ($) # of projects value ($)
research projects 80 31,750,379 13 502,746 0 0 1 1,550,000 94 $33,803,125
training/skills development 38 7,638,958 22 1,233,922 1 39,000 1 200,000 62 $9,111,880
loan top up 8 6,351,222 1 23,178 0 0 0 0 9 $6,374,400
activity unclear/other capacity 43 4,310,046 16 824,811 1 5,000 1 200,000 61 $5,339,857
strategic planning 40 4,411,107 13 395,232 0 0 0 0 53 $4,806,339
feasibility studies 24 2,448,622 5 203,085 0 0 0 0 29 $2,651,707
conferences/workshops/summits/forums/symposiums 11 773,454 28 1,141,374 6 168,915 0 0 45 $2,083,743
organizational development
consultation/expert knowledge 11 1,281,837 4 247,400 5 175,467 1 129,388 21 $1,834,092
e-business products/solutions 7 806,799 3 97,759 1 25,000 0 0 11 $1,013,610
TOTAL 262 $59,772,424 105 $4,669,507 14 $413,382 4 $2,079,388 385 $67,018,753

As shown in Table 5, overall, the most frequent type of capacity investment was research projects. In fact, fifty percent of SINED capacity expenditures went towards research projects largely in the area of geoscience. Training/skills development and loan top-ups were the second and third most frequently supported capacity projects overall.

Also interesting to note is the difference between the type of capacity projects supported by the separate funding streams. Targeted Investment Program and Pan-Territorial Fund capacity expenditures went towards research projects, while Innovation and Knowledge capacity projects were mostly in training/skills development and conferences/workshops/etc. Most of Partnership Advisory Forums investments in capacity went towards consultation/expert knowledge and conferences/workshops/etc.

Given that a large proportion of projects covered by the sector case studies were capacity projects, some of the impacts of capacity projects are discussed in sections a) to g) above.

5. Evaluation Findings – Performance - Efficiency and Economy

Issues of efficiency and economy were explored in key informant interviews with SINED staff and recipients, the document and file review and the literature review. Interviewees were asked about the strengths and weaknesses of SINED, about their experiences with the development of the Territorial Investment Plans and the processes in place to obtain project funding. Responses were supplemented with the document and file review.

Some inefficiencies in delivery were identified. While key informants identified SINED staff as a clear strength of SINED, it is clear that there have been some difficulties attracting and retaining staff. The limited or reduced sub delegation of ministerial authority for project approvals has caused significant problems both internally within CanNor and externally with recipients. There is evidence that the transition to the new agency has resulted in some disconnect between management and operations. Finally, the similarities between SINED programs calls into question the need for four separate programs.

Finding: SINED staff were identified by recipients as a strength of SINED.

One third of recipients identified SINED staff (without prompting) as a strength. They described regional staff as passionate, engaged, well networked, supportive and knowledgeable of issues and communities. EPMRB's research also determined that there is a good level of trust and cooperation between CanNor and recipients (i.e., territorial governments). For example, Government of Nunavut staff interviewed were of the view that the project proposal assessments were fair and constructive, with SINED staff providing assistance in preparatory as well as subsequent stages. In addition to providing assistance, SINED staff also added a good challenge function.

Finding: SINED has experienced difficulties attracting and retaining staff, which has resulted in increased workloads, has impacted planning, performance measurement, monitoring and reporting, and has reduced support to project proponents

Twenty-nine percent of SINED staff highlighted the difficulty of attracting and retaining employees. Delivery of the SINED program requires people with a background in economic development who are willing to live in the North. As mentioned in the Program Profile in Section 1.2, CanNor has been facing the challenge of filling vacancies since the beginning. The consequences of staff shortages identified by SINED staff were as follows: a lack of resources for proper planning, monitoring, reporting and performance measurement; increased workloads; and, reduced support for project proponents.

While a number of key informants spoke of heavy workloads, EPMRB was not able to verify workload levels with available data. To provide some perspective on workloads, the average case load for SINED projects per year was calculated but deemed inconclusive due to inconsistencies in how workloads are assigned in the territoriesFootnote 65. The evaluation did not examine the ideal case load, but based on the key informant interviews, a case load of 20 files was too heavy.

Finding: The engagement process for Territorial Investment Plans is collaborative in nature but many recipients in the Northwest Territories felt the engagement process for the second round of SINED was not as well orchestrated as the first round.

Forty-three percent of SINED staff identified the engagement process (for identifying investment priorities and establishing Territorial Investment Plans) as a SINED strength. They argued that the engagement process increased participant understanding of issues facing sectors, allowed Northerners to identify investment priorities, provided an opportunity to network and combine/complement efforts and helped avoid duplication of projects and programs. While recipients interviewed tended to agree with the importance of the engagement process, they did identify some areas for improvement.

Fifty-four percent of recipients in the Northwest Territories interviewed felt the SINED engagement process in 2009 was inferior to that of 2005. While the level of involvement was strong, they expressed disappointment with the lack of follow-up. As a result, some representatives from the territorial government felt that the Plan for round two of SINED did not reflect their priorities.

Finding: The current project approval process is considered by proponents as being slow. Reduced sub-delegation of ministerial authority has increased delays, increased workloads and reduced transparency in decision making.

As mentioned in Section 1.2.6 of the Program Profile, after limited or reduced sub-delegation of ministerial authorityFootnote 66 was introduced in November 2010, only the Minister could approve SINED expenditures. While the reason for removing the delegation of authority for SINED projects was never made clear, the impact of this change was significant.

Fifty percent of SINED staff interviewed indicated that since the introduction of limited or reduced sub delegation of ministerial authority for project approvals, there have been increased delays and more paperwork for Economic Development Officers. To illustrate, ministerial approvals require the preparation of formal briefing notes for decision. In addition, when questions arise in the Minister's Office about a recommendation for approval, briefing notes must be prepared in response, which engages the limited resources.

The length of delays could not be confirmed due to a lack of data on the time it took projects to proceed through the approvals process. Additional analysis is required to determine the length and source of delays and to explore possible solutions.

SINED staff interviewed also indicated that the process for ministerial approvals has raised concerns about a lack of transparency. As explained in the Program Profile in Section 1.2, projects that are recommended for approval meet both eligibility requirements and funding priorities, and are also based on an analysis of the project proponent's ability to deliver. However, when the Minister does not approve projects and no explanation is provided, SINED staff was left guessing as to the reasons why; they were unsure of how to assess for recommendation similar projects in the future; and, they could not explain to project proponents the reasons why their project was not approved.

Some SINED staff and recipients were of the opinion that projects were not being approved in cases where they were not related to resource extraction. They both expressed frustration and shared their perspective that project approvals did not appear to be based solely on the analysis of the project's merit.

Finding: Delays in approvals and funding, disrupted the administration of recipient organizations, caused financial strain for recipients, increased project costs and interfered with the timely delivery of a quality product.

Seventy-four percent of recipients interviewed identified delays in project approvals and funding as an issue with negative consequences. Citing examples, many recipients spoke of submitting proposals in the spring and receiving project approvals or funding in the fall or winter with instructions to spend the money by March 31. This resulted in an increased administrative burden to recipients as they were forced to revise their project planning and reassign resources to meet new timelines. For example, in the case of geoscience, summer field work involves the coordination of researchers, equipment, transportation, camp staff and more. Delays in funding resulted in huge logistical problems. Shortened timelines also reduced the quality of deliverables because scope of projects often had to be reduced along with participation levels.

Delays also equalled increased project costs. EPMRB learned from some proponents that they were forced to engage consultants in order to meet revised project timelines, which increased costs. In one situation, a project involving construction, did not receive funding until after winter arrived. As a result, it took an exorbitant amount of time to break through snow/ice to get the project underway, which resulted in inefficiencies and increased costs.

If they were able, project proponents used their own money to cover costs (by borrowing or reallocating funds from other projects or programs) until SINED funding came in. Resorting to this approach allowed them to move their projects forward as quickly as possible, but it also caused an administration and financial burden as the organizations were using their own financial resources to cover costs, reallocate funds and resources, revise budgets or access loans. One project proponent reported nearly going bankrupt on two occasions, while another reported paying $12,000 in interest in order to access a line of credit. Several non-government organization representatives said they had to forego several months' salary to start a project before SINED funding had arrived. Organizations, especially non-government organizations, that do not have the financial means to cash-manage a project (or support it using their own resources), experience further delays, lose momentum and also risk not being able to complete a quality project on time. In some cases, proponents, because of the time the funds came in, could not spend all the money allocated and had to cancel or postpone a project because of unreasonable timelines.

Once again, the time taken to process contribution agreements could not be calculated due to a lack of data. Additional analysis is required to determine the length and source of delays and to explore possible solutions.

Finding: There are many similarities between Targeted Investment Program, Innovation and Knowledge, Partnership Advisory Forums and Pan-Territorial Fund, calling into question the need for four component programs.

The document review revealed many similarities between Targeted Investment Program, Innovation and Knowledge and Partnership Advisory Forums, including similar goals for the four programs, similar eligible recipients and identical eligible applicants. In addition, as EPMRB noted, the four programs were funding similar projects. For example, what were clearly innovation projects were funded under Targeted Investment Program, when Innovation and Knowledge is meant to support these types of projects. Conferences and workshops were funded under Targeted Investment Program, Innovation and Knowledge and Partnership Advisory Forums. All programs fund the same sectors identified in Territorial Investment Plans, although Innovation and Knowledge and Pan-Territorial Fund tend to fund more projects outside these sectors. There was also evidence that funds were being transferred between programs as evidenced by the difference between SINED planned/forecast spending (Table 1) and SINED actual spending (Table 2).

There were some differences identified between programs. Targeted Investment Program investments were driven by the Territorial Investment Plans, but the other programs generally supported the same sectors although Innovation and Knowledge and Pan-Territorial Fund did fund more projects outside Targeted Investment Program sectors as shown in Table 3. The limits on the amount of funding under each program varied and there were slight differences in eligible projects, however, if a project was not permitted by one program it was likely eligible for funding under another. These differences are so slight that separate programs do not seem warranted as they cause confusion among applicants. The transfer of funds between programs involves extra work for an already busy staff.

Finding:The transition to the new CanNor Agency has resulted in some disconnect between management and operations.

As described in Section 1.2.6 of the Program Profile, the transition of SINED from AANDC to CanNor has been challenged by increasing demands to deliver additional programming, more profile, increased paperwork, staff shortages, cutbacks and high staff turnover. EPMRB notes that the transition represented a significant change requiring the creation of a new northern development agency in the midst of heightened federal government emphasis on cost savings and budgetary restraints. The transition of employees into a new agency, in the middle of the fiscal year, while continuing to execute the wider day-to-day responsibilities (e.g. reviewing proposals, developing funding agreements, monitoring and reporting, etc) was a challenge. However, best efforts were made to maintain the best possible service to federal clients, both internal and external.

Fifty percent of SINED staff interviewed spoke of a "disconnect" between CanNor management and operational staff, which manifested after the transition. SINED staff interviewed observed that CanNor managers were often not familiar with economic development or northern issues. The managers were transient, often temporary, and many were not respectful of processes that were already in place, such as the engagement process, or investment plans; neither did they recognize the importance of local relationships and local support. For example, in one territory, new management ordered a mass call for proposals, which is contrary to the known practice; a mass call raises expectations among potential recipients and also results in a lot of additional work for them and SINED staff alike.

SINED staff also mentioned that the new CanNor managers, though they turned out to be temporary, seemed to favour and push resource-sector type projects (i.e., mining and energy) and did not acknowledge the importance of diversification to protect the economy from the boom-bust cycle of resource development. Recognizing that diversification is one of the key tenets of SINED, this created conflict between SINED Economic Development Officers and the CanNor managers with respect to the importance of other sectors such as fisheries, cultural industries and tourism, all of which have huge potential and are more closely aligned with local culture.

The key informants were selected on the basis of their knowledge and expertise relative to the range of activities associated with the design, development, implementation, and monitoring of SINED. Nonetheless, aside from the information obtained from the key informant interviews, EPMRB was unable to identify additional evidence that supports the "disconnect" identified above. However, given the pan-northern mandate of CanNor and the negative impact that the transition had in other areas of program delivery, this finding was deemed important for inclusion in this final report. It is expected that, drawing attention to this issue will serve as a caution or lesson for the possible delivery of any future similar undertakings.

6. Evaluation Findings – Other Issues

Finding:The literature review concluded that SINED is consistent with international best practices for short-term policies to improve economic development in northern and remote regions. However, without significant, coordinated, long-term investments in the North, SINED is unlikely to result in a sustained change in economic conditions.

The literature review determined that the most successful short- and medium-term approaches for promoting economic development in northern and remote regions have an industry focus, explore opportunities, are north-centred, incorporate regional input, build capacity and support specific projects. While SINED is aligned with these best practices, it is clear that the challenges in Canada's north are too great for a small program to meet over a short to medium-term timeframe.

Countries with sustained success such as Sweden, Norway and Finland demonstrate that long-term commitments to northern regions are preferred to short-term policies and investments. For several generations, Canadians have accepted, with little debate, the assertion that the expense of developing the North is not justified by the small regional population and high costs of operating. In contrast, Scandinavian countries (noting differences between them and Canada in terms of population and land sizes, etc.) were determined to ensure that northern and remote communities enjoy the same benefit as other citizens. Regional economic development strategies for the Canadian north, fall short of Scandinavian commitments in several key areas, for instance, the absence of regional research, high-end training capacity in the region, transformational and sustained investments in infrastructure, etc. Regional economic development strategies for the Canadian North, therefore, lag behind international standards for northern economic development in terms of long-term approaches.

Canada's north needs sustained long-term investment to build the foundations for economic development because, in order for a program like SINED to achieve the greatest results, the preconditions of success must be in place. For example, initiatives designed to support the "new economy" have difficulty gaining traction in the North in large measure because of difficulties with technological infrastructure and the receptor populationsFootnote 67. The research also shows that communities in long-term economic distress are poor candidates for place-basedFootnote 68 support. As a result, the Government of Canada needs to work with local authorities to build the foundations of success such as leadership and institutions, a skilled workforce, infrastructure, entrepreneurial mindset and healthy, vibrant communities. With these foundations in place, SINED funding will have the greatest impact.

The literature review found that human capital is critical to economic growthFootnote 69 but requires long-term investment not just in skills development and adult basic education, but also in elementary and high-schoolFootnote 70. A northern university was also suggested in the literature review. There is ample evidence that institutions play a vital role in shaping their host communities and providing region-wide capacity building and other benefits. Significant investment in research, post-secondary education and infrastructure are required to promote the development of new-economy jobs and ensure local businesses can compete in the global high-tech workplaceFootnote 71 The North is the only region in Canada without a university and this is a major shortcoming in terms of long-term regional development. There is, of course, a valuable network of regional colleges; however, with the exception of Yukon College, they focus largely on adult basic education and skills development and do not have the same long-term impact as universities.

Improvements to quality of life in northern centres are also critical for success as they are pivotal in encouraging highly qualified personnel to come to a region and stay. Local amenities can be transformational in terms of loyalty to a region and urban residents most often cite attractive lifestyle as a reason to stayFootnote 72. Restaurants, coffee shops, health facilities, schools, recreation facilities, parks and local attractions are thus important to economic development.

Foundational investments in infrastructure that benefit multiple sectors are also needed. The literature review showed that infrastructure investments not focused on a specific venture are more successfulFootnote 73. Hastily conceived and poorly executed projects can distort regional economies and hamper economic development for generations.

Considering the above, the literature review suggests the Government of Canada should have more significant, coordinated, long-term, foundational investments in education and infrastructure and that SINED should focus on the following:

  1. Areas with strong fundamentals - such as projects and sectors with existing strength and strong foundations and communities where the pre-conditions for success are present
  2. Capacity building – the importance of leadership and entrepreneurship to economic development was emphasized by many researchers who argued that it should be a priority given the significant returns on investmentFootnote 74
  3. Aboriginal groups – the literature review found that projects with indigenous populations yield more success because they are committed to the region and less transient than the non-indigenous populationFootnote 75. Projects supported by Aboriginal groups have a significant advantage if the other preconditions for success are present.
  4. Evidence-based decision making – investment decisions based on economic analysis and modeling. The current engagement process for developing Territorial Investment Plans, while important for collecting input, establishing trust and gaining buy-in, should not replace careful and considered economic planning and policy making.
  5. Alternative measures of success – recognizing that for Aboriginal populations, who have other sources of sustenance, including country food, and cultural reasons for pursuing non-income activities, traditional measures of regional economic success such as employment and income do not apply in the NorthFootnote 76. As a result, alternate economic models may be required to accommodate the lifestyle choices made by significant numbers of northern residentsFootnote 77.

7. Conclusions

SINED is consistent with federal government priorities and roles and responsibilities and responds to a continued need for foundational investments supporting economic development in the North. SINED funding complements what is available through territorial governments and other federal departments and should continue given the limed resources and difficulties attracting private-sector investment.

While some quantifiable measures of success were documented for particular sectors, without a process to properly monitor the outcomes of SINED investments, it is not possible to assess the impacts of these investments. SINED needs a performance measurement strategy, which identifies: outputs; outcomes and indicators; and, outline a process for collecting performance information. This performance information can be used for ongoing management, reporting on results and the early identification of problems and/or inefficiencies.

The evaluation identified inefficiencies in the project and funding approvals, program design and tracking tools. In addition to negatively impacting internal operations, inefficiencies in these areas have also negatively impacted recipient organizations and the cost, quality and timeliness of projects. SINED should apply lessons learned and best practices from regions and from Regional Development Agencies to address issues impacting efficiency and economy.

The disruption caused by the transition of SINED to CanNor, the additional workload and the disconnection between SINED management and operations must be recognized and addressed immediately. A staffing strategy is needed to identify responsibilities and appropriate resource levels for the effective delivery of SINED and to attract and retain staff. The strategy should target individuals with the right skills (e.g., new business graduates), address barriers to moving north (e.g., housing) and provide incentives for moving north (e.g., training).

Finally, there is a great need for foundational investments in the North to support economic development. SINED provides essential support for capacity building in existing sectors and economic diversification, and has adapted well to northern economic development strategies. However, the need in the North is greater than one economic development program. In order to achieve real change, SINED must be part of a long-term, sustained, coordinated approach to northern development.

8. Recommendations

It is recommended that CanNor:

  1. Introduce processes, systems and resources to properly monitor and manage performance (effectiveness, efficiency and economy) and report on the impact of SINED investments.
  2. Review the approach (program design) to achieving outcomes based on lessons learned and best practices. Consider collapsing the four programs into one or better distinguishing between them.
  3. Introduce processes/procedures to improve approval time and delivery of funding.
  4. Update policies and procedures manual.
  5. Develop a Human Resources Plan / Recruitment Strategy to attract and retain staff and ensure CanNor has needed resources (with relevant experience) to meet program delivery demands.

Appendix A – Evaluation Matrix
Relevance

Evaluation Matrix
Questions Indicators Methods/Sources
Relevance
Continued Need
1) Is there a continued need for SINED?
  • Stakeholder support for SINED
  • Key informant interviews with recipients
  • Availability of other sources of funding for economic development
  • Literature review
  • Key informant interviews with recipients
  • Sector performance - Threats and opportunities facing sectors
  • Sector case studies
2) Are the conditions that led to the creation of SINED still present?
  • Evolution of employment rates in public vs. private sector between 2005 and 2012
  • Literature review (Territorial Outlooks, Northern Economic Index)
  • Evolution in economic outlook for 2005, 2009 and 2012
  • Evolution of socio-economic conditions (income levels, employment)
Alignment with federal government priorities and departmental strategic outcomes
3) To what extent is SINED consistent with: federal government priorities; CanNor strategic outcomes; and the outcomes of regional economic development agencies?
  • Federal announcements and investments in the North
  • Literature review (Canada’s Economic Action Plans, Northern Development Strategy Economic Chapter, Speeches from the Throne, Budget commitments)
Alignment with federal government roles and responsibilities
4) What is the role of the federal government in advancing economic development? Is SINED aligned with federal roles and responsibilities?
  • Federal jurisdiction in the North
  • Literature review (Analysis of government roles and responsibilities related to the North)
5) What are the roles and responsibilities of territorial governments in economic development? Is there any overlap or duplication with SINED?
  • Territorial government’s role in economic development
  • Literature review
  • Analysis of government roles and responsibilities related to economic development
  • Overlap or duplication of responsibilities between SINED and territorial government
  • Literature review
  • Key informant interviews with recipients and SINED staff
6) What are other government departments doing in the area of northern economic development? Is there any overlap or duplication with SINED?
  • Overlap of responsibilities between SINED and other government departments
  • Literature review
  • Key informant interviews with recipients and SINED staff
Performance (Effectiveness, Efficiency and Economy)
Effectiveness
7) Has SINED been achieving it's intended outcomes?
a) Expanded publicly accessible information (knowledge base)
  • Incremental geographic area assessed/sampled for resource potential
  • Case study (geoscience)
  • Amount of new exploration in areas that were scanned and mapped
  • Case study (geoscience)
  • Literature review (The Future of Mining in Canada’s north – Centre for the North, January 2013)
  • Increase in area covered by resource stock assessments
  • Increase in number of data holdings more readily available to the public
  • Use of data holdings
  • Increase in number of sectors with a strategic plan
  • Case study (fisheries)
b) Expanded multi-user infrastructure
  • Increase in area serviced by multi-user infrastructure
  • Case study (broadband, energy, and alternative energy)
c) Increased access to capital
  • Growth in access to capital by business
  • Case study (small and medium-sized enterprise)
d) Increased ability of Northerners to respond to economic development opportunities (capacity development)
  • Impact of capacity investments
  • Summary of SINED investments in capacity, i.e., types of investments and expected results
  • Capacity case study
e) Increased partnerships with northern governments and organizations
  • Impact of partnerships with northern governments and organizations
  • Partnership Advisory Forums and PTF case studies
  • Case studies (all)
  • Opinions of engagement process for Territorial Investment Plans
  • Key informant interviews with recipients and SINED staff
f) Increased private sector investment and expanded business and employment opportunities (leveraged funding)
  • Leveraged funding
  • Case studies (all)
  • Job creation
  • Case studies (all)
g) Increased sustainability of principle economic drivers
  • Growth in resource based industries (mining, oil and gas, tourism)
  • Case studies (geoscience, mining, fisheries, and energy)
  • Stability of resource based industries (avoiding boom bust cycle)
  • Case studies (geoscience, mining, fisheries, and energy)
h) Increased economic development and diversification
  • Growth in employment
  • Growth in income levels
  • Increased entrepreneurship
  • New business development and growth
  • Literature review
  • Case studies (all)
  • Increased investment
  • Increased diversification
  • Literature review
  • Case studies (all)
i) Strong, stable, territorial economies for the benefit of Northerners and all Canadians
  • Growth in employment
  • Growth in income levels
  • Increased entrepreneurship
  • New business development
  • Literature review (Territorial Outlooks, Northern Economic Index)
  • Increased investment
  • Increased diversification
  • increased own-source revenue
  • Literature review (Territorial Outlooks, Northern Economic Index)
8) Has the design of SINED contributed to the achievement of the intended outcomes?
  • Perceptions on SINED design and delivery
  • Key informant interviews with recipients and SINED staff
  • Degree to which SINED design is aligned with best practices identified in the literature review
  • Literature review
  • Document review
9) Have there been unintended outcomes (positive or negative)?
  • Stakeholders identify unintended outcomes
  • Key informant interviews with recipients
10) Is a system in place to identify, collect and report on: (a) risk; and (b) performance measurement data?
  • System in place to identify, collect and report on risk and performance measurement
  • Document review
  • Key informant interviews with SINED staff
11) Is the performance measurement data collected being used in decision making?
  • Extent to which data is used: for day to day management of SINED programs, planning; to make adjustments to incorporate best practices; to report on results
  • Key informant interviews with SINED staff
  • Document review (program data, planning documents, records of program modifications, reporting)
  • SINED reports contain performance data
  • Document review
Effectiveness
12) Has SINED optimized their processes over time? In other words, has SINED been able to deliver a greater amount and quality of outputs with the same inputs over time? Or the same amount of outputs with less inputs?
  • Number of proposals processed versus the number of employees
  • Amount of time required to assess/process proposals over time – quality of proposals
  • Amount of time required to obtain approvals over time
  • Document review (staffing, process times recorded in project files)
  • Key informant interviews with SINED staff
13) Were there any external factors that hindered program delivery or lead to programming success?
  • Identification of significant unplanned factors that had an impact on inputs and outputs, and the achievement of outcomes
  • Key informant interviews with recipients and SINED staff
  • Document review
14) How can greater efficiencies be gained?
  • Suggestions on how to improve SINED design and delivery
  • Identification and degree of implementation of best practices
  • Key informant interviews with recipients and SINED staff
  • Best practices identified in other jurisdictions that could result in efficiencies in SINED
  • Literature review
  • Level of implementation of recommendations from previous evaluation
  • Document review
Alternatives
15) Are there alternatives to the design and delivery of SINED which could achieve the same results more efficiently and/or economically?
  • Identification of alternative approaches for program delivery
  • Identification of opportunities to reduce overall operating costs (i.e., sharing of services with AANDC)
  • Key informant interviews with recipients and SINED staff
  • Literature review
  • Alignment of SINED design and delivery with emerging policies/strategies for promoting economic development in circumpolar or remote regions
  • Literature review
Date modified: