- New Zealand
- New Zealand Qualifications Authority (NZQA)
Diploma of higher education
1. Origin, Institution name
2. Institution website
Graduation diploma from a non-university tertiary education organisation.
4. Good practice - Zewnętrzne zapewnianie jakości
The good practice refers to the quality assurance process in non-university tertiary education organisations in New Zealand. It encompasses all educational institutions operating in this area (higher vocational education schools, private training establishments, polytechnics and institutes of technology).
The system has been functioning in its current form since 2009 and is the responsibility of a government agency – the New Zealand Qualifications Authority (NZQA). The above-listed educational institutions are legally required to be in this system, use self-assessment procedures and have cyclical external evaluations performed in accordance with the Education Act of 29 September 1989 . Each institution of this type and its offered curricula must be registered and approved by NZQA.
In the context of searching for good practices, a key aspect of the system is the focus on cooperation between the institutions and NZQA, as well as engaging the institutions in the evaluation process and developing standardised guidelines.
4.2. New Zealand Qualifications Authority (NZQA)
The NZQA is the agency playing the key role in the entire system. The main purpose of its operation is to support New Zealand learners and social development by ensuring the credibility of attained qualifications and improving them, as well as by increasing their recognition at the national and international levels.
- is involved in quality assurance in the sector of non-university tertiary education,
- monitors the system of internal evaluation in secondary education,
- manages the certification system determining the competence and qualifications of students of public secondary education institutions,
- determines educational standards in all areas to which other governmental institutions have not been assigned,
- acts as an expert authority responsible for defining and describing qualifications.
The Quality Assurance Division is responsible for the operation of the NZQA. It has 6 teams responsible for the following areas:
- quality assurance strategy,
- risk management,
- New Zealand Qualifications Framework (which also functions as the qualifications register),
- education programmes for the Maori community.
4.3. The Premises of the System
The quality assurance system in education and the external evaluation of New Zealand’s non-university tertiary education organisations relies on four premises, which set forth the framework of activities undertaken by NZQA:
- High level of trust and responsibility. The foundation for the functioning of the system is cooperation and communication, as well as building an atmosphere of trust and joint responsibility for the educational performance of the entire tertiary education sector. This means that the NZQA respects the autonomy of the evaluated institutions, their control over internal processes and thus limits its interventions to the necessary minimum. At the same time, the agency believes that every institution is responsible for ensuring a high quality of teaching, the outcomes achieved by students and course participants, and the ongoing improvement of services. The condition for granting extensive autonomy is confidence with respect to the high level of competence of individual institutions, verified through external evaluations.
- Dynamic understanding of quality. The definition of quality depends on the specific situations and contexts, and in tertiary education, the key element is the knowledge acquired by learners, the utility of the awarded qualifications and the degree to which education translates to long-term positive results (for learners and all of society). The dynamic understanding of quality is consistent with international trends, where we see a departure from quality control (compliance of educational activities with designated standards) in favour of ongoing improvement and striving for excellence.
- Focus on results. The classic approach to quality assurance focuses on institutional potential, actions undertaken by the institutions and systemic issues, with the assumption that the fulfilment of designated standards is tantamount to a high quality of teaching. The evaluation approach to quality assurance emphasises educational outcomes and key processes contributing to their accomplishment. The study covers the teaching process and activities performed by the institution, yet the point of reference is always the final results. The main criteria for the functioning of an institution are efficiency and utility of the internal system to accomplish the stipulated educational outcomes. Educational achievements are affected by the competence of students and course participants, the qualifications of the personnel and the institution’s budget.
- Flexibility of evaluation. Evaluation must be flexible and adaptable to diverse conditions. This makes it possible to include a broad spectrum of educational institutions in the evaluation of their activities. The evaluation process is adjusted to the specific nature of individual institutions, the objectives set before them, with the simultaneous observance of its methodological correctness and the ability to compare results. This is particularly important in the context of institutions supporting the Maori community, which is guided by different values and criteria of success. The NZQA has developed a separate system of evaluation and monitoring for this community, which shows that the system is adjusted to the specific needs of individuals.
As may be seen, the evaluated institutions are involved in the process of evaluating their operations and accreditation. Their role is not exclusively limited to the provision of documents and data. They are required to perform systematic self-assessments, the results of which - if they fulfil the adopted criteria and standards - are recognized by the NZQA. The way in which the self-assessment is performed is reflected in the evaluation of the reliability of a given institution - the higher the competence in internal quality assurance, the greater the extent to which its results will be taken into account in the evaluation of the educational quality at a given institution. At the same time, this is conducive to building trust between the institutions and the NZQA. Effective and verified institutions are also encouraged to share their conclusions and developed solutions so that others may also use them. NZQA focuses then on improving educational quality in institutions struggling with problems or in new institutions in the system.
An external evaluation of an institution is performed with the use of standardised tertiary evaluation indicators and a grading scale that allows the educational quality of various institutions to be compared and the changes occurring to be monitored over the long-term. Certified institutions learn self-assessment that is focused on constant improvement. It is important to note that this takes place in an atmosphere where finding and solving problems does not challenge the credibility of an institution in the eyes of NZQA.
4.4. Evaluation System in Non-University Tertiary Education Organisations
NZQA requires every institution to undergo an evaluation in 4-year cycles. The Agency develops and publishes a generally available list of institutions being evaluated in a given year. The evaluation may be initiated at the request of a tertiary education organisation (e.g. in case of changes in the management) or at the request of another governmental agency (e.g. when complaints have been received pertaining to the operation of a given institution).
The process of self-assessment of an educational institution is the starting point for an external evaluation. Representatives of the evaluated educational institutions are included in every stage of the evaluation and quality assurance process via informing, consulting and cooperation.
The functioning of the New Zealand quality assurance system in non-university tertiary education can be divided into four main stages:
1. entry stage: recognition and certification of non-university tertiary education organisations, description and identification of qualifications awarded in a given institution and accepting the presented curricula. At this stage, the institution also provides its consent for NZQA oversight and participation in the evaluation process.
2. quality assurance stage: monitoring educational performance (current educational results, internal consistency of the curricula) at various levels defined in the New Zealand Qualifications Framework, conducted by external experts.
3. external evaluation stage: the overall evaluation of an institution (educational outcomes, educational process) and also detection of potential problems and areas requiring improvement. Based on the evaluation, a generally available report is created, and the institution is included in one of four categories of “confidence” (based on their educational achievements and capability in self-assessment).
4. risk estimation and risk management stage: the collected data (derived from the external evaluation, site visits, delivered documents and potential complaints) are the basis for determining the level of risk to an institution of failing to uphold teaching standards, protect consumers’ rights and comply with New Zealand regulations.
The purpose of external evaluation is not to obtain results that are fully representative and statistically significant. In principle, this would be practically impossible due to the mostly qualitative nature of the evaluation and its methods. Therefore, the final evaluation of an institution (the aspect of capability in self-assessment and educational achievements) is formulated as the confidence level that the evaluation team has with respect to the institution’s fulfilment of specific criteria. The use of uniform tools is aimed at guaranteeing the highest possible reliability of the final results and at avoiding situations in which one aspect of the institution’s functioning or the opinion of a given evaluator may distort the overall evaluation of the institution.
The process is a compromise between the flexibility of the evaluation scheme, its adjustment to a given institution’s specific situation and the standardisation of evaluation results, enabling individual institutions to be compared. Therefore eventually, a number of qualitative studies lead to the preparation of an extensive quantitative database, containing the most important information about all non-university tertiary education institutions in New Zealand.
Educational institutions are obliged to perform self-assessments focused on:
- identifying the needs of pupils, students, course participants and stakeholders,
- evaluating organisational processes and practice,
- using the acquired knowledge for actual improvement of the outcomes and achievements of pupils, students and course participants.
The solutions applied by individual institutions are not regulated, but they have to fulfil the general premises. They should be transparent, credible and consistent, whereas their scope should refer to the diagnosis of stakeholders’ needs, the evaluation of the effectiveness of institutional processes (applied practice, solutions, compliance with regulations pertaining to teaching and administrative aspects), as well as educational achievements (consistent with the premises of the curriculum) and their impact on learners (chances in the labour market, utility of awarded qualifications, etc.). The data generated by the self-assessment are used to identify areas where improvements may be needed and to formulate precise recommendations that are to be implemented in practice. The NZQA does not determine and does not recommend any specific methods or self-assessment schemes. It only requires the provision of answers to six key evaluation questions listed below, which pertain to two aspects: educational achievements and the processes supporting them.
Questions pertaining to educational achievements:
- How well do students achieve? The question refers to the immediate educational outcomes (receipt of new qualifications, acquisition of knowledge and skills) and their long-term impact on learners (increased competitiveness in the labour market, commencement of further education, improvement of social or economic status, etc.).
- What is the value of the outcomes for key stakeholders, including students? This question reflects a broader outlook on the educational outcomes and includes the potential employers or the local community in the evaluation. In this context, the utility of the “educational product” offered by the educational institutions to stakeholders is evaluated and the manner in which the attained qualifications influence the functioning of the entire community.
Questions pertaining to the educational process:
- How well do programme design and delivery, including learning and assessment activities, match the needs of students and other relevant stakeholders? This question should allow the institution to determine the degree to which it systematically diagnoses and analyses stakeholder needs and the degree to which such a diagnosis is accurate and is significantly taken into account in the process of designing and implementing curricula.
- How effectively are students supported and involved in their learning? This question refers to the methods used in a given curriculum (application of active methods, e-learning, independent projects entrusted to learners, additional activities offered at the campus, etc.). It takes into account the interactive nature of classes, the proportion between work in class and independent work, innovation and the comprehensiveness of applied teaching methods.
- How effective are governance and management in supporting educational achievement? This question is used to check the degree to which the support and communication offered by the institution allows the needs of learners to be determined and met, as well as sustaining their involvement.
- How effectively are important compliance accountabilities managed? This question emphasises the evaluation of institution’s management of complying with legal and procedural requirements.
4.6. External evaluation
The evaluation process performed by the NZQA relies on key evaluation questions, tertiary evaluation indicators and scales. All evaluation studies conducted by the NZQA have a common basis. The key evaluation questions (which also form the framework for the self-assessment) constitute only the framework of the evaluation and are specified more precisely during each individual evaluation. The tertiary evaluation indicators determine the scope of the analysis more precisely, whereas standardised scales allow the degree of the performance of every indicator to be determined.
The tertiary evaluation indicators are developed on the basis of continually updated data and the experiences of NZQA, as well as the experiences of other quality assurance organisations from abroad. Similarly to the aforementioned study questions, they are divided into two groups:
- indicators pertaining to the educational process,
- indicators pertaining to the educational outcomes.
In the group of indicators pertaining to the educational process, the following are distinguished:
- indicators on the management effectiveness of a given institution, i.e. regulations and procedures (hiring employees, self-assessment, observance of legal and ethical standards), as well as the transparency of the institution’s policies with respect to educational objectives and planned development,
- indicators on the adequacy of the educational offer with respect to the situation in the labour market, the potential needs of students and course participants, and engagement of the local community (a significant criterion in the case of curricula addressed to the Maori community),
- indicators on the quality and effectiveness of the educational process, including openness and consistency of the educational offer, developing educational opportunities for students and course participants (availability of additional courses), engagement of students and developing possibilities for the practical application of acquired knowledge, as well as a system of feedback between students/course participants and teachers.
Indicators describing the teaching outcomes are expressed in three dimensions:
- economic (employment, increase in remuneration, professional promotion),
- social (higher self-assessment, establishing relations, participation in the local community),
- cultural (development of identity and increase in involvement in the local community, which is particularly significant in the case of the Maori community).
The comprehensive system of operationalised indicators allows a certain vision to be developed of an effectively operating system of education and referring to it when analysing the results achieved by individual institutions.
The external evaluation is divided into four stages:
- determining the scope and plan of the evaluation,
- performing the evaluation,
- formulating conclusions pertaining to educational achievement and capability in self-assessment,
- issuing the evaluation report.
These stages are discussed in detail below.
4.6.1. Stage one: determining the scope and plan of the evaluation
The external evaluation is performed by a team of a minimum of two evaluators from the NZQA. If the evaluated institution is large, more evaluators are delegated (in some cases, evaluations are performed by six persons). The lead evaluator - the team manager is responsible for determining the exact scope of the evaluation and the effective management of the team and available resources. He/she should conduct the process in a manner guaranteeing the credibility and consistency of the performed evaluation. Determining the evaluation plan consists of several steps.
The first step is reading all the documents available in the NZQA pertaining to the given institution and understanding the principles and objectives of its functioning. As part of desk research, the evaluators use:
- self-assessment reports delivered by the institution,
- its profile available in the NZQA archives,
- business plans of the institution,
- annual reports and other reporting documents.
The second step is selecting the number of focus areas to evaluate. This depends on:
- the size and degree of the institution's complexity,
- number of offered curricula,
- resources at the evaluators’ disposal.
A focus area to be evaluated may be, for example, a selected curriculum, applicable procedures or the self-assessment system. The choice of focus areas depends on their internal complexity and possibility of verification. According to the NZQA guidelines, the number does not have to be high in order to ensure full representativeness for a given institution, but they should encompass a significant portion of the activities undertaken by such an institution. NZQA may designate certain focus areas as compulsory for every educational institution at a given time.
The basic factors that the evaluators should take into account when determining the focus areas include:
- the self-assessment procedures of the institution: their results and priorities,
- the type of institution and its size (number of students and course participants, employees, courses offered, etc.),
- problems (past and present) in the functioning of the institution,
- governmental guidelines relating to educational policies, recommendations of the NZQA, stakeholders’ needs.
Table 1. Number of focus areas recommended by NZQA, depending on the size of institution
|Size of the educational institution||Suggested number of focus areas|
|<20 students/course participants, 1 location, 1–2 curriculum/curricula||1-2|
|20–100 students/course participants, 1 location, 2 and more curricula||2-4|
|100–500 students/course participants, 1 – 2 locations, 6 and more curricula||3-6|
|500–1000 students/course participants, greater number of locations, several curricula||6-8|
|1000 and more students/course participants, greater number of locations, several curricula||8-10|
When examining non-university tertiary education organisations, NZQA distinguishes two approaches to evaluation: programme evaluation and institutional evaluation. The evaluators, when choosing the focus areas, should maintain balance between them. These approaches are understood as follows:
- programme evaluation: examination of a given curriculum, starting from its premises and internal logic, through the applied teaching methods, up to the final educational outcomes,
- institutional evaluation: examination of applied procedures affecting the educational achievements and satisfaction of students/course participants (effectiveness of management, amenities and teaching aids for students/course participants, methods of collecting feedback and self-assessment, etc.).
The third step is the commencement of initial consultations with representatives of the evaluated educational institution. These consultations provide the opportunity to describe and explain the internal dynamics of the institution, its objectives and the specific nature of its operations, allowing NZQA evaluators to obtain a relatively full picture of its functioning.
Representatives of institutions often include directors, management board members or persons designated by them. In line with the recommendations of NZQA, the lead evaluator should contact (via telephone or in a meeting at the headquarters of the institution) representatives of the institution as early as possible - preferably six weeks before the start of the evaluation.
In the course of the consultations, the following aspects are discussed:
- the effectiveness of the self-assessment and its results (based on the delivered documents),
- proposals of focus areas to be evaluated as suggested by the institution’s representatives,
- proposals of focus areas to be evaluated as indicated by the evaluators on the basis of available materials (along with justification),
- an initial outline of the evaluation plan.
The scope of the evaluation has to be sufficiently broad to obtain reliable answers to the key evaluation questions and to formulate a general assessment of the functioning of the institution. Of greatest important in the evaluators’ consultations with the representatives of institutions is the fact that the evaluators learn about the results of the self-assessment and have the opportunity to directly obtain additional information and clarifications, which helps to determine the capacity of self-assessment in the evaluation.
The fourth and the last step is determining the schedule of the evaluation and preparing a final plan. This includes the key evaluation questions as well as the methods and techniques for compiling and analysing the data. NZQA assumes that the highest level of the evaluation plan is the focus area being evaluated. The key evaluation questions are assigned to each of such areas. There are also lower level questions (used later in the interview scenarios) along with sources of data which are needed for a credible evaluation. Support and good communication with the evaluators is intended to help in identifying the required desk research data and the persons who should be contacted for interviews.
Thus, the logic of the evaluation can be described as follows:
- procuring data for every focus area to be evaluated,
- analysing and processing data for each focus area to be evaluated,
- recapitulating and analysing the conclusions with respect to their consistency and reliability,
- developing a synthesis of the information obtained and the conclusions leading to the evaluation of the educational achievements of the institution and its capability in self-assessment,
- discussing the emerging themes and conclusions with representatives of the institution as the evaluation is being performed,
- holding a meeting to recapitulate the results: presentation of the conclusions and recommendations and discussing them with the representatives of the educational institution, as well as clarifying potential ambiguities and disagreements.
4.6.2. Stage two: performing the evaluation
With the evaluation plan prepared, the evaluators start the field study. The collected data are analysed and interpreted by more than one evaluator in order to achieve the most objective conclusions. NZQA does not specify the exact duration of this stage, according to the available reports, it usually takes 2 – 3 days.
The evaluators are responsible for documenting the entire process (analysed documents, conducted interviews, encountered problems, etc.). If they are using materials that were made available by the educational institution, they should use citations in the documents with a note about the manner in which a given piece of information was used in the evaluation. The evaluated institution has the right to inspect such documents.
In line with the recommendations of NZQA, the main material for the analysis is the information derived from the self-assessment. The evaluators analyse it not only with respect to the regularity and complexity of the self-assessment, but primarily its effectiveness – the reactions to the problems detected, the use of the obtained data to improve the functioning of the institution and to advance educational achievements. Educational institutions are responsible for reporting and presenting their results and accomplishments to NZQA. Their role is the active provision of data, and not the passive submission to evaluation and assessment.
In the New Zealand system, an institution’s self-assessment is not a declaration of minor significance. A credibly prepared self-assessment is the basis for issuing the final assessment of the institution's functioning (within the scope of educational achievements and capability in self-assessment) by the evaluators. Such an approach is evidence of the fact that NZQA has significant trust in the evaluated institutions, provided they fulfil their obligations.
If the documents from the self-assessment are deemed insufficient (due to detected irregularities, gaps in the evaluation system, lack of reflection about the results), the representatives of the evaluated institutions and other stakeholders (students, teachers) are engaged in the external evaluation to a greater extent to expand the information already available. In special cases (lack of cooperation on the part of the decision-makers, hindering the evaluation and access to information), the evaluators may conduct evaluation activities without the participation of the representatives of the educational institution.
As the data is being analysed, interviews are also conducted with representatives of the evaluated institution. NZQA documents do not specify the level of their standardisation. Nevertheless, it seems that they have the form of individual in-depth interviews. Their aim is to discuss the processes occurring in the institution and to provide information about its plans and priorities. In the course of an interview, the evaluators should determine the degree to which a given institution is capable of effectively identifying the needs of students and other stakeholders (employees, employers) and responding to them.
The received data are helpful in providing answers to each of the questions asked in the course of the evaluation (both the key evaluation questions, as well as the detailed one within every focus area). With the aim of maintaining the comparability of results, each question is answered by selecting one of the options:
or a remark is made that the data are insufficient to answer the question.
The mark is accompanied each time by a brief standardised descriptive assessment, which explains its meaning.
At the end of the field study, the representatives of the evaluated institution are invited to take part in a meeting with the evaluators, during which the course of the study is discussed as well as the initial conclusions. The meeting forms yet another – the last – opportunity for clarification – as part of the dialogue between the evaluators and representatives of the institution being evaluated – of the detected problems and ambiguities.
The presentation of the main conclusions and findings, even before they are written and issued in a report, is meant to guarantee that the received grade will not be a surprise and will not be unjustified. Direct contacts with the evaluators ensure that their conclusions are understood and effectively used to improve the functioning of the institution in the future.
4.6.3. Third stage: formulating conclusions pertaining to educational achievement and capacity in self-assessment
Field studies allow the evaluators to obtain the greatest amount of data on all areas included in the evaluation and the best understanding of the processes taking place in the institution. On the other hand, at the stage of formulating conclusions, the standardisation and synthesis of the answers received is emphasised.
The evaluators put the material collected in order, moving from answers to detailed questions to the key evaluation questions and from conclusions pertaining to individual areas being evaluated to the evaluation of the entire institution. Even though the grade scale remains unchanged, the descriptive grades are changed, depending on what they refer to (educational outcomes, capacity of self-assessment) and at which scale (a single area, the entire organisation).
NZQA distinguishes three activities as part of the process of formulating conclusions:
- evaluation of educational performance and the capacity of self-assessment within the individual focus areas being evaluated,
- evaluation of educational performance and the capacity of self-assessment in the context of every key evaluation question,
- the final evaluation of educational performance and the capacity of self-assessment at the level of the entire institution.
The main tool that makes it easier for the evaluators to formulate conclusions (from individual questions and areas up to the institutional level) is the CORE outline. This procedure is meant to guarantee the credibility of the final results.
Table 2. CORE outline
|Concentration of ratings||Determine the mode and the median of grades|
At this stage, the evaluation team may look at earlier findings and the performed data analysis once again and/or adjust the final conclusions. Differences in opinions among the evaluators should be discussed.
The evaluators must remember about the hierarchy of the formulated conclusions: a high evaluation of educational outcomes in the context of a given study question or a study area does not necessarily automatically mean a high evaluation of overall educational performance at the level of the entire institution. It is only the analysis of all conclusions that allows the full picture of institution’s operations to be captured. The conclusions contained in the report must be the result of a discussion of the entire team of evaluators, however one researcher is designated to prepare it. The evaluators have to be ready to defend their positions and to justify every grade.
4.6.4. Stage four: issuing the evaluation report
The purpose of preparing the final report is the transparent and detailed presentation of the conclusions from the evaluation, along with the relevant justifications. The document is addressed to a broad group of recipients – NZQA, representatives of the evaluated institution, students/course participants, Higher Education Committee, experts and other persons interested in the situation in the education market.
Apart from a summary of the institution’s functioning, the report is intended to form the basis for its future development. The numerous recommendations and guidelines included in the report pertain to areas requiring improvement. This is significant due to the fact that the conclusions from the current evaluation become part of the material that is reviewed for the next evaluation. This is intended to reflect the importance of the systemic nature of quality assurance in the educational institutions.
Depending on the cathegory of reliability (see: Section 4.3) the scope of NZQA’s oversight of a given institution changes – the highest grade is subject to the classic evaluation in a four-year cycle, whereas an institution placed in a lower category is evaluated more frequently (an institution may be subject to an evaluation even every 6 – 12 months).
An institution is included in the highest category I if it receives the highest grades with respect to the educational achievement and capability in self-assessment, or the highest grade in educational achievement and a good evaluation of capability in self-assessment.
Category II includes institutions that receive the highest grade in capability in self-assessment and a good evaluation of educational achievement or good grades in both these areas.
Category III includes institutions with educational achievement or capability in self-assessment evaluated as adequate, yet requiring improvement.
The last, category IV, is reserved for institutions where serious problems were detected in relation to educational achievement or the process of self-assessment.
Institutions from categories III and IV are subject to an additional evaluation to verify whether sufficient measures were taken to improve the problems. If this additional evaluation also ends in a negative grade, NZQA may take further legal steps, including the withdrawal of accreditation or (in special situations) sending the case to court.
 Education Act 1989, Journal of Laws 1989, No. 80, amended on 2 January 2018. Cf.: http://www.legislation.govt.nz/act/public/1989/0080/latest/DLM175959.html
In the New Zealand system, the costs of performing the external evaluation are borne by the evaluated tertiary education organisations. Every evaluation is priced individually, and the final amount depends on the outlay of work of the evaluators. NZQA covers the costs of travel of the evaluators for the site visit and unforeseen expenses, irrespective of the operation of the evaluated institution.
The cost of the evaluation is presented to a given institution at the stage of specifying the study plan. The four-year cycle of performing the external evaluation helps to distribute the incurred costs. In its published materials, NZQA states that it makes every effort to make the costs proportional to the scope of the study, and to make the process transparent and consistent.