- The Norwegian Agency for Quality Assurance in Education (NOKUT)
Diploma of higher education
1. Origin, Institution name
2. Institution website
Diploma of higher education
4. Good practice - Zewnętrzne zapewnianie jakości
4.1. Background Information
The good practice pertains to the system of quality assurance and validation in higher education institutions (HEI) in Norway with special attention given to the modes of performing evaluation. The system is managed by the Norwegian Agency for Quality Assurance in Education (NOKUT), operating on the basis of regulations issued by the Norwegian Minister of Education and Research.
The quality assurance system in Norway consists of a number of elements, including data collection on the quality of teaching, the supervision of HEIs, their accreditation and the awarding of distinctions and prizes to particularly distinguished HEIs. Since 2003, NOKUT has also been implementing cyclical in-depth evaluations of study programmes and internal quality assurance systems at university and non-university higher education institutions (including polytechnics and tertiary vocational schools). As of 2014, the annual Studiebarometeret survey has been included in the cyclical evaluations conducted. The survey is sent to all students in Norway, which enables them to assess the study programmes they attend.
The purpose of the quality assurance system is not only to supervise the functioning of the higher education system, but primarily its improvement, among others, by providing information, as well as stimulating the development of quality in educational services and the recognition of diplomas awarded by foreign universities. Comprehensive and broad-range activities are aimed at contributing to building society’s trust in the education system of Norway.
Meeting the objective above is ensured by basing the quality assurance system, including the design of the research for performing evaluations, on the following premises:
- transparency of the research process: the evaluated institution is informed both about the methods and the tools used during the evaluation, and its effects;
- involvement of representatives of evaluated institutions in the evaluation process: informing about the proposed evaluation criteria, details of the plan and course of the evaluation, presenting and consulting initial report results;
- involvement of representatives of evaluated institutions and other stakeholders in the process of developing research tools: conducting exploratory and pilot studies to collect feedback about the transparency, structure and adequacy of the research tools proposed by NOKUT; additionally, as in the case of Studiebarometeret, the structure and the content of the questionnaire is consulted each year with experts;
- evaluation of the reliability and utility of the data from the internal evaluation of the institutions: the evaluated institutions perform self-assessments of their activities and the quality of the education, and subsequently the process of self-assessment and its results are analysed by external experts;
- involvement of external experts: use of the competence and experience of experts within the scope of quality assurance facilitates obtaining reliable and comparable results.
In line with these assumptions, responsibility for ensuring the quality of education is placed to a greater degree on the educational institutions themselves, however, NOKUT still plays a supervisory and monitoring role in the system. This reduces its workload, allowing the Agency to intervene in those organisations that require it. The adopted approach also reinforces trust in the supervising institution as one that recognize the autonomy and competence of the evaluated higher education institutions. Trust is also fostered by NOKUT’s emphasis on dialogue and cooperation with education system stakeholders: students, the evaluated institutions, representatives of the labour market, and other interested entities.
As a result, the solutions functioning in Norway offer an interesting example from the point of view of developing an institutional culture of quality assurance. Its foundation consists of a comprehensive and reliable self-assessment, the engagement of the evaluated institutions in the evaluation process, as well as the promotion of good practices.
4.2. Elements of the Quality Assurance System in Higher Education in Norway
The tasks performed by NOKUT in the quality assurance of higher education are conducted in eight areas listed below, which are considered the most important elements of the quality assurance system:
- Data collection on the quality of education: developing and managing databases about educational institutions and using this information to prepare analyses and statistics to obtain an overall picture of the status of higher and tertiary vocational education in Norway;
- Supervision of educational institutions and programmes: evaluation of how institutions manage their internal procedures and quality assurance systems, as well as how these systems are developed, by:
- conducting thematic audits: encompassing several different institutions, enabling comparisons to be made about the quality of education in a specific area of education;
- performing supervision based on education quality indicators and information obtained by NOKUT: a procedure initiated when a given institution does not apply specific indicators and has an inadequate level of teaching.
- Accreditation of higher educational institutions and teaching programmes: evaluating whether a higher educational institution or its study programme(s) fulfils the standards and criteria set forth by the Ministry of Education and Research and NOKUT. These standards refer to the scope of activities, personnel, infrastructure, organisation, participation in research projects and domestic or international networks of higher education institutions. Two types of accreditation awarded to institutions are distinguished:
- accreditation of selected study programmes: compulsory and performed in all formally recognized higher education institutions in Norway (apart from exceptions determined in the act) for society at large; it is awarded for an unlimited period of time, whereas it may be revoked, e.g. after performing supervision;
- accreditation of an educational institution: voluntary, however attaining this type of accreditation provides an institution with the authorisation to independently offer study programmes and award academic degrees and diplomas.
- In-depth external evaluation: evaluations are performed independently by NOKUT or in cooperation with other partners. Their aim is not only to assess the quality of the education in institutions (or areas of education), but also to develop the basis for further studies and capturing the process of change or stagnation in this respect. Furthermore, the projects implemented by NOKUT jointly with other state agencies and educational institutions may form the basis for developing systemic solutions in selected areas of education (e.g. integrated model of evaluation at HEIs encompassing both the quality of education as well as its outcomes in terms of the research undertaken by students and graduates).
- Studiebarometeret survey: an annual national survey of students. The barometer focuses on assessing the quality of the education in the student’s education facilities, but it does not contain questions about the assessment of educational institutions or the quality of studying in them. The information obtained from the survey is made available to the educational institutions, which frequently incorporate them in the results of their self-assessments. This enables the institution to obtain a more comprehensive picture of the activities they conduct by having it supplemented with the students’ point of view.
- Centres for Excellence in Higher Education initiative: a prestigious group of higher education institutions, whose efforts to ensure the quality of education and implement innovative approaches are treated as models to be followed. Candidates for the initiative are reviewed by experts appointed by NOKUT, and the selection criteria include an effectively operating internal quality assurance system, as well as the sharing of knowledge and experience with other centres.
- Quality in Education Prize (Utdanningskvalitetsprisen): annual prizes awarded by NOKUT to higher educational institutions distinguished by their efforts to ensure the quality of education.
- Dissemination and ensuring public access to completed evaluations: providing the broadest group of recipients with information about the condition of Norwegian education. NOKUT publishes the results of studies on its website and disseminates them via newsletters, publications on blogs, etc. Furthermore, it organises debates and meetings for representatives of educational institutions, politicians and experts, offering an opportunity to discuss various aspects of the quality of teaching in education, sharing knowledge and learning about other points of view.
In pursuing its mission, NOKUT emphasises dialogue and cooperation with students, educational institutions, employers, other interested social partners and governmental agencies. Meetings integrating the education community also play an important role in presenting the results of NOKUT’s studies and the trends they exhibit in education and its quality. Representatives of various stakeholder groups have the opportunity to learn about different aspects of quality assurance in education, and as a result, the activities they undertake may be more responsive and effective (legislation can be better adjusted to the needs and specifics of tertiary vocational education, internal self-assessment procedures may better contribute to the premises of national education policy and comply with all formal requirements). This also provides opportunities to establish new contacts and builds trust between NOKUT and various stakeholders groups involved in education (transparency of research, openness to discussions on conclusions and results).
Details on two of the elements of the quality assurance system for higher education in Norway are presented below:
- the audit, i.e. in-depth external evaluation of study programmes and education institutions
- the Studiebarometeret survey.
4.3. Audit: In-Depth External Evaluation of Teaching Programmes and Educational Institutions
Audits performed by NOKUT of selected study programmes and educational institutions with respect to internal quality assurance and self-assessment take place at least once every six years. If any shortcomings are detected, NOKUT designates a deadline for correcting them. After this time, another evaluation is performed in order to verify implementation of the recommendations (this usually takes place 6 months before the end of the main study).
NOKUT invites external experts to perform the audit. This enables the involvement of persons who are highly competent in evaluation and familiar with the relevant area of education. It also ensures the objectivity of formulated opinions and conclusions. The candidates for the role of expert are assessed by NOKUT. On this basis, an expert committee of several members is formed and authorised to conduct the audit. The committee is responsible for meeting specific objectives and evaluation criteria as well as for presenting the final conclusions and recommendations to the NOKUT Board . Members of the committee sign a declaration ensuring their impartiality, the lack of any relationship with the evaluated institution and upholding the confidentiality of the audit and its results until the report is published (they cannot speak in public about the audited institution before the audit is concluded). The expert committees include highly qualified scientists from Norwegian and foreign universities, also sometimes representatives of foreign quality assurance agencies in higher education. NOKUT provides only general guidelines for the audit, leaving the expert committees with significant freedom in performing the evaluation and determining the techniques to be used.
The main purposes of the in-depth external evaluation performed by NOKUT are:
- to evaluate the educational outcomes and make recommendations to improve the quality of education
- to evaluate the internal evaluation system of the given institution
- to refer to the premises and results of specific study programmes defined in the NorwegianQualifications Framework (in the case of tertiary vocational study programmes)
An important element of quality assurance in the Norwegian system is the self-assessment performed by the educational institution. The results are provided in a self-assessment report, which is the subject of the in-depth external evaluation in the desk research phase (see Point 4.3.2). NOKUT recommends that the self-assessment be used as a tool for diagnosing the needs of an institution, to enhance its effectiveness and contribute to making rational decisions about the directions of its development. Self-assessment understood in this way acquires an institutional dimension and should be a process that includes the continuous collection of data. This complies with the premises of the evaluation system, where a comprehensive self-assessment and involvement of institutions in the process of evaluation constitute the foundation for building a culture of quality assurance. Additionally, the participatory nature of self-assessment is emphasised. It should engage all groups of stakeholders, both students and employees at various levels.
NOKUT does not determine the methods and plans that institutions should use for self-assessment. The solutions adopted by individual HEIs have to fulfil the premises of the process, yet the mode of its performance is an individual and non-regulated issue (in some situations, institutions receive a form with questions formulated by NOKUT containing the evaluation criteria). However, the starting point is an assessment of its own activities on the basis of indicators and standards set forth by NOKUT. Tools used for self-assessment include questionnaires for class participants and forms listing the academic credits achieved, which is completed after exams and tests. Based on the data received, the institutions prepare their own reports, which are later made available to the Agency.
4.3.2. Desk Research
HEIs in which in-depth external evaluations are performed are responsible for providing the self-assessment report to NOKUT. The results of the self-assessment are given to the Agency together with other required documents (study programmes, financial reports, educational materials, test and examination templates, etc.). NOKUT often entrusts the analysis of the submitted documents to committees composed of foreign experts. Based on the desk research performed in this manner, an initial report is prepared containing the conclusions and assessment of the reliability of the performed self-assessment. The information received is verified during the next stage of the audit: a study visit to the audited institution, conducted by the expert committee appointed by NOKUT.
4.3.3. Study Visit and Report Preparation
After completion of the analyses described above and preparation of the initial report, the expert committee conducts a study visit, i.e. a field study of the education institution being evaluated. During the visit, the conclusions drawn on the basis of the institution’s self-assessment are verified. One of the premises of the Norwegian quality assurance system is a flexible approach to the research procedures and techniques: the exact course of a study visit depends on the specific nature of the evaluated institution or study programme. An individual approach to the evaluated HEI means that there are no standardised, universal research tools. However, the experts most frequently use qualitative methods:
- interviews: individual in-depth interviews or focus group interviews conducted with employees of the institution, students and participants of training sessions and workshops
- class visitations.
The standard methodology is sometimes expanded to include additional elements depending on the need, e.g. additional analyses and/ or review of examination documents. NOKUT decides on the scheduling of a given institution’s audit and determines its duration with the HEI and the experts’ committee.
During the study visit, the expert committee stays in contact with the representatives of the audited institution. In line with the premises of the quality assurance system, this is one way of increasing the involvement of the HEI in the audit. By providing information about its activities and presenting the initial results, NOKUT ensures that the procedures are transparent. This also increases mutual trust between the Agency and the educational institution. By participating in the experts’ work, the institution is able to more fully understand the formulated conclusions and to use them effectively to improve its functioning, which, in turn, is conducive to a better higher education system and quality of education in Norway.
After the end of the study visit, the expert committee prepares a final report of the complete audit of the institution. If any shortcomings are detected, the document also includes recommendations pertaining to areas requiring improvement, e.g. elements of the internal quality assurance system or the self-assessment process.
By rewarding and promoting, NOKUT distinguishes institutions which stand out in terms of the quality of their education and have effective quality assurance mechanisms. This is aimed at disseminating good practice and persuading other institutions to make an effort to pay close attention to the quality of their activities. It is also an incentive for becoming involved in developing a consistent quality assurance system in tertiary education.
4.3.4. Results of the In-Depth External Evaluation and the Accreditation Process of Educational Institutions
The results of the in-depth external evaluation may affect the accreditation of an educational institution. If any gaps or shortcomings are detected, NOKUT designates a deadline for introducing the needed changes. The time allotted to make the improvements depends on the importance and the scope of the detected problems in the education system. It can range from 6 months to 2 years. After this time, a follow-up evaluation is performed in order to verify if the provided recommendations were implemented and whether they contributed to the improvement of the situation in a given institution.
If an educational institution fails to follow the recommendations of NOKUT and also receives a negative assessment during the follow-up evaluation, the Agency undertakes specific steps: it may, for example, withdraw the accreditation for a specific study programme or for the entire institution (which requires that a representative of the Ministry of Education and Research countersign and confirm the decision).
After losing accreditation, the next consequence of failing to follow the recommendations of NOKUT is the loss of public financing.
If the accreditation for a given study programme has been revoked, the evaluated institution is responsible for making it possible for its students to complete the commenced study programme and to attain the qualifications.
The solution proposed by the educational institution to resolve the situation must be accepted by NOKUT. The Agency then assumes the role of an adviser and consults the possibilities of how the problems may be solved with the evaluated institution, and also works together with it while the improvements are being implemented.
The number of programmes rejected or selected for a follow-up evaluation shows that the Norwegian quality assurance system is not only a formality – the applied evaluation criteria and NOKUT standards place high expectations on education institutions and study programmes. This makes it a real quality assurance tool for education and formation of a culture focused on continuous development.
The Studiebarometeret survey is not compulsory, yet according to data provided by NOKUT, few institutions in Norway decline to participate in it (e.g. military academies, some private higher education institutions). After an initially sceptical approach on the part of the education sector, today the survey is positively viewed by all institutions participating in it, primarily because they are able to use the generated data. The Studiebarometeret results allow NOKUT to perform a general evaluation of the study programmes offered in a given institution. The survey results are also analysed by the Ministry of Education and Research to identify areas requiring intervention or monitoring.
Information provided to the Agency by the Norwegian Association of Higher Education Institutions and the National Union of Students of Norway indicates that the survey contributes to improving the quality of education and developing education institutions in three ways:
- NOKUT makes databases available to educational institutions enabling them to supplement data from their self-assessments. As a result, their picture of the functioning of their study programmes is supplemented with students’ opinions. Many universities include Studiebarometeret among the tools used for internal quality assurance.
- The survey allows a comparison to be made of students’ opinions about the study programmes offered at various universities. This enables study programmes in similar fields to be easily compared, as well as to evaluate and compare the education institutions.
- The results of the Studiebarometeret survey are published and made available to all interested parties, which significantly affects the functioning of the evaluated educational institutions. Good results serve to promote a HEI and positively impact its employees. On the other hand, negative results stimulate the institutions to undertake improvements and introduce changes to perform better in the next edition of the survey.
The survey is conducted by using an Internet questionnaire (computer-assisted web interview – CAWI) sent to students by e-mail (in October or November). Databases with students’ addresses are procured from the educational institutions. The questionnaire is standardised and the respondents answer the same set of questions. This approach allows the results of various institutions and study programmes to be compared. NOKUT analyses the collected data. The final report from the survey is published at the beginning of the next year.
NOKUT undertakes a number of marketing activities aimed at promoting knowledge about the Studiebarometeret survey and encouraging students to take part in it. One of the ways of reaching young respondents was by designing the survey website for use by mobile devices, which allows the questionnaire to also be conveniently displayed and completed on telephones and tablets. Students were also informed and encouraged to take part in the survey by their educational institutions. Offering gift coupons is an additional incentive, which several dozen survey participants win every year.
In 2015, the response rate was 47%, which is a good result for an Internet questionnaire. This enabled NOKUT to assess 73% of all study programmes offered in Norway (bachelor and master’s degree studies). Norwegian research institutions can also access the database (without personal data).
 The Board of the Norwegian Agency for Quality Assurance in Education is the highest authority managing the Agency. Its members include representatives of the academic community, employers’ and business organisations, international research centres, student organisations, as well as NOKUT.
5.1. Tools for Conducting the Audit, i.e. the In-Depth Evaluation
NOKUT uses various research methods in the audit, adapting them to the specific nature of the institution or study programme and the character of the conducted study. Sometimes, a combination of various qualitative methods are used, in other cases – a combination of qualitative and quantitative methods.
- Combination of several qualitative methods. This may be a combination of desk research and materials from interviews and class visitations gathered during a study visit. The collected data are analysed by external experts. By compiling information from various sources, a more precise assessment of their reliability and utility is possible. The engagement of a higher number of experts reduces the risks to research results, e.g. observer bias, relating to the final version of conclusions and assessments.
- Combination of qualitative and quantitative methods. The Norwegian quality assurance system uses a combination of the two types of methods to assess the Centres for Excellence in Education. Desk research is conducted to start, and a questionnaire is sent to individual educational institutions. Later, based on the questionnaire results, individual in-depth interviews are held with representatives of the institutions. The next stage is holding focus group interviews. This research structure assumes that quantitative methods will be used to narrow the group of institutions to be further investigated by using qualitative methods. The main purpose of such activities is to distinguish outstanding education centres, to identify good practices and collect detailed and valuable data in this respect. Educational institutions applying for the title of Centre for Excellence in Education respond to questions pertaining to the main purposes of their study programmes, the aims and tasks relating to their application to the competition, and their impact in the area of education. They are also assessed in the categories of sustainable development and self-improvement strategies.
As mentioned before, research procedures are adapted to the specifics of a given institution or the assessed study programme. The application of various research methods leads to insights on a given issue from various perspectives, yet at the same time, it negatively affects the level of standardisation and comparability of the obtained results in individual years. The collected data offer a basis for evaluation, yet they do not provide a fully consistent picture of the entire area of higher education in Norway. Furthermore, expanding the scope of the study with additional issues relating to the educational institution or study programme must be strictly controlled. The expert committee should not lose sight of the main purpose of the audit, and thus a reliable and comprehensible summary of the educational results of a given institution and its competence in the area of internal quality assurance.
5.2. Questionnaire for the Studiebarometeret
The questionnaire was prepared by a group of education and evaluation experts and based on foreign experiences. First of all, it was tested in two pilot tests with students from various HEIs pursuing a variety of study programmes. The questionnaire was well received by the respondents and showed that valuable information could be gained from it. Changes were then introduced to the scenario and the final version was prepared. Furthermore, the tool is continually amended on the basis of feedback collected on an ongoing basis.
The Studiebarometeret survey consists of questions from several thematic areas:
- academic teaching and guidance
- education conditions
- students’ involvement in and impact on the study programme
- possibilities of inspiring students with the study programmes offered
- professional internships
- utility in professional life
- assessment system and assignment of work
- learning outcomes
Additionally, students are asked about the level of their general satisfaction with the study programme.
Every year, the questionnaire is changed (editing the cafeteria of answers, adding or removing questions); therefore, the results on various study programmes or educational institutions are fully comparable only within a given year. The questionnaire uses a five-point Likert scale (with the additional option “I don’t know/NA” when this is justified).
The data made available by universities enables NOKUT to send the questionnaire to almost the entire student population of Norway (99%). The survey has a high response rate. It was noted that the questionnaire is completed more often by women, who assess their satisfaction with educational services at a level higher than men. The differences are not great, but statistically significant. This allows trends to be determined, as well as existing factors and deviations that should be taken into account when formulating conclusions from the survey.
The evaluation activities performed by NOKUT are financed from budget funds allocated by the Ministry of Education and Research. The documents made available do not indicate that the evaluated institutions participate in these costs.
The research generates high costs because of its scale (it covers the entire area of higher education in Norway) and the external experts hired for its performance, who are often foreign and use diversified research tools. Nevertheless, it seems that these expenses are justified in the context of achieving the mission and accomplishing the objectives of the quality assurance of education.