Observation in simulated conditions (simulation)

Observation in simulated conditions (simulation)

Characteristics

Characteristics

Observation in simulated conditions (simulation) analyses a candidate's performance in conditions imitating real life, created for the needs of the validation process.

The use of observation in real-life conditions can sometimes involve a dangerous activity, be too time-consuming or raise an ethical dilemma for the observer (e.g. care for an elderly person). In such a situation, it can be replaced with observation in simulated conditions. In this case, the person designing the validation develops and provides the assessment conditions that allow the behaviour of the candidate to be observed in unusual or dangerous conditions. A simulation also makes it possible to confirm the social competences needed to build and maintain proper relationships in the workplace or the skills of implementing tasks using specialised (often expensive) equipment, which further supports its use.

Observation in simulated conditions is suitable for confirming a wide range of learning outcomes. They include interpersonal, analytical, communication and organisational skills and competences, as well as the knowledge on which the use of these skills is based. This method can be used to identify and assess the learning outcomes required to create products or achieve specific results (e.g. technical, manual, artistic, fitness-related), as well as those that are not subject to direct observation (abstract, hidden), but whose achievement can be inferred, for example, based on the results of the performed work.

The simulation method is valid for learning outcomes that are directly observable (skills and some social competences). Otherwise, it should be combined with other methods. It is also recommended when confirming complex social skills and competences at levels 48 of the Polish Qualifications Framework (PQF). This allows the “examiner's effect” to be minimised, i.e. assessing the candidate on the basis of an overall impression or always awarding extreme or average grades.

Assessment in simulated conditions requires designing observations in such a way that it is possible to control the process and react to undesirable events that are not part of the assessment. To this end, a scenario should be prepared before applying simulation techniques, which includes:

• the type of learning outcomes being confirmed,
• the conditions ensuring compliance with safety rules and ethical standards,
• procedures for responding to events that exceed the planned scenario of the method.

In addition, before using the method, one has to make sure that the candidate understands the assignment and is provided with all necessary tools or aids required to perform the task.

A study by Cedefop (an agency supporting the development and promotion of vocational education and lifelong learning in the European Union) shows that in recent years, observation in simulated conditions is being used with increasing frequency. In 2010, it was used in six European countries to validate learning outcomes, and four years later – in already 12. This method is used by employers as part of recruitment (34% of companies surveyed in 10 countries) or for employee assessment (26% of companies). It is more willingly used by large companies.

The information obtained on achieved learning outcomes by observing under simulated conditions is collected with the same tools as when observing in real-life conditions (notes, work samples and scales). Taking into account the results of the quantitative tools used, including performance scales (e.g. possible and correct final result – for a practical task, simulation game or project, average time – for a fitness test), increases the probability of a reliable validation.

An observation in simulated conditions may involve a practical task that can be solved in several ways, and the scenario does not indicate which is the correct solution. The auditor should examine the solution in an impartial manner, even if he/she prefers a different method than the one used by the candidate.

Validation stages in which the use of the method is recommended

Validation stages in which the use of the method is recommended

Cedefop data show that simulation is almost always used in the assessment stage in combination with other methods (e.g. debate, interview, analysis of evidence and statements, presentation, observation). It is also used in the identification and documentation stages in only two countries (Portugal and the United Kingdom).

Scope of the learning outcomes that may be assessed using the method

Scope of the learning outcomes that may be assessed using the method

Information on the learning outcomes that can be confirmed through observation in simulated conditions is for guidance only. It is based on an analysis of the required learning outcomes in the Polish Qualifications Framework (second stage generic descriptors typical for vocational education and training).

It should be remembered that the choice of a given method must always result from the analysis of the learning outcomes for a given qualification.

Knowledge

Observation in simulated conditions is focused on analysing an activity, and the time of its duration is often limited. Therefore, it enables evidence to be gathered on acquired basic knowledge, usually from PQF levels 1–4 in reference to:

• theories and principles (aspects of the PQF: methods and solutions, business activity),
• phenomena and processes (characteristics and conditions),
• organising work (methods and technologies, organisational solutions),
• tools and materials.

In the case of complex learning outcomes (from higher PQF levels), focused on a deeper knowledge of the work environment, it is recommended that simulation be combined with other methods.

Skills

Observation in simulated conditions is recommended primarily for this category of learning outcomes. It enables the confirmation of all types of skills contained in the PQF descriptors typical for vocational education and training. It does not require additional assessment using other methods.

The higher the complexity of the learning outcomes (the higher the PQF level) to be assessed, the more complex assessment tools or expensive technologies and devices may be needed for the simulation. Some of the learning outcomes in this category may require the candidate to perform tasks in which occupational health and safety rules need to be followed.

Social competences

Observation in simulated conditions is recommended to assess learning outcomes at lower PQF levels relating to compliance with rules, cooperation (communication) and responsibility.

This method works well in assessing learning outcomes relating to communication in the work environment. At PQF level 3, it is possible to confirm them using a simulation game involving role playing.

However, an accurate assessment of such outcomes at PQF level 4 requires taking into account information obtained using other methods (analysis of evidence, debate, interview, presentation or observation in real-life conditions).

Some social competences, such as loyalty, are associated with characteristics that can be confirmed only through the use of the analysis of evidence, interview or long-term observation. It is relatively difficult to use simulation to confirm readiness to cooperate, and especially to assess communication in the professional community (e.g. with external companies).

The suggested learning outcomes that can be confirmed using the Observation in simulated conditions method

Strengths and weaknesses of observation in simulated conditions

Strengths and weaknesses of observation in simulated conditions

Strengths

Weaknesses

allows the confirmation of learning outcomes, whose observation in real-life conditions could be dangerous, time-consuming or would raise ethical concerns (e.g. care for an elderly person) suitable for confirming learning outcomes from the skills category during the simulation, the assessment may relate to the way of performing a task, the behaviour of the candidate, the products of his/her work can be used together with other methods how the learning outcomes were achieved does not affect the course of observations in simulated conditions each candidate being observed in simulated conditions is assessed in the same way

the use of the simulation is limited to the assessment stage can be expensive in the case of qualifications requiring skills in handling specialised equipment, apparatus and machines; it is very expensive when it requires the use of specialised devices the duration of the observation in simulated conditions is limited – the method may not be accurate if the learning outcomes are observable only after a longer period of time (e.g. loyalty, developing professional relationships, knowledge of trends in the industry) the results of the observations are particularly susceptible to distortion due to the “examiner’s effect”, i.e. assessing a candidate on the basis of an overall impression or always awarding extreme or average grades

Limitations of using the method

Limitations of using the method

From the perspective of the candidate, using the simulation method may be limited by his/her personality traits or special needs. Techniques that require professional roles to be played in front of observers may block people who are introverted or find public speaking stressful.

The use of techniques that require the creation of a product or work may be difficult for people with reduced mobility. Sometimes it is enough to give the candidate more time to complete the task.

If necessary, the simulation can be adapted to the needs of the candidate. However, the assessment scenario should not be changed to the extent that the accuracy or reliability of validation is affected. Sometimes it may turn out that the use of other methods to confirm the same learning outcomes (e.g. analysis of evidence and statements) will be better for a candidate with special needs.

Due to the need to comply with occupational health and safety rules, the use of some techniques may limit the field of view of the assessor during the observation. If the assessor must remain a safe distance away from the area where the candidate is performing a task during assessment, then the use of sound and image recording equipment is recommended to register the activities.

Required human, organisational and material resources

Required human, organisational and material resources

When conducting simulations, the assessor usually collects information about the candidate. An external observer may also be involved; he/she will use the indicated assessment tools. The results collected in this way are subject to further analysis by the assessor. The observer should also ensure that the assessment process does not deviate from the designed scenario. The need to train staff in gathering information about the candidate (use of assessment tools) and providing a reliable assessment of the learning outcomes, as well as assuming a neutral attitude towards the candidate may increase the costs of simulation.

Applying this method on a larger number of candidates can be time-consuming and may require hiring multiple assessors or using recording equipment that will enable the review process to be recreated during the assessment stage.

If there is a need to comply with specific safety rules, the awarding body should ensure the presence of a first aid provider.

The costliest aspect of using observation in simulated conditions may be the needed organisational and material resources. In order for this method to be reliable and close to real-life conditions, the equipment and software used in the work of the given qualification must be provided. If simple skills are being confirmed (e.g. calculation), the costs of using this method do not have to be high. However, in the case of qualifications relating to health protection or public safety services, the simulation budget can increase by the costs of equipment or simulators.

Required materials: specialised tools and machines used at workplaces relating to the confirmed qualification; a stopwatch or watch for measuring the time of performing activities; voice recorder, camera or photo camera.

When observing the activities requiring compliance with occupational health and safety rules, it is advisable to wear protective head (e.g. caps, helmets, goggles, masks, ear protections, earmuffs) and body gear (overalls, aprons, vests, gloves, knee pads, footwear protectors). It is also advisable to equip such a simulation site with first aid supplies and items (first aid kit) or fire-fighting equipment (appropriate type of fire extinguisher, fire blanket).

If the simulation requires the use of a program or computer application, then the assessment site should be equipped with the necessary devices (tablet, computer, screen, mouse, printer, etc.) and the software mentioned in the description of the qualification. In some situations (e.g. when checking the ability to search, compare, evaluate and analyse the information needed to perform professional tasks), the candidate should be provided with access to sources, e.g. the Internet.

Possibilities of combining observation in simulated conditions with other methods

Possibilities of combining observation in simulated conditions with other methods

The method, which can supplement the simulation is the analysis of evidence and statements – primarily documents proving that the candidate has achieved some of the learning outcomes (professional certificates, employment certificates, certificates of completing training, etc.).

When confirming social competences, in addition to simulation, such methods as unstructured and structured interview, unstructured and structured debate, presentation can be used. In the case of knowledge – the theoretical test can be used (in particular when assessing learning outcomes from PQF levels 5–8). This applies mainly to communication in the broader professional community and knowledge relating to familiarity with and understanding of theories, phenomena and processes occurring in the workplace or the properties, principles and characteristics of the materials used and the products being made.

Example techniques

Example techniques

A technique is a way of doing a particular task in a given method, used to collect and analyse data proving that a person has achieved the learning outcomes.

In terms of techniques, simulation is a very diverse method. The type of technique depends not only on the category of learning outcomes, but also on the industry to which the qualification belongs. Candidates applying for qualifications relating to, for example, accounting, healthcare or the arts may be asked to complete a practical task or project. The common element is a scenario-based assessment of the task being performed, taking into account whether it is being done in accordance with the procedures or customary practices of a given industry.

Practical task

A particular technique used during simulation is to perform a practical task. Through its use, it is possible to assess learning outcomes mainly in the category of skills. The result of the task can be a work sample, e.g. a product, service or documentation. Not only is the final result assessed, but also the method of work (e.g. the chosen method of reaching the solution).

Unlike other techniques, a practical task allows a small part of the work to be checked, because it is limited by time or organisational resources.

One of the varieties of a practical task is the physical fitness test. This can include the performance of a single exercise or a series of exercises to check physical fitness under certain conditions. In this case, not only is the performance of the fitness task assessed, but also the level of its performance, measured by the result obtained by the candidate.

In the case of learning outcomes relating to an intangible piece of work, including an artistic work (e.g. a musical piece), only the implementation process is assessed. The type of the selected work (e.g. due to the pace) as well as the assessment criteria themselves (e.g. general impression, technical level, expression, performance quality) can influence the final result.

Due to the specificity of these types of skills, performance evaluation may be subjective. This can be prevented by including experts in a given field among the assessors and training them in the use of the assessment tools. A good solution may be to record the performance of the work, which will allow it to be reproduced at a later time and possibly discussed.

A case study is a particular type of practical task. This technique can be used during observation in simulated conditions and in a written theoretical test.

Box 1. Case study

Case study has many definitions. The term refers, among others, to research methods used in social sciences and to didactic methods. For this reason, it can be understood differently by different persons.

With regard to validation, a case study is a technique for confirming learning outcomes, and consists of completing a task or answering questions on the basis of information contained in a description of a specific situation (real or imaginary). The result of the case study can be, among others, a diagnosis of a given event, a proposal of solving problems presented in the case description, predictions about the further development of the situation or a proposal of action.

A case study can help to assess, among others, the ability to analyse information, draw conclusions, think critically and apply knowledge in practice. For example, a candidate can determine the reasons for a given situation, indicate the main problems raised in the case description, determine the relationship between the facts, predict or assess the effects of the described activities / events.

Strengths of the practical task:
it is a standardised tool for assessing candidates; it enables a part of the work to be assessed, thereby shortening the assessment process.
Weaknesses of the practical task:
in the case of confirming certain skills, the decision of the assessor may be biased by tendencies relating to the examiner's effect, which means that before the application of this technique, the assessors should be properly trained; the high costs of standardising the tool; sometimes the organisational and material costs are quite high.

Project

The project allows the learning outcomes required to make a product or provide a service to be assessed. Simulation conducted with this technique may require the candidate to reconstruct a given process (e.g. assemble a computer from provided parts and start it) or create a new object / service (e.g. develop new methods and technologies in the workplace). In the first case, the assessment is based on proceeding with the task in accordance with the procedures used in the workplace, for example, by applying increased security measures. In the second case – creative skills or the application of the knowledge and skills needed to solve the problem.

The use of the project technique requires the goal and presentation of the problem context to be precisely formulated. The content of the project task should specify which parameters will indicate its correct execution. Candidates should also be provided with the necessary tools and accessories as well as the time required to complete all stages of the task in stress-free conditions. The time devoted to implementing the project should include several stages leading to the solution of the design task.

Box 2. Stages of solving a project task

The project task is usually solved in the following stages:

1. diagnosing the situation – based on an analysis of the description of the task and an understanding of its context (possibilities and limitations),
2. proposing solutions – based on evaluating the possibilities of using the available tools,
3. critically analysing the variants – based on assessing the adequacy and usability of each solution,
4. selecting and implementing the project that best meets the assumed conditions of the task.

The final effect of the project can take several forms – each of them enables the confirmation of various learning outcomes, depending on the level of performance. The following can be identified:

• a product (service or item) that is a reflection of another product;
• a model, i.e. a simplified, theoretical version of a new product that meets the performance parameters specified in the task, reflects the capabilities of this product, but does not take into account its practical operation (e.g. decision model, physical model);
• a prototype, i.e. a project created to demonstrate the candidate's ability to create a new item or service that meets the parameters specified in the task and functions according to the specified assumptions.

Box 3. Application of the project

With reference to the Polish Qualifications Framework, the project may be an appropriate technique for assessing skills, e.g. those relating to planning and revising plans.

The individual levels of project implementation correspond to the successive levels of PQF for these types of skills. This is illustrated by the following example.

If the project task aims to confirm whether the candidate is able to:

• “plan very simple elements of occupational activities under the direction of a more experienced person” (PQF level 1),
• “plan simple occupational activities under the direction of a more experienced person and revise the plan according to circumstances” (PQF level 2),
• “prepare a plan of carrying out not very complex occupational tasks and revise it according to circumstances” (PQF level 3), or
• “prepare a plan of carrying out moderately complex occupational tasks under routine conditions and revise it according to circumstances for oneself and the team one is leading” (PQF level 4),

then the instruction to make a product will be appropriate. This can be a work schedule of people employed in various shifts at a company or a plan of tasks of individual employees in a given time period.

If the candidate is able to show that he / she can:

• “prepare a plan of carrying out occupational tasks taking into account variable, predictable conditions and revise it according to circumstances for oneself and the team one is leading” (PQF level 5), or
• “prepare a plan of carrying out occupational tasks taking into account variable, not fully predictable conditions and revise it according to circumstances” (PQF level 6),

it will be appropriate to instruct the candidate to create a model that takes into account possible changes in the conditions of a given workplace. In the above example, the plan may include the hiring and integration of a new employee who is replacing someone or when a team’s tasks are reorganised so that they can take on and perform unscheduled, urgent tasks.

The instruction to create a prototype would be appropriate in a situation where the candidate must show that he/she is able to:

• “develop a strategic plan for a work team / organisation relating to the work of the occupation" (PQF level 7) or
• “produce a development strategy relating to the work of the occupation, design the legal and organisational conditions to benefit the development of the occupation” (PQF level 8).

In this situation, analytical skills must be used to develop a plan of action for a larger group of employees, focused not only on the development of the candidate’s own company, but also its environment.

Project strengths:
it can be used to assess the learning outcomes of several people simultaneously; if it is used to assess the learning outcomes of several people, skills can be observed, such as the ability to work in a group, critical thinking, argumentation and creativity.
Project weaknesses:
producing a model or prototype can be time-consuming; sometimes it may require testing and adjustments before obtaining the final version; if the technique is used to assess a group of candidates, it may not be accurate for persons who are reluctant to work in a group.

Simulation games

Simulation games relate to the ability of a candidate to assume a professional role. They usually require the solution to a problem of a group of employees or a company. The game could be about a situation in which the candidates' have to solve a problem and take a decision in a matter relating to the operation of the company, or secure an agreement among interested parties.

The game should be based on a specific scenario that takes into account the topic, the roles and how they are allocated (e.g. randomly). In addition, it is important to provide participants with help or access to information that will enable them to reach a solution.

The game begins with a presentation of the situational context, e.g. in the form of a short story introducing the task, and an explanation of the rules for assessing the learning outcomes. The introductory stage is extremely important – it allows candidates to enter the situation, get to know the conditions and factors that may affect the way a solution can be reached, to play the professional roles of the individual participants and the relationships between them. Before starting the game, one has to make sure that participants understand its rules and objectives.

The game is played by assuming roles. One of them is the moderator (one of the assessors) – a person who does not directly participate, but who monitors the game (controls the time, provides explanations as needed or moderates if the discussion deviates from the subject; ensures compliance with the rules, and if they are violated – reminds the participants about the rules). In the course of the game, the moderator should maintain a neutral attitude, limit his/her activity and not influence the situation. When intervention is needed, he/she should not evaluate the participants, advise or guide them to a solution.

At the end, the moderator conducts a summary exercise, so that the participants can distance themselves from their roles and discuss the course of the game. The final phase is aimed at defusing emotions and is not to be assessed. It can be held in the group immediately after the end of the game, it can also be conducted with each candidate individually. Participants themselves can:

• discuss problems that arose while completing the task;
• assess the impact of the factors that led to the result;
• present other possible solutions;
• individually assess the outcome of the game.

The context of assessing a number of persons is specific to this technique. Regardless of the situation defined in the scenario, phenomena specific to group processes may occur, such as rivalry, coalitions, dominant or passive attitudes.

The result of the game is therefore not only the solution to the problem, but also – indirectly – the creation of a relationship between the participants, sometimes based on a hierarchical division (e.g. employees vs. employers, winners vs. losers). The latter results should not be assessed, which makes the task very difficult for the assessors. It should also not take into account the acting skills of the candidates.

Simulation games are suitable for confirming skills and social competences, especially cognitive and communication skills (including the ability to argue, convince others and work in a group). They can be used interchangeably with the interview and the debate, especially when assessing a larger number of candidates.

To evaluate the game, in addition to the results from a sheet containing statements or scales, it is worth using an audio recording. This is very useful when several people are assessed at the same time. In the event controversial issues emerge, it is then possible to recreate the course of the game and re-observe parts of it, which is very important if the number of observers is smaller than the number of candidates.

Strengths of the simulation game:
it allows several people to be assessed at the same time; it is suitable for simulating short situations that require the assessment of learning outcomes relating to cooperation or communication.
Weaknesses of the simulation game:
is susceptible to factors other than the candidate's learning outcomes (e.g. lack of observers’ impartiality, group processes); the presence of many observers or recording equipment may be stressful for the candidates.

Examples of the use of observation in simulated conditions

Examples of the use of observation in simulated conditions

Examples for the application of the observation in simulated conditions are contained in the Database of Good Practices. Validation, ensuring the quality of validation and certification, especially the carpenter's in Germany.

Bibliography

Bibliography

Cedefop (2014). Use of validation by enterprises for human resource and career development purpose. Cedefop reference series No 96. Luxembourg: Publications Office. Downloaded from: http://www.cedefop.europa.eu/files/3065_en.pdf

Cedefop European Commission; ICF (2017). European inventory on validation of non-formal and informal learning – 2016 update. Synthesis report. Luxembourg: Publications Office.

Łoś, E. and Reszka, A. (2009). Metody nauczania stosowane w kształtowaniu kompetencji kluczowych. Matematyka. Podręcznik metodyki operacyjnej [Teaching methods used in shaping key competences. Mathematics. Manual of operational methodology]. Lublin: Innovatio Press Wydawnictwo Naukowe Wyższej Szkoły Ekonomii i Innowacji.

Websites:

Cedefop. European database on validation of non-formal and informal learning [database]; http://www.cedefop.europa.eu/en/publications-and-resources/data-visualisations/european-database-on-validation-of-non-formal-and-informal-learning