Observation in real-life conditions
Observation in real-life conditions analyses a candidate's performance in the actual conditions of doing the tasks specified in the description of a qualification.
The person observing – usually the assessor – has limited influence on the conditions in which the observation is conducted, because they are provided by someone else (usually the employer). Observers, however, largely influence the processes of collecting and analysing information about the learning outcomes achieved by the candidate.
In various European countries, this method is most often used to identify or confirm selected learning outcomes. Frequently, the candidate's activities in the workplace or a part of his/her work are the subject of the observation, if such specific activities are important for a given qualification. This method can be used not only in the case of sectoral or craft qualifications, but also artistic and sports qualifications.
Observation in real-life conditions allows information to be collected on learning outcomes from all three categories: knowledge, skills and social competences. However, it is most accurate when it assesses learning outcomes that can be directly observed (skills). To identify other categories of learning outcomes, observation should be combined with other methods.
Observation in real-life conditions can take various forms. More information about different variations of observation can be found in the "Example techniques" section.
Validation stages in which the use of the method is recommended
Observation – and specifically different techniques of this method – can be used at all stages of validation.
At the identification stage, it serves to recognise the candidate's knowledge, skills and social competences. In the documentation stage, completed observation sheets (or other forms of recording the results of observations) can be evidence of the candidate's learning outcomes. Most often this method is used in the assessment stage.
This is confirmed by research conducted by Cedefop (an agency supporting the development and promotion of vocational education and lifelong learning in the European Union). According to data from 2016, three out of the 34 countries in the study used this method to identify learning outcomes. Observation was very rarely (1 country) selected in the documentation stage. In contrast, this method was used, often in combination with others, in 10 countries in the assessment stage. When used as an auxiliary method, only some learning outcomes are observed, mainly from the category of social competences and skills.
Scope of the learning outcomes that may be assessed using the method
Information on the scope of the learning outcomes that can be confirmed by observation in real-life conditions is for guidance only. It is based on an analysis of the required learning outcomes in the Polish Qualifications Framework (second stage generic descriptors typical for vocational education and training).
It should be remembered that the choice of a given method must always result from an analysis of the learning outcomes for a given qualification.
In a broad sense, observation consists of gathering information that can be used to confirm the achievement of a candidate's learning outcomes relevant to a given qualification and to analyse his/her behaviour, manner of communicating and performing tasks in the actual conditions of professional activities. Therefore, it is recommended for confirming skills and social competences, but can also be used to identify the scope and depth of a candidate's knowledge.
An appropriately long observation allows learning outcomes to be recognised relating to the tools and materials used in the workplace.
In the case of knowledge and an understanding of theories, principles, phenomena and processes as well as the organisation of work, this method can be used alone, but only at lower framework levels (PQF 1-3). Confirming knowledge from higher levels (PQF 4-8) using observation would be very time-consuming. It is therefore recommended either to combine it with other methods or to replace it with a theoretical test.
However, it is not recommended for confirming knowledge relating to organising work in the aspects of "methods and technologies" and "organisational solutions" at PQF levels 7 and 8.
Observation is useful in confirming all skills acquired at each PQF level. It does not require using other methods. However, the more complex the learning outcome (the higher the PQF level), the more time-consuming and expensive the observation can be.
Demonstrating some of the learning outcomes from the knowledge category requires the candidate to perform complex tasks and the ability to work in variable and unpredictable conditions. A sufficiently long observation can make it possible to observe key information that confirms (or does not) these skills, and confirming them in real-life conditions is much more accurate than using other methods.
Observation can be applied to all social competences at every PQF level in all its descriptive categories: following rules, cooperation, and responsibility.
This method can be used at the identification stage to recognise social competences. However, determining the acquisition of social competence learning outcomes during the assessment stage should be justified in writing or presented with the use of a scale.
Only longer term observation can determine whether a candidate has different types of social competences (they may not be immediately visible). Therefore, in some cases it is advisable to use other methods, especially those aimed at analysing statements (e.g. interviews) or documents.
Strengths and weaknesses of observation in real-life conditions
- the course of observation can be adjusted to the needs of the candidate applying for the qualification (e.g. by conducting it in the workplace of the candidate)
- it allows validation to be adapted to the candidate's needs (including for applying for validation) and limitations
- it is more accurate than other methods in recognising all categories of learning outcomes and can be applied at all stages of validation,
- the actions, behaviour and effects of the candidate's work can be observed
- it can be used together with other methods – in such cases, only some of the learning outcomes are confirmed using observation
- does not require special financial expenditure for its organisation and needed materials because it is conducted in the real-life conditions provided by an employer
- observation is time-consuming (especially if it is the only validation method used). It requires the observer to focus on one candidate and to collect and analyse detailed data on the subject
- this method is not suitable for performing validation for several persons at the same time
- the results of observations are particularly susceptible to distortion as a result of the “examiner's effect”, i.e. assessing a candidate on the basis of an overall impression or always awarding extreme or average grades
- the candidate’s awareness that he/she is being observed may affect his/her behaviour and, as a consequence, distort the results
Limitations of using the method
The use of observation in real-life conditions may be limited or impossible due to the need to comply with special occupational health and safety regulations or difficult conditions, such as working at heights or performing tasks relating to health services.
Observation can be completed only when the collected data enable the achievement of learning outcomes for a given qualification to be assessed. Meanwhile, performing certain activities may depend, for example, on the season of the year or require unusual conditions. This may limit the use of this method or require that it be combined with others.
A limitation may also be the length of time of the validation process in situations where observation is the only method used at the assessment stage (e.g. due to the fact that some activities in the workplace are rarely performed or take place at great intervals). This particularly applies to learning outcomes for knowledge and social competences (some of which can be observed only after over a longer period of time, while others can only be indirectly inferred on the basis of confirmed skills). This occurs regardless of how complex these learning outcomes are (their PQF level), but can be circumvented by adding a method (e.g. analysing evidence) or by replacing observation with another method.
Required human, organisational and material resources
The observer can be the validation counsellor at the identification stage or the assessor at the assessment stage. An external observer may also be involved to collect data – whose experience must be taken into account, because the level of detail of the information that can be obtained depends on this person. The quantity and quality of collected information will, in turn, be very important in the assessment stage. It should be remembered that the observer's role requires proper training in collecting data (making detailed notes, using tools), reliably confirming the learning outcomes and adopting a neutral attitude towards the candidate.
The observer should collect observations showing that the candidate can perform the given assignments. He/she cannot base the assessment on intuition or beliefs based on observations of the candidate's work in other conditions. This means that validation personnel must have completed proper training.
This is particularly important in the case of observation, because the results can be affected by a number of factors independent of the learning outcomes, e.g. the nature of the information being collected. If there are too many unusual situations (e.g. for a given workplace), the observer may focus on assessing how the candidate responds to them, not on the learning outcomes.
Depending on the purpose of using observation to validate learning outcomes, information about the candidate can be collected and documented using a variety of tools. At the stages of identification or documentation, the validation counsellor may make observation notes or use sound recording equipment or images (after obtaining the candidate's permission to make such recordings).
It is important that the data collected is very detailed and covers a wide range of observed activities of the candidate or the effects of his/her work. This helps to counteract the “examiner's effect”. One must ensure that the same assessment sheet is used to evaluate all candidates. It is also important that the assessment criteria in the sheet refer only to the learning outcomes being confirmed.
Because the observation takes place in the natural working environment of the candidate, no special conditions are needed to use this method.
Materials, which may be needed: paper and a pen to take notes; a stopwatch or watch for measuring the time of performing activities; voice recorder, video or photo camera to record the sound or images.
When observing activities that require compliance with specific occupational health and safety rules, it is advisable to use protective gear (e.g. caps, helmets, goggles, masks, ear protections, ear muffs, overalls, aprons, gloves, footwear protection).
Possibilities of combining observation in real-life conditions with other methods
Observation makes it possible to collect information regarding both the activity, the result of the work, behaviour, as well as the way the candidate communicates. Therefore, it should be used at the assessment stage with other methods suitable for checking those aspects of the candidate's work and related learning outcomes.
The factor that determines the applicability of observation is the convergence of conditions specified in the description of a given qualification with actual conditions, e.g. those prevailing at the candidate's place of work. For example, if the candidate works on machines other than those indicated in the description of the qualification, then other methods should be used to confirm the learning outcomes (e.g. simulation, analysis of evidence and statements, presentation, unstructured or structured interview).
According to Cedefop, 10 countries combine observation with other methods at the assessment stage, e.g. unstructured and structured debates, unstructured and structured interviews, analysis of evidence and statements, presentation and observation in simulated conditions.
It is also possible to supplement this method with the theoretical test (especially with the technique of oral examination).
A technique is a way of doing a particular task in a given method, used to collect and analyse data proving that a person has achieved the learning outcomes.
The information collected in this type of observation is used to confirm whether the candidate has achieved the learning outcomes relevant to the qualification. In order to gather specific information, the assessor controls the conditions of the observation, for example by discussing the tasks that will be assigned at work with the candidate's supervisor on the day of the observation. As a result, specific learning outcomes can be observed.
Strengths of controlled observation:
- allows complex learning outcomes to be confirmed during one observation;
- increases the certainty of confirming learning outcomes, regardless of unpredictable factors;
- quick to use, saves times during the assessment by allowing many learning outcomes to be confirmed in a short time.
Weaknesses of controlled observation:
- interference during the observed situation may affect the behaviour of the candidate, which could differ in a natural situation;
- it may be difficult to negotiate the tasks assigned to the candidate with the employer.
It is conducted without any interference - the assessor observes the situation, without prior arrangements with the candidate or the supervisor. This technique works best at the identification stage of validation.
Strengths of uncontrolled observation:
- this technique is used in a natural situation, so the behaviour and reactions of the candidate are credible;
- it can be used when observing the candidate's performance or the product of his/her work.
Weaknesses of uncontrolled observation:
- in using this technique, the assessor has no influence on the tasks the candidate will perform, therefore, the candidate will not always have the opportunity to demonstrate all the learning outcomes (therefore it is recommended that this technique be combined with other ones);
- due to its lower predictability, it is time-consuming;
- one observer (counsellor or assessor) is unable to identify or assess the learning outcomes of many candidates at the same time.
In addition to the above variants, there are many different variations of observation method. The table below shows the types of observations categorised by several key aspects.
Table 1. Variations of observation
ASPECT VARIATIONS OF OBSERVATION
The candidate's knowledge about being observed
- overt – when the candidate knows and agrees to be watched
- covert – when the candidate is not aware that he/she is being watched and that his/her learning outcomes are being confirmed
The degree of interference of the observer (e.g. assessor) with the environment
- participatory – when the observer becomes, for example, one of the candidate's colleagues or customers
- non-participatory – when the observer observes the candidate operating in his/her environment from the outside
Contact of the observer with the candidate
- indirect – when the effects of the candidate's work are subject to observation (e.g. documents, recordings, objects)
- direct – when the observer has contact with the candidate (he/she may disrupt the process of collecting information about the candidate, for example, by asking questions)
Method of collecting information about the candidate
- uncontrolled – conducted without systematising tools (e.g. when the goal is to identify learning outcomes)
Not all variations of observations can be used to validate learning outcomes. One of the basic assumptions of validation is the transparency of procedures and informing the candidate about the course of the process. This excludes the use of hidden observation.
In turn, the participation of an observer (e.g. an assessor) at some stages of the candidate's work could affect its course, disturb the naturalness of the conditions and the final result. There are also situations in which the use of participatory observation is limited by health requirements or specific safety rules. Therefore, the use of non-participatory observation is recommended for validation.
Examples of the use of observation in real-life conditions
Examples for using observation in real-life conditions are presented in the Database of Good Practices. Validation, ensuring the quality of validation and certification, especially waste management – hazardous waste transport qualification in Scotland – portfolio technique.
Dubiecka, A., Szaleniec, H. and Węziak, D. (2006). Efekt egzaminatora [Examiner effect]. Downloaded from: www.oke.krakow.pl/inf/filemgmt/visit.php?lid=1509
Cedefop, European Commission and ICF (2017). European inventory on validation of non-formal and informal learning – 2016 update. Synthesis report. Luxembourg: Publications Office.
Łoś, E. and Reszka, A. (2009). Metody nauczania stosowane w kształtowaniu kompetencji kluczowych. Matematyka. Podręcznik metodyki operacyjnej [Teaching methods used in shaping key competences. Mathematics. Manual of operational methodology]. Lublin: Innovatio Press Wydawnictwo Naukowe Wyższej Szkoły Ekonomii i Innowacji.
Frankfort-Nachmias, Ch. and Nachmias, D. (2001). Research methods in social sciences. Poznań: Zysk.
Szaleniec, H., Kondratek, B., Kulon, F., Pokropek, A., Skórska, P., Świst, K., Wołodźko, T. and Żółtak, M. (2015). Efekt egzaminatora w ocenianiu prac maturalnych z jz jra polskiego i matematyki [The examiner's effect in assessing the general secondary school completion (Matura) exams in Polish and mathematics]. Warsaw: Educational Research Institute Downloaded from: http://pwe.ibe.edu.pl/opisy/efektEgzaminatora/Efekt%20egzaminatora.pdf
Cedefop. European database on validation of non-formal and informal learning. http://www.cedefop.europa.eu/en/pu blications-and-resources/data-visualisations/european-database-on-validation-of-non-formal-and-informal-learning