Analysis of evidence and statements
Characteristics
The analysis of evidence and statements involves examining the documents and products of a given person in terms of how they provide the evidence of a candidate's achievement of selected learning outcomes.
Table 1. Examples of evidence and statements that may be analysed
Sample evidence | Sample statements* |
---|---|
|
|
* Statements of a given person (or of persons from their communities) about his/her learning outcomes; for this reason, they should never be the only basis for a decision to award qualifications |
The evidence and statements being analysed should be:
- relevant to the learning outcomes that will be confirmed – this means that activities enabling evidence and statements to be generated about the candidate should refer to specific knowledge, skills and social competences, and if the aim of the analysis is to assess the learning outcomes of the qualification – they should prove that the specified assessment criteria are fulfilled;
- authentic – they should relate to or be produced by the candidate; if the evidence provided was made together with others, the elements (or scope) produced by the candidate should be clearly indicated;
- sufficient – the evidence and statements should relate to all learning outcomes selected for analysis; at the same time, they should provide enough information to determine the candidate's knowledge, skills and social competences;
- current – in principle, the evidence and statements should allow the candidate to demonstrate his/her current ability to undertake the activities described in the learning outcomes, and as a consequence, should – if possible – be recently acquired.
In order to be able to analyse the evidence and statements in the assessment stage, the awarding body should indicate:
- the types of evidence that are acceptable and the forms in which they should be presented,
- additional criteria to be met (if required),
- the rules of assessment (the basis for issuing the decision).
The purpose of the analysis of evidence and statements may vary depending on the stage of validation in which it occurs:
- At the identification stage, it can help the candidate or the validation counsellor determine the learning outcomes already achieved by the candidate.
- At the documentation stage, it allows the candidate or counsellor to select the relevant evidence in a format that enables it to be presented to a third party.
- At the assessment stage, it is used by the validation assessor to determine whether the achieved learning outcomes have been sufficiently confirmed.
One of the most important skills of an assessor using this method is the proper interpretation and evaluation of evidence and statements, which can take many forms, e.g. documents, audio-visual material, photos, work samples.
Only those items indicating that the candidate possesses the learning outcomes to be evaluated should be taken into account. The same evidence (statement) can be used for several qualifications. It can also relate to various learning outcomes.
Example
The candidate can include a video recording in his/her portfolio. If it serves as evidence documenting the process of creating a piece of work, the assessor should not be judging its aesthetic values.
However, if the ability to make video recordings as part of an artistic qualification is being assessed, the content of the film is potentially less important than its aesthetic qualities.
The validation stages in which the use of the method is recommended
The method of analysing evidence and statements can be used in each of the three validation stages.
Identification stage
In the process of identifying knowledge, skills and social competences, it may turn out that the candidate has work samples, certificates of completed courses, references and other documents testifying – indirectly or directly – to his/her learning outcomes. Analysing this evidence can help in determining:
- the knowledge, skills and social competences the candidate has achieved,
- whether these learning outcomes meet the requirements specified in the qualification,
- other qualifications that the candidate may be able to attain.
The analysis of evidence and statements in identifying learning outcomes is made by the candidate or the validation counsellor. It plays an auxiliary role. This means, among other things, that the analysis can be conducted even if it is not included in further stages of validation, as well as when the candidate is not seeking to have a qualification awarded.
Documentation stage
At the documentation stage, evidence is gathered to prove that the candidate has achieved the required learning outcomes, including the knowledge, skills and social competences distinguished at the identification stage. The end result should be the preparation of evidence so that it can be presented to a third party (e.g. evaluated during assessment or presented to a potential employer in a portfolio).
Analysing the evidence and statements prepared by the candidate at this stage should result in the selection of those that:
- confirm the learning outcomes required for the qualification being sought by the candidate,
- meet the requirements regarding the form and criteria specified by the awarding body.
The analysis at this stage is done by the candidate, however, the assistance of a validation counsellor with this process is indicated.
Assessment stage
The validation assessor analyses the evidence and statements accepted for assessment in relation to the learning outcomes of the qualification. His/her task is to determine whether the submitted documents and products (e.g. work samples) show that the candidate has the required knowledge, skills and social competences.
Scope of the learning outcomes that may be confirmed using the method
The analysis of evidence and statements can be conducted with reference to the knowledge, skills and social competences of all levels of the Polish Qualifications Framework (PQF).
Knowledge: Some of the evidence analysed using this method will be the result of activities or the documentation of a process. At lower PQF levels (1-4), it is often impossible to directly demonstrate knowledge. In such cases, an additional method may be used (e.g. interview, test) or statements certified by third parties (e.g. by employers) are included.
At the same time, the preparation of a portfolio or writing a self-reflective essay allows the depth of understanding to be shown, which is particularly useful in relation to complex issues and qualifications at higher PQF levels (5–8).
Skills: Depending on the piece of evidence, conclusions can be made about skills from all PQF levels.
Social competence: The method of analysing evidence and statements can be particularly useful for confirming certain social competences. The assessment process can be based on both evidence (e.g. observation result, recording of behaviour, description of volunteer career), as well as on statements (e.g. opinion of other people, reflective essay). It should be remembered that statements should not be the only basis for a positive validation result.
Strengths and weaknesses of the analysis of evidence and statements
Strengths | Weaknesses |
---|---|
|
|
Limitations of using the method
Awarding bodies may be limited in using the analysis of evidence and statements because of a concern about the reliability of validation. The assessor considers only documents and products presented by the candidate. In such a situation, there is a risk that the assessment will be incomplete, especially if some of the evidence and statements are rejected (e.g. because they did not meet the formal requirements). The authenticity of the documents and products presented by the candidate may also be questioned.
In both cases, one of the solutions may be the involvement of a validation counsellor in the process of preparing evidence and statements. This is also recommended because gathering sufficient proof and statements about acquiring learning outcomes requires good knowledge of the qualification’s description. Although the need to involve additional human resources may in itself be discouraging for the awarding body, the counsellor's participation may also bring a number of benefits.
Box 1. Advantages of having a validation counsellor participate in the process of analysing evidence and statements
The participation of a validation counsellor in the process of analysing evidence and statements has the following advantages:
- the perspective of obtaining support may encourage people who would not have applied for validation without it;
- the risk of rejection of the evidence and statements presented by the candidate is reduced, e.g. in situations when the documentation does not comply with the formal requirements or is incomplete, because the counsellor assists with the selection and can indicate what is lacking before proceeding to the assessment stage;
- the counsellor supports the candidate in collecting the evidence to prove that the required learning outcomes are achieved and in matching the portfolio to the requirements included in the qualification, thereby reducing the risk of an incorrect assessment of the material by the assessor;
- the risk of the candidate providing false evidence and statements is reduced.
The difficulty in using the analysis of evidence and statements may be its limitation in using the techniques and tools for generating evidence. For example, in order to be able to analyse the results of observation in the workplace, the assessor first needs to communicate with the employer to conduct the observation. This limitation occurs at the stage of documenting learning outcomes.
A factor hindering the recommendation of this method by entities describing qualifications may be the low level of trust in its accuracy. On the other hand, Cedefop (an agency supporting the development and promotion of vocational education and lifelong learning in the European Union) data from 2016 shows that the portfolio – understood as a collection of evidence – was used in 27 out of 34 European countries, and the analysis of statements – in 24 of those countries included in the study. In addition, for greater validation accuracy, the analysis of evidence and statements can be combined with other methods. For example, in France, the basis for validation is the assessment of a dossier prepared by the candidate which, in justified cases, is supplemented by an interview with the examination board.
Required human, organisational and material resources
In order to be able to apply the method of analysing evidence and statements, the participation of a properly trained assessor is necessary. This person should have the knowledge and skills needed to correctly interpret the evidence and statements and to draw conclusions about the candidate's learning outcomes. If the method is to be combined with other methods / techniques, the assessor should know how to use them.
Additionally, the participation of a properly trained validation counsellor is recommended. This person should have the knowledge and skills to:
- indicate the types of documents and products that can prove the knowledge, skills and competences of the candidate;
- draw conclusions based on evidence and statements about the candidate's learning outcomes;
- correctly interpret the presented evidence and statements;
- present a list of evidence and statements to the candidate that the awarding body allows and help in their selection and preparation.
In addition, it is advisable to appoint a person(s) responsible for processing the candidate’s application.
The following may be useful in analysing the evidence and statements:
- information materials for the candidate about validation, including the requirements of awarding bodies regarding admissible evidence and statements,
- required forms (e.g. application, portfolio template),
- technical means necessary to review the submitted documents and products (e.g. a CD / DVD player, a computer),
- material resources in case additional methods are used.
The costs of using this method can consist of:
- preparation and improvement of the validation scenario taking into account the analysis of evidence and statements;
- provision of human resources – this will depend on the number and type of personnel needed for the process (is a validation counsellor, person receiving the application, providing information, etc. needed in addition to the assessor?) or the number of people participating in the process (e.g. several assessors, several counsellors), as well as (indirectly) the number of candidates;
- providing the technical means (e.g. a CD / DVD player to view a recording, a computer to read an ePortfolio);
- storage for the evidence and statements – depending on the internal arrangements of the awarding body;
- additional costs if the analysis of evidence and statements is to be combined with other methods.
Possibilities of combining the analysis of evidence and statements with other methods
The analysis of evidence and statements can be supplemented by an unstructured interview or structured interview – this is used, for example, in France, the Netherlands and Portugal. In case of doubts, the counsellor or assessor is able to directly interview the candidate. These methods can be combined at all stages of validation.
In addition, the analysis of evidence and statements can be combined with a structured i unstructured debate. At the assessment stage, the analysis can be combined with a theoretical test, observation in simulated conditions (practical tasks), observation in real life conditions or a presentation – depending on which learning outcomes are to be confirmed. The results of a skills audit (conducted at the stage of identifying and documenting learning outcomes) can also be analysed.
Example techniques
A technique is a way of doing a particular task in a given method, used to collect and analyse data proving that a person has achieved the learning outcomes. For example, a portfolio is one way of collecting and analysing data to prove that a person has achieved the learning outcomes.
Portfolio
The portfolio can be used:
- in the learning process – as a method of education that allows a person to evaluate past achievements and plan and then implement further educational, professional, personal development,
- as a collection of a person’s products.
In both cases, the structure, choice of content, form of evidence confirming the achievement of learning outcomes and the way of assessing the portfolio will be different. Because validation does not take into account the educational process, the portfolio in the Catalogue of Validation Methods is understood as a collection of a candidate’s products to be analysed as part of validation.
Before a candidate starts preparing the portfolio, the awarding body should specify:
- the form (paper or electronic form) and structure of the portfolio;
- the standards for the acceptability of evidence confirming the achieved learning outcomes, including their form, quality and sufficiency;
- standards for assessing both the individual items of evidence and the portfolio as a whole.
The structure of a portfolio may differ due to its purpose, group of recipients and even type of qualification. Therefore, there is no single, universal portfolio design. The following is one of the possibilities.
Box 2. Sample portfolio structure
The portfolio may contain:
- title page
- table of contents
- a declaration that the information and documents confirming the presented facts in the portfolio are true,
- contact details,
- candidate’s details,
- list of documents,
- evidence and statements of having achieved the learning outcomes.
It is advisable that the candidate receive support in the process of preparing the portfolio (e.g. from a counsellor).
Persons who prepare evidence and statements on learning outcomes (both the candidate and the persons supporting him/her, e.g. a counsellor) must:
- know the standards under which validation will take place (e.g. description of the qualification);
- know the criteria of the awarding body for admitting evidence;
- use appropriate tools and technologies if the portfolio is electronic.
Strengths of the portfolio |
---|
|
Weaknesses of the portfolio |
---|
|
Evaluation of a piece of work
A special case of the evidence analysis is the evaluation of a piece of work. It consists of a judgement being issued about whether or not the candidate has achieved the required learning outcomes based on submitting a product or piece of work, made as an assignment and in accordance with an assessor's instructions. This means that as a rule, the candidate cannot present the best sample of his/her work for the assessment, which is possible in the case of a portfolio.
The candidate does not have to prove his/her skills under controlled conditions (in an examination room) before the assessor. The achievement of the learning outcomes is determined indirectly, based on fulfilling the criteria for producing a given product or piece of work. An example is the evaluation of a work of art.
Examples of the use of evidence analysis and statements
The Database of Good Practices. Validation, Ensuring the Quality Of Validation and Certification provides examples of using the analysis of evidence and statements.
Among others, the following qualifications use this method:
- hairdresser in France – portfolio technique,
- waste management: hazardous waste transport in Scotland – portfolio technique,
- administrative assistant in Italy – portfolio technique.
Below is an example developed on the basis of the materials of the project „Validation of skills and knowledge for strengthening the positions of low-qualified employees in the labour market – ValidAid”
Name |
An international project aimed at validating the competences of low-skilled persons in the IT, finance and trade sectors The project was conducted from October 2010 to September 2013. |
---|---|
Country | Austria, Bulgaria, France, Iceland, Lithuania and Portugal |
Context of its development | The project was implemented by a consortium of 7 institutions from 6 countries, including universities, research institutes, counselling and advisory bodies and training institutions. |
Name of the qualification | IT, finance and trade salesperson |
Brief description of the method used |
The aim of the project was to prepare a “Certification Portfolio” that will be used in the validation process to confirm the learning outcomes needed by a salesperson in the IT, finance and trade sectors. The content of the portfolio includes:
The preparation of the portfolio is aimed at presenting the candidate's ability to perform a range of professional tasks in the field of sales. Thus, the candidate must reflect and assess him/herself before proceeding with validation. The portfolio is assessed by a suitably trained assessor. |
Used individually or with the assistance of a counsellor | At the identification and documentation stages, the portfolio can be prepared independently by the candidate. However, it is recommended that a counsellor assist in this process and, in addition to the candidate's self-reflection, his/her superior should be consulted regarding the content. |
Factors underlying the choice of the method | Preparation by the candidate of the Certification Portfolio is one of the stages of the validation process. The portfolio forms the basis for an interview with the assessor. |
The procedure |
The candidate should use a computer to develop the portfolio. In order to facilitate this task and to ensure that no important information is left out, the counsellor should provide the candidate with a portfolio template. As a result, all portfolios will have the same structure, which will facilitate the assessment process. The counsellor monitors the preparation of the portfolio and supports the candidate throughout the entire process. Job descriptions are designed to provide a clear picture of the candidate's tasks in the workplace (retail store) and the skills that are necessary for their implementation. It is therefore important to accurately describe his/her work, taking into account all relevant details. In order to obtain a comprehensive picture of the candidate's abilities, in addition to defining vocational tasks, the work process must be described from several different perspectives, such as:
It is also worth describing how candidates perform their tasks in different contexts, for example are they able to:
A portfolio thus prepared allows candidates to perform a comprehensive self-reflection on their knowledge, skills and social competences, as well as to present them to third parties, including the assessor. Then, the assessor interviews the candidate, based on the content of the portfolio and the candidate’s self-assessment. The purpose of this meeting is to determine which learning outcomes should be assessed and how. The methods are: observation in simulated conditions (case study, role play), analysis of work samples, observation in real life conditions (in the workplace), theoretical test and interview. The assessor issues a decision, taking into account the results of the methods used and compares them with the self-reflection of the candidate prepared in the portfolio development process. |
Results of the validation process |
If the results of the assessment coincide with the candidate's self-reflection, the assessment is considered successful. If there are any discrepancies, the assessor determines their scope. Then he/she either states that the candidate does not have the required learning outcomes, or allows him/her to perform additional tasks. |
Organisational and material resources |
Material resources To prepare the portfolio, a computer and a form provided by the counsellor are needed. Human resources Counsellor The main tasks of the counsellor are:
The role of the counsellor should first be proposed to persons with competences in the field of counselling and advising. The training of counsellors should include conducting validation with an emphasis on preparing a portfolio. In addition, counsellors should be involved in developing and improving validation tools. Assessor The assessor should be impartial and have no previous contact with persons who apply for validation. The assessor may be an external trainer or an expert employed in a trade union, academic organisation, industry association, etc. The assessor's role may be performed by, among others, vocational teachers and specialists in the field of human resources management. Such a person must:
Assisting assessors When one of the validation methods is observation in the candidate's workplace, the participation of persons acting on behalf of the assessor may be required. In some circumstances, the role of such assistant assessors may be given to the direct superior of the candidate. |
Limitations of use | The described procedure of preparing the portfolio was developed for the needs of the project and addressed to a specific group of people (salespersons in the IT, finance and trade sectors). |
Notes | The portfolio (analysis of evidence and statements) was only one of the methods used in validation within the project. Others include observation and tests (including case studies). |
Bibliography
Bacia, E. (2014). Validation of learning outcomes acquired outside the formal education system as a new policy challenge for lifelong learning. Warsaw: Educational Research Institute Downloaded from http://kwalifikacje.edu.pl/wp-content/uploads/publikacje/PDF/Walidacja.pdf
Barrett (2010). Balancing the Two Faces of ePortfolios. Educação, Formação & Tecnologias, 3 (1), 6-14. Downloaded from http://eft.educom.pt/index.php/eft/article/viewFile/161/102
Białek, M., Brudny, M., Matysiak, W. and Wykrotka, M. (2014). Preliminary guidelines for the identification and documentation of learning outcomes, including methods and tools. Kraków: Provincial Labour Office [unpublished material].
Butler, SM and McMunn, N.D. (2006). A Teacher's Guide to Classroom Assessment. Understanding and Using Assessment to Improve Student Learning. San Francisco: Jossey-Bass.
Duvekot, R. and Schuur, K. (eds.). (2014). Building Personalized Learning. A handbook for creating a common theoretical background on concepts regarding personalized learning. Vught: EC_VPL.
EuroPortfolio (2015). Implementation of the e-portfolio. Teacher Guide. Downloaded from: http://www.eportfolio.eu/wiki/index.php/Wdro%C5%BCenie_e-portfolio._Przewodnik_dla_nauczyciela
Finger, G. and Jamieson-Proctor, R. (2009). Assessment Issues and New Technologies.: ePortfolio Possibilities. In: C. Wyatt-Smith, J.J. Cumming (ed.), Educational Assessment in the 21st Century. Springer.
Gmaj, I., Grzeszczak, J., Leyk, A., Pierwieniecka, R., Sławiński, S., Tauber, M. and Walicka, S. (2016). Validation – new opportunities for acquiring qualifications. Warsaw: Educational Research Institute
Gmaj, I., Grzeszczak, J., Leyk, A., Pierwieniecka, R., Tauber, M. and Walicka, S. (2017). Foreign validation systems. Overview of solutions. Warsaw: Educational Research Institute
Joostenten Brinke, D., Sluijsmans, D. M. and Jochems, W. M. (2010). Assessors' approaches to portfolio assessment. Assessment of Prior Learning procedures. Assessment & Evaluation in Higher Education, 35 (1), 55-70. Downloaded from: http://dspace.ou.nl/bitstream/1820/3073/1/Joosten-ten%20Brinke%20et%20al_2010_AEHE.pdf
Kubiszyn, T. and Borich, G. (2010). Educational Testing and Measurement. Classroom Application and Practice. New York: Wiley.
Lynch, B. and Shaw, P. (2005). Portfolios, power and ethics. TESOL Quarterly, 39 (2), 263-297. Downloaded from: http://onlinelibrary.wiley.com/doi/10.2307/3588311/epdf
Muchacka, B., Kaleta-Witusiak, M. and Walasek-Jarosz, B. (2013). Self-reflection and analysis of own work. Casebook with tips for practitioners. Kielce: Staropolska Szkoła Wyższa w Kielcach.
NVR (2010). E-tools and Validation – areas of significance and dilemmas. Downloaded from: http://biblioteka-krk.ibe.edu.pl/opac_css/doc_num.php?explnum_id=413
Rumsey, D. (1994). Assessment practical guide. Canberra: Australian Government Publishing Service. Downloaded from: http://www.voced.edu.au/content/ngv%3A39513
Strivens, J., Waranet Strivens, R., Wad, R., Guàrdia, L., Maina, M., Barberà, E., Alsina, I. and Wolf, B. (2014). E-Portfolio competency recognition and accreditation framework [working document]. Downloaded from: https://docs.google.com/document/d/1OEIHt9u1H6WtjHDc1gZjOj58h4HsQtm4mRqemsj1cjs/edit
Szymczak, J. (2015). Model of reflexivity development in (future) early school education teachers – assumptions, essence, design motifs and giving it a practical dimension. Problems of Early Education, 1 (28), 140-153.
ValidAid (2010). Final version of methods and tools for evaluation of the low-qualified employees who will take part in the validation procedures in sector "Trade", "IT and Communications" sector and "Finance". Work Package, 6, 23 -24. Downloaded from: http://www.validaid.eu/images/validaidkit/methodstools.pdf
Websites:
Cedefop. European Inventory on validation of non-formal and informal learning [database]. Downloaded from http://www.cedefop.europa.eu/en/events-and-projects/projects/validation-non-formal-and-informal-learning/european-inventory
Europortfolio portal: http://www.eportfolio.eu/
Reflection for learning: https://sites.google.com/site/reflection4learning/Home