Here is the write up of some research I undertook in 2008 when I was studying for my Level 5 qualification.
Does the level 2 numeracy test validly assess a student’s ability to operate at that level?
Executive summary
Level 2 numeracy students were tested in exam conditions, with questions referenced to the Core Curriculum. Students achieving the 50% pass mark were included in analysis. Answer papers were collated showing which questions students got right and incorrect responses. The reading age of each question was calculated using three methods. Scatter diagrams were used to compare readability with student success. Questions that students found difficult were investigated. A questionnaire, including an open question for respondents to make comments about the Level 2 test, was distributed to colleagues. Fifty-eight responses were received.
Analysis showed that it is possible to pass this Level 2 test with a very weak understanding of the core curriculum. A hypothetical student unable to answer number questions above Level 1, but correctly responding to measure, shape and data handling questions could score 63%. This shows that a student can pass the Level 2 test without understanding much of the Core Curriculum.
One major concern raised by respondents was readability of test questions. The correlation between readability of questions and student success at a particular question is not significant. Care is taken in test design to ensure readability scores are low. However for ESOL students recognising words such as “turbines” or “emissions” may be difficult. Illustrations would be helpful to give visual clues.
Analysis shows that students did have enough time to complete the test.
Some questions were ambiguous from a mathematical point of view.
The questionnaire shows a very high level of dissatisfaction with the test. Increasing the readability of questions was the most popular improvement, followed by replacing multi choice questions with single answer questions.
Thirty-nine people made negative comments about the test. There were 6 main themes to these comments.
| Issue |
Number of comments |
| Low pass mark/Passing does not guarantee competence |
16 |
| Readability |
11 |
| Multiple choice format |
5 |
| Big gap between L1 and L2 |
4 |
| Teaching to the test |
3 |
| Tests have recently got harder |
3 |
Comments show the Level 2 test is losing credibility amongst employers. It is vital that the decision makers currently working on the Functional Skills standards and qualifications get it right. There should be an enquiry into why the pass mark has dropped from 75% to 50%. A qualification will not be valued unless it indicates that the person holding the certificate has attained certain skills. The Level 2 test does not do this.
Readability is a massive issue with this test. It would be a shame to lose the real life context of questions, so one way forward would be, for questions to be illustrated with photographs that would help ESOL students to interpret the more unusual words.
It is the ease, convenience and lack of expense that attracts examiners to multi choice question papers. However, with modern technology it is now not difficult to design tests that require the examinee to enter their own answer. I recommend that the exam boards investigate the possibility of transferring from multi choice for at least half the test.
Respondents are concerned about the gap between Level 1 and 2. Problems arise when students who lack competence at Level1 pass a test. They struggle with Level 2 because they haven’t grasped the concepts they should be building on. Increasing the pass mark at Level 1 would solve this.
Teaching to the test is problematic. When tutors know that topics such as probability are never on the test, it is tempting not to cover these. I recommend that the Level 2 Test should cover the full breadth of the core curriculum.
There is evidence that tests have recently got harder. Rather than making the test more difficult with a low pass mark, I recommend that the test should revert to previous levels of difficulty, but with a higher pass mark. Employers would then be assured that students with the certificate are competent at this level of numeracy.
