Before an assessment can tell us what children know and can do, there is another important question to answer: do children understand the assessment itself and is the content being assessed class/grade appropriate?  

This question is at the heart of cognitive testing currently being undertaken as implementation of PlayMatters 3.0 kicks off in Ethiopia and Uganda. 

During the exercise, trained assessment teams work individually with Grade 4 learners, observing how they respond to questions and asking carefully structured follow-up questions. Rather than focusing only on whether an answer is correct, assessors seek to understand how learners interpreted the question, whether the instructions were clear, which words or phrases were difficult, and how they arrived at their response. 

The process is an important step in preparing the tools that will underpin measurement and learning during PlayMatters 3.0. 

Testing the questions before measuring learning 

PlayMatters uses the HALDO assessment tool to examine learning across literacy, numeracy and social-emotional learning (SEL). Before the tool is piloted in the targeted local languages and locations,  and subsequently used for measurement, itundergoes contextualization, translation and technical review. 

Cognitive testing provides an additional layer of quality assurance. 

The process examines whether learners understand and interpret assessment items as intended and helps identify potential challenges related to comprehension, translation, cultural appropriateness and administration that could affect the validity and reliability of the tools.  

During cognitive interviews, assessors use questions such as: What is this question asking you to do? What does this word or phrase mean to you? Was there anything that was difficult to understand? How did you decide on your answer? 

They also ask whether the situations described in questions are familiar to learners and whether there might be a better way of expressing particular words or phrases in the languages they use.  

These conversations allow the assessment team to look beyond children's responses and understand how children interpret the assessment itself. 

Listening to learners to strengthen measurement 

An important part of cognitive testing is creating space for learners themselves, as well as teachers, to provide feedback on the assessment. 

Learners are asked which questions they found easy or difficult to understand, whether they encountered unfamiliar words, whether instructions were clear and whether examples reflected situations they recognized. They can also suggest how questions might be expressed more clearly.  

This feedback matters because an assessment question can only generate meaningful information if learners understand what it is asking. 

This becomes particularly important when assessment tools are being adapted across different languages and contexts. Translation involves more than replacing one word with another; concepts, examples and expressions must also make sense to the children who encounter them. 

In Uganda, for example, the cognitive testing exercise involved 180 Grade 4 learners and their teachers, covered different language contexts including Aringa, Madi, Lugbara, Acholi, Runyankore, Kiswahili and Juba Arabic.  

In Ethiopia, the exercise was conducted in all the PlayMatters targeted regions of Oromia, Tigray, Somali, Benishangul-Gumuz and Gambella. In each region, two schools were covered, reaching a total 10 schools. In each school, 2 teachers and 10 learners were engaged in the exercise. Overall, a total of 20 teachers and 100 children participated.  

Overall, the cognitive assessment process provides an opportunity to identify issues early and strengthen the assessment tools before they are used more widely. 

From cognitive testing to stronger assessment tools 

What happens after these conversations is equally important. 

Assessment teams document questions or instructions learners find confusing, unfamiliar words and phrases, translation issues, requests for clarification and other signs of uncertainty. Findings are reviewed to identify recurring patterns across teacher, learners, schools and language groups.  

An assessment item may then be considered for revision where, for example, learners consistently misunderstood it, wording is ambiguous, a translation does not convey the intended meaning, or an example is unfamiliar within the local context. Importantly, decisions to revise items are based on evidence from teachers’ feedback, learner responses, observations and feedback.  

The findings from cognitive testing are consolidated into a report and item revision matrix, informing revisions to the HALDO tools before pilot testing.  

Building the evidence foundation for PlayMatters 3.0 

Cognitive testing may be one of the less visible parts of starting a new program, but it plays an important role in establishing the foundation for credible measurement. 

For PlayMatters 3.0, the work underway in Ethiopia and Uganda forms part of the preparations for establishing baseline measurement and the broader monitoring, evaluation and learning approach that will enable the project to understand change over time. 

The exercise is therefore about more than refining individual questions. It is about strengthening confidence that the tools eventually used to measure literacy, numeracy and social-emotional learning are clear, appropriate and capable of generating meaningful information about children's learning. 

As PlayMatters 3.0 moves into implementation, evidence generated through monitoring, evaluation, research and learning will be important for understanding what is working, informing program adaptation, and contributing to wider learning about Learning through Play. 

And that evidence begins with getting the measurement right. 

Before asking what has changed, PlayMatters is first making sure it can measure that change well.