Robust, reliable information

The LNAAT is an online adaptive tool primarily designed to provide robust and reliable information on the reading, writing, numeracy and vocabulary skills of adults.

This information informs the development of learning interventions that match learners’ needs and strengthen their literacy and numeracy skills. The LNAAT also allows learners to track their progress over time and enables educators and organisations to report on progress made by groups or cohorts of learners.

It provides:

• reading, writing, numeracy and vocabulary assessments linked to the Learning Pogressions
• hundreds of assessment questions, using New Zealand contexts relevant for adults
• adaptive (where the computer alters the difficulty of questions in response to learner answers) and non-adaptive assessments (online and printed)
• reports for learners, educators, organisations and the Tertiary Education Commission (TEC)
• a reliable infrastructure built and tested to IT industry standards.

Developed by a consortium in 2010

The LNAAT was developed for TEC in 2010 by the New Zealand Council for Educational Research (NZCER) who led a consortium involving NZCER, the Australian Council of Educational Research (ACER) and Fronde. The development process drew on the collective curriculum and assessment expertise of NZCER, ACER and TEC, and the ICT expertise of Fronde.

Assessment questions – a key component

At the heart of the LNAAT is a database of more than 2,000 reading, writing, numeracy and vocabulary questions. Each assessment question in the database is associated with a detailed description (or signature) that includes information about its links to the Learning Progressions, calibration values and usage history. The ability to add to and maintain this database over time is a central feature of the LNAAT.

The questions were developed using procedures employed by ACER’s professional test development teams. A key component of these procedures is “panelling”. Panelling is a team approach to reviewing assessment materials. It is a rigorous, robust quality-control mechanism, based on the importance of exposing material to multiple viewpoints.

The panelling process involved three to five expert test developers jointly reviewing all aspects of draft assessment material with a view to accepting, modifying or rejecting the material.

The questions asked of the material during the panelling process included:

• Is it relevant to New Zealand adults, especially those in the workplace? Is it interesting? Is it worthwhile? Is it of some importance?
• Is it coherent? Unambiguous? Clear? Is it at the right level in relation to the Learning Progressions?
• Does it breach any ethical, cultural or other sensitivities?
• How will this material stand up to public scrutiny (including project stakeholders and the wider community)?
• Is it likely to be biased, ie, is it likely to be easier or harder for certain subgroups for reasons other than differences in the ability being measured?
• Is the reading level at the appropriate level?

An extensive trialling process was used to ensure the robustness of the assessment questions, that they discriminate appropriately between learners, and that they function together to define the underlying dimensions (reading, writing or numeracy) that they are intended to assess.

To do this, each assessment question was trialled by about 150 learners. The trial process was also used to provide qualitative information about:

• the relevance of the materials for educators and learners, and
• the degree to which the materials appropriately engage adults from a range of social and cultural backgrounds working and learning in a range of contexts.

Scales that help show progress

The LNAAT measures achievement and progress using Rasch measurement scales that map the progression of competency in reading, writing, numeracy and vocabulary.

Using Rasch measurement scales means that assessment results for different learners, or for the same learner at different time points, can be compared regardless of which actual questions were used in any of the individual assessments.

As well as locating the achievement level or competency of different learners, Rasch measurement scales let us locate the knowledge and skill levels needed to correctly answer each different question in the LNAAT. The location of a question on the scale is sometimes thought of as the difficulty of the question. This means that a learner's location on the scale can be understood in terms of the different types of questions that are located at the same place on the scale.

Rasch measurement scales are known as equal-interval scales. Each unit on an equal interval scale represents the same amount of change. This makes them ideal for quantifying change.

Supporting educators and learners

The primary purpose of the LNAAT is to support educators and learners in their teaching and learning of reading, writing, numeracy and vocabulary.

It provides information to help “know the learner”. This information can then be used to provide the next learning step. This learning step is made in relation to what the learner currently knows and what the learner needs to know.

While the LNAAT provides a strong starting point for understanding the literacy, numeracy and vocabulary skills of learners, it may be necessary to use other assessment tools or techniques to probe further into the literacy and numeracy skills of some learners.