assessANDfeedback

When the teacher finds each learners firm learning foundations on a topic and uses that data to decide where to start teaching that topic, the teacher can be confident that the learners will retain new teaching because

• the teacher is teaching on firm learning foundations,

• the app schedules a retrieval practice programme for each bite (layer) of new learning and

• that when feedback needs to be given, it has the best chance of success, because the teacher and learner are entering into a feedback-dialogue and the app adjusts (reduces) the interval between asks of the similar questions from a layer in the retrieval practice schedule, so that the learner is most likely to remember and be able to apply the recent feedback.

A traditional pre assess process which asks all the learners the same questions at the same time - whether they answer within a test, selecting from multiple choice options or using mini white boards - makes many learners uncomfortable. “Our cohort” are often especially uncomfortable, so they often undermine its effectiveness by quietly not engaging, copying or otherwise avoiding answering, perhaps by claiming they know everything or nothing or perhaps by asking - when will I use this in life?

That’s why timely practice ensures that

• pre assess and retrieval practice are "low stakes". Learners, over time, learn their assessment is formative (to help their learning) not summative (to judge them against a standard). We need to teach our cohort - who no matter what the standard was, have learned that they won’t meet that standard - that when they use timely practice we are in the business of assessing them to help them learn better. (We probably won’t be able to avoid summative testing entirely.)

• it may take some time for learners to begin to feel comfortable, what the teacher can do to help is keep sticking with the message: “I’m finding out what will be easiest and most valuable to teach next” and “I’m finding out how well you can recall recent learning, so that you can recall it more easily in the future” and encouraging learners to “remember to use a question mark, if you need to”.

We don’t ask each learner a question from every layer in every topic because

• it would take too long: we initially rule out some key layers that we consider too easy or too hard (based on the Level for Learner the teacher inputs into the app) and because

• it would be too traumatic for learners. Hence the teacher can say to the class “Everybody will get some too easy, some OK and some too hard questions in their learned and remembered section, this is to help the app find out for the teacher, what to teach you soon.”

Instead we ask

• questions on a few key layers of a topic, and

• any that are answered 100% correctly we ask an additional verify question.

This gives us a broad brush stroke picture of the learner’s skills and learning gaps. We then ask

• responsively questions from easier or harder key layers, and

• any that are answered 100% correctly we ask an additional verify question,

• then we sometimes ask questions from some scaffold layers and

• any that are answered 100% correctly we ask an additional verify question.

This trial and improvement process means we can be fairly confident we are finding each learners' firm learning foundations from each topic.

It’s easy to teach too much and/or too hard and if we do

• the learners will forget the new teaching and will inevitably become progressively more despondent about their ability to learn,

• the retrieval practice system that timely practice uses would require more feedback capacity than the teacher has available and so only learning that the learner requires no feedback for would be retained 😢

The pre assess process is set up so that the teacher can create and set

• an assessment assignment for one whole lesson or

• shorter assessment assignments to be completed in say 10 to 15 minutes of a fortnight of consecutive lessons,

After this, the timely practice app can show the teacher firm foundations for the first few topics.

The app is intended to hold the teacher back from teaching too much and/or too hard, meaning

• the learners generally retain their learning until the next lesson, and when this happens regularly learners become progressively more motivated,

• very little feedback is required to ensure the retrieval practice system that timely practice uses ensures learners retain new learning for the long term

Preparing for lessons requiring a warm up assignment,

1. Create a P&P (planning and preparation) session - the type will be warmup and the date will be the date of the lesson when timely practice pre assess will happen

2. Open that P&P session and tap Continue to move past the Assess t.p. section (this is the first assignment, so there are none to assess yet).

3. Change the nominal number of questions (if you want a longer or shorter assignment).

4. Leave both retrieval practice and pre assess assignments ticked (this is good practice for 2 reasons which are explained later)

5. Tap the Create button, tap Continue

6. Select the type of Download you prefer: duplex concatenated (all learners assignments printed double sided with additional blank pages to easily separate them), simplex (single sided) or each learner separately. Click the appropriate green arrow(s).

7. The assignments will be found in your downloads folder.

1. Quick start: the school could allocate

   • one lesson to do a pre-assess only assignment as a test (say 20 -30 questions)

   • for the next lesson create a slightly shorter pre-assess only assignment (say 15 to 20 questions), this will primarily verify the data collected in the first lesson. The teacher can begin this lesson with learner training on what to expect from their timely practice assignment this lesson: a second verify question on answers they got correct and an easier question on topics which they found the questions too hard. The teacher should tell the learners that they tricked them, yesterday’s test wasn’t a test it was doing what it said at the top of the assignment - finding out what they already learned and remembered. At the end of the lesson the teacher can explain about teaching, learning and memory and a little about what and how they will learn next lesson.

   • for the third lesson the teacher could prepare to teach the class using timely practice. Of course for any learners absent from either of first 2 lessons would have insufficient data for retrieval practice to follow up the 3rd lesson’s teaching in the 4th lesson.

2. Slower start: the school could allocate 10 to 15 minutes of lessons for pre assess (replacing lesson warm up) over a fortnight or longer. In the rest of these lessons the teacher will teach using the current scheme of learning. (To save outside of lesson assessment time, the teacher could do some of the assessment of the last lesson’s assignments whilst the class is doing their new assignment this is only OK because the teacher doesn’t need the pre assess to be efficient, because the warm up lasts for a fortnight or longer). This strategy is particularly useful for 2 weeks revision before a school scheduled end of term/year test.

3. A small group preparing for intervention tutoring could be pre assessed in 1 lesson: with the first 8 question assignment created prior to the lesson, and then 3 episodes of assess + but no feedback + create next + do next PDF assignment, within the lesson. Provided the group is small e.g. 3 learners and the teacher/tutor has access to a tablet device and printing facilities, the teacher should be able to assess, create, print and supervise the learners.

Pre assess will feel less threatening than traditional tests as the learners will become aware of

• questions they can do returning, and

• questions they couldn’t do being dropped.

We are just refining our auto pre assess facility so that for each topic theme

• 4 priority topics are pre-assessed over a range of levels (so most learners will see some easy, some OK and some too hard questions on these topics) and

• the remaining topics are assessed in a more responsive way (so most learners will see just one level in the first assignment on this topic)

in both cases the app will ensure it continues to find the sweet spot (between too easy and too hard) where teaching should begin.

How long the pre assess takes will depend on the attainment of the class and the intensity at which the school wants to do the pre assess process.

Our new auto pre assess makes it much easier to avoid pre-assess of what has been taught within the last 4 weeks.

A learner’s column in the Plan Teaching topic dashboard

• which has some mid grey and darker blue-grey squares - indicates that pre assess is not complete;

• which only shows light grey squares - indicates that the app does not intend to pre assess the topic (judging all the layers either too easy or too hard);

• which shows a mix of white and light grey squares - indicates the app thinks the learner could learn layers from the topic, but has not learned any yet;

• the (often fuzzy) line between the shades of blue (perhaps mixed with light grey) in the lower part of the column and the white (usually mixed in with light grey) in the upper part of the column is the (often fuzzy) “goldilocks” point where teaching should begin.

Preparing for lessons requiring a warm up assignment,

1. Create a P&P (planning and preparation) session - the type will be warmup and the date will be the date of the lesson when timely practice pre assess will happen

2. Open that P&P session and go to the Assess t.p. section and select a learners name

3. Assess each question of the learners' completed/partially completed assignment using

   • tick: fully correct

   • BELL (BEst Learned Later): almost any other attempt e.g. partially correct, incorrect, missed out, written a ? etc

   • reset: learner ran out of time or you think the learner didn’t see the question (e.g, small question sandwiched between larger questions, you just can’t read the learners handwriting e.g. they orginally wrote 5 they’ve overwritten with 3, or was it the other way around?)

   • once all the questions in a learners assignment are assessed, tap the floppy disk symbol and assess the next learner’s assignment

4. Once all completed/partially completed assignments have been assessed, make a note of who has an outstanding assignment (and check you have the paper copy) and tap the Continue button

5. If you need to, change any learners Pace for Practice, (this will ensure most students complete their assignment within a minute or two of each other). Each learner’s Pace for Practice slider, is found within each learner’s data within the Classes/Learners' tab. Then, return to the Planning & Preparation Tab, for the next lesson and return to the Create t.p. section.

6. Change the nominal number of questions (if you want a longer or shorter assignment), and tap to un tick the names of the learners who have an outstanding assignment.

7. Leave both retrieval practice and pre assess assignments ticked (this is good practice for 2 reasons which are explained later)

8. Tap the Create button, tap Continue

9. Select the type of Download you prefer: duplex concatenated (all learners assignments printed double sided with additional blank pages to easily separate them), simplex (single sided) or each learner separately. Click the appropriate green arrow(s).

10. The assignments will be found in your downloads folder.

The decision between feedback (FOB or FOA) and BELL (BEst Learned Later) can be complex, as we are deciding on the best use of the teacher’s and learner’s lesson time.

If the learner made errors/didn’t answer a recently taught topic, usually we will use the correct feedback symbol.

However if the learner made errors/didn’t answer a topic they’ve had feedback on a few times already, then it’s useful to decide “shall I BELL this layer?”

The question isn’t about just whether the teacher and learner can use the feedback process to “get the learner to be able to ask similar questions in the layer” but also about balancing the cost in lesson time allocated and the cost to the learner’s limited supply of motivation.

Questions which might help decide are

• does the learner still have some motivation for the feedback process?

• given the number of learners in the class/group - and how much time the teacher normally has to give each learner feedback - is giving more feedback on this layer sensible? e.g. a tutor with fewer learners in their group, might give choose feedback whereas a teacher with a larger class might choose best learned later,

• thinking about the next fortnight, will there be enough time to embed the feedback into the learner’s long term memory? e.g. if there is only 2 days before a 2 week holiday, then the learner is unlikely to retain the benefits of feedback but if there is more than 2 weeks before the next holiday, the teacher might want to try to give a bit more feedback.

• if the feedback on a layer isn’t going well, the teacher and learner are likely to remember, perhaps with a slightly sick feeling in the stomach, how the previous feedback-dialogue attempts went. They should both be honest about whether the feedback is working, in a case like this, deciding on BELL will open up more time and more motivation for other feedback.

• the more feedback assessments per assignment, the less likely each is to result in embedded learning - usually 3 feedbacks per assignment is enough, so if a layer which has already had several feedback dialogues is the only layer needing feedback perhaps continue, but if there are 2 or 3 other layers needing feedback perhaps BELL the layer which has had several feedback’s already.

The purpose of marking may be thought to be

• To make sure there are no remaining written errors on the page: the reason might be that then the learner can revise from their exercise books. This is not a good use of time, because we want learning to be improved soon, rather than hope learners will return and revise at a later date.

• To give hints or clues or model answers to help the learner. However the teacher must guess what the learner was thinking/ not thinking when the learner made their error(s). This is not a good use of time because often the teacher will guess wrong, or the learner won’t read, or understand if they even read what the teacher has written.

• The teacher gives value judgements in the hope of changing learners effort/motivation in lessons. This is not a good use of time, because value judgements, even positive ones, may decrease rather than increase the likelihood of learners learning from suggestions and the learners motivation.

• A repetition of all or part of the original teaching. This is not a good use of time, because if the teaching of the lesson, didn’t lead to embedded learning, the teachers explanation - which is necessarily less detailed and can’t involve assessment for learning, as the lesson could - is less likely to lead to embedded learning.

Although the desired marking purpose may sometimes be achieved, the probability that any of these purposes will effectively embedding learning is low, and the cost to the teacher is high.

This saves the teacher a little time for each question.

Teacher’s may find suppressing their urge/habits to write more a little difficult at first - but stick with it - not only will it reduce the teacher work load in non-directed time, it will also make retrieval practice more effective at embedding learning.

_{The only exception to this rule, is for example, if the learner makes an error in a complex multistage word problem, the teacher might want to write down the numerical answer (on the learner's page, or on the teacher’s lesson plan page) to save the teacher having to work out the answer to check the learner's correction.}

If the assessment outcome is a tick, BELL (BEst Learned Later) or reset: there is no need for feedback in the next lesson.

If the assessment outcome is FOA feedback on attempt or FOB feedback on blank: then we expect the learner to try and self correct or get peer to peer help or initiate a feedback-dialogue with the teacher, within the rules of the classroom.

By assessing and where necessary, engaging in personalised feedback dialogue in the lesson, we gain multiple advantages:

1. The teacher is replacing the time and effort they would spend on marking, non-directed time, with only the possibility that they may need to spend directed time, inside of lesson time, on feedback. The teacher need not feel guilty, as not marking + giving feedback in the lesson (if necessary), helps the learner learn better.

2. The non-directed time spent by the teacher will be less - today - because this kind of assessment is far quicker than marking, and will be less - in the future - because feedback is far more likely to be successful and so similar questions will be asked less frequently and answered more accurately.

3. The learner has a chance to self correct or self reflect or get peer-to-peer help: so the teacher may not need to spend lesson time giving feedback or failing that, the learner has time to read the question again and will be primed to adjust their chunk or mental schema (Kornell et al 2009) during the feedback-dialogue.

4. With in-lesson feedback-dialogue the learner is far more likely to engage with the teacher than they are likely to engage with the teacher's marking.

5. Without marking by the teacher, the teacher and learner can start the feedback-dialogue from a common place.

6. Within the feedback-dialogue the teacher has a chance to learn about learner's past thinking and/or influence the learner's future thinking, more effectively than with marking.

7. Feedback-dialogue makes excellent, non threatening, feedback for the teacher on fine detail ways to improve future teaching. It allows the teacher to gain decades worth of high quality teaching experience within a much shorter span of time.

8. Feedback-dialogue provides an opportunity for the teacher to help the learner to better deal with the emotions brought up by errors e.g. to move on from self-criticism or making excuses or blaming others and to help the learner reflect about their question reading or process or problem solving skills i.e. help the learner grow a growth mindset.

9. Feedback-dialogue is likely to increase the learner's motivation whereas marking is likely to decrease it.

10. Sometimes during feedback-dialogue the teacher and/or learner will realise that the layer is BELL BEst Learned Later - and this is also a productive use of the teachers and learners time - because now they can stop putting their time and effort into making this layer stick and instead put their efforts in to embedding other layers which are far more likely to stick. After all we can’t expect to go from a place where most teaching is forgotten to a place where absolutely all teaching becomes embedded learning. Surely a process where almost all teaching becomes embedded learning should satisfy us - there is no shortage of other layers to teach - before returning, to see if this layer will stick in a few months time.

The main disadvantage (such as it is) is that there is very little in the way of an easy to follow paper trail of the efforts the teacher has put in. There is the record that feedback dialogue has been requested, and possibly the corrected workings by the learner. In future assignments we usually see the learner being able to answer correctly similar questions and there is the record of the subsequent progress on the layer within the timely practice app. If the teacher must be checked up on, then the “checker-upper” must do the work to find out if the feedback is successful or not. This counterposes with a marking trail - where the teacher is doing the work of demonstrating that they have tried - but cannot possibly demonstrate that they have been successful - in enabling the learner to embed the learning.

Not giving hints etc will pay dividends, as the answer space can be used to answer the question

• by the learner and teacher to model answering if the learner gets peer-to-peer help or feedback-dialogue with the teacher

• by the learner if the learner can actually do the question - perhaps they just didn’t see it, they got distracted, they were having “an off-day” etc

** or other symbol used to show that the answer is not fully correct but that the teacher wishes the learner to try and self correct/get peer help/get feedback

Even though many questions will take the teacher "next to no time" to work out the correct answer, the teacher should use the answers provided by the app, because 

• the small bits of extra time to read the question and work out the correct answer add up,

• it distracts the teacher from the assessment task - i.e. how well is this learner able to apply their learning - it is harder, and therefore takes fractionally longer, for the teacher to switch back and forth between doing maths and assessing maths.

1. The learner has built an incomplete chunk in long-term memory - despite timely practice layers being small and therefore easier to learn, sometimes the learner will need more support to build a chunk - the teacher, via feedback-dialogue, should work with the learner to find what is missing and help the learner fix it. See chunk-basedTheory(4) , for more about this.

2. The learner has not replaced/adapted an incorrect chunk built some time ago, with a new/adapted chunk in long-term memory. This is different from 1. in that the chunk to do the old incorrect method hasn’t been overwritten, despite perhaps a new chunk being built in the lesson. So here we are working on changing the trigger i.e. the learner choosing the new correct chunk, rather than the old incorrect one. The best way to do this, is to offer a reason why the old incorrect one is incorrect or inefficient which chimes with the learner’s understanding.

3. The learner is still reliant on some of the “unacknowledged scaffolding'' of the lesson e.g. placement of workings out on the page or use of a diagram etc. See chunk-basedTheory(4) for more about this.

4. The learner has misread the question or poorly applied their numeracy skills when answering the question. This is likely to be due to working memory overload - all learners, even the most able A level maths learners experience it. As they are learning and working through something new and hard, they are unable to accurately apply skills which are usually easy for them. I sometimes describe this effect to students as their brain isn’t very good at easy thinking and hard thinking at the same time. The best we can offer learners as they practise, is that they can look through their workings out for accuracy periodically. Sometimes I suggest they write “check for accuracy” on the answer line, as an aide memoire, for when they think they have solved the problem. We need to encourage learners to realise that making “silly mistakes” is often a sign of hard learning going on, not a sign that “they can’t even do the easy maths”.

I wish I could say that is why timely practice expects the teacher to assess assignments between one lesson and the next, so that feedback is delayed by at least one sleep. However the main reasons for this fundamental decision are:

• to reduce the need for learners to having one computing device per learner and the need for reliable access to wifi in every lesson,

• to get learners to write by hand their workings and diagrams, just as they must in a test (which may also be more effective at embedding learning).