limited topic trial - is on hold for most of 2021-22

The big picture question

Does teaching with timely practice embed significantly more learning than teaching following our
current scheme of learning and program of study?

The limited topic trial will

  1. measure the learners learning gain from teaching 
    1. using the school's current scheme of learning,
    2. using a bespoke timely practice scheme of learning, which replaces another part of the school's scheme of learning.
  2. teach the teacher how to use the timely practice app.

The limited topic trail will be free for:

  • training,
  • creation of scheme of learning,
  • use of timely practice app,
  • use of teaching resources;

but we may charge for

  • travel, accommodation and sundry related expenses if we need to visit your school.

same teaching skills

The skills required in teaching a traditional maths lesson and a timely practice maths lesson are the same:

  • the ability to help learners hang new learning on existing learning,
  • clear explanations,
  • knowing when to use closed and open questioning,
  • give effective feedback and
  • empathy.

The difference between teaching traditional maths lessons and timely practice maths lessons are in timing

different timings for teaching and learning


traditional maths teachingtimely practice maths teaching
scheme of learning 
  • topics taught once per year
  • a few layers from a topic taught per week
  • ≈ 60 topics ÷ almost 40 weeks = one or two topics per week
  • only one layer from a topic taught per term,
  • one or two different topics are taught per hour,
  • ≈ 4 topics per week x 40 weeks ≈  160 topics per year 
  • ≈ 160 topics per year ÷ 60 topics → topics taught 2 to 3 times per year
start of lesson
  • whole class
    • general maths skills or
    • reminder of pre-requisites for main part of the lesson
  • 5 to 10 minutes
  • personalised timely practice on
    • variety of questions from many topics
  • whole class/small group/personalised feedback on problem areas
  • 10 to 25 minutes
teach new learning
  • whole class
  • on one topic
  • 5 to 20 minutes
  • whole class
  • remind pre-requisites before teaching one layer from a topic
  • sometimes there is enough time for the teacher to teach and the learners to practise a layer from second topic 
  • 5 to 10 minutes per topic
practise new learning
  • there is likely to be some differentiation between learners
  • questions for each learner get progressively harder
  • 15 to 35 minutes
  • there is likely to be some differentiation between learners
  • all the questions for each learner are at a similar level of difficulty
  • 5 to 20 minutes
practise prior learning
  • sometimes/often at the start of the lesson 
  • often/usually prior to tests
  • usually at the start of the lesson
  • almost every maths lesson
feedback
  • is usually written
  • is usually given every 2 to 3 weeks
  • is usually oral
  • is given up to every maths lesson (but only when required)
after feedback
  • common mistakes may trigger extra teaching
  • practice questions are set more frequently
assessment after learning
  • homework
  • end of unit test
  • extra practice questions done completed in lessons, some time after teaching, are assessed promptly
  • frequency of practise responsive to accuracy and retention

different timings for assessment

Depending on when we assess learners (especially low attaining learners) we will see a very different picture of how well teaching has become learning. 

For low attaining learners (those not expected to gain a grade 4 or higher at GCSE)
time elapsed after teachingtraditional maths teachingtimely practice maths teaching
at the end of the lesson
  • almost all learners have learned what was taught
  • almost all learners have learned what was taught

next maths lesson

  • some learners have forgotten significant details 
  • most learners remember most learning
  • some learners remember everything
  • some learners have forgotten significant details 
  • most learners remember most learning
  • some learners remember everything
a month
  • some learners have forgotten most learning 
  • most learners have forgotten something
  • a few learners remember everything
  • a few learners have forgotten something
  • most learners remember everything
3 months
  • some learners have forgotten everything
  • most learners remember some learning
  • very few learners remember everything
  • most learners remember everything
  • most learners have mastered the learning

to measure embedded learning we must delay summative assessment

Condition A: For an effective trial we must delay testing for at least 4 weeks, (ideally perhaps 7 weeks - a summer holiday's break) after teaching / any revision work/ any end of unit tests which the school normally uses.

pre assessment must be robust

From previous trials we know that one question asked and answered isn't sufficient to decide whether a learner already knows a skill or not. Only asking one question makes calculating learning gain open to teachers judgement and to luck (see the pre assess must be robust (info) box at the bottom of the page).

Condition B: For a simple yet accurate pre assess use timely practice pre assess assignment and avoid false positives by asking a second question on every layer which appears to be known via a timely practice classwork assignment. In the trial the assessment choices for pre assessed layers (bites of learning) are perfect or not.

Condition C: We must keep the teaching and learning in the business as usual (BAU) and the timely practice (TP) lessons separate. Therefore each learner will have two timely practice profiles, one for all the topics which will be taught in the the BAU lessons and one for all the topics which will be taught in the TP lessons.

  • The BAU profile will find and verify the layers (small bites of learning for a topic) which the learner "already knows" in the BAU units and will be used to create the "delayed summative assessment" - which will assess both what the learner already knew and the new learning taught in the BAU units. 
  • The TP profiles will find, verify and then more deeply embed into long term learning what the learners already knew in the TP topic themes, ensure that the new learning in the TP lessons is practised at intervals to ensure retention and  to assess that learning in the "delayed summative assessment"

make the trial as easy as possible for teachers

  1. disrupt the teaching of maths as little as possible - so the teacher will teach the BAU units from the existing scheme of learning, programs of study and resources

ensure mastery learning is used for the timely practice lessons

  1. teach on firm learning foundations. i.e. the pre assess must be used to plan teaching 
  2. the teacher must teach only one layer from each topic 

ensure practice questions are timed to efficiently embed learning in the timely practice lessons

  1. the learner should do less practice questions on the learning of the lesson directly after teaching (this is fewer practice questions than the teacher would normally do directly after teaching),
  2. the learner must do retrieval practice questions at close to the optimum times calculated by the app (i.e. a timely practice assignment must be done every maths lesson for at least one month after new teaching and twice reducing to once a week within the second month after teaching),

Condition D: For the timely practice (TP) lessons the teacher should follow the scheme of learning which timely practice will create after the pre assess process. This scheme of learning will

  • teach all the prioritised topics from the existing scheme of learning along with most of the other topics from the topic theme,
  • ensure that the fundamentals of mastery learning and retrieval practice are applied.

Condition E: For the correct timing of the timely practice questions, the learners will need to do a short timely practice assignment for 5 to 10 minutes within the subsequent BAU units. The teacher will need to give feedback during these 5 to 10 minutes sometimes to some learners.  The 5 to 10 minutes of timely practice and feedback can to some extent replace the warm up of the maths lesson. To compensate for the fact that timely practice (TP) business is taking up BAU lesson time, the duration allocated to teach the BAU units must be extended and this time must be taken from the time allocated to teach the TP topic themes, (TPTT). 

mimic a tightly spiralled scheme of learning in the timely practice lessons

A 1 term trial means the teacher can only teach each topic once. The teacher may want to return to teach more on these topics after the trial is finished.

A 2 term trial means the teacher can return to teach more on each topic in the second term. This will give a clearer picture of how timely practice works.

Condition F: The school may choose a 1 term or a 2 term trial. The school may hedge their bets and decide whether to continue the 1 term trial into term term 2, after the first term. If this is considered a possibility, then the we must plan in advance which units will be taught in the business as usual lessons over the two terms, prior to starting the 1 term trial. See Condition C. 

ensure that it is decided in advance which topics are BAU topics and which are TP topics

There are 5 topic themes: number, word and proportion, algebra, geometry and measure, and probability and statistics.

Condition G: The BAU units will come from 2 topic themes, the TP topics will come from the other 3 topic themes (at least 2 if not all 3 topic themes will be taught). There must be a strict separation of topics, to ensure that the learning gain the trial calculates can be attributed fairly between the BAU and TP lesson time.

Summary of conditions

Condition A: delay final testing for at least 4 weeks/ ideally 7 weeks after teaching / any revision work/ any end of unit tests which the school normally uses.

Condition B: in pre assess each layer must be answered twice correctly, to count as "already knows"

Condition C: we must ensure that the BAU and the TP lessons do not overlap teaching content, so each learner will have a BAU profile and a TP profile for assessment purposes.

Condition D: BAU lessons are taught as normal except for condition E, TP lessons are taught following the TP scheme of learning - which will be created after the pre assess process 

Condition E: 5 to 10 minutes of most BAU lessons will be used to review TP skills, so BAU units will have an extra lesson (or two) per unit and TP will have one (or two) fewer lessons per unit

Condition F: The school may choose a 1 term or a 2 term trial. 

Condition G: The BAU units will come from 2 topic themes, the TP topics will come from 2 or 3 topic themes - there must be a strict separation of topics. 

Decisions for this trial

For conditions C and G we must split up the maths content (topics) to be taught into BAU and TP.

The BAU units must be selected from only 2 topic themes. 

The TP units must be allocated the the other 3 topic themes.  timely practice is hungry for topics to teach, because we use a breadth first SOL how this impacts on the trial is explained in more detail below the table of topics found in each topic theme.

The school can choose which of the units in the period of the trial will be BAU will be taught in term 1 (and which might be taught in term 2) - these must be drawn from only 2 topic themes. An example follows on how this might be done, but first we'll look at

Which topics are in each of the topic themes

Which topics the learners might be ready to learn

Number FDPAlgebraGeometry and MeasureProbability and Statistics
Number Word and ProportionAlgebraGeometry and MeasureProbability and Statistics

Very low attaining learners are likely to be ready to learn just a few topics from each topic theme ...

e.g. only ready to learn 6 topics e.g. ready to learn 8 topics e.g. ready to learn 5 topics e.g. ready to learn 6 topics e.g. ready to learn 3 topics 

... whereas some older and more highly attaining learners may be ready to learn considerably more topics from each topic theme, 

these topics are prefixed with the word "perhaps" e.g.  "perhaps BiDMAS" means that it is likely that a number of low attaining learners will not be ready to learn any of the topic "value of: index" in the trial.

There may also be some topics which learners no longer need to learn e.g. "solving ready or perhaps solve" means the learner will no longer need to learn solving ready once they are ready to learn solve

perhaps up to 13 topicperhaps up to 15 topicsperhaps up to 12 topicsperhaps up to 13 topicsperhaps up to 9 topics

Why does TP need 3 topic themes?

As already mentioned

  • the BAU units must be selected from only 2 topic themes
  • timely practice is hungry for topics to teach, because we use a breadth first SOL 

For a very low attaining learner - if only two topic themes are chosen

  • e.g. Algebra and Probability and Statistics the learner may only be ready to learn 5 topics from Algebra and 3 from Probability and Statistics, 
  • e.g. Word and Proportion and Geometry and Measure the learner may only be ready to learn 8 topics from Word and Proportion and 6 topics Geometry and Measure,

in both cases a further topic theme will be required to give the teacher enough to teach. 

For learners at the lower quartile in KS4 - two topic themes may well give the teacher enough to teach 

  • e.g. if Algebra and Probability and Statistics are chosen, the learners may well be ready to learn 12 topics from algebra and 8 or 9 from Probability and Statistics, 
  • e.g. if Word and Proportion and Geometry and Measure are chosen, the learners may well be ready to learn 12 to 15 topics from Word and Proportion and 11 to 13 topics Geometry and Measure, 

then perhaps a further topic theme won't be required. However if there is even one learner who needs to be taught more topics (i.e. from a third topic theme) - we want to ensure that the trial doesn't hold that learner back. Hence we want timely practice to have 3 topic themes to choose from, meaning BAU must be restricted to 2 topic themes. Which is why

  • the BAU units must be selected from only 2 topic themes
  • and the TP units must be allocated the the other 3 topic themes. 

This sample SOL is from KS4, see below, where the first 2 units will have been taught before the trial starts. 

  • 2 of the units from units 3 to 12 must be allocated to BAU for the 1 term trial,
  • 4 of the units from units 3 to 12 must be allocated to BAU for the 2 term trial

but these must come from only 2 topic themes. 

A colour coded Key is used Number, Word and Proportion, Algebra, Geometry and Measure, Probability & Statistics 


One option would be

BAU unit TP topic theme 

Unit 3: simplify and substitute

1st term of trialthis can't be taught

Unit 4: angle rules

1st term of trialthis can't be taught

Unit 5: averages and graphs

this can't be taughtProbability & Statistics

Unit 6: FDP and ratio

this can't be taughtWord and Proportion
Unit 7: sequences2nd term of trial this can't be taught

Unit 8: data graphs

this can't be taughtthis is already chosen

Unit 9: shape properties and calculating space

2nd term of trial this can't be taught

Unit 10: equations

AFTER TRIALthis can't be taught

Unit 11: transformations 

AFTER TRIALAFTER TRIAL
Unit 12: probabilityafter the trial it may not be necessary to teach this
return to Unit 1: Calculating
Number

The TP topic themes are  Probability & Statistics and Word and Proportion and Number



Another option would be

BAU unit TP topic theme 

Unit 3: simplify and substitute

this can't be taughtAlgebra

Unit 4: angle rules

1st term of trialthis can't be taught

Unit 5: averages and graphs

this can't be taughtProbability & Statistics

Unit 6: FDP and ratio

1st term of trialthis can't be taught
Unit 7: sequencesthis can't be taughtthis is already chosen

Unit 8: data graphs

this can't be taughtthis is already chosen

Unit 9: shape properties and calculating space

2nd term of trial this can't be taught

Unit 10: equations

this can't be taughtthis is already chosen

Unit 11: transformations 

2nd term of trial this can't be taught
Unit 12: probabilityafter the trial it may not be necessary to teach this
return to Unit 1: Calculating
Number

The TP topic themes are Probability & Statistics and Algebra and Number


This sample SOL is from KS4 - it is heavy on algebra, but light on number, so TP may need to teach a third Number topic theme to the lowest attaining learners. This is likely to be a review of the first unit of the year, calculating. However if all the class are ready to learn most topics in the two chosen topic themes, then only two topic themes will be allocated to TP.


Assumptions for this example trial

FYI we can devise a bespoke trial to suit a schools unit length, lesson length and term length.

Teaching

Assuming in the 1 term trial:

  • 4 units are taught in a term
  • each unit will be taught for 3 weeks
  • each week has 4 maths lessons and each maths lesson is 1 hour.

i.e. 12 week term with 48 maths lessons. 

For a fair comparison, TP will need to use 5 to 10 minutes of each BAU maths lesson (perhaps replacing the warm up), see Condition E, i.e. 60 to 120 minutes, i.e. 1 to 2 extra lessons. 

So each BAU unit will have 12 + 2 lessons and each TP topic theme will have 12 - 2 lessons

BAU will have 14 lessons to teach each unit 

TP will have a total of 20 lessons - this may be split into 2 units or 3 units (depending whether the teacher will teach 2 topic themes or 3 topic themes).

Assessment

Accurate assessment requires verification with timely practice 

We need to accurately measure longer term learning gain. This means we need

  • an accurate measure of what each learners "already knows" and
  • to wait at least a month (but ideally half a term / the duration of the summer holiday) before we do the "delayed summative assessment" 

 to be sure our calculation of learning gain is accurate as possible.

timely practice makes this easier than it could be, however the design philosophy of timely practice is slightly different - timely practice uses data to ensure that the learner embeds the learning for the long term, so timely practice does assessment for learning - whereas for the trial we want to use timely practice to do "delayed summative assessment". 

The main thing we need to do is ensure that the pre assess data is accurate - meaning each learner will do a second verification question on any layers, the assessment of the pre assess assignment showed they "already know".

pre assess must be robust

Many low attaining learners have non predictable learning gaps  - so what to a maths teacher looks like two similar questions (a) and (b)

  • one learner may find (a) easy for them but (b) too hard for them and
  • another leaner may find the opposite (a) too hard, but (b) easy for them

Learning from the past

When we have measured learning gain in the past, we find that after asking only one question on a topic we may think the learner already knows a layer - but by asking two more questions on the topic - we find that they do not. We refer to this as a false positive.  In our intervention trial we used a points system, one point for partially knows, two points for fully knows - but this was arguably open to interpretation and made measuring learning gain complex. For some learners up to 50% of the layers we tested gave false positives, for other learners the proportion was negligible. 

Applying to the future

For the limited topic trial, we will require every layer to be asked and answered independently and accurately twice before we judge the layer as "already knows". This will make it easy for

  • the teacher to assess each question (perfect or not),
  • the results of the trial to be analysed (one point per layer - no drilling down to find the learner's past performance on answering each layer),
  • the teacher to plan teaching during the timely practice parts of the trial (teacher sees only what the learners already know well),
  • the teacher to give feedback during the timely practice parts of the trial (not wasting time giving feedback on what it turns out the learner doesn't know).

It will also make

  • it fair when comparing the learning gain with the BAU taught topics and the TP taught topics,
  • easy for schools to use the assessment for learning data from the BAU taught topics should they decide to continue to use teach timely practice to teach. 

Advantages of using timely practice for assessment

  1. Assessment is quick, as the app presents the answers to the teacher, together with the assessment code options for pre assess the teacher has the choice of or the app then collates all the data - and can easily create a second assignment including only questions the learner seemed to already know, the verification assignment.
  2. The topics are split into small bite size pieces of learning - which is ideal for low attaining learners. Partial learning can be measured. For example we can see which learners can write down a coordinate in the first quadrant or the first, second and fourth quadrant or all 4 quadrants. As teachers, we already know that to take a low attaining learner, from not being able to write any coordinate accurately to being able to write any coordinate in any quadrant accurately is "a very big ask". However finding where each learner is on the "writing a coordinate accurately ladder" and moving them one rung up the ladder - and ensuring they don't slip back when we aren't watching - means that we can return at a later time to teach a little more. This also means small learning gains can be measured.
  3. The teacher doesn't need to collate the data - calming though filling in a ROG spreadsheet can be, it probably isn't the best use of a teacher's time.
  4. Learners can very rarely copy, as they very rarely get the same questions - therefore assessment doesn't mean moving all the chairs and tables!

Disadvantages of using timely practice

  1. Teachers can't use the "mark a page at a time" method for assessing tests - as each learners assessment assignment will be different.

Finding out in fine detail what each learner already knows, and what they have learned is time consuming

If we are to do a trail and get accurate data then there is no way around this. In the trial we need to find all the learning which is fuzzy or partial (and with low attaining learners this can be a lot of their learning), hence the need for verification, which increases the time that assessment takes. 

On the plus side, if the school goes on to use timely practice - all the data collected can be used - and the classes in the trial can skip the Finding and firming learning foundations phase and go straight to the Teaching on firm learning foundations phase.

FYI timely practice is not normally as time consuming when collecting assessment for learning data, as it will be for this trial; because timely practice is in the business of embedding learning rather than measuring how much learning it can take the credit for and how much learning prior teaching can take the credit for. 

What's a snapshot

After the verification question has been assessed, we can take a "snapshot" of each learner's learning profile (print out or copy and paste on to a spreadsheet) to accurately record the assessment and then merely count the "Number of layers correct" for each unit or topic theme. 

What time will be required to do the assessment?


BAU units (business as usual)TPTT (timely practice topic themes)
Pre assess up to 1 lesson per unitup to 2 lessons per topic theme
Verify pre assess up to 1 lesson per unitno extra lesson time - all time will come from timely practice allocated lesson time

Post assess

as school's existing SOLnone
"delayed summative assessment"up to 1 lesson per unitup to 2 lessons per topic theme

Why does timely practice need more lesson time to do pre assess?

If every learner is given a test on every layer (bite of learning) which any learner may know from every topic in the BAU unit this could be

  • 4 questions from each of 5 topics i.e. a maximum of 20 questions in the BAU units

If every learner is given a test on every layer (bite of learning) which any learner may know from every topic in the TP topic theme this could be

  • 4 questions from each of 12 topics i.e. a maximum of 48 questions in the TP topic theme

How much time is needed for assessment for the one term trial?

  • the extra time to assess the BAU units will be up to 6 hour lessons 
  • the extra time to assess the TP topic themes will be up to 6 hour lessons 

Although very low attaining learners will require much less, probably a total of 3 hours.

How much time is needed for assessment for the two term trial?

  • the extra time to assess the BAU units will be up to 8 hour lessons 
  • the extra time to assess the TP topic themes will be up to 6 hour lessons 

Although very low attaining learners will require much less, probably a total of 4 hours.

Calculation of learning gain

Heres how we propose to calculate learning gain.

Calculation of BAU learning gain

GAIN = Number of layers correct in formal assessment of BAU1 

PLUS Number of layers correct in formal assessment of BAU2

MINUS Number of layers correct in BAU1 pre assess

MINUS Number of layers correct in BAU2 pre assess

Calculation of TP learning gain

GAIN = Number of layers correct in formal assessment of TPTT1 (timely practice topic theme)

PLUS Number of layers correct in formal assessment of TPTT2

PLUS Number of layers correct in formal assessment of TPTT3 (if appropriate)

MINUS Number of layers correct in TPTT1 pre assess

MINUS Number of layers correct in TPTT2 pre assess

MINUS Number of layers correct in TPTT3 pre assess (if appropriate)

The trial itself

Please refer to the one or two term limited topic trial scheme of learning page

FYI

pre assess must be robust

Many low attaining learners have non predictable learning gaps  - so what to a maths teacher looks like two similar questions (a) and (b)

  • one learner may find (a) easy for them but (b) too hard for them and
  • another leaner may find the opposite (a) too hard, but (b) easy for them

Learning from the past

When we have measured learning gain in the past, we find that after asking only one question on a topic we may think the learner already knows a layer - but by asking two more questions on the topic - we find that they do not. We refer to this as a false positive.  In our intervention trial we used a points system, one point for partially knows, two points for fully knows - but this was arguably open to interpretation and made measuring learning gain complex. For some learners up to 50% of the layers we tested gave false positives, for other learners the proportion was negligible. 

Applying to the future

For the limited topic trial, we will require every layer to be asked and answered independently and accurately twice before we judge the layer as "already knows". This will make it easy for

  • the teacher to assess each question (perfect or not),
  • the results of the trial to be analysed (one point per layer - no drilling down to find the learner's past performance on answering each layer),
  • the teacher to plan teaching during the timely practice parts of the trial (teacher sees only what the learners already know well),
  • the teacher to give feedback during the timely practice parts of the trial (not wasting time giving feedback on what it turns out the learner doesn't know).

It will also make

  • it fair when comparing the learning gain with the BAU taught topics and the TP taught topics,
  • easy for schools to use the assessment for learning data from the BAU taught topics should they decide to continue to use teach timely practice to teach. 

About prioritising topics

We recommend schools follow their existing scheme of learning, so these are only sample units.

In most schools a topic theme will be taught in 2 to 3 units during the year.

Here are some examples of what topics may be taught in some a few units.

Number Word and ProportionAlgebraGeometry and MeasureProbability and Statistics
BAU unit e.g.BAU unit e.g.BAU unit e.g.BAU unit e.g.BAU unit e.g. 

calculating

equivalence FDPR

simplify and substitute

shape properties and angle rules

data graphs

BAU unit e.g.BAU unit e.g.BAU unit e.g.BAU unit e.g.BAU unit e.g. 

types of numbers

calculating with FDPR

sequences and graphs

transformations and calculating space

averages and probability 


BAU unit e.g.BAU unit e.g.


word problems

not suitable - pre assess takes too long

equations



Once a topic theme is allocated to TP then the topics in the unit it replaces the topics from the unit may be prioritised, if there might not be time to teach all the topics from the topic theme - with the proviso, that topics will be only taught if the pre assess process says the learners have sufficient pre requisite skills that we can be (relatively sure) that the learners will retain the learning.