Q: When is a target not a target? A: When its a statistical norm #OnTheSick
Details have begun to leak about the Dispatches investigation into the Work Capability Assessment scheduled to air on monday night, then followed by a Panoram investigation into the same subject. Dispatches report that they have sent an undercover GP, Dr Steve Bick through the complete Atos training process filming undercover.As news leaks, reputable news outlets are reporting that this shows this must be proof of targets built into the WCA system, presumably in the contract the DWP issued to Atos. However, this is not quite the case, what is being reported based on evidence from undercover investigation is 'targets that aren't targets', ie statistical norms. What this means is that there are effectively targets built into the system, but they are targets that are not technically targets because the targets seem to be being imposed by a potentially false baseline average, for which there appears to be no source of original data or evidence at this time.
This means there are essentially two possibilities; either the government, DWP, ministers and Atos have all repeatedly lied on record about the existence of targets, or they are telling the truth and there aren't actually any targets, because they aren't called targets, they are called statistical norms.
Atos are an international IT company, their business has been built upon using systems of statistical norms to determine trends. This can be seen as acceptable if for example Atos were running the contract for a telecoms system. A company who want to understand the different types of phone calls made at different times would clearly benefit from this kind of statistical analysis, which is presumably why telecoms giants use the Atos systems. However, human beings, particularly sick, disabled and vulnerable human beings do not fit neatly into any kind of statistical system. It is impossible to predict who will come through the door in any given day to see an HCP for their WCA, quite unlike telephone calls which tend group into averages - the way we see reflected in the kind of special deals offered by British telecoms companies who use such data to offer cheap calls after 7pm.
The consequences of getting the baseline figures wrong to work out an average for telephone calls might be significant financially for businesses, and impact on consumers by skewing the 'special deals' based on these averages, but they do not have the potential to have a devastating harm on individual lives in the way those exact same averages do when used to assess sick and disabled people for eligibility for benefits.
So, what really happens on the frontline for those being assessed for benefits and those doing the actual assessments?
A system of statistical norms is used to ascertain how many people a health care professional assigns to the different eligibility groups in Employment Support Allowance; the support group for those most unwell and not expected to perform any work, the Work Related Activity Group for those expected to be able to do some work at some point in the future, and those found fit for work. Using these 'norms' or averages, Atos monitor closely each HCP's performance, and as explained by Dr Bick, any who assign more people to those groups than the norms suggest they should are then audited.
Although I have been unable to find any evidence that HCP's are fined or formally disciplined for putting too many people in the support group, it is clear that frontline workers see this as a punitive process and are constantly attempting to juggle the requirements of the claimants they assess with the requirement to stay within a statistical norm. This average appears to be based on national rather than regional data, so an HCP assessing people in Croydon would be held to the exact same average that an HCP assessing people in the Outer Hebrides would. That's quite an extreme example, but demonstrates that using a system of averages creates a skewed picutre as it does not account for regional variations in population size, health conditions and the local economy.
At this time, no-one knows what original data these statistical norms, or targets that are effectively targets, but just aren't called targets, are based upon. It may be that data originated from the DWP, based upon guesstimates from the old Incapacity Benefit claims, but it may be that Atos only ever had access to an average data sample to build their national averages upon. Frankly, unless the DWP and/or Atos radically alter their stance and start communicating clearly about the details of the WCA system and outsourcing process then we may never know the answer to this key question.
The next question is of course whether these 'norms' take any account of different conditions, ie are there different sets of norms for people with Mental Health conditions, fluctuating conditions, sensory disabilities or Learning Disabilities? The answer is presumably not, as HCP's, confirmed by DWP and Atos quotes are outlining a simple one system of averages based upon crude numbers of total people seen.
That would lead us to wonder, how can a norm possibly be a norm for an overall client group without accounting for all these different conditions? Perhaps the DWP or Atos will see fit to share the answer to that question, but I shan't be holding my breath waiting.
Then that leads us to some even bigger questions, the DWP record outcome data for WCA's, to use as part of the official statistics relating to the benefit ESA. This data is of course based on outcomes from a system which uses Atos assessments held to unverifiable in origin norms, decisions almost always confirmed in practice by the DWP.
So, if the whole assessment system is based on a system of averages that have no regional or condition related breakdowns in them, then that flawed system is used to measure the official figures of those eligible for benefits we then have some very serious questions to ask about the validity of those final figures. Figures remember that seem to be built on an average assumption of how many people should receive a particular level of benefit, without ever leading us to the original data, if any, used to predict these outcomes.
This could mean that four years on from the introduction of a benefit which has caused terrible distress and fear to sick and disabled people having to claim it, been implicated as a factor in multiple suicides and used to make presumptions about work and growth figures has been based on nothing more than a guesstimate, that became an average, that became a norm, which became a target that isn't a target.
29 comments:
Great post. ATOS Healthcare is a really obnoxious company. It sends my blood pressure up every time I hear about them. They will do well though, it seems they're sucking up now by sponsoring the Olympics.
I just don't understand how you can create a statistical norm for those not entitled based on the fact that those referred by default have been sent there by a practitioner of some form or another, thus removing the individual from the stastical normal population. There is no way to extrapolate from an unknown /facepalms
RosieB: Quite...the flaws in this process are astonishing. The key Q's will become whether DWP understood all this and thus provided the norms to Atos in the full knowledge that it would effectively create a target system that's deniable because they aren't technically targets or whether its just a series of stupid assumptions and misunderstandings throughout various different parts of the process. Much as my inner conspiracist would like it to be the former, I rather suspect its much more the latter
I notice when asked about targets Chris Grayling is very emphatic about the fact there are "No FINANCIAL Targets" here - His reply to Q 268
http://www.publications.parliament.uk/pa/cm201012/cmselect/cmworpen/1015/11060802.htm
Perhaps there are targets other than FINANCIAL ones?
I think it's interesting seeing this, having also heard so many stories of people with obvious serious disability being denied the top rate of support and then being successful on appeal.
It almost makes me wonder if there are doctors working for Atos who, knowing they have to meet targets, deny the top rate of support to people who they know will get through on appeal in order to grant the top rate to cases they're unsure would succeed on an appeal?
Thinking about it, actually, I doubt that would happen given that they'd effectively be denying support to those people until their appeals went through. I have to admit though that I'm astonished so many doctors seem to just go along with this. Is it really what they got into medicine for?
Emer Grey: Interesting points. As far as I can tell, that doesn't happen...it seems to be more that HCP's are concerned with the audit for total no of ppl in support group. Now, this is obviously skewed because we're being told this by HCP's who have concerns about this and want the wider public to know those concerns, so it's not a clear picture of overall HCP's,or what Atos as a company think or the DWP's perspective
However, these HCP's are assessing ppl based on criteria set by the govt, via DWP. These criteria were deliberately changed to redefine the level of sickness or disability at which benefit qualifications come in, this was deliberately done by Gov/DWP to reduce the numbers of ppl qualifying for the benefits. I think we can intimate that medical professionals don't see this as appropriate, as so many are now trying to speak out. That Atos use the Official Secrets Act has no doubt influenced this and HCP's might have whistleblown much earlier had they not been fearful of national security 'threats' to themselves
Let's not forget these tests were devised with input from the American insurance company Unum)or UnumProvident, I forget what they were calling themselves back then), a company with an established reputation for criminality in the field of disability denial. I think this significantly alters the balance of probability in this regard. Why would you consult with criminals if you weren't planning on breaking the law yourself?
The Government announced before the IB migration started that their "reforms" would save £2bn pa. Simple logic dictates that there must be targets (oops I mean statistical norms!) that are in place in order to meet that overall savings target.
They have already announced how much will be saved under PIP, even before it has been introduced, so the same applies there.
OK, get yer brains warmed up...
This is an extract from the blog of the amazing Declan Gaffney, and quotes the DWP-Atos contract.
Data for WCA decisions only runs up to May 2011, but even so it shows a remarkable shift in the outcomes over time. For completed initial assessments, summarised in the attached table, the proportion found fit for work was declining even before Harrington’s first report was published, falling by some 10 percentage points since October 2009. And the percentage of claimants being placed in the ESA support group- those who are not expected to seek work because of the severity of their condition- went from 11% in October 2010 (just before publication of the review) to 16% by May 2011. One of the results is that DWP has been obliged to revise up its projected expenditure on ESA by £1bn (about 10%), as revealed by the Office for Budgetary Responsiblity in November.
In its original contract with Atos Healthcare to deliver the WCA assessment, DWP had specified that it expected about 11% of ESA claimants to go into the support group. This is what the contract says: ‘It is estimated that approximately 11% of new customers will fall into the support group. Atos Healthcare MUST base their solutions and costs the [sic] figures in Appendix 8.’ (Contract between DWP and Atos Healthcare, Request for Proposal, section 2.2.2: Appendix 8 shows DWP’s modelling of the caseload, including the 11% estimate for the support group). This is what the WCA proccess was delivering until Harrington’s review. Given the major changes since the review, it now seems very likely that the expectations incorporated in the contract had a significant impact on the outcomes for claimants.
The implication is that thousands of ESA claimants have been wrongly denied unconditional support since October 2008 due to faulty assumptions about the number of claimants with serious work-limiting conditions on the part of DWP ministers and officials, embedded in the assessment process and the contract with Atos healthcare. This interpretation is supported by the high rate of successful WCA appeals.
http://lartsocial.org/DLAreform
Now look carefully at the 11% figure.
If the total claiments are being placed into 3 groups, pretty evenly matched, what effect does the 11% have?
Well it drags the "norm" downwards.
The "norm" they WANT for new claiments is included in the figures for ALL claiments, and the two are not seperate - or at least we have no evidence that they are.
So if the "norm" is reduced, then the Atos examiner has to reduce the total number of people put in support.
The "norm" that MAY have been originally generated may be real, it may be a "norm" generated by preset ideals, by a false baseline. (I am sure we will find out).
But adding on this blatent skewing factor WILL reduce the "norm" across the board, thus leaving Atos assessors in no other position but to realign their results accordingly.
This is concurrent to the migration of IB claiments, claiments with extensive medical proof and histories that would statistically be pushing the "norm" upwards.
I am sure that given a few days we will have more answers to the questions posed, but thanks to Declan for being, as always, ahead of the game.
This denial is totally disingenuous. There are well established 'targets' See this in Daily Express back in 2010:
'Mr Duncan Smith told a work and pensions committee that his department estimated that 23 per cent of these people would be judged “fit to work”.
A further 58 per cent would be moved to Employment Allowance but would be required to make some preparation for employment.
Only 19 per cent would qualify for a higher “support” payment because they were unable to carry out any form of work' http://www.express.co.uk/posts/view/200030/500-000-benefit-scroungers-will-be-made-to-seek-work/500-000-benefit-scroungers-will-be-made-to-seek-work500-000-benefit-scroungers-will-be-made-to-seek-work500-000-benefit-scroungers-will-be-made-to-seek-work500-000-benefit-scroungers-will-be-made-to-seek-work500-000-benefit-scroungers-will-be-made-to-seek-work
The 'estimates' are slightly different in this Work ans Pensions Committee business report, on the reassessment trials:
' The Papworth Trust pointed out that, at the start of the reassessment process, the Government had estimated that:
65% of claimants would be placed in the WRAG
20% would be placed in the Support Group
15% would be found fit for work.[196]'http://www.publications.parliament.uk/pa/cm201012/cmselect/cmworpen/1015/101509.htm
It is obvious that ATOS have been too gung-ho is declaring people fit for work on the actual figures:
'38% (616 individuals) were placed in the WRAG
30% (484 individuals) were placed in the Support Group
32% (526 individuals) were found fit for work and not entitled to ESA'
This is a case of 'you say estimates, I say targets'.
Are they now planning to do exactly the same thing with PIP with the decision to make 20% savings before even seeing a single claimant?
We know that seeing an "independent health professional" is expected to be part of the process in the name of objectivity. Will each one be expected to "pass" 20% less people than would have got DLA for instance? This needs urgently clarifying.
Aagghh nooo.... it's so completely bonkers and yet it makes perfect sense now you put it like that. Esp what RosieB said. The people who gave Atos the contract on this basis are evil through and through. Allowing your own stupidity to direct you to an outcome you already wanted is possibly the worst thing that a human being can do.
I'll probably get roundly thumped for posting this, so let me first say this comment is not supporting Atos - I know not a great deal about what they do or how they operate. Rather, I want to just point out that the use of norms is reasonable under certain circumstances.
Firstly, it's wrong to suggest that because humans are involved and you can't predict on a one off basis who will come through the door that you can't use statistics to make more general predictions. That's exactly what statistics can be used for. I can't predict the winner of a single horse race, but if the odds on a horse are 10/1, I can predict the horse will win once in every 10 starts or so. Statistics is about the aggregation of observations to allow general case predictions to be made.
If you aggregate outcomes (say the % of awards made) from a group of people doing similar things, you tend to get a bell curve, with clustering of performance around a central point.
Some portion of the bell curve will be the "norm", centred on the peak. It's perfectly normal and in fact sensible to audit performance outside that range. Auditing is about checking and explaining, not about changing outcomes - operation of random chance or sampling variations may explain what happened, or there may be procedural differences.
So the aim of a norm based analysis is to normalise performance (fairly obviously), not specifically to shape it. The only way you can move the bell curve as a whole is to issue specific instructions (targets) to the entire population of practitioners to change the way they ALL assess cases. Otherwise it's about improving consistency overall within the group, and it's a perfectly reasonable management method.
Actually the baseline is fairly unimportant, because it will shift to the true position based on observations very quickly if its in the wrong place.
Normalisation of performance can be abused, obviously, and signalling requested outcomes with menaces is a way of doing that. That's the evidence to look for, rather than the use of norms themselves. If Atos are telling groups they expect only 1 out of 7 claims to succeed, then that is clearly a target. If they are explaining what the norm is and under what conditions an audit will take place, that is also a target, implicitly. But if they're measuring norms and auditing differences with a feedback loop into norm setting, there's a strong case to say that it's just responsible process.
Anon 1:20pm: As I understand it the issue is one of requested outcomes with 'menaces'. It's certainly perceived that way by all the HCP's I've been able to speak to.
Anon 1.20pm See quotes from HCP's here http://www.guardian.co.uk/society/2012/jul/27/disability-benefit-assessors-film
But the norm had to be taken from somewhere and specifically from what existed before, because the 3 categories are artificial and do not exist outside DWP. The comment of anon is interesting, because it implies that the norm would shift to the true position if it were in the wrong place. But it is in the wrong place for disabled people and in the right one for DWP if the desired outcome is to get as many people as possible from disability benefit.
Remember when Lord Freud began his report into Welfare Reform and wrote his report 3 weeks after declaring he knew nothing about the welfare system?
He said then that he "thought" about two thirds of those on IB were in fact fit for work.
That's it. Lord Freud's guesstimate based on a sytem he knew nothing about and voila, we have a policy. No evidence, no research - just his "thinking".
He seemed to have no idea that people on IB had ALREADY been through a testing process, said by the OECD to be the most stringent test for disability benefits in the world. And that most were put through it again at regular intervals.
Not long ago, Grayling admitted they were "surprised" at just how sick many of the people going through the new system were. The Workfare companies were complaining they were not getting enough people well enough to be able to put into jobs to reach their maximum £14,000 bonus.
Well, who knew?? People on IB are actually sick, shock, horror.
OK, just a quickie.
A race horse at 10/1 does not mean that statistically it will win one race in ten.
What it means is that given the horse's previous performance over that distance and ground conditions, and comparing that to the previous history of the other runners it has a 1 in 10 chance of winning.
And if it doesn't win, by your logic it will then win one of the next 9 races, so be 9/1.
What will happen is that the next race it will probably be at 20/1.
Neither humans or horses fit those neat little boxes, and every time someone walks through the door it is a UNIQUE event, which should not be influenced by previous events.
The chances of a patient walking through the door of Atos and getting support group is around 1 in 3, however they too are using a system of counting the rest of the field, trying to make a unique event part of a statistical prediction.
If the HCPs are deviant from the normb overall, then under pressure from management the HCPs have to assess with that in mind.
Its quite clear that the HCP has two options, assess accurately and face punitive recourse, or assess in a way that moves their assessments closer to the norm, satisfied that there is an appeal process, and no comeback on them from that appeal process.
The fact that the "norm" is NOT allowed to deviate, or move, means that should there be a terrible disaster, such an a random outbreak of almost everyone's heads falling off, some of those people with no heads will be found to be fit for work - around 33% of them.
And that is clearly nonsense.
I agree with the RT Honorable Chris Grayling about there being 'no financial targets';
They will spare no cost to see the disabled & poor of the land extinguished. For them it is worth the financial price.
I figured this one out a couple of years ago, as soon as it came out that ATOS audited HCPs against expected norms. It immediately creates a pressure on the individual HCPs to conform to the target quota, and punishes them if they do not. Whatever the language used, it creates a de facto target and anyone with even basic experience in quality systems or business management systems would have been well aware of it (someone remind me, what is it the rest of ATOS does - oh, business management systems, you don't say!). The protests that they aren't targets just aren't believable and ATOS will have been well aware of how they would function, as would DWP.
And when you consider that DWP have recently admitted that they did not have a clear understanding of the population of people being declared into the WRAG, it becomes very clear that fixed targets are indefensible.
Quite right that we take every possible opportunity to point out the flaws, inconsistencies and errors in the Government’s welfare programme, but don’t forget how thick-skinned and lacking in integrity it is. Grayling et al will attempt to argue the statistical toss here and their underlying altruistic intensions in their usual disingenuous manner until they back themselves into the inevitable corner at which point they will have no qualms about changing their tack completely. They have recently done exactly this with WCA audio recording.
Much of this is post decision rationalisation – they decided long ago what they wanted to achieve and since have desperately being trying to find a way of supporting it. I have yet to find one single aspect of the changes they have implemented that is GENUINELY evidence-based. If you take the trouble to look closely and challenge, the truth is that for example:
• There is no evidence to support descriptor changes or the correlation they assume between ‘typical day’ activities and capability to hold down a job.
• The research they rely on is at best inconclusive, at worst argues AGAINST their proposition.
• The reports they quote usually start with a disclaimer, even when they have commissioned them : “ These views are those of the author(s), not necessarily . . . . .
• The organisations and “experts” they cite may well have been consulted, but most certainly have NOT signed off the end result and do not therefore necessarily support it.
• Etc.
It is of course a thinly veiled charade to hide what is nothing other than a political dogma – they will not say outright that we are all benefit scroungers, but will happily construct a regime based on the belief that we are. If there was a genuine scientific basis for their approach, they would not have to approach it in such and underhand and deceitful way (the hypothetical wheelchair is an absolute classic!). If they genuinely believe that I am able to work, want to work and the “system” is preventing me, why adopt such a heavy-handed, autocratic, prescriptive, we-know-best philosophy? Why not just ask me what I’d like to do and help me do it? Why not build a process based on early consensus rather than one based on conflict and intimidation? It is obviously because you don’t trust me.
There do not have to be specific targets – the Government’s overall message is perfectly clear. If in doubt declare FFW – you may be wrong, but we’ll sort that out later. The end justifies the means and the amount of collateral damage along the way is perfectly acceptable!!
anything to do with this company atos is dicky and not to be trusted they lied al the way through and anyhting they sprout out taken with a lot of salt ,they will never tell the truth and its a sad reflection on the goverment allowing this company to prosper figures are meaningless only to matter to those who have been abused by this company and hope they shown the door with big fines to follow no atos abuses them who needed help not a kicking unbeleavible jeff3
When we are dealing with sick people, all this talk of norms is obscene. It's a way to distance the healthy from the sick, the well-off from the unfortunate. There are risk factors for cancer, but every year there are people who get cancer who don't have any of the risk factors. Are we going to start denying treatment for lung cancer to the minority who get lung cancer despite never having smoked?
A bell curve is the sum of a lot of little probability curves belonging to all the people in the target population. 37 deg. C (or 98.6 F) may be the normal body temp for a whole population, but not for individuals. There are many who run low or high. That's how you get the tail. But they are no less legitimate!
If your natural body temp runs low, then you could have a fever, but compared with the median for all people, it would look normal. For THAT person, what is normal for the population is not normal for them.
You cannot deny help just because a person's own little probability curve falls in the tail of the population's probability curve.
The bald truth is that ATOS does not deal in probabilities that people might be disabled. It deals in probabilities that people who say they are disabled are liars.
Insurance companies have a lot of power today, and they love to popularize the concept of "moral hazard" as it applies o their clients. What they don't say is that there is enormous moral hazard on THEIR side.
There is also risk in denying disability - the risk that someone who is very sick is wrongly denied. Where's their estimate of THAT?
You have a powerful entity making arbitrary decisions about who is cheating and who is not - is it any surprise that the powerless are left hurt by this process?
The essay noted that using national averages as endpoints discriminates against the poor. It also discriminates against ethnic groups who have higher risks for some conditions, and by gender because many diseases associated with moral hazard, with so-called self-reported or medically unexplainable symptoms, are in the category of autoimmunity and hit women far harder than men.
Who's the best judge of whether someone is disabled? Doctors, paticularly the doctor who works with that individual. And usually, the individual himself/herself.
The people who assume there's rampant cheating are sayng something about human nature. I wonder if they are saying more about THEIR nature than ours. Men who cheat on their wives often say that "all men cheat" - when actually, men who cheat on their wives are in the minority.
But they know all this. They got that contract by effectively guaranteeing it would be cost-effectivel that ATOS knows better then your doctor, than your family, than the people who put you on support in the first place.
To bump so many sick people off, to leave them helpless, is too cruel for most people - so you turn them into things. Statistics. Numbers.
Why not just tattoo it into our arms while you're at it? "Undeserving". It's so much easier to remember us that way.
I knew I had this somewhere. Freud based his 'guestimate'on the number of people claiming benefit in 1980. He made it clear that numbers on IB should be reduced by 2/3 rds. Here is a very offensive article explaining it from 2007
http://www.telegraph.co.uk/news/politics/1577312/1.9m-on-benefit-should-go-back-to-work.html
the targets can be found easily, look at the changes made by the last labour government when they changed the threshold points of claimants and then shifted them all onto incapacity and income support. Caroline flint and peter hain were the government appointed lackies involved and it cause thousands of people hardship, it cause people to live off sanctioned rates, it also led to a increase in people killing themselves in public places and eventually led us here today.
So if these norms are national, someone in Westminster, for instance, would stand a much better chance of getting into the support group(even with exactly the same level of disability) than someone in Glasgow, given the huge differences in overall health and life expectancy. Oh, and levels of wealth. Funny, that.
By the way, the DWP has an entire Department of Statistics. I had a Statistician friend who worked at one of their buildings in Sheffield. So they should know their way around stats or have minions who do.
Niahm
Like Anon, I am hardly a fan of Atos or DWP, but there is a distinct difference between estimates and targets. For the purposes of budgeting, forecasting and tendering, it is quite reasonable for estimates to be used, for tendering to ensure a level playing field and that tenders are returned on an exactly equal footing. If however they turn out to be inaccurate and result in a cost that was not expected, turning them into targets is something quite different.
I make this rather uninteresting point simply because I believe we must keep our offensive on these policies on an accurate basis or we just create opportunities for Grayling et al to drift away from the main issues. More interesting is where the estimates came from.
Like Anon, I am hardly a fan of Atos or DWP, but there is a distinct difference between estimates and targets. For the purposes of budgeting, forecasting and tendering, it is quite reasonable for estimates to be used, for tendering to ensure a level playing field and that tenders are returned on an exactly equal footing. If however they turn out to be inaccurate and result in a cost that was not expected, turning them into targets is something quite different.
I make this rather uninteresting point simply because I believe we must keep our offensive on these policies on an accurate basis or we just create opportunities for Grayling et al to drift away from the main issues. More interesting is where the estimates came from.
Post a Comment