A Shocking Story of Marks Tampering and Inflation: Data Mining CBSE Scoring for a decade

The Learning Point‎ > ‎Examination Results 2013: Class 10/Class 12/ICSE/ISC/CBSE/IIT JEE‎ > ‎

A Shocking Story of Marks Tampering and Inflation: Data Mining CBSE Scoring for a decade

Today, I'd like to bring closure to the series of Data Analysis experiments which I have been conducting with the school leaving scores of high-school students in India, attained in their final assessment at the end of Class 12. It is something which I have been doing since 2012, as a side-project. Ideally, one likes to bring closure to issues, when you have answers to most questions, but in my mind, the data mining experiments which I conducted, have spawned more and more questions, the deeper I went into the issue, and I haven't found a single answer yet. Addition (June 2014): I have analyzed 2014 data as well, and the marking problems have only increased.

The only thing I now know for sure (not that we didn't know this in an anecdotal way) is, that the ICSE-ISC and CBSE boards have a long drawn history of tampering, manipulating and inflating marks in an erratic and arbitrary manner; apart from rigging scores to make sure that even people who were supposed to fail in the examination, end up with a pass grade, so that political masters can be kept happy. Analyzing the histograms of the former (specially) shows that as much as 20 marks are being increased in some subjects.

It has been approximately two months since I wrote this blog:Exposing CBSE and ICSE: Statistical Insights into the True Lies on your Marksheets. While it is likely that you didn't read my post, if you're a young person in India, there's a high probability that you either read or heard about Debarghya Das's viral blog post about Hacking into the Indian Education System.

In the mean-time, there was an open-feud between the Indian Statistical Institute and the CBSE regarding board and entrance examinations and their normalization. Some documents, which were previously confidential and kept a secret from the public, were dumped into the open and one specific analysis from a reputed academic confirms what our blog posts had revealed: that mark tampering was rampant, it was evident from the curves, this feedback was known to authorities and they chose not to do anything about it. Please note that this comment was made only in the context of the distorted CBSE scores he saw (the rest of the scores he inspected were State Board scores, which, by and large, resembled known distributions. He wasn't provided the ICSE scores, though that would open the biggest can of worms.)

http://www.isical.ac.in/~jeexiiscore_normal/PAP04_ACER.pdf

A comment from Dr. Glenn Rowley from the Australian Council for Educational Research, about the CBSE Scores in Physics, Chemistry and Mathematics for 2012:

"These bear no resemblance to any known distribution, and defy explanation. They appear to

contain at least two separate populations. One is small, nearly normal, and at the lower end

of the reported scores. The other is larger and is almost completely separate from the first in

Physics and Chemistry and completely separate in Mathematics. For each subject, the

second, larger distribution has peaks that suggest human intervention to raise candidates over some score, such as a passing score or perhaps the score that required to gain some reward,

such as course selection or a scholarship. In Mathematics the distribution approximates a

uniform distribution, suggesting that the bulk of the scores submitted were rankings or

percentiles.

In the absence of any explanation for these irregularities, I cannot recommend that such data

could contribute usefully to tertiary selection. AIEEE Total scores appear to be quite wellsuited for tertiary selection, and to form any combination with scores distributed like these

would detract seriously from their effectiveness.

Fortunately, these problems were absent from the 2102 data that were provided next. The

remainder of the report will be based on 2012 data, which were free of these irregularities"

Today, I present scoring histograms, constructed from ten years of data mined from the web.

All of this has been mined from publicly available and accessible information, which has been made easier by the fact that most CBSE schools now have websites, and share details about their current and past results as well. Sites with a robots.txt were not mined for this process. Nearly six million records have gone into constructing these histograms. These are definitely not the complete set of records , but they are enough to provide us a very large sample, to identify trends in scoring patterns.

I will briefly summarize my findings over here, and you can take a look at my graphs and confirm them.

1. Distributions of scores in examinations are not expected to be erratic or jagged or spiky, nor do they have sudden and suspicious peaks or missing bars. I certainly don't expect to see a perfect bell shaped curve, but I do expect to see something which resembles it. It is fine if one side of the curve has a much steeper taper than the other, or if the curve is skewed - but the curve is expected to be reasonably smooth even if it is a bit lumpy at certain points (which possibly points to mixtures).

2. I present graphs of CBSE scores in a number of subjects, for the past decade (ten years of examinations from 2004-2013). Note the increasing level of distortion (as reflected by the increasing standard deviation as well) from the expected bell shaped curve year after year.

3. It is understandable that there will be a bit of a spike around the pass-mark (33 or so). Since evaluation is rather subjective, an examiner might not want to accidentally fail a student who scores even, say, 30 marks. Fair enough. What does not have any good explanation is the other spikes, specially the most frequent (and massive) spike at the 95 percent mark which is a routine feature of the graphs of subject scores specially in the 2011-2013 period. Let us walk through the histograms for a couple of subjects and see how scoring has, "evolved" with time.

4. Let us begin with an inspection of the English Core graphs. Notice how the graph for the year 2004 is rather close to what one would expect in a public examination taken by hundreds of thousands of students. There's a gap followed by a spike around the pass mark which is understandable (as I've explained in point 3). There's an unexpected spike at 75. But for the most part, the graph has tapering sides and a prominent hump. Over the years, however, the graph becomes spikier and spikier and starts to flatten out a bit, and finally in the year 2013, notice that massive spike at 95 where almost five percent of everyone who appeared for CBSE-2013 ended up with exactly the same score (95). Notice how the spike at 95 is non-existent in the 2004-2009 period and suddenly starts to appear in the graph from 2010 and finally shoots up like a beanstalk in the 2013 graph. Notice the gradual flattening of the curve from what was approximately a bell, to something artificially flattened out. Also notice, how the average score in this period increased by around 6 marks and the standard deviation increased by more than 3 marks (which is noticeably reflected in the visible increase of proportion of students in the highly-competitive zone of those scoring 85 or 90 or more).

5. Now let us move on to the Mathematics Score distributions beneath it. The 2004 curve is hardly a bell or a skewed-bell or a tapering-bell, but it at least shows something which resembles a bit of a hump. Notice the increasing size of the spike at 95, which appears after 2008. The 2013 curve has multiple spikes. And it resembles nothing close to what one would expect in a public examination taken by a large number of students. Also note the abnormally high standard deviation of the curve (increases from ~23 in 2004 to ~27 in 2013). Note the almost equal size of the histogram bars. And here I draw attention to the statement made by Dr. Glen Rowley in the document which I mentioned above: In Mathematics the distribution approximates a

uniform distribution, suggesting that the bulk of the scores submitted were rankings or

percentiles. It is completely unacceptable that a national board does something as deceptive and manipulative as this without a transparent explanation. The curves for Economics and Business Studies are on similar lines, again with a suspiciously high standard deviation, starting off with a somewhat curvy shape in 2004 and getting completely distorted and surprisingly flattened in 2013, with rectangular blocks and sudden, massive spikes in 2013, resembling the silhouette of the skyline of New York City more than a scoring distribution.

6. Notice the Physics, Chemistry, Biology scores and the surprising rise or peak around 50 or 55. And, once again, there's a weird spike at the 95 mark, which starts to show up in the plots of scores from 2009 and later. I have a theory for the surprising rise though it is hard to be certain about the exact reason. These three subjects have a practical component of 30% weight-age To pass, a student needs to score the pass mark in both theory as well as the practical. Assuming the board promotes students who are a bit(?) below the pass-mark, to a score of 22/70 in the Theory paper (which is the minimum required to pass in the Theory section). The practical examination scores are out of 30, and you probably just need to copy the answers from the person who copied the correct answers from someone else in the practical test: and wow, you get a nice 28-30 marks on 30. And the total of the barely-passed Theory score to the flying-colors Practical score works out to ~(22+30) = 52. Which explains the kind of sudden peak in the 50-55 region for these three subjects. Now, let's get back to what Dr. Glen Rowley said about the Physics and Chemistry scores. "....They appear to contain at least two separate populations. One is small, nearly normal, and at the lower end of the reported scores. The other is larger and is almost completely separate from the first in Physics and Chemistry and completely separate in Mathematics. For each subject, the second, larger distribution has peaks that suggest human intervention to raise candidates over some score, such as a passing score or perhaps the score that required to gain some reward, such as course selection or a scholarship."

I don't think that it is a co-incidence that it is on the left-half of these curves, where a lot of distortion is present. After all, that is the range where maximum tampering is required to inflate and tinker results and pass percentage rates.

7. History and Geography have a bit of a compression and sudden tower, somewhere in the fifties, but for the most part these graphs look much more authentic than the rest of the subjects.

8. Notice how every subject has its own shape and structure which gradually gets distorted more and more as we move from 2004 to 2013. The peaks, spikes and valleys occur at somewhat similar places, year to year.

9. The curves for Geography, History, Physical Education and Computer Science are pretty much in line with what one would expect in a scoring distribution barring a few topological anomalies. The reason is not too hard to decipher: coincidentally all of these are subjects, where the average score is either in the early seventies or the late sixties - much better than the average performance in other subjects. Since the scores were already pretty good, there wasn't really much of a need to tamper these marks.

10. The big spike at 95 and the sudden drop after it is actually a big cover up. Scores are presumably inflated and pushed till a certain point and not beyond that. Why might this be the case? Because if too many people suddenly jumped the 95 barrier as well, college admission cut-offs would soar upto 100% immediately, everyone would be curious to see what kind of examinations result in such high marks and all eyes of suspicion and scrutiny would be on the board. Coincidentally, 95 also happens to be the mark after which the equally generous ISC-ICSE board becomes extremely stingy about the marks it hands out as you might find in the graphs which I plotted over here in my last article Exposing CBSE and ICSE. I'm wondering if there is some unofficial agreement between these boards about how scores are to be rigged in this unique educational match fixing game.

11. In curves like those for Physics, Chemistry, Biology for the 2004-2009 period there's one surprising peak around the 50-55 mark range. Note that it transforms into two close peaks around 2010, which become more and more prominent around 2013. And the curvy side on the ride, becomes almost like a straight edge of a triangle (ending with a spike at 95) as time passes.

12. We've discussed these strange scoring patterns enough for now. What are the implications which come to my mind?

(a) When it comes to college admissions, students from other boards (state boards) will be unfairly disadvantaged.

(b) Not only students from other boards, but good students from CBSE (or for that matter ICSE) themselves will be unfairly disadvantaged because there will be a sudden compression of marks in the high-scoring range, because of which the examination loses its ability to differentiate between good and very-good students. Each mark lost will result in a sudden, unexpected drop of percentile in the 90+ zone.

(c) Students, parents, schools and employers have reasons to have no confidence on the scores which end up on marksheets.

13. Some might jump on me and say, that it is extremely naive to apply the central limit theorem blindly and expect a perfect bell shaped Gaussian. I am not expecting an ideal Gaussian from a Math book. I am only expecting something which resembles it. It might be lumpy (different kinds of student groups: leading to mixtures) and it might be skewed (depending on how easy or difficult the examinations were) and it might be slightly uneven (depending on the marks assigned for each question in the exam and the scoring blocks). But there is certainly no reason to accept a complete breakdown of the Gaussian structure. I am not alone in my assumption.

Once again I quote this line from Dr. Glen Rowley's report: "In populations of the size we are dealing with here, distributions of achievement will be normal, or approximately so".

14. The Boards are taking cover of what they call a Moderation Policy to make adjustments to scores. That is a big cover up as Dr. Dheeraj Sanghi, Professor of Computer Science at IIT Kanpur has nicely explained in his blog. His blog is no longer public but you can take a look at his cached post over here.

I think I have put enough numbers and data into the public domain and this is my last post on this topic for now. Administrators should stop behaving like bulls in china shops by tinkering around with marking patterns and completely randomizing our already broken and non-transparent education system. And HRD should stop announcing bizarre policies to make comparisons and rankings across 40 different boards.