The accusation is - three prominent Western think-tanks have used less than shoddy methodology to declare Indian as ranking lowly on democratic parameters.
The Working Paper can be accessed here.
The Economic Advisory Council to the Prime Minister of India (EAC-PM) has recently come up with a report titled Why India Does Poorly On Global Perception Indices to assess the credibility of global expert opinion based indices. The report, authored by economists Sanjeev Sanyal and Aakshanksha Arora, primarily questions the methodology and structure of three major indices - Freedom in the World Index, Economist Intelligence Unit’s Democracy Index and the Variety of Democracy Indices.
The prime argument of the report deals with the idea that there is a lack of objective basis to these indices due to the methodology they adopt which makes their assessments very subjective. Majority of the data obtained through the indices is based on the perceptions and opinions of a ‘handful of experts’, the authors assert. While these indices claim to measure the subjective processes of ‘democracy’ and ‘freedom’ , it becomes all the more important to question their methodology and identify their ‘objective’ conclusions.
The scores obtained by states in these indices are also used to have an objective comparison among different states. Sanjeev Sanyal and Aakanksha Arora claim in this regard that the criterion used to obtain scores are based on ‘expert’ perceptions and may not be viable to have an objective comparison among the states being measured. Moreover, the factors of democracy and freedom are also subjectively perceived by the people residing within the states, therefore solely basing the results on the opinions of experts may yield inaccurate results.
Now, the other major subject of this report deals with the fact that India has seen a repeated decline in its position obtained in these indices. It is important to consider what this could mean for the perception of the country and whether it could have a rather tangible impact in terms of the country’s economic standing as well as sovereign ratings. To understand this further, let us have a deeper look into the indices at the focus of this report and how they assess India’s position.
Freedom in the World Index
This index primarily deals with scoring the countries based on two parameters- political rights and civil liberties.
The annual report is a product of Freedom House which is a non governmental organisation based in New York, United States. The 2022 report charts out a gradual decline in democratic institutions around the world and claims that 8 out 10 individuals live in a ‘partly free’ country. This brings us to the criteria of categorization that the report uses based on the scores obtained.
The countries being assessed fall into one of the three categories - Free, Partly Free and Not Free. This particular index reveals that India has declined about 11 points in its aggregate score and comes under the category of “partly free”.
This brings us to another pertinent question, how are these conclusions drawn?
The Economic Advisory Council, in its report, suggests that the Freedom in the World Index uses a set of twenty five questions which are answered by key experts, consultants and analysts. One assessment of the final report would make it clear that there is no mention of who these experts or consultants are and what the basis of their responses to the given questions would be. This is an important aspect to consider in this regard because of the kinds of questions that are eventually asked in the assessment. It is revealed that the questions are quite subjective and would highly depend on the perception level as well as the context of the person answering them.
Another important issue of contention pointed out by the authors is that the index arbitrarily portrayed Jammu and Kashmir as a separate territory to India. The Union Territories were even given a separate rating and were classified as “not free” by the Freedom in the World Index.
The Economist Intelligence Unit (EIU)- Democracy Index
The next index in the list assessed by the authors of the report in question is the EIU - Democracy Index. The index primarily focuses on the democratic processes within the states and the situation of political culture as well as participation. The latest issue of this index claims that “less than half of the world lives in a democracy”.
There are five categories that this index majorly looks into - Electoral process and pluralism, Functioning of government, Political participation, Political culture and Civil liberties. There are about sixty questions spread across these parameters and the final scores obtained is the average of the scores of these individual categories.
Now, based on these scores obtained, the countries are divided into four categories. The countries could be classified as - full democracies, flawed democracies, hybrid democracies and authoritarian regimes.
An analysis of the questions posed by the index reveals that they can hardly be objectively scored. For example, questions such as “how pervasive is corruption” would require a much more in-depth answer than just a number across the scale. Not to mention that these questions are again answered by the people who are claimed to be ‘experts’ but still their identity is not revealed by the actual report. It could be very legitimately questioned as to how capable the experts themselves are in answering these questions with utmost accuracy and what definite information are they basing their responses on.
The next important aspect of this index is the ‘public opinion poll’. One big fundamental flaw as highlighted by the authors of the FAC report is that the suitable public-opinion surveys are only available for a few questions in the index and the rest of the questions are again just based on the expert opinions. This factor in itself points to the inconsistencies that exist while developing the analysis for this index.
When it comes to the position of India within this index, it is revealed that the country comes under the category of a ‘flawed democracy’. This has been the case since the very first edition of the index back in 2006. The scores obtained by the country have also declined over the period and majorly in the parameters of civil liberties, political culture as well as electoral process and pluralism. The categorisation of various countries is further questioned by the fact that there are countries like Malaysia with a higher score than that of India. The example of Malaysia is particularly highlighted in the report because of the ‘special’ political status that is accorded to the indigenous Malays and several other tribes in the country’s constitution. For an index measuring the democratic process and political participation, this should have been an important factor to consider.
The Variety of Democracies (V-DEM) Indices
The last category of indices assessed in this report are the ones produced by the Variety of Democracy Institute in Sweden. These indices include a set of different indices that are used to measure the various components of democracies. They involve- Liberal Democracy Index, Electoral Democracy Index, Liberal Component Index, Egalitarian Component Index, Participatory Component Index and the Deliberative Component Index. More information on these specific indices can be found here.
The latest issue of the indices show that the county score for India has reduced primarily in the Liberal Democracy Index, Electoral Democracy Index as well as in the Deliberative Component Index.
The inconsistent basis of these measures are pointed out in the report by Sanjeev Sanyal and Aakansksha Arora. They suggest that some of the elements in the index cannot be used to measure all the countries on the same base. For example, the fact that the Participatory Component Index uses Direct Vote as a base to establish public participation in a democracy. A direct voting system may be possible for smaller countries but it is simply not a viable option for a national level political participation in a country like India. This brings to question how this particular parameter can be an equal source for comparison among the given countries.
The different sets of inconsistencies found in the indices analysed in the report and the ambiguity of their methodology poses certain question marks on the reliability of the data.
How reliable are these indices?
The dwindling scores of India in these indices might seem worrisome on the surface but it is also important to understand the process behind the development of these indices. It has been established that there is a severe lack of objectivity in the results produced by their data.
However, whatever the process of these indices may be, it cannot be ignored that their results and rankings of the states have an eventual impact on the international perception of the countries in question. So, even though the questions in the measures are answered by the experts based on their ‘opinions’, they end up having a tangible effect on the countries. The data can be used to influence the important international rankings such as the World Bank’s Worldwide Governance Indicators (WGI). Thus, it is very important to repeatedly question the methodology of these independent indices.
The lack of transparency over the methodology chosen for their studies is a prime factor that leads to a questionable reliability of the indices. There is no information on who these experts are, what they specialise in and how they are chosen.
As far as the rankings of India are concerned, in order to have a context specific overview of the processes of democracy and freedom, it would be imperative for India to rely more on the institutions at home than the ones in the west claiming to have an expert opinion on this country. The argument of the authors follows that these perception based indices often base their ‘opinions’ on a handful of media reports as well as just focus on the information that confirms their pre-existing biases. This factor could be effectively countered if the independent Indian research organisations carry out their specific empirical studies.