Gallup poll. [Graphic=Yonhap News]

Polls have long been used by politicians and the media as a 'barometer of public sentiment'. But how reliable are these figures? 

While there have been no confirmed cases of manipulation in national opinion polls, there is also no external method to verify whether manipulation has occurred. This impossibility of verification is at the core of the debate surrounding the reliability of polls. 

The problem lies not in a conspiracy by a specific organization, but in the flaws of the verification structure throughout the survey design and results processing, and the media's practices of not adequately scrutinizing these flaws.

This fact-check is not intended to determine the truthfulness of individual poll figures. Instead, it examines where and in what structure the distrust surrounding opinion polls has been generated.

To this end, the debate has been organized into three structural issues: institutional, statistical, and reporting practices.

The reduction in the number of issues is not to downplay the problem, but to explain the causes of recurring suspicions as structural issues rather than human ones.

There are disclosure standards, but no verification standards

Claims such as "the Election Commission controls the sample" or "state agencies design public opinion" are repeatedly made regarding the results of opinion polls. Who actually creates and controls the sample, the starting point of opinion polls?

In opinion polls, the entity that actually extracts the sample is the polling agency. 

The sampling frame for mobile phone polls is the anonymized number (virtual number) generated by telecommunications companies, which is provided without any combined political orientation or opinion information. Telecommunications companies only apply administrative and technical criteria such as the minimum age (18 years or older) and regional scope, and do not constitute samples by age group, gender, or political orientation. 

The Election Commission's Polls Scrutiny Committee only processes and manages this, and does not design samples or provide them reflecting political orientation. 

In landline surveys, number generation (RDD), sample composition, and determination of survey ratios are also within the judgment of the survey agency.

The claim that state agencies manipulate opinion poll samples is not institutionally confirmed. 

However, this does not mean there is no problem. 

While disclosure standards exist in Korea, there are no independent verification standards to filter the validity of survey results. This structure itself breeds distrust.

Response rates and weights are disclosed but cannot be verified

The phenomenon of poll results sharply rising or falling in a short period is repeated. Is this a drastic shift in public sentiment, or a limitation of the statistical structure?

The average response rate for opinion polls is generally around 3-5%. The results are determined by the small number of people who choose to respond among the many randomly contacted.  

Weights by gender, age, and region are then applied, and the opinions of groups with few respondents are reflected more broadly. Korea's disclosure standards mandate the disclosure of response rates but do not set a minimum response rate. There are no mechanisms to restrict disclosure regardless of whether the response rate is 3% or 10%.

The same applies to weights. 

At the time of disclosure, it is sufficient to state whether weights were applied and an overview of the variables used; there is no obligation to disclose the weighting values for each group or a comparison of figures before and after applying the weights.

Response rates and weights are key factors that influence the results of opinion polls, but Korea's system treats them as formal disclosure items rather than subjects for verification. 

The outside cannot distinguish whether the results are based on the actual distribution of responses or on weight adjustments. The fact that manipulation has not been proven and the fact that manipulation cannot be verified coexist. 

The media's uncritical acceptance has fueled distrust

How has the media handled these structural limitations?

In practice, the media does not apply its own statistical judgment criteria when quoting opinion polls. 

Reviews on whether the response rate is excessively low or whether weights have significantly influenced the results are rarely conducted during the article writing process. Instead, the media relies on the recognition and reputation of the polling organization as a basis for trust, rather than the survey methods or figures.

As a result, if a particular polling organization is perceived as 'authoritative,' its survey results are reported regardless of design limitations. Conversely, surveys by lesser-known organizations are often excluded even if they use the same methodology.

This is less a problem with the polling organizations and more a result of the media failing to establish its own standards for judgment. 

In the absence of response rate benchmarks and weight verification standards, the media has abandoned statistical judgment and used the names of polling organizations as a substitute for verification. As a result, distrust has accumulated.

Q&A on the possibility of manipulation 

Q. Have there been any confirmed cases of manipulation in opinion polls?

A. While there have been no confirmed cases in national surveys, they do exist in regional surveys.

Q. Then why is the debate over reliability repeated?

A. Although there have been no confirmed cases of manipulation in national opinion polls, there is no external method to verify whether manipulation has occurred. 

There are disclosure standards, but no verification standards to filter the impact of response rates and weights on the results. This gap amplifies distrust.

International Comparison

In the United States, the government does not intervene in opinion polls beforehand. Instead, if survey methods and results are not transparently disclosed, they lose credibility in academia, the media, and the market. The EU also does not provide samples from the state and emphasizes post-verification and accountability for disclosure.

In contrast, Korea has a system of prior management, but there is no way for external parties to verify whether this management has been appropriate. It is a structure of control without verification.

Minimum Solutions to Reduce the Impossibility of Verification

The crisis of confidence in opinion polls is not a matter of manipulation but of a verification gap. 

While the legitimate principle of protecting personal information prevents the disclosure of raw data, this does not necessitate the impossibility of all verification. There are clearly areas that can be verified without infringing on personal information.

Realistic alternatives include disclosing procedural logs (number of call attempts, drop-out rates, quota closing times) in aggregated form instead of raw data, and presenting changes and impact assessments before and after weight application, rather than the calculation formula for weights.

Setting a minimum response rate threshold, non-disclosure verification of some surveys by a third party, and, above all, the media establishing its own citation standards can be the starting point for regaining trust.

Opinion polls are not public sentiment itself, but an imperfect tool attempting to estimate public sentiment. 

The path to regaining trust lies not in stronger control, but in social consensus on how far to open verifiable structures. 

A system that explains how numbers are created should come first, rather than demanding that people believe the numbers.

By Kim Young, Reporter