J Korean Acad Fam Med Search

CLOSE


Korean J Fam Med > Epub ahead of print
Lee and Oh: Health Information Quality of Online Newspaper Articles in Korea

Abstract

Background

The Web is an important source of health information, but the quality of such online information is highly variable. This study evaluates the quality of health articles published on Naver News, Korea’s most popular portal, using the Health Information Quality Assessment Tool (HIQUAL).

Methods

We collected 712 health-related articles published on Naver News from May 1 to 7, 2023. After applying exclusion criteria, we selected 116 articles for analysis. Two clinicians independently assessed the quality of these articles using the HIQUAL, which scores articles based on five domains: “reliability,” “usefulness,” “understandability,” “sufficiency,” and “transparency.”

Results

Overall article quality was generally considered recommendable (mean±standard deviation: 7.52±2.00). “Usefulness,” one item of “reliability,” and “understandability” were the three items with the highest levels of satisfaction. “Sufficiency” criteria for costs, risks, and benefits received low scores. Quality scores for articles focused on health risk factors and intervention showed a statistically significant difference.

Conclusion

While the overall quality of health information in Korean online newspaper articles is acceptable, room for improvement remains in some areas, particularly with regard to the fair presentation of costs, risks, and benefits. The study highlights the need for ongoing quality improvement and evaluation initiatives for online health information.

INTRODUCTION

Since the birth of the internet, one issue that has received continuous attention is how to obtain health information online [1]. In the past, information that first appeared in the scientific literature was redistributed through the mass media [2], and people relied solely on traditional media and advice from a physician for their health information. Today, however, people seek information online before consulting a physician [3]. Advances in digital media technology have made the internet more accessible, creating additional opportunities to search for health information that was not readily available before, regardless of its reliability or accuracy. The internet has significantly changed how people obtain health information [4].
Experts believed that information consumers would value the same things in evaluating the quality of online information as they did in traditional print publications: authorship, sources, disclosure of conflicts of interest, and timeliness [1]. However, owing to poor access to complete information and the need for high levels of reading comprehension [5], readability and design were also considered important factors [6]. This underscores the role of primary healthcare providers as evaluators and trusted providers of health information in the face of a large information gap between providers and consumers [7].
As can be seen, many factors contribute to the quality of online health information. Moreover, the accuracy and reliability of health information varies considerably and can be misleading [8,9]. The same is true for press releases reporting medical research findings [10-13]. Health information is also considered a social determinant of health because it can have negative consequences if it is incomplete or inaccurate [14].
Multifaceted tools such as HONcode and DISCERN have been developed to assess, and ultimately improve, the quality of health information [15-17]. Evaluation tools have also been developed in Korea [18]. As a result, several studies have evaluated online health information on specific topics, allowing for comparability. However, general-purpose tools that can evaluate various types of online sources are generally rather complex, require a long time to evaluate, and have not proven reliable [19]. The Health Information Quality Assessment Tool (HIQUAL) is the only validated tool to assess the quality of information in online newspaper articles [20,21], but not one study has evaluated the quality of health information in online newspaper articles using this tool.
Therefore, this study seeks to assess the quality of online health-related articles published and distributed through a representative web portal in Korea using a validated tool.

METHODS

1. Collecting Articles for the Study

Naver, the most popular internet portal in South Korea, provides a news service by collecting articles from media organizations, dividing them into sections according to the categories classified by each media organization, and listing them in order of publication time. The health information section of the “Life & Culture” section of Naver News includes online news that cites new scientific research. This study has focused on articles published in the “Health” section of Naver News from May 1 to 7, 2023. Using the R-based package “N2H4” developed for collecting information on Naver News articles for research purposes, we collected the following information: Naver article uniform resource locator (URL), original article URL posted on each newspaper’s website, publication date, newspaper, article title, and part of the article text. This study met the exclusion criteria for review by the institutional review board as an observational study without human participants.

2. Selection and Categorization of Articles

The list of collected health articles was screened by a single evaluator (H.L.) by reviewing the titles and text according to specific criteria (Figure 1). Specifically, articles were excluded if (1) they did not contain health information such as policy trends or personnel transfers; (2) they contained health information but did not convey relatively new information; (3) they conveyed relatively current health information but did not meet the classification of health risk factors, diagnosis, treatment, and so forth (e.g., disease epidemiology, new drug release); and (4) they were published in different newspapers, but a significant portion of the article text overlapped, and the evaluation results were the same. The relative recency of an article was judged differently depending on the state of knowledge on the topic. For guidelines, the most up-to-date version was used as the standard, and for individual studies, publication within a period of at least 3 years was used as the standard, with a certain degree of adaptability depending on the topic and content. Of the 712 articles collected, 179 were eligible for analysis. The articles were categorized into three groups based on their content: risk factors, diagnostics, and interventions.
The selected and categorized articles were independently assessed by two clinicians (H.L., S.W.O.) experienced in assessing the quality of newspaper articles. If the reviewed articles were deemed inappropriate for inclusion in the study through discussion after the evaluation, they were excluded from the study. In cases of disagreement, the scores were reconciled through discussion to finalize the results. Finally, 116 articles were analyzed (Supplement 1).

3. Evaluation Tool

The information quality of each online newspaper article was assessed using the HIQUAL, which has demonstrated reliability and validity for assessing the quality of online newspaper articles. The HIQUAL consists of 10 items in five domains: “reliability,” “usefulness,” “understandability,” “sufficiency,” and “transparency” (Figure 2, Supplement 2). Each item is scored 0/1 or NA if not applicable.
To determine the extent to which each item in the evaluation tool was satisfied in the articles studied, we expressed the number of articles for which the item was satisfied, and the number of articles evaluated with that item, in other words, 116-NA in this study, as fractions and percentages. The total score, which represents the quality of the information in each article, is calculated by simply adding up the scores of the individual items. However, if any items are rated NA, the total score is calculated out of 10 by dividing the value added in the same way by the actual number of items rated, in other words, 10-NA, and then multiplying it by 10. In this case, the total score may have a decimal point. The same method was used to calculate the domain score. On the basis of the total score, we categorized the level of recommendation as “not recommended” for a score of 6 or less, “recommended” for a score of greater than 6 and up to 8, and “highly recommended” for a score of greater than 8 [21].

4. Statistical Analysis

All statistical analyses were performed using R ver. 4.3.1 (R Foundation for Statistical Computing, Vienna, Austria). Descriptive statistics such as frequencies, proportions, means, and standard deviations were used to characterize the quality score data. The Shapiro-Wilk normality test confirmed that the data did not follow a normal distribution, so the Kruskal-Wallis test was conducted to test the difference in total scores by article classification, and the Wilcoxon-Mann-Whitney test with Bonferroni correction was conducted as a post-hoc analysis. The corrected P<0.016 (0.05/3) was used as the statistical significance level.

RESULTS

The final quality scores for the 116 selected articles are shown in Table 1. Of the 10 items that make up HIQUAL, “understandability” had the highest item-level satisfaction rate, at 98.3% (114/116). Item 5, “usefulness,” which assesses the reader’s ability to use the information, and item 1 of “reliability,” which examines whether the information is medically valid and based on scientific evidence, had similarly high levels of satisfaction, at 92.2% (107/116). In contrast, item 7 of “sufficiency,” which assesses whether the benefits and harms of the information are presented together, and item 9 of “sufficiency,” which examines whether the costs associated with the information are explained, showed low item-level meet rates of 28.6% (10/35) and 30.0% (9/30), respectively. In addition, items 2 and 3 of “reliability,” which examine whether the source of the information is provided, the timing of the information, and whether the information is current, also showed low levels of satisfaction at 56.9% (66/116) and 59.5% (69/116), respectively. The overall quality rating score indicated that the quality of online newspaper articles was generally acceptable (mean±standard deviation: 7.52±2.00). The number of articles that could be highly recommended was 47 (Table 2).
We divided the 116 articles analyzed into three categories: 56 articles fell into the category of health risk factors (48.3%); seven articles fell into the category of diagnosis (6.0%); and 53 articles fell into the category of interventions (45.7%). The mean and standard deviations of the total score of the quality assessment were 8.23±1.80, 6.97±2.17, and 6.84±1.95, respectively, and the difference in the total score according to article categorization was statistically significant between the health risk factor and intervention groups (Figure 3).

DISCUSSION

Given the importance of determining how often individuals use online health information [22], the need to improve information quality has been emphasized on the assumption that information can have a significant impact on individuals’ health behaviors. To this end, many tools have been developed and applied, including key factors from both the provider and the consumer of information [15-17]. In this context, this study evaluated the quality of articles published in the health information section of Naver News, the most-used portal in Korea, during the first week of May 2023 using the HIQUAL tool. A total of 712 articles were published, of which 116 articles from 27 newspapers were finally selected and analyzed.
The results indicated that the overall quality of health information articles with new content was good. Nearly 40% of the articles were highly recommended, and less than 30% were not recommended, although the criteria for recommendation levels were somewhat arbitrary. This seems to be consistent with the reality that online news portal websites generally provide better-quality information than general websites [23].
If we look at each HIQUAL item, we find, first, that almost all articles satisfied the standard of “understandability,” which assesses whether the information provided is understandable at a layperson level. The same level of high satisfaction was observed with respect to “usefulness,” which reflects whether the reader can accept and use the information in the article, and item 1 of “reliability,” which examines whether the content is reasonable given existing textbook medical knowledge and is presented at an appropriate level of evidence. Taken together, the well-received items suggest that evidence-based knowledge is selected and presented in a way that readers can understand and use. This differs from studies evaluating online health information for pediatric epilepsy, preterm infants, and hand osteoarthritis, which reported low levels of “trustworthiness,” “quality,” and “understandability.” This divergence may be due to differences in the categories of information studied and in the evaluation tools [23-25].
Notably, less than one-third of the articles assessed for sufficiency stated the benefits, risks, and costs of the information provided, and that a significant number of articles contained errors or exaggerations. In particular, the statistical significance of the difference in scores between risk factor and intervention articles confirms that gaps identified in previous studies remain and that newly introduced interventions have an incentive to emphasize their benefits and effectiveness, as does the fact that all five articles that were not recommended fell into the intervention category [11-13]. Furthermore, the finding that nearly half of the articles did not clearly indicate the source and timing of the information is consistent with the results of an evaluation of online information on colorectal cancer in Korea and indicates that further efforts are needed to improve the quality of information [26].
We tested whether scores differed based on article classification in each of the five domains for all articles evaluated, and found that only the difference in scores for “transparency” was statistically significant (Supplement 3). Therefore, the difference in scoring on “transparency” may have contributed to the difference in total score.
To our knowledge, this study is the first to use a validated tool for online health articles in Korea, but it has several limitations. First, this study was conducted by selecting a specific, arbitrarily short time period. To eliminate the bias caused by selecting only articles published in a specific time period, all published articles were screened according to predefined systematic criteria, but the bias may have remained. We believe that screening and selecting published articles by random sampling or systematic clustering may be one way to eliminate this bias. Second, this study focused on targeted articles with relatively new medical content. Many articles about diseases or health-related information of high interest to a population contain no new findings, so numerous articles were excluded from the study. One additional limitation is that the quality of health information on common diseases was not evaluated in a sufficient number of articles. On the other hand, one strength of the study is that the evaluation category was limited to articles containing new information, which is known to be more likely to distort information, in order to identify specific points for improvement. Third, a high proportion of subjects rated NA on items in the “sufficiency” domain. We believe this is due to the inclusion of a large number of articles on health risk factors, for which it is not appropriate to evaluate all health-related articles published through the portal to determine benefits, harms, alternatives, and costs. Therefore, articles dealing with interventions or diagnostics to which all HIQUAL items are applicable should be the focus of future research. Fourth, the number of articles in the diagnosis category was too small to conduct a statistical analysis with high power. As is well known, it takes a considerable amount of time to develop new diagnostics in the medical field; nevertheless, a study with articles published over a longer period of time would be required to evaluate and discuss in depth all areas of the tool in the diagnosis category. Lastly, this study presents results from the perspective of two healthcare professionals. However, because the “understandability” items are intended to be scored from a layperson’s perspective, some form of proxy estimation may have occurred, resulting in an overestimation of the “comprehensibility” scores. While a full correction is unlikely, we believe that some correction may have been possible with a larger number of raters. Nevertheless, this study’s value consists in the following: it provides specific and objective data by classifying and rating the quality of online newspaper articles published on the most popular portal site in Korea using a standardized rating tool with demonstrated validity and reliability by two or more knowledgeable clinicians.
The overall good quality of the articles combined with the lack of improvement in terms of credibility and sufficiency suggests that our understanding of how to assess the quality and credibility of online health information remains limited. A recent meta-analysis study also reported that the quality of health information has a greater impact on health behavior intentions than on actual health behaviors, and that differences in individuals’ information use related to the quality of health information are minimal [27].
Therefore, it may be that the provider of online information and the surrounding support system available to the consumer, such as health professionals [28] or technological tools [29], are more important than the individual consumer. Since the time-consuming application of quality assessment tools has been identified as a blind spot [19], the application of technology to facilitate the two-way effort of improving health literacy at the individual level while simultaneously improving the quality of information could have many benefits [30].
In the future, efficient and ongoing evaluation will be needed to generate representative findings from a larger number of articles with a longer follow-up period, and to more closely analyze the factors necessary to improve the quality of online health information.

Notes

CONFLICT OF INTEREST

Seung-Won Oh is an editor of the Korean Journal of Family Medicine but had no role in the decision to publish this article. Except for that, no other potential conflict of interest relevant was reported.

SUPPLEMENTARY MATERIALS

Supplementary materials can be found via https://doi.org/10.4082/kjfm.23.0197.
Supplement 1.
List of health-related articles selected for study (published between May 1 and 7, 2023).
kjfm-23-0197-Supplementary-1.pdf
Supplement 2.
Health Information Quality Assessment Tool for health-related newspaper articles (Korean version).
kjfm-23-0197-Supplementary-2.pdf
Supplement 3.
Comparison of the quality scores between the categories and domains.
kjfm-23-0197-Supplementary-3.pdf

Figure. 1.
Schematic overview of the selection process for articles in the online newspaper.
kjfm-23-0197f1.jpg
Figure. 2.
Health Information Quality Assessment Tool for newspaper articles (English translated version). NA, not applicable.
kjfm-23-0197f2.jpg
Figure. 3.
Comparative boxplot of news article quality scores across three categories (N=116). P-values were calculated by Wilcoxon-Mann-Whitney test. NS, not significant. *P=0.00031.
kjfm-23-0197f3.jpg
Table 1.
Quality scores of the articles evaluated by Health Information Quality Assessment Tool
No. (%) Domain
Total
Reliability Usefulness Understandability Sufficiency* Transparency
Satisfaction - Item 1: 107/116 (92.2) Item 5: 107/116 (92.2) Item 6: 114/116 (98.3) Item 7: 10/35 (28.6) Item 10: 98/116 (84.5) -
Item 2: 66/116 (56.9) Item 8: 24/35 (68.6)
Item 3: 69/116 (59.5) Item 9: 9/30 (30.0)
Item 4: 73/116 (62.9)
Score - 2.72±1.25 0.92±0.27 0.98±0.13 1.13±0.91 0.84±0.36 7.52±2.00
Category of article
 Risk factor 56 (48.3) 2.86±1.24 0.95±0.23 1±0 1±0 0.96±0.19 8.23±1.80
 Diagnosis 7 (6.0) 3.29±1.11 0.71±0.49 1±0 0.4±0.89 0.71±0.49 6.97±2.17
 Intervention 53 (45.7) 2.49±1.25 0.92±0.27 0.96±0.19 1.26±0.89 0.74±0.45 6.84±1.95

Values are presented as number of articles (% of article satisfied each item or of article by category) or mean±standard deviation.

* Analysis was performed excluding not applicable values.

Table 2.
Number of articles according to quality score level (N=116)
Total score Risk factor (n=56) Diagnosis (n=7) Intervention (n=53) Articles by score level
≤6 11 (19.7) 4 (57.1) 5 (9.4) 34 (29.3)
6–8 12 (21.4) 1 (14.3) 36 (67.9) 35 (30.2)
>8 33 (58.9) 2 (28.6) 12 (22.6) 47 (40.5)

Values are presented as number of articles (% of articles in each category).

REFERENCES

1. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: caveant lector et viewor: let the reader and viewer beware. JAMA 1997;277:1244-5.
crossref pmid
2. Phillips DP, Kanter EJ, Bednarczyk B, Tastad PL. Importance of the lay press in the transmission of medical knowledge to the scientific community. N Engl J Med 1991;325:1180-3.
crossref pmid
3. Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, et al. Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med 2005;165:2618-24.
crossref pmid
4. Chen YY, Li CM, Liang JC, Tsai CC. Health information obtained from the Internet and changes in medical decision making: questionnaire development and cross-sectional survey. J Med Internet Res 2018;20:e47.
crossref pmid pmc
5. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001;285:2612-21.
crossref pmid pmc
6. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287:2691-700.
crossref pmid
7. Diaz JA, Griffith RA, Ng JJ, Reinert SE, Friedmann PD, Moulton AW. Patients’ use of the Internet for medical information. J Gen Intern Med 2002;17:180-5.
crossref pmid pmc
8. McGrath BM, Kapadia RK. Is the medium distorting the message?: how the news media communicates advances in medical research to the public. Dalhous Med J 2009;36:11-7.
crossref
9. McCaw BA, McGlade KJ, McElnay JC. Online health information: what the newspapers tell their readers: a systematic content analysis. BMC Public Health 2014;14:1316.
crossref pmid pmc pdf
10. Sumner P, Vivian-Griffiths S, Boivin J, Williams A, Venetis CA, Davies A, et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ 2014;349:g7015.
crossref pmid pmc
11. Schwitzer G. A guide to reading health care news stories. JAMA Intern Med 2014;174:1183-6.
crossref pmid
12. Haneef R, Lazarus C, Ravaud P, Yavchitz A, Boutron I. Interpretation of results of studies evaluating an intervention highlighted in Google Health News: a cross-sectional study of news. PLoS One 2015;10:e0140889.
crossref pmid pmc
13. Buhse S, Rahn AC, Bock M, Muhlhauser I. Causal interpretation of correlational studies: analysis of medical news on the website of the official journal for German physicians. PLoS One 2018;13:e0196833.
crossref pmid pmc
14. Morley J, Cowls J, Taddeo M, Floridi L. Public health in the information age: recognizing the infosphere as a social determinant of health. J Med Internet Res 2020;22:e19311.
crossref pmid pmc
15. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999;53:105-11.
crossref pmid pmc
16. Provost M, Koompalum D, Dong D, Martin BC. The initial development of the WebMedQual scale: domain assessment of the construct of quality of health web sites. Int J Med Inform 2006;75:42-57.
crossref pmid
17. Robillard JM, Jun JH, Lai JA, Feng TL. The QUEST for quality online health information: validation of a short quantitative tool. BMC Med Inform Decis Mak 2018;18:87.
crossref pmid pmc pdf
18. Kim MJ, Kang NM, Kim SW, Rhyu SW, Chang H, Hong SK, et al. Development of an evaluation checklist for internet health/disease information. J Korean Soc Med Inform 2006;12:283-92.
crossref pdf
19. Dobbins M, Watson S, Read K, Graham K, Yousefi Nooraie R, Levinson AJ. A tool that assesses the evidence, transparency, and usability of online health information: development and reliability assessment. JMIR Aging 2018;1:e3.
crossref pmid pmc
20. Lee N, Oh SW, Cho B, Myung SK, Hwang SS, Yoon GH. A health information quality assessment tool for Korean online newspaper articles: development study. J Med Internet Res 2021;23:e24436.
crossref pmid pmc
21. Cho B, Myung SK, Oh SW, Hwang SS, Yoon GH, Lee N, et al. Development of standardized assessment tools and methods for systematic evaluation of health information. Seoul: Korean Medical Association;; 2019.

22. Lemire M, Pare G, Sicotte C, Harvey C. Determinants of Internet use as a preferred source of information on personal health. Int J Med Inform 2008;77:723-34.
crossref pmid
23. Hong SW, Kang JH, Park JH, Park HJ, Kim E. Quality and readability of online information on hand osteoarthritis. Health Informatics J 2023;29:14604582231169297.
crossref pmid pdf
24. Cerminara C, Santarone ME, Casarelli L, Curatolo P, El Malhany N. Use of the DISCERN tool for evaluating web searches in childhood epilepsy. Epilepsy Behav 2014;41:119-21.
crossref pmid
25. Dol J, Richardson B, Boates T, Campbell-Yeo M. Learning to parent from Google?: evaluation of available online health evidence for parents of preterm infants requiring neonatal intensive care. Health Informatics J 2019;25:1265-77.
crossref pmid pdf
26. Sohn DK, Choi HS, Lee DU, Lee SJ, Lee JS, Lee YS. The quality evaluation of information of websites on colorectal cancer using the DISCERN instrument. J Korean Soc Coloproctol 2005;21:247-54.

27. Zhang Y, Kim Y. Consumers’ evaluation of web-based health information quality: meta-analysis. J Med Internet Res 2022;24:e36463.
crossref pmid pmc
28. Ioannidis JP, Stuart ME, Brownlee S, Strite SA. How to survive the medical misinformation mess. Eur J Clin Invest 2017;47:795-802.
crossref pmid pdf
29. D’Cruz MJ, Kini RB. The effect of information asymmetry on consumer driven health plans. In: Wang W, Li Y, Duan Z, Yan L, Li H, Yang X, editors. Integration and innovation orient to E-Society. Boston (MA): Springer; 2007. p. 353-62.

30. Cordero DA Jr. Promoting health literacy: preparing for future health crises. Korean J Fam Med 2023;44:124-5.
crossref pmid pmc pdf


ABOUT
ARTICLE CATEGORY

Browse all articles >

BROWSE ARTICLES
INFORMATION FOR AUTHORS AND REVIEWERS
Editorial Office
Room 2003, Gwanghwamun Officia, 92 Saemunan-ro, Jongno-gu, Seoul 03186, Korea
Tel: +82-2-3210-1537    Tax: +82-2-3210-1538    E-mail: kjfm@kafm.or.kr                

Copyright © 2024 by Korean Academy of Family Medicine.

Developed in M2PI

Close layer
prev next