Skip to main content

Advertisement

SpringerLink
Log in

Equivalence of electronic and paper-based patient-reported outcome measures

  • Review
  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Aim

Electronic formats (ePROs) of paper-based patient-reported outcomes (PROs) should be validated before they can be reliably used. This review aimed to examine studies investigating measurement equivalence between ePROs and their paper originals to identify methodologies used and to determine the extent of such validation.

Methods

Three databases (OvidSP, Web of Science and PubMed) were searched using a set of keywords. Results were examined for compliance with inclusion criteria. Articles or abstracts that directly compared screen-based electronic versions of PROs with their validated paper-based originals, with regard to their measurement equivalence, were included. Publications were excluded if the only instruments reported were stand-alone visual analogue scales or interactive voice response formats. Papers published before 2007 were excluded, as a previous meta-analysis examined papers published before this time.

Results

Fifty-five studies investigating 79 instruments met the inclusion criteria. 53 % of the 79 instruments studied were condition specific. Several instruments, such as the SF-36, were reported in more than one publication. The most frequently reported formats for ePROs were Web-based versions. In 78 % of the publications, there was evidence of equivalence or comparability between the two formats as judged by study authors. Of the 30 publications that provided preference data, 87 % found that overall participants preferred the electronic format.

Conclusions

When examining equivalence between paper and electronic versions of PROs, formats are usually judged by authors to be equivalent. Participants prefer electronic formats. This literature review gives encouragement to the further widespread development and use of ePROs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Center for Drug Evaluation and Research (CDER). (2009). Guidance for industry—patient-reported outcome measures: Use in medical product development to support labeling claims. Silver Spring: Food and Drug Administration.

    Google Scholar 

  2. Leidy, N. K., & Vernon, M. (2008). Perspectives on patient-reported outcomes: content validity and qualitative research in a changing clinical trial environment. Pharmacoeconomics, 26(5), 363–370.

    Article  PubMed  Google Scholar 

  3. Marshall, S., Haywood, K., & Fitzpatrick, R. (2006). Impact of patient-reported outcome measures on routine practice: A structured review. Journal of Evaluation in Clinical Practice, 12(5), 559–568.

    Article  PubMed  Google Scholar 

  4. Lee, S. J., Kavanaugh, A., & Lenert, L. (2007). Electronic and computer-generated patient questionnaires in standard care. Best Practice and Research Clinical Rheumatology, 21(4), 637–647.

    Article  PubMed  Google Scholar 

  5. Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429.

    Article  PubMed  Google Scholar 

  6. Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value in Health, 11(2), 322–333.

    Article  PubMed  Google Scholar 

  7. Arthur, C. (2012). The history of smartphones: timeline. The Guardian, 24 January 2012 (Online). http://www.theguardian.com/technology/2012/jan/24/smartphones-timeline. Accessed 2 May 2014.

  8. McLellan, C. (2014). The History of Tablet Computers: A timeline (Online). http://www.zdnet.com/the-history-of-tablet-computers-a-timeline-7000026555/. Accessed 2 May 2014.

  9. Salaffi, F., Gasparini, S., & Grassi, W. (2009). The use of computer touch-screen technology for the collection of the patient-reported outcome data in rheumatoid arthritis: Comparison to standardised patient questionnaires. Clinical and Experimental Rheumatology, 27(3), 459–468.

    CAS  PubMed  Google Scholar 

  10. MacKenzie, H., Thavaneswaran, A., Chandran, V., & Gladman, D. D. (2011). Patient-reported outcome in psoriatic arthritis: A comparison of web-based versus paper-completed questionnaires. The Journal of Rheumatology, 38(12), 2619–2624.

    Article  PubMed  Google Scholar 

  11. Wu, R. C., Thorpe, K., Math, M., Ross, H., Micevski, V., Marquez, C., Straus, S. E. (2009). Comparing administration of questionnaires via the Internet to pen-and-paper in patients with heart failure: Randomised controlled trial. Journal of Medical Internet Research (Online), 11(1). http://www.jmir.org/2009/1/e3/. Accessed 1 May 2014.

  12. Juniper, E. F., Riis, B., & Juniper, B. A. (2007). Development and validation of an electronic version of the Rhinoconjunctivitis quality of life questionnaire. Allergy, 62(9), 1091–1093.

    Article  CAS  PubMed  Google Scholar 

  13. Ring, A. E., Cheong, K. A., Watkins, C. L., Meddis, D., Cella, D., & Harper, P. G. (2008). A randomised study of electronic diary versus paper and pencil collection of patient-reported outcomes in patients with non-small cell lung cancer. The Patient: Patient-Centered Outcomes Research, 1(2), 105–113.

    Article  Google Scholar 

  14. Juniper, E. F., Langlands, J. M., & Juniper, B. A. (2009). Patients may respond differently to paper and electronic versions of the same questionnaires. Respiratory Medicine, 103(6), 932–934.

    Article  PubMed  Google Scholar 

  15. Naus, M. J., Philipp, L. M., & Samsi, M. (2009). From paper to pixels: A comparison of paper and computer formats in psychological assessment. Computers in Human Behavior, 25(1), 1–7.

    Article  Google Scholar 

  16. Ribeiro, C., Moreira, L., Silveira, A., Silva, I., Gestal, J., & Vasconcelos, C. (2010). Development and use of touch-screen computer-assisted self-interview in Portuguese patients with chronic immune disease: Evaluation of an electronic version of SF-36v2. Acta Reumatológica Portuguesa, 35(2), 208–214.

    PubMed  Google Scholar 

  17. Matthew, A. G., Currie, K. L., Irvine, J., Ritvo, P., Mina, D. S., Jamnicky, L., Nam, R., Trachtenberg, J. (2007). Serial personal digital assistant data capture of health-related quality of life: A randomised controlled trial in a prostate cancer clinic. Health and Quality of Life Outcomes (Online), 5(38). http://www.hqlo.com/content/5/1/38. Accessed 30 April 2014.

  18. Schefte, D. B., & Hetland, M. L. (2010). An open-source, self-explanatory touch screen in routine care. Validity of filling in the bath measures on ankylosing spondylitis disease activity index, function index, the health activity index, function index, the health assessment questionnaire and visual analogue scales in comparison with paper versions. Rheumatology, 49(1), 99–104.

    Article  PubMed  Google Scholar 

  19. Handa, V. L., Barber, M. D., Young, S. B., Aronson, M. P., Morse, A., & Cundiff, G. W. (2008). Paper versus web-based administration of the pelvic floor distress inventory 20 and the pelvic floor impact questionnaire 7. International Urogynecology Journal, 19(10), 1331–1335.

    Article  Google Scholar 

  20. Swartz, R. J., Moor, C. D., Cook, K. F., Fouladi, R. T., Basen-Engquist, K., Eng, C., & Taylor, C. L. C. (2007). Mode effects in the center for epidemiological studies depression (CES-D) scale: Personal digital assistant versus paper and pencil administration. Quality of Life Research, 16(5), 803–813.

    Article  PubMed  Google Scholar 

  21. Carlbring, P., Brunt, S., Bohman, S., Austin, D., Richards, J., Öst, L.-G., & Andersson, G. (2007). Internet versus paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Computers in Human Behaviour, 23(3), 1421–1434.

    Article  Google Scholar 

  22. Clayer, M., & Davis, A. (2011). Can the Toronto extremity salvage score produce reliable results when used online? Clinical Orthopaedics and Related Research, 469(6), 1750–1756.

    Article  PubMed Central  PubMed  Google Scholar 

  23. Frennered, K., Hägg, O., & Wessberg, P. (2010). Validity of a computer touch-screen questionnaire system in back patients. Spine, 35(6), 697–703.

    Article  PubMed  Google Scholar 

  24. Oliveira, A., Ferreira, P. L., Antunes, B., & Pimentel, F. L. (2011). OnQol: Electronic device to capture QoL data in oncology: Difference between patients 65 years or older and patients younger than 65 years of age. Journal of Geriatric Oncology, 2(4), 253–258.

    Article  Google Scholar 

  25. Saunders, G., Forsline, A., & Jacobs, P. (2007). The Attitudes towards Loss of Hearing Questionnaire (ALHQ): A comparison of paper and electronic formats. The Journal of the American Academy of Audiology, 18(1), 66–77.

    Article  Google Scholar 

  26. Silveira, A., Gonçalves, J., Sequeira, T., Ribeiro, C., Lopes, C., Monteiro, E., & Pimentel, F. L. (2011). Computer-based quality-of-life monitoring in head and neck cancer patients: A validation model using the EORTC-QLQ C30 and EORTC-H&N35 Portuguese PC-software version. Acta Médica Portuguesa, 24(S2), 347–354.

    PubMed  Google Scholar 

  27. Wright, E. P., Kiely, M., Johnston, C., Smith, A. B., Cull, A., & Selby, P. J. (2005). Development and evaluation of an instrument to assess social difficulties in routine oncology practice. Quality of Life Research, 14(2), 373–386.

    Article  CAS  PubMed  Google Scholar 

  28. Beaumont, J. L., Victorson, D., Su, J., Baker, C. L., Wortman, K., Shah, H., & Cella, D. (2011). Examining web equivalence and risk factor sensitivity of the COPD Population Screener. Value in Health, 14(4), 506–512.

    Article  PubMed  Google Scholar 

  29. Bernstein, A. N., Levinson, A. W., Hobbs, A. R., Lavery, H. J., & Samadi, D. B. (2013). Validation of online administration of the sexual health inventory for men. The Journal of Urology, 189(4), 1456–1461.

    Article  PubMed  Google Scholar 

  30. Bruce, B., & Fries, J. F. (2011). Internet versus mailed administration of the health assessment questionnaire disability index. 63(10), pp. S1–S1 256.

  31. Chang, Y-J., Chang, C-H., Peng, C-L., Wu, H-C., Lin, H-C., Wang, J-Y., Li, T-C., Yeh, Y-C., Liang, W-M. (2014). Measurement equivalence and feasibility of the EORTC QLQ-PR25: paper-and-pencil versus touch-screen administration. Health and Quality of Life Outcomes (Online), 12(23). http://www.hqlo.com/content/12/1/23. Accessed 29 April 2014.

  32. Chen, H-L., Tien, S-W., Shih, C-C. (2011). Paper questionnaire versus Web questionnaire for clinical research impact using the short form of the UDI-6, HQ-7, PISQ-12. International Conference on Engineering and Business Management (EBM2011), 1–6, pp. 1293–1297.

  33. Chen, T-h, Li, L., Sigle, J. M., Du, Y.-P., Wang, H.-M., & Lei, J. (2007). Crossover randomised controlled trial of the electronic version of the Chinese SF-36. Journal of Zheijang University Science B, 8(8), 604–608.

    Article  Google Scholar 

  34. Chen, W.-C., Wang, J.-D., Hwang, J.-S., Chen, C.-C., Wu, C.-H., & Yao, G. (2009). Can the web-form WHOQOL-BREF be an alternative to the paper-form? Social Indicators Research, 94(1), 97–114.

    Article  Google Scholar 

  35. Chen, T-h, & Li, L. (2010). Pilot study of equivalence between the electronic and paper version of the Chinese SF-36. Journal of Happiness Studies, 11(2), 151–161.

    Article  Google Scholar 

  36. Dalal, A. A., Nelson, L., Gilligan, T., McLeod, L., Lewis, S., & De Muro-Mercon, C. (2011). Evaluating patient-reported outcome measurement comparability between paper and alternative versions, using the lung function questionnaire as an example. Value in Health, 14(5), 712–720.

    Article  PubMed  Google Scholar 

  37. Dinkel, A., Berg, P., Pirker, C., Geinitz, H., Sehlen, S., Emrich, M., et al. (2010). Routine psychosocial distress screening in radiotherapy: implementation and evaluation of a computerised procedure. British Journal of Cancer, 103, 1489–1495.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  38. Gudbergsen, H., Bartels, E. M., Krusager, P., Waehrens, E. E., Christensen, R., Danneskoild-Samsöe, B., Bliddal, H. (2011). Test-retest of computerised health status questionnaires frequently used in the monitoring of knee osteoarthritis: a randomised crossover trial. BMC Musculoskeletal Disorders (Online), 12(190). http://www.biomedcentral.com/1471-2474/12/190. Accessed 29 April 2014.

  39. Hedman, E., Ljótsson, B., Rück, C., Furmark, T., Carlbring, P., Lindefors, N., & Andersson, G. (2010). Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation. Computers in Human Behaviour, 26(4), 736–740.

    Article  Google Scholar 

  40. Heiberg, T., Kvien, T. K., Dale, Ø., Mowinckel, P., Aanerud, G. J., Songe-Møller, A. B., et al. (2007). Daily health status registration (patient diary) in patients with rheumatoid arthritis: A comparison between personal digital assistant and paper-pencil format. Arthritis Care & Research, 57(3), 454–460.

    Article  Google Scholar 

  41. Holländare, F., Andersson, G., Engström, I. (2010). A comparison of psychometric properties between Internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. Journal of Medical Internet Research (Online), 12(5). http://www.jmir.org/2010/5/e49/. Accessed 1 May 2014.

  42. Hollen, P. J., Gralla, R. J., Stewart, J. A., Meharchand, J. M., Wierzbicki, R., & Leighl, N. (2013). Can a computerised format replace a paper form in PRO and HRQL evaluation? Psychometric testing of the computer-assisted LCSS instrument (eLCSS-QL). Supportive Care in Cancer, 21(1), 165–172.

    Article  PubMed  Google Scholar 

  43. Howell, R. T., Rodzon, K. S., Kurai, M., & Sanchez, A. H. (2010). A validation of well-being and happiness surveys for administration via the Internet. Behaviour Research Methods, 42(3), 775–784.

    Article  Google Scholar 

  44. Lalanne, C., Herrmann, S., Armstrong, A. R., Cheung-Lung, C., Schwartz, Y., Chassany, O., & Duracinsky, M. (2013). Paper-based and electronic assessment of health-related quality of life specific HIV disease: A reliability study with the PROQOL-HIV questionnaire. Value in Health, 16(7), A362.

    Article  Google Scholar 

  45. Lee, E.-H. (2009). Touch-screen computerised quality-of-life assessment for patients with cancer. Asian Nursing Research, 3(1), 41–48.

    Article  PubMed  Google Scholar 

  46. Lee, E.-H. (2009). Computerised measurement for asthma-specific quality of life: Comparison with a conventional paper-and-pencil questionnaire. Journal of Korean Academy Nursing, 39(6), 781–787.

    Article  Google Scholar 

  47. Lee, E-H., Lee, Y.W., Lee, K-W., Kim, D.J., Kim, Y-S., Nam, M-S. (2013). Measurement equivalence of touch-screen computerised and paper-based diabetes-specific quality-of-life questionnaires. International Journal of Nursing Practice (Online). http://onlinelibrary.wiley.com/doi/10.1111/ijn.12184/abstract. Accessed 30 April 2014.

  48. Minard, J. P., Thomas, N., Olajos-Clow, J., Juniper, E. F., Jiang, X., Jenkins, B., Taite, A. K., Turcotte, S., Lougheed, M. D. (2011)a. Validation of an electronic version of the Pediatric Caregiver’s Asthma Quality of Life Questionnaire (PACQLQ). American Journal of Respiratory and Critical Care Medicine (Online),183. http://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A1432. Accessed 30 April 2014.

  49. Minard, J.P., Thomas, N., Olajos-Clow, J., Juniper, E. F., Jiang, X., Jenkins, B., Taite, A. K., Turcotte, S., Lougheed, M. D. (2011)b. Validation of an electronic version of the Mini Pediatric Asthmas Quality of Life Questionnaire (Mini PAQLQ). American Journal of Respiratory and Critical Care Medicine (Online). http://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A1433. Accessed 30 April 2014.

  50. Olajos-Clow, J., Minard, J., Szpiro, K., Juniper, E. F., Turcotte, S., Jiang, X., et al. (2010). Validation of an electronic version of the Mini Asthma quality of life questionnaire. Respiratory Medicine, 104(5), 658–667.

    Article  CAS  PubMed  Google Scholar 

  51. Oliveira, A., Ferreira, P. L., Antunes, B., & Pimentel, F. L. (2010). Quality of life in oncology: Electronic device to collect data. Acta Médica Portuguesa, 23(6), 1017–1024.

    PubMed  Google Scholar 

  52. Parnell, B. A., Dunivan, G. C., Connolly, A., Jannelli, M. L., Wells, E. C., & Geller, E. J. (2011). Validation of web-based administration of the pelvic organ prolapse/urinary incontinence sexual function questionnaire (PISQ-12). International Urogynecology Journal, 22(3), 357–361.

    Article  PubMed  Google Scholar 

  53. Raat, H., Mangunkusumo, R. T., Landgraf, J. M., Kloek, G., & Brug, J. (2007). Feasibility, reliability and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): Internet administration compared with the standard paper version. Quality of Life Research, 16(4), 675–685.

    Article  PubMed Central  PubMed  Google Scholar 

  54. Ribeiro, C., Silveira, A., Silva, I., Ribeiro, C., Gestal, J., & Vasconcelos, C. (2011). Computerised information-gathering in patients with lupus: An initial evaluation of an electronic version of the short form 36 version 2. Lupus, 20(4), 402.

    Google Scholar 

  55. Richter, J. G., Becker, A., Koch, T., Nixdorf, M., Willers, R., Monser, R., et al. (2008). Self-assessments of patients via Tablet PC in routine patient care: Comparison with standardised paper questionnaires. Annals of the Rheumatic Diseases, 67(12), 1739–1741.

    Article  CAS  PubMed  Google Scholar 

  56. Sage, J. M., Ali, A., Farrell, J., Huggins, J. L., Covert, K., Eskra, D., et al. (2012). Moving into the electronic age: Validation of rheumatology self-assessment questionnaires on tablet computers. Arthritis and Rheumatism, 64(S10), S1102.

    Google Scholar 

  57. Salaffi, F., Gasparini, S., Ciapetti, A., Gutierrez, M., & Grassi, W. (2013). Usability of an innovative and interactive electronic system for collection of patient-reported data in axial spondyloarthritis: Comparison with the traditional paper-administered format. Rheumatology, 52(11), 2062–2070.

    Article  PubMed  Google Scholar 

  58. Schemmann, D., Rudolph, J., Haas, H., & Müller-Stromberg, J. (2013). Validation and patient acceptance of a touch tablet version of the iHOT-12 questionnaire. Arthroscopy, 29(12), E188.

    Article  Google Scholar 

  59. Sjöstrom, M., Stenlund, H., Johansson, S., Umefjord, G., & Samuelsson, E. (2012). Stress urinary incontinence and quality of life: A reliability study of a condition-specific instrument in paper and web-based versions. Neurology and Urodynamics, 31(8), 1242–1246.

    Article  Google Scholar 

  60. Twiss, J., McKenna, S., Graham, J. E., Swetz, K. M., Sloan, J., & Gomberg-Maitland, M. (2013). Assessing measurement equivalence of different forms of administration of the Cambridge Pulmonary Hypertension Outcome Review (CAMPHOR) using Rasch analysis. Value in Health, 16(7), A606.

    Article  Google Scholar 

  61. Varni, J. W., Limbers, C. A., Burwinkle, T. M., Bryant, W. P., & Wilson, D. P. (2008). The ePedsQL™ in Type 1 and Type 2 diabetes: Feasibility, reliability, and validity of the Pediatric Quality of Life Inventory™ Internet administration. Diabetes Care, 31(4), 672–677.

    Article  PubMed  Google Scholar 

  62. Vinney, L. A., Grade, J. D., & Connor, N. P. (2012). Feasibility of using a handheld electronic device for the collection of patient reported outcomes data from children. Journal of Communication Disorders, 45(1), 12–19.

    Article  PubMed Central  PubMed  Google Scholar 

  63. Young, N. L., Varni, J. W., Snider, L., McCormick, A., Sawatzky, B., Scott, M., et al. (2009). The Internet is valid and reliable for child-report: An example using the Activities scale for kids (ASK) and the Pediatric quality of life inventory (PedsQL). Journal of Clinical Epidemiology, 62(3), 314–320.

    Article  PubMed  Google Scholar 

  64. Zimmerman, M., & Martinez, J. H. (2012). Web-based assessment of depression in patients treated in clinical practice: Reliability, validity and patient acceptance. Journal of Clinical Psychiatry, 73(3), 333–338.

    Article  PubMed  Google Scholar 

Download references

Conflict of interest

AYF is joint copyright owner of the DLQI.

Author information

Authors and Affiliations

  1. Centre for Socioeconomic Research, School of Pharmacy and Pharmaceutical Sciences, Cardiff, UK

    Niloufar Campbell

  2. Department of Dermatology and Wound Healing, School of Medicine, Cardiff University, Glamorgan House, Heath Park, Cardiff, CF14 4XN, UK

    Faraz Ali & Andrew Y. Finlay

  3. Department of Pharmacy, University of Hertfordshire, Hatfield and Institute for Medicines Development, Cardiff, UK

    Sam S. Salek

Authors
  1. Niloufar Campbell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Faraz Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrew Y. Finlay
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sam S. Salek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Faraz Ali.

Appendix

Appendix

See Table 3.

Table 3 Reviewed articles
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Campbell, N., Ali, F., Finlay, A.Y. et al. Equivalence of electronic and paper-based patient-reported outcome measures. Qual Life Res 24, 1949–1961 (2015). https://doi.org/10.1007/s11136-015-0937-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-015-0937-3

Keywords

Search

Navigation