Translation and validation of the instruments DigiHealthCom and DigiComInf

Introduction 

Digitalisation within health, care and social services has increased, both nationally and internationally (1). The World Health Organization (WHO) defines digital health as the use of digital technologies to improve health and healthcare services (1, 2).

In Norway, digitalisation is a key driving force in the development of the health service. Digitalisation is intended to contribute to a patient-centred, effective and integrated health service that supports the work of healthcare personnel, strengthens patient involvement and promotes organisational sustainability (3, 4).

Digitalisation has changed the roles and tasks of healthcare personnel. The increased focus on patient-centred care requires a greater degree of collaboration (5, 6), while digital solutions require the safe, ethical and practice-oriented management of patient data (7). Since 2019, the Norwegian Directorate of Health has surveyed healthcare personnel’s use of, attitudes to and satisfaction with digital health services using a self-developed questionnaire (8). 

In 2024, communication was mostly via telephone or in written form, while remote consultations served as a supplement. Seventy-nine per cent of respondents reported having the necessary skills, and 63 per cent believed that digital support simplifies their work. Seventy-eight per cent felt that training is adequately facilitated, while only 40 per cent had received training in recent years. Those with the most experience also had the most positive attitudes to digitalisation (8).

The development and implementation of digital health services is a clear political priority in Norway. The emphasis is on digital solutions making the working day easier, not more difficult. Nevertheless, their use is hindered by healthcare personnel’s lack of experience, confidence, competence and training. Their digital competence is crucial for successful implementation (9).

In 2019, helsekompetanse was established as the Norwegian term for ‘health competence’. The term refers to the patient’s ability to search for, understand, appraise and apply health information in order to make informed decisions about their health (10). 

‘Digital health literacy’ and ‘eHealth literacy’ build on this concept and refer to the ability to understand and use digital health information (11, 12). According to the Official Norwegian Report on future competence needs (13), digital literacy encompasses skills, knowledge, understanding, values and attitudes related to technology, including insight into data protection, information security and troubleshooting. For healthcare personnel, this involves delivering health care of a professional standard, safeguarding ethical considerations and collaborating effectively on digital platforms and across different systems (9, 14).

To ensure the successful implementation of digital health services, various aspects of healthcare personnel’s digital competence must be mapped in order to identify positive and negative factors and to tailor appropriate measures (15).

In a review of previous research, Jarva et al. (15) at the University of Oulu in Finland found that existing measurement instruments capture certain aspects of digital technology in health care, but not healthcare personnel’s overall digital health competence. They therefore developed two standardised instruments to measure healthcare personnel’s digital health competence (which they refer to as ‘digital health competence’). 

The Digital Health Competence (DigiHealthCom) instrument measures healthcare personnel’s self-assessed digital health competence, while the Aspects Associated with Digital Health Competence (DigiComInf) instrument maps educational and organisational frameworks in the health service, such as support from management, training, practices for digital competence development and collegial influence (Figure 1). Both instruments demonstrated satisfactory validity and reliability (15).

In this study, digital health competence is understood as a multifaceted concept based on perspectives from general competence, health literacy and digital health competence. Digital health competence refers to the ability to use digital tools in a safe, ethical and effective manner, both in interactions with patients and in collaboration with other healthcare personnel. The term encompasses technical skills, understanding of digital health information, data protection and information security, as well as the ability to reflect, collaborate and adapt to digital ways of working (9–13, 16, 17). 

Objective of the study

The objective of our study was to translate and validate the DigiHealthCom and DigiComInf instruments for use in a Norwegian context by researchers and clinicians in order to measure healthcare personnel’s digital health competence. 

Method 

The translation and validation of the instruments were conducted in two phases: phase 1 involved a sequential translation procedure, and phase 2 comprised validation of the instruments through content validity analysis using the Content Validity Index (CVI). The CVI is calculated based on the assessments of an expert panel for each item in the instruments. This method was also used by the original developers of the instruments and is well suited when there is no gold standard for comparison (18).

Expert panel

A total of eight experts participated in one or both rounds of the CVI assessment of the instruments. The experts had backgrounds in nursing or other health and social care fields and met at least one of the following criteria: experience from primary care and/or the specialist health service, experience with remote counselling’, experience in the translation of questionnaires, and/or experience in using digital solutions in clinical practice.

Following the recommendations of Polit et al. (18), we conducted an additional assessment round. Six experts participated in the second round; two of them had also participated in the first round, while four were new participants.

Description of the instruments

The Digital Health Competence (DigiHealthCom) instrument measures healthcare personnel’s self-assessed digital health competence. It consists of five dimensions with a total of 42 items: human-centred remote counselling competence (16 items), digital solutions as part of work (9 items), information and communication technology (ICT) competence (5 items), competence in utilising and evaluating digital solutions (8 items), and ethical competence related to digital solutions (4 items).

The Aspects Associated with Digital Health Competence (DigiComInf) instrument maps educational and organisational factors related to digital health competence. It consists of three dimensions with a total of 15 items: support from management (6 items), organisational practices as part of digital competence development (4 items), and colleagues’ adoption and influence (5 items) (Figure 1 and Appendix 1 - in Norwegian).

Items in both instruments are answered using a four-point Likert scale: 1 = completely disagree, 2 = partially disagree, 3 = partially agree, and 4 = completely agree. Mean scores are calculated for each dimension by summing the responses and dividing by the number of items. Scores range from 1 to 4, with higher values indicating a higher level of self-assessed digital competence. 

Phase 1: translation 

We translated and adapted the Norwegian versions of DigiHealthCom and DigiComInf through a sequential process in accordance with international recommendations (20) (Appendix 1 – in Norwegian). First, a certified bilingual translator translated the instruments into Norwegian. Meanwhile, one of the authors produced an independent translation. We discussed discrepancies between the translations with reference to the original to ensure both clinical and linguistic accuracy. The two versions were then compared and merged into Version 1 (V1).

Another government authorised translator back-translated V1 into English. The authors and a Finnish-speaking colleague compared the back-translated version with the original Finnish version to assess the accuracy of the Norwegian translation. After adjustments, Version 2 (V2) was produced and sent to an expert panel for assessment in the first round. 

Based on feedback from the expert assessment in round 1, we adjusted certain linguistic formulations. In consultation with the original authors, the term kunden (the customer) was replaced with pasienten (the patient), and the instruments were revised as Version 3 (V3).

In terms of terminology related to digital health services, we chose to use the term fjernrådgivning (remote counselling) in the Norwegian version, as it best captures the breadth of the original concept. The term was explained in the introduction to the questionnaire and in the information provided to the expert panel. It was clarified that fjernrådgivning encompasses various forms of digital communication, including remote monitoring, remote medical monitoring, digital monitoring and digital home monitoring.

Following these linguistic adjustments, a new expert assessment was conducted (round 2). Based on the feedback from this round, the final Norwegian version of the instruments, Version 4 (V4), was completed (Appendix 1 – in Norwegian).

Phase 2: content validity 

CVI is a method for assessing the content validity of questionnaires, scales or other measurement instruments. It is particularly useful when developing new instruments or adapting them to new languages, cultures or contexts. It is also valuable when there is no gold standard instrument, or no established criteria, such as a previously validated instrument (18).

CVI evaluates whether the items are relevant and adequately represent the concept they are intended to measure. Through expert assessments, a quantitative value is calculated that summarises the consensus on the content of the instrument. CVI is thus a key tool for ensuring quality and applicability (18, 21, 22).

CVI is calculated based on the assessments of an expert panel. The recommended panel size is three to ten experts (22). The experts evaluate each item according to two dimensions: 

• Relevance: the item’s relevance to the concept being measured
• Clarity: the clarity and comprehensibility of each item 

The results of these assessments are then used to calculate the CVI values. CVI is calculated both at the item level (item CVI [I-CVI]) and for the scale or instrument as a whole (scale-level CVI [S-CVI]) (18, 21, 22).

I-CVI: evaluation of individual items

I-CVI evaluates each item for relevance and clarity based on the expert panel’s assessment (18, 21, 22). The experts rated the items on a four-point Likert scale, with relevance ranging from not relevant (1) to highly relevant (4) and clarity from unclear (1) to very clear (4) (Table 1).

I-CVI indicates the extent of agreement on relevance and clarity, with values ranging from 0 to 1. Values ≥ 0.78 indicate that the item is relevant and clear, values between 0.70 and 0.78 indicate a need for revision, and values < 0.70 suggest that the item should either be revised or considered for removal (18, 21, 22).

S-CVI: evaluation of the instrument

S-CVI is calculated to evaluate the instrument as a whole or subgroups of items within an instrument. The aim is to assess the extent to which the instrument adequately covers the concept it is intended to measure (22). The following main types of S-CVI are used: 

• S-CVI/Ave (average): This method calculates the average of the I-CVI values for all items in the instrument. It provides an overall measure of how well the items represent the content of the instrument. An S-CVI/Ave value of ≥ 0.90 is considered to indicate excellent content validity (18, 21, 22).
• S-CVI/UA (universal agreement): This method indicates whether there is full agreement among the experts that each item in the instrument is relevant and represents the phenomenon the instrument is intended to measure. Values ≥ 0.80 are considered acceptable for universal agreement (18, 21, 22).

Ethical considerations 

The study was registered with Sikt – the Norwegian Agency for Shared Services in Education and Research (reference number 745044).

Results

DigiHealthCom and DigiComInf were validated in two rounds of expert assessments. The results indicate an overall improvement in both relevance and clarity. 

I-CVI: analysis of individual items in the instruments

Relevance

For DigiHealthCom, 40 of the 42 items achieved acceptable relevance in round 1. Two items were below the threshold (< 0.78): item 12 in domain 1 (human-centred remote counselling competence), which received an I-CVI value of 0.4, and item 3 in domain 5 (ethical competence related to digital solutions), which received an I-CVI value of 0.6.

After revisions and a new expert assessment in round 2, the results showed improvement. At that stage, 41 of the 42 items had an acceptable value. Item 12 in domain 1 remained below the threshold, while the remaining items scored ≥ 0.78 (Table 2).

For DigiComInf, all items achieved approved I-CVI values of ≥ 0.78 for relevance; only item 5 in domain 1 (support from management) lacked full agreement, but its value was nevertheless above the threshold. In round 2 of the expert assessment, the I-CVI value for this item increased to 1.

Item 4 in domain 2 (organisational practices as part of digital competence development) decreased to 0.8, but remained ≥ 0.78. Overall, all 15 items achieved acceptable values in both rounds, and the instrument demonstrated satisfactory content validity for relevance.

Clarity

For DigiHealthCom, 35 of the 42 items achieved a satisfactory clarity value in round 1. Seven items were below the threshold (< 0.78) and were recommended for revision (18) (Table 2). After revisions, 40 of the 42 items achieved satisfactory clarity values in round 2. Two items still had I-CVI values below the threshold. The remaining items achieved values of ≥ 0.78, and several reached a value of 1, indicating full agreement.

In DigiComInf, two items in domain 1 achieved a value of 0.8 in round 1, while the remaining items had a value of 1, indicating full agreement. Following the expert assessments in round 2, all items achieved a value of 1, indicating full consensus among the experts.

S-CVI: evaluation of the instruments

Overall content validity was evaluated using S-CVI analysis. The results show the development of DigiHealthCom and DigiComInf between two expert rounds (Table 3).

S-CVI/Ave: average score for all items in the instrument

In DigiHealthCom, the S-CVI/Ave for relevance increased from 0.93 to 0.94 from round 1 to round 2, indicating a slight improvement, while clarity increased from 0.80 to 0.93. Both values in round 2 were above the threshold and indicate good content validity. DigiComInf showed high values for relevance (0.98) and clarity (0.97) in both rounds (Table 3).

S-CVI/UA: universal agreement across the items in the instrument

In DigiHealthCom, the S-CVI/UA for relevance decreased from 0.71 to 0.70, below the threshold of 0.80. Clarity increased substantially from 0.23 to 0.64, but remained below the threshold. For DigiComInf, the S-CVI/UA for relevance decreased from 0.92 to 0.89, and for clarity from 0.89 to 0.80; however, both values remained above the recommended level after round 2 (Table 3).

Discussion

In this study, we translated and validated two instruments: Digital Health Competence (DigiHealthCom), which measures healthcare personnel’s self-assessed digital health competence, and Aspects Associated with Digital Health Competence (DigiComInf), which maps educational and organisational factors related to digital health competence. The translation process was in accordance with international recommendations. Content validity was assessed in two expert rounds using the Content Validity Index (CVI). 

The analysis of individual items (I-CVI) showed that most items in both instruments achieved satisfactory values for relevance and clarity after revisions, and that only a few items were below the recommended threshold. At the scale level, the S-CVI analysis demonstrated that both instruments achieved the desired value for average content validity (S-CVI/Ave ≥ 0.90). DigiComInf also achieved acceptable universal agreement among the experts (S-CVI/UA ≥ 0.80), while DigiHealthCom was somewhat below this threshold for both relevance and clarity.

On this basis, we concluded that the instruments are well suited to measuring healthcare personnel’s digital health competence in a Norwegian context. We recommend further evaluation of the instruments’ reliability, validity and sensitivity.

Expert assessment in two rounds: advantages and challenges

In this study, the instrument validation process consisted of two rounds of expert assessments and a content validity analysis (CVI). A total of eight experts participated, some in both rounds and others in only one. Carrying out two rounds of expert assessments is in line with the recommendations of Polit et al. (18).

This approach makes it possible to assess which experts should be retained in the further process, based on their understanding and potential biases. However, differing expert assessments can have both a positive and negative effect on the validation process. One advantage of two rounds is that weaknesses can be identified and revised before a subsequent round confirms whether the changes have actually led to improvements (18). 

Our findings support this approach, as minor adjustments to the wording increased the I-CVI values for several items (Table 2). Nevertheless, we could not implement all suggestions from the expert panel, as some changes would have affected the content and made international comparability more difficult.

One challenge of conducting two rounds of expert assessments is that this requires additional time and resources. Extensive revisions can also result in items being overworked and losing their original meaning. There is also a risk of inconsistent results between the two rounds, or of reduced engagement in round 2 (22). However, reduced engagement can be mitigated by including new experts in the second round, as was done in this study.

Content validity in DigiHealthCom and DigiComInf

After the first round of expert assessments, seven items did not achieve satisfactory I-CVI values. Following revisions, round 2 showed a clear improvement, with all but one item achieving acceptable values. The remaining item did not achieve satisfactory values for either relevance or clarity.

S-CVI provides an overall assessment, in which S-CVI/Ave (average) and S-CVI/UA (universal agreement) represent two different perspectives. Polit et al. (18) discuss the different methods for calculating content validity at the scale level and point out that S-CVI/Ave is more robust and informative, as it takes all individual items in the instrument into account. They recommend S-CVI/Ave as the most appropriate measure for assessing content validity, with a standard of ≥ 0.90 for an acceptable level.

In our study, both instruments achieved an S-CVI/Ave value of ≥ 0.90. However, DigiHealthCom did not achieve an S-CVI/UA value of ≥ 0.80 for either relevance or clarity. Although clarity improved, the instrument achieved an S-CVI/UA value of 0.70 for relevance and 0.64 for clarity, which is below the desired threshold of 0.80. S-CVI is useful, but has limitations (18). 

S-CVI/Ave, which calculates the average of all items, can be diluted if some items have low I-CVI values. S-CVI/UA is a stricter measure, requiring full agreement among all experts for an item to be considered approved. Even a single item without agreement can lower the value (18).

Missing values from experts can particularly affect S-CVI/UA, as the method requires all experts to evaluate every item. In our study, some experts skipped certain items, probably unintentionally. These missing values are interpreted as lack of agreement, since the requirement for full consensus is not met. As a result, missing values may have contributed to DigiHealthCom not reaching the threshold for S-CVI/UA. Appendices 2 and 3 (in Norwegian) show items with missing values (18, 22).

Further use of DigiHealthCom and DigiComInf

Although DigiHealthCom did not reach the threshold for universal agreement, the content validity for all items except one was acceptable. Meanwhile, both DigiHealthCom and DigiComInf achieved satisfactory S-CVI/Ave values, confirming good overall content validity. A limitation is the missing data for certain items, which may have artificially lowered values, particularly for S-CVI/UA. This may indicate a need for revision, further testing and possibly a shortened version of the instrument at a later stage.

Following translation and validation, DigiHealthCom and DigiComInf can be used to map digital health competence in the health service. The instruments are suitable for research, quality development and competence mapping. They can also help identify competence needs and provide a basis for targeted measures to further develop digital health competence in the health service.

Strengths and limitations of the study

The study has several strengths. The translation followed recognised principles for forward and back translation. Content validity was assessed in two expert rounds, which made it possible to revise the items. Using both I-CVI and S-CVI provided a thorough evaluation of individual items as well as the extent to which the instruments collectively cover the defined aspects of digital health competence.

A particular strength of the CVI is the combination of quantitative assessments in the form of I-CVI and S-CVI values with qualitative expert comments. Together, these contributed to a nuanced evaluation of the items and provided a clear basis for improvements related to wording, conceptual precision and content relevance (22).

A limitation is the missing values in the expert panel, which particularly affected S-CVI/UA, where full agreement is required. This should have been addressed, for example by asking the experts to review their responses before submission. This could have clarified whether low values were due to genuine disagreement or unintentional omission. A small expert panel may limit generalisability, although the number in our study was in line with recommendations (22). 

Conclusion

In this study, we translated and validated the international instruments Digital Health Competence (DigiHealthCom) and Aspects Associated with Digital Health Competence (DigiComInf) for a Norwegian context. The translation process consisted of forward and back translation. Content validity was assessed in two rounds of expert evaluations using the Content Validity Index (CVI).

Overall, the findings indicate that the translation and validation resulted in two valid and contextually adapted instruments suitable for use in the Norwegian health service, while retaining features that make them comparable with the international versions of the instruments, allowing results to be compared across countries. DigiHealthCom can be used to map healthcare personnel’s self-assessed digital health competence, while DigiComInf provides insight into organisational and educational factors that impact on digital competence development.

Acknowledgements

We would like to thank Gunnar Ellingsen, Monica Evelyn Kvande, Gunn Pettersen, Jill-Marit Moholt, Bente Norbye, Gunhild Brørs, Line Silsand and Mariell Waleniussen Borge, who participated in the content validity evaluation in the CVI analysis. Their professional insight and valuable feedback were crucial for adapting the Norwegian versions of the instruments. We greatly appreciate their effort and commitment, which helped strengthen the clinical and linguistic precision of our work.

The authors declare no conflicts of interest.

Open access CC BY 4.0