Validity and reliability of the evaluation tool of TCM confidence in students with TCM learning experience

Rui-Min Ji,Xio-Ning Yn ,Xing-Yu Liu ,Jin-Nn Wng

aSchool of Nursing, Shanxi University of Chinese Medicine, Jinzhong, Shanxi 030619, China

bThird Clinical College, Shanxi University of Chinese Medicine, Jinzhong, Shanxi 030619, China

Abstract: Objective:This study aimed to design an evaluation scale of traditional Chinese medicine (TCM) confidence and then evaluate its validity and reliability.Methods:Three parallel samples with TCM learning experience were included in this current study.Randomly selected 500 nursing students were included factor analysis and reliability and validity analyses.After 4 weeks,200 nursing students were recruited to retest the reliability.The evaluation tool of TCM confidence included 6 dimensions and 25 entries.Results:Cronbach’s α coefficient of the evaluation scale was 0.95 and that of the 6 observed aspects were higher than 0.90,and splithalf reliability was 0.81.For the retest reliability,Cronbach’s α coefficient of the evaluation scale was 0.93 and that of the 6 examined aspects was all over 0.80.The index of content validity Scale Content Validity Index (S-CVI) was 0.92 and Item Content Validity Index(I-CVI) was 0.83–1.00.Conclusions:Observed results may indicate that the designed tool can be used to assess the TCM confidence level of students with TCM studied experience and also can evaluate results of TCM education during studying TCM lectures.

Keywords: Chinese medicine confidence • Evaluation tool of Chinese medicine confidence • student with TCM experience • traditional Chinese medicine • validity and reliability

1.Introduction

Traditional Chinese medicine (TCM) is one of the great medical achievements,which has been developed more than 5000 years in China.1It was a medical science formed and matured empirically during Chinese people’s daily life and against diseases.TCM has its own special characteristics assimilating from Chinese culture.It uses a holistic view to diagnosis and treatment of diseases by combining individuality and natural and social environments together.2

TCM has been focusing on disease prevention and health promotion,and been making great contributions to the reproduction and survival of the Chinese nation.Its medical achievements are not just because it is an effective treatment method but also because of emphasis on regulating people’s life pattern.3Recently,COVID-19 has been rampant worldwide.TCM is acknowledged globally as an effective method for treatment and prevention.4

In light of these contributions,the Chinese government has issued a large series of policies and laws to inherit and advance TCM.5One of the important strategies is setting up TCM institutions and TCM subjects in medical institutions to play a key role in cultivating TCM servicers.The number of graduated and current students has been increasing gradually.6However,there is no evaluation tool to evaluate the study results– the confidence level of TCM.Therefore,this current study aims to design an evaluation scale and then evaluate its validity and reliability.

2.Methods

2.1.Participant recruitment

The participants of this current study were nursing undergraduates who have studied TCM in the Shanxi University of Chinese Medicine.Three levels of students were included– sophomore,junior,and senior,and first-year students were excluded because of lack of learning experience of TCM.Three parallel samples were included: 200 students for pretesting and identifying the original entry,500 students for factor analysis and validity and reliability analyses,and 200 students for verifying validity and reliability.7Every participant was voluntarily recruited by tutors notification and confidentially maintained throughout the present study.Ethical approval was obtained from the Shanxi University Human Ethnics Committee (No.LL038).

2.2.Questionnaire development

The questionnaire was designed based on talent training objectives of TCM and documented studies of TCM education,8–11including 46 items.Seven TCM specialists were involved to evaluate the content validity of the items,and after the assessment,34 items were selected.

2.3.Data collection

Data were collected online through the Questionnaire Star Website from August to November 2020.Responses to the examined questions were rated on a Likert scale with 5 degrees,with values ranging from 1 to 5.The mean score was positively related to the confidence level of TCM.

2.4.Data analysis

Results were described and analyzed using Statistics Package for the Social Sciences (SPSS) version 22.0(IBM Corporation,Armonk,New York,United States).Qualitative data were represented by frequency and percentage (%),and quantitative data were represented by mean ± standard deviation (M ± SD).Exploratory factor analysis (EFA) was performed by using the principal component method with maximum variance factor rotation,and the characteristic root >1 was used as the standard for extracting common factors.The reliability was evaluated by Cronbach’s coefficient for internal consistency,and Pearson’s intraclass correlation coefficient (ICC) for test–retest reliability.Confirmatory factor analysis (CFA) was performed by AMOS 24.0 software(IBM Corporation,Armonk,New York,United States)for evaluating structure validity and content validity.12The evaluation of the fitting model was achieved by χ2/freedom degree (DF),root mean error of approximation(RM-SEA),and goodness-of-fit index (GFI),comparing fitting index (CFI) and normed fit index (NFI).13AP-value～ 0.05 was considered significant.

3.Results

3.1.Demographic characteristics

The participants were nursing students recruited voluntarily.The majority were female,and firstyear students were not included because of lack of experience of Chinese lectures.The specific information is presented in Table 1.

3.2.Validity test

3.2.1. Structure validity

The EFA results showed that the Kaiser–Meyer–Olkin(KMO) value was 0.704,and the Bartlett spherical test value was 1442.52 (P<0.01),which indicated that the data were suitable for factor analysis.Eight common factors were extracted according to the characteristic root >1,after principal component analysis with maximum variance factor rotation.A factor load <0.5 and double load entries were removed,based on the post-rotation component matrix.In this study,2 common factors and 9 entries were removed,yielding 6 common factors,with 25 entries finally reserved.Specifically,6 common factors are “the cognition of TCM culture and theory” including 6 entries,“advantages and characteristics of TCM diagnosis and treatment” including 6 entries,“feelings about TCM achievements” including 3 entries,“personal behaviors related to TCM” including 3 entries,“attitudes towards the association between TCM and Western medicine” including 2 entries,and “attitudes towards the future of TCM” including 5 entries.The variance percentages of 6 common factors were 17.21%,18.16%,12.22%,13.18%,10.19%,and 12.68%,and the cumulative variance contribution rate was 83.64%.The factor load of each item is shown in Table 2.CFA results showed that the fit index of the model was χ2/df=3.01,RM-SEA was 0.082,GFI was 0.92,CFI was 0.96,and NFI was 0.94,which indicated that the model fitted well.

3.2.2. Content validity

Content validity was analyzed from 2 aspects,namely,Item Content Validity Index (I-CVI) and Scale Content Validity Index (S-CVI).The results showed that the I-CVI was 0.83–1.00 and the S-CVI was 0.92.

3.2.3. Discriminant validity

In this study,the differential validity was used to evaluate criterion validity.The differences in scores of the total content and of each dimension between 2 groups(group 1: willing do TCM related work in future;group 2: do not willing do TCM-related work in future) were analyzed by using the independent sample t-test.The results are provided in Table 3.

3.2.4. Reliability test

Cronbach’s α coefficient of the questionnaire was 0.95 and that of the 6 examined aspects was 0.92,0.92,0.94,0.96,0.93,and 0.97,respectively,and split-half reliability was 0.81.These results were obtained through analyzing 500 validated questionnaires.After 4 weeks,the retest reliability was performed by analyzing 200 questionnaires.Cronbach’s α coefficient of the questionnaire was 0.93 and that of 6 examined aspects was higher than 0.80 (i.e.,0.82,0.90,0.85,0.93,0.81,and 0.90,respectively).

3.2.5. Contents of the questionnaire

The contents of the questionnaire are shown as follows(Table 4).

Table 4.Contents of the questionnaire.

4.Discussion

The great contributions have been made by TCM for the public health.The Chinese government and related departments have been putting efforts to enhance TCM and to facilitate greater understanding and acceptance of TCM,with the aim of providing medical services to global people.One of the strategies is cultivating TCM talents,who are the major contributors to TCM.It is essential for these individuals to have confidence in TCM themselves before they can make contributions to its development.5

However,the scientific research in this field is very few.There is currently no evaluation tool of TCM confidence.Therefore,it seems that this present research did a valuable work.The evaluation tool was designed based on talent training objectives of TCM,documented studies,and suggestions of TCM experts.8–11Finally,the content was decided to include 6 aspects—the cognition of TCM culture and theory,advantages and characteristics of TCM diagnosis and treatment,feelings about TCM achievements,personal behaviors related to TCM,and attitudes toward Western medicine and the future of TCM.

In the current study,the validity and reliability of the evaluation tool of TCM confidence were tested.Generally,the reliability is evaluated using Cronbach’s coefficient and test–retest reliability.Validity is evaluated mainly based on 3 aspects,namely,content validity,structure validity,and differentiation validity.Cronbach’s α coefficient of the evaluation scale was 0.95 and that of the 6 observed aspects was higher than 0.90,and split-half reliability was 0.81.For the retest reliability,Cronbach’s α coefficient of the evaluation scale was 0.93 and that of 6 examined aspects was all over 0.80.These results met the required standards of the internal consistency coefficient of the evaluation scale >0.8 and of examined dimension >0.75.These results may indicate that the evaluation tool of TCM confidence has good reliability and stability.The tested validity indexes in this current study showed positive results as well.Specifically,content validity: in this study,the S-CVI was 0.92 and the I-CVI was 0.83–1.00 compared to documented standards of the S-CVI of 0.90 and I-CVI >0.78;structural validity: in this study,χ2/df was3.01,RM-SEA was 0.082,GFI was 0.92,CFI was 0.96,and NFI was 0.94 compared to documented standards of RM-SEA >0.08,GFI,CFI,and NFI >0.90;differentiation validity: in this study,statistically significant differences were observed in scores of the total content of the evaluation tool and in each observed aspects between 2 groups (P<0.001).12,13

5.Conclusions

In this study,the evaluation tool of TCM confidence includes 6 dimensions and 25 entries demonstrating good validity and reliability.The results suggest that the designed tool can be used to assess TCM confidence levels of students with TCM experience and can also evaluate outcomes of TCM education when studying TCM lectures.Hence,this evaluation tool may not only reflect the TCM confidence level of TCM students but also show weaknesses of TCM education.This helps educators find problems,devise strategies to improve it,and finally contribute to TCM.However,this evaluation tool of TCM confidence is a subjective evaluation scale,which may limit to show objective abilities of TCM.Therefore,it may be necessary to develop both subjective and objective evaluation scales in future.

Ethical approval

This study was approved by the Shanxi University Human Ethnics Committee (IRB approval number:LL038).

Conflicts of interest

All contributing authors declare no conflicts of interest.