Comparison of the clinicians’ and experts’ assessments of rehabilitation service needs for patients in the middle of China

Study design

We conducted a multicenter clinical vignette-based survey from May to October 2024 in Anhui Province, located in the middle of China. This province was chosen due to its relatively underdeveloped economy, limited medical resources, and the availability of comprehensive institutional data. A total of 40 clinical vignettes, primarily representing rehabilitation cases, were selected from our previous study for evaluation by clinicians and experts¹⁷. Each clinical vignette was presented in a standardised and structured medical record format, including patient demographics, chief complaint, history of present illness, past medical history, personal history, family history, epidemiological history, and findings from physical examination. This structured information was sufficiently detailed to allow raters to independently assess each patient’s rehabilitation service needs. All vignettes followed a consistent structure to minimize information bias and ensure comparability across raters. An example of the clinical vignette is provided in Appendix 1. The inclusion criteria for clinicians were: (1) relevant clinical experience and (2) consent to participate. The inclusion criteria for experts were: (1) at least 20 years of clinical experience in rehabilitation medicine, (2) a senior professional title (e.g., Associate Chief Physician or higher), (3) consent to participate. Clinicians in China follow a standardised certification and training system. To provide the professional clinical service, all clinicians must first complete a medical degree (3–8 years in duration depending on the training model), pass the National Medical Licensing Examination (NMLE), and obtain a practicing certificate. They are also required to undergo standardised residency training, the duration which varies according to their level of prior medical education. Professional titles (junior, intermediate and senior) are awarded based on a combination of academic qualifications, years of clinical practice, and passing professional qualifying examinations.

Sample

A three-stage approach was employed to select the sample for this study. First, 40 vignettes were chosen from a database of over 2,000 medical vignettes stored in the mini-program developed in our previous study¹⁸. This mini-program was built based on a novel clinical tool developed by our research team to assess patients’ rehabilitation needs and determine appropriate service referral levels. The structure and logic of this tool are illustrated in Fig. 1. These 40 vignettes represented three different levels of healthcare institutions in Yunnan Province, including one county hospital, one township health center, and two village health centers. Second, 174 clinicians were recruited from three levels of healthcare institutions in Anhui Province, which included county hospitals, township health centers, and village health centers. After excluding 10 invalid questionnaires, 164 valid clinician responses remained, yielding an effective response rate of 94.2%. Third, three rehabilitation experts were selected to assess the clinical vignettes. The experts were internally recruited from our study team using a convenience sampling method. Eligible experts were required to have at least 20 years of clinical experience in rehabilitation medicine, hold a senior professional title, and consent to participate. Their extensive experience and expertise helped ensure the reliability and credibility of the reference standard evaluations. As a result, the final analytical sample included 167 participants: 164 clinicians and three experts.

(a) The novel rehabilitation tiered service tool for rehabilitation patients distribution in previous study. (b) The novel clinical tool for distributing rehabilitation patients in China developed by our research group. * Note: The tool was developed by our research group. (a) Was the tool, (b) was the diagram and logic path of the tool. ^a The Longshi-scale was an evaluation method of self-care ability among disabled people, which was approved by the National Standards Commission of China in 2018 (GB/T37103-2018). ^b The patient should be referred if his/her functional status did not change for over one month. ^c The disease course could be different in different cities. ^d Multiple dysfunctions indicated patients who had conscious impairments, or in addition to motor impairment, they also had any one or more cognitive impairment, speech impairment, swallowing impairment, or cardio-pulmonary impairment. This tool classifies patients based on rehabilitation needs. The ‘only movement disorders’ category refers to patients with motor impairments but no cognitive or speech disorders. These cases are considered mild and suitable for primary healthcare, as referring them to higher-level institutions would unnecessarily use up medical resources.

Procedure

In previous studies, data were collected using a mini-program tool on WeChat, which securely stored the information in the cloud (Kanghui platform). This setup allowed for real-time access and efficient management by the researchers. The study was conducted in three phases.

Phase 1: participant inclusion and vignette screening

First, vignette data from Yunnan Province were selected from among the 28 provinces and cities included in the previous multicenter study. Vignettes were then exported from the databases of four institutions in Yunnan Province: one tertiary hospital, one secondary hospital, and two primary care institutions. 10 vignettes were randomly selected from each institution, resulting in a total of 40 vignettes for subsequent evaluation. Patient identities and addresses were anonymized to protect privacy and minimize bias. Subsequently, 174 doctors from various healthcare facilities (county hospitals, township health centers, and village health centers) in Yunnan province were invited to assess clinical vignette data developed for this study. Each doctor independently evaluated 40 vignettes presented as image-based excerpts from patient medical records. After excluding invalid responses, 164 valid assessments were included in the follow-up analysis.

Phase 2: establishing a reference standard

To establish a reference standard, two experienced rehabilitation experts were invited to independently evaluate the rehabilitation service needs of patients based on the 40 vignettes. However, 21 of the 40 vignettes displayed discrepancies between the evaluations of the two experts, and the initial inter-rater consistency among the experts was low. Therefore, a structured consensus process was implemented: another expert was invited to join the original two experts to review and discuss the vignettes with inconsistent assessments. Through systematic discussion in a closed session, the three experts reached a final consensus for each disputed case, which served as the reference standard for evaluating the clinicians’ assessments. This staged consensus approach—consisting of independent assessments followed by group discussion—is commonly adopted in rehabilitation and referral research when no objective gold standard exists^19,20,21,22.

Phase 3: consistency of clinicians’ assessments of rehabilitation service needs

The 164 clinicians were asked to evaluate the rehabilitation service required for the same patient vignettes, relying solely on their own clinical experience, without considering any external factors. The clinicians’ assessments were then compared to the expert-established reference standard to evaluate their consistency. Additionally, the consistency across different levels of healthcare institutions was also exmined.

Measurement

General characteristics

This study included 164 clinicians, and their general characteristics were collected, including age, gender, years of experience, organization level, and other relevant variables. Additionally, each of the 40 patient vignettes contained standardised information such as gender, age, diagnosis, dysfunction, disease duration, and the stability of the current condition, among other details.

Clinicians’ subjective assessment outcomes

Clinicians’ subjective assessments were based on their clinical experience and the standardised information provided in the patients’ vignettes. This included patient complaints, disease progression, imaging results, type of dysfunction, and other relevant data. Clinicians were not instructed to follow any predetermined standard when deciding on the rehabilitation treatments they deemed necessary for each patient. This approach was designed to reflect the typical evaluation process they would use in routine clinical practice. The objective was to optimize clinicians’ decision-making while preserving the authenticity of real-world consultation scenarios. The assessments made by clinicians and experts were categorized into six distinct outcomes:

1. (1)

   Outpatient rehabilitation treatment: patients’ rehabilitation needs can be met without hospitalization.

2. (2)

   Other clinical departments: if the consultation showed no need for rehabilitation, patients might be advised to visit other clinical departments for further evaluation or treatment.

3. (3)

   Inpatient rehabilitation in primary care: this category includes community health service centers or township health centers that provide basic rehabilitation services for patients in the recovery phase.

4. (4)

   Inpatient rehabilitation in secondary care: this category comprises rehabilitation hospitals or departments within secondary general hospitals that offer comprehensive rehabilitation services while engaging in teaching and research.

5. (5)

   Inpatient rehabilitation in tertiary care: these are specialized institutions that provide advanced rehabilitation services, conduct high-level scientific research, and offer educational programs for professionals, typically affiliated with regional or national hospitals.

6. (6)

   Nursing homes or long-term care institutions: patients whose conditions extend beyond the coverage period of insurance or Medicare may be referred to care homes or nursing homes for long-term care.

Statistical analysis

The primary focus of the analysis was to assess the consistency between the clinicians’ subjective evaluations of rehabilitation service needs and the expert-established reference standard. Additionally, we examined the consistency in evaluation outcomes between rehabilitation experts and clinicians at various healthcare institutions. Descriptive statistics were used to analyze the clinicians’ general characteristics. Continuous variables were expressed as median values (IQR), while categorical variables were presented as frequencies and proportions. To assess the initial consistency among experts before consensus, Fleiss’ Kappa was calculated. Given the low initial consistency, a structured consensus process was implemented to finalize the reference standard. Furthermore, Kendall’s coefficient of concordance (W) was used to examine the overall consistency between clinicians and experts, and to compare consistency across different healthcare levels. This non-parametric statistic is appropriate for measuring consistency among multiple raters when the variables are ordinal. Given that the rehabilitation service needs in this study were categorized into six ordinal levels and each vignette was rated independently by clinicians, Kendall’s W provided a robust and interpretable measure of inter-rater reliability across different healthcare settings.

Ethic approval

The study was approved by the Ethics Review Office of Shenzhen Second People’s Hospital (Ethics No.2023-226-02PJ). All methods were performed in accordance with the relevant guidelines and regulations. All participants provided informed consent before their inclusion in this study.