Tool components
The final tool compromised four main types of codes: administrative codes; safety-netting contextual codes; safety-netting advice codes; and an additional optional set of problem contextual codes. Administrative codes recorded assigned study identification number, how many problems were raised during consultation, and the type of problem using the International Classification of Primary Care (ICPC-2) classification.31 These administrative codes were based on information from the original ‘One in a Million’ study and not included in the IRR testing.
Contextual codes recorded elements that have been described as key features of the broader term ‘safety netting’ such as the communication of planned follow-up, diagnostic uncertainty, and the expected time course of a problem,7 but these features would not be coded as ‘safety-netting advice’ as standalone statements. As this study aimed to capture all statements of uncertainty around the diagnosis, the authors opted to include both direct, ‘I’m not sure what the diagnosis is’, and indirect, ‘I think it’s x ’, statements of uncertainty.32
Codes that evaluated the specific details of the safety-netting advice formed the bulk of the coding tool. An example of how these codes are assigned to an extract of a GP giving safety-netting advice is provided in Box 2. As multiple consultation models teach that safety netting is delivered towards the end/during the closing-down phase of a consultation,6,33 the authors wanted to be able to capture when in the consultations GPs were actually giving safety-netting advice and who was initiating the advice-giving sequence. The phases of the consultation based on the original classification described by Byrne and Long (Phase II to Phase VI)34 were recorded but links to the corresponding phases for the Calgary–Cambridge model were also provided.33
During development work the authors also observed doctors giving safety-netting advice that only applied to their treatment or management plan, therefore a code to differentiate between problem and treatment safety-netting advice was generated, as demonstrated in Box 3.
| Problem | ‘So, reassured about this, but come back to us if it seems to be changing.’ |
| Treatment/management plan | ‘Antiinflammatories are really good at pain thinning, but they’re bad at irritating the lining of the stomach. And in the worst case it can cause an ulcer and bleeding. So, if you’re getting indigestion pains, coughing up blood, or your stool is very dark and black and sticky, you must stop the naproxen and come and see me straight away.’ |
| Both | ‘Yes, well, any problems, come back.’
‘And of course, if things are getting worse rather than better in the meanwhile, or any problems with the antibiotics, we’ll see you before.’ |
Box 3. Examples of safety-netting advice for a problem, for a treatment or management plan for the problem, or both
The number of conditions or symptoms the doctor had warned the patient to look out for, the action they should take if those symptoms developed, and how quickly they needed to take such action was recorded. Regarding the course of action recommended, the codes included three different options: patient-focused action, ‘you must come back’; doctor-focused action, ‘I’d like to have another look at it ’; and both doctor- and patient-focused action, ‘you must come back so I can have another look at it ’.
In addition, a code to separate generic from specific advice was generated, full criteria for which are described in detail in the codebook (Supplementary Table 1). Briefly, generic advice could potentially apply to multiple problems or management plans, for example, ‘any problems’, ‘issues’, ‘it gets worse’, or asking the patient to return if their condition did not get better but without giving a specific time frame; whereas specific advice included new symptoms such as ‘chest pain’, ‘cough up any blood’, or asking them to return if their symptoms persisted but included a time frame ‘If it’s not better in 2 weeks then come back ’.
Finally, how the patient responded to the advice, if the patient asked any questions about the advice, if the doctor gave any written safety-netting advice, and whether the doctor documented that they had given safety-netting advice in the medical records was assessed.
Multiple other contextual codes were included to help identify which types of problems were associated with higher or lower rates of safety-netting advice, such as the nature of the problem, for example, ‘acute’ or ‘chronic’. As these codes were not directly related to ‘safety netting’ the authors opted to include these in an optional section.
Collapsed codes and repeat evaluation
Inter-rater testing demonstrated that some variables were too difficult for coders to reliably distinguish between. For example, when assessing how patients responded, coders found it difficult to differentiate between codes for acknowledgements ‘yeah’, positive assessments ‘great, fine ’, and acceptance ‘OK, all right, sure ’. These codes were subsequently collapsed into a single ‘acknowledgement or acceptance’ code. Two codes (the action advised and the timescale of action) that were deemed to have performed substandardly but were judged to be essential to retain underwent further refinement and were evaluated in 10 further randomly selected consultations that contained safety-netting advice (13 problems, 25 discrete episodes of safety-netting advice) using the relevant parts of the written transcripts only. The authors also wanted to differentiate whether coders thought the safety-netting advice and follow-up plans had been fully or only partially documented in the patient’s medical notes and if diagnostic uncertainty and the expected time courses of illness were delivered with the safety-netting advice or at a separate part of the consultation, but IRR scores demonstrated that in its current format coders could not reliably discriminate to this level of detail.
Inter-rater reliability scores
At the consultation level, coders agreed on the presence or absence of safety-netting advice for 32/32 consultations (100% κ = 1.0). At the problem level, coders agreed on the presence or absence of safety-netting advice for 49/55 problems (89% κ = 0.77). The ICC for the number of separate times safety-netting advice was discussed in each consultation and for each medical problem was 0.88 and 0.73 respectively. A contributing factor towards the lower ICC for safety-netting advice per medical problem was if generic safety-netting advice, for example, ‘any problems let me know’, was not listed under all the problems it could have applied to.
Incidents where one coder missed an episode of safety-netting advice but incorrectly labelled a non-safety-netting contingency plan (exclusion criteria, Box 1) occurred once for coder 1 and three times for coder 2. This only positively affected the IRR results for the presence or absence of safety-netting advice for one consultation and one medical problem. Because agreement scores do not differentiate between correct agreements and false agreements, the authors also reported how many of the 51 discrete episodes of safety-netting advice each coder correctly identified and any false positives. Full details of correct and false positive identification of safety-netting advice by coders are shown in Supplementary Table 3. Coder 1 correctly identified 48/51 (94.1%) episodes of safety-netting advice, whereas coder 2 correctly identified 45/51 (88.2%) of episodes.
The review process identified 51 separate episodes where the GPs gave safety-netting advice for 35 problems in 24 consultations. Table 1 demonstrates the IRR scores assessed in the coding tool. One code ‘does the safety-netting advice apply to this problem or multiple problems?’ was wholly dependent on the screening for the presence or absence of safety-netting advice for each problem, therefore IRR was not reassessed. The mean average unweighted percentage agreement was 88% (90% weighted) and mean average κ score was 0.66 for the final tool.
