Commissioners of artificial intelligence (AI) for health care, clinicians who use it, and patients who receive care including it may wish to ask themselves, ‘What do I need to understand to commission, or use, healthcare systems that include automation, machine learning, or artificial intelligence, in a way that I trust, and that promotes trust for others? What will the public expect to have been considered?’ This can be a challenge as technology is often created, and implemented, before the debate has been had, and people have developed informed views. We outline some key questions.
Some background
Advances in AI have the potential to transform medicine by generating novel cures, improving diagnostics, making care more accessible, reducing costs, and alleviating the workload of clinicians. AI refers to a broad field of science encompassing not only computer science but also psychology, philosophy, linguistics, and other areas. It is concerned with getting computers to do tasks that would normally require human intelligence. The term AI is often used as a catch-all term to include any automated device or website, machine learning, or the achievement of artificial sentience. Machine learning is a branch of artificial intelligence that allows computer systems to learn directly from examples, data, and experience. Through enabling computers to perform specific tasks intelligently, machine learning systems can carry out complex processes by learning from data, rather than following pre-programmed rules. Natural language processing takes an advanced neural network (in this case a network of powerful computers) to analyse and recreate human language. When an AI algorithm is trained to interpret human communication, it is called natural language processing. This is not only useful for chat bots and translation services, but is also represented by AI assistants such as Alexa. The development and application of AI are rapidly advancing: both ‘narrow AI’, where only a limited and focused set of tasks are conducted; and ‘broad’ or ‘broader’ AI, where multiple functions and different tasks are performed. Broad AI has been discussed as having the potential to be an existential risk to humans. Such official guidance as there is for healthcare stakeholders is mainly aimed at those who wish to develop, sell, or purchase AI destined for clinical use, rather than for clinical users. Clinical users have been advised to consult their professional bodies (this would include the General Medical Council, British Medical Association, and the royal colleges in the UK).
Public trust
What kinds of concern might the public have with an automated system in place of a human healthcare professional? In their book The Future of the Professions, Richard and Daniel Susskind highlight eight key concerns for when a professional service is replaced by an automated one:
Loss of trustworthy institutions: is the automated system provided by a reputable and reliable institution?
Loss of moral character: is the automated system seen as a public benefit or a profit-making product? In the case of AI, does the sale of products or acquisition of patient data for use or sale affect trust?
Loss of old way of doing things: when a helpful person to see or speak to is replaced by an online form or a bot, is this something that is acceptable?
Loss of the personal touch: does standardisation of processes result in fairness or unfairness, and do clients feel connected or abandoned?
Loss of empathy: is the system able to recognise and accommodate for distress? Is it ‘coldly’ clinical or emotionally unresponsive?
Loss of jobs: is the system offered primarily as a way to balance a budget by reducing staff numbers?
Loss of pipeline of experts: who backs up, supervises, and helps to train the automated system?
Loss of future roles: what will people do instead of being doctors, nurses, and receptionists?
Susskind and Susskind consider these to be addressable concerns. Moreover, they argue that, for many people, automation is the only way that services will remain or become accessible. This could be for financial reasons (professionals are expensive) or social ones (machines can operate 24/7).
Alignment of AI technologies
The use of automated systems can be considered in terms of four broad groups of stakeholders:
the creators/vendors of an automated system;
the policymaker and/or commissioner of a service using the system;
the human team who would ordinarily deliver part or all of the service; and
the recipients or users of the service.
Stakeholders may have different goals, beliefs, values, and material needs or constraints (conditions). Medical journals have discussed the alignment problem: where AI is poorly aligned with the needs of the public and values of society, it can present a public health risk. So ask yourself: Have I considered how the system in question benefits or harms: service users, healthcare staff, the institutions delivering the service, and wider society? Do I know:
That the data on which the system is trained adequately represent the service user population? For example, are different genders, age groups, sexes, and ethnicities represented in the data used to develop an automated application/AI?
That the system is fair and are there ways of identifying and adjusting for systematic bias?
What transparency is in the system so that errors and failings can be addressed and learned from?
What accountability for reasonableness is in a system? (Sharing reasons why the system has made a decision, which then allows appeal to a human supervisor and/or allows a decision to be understood/complied with.)
What is the ability of human intervention to change the goals of the system in order to prevent significant or fatal errors?
What safeguards are there against unwise or mischievous adjustment of the system?
Is there any risk that a system will amplify human mistakes or failings?
How does the system measure success? Do metrics used for success or failure align well enough with population benefits and harms?
Policymakers need to consider what trade-offs are relevant in the deployment of AI. It is a mistake to trade values, for example, respect for autonomy versus benefit and harm, rather than instances of those values, for example, more people can access medical advice versus there is less access to advanced professional care. Does use of an automated application create digital democratisation or digital discrimination?
Accountability, risk, and assurance
Many automated systems are deployed in high-volume and low-risk settings, meaning they can process many thousands more interactions than is possible by humans, but only those where there are clear decision-making pathways, low risk of harm, and lower stakes of harm in the event of system error. All these elements — clear pathways, low risk, and low stakes — need to be considered. Policymakers, commissioners, and clinicians need to understand their responsibilities and accountability when working with AI. Consider:
What legal liability does the designer or vendor of the technology accept? What liability is passed to institutions and individuals that use a technology?
What is the potential for a technology to be misused either through error or by bad actors? Can a system be ‘gamed’ to produce particularly desirable outcomes?
Has the designer or vendor of the relevant technology identified any inherent risks that have been managed or mitigated? Are there any residual risks that cannot yet be managed?
Has the designer or vendor of the technology met minimum legal compliance to be first to market or gone beyond this? For example, does the technology take account of legally protected characteristics in terms of equality and diversity, or take into account other sources of inequity, such as being a carer or having low socioeconomic status?
Are there clear autonomy safeguards?
Where are data held? Are they held in ways that comply with NHS expectations for information governance?
What happens to data collected by the system? Are they used for purposes other than provision of the service, for example, for improvement of the service, or ‘in trust’ for end users to decide what to do with? AI can use data in unacceptable or inscrutable ways.
Are data ever sold on?
What happens if the designer or vendor sells their company?
Opportunity costs and hidden harms
Commissioners and users of AI for health care should be imaginative about the potential costs of using AI in the long term and to third parties, as well as broader impacts beyond the vendor–client relationship. For example:
Are cost savings generated by a technology reduced or negated by unintended consequences of the technology and/or system being risk averse? Even if it does not, does a successful automated service generate uncosted or unmet demand elsewhere in a health system?
Does the environmental benefit of an app delivering advice by phone rather than patients and staff travelling to a clinic match the environmental cost of creating and running the computing facilities required?
Do unnuanced ethical principles pose predictable but unforeseen risks, for example, does maximising welfare systematically discriminate against vulnerable groups?
A recipe for success
Jessica Morley describes four unifying concepts (utility, usability, efficacy, and trust) in her work on an algorithmically enhanced healthcare service, arguing that we should be sceptically optimistic. We applaud her realist approach to appraising the practical and moral issues raised by AI. Drawing on this, and in summary, the key questions to ask yourself are:
Is this technology useful? Does it deliver something that patients or the healthcare service needs? Does it actually do what it claims? Beware the hype and seek real-world evidence of effectiveness and safety.
Is it useable? What is the experience of the practitioner, patient, or carer, and would they use this over existing solutions? Does it produce results that can be easily interpreted by users?
Will it be used? This is by far the most complex category and depends on a range of factors from trust in the product or the company producing it, to the ease and cost of implementation. For example, key questions might be:
Would you recommend it to your colleague or family member?
How easy is it to embed within care pathways or existing workflows?
Do time-or resource-saving benefits outweigh investment costs?
These notes are intended as a brief guide to some key issues for the clinical users, as well as the shapers, drivers, creators, and embedders of AI in health care, to provoke further reading and discussion in a rapidly developing field.
Footnotes
This article (with references) was first posted on BJGP Life on 25 Sep 2024; https://bjgplife.com/briefingonai
Competing interests
The authors were funded to explore the ethical acceptability and trustworthiness of AI in health care by SBRI Phase II funding to Ufonia. The views expressed are solely those of the authors.
- © British Journal of General Practice 2025
