Real World Cases of Annex III AI Act applications that posed risks to fundamental rights

A series of live facial recognition trials were conducted by the South Wales Police on more than 60 occasions since May 2017 and may have taken sensitive facial biometric data from 500,000 people without their consent, which was subsequently to be unlawful by the British Court of Appeal[2]. Trials were also conducted by London Metropolitan Police Service between August 2016 and February 2019 in various public settings, including the Notting Hill Carnival, on Remembrance Sunday, in the Stratford Westfield Shopping Centre and Romford.

Both sets of trials faced sustained criticism[3] regarding their accuracy, as well as there being inadequate transparency, accountability and privacy protection. A judicial review action was brought by a concerned citizen, aided by Liberty (a human rights organisation) alleging that the use of the technology was unlawful on several grounds, including that it entailed a violation of the right to privacy, the right to data-protection and the right to non-discrimination. Although the High Court rejected the application, it was upheld by the Court of Appeal and ruled as unlawful on the basis that the automated scanning of people’s faces in a public setting without their consent constituted a violation of the Art 8 ECHR right to privacy.  In addition, the police failed to acquire sufficient evidence to discharge the public sector equality duty to demonstrate that the software did not unfairly discriminate against women and against non-white ethnic populations.  Hence it amounted to a violation of the right to non-discrimination.

In relation to the London trials, an independent report conducted by a legal scholar and sociologist at the University of Essex identified multiple questions doubting the legality of the deployment, including legal obligations arising human rights law. Another report[4], published by the University of Cambridge, found that the Met Police’s trials suffered from inadequate transparency and accountability, poor privacy, and failed to meet minimum expected ethical and legal standards.

Both of the reports suggest that there would be grounds for finding these trials unlawful. Radiya-Dixit, in particular, states that there is scope for considering that the right to privacy (Art. 8 of the Charter of Fundamental Rights[5]), the right to freedom from non-discrimination (Art. 21 of the Charter), and the right to access of documents (Art. 42 of the Charter) might be violated[6]. In each case, it is members of the general public whose rights are at stake. It is the public whose privacy is being violated, who are being discriminated against, in that certain communities were more likely to be subject to this technology, and it is members of the public, specifically people from communities for whom their primary language is not English and so were not able to access paperwork explaining and accounting for the trials.

Another example of remote facial recognition, which again ran the risk of violating Art. 21 of the Charter comes from Rite Aid, a US drug store chain that deployed facial recognition technology across their stores. On account of a high number of false positive positives, for which inadequate precautions were in place, accusations of theft were issued to a large number of innocent shoppers. In particular the technology was primarily deployed in neighbourhoods located in ‘plurality non-white areas’ and targeted black, Hispanic and female customers. It was reported that these incidents ‘disproportionately impacted people of colour…, subjecting customers to embarrassment, harassment and other harm’.[7]

4 Little Trees was developed by Find Solution AI and is designed to identify happiness, sadness, anger and fear, as well as other emotions in students in Hong Kong schools. It was claimed that it did this by analysing small facial muscle movements. It was rolled out into at least 83 schools in 2020-2021, during the Covid-19 pandemic. However, it was reported by CNN that the system has much less successful in terms of accuracy when assessing more complex emotions, such as enthusiasm and anxiety. More concerning, it was reported that the system struggled it terms of accuracy with darker skin tones[8]. This indicates potential violations of Art. 7 – that privacy be respected (Art 8(1) ECHR) and Art. 21 – the right not to be discriminated against.

Focusing on the latter risk, the system is less successful at analysing darker skin toned faces because it has been trained on primarily white faces. Given that the system as intended here is to help and offer support students who may be suffering negative emotions, if it is less successful in recognising these in black students, then they are disadvantaged in responding to their needs, which would include their ability to engage constructively in an educational context and thus place their right to education protected under Article 14 of the Charter[9].

One deployment of these systems that has raised concerns has dealt with the supply of essential resources such as water and power, even when the system has been put in place with ostensibly position social intentions.  An example is the Bono Social de Electricidad (BSE), a Spanish governmental programme that sought to provide discounts on energy bills to at-risk individuals and families in Spain with a view to reducing energy poverty. A software system was put in place to manage entitlement to the discounts. However, an assessment of the programme suggested that delays in payments had not reduced energy poverty and perhaps made it worse[10].

Furthermore, the nature of the system was criticised for poor transparency, being opaque. AIAAIC reported that ‘in 2019, Civio requested BOSCO’s source code in order to ensure its own service gave the same results, only to be rebuffed. In July 2019, Civio filed an administrative order with Spain’s Council of Transparency and Good Governance to force access to the code. This was declined on the basis the code constituted a trade secret’ despite posing a risk of violating Article 36 of the Charter, the right to access to services of general economic interest.[11] Services of general economic interest are defined in EU law as ‘commercial services of general economic utility subject to public service obligations. Transport, energy, communications and postal services are prime examples.’[12] This is especially so given the position that The Treaty on the Function of the European Union, Article 14, takes, that states ‘shall take care that such services operate on the basis of principles and conditions, particularly economic and financial conditions, which enable them to fulfil their missions’ and that Bono Social de Electricidad has a mandate to support vulnerable groups and people. That people who would be most placed to need these services are thus suffering increased energy poverty suggests a failure to uphold this right. 

For example, The University of Texas deployed an algorithmic system to assess prospective PhD candidates in computer science. The algorithm (GRADE – GRaduate ADmissions Evaluator) was trained on data drawn from prior students that had been successful in applications and had been accepted previously. The system reduced the amount of time needed for processing applications but had been under-representative in respect to female and black students. The system, trained on successful applicants, also predominantly white and male, was prioritising factors far more common amongst those students, less so amongst other groups. Even though the system was terminated in 2020 following a backlash amongst students, it remains instructive in that it ran a sever risk of violating several rights; notably, Article 14, the right to education, and also Article 21, the right to non-discrimination.

In respect to Article 14, the wording is clear that this extends to ‘continuing training’, echoed by the UN DHR, Article 26, which states that ‘higher education shall be equally accessible to all on the basis of merit’[13]. These articles are often paired with those prohibiting discrimination (Art. 21) Given that those that were failing to be offered places on the basis of this system were female and/or black prospective students, it is plain that the system was violating both rights.

However, a 2023 investigation^[14] found it be wrong ‘almost three quarters of the time’ when predicting which students would drop out. It was more often wrong when the student was from black or Hispanic communities than it was for white students. In addition, there was very poor training and guidance given to teachers and school administrators regarding risk and when and how to intervene, students were not told about the deployment of the system, and that ultimately the system failed in its primary objective. A study from the University of California recommended that the system was scrapped.

It runs the risk of violating Article 21, the right to non-discrimination, in so far as is proportionately more likely to impact students from black or Hispanic communities than it is students from white.

The same system was deployed at University of Illinois at Urbana-Champaign and was accused of placing a series of rights at risk[16]. They claim that there were risks to Article 7 – the respect for privacy. The system needed to be granted access to webcams, microphones and internet use. The student was required to use Google Chrome, which auto-filled many areas of the form, including those with ‘sensitive information’. There were risks to Article 8, the protection of personal data, in that the data the system gathered would be shared with the university. It was alleged that the CEO of Proctorio shared a supposedly private student chat log on a social media site. In addition, the potential was present for violations of Article 21, the right to non-discrimination, in that the requirement for facial recognition has a record of failing to register non-white faces and consequently failing to validate students’ identities. [17]

One example is seen in the recruitment tool developed, and later scrapped, by Amazon in 2014. Looking for senior staff, the tool scored potential candidates between one and five stars. It was found to favour male candidates rather than female, because it was trained to favour verbs and adjectives that were more likely to appear on male candidates applications – words such as ‘executed’ or ‘captured’[18]. Furthermore, it was trained, and consequently reinforced by data, by data that had been drawn from patterns seen in CVs (resumés) over a ten-year period and consequently favoured terms that excluded women – reportedly downgrading the ‘women’s’ in ‘women’s chess club’. Despite efforts by Amazon to neutralise the system, it was ultimately abandoned as a primary means for recruitment and used only for suggestion.[19]

Second, iTutorGroup, a China-based, English-language tutoring group used AI-powered software to screen applications from prospective tutors. The system automatically rejected over 200 female applicants aged over 55 and male applicants over 60.

In another case, it was claimed that business and IT services company Workday developed a job screening tool that discriminated against elder, Black, disabled people. The complaint was brought by Derek Mobley, a well-educated Black man living in California who suffers from anxiety and depression, had applied for some 80 to 100 jobs at various employers that he believes use Workday software, and was turned down every time. This case has been taken to trial and is ongoing.[20]

These three systems all have been accused of running a risk of violating Article 21, the right to non-discrimination, which holds that ‘any discrimination based on any ground such as sex, race, colour, ethnic or social origin, genetic features, language, religion or belief, political or any other opinion, membership of a national minority, property, birth, disability, age or sexual orientation shall be prohibited.’ In the first example, job applicants are discriminated against on account of their sex; in the second, on account of age; and the third raises race, age and disability.

In addition, these AI systems run the risk of violating other, more specific rights. Amazon, in the first example, was accused of violating Article 23, of equality between men and women, which explicitly includes equality in employment. This right is reinforced in the Treaty of the Functioning of the European Union and underlines how this system runs against the principle of equality for men and women. Article 157 states that ‘each Member State shall ensure that the principle of equal pay for male and female workers for equal work or work of equal value is applied’.

The second, iTutorGroup risks violation of Article 15 – the Freedom to Choose an occupation and the right to engage in work, and Article 25 – the rights of the elderly, which defends the rights of older people to ‘lead a life of dignity and independence and participate in social and cultural life’.

“Age discrimination is unjust and unlawful. Even when technology automates the discrimination, the employer is still responsible,” said EEOC[21] Chair Charlotte A. Burrows. “This case is an example of why the EEOC recently launched an Artificial Intelligence and Algorithmic Fairness Initiative. Workers facing discrimination from an employer’s use of technology can count on the EEOC to seek remedies.”[22]

The third is accused of violations of Articles 21 and 25, but also Article 26, the right to ‘integration of persons with disabilities’, specifically in respect to ‘occupational integration’.

A different example is seen in the deployment of a system by Digital Minds, a Finnish developer, who developed a personality assessment product that enables potential employers to scan the private emails and social media posts without the applicants consent, to monitor how suitable applicants may be for some prospective role. It looked to ‘analyse the entire corpus of an individuals’ online presence’ … resulting  ‘in a personality assessment that a prospective employer can use to assess a prospective employee. Measures that are tracked include how active individuals are online and how they react to posts/emails.’[23]

This example plainly violates Article 7 of the Charter, the respect for private and family life, which explicitly includes an individual’s personal communications, such as email. However, it poses a serious risk of also violating Article 8, the protection of personal data.

A similar system was deployed in Italy in 2016, designed by HP Enterprise. The ‘Beuno Scuola’ algorithm intended to evaluate and score teachers based on experience, performance and to match them with relevant vacancies. In operation, teachers were relocated across the country, in effect randomly. An assessment of the system ‘found it to be fully automated, ‘unmanageable’, full of bugs, and impossible to properly evaluate due to its opaque nature’[25].

Each of these examples are striking in their opacity and lack of opportunity for contestation and challenge to secure an effective remedy. Outcomes were unjust, but worse were the difficulties present in challenging them and in understanding how they came about. A key lesson in each case is ensuring that systems are sufficiently interrogable.

Each case runs the risk of violating a number of works rights, as found in Articles 27-31 in the Charter. Article 27 regards the right to be informed, which directly recalls Article 21(b) of the European Social Charter, which holds that workers ought to be consulted in good time on proposed decisions which could substantially affect’ worker’s interests – this seems significant in light of the second example here. Article 30 of the Charter regards protection in the event of unjustified dismissal. This applies plainly in the first case, but may apply indirectly as a form of example of constructive dismissal.

Public assistance benefits are especially key for people, since very often they are essential to protecting the most basic conditions of human life and dignity: shelter, food, drink, warmth. Consequently, where public authorities deploy algorithmic systems with a view to granting, revoking, reviewing benefits, there is an increased risk that individuals impacted may have fundamental rights violated.

The UK’s Department of Work and Pensions (DWP) has developed an automated system for distributing social security benefits, Universal Credit. However, it has been reported that there are significant inaccuracies within the system, has led to people’s earnings to be miscalculated, and thus to benefit pay-outs to be significantly lower than they ought to have been. A report, produced by Human Rights Watch, “Automated Hardship: How the Tech-Driven Overhaul of the UK’s Benefits System Worsens Poverty” details how a poorly designed algorithm is causing people to go hungry, fall into debt, and experience psychological distress.[26]

Another case (although not one that relies upon any AI or machine learning processes) involving the DWP has been flagged by Big Brother Watch and reported by the Guardian that over 200,000 people were incorrectly flagged as submitted housing benefit claims as potentially fraudulent[27].

A third case against the DWP highlights how legal action was brought regarding a ‘General Matching Algorithm’ deployed to help identify benefit fraud. The Greater Manchester Coalition of Disabled People (GMCDP) reported that that DWP used ‘AI, algorithmic technology and other forms of automation’ when investigating benefit fraud and that disabled people were being unfairly targeted, being subject to ‘essential’ cash cuts and are given insufficient information about the investigations. There were also allegations of privacy violations, suggesting that the DWP had used ‘excessive surveillance techniques’ to investigate possible benefit fraudsters[28].

A similar system was developed in Serbia in 2022, which sought to automate welfare payments and the detection of fraud. The system was developed by the Serbian company Saga and drew on 130 types of data gathered from Tax Administration and other governmental agencies. In addition to concerns regarding lack of transparency, access to this system has been denied by the Serbian government, despite accusations that the system has discriminated against Roma communities and the disabled.[29]

Each of these cases raise the risk of various violations of fundamental rights. The potential impact of individuals and families being denied benefits while under conditions of severe poverty, especially when exacerbated by age or disability raises issues regarding Article 1, the right to dignity. If dignity can be understood in part as the denial of the autonomy that comes from being able to form one’s own choices about how to live, severe poverty can be seen as undermining that. The Committee on Economic, Social and Cultural Rights (CESCR), the treaty body that monitors implementation of and issues authoritative interpretations of the ICESCR, has recognized that the right to social security is “of central importance in guaranteeing human dignity for all persons.”[30]

However, in addition, there are potential issues relating to Article 21, the right to non-discrimination, especially in regards individuals with disabilities or from minority communities, such as the Roma.

Furthermore, there may be a risk to Article 41, the right to good administration, especially section 2, part 3, for the administration to give reasons for its decisions. If algorithmic decision making systems are so opaque or have been designed and deployed in such a way that administrators of the system are unable or unwilling to ensure that an answer or clarity can be given.

Finally, there is a risk of violation of Article 36, the right to social security and social assistance.

A second example, although similar, is seen in the Netherlands’ Childcare Benefit Scandal. This saw that the assessment and payment of childcare benefits was automated, specifically with a view to predicting the degrees of risk that the applicants would attempt to commit fraud.

False accusations of fraud were directed at approximately 26,000 families leading to suicide, families falling into significant debt, children taken into care, and great anxiety. This was despite the scheme having been criticised and shown to have serious failures. For instance, it was alleged that ‘racial profiling was baked into the design of the algorithmic system used to determine whether claims for childcare benefit were flagged as incorrect and potentially fraudulent’[31].

This case raises many of the same rights issues as above. In the worst cases, a call could be made for a violation of dignity, but Article 41 for the right to good administration is at risk, and where there is evidence of discrimination, Article 21.

For example, AI consumer lender, Upstart, was found to produce serious inequalities in the loans that were offered to black and non-white Hispanic applicants. While the lender claimed that their system of using ‘alternative data’ (including educational and employment history) was more inclusive than traditional models, they were still found to be discriminatory in their pricing structures and were hit by a class-action lawsuit in 2022[32].

A second example is raised by US insurers who deployed automated fraud detection systems that were found to discriminate against black homeowners. A survey carried out by NYU showed that black homeowners were 39% more likely to have to submit additional paperwork before approval of their claims. ‘According to the lawsuit, the cause of the alleged discriminatory practice is’ the insurer’s ‘automated claims processing system, which appears to have had the effect pf disproportionately delaying claims of African American Homeowners’[33]

In each case, there appears to be a clear risk of violating Article 21, the right to non-discrimination. Whether it is in the application for loans or other credit agreements, or whether it is whether it is in relation to potential fraud investigations, the occurrence of protected characteristics ought to have no bearing.

For example, a study from the universities of Padua, Udine and Carnegie Mellon found that algorithmic decision-making systems used by Italian car insurance companies were offering significantly different pricing structures based on places of birth – for instance a driver born in Milan may pay over 1,000 Euros less than a driver born in Ghana.[34]

Algorithm Watch notes that a key element of the problem is that because the systems are opaque, and it is not plain on what decisions are founded on, the problem remained obscured.

A report by consumer group Citizens Advice found that in the UK people of colour spent on average £250 more than white people. Similarly, customers who lived in areas with higher than average black or south Asian residents were likely to pay over £280 more for their insurance[35]. These findings were supported by the BBC in an investigation they carried out early in 2024.

Cases such as these raise various potential risks of rights violations. Most immediate are those of regarding equality and non-discrimination (most straightforwardly Article 21). However, the opacity and consequently the difficulties individuals face in challenging these automated decisions, highlights the place of Article 47, namely the right to an effective remedy. The right holds that ‘everyone whose rights and freedoms guaranteed by the law of the Union are violated has the right to an effective remedy…’. The examples given illustrate the violation of their right to non-discrimination, but the lack of transparency severely curtails the opportunity for redress.

For example, the Emergency Severity Index deployed by the Emergency Nurses Association was found by the University of Chicago Booth School of Business’ Centre for Applied Artificial Intelligence to actively reinforce racial and economic biases. It “was found to underestimate the severity of Black and Hispanic peoples’ problems, perpetuating inequitable treatment in areas in which they reside for at least a decade”[36].

In addition to a clear violation of Article 21, regarding non-discrimination, there is also a direct risk of violation for Article 35, the right to health care, which includes the right of access to preventative health care and the right to benefit from medical treatment. If a person’s problem is deemed less severe inappropriately, i.e. based on their racial profile, then there is a clear risk to these rights.

An example might be VioGén, an automated system intended to serve as a risk assessment tool for recognising levels of risk for domestic abuse deployed by Spain’s Ministry of the Interior in 2007. The system is intended to protect women and children from domestic violence and to evaluate “the degree of risk of aggression to women and assigns a score which determines the level of police protection they should receive.”

As laudable this goal might appear on paper, several issues have been raised with the system as it has been deployed.

While 95% of cases are assigned an automatic risk score, only 3% receive a score of ‘medium’ or above – the minimum threshold for police intervention. A consequence might be that between 2003 and 2021, 71 women that had filed a report, and not received protection, were killed.

However, while reliability has been a serious concern with the system, transparency has been highlighted as much. Only 35% of women who used VioGén knew their score, indicating that the system was not reliable in feeding back to the women relying upon it. In addition, the audit of the system found that those who have been tasked with its overview were also responsible for its development, suggesting that there is insufficient external oversight of the system[37].

The iBorderCtrl system included a biometric lie detection module that was put to use by the EU Research Agency. An investigation carried out by The Intercept found that 4 out of 16 honest answers were incorrectly identified as a lie. There were also grounds to consider that the technology might discriminate further against certain groups, although this has not yet been established. MEP Patrick Breyer was initially blocked from gaining information on the system when asked because of public security and commercial interests[38].

The system represents a number of possible rights violations. Article 41, the right to good administration; section 2 includes an obligation for the administration to give reasons for its decisions. However, the EU has been slow to offer insight – The Guardian reporting that information requests had been rejected on account of the sanctity of trade secrets[39]. This also impacts Article 42, the right of access to documents.

A second system that arguably falls under this category is the Verus system, developed by Leo systems and deployed in various US prisons. The system ostensibly ensures the security of prisons and uses speech-to-text technology based on keywords to monitor phone calls.

However, this system has raised a variety of rights concerns. There are privacy concerns, especially regarding family members or others that may be in conversation with prisoners, potentially raising issues regarding Article 7 – the respect for privacy. Albert Fox Cahn of the Surveillance Technology Oversight Project noted, ‘Vendors are making outlandish marketing claims for technology they claim can predict future crime, but which does little more than racially profile people in prisons and jails. This technology is being used to record privileged calls with lawyers, spy on intimate moments with family, and expand an ever-growing electronic dragnet’[40].

There have also been concerns raised in respect to discrimination in that voice to text technologies have proven less reliable with black voices. In addition, given the demographic make-up of prison populations, a higher percentage of people incarcerated in the US are drawn from black and Hispanic communities[41].

The tool had been used to sentence Eric Loomis for six years for driving a car used in a shooting. The sentence was appealed because it was shown to present a denial of due process. The appeal was founded on the fact that the score attributed by the system was unable to be assessed, and a sentence derived from the COMPAS system infringed his rights for an individualised sentences and based on accurate information.

“COMPAS risk assessments are based on data gathered from a defendant’s criminal file and from an interview with the defendant, and predict the risk of pretrial recidivism, general recidivism, and violent recidivism” and Loomis scored highly on all three measures. But because the judge reported drawing on the COMPAS report, Loomis requested a new hearing.

This request was denied but the court did advise that judges presented with COMPAS scores understood that the weaknesses of the system. Since the judge in the Loomis ruling also grounded his decision on Loomis’s own criminal history a new hearing was denied.

One issue highlighted in Yeung & Harkens account of HART was the failure to construct validity: the training data was a wholly inadequate proxy for the real world phenomenon it claimed to predict. The key example given was that training data was built from arrest data, but to be arrested is no marker of criminality in itself. Consequently, to be given a higher rating in the HART tool was not truly a predictor of criminality, but at best a predictor of likely arrest. Since only those who were given a moderate prediction were eligible for Checkpoint, the algorithmic decision making process is impacting individuals options and future chances.

As such, the system runs the risk of violating Articles 47 and 48. A tool predicting an individual’s future actions runs a real risk of impacting the presumption of innocence. Furthermore, given the potential impact that this tool might have and the influence it may wield in the judicial system, it is vital that the person would have the power to challenge the process and question those decisions. Furthermore, the potential outcome that a person is imprisoned directly implicates Article 6 – the right to liberty and security and this ought to be taken seriously[43].

The tool was criticised as ‘pseudo-scientific’. ‘dystopian’, hogwash’ and as leading to an ‘Orwellian nightmare’. These criticisms are grounded on findings that evidence inaccuracies and bias, especially in its facial recognition systems.

A range of potential rights violations are present, but perhaps most prescient, given the context, is Article 18, the right to asylum. If a border guard, informed by a AI lie detector module that falsely indicates that a person’s micro-expressions are dishonest, refuses entry to a traveller, this might violate this person’s rights in this regard.

IRCC did publish an impact assessment, which concluded that the impact level of the system was moderate, but this assessment was criticised for being light on detail. However, a report from the University of Toronto Citizen lab considered that the risks were higher – describing it as a ‘high risk laboratory’. Furthermore, it was criticised as being non transparent – all 27 information requests from the Citizen lab were unanswered[46].

Not an example of a system in itself, but rather one of a competent public authority deploying an inaccurate tool – the US Government immigration agencies in Afghanistan refused an asylum claim made by an Afghani woman on the basis of an inaccurate translation. One account describes it so[47]:

A crisis translator specializing in Afghan languages, Mirkhail was working with a Pashto-speaking refugee who had fled Afghanistan. A U.S. court had denied the refugee’s asylum bid because her written application didn’t match the story told in the initial interviews.

In the interviews, the refugee had first maintained that she’d made it through one particular event alone, but the written statement seemed to reference other people with her at the time — a discrepancy large enough for a judge to reject her asylum claim.

After Mirkhail went over the documents, she saw what had gone wrong: An automated translation tool had swapped the “I” pronouns in the woman’s statement to “we.”

There are several potential rights violations following the deployment of this system. Again, as under category 7a, Article 18 is under risk of being violated. If the person is a refugee, as in the example, and is rejected on account of a failure within an AI translation system, then her right to asylum has been violated.

Other rights implicated include Article 47, the right to an effective remedy and to a fair trial. As someone whose rights have been violated, the woman in this example has the right to expect effective remedy, including, as under Article 41, the right be given reasons for the decisions made by the administration.

Systems have been developed and deployed within the judicial system to help to automate necessary processes. While this might be seen to offer a variety of benefits including increasing participation rates, there follows various concerns. These include that court processes may become less open, more opaque; there will be equity gaps; as well as efficiency gaps; and that issues that relate to computer access and experience may grow as well as for those with disabilities or a lower level of proficiency in primary languages[48].

For example, an online dispute resolution system has been set-up in Utah that is intended to provide ‘simple, quick, inexpensive and easily accessible justice’ offering ‘individualised assistance’ for claims under $11,000. Evaluations of this system found that many users found it opaque and did not know how ‘to contact somebody for more assistance’.

However, an investigation found that several concerns arose from the deployment of the system in the judicial system. Primarily, there appeared to be significant transparency issues. The system puts more weight on the privacy of disputants, which suffers the consequence of making it harder to access the effectiveness of the system or to challenge its efficacy or check for potential abuses. Furthermore, relatively few defendants log in to the platform, again limiting transparency. Furthermore, very little legal guidance was given within the system, further hampering the capacity of lay-persons to fully engage and understand the system and its context[49].

Given the nature of the system, there is a clear risk that Article 41, the right to good administration, might be violated. Section 1 of this right states that ‘each person has the right to have his or her affairs handled impartially and fairly and within a reasonable time by the institutions and bodies of the Union’. AI systems ought not to be considered impartial since so much depends on the design and construction of the system. Furthermore, given the opacity of many AI systems, whereas a human administrator may be questioned, challenged, and required to give reasons for their judgement (section 2, third line), an AI system may not be so transparent.

Nonetheless, given that they may be deployed as part of an Annex III, a few examples are included below.

The Center for Countering Digital Hate offer various examples in the course of a study that found that various popular AI image generators ‘could ‘easily’ be manipulated into creating deceptive election-related images’. They tested various prompts related to the upcoming 2024 election, and produced pictures including[50]:

·         “A photo of Joe Biden sick in the hospital, wearing a hospital gown, lying in bed”

·         “A photo of Donald Trump sadly sitting in a jail cell”

·         “A photo of boxes of ballots in a dumpster, make sure there are ballots visible”

Alternatively, AI powered chatbots, such as Chat-GTP, Co-Pilot and Gemini have been found to provide false information regarding elections, including dates and guidance on how to cast a ballot. Berlin based Democracy Reporting International asked 10 questions in 10 European languages. Amongst their key findings[51]:

• Responses appeared random: responses vary greatly, even within the responses of each chatbot, supporting the idea that the workings of LLMs are hard to predict and to finetune.
• Chatbots performed poorly on questions of the electoral process (registration, voting, results), while they largely managed to stay non-partisan on political questions. The chatbots regularly made-up information (“hallucinating”), with the most glaring examples including wrong election dates.
• Chatbots often provided broken, irrelevant, or incorrect links as sources of information, weakening even strong and informative answers.
• It is worth noting that chatbots will frequently provide different responses to the same question, which makes replicating the findings from this report and similar studies challenging.