13 January 2021
An open letter to the European Commission officials responsible for the EU's forthcoming proposal for a regulation on artifical intelligence calls for clear "red lines", limitations and safeguards to prevent the deployment of "artificial intelligence" (AI) systems that breach fundamental rights and undermine democratic standards.
The letter (full-text below) looks a range of uses of AI systems "that are incompatible with a democratic society and must be prohibited or legally restricted in the AI legislation," including biometric mass surveillance (such as the deployment of facial recognition systems in public spaces), the use of AI for border control and migration management purposes, predictive policing and the use of algorithms for risk assessment in the criminal justice system.
The letter was coordinated by European Digital Rights (EDRi) and has been signed by 61 organisations, including Statewatch. EDRi is also coordinating the Reclaim Your Face campaign, calling for a ban on biometric mass surveillance.
A European Citizens' Initiative (ECI) on the issue of banning mass biometric surveillance was recently accepted by the European Commission. It calls on the Commission to adopt a legal act "to permanently end indiscriminate and arbitrarily-targeted uses of biometric data in ways which can lead to mass surveillance." If the ECI gathers over a million signatures, it must be considered by the Commission; it opens for signatures in February.
The EU is already working on AI tools for deployment in its migration management and border control systems, as highlighted in the Statewatch report Automated Suspicion ,which examines the forthcoming regime of "travel authorisations" and changes to the EU's visa processing system. A successful campaign against such uses of AI would require changes to these tools, which exacerbate the risks of discrimination and errors in decision-making.
The European Commission is expected to publish its legislative proposal on AI by the end of March.
Full-text of the open letter (also available here, link to pdf)
12th January 2021
Dear Executive Vice-President Vestager,
Dear Vice President Jourová,
Dear Commissioner Breton,
Dear Commissioner Dalli,
Dear Commissioner Reynders,
Dear Commissioner Johansson,
cc: Lucilla Sioli,
cc: Juha Heikkilä
We, the undersigned, write to restate the vital importance of clear regulatory red lines to prevent uses of artificial intelligence which violate fundamental rights. As we await the Commission’s legislative proposal on artificial intelligence, expected from Directorate-General CONNECT during Q1 of 2021, we emphasise that regulatory limitations form a necessary part of a fundamental rights-based artificial intelligence regulation.
EU Member States and EU institutions have an obligation under the Charter of Fundamental Rights of the European Union and the European Convention on Human Rights to ensure that each person’s rights to privacy, data protection, free expression and assembly, non-discrimination, dignity and other fundamental rights are not unduly restricted by the use of new and emerging technologies. Without appropriate limitations on the use of AI-based technologies, we face the risk of violations of our rights and freedoms by governments and companies alike.
The development of AI offers great potential to benefit people and society. However, socially-beneficial innovation can only be achieved when we guarantee that uses are safe, legal, and do not discriminate. The European Union now has the opportunity and the responsibility to ensure democratic oversight and clear regulation before technologies are deployed. Europe’s industries - from AI developers to car manufacturing companies - will also benefit greatly from the regulatory certainty that comes from clear legal limits and an even playing field for fair competition.
We, the undersigned, call for regulatory limits on deployments of artificial intelligence that unduly restrict human rights. In addition to strong enforcement of the General Data Protection Regulation (GDPR) and safeguards such as human rights impacts assessments, software transparency and the availability of datasets for public scrutiny, it is vital that the upcoming regulatory proposal establishes in law clear limitations as to what can be considered lawful uses of AI, to unequivocally address the following issues:
In particular, we call attention to specific (but non-exhaustive) examples of uses that are incompatible with a democratic society and must be prohibited or legally restricted in the AI legislation:
1. Biometric mass surveillance:
Uses of biometric surveillance technologies to process the indiscriminately or arbitrarily-collected data of people in publicly-accessible spaces (for example, remote facial recognition) enables mass surveillance and creates a ‘chilling effect’ on people’s fundamental rights and freedoms. Any deployment of biometric surveillance in public or publicly-accessible spaces amounts to, per definition, the mass indiscriminate processing of biometric data. Such use of biometric mass surveillance intrudes the psychological integrity and well-being of individuals, in addition to the violation of a vast range of fundamental rights. As emphasised in EU data protection legislation and case law, such uses are neither necessary nor proportionate to the aim sought, and must therefore be clearly prohibited in the AI legislation through an explicit ban on the indiscriminate or arbitrarily-targeted use of biometrics which can lead to mass surveillance. This will ensure that law enforcement, national authorities and private entities cannot abuse the current wide margin of exception and discretion currently possible under the existing general legal principles of a prohibition on biometric processing.
2. Uses of AI at the border and in migration control:
The increasing examples of AI deployment in the field of migration control pose a growing threat to the fundamental rights of migrants, to EU law, and to human dignity. Among other worrying use cases, AI has been tested to purportedly detect lies for the purposes of immigration applications at European borders and to monitor deception in English language tests through voice analysis, all of which lack credible scientific basis. In addition, EU migration policies are increasingly underpinned by AI systems, such as facial recognition, algorithmic profiling and prediction tools for use within migration management processes, including for forced deportation. These use cases may infringe on data protection rights, the right to privacy, the right to non-discrimination, and several principles of international migration law, including the right to seek asylum. Given those concerns and the significant power imbalance that such deployments exacerbate and exploit, there should be a ban or moratorium on the use of automated technologies in border and migration control until they are independently assessed to determine compliance with international human rights standards.
3. Social scoring and AI systems determining access to social rights and benefits
AI systems have been deployed in various contexts in a manner that threatens the allocation of social and economic rights and benefits. For example, in the areas of welfare resource allocation, eligibility assessment and fraud detection, the deployment of AI systems to predict risk, verify people’s identity and calculate their benefits greatly impacts people’s access to vital public services and has a potentially grave impact on the fundamental right to social security and social assistance. This is due to the likelihood of discriminatory profiling, mistaken results and the inherent fundamental rights risks associated with the processing of sensitive biometric data. A number of examples demonstrate how automated decision-making systems are negatively impacting and targeting poor, migrant and working class people, including the deployment of SyRI in the Netherlands and the use of data-driven systems in Poland to profile unemployed people, with severe implications for data protection and nondiscrimination rights. Further, uses in the context of employment and education have highlighted highly-intrusive worker and student surveillance, including social scoring systems, intensive monitoring for performance targets, and other measures which limit work autonomy, diminish well-being and limit workers’ and students’ privacy and fundamental rights. There have also been cases of discriminatory use of AI technologies against persons with disabilities by state and private entities in the allocation of social benefits and access to education. The upcoming legislative proposal must legally restrict uses and deployments of AI which unduly infringe upon access to social rights and benefits.
4. Predictive policing:
Uses of predictive modelling to forecast where, and by whom, certain types of crimes are likely to be committed repeatedly score poor, working class, racialised and migrant communities with a higher likelihood of presumed future criminality. As highlighted by the European Parliament, deployment of such predictive policing can result in “grave misuse”. The use of apparently “neutral” factors, such as postal code, in practice serve as a proxy for race and other protected characteristics, reflecting histories of overpolicing of certain communities, exacerbating racial biases and affording false objectivity to patterns of racial profiling. A number of predictive policing systems have been demonstrated to disproportionately include racialised people, in complete disaccord with actual crime rates. Predictive policing systems undermine the presumption of innocence and other due process rights by treating people as individually suspicious based on inferences about a wider group. The European Commission must legally prohibit deployments of predictive policing systems in order to protect fundamental rights.
5. Use of risk assessment tools in the criminal justice system and pre-trial context
The use of algorithms in criminal justice matters to profile individuals within legal decision-making processes presents severe threats to fundamental rights. Such tools base their assessments on a vast collection of personal data unrelated to the defendants’ alleged misconduct. This collection of personal data for the purpose of predicting the risk of recidivism cannot be perceived as necessary nor proportionate to the perceived purpose, in particular considering the implications for the right to respect for private life, and the presumption of innocence. In addition, substantial evidence has shown that the introduction of such systems in criminal justice systems in Europe and elsewhere has resulted in unjust and discriminatory outcomes. Beyond this, it may be impossible for legal professionals to understand the reasoning behind the outcomes of the system. We argue that legal limits must be imposed on AI risk assessment systems in the criminal justice context.
These examples illustrate the need for an ambitious artificial intelligence proposal in 2021 which foregrounds people’s rights and freedoms. The signatories of this letter call for the legislative proposal on artificial intelligence to include:
We look forward to a legislation which puts people first, and await your response about how the upcoming artificial intelligence proposal will address the concerns outlined in this letter.
European Digital Rights (EDRi), including:
Bits of Freedom
Chaos Computer Club
D3 - Defesa dos Direitos Digitais
Electronic Privacy Information Center (EPIC)
AI Now Institute, NYU
App Drivers and Couriers Union (ADCU)
Associazione Certi Diritti
Associazione Luca Coscioni
Associazione per gli Studi Giuridici sull'Immigrazione
Big Brother Watch
Center for Intersectional Justice (CIJ)
Dutch Section - International Commission of Jurists (NJCM)
Each One Teach One (EOTO) e.V.
European Disability Forum
European Evangelical Alliance (EEA)
European Network Against Racism (ENAR)
European Network On Religion and Belief (ENORB)
European Roma Grassroots Organizations (ERGO) Network
European Youth Forum
Federation of Humanitarian Technologists
Fundación Secretariado Gitano
Greek Forum of Migrants
Human Rights Watch
International Committee on the Rights of Sex Workers in Europe (ICRSE)
International Federation for Human Rights (FIDH)
International Decade for People of African Descent, Spain
Ligue des Droits Humains
Montreal AI Ethics Institute
Open Society European Policy Institute
Platform for International Cooperation on Undocumented Migrants (PICUM)
Ranking Digital Rights
Refugee Law Lab, York University
Save Space e.V.
Stop Ethnic Profiling Platform Belgium
StraLi - for Strategic Litigation
UNI europa – European Services Workers Union
University College Dublin Centre for Digital Policy
Image credit: Ars Electronica, published under a CC BY-NC-ND 2.0 licence.
Spotted an error? If you've spotted a problem with this page, just click once to let us know.
Statewatch does not have a corporate view, nor does it seek to create one, the views expressed are those of the author. Statewatch is not responsible for the content of external websites and inclusion of a link does not constitute an endorsement. Registered UK charity number: 1154784. Registered UK company number: 08480724. Registered company name: The Libertarian Research & Education Trust. Registered office: c/o MDR, 88 Fleet Street, London EC4Y 1DH, UK. © Statewatch ISSN 1756-851X. Personal usage as private individuals "fair dealing" is allowed. We also welcome links to material on our site. Usage by those working for organisations is allowed only if the organisation holds an appropriate licence from the relevant reprographic rights organisation (eg: Copyright Licensing Agency in the UK) with such usage being subject to the terms and conditions of that licence and to local copyright law.