governments' use of automated decision-making systems reflects systemic issues of injustice and inequality
Software and technology can process large amounts of data instantaneously, making them highly attractive for government use. Credit: Shutterstock

In 2019, former UN Special Rapporteur Philip Alston said he was worried we were “stumbling zombie-like into a digital welfare dystopia.” He had been researching how government agencies around the world were turning to automated decision-making systems (ADS) to cut costs, increase efficiency and target resources. ADS are technical systems designed to help or replace human decision-making using algorithms.

Alston was worried for good reason. Research shows that ADS can be used in ways that discriminate, exacerbate inequality, infringe upon rights, sort people into different social groups, wrongly limit access to services and intensify surveillance.

For example, families have been bankrupted and forced into crises after being falsely accused of benefit fraud.

Researchers have identified how facial recognition systems and risk assessment tools are more likely to wrongly identify people with darker skin tones and women. These systems have already led to wrongful arrests and misinformed sentencing decisions.

Often, people only learn that they have been affected by an ADS application when one of two things happen: after things go wrong, as was the case with the A-levels scandal in the United Kingdom; or when controversies are made public, as was the case with uses of facial recognition technology in Canada and the United States.

Automated problems

Greater transparency, responsibility, accountability and public involvement in the design and use of ADS is important to protect people’s rights and privacy. There are three main reasons for this:

these systems can cause a lot of harm;they are being introduced faster than necessary protections can be implemented, and;there is a lack of opportunity for those affected to make democratic decisions about if they should be used and if so, how they should be used.

Our latest research project, Automating Public Services: Learning from Canceled Systems, provides findings aimed at helping prevent harm and contribute to meaningful debate and action. The report provides the first comprehensive overview of systems being canceled across western democracies.

Researching the factors and rationales leading to cancelation of ADS systems helps us better understand their limits. In our report, we identified 61 ADS that were canceled across Australia, Canada, Europe, New Zealand and the U.S. We present a detailed account of systems canceled in the areas of fraud detection, child welfare and policing. Our findings demonstrate the importance of careful consideration and concern for equity.

governments' use of automated decision-making systems reflects systemic issues of injustice and inequality
There are a range of factors that influence decisions to cancel the uses of ADS systems. Credit: Data Justice Lab, Author provided

Reasons for cancelation

There are a range of factors that influence decisions to cancel the uses of ADS. One of our most important findings is how often systems are canceled because they are not as effective as expected. Another key finding is the significant role played by community mobilization and research, investigative reporting and legal action.

Our findings demonstrate there are competing understandings, visions and politics surrounding the use of ADS.

Hopefully, our recommendations will lead to increased civic participation and improved oversight, accountability and harm prevention.

In the report, we point to widespread calls for governments to establish resourced ADS registers as a basic first step to greater transparency. Some countries such as the U.K., have stated plans to do so, while other countries like Canada have yet to move in this direction.

Our findings demonstrate that the use of ADS can lead to greater inequality and systemic injustice. This reinforces the need to be alert to how the use of ADS can create differential systems of advantage and disadvantage.

Accountability and transparency

ADS need to be developed with care and responsibility by meaningfully engaging with affected communities. There can be harmful consequences when government agencies do not engage the public in discussions about the appropriate use of ADS before implementation.

This engagement should include the option for community members to decide areas where they do not want ADS to be used. Examples of good government practice can include taking the time to ensure independent expert reviews and impact assessments that focus on equality and human rights are carried out.

governments' use of automated decision-making systems reflects systemic issues of injustice and inequality
Governments can take several different approaches to implement ADS systems in a more accountable manner. Credit: Data Justice Lab, Author provided

We recommend strengthening accountability for those wanting to implement ADS by requiring proof of accuracy, effectiveness and safety, as well as reviews of legality. At minimum, people should be able to find out if an ADS has used their data and, if necessary, have access to resources to challenge and redress wrong assessments.

There are a number of cases listed in our report where government agencies’ partnership with private companies to provide ADS services has presented problems. In one case, a government agency decided not to use a bail-setting system because the proprietary nature of the system meant that defendants and officials would not be able to understand why a decision was made, making an effective challenge impossible.

Government agencies need to have the resources and skills to thoroughly examine how they procure ADS systems.

A politics of care

All of these recommendations point to the importance of a politics of care. This requires those wanting to implement ADS to appreciate the complexities of people, communities and their rights.

Key questions need to be asked about how the uses of ADS lead to blind spots because of the way they increase the distancing between administrators and the people they are meant to serve through scoring and sorting systems that oversimplify, infer guilt, wrongly target and stereotype people through categorizations and quantifications.

Good practice, in terms of a politics of care, involves taking the time to carefully consider the potential impacts of ADS before implementation and being responsive to criticism, ensuring ongoing oversight and review, and seeking independent and community review. Provided by The Conversation

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Citation: Governments’ use of automated decision-making systems reflects systemic issues of injustice and inequality (2022, September 22) retrieved 22 September 2022 from https://phys.org/news/2022-09-automated-decision-making-issues-injustice-inequality.html This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Keyword: Governments' use of automated decision-making systems reflects systemic issues of injustice and inequality

TECH'S NEWS RELATED

Generating New Materials by Mimicking Fundamental Rules Hidden in Nature’s Growth Patterns

Caltech researchers developed a framework to design new materials that mimic the fundamental rules hidden in nature’s growth patterns. Credit: Caltech Inspired by the way termites build their nests, scientists at the California Institute of Technology (Caltech) developed a framework to design new materials that mimic the fundamental rules ...

View more: Generating New Materials by Mimicking Fundamental Rules Hidden in Nature’s Growth Patterns

Digital Assets And Fintech Are The Evolution Of The Financial Industry

Influence Of Financial Technology And Digital Assets The Growth Of Fintech And Online Shopping A Rising Need For Data Protection A Crashing Tide Of Cryptocurrencies Fostering Economic Growth In Conclusion (Photo : Kanchanara on Unsplash) The birth of a new business exemplifies the world’s ongoing effort to adapt to ...

View more: Digital Assets And Fintech Are The Evolution Of The Financial Industry

New Hacker Technique Lets Attackers Disable Antivirus Solutions and Infect the Users Device Anyway

A New Method was Discovered and Being Used by Hackers to Bypass Antivirus Solutions The Vulnerability Method was Known as CVE-2019-16098 BlackByte was Found Using the Method to Disable Over 1,000 Drivers Sophos Gave Tips on How Organizations can Protect Themselves Against This Method Lazarus Group was Spotted Using ...

View more: New Hacker Technique Lets Attackers Disable Antivirus Solutions and Infect the Users Device Anyway

Honda Reveals the Design of the Electric Prologue SUV

The Prologue SUV Hybrids and Electric Vehicles Today Honda has finally revealed more details about the Prologue SUV, according to Engadget. Perfect for vehicle owners who don’t like flashy designs, this electric vehicle may just be the perfect choice for you. (Photo :  -/AFP via Getty Images)The interior of ...

View more: Honda Reveals the Design of the Electric Prologue SUV

Learn a new language, skills and more with this self-starters bundle

Let’s face it: Most of us take the internet for granted. Right here in our pocket is a portal to most of the world’s accumulated knowledge, and what do we use it for? Games and Keanu memes (not that there’s anything wrong with either). Instead of looking to the ...

View more: Learn a new language, skills and more with this self-starters bundle

Rocket Report: Falcon Heavy launch on tap; South Korea seeks Russia alternative

Also, the Ariane 6 rocket is inching closer to the launch pad.

View more: Rocket Report: Falcon Heavy launch on tap; South Korea seeks Russia alternative

Food from the printer

Fibrous structures Products for lung patients and soldiers Making meat juicier Immediate consumption Read more about printing food Read more Wageningen World stories Education & Programmes Research & Results Longread Printing food sounds futuristic, but that future is approaching fast. And Wageningen is at the forefront; the latest success ...

View more: Food from the printer

How does the Pixel 7 stack up against Apple's iPhone 14 and the Samsung Galaxy S22?

The Pixel 7 line-up is now official but you may ask yourself how it stacks up against the competition. Here is a quick comparison of the specifications of Google’s latest flagship with Apple’s iPhone 14 line-up, and Samsung’s popular S22 series. The brass tacks: Pixel 7 and Pixel 7 ...

View more: How does the Pixel 7 stack up against Apple's iPhone 14 and the Samsung Galaxy S22?

In Apple's shadow, Google takes new route to face recognition on Pixel phones

Elon Musk and Hu Xijin face off on Twitter as Tesla founder claps back at Chinese nationalist influencer over Russia-Ukraine war comments

Octopus Ventures bolsters deep tech investor team

NASA Research Says Moon May Have Formed in Mere Hours

Can Cats and Coyote Co-Exist? Surprising Findings From Suburban Los Angeles

iPhone should switch to USB-C, says iPod inventor Tony Fadell

New solution from Silicon Labs streamlines Amazon Sidewalk developer journey

iCatch Technology AI imaging SoC driving electronic mirror and ADAS advancement

3 Gaming and social trends to watch out for in the metaverse

UK AI investment reaches record $3.6bn despite economic downturn

EU says Shopify will improve platform to make online shopping safer

Pixel 7: facial recognition is finally available but there is a catch

OTHER TECH NEWS

;