Why an AI needs rules

Algorithms now permeate our everyday digital lives more than we often realize: From risk assessment for a loan, to product recommendations on e-commerce sites, to image optimization in Adobe Photoshop - automated decision-making tools analyze us, give us suggestions, and thus also subtly steer us. But do they do so entirely without ulterior motives, and who decides about them?

Automated decision support and algorithm bias

Systems for automated decision-making also include "chatbots, data-based targeted advertising, or navigation systems," as Jessica Wulf of AlgorithmWatch explains. This nonprofit organization conducts research on AI-enabled services and their societal impact. Research on algorithm-driven decisions is important, because often in everyday life we don't even notice that an AI has (helped) decide at all. For example, in job postings that are more often shown to female user groups than male users, as AlgorithmWatch demonstrated in an experiment.

Rare extreme cases of prejudice illustrate all the more how complex the issue actually is: a study two years ago showed that image recognition systems, such as those used in autonomous vehicles, were less able to recognize people of color. But what was the reason for this so-called algorithm bias? First, the image recognition software was fed too few people with dark skin. Second, the AI did not give enough weight even to the few photos with people of color. In another recent case at Harvard's U.S. University Hospital, algorithms prioritized physicians for Covid 19 vaccination who worked from home over nurses on site at the clinic. What had gone wrong? Hospital staff were not involved in the development of the vaccination prioritization algorithm, nor were they able to provide feedback when the algorithms were deployed.

Recognizing discrimination risks in the first place

Until such a guideline is put into practice, the users are in demand. Everyone should take a very close look at automated decision-making aids and also question them. But this is exactly where the problem lies. Help is coming from Unding, which is also supported by the Bertelsmann Stiftung - and from AutoCheck, a new project from AlgorithmWatch. Until July 2022, project manager Jessica Wulf and her team will be working on guidelines for anti-discrimination counseling centers to be able to recognize discrimination risks in the first place. She reports: "It is very difficult to notice that I am being discriminated against by an automated system. For one thing, because of the lack of information that an automated system is being used here. For another, I don't have comparisons to other people and their results to determine that I've been treated differently - and possibly discriminatorily." At the moment, the team is tracking down case studies and also relies on tips from users. Specifically, for example, AutoCheck was informed that some images of black people appeared under the label "animal" on an iPhone 8, meaning they were automatically assigned to that category. "The results of the project, instructions for action and workshop concepts, are primarily aimed at employees of anti-discrimination counseling centers. Their competencies are to be supported so that they are better able to assess and recognize risks. This should provide better support for those affected in concrete cases of discrimination," says Jessica Wulf. Making people aware of possible discrimination through automated decision-making aids is therefore at least as important and fundamental work as a set of rules for the development of these decision-making aids. But if we're going to start questioning, let's be more fundamental: Jessica Wulf calls for more social discussion about whether and where we can use automated systems sensibly - and where it's better not to. 

Text: Verena Dauerer