How to see your system clearly

A guide for making sense of your system.

Shady Sharify / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/tion...

Overview

Once you start interrogating and poking holes in the frames and narratives that hide power in technology, you’ll find a system. Basically, a ‘system’ exists wherever people are trying to meet a need or solve a problem. So a neighborhood trying to foster community or improve safety is a system. A constellation of service organizations trying to reduce homelessness is a system. A group of non-profits trying to ensure ‘AI’ serves the public interest is a system.

Systems are all around you but often invisible. You just see a recurring output, and, if you care about changing the system, you have to make visible how the interactions of the different parts of the system (people, technologies, organizations, cultural norms, etc.) produced it. And you have to do so in a way that gets the key stakeholders in the system on the same page about what it looks like, how it behaves, etc. Seeing your system clearly is about developing enough awareness so that you can strategically intervene amidst uncertainty — and, intervene in a way that takes account of how the system and its stakeholders are likely to respond. Which. Is. Hard.

Some groups only see part of the system. Different groups will have different ways of seeing (e.g. different backgrounds, expertise, epistemologies, etc.), which will shape how they see and what they ignore. Others will deny that they’re part of the system -- ignorance really is bliss when the system is tilted in your favor. How different groups make sense of data (e.g. what they can say, what they can’t) and technology (e.g. the values and beliefs reflected in it, how it impacts different communities, how it shifts power, etc.) will fundamentally shape how they see their system. Moreover, some systems are harder to see clearly because they’re complex and uncertain, behaving in ways you can’t predict. If people are involved, then the system includes hierarchies, power dynamics, cultural norms, and status, which tend to hide in plain sight.

Seeing the system clearly requires not just that we interrogate the frames and narratives that hide the real problem but that we map the stakeholders, cultural norms, technologies, and power dynamics that make up the system, and keep it in place. (This overview doubles as a plug for my course, Sociotechnical Systems Change, because we can’t meaningfully change our system without seeing it clearly.)

E.G.

In 2014, Amazon developed an algorithmic recruitment tool to support its hiring process. Amazon hoped the tool would increase efficiency while avoiding bias against women by eliminating social identity markers. The recruitment tool was trained on the previous 10 years of employment data, and technology was —and is — a male-dominated sector. So the algorithm assigned higher scores to men.

While we’re often quick to focus narrowly on the algorithmic impact, there is a system producing this outcome, including things like hiring practices and corporate culture, data that reflect and reproduce societal biasses, gender norms, ideas about meritocracy, etc. But if you’re part of the system, it might be challenging to see your hiring practices clearly. Or if you’re a trained engineer or statistician, you might see the biases as statistical, not social and systemic. Or if you’re a man who benefitted from this system, you might not see the social and cultural norms that nudged you along the way and defend the existing hiring practices as ‘fair’ or meritocratic.’ Or you might see them clearly, which creates a different set of dynamics.