TEC Talks Podcast Transcripts

14. Our Data Privacy and the Issue With Inferences (November 30, 2022)

Guest: Ignacio Cofone (McGill University)

Transcript

Kirsten Martin  0:01 
(voiceover) Hey, it's Kirsten. Just wanted to let you know that this is our last episode of 2022, but we'll be back early next year with more TEC Talks and more great guests. Thank you for listening, and happy holidays. (end voiceover)

Kirsten Martin  0:15
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we talk about an important, idea, paper, article, discovery in tech ethics, and today, I'm so happy to be joined by Ignacio Cofone. Ignacio is an assistant professor and Canada Research Chair in Artificial Intelligence Law and Data Governance at McGill University's Faculty of Law. His research focuses on privacy harms and on algorithmic decision-making to explore how the law should adapt to technological and social change. He was previously a research fellow at the NYU Information Law Institute, a resident fellow at the Yale Law School Information Society Project, and a legal adviser to the city of Buenos Aires. He has a joint Ph.D. from Erasmus University in Rotterdam and Hamburg University, and a J.S.D. from Yale Law School, which is a doctor of science of law, which I did not realize. But that was very on-point.

So today, we're gonna take a deeper dive into your Illinois Law Review article "Privacy Standing." And I thought we could start with this idea--so what I liked about this article was you really kind of slow down what we're talking about when we talk about a loss of privacy. So we sometimes just say, That's a privacy loss. And we just kind of flippantly talk about this, and we try to disentangle what this means. And you kind of ask us to slow down as to, like, each of the piece parts about what's going on with a loss of privacy, privacy harms, and privacy injuries. And what I thought would be helpful is to find out what--usually there's a problem that you saw, either an incident that occurred or a problem within the theory that you were trying to solve. Like, what was the problem that you saw going on that you were like, Okay, I need to figure out how to try to solve this.

Ignacio Cofone  2:00  
Yeah, that's actually a good question. Well, first of all, thank you so much for having me here; I'm very excited to talk about the article with you. So I think the problem that I was trying to solve with this article is a consistent line of terrible Supreme Court jurisprudence (Kirsten laughs) regarding privacy harms. And how can one navigate it cohesively, and how can one comply with especially federal courts with constitutional requirements, and with Supreme Court requirements of dubious constitutional value, and still give some recognition to people's privacy injuries?

Kirsten Martin  2:34
So you're saying, like, the people, the courts were really struggling with how to identify or make decisions around privacy when they were trying to identify a privacy loss. Was a harm required? You know, what type of injury counts to say that you have standing? Maybe it would help to kind of back up and say, What--because the article is actually called "Privacy Standing," which requires harm. But maybe it would be worth backing up and saying, Why is privacy standing important?

Ignacio Cofone  3:02
Privacy standing is important because it determines to which extent people will get compensated for the harms that they suffer regarding their privacy, and regarding a bunch of other consequential harms that are linked to their privacy, such as their reputations, such as being discriminated, such as financial harm. And it is important because the extent to which we allow it also will determine the disincentives that corporations have to create those harms or to not have care to prevent those harms in absence of regulatory enforcement. And regulatory agencies are very important in privacy, but they have limited powers, and they have limited resources and limited attention. So giving people the right to sue when they were harmed is important for protecting our data rights.

Kirsten Martin  3:47
And so to be able to say that you have standing to sue--to say, You hurt me, I want something back--you have to actually prove, show that you had an injury, that the organization caused that injury, and that it's fixable in some way, that there's some redressability. Is that right?

Ignacio Cofone  4:04
Yeah, that's right. If we get into legalese--

Kirsten Martin  4:06
No, yeah, yeah.

Ignacio Cofone  4:07 
No, no, but if we get into legalese, we usually would call [it] "standing" only when it's for statutory privacy, and when it's tort law, we don't usually call it "standing."

Kirsten Martin  4:15
Okay.

Ignacio Cofone  4:16
But the principle of being, of importance of the right to sue applies to both, and the requirement of harm that you're bringing up, that's a really insightful point because courts have been really adamant about it in statutory law. But in tort law, they recognize that if someone invaded your privacy, then you must have suffered a privacy harm and the evidentiary requirements are different. And there's a lot of room to learn from tort law as to how courts apply statutory law.

Kirsten Martin  4:42
Okay, okay. So for the layman, because like the--and this gets a little bit ahead when you talk about the different types of, like, injuries that you can feel from privacy and the different types of harms--but the idea is that sometimes you can just have a harm just from the loss of privacy; you don't have to say I had a reputational harm, you don't have to say any of the kind of later-on harms that you get from the loss of privacy. It's the loss of privacy itself that can be considered a harm. Is that fair?

Ignacio Cofone  5:11 
That's fair. Or at least that's what the law should do.

Kirsten Martin  5:14
That's what the law should do. (laughs) Fair enough, fair enough, fair enough. So one of the first things you define is a loss of privacy, which seems very subtle when you define it, but it is actually super important. And so how, can you talk about how you got to the idea of, like, what constitutes a loss of privacy or a diminished--it's on a continuum, but, like, having less privacy?

Ignacio Cofone  5:35
Yeah, I got to this definition of loss of privacy by thinking about inferences. So many harms that people go through happened not because of collected information or because of information that they shared, but because of information that was inferred about them. And most of the binary definitions of privacy, such as privacy as secrecy, don't capture that. And they don't capture the fact that privacy may be lost in different graduations and that if I share some information with one person, that doesn't mean that it is not, that it is not private anymore because that's to keep expectations of privacy towards other people. And say that I tell you something, and then you tell everyone Notre Dame what I told you, it is not the case that I didn't lose any privacy because the information wasn't private anymore; it may very well be the case that I lost privacy, at least descriptively. Then we can get into into a conversation about whether it harmed me. But that thing that seems minor is important to avoid dismissing people's injuries.

Kirsten Martin  6:34
Right. And so the casual way that people have talked about loss of privacy is actually an idea around secrecy, which you mentioned, or sometimes control. But the same idea goes towards, if you hand over privacy to someone or an organization, disclose it in some way--either typing it into a box on their webpage or their app, telling it to somebody, it could be all sorts of ways--that that's actually where the privacy loss occurred. You know, in, like, handing over your data. And you're saying that actually, that what they do with the data can actually constitute a new privacy loss, you know, that when they hand over the data, there's still a privacy expectation about how that will be used or shared, and the loss of privacy depends on whether somebody else actually gains "probabilistic information about the observed." Which gets away from this idea of whether they actually collected no data about you. So you could have not shared any additional data to me, but I figured out more information about you, and that can still constitute a privacy loss. Is that--

Ignacio Cofone  7:37
Exactly. Yeah, that's exactly right. And so many privacy harms happen not because of information that is collected from me, but collected from people similar to me, or people--or information that is inferred about me from other data that I disclosed.

Kirsten Martin  7:51
Right, yeah, right. And so I think this is where attempts to fix the handoff of information--whether it's, you know, giving me rights to the data, you know, having me own my data, you know, in some sort of like web 3.0 idea of making it harder and harder for you to get my data--doesn't actually address in many ways the majority of the privacy issues that are going on right now with the way that data is aggregated and inferences are drawn, in that I don't have to share any data for a company to figure out a lot of information about me. Is that--

Ignacio Cofone  8:28
Yeah. Data ownership would even make the problem worse because privacy law already focuses too much on the moment of collection and not enough on inferences and processing. And data ownership would shift all the regulatory burden onto the moment of collection, which worsens all the problems that we know in privacy law, such as asymmetry of power, asymmetry of information, the importance of inferences, and the wide scope that corporations have to do whatever they want with our data after they obtain it.

Kirsten Martin  8:58
Oh, good, good, good. Yes, right. So not only is it not solving all of the problem, but it can be a distraction and get companies to only focus on this one thing, which they're too focused on anyway, which is this handoff of information. And it could mistakenly make people think that their privacy is being respected when it's not. You know what I mean? Like, so they could actually have this quote "solution," a privacy solution, that really does absolutely nothing to stopping companies from knowing a lot of inferential data about you, information about you, that doesn't rely on you sharing any of that information. That's a great point, you know, that it's not only a distraction, it's not only addressing the problem, it's actually--it can be making it worse by focusing on it.

Ignacio Cofone  9:40
And that's why corporations are always so happy to say, Yes, you should own your data (Kirsten laughs), you should have control over your data, and politicians funded by them are always very happy to say, Yes, we should totally give people control over their data and have them own their data. Because it won't really help them.

Kirsten Martin  9:53
Right. Yeah, and this is where, this is your own words, so I'm just going to quote back to you, but you said--the cases that you go through, and you go through cases that are great, and you say "highlight a major policy aspect of privacy harm decisions: it is rare that a piece of disclosed personal information is the information that produces the harm to the person." Like, so it's often not the issue of what was disclosed, it's what they figured out about you based on that or other information that was there. Okay, so then, so we have this idea of the loss of privacy that's a little bit different. It's a slight shift, but it's kind of known in the literature that this occurs. And so then you move from privacy loss to how to figure out a privacy harm--like, pulling those two concepts apart and defining one and defining separately. So can you talk a little bit about what you talk about with privacy harms?

Ignacio Cofone  10:41
I think it is worth separating privacy loss from privacy harm so that we have a descriptive element that correctly captures inferences that may be over-inclusive as to what we want to protect, and then we narrow it down with the values that we believe that privacy protects, such as autonomy or expectations. So I think the main difference between a privacy loss and a privacy harm revolve[s] around social expectations and social norms. So a privacy loss that is a privacy harm is one that happens contrary to our social rules about what is acceptable and what is not acceptable to collect or to share about other people and that interferes with the different values that the literature has discussed privacy protects, such as intimacy, autonomy, and people's well-being.

Kirsten Martin  11:27
Mm hmm. And you you also make a move to separate out privacy versus consequential harms.

Ignacio Cofone  11:33
Yeah, I think that is really important, and it is an aspect that courts often don't see. So oftentimes, when courts require some type of harm to provide standing, what they mean is that they require a harm that is not privacy. So they ask, like, Has your reputation been harmed? Have your finances been harmed? Have you been physically hurt? Have you been discriminated [against]? And a lot of those things are really important, and all those are values that protecting our personal information does feed into, and they are interests that our privacy protects, but it may very well be the case that someone breached your privacy, and none of that happened. And the reason why it is practically important to distinguish that is that oftentimes when those things happen, they happen really far down the road. And once they happen, we cannot point to a causal link to sue. So you might think, for example, a data breach, a large data breach, like the one Equifax had. If you're a victim of the Equifax breach, and then you sue, then the judge will tell you--if you get one of the judges that require harm--the judge will tell you, Well, have you suffered identity theft or credit card fraud? And then you say actually, No, not yet. But that's something that can happen to you in five or 10 years. And when it happens, you won't be able to trace it back to the Equifax hacks. So now you get no remedy because it didn't happen yet, and then you get no remedy because you cannot establish the link.

Kirsten Martin  12:56
So you're stuck. And I think this was a great move--it does a lot. So moving, saying that there's a privacy harm, which is the harm from the loss of privacy, and that can be around dignity, autonomy, all the reasons why, the normative reasons why we think privacy is important. And we justify those in lots of different ways for individual[s] in the society and why it's important. And moving the word "privacy harm" off of the consequential harm. So these are the downstream effects from a loss of privacy. And so this, this could be, I'm unable to get a job, you know, my reputation has been harmed in this way, I was denied a mortgage--like, things that we can put in a spreadsheet, and we can sum up, and we can say, This is actually the value of these consequential harms. And the move is wise because we previously, in calling them privacy harms, we actually placed too high of a burden to be able to quantify the harm of actual privacy harms, if that makes sense. So we tended, when we called them privacy harms for the last 20 years or more--I mean, like, for a long, long time, we would call them privacy harms, and people did work like Ryan Calo about objective versus subjective privacy harms and trying to tease out what was going on under the general rubric of privacy harms. But what this move does is it allows us to talk about consequential harms as a different category with a different name, and so we're less likely to hold them up against one another to say, Well, you have a lot of these funny things that really aren't quantifiable in the courts, or to an organization to say, Well, what harm is there? What's the privacy harm? When some were quantified and some were not, it was a tough battle. (laughs) And in calling them consequential harms, it really is helpful for organizations, too, to be able to say, like, Look, there's a privacy harm in and of itself; we can also talk about consequential harms. But this is considered, you know, harmful, and we don't even have to put a number on it.

Ignacio Cofone  14:49
Yeah, and some institutions like the Knight Institute, for example, have done really productive litigation with mixed results by telling judges, because all these consequential harms materialize later on and then we kind of prove the costs, what we should take into account is the risk of those consequential harms. And that's way better than the status quo. However, it has two limitations. The first limitation is that it's a larger departure from the existing legal requirements than just recognizing that privacy harm also exists and is different from those, because then we'd have an actual harm to tell federal courts that they need to recognize. And second, as you were saying, oftentimes proving the risk is really difficult.

Kirsten Martin  15:30
Right.

Ignacio Cofone  15:30
And if we have a recognition of the privacy harm, then we could remedy that, and then remedy the actual consequential harms downstream.

Kirsten Martin  15:37
I just think that, it's a move that you make within the article, and I will use it a lot because I think that it's really helpful to kind of cordon them off as separate and consequential, but not exactly--but leave privacy harms for the label of actually the harm from the privacy loss. Okay. And then you make a third move. (Ignacio laughs) So it was the privacy loss move, the privacy harm move, and you make a third move from privacy harm to actionable privacy injury. So can you talk about that kind of subtle difference?

Ignacio Cofone  16:05
Yeah. So that's mainly to satisfy the law that federal courts have to follow from the Supreme Court.

Kirsten Martin  16:11
Okay.

Ignacio Cofone  16:11
So if we didn't have Article III constitutional requirements, as delineated by the Supreme Court, so if we were at a state court, for example, then possibly we could say that privacy loss plus privacy harm is enough to sue. But we do have a requirement of showing, in most legal contexts, that whatever privacy harm accrued has to be the consequence of a legal wrong to be actionable.

Kirsten Martin  16:37
Got it.

Ignacio Cofone  16:37
There are different ways to define that legal wrong. One way to define that legal wrong is the breach of a statute. But judges could get more creative in satisfying that legal wrong by interpreting a tort more expansively, particularly if they're state courts.

Kirsten Martin  16:53
Okay. Got it. I think that the issue of privacy harm, I don't think a lot of people realize how important it is to movement around regulations or taking cases to kind of show some sort of harm that went along with the privacy loss, you know, previously. That it's really difficult, especially with before, when they were just focused on contracts and not really open to talking about torts and harms, and would really, really overly focus on this handoff of information and whether or not there was some sort of breach of what was promised. You know, versus understanding that, like, harms can come regardless of that handoff of information when no information was [inaudible] gathered about us.

Ignacio Cofone  17:36
Yeah, there is some historical, and particularly in the US, notice and choice has such--

Kirsten Martin  17:42
Yeah.

Ignacio Cofone  17:43
Has had such a prevalence in the way that we think about privacy. Now, we think it's almost silly to think about privacy in terms of, Well, companies have to be really clear about what they do, and as long as they're clear, they can do whatever they want. And there are movements to depart from that. They're really important principles, particularly in state legislation that have been enacted. But I think even the most robust versions of the legislation that we have, such as the CCPA, have room for improvement if they move further away from recognizing that people kind of waive things through consent and have a larger focus on actually reducing the harms that happen.

Kirsten Martin  18:22
Well, and I think your point about ownership of data as being a distraction, I mean, notice and choice in many ways is the initial distraction (Ignacio laughs), like where it actually did a lot of damage by--I mean, I always jokingly say, you know, we told them to just focus on consent and notices, and businesses did it for decades. I mean, that's what we told them to do, and then, and we told them, you know, whatever you do, don't make a falsehood in that notice. And so they just made no claims whatsoever in their notices. And so, in some ways, they were extremely responsive, you know, to the regulators in the United States, in that they did whatever the FTC and others wanted, you know, in that realm.

Ignacio Cofone  19:02
There are some amazing cases, like the Snapchat cases, that got sanctioned by the FTC for over-promising, and then as a consequence, had actually minimal changes in the way to handle the data and large changes in how they described that they were handling the data.

Kirsten Martin  19:15
(laughing) Right, right, exactly. That's what I mean, like they'll do, they're very responsive. In some ways, it's why I always say, like, whatever we decide as a society is important to regulate or court cases to bring and adjudicate, businesses will respond. And in some ways, like, this idea that they're gonna go under if we ever do X, I always am like, Oh, they will figure this out. They figured this out in the past, and they actually are extremely responsive. And in some ways, that's a great example of like, Oh, the notice was too clear? (laughs)

Ignacio Cofone  19:52
(laughs) Yeah, exactly, yeah, yeah, yeah.

Kirsten Martin  19:53
I can fix that, I can actually make it obscure and actually, like, lessen the amount that I'm promising you that I would ever provide in terms of privacy protection.

Ignacio Cofone  20:03
Yeah, particularly when the regulations address not the business model itself but aim at reducing the undesirable consequences from the business model such as privacy harm, then businesses can adjust. It would be more difficult to tell a business, Oh, it should just all stop collating data and making inferences because then it would be [inaudible] the data. But if you say something like, Well, you may be liable more often for the harms that you produce to people when you're making money, then that is the way that the law deals with so many other areas where companies are making money and producing unintended harms.

Kirsten Martin  20:36 
Right, right. And in some ways, and this is where businesses are, you know, just not, what we were talking about beforehand, just not keeping the collected data and making inferences about it, and we don't have a method of dealing with that right now. And even though there's a loss of privacy, they have new knowledge about us that we did not give them, you know, or did we think about them having and there could be a harm from it--or there is a harm from a privacy loss, and there could be actually an injury as well. And we are not thinking about--or we're thinking about it, but we don't, we're not used to thinking about how to do that in the law. You know what I mean? Like, so we're struggling.

Ignacio Cofone  21:13
And the ways that we have to think about it, like the purpose limitation principle and the data minimization principle, are so onerous that then we end up not taking them very seriously. So the purpose limitation principle, for example, would be great to address some of that, to say that, Well, they have to have it for a certain purpose, and then they have to acquire consent again, whatever that means, and it's not that difficult to have a different purpose. But then, because companies often don't know what purpose they're going to give the data later on, we end up accepting very wide purposes, like "marketing." So we end in this weird dichotomy [inaudible]. If we took the purpose limitation principle, which is actually protected very seriously, then the cost would be enormous. So we take it not seriously at all, and then it's almost [inaudible] having it, while other solutions that try not to address the business goal directly but then discern consequences from it, like a more robust liability system, would be better in that sense.

Kirsten Martin  22:05
Yeah, that's a great point. I remember there's, if you ever read back into like the 1960s, you know, economics of information stuff that all this is built upon--like, so all the notice and choice is kind of built upon these, like, three people in Chicago, you know, in the '60s. (Ignacio laughs) But they assumed that we would never buy and sell data, that it would always be--they literally assumed it would always be cheaper to always just ask Kirsten again for the data.

Ignacio Cofone  22:30
Yeah.

Kirsten Martin  22:31
And they never imagined a data market. They literally said, like, Well, no one would ever buy and sell it, it would always be better just to go back to the individual and ask them for that data again. Because they thought that the only place that you could actually hold onto that much data is the government. So this idea of, like, asking permission and that being, like, the way that you can make sure that you trust whoever has your data, and that's the fix, was literally premised on an old marketplace, you know, that just doesn't exist anymore.

Ignacio Cofone  23:02
Yeah, absolutely. Even when I speak with economists today, sometimes they don't immediately see how is this different from a normal standard form contract, like the one that you may have with your internet service provider where you don't read the contract. And then it's like, Well, you know, the harms are unknown, the set of potential negative consequences to you are unknown, the risk is unknown. And it's different than when you know exactly how much you're paying for internet, and then you may not read the small print and other clauses.

Kirsten Martin  23:27 
Well, when you have a hammer, everything's a nail, so (both laugh)--everything comes back to a contract and transaction costs. So I don't know. But even within those, you can show that it's not working, like, if you actually look and show, like, how the market works. And I, what I think is the future, and you're really hammering it here, and I know you have a book coming out is this "problem"--and I'm putting that in air quotes, even though it's not visual--the problem of inferences drawn about us. And I just think that that, like, talking about that is the issue. You know what I mean? Like, and getting in front of it and figuring out what shifts we need to do in the way that we even think about knowledge about us is kind of the point. And I'll just use a quote from, this is from you, so I'm not making this up. But what you said is, "Most often, information acquires its harmful characteristic through the process of aggregating different pieces of personal information and inferring new information out of them. In other words, harmful information is rarely collected information and is frequently inferred information produced by aggregating different pieces of seemingly inoffensive collected information." And I think that's key, is that a lot of times the data that is drawn seems inconsequential, you know, it's this exhaust that it's hard to point to the collection as actually the problem.

Ignacio Cofone  24:39
Sometimes even the inferences made about you are not on the basis of only the information collected from you, but information collected from others. So consent is absolutely irrelevant.

Kirsten Martin  24:48
Right, right. In fact, they might only need a few pieces of inconsequential information that I think they have and they should have, like, let's just say my age and my gender and where I went to school or something like that. But they know a ton about a million people that are just like me, and so then they're able to draw inferences about me. You go onto say, "When we share something about ourselves, we simply do not know what other information is out there for malicious actors to aggregate and use." And then, "Thus, if no remedy is provided for each illegally disclosed piece of information that contributes to the aggregation, but then the aggregation that can produce harm is invisible to the law, no remedy will ever be provided." And this is like this sticking point that we're at, where the collection was, what we're going to call unproblematic. You know what I mean? Like, or maybe it didn't even occur, there was no collection of data. And yet the aggregation of that data is literally invisible to the law because we don't see it as something that we should care about. The entire process, there's no remedy. Because it doesn't see it as a problem. And yet that, as you point out earlier, that's actually where the majority of the harms are occurring, is from this knowledge that's generated about us. So I think, I really appreciated the article just in kind of very clearly articulating and really walking through, like, why this is a problem right now and our lack of a remedy for it. Which is important because sometimes you can say, Well, we have a solution for that, this already exists. But that's just not the case right now.

Ignacio Cofone  26:15
Yeah. Even laws that are coming out recently, and there's some recognition of inferences in California, but other than California right now, even laws that are coming out now like the proposed bill in Canada that are quite robust and have quite a robust set of data rights, don't explicitly recognize inferences. Huge gap.

Kirsten Martin  26:34
Hopefully people will start doing more studies, also, like, writing about it because I really think that it like is an all-hands-on-deck type of situation. But before--I won't take any more of your time, and I would love to have you back when your book is out because I know it'll cover how to address inferences and stuff like that.

Ignacio Cofone  26:49
I'd be very happy to, thank you.

Kirsten Martin  26:50
Is there anyone--we always ask if there's anybody that we should be paying attention to, someone that you're like, Oh, I can't wait to see what they write next, young scholars in law or outside, any discipline?

Ignacio Cofone  27:02
Yeah, that's a tough question because there's so many good ones.

Kirsten Martin  27:04
I know.

Ignacio Cofone  27:05
So if I think about people in law, which are the people that I know better, I really like--well, first of all, Salome Viljoen's work, you already spoke with her. And two other scholars that I always pay attention to what they write are Rebecca Wexler and Margot Kaminski.

Kirsten Martin  27:20
Oh, that's great. Yeah, those are great suggestions. Yeah, it's so funny, especially, I always say it's a young person's game. But like, there's always, there are a lot of young scholars in the area just doing such interesting work from all different areas. And I always love reading a new thing, and I'm like, Oh, I never thought about that, that's so interesting. (Ignacio laughs) Like, they just think of, like, such different takes on things, and it's a great community, I have to say, in that respect, in that there's so much kind of work to do that it seems very collaborative, in that people just want more good ideas out, and they're not trying to scrape and, you know, no one feels impinged upon in their ideas, you know, when a new idea comes out.

Ignacio Cofone  27:59
Yeah, we're really lucky. Both the junior and the senior people working in this field have been so generous in feedback and in helping out with ideas.

Kirsten Martin  28:07
Oh, I agree. I mean, when I first started going to the Privacy Law Scholars Conference, I just, I was kind of like, it was an eye-opener to me how you could be really constructive and helpful and come out with new ideas all the time. And you'd have these people asking such interesting questions to kind of push your thinking, and it was just very collaborative and constructive.

Ignacio Cofone  28:24
Well, I remember the first PLSC that I went, I think it was 2014, I had a really productive conversation with you about the paper that I was trying to write. (both laugh)

Kirsten Martin  28:31
Oh, good. Yeah. Yeah, yeah, well, that's how we met, at PLSC. I mean, I always joke they should have like a, you know, people that come out of PLSC and, like, start writing together and do things together because, like, there's all these things that come out of it. They could, like, have a tree that shows all the people that come out of it, especially across disciplines. Well, thank you so much. I really appreciate you coming on and talking and stuff like that. So I know it's super busy.

Ignacio Cofone  28:57
Thanks. This was fun.

Kirsten Martin  28:59
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

13. AI, Anti-Discrimination Law, and Your (Artificial) Immutability (November 16, 2022)

Guest: Sandra Wachter (University of Oxford)

Transcript

Kirsten Martin  0:03  
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we discuss an important idea, paper, article, discovery in tech ethics, and today, I'm so happy to be joined by Sandra Wachter. Sandra is a professor of technology and regulation at the Oxford Internet Institute at the University of Oxford, where she researches the legal and ethical implications of AI, big data, and robotics as well as internet and platform regulation. Her current research focuses on profiling, inferential analytics, explainable AI, algorithmic bias, diversity, and fairness as well as governmental surveillance, predictive policing, human rights online, and health and medical law. She studied law at the University of Vienna and has a Ph.D. in technology, intellectual property, and democracy. So we take this, the idea of this series is to take one idea or case and examine the larger implications for the field of technology ethics, and today, we're gonna do a deeper dive into your article in the Tulane Law Review called "The Theory of Artificial Immutability: Protecting Algorithmic Groups Under Anti-discrimination Law." So I thought we could start with kind of the, I'm putting this in air quotes, the "problem" that you were trying to address. And when you--that's usually how these things start is, you see a problem, and then you're trying to figure out how to address it. So what was that problem that you saw that you're trying to address with this paper?

Sandra Wachter  1:24
Yes, so I'm very much interested in the question of algorithmic accountability and how we can make those systems work for us rather than against us. And I think most of us will be aware that one of the issues that always comes up with AI is that they can be discriminatory. And I think many, many fantastic people have written on this. One of the reasons why this is the case is because, you know, it's collecting historical data, and therefore it's transporting inequalities from the past into the future. And so very often you have data that links back to protected attributes, such as ethnicity or gender, sexual orientation, religion, age, those kinds of things. And so there's very interesting research going on, and I have done my share fair in this area as well. But then I came to realize that another thing is happening as well, that algorithms are not just grouping us similar to protected attributes; they're also grouping us according to groups that fall outside of non-discrimination law. So for example, you could be applying for a job, and if you were to use a browser such as Internet Explorer or Safari, you're more likely to get rejected than if you used Chrome or Firefox, for example. 

Kirsten Martin  2:44
Mm hmm.

Sandra Wachter  2:45
And so that's quite interesting because obviously, browser usage or Safari user group is not something that is protected under the law. But nonetheless, it's holding you back in the same way. And so I started to realize that AI is grouping us in all those different types of groups, making very important decisions about us, yet the group that I'm part of has no protection under the law. And so I got interested in this topic.

Kirsten Martin  3:13
That's great. Right. And I think, because sometimes, and especially in the more technical work, we see people really focusing on protected classes as a shorthand for lots of things--like, if we could fix this protected class, but the problem is always that there's a lot of things going on around grouping. And you mentioned two groups that are, that you see, these algorithmic groups, and I didn't know if you could explain--I thought that was interesting how you categorized these two types of algorithmic groups that you were worried about, or that we should be thinking about, and if you could explain those. I think one is, like, non-protected, and then the other, incomprehensible.

Sandra Wachter  3:48
Yes, exactly. It makes sense to categorize them because it just shows how deep the problem actually goes. So there are algorithmic groups that are created by an algorithm that fall outside non-discrimination law because they're not seen as a protected attribute. This could be, I'm not getting a loan because I have a dog, dog ownership as a group.

Kirsten Martin  4:11
Right.

Sandra Wachter  4:11
It could be Safari user, it could be fast-scrollers, it could be football players, sad teenagers, video gamers. Those are all groups that are already being used to make decisions about people, but those groups don't have any protection. And the other side, there are algorithmically created groups where we don't even have human understanding or human language to describe what's going on, yet they are being used to make decisions about you. So it's, you know, electronic signals that you send out by your computer where we don't have a human concept to describe what that actually is. And so protection for that is even harder because I don't even have a social concept to describe what's going on.

Kirsten Martin  4:57
Right. And so I think, and this is where I thought, what I really liked about the paper--so it was kind of, you could see this process of, there seems to be something wrong that we're using these groupings to make a decision about someone. They're not fitting into these classes that we've decided are protected, and so before jumping to saying, Why should we protect them? You take the step and say, What were we trying to do with discrimination law and protected classes? Like, what was the point, like, what work was that doing for us, and could that actually apply in that situation? And so it just--anyone who's interested, I would say, there's a great explainer in some ways, like, a small literature review of, What is the theory behind and the reason why we have discrimination law? That really allows you to come away and say, Okay, this is the work that discrimination law is doing for us. And there are places that this might actually apply aside--is that a fair point?--aside from protected classes.

Sandra Wachter  5:56
Yes, absolutely. I think you have actually just summarized it better than I did in the paper. (both laugh) That's absolutely correct, yes. I did try to find out what makes a group worthy of protection and came to the conclusion that those either nonsensical groups or non-protected groups just don't really fit into that concept. And I started thinking about, What's the underlying purpose? What does the law actually want for us? What would society look like if the law got its way? And so at the very, very basic level, right, the idea is that the law wants you to be independent and self-sustaining. The law wants you to be your own person, to steer your path in life and make your luck, basically. 

Kirsten Martin  6:40
Mm hmm.

Sandra Wachter  6:40
And that includes various things. That means you should have the ability to get an education. That means you should have the ability to train and learn for a profession, get a job at some point. It means that you're entitled to health care. It means that you should have enough food and shelter. And so if you think about those sectors, those are really needed to be an independent, self-sustaining person. And so the law has decided that the protected attributes are protected because we humans have used them as reasons to pull people back from getting education, from getting health care, from realizing their life goals, right? And it just so happened that those were the criteria that we decided upon, why you're holding people back, because we have biased beliefs about gender and ethnicity. Well, algorithms [are] also holding us back, but just using different criteria that we would never think of. We would never think of holding somebody back because of their browser usage. But the effect is the same, right? I'm being held back based on an attribute that I actually really don't have any play in acquiring; it is something that is assigned to me without me being aware of it.

Kirsten Martin  7:52
Right. And this is where I think it has, like, a general application. So it has--it's definitely for the law, and kinda saying like, We need to start thinking about this a little bit differently. And I think we sometimes talk, like in business or just in general, about, Well, is this discriminatory? Is this a discriminatory behavior? And it immediately goes to protected classes: Well, it's not discrimination if it's not protected classes. As if they have to go together. And what you do is say, Hold on, like, what was the end goal? What was the law envisioning for us when it was written? What world did it want to create? And is there another way that we're actually undermining that world in some way that we need to actually shore this up in some way? And I like, this is a good quote is when you say--this is not my words, these are yours: "All [of] these views have one thing in common: discriminatory behaviors carry with them an assumption of moral superiority." This means that discriminatory behavior "demeans an individual, considers them of lower moral value, promotes negative stereotypes and prejudice, treats them with disrespect, and has a negative impact on their deliberate freedom, autonomy, dignity, or life choices or benefits." And so, this is the idea of why discriminatory behavior is wrong; you know, so this is, like, the wrongness of discriminatory behavior. And it doesn't ever mention protected classes in that. I mean, there's no, you know, only according to religion, national origin, ethnicity, right? It doesn't say anything about that. It just has a general vision of, like, behavior that would be discriminatory, and then--and why people deserve protection based on, you're gonna say is arbitrary groupings that we might have.

So I would say, in general, that was a great summary of kind of why we have discrimination law, and then why we intended to do it for protected classes. And I thought maybe you could take a moment to talk about kind of your step from discrimination--this is what it does, this is why we avoid it, this is the vision for the future--and then why it should apply to these groupings. Or just, not discrimination law, they don't have to be protected classes because we couldn't enumerate them all. But why we need to start thinking about them as needing protection.

Sandra Wachter  10:02
Yes. So I actually borrowed a little bit from the literature in terms of terminology. So I chose the word "artificial immutability." And so that's borrowed from one idea that certain protected attributes are protected because you had no hand in acquiring them, and you wouldn't be able to change them. That includes, for example, age. That includes ethnicity. And so the law always thinks you should only be based on actions that you do rather than things that you have no control over. And so I borrowed that because, you know, we would usually think about immutable characteristics as something that is--has somewhat of a quote-unquote "natural source," maybe, right, that has been given to you. And so I thought about, it's artificially created immutability.

Kirsten Martin  10:55
Right.

Sandra Wachter  10:56
So it's not in the sense that I was maybe born with it, but it was assigned to me. And there are various ways of making it de facto immutable. So for example, if I don't know what criteria are being used to make hiring decisions, then they are immutable, de facto immutable, because I can't actually control any of those things, right? If you're using facial recognition software, right, to decide if I should get a loan, that doesn't mean I can move my retina differently, even though that's a decision criteria, right? Or the sweat that I have on my face. And so it means that the criteria that are being used are assigned to me, but they're immutable to me; I cannot actually change it. And traditionally, that's a problem. Because you know, if you say you have to have good grades to get into university, that means, you know, you're gonna study, you're gonna prepare well, and then you're gonna have good grades, and then you go into a good college or get a good job, right? There's control over that criteria. With moving your retina, that's never going to be possible, I'm never going to be able to control my heartbeat in the same way. And so algorithms just create a new type of immutability that has the same effect, as in I cannot control it, but it's just artificially assigned to me rather than given by birth.

Kirsten Martin  12:16
And that was, like, a kind of "Aha" in the paper. I mean, there was an "Aha" around discrimination. But I liked the idea of, we all know that protected classes, the reason why we don't like people to be discriminated against for protected classes, it might be historical reasons, but it's also because you can't change it. It's immutable. It's the idea that it's an attribute of you, and it shouldn't be, and we should be choosing you based on something else that you had some control over. But what your point was, and you do a great job of, like, identifying also why something should be, is considered immutable. Like, if we don't know about it, we can't change it, if something is kind of sticky in some way. And why would that be any different in the law or how we think about things that we shouldn't be discriminated against or treated differently based on something that we haven't been given a chance to change if we could?

Sandra Wachter  13:04
Yes, exactly. 

Kirsten Martin  13:05
And so could you say a little bit more about what makes something immutable?

Sandra Wachter  13:09
Yes. So I came up with a couple of criteria that make something, in my opinion, immutable. So it could either have to do with opacity, as in, I don't really know what the decision criteria are therefore I have no control over. It might be that the decision criteria are too vague. So I could tell you, you know, Your friends on Facebook have an impact on whether you get a loan; that doesn't really give me the ability to know who's a good friend. So again, I have really no control. How can I put my best foot forward if I don't know what's a good Facebook friend? Stability, right? If the criteria are constantly changing, I have no control over that process. I know good grades get me into university, but what if that changes, and at some point, it's my dog, and the year after it's my browser, and the year after it's my retina. So I can't actually prepare and have control over the path in my life if that's constantly changing. Involuntariness is another one. Again, face recognition software that measures how your retina moves, that I can't control the sweat on your face, your heartbeat. And the last one is if there's no social concept for the words. So that comes back to those two groups, like, the dog owners where there's a word for it, but it's not protected, and then those human un-understandable groups where there is no social concept. So if I don't even have a human understanding of what that group actually means, how can I put a good loan application--

Kirsten Martin  14:32
Yeah, right, yeah.

Sandra Wachter  14:33
--[based on] electronic signals. And so those are very different types of immutability than we would usually think when we think about age, for example. But that's why I'm saying they're artificially created because they're in effect the same, they're de facto immutable, because I have no control over them.

Kirsten Martin  14:48
And it also goes towards, like, if you don't, if you can't--I don't want to use the word "explain"--the idea of not having a word for "why." So if we either, if we don't know why you were denied a job, it's as fuzzy as anything, and so how are we--there's some immutable attribute of you that we have now made a decision based on. But we can't explain it, so how would you, by definition, ever be able to change it in the future or get better at it if we're not able to explain it to you? And I think that the other thing that's so interesting is, these are constructed, right? Like you're saying, they're artificially created, they're constructed by the organization that either designs, develops, or deploys the AI system. And so in some ways, they're creating their own problems. I mean, they create their own problems with regular old discrimination, too. But you can almost see each of those being like, How could we possibly try to make better decisions not on immutable attributes of people? How could we not create an immutable attribute in our design of AI? So in one way, it speaks to regulators to say whether or not--should this be something that we're looking at for regulation if people are making, organizations are making decisions based on immutable attributes? Because we've said in the past that we don't like that. But then you could ask organizations, Why are you creating an immutable attribute? You know what I mean? And this is how you're making that immutable attribute. Is that fair? Like it could have design implications and regulatory implications?

Sandra Wachter  16:13
Yes, I think it could have both. And I think it will be in the interest of both sides to actually dig deeper. Because I think, you know, figuring out why an immutable characteristic is relevant to a decision at hand is just normatively something that is valuable, but also just something that could be very interesting for a company to know. You know, is dog ownership really a good predictor for repaying a loan or not? (Kirsten laughs) It would be a good thing to know more about that, right? And unfortunately, very litttle is done at the moment to find out what the causal relationship between the data points is. Because very often, it's just good enough to rely on correlation rather than causation. This is not to say that immutable characteristics are always problematic.

Kirsten Martin  16:55
Right, right.

Sandra Wachter  16:56
I would just apply a similar idea to how we deal with immutable characteristics in traditional settings. So for example, on the face, prima facie, yes, immutable characteristics is always going to be a problem because you cannot control it. But there are exemptions if you can justify it. For example, age is an immutable characteristic. But we have laws against child labor that are based on age. And that's a good thing, right?

Kirsten Martin  17:22
Right, right.

Sandra Wachter  17:23
Other characteristics, we have, for example, that you have to have perfect eyesight when you're a pilot--immutable characteristic, but there is a reason that this is acceptable, for example, right? We have schools for gifted children on a particular IQ that you can't change, right? And so there are situations in society where it is acceptable to use immutable characteristics, but you need to explain why it is acceptable to use a characteristic that you cannot change. And when you can do that, then I'm happy for you to use an artificial immutable characteristic if you can tell me why it's acceptable. I cannot change it. (laughs)

Kirsten Martin  18:03
Right. Yeah, no, I think it's, I think it's brilliant, really. I mean, I just think it--it's a great example of trying to say, you know, Why do we care about these protected classes or this one category of things? And even though it looks differently, it looks different right now--it seems like, Oh, it's not a protected class--well, is it doing the same work as a protected class? And should our same thinking about the law around protected classes apply to the way we think about artificial immutable traits? And so that's what I really liked about it, is that the decomposing, you know, what we're doing with discrimination law, and then kind of reapplying it to another area to say, Look, this is also similar to these protected classes in certain ways. And then by doing it this way, I have to say, just by--especially enumerating the ways it's immutable--it's super helpful just the way that you said it because it would allow someone to say, How am I contributing to this being immutable? And, Can I justify any one of these types of immutability that I'm making an attribute? And so I think that it's really in that way prescriptive, as well--not only by saying, you know, This is what we should be doing, but saying, If you don't want immutable attributes, don't make them. You know what I mean? Or justify it, you know? Or justify why, and maybe there's a good business case. And then the last bit is, of course, if you can't articulate why you're making a decision, you know, it falls into the fifth group, you know, which is, it's immutable. You know, if all intents and purposes, and so that's not where you want to be. And so it gives organizations an incentive to be able to articulate why they're making a decision, which I always like. (laughs) Like so that's, that's a good thing.

And I'll end with, you have a great quote, I'll just read it from, it's towards the end of the paper, and you say: "Algorithms, as opposed to humans, do not make decisions based on prejudices or the idea of inferior worth, but in the same way they prevent people from accessing goods and services. They do this by creating groups that effectively act as immutable characteristics that individuals are unable to change and have no control over. As a result, individuals lose the ability to exercise their rights and freedoms and gain access to goods and services. Therefore, the harm is the same as that originally imagined in an anti-discrimination law, only the mode and process of bringing about [that harm] are different." And I just thought that's a great encapsulation of the article and kind of why it's important because a lot of times these algorithms are actually being used on rights and freedoms and taking away access to goods and services and important decisions like employment, credit, getting into schools. So I loved it.

Sandra Wachter  20:40
Thank you so much. I'm very glad you liked it.

Kirsten Martin  20:43
Oh, yeah, yeah, no, and it's, the thing is, I think, you know, it's in Tulane Law Review, so it's obviously in a law review, and it's based on anti-discrimination law and discrimination law. But I really think that the implications are much broader, the way that people need to start thinking about the groupings that they're creating. And that's where I think it has, like, kind of a general application to tech ethics. And speaking of which, are there, we always like to end with a wrap-up of, is there anyone kind of in the area of tech ethics broadly--I mean, a lot of us read across disciplines--so is there anyone in the area of tech ethics that you think, Oh, I can't wait for them to write again or see what they're presenting, and anyone we should keep an eye on?

Sandra Wachter  21:21
Yes, I would highly recommend to keep an eye on Dr. Amy Orben, who is now at Cambridge. She's a psychologist, and she's interested in the questions of online harm--online harms broadly, but particularly on social media on young people and mental health issues. And so she is doing extremely interesting, insightful, peer-reviewed empirical work to actually figure out what the harm is that people experience when they are experiencing life online.

Kirsten Martin  21:54
Oh, that's great. That's great. And super important because I don't think they're--they're obviously not capturing, "they" being platforms, aren't capturing all the costs that they create when they're making money online. So it's helpful to have someone enumerate exactly what those harms might be. That's a great recommendation. Well, gosh, Sandra, thank you so much. I really appreciate you coming on and talking just briefly about your paper. Thanks so much for taking the time.

Sandra Wachter  22:14
Yeah, thank you so much for the invitation. Such a pleasure to be here. Anytime again.

Kirsten Martin  22:18
Oh, I will take you up on it.

(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

12. Algorithmic Fairness is More Than a Math Problem (October 19, 2022)

Guest: Ben Green (University of Michigan)

Transcript

Kirsten Martin  0:03
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we talk about an important idea, paper, article, discovery in tech ethics, and today, I'm so happy to be joined by Ben Green. Ben is an assistant professor in the Ford School at the University of Michigan and a postdoctoral scholar in the University of Michigan Society of Fellows. He has a Ph.D. in applied mathematics with a secondary field in science, technology, and society from Harvard. He studies social and political impacts of government algorithms with a focus on algorithmic fairness, smart cities, and the criminal justice system. So today, we're going to take a deeper dive into this article in Philosophy and Technology, which is forthcoming, called "Escaping the Impossibility of Fairness: From Formal to Substantive Algorithmic Fairness." And I'll just say, I think my favorite part is Figure 2--but we'll get to that in a little bit--because I think it really is a nice visual as to the role of formal fairness. But, well, I don't want to skip too far ahead, but it's great. And so I thought this is, in general, talking about this move from formal fairness or mathematical fairness to something more substantive, and so I thought maybe you could start with the problems that you identify rightly about the dominant approaches to fairness that rel[y] on mathematical modeling, and that gets us into this impossibility idea of fairness, where we see ideas of fairness as incommensurable, before kind of going to your solution.

Ben Green  1:26
Yeah. Well, thanks so much for inviting me on. And I think that's a great place to start. I'm excited to have this conversation about this paper. Yeah, you know, I think this paper really springs out of trying to figure out, where does the field of algorithmic fairness go? What is a positive agenda for this field? It [ha]s often been caught between sort of two camps. On the one hand, you have the computer scientists, who say, Fairness is a problem, we can formalize fairness, we can create a definition for fairness. And then you have critical scholars in philosophy and STS and other fields that say, Of course you can't do that, of course you can't, you know, characterize this very complex notion of fairness into a single mathematical metric, here are all of these problems that arise when you try to do that. And so there's this sort of deep tension between, what do we do, right? On the one hand, there's these attempts to formalize the problem; on the other hand, there are critical scholars saying that that approach doesn't actually get us that far.

And one of the major issues that has come up through this formalization approach is what's been called the impossibility of fairness. And this is a mathematical result that sort of shows the incompatibility of different mathematical notions of fairness. And in particular, there's sort of the two major notions of fairness that often are talked about, the first being an idea of calibration, which is that people who are predicted to have an outcome at similar rates should be treated similarly. And the other notion would be something along the lines of equal false-positive and false-negative rates for different groups, saying that, you know, if two groups are equally, if someone is not going to commit a certain outcome, then they should be treated similarly; there should not be more false positives or false negatives across different groups. And this debate came up most notably when it was really sparked by the ProPublica article about the COMPAS algorithm, their machine bias report that showed that this algorithm that was used in pre-trial and sentencing made false-positive predictions about Black defendants, saying that--falsely predicting that Black defendants were going to recidivate at a much higher rate than they did, than the algorithm did for white defendants. And so it sparked this question of, you know, how do you navigate between this tension of wanting to, you know, get rid of these different false-positive rates, but then that's in tension with, you know, treating people the same based on their prediction. So that's really this impossibility result that has, I think, stymied the field's progress.

Kirsten Martin  4:06
And I think it also has real implications. Like, I feel it when I'm talking to people in the business school or anyone that's kind of looking at more applied data science, and they use this as a kind of excuse as to, like, We're doing the best we can, and if I just kind of, I just have to choose one of these random mathematical definitions of fairness, and therefore that's what fairness is, you know? And so I see that from the titles of "the impossibility of fairness," people glom onto that and say, Well, it's impossible, you know that, right? It's impossible, that we can't do this. And so it's kind of had this downstream effect into applied work from the computer scientists noting that if we say that we want to have the same types of mistakes for the same type--that all people have error rates about the same versus good predictions about the same, that it's hard mathematically to get that to work. But that doesn't mean that we don't stop trying, or in your case to say, Maybe those aren't the right definitions of fairness, and that's not all there is.

Ben Green  5:01
Yeah, and that's a great point. And I think that was part of what motivated me to really dig into this impossibility result more fully was having a lot of conversations like what you're describing, where computer scientists or even policymakers are saying, Well, it's just a matter of trading one versus the other, we have to pick one, or we have to figure out the right balance, and that's really the best that we can do, this sort of shrug-your-shoulders, accept-your-fate approach. Which on one level is mathematically correct but I think felt to me like it was really limiting our ability to think creatively about how we could try to promote equality with algorithms.

Kirsten Martin  5:39
Right. Yeah. And you do a nice job of saying, you know, just that the problems with the formal, that one, we're ignoring these philosophical meanings, but we're also just naturally limited by the tools of the field, which I like your quote from Dewey, which I always misquote by saying, A problem defined is a problem half-solved, which is, like, the much shorter version of it. But the longer version, which you nicely quote, is, "The way in which a problem is conceived decides what specific suggestions are entertained and which are dismissed." And so we automatically by defining a problem as a mathematical problem that we need to solve kind of go to it with only those tools that we have, and then the answer we get, it's incommensurable, is the answer that we have, and we just stop looking, we kind of dismiss these other ideas. And your great point is, which [is] leading to myopic suggestions for how to promote fairness in practice, which I think we see the results of.

Ben Green  6:31
Yeah, and I think, you know, a lot of this is--I don't get too much into, you know, the Dewey of it all in this paper (Kirsten laughs), but I think there's more that I've been thinking about and hope to write on sort of bringing in Dewey's pragmatism into thinking about the role of algorithms and how we design algorithms, really shifting away from trying to focus on these abstract conceptions of ends in and of themselves. When we--you know, from computer scientists trying to define fairness, then the conversation centers around the definitions of fairness, these mathematical conceptions, and even our arguments, you know, sort of focus on that level to say, Are algorithms inherently biased? Can we make fair algorithms? And it's very much focused on the algorithm itself [inaudible] because of the way that the problem is conceived. And what I am pushing with this article and some of my other work is to say that we really shouldn't be worrying about that mathematical construct so much, except to say, How does this actually affect the real world? How does this work in practice? Rather than asking is this algorithm inherently fair or biased, we should be thinking more in terms of, to what extent do algorithms improve or hurt efforts to promote equality? Right? Sort of shifting from a debate of, This represents fairness versus, No it doesn't, to: How can we have algorithms that play more productive roles in promoting equality? And whatever we're doing on the formalization side is instrumental to that downstream goal.

Kirsten Martin  8:03
To that goal. Right, right, right, right. To kind of shift what the goal is versus a technical outcome is to say, What do we want to have in this decision space? Like, so what is, what is our goal for this type of decision? And then have that kind of, you know, filter downstream as to whatever, and it would almost become an engineering input of, like, the assumptions that you need to make.

Ben Green  8:22
Yeah, because there's a huge gap between the, you know, these sorts of notions that we put into algorithms and the real-world impacts. And that's part of what critiques of algorithmic fairness, including my own here, are pointing to is that, an algorithm can satisfy a notion of fairness that's pretty standard, and yet actually, by many accounts, be re-entrenching inequality. So if that's the case, clearly we can't just be relying on these definitions, we need to find other approaches.

Kirsten Martin  8:50
Right, I thought that was the--I thought that was the more powerful part of the piece was to say, not just that they're, not just that they're not helpful, but they could, by using them, be doing more harm. Like, by relying on them, if we're kind of generating disparities in social and material resources, you know, it's kind of reinforcing systematic problems that we have in society, then they're actually not--not that they're ever neutral, but they're not that they're not that helpful, they're actually harmful in some ways, like, by relying on them too much. So I thought, I didn't know if you could talk a little bit about the move, which you pull on from formal versus substantive equality, as to how you see that difference and how you kind of take that conversation in formal and substantive equality and then bring it over to the fairness in algorithms discussion.

Ben Green  9:35
Yeah, so what I'm looking at with--yeah, so sort of thinking about, Where are the problems in how algorithmic fairness is conceived? And it's really two intertwined issues, the first being the issue around really trying to make this a mathematical problem and define fairness mathematically, the second piece being the limiting scope of treating fairness really as just a matter of inputs and outputs within a specific decision-making process. And that really maps onto ideas of formal equality, which are striving to sort of, yeah, create--you know, sometimes this is quite similar to procedural equality--to create, you know, fair procedures, decision-making processes that say, you know, Two people who are alike in terms of their input attributes for this decision should be treated similarly within the bounds of this process, and everything upstream and downstream of that decision-making process is ignored. And I think that that sort of maps onto a lot of the issues that we can see within the formal approaches. And part of how I bring in substantive equality is by looking at how some philosophers and legal scholars have responded to similar types of tensions within efforts to promote equality that actually look a lot like the impossibility of fairness. Where within a more formal frame of analysis, it looks like we're caught between these two completely incompatible ways of trying to promote equality, but actually, if we take a zoomed-out approach, if we take a more substantive approach that looks at social hierarchies and downstream impacts and sort of the real-world complexity of the situation and thinks about reforms beyond just the scope of decision-making procedures, we can actually see that what looked like a fundamental tension is not so intractable after all. So I'm bringing in ideas from those scholars to say, Here are some ways to think about how we might approach reform in these settings where it looks like we're stuck with a deeply intractable zero-sum dilemma.

Kirsten Martin  11:42
So almost, like, as a visual--which I think this actually goes towards, like, other things with algorithms as to, like, efficiency claims and stuff like that--is that a lot of times, we're looking in a very, very small box around just, like, the data that we've received and then the model, and then maybe an outcome. And we're trying to--I'm using my hands, and no one can see me except for Ben--but with very, very small, and we're trying to optimize or figure out the fairness within that. So this is like the error rates being the same or the true positives being similar for similar groups or something along those lines. But when all of a sudden, that's all the outcome and how we treat people, how the datasets were actually created is actually embedded within this larger system that we need, if we kind of expand out to see what's going on--instead of just taking that as, like, a null, like a given--if we expand out to see what's going on, one, we're not gonna use mathematical models for that, but also the problems that we're trying to solve might be different. And the issues that we might be facing with the data might be different, as well. So our solutions and the problems are going to be different if we expand out. And that's similar to issues around equality, where we're trying to--I don't know, university admissions, where we're trying to, like, micromanage little tiny things about someone's application versus saying, Well, who's even taking the SAT? And which schools are we even--like, kind of broadening out further to say, What else is going on in society that we need to understand how, instead of looking only at SAT scores, looking, going more broadly, and trying to understand what else is going on in the world to create the imbalances that we're seeing in SATs or GPAs or whatever that might be. Is that a fair characterization of kind of broadening it out?

Ben Green  13:20
Yeah, no, I think that that captures it really well. And I think, with respect to algorithms, gets to this point around, you know, are we trying to define these sort of mathematical formalisms about algorithms as somehow a property of algorithm, which maybe makes sense in some internal world, or are we actually trying to figure out how we can use algorithms as tools for improving society? And if we're in the first world, then sure, we can define fairness, we can treat inputs and outputs and think about fairness as this mathematical construct. But if we're actually interested in real-world impact, we have to look at what's happening in practice, we have to take this broader scope, especially because just relying on these mathematical notions of fairness, it doesn't actually lead to deeper equality when implemented in the real world. So we really have to think in this deeper way, and I think yeah, there are a lot of lessons from ideas about equality and ideas about political reform and theories of change to draw on and say, Why do certain types of reforms, you know, lead in one direction and certain types of reforms lead in another direction? I think we can really pull on that, you know, especially when we're talking about algorithms like pre-trial risk assessments or predictive policing algorithms or welfare algorithms, where these are directly operating on these high-stakes decisions and on people's lives. But also for the developers a major motivation is that they are contributing to improving society. The goal isn't just to make a fancy mathematical tool for, you know, theoretical computer science. They're working on an applied problem with an implicit or explicit normative goal behind that when they're developing these systems. So I think, you know, it's important to carry that normative lens all the way through not just to a high-level motivation but also to how do we evaluate what these tools are doing and whether or not they're actually working.

Kirsten Martin  15:18
That's a great point. Because I think, like, the idea of, like, what it means to actually work in the real world is one of those things that we just don't pay enough attention to, and that we need to pay more attention to. What I always say is--which is extremely odd. Because in other places in an organization--like when we automate a manufacturing line--we have to take into consideration all these laws and norms and, like, actually what's going on within the context and make sure that it's not creating problems in the manufacturing line, if it's not hurting employees. I'm not saying that businesses do this all the time, but they should. (laughs) And so I think that it's just one of those things that we looked at whether it optimized or was efficient in such a narrow, small little box and not really understand in what way are their downstream effects, or even, like, what are the harms of additional true positives to someone who's not committing or people think is committing fraud in welfare? And especially since you look in the government context, it's always someone that's taking away someone's rights, right? So we have someone, like, so it's like a higher impact on someone, and it's not really clear how we're measuring whether or not that AI is actually successful, or what success even means in that case. I don't know, do you know Apryl Williams? She does stuff on reparative--

Ben Green  16:28
Mm hmm.

Kirsten Martin  16:28
Yeah, so she has a workshop coming up this weekend on reparative algorithms with that same kind of idea of looking at some sort of larger goal and not just trying to optimize on something kind of very narrow and small, but looking at something that's more of a social justice repairing what's out there versus just trying to make whatever is going on faster, which is kind of where we are right now. So I don't know, could you speak a little bit about your alternative, which is, you know, focusing on substantive? I recommend everyone look at Figure 2, which shows the role of--we don't have to go through Figure 2, but what I liked about Figure 2 is that I think it's a great, like, actual visual reminder of all the work that needs to be done around substantive ideas of fairness. Like, there's three times the boxes in Figure 2 around substantive fairness, and formal is really like an offshoot of a discussion. And so it really shows everything that we're missing by ignoring questions about substantive fairness and our kind of preoccupation with this one lane of formal.

Ben Green  17:31
So yeah, the approach that I describe in the paper is called substantive algorithmic fairness. And I characterize it into two different types of responses. There's the relational response and the structural response. And we can think about this in terms of a more substantive diagnosis of the problem of even the impossibility of fairness, which comes up, you know, most first in Figure 1 as a sort of prelude to the second figure, which is more of a how-might-you-implement-this-method-in-practice. But when we're looking at something like the impossibility of fairness, we can see sort of upstream and downstream issues that create this tension. Upstream of a decision-making process, we have what I call a relational harm. We have social hierarchies that say, you know, there are deep inequalities in terms of educational outcomes or how likely different communities are to be arrested or to commit crimes in the future. And, you know, it's important to note that these are not just issues of the data being biased in the sense of misrepresentation, but also that the data is often accurately picking up on real-world enduring inequalities. And I think that's an important point of this relational harm, where often we talk about the data's bias just to mean, well, the data's not--you know, is misrepresenting the reality. But often the reality is deeply unequal, and the data actually accurately captures that.

Kirsten Martin  19:01
Right. 

Ben Green  19:01
So we can't just have the issue be a concern about misrepresentation. So that's really this issue of the relational harm. And then downstream of a decision-making process, we have what I call the structural harm of how--what are the implications of the decision that the algorithm is informing? And typically, as you already hinted at, often these decisions are punishing individuals or restricting benefits from individuals who are judged negatively. So essentially here, downstream we're re-entrenching the inequality that exists by saying, Oh, you didn't, you know, you have lower SAT score likely because you have all, you know, because your social group was oppressed, that also now means that you, rather than getting support, you are now not able to get into college, you're not able to get this job or something like that. So we have the downstream harm that is further magnifying the upstream inequality. You know, sort of if you think about this larger chain of a process, when we're just looking at the decision-making process itself, which is the case in formal equality or formal algorithmic fairness, we're not able to account for either of these concerns. And the goal of substantive algorithmic fairness is to bring both of these types of concerns into view.

Kirsten Martin  20:21
I have to say, I was really impressed by how early you got to your answer, and how much time you spent with it. Because the vast majority of papers get to, like, here's everything that's wrong with what you all are doing--and then implications and discussion. You know what I mean? And so I want to applaud you for sticking with your solution as long and as detailed as you do. Because every time you do that, you are, like, laying yourself out to be then later critiqued. (both laugh) Even though we're academics, and we invite that on the regular--like, that's just kind of our job--it's still, it is not an easy thing. And I even looked at the number of, like, the pages, and I was like, Oh, you got there, like, by page 15 (Ben laughs), I'm aready talking about your solution. And I was like, That's impressive. Because I always tell people, Get to the halfway point, talk about what you're going to do differently. And you did it, and you spend a lot of time with it, and very detailed in applying it. So I mean, it's one thing to propose, like, here's a way we can think about this--we should be thinking about this differently, even stronger--we should be thinking about this differently. And by the way, and this is how you would do it. Like, this is, I'm going to walk you through what that would mean in a very specific case and kind of what the repercussions of doing that. So taking the COMPAS idea, the sentencing algorithm, and then saying like, What would that mean in this situation? What would it mean in other situations? And I think--I really recommend that people read it because even if you're not interested in it, just to see, like, how you can actually put forward an idea and then defend it, you know, for a lengthy amount of time. (both laugh) I don't know what the reviewer said, but--

Ben Green  21:53
Yeah, it's funny--I appreciate that you picked up on that and enjoyed that. Because I would say in successive drafts of this paper, and it went through many drafts, the positive agenda expanded and got larger and larger and started earlier and earlier, and the critique element got shorter and shorter. Because yeah, increasingly, it felt like--if the paper is more, there's a lot of papers that, yeah, helpfully point out things wrong and then, right, have a very brief section of, Here are some high-level steps forward. I've written papers like that, too.

Kirsten Martin  22:27
Right, same.

Ben Green  22:28
They don't give a clear, positive agenda. And so yeah, my goal was much more to provide that, which then means, oh, you have to keep adding more and more layers to the positive agenda (Kirsten laughs) so that it feels fully fleshed out. It's always easier to do critique--

Kirsten Martin  22:43
Oh yeah. (laughs)

Ben Green  22:43
Than I think to make proposals. But yeah, I'm hoping that this can be, you know--hopefully now as opposed to earlier drafts, it has enough meat on its bones, so to speak, that it's both, yeah, more to think about how it could be implemented in practice. Even Figure 2 is something I added quite late in the process to really try to distill it down to say, What are the actual steps that someone would follow here? As opposed to just sort of a higher level conceptual frame? 

Kirsten Martin  23:10
Yeah, I mean, it's not often you see figures in a philosophy paper. So I--kudos to that one, as well. Because I used to joke, they wouldn't even know how to process it normally. (both laugh) They probably were like, What? What is this JPEG? I don't even understand what this is.

Ben Green  23:23
We'll see what happens when I go through the proofs. (laughs) I'm sure they'll have no problems.

Kirsten Martin  23:28
No, I'm sure, I'm sure. You know the other, what it's a good reminder of, though, is that in some ways--there's this great philosophy of science, Richard Rudner, who talks about the value-ladenness of science, and people have applied it to technology. And the idea that he says is that, Look, you're making these value-laden decisions throughout; by ignoring them--so by ignoring your more substantive definitions of fairness, of algorithmic fairness--it doesn't mean that you're not actually taking a stand on them. You're just doing it, in Rudner's words, unmanageably. So you're not actually being thoughtful about this because you're inheriting them no matter what, and you're contributing to them, or you're interacting with them no matter what. It's a matter if you're kind of managing that interaction in the way that you deal with it on the front end, or as your point, are you contributing to it later on downstream as well? And the visual, it really helps with that. So I will just say that I think this kind of summarizes your paper: "Substantive algorithmic fairness presents a new direction for algorithmic fairness: away from formal mathematical models of 'fair' decision-making and toward substantive evaluations of how algorithms can and cannot promote justice." Which happens early in the paper, but I thought it was a great summary of kind of what the point is and also really focuses on the main point of the paper, which is to offer this new vision or different vision of how we think about algorithmic fairness, and also the role of mathematical fairness as a piece of the overall understanding of fairness.

So we usually end with anyone that you know in the field--you know, broadly construed--that we should be paying attention to. And this can be from any discipline whatsoever. I'm very agnostic to discipline. So if there's anyone that you really, when you think, Oh, I can't wait to see what they write about next. Is there anyone we should be paying attention to?

Ben Green  25:18
Absolutely. I think there's, you know, a number of people that I think are really doing interesting work at the intersections of AI, ethics, race, and the real-world social impact. So Rashida Richardson is one who's a legal scholar at Northeastern, and she writes really great articles about sort of the role of race in how AI is used in police departments and government bodies generally. Anna Lauren Hoffmann is at University of Washington writing some of the critical articles on fairness that I mention, refer to, and cite in the article, but really pointing to the gap between these sort of abstract ideal conceptions and some of the real-world issues of inequality. Lily Hu is a friend who is a philosopher at Yale writing about issues of race and social science and AI in terms of how we construct race and issues of causality that I always really enjoy reading. And then Rodrigo Ochigame, who is a professor at the University of Leiden writing about, similarly to many of the others, about the limits of AI ethics and AI fairness and trying to bring in more of a historical approach, an STS approach, an anthropological approach, to understanding, where do these conceptions come from? What sorts of impacts do they have when deployed by political actors and engineers into practice?

Kirsten Martin  26:51
Oh, that's great. Well that's, you covered your bases. I saw when you cited Anna Lauren Hoffman, I love that paper of hers from a number of years ago where she really just takes it to task on the focus on discrimination, and it was just so pointed and on point that I was--it's so refreshing to have someone just kind of tackle that straight on. But thank you very much, that's really helpful. Well, thanks so much for being here. I really appreciate you taking the time out of your day. I know it's super busy, so thank you very much.

Ben Green  27:17
Yeah. Thank you so much. This was a great conversation. Thanks for having me on.

Kirsten Martin  27:23
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more visit techethics.nd.edu.

11. Provoking Alternative Visions of Technology (October 5, 2022)

Guest: Daniel Susser (Penn State University)

Transcript

Kirsten Martin  0:03
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we discuss an important idea, paper, article, or discovery in tech ethics, and today, I'm so happy to be joined by Daniel Susser. Daniel is an assistant professor in the College of Information Sciences and Technology [at Penn State]. A philosopher by training, he works at the intersection of technology, ethics, and policy. His research aims to highlight normative issues in the design, development, and use of digital technologies, and he's currently focused on questions about privacy, online influence, and automated decisionmaking. Today, we're going to take a deeper dive into your article in Surveillance & Society that's just recent--super short, but I really liked it--on "Data and the Good?" And I-- well, I like the fact that, we can get into the meat of it, but pretty much what I took you to be saying is that privacy law scholars and surveillance studies scholars, which don't always operate, other people might not realize but they kind of build on different areas and their goals are different. Even though we might think of them as similar from the outside, they actually are very distinct. But they both talk about the broader implications of surveillance for individuals and society but don't focus enough on what should be happening. So focusing on what shouldn't be happening, but not so much on what should be happening, and I didn't know if you could speak a little bit about that, both where they're focused now and where you think they should be focused.

Daniel Susser  1:26
Yeah, thanks. So I mean, one, thanks for having me, Kirsten. I'm really excited to be here, and I'm glad you enjoyed the piece. You know, just sort of as context, the short essay was part of a, like a dialogue series in Surveillance & Society. And so the idea was for a few pieces to sort of speak together on some kind of theme--really short, sort of like provocation pieces. So that's the spirit in which I wrote it. And the prompt was the intersection of privacy law scholarship and surveillance studies scholarship. So that's where that focus came from. And you know, Scott Skinner-Thompson at the University of Colorado invited me to participate and so had me just sort of thinking about where these two different fields--which, as you say, sort of speak to similar issues often but generally usually coming from very different theoretical backgrounds and disciplinary perspectives--sort of what happens at the intersection of these two spaces. And like you said, I was just really struck as I was thinking about it by how much really amazing, trenchant, I think super important work has been done and is being done to point out the kinds of harms that data collection and data-driven technologies can produce, and to level, I think, really meaningful and, like, devastating normative critiques against these kinds of harms. But much less--I mean, there is some work, which I try to point to in the paper and I'm sure we'll talk about, but just much less work that really tries to put forward an alternative vision for what these technologies can do for us. And I have to say, I mean, this is--I do not mean this piece as an attempt to cast stones. I think my own work reflects this same kind of bias, where I have been mostly focused on critique and less on putting forward a kind of substantive vision. And so in part, this is kind of like a mea culpa. (Kirsten laughs) I want to be, in my own work, be thinking about sort of how we could do this more.

Kirsten Martin  3:21
I think for a lot of us, and I would say the same about my own work--you know, we both have written about manipulation, we both write about privacy. I think a lot of it is, you see something wrong going on, and you're like, Oh, I want to enumerate why this is wrong and help out and explain, these are the things that are wrong. And then you get to the end of the paper, and then that's just it. (laughs) Or maybe the target is a different type of journal outlet, where you're supposed to be designing perhaps, you know, regulation or policy that might be different. And I thought was also interesting is you do a great job of identifying, like, why the fields steer clear. You know, so why--like, this is a commonality of surveillance studies and privacy law that we steer clear of enumerating what a good technology would look like given the theories that I just used to explain why this went wrong. And some of it's like, it's just not the job of law to do that. These law professors were not brought up to identify, and the law scholarship is not there to identify, procedural norms that should occur within the law and not a design decision, which is really left up to me in a business school or you in an information school. Right?

Daniel Susser  4:30
Yeah, a hundred percent. I mean, I think both of the things that you just pointed to were really, like, important as I was thinking through this piece. On the one hand, right, like I think the motivation for really focusing on diagnosing harms is completely understandable and I think justified in a lot of cases because there are so many of them, and they are so incredibly worrying that, you know, it's perfectly understandable why so many of us, I think, really home in on that and think like, Let's point out what these potential harms are and think about how we can mitigate them. And then yeah, the sort of middle part of the piece, where it's like a bit speculative.

Kirsten Martin  5:06
Yeah, right right.

Daniel Susser  5:07
But I identify what I imagine are some sources for this hesitation to sort of put forward a substantive, positive vision. Which yeah, in the context of legal scholarship, I think the sort of liberal sort of foundations of American legal scholarship and most Western legal scholarship does exactly as you said, which is sort of assume that the law's role is really to provide a set of rules within which otherwise autonomous people can kind of live out their life in whichever way they see fit. And there's a kind of implicit injunction, sometimes an explicit injunction, against law sort of intervening in our lives in a more substantive way, which again, I don't, I don't reject that. I think that's, like, a perfectly--there's a lot of wisdom to that perspective. And likewise, I think, you know, some people who work not in a liberal tradition but in a more radical tradition are really worried about, you know, if we put forward new substantive conceptions of how these technologies should be operating, are we going to maybe, you know, inadvertently reinscribe racist or colonial or imperialist or other kinds of oppressive values into these technologies? And so there's a real worry about, I think, normative prescription at all coming from some of those traditions, especially in the surveillance studies literature. And I think those worries are perfectly understandable, too. But, you know, for reasons I sort of lay out in the second part of the paper, I think we should be emboldened, and we should sort of, even in the face of those kinds of real concerns, work to put forward a substantive vision. Because the alternative, as I suggest in the paper, is that we are just beholden to the technology industry's vision for the future. And so I think if we don't put forward our own alternative conceptions, the best we can hope for is a kind of harm reduction. And I think that we can do better than that.

Kirsten Martin  7:00
Right, and I like what you used, [Langdon] Winner, to identify that "particular technologies," I'll just quote, "generate specific 'forms of life.' By 'provid[ing] structure for human activities,'" they "'reshape that activity and its meaning.'"And "'As they become woven into the texture of everyday existence, the devices, techniques, and systems we adopt shed their tool-like qualities'" and "'become a part of our very humanity.'" And so I--which is beautifully written, I have to say--but it also goes to, like, how important it is to be speaking of an alternative vision of, like, where you think the future should be or what world are you trying to create with the design of this technology? That that's--I like to say, That's where the magic happens is in design and development. And in some ways, kind of harm reduction or mitigation at the end is unsatisfying because it is--it's important, and we should still do it, but the main idea is, like, How would we have done this differently? Because that's actually where the important decisions, building on Winner, that's where the important decisions are being made.

Daniel Susser  8:00
Yeah. Isn't that quote so good?

Kirsten Martin  8:02
Yes. 

Daniel Susser  8:02
He's such a good, good writer. Yeah, I think there is a, I mean, I totally agree with everything you just said. I mean, I think there's a vein of scholarship in political theory and political philosophy that has of late really sort of focused on the sort of diagnosis of harms and mitigation of harms as sort of a more tractable set of problems that we can deal with, and sort of move away from what has traditionally been called sort of ideal theory in politics that tries to put forward these kinds of more substantive, positive visions. And I think in certain domains, that makes a lot of sense. Because what we--like, the urgent problem is just to, you know, create less-racist institutions or to create a politics that is less polarizing or something like that.

But the thing I take away from Langdon Winner's work, from the work of Phil Agre, and other people that I cite in the paper--sort of really classical texts in science and technology studies, philosophy of technology--is that technologies are, you know, we use them in this way that feels like they are just sort of enhancing our ability to realize our own ends. But what the scholarship really teaches us--and we have, you know, decades now of work that really tries to make this argument, and I think makes it really powerfully--is that that's never all that's going on. Technologies are always world-building in this way, and I think the language, you know, Langdon Winner takes this language from Wittgenstein, of forms of life, that technologies provide a kind of form of life. And what he means is that they structure our activity and the meaning that we imbue that activity with in various different ways that really impacts how we live our lives and experience our lives individually, how we organize socially and collectively. It impacts the kinds of political institutions that we're able to create and maintain. And so because technologies are always already doing that kind of positive, like, constructive work in the background, I think it just raises the question for us: If technologies are world-building, like, what are the worlds that we want our technologies to build? And I think that for most of us, while it's true that we would like them to be less-unjust worlds than the ones that we're currently experiencing, I think we want more than that. And, you know, I think we want a more democratic world, I think we want a more egalitarian world. And I think that that requires thinking not just about sort of removing opposition to democratic movements or removing obstacles to egalitarian social relations but actually, like, positively building technologies in ways that advance those goals.

Kirsten Martin  10:50
That's great. Yeah, I think your paper is a great call to not only identify what's going on and also why it's wrong--so what exactly is going on when we get this instinct that something seems off--and then you identify some scholars, like you said a minute ago, like, you identify some scholars that are kind of moving in this direction to say, like, What's this alternative vision of what we actually want the technologies to support? Versus saying, Stop doing this, modify this.

Daniel Susser  11:20
Yeah, absolutely. There--you know, I mention a number of folks in the paper who I think are moving in this direction. I think, you know, there's amazing work by Ruha Benjamin. She has an amazing book called Race After Technology, where she advances this vision of tech abolition, which sounds like a purely negative project, but in Benjamin's view, you know, she says something to the effect of, Abolition is never just about sort of destroying the oppressive system. It's also about envisioning a new one. And I think that that's really crucial. I'm really excited about her new, forthcoming book--I think it's coming out, like, next month or the month after--called Viral Justice, where it looks like she's going to be engaging in that kind of work, as well. I'm not sure if this is the point where you want me to sort of name other folks that I mention.

Kirsten Martin  12:05
Oh, yeah, we should. That's a, that's a good time, that's a good place to do it. Yeah. Especially because you name some of them in your paper.

Daniel Susser  12:11
Yeah, for sure. So, you know, another person I mention in the paper, Salomé Viljoen, who is a legal scholar at the University of Michigan, I think is doing really amazing work that really reflects, I think, exactly this kind of perspective that we've been talking about. It is on the one hand, you know, it produces a critique of the existing order, but then also really tries to push us in the direction of imagining what a new sort of legal order would look like. Salomé's focus is on thinking about data not just as sort of about individual people, but rather as a medium for producing social relationships. And in particular, she wants us to think about what it would mean for data to produce more democratic, egalitarian social relationships. And she offers some really useful concepts for thinking that through.

Another book that I just have to give a sort of shout-out to, James Muldoon has a new book called Platform Socialism, which does a really incredible job of thinking about how we can sort of, like, leverage the kinds of tools and techniques that we have in order to produce a radically different political economic order. And you know, not everyone is going to be on board with platform socialism. (Kirsten laughs) That's sort of a contentious set of political views. 

Kirsten Martin  13:27
Right right right.

Daniel Susser  13:28
But even for those people who maybe don't want to take on board all of that politics, as an example of work that can provide just a completely different set of conceptual tools for helping us think through these kinds of questions to sort of build what I call in the paper, I take the language from Sheila Jasanoff, of sociotechnical imaginaries. And I say that, you know, we need folks to help us develop new sociotechnical imaginaries so that we're not kind of beholden to those of Silicon Valley. And so James Muldoon's book I think really offers some amazing conceptual tools that helps us think in different ways from the ways that we're used to. I should also just say because you brought him up, I mean, I just cannot recommend enough to people who might be listening to this some of the classics--

Kirsten Martin  14:15
I know.

Daniel Susser  14:15
--in philosophy of technology and science and technology studies. So you know, Langdon Winners' book, The Whale and the Reactor, I teach it every semester, and every time I go through it, I, like, find new stuff that makes me think really hard even though it was written, you know, 40 years ago. Phil Agres' work. And there's also a really great, a really great blog, like, Substack, if people are interested in such things by Michael Sacasas called the Convivial Society, where he takes a lot of--he's a really wonderful reader of these kinds of classic philosophy of technology writers, and he does really amazing work sort of translating their insights for contemporary problems. So, highly recommend that.

Kirsten Martin  14:53
That's great. That's a great list of people to look for, both very current and then, like, the classics. I know I'm going up--Apryl Williams' work on reparative algorithms, which is the idea of not just attempting to get to whatever status quo is and not doing further harm, but actually trying to empower people at the margins through the design of your AI. And so there is work--now she's not in surveillance studies or PLSC, or privacy law scholars. Which maybe is to the point that there's, you know, interesting reparative work that's going on where you see a vision of the future, and, like, how can we actually design our technologies to enact that vision of the future? Because we're, I think the power of Winner and others like Winner, Langdon Winner, is that, what he would say probably is that we're doing it anyway. And so whether you're thoughtful about it or not, and why don't we just be more thoughtful and manage it in a better way, which is kind of your call to arms for this entire endeavor, which I really liked. I'll just say this is your words, not Langdon Winner's I think, from your paper, but-- "I'm suggesting that we contemplate new goals. In addition to diagnosing and mitigating the risks of data-driven technologies, privacy and surveillance scholars ought to contest the technology industry's vision of the technological future we are striving to achieve by offering competing visions of our own." And I really think that that was a great summary of, like, the call of saying kind of, you know, at the end, kind of push ourselves to come up with what's an alternative vision or what would be the steps of identifying an alternative vision. And so I really--I really thank you for writing it. It was a great piece.

Daniel Susser  16:24
Oh, thanks so much. I really appreciate you taking the time to sort of, like, think through it with me and to highlight these pieces. I hope it is--it was meant as a provocation, so I hope it provokes people. I'm excited to see what it yields.

Kirsten Martin  16:36
Right. Yeah, well, we might as well amplify the provocation. (Daniel laughs) I always love a good provocation. So thank you very much for coming.

Daniel Susser  16:41
Oh, thanks so much, Kirsten. 

Kirsten Martin  16:44
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more visit techethics.nd.edu.

10. Moving Data Governance to the Forest From the Trees (September 21, 2022)

Guest: Salomé Viljoen (University of Michigan)

Transcript

Kirsten Martin  0:03
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center or what we like to call ND TEC. In these discussions, we discuss an important idea, paper, article, discovery in tech ethics, and today I'm so happy to be joined by Salomé Viljoen. Salomé is an assistant professor of law at the University of Michigan. She studies the law governing digital information and [is] interested in how information law structures inequality and how alternative legal arrangements might address that inequality. And we're gonna talk about some of that work today. Salomé's academic work has appeared in the Yale Law Journal and the University of Chicago Law Review Online, as well as technical avenues such as the ACM Conference on Fairness, Accountability, and Transparency. So today, we're going to take a deeper dive into this article in the Yale Law Journal, "A Relational Theory of Data Governance." And I will just say at the beginning, I highly recommend this article. And the reason is that I think it does such a great job of being deceptively simple in the language on a really complex idea. And so I think if you don't know the background, and I think you do a great job of giving that background, of the nuanced turn that you make in how we look at privacy and data governance and the frailties of what we do right now. And that this different approach to data governance, which I thought maybe you could talk a little bit about the old way of thinking, or what we might like to think is the old way, the existing way of thinking, focused on individual control and data-subject financial gain, which I thought--that's the current way that we think about governing privacy, governing data, and how they're both limited. And then we can get into kind of your, the solution, which is more relational. But can you say a little bit more about this current way of thinking that we have around privacy or data governance more generally?

Salomé Viljoen  1:52 
Yeah. And first of all, thank you so much for the kind words about the piece. (laughs) It did not come out that way the first time.

Kirsten Martin  2:01
Oh, it never does. (laughs)

Salomé Viljoen  2:02
I think a lot of early draft reading by people near and dear to my sort of intellectual journey, who helped make it as clear as it is. So yeah, I'm really happy, heartened to hear that and, you know, all thanks get passed on to my behind-the-scenes friends and mentors and colleagues. So I'll sort of just quickly recap some of what I sketch out in the piece about, like, sort of how our sort of, like, typical intuitions are formed or sort of how we typically approach thinking about the kinds of legal interests or claims we have about information in that kind of traditional privacy register.

So the old way, as you put it, or sort of the traditional way is to sort of think about privacy as governing--I like to think of it as like, it's like a little house around all of us, our little sphere of autonomy. And privacy is sort of what protects that bubble. And so there's this feeling of like, much like we would say, you know, The difference between someone being in my house legitimately or illegitimately is, like, the difference between someone being, like, a trespasser and a guest. It's like, did I invite them in? And somewhat intuitively, we just kind of extend that to this idea of like, I have my little personal bubble. And the difference between whether or not, like, a company or, like, a government surveiller or whatever is, like, entering my little sphere, my little personal sphere, legitimately or illegitimately, is also, like, Did I invite them in? And the law sort of tends to think about that sort of, like, consent relationship in these two big--I'm generalizing, but like two big buckets.

Kirsten Martin  3:45
Yup, yup.

Salomé Viljoen  3:45
So one of them is like, If you didn't, if I didn't invite you in, and you're in, then you have sort of, like, undermined some sort of property interests that I have, and you owe me money. (laughs) Or you have sort of, like, undermined some sort of dignitary right that I have to sort of, like, govern my own sphere of autonomy. And that's not like a quantitative deficiency that you can make up with in money, but it's sort of like a qualitative deficiency, that you've wronged me in this kind of expressive or dignitary way. And what I recap in the earlier parts of the article are sort of saying, you know, there's like, reasons to think about privacy in these two ways. That, like, a lot of our life being datafied either as kind of this quantitative deficiency that companies are getting rich off of this valuable resource, which is like them violating our sphere of integrity, and we deserve a piece of that pie, or you know, kind of the dignitary account, that they're sort of eroding that sphere of autonomy, sort of my personal sphere, they're, like, trampling through my house all the time, and this is a dignitary violation, and the response needs to be to, like, thicken up those walls and shore up the door and make sure that when I let somebody in, I really mean it, right? Like, strengthening up consent. Yeah.

Kirsten Martin  4:56
And so that would be, so the equivalent if we're going to extend this would be, so then the solutions are things like, pay me for my data.

Salomé Viljoen  5:03
Yes. 

Kirsten Martin  5:03
You know, so like, that's my undermining of the property idea. So before you come in the house, pay me for it, and then you can come in my house. Or shoring up the walls of, like, explicit notice.

Salomé Viljoen  5:14
Yes.

Kirsten Martin  5:15
So this is the notice and consent idea fixing--and that "fixes," and I'm putting this in air quotes even though we're just audio--this "fixes" the dignitary harm of, like, not being asked permission first. And so, like, kind of forcing it on that way. And so this is, and this is, what I liked about it is it really does encapsulate the two arguments. I get told, especially in a business school all the time, Well, the fix is, why don't they just pay us for our data? And you're kind of like, that just does not get to what's going on. But what I liked about what you said is, and this is the aha moment, is that these miss, and I'm just going to quote you here, the point of data production in a digital economy to put people in population-based relations to one another. And so we're, we're treating this as like one-off transactions, where the way the structure of this market or this economy works is actually in order to gain population-level insights, and so the harms might be at that level, as well. And so what I liked about it is you're kind of like, we're having these fixes that don't work it the way the world is actually working. So that's the turn that you're making, right, is to talk a little bit--it's a different, it's a different approach.

Salomé Viljoen  6:23
Yeah. And so, you know, I think a lot of what I'm trying to do initially in this piece is just to say, like, Look, this little model of, like, the house and whether or not you think it's, like, a toll that I ought to charge people for coming through my house or, like, thickening up the walls, that's all well and good. That's just not actually an accurate model of this economy and how it actually works, and why companies and governments are interested in collecting information about us to begin with. But also, like, how insights about us that might trigger us being concerned about the people traveling through our little personal house or whatever actually even works. And it's not just about the harms; it can also be about the benefits, which is sort of another--

Kirsten Martin  7:07
Exactly.

Salomé Viljoen  7:07
--aspect of the piece. So yeah, I mean, I think a lot of the reorientation that I'm trying to do is just to say that--it seems really stupid and obvious in some sense to point out but, like, any data scientist would be like, Yeah, duh--but you know, information about me is never just--like, the way it's, the reason it's collected and the way that it's operationalized and used in the digital economy is never just about me qua me. Like, they don't collect information about Salomé that they stick in the Salomé folder to make decisions about Salomé; they collect information about Salomé that's like, you know, goes into forming populations. So millennial like, woman, like, likely to own a cat (laughs), like, Netflix user, and it's that sort of population pastiche that then allows them to derive insights that are like, Women millennial cat-owning Netflix users are likely to be interested in purchasing this, like, stupidly expensive Japanese ceramic cat bowl. (both laugh) What's sort of important about that from, like, a legal perspective is that insofar as I have, like, a legally cognizable interest that the law is going to, like, protect and give me a claim over, it doesn't attach to that one-off individual transaction. It attaches at the point of, like, those populations being formed and insights being derived from those populations and choices that are being made about those populations. And so that's just a very different way for us to be thinking about what kind of legal interests we're in the business of protecting when we say that people have legal interests in information about them.

Kirsten Martin  8:49
Right, yeah. And I, to this point, which you talk about is like, and what I liked about this is that you talked about that relational aspect of data production gets to both the social value that's created--so the point of, like, kind of like great insights that we can get from big data--but also then the harm of the data collection and use in a digital economy. And so we're looking at the harm at these individual-level transactions, where the real action is happening in this, like, in the neighborhood behind. And so, and in some ways whether or not I consented to the data collection or not, by the way it works, they can still draw inferences about me. So I might actually still be harmed in the same way as everyone else in the group. And so it doesn't, my dignity might be maintained at that one point, but it's actually the value and the harm that's being brought forward by this is actually happening no matter what. And so it really--I almost had this visual of all of us looking at like a needle or like the tree, like the leaves in the forest, but, like, missing the forest. You know what I mean? Like, so we're looking at these little pieces and trying to understand, How do we govern this individualized transaction? And we've kept our eye off the ball. I mean, not everyone (both laugh). There's many people who are looking at it. But the real action is on data governance broadly.

Salomé Viljoen  10:03
Yes. 

Kirsten Martin  10:03
And so not on these individual privacy things. And so I just think that that--and what I liked about it also is that you're talking about the social value and the harm that comes from it, as well.

Salomé Viljoen  10:12
Yeah, I mean in a lot of--again, it's like a lot of these points (laughs) seem actually, you know, I think to me quite clear and hopefully are clearly expressed in the piece, which is like, the thing that makes data collection about people valuable is the fact of our sociality. Like, the fact that I do share meaningful similarities with people who live in the same neighborhood as me or are also millennials or are also women. Like, that's what it means to be socially constituted beings, like, who we are like has a great deal to say about who we are as people, that we exist in relation with one another and in community with one another, which is basic to what it means to be a person. And that's what makes population-level data analysis--like the "big" of big data--socially valuable, is that we can be meaningfully understood in this population-level way. And I think that's like a profoundly--it's not bad or good, it's that's just, like, what it means to be a person. (laughs) But you know, as you sort of hinted at, I think a lot of what I'm sort of trying to reorient is, like, what like you said, those sort of from the leaves to the forest, is to say, like, that is what it means to be a person, and by the same token, like, a lot of our analog social relations are currently marked by conditions of exploitation and domination and oppression and marginalization. So it shouldn't surprise us that the sort of digital recreations and rematerializations and sort of remanifestations of the way we relate to one another--

Kirsten Martin  11:48
Inherits those problems, right.

Salomé Viljoen  11:49
--also become marked by conditions of domination and exploitation and alienation. And that's just a different account of what it is that we might be worried about in the process of datafication than, Did someone enter my house with permission or not?

Kirsten Martin  12:04
And I think, and this is a subtle--what I like also is, by us focusing and asking firms to focus on this handoff of information in paying someone and getting value for the data that they collected, this tradeoff idea, or the notice, give them enough notice, and that as long as they consent, you can do whatever you want. But the thing is that after you get the data, you can do whatever you want. That's the focus on both the dignitary reforms and the property-like refo rms--like, so either they're paying someone for the data or the notification for data--is that it really never gave any guidance or discussions around what you do with the data afterwards, right? And so this is where we've kind of taken our eye off the ball, so to speak, in private firms and businesses is that we have just said, Well, as long as I've gotten the data, I can do whatever I want. And what you nicely, like, literally take the lens and say--not that you say, I don't care how you get the data--but pretty much, like, stop focusing so much on this one thing. Like, we have enough rules around that, that's fine. What we need to focus on is this entire cloud or forest and what's going on there, and that's where we need to focus. Because that's where the value is being created, that's where we're recreating these relationships that are exploitive. You know, that's where all the action is happening. And we're so fixated on this handoff or trading that we're missing giving guidance on the rest of it. Is that fair? 

Salomé Viljoen  13:24
Yeah, absolutely, yes. I mean, again, the way I sort of frame it for legal audiences is like, consent works as, like, a thing we do in law. Like, you know, again, to grant the imprimatur of legitimacy to, like, Is this person in my house or, like, you know, whatever, when I am the agent affected by my own decision. I'm teaching contracts next semester, and when I teach my students contracts, I'm going to be like, A contract is bound by consent--like, you agree to the terms of the contract, and then you're bound by your contract because we think that consent is doing important moral, political, and legal work. Which is to say, I had an opportunity to negotiate these terms, I agree to them, now I need to be bound by my promise. Consent doesn't work if I'm consenting on behalf of another, or another is consenting on behalf of me. But because of how the data economy works, we are functionally consenting on behalf of others, and others are consenting on our behalf irreducibly all of the time. So from a legal perspective, I'm sort of just intervening to say like, We're doing the wrong thing. (laughs) And that follows very nicely to your point, which is to say, like, I don't even know what it means from, like, I'm granting a stamp of legitimacy on consenting to a data flow when if I were to never consent to any of that information being granted, if I were to, like, fully exclude myself from the digital realm--which would be extremely difficult to do--the accuracy of inferences and predictions being made about me, because of how many people I share population features with, would not go down at all. (laughs)

Kirsten Martin 14:58
Right, right. Yeah. They can still know what your concerns are, you know what I mean? Like, so it's just one of those things where--

Salomé Viljoen  15:03
They can know what my concerns are, they can know where I live.

Kirsten Martin  15:06
Yup.

Salomé Viljoen  15:06
They can know what shows I'm likely to watch, they could know which food I'm likely to be in favor of, they could know with a great degree of certainty whether or not I'm pregnant or likely to become pregnant. Like, very intimate decisions that we think of in that dignitary frame. You know, it's very intuitive for us to think about a set dignitary frame, and it's, like, scary to think that those sorts of insights aren't up to us. But they aren't. Which point exactly to, as you said, like, this entire kind of back-end process of how this information is being used, what sorts of insights are being made about it, how are those driving business decisions? A lot more attention and focus I think there is clearly kind of one implication of the work.

Kirsten Martin  15:48
Yeah, and because that's when--I think you're right to take a critical lens to, like, what's going on with the data to say, right now, I always joke, you know, data, a natural place to look would be white-collar crime for predictive analytics. That would, it'd be pretty natural. If you asked people, you know, at the SEC about what they look for with white-collar crimes and, you know, they look for very small changes in expense accounts and all this other stuff. But we don't use it for that, you know what I mean? Like, we use it for other things, you know, like, you know, predicting who's going to be a criminal at the border, we use it for, you know, deciding who's going to get welfare by the state, you know, who's going to, what their criminality potential is. So I think it's smart to kind of focus more on that. So we talked about, like, what doesn't work, and I want to make sure we get to, like, part of the ideas of the solution. And so could you say a little bit more about the alternative approach, which is data as a democratic medium, as an alternative approach to the way we think about it?

Salomé Viljoen  16:45
Yeah. So I think your prompt about white-collar crime is a really nice sort of segue into this. So, I think there's sort of two parts to that solution. One is sort of a theoretical reframing, which is to say, as we sort of talked about a little bit earlier, like, when you kind of get out of this permission or payment model to focus on the relationships that are sort of being rematerialized in our data flows. So what it means to be put in relation with other people and what the sort of quality of those relationships are. One part of what it means to think about governing those relationships more justly is to say, Are they relationships that, were they happening in an analog world and we were able to sort of look at them and evaluate them, we would be like, These are relationships where these two entities or this population of people have relatively equal standing in what this population that they're being brought together in is and what it means to them and what the stakes of that are. Which is to say there are sort of, like, procedural and substantive conditions we would want to see being met. And that's in part what I mean by democratic, is like, is everyone in that sort of relation, are their interests being taken into account, are we thinking of the people that we're being put into relationship with as our moral and political equals?

And that doesn't always mean that you have to, like, have a full democratic process for every data flow. That would not be realistic, obviously. (laughs) But it does mean that certain things where we have, like, every reason to believe that a certain group of people are very willing to give this information because they enjoy a high status and low risk, but it has clear implications for people that enjoy a lower status and will shoulder much greater risk, that we would look at that data flow with far more skepticism. And here I'm thinking about things like facial biometric data. Like, a lot of facial biometric data is being collected by people who want to move through the airport faster because they're business people who travel a lot. They're fairly secure in our social structure, they enjoy a high degree of status and standing. Those systems are being trained on face data that then are increasingly used and incorporated in fairly carceral systems at the border, in enforcement. So they're being placed in relation with people who enjoy much lower status in our social structure and face far greater risk from being placed in those data relations. So we don't have to necessarily, like, convene some democratic process of everyone who has a face. (laughs) But we have to think about what the quality of that data relation is and if it's the kind of thing that sets off alarm bells about likely not materializing a relationship in which we are treating people as having equal political standing and taking their interests into account equally. So that's kind of the substantive and the sort of procedural component of what I mean by a democratic data relation.

Kirsten Martin  19:39
That's great. That's great. I remember, I was talking--what I liked about it is, it's a common--I remember, like, getting a review back on a paper (laughs), which everyone remembers the reviews back from papers. But the answer to them was, like, Well, couldn't the subject of this predictive analytics program, like, just complain? Like, What do you mean their interests aren't being taken into account; like, they have to be taken into account. And I was like, Oh, why? I mean, like, there's no, I mean-- 

Salomé Viljoen  20:05
You're like, On the basis of what? (laughs)

Kirsten Martin  20:07
Yeah, right, exactly. I mean, not all of us are kind of, like, thinking, like, just every actor has an equal amount of power and can transact openly in the marketplace. And so if you see people have different political standings in the marketplace and different abilities to actually transact or bargain, then you know, you could easily have people that are being only subject to the predictive analytics program and have no voice whatsoever and no ability to have any voice. And no recourse--you know, there's no appeal process or anything along those lines. And so it reminded me when I was bringing up the white-collar crime example of, like, if you train something ought to, you know, find white-collar criminals or instances of abuse or something like that, and then trained it on welfare. You know what I mean? Or something else of the state, where then all of a sudden, you're taking it from the private company to the state, you're taking it from people who have--you know, are lawyers or have lawyers at the ready to people who have no ability or not trusted at all. And so you can easily see that same idea of training something in one area and then putting it onto another.

Yeah. And I think what really resonates when you read this is how you capture how the world is working right now, both in the benefits that it's creating and the harms, and saying that this idea of data governance covers both. You know what I mean? So it--because you can't find solutions that don't understand how the benefits are being accrued, because otherwise, the market will go after those benefits. And so you have to kind of understand and encapsulate how the market is actually working or the industry is working at the moment. And so that's what I liked about your paper is that it really does capture the way the world is working right now and where the action is, both the benefits and the harms. And then second, I'll just say that it really, what it offers, and you even say it offers two things, it better reflects, you know, the economic value being created but also it gives us another idea about what the harm is, that it's different than a property or dignitary harm, that the harms can be exploitation and other harms in the marketplace that exist outside the online world or outside the big data world that are just being reinforced in the data world. And so that, I think you did both those things really, really well. 

Salomé Viljoen  22:23
Thank you.

Kirsten Martin  22:23
And I really commend you on the paper. And I'll just say that if people get a chance to read it, there's a great example, and we'll run out of time to talk about it, but when you talk about water corp and water org.

Salomé Viljoen  22:33
Yeah. (laughs)

Kirsten Martin  22:33
And how--it's a really thought-provoking example of where the property and the consent or dignitary approach fails, and how people who espouse that have a problem navigating between a government-run entity that's trying to conserve water versus someone else that's actually trying to exploit you for your water use. And so you really do a great job of using those examples, as well. 

Salomé Viljoen  22:56
Thank you. You know, I think definitely for me, part of the kind of intellectual project of this piece was to try to provide a theoretical account of data governance that really did capture both. Like, really was able to provide an account for what is beneficial about datafying, you know, about trying to understand how we are as a society and how we relate to one another in ways that can be extremely helpful and extremely beneficial, while at the same time still providing a very compelling account of why a lot of people are justified in feeling extremely skeptical about how a great deal of datafication has in fact been realized and the sorts of results that it's had for some of the most vulnerable communities. And so really sort of--again, we all probably think our theoretical accounts are good, and we should hopefully all think they're good and compelling; otherwise, it's like, Why spend years of your life trying to articulate them?

Kirsten Martin  23:56
(laughs) True.

Salomé Viljoen  23:56
But I do think what's really nice about focusing on relations is you can really see both. You can say, you know, if we can really use this profoundly, I think, powerful way of understanding how we relate to one another to try to intervene on those relations of inequality, like you said, to try to understand things like white-collar crime, to understand things like patterns of my water usage and how they might meaningfully be reduced, to try to understand things like how I as a worker am connected to a global supply chain and to try to, you know, develop solidarity across that. Like, that requires information, that requires us to, like, know what we don't already know and to see those ways that we are deeply bound up in the project of one another's social constitution. That's very powerful. But at the same time, if those data relations are sort of recreating relations of exploitation, relations of domination, relations of imperialism, it's important to sort of pinpoint that as the account of what is being harmful right now because we don't have the tools individually to sort of, like, individually consent our way out of that. In related work on this topic, and maybe in the paper too, I've sort of said, you know, if you think about something like, I use this example of location data taken from a global Muslim prayer app being used by the US military to track the location of Muslims around the world. Like, what's wrong about that is not, like, they didn't get my consent. (laughs)

Kirsten Martin  25:26
Right. Right, right, right.

Salomé Viljoen  25:29
What's wrong about that for a lot of people, a lot of Muslims around the world who are understandably expressing outrage about this on the internet, was this idea of, like, being drafted into the project of oppression of your own community just by having used a prayer app.

Kirsten Martin  25:47
Right, right. 

Salomé Viljoen  25:48
That's kind of a relational account of what's wrong with that, you know?

Kirsten Martin  25:51
Yeah, and that's a great example. I mean, and I like the idea of, like, you can't consent your way out of that problem or pay off someone enough to get their data. Like that's, that's not going to fix the problem because that's actually not where the harm was. It wasn't, the consent wasn't the issue. It was, like, the use of the data in that way. And you're right, there is that additional--there's a friend of mine, Vikram Bhargava, he has a paper on social media addiction. And he argues the same type of thing, which is, it's the use of your data almost against your people or yourself, that's a different type of harm than even the other things that we're talking about. Like, it's an issue of, like, I'm actually contributing to this problem that I am trying to fight in other areas of my life. And so, which is an issue of the use of that data. And I think that you do a great job of explaining that. And I think there's a lot more to say--I mean, yeah, there's a lot more (Salomé laughs) to say about that. And I think that you, I look forward to everything that you're going to do in the future in this area, as well, because I really, I can't recommend it enough. And the intro and even the first, like, 10 pages really encapsulates it, and then you can read the rest of it and look for water corp, as well, as an example. (Salomé laughs) So before, like, I always ask this question, and so is there anyone in the area of tech ethics that we should be paying attention to, younger scholars in particular, and it can be from any field. I'm kind of indifferent to fields. But if there's anyone that you kind of like, Oh, I can't wait to see what they're writing about next.

Salomé Viljoen  27:14
Yeah, so a colleague of mine who's writing in the European context about smart cities, Beatriz Botero. Her work is really great and definitely worth checking out. A colleague of mine up at McGill, Ignacio Cofone, is working on a book on privacy that I'm, you know, his work is always very good and very thoughtful. A friend of mine, Elettra Bietti, works a lot on sort of antitrust and digital platforms. 

Kirsten Martin  27:42
Oh, interesting.

Salomé Viljoen  27:42
So she's definitely worth checking out. I've only given you legal scholars at this point, which is very closed--

Kirsten Martin  27:47
Oh, that's okay, that's what you need to worry about. (both laugh) That's okay. Yeah, that's okay. I totally understand. I think, you know, and I think, I would just say one last thing about what I like about your move to data governance as well, not just privacy, is that sometimes to capture your harms that you're identifying and the solutions, we twist the word privacy in so many ways and, like, try to stretch it because we can't figure out how to talk about the harms that you articulate so well. And so, I've really liked the move to "data governance"--I mean, data governance has existed as a field, so I don't mean to say just that. But to kind of explicitly link the two and then kind of say, Look, we have to talk about it in this way because this is really where the action is. Because sometimes we want to, because it's about data, we want to use "privacy," and then we're talking about inferential privacy, and then we're talking about you know, like, and it just--sometimes it's just worth saying this is more about data governance, and there's privacy issues within it, but there are other things going on as well.

Salomé Viljoen  28:47
Totally. 

Kirsten Martin  28:47
And so I think it's super important.

Salomé Viljoen  28:50
Yeah, I'm very deliberate about using "data governance" everywhere. Because I think, exactly as you said, there are privacy interests being implicated, but there aren't only privacy interests being implicated when we talk about data governance. Oh, one more name that I absolutely can't recommend enough is a colleague of mine, Amanda Parsons.

Kirsten Martin  29:11
Okay. 

Salomé Viljoen  29:11
She is actually an international tax scholar, but she writes a lot about sort of the global justice, sort of global tax perspective on taxing digital assets. So how do we tax wealth created from data collection?

Kirsten Martin  29:25
Oh, interesting.

Salomé Viljoen  29:26
Sort of from like a global perspective.

Kirsten Martin  29:28
Oh, that's super interesting. 

Salomé Viljoen  29:29
Yeah, she's kind of a different perspective on things. But yeah, has written great stuff about it and is very thoughtful. 

Kirsten Martin  29:38
Oh, that's interesting.

Salomé Viljoen  29:38
And I think we're not always thinking about, like, there's like this whole other bucketed problem, which is, like, tax (Kirsten laughs), which is our traditional redistributive mechanism. And yeah, so she has a great perspective.

Kirsten Martin  29:49
I've been going to the--I'm not a lawyer, but I've been going to the Privacy Law Scholars Conference for, I don't know, like 10 years, and I always call lawyers, like, the tip of the spear. Like they (Salomé laughs), they just are so, because they're good at issue-spotting, and I realized that talking to Casey Fiesler at University of Colorado, and she talked about issue-spotting as a mechanism of looking for things. And I realized that is how you're trained is to be issue-spotters.

Salomé Viljoen  30:09
We are literally trained to do that, yeah.

Kirsten Martin  30:11
Exactly. Like, that's part of, that's what you learn in law school. And so they're just always kind of looking around and trying to understand what's going on in the real world, and then bringing it back. And other disciplines just aren't always so nose-to-the-ground of actually trying to find the current thing that's going on. And so I always find it fascinating--and you guys write really well, too, so it's easy to follow. (both laugh) Well, some of you do, but like, others are just really long. But that's okay too. (both laugh) Well, gosh, thank you so much. I really appreciate it.

Salomé Viljoen  30:44
No, thank you so much for a great conversation. I really enjoyed it.

Kirsten Martin  30:48
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu

9. It’s AI, Not a Personality Detector (Part 2) (September 7, 2022)

Guests: Luke Stark (Western University), Jevan Hutson (Hintze Law)

Transcript

 

Kirsten Martin  0:03
(voiceover) Hey, it's Kirsten. This is part 2 of my conversation with Luke Stark, an assistant professor in the Faculty of Information and Media Studies at Western University in London, Ontario, and Jevan Hutson, an associate at Hintze Law PLLC. We’re talking about their paper “Physiognomic Artificial Intelligence,” which appeared in the Fordham Intellectual Property, Media and Entertainment Law Journal.

In part 1, we started with the troubling history of physiognomy and phrenology. These two pseudosciences were widely discredited in the 20th century, but their notions that people’s external appearances can be a way to access internal truths about them have made a comeback in the form of AI systems that purport to be able to perform this type of analysis.

Here, we pick things up with me asking Jevan about the menu of regulatory options he and Luke propose in the paper to remedy the fundamental problems with this physiognomic AI. (end voiceover)

So, Jevan, I thought maybe you could answer this, talk a little bit about some of the remedies. The law I thought was interesting, like, the different--I'll say the mechanisms that we might approach this. One small--it's not a small detail--but I was surprised to see, and it's an anecdote kind of in the beginning, that there were competing bills in Washington state legislature around facial recognition technology, which excluded facial analysis and emotional extrapolation from the definition. It was (laughs)--just, that amazed me because it's one of the major problems. It just took a whole bunch of the problems of facial recognition and said we're just excluding this from the bill. But anyway, there are other remedies. But that was an interesting example.

Jevan Hutson  1:41
Yeah, well, I mean in many ways, this sort of, I mean this paper kind of came out of those sort of initial legislative discourses around facial recognition in Washington state, where we had sort of one side of sort of community advocates and impacted communities advocating for sort of the abolition or banning or prohibition of facial recognition technologies, where on sort of the other side of the aisle, we had many of the larger companies here in Washington state attempting to, one, scope regulation to simply facial recognition. Because to expand it to any other form of facial analysis would impact (laughs) a host of other artificial intelligence and machine learning tools that they are either currently working on, developing, or currently use. So their sort of goal was to sort of keep the conversation to facial recognition because to open it up to facial analysis, so to speak, would both hurt the bottom line and otherwise sort of impede other efforts, which, in many ways here inspired the paper, where it's like, Well, all of those other things are, like, fundamentally worse. (laughs)

Kirsten Martin  2:39
Right.

Jevan Hutson  2:39
Like, facial recognition is bad, like, it's bad, it's contemptible, is worthy of prohibition as communities and advocates have laid out over years, right? But this entire other arena is deserving of, like, an incredibly higher level of scrutiny from lawmakers and regulators. And so for us, in terms of sort of developing, How do we address physiognomic AI? Obviously, we situate our response with, in sort of the field of abolition, where obviously the goal is to abolish physiognomic AI systems. We fundamentally reject computer vision applied to humans. And so we have to think about, okay, what sort of mechanisms, at least in the United States, might allow us to sort of effectuate the abolition of physiognomic artificial intelligence.

And sort of first and foremost, we sort of start with consumer protection law, which in the United States has really been sort of the active area for thinking about sort of unfairness in AI, and how, you know, we might prevent these systems or otherwise regulate them. Obviously, sort of reflecting on some of the limitations of consumer protection law, we sort of aim for legislators actively sort of enshrining physiognomic AI as being an unfair and deceptive per se, effectively where consumer protection regulators don't have to go through arguing why physiognomic AI is, like, unfair or deceptive under extant standards. And as I think we've laid out, it's like, these are fundamentally morally bankrupt systems that, you know, federal regulators should have the ability to sort of sidestep, like, a massive investigative process because, you know, HireVue can throw research papers (laughs) from various institutions and sort of engage in this sort of unnecessary process. And so we sort of talk about different ways in which they might be sort of legislated as being unfair and deceptive. We talk about sort of some of our efforts that we undertook in Washington state to attempt to do that. In Washington state, for example, there's like 160 different practices under state consumer protection law that are sort of fundamentally unfair and deceptive, you know, sort of per se under law.

Kirsten Martin  4:31
And so if I, just to understand that really fast, the argument is--which I like--is that this is snake oil, to use Arvind's [Narayanan] term, this is snake oil, and as a class of computer vision offerings, of AI offerings, it should just be, like, labeled as unfair and deceptive. To claim that you can identify someone as trustworthy, happy, sad, whatever it might be, anything, possible shoplifter, whatever the purported computer vision AI is saying, it's not possible. It's pseudoscience. And so it would be considered deceptive. Yeah, I mean, that makes sense to me given your paper, so I can understand why you make the argument.

Jevan Hutson  5:12
(laughs) Yeah, all I'd add to that is sort of the levers there, whether that's the FTC making a policy statement, engaging in Magnuson Moss rulemaking, or rather just lawmakers just fundamentally enshrining it to, you know, give further direction to consumer protection authorities.

Luke Stark  5:30
Just really quickly, I mean, one reason why it's snake oil, right, is the kind of broken telephone that happens between academic papers or, you know, actual machine learning results, right? Which often are pretty narrow, and you know, they present a kind of statistical--you know, a probability range that then somehow becomes this kind of like 100% we can do X, Y and Z, right? So whenever, just whenever you're looking at one of these claims by a company, you gotta really break down exactly on what experimental basis are they making the claim, you know, the claim in the first place, and then of course on top of these kind of conceptual problems that backs up the argument. But that's, you know, it's hard to do that. You have to have a lot of data, you have to have the data from the companies themselves. And so that's why, right, I think we can say to the FTC, Look, you know, this is just not accurate, this is unfair.

Kirsten Martin  6:25
Right.

Jevan Hutson  6:25
And it builds on some of the stuff that the FTC has already put out with respect to guidance around AI, where they're like, Don't overstate the findings, don't overstate the sort of capacity of your AI, do not state that your AI is bias-free. (Kirsten and Jevan laugh) And here, we sort of think, like, this is an arena of sort of AI and ML technologies that just need to be fundamentally off limits. Like, they are fundamentally unfair, they are fundamentally deceptive, on so many different levels. And it's sort of represents an area that we think the FTC and state consumer protection regulators can sort of say, This is a red line. Like, facial analysis onward that is attempting to predict these particular characteristics is a no-go, and we're going to effectively police it under, you know, federal or state consumer protection law. And we think that might be a very helpful incentive (laughs) to maybe stop, you know, larger institutions and smaller institutions from proceeding to developing and deploying these systems.

Kirsten Martin  7:19
I think, yeah, that makes a lot of sense.

Jevan Hutson  7:21
But on top of that, you know, we unpack sort of other mechanisms as well. Our goal is really to offer sort of a menu and a toolkit of options to policymakers because, you know, no body of law is necessarily perfect. And so we also talk about, you know, potentially enshrining this in the context of biometric law, we can imagine in sort of the context of BIPA adding a sort of another categorical prohibition. We also talk about anti-discrimination law and building on some work we also had done here in Washington state around attempting to prohibit these systems in places of public accommodation. As sort of a broad-brush prohibition, this would obviously touch on a host of both quasi-private and public institutions, you know, that shape access to basic social goods, but also just, you know, impact people's dignity, right? The ability to go into a store or to go about public life and not be impacted by systems that are trying to categorize you as being suspicious or criminal or even just categorizing you along particular protected class lines. You know, that impacts our ability to enjoy public life, and it really calls into, you know, some of the similar questions under, you know, civil rights law. And so, our goal is to offer a menu of options. (Kirsten and Jevan laugh) Physiognomic AI raises enough concern that we need to exhaust legal methods that we at least have here in the US. I think we have existing tools that could allow us to abolish physiognomic AI, but obviously, they're worthy of new potential frameworks and paradigms. But I think we have a toolkit; it requires a political will that I think is, again, part of the, we hope to be, you know, the conversation and discourse around, like, how we prevent both facial recognition and larger, you know, facial analysis systems that [inaudible] physiognomic artificial intelligence,

Kirsten Martin  8:55
Yeah, I liked how you gave a menu of options, with the first bucket being unfair and deceptive, which given your paper makes a lot of sense. And I thought the idea of biometric was a good--I mean, they're all good avenues, it gives people a couple of options.

Luke Stark  9:08
I mean, isn't it funny that, you know, we have, in some ways, it almost feels like we're overdoing it because we have all of these options, and we point to all the ways that these technologies run afoul of all these different kinds of regulation. But I mean, it's odd that that would be--you know, this is so obviously a problem across so many different vectors and facets that it makes it sort of all the more amazing that these technologies are not just continuing to exist but are being still being feted largely. Now of course there's been a lot of pushback in the last couple of years, but, you know, not enough. I really analogize these technologies to a rotten onion, where the counter arguments from these technologies' proponents tend to kind of involve these different layers. So you talk about the problems of use. We know there are lots of issues with the use of these technologies around, you know, discrimination, the chilling of political speech, etc. Well, the response from technologists in favor of these technologies is, Oh, well, we'll just improve them--[inaudible] so sorry, that's not the response. The response to the use case is, Well, we have nothing to do with that, we can't, we have no ability to change how people use these technologies, which is obviously not true.

Kirsten Martin  10:18
They patent it. I mean, they patent it. You know what I mean?

Luke Stark  10:20
They patent it, exactly. (Kirsten laughs) And then they're the technical issues. And the response to the technical issues around accuracy is, Oh, well, we'll just get more data, it'll be better, will be better. And so that's, so that's the second layer of the rotten onion. But then I think that's why it was so important to make this point in the paper: At a conceptual level, quantifying the human face and body and putting those quantifications into kind of categories and classifications and conversation with each other is, it just automatically opens up the door to things that we would recognize as racism and sexism and other forms of discrimination. And so at a conceptual level, like even the center of that onion is rotten, and that's why we're so committed to all of these different legal remedies because the whole edifice is just not good.

Kirsten Martin  11:07
Right. And I think that's where you can feel this huge disconnect, and where I think actually the paper does a nice job of saying that within people that study AI ethics or just in general, it is just a non-starter. You know what I mean? Like it's, you see it, and it's offensive on so many levels, and you do a great job of explaining how offensive it is, both as like a dignity harm, as actual harm to people, as actually just doesn't even work, but we're still kind of going through the motions and harming people for no benefit of it whatsoever. And yet, the other side is almost extremely the opposite, the opposite meaning its commercialization and use within governments and businesses and even research as an endeavor continues to proliferate. And they're kind of shocked-shocked when there's pushback. Out of all the areas that we all look at, it seems to be this one area where the two sides seem so far apart. Versus, like, the use of consumer data, just generally, even those within organizations have to say, Well, I mean, there are no laws. I mean, they kind of know that it's not great, right? Like, they kind of admit it, but this one, they don't seem to admit that it's wrong at all. I mean, there's Wall Street Journal articles about it, the Washington Post writes about it, it just seems to proliferate with how you could get through, how can we categorize students? How can we look at people who are applying for a job? How can we look at people en masse when they're in the airport? How can we identify future terrorists? Like all of this--who could be, who's actually going to be trustworthy to give them a loan? All these uses that you would think--and the matchmaking that you point out--are based on this pseudoscience that as you rightly put, the "scientifically baseless, racist, and discredited pseudoscientific fields--which purport to determine people's characters, capabilities, and future prospects based on their facial features or the shape to their skulls--should be anathema to any researcher or product developer working in computer science today." And so we see it and think: anathema. You know what I mean? This is like a--and yet. And yet. You do a great job of listing all the ways it keeps going.

Luke Stark  13:11
I think one thing just really quickly to say, and this speaks to this idea that the physiognomic impulse, you know, went sort of underground but didn't disappear between, let's say, 1945 and, you know, 2005 or whatever it was, right, is that I think a lot of scientific disciplines--evolutionary biology, certain parts of anthropology, right--a lot of scientific disciplines have kept vestiges of this kind of physiognomic idea. Or have not interrogated closely, you know, what they're actually studying and what the proxies that they think they're looking at, you know, and what they think tell them things actually tell them. And I think that's--you know, so I think actually a lot of people in the sciences, they wouldn't agree with phrenology, they wouldn't agree with some of these kind of crass physiognomic ideas, but there would be a kind of belief that you can tell certain things, maybe quite limited things, about individuals based on, you know, exterior traits. And again, informally, right, humans do that all the time, and we are more or less good at it, often less good at it. So I think that's part of why this keeps going. 

Kirsten Martin  14:22
I agree.

Luke Stark  14:23
I think we in the sciences, or broadly in the sciences and social sciences, really need to have the conversation and really dig into the assumptions in some of those fields. Because I, you know, there's been lots of great critical work in STS more broadly that makes the point that, you know, none of these fields have been studying the things that they think they're studying, right? They're not, the proxy data that they're using is not necessarily determinative, some of the underlying assumptions that are being made are indeed racist and sexist. And so, you know, this is a bigger conversation for the quote-unquote human sciences to be having. Some folks in those those fields are, absolutely, but not everybody is. And so when those more retrograde notions from those fields get applied in computer vision, that's where you get these big problems.

Kirsten Martin  15:11
That's great. Yeah, no, thank you. And I appreciate the work and the thoroughness by which you both go through the problem and the scope of the issue in general. And so as I usually wrap up, I like to ask people if there is anyone in the field of tech ethics--it can be in your discipline or someone totally outside your discipline--that you look forward to hearing from. It can be a young scholar, someone that is, like, you look forward to seeing what their work is or there's new work that you'd like to talk about.

Luke Stark  15:38
Oh, there just so many awesome folks. I mean, I'll give a couple of shout-outs to Canadians, fellow Canadians, that I really love. One is Deb Raji, who I think [whose] work is probably well-known by now but who is doing awesome things at the Mozilla Foundation on these topics. The second is Catherine Stinson. She's in philosophy and computer science at Queen's University, and she has a neuroscience background and is really working hard to produce, you know, empirical studies that kind of systematically work through the methodology of some of these physiognomic AI papers and debunk it. So you know, so that's really, really awesome work too. So they're--there are so many awesome, amazing voices in this space, it's wonderful. It's wonderful that there's so much good work happening right now.

Jevan Hutson  16:20
I'd add to that, I mean, to Luke's point to Deb, to Deb's paper at FAccT, "The Fallacy of AI Functionality," I think really hits home to some of this that, you know, regulators really need to consider. Like, how do we respond to AI when we let go of the assumption that it fundamentally works? Outside of the academic space, I want to give a shout-out to Jennifer Lee at the ACLU of Washington, who's been doing, I think, some really great work on the policy front and trying to expand not only, say, like the express definition of facial recognition but thinking about how we can drive sort of community-centric tech policy and sort of expanding, you know, the traditional discourses around what's possible and leveraging sort of existing regulatory frameworks to, say, prohibit physiognomic AI as well as facial recognition and other problematic tools.

Kirsten Martin  17:07
Well, gosh, I really appreciate it, and thanks so much. It's great to talk with you. And I really love the paper. And I'd been, I knew Luke was working on work that was around this area, so I was excited when it came out. And I'm really glad that you guys did it, and thanks for coming on and spending, you know, 40 minutes with me and stuff. But anyway, thank you very much for coming.

Luke Stark  17:24
It's such a pleasure. Thanks so much. 

Jevan Hutson  17:25
Of course. Great to finally meet you.

Kirsten Martin  17:28
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

 

8. It’s AI, Not a Personality Detector (Part 1) (August 24, 2022)

Guests: Luke Stark (Western University), Jevan Hutson (Hintze Law)

Transcript

Kirsten Martin  0:00
(voiceover) Hey, it's Kirsten. Just a quick note to tell you that this episode marks a first for TEC Talks: We're still talking about one paper, but this time, I'm joined by both of the authors. So with two guests, our conversation went a little longer than usual, so we've decided to break it into two parts. Here's part 1. (end voiceover)

Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we discuss an important idea, paper, article, discovery in tech ethics, and today I'm so happy to be joined by two people, Luke Stark and Jevan Hutson. Luke is an assistant professor in the Faculty of Information and Media Studies at Western University in London, Ontario. He researches ethical, historical, and social impacts of technology such as artificial intelligence and machine learning. Jevan is an associate at Hintze Law; Jevan's practice focuses on the intersection of privacy, security, and data ethics, with a particular experience in emerging ethical and policy issues in AI, machine learning, and computer vision. You can see where your expertise overlaps, which is great. So our goal in this series is just to take one idea, one article, and really do a deeper dive and understand the larger implications for the field of tech ethics just generally. And today, we're gonna do a deeper dive into this great article, physio--"Physiognomic Artificial Intelligence." So I'll mispronounce that a couple of more times today.

What I really liked about this paper is, there's been so much in the commercial area that we see, just online and in research, attempting to identify something about someone's face--like, so using facial recognition and computer vision, with physiognomy being the study of features of the face or the form of the body. And what I like, you differentiate these two ideas, and I'm hoping that one of you can go through a little bit of the history of both, is that one is the practice of using people's outer appearance to infer interior characteristics. So you can see how computer vision would be used to do this: if someone's trustworthy, if they're anxious, if they're upset, if they're happy, if they're smiling enough. Whereas phrenology is the study of, mainly around the human skull. So it's just a subset of this larger area. And I was hoping, if you could, give a brief background on these two concepts and the racial history of these two concepts.

Luke Stark  2:20
Sure, I can jump in on that, being, since being, I suppose, the historian, the labelled historian in the duo. (Kirsten laughs) And I want to--I mean, first of all, thanks so much for having us. This is great, it's great to be able to chat. And I also want to flag how much great historical work there has been on physiognomy and phrenology in, especially in its kind of 19th-century heyday, or at least its first heyday. I think one of the things we're arguing in the paper, unfortunately, is that it's coming back and having a bit of a second heyday, which is really too bad given it was kind of discredited once already back 100 years ago. So yeah, so physiognomy is this idea of being able to tell inner characteristics from exterior signs or signals and has to do with the face and the body. It also--and this really interestingly, you know, we realized when we were doing the research--it also in the 19th century had to do with things like makeup and how you dress and this whole array of things. So, so physiognomy, you know, it was, it wasn't just about a kind of biologically essentialist kind of set of questions. It was also, it was sort of about this general science of signs.

This idea of conjecturing about people and making inferences about people obviously, you know, has a long history; people make inferences about each other all the time. And physiognomy in the 19th century, again, was this broad class of categories. It was actually quite hard to make scientific. And that's one of the points that Sharrona Pearl, who's written a great book about this topic, makes. Phrenology, as a subset of this kind of physiognomic impulse, was the first, I guess you could say, sort of structured attempt to claim scientific validity about some aspect of the exterior of the human that then could be used to kind of understand things about the interiority of the human, about somebody's intelligence, about their personality. And that's kind of where things really--well, I mean, you could say physiognomy, you can often go wrong there too, and we could talk more about that. But phrenology in the late 19th century became this kind of, this kind of pseudoscience, right--it was seen as a science at the time by some people--that tried to find these correlations between the shape of the head, the shape of the skull, and various kinds of character traits and virtues. And this history was incredibly racialized, right? Racialized and also gendered, right? So the idea of what kind of skull was the kind of virile or responsible skull, you know, really tracked to racist ideas at the time. The idea of, people won't probably know about this, you know, the idea of the size of the forehead as an indicator of intelligence meant that all sorts of kind of totally spurious correlations got made around ethnic and racial background and intelligence.

But another thing I think it's important to flag about physiognomy and phrenology in the 19th century that is relevant to today's AI systems is that often these pseudosciences were being put forward by people who consider themselves social reformers, who consider themselves to be liberals, to be progressive. Because this was a science, you know, it was an organized way of classifying people. You know, phrenologists and physiognomists, phrenologists especially sort of said, Well, we don't think everybody can possibly be, you know, not everybody can be at the same level of intelligence, but if we can identify people who are of lower intelligence, we can help them in different ways than we would other people. So there were these kinds of putatively, supposedly liberal ideas that were pushing these technologies. But of course actually, it's totally--I mean, fundamentally, it's both fundamentally unsound and fundamentally racist, classist, sexist, and fundamentally wrong, right? And that's why it was, you know, by the early part of the 20th century, mostly discredited. And so it's alarming to see physiognomy broadly understood, you know, really take so much, take up so much oxygen in AI these days. And even phrenology, even like, even the specific act of looking at the shape of the skull, I think there was an IBM patent that talked about doing this a couple of years ago.

Kirsten Martin  6:20
Interesting.

Luke Stark  6:21
So unfortunately, this idea that you can tell certain intrinsic truths about the interiority of the individual from their exterior, their exterior features, is now unfortunately again really popular.

Kirsten Martin  6:33
It was just today, and [these] tweets could have been going around yesterday, but there was a whole bunch of people looking at a profile of--there was like a new logo for an organization where they use three people's profile pictures, and it was very much like phrenology in trying to, people were saying that those are male heads, those are female heads, and they were not gendered at all. And then other people were putting their own face up next to it and then all of a sudden trying to gender and misgender those people. And so it was just, it was amazing. Now that's not AI; this was just someone taking--but the idea of phrenology never dies. And it was immediately gendered. It was, like, it wasn't even a beat, and then it was used in a pejorative way, you know, even though it wasn't, it was just someone's logo.

Luke Stark  7:17
One thing I would say, and maybe Jevan, you have things to say about this too, is that one of my worries about all of these kinds of physiognomic AI systems and their, you know, their advertisement and their kind of hyping in the tech community is that it's getting people to think physiognomically when they really shouldn't be. It's promoting this idea of physiognomy in ways that, you know, no one would otherwise think of and giving people this kind of totally spurious set of characteristics and connections to latch onto. I think that's one of the many big worries I have about these technologies.

Kirsten Martin  7:51
I think that's a great point. And Jevan, and I didn't know if you could talk a little bit about some of the current either commercial ideas or research that you guys saw and thought, Well, one, we need to write a paper on this. There's always something that you read that annoys you and thinks I need to do something. (all laugh) And then probably even more that came on along the way as you were writing this. Because it's a great overview, I have to say, if you're ever interested of like, Does this really occur? And you both do a very good job of cataloging, it does occur, and it's prevalent. You know what I mean? Like, this is very popular.

Jevan Hutson  8:25
Indeed. And in many ways, that's what inspired the paper was the sort of every week, the sort of Twitter warfare of, Oh my God, why are we automating this again? And going down sort of a similar rabbit hole that in many ways we need to diagnose this so we can sort of stop the process. But in terms of some of these examples, we'll sort of start from, sort of two in the research context, which talk about sort of basically predicting sexuality from the human face as well as political orientation, which are manifestly unsound. But as we categorize in the paper, in terms of commercial applications, we look at sort of first and foremost sort of labor and employment, particularly the proliferation of automated video interviewing tools that purport to scan individuals' faces to determine, you know, particular characteristics, like whether they're adventurous or cultured or resourceful or intellectual, all based off, you know, maybe a 30-second clip. Which, you know, research since this paper has already shown that, you know, manipulating a background, like putting in a bookshelf, can manipulate these particular determinations. But it's particularly concerning when you see that Fortune 500 companies are leveraging these technologies for sort of entry-level interviews or sort of initial barriers to employment.

We also sort of break down other categories in, you know, like teaching and education where folks are attempting to sort of scan students' faces to gauge attention or participation in the classroom, as well as in instances like criminal justice, which sort of interplay with other uses of facial recognition where the goal is to sort of expand into other forms of facial analysis. We also talk about other commercial applications, particularly around like shoplifting, where it moves away from, say, identifying a particular individual who might be shoplifting to, say, predicting or attempting to assess the likelihood of an individual from shoplifting. And altogether sort of, not only are these technologies in and of themselves concerning, but what is so concerning is that they are proliferating across multiple domains that have significant impacts for individuals at an individual level, but also at like a broader socioeconomic level if, you know, an entire class of folks are subject to AI video interviewing tools that purport to, you know, to take out these particular characteristics. Like, that might disparately impact particular racial groups, particular ethnic groups, persons with particular disabilities. And sort of for us, we were talking about this yesterday, sort of the next goal now is, maybe we really need a full count of the ways in which these exist in the wild.

Kirsten Martin  10:52
I saw that, yeah.

Jevan Hutson  10:53
But I think for us is at least beginning to sketch that these play out across important domains that also implicate, you know, civil rights laws, other important laws. Because many of these are public accommodations based in, like, the public where people have access but also mediate opportunities like jobs and, you know, other important social and economic opportunities.

Kirsten Martin  11:14
Right. Yeah, I saw because, like, employment, policing, education. You know, there is commercial, but I think it's also interesting for anyone listening, if you're teaching in this area, the HireVue emotion recognition case you cite by Drew Harwell in Washington Post, it's a great case just to teach. And I find the students are extremely upset about it. And they really do a great job at that age, say 20 to 27 if I'm teaching the master's students, to really, they're really quite good at identifying some of the more individual harms--not always societal harms, but a lot of the individual harms.

Jevan Hutson  11:48
I think, as well, I mean, students over the pandemic and previously being subject to, you know, automated proctoring tools that are attempting to, you know, scan faces to determine whether someone is suspicious, whether they're cheating, all sort of fall under, you know, sort of the penumbra of physiognomic artificial intelligence when we're attempting to scan the human body and make these determinations that we simply can't make using artificial intelligence and machine learning and that ultimately, I think students bear the brunt of. They witness how bluntly these systems don't work, how the failures of these systems have, like, abject consequences for folks, whether it's failing an exam, or the folks who've been trying to take the bar exam over the past three years and the nightmares of trying to deal with those particular facial analysis systems. But I think students, particularly students who have gone through the pandemic, sort of feel the brunt and impact of these technologies.

Kirsten Martin  12:40
Yeah. And I do also, this is, I mean, there is an immediate harm: You don't get a job, you're misidentified in some way. You also, before you get to the possible remedies--including banning it--but the possible remedies, you do a nice job also of talking about the harms. Like the, I'm putting "harms" in, like, a lowercase letter, harms. Like so, just a broad idea of the wrongs of using computer vision to guess something about someone and all that encompasses. Because it's more than just not getting the job. It can be an insult to the dignity in a different way.

Luke Stark  13:16
Yeah. So one thing, one thing that's really interesting about this conversation, and actually, when I talk about this or I give interviews about this topic, often what people say to me is they say, Well, humans can make inferences about people, we do that all the time, so why can't the computer? Right? And this is actually something that one of the famous papers, sort of the kind of famous physiognomic AI papers, the one about sexual orientation, they say this in the paper, which we talk about. You know, they sort of say, Well, humans can do this, and so, you know, of course, if computers do it at scale, they'll do it better than humans. (Kirsten laughs) And this is, this is like a, I think from a kind of philosophy of science, history of science perspective, this, like, rests on a kind of fundamental misreading or misunderstanding of two kinds of different epistemological modes, right? That conjecture about the past is always backward-looking, right? It always involves--if it's subjective, it's going to involve some kind of positionality, some kind of bias, right? And it's not, you know, so you can't make these kinds of claims, you know, across many data points and going from a kind of high level percentage back to any one individual, right? That's called ecological fallacy in statistics.

And of course, at a kind of common sense level, we can think about our own lives. We all know that we make inferences not always so well about people, and even not so well about people we know and love, right? You know, much great art and literature is all about the misconceptions and, you know, problematic inferences that, you know, a spouse makes about another, that children make about parents. Like, we know, we're not good at this, even with the people we know really well and esteem really well. And so this kind of inference-making, I think just, you know, just doesn't, it doesn't work in the same way because computers can't put together the kind of longitudinal data they often hold on individuals with the sorts of holistic judgments or kind of potential misjudgments that humans can do easily, right? It's just two different epistemological realms. And so anybody claiming that computers can do that the way humans can is selling you, you know, a little snake oil. And so yeah, and so when you build these big apparatuses of judgment, of automated judgment or automated decision-making, on this core erroneous assumption, this wrong assumption, then the whole edifice just crumbles, right? It's built on sand. So I think that's one of the things we're trying to really get at in this paper, which is it's not just that use cases can be a problem. It's not just that, like, there are technical issues around precision, you know, in terms of, you know, there's been lots of work on that in the context of AI fairness. It's that conceptually, these technologies just are not appropriate. They just don't do the thing they say that they claim to do. And you know, and as such, they're just kind of conceptually bankrupt.

Kirsten Martin  15:58
Right. And I like your point in the paper that because we think we can do the same, that we can guess someone's gender, race, we can guess if they're trustworthy. If I'm a retailer, I think I can guess who's shoplifting, who's untrustworthy. That we think we can judge others,' you say characters, psychological states, and demographic traits, it makes it attractive, we think that the AI should be able to do the same thing. It kind of explains the attraction of being like, Oh, I knew I could do that. It reminded me actually of predictive policing, that police officers think that they know where the crime occurs, and who's going to commit the crime. And so why couldn't you automate that? I mean, they do it all the time. You know what I mean? Like, and it's the same type of like, Well, I can guess who's trustworthy, I can guess what gender someone is, I can guess their race, ethnicity, and so I think it's actually possible. Even though you're pointing out and do a great job in the paper to say, This has always been pseudoscience. Like, the entire endeavor has been pseudoscience.

Luke Stark  16:54
Yeah, a police officer might think that, but they're wrong. They probably can't.

Kirsten Martin  16:59
(laughs) No, right, yeah. Right.

Luke Stark  17:01
Or anybody. I mean, I think it's really telling, I think it's really telling that the figure we would think of in terms of, like, this inferential mode is Sherlock Holmes, right? A detective--we don't mention this in the paper, but there's some other stuff I've been working on, this comes up. You know, Sherlock Holmes, the detective, detective stories really prime us to believe that some special individual can infer things with, you know, uncanny accuracy. And of course, Sherlock Holmes is fictional. Of course every fictional, every fictional detective from Conan Doyle to Agatha Christie to Jo Nesbo, of course they can do a good job inferring. The narrative that they've been placed in means that they can do a good job. But humans can't do that, right? We don't live in a narrative like that. We create narratives in our brains and socially, but we don't do it. It's not the same as a novel.

The other thing, and this isn't in the paper, but Jevan, I was thinking about this the other day, is that when we interact with people, it's not a one-way street, right? So when we're interacting with people we know or getting to know or talking to, both sides are sending out inferential signals in ways that are making it easier for the other to, you know, to pick up on cues, whether they're emotional or social or whatever, right? Often in ways we're not wholly conscious of. But it's not like, you know, I'm sitting there kind of scanning somebody who's just going about their business. I'm getting a lot more information if the other person is actively trying to engage communicatively. And I think that's, that's another point for why these systems just are not, you know, just are not good at what they do.

Jevan Hutson  18:35
I'd add a note there that don't take that as impetus for further engagement with physiognomic AI systems. (laughs)

Luke Stark  18:40
Yeah, no, no, no indeed not. No but I mean, but again, that kind of engagement, right? I mean, one of the interesting things about all of this inferences, right, is that it does then shape people's attempts to do--you know, create more interactive digital systems, and that's a whole other set of problems and a whole other mess.

Kirsten Martin  19:00 
So the design, the fact that we need to feed the beast, will actually make a design for more opportunities for people to interact facially with--or walking or whatever--with computer vision because they need to get more data. If the answer is it's not accurate, then they might think, I just need more data. But there's more wrong. I mean, one is that more data is not going to fix this problem. That's not, the issue isn't seeing more people.

Luke Stark  19:24
Exactly.

Kirsten Martin  19:24   
It's just that it actually doesn't work. I think the other thing that comes out is that there are certain guesses about someone that we want to--at least in the United States, and I think this is the same way with the census in Canada--that self-identification is like a crucial part of it. So you're able to self-identify your race, ethnicity, gender, sexuality-- that's the, we have a strong instinct in that way, even when it's not legally required, that it feels offensive if someone's guessing that about you. And so, that always comes out when I teach this in class; you'll have people who have been misidentified in any of the categories really get upset at this idea that computer vision is guessing, even if it's just for marketing purposes. The idea that a database has scripted them as being Hispanic or Caucasian/white when they're the other, you know, especially someone who is Hispanic who actually is being written as white. They're like, I--you know, it is a sensitive subject for them. And so they get very upset about it, which I think is, it's an interesting harm that is not easy to point to. It's just a dignity harm.

Luke Stark  20:31
Yeah. And I think, I mean, one of the things that I take as really fundamental--and I get this from science and technology studies, and I get this especially from people like [first name inaudible] Gandhi and Susan Leigh Star and Geoff Bowker's work, right, is that categories are always political, right? The census is a great example. The census has been political since the census was founded, right? 

Kirsten Martin  20:48
Yes, right. 

Luke Stark  20:49
You know, authority, you know, government authorities make, putting you into categories and you having to navigate which category you--you know, even if you self-identify, you know, navigating which of the kind what Lisa Nakamura calls menu-driven identities, which to plug yourself into. That's always about power, it's always about how those categories get decided, who has to tick something in the "Other" box and write something in, right? So yeah, so now we've just got this on, on steroids. We've got menu-driven identities all over the place, and inferences about those menu-driven identities, which are in some ways doubly, I think, doubly troubling. They're troubling because they make an inference about the identity. And they're troubling because they then, you know, they assume that there's a set of kind of concretized identities that one could be put into, right, even if they make the wrong choice. So yeah, that's a bigger problem than just physiognomic AI. That's a bigger problem with all sorts of ways that we track, you know, we collect data about people.

Kirsten Martin  21:43
Right, it doesn't have to be through, you're saying it doesn't have to be through computer vision that you're guessing someone's gender. It's by other things, it could be their purchasing behavior or whatever it might be. Yeah.

Luke Stark  21:53
Exactly. Yeah, and I think one of the things that I'm increasingly convinced about with AI is that, you know, AI is only the latest, it's only the latest symptom of this bigger question about classification. And, you know, that has really been the hallmark of 100, 150 years of modern bureaucracies. Which makes it, like, feel like a big problem. But. (Luke and Kirsten laugh) But it is. I mean, the fact that physiognomy and phrenology were big 100 years ago, and then for various reasons, they dropped out of conversation, although they were still kind of there. And now they're back in the conversation. I think that just, that's a testament to having to think about the kind of normative and value, you know, value bases of a lot of classifying techniques and a lot of the kind of numerization that we base our societies on.

Kirsten Martin  22:41
Right, right right right. This is an interesting thing to talk about just as one example of a larger problem that's going on with this classification. Who decides the classifier? How can we quantify someone within that classification scheme? How is that menu-driven classification working, and who decides it? And the census is an interesting example in a larger scale because it's obviously politicized because it's literally political institutions that are doing it. (laughs) But this is just powerful corporations that are doing it and having classifications that work for them. Otherwise they're not going to use them. I mean, they're not going to necessarily use it if it doesn't work for them.

(voiceover) That does it for part 1. In part 2, I’ll talk with Luke and Jevan about the menu of regulatory options they propose in the paper to remedy the fundamental problems with physiognomic AI. I hope you’ll join us.

TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

7. When Privacy is a Facade for Data Extraction (August 10, 2022)

Guest: Ari Waldman (Northeastern University)

Transcript

Kirsten Martin  0:03  
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we talk about an important idea, paper, article, book, discovery in tech ethics. And today, I'm so happy to be joined by Ari Waldman. Ari is a leading authority on law, technology, and society and is a professor of law and computer science at Northeastern University. He directs the School of Law's Center for Law, Information, and Creativity and studies how law and technology affect marginalized populations, with particular focus on privacy, misinformation, and the LGBTQ community. Ari has both a Ph.D. and a law degree, which I really want to emphasize only because it really shows up in this book that we're going to focus on. So today, we're going to take a deeper dive into the book Industry Unbound: The Inside Story of Privacy, Data, and Corporate Power. So what I loved about this book was that you weave in on-the-ground interviews with the research background that someone in this space should know about. And as I was reading this, I thought, so if you're a researcher interested in privacy, you covered these major tensions, both in theory and in law, that someone really should be aware of before they go into this space, and cite all the relevant people. If you're a practitioner, you can see how your organization, though, manifests or puts into action these current tensions that are going on within theory and research. And so I just want to say that as, like, if we're thinking about this space, I thought it was a really great way that you navigated, like ran down that pole, this very narrow thread that you could actually go through. Which I know is extremely difficult to make something, the writing, accessible for like what we would call a normal person or just a regular practitioner, but also with the substance and the depth of someone in academia. So I just want to applaud you for that. I thought it was great.

Ari Waldman  1:49  
Well, thank you so much. Thank you so much for having me and for doing this. I really appreciate the opportunity to talk about not only the book, but also talk about it with you because you're such an expert in this field, as well, and we have so many overlapping interests in this area. And particularly, I'm very flattered by what you say about bridging the two areas of research. So that's what my goal was. And I think that privacy scholarship in general, and I have to say most scholarship in general, is better when we can bring in different perspectives. This book in particular benefited from not only training and qualitative methods and how to learn or extract information from observing people doing their jobs, but also from connecting what people do with their jobs to ongoing theories and approaches and understandings about how organizations work. But also the relationship between law and organizations, and the relationship between theory and practice, right? These are things that some law professors, they will all--they'll write in one area, and then other people will write in an entire other area. My goal was to bring those two together.

Kirsten Martin  3:03  
Yeah, and I love it because I--so the opening chapter has these series of interviews, which were I'm sure, like, and I would love to get your take on what kind of surprised you the most. So you get these series of interviews, which are really--they're short, like, so the excerpts are short, but they're so rich with, like, fascinating tensions that you kind of pull the thread throughout. The bro meeting, the claim that someone else does that. 

Ari Waldman  3:25  
(laughs) [The bro meeting] was one my favorities.

Kirsten Martin  3:26  
Yeah, I know, and they call it a "bro meeting." That's not you, like, infusing that into it. It was, like, they call it the "bro meeting." The "someone else does that" idea, the idea that they did privacy when they were an undergrad intern. (Ari laughs) The focus on encryption. And of course, like, this pervasive theme of, like, privacy is important, but then quote, "I have no idea what it means." And I just want to just first ask, was there anything in those interviews that surprised you? Like, was there one thing that kind of you were like, Oh, my gosh, I can't believe this is going on.

Ari Waldman  3:56  
(laughs) So, so many things. So you're right, those are excerpts of some interviews or observations that happened over a really long period of time. It's retold almost like it's a day-in-the-life-of, and I intended it in part, as a law professor, I was thinking, Oh, this will be a really good issue-spotter. So for people who are in the field, you can see what's going on and you can check off, Oh, here's a thing or here's something or here's something to guard against. So I think that the one thing that I couldn't stop laughing at for a really long time, even though it's not really that funny, was the bro team thing, right? So this was a room that--so in these kind of tech offices, there's open plans, and then you can reserve a conference room for a meeting, and you would put a little sign up or the the computer would have a sign. It's like from 10am to 12pm, this is such and such meeting. And the name of their meeting was the bro team meeting. (Kirsten laughs) And that was like--which meant that that was the official name of this meeting because someone's assistant put it into a system, and then it fed onto the computer screen. And then--because obviously it was only dudes in this meeting. That I think was the funniest thing. 

But what was most amazing, I think were two--most amazing from a sociological perspective were two things. One, the persistent belief that, Well, this is just what we do. Like, that there is a definition or a conception, and that is the limit of what privacy work is. And then the second--and I'll talk a little bit about that in a second--and then the second fascinating thing was the level of false consciousness. So I'll start with the second one. So I remember having a meeting with a group of privacy professionals at a mid-sized tech company. So most of the book, I would say a good chunk of the book is based on an ethnography of three technology companies, in addition to many more interviews with current and former employees of the, like, really big ones. And one of these meetings, they were telling me all about how much work that they've been doing, Look at our work product, oh my God, we've been so busy in the run-up to the GDPR and after the GDPR, we've been so busy. And they even showed me stacks and stacks of papers in this conference room--like physical paper, I didn't even expect to see, like, physical paper anymore--but physical paper that included impact assessments and new policies and all this kind of stuff. And then when I asked afterwards, Okay, that's amazing, that's great, what kind of impact has all of this work had? And their answers were, Well, look at the privacy policy that we changed. And I said, I get that, but what about, what kind of impact are you having on, like, engineers understanding privacy? Or better yet, on the products that you create? What kind of impact are you having? [They said,] I don't know what you're asking.

Kirsten Martin  6:59  
Right.

Ari Waldman  6:59  
There was this lack of understanding that their work is not just work; it's supposed to have impact as opposed to just, like, taking up space, right? So the inability to see that all of this paperwork was not having a material impact on the privacy defaults or the work that's being done with respect to privacy and designing new products was bananas. Another crazy thing was, when I would talk to people, there would be these strange--strange examples of dissonance. So, like, for example, I'm in a meeting with a group of lawyers. This was a little bit of a larger company, large enough that they're involved in quite a bit of litigation when it comes to privacy. And they run a platform. And the general counsel made a very clear statement in our meeting that they believe that people retain their privacy in material that they may disclose on the platform, especially if they use their settings and they actively take control of their privacy. And after a very long speech and all this information that they were giving me about how people still have privacy on their platform, I said, That's great; how do you explain this brief, which I gave them, where you specifically say--I mean, you're the author of this brief--you said that once people disclose information on their platform, they lose their privacy interest. And the general counsel said, Well, nothing prevents me from being the most zealous advocate for my client's interests. And then he stormed out of the room in a huff. Like, this was a childish reaction. This is a grown man, like, this is a childish reaction to someone challenging you and leaving the rest of his staff to clean up his mess. And he stormed out in a huff screaming obscenities, right?

Kirsten Martin  8:52  
Oh my God.

Ari Waldman  8:53  
You get those kinds of reactions as well as I got reactions of people who were really--they really believed that their privacy team was doing such good work or that their machine learning team had integrated privacy. But again, like the other example I told, like, they couldn't point to a specific material example that showed that privacy was somehow better represented or better protected in the products that they created.

Kirsten Martin  9:21  
Right. And I think, and this is where I thought it was really interesting how when you open up the piece on discourse, and you can see how the idea of discourse as power--that you say, Tech companies inculcate definitions of privacy that are so narrow and corporate-friendly that they end up serving as facades for data extraction, which I thought really encapsulated the problem really well. And then you can see, as you said, like how that kind of filters down--both you can see how it, you show how it manifests into the law, which is interesting. But then you can also see how it shows up in the organization, and this focus on notice and choice, which we all know happens in theory, but usually privacy scholars have kind of--the ones that we hang out with (Ari laughs)--have moved on from that, you know? So I shouldn't say all privacy scholars because I still run into it. But there's, the back and forth that you have in there are so, so common about the focus on notice as being this panacea for respecting people's privacy. And it's so pervasive. And really, once you realize that that's their focus and the discourse around notice, it gives them this facade, but it also means why would you talk to the engineering team? They have nothing to do with notice. It kinda gives them the excuse as to what do they have to do with that, other than telling me what I need to provide in the notification so I don't commit fraud? That's such a bare minimum.

Ari Waldman  10:44  
That's a really good point. And you're right that--I think what the real impact or work of this book is, is to show about the lack of trickle down, or how--sometimes the lack of privacy discourse trickling down--but how pro-industry or pro-data-extractive discourses do trickle down. And you know, sometimes it's in the mundane, right? If you are, if you're a junior attorney or a junior privacy professional, your boss is the one that's giving you assignments, and your boss decides that, Well, your job is going to be to write a transparency report, or your job is going to write a brief on this transparency issue, or that you're going to assign a team to focus on how can we be more transparent? Or how can we make this clearer, right? So sometimes, it's very mundane; we all get assignments from our bosses. But the work that's done has a performative or normalizing effect, in that, you can come in to a job like this really believing and truly believing that you're going to change things, you can see yourself as a privacy advocate. And then even if you have a robust view of privacy coming in, you are constrained by how the organization works, by how you get assignments, by the work that you're given, by the opportunities, by your budgets, right? So I spoke to privacy professionals, like chief privacy officers, whose entire budgets are split between law, IT, and the compliance department. And you don't need to be an expert on organizational budgeting to recognize that that's not good for the privacy department, right? Not that it's automatic that the lawyer or the compliance person or the IT guy, it's not that they're automatically anti-privacy voices. They may be, but they don't need to be for that structure to hamper or hinder the privacy office, right?

So the work that people do have a normalizing effect, such that I interviewed people--this was, I think, a really good example--someone who puts on LinkedIn, saying that people hire me when they care about privacy, right? And that's actually, that would be really good if we had people like that, right? (Kirsten laughs) People who are experts in, you know, in engineering privacy, people who are experts at integrating privacy into every element of a company, that would be really good. But then when I talked to this person and asked about the actual work that they do in a company once they hired them--luckily, I met this person right after they did a three-year stint at one company and then were moving onto another. The world of work product were things like trainings, privacy policies, impact assessment templates, which, like, are checkboxes. And then more, like, and then describe their work as almost like a consultancy: I had an open-door policy, people would come to me, and I would explain and we would go over these things. Well, what about something more affirmative? Or what about, you know, changing defaults? Or did you get involved in design meetings; [they were like,] Oh, no, that's not what we do, right?

Kirsten Martin  13:53  
That's right, yeah.

Ari Waldman  13:54  
So even people who go so far as to put on LinkedIn that people hire them when they care about privacy has such a limited, constrained vision of what their job and what privacy is. And that comes from years of being in an environment where you're only given a small world of things to do.

Kirsten Martin  14:14  
Yeah, and that type of--the focus, keeping the law and the privacy people on the outside, and them really saying things like, That is not my job, like, I don't go into those meetings. Or if I talk too much, then they're not going to invite me back--like I can't, they're just going to ice me out and they're not going to include me in these meetings if I'm too negative or if I start talking too much, so I really have to watch what I say. And it really, and you bring this up, it really reminded me of issues around discrimination that we see at universities--like, the chief discrimination or diversity officer and where that person is positioned, how much is that operationalized into, like, what faculty and chairs do, versus is it just, like, somebody without a budget? And those are things that we look to within universities to be like, Oh, how serious are they about this? And it really reminded me of that same idea, where if you have just one person centralized or an office centralized, and you're like, Okay, everybody worry about privacy. Like, it just doesn't, it's the same thing as having a chief diversity officer but then never training the hiring manager as to how to hire.

Ari Waldman  15:17  
Exactly right.

Kirsten Martin  15:18  
Or not having any policies. And so it really reminded me of that same--and also environmentalism. Like so, back in the old days, in fact the exact same tensions around, first of all, it's anti-innovation. So the exact same argument that, like, If you put this into place, EPA, there is no way we can make cars or steel, that we are the dominant force in the industry, we will go out of business, like, it is impossible for us to survive. And so that same, Well, what do you expect us to go out of business? Like, you got those comments from people when asked about privacy. Like, What, do you think we're supposed to not be profitable? I mean, that's the only way, we can't take those things into consideration. And you just saw these same old tensions, you know, of like anti-innovation, you know, that's for someone else to decide. The good news is we got over those things with environmental issues and discrimination, it's just a matter of kind of, like, how long is it going to take them to actually absorb this?

Ari Waldman  16:13  
That's true, I mean, you're right to spot a lot of this in other industries. And I'm very clear about that, like, these things are not unique to the privacy world.

Kirsten Martin  16:22  
Right.

Ari Waldman  16:23  
It's just that many scholars and many practitioners don't think it's happening in privacy. Like, there is a whole book written in this area that says that chief privacy officers are actually doing the opposite, that they are actually filling the void left by privacy law with more aggressive, going over and above what the law requires.

Kirsten Martin  16:44  
Yeah, that's not happening. (laughs)

Ari Waldman  16:46  
Even assuming that that's true, right--like the book that I wrote, the scholarship that I've been doing, and the research that I've been doing inside these companies can assume that that is true, the chief privacy officers are actually doing this. But the point is, chief privacy officers do not design anything, right? They don't do the on-the-ground work of lawyering and compliance as well as design. That means that if they are doing that kind of stuff, why isn't it trickling down, right? So, and that kind of stuff is still happening: the overemphasis on compliance for compliance's sake, the weaponization of compliance to achieve industry goals as opposed to non-discrimination goals, the role of industry in kind of using law to immunize themselves. Like, all of these things exist elsewhere. There's great research as well as book done by Laurie Edelman at Berkeley, her book in 2016 called Working Law, won the Law & Society Association's book prize, it talked about what I talk about in the book: legal endogeneities, her theory about the way in which law develops endogenously from the ground up but also is weaponized against non-discrimination. Because her book is about these diversity offices and how--

Kirsten Martin  18:00  
Oh, interesting, yeah.

Ari Waldman  18:00  
Instead of, you know, Title VII requires equality or non-discrimination on the basis of sex when it comes to hiring, but how companies have interpreted that is to, Let's create a policy, let's create an appeals process, let's create a DEI office. And because those offices, those internal structures, have been legitimized as in fact what the law requires as opposed to actual equality, then it's been one of the reasons why equality under Title VII hasn't worked out so well. And it's the same, it's a similar story here. In fact, the legitimization of these internal policy, internal structures like impact assessments and compliance and record-keeping and offices, not only have those been legitimized ex-post by the law, they have actually been printed black-and-white in the law itself. Like, that's what the GDPR is, right? The GDPR is a series of--sorry, the General Data Protection Regulation in Europe, which everyone talks about as the strictest privacy law in the world--is just a series of individual rights and compliance obligations. And as I write in a California Law Review article coming out soon, those compliance obligations are not innovations of regulators; those are the same compliance innovations, so compliance processes, that companies have been engaging in for at least 15 years, right? So the law has been endogenously created by the very processes that companies have used to try to legitimize their data extraction without actually undermining their business model.

Kirsten Martin  19:35  
Right. So because the idea is that by focusing on process, by focusing on we need to have these impact assessments or appeal processes or ways that people can ask questions and they have rights of asking what data they have, that doesn't do anything to stop the flow of the data; it just is, like, a wrapping around it that we have to include. And so that's why you have great quotes from the lawyers that are kind of like Oh, great, you know, another privacy law, like, I've got more business now. (Ari laughs). You know, the idea is that, that there's just, like, this idea of compliance through processes, of those papers that are produced to kind of prove that they did the process then to report that out. But it doesn't really have to change the underlying business model because that can just keep on going. But they can have this veneer of saying, Oh, look at my privacy policy, I just updated it last week, and we did it based on all these things. And so they have these kind of, like, mannequins of privacy, you know, out there kind of pretending to do it. And yet, it's not really the underlying decisions aren't even changed at a grand scale because they're still big data extraction industries. But then even as you show in the book at the smallest scale, in that they never go and talk to the engineers or the computer scientists that are actually doing the actual design. 

Ari Waldman  20:49  
That's exactly right.

Kirsten Martin  20:50  
And even later on in the book, where you have people that were hired as computer scientists in privacy--so there were a couple of people who were told, We want you to be in on those meetings. It didn't always come to be, you know what I mean? Like, so they were, you know, or they came in later after design, and they're like, Well, I understand what you did, but it'd be so expensive to change it now. Which is exactly why we all focus on design because that's where the magic happens. Like, the magic is in design. It's where you have the most play with what's going on, and things get harder to change as you go along. But that's not exactly what they're focusing on.

Ari Waldman  21:24  
Right. That's exactly right. So one of the key characteristics of organizational structures that people in power use to keep the workers weak--this is not just true in a big tech company, it's true in any organization, like you can go as far down as, like, a factory--is you keep teams separate. And there are some totally valid efficiency and productivity reasons why you might want to do that. There's a ton of business literature on that--like, small teams are more efficient. And also engineers in particular are a class of persons that, you know, love to focus on this one cool and really intense problem, and they'll, with small teams, they'll go at it. And they'll become territorial about it. So what companies do is they create these barriers of information flow within their organization.

I'll give you three very quick examples. One example is what you mentioned about the privacy engineers. These companies make a lot of hay about, Oh, we're hiring all these privacy engineers to really integrate privacy and design. But they're a separate team, and that separate team is not just isolated from the other teams, but also their work is more auditing at the end of the process. And as you know, like, no one, even if it's possible to go through millions of lines of code for a product, no one at the end of the process is going to be in a position to say, Stop the presses, we're not releasing this product until you fix all of these things. Another example is that there are companies out there that have specific hierarchy reporting rules--we all have reporting structures in all organizations in which we work. There are companies out there that specifically say that a lawyer should not, cannot be in a design meeting, and that when something from the designers has a question, or maybe it's a question about privacy or maybe it's a question about law, it has to trickle up, it has to go through the manager and then the manager and then the product manager, and then product manager can meet with the lawyer. And you might see that, and that's how the organizational structure, that's how their reporting works, and they have to follow it. And you might say that, Well, that's really bad. Like, why would anyone agree to that? Why would a lawyer who--not saying that lawyers are always good (laughs)--but obviously, why would a lawyer agree to do something like that if their job is to represent these people and give them advice? Well, because the most insidious thing about all of these organizational structures is not that they're just imposed from the top; they actually reflect ongoing, whether it's insecurities or egos or the personality and social structures of the people that they're organizing. So lawyers--as a law professor, I know, like, I teach a privacy course where we talk about but need no background expertise in computer science and how computers work. And yet a lot--I always have a lot of students who say, I went to law school because I don't want to do math, right? 

Kirsten Martin  24:16  
Right. Right, right, right. 

Ari Waldman  24:19  
So you have a lot of lawyers, and many of whom I spoke to said, I don't want to be in a design meeting, I don't know what they're doing, I'm just going to be in the way. And then the flip side of that is the engineer will say, I don't want a lawyer here, they're just going to slow me down. So this separation, it taps into these perceptions that people already have that then metastasize because they set the norm that this is what's appropriate. And so they feel--so when I when I would suggest, for example, Well, why don't, if you feel that you are better at spotting privacy issues as they come up in the design process, why aren't you there in the design meeting, and then the person said, I never even thought of that. Like, That's not what we do, like, that's not what lawyers do, lawyers aren't inside design meetings. And it's just baffling. And it follows again, and the last thing, and previously, what we talked about earlier, that there are these limitations. There's this narrow band of possibilities that describe everyone in this industry, such that when you propose something, it's not just that, Oh, that's interesting, I never thought of that. It's like, That's not possible, that's not this--that's not this world, that's something in the multiverse, right? Like, that's not, that's not even in this world. And that also suggests to me that the way this has been constructed is through normalization and performance. Which really takes us to, well, what do we do about it, is that, honestly, we really only need one company, one person, one state law that thinks about in a totally different way and starts a new norm.

Kirsten Martin  25:54  
Yes. Yeah. And you know, it's so funny, that happens so often. There's a guy Tom Donaldson that wrote about this in California Management Review about the Pelican Gambit, which is his version of, like, within a given industry, there are times that the bar just goes low, you know what I mean? And it kind of normalizes. And what you just need is someone to do it differently--one, you just need one to poke their head out and say, Oh, this is a different way of doing it, and put a stake in there. And then others are like, Wait, wait, wait, what's that? And then they, they kind of ask questions. They're like, Oh, that's interesting, like, oh, is that an advantage? You know what I mean? Like, and then they kind of, and then it kind of slowly comes out. But you can get these stabilizing forces within industry because they're such copiers, you know I mean? And plus, they move back and forth, and there are reasons why, as you point out, that they're siloed like this because they just don't feel comfortable with the other person's space. And they don't like to think of themselves as doing that space. Like, if you're a computer scientist, you don't like to think about the law necessarily. (Ari laughs) Makes you feel uncomfortable. I mean, there's a lot of ambiguity and so they focus on that. That's the other quote that would come up a lot is, like, I can't--they would literally say, I cannot design for privacy because it's too ambiguous. And you're kind of like, Well, you're doing it whether you think about it or not. Like, so you're either doing it well or you're not. (Ari laughs) 

But it was, the opening chapter I just found, like, super interesting. But then, like, as you--you find it interesting just as a someone in this space because you can see all of these things manifesting themselves in these conversations. But then as you kind of pull them along, you do a great job of, like, identifying, you know, the slippage that we have around privacy versus innovation, notice and choice, we didn't even talk about this, but encryption. I mean, the way that they slip into encryption and security as what privacy is, it just--any time there was a back and forth, it either dissolved into notice, or any time there was a back and forth, it dissolved into encryption. Like, it either went to, like, one of those places--

Ari Waldman  27:50  
You don't even notice. 

Kirsten Martin  27:51  
Of like throw up the hands, and, like, this is my settling point on privacy, which is not privacy. (both laugh) But it was-- 

Ari Waldman  28:00  
It goes so quickly, sorry to interrupt, but that conflation goes so quickly, you blink and you don't even notice, right? Someone, a chief privacy officer will be so excited about all of the work that they're doing in the wake of, you know, data misuse, but all of it will be about encryption. And then you have this, Huh, wait a minute, you just listed all of these five things, but that was just, you know, making the database more secure; how about privacy stuff? And then they'll say, just like in repeating the mantra that we've talked about before, it's like, What do you mean? Like, That's not what we do here.

Kirsten Martin  28:33  
And what I love about the privacy versus security thing, which I always tell my students is, what it says is that you can't let someone steal the data. I mean, that's security. But you can sell it to them.

Ari Waldman  28:33  
Right. (laughs) 

Kirsten Martin  28:37  
Like the exact same people, you can sell it to them. But like, and it's just such a backward way of thinking about privacy as just not letting someone steal it. But then selling them the same data is okay; that doesn't make any sense. And so anyway, it's just, it's a fascinating--what I liked about the book was, again, [this] kind of grounding it very much in these ongoing conversations, seeing how the industry not only gets their discourse into the law, which we all talk about, but then to see the law kind of influencing the organizational structure and exactly how the discourse occurs in there, as well. You can kind of see and I'll just say like, one, this is a final quote, which I just thought wrapped up the book so well, which is, "Through a long campaign to inculcate corporate-friendly discourses about privacy, the information industry tilted our legal consciousness away from privacy and enlisted even those employees who see themselves as privacy advocates in their data-extractive missions." Which I thought really summarized the book--obviously you wrote it towards the end, so you're a good writer. (both laugh) But it really--like what I liked about it is it also talks about our legal consciousness, not just the laws but in the way that scholars talk about it. We both run into this when you run into--like, I was talking to a political scientist, who was just adamant that notice and choice was the only way that you could ever talk about privacy. And what's interesting, like, when you say something about, Well, isn't true choice when they actually know what's going on? And then they just stop. You know, because they know that they don't know what's going on, and so it's kind of like--but then they just want the conversation to go away. Like, there's no, like, alternative, it just kind of stops.

So I really, I can't wait to see kind of more that you do in this area. I love all of your work, obviously. But I really liked this book a lot. And I liked it both for practitioners and for academics who are interested in privacy, just because you really do a nice job of making even the research accessible because you write about it in, like, very common language that people can read. But then the citation is in the back of the book, which is like a gazillion pages long. (Ari laughs) Which is good because you can go back and see what is that article by Danielle Citron or whoever it is. But you can kind of get the idea from the book itself. But I wanted to ask, I always end with this, but is there anyone else in this area of tech ethics that you in particular--it can be from any discipline that you kind of think is, you know, interesting writing; I know it's really hard to pick just one but you have a ton in your book--but are there any new people coming up that you're really looking forward to hearing from?

Ari Waldman  31:03  
Yeah, absolutely. So let me split them into senior scholars, really important voices, and some new up-and-coming scholars. So I already mentioned Laurie Edelman and her work on legal endogeneity in the Title VII space. A lot of this book is really indebted to and I see it more as the practical application of properly understanding how industry works from Julie Cohen's book on informational capitalism. Her book is called Between Truth and Power--it came out in 2019--which is a really deep and expansive study about the political economy of informational capitalism. Some new names, up-and-coming scholars or people who have already arrived, even though they are so young. I'm thinking in particular of Salome Viljoen, who is about to start as a law professor at the University of Michigan. Her work, she published an article on democratic data in the Yale Law Journal, her work on the political economy of data is really second to none. There's also Alicia Solow-Niederman, who is just starting at the University of Iowa as a law professor. Her work, very interesting, is focused on ethics and machine learning and the crossover appeal and multidisciplinary areas there. I'll also mention work by people like Rashida Richardson. Rashida is now at the FTC and the Office of Technology Policy under Alondra Nelson, but she has some policy and scholarly work out there on data and discrimination and AI and discrimination.

And then also related to this, although it seems like it's a different field, I'll also mention two people who are doing work in the privacy and Fourth Amendment space. And the reason why it seems like a different field is because a lot of this book is about private companies that collect data. But as everyone knows, the government has a lot of access to that information, very easy access. So Kate Weisburd at GW is a really outstanding scholar focusing on the kind of social elements of government surveillance. She wrote a great article on electronic ankle monitors.

Kirsten Martin  33:14  
Oh, right, yes yes.

Ari Waldman  33:15  
And how, you know, data collectors and their intimate relationships with companies and also how they sort of deputize private people, like, even family members of the incarcerated or formerly incarcerated, into surveillance. And also one of the best Fourth Amendment scholars out there is Matt Tokson, Matthew Tokson at the University of Utah, who's doing extraordinary work about what the Fourth Amendment means these days, especially in an informational capitalism area, informational capitalistic space. And we, Matt and I have an article out in the Michigan Law Review specifically connecting our work from some of the stuff that's going on in this book with his work on the Fourth Amendment. So I think a lot of those--there's so many others, as well.

Kirsten Martin  33:58  
Yeah. So gosh, thank you so much. And we'll have you back when you have another article out because I always love to see your work. 

Ari Waldman  34:03  
Thank you so much for having me. This has been--I'm so grateful not just to you for inviting me, but also for the deep engagement that you showed with the book. I really appreciate that. It's always fun talking with you, especially about our work.

Kirsten Martin  34:15  
Aww, you're the best. Thanks so much, Ari.

Ari Waldman  34:17  
Thank you.

Kirsten Martin  34:18  
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

6. Lost in Translation: When Machines Learn Language (July 27, 2022)

Guest: Amandalynne Paullada (University of Washington)

Transcript

 

Kirsten Martin  0:03  
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we discuss an important idea, paper, article, discovery in tech ethics, and today, I'm so happy to be joined by Amandalynne Paullada. She's a postdoctoral fellow at the University of Washington Department of Biomedical Informatics and Medical Education. She also just completed her dissertation in computational linguistics in 2021, also at the University of Washington, where she studied the impact of natural language processing. In this series, our goal is just to take one article or idea and examine the larger implications. And for today, we're going to take a deeper dive into your recent article in The Gradient called "Machine Translation Shifts Power," which was a runner-up for the inaugural Gradient Prize. And I like this article for many reasons. The opening is scary, it has--the idea is that technology is being used to translate languages, which we know of--I mean, so we kind of are used to doing this with our Google or anything like that. But you provide a frightening example from DHS, where extreme vetting will also include using Google Translate to facilitate DHS being able to read social media posts, and then being used at the border. I thought maybe you could talk briefly about that example that you opened with, but also what I liked about this is that you put it into the historical context of how the practice of translation has long been shaped by power asymmetries--like, that it's been a tool in many ways. And so I think, the more that we automate this, the more we forget that this is, it's been a tool or a tactic that's been used. So if you could say a little bit about the opening and then some of the history.

Amandalynne Paullada  1:41  
Sure. So I'll also mention something kind of funny about that example, is that the extreme vetting that happens at the US border--I don't actually know if it's still happening--but where foreigners who wanted to gain entry into the US either for travel or for work or whatever, they were also vetting social media posts in English. And there was a British national who was denied entry because he had tweeted something about wanting to tear up or destroy the US, and this was slang saying that he wanted to party. So it's funny that even within English--

Kirsten Martin  2:13  
Oh, interesting.

Amandalynne Paullada  2:13  
Like, the same language that was being spoken, you know, predominantly in the US, like, there can still be these really grave misunderstandings, even when there isn't a language barrier. And so, yeah, I think that's just to say that the issue isn't just that we're deploying translation to kind of surveil people's casual social media posts, but even within the same language, there can be these grave misunderstandings. But yeah, I think part of what led me to write this article was reading about that and finding it sort of disturbing that, you know, you can be subject to having your social media scanned and potentially mistranslated or taken out of context in such a way that you could just be denied entry and have to go back and, yeah. What was the second part of your question?

Kirsten Martin  2:59  
Just about, like, there's also been this history of the use of machine translation to exert power. What I like about people that look at the history is it's not really the technology being used--[although] there's problems with the technology that we're using with machine translation--but this use of translation to exert power has existed for forever, as long as we've been translating.

Amandalynne Paullada  3:20  
Yeah, so I was interested in looking at when the first kind of machine translation efforts in the US were being undertaken, and a lot of it was to kind of be able to translate Russian military research for sort of--kind of gaining intel about, during the Cold War, what Russian scientists were working on. So I think just kind of, like, the very assumption that you can use technology to translate a language that you don't understand--there's sort of this, like, presumption that you should have access to some document, and if you don't understand it, you should be able to kind of reveal what it quote "really" says, without having to spend the time to learn the language or the cultural references or whatever. It's sort of this like--I guess in a sort of sinister and maybe dark way, it kind of reminds me of, there's this sort of trend of computer vision applications where it's like filling in a missing part of an image or, like, trying to generate what would this person look like without clothes on. So it's like something that you don't have access to, but you're trying to use technology to say, I would like access to this information or to generate something that would otherwise cost me some effort to figure out or to get access to.

Kirsten Martin  4:37  
Oh, interesting. Right. So this is, I mean, in some ways, you're saying that, if I understand correctly, that we could take the time to actually get to know the language and do it in a more thoughtful way. And then this is, in some ways, like filling in the gaps of an image, it's somewhat using machine translation to fill in the gaps of a language, which is also a culture.

Amandalynne Paullada  4:56  
Yeah. Like, I think for me, kind of like the dystopian future technology that really worries me is, like, everyone has, like, something--I mean, I guess this was in Hitchhiker's Guide to the Galaxy, the Babel Fish, where it's just like a little thing that sits in your ear, and you can just eavesdrop on any conversation in any language. And I think that really worries me because I think right off the bat, when you're eavesdropping on somebody, it's easy to take things out of context. But sometimes people code switch into a different language than the dominant one surrounding them for, like, safety reasons or just because they want to, you know--it's sort of a choice that you make to try to obscure what you're saying. And if someone has universal access to that, putting aside whether the translation is accurate or not, if someone believes that they can just eavesdrop on any language, I think that's pretty dangerous.

Kirsten Martin  5:42  
Interesting. Because you note that, I think in the past, that machine translation has, you say, exerted power over subordinate groups, those that don't necessarily control the technology, but are the subject of it. I think the term you use was linguistic suppression or forced translation, which is a little bit like this idea of forced translation of not having your own language to yourself, the idea of eavesdropping almost.

Amandalynne Paullada  6:07  
Yeah, I guess there, what I was referring to also was even before, you know, the concept of machine translation existed, colonial subjects were forced to learn the languages of, you know, whoever was colonizing them, and were punished sometimes--like, corporally punished--for speaking their native languages. And so there's a lot of--I mean, I guess this is sort of worldwide, a lot of indigenous languages were, like, really beaten out of people because they were forced to learn the language of the colonizers. And so there's a lot of, like, forgetting or a lot of language loss that has happened because of this sort of, like, We need to be able to manage you and understand you, so you have to speak our language. Yeah, I feel like that's sort of an abstract way of putting that. 

Kirsten Martin  6:55  
Yeah. Well, and I think you can see the power in a few different ways. So there's the, you know, the decision as to whose language is dominant; you mentioned that even the decision as to which language to prioritize is political, like the Russian example that you gave previously, where we were worried about the Soviet Union. And so we prioritize certain languages to learn those translations before others that might have been for other reasons and not for wartime reasons. And then this idea, which I had not thought about, that Google's history, that there are certain companies that based on just the resources and their assets like Google, that they just have this unbelievable advantage now to be the go-to place for translation because of the text that they have from decades of collection of it.

Amandalynne Paullada  7:38  
Yeah. And I think, like, I don't want to deny that it's useful to have free access to translation software. Like, I certainly use it quite a bit, and I'm sure a lot of people do. But yeah, there is this sort of--there was an article that I mentioned in my article--and I'm forgetting the author, I'm sorry--but talking about how this sort of automatic translation industry has, like, shifted away a lot of the industry towards these tech giants and away from people who are professional interpreters and translators. And I think--I know of some situations where it would be a lot more, I mean in really like high-stakes scenarios--where it would be nice to have a human interpreter to have a little more sensitivity to talk about a really sensitive topic. But instead, like, they're under-resourced or unable to find them.

And so there's this reliance on technology, which even if it's--I guess something that I have less expertise in is different theories of translation. And I think certain kinds of texts, like I don't think you can get a perfect word-to-word translation. Like, I don't think that the algorithms that do machine translation now can convey certain things, like how to convey something in a culturally sensitive way. Like, you can translate something word-for-word or sentence-for-sentence and get an approximation of, This is literally what is being said. But that doesn't take into account, I think, culturally relevant ways of communicating things. And I think my worry is that people will say, Oh, well, we can build that into the technology too, and I don't know if you can. Because it's like a really contextual and subtle thing. So I think making machine translation tools more adaptable to real situations in which they're being used, I think people are working on that. And I think that makes a lot more sense where, like, you could have someone who was a professional translator who has an assistant--or sorry, an automated assistant--to kind of help them partly do part of the job. And then they can do the rest of the, you know, the culturally relevant translation or the--like, keeping a person in the loop to make sure that things are being conveyed not just accurately, but also in a way that will, you know, be socially and culturally relevant.

Kirsten Martin  9:44  
Right. And we see this, as you point out from the beginning, that we see this even with English-to-English translation. So within the United States, you could have different cultures or different groups of people, you could have something that's a dialect within English and then have, like, the example you had from someone from England saying that they're going to tear up the United States, and all they meant was that they wanted to party, they didn't mean that they were a terrorist. And this idea can happen with content moderation along those lines. And you could see how it could be even worse with actual translation from a foreign language, where if we're having trouble English-to-English translation, that we might have trouble with French to English, but then even more so, like, a smaller language, which has a smaller footprint, so fewer people that speak it. And so this, you're saying this problem, we face this all over the place, and we see it even more so in this area of machine translation of foreign languages, which makes sense. I mean, since we're having trouble with English, you would imagine that we're going to have trouble with other languages, as well.

Amandalynne Paullada  10:44  
Yeah, I think there's also--this kind of reminds me of like a sort of concept from, like, literary critique of, like, paranoid reading. Like, I think a lot of these really high-stakes scenarios are because the person consuming the translation is a paranoid reader, like they want to find something. Like, I think when you're dealing with, like, the Department of Homeland Security, I think by off the bat, this is sort of going to be a paranoid reading of whatever action or text or whatever they interpret in someone who's just trying to enter the country.

Kirsten Martin  11:15  
Yeah, no, that's an excellent point. So it's different if I'm looking for a translation because I'm trying to find a restaurant, and I'm not a paranoid reader at that point; I'm trying to look something up. 

Amandalynne Paullada  11:25  
Yeah. 

Kirsten Martin  11:25  
But this idea of, that the reader actually has a disposition or an objective in trying to find something. So that's the point of the translation, is that they have some sort of goal or point of view, kind of a worldview that they're bringing to the situation. And so that would be different when you're looking at--even content moderation, it's the same thing, we're looking for problems. And if you're DHS, you're definitely looking for a problem. That's the entire reason why you're there. And that we have to understand in those high-stakes situations how these translations could be read. 

Amandalynne Paullada  11:58  
Yeah. 

Kirsten Martin  11:59  
And who bears the cost of a mistranslation, whether it's the developer or the actual user, the subject coming across the border, for that matter. 

Amandalynne Paullada  12:05  
Yeah, for sure. 

Kirsten Martin  12:06  
What I liked about, which I didn't understand previous to this--and this is because I'm not a linguist, so I wouldn't know this--but it was interesting to read about, one, the history as to how translation and machine translation in general is about politics and power: the language that's being chosen, which ones are being prioritized, how it's used to subordinate different groups, how the different ideas of, like, the different people that are even able to translate. So whether you have like unique knowledge or if you're Google, and you just have more text, that these are all issues of power, that they've always been going on. And so now we have this situation with machine translation. And [what] I also thought was interesting was this other issue, which you kind of touched upon with this idea of not being culturally sensitive. Could you say a little bit--you mentioned there was a lawsuit against Microsoft. Which I just thought was interesting because it says how someone who is a person of that language might think about their translation.

Amandalynne Paullada  13:06  
So that was an article--from, I guess, maybe 10 years ago or more--that was talking about how Microsoft released a version of Windows in Mapudungun, which is an indigenous language spoken in Chile. And there was, I think Microsoft had collaborated with the Chilean government, but not the actual, like, Mapuche tribe. And I think there's this sort of conflation or misunderstanding that, you know, like, indigenous groups worldwide are not necessarily represented by the state whose borders they live in. And so I think they, you know, got what they believed was consent from the Chilean government, but not from the actual tribe, if I'm remembering the details correctly. And so, yeah, I think if you're dealing with like a global north or Western entity, like, really your only recourse is to issue a lawsuit. Because--it's funny, I sort of wonder about these sort of Western legal tools like licensing or lawsuits, I mean, like, what other way can you get enforcement or attention on an issue? I'm still kind of thinking about this myself, like, what do you do in that situation to enforce, like, consent or a use that the people who speak the language find appropriate? I don't know. I'm still thinking about kind of ...

Kirsten Martin  14:26  
What tools to communicate. Yeah. I mean, sometimes I take lawsuits, when I read that I thought, to me what it illustrated was the offense that was taken at, one, the lack of consent to be translated. And then second, the kind of indignities of how the translation went. You know, and it made me think differently. So in that way to show that the people of a language would see this differently than the power that's saying, Oh, you're so lucky, we're going to translate you. Where they might think, one, you did it poorly, and you never asked. And so I think that that's what that lawsuit kind of spoke to is that when we're thinking about translation, there's a lot more going on than just, Oh, now the Western world can kind of translate your language. The people that are being translated might not think of it that way.

Amandalynne Paullada  15:18  
Yeah. I mean, I guess another thing I wonder about, too, is, like, who does speak for a language? Like, I could imagine that, you know, for languages that don't have a lot of speakers--I guess, like, different languages have different kind of governing bodies, and not all of them do have this. But yeah, even for languages with, like, hundreds of thousands of speakers, I sort of wonder, is there-- there's probably not even consensus among them, you know, what to do with the language or how to document or not document it or whatever? Yeah, I think that, too, is something I wonder about. Like, I think there's different, within different cultures and linguistic communities, there's probably also different ways of having consensus of, like, Yeah, how do we want to communicate our language and share it or not share it? I'm hoping to spend some more time also theorizing, like, what is a language to the people who speak it? And who gets to kind of set the terms of when a language gets to be used?

Kirsten Martin  16:15  
Yeah, no, I think what's interesting about it is the use of--one, I think what's interesting is that the use of machine translation just as its own, but then the general use of large language models and all the issues that come with it carry a lot of the same issues that you find here. I liked what you said, "Translation remains a political act, and data-driven machine translation developments, largely centered in industry, complicate the mechanisms by which translation shifts power." And we could say, I could have said that same thing and talked about large language models at the same time. And I think that that's what, in some ways, this one small issue around DHS--which wasn't small, I just mean, like, a news article--really, one instance of the use of machine translation really opened up, and there was a lot more going on. At one point, you mentioned that we're trying to, like, uncover what's made obscure. And you were talking about something else, but I just thought in general, what your article did was try to uncover what was obscured by just this idea of a machine translation tool, was that there's a lot more going on with power and politics. And in fact, those things going on with power and politics have actually been going on for decades in this space. And this is just the latest manifestation of that.

Amandalynne Paullada  17:23  
Yeah, yeah. In some ways, I almost felt like, when I was writing this article, I sort of struggled with, like, What am I saying that's new here? Like, this is the same thing for almost every technology. (laughs)

Kirsten Martin  17:30  
Right, yeah. But I think the people in the technology don't realize that, right? Like, so the people that are developing it all the time don't know that history, and so that we face this a lot. And so that's why it's helpful, I think, to say, We've actually done this before, and this is what we have to watch out for, we have to be thoughtful about these issues, because translation is not some sort of neutral act. It's incredibly political, and it's shifting power all the time. Which I just think is a good reminder when we're talking about, like, a quick search of your social media, which happens to be, even in English, there's a problem. 

Amandalynne Paullada  18:05  
Yeah. 

Kirsten Martin  18:05  
Well, great. So normally I ask people what made them write the story, but you obviously had a specific article, which was great, that was, like, the impetus for the article. But I wondered, in general, in this area, like who in tech ethics, who should we be paying attention to in the area of tech ethics? And are there any particular scholars that you're following right now? Or who should we be watching out for?

Amandalynne Paullada  18:27  
Yeah, so I wrote down a few names in preparation. So there's an article that just came out that I'm excited to read by Chelsea Barabas called "Refusal in Data Ethics." That's the main title, but there's a subtitle too. But yeah, I'm excited to read that one. I guess on this--I guess this is, like, less ethics-specific, but more kind of, like, around theorizing data and the current kind of big data moment. So there's a short-ish article by Ranjit Singh called--something about the decolonial turn, let me see if I can find it; oh, "The decolonial turn is on the road to contingency"--that's sort of, like, rethinking this thesis around data colonialism. And then the last one I'll say that I've been recommending to a lot of people lately is a talk by Danielle Carr that's on YouTube called "But is it Labor?" And this is also sort of talking through, like, different ways of thinking about data. Like, I think there's a lot of people trying to theorize, like, what is data that--especially in this moment, where, like, everything is digital, and we're just sort of constantly, by merely existing, just generating streams of data that could be worth something to somebody--like theorizing, are we doing labor when we do that? Or like, do we own this data, and in what sense do we own it? So yeah, this is something that I've been thinking about a lot lately and trying to understand a little bit more of and trying to retheorize. I think I've changed a lot of my thinking around, you know, data broadly construed in the last year.

Kirsten Martin  19:57  
Yeah, that's interesting. I have a friend of mine, Tae Wan Kim, who wrote about, and this is within a business school, but his argument was a little bit about users actually need to be thought of as actually investors. As in, they're investing, they're in some ways more invested in the company than anybody else because they've given their data over. And now it's being used in this way, similar to, like, capital by stockholders. And so he makes almost an argument that you have to be, you should be obligated to them in the same way that you feel obligated as a firm, as a company, to other people. 

Amandalynne Paullada  20:32  
Mm, interesting.

Kirsten Martin  20:32  
And I think it's useful to stop thinking of people as just users. And so labor or investors are ways of making them seem thicker than mere users that just drop off their data and leave because that's not what they're doing. And so I think that's a great point that we need to start theorizing about people and also their data in different ways. Because the way that we do it now is fairly abstract, both the person and the data. So that's a great point. Well, gosh, those are great recommendations. I really appreciate you taking the time. And I know you have a lot more to write in this area, so I look forward to seeing even more in the future. And so I'm sure we'll have you back with your next article or something along those lines. So with that, thank you very much. I really appreciate it.

Amandalynne Paullada  21:12 
Thank you so much for having me.

Kirsten Martin  21:15
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

 

5. Creative Speculation: Computer Science Taps Science Fiction (July 13, 2022)

Guest: Casey Fiesler (University of Colorado Boulder)

Transcript

 

Kirsten Martin  0:02  
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we discuss an important idea, paper, article, or discovery in tech ethics. And today, I'm really excited to be joined with Casey Fiesler. Casey is an assistant professor in the Department of Information Science as well as Computer Science, by courtesy, at the University of Colorado in Boulder in the College of Media, Communication and Information. She has additional affiliations with Silicon Valley Flatirons, or Silicon Flatirons, excuse me, at the Law School and the Atlas Institute. Her area of focus includes big data research ethics, ethics education, ethical speculation in technology design, technology empowerment for marginalized communities, and broadening participation in computing. We're fortunate to have Casey here--she has a Ph.D. and a law degree, which is always helpful when we're talking about tech ethics and policy--and so we'd like to take a deeper dive into your article about "Innovating Like an Optimist, Preparing Like a Pessimist: Ethical Speculation and the Legal Imagination." This really--I mentioned your Ph.D. and your law degree because this combines both in this area. So I think it was really interesting to think about how your legal background actually informed how you were thinking about teaching ethics to computer scientists, which it's always fun when, like, some weird part of your background all of a sudden pops up in a different area. And so this article starts with this idea of unanticipated consequences, which is kind of the bedrock of what we're dealing with right now, or claiming of unanticipated consequences. And I didn't know if you could say a little bit more about this and the difficulty of foresight and the challenge that it might create.

Casey Fiesler  1:41  
Absolutely. So a few years ago, there was a blog post written by the Future Computing Academy, which is part of ACM, and they were suggesting that maybe writing about the unintended consequences of research should be a required part of computer science publications. And this kind of idea has come up as well because NeurIPS, the AI conference, has started asking for broader impacts statements. And one of the criticisms that I often hear of these kinds of proposals is, Well, how are we supposed to see the future? (both laugh) How can you possibly expect a researcher or even someone building technology to imagine downstream consequences or the ways that bad actors might take advantage of or use what we're creating? And I think that that kind of speculation is actually not as difficult as people make it out to be. And I think this is in part because--I'm obviously not suggesting that anyone should be able to imagine all potential things that could go wrong and try to mitigate all of them. But I do think that we've learned enough from the past and have a sense of the kinds of things that can go wrong, to the point where some of that kind of anticipation of consequences should be something that we're able to do

Kirsten Martin  3:31  
What I liked about it, I'll just say in normal products, we in business--I'm in a business school, and so we have expectations that we don't hold, for example, Tide Pods responsible, Tide responsible, for when a whole bunch of teenagers decide to start eating them. But, you know, once they saw that, then they put out campaigns around it, they made sure that there were greater safety issues because they realized they were attractive to smaller children to possibly eat, you know. We have these issues with secondary or unintended consequences or uses in other areas, and we don't just throw up our hands and say, Oh, well, I guess we can't do anything about Sudafed and meth. You know, we get in there, and we put in either regulations or sometimes design decisions around how we could mitigate the unintended uses. That being said, I think you make a great point about how there might be reasons why it's a little bit more difficult or why this might occur with these unintended consequences. If I understood correctly, there is this where you borrow from Robert Merton's theory of unintended consequences about this inability to anticipate every eventuality, but also the kind of reliance on old methods of analysis, kind of having a tool and going back to it over-- even when things have changed, and you might need to think about a new tool.

Casey Fiesler  4:53  
Yeah, and this is one of the things that I think connects well to some of the challenges that we see in law, as well. Because the law as a discipline and a practice largely relies on precedent, and technology, as a general rule, moves much faster than the law does. And so something that frequently happens when we get new technologies is that we're trying to apply older laws to them or create new laws that look a lot like old ones. (laughs) So I think that this idea of, like, reactionary ethics or reactionary design or policy happens a lot, both because of this kind of slowness for things to catch up to technological advances, both with respect to not knowing what kinds of things that might go wrong and maybe not even having the tools, and maybe those are regulatory tools, to deal with those kinds of things right away.

Kirsten Martin  6:03  
So can--you have a solution. Well, it's kind of a solution. It's this idea of--I mean, it's not going to solve everything--but this idea of creative speculation. Both issue-spotting from legal analysis and then the idea of imagination as a possible tool for computer scientists to use in design. And I didn't know if you could say a little bit more about that, just for so many of us that teach people in data analytics or data science or computer science, about how we can help or think through how to give them tools to deal with unanticipated consequences, or think through as many as you can.

Casey Fiesler  6:41  
Yeah, so I definitely think this isn't a solution necessarily [so much] as one tool that we might consider, particularly in the educational process. So you mentioned issue-spotting, and this is a term from legal education. The way that many exams work in law school is that you get this very long fact pattern like, Here's, you know, two pages of something that happened. And then at the end, it just says, Please discuss all possible torts claims. (both laugh) Or something like that. And so one of the things that you learn to do in law school is take this very complicated situation and see the legal issues. And that's actually, that's not a natural thing. It is a skill that I think people can learn through practice. And of course, that's how the law often works in the real world. 

And so when I started teaching, particularly ethics-related classes, I started thinking about it partially in this way. Let's say you're designing a new technology; can you do that same sort of thing? Like, imagine the kinds of things that might happen or the kinds of issues you might see. I mean, this is the kind of thing that happens when, you know, someone has a startup, and they're building this new technology, and maybe they want to hire someone to, you know, be an ethics consultant. And they're like, Just tell me what the problems might be, what are the kinds of things that might be issues here? And one way that I think you can do that is looking at patterns from the past and things that often happen. And so this is why I use current events very heavily in my classes, not even just my ethics classes, but in general I think it's really important to be able to see how things connect to the real world.

But also this idea of a sort of creative speculation. And this doesn't mean turning all computer science majors into science fiction writers (Kirsten laughs), though I do think that ways to help technologists think a little more like science fiction writers might be good. Like, there's this quote from Isaac Asimov about how science fiction writers are good at foreseeing inevitable catastrophes, but though catastrophes are inevitable, solutions are not. And science fiction writers don't tend to be the ones tasked with coming up with solutions. (laughs) So how can we help the people who should be coming up with solutions think a little more about the inevitable catastrophes? And so one of the things that I'm trying to do in my research, and this is most recently supported by an NSF CAREER grant, is to think about creating tools for the technology design process for doing this kind of speculation. And also how that can work well in the classroom.

Kirsten Martin  9:53  
Oh, super interesting. That is interesting. I think that, now that you mentioned it, I used to call--I go to the Privacy Law Scholars Conference or any of the legal work around technology--and I call them the tip of the spear. Like, they tend to, they're good issue spotters, and so they tend to see the issues where everyone else is just saying, That just seems wrong. They're very good at identifying, you know, interests and rights that are at stake, you know? And usually with their old tools, so I'm not saying that it's perfect, but they are really good at issue-spotting. And it would be helpful to have that as a skill to engineers and computer scientists to think through issue-spotting in general, to almost red team their own design, you know, around ethical issues, just like they do with security. We're used to doing that with security, but they don't tend to think about it with their own design. They're optimists. But I like the way that you say that you can, I liked your ending when you say, "I am an optimist who uses pessimism to prepare"--which as someone who's called a pessimist sometimes, I just say I'm a realist that tries to prepare--and the "preparation is speculation for what the world could be, and how it could be better."

When we were talking before we started this, is that what I liked about this article was a lot of times when you're issue-spotting or critiquing or identifying issues that someone needs to address--unfairness issues, people being used as a mere means, anything along those lines--it can sound as if you're very negative about all technology, as if the technology itself should not exist at all. That there's no benefit to social networks or something along those lines. And yet, that's not, for the vast majority of people that are critiquing, that's not actually what they're saying. They're trying to figure out how to design better and make the technology better. So I appreciated that, you know, general idea of trying to help it with design versus getting rid of technology in general.

Casey Fiesler  11:50  
And I also think that, I think that if you are in any field, and you love your field of study or what you're doing, that it is your obligation to critque it because you should want it to be the best that it can possibly be. And I am very clearly not anti-technology. (laughs) I study what I study, and I think a lot about the harms of technology because I love it so much, and I want it to be better. And I think about this a lot, in particular with respect to things like the internet and social media and online communities, which is sort of my original area of study. And I talk a lot about social media ethics, and then people seem sort of surprised that I'm on Twitter and TikTok so much. 

Kirsten Martin  12:47  
(laughs) Right, right.

Casey Fiesler  12:47  
Like, Don't you think that these social media platforms are evil? No, I do not. If I thought they were evil, I wouldn't bother. I would be pushing for everything to shut down, not to try to make things better. And I also think it's really important that we consider--I think it's really important that we don't focus so much on the harms that we try to fix things in ways that break things that are good. (laughs)

Kirsten Martin  13:22  
Oh, good, yeah. 

Casey Fiesler  13:23  
And so for that reason, I also think it's really important to study the good things and the bad things about technology at the same time.

Kirsten Martin  13:30  
That's great. Yeah, I was just, a friend of mine at Michigan just wrote a paper with others on algorithmic reparations with this idea--she does a lot of work on the good side of, like, social media use, or in this case, trying to not just avoid unfairness or discrimination, but, like, how could you actually use the design to fix the issues that are in the world, so make it more of a positive impact. And I agree with you because just as we can do design decisions that are bad--you know, a bad recommendation algorithm--we could actually look at how to make it good, you know? So instead of having to avoid the bad. And so that's, it's the same design decisions or the same category of design decisions that we're in, and you can just look at it in two different ways. Yeah, I really, I just enjoyed this. I like the overall look of it--I mean, the overall approach that you had of, you know, being an optimist, but trying to think like a pessimist to make it better, and that it's okay to be an issue spotter and trying to think through what the problems are to make it better while liking the technology and using the technology. Because I agree with you. I think my students are amazed when I make references that my TikTok is very different than my daughter's TikTok. I think they're shocked that I know TikTok or that I, you know what I mean, that I'm following dogs all the time. You know, they're just surprised, given all the issues that we're spotting all the time and that you can still have the good parts of technology without taking all of the bad.

So I wanted to wrap up, I usually ask this question at the end, just to kind of highlight people that we might not be paying attention to. So if I didn't know if there was anybody that in particular that you're paying attention to in the area of tech ethics, just currently right now. It could be a doctoral student or someone that you are following along, always looking forward to what they write about it, you know, it's always different or something like that. If there's anyone that you recommend. 

Casey Fiesler  15:18  
Something that I think I'd like to see even more attention paid to is the work that's happening in computing education and technology education in this area. And one thing that I've been thinking a lot about lately is how "ethics" is sort of shorthand for a constellation of things like responsibility and justice. And so some work that's happening in the computer science education field along these lines that I really like is, Amy Ko is doing a lot of work on social justice in computing education, and Nicki Washington is looking at it as cultural competency in computing education. And I think that those are both interesting ideas. And I have done a lot of work on sort of, you know, ethics, which again, I feel like it's this sort of umbrella shorthand term for thinking about how we're talking about the stuff in computing education. But sometimes I think the word that I'm actually looking for here is justice, which I think is really important as well. So those are two people who have been doing good work in that area.

Kirsten Martin  16:33  
That's great. Thank you. I agree with you, I think--I [was] an engineer undergrad, and I have, two of my kids went through engineering college, or one's in the middle of it. And I would say that I would love for the ethics training of engineers to be less of a sidenote in their senior-year design thesis and more of an integral part that they have touches a couple times. So I would, I would love to have it be a little bit more robust than it currently is. So I think that's great.

Casey Fiesler  17:02  
Also, for people who are interested in that as a general thing--like, ethics sort of integration--the Responsible Computer Science Challenge, which was a funded initiative, but there are a large number of, 18 or 20, universities in the US who are part of that who've all created various types of ethics-related curriculum to integrate into technical classes.

Kirsten Martin  17:23  
Oh, that's great. 

Casey Fiesler  17:24  
And so there's a lot of people working along these lines, which I also think is really important because I think it's important that we teach these things, not as just a specialization but as an integral part of what every technologist is doing.

Kirsten Martin  17:39  
Oh, I agree, I couldn't agree more. So I really appreciate you taking the time, and we'll always look forward to seeing your work in the future. But thank you very much. I appreciate you coming. 

Kirsten Martin  17:48
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

4. An Evolutionary Case for Better Privacy Regulations (June 22, 2022)

Guest: Laura Brandimarte (University of Arizona)

Transcript

 

Kirsten Martin  00:02
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we discuss an important idea, paper, article, or discovery in tech ethics. And today, I'm so happy to be joined by Laura Brandimarte. Laura is an assistant professor of management information systems at the University of Arizona Eller College of Management. I met Laura during her time at Carnegie Mellon University, where she earned her Ph.D. in public policy and management, and then was there for two years as a postdoc. Her areas of research include privacy, the psychology of self-disclosure, and the social dynamics of privacy decision-making and information-sharing. In this series, our goal is to take one article, idea, or case and examine the larger implications for the field of tech ethics, including what someone who, for example, is hanging out in North Carolina, reading hard copies of The Wall Street Journal and possibly looking at the weather, like my dad, should know about the idea, just as an example. So today, I'd like to really take a deeper dive into your article that's pretty recent in 2022 in Science with Alessandro Acquisti and Jeff Hancock about how privacy's past may shape its future. 

And, I have to say, obviously I met you at like a privacy law scholars conference and through Alessandro around privacy. And what I really liked about this article was that rather than focusing on, like, the technology du jour of some disruption to what our notions of privacy and security are, that you all tried to argue that there's evidence for us seeking to manage our boundaries of private and public that spans, like, time and space. That we've been constantly doing that. And I didn't know if you could speak a little bit more about, in the paper, how you guys argue that this is not a novel issue that we're dealing with, that we've actually been doing this a while.

Laura Brandimarte  01:44
Yes. Well, first of all, thank you so much, Kirsten, it's such a pleasure to be here. I love this, this series, and I'm really thankful to you for inviting me. So, it is true that we argue that in the paper, but it's not really our argument. So, there have been people in the past, since the '70s actually, that have been talking about how our management of privacy is something that spans cultures and times. So the main, I guess, reference that I would suggest if anyone is interested in these topics is Irwin Altman. He's a psychologist, a sociologist and psychologist, and he's the first one to have said, I think we even quote him in the paper, he said that privacy is at the same time universal but also culturally specific. And what he means by that is that every one of, you know, every different culture, in every different time, may have different representations of what privacy means to them, to their people. But there is a, that we were able to observe, a constant search, in one way or the other, for privacy intended as this ability to manage what is public and what is private. We don't mean to say that people in all ages, at all times, in all circumstances want to protect their privacy-- that's not what privacy management is about. 

Obviously, there are tons of situations where it's great to share, right? Not just personal information, but also intimacy, experiences. There are immense benefits from it, and there's lots of literature in psychology that suggests that this is true. And there's even some interesting work in neuroscience that says that, apparently, when people disclose intimate information, there are parts of the brain that fire that are the exact same that fire when you get a monetary reward. So disclosure, it's really intrinsically rewarding. So privacy management is not about, you know, protecting your space necessarily; it's more about being able to moderate when to open and when to close, depending on the situation. And that is true of lots of cultures, and I'll be happy to give references from ethnographical studies, you know qualitative-type research, that suggests that this is true in many cultures and many times during the evolution of the human species.

Kirsten Martin  04:05
That's what I found so, like, clearly written and well-argued was this idea that we sometimes over focus on privacy as this protective measure, the hiding part of it. And we know that sometimes we rely on that in the scholarship too much, you know what I mean? This idea of inaccessibility as the definition and the construct of privacy, and what I liked about it, and what you just said, is that there's this beneficial side of privacy, meaning having a norm around information as it's shared because we need to share information that's then confidential or protected. Like, so we need to be able to share something, as you point out, like be able to lower our voices and speak privately, you know? And that you'll still share the information without it going anywhere. And this is a way it's both protected and beneficial, if I understood.

Laura Brandimarte  04:54
Yes, absolutely. And I think that part of this is our fault, right? As privacy researchers, we sort of, in the last, you know, 20 years since probably the evolution of Web 2.0, like the commercial side of the internet, and we've seen how in a lot of situations it's easy to disclose a lot about ourselves without actually realizing the risks associated with it. So a lot of us privacy researchers in the last few years have focused on these risks associated with disclosure. And so, I think that part of that conception that privacy management is about hiding and, you know, protecting is, you know, in part our fault. But I think that it's because--you know, it's been the result of what we've seen in the evolution of the internet, right? So companies have indeed become responsible for lots and lots of campaigns about data collection. They have invested more and more into data harvesting, not necessarily without the knowledge and consent of users, so we have seen this as sort of a trend in the last few years. And so that's why I think that the attention of the researchers has been in that area. But I am a firm believer in the fact that privacy, protecting privacy, is not about building boundaries, you know, or raising walls. It's about managing them. It's about deciding what to disclose, and when, and in which situations.

Kirsten Martin  06:19
Right. And where can it go once you disclose it? So, if I share something with you in the offline world, as you say, like, we have muted voices. If we're in a crowded room, and I'm telling you something private, you would understand by my muted voice that it wasn't actually meant for broad consumption. And what I thought was interesting is when you said there's, in the paper, that we don't have an analogue to that in the digital world. There is no pulling you close to say, Hey, I just want to tell you something privately, or just these three groups of people, that our technology or social media is not actually set up for that right now. It's almost like releasing of these boundaries, so that if you tell one, you tell them all, like this idea.

Laura Brandimarte  06:58
Exactly, Kirsten. This is a great way to put it. So there has been more and more work in usable security and privacy recently to build ways to communicate privately. So you see foundations like Signal, for instance, which is sort of this alternative to WhatsApp or, you know, Meta, Facebook, whatever, which is based on this open-source, end-to-end encrypted communication where you can communicate privately with somebody so that you are sure that if you're saying something to one person, you're just saying it to the one person and to nobody else. So there are certain guarantees that encryption can provide. But for sure, it's difficult to do the same kind of thing outside of end-to-end encryption. Unless there is end-to-end encryption, if you use some kind of intermediary to communicate with somebody else, there is always you, the person that you're trying to reach, and the intermediary in the middle. And when there is the intermediary in the middle, unless there is regulation that imposes that intermediary not to share that information with third parties, eventually, if it's profitable for them, they will. And so I think that that's what happened with the data market, especially in the United States, which for a large part is still unregulated. There are certain regulations with specific types of data. But in general, there is no yet umbrella of federal regulation about data in general. And so, it's really difficult to know, once you shared something with a company, maybe you trust that one company with your data, but then that company a lot of times ends up sharing it with other third parties without you knowing. 

And I guess my research has been trying to do something about that. We've been trying to develop something that might help essentially trace the flow of the data in the data markets. We're trying to raise the veil, you know, that hides all of this unknown data market that exists among companies and data brokers. But it's something that really, I think, requires a shift in the mentality. So, companies might--you know, they have to understand that it's not just about harvesting as much data as possible because at some point you might be able to do something with it. It's more about collecting what information is necessary in order to provide a good service to the user, and I think that that mentality shift still hasn't happened, especially in the United States, not yet.

Kirsten Martin  09:21
Right. And I think the term that I've heard some of you use privately but also in this paper is the "responsibilization" of this focus of, in the United States, we've really put a focus on over-emphasizing the responsibility of the individual to navigate, as you mentioned, this really unknown market of data that's hidden. Even us researchers don't know it--like, you would think that we should know where 60% of the internet traffic is going because it's all this trading with data traffickers back and forth. Yet we don't actually have that information. And it's this focus in the United States on, Well, you, the individual, should have known better when you were sharing information with that website or that app or you downloaded something, as to everything that was going to be going on. Versus, this shift that you're talking about, which is asking the intermediary--the website, the app, the social media company--what are you responsible for? Like, what data are you gathering? How are you using it? You should be able to justify this; why are we asking the individual to make all these decisions? Which is a shift in our mentality here in the United States because that is not how we think about things right now.

Laura Brandimarte  10:26
Yes, exactly. So we argue, essentially, that there are many reasons why we really can't make the user responsible for making the best possible privacy decision for themselves. One of the reasons is asymmetric information, for instance. So people, when we use the internet, we don't use the internet with the primary goal of protecting our privacy, right? Or having our security guaranteed. We go on the internet because we need to do things, we need to do stuff--we need to take classes, we need to buy things, we need to teach, we need to do things. And so privacy and security will never be our first concern, and that is the first reason. 

The second is that we don't really know, like we mentioned, what is going to happen once we share information on the internet. It's not really clear where that information is going. Even if we did know where the information is going, we wouldn't really understand what are all the risks and benefits associated with that disclosure. Yes, we all understand that, you know, if we sign up for a loyalty program at our, you know, grocery store, we would get some points or, you know, we get discounts. But then on top of that, what else? What happens? So the consequences are not really clear of what is going to happen. And third, we tend to skip, you know, all the information that we're provided by companies that are just trying to abide by the law. If the law focuses so much on notice and consent, which is what has happened in the United States, essentially, companies are in the clear as long as they have a long privacy policy that explains, you know, details about where the data is going and what [it]'s being used for. 

Kirsten Martin  12:06
Or no details, no details. Like, they could just say, Not much. Yeah, right.

Laura Brandimarte  12:11
Exactly. It's not really transparency, right? So the core of the law, it's just about providing information to the people so that the people can make good decisions for them. But we can't do that, right? We're not going--there's lots of work, Lorrie Cranor is a great researcher at Carnegie Mellon, she's done incredible work on the economic costs of reading privacy policies. If each one of us, just in the United States, were to read all the privacy policies of all the websites that, you know, we interact with, basically, we would end up losing something in the billions of dollars per year, which is insane obviously. So that approach of giving the users all the responsibility to make decisions when it comes to privacy is really--it's not effective, it doesn't really help the consumer at all. It only provides a lot of weight and responsibility to the user, which they can't handle.

Kirsten Martin  13:03
That's great. And what I liked in the paper was the suggestion, I liked the analogy that you all made to cars, right? And so when we started going faster with cars--and we can see the analogy of our information is flowing faster online--we had to, all of a sudden we had technological improvements, we could go a lot faster, and we added laws and technical fixes. You know what I mean? Like, we didn't just say, Oh, wow, a lot more people are dying, you know, like what are we going to do? We actually started doing things about it, both regulating how drivers acted, but also regulating how car manufacturers behaved, and then having technical fixes. And I didn't know if you could speak a little bit about that. I like that parallel because sometimes we think there's no way to regulate this or there's nothing that we can do or no fixes. And yet we've dealt with these things before.

Laura Brandimarte  13:48
Yes, that's an analogy that I owe 100% to my coauthor Alessandro Acquisti, who was also my advisor. I think it's a great analogy, it makes perfect sense, right? So, like you said, the development of technology in cars has made it so that at some point, cars were becoming more and more dangerous, and we weren't able to handle them. And what happened when, you know, those technologies arrived, when those new technologies arrived--it's not like we trained people to be more responsive, for instance. We didn't rely, we didn't put the responsibility on them. We simply imposed regulations that said, Okay, on this particular roads, you can't go faster than so and so, and if you do go faster, you're gonna get fined for it. And similarly, we said to the manufacturers, You have to build particular technological solutions to help the users because they are not to be considered responsible for the development of technology, right? So you add ABS systems, like braking systems, right? Or airbags, for instance. And you make seatbelts mandatory, right? So there are specific regulations that can help the relationship between humans and technology when technology moves so fast that humans can't keep up essentially. So there are certain things that we can do at the regulatory level, which we have done, like you said, in the past for similar issues. 

So our argument is, why not do that in the realm of privacy, as well? Other countries have done that, all kinds of countries--you know, not just Europe, but China has developed their own regulation for internet privacy, Australia have their own. So it's not just, you know, GDPR; everybody's so focused nowadays on the General Data Protection Regulation, but it's not just Europe. It's a lot of other countries. So why shouldn't or couldn't the U.S. do a similar thing where there is an intervention at the federal level, allowing therefore for a general protection of data that goes beyond specific regulations of, I don't know, health-related data or educational records and so on and so forth. There are these types of data that are protected in the United States, but why not make it general so that data is data essentially, and so the data market itself is regulated?

Kirsten Martin  16:05
Well, and especially because the data market itself really doesn't have any market pressures--you and I are in a business school, so we talk about these things--and so the data market itself is odd because there's no consumer pressure, there's no real SEC pressure. I mean, like, all the normal pressure points that we might see of the market correcting the bad behavior of these data traffickers that are hidden behind the scenes trading information, we don't have those normal mechanisms. And it's also interesting from the car industry standpoint because our automotive industry, at least in the United States, similarly argued against regulations for decades, you know, and said that if you regulate us, like, we will go out of business and there's absolutely no way that we'd be able to continue to make cars. So this--it's a normal give and take, you know what I mean? Technology gets faster, people start getting hurt, you know what I mean? Like, we try to figure out what to do about it, the industry says, There's no way, otherwise we'll go out of business. And then somehow they persevere, you know what I mean? Like, they kind of make it through.

I really liked the paper mainly--I mean, for many reasons. Like, I love the historical nature of our need, as you said, "The history of privacy tells us that the drive to maintain a private space may be as universal as the drive to commune." And I think that this shift of thinking about privacy as a need to be secluded--you have to either be secluded or interact--versus this drive for privacy that we've always had [that] is actually tied in with the need to interact, to commune with people. And so it's not seclusion, it's the opposite of that. It's the idea that we should be able to commune, have a community, talk, develop ourselves in society by interacting with other people within understandings about how that information will be used. It's actually the opposite of a way we've been thinking about it. And that we have this need, as you said, to carve out those spaces, even when the odds are stacked against us, which I just thought was a great summary of the article.

Laura Brandimarte  17:54
Yeah, thank you, Kirsten. I think that you summarized it even better than we wrote it. (laughs)

Kirsten Martin  17:57
(laughs) Oh, no, I was reading from it.

Laura Brandimarte  18:02
Really though, that's what we believe, right? So, whenever you confide in somebody, right, you tell them something about yourself, essentially what you're doing is you're trusting them, that the information that you give to them is going to stay between the two of you. And that kind of relationship is--why shouldn't that kind of relationship be similar to what happens with, I don't know, Google or Facebook or other companies, right? You should be able to trust the companies that you interact with. Because after all, you give them--you know, we give Amazon our addresses, our credit card numbers, so there is an embedded trust relationship between the consumer and, you know, commercial companies obviously. So I think that what customers would want when it comes to exchanging data with other companies is precisely that kind of trust; we would like for these kinds of transactions to happen in the realm of trust. So you should be able to share information with your provider without the fear that that provider is going to share that information with third parties. And it doesn't matter whether we consider that information to be intimate or sensitive because nowadays, anything can be translated into something sensitive or important. 

When Cambridge Analytica happened, nobody had thought about the consequences of sharing your psychological tests, right? So people were lured into disclosing this information because they were asked a simple question such as, you know, What kind of type of person you are, do you want to know what profile, psychological profile you have? Answer these five questions, and we'll tell you. That seemed pretty tame, right? And then of course, we realized that that was not that tame. So it's not about necessarily disclosing information that is sensitive; nowadays anything, one way or the other, can turn out to be sensitive. And so I would like for that relationship that you have with companies to be based on trust. And I'm afraid that unless--in that respect, our incentives are not really aligned. There's more that we have to gain from that kind of relationship than companies have as of now, you know, because of the lack of regulation and because of the way that the data market is structured. So unless we find a way to align those incentives, it's going to be hard to establish that trust relationship with other companies. And we argue that one way to do that, to align those incentives, is indeed via regulation.

Kirsten Martin  20:32
That's great. Yeah, I like a lot of the work that you do with your coauthors and by yourself, and so I really appreciate you taking the time to talk about it. And I always look forward to seeing what you guys are writing about next and what you're writing about next. And I want to just wrap up by asking, like, who else should we be paying attention to in the area of tech ethics, with the idea of, you know, not necessarily only within our discipline, but also just any discipline that you--are there scholars that you're following right now that you're waiting to see what they write next?

Laura Brandimarte  21:01
Yes, absolutely. This is something that I'm actually very excited about; tech ethics is, you know, it's what I do, and so I'm very passionate about this. And I'm really looking forward to hearing from a Ph.D., recent Ph.D. graduate, very recent, Joy Buolamwini, she just recently graduated from MIT. She does excellent work in tech ethics. 

Kirsten Martin  21:23
She's great.

Laura Brandimarte  21:24
Yeah, her dissertation is basically, it's a study of algorithmic bias in the field of computer vision. And her story, I actually got to know about her, I don't know her personally, but I got to know about her from a documentary that I highly recommend to anyone who's interested in algorithmic bias and tech ethics in general. It's called Coded Bias. And in that documentary, she tells her story of how she started her Ph.D. She was just studying a computer vision algorithm with face recognition essentially, so the capabilities of algorithms to detect faces. And she noticed that--she is a Black student, she was a Black student, and the algorithm couldn't recognize her face. Then she put a white mask on, and all of a sudden, the algorithm was able to detect the face. And so she said, There's something wrong here, there is something really, really wrong. And so she started her whole Ph.D. thesis based on that idea that sometimes because of the way that algorithms are trained and built, they have an inbuilt bias, which is probably unintended, right? We're not saying that companies build algorithms in order to be biased, in order to cause bias, but it's just because of the way that they're trained, they end up provid[ing] biases in their results. And so we have to be aware of those kinds of biases, and I think that Joy is really an excellent researcher who's done really, really great work up to now. So I'm definitely looking forward to seeing what she's doing next.

Kirsten Martin  22:55
Yeah, that's great. That's great. Well, gosh, I really, I so appreciate your time and everything. And I hope you'll come on again--like, I'll watch for your next paper and then send you another email to say, Hey, you want to come on and talk about this?

Laura Brandimarte  23:07
(laughs) I would be honored, Kirsten. Thank you so much.

Kirsten Martin  23:10
This is great. I always enjoy your work, and I really appreciate you taking the time, so thanks so much.

Laura Brandimarte  23:14
Thank you so much, Kirsten. And also, I appreciate your work, as well. It's fantastic, you're definitely a pioneer. So thank you for what you do.

Kirsten Martin  23:23
You're so nice. Thank you so much.

Kirsten Martin  23:25
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

 

3. Not the (Speech) Chilling Effect We Think (June 1, 2022)

Guest: Suneal Bedi (Indiana University)

Transcript

Kirsten Martin  0:03  
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we discuss an important idea, paper, article, discovery in tech ethics, and today, I'm so excited to be joined by Suneal Bedi. Suneal is an assistant professor of business law and ethics at the Kelley School of Business at IU, Indiana University, where his research focuses on intellectual property, marketing law/ethics, and brand strategy. He teaches classes in business ethics, corporate law, and fashion law and ethics. His research employs lots of different methods because of his many degrees that he has--which is super helpful when you're looking at tech ethics--to answer business-relevant questions that sit at the intersection of law, marketing, and public policy. And many times his work makes its way into tech ethics, specifically around marketing, marketing decisions, markets, and the decisions of firms.

And in this series, our goal is to take this one idea and examine the larger implications for the field of tech ethics. And today, we want to take a deeper dive with you into this article that you recently had on "The Myth of the Chilling Effect" in the Harvard Journal of Law & Technology. And I really liked not only the findings of this paper, but I really like how you handled the setup or the grounding, as we would call it. And so like, why, theoretically, is this an interesting thing to talk about, and you tackle some of our favorite ideas in tech ethics that we're always batting away, which is both the First Amendment and Section 230, which I thought was great. And so the article starts with this idea of the First Amendment and the idea of the chilling effect. And I thought you summarized this really well with, the theory is that "if citizens are not confident of exactly what speech is being limited, they may overregulate their speech in fear of sanctions, and hence may chill, in quotes, their speech in an unnecessary way." And what I liked was that you were trying to take this idea from the First Amendment and see how it was operative for private companies. And I thought maybe you could talk a little bit about this because the First Amendment applies to the government. A lot of times it's--I'm sure frustrating to you, too often just thrown around with companies. And so I thought maybe you could just take a second and talk about that issue, you know, how you dealt with it in this paper. Because I thought it was brilliant.

Suneal Bedi  2:13  
For sure. You know, first thanks for having me on, I'm really excited to be here. So it's really interesting. We always say the First Amendment, and we talk about the First Amendment in response to private companies, but as you said, it doesn't apply, right? So the First Amendment, the text of the First Amendment, says Congress shall not make a law. So the idea is the First Amendment only applies when Congress is acting--when it's the Senate, when it's the legislative, the House, or sort of any government entity. It doesn't really apply when in private individuals say things. So, for example, you know, I used to be at a law firm. That law firm might tell me in my contract, You aren't allowed to say things that are bad about the law firm. Perfectly fine, not a violation of the First Amendment. It doesn't even implicate the First Amendment. And so often, you know, we have people kind of misnomering--

Kirsten Martin  3:05
Always.

Suneal Bedi  3:08  
As applies to private entities. But it doesn't. That being said, even though the First Amendment as a legal principle doesn't apply, some of the effects of speech regulation that are talked about in the sort of public government arena can certainly make their way into the private arena. And that's sort of the impetus of both the thought processes of the article and the methodology of the article to really see, okay, we have this argument that people make about the chilling effect in the government realm, you know, and people are starting to make that same argument--even though it doesn't violate the First Amendment necessarily--they're still trying to make that argument in the private realm. And so the idea is, can I test that and see what actually happens?

Kirsten Martin  3:53
So in some ways, to be most charitable to people that throw around the First Amendment with private companies, what you're picking up on is the fact that they're worried about the same chilling effect by content moderation if it's arbitrary or over aggressive or something along those lines, that our speech could be chilled. Which we might decide is good or bad or something along those lines. But that's what they're trying to get at with these claims of the First Amendment of really, they're just worried about speech being chilled in a way that's not good, or it's something that we don't--not appropriate or something that we don't want. 

Suneal Bedi  4:30  
Yeah, yeah, that's really--I think that's what's going on if we're being charitable to people who talk the First Amendment with private. It certainly is not about legality, well, it's illegal, and that's why it's relevant for tech ethics. It really comes down to a question of ethics on, is this sort of the types of things that we want businesses to do independent of what's legal. But I want to sort of, you know, clear sort of, ground-clearing a little bit about the chilling effect. Because I think sometimes it's a little bit confused where, you know, we think of, Oh, well sometimes the chilling effect, sometimes there's not, is it right, is it wrong? You know, there's two effects that happen when you say people can't say something, right? So for example, let's say, if I say, you know, Students are not allowed to say Professor Bedi abuses animals. I don't, right? But let's say that's the rule. Okay. Well, then, you know, whoever is my student is not allowed to say, you know, Professor Bedi kicked that dog that he saw. That would be a lie, and it would go against sort of what the rule is. All right. And that's fine. That's not really chilling. The problem is that rule may also sort of prevent a student from making a joke. Maybe the student wants to say, Oh, well, Professor Bedi's like Michael Vick, he's so aggressive. Okay. Well, now, they're not saying anything about, you know, animal abuse, they're not lying or anything. But it's close. And that maybe we would be okay with them saying, but the fact is, this sort of rule prevents them from saying that. That's the chilling effect. So every time we have a rule, it certainly affects people's speech. But it might affect people's speech in the exact way we want it to. Great. But then it also might affect in ways that we don't anticipate or we don't intend. And that is the chilling effect. And that's what I'm trying to sort of capture. And that's what people are saying is problematic with these private companies enacting sort of speech regulations.

Kirsten Martin  6:24  
So right, so if I understand, even with a surgical strike content moderation--which is, you're not allowed to say, hey, that I abused dogs or something along those lines--but there could be this shadow or cloud that when the surgical strike hits, it's like kind of like the flume that comes out from it that's going to capture other things that we don't intend for it. We just are saying you can't defame me in some way. But we might actually capture--it reminds me of, like, with content moderation on social networks sometimes, the people who are trying to fight, like, white supremacy or racist comments get moderated out of calling it out. That's not the intent. You know what I mean? But it's very difficult to enact a policy without either it being chilled or, like, there being some sort of, like, overflow from it. 

Suneal Bedi  7:11  
Yeah, exactly. 

Kirsten Martin  7:12 
And people being afraid to talk about it

Suneal Bedi  7:13 
Correct. And the idea is that, you know, the more narrow we get, right, you aren't allowed to use the word ess h i tee. Okay, that's probably--maybe that would chill saying the word "shoot." Okay, but we don't really care, right? But the more narrow we get, the less we have these flume sort of effects, these chilling effects. But then the problem is, there's a balance. Because the more narrow we get, the less, really, that the speech regulation is doing. The broader we say, you know, You can't say racist remarks. Okay, that's covering a lot of stuff. But is it covering jokes? Is it covering, you know, real dialogue about race that isn't necessarily harmful? Potentially. And that is--if people don't want to say that because they're afraid, that's the chilling effect coming into play.

Kirsten Martin  7:57
Yeah, that's great. So then, the other bane of our existence, the misunderstood Section 230 makes a little appearance there for briefly. So I didn't know if you could just speak just briefly about Section 230. Just because it's similar to the First Amendment, it's kind of, like, batted around like a Wiffle bat, and people don't understand. 

Suneal Bedi  8:13 
I'm certainly not a 230 scholar, but what I sort of get from it is this idea that we cannot hold tech companies liable for what is said on their platform, right? That's the first sort of cut at it, we can't hold them liable. Okay. And that's sort of fine. I think that's, you know, they're not responsible for what sort of customers or people are tweeting or writing on Facebook, they can't be held responsible, that's hard. The other cut at it is because of that, we also actively give them sort of freedom to limit how people are speaking on their platform. We specifically say, You can kind of do whatever you want because you're in this weird, sort of tech realm, right? Where you might see other sort of public resources we don't say. So for example, if, you know, you owned a mall, you might not be able to say, Well, no protesting at this mall. You know, the court has said, Listen, that's sort of so public in nature that we're going to say you can't sort of violate the First Amendment, or you can't have speech restrictions. Section 230 effectively says tech companies can do really whatever they want with the content on their platform. They can limit it; they can also not limit it, and we're not going to hold them responsible for either that decision.

Kirsten Martin  9:35
Yeah, right, and so it actually gives them the freedom to do that, to do content moderation. Because otherwise, we, society, would see them as meddling with the content in some way, getting their hands in in, like, a Lockean way, you know, kind of mixing their property with that property and all of a sudden having some ownership as to what the content was. And so what they said was No, no, no, we want you to be free to moderate the content if you so desire, to the way that you wish. You know, one can be Twitter, one can be Facebook, you make different decisions, that's all fine. 

Suneal Bedi  10:06
You don't have to; you can do whatever you want. You can moderate, or you can not moderate. We're gonna say it doesn't violate any law, and actively, we're gonna give you the power to do it. That's effectively what 230 is doing.

Kirsten Martin  10:17
Right. Yeah. And in some ways, I mean, it makes it so [it's] less a business law decision, it's a business ethics decision. So, like, whatever you want to do has to be strategic and ethical because they're not--the lawyers, I mean, they'll just say, Yes, you can. (laughs) So it doesn't really give you the should, you know, of what you should do.

Suneal Bedi  10:32
And that's what I think is so interesting about on the tech side, these content moderation decisions. And when people say, Well, it's about the First Amendment. It's actually not, it's literally a 100% ethical decision. Now, are you balancing--you know, you're balancing interests of various stakeholders. Maybe it's society, maybe it's customers, maybe it's shareholders, maybe it's justice, maybe it's diversity. You know, you're balancing--that's literally what business ethics is, right? So this is in some ways a pure tech ethics decision, which is what makes it really cool.

Kirsten Martin  11:03
Yeah, that's true. I hadn't really thought of it that way, is that what's so interesting about it is--and what's interesting about it in some content moderation decisions within social media companies--it's actually more the lawyers making the decision. You know, the engineers are actually proposing ways to moderate the content, as we saw with the Facebook Papers, and the lawyers are stepping in. But it's really, that might not be the expertise that they need to be making those types of decisions. Because they're only going to say, Yes, you can, because the law gives them the freedom to do it. But it doesn't really give you the should.

Suneal Bedi  11:33
Exactly. And that's the ethics question, which I find it so fascinating. That's why it's such a cool sort of topic, I think. 

Kirsten Martin  11:39
Right. So then, so that's the background, which I think you make a great case on this. And then the study I thought was brilliant because, and you should explain it better than me, but you gave people--like, you actually tested whether or not people would moderate their content based on different policies. Is that right? Like content moderation policies, like a zero, strict, and general type of thing?

Suneal Bedi  12:02  
Exactly. And so the idea, you might say, Well, what we really want is, what we really would want is the government to say, You can't say XYZ, and then measure what happened. That's so hard to do, economists do that. (Kirsten laughs). You know, that's strictly, event studies, that's really hard to do. So the question is, can we sort of in a private setting mimic content moderation and then see what people do? And so what I did is I allowed people to write sort of Yelp-like reviews, like negative reviews about some sort of dining experience they had. It was during COVID, so I figured a lot of people had, you know, takeout or Uber Eats, a lot of dining experiences. So I said, You know what, write a really negative review, just write whatever you want. And then some people I said, There is no restriction, write whatever you want. In others, I said, You may not use certain words. And I chose very--you know, I chose words that were generally cuss words or really highfalutin vocabulary words that people wouldn't use.

Kirsten Martin  13:04
These are words like, just to say, like repugnant, putrid, gruesome, ess h i tee, the f word, that type of thing.

Suneal Bedi  13:10
Those type of words, right. And then I gave a third, it was a third sort of instruction where, You can't say general categories of words. And I said, you know, harmful language, racially charged language, offensive language. The idea would be that in the condition where I don't tell them anything, they're gonna say whatever they want. Okay. In the condition where I tell them, You can't say these words, they're not going to say those words. Fine. And in the condition where I say, You can't say these categories of words, ideally, they wouldn't say those categories of words, which is fine. And that's sort of, and that's not a chilling effect, that would be the actual effect that we want. The chilling effect is an effect beyond that, right?

And so the question is, how do we isolate that? Well, the way I isolated it was, I tried to create these speech restrictions so that they would actually have no effect. So for example, if I tell you, you know, go out and write about, you know, your negative dining experience, but don't say the word "Professor Bedi," you might be like--

Kirsten Martin  14:12 
Right, right, how hard is that? (laughs)

Suneal Bedi  14:12
I never would say that, anyway. Exactly. So if I say don't say that, it shouldn't change your speech. So that was the idea. The idea was to choose sort of words--ess h i tee, repugnant, putrid--words that nobody really uses based upon pretesting, or say, Don't use racially charged language. No one said anything racist, no one was ever going to say--so really, if there were no chilling effect, we should see the speech be consistent across all of those three things because it's not actually doing anything.

Instead, what I found, is I found people did change their speech, even though the speech restrictions shouldn't have actually changed their speech, it shouldn't require them--because they weren't going to say those words anyways. No one was gonna say something racist when they were talking about an Uber Eats experience. They just weren't gonna say it. So the question is, well, how did they change it? And how do we sort of contextualize that? And what it turns out is, people actually used more positively toned words in order to sort of--you know, the person that's making the decisions, in order to maybe trick the person making the decision, in order to get around the content moderation--for whatever reason, they overcorrected their speech. They didn't need to correct it at all, and they overcorrected it. But they overcorrected it in a way that made the speech more positively sort of charged. And you might say, Okay, that's bad. That's a bad thing because what we wanted was negative reviews, and so now the chilling has made it more positive.

So here would be a, you know, a political example. You might say, Don't lie about politicians. And now people aren't lying about politicians, but they're also not being as critical as they would be, right? So you're like, Oh, that's bad, that's bad. So I said, Well, let's see if that's actually true. I then took those negative reviews and had people rate those negative reviews. And it turns out in each category, the reviews were equally negative, which means that the message that was being communicated by those negative reviews was equally negative. The message was communicated. It didn't matter that I said, Don't say racially charged language. People could still communicate what they wanted. But they did so in a slightly more positively toned manner. So if we take that into, like, outside of this negative review context, that may mean, and as I suggest, as I argue, that actually, these sorts of content moderation that maybe Twitter, Facebook, Instagram are doing will allow people to still express what they want to express, but do so in a way that is, you know, more civil, in a way more positive, that it could actually, you know, not only not chill speech, but actually promote more general dialogue and more robust dialogue around controversial topics.

Kirsten Martin  17:05 
That's what I found so fascinating about this finding was that, like, there was no change in the likelihood to patronize the restaurant, so the actual, like, likelihood to go back. So they conveyed the same substance of what they wanted to convey; they just did so with more civil speech. Which as I jokingly said to you before this started, which my favorite lawyer, what I was telling him about your story, said to me, Isn't that the whole point of content moderation, is that you have a civil place that people feel safe to be on? And I thought that that was, like, such a--well, one, the study design is simple, which is always the best. But also the way that you did the analysis was so clear of, you know, kind of this idea of tone versus substance, and that there was a chilling effect, but it wasn't the chilling effect that we're all afraid of. This was a chilling effect to make sure that they weren't instead of not saying repugnant, they also didn't say awful. You know what I mean? Like, that they used different words in order to convey what they were trying to say.

Suneal Bedi  18:05
And you know, often what we say, and particularly in the private context, and we say, Listen, if Twitter says, You can't say false statements, you can't say derogatory statements, then people actually aren't going to do anything. They're just going to be like, You know what, not gonna say anything, deactivate my account. That's also part of the chilling effect, and people get afraid. That was quite the opposite. Nope, people had the option to exit, they could've said, Okay, I'm not, I don't, fine, I won't get paid. I won't do this. No, people said, You know what, I'm gonna say it, I'm gonna say what I want to say, but I'm gonna find a way to say it around your restriction, and it turns out the way I'm going to say it, it's actually kind of in a positive manner. And you can imagine this applying to sort of dialogue about politics, when somebody wants to criticize one party or another party or want to criticize a politician, rather than sort of saying, the chilling effect being, Oh, well, they're not going to say anything at all. Instead, we say, No, they will say it, but they'll find a way to say it in a way that is more civil, which will ideally create more conversation, which will then sort of create more robust dialogue, which is exactly what we want, right? 

Kirsten Martin  19:13
Yeah. And that's the other part of the chilling effect that we always, or people that spout the First Amendment about these things with a chilling effect ignore--you also point out that we don't talk enough about the chilling of speech when there's no content moderation, of the target of hate speech. And that could be racial, ethnic minorities, women, LGBTQ+, and it doesn't even have to be that you're a member of a specific group. If I see racialized speech-- I'm white--that would upset me to feel unsafe on the platform; there I could be targeted as well. And that is a type of chilling effect that we don't talk about, about the absence of it. So just like we have surgical strikes with content moderation, every bullying, you know, tweet or every bullying post on Facebook or a Facebook group that's about hate speech is really the same type of chilling effect, if not larger, to an entire group of people that we just don't even see.

Suneal Bedi  20:08
Yeah, you know what's funny, Kirsten, is I didn't even call that a chilling effect. But now that you've said it that way, I think that is exactly what that is. Whereas content moderation has a both a chilling effect when you're moderating content, but it also has a chilling effect when you're actively not moderating.

Kirsten Martin  20:24
Right, exactly. 

Suneal Bedi  20:26
And so the question is, what are these two poles, and how do we sort of balance these poles? And, you know, my study suggests, well, maybe we can sort of strike the balance in terms of moderation because it will allow people to have a conversation, and it won't sort of chill those people, and they won't sort of leave the platform. They'll just change their speech slightly, and that will actually maybe create more robust dialogue.

Kirsten Martin  20:48
Right. Yeah, exactly. And that's what, because what I liked about it is that you kind of showed a path forward on the ability to do content moderation, to make people feel better without actually--it doesn't have to change the substance. There's better and worse ways to do content moderation. And we see this in practice, we see the difference of content moderation on Facebook versus Twitter, and you can see the actual implications of it. You know, where do white nationalist groups go, where do they not go? Who's upset about Twitter's policies, who's not upset? You know, you can kind of see who likes these decisions and who doesn't. And so the difference of the way you tested content moderation is important, in that it's not just some versus none; it was different types and to think that through.

Suneal Bedi  21:30
The idea is that historically, we've thought--and this ties back to the First Amendment--we've thought government restrictions that are very broad are wrong because they're gonna have more of a chilling effect than the ones that are, you know, more precise strikes. And that very well may be true, I kind of show that a little bit to some degree. But that broad sort of restriction may still chill in ways that we're okay with. And I think that's the idea. And that's why in the private context, it's a little bit different than the public context. Because in the public context, when you think about the government, the government can't take a stand on a specific type of speech, the government cannot say, you know--

Kirsten Martin  22:11
Well, they shouldn't. (laughs)

Suneal Bedi  22:14
They shouldn't. Legally, they shouldn't, they can't really say, Well, white supremacy speech is bad, and, you know, diversity speech is good. They can't say that.

Kirsten Martin  22:20
Right, right. 

Suneal Bedi  22:22
But private companies can. Why not? 

Kirsten Martin  22:24
Yeah, right. 

Suneal Bedi  22:25
Of course they can say that. You would ask a private company, Is white supremacy speech good? They're gonna say no, right? Obviously.

Kirsten Martin  22:31
Fingers crossed. (laughs)

Suneal Bedi  22:34
They should. You would think. So they can.

Kirsten Martin  22:34  
Yeah, yeah.

Suneal Bedi  22:34
And so the idea is that these companies can make these sorts of decisions. They're an ethics decision. But they can favor speech, they can favor certain types of speech, and so we are allowed to do that. The question is, how do we strike that balance, right, and what are the effects? What are the externalities when we do that? And I think that, you know, my research shows, well, they're not as sort of overblown as people think they are. The externalities not only might be small, but actually might work in our favor.

Kirsten Martin  23:02  
They might be positive. Right. Yeah, they might be positive externalities.

Suneal Bedi  23:02  
They might actually work in our favor. Yeah, exactly. 

Kirsten Martin  23:07
Right, exactly. Especially if done well. And I think--well, you summarize it, I'll just say this, you summarize it really well: "If the chilling effect can change the tone of how individuals speak while keeping the content of their speech the same, it could also promote participation in exchanges of ideas from those who would otherwise be offended. The longstanding legal principle then may be reframed as a way to promote more robust speech activity rather than deter it." And I think that's the part that I think is missing, is that the lack of content moderation, as we were talking about, chills speech. It might not be speech that isn't prioritized in society, but these are speeches of marginalized groups that--and they're not, I don't mean marginalized like there aren't very many of us. There's lots of us. I mean, it's just that they're not being prioritized. And it's important to realize that the lack of content moderation actually disincentivizes people from going on, you know, that it actually has less speech because of it. 

Suneal Bedi  24:02  
Yeah, I mean, I think you actually said it right, and in hindsight, I wish I would have said it myself in the paper, that lack of content moderation does chill. And the idea is, who are we chilling in that? As you say, we're often chilling marginalized groups, we're chilling groups that have been historically marginalized, right? And the ones that we actually, the groups that we want to promote having a sort of a different sort of perspective or bringing their perspective because historically, they haven't had the platform. And it's actually those groups that we are preventing, or at least creating an environment that they are self-selecting out of our social media platforms. You know what, maybe the idea is, okay, yeah, we can sort of require, you know, more majoritarian groups to have to tailor their speech a little bit, but not so much that they can't say what they want, in order to get, you know, marginalized groups to really be able to be active on these platforms. I mean, I think that's sort of a tradeoff that I think, you know, an ethics tradeoff that tech companies can sort of seemingly make and reasonably make.

Kirsten Martin  25:07  
Right. And they can if they decide that's the values and the mission of the organization. So if Twitter decides, this is the type of speech that I want to have on my platform, these are the types of group I want people to feel safe--which they seem to have done that fairly well in the past--then I'm going to content moderate one way. If another social media company decides to do nothing, it's interesting to see, we almost have our own experiment. We can see who's on those, you know, and so you can kind of see, where do people feel safe and want to be? And it's where there's some content moderation--not a lot, but a little bit, so that it's a place where they don't feel chilled in that way. 

Suneal Bedi  25:40 
Exactly, exactly.

Kirsten Martin  25:40  
Yeah. So I just, I could keep you here, and I would love to talk more, but I know we all have things to do and classes to teach and stuff like that. But I wanted to ask one last question, just like in the area of tech ethics, just broadly, like if there's anybody that you're really paying attention to or someone that you want to highlight, who you look forward to seeing what they write about or anything along those lines.

Suneal Bedi  26:00  
Yeah, I mean, well, first, I'm really interested to see what's going to happen with Elon Musk and Twitter. (Kirsten laughs) I'm very curious if this sale goes through, particularly if the (inaudible) board goes away. You know, what is Twitter going to do with its content moderation? Because it's hotly contested. I do think, and I know you had him already on a podcast, but my good friend Vikram Bhargava--

Kirsten Martin  26:22  
Oh, Vik's great.

Suneal Bedi  26:22
He's doing really interesting stuff in tech ethics. And you know, I would encourage anybody to read a lot of his work, not just what he discussed on your podcast, but some of his other stuff. And so I would definitely, those are two: Elon Musk and Vikram, if I can put them.

Kirsten Martin  26:35
(laughs) They're like this; I'm crossing my fingers. Yeah. That's great. Yeah, I'm looking forward to seeing what happens with that as well. I think it's a little bit scary that he thinks things need to change because Twitter is kind of known in this space as, they're not perfect, but they're trying, and they're heading in the right direction. Like, they're constantly working, they fight to protect their subjects and their users from being outed with anonymity. But anyway, we'll see what happens. But gosh, Suneal, I really appreciate you being here and us being able to talk about this. I look forward to the next time you write in this space. I'm more than happy to have you back, and we can talk about your next article. And I hope to see you at the ethics conferences that we go to and all that kind of stuff. Oh, and our workshops.

Suneal Bedi  27:18
Of course, absolutely. Thank you so much, Kirsten, for having me. I appreciate having the opportunity to share my research and really enjoy what Notre Dame is doing with the tech ethics podcast. 

Kirsten Martin  27:28 
Oh, great. Thank you so much. Thanks a lot.

Kirsten Martin  27:31
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

2. Don’t Take the Data and Run (May 11, 2022)

Guest: Katie Shilton (University of Maryland, College Park)

Transcript

Kirsten Martin  0:00  
(voiceover) Hey, it's Kirsten. A quick note about today's episode: You'll hear my guest, Katie, and I use the term "IRB" on several occasions. In case you're not steeped in academia life like we have been, I figured it would be helpful for you to know that this stands for the Institutional Review Board, and it's the body at a university that ensures research studies meet appropriate ethical standards. Anytime we do research, we have to submit the proposal to this IRB, and then they approve it. Okay, let's get started. (end voiceover)

Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we like to call ND TEC. In these discussions, we discuss an important paper, idea, article, discovery in tech ethics, and today, I'm so happy to be joined by Katie Shilton. Katie is an associate professor in the College of Information Studies at the University of Maryland College Park and leads the Ethics and Values in Design Lab. Her research explores ethics and policy for the design of information technologies, and she's the PI of the PERVADE project, a multi-campus collaboration focused on big data research ethics, and we're going to talk a little bit about that today. She also works on things like developing privacy-sensitive search for email collections, analyzing ethical cultures and computer security research, building tools to facilitate ethics discussions, also called value levers--and I cite that all the time. And I like to say her work on value levers about opening up ethical discussions in development. And I know Katie from our work on privacy together and some other conferences where we met each other, and we kind of take it from, I'm from a business school and she's from the I school, and we kind of approach the same subjects from different points of view. Today, we're gonna take a deeper dive into your article with a few others--and the others include Emanuel Moss, Sarah Gilbert, Matthew Bietz, Casey Fiesler, Jacob Metcalf, Jessica Vitak, Michael Zimmer, like a cast of 1000s, but it's all the people from the PERVADE project--about "Excavating awareness and power in data science: A manifesto"--which got me there--"for trustworthy, pervasive data research," which is in Big Data & Society already. Can you talk a little bit about the PERVADE project on data research ethics? Because it kind of comes out of that whole entire endeavor.

Katie Shilton  2:11  
Yeah, absolutely. So right, this is sort of our first joint project--or our joint paper-- from the PERVADE project, which has been going on for four years, five years, what is time anymore? (laughs) But it's a collaboration, it's a six-campus collaboration that's funded by the National Science Foundation. And the idea was to ask empirical questions in data ethics and in big data ethics because, you know, five or six years ago, when we wrote this proposal, there was lots of good work out there sort of saying, There's a problem in data ethics, right? We don't know what to do. We don't know what best practices are for using big data in research. And there were starting to be some sort of recommendations from ethicists, folks working in research saying, Oh, you know, we should do this or this. But there were a whole bunch of unanswered empirical questions like, What are researchers--How are researchers grappling with their data practices on the ground, right? Like, what are they doing in their communities? What's acceptable in given communities? What are IRBs and regulators doing-- 

Kirsten Martin  3:15  
Oh, interesting.

Katie Shilton  3:15  
--about research ethics in this space? You know, five or six years ago, we just really didn't have a handle on whether IRBs were even seeing this stuff. And then what do the people who are documented in the data think about being part of research studies, right? So what do Twitter users think, what do Facebook users think about their data being used for research? And so we wanted to ask all of those questions, and we needed a big group of people with lots of different expertise to sort of do that together to look at all three stakeholder groups. And that's what we've been doing. And so, you know, over the course of the project, we studied IRBs, we studied users on multiple social media platforms, we sent them vignettes, which is something you and I have done together in the past to sort of get their ideas about, you know, what factors make a difference. And what sort of contextual factors make a difference to them, you know, in saying, Yes, this is a cool use of my data for research, or like, What, what are you doing? No, no, no. Right? And so trying to figure out what makes a difference for people. And then we've been interviewing researchers, that's been sort of the last phase, is interviewing big data researchers about sort of how they're grappling with ethics on the ground. So none of us are--well, lots of us dabble in data ethics, but primarily, we're social scientists. And we're really interested in, like, what are people doing? And then can we kind of map across the three to start to make recommendations? And that's what this paper does.

Kirsten Martin  4:26  
Got it.

Katie Shilton  4:27  
It's our first sort of attempt to say, Okay, we have this body of research about when people mind, which is when--and it turns out people generally mind when they don't know what's happening, which is most of the time, right? Like, Twitter users, Twitter is kind of like the model organism for computational social science, right? Twitter is used in research all the time because it's open, right? It's public-facing, it's easy to get tweets, you can buy tweets from Twitter, right? And so it's used all the time. Turns out Twitter users don't really know that.

Kirsten Martin  4:56  
No. Right, yeah.

Katie Shilton  4:57  
And they expect to be asked for consent--which, you know, expectation is not the same as ethics, right? I'm not gonna say that every study--and so that was the first thing we started to unpack, is well, when do you need to ask for consent, right? When is it ethical, or are there other forms of engagement you could try? Could you try letting people know about your study afterwards, right, sending them the results? Could you talk to community leaders and get buy-in in a particular community as opposed to, you know, trying to get consent from all 6 million people you included in your study, right? Which may or may not actually be feasible. So trying to sort of unpack some of that. One of the concepts we talk about in our paper is awareness, and trying to figure out, you know, how aware are people that their data is being used, and the more aware people are, the sort of less you have to do as a researcher, right, to sort of navigate that space.

Kirsten Martin  5:45  
Interesting.

Katie Shilton  5:46  
But you know, if you're working in a space where--and so one of the things we do in the paper is we have these sort of four quadrants of data collection, and you know, some of them are public-facing. Oh, I don't have the diagram in front of me, I should (laughs) But you know, they're sort of, some are more secret than others, right? 

Kirsten Martin  6:02  
Yeah, right. 

Katie Shilton  6:02  
So the text you send your spouse, if somebody used that in research, you would be really shocked, right? Because you're not expecting that to be. But you know, Twitter stuff is public-facing. People have, they know they have an audience, right? They're creating that differently. It doesn't mean they know it's being used in research, but it's--you sort of have less obligation for that sort of public-facing things where you know you're leaving a trail. And then we have all kinds of pervasive data where you don't even know you're leaving a trail, right? So your DVR is recording all kinds of data about you and your viewing habits and, like, what you like and how loud you like the volume on your TV to be, right? You're not even aware that that's being collected. That is definitely being used for research somewhere. (laughs) But that is what we call, like, that's the spy data, right?

Kirsten Martin  6:44  
Or that we think is private, but is just kind of this exhaust that we're throwing off.

Katie Shilton  6:50
Exhaust. Exactly.

Kirsten Martin  6:52
That then can be used. And I thought--so this is what was so interesting. Like, so the whole entire premise why, like, something's needed as this is--we always call it, like, big data ethics, with the emphasis on big, as if the bigness was important, right? Like, that the size of the data was the distinguishing feature. And I thought what was interesting about the beginning of the paper was you kind of said, Yeah, that's actually not that interesting. You know, like, so we've been dealing with big data sets from, like, the Census Bureau, for example, for decades. What's interesting is not the bigness of the data set, but just this ubiquitous nature of the data that's being collected about us. And also, I didn't know if you could say more about this, because it's almost like this surveillance data that's collected about us both in public and in private. So the DVR example is a great private example that we think is private, but it's actually with another party, it's with our DVR and everybody else. But Twitter, we kind of know, or Facebook, we kind of know is more public-facing. But it's also this idea that it then destabilizes, which I thought was so interesting, this relationship between researchers and subjects, and I didn't know if you could say more about that. That it's really not the bigness. It's the way we collect and then share data that's almost putting distance between the users, or the research subjects, and the researchers. And that's the problem. Is that right?

Katie Shilton  8:14  
Yes. So our traditional--the legislation around how we handle ethical research with humans in the United States, and internationally in many, many places, is based on sort of an assumption of the lab model of doing research, right? It's based on an old assumption that was built for biomedical research. But that is not only sort of in the regulation space how we handle research ethics, right--you get consent with a consent form, that is a thing that you've signed in a doctor's office a million times before; we all sort of recognize it may or may not be meaningful, that's a different conversation--and we have a set of expectations that we've developed over time based on that model, right? So that "research" in people's minds equates with, I'm going to meet you, right, I'm gonna know you, I'm gonna, like, know what institution you're from, I'm gonna decide if I trust you, and I'm gonna sign a consent form, right?

But in the big data world, that's all been moved away, right? Or like, sort of, we use this excavating metaphor: It's been flattened, right? Flattened over, paved over, right, or something, by all of the available data that you can just--you can talk with a company and get your hands on, you can scrape and get your hands on, right? There are a million ways researchers can get data about people now that don't involve talking to those people. And so we don't have a paradigm for ethical use of that data. Like we just--it's a totally different, our analogies start to break down. And one of the things we've seen with IRBs is that's where they struggle, right? They struggle to figure out how to give good advice about how to, like, deal with that data because it's, you're just like--so the analogy we end up making in the paper, and this comes from our social science backgrounds, but we were reminded of nothing so much as anthropologists. And ethnographers specifically, who have long dealt with how to get entree into communities, how to figure out how to study groups of people, and then have had to combine all of these many different kinds of data streams. I mean, they are the instrument, not an algorithm, right? So it's a little bit different. But--well, there are few things that are different. But anyway, the analogy is that they are used to these multiple data streams and the inferences, the kinds of inferences that people don't necessarily see coming about themselves, right? Like, that's very much the provenance of ethnography. But ethnographers are there, right? They're in the space.

And so what do we do when you are the big data ethnographer, right? You are observing all these data streams, you're making meaning from multiple different kinds of data. But nobody knows you're there. And in ethnographic terms, that would not be okay, right? And there have been sort of famous examples of researchers going into spaces and studying them without telling anyone they were doing it, without identifying themselves as a researcher. And generally, that's pretty frowned upon now in anthropology and sociology. 

Kirsten Martin  10:58  
Interesting.

Katie Shilton  10:58  
Yeah, it's, you know, negotiating your presence in the space is a big part of research. And so what we're asking is, could that be true of big data research, as well, right, or pervasive data research, because the bigness is not the thing, exactly as you said. It's the ease of getting it without talking to people. And so what does renegotiating that trust relationship look like? And so one of the things we're doing when we're talking to big data researchers now, data scientists, in our interviews is asking them, What are you doing to navigate this? How are you making yourself known to the communities that you're in? And how are you thinking about power relationships?

Kirsten Martin  11:37  
Yeah, that is the second part.

Katie Shilton  11:38  
Yeah, because ethnographers have long dealt with the history of their field, which is one of colonialism. Ethnography was a tool of colonialism, it was European and American scholars serving as sort of tools of colonial governments. And so, you know, anthropologists in the '70s, '80s, and '90s really, had to disown that history and say, What are we going to do differently? And the answer was a number of things. Among them--so diversifying the field was of course really important to grappling with that legacy, but so is reflectiveness and self-reflection as part of research methods. So in anthropological papers, you get explicit statements about the power of the researcher and their power relative to the community, why they belong, that can be why they were the right person to study a particular community. And we would love to see more of that in data science, right?

Kirsten Martin  12:29  
So more justifying their presence in the work that they're going to do.

Katie Shilton  12:32  
That's right.

Kirsten Martin  12:32  
Like to almost, like, go through this reflective exercise to say, Like, I've thought about these power relationships, and, like, taking advantage of people that didn't know they were giving me their data or the Other that I'm studying, and this is why I can still justify it. So almost that type of reflection in written word. That's interesting, that's interesting.

Katie Shilton  12:53  
And making that a habit, right, making it a habit to say, you know, Here's how I thought about this, here's my theory of power, right? Because power is not--that's not, we realize this is a big ask, but there are some great resources out there for thinking through data and power right now. And so, you know, have a theory of power, have an idea about, like, how you relate to the population that you are studying or the group that you're studying. Or maybe use your skills to study up. And that's another thing that ethnographers have done a lot of: study systems or people more powerful than you, right? So study the architects of a financial system, right? Study judges. Study Facebook. (laughs)

Kirsten Martin  13:30  
Yeah, I often point out that identifying financial fraud is actually ripe for use of AI. And yet, we don't do it, you know what I mean? Like, we're more likely to try to find people committing welfare fraud, you know what I mean? Like, that's the tools of where we're focused. So, to that point, even for non-big data researchers, we can use these tools, you know, in different directions, so to speak. And I thought that was--what I thought was so interesting about it was, one, this kind of, like, general assumption that we have, and you're right, I mean, that we go through IRB, and we have these informed consent, and we assume that there's an established relationship, even if it's quote "remote," between an individual subject and a researcher, where I identify myself from an institution with the IRB number on it, and it's like this establishment of trust, and then they opt into the study. And this distancing that goes on and, like, the lack of awareness, so one, how can we rethink about consent? And then also this interesting idea of, like, the power dynamic that's going on, and how can we think through, you know, reflecting on that and our position in it. It's interesting that the problem is distance, and the pervasive collection of data, and the answer is, like, looking to a group, ethnography, that's close. But the same type of, like, multiple streams of data, but, like, how did they deal with that? 

And when I was talking before we got on, I was thinking, that's when I realized the connection to the way--so companies, the theory of why people give information to companies is actually the same type of theory of why people give information to researchers. And so that's what I thought was so interesting about this paper also is the implications for market. So we assume--like all of our theories of privacy, the flow of information, the economics of information--they all assume that a consumer gives data to a singular company, and that they would never give data to a company that they didn't trust. That's just a blanket assumption that we made in 1960. Because we didn't fathom data capitalism. We just didn't think about the fact that it'd be cheaper to get information about Katie Shilton from someone else, rather than ask her.

Katie Shilton  15:36  
Right.

Kirsten Martin  15:36  
Like, literally in words, put that into the theory, like, it would never be cheaper to go to a data aggregator, you would always go to the person, which is just flat out not correct right now. And you can see the parallels to what you're grappling with here is that, for a company, when you've gathered information about a person and never asked them for it, you have the same question of a big data researcher that, like, What's meaningful consent in this area? I can't ask them, you know what I mean? So how would I think through whether or not--like, how do I think through consent in this area if they would hypothetically give it to me? You know, how do I think about whether I'm taking advantage of a power relationship? Or can I justify it in some other way? And that's where I saw the parallels, actually, to the way companies have to deal with this. If you take a big data researcher at, you know, Milwaukee or Maryland or Colorado, and they're grappling with this before they go through IRB, you could ask the same question about the big data researcher in Facebook or Twitter and what they're doing with the data, if that makes sense. So just based on who's paying their salary, it doesn't really change the ethical obligation to the person that's the subject matter.

Katie Shilton  16:48  
I think that comparison is so interesting, I hadn't thought about it; it completely makes sense. And actually, interestingly, early in the days of this paper, we were using literature from the sort of trusted business lit.

Kirsten Martin  17:02  
Oh interesting.

Katie Shilton  17:02  
And I think you and I had talked about it a little bit, and just because I thought it was helpful, and we actually ended up making a flip away from trust to trustworthiness because we felt like we couldn't solve this problem of trust in research, right? Sort of, that feels like a deep, intractable problem. What we can do is encourage researchers to be trustworthy, and that is sort of the best we can do, right? Or me, you know, personally and the PERVADE team. But I think you're right there are deep, deep problems right now around data and trust, for good reason. And part of it is that our assumptions about how trust works around data that are built into the system are just all wrong. They've all been broken. (laughs) And so what do we do in--you know, what do we replace them with?

Kirsten Martin  17:46  
Yeah, right. And the other--so one is like this problem that we have, and that the second was, well, the whole entire thing of like what's wrong, like the idea that it's not the bigness that's the issue, it's this idea of the pervasive kind of data exhaust that we're picking up all the time, is the ethical problem that you have to grapple with. Because we tend to focus on the bigness, you know, the four V's, you know what I mean? Like, and that's what's differentiating it, and it's kind of like, that's not what's differentiating it, it's actually something else. And so that's something else that ports over to anyone, whether they're at a university or not, when we're thinking about what's the problem. And I thought this is, actually I'll just give this quote because I liked it so much, when you said, "The challenge for pervasive data research"--so any data research company or university--"is not only to center discussions of awareness and power in its research practice, but also to dig out from this disembodiment: to find ways to excavate and retexture modes of trust building." So don't just kind of take the data and run with it. But, like, think about how you can act in a trustworthy way, like this disembodiment of the data as if it's not a person, you know what I mean? Like, so grapple with it as if they're people and think about, like, what would you do in this situation if you were looking at a human? So stop the disembodiment, which I think is just in general a good rule, but I mean, I think it really, it leads to different types of questions that you would ask as a researcher, whether a university or a company, wherever it might be.

Katie Shilton  19:15  
Yeah.

Kirsten Martin  19:16  
Well, I could go on and on about your paper, but I know you've got other things to do. But I would, I know you have more stuff that you're always doing, so I'll definitely come back for another paper, the next thing that comes out of the project since it's ongoing. But I always like to ask, because we're both in the era generally of tech ethics, kind of broadly construed, but like who is it that you're currently looking forward to hearing from or you're like, Oh, that's so interesting, like I wish I thought of that. Those are the people that I always find fascinating, that I try to see [because] they're so insightful. So I didn't know if there's anyone that we should keep an eye on.

Katie Shilton  19:47  
Oh yeah, definitely. So a couple of folks spring to mind sort of immediately. The first is work in a more philosophical vein. So I'm not a philosopher by training, but I'm really inspired by work in data ethics, information ethics. And so Anna Lauren Hoffmann is at the University of Washington, and she has a recent paper around ideas of inclusion as not--this is in New Media & Society--of inclusion as not a solution in this--so, you know, inclusion tends to be sort of like a first line--and we see this with participation, too, but her paper is about inclusion. And about the sort of double-edged sword of visibility, right? Including people in datasets is not necessarily the answer-- 

Kirsten Martin  20:31  
The answer. Oh interesting.

Katie Shilton  20:32  
When the problem is, and she uses this term data violence--like the violence that is done to people from data and datafication--inclusion is not a solution to that problem, right? And so it's a really nice paper, as I think we think more and more--especially for researchers, right-- you hear this a lot, this idea that, you know, researchers are, Oh, no, we don't have data on X or Y group, right? And they're not in our--and then we can't make new knowledge about them. And yes, that can be a problem. Absolutely. Medical contexts, you know, there are all kinds of fairness problems, you know, with not having a group in your data. But that is also, in an age of datafication, not always a bad thing, right? 

Kirsten Martin  21:08  
Yeah, yeah.

Katie Shilton  21:10  
Like, I think researchers have so much to grapple with there. And so when I talk, you know, about their use of power, like that's one place I send them is to Anna's work. So I think that's really exciting. Another really sort of interesting, exciting area of work that I can't wait to see more of is coming out of a group at the University of Washington. I think the PI is Emily Bender, but Amandalynne Paullada is the first author as a student, I believe a first author on a really interesting paper around data and its discontents. And so it's called "Data and Its (Dis)contents." 

Kirsten Martin  21:44  
That's a good name.

Katie Shilton  21:45  
Yes, isn't it? (both laugh) And it's about the training data that machine learning is based on and the sort of practices that go into making training data because--so you know, I'm often talking to researchers about making good data because they're making their own data, but so much ML data, machine learning data, is sort of inherited, right? There are these big training datasets that are just kind of out there. And like, who knows who's in them, or what's in them, or what the conditions of collection were, the labor that went into them. And so there's sort of groups that are starting to ask those questions and study those data practices. And I think that area of work is super, super exciting.

Kirsten Martin  22:23  
Those are great recommendations. I love that work because it kind of shows--I think people think that the data that the AI and machine learning is being trained on is somehow, because it's big, it's good. (cross-talk) And so the badness just gets like taken out in the wash or something? Like, they just think it's noise, you know, and not realizing that it was curated by humans, labeled by humans, you know what I mean? Like so, I think that kind of work is just so important to, like, an underlying assumption that goes on when we think about these models.

So, you guys all probably heard my dogs wanting to go outside, they're wrestling with me with my one hand. (both laugh) If you could see me--well, you [Katie] can see me--but my hand is trying to keep my one puppy silent. But I really appreciate you taking the time to talk about this. I love the paper. I think if anyone was gonna take anything away from it, the idea that it's not the bigness that's the problem, and that it's the way that the data collection, the pervasive data collection is the problem is, like, the biggest takeaway to undermine the "big" focus on data that we have. And so--I mean, and the rest of it's good, too, but I would just say that was like a kind of a big, insightful moment. And I thought it was hugely important also to set up the rest what the solution was, then, is not--look, we're not going to solve the "big" problem, what we need to solve is the other problem, which is this pervasive hidden data collection, and that it puts this distance between the researcher doing the data analysis and the subject. And that's true of all of us.

Katie Shilton  23:49  
Puts us in the team of surveillance or the camp of surveillance. 

Kirsten Martin  23:52  
That's right.

Katie Shilton  23:53  
What we are doing is also surveillance, right? And so when we have people reacting against surveillance, sure, by governments is a problem. But as we've seen and as some wonderful scholarship's illustrated, surveillance by corporations is also a problem. Surveillance by researchers could be a problem, as well. We're using those tools, so we need to be really, really reflexive about how we're using them. Because there are tools that are associated with a lot of things people don't trust.

Kirsten Martin  24:17  
That's right. That's right. And so, like, you have to understand that if you're buying into that surveillance data, that you're also becoming part of that system, and how does that then have implications for all research and you as a researcher, which I think is great. Well, thank you. I really appreciate your work. I do so much. And thanks for taking the time to talk about it. It's been awesome.

Katie Shilton  24:37  
My pleasure. Thanks for having me.

Kirsten Martin  24:38  
Thanks so much.

(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.

1. Social Media Addiction: Adding Insult to Injury (April 20, 2022)

Guest: Vikram Bhargava (George Washington University)

Transcript

Kirsten Martin  0:03  
Welcome to TEC Talks, a podcast about the impact of technology on humanity. I'm Kirsten Martin, the director of the Notre Dame Technology Ethics Center, or what we call ND TEC. In these discussions, we discuss an important idea, paper, article, discovery in tech ethics, and today, I'm so happy to be joined by Vikram Bhargava. Vik is an assistant professor of strategic management and public policy at the George Washington University School of Business. Vik's research centers around the distinctive ethics and policy issues that technology gives rise to in an organizational context.

In the series, our goal is to take one article, idea, or case and examine the larger implications for the field of tech ethics at large--including what someone who, for example, is hanging out in North Carolina, reading hard copies of The Wall Street Journal and The New York Times, should know about the idea, just as an example. Happens to be my father that still reads hard copies of The New York Times and The Wall Street Journal.

And today, I wanted to take a deeper dive into that article you had with Manuel Velasquez, the "Ethics of the Attention Economy: The Problem of Social Media Addiction." I just so enjoyed this idea when I first saw it presented and then when I saw it in print, and I just thought you did, like, such an interesting job of figuring out why it feels kind of more wrong that these companies get users addicted to their product. And so when we see with the Facebook Papers, right, like this continuing more evidence of this issue around social media and user engagement as the optimized outcome, and I thought, could you say a little bit more about, one, I guess how the wrongs are similar to other industries that we're used to thinking of--like sin industries of, like, tobacco or gambling? But then also, like, what's more wrong? Does that makes sense? (laughs) Like, what's the more "wronger" thing that's going on here?

Vikram Bhargava  1:48  
Thanks for having me, first of all. You know, I am excited to be here. And you actually get to the question that inspired me in a lot of ways, which is that, Look, we've had businesses that have sold addictive products for decades now, longer even. And it's not enough, for example, to say, Well, look, here's a product that's harmful. Because there are lots of things that are harmful that might not raise unique issues. So the question I'm interested in is, Well, what, if at all, is unique or distinctive about social media companies addicting their users? And there are a lot of ways in which social media companies resemble other more familiar sort of "sinindustries," so to speak, and there are ways in which they're different. So I wouldn't quite put it as they are worse than these familiar addictive products--let's say cigarettes or alcohol or even harder drugs--but rather that they raise certain issues, ethical issues, that were not seen with these other kinds of products.

So, to give you an example of this, you know, I just got a cup of coffee this morning at Starbucks. And let's suppose that I finished drinking this cup of coffee at the coffee shop, and I threw it away. Now, I'm using this example that I'm going to build on here to illustrate a unique component of social media, which is called adaptive algorithms, which is the idea that the more you use the platform, the more the platform itself adapts to you, which gets you to use it more, and then this gives it more data, and this further adapts to you. So it's a tailoring process to the individual. So now returning to the coffee example. So, you know, let's say I finish up my cup of coffee, I toss it out at the coffee shop, and unbeknownst to me, they go in the trash can, they grab my cup of coffee, seal it up in a Ziploc bag, and send it off to the regional lab for processing to figure out a little bit more about the biological attributes that incline me toward addiction. So then they sort of mildly tweak their coffee recipe to make it slightly more addictive the next day when I come in to have coffee. So again, I come in to have coffee, I finish my cup of coffee, slightly more addictive, again I throw it away, and then again, they seal it up in a Ziploc bag, and they send it off to a lab for processing. Now this process happens over and over and over and over until, let's say a few weeks later, I come back to the coffee shop. And now I'm given a highly addictive cup of coffee. And not only that, they use me against myself.

Kirsten Martin  4:28  
Mm hmm.

Vikram Bhargava  4:29  
They sort of implicated me in the very process of addicting myself. So it's something like, Not only are we going to do this bad thing to you, we're going to get you to help us do this bad thing to yourself.

Kirsten Martin  4:41  
Right, right, right, right.

Vikram Bhargava  4:42  
Now a key point here is, I was aware at the front end that caffeine has some mildly addictive potential, right? So I might have even been aware of that. But that doesn't mean that I endorsed being given a highly addictive cup of coffee, nor being used in that very process. So, you know, a lot of times people might point out, Well, aren't users sort of consenting to this in their agreement? Well, you know, the scope of what they're consenting to is relevant in a lot of ways, right? The social media that we signed up for, at the outset--Facebook was a very, very different product at the outset than it is right now.

Kirsten Martin  5:17  
Right.

Vikram Bhargava  5:17  
So you know, this example is a way in which--you know, you might think, Well, I still don't see why this is different from cigarettes; isn't it the case that with cigarettes, the more you use cigarettes, the more addicted you become? Now, it is true that the more you use a given product, the more addicted you become to it. But that's not the point I'm making here. The point, rather, is that the more you use social media, the more addictive it becomes. So it becomes--it would be like a cigarette that each time you smoke the cigarette, the cigarette itself increased in nicotine content, and you help them do that. So that's the way in which, I think--that's one way; there are other ways in which social media addiction is unique--but you know, that is a salient way, I think.

Kirsten Martin  6:01  
I love it. So, like, to take the cigarette example, it'd be like if it was monitoring on each of your cigarettes--but they don't tell you this. Because they're not--you still buy the same pack of cigarettes, it's Vik's pack of cigarettes, it looks on the outside like the same pack of cigarettes, it's just that the actual nicotine amount is now tailored to you, to get you to want more now. Now maybe you need a hit early, and so the rush is earlier in the cigarette versus someone else that needs the hit later, and the cigarette, they put the nicotine way at the end or something along those lines. But it's this tailoring based on your input. But the product looks the same. Like, for all you know, you're just getting the same cigarette over and over again. And yet they're kind of using your own information to get you--for you to have a negative effect, in some ways.

Vikram Bhargava  6:47  
Yeah. So you know, it's something like, they are of course harming you in doing this. But there's an added insult to the harm of using you against yourself, right? It's something-- you know, this is an entirely different degree example, of course, but it's something like not just murdering somebody, but getting the person you're about to murder to dig their own grave before you do.

Kirsten Martin  7:08  
Right, right. Yeah. And I think you call it the "insult to injury argument" in the paper, right?

Vikram Bhargava  7:14  
That's right.

Kirsten Martin  7:14  
It's demeaning in a certain way, in a way that's different than just addiction, addictive products or harmful products that may hurt you. But you kind of know what's going on with cigarettes and you understand what the, you know, with the warning labels. But this kind of, like, adding insult to injury is a level of demeaning to use your own information to further addict you, which I thought kind of captured why we, you know, why do we think it's wronger? You know, more wrong? Why do we have this visceral reaction to this? And I was like, That's it, you know, how are they--it is different in a way. 

Vikram Bhargava  7:48  
Right. You know, like, in a lot of ways, I think people have sort of correctly accused me of having an understated public policy response that I defend in the paper itself, right? You know, I don't call for a heavy-handed regulatory response in the paper--in part, not because I don't think it's in order, but in part because I think there's a lot more empirical study that's required, both of the very phenomenon of social media addiction but also, you know, I think any time you want to legislate or regulate something, there are important empirical questions that need to be addressed, right? And I of course didn't have the--you know, they haven't been addressed yet.

Kirsten Martin  8:26  
Right.

Vikram Bhargava  8:27  
There is a way in which even with--you know, a lot of times people will say, How is it that this is addictive? With cigarettes or heroin, this is something that's affecting your body directly. But the kind of key point is that there's nothing, you know, special about the fact that it's affecting your body. The important thing is certain brain changes, and brain changes can be effected through the body or otherwise, right? We have certain responses, dopamine responses and other kinds of responses, whether or not it enters our body through, you know, a physical injection, and we can have psychological responses to this, as well. This is why, you know, gambling is widely regarded to be an addiction now. Now an important way in which even on slot machines--which people often draw a parallel to now with social media--with slot machines, they're not, at least in the United States, legally permitted to change the odds on you while you are playing.

Kirsten Martin  9:22  
Right.

Vikram Bhargava  9:23  
There's no such issues with social media, for example, right? Let's say they sense that you're probably going to log off, they can just hit you with the exact kind of content that would prevent that at that point, with no issue.

Kirsten Martin  9:37  
And I suppose the other thing gets to the exploitation argument that you make. We can avoid--I mean, I think someone who's addicted to gambling would say there's always opportunities to gamble on, like, whether a light turns red before a certain time. So putting that aside, there aren't slot machines everywhere, and you can avoid cigarettes now more than ever. Which gets to something else that's kind of different about social media, right, than other addictive industries that you've mentioned.

Vikram Bhargava  10:01  
That's right. So I think there's a sense in which--actually, in philosophical discussions of exploitation, the relationship between a drug dealer and addict is often given as an example of exploitation. This is a sort of garden-variety way in which, you know, they could jack up the price, this person is desperate, so to speak.

Kirsten Martin  10:22  
Right.

Vikram Bhargava  10:24  
Now, there's a way in which the kind of exploitation that's going on with social media addiction is much more invidious for the very reasons that you sort of sketch. Which is that--although you know, I don't mean to suggest that it's easy for an addict to quit cigarettes or to quit alcohol; it is, of course, a challenge--but rather, the point is that one can get on fairly successfully in contemporary life without ever smoking cigarettes, right? A little more difficult with alcohol, but even that, it doesn't really hinder too many professional, social opportunities, etc. And if we're talking about the case of teens, for example, you know, maybe there's going to be teens who sort of sneak into their parents' alcohol cabinet, or you know, take a puff of a cigarette while they're at a party. But nothing bad would happen if they did not do that until they like, say, graduated from college or graduated from high school.

There's something different in the case of social media, which is that every college class that I've taught right now--so these are, you know, Gen Z students--there's always at least 10% or so of them who have had history teachers who've given assignments such as, Make a Twitter profile for this historical figure. And they're, you know, important--in order to succeed on school assignments in a lot of ways, it requires being a part of social media, right? And so it's not this, you know, these deviant teens, it's the very, you know, the ideal academically, socially successful teens, in a lot of ways have legitimate grounds to be on social media. And, you know, it's not just children. I mean, I think these days there are more and more politicians who are announcing, you know, global governance decisions, public policy decisions, and if you want to be a sort of participant in the democratic process, in a lot of ways you need to be attuned to this on social media. So if you want to be, let's say a part of--there are plenty of, for example, job search websites that require you to log in with a social media platform, dating apps, etc.

Kirsten Martin  12:26  
Right.

Vikram Bhargava  12:26  
So there are a lot of different social goods that either require you to join using a social media platform, or it's the sort of most straightforward way to do so. So the way in which this is more, you know, the exploitation is more invidious, is that there are legitimate grounds for why we need to be on the Internet more generally, and also even plausibly on social media in particular. So it's that even if you're not addicted yet, it's just they have innumerable opportunities to get you, so to speak.

Kirsten Martin  12:55  
Right, right.

Vikram Bhargava  12:56  
And this is in a lot of ways different, right? We wouldn't lose out on much if we never smoked cigarettes in our lives. But arguably--I mean, certainly with Internet in contemporary life, and you might correctly say, Well, look, the Internet is not the same as social media. But the trouble is, you know, we know from the research on environmental cues, you're just one step away. It's like, you know, you can go to a bar as an alcoholic and have alcohol [be] there, but it's the environmental cues make it all the more difficult. And, you know, this is a sort of perpetual challenge, basically, for social media, given that there are a lot of legitimate uses of it.

Kirsten Martin  13:34  
Right. Yeah, and especially--I mean, this was written, if I remember, I mean, I saw it presented before the pandemic--but it was written and published before the pandemic, in which social media has only become more important, especially for teens and young adults, since the pandemic, because that's the only way that they actually are socially interacting. So it was actually written before that, but you could even make a stronger argument now, in many ways, with the reliance on people being remote. And now that they're used to it, it's just so much easier for them to interact that way, and they have friends from long distances, so it's kind of become even more normal.

And so--well, you and I are in business schools, so of course, the first question is, Why can't the market just fix this? And I think you have a great answer for that, that there's reasons why the market can't fix it. You know, because that's what we're supposed to ask, right? Like, what's the market force that could actually get this thing to move? Or what's that fix? And so I think your point is, really, there's not much incentive right now around the market to push to not focus on engagement, right?

Vikram Bhargava  14:32  
Right. So there's a way in which right now the debate around social media addiction--and not just addiction, but a bunch of the different woes associated with social media-- there are people who sort of say, Well, you know, there are two options here. You know, we either get rid of this attention economy business model in general, and get rid of social media, or you know, if we have social media, just no attention economy, you know, business model, or you know, something like, we just have to get rid of social media. And I think there is an intermediate position in some ways.

Kirsten Martin  15:10  
Yeah, yeah. Maybe the NFTs will all fix it. (both laugh) I'm just kidding. Yeah, I'm just kidding. I really liked this paper. I mean, what I like about it is it kind of, it's a microcosm of the wrongs of using your data against you. And this happens in other areas and other places where people are using someone's, an individual's own data, but not in their interest. And I thought that this was--like, the way the argumentation went in the paper, first of all, it's really clear. And then also, I just like how it really encapsulated this idea of this adding insult to injury of using your own data against you, which really resonated of kind of saying, Yes, that's what feels more wrong, you know? Like, that's the thing.

Vikram Bhargava  15:52  
Thank you. And actually, you know, that point that you mention is actually, I think, directly related to this incentive structure. The trouble right now is that, of course, social media has played a role in important positive social movements, Arab Spring and so on.

Kirsten Martin  16:07  
Right.

Vikram Bhargava  16:07  
So I don't mean to suggest there haven't been positives associated with it. There is a reasonable challenge--some people say, Well, look, we've had attention economy business models before; television and radio are attention economy business models, right? As is journalism.

Kirsten Martin  16:20  
Yup, yup.

Vikram Bhargava  16:21  
So there is a real challenge of, Well, what's going on here? And I think there is an important difference, and this is that. So as I see it, the way forward--you know, this is a sort of hasty sketch--is that if social media companies want to continue on the attention economy business model, then I think that the tailoring needs to be limited. It shouldn't be hyper-tailored to the individual. If they want to continue to tailor it to the individual, then I think it should be on a membership model, right?

Kirsten Martin  16:53  
Oh, interesting. Right, it's because, it's the, it's exacerbating. So the one, the attention economy model of you get paid based on engagement, so I need to keep you on longer. 

Vikram Bhargava  17:04  
Right. 

Kirsten Martin  17:05  
Dovetailing with this insult-to-injury adaptation of the engagement of, like, kind of how you actually are affected by the social media, that that's just--it's like a tornado, it's just making it worse. It's like two streams coming together. And so I see what you're saying. And so you can do one but not both. You know what I mean?

Vikram Bhargava  17:24  
Exactly.

Kirsten Martin  17:25  
So pick your poison is one way to moderate the kind of evils that come with the wrongness that comes with both. That's super interesting.

Vikram Bhargava  17:33  
So as like a slogan, If attention economy, then no tailoring; if tailoring, no attention economy.

Kirsten Martin  17:39  
Exactly. Yeah, right, right. I mean, I think what's interesting about that is you could see that a lot of different places. I mean, that's what I liked about the paper is, like, it's a very specific paper on one area, but you can see it with the attention economy and adaptive architecture as kind of one of those, like, constantly--and the use of consumers' data against them--as this constant problem that we have. You know, so it's kind of a, it's not just social media addiction, you can take these ideas and put them onto other things.

So yeah, well, I'm always excited to hear your stuff. But, like, so tell me--I want to wrap up just because I know you are super busy--but who are you paying attention to right now in tech ethics? Or you kind of look forward to hearing what they write about? There's tons of people in this space; it's kind of an area where you can just highlight a few people that we should be watching out for.

Vikram Bhargava  18:25  
That's right. So you are right, that, you know, there are tons of new people, new talent coming into the space. And there are, you know, a host of different issues to address. I think one of the papers, and I think I've mentioned this to you before, this is a paper and also a book, is by a political theorist, Sonu Bedi up at Dartmouth.

Kirsten Martin  18:45  
Oh, yeah, he's great.

Vikram Bhargava  18:47  
And he has a book and a paper, the book is entitled Private Racism. And he has a paper about the role of race-based filters in dating app algorithms. And, you know, a lot of the dating apps right now, they offer various different race-based filters where you can filter out the prospective romantic or sexual partners by race. And he explores the ethics and law of this in a really fascinating, clear way. And I think it is, you know, it's a topic that affects huge chunks of our students, huge chunks of the population. This is how people are dating right now. It's a core part of contemporary life. And in a lot of ways, I think people have sort of regarded this aspect of our life as sort of insulated from the possibility of bigoted preferences. And it raises, I think, fascinating, challenging questions that, you know, in some cases, uncomfortable ones, but I think it does it in all the right ways. So I think Sonu Bedi's work on this topic of private racism I think is really quite excellent.

Kirsten Martin  19:50  
Yeah, that's a great suggestion. He's great. So thank you. Well, gosh, let me know when you have another thing out. I always watch for it. And I know we always, we see each other when we do our workshops together, so I always am keeping track of what's going on. So I really, I can't thank you enough for coming and kind of just doing a little bit of a deep dive on one idea just to make it kind of broad for the entire tech ethics community. So I really appreciate you taking the time out of your day, Vik, and thanks so much, and I'll see you soon.

Vikram Bhargava  20:16  
Thank you so much for having me. It was fun.

Kirsten Martin  20:19
(voiceover) TEC Talks is a production of the Notre Dame Technology Ethics Center. For more, visit techethics.nd.edu.