In this lecture, we're going to talk about AI governance. How do we ensure we can get the benefits of AI, without experiencing some of the risks we discussed. The three main principles that I've advocated in my book are humans guide to machine intelligence, and their user control, transparency and audits. Let's look at each of them in turn. The first is controls. This is the idea of giving users some control over the way algorithms make decisions for or about them. In other words, keeping a human in the loop. Let's consider some examples. First let's look at Facebook newsfeed. Now Facebook's newsfeed faced some criticisms in 2016, because there were some false new stories being circulated on the platform. In response to that, Facebook implemented many changes. One of the critical changes was that, they allowed users to be able to flag posts in their newsfeed that they felt were either false news or offensive in some way. This feature gives users control, and allows users to give feedback to the algorithm itself about certain decisions it's making that is problematic or incorrect, and this helps the algorithm learn as well. In fact in reality, over the last year, this feature has actually helped Facebook detect many problematic news posts. Now, even though control sounds like a simple idea, there are many nuances here, and one has to approach it with caution. An example of why one needs to approach it with caution comes from, again Facebook. In 2015, Facebook released several mixer-style newsfeed controls, so users could control precisely what kinds of posts show up in their news feed. For example, they could say, show me more posts from these kinds of friends or less posts from these other friends, they could say, show me more or less relationship status messages, or don't show me messages with profile changes and so on. Facebook tested this feature with a few users. They found that user satisfaction among these users actually grew. However, user engagement went down, meaning that these user spent less time on Facebook, they engaged with fewer posts, they clicked on fewer posts and liked fewer posts. All of this suggested that the algorithm, was not as effective at showing users the kinds of posts that they would find interesting. We have actually seen this play out in multiple other settings where algorithms and AI in particular, tend to perform quite well in an autonomous mode, but when users can have some control, some of the performance actually dips. So while I'm suggesting it's good to give users control, giving users a lot of control has one risk, which is that performance can go down. One has to really think about how do you design a system, that allows users to have control, and help flag problems with the algorithms when they arise, but at the same time not suffer from some of the performance issues with too much user control. Now my research as well as the researchers, many others suggests that there is a way to design some of these controls. A recent study conducted by some of my colleagues at Wharton evaluated the impact of user control on trust. In their experiment, users were asked to predict the score that high school students will receive in standardized tests, based on some information they had about these high-school students. Users were allowed to consult algorithms. These users or subjects in the experiment were divided into four groups. The first group, had no control over the algorithm. They had to decide whether or not they wanted to use the algorithm, but if they decided to use the algorithm, then whatever the algorithm suggested, that would be their choice. Groups 2 and 3 had very limited control over the algorithm. For example, group 2 was able to overrule the algorithm or AI, in some instances, but very few instances in general, and group 3 was allowed to overrule the algorithm or the model, by changing the algorithm's predictions by a small amount. The last group had complete control over the algorithm. Then they evaluated users trust in the algorithm. How often did they want to use the algorithm, or what percentage of these users wanted to use the algorithm. They found that users with no control, had low trust. They were less willing to use the algorithm. Users with even a little bit of control, had very high trust. They were willing to use the algorithm much more. Interestingly, they found that the amount of control didn't matter. Whether users had little control, or a lot of control, the trust levels continue to be high and similar. In short, what this research is showing, is that a little bit of control goes a long way in establishing user trust, but you don't necessarily need a lot of control. On the other hand, research suggests that performance sometimes drops with control. Bringing the two together, the implication is that we need to give users just enough control, so they can overrule the algorithm or have some control over how the algorithm makes decisions for or about them, but it doesn't mean that, the algorithm cannot make decisions on its own. In particular, in this instance, the idea was to give users the ability to overrule the algorithm, when they saw there was a problem. The idea on Facebook again, is that the algorithm figures out how to rank order posts in a news feed but users can give feedback and say or indicate when certain problematic post show up. Here again, the idea is that when things go wrong, users can give feedback. The idea of user control is essentially to give users enough options in the product design and in the user interface so that they can guide the algorithm's performance and in fact, even overrule it when they notice something is going wrong. The next principle is that of transparency. Which is the idea of giving users enough information on how algorithms are making decisions for or about them. Now, transparency is an interesting idea and there are many interpretations of it. One interpretation of transparency is the idea that we have to reveal the source code of the algorithm. This is also sometimes referred to as technical transparency. When there was a crash in the US stock market in 2010, there was an investigation done of the stock market flash crash, and the investigation suggested that automated trading algorithms had a role to play in the flash crash. In 2015, the CFTC, or the Commodities and Futures Trading Commission, had a ruling and suggested a regulation where they suggested that the Department of Justice could force traders to reveal the source code of their algorithms. The traders and most industry professionals resisted this because it would cause them to reveal their proprietary IP, and in fact, in the end, the ruling was modified so that these companies were not forced to share their source code. In 2017, New York City also proposed a bill on automated decisions. This bill would have required that all vendors who submit software that is used by the city government and that is used by the city government to make automated decisions, would be required to reveal their source code and make it publicly accessible. The vendors again, resisted because it would cause them to give up their proprietary IP, and furthermore, it would allow hackers to evaluate their code and identify vulnerabilities in their code and attack the code itself. Now, in this instance as well, the ruling was modified or the bill was modified, and it was decided that technical transparency is not the best solution. In many instances, we have seen the idea of transparency should not be technical transparency meaning revealing the source code to the public. In fact, it's not even clear, it adds any value because it's not clear an average layperson can actually evaluate the source code and act on it. Now the kind of transparency that might be useful is something a little more high level. A recent study by researcher Rene Kizilcec evaluated how user trust changes with transparency in the algorithm. In the experiment that was run by the researcher, students were provided a grade from an assignment and the grade was determined algorithmically. Some students receive no information about how the algorithm decided its grade. Trust was very low. Another group of students got some minimal information on how the algorithm decided grades. In other words, it was given high-level information on the factors that the algorithm weighed and some basic explanation that provided them intuition on the algorithm's design. Trust went up dramatically when this kind of transparency was provided. A third group received very high amount of transparency, where they were given detailed formulae that was used in the algorithm and very detailed specifics. Trust actually remained low with this group. This is partly because these users were not able to evaluate that complex information and make use of it. All of this suggests that for end-users, again, we don't need a lot of transparency we just need basic information such as: Was an algorithm used to make decision? What kinds of data are being used by the algorithm? What variables were considered, and what variables were most important for this decision? This is an idea that is becoming more and more important. There's a whole sub-field in machine learning which is focused on interpretable decisions or interpretable machine learning. There are two main ideas within this field, global and local interpretability. Global interpretability is about providing high-level feedback to users or anyone else about what are the most important variables or factors that are driving the algorithm or the model's decisions or predictions. Local interpretability is about providing feedback on what are the most important variables or factors driving a particular decision. For example, if an individual's loan application was denied, what were the specific factors that resulted in this person's loan application being denied? A lot of regulation is moving in this direction. I previously mentioned EU has a regulation called GDPR, which gives consumers a right to explanation. If companies are using advanced machine learning such as neural nets or random forests, then they would need to provide explanations to consumers, and the field of interpretable machine learning will become relevant because it allows them to give high-level explanations to consumers about the factors driving the model's predictions, or decisions. Now transparency has another interesting aspect to it. We've talked about transparency as it relates to the end consumer, but there's also transparency as it relates to managers and data scientists. Now, when a data scientist is using a very complex model, like a neural net to make a decision, like loan approval decisions, even the data scientist might not know what are the factors that are driving the model's performance. Interpretable machine learning is also relevant not just for the end-user, but also for the data scientists or managers who are actually deploying these algorithms. Here again, global interpretability and local interpretability matter. Global interpretability as I said, is essentially this idea of whether we can explain at a high level what are the most important variables that are driving a model's predictions. For a loan approval decision, it might be feedback that the loan approval is determined primarily by the applicant's income, and second by their credit history, and third by some other variable and so on. Local interpretability as I mentioned is about, getting feedback on the most important variables driving a particular decision. If my loan application was not approved, then local interpretability would involve explaining what are the factors that resulted in cortex loan application being denied. Now, this is becoming an increasingly important field within machine learning. There are many open source tools and third-party vendors that are offering solutions that help companies offer interpretability or add interpretability as part of their machine learning activities. This can not only provide explanations to consumers and increase trust, it can also increase trust among managers who are trying to deploy these systems, and they can also be valuable tools for debugging. The third principle that I advocate is the idea of auditing algorithms, especially in high stake settings. There is in fact, an ongoing regulatory proposal here in the US Congress, which is called the Algorithmic Accountability Act. Which if passed, would require very large companies to actually evaluate their high risk automated decision systems, meaning machine learning models that are used to make high-risk decisions like loan approvals or resume screening and so on, to have them audited for things like accuracy and fairness. Now, this regulation has not yet been passed, and it is not clear whether it will be passed or not in its original form. But independent of that, forward-looking companies should not wait for regulation, but should take proactive steps so that they can win consumer trust, prevent problems with the algorithms, and also ensure that their systems are robust for the long-term. Let's talk a little bit about what an audit process might look like. An audit process begins by first creating an inventory of all machine learning models that are being used by an organization. Next, for each model, we identify what use cases are as the model being used for. Sometimes the model might be used for very simple decisions. Sometimes a model might be used for more complex decisions. We identify the use cases. We also identify who's the developer of the model, who's the business owner of the model, which division owns it, which individual in the division is perhaps responsible for it? Each model might be given a risk rating, which might assess what are the social and financial risks if the model goes wrong. This might be the basis of a decision of whether an audit is needed. If the risk rating of a model is relatively low or moderate, perhaps we don't need an audit process. But if it's a high risk model, then an audit process may be initiated. The audit can be conducted by internal experts or by an external expert who's broaden. An audit would look at a number of factors. For example, they would start by looking at the inputs that are going into model, what is the quality of data? Are there biases in the training data? It would look at the model itself, it would benchmark the model against alternative models and confirm it does better than the alternative models, it would also stress test the model against simulated data and confirm that the model works well even when the data looks different than the training data. Lastly, we have outputs. The idea here is to look at the decisions or predictions made by the model, evaluate it, look at explanations or interpretations so that we understand if the model is weighing the right factors more heavily. We might also try and look at outliers in the predictions. This is all collectively part of an audit process. As I've mentioned, there are many risks with machine learning models that are automating decisions. Issues range from bias to not understanding the models, and an audit process can go a long way in stress testing the model before you actually deploy them. In summary, there are many ways to think about managing risks. Control is an important aspect. Make sure there's a human in the loop, give some humans control over the model. The second is transparency, which is the idea of explanations about how the model works. Lastly, we have the notion of audits so as to stress test the model. In fact, when you design the team of data scientists, you can think about a model developer that is focused on developing the machine learning model. Then you might have a data science QA process. This is much like when software development, you have software engineers creating the software and then you have testers or test engineers who are testing the software. Similarly, you could have a data science QA process that is testing the model. For super high-stakes model, there might be a more deeper or more involved testing, which is essentially what an audit is all about. That is a set of frameworks that can help us manage or govern some of the risks associated with AI.
