[NOISE]. >> So today we're going to be talking about how to run a usability study. This is a fundamental part of analyzing the usability of a system, whether it's a security system, a website, or a mobile application. Usability studies analyze how useable a system is. When you're evaluating it, you want to look at the aspects of usability that we talked about earlier. That's speed, efficiency, learnability, memorability and user preference. We're going to study those by having users sit down in front of your system, use it, do some normal tasks, and collect data on each or some of those aspects of usability. When you're testing the usability of security, it's important to realize that security is rarely the task that users set out to accomplish. Security should just be part of the background. So good security is really a seamless part of other tasks that users are trying to accomplish. And we'll see some examples of good and bad tasks that can test how useable the security features are without trying to put them too far forward. So the useability study process goes like this. First you define tasks, and their importance. We talked about tasks in the first week of class. And the important thing here is that we come up with a set, usually around five, maybe a few more or a few less, that represents the normal things people would do with the system. You want to find the most obvious tasks and put those first. For example, if you were studying Google, one of the main tasks you should have on your usability test would be to have people conduct a web search. After you develop your set of tasks, and we'll talk more about that in a minute, you need to develop questionnaires. Some of those will collect data ahead of time, that has things like demographics and experience. Post test questionnaires that go after the study, ask people for their feedback about how easy it was to use the system and what they liked and didn't like. When you select tasks, you're really doing the most critical part of running a usability study. People will sit down and they'll do the task that you give them. So they need to be well chosen and well articulated. As I said before, you want to pick tasks to be the most important things that a person would do with an interface. When you're presenting tasks, you want to make sure that you avoid some common pitfalls that people often have when they're learning how to do useability studies. First, you need to present your tasks as an actual task action, and not a question. This is a common mistake that people make when they're learning how to do usability studies. A good task would be something like create an itinerary with flights from BWI to Las Vegas, departing December 30th and returning January 5th. It's specific and it's an action that the user would go into a website to take. A bad task is, how many flights are available from BWI to Las Vegas, departing on December 30th and returning on January 5th? People don't go to a site to count the number of available flights. So your task shouldn't ask them for a number. People do go to a site to find flights that are available, so the task should reflect that. And this is a lesson you want to keep in mind for all of the work that you're doing when you're building tasks. Ask yourself, is this something the user would go to the site to do? And really look at the way your task is presented. You also want to be specific. So, the previous example I gave you is specific. Another good example would be to find the calories, vitamins and minerals in a one cup serving of broccoli. That's a specific task. A bad task would be, find nutrition information. You haven't said what information you want them to find, you don't know what the food is they're looking for, and that leaves users kind of wondering what they're supposed to do. They don't know when they've completed the task and they're not sure what you're looking for. So, your user shouldn't have to do any work to figure out what you want them to accomplish. That said, it's important that your not giving instructions in your task. So, another example of a good task would be to use Google Maps to find a street view of the Empire State Building. A bad tasks that is too specific would really give instructions on how to do that. It might say, like go to maps.google.com, in the search box enter 350 5th Avenue, New York, New York, click on search. Then using the zoom scale on the left side of the map, click the person icon to see street view. That's just telling the person exactly what to do. So you're not testing the system, you're really testing if the user can follow your instructions. So you want to make sure that your tasks are a thing a user would come to the site to do. They wouldn't come to the site to follow that set of instructions, but they would come to find that street view. Finally, you don't want to be vague or provide tiny insignificant tasks. So another good task would be, using Google Maps, find a close up view that shows just the block with the Empire State Building. A bad task would be, zoom on a Google Map. It's true that people might zoom when they come to the site, but the reason for going to Google Maps is not to execute the zoom feature. That's used as part of some greater task. So, you want to make sure that you don't focus on little features that you want to know the usability of and make a task that's not actually a realistic thing that a person would come to the site to do. You want to choose represented tasks that reflect the most important thing users would do in the interface. So for Google, as I mentioned, that could include things like searching the web, searching for images, creating an account, looking for directions. Bad example of task could be to do five basic web searches for different things, so search for broccoli and search for information on sheep and search for the date of the next full moon. You're just having them do the same task over and over again with slightly different parameters. So your task should really cover the breadth of things that people are going to do on the site and highlight the most important ones. When you're looking at security tasks, it's important to remember that security is almost never a task in and of itself. So if we were looking at a secure system, like a banking website, a good task could be check your balance or make a transfer. A bad task will be, log in to your account. Now it's true that people will have to log in into their account to do something on the site. But, no one ever goes to their banking site where the goal is, I'm going to log in, and that's it. They may have to log in in order to be able to take some of the other functionality. But, logging in by itself is not a primary goal that someone has with the site. So that's a security feature that we may want to test the usability of. But you can't pretend that it's a task or a main goal that a user would have on the site. Instead, security should neatly run into the flow of normal tasks that people are trying to do. In this case, if someone wants to check their balance and they're not logged in, they'll have to go through the log in procedure and they'll be able to see how useable it is, but they shouldn't focus on logging in as their main goal. They should focus on the actual tasks that they want to perform on the site. Once you have your tasks developed, the two things that you have left to build and pre and post-test questionnaires. Pretest questionnaire will have any relevant background questions that you want to ask your subjects. That could include things like age, gender, education level, experience with the particular type of website. You may also want to ask more specific questions like whether people are color blind, if the user has children. It depends on the site that you are trying to test and the functionality that you're interested in your tasks. Don't ask anything that's irrelevant. But if something is really important for the kind of site you're testing, put it on that pretest questionnaire so you have it as background. Your post test questionnaire is going to ask questions about, what people's experience were doing the tasks. You may ask them to rate how difficult or frustrating or easy it was to complete the tasks. We often ask people to do this on what's called a Likert scale. Which is a five or seven point scale where they can just circle a number. The end points are labeled, like, easy to difficult, or frustrating to intuitive. That questionnaire goes at the end and we'll see some examples of that later in this lecture. Once you've got the tasks and the questionnaires, your evaluation is going to take place. The users are given the list of tasks after the completing the pretest questionnaire, and they sit down with a system and just go through and do each of the tasks. Their interaction is governed by some different ways of observing them, what we would call observation protocols. There's three major observation methods or protocols that we do when we're studying usability tests. One is called the silent observer. In this example, you, the experimenter basically stand behind the subject who's undertaking the task and watch them do it. They don't say anything, you don't ask them any questions, you just watch and take notes. So, we're going to pretend we're doing some usability studies. You get to be the experimenter and I get to be your subject. I'm going to sit here using the system, and you'll observe me. I'll let you see both from the experimenters perspective that you're seeing here, and you can also see some clips of what I'm doing on the screen. The task that we're going to have me complete here is to look at the dashboard of courses that I have set up in Coursera. So this is a system you should all be familiar with. We're going to go through each of the different observation methods, and we'll start with silent observer. With the silent observer method, I'm simply going to try and log in and see the list of courses. You're not going to hear me say anything. You're just going to watch. So we'll do this pretty quick. [SOUND]. And that's it. This is my list of courses. The second method, is the think aloud method. This can be a little uncomfortable for users. But it's really helpful and insightful for you. In the think aloud method, users actually sit down and talk though each of the things that they are doing. Every part of their thought process gets articulated, and so you get to see what they're thinking. A lot of times you'll find that when you are watching people they'll do stuff that you don't expect. And you have no idea why they did it. But in the think aloud method, they actually explain what they're doing. So you can get some more usability insights. So here we are, back at the home screen and now we're going to go through what it would be like to do, I think, a loud method. I'm going to do exactly the same task but you're going to see what it would be like if I'm using that protocol. Okay, so I need to see my list of courses and for that I probably need to log in. I see that there's a Sign In button at the top, so I'll click that. I think my email for this is my main university email that I use common password. All right so now I'm here on the main page. I think this is a list of my courses here. And, if I want to see if there's other ones, I clicked, put my mouse over my account, up here, that's a pretty normal place to find it. And I see there's this Course Dashboard, if I click on that, nothing happens. Maybe that's where I am already. So I'm going to say that's it. I'm done with that task. And that's what you would have subjects do for every task through to completion if they're doing the silent observer method. And then finally there is a constructive interaction method. In this way of observing people you can have two user working together or you can be asking questions and offering suggestions without telling the user how to do the task. It allows a little more interaction, and you can see how people work through problems that they may have using your application. Finally, there's constructive interaction. Now, this is a little harder for me to simulate, since there's just me here, and you, the experimenter, can't talk to me and I don't have a partner to work with. But I'll show you an example of some of the questions I might ask and you can think of some answers that you would give me. Either as the experimenter or if you were a partner sitting down and working with me. Okay, so I'm going to sign in using this button up here. We can use my account. Okay, so I get here and I can see that I have a section here called Your Courses. But I'm not sure if that's all of them. So do you think there's somewhere else that maybe I should go to look at this? We could try going here, and looking at the Course Dashboard, and that doesn't seem to go anywhere, so maybe this is it, maybe this is the main place that we want to look for that. And so that's it. Those are the kinds of questions that I would ask and if I had a partner or are participating experimenter working with me, we would answer those questions, maybe offer some feedback on the actions that I was taking. So there you go, that's constructive interaction. When you're done with the test, so everyone has gone through, done all the tasks, and you've taken your notes, you'll administer your post test questionnaire, and then a lot of times we try to do an interview. This can be really short. But, we often find when we try to run useability studies, that people aren't willing to write you long paragraphs describing their experiences even if they have thoughts. Because it's just, not fun and it takes them a lot of time. But if you ask them questions, if you just ask, what did you think of the site as a general question, or if you have some more specific ones, they are a lot more likely to tell you in an interview format. Take notes or, if you're making video, you can go ahead and record that, and that'll give you some extra things to think through when you're analyzing the usability of your site. Once you've done all that you want to report your results so you take your evaluation, everything that all of your users have done, report that out. You want to summarize the experience with your users. Now you can do a usability test with only one or two people. That's a little bit small because you may not catch all the usability errors but it turns out you only need a few more than that to actually cover a lot of the usability problems with an application. Five or six people often will give you all the insight that you need. You want to summarize their experiences, emphasize your insights with specific examples or quotes. So, if people had a lot of trouble with one particular task, take some quotes from your exit interview. Take some things that they maybe said while they were doing the task and report that. And then, offer some suggestions for improvements on the tasks that gave people real difficulty. So that may be it turned out logging in was quite hard. You want to make sure that you can offer some suggestions on how to make that easier. Finally here are some lessons that we can take from how you actually conduct a useability study and what it's good for. You want to look at what parts of a site or application are easy to use. That's ultimately your goal. As a designer you may not really understand that. But your users will be able to go through the tasks and tell you what's easy and what's hard. You can come away with that with an understanding of how usable the site or application is for each one of the tasks that you have people complete. You can also come up with suggestions about what improvements can be made to improve usability. And in the security context you can get additional insights into how you can make the security a more seamlessly integrated part of the site as a whole. So usability studies are a great tool for understanding these usability problems. They're very low cost. They're easy to run, and with a little investment in time, you can actually identify and correct a lot of usability problems with an application you're building, just by running this simple test.
