[MUSIC] For all practical purposes, any file that's not a text file can be considered a binary file. Binary files can contain all kinds of data, but most typically they store numbers. And there are many ways to represent these numbers in a computer. The lesson on data types gives you a hint. But MATLAB makes it easy to deal with these different types when we want to read or write a binary file. All we need to know is the type. But more on that later. Binary files need to be opened and closed with the same functions, fopen and fclose, that we used with text files. The permission's similar too. The only difference is that we omit the t, which you'll remember means text file. In other words, the default type for these two functions is the binary file. The w, the r, the a, and the plus characters have the same meanings as before. Here's an example that'll show you why we have to know the data type. The write_array_bin function takes one array of doubles and a file name as input arguments, and it writes the elements of the array into a binary file. We start by opening the file and checking for an error, as we did before with the text file. Now we're ready to write the numbers into the file. So let's see that code. That's it! The built-in fwrite function takes the file identifier, the array to be written, and a string that specifies the type of the elements in the array and does it's job. That's all there is to it. The only way to mess this up is to give the wrong data type. That's unlikely to happen if you created the data, but it can happen if someone else wrote the function that passes the data in. And it's very easy to get the wrong type later when you're reading the file, because, unless you included the type in the file name when you wrote it, there's just no way to determine the type and numbers inside the file. And, if you specify a type that's different from the type in the file when you read it, the result will be just nonsense. Once we've written the array to the file, the only thing left to do is to close the file with fclose. So putting data into a binary file is easy. And getting the data out of this file is just as easy. Consider the function read_bin_file, which takes the file name and the data type as input, and returns as output the array that was read from the file. We start the same way we started twice before. We open the file with fopen and we handle any errors. Reading the array is another one liner. The fread function takes the file identifier, the number of items to read, and the data type, and it returns the values it read. Here, instead of specifying the exact number of items, we specify inf meaning, infinity. That causes it to read everything in the file. That's the easy way to get all of them, and it's what you do when you don't know or don't care how many numbers are there. Finally, as always, we close the file. By the way, it's important that our program close the file, whether it be a text file or a binary file, because if it doesn't, then the file can't be reopened, and other programs will be blocked from accessing it. Okay, so far so good. We'll soon see that things can get a little more complicated, but first let's just try these functions out in my lab. Let's create a data array first. As usual, I initialized the pseudorandom number stream so you can duplicate what I do. Now let's look at the function write_array_bin that I showed you in PowerPoint. It's over here in the current-folder window right there. And let's see. It looks just as simple as it did on the slide. It opens a file, who's name is given as an input argument, checks for an error; everything's okay. Then it simply writes into that file using the file ID and writes the first argument in as type double, and it closes the file. So let's run it and use Data as the input array. And I'm going to store it in a file called datafile.dat. Now that's not over here but watch right here where this arrow is, in this vicinity as I hit return ...[CLICK]...and the file appears. Now let's try to look at the contents of this file. We'll use the function that I showed you in PowerPoint called view_text_file, which I happen to have right here. And here it is in the editor window. There. Wait, let me show you something really nice about the editor. If we click View here, and come over here to the left, we see this little divided-screen icon here, named Left/Right. I'm going to click that and watch what happens over here on the screen; view_text_file moved over here to the right. Now these two files are lined up for easy side-by-side comparison. With this arrangement you can see that they start out almost exactly the same. Both open the file the same way. This one uses w+, this one uses rt. Then they check for an error. It checks for an error. write_array_bin, does its work in one line. But working with text files is more complicated than working with binary files. So, view_text_file's work is complicated enough to warrant a couple of explanatory comments here. Then they both call fclose and close with the fid, the file ID. So let's use view_text_file to look inside data.dat. Woah! Okay. I'd call that garbage. And that's what we should expect. Any time we try to read a binary file as a text file, we're going to get garbage because the binary file is not encoded with the text-encoding scheme. In this particular case it's encoded as a string of doubles. What we should have done is use the function I showed you in PowerPoint called read_bin_file, which is designed to read doubles stored in a binary file, reads it into an array. Here it is, right over here. I double-clicked it, and it popped up on the right on top of view_text_file. Well these two functions are almost identical. They both open the file. This one uses a w+, this one uses an r, they check for the errors, and each uses one line: fwrite to write the file or fread to read the file. Then they close the file. That's how simple it can be to write and read binary files. Okay, let's use read_bin_file to read datafile.dat the right way. Remember that the first argument is the file name; the second argument is the type of the numbers that are written into the file. That type is given to fread, which you'll remember requires the data type as its third argument and requires the number of values to be read as the second argument. We've specified infinity here, which means that it reads all of them in the file. So, there. We should now have two variables in our workspace with the same values in them named Data and X. Let's check that with whos. Well, they're both there, Data and X, and they're both doubles. But you may be surprised to find that X is a vector, a 120-by-1 column vector, and not a 10-by-12 array like the original. So what's going on? Well, inside write_array_bin here, when fwrite copied the input array into the file, it wrote each element, one after the other, in one long stream of numbers with nothing written about the dimensions of the array that it came from. So the data file contains 120 doubles arranged like one long vector. Then inside read_bin_file over here, when fread retrieved those numbers, it's simply read them in the way they were arranged in the file as a vector. So A ended up as a vector. If we want to preserve the dimensions of the variables when using binary files, we have two options. We either try to remember that a file with a certain name contains an array with such- and-such dimensions, or we write the dimension information into the file as well. Writing the dimensions into the file is a much better option, but it still requires us to remember the format of the file. Let me explain. Suppose we decide to write out the dimension information as the first few values that we write in the file and follow those values by the contents of the array. We still have to remember that we've done that. There's no way to figure that out by examining the file itself. I'm sure you see the problem. Yeah, maybe not. Okay, let's do an example. Let's look at a modified version of write_array_bin over here that stores the dimensions in the file along with the data. It's called write_dims_array_bin. I'm going to load it in the left side of the editor this time. And I do that by clicking the function over here, you know, to make it the active one, and then double-clicking the file over here. There, it shows up here. It's all fancy with a little header I put into it. And, as I said, it's a modification of write_array_bin over here. Let's move that over to the right so we can compare them. It's pretty simple, the original version. This is a little more complicated. The first modification is that we now use the size function here to get the dimensions of A, the input argument. Here, we didn't do that. We put them into the vector dims. So for example if A is a nine-by-five-by-six array, dims would contain the numbers nine, five, and six. For a scalar, dims would contain a one and another one, and so forth. The second modification is that we call fwrite, instead of one time, three times. One call is to write the number of dimensions of A. The number of dimensions is equal to the number of elements in dims. So it's equal to the length of dims. And we write it as a double. The second call puts the list of the dimensions into the file after the number of dimensions. An important feature of fwrite is that every time you call it it starts writing at the point in the file where its previous call left off. So a given call doesn't write over what was written by the previous calls. And there is a third call, and that's when we write the elements in after the list of dimensions. And really that's all there is to it. Then we close the file. I'm going to close this one. We're done with it. And I'm going to close view_text. We're done with that. And now I'm going to look at a companion function that reads files written by write_dims_array_bin. I'm going to put it over here on the right so we can see the similarities between the new function I'm going to show you and write_dims_array_bin. It's called read_dims_array_bin. So, let's put that in, and here it is, again with the fancy header. And let's compare these. We open the file using an r instead of w+, we check for an open error, and then we begin doing things in earnest. First, we read one element of the file. That what this second argument--one--says to fread, and we store it in n. You'll remember that the first number we wrote was the number of dimensions. So, this n now will hold the number of dimensions. The next thing we do is tell fread, we want to read n elements. So if it was a three-dimensional array, we'd be reading three dimensions, and we'd read them into dims. And then we call a third time. So, just as we called three times the fwrite function, here we call three times the fread function, and the third time we read the elements of A. Note that this last call doesn't tell fread how many items to read. In that case, it just reads in everything to the end of the file. After we've read these elements into A, there is one problem. We did this before, and remember A turns out to be a vector. Well, we want A to have a dimension specified in dims. So, what do we do about that? Well, there happens to be a function, a built-in function, called reshape that will take a vector like this, and if you give it the dimensions, it will produce as an output, an array with the same elements but with the specified dimensions. If you look really closely here, you'll see the transpose operator. That's because this vector in reshape has to be a row vector, but fread returns a column vector. Okay, that's it for the read_dims_array_bin function except for closing the file, which it does just the same way that write_dims_array_bin did. And we're done. All right. Let's try this new pair of functions. Let's write Data here into a file again, but this time we'll name it dims_array.dat to remind us that the file contains both the dimensions and the elements of an array, in that order. And we'll read it back in the variable Y. Well, here goes the write. All right, if things worked properly, over here in this file that we just wrote, the number of dimensions, the list of dimensions, and the elements of data should've been written into the file. Let's try reading the array. Well, if everything went according to plan, Y should contain an array that's identical to the array in Data. Let's cross our fingers and look at our variables using whos. Well, as you can see, data is 10-by-12, and Y is 10-by-12--a little better than X was. And they're doubles, of course. What about their elements? I wonder if they're the same values. There's a way to check that. There's a function, a built-in function, called isequal, which will return true if and only if its two arguments have the same size and contain the same values. So let's try that. Is Y equal to Data? One is true. Excellent! Let me emphasize a key point again here. This particular pair of functions, write_dims_array_bin and read_dims_array_bin work properly only if we remember the following things about the format of any file that they work with. First, all the data in the file must be doubles. And second, the file must contain the number of dimensions and a list of those dimensions, and then the elements. That's what write does; that's what read expects. We shouldn't rely on remembering these rules. We should write them into comments in a header at the beginning of each function, just as we've done here, so that we can jog our memory with the help function, like this. And here we see, write_dims_array_bin. It writes a dimensioned array in binary. And then we give an example. It writes the number of dimensions of A, then a list of the dimensions, and then the elements of A into a file named fname, encoded as doubles. And there's a similar explanation down here for read_dims_array_bin. And we should do one more thing. In the folder in which we store the files that are written in this format, we should include a text file that describes the format. You may not believe me yet, but if you write formatted files and let them sit there a while, and then go back a few weeks or even just a few days, and try to use them, you'll become a believer. You'll see that you actually do need to go to such lengths to make sure that you'll remember all this when you need to use these functions and use the files written by write_dims_array_bin later, and so that you can help other people working on the same problem to understand these rules, as well. We may even want to make up a new file extension for these files to differentiate them from other binary files, say .dar for dimensioned array or something like that. In fact, let's do one more example with a three-dimensional array instead of a two- dimensional one. Great! B and Beta are both six by-two-by-five arrays. And are their values equal? Let's see that. Success again. So by writing a binary file in this special format, we can read it later and retrieve an array of any dimensions without having to remember its dimensions. Sure, our binary file format maybe a little more complicated than a simple stream of elements, but I think you'll agree that it's worth the complication. And formatted binary files can get even a little more complicated. The binary files we've been writing so far have used just one type of encoding, double. Sometimes a binary file needs to contain more than one type. It might have doubles and ints, or chars and singles, for example. To read or write such files, it's necessary to know how many of each type there are and what order they're in. So once again, we need to keep up with the format of the file, this time to keep track of the types. Let's de-clutter things and look at an example. We can close every file in the editor with one click over here on this little x. Gone. And we'll clear the workspace and clear the command window. There we go. Okay, I've got a couple of functions to show you, custom_write_bin, it's right here, and custom_read_bin. And again, I'm going to put the writing function on the left and the reading function on the right over here. Let's look at the one on the left first, custom_write_bin. This function calls fopen. It opens a file as a binary file for writing. And it writes to that file by calling fwrite five times. And in those five calls, it writes the following types: int16, three of them. some chars, some singles, some int 32s, and some singles, in that order. The first three values, this n1, n2, and n3, tell how many chars there are, how many singles there are, and how many int32s there are. In a minute or two, when we look at how this file must be read, we'll see that we need to store these three numbers in the file, but we don't need to store the number of singles written at the end here. So we leave that out. Now let's look at the arguments up here. The last argument provide the filename. The first four arguments are vectors containing the data to be written. Notice that the types of these data are not specified anywhere. These arguments can be of any type. But when they're written into the file, they'll be converted respectively into chars, singles, int32s and singles again. The conversion process may cause some of the values to be changed if those values are not in the range of the new type. You may remember us seeing this happen back in lesson seven when the ranges were clipped. For example, d1 is going to be written as a char. And if it contains a negative number, it will be converted to zero, because negative numbers are outside the range of a char. The smallest number a char can hold is zero. These lines right here, n1 equals length d1, n2 equals length d2, and n3 equals length d3, determine the number of chars, singles, and int32s that need to be written in the file. And they're written in this command right here, in the format int16. So they become the first three numbers in the binary file. Then each of the remaining four fwrite calls writes one of the four vectors, one by one, using the required types, into the file. Now let's look at the function for reading the data from the file that was written by custom_write_bin. It's over here, custom_read_bin. It opens the file for reading as a binary file. And it reads from that file by calling fread, one, two, three, four, five times. One for each of the five groups of values that custom_write_bin wrote with its five calls to fwrite over here. First, we read 3 int16s into nums with this command here. The second argument tells fread to read three values, and the third argument tells it that these values are encoded in the file using the type int16. All together, six bytes will be read in three groups of two bytes, each of which is decoded as one int16. In the next command here, we use the first element of nums to tell fread how many chars we want to read into o1. It may be surprising to see the conversion function char here used in this command to convert what's returned by fread. You might have expected, since fread is reading something of type char, it would return the value in type char, but that's not how it works. By default, the values that fread returns are always of type double. We want to return the chars that were read from the file when this custom_read_bin is done, in o1 as a string, so we converted the doubles here into chars. You might notice also this transpose here. That's because fread returns a column vector, and a string has to be a row vector. So we use the transpose to convert that column vector into a row vector. So fread, by default, returns doubles, and we converted them to a char. However, it is possible to tell fread directly to return values of any type you want by altering this third argument here, char, to include both the input type, char in this case, and the output type, which we want to be char, by putting those two types in this string, separated by an equals, greater-than. Let me show you what I mean. This means input is char and it's converted to a char. Well, it's not necessary if the input and output values specified in this way be the same. For example, if we wanted to return, I don't know, this last value here as a int64. We could do this. It's a very nice, flexible way of handling type conversion in file input/output. But for now, we're going to go back the way we had it. I'm going to hit this undo a time or two. And now it's back to the way it was. Now after the chars have been read on this line, we read from the file a third time. This time we use the second element of nums to tell us how many values to read, and we tell it that they're encoded as singles. A fourth call uses the third element of nums and says that we want to read that many int32s. Then we come to the last call. As we mentioned while we were talking about custom_write_bin over here, we don't need to know how many singles there are at the end because we can just simply read all the way to the end of the file. So we need to tell fread to do that. What we do is just omit the number here. When fread discovers that the second argument is not a number but is a string, it knows that the string gives the type, and that we want to read to the end of the file. So it gets all the singles at the end, and puts them in o4. These four variables are return by custom_read_bin. Well it's time to try our pair of function, so let's go down here to the command window and do it. [SOUND] [CLICK_SOUNDS] Okay, I'm going to hit this and if you watch over here, you'll see the file appear. There. Right there. Now let's try reading it with custom_read_bin. And there's our data. Well, but does our data have the right values, the right types, the right shape, and so on? Well, first off, it's apparent that o1 has the right type. It's a string. You can tell that because it's printed as a string. The other three variables, o2, o3, and o4, have the right values if you check them here. But it is a little disappointing to see that they're column vectors when the inputs were all row vectors. This happened because fread, as we mentioned, returns only column vectors. We could change this pair of functions so that the orientations of the input that came over here to custom_write_bin, and the output that was produced by custom_read_bin, agree with each other. We'd have to store information about the orientation of these vectors, you know, somehow in the file. As we did before with write_dims_array_bin And of course we'd need to change custom_read_bin over here to interpret that information and orient its output accordingly. But we're not going to do that, not with this exercise. All we are going to do now is to check to see if the types of o2, o3, and o4 are correct. We'll do that with the class function: double, double, double. Well, are they correct or not? Weren't we going to get a single, an int32, and a single? Well no. We stored them as singles, int32s, and singles, but when we read them back, they come back as doubles unless we do the conversion that we were talking about before, and we didn't do that. So we started out with one string and three doubles, we stored them as char single int 32 single, and then we ended up with doubles again. You may be wondering about the point of saving doubles into a file using these various other data types, and then reading them back into doubles again. Well, we certainly could have had all the last calls here return the same types that they read. But we chose not to do that in order to illustrate a point. We're letting them come back as doubles, because the default data type in MATLAB is double. And it's usually easier to use doubles in calculations because a double holds such a wide range of numbers. On the other hand, other types, while they're more difficult to use in calculations, take up less space in a file. The int32 for example, takes up only 32 bits. Where a double takes up 64. If we know that the data can fit in the less space, then we can use less space by choosing just the right type. If we know that an array contains only non-negative integers ranging up to, say, 50,000, for example, it makes sense to store the data on disk using the uint16 type because its range is 0 to 65,535. An int16 won't work because it goes up to only 32,767 to leave room for negative numbers. A double would handle it, but it takes eight bytes, while a uint16 takes only two, thereby cutting down the file size by a factor of four. And keeping files small can be important when you're dealing with big data, not just to save memory on your machine, but to save time when you upload that data to the web for someone else to access. So now you know how to read and write information to files, and you can do it in a way that suits the problem at hand, whether you're using MATLAB's, mat-files to save your work from one session to another, Excel files to interface with spreadsheet applications, text files to allow humans to read them, or binary files to get the highest efficiency. File IO is the last piece of the puzzle that gives your programs access to the world's vast data resources, and allows the world to benefit from the results of your programming. And that brings us to the end of lesson eight, and with the end of lesson eight, we bring our course to a close. On behalf of Akos Ledeczi, Robert Tairas, the people of Vanderbilt's Institute for Digital Learning, and myself, I'd like to thank you for taking part in Computer Programming with MATLAB. If you've followed along with us from the beginning, taken the quizzes, and worked the exercises, then you're a MATLAB programmer. Congratulations! And welcome to the club! Whether you took this course for a programming class, for your job, or to get a job, or if you just talk at simply out of intellectual curiosity, you now know how to program a computer. And you're part of the worldwide community that program's in MATLAB. But wait! Turn down the music. This is not the end. It's really just the beginning. Programming is a lifetime thing. Those of us who have been programming for 10 or 20 or 40 years will all tell you the same thing. We're still learning today. I learned something new every one of the ten wonderful years that I taught MATLAB to those incredibly bright and inquisitive first-year students at Vanderbilt. And I'd been programming for 30 years before that. You learn by doing, and the more you learn, the more satisfying it becomes, both because you write better programs and because you're more valued by the world when you do. If I can offer some advice for your life long learning, it would be to do two things. First, find real problems that need real solutions and solve them. Textbook problems are good of course, and they're focused, but the thrill that will keep you hanging in there, trying to make a program work while you look for those frustrating bugs, is the knowledge that you are the first person to solve the problem. Second, read code written by more experienced programmers. You can find lots of that on the web in many programming languages. But a very convenient place to find really good MATLAB code is in MATLAB itself. Many of the built-in functions are actually open source, meaning that anyone can look at the code anytime they want to. The command, "edit", will let you look at them. Type "edit num2str", for example, n-u-m-2-s-t-r. An m-file containing num2str will pop up immediately in the editor, and you can see how the math works programmers solve the problem of converting a number to a string. On the web, a great place to go to see the handiwork of other MATLAB programmers is MATLAB Central. There you'll find code contributed by people from all over the world, and with a bonus: There are often commentaries included that are written by other programmers. Okay, you can bring up the theme music now. [MUSIC] That's the end of my advice and the end of the course. It's time for you to begin the next phase of your programming career. If this course has helped you get to that next phase, then we've achieved the goal that we set ourselves. That goal was simply to use MATLAB to make computer programming more accessible to you. We've done our best. The rest is up to you. We wish you well. [MUSIC]
