This particular lecture is going to be devoted to a very specific and pretty new topic in experimental design, which I consider to be a subset of the response surface modeling world. And this is experiments with computer models. Now we often think, in fact, we almost always think of applying designed experiments to a physical process such as chemical vapor deposition or wave soldering or machining or even to some e-commerce type experiment. But designed experiments can be successfully applied to computer simulation models as well. And in these kinds of applications, we use the data from the experimental design to build a model of the computer simulation model. In other words, what we might call a metamodel and then optimization is carried out on the metamodel. And the idea here is that if the computer simulation model is a really good faithful representation of the real system, then optimization of the metamodel will give you a very good determination of optimum conditions for the real system. When we dig into the science of simulation, we find that there are really two kinds of simulation models. Stochastic simulation models, in which the output responses are random variables and there are a lot of examples of these kinds of simulation models. Factory planning and scheduling models that are used in the industry. Traffic flow simulators used in traffic and civil engineering. Monte Carlo simulations from different probability distributions to study complex mathematical phenomena. Sometimes the output from a stochastic simulation model is in the form of a time series. So, there are many kinds of simulation models that are stochastic in output and for those kinds of experiments, very often standard designs often work fairly well. Although there have been some specialized techniques that are sometimes used to study these problems, standard designs are very often the basis of applying design experiments to stochastic simulators. But this other type of simulation model in that type of situation, the output is deterministic and usually these are models such as finite element models or computational fluid mechanics models or computational thermodynamic or heat transfer models. There's no stochastic or random element in the output and here's a situation where that kind of problem might arise. This is the compressor containment portion of a jet aircraft turbine engine and it's a metal part that is surrounds the combustion chamber and if say a blade fails on the engine, this system is supposed to contain it within the core of the engine. So it will not come flying out through the nacelle and and contact the aircraft and potentially damage the aircraft or go into the aircraft and damage or hurt people. This containment component has a series of parts and here are the parts. One, two, three, four, five, six, seven, eight distinct parts. And each of these parts can be described in terms of a set of specifications like their size, thickness, diameter, contour, the type of material they're made from. There are several design parameters associated with each of these components. And typically the way the designer would study these components is to build a computer model of this device, this system and then use a finite element analysis model to study how the stresses flow through this part when a failure occurs. And the computer models that do this are very, very widely used in engineering design, the automobile industry, the aircraft industry. Lots of people use these and they are very effective design tools because they're an efficient way to study how a system works without having to actually build expensive prototypes and test prototypes. These are very elaborate computer models and they are often highly detailed and they can take a very long time to execute, to run. So, testing or experimentation with these models is something that we need to do efficiently and it turns out that you could use optimal designs for this. Or you can even use standard designs if you think a polynomial model would be appropriate but in many cases polynomial models are not appropriate because the response surfaces generated from these experiments are pretty complicated. And modeling them with a low order polynomial might not be very appropriate and in fact, even in situations when a low order polynomial might be appropriate, standard designs really don't work very well in terms of the prediction quality that you get in your final model. So, many experimenters in this field use what we call space-filling designs and these are designs that are widely used. You could use them for polynomial models, but they're very widely used for non polynomial models and there are several classes of these designs that that are very popular and that are widely used. Here are four classes of designs that I'm going to illustrate. The Latin hypercube design. The sphere packing design. Uniform designs and a maximum entropy design. Upper left hand corner is a 10-run Latin hypercube design. And the reason we call this a Latin hypercube is if you projected this design down onto either the X1 Axis or the X2 axis, you would have an even distribution of points across that axis. It would have the Latin Square type property of equally spaced levels across this axis and across this axis. This is one of the oldest types of space filling designs and generally I find it to be a very good design. It may be one of my favorite choices. On the right is a 10-run sphere packing design. Now ,the way this is done is to imagine putting spheres in here. And you try to put these spheres, As far away from each other as possible. And these are the centroids of the spheres and I've really done a terrible job of sketching my spheres but the whole idea here is that you want to get these ten spheres in here. And you want the centroids of those spheres, the centers, to be as far away from each other as possible, that fills the region. Here's a uniform design. So the idea here is that these runs are uniformly distributed over the design space. This is maybe not the world's best example of a uniform design but it is typical of what you get when you use a uniform design generator because, Even though the design points are distributed randomly over the region and uniformly over the region, the distance between the points may not be uniform at all. It turns out that they actually have an exponential distribution and then here's a maximum entropy design. Remember, entropy is a measure of scatter or spread and what we're doing here is we're trying to spread these points out as far as possible. But to also ensure that there are no replicates upon projection because when you project points down like this, you don't want to have replication because the runs at those two points will be different not because of random error but because of something different that's going on in the computer model. There is no random error. What kind of model do we typically fit to this data? Well, you can fit polynomials although most people don't do that. Usually some type of spatial correlation model is used. Some people call these Kriging models because these spatial correlation models or Kriging models were first used in seismographic type studies. The Gaussian process model is probably one of the most widely used types of models and this is the general form of the Gaussian process model, constant plus this quantity z of x and the correlation matrix between these variables in x is of the form that you see here. It's sigma square times a correlation matrix R of theta where the form of the elements in R of theta are given by this equation. This is an exponential correlation function. There are other forms, but the beauty of this particular model is that there's only one parameter for each dimension. The thetas are the design parameters and there's only one parameter for each dimension. Here is the prediction equation for a Gaussian process model and it is a prediction in terms of this correlation Matrix R. This is often a very good choice for a deterministic computer model because these Gaussian process models give a perfect fit to the sample data. So your fitted model will go through your design points perfectly, here's an example. And in this example, we're studying the temperature in the exhaust from a jet turbine engine and we're looking at the temperature at different locations in the plume, that is that shoots out behind this engine. And there's a computational fluid dynamics model that is used to produce this result. There are two design parameters of interest. The location in the plume, the X and Y coordinates, the researchers here referred to the Y-axis as the R-axis or the radial axis. We're only really looking at two dimensions because the theory is that these plumes are symmetric. Both of these location parameters were coded to a minus one to plus one interval and the experimenters used a 10-run sphere packing design. This is the 10-run sphere packing design that they used. It's very similar to the one I showed you earlier. And here are the data from the experiment, here are the coordinates of the X and R-axis for each of these points and here are the observed temperatures. We use JMP to fit the Gaussian process model to the temperature data. And, Here's the actual by Predicted Plot for this experiment and this is a jackknife prediction. That means that the way these predicted values were obtained was by eliminating that design point, fitting the model to the remaining nine points and then using it to predict the value that you see here. And in a perfect world, these would lie exactly along a straight line. Well, they don't exactly, there's kind of an indication that at the upper end of the temperature range, maybe this model doesn't fit quite as well as it does at the lower end of the range. Here are the design parameters. These are the estimates of theta and the larger the value of theta, the greater the sensitivity, the greater the amount of curvature in the system. And you can see that by looking at the picture down at the bottom of the page. This is the Contour Profiler and you notice that there's much more dramatic curvature in the R-axis than there is in the X-axis and that's a reflection of the large difference in the parameter estimates. The Theta estimates along the X-axis and the R-axis. There's a couple of orders of magnitude of difference in those estimates. So there's much greater curvature along the R-axis that there is on the X-axis. And this also shows you why a low order polynomial would be a really tough way to try to model this because there are a number of bumps and turns and curves in this surface. And trying to model that with a low order polynomial would be very difficult. Here is the prediction equation for this model and this prediction equation is the fitted Gaussian process model and this equation would produce an exact fit. All you have to do is to plug in any value of the X-axis and the R-axis that corresponds to a design point and the predicted value from this equation will match the observed design point exactly. This kind of property is something that deterministic computer and modeling people really like because they don't feel that it's very logical for the model to not basically predict the same value that you observed at a particular design point because there is no error in the system. There is no random or experimental error in the system. These computer experiments are growing widely in use in many, many aspects of engineering design and product development. And this is an area that I think will continue to grow and to continue to be of considerable importance in experimental design for years to come.
