At this point, we want to look first at the pre-post test process of evaluating training outcomes. A pre-test can be planned as part of the activities during the very first session of the training program. Maybe an hour or could be set aside the first morning. Or if you have a registration going on the day before, maybe there could be a quiet room, where people can take the pre-test as part of the registration activities. we want to make sure that this provides, you know, that we have an opportunity, a timing that is not going to eat into the main schedule. But we want to make sure that we have an opportunity to get this information. Now, a pre-test would be slightly different than a baseline survey. A baseline survey of potential trainees may include many people who are not actually attending the current training session. So, what the value of a pre-test can do is that you can have information on specific individuals, both before and after. And measure, in fact, does that individual's score on the test change, does it improve? There are various ways of doing this. Again, the question should be, based on our diagnostic findings, on our baseline findings. And on the specific content that's going to be presented as evidence in our training objectives. These tests generally focus more on knowledge. we can certainly link in skills if we're talking about the skill of preparing oral rehydration [INAUDIBLE] solution, we can certainly ask people to list the steps involved in correct order and the amounts involved. But it's usually these tests do not show knowledge. Now we have an example where there are possibilities of doing skill tests one of our MPH students in Ibadan wanted to train teachers on giving visual acuity test to their pupils. And he actually provided the materials and asked the teachers to demonstrate it or as you can imagine, prior to the training, most of them just looked at the materials on the table and didn't know what to do with them. But at least there was a skill baseline pre-test that could be used to compare with post-test. And again, the post-test to be useful has to be the exact same questions, exact same procedures that occurred at baseline and pre-test that would be repeated again at the last session. So just as you've scheduled time during the first day or the registration for people to take the pre-test, you also have to schedule time at the end of the session for them to take the post-test. One thing just to note for confidentiality, you may want to assign participants a secret number that only they know. And they put that secret number on their pre-test and post-test so that you can compare them but not necessarily divulge whose results. On the other hand, if, if the participants want their results back, they want to compare them, then clearly you would want to put their names on it, so that you could return the the tests to them. So this has to be decided as part of the training committee plans. The tests have various types of questions on them. These can include open-ended questions, multiple choice questions, and attitude, opinion statements, or items. I need to balance the types of questions that we use in our tests. We want to cover the scope of the knowledge, but we want to make sure these tests are simple enough that the participants can complete them in 30 minutes or less. As we mentioned before, you need to schedule time to do these tests, and you don't want to eat into the process. You don't want to make the trainees feel too nervous or worried about a testing procedure, so you want to have it relatively simple, straightforward, and consuming little time. Open-ended questions are useful in terms of people describing processes or giving their opinions. it takes a little more effort to score these kinds of questions, you have to develop a coding guide. But it lets the trainees put down information in their own words. Examples of open-ended questions could be, list all the ingredients that can be mixed together for a homemade solution for oral rehydration or describe the signs of, danger signs of dehydration. Or mention the steps in order, for making homemade ORS. Open-ended interviews have been very useful when you have trainees who have a low literacy skills. Where they can't, you can't give them a four or five-page test and expect them to sit down and read it. And fill it out, even if it's in a local language. So, in situations like that, what we've done is arrange a group of trainers to serve as, just like interviewers, and provide rooms or space where the trainees can one by one by one go and be interviewed with the open-ended questions, and then their results recorded. Multiple choice questions. You all have experienced those before. They give you options. And you obviously mark on the option which is correct. Now, there are different ways of doing multiple choice. You can have a situation where only one option is correct. Or you could have a situation where many options are correct. And, again, the scoring depends on how you do this. If you agree in advance that only one option will be correct and you have a ten-item multiple choice questionnaire, then what we would have is a situation where the maximum score would be 10. There would only be one right answer for each question. But if you have ten items on ten questions, and each question has a choice of five items, as we see here. what are all the, what are the ingredients that one can mix together. We have five different ingredients. Sugar, soft drink, salt, vegetables, water. So they have to decide if each of these is right or wrong. Almost like a true-false. And if you had ten questions like this, with five options, you would really have a score of 50 possible correct answers. So clearly, you think about it, though look at this yourself and think what would you score? What's your right? What's your wrong? What's your wrong? Okay, the homemade solution that we've been teaching people includes sugar, but not soft drinks. Soft drinks would be bad for a child with diarrhea. Salt is another ingredient. Vegetables per se, solid food. Continued feeding might be possible for oral rehydration, but it wouldn't be part of a solution. You can't put a zucchini or, or a potato in, in a solution easily. And then the ingredient water, of course, that makes a solution. So, sugar, salt, and water should be ticked. And soft drink and vegetables would be left blank. And that's how you would score it. Another type of question that you can have on your pre- and post-tests are attitude items. If you're trying to encourage people to value or have a positive attitude about oral rehydration, then you could use some of these questions. We've found, for example, that health workers were uncomfortable at first when oral rehydration was instituted. They wanted to prescribe drugs. They didn't feel that were doing their job properly, unless they were prescribing drugs for children with diarrhea. And it took a lot of effort not just to teach them the reasons for this, the importance of replacing electrolytes, the fact that many diarrheals are caused by viruses that don't respond to antibiotics. the issue of antidiarrheal drugs clogging up the intestines and holding bacteria in or viruses in. So, it was more than just teaching them those facts. It was also, you know, letting them think about the fact that this was a good, a valuable, a useful treatment and that they would be doing the best for mothers. So here's a simple attitude test with, with four items mentioned. And you can obviously check or tick which one you agree with or disagree with or are uncertain. You can frame some of these in positive ways. mothers can be trusted to mix ORS at home. And then a few items later you can reverse it just to see if the people are being consistent and say we can't trust mothers to mix ORS or take care of their children with diarrhea at home. So you can, can have that. And then, of course, in your scoring process, if a question is phrased in a positive way, agree might be three points, uncertain, two, and disagree, one. But if it's framed in a negative way, agree may be one point, uncertain two points, and disagree with B3. So, you in terms of getting a final score, you have to be aware. With any kinds of testing procedures, just as we talked about. education materials, tests also need to be pre-tested to see if people understand the questions if the questions are easy to answer if people are responding in ways that make sense. If [INAUDIBLE] you have a question where 99% of the people agree with it or disagree with it, it's not a very good test question or it also shows that people you know, are already having the attitudes that you're going to train them on. So you want to make sure that you get a sample of people to try out the test first, to see if it works. Some examples of how we've used tests, include the patent medicine vendor training in a rural community where I worked in Nigeria. what we did, of course, was a we did a baseline survey to find out what they were thinking and doing in their shops. but we also had a pre-test, the, as we mentioned before, a training committee was formed among the patent medicine vendors, so that they could plan activities. They, in addition to the information that we gathered during our baseline survey, that committee members requested certain lessons on certain illness and skills. Like reading a prescription. We had a situation where either the clerk in the shop or the shop owner attended, usually it was not both because they didn't want to leave the shop vacant. And then we did a pre/post test. We had a, in addition, we had a control community where it was about 25 miles away in another local government. And so, we interviewed patent medicine vendors there at about the same time we did the pre-test for the ones in our intervention area. And then, did a post-test after the training in both places. In terms of the pre-test, you can see from graph, that the trainees and the control patent medicine vendors had very similar scores. And then, afterwards, the post-test shows a significant increase in the knowledge of the trainees, the people who went through the program. And the value of this, as you've probably learned from your various epidemiological courses is that if we just have the trainees alone, a pre-test post test, we couldn't be sure if it was really our training or maybe they enjoyed the experience. Maybe they were upset by the initial questionnaire and were inhibited. Maybe during the course of the training they went out and were exposed to other activities that increased their knowledge. So by having a control group, we can say that people under the similar type of people, you know, whether they experienced something in the environment that increase their knowledge or not, or whether the changes that we're observing at post-test can be really attributed to the training program. One thing, another thing that was important, as they said before, if you can identify the people who are in the training program, and either give them some sort of identi, ID number, so you can compare pre and post. one thing that we found is that the people who took the pre-test and post-test were not exact, were not identical. And this happens. People don't show up for all the sessions. Some people join late. So we found that 33 took the pre-test, 37 took the post-test. but 28 of them took both tests. So, in addition to looking at the gross results of the 33 pre-tests versus the 37 post-tests as you saw in the graph in the previous slide. We could also do what we call a paired t-test on the 28 simply subtracting the [INAUDIBLE] pre-test from the post-test and getting a mean for that and seeing if this difference between the two scores was different than zero. And we found that the average pre-test for this group was 46. The post-test was 71 and by subtracting each individual's pre-test from post-test, found that these [INAUDIBLE] these differences were significantly different than zero or zero meaning no change, as we saw in the control group. So, ideally if you, for yourselves, as, as trainers and evaluators, you would want to see, did the training actually make a difference? To the individuals, not just the whole group. Because the whole group, you don't know some may have lost knowledge for some reason. Some may have gained more than others what was going on. So, this being able to pair up the results and look at it gives you more confidence that the improvement did occur with each, individual trainee. Another patent medicine vendor training program was conducted in Kenya by a USAID project called the Quality Assurance Project. Their goal was to equip the patent medicine vendors with customized job aids to communicate new malaria guidelines to put drug outlets. So there are a number of private drug shops around in the community. the country had changed from chloroquine to sulfadoxine-pyrimethamine because of drug resistance. And they wanted to make sure that patent medicine vendors who dispense quite a lot of these countries' drugs understood this change. So again, the idea was that if you train the patent medicine vendors, they would be in a good position to educate their customers on the new drugs, malaria recognition, the importance of prompt treatment. But also the trainees who would, in turn, be able to train other patent medicine vendors. So this was sort of an outreach type of program. They were tested on specific areas of knowledge about malaria. And as we can see in the chart on this table there was an intervention group, there was a control group. And interestingly enough can see that there are some elements of knowledge that are, must be common knowledge in the community. The very first thing, that fever is a symptom of malaria and see that both groups knew that. Majority, we ask majority knew that. the issue of the new drug SP is also known commercially as Fansidar. You can see the items about Fansidar, which was the focus of training and how to use this new drug, were all higher for the intervention group compared to the control group that did not have this training. So, again, not only do we have a gross score that we can compare, but we can also look at specific items. And we can look at that and say, okay, well, overall at the end of training a number of these items, 95% for you shouldn't sell drugs below the exist, the correct dose. Again, recognizing the, the symptoms noting that the Fansidar is a very effective drugs. At the same time you can see that still, about a quarter of them didn't really believe or understand that Fansidar could be sold in their shops, that you didn't have to get it at a pharmacy. And we can see that 17% still didn't know that Fansidar, a single dose, was enough for treatment. That it's packaged that way so you just take it once. So there were still some gaps but again, you know, the, this overall improvement compared to the control group, but it does tell you that you may need to follow up with some of the trainees. So by having knowledge specific results, you know item specific results, you can see where the training was successful and where there are still some knowledge gaps. Another thing that this Kenyan program showed was that there are different categories of trainees. They all may have been in one room together, but it was found that the response was different by educational level. And even though in all cases, as you can see the darker green bar shows that the control of the control group, I mean the intervention group, did better than the control group. You can also see differential among the intervention and the control respondents in terms of their level of knowledge by their level of education. Std 1-8 is like primary school. Form 4 is like, sort of like junior high. people that have let's see, then you can see that then senior high. Also whats interesting is that the people with high school education were more knowledgeable to begin with. And so the intervention group, although they gained over the control group, it wasn't that great a gain. Whereas, those who had less education, only primary education, they really gained a lot more by this program. So we have to recognize that people of, of different levels of education perform differently and we have to make sure that our training materials and exercises are geared toward all these differences. That with the program is comprehensible by people who have only finished primary school as easy as it is for people who have finished high school. As I mentioned earlier, one of our MPH students for his dissertation did a program on training primary school teachers to test for visual acuity. He was an optometrist himself when he came to the program. And he was interested in preventative work community-based work. He developed both a knowledge test as well as performance test. The 17-point knowledge test that has things like what structure of the eye is responsible for image formation? What are three common causes of blindness? Name two ways of detecting visual problems in student. And just as we have done before, we did a pre-test, post-test with both intervention schools teachers and intervention schools and teachers in control schools. And as you can see here at baseline, the scores were on average four out of 17. At post test the the trainees achieved almost over 13 points on average out of 17. But you can see also that the the control group did increase their knowledge by, on average, about two points. And this is one of the things that happens in, in the testing process by exposing the intervention teachers to the pre-test itself. They thought about these questions. They became curious. They may have asked people around them, well, I had this test today, what is this about? And so, the testing process itself stimulated some knowledge acquisition. But clearly it was not of the level that the people who experienced a real training program. In terms of training affecting attitudes, we have an example from the CDT community-directed treatment in onchocerciasis control. What was involved was training local government, district health workers, to carry out the program. We've talked about this before. But it involves organizing community meetings explaining the program to the community encouraging the community to take responsibility for distributing it's own ivermectin. Which involves volunteers who would be trained that involves community collecting as drugs, community collecting the records, statistics, reporting the community managing side effects. These types of things. And so, the program hands over much of the responsibility for the day-to-day delivery of the program to the community. Which is why it's called community directive. This idea of handing over some control to the community was seen as somewhat threatening to some of the healtworkers. They were concerned that the community may not do it correctly, that they may not, if anything went wrong, they would be blamed. And so during prior to the intervention of this program we had a series of attitude statements with the health workers. The health workers were trained to organize the program in their local government. And then [INAUDIBLE] we followed up. Some of the statements with they could agree-disagree included communities are quite capable of managing the distribution of ivermectin. And in contrast onchocerciasis control should best be run by district. So these are some things that we asked them in addition. Community involvement in ivermectin distribution saves time for the health workers who could be doing other things. So, again, seeing a different value, seeing it in a positive light. Health workers in this community cannot handle ivermectin because they're distribution because they're too busy. They'r overworked. And then, again, health workers do not believe that community directed distribution of ivermectin is the best way to make it available to people. In contrast to programs that have had it based in health centers per se, or the health workers going out and handing it out themselves. So their response to these statements showed whether they favored the program or not. And as we can see in the chart, we have a pre-intervention survey, where we have all of the health workers in the study districts combined. And this was the distribution of their attitude scores. it was reconfigured to be on a scale of minus 25 to 25 whereas the positive side indicated a positive attitude. So, in other words, the mid, midline was set at zero. We see that, after the intervention, we have two types of intervention. One, where we train the health workers simply to go out into the villages and do their normal mobilization and helping the village setup it's own program and decisions. And then, we had an enhanced version where we, in addition to the village meetings, we brought villagers together at central places to have stakeholder meetings where they could interact more freely with the health workers. In both of these situations, the health workers improve their attitudes toward this community-directed distribution. But those who had the additional intervention of a stakeholder's meeting where there was more interaction with the community developed a more positive attitude. But, just going through the process, experiencing the, the training, then going out in the field and applying it. Interacting with the community. They recognize that the community didn't have the resources and, and the willingness to help carry out the program, and their attitudes improved. So, here is an example of measuring that. Another variable that we often look at, in terms of training programs, is not just enhancing people's skills but people's self-confidence or self-efficacy that they can carry out a skill or a program. And this is something we can clearly measure before people leave the training site. The actual application of the skill we may not know until they get back to the classroom, to the community, wherever they're working. The visual acuity testing looked at this question of self-efficacy. And as you can see, we asked the teachers did they, how confident did they feel in four things. Recognizing a child with visual problems, using the visual acuity chart, interpreting the chart results, and knowing when to make a referral. And so, of course the options are they're very capable, capable, unsure, not capable. And they could check what they, what they felt. And we did this again before and after. One thing that was very interesting is that prior to the intervention, the groups were very similar. The intervention group, basically, doubled their perception of self-confidence. Whereas, interesting enough, the control group decreased. So the, confrontation of the test, thinking about these things, thinking about what's involved, actually made those who did not receive the training feel less secure in their ability to, to identify visual problems. So, they recognize their limitations more after taking the test. So we did see changes in, because of exposure, we had changes in people's confidence. And, again, exposure to the test itself, especially the skill test made the teachers uncomfortable and, and feel less about their ability to, to do this without training. In summary, we can see that pre-/post-test document are knowledge gains. If we match the results, we have an identification number of the trainees, we can match pre- and post-test results and be really sure that individuals did gain knowledge. And then using a control group definitely helps us know whether the changes we've seen were due to the effect of the training program and not some other intervening factor. For example, the training that was done for the teachers was not done all in one session. It was done over a couple weeks, so could have been something going on in the environment where teachers could have learned more about visual acuity. having a control group makes sure that, if we see changes, it's likely to have occurred because of the, the training program. The issue of control groups especially if your implementing a program that is ultimately designed for all of the health workers If you do it in phases, your control group could be people coming for the next round of training, so that your not excluding people from the program, ultimately. Also, in terms of testing, we're looking at not just the overall summary scores to see if there was improvement, but looking at individual test questions to identify areas of strength or weakness in the training program itself. And finally, we need to consider the results in terms of the trainee's background. Does their educational level, for example, affect their scoring, affect their ability to absorb the knowledge that's being provided in the training, and how we can accomplish that? We've talked about tests. In our next section, we will be looking at other methods of gaining information.
