This week you will be writing the first portion of the program evaluation report. It will be helpful to skip ahead to chapter 9 in the textbook, starting w

Welcome to the second week of Program Evaluation. Just to reintroduce myself I’m doctor Wade. We’re going to get started even though no one’s here. I’ve been told to stay calm about that. It’s the small class, and if you can’t make it, you’ll watch the recording. So I’m going to jump right in as someone were here. Oh, I actually see a student. Go to see you this week. I appreciate that. It’s good to see you too. It’s hard to find the link to get in. Oh, really? Yeah, I wasn’t up for a while, and so I had to keep logging in and coming back, and I went to my e mail. So I just I just kept trying, and eventually it was there. Yeah. Caltur Alive. You were just clicking on altura Alive? Yeah. Okay. Not sure why, but I’m glad we’re together now. Yeah. I’ve started recording, and I let everyone know that it’s got to be okay. The weeks you can make it, the weeks you can. I’ve been told to not worry because it’s a small class. So I’m glad you can be here. And if you have questions, you’ll be able to raise them that way. Okay. But I am going to start to go through the material. I’ll go ahead and. But pop in, if you have any questions. It’s an odd class because what we go through is essentially your assignment for the week. We’re building a report that you need to turn in in week seven. Each week now starting this week, your assignment will be part of that report. And your discussion, which you’ll complete before your assignment will be part of clarifying what should go in the report. Then hopefully I can give you feedback by Thursday or Friday, each week. Depending when you turn in your discussion, and that should help you finalize your report by Sunday. So what we’re going to do is remind you that this is not an APA report. You won’t have all the same sections that you have in the APA paper. Some of the sections will be the same, but some will be quite different. The idea is that this is a report that you would give to a board of directors of an agency you were working for or that you would give to a funding agency that you were trying to get money for the particular nonprofit that you were looking at. And so it wouldn’t look like a typical research paper. It wouldn’t look like something in the academic world that we would produce, like you’re typically used to at Kaiser, producing APA papers. We’ll follow the guidelines starting in Chapter nine on page 220. Even though this course doesn’t assign Chapter nine early, it assigns it by week seven. You’ll be referring to it throughout the semester as you work on the varying assignments. So please look ahead to page 220 in your book as you start to do your assignment this week. Really, you’re going to be writing this report in about six weeks flat because we started this week in week two and you turn it in in Week seven. I know you started to think about it last week and you’ve identified your agency. But some of you are re rolling on that and working a little harder on a second agency if the first one didn’t work out. This is the first week you really have to produce something for the final project. I will give you feedback on your assignment in the next two or three days and your discussion, hopefully by Wednesday for sure. Last week’s discussion. This week’s discussion, I’ve got to give you until at least Wednesday to start posting there. So what you’re going to do this week is based on either the interview you did or the review of the website that you did, depending if you had access to someone to talk to, or if you just had to review a website for an agency. You’re going to start to write about the one or maybe two services that you would evaluate for effectiveness. The whole goal of program evaluation is to see how effective the agency is at the outcomes they’re interested in doing, they’re interested in these treatments. How did the outcomes work out? Are they effective or not? The very first thing you’re going to do this week is your discussion. Again, based on the information you obtained from your interview with a key personnel or at the agency last week or the evaluation you did of the website, you’re going to identify one or two areas of service to evaluate for effectiveness. Although the agency may offer many services, keep in mind that this course is very brief, a total of eight weeks, but again, you’ll be getting this project done and about five or six weeks. Therefore, just one or maybe two key services should be selected. You don’t have to choose the second one, but if you feel like in the flow of your work, that’s what you want to do, then you’re welcome to it. What will you do this week? You’ll do parts A, B, C, and D of the discussion. You’ll explain the following for each service that you’ll evaluate. Again, probably one service, maybe two. Um, so based on that, you’ll set out to talk about what program goals you have? That is, what is the purpose that the service is providing, right? Is the service trying to reduce mental health symptoms? Is the program or agency trying to reduce substance use? Is it trying to increase educational attainment or passing the GED? What are the outcomes that are important? What is the program goal for the agency? You might pick a secondary goal. Maybe if you’ve seen a reduction in mental health symptoms or a reduction in substance use, you’d secondarily look at a goal like family engagement or family involvement, or employment or something like that. But primarily, you’re interested in the reduction of these symptoms or the reduction of use, or maybe if we’re looking for increases in education, GED, pursuing higher Ed, et cetera. The primary goal is the reduction of something. A secondary goal might be on the level of functioning. Again, functioning well within a family, functioning well within an employment situation. The first goal or part A of the discussion is to identify that program goal. What are you trying to see if the agency is effective at doing? Are the effective at reducing mental health symptoms? Are they effective at reducing substance use, increasing education, increasing foster placement, your name. Then we look at part B, the evaluation question. Now you’re going to restate the goal in terms of a research question. Your research question may be, is X Y Z agency, effective in significantly reducing the symptoms in their clients or reducing substance use in their clients or increasing educational attainment in their clients. The evaluation question is a restatement of the goal in terms of a question. We’re going to get into what measurements you might use? If you’re interested, for example, in substance use, you’ll probably use something from NTA. If you’re interested in mental health reduction, symptom reduction, you’ll look at either mood disorders or anxiety or whatever your agency mostly treats. Okay. The key here is that you’re going to want to see, is there an effectiveness? Is there a change between the intake value and the discharge value? Are their symptoms higher significantly when they come in, then when they leave? Is their substance use higher when they come in at intake than when they leave at discharge? The reverse. If you’re looking at something in the education realm, are they doing better off at discharge in terms of educational attainment than they were at intake? If you’re looking at a inpatient agency, some rehabilitation or detox program where they come in and they have no access to substance abuse or substances while they’re in the agency. Then you’re going to want to look not just at intake and discharge, but probably something more like intake and six months after they leave. Six months after discharge. But we’ll talk more about that when we get to measurement. One of the notes I want to make here is that week five is going to be a particularly heavy week of work. It’s when you’re going to make up the data. We’re not going to actually collect the data, as I said last week. You’re going to have to make up the data. And so as I’ll note a couple of times tonight, Week five is a particularly busy part of the course. But back to this program evaluation question, this is going to tell us whether what the administration is doing is effective or not. So the program evaluator, that’s us. We’re the ones hired to come in to be the program evaluators, we’ll tell the administration what measures we want them to assess. But we won’t do the assessment ourselves. The assessment will be done by if this were a real project and we were really collecting data. The program evaluator would tell the administration what to assess. Then we would deal with the data. So we’d let them collect the data? We would assess the data. Again, because we’re not doing this project in reality, we’re just making it up, I want to be clear about what our role would be at each stage. We tell them what they should be doing, but they do the administration piece of it. We’re the research designer, but not the collector. Okay. Any questions on A or B so far? And I know that puts all the pressure on one student, but I’ll just keep going unless I see a question. Well, I do have a slight question because as you know, I’ve had to change my agency Ag So I think maybe I should wait till at the end because the agency I the input and outcome of the evaluation is going to be a little bit different. Well, the e the research question is going to be completely different. Because you’re still using the same agency and just not talking to them or because you’re using a brand new agency. Brand new agency. I actually worked for a nonprofit and decided, well, you know what? I’ll just go with that. Yes. Yes. Didn’t know what is the goal that you’re going to measure the effectiveness of? Well, it’s most likely depression. It’s a nonprofit for hospice. So I thought it would be a little bit difficult with the input and output, the results. But is that one of the aims of hospice to reduce depression? In the families, yes. In the family members. Okay. So your unit of measurement won’t be the person who’s actually in hospice, but in the caregivers. Right. But I was looking at a specific program with the Veterans Administration. They have other programming. So that’s Yes. Maybe I’m going to kind of trail off into a specific programming. I want to be very specific here. Yes. More specific you can be the better. And so if it’s with veterans families, reductions of depression, is that something they already measure or that you would wish to measure? They already measure. I’m trying to get as detailed as possible because it can go really big. That’s it initially. Yeah. So that’s why I figured, You know what? I’m going to Yeah. Let’s just really, really narrow it. Yeah. But I that the information to you by in our assignment. I’m like, put it together, and we’ll see how it goes. Okay. So between the discussion and the assignment, when we talked yesterday briefly on e mail, you went in the new direction for your assignment. Yeah, very great. Great. Okay. And I will give feedback on that. And I do agree, the more specific you can go, the better. Yeah. It was a fear. It’s a fear. I get it. I get it. All right. So we’re good on A and B, at least in terms of what you know you need to do. Yeah. It’s clear. Okay. Let me move on to C then. Here you’re going to provide what evidence you’re going to present. What is the standard that must be reached in order to demonstrate that the goal has been accomplished? What is the standard? Well, in the example you just used, it would be a significant reduction in mental health symptoms for a caregiver. That would be the standard, significant reduction in symptoms for a caregiver. For others of you, it could be significant reduction in substance use. Again, at intake and at discharge, or if it’s inpatient, then it would be at intake and at maybe six months post discharge. Um If you were working in an education setting, maybe it would be a significant increase in the education attained. What standard would have to be reached between intake measurement and the discharge measurement for you to have evidence that demonstrated the goal had been accomplished. Finally, data collection, without yet talking about the measure. I know this is a little fuzzy, but we have to introduce this in a way in our discussion that we’re starting to make sense of what we’re trying to do before we narrow down that measurement. Here, for data collection, we ask ourselves, what variable or variables, but we want to stay as few as possible, need to be measured in order to provide the evidence. So if we’re looking at reduction of depression, we would need to measure the variable of depression in the caregiver in the case that Lanette just used. So for mental health, you’d want to be as specific as possible if you’re studying anxiety reduction. Or specific as possible if you’re studying mood disorder reduction. If you need some broader symptoms like mental health symptoms generally, we’re going to want to look at broader measures. Here’s where you’re trying to really scale things to the appropriate place. If, for example, you were looking at trauma, You’d want to look at a measure of PTSD, right? And therefore, if you wanted to see a reduction in PTSD from intake to discharge, then the data collection would have to be around the variable of PTSD, right? It’s all, I know, very small and incremental. Okay, I just said this in C, it seems like, but in D, we’re really specifying, right? So, essentially, you want to really gear your assessment to what the program treats, to what outcomes the program is interested in. That’s your discussion for the week. Remember to support your discussion with at least two scholarly sources that focus on evidence based practice in social service agencies. You want to go out there and do research. Yes, Fink can be one of your scholarly sources, but we’d like to get beyond that. Fink being our textbook author. It’s really pretty easy to Google search and to go into our library proquest and other kinds of searches to look at social service agencies specifically, evidence based practice, to say, here’s the reason why we want to collect data that looks for evidence about the program’s goals, because that’s what agencies do. And we cite in a source that says, this is what agencies do. They go out there. They have a goal to see that their treatment is effective. They gather evidence about that after posing a research question and they collect data specific to the evidence they want to have. Don’t forget those two scholarly sources. They can come up either in your first or in your response in the discussion, but at least one in your initial posting. You should have at least one scholarly source there, and then in follow ups, you could put another one if you didn’t fit two in. But I think you’ll fit two in quite easily. Now we move on to the assignment. And this assignment is going to be the first part of what you turn in in Week seven. What we’re going to do is we’re going to build your report in Weeks two, three, four, five, and six. You’re going to get feedback on that portion of the report. You’re going to fix it up a little, and then in week seven, you’ll present it as a finished product with all the sections put together. So you’ll be writing the first portion of the program evaluation report. Again, it’ll be helpful to skip ahead to Chapter nine in the textbook, starting with page 220. I’m talking about the electronic version, but I’m pretty sure the page numbers map on if you’re not using the electronic version. At this point in your evaluation, you should be able to complete the introduction, which includes three sections, and I’m going to talk about each of those sections now. Section one will be the problem. Section two will be the program, and Section three, which is only going to be one sentence long, will be the evaluation. In Section three, you’ll incorporate the goals and objectives that you discussed in the discussion this week. There’s a reason we do the discussion first. It’s a clarifying assignment, and then you’ll go into the report production. Here’s what you’ll do. You’ll start with a title for your report and that title should be pretty simple. It should simply say program evaluation of XYZ agency or the name of the agency, Colon program evaluation. One of those two titles, program evaluation of XYZ agency. That should be centered and bolded. Then you should move on to or that should be done as a title page, me. That should be done as a title page. That’s why you would include information like your name, the program you’re in, the university you affiliated with, a typical title page. Then you move on to the second page of the report and you start with introduction. In this case, we’re actually going to use the word introduction. After all the years you’ve been trained, don’t use the word introduction in an APA paper. This isn’t an APA paper. We are going to use APA style citations and an APA style reference page. But otherwise, it’s not an APA style paper, it’s not an academic paper. You’ll use the word introduction and you’ll center and bold that. And then you’ll describe the agency to us to me, I guess. Describe the agency by saying XYZ agency is located in Fort Lauderdale, Florida, houses 100 homeless individuals and ten families. I’m going back to the Broward report that we used last week, the PowerPoint example that was provided. You can say they provide A B and C services, and this program evaluation will provide or assess the effectiveness of and then you say. What are you going to assess? You’re going to assess, maybe in the case that we just we’re talking about, mental health treatment, programs at the agency, or depression reduction in hospice as laid out by Lanette. Then you go into Section one called the problem. Your introduction is going to have three sections. Section one is called the problem. I would justify the problem and maybe underline it. Again, this is not APA style, but we want to have some technical organization to it. The problem essentially is why we should care about this program evaluation. And why is it important, right? Why should we care about the work of the agency? Justify the agency, explain its importance. This report is probably going to go to funders to people who want to make decisions about what agencies are worthy of their money. And so the first thing you have to do is say, you know, this is a really prevalent problem. Homelessness in Fort Lauderdale, depression among veterans in Hospice, or among the caregivers for veterans in Hospice. You can get information on this prevalence from websites, like the agency you work for. But even better is if you can back it up by folks like the Centers for Disease Control, the CDC, the National Institute of Mental Health, NIMH, the National Institutes of Health, NIH, NTA, the National Institute for Drug addiction, So NIMH, NIH, NTA, the CDC, these are the kinds of places you can go for some data on the prevalence of the problem that the agency is trying to address. What are the implications or outcomes if we don’t address what this agency helps with, right? If depression continues in this group of individuals, what are the likelihoods of suicide? What are the likelihoods of broken families of substance abuse, of job issues? You can make the case that this is a very prevalent problem, and if we don’t address it, there are these outcomes that are quite serious. You probably will only need about one or two paragraphs for that section on the problem, one or two paragraphs. Then you move on to the program, Section two. You might have one or two paragraphs here, possibly three paragraphs, but program evaluation reports are about being concise. The more you can pack into the fewer sentences, the better. Okay? Here, again, you’re going to give the justification for something. But here it’s the justification for the type of treatment that the agency engages in. What is the solution or the predictive best answer for the problem you’ve just described. Now that you’re describing the program, you want to be specific. You want to say there’s research evidence for the effectiveness of this treatment for this population. We know that by using X program, we reduce y symptoms among z population. You want to demonstrate that the treatment is really the treatment excuse me, that the agency is providing is very effective. You want to show from the research literature that the treatment that the agency is providing is very effective and evidence based, effective and evidence based? What does the organization use? Does it use cognitive behavioral therapy? Does it use an eclectic form of therapy? Does it what’s out there for it to be saying that it uses? Then what evidence is there in the literature that that’s effective for the symptoms they’re trying to reduce? Then we move on to Section three, which is the evaluation, literally, this is going to be one sentence long. Here you’ll incorporate the goals and objectives that you discussed in the discussion this week. I’m going to read a sentence to you that’s going to be very familiar in all your papers. This is the sentence I expect to see pretty much in the evaluation section, which again, you left justify, you underline just as you did for the program, and you did for the problem. That sentence should read something like this. This program evaluation will demonstrate the effectiveness of X Y Z program treatment by determining, and I’ll repeat this sentence. If there is significant reduction in A B and C symptoms, we’re going to stick with one symptom, maybe two by certain dates, and those dates would be the intake time and the discharge time, or the intake time and six months after discharge. You go back to the program goal. You stated in your discussion, and you write a sentence like this. This program evaluation will demonstrate the effectiveness of XYZ’s program treatment by determining if there is a significant reduction in ABC symptoms by the dates of intake and discharge or six months apart at intake and discharge. If it’s not a significant reduction you’re looking for, but a significant increase like foster assignment or educational attainment, then you, of course, don’t want to say significant reduction, you want to say significant increase. Again, the discussion leads to the assignment. The assignment will be maybe a page and a half, two pages double spaced at the most. Because remember, you have many sections to write, and at the end of this, we don’t want the report to be any longer than 15 or 16 pages. It’s very important when you reach out to a funding agency, when you communicate with the board of directions, that you’re concise, that you don’t go on and on and on and on, but here we go, the problem, the program, the evidence, et ce Excuse me, the evaluation, and so on. Be concise. If you’re looking for a few more details about what this project is going to be and how you’re going to do it, we are going to have about 100 participants in our fake data in our made up data. But I will explain through several videos how to do that. For example, you’ll probably only have to make up about ten rows of data, and then we’ll show you how to copy that ten rows nine more times. You won’t be hand entering 100 rows of data, rather, you’ll be entering ten rows of data and then copying them nine more times. You are going to turn in a separate document each week. As you’re building this report, I don’t want to see this week’s work attached to next week’s work. I just want to see this week’s work for this week, next week’s work for next week. Week four’s work, in week four, and so on, until week seven when you’ll put the whole thing together. One thing that you could have running along each time is the references section. If you provide references this week and you start to build your references section, and then you have more and you will have more references next week, you can just add that into the references section, And I’ll try to remember, not to say something like, Oh, you cited something that you didn’t have in your writing this week, you know, because then you’ll want to say, Well, I had it in the writing last week. But I want you to probably keep building that references section as you go so that it’s nicely organized and alphabetized, and all those things are done well by the end. I don’t want you to have to sort of cut and paste later. But all the other sections should be handed in separately each week and then compiled in week seven. Speaking of seven, remember to use the seventh edition of the APA manual. That’s what was listed in the syllabus, but there were a few changes between Addition six and addition seven, like when you use and this thing. Make sure you’re using the seventh edition. It’s what you’ll use for your dissertation. The more familiar you can become with that the better. Okay. Any questions to put this on Lynette again? Is this clear for this week, or do you need a little more time talking about your project? I think it’s clear for this week. I just have to kind of make sure I have everything in order. And narrowed down. Yeah, because it was still last minute. Yeah. That’s tough. But fortunately, you work for an agency, and I know it’s a big one, and it has a lot of aims. But I think if you narrow it down and you focus on what’s asked for each week, you’ll be okay. Yeah. Yeah, absolutely. Thank you. To get ahead of yourself. Thanks for being here tonight. I appreciate it. Okay. Thank you. You. You too. Bye bye. Bye bye.

,

3

Week 2 Discussion Evaluation

Student Name

Institution

Course Name

Instructor

Date

Patients’ experiences in community mental health hospitals have continued to deteriorate, and many people in the community are reporting increased difficulties in accessing these services (Torjesen, 2022). Based on the information obtained from the interview with key personnel at the agency last week, Mental Health America offers various range of services to improve mental health involves numerous services such as promoting community support and well-being, reducing stigma, and increasing access to mental healthcare. This week, two areas of focus that I will evaluate for effectiveness in the agency will be increasing access to mental health and reducing stigma.

1. Reducing Stigma

Program Goal: The goal of this program is to ensure that the agency focuses on mental health stigma reduction, this will be achieved by fostering an inclusive community environment and educating members of the public.

Evaluation Question: To what degree do MHA’s educational initiatives impact the reduction of mental health stigma and discrimination in the community?

Evidence: According to Colizzi et al. (2020) prevention, promotion, and early intervention strategies on mental health have a significant impact on the well-being and health of the people in the community. Therefore, the success of this program will be demonstrated I post-intervention research shows that there is a significant reduction in stereotypes, and misconceptions about mental health when compared to the data before the program.

Data Collection: To effectively evaluate the success of the program, data on variables such as attitudes toward mental health patients and rates of stigma pre- and post-intervention will be key.

2. Increasing Mental Healthcare Access

Program Goal: The goal of this program will be to improve access to mental healthcare services for populations that have limited access by removing systemic, financial, and geographic barriers that prevent them from accessing the care.

Evaluation Question: How much does MHA contribute to improving access to mental health services for hard-to-reach populations?

Evidence: The program will measure improvement in mental health access, the success of this program will be demonstrated through the change in numbers, an increase in mental health access among the minority and underserved populations and a reduction in the amount of time they spend waiting will be a crucial measure of success.

Data Collection: Various variables will be crucial in highlighting the desired changes, these variables include several referrals made compared to those that were fulfilled within a specific time, rates of utilization of the service, and demographics of the mental health patients.

Generally, when collecting client data for both services, you should be mindful of maintaining confidentiality and obtaining informed consent as stated in the ethical standards. The evaluation process can be based on the guidelines for evidence-based practices in program evaluation provided by Fink (2015) and used in the current study. Stigma reduction and access enhancement are two of the main goals of MHA and it is important to appreciate the extent to which MHA is achieving these goals. The evaluation will be evidence-based and outcomes-based with case measurement of change in attitude within the target population or community and the effective rates of utilization of the services offered. Ethical standards such as the guarantee of confidentiality and informed consent will be used to ensure privacy.

References

Colizzi, M., Lasalvia, A., & Ruggeri, M. (2020). Prevention and early intervention in youth mental health: is it time for a multidisciplinary and trans-diagnostic model for care?  International journal of mental health systems,  14, 1-14. https://doi.org/10.1186/s13033-020-00356-9

Fink, A. (2015). Evaluation fundamentals: Insights into program effectiveness, quality, and value (3rd ed.). Thousand Oaks, CA: Sage.

Torjesen, I. (2022). Access to community mental health services continues to deteriorate, survey finds.  BMJ: British Medical Journal (Online),  379, o2585. https://doi.org/10.1136/bmj.o2585

Last Completed Projects