What Counts As Success? Assessing The Impact Of Civics In Higher Ed

- Hi everybody. Welcome to What Counts To Success, assessing the Impact of Civics in Higher Ed. I'm Peter Levine from the Tisch College of Civic Life at Tufts University, and this is a alliance for Civics in the Academy webinar. So just a little bit of intro for those who haven't joined these before. First of all, the Alliance for Civics in the academy is about a year old, was launched about a year ago, has grown very rapidly and with wonderful people, and membership is open. So we urge you to join us. There's an application link, I think, yes, I think it will be put in the chat or it will be in a minute. One of the things we've been doing, but we've been doing some other things that I'll mention is a series of a webinar series. So this is something like the fourth. And in addition to that, the Alliance also provides seed grants for members. So if you join, you're eligible to apply for seed grants. And the seed grants are basically for hosting meetings. And there've been a bunch of those. Also the Alliance disseminates practical resources on the website. There are things like syllabi, program descriptions and publishes, publishes, often republishes, but sometimes original things. Contributions by members of the Academy of This Alliance, specifics in the academy on topics having to do with civic education in higher education. It's a diverse community. We really tried to build it that way and recruit that way. It's, there's a lot of different approaches to civic education, a lot of different philosophical views, political views, and it's meant to be a robust place for this kind of conversation, including on these webinars. You can, especially if you've joined, you can send us your own resources, syllabi and commentary. And you can do throw, you can do throw through the website or you can also email Hoover aca@stanford.edu. The next webinar, write it down right now is on March 4th at also at at noon Eastern nine Pacific. It's with, it's about how universities can strengthen civic education in K 12 schools. So it's about the nexus between higher education and K 12. And the wonderful panel is Jennifer McNabb, Joshua Wood Dunn, Mira Levinson and Jenna Story. An excellent panel. Today though, the conversation, well actually it's not completely distinct from that, but it's a little different. It's about measurement, assessment and those things. And we have a all star panel. I think these are also my friends, so hi every hi to you. I just want to introduce 'em and then we'll jump into the conversation. So in, I guess let's say in alphabetical order, Joe Khan is the Dutton Presidential Professor for Education, policy and Politics at the use at uc, Riverside and director of the Civic Engagement Research Group, which is a very influential and prolific organization. Joe chaired the MacArthur Foundation's Youth and Participatory Politics Network and was on the American Academy of Arts and Sciences Commission on the practice of Democratic citizenship, which has really laid a roadmap that a lot of us look to for how to recover American democracy. He's a member of the National Academy. He's a and a fellow of the American Education Research Association. I wanted to mention that just because it's relevant to our conversation today. Joe's 2004 article with Joel Westheimer called What Kind of Citizen Includes measurement models for different kinds of citizenship. And so it's a foundational text. It's been cited nearly 5,000 times in subsequent literature and it's very widely taught. And right now I understand that Joe's partnering with our friend Dave Campbell and Dave, our two friends, Dave Campbell and David Kidd, on a project to develop new measures of civic outcomes. So if I stick with alphabetical order, trig Phite is a research professor in higher education at Ball State University of Indiana and associate director of the Center for Economic and Civic Learning there, right this year he's also a fellow at the ho, at the a National Civics Fellow at the Hoover Institution. He co-founded and has led and I think continuously lead right to the Twin Cities based Institute for Public Life and Work, which promotes and rethinks civic life. He's historian of US politics and foreign policy, especially in the progressive eras written about William James and Woodrow Wilson. He used to be for eight years, the Editor of The Good Society, which I just got my latest edition of. This is the Journal of Civic Studies and it has focused the past five years on developing, expanding what's called Third Way Civics, mostly in the state of Indiana but beyond. And we, you can talk about that in a few minutes. And then Rachel Wall is Associate professor of Education at University of Virginia. Her website I thought captured this nicely. She examines how ideas and ideals, how ideas and ideals spread through education and advocacy, particularly in regard to state and civil society efforts to influence each other, which is a really nice framing for what we're talking about today because how we measure something is a way that we try to spread ideas and ideals through society. Her 2017 book with Stanford is entitled Just Violence, torture and Human Rights in the Eyes of the Police. And it's a, it's a based on close interviews with Indian police officers who may have been involved in violations of human rights. And the fascinating study of how they process that and how they think about it. Her forthcoming book, which I got to read a manuscript of and really learned a lot from and enjoyed, is called Keeping Our Enemies Closer Political Dialogue and Polarized Democracies. And I just wanna say Rachel also both studies and writes about kind of deliberative processes and discussions both on campus but also off. And that that's a whole, a whole set of questions about how to measure that and how to measure the outcomes of it as well. So those are our colleagues. You will get to ask questions of them because I will make sure that we start looking at the q and a after a bit, but we're gonna start talking among ourselves first. And I think what I'm gonna add do is pose a question to each of the colleagues first in order, but then you guys should always feel free to jump in with dialogue with each other. So my fir my first question was for Joe. So Joe and I worked together for most of this, most of this century. And I've watched Joe building the field, especially of K 12 civic education, obviously meanwhile teaching in college, but building the field of K 12 civic education and often intervening there by developing measures, assessing measures, advocating them, but in complicated ways. So I just want you to reflect on that. I mean, maybe my question is, why have you done that? Why, why, why do that? Why try to intervene in that way?

- Yeah, thanks Peter. And thanks to everybody for this opportunity, which I really appreciate. And as you're is sort of embedded in, in your statement, Peter, I've done some work in higher ed, but most of my work has been in K 12. So hopefully the similarities and differences will be helpful. In thinking about this to your big question, there probably are a couple of answers that overlap. So one of them is that, you know, in a, in a you sort of simple way, I buy into enlightenment logic, like you should study and see what the impacts are and you should be informed by what you learn. And rationality, you know, should play a role and evidence should play a role in all of that. And, and in K 12 that's important because both funding in civics has been so slim and funding for research in civics has been so slim that there often isn't that much empirical evidence about the impacts of things. And you know, when people often don't have a lot of money for research, they study their own practice, they study true teachers, they do things that can be very, you know, can lead to important insights, but but that are limited in their ability to make sort of broader or general statements about those. And so some of the work is, is I do it just 'cause I, I think it's really exciting and important to have quantitative measures and quantitative data about what's happening for most kids or for a representative sample of kids in a given context. And, and to see what are the relationships that that you have, especially between learning opportunities and outcomes. I think a lot of times when we think about measures, we think about measuring outcomes and that's obviously important, but, but sometimes that leads to not measuring learning opportunities and learning opportunities are the thing we actually have more control over. And, and that's what gives, you know, our students, you know, what, what they came to school for. And so, and so I think one of the interventions in a sense that you're talking about in, in the work has been to really highlight measures of learning opportunities and try to get people to focus more on that. And then the last thing I'd say is, you know, well I love the idea that evidence and science can drive, you know, decision making and practice to some degree. It's also just because I think both quantitative and qualitative work and efforts to measure things carefully can promote reflection. And so a lot of the tools that we've developed have not so much been measures that would allow for causal claims about how doing X in your classroom leads to y as much as they've been ways to collect indicators of things that the people within the learning community, the professionals or the students care a good bit about. And then to look collaboratively at that information with the hope that it will help foster the kind of community that will lead to growth and and developmental practice. So I'll leave it there, but

- Thank you. Yeah, thank you. Perfect. And we'll come back in particular, we'll make sure that we address the limitations of what you call the enlightenment kind of paradigm. 'cause I know all of us see both needs to think that way and also limitations, so we'll, we'll, we'll talk about that Trig, I mentioned third way civics, but I just named it, I didn't try to define it or let alone justify it. Do you want to, do you wanna take a a a minute to talk about third way civics and maybe a little bit about the role that measurement is already playing in it, but I I think also people should just hear about third way civics.

- Yeah, thank you. And, and thanks everyone for, for joining us. Yes. Third way civics, or three WC is a shorthand I, I use and others use was designed first as a way to provide undergraduates with a nuanced account of American history and politics that was sort of designed to transcend the, what we then thought were raging history and civics wars, which are continuing to rage. It has at the same time allows students opportunity to develop the skills to navigate difficult histories, but also their, the shared social presence that they share with, with their fellow citizens, the, the people who make up their communities. We very soon expanded our focus realizing that what we were doing that was really powerful was changing the way that instructors taught and helping them find ways to organize their courses in their classrooms so that students were independently and collaboratively wrestling with information and generating meaning the same way we want them. And this is a normative project in this way, the same way we want them to interact with meaning and other people out in society. So really habituating them, developing the character and just the habits of learning and then acting on learning in a particular way that is not, that is not solely collective. It takes independent inquiry and reflection, but always takes those next steps of collaborative inquiry and negotiation across differences and developing Peter, what you might call a, a, a grasp of the facts and the values and the strategies that might be possible for, for you and your peers to come to some common way forward when it comes to understanding an issue or taking action. So that's what we do in Third Way civics, it's really focused on immersive and deep kind of culture change amongst the professoriate, the teaching, the teaching core in undergraduate education. And of course we wanna measure this because we want to understand, first of all, do students, we want to make the case that students respond positively to this. We want to make the case in a higher ed environment that is so focused on promising high paying jobs and the right credentials for the future job market. We want to make the case that actually students want to live publicly meaningful lives too. They, they want college to offer them a chance to develop their civic character and their civic skills without indoctrinating them telling them what to think, how to act. So we measure that, we try to measure that enthusiasm among students and we do that through enrollments. We do it through student feedback. We do it through the, through very carefully working with our faculty to record their own observations about student engagement. And my enrollments were up again the next semester. And, but we also want to measure the types of knowledge and more importantly, the way the program has developed the, the skills and dispositions or or values that change among students. And we do that primarily through pre and post surveys, which of course are problematic because it's students reporting on themselves. But we found a few things. We do think it's important if a student starts out in class or many students start out saying, I don't really think structuring a society to promote all this kind of democratic interaction is really all that important 'cause what has it done for us? And then they end the class by saying, well, I'm this much more committed to the importance of that than I was before. We think that's an important finding. And then we, we work of course with faculty to, to ask them, do you see this in your classroom? Do you see students practicing habits that we want them to practice getting excited about what they can discover when they open up their minds to new, even previously unpalatable ideas, et cetera. So we do that because we want to be able to demonstrate that things are working. Like Joe said, we want to, we want information, we want people to reflect on the changes and also the, the values that they are developing. We also though just really want to generate enthusiasm and civic hope. And so we, we rely a lot just on storytelling as well. I, I used to work for the Minnesota Humanities Center and the Minnesota Humanities Center. I live in Minneapolis and the Humanities Center helped to incubate this initiative. And, you know, we were all about the power of storytelling to catalyze change. And I think we all, if we take a step back from the pressures of showing that, oh my gosh, every student is going to have this wonderful transformative experience, well sometimes you just need a few people to experience a major transformative change to alter the course of history or many people to have just a few s make a few small changes in the way they value something or practice living with other people. So we really try to generate enthusiasm and use stories also to open up people's imaginations so that we are not bound by the particular formula of third way civics that we are using currently, but instead can kind of glean from people's stories. This is what really worked serendipitously in my classroom. So that's our general, that's third way civics in a nutshell. And our, and in the way we, we use thank you measurement to,

- And there's a no, it's great. Everything you said is helpful and lots of different angles on this. So first of all, you're measuring students and faculty, you're thinking there's different timescales. I mean, to change the course of history is a different timescale than a pre-post. And there's, there's the question of the values. So there's a very value driven normative conception of what positive changes. Sometimes people can debate what, what, what changes positive and also, and there's a issue here about kind of what are we neutral about and what are we advocating, right? Which is just a really good question, Rachel. So at least the work that I know of yours most directly is more about people involved in dialogue and deliberation in different settings including campus. So, but also not including campus. So I'm wondering how you think about that. So do you think about it more as a means to the end of certain kinds of outcomes that people should deliberate so they can get certain kinds of outcomes or do you think of it as more of an outcome we're trying to build deliberative citizens? And also how does you also do a lot of extremely valuable qualitative research and qualitative research is a form of measurement, but I wonder how you think about measurements and assessment in, in the work that you do.

- Sure, those are great questions both in answer to the first question, is it a means to an end or is it the end? I see it as both. I think that being able to talk to each other is obviously a good of democracy, but it's also something that in my research I have found leads to other goods of democracy. And when I started the research, it was because I really was curious whether it was a means to any end. I, we are all surrounded, I think in spite of the lack of examples of it, by the idea that we should be able to talk to people who disagree with us, that it's an important civic skill, but often without a lot of clarity on exactly why, what the good of it was other than making it less likely that we resort to violence against people we don't like. And so I started doing this in-depth qualitative research on different kinds of conversations, as you said, in different kinds of public settings. And most relevantly for this case on university campuses, students who are intentionally brought together for two hour discussions in some cases over the course of a semester, in some cases, one-off sessions seated at tables with an ideologically diverse group of peers for these pretty structured conversations in which they're asked to try to understand each other. And I really didn't know whether it was going to do any good in terms of a skill beyond itself and means to any end. I also wondered whether it might do some harm. I was sympathetic with the idea that it might lead to a kind of moral equivocation in which students realized that there's nuance and then they ended in paralysis and also was sympathetic with the idea that it might just lead to a kind of exploitation of students who were representative of diverse views, but less able to articulate themselves in high status language. And so I was really curious about what the ends would be. And what I found somewhat to my surprise, because I wasn't expecting it, was that almost ubiquitously across different kinds of settings, but most obviously in college settings, students came to not shift in their political beliefs, although that surprises, but to shift pretty dramatically in their feelings about the, the peers that they disagree with that students did in a pretty significant way come to see the people they disagree with as morally recognizable as somebody whose motivations were relatable, even if not persuasive. And this, I found this to be in three different stages, you could say one, the most common was personalization, just kind of realizing the individuality and humanity of the people they most disagreed with. Two, also very common was recognition that others are well intentioned, recognizing that there is a moral source at the heart of other people's views, often even if it's not one that you adopt as your own. And third le this was the, the least ubiquitous and the most robust, but also not infrequent, was students realizing that there was actually coherence behind others moral views, at least sometimes even though they were no more persuaded by those views. And basically what I saw through these interviews over years is that it has the potential not just to reduce the tendencies toward violence in any kind of pluralistic society, but also potentially to enhance students' desire for broader inclusion to give students the sense not only that other people who vote differently than them have a right to political participation, but actually it's good for them to have to be included in political society. So I do think that it, it has the potential to reduce support for policies that effectively disenfranchise the other side to reduce support for conspiracy theories that arise when you can't believe that anybody would've possibly voted for the other side. So I, and and also I think there are clearly epistemic and ethical benefits to these kinds of conversations. So that's an answer, a long answer to the question of yes, it's an important civic skill, but also I do think it has these other benefits to democracy.

- Thank you. So actually I think what I wanna say to sort of everybody and the, all our attendees and everything is this is to stimulate you to say things in the q and a to type things in the q and a. So I hear Rachel kind of laying out a pretty compelling model in which there's a learning, as Joe said earlier, there's a learning opportunity which is these structured dialogues and there's a set of outcomes at the individual level that have to do with at mostly attitudes and understanding and that she's measuring it. And it is interesting, it reveals things that we didn't know before and then there's longer term outcomes that has to do with perhaps less violence and less disenfranchisement. So that's a neat model. The what what I wanna say is people have other models as well in this field and in maybe the four of us, including Rachel, have other models as well. So a model in which students learn how the constitution is structured and as a result support constitutional rights more or whatever, there's a, or they learn about the classical heritage of Republican and self-government and as a result they learn certain things and and so on. So there's a lot of different models. So what I'm just trying to ask you to do as a participant in this webinar is think about the model that matters the most to you, and maybe you can share that or ask about it. We, what, what I think we should talk about as a panel though next is sort of the, the measure, the role of measurements in, in the larger strategies for, for change. So, you know, when we were planning this, Joe, you mentioned Tony Bri's kind of maxim, you, you can't do anything at, you can't make anything happen at, I don't think this is how he exactly said it, but you can't make anything happen at scale unless you have a measurement process that's effective. You can't persuade stakeholders to invest in something unless you can measure it and show that you can affect things. But I I wanted to hear you talk about that. I mean, do you agree with that? Is it, do you agree with that? A hundred percent. Do you agree with that 50% you've worked in that domain for a long time?

- Yeah, so I, I think it is a powerful idea that if you wanna improve something at scale, having a way to measure what that thing is is super valuable. I do largely think that's really, that's different though than saying that's all you need or if you have that you won't get unintended negative consequences. And so I think embedded in what you're, I think trying to get at here is, is just that what we need is a kind of nuanced stance towards measurement. We need a stance that can recognize dynamics that are good, you know, so for example, I think it's pretty clear that if you don't measure anything, it's unlikely that you're gonna reach your full potential in terms of growth, right? If we just say everybody should do what they wanna do, and we're not gonna try to have any forms of accountability or any forms of reflection, you're, you're probably taking something away from your toolkit towards happening. And I think it's probably also true that if you say we're gonna try to create some measures that while imperfect we're gonna make super high stakes and we're gonna try to drive everybody towards those, you know, again, we're probably gonna create unintended negative consequences that are really problematic. And so what what we want is a kind of nuanced stance towards this that is really reflective that says, well suppose we used this measure and suppose we attached, you know, this goal to engagement with this measure, whether it's some form of accountability or whether it's some form of reflection, is it likely to help move things in a good direction? And, and that's, you know, that would be sort of where I would come down that, that there's gotta be a conversation that looks like that. And I think related, and maybe this is a corollary to Tony Bri's comment, you're not gonna improve at scale if you don't create a professional community in which people engage with ideas productively. So, you know, a lot of us spend time in higher ed, a lot of us wouldn't define the communities that surround higher ed governance as all that functional certainly at times. And I think at the same time that we wanna be thinking about measurement, we wanna be thinking about what's the context in which that measurement sets what's the voice that people have in shaping it and how that stuff is engaged with. And that if we bring those two things together, I think we're much better positioned to move measurement forward.

- We talked on the minute call about sort of po contrasting so that Tony Bri said, I didn't quote him well, but he said something about you have to measure to make social change, but there was also Donald T. Campbell, the, the very wonderful social social psychologist who said Campbell's law, the more any quantitative social indicator is used for decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to measure. And I think we were all trying to figure out how to kind of navigate that. So there's the idea that if you don't measure it, you can't change it. And there's also the idea that if you measure it, you corrupt it. And I think Joe, Joe, yeah, I wanna call on Trigon and Rachel, I think Joe's part of Joe's answer is you need to have an explicit discussion among people who can make a difference. Not just a seminar conversation on the side, but a conversation among the people who can make a difference about how the measurements are being used. Chig, what do you think?

- Well, I was just gonna say, yeah, I, I I'm gonna say something that's kind of glib, but that I think is important and then I'll try to illustrate it. I mean, it depends on how you're defining measurement. I mean, and, and not to get hung up on the one thing that is either easiest to measure or that you think sadly a funder, some number that you think they're going to want. For instance, what we try to do in third way civics is basically put students through the type of experience that Rachel is describing, but have them do it every day in class while also learning something that's interesting or useful to them. We can measure how much knowledge or the way people's views of democracy change, and it may be impressive gains or not, but if we don't also pay attention to the stories of what those students do outside of class, you know, the student who reports not really caring that much more about democracy as a form of government over others, but organizes with his liberal progressive classmate, you know, a weekly get together in their residence hall for people to talk about the things that are bothering them across these differences, that to me is a much more important measure or outcome evidence of some kind of outcome than, you know, of all those students who are gathering, you know, to what degree did they change in their reported, you know, habit of checking multiple news sources or something like that. So I think the danger of fetishizing the measurement is, is what you have to avoid without throwing measurements out of the window entirely.

- Rachel, what are you thinking about this? I I I can imagine, well, what are you thinking about instead of my guessing what you're thinking? What are you thinking about it?

- Sure. I have my own methodological biases as a qualitative researcher where I can't imagine understanding these particular processes without long probing conversations with students. Although again, that's just my own methodological bias. But when I set out to understand what was happening for students, I really did not want to measure predetermined outcomes. I didn't want to measure whether a particular, you know, whether they were more civically engaged or whether they were more, you know, positive about that partisan outgroup. I really wanted to start from scratch and understand what the meaning is for students. And I still do the interviews in this way, even though I now have a sense of what outcomes are happening. I still, in my current research, conduct my interviews as if I am starting from scratch again, because I'm really trying to understand for the individual students how they're constructing meaning of the experience and how they're translating it, if at all, into their lives. And so, you know, it's, it's time intensive and it's small and, you know, it's, it's not at scale. You know, I I interview maybe 50 students for each wave of interviews, but it's really trying to understand on an individual level what the meaning is and how it's operating in their lives. So it's, yeah.

- How do, how do you navigate the question of values and ideals in your work that, I mean, I don't know the answer, what you can, I don't know what you're gonna say, but it's, I'm super interested in your answer because I mean, among of the things you, you're, you know, well, very well known for having sat down with police officers who are involved in, in torture and hearing their perspective on it. So one read would, which I don't think is the right read, would be that you're, you're value free and you're just, you're just, listen what your, your job is to tell us what they say and that's it. You're a scientist. I I don't think that's right though. So what, how do you navigate that?

- Yeah, I mean, I, that's a great question. I, I think a lot about how being a researcher is very similar to being a dialogue facilitator in the sense that in both instances you might have deep commitments and yet revealing them would undermine what you're trying to accomplish. And I did worry about this a lot when I was interviewing people who had engaged in torture and other forms of illegal violence. I often worried that I was legitimizing it by asking lots and lots of questions that were judgment free because I really wanted to understand the circumstances and beliefs that enabled it and believed that were my judgment to seep through. I wouldn't get to that understanding. And I also think of myself that way when I'm doing research. And also probably more relevant to everyone in this webinar when I'm teaching, I teach a lot of political dialogue classes where students are practicing these skills and talking about controversial issues and frequently expressing views that I deeply disagree with. But my sense, although I don't think this is obvious or inevitable, is that the more curiosity I give to students and to people with whom I disagree, the more curiosity they give to their own views and their own assumptions and the less defensive they become, and that the more, particularly from a position of an educator who occupies an authority position, the more I am seem to be condemning, the more students or really any person will button up defend themselves or pretend to think differently and not reflect at all on their own beliefs. So,

- And reflecting on their own beliefs might be, might be the core ideal here, right?

- Yeah. So one, one thing I found in the research is that although most students don't change their minds about the actual political views, they hold those students who did change their minds did it in part because they felt like students were asking them questions of curiosity. They started to ask themselves those questions rather than feeling like they had to defend themselves.

- And curiosity might be a virtue you're going after.

- Yeah,

- We could keep, we could keep talking in that path, but I think I, well also take us into different one, which is there's this question floating here about whether, whether you can do something with relatively concrete quantitative measures at scale to make a difference. And so actually Joe, did you wanna talk about the Chicago schools example about how Chicago schools use, you use? I think it's still going on civic indicators to improve

- Yeah,

- K12 studies. Yeah.

- Give sort of two examples and, and I think the risk of examples, people that imagine, oh, those are the ways to do it and, and really, you know, there are a multiplicity of ways to do things that obviously move us towards some goals and see, so you really, it's not a, there's not, I don't wanna imply that this is the way to do something. I'll, I'll start with a, with a, with a, a sort of small example that also speaks to something someone put in the chat where they were asking about authoritarian governance. So in, in Chicago public schools, they've, for the past more than 10 years now, had a huge commitment, at least at the district level to what they call student voice. So their student voice committees at all high schools and at at most elementary schools, you know, basically structures that are designed to make sure that young people can share their views on the operation of the school. I think it won't be a surprise that many people worry that student voices often a performative commitment and not a super substantive commitment. And that it's hard to really reflect on how well is this working? When you're talking about a system with literally hundreds of thousands of kids across many hundreds of schools, you can find lots of examples that are really compelling, but you wonder, so how is it going and how does it relate to stuff? So we work with a group of stakeholders within the system to come up with two questions that we could put on a survey. And they're relatively simple questions. You know, to Rachel's point, they don't get us into a lot of the depth that we'd like in terms of fully understanding this, but we thought they were helpful. One was just when students in class express a concern, the teacher is responsive. And the second question was, when students in the school express a concern, the school leadership is responsive. And literally it's not that we know well, if on a five point Likert scale the answer is 4.1, you know, the average answer is 4.1, then we know something particular. It is more that that opens up the possibility for a conversation and it opened up the possibility for analysis. So one, you do get variation across classrooms and schools with respect to the way kids answer those questions. So then you can start asking, well, are there features of schools that relate to that variation? And that can be a productive thing for a district to reflect on. You can find out, which we did, and this may not surprise people, that actually attendance is meaningfully better at schools where people say, when the, when students at the school express a concern, the administration is responsive. Well, in a district like Chicago where attendance is a really big deal, that gave some energy and some rationale for better funding the student voice initiatives. So I don't wanna argue that it's the end of the discussion, but there are ways in which that kind of an indicator can be useful in the context of a, you know, a meaningfully functional community with plenty of dysfunction embedded in it too, to, to make a little bit of progress. A related kind of agenda, which I guess I might call low stakes assessment, is that the district gives schools, I think it's 19, it may have changed at this point, different goals that they can pick from. And they're supposed to pick three goals and focus on them for two years, and they have to come up with indicators which can be qualitative as well as quantitative of the degree to which they're achieving those goals and how they're going about it. And so they basically create a development plan for two years to reflect on three goals that they pick out of a list that, I think it's 19 civics was one of the goals that, that a district could pick. And again, I think that's a relatively productive starting point for encouraging schools to create a professional community where they're aiming at something meaningful for their students. I wouldn't say it's the only thing you need to do in school governance, but it's not, I think it's probably a productive thing to do as a part of a school governance game plan. You can then imagine bringing together the schools that all picked, you know, option seven, how are you measuring it? How are you thinking about it? What are you seeing? In other words, you're creating the possibility for more productive community-based conversations about how to move forward on something that's meaningful.

- I think both of your examples are compatible with a worldview in which you're pretty trusting of the professionals in the schools to make change, right? So, 'cause we do have a different kind of education reform model where the idea is to have some kind of high stakes test that changes incentives whether people want to or not. And we have that in civics. We have states, we have something like eight states that have, maybe it's more have high stakes civics exams and you have to, it matters to the student whether they pass and there are calls for that at the college level. Do we, what do you all think about that kind of, Joe, am I right and say, in making that distinction, you both of your examples were kind of,

- I think that there is a place for trust and professional community in those organizations. Yes. That doesn't mean that's the only thing I would do, right? Yeah. In, in other words,

- Yep.

- You can understand parents and community members, people who are putting money into these systems, wanting some forms of accountability that don't rely, you know, that don't use those mechanisms. But I, I

- Agree, - Skeptical of the idea that that's the only way to move a system, just as I would be skeptical of the idea that you shouldn't have any other forms of accountability beyond ones that relies you say on on trust. Yep.

- I mean, I think if you don't trust the professionals, it doesn't happen because it, it, they just find if they are not trusted and feel like they are being asked as professionals to be part of this work, they will just find the easiest way to check the box. And that is exactly what they'll be modeling for their students as well. It doesn't mean that you don't monitor or don't invest in particular training for these professionals to help them do something that they have not perhaps been socialized or ever trained to do. But I really think if you, if you can't find a way to both trust and hold the professionals accountable, which is a way, both of which are ways of saying your work is important and matters, you as a professional have value, value enough to be trusted, but also to be held accountable. I don't, I don't think you can just rely on a test or a series of activities that won't just become hoops and rote.

- But Rachel, I wanted to make sure you had a chance to weigh in on this question if you wanted to.

- No, I think they've covered it.

- Oh, okay. Well, so one, we do have questions in the q and a and more are welcome. I think one of or two of the questions actually are, are, are we sort of staring at the questions? Two of the questions actually relate to the same issue, which is, which is basically power. So there, there are powerful institutions such as state governments or boards of trustees in the case of universities which have agendas. And I, you know, the governor of Florida and the state of New Yorker invoked in two different comments by two different people that may not be on the same page, I don't know. But the, the question is, so deciding what to measure is also a way of deciding what to control and how to control it. Are we, but it's also a way for a demi democratic community to control the institutions that Joe just, Joe just alluded to, that the people of Chicago might like to know how their schools are doing. So measurement is a method of control. It's also potentially a dangerous one. Or one way to ask this question is, is, is the current moments the last year or the last 10 years changing your view of how to think about measurement? For instance, one, one reaction might be let's stop trying to measure things at scale because it's too evident that it can be controlled by powerful people's agendas. I

- Think what your question is getting at Peter is that measure it, measuring something doesn't solve the political questions of whether you're building a functional professional community, a a functional and inclusive and just democratic community measurement can be used in authoritarian context, measurement can be used in much more democratic context. That's a different but related problem that you have to be focused on. So the person raises, one of the people raised a very interesting question, I think about New York City, Steven. Yeah, Yeah. You need to make sure you, you should work towards having a department of education in New York that, that has stakeholders with varied views and people who are part of that dialogue who are pushing to say, these are the, these are the kinds of things we wanna be careful about. Oh, it looks like your measurement, whether you're talking about what DeSantis is doing or what someone in in New York is doing looks problematic or this looks good. You wanna be bringing in voices to try to make that better and avoid those potential pitfalls. That measurement isn't gonna be the answer there. And I don't think getting rid of measurement is gonna solve the problem of a dysfunctional or non-inclusive community or anti-democratic community regardless.

- I think two, two responses that I would offer would be, one, don't confuse, don't shrink evaluations solely to measurement. So you, you, you, you, you evaluate programs in lots of ways that, that include but are not exhausted by measurement. And one way to operationalize that is to take a developmental evaluation approach. And this is another suggestion where you are in many forms, you're not only kind of responding to data as you get it, rather than sticking to your predetermined outcomes, which is a wonderful point that Rachel made about the value of qualitative, her qualitative work. You also, in my ideal developmental evaluation setup are ensuring that the people involved the target impact population are also in real time evaluating and responding to what is happening. And, and you build that into the, the project, the goal, the work, whatever you wanna call it from the beginning as a way to counter, now none of these things are perfect and they're, they're difficult and messy and you may have to really, really fight, but as a way to counter that act of using measurement to control from an external or a hierarchical direction.

- Can we think about this a little bit strategically as an alliance for a moment? So we, we, this is the alliance for civics in the academy and I think you guys are members, if you're not, you need to join. So what, what should a group like the Alliance be doing? So let me just say the alliance is mostly targeting for a recruitment teachers of things in the college that have civic implications. So it's, it's interdisciplinary, but mostly it's professors or instructors. It's not a network of universities, for example, from that perspective. But these are people who also really, in order to join you, you ought to be committed to the mission of civic education because that's what it says in the mission segment. Although you can have a different views of what is central to that. So where as a group of people who care about this, who are mostly positioned as educators, should we be putting our attention when we think about measurement? Because we could think about measures that voluntarily faculty members would choose to use in their own classrooms. We could think about demanding that our institutions measure things not necessarily for our own classrooms. We could be thinking about our disciplines. Certainly the American Political Science Association has had various forays into this over the years. That's the discipline association that I identify with. But what should we, so my question is strategic, like what we've talked about some, some of the pros and cons and limitations of measurement, but we've also, nobody's said there, they think that we should just scrap it. So what should people position where we are advocate for or do

- I think one of the most valuable things that we could do in that regard is sharing means of measurements of both the measures themselves and means of measurement. I think it is the case that many people working in this field are reinventing wheels in terms of trying to measure the, not just the outcomes, but as, as I think Joe put it, the learning opportunities inherent in what they're doing and making widely available. The quantitative and qualitative measures that others are using to measure their civic work and higher ed and sharing in a easily accessible way how they're using these measures and what they're learning from them. And having some kind of user friendly inventory of what has been useful in measuring what kind of thing would itself be, I think more helpful than trying to promote measurement. How,

- How good do you think, so both Rachel and t trig have really drawn our attention to it broadly when kind of educational opportunity or input, which is discussion across difference in trig's case, especially often provoked by texts. But anyway, maybe also Rachel, but how good do we think the measurement toolkit is? I mean, is it, or do we need to start from scratch or is it basically that people need to make sure they're, they're, they're using the good measures that are out there already?

- Well, one thing I would like to do is find a way to, well this is a complicated question to either demonstrate that students want to develop their civic character skills, they want college to be a place where they can, you know, enjoy the kinds of epistemic and just sort of civic age civically age agentic benefits that Rachel describes coming out of, you know, her work. People say, well, students aren't asking for that. Well, they might just not know they're allowed to want that from college. The other thing we might think of in as a group is how unified do we need to be around our own normative commitments and what are those baseline commitments? I personally think if students don't want that, my normative commitment is we as educators should encourage them to value that type of development as part of education, whether they want it or not, and whether employers want it or not. I think they do, but if they don't, is that, whether that's wise or not for a group like this to wise territory to enter to, to all agree on that kind of fundamental No, that's what we're here for. That's not for me to say,

- Although we do, we do actually have a statement of values that you have to, you have to approve to join. So in some ways sure, it's not a, it's not a neutral group, although it's neutral about a bunch of questions within, within that there are a lot of

- Detailed questions. Yeah. Peter, this, this is slightly come coming orthogon to your question, but I I do think of

- Course, - So there's research on the use of research and one of the things they find people who do this is that folks like us are much more likely to engage productively with research on questions we don't already have strong views on. Right? Yeah. We have a curiosity about something but aren't quite sure we're much more likely to be interested in and engage productively with research. I could imagine this kind of an organization pulling together people who share an interest in a question but are not particularly locked in on the answer.

- Yep.

- And the format I think might be more productive than some of the ways we say we gotta collect some data to show people what we already know to be true.

- Great. There might be a reason to also collect data to show people that we know should that reason for that, but, but it's not as interesting for sure. There's a couple of kind of relevant to your point, there's a couple of relevant suggestions in some ways in the chat. Victoria Sutton is talking about comparative research where you look at, for example, online offline treatments of the same topic. So that's, that's something that people, they might have guesses about, but to me at least that's an open question, what can you do with online? And then a couple different people have talked about sort of long, long, longer term processes that we wanna measure. So what happens if, if students, for example, develop foundational knowledge and that leads them to become more interested in discussions and then later they become active participants. So there could be really unanswered questions that are a little bit longer timescale. We kind of know that if you, if you teach somebody the what the Bill of Rights, they'll learn what the Bill of Rights is. As a colleague said years ago, if you teach them, they will learn. But the, the the, but there might be questions about the longer term developmental trajectories that we really don't feel we know the answer to at all.

- And I can speak a little bit about that, Peter. I think one complexity in that is that it is often in interaction with what happens to happen in students' lives in the period that follows whatever it is we're offering them. I did longitudinal research on some of my dialogues in which I followed up with students three to four years later. And I found that so much of the meaning of these civic discourse sessions was apparent only to students in retrospect because of events that happened in their lives later and for those students who had certain kinds of events happen in their lives later. So it's in interaction with students' lives and you know, that makes sense, but I can see it in detail when we do talk to them years later.

- Yep. No, I remember, I remember at least one story from your book, which is pretty powerful. Right, right, right. And there's a, so there's also a question about sort of, there's a question about accountability here. What are, what are we asking individual faculty members to be responsible for? Not for the long-term effects on the students, if that's very interactive, but very dynamic. Just wanna make sure we're hitting in a, in a about a minute, I'm gonna end this with some, with some announcements, but do we make, do we get major issues here in the chat? There are, there are some good, there are some good suggestions that I think we'll harvest those for actual activities for our, for the alliance certainly including simply continuing to provide a platform for sharing approaches to measurement and things that are measured. Yeah, so probably I should, I should wind this up. So I wanna, I wanna, first of all really thank Joe Trigg and Rachel, great people, great conversations, wise people and worth, worth reading all their work. The next episode of this webinar series is March 4th. Again, it's how universities can strengthen civic education in K 12 schools. Be moderated by Jenna, story from AI and panelists will be Jennifer McNabb, Joshua Dunn and Mira Levinson, and that'll be a great one. You can submit your own resources, commentary, research and questions to Hoover aca@stanford.edu. You can apply for membership on the ACA membership page. You can also nominate a colleague for membership because you think they should join. And that's been one of the ways that this has grown and that's really a valuable thing to do. In fact, you should take a moment to just seriously think about people who should join. And I think that's all for our, for our show.

- Peter, can I just really quickly say IIII was just going through and trying to respond to direct questions to me, just sending those people my email. Salvatore, will you send people or our contact information if they wanna reach out

- Good

- Or mine at least.

- And I, I, I wanna say one more. Good. I think it sounds like everybody's willing to be contacted, including me. Definitely. And also, I I should say on April 10th at Tufts University, the Alliance Civics in the academy is sponsoring a, gathering a quite a broad day long summit on civic education. And that's actually open to anybody and is free. So please kind of Google that and you're very welcome to come. And that's, I I will, I will make a pitch and say that, that we, I think that's the gonna be the broadest conversation about civics and higher education that I know of because it's gonna be bringing together people who do sort of community engaged work, people who do curricular work and people who do research on democracy from research centers. And I think those are three major streams of work that haven't been too much in dialogue. And also I'm just real excited by the people who are gonna be speaking because they exemplify really exciting programs from across the country for and by April in Massachusetts it might be possible to walk around, which it's not really now. So everybody should come to the Tufts. And I think with that we will wrap this webinar. Thanks everybody for joining us.

- Thank you.