Our State Has an SLDS Now What? Bridge Event Webinar Tuesday, February 25, 2014, 2:00 3:30 p.m. (Central Time)

Transcription

1 Bridge Event Webinar Tuesday, February 25, 2014, 2:00 3:30 p.m. (Central Time) REL Southwest Facilitators: Vicki Dimock, PhD, REL Southwest Director, SEDL Jackie Burniske, REL Southwest Dissemination and Collaboration Lead, SEDL Presenters: Nancy J. Smith, PhD, Principal Consultant, DataSmith Solutions, LLC, Austin, Texas Bethann H. Canada, Educational Information Management Director, Virginia Department of Education John Hughes, PhD, REL Southeast Deputy Director, Tallahassee, Florida To view a video of this archived webinar: Transcript: Presentation by Dr. Nancy J. Smith Dr. Vicki Dimock: Slide 2: Okay, first of all I want to introduce to you Dr. Nancy Smith. She is the principal consultant with DataSmith Solutions in Austin, Texas, and she has a great deal of experience in providing technical assistance to numerous states and organizations on issues around data governance and improving the development and use of statewide longitudinal data systems. Dr. Smith was the director of the Statewide Longitudinal Data System Grant Program at the U.S. Department of Education, and she s a cofounder and deputy director of the Data Quality Campaign, so she has a lot of experience and background around longitudinal data systems. She s also written a number of white papers on research capacity, adjusted cohort graduation rates, federal data collections, and the impact of statewide longitudinal data systems. She began her career in data governance while she was a systems analyst at the Texas Education Agency, and she earned her PhD from The University of Texas in Austin. I guess I better not say Hook em Horns, right? So we welcome Dr. Smith, and I m going to turn the webinar over to her now. Thank you.

2 Dr. Nancy Smith: Slide 3: Thanks, Vicki, and thanks to all of you for participating. We re up to 120 folks now online. That s a fabulous turnout, and I apologize in advance if my voice crackles but we ll just keep going, slogging along. And just a reminder, we are going to do a Q&A and discussion session after the presenters, and if things pop up for you, thoughts, questions, please type them into the chat pod at the bottom of the screen. We are monitoring those and we ll select some of those questions to discuss. So onward and upward. So you have an SLDS. I know a lot of you, especially those of you involved in the state education agencies, know that we ve spent years building and designing and talking about and starting to collect the data and connect longitudinal data. But now what? It s not enough to build the system and collect the data. If it was, we didn t need to go to student-level data systems. We were already collecting tons of data before that. The real purpose of this is to figure out the various and often many ways that we can use this longitudinal data to help inform policymakers and practitioners, ultimately to improve student outcomes. But what do we mean by data use, data sharing, and how can we support all of that? What type of staff and resources does it take? What skill sets does it take to support robust data use, data sharing agreements? As I ve worked through the years with various states in various capacities, I hear a lot of the same questions and conversations going on. (There we go. We re trying to coordinate the slides here.) Anyway, so I hear many of the same questions and conversations going on, and I ve identified five kind of overarching questions that I want us to think about during today s discussion. These address a lot of the issues that I hear as I work with different states. The first one, what data uses need to be supported? And I would say that s both internally to SEAs and LEAs and externally with researchers, with other policy advocacy organizations. What are valid uses of the data in the SLDS? And while I say the SLDS, we know that that mostly holds administrative data. We all know that there are a lot of other data sources that state departments of education manage and operate and store the data about. So it s really the bigger picture. For me an SLDS is the whole 2

3 system of data collections. So what are valid uses? Just because you have the data it doesn t mean it s appropriate to use it in any old way. Next, what are the best processes for managing data requests from outside researchers and organizations? As you all know, you re living it, there are limited staff and resources available to you, so any processes that are developed need to be very, very efficient. You need to figure out which staff to do what and how to be the most efficient. But you really do need to come up with really good, sound standardized processes, otherwise it can be haphazard in who responds to what question or what data request. You don t know if you re using the same data source for the similar questions, all of that. You would just need to come up with good processes for managing data requests and data sharing. The third one, how do you guide data use, research, and analytics to improve education outcomes? Along these lines some of the discussions have been: What are the priority analyses for SEAs and LEAs to consider? What types of information do policymakers need? How do you set your priority research agenda if you have one? How do you figure out which data requests to respond to with your limited resource? The fourth question, how do you prioritize competing data policies and needs? There are many more questions and desires and needs and requests than you really probably have the staff or the time to answer. So how do you prioritize what gets attention and what gets responded to? Some of the requesters you re always going to have the governor and the legislature, and other state policymakers, with information requests. You ve got federal- and state-mandated reporting, which are no small activities. You ve got priority research information needs, hopefully, and statewide reports that may or may not be mandated, but you do historically. And now that we ve advertised the SLDSs we ve got a lot of other folks requesting the data. So internally to the SEAs, how do you manage all those requests, how do you decide what to respond to? And then externally, how do you communicate so that external research organizations can help monitor themselves and what they ask for and what timeline they re looking at? 3

4 Then finally, I don t know the answer to this one but I ll just pose the question. How do you sustain funding and other support for the system? How do we get state line item budgets to support this, or can we? What program-area funding can support this? What do you charge for your data sharing, the data sets that you share? Are there license fees? I don t know, but I do want to pose that because I know it s something all of you are having to deal with. And it s a goal of this webinar, especially during the discussion session, for us all to help each other think through and learn from each other on some of these issues. Slide 4: I just want to highlight a couple of resources. A few years ago, back in 2010ish, the U.S. Department of Ed and the Forum developed a series called Traveling Through Time, all about SLDSs. There are four books involved, and I think they re really pretty timeless, especially the third volume, which covers data governance, improving data quality and, certainly very important, how to secure data and protect the individual. There is a lot of very good information that I don t think will ever go away. I think we re always going to have to stay on top of how we can improve our processes to improve data quality, how to keep our data governance up to date, and privacy and security and confidentiality will always be there. Slide 5: So I think that s a great resource as well as the fourth book, Advanced SLDS Usage. This has a lot of good information, including four case studies on how to use data, how different states in the district have used data. I hope that these serve as good, thought-provoking case studies as you tackle, both from the research side and the SEA and LEA side, how you re going to use the data. Keep in mind what data is in the SLDS, predominantly administrative data. But how can we use it, how can we partner with folks to get the data analyzed and reported? This is why we asked Bethann to be our co-presenter on this, because she s been very creative in how she s tackled some resource limitations in developing partnerships in data use and trying to figure out how to help policymakers and educators. So I think these are good, timeless resources for you all to reference and they can be found on the NCES website. We ve got the link there. 4

5 Slide 6: The next couple of resources I want to highlight, this Six Key Uses of Longitudinal Data, a little bit of self-promotion but that s not really why it s up here. It s just a two-page handout. You can download it. Lynn and I and Chris Dougherty worked on this back in the day, early DQC days when we were all at NCEA. We had done a lot of research and analysis with longitudinal data because we had access to it in Texas and this highlights different ways of looking at data, four different audiences using different analyses. These types of uses involve both state and local level, internal/external uses. Maybe you can analyze things at a campus/district/state level. You can probably combine state and local data. I think progress monitoring you get monitoring the progress of students or cohorts of students along time. Diagnosis and prescription how can the data help you figure out how to intervene? Internal benchmarking what areas of success do we have within a school or district in certain areas? External benchmarking this really gets a lot at what we re doing here and a lot of the webinars that I ve been involved in the past few years how to identify best practices and adapt strategies that others are using to help improve our processes. Predictive analyses we ve got our early warning indicator systems where we try to predict dropouts or the likelihood of graduates being ready for college and career. Then evaluation we can evaluate programs, evaluate policies, this usually involves more complicated statistical analyses, beforeand-after analyses, comparison groups, more statistical modeling. Depending upon what you want to do, you can see that the types of analyses, the type of skill set, the type of data you need will be different, and all of these are very valid and necessary uses. It s not easy to manage all of them, but we ve got to figure this out. We ve got to start figuring out how we re going to do that. Slide 7: The next resource that I want to highlight is Leveraging the Power of Longitudinal Data Systems, the DQC publication, and in it we highlighted another way of looking at five different uses of data. We ve got analytics, business intelligence, data mining, evaluation, and research. These kind of run the gamut from just simple descriptive statistics, snapshot reports, or 5

6 there may be what looks like a snapshot report based on a whole cohort of students, all the way to evaluation and research which might involve randomized control trials or comparison group analyses. Why I bring this up is, depending upon which types of use you re focused on, you really need to pay attention to having the right data and the right format and the right skill set for the analyst to validly and reliably analyze the data in these ways, produce the reports that you re looking for. That s one of my key things that those of you who know me know I harp on is, do you have the right skill set of analysts to do the different types of analyses you re looking at? A business analyst or business intelligence analyst can do a whole lot, but are they trained for a more sophisticated, complicated randomized control trial study? We need to make sure that we have the right people and frankly SEAs, LEAs, you have limited money for staff and so you might have to partner with external researchers or others to get these analyses done. That s universities, the Regional Educational Lab, external organizations these are the folks who have a lot of the statistician type folks on hand and there are ways to partner to get a winwin situation. So that s enough from me and my soapbox. I know those of you who know me, you ve heard this before. But I do think that these are useful tools and I really do think that we need to pay attention to how is the data going to be used, and do we have the data in the right format that allows the proper analyses, and who is best trained to do those analyses? Continued: See next transcript 6