What is an Implementation Study?
The goal of an implementation pilot is to establish concrete guidelines for how an edtech tool should be used by educators and learners in an authentic context that creates and sustains long-term, quality engagement with the tool. Through this process, we’re trying to ensure that educators and learners are using the tool exactly as it’s meant to be used, so that it's not purchased then sitting on the proverbial shelf.
Who Are Implementation Studies For?
Implementation studies are ideally for mid- to later-stage companies. The definition of stage here is moreso defined by number of users, number of district licenses, and years active; it’s less defined by ARR or funding stage.
So, if you’re a large company rolling out a new product or feature, implementation may not be the best choice, and, conversely, if you’re a smaller organization with a group of dedicated users and want to really understand best practices for product engagement before you expand, then implementation is the right choice.
Concretely, the product should have a relatively sizable user-base (~20+ districts). Additionally, the edtech company should have evidence (through usability testing) that the product is easy to use and free of all technical glitches and have evidence (through feasibility studies) of successful use cases in and with the target context.
Why Conduct Implementation Studies?
Increase the Usage of Your Edtech Tool
Most edtech entrepreneurs didn’t decide to get into this very difficult industry to be satisfied by making sales. They care more that their product is being used by teachers and actually making a difference in the classroom for students. The hard truth, however, is that a troubling percentage of edtech tools are purchased by the district then never activated or are underutilized.
According to LearnPlatform, on average, school districts access over 1,400 edtech tools every month. Additionally, according to the State of Edtech report by SETDA, 57% of respondents report that they have a lot of edtech programs/products but they aren't always used effectively.
The aim of an implementation study is to understand what is needed in the way of training, classroom implementation, and integration in order to maintain high levels of quality engagement with the product over the course of a semester and year-over-year. An implementation study can give edtech companies clear, evidence-based guidelines for classroom best practices.
Demonstrate Evidence to School Districts
The wild west of edtech is over. Gone are the days when a flashy UI and a compelling sales pitch will land an edtech company large contracts with districts. Teachers and administrators are more savvy about research than ever before, and they will demand it before they purchase a product.
The widespread transition to remote learning in 2020 caused record levels of edtech investment along with a record number of edtech tools being used every day. This growth in purchases of edtech tools was also aided by the government monies through ESSER funds; however, as the dust settles and those funds run out, schools are realizing that they are under contract for a massive number of edtech tools that teachers and students don’t use.
Implementation studies are a way to show a school district that you’ve done the hard work to find out why your edtech tool might have been underutilized, you fixed it, and developed a set of research-backed implementation best practices to ensure that once the tool is purchased it won’t collect dust. If the research is done through a third-party, it only increases the credibility of your claims.
Prepare for Efficacy Research
Efficacy studies are long and expensive. If you’ve prepared for it, they’re worth every second and every penny. If you aren’t prepared, they can be a huge waste of time and resources. If you truly want to understand how your product is impacting the outcomes that you care about, the outcomes in the very last column of your logic model, then implementation research is an essential step on that path.
Implementation studies help you get to a point where your tool is being used with fidelity no matter the context. This may necessitate different implementation practices for different contexts. Perhaps, large class sizes have different implementation practices than small class sizes; however, within those specific contexts, teachers are using the tool in the same way and students are having the same experience.
Once you establish that type of fidelity, efficacy research becomes a much more reasonable use of time and money, as it controls for other possible variables that can impact outcomes and ensures a sufficient amount of usage has been reached to actually impact outcomes.
At that point you can get “good” results from an efficacy study. “Good,” meaning that you have ANY results and don’t end up at the end of the study with a null outcome because there’s too much variation in usage across different contexts.
Guiding Questions for an Implementation Study
Specific research questions can vary widely from one implementation study to another, but there are two broad themes common to all of them.
What are the most effective techniques to implement the product into the classroom?
This question is focused on understanding how teachers, administrators, and students actually use the product. We’re trying to uncover what successful implementation looks like and how those educators that are most successful recommend using the product. Is the tool best used for whole group instruction or should it be used in small groups? Is it better as a supplemental tool, if so, should it be used as homework or maybe three times a week in class for 30 minutes? Or could it be used as a curriculum substitute?
How do contextual factors influence implementation success or failure?
Contextual factors encompass the facets of a school or learner population that influence its success or failure. This is meant to uncover best practices for different types of learning environments and school characteristics. This could encompass demographic information, school size, class size, or geographic locality, among other factors.
How to Prepare for Implementation Pilot?
Conduct usability testing
The first way to prepare for implementation research is to conduct usability testing on your product. Testing the usability of your product is about understanding if your edtech tool is fully functional and people enjoy the user experience. Compared to implementation research, usability testing drills down into more specific feature sets of the product and may include task analyses or user interviews. Conducting usability testing before implementation research ensures that there are no blatant issues with the functionality of your product that may hinder a successful implementation study.
Integrate your edtech tool
Second, it’s imperative that your tool is interoperable with the variety of tech stacks that school districts use. Basically, make sure that your tool is integrated with other tools like Clever and Google classroom so that there are no unforeseen obstacles when districts try to integrate your tool with their inveterate technology. These integrations should feel seamless to the user because that’s, most often, their first impression of your product.
Develop implementation guidelines
Finally, before conducting implementation research, you need to develop testable implementation guidelines. This is your preliminary set of best practices that the implementation study will test. The implementation guidelines are how you expect the person interacting with your product to use it. If you don’t already have an idea of how you want teachers, students, or administrators to use the product, then feasibility research can help develop those guidelines.
Implementation research is an essential part of the research journey. It can open up opportunities for large-scale efficacy studies and help get your product in schools. Most importantly, though, it’s a way to ensure that no matter the context, your product is being used in a quality way with fidelity by teachers and students and not just another piece of tech in a school district’s stack.
To learn if implementation research is appropriate for your company, visit our webpage and fill out an intake form.