How effective are your ABA therapy sessions? A question commonly asked during self-reflection, supervision, and parent feedback. However, how can you accurately measure the effectiveness of your interventions if they aren’t being implemented correctly?
Treatment integrity, otherwise known as clinical integrity, refers to the degree to which your interventions are delivered precisely as designed. In addition to ensuring your hard thought-out plans come to fruition, this is crucial for ensuring ethical practice, maximizing client outcomes, and upholding the reputation of ABA as a whole.
Based on one of our highly acclaimed continuing education (CE) webinars, this article dives into the nuances of treatment integrity in ABA, including practical examples of how technology is being leveraged to improve it.
Treatment Integrity Explained: Why?
During the aforementioned webinar, we asked the audience about the roles for which they are currently monitoring clinical integrity. We discovered that 41% of the participants had tools in place to measure clinical integrity for RBTs, while 26% have similar tools for BCBAs. While this suggests that organizations are adopting these practices, 15% of attendees reported not having any formal tools in place. This highlights the ongoing need for additional resources surrounding clinical integrity.
The BACB emphasizes the importance of integrity in its Core Principles and Code of Ethics. Core Principle 3 requires behavior analysts to behave with integrity, holding themselves and their supervisees accountable for accurate implementation.
Think about it: if your carefully crafted behavior plans aren’t being implemented as intended, how can you truly measure their effectiveness?
When you introduce a new skill acquisition program or behavior plan, it’s essential to monitor its implementation closely. This includes observing your trainees and supervisees to ensure they are carrying out the plan accurately. By doing so, you can be confident that the collected data truly reflects the effectiveness of your interventions.
Imagine a scenario where a behavior plan calls for specific reinforcement procedures, but they’re not being implemented consistently. Or perhaps the plan specifies avoiding eye contact during a particular behavior, yet the RBT inadvertently makes eye contact. These seemingly minor discrepancies can significantly impact the outcome. That’s why it’s not enough to simply verify initial implementation, you must maintain ongoing monitoring and conduct regular integrity checks.
Measuring Treatment Integrity: What and how?
So, what specific areas should you be monitoring to determine treatment integrity? Keep in mind that these lists are not exhaustive but highlight common areas for measurement.
Measuring RBT integrity
When it comes to measuring RBT integrity, the devil is in the details. It’s not enough to simply observe your team; you must focus on specific areas that directly impact the quality of your ABA services.
- Initial Training Competency Regardless of the RBT’s experience level, each company and BCBA does things slightly differently, leading to varying skillsets. Starting with an initial training competency allows you to document that you have observed the RBT demonstrating the specific skills necessary to perform your programs effectively. This can evolve into a maintenance competency to ensure those skills are retained over time, similar to what you would do with your clients.
- Data Collection Integrity Accurate data collection is the cornerstone of measuring fidelity and effectiveness. Is what you would count as an independent trial the same as what your RBT would? A strong interobserver agreement (IOA) with your RBT is essential to validate the accuracy of your data. If there’s a discrepancy between your data collection and your RBT’s, it may indicate a need for additional training or clarification of program instructions.
- Program Implementation Are your team members following your program instructions to the letter? This includes everything from specific reinforcement procedures to prompt hierarchies. Regular observations and feedback can help identify any areas where implementation may be inconsistent.
Additional impacts to integrity:
While these areas might not be direct measures of treatment integrity, they still impact your supervisees’ ability to carry out your programs as designed.
- Pairing and Reinforcement Pairing yourself as a reinforcer is a fundamental skill for any ABA professional. Are your team members effectively building rapport with clients and using positive reinforcement strategies? Observing their interactions with clients can provide valuable insights into their pairing and reinforcement techniques.
- Session Notes Session notes are crucial documentation for tracking progress, serving as a reference for the treatment fidelity of past sessions and guiding the needs of future sessions. Ensure your team members are writing clear, concise, and accurate session notes.
BCBA self-measurement
As a BCBA, it’s not just your RBTs whose work needs to be monitored. By measuring your own performance, you can ensure that you’re delivering high-quality, ethical, and effective services, consistent with your plans.
One crucial area to consider is supervision quality. Are you using behavior skills training effectively when training on new programs or behavior plans? Are you providing an adequate number of supervision hours, both in terms of quantity and quality? BACB Code of Ethics Item 4.08 speaks to the importance of performance monitoring and feedback in improving ABA practice. Meaning, if you want your supervisees to perform effectively, ensure you’re making the most of your supervision time by providing meaningful feedback and guidance.
Another important aspect is program instruction. When you create a new skill acquisition program, are your instructions clear and concise enough for a new or substitute RBT to come in and quickly follow? Are you including all the necessary components, such as clear SDs, desired responses, error correction procedures, data collection methods, and mastery criteria?
Additionally, consider your data-driven decision-making. Are you consistently reviewing client data to identify trends and make informed decisions about program modifications? If you notice that progress is not being made over a period of time, do you conduct a program checklist on the RBT? Are you making modifications to the program, trying different methods, or implementing new prompting strategies and reinforcers? Or is the program becoming stale and stagnant while no progress is achieved? It’s important to determine if the issue could be with the program itself rather than how it’s being implemented.
Finally, ensure that your treatment plans are comprehensive, individualized, and compliant with insurance requirements. As you know, insurance payors have specific guidelines, and securing approval for the hours you’re requesting can be difficult. Creating a treatment plan checklist tailored to the requirements of each insurance payor can help ensure that your treatment plans are being accepted by those payors.
Also, avoid submitting the same treatment plan under different names for multiple clients (even if their cases are similar). Each plan should be individualized for the needs of each client, in order to be effective and to meet the requirements of the insurance payor.
Remember, this is a sample list of high-level areas you can monitor for your own clinical integrity; however, feel free to personalize these to the needs of your own practice.
Treatment Integrity Checklists
Now that you know what to measure, try to create a system to ensure consistent monitoring. A treatment integrity checklist can be a valuable tool in this process. Here’s a breakdown of how to create and implement one.
- Develop a Collaborative Checklist: Make sure RBTs, BCBAs, and BCaBAs are all involved in the creation of the checklist. This collaborative approach fosters buy-in and ensures that the checklist aligns with your team’s specific needs and practices.
- Clear Expectations and Training: Before conducting an observation, review the checklist with the clinician. This ensures transparency and sets clear expectations for what will be observed and evaluated. Provide comprehensive training on all areas of the checklist, including hands-on practice, resources, and opportunities for feedback.
- Targeted Observations: During observations, focus on specific areas of the checklist that are relevant to the current session. If certain skills or behaviors are not present, identify opportunities for future observations or role-play scenarios.
- Leveraging Electronic Data Collection: Utilize your electronic data collection system to review non-observational items like program quality and interobserver agreement (IOA). This can be a time-efficient way to monitor treatment integrity without requiring in-person observations.
The following examples illustrate items that can be included in various checklists. Please note that these are not meant to be comprehensive checklists and should not be used as such. Additionally, keep in mind that a wide range of checklists can be created based on specific needs, such as behavior plans, skills, programs, or levels.
Leveraging Technology: Where?
Technology can be an effective tool to measure and improve integrity without adding significant work for you and your team. As a matter of fact, during our Clinical Integrity CE webinar, we asked attendees how they collect data today and found that over 70% use some sort of technology, whether that be data collection software or Microsoft Excel. However, the poll also revealed that over 20% of attendees were still using pen & paper to collect and analyze their data.
While you can create and complete checklists with paper and pen, technology allows you and your team to save time by streamlining communication and documentation in one central location. This approach reduces errors and minimizes the risk of lost files, while allowing you to easily visualize data by automating graphs and reports.
For example, you can see in the image below that Catalyst’s ABA data collection software includes a form builder that allows you to easily create custom forms and enter responses. That data is then stored in the system for immediate analysis by any team member with access.
Additionally, you can easily create custom programs, such as the skill acquisition example below. The template format ensures that all required information is included when building a program.
Takeaway
By carefully measuring and monitoring key aspects of your team’s performance, you can identify areas for improvement and make data-driven decisions to enhance outcomes for your clients, with technology serving as a powerful ally in this process.
To learn more about using technology in clinical practice, watch our free CEU webinar “Measuring Clinical Integrity through Electronic Data Collection”, hosted by Lainey Posthumus, M,Ed., BCBA.