Scorpion is a secure exam development tool that opens up new ways for project managers to collaborate with subject matter experts at every stage of exam development. This document provides an overview of its core features and how they are used.
Caveon, the industry leader in test security, developed Scorpion first and foremost as a secure environment for developing exam items. Project managers have complete control over who can access their exam content and which tasks users are authorized to perform. When a task is complete, the project manager can remove access to all users who are no longer active in the project. Additional security features include (but are not limited to) multi-factor authentication, password strength grade, and end-to-end SSL encryption for data in transit.
Scorpion accommodates users of all skill levels. Its straight-forward, intuitive user interface allows users of any experience level to focus solely on performing their tasks, not on learning a new tool.
Users with Scorpion, managers can invite subject matter experts to their projects, providing a secure and intuitively-designed environment for these users to perform their assigned tasks. Because these subject matter experts are often engaging in test development for the first time, Scorpion’s flexible, built-in communication and workflow mechanisms allow project managers to easily guide these users through the process of each unique project.
Due to the ease of being able to import and export data at any step in the process, Scorpion is an excellent companion to whatever item banking and delivery platforms you currently use.
Project managers can use Scorpion’s built-in project tracking features, including visible progress markers (phases and levels, target counts, number short, and more – all on a single page). Pie charts show progress for item reviewers, and number counts are provided at each stage.
Exam Developer Access
Scorpion can be accessed at the following website: https://scorpion.caveon.com. Once an account is created, a user is able to create a new project. To access a pre-existing project, a user must be invited to the project by an assigned project manager for the exam.
The settings tab allows project managers to set project customizations. It is here that project managers can establish project settings and proficiency levels, archive projects, and invite additional project managers.
Scorpion makes blueprinting – an often arduous task – fast and easy. Data is either manually entered or imported into the system. Project managers then invite participants to take part in the fully-customizable blueprint review. Once all data is collected, the ratings can easily be exported from Scorpion into a different tool.
Blueprinting can be performed at the competency and objective levels, and the system provides for automatic blueprint creation using even splits or item counts. Blueprint content can also be uploaded into Scorpion or exported. Objectives and competencies can be entered and edited singly or in bulk.
Target Numbers per Form
Form target numbers are easily set with a simple pop-up where the project manager can set form counts, items per form, and items to share across forms.
Item Creation and Storage
Scorpion supports many item types, including Discrete Option Multiple Choice (DOMC) and Caveon SmartItems. The item writer chooses an item type from a drop-down menu or uses the default multiple choice item type, and the system provides the appropriate item template. Any form can have multiple item types, and types can be changed even after an item is written.
- Multiple Choice
- Build List
- Short Answer
- Hot Spot
Scorpion offers two different methods to track your items throughout the development process: Phases and Status
Scorpion offers phases that indicate at which stage of the exam development process each item is. Project managers can fully customize these phases for any project. Additionally, the phases can be created not only before item development, but during, providing agility for project managers.
Phases can be designated as “locked” to prevent items in locked phases from being edited.
Past Versions of Items
Scorpion saves a copy of an item at each stage in the item development process, when a phase is changed. If a writer or administrator wants to see how the item has changed over time or consider whether an earlier version of an item is better, a simple click displays each version. The system makes it easy to roll back to an earlier version if the user decides to do so.
For ease of communication between the item development team and SMEs, Scorpion offers Statuses. At each phase, a color-coded status indicates if the items are ready for the next phase (green), in progress (default / gray), on hold (red), or rejected (dark red). Indicating item status is accomplished by using a simple drop-down menu.
Subject Matter Expert (SME) Access and Limitations
During item writing SMEs are able to indicate the status of an item, but only project managers have the ability to move an item to a new or different phase. At the end of the writing and/or review phases, SME access is easily removed in the interests of exam security.
Storage/ Archive Projects
Once all exam development activities are complete (blueprint, item writing, reviews, cut score setting, forms build, and publication), item banks, review information, and forms build data stay securely housed in Scorpion.
All backups and server snapshots are stored on Amazon Glacier and are encrypted using 256-bit symmetric keys. In addition, standard precautions are in place to prevent cross site scripting and cross site request forgery attacks.
After item development and reviews are complete, the project manager can use the psychometrics section in the dashboard to build final examination forms. The psychometrics section of each project provides access to a wealth of information, including:
- Modified Angoff, or other review conducted in Scorpion
- Pvalue if the exam is delivered in Caveon SEI
- Item difficulty across forms created, based on difficulty data selected
- Standard deviation from the mean
- Form counts, items per form, items shared across forms, enemies and more.
Modifications to a form, if an item is revised or a bank undergoes a new review after changes, are easily made with a click of a button. Items can be moved between forms, and forms can be split with little effort by the project manager.
All findings can be exported as an Excel report at anytime throughout the form creation process.
Reviews and Surveys
Reviews and surveying are simplified using the Scorpion tool. You can choose to manually enter or import your data and, once set up, invite participants to take part in a fully customizable review. The review has a straightforward interface, allowing users of all skill levels to participate. Once all data has been collected, the ratings can easily be exported from the system into your preferred tool.
Types of Reviews:
- Blueprint review
- Item review
- Preset and custom questions available with a variety of response options:
- Sharing reviewer comments
- Standard Setting