F5 certification bridges the gap between Networking and Advanced Application Layer Stack. It takes about 8-12 months to develop a test. I was fortunate to be part of the Item Development Workshop (IDW) for F5 201v2 exam and wanted to share some of the information I learned during the IDW.
Key Development Concepts utilized during the IDW:
Reliability: Consistent and precise questions.
Fairness: Does not put any group under disadvantage.
Validity: Accurately and appropriately measures what is relevant.
Reliability is not just related to the individual items but the exam as a whole. In essence, reliability of an exam is measured by the consistency of an individual’s score over multiple attempts, assuming the individual’s ability hasn’t changed substantially over the many attempts.
Validity is similarly not just about an item but the overall exam. Validity is how well the proposed purpose of the exam meets the outcomes of the exam.
Minimal Competency:
Minimally Qualified Candidate (MQC) is someone who meets the minimum requirements defined by the syllabus. A rough definition of MQC is that of an MQC Lawyer who may or may not have the skills to become a Supreme Court Judge but society is comfortable with them practicing law as the MQC Lawyer satisfies widely accepted qualification standards.
Cognitive Complexity and Difficulty:
The difference between low and high difficulty is knowing pi is 3.14 or 3.1415926535. Cognitive Complexity is using pi to find the area of circle with specific radius. Based on the blueprint of the exam, up to the 3xx level exams, only Remember and U/A were used extensively in the topics. A/E shows up more in the 4xx level exam. The difference in cognitive complexity for multiple topics are provided in the blueprint of the exam.
Cognitive Complexity:
- Remember (R)
- Understand / Apply (U/A)
- Analyze / Evaluate (A/E)
- Create (C)
Remember:
The Remember cognitive complexity generally tests rote memorization and information retrieval. There is a general preference against Remember (R) questions. So, instead of asking a question to list the TCP flags, a question that requires understanding of TCP flags in order to answer the question is preferred.
Understand/Apply:
U/A is utilized to test application of concepts within standard operations. U/A requires an understanding of processes and the ability to pick the right process to solve a problem while being able to compare multiple processes.
Analyze/Evaluate:
A/E tests the ability integrate new information with existing information to provide answers. Diagnose a problem and understand the relationship between the concepts and how one concept influences other concepts.
Create:
Create (C) tests the ability to create new products/solutions by utilizing new or existing concepts.
Items:
Items are the questions and possible options that show up in the exam. An item consists of a Stem and Options. Stem is a combination of Problem statement and Question Statement. Options can be Distractor options or Key option(s). The Key option(s) is the right answer in the item.
Remember requires only question statement. Problem statement is required for U/A, A/E and C. An ideal MQC should be able to determine the Key without having to read the options. This is one of the reasons why time is an important aspect to differentiate the competence of an exam taker. An ideal MQC should know the answer without looking at the options. Others may have to check all the options which means they will end up spending more time per question and may run short on time.
The Stems were constructed with positive words (Positive Construction). Almost all the stems eschew negative words like NOT, NEVER, EXCEPT that could potentially lead to a wrong answer as the candidate may miss the key negative word while reading the Stem.
Each item is intended to focus on a single trait that is being tested instead of multiple traits as much as possible. A trait is a subtopic that is utilized within the blueprint. Higher complexity question could have multiple traits utilized. The item is intended to be congruent to cognitive complexity & content identified in the exam blueprint without introducing any irrelevant variance that is not required to answer the question.
Response options (Distractor options and Key options) should be similar in terms of length and logic in order to prevent the option from being an obvious wrong/right answer.