NAFI Pegs Lower Test Scores on Question Bank Changes
March 10, 2011 — An unusually high number of pilot-applicant failures in recent weeks was due to apparent changes to airman knowledge test-bank questions made without notifying industry stakeholders, and the National Association of Flight Instructors (NAFI) has asked the FAA to clarify or rescind them.
“We learned last week that the FAA’s Airman Testing Standards Branch recently implemented changes to the banks of questions the agency uses to compile knowledge tests for pilot candidates,” NAFI Executive Director Jason Blair said. “These include the airline transport pilot, flight engineer, and fundamentals of instruction tests - and possibly more - and they involved significant revisions to test-bank questions.”
NAFI and AOPA submitted a letter to the FAA to request that “the knowledge test banks be reverted back to the questions in place prior to the recent update (and) that students who failed the exam since the changeover be allowed to retest and have the initial failure expunged from their record.” The associations also asked for a more orderly, transparent transition to new test-bank questions over a short period of time, giving them the opportunity to notify their members that such modifications were being made. That would allow training providers and students to adapt learning processes and avoid surprising applicants with unexpected test material, Blair said.
While the actual test questions have not been public for nearly seven years, commercial companies produce study guides with historically accurate representations of test questions that pilot applicants can use to study for the exams, Blair said. Test-prep vendors also provide sample exams that allow pilot applicants to practice taking the knowledge exams - not only to reduce pre-test nervousness, but also to gauge a close approximation of their final knowledge-test scores.
However, NAFI learned that, because of the nature of changes that have apparently been made to the test banks, in some tests, the rate of failure on knowledge tests has increased significantly over the past month. For example, some university flight programs have indicated that they’re experiencing diverging results between their own practice tests and results on the actual instrument pilot knowledge tests their students are taking.
“We fully support the FAA’s efforts to improve the quality of the knowledge tests,” Blair said. “It’s important that applicants aren’t simply studying ‘sample test’ questions at the rote memorization level to pass a knowledge test, and we agree that a focus on learning material and skills to the correlative level is the best method to teach applicants to be safe, knowledgeable pilots. However, we’re concerned that the test changes were made without any notification to the industry.”
That hasn’t provided stakeholders with the opportunity to promote the testing changes, which eliminates any opportunity for applicants and instructors to adjust their study and teaching strategies to accommodate the new content, Blair said. The result isn’t an improvement in teaching or learning efforts. Instead, it represents a wasted testing experience that results in unrecoverable costs for applicants who had prepared for a test that was not what they expected.
NAFI is working with the FAA and other aviation organizations to determine the scope of the changes and how they will affect applicants’ preparation for knowledge tests. “We strongly advise our members and other flight instructors to be aware that that the test-bank questions for some knowledge tests may have changed,” Blair said. “Instructors and students should also remember that students’ primary preparation for a knowledge test should focus on knowledge, not memorization of answers from sample test banks. Yet, while we support the FAA’s efforts to improve the rigor of the testing process, and we’re sincerely interested in promoting the process, and to do this effectively, we would like the FAA to include industry partners such as NAFI, AOPA, and other flight training stakeholders in efforts to spread the message about upcoming changes.”