Test page

From Sigma

Jump to: navigation, search

Simulation validation is an important process for a simulation model to gain credibility as an accurate, working model. There are several methods that Bio G uses to validate their models.

Contents

Model Verification

TO EDIT: Model double checking, basically. Include "quick and dirty" cycle time calculations.

Model Validation

External model validation involves (include notes from Sigma 131 reader).

Schruben's Protocol

Schruben's Protocol, (also known as a Simulation Turing Test) is a procedure that exposes shortcomings in a simulation model as it seeks to establish a model’s credibility with its users. It was developed by Professor Lee Schruben in 1980 [citation needed] as means to ensure model accuracy of simulated systems.

Schruben called this a "Simulation Turing Test” after a conceptual test, suggested by Professor Alan Turing in the 1950s, which might be used to tell machines from humans [citation needed]. However, Schruben’s Protocol has the objective of improving a simulation’s validity with direct user input. It relies on the judgments of a team of subject matter experts (SME’s) who are challenged to distinguish simulated system performance from actual system performance. These SME's are typically the potential users of the simulation model.


The protocol as applied at Genentech is as follows:

  • SME’s are told the source of the real performance data and the details of the campaign during this time (Jan 2007, Herceptin running at 4.7 runs per week).
  • Users are told that the test is of the simulation, not of their own knowledge.
  • SME’s are presented with between 10 and 15 sample outputs in the exact format the plant uses for the output, and asked to say whether the output is real (i.e. actual plant historical data) or simulated (a simulation of Herceptin at 4.7 runs per week).
  • SME’s write their answers, without conferring, on a standardized answer template including a two sentence reason for their answer.
  • At the end, the answers are compared in a group setting. Notes of further group reasons for answers are taken by the team.
  • The real answer is revealed and an additional chance for feedback is given.
  • Written answers are collected and collated and the model corrected and enriched as suggested by the panel of SME's.
  • The protocol is repeated as necessary to establish a trustworthy, and trusted, model.

References

Sigma Reader IEOR 131 Schruben, L.W., “Establishing the Credibility of Simulation,” //Simulation//, 34 (3), (1980), 101-105. Turing, Alan (October 1950), "Computing machinery and intelligence", Mind LIX (236): 433-460.

Personal tools