Software Measurement Validation

Validating Software Measurement

Simply collecting metrics isn’t enough; we need confidence that our measurements are meaningful and our predictions based on them are accurate. Validation is the process of ensuring this confidence. It typically involves two aspects:

• Validating the measurement system (the metric itself).
• Validating prediction systems (models using metrics).

Validating Measurement Systems

This step ensures that a metric accurately represents the specific attribute it claims to measure. It’s about checking if the metric behaves consistently with our understanding of the attribute.

Formally, this involves demonstrating that the metric satisfies the ‘representation condition’ from measurement theory – meaning it preserves the qualitative relationships observed in the real world. For example:

• If we believe Program A is ‘more complex’ than Program B, a valid complexity metric m should yield m(A) > m(B).
• If we combine two independent code modules P1 and P2, a valid length metric m might be expected to satisfy m(P1 + P2) = m(P1) + m(P2) (additivity).

Validation techniques include:

• Theoretical Validation: Showing the metric conforms to mathematical properties expected of the attribute (like the additivity example for length).
• Empirical Validation (Convergent Validation): Showing that the metric correlates strongly with other accepted (and hopefully already validated) measures of the same attribute. For instance, correlating a new complexity metric with Cyclomatic Complexity or subjective developer ratings of complexity.
• Expert Judgment: Assessing whether the metric’s results align with the consensus understanding of experienced practitioners.

Example: Validating ‘Lines of Code’ (LOC) as a measure of ‘Length’. We can theoretically argue it satisfies basic ordering (longer programs have higher LOC). Empirically, we might find it correlates well with other length measures like character count. However, its validity as a measure of ‘Effort’ or ‘Complexity’ is much weaker and requires separate, more rigorous validation, often showing poor correlation.

Validating Prediction Systems

Prediction systems use mathematical models (often incorporating validated metrics) to estimate future outcomes, such as project effort, cost, schedule, defect counts, or system reliability.

Validation here means establishing the accuracy of the model’s predictions in a specific context. This is primarily an empirical process:

• Collect historical data from past projects within the target environment (e.g., data on project size metrics, actual effort, actual defects).
• Apply the prediction model to the input data from these past projects.
• Compare the model’s predictions against the actual known outcomes.
• Use statistical techniques (e.g., Mean Magnitude of Relative Error - MMRE, Prediction Level - PRED(l)) to quantify the prediction accuracy.

Key considerations:

• Context Dependency: Prediction models are often highly sensitive to the environment (team skills, process maturity, technology stack). A model validated in one context may not be accurate in another.
• Data Quality: The accuracy of validation depends heavily on the quality and relevance of the historical data used.
• Stochastic vs. Deterministic: Some predictions are inherently uncertain (stochastic), like defect arrival rates. Validation assesses if the predictions fall within an acceptable range of probability, rather than expecting exact matches.
• Model Calibration: Often, validation leads to calibrating or tuning the prediction model parameters to better fit the local context.

Example: Validating a COCOMO-based effort estimation model. We would gather data from past projects (size in KLOC or Function Points, actual effort). We’d run the model using the size inputs and compare its effort predictions to the actual effort recorded for those projects. If the average error is acceptably low for our purposes, we might consider the model validated for use on similar future projects in our organization.

Validation is not a one-time activity. As environments change, metrics and prediction models should be periodically re-validated to ensure their continued relevance and accuracy.