Reliability Growth — theory

Finding and fixing defects during flight test. Based on C. P. Hoult, Reliability Growth During Flight Test, 2013, which reprises W. Schaechter and C. P. Hoult, Economic Considerations in Sounding Rocket Utilization, 1967.

Open the calculatorOriginal memo (PDF)

Overview

A new rocket carries defects in its design and workmanship that no analysis or ground test has caught. Some of them are lethal: if one shows up in flight, the flight fails. Each test flight gives these defects a chance to show up, and a defect that shows up can be found and fixed before the next flight. Reliability therefore grows from one flight to the next.

The model is deliberately simple. It supports management decisions: how much development analysis and testing to do, how much quality assurance, and how deep the failure-mode analysis and the flight instrumentation should go. More detailed models exist, but they need more parameters estimated in advance, so in the end they are no more useful.

The model

Defects occur independently of one another with binomial statistics, and the analysis is done on an expectation basis. Let

  • D_n be the expected number of lethal latent defects left after n flight tests (D_0 at the first flight);
  • p the probability that any one latent defect shows up in a given flight;
  • d the probability that a defect which shows up in a test flight is detected, characterised and corrected before the next flight;
  • T the number of flight tests and R the reliability.

Reliability is the probability that no defect shows up in a flight. For the first flight

R_0=(1-p)^{D_0}

In the nth flight, D_{n-1}p defects are expected to show up, and D_{n-1}p\,d of them are fixed. So

D_n=D_{n-1}-D_{n-1}p\,d=D_{n-1}(1-p\,d),\qquad D_n=D_0(1-p\,d)^n

and the reliability of the flight that follows T flight tests is

R_T=(1-p)^{D_T}=(1-p)^{D_0(1-p\,d)^T}

Each test removes on average the fraction p\,d of the defects still present. The product shows the two levers: p is set by the vehicle and its flights, while d is set by how well each flight is instrumented, recovered and investigated.

The chance p acts both ways. Taking logarithms, \ln R_T=D_0(1-p\,d)^T\ln(1-p): a large p ruins the first flight but removes defects quickly, while a small p does little harm on any one flight but lets defects survive the tests. After a few tests the reliability is therefore lowest for an intermediate p; the plot of reliability against p shows where the design sits.

Schaechter and Hoult studied a set of sounding-rocket development histories and found D_0\approx4, p\approx0.6 and d\approx0.8 to be representative. With these values the first flight succeeds with probability 0.026; after five tests the next flight succeeds with probability 0.87, and after ten with 0.995.

Tests needed for a target reliability

The flight after the tests meets a target reliability R^* when no more than

D^*=\frac{\ln R^*}{\ln(1-p)}

latent defects remain. Solving D_0(1-p\,d)^T=D^* for T and rounding up to whole flights gives

T^*=\left\lceil\frac{\ln(D^*/D_0)}{\ln(1-p\,d)}\right\rceil

If the first flight already meets the target (D_0\le D^*) no test is needed. If nothing that shows up is ever fixed (d=0), the reliability never grows and no number of tests is enough. The plot of T^* against d shows how strongly instrumentation and failure investigation shorten a campaign.

Expected failures during the campaign

Flight n+1 fails with probability 1-R_n. The expected number of failed flights among the first T is the sum of these chances:

F_T=\sum_{n=0}^{T-1}\left[1-(1-p)^{D_n}\right]

It is a useful figure for budgeting vehicles and schedule. It levels off once most defects are fixed, while the reliability of each further flight keeps approaching one.

Verdict and fixes

The memo gives the reliability after a test campaign and the number of tests that reaches a target; the target is yours. The calculator compares the failure chance of the next flight, 1-R_T, with the one your target allows, 1-R^*: safe up to it, marginal up to twice it, unsafe beyond (this site’s bands, not a published limit).

The fixes follow exactly from R_T=(1-p)^{D_0(1-pd)^T}: fly T^* tests; or, keeping your planned T, raise the fixing rate to d_{need}=[1-(D^*/D_0)^{1/T}]/p; or start with no more than D_{0,need}=D^*/(1-pd)^T latent defects.

Verification

tests/relgrowth.test.js checks the following:

  • the reliability for 0 to 25 flight tests against the reference table for D_0=4, p=0.6, d=0.8;
  • the closed form D_n=D_0(1-p\,d)^n against the flight-by-flight recursion;
  • that T^* is the smallest whole number of tests that meets the target;
  • the expected-failure sum and its limit with no fixing.

Notes on the method

  • Representative values. The memo’s text quotes D_0=6 for its typical results, but its plot and table start at R_0=0.4^4=0.0256, i.e. D_0=4. The calculator’s default follows the plot and table.
  • Expectation basis. All numbers are expected values; the actual number of defects and of failures in a campaign scatters about them. No confidence bounds are given.
  • Added results. The target inversion T^* and the expected failures F_T follow directly from the model’s equations.
  • Not modelled. Defects introduced by fixes or design changes, defects that are not lethal, and different values of p for different defects.

References

  1. C. P. Hoult, “Reliability Growth During Flight Test,” RS&T, 20 July 2013. View in library
  2. W. Schaechter and C. P. Hoult, “Economic Considerations in Sounding Rocket Utilization,” Proceedings of the Sounding Rocket Vehicle Technology Specialist Conference, Williamsburg, VA, February 1967.