On resampling and uncertainty estimation in linear system identification

Linear System Identification yields a nominal model parameter, which minimizes a specific criterion based on the single input-output data set. Here we investigate the utility of various methods for estimating the probability distribution of this nominal parameter using only the data from this single experiment. The results are compared to the actual parameter distribution generated by many Monte-Carlo runs of the data-collection experiment. The methods considered are collectively known as resampling schemes, which include Subsampling, the Jackknife, and the Bootstrap. The broad aim is to generate an empirical parameter distribution function via the construction of a large number of new data records from the original single set of data, based on an assumption that this data is representative of all possible data, and then to run the parameter estimator on each of these new records to develop the distribution function. The performance of these schemes is evaluated on a difficult, almost unidentifiable system, and compared to the standard results based on asymptotic normality. In addition to the exploration of this example as means to evaluate the strengths and weaknesses of these resampling schemes, some new theoretical results are proven and demonstrated for Subsampling schemes.