Flexible Least Squares:
A Multicriteria Optimization Method
for Model Specification

Last Updated: 9 June 2024 Site Maintained By:   Leigh Tesfatsion   Prof. Emerita of Economics   Courtesy Research Professor    of Electrical & Comp. Eng.   Heady Hall 260   Iowa State University   Ames, Iowa 50011-1054   tesfatsi AT iastate.edu FLS Intro Article FLS-TVLR Article GFLS Intro Article

• Overview
  • What is Flexible Least Squares (FLS)?
  • The Basic FLS Approach
  • FLS for Time-Varying Linear Regression (FLS-TVLR)
  • FLS for Approximately Linear Systems (GFLS)
  • FLS for K-Dimensional Goodness-of-Fit Criterion Functions (KFLS)
  • FLS and Kalman Filtering
  • FLS as a Diagnostic Method for Model Specification
• Publications Developing the Basic FLS Methodology
• FLS Comparative Testing
• FLS Software & Manual Availability
• Illustrative FLS Applications

Overview

What is Flexible Least Squares (FLS)?

Flexible Least Squares (FLS) is a multicriteria optimization method for model specification.

FLS permits users to identify the "Pareto frontier" of all efficiently estimated models conditional on: (i) a given theory; (ii) a given data set; and (iii) a designated collection of one or more goodness-of-fit metrics.

FLS does not require the imposition of problematic stochastic assumptions on "residual error terms" that in fact arise from deterministic model mis-specification.

The Basic FLS Approach

Any real-world system that a researcher attempts to model will inevitably behave in a manner that is incompatible to some degree with the theoretical assumptions the researcher has incorporated in the model. "All models are wrong, but some are useful." (George E.P. Box, 1987)

These theoretical assumptions typically fall into four conceptually-distinct categories: (1) dynamic assumptions postulating constraints on the changes in state variables over time; (2) measurement assumptions postulating relationships between observed values and model-predicted values; (3) cross-sectional assumptions postulating relationships among simultaneously determined endogenous variables; and (4) stochastic assumptions postulating constraints on the realizations for variables assumed to be randomly generated.

Discrepancies between the theoretical assumptions (1)-(4) and the actual real-world system of interest are called model specification errors.

The model specification errors arising for any given modeling of a real-world system are not necessarily commensurable in terms of a single empirically-meaningful scalar metric. For example, there is no particular reason to think that a measurement specification error arising from the use of an imperfect measurement device can be directly compared with a dynamic specification error that arises because the dynamic relationship between successive system states has incorrectly been assumed to take a linear rather than a nonlinear form.

Consequently, given a model incorporating conceptually distinct types of theoretical assumptions, the fitting of this model to a given data set is intrinsically a multicriteria optimization problem. Any such fitting will result in conceptually distinct types of model specification errors indicating the extent to which conceptually distinct types of theoretical relationships are incompatible with the data set. An econometrician undertaking the fitting would presumably prefer each type of model specification error to be small. However, beyond a particular point a further decrease in one type of model specification error will typically come at the cost of an increase in another.

One way to proceed here would be to induce commensurability among model specification errors by assuming they are governed by a joint probability distribution. An important drawback to this approach is that different modelers will typically have different prior conceptions regarding what constitutes a meaningful assignment of probability assessments, conceptions that cannot be brought to conformity on the basis of available empirical evidence.

In a series of studies summarized in Kalaba and Tesfatsion (CSDA,1996), Bob Kalaba and I develop an alternative approach to this multicriteria optimization problem, referred to as Flexible Least Squares (FLS). As clarified below, The FLS approach accommodates a wide range of views regarding the appropriate interpretation, measurement, and estimation of model specification errors.

FLS for Time-Varying Linear Regression (FLS-TVLR)

At one end of the range of FLS applications, relying solely on simple "smallness" priors for model specification errors, we develop FLS for time-varying linear regression.

For illustration, consider a simple linear regression relation y = bX with a scalar dependent variable y, a K-dimensional coefficient vector b, and a K-dimensional vector X of explanatory variables. Suppose this linear regression relation is postulated to explain a time-series data set T consisting of scalar dependent-variable observations y₁,...,y_N and K-dimensional vectors X₁,...,X_N of explanatory variables recorded for time periods n = 1,...,N. The goal is to study the extent to which the null theoretical hypothesis of a linear regression relation y = bX with a constant coefficient vector b is incompatible with the time-series data set T.

Each estimated model M(T) for the postulated linear regression relation y = bX, conditional on the time-series data set T, is characterized by an estimate (b*₁(T),...,b*_N(T)) for the sequence (b₁,...,b_N) of coefficient vectors b_n for time periods n = 1,...,N. Two conceptually distinct types of specification errors, dynamic and measurement, can be associated with M(T).

• The dynamic specification errors consist of the differences [b*_n+1(T) - b*_n(T)] between coefficient-vector estimates b*_n(T) and b*_n+1(T) for successive time periods n = 1,...,N-1, relative to the null of a constant coefficient vector b for all time periods n = 1,...,N.

• The measurement specification errors consist of the differences [y_n - b*_n(T)X_n] between the actual observed outcome y_n and the theoretically predicted outcome b*_n(T)X_n for time periods n = 1,...,N, relative to the null of a linear regression relation y = bX.

Let these dynamic and measurement model-specification errors, in squared form, be separately aggregated into squared-error sums RD(T) and RM(T). As explained with care in Kalaba and Tesfatsion (1989a), the basic FLS objective for time-varying linear regression is to determine the Residual Efficiency Frontier REF(T), a construction analogous to the concept of a Pareto Efficiency Frontier in standard economic analysis.

Specifically, the Residual Efficiency Frontier REF(T) consists of all estimated models M(T) whose corresponding squared-error sums (RD(T),RM(T)) for dynamic and measurement model specification errors are minimal in the following vector sense:

Given any estimated model M(T) along REF(T) with corresponding squared-error sums (RD(T),RM(T)), there does not exist any other estimated model M#(T) whose corresponding squared-error sums (RD#(T),RM#(T)) are strictly smaller than (RD(T),RM(T)) in the following vector sense: RD#(T) does not exceed RD(T), RM#(T) does not exceed RM(T), and either RD#(T) is strictly smaller than RD(T) or RM#(T) is strictly smaller than RM(T).

Consequently, as illustrated in the figure appearing at the top of this website, REF(T) is analogous to a Pareto Efficiency Frontier of estimated models M(T), given the time-series data set T.

Finally, note that the estimated model M^o(T) corresponding to ordinary least squares (OLS) linear regression with a time-invariant estimate b^o(T) for the coefficient vector b is obtained at the limit point of REF(T) where RD(T) = 0 and RM(T) attains its largest possible value. Starting from this limit point,the measurement error RM(T) successively declines along REF(T) as RD(T) is permitted to increase from 0, i.e., as the dynamic specification errors [b*_n+1(T) - b*_n(T)], n = 1, ..., N-1, are permitted to increase in size.

A Fortran program for FLS-TVLR has been incorporated into the statistical packages SHAZAM and GAUSS. See the software section, below, for details.

FLS for Approximately Linear Systems (GFLS)

As detailed in Kalaba and Tesfatsion (1990a), we also developed a Generalized Flexible Least Squares (GFLS) method for state-space models whose postulated dynamic and measurement relationships constitute a general linear-affine system of equations. The concept of the Residual Efficiency Frontier REF(T) is correspondingly generalized to a Cost-Efficient Frontier CEF(T) for which costs are separately assessed for dynamic and measurement specification errors.

A Fortran program for GFLS has been incorporated into the statistical package GAUSS. See the software section, below, for details.

FLS for K-Dimensional Goodness-of-Fit Criterion Vectors (KFLS)

More generally, suppose a theoretical state-space model has been conjectured as a possible explanation for a given time-series data set T. Suppose, also, that a modeler has specified a K-dimensional vector of incompatibility cost functions for measuring the degree of incompatibility between theory and data in accordance with K different goodness-of-fit criteria, where K is greater or equal to 1.

In a succession of studies that are listed below in the publications section, and summarized in Kalaba and Tesfatsion (1996), Bob Kalaba and I developed what is here referred to as KFLS, an extended FLS method that handles this K-dimensional goodness-of-fit problem.

More precisely, KFLS is a constructive procedure for the determination of a Cost-Efficient Frontier CEF(T) for this problem. The points along CEF(T) correspond to the family of all estimated state-space models that are equally efficient with respect to achieving vector-minimalization of these K incompatibility cost functions.

In addition, we obtain a recurrence relation for CEF(T). Let T_n denote the time-series data in T pertaining to time periods 1,...,n. For each n, the recurrence relation expresses CEF(T_n+1) as a function of CEF(T_n) together with a K-dimensional vector of incremental incompatibility costs associated with new data obtained between n and n+1.

We show that KFLS encompasses a number of other state-space estimation methods. For example, FLS and GFLS are special cases of KFLS with K=2. Moreover, KFLS reduces to the standard Kalman Filter for the special case in which all model specification errors are assumed to be stochastic disturbances to otherwise correctly specified theoretical relationships, thus permitting the derivation of a single (K=1) real-valued incompatibility cost function in the form of a posterior probability distribution.

Finally, we also clarify the interesting relationship between KFLS and the general-to-specific econometric methodology advocated by David Hendry, among others, and between the KFLS CEF construction and the information contract curve (global sensitivity analysis) approach developed by Ed Leamer.

FLS and Kalman Filtering

As discussed more fully in (Kalaba and Tesfatsion, 1990b), it is logically incorrect to equate FLS for time-varying linear regression with Kalman Filtering (KF).

FLS for time-varying linear regression does not require probability assumptions either for its motivation or for its solution. As previously explained, its goal is to characterize the set of all possible sequences of coefficient-vector estimates that achieve vector-minimal incompatibility between imperfectly specified theoretical relationships and a given time-series data set T.

In contrast, KF applied to a time-varying linear regression problem conditional on a time-series data set T is a recursive method for the evaluation of the T_n-conditional moments of the period-n coefficient vector for each time period n, where T_n denotes the time-series data in T pertaining to time periods 1,...,n. The time-varying linear regression model is expressed as a stochastic linear state-space model with coefficient vectors identified as state vectors. The general form of the stochastic model is assumed to be correctly and completely specified. In particular, "residual error" terms are assumed to be stochastic disturbance terms governed by a joint probability distribution whose general form is common knowledge.

Nevertheless, FLS was originally motivated by a puzzling aspect of KF: Why has KF proved to be useful in empirical practice, even when its assumptions are not even remotely satisfied? The original goal of FLS was to get to the heart of the matter by stripping away non-essential aspects of KF that complicate its application without adding to scientific understanding.

The key non-essential aspect of KF was soon revealed to be overly strong distributional assumptions for "residual error" terms.

The KF presumption that "residual error" terms are governed by a joint probability distribution permits the use of scalar posterior probability distributions for state moment estimation. However, this presumption hides the multicriteria aspects of the underlying estimation problem, i.e., the possible presence of multiple conceptually distinct types of model specification errors.

In contrast, as detailed above, FLS for time-varying linear regression sets out the underlying multicriteria estimation problem in explicit bare-boned terms. Given a time-series data set T, FLS solves for the entire Residual Efficiency Frontier REF(T) of estimated linear regression models that are efficient in the sense that measurement specification errors RM(T) are minimal for each given level RD(T) of dynamic volatility in the estimated regression coefficients.

Thus, for a researcher unsure about model specification errors, an FLS-generated REF provides a way to obtain a fuller picture of the researcher's true "cone of uncertainty" for the model. If additional knowledge becomes available constraining the possible sizes of the model specification errors, the REF can be correspondingly narrowed. Otherwise, restricting attention to a proper subset of the REF is an arbitrary decision.

This understanding leads to the recognition that KF and FLS represent two endpoints of a continuum of approaches to time-varying linear regression ranging from relatively strong distributional priors (KF) to simple "smallness" priors (FLS) for dynamic and measurement model specification errors. All of these approaches, from KF to FLS and everything between, should be routinely available in the toolkits of statisticians and econometricians as they attempt to gain a better understanding of real-world systems. The determination of a "best" approach should then depend on the extent to which different forms of prior restrictions on model specification errors can be scientifically justified.

FLS as a Diagnostic Method for Model Specification

In FLS studies, the full display of the Residual Efficiency Frontier (REF) or Cost-Efficient Frontier (CEF) can facilitate the discovery of true model structure. The qualitative persistence all along the REF/CEF of particular forms of time variation in FLS-estimated state vectors gives confidence that these time variations reflect true attributes of the underlying data generating mechanism.

For example, as reported in Kalaba and Tesfatsion (1989a), FLS studies were conducted for time-varying linear regression systems characterized by step-function, elliptical, sinusoidal, and other types of time-variation in the regression coefficients. It was found that these true underlying time variations were correctly displayed by the FLS-estimated regression coefficients all along the REF, albeit in increasingly flattened form as the limit point was approached where the dynamic cost RD becomes zero (i.e., where the OLS solution is obtained).

Restriction to a subset of the FLS-estimated models along a REF/CEF is arbitrary unless additional prior information is available to supplement the given theoretical model and time-series data set. In some cases a researcher might indeed have prior information about a system under study that permits the researcher to assess ex post which estimated models along a REF/CEF have physically sensible interpretations and which do not. The researcher might then be able to use these assessments to limit attention to a proper subset of the REF/CEF.

NOTE: Robert E. Kalaba, great scholar, mentor, colleague, and friend, died on 9/29/2004.

Publications Developing the Basic FLS Methodology (Chronological Order)

• Luis Frank (2023), "Time Series Reconciliation Through Flexible Least Squares Estimation" (pdf,383KB), Cuadernos Del CIMBAGE No. 25, June 2023, 1-14.

  Summary: This paper improves a recently developed procedure for using Kalaba and Tesfatsion's Flexible Least Squares (FLS) method to reconcile time series consisting of data measured at differing frequencies. In the previous version of the procedure the author assumed prior knowledge of the weighting parameter mu (µ) appearing in the FLS "incompatibility cost function" used to generate the FLS Residual Efficiency Frontier (REF), i.e., the Pareto frontier consisting of all efficiently estimated time-varying linear regression models conditional on a given theory, time-series data set, and goodness-of-fit metric. In the new version of the procedure, this parameter is estimated from sample data using an iterative method based on the Newton-Raphson algorithm. For demonstration, the new method is used to reconcile the monthly growth rates of the Monthly Economic Activity Estimator (EMAE) for Argentina with the quarterly growth rates of the Gross Domestic Product for Argentina.

  Yunxia Ni, Lixing Lv, Yuejiang Ji (2022), "Flexible Least Squares Algorithm for Switching Models" (pdf,981KB), Hindawi Complexity, Vol. 2022, Article ID 2605700, October 2022, 11pp.

  Summary: Inspired by the flexible least squares approach first developed by Kalaba and Tesfatsion, this paper proposes a recursive flexible least squares algorithm for switching models with two considered types of estimation cost: (i) cost due to neighboring two-parameter estimation error; and (ii) cost due to output estimation error. Convergence properties and simulation results are provided to support the claim that this algorithm provides several advantages over traditional approaches to the estimation of switching models.

  Jing Chen, Liuxiao Guo, Manfeng Hu, Min Gan, Quanmin Zhu (2021), "Augmented Flexible Least Squares Algorithm for Time-Varying Parameter System" (pdf,1.1MB), International Journal of Robust Nonlinear Control, December, Wiley Online Library, 1--19.

  Abstract: "This study proposes an augmented flexible least squares (FLS) algorithm for time-varying parameter systems. The parameter estimates, obtained by minimizing the squared residual measurement and dynamic errors, can catch the true values through a penalized term/weight. The algorithm associated properties are analyzed accordingly. By absorbing all into time varying parameters, the algorithm can convert complex nonlinear processes into various linear relations in time varying parameters. Thus, it can be extended to many kinds of systems. Compared to the classical FLS algorithm, the algorithm proposed in this article has less computational efforts and concise structures. To show the effectiveness of the algorithm and help the readers to follow systematically, this study provides several simulation examples."

  A. H. Bura, Bo Chen, and Li Yu (2012), "Error-Driven Adaptive, Virtual Machine Model-Based Control with High Availability Platform", Proceedings, 2012 11th International Conference on Machine Learning and Applications, 13-17.

  Abstract: "An error-driven adaptive model-based control system, for optimizing machine or assembly plant performance and operation under normal and fault conditions, is proposed. ... Using Virtual Machine Model concept, the method is achieved in three steps: state prediction, fault detection & sensor measurement and system online update or correction. A combination of flexible least square algorithm and adaptive Kalman filtering method are implemented to learn and predict system behavior. The experimental results show that the proposed model and algorithms can efficiently identify faulty components, reduce noise errors injected by sensors/system and thus providing self healing."

  Benjamin Hamidi, Bertrand Maillet, and Paul Merlin (2011), "A Robust Time-Varying Style Analysis for Hedge Funds based on Dynamic Quantiles" (pdf,563KB), CES/CNRS Working Paper, Paris, February.

  Abstract: Financial management styles are often active, time-varying, and dependent on market opportunities. This paper adapts the time-varying flexible least squares (FLS) approach to L-estimators based on quantile regression, resulting in a time-varying multi-quantile robust approach for financial management style analysis. The approach is used to study hedge fund strategies as defined in the HFR database, using monthly data from January 1995 to January 2010.

  Giovanni Montana, Kostas Triantafyllopoulos, and Theo Tsagaris (2009), "Flexible Least Squares for Temporal Data Mining and Statistical Arbitrage" (pdf,348KB), Expert Systems with Applications 36(2), Part 2, 2819-2830.

  Abstract: "A number of recent emerging applications call for studying data streams, potentially infinite flows of information updated in realtime. When multiple co-evolving data streams are observed, an important task is to determine how these streams depend on each other, accounting for dynamic dependence patterns without imposing any restrictive probabilistic law governing this dependence. In this paper we argue that flexible least squares (FLS), a penalized version of ordinary least squares that accommodates for time-varying regression coefficients, can be deployed successfully in this context. Our motivating application is statistical arbitrage, an investment strategy that exploits patterns detected in financial data streams. We demonstrate that FLS is algebraically equivalent to the well-known Kalman filter equations, and take advantage of this equivalence to gain a better understanding of FLS and suggest a more efficient algorithm. Promising experimental results obtained from a FLS-based algorithmic trading system for the S&P 500 Futures Index are reported."

  Claudio Morana (2009), "An Omnibus Noise Filter" (pdf,237KB), Computational Statistics, Vol. 24, 459-479.

  Abstract: "A new noise filtering approach, based on flexible least squares (FLS) estimation of an unobserved component local level model, is introduced. The proposed FLS filter has been found to perform well in Monte Carlo analysis, independently of the persistence properties of the data and the size of the signal to noise ratio, outperforming in general even the Wiener Kolmogorov filter, which, theoretically, is a minimum mean square estimator. Moreover, a key advantage of the proposed filter, relatively to available competitors, is that any persistence property of the data can be handled, without any pretesting, being computationally fast and not demanding, and easy to be implemented as well."

  Michael Markov, Vadim Mottl, and Ilya Muchnik (2004), "Principles of Non-Stationary Regression Estimation: A New Approach to Dynamic Multi-Factor Models in Finance" (pdf,1.5MB), DIMACS Technical Report 2004-47, October.

  Abstract: "We review existing financial multi-factor models from the standpoint of their performance in detecting hidden investment portfolio dynamics. Using practical examples, we present and analyze the shortcomings of these models in detecting both gradual and rapid changes in investment portfolio structure. We then lay the groundwork for a new approach, which we call Dynamic Style Analysis (DSA), representing a true time-series multi-factor portfolio analysis model. At the core of the methodology, we present a new dynamic regression model, which we call Constrained Flexible Least Squares (CFLS). One of the most important features of the DSA model is that it is fully adaptive, i.e., all model parameters are determined from data. The major concepts of the new methodology are gradually introduced and applied to analyses of both model portfolios and well-known public US mutual funds."

  M.J. Manohar Rao (2000), "Estimating Time-Varying Parameters in Linear Regression Models using a Two-Part Decomposition of the Optimal Control Formulation" (pdf,201KB), Sãnkhya: The Indian Journal of Statistics Vol. 62, 433-467.

  Abstract: "This paper discusses an econometric technique based on optimal control theory which, by employing a variation of the near-neighborhood search problem, is seen to be suitable for the type of research that requires estimating time-varying parameters for linear regression models. The methodology is based on the characterization of the time-varying parameter (TVP) problem as an optimal control problem, with an explicit allowance for welfare loss considerations, which leads to an algorithm capable of updating the values of the time-varying parameters as well as their covariance matrices. The technique adopts an instruments-targets approach, with the initial condition and the emphasis on parameter flexibility being the instruments; and the total welfare loss and the norm of the error vector being the targets. The methodology is a blend of the flexible least squares and Kalman filter techniques."

  Helmut Lütkepohl and Helmut Herwatz (1996), "Specification of Varying Coefficient Time Series Models via Generalized Flexible Least Squares," Journal of econometrics 70 (1), 261-290. Available from ScienceDirect.Com using doi:10.1016/0304-4076(94)01692-5

  Abstract: "Flexible Least Squares (FLS) is a method for recursively estimating the time paths of the coefficients of a regression model with time-varying coefficients. In its standard form the FLS solution is capable of capturing smooth changes of the coefficients over the sample period. For time series models erratic coefficient changes, for instance, due to seasonal variation, are possible. A generalization of FLS is proposed which can account for such phenomena. The method is applied to artificially generated as well as real economic data. Specifically, West German income and consumption time series are analyzed in some detail."

  Robert Kalaba and Leigh Tesfatsion (1996), "A Multicriteria Approach to Model Specification and Estimation" (pdf,1.6MB), Computational Statistics and Data Analysis, Vol. 21, pp. 193-214. The published article is also available from Science Direct.

  Abstract: This study considers why multicriteria techniques have not been widely adopted in econometrics to date. It then reviews a multicriteria Flexible Least Squares (FLS) approach to model estimation developed earlier by the authors for which the basic objective is state filtering, i.e., sequentially estimating the sequence of states through which a process has passed as time proceeds and new process data is obtained. Suppose a model (theory) for a given process has been proposed, and the modeler has specified a vector of theory-data incompatibility cost functions for measuring the degree of incompatibility between the given model and observed process data. The FLS approach then involves the constructive determination and sequential updating over time of the Cost-Efficient Frontier (CEF), the family of all estimated forms of the model that are equally efficient with respect to achieving minimal theory-data incompatibility costs conditional on the current set of observed process data. This FLS approach is shown to encompass a variety of previously proposed model estimation methods as special cases: for example, FLS for time-varying linear regression problems; Generalized FLS (GFLS) for approximately linear systems; the Larson-Peschon Filter; the Swerling Filter; the Viterbi Filter; and the Kalman Filter. Connections with Hendry and Richard's general-to-specific econometric methodology and Leamer's "information contract curve" (global sensitivity analysis) are also clarified.

  Robert Kalaba and Leigh Tesfatsion (1993), "A Multicriteria Approach to Dynamic Estimation," Chapter 18, pp. 288-300, in R. Day and P. Chen (eds.), Nonlinear Dynamics and Evolutionary Economics, Oxford University Press.

  Abstract: This chapter reviews work by the authors on the multicriteria Flexible Least Squares (FLS) approach to model estimation.

  Robert Kalaba and Leigh Tesfatsion (1991), "A Unified Approach to Dynamic Estimation" (pdf,710KB), Information Sciences Vols. 57-58 (September-December), pp. 159-169. The published article is available from Science Direct.

  Abstract: Discrepancies between assumed dynamical models and observations are often handled by making further probabilistic assumptions, a tactic which has both strengths and weaknesses. A re-examination of filtering and smoothing is conducted, and an alternative multicriteria Flexible Least Squares (FLS) approach is advanced that does not require a probabilistic interpretation for discrepancy terms. This approach involves vector minimization as a key ingredient, and it specializes to the well-known Kalman, Viterbi, Larson-Peschon, and Swerling filters.

  Robert Kalaba and Leigh Tesfatsion (1990a), "Flexible Least Squares for Approximately Linear Systems (pdf,1.2MB), IEEE Transactions on Systems, Man, and Cybernetics 20(5), 978-989. The published article is also available from IEEE Xplore.

  Abstract: The problem of filtering and smoothing for a system described by approximately linear dynamic and measurement relations has been studied for many decades. Yet the potential problem of mis-specified dynamics, which makes the usual probabilistic assumptions involving normality and independence questionable at best, has not received the attention it merits. This study proposes a probability-free filter that meets this mis-specification problem head on, referred to as Generalized Flexible Least Squares (GFLS) for approximately linear systems. A Fortran program implementation is provided for GFLS, and references to simulation and empirical results are given. Although GFLS has close connections with the standard Kalman Filter, it is concretely demonstrated that there are also important conceptual and computational distinctions. The Kalman Filter provides a unique estimate for the state sequence, conditional on maintained probability assumptions for discrepancy terms. In contrast, the GFLS filter provides a family of state sequence estimates, each of which is vector-minimally incompatible with the prior dynamical and measurement specifications.

  NOTE: A program implementation for GFLS has been incorporated into the statistical package GAUSS. See the section on Software & Manual Availability below.

  Robert Kalaba and Leigh Tesfatsion (1990b), "A Further Note on Flexible Least Squares and Kalman Filtering" (pdf,174KB), Journal of Economic Dynamics and Control 14(1), February, 183-185. The published article is available from Science Direct.

  Abstract: The logical relationship between the multicriteria Flexible Least Squares (FLS) method and Kalman Filtering is clarified.

  Robert Kalaba and Leigh Tesfatsion (1990c), "An Organizing Principle for Dynamic Estimation," Journal of Optimization Theory and Applications 64(3), March, 445-470. The published article is available from SpringerLink.

  Abstract: This study develops a general multicriteria Flexible Least Squares (FLS) framework for the sequential estimation of process states. Three well-known state estimation algorithms (the Viterbi, Larson-Peschon, and Kalman Filters) are derived as monocriterion specializations. The FLS framework is used to clarify both Bayesian and classical statistical procedures for treating potential model specification errors. Recently developed bicriteria specializations (FLS, GFLS), explicitly designed to take model specification errors into account, are also reviewed. The latter specializations concretely demonstrate how the FLS framework can be used to construct estimation algorithms capable of handling disparate sources of information coherently and systematically, without forced scalarization.

  Leigh Tesfatsion and John Veitch (1990), "U.S. Money Demand Instability: A Flexible Least Squares Approach" (pdf,1.4MB), Journal of Economic Dynamics and Control 14(1), February, 151-173. The published article is also available from Science Direct.

  Abstract: This study uses the Flexible Least Squares method for time-varying linear regression to investigate coefficient stability for the Goldfeld U.S. money demand model over the volatile period 1959:Q2 to 1985:Q3. The only constraint imposed on coefficient variation over time is a smoothness prior. Nevertheless, the time paths traced out by the FLS coefficient estimates exhibit systematic idiosyncratic time variations as well as simultaneous shift movements in 1974 during the time of the first oil price shock. Moreover, the FLS estimates also indicate that the "unit root" non-stationarity problem reported by OLS money demand studies disappears if the coefficients are allowed to exhibit even small amounts of time variation.

  Robert Kalaba and Leigh Tesfatsion (1989a), "Time-Varying Linear Regression Via Flexible Least Squares" (pdf,2.4MB), Computers and Mathematics with Applications, Vol. 17 (8/9), pp. 1215-1245. The published article is available from Science Direct.

  Abstract: This study develops in detail a multicriteria Flexible Least Squares (FLS) method for time-varying linear regression. The basic FLS objective is to determine the Residual Efficiency Frontier (REF), that is, the set of all coefficient trajectory estimates that yield vector-minimal sums of squared residual dynamic errors conditional on a given set of observations.

  NOTE: A FLS program implementation for time-varying linear regression has been incorporated into the statistical packages GAUSS and SHAZAM. See the section on Software & Manual Availability below.

  Robert Kalaba and Leigh Tesfatsion (1989b), "Sequential Nonlinear Estimation with Non-Augmented Priors," Journal of Optimization Theory and Applications 60(3), March, 421-438. The published article is available from SpringerLink.

  Abstract: A "flexible least cost method" is proposed for investigating the basic compatibility of theory and observations in the absence of valid or known stochastic characterizations for residual error terms.

  Robert Kalaba and Leigh Tesfatsion (1988a), "The Flexible Least Squares Approach to Time-Varying Linear Regression" (pdf,360KB), Journal of Economic Dynamics and Control 12(1), March, 43-48. The published article is available from Science Direct.

  Abstract: This study proposes a Flexible Least Squares (FLS) method for state estimation when the dynamic equations are unknown but the process state evolves only slowly over time. A smoothness prior is introduced in place of an explicit specification for the unknown dynamic equations governing the evolution of the process state. Simulation experiments illustrating the method are presented.

  Robert Kalaba and Leigh Tesfatsion (1988b), "Exact Sequential Filtering, Smoothing, and Prediction for Nonlinear Systems" (pdf,1.4MB), Nonlinear Analysis 12(6), June, 599-615. The published article is available from Science Direct.

  Abstract: This study develops two algorithms for the exact sequential updating of the optimal solution for a general discrete-time nonlinear least squares estimation problem as the process length increases and new observations are obtained. One algorithm proceeds via an imbedding on the process length and the final state vector. The second algorithm proceeds via an imbedding on the process length and the final observation vector. Each algorithm generates optimal (least cost) filtered and smoothed state estimates, together with optimal one-step-ahead state predictions.

  Robert Kalaba, Karl Spingarn, and Leigh Tesfatsion (1981), "A Sequential Method for Nonlinear Filtering: Numerical Implementation and Comparisons," Journal of Optimization Theory and Applications 34(4), August, 1144-1149. The published article is available from SpringerLink.

  Abstract: Exact equations are presented for sequentially updating the optimal solution for a discrete-time analog of the basic Sridhar nonlinear filtering problem as the process length increases and new observations are obtained. A tabular method is described for implementing numerically the sequential filtering equations. The accuracy and efficiency of the tabular method are illustrated by means of several numerical examples.

  Robert Kalaba and Leigh Tesfatsion (1981), "An Exact Sequential Solution Procedure for a Class of Discrete-Time Nonlinear Estimation Problems" (pdf,595KB), IEEE Transactions on Automatic control, Vol. AC-26, pp. 1144-1149. The published article is available from IEEE Xplore.

  Abstract: This study develops an exact procedure for sequentially updating the optimal solution for a general discrete-time nonlinear least squares estimation problem as the process length increases and new observations are obtained.

  Robert Kalaba and Leigh Tesfatsion (1980), "A Least-Squares Model Specification Test for a Class of Dynamic Nonlinear Economic Models with Systematically Varying Parameters," Journal of Optimization Theory and Applications 32(4), December, 537-567. The published article is available from SpringerLink.

  Abstract: This study develops a least-squares measure for simultaneously testing the basic compatibility of prior dynamical, observational, and distributional model specifications against actual data for a class of dynamic nonlinear economic models with parameters explicitly modeled as nonlinear functions of endogenous and exogenous variables. Using invariant imbedding techniques, an algorithm is derived for sequentially updating the optimal least-squares estimates for parameters, endogenous variables, and squared residual modeling error sums as the duration of the process increases and new observations are obtained.

FLS Comparative Testing

• Zsolt Darvas and Balázs Varga (2012), "Uncovering Time-Varying Parameters with the Kalman-Filter and the Flexible Least Squares: A Monte Carlo Study" (pdf,2.3MB), Working Paper 2012/4, Department of Mathematical Economics and Economic Analysis, Corvinus University of Budapest, Hungary, December.

  Abstract: "Using Monte Carlo methods, we compare the ability of the Kalman-filter, the Kalman smoother and the flexible least squares (FLS) to uncover the parameters of an autoregression. We find that the ordinary least squares (OLS) estimator performs much better than the time-varying coefficient methods when the parameters are in fact constant, but the OLS does very poorly when parameters change. Neither the FLS, nor the Kalman-filter and Kalman-smoother can uncover sudden changes in parameters. But when parameter changes are smoother, such as linear, sinusoid or even random walk changes in the parameters, the FLS with a weight parameter 100 works reasonably well and typically outperforms even the Kalman-smoother, which in turn performed better than the Kalman-filter."

• Andreas Kladroba (2005), "Flexible Least Squares Estimation of State Space Models: An Alternative to Kalman Filtering?" (pdf,126KB), Working Paper 149, Universitat Duisburg-Essen.

  Abstract: "In 1990 Kalaba/Tesfatsion developed a Flexible Least Square (FLS) approach for estimating state space models as an alternative to Kalman filtering. In this paper we ask whether FLS is really an alternative. For answering this, we use a simulation using FLS as a regression model with time varying parameters. We will estimate this model with Kalman filtering and two variations of FLS. In a second step we will mis-specify the so-called hyperstructure of the model and we will prove how the two ways of estimating (Kalman filtering and FLS) react to this mis-specification."

  Important Note: The findings of this paper have been misreported in several subsequent studies. What, in fact, Kladroba concludes (p. 10) is that "(FLS) is a method which achieves the same or better estimation results like Kalman Filtering and which does not cause more problems than Kalman Filtering." For a detailed discussion focusing on distinctions and commonalities between Kalman filtering and FLS, see Kalaba and Tesfatsion (1990, Section 4)

FLS Open Source Software & Manual Availability

• FLS for Time-Varying Linear Regression (FLS-TVLS):

  FLS-TVLR has been implemented by means of an FLS-TVLR Fortran Program developed by Kalaba and Tesfatsion (the copyright holders), released as free open-source software under the terms of the Artistic License Agreement (html).

  Programs implementing FLS-TVLR are also available in the statistical packages SHAZAM and GAUSS - TSM (Time Series Methods).

  An R Implementation of FLS-TVLR by Bryan Wayne Lewis is available at GitHub, released under an open source (GPL) license.

  An R/C++ Implementation of FLS-TVLR by Gediminas Bagdonas is available at GitHub, released under an open source (GPL-3) license. Original release: May 16, 2019.

• FLS-TVLR Manual:

  Robert Kalaba and Leigh Tesfatsion, "Time-Varying Linear Regression Via Flexible Least Squares" (pdf,2.4MB), Computers and Mathematics with Applications Vol. 17 (1989), 1215-1245. The published article is available from Science Direct.

• Generalized FLS for Approximately Linear Systems (GFLS):

  GFLS has been implemented by means of a GFLS Fortran Program developed by Kalaba and Tesfatsion (the copyright holders), released as free open-source software under the terms of the Artistic License Agreement.

  Programs implementing GFLS are also available in the statistical package GAUSS - TSM (Time Series Methods) under the category Estimation Tools for Time Series Analysis.

• GFLS Manual and Help File:

  Robert Kalaba and Leigh Tesfatsion, "Flexible Least Squares for Approximately Linear Systems" (pdf,1.2MB), IEEE Transactions on Systems, Man, and Cybernetics, Vol. 20, No. 5 (1990), 978-989.

  A GFLS Help File is also available.

Illustrative FLS Applications

• Ufuk Akcigit, Raman Chhina, Seyit Cilasun, Javier Miranda, Eren Ocakverdi, Nicolas Serrano-Velarde (2023), "Intuit QuickBooks Small Business Index: A New Employment Series for the US, Canada, and the UK", National Bureau of Economic Research (NBER) Working Paper 31350, June, DOI:10.3386/w31350.

• Aynur Alptekin, David C. Broadstock, Xiaoqi Chen, Dong Wang (2019), "Time-Varying Parameter Energy Demand Functions: Bench-marking State-Space Methods Against Rolling Regressions", Energy Economics, August. DOI:10.1016/j.eneco.2018.03.009

• Carlos A. Arango A., Martha A. Misas A., and Juan Nicolández (2004), "La Demanda de Especies Monetarias en Colombia: Estructura Y Pronóstico" (pdf,538KB), Banco de la República, Borradores de Economia, No. 002964, September.

• Janis Berzins, Crocker H. Liu, Charles Trzcinka (2013), "Asset Management and Investment Banking", Journal of Financial Economics, Vol. 110, Issue 1, October, 215-231 (ScienceDirect).

• Jon R. Bond, Richard Fleisher, and B. Dan Wood (2003), "The Marginal and Time-Varying Effect of Public Approval on Presidential Success in Congress" (html), The Journal of Politics 65 (1), 92--110.

• Adam Brinker, Joe Parcell, and Kevin Dhuyvetter (2007), "Cross-Hedging Distillers Dried Grains: Exploring Corn and Soybean Meal Futures Contracts" (pdf,274KB), Proceedings, NCCC-134 Conference on Applied Commodity Price Analysis, Forecasting, and Market Risk Management, Chicago, IL.

• Trevor C. Brown, David J. Miron, Susannah L. Brown, and Shane M. Kendell (2016), "Time- and Temperature Varying Activation Energies: Isobutane Selective Oxidation to Methacrolein over Phosphomolybdic Acid and Copper(II) Phosphomolybdates" (pdf,1.6MB), Catalysts 6(9), 137: doi:10.3390/catal6090137.

• Trevor C. Brown, David J. Miron, Abdullah K. Alanazi, and Cam Le Minh (2013), "Rate Parameter Distributions for Isobutane Dehydrogenation and Isobutene Dimerization and Desorption over HZSM-5" (pdf,625KB), Catalysts, Vol. 3, 922--941, doi:10.3390/catal3040922.

• Cansin Kemal Can (2023), "Estimating Bohn's Fiscal Sustainability Model with Temporal Variation: Evidence from Turkey" (Preprint,pdf), Prague Economic Papers 32(1), 61--83, doi:10.18267/j.pep.822

• Nuno Cassola and Claudio Morana (2010), "Comovements in Volatility in the Euro Money Market" (Science Direct Site), Journal of International Money and Finance 29(3), April, 525--539.

• Lars-Erik Cederman (2001), "Exploring the Dynamics of the Democratic Peace", Journal of Conflict Resolution 45(6), 818--833.

• Kaiyi Chen, Ling T. He, R.B. Lenin (2016), "Time Variation Paths of Risk Sensitivities of Bank Stocks in the Past Two Decades" (DOI), The Journal of Risk Finance, Vol. 17, Issue 4, 446--445.

• Jim Clayton and Greg MacKinnon (2001), "The Time-Varying Nature of the Link Between REIT, Real Estate and Financial Asset Returns," Journal of Real Estate Portfolio Management, January--March Issue.

• Zsolt Darvas and Balázs Varga (2014), "Inflation Persistence in Central and Eastern European Countries" (link), Applied Economics 46(13), 1437--1448. DOI: 10.1080/00036846.2013.875113

• Jeffrey H. Dorfman and Kenneth A. Foster (1991), "Estimating Productivity Changes with Flexible Coefficients," Western Journal of Agricultural Economics 16 (2), 280--90.

• Markus Ebner and Thorsten Neumann (2005), "Time-Varying Betas of German Stock Returns" (html), Financial Markets and Portfolio Management 19(1), 29--46.

• Markus Ebner and Thorsten Neumann (2008), "Time-Varying Factor Models for Equity Portfolio Construction" (html), The European Journal of Finance 14(5), July, 381--395.

• Akhter Faroque and William Veloce (2008), "Fundamentals versus the Leading Index: The Forecasting of Canada's Output Growth Since 1991, an Encompassing Approach", Applied Economics, November.

• Luis Frank (2022), "Time Series Reconciliation Through Flexible Least Squares Estimation", Cuadernos Del CIMBAGE No. 25, June 2023, 1--14.

• Jason R. V. Franken and Joe L. Parcell (2003), "Market Integration: Case Studies of Structural Change" (pdf,311KB), NCR-134 Conference on Applied Commodity Price Analysis, Forecasting, and Market Risk Management, St. Louis, Missouri, April 21--22.

• Hayette Gatfaoui (2007), "Are Credit Default Swap Spreads Market Driven?," 4th AFE-QASS Conference, Samos, Greece, July.

• Hayette Gatfaoui (2006), "Credit Risk and Market Risk: Analyzing US Credit Spreads" (pdf,956KB), Working Paper, Rouen School of Management, Economics and Finance Department, Mont-Saint-Aignan Cedex, France, August.

• Miguel I. Gómez, Eliana R. González, and Luis F. Melo (2012), "Forecasting Food Inflation in Developing Countries with Inflation Targeting Regimes" (Oxford Journal Site), American Journal of Agricultural Economics 94(1), 153--173. doi: 10.1093/ajae/aar122

• Eliana González, Miguel I. Gómez, Luiz F. Melo, and José Luis Torres (2006), "Forecasting Food Inflation in Developing Countries with Inflation Targeting Regimes: The Colombian Case" (download site), Banco de la República, Borradores de Economia, No. 002735, October.

• Ling T. He (2001), "Time Variation Paths of International Transmission of Stock Volatility -- U.S. vs. Hong Kong and South Korea," Global Finance Journal 12 (1), 79--93.

• Ling T. He (2004), "Instability of Risk Loadings of Industrial Stocks" (pdf,217KB), Journal of Business and Economics Research 2(2).

• Ling T. He (2005), "Instability and Predictability of Factor Betas of Industrial Stocks: The Flexible Least Squares Solutions", Quarterly Review of Economics and Finance 45(4-5), 619--640.

• Todd Hubbs, Todd H. Kuethe, and Timothy G. Baker (2009), "Evaluating the Dynamic Nature of Market Risk" (pdf,1.3MB), Proceedings, NCCC-134 Conference on Applied Commodity Price Analysis, Forecasting, and Market Risk Management, St. Louis, MO.

• Zeng Jiusun (2012), "Identification of Linear Parameter Varying System using Flexible Least Squares," IEEEXplore Digital Library, Control Conference, Hefei, July 25--27.

• Peter Kennedy (2008), A Guide to Econometrics: Sixth Edition (Publisher's Book Guide), MIT Press, 623 pp.

  Note: Here is a sample quote from this delightfully honest and informative introduction to econometrics for those interested in science with practice: "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise."

• Todd H. Kuethe, Kenneth A. Foster, and Raymond J.G.M. Florax (2008), "A Spatial Hedonic Model with Time-Varying Parameters: A New Method Using Flexible Least Squares" (pdf,285KB), Selected Paper, American Agricultural Economics Association Annual Meeting, Orlando, Florida, July 27--29.

• Manish Kumar (2011), "A Time-Varying Parameter Vector Autoregression Model for Forecasting Emerging Market Exchange Rates" (pdf,364KB), International Journal of Economic Sciences and Applied Research 3(2), 21--39.

• Andrew C. Lee and Man-Keun Kim (2005), "Time Varying Coefficients: An Application of Flexible Least Squares to Cattle Captive Supply," Selected Paper, American Agricultural Economics Association Annual Meeting, Providence, Rhode Island, July 24--27.

• Daniel Li, Michael Markov, and Russ Wermers (2013), "Monitoring Daily Hedge Fund Performance When Only Monthly Data Is Available" (pdf,435KB), Journal of Investment Consulting 14(1), 57--68. (Winner of the JIC 2012 Academic Paper Competition)

• Zhibao Li, Xiaoyu Zeng, Jun Ni, Yuzhu Guo, Hua Zhang, Yang Li, Yina Ma, Fangan Meng (2023) "Dynamic Functional Connectivity Assesses the Progression of Parkinson's Disease" (pdf,13MB), Innovation Medicine 1(2):200027, September 21, 2023.

• Helmut Lütkepohl and Helmut Herwatz (1996), "Specification of Varying Coefficient Time Series Models via Generalized Flexible Least Squares," Journal of Econometrics 70(1), 261--290.

• Helmut Lütkepohl (1993), "The Sources of the U.S. Money Demand Instability," Empirical Economics 18(4), 729--743.

• Vêlayoudom Marimoutou, Denis Peguin, and Anne Peguin-Feissole (2009), "The `Distance-Varying' Gravity Model in International Economics: Is the Distance an Obstacle to Trade?" (pdf,333KB), Economics Bulletin 29(2), 1157--1173.

• Cam Le Minh, Abdullah K. Alanazi, David J. Miron, and Trevor C. Brown (2012), "Carbon-Carbon Bond Cleavage and Dehydrogenation of Isobutane Over HZSM-5 at Low Pressures and Temperatures" (pdf,365KB), Catalysis Letters 142(12), 1470--1473.

• David J. Miron, Shane M. Kendell, Alaa M. Munshi, Abdullah K. Alanazi, and Trevor C. Brown (2013), "Time-Varying Flexible Least Squares for Thermal Desorption of Gases" (pdf,550KB), International Journal of Chemical Kinetics" 45(6), 374--366. DOI: 10.1002/kin.20772

• Giovanni Montana, Kostas Triantafyllopoulos, and Theo Tsagaris (2009), "Flexible Least Squares for Temporal Data Mining and Statistical Arbitrage" (pdf,348KB), Expert Systems with Applications 36(2), Part 2, 2819--2830.

• Claudio Morana (2009), "On the Macroeconomic Causes of Exchange Rate Volatility" (pdf,518KB), International Journal of Forecasting 25(2), April-June, 328--350. doi:10.1016/j.ijforecast.2009.01.013

• Claudio Morana (2009), "Realized Betas and the Cross-Section of Expected Returns", Applied Financial Economics 19(7), 1371--1381.

• Claudio Morana (2005), "The Japanese Deflation: Has it Had Any Real Effects? Could It Have Been Avoided?" (pdf), Applied Economics 37, 1337--1352.

• Christopher John (C.J.) O'Donnell (2006), "Some Econometric Options for Dealing with Unknown Functional Form" (pdf), Working Paper, School of Economics, University of Queensland, Bridbane QLD 4072.

• Joe Parcell (2003), "An Empirical Analysis of the Demand for Wholesale Pork Primals: Seasonality and Structural Change", Journal of Agricultural and Resource Economics 28(2), 335--48.

• Michael C. Poray, Kenneth A. Foster, and Jeffrey H. Dorfman (2001), "Measuring an Almost Ideal Demand System with Generalized Flexible Least Squares," (pdf,113KB), Working Paper 01-01, Department of Agricultural Economics, Purdue University.

• A. Ramachandra Rao (1995), "Multivariate Flexible Least Squares Analysis of Hydrological Time Series" (pdf,381KB), Modelling and Management of Sustainable Basin-Scale Water Resource Systems, Proceedings of a Boulder Symposium, July 1995, IAHS Publication No. 231, pp. 359--366.

• Houston H. Stokes (2013), "Money Balances in the Production Function: Nonlinear Tests of Model Stability and Measurement Issues - Two Sides of the Same Coin?" (pdf,549KB), Journal of Economic Asymmetries 10, 101--114.

• Stâle Størdal and Darius M. Adams (2005), "Testing for Variation in the Western Oregon Softwood Log Price Structure," Canadian Journal of Forestry Research 35(3), 713--723. doi:10.1139/x05-008

• Leigh Tesfatsion and John Veitch (1990), "U.S. Money Demand Instability: A Flexible Least Squares Approach" (pdf,1.4MB), Journal of Economic Dynamics and Control 14(1), 151--173.

• K. Triantafyllopoulos and G. Montana (2011), "Dynamic modeling of mean-reverting spreads for statistical arbitrage", Computational Management Science 8(1), 23--49, April.

• Benu Varman and Wolfgang Schneider (1989), "Measuring Capital Flight: A Time-Varying Regression Analysis," Universitat Kiel Discussion Paper No. 82/89, December.

• Curt Wells (1996), The Kalman Filter in Finance (Chapter 3:FLS), Series: Advanced Studies in Theoretical and Applied Econometrics, Vol. 32, 192 pp., Hardcover, ISBN: 978-0-7923-3771-3.

• B. Dan Wood (2000), "Weak Theories and Parameter Instability: Using Flexible Least Squares to Take Time-Varying Relationships Seriously," American Journal of Political Science 44(3), 603--618.

• Chiang Yat-Hung, So Chun-Kei Joinkey, and Tang Bo-Sin (2008), "Time-varying performance of four Asia-Pacific REITs" (html), Journal of Property Investment and Finance 26(3), 210--231. DOI: 10.1108/14635780810871605

• Veli Yilanci, Taner Akan, and Ali Haydar Isik (2023), "Time-Varying Disaggregation of the Income-Emissions Nexus: New Evidence from the United Kingdom" (pdf,6.2MB), Journal of Environmental Management 348, 119202.

• Zsuzsanna Zsibók and Balázs Varga (2012), Inflation Persistence in Hungary: A Spatial Analysis (pdf,6.5MB), Working Paper 2012/3, Department of Mathematical Economics and Economic Analysis, Corvinus University of Budapest, Hungary, July.

Copyright © Leigh Tesfatsion. All Rights Reserved.