Sensitivity analyses and meta-modeling in complex decision support tools for transportation asset management
Date
2025
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
The importance of decision support tools in transportation asset management (TAM) cannot be over-emphasized. Decision support tools have been developed to solve a variety of TAM problems, yet Departments of Transportation (DOTs) also continue to develop new decision support models without necessarily measuring the perceived benefits from the new model viz-a-viz existing models. Furthermore, some studies, as well as interactions with practitioners, have shown that the agencies sometimes revert to simple, legacy tools due to the blackbox nature of newer, more complex models, as well as their analytical and computational requirements. Efforts should be made to encourage appropriate use of the available complex tools. ☐ This research involved two objectives. The first objective explored the role of verification and validation of decision models in TAM through variance-based sensitivity analysis techniques. The sensitivity analysis helps to understand how model parameters and variables influence policies and decision, thereby reducing the skepticism inherent in the use of complex models. The second objective proposed and implemented a meta-model framework as a higher-level tool to facilitate the matching of a set of decision support tools to appropriate decision contexts. ☐ In the first objective, a global sensitivity analysis technique, called Sobol’s method is used to identify the importance of the model’s input variables and parameters of a case-study complex decision support tool. The analysis showed that, unlike local methods, where the sensitivity of the model’s outcomes to the inputs are examined one at a time, the Sobol method showed a superior ability to identify the interactions among parameters and their importance. That is, the analysis showed that one must be careful in dealing with the parameters in complex decision support tools as there is a tendency to erroneously underestimate the influence of some parameters on the output when they are considered independently as opposed to their combined effects which may reveal that they are otherwise consequential to the outcome of the model. ☐ In the second objective, a machine-learning-based meta-model procedure for selecting appropriate support tools for specific decisions according to the scenarios encountered was demonstrated. Three case study decision support tools were applied to different decision scenarios that are characterized by the transportation network variables. A model assignment rule for selecting the most appropriate tool for each scenario was defined using the differences in the outputs from the different tools. This produced a labeled dataset for each decision scenario. A chi-square test performed on this data revealed a significant association between the congestion levels on the networks and the choice of decision thereby reinforcing a hypothesis that the complex tool may be favored in congested networks. A cluster analysis indicated that two of the three models were dominantly favored in all the clusters identified, suggesting that the asset manager may focus less on the third model. The labels of the scenarios were reassigned disregarding the least dominant model and a classification model was trained to predict the appropriate tool to use for future scenarios. Five classification algorithms, logistic regression, support vector machine, k nearest neighbor, decision trees and random forest, were trained. Principal component analysis was to reduce the dimensionality of the data and the principal components were used as the independent variables. The findings show that it is possible to reduce the catalog of decision support tools available to solve the same problem by identifying the ones that will not apply to a group of scenarios with similar characteristics. The performance of the meta-model classifier also shows that the proposed framework can predict the most preferred decision support tools to use for previously unseen scenarios. ☐ In summary, complex decision support tools have their place in TAM. However, their use is not always warranted. Furthermore, users of such models should conduct rigorous local and global sensitivity analysis to verify that the models behave as expected and to assist in parameter selection. This research demonstrated the value of such rigorous sensitivity analysis and the meta-model framework in the context of three decision tools and a selection of toy networks. Additional research is needed to determine whether these conclusions apply to other decision-support tools and larger networks.
Description
Keywords
Decision support tools, Metamodeling, Sensitivity analysis, Transportation asset management, Complex decision support tools