From aeafe4d48ca14bb1a959550292cfaa4bcca003eb Mon Sep 17 00:00:00 2001 From: slilonfe5 Date: Fri, 3 Apr 2026 17:08:52 -0400 Subject: [PATCH 01/19] Updated covariance.rst file --- .../analysis/parmest/covariance.rst | 102 ++++++++++++++++-- 1 file changed, 94 insertions(+), 8 deletions(-) diff --git a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst index b7142e0e103..531edbad32a 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst @@ -1,14 +1,20 @@ .. _covariancesection: +Uncertainty Quantification +========================== +The goal of parameter estimation (see :ref:`driversection` Section) is to estimate unknown model parameters +from experimental data. Uncertainty quantification is required to ensure that the estimates of the parameters are +close to their true values. This parameter uncertainty can be computed using four methods in parmest: +covariance matrix, likelihood ratio test, bootstrapping, and leave-N-out. + Covariance Matrix Estimation -============================ +---------------------------- -The goal of parameter estimation (see :ref:`driversection` Section) is to estimate unknown model parameters -from experimental data. When the model parameters are estimated from the data, their accuracy is measured by -computing the covariance matrix. The diagonal of this covariance matrix contains the variance of the -estimated parameters which is used to calculate their uncertainty. Assuming Gaussian independent and identically -distributed measurement errors, the covariance matrix of the estimated parameters can be computed using the -following methods which have been implemented in parmest. +The uncertainty in estimated model parameters can be quantified by computing the covariance matrix. +The diagonal of this covariance matrix contains the variance of the estimated parameters which is used to +calculate their uncertainty. Assuming Gaussian independent and identically distributed measurement errors, +the covariance matrix of the estimated parameters can be computed using the following methods which have been +implemented in parmest. 1. Reduced Hessian Method @@ -138,4 +144,84 @@ e.g., >>> pest = parmest.Estimator(exp_list, obj_function="SSE") >>> obj_val, theta_val = pest.theta_est() >>> cov_method = "reduced_hessian" - >>> cov = pest.cov_est(method=cov_method) \ No newline at end of file + >>> cov = pest.cov_est(method=cov_method) + +Bootstrapping +------------- + +Bootstrapping is a non-intrusive method that uses resampling to approximate the variance of the parameter estimates. +By repeatedly fitting the model to resampled datasets, the parameter uncertainty (e.g., variance or covariance) +can be computed without relying on strong distributional assumptions. This method is summarized as follows: + +1. Step 0: Define the input data + + Given input data: :math:`[y_1, \dots, y_n]` + +2. Step 1: Generate :math:`B` artificial datasets through sampling + + Sample with replacement from the original data to create :math:`B` bootstrap datasets: + + .. math:: + \left\{y_1^{(1)}, \dots, y_n^{(1)} \right\}, \dots, \left\{y_1^{(B)}, \dots, y_n^{(B)} \right\} + +3. Step 2: Compute the estimator of the parameters + + Fit the model to each bootstrap dataset to obtain parameter estimates: + + .. math:: + \left\{ \hat{\theta}^{(1)}, \dots, \hat{\theta}^{(B)} \right\} + +4. Step 3: Compute the approximate variance of the parameter estimates + + The variability across bootstrap estimates approximates the estimator variance: + + .. math:: + \text{Var}(\hat{\theta}) = + \frac{1}{B} \sum_{j=1}^{B} \left(\hat{\theta}^{(j)}\right)^2 + - + \left( + \frac{1}{B} \sum_{j=1}^{B} \hat{\theta}^{(j)} + \right)^2 + +The example code for this method will soon be provided. + +Likelihood Ratio Test +--------------------- + +The likelihood ratio test is a non-intrusive method that compares how well two parameter sets explain the +observed data: an unconstrained set and a constrained set defined by the null hypothesis. It is commonly used +to assess whether restricting parameters significantly degrades model fit. This method is summarized as follows: + +1. Step 1: State the hypothesis + + Null hypothesis: :math:`\theta \in \Theta_0` vs. Alternative hypothesis: :math:`\theta \notin \Theta_0` + +2. Step 2: Define the test statistic + + The test statistic is the ratio of the maximum likelihood under the full parameter space to that under the + constrained space: + + .. math:: + \lambda_n = + \frac{\sup_{\theta \in \Theta} L(\theta; y_1, \dots, y_n)}{ + \sup_{\theta \in \Theta_0} L(\theta; y_1, \dots, y_n)} + +3. Step 3: Define the decision rule + + Reject the null hypothesis if: + + .. math:: + \lambda_n > c_{\alpha} + + Equivalently, using maximum likelihood estimates: + + .. math:: + \lambda_n = + \frac{L(\hat{\theta}; y_1, \dots, y_n)}{L(\hat{\theta}_0; y_1, \dots, y_n)} > c_{\alpha} + +The example code for this method will soon be provided. + +Leave-N-Out +----------- + +The documentation, description, and example code for this method will soon be provided. \ No newline at end of file From 74369fd6b686bea09da6d0fe559089e151746314 Mon Sep 17 00:00:00 2001 From: slilonfe5 Date: Tue, 7 Apr 2026 11:00:22 -0400 Subject: [PATCH 02/19] Added new files --- .../analysis/parmest/covariance.rst | 12 +++- .../analysis/parmest/estimability.rst | 55 +++++++++++++++++++ .../explanation/analysis/parmest/index.rst | 2 + .../analysis/parmest/objective.rst | 7 +++ 4 files changed, 73 insertions(+), 3 deletions(-) create mode 100644 doc/OnlineDocs/explanation/analysis/parmest/estimability.rst create mode 100644 doc/OnlineDocs/explanation/analysis/parmest/objective.rst diff --git a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst index 531edbad32a..29c9d91f47e 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst @@ -183,7 +183,9 @@ can be computed without relying on strong distributional assumptions. This metho \frac{1}{B} \sum_{j=1}^{B} \hat{\theta}^{(j)} \right)^2 -The example code for this method will soon be provided. +.. note:: + + The example code for this method will soon be provided. Likelihood Ratio Test --------------------- @@ -219,9 +221,13 @@ to assess whether restricting parameters significantly degrades model fit. This \lambda_n = \frac{L(\hat{\theta}; y_1, \dots, y_n)}{L(\hat{\theta}_0; y_1, \dots, y_n)} > c_{\alpha} -The example code for this method will soon be provided. +.. note:: + + The example code for this method will soon be provided. Leave-N-Out ----------- -The documentation, description, and example code for this method will soon be provided. \ No newline at end of file +.. note:: + + Detailed descriptions and example code for this method will be added in a future update. \ No newline at end of file diff --git a/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst b/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst new file mode 100644 index 00000000000..4cf40728689 --- /dev/null +++ b/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst @@ -0,0 +1,55 @@ +.. _estimabilitysection: + +Estimability Analysis +===================== + +After estimating the model parameters with their associated uncertainty, as demonstrated in the +:ref:`driversection` and :ref:`covariancesection` Section, estimability analysis is required to identify +parameters that cannot be reliably estimated from the available data due to limitations in the mathematical +model structure. If such parameters are identified, the model may need to be reformulated, replaced with an +alternative structure, or augmented with additional prior information. In parmest, estimability analysis can +be performed using eigen-decomposition of the parameter covariance matrix, profile likelihood methods, or +multi-start initialization routines. + +Eigen-decomposition +------------------- + +The estimability of model parameters can be analyzed through eigen-decomposition of the covariance matrix +obtained from parameter estimation. This covariance matrix quantifies parameter uncertainty and captures both +parameter variances and correlations. Eigen-decomposition of this matrix identifies principal directions +in parameter space along which uncertainty is largest or smallest. These directions provide insight into +parameter identifiability and reveal combinations of parameters that are either structurally identifiable or +non-identifiable based on the underlying model formulation. + +.. note:: + + Detailed descriptions and example code for this method will be added in a future update. + +Profile Likelihood +------------------ + +Profile likelihood analysis evaluates parameter estimability by systematically varying one parameter while +re-optimizing the remaining parameters to maintain consistency with the model and observed data. This approach +is closely related to likelihood ratio–based uncertainty quantification and provides a robust characterization +of practical identifiability through the shape of the likelihood surface. In addition, it can reveal structural +non-identifiability when flat or unbounded profiles indicate parameter combinations that are not uniquely +determined by the model formulation, particularly in nonlinear systems. + +.. note:: + + Detailed descriptions and example code for this method will be added in a future update. + +Multi-start Initialization +-------------------------- + +Multi-start initialization assesses parameter estimability by exploring a range of initial guesses. Because +parameter estimation problems are often nonlinear and may exhibit multiple local minima, different initializations +can lead to different parameter estimates. By solving the estimation problem from multiple starting points, one can +evaluate the robustness of the solution and identify potential issues related to non-convexity or non-identifiability. +Consistent convergence to a unique solution across initializations suggests that the parameters are structurally +identifiable within the model formulation, whereas sensitivity to initialization indicates potential estimability +issues. + +.. note:: + + Detailed descriptions and example code for this method will be added in a future update. \ No newline at end of file diff --git a/doc/OnlineDocs/explanation/analysis/parmest/index.rst b/doc/OnlineDocs/explanation/analysis/parmest/index.rst index 4616a2134d4..54e85a67071 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/index.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/index.rst @@ -27,4 +27,6 @@ Index of parmest documentation graphics.rst examples.rst parallel.rst + objective.rst + estimability.rst api.rst diff --git a/doc/OnlineDocs/explanation/analysis/parmest/objective.rst b/doc/OnlineDocs/explanation/analysis/parmest/objective.rst new file mode 100644 index 00000000000..164c5a5c0ff --- /dev/null +++ b/doc/OnlineDocs/explanation/analysis/parmest/objective.rst @@ -0,0 +1,7 @@ +.. _objectivesection: + +Objective Options +================= +.. note:: + + Detailed descriptions and example code for the objective options in parmest will be added in a future update. \ No newline at end of file From 1eb8c7ffa70eac212ee7c4c8bfb29a41752201ff Mon Sep 17 00:00:00 2001 From: slilonfe5 Date: Tue, 7 Apr 2026 12:06:35 -0400 Subject: [PATCH 03/19] Added and updated documentation files --- .../explanation/analysis/doe/guide.rst | 230 ++++++++++++++++++ .../explanation/analysis/doe/multexp.rst | 8 + .../explanation/analysis/doe/objective.rst | 8 + .../explanation/analysis/doe/overview.rst | 131 ++++++++++ .../explanation/analysis/doe/uncertainty.rst | 10 + .../analysis/parmest/covariance.rst | 6 +- .../analysis/parmest/objective.rst | 3 +- .../explanation/analysis/parmest/overview.rst | 6 +- 8 files changed, 396 insertions(+), 6 deletions(-) create mode 100644 doc/OnlineDocs/explanation/analysis/doe/guide.rst create mode 100644 doc/OnlineDocs/explanation/analysis/doe/multexp.rst create mode 100644 doc/OnlineDocs/explanation/analysis/doe/objective.rst create mode 100644 doc/OnlineDocs/explanation/analysis/doe/overview.rst create mode 100644 doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst new file mode 100644 index 00000000000..cb14c08d6e0 --- /dev/null +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -0,0 +1,230 @@ +.. _startguide: + +Quick Start Guide +================= + +To use Pyomo.DoE, a user must implement a subclass of the :ref:`Parmest ` ``Experiment`` class. +The subclass must have a ``get_labeled_model`` method which returns a Pyomo `ConcreteModel` +containing four Pyomo ``Suffix`` components identifying the parts of the model used in +MBDoE analysis. This is in line with the convention used in the parameter estimation tool, +:ref:`Parmest `. The four Pyomo ``Suffix`` components are: + +* ``experiment_inputs`` - The experimental design decisions +* ``experiment_outputs`` - The values measured during the experiment +* ``measurement_error`` - The error associated with individual values measured during the experiment. It is passed as a standard deviation or square root of the diagonal elements of the observation error covariance matrix. Pyomo.DoE currently assumes that the observation errors are Gaussain and independent both in time and across measurements. +* ``unknown_parameters`` - Those parameters in the model that are estimated using the measured values during the experiment + +An example of the subclassed ``Experiment`` object that builds and labels the model is shown in the next few sections. + +This guide illustrates the use of Pyomo.DoE using a reaction kinetics example (Wang and Dowling, 2022). + +.. math:: + :nowrap: + + \begin{equation} + A \xrightarrow{k_1} B \xrightarrow{k_2} C + \end{equation} + + +The Arrhenius equations model the temperature dependence of the reaction rate coefficients :math:`k_1` and :math:`k_2`. Assuming a first-order reaction mechanism gives the reaction rate model shown below. Further, we assume only species A is fed to the reactor. + +.. math:: + :nowrap: + + \begin{equation} + \begin{aligned} + k_1 & = A_1 e^{-\frac{E_1}{RT}} \\ + k_2 & = A_2 e^{-\frac{E_2}{RT}} \\ + \frac{d{C_A}}{dt} & = -k_1{C_A} \\ + \frac{d{C_B}}{dt} & = k_1{C_A} - k_2{C_B} \\ + C_{A0}& = C_A + C_B + C_C \\ + C_B(t_0) & = 0 \\ + C_C(t_0) & = 0 \\ + \end{aligned} + \end{equation} + + + +:math:`C_A(t), C_B(t), C_C(t)` are the time-varying concentrations of the species A, B, C, respectively. +:math:`k_1, k_2` are the rate constants for the two chemical reactions using an Arrhenius equation with activation energies :math:`E_1, E_2` and pre-exponential factors :math:`A_1, A_2`. +The goal of MBDoE is to optimize the experiment design variables :math:`\boldsymbol{\varphi} = (C_{A0}, T(t))`, where :math:`C_{A0},T(t)` are the initial concentration of species A and the time-varying reactor temperature, to maximize the precision of unknown model parameters :math:`\boldsymbol{\theta} = (A_1, E_1, A_2, E_2)` by measuring :math:`\mathbf{y}(t)=(C_A(t), C_B(t), C_C(t))`. +The observation errors are assumed to be independent both in time and across measurements with a constant standard deviation of 1 M for each species. + + +Step 0: Import Pyomo and the Pyomo.DoE module and create an ``Experiment`` class +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. note:: + + This example uses the data file ``result.json``, located in the Pyomo repository at: + ``pyomo/contrib/doe/examples/result.json``, which contains the nominal parameter + values, and measurements for the reaction kinetics experiment. + + +.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py + :start-after: # === Required imports === + :end-before: # ======================== + +Subclass the :ref:`Parmest ` ``Experiment`` class to define the reaction +kinetics experiment and build the Pyomo ConcreteModel. + +.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py + :start-after: ======================== + :end-before: End constructor definition + +Step 1: Define the Pyomo process model +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The process model for the reaction kinetics problem is shown below. Here, we build +the model without any data or discretization. + +.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py + :start-after: Create flexible model without data + :end-before: End equation definition + +Step 2: Finalize the Pyomo process model +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Here, we add data to the model and finalize the discretization using a new method to +the class. This step is required before the model can be labeled. + +.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py + :start-after: End equation definition + :end-before: End model finalization + +Step 3: Label the information needed for DoE analysis +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We label the four important groups as Pyomo Suffix components as mentioned before by +adding a ``label_experiment`` method. This method is required by Pyomo.DoE to identify +the design variables (experimental inputs), measurements, measurement errors, and +unknown parameters in the model. + +.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py + :start-after: End model finalization + :end-before: End model labeling + +Step 4: Implement the ``get_labeled_model`` method +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This method utilizes the previous 3 steps and is used by `Pyomo.DoE` to build the model +to perform optimal experimental design. + +.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py + :start-after: End constructor definition + :end-before: Create flexible model without data + +Step 5: Exploratory analysis (Enumeration) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +After creating the subclass of the ``Experiment`` class, exploratory analysis is +suggested to enumerate the design space to check if the problem is identifiable, +i.e., ensure that D-, E-optimality metrics are not small numbers near zero, and +Modified E-optimality is not a big number. +Additionally, it helps to initialize the model for the optimal experimental design step. + +Pyomo.DoE can perform exploratory sensitivity analysis with the ``compute_FIM_full_factorial`` method. +The ``compute_FIM_full_factorial`` method generates a grid over the design space as specified by the user. +Each grid point represents an MBDoE problem solved using the ``compute_FIM`` method. +In this way, sensitivity of the FIM over the design space can be evaluated. +Pyomo.DoE supports plotting the results from the ``compute_FIM_full_factorial`` method +with the ``draw_factorial_figure`` method. + +The following code defines the ``run_reactor_doe`` function. This function encapsulates +the workflow for both sensitivity analysis (Step 5) and optimal design (Step 6). + +.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_example.py + :language: python + :start-after: # === Required imports === + :end-before: if __name__ == "__main__": + +After defining the function, we will call it to perform the exploratory analysis and +the optimal experimental design. + +.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_example.py + :language: python + :start-after: if __name__ == "__main__": + :dedent: 4 + +A design exploration for the initial concentration and temperature as experimental +design variables with 9 values for each, produces the the five figures for +five optimality criteria using the ``compute_FIM_full_factorial`` and +``draw_factorial_figure`` methods as shown below: + +|plot1| |plot2| + +|plot3| |plot4| + +|plot5| + +.. |plot1| image:: example_reactor_compute_FIM_D_opt.png + :width: 48 % + +.. |plot2| image:: example_reactor_compute_FIM_A_opt.png + :width: 48 % + +.. |plot3| image:: example_reactor_compute_FIM_pseudo_A_opt.png + :width: 48 % + +.. |plot4| image:: example_reactor_compute_FIM_E_opt.png + :width: 48 % + +.. |plot5| image:: example_reactor_compute_FIM_ME_opt.png + :width: 48 % + +The heatmaps show the values of the objective functions, a.k.a. the +experimental information content, in the design space. Horizontal +and vertical axes are the two experimental design variables, while +the color of each grid shows the experimental information content. +For example, the D-optimality (upper left subplot) heatmap figure shows that the +most informative region is around :math:`C_{A0}=5.0` M, :math:`T=500.0` K with +a :math:`\log_{10}` determinant of FIM being around 19, +while the least informative region is around :math:`C_{A0}=1.0` M, :math:`T=300.0` K, +with a :math:`\log_{10}` determinant of FIM being around -5. For D-, Pseudo A-, and +E-optimality we want to maximize the objective function, while for A- and Modified +E-optimality we want to minimize the objective function. + +In this sensitivity analysis plot (heatmap), we only varied the initial +concentration and the initial temperature, while the temperature at other time +points is fixed at 300 K. + +.. math:: + :nowrap: + + \[ + T(t) = \begin{cases} + T_0, & t \le 0.125 \\ + 300\ \text{K}, & t > 0.125 + \end{cases} + \] + +If :math:`T_0 = 300\ \text{K}`, the reaction is conducted under strictly isothermal +conditions. Because the temperature is constant, the sensitivities of the species +concentrations with respect to the Arrhenius parameters (:math:`A_i` and :math:`E_i`) +become linearly dependent. This high correlation means the effects of the +pre-exponential factor and the activation energy cannot be uniquely distinguished +from the measurements. Consequently, the Fisher Information Matrix (FIM) becomes +ill-conditioned, resulting in a near-zero determinant and a very large condition number. + +To break this correlation and make the parameters identifiable, introducing a time- +varying temperature profile (for example, a temperature step or a ramp) is required. +As shown in the heatmap, when the initial temperature :math:`T_0` differs from the +subsequent 300 K baseline, such a temperature change breaks the linear dependence, +yielding a well-conditioned FIM and identifiable parameters. + + + +Step 6: Performing an optimal experimental design +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In Step 5, we defined the ``run_reactor_doe`` function. This function constructs +the DoE object and performs the exploratory sensitivity analysis. The way the function +is defined, it also proceeds immediately to the optimal experimental design step +(applying ``run_doe`` on the ``DesignOfExperiments`` object). +We can initialize the model with the result we obtained from the exploratory +analysis (optimal point from the heatmaps) to help the optimal design step to speed +up convergence. However, implementation of this initialization is not shown here. + +After applying ``run_doe`` on the ``DesignOfExperiments`` object, +the optimal design is an initial concentration of 5.0 mol/L and +an initial temperature of 494 K with all other temperatures being 300 K. +The corresponding :math:`\log_{10}` determinant of the FIM is 19.32. diff --git a/doc/OnlineDocs/explanation/analysis/doe/multexp.rst b/doc/OnlineDocs/explanation/analysis/doe/multexp.rst new file mode 100644 index 00000000000..0c943b7e010 --- /dev/null +++ b/doc/OnlineDocs/explanation/analysis/doe/multexp.rst @@ -0,0 +1,8 @@ +.. _multexperiments: + +Multiple Experiments +==================== + +.. note:: + + Detailed descriptions and example code for simultaneous design of multiple experiments will be added in a future update. \ No newline at end of file diff --git a/doc/OnlineDocs/explanation/analysis/doe/objective.rst b/doc/OnlineDocs/explanation/analysis/doe/objective.rst new file mode 100644 index 00000000000..28bd39972e9 --- /dev/null +++ b/doc/OnlineDocs/explanation/analysis/doe/objective.rst @@ -0,0 +1,8 @@ +.. _objectives: + +Objective Options +================= + +.. note:: + + Detailed descriptions and example code for the objective options in Pyomo.DoE will be added in a future update. \ No newline at end of file diff --git a/doc/OnlineDocs/explanation/analysis/doe/overview.rst b/doc/OnlineDocs/explanation/analysis/doe/overview.rst new file mode 100644 index 00000000000..2c25f5d02fc --- /dev/null +++ b/doc/OnlineDocs/explanation/analysis/doe/overview.rst @@ -0,0 +1,131 @@ +Overview +======== + +Model-based Design of Experiments (MBDoE) is a technique to maximize the information +gain from experiments by directly using science-based models with physically meaningful +parameters. It is one key component in the model calibration and uncertainty +quantification workflow shown below: + +.. figure:: pyomo_workflow_new.png + :align: center + :scale: 90 % + +The parameter estimation, uncertainty analysis, and MBDoE are +combined into an iterative framework to select, refine, and calibrate science-based +mathematical models with quantified uncertainty. Currently, Pyomo.DoE focuses on +increasing parameter precision. + +Pyomo.DoE provides the exploratory analysis and MBDoE capabilities to the +Pyomo ecosystem. The user provides one Pyomo model, a set of parameter nominal values, +the allowable design spaces for design variables, and the assumed observation error model. +During exploratory analysis, Pyomo.DoE checks if the model parameters can be +inferred from the postulated measurements or preliminary data. +MBDoE then recommends optimized experimental conditions for collecting more data. +Parameter estimation packages such as :ref:`Parmest ` can perform +parameter estimation using the available data to infer values for parameters, +and facilitate an uncertainty analysis to approximate the parameter covariance matrix. +If the parameter uncertainties are sufficiently small, the workflow terminates +and returns the final model with quantified parametric uncertainty. +If not, MBDoE recommends optimized experimental conditions to generate new data +that will maximize information gain and eventually reduce parameter uncertainty. + +Below is an overview of the type of optimization models Pyomo.DoE can accommodate: + +* Pyomo.DoE is suitable for optimization models of **continuous** variables +* Pyomo.DoE can handle **equality constraints** defining state variables +* Pyomo.DoE supports (Partial) Differential-Algebraic Equations (PDAE) models via :ref:`Pyomo.DAE ` +* Pyomo.DoE also supports models with only algebraic equations + +The general form of a DAE problem that can be passed into Pyomo.DoE is shown below: + +.. math:: + :nowrap: + + \[\begin{array}{l} + \dot{\mathbf{x}}(t) = \mathbf{f}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{y}(t), \mathbf{u}(t), \overline{\mathbf{w}}, \boldsymbol{\theta}) \\ + \mathbf{g}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{y}(t), \mathbf{u}(t), \overline{\mathbf{w}},\boldsymbol{\theta})=\mathbf{0} \\ + \mathbf{y} =\mathbf{h}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{u}(t), \overline{\mathbf{w}},\boldsymbol{\theta}) \\ + \mathbf{f}^{\mathbf{0}}\left(\dot{\mathbf{x}}\left(t_{0}\right), \mathbf{x}\left(t_{0}\right), \mathbf{z}(t_0), \mathbf{y}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\theta}\right)=\mathbf{0} \\ + \mathbf{g}^{\mathbf{0}}\left( \mathbf{x}\left(t_{0}\right),\mathbf{z}(t_0), \mathbf{y}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\theta}\right)=\mathbf{0}\\ + \mathbf{y}^{\mathbf{0}}\left(t_{0}\right)=\mathbf{h}\left(\mathbf{x}\left(t_{0}\right),\mathbf{z}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\theta}\right) + \end{array}\] + +where: + +* :math:`\boldsymbol{\theta} \in \mathbb{R}^{N_p}` are unknown model parameters. +* :math:`\mathbf{x} \subseteq \mathcal{X}` are dynamic state variables which characterize trajectory of the system, :math:`\mathcal{X} \in \mathbb{R}^{N_x \times N_t}`. +* :math:`\mathbf{z} \subseteq \mathcal{Z}` are algebraic state variables, :math:`\mathcal{Z} \in \mathbb{R}^{N_z \times N_t}`. +* :math:`\mathbf{u} \subseteq \mathcal{U}` are time-varying decision variables, :math:`\mathcal{U} \in \mathbb{R}^{N_u \times N_t}`. +* :math:`\overline{\mathbf{w}} \in \mathbb{R}^{N_w}` are time-invariant decision variables. +* :math:`\mathbf{y} \subseteq \mathcal{Y}` are measurement response variables, :math:`\mathcal{Y} \in \mathbb{R}^{N_r \times N_t}`. +* :math:`\mathbf{f}(\cdot)` are differential equations. +* :math:`\mathbf{g}(\cdot)` are algebraic equations. +* :math:`\mathbf{h}(\cdot)` are measurement functions. +* :math:`\mathbf{t} \in \mathbb{R}^{N_t \times 1}` is a union of all time sets. + +.. note:: + + Parameters and design variables should be defined as Pyomo ``Var`` components + when building the model using the ``Experiment`` class so that users can use both + ``Parmest`` and ``Pyomo.DoE`` seamlessly. + +Based on the above notation, the form of the MBDoE problem addressed in Pyomo.DoE is shown below: + +.. math:: + :nowrap: + + \begin{equation} + \begin{aligned} + \underset{\boldsymbol{\varphi}}{\max} \quad & \Psi (\mathbf{M}(\boldsymbol{\hat{\theta}}, \boldsymbol{\varphi})) \\ + \text{s.t.} \quad & \mathbf{M}(\boldsymbol{\hat{\theta}}, \boldsymbol{\varphi}) = \sum_r^{N_r} \sum_{r'}^{N_r} \tilde{\sigma}_{(r,r')}\mathbf{Q}_r^\mathbf{T} \mathbf{Q}_{r'} + \mathbf{M}_0 \\ + & \dot{\mathbf{x}}(t) = \mathbf{f}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{y}(t), \mathbf{u}(t), \overline{\mathbf{w}}, \boldsymbol{\hat{\theta}}) \\ + & \mathbf{g}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{y}(t), \mathbf{u}(t), \overline{\mathbf{w}},\boldsymbol{\hat{\theta}})=\mathbf{0} \\ + & \mathbf{y} =\mathbf{h}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{u}(t), \overline{\mathbf{w}},\boldsymbol{\hat{\theta}}) \\ + & \mathbf{f}^{\mathbf{0}}\left(\dot{\mathbf{x}}\left(t_{0}\right), \mathbf{x}\left(t_{0}\right), \mathbf{z}(t_0), \mathbf{y}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\hat{\theta}})\right)=\mathbf{0} \\ + & \mathbf{g}^{\mathbf{0}}\left( \mathbf{x}\left(t_{0}\right),\mathbf{z}(t_0), \mathbf{y}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\hat{\theta}}\right)=\mathbf{0}\\ + &\mathbf{y}^{\mathbf{0}}\left(t_{0}\right)=\mathbf{h}\left(\mathbf{x}\left(t_{0}\right),\mathbf{z}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\hat{\theta}}\right) + \end{aligned} + \end{equation} + +where: + +* :math:`\boldsymbol{\varphi}` are design variables, which are manipulated to maximize the information content of experiments. It should consist of one or more of :math:`\mathbf{u}(t), \mathbf{y}^{\mathbf{0}}({t_0}),\overline{\mathbf{w}}`. With a proper model formulation, the timepoints for control or measurements :math:`\mathbf{t}` can also be degrees of freedom. +* :math:`\mathbf{M}` is the Fisher information matrix (FIM), approximated as the inverse of the covariance matrix of parameter estimates :math:`\boldsymbol{\hat{\theta}}`. A large FIM indicates more information contained in the experiment for parameter estimation. +* :math:`\mathbf{Q}` is the dynamic sensitivity matrix, containing the partial derivatives of :math:`\mathbf{y}` with respect to :math:`\boldsymbol{\theta}`. +* :math:`\Psi` is the scalar design criteria to measure the information content in the FIM. +* :math:`\mathbf{M}_0` is the sum of all the FIMs from previous experiments. + +Pyomo.DoE provides five design criteria :math:`\Psi` to measure the information in the FIM. +The covariance matrix of parameter estimates is approximated as the inverse of the FIM, +i.e., :math:`\mathbf{V} \approx \mathbf{M}^{-1}`. +We can use the FIM or the covariance matrix to define the design criteria. + +.. list-table:: Pyomo.DoE design criteria + :header-rows: 1 + :class: tight-table + + * - Design criterion + - Computation + - Geometrical meaning + * - A-optimality + - :math:`\text{trace}(\mathbf{V}) = \text{trace}(\mathbf{M}^{-1})` + - Minimizing this is equivalent to minimizing the enclosing box of the confidence ellipse + * - Pseudo A-optimality + - :math:`\text{trace}(\mathbf{M})` + - Maximizing this is equivalent to maximizing the dimensions of the enclosing box of the Fisher Information Matrix + * - D-optimality + - :math:`\det(\mathbf{M}) = 1/\det(\mathbf{V})` + - Maximizing this is equivalent to minimizing confidence-ellipsoid volume + * - E-optimality + - :math:`\lambda_{\min}(\mathbf{M}) = 1/\lambda_{\max}(\mathbf{V})` + - Maximizing this is equivalent to minimizing the longest axis of the confidence ellipse + * - Modified E-optimality + - :math:`\text{cond}(\mathbf{M}) = \text{cond}(\mathbf{V})` + - Minimizing this is equivalent to minimizing the ratio of the longest axis to the shortest axis of the confidence ellipse + +.. note:: + + A confidence ellipse is a geometric representation of the uncertainty in parameter + estimates. It is derived from the covariance matrix :math:`\mathbf{V}`. + +In order to solve problems of the above, Pyomo.DoE implements the 2-stage stochastic program. Please see Wang and Dowling (2022) for details. \ No newline at end of file diff --git a/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst b/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst new file mode 100644 index 00000000000..50b31dbe040 --- /dev/null +++ b/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst @@ -0,0 +1,10 @@ +.. _paramuncertainty: + +Parameter Uncertainty +===================== + +.. note:: + + Detailed descriptions and example code for experiment design under parameter uncertainty will be added in a + future update. + diff --git a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst index 29c9d91f47e..971c411c358 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst @@ -185,7 +185,7 @@ can be computed without relying on strong distributional assumptions. This metho .. note:: - The example code for this method will soon be provided. + The example code for this method will soon be provided. Likelihood Ratio Test --------------------- @@ -223,11 +223,11 @@ to assess whether restricting parameters significantly degrades model fit. This .. note:: - The example code for this method will soon be provided. + The example code for this method will soon be provided. Leave-N-Out ----------- .. note:: - Detailed descriptions and example code for this method will be added in a future update. \ No newline at end of file + Detailed descriptions and example code for this method will be added in a future update. \ No newline at end of file diff --git a/doc/OnlineDocs/explanation/analysis/parmest/objective.rst b/doc/OnlineDocs/explanation/analysis/parmest/objective.rst index 164c5a5c0ff..ccce0eb8ac2 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/objective.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/objective.rst @@ -2,6 +2,7 @@ Objective Options ================= + .. note:: - Detailed descriptions and example code for the objective options in parmest will be added in a future update. \ No newline at end of file + Detailed descriptions and example code for the objective options in parmest will be added in a future update. \ No newline at end of file diff --git a/doc/OnlineDocs/explanation/analysis/parmest/overview.rst b/doc/OnlineDocs/explanation/analysis/parmest/overview.rst index 7f6f23c9140..7a3f7538fde 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/overview.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/overview.rst @@ -12,10 +12,12 @@ Functionality in parmest includes: * Model-based parameter estimation using experimental data * Covariance matrix estimation -* Bootstrap resampling for parameter estimation +* Bootstrap resampling for uncertainty quantification * Confidence regions based on single or multi-variate distributions -* Likelihood ratio +* Likelihood ratio test * Leave-N-out cross validation +* Regularization for objective function improvement +* Multi-start initialization optimization * Parallel processing Background From dbe9f2caf498ad17873bbbe1f980c0ff7bc3c4e9 Mon Sep 17 00:00:00 2001 From: slilonfe5 Date: Tue, 7 Apr 2026 14:30:02 -0400 Subject: [PATCH 04/19] Added abstraction.rst and updated doe.rst --- .../explanation/analysis/doe/abstraction.rst | 8 + .../explanation/analysis/doe/doe.rst | 372 +----------------- 2 files changed, 20 insertions(+), 360 deletions(-) create mode 100644 doc/OnlineDocs/explanation/analysis/doe/abstraction.rst diff --git a/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst b/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst new file mode 100644 index 00000000000..4aabe3a9f3a --- /dev/null +++ b/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst @@ -0,0 +1,8 @@ +.. _abstractexp: + +Experiment Abstraction +====================== + +.. note:: + + Detailed descriptions and example code for experiment abstraction in Pyomo.DoE will be added in a future update. diff --git a/doc/OnlineDocs/explanation/analysis/doe/doe.rst b/doc/OnlineDocs/explanation/analysis/doe/doe.rst index 77fc8b1a08a..7cefb99e418 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/doe.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/doe.rst @@ -21,363 +21,15 @@ If you use Pyomo.DoE, please cite: "Pyomo.DOE: An open‐source package for model‐based design of experiments in Python." AIChE Journal 68.12 (2022): e17813. `https://doi.org/10.1002/aic.17813` -Methodology Overview ---------------------- - -Model-based Design of Experiments (MBDoE) is a technique to maximize the information -gain from experiments by directly using science-based models with physically meaningful -parameters. It is one key component in the model calibration and uncertainty -quantification workflow shown below: - -.. figure:: pyomo_workflow_new.png - :align: center - :scale: 90 % - -The parameter estimation, uncertainty analysis, and MBDoE are -combined into an iterative framework to select, refine, and calibrate science-based -mathematical models with quantified uncertainty. Currently, Pyomo.DoE focuses on -increasing parameter precision. - -Pyomo.DoE provides the exploratory analysis and MBDoE capabilities to the -Pyomo ecosystem. The user provides one Pyomo model, a set of parameter nominal values, -the allowable design spaces for design variables, and the assumed observation error model. -During exploratory analysis, Pyomo.DoE checks if the model parameters can be -inferred from the postulated measurements or preliminary data. -MBDoE then recommends optimized experimental conditions for collecting more data. -Parameter estimation packages such as :ref:`Parmest ` can perform -parameter estimation using the available data to infer values for parameters, -and facilitate an uncertainty analysis to approximate the parameter covariance matrix. -If the parameter uncertainties are sufficiently small, the workflow terminates -and returns the final model with quantified parametric uncertainty. -If not, MBDoE recommends optimized experimental conditions to generate new data -that will maximize information gain and eventually reduce parameter uncertainty. - -Below is an overview of the type of optimization models Pyomo.DoE can accommodate: - -* Pyomo.DoE is suitable for optimization models of **continuous** variables -* Pyomo.DoE can handle **equality constraints** defining state variables -* Pyomo.DoE supports (Partial) Differential-Algebraic Equations (PDAE) models via :ref:`Pyomo.DAE ` -* Pyomo.DoE also supports models with only algebraic equations - -The general form of a DAE problem that can be passed into Pyomo.DoE is shown below: - -.. math:: - :nowrap: - - \[\begin{array}{l} - \dot{\mathbf{x}}(t) = \mathbf{f}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{y}(t), \mathbf{u}(t), \overline{\mathbf{w}}, \boldsymbol{\theta}) \\ - \mathbf{g}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{y}(t), \mathbf{u}(t), \overline{\mathbf{w}},\boldsymbol{\theta})=\mathbf{0} \\ - \mathbf{y} =\mathbf{h}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{u}(t), \overline{\mathbf{w}},\boldsymbol{\theta}) \\ - \mathbf{f}^{\mathbf{0}}\left(\dot{\mathbf{x}}\left(t_{0}\right), \mathbf{x}\left(t_{0}\right), \mathbf{z}(t_0), \mathbf{y}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\theta}\right)=\mathbf{0} \\ - \mathbf{g}^{\mathbf{0}}\left( \mathbf{x}\left(t_{0}\right),\mathbf{z}(t_0), \mathbf{y}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\theta}\right)=\mathbf{0}\\ - \mathbf{y}^{\mathbf{0}}\left(t_{0}\right)=\mathbf{h}\left(\mathbf{x}\left(t_{0}\right),\mathbf{z}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\theta}\right) - \end{array}\] - -where: - -* :math:`\boldsymbol{\theta} \in \mathbb{R}^{N_p}` are unknown model parameters. -* :math:`\mathbf{x} \subseteq \mathcal{X}` are dynamic state variables which characterize trajectory of the system, :math:`\mathcal{X} \in \mathbb{R}^{N_x \times N_t}`. -* :math:`\mathbf{z} \subseteq \mathcal{Z}` are algebraic state variables, :math:`\mathcal{Z} \in \mathbb{R}^{N_z \times N_t}`. -* :math:`\mathbf{u} \subseteq \mathcal{U}` are time-varying decision variables, :math:`\mathcal{U} \in \mathbb{R}^{N_u \times N_t}`. -* :math:`\overline{\mathbf{w}} \in \mathbb{R}^{N_w}` are time-invariant decision variables. -* :math:`\mathbf{y} \subseteq \mathcal{Y}` are measurement response variables, :math:`\mathcal{Y} \in \mathbb{R}^{N_r \times N_t}`. -* :math:`\mathbf{f}(\cdot)` are differential equations. -* :math:`\mathbf{g}(\cdot)` are algebraic equations. -* :math:`\mathbf{h}(\cdot)` are measurement functions. -* :math:`\mathbf{t} \in \mathbb{R}^{N_t \times 1}` is a union of all time sets. - -.. note:: - Parameters and design variables should be defined as Pyomo ``Var`` components - when building the model using the ``Experiment`` class so that users can use both - ``Parmest`` and ``Pyomo.DoE`` seamlessly. - -Based on the above notation, the form of the MBDoE problem addressed in Pyomo.DoE is shown below: - -.. math:: - :nowrap: - - \begin{equation} - \begin{aligned} - \underset{\boldsymbol{\varphi}}{\max} \quad & \Psi (\mathbf{M}(\boldsymbol{\hat{\theta}}, \boldsymbol{\varphi})) \\ - \text{s.t.} \quad & \mathbf{M}(\boldsymbol{\hat{\theta}}, \boldsymbol{\varphi}) = \sum_r^{N_r} \sum_{r'}^{N_r} \tilde{\sigma}_{(r,r')}\mathbf{Q}_r^\mathbf{T} \mathbf{Q}_{r'} + \mathbf{M}_0 \\ - & \dot{\mathbf{x}}(t) = \mathbf{f}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{y}(t), \mathbf{u}(t), \overline{\mathbf{w}}, \boldsymbol{\hat{\theta}}) \\ - & \mathbf{g}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{y}(t), \mathbf{u}(t), \overline{\mathbf{w}},\boldsymbol{\hat{\theta}})=\mathbf{0} \\ - & \mathbf{y} =\mathbf{h}(\mathbf{x}(t), \mathbf{z}(t), \mathbf{u}(t), \overline{\mathbf{w}},\boldsymbol{\hat{\theta}}) \\ - & \mathbf{f}^{\mathbf{0}}\left(\dot{\mathbf{x}}\left(t_{0}\right), \mathbf{x}\left(t_{0}\right), \mathbf{z}(t_0), \mathbf{y}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\hat{\theta}})\right)=\mathbf{0} \\ - & \mathbf{g}^{\mathbf{0}}\left( \mathbf{x}\left(t_{0}\right),\mathbf{z}(t_0), \mathbf{y}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\hat{\theta}}\right)=\mathbf{0}\\ - &\mathbf{y}^{\mathbf{0}}\left(t_{0}\right)=\mathbf{h}\left(\mathbf{x}\left(t_{0}\right),\mathbf{z}(t_0), \mathbf{u}\left(t_{0}\right), \overline{\mathbf{w}}, \boldsymbol{\hat{\theta}}\right) - \end{aligned} - \end{equation} - -where: - -* :math:`\boldsymbol{\varphi}` are design variables, which are manipulated to maximize the information content of experiments. It should consist of one or more of :math:`\mathbf{u}(t), \mathbf{y}^{\mathbf{0}}({t_0}),\overline{\mathbf{w}}`. With a proper model formulation, the timepoints for control or measurements :math:`\mathbf{t}` can also be degrees of freedom. -* :math:`\mathbf{M}` is the Fisher information matrix (FIM), approximated as the inverse of the covariance matrix of parameter estimates :math:`\boldsymbol{\hat{\theta}}`. A large FIM indicates more information contained in the experiment for parameter estimation. -* :math:`\mathbf{Q}` is the dynamic sensitivity matrix, containing the partial derivatives of :math:`\mathbf{y}` with respect to :math:`\boldsymbol{\theta}`. -* :math:`\Psi` is the scalar design criteria to measure the information content in the FIM. -* :math:`\mathbf{M}_0` is the sum of all the FIMs from previous experiments. - -Pyomo.DoE provides five design criteria :math:`\Psi` to measure the information in the FIM. -The covariance matrix of parameter estimates is approximated as the inverse of the FIM, -i.e., :math:`\mathbf{V} \approx \mathbf{M}^{-1}`. -We can use the FIM or the covariance matrix to define the design criteria. - -.. list-table:: Pyomo.DoE design criteria - :header-rows: 1 - :class: tight-table - - * - Design criterion - - Computation - - Geometrical meaning - * - A-optimality - - :math:`\text{trace}(\mathbf{V}) = \text{trace}(\mathbf{M}^{-1})` - - Minimizing this is equivalent to minimizing the enclosing box of the confidence ellipse - * - Pseudo A-optimality - - :math:`\text{trace}(\mathbf{M})` - - Maximizing this is equivalent to maximizing the dimensions of the enclosing box of the Fisher Information Matrix - * - D-optimality - - :math:`\det(\mathbf{M}) = 1/\det(\mathbf{V})` - - Maximizing this is equivalent to minimizing confidence-ellipsoid volume - * - E-optimality - - :math:`\lambda_{\min}(\mathbf{M}) = 1/\lambda_{\max}(\mathbf{V})` - - Maximizing this is equivalent to minimizing the longest axis of the confidence ellipse - * - Modified E-optimality - - :math:`\text{cond}(\mathbf{M}) = \text{cond}(\mathbf{V})` - - Minimizing this is equivalent to minimizing the ratio of the longest axis to the shortest axis of the confidence ellipse - -.. note:: - A confidence ellipse is a geometric representation of the uncertainty in parameter - estimates. It is derived from the covariance matrix :math:`\mathbf{V}`. - -In order to solve problems of the above, Pyomo.DoE implements the 2-stage stochastic program. Please see Wang and Dowling (2022) for details. - -Pyomo.DoE Required Inputs -------------------------- -To use Pyomo.DoE, a user must implement a subclass of the :ref:`Parmest ` ``Experiment`` class. -The subclass must have a ``get_labeled_model`` method which returns a Pyomo `ConcreteModel` -containing four Pyomo ``Suffix`` components identifying the parts of the model used in -MBDoE analysis. This is in line with the convention used in the parameter estimation tool, -:ref:`Parmest `. The four Pyomo ``Suffix`` components are: - -* ``experiment_inputs`` - The experimental design decisions -* ``experiment_outputs`` - The values measured during the experiment -* ``measurement_error`` - The error associated with individual values measured during the experiment. It is passed as a standard deviation or square root of the diagonal elements of the observation error covariance matrix. Pyomo.DoE currently assumes that the observation errors are Gaussain and independent both in time and across measurements. -* ``unknown_parameters`` - Those parameters in the model that are estimated using the measured values during the experiment - -An example of the subclassed ``Experiment`` object that builds and labels the model is shown in the next few sections. - -Pyomo.DoE Usage Example ------------------------ - -We illustrate the use of Pyomo.DoE using a reaction kinetics example (Wang and Dowling, 2022). - -.. math:: - :nowrap: - - \begin{equation} - A \xrightarrow{k_1} B \xrightarrow{k_2} C - \end{equation} - - -The Arrhenius equations model the temperature dependence of the reaction rate coefficients :math:`k_1` and :math:`k_2`. Assuming a first-order reaction mechanism gives the reaction rate model shown below. Further, we assume only species A is fed to the reactor. - -.. math:: - :nowrap: - - \begin{equation} - \begin{aligned} - k_1 & = A_1 e^{-\frac{E_1}{RT}} \\ - k_2 & = A_2 e^{-\frac{E_2}{RT}} \\ - \frac{d{C_A}}{dt} & = -k_1{C_A} \\ - \frac{d{C_B}}{dt} & = k_1{C_A} - k_2{C_B} \\ - C_{A0}& = C_A + C_B + C_C \\ - C_B(t_0) & = 0 \\ - C_C(t_0) & = 0 \\ - \end{aligned} - \end{equation} - - - -:math:`C_A(t), C_B(t), C_C(t)` are the time-varying concentrations of the species A, B, C, respectively. -:math:`k_1, k_2` are the rate constants for the two chemical reactions using an Arrhenius equation with activation energies :math:`E_1, E_2` and pre-exponential factors :math:`A_1, A_2`. -The goal of MBDoE is to optimize the experiment design variables :math:`\boldsymbol{\varphi} = (C_{A0}, T(t))`, where :math:`C_{A0},T(t)` are the initial concentration of species A and the time-varying reactor temperature, to maximize the precision of unknown model parameters :math:`\boldsymbol{\theta} = (A_1, E_1, A_2, E_2)` by measuring :math:`\mathbf{y}(t)=(C_A(t), C_B(t), C_C(t))`. -The observation errors are assumed to be independent both in time and across measurements with a constant standard deviation of 1 M for each species. - - -Step 0: Import Pyomo and the Pyomo.DoE module and create an ``Experiment`` class -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. note:: - - This example uses the data file ``result.json``, located in the Pyomo repository at: - ``pyomo/contrib/doe/examples/result.json``, which contains the nominal parameter - values, and measurements for the reaction kinetics experiment. - - -.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py - :start-after: # === Required imports === - :end-before: # ======================== - -Subclass the :ref:`Parmest ` ``Experiment`` class to define the reaction -kinetics experiment and build the Pyomo ConcreteModel. - -.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py - :start-after: ======================== - :end-before: End constructor definition - -Step 1: Define the Pyomo process model -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The process model for the reaction kinetics problem is shown below. Here, we build -the model without any data or discretization. - -.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py - :start-after: Create flexible model without data - :end-before: End equation definition - -Step 2: Finalize the Pyomo process model -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Here, we add data to the model and finalize the discretization using a new method to -the class. This step is required before the model can be labeled. - -.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py - :start-after: End equation definition - :end-before: End model finalization - -Step 3: Label the information needed for DoE analysis -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -We label the four important groups as Pyomo Suffix components as mentioned before by -adding a ``label_experiment`` method. This method is required by Pyomo.DoE to identify -the design variables (experimental inputs), measurements, measurement errors, and -unknown parameters in the model. - -.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py - :start-after: End model finalization - :end-before: End model labeling - -Step 4: Implement the ``get_labeled_model`` method -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This method utilizes the previous 3 steps and is used by `Pyomo.DoE` to build the model -to perform optimal experimental design. - -.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py - :start-after: End constructor definition - :end-before: Create flexible model without data - -Step 5: Exploratory analysis (Enumeration) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -After creating the subclass of the ``Experiment`` class, exploratory analysis is -suggested to enumerate the design space to check if the problem is identifiable, -i.e., ensure that D-, E-optimality metrics are not small numbers near zero, and -Modified E-optimality is not a big number. -Additionally, it helps to initialize the model for the optimal experimental design step. - -Pyomo.DoE can perform exploratory sensitivity analysis with the ``compute_FIM_full_factorial`` method. -The ``compute_FIM_full_factorial`` method generates a grid over the design space as specified by the user. -Each grid point represents an MBDoE problem solved using the ``compute_FIM`` method. -In this way, sensitivity of the FIM over the design space can be evaluated. -Pyomo.DoE supports plotting the results from the ``compute_FIM_full_factorial`` method -with the ``draw_factorial_figure`` method. - -The following code defines the ``run_reactor_doe`` function. This function encapsulates -the workflow for both sensitivity analysis (Step 5) and optimal design (Step 6). - -.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_example.py - :language: python - :start-after: # === Required imports === - :end-before: if __name__ == "__main__": - -After defining the function, we will call it to perform the exploratory analysis and -the optimal experimental design. - -.. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_example.py - :language: python - :start-after: if __name__ == "__main__": - :dedent: 4 - -A design exploration for the initial concentration and temperature as experimental -design variables with 9 values for each, produces the the five figures for -five optimality criteria using the ``compute_FIM_full_factorial`` and -``draw_factorial_figure`` methods as shown below: - -|plot1| |plot2| - -|plot3| |plot4| - -|plot5| - -.. |plot1| image:: example_reactor_compute_FIM_D_opt.png - :width: 48 % - -.. |plot2| image:: example_reactor_compute_FIM_A_opt.png - :width: 48 % - -.. |plot3| image:: example_reactor_compute_FIM_pseudo_A_opt.png - :width: 48 % - -.. |plot4| image:: example_reactor_compute_FIM_E_opt.png - :width: 48 % - -.. |plot5| image:: example_reactor_compute_FIM_ME_opt.png - :width: 48 % - -The heatmaps show the values of the objective functions, a.k.a. the -experimental information content, in the design space. Horizontal -and vertical axes are the two experimental design variables, while -the color of each grid shows the experimental information content. -For example, the D-optimality (upper left subplot) heatmap figure shows that the -most informative region is around :math:`C_{A0}=5.0` M, :math:`T=500.0` K with -a :math:`\log_{10}` determinant of FIM being around 19, -while the least informative region is around :math:`C_{A0}=1.0` M, :math:`T=300.0` K, -with a :math:`\log_{10}` determinant of FIM being around -5. For D-, Pseudo A-, and -E-optimality we want to maximize the objective function, while for A- and Modified -E-optimality we want to minimize the objective function. - -In this sensitivity analysis plot (heatmap), we only varied the initial -concentration and the initial temperature, while the temperature at other time -points is fixed at 300 K. - -.. math:: - :nowrap: - - \[ - T(t) = \begin{cases} - T_0, & t \le 0.125 \\ - 300\ \text{K}, & t > 0.125 - \end{cases} - \] - -If :math:`T_0 = 300\ \text{K}`, the reaction is conducted under strictly isothermal -conditions. Because the temperature is constant, the sensitivities of the species -concentrations with respect to the Arrhenius parameters (:math:`A_i` and :math:`E_i`) -become linearly dependent. This high correlation means the effects of the -pre-exponential factor and the activation energy cannot be uniquely distinguished -from the measurements. Consequently, the Fisher Information Matrix (FIM) becomes -ill-conditioned, resulting in a near-zero determinant and a very large condition number. - -To break this correlation and make the parameters identifiable, introducing a time- -varying temperature profile (for example, a temperature step or a ramp) is required. -As shown in the heatmap, when the initial temperature :math:`T_0` differs from the -subsequent 300 K baseline, such a temperature change breaks the linear dependence, -yielding a well-conditioned FIM and identifiable parameters. - - - -Step 6: Performing an optimal experimental design -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In Step 5, we defined the ``run_reactor_doe`` function. This function constructs -the DoE object and performs the exploratory sensitivity analysis. The way the function -is defined, it also proceeds immediately to the optimal experimental design step -(applying ``run_doe`` on the ``DesignOfExperiments`` object). -We can initialize the model with the result we obtained from the exploratory -analysis (optimal point from the heatmaps) to help the optimal design step to speed -up convergence. However, implementation of this initialization is not shown here. - -After applying ``run_doe`` on the ``DesignOfExperiments`` object, -the optimal design is an initial concentration of 5.0 mol/L and -an initial temperature of 494 K with all other temperatures being 300 K. -The corresponding :math:`\log_{10}` determinant of the FIM is 19.32. +Index of Pyomo.DoE documentation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. toctree:: + :maxdepth: 2 + + overview.rst + guide.rst + abstraction.rst + objective.rst + multexp.rst + uncertainty.rst \ No newline at end of file From 46aba836bf77a025e626c5d028ee496d80c131fd Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:39:20 -0400 Subject: [PATCH 05/19] Update doc/OnlineDocs/explanation/analysis/doe/overview.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/doe/overview.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/overview.rst b/doc/OnlineDocs/explanation/analysis/doe/overview.rst index 2c25f5d02fc..0a7c4a0b1a4 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/overview.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/overview.rst @@ -6,9 +6,9 @@ gain from experiments by directly using science-based models with physically mea parameters. It is one key component in the model calibration and uncertainty quantification workflow shown below: -.. figure:: pyomo_workflow_new.png +.. image:: pyomo_workflow_new.png :align: center - :scale: 90 % + :width: 90 % The parameter estimation, uncertainty analysis, and MBDoE are combined into an iterative framework to select, refine, and calibrate science-based From 0dccc5b6afe05cbae743c5343f2c7aa1fd3fc31d Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:40:00 -0400 Subject: [PATCH 06/19] Update doc/OnlineDocs/explanation/analysis/doe/guide.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/doe/guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index cb14c08d6e0..cf540f0942f 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -4,7 +4,7 @@ Quick Start Guide ================= To use Pyomo.DoE, a user must implement a subclass of the :ref:`Parmest ` ``Experiment`` class. -The subclass must have a ``get_labeled_model`` method which returns a Pyomo `ConcreteModel` +The subclass must have a ``get_labeled_model`` method which returns a Pyomo :class:`ConcreteModel` containing four Pyomo ``Suffix`` components identifying the parts of the model used in MBDoE analysis. This is in line with the convention used in the parameter estimation tool, :ref:`Parmest `. The four Pyomo ``Suffix`` components are: From 459932e87947fce3e3669dae6771a37505e4657a Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:40:17 -0400 Subject: [PATCH 07/19] Update doc/OnlineDocs/explanation/analysis/doe/guide.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/doe/guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index cf540f0942f..fe27e838eb1 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -5,7 +5,7 @@ Quick Start Guide To use Pyomo.DoE, a user must implement a subclass of the :ref:`Parmest ` ``Experiment`` class. The subclass must have a ``get_labeled_model`` method which returns a Pyomo :class:`ConcreteModel` -containing four Pyomo ``Suffix`` components identifying the parts of the model used in +containing four Pyomo :class:`Suffix` components identifying the parts of the model used in MBDoE analysis. This is in line with the convention used in the parameter estimation tool, :ref:`Parmest `. The four Pyomo ``Suffix`` components are: From f6b4364da82ded6df0e56f351d9f044305dc259e Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:40:36 -0400 Subject: [PATCH 08/19] Update doc/OnlineDocs/explanation/analysis/doe/guide.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/doe/guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index fe27e838eb1..0ab09ff4230 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -7,7 +7,7 @@ To use Pyomo.DoE, a user must implement a subclass of the :ref:`Parmest `. The four Pyomo ``Suffix`` components are: +:ref:`Parmest `. The four Pyomo :class:`Suffix` components are: * ``experiment_inputs`` - The experimental design decisions * ``experiment_outputs`` - The values measured during the experiment From b33cd073b145f669be5d51152351255141522411 Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:41:07 -0400 Subject: [PATCH 09/19] Update doc/OnlineDocs/explanation/analysis/doe/guide.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/doe/guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index 0ab09ff4230..d384b099a92 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -3,7 +3,7 @@ Quick Start Guide ================= -To use Pyomo.DoE, a user must implement a subclass of the :ref:`Parmest ` ``Experiment`` class. +To use Pyomo.DoE, a user must implement a subclass of the :ref:`Parmest ` :class:`Experiment` class. The subclass must have a ``get_labeled_model`` method which returns a Pyomo :class:`ConcreteModel` containing four Pyomo :class:`Suffix` components identifying the parts of the model used in MBDoE analysis. This is in line with the convention used in the parameter estimation tool, From 617c49c9cdc5272afa466b9b92b151bd9e0415b7 Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:41:31 -0400 Subject: [PATCH 10/19] Update doc/OnlineDocs/explanation/analysis/doe/guide.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/doe/guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index d384b099a92..1da7952e4fd 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -14,7 +14,7 @@ MBDoE analysis. This is in line with the convention used in the parameter estima * ``measurement_error`` - The error associated with individual values measured during the experiment. It is passed as a standard deviation or square root of the diagonal elements of the observation error covariance matrix. Pyomo.DoE currently assumes that the observation errors are Gaussain and independent both in time and across measurements. * ``unknown_parameters`` - Those parameters in the model that are estimated using the measured values during the experiment -An example of the subclassed ``Experiment`` object that builds and labels the model is shown in the next few sections. +An example of the subclassed :class:`Experiment` object that builds and labels the model is shown in the next few sections. This guide illustrates the use of Pyomo.DoE using a reaction kinetics example (Wang and Dowling, 2022). From 1ae1e010cb971384b453e04e6759491202096f64 Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:42:25 -0400 Subject: [PATCH 11/19] Update doc/OnlineDocs/explanation/analysis/parmest/covariance.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/parmest/covariance.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst index 971c411c358..ff0d07fb133 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst @@ -3,8 +3,9 @@ Uncertainty Quantification ========================== The goal of parameter estimation (see :ref:`driversection` Section) is to estimate unknown model parameters -from experimental data. Uncertainty quantification is required to ensure that the estimates of the parameters are -close to their true values. This parameter uncertainty can be computed using four methods in parmest: +from experimental data. Uncertainty quantification then aims to characterize +how close the parameter estimates are to their true (unknown) values. +This parameter uncertainty can be computed using four methods in parmest: covariance matrix, likelihood ratio test, bootstrapping, and leave-N-out. Covariance Matrix Estimation From b7e2742822b6273c372fcb76cfaa7e02fa6dd641 Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:42:56 -0400 Subject: [PATCH 12/19] Update doc/OnlineDocs/explanation/analysis/parmest/covariance.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/parmest/covariance.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst index ff0d07fb133..edbfb7d6a07 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst @@ -12,7 +12,7 @@ Covariance Matrix Estimation ---------------------------- The uncertainty in estimated model parameters can be quantified by computing the covariance matrix. -The diagonal of this covariance matrix contains the variance of the estimated parameters which is used to +The diagonal of this covariance matrix contains the variance of the estimated parameters, which is used to calculate their uncertainty. Assuming Gaussian independent and identically distributed measurement errors, the covariance matrix of the estimated parameters can be computed using the following methods which have been implemented in parmest. From 941b307e0ff75a597a78e3d471cf5c19ed2ab86a Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:43:23 -0400 Subject: [PATCH 13/19] Update doc/OnlineDocs/explanation/analysis/parmest/estimability.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/parmest/estimability.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst b/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst index 4cf40728689..ff72dd505fe 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst @@ -4,7 +4,7 @@ Estimability Analysis ===================== After estimating the model parameters with their associated uncertainty, as demonstrated in the -:ref:`driversection` and :ref:`covariancesection` Section, estimability analysis is required to identify +:ref:`driversection` and :ref:`covariancesection` Sections, estimability analysis is required to identify parameters that cannot be reliably estimated from the available data due to limitations in the mathematical model structure. If such parameters are identified, the model may need to be reformulated, replaced with an alternative structure, or augmented with additional prior information. In parmest, estimability analysis can From 4766bacace48bcf6c04e2637b263daed7df3316a Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:43:56 -0400 Subject: [PATCH 14/19] Update doc/OnlineDocs/explanation/analysis/doe/guide.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/doe/guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index 1da7952e4fd..3851c2f159f 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -94,7 +94,7 @@ the class. This step is required before the model can be labeled. Step 3: Label the information needed for DoE analysis ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -We label the four important groups as Pyomo Suffix components as mentioned before by +We label the four important groups using Pyomo :class:`Suffix` components as mentioned before by adding a ``label_experiment`` method. This method is required by Pyomo.DoE to identify the design variables (experimental inputs), measurements, measurement errors, and unknown parameters in the model. From 2ccaf8878c3984f9e4588c55fba2a4284bb8a912 Mon Sep 17 00:00:00 2001 From: Shammah Lilonfe <158379837+slilonfe5@users.noreply.github.com> Date: Sat, 9 May 2026 15:44:27 -0400 Subject: [PATCH 15/19] Update doc/OnlineDocs/explanation/analysis/doe/guide.rst Co-authored-by: John Siirola --- doc/OnlineDocs/explanation/analysis/doe/guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index 3851c2f159f..3aee2bca856 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -116,7 +116,7 @@ to perform optimal experimental design. Step 5: Exploratory analysis (Enumeration) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -After creating the subclass of the ``Experiment`` class, exploratory analysis is +After creating the subclass of the :class:`Experiment` class, exploratory analysis is suggested to enumerate the design space to check if the problem is identifiable, i.e., ensure that D-, E-optimality metrics are not small numbers near zero, and Modified E-optimality is not a big number. From f73b78d63c1545f5fae497bf8e2577f75f362e9a Mon Sep 17 00:00:00 2001 From: slilonfe5 Date: Mon, 11 May 2026 13:51:03 -0400 Subject: [PATCH 16/19] Implemented comments from John and Shilpa --- .../explanation/analysis/doe/abstraction.rst | 2 +- .../explanation/analysis/doe/guide.rst | 32 ++++++++++--------- .../explanation/analysis/doe/multexp.rst | 2 +- .../explanation/analysis/doe/objective.rst | 2 +- .../explanation/analysis/doe/overview.rst | 13 ++++---- .../explanation/analysis/doe/uncertainty.rst | 2 +- .../analysis/parmest/covariance.rst | 2 +- .../analysis/parmest/estimability.rst | 6 ++-- .../analysis/parmest/objective.rst | 2 +- .../explanation/analysis/parmest/overview.rst | 18 +++++++---- 10 files changed, 43 insertions(+), 38 deletions(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst b/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst index 4aabe3a9f3a..a169599b4a9 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst @@ -1,4 +1,4 @@ -.. _abstractexp: +.. doe_abstract_exp: Experiment Abstraction ====================== diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index 3aee2bca856..89254073419 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -1,4 +1,4 @@ -.. _startguide: +.. doe_start_guide: Quick Start Guide ================= @@ -10,9 +10,9 @@ MBDoE analysis. This is in line with the convention used in the parameter estima :ref:`Parmest `. The four Pyomo :class:`Suffix` components are: * ``experiment_inputs`` - The experimental design decisions -* ``experiment_outputs`` - The values measured during the experiment -* ``measurement_error`` - The error associated with individual values measured during the experiment. It is passed as a standard deviation or square root of the diagonal elements of the observation error covariance matrix. Pyomo.DoE currently assumes that the observation errors are Gaussain and independent both in time and across measurements. -* ``unknown_parameters`` - Those parameters in the model that are estimated using the measured values during the experiment +* ``experiment_outputs`` - The variables that are being measured +* ``measurement_error`` - The error associated with the measured value of the experimental outputs. It is passed as a standard deviation or square root of the diagonal elements of the observation (measurement) error covariance matrix. Pyomo.DoE currently assumes that the observation errors are Gaussian and independent both in time and across measurements. +* ``unknown_parameters`` - Those parameters in the model that are estimated from the experimental outputs An example of the subclassed :class:`Experiment` object that builds and labels the model is shown in the next few sections. @@ -84,8 +84,8 @@ the model without any data or discretization. Step 2: Finalize the Pyomo process model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Here, we add data to the model and finalize the discretization using a new method to -the class. This step is required before the model can be labeled. +Here, we add data to the model and discretize it. This step is required before +the model can be labeled. .. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py :start-after: End equation definition @@ -94,10 +94,10 @@ the class. This step is required before the model can be labeled. Step 3: Label the information needed for DoE analysis ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -We label the four important groups using Pyomo :class:`Suffix` components as mentioned before by -adding a ``label_experiment`` method. This method is required by Pyomo.DoE to identify -the design variables (experimental inputs), measurements, measurement errors, and -unknown parameters in the model. +This step formally labels the Pyomo model with the experimental inputs (design variables), +experimental outputs (measurements), measurement errors, and unknown parameters. The labeling +of these four important groups is performed using Pyomo :class:`Suffix` components +(as discussed earlier) by defining a ``label_experiment`` method. .. literalinclude:: /../../pyomo/contrib/doe/examples/reactor_experiment.py :start-after: End model finalization @@ -171,11 +171,13 @@ five optimality criteria using the ``compute_FIM_full_factorial`` and .. |plot5| image:: example_reactor_compute_FIM_ME_opt.png :width: 48 % -The heatmaps show the values of the objective functions, a.k.a. the -experimental information content, in the design space. Horizontal -and vertical axes are the two experimental design variables, while -the color of each grid shows the experimental information content. -For example, the D-optimality (upper left subplot) heatmap figure shows that the +The heatmaps show the variation of a FIM-based objective function +(specified by the user) over a grid of the experimental design space. +Therefore, the heatmaps are a representation of the experimental +information of various design conditions. Horizontal and vertical axes +are the two experimental design variables, while the color of each +grid shows the experimental information content. For example, +the D-optimality (upper left subplot) heatmap figure shows that the most informative region is around :math:`C_{A0}=5.0` M, :math:`T=500.0` K with a :math:`\log_{10}` determinant of FIM being around 19, while the least informative region is around :math:`C_{A0}=1.0` M, :math:`T=300.0` K, diff --git a/doc/OnlineDocs/explanation/analysis/doe/multexp.rst b/doc/OnlineDocs/explanation/analysis/doe/multexp.rst index 0c943b7e010..79c676f4db3 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/multexp.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/multexp.rst @@ -1,4 +1,4 @@ -.. _multexperiments: +.. doe_multexperiments: Multiple Experiments ==================== diff --git a/doc/OnlineDocs/explanation/analysis/doe/objective.rst b/doc/OnlineDocs/explanation/analysis/doe/objective.rst index 28bd39972e9..11bd5d3c99c 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/objective.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/objective.rst @@ -1,4 +1,4 @@ -.. _objectives: +.. doe_objectives: Objective Options ================= diff --git a/doc/OnlineDocs/explanation/analysis/doe/overview.rst b/doc/OnlineDocs/explanation/analysis/doe/overview.rst index 0a7c4a0b1a4..d1ada117fdb 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/overview.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/overview.rst @@ -17,15 +17,14 @@ increasing parameter precision. Pyomo.DoE provides the exploratory analysis and MBDoE capabilities to the Pyomo ecosystem. The user provides one Pyomo model, a set of parameter nominal values, -the allowable design spaces for design variables, and the assumed observation error model. -During exploratory analysis, Pyomo.DoE checks if the model parameters can be -inferred from the postulated measurements or preliminary data. -MBDoE then recommends optimized experimental conditions for collecting more data. +the allowable design spaces for design variables, and the postulated observation error model. +During exploratory analysis, Pyomo.DoE checks experimental design conditions that will provide +the most informative experimental outputs (i.e., data) to estimate the model parameters. Parameter estimation packages such as :ref:`Parmest ` can perform parameter estimation using the available data to infer values for parameters, and facilitate an uncertainty analysis to approximate the parameter covariance matrix. -If the parameter uncertainties are sufficiently small, the workflow terminates -and returns the final model with quantified parametric uncertainty. +If the parameter uncertainties are sufficiently small (i.e., at a level acceptable to the user), +the workflow terminates and returns the final model with quantified parametric uncertainty. If not, MBDoE recommends optimized experimental conditions to generate new data that will maximize information gain and eventually reduce parameter uncertainty. @@ -57,7 +56,7 @@ where: * :math:`\mathbf{z} \subseteq \mathcal{Z}` are algebraic state variables, :math:`\mathcal{Z} \in \mathbb{R}^{N_z \times N_t}`. * :math:`\mathbf{u} \subseteq \mathcal{U}` are time-varying decision variables, :math:`\mathcal{U} \in \mathbb{R}^{N_u \times N_t}`. * :math:`\overline{\mathbf{w}} \in \mathbb{R}^{N_w}` are time-invariant decision variables. -* :math:`\mathbf{y} \subseteq \mathcal{Y}` are measurement response variables, :math:`\mathcal{Y} \in \mathbb{R}^{N_r \times N_t}`. +* :math:`\mathbf{y} \subseteq \mathcal{Y}` are measured variables (i.e., experimental outputs), :math:`\mathcal{Y} \in \mathbb{R}^{N_r \times N_t}`. * :math:`\mathbf{f}(\cdot)` are differential equations. * :math:`\mathbf{g}(\cdot)` are algebraic equations. * :math:`\mathbf{h}(\cdot)` are measurement functions. diff --git a/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst b/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst index 50b31dbe040..cabb4f226dd 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst @@ -1,4 +1,4 @@ -.. _paramuncertainty: +.. doe_param_uncertainty: Parameter Uncertainty ===================== diff --git a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst index edbfb7d6a07..7e630f3ef8c 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst @@ -1,4 +1,4 @@ -.. _covariancesection: +.. parmest_covariance_section: Uncertainty Quantification ========================== diff --git a/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst b/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst index ff72dd505fe..d6d56c77d4d 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst @@ -1,12 +1,12 @@ -.. _estimabilitysection: +.. parmest_estimability_section: Estimability Analysis ===================== After estimating the model parameters with their associated uncertainty, as demonstrated in the -:ref:`driversection` and :ref:`covariancesection` Sections, estimability analysis is required to identify +:ref:`driversection` and :ref:`parmest_covariance_section` Sections, estimability analysis is required to identify parameters that cannot be reliably estimated from the available data due to limitations in the mathematical -model structure. If such parameters are identified, the model may need to be reformulated, replaced with an +model structure. If such parameters exist, the model may need to be reformulated, replaced with an alternative structure, or augmented with additional prior information. In parmest, estimability analysis can be performed using eigen-decomposition of the parameter covariance matrix, profile likelihood methods, or multi-start initialization routines. diff --git a/doc/OnlineDocs/explanation/analysis/parmest/objective.rst b/doc/OnlineDocs/explanation/analysis/parmest/objective.rst index ccce0eb8ac2..56a63fb055a 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/objective.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/objective.rst @@ -1,4 +1,4 @@ -.. _objectivesection: +.. parmest_objective_section: Objective Options ================= diff --git a/doc/OnlineDocs/explanation/analysis/parmest/overview.rst b/doc/OnlineDocs/explanation/analysis/parmest/overview.rst index 7a3f7538fde..8a0598bc935 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/overview.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/overview.rst @@ -32,14 +32,13 @@ a vector, :math:`\boldsymbol{\theta}`, to use in the functional form \boldsymbol{\varepsilon}_i \quad \forall \; i \in \{1, \ldots, n\} where :math:`\boldsymbol{y}_{i} \in \mathbb{R}^m` are observations of the measured or output variables, -:math:`\boldsymbol{f}` is the model function, :math:`\boldsymbol{x}_{i} \in \mathbb{R}^{q}` are the decision +:math:`\boldsymbol{f(\cdot)}` is the model function, :math:`\boldsymbol{x}_{i} \in \mathbb{R}^{q}` are the decision or input variables, :math:`\boldsymbol{\theta} \in \mathbb{R}^p` are the model parameters, :math:`\boldsymbol{\varepsilon}_{i} \in \mathbb{R}^m` are measurement errors, and :math:`n` is the number of experiments. -The following least squares objective can be used to estimate parameter -values assuming Gaussian independent and identically distributed measurement -errors: +The following least squares objective can be used to estimate model parameters +from data assuming that the measurement errors follow a Gaussian distribution: .. math:: @@ -49,6 +48,9 @@ where :math:`g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta})` can be: 1. Sum of squared errors + If the measurement errors (which are assumed to follow a Gaussian distribution) are independent + and identically distributed, the objective function can be defined as the sum of squared errors + .. math:: g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta}) = @@ -57,6 +59,10 @@ where :math:`g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta})` can be: 2. Weighted sum of squared errors + When the measurement errors are correlated and their covariance + matrix, :math:`\boldsymbol{\Sigma}_{\boldsymbol{y}}`, is known a priori, the objective + function is defined as the weighted sum of squared errors + .. math:: g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta}) = @@ -64,9 +70,7 @@ where :math:`g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta})` can be: \right)^\text{T} \boldsymbol{\Sigma}_{\boldsymbol{y}}^{-1} \left(\boldsymbol{y}_{i} - \boldsymbol{f}(\boldsymbol{x}_{i};\boldsymbol{\theta})\right) -where :math:`\boldsymbol{\Sigma}_{\boldsymbol{y}}` is the measurement error covariance matrix containing the -standard deviation of the measurement errors of :math:`\boldsymbol{y}`. Custom objectives can also be defined -for parameter estimation. +Custom objectives can also be defined for parameter estimation. In the applications of interest to us, the function :math:`g(\cdot)` is usually defined as an optimization problem with a large number of From 40faff70a5af6eaaf5b672913bee5b26bde490a9 Mon Sep 17 00:00:00 2001 From: slilonfe5 Date: Tue, 12 May 2026 10:25:29 -0400 Subject: [PATCH 17/19] Implemented feedback from John and Shilpa --- .../explanation/analysis/doe/guide.rst | 18 +++++++++--------- .../explanation/analysis/doe/overview.rst | 2 +- doc/OnlineDocs/reference/bibliography.rst | 4 ++++ 3 files changed, 14 insertions(+), 10 deletions(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index 89254073419..2fffe2dccca 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -16,7 +16,7 @@ MBDoE analysis. This is in line with the convention used in the parameter estima An example of the subclassed :class:`Experiment` object that builds and labels the model is shown in the next few sections. -This guide illustrates the use of Pyomo.DoE using a reaction kinetics example (Wang and Dowling, 2022). +This guide illustrates the use of Pyomo.DoE using a reaction kinetics example ([WD22]_). .. math:: :nowrap: @@ -117,17 +117,17 @@ Step 5: Exploratory analysis (Enumeration) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ After creating the subclass of the :class:`Experiment` class, exploratory analysis is -suggested to enumerate the design space to check if the problem is identifiable, -i.e., ensure that D-, E-optimality metrics are not small numbers near zero, and -Modified E-optimality is not a big number. +suggested to systematically enumerate the experimental design space and identify regions +that provide high information content about the model parameters, as quantified by +the A-, D-, E-, and ME-optimality criteria. Additionally, it helps to initialize the model for the optimal experimental design step. -Pyomo.DoE can perform exploratory sensitivity analysis with the ``compute_FIM_full_factorial`` method. -The ``compute_FIM_full_factorial`` method generates a grid over the design space as specified by the user. -Each grid point represents an MBDoE problem solved using the ``compute_FIM`` method. +Pyomo.DoE can perform exploratory sensitivity analysis with the :meth:`compute_FIM_full_factorial` method. +The :meth:`compute_FIM_full_factorial` method generates a grid over the design space as specified by the user. +Each grid point represents an MBDoE problem solved using the :meth:`compute_FIM` method. In this way, sensitivity of the FIM over the design space can be evaluated. -Pyomo.DoE supports plotting the results from the ``compute_FIM_full_factorial`` method -with the ``draw_factorial_figure`` method. +Pyomo.DoE supports plotting the results from the :meth:`compute_FIM_full_factorial` method +with the :meth:`draw_factorial_figure` method. The following code defines the ``run_reactor_doe`` function. This function encapsulates the workflow for both sensitivity analysis (Step 5) and optimal design (Step 6). diff --git a/doc/OnlineDocs/explanation/analysis/doe/overview.rst b/doc/OnlineDocs/explanation/analysis/doe/overview.rst index d1ada117fdb..f108b05920c 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/overview.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/overview.rst @@ -127,4 +127,4 @@ We can use the FIM or the covariance matrix to define the design criteria. A confidence ellipse is a geometric representation of the uncertainty in parameter estimates. It is derived from the covariance matrix :math:`\mathbf{V}`. -In order to solve problems of the above, Pyomo.DoE implements the 2-stage stochastic program. Please see Wang and Dowling (2022) for details. \ No newline at end of file +In order to solve problems of the above, Pyomo.DoE implements the 2-stage stochastic program. Please see [WD22]_ for details. \ No newline at end of file diff --git a/doc/OnlineDocs/reference/bibliography.rst b/doc/OnlineDocs/reference/bibliography.rst index 46164c64030..ff84ae0d0ba 100644 --- a/doc/OnlineDocs/reference/bibliography.rst +++ b/doc/OnlineDocs/reference/bibliography.rst @@ -179,3 +179,7 @@ Bibliography .. [NW99] Nocedal, Jorge, and Stephen J. Wright, eds. Numerical optimization. New York, NY: Springer New York, 1999. + +.. [WD22] Wang, Jialu, and Alexander W. Dowling. "Pyomo.DOE: An open‐source + package for model‐based design of experiments in Python." *AIChE Journal*, + 68(12), e17813. 2022. DOI `10.1002/aic.17813Y `_ From 57e5bca4331ddc2a1c2c33dd64f11785a504b76b Mon Sep 17 00:00:00 2001 From: slilonfe5 Date: Tue, 12 May 2026 11:18:28 -0400 Subject: [PATCH 18/19] Minor update to the documentation files --- doc/OnlineDocs/explanation/analysis/doe/abstraction.rst | 2 +- doc/OnlineDocs/explanation/analysis/doe/guide.rst | 2 +- doc/OnlineDocs/explanation/analysis/doe/multexp.rst | 2 +- doc/OnlineDocs/explanation/analysis/doe/objective.rst | 2 +- doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst | 2 +- doc/OnlineDocs/explanation/analysis/parmest/covariance.rst | 2 +- doc/OnlineDocs/explanation/analysis/parmest/estimability.rst | 2 +- doc/OnlineDocs/explanation/analysis/parmest/objective.rst | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst b/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst index a169599b4a9..57e113c1eed 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/abstraction.rst @@ -1,4 +1,4 @@ -.. doe_abstract_exp: +.. _doe_abstract_exp: Experiment Abstraction ====================== diff --git a/doc/OnlineDocs/explanation/analysis/doe/guide.rst b/doc/OnlineDocs/explanation/analysis/doe/guide.rst index 2fffe2dccca..f7c593a00f6 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/guide.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/guide.rst @@ -1,4 +1,4 @@ -.. doe_start_guide: +.. _doe_start_guide: Quick Start Guide ================= diff --git a/doc/OnlineDocs/explanation/analysis/doe/multexp.rst b/doc/OnlineDocs/explanation/analysis/doe/multexp.rst index 79c676f4db3..0687b37336b 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/multexp.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/multexp.rst @@ -1,4 +1,4 @@ -.. doe_multexperiments: +.. _doe_multexperiments: Multiple Experiments ==================== diff --git a/doc/OnlineDocs/explanation/analysis/doe/objective.rst b/doc/OnlineDocs/explanation/analysis/doe/objective.rst index 11bd5d3c99c..7a43356c72b 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/objective.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/objective.rst @@ -1,4 +1,4 @@ -.. doe_objectives: +.. _doe_objectives: Objective Options ================= diff --git a/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst b/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst index cabb4f226dd..37076552166 100644 --- a/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst +++ b/doc/OnlineDocs/explanation/analysis/doe/uncertainty.rst @@ -1,4 +1,4 @@ -.. doe_param_uncertainty: +.. _doe_param_uncertainty: Parameter Uncertainty ===================== diff --git a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst index 7e630f3ef8c..363f54c5684 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/covariance.rst @@ -1,4 +1,4 @@ -.. parmest_covariance_section: +.. _parmest_covariance_section: Uncertainty Quantification ========================== diff --git a/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst b/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst index d6d56c77d4d..4710990e9e0 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/estimability.rst @@ -1,4 +1,4 @@ -.. parmest_estimability_section: +.. _parmest_estimability_section: Estimability Analysis ===================== diff --git a/doc/OnlineDocs/explanation/analysis/parmest/objective.rst b/doc/OnlineDocs/explanation/analysis/parmest/objective.rst index 56a63fb055a..29c6126cb37 100644 --- a/doc/OnlineDocs/explanation/analysis/parmest/objective.rst +++ b/doc/OnlineDocs/explanation/analysis/parmest/objective.rst @@ -1,4 +1,4 @@ -.. parmest_objective_section: +.. _parmest_objective_section: Objective Options ================= From 0ba518943e0616eb322a9db30dfef8ed7b61a968 Mon Sep 17 00:00:00 2001 From: slilonfe5 Date: Tue, 12 May 2026 13:28:23 -0400 Subject: [PATCH 19/19] Updated reference in bibliography.rst --- doc/OnlineDocs/reference/bibliography.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OnlineDocs/reference/bibliography.rst b/doc/OnlineDocs/reference/bibliography.rst index ff84ae0d0ba..2c98bac63d6 100644 --- a/doc/OnlineDocs/reference/bibliography.rst +++ b/doc/OnlineDocs/reference/bibliography.rst @@ -181,5 +181,5 @@ Bibliography optimization. New York, NY: Springer New York, 1999. .. [WD22] Wang, Jialu, and Alexander W. Dowling. "Pyomo.DOE: An open‐source - package for model‐based design of experiments in Python." *AIChE Journal*, + package for model‐based design of experiments in Python", *AIChE Journal*, 68(12), e17813. 2022. DOI `10.1002/aic.17813Y `_