Blue Plains Advanced Wastewater Treatment Plant (BP AWWTP) employes High-rate Contact Stabilization (HiCS) for secondary treatment following chemically enhanced primary treatment (CEPT). It precedes the tertiary treatment where nitrogen removal is performed. In addition to the primary effluent, the HiCS system receives two return flows that are hypothesized to influence performance: (i) the solids processing line (SPB) produced during the thickening and dewatering processes, and (ii) waste nitrification sludge (WNS) from the downstream tertiary nitrification process. At BP AWWTP, it was observed that bio-augmenting the A-stage HiCS process with WNS may positively influence settleability and overall process performance, whereas supernatant return flow from solids processing (SPB) may negatively influence process stability. 

Unfortunately, current mechanistic modeling approaches are inadequate for robust prediction of dynamic behavior in HiCS. Therefore, the ultimate goal of this work is to develop a data-driven tool to control the secondary treatment process to keep its effluent suspended solids (ESS) below 25 mg/L. 

This goal can be achieved through a two-step process:  

Objective 1: Develop a data-driven model that links HiCS operational conditions and ESS. For this model to serve the goal of controlling the process ESS, the project team subjected model developement to two specific constraints: 

• Successfully capture ESS daily dynamics. 

• Identify sensitivities to changes in the manipulated variables (SPB and WNS flow rates). 

Objective 2: Based on the developed model, develop a decision-making tool to inform operators with effect and optimal SPB and WNS flow rates to control ESS below 25 mg/L.

These variables were collected from multiple sources using different methods and with different collection frequencies. The DCS-based data are collected from online sensors and available with high frequency on a daily basis. This includes different flow rates like SPB flow rate (SPB_Q) and WNS flow rate (WNS_Q), dissolved oxygen (DO) concentration via online sensors, and mixed liquor suspended solids (MLSS) probe data. The second group of the data is collected and measured manually by the DC Water Lab team, but is available at a relatively high frequency (more than once a week). This includes parameters like lab-based MLSS measurements, primary effluent total suspended solids (PE_TSS), and total soluble phosphorus of the primary effluent (PE_TSP) and secondary effluent (SE_TSP). The last group of data sources requires more complex experimental measurements and is performed by the DC Water Research team, and is therefore available in a less frequent and a less systematic manner (generally available once a week.) This includes parameters like threshold of flocculation (TOF), sludge volume index (SVI), settling flux curves, and COD and nitrogen fractions for the primary effluent and secondary effluent. To complement historical data, a tailored 16-week experimental plan was designed and implemented to intentionally change the WNS and SPB control points and measure its impact. The purpose of this data collection campaign was to impose sufficient systematic variation in the WNS and SPB return flows to robustly assess their influence on ESS, thereby aiding in robust model development. This experimental campaign was completed at the end of November 2022.

A major challenge with the data is the high degree of missingness and inconsistency in such missingness. This challenge was multiplied with data during COVID periods, where sampling frequency and routine varied significantly. Missingness was assessed using missingo library as shown below

The main data preparation or pre-processing step was data imputation. For such a process, For data imputation, a multi-step approach was adopted. In the first step, variables were imputed using linear interpolation. This was restricted to a maximum of 2 missing consecutive days in a week (7 days). To validate the assumption that the weekly data follows a linear pattern, the data was split in weeks and a linear model was fitted into each week and the goodness of fit was evaluated to confirm its suitability. In the second step, a multivariate imputation was used to leverage the high dimensionality and co-linearity in the dataset. First, variables were selected using maximum relevance minimum redundancy (mRMR) method. Next, an optimized Huber regression model was trained using the existing points and the fitted model was used to predict the missing values. Huber regression is a robust linear regression technique used in statistics and machine learning for modeling the relationship between variables in the presence of outliers and influential data points. A threshold of prediction accuracy (validation R2 above 0.5) was set to avoid imputing variables using inaccurate models. An example of the results of the multivariate imputation is shown in Figure 5.4.4. The last step was using the moving average for variables expected to remain constant for a specific period of time. This period of time was determined according to process knowledge. For primary effluent characteristics, no imputation was performed at this step as there is no prior knowledge of changes in influent characteristics. Seven days moving average were used to impute variables related to SPB and WNS streams as they are generated based on relatively slow processes, while 3 days moving average were used for the HiCS process conditions like sludge characteristics as this was the average SRT of the system.  

Given the stochastic and dynamic nature of the ESS, it was smoothed by using a 7-day rolling average. This fits well with the project objectives as the goal is to make weekly or bi-weekly decisions on SPB and WNS flow rates. The dataset was normalized using a Min-Max scaler, to ensure all values were between 0 and 1, which significantly improved model performance by ensuring that all variables contribute equally.

Feature selection was performed in multiple steps. The first step is selecting variables with missing ratio equal or less than 50%. Such variables were used in preliminary algorithm selection and as a starting point. Then, the artificial contrasts with ensembles (ACE) method was used for the selection of the key variables from the variables with a missing ratio below 50%. ACE is unsupervised and does not require a pre-selected number of variables or pre-determined stopping criteria. It works by first estimating feature importance using an algorithm of ensemble of trees (RF was used). Then, for each variable in the dataset, an artificial random variable (called “artificial contrast variable”) is created from the same distribution as its corresponding variable but is independent of the target. A variable is eliminated if it has lower importance, that is artificial contrast. A snippet of the code is shown below.

Following, a rigorous hyperparameter optimization (HPO) was performed along with different methods for data imbalance treatment. Below is the hyperparameter space used for optimization.

An observation from data exploration is data imbalance. This is a typical challenge with data obtained from wastewater treatment. In most of the days, the process is performing well, and ESS is below 25 mg/L, as a result, the algorithm prioritizes predicting the common low values and ignores the high values. To fix this issue, several imbalance treatment methods were tested, including Synthetic Minority Oversampling Technique (SMOTE), Adaptive Synthetic Oversampling (ADASYN), weight-adjusted loss function, and quantile transformation of the target. SMOTE is an oversampling method that creates synthetic samples using k-nearest neighbors of the minority instances in the original dataset, thus addressing imbalances in the target variable distribution. ADASYN is similar to SMOTE, but instead of oversampling the minority instances of the data, it generates more samples similar to those that are difficult to learn. A weight-adjusted loss function assigns different weights to instances based on their representation in the dataset, penalizing errors associated with the less represented instances more heavily to improve model performance. This was performed by assigning weights equal to the inverse of the probability density function of the training data or by assigning higher weights to values beyond the interquartile range. Quantile transformation of the target involves transforming the target variable to follow a uniform or normal distribution, stabilizing variance and making the modeling process more robust to imbalanced data.

To train and evaluate models based on different data imbalance methods, Nested cross validation was used to compare models.

In the actual project, 10 different algorithms were tested. However, in the modelling notebook only support vector machines are used as an example. The best results were obtained with weight adjusted loss function. This applies to both goodness of fit indicated by R2 and RMSE, and degree of overfitting measured as the percentage of the decrease in R2 from training to testing. 

Below are the plots for such model “SVM_W3”.

For this model, permutation feature importance was determined to understand which variables are more valuable to the model. This is particularly significant because the ultimate goal is to use this model to advise on SPB and WNS flow rates. Below is snippet of the code and the results

Project overview video

In addition, the model can be fed the current conditions (the input variables) as a new instance in addition to all possible flow rates for WNS and SPB, and the algorithm will generate different predictions according the WNS and SPB values. These values can be plotted in a heat map similar to the one shown below. The center of the heatmap can be adjusted based on the current ESS. This heat map will answer both questions; if all values are similar to the existing ESS, this means WNS and SPB return flows are not the main contributors to ESS. For the second question, the heat map will identify the zones of operation where low ESS is expected under the existing conditions.