Machine learning-enabled maternal risk assessment for women with pre-eclampsia (the PIERS-ML model): a modelling study (2024)

Associated Data

Supplementary Materials

Data Availability Statement

Summary

Introduction

Complicating 2–4% of pregnancies, pre-eclampsia (defined as new-onset hypertension at or after 20 weeks’ gestation, accompanied by either new-onset proteinuria, other maternal target organ damage, or evidence of uteroplacental dysfunction)^{1, 2} remains a leading global cause of maternal mortality and life-threatening morbidity.^{1, 2, 3, 4, 5} More than 99% of the annual 46 000 pre-eclampsia-related maternal deaths occur in low-income and middle-income countries (LMICs).⁶

In pregnancies complicated by pre-eclampsia, it is clear that perinatal survival without major morbidity is largely related to gestational age at birth.⁷ However, the burden of adverse maternal outcomes is spread across gestation. Although maternal risks are proportionately greater with the earlier onset of pre-eclampsia,^{5, 8, 9} the population level burden of maternal risk is borne by the 75–80% of cases of pre-eclampsia that arise at term.¹⁰

The sole method of initiating recovery from pre-eclampsia is delivery of the placenta.² At term, the focus is initiating birth.¹¹ Before term, women and their maternity care providers balance maternal risks from evolving disease with prematurity-related perinatal risks by subjectively integrating ongoing assessments of symptoms, signs, and laboratory tests.² In busy maternity units, considerable experience informs decisions; however, most women with preterm pre-eclampsia are managed, at least initially, by relatively inexperienced maternity care providers. For many LMICs and disadvantaged high-income country populations, access to comprehensive obstetric and newborn care is limited.

To optimise maternal outcomes in pre-eclampsia, we need objective, time-of-disease maternal risk assessment to inform decision making during the following 48 h, wherever that woman lives. We previously used logistic regression to develop a model—fullPIERS (pre-eclampsia integrated estimate of risk)—for high-income countries.⁸ In this study, we aimed to harness the strengths of machine learning-based classifiers to test the hypothesis that it is possible to develop and externally validate a novel globally-relevant PIERS-ML model using information routinely-available at presentation with pre-eclampsia.

Methods

Results

Data were available for 8843 eligible women with pre-eclampsia, who were recruited from 53 institutions (appendix pp 2–4) in 11 countries (Australia, Brazil, Canada, Fiji, Finland, New Zealand, Pakistan, South Africa, Uganda, UK, and USA), with a median of 1244 (IQR 693·5–1983·5) women per included cohort.^{5, 8, 12, 19, 20, 21} Other cohort details have been published previously.^{5, 8, 12} For external validation from two new institutions, data were available from an additional 2901 women, and collected as part of a prospective observational study.

Health system and individual-level characteristics of women who experienced an adverse maternal outcome differed from women who did not (table 1; appendix pp 6–7). Women who experienced adverse outcomes were more often cared for in countries with lower per capita gross domestic products. These women were younger, more likely to have multiple pregnancies, and presented at an earlier gestational age; less often, their past history was complicated by chronic hypertension and their pregnancies by gestational diabetes. The study population had similar proportions of women from White, Asian, or Black ethnic backgrounds, regardless of complications.

At presentation with pre-eclampsia, women who subsequently developed an adverse maternal outcome (vs those who did not) differed in their symptom profile, signs, and laboratory results, although results largely overlapped (table 1). Compared with women who did not experience outcomes, women who did experience outcomes were more often symptomatic, had lower weight, higher blood pressure, lower oxygen saturation, higher dipstick proteinuria, and more perturbed laboratory results; differences in laboratory results were not changed significantly by imputation. Women who subsequently developed an adverse outcome more often received antenatal corticosteroids, antihypertensives, and magnesium sulphate. Compared with the babies of women who did not experience outcomes, babies of women who did were born earlier and of lower birthweight, and more often died.

Table 2 shows that 590 (6·7%) of 8843 women had an adverse maternal outcome within 2 days of first assessment, 813 (9·2%) within 7 days of assessment, and 1083 (12·2%) at any time before primary discharge. Most adverse outcomes were cardiorespiratory, haematological, hepatic, or placental. There were two maternal deaths, neither within 48 h of first assessment. In the external validation group, 83 (2·9%) of 2901 women had an adverse maternal outcome within 2 days, 99 (3·4%) within 7 days, and 121 (4·2%) at any time before primary discharge. Most adverse outcomes were central nervous system, haematological, or placental.

Data from a random selection of 6633 women were used for model development. The following variables were excluded due to at least 60% missingness: total bilirubin, urinary protein to creatinine ratio, international normalised ratio, and lactate dehydrogenase (appendix pp 8–9). The remaining variables either contained no missingness or fewer than 60% missing at random or missing completely at random observations.

Figure 1 contains a plot of the Shapley values for a single random forest on its corresponding validation dataset. Although most Shapley values for all variables were close to zero, high values of serum creatinine, national maternal mortality ratio, aspartate and alanine transaminases, and low values of platelet count, oxygen saturation, national per capita GDP, and haematocrit had high Shapley values, meaning an increased predicted probability (appendix pp 21–22). Figure 2 shows the relative importance of the 18 PIERS-ML model variables. Platelet count was of greatest importance and national MMR was of least importance. All target organ systems were included, with the following numbers of covariates: cardiorespiratory (N=3), renal (N=3), hepatic (N=2), and haematological (N=4). In addition, there were variables representing health systems (N=2), demographic characteristics (N=2), and anthropometry (N=2).

Open in a separate window

Figure 1

Shapley values for the PIERS-ML variables in a single random forest on the corresponding development dataset

Shapley values represent the feature value's contribution to the individual's predicted probability. Positive Shapley values increase the predicted probability, negative values decrease the predicted probability, and values of zero show no change to the predicted probability. EDD=expected date of delivery. GDP=gross domestic product. SpO₂=oxygen saturation by pulse oximetry.

Open in a separate window

Figure 2

PIERS-ML variables ranked by importance within the random forest model based on Gini index, compared with the least important variable (National MMR)

Random Forests enable examination of feature importances, which is the mean of the amount the Gini Index (or node impurity) decreases by in each tree at the split that uses the feature. The more the Gini Index decreases for a feature, the more important it is. This figure rates the features from 0–100, with 100 being the most important. EDD=expected date of delivery. GDP=gross domestic product. SpO₂=oxygen saturation by pulse oximetry.

From the remaining 2210 (24·9%) of 8843 women, 1107 women informed selection of cutoff points for the risk strata according to likelihood ratios, and the remaining 1103 women informed PIERS-ML model validation according to the selected cutoff points. Calibration was assessed on all 2210 women in the internal validation group; although the linear calibration curve appeared to be close to an intercept of 0 and slope of 1, the Spiegelhalter p value of 0·0026 showed the model to not be optimally calibrated (figure 3). As calibration did not improve upon recalibration (appendix pp 22–23), we chose to prioritise stratification into risk classification groups to inform clinical decision making. The PIERS-ML model accurately stratified risk for adverse maternal outcomes within 2 days (table 3), with an AUROC of 0·80 (95% CI 0·76–0·84 vs the currently used logistic regression model, fullPIERS: AUROC 0·68 [95% CI 0·63–0·74]; figure 4). The precision-recall curve of the PIERS-ML model on the validation data was very close to a straight line, meaning that there is no single probability threshold with both good precision and good recall; therefore, using a single cutoff point to classify patients into an outcome group and a no outcome group would be inaccurate (figure 5).

Open in a separate window

See Also

untitled

Figure 3

Calibration plot for the PIERS-ML model

Predictions on the validation and testing datasets, in increasing order, were binned together into ten groups of 221 predictions. Event rates (observed risk) were calculated along with confidence intervals and plotted against the mean predicted probability per group to create the dot and whisker plot. Smooth lines were plotted using the individual predicted probabilities and yes and no outcomes, with Linear (red) and Loess (blue) methods. Cox calibration intercept and slope, Brier score, and Spiegelhalter Z scores were calculated.

Open in a separate window

Figure 4

PIERS-ML area under receiver-operator characteristic for adverse maternal outcomes within 2 days of initial assessment using data within 1 day of initial assessment and before the occurrence of any outcome

Area-under-the receiver-operator characteristic of 0·78 (95% CI 0·73–0·82).

Open in a separate window

Figure 5

Precision-recall plot for the PIERS-ML model

Of the 1103 women in the internal validation group, 329 (29·8%) women were classified as being at very low risk (eight [0·7%] women) or low risk (321 [29·1%] women) during the 2 days (table 3). Among women at very low risk, adverse maternal outcomes were very infrequent: zero within 2 days, and one (12·5%) of eight women each within 7 days or at any time. Among women at low risk, adverse maternal outcomes were also infrequent: seven (2·2%) of 321 women within 2 days, 13 (4·0%) women within 7 days, and 21 (6·5%) women at any time. The PIERS-ML model correctly classified women at very low and low risk of adverse outcomes within 2 days, but not within 7 days or at any time.

98 (8·9%) of 1103 women were classified as being at high (87 [7·9%] women) or very high risk (11 [1·0%] women) of an adverse maternal outcome within 2 days (table 3). Among women at high risk, adverse maternal outcomes were frequent: 23 (26·4%) of 87 women within 2 days, 25 (28·7%) women within 7 days, and 28 (32·2%) women at any time. Among women at very high risk, adverse maternal outcomes were very frequent, with all ten (90·9%) of 11 women's outcomes occuring within 2 days. The PIERS-ML model correctly classified women at high and very high risk of adverse outcomes within 2 days, 7 days, or at any time.

Among 676 (61·3%) of 1103 women classified as being at moderate risk, adverse maternal outcomes occurred in 36 (5·3%) women within 2 days, 58 (8·6%) women within 7 days, and 76 (11·2%) women at any time, all within the anticipated range of outcomes based on uninformative likelihood ratios (table 3).

The event rate in the external validation group was 2·9% (83 of 2901 women) in 2 days, 3·4% (99 women) in 7 days, and 4·2% (121 women) at any point, compared with 6·7% in 2 days, 9·2% in 7 days, and 12·2% at any point in the combined dataset. From the 2901 women in the external validation cohort, nine (0·3%) women were classified as being at very low risk, 1512 (52·1%) women were classified as being at low risk, 1324 (45·7%) women were classified as being at moderate risk, 52 (1·8%) women were classified as being at high risk, and three (0·1%) women were classified as being at very high risk. The PIERS-ML model accurately stratified risk for adverse maternal outcomes within 2 days with women assigned to each stratum experiencing an outcome rate within the predicted range, and an AUROC of 0·76 (95% CI 0·71–0·82; table 3). Although the event rates were lower in the external validation set than the development data, the model still correctly classified women at very low risk and low risk, as well as those at high risk and very high risk.

Excluding mean platelet volume from the set of variables significantly altered the PIERS-ML model's performance, resulting in 59·5% (656 of 1103 women) in the moderate risk stratum, and loss of reassurance for the very low risk and low risk strata beyond 2 days (table 3), although the AUROC remained unchanged (0·80 [0·76–0·84]). Including fibrinogen and activated prothrombin time as variables neither improved the model's performance (AUROC 0·80 [0·76–0·84]) nor its risk stratification capacity (moderate risk stratum 64·3% (709 of 1103; table 3); as there was no significant difference between model performance, the model with no coagulation variables was chosen as coagulation variables are often not available or expensive to obtain if not necessary. The original fullPIERS model had poorer performance in the combined validation dataset (table 3).

The PIERS-ML model had strong performance in ruling out (very low risk and low risk strata) and ruling in (high risk and very high risk strata) subsequent seizures of eclampsia (table 4). For the 2210 women in the full internal validation group, stillbirth risks varied by risk strata, such that women at very low risk and low risk of adverse maternal events had few stillbirths, whereas those at moderate risk had 2·8% (32 of 1148) stillbirths, high risk had 11·6% (16 of 138 women) stillbirths, and very high risk had 0·0% stillbirths (table 4).

Removing renal and haematological components of the outcome did not significantly alter PIERS-ML performance (appendix pp 10–11). Alternative machine learning strategies were considered but did not show improved performance (appendix pp 12–14). The complete case analysis showed a reduced AUROC but with wider confidence intervals that included the AUROC of the imputed model. The validation with mean imputation had both similar number of patients per risk strata and similar number of outcomes as with imputation (appendix pp 15–16, 25).

Discussion

This study included 8843 women from 11 countries presenting for first assessment of pre-eclampsia and used the random forest method to develop and internally validate an 18-variable model for maternal risk stratification, applicable for LMICs and high-income countries. A further 2901 UK resident women contributed data to a fully external validation dataset.

The PIERS-ML model has identified nearly 40% of women with pre-eclampsia for whom care should be altered. The 29·8% (329 of 1103 women) and their families and maternity care providers identified as being at very low risk (0·7% [eight of 1103 women]) or low risk (29·0% [321 women]) can be reassured that it is very unlikely that adverse maternal events will occur within 2 days. However, for the 8·9% (98 women) identified to be a high risk (7·9% [87 women]) or very high risk (1·0% [11 women]), a timely clinical response can be justified based on a substantial risk of an adverse maternal event within 2 days, or for women at very high risk, at any time. Identifying these women can inform discussions about place of care, transfer of care, antenatal and postnatal surveillance, co-interventions, and timed birth.

Strengths of our study include the large sample size, which was, to our knowledge, larger than any previous study modelling adverse outcomes in women with pre-eclampsia.^{5, 8, 12, 20, 23} We tested a list of variables with clinical external validity and availability, including all target organ systems, except for the central nervous system. Clinical central nervous system predictors rely on either the subjectivity of symptoms or the questionable reproducibility of deep tendon reflexes and clonus, particularly in pregnancy. Also, the PIERS-ML model does not include direct measures of coagulation; these are not routinely performed, and their addition neither altered model performance nor warranted related costs in women with pre-eclampsia. Machine learning is suitable for managing many variables, without assumption with respect to interactions and mediation, and addresses concerns regarding collinearity. Trade-offs were considered between model performance, complexity, and face validity. In addition, machine learning algorithms can use all available predictor variables as inputs as variables that are not predictive of the outcome will have little to no effect on the predicted probabilities. Therefore, a machine learning-based model with more variables than necessary should not have degraded performance in the context of this application. However, including unnecessary variables means that included variables might not contribute substantially to the predictions. To make the model more pragmatic in a clinical setting, we have chosen to use feature selection to remove unimportant variables avoiding unnecessary data collection or clinical tests.

There are significant differences between our PIERS-ML model and the recently published Charité machine learning-based model.²³ Schmidt and colleagues²³ used data from 1647 women with pre-eclampsia admitted to a single high-income country institution, and those data were modelled against a composite maternal and fetal outcome, despite rates of adverse maternal and perinatal outcomes not tracking together.^{24, 25} Although they applied different machine learning methods, they used approximately 24 observations (internally normalised to multiples of the median) per candidate variable, compared with approximately 230 observations per variable in the PIERS-ML dataset. They undertook no imputations, thereby assuming that missingness is predictive of either outcome or no outcome. In addition, we fitted models using ten-fold cross-validation to minimise overfitting and validated the model in an independent 12·5% (vs 10% from the Charité model) of the dataset. Importantly, our risk stratification was data driven in ruling out or ruling in the risk of adverse maternal outcomes. The main strength of the study by Schmidt and colleagues²³ was the inclusion of angiogenic markers, whereas the main strengths of our study are the diversity of data, the use of multiple imputation, and external validation.

Missing values were common in our dataset; however, where appropriate, missingness was handled by multiple imputation, with chained random forest to minimise bias. Of note, development and validation datasets were imputed separately, thereby creating independent model development and validation datasets. Some variables (namely bilirubin, urinary protein to creatinine ratio, international normalised ratio, and lactate dehydrogenase) had to be excluded due to high levels of missingness. This limitation reflects current clinical practice. Modelling on complete data from a representative and diverse sample population will always give a more accurate result than modelling on a dataset with a substantial amount of missingness. However, obtaining such a dataset would be very difficult as many of these variables are not regularly collected in clinical practice from patients with pre-eclampsia, especially close to full term. Additionally, international normalised ratio and lactate dehydrogenase were not expected to be strong predictors, whereas urinary protein to creatinine ratio would be anticipated to outperform the readily available dipstick proteinuria.

The composite adverse maternal outcome for the PIERS model was Delphi-derived, similarly to the core maternal outcome list (iHOPE) for pre-eclampsia.²⁶ However, there are differences: only PIERS includes uncontrolled hypertension, inotropic support, myocardial ischaemia or infarction, hepatic dysfunction, or transfusion, and only the iHOPE outcome set contains elevated liver enzymes, postpartum haemorrhage, and admission to intensive care. To confirm model performance, as some factors are both predictors and components of the combined outcome (eg, serum creatinine and platelet count), we assessed the PIERS-ML model excluding renal or haematological components of the outcome, or both (appendix pp 10–11).

The PIERS-ML model also proved to be highly effective on external validation in a UK resident cohort, achieving the predicted outcome rates in the risk classification strata expected from internal validation. The model could identify women for whom a timely response can be justified very well while reassuring women who were predicted very low risk who had no outcomes at any point. However, the full range of the components of the combined maternal outcome were not readily available, and some outcomes (eg, renal) were conflated within the dataset as those data had been collected previously. Although this introduces the possibility of underestimating the outcome rate as defined by the original PIERS combined outcome, we are confident in the model's ability to classify into risk strata, especially for ruling in outcomes in the high risk and very high risk strata.

Multivariable model-based risk stratification of women with pre-eclampsia is recommended by national and international clinical practice guidelines.^{1, 27, 28} With access to full laboratory facilities, models have been based on either logistic regression or a survival model for time-to-adverse event.^{8, 20} Women with a hypertensive disorder of pregnancy (including pre-eclampsia) in LMICs, without ready access to laboratory tests, benefit from the demographics, symptom, and signed-based miniPIERS model, with model performance improved by pulse oximetry.^{5, 29}

The PIERS-ML model improves on previous models in several ways. First, to our knowledge, the PIERS-ML model is the first model for risk stratification in pre-eclampsia that has been developed using machine learning from women with pre-eclampsia living in LMICs (sub-Saharan Africa, South America, south Asia, and Oceania—areas of the world where more than 99% of pre-eclampsia-related maternal mortality occurs)⁶ and high-income countries. We are unaware of another model that has included data from 11 low-income, middle-income, and high-income countries.

Uniquely, the model includes national per capita gross domestic product and maternal mortality ratio, which are variables that adjust for location, avoiding adaptation of models to local outcome rates, particularly in which information governance and research resources are absent or limited. Adjusting for local settings was required for original fullPIERS model validation in LMICs.¹⁸ Therefore, we were not surprised by the poor performance of fullPIERS within this combined dataset. This approach to create auto-adjustment for setting should be validated in further geographies.

Second, the PIERS-ML model does not include maternal symptoms, the inclusion of which was criticised as a weakness of previous models,⁸ given the subjective nature, variable definitions, and inconsistent documentation of symptoms in health records. However, mean platelet volume, which is a marker of platelet consumption and release of immature platelet forms, is important within the PIERS-ML model.³⁰ Haematology analysers routinely measure, but many laboratories do not report, mean platelet volume; our findings suggest mean platelet volume should be reported for all hypertensive pregnant women.

Third, although the PIERS-ML model does not include symptoms of central nervous system involvement, the model has clinically-relevant performance in identifying women at both least (very low risk and low risk strata) and greatest (high risk and very high risk strata) risk of developing eclampsia, especially within 7 days of initial assessment (table 4). These results could guide the targeted use of magnesium sulphate and reduce the number-needed-to-treat, for the prevention of eclampsia.³¹ Depending on health system resilience, women in the moderate risk strata might or might not be considered for magnesium sulphate prophylaxis. For women with disease onset before 34⁺⁰ weeks’ gestation, a loading dose of magnesium sulphate should be administered to reduce the risk of prematurity-related cerebral palsy.³²

Fourth, women in the very low risk and low risk strata were very unlikely to suffer a stillbirth (table 4), and we believe that such women can be appropriately reassured by our model. All intrauterine fetal deaths were noted within 2 days of admission with pre-eclampsia. However, it was notable that the women in the moderate risk and high risk strata bore the greatest risk of stillbirth, presumably as maternity care providers were sufficiently concerned by the condition of women in the very high risk stratum to intervene for either maternal or fetal indications, or both, or in response to the woman experiencing an adverse maternal event. In addition, the very high risk stratum had a small sample size; therefore, performance in predicting stillbirth could not be accurately assessed.

Finally, the PIERS-ML model has been externally validated with very good performance in an independent cohort of women with pre-eclampsia admitted to hospitals that did not participate in the fullPIERS development and validation projects, and where neither fullPIERS nor miniPIERS were used in routine clinical practice.

There are three important lines of investigation that follow from our work. The first is that use of this method offers the potential for the accuracy of the model to improve over time as data accumulate and the model learns with regularly scheduled manual updates provided to model users. Amassing such data is feasible, as all individual-level variables in the PIERS-ML model are part of routine clinical and laboratory assessment of women with pre-eclampsia in well-resourced settings, and available from electronic health records in real-time. The second line of investigation is to explore whether addition of new markers could improve the PIERS-ML model performance. As markers of uteroplacental dysfunction of pre-eclampsia, angiogenic markers are used increasingly in the investigation of women with suspected pre-eclampsia, at first presentation, for ongoing surveillance³³ (although performance in trials has been variable^{34, 35}), and within the Charité model.²³ If independently informative, angiogenic markers could be incorporated into the model during clinical implementation. An ophthalmic artery Doppler could provide useful information about the less-accessible intracranial circulation.³⁶

Third, to determine how best to assess evolving risk among women with pre-eclampsia, especially those women at moderate risk who will require ongoing, close surveillance, particularly during the first 7 days when most adverse outcomes occur. PIERS-ML, and other models, perform best during the first 2–7 days; women with pre-eclampsia could be expectantly managed for up to 4 weeks.² Although repeated risk stratification has been recommended,²⁷ future work should examine replacing this serial static model approach with a new dynamic approach that accounts for changes over time.

The PIERS-ML model uses the power of machine learning to develop a new effective risk stratification tool for international use. Using only variables routinely assessed in pregnant women with pre-eclampsia, this tool can stratify women with the condition into clinically-relevant risk strata, to guide clinical care, and is amenable to future inclusion of new biomarkers and predictors.

Data sharing

The PIERS data are de-identified participant-level data. As permitted by existing data sharing and collaboration agreements, the data will be available to academically active entities (eg, universities, non-governmental organisations, and multilaterals), with the PIERS Principal Investigator (Peter von Dadelszen), or named delegate, as a named co-investigator, for the purposes of pregnancy hypertension-related research and within the limits of the informed consent obtained. Access will be through the Principal Investigator, or named delegate, contacted at pvd@kcl.ac.uk. A full data dictionary and all study documents will be available. Access will be through written application. The complete data sharing statement for the PIERS-ML model is in the appendix (p 11).

Declaration of interests

TM-C, KK, PM, SJEB, LAM, and PvD acknowledge that the intellectual property related to the PIERS-ML model has been registered, and that the inventors have no financial benefit from the use of the model based on the transfer. TM-C was funded by the University of Strathclyde, through the STRADDLE (University of Strathclyde Diversity in Data Linkage) Centre for Doctoral Training. All other authors declare no competing interests.

Acknowledgments

This study was supported by a grant from the Fetal Medicine Foundation (charity number 1037116). The PIERS datasets were primarily funded by operating grants from the Canadian Institutes of Health Research and the Bill & Melinda Gates Foundation. Funders had no role in the design, analyses, interpretation, or manuscript preparation for this study.

Contributors

PvD, KK, PM, and LAM designed the concept of the PIERS-ML project and had primary oversight of these analyses. TM-C and PvD co-wrote the first draft of the manuscript, and all authors contributed to the revisions and approved the submitted version. RA, KHN, OI, and AS collected the external validation cohort data. TM-C managed the central data set and was responsible for the statistical analyses with support from KK, PM, and SJEB. All authors had full access to all data in the study and had final responsibility for the decision to submit for publication. TM-C, KK, and SJEB accessed and verified the data.

Contributor Information

PIERS Consortium^†:

Supplementary Material

References