In their manuscript, titled ‘Towards resolving poor performance of mechanistic soil organic carbon models’, Wang et al. describe their results of a study comparing the performance of two mechanistic soil organic carbon models (MIMICS and the newly-proposed MES-C) to a machine learning approach (random forest). They use the mechanistic models and random forest models to predict the amount of soil organic carbon for multiple locations around the world, after which they compare model results to depth profiles of SOC from the WoSIS database. Based on their results, the authors conclude that random forest models consistently outperformed mechanistic models. In addition, they used interpretable machine learning methods to conclude that both mechanistic soil organic carbon models perform poorly due to lack of accounting for key variables (most notably CEC), while the role of existing variables is underrepresented. They compare the controlling factors in the mechanistic models to factors controlling the SOC content at the global scale, and find that, for example, the role of the clay and silt content of the soil is overestimated by the models, compared to observations. Furthermore, the authors asses the sensitivity of model outcomes along a range of predictor values, and study how pairs of predictors affect model outcomes, compared to observations.

The development and application of mechanistic soil organic matter models, among which MIMICS, has gained much importance over the past two decades. Therefore, the assessment of model performance and the relative importance of model drivers in making predictions of the SOC content, compared to observations, is needed to improve these models to increase confidence in their outcomes. I appreciate the effort done to collect the data, develop a new model and apply it, together with MIMICS, at the global scale, which is a challenging endeavor. The manuscript is well-written and well-structured. The abstract conveys the main messages and the introduction provides sufficient background to the topic, although I would encourage the authors to include more information about previous studies assessing controls of the SOC content at the global scale, which is the topic of their study. Although the methods section describes the performed analyses and data collection strategy, much information about the performed model simulations is missing to assess the quality of the performed simulations (see below). The results section concisely describes the results, but I would encourage the authors to show the model results in a graphical way, for example using scatterplots. The discussion touches upon the main results, but mainly sections 4.3 and 4.4 lack clear messages, and are rather a summing up of the relations the authors found.

Based on my evaluation I recommend major revisions, mainly due to (1) the lack of detailed information which would be necessary to evaluate the performed model simulations and (2) the nearly complete attribution of the mismatch between model results and measurements to the structure of the model, without accounting for uncertainties in input data at the global scale (see below). My main concerns about the manuscript are the following, while detailed feedback is provided below.

• The authors propose a new mechanistic soil organic matter model that is more complex than MIMICS, as it also incorporates SOC in aggregates. However, the authors do not have any data on the distribution of measured SOC among different pools, so they have no data to either constrain the distribution of simulated SOC among the different model pools, or to evaluate this simulated distribution. Therefore, I invite the authors to justify the use of such a (complex) model at the global scale, given this lack of data. The evaluation of the simulation results based on data for only total SOC inevitably leads to overparameterization and equifinality. This is an important, but generally overlooked, aspect of environmental models, the consequences of which for the model simulations should be discussed in the manuscript.
• The authors attribute the ‘poor performance’ of the mechanistic models exclusively to the structure of these models. However, there are many more uncertainties leading to a mismatch between model results and measurements:
  • Uncertainties related to model drivers, which were extracted from global data sets. For example, NPP data derived from MODIS cannot be expected to correctly represent C inputs from vegetation to the soil, as, for example, a substantial part of aboveground litter C will be respired before it can enter the soil. A model can never be better than the input data, so accounting for uncertainties in the data is important.
  • The applied mechanistic models are too complex compared to the data available for model evaluation, i.e., they are overparameterized. Six parameters are evaluated while only one output (total SOC) is evaluated (it seems, as this is not described clearly in the manuscript). This inevitably leads to large uncertainties about model results.

These aspects should be discussed, because it is evident that an overparameterized model with uncertain inputs will lead to a ‘poor performance’.

• The methods related to the application of the mechanistic models are not sufficiently described. For example:
  • Down to which depth were the simulations performed? To 1.5 m, as this is the maximum depth down to which measurements were available?
  • Were the simulations performed at a vertical resolution of 30 cm?
  • For how long were the simulations run? Was there a spin-up period, followed by a run with actual temperature and precipitation data?
  • In which programming language were the models coded?
  • With which time step were the simulations performed? Which solver was used to solve the differential equations?
  • How were locations with a different land use treated? Was there a different vertical resolution for C inputs per land use?
• No sufficient information about model evaluation is provided. As the authors conclude that the mechanistic models perform poorly compared to machine learning approaches, a thorough model evaluation is important to support this conclusion. For example:
  • What was the simulated turnover time of SOC at different soil depths? Was this in line with measurements (e.g., DOI: 10.1126/science.aad4273) Did this decrease with soil depth, as is generally observed?
  • How was the simulated amount of SOC distributed among the different pools? Was this distribution in line with observations?
• The authors should justify why they chose to perform a global study, instead of focusing on a smaller region with less uncertainties about input data for the models and more detailed data for model evaluation.

Detailed feedback

Abstract

L23: Would be good to mention which ‘existing variables’ these are

L29: What do you mean with a ‘simple trend’?

Introduction

L45-55: It is not clear what the ‘effectiveness of land management strategies’ has to do with the ability to accurately estimate SOC content

L73: The study by Gurung et al. is a study on daycent, so not applicable as a reference for the ‘poorly constrained parameter values of microbial explicit models’

L83: Georgiou et al. (2022) is not a modelling study, so not appropriate as a reference for this sentence.

L107: ‘accuracy’ of what?

L111: R2 is not a measure of model performance (see e.g. https://en.wikipedia.org/wiki/Anscombe%27s_quartet), so please remove this throughout the manuscript when used for this purpose.

Methods

General methods: Does the WoSIS database contain SOC stocks, i.e., a combination of OC% and bulk density, for all profiles that were used? If not, how was OC% converted to SOC stocks? Although it is not described in the manuscript, I assume absolute amounts of SOC were simulated? To compare these to observations of OC% in WoSIS, these would need to be converted to OC%. How did this happen? This information should be provided in the manuscript.

L143-144: where did the values for the parameter to distribute litter inputs over metabolic and structural litter come from? As this is based on litter quality, was this value different for different types of vegetation?

L143-158: more information about the chosen version of MIMICS is needed. What did the soil moisture scalar look like? This is important information, as you discuss the limited ability of MIMICS to account for moisture later in the manuscript. How was bioturbation simulated? Was the magnitude of diffusion the same for every land use and vegetation type? Were the same parameter values used for all simulated depths? How was, for example, the generally-observed decrease in the rate of SOC cycling with depth accounted for?

L181-182: In MIMICS, doesn’t the physically-protected SOC pool represent mineral-associated OC?

L183: this knowledge is not ‘recent’ but has been known for decades

Section 1.2.1: as you describe a new model (although based on MIMICS), I encourage the authors to describe the equations that they added to MIMICS in the main manuscript, not only in the supplement.

Section 2.2.1: did you check for autocorrelation between the predictors before performing the analyses?

L223-225: did this ‘negative exponential function’ have different parameters for different vegetation types?

L233: weighted based on what?

L234: what do you mean by ‘standardized’?

L235: ‘mean annual values’: over which time period where these means calculated?

L248: what is ‘SOC profile data’? Only OC%, or also bulk density, to convert OC% into stocks?

L255: so all profiles that were used had data down to 1.5 m? Please clarify this in the manuscript

L257: what happened to 1x1 km grid cells that did not have any profiles in WoSIS? Were these cells not simulated?

L262: It is not clear how a cumulative SOC content (in g/kg) can be calculated, as you can only calculate a cumulative profile by summing absolute values, not concentrations. Please clarify.

L266-267: If no data below 60 cm was present, where these deeper layers also simulated? How did you resample the data to 0-30, 30-60, … intervals? Were OC% weighted based on bulk density? Or were averages of the OC% used? Please clarify.

Fig. 1: Why is there a sharp boundary latitude below which no data are present?

Section 2.3:

• More information about the parameter optimization process is needed:
  • Which data were used to optimize model parameters? Only total SOC%? If so, how can you assess if the distribution of simulated SOC over the different model pools was correct?
  • Which measure was optimized? RMSE?
  • Which program was used to optimize model parameters?
• Did you use one constant desorption parameter for the entire globe, or per ‘cluster’? If so, the turnover rate of SOC is very unlikely to be simulated correctly, as this is different in different ecosystems/soil types/… Please clarify.
• Please provide the optimized values of the calibrated parameters.

L285: what do you mean by ‘relatively sensitive’? How was this sensitivity calculated?

L300: what do you mean by ‘for reference’

L302-303: R2 is not a measure for model performance, and should not be used as such (see above). Please remove and perform the analysis without this measure.

Results

Section 3.1: scatterplots (measured vs modelled) would be necessary to evaluate model performance. I encourage the authors to provide these.

L352: Clarify what you mean by ‘out-of-sample’

L357-358: how do you see in Table 2 that that the predictability decreases with depth, since all measures decrease with depth? Better would be to use relative measures (scaled by the measured OC%).

Table 2: I encourage the authors to remove R2 from the table, as this is not a measure of model performance (a very bad simulation, for example a systematic over- or underestimation, can have an R2 value close to 1).

Fig. 3: how is it possible that in a) NPP has a ranking far below AMT, while in b) (same data but fewer predictor variables) this order is reversed?

Fig. 3: Is the variable Soil Temp different from AMT? Why is this variable not present in plot a)? Please clarify.

L547-549: While this is partially true, there is still a substantial correlation between clay+silt content and maximum potential mineral-associated SOC (see https://doi.org/10.5194/soil-10-275-2024). I encourage the authors to put some nuance to this statement.

Discussion

L503 + 511 + 517-518: Please remove these statements or use a performance measure different than R2 (see above)

L534: how should this be done? A substantial part of CEC comes from organic matter itself (see for example Solly et al., which you cite), which is not discussed in the manuscript. Soils with a high SOM content typically have a higher CEC value, so it seems that including CEC as a model driver will not improve the mechanistic basis for these models. Please discuss.

Technical feedback

L50: ‘is’ => ‘was’

L77: ‘progress’ => ‘process’?

L78: ‘limited’ => ‘few’?

L88-89: something seems wrong with this sentence

L138: ‘variable’ => ‘variables’

You use AMT as an abbreviation for mean annual temperature, why not stick to the conventional MAT abbreviation?

L299: ‘permafrost’ => ‘frozen’?

L554: ‘perform’ => ‘performs’

Wang et al. compare the performance of mechanistic SOC models and machine learning (ML) models on capturing the spatial variations of global SOC across different soil layers. Their findings reveal that some key environmental facotrs controlling SOC are still missing or have not been well represented in existing mechanistic SOC models. The topic of this study is very interesting and their findings provide valuable insights for improving existing mechanistic models. The manuscript is well organized and the results are well represented.

My main concerns are related to the inadequate description on the data and methods used in this study, as well as the interpretation of some analysis results:

1）Vegetation type strongly affects the stoichiometry and physiochemical property of litter input. Do you consider the vegetation type at each observation site when dividing the NPP into different litter pools? If yes, please give a description in the Method section. In addition, have you tested the RF model which includes the land use type as an predicting factor?

2）Both PVI and SHAP can be used to identify the relative importance of different environmental factors on controlling SOC. Why PVI but not SHAP is used here? Will the key predictors for SOC will be different if the authors use the SHAP approach?

3）I am wondering whether the analysis based on ML approach can provide the reliable implications on the controlling factors and underlying mechanism associated with the formation and decomposition of SOC. For example, based on the results in Fig. 3, NPP seems to be less important than climatic factors on predicting SOC. Ignoring NPP in ML model may have a small effect on the simulated SOC. But should we ignore the NPP in the process-based model? In reality, NPP cannot be ignored at all even it ranks last in the PVI analysis. We still should develop process-based SOC models based on the understandings in SOC formation and decomposition processes, rather than based on the statistical importance of environmental factors. My concern is that what implications from the  analysis using ML are true in reality, or what implications should be incorporated into the process-based model?

4）Although this study did a analysis on the environmental controls of SOC and the related mechanism. There are still large limitation & uncertainties in the methods and data used in this study. I strongly the authors add a section to discuss the limitation and potential uncertainties in this study.

Minor comments:

L22: The poor performance

L40: ## between plant carbon inputs and outputs through soil heterotrophic respiration and leaching ##. Or, ## between carbon inputs in forms of plant litter and root exudates and outputs through soil heterotrophic respiration and leaching ##.

Fig. 2 Why there is no observation sites in the south Hemisphere? Or it is because the map in Figure 2 has not been correctly showed?

Fig.3 Terrain factor is only represented by elevation. Have the authors tried to include more factors such as slope steepness?

Fig. 5. It seems the relationships between SOC and NPP revealed by MIMICS and MES-C model are also nonlinear. SOC increases with NPP quickly when NPP is smaller than 1000 g C m-2 yr-1, then increases slowly with increasing NPP.

Overall Comments:

This work compares drivers of soil organic carbon in observations (using two random forest models) and in two mechanistic models (MIMICS and the new MES-C model) with the goal of identifying poorly represented or missing drivers in mechanistic models. Overall, I think the goal of this work is admirable and the authors present some interesting results highlighting CEC as a missing driver and different types of relationships between NPP and SOC for observations (nonlinear) and models (linear) which provide areas for models to improve. The paper would benefit from clearer language, more detailed methods, and some re-organization. My comments below explain in more detail but I think the way the observations/random forest models are referred to needs to be more consistent and clear for readers; the choice of which models to present in each figure needs to be clarified; the description of MES-C (which seems to be a brand new model) needs to be much more detailed; and the discussion would benefit from a reorganization to either combine with the results or differentiate itself from the results to focus on more high level findings. I think this paper has a lot of great information in it that will be useful to readers once there is some more detail and organization.

Specific Comments: 

Title: I feel this title is a little too broad for the paper scope, especially since only two relatively similar models are investigated. Perhaps something a bit more specific either about comparing to an ML model or about environmental drivers? For example, the title could be: “Comparative ability of two mechanistic soil biogeochemical models in representing environmental drivers of soil organic carbon” or something along those lines.

Abstract: It would be helpful to mention the scale of the study (globally-distributed) in the abstract.

Lines 50-51: I am not sure I agree with this statement. We’ve seen a proliferation of microbially-explicit SOC models in the last decade, many of which are cited throughout the introduction. Perhaps it’s better to highlight that these models are relatively under tested, especially compared to Century or RothC.

Line 69: It could be helpful here to mention that we might have more confidence in microbially-explicit models, given they represent microbes, which we know empirically to be important to SOC cycling. See Bradford et al., 2016 for commentary on having confidence in model projections: Bradford, M.A., Wieder, W.R., Bonan, G.B., Fierer, N., Raymond, P.A., Crowther, T.W., 2016. Managing uncertainty in soil carbon feedbacks to climate change. Nature Climate Change 6, 751-758.

Paragraph at line 104: I appreciate this paragraph overviewing benefits and drawbacks of ML/AI. One thing I feel is missing here is finding a sufficiently large training dataset to use AI/ML techniques to predict effects under climate change or conditions they are poorly trained on. Process-based models assume the underlying processes/equations won’t drastically change under novel conditions, such that equations in that model will still hold in unseen conditions, but my understanding is that AI/ML techniques don’t have that same strength. I admittedly have more knowledge on the process-based modeling side here, so I would be curious if the authors think this should be addressed.

Line 148: More current versions of MIMICS use reverse M-M kinetics (see Wieder et al., 2018 and forward). This could be considered in the discussion.

Wieder, W.R., Hartman, M.D., Sulman, B.N., Wang, Y.P., Koven, C.D., Bonan, G.B., 2018. Carbon cycle confidence and uncertainty: Exploring variation among soil biogeochemical models. Global change biology 24, 1563-1579.

Line 183: It seems MES-C is a new model the team developed – is that correct? I found it a bit surprising to have a new model in this work and not have that introduced in the introduction. Assuming this is indeed a new model, it would be helpful to have much more information on philosophy of the model, the equations that make up the model, what environmental variables it uses, and how parameter values were determined.

Lines 183-189: I am struggling with the fact that there is no particulate organic matter-like pool in MES-C. Could the authors address this? In the current structure there is no representation of “lower quality” or “chemically recalcitrant” soil organic matter, which is a considerable proportion of soil organic matter in many ecosystems.

Table 1: I suggest using MAP and MAT instead of AP and AMT, respectively, to make it easier for the reader to understand these drivers in the results.

Section 2.3: It seems the authors are choosing different parameter sets for each of the 12 clusters – is that correct? Parameters in both models have environmental drivers that modify fluxes and so these additional changes to parameters based on environmental differences seem a bit redundant. However, I also see how it could improve model fit. To that end, I would appreciate the authors reflecting on this choice and if kept, adding justification to the methods.

Line 304: Nice, I like this AIC approach for accounting for different numbers of predictors!

Line 314: Why not RF_env?

Line 315: I believe the following sections are the XAI techniques right? Could that be said explicitly here for the reader?

Line 352: Here is where I think the random forests models of SOC for comparison to MIMICS and MES-C and the random forest models used for XAI, that the reader just finished reading about, get confusing. In addition, the random forest models for SOC are later in this paragraph called “ML models”. I think using consistent, clear and distinct language is going to be needed for the reader. I suggest either calling the initial random forest models “ML models” consistently or only referring to them as RFenv and RFinp so its super clear to the reader which random forest models are being referred to at any given time.

Table 2: Please clarify your SOC units (and thus the MAE and RMSE units). Grams of what per kilogram of what? Please clarify this throughout the paper.

Figure 3 and associated text: Panel (a) is RFenv and (b) is RFinp right? As noted above, please use consistent language to help the reader. I am also wondering why (a) is stylized differently than the other panels? I understand why (a) might be vertical for space reasons but I find it a little misleading that is looks different when its displaying the same information. I believe panels b-c should be colored as panel a for consistency and to help the reader understand that these plots are showing the same things for different models.

Figure 4 and associated text: Why is RFenv missing here?

Lines 395-397: Please be careful to not use causal language here and throughout. Instead, the authors could say: higher soil temperature was associated with lower SOC and higher silt and clay was associated with higher SOC.

Lines 395-404: I wonder if simple linear regression would aid in interpretation of these plots. Currently, it seems interpretation is based on the smoothness of the color gradient, and it could be more objective to use r-squared values here.

Section 3.4: Which observed data is presented here? RFenv or RFinp?

Figure 5: Could the short ticks at the bottom of each axis be described more? Which percentiles do they represent?  Also, could the authors justify why high SOC values are not shown?

Section 4.2: It could be useful to comment in this section that CEC is not as widely or easily measured as clay and silt, making it harder to use in models applied at site scales where site data is generally used for model inputs. Thus, it might be suggested to measure that more frequently.

Like 563: Is the weaker response or sharp decrease more pronounced in MES-C?

Lines 563-567: I am not sure I agree with this. The authors seem to be conflating steady state associations between SOC and temperature with transient sensitivities, and I am not sure those would necessarily be the same.

Lines 583-586: I suggest softening this language since Cotrufo et al., 2013 is a prediction with mixed support in the literature.

Lines 591-594: How was litter quality handled in this study as a model input? Did it vary with space? The quality of litter is a very influential input in MIMICS (and presumably MES-C given their similar structure) and so addressing this input would be important here.

Discussion, overall: There seems to be quite a bit of restating the results in the discussion – I wonder if these could be combined to reduce repetition? Or, if not, could the discussion try to be a bit broader, focusing on consistent findings across methods, rather than covering each method individually, which lends itself to a restating of the results?

Line 647: I think this paper would benefit from a “Limitations” section that addresses the fact that only two SOC models that are relatively similar were addressed here and that it seems an important driver in at least MIMICS and maybe also MES-C is not evaluated (e.g., litter quality).

Line 649: Two random forest models?

Line 658: MIMICS doesn’t have pH representation – could that be stated clearly here?