The DEnKF is based on the Kalman filter analysis equation, of which various flavours have had general success in state estimation in other marine BGC DA problems (e.g. Hu et al., 2012; Ciavatta et al., 2016). The derivation of the DEnKF is given in Sakov and Oke (2008) and is a modification to the traditional Kalman filter equation: xa=xf+K(Y-Hxf), where x is the model state, Y is the vector of observations, H is the observation operator and the superscripts of “a” and “f” denote the analysis and forecast fields respectively. In this study, we only use a subset of the state variables in the DEnKF update; those variables not included in the state vectors denoted in Table 1 are not altered by the state update. The forecast innovations (F) are defined by F=(Y-Hxf). The Kalman gain matrix, K, is given by K=lPfHTHPfHT+R-1, where l is the localisation operator, applied in the form of covariance localisation (Gaspari and Cohn, 1999; Sakov and Bertino, 2011), with an isotropic localisation radius of 60 km, and R is the observation error covariance matrix. Off diagonal elements of R are set to 0, and the diagonal elements of R are the sum of the representation error, difference in kind error and analytical measurement error, termed Etot. These error terms are discussed in detail in Sect. 2.5.2 (additionally see Schaeffer et al., 2016, for a further discussion of these error sources relating to BGC variables and methods to estimate them from glider data).

The background error covariance matrix, P, is given by Pf=1m-1∑i=1mXif-xfXif-xfT=1m-1AAT, where m is the ensemble size and i denotes the ith member of the ensemble. The background error covariance is approximated by a 36-member dynamic ensemble, where Xif is the ith ensemble member and x is the ensemble mean. We acknowledge that a 36-member ensemble is small, but, as demonstrated in the results section, this small ensemble performs adequately for the assimilation of a univariate observation type. The ensemble size will need to be increased to assimilate multivariate observations. To avoid negative values and normalise the state, we log-transform the state before forming the state vector. The application of the log-transform to biological variables is discussed in Parslow et al. (2013) and is commonly used in other BGC assimilation schemes (e.g. Ciavatta et al., 2014). The background error covariance matrix is never computed; rather a series of anomaly fields are constructed and denoted by A. We then construct the Kalman gain matrix in the observation subspace as per Sakov and Oke (2008): A=[A1…Am], where the ith anomaly field is given by Aif=Xif-xf, where x is the ensemble mean and is updated via Eq. (5) and each anomaly field is updated by Aia=Aif-12KHAf. The full analysed ensemble is then given by Xia=Aia+xa. The assimilation system iterates through time using a five-day forecast duration. The assimilation system is cycled by calculating the analysis fields at time t, using the forecast from the previous cycle, Xf(t), and observations Y(t) at time t, using all observations that fall within a window of t ±3 h. The numerical model is initialised using the analysis fields Xa(t) and the next 5-day forecast is made. This forecast at t+5 days, Xf(t+5), is then used in the next assimilation cycle.

The DEnKF requires the ensemble to be perturbed in such a way that it captures the main source of error. These perturbations are introduced in a way that captures our prior understanding of the dominant errors. In this system we expect that errors will stem from uncertainty in the initial conditions (ICs) as per most assimilation systems. Additional sources of error can stem from uncertainty associated with BGC process parameters, which has been discussed at length in Parslow et al. (2013), and river boundary conditions. In this study we have ignored the effects of uncertainty that propagate from the model physics (hydrodynamics), short-wave radiation forcing and open ocean boundary fluxes of BGC tracers, but we do acknowledge that they contribute to the overall uncertainty in the predicted system state.

In the context of this study, we have introduced perturbations to the ensemble by sampling ICs randomly from a 4-year run (where T= 4 years) of the BGC model: X(t=0)i∼UniformX(t=0),X(t=T)fori=2…m, where X(t=0)i is the initial condition for the model state for the ith member sampled for a uniform distribution with no replacement, and X(t=0)i=1 is the ensemble mean. Sensitivity experiments have shown the model is sensitive to perturbations in the quadratic zooplankton mortality rate coefficient for small (mQ,ZS) and large (mQ,ZL) zooplankton with units of d-1 (mg N m-3)-1. These are considered system parameters and are as such uncertain. To this end we have perturbed the ensemble by sampling space- and time-invariant parameters from mQ,ZL,i∼LN0.012,1fori=2…m,mQ,ZS,i∼LN0.007,1fori=2…m, where LN is a log-normal distribution, and respective means of 0.012 and 0.007 are values as used in the control run (Baird et al., 2016a) and are given relatively broad standard deviations of 1 for both parameters. The river nutrient and sediment loads were altered by a time-invariant scaling factor (θ) to all rivers: θNO3,i∼N1,0.3fori=2…m,θNH4,i∼N1,0.3fori=2…m,θDIP,i∼N1,0.3fori=2…m,θFineSed,i∼N1,0.3fori=2…m, where N is a normal distribution truncated at 0. Each ensemble member has their load (Qi) scaled according to Qi=θiQcontrol, where Qcontrol is the load entering in the control run.

The assimilation system was assessed using five assimilation system configurations (Table 1) using a subset of the full model state in the assimilation state vector and corresponding diagonal elements (Etot; see below) of the observation error covariance matrix (R). The assimilation experiments were conducted on Raijin, a super-computer hosted at the National Computational Infrastructure (NCI; http://nci.org.au/). The assimilation cycle progressed via a two-step process. The first step generated the forecast fields by integrating the ensemble forwards on 36 nodes of the system. Each node has a dual 8 core processor, with 16 Gb of RAM. This step takes approximately 90 min to simulate 5 model days. The second step generates the analysis fields and updates model scripts to allow for the next forecast cycle. The analysis step is undertaken on a single high-memory node and requires 800 Gb of RAM and takes 50 min using 16 cores.

We only allow the observations to update the variables contained in the assimilation state vector. The analysed assimilation state vector is then inserted into the full model state vector.

There are three sources of observational error (Etot) that must be accounted for when relating remotely sensed observations to a modelled state variable:

Representation errors (ER) arise due to the approximation that the modelled tracer quantities are an average over a whole model cell. This can be thought of as unresolved spatial variability. In the case of the MODIS observations, we set ER to be the standard deviation of all the 1 km log-scaled observations that fall within a 4 km model grid cell.

The sum of these error types can be large and forms the diagonal element of the observation error covariance matrix (R). The larger Etot is the lower the impact of the observations in the assimilation system. If we can remove the difference in kind error (ED), then we only have representation error and analytical/sensor/processing error. Given that Level 3 OC remote-sensing products rely on empirical/statistical relationships, ED dominates; therefore if we can minimise ED (or remove it entirely), then the information content of the observations increases. Conversely, if we do not have a large enough observation error, we run the risk of overfitting the observations and generating unrealistically large increments for the unobserved model state variables.

Experiment 1 (EXP1) was designed to test the assimilation system under the assumption that modelled surface total Chl a (the sum of small and large phytoplankton Chl a, benthic Chl a and Trichodesmium Chl a) was equivalent to the Chl a from MODIS-observed OC3M. This experiment is analogous to those of Natvik and Evensen (2003), Gregg (2008) and Ford et al. (2012). This is a reasonable assumption in offshore waters, but OC3M is known to be unreliable in coastal waters where sediments (e.g. TSS), bottom reflectance and CDOM cause artificially high OC3M values. We expect that EXP1 will contain difference in kind errors that stem from the assumption that surface Chl a is equivalent to observed OC3M. The observation error prescribed for this experiment is considered to be the lower bound of OC3M errors as reported in Qin et al. (2007).

In experiments 2–4 (EXP2-4) we assume that the simulated OC3M (i.e. calculated from simulated RSR) is equivalent to the observed OC3M (i.e. calculated from observed RSR) and it is used as an input into the observation operator. The simulated OC3M (Eq. 21) contains the signature from simulated TSS, CDOM and bottom reflectance as per Sects. 2.3, 2.4 and Appendix A. Thus for EXP2–4, the observed and simulated OC3M contain no difference-in-kind errors (although we leave ED high during EXP2 for experimental purposes). In other words, the configuration used for EXP2 is the same as EXP1, except the effect of difference in kind error has been removed by using simulated OC3M, in place of total surface Chl a. In EXP3, we reduce the observation error to account for the reduction in ED, and in EXP4 we add additional variables to the assimilation state vector.

In EXP5, we assimilate observed RSR at 551 nm (R551) using the simulated RSR at the equivalent wavelength. Typically errors in RSR as reported in Schroeder et al. (2007) are in the order of 10–20 %, and we have used 0.2 for this experiment.

In all experiments we have generated a set of super-observations (Cummings, 2005; Oke et al., 2008b) by spatially averaging the 1 km observations onto the 4 km model grid. Prior to generating the super-observations in experiments 1-4 (EXP1-4), the observed atmospherically corrected RSRs (Schroeder et al., 2007) at 443, 488 and 551 nm were transformed into a single observation using the OC3M algorithm: OC3M=10(a0+a1.B+a2.B2+a3.B3+a4.B4), where a0, a1, … a4 are a set of empirically determined coefficients (e.g. NOMAD version 2, http://seabass.gsfc.nasa.gov/wiki/article.cgi?article=NOMAD) with the respective values of a0= 0.2424, a1=-2.7423, a2= 1.8017, a3= 0.0015 and a4=-1.2280 (http://oceancolor.gsfc.nasa.gov/cms/atbd/chlor_a). B is B=log10(Rrs,λ1Rrs,λ2). Rrs,λ1 and Rrs,λ2 are determined by the absolute magnitude (whichever is greater) of the remote-sensing reflectance, and Rrs,λ1 is either the band centred on 443 or 488 nm and Rrs,λ2 is the band centred on 551 nm. We apply the OC3M algorithm (Eq. 21) to both the observed and simulated RSRs.

The assimilation system preserves the stoichiometry of the small and large phytoplankton (referred to as PhyS Chl a and PhyL Chl a respectively) as follows. In the BGC model, each phytoplankton cell (small, large, benthic or Trichodesmium) is represented by a quantity of structural material, B, and reserves of nitrogen, RN, reserves of phosphorus, RP, reserves of energy, RI, and an intracellular chlorophyll a concentration, ci. Our intention in the assimilation is to change the number of cells, as quantified by B, not the physiological status of the cell, as represented by RN, RP, RI and ci. Since the reserves are quantified as the total of these reserves across the entire population, each of the reserves is changed by the same proportion as the biomass. Once the analysed quantity is determined (e.g. PhyL Chl a and PhyS Chl a), the quantities of RN, RP, RI and B are updated such that the respective ratios prior to assimilation are preserved.

The forecast innovations (Eq. 6) for the five assimilation system configuration experiments are shown in Fig. 6:

EXP 1 (black lines, Fig. 6) assumes that total surface Chl a is equivalent to observed OC3M and is used to calculate the forecast innovations, which are then used to update small and large phytoplankton Chl a. An 80 % error in the ANN-observed OC3M observation is prescribed on the diagonal elements of R.

Of the experiments that assimilated OC3M, the configuration used for EXP4 (Fig. 6, blue line) gave the lowest mean absolute forecast innovation. While the EXP5 configuration (noting that this experiment assimilates R551, not OC3M) beat EXP4 with a lower mean absolute innovation, these innovations are for R551, not OC3M, and therefore cannot be directly compared. An assessment against independent (non-assimilated) in situ observations is given in Sect. 3.2.2–3.2.5 to assess the relative performance between the non-assimilating control run and EXP1 (glider only), EXP2, EXP3 (glider only), EXP4 and EXP5.

The control and assimilating runs were compared to withheld data obtained from an ocean glider that was deployed on 26 May 2013, and recovered on 4 August 2013. The glider track largely followed the shelf break and headed in a south-easterly direction. To make a comparison between glider observations and the model, we take a subsample of glider observations centred at the time of model output, with a time window of 2 h. For each glider observation that falls within this time period, we find a corresponding 3-D cell from the model and extract the equivalent model solution.

A persistent feature of the observed Chl a when interpolated onto the model time–space grid (Fig. 7, top panel) is the relatively low values in the upper 100 m of water column. For much of the record, there is a persistent weak deep chlorophyll maximum (DCM) that is centred at between 80 and 120 m depth. Rarely do concentrations in the upper 80 m exceed 0.5 mg m-3. When the control run is examined, Chl a in the top 80 m regularly exceeds 1 mg m-3, and DCMs, when they exist, are located between 30 and 60 m deep. A detailed analysis of the control run demonstrates the model is able to reliably produce DCMs (contained in the skill assessment report at https://research.csiro.au/ereefs/models), but, in this particular location in time and space, the model does not generate one consistent with the observations. Using the EXP1 configuration, the assimilation system does not improve the solution when compared with the control run. The EXP2 configuration marginally improves the solution by reducing the mixed layer Chl a. However, using the EXP4 and EXP5 configurations, the assimilation system substantially improves on the control when compared to the glider observations. The difference between the EXP2 and EXP4 configuration is the reduction in observation error and the inclusion of additional variables in the assimilation state vector. Regardless of the assimilation configuration used, the assimilation system cannot place the DCM in the correct location because the remote-sensing observations provide no information about such a deep feature. The remote-sensing observations do remove the bias in the upper 80 m with concentrations in the assimilating run ranging between 0.1 and 0.3 mg m-3 (Fig. 7). The configuration used in EXP5 captures a weak DCM that persists for 7 days from 7 to 14 July 2013 and also maintains the lowest mixed layer Chl a for the feature captured on from the 7–13 July 2013 (Fig. 7: bottom panel). Assimilating OC3M still overpredicts the mixed layer Chl a.

A comparison between non-interpolated individual profiles from the glider and equivalent sampling of the model shows substantial unresolved subgrid-scale variability (or instrument noise). Figure 8 (top left) shows that in the upper 70 m the observed Chl a as measured by the fluorometer ranges between 0.14 and 0.3 mg m-3, with a mean value of 0.18 mg m-3. There is no quenching evident in this or other profiles, as the observed fluorescence and backscatter rates are constant in the upper 50 m. Within each of the subplots, the glider profiles collected over the 2 h window may sample multiple model grid cells. However, due to the relatively slow horizontal glider speed, these glider profiles fall within two adjacent model cells. In most cases, the control and assimilating runs gave indistinguishable solutions between adjacent cells with the exception of Fig. 8 (top left). In all cases the assimilation of observations of both OC3M and R551 has reduced the error in the simulated Chl a when assessed against independent glider data. The noise in the fluorometer observations appears to range between 0.04 and 0.08 mg m-3.

The Chl a root mean square difference (RMSD) for each layer of the model, and aggregated in time for all glider observations, is shown in Fig. 9. Above 80 m, there is a substantial reduction in RMSD between the control (cyan) and assimilating runs (EXP2-red, EXP4-blue and EXP5-magenta). EXP3 behaved similarly to EXP2. The RMSD from EXP1 (dashed black) is marginally higher than the RMSD in the control run. The assimilating runs (EXP4 and EXP5) have an RMSD of between 0.10 and 0.17 mg m-3 compared with 0.30 to 0.41 mg m-3 in the control run. In the upper 60 m, the assimilation of R551 (EXP5) results in a minor improvement in RMSD when compared with the assimilation of OC3M (EXP4). Below 80 m, the RMSD profile is similar between the control and assimilating runs. However, the assimilation of R551 results in the lowest RMSD below 80 m. The largest impact of assimilating remote-sensing observations is in the top 80 m of the water column.

The AIMS moorings were deployed at the 13 GBRMPA MMP sites in the shallow inshore regions of the Great Barrier Reef Lagoon (Fig. 1). The control run typically had RMSDs of between 0.4 and 0.6 mg m-3 for in situ Chl a (Fig. 10; upper panel). In all cases the EXP2 configuration performed worse than the control run. In most cases the EXP4 configuration reduced the RMSD of in situ Chl a by 0–10 % when compared with the control run. The assimilation of R551 (EXP5) was more variable. Some sites improved, whereas others were degraded by the assimilation of this observation type when compared with the control run.

The TSS RMSD varies widely across all the GBRMPA MMP sites (Fig. 10; lower panel), driven by the strong variation in magnitude of the spring–neap tidal forcing. The combination of the initialization of the ensemble and perturbed forcing caused the ensemble mean for EXP2 to have a substantially reduced RMSD when compared with the control run. The inclusion of the TSS constituents in the state vector in the assimilating model further reduced the RMSD, and has generated realistic time-varying correlations between the observed OC3M/R551 and inshore TSS. These cross correlations allow for the correction of simulated TSS from OC3M/R551 observations. The performances of EXP4 and EXP5 were indistinguishable, but the inclusion of TSS in the assimilation state vector was beneficial.

Within the GBR region, there are two IMOS NRS sites (Yongala and North Stradbroke, NS, Fig. 1). The dissolved inorganic nutrients of NO3, NH4 and DIP are taken monthly. At Yongala, water samples are taken at the surface (0 m), 10 m, 20 m and bottom (26 m). At NS, samples are taken at the surface (0 m) and 10 m. It should be noted that there are only three to four samples per depth at each site during the 4-month simulation period. At Yongala, typical control run RMSDs for NO3 range from 5 to 12 mg m-3. Assimilating OC3M/R551 halved these errors using the EXP4/5 configurations. The improvement at NS is evident with improvements found in the upper 10 m of the water column, with EXP4 and EXP5 giving indistinguishable results. The EXP2 configuration yielded higher RMSDs than EXP4 and EXP5.

With the exception of the surface samples at Yongala, the assimilation system improved the prediction of NH4 at all depths for each site. Most notably was the 70 to 90 % reduction in RMSD at the deeper locations at Yongala. EXP4 and EXP5 outperformed EXP2 in all cases for NH4. There were marginal improvements to DIP, which displayed a 0 to 30 % reduction in RMSD across all sites. EXP4 resulted in lower DIP errors at NS, but EXP5 resulted in lower DIP errors at Yongala. EXP2 had the lowest DIP RMSDs.

The direct assimilation of RSR (EXP5: R551), or a function of RSR, f(RSR) (EXP4: OC3M), improved the model solution when compared with the control run against three independent non-assimilated in situ observational datasets. In most cases the EXP4 and EXP5 configurations outperformed the EXP2 configuration, which was to be expected due to the additional variables included in the assimilation state vector. The subtle differences between the two approaches will be discussed in Sect. 4. Based on these findings, the preferred assimilation system for state estimation on the GBR is that used in EXP4. A further examination of the EXP4 forecast error statistics and example forecast, analysis and increment field is given Sect. 3.3.

The sum of the surface Trichodesmium Chl a, PhyS Chl a and PhyL Chl a biomass differs substantially from the simulated OC3M as shown in Fig. 5. The assimilation system updates all of the state variables included in the assimilation state vector. Simulated OC3M is a diagnostic variable that is a function of all the optically active dynamic state variables as described in Sect. 2. To demonstrate the impact on the dynamic variables of PhyS and PhyL, results from the forecast step, and the assimilation update, are presented in Figs. 13, 14 and 15. Cycle 22 was chosen to demonstrate the spatial impact of the system as it is representative of the last 10 cycles and was relatively cloud free.

The simulated OC3M forecast field for cycle 22 (12 September 2013) displays elevated OC3M in the shallow near-shore environment throughout the whole of the GBR region and southern shelf of Papua New Guinea (PNG; Fig. 13). Additional features are elevated OC3M in the vicinity of the central and southern fringing reefs and a plume originating from the eastern region of PNG. Offshore oceanic waters generally have low OC3M of 0.2 mg m-3 or less. There is some evidence of mesoscale blooms in the northern and southern sections of the domain. Observed OC3M is overlaid on Fig. 13 (left). Where there is a difference in colour, the simulated OC3M differs from the observed OC3M.

The forecast surface layer fields for PhyS and PhyL appear substantially different to the simulated OC3M field (Fig. 13). The differences near the coast are where TSS and CDOM are known to cause artefacts in OC3M. While there are patchy blooms of small phytoplankton at various locations within the domain, rarely does the PhyS Chl a exceed 0.5 mg m-3. The exception to this is in the inner central coastal region of the GBR and in the vicinity of the Fly River plume on the south coast of PNG. Similarly, the PhyL Chl a remains very low for large areas of the domain; however, in regions with additional nutrient supply (e.g. in upwelling regions, mesoscale eddies and some river mouths) blooms do occur.

When the observed OC3M is assimilated, increment fields are calculated using Eq. (5) and are presented in Fig. 14 for simulated OC3M, PhyS Chl a and PhyL Chl a. The innovations are overlaid on the increment field for OC3M to give an indication of how well we are fitting the observations. In areas where the model overpredicts OC3M, the increments will be negative. In areas where the model underpredicts OC3M, the increments will be positive. The increments and innovations here are presented as a fractional change with respect to the background (forecast) field.

For this particular analysis cycle, it appears that the model is underestimating inshore OC3M by up to 10–30 % and overestimating OC3M by upwards of 50 % offshore (Fig. 14, left). By using the background ensemble correlation structure, the increments applied to PhyS biomass increase its concentration in the inner lagoon by up to 20 %, and substantially increase the PhyS biomass offshore of the central outer reefs by more than 50 % (Fig. 14, centre). It should be noted that the increments being applied to the background fields contain meso- and sub-mesoscale information. Significantly, features such as upwelling filaments, eddies and plumes are maintained through the assimilation procedure, demonstrating that they are allowed to dynamically evolve in the assimilation system. The increments applied to PhyL biomass (Fig. 14, right) differ substantially to those of PhyS biomass (Fig. 14, centre). For large areas of the domain, the assimilation system decreases the PhyL biomass by up to 50 %, whereas there are some areas in which it increases. These areas correspond to regions where a bloom may be occurring (there is a small westward shift in the major bloom off the Papua New Guinea coast). The increment applied to the central region of the domain, offshore of the outer reefs, is linear and coherent and likely a result of shifting a dynamic feature such as an upwelling-induced bloom to better match observations.

When the increments contained in Fig. 14 are applied to the forecast fields, the resulting analysis field for simulated OC3M better fits the observed OC3M, with a substantially reduced error inshore and in the vicinity of the outer reefs (Fig. 15). The difference between simulated and observed OC3M is small in the deeper offshore regions and shallow sections of the lagoon. The greatest error in OC3M occurs in the central lagoon and the outer reefs where spatial variability is highest. The corresponding analysis fields for small phytoplankton and large phytoplankton are contained in Fig. 15. There are elevated concentrations of small phytoplankton biomass in the near-shore region near river mouths and the outer fringing reefs. The large phytoplankton biomass is concentrated in the region of Broad Sound, the Fly River plume and the Papua New Guinea upwelling. Each of these features was predicted by the forecast, as little biomass is added or subtracted by the assimilation update. However, there is substantial removal of large phytoplankton biomass from the northern and central offshore regions. This leaves very little large phytoplankton biomass present in substantial areas of the domain during this particular analysis cycle.