Closure Methodologies

flux-data-qaqc currently provides two routines which ultimately adjust turbulent fluxes in order to improve energy balance closure of eddy covariance tower data, the Energy Balance Ratio and the Bowen Ratio method.

Closure methods are assigned as keyword arguments to the QaQc.correct_data method, and for a list of provided closure options see QaQc.corr_methods. For example, if you would like to run the Bowen Ratio correction routine assuming you have succesfully created a QaQc object,

# q is a QaQc instance
q.correct_data(meth='br')

The other keyword argument for QaQc.correct_data allows for gap filling corrected evapotranspiration (\(ET\)) which is calculated from corrected latent energy (\(LE\)). By default the gap filling option is set to True, more details on this below in Step 9, optionally gap fill corrected ET using gridMET reference ET and reference ET fraction.

Tip

All interactive visualizations in this page were created using Plot.line_plot, Plot.add_lines, and Plot.scatter_plot which automatically handle issues with utilizing the mouse hover tooltips and other bokeh.plotting.figure.Figure features.

Data description

The data for this example comes from the “Twitchell Alfalfa” AmeriFlux eddy covariance flux tower site in California. The site is located in alfalfa fields and exhibits a mild Mediterranean climate with dry and hot summers, for more information on this site or to download data click here.

Energy Balance Ratio method

The Energy Balance Ratio method (default) is modified from the FLUXNET methodology (step 3 daily heat processing). The method involves filtering out of extreme values of the daily Energy Balance Ratio time series, smoothing, and gap filling. Then the inverse of the filtered and smoothed time series is used as a series of correction factors for the initial time series of latent energy (\(LE\)) and sensible heat (\(H\)) flux time series.

All steps, abbreviated

Below is a step-by-step description of the Energy Balance Ratio correction routine used by flux-data-qaqc. More details and visual demonstration of steps are shown below.

Step 0 (optional): optionally filter out poor quality data first if quality control (QC) values or flags are provided with the dataset or other means. For example, FLUXNET data includes QC values for \(H\) and \(LE\), e.g. H_F_MDS_QC and LE_F_MDS_QC are QC values for gap filled \(H\) and \(LE\). This allows for manual pre-QaQc of data.

Step 1: calculate the Energy Balance Ratio (EBR = \(\frac{H + LE}{Rn – G}\)) daily time series from raw data.

Step 2: filter EBR values that are outside 1.5 times the interquartile range.

Step 3: for each day in the daily time series of filtered EBR, a sliding window of +/- 7 days (15 days) is used to select up to 15 values.

Step 4: for each day take a percentile (default 50) of the 15 EBR values. Check if the inverse of the EBR value is \(> |2|\) or if the the inverse of the ratio multiplied by the measured \(LE\) would result in a flux greater than 850 or less than -100 \(w/m^2\), if so leave a gap for filling later.

Step 5: if less than +/- 5 days exist in the sliding 15 day window, use the mean EBR for all days in a +/- 5 day (11 day) sliding window. Apply same criteria for an extreme EBR value as in step 4.

Step 6: if no EBR data exist in the +/- 5 sliding window to average, fill remaining gaps of EBR with the mean from a +/- 5 day sliding window over the day of year mean for all years on record, i.e. 5 day climatology. Calculate the 5 day climatology from the filtered and smoothed EBR as produced from step 5. Apply same criteria for an extreme EBR value as in steps 4 and 5.

Step 7: use the filtered EBR time series from previous steps to correct \(LE\) and \(H\) by multiplying by the energy balance closure correction factor \({EBC_{CF}} = \frac{1}{EBR}\), where EBR has been filtered by the previous steps. Use the corrected \(LE\) and \(H\) to calculate the corrected EBR.

Step 8: calculate corrected \(ET\) from corrected \(LE\) using average air temperature to adjust the latent heat of vaporization.

Step 9 (optional): if desired, fill remaining gaps in the corrected \(ET\) time series with \(ET\) that is calculated by gridMET reference \(ET\) (\(ETr\) or \(ETo\)) multiplied by the filtered and smoothed fraction of reference ET (\(ETrF\) or \(EToF\)).

Step 0, manual cleaning of poor quality data

Below we can see that the daily time series of net radiation (\(Rn\)) has some periods of poor quality data. This is a common issue due, e.g. to instrumentation problems, that cannot always be avoided. In this case the sensor did not record values at night (or they were not provided with the data) when \(Rn\) values are lower for several days (e.g. around 8/26/2014) which resulted in overestimates of daily mean \(Rn\) during these periods. Although these days can automatically be filtered out by the QaQc class, the example below shows a way of manually filtering them because in other cases outliers in the daily data may not be caused by resampling of sub-daily data with systematic measurement gaps. The main point is that manual inspection and potentially pre-filtering of poor quality data before proceeding with energy balance closure corrections is often necessary.

There are several ways to conduct manual pre-filtering of poor quality meterological time series data, to filter data based on input quality flags or numeric quality values see Quality-based data filtering.

flux-data-qaqc also allows for filtering of poor quality data on the fly as shown in this example. In other words, we simply filter out the periods we think have bad data for \(Rn\) within Python before running the closure correction. After manually determing the date periods with poor quality \(Rn\), here is how they were filtered oiut before running the correction:

>>> import pandas as pd
>>> import numpy as np
>>> from fluxdataqaqc import Data, QaQc
>>> d = Data('Path/to/config.ini')
>>> # days with sub daily gaps can be filtered out automatically here,
>>> # see "Tip" below the following plot
>>> q = QaQc(d, drop_gaps=False)
>>> # rename dataframe columns for ease of variable access, adjust
>>> df = q.df.rename(columns=q.inv_map)

Here were the dates chosen and one way to filter them,

>>> # make a QC flag column for Rn
>>> df['Rn_qc'] = 'good'
>>> df.loc[pd.date_range('2/10/2014','2/10/2014'), 'Rn_qc'] = 'bad'
>>> df.loc[pd.date_range('8/25/2014','9/18/2014'), 'Rn_qc'] = 'bad'
>>> df.loc[pd.date_range('10/21/2015','10/26/2015'), 'Rn_qc'] = 'bad'
>>> df.loc[pd.date_range('10/28/2015','11/1/2015'), 'Rn_qc'] = 'bad'
>>> df.loc[pd.date_range('7/23/2016','7/23/2016'), 'Rn_qc'] = 'bad'
>>> df.loc[pd.date_range('9/22/2016','9/22/2016'), 'Rn_qc'] = 'bad'
>>> df.loc[pd.date_range('3/3/2017','3/3/2017'), 'Rn_qc'] = 'bad'
>>> # filter (make null) based on our QC flag column for Rn
>>> df.loc[df.Rn_qc == 'bad', 'Rn'] = np.nan
>>> # reassign to use pre-filtered data for corrections
>>> q.df = df

The resulting energy balance component plot with \(Rn\) filtered:

Tip

In this case, the issues with \(Rn\) were caused by resampling 30 minute data with systematic night-time gaps. These sort of issues can be automatically handled when creating a QaQc object; the keyword arguments drop_gaps and daily_frac to the QaQc class are used to automatically filter out days with measurement gaps of varying size, i.e.,

>>> d = Data('path/to/config.ini')
>>> q = QaQc(d, drop_gaps=True, daily_frac=0.8)
>>> q.correct_data()

This would produce very similar energy balance closure results as the manual filter above. Another more fine-grained option would have been to flag the days with gaps in the sub-daily input time series that you would like to filter by Data.apply_qc_flags.

Note

The remaining step-by-step explanation in this page uses the pre-filtered input time series, however results of the energy balance closure correction without pre-filtering outliers of \(Rn\) are also shown in plots for the final steps (8 and 9) for comparison. If you now ran:

>>> q.df = df
>>> q.correct_data()
>>> q.plot(output_type='show')

This will directly produce the same output of step 9 using the pre-filtered data.

Steps 1 and 2, filtering outliers of EBR

Calculate daily EBR = \(\frac{H + LE}{Rn - G}\) time series and filter out extreme values that are outside 1.5 the interquartile range. Note, in flux-data-qaqc this is named as “ebr”.