CALIPSO Quality Statements:
Lidar Level 2 Cloud and Aerosol Profile Products
Version Releases: 3.01, 3.02
|
CALIPSO Quality Statements: |
|
This document provides a high-level quality assessment of the cloud and aerosol profile products derived from the CALIPSO lidar measurements, as described in sections 2.5 and 2.6 of the CALIPSO Data Products Catalog (Version 3.2) (PDF). As such, it represents the minimum information needed by scientists and researchers for appropriate and successful use of these data products. We strongly suggest that all authors, researchers, and reviewers of research papers review this document periodically, and familiarize themselves with the latest status before publishing any scientific papers using these data products.
These data quality summaries are published specifically to inform users of the accuracy of CALIOP data products as determined by the CALIPSO Science Team and Lidar Science Working Group (LSWG). This document is intended to briefly summarize key validation results; provide cautions in those areas where users might easily misinterpret the data; supply links to further information about the data products and the algorithms used to generate them; and offer information about planned algorithm revisions and data improvements.
The structure and content of the version 3 profile products is significantly different from previous releases. In particular, the aerosol profile products have been completely restructured, and now are reported on the same 5-km spatial grid as the cloud profile products. The horizontal resolution of the cloud and aerosol profile products is now identical to the horizontal resolution of the 5-km cloud and aerosol layer products, thus enabling a one-to-one match between the bulk optical properties of any layer (e.g., optical depth) and the profile data (e.g., extinction coefficients) from which the bulk properties were derived. The profile times and latitude/longitude coordinates are now reported identically in the 5-km profile products and the 5-km layer products. Uncertainty estimates are now provided for all profile data derived from the Level 1 lidar attenuated backscatter data, and several new diagnostic and quality assurance parameters have been added. The full extent of the changes made is identified and explained in the paragraphs below.
Because validation for different parameters can require different levels of effort, and because the uncertainties inherent in some retrievals can be substantially larger than in others, the maturity levels of the parameters reported in the profile products files are not uniform. Therefore, within this document, maturity levels are provided separately for each scientific data set (SDS) included with the data files. The data product maturity levels for the CALIPSO layer products are defined in the table below.
| Beta: | Early release products for users to gain familiarity with data formats and parameters. Users are strongly cautioned against the indiscriminate use of these data products as the basis for research findings, journal publications, and/or presentations. |
| Provisional: | Limited comparisons with independent sources have been made and obvious artifacts fixed. |
| Validated Stage 1: | Uncertainties are estimated from independent measurements at selected locations and times. |
| Validated Stage 2: | Uncertainties are estimated from more widely distributed independent measurements. |
| Validated Stage 3: | Uncertainties are estimated from independent measurements representing global conditions. |
| External: | Data are not CALIPSO measurements, but instead are either obtained from external sources (e.g., the Global Modeling and Assimilation Office (GMAO)) or fixed constants in the CALIPSO retrieval algorithm (e.g., the 532 nm calibration altitude). |
| NOTICE! |
| The backscatter and extinction coefficients reported in the version 3.01 release of the CALIPSO profile products are provisional data products. As such, they have been compared with a limited set of independent measurements and obvious errors have been repaired since version 2.01. The version 3 products also contain numerous data quality flags and uncertainty estimates that were not available in prior releases. Users of the profile products are highly encouraged to apply this new QA information as appropriate when conducting scientific analyses using the CALIPSO data products. |
The CALIPSO Cloud and Aerosol Profile Products report profiles of particle extinction and backscatter and additional profile information (e.g., particulate depolarization ratios) derived from these fundamental products. Layer optical depths are reported in the Cloud and Aerosol Layer Products. The layer optical depths are derived from the same retrievals that are used to compute the extinction and backscatter profiles in the profile products. All of these extinction products are produced using the same basic algorithm (Young and Vaughan, 2009).
There are layers for which the optical depth can be reliably measured directly from the CALIOP backscatter signal. For these layers, the measured optical depths are reported in the layer products, and the layer two-way transmittance is used to constrain the extinction solution, so that an optimal estimate of the layer lidar ratio is retrieved. These constrained solutions are the most reliable retrievals, and can be identified by examining the corresponding profile of extinction QC flags. The version 3 retrieval analysis has been improved to increase the amount of constrained retrievals while still maintaining good data quality. Uncertainties of the other, unconstrained, retrievals are larger, and primarily depend on how closely the lidar ratio initially assumed by the algorithm agrees with the true lidar ratio, and on how well the attenuation of overlying layers has been estimated. Errors in the initial lidar ratio and attenuation correction propagate systematically and non-linearly into subsequent retrievals, so that extinction errors do not average out but instead produce biases.
In the case of opaque layers - i.e., where the lidar signal does not penetrate to the base of the layer - the reported optical depth refers only to the upper portion of the layer where there is measurable lidar signal. About 20% of cloud layers are identified as being opaque, as are a few very dense aerosol layers. In these cases CALIOP cannot measure the true layer base height and thus underestimates the true layer optical depth. The Opacity Flag in the Cloud and Aerosol Layer Products identifies layers which are opaque. In the profile products, opaque layers are identified by the extinction QC flags. About 90% of cloud retrievals and 99% of aerosol retrievals conclude successfully, in the sense that a physically possible solution is found (retrieved extinction is finite and non-negative for non-negative inputs). The results of the remaining retrievals can also be found in the data products, but are non-physical. For these cases, the extinction QC flags will enumerate the condition(s) that caused the failure. New in version 3.01, the layer Extinction QC Flags are reported on a profile-by-profile basis. Users are highly encouraged to make use of the extinction QC values when using the data products.
The extinction retrieval will, if necessary to avoid a non-physical solution, adjust the initial lidar ratio as required to produce a physically plausible solution. The most common reason for adjusting the lidar ratio is that the extinction solution diverges and the retrieved values tend toward infinity when the assumed lidar ratio is too large. For weakly scattering layers the lidar ratio is most often left unchanged, as a physical solution is usually obtained on the first iteration. However, if the initial lidar ratio is much larger than the true value, divergence can occur even for optically thin (optical depth < 0.5) layers. When divergence occurs the lidar ratio is automatically decreased and the retrieval is repeated until a convergent solution is obtained.
In the current implementation of the extinction retrieval algorithm, the lidar ratio adjustment scheme may result in unrealistic retrievals in aerosol layers in cases where the initial divergence is due to a misclassification of aerosol type. When divergence occurs, it can be because the true lidar ratio is much less than the lidar ratio selected by the algorithm, or that the attenuation of overlying layers was overestimated, or both. Aerosol retrievals where the final lidar ratio is not equal to the initial lidar ratio should be regarded carefully by the data user. The lidar ratio is adjusted in only about 5% of the aerosol retrievals.
Users should not be unduly pessimistic about the quality and usability of the CALIPSO optical depth estimates. Figure 1 (below) shows a preliminary comparison of CALIPSO version 3 aerosol optical depths with the optical depths derived from MODIS for all daytime measurements acquired during January 2007. The comparison is generally good, with MODIS appearing to slightly over-estimate values at the lower end of the optical depth range.
|
Figure 1: Comparison of CALIPSO aerosol optical depths to those derived from MODIS
(Preliminary - January 2007, daytime data, final lidar ratio = initial lidar ratio only) |
 |
A histogram of cirrus optical depths from the Cloud Layer Product shows a bimodal distribution (see Figure 2, left panel). The peak on the right, at an optical depth of ~2.5, is an artifact due to the behavior of the retrieval algorithm when the initial retrieval diverges and the lidar ratio is reduced to produce a convergent solution. This happens most often in totally attenuating (i.e., opaque) clouds, and when the true cirrus lidar ratio is significantly smaller than the initial value assumed by the algorithm. The second peak disappears when we consider only those solutions for which the lidar ratio was unchanged (i.e., the extinction QC flag = 0; see Figure 2, right panel). In any case, errors grow rapidly when the true cirrus lidar ratio is different from the assumed value. Errors in optical depth for thin cirrus (optical depth < 0.5) are on the order of 41% for unconstrained retrievals for clouds composed of random oriented crystals.
|
Figure 2: CALIPSO cirrus cloud optical depths
(v3.01 - August 2006, nighttime only) |
|
 |
 |
Although the production code applies the extinction retrieval algorithm to all layers detected, the CALIOP extinction retrieval algorithm was developed for retrievals of aerosol and ice clouds, not water clouds. In version 2 the multiple scattering factor for water clouds was set to unity, resulting in large errors in retrievals of extinction and optical depth. In version 3, a value of η532 = 0.6 is used. Based on Monte Carlo simulations of multiple scattering, this value appears to be appropriate for semitransparent water clouds (τ < 1). (It is purely coincidental this is the same value used for ice clouds.) For denser water clouds (τ > 1) the multiply-scattered component of the signal becomes much larger than the single-scattered component, η532 becomes dependent on both cloud extinction and range into the cloud, and the retrieval becomes very sensitive to errors in the multiple scattering factor used. In these cases the multiple scattering cannot be properly accounted for in the current retrieval algorithm and retrieval results are unreliable.
The version 3 CALIPSO data processing system now generates profile products for clouds and aerosols at the same spatial resolution. The cloud and aerosol profile products are reported at a uniform spatial resolution of 60-m vertically and 5-km horizontally, over a nominal altitude range from 20-km to -0.5-km for clouds, and over a broader altitude range of 30-km to -0.5-km for aerosols. Profile data for all features detected in the stratosphere are reported in the aerosol profile product, thus users seeking to obtain CALIPSO retrievals of backscatter and extinction profiles of polar stratospheric clouds are directed to the aerosol profile products. Due to constraints imposed by CALIPSO's on-board data averaging scheme, the vertical resolution of the aerosol profile data varies as a function of altitude. In the tropospheric region between 20-km to -0.5-km, the aerosol profile products are reported at a resolution of 60-m vertically, and in the stratospheric region (above 20-km), the aerosol profile products are reported at a resolution of 180-m vertically. In the text below we provide brief descriptions of individual data fields reported in the CALIPSO cloud and aerosol profile products. Where appropriate, we also provide an assessment of the quality and accuracy of the data in the current release. The data descriptions are grouped into several major categories, as follows:
Optical depth of all clouds, aerosol, or stratospheric layers within a 5 km column. The optical depths are obtained by integrating the 532 nm cloud/aerosol/stratospheric extinction profile, reported in these profile products. For aerosols and stratospheric layers the optical depth is provided at both 532 and 1064 nm wavelengths.
The column optical depth uncertainties provide the only unique QA indicator for each column optical depth product; no other QA or data quality flags are provided. Therefore, users are strongly encouraged to make use of the Extinction_QC ,CAD_Score, cloud water phase, cloud water phase QA values (both found in the Feature_Classification_Flags in the layer products or the Atmospheric_Volume_Description in the profile products), Column_IAB_Cumulative_Probability, and feature opacity information to make their own quality assessment of the column depths.
The column optical depths are a column integral product. Any large uncertainties or poor extinction retrievals from layers within the column (i.e. clouds, aerosols, or stratospheric features) will propogate downward and may impact the quality of all (i.e. cloud, aerosol, and stratospheric) the column optical depths in the column. The following paragraphs outline notes with regard to specific data products that users should be aware of when using column optical depth data.
Opacity: We remind data users that the CALIPSO lidar is only capable of penetrating to the surface if the total column optical depth is less than ~5. (Note that this value takes into account the contribution of multiple scattering.) If the column is opaque to the lidar, then the reported column optical depths are a measure to the apparent base of the lowest feature observed. Feature opacity can be determined by inspecting the extinction QC flag for the lowest extinction coefficient in any 5-km column; if bit 5 (value = 16) is set then the feature is totally attenuating.
Extinction QC: The extinction QC values in the column should be examined to determine if any of the extinction retrievals were bad. Users are reminded that any poor extinction retrievals in the column may impact the quality of all column optical depths. In general, solutions where the final lidar ratio is unchanged (extinction QC = 0) or the extinction solution is constrained (extinction QC = 1) yield physically plausible solutions more often. Conversely, retrievals tend to be more uncertain in those cases where the lidar ratio for either wavelength must be reduced.
CAD_Score and feature sub-type: Features with low absolute CAD_Scores, "special" CAD_Scores, or uncertain aerosol type classifications may impact the quality of the column optical depths. For example, if the top-most feature in the column has a low absolute CAD_Score it is possible that the assigned lidar ratio may be incorrect; this would impact the extinction retrieval for that feature which would lead to an incorrect rescaling of all the data below that feature.
Cloud phase: If there are clouds in the column that are found to have horizontal oriented ice (HOI) crystals it is likely that the quality of the column optical depths are low. The anomalously high backscatter from HOI clouds generally makes the extinction retrieval more difficult. Because all the data below the HOI cloud is rescaled by the retrieved optical depth, the extinction data below could be suspect.
Calculation of the layer optical depth uncertainty is an iterative process. On some occasions when the SNR is poor, or an inappropriate lidar ratio is being used, the iteration will attempt to converge asymptotically to positive infinity. Whenever this situation is detected, the iteration is terminated, and the layer optical depth uncertainty is assigned a fixed value of 99.99. Any time an uncertainty of 99.99 is reported, the extinction calculation should be considered to have failed. The associated optical depths cannot be considered reliable, and should therefore be excluded from all science studies.
Note: optical depth uncertainties are reported as absolute errors, not relative errors.
| Figure 3: Distribution of γ′column at 532 nm |
 |
Reports the fraction (by area) within each 5 km horizontal * 60 m vertical range bin containing aerosols or clouds in the profile products. The Aerosol and Cloud Layer Fractions are conceptually identical and the same procedure is used to calculate both quantities.
Since the array elements of the profile products can be larger than the native resolution of the extinction retrieval (5 km * 30 m vs. 5 km * 60 m in the lower troposphere), and because atmospheric features can be identified at horizontal resolutions of 1 km and 1/3 km, the atmospheric composition within each profile range bin is not guaranteed to be homogeneous. Thus, the Aerosol and Cloud Layer Fractions report the fraction (by area) of each 5 km * 60 m profile range bin identified as containing aerosols or clouds by the Scene Classification Algorithms. By referencing Cloud Layer Fraction, the fractional amount of cloud clearing performed within each profile range bin of aerosol backscatter and extinction can be determined.
| Figure 4: Cloud clearing scenarios for strongly scattering clouds detected at single shot resolution. Red indicates clouds detected at 1/3 km resolution, blue indicates clouds found at 1 km or coarser resolution, yellow indicates an aerosol layer found at 5 km resolution, and white indicates clear air. Scenarios: Clouds embedded in aerosol (upper panel), clouds embedded in clear air (middle panel), and dense clouds embedded in within a weakly scattering cloud layer (lower panel). Each row extends 5 km horizontally and 30 m vertically. Each column extends 1/3 km horizontally. |
 |
Shown in Figure 4 are 3 possible scenarios illustrating how the Cloud Layer Fraction would be reported for each 5 km x 60 m cloud profile range bin. There are at most 30 single shot (1/3 km x 30 m) cloud layers in each 5 km x 60 m cloud profile range bin - fifteen 1/3 km horizontally and two 30 m vertically. In the top panel, red indicates clouds found at 1/3 km resolution and the yellow indicates an aerosol layer found at a 5 km horizontal resolution after the 1/3 km clouds had been removed. In this case, the cloud fraction for the top row would be 11/30 = 0.36. In the middle panel, no features were detected at any coarser spatial resolution after the 1/3 km features were removed. The Atmospheric Volume Description for the 5 km average would report the cell as being "clear air", but the cloud fraction for the top row would still be 11/30 = 0.36. In the lower panel, a cloud was detected in the data remaining after all 1/3 km features had been removed. In this case the cloud fraction would be 1 for all rows shown.
The Aerosol and Cloud Layer Fractions must be values between zero and one, yet both layer fractions are reported as integers between 0 and 30. For example, a value for the Aerosol_Layer_Fraction reported as 11 would indicate a fraction of 11/30 = 0.367.
Atmospheric Volume Description is a profile descriptive flag containing the Feature Classification Flags associated with each 5 km x 60 m (or 5 km x 180 m) range bin in the Profile Products. The Feature Classification Flags provide assessments of (a) feature type (e.g., cloud vs. aerosol vs. stratospheric layer); (b) feature subtype; (c) layer ice-water phase (clouds only); and (d) the amount of horizontal averaging required for layer detection. Note that the interpretation of final three bits in the atmospheric volume description (i.e., the averaging required for detection) is slightly different from the interpretation that would be used for the feature classification flags. These differences are summarized in the table below.
| Value | Atmospheric volume description | Feature Classification flag |
|---|---|---|
| 0 | not applicable | not applicable |
| 1 | 5 km | 1/3 km |
| 2 | 20 km | 1 km |
| 3 | 80 km | 5 km |
| 4 | 5 km w/ subgrid feature detected at 1/3 km | 20 km |
| 5 | 20 km w/ subgrid feature detected at 1/3 km | 80 km |
| 6 | 80 km w/ subgrid feature detected at 1/3 km | not used |
| 7, 8 | not used | not used |
Atmospheric Volume Description, CAD Score and Extinction QC [532|1064] are all profile descriptive flags that are stored in the Level 2 Profile Products in the same manner explained here.
Ideally, each profile descriptive flag would be an array of the size [# altitude bins, # profiles] with each array element providing a complete description of the range-resolved atmospheric state. However, because the range resolution of the Level 1 profile data below ~8.3-km is 30 m, and because the feature-finder, scene classification, and extinction algorithms all operate at this finer spatial resolution, providing a genuinely complete description of the atmospheric state for each 60 m Level 2 range bin requires that the profile descriptive flags be stored as 3-D arrays of the size [#profiles, # altitude bins, 2]. The first dimension, [ : , : , 1], corresponds to the standard altitude array of the Profile Products. Thus, below 8.3 km, the first dimension contains the descriptive flags of the higher of the two full resolution (30 m) bins that comprise the single 60 m bin reported in the Profile Products. Meanwhile, below 8.3 km, the second dimension [: , : , 2] contains the descriptive flags for the lower of the two 30 m range bins. Above 8.3 km, where the range resolution of the Level 1 data is 60 m or greater, the descriptive flags for each single 60 m (or 180 m) range bin are replicated in both array elements.
| Figure 5: Wholly fictitious but heuristically useful schematic of layer detection results for a data segment extending 80-km horizontally and 480-m vertically. Yellow/orange/brown indicates an aerosol layer detected at horizontal averaging resolutions of, respectively, 80, 20 or 5 km. Shades of blue likewise represent clouds at 80, 20, and 5 km. Red represents a surface detected layer at 5 km, and the white regions are (presumably) clear air, where no features were found. The right-hand side of the figure shows the atmospheric volume descriptor for columns 1 and 16. |
 |
Note that this field is stored in a similar format as the Atmospheric Volume Descriptor.
The extinction QC flags are bit-mapped 16-bit integers, reported for each layer and for each wavelength for which an extinction retrieval was attempted. Aerosol extinction is computed for both wavelengths; cloud extinction is only reported at 532 nm. The information content of each bit is as follows
| Bit | Value | Interpretation |
|---|---|---|
| 1 | 0 | unconstrained retrieval; initial lidar ratio unchanged during solution process |
| 1 | 1 | constrained retrieval |
| 2 | 2 | Initial lidar ratio reduced to prevent divergence of extinction solution |
| 3 | 4 | Initial lidar ratio increased to reduce the number of negative extinction coefficients in the derived solution |
| 4 | 8 | Calculated backscatter coefficient exceeds the maximum allowable value |
| 5 | 16 | Layer being analyzed has been identified by the feature finder as being totally attenuating (i.e., opaque) |
| 6 | 32 | Estimated optical depth error exceeds the maximum allowable value |
| 7 | 64 | Solution converges, but with an unacceptably large number of negative values |
| 8 | 128 | Retrieval terminated at maximum iterations |
| 9 | 256 | No solution possible within allowable lidar ratio bounds |
| 16 | 32768 | Fill value or no solution attempted |
The bit assignments are additive, so that (for example) an extinction QC value of 18 represents an unconstrained retrieval (bit 1 is NOT set) for which the lidar ratio was reduced to prevent divergence (+2; bit 2 is set), and for which the feature finder has indicated that the layer is opaque (+16; bit 5 is set). For the version 2.01 release, bits 10-15 are not used. Complete information about the conditions under which each extinction QC bit is toggled can be found in the CALIPSO Extinction Retrieval ATBD (PDF)
Note that this field is stored in a similar format as the Atmospheric Volume Descriptor.
The cloud-aerosol discrimination (CAD) score, which is reported in the 1-km and 5-km layer products, and now in version 3, the 5 km cloud and aerosol profile products, provides a numerical confidence level for the classification of layers by the CALIOP cloud-aerosol discrimination algorithm. The CAD algorithm separates clouds and aerosols based on multi-dimensional histograms of scattering properties (e.g., intensity and spectral dependence) as a function of geophysical location. In areas where there is no overlap or intersection between these histograms, features can be classified with complete confidence (i.e., |CAD score| = 100).In the current release (version 3), the CAD algorithm uses newly developed five-dimensional (5D) probability density functions (PDFs), rather than the three-dimensional (3D) PDFs used in previous versions. In addition to the parameters used in the earlier 3D version of the algorithm (layer mean attenuated backscatter at 532 nm, layer-integrated attenuated backscatter color ratio, and altitude), the new 5D PDFs also include feature latitude and the layer-integrated volume depolarization ratio. Detailed descriptions of the CAD algorithm can be found in Sections 4 and 5 of the CALIPSO Scene Classification ATBD (PDF). Enhancements made to incorporate the 5D PDFs used in version 3 release are described in Liu et al., 2010 (PDF). For further information on the CAD algorithm architecture and the three-dimensional (3D) PDFs used in versions 1 and 2 of the data products, please see Liu et al., 2004 (PDF) and/or Liu et al., 2009.
The standard CAD scores reported in the CALIPSO layer products range between -100 and 100. The sign of the CAD score indicates the feature type: positive values signify clouds, whereas negative values signify aerosols. The absolute value of the CAD score provides a confidence level for the classification. The larger the magnitude of the CAD score, the higher our confidence that the classification is correct. An absolute value of 100 therefore indicates complete confidence. Absolute values less that 100 indicate some ambiguity in the classification; that is, the scattering properties of the feature are represented to some degree in both the cloud PDF and in the aerosol PDF. In this case, a definitive classification cannot be made; that is, although we can provide a "best guess" classification, this guess could be wrong, with a probability of error related to the absolute value of the CAD score. A value of 0 indicates that a feature has an equal likelihood of being a cloud and an aerosol. Users are encouraged to refer to the CAD score when the cloud and aerosol classification results are used and interpreted.
Beginning with the version 2.01 release, several "special" CAD score values have been added. These are listed in the table below. Each of these new values represents a classification result that is based on additional information beyond that normally considered in the standard CAD algorithm.
| CAD score | Interpretation |
|---|---|
| -101 | negative mean attenuated backscatter encountered; layer is most likely an artifact, and its spatial and optical properties should be excluded from all science analyses. |
| 101 | initially classified as aerosol, but layer integrated depolarization mandates classifying layer as cloud (version 2 only; obsolete in version 3) |
| 102 | layer exhibits very high integrated backscatter and very low depolarization characteristic of oriented ice crystals (version 2 only; obsolete in version 3) |
| 103 | layer integrated attenuated backscatter at 532 nm is suspiciously high; feature authenticity and classification are both highly uncertain |
| 104 | layers with CAD scores of 104 are boundary layer clouds that were found to be opaque at the initial 5-km horizontal averaging resolution used by the layer detection algorithm; however, these layers are not uniformly filled with high-resolution clouds (i.e., layers detected at a 1/3-km horizontal resolution), and the 532 nm mean attenuated backscatter coefficient of the data that remains after cloud clearing is negative. Studies examining the spatial properties and distributions of clouds can safely include the spatial properties of these layers; however, the associated measured and derived optical properties should be excluded from all science studies. |
| 105 | a CAD score of 105 designates a layer detected at one of the coarser averaging resolutions (20-km or 80-km) for which the initial estimates of measured properties have been negatively impacted by either (a) the attenuation corrections applied to account for the optical depths of overlying layers, or (b) the extension of the layer base altitude are boundary layer clouds that were found to be opaque at the initial 5-km horizontal averaging resolution used by the layer detection algorithm; however, these layers are not uniformly filled with high-resolution clouds (i.e., layers detected at a 1/3-km horizontal resolution), and the 532 nm mean attenuated backscatter coefficient of the data that remains after cloud clearing is negative. Studies examining the spatial properties and distributions of clouds can safely include the spatial properties of these layers; however, the associated measured and derived optical properties should be excluded from all science studies. |
 |
Ice clouds: for the CALIOP viewing geometry, simulations show multiple scattering effects are nearly independent of range and, as parameterized in the CALIOP retrieval algorithm (Winker et al. 2009), are nearly independent of extinction. In version 2, ice clouds were assigned a range-independent multiple scattering factor of η532 = 0.6. Validation comparisons indicate this is an appropriate value and the same value is used in version 3.
Water clouds: in version 2 the multiple scattering factor for water clouds was set to unity, resulting in large errors in retrievals of extinction and optical depth. In version 3, a value of η532 = 0.6 is used. Based on Monte Carlo simulations of multiple scattering, this value appears to be appropriate for semitransparent water clouds (τ < 1). (It is purely coincidental this is the same value used for ice clouds.) For denser water clouds (τ > 1) the multiply-scattered component of the signal becomes much larger than the single-scattered component, η532 becomes dependent on both cloud extinction and range into the cloud, and the retrieval becomes very sensitive to errors in the multiple scattering factor used. In these cases the multiple scattering cannot be properly accounted for in the current retrieval algorithm and retrieval results are unreliable.
Aerosols: simulations of multiple scattering effects on retrievals of aerosol layer optical depth indicate the effects are small in most cases. There is uncertainty in these estimates, however, due to poor knowledge of aerosol scattering phase functions. Validation comparisons conducted to date do not indicate significant multiple scattering effects on aerosol extinction profile retrievals. Multiple scattering effects may become significant in dense aerosol layers (σ > 1 /km), but in these cases retrieval errors are usually dominated by uncertainties in the lidar ratio or failure to fully penetrate the layer. In version 3, as in version 2, multiple scattering factors for both wavelengths are set to unity.
Ice water content (IWC) is reported for all ice clouds detected in the CALIPSO measurements. IWC values of 0-0.54 gm,-3 account for 99.5% of the values measured by CALIOP. Above 8.25 km, IWC values > 0.767 gm-3 cannot be measured.
Background: Cloud ice water content is a provisional data product that is calculated as a parameterized function of the CALIOP retrieved extinction by ice cloud particles:
| (1) |  |
Here, σ is the 532 nm volume extinction coefficient in km-1, and c0 = 119 gm-3 and c1 = 1.22 are coefficients derived from an observed empirical relationship between lidar extinction and an extensive set of in situ measurements of cloud particle properties from numerous field campaigns [1]. The relationship between 532 nm extinction and IWC was developed using IWC data between 0-1.0 gm-3 with temperatures between -70 and 0 °C. Cloud ice amount has been shown to vary with temperature, cloud particle size distribution, and by location inside a cloud. A temperature-dependent parameterization is being considered and tested for the next CALIOP data release. The effect of particle size distribution on IWC as seen by CALIOP is also currently being evaluated by comparison with in situ cloud data. Preliminary results show that CALIOP IWC has sufficient spatial resolution and precision to realistically resolve cloud morphology. A more detailed preliminary evaluation of the CALIOP version 3 IWC is available as an ILRC extended abstract [2], which includes CALIOP IWC probability distributions and example browse images. For a brief discussion containing critical information needed to intelligently use CALIOP IWC, please see the "data screening" section, below.
To summarize, users that do not wish to dig more deeply are recommended to use data of type=2 with phase=1 (atmospheric volume descriptor) and an extinction quality flag value of 0 or 1. IWC data from the highest cloud layer (highest signal-to-noise ratio), relative uncertainty of < 2 and IWC of < 0.026 gm-3 are likely to be the most reliable.
References:
IWC uncertainty has a range of 0-99.99 gm-3, and is derived directly from the extinction retrieval uncertainty. The IWC fractional uncertainty probability distribution is shown in [1]. This is the estimated CALIOP measurement uncertainty, and does not characterize uncertainty in the IWC parameterization. Users wishing to understand extinction retrieval uncertainty should consult the summary descriptions at the top of this page.
Comparison of CALIOP IWC with in situ data from various field campaigns, and with CPL lidar data is underway. Further, we are working on comparisons between CALIOP IWC and IWC from CLOUDSAT and MLS. We anticipate sharing the results from these comparisons in future quality summary updates.
Future IWC parameterizations may include temperature dependency, based on further in situ data comparisons. Currently suggested temperature-dependant parameterizations do not produce good results for high-altitude tropical clouds [6] and therefore are not used.
References:
| Lidar Level 2 Cloud and Aerosol Profile Information Half orbit (Night and Day) averaged cloud and aerosol profile data and ancillary data |
|||
|---|---|---|---|
| Release Date | Version | Data Date Range | Maturity Level |
| December 2011 | 3.02 | November 1, 2011 to present | Validated Stage 1 |
| May 2010 | 3.01 | June 13, 2006 to February 16, 2009 March 17, 2009 to October 31, 2011 |
Provisional |
The CALIPSO Team is releasing Version 3.02 which represents a transition of the Lidar, IIR, and WFC processing and browse code to a new cluster computing system. No algorithm changes were introduced and very minor changes were observed between V 3.01 and V 3.02 as a result of the compiler and computer architecture differences. Version 3.02 is being released in a forward processing mode beginning November 1, 2011.
Version 3.01 of the Lidar Level 2 data products is a significant improvement over previous versions. Major code and algorithm improvements include
In addition to the numerous algorithm updates, several new parameters have been added to the profile products. These include
The sections below highlight important changes to the layer detection, scene classification, and extinction algorithms that have implications for the overall quality of the Lidar Level 2 data products.
| Figure 6: Histograms of CAD scores for Version 2 (red) and Version 3 (blue) |
 |
Figure 7 (below) presents the relationship between the CAD score and the layer IAB QA factor, which provides a measure of the integrated attenuated backscatter overlying a cloud or an aerosol layer. A layer IAB QA factor close to 1 indicates that the atmosphere above the layer is clear. Decreasing values indicate the increasing likelihood of overlying layers that have attenuated the signal within the layer under consideration, and thus decreased the SNR of the measurement. A layer IAB QA factor near 0 indicates total attenuation of the signal. As seen in the figure, the IAB QA is highest for high magnitude CAD scores and slopes down gradually for small CAD score magnitudes. This relationship reflects the fact that the presence of overlying features tends to add difficulty to the cloud-aerosol classification task, and therefore reduces the confidence of the classifications made. The dip between -10 and 20 represents features that are outliers in the 5-D CAD PDFs, and indicates that these outliers most often lie beneath other relatively dense features. The cloud layers with special CAD scores (103 and 104) have the smallest IAB QA values. The relatively big value at CAD = 0 corresponds to the features having zero CAD values at high altitudes where the probability of the presence of overlying features is low. At high altitudes the separation of clouds and aerosols is not as good as at low altitudes because of the presence of subvisible cirrus clouds.
| Figure 7: Relation between CAD score and Layer IAB QA Factor |
 |
Overall, because of the better separation between clouds and aerosols in the 5D space, the 5D CAD algorithm significantly improves the reliability of the CAD scores. The improvements include:
Dense aerosol layers (primarily very dense dust and smoke over and close to the source regions), which are sometimes labeled as cloud in the V2 release, are now correctly identified as aerosol, largely because of the addition of the integrated volume depolarization ratio to the diagnostic parameters used for cloud-aerosol discrimination. In addition, in the open oceans, dense aerosols that were previously classified as clouds are now frequently observed in the marine boundary layer. Improvements are also seen for these maritime aerosols. Note, however, dense dust/smoke layers found at single-shot (0.333 km) resolution will be classified as cloud by default. This issue will be revisited for post-V3 releases.
Because the V2 CAD algorithm used a latitude-independent set of 3D PDFs, a class of optically thin clouds encountered in the polar regions that can extend from the surface to several kilometers were sometimes misclassified as aerosols. In version 3, these features are now correctly classified as cloud.
Correct classification of heterogeneous layers is always difficult. An example of a heterogeneous layer would be an aerosol layer that is vertically adjacent to a cloud or contains an embedded cloud, but which is nonetheless detected by the feature finder as a single entity in the V2 release. By convention, heterogeneous layers should be classified as clouds. The version 3 feature finding algorithm has also been improved greatly, and can now much better separate the embedded or adjacent single-shot cloud layers from the surrounding aerosol. This improvement in layer detection contributes significantly to the improvement of the CAD performance.
Some so-called features identified by the layer detection scheme are not legitimate layers, but instead are artifacts due to the noise in the signal, multiple scattering effects, or to artificial signal enhancements caused by non-ideal detector transient response or an over estimate of the attenuation due to overlying layers. These erroneous "pseudo-features" are neither cloud nor aerosol and are distributed outside of the cloud and aerosol clusters in the PDF space. The V3 CAD algorithm can better identify these outlier features by assigning a small CAD score (the bump between -10 and 20 in the V3 CAD histogram) and classify most of them as cloud by convention. A CAD threshold of 20 can effectively filter out these outliers.
The extinction products are produced by first identifying an aerosol type and then using the appropriate values of Sa and the multiple scattering factor, η(z). Note that multiple scattering corrections have not yet been implemented for the current data release, so that η(z) = 1 for all aerosol types. The accuracy of the Sa value used in the lidar inversions depends on the correct identification of the type of aerosol. In turn, the accuracy of the subsequent optical depth estimate depends on the accuracy of Sa.
The underlying paradigm of the type classification is that a variety of emission sources and atmospheric processes will act to produce air masses with a typical, identifiable aerosol 'type'. This is an idealization, but one that allows us to classify aerosols based on observations and location in a way to gain insight into the geographic distribution of aerosol types and constrain the possible values of Sa for use in aerosol extinction retrievals.
The aerosol subtype product is generated downstream of the cloud-aerosol discrimination (CAD) scheme and, therefore, depends on the cloud-aerosol classification scheme in a very fundamental way. If a cloud feature is misclassified as aerosol, the aerosol subtype algorithm will identify this 'aerosol' as one of the aerosol subtypes. The user must exercise caution where the aerosol subtype looks suspicious or unreasonable. Such situations can occur with some frequency in the southern oceans and the polar regions.The version 2 algorithm included a rudimentary ability to identify a specific subset of high confidence instances of HOI. These clouds were classified as ice clouds, and flagged with a 'special CAD score' of 102, indicating that they had been further classified as HOI. The new version 3 algorithm implements a much more sophisticated scheme for recognizing HOI that correctly identifies many more instances of these sorts of ice clouds. The special CAD score of 102 is no longer used to identify these layers. Instead, the "ice cloud" and "mixed phase cloud" classifications have been eliminated, and replaced as shown in the table below.
| Value | Version 2 Interpretation | Version 3 Interpretation |
|---|---|---|
| 0 | unknown/not determined | unknown/not determined |
| 1 | ice | randomly oriented ice (ROI) |
| 2 | water | water |
| 3 | mixed phase | horizontally oriented ice (HOI) |
The Ice/water Phase QA flags have also been redefined slightly for version 3, as follows:
| Value | Version 2 Interpretation | Version 3 Interpretation |
|---|---|---|
| 0 | no confidence | no/low confidence |
| 1 | low confidence | phase based on temperature only |
| 2 | medium confidence | medium confidence |
| 3 | high confidence | high confidence |
A confidence flag of QA=1 indicates the phase classification is based on temperature. Initial classification tests are based on layer depolarization, layer-integrated backscatter, and layer-average color ratio. Layers classified as water with temperature less than -40 C are forced to ROI and given a confidence flag of QA=1. Layers classified as ROI or HOI with temperature greater than 0 C are forced to water and also given a confidence flag of QA=1. Clouds for which the phase is ‘unknown/not determined’ are assigned a confidence value of 0 (no/low confidence).
Layers classified as HOI based on anomalously high backscatter and low depolarization are assigned QA=3. These layer characteristics are rarely detected after the CALIOP viewing angle was changed to 3° in November 2007. The version 3 algorithm computes the spatial correlation of depolarization and integrated backscatter, and uses this as an additional test of cloud phase. Layers classified as HOI using this test are assigned QA=2. The spatial correlation test is responsible for the majority of the layers classified as HOI. These layers typically have higher backscatter than ROI but similar depolarization, and are common even at a viewing angle of 3°. We interpret this as clouds with significant perpendicular backscatter from ROI but containing enough HOI to produce enhanced backscatter. These layers tend to be found at much colder temperatures than the high confidence HOI (see Hu et al. 2009).
PLEASE NOTE: Users of the CALIOP optical depths should read and thoroughly understand the information provided in the Profile Products Data Quality Summary. This summary contains an expanded description of the extinction retrieval process from which the layer optical depths are derived, and provides essential guidance in the appropriate use of all CALIOP extinction-related data products. Validation and improvements to the profile products QA are ongoing efforts, and additional data quality information will be included with future releases.
