This document provides a high level quality assessment of the CALIPSO lidar
level 1.5 expedited data product, derived from CALIPSO lidar level 1B, level 2
aerosol profile, and level 2 vertical feature mask expedited products, as described
in section 6.0 of the CALIPSO Data Products Catalog (Version 3.3) (PDF). As such,
it represents the minimum information needed by scientists and researchers for
appropriate and successful use of these data products. We strongly suggest that
all authors, researchers, and reviewers of research papers review this document
periodically, and familiarize themselves with the latest status before publishing
any scientific papers using these data products.
These data quality summaries are published specifically to inform users of
the accuracy of CALIOP data products as determined by the CALIPSO Science Team
and Lidar Science Working Group (LSWG). This document is intended to briefly
summarize key validation results; provide cautions in those areas where users
might easily misinterpret the data; supply links to further information about
the data products and the algorithms used to generate them; and offer
information about planned algorithm revisions and data improvements.
Additionally, since lidar level 1.5 is a merged data product, this document
describes and assesses attributes specific to the level 1.5 product. Users are
advised to consult the Data Quality Summaries for the
lidar level 1B,
lidar level 2
aerosol profile, and
lidar
level 2 vertical feature mask products for detailed quality information of
the inputs for level 1.5.
The CALIPSO lidar level 1.5 expedited data product is currently available by
subscription only. Since it is designed for near-real time applications,
granules are produced in forward processing and only the most recent ten
days are accessible via ftp. Please contact the CALIPSO Principal
Investigator
David Winker
and CALIPSO Data Manager
Pat Lucker
to obtain a subscription to access level 1.5 expedited data.
The initial release of the CALIPSO lidar level 1.5 expedited data product is
considered beta, as defined in the table below. Even though level 1.5 is a
merged product based on level 1B and level 2 data products which are at higher
maturity levels, the algorithms used for cloud-clearing and re-averaging are
still considered beta, thus the maturity of all scientific data sets reported
by level 1.5 are uniformly classified beta for this release.
Maturity Level Definition
Beta:
Early release products for users to gain familiarity with data formats and
parameters. Users are
strongly cautioned against the use of these data products as the basis for
research findings, journal publications, and/or presentations.
Vaughan, M., C. Trepte, D. Winker, M.
Avery, J. Campbell, R. Hoff, S. Young, B. Getzewich, J. Tackett, and J. Kar,
2011: "Adapting CALIPSO Climate Measurements for Near Real Time Analyses
and Forecasting", Proceedings of the 34th International Symposium
on Remote Sensing of Environment, [Available at
Vaughan, et al (PDF)].
The CALIPSO lidar level 1.5 expedited data product is an unvalidated,
beta-quality product. Though it is a merger of existing CALIPSO data products
at higher maturity levels, the algorithms used to cloud-clear and re-average
these products in order to produce level 1.5 data are considered beta. As
such, level 1.5 data may contain any number of errors and/or inconsistencies.
Users are cautioned against the indiscriminate use of these data products;
using them as the basis for research findings, journal publications, and/or
presentations at scientific conferences is strongly discouraged.
The CALIPSO lidar level 1.5 expedited data product contains continuous,
calibrated, and geo-located profiles of attenuated backscatter and aerosol
extinction coefficients that have been screened of clouds, overcast, surface,
subsurface, totally attenuated, and invalid features before being averaged to
20 km horizontal x 60 m vertical resolution. It is a synthesis of CALIPSO
lidar level 1B expedited and lidar level 2 expedited aerosol profile products
with the lidar level 2 expedited vertical feature mask which is used for
feature identification and subsequent screening. Thus, it is deemed a
"level 1.5" expedited product and it is currently available by
subscription only.
Quality statement for expedited data used as input for level 1.5 data product
Input granules for the beta release of lidar level 1.5 expedited data
product are CALIOP lidar level 1B, level 2 aerosol profile, and level 2
vertical feature mask expedited products, version 3.02. Expedited CALIPSO data
products are designed to be generated faster than standard CALIPSO data
products for near-real time forecasting purposes. Two issues impact the quality
of expedited level 1B processing versus standard processing: calibration and
geolocation. As discussed in Vaughan et al. (2011), mean
differences between version 3.01 expedited and standard calibrations are less
that 1% for 532 nm (though extreme differences are in excess of 10%). Altitude
registration differences are no greater than 60 m and are less than 30 meters
97% of the time.
Differences in calibration,
Global Modeling
and Assimilation Office (GMAO) molecular model used, and geolocation in
expedited processing versus standard processing also affects level 2 expedited
data products. In particular, differences in the GMAO molecular model used in
the calibration region impacts the ability to detect faint layers in expedited
processing, but has little effect on more robust layers. Cloud-aerosol
discrimination can also be influenced due to all three of these issues. In
comparing level 2 vertical features masks from expedited versus nominal
processing, differences in layer detection occurred for ~3% of all range bins -
with slightly higher differences at nighttime. Changes in layer classification
occurred for ~1% of samples detected at night and ~6% of samples detected
during the day.
Impact of using expedited data as input for level 1.5 data
The consequence of using expedited data products instead of standard data
products as inputs for level 1.5 expedited processing manifests itself in the
magnitude of attenuated backscatter and aerosol extinction coefficients via the
expedited calibration coefficients. As described above, level 1.5 expedited
attenuated backscatter coefficients should be comparable to their standard
processing counterparts to within 1-2%. However, the aerosol extinction coefficients
are derived from the solution of a highly non-linear equation whose outputs are
very sensitive to changes in its inputs. Thus, large discrepancies can exist
between aerosol extinction coefficients in expedited versus standard processing.
Cloud-clearing with the expedited vertical feature mask should be robust
compared to the standard vertical feature mask since discrepancies in layer
detection occurs for mostly for faint layers (optically thin aerosol layers
and sub-visible cirrus); i.e., optically thick cloud layers are detected with
nearly equal regularity in the expedited products and are screened out
accordingly. The primary objective of level 1.5 processing is to remove clouds
and surface features from the attenuated backscatter signal. Since these
features are readily detectable with both expedited and standard level 2
processing, cloud and surface clearing are expected to be effective for both
processing methods.
Product Descriptions
CALIPSO lidar level 1.5 expedited data products provide profiles of
attenuated backscatter and aerosol extinction coefficients that have been
screened of clouds, overcast, surface, subsurface, totally attenuated, and
invalid features. Data is averaged to a uniform 20 km horizontal x 60 m
vertical resolution and is reported from -0.5 km to 20 km in altitude. Feature
classification as determined by the CALIPSO scene classifier is reported by the
L2 Feature Type array. Descriptions of the individual data fields reported in
level 1.5 are provided below and, where appropriate, an assessment of data
quality for the current release is given. Data descriptions are grouped into
several major categories, as follows:
Time, expressed in International Atomic Time
(TAI) at the temporal midpoint of each 20 km horizontal resolution
profile (average time of the 30th and 31st consecutive laser shots). Units
are in seconds, starting from January 1, 1993.
Profile UTC Time
Similar to the Profile Time, but for time expressed in Coordinated Universal
Time (UTC), and formatted as "yymmdd.ffffffff", where "yy" represents the
last two digits of the year, "mm" and "dd" represent the month and day,
respectively, and "ffffffff" is the fractional part of the day.
Day Night Flag
Indicates the lighting conditions at an altitude of ~24 km above mean sea
level; 0 = day and 1 = night. In lidar level 1.5 expedited profiles where
the day/night terminator is crossed within the 20 km horizontal averaging
interval, Day Night Flag is assigned a value of 2 for "day and night".
This occurs in the expedited products because expedited granules are
90 minutes in duration and contain data recorded at both day and night.
Geodetic latitude, in degrees, at the temporal midpoint of each 20 km
horizontal resolution profile (average of laser footprint latitudes at
the 30th and 31st consecutive laser shots).
Longitude
Longitude, in degrees, at the temporal midpoint of each 20 km horizontal
resolution profile (average of laser footprint longitudes at the
30th and 31st consecutive laser shots).
This HDF metadata field defines the altitudes of the 345 range bins to which
the lidar level 1.5 profile products are registered. Altitudes extend from
approximately -0.5 km to 20.2 km with a vertical resolution of 60 meters.
This is a 32-bit integer generated sequentially for each single-shot profile
record in the level 1B products. Each profile ID is unique within each
granule. For level 1.5 data products, only the first and last profile ID for
each 20 km resolution profile are reported (corresponding to the
1st and 60th laser shots).
Mean molecular number density, in molecules per cubic meter, reported for the
temporal midpoint of each range bin in each 20 km horizontal resolution
profile (i.e., the average of the molecular number densities reported at the
30th and 31st consecutive single shot profiles for each
range bin). Derived from the GEOS-5 data product provided to the CALIPSO
project by the
GMAO Data
Assimilation System and logarithmically interpolated from the 33
standard altitudes levels reported in the lidar level 1B products
(Met Data Altitudes) to the 345 standard altitude levels
reported in the lidar level 1.5 product
(Lidar Data Altitudes) prior to computing
the temporal midpoint average.
Ozone Number Density (external)
Mean ozone number density, in molecules per cubic meter, reported for the
temporal midpoint of each range bin in each 20 km horizontal resolution
profile (i.e., the average of the ozone number densities reported at the
30th and 31st consecutive single shot profiles for each
range bin). Derived from the GEOS-5 data product provided to the CALIPSO
project by the GMAO Data Assimilation System and logarithmically
interpolated from the 33 standard altitudes levels reported in the lidar
level 1B products
(Met Data Altitudes) to the 345 standard altitude levels
reported in the lidar level 1.5 product (Lidar
Data Altitudes) prior to computing the temporal midpoint average.
Pressure (external)
Mean pressure, in millibars, reported for the temporal midpoint of each range
bin in the 20 km horizontal resolution profile (i.e., the average of the
pressure reported at the 30th and 31st consecutive
single shot profiles for each range bin). Derived from the GEOS-5 data
product provided to the CALIPSO project by the
GMAO Data
Assimilation System and logarithmically interpolated from the 33 standard
altitudes levels reported in the lidar level 1B products
(Met Data Altitudes) to the 345 standard altitude levels
reported in the lidar level 1.5 product (Lidar
Data Altitudes) prior to computing the temporal midpoint average.
Temperature (external)
Mean temperature, in degrees C, reported for the temporal midpoint of each
range bin in the 20 km horizontal resolution profile (i.e., the average of
the temperature reported at the 30th and 31st
consecutive single shot profiles for each range bin). Derived from the GEOS-5
data product provided to the CALIPSO project by the
GMAO Data
Assimilation System and linearly interpolated from the 33 standard
altitudes levels reported in the lidar level 1B products
(Met Data Altitudes) to the 345 standard altitude levels
reported in the lidar level 1.5 product (Lidar
Data Altitudes) prior to computing the temporal midpoint average.
Average of the surface elevation of all laser footprints within each 20 km
horizontal resolution profile, in kilometers above local mean sea level, obtained from the
GTOPO30 digital
elevation map (DEM).
Surface Elevation stDev
Standard deviation of the surface elevation of all laser footprints within
each 20 km horizontal resolution profile, in kilometers above local mean
sea level, obtained from the
GTOPO30 digital
elevation map (DEM).
Number of full resolution samples averaged within each profile range bin
after screening clouds, overcast, surface, subsurface, totally attenuated,
and invalid features. Here, "full resolution" is taken to mean
single shot (~1/3 km) horizontally and 30 meters vertically. Since level 1.5
profile range bins are averaged to 20 km horizontal x 60 meters vertical,
there are at most 120 full resolution samples averaged per lidar level 1.5
range bin.
L2 Feature Type
Reports the results of the CALIPSO level 2 layer detection and scene
classification algorithms for tropospheric features within each lidar level
1.5 profile range bin. For each 20 km horizontal x 60 m vertical lidar level
1.5 profile range bin, the L2_Feature_Type reports a 4-element array of 8-bit
unsigned integers. Each of these four array elements corresponds to a 5 km
horizontal x 60 m vertical segment of data that was considered for averaging.
Values for L2_Feature Type are as follows:
Value
L2 Feature Type
Value
L2 Feature Type
0
invalid (bad or missing data)
11
mixed aerosol
1
totally attenuated
12
cloud-cleared mixed aerosol
2
surface
13
cloud-cleared clean marine
3
subsurface
14
cloud-cleared dust
4
cloud
15
cloud-cleared polluted continental
5
clean marine
16
cloud-cleared clean continental
6
dust
17
cloud-cleared polluted dust
7
polluted continental
18
cloud-cleared smoke/biomass burning
8
clean continental
19
"clear air"
9
polluted dust
20
cloud-cleared "clear air"
10
smoke/biomass burning
21
overcast
Overcast is defined as all range bins beneath the highest cloud in a
profile. The convention for defining clouds and overcast features with
L2_Feature_Type is to classify the entire continuous vertical extent of the
highest cloud in a profile as cloud (4) and then classify all range bins
beneath that cloud as overcast (21) until the first totally attenuated,
surface, or subsurface range bin.
Cloud-cleared aerosols refer to aerosols that have been detected after 1/3
km and/or 1 km horizontal resolution clouds have been cleared by the level 2
scene classification algorithms. Similarly, cloud-cleared "clear
air" refers to segments of "clear air" where 1/3 km and/or 1
km resolution clouds have been cleared.
Mixed aerosols identify 20 km x 60 m resolution elements containing more
than one aerosol type. Cloud-cleared mixed aerosols identify 20 km x 60 m
resolution elements containing more than one aerosol subtype where at least
one is cloud-cleared as defined above.
Note that stratospheric features are not reported in the level 1.5
products. The convention for handling stratospheric features is as follows.
Between 60° N and 60° S, medium and high confidence stratospheric
features are reported as "clear air"; all other stratospheric
features are assumed to be clouds. Poleward of 60°, all stratospheric
features are assumed to be clouds (e.g., PSCs).
IMPORTANT: regions of data in the lidar level 1B
and lidar level 2 aerosol profile products having L2_Feature_Type values of
0 through 4 and 21 are NOT INCLUDED when computing the cloud-cleared averaged
data that is recorded in the attenuated backscatter and aerosol extinction
coefficient profiles.
Extinction QC flags are bit-mapped 16-bit integers reported for each profile
range bin where an extinction solution was attempted. It summarizes the
final state of the extinction attempt. For the level 1.5 products, the
maximum value of all screened extinction QC flags (taken from the
lidar level 2 aerosol profile product) within each 20 km
x 60 m resolution profile range bin is reported. Complete information about
extinction QC flags and their bit assignments can be found in the
CALIPSO Extinction Retrieval ATBD
(PDF).
Total Attenuated Backscatter 532 Mean
Perpendicular Attenuated Backscatter 532 Mean
For each variety of lidar level 1B expedited attenuated backscatter
coefficients listed above, this is the mean of those attenuated backscatter
coefficients within each 20 km horizontal x 60 meter resolution level 1.5
profile range bin after clouds, overcast, surface, subsurface, totally
attenuated, and invalid features have been screened out, in units of
km-1sr-1.
Total Attenuated Backscatter 532 Median
Total Attenuated Backscatter 532 stDev
Perpendicular Attenuated Backscatter 532 Median
Perpendicular Attenuated Backscatter 532 stDev
For each variety of lidar level 1B expedited attenuated backscatter coefficients
listed above, this is the median and standard deviation of those screened
attenuated backscatter coefficients within each 20 km horizontal x 60 meter
resolution level 1.5 profile range bin after clouds, overcast, surface, subsurface,
totally attenuated, and invalid features have been screened out, in units of
km-1sr-1. Since the lower troposphere (-0.5 km to 8.2 km)
and upper troposphere (8.2 km to 20.2 km) are averaged to different resolutions
in lidar level 1B products, the lower troposphere data are first averaged to
upper tropospheric resolution prior to computing the median and standard
deviation. Thus, these statistics should be regarded as the median and standard
deviation of those screened 1 km x 60 m resolution attenuated backscatter
coefficients within each 20 km x 60 m lidar level 1.5 profile range bin.
Total Attenuated Backscatter 532 Uncertainty
Perpendicular Attenuated Backscatter 532 Uncertainty
For each variety of lidar level 1B expedited attenuated backscatter
coefficients listed above, this is the uncertainty on the mean of those
screened attenuated backscatter coefficients within each
20 km horizontal x 60 meter resolution level 1.5 profile range bin after
clouds, overcast, surface, subsurface, totally attenuated, and invalid
features have been screened out, in units of km-1sr-1.
Attenuated backscatter uncertainty is an estimate of random error based on
level 1B noise scale factors
(Liu et al., 2006 (PDF)). It is first calculated at
nominal level 1B range resolution using procedures outlined in
Uncertainties for Attenuated Backscatter (PDF) before the
uncertainty on the mean σμ is calculated for each 20 km x
60 meter profile range bin using (Bevington & Robinson, 1992),
Here, σi is the attenuated backscatter uncertainty at a
level 1B profile range bin i, and N is the number of level 1B profile range
bins within the 20 km x 60 m interval remaining after clouds, overcast, surface,
subsurface, totally attenuated, and invalid features have been screened out.
Extinction Coefficient 532 Mean
Extinction Coefficient 532 Median
Extinction Coefficient 532 stDev
Mean, median, and standard deviation of all lidar level 2 expedited aerosol
profile product extinction coefficients at 532 nm within each 20 km
horizontal x 60 meter resolution level 1.5 profile range bin after clouds,
overcast, surface, subsurface, totally attenuated, and invalid features have
been screened out, in units of km-1. Since lidar level 2 aerosol
profile products are reported uniformly at 5 km horizontal x 60 m vertical
resolution, these statistics should be regarded as the mean, median, and
standard deviation of those screened 5 km x 60 m aerosol extinction
coefficients within each 20 km x 60 m level 1.5 profile range bin.
Extinction Coefficient 532 Uncertainty
Uncertainty on the mean of all lidar level 2 expedited aerosol profile
product extinction coefficients within each 20 km horizontal x 60 meter
resolution level 1.5 profile range bin after clouds, overcast, surface,
subsurface, totally attenuated, and invalid features have been screened out,
in units of km-1sr-1.
Computed similar to attenuated backscatter uncertainties, except in this
case the extinction coefficient uncertainty σi is reported
by the level 2 aerosol profile products for each 5 km x 60 m profile range
bin i. Thus, the uncertainty on the mean σμ is
given by (Bevington & Robinson, 1992),
Here, N is the number of lidar level 2 aerosol profile range bins
within the 20 km x 60 m interval remaining after clouds, overcast, surface,
subsurface, totally attenuated, and invalid features have been screened out.
Mean of the expedited lidar parallel calibration constants at 532 nm, as
defined in Section 3.1.2 of the
Lidar Level I ATBD (PDF) within each 20 km horizontal
resolution profile. Expedited calibration procedures are described in
Vaughan et al. (2011). Units are
km3·sr·count.
Calibration Constant Parallel Uncertainty 532
Mean of the uncertainties due to random noise for expedited 532 nm parallel
calibration constants within each 20 km horizontal resolution profile.
Computed based on the 532 nm
noise scale factors using equation 4.24 in Section 4.3.2
of the CALIPSO
Lidar Level I ATBD (PDF). Units are
km3·sr·count.
Calibration Constant Perpendicular 532
Mean of the expedited lidar perpendicular calibration constants at 532 nm
within each 20 km horizontal resolution profile. Computed using equation 5.6
in Section 5.1.2 of the
Lidar Level I ATBD (PDF). Expedited calibration
procedures are described in Vaughan et al. (2011).
Units are km3·sr·count.
Mean of the uncertainties due to random noise for expedited 532 nm calibration
constant within each 20 km resolution profile. Computed based on the 532 nm
noise scale factors using equation 5.17 in Section 5.2 of
the Lidar Level I ATBD (PDF). Units are
km3·sr·count.
Molecular attenuated backscatter coefficients at 532 nm, in units of
km-1sr-1, at the temporal midpoint of each 20 km horizontal
resolution level 1.5 profile derived from modeled molecular and ozone number
densities provided by the
GMAO Data Assimilation System.
Specifically, the GMAO molecular and ozone number densities are logarithmically
interpolated from the 33 standard altitudes levels reported in the lidar level 1B
products (Met Data Altitudes) to the 345 standard altitude levels
reported in the lidar level 1.5 product
(Lidar Data Altitudes). The number
densities at the temporal midpoint of each 20 km horizontal resolution
profile are selected (i.e., the average of the number densities reported at
the 30th and 31st consecutive single shot profiles for
each range bin).
Backscatter at 532 nm is computed by multiplying the modeled molecular
number densities by the 532 nm molecular
backscatter cross section. Two-way transmittance due to molecules and
ozone is obtained by first calculating extinction coefficients using the
532 nm molecular extinction cross section
and the ozone absorption cross section at 532
nm, converting these extinction coefficients to optical depths using
trapezoidal integration, and then calculating two-way transmittance.
Molecular model attenuated backscatter is the product of the calculated
backscatter and two-way transmittance.
An 80-byte (max) character string specifying the product name. For CALIPSO
lidar level 1.5 data products, the value of this string will be
"Level1.5_Science".
Date Time at Granule Start
A 27-byte character string that reports the date and time at the start of
the file orbit segment (i.e., granule) from the level 1B granule. The format
is yyyy-mm-ddThh:mm:ss.ffffffZ.
Date Time at Granule End
A 27-byte character string that reports the date and time at the end of the
file orbit segment (i.e., granule) from the level 1B granule. The format is
yyyy-mm-ddThh:mm:ss.ffffffZ.
Date Time at Granule Production
A 27-byte character string that defines the date and time of the level 1.5
granule production. The format is yyyy-mm-ddThh:mm:ss.ffffffZ.
Reports the
initial lidar ratios
at 532 nm and 1064 nm wavelengths used for aerosol extinction retrievals in
the lidar level 2 aerosol profile product.
This is the beta release of the lidar level 1.5 expedited data products. It
is a synthesis of three CALIPSO lidar expedited data products: level 1B, level 2
aerosol profile, and level 2 vertical feature mask. As such, the quality of
feature screening (cloud, surface, etc) and magnitude of reported attenuated
backscatter and aerosol extinction coefficients are heavily dependent on the
quality of these input data products - specifically the accuracy of level 1B
calibration and the strength of level 2 feature classification. Users are
advised to read the data quality summaries of these CALIPSO data products.
For this beta release, input granules used to generate the level 1.5 expedited
data product are expedited version 3.02 lidar level 1B, lidar level 2 aerosol
profile, and lidar level 2 vertical feature mask.
