Despite (or maybe because of my feeble explanations) the information contained here, I'm sure those who use these files will have at least some questions. Here is my contact info, please do not hesitate to get in touch with me. I am looking forward to working with GEWEX RFA participants. It is best to contact me via e-mail, because I travel a lot and am often not there to answer the phone or get voice-mails. All papers listed in "References" are available on request. Cheers, Chuck Dr. Charles N. Long WMO BSRN Project Manager NOAA ESRL GMD/CIRES 325 Broadway Street Boulder, CO 80305 e-mail: Chuck.long@noaa.gov telephone: 303-497-6056 ============================================================= Notes on the output files of the latest Flux Analysis code: These results are from the "next generation" of flux analysis based on the clear-sky detection and fitting techniques described in Long and Ackerman (2000), Long and Gaustad (2004), Long (2004, 2005), Long et al., (2006), and Barnard and Long (2004). Whereas the former analysis code dealt only with the SW portion of the downwelling radiative energy budget, this updated code now also includes the LW. This effort is a work-in-progress, and not all of the methodologies have undergone peer review. I am releasing these results to those interested with the understanding that some of these variables (as described below) are at this point preliminary results only. Calculated variables that are considered "solid": Estimates of clear-sky downwelling GlobalSW, DifSW, DirSW; SW fractional sky cover; Cloud optical depth for sky cover > 0.95; effective Cloud transmissivity; clear-sky downwelling LW and clear-sky broadband effective emmisivity. Calculated variables that are considered "good": LW sky cover, clear-sky upwelling SW Some calculated variables "not yet proven": Clear-sky upwelling LW, Cloud temperature and height estimates Notes: The cloud optical depth estimates are based on a technique by Barnard and Long (2004). This technique, an empirically derived relationship based on the results of Min and Harrison (1996), is officially only valid for overcast skies (sky cover > 0.95) of liquid water clouds. Thus the current output includes cloud optical depth only for sky cover > 0.95 for now. Also, recent comparisons conducted as part of the ARM CLOWD project suggest that the Min and Harrison technique itself tends to overestimate the cloud optical depth for thinner clouds (Tau < 5) (Dave Turner, personal communication), thus so does the Barnard and Long technique. Finally, these are "effective" optical depths in that they assume a single uniform liquid cloud layer with an effective radius of 10 microns and an assymetry parameter of 0.87. The estimated clear-sky downwelling LW is derived from a technique based on Brutsaert (1975). Unlike the Brutsaert formulation, we use the known clear-sky periods and the corresponding measured clear-sky downwelling LW to calculate lapse rate coefficients. We then interpolate these calculated lapse rate coefficients for cloudy periods, similar to the SW technique. Preliminary comparisons show that about 85% of the estimated clear-sky LW falls within 5 W/m^2 of the corresponding clear-sky measured LW (Long, 2004). The added uncertainty due to interpolation has yet to be determined, but preliminary model comparisons indicate an agreement similar to that above. There is a known "problem", however, in that the only information available for LW estimation is surface measurements. For those times of abrupt major changes in temperature or humidity profiles significantly differing from the data the lapse rate coefficients were determined from, such as cold front passages, the clear-sky LW estimates will exhibit greater error. Fortunately, these conditions occur infrequently. The LW effective sky cover is from a technique developed by Durr and Philipona (2004), but with some differences. Durr and Philipona use a climatologically derived and applied formulation for clear-sky effective broadband LW emissivity, whereas those here are derived from surrounding clear-sky data. In addition, Durr and Philipona use a calculation of downwelling LW standard deviation for the hour preceding the time of interest in their sky cover prediction, where here I use a running 21-minute standard deviation centered on the time of interest. The varible is deemed as the "effective LW sky cover" in that the downwelling LW at the surface is insensitive to high and thin clouds, thus the sky cover is essentially most representative of the amount of low and mid-level cloudiness (Long, 2004). The original Durr and Philipona retrieval is in Oktas, so the inherent uncertainty is at least 1/8 of sky cover. I use an 11-minute running mean to smooth the results. ARM is working on fielding an Infrared Sky Imager that should provide the data needed to refine the (or even develop a new) approach, similar to how I used TSI data to develop the SW sky cover technique. CSWup - There are identified problems associated with guesstimating upwelling SW measurements using only detected clear-sky measurements, and then interpolating fit coefficients as we do for the downwelling SW (Long, 2005). For instance, when it snows, it's cloudy, thus the "fit" is way off until the next "clear enough" day for fitting after the snow event. This introduces a large error during the period, and for times of snow melt. Data show that the bi-directional reflectance function also changes over time depending on the surface characteristics. Thus, the current procedure for estimating clear-sky upwelling SW is to look through the data and take a daily average for all data from 1100 through 1300 local standard time. This captures, at least on a daily basis, the major changes in surface albedo such as those from snow accumulation or snow melt. A second pass through the data then uses the "daily noon average" as a constant, and determines a function for any data that include at least 25% of the total SW produced by the direct component (i.e. significant direct sunlight producing the bi-directional nature of the albedo dependence) using the cosine of the solar zenith angle as the independent variable. Again, these fit coefficients are interpolated for days when insufficient data are available for fitting. The function is then multiplied times the estimated clear-sky SWdn to produce a continuous estimate of clear-sky SWup. My examination of these results so far suggest this technique does pretty much eliminate the "gotcha" of it always being cloudy when it snows, and does a better job than just multiplying the measured albedo (SWup/SWdn which often behaves erratically through time depending on whether the direct sun is blocked by cloud or not) times the clear-sky SWdn. CLWup - The clear-sky upwelling LW uses the same detected SW and "LW effective" clear-sky data to empirically derive fit coefficients that are again interpolated for cloudy periods (Long, 2005). In this case, since the upwelling LW is tied to the total surface energy exchange including latent and sensible heat,the independent variables used are the downwelling LW, the net SW, 2 meter relative humidity, and wind speed. These last are used as surrogates to help account for the unknown relative changes in surface sensible and latent heat exchange. Comparisons show that over 90% of the estimations agree with detected clear-sky LWup measurements to within 5 W/m^2. Though estimation of the accuracy of the interpolated values has yet to be investigated, visual inspection indicates that the results appear reasonable. Cloud field temperature and height estimates - these are "work in progress". I use the measured and clear-sky estimated LWdn, the LW sky cover amount, and Independent Pixel Approximation arguments to estimate the LW effective radiating ("cloud") temperature. The uncertainty in this estimation is largely driven by the uncertainty associated with the LW effective sky cover. The value generated assumes a single layer of cloudiness covering the "LW sky cover" portion of the sky, and with uniform radiating properties. Thus this value is best described as an "effective cloud field radiating temperature" with all the assumptions that the word "effective" usually implies. In addition, given a good cloud radiating temperature estimate, one must then figure out how to reasonably translate that temperature to a cloud height. I use here the difference between the estimated cloud field radiating temperature and the ambient air temperature, and a simple 10-degree-C-per-km lapse rate to estimate the effective cloud field radiating height. Note that the imaginary "radiating surface" relates approximately to about one optical depth into the cloud, and so is NOT located at the same height as the cloud physical boundary as would be determined by a lidar or cloud radar. Also, this is an estimate as if all cloudiness were in a single uniform layer. Again, this is a work in progress. Use these at your own risk for now. Monthly Diurnal and Monthly Average files: There are many ways to skin a cat, and there has been considerable discussion with reference to producing monthly averages and what to do about missing data as part of the surface RFA effort. Somehow trying to manufacture numbers to "fill in" for missing data has always made me nervous. For this data set, I have chosen not to attempt any "filling in", but only to use the data available, with a minimum limit in order to produce an average. For the monthly diurnal cycle files, I put all available data into 15-minute bins, and then if there is at least half of the possible data available, take an arithmetic averge of the values in the 15-minute bin. Once a month's diurnal cycle is produced, then again if there is at least half the possible data, then a monthly average for that value is produced. This is about the simplest way of averaging without "filling in", while at the same time mitigating the problem of WHEN (i.e. what solar elevation angles) especially solar variables are missing that is inherent in using just a straight arithmetic average of all available data to produce a monthly average. The choice of "if half the data are available" is arbitrary. In all cases for the monthly diurnal and average files, the actual number of data used in a given average is provided so that those who want a tighter restriction can screen for it. In the case of the monthly diurnal files, a corresponding file is provided that lists the number of data used. For example, the monthly diurnal file for Barrow, Alaska (BAR_diurmnth.txt) includes all the average values themselves, while the corresponding "number of data" file (BAR_diurmnth_ndt.txt) includes the data counts. For the monthly average files, both the average values, and the number of data counts are all in the same file. Thus again, those who want to screen more tightly for whether to use a particular average value can do so with the information provided. All files (monthly diurnal and monthly average, 2004 15-minute) are provided in ASCII format, with each row of data time stamped at the beginning of the row followed by the variables. Variables are in colomns, with a "header row" describing the variable in that column. Descriptions of the header abberviations are given below. The year 2004 15-minute files are provided in daily files, i.e. one day of data per file, and these files have been tar bundled for each site that I had access to 2004 data for. Output Files Header Description: In all the files provided for the RFA (15-minute data for 2004 [*_2004.TAR files], monthly diurnal cycles at 15-minute resolution [*.diurmnth.txt files], monthly averages [*.avg_mnth.txt files]), variable names are mostly standardized. The following is a listing of the common column header abbreviations, and a description of the variable so labeled: YYYYMM year and month of year (used in monthly average and diurnal files) hhmm hour and minute (used in monthly diurnal files, based on LST) Zdate date in YYYYMMDD format, based on GMT Ztim time in hhmm format, based on GMT Ldate date in YYYYMMDD format, based on LST Ltim time in hhmm format, based on LST CosZ Cosine of the solar zenith angle AU earth-sun distance in AUs SWdn best estimate downwelling SW from sum or global pyranometer (W/m^2) CSWdn estimated clear-sky downwelling SW (W/m^2) LWdn downwelling LW from pyrgeometer (W/m^2) CLWdn estimated clear-sky downwelling LW (W/m^2) SWup upwelling SW from pyranometer (W/m^2) CSWup estimated clear-sky upwelling SW (W/m^2) LWup upwelling LW from pyrgeometer (W/m^2) CLWup estimated clear-sky upwelling LW (W/m^2) DifSW measured downwelling diffuse SW (W/m^2) CDifSW estimated clear-sky downwelling diffuse SW (W/m^2) DirSW measured downwelling direct SW (W/m^2) CDirSW estimated clear-sky downwelling direct SW (W/m^2) LWScv estimated LW effective fractional sky cover SWScv estimated SW (total) fractional sky cover CldTau estimated effective visible cloud optical depth CldTrn estimated effective SW cloud transmissivity (SWdn/CSWdn ratio) CldTmp estimated effective cloud radiating temperature (K) CldHgt estimated effective cloud height (km) Tair air temperature (K) VPrs vapor pressure (mb) RH Relative Humidity (%) RHfac RH-based adjustment to Ec to account for haze formation Ec effective clear-sky LW emissivity using Ta Wspd Wind speed (m/s) Wdir Wind Direction (degrees from North) WARNING: these are arithmetic averages, NOT PROPER VECTOR AVERAGES Aprs Air pressure, usually in mb, but sometimes in hPa LWlw (2004 15-minute files) Contribution to CLWup from CLWdn variable (W/m^2) SWlw (2004 15-minute files) Contribution to CLWup from SWnet variable (W/m^2) RHlw (2004 15-minute files) Contribution to CLWup from RH variable (W/m^2) Wslw (2004 15-minute files) Contribution to CLWup from Wspd variable (W/m^2) NSWClr (monthly average and diurnal files) number of SW (i.e. totally) clear-sky data detected NAllClr (monthly average and diurnal files) number of SW plus LW (i.e. "LW Effective") clear-sky data detected ClrF (2004 15-minute files) number of SW plus LW clear-sky detected PossN (2004 15-minute files) number of time stamped lines in input file, so possible number in each individual average NPoss (monthly average and diurnal files) possible number of data Minnum (monthly average and diurnal files) minimum number required to produce an average NOTE: In the monthly average files, the count of number of data used in a particular average is labeled with an "N" in front of the variable header abbreviation. For example, for SWdn, the number of data is labeled "NSWdn". For the monthly diurnal files, a separate file with the same stem, but extension "_ndt.txt" gives the number of data counts with the same header abbreviation as for the variable itself in the monthly diurnal data file. For example, the column abbreviation "SWdn" is in both the monthly diurnal ".txt" and "_ndt.txt" files. There may be other columns of data if the provider used the option to include up to 20 extra variables. Hopefully the column header abbreviations in this case are self-explanatory as to what the variables are...if not, contact me for more info. ===================================================================================== REFERENCES: Barnard, J. C., and C. N. Long, (2004): A Simple Empirical Equation to Calculate Cloud Optical Thickness Using Shortwave Broadband Measurement, JAM, 43, 1057-1066. Brutsaert, W., (1975): On a Derivable Formula for Longwave Radiation from Clear Skies, Water Resour. Res., 11(3), 742ˆ 744. Durr B. and R. Philipona, (2004): Automatic cloud amount detection by surface longwave downward radiation measurements, JGR, 109, D05201, doi:10.1029/2003JD004182. Long, C. N. and T. P. Ackerman, (2000): Identification of Clear Skies from Broadband Pyranometer Measurements and Calculation of Downwelling Shortwave Cloud Effects, JGR, 105, No. D12, 15609-15626. Long, C. N., T. P. Ackerman, K. L. Gaustad, and J. N. S. Cole, (2006): Estimation of fractional sky cover from broadband shortwave radiometer measurements, JGR, In Press. Long, C. N. and K. L. Gaustad, (2004): The Shortwave (SW) Clear-Sky Detection and Fitting Algorithm: Algorithm Operational Details and Explanations, Atmospheric Radiation Measurement Program Technical Report, ARM TR-004, Available via http://www.arm.gov/publications/techreports.stm. Long, C. N., (2004): The Next Generation Flux Analysis: Adding Clear-sky LW and LW Cloud Effects, Cloud Optical Depths, and Improved Sky Cover Estimates, 14th ARM Science Team Meeting Proceedings, Albuquerque, New Mexico, March 22-26, 2004. Long, C. N., (2005): On the Estimation of Clear-Sky Upwelling SW and LW, 15th ARM Science Team Meeting Proceedings, Daytona Beach, Florida, March 14-18, 2005. Min, Q., Harrison, L. C., (1996): Cloud Properties Derived from Surface MFRSR Measurements and Comparison with GOES Results at the ARM SGP Site, Geophysical Research Letters, Vol. 23, pp. 1641-1644.