This documentation in for Version 2 of the HEPSA data file stored in the DAAC archive. Contact: Dr. J. D. Winningham (PEM PI) Southwest Research Institute PO Drawer 28510 San Antonio, TX 78228 Date: September 8, 1997 Written by: Dr. R. A. Frahm Southwest Research Institute PO Drawer 28510 San Antonio, TX 78228 Modification Date: Modified by: General Information: The HEPSA data file contains the electron high-resolution spectral data converted to number intensity units from the HEPS telescopes mounted on the zenith UARS boom. HEPS electron data covers the rough range from 35 keV to 5 MeV. There are four telescopes mounted in different directions. These measure -15 deg, +15 deg, +45 deg, and +90 deg with respect to the spacecraft -z-axis and along the spacecraft +x-axis. The HEPS electron units accumulate a spectrum in 4.086 sec. See Winningham et al., J. Geophys. Res., 10649-10666, 98, 1993 for a description of the instrument. There are two HEPS units in this file (labeled HEPS1 and HEPS2), each containing two telescopes (labeled telescope 1 and telescope 2). Each telescope contains a stack of solid state crystals. The signals from each of the crystals are combined by energy processing electronics of the instrument to yield two logical crystals or detectors, called the DE and EE detectors. The DE detector has the rough energy range from 35 keV to 300 keV and the EE detector rough energy range is from 300 keV to 5 MeV. These designations are used in the description of HEPS electron data. The HEPSA data file is written in network binary format. HEPSA binary files have two types of records. The first is a header record which occurs only at the beginning of the file. After this header record comes a variable number of data records. Through out the HEPSA binary files, values of -1.0e-31 is used as a fill value. If this value occurs in a field, it means that that field is invalid. Values of +1.0e+31 are used to indicate that the data was excluded due to errors not directly related to the instrument. Values of +1.0e+31 are to be considered fill values. The HEPSA data file contains two types of records. The first type occurs as the first record in the data file and is called the header record. The second type of record occurs multiple times and is referred to as the data record. The header record contains that information which is common to each data record and applies to every data record. Other values which change their state more than once are stored in the data record. The header record contains the center energy values for each energy channel, the energy width of each channel, and the fractional error matrix used to determine the amount of error in each count value. The data record contains the start and stop time for each sample (year, day, millisecond of day), spacecraft parameters at the center of the data sample (latitude, longitude, altitude, invariant latitude determined at 600 km, magnetic solar time determined at 600 km, solar zenith angle), particle pitch angle at the center of each telescope, the differential number flux (or intensity), telescope quality, and the 8-bit compressed raw telemetry value. Quality values should be zero. A value of quality which is non-zero (including fill values) means that the entire sensor data is not valid, no matter what values occur within the data fields. The quality values use different numbers to indicate where failure occurred during processing and are used for diagnostic purposes only. Global Structure Definitions: Header Record Structure: float eh1t1de_eng[16]; 16 values of electron HEPS 1 Telescope 1 DE sensor center energy data in [eV] float eh1t1ee_eng[16]; 16 values of electron HEPS 1 Telescope 1 EE sensor center energy data in [eV] float eh1t2de_eng[16]; 16 values of electron HEPS 1 Telescope 2 DE sensor center energy data in [eV] float eh1t2ee_eng[16]; 16 values of electron HEPS 1 Telescope 2 EE sensor center energy data in [eV] float eh2t1de_eng[16]; 16 values of electron HEPS 2 Telescope 1 DE sensor center energy data in [eV] float eh2t1ee_eng[16]; 16 values of electron HEPS 2 Telescope 1 EE sensor center energy data in [eV] float eh2t2de_eng[16]; 16 values of electron HEPS 2 Telescope 2 DE sensor center energy data in [eV] float eh2t2ee_eng[16]; 16 values of electron HEPS 2 Telescope 2 EE sensor center energy data in [eV] float eh1t1de_ede[16]; 16 values of electron HEPS 1 Telescope 1 DE sensor channel width data in [eV] float eh1t1ee_ede[16]; 16 values of electron HEPS 1 Telescope 1 EE sensor channel width data in [eV] float eh1t2de_ede[16]; 16 values of electron HEPS 1 Telescope 2 DE sensor channel width data in [eV] float eh1t2ee_ede[16]; 16 values of electron HEPS 1 Telescope 2 EE sensor channel width data in [eV] float eh2t1de_ede[16]; 16 values of electron HEPS 2 Telescope 1 DE sensor channel width data in [eV] float eh2t1ee_ede[16]; 16 values of electron HEPS 2 Telescope 1 EE sensor channel width data in [eV] float eh2t2de_ede[16]; 16 values of electron HEPS 2 Telescope 2 DE sensor channel width data in [eV] float eh2t2ee_ede[16]; 16 values of electron HEPS 2 Telescope 2 EE sensor channel width data in [eV] float h_err[256]; 256 values consisting of a fraction of error assigned to each HEPS telemetry value, including Poisson statistics, telemetry compression, and instrument errors [unitless] Data Record Structure: long syear; year the data started accumulating [UT] long sday; day of year the data started accumulating [UT] long smsec; millisecond of day the data started accumulating [UT] long eyear; year the data stopped accumulating [UT] long eday; day of year the data stopped accumulating [UT] long emsec; millisecond of day the data stopped accumulating [UT] float cglat; latitude at center of sweep [degrees] float cglon; longitude at center of sweep [degrees] float cgalt; altitude at center of sweep [km] float cil600; invariant latitude determined at center of sweep for 600 km [degrees] float cmst600; magnetic solar time determined at center of sweep for 600 km [degrees] float csza; solar zenith angle at center of sweep [degrees] float cpa[8]; 8 values of pitch angle at center of sweep [degrees] - assignment (DE and EE crystals have the same look directions): cpa[0] = HEPS 1 Telescope 1 DE cpa[1] = HEPS 1 Telescope 1 EE cpa[2] = HEPS 1 Telescope 2 DE cpa[3] = HEPS 1 Telescope 2 EE cpa[4] = HEPS 2 Telescope 1 DE cpa[5] = HEPS 2 Telescope 1 EE cpa[6] = HEPS 2 Telescope 2 DE cpa[7] = HEPS 2 Telescope 2 EE float eh1t1de_dnf[16]; 16 values of electron HEPS 1 Telescope 1 DE differential number flux or intensity [(cm**2 sr s eV)**-1] float eh1t1ee_dnf[16]; 16 values of electron HEPS 1 Telescope 1 EE differential number flux or intensity [(cm**2 sr s eV)**-1] float eh1t2de_dnf[16]; 16 values of electron HEPS 1 Telescope 2 DE differential number flux or intensity [(cm**2 sr s eV)**-1] float eh1t2ee_dnf[16]; 16 values of electron HEPS 1 Telescope 2 EE differential number flux or intensity [(cm**2 sr s eV)**-1] float eh2t1de_dnf[16]; 16 values of electron HEPS 2 Telescope 1 DE differential number flux or intensity [(cm**2 sr s eV)**-1] float eh2t1ee_dnf[16]; 16 values of electron HEPS 2 Telescope 1 EE differential number flux or intensity [(cm**2 sr s eV)**-1] float eh2t2de_dnf[16]; 16 values of electron HEPS 2 Telescope 2 DE differential number flux or intensity [(cm**2 sr s eV)**-1] float eh2t2ee_dnf[16]; 16 values of electron HEPS 2 Telescope 2 EE differential number flux or intensity [(cm**2 sr s eV)**-1] unsigned char eq[8]; 8 values of electron sensor quality assignment: eq[0] = electron HEPS 1 Telescope 1 DE eq[1] = electron HEPS 1 Telescope 1 EE eq[2] = electron HEPS 1 Telescope 2 DE eq[3] = electron HEPS 1 Telescope 2 EE eq[4] = electron HEPS 2 Telescope 1 DE eq[5] = electron HEPS 2 Telescope 1 EE eq[6] = electron HEPS 2 Telescope 2 DE eq[7] = electron HEPS 2 Telescope 2 EE Note: values not equal to zero indicate bad and the data from that sensor should not be used unsigned char eh1t1de_raw[16]; 16 values of electron HEPS 1 Telescope 1 DE raw telemetry value (used in conjunction with the error table stored in the header record to determine the fractional error in the data value) unsigned char eh1t1ee_raw[16]; 16 values of electron HEPS 1 Telescope 1 EE raw telemetry value (used in conjunction with the error table stored in the header record to determine the fractional error in the data value) unsigned char eh1t2de_raw[16]; 16 values of electron HEPS 1 Telescope 2 DE raw telemetry value (used in conjunction with the error table stored in the header record to determine the fractional error in the data value) unsigned char eh1t2ee_raw[16]; 16 values of electron HEPS 1 Telescope 2 EE raw telemetry value (used in conjunction with the error table stored in the header record to determine the fractional error in the data value) unsigned char eh2t1de_raw[16]; 16 values of electron HEPS 2 Telescope 1 DE raw telemetry value (used in conjunction with the error table stored in the header record to determine the fractional error in the data value) unsigned char eh2t1ee_raw[16]; 16 values of electron HEPS 2 Telescope 1 EE raw telemetry value (used in conjunction with the error table stored in the header record to determine the fractional error in the data value) unsigned char eh2t2de_raw[16]; 16 values of electron HEPS 2 Telescope 2 DE raw telemetry value (used in conjunction with the error table stored in the header record to determine the fractional error in the data value) unsigned char eh2t2ee_raw[16]; 16 values of electron HEPS 2 Telescope 2 EE raw telemetry value (used in conjunction with the error table stored in the header record to determine the fractional error in the data value) Variable Lengths: A float type is the same length as a real*4 variable. A long type is the same length as an integer*4 variable. A char type is the same length as a byte variable. An unsigned char means that all 8-bits of the byte are to be interpreted as a number (do not interpret a bit as a sign). Timing of Data in the File: HEPS is a parallel acquisition instrument. the data accumulation time for each energy level is 4.096 seconds. Each energy level is accumulated simultaneously. HEPS has coincidence electronics which discriminates against particles which enter the telescope from outside of the collimator. This discrimination happens in the course of routine processing and accumulation within the instrument. Thus, HEPS data represents only those particles which enter the collimator. Data are accumulated for a specific time interval. This interval has resolution down to the millisecond. The data record includes the start time of accumulation and the stop time of accumulation. Values are given in year, day, and millisecond of day in Universal Time. Time has been corrected for the specific offsets of UARS telemetry and the UARS clock to reflect the correct times of accumulation. UARS O/A Data in the File: Six parameters are retrieved from the UARS O/A data and stored with this data file. They are the spacecraft geodetic latitude, east longitude, altitude, invariant latitude, magnetic solar time, and solar zenith angle. Invariant latitude and magnetic solar time are computed for 600 km altitude and are determined at the latitude-longitude position of the spacecraft. The units of geodetic latitude, east longitude, invariant latitude, and solar zenith angle are [deg]. Spacecraft altitude is given in [km] and magnetic solar time is given in [hour]. Spacecraft parameters are determined at the center of the data accumulation interval. HEPS Data in the File: The data of relevance in the HEPSA files are the spectral number flux values (also called intensities). They have units [(cm**2 sr s eV)**-1] and are acquired within the corresponding energy channel listed in the header record. The energy channel is identified by its center energy, given in units of [eV]. The width of each energy channel is also given in the header record in units of [eV] and represents the range over which the flux value was measured. The boundaries of the energy channel are determined by taking half of the energy channel width about the channel's center energy. For each measurement time period, the particle pitch angle at the center of each telescope is given. This value is the angle between the particle direction (measured by the detector -- opposite to the detector normal) and the magnetic field. The magnetic field used was determined by the 3-axis PEM science magnetometer to a resolution of 2 gamma. Precipitating particles have pitch angles less than 90 deg if the spacecraft is in the northern hemisphere and greater than 90 degrees if the spacecraft is in the southern hemisphere. The error in the measurement of the differential number flux is determined from the heps error look-up table and the raw telemetry 8-bit value. The raw telemetry 8-bit value is used as an index into the error table. The resulting value is the fractional amount of the flux which represents the standard deviation of the measurement. Table indices run from 0-255 (the raw telemetry 8-bit values run from 0-255). If interperting this table in a language which has indices starting at one, make sure you align the raw index with the first value in the table (you will add one to the raw 8-bit telemetry index in this case). So, the standard deviation of the measurement is the measurement value times the fractional amount of error determined from its corresponding raw 8-bit telemetry number. The measurement value has an associated error which is from the sum of the measurement and its standard deviation to the difference between its measurement and the standard deviation. With each sensor is a quality indicator. The quality indicator should be zero for good data. If the quality flag is not zero (for any reason), the entire data for that sensor should be considered bad and not used. Bad data quality is an indication that something went wrong during data correction procedures or to note times when data is considered contaminated and should be eliminated. HEPS Data Corrections: HEPS data consists of the 8-bit raw telemetry values that were measured on the spacecraft. These numbers, along with other instrumental and spacecraft parameters, are used to adjust the HEPS data to correct for instrumental effects. These effects are: (1) sunlight contamination, (2) live time correction, (3) background correction, (4) contamination due to falsely identified proton signals, (5) linear correction region. Each of these effects are corrected in this data and their correction is discussed below. Sunlight Contamination: When the HEPS telescopes view the sun, the solar UV penetrates the thin film window and deposits energy in the first crystal of the HEPS solid state detector stack. This is effectively the DE detector. When solar contamination occurs, the DE detector values are invalid because measurements are dominated by solar contamination. The values of the contaminated data still are processed by the instrument electronics and thus take up time processing bad signals. The DE count rate effects the remaining crystals, leading to an effect in the EE detector. Even though the DE data signal is excluded, its effect is not excluded during data processing. The EE detector measurements are adjusted for this contamination. Live Time Correction: When the instrument receives a signal, it must take time to process that signal. It can not receive or process more than one signal at a time. This produces times when the instrument is effectively dead. These times become a significant fraction of the detector's signal rate as the frequency of the signals becomes large. This translates into the instrument missing measurements because it is dead while busy processing another signal. The amount of time the instrument is live and ready to process signals is related to what is happening in the remainder of the telescope. HEPS data from both species (protons and electrons) is examined to determine the amount of live time correction that is needed. The data in this file reflects the results of correcting for detector live time. Background Correction: Low level noise is generated by solid state crystals. The HEPS units are susceptible to this noise because temperature effects. Background signals which develop because of thermal noise are present, but small. HEPS background noise increases as the temperature of the telescope increases. Temperature produces effects which are seen at the lowest count rates and these have been seen to vary through out the orbit. The HEPS units contain temperature sensors which allow the temperature dependent correction to occur. Background effects are removed by this temperature dependent correction. Falsely Identified Proton Signals: Low-energy protons can enter the detector and are stopped in the same crystal which measures the lower DE electrons. These signals are falsely identified as coming from electrons instead of from low-energy protons. The HEPS Low Energy Proton (LEP) telescopes make measurements close to the range of those falsely identified protons. The data from the LEP telescope is used to extrapolate the energy spectrum of the protons causing the falsely identified signals. Using this projected spectrum, the amount of signal from low-energy protons that deposit energy within the electron crystal is identified and removed from the electron signal. The energy of the protons creating the false signal in the electron channel is below the proton threshold for that telescope, but greater than the threshold that it takes for protons to penetrate the barrier of the telescope. Linear Correction Region Defined corrections to HEPS do not produce accurate corrected numbers when the count rates become high. This is due to correction algorithms becoming non-linear. To date, non-linear corrections to the HEPS data have not been defined; however limits has been placed on each telescope count rate beyond which linear corrections are not valid. Quality indicators reflect this condition. Some additional HEPS data should be able to be recovered when non-linear data corrections become defined. Instrument Related Effects in Data Not Yet Removed: There are several effects seen in the electron data which are known and have yet to be removed. They are (1) increase in flux at the DE-EE detector boundary, (2) flux limitation at low DE energies due to high count rates, (3) high count rates causing amplifier parallelization to dominate giving false quality, and (4) incorrect flux in lowest energy DE detector channel.. The corrections required are involved and in the case of (2), highly non-linear. Correction of these effects are not easy, but are some what correctable. DE-EE Detector Boundary: Bremsstrahlung is generated at the DE-EE interface (about 300 keV). This extra energy is transmitted preferentially in the forward direction, which is into the EE detector. This causes more energy to be deposited into the EE detector than what should be deposited. The result is often seen in the data as a spike of about twice the actual value. The actual magnitude of the Bremsstrahlung is spectral dependent. There are times when no detectable Bremsstrahlung is seen; however there are other times that this spike stands out clearly. Daily averages of the HEPS data seem to emphasize this spike, particularly in the radiation belts. Flux Limitation at Low DE Values: At the low DE energies, the spectrum can become exceed the limits on the instrument corrections. Instrument corrections are linearly based and the flux exceeds the region of linearity. This can be seen as a flattening of the spectral shape so that it approaches a slope which is parallel to the one count flux level. The spectral rotation and the amplitude of the spectrum are highly dependent on the details of spectral shape. Symptoms of approaching count rates which cause trouble are a curvature in the number flux spectrum at the lowest HEPS DE energy channels. Amplifier parallelization-False Quality: At extremely high count rates, the amplifiers within the HEPS instrument exhibit parallelization. Here the amplifier no longer produces a count which is indicative of the large counting rates, but outputs a count much less than the rate of pulses being sent into the amplifier. The total count passed to the instrument energy processors becomes low enough that its value becomes acceptable as a good quality count rate. In the DE detector, this effect is seen as a decrease in flux at the lowest DE energies while the number flux remains high at the higher DE energies (the number flux in the DE detector region looks like it has the opposite slope to what it should have produced). In principle, identification of regions where amplifier parallelization occurs should be correctable. At present, there is no method to automatically identify and correct times when amplifier parallelization occurs. When amplifier parallelization does occur, count rates become reduced to within the region identified as good quality. Thus, the quality indicator does not indicate times of amplifier parallelization. Identification of times of amplifier parallelization is not easy. It is recommended that the number flux spectrum be examined from HEPS and the MEPS electron unit at the nearest pitch angle for times around that in question. Due to the physics of how the electron distribution is generated and evolves, sharp discontinuities should not exist. There should also be a smooth transition between number flux values and these numbers should be decreasing in the energy range of 10 keV and above. By examing the spectral data on before and after the suspected amplifier parallelization, the development of electron spectra should be fairly continuous. Be very suspicious if you see the HEPS DE spectra showing bad quality when data before and after the amplifier parallelization event is examined and they show DE spectral flattening. Incorrect Flux in the Lowest DE Energy Channel At times it has been observed that the flux in the lowest energy channel of the DE unit is less than the flux in the next highest DE channel. This is a falsely reproduced data value and results from being the first energy channel within the upper crystal. The effect seems to be prominent at times when there is high flux in the lowest DE energy channels. This effect is identified when the intensity of the lowest DE energy channel is less than the intensity of the second DE energy channel. Data Reconstruction: The following "c" procedure is used to read the HEPSA data files. First define the header and data structures of a record: /******************************************************************************/ /* Header Record stores the Center Energies, Energy Widths, and the error */ /* fraction table used to determine the fraction of the differential number */ /* which is the standard deviation. */ /******************************************************************************/ struct HEAD { float eh1t1de_eng[16]; float eh1t1ee_eng[16]; float eh1t2de_eng[16]; float eh1t2ee_eng[16]; float eh2t1de_eng[16]; float eh2t1ee_eng[16]; float eh2t2de_eng[16]; float eh2t2ee_eng[16]; float eh1t1de_ede[16]; float eh1t1ee_ede[16]; float eh1t2de_ede[16]; float eh1t2ee_ede[16]; float eh2t1de_ede[16]; float eh2t1ee_ede[16]; float eh2t2de_ede[16]; float eh2t2ee_ede[16]; float h_err[256]; }; /******************************************************************************/ /* Data Record stores the Sweep Start and Stop times, Center O/A Values, the */ /* Center Pitch Angle Value of the Sensor, differential number flux (or */ /* intensity) Value, Sensor quality indicator, and the RAW count (used as */ /* an index for determining the error in the measurement). */ /******************************************************************************/ struct DATA { long syear; long sday; long smsec; long eyear; long eday; long emsec; float cglat; float cglon; float cgalt; float cil600; float cmst600; float csza; float cpa[8]; float eh1t1de_dnf[16]; float eh1t1ee_dnf[16]; float eh1t2de_dnf[16]; float eh1t2ee_dnf[16]; float eh2t1de_dnf[16]; float eh2t1ee_dnf[16]; float eh2t2de_dnf[16]; float eh2t2ee_dnf[16]; unsigned char eq[8]; unsigned char eh1t1de_raw[16]; unsigned char eh1t1ee_raw[16]; unsigned char eh1t2de_raw[16]; unsigned char eh1t2ee_raw[16]; unsigned char eh2t1de_raw[16]; unsigned char eh2t1ee_raw[16]; unsigned char eh2t2de_raw[16]; unsigned char eh2t2ee_raw[16]; }; Define the other variables that you will need to access data from binary reads on your system (don't forget the proper include files): #include #include #include #include #include #include #include int file_ptr, empty; char bfilename[80]; Then within your main routine or subroutine in which you read the HEPSA file, assing a pointer to your data definitions: struct HEAD header_rec; struct DATA data_rec; Open the desired data file for binary access: sprintf(bfilename,"PEM_HEPSA_1991313_V02.DAT"); if ((file_ptr = open(bfilename, O_RDONLY,0)) == -1) exit; Read the reader record: if ((empty=read(file_ptr,(char*)&header_rec,sizeof(header_rec))) < 0) exit; Read the data records: while ((empty=read(file_ptr,(char*)&data_rec,sizeof(data_rec))) > 0) { /* process the data records here */ } Don't forget to close the file when you are through: close(file_ptr); An example HEPSA binary read routine exists in the separate file called read_hepsa_binary.c.