MARS EXPLORATION ROVERS

PROJECT

   

Operations Product Generation Subsystem (OPGS)

Experiment Data Record (EDR) and

Camera Reduced Data Record (RDR)

Archive Volume

Software Interface Specification (SIS)

 

VERSION 1.7

 

 

 

JPL D-27086

July 25, 2005

 

Jet Propulsion Laboratory

California Institute of Technology


TABLE OF CONTENTS

 

1. Introduction............................................................................................... 1

1.1. Purpose and Scope................................................................................. 1

1.2. Content Overview.................................................................................. 1

1.3. Applicable Documents and Constraints....................................................... 2

1.4. Relationships with Other Interfaces............................................................ 2

2. Archive Volume Contents.............................................................................. 2

2.1. Root Directory Contents.......................................................................... 2

2.2. Data Directory Contents and Naming.......................................................... 3

2.3. Index Directory Contents......................................................................... 3

2.4. Document Directory Contents................................................................... 3

2.5. Catalog Directory Contents...................................................................... 4

2.6. Label Directory Contents (optional)........................................................... 4

2.7. Software Directory Contents (optional)....................................................... 4

2.8. Calib Directory Contents (optional)........................................................... 5

2.9. Geometry Directory Contents (optional)...................................................... 5

2.10. Browse Directory Contents (optional)....................................................... 5

2.11. Extras Directory Contents (optional)......................................................... 5

3. Archive Volume Format................................................................................. 6

3.1. Disk Format.......................................................................................... 6

3.2. File Formats.......................................................................................... 6

3.2.1. Document File Format..................................................................... 6

3.2.2. Tabular File Format......................................................................... 6

3.2.3. PDS Label Format.......................................................................... 7

3.2.4. Software File Format....................................................................... 7

3.2.5. Catalog File Format........................................................................ 7

3.2.6. Science Data File Formats................................................................. 7

4. Archive Volume Generation............................................................................ 8

4.1. Data Transfer, Validation Methods, and Peer Review....................................... 8

4.2. Interface Media Characteristics.................................................................. 8

4.3. Backup and Duplicates............................................................................ 8

4.4. Labeling and Identification...................................................................... 9

5. Support Staff and Cognizant Persons................................................................. 9

5.1. Data Providers...................................................................................... 9

5.2. PDS Contacts...................................................................................... 10

 

Appendix A – MER OPGS EDR AND CAMERA RDR ARCHIVE VOLUME CONTENTS

Appendix B – MER OPGS CAMERA RDR MOSAICS VOLUME CONTENTS

Appendix C – MER OPGS CAMERA RDR MESHES ARCHIVE VOLUME CONTENTS

 


DOCUMENT CHANGE LOG

 

Change

Date

Affected Portions

Initial Release, Version 1.0

06-13-04

All

Version 1.1

06-15-04

Updated Volume Set Ids, in Appendices, to contain "OP"

Version 1.2

06-23-04

Added Anaglyph datasets, RMC dataset. Removed Hazcam mosaic dataset.

Version 1.3

07-02-04

Removed references to Mini-TES throughout the document.

Version 1.31

07-11-04

Added browse file .LBL requirement and renamed the dataset catalog files.

Version 1.4

11-01-04

Added 6 new RDR datasets.

Version 1.5

03-01-05

Removed the 6 new RDR datasets and added the SOLAR RDR dataset. Rearranged the DATA directory structure for Appendix A.

Version 1.6

05-09-05

Moved the .svf and .rvf files to the EXTRAS directory.

Version 1.7

07-25-05

Removed the .svf and .rvf files from the EXTRAS directory as well as the RMC_SIS from the documents directory. Instead, they will become a new volume of ancillary data.

 

 

 

 


TBD ITEMS

 

Section

Description

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


ACRONYMS AND ABBREVIATIONS

 

APXS

Alpha Particle X-ray Spectrometer

ASCII

American Standard Code for Information Interchange

CODMAC

Committee On Data Management And Computation

CORNELL

Cornell University

DESCAM, DES

Descent Camera

DVD

Digital Video Disc

EDR

Experiment Data Record

HAZCAM, HAZ

Hazard Avoidance Camera

HTML

HyperText Markup Language

ILUT

Inverse Look-Up Table

IMG

Image

ISO

International Standards Organization

JPEG, JPG

Joint Photographic Experts Group

JPL

Jet Propulsion Laboratory

MB

Mšssbauer Spectrometer

MER

Mars Exploration Rover

MI

Microscopic Imager

MIPL

Multi-mission Image Processing Laboratory

NASA

National Aeronautics and Space Administration

NAVCAM, NAV

Navigation Camera

PANCAM, PAN

Panoramic Camera

NSSDC

National Space Science Data Center

OPGS (OPS)

Operations Product Generation Subsystem (Operations)

PDF

Adobe¨ Portable Document Format

PDS

Planetary Data System

RAT

Rock Abrasion Tool

RDR

Reduced Data Record

RMC

Rover Motion Counter

SIS

Software Interface Specification

TBD

To Be Determined


GLOSSARY

 

Archive – An archive consists of one or more data sets along with all the documentation and ancillary information needed to understand and use the data. An archive is a logical construct independent of the medium on which it is stored.

Archive Volume, Archive Volume Set – A volume is a unit of media on which data products are stored; for example, one CD-ROM or DVD-ROM. An archive volume is a volume containing all or part of an archive; that is, data products plus documentation and ancillary files. When an archive spans multiple volumes, they are called an archive volume set. Usually the documentation and some ancillary files are repeated on each volume of the set, so that a single volume can be used alone. 

Catalog Information – Descriptive information about a data set (e.g. mission description, spacecraft description, instrument description), expressed in Object Description Language (ODL), which is suitable for loading into a PDS catalog.

Data Product – A labeled grouping of data resulting from a scientific observation, usually stored in one file. A product label identifies, describes, and defines the structure of the data. An example of a data product is a planetary image, a spectrum table, or a time series table.

Data Set – An accumulation of data products. A data set together with supporting documentation and ancillary files is an archive.

 


1. Introduction

1.1. Purpose and Scope

This Software Interface Specification is intended to be used by those who wish to understand the format and content of the Mars Exploration Rovers (MER) Archives. Typically, these individuals would be software engineers, data analysts, or planetary scientists.    

The specifications in this document apply to the MER Operations Product Generation Subsystem (OPGS) Experiment Data Record (EDR) standard product archive volumes , MER Camera Reduced Data Record (RDR) standard product archive volumes, MER Camera RDR Mosaics archive volumes, and MER Camera RDR Meshes archive volumes that are generated by the MER Project.

The MER Archives are intended to be stored online for electronic distribution. The online version will conform to the structure described in this document. In addition, copies of the archives will be stored on physical media such as CDs or DVDs for long-term preservation. The requirements for these physical copies are described in section 4.

1.2. Content Overview

The MER OPGS EDR Archive volume set consists of the MER raw data products acquired and used during ground operations. The MER Camera RDR Archive volume sets consist of MER OPGS EDR derived data products. All archives contain documentation and other ancillary material.

The MER raw data products are OPGS EDRs, produced from telemetry data from instruments onboard the MER Project rovers. Telemetry data is processed into data records (CODMAC Level 2), with attached dual PDS/VICAR labels, by MIPL of the Jet Propulsion Laboratory. The MER Camera RDR products are produced from OPGS EDRs for the Pancam, Navcam, Hazcam, and MI instruments.

MIPL is the producer of all OPGS EDR and Camera RDR data records and is responsible for assembling the archives. The PDS is responsible for validating the archives for compliancy of structure and format against PDS specifications. The various MER instrument science teams are responsible for reviewing the archive in terms of science validity and integrity.

This Software Interface Specification (SIS) describes the format, content, and generation of the Mars Exploration Rover (MER) Archives. Section 2, Archive Volume Contents, describes the general structure of archive volumes and the contents of each file. Section 3, Archive Volume Format, describes the file formats used on the archive volumes. Section 4, Archive Volume Generation, describes the procedure for transferring data products to archive media. Section 5, Support Staff and Cognizant Persons, lists the individuals and institutions responsible for generating the archive volumes. Finally, Appendices A-D, describe the specific identifiers, specifications, and structure of the archive volumes produced along with a listing of any relevant documentation such as the Data Product SISs and schedules for release of data products.

1.3. Applicable Documents and Constraints

This Archive Volume SIS is intended to be consistent with the following documents:

1.    Mars Exploration Program Data Management Plan, R. E. Arvidson and S. Slavney, Rev. 3, March 20, 2002.

2.    Mars Exploration Rover Project Archive Generation, Validation and Transfer Plan, R. E. Arvidson and S. Slavney, JPL D-19658 Rev. B, July 14, 2004.

3.    Data Product SIS [Please refer to appendices for instrument specific SISs.]

4.    Planetary Data System Data Preparation Work­book, February 17, 1995, Version 3.1, JPL D-7669, Part 1.

5.    Planetary Data System Standards Refer­ence, August 1, 2003, Version 3.6, JPL D-7669, Part 2.

6.    ISO 9660-1988, Information Processing - Volume and File Structure of CD-ROM for Information Exchange, April 15, 1988.

7.    Universal Disk FormatŠ Specification, Revision 1.02, August 30, 1996, Optical Storage Technology Association (OSTA).

1.4. Relationships with Other Interfaces

This Archive Volume SIS could be affected by changes to the design of any of the MER standard data products (Applicable Document #3 ).

2. Archive Volume Contents

This section describes the general contents of the MER OPGS EDR and Camera RDR Archive volumes, including directory names, file names, file contents, file types, and organization responsible for providing the files. Volume set specific MER archive contents can be found in the appendices.

The MER Archives are organized with each volume set on a separate logical volume (Several small data sets may be stored together on one physical volume, and a particularly large data set may span more than one physical volume).  Each logical volume includes the required directories listed below, and may or may not include some or all of the optional directories.

2.1. Root Directory Contents (required)

Files in the Root Directory  include an overview of the archive, a description of the volume for the PDS Catalog, and a list of errata or comments about the archive. The following files are contained in the Root Directory.

 

File Name

File Contents

File Provided By

AAREADME.TXT

Volume content and format information

(e.g., PDS Node or Instrument Team)

AAREADME.HTM

Hypertext version of AAREADME.TXT (optional)

PDS Node

AAREADME.LBL

A PDS detached label that describes both AAREADME.TXT and AAREADME.HTM (optional, could be attached to AAREADME.TXT).

PDS Node

ERRATA.TXT

A cumulative listing of comments and updates concerning all archive volumes published to date

Data provider or
PDS Node

VOLDESC.CAT

A description of the contents of this volume in a PDS format readable by both humans and computers

PDS Node

2.2. Data Directory Contents and Naming (required)

Contents and naming scheme of the data sub-directories for specific instruments is described in the appendices. Data file naming format and nomenclature is described in each instrumentŐs respective Data Product SIS [Applicable Document #3].

2.3. Index Directory Contents (required)

Files in the Index Directory are provided to help the user locate products on this archive volume and on previously released volumes in the archive. The following files are contained in the Index Directory.

 

File Name

File Contents

File Provided By

INDXINFO.TXT

A description of the contents of this directory

PDS Node

INDEX.TAB

A table listing all data products on this volume

PDS Node or Data Provider

INDEX.LBL

A PDS detached label that describes INDEX.TAB

PDS Node or Data Provider

CUMINDEX.TAB

A cumulative listing of all data products on this volume and on previous volumes in this set

PDS Node or Data Provider

CUMINDEX.LBL

A PDS detached label that describes CUMINDEX.TAB

PDS Node or Data Provider

2.4. Document Directory Contents (required)

The Document Directory contains documentation to help the user understand and use the archive data. The following files are contained in the Document Directory.

 

File Name

File Contents

File Provided By

DOCINFO.TXT

A description of the contents of this directory

PDS Node

DPSIS.ASC or .HTM

The Data Product SIS as text or hypertext

Data Provider

DPSIS.PDF

The Data Product SIS as a PDF file

Data Provider

DPSIS.LBL

A PDS detached label that describes both DPSIS.TXT(HTM) and DPSIS.PDF

PDS Node

ARCHSIS.ASC or .HTM

The Archive Volume SIS (this document) as text or hypertext

PDS Node or Data Provider

ARCHSIS.PDF

The Archive Volume SIS (this document) as a PDF file

PDS Node or Data Provider

ARCHSIS.LBL

A PDS detached label that describes both ARCHSIS.TXT(HTM) and ARCHSIS.PDF.

PDS Node

[*.ASC files]

Other Documents

Data Provider

2.5. Catalog Directory Contents (required)

The files in the Catalog Directory provide a top-level understanding of the mission, spacecraft, instruments, and data sets. The files in this directory are coordinated with the PDS data engineer, who is responsible for loading them into the PDS catalog. The following files are found in the Catalog Directory.

 

File Name

File Contents

File Provided By

CATINFO.TXT

A description of the contents of this directory

PDS Node

DATASET.CAT

Data set information for the PDS catalog

Data Provider

INSTHOST.CAT

Instrument host (i.e., spacecraft) information for the PDS catalog

MER Project

INST.CAT

Instrument information for the PDS catalog

Data Provider

MISSION.CAT

Mission information for the PDS catalog

MER Project

PERSON.CAT

Personnel information for the PDS catalog (Team and PDS personnel responsible for generating the archive)

Data Provider

REF.CAT

References mentioned in other *.CAT files

Data Provider

SOFTWARE.CAT

Software information for the PDS catalog

Data Provider

2.6. Label Directory Contents (optional)

The Label Directory contains files that describe data format and organization. These files are referred to in the PDS labels that accompany the data products. They are "include" files that are intended to be parsed as if they were part of the PDS labels that refer to them.  The following files are contained in the Label Directory.

 

File Name

File Contents

File Provided By

LABINFO.TXT

A description of the contents of this directory

PDS Node

[*.FMT files]

Format files

Data Provider

2.7. Software Directory Contents (optional)

The Software Directory contains utilities or application programs to aid the user in viewing or extracting data from the data product files. The following files are contained in the Software Directory.

File Name

File Contents

File Provided By

SOFTINFO.TXT

A description of the contents of this directory

PDS Node

Software files

Software files, applications, source code, etc.

Data Provider

 

2.8. Calib Directory Contents (optional)

The Calib Directory contains calibration files used to process the data products, or calibration data needed to use the data products. The following files are contained in the Calib Directory.

File Name

File Contents

File Provided By

CALINFO.TXT

A description of the contents of this directory

PDS Node

Calibration files

Image Calibration Files

Data Provider

 

2.9. Geometry Directory Contents (optional)

The Geometry Directory contains files needed to understand observation geometry. The following files are contained in the Geometry Directory.

File Name

File Contents

File Provided By

GEOMINFO.TXT

A description of the contents of this directory

PDS Node

Geometry files

Spacecraft Geometry files

Data Provider

 

2.10. Browse Directory Contents (optional)

The Browse Directory contains reduced-size, easily viewed versions of data products to be used to help identify products of interest. The following files are contained in the Browse Directory.

File Name

File Contents

File Provided By

BROWINFO.TXT

A description of the contents of this directory

PDS Node

browse_image.JPG

Thumbnail size versions of the full resolution image files

Data Provider

browse_image.LBL

A PDS detached label that describes browse_image.JPG

Data Provider

 

2.11. Extras Directory Contents (optional)

The Extras Directory contains documentation, utility programs, or other materials that the user may find helpful, but that are beyond the scope of the required elements of the archive. The contents of this directory are exempt from PDS requirements for labeling, etc. The Extras Directory is intended for "value-added" material, handy to have but not crucial for understanding the data. An example would be a set of web pages for displaying the browse data. Since the directory is nonstandard, a thorough explanation of its purpose should be included. The following files are contained in the Extras Directory.

File Name

File Contents

File Provided By

EXTRINFO.TXT

A description of the contents of this directory

PDS Node

[other files]

 

Data Provider

 

3. Archive Volume Format

This section describes the format of the MER OPGS EDR and Camera RDR Archive Volumes. Data that comprise the Archive will be formatted in accordance with Planetary Data System specifications [Applicable Documents #4 and #5].

3.1. Disk Format

Archive Volumes have a digital video disk (DVD) format that is compatible with the computer operating systems MS‑DOS, Macintosh, and SunOS. The volume format is in accordance with ISO 9660 level 1 Interchange Standard or level 2, if any file names are longer than 8.3. [Applicable Documents #6 and #7].

3.2. File Formats

This section describes file formats for the kinds of files contained on Archive Volumes.

3.2.1. Document File Format

Document files with the .TXT suffix exist in the Root, Index, Software, Catalog, Document, and Label directories. They are ASCII files which may have embedded PDS labels. Lines in a .TXT file end with a carriage return character, <CR> (ASCII 13) and a line feed character, <LF> (ASCII 10). PDS recommends plain text files have line length restricted to 80 characters or fewer, including the <CR><LF>. This allows the files to be readable under various operating systems.

Documents in the Document directory may contain formatting and figures that cannot be rendered as ASCII text. Therefore each document is given in two formats, hypertext and PDF. The hypertext file contains ASCII text plus hypertext markup language (HTML) commands that enable it to be viewed in a Web browser such as Netscape Navigator or Microsoft Internet Explorer. The hypertext file may be accompanied by ancillary files such as images and style sheets that are incorporated into the document by the Web browser. The second format, PDF (Portable Document Format) is a proprietary format of Adobe Systems Incorporated that is frequently used for distributing documents. Adobe offers free software, Acrobat Reader, for viewing PDF files.

3.2.2. Tabular File Format

Tabular files (.TAB suffix) exist in the Index directory and in any data directory where the data consists of ascii tables. Tabular files are ASCII files formatted for direct reading into many database management systems on various computers. All fields are separated by commas and character fields are enclosed in double quotation marks ("). (Character fields are padded with spaces to keep quotation marks in the same columns of successive records.) Character fields are left justified, and numeric fields are right justified. The "start byte" and "bytes" values listed in the labels do not include the commas between fields or the quotation marks surrounding character fields. The records are of fixed length, and the last two bytes of each record contain the ASCII carriage return and line feed characters. This allows a table to be treated as a fixed length record file on computers that support this file type and as a text file with embedded line delimiters on those that don't.

         All tabular files are described by PDS labels that are either embedded at the beginning of the file or detached. If detached, the PDS label file has the same name as the data file it describes, with the extension .LBL; for example, the file INDEX.TAB is accompanied by the detached label file INDEX.LBL in the same directory.

3.2.3. PDS Label Format

                           All data files in the archive have PDS labels as detached files or embedded at the beginning of the file. For examples of PDS labels for each type of data product, see the Data Product SISs [Applicable Document #3].

         A PDS label, provides descriptive information about the associated file.  The PDS label is an object-oriented structure consisting of sets of 'keyword=value' declarations. The object to which the label refers to (e.g. IMAGE, TABLE, etc.) is denoted by a statement of the form:

                  ^object = location

in which the carat character (^, also called a pointer in this context) indicates where to find the object. The location is an integer representing the starting record number of the object (the first record in the file is record 1). Below is the format for the ^object definition.

                  ^object = n

where n is the starting record or byte number of the object, counting from the beginning of the file (record 1, byte 1).

3.2.4. Software File Format

Software is provided in a Zip-compressed file with a detached PDS label as specified in the PDS Standards Reference, chapter 20, Zip Compression. The Zip file includes all files required to use the software, including user manuals.

3.2.5. Catalog File Format

         Catalog files (suffix .CAT) exist in the Root and Catalog directories. They are text files formatted in an object-oriented structure consisting of sets of 'keyword=value' declarations.

Each line in a catalog file must be terminated by the two-character carriage-return/linefeed, <CR><LF>, sequence (ASCII decimal character codes 13 and 10, respectively). PDS requires catalog files have line length restricted to 72 characters or fewer including the <CR><LF>, to accommodate PDSŐ internal database requirements.

3.2.6. Science Data File Formats

See the Data Product SIS for each OPGS EDR product for descriptions of the data file formats.

4. Archive Volume Generation

4.1. Data Transfer, Validation Methods, and Peer Review

Data provided to the MER science teams will meet the specifications detailed in the Data Product SISs [Applicable Document #3].

The OPGS EDRs and Camera RDRs will be generated by MIPL of the Jet Propulsion Laboratory. MIPL is responsible for the assembly and production not only of PDS formatted data, but of complete, PDS-compliant archive volumes, e.g. DVDs. The MER science teams, with the assistance of their respective PDS Nodes, are responsible for the documentation and ancillary files pertaining to their data.

Before final delivery of the archive to PDS Central Node, the PDS Nodes will conduct both peer review and validation.  Peer review may be performed on the volume set as a whole, but validation must be performed on every individual volume.  The Central Node may perform additional validation once the volume has been received.

The purpose of the peer review is to confirm that the archive will be useable by members of the science community, both present and future, who are not familiar with the mission and/or instrument.  Reviewers include members of the PDS, a distributed representation of the project science teams, and members of the science community not associated with the mission. 

The purpose of validation is to verify that each volume adheres to PDS standards and to this Archive Volume SIS.

4.2. Interface Media Characteristics

All volumes in the MER Standard Product Archive conform to ISO 9660 standards [ISO 9660, 1988] and UDF standards [OSTA UDF Specification, Rev. 1.02, 1996].

4.3. Backup and Duplicates

At an absolute minimum, three physical media copies of each volume will be produced on write once physical media and delivered to the PDS Central Node, the respective PDS Node, and the NSSDC.

Until validated masters have been produced, or three validated write-once copies have been received by the PDS, volume contents shall be stored by the archive producer either on magnetic disc or write-once physical media.

4.4. Labeling and Identification

The two MER rovers are identified as shown in the table below.

 

Rover Name

Rover ID

Launch Date

Arrival Date

Spirit

MER 2 (MER A)

June 10, 2003

January 4, 2004

Opportunity

MER 1 (MER B)

July 7, 2003

January 25, 2004

 

Please refer to appendices for instrument specific labeling scheme of archive volumes.

5. Support Staff and Cognizant Persons

5.1. Data Providers

Ray Arvidson
MER Data and Archive Working Group Chair
Washington University
1 Brookings Drive, Campus Box 1169
St. Louis, MO 63130
USA
arvidson@wunder.wustl.edu

Jim Bell
MER Pancam Payload Element Lead
Dept. of Astronomy
402 Space Science
Cornell University
Ithaca, NY  14853
USA
jfb8@cornell.edu

Ken Herkenhoff
MER Microscopic Imager Payload Element Lead
U.S. Geological Survey
2255 N. Gemini Drive
Flagstaff, AZ  86001
USA
kherkenhoff@usgs.gov

Rudolf Rieder
MER APXS Payload Element Lead
Max Planck Institute
rieder@mpch-mainz.mpg.de

Gšstar Klingelhšfer
MER Mšssbauer Payload Element Lead
University of Mainz
klingel@mail.uni-mainz.de

Steve Gorevan       
MER Rock Abrasion Tool Payload Element Lead
Honeybee Robotics
gorevan@honeybeerobotics.com

5.2. PDS Contacts

Rafael Alanis
PDS Imaging Node
Jet Propulsion Laboratory
MS 168-414
4800 Oak Grove Drive
Pasadena, CA  91109
rafael.alanis@jpl.nasa.gov

Ed Guinness
PDS Geosciences Node
Washington University
1 Brookings Drive, Campus Box 1169
St. Louis, MO  63130
guinness@wunder.wustl.edu

Myche McAuley
PDS Imaging Node
Jet Propulsion Laboratory
MS 168-414
4800 Oak Grove Drive
Pasadena, CA  91109
myche.mcauley@jpl.nasa.gov

Susan Slavney
PDS Geosciences Node
Washington University
1 Brookings Drive, Campus Box 1169
St. Louis, MO  63130
slavney@wunder.wustl.edu

Betty Sword
PDS Central Node MER Data Engineer
Jet Propulsion Laboratory
MS 171-264
4800 Oak Grove Drive
Pasadena, CA  91109
betty.sword@jpl.nasa.gov