Leveraging Cloud Computing to Improve Storage Durability, Availability, and Cost for MER Maestro
- Created: Saturday, 01 September 2012
The Maestro for MER (Mars Exploration Rover) software is the premiere operation and activity planning software for the Mars rovers, and it is required to deliver all of the processed image products to scientists on demand. These data span multiple storage arrays sized at 2 TB, and a backup scheme ensures data is not lost. In a catastrophe, these data would currently recover at 20 GB/hour, taking several days for a restoration.
A seamless solution provides access to highly durable, highly available, scalable, and cost-effective storage capabilities. This approach also employs a novel technique that enables storage of the majority of data on the cloud and some data locally. This feature is used to store the most recent data locally in order to guarantee utmost reliability in case of an outage or disconnect from the Internet. This also obviates any changes to the software that generates the most recent data set as it still has the same interface to the file system as it did before updates.
This software provides a seamless integration between existing software tools that would enable any mission across NASA to leverage the capability with minimal customization. It also unleashes a virtually limitless amount of storage and delivers it to projects without having to worry about provisioning, managing, and backing up large storage arrays.
The software integrates with Amazon Simple Storage Service (Amazon S3) service to provide the aforementioned solutions. By integrating with S3, unprecedented durability is delivered to the storage system with 99.999999999% data retention rate. Furthermore, it is a self-healing replication system that repairs objects automatically if they are ever lost. Since data is stored on a perobject basis rather than a file system mount, correlated loses of objects are extremely unlikely and recovery of each object is fast. This also reduces reliance on a single file system, where an outage can take the system offline for extended duration. The solution, built on cloud computing technology, reduces MER Maestro’s storage costs by over 80%. Most importantly, the solution is completely server-side, providing a seamless integration with existing clients without modifying any of their code or redelivering code.
An HTTP proxy was built that enables clients to access large amounts of data on S3 securely, and without any changes to existing software. The proxy caches information and is capable of accessing data from local channels as well as on S3. This enables the proxy to serve the most recent data from local storage, while the older archived data is retrieved ondemand from S3. The data stored on S3 is private and can only be accessed by the proxy. Furthermore, the proxy authenticates its users through JPL LDAP, and verifies their membership in a specific group before giving them access to the data.