cloudThing logo in white
Menu open icon
cloudThing logo in white
Menu closed icon
email: info@cloudthing.com
tel: +44 (0) 121 393 4700

General

Generally useful pages

Sectors

We know loads about this stuff

What we do

The Building Blocks for cloudThing Magic

NASA satellite orbiting the earth with a view of the globe, clouds and sea in the background

Oops! NASA Forgot About AWS’s Egress Data Costs

NASA is upping its storage to 215 petabytes but forgot about retrieval costs AWS charge for taking data back from the Cloud

It was widely reported last year that NASA, after an extensive consultation period, had chosen AWS (Amazon Web Services) to run their Earthdata Cloud, a repository for data for the Earth Science Data and Information System (ESDIS) that collates all information collected from NASA’s missions.

NASA made the jump to AWS after realising that, by 2025, they’d have increased from around 32 petabytes of data to nearly 250 petabytes (which is a lot!)

Unfortunately, either no one told them or they forgot about the associated costs of retrieving the data they feed into AWS.

 

For anyone unsure, egress charges are a cost occurred on AWS when moving data from the Cloud to another area, in this instance we could imagine a local workstation for a NASA engineer or scientist.

Most AWS subscriptions will have them charged on top of a clients monthly Cloud bill and the more data that you retrieve… the more you’ll have to pay extra.

Previously NASA was storing all its data on prem, in 12 DAACs (Distributed Active Archive Centres) and was expecting to complete the transfer to AWS in the next couple of years.

 

If you’re wondering why NASA suddenly needs another 215 petabytes of information, it’s down to the fifteen upcoming missions they have planned, each of which is expected to produce around 100 terabytes of information… daily!

Those missions will include the launching of NISAR, the NASA-ISRO Synthetic Aperture Radar as well as the Surface Water and Ocean Topography (SWOT) satellites, with all data collected being directly uploaded to their Earthdata Cloud.

 

NASA’s main goal behind the migration was to have all their data available in the Cloud rather than siloed away in different geographical locations as it currently is.

This will give NASA’s scientists a huge boost… if they can afford the new costs!

 

The office of the Inspector General of NASA ordered a full audit of the project earlier this month, concluding that the Earth Observing System Data and Information System (EOSDIS), which makes the information from ESDIS available, hadn't properly taken in to account the additional costs egress charges would incur:

open quote mark

Specifically, the Agency faces the possibility of substantial cost increases for data egress (i.e., when end users download data from a network to an external location) from the cloud. Currently, when end-users access and egress data through a DAAC there is no additional cost to NASA other than maintaining the current infrastructure. However, when end users download data from Earthdata Cloud, the Agency, not the user, will be charged every time data is egressed. Ultimately, ESDIS will be responsible for both cloud costs, including egress charges, and the costs to operate the 12 DAACS.

In addition, ESDIS has not yet determined which data sets will transition to Earthdata Cloud, nor has it developed cost models based on operational experience and metrics for usage and egress. As a result, current cost projections may be lower than what will actually be necessary to cover future expenses and cloud adoption may become more expensive and difficult to manage.

Collectively, this presents potential risks that scientific data may become less available to end users if NASA imposes limitations on the amount of data egress for cost control reasons.

Office of the Inspector General of NASA

close quote mark

Perhaps even more worrying though was the fact that the report also highlighted the fact that the  Evolution, Enhancement, and Efficiency (E&E) panel chosen to review the DAACs didn't even attempt to identify potential cost savings (after the Mission Support Council removed the need to do so), didn't follow the National Institute of Standards and Technology's (NIST's) data integrity standards and lacked independence, with six of the twelve panel members working on ESDIS.

 

Following that the Office of the Inspector General for Nasa has made three recommendations…

 

  • As soon as NISAR & SWOT are up and running and providing sufficient data, a complete and independent analysis needs to be conducted  to determine the long-term financial sustainability of supporting the Cloud-migration to AWS whilst also maintaining the current DAAC footprint
  • Incorporate in appropriate Agency guidance language specifying coordination with ESDIS and OCIO early in a mission's life cycle during data management plan development
  • Ensure all applicable information types are considered during DAAC categorisation, that appropriate premises are used when determining impact levels, and that the appropriate categorisation procedures are standardised. 

Not quite ready to get back to work just yet?

Generate & Nurture Leads With Microsoft Dynamics 365 Sales

Case Study: Avery Weigh-Tronix’s Cloud Migration & Future Proofing

Working From Home? The UK Can Handle It

Contact us

Migrating to the Cloud may seem like an easy task but as you can see... even organisations like NASA can make mistakes. If you'd like any help or advice then feel free to get in touch with cloudThing today

Name

*

Company Name

*

Email Address

*

Telephone Number

Is there anything else you'd like us to know?

© cloudThing 2020

email iconinfo@cloudthing.com
© 2020 Copyright cloudThing ltd. All rights reserved. Company registered in England & Wales no. 7510381, VAT no. 152340739