How The Server Labs helps the European Space Agency use the cloud to process astronomic data from two billion stars.
In the run-up to launching its global space astrometry mission Gaia, the European Space Agency (ESA) tasked The Server Labs with building a platform to provide the massive data processing capability required. The Server Labs (TSL) built a grid/distributed computing system and ported it to the AWS cloud.
The European Space Agency’s ambitious Gaia project, a global space astrometry mission, creates a 3D map of unprecedented size and precision charting the composition, formation and evolution of over two billion stars, around 1% of our galaxy.
ESA estimated that if it took one millisecond to process one image, it would take 30 years of data processing time on a single processor. In terms of cost, processing the full data set with six years of data would have cost over one million Euros using inhouse data processing. So ESA asked TSL to help them find an efficient way to process the data.
In order to process the massive volumes of data from all the satellite observations (2 billion stars x 80 observations x 10 readouts) the TSL team built the Astrometric Global Iterative Solution (AGIS), a grid/distributed computing system based on data processing trains.
TSL began testing the technology on ESA’s in-house cluster. But the amount of data was increasing each month and year, and they needed a more scalable, cost effective solution. The magnitude of data and processing power required and the fact that the processing for AGIS was not continuous made the project an ideal candidate for the cloud.
In order to process the massive volumes of data from all the satellite observations (2 billion stars x 80 observations x 10 readouts) the TSL team built the Astrometric Global Iterative Solution (AGIS), a grid/distributed computing system based on data processing trains.
TSL began testing the technology on ESA’s in-house cluster. But the amount of data was increasing each month and year, and they needed a more scalable, cost effective solution. The magnitude of data and processing power required and the fact that the processing for AGIS was not continuous made the project an ideal candidate for the cloud.
Paul Parsons, CTO of The Server Labs describess the migration process: “It took us 20 man-days to port the software to EC2, and most of that time was spent configuring Oracle. To move the architecture to AWS, we created a 64-bit Amazon Machine Image (AMI) running Oracle Database 11g Enterprise Edition using Automated Storage Management (ASM) on top of Amazon Elastic Block Store (EBS)."
For the grid software, TSL created another AMI capable of running the three different types of data train used in AGIS. Paul Parsons says: "As all the software is written in Java, this process was quite straight forward. To get everything running we had to change only four lines of code to solve a thread synchronization problem that only occurred in virtual machines.”
The project has helped ESA to process data from a stellar census of over one billion stars in our galaxy, with the cloud platform performing six times faster than the equivalent in-house system. Not only is the ASW cloud platform faster, but it has delivered savings of over 50% when coompared to the cost of running the processing on an in-house platform.
Stars in the stellar census
AWS is faster than in-house computing platform
Financial savings by processing in the cloud
The move to the AWS cloud gave ESA substantial business and technical benefits:
The move to the AWS cloud gave ESA substantial business and technical benefits:
The Gaia AGIS Cloud experiment has been very successful for us. Bringing the data processing to the cloud provides us with savings of up 50% compared to using in-house hardware. Also, we can scale to far more processors than we could have in-house meaning that we can finish the job sooner.
Gaia Science Operations Development Director at European Space Agency
You can download the case study in PDF format from here
You can download the case study in PDF format from here
