Descartes Labs Achieves #41 in TOP500 With Cloud-Based Supercomputing Demonstration Powered by AWS

Descartes Labs, Inc. today announced a new cloud supercomputing achievement with a TOP500 run on AWS of 9.95 petaflops using virtualized Amazon Elastic Compute Cloud (Amazon EC2) instances. The TOP500 organization uses the LINPACK Benchmark”a test that involves solving a dense system of linear equations”to rank the performance of the 500 most powerful commercially available computer systems. Descartes Labs improved its previous cloud record of 1.926 petaflops set in its 2019 TOP500 submission (#136 ranking) with a 417% HPL performance improvement over two years, and improved performance by a factor of 10 relative to the performance gains achieved by traditional, on-premises supercomputers in the same two-year period. For 2021, the overall computation took approximately nine hours and performed 3 x 1020 total floating point operations across 172,692 Intel Xeon Scalable Processor cores.

The volume of global sensor data processed by Descartes Labs requires cutting-edge cloud technology capable of running High Performance Computing (HPC) and Machine Learning (ML) workflows at scale. This allows the Descartes Labs engineering teams to compute, analyze, and store near real-time data in order to deliver mission-critical business analytics at a global scale to customers in the agriculture, mining, defense and intelligence, and consumer packaged goods (CPG) industries. This capability is becoming increasingly important for applications that rely on a combination of geospatial analytics on a global level, such as environmental monitoring, severe weather tracking and forecasting, multimodal data fusion, and climate change analysis. An example of this mission is the six petabytes of Sentinel-1 radar data recently processed by Descartes Labs to help track deforestation at continental scale, understand where wildfire risk is highest, and monitor regenerative agriculture practices.

"Having the vast computational capacity offered by AWS, connected to tremendous amounts of storage that's accessible at very high speeds, lets you do things that you just couldn't do before in terms of data analysis and understanding the reality of our planet or the universe as a whole," said Mike Warren, Descartes Labs Cofounder and CTO. "Our cloud usage today spans the range of analysis of earth observation datasets at the petascale, particularly moving beyond just optical imagery into more esoteric types of earth observation such as radar, InSAR, and AIS data."

In collaboration with the AWS HPC team, Descartes was able to spin up the resources needed for its TOP500 run using general-purpose, publicly available Amazon EC2 instances across C5/M5/R5 families. Thanks to the scale and elasticity of the AWS Cloud, Descartes Labs was able to scale up infrastructure for this test without affecting its production workflows. Additionally, due to the elasticity of Amazon EC2's compute resources, Descartes was also able to test out different cluster sizes with multiple HPL runs spanning 1 petaflops to 8.5 petaflops, leading up to the final 9.95 petaflops result.

"The fact that Descartes Labs was able to use Amazon EC2 to build a 9.95 petaflops supercomputer ranking #41 on the TOP500 list is a truly impressive accomplishment," said Dave Brown, Vice President, Amazon EC2, AWS. "AWS is proud to support Descartes Labs in their mission to develop planetary scale geospatial insights for their customers. We expect the demand for instant data and insights to grow into the future, and this is a powerful demonstration of the value that AWS can bring to customers and partners on any scale."

With petabytes of earth observation and ground sensor data moving at the speed of collection, it's now the data that's driving the computational needs, not the other way around. The architecture of a traditional supercomputer is often limited in terms of cluster size and input/output (I/O). Having flexible computational capacity, connected to tremendous amounts of storage in the cloud, all accessible at very high speeds, lets Descartes Labs carry out science that the world couldn't accomplish before.

The LINPACK benchmark used in this supercomputing demonstration helps to approximate how fast a computer will perform when solving "tightly coupled" real-world problems, such as weather forecasting. By building systems that can run the LINPACK benchmark at high performance, Descartes Labs can use the same systems to model the weather quickly, accurately, and over multiple locations simultaneously to enhance agricultural yield forecasts.

Such cutting-edge science has been instrumental in Descartes Labs' collaboration with customers like CJ OliveNetworks, an organization that recently migrated all of CJ Group Companies' IT systems to AWS. The company leveraged Descartes Labs technology to carry out a purchasing innovation project to predict sugar input prices for CJ CheilJedang, which imports 600,000 tons of raw sugar per year. CJ has achieved successful business innovation from this project through satellite imaging technology, climate, and vegetation data analysis.

Says CEO of OliveNetworks and Group Chief Digital Officer In hyok Cha, "I wish to congratulate both Descartes Labs and AWS on achieving such a large scale computation to support the infrastructure work behind planetary sustainability initiatives, and also wish to continue to collaborate with the two companies to develop more innovative earth observation use cases and business applications."

The transition from on-premises to cloud computing mirrors the transition from just analyzing an organization's internal data to incorporating large volumes of geospatial and sensor data from the world beyond. Planetary scale data needs planetary scale compute. Accordingly, the size of this "outside the firewall" data requires the kind of cloud infrastructure Descartes Labs has carefully developed and used for this most recent TOP500 run. The company looks forward to helping others do the same.

"The cloud is where all the latest technology is showing up first. You're seeing all the latest processors from Intel and AMD in the cloud before you have access to them in your on-prem supercomputers," said Warren. "The investment in the networking technology to support these hyperscale applications in the cloud is turning out to be quite effective for traditional tightly coupled supercomputing applications as well. Now, the big push is into exascale in HPC. But at the same time, machine learning and deep neural networks are making huge progress in other types of problems. I think those worlds may come together, or they may drift further apart. And that's one of the big questions for the next decade."

Visit descarteslabs.com or learn more about our 2019 TOP500 achievement.

About Descartes Labs

Descartes Labs is a geospatial intelligence company that performs scientific analysis of geospatial, remote sensing, and diverse complementary data sets to enable sustainable sourcing best practices, commodity price forecasting, and efficient mineral exploration for leading CPG, Agriculture, and Mining companies. Our SaaS platform automates the analysis of geospatial imagery for our users, enabling planetary scale analysis through artificial intelligence and machine learning. The company also supports a diverse set of federal government efforts to curate, analyze, and provide unique actionable insights from geospatial data. For more information, visit www.descarteslabs.com or follow us on Twitter or LinkedIn.

Source: https://www.newswire.com/news/descartes-labs-achieves-41-in-top500-with-cloud-based-supercomputing-21419141

Tags: