Creating ChatGPT-style tools with Earth observation
Creating ChatGPT-style tools with Earth observation Consider being able to query a chatbot questions like, “Are buildings sinking in my street?” or “Can you create me a highly precise map for crop cultivation in Kenya?” And suppose that the data that is returned is based on credible Earth observation data and is supported by science.
ESA is creating AI applications that will transform retrieval of data in Earth observation, in collaboration with technology partners, to bring such a tool to reality.
An electronic assistance program for information
Every day, earth observation produces enormous amounts of important data, but it is challenging for humans to make sure we get the most out of that data. Thankfully, artificial intelligence (AI) makes it easier to work with such vast and complicated datasets, detecting important characteristics and presenting the data in an approachable way.
For instance, I*STAR, a project supported through the ESA InCubed program, created a platform that use AI to track current occurrences such as earthquakes and volcanic eruptions, enabling satellite operators to autonomously schedule the next data collections for clients.
InCubed once more provides support to the SaferPlaces AI tool, which combines satellite data and in situ measurements to produce flood maps for preparedness teams. During the Italian region of Emilia-Romagna’s floods last year, SaferPlaces played a critical role in damage assessment operations.
AI has advanced remarkably in the previous several years, even stunning specialists in the area with the capabilities of ChatGPT and Gemini. Building a ChatGPT-style text-based inquiry using Earth observation data is a logical next step to take advantage of this disruptive discovery and seize the opportunities made possible by this technology.
ESA is now working with a number of partners in the meteorological, computer, and space domains to develop natural language skills for an Earth-observing digital assistant. These capabilities will allow the assistant to comprehend human questions and provide responses that are similar to those of a person.
Naturally, nevertheless, there are a lot of puzzle pieces to fit together in order to build a digital assistant of this kind. Let’s start with the foundation model, which is the engine that powers the whole thing.
While more conventional machine learning requires the computer to be fed massive amounts of labeled data, frequently by a human, the powerful AI models operate by learning and improving over time.
Now for foundation models, which operate in a completely different way. A foundational model is an automatic learning model that learns on a substantial and diverse set of unlabeled data sources, mostly without human supervision. Although foundational models are quite generic, they can be customized for particular uses.
The outcome is a versatile, potent AI engine. Since foundation models were first introduced in 2018, they have greatly impacted several businesses and society at large by revolutionizing machine learning.
There are many current projects at ESA Φ-lab to develop foundation models specifically for Earth observation applications. These models make use of data to offer information on ecologically important subjects including mitigating the effects of extreme weather events and methane leaks.
Starting in early 2023, PhilEO is one foundational model project that is now nearing maturity. To encourage cooperation, progress the field, and guarantee that the derived the basis model is thoroughly validated, the Earth observation group will soon have access to the PhilEO model itself as well as a system for evaluation based on worldwide Copernicus Sentinel-2 data.
The Richat Structure is the kind of structure which the PhilEO system has become adept at identifying on its own without human guidance.
The interaction between humans
The human piece of the jigsaw puzzle is being studied by several ESA efforts. These involve developing a digital assistant that can answer questions from users in plain language, analyze the appropriate data using Earth observation foundation models, and provide a response in text or visual format.
A forerunner project called Digital Twin of Earth has recently shown that its virtual assistant prototype can do multimodal tasks, including information comparison by searching across several data archives like Sentinel-1 and 2.
Beginning in April, an ESA Π-lab project will investigate the processing of natural languages for information extraction and analysis from certified Earth observational text sources, as well as for understanding questions from experts and non-expert users. Ultimately, this endeavor will result in the creation
“It’s a fascinating idea to have an Earth observation computer program that can offer a wide range of information from multiple sources, and these initiatives demonstrate that there are several essential foundational elements to establish in order to accomplish that goal,” says Giuseppe Borghi, Head of ESA’s Π-lab.
“With Philadelphia and the digital companion prototype already making such encouraging strides, I am confident that the new projects will soon produce game-changing outcomes.”
