The ITI: Instrument to Instrument Translation tool rapidly aligns data from different instruments: combining insights and unlocking undersued potential.  

There are a multitude of satellites producing myriad data.  Scientists need curated access to data to answer vital questions about the Sun, the Earth and more. 

Created in 2022 to align solar observatory data, including SDO/AIA+HMI, Hinode/SOT, SoHo/EIT+MDI, STEREO/EUVI, and KSO, providing stable enhancement of solar images and restoration of long-term image time series [Jarolim et al. 2023].

ITI harmonizes data across multiple instruments; enabling scientific queries for Heliophysics and Earth Science. It represents a first step towards multi-instrument ‘sensor webs’ and is a key element of Trillium’s Network Intelligence in Orbit initiative. NIO.space

The team

Christoph Schirninger, Robert Jarolim, J. Emmanuel Johnson , Anna Jungbluth,  Richard Galvez, Lilli Freischem, Cormac Purcell, Noah Kasmanoff, Anne Spalding 

Acknowledgements

This research is funded through a NASA 22-MDRAIT22-0018 award (No 80NSSC23K1045) and managed by Trillium Technologies, Inc (trillium.tech)

NASA has more than a dozen Earth science satellites in orbit. They help NASA study the oceans, land and atmosphere. Credits: NASA

ITI Heliophysics Goals

ITI is built on unpaired image-to-image translation, which enables a wide range of applications, where no spatial or temporal overlap is required between the considered datasets.This approach enables instrument intercalibration, image enhancement, mitigation of quality degradations, and super-resolution across multiple wavelength bands. 

We highlight ITI as a general tool for Heliospheric applications and demonstrate its capabilities by applying it to data from Solar Orbiter/EUI, PROBA2/SWAP, and the Solar Dynamics Observatory/AIA in order to achieve a homogenous, machine-learning ready dataset that combines three different EUV imagers. 

The direct comparison of aligned observations shows the close relation of ITI-enhanced and real high-quality observations. The evaluation of light-curves demonstrates an improved inter-calibration.

Solar Orbiter EUI

  • Intercalibration Full Sun Imager (FSI) to SDO AIA

  • Super-resolve AIA observations with data from High-Resolution Imager (HRI)

PROBA2 SWAP

  • Intercalibration SWAP to AIA

ITI Earth Science Goals

While ITI was originally developed by for heliophysics applications and to homogenize solar observations, we are now extending this approach to homogenize image data collected by Earth-observing satellites. 

Our approach can be applied to unpaired images and does not require input and target observations to be spatially or temporally aligned.

This enables the translation between instruments without shared observing periods (e.g. historic datasets, different observing times or measurement cadences) or spatial coverage (e.g. different vantage points or satellite orbits). 

To fully demonstrate the framework potential, we apply the ITI tool in two Earth observation use cases: (1) intercalibration of geostationary satellites, and (2) translation between geostationary and polar-orbiting satellites.

Translation between geostationary & polar orbits

  • GOES-16 / ABI

  • TERRA & AQUA / MODIS

Translation between geostationary orbits

  • GOES-16 / ABI

  • MSG / SEVIRI

ITI Framework

The ITI tool is a high technology readiness level (TRL) framework that can be applied to multiple research domains.  Our ITI framework is built on generative models that enable intercalibration, image enhancement, degradation correction, and super-resolution across multiple spectral bands. Our model consists of two neural networks, with the first network generating synthetic low-quality images from high-quality target observations, and the second network learning to invert the image degradation to reconstruct the original input. Image degradation is performed via a discriminator to match the characteristics of lower quality input images that need translating to the target image domain. Since the networks are competitively trained against each other, the model learns to create and correct increasingly diverse and realistic low-quality images.

The open-source framework is developed to download and pre-process the original satellite data into machine-learning ready datasets. Benchmarking and validation metrics are available to evaluate model outputs.  Following the standard software engineering practices, this framework is broken into the following components: a) data ingestion, b) data preprocessing, c) machine learning ready data, d) model training, e) model inference and predictions, f) visualization, g) validation and analysis, h) documentation and tutorials, and i) API access and hosting to such models and data products.

The ITI tool is divided into three projects hosted on GitHub

Get involved

ITI can be used for a myriad of downstream applications including detection, estimation and feature extraction.   

To use ITI, start by Reading the Documentation. (Note the documentation is evolving as we build the tool and will include tutorials and codelabs in the near future)

Email anne@trillium.tech for any feedback or questions on this project.