Jet Propulsion Laboratory

OIIP: Oceanographic In-situ data Interoperability Project

"Leveraging available technologies for improved interoperability and visualization of remote sensing & in-situ oceanographic data at PO.DAAC"

OIIP Logo

Overview

The Oceanographic In-situ data Interoperability Project (OIIP) is a two-year technology development project initiated in September 2016 and funded under NASA's Advancing Collaborative Connections for Earth System Science (ACCESS) Program. This collaborative project aims to leverage and extend as necessary a set of available, higher TRL (Technology Readiness Level) informatics technologies for the improved integration of oceanographic in-situ and satellite datasets. The earth science enterprise increasingly relies on the integration of multivariate data from diverse observational platforms to enable research, mission calibration/validation, and decision support. The project seeks to enhance technical capacity in support of this broader objective by developing tools and approaches addressing key interoperability issues that are particularly acute for field campaign data sets.

Team

The project is a collaborative venture between the Jet Propulsion Laboratory (JPL), UCAR/Unidata, and the Large Pelagics Research Center (LPRC) at the University of Massachusetts, Boston. Members of the project team include: V. Tsontos (PI) and co-investigators C. Thompson, N. Quach (JPL), C. Lam (UMASS/LPRC) and S. Arms (UCAR/Unidata), and JPL developers F. Platt and J. Roberts. OIIP also acknowledges the involvement and support of our external project collaborators Wildlife Computers.

Goals

Overarching goals of the project include:

Enhance available earth science data technologies to address key interoperability and data challenges associated with oceanographic in-situ datasets, focusing on marine animal electronic tagging data as a representative (but also more challenging) use case. OIIP technology components include: the NCEI/NODC netCDF templates, Unidata's ROSETTAand THREDDS software, JPL's Common Mapping Client (CMC) and Data management & Archive System (DMAS) technologies, and the Tagbase relational database system (LPRC).
Develop improved PO.DAAC capacity to support NASA field campaign data via the integration of these technology components within system workflows with a view to operational DAAC infusion
Illustrate the value-added of satellite-in situ/tag data integration via an enhanced Web-based visualization tool
Open Source release of software components for use by the broader earth data informatics community upon the conclusion of the project

Technology Enhancement & Integration

Extensions to the software technology stack leveraged by the OIIP project are summarized below. Additional tasks will include the integration of these in a test-DMAS environment to demonstrate data processing workflows end-to-end. It is important to note that while electronic tagging datasets serve as a thematic focus for OIIP, all capabilities will also be tested against more "conventional" in-situ data holdings available at the PO.DAAC, such as the SPURS field campaign datasets.

Component	Description/Extensions
NCEI/NODC .nc templates	NCEI/NODC netCDF "spatial feature class" templates (8) are an implementation of CF/ACCD standards for the representation of discrete sampling geometry (point, profile, trajectory, time series etc.) data within the netCDF self-describing file format. OIIP will review the adequacy of the templates for a range of tagging datasets while additionally provide implementation recommendations regarding representation of Domain specific metadata, Geolocation uncertainty, Data summaries.
ROSETTA	Unidata's ROSETTA tool is a generalized web-based capability for conversion of unstructured columnar ASCII data files to CF/ACDD compliant netCDF files. It's GUI wizard provides a guided, step-wise process for conversion and augmentation of file metadata by user. OIIP will: extend ROSETTA to support the NODC .nc templates and community specific metadata framework, and implement a RESTful web service automation interface. Integration of ROSETTA with PO.DAAC DMAS will also be pursued.
THREDDS	Unidata's "Thematic Real-time Environmental Distributed Data Services" (THREDDS) Web server technology provides metadata and data access for scientific datasets using a variety of remote HTTP-based data access protocols. Extensions undertaken as part of OIIP will include support for point, profile, trajectory spatial data types, plus integration of the nc-SOS (OGC Sensor Observation Service standard) plug-in.
CMC Web Visualization	JPL's Common Mapping Client (CMC) is a software framework and reusable set of components for efficiently developing and deploying Web-based mapping applications for the exposure of earth science data. During OIIP, CMC capabilities will be extended to access, integrate and display in-situ and satellite-derived datasets, providing synchronized horizontal and vertical profile views of data and their evolution over time.
DMAS	The Data management & Archive System (DMAS) is an integrated set of software components that comprise JPL PO.DAAC's data archival and distribution architecture. Work on DMAS undertaken as part of OIIP will focus on extensions to the DMAS internal metadata model for comprehensive CF/ACDD/NODC support, data handler extensions, Rosetta RESTful-Web Service development and Rabbit-MQ middleware enhancement.
Tagbase	Tagbase is a relational database system for the management of (non-acoustic) electronic tagging datasets developed to assist smaller research groups to manage their data collections. While being simple to use, it provides a comprehensive data model and multi-tag, multi-manufacturer support. During OIIP, Tagbase will be ported to the PostgreSQL open source RDMS, and its stored query procedures will support the CMC-Visualization tool on the backend.

Approach

The approach to project implementation is best encapsulated as "Community, Data Technologies, Standards, Team". OIIP places particular emphasis on community engagement and community driven approaches. We see this as critical to robust use case and requirements development. Collaboration and coordination with stakeholders is deemed essential for buy-in on technology and standards development aspects. Stakeholders with whom we are in communication include NCEI/NODC, IOOS-ATN, SPURS, PODAAC, ESDWG Data Interoperability Working Group, CF2. Additionally, OIIP has partnered and is working closely with a leading e-tag instrument manufacturer (Wildlife Computers) with the aim of ultimately implementing interoperable file/metadata standards recommendations sustainably at source.

Documentation

Software

Code developed under the OIIP project is available Open Source via the links below to their respective repositories on Github:

OIIP Data Viewer: https://github.com/oiip/oiip-data-viewer
Tagbase-Server: https://github.com/tagbase/tagbase-server
ROSETTA: https://github.com/Unidata/rosetta
THREDDS: https://github.com/Unidata/thredds

The OIIP project encourages further active community development of this software. Any comments/questions can be addressed to oiip@jpl.nasa.gov

Media

OIIP Visualization Tool Feature Demo

OIIP Visualization Tool Demo with Saildrone Baja Campaign Data

OIIP end-to-end Oceanographic in situ Data Workflow
Demo

Overview of the Electronic Tag Universal File Format
(eTUFF)

OIIP Tagbase Overview & Deployment Tutorial

"ROSETTA": Web-based netCDF Data Conversion Tool
from UCAR/Unidata

OIIP Rosetta Automated Batch Conversion Workflow

OIIP Rosetta eTuff Workflow

Presentations

"Leveraging Available Technologies for Improved Interoperability and Visualization of Remote Sensing and In situ Oceanographic data at the PO.DAAC". Presentations at the IATTC and NOAA/SWFSC, 2016, San Diego, CA, 26-27 October
"Leveraging Available Technologies for Improved Interoperability and Visualization of Remote Sensing and In situ Oceanographic data at the PO.DAAC". Poster presented at the AGU 2016 Conference, session IN51C-1862, San Francisco, CA, 12-16 December, 2017
"Leveraging Available Technologies for Improved Interoperability and Visualization of Remote Sensing and In situ Oceanographic data at the PO.DAAC". Poster presented at the NASA ESDSWG 2017 Meeting, Annapolis, MD, 21-23 March, 2017
"OIIP: The Oceanographic In-situ data Interoperability Project". Presented at the PO.DAAC Annual User Working Group meeting, Pasadena, CA, 12-13 April, 2017
"OIIP: The Oceanographic In-situ data Interoperability Project". Presented at the 68^th International Tuna Conference, Lake Arrowhead, CA, 15-18 May, 2017
"OIIP: Oceanographic In Situ Data Interoperability Project". Presented at the IOOS-ATN Workshop Meeting, MBARI, Monterey, CA, 10 August, 2017
"Extensions to Better Support the Needs of In-Situ and Observational Communities". Presented at the CF2.0 netCDF Earthcube Workshop, UCAR, Boulder, CO, 6-8 September, 2017
"The Oceanographic In Situ Data Interoperability Project (OIIP): A Year in Review". Presented at the IEEE Oceans 17 MTS Conference, Anchorage, AK, 18-22 September, 2017
"Earth Science Data Standards Considerations to Better Support the Needs of in situ Data and Observational Communities". Presented to the NASA ESDSWG Data Interoperability Working Group, 15 November, 2017.
"Integrated Visualization of Multi-sensor Ocean Data across the Web". Presented at the 2017 AGU Fall Meeting, New Orleans, LA, 11-15 December, 2017.
"Tuna and Data Standards: The Use of Rosetta in the Oceanographic In Situ Data Interoperability Project (OIIP)". Presented at the 98th Annual AMS Meeting, Austin, TX, 7-11 January, 2018.
"Addressing the Challenge of Electronic Tagging Data Interoperability: The Oceanographic In situ data Interoperability project (OIIP)". Presented at AGU/ASLO Ocean Sciences 2018, Portland, OR, February 2018.
"Enabling Interactive Synthesized Visualization, Exploration, and Access of In Situ and Satellite Ocean Datasets". Presented at AGU/ASLO Ocean Sciences 2018, Portland, OR, February 2018.
"Status of the Oceanographic in situ Data Interoperability Project (OIIP)" presented at PODAAC UWG meeting (5/16/2018, Pasadena, CA) with also a live demo of our CMC Web-visualization tool.
"Status of the Oceanographic in situ Data Interoperability Project (OIIP)" presented and OIIP workshop held at the 69th International Tuna Conference (Lake Arrowhead, CA, 23 May, 2018).
"Introducing the Oceanographic In-situ data Interoperability Project (OIIP)" presented at the Open Geospatial Consortium (OGC) TC/TP Meeting, ESS-DWG session , Fort Collins, CO, 4 June, 2018).
"The Oceanographic In-situ data Interoperability Project (OIIP)" presented in IBLS (International Bio-logging Society) Data Standardization Working Group Meeting teleconference, 12 June, 2018.
"OIIP Workshop: Web-based Tools and Data Standards for Electronic Tagging and In situ Datasets" presented at ESIP-summer meeting, Tucson AZ, 17 July, 2018.

Contacts

Inquiries on the OIIP project can be addressed to: oiip@jpl.nasa.gov

Follow @OIIP16

Acknowldegements

The project is funded under NASA Grant ACCESS15-017. We acknowledge the support of NASA EOSDIS Advancing Collaborative Connections for Earth System Science (ACCESS) Program.