Up-to-date and accurate maps of the waterways are a prerequisite for safe and efficient shipping in Germany. Authorities such as the Federal Maritime and Hydrographic Agency, but also private harbor operators, are obliged to provide these maps in an up-to-date form at all times. Primarily, the reason is to prevent accidents caused by incorrect or outdated nautical charts. This includes the classification of federal waterways; their length in kilometers; existing locks, lifts, and barrages; and other surrounding waters, places, and borders.

Currently, mapping is carried out by conventional ships with trained personnel, which is time-consuming and expensive. This can result in long intervals between measurements at a particular location.

Semi-automated monitoring of waterways by autonomous platforms with obstacle avoidance and traffic detection could optimize this process and provide maps in less time. To this end, the Fraunhofer Institute of Optronics, System Technologies, and Image Exploitation (IOSB) has developed an autonomous surface vehicle capable of surveying bodies of water as part of a three-year in-house research project. For this purpose, the riverbed is recorded using sonar, and the bank areas and surface structures are mapped using an optical system with two integrated industrial cameras from IDS Imaging Development Systems. The information obtained is then merged and transferred to a 3D map of the surroundings.

Application

Precisely surveying water bodies is a demanding task. TAPS, the Fraunhofer IOSB’s semi-automatic direction-finding system for rivers and lakes, is able to perform this task both under and above water.

The vehicle measures approximately 2 x 1.5 x 1 meters and travels along the relevant waterway while automatically avoiding obstacles, whether stationary or moving. At a speed of 2 knots—which corresponds to about 3.7 kilometers per hour—such a mission can last up to 20 hours. For visual mapping of the shore areas, TAPS has two cameras attached to a mast, each pointing to starboard or port with fields of view that don’t overlap. Due to their high resolution, they enable a visual inspection of relevant infrastructure, such as quay walls, as well as 3D modeling of the shore area based on the recorded image data.

3D model of a surface structure

“We use an automated system for intelligent image capture,” says Boitumelo Ruf, expert in photogrammetry in the autonomous robot systems research group at Fraunhofer IOSB. “As soon as one or both cameras are focused on a predefined area of interest, image recording is started. The vehicle’s own movement is also used to store only image data that is recorded from different angles and therefore offers added value in terms of content.”

GNSS (global navigation satellite systems, such as GPS) and IMU (inertial measurement unit for position determination) data serve as the basis for determining the position and orientation of the TAPS platform. “When the images are captured, they are enriched with the current GNSS position data. The latter are required later for the assignment of precise coordinates,” says Ruf.

The watercraft autonomously maneuvers around a leaking inflatable boat.

After data acquisition, the recorded images are transmitted together with the GNSS data to a ground control station, where a photogrammetric reconstruction is carried out.

Ruf describes the procedure in more detail: “Among other things, we use the COLMAP photogrammetric toolbox. It uses outstanding image features to first compare the input images, then calculate their relative positions and create a true-to-life 3D model of the environment. We then use the tool to match the images pixel by pixel, i.e., we search for corresponding pixels and merge them exactly. The result is a dense 3D point cloud that is georeferenced using the GNSS positions, i.e., provided with the corresponding current coordinates.” The 3D model can then be used for other tasks, such as visual inspections or bank monitoring.

Robust industrial cameras defy wind and weather

On the camera side, the Fraunhofer Institute relies on two uEye FA industrial cameras from IDS. The robust and resilient models with PoE are ideal for demanding environments. Camera housings, lens tubes, and the screwable connectors meet IP65/67 requirements and thus are optimally protected against dirt, dust, and splash water.

The model uses the IMX304 large-format 1.1 in. CMOS sensor from Sony and delivers very clear, noise-free images with a resolution of 4,096 x 3,000 pixels. Thanks to its remarkable dynamic range and high sensitivity, the global shutter sensor of the second pixel generation of the Pregius series is particularly suitable for metrology applications.

It was precisely these characteristics that were decisive for the Fraunhofer IOSB’s choice of camera. In addition to durability and weather protection, important requirements for the cameras included a compact form factor and high image resolution to enable detailed visual inspections. The sensor also offers a high dynamic range to capture detailed images in both sunny and shady areas.

Integration via the standard GigE Vision interface enables the development and use of one driver for several systems with different cameras.

TAPS has two uEye FA industrial cameras from IDS mounted on a mast to visually map the shoreline areas.

Outlook

A shortage of skilled labor is also a major issue in surveying waterways. Automated mapping of the waterways could not only reduce costs but also the time and, above all, the labor required. The aim of the TAPS project was to develop a prototype for this task and demonstrate its potential in relevant scenarios. As a research organization, Fraunhofer typically develops technologies up to a level at which the functionality and suitability of the system have been proven. The next step is now cooperation with industrial partners.

“We are actively looking for partners to take this final step toward commercialization together,” says Janko Petereit, head of the autonomous robot systems research group at Fraunhofer IOSB. “In view of the positive results of our initial tests, and the increasing demand for accurate, up-to-date waterway maps, we expect strong demand for our technology in the coming years. The knowledge gained can be used in a variety of ways. Autonomous watercraft can be used to transport people and goods by sea or inland waterways. Excavation of waterways and autonomous surveying of fairways are also conceivable.”

The project demonstrates the high potential of the interaction between robot systems and digital image processing. This promising technology could provide new ways of surveying shallow waters in the future, particularly with regard to visual environment detection and scene interpretation.

About Fraunhofer Institute

The Fraunhofer Institute of Optronics, System Technologies and Image Exploitation aims to efficiently support people in making informed decisions, optimizing processes, and intelligently controlling autonomous systems with its work in the three core competencies that give the institute its name. This involves developing new types of vision systems, including the necessary laser light sources, optimal use and networking of sensors, and processing the resulting data streams.