Detecting (underwater) damage in ports is no easy feat. By using specialized point cloud scanners, trained professionals manually compare underwater snapshots through time and detect damage or other irregularities through careful inspection. This inspection of collected point clouds is very labor-intensive; that’s why at In The Pocket, we’ve devised a scalable pipeline using machine learning to automatically detect and signal irregularities in underwater point clouds.

Point cloud scanners gather a large amount of three-dimensional points to represent the environment they capture. All points in their simplest form basically consist of three numbers (the x, y and z coordinates) that together form a mathematical representation of the surfaces of the captured environment. These points can intuitively be used to visualize the captured environment, just like this point cloud of the Colosseum:

Point cloud technology is used in a variety of sectors besides traditional 3D engineering. Some examples are the manufacturing industry, mining industry, urban planning sector, construction and the gas & oil industry to monitor large-scale assets such as plants, properties and structures. The point clouds are used to inspect and determine the condition of large systems, which also make them a great asset for predictive maintenance and safety assessment.

In our case, point cloud capturing technology is used to take snapshots of the underwater environment of ports at different moments in time to subsequently use these for visual inspection by humans. If irregularities are found, divers are sent to investigate the situation. Since it is a time-consuming task to compare point clouds, comparisons for the same place only happen at timescales within an order of months or years. But what do you do when you want to catch anomalies much sooner, for example within the week? This is where our automatic irregularity detection pipeline comes into play. It can compare all available point clouds of all places at the same time, with (much) faster speeds than humans. This way it is possible to assess if an irregularity has occurred very soon after a new point cloud is available, without having it sitting idly in a queue waiting for human inspection. With a higher check-up frequency, the quality of day-to-day port operations improves noticeably.

How do we do this? At the core of our pipeline is the battle-hardened classic machine learning algorithm, called nearest neighbors search. Put very simply, this algorithm compares every point in the new point cloud with every point in the old point cloud, and calculates the distance to the nearest (old) point as an indication of irregularity. By using things like smart indexing and GPU-parallelization, this computationally time-complex process is made more efficient to adapt to the large point clouds that occur in real-life situations. Different point cloud comparisons can be run in parallel on the cloud, effectively removing the waiting queue for processing. A visual example of our algorithm finding three (artificial) irregularities looks like this:

Besides a visual presentation of the anomalies, the pipeline should provide a suggestion at which point the irregularity is actually occurring. One possibility is to find the points where the nearest-neighbor distance reaches a local maximum (using first and second spatial derivatives on the distance function), and suggesting these points as representations of the irregularities.

Visual representation of a fictitious distance function, where the horizontal two axes represent the spatial coordinates (because humans cannot see in 4D, unfortunately), and the vertical axis represents the nearest-neighbor distance. At the top (red), the first spatial derivative is zero, and we use the second spatial derivatives to determine that the corresponding point is in fact a maximum (and not just part of, e.g., the null plane).

To avoid too many false positives caused by random small distance fluctuations (creating irrelevant local maxima), we voxelize both the old and new point cloud to the same voxel size, creating a more rigid grid of point clouds. Voxelizing is basically turning reality into Minecraft: our messy underwater point cloud with many redundant points gets transformed to a strictly rigid ‘blocky’ structure, where every point represents one volume cube with predefined size, called a voxel (volume + pixel). This effectively flattens the distance function through space, except where non-negligible irregularities that are larger than the voxel size occur, reducing the amount of false positives. This doesn’t mean that there won’t be any false positives left, but in this case it is better to have some false positives, than to miss an irregularity. A human operator during quality control can simply dismiss a falsely proposed irregularity every now and then based on his experience. This could for example be the same irregularity showing up twice in close proximity, or an irregularity that in context is nothing to worry about (e.g., a ship that was simply not there in the previous scan). The opposite case is worse: when an irregularity is not picked up, quality control will not easily look beyond the proposed irregularities, and the fault will not get addressed.

Adding this all up, we create a robust pipeline that decreases the amount of manual inspection labor, increases the inspection frequency, and overall innovates the way of working in port change detection. These techniques are not restricted to ports alone: other application domains using point clouds can benefit greatly from this pipeline as well. If you have a use case that would benefit from our algorithms, don’t hesitate to contact us!