Experience test-driving NDIToolbox in the field (or the depot / hangar to be more accurate) showed me that there is a ton of NDE sensor data out there and that it can take forever and a fortune to analyze manually. I’d experimented with algorithms to automatically flag possible indications of damage in the data when I was working on NDIToolbox and a project for the Air Force, but I’d never really gone beyond the proof of concept stage. Until recently I didn’t have a good handle on how to make it multi-processor and/or distributed, either – sitting in a depot for an hour waiting for a file to load has taught me single-threaded analysis isn’t feasible.
Enter Akka. I’ve written code in Spark and Storm, but Akka seems to impose fewer restrictions on development. Throw in some handmade pyramid, sliding window, and convolution implementations; add Apache Mahout for machine learning; and a Fault-Tolerant NDE Data Reduction Framework is born.
Two months or so in to the project and there’s a rather long demonstration of “Myriad” in action available, in which we train a machine learning model to automatically detect indications of damage in ultrasonic sensor data. I hope to have a few more demos of calling external apps or building a Myriad P2P cluster soon, stay tuned!