chriscoughlin.com
stochastic arrangement of 1's and 0's
  • Main
  • About Chris
    • Resume
    • Publications And Presentations
  • Stuff To Get
    • NDIToolbox
    • PODBrowser
    • SkinDepth
    • Demos
    • Myriad

Recent Posts

  • Application of the Actor Model to Large Scale NDE Data Analysis
  • Smart Structures and NDE for Industry 4.0
  • Myriad On The Server

Categories

RSS Elsewhere…

  • Chris Coughlin pushed to project branch master at Chris Coughlin / MyriadTrainer
  • Chris Coughlin pushed to project branch master at Chris Coughlin / Myriad
  • Chris Coughlin pushed to project branch master at Chris Coughlin / Myriad
April 24 2013

NDIToolbox April 2013 Update

crc development, NDE, NDIToolbox, Python, science

Development on TRI‘s nondestructive evaluation data analysis software NDIToolbox has slowed of late as we’ve gotten closer to our goal for functionality and as we get ready to do an honest-to-goodness field test later this year on a QA line. Nevertheless I’m still plugging away at it whenever I get the chance, and today I’ve got the latest and greatest available with two new features:  support for multiple datasets in Winspect data files and a new “batch mode.”

The batch mode feature lets you run an NDIToolbox plugin on a set of input files, optionally spawning multiple processes to speed things up.  If you have a ton of data files and you’re doing the same number crunching over and over, just point NDIToolbox to the files and the plugin and let it do it for you.  You don’t have to convert your data files to HDF5 before using batch mode; as long as the file format(s) are supported by NDIToolbox it’ll fetch the data and run the plugin automatically.  More info on batch mode available here from my mirror of the NDIToolbox docs.  If you’re going to use batch mode’s multiprocessing, be sure to read up on the requirements (basically, don’t have really huge data files).

As usual, I’d recommend using the conventional Python version of NDIToolbox if you can.  If you’re on Windows and don’t want to install Python (or you want to run from a thumb drive), the Downloads section of NDIToolbox’s Bitbucket page has a Windows installer and a compiled version available, no Python required.

If you’re writing a plugin there’s one additional step required to support the new batch mode.  Since more than a few nondestructive testing system file formats like UTWin’s CSC or WinSpect’s SDT can have multiple datasets in a single file, batch mode will send your plugin a dict of all the datasets it finds in a given input file.  So you’ll need a bit of code to see if you’ve been passed a single dataset (conventional user interface) or a container full of datasets (batch mode). There’s a few ways to do this but one of the most straightforward is to look for a “keys” attribute like so.

if hasattr(self._data, "keys"):
    # Dict of data provided - batch mode
    for dataset in self._data:
        # Execute plugin on every dataset
        self._data[dataset] = your_analysis_function(self._data[dataset])
    # You could alternatively execute on one particular type of data
    # e.g.
    # if dataset == "waveform":
    #   self._data = your_analysis_function(self._data[dataset])
else:
    # A single dataset was provided
    self._data = your_analysis_function(self._data)

You could also just check to see if you were passed an actual dict, courtesy isinstance(). I’d recommend against doing that for now though – better to just assume it’s an associative container of some sort rather than hard-wiring an expectation of an actual dict.

Align The Laser NDIToolbox June 2013 Update

Related Posts

career, data, Java, Machine Learning, Myriad, NDE, news, science

Application of the Actor Model to Large Scale NDE Data Analysis

career, data, Java, Machine Learning, Myriad, NDE, news, science

Smart Structures and NDE for Industry 4.0

data, development, Machine Learning, Python, Spark

PySpark Serializers