Category: Python

PySpark Serializers

, , , ,

One of the habits I’ve formed using PySpark is to have a look at the source code, not just to get a better feel for what’s changed in a new release but also to supplement / supplant the documentation. For instance, if you peruse the docs for serialization, you might be left with the impression that your choices are either fast but limited datatype support or always-works but slow, when there’s actually quite a few serializers available. In fact you can get the best of both worlds if you use the AutoSerializer, which defaults to MarshalSerializer if supported for faster operations but falls back to PickleSerializer when required. I’ve had good results combining AutoSerializer with CompressedSerializer, which compresses / decompresses the data on the fly:


import pyspark
from pyspark.serializers import CompressedSerializer, AutoSerializer
sc = pyspark.SparkContext(conf=config, serializer=CompressedSerializer(AutoSerializer())

As always with [un-/under-]documented API features you should be cognizant of the fact that they could disappear without any warning, but that said these serializers are used elsewhere in PySpark so you’re probably good…for a while anyway. 🙂

Spark Streaming concurrentJobs: Necessary But Not Sufficient

, ,

Just a future reminder for myself and anyone else mildly surprised by this configuration option, if you want concurrent jobs in Spark Streaming you need to set concurrentJobs to a value of more than 1 as expected e.g.

conf.set("spark.streaming.concurrentJobs", "2")

But you also need to switch to a “fair” scheduler:

conf.set("spark.scheduler.mode", "FAIR")

Otherwise you’re using the default FIFO scheduler, so you’ll have multiple sequential jobs. Seems like naming the option concurrentJobs is a little misleading in that regard.

Serializing NumPy and Pandas in PySpark

, ,

I’ve been working on an application with PySpark and Kafka lately, and I’ve run into issues trying to push NumPy and Pandas objects around in Spark.  PySpark uses Py4J to connect Python to the underlying JVM, and Py4J doesn’t serialize NumPy arrays, data types, etc. for performance reasons (IIRC). There’s probably a better way to handle this but a quick solution is to just convert the “un-serializable” objects into their vanilla Python equivalents, which is what to_primitive does.

Just call to_primitive(some_unserializable_object) before you start tossing it around in Spark.

Calling Python From Myriad

, , , , ,

Myriad might be a Java toolkit but that doesn’t mean you can’t use MATLAB, Python, R, or anything else with it for that matter.  The NASA Snapshot has support for redirecting standard input and output so anything that can print to the screen and read input will work.  The API entry for ExternalROIFinder has all the details but basically all you need to do is have your code print myriad: True when it finds a Region Of Interest and the library takes care of the rest.

Here’s a quick video demonstration – I trained a model in scikit-learn to detect damage in ultrasonic data and wrote a simple Python app to handle input and output.

This was done in an early build of Myriad Desktop – more recent versions include support for configuring a Python or MATLAB interpreter instead of making you browse to the binary yourself.

If your app takes a while to get started, you can either structure it as a long running app to amortize the startup cost or you can simply create many more Myriad workers to run  it.  That way even if each individual call to your code takes a while to complete, the rest of the data processing pipeline isn’t being held up.

Thanks, WIMS-AECL!

, , ,

Working with some Log ASCII standard (LAS) data the other day made me realize that I owe a belated thank-you to Canadian Nuclear Laboratories (CNL) for allowing me to write the WIMS-AECL (Winfrith Improved Multigroup Scheme Atomic Energy Of Canada Limited *phew*) post processor “WIMSpp”.  It might not ever find widespread use outside CNL (or inside CNL for that matter), but I found it to be worthwhile for a couple of reasons.

  1. A couple of months of head down, solve a problem with Python work is always appreciated.
  2. After the NDIToolbox project had wrapped up, I’d had a couple of ideas on how to handle plugins in a self-contained (no Python install) Python app for Windows users and WIMSpp gave me the opportunity to test them out.
  3. Learning how to parse WIMS-AECL unstructured data (randomly word-wrapping data tables for the win!) means parsing LAS was easy.  🙂

OpenCV3 & Anaconda3 in Arch

,

An addendum to the excellent writeup over at River City Labs about getting OpenCV3 up and running under Anaconda 3 – if you’re using Arch or an Arch derivative like Manjaro, you might also need to install Pango if you get ImportError messages along the lines of

/usr/lib/libpangoft2-1.0.so.0: undefined symbol: FcWeightToOpenType

 

Here’s the sequence of conda directives that worked for me:

conda install -c https://conda.binstar.org/menpo opencv3
conda install -c asmeurer pango

Seems that Arch’s fontconfig package is also too recent for Anaconda.

Quandl For Python

, ,

I’ve been playing with Quandl’s Python API for a weekend or two, and I have to say it’s been great.  You know when you hear the word “API” and you mentally picture a few days of wading through Javadocs and wiki diving?  This isn’t that.  It takes all of a second to get something going.

import numpy as np    
import matplotlib.pyplot as plt
plt.style.use('ggplot')
import Quandl
import pandas as pd
auth = 'SuperSecretKey'

ufo_sightings = Quandl.get("NUFORC/SIGHTINGS", 
                           trim_start="1945-01-01", 
                           authtoken=auth)
meals = Quandl.get("USDATA/SCHOOL_BREAKFASTS", 
                    authtoken=auth)
meals = meals.dropna()
ufo_sightings['year'] = ufo_sightings.index.year
ufo_sightings['month'] = ufo_sightings.index.month
ufo_sightings_subset = ufo_sightings[np.logical_and(ufo_sightings['year'].isin(meals.index.year), 
                                                    ufo_sightings['month'] == 1)]

fig, ax1 = plt.subplots()
ax2 = ax1.twinx()
fig.set_size_inches(16, 9, forward=True)
line1=ax1.bar(ufo_sightings_subset['year'], 
              ufo_sightings_subset['Sightings'], 
              label='UFO Sightings')
line2=ax2.plot(meals.index.year, 
               meals['Free'], 'o-.', 
               label='Free School Breakfasts')
ax2.set_ylabel('Breakfast Expenditure $\\times 10^6$ USD')
ax1.set_ylabel('Number of Sightings')
plt.legend([line1, line2[0]], 
           [line1.get_label(), line2[0].get_label()], 
           loc='upper left')
plt.xlim([1970, 2012])
plt.title('UFO Sightings vs. US Free Breakfast Program, {0}-{1}'.format(min(meals.index.year), 
                                                                        max(meals.index.year)))
ax1.set_xlabel('Year')
plt.show()
Please don't take this seriously
COINCIDENCE?!?!  Yes.  Yes it is.  Please don’t take this seriously.

Anyway, if you’re looking for an easy way to get data it’s worth a look.

On People Scraping

, ,

Having written one or ten people scrapers this week, I’ve come to realize a few things.  Posted here so I have a reference for the next time it comes up.

  1. Requests + BeautifulSoup is a great combo.
  2. You’re still going to want to keep re handy.
  3. You’ll want Unidecode too.
  4. You’re probably thinking a straight dict for a person, but go with defaultdict.  Use an empty string for the default e.g. person = defaultdict(lambda: '""').  Makes it easy to normalize across sites.
  5. Man there’s a lot of broken HTML out there.

Machine Learning Weather Forecasting

, , , ,

Real vs. Model Mean Temperature
A plot comparing real mean temperature vs. an early model’s predictions.

Over the holidays I was a little peeved at how inaccurate the weather forecast seemed to be (I swear the uncertainty was +/- 10 degrees C), and that got me wondering about how machine learning would fare.  The basic model came together pretty quickly and today I’ve got a bare-bones demo up on PythonAnywhere for a few locales.  The plot on the right compares an early (i.e. pre-tweaking) model’s predictions for mean temperature vs. actual temperature readings for a weather station in Petawawa Ontario.  I’ve tweaked the model’s settings a bit since this plot so hopefully I’ve squeezed a bit more accuracy out of it, but the standard disclaimers apply:  work in progress, don’t plan outdoor weddings or tuna fishing expeditions around its forecast, etc.

Reading NCDC Weather Data

, , ,

The current version of the weather station data from the National Climatic Data Center is fairly straightforward to read with Python and pandas, but the “legacy” file format as used on the old NNDC site and elsewhere is a bit more work.  So in case it’s useful here’s a bit of code that I’ve had some luck with; just call the read_ncdc function to get a slightly cleaned up pandas DataFrame for your number-crunching pleasure.

Load More