Just a future reminder for myself and anyone else mildly surprised by this configuration option, if you want concurrent jobs in Spark Streaming you need to set
concurrentJobs to a value of more than 1 as expected e.g.
But you also need to switch to a “fair” scheduler:
Otherwise you’re using the default FIFO scheduler, so you’ll have multiple sequential jobs. Seems like naming the option
concurrentJobs is a little misleading in that regard.