Laravel Parallel Job Processing Tip.

Whilst recently working on porting a legacy client application to Laravel we needed to implement a pipeline of jobs to generate a temporary file, run several processes on the file and then remove the file afterwards. As per the Laravel docs, this is possible via Job Chaining which:

Job chaining allows you to specify a list of queued jobs that should be run in sequence after the primary job has executed successfully

In this use case we a Job class ProcessFileJob that dispatches a chain of Jobs that will be worked through in order:

final class ProcessFileJob implements ShouldQueue
{
    ...
    public function handle(): void
    {
        Bus::chain([
            new CreateTemporaryFileJob(),
            new DoSomethingToFileJob(),
            new DoSomethingElseToFileJob(),
            new CleanUpAfterWardsJob(),
        ])->dispatch();
    }
}

This is fine and works but it became apparent that DoSomethingToFileJob and DoSomethingElseToFileJob are not dependent on each other. Further as they are both long running tasks it would be nice if they could run in parallel so we can potentially clean up sooner! We can achieve this by delegating the processing of the parallel job chains into another Job with its own batch like this:

final class ProcessFileJob implements ShouldQueue
{
    ...
    public function handle(): void
    {
        Bus::chain([
            new CreateTemporaryFileJob(),
            new HandleParallelJobs(), // replace two blocking jobs with one parallel handler
            new CleanUpAfterWardsJob(),
        ])->dispatch();
    }
}
final class HandleParallelJobs implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable;
    public function handle(): void
    {
        $parallelJobChains = [
            [
                new DoSomethingToFileJob(),
            ],
            [
                new DoSomethingElseToFileJob(),
            ],
        ];
        Bus::batch($parallelJobChains)->dispatch();
    }
}

Moving the parallel process into another Job means we can verify this is behaving as expected through a discrete and succinct unit test like this:

test('job creates batch with parallel chains as expected', function () {
    Bus::fake([
        DoSomethingToFileJob::class,
        DoSomethingElseToFileJob::class,
    ]);
    HandleParallelJobs::dispatch();
    Bus::assertBatched(function (PendingBatch $batch) {
        // assert that we have two chains
        return $batch->jobs->count() === 2 
            // assert one chain for DoSomethingToFileJob
            && is_array($batch->jobs[0]) 
            && $batch->jobs[0][0] instanceof DoSomethingToFileJob
            // assert one chain for DoSomethingElseToFileJob
            && is_array($batch->jobs[1]) 
            && $batch->jobs[1][0] instanceof DoSomethingElseToFileJob;
    });
});

So now our Jobs can be process more efficiently without un-necessary blocking, nice! What patterns do you use to process complicated Job flows? What ways would you test they’re working as you’d expect? Let us know on Twitter or Linkedin!