November 29, 2014
Hot Topics:

Python 3.2 Concurrent Programming Features, Page 2

  • April 13, 2011
  • By Edmon Begoli
  • Send Email »
  • More Articles »

Mapped Submission in Python 3.2

In addition to submitting a function with parameters to an executor, we can alternatively "map" a function onto a list of arguments (more precisely, map a function onto an iterable list of arguments).

In the example below, I use the executor to asynchronously apply the function search_file to each of the file names provided by iterable files (a sequence of file name strings) and get the future result as the outcome of each map application within an iteration of the for loop.

def mapped_search():

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:

for result in executor.map( search_file, files ):

print( result )

Waiting for All Futures to Finish

Finally, let's look at how the wait function, available as the module level function, can be used to process results of all completed threads at the common synchronization point.

In the function parallel_search shown below, I submit a list of files into the executor to get a list of futures. I then wait for all futures to complete using the function wait.

def parallel_search():

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:

futures = [ executor.submit( file_search, search_target, "Synergy" )

for search_target in files]



results = concurrent.futures.wait( futures )



for completed in results.done:

print( completed.result() )

Function wait by default indefinitely waits for all futures to complete, but it can alternatively accept the Timeout parameter. The parameter represents the number of seconds and specifies how long to wait before throwing an exception. It also can accept the return_when parameter, which specifies the criteria for waiting. The parameter can be set to one of the following three conditions: ALL_COMPLETED (set by default), FIRST_COMPLETED and FIRST_EXCEPTION.

The function wait returns a named tuple consisting of done and not_done sets of futures.

Done contains all the futures that have completed by the time wait has returned. These two sets are most pertinent with FIRST_COMPLETED or FIRST_EXCEPTION option set.

Processing Futures Upon Their Completion

Use the as_completed method if your concurrent programming task demands processing of individual futures as they complete their execution and in order of completion. The function as_completed accepts a list of futures and returns iterable which returns futures iteratively as each of the futures completes and in order of completion. as_completed accepts the optional Timeout parameter as well.

for future in concurrent.futures.as_completed( search_futures ):

process_results( future )

Other Python 3.2 Features

This article focused on threads, but the concurrent.futures package also offers ProcessPoolExecutor objects for instantiation and execution of processes instead of threads. The pattern of use is the same as with threads, except for the different executor object use and a requirement that all submitted tasks must be picklable.

In the terminology of the operating system, the term "process" refers to an operating system level task that has its own stack and heap space and that is managed by the operating system scheduler. "Threads," however, can be purely within-process tasks that may or may not be visible to the operating system at all. Visibility of the threads depends on the implementation of the operating system, the compiler, the threading library used and the virtual machine that runs the interpreted program -- if applicable. System resources wise, tasks are more intensive and more expensive to manage and instantiate.

Futures also capture exceptions, and they can be checked for using the method exception() of the future object. The wait() and as_completed() functions can raise TimeoutErrors when Timeout is specified, so these need to be checked as well.

About the Author

Edmon Begoli is a software architect with 14 years of professional experience on large commercial and public software projects. He is a member of the R&D staff at Oak Ridge National Laboratory.


Originally published on http://www.developer.com.

Page 2 of 2



Comment and Contribute

 


(Maximum characters: 1200). You have characters left.

 

 


Enterprise Development Update

Don't miss an article. Subscribe to our newsletter below.

Sitemap | Contact Us

Rocket Fuel