I’ve been bashing my head against this topic for months now on some level. I’ve looked around the web at people’s horrible explanations of how to use the threading module in Python and now that I figured it out for myself, let me unleash my horrible explanation on the world. 🙂
In order to properly use threading, first off, you need a task that can be passed off to a thread and done in the background in parallel with other tasks. For example, crawling a website. A crawler could grab a page, parse all of the links for that page and then pass all of the links off to process threads to speed up the process.
My problem before was that I was thinking about threading all wrong. I was thinking that you launch the threads via the process you are trying to accomplish so in other words something like:
def thread(url):content = crawlPage(url)return content
^^^ BAD!!! That doesn’t work.
The proper way to think about threads is that you are creating empty background while loops. Those loops continue running and picking things out of your queue to process. If there is nothing in the queue, they’ll just indefinitely loop until something shows up. The code I’m going to show you below has been distilled down to what *I* think you need for a good thread implementation. It has several great features such as monitoring, throttling(upwards only) and a queue.
There is some fluff in this code for purpose of demonstration that could be pulled out when it comes time for implementation but I would suggest you download/retype this code, run it and tweak it a bit to see what happens with it. I pretty much promise you that you’ll get a better feel for how threads work after messing with this code for a bit.
#!/usr/bin/python import time import Queue import threading import random # create the queue that will feed the background threads q = Queue.Queue() # this starts and individual thread that continuously # pull jobs out of the queue def backgroundThread(): # this while loop keeps the thread running continuously while True: # get the job out of the queue job = q.get() # do the job print "Job output: %s" % job # pause so the loop doesn't spike your cpu load time.sleep(5) # report back to the queue that the task is finished q.task_done() # this launches the necessary number of background threads def launchThreads(num_threads): for i in range(num_threads): # create the thread t = threading.Thread(target=backgroundThread) # setting daemon mode makes threads cease when # main thread is terminated t.setDaemon(True) # start the thread t.start() print "Now launching %s" % t.name # launch 2 background threads launchThreads(1) # Main loop while True: #this spins up more workers if queue gets too big if q.qsize() > 10: launchThreads(1) # this will be results of another process or a list/file iteration job = random.uniform(1,300) # push the job into the queue. q.put(job) # this slows down the queue filling up for demonstration time.sleep(random.uniform(0,1)) # shows the size of the queue print "%s jobs in queue" % q.qsize() # this prints a list of the threads currently running for thread in threading.enumerate(): print thread
If downloading the code works better for you, here’s a link to it. threading-demo.py
