Tag Archive: python

Simplified Python Multithreading

I’ve been bashing my head against this topic for months now on some level.  I’ve looked around the web at people’s horrible explanations of how to use the threading module in Python and now that I figured it out for myself, let me unleash my horrible explanation on the world.  🙂

In order to properly use threading, first off, you need a task that can be passed off to a thread and done in the background in parallel with other tasks.  For example, crawling a website.  A crawler could grab a page, parse all of the links for that page and then pass all of the links off to process threads to speed up the process.

My problem before was that I was thinking about threading all wrong.  I was thinking that you launch the threads via the process you are trying to accomplish so in other words something like:

def thread(url):
    content = crawlPage(url)
    return content

^^^ BAD!!!  That doesn’t work.

The proper way to think about threads is that you are creating empty background while loops.  Those loops continue running and picking things out of your queue to process.  If there is nothing in the queue, they’ll just indefinitely loop until something shows up.  The code I’m going to show you below has been distilled down to what *I* think you need for a good thread implementation.  It has several great features such as monitoring, throttling(upwards only) and a queue.

There is some fluff in this code for purpose of demonstration that could be pulled out when it comes time for implementation but I would suggest you download/retype this code, run it and tweak it a bit to see what happens with it.  I pretty much promise you that you’ll get a better feel for how threads work after messing with this code for a bit.


import time 
import Queue 
import threading 
import random

# create the queue that will feed the background threads
q = Queue.Queue() 

# this starts and individual thread that continuously 
# pull jobs out of the queue
def backgroundThread():
# this while loop keeps the thread running continuously
    while True: 
        # get the job out of the queue
        job = q.get() 
        # do the job
        print "Job output: %s" % job
        # pause so the loop doesn't spike your cpu load
        # report back to the queue that the task is finished

# this launches the necessary number of background threads
def launchThreads(num_threads):
    for i in range(num_threads): 
        # create the thread
        t = threading.Thread(target=backgroundThread) 
        # setting daemon mode makes threads cease when
        # main thread is terminated
        # start the thread
        print "Now launching %s" % t.name
# launch 2 background threads

# Main loop
while True:
    #this spins up more workers if queue gets too big
    if q.qsize() > 10:

    # this will be results of another process or a list/file iteration
    job = random.uniform(1,300)

    # push the job into the queue.

    # this slows down the queue filling up for demonstration

    # shows the size of the queue
    print "%s jobs in queue" % q.qsize()

    # this prints a list of the threads currently running
    for thread in threading.enumerate():
        print thread

If downloading the code works better for you, here’s a link to it.  threading-demo.py

The O’Reilly School of Technology recently started offering a 4-part Python certificate course.  I have just recently completed the first part of the course and wanted to share some of my impressions.  The course is ‘cloud-based’ and served to you over an RDP connection.  They provide you links to Mac OSX and Windows versions of the RDP client so either way you are covered.  When you log in, you are dumped into the Eclipse IDE environment and walked through how to navigate the environment and set it up for your use.  Love or hate Eclipse, you will be forced to use it through this course.  I personally did not like Eclipse too much before I signed up for this course.  Now that I’ve been using it a lot more I have decided it’s not so bad and I could probably grow to like it at some point.  The further into the course you get, the more you realize why they made the decision to use Eclipse.  Largely because it allows them to embed a web browser and they can run their other courses for Java, etc all off of Eclipse.  Anyhow, enough about Eclipse…

My first impression of the course was not really so good.  I kinda felt like I was just reading an abridged version of a $50 book.  But after completing a few lessons I can see the value in essentially being forced to apply what you learn in each lesson.  Is it worth the $300-$400($300 if you happen to catch it on sale)?  Well that depends on your specific needs.  If you want to go for the full certificate, you will spend $1200-$1600 because there are 4 separate courses you need to complete in order to obtain the cert.  On top of that, there is a $15 monthly lab fee for using the system where the course is hosted.  $1600 MIGHT be cheaper than your local university or perhaps your local university doesn’t even offer Python in the first place.  I know some folks that were recently enrolled in a prominent local university’s Python course and they ended up feeling it was a complete waste of their time.  Apparently the teacher kind of sucks.  That’s good money down the drain unfortunately…  Fortunately their employer is paying that bill.  The OST course is nice because you can just try one module of it and if you don’t like it, don’t go any further.  There is also a 7 day money back guarantee.  Keep in mind that if you have completed more than 40% of the course in that first 7 days, you will no longer be eligible for the money back guarantee.

For me, I’m in it for the knowledge.  I want to learn to be a better programmer and I think Python is a great language to help with that goal since it’s so deep and versatile yet simple enough to understand the fundamentals with a little bit of effort.  If your company will pay for you to obtain a Python cert with O’Reilly, I would say go for it.  No question.  If it’s your own money you are spending, I would suggest trying to go through a book such as Learning Python by Mark Lutz.  That is what I started with and I’m glad I did.  It allowed me to fly through the O’Reilly course and get more value for my money since the course was reinforcing and clarifying concepts that I had already come across in the book.  I wouldn’t mind having the Python certificate and learning the rest of the knowledge the course has to offer but I’m not so sure I want to spend another $900-$1200 to get to that point right now.

As far as specifics of the course, I felt that it started out with a slightly higher than beginner level.  If you’ve never programmed before and never even tried to play around with Python, you might get lost in place.  There is an instructor available via email but there is a lag if you are working on the course after business hours.  The course makes a couple assumptions about prerequisite knowledge.  My suggestion would be to at least skim a Python book even if you don’t punch in code out of the book as you read it.

I spent several weeks deciding whether I wanted to do this Python course.  I discussed it over with a few friends and what finally convinced me to just give it a shot was the words of one person in specific…. “Well, it can’t hurt any now can it?  It won’t make you any worse at Python, right?”  True enough.  In the end I definitely feel like I have a much deeper understanding of Python and I can now converse with other Pythonistas in a semi-coherent manner.  So for me, yes, it was worth the $300.  As an added bonus, you get a free ebook of one of their beginner Python books.  For me it was “Head First Python by Paul Barry” which incidentally is a fairly new release for them.  On top of that, I got a discount code for 40% off of any of their printed books.  That sweetens the deal if you are in the market for any of the O’Reilly books.

Bottom line is that I’m happy with my purchase and would do it again given the choice.  If you have completed this course yourself or are thinking about taking it, please feel free to post a comment down below.  I would be interested in some additional perspective.

Now I wish they could figure out how to do one for developing iPhone apps 😉

I’ve been working on learning Python lately and there is a boatload of good information out there that is scattered all over the Internet.  Luckily in the spirit of open source software, most of the information is free.  Some of it can be had in more convenient packaging for a low price alternatively.  Personally, my primary focus is Python 3.0 since that is the future and I don’t expect to be working on any legacy Python code before I learn Python much, much better.  By the time I get to that point of confidence I’m sure I’ll be familiar with the nuances of Python 2.6-2.7 though.  Python 2.7 is the last major release of the Python 2.0 series as of this writing.  The developers intend to do bug fixes for 2-5 more years but there will be no new features added.  If you insist on using 2.7 or 3.x is not available, you are not totally S.O.L. though.  There is a library called “future” that has many features from the newer series so you’ll be able to use them in your 2.x code.  Anyways, on with some of the resources…


I’m heavy into podcasts since it’s something constructive I can listen to while cooking dinner, driving, etc.  There happen to be three great Python podcasts out there that I know of(if you know of any other current ones, let me know). Most of the resources that I am going to share with you below were parsed by me from listening to these podcasts.

Python 411 – The first one I checked out was Python 411.  This one is a one-man show but he does a great job keeping it interesting by interviewing developers and other folks that use Python in their day to day jobs.  He branches off into scientific and sometimes philosophical realms but it’s all going to be interesting too the average nerd who is interested in learning Python anyhow.

From Python Import Podcast – This was the second Python podcast I discovered.  It’s relatively new to the scene and features 3 slightly jaded developers who are obviously good friends aside from coworkers.  They tend to rant on some higher level topics but I think it’s valuable to listen to anyhow if you are interested in taking Python further than just a hobby.

A Little Bit of Python – This podcast features several core pillars of the Python community and they discuss all kinds of topics from cons to newbie items.  There are a few good interviews as well with some other large names in the Python community.  If Guido were to come onto a podcast, this would probably be the one.  As a bonus, this podcast has a bit of European flair as well.

Reading Material:

Unfortunately, more than just podcasts will be required to learn Python. There are PLENTY of great books out there but most of them are still for Python 2.x.  Here are a few resources that I have managed to hunt down for Python 3.

Python Docs – The obvious place to go is the official Python docs and if you look in there, you will see the official Python Tutorial.  If you have an iPod Touch, iPhone or iPad, you can also purchase(from the app store for $.99) an offline version of the Python 2.6 docs.  Even though it’s for 2.6, it’s still worth having even if you are focused on Python 3.x.

Learning Python by Mark Lutz – I have seen good and bad reviews of this book but I personally think it’s a great place to start.  The bad is that there is a lot of repetition and he teaches you the wrong way to do things before teaching you the correct way but on the converse, it’s good to know that there are so many ways to do things in Python since it can spark your creativity if you are not a veteran developer.  If you are very experienced in some other language, perhaps you’ll find this book too slow.  For me, it’s been mostly a good read.  SOME of the examples in the book are the most clear cut ones I can find on there topics anywhere.  If you have an iOS device, you can purchase a standalone e-version of the book for under $5.

Instructional videos:

If you learn better by watching it done, this might be just the ticket.  There are several excellent Python specific video sites out there so it’s worth a bit of your time to check out some of the videos.

Showmedo – This site was started by a guy that was trying to put together some distance learning materials for a company.  The videos are all provided by individuals and are all rated and monitored for quality.  This site doesn’t have EVERYTHING that I’ve looked for but there is a lot of good info here.  Be aware that SOME(not many) of the videos cost money to watch.

Python Miro Community – This site has video archives of several pycons on top of several other categorized videos covering other Python-related subjects.  I just found this site as of today so I have not had time to make a quality assessment yet but the interface looks great and the some of the categories look quite promising.

Interactive Learning:

If you learn better by doing, this is something you should check out.

Python Koans – The Python Koans are based on the Ruby Koans.  What’s a Koan?  According to the wiki, “It consists of a story, dialogue, question, or statement, the meaning of which cannot be understood by rational thinking but may be accessible through intuition“.  That describes the theme of the koans but in practice, it’s a series of tests build upon each other.  So when you run the Koan, it will fail.  You will then need to jump into the code and figure out what is wrong with it to make it pass the test.  There are MANY MANY tests here.  Even if you are a cracker jack Python wiz, it would probably still take you 2-3 hours to plow through the Koans.  They are available for Python 2 and Python 3.  When you download the package, you get both.  The Python Koans are probably the quickest way to jump into coding some Python.  The tests start out at a very basic, beginner level and if you pay attention to HOW the tests fail, you will see clues to help you succeed.

O’Reilly 3-part Python certificate course – This is a 4-part course(at $400 per part) that is offered remotely and somewhat interactively by the O’Reilly School of Technology.  When you complete the third part of the course, you will receive a certification in Python from the University of Illinois.  I cannot vouch for this course since I have not taken it but I am certainly considering it.  The price is a little steep but still MUCH cheaper than the one offered locally to me by the University of Washington.  The course is brand new so it focuses on Python 3.  I looked around online for some reviews and the O’Reilly School has had some mixed reviews.  Most of them stated the the material was outdated(won’t be a problem here) or that the course was not challenging enough(ymmv).  Whatever the case, at the end, you will receive a certification for what it’s worth.

Python Meetups – There are 95 groups in 60 cities in 12 countries.  Surely you are near one of these.  I have not gone to one yet but I do look forward to it.  There are 3 groups in the Seattle area alone!

Python in 5 minutes

I will start out by saying this title is a little misleading but I’ve been trying to learn some Python lately and I have figured out some things that will help you get well on your way to learning Python in about 5 minutes. I won’t say that I know Python well at this point but what I DO know is how to figure a lot of things out without cracking open the manual. For starters, get Python installed. I’ll leave that part up to you. After that, you’ll want to open the Python interpreter by typing python at the command line in your particular operating system. You can’t save your code while you are in the interpreter but you can safely play around in your own personal sandbox.

First lesson – Strings: At the prompt, type in something like:

>>> s=’Spam!’

That will put a value into memory that you can play with.

Second lesson – dir() & help(): Not only is “s” a string, it’s also an object.  This lends itself to all kinds of neat things that I will let you figure out on your own but one of the coolest features is that you can tack string attributes on it very easily.  How do you know which attributes will work?  This is where dir() comes in.  At the interpreter prompt, type:

>>> dir(s)

This will show you a huge list of attributes that can perform operations on your string.  For instance, you can use:

>>> s.swapcase()

This will print something like ‘sPAM!’ at on your screen.  If you want to change the variable, you can use:

>>> s=s.swapcase()

Now what if you see another attribute in the list who’s function isn’t self explanatory?  This is where help() comes in.  Say that you don’t know what the “zfill” attribute of a string means…  Type:

>>> help(s.zfill)

You’ll get the exact explanation of what the attribute does right at your prompt.  You can also simply do a help(s) but you’ll get pages of help that will scroll off your screen.

There is a LOT more to python than this of course but these couple of bits of knowledge will help you learn a lot of python’s tricks just by playing around with code at the interpreter prompt.  Of course you can’t go wrong reading the python documentation either.  It’s pretty good and thorough.  I wouldn’t suggest buying a book until you give that a shot.  If you do still need a book, I like Learning Python: Powerful Object-Oriented Programming.

Powered by WordPress. Theme: Motion by 85ideas.