Upgrading & migrating pip packages, en masse

Upgrading, faster

Pip, the Python package management system, still lacks an easy way to update all installed packages. The “upgrade-all” ability has been in the works for nearly 4 years now.

In the meantime, many simple hacks have evolved to meet the demand. They’re all simple, and quite slow.

About six months ago I wrote a fast Python script to upgrade all local pip packages.

The idea is simple.


import pip
import queue

Then, query pip for the list of installed packages:

def buildQueueOfInstalledPackages():
    distQueue = queue.Queue()
    for dist in pip.get_installed_distributions():
    return distQueue

Here is where my script gets interesting:

        queueOfPackagesToUpdate = buildQueueOfInstalledPackages()
        queueOfFailedPackages   = queue.Queue()
        if sys.version_info.major == 3 and sys.version_info.minor > 3:
            numCPUs = os.cpu_count()
            print("you're using an out-of-date version of python! Please upgrade. I'll set the number of threads to 8, as I can't query the cpu_count")
            numCPUs = 8
        [threads.append(installerThread(queueOfPackagesToUpdate, queueOfFailedPackages, "install -U")) for _ in range(numCPUs)]
        for thread in threads:

I’m doing it in parallel.

The problem with upgrading pip packages is twofold. Firstly, it doesn’t have an --upgrade-all option (hence my script); Secondly, the process of upgrading each package is not just slow, but a single-threaded, cpu-bound operation.

This script is actually pretty straightforward. For each item in the queue, it spawns an instance of pip with “pip install -U package_name”, where  package_name is the name of the package. Each process will use 100% of a single core.  On versions of Python below 3.4,  it defaults to 8 threads – if you’re having problems, then you may need to tweak numCPUs yourself.

There are many easy optimizations remaining, but this is only intended as a stopgap, until pip gets the--upgrade-all option that it needs.

Migration, simpler

My script also makes it much easier to migrate a set of locally installed packages to a newer Python installation.

Run it in the source Python installation with the argument --writeToFile, and --readFromFile in the destination Python installation.

The script will write the list of packages to a file in the working directory, and then read the same file. It’s not exactly elegant, but it works.

Go grab it on GitHub:


I welcome feedback, and enjoy pull requests!

~ by Alexander Riccio on December 16, 2014.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Lucky's Notes

Notes on math, coding, and other stuff


Abandoned places and history from the five boroughs and beyond.

Open Mind

Science, Politics, Life, the Universe, and Everything

I learned it. I share it.

A software engineering blog by György Balássy

Untapped Cities

Rediscover your city: Urban discovery and exploration in NYC and around the world

Bit9 + Carbon Black Blog


The Electric Chronicles: Power in Flux

If someone ever tells you that you don't need more power, walk away. You don't need that kind of negativity in your life.

Ted's Energy Tips

Practical tips for making your home more comfortable, efficient and safe

love n grace

feel happy, be happy

Recognition, Evaluation, Control

News and views from Diamond Environmental Ltd.

greg tinkers

Sharing the successes and disasters.

Sam Thursfield's Blog

I want music in my life not questions!

Cranraspberry Blog

Sharing the things I love


Advances in biological systems.

The Embedded Code

Designing From Scratch

Sean Heelan's Blog

Program analysis, verification and security


Connecting Research, Policy, and Practice in Education


A Group Complaint about Law, Liberty, and Leisure

Warners' Stellian Appliance

Home & Kitchen Appliance Blog

Bad Science Debunked

Debunking dangerous junk science found on the Internet. Non-scientist friendly!

4 gravitons

The trials and tribulations of four gravitons and a postdoc

Strange Quark In London

A blog about physics, citylive and much procastination

The Lumber Room

"Consign them to dust and damp by way of preserving them"

In the Dark

A blog about the Universe, and all that surrounds it

andrea elizabeth

passionate - vibrant - ambitious

Probably Dance

I can program and like games

a totally unnecessary blog

paolo severini's waste of bandwidth

Musing Mortoray

Programming and Life

PJ Naughter's space

Musings on Native mode development on Windows using C++

  Bartosz Milewski's Programming Cafe

Category Theory, Haskell, Concurrency, C++

Brandon's Thoughts

Thoughts on programming

David Crocker's Verification Blog

Formal verification of C/C++ code for critical systems

10 Minute Astronomy

Stargazing for people who think they don't have time for stargazing.

One Dev Job

notes of an interactive developer

Enterprise Architect, IoT, Cloud, Mobile Apps, Technology Evangelist, Technical Pre-Sales, Business Evangelist, Speaker

Coder/Architect for IoT, Cloud Technologies and Mobile Apps, Azure Cloud, Amazon Cloud, Windows Phone 10 Apps, iPhone Apps, Scrum Master, Business Evangelist, Mobile apps developer in iOS and Windows 10 UWP, Azure IoT Hub, Machine Learning, Stream Analytics, Azure Mobile Service, APM Tools

The Angry Technician

No, the Internet is not broken.

Kenny Kerr

Author • Systems programmer • Creator of C++/WinRT • Engineer on the Windows team at Microsoft • Romans 1:16

IT affinity!

The Ultimate Question of Life, the Universe, and Everything is answered somwhere else. This is just about IT.


The ramblings of a crazed mind

Molecular Musings

Development blog of the Molecule Engine

%d bloggers like this: