Upgrading & migrating pip packages, en masse

Upgrading, faster

Pip, the Python package management system, still lacks an easy way to update all installed packages. The “upgrade-all” ability has been in the works for nearly 4 years now.

In the meantime, many simple hacks have evolved to meet the demand. They’re all simple, and quite slow.

About six months ago I wrote a fast Python script to upgrade all local pip packages.

The idea is simple.


import pip
import queue

Then, query pip for the list of installed packages:

def buildQueueOfInstalledPackages():
    distQueue = queue.Queue()
    for dist in pip.get_installed_distributions():
    return distQueue

Here is where my script gets interesting:

        queueOfPackagesToUpdate = buildQueueOfInstalledPackages()
        queueOfFailedPackages   = queue.Queue()
        if sys.version_info.major == 3 and sys.version_info.minor > 3:
            numCPUs = os.cpu_count()
            print("you're using an out-of-date version of python! Please upgrade. I'll set the number of threads to 8, as I can't query the cpu_count")
            numCPUs = 8
        [threads.append(installerThread(queueOfPackagesToUpdate, queueOfFailedPackages, "install -U")) for _ in range(numCPUs)]
        for thread in threads:

I’m doing it in parallel.

The problem with upgrading pip packages is twofold. Firstly, it doesn’t have an --upgrade-all option (hence my script); Secondly, the process of upgrading each package is not just slow, but a single-threaded, cpu-bound operation.

This script is actually pretty straightforward. For each item in the queue, it spawns an instance of pip with “pip install -U package_name”, where  package_name is the name of the package. Each process will use 100% of a single core.  On versions of Python below 3.4,  it defaults to 8 threads – if you’re having problems, then you may need to tweak numCPUs yourself.

There are many easy optimizations remaining, but this is only intended as a stopgap, until pip gets the--upgrade-all option that it needs.

Migration, simpler

My script also makes it much easier to migrate a set of locally installed packages to a newer Python installation.

Run it in the source Python installation with the argument --writeToFile, and --readFromFile in the destination Python installation.

The script will write the list of packages to a file in the working directory, and then read the same file. It’s not exactly elegant, but it works.

Go grab it on GitHub:


I welcome feedback, and enjoy pull requests!

~ by Alexander Riccio on December 16, 2014.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Shelly L. Miller

I am an environmental engineer. I teach and research urban air pollution.

Lucky's Notes

Notes on math, coding, and other stuff


Abandoned places and history in the five boroughs

Open Mind

KIDS' LIVES MATTER so let's stop climate change

I learned it. I share it.

A software engineering blog by György Balássy

Kitware Inc

Delivering Innovation

The Electric Chronicles: Power in Flux

If someone ever tells you that you don't need more power, walk away. You don't need that kind of negativity in your life.

Ted's Energy Tips

Practical tips for making your home more comfortable, efficient and safe

love n grace

feel happy, be happy

Recognition, Evaluation, Control

News and views from Diamond Environmental Ltd.

greg tinkers

Sharing the successes and disasters.

Sam Thursfield

Software and technology from Galicia, Spain

Cranraspberry Blog

Sharing the things I love


Advances in biological systems.

The Embedded Code

Designing From Scratch

Sean Heelan's Blog

Software Exploitation and Optimisation


Connecting Research, Policy, and Practice in Education


A Group Complaint about Law, Liberty, and Leisure


Home & Kitchen Appliance Blog

Bad Science Debunked

Debunking dangerous junk science found on the Internet. Non-scientist friendly!

4 gravitons

The trials and tribulations of four gravitons and a physicist

Strange Quark In London

A blog about physics, citylive and much procastination

The Lumber Room

"Consign them to dust and damp by way of preserving them"

In the Dark

A blog about the Universe, and all that surrounds it

andrea elizabeth

passionate - vibrant - ambitious

Probably Dance

I can program and like games

a totally unnecessary blog

paolo severini's waste of bandwidth

Musing Mortoray

Programming and Life

PJ Naughter's space

Musings on Native mode development on Windows using C++

  Bartosz Milewski's Programming Cafe

Category Theory, Haskell, Concurrency, C++

Brandon's Thoughts

Thoughts on programming

David Crocker's Verification Blog

Formal verification of C/C++ code for critical systems

10 Minute Astronomy

Stargazing for people who think they don't have time for stargazing.

One Dev Job

notes of an interactive developer

Chief Cloud Architect & DevSecOps SME, Enterprise Architect, Agile Coach, Digital Transformation Leader, Presales & Tech Evangelist, Development Manager, Agilist, Mentor, Speaker and Author

TOGAF Certified Enterprise Architect • AWS Cloud Certified Solutions Architect • Azure Cloud Certified Solutions Architect • Scrum Alliance: Certified Scrum Professional (CSP), Certified Agile Leadership I (CAL 1), CSM, ACSM • Kanban Management Professional (KMP I & KMP II), Certified Enterprise Agility Coach (CEAC) • SAFe: Certified SAFe Architect, SAFe DevOps, Release Train Engineer (RTE), SAFe Consultant (SPC) • Certified Less Practitioner (CLP), Six Sigma (Greenbelt), Training from the Back of the Room (TBR) Trainer • Certified Agile Coach & Facilitator: ICP-ACF & ICP-ACC

The Angry Technician

No, the Internet is not broken.

Kenny Kerr

Creator of C++/WinRT and the Windows crate for Rust • Engineer on the Windows team at Microsoft • Romans 1:16

IT affinity!

The Ultimate Question of Life, the Universe, and Everything is answered somewhere else. This is just about IT.


The ramblings of a crazed mind

Molecular Musings

Development blog of the Molecule Engine

%d bloggers like this: