Upgrading & migrating pip packages, en masse
Pip, the Python package management system, still lacks an easy way to update all installed packages. The “upgrade-all” ability has been in the works for nearly 4 years now.
In the meantime, many simple hacks have evolved to meet the demand. They’re all simple, and quite slow.
About six months ago I wrote a fast Python script to upgrade all local pip packages.
The idea is simple.
import pip import queue
Then, query pip for the list of installed packages:
def buildQueueOfInstalledPackages(): distQueue = queue.Queue() for dist in pip.get_installed_distributions(): distQueue.put(dist) return distQueue
Here is where my script gets interesting:
queueOfPackagesToUpdate = buildQueueOfInstalledPackages() queueOfFailedPackages = queue.Queue() if sys.version_info.major == 3 and sys.version_info.minor > 3: numCPUs = os.cpu_count() else: print("you're using an out-of-date version of python! Please upgrade. I'll set the number of threads to 8, as I can't query the cpu_count") numCPUs = 8 [threads.append(installerThread(queueOfPackagesToUpdate, queueOfFailedPackages, "install -U")) for _ in range(numCPUs)] for thread in threads: thread.start()
I’m doing it in parallel.
The problem with upgrading pip packages is twofold. Firstly, it doesn’t have an
--upgrade-all option (hence my script); Secondly, the process of upgrading each package is not just slow, but a single-threaded, cpu-bound operation.
This script is actually pretty straightforward. For each item in the queue, it spawns an instance of pip with “pip install -U package_name”, where package_name is the name of the package. Each process will use 100% of a single core. On versions of Python below 3.4, it defaults to 8 threads – if you’re having problems, then you may need to tweak numCPUs yourself.
There are many easy optimizations remaining, but this is only intended as a stopgap, until pip gets the
--upgrade-all option that it needs.
My script also makes it much easier to migrate a set of locally installed packages to a newer Python installation.
Run it in the source Python installation with the argument
--readFromFile in the destination Python installation.
The script will write the list of packages to a file in the working directory, and then read the same file. It’s not exactly elegant, but it works.
Go grab it on GitHub:
I welcome feedback, and enjoy pull requests!