The issue with my email has been resolved. You may now email me at email@example.com.
Due to some sort of mixup with Hover, who manages my email address, firstname.lastname@example.org is at the moment, a non-functional email address, and has been so since early morning February 12. They have not responded to my emails asking for help. Will update soon.
A brilliant hack. And available on GitHub!
Originally posted on Hackaday:
[Charlie] was killing some time hacking on some cheap FPGA dev boards he bought from eBay. Initially, he intended to use them to create HDMI ports for a different project before new inspiration hit him. Instead, he added an HDMI port to Neo Geo MVS games. The Neo Geo MVS was a 90’s arcade machine that played gems like the Metal Slug and Samurai Showdown series. [Charlie] has a special knack for mods, being featured on Hackaday before for implementing Zork on hardware and making a mini supergun PCB. What’s especially nice about his newest mod is that the HDMI outputs both audio and video.
[Charlie] obtained the best possible video and audio signal by tapping the digital inputs to the Neo Geo’s DACs (digital-to-analog converter). The FPGA was then used to convert the signals to HDMI, maintaining a digital signal path from video generation to display. While this…
View original 249 more words
Microsoft’s SAL started nearly ten years ago at Microsoft as part of a major push for code quality, and (more visibly) preventing bugchecks. In its earliest versions, it appears to have been effectively restricted to the Windows core codebase, with kernel-mode-driver developers following suit. “Unleashing the Power of Static Analysis” says that here it helped developers fix more than twenty thousands bugs in Windows Vista, and more than 6,500 bugs in Microsoft Office 12. Of these, they found one buffer overrun per twenty annotations!
Way back in April (2014), I was in an Object Oriented Programming class, and I’d just started to master C++. With WinDirStat I had more than just an interesting problem to work on, but a real challenge to take on. But enough about WinDirStat, that’s for another article.
Side note: 90% of the slowdown in WinDirStat was NOT related to actually walking a directory tree, and was fixed by swapping a few data structures. Directory walking was however, still slow.
On windows there are two obvious ways to recursively enumerate files and directories. The first, provided by the Windows API, is with FindFirstFile/FindFirstFileEx & FindNextFile. This is a C/C++ API that will, on every call, (a) fill a predefined structure with information about a single file, or (b) fail and set the last error to ERROR_NO_MORE_FILES (the terrible behavior typical of the Windows API). The second, provided by MFC, is the CFileFind class. CFileFind is nothing more than a convenience wrapper for FindFirstFile, but with some utility functions & overhead.
The biggest downside to both methods is that they require a system call for every file. Internally, FindNextFile opens a file handle and then calls NtQueryDirectoryFile with said handle. This is terribly inefficient, especially if 8dot3 name creation is enabled. Back to the documentation.
Aha! With FindFirstFileEx, we can ask for only the relevant information (with FindExInfoBasic), and even better, there’s a mysterious flag: “FIND_FIRST_EX_LARGE_FETCH”, which is described to mean: “Uses a larger buffer for directory queries, which can increase performance of the find operation.”
Of course, the MSDN documentation provides “just the facts” (as Raymond Chen describes it), and nothing about where FIND_FIRST_EX_LARGE_FETCH should be used or what performance benefit it might provide. Raymond Chen offers some theory, but no hard numbers (and some comments suggest that it makes no difference). There’s also a poorly formatted Visual Studio user suggestion, which suggests using the USN (update sequence number) journal, which is not a viable option.
Perhaps most interestingly, FIND_FIRST_EX_LARGE_FETCH is used in the Chromium codebase!
The source even mentions that it “should speed up large enumerations”, but provides NO evidence.
Sidenote: the USN journal is not viable as it stores only a log of changes. There is no structure information, and it isn’t guaranteed to hold information about every file on the drive. Reconstructing the filesystem tree from the USN journal is an extremely complex task, and I doubt it could be done quickly, if at all.
The truth is: no matter what flags you set, you will see roughly the same picture:
Nearly all of the program’s time is spent in NtOpenFile & NtQueryDirectoryFile, and a single core (I have 8 logical) is pegged at 100%.
std::async as a force multiplier
Before I get to the hard numbers, I want to mention that there’s a parallel option. God damn do I love C++11.
Nearly three years ago, a team from Carnegie Mellon’s Robotics Institute, Mines ParisTech, and Texas Instruments, approached the problem of driving in the rain, with a brilliantly simple idea.
We’ve all been there – come nighttime, your headlights shine as brightly as ever, but at the rain, not the road. The road is still dark, but your eyes are adjusted for bright light.
Their idea is to simply project light precisely around the rain.
Sound hard to you? Well, it turns out that someone’s already solved the hardest part of the problem:
Pip, the Python package management system, still lacks an easy way to update all installed packages. The “upgrade-all” ability has been in the works for nearly 4 years now.
In the meantime, many simple hacks have evolved to meet the demand. They’re all simple, and quite slow.
About six months ago I wrote a fast Python script to upgrade all local pip packages.
The idea is simple.
import pip import queue
Then, query pip for the list of installed packages:
def buildQueueOfInstalledPackages(): distQueue = queue.Queue() for dist in pip.get_installed_distributions(): distQueue.put(dist) return distQueue
Here is where my script gets interesting:
This is beautiful :)
Originally posted on Gigaom:
Drones get a bad rap from the FAA but there’s growing evidence that more unmanned aircraft in the sky would do more good than harm. We’ve already seen how drones can save the day in search-and-rescue situations, and now a Dutch student is showing people how the devices, which can weigh under 5 pounds, could be a game-changer in medical emergencies.
Alex Momont, an engineer at the Technical University of Delft, has created an airborne defibrillator-delivery system that can reach anyone with a five-square-mile area in less than minutes. The school has posted this remarkable video showing how it works:
The so-called “Ambulance Drone” was the result of Momont’s Master thesis research. On his website, he likens the project to a medical toolbox:
[blockquote person=”” attribution=””]”The first minutes after an accident are critical and essential to provide the right care to prevent escalation. Speeding up emergency response can prevent deaths and accelerate recovery dramatically. This is notably true…
View original 160 more words
I have a post in the works about the performance of enumerating a directory with FindFirstFile/FindFirstFileEx, and CFileFind. I also investigate the various performance “tricks” (more like myths) used to speed these APIs up.
Two key findings:
- They’re actually fairly – but not terribly – fast
- FIND_FIRST_EX_LARGE_FETCH doesn’t do what you think it does.
Earlier this week I tracked down an insidious bug in Notepad++.