I will be out of normal contact until Wednesday March 25th

•March 17, 2015 • Leave a Comment

I’ll be away from my computer, but not my phone. This means ( sadly ) no compilation, and thus, no development.

alexander@riccio.com is BACK UP!

•February 21, 2015 • Leave a Comment

The issue with my email has been resolved. You may now email me at alexander@riccio.com.

ATTENTION: alexander@riccio.com is not reachable!

•February 14, 2015 • Leave a Comment

Due to some sort of mixup with Hover, who manages my email address, alexander@riccio.com is at the moment, a non-functional email address, and has been so since early morning February 12. They have not responded to my emails asking for help. Will update soon.

HDMI Audio and Video for Neo Geo MVS

•February 13, 2015 • Leave a Comment

A brilliant hack. And available on GitHub!

Preventing bugs, and improving code quality with Microsoft SAL (Part 1, Prologue)

•February 10, 2015 • Leave a Comment
_Post_invalid_ here is kind of a hack, but it works!

You’re using an invalid handle!

Microsoft’s SAL started nearly ten years ago at Microsoft as part of a major push for code quality, and (more visibly) preventing bugchecks. In its earliest versions, it appears to have been effectively restricted to the Windows core codebase, with kernel-mode-driver developers following suit. “Unleashing the Power of Static Analysis” says that here it helped developers fix more than twenty thousands bugs in Windows Vista,  and more than 6,500 bugs in Microsoft Office 12. Of these, they found one buffer overrun per twenty annotations!

Continue reading ‘Preventing bugs, and improving code quality with Microsoft SAL (Part 1, Prologue)’

How fast are FindFirstFile/FindFirstFileEx, and CFileFind – actually?

•January 13, 2015 • 1 Comment

For the past several months, I’ve been working on a fork of WinDirStat, the extremely popular disk usage analysis program.

WinDirStat is fantastically useful, but also fantastically slow. A well populated drive will lock WinDirStat for anywhere from fifteen minutes to a few hours, or worse (mirror).

Way back in April (2014), I was in an Object Oriented Programming class, and I’d just started to master C++. With WinDirStat I had  more than just an interesting problem to work on, but a real challenge to take on. But enough about WinDirStat, that’s for another article.

Side note: 90% of the slowdown in WinDirStat was NOT related to actually walking a directory tree, and was fixed by swapping a few data structures. Directory walking was however, still slow.

On windows there are two obvious ways to recursively enumerate files and directories. The first, provided by the Windows API, is with FindFirstFile/FindFirstFileEx & FindNextFile. This is a C/C++ API that will, on every call, (a) fill a predefined structure with information about a single file, or (b) fail and set the last error to ERROR_NO_MORE_FILES (the terrible behavior typical of the Windows API).  The second, provided by MFC, is the CFileFind class. CFileFind is nothing more than a convenience wrapper for FindFirstFile, but with some utility functions & overhead.

The biggest downside to both methods is that they require a system call for every file. Internally, FindNextFile opens a file handle and then calls NtQueryDirectoryFile with said handle. This is terribly inefficient,  especially if  8dot3 name creation is enabled. Back to the documentation.

Like many Windows APIs, there's an "extended" version.

Like many Windows APIs, there’s an “extended” version.

Aha! With FindFirstFileEx, we can ask for only the relevant information (with FindExInfoBasic), and even better, there’s a mysterious flag: “FIND_FIRST_EX_LARGE_FETCH”, which is described to mean: “Uses a larger buffer for directory queries, which can increase performance of the find operation.”

Of course, the MSDN documentation provides “just the facts” (as Raymond Chen describes it), and nothing about where FIND_FIRST_EX_LARGE_FETCH should be used or what performance benefit it might provide. Raymond Chen offers some theory, but no hard numbers (and some comments suggest that it makes no difference). There’s also a poorly formatted Visual Studio user suggestion, which suggests using the USN (update sequence number) journal, which is not a viable option.

Perhaps most interestingly, FIND_FIRST_EX_LARGE_FETCH is used in the Chromium codebase!

Yup, Chromium is using FIND_FIRST_EX_LARGE_FETCH.

Yup, Chromium is using FIND_FIRST_EX_LARGE_FETCH.

The source even mentions that it “should speed up large enumerations”, but provides NO evidence.

Sidenote: the USN journal is not viable as it stores only a log of changes. There is no structure information, and it isn’t guaranteed to hold information about every file on the drive. Reconstructing the filesystem tree from the USN journal is an extremely complex task, and I doubt it could be done quickly, if at all.

The truth is: no matter what flags you set, you will see roughly the same picture:

Note the "Functions Doing Most Individual Work" table.

Note the “Functions Doing Most Individual Work” table.

Nearly all of the program’s time is spent in NtOpenFile & NtQueryDirectoryFile, and a single core (I have 8 logical) is pegged at 100%.

std::async as a force multiplier

Before I get to the hard numbers, I want to mention that there’s a parallel option. God damn do I love C++11.

Continue reading ‘How fast are FindFirstFile/FindFirstFileEx, and CFileFind – actually?’

The future of automotive headlamps

•January 8, 2015 • Leave a Comment

 Nearly three years ago, a team from Carnegie Mellon’s Robotics Institute, Mines ParisTech, and Texas Instruments, approached the problem of driving in the rain, with a brilliantly simple idea.

We’ve all been there – come nighttime, your headlights shine as brightly as ever, but at the rain, not the road. The road is still dark, but your eyes are adjusted for bright light.

Their idea is to simply project light precisely around the rain.

Sound hard to you? Well, it turns out that someone’s already solved the hardest part of the problem:

Continue reading ‘The future of automotive headlamps’

Upgrading & migrating pip packages, en masse

•December 16, 2014 • Leave a Comment

Upgrading, faster

Pip, the Python package management system, still lacks an easy way to update all installed packages. The “upgrade-all” ability has been in the works for nearly 4 years now.

In the meantime, many simple hacks have evolved to meet the demand. They’re all simple, and quite slow.

About six months ago I wrote a fast Python script to upgrade all local pip packages.

The idea is simple.

First:

import pip
import queue

Then, query pip for the list of installed packages:

def buildQueueOfInstalledPackages():
    distQueue = queue.Queue()
    for dist in pip.get_installed_distributions():
        distQueue.put(dist)
    return distQueue

Here is where my script gets interesting:

Continue reading ‘Upgrading & migrating pip packages, en masse’

Ambulance drone can help heart attack victims in under 2 minutes

•November 30, 2014 • Leave a Comment

This is beautiful 🙂

Gigaom

Drones get a bad rap from the FAA but there’s growing evidence that more unmanned aircraft in the sky would do more good than harm. We’ve already seen how drones can save the day in search-and-rescue situations, and now a Dutch student is showing people how the devices, which can weigh under 5 pounds, could be a game-changer in medical emergencies.

Alex Momont, an engineer at the Technical University of Delft, has created an airborne defibrillator-delivery system that can reach anyone with a five-square-mile area in less than minutes. The school has posted this remarkable video showing how it works:

The so-called “Ambulance Drone” was the result of Momont’s Master thesis research. On his website, he likens the project to a medical toolbox:

[blockquote person=”” attribution=””]”The first minutes after an accident are critical and essential to provide the right care to prevent escalation. Speeding up emergency response can prevent deaths and accelerate recovery dramatically. This is notably true…

View original post 160 more words

PREVIEW: How fast are FindFirstFile/FindFirstFileEx, and CFileFind – actually?

•September 28, 2014 • Leave a Comment

I have a post in the works about the performance of enumerating a directory with FindFirstFile/FindFirstFileEx, and CFileFind.  I also investigate the various performance “tricks” (more like myths) used to speed these APIs up.

 

HowFastAre_preview

(sneak peek)

 

Two key findings:

  1. They’re actually fairly – but not terribly – fast
  2. FIND_FIRST_EX_LARGE_FETCH doesn’t do what you think it does.
 
Lucky's Notes

Notes on math, coding, and other stuff

AbandonedNYC

Abandoned places and history in the five boroughs

Open Mind

KIDS' LIVES MATTER so let's stop climate change

I learned it. I share it.

A software engineering blog by György Balássy

The Electric Chronicles: Power in Flux

If someone ever tells you that you don't need more power, walk away. You don't need that kind of negativity in your life.

Ted's Energy Tips

Practical tips for making your home more comfortable, efficient and safe

love n grace

feel happy, be happy

Recognition, Evaluation, Control

News and views from Diamond Environmental Ltd.

greg tinkers

Sharing the successes and disasters.

Sam Thursfield's Blog

I want music in my life not questions!

Cranraspberry Blog

Sharing the things I love

Biosingularity

Advances in biological systems.

The Embedded Code

Designing From Scratch

Sean Heelan's Blog

Program analysis, verification and security

EduResearcher

Connecting Research, Policy, and Practice in Education

Popehat

A Group Complaint about Law, Liberty, and Leisure

Warners' Stellian Appliance

Home & Kitchen Appliance Blog

Bad Science Debunked

Debunking dangerous junk science found on the Internet. Non-scientist friendly!

4 gravitons

The trials and tribulations of four gravitons and a postdoc

Strange Quark In London

A blog about physics, citylive and much procastination

The Lumber Room

"Consign them to dust and damp by way of preserving them"

In the Dark

A blog about the Universe, and all that surrounds it

andrea elizabeth

passionate - vibrant - ambitious

Probably Dance

I can program and like games

a totally unnecessary blog

paolo severini's waste of bandwidth

Musing Mortoray

Programming and Life

PJ Naughter's space

Musings on Native mode development on Windows using C++

  Bartosz Milewski's Programming Cafe

Category Theory, Haskell, Concurrency, C++

Brandon's Thoughts

Thoughts on programming

David Crocker's Verification Blog

Formal verification of C/C++ code for critical systems

10 Minute Astronomy

Stargazing for people who think they don't have time for stargazing.

One Dev Job

notes of an interactive developer

Chief Cloud Architect & DevSecOps SME, Enterprise Architect, Agile Coach, Digital Transformation Leader, Presales & Tech Evangelist, Development Manager, Agilist, Mentor, Speaker and Author

TOGAF Certified Enterprise Architect • AWS Cloud Certified Solutions Architect • Azure Cloud Certified Solutions Architect • Scrum Alliance: Certified Scrum Professional (CSP), Certified Agile Leadership I (CAL 1), CSM, ACSM • Kanban Management Professional (KMP I & KMP II), Certified Enterprise Agility Coach (CEAC) • SAFe: Certified SAFe Architect, SAFe DevOps, Release Train Engineer (RTE), SAFe Consultant (SPC) • Certified Less Practitioner (CLP), Six Sigma (Greenbelt), Training from the Back of the Room (TBR) Trainer • Certified Agile Coach & Facilitator: ICP-ACF & ICP-ACC

The Angry Technician

No, the Internet is not broken.

Kenny Kerr

Author • Systems programmer • Creator of C++/WinRT • Engineer on the Windows team at Microsoft • Romans 1:16

IT affinity!

The Ultimate Question of Life, the Universe, and Everything is answered somwhere else. This is just about IT.

Eat/Play/Hate

The ramblings of a crazed mind

Molecular Musings

Development blog of the Molecule Engine