Using SAL in the SQLite API

•May 10, 2015 • Leave a Comment

I’ve just finished adding annotations to a huge portion of the SQLite API, and I see a LOT of potential. This is going to be awesome.

See the changes that I’ve made so far on GitHub:

I’m not a core SQLite dev, so I just hacked on the amalgamation ūüėä

Preventing bugs, and improving code quality with Microsoft SAL (Part 2, custom preconditions for structs & objects)

•April 2, 2015 • Leave a Comment

Note: At the end of part 1, I’d suggested that part 2 would be about invalid handles.¬†This post however isn’t about invalid handles. That’ll be the next part in this series.

I very frequently come across code like this:

void dosomething( _In_ somestruct* in ) {
    //Not detected at compile time

The function, dosomething, expects/requires a specific condition. So, the author asserts it, and resolves to test every possible code path, thus verifying code correctness. When all their tests pass, they build in release mode, and the assert disappears.

Some time later, someone calls dosomething under different conditions, all of their tests pass, and they release the modified code.  Users start seeing their data mysteriously corrupt itself, and their once-reliable program is crashing in ways that nobody can reproduce. After several days worth of investigation, developers thoroughly audit the code, and spot the new (and incorrect) code path. They refactor the faulty code path, add a VERY LOUD comment to dosomething, and forget about the whole thing.


Some number of months¬†later, the same thing happens. This time, a developer has a bright idea: Let’s return an error code! Whoever wrote the code without a proper check for invalid arguments is an idiot!

The function now looks like this:


HRESULT dosomething( _In_ somestruct* in ) {
    //Not detected at compile time
    if ( !in->some_precondition ) {
        return E_INVALIDARG;

We’re better off now, or¬†at least we think so.

Now of course, everybody has to check the return code, which means more complexity, more ways to go wrong, and more code paths to test. Since there are far too many code paths to test every single one, some end user might still get to one of the untested, and also invalid, path.

Yet a third developer comes along, and thinks: What a silly C programmer. Someone will inevitably ignore the return value and the error will go unnoticed. Even worse, because I have to check the return value every single time, I can’t compose this function, and thus it’s less practical. This is an exceptional case, so we should treat it as one.

The function now looks like this:

void dosomething( _In_ somestruct* in ) {
    //Not detected at compile time
    if ( !in->some_precondition ) {
        throw std::invalid_argument( "Function preconditions not met!" );


We’re a little bit better off now, at least we’re pretty sure that we are.

Now, exception safety becomes an issue. We still have to test every possible code path to make sure that none of them incorrectly call dosomething. We still can’t be confident about it.

Perhaps somebody else comes along, and writes a wrapper object, thus only “allowing” correct uses of dosomething. Now everybody has to rewrite their code use that wrapper.

Complexity has increased, this function has been refactored thrice, and we still haven’t even found a way to be code-correctness-confident, at compile-time,¬†for all the uses of dosomething.

The cycle repeats, no end in sight.

Nearly every program stops there – they choose error codes or C++ exceptions, to carefully handle bad logic WHEN it happens.¬†Preventing them from happening, well that becomes an issue of “professionalism” and “discipline”.


Neither “professionalism” nor¬†“discipline”, actually prevent the mistakes of logic, but blaming the mistake on a given individual’s moral or personal failings works nicely to…well I’m not actually sure; I have noticed it is a fairly pervasive and pathological mindset in case studies of engineering disasters (Therac-25, Toyota electronic throttle¬†malfunctions, others). But, I digress.

¬†A few examples from Microsoft’s headers

One of Microsoft’s motivating¬†factors in developing SAL, was the infamous blue screen. Drivers. particularly kernel-mode drivers, are extremely sensitive to misuse. ¬†A failure in the kernel can mean system instability, serious security vulnerabilities, or massive data corruption. Furthermore, the kernel APIs that drivers rely on are often very complex, interacting with many components of hardware and software – and thus very hard to get right.

Continue reading ‘Preventing bugs, and improving code quality with Microsoft SAL (Part 2, custom preconditions for structs & objects)’

A hydrogen bomb detonated against your eyeball

•March 24, 2015 • Leave a Comment

10 Minute Astronomy

…would deliver less energy to your retina than a supernova observed from a distance of one astronomical unit (AU; the distance from the Earth to the sun). How much less? From this XKCD What If:

Which of the following would be brighter, in terms of the amount of energy delivered to your retina:

A supernova, seen from as far away as the Sun is from the Earth, or

The detonation of a hydrogen bomb pressed against your eyeball?

Applying the physicist rule of thumb suggests that the supernova is brighter. And indeed, it is … by nine orders of magnitude.

That rocked me back on my heels. And it got me thinking: how far away would one have to be for a supernova to be only as bright as an h-bomb pressed against one’s eyeball?


Radiated energy is subject to the inverse-square law, by which intensity of radiation…

View original post 327 more words

Trace Your Book or Kindle with the FingerReader

•March 19, 2015 • Leave a Comment

The MIT media lab does some pretty damned cool things.

Although, I can imagine a few improvements ūüėČ

I will be out of normal contact until Wednesday March 25th

•March 17, 2015 • Leave a Comment

I’ll be away from my computer, but not my phone. This means ( sadly ) no compilation, and thus, no development. is BACK UP!

•February 21, 2015 • Leave a Comment

The issue with my email has been resolved. You may now email me at

ATTENTION: is not reachable!

•February 14, 2015 • Leave a Comment

Due to some sort of mixup with Hover, who manages my email address, is at the moment, a non-functional email address, and has been so since early morning February 12. They have not responded to my emails asking for help. Will update soon.

HDMI Audio and Video for Neo Geo MVS

•February 13, 2015 • Leave a Comment

A brilliant hack. And available on GitHub!

Preventing bugs, and improving code quality with Microsoft SAL (Part 1, Prologue)

•February 10, 2015 • Leave a Comment
_Post_invalid_ here is kind of a hack, but it works!

You’re using an invalid handle!

Microsoft’s¬†SAL started nearly ten years ago at Microsoft as part of a major push for code quality, and (more visibly) preventing bugchecks. In its earliest versions, it appears to have been¬†effectively¬†restricted to the Windows core codebase, with kernel-mode-driver developers following suit.¬†“Unleashing the Power of Static Analysis” says that here it helped developers fix more than twenty thousands bugs in Windows Vista, ¬†and more than 6,500 bugs in Microsoft Office 12. Of these, they found one buffer overrun per twenty¬†annotations!

Continue reading ‘Preventing bugs, and improving code quality with Microsoft SAL (Part 1, Prologue)’

How fast are FindFirstFile/FindFirstFileEx, and CFileFind – actually?

•January 13, 2015 • 1 Comment

For the past several months, I’ve been working on a fork of WinDirStat, the extremely popular disk usage analysis program.

WinDirStat is fantastically useful, but also fantastically slow. A well populated drive will lock WinDirStat for anywhere from fifteen minutes to a few hours, or worse (mirror).

Way back in April (2014), I was in an Object Oriented Programming class, and I’d just started to master C++. With WinDirStat I had ¬†more than just an interesting problem to work on, but a real challenge to take on.¬†But enough about WinDirStat, that’s for another article.

Side note: 90% of the slowdown in WinDirStat was NOT related to actually walking a directory tree, and was fixed by swapping a few data structures. Directory walking was however, still slow.

On windows there are two obvious ways to recursively enumerate files and directories. The first, provided by the Windows API, is with FindFirstFile/FindFirstFileEx & FindNextFile. This is a C/C++ API that will, on every call, (a) fill a predefined structure with information about a single file, or (b) fail and set the last error to ERROR_NO_MORE_FILES (the terrible behavior typical of the Windows API).  The second, provided by MFC, is the CFileFind class. CFileFind is nothing more than a convenience wrapper for FindFirstFile, but with some utility functions & overhead.

The biggest downside to both methods is that they require a system call for every file. Internally, FindNextFile opens a file handle and then calls NtQueryDirectoryFile with said handle. This is terribly inefficient,  especially if  8dot3 name creation is enabled. Back to the documentation.

Like many Windows APIs, there's an "extended" version.

Like many Windows APIs, there’s an “extended” version.

Aha! With FindFirstFileEx, we can ask for only the relevant information (with¬†FindExInfoBasic), and even better, there’s a mysterious flag: “FIND_FIRST_EX_LARGE_FETCH”, which is described to mean: “Uses a larger buffer for directory queries, which can increase performance of the find operation.”

Of course, the MSDN documentation provides “just the facts” (as Raymond Chen describes it), and nothing about where¬†FIND_FIRST_EX_LARGE_FETCH should be used or what performance benefit it might provide. Raymond Chen offers some theory, but no hard numbers (and some comments suggest that it makes no difference). There’s also a poorly formatted Visual Studio user suggestion, which suggests using the USN (update sequence number) journal, which is not a viable option.

Perhaps most interestingly, FIND_FIRST_EX_LARGE_FETCH is used in the Chromium codebase!

Yup, Chromium is using FIND_FIRST_EX_LARGE_FETCH.

Yup, Chromium is using FIND_FIRST_EX_LARGE_FETCH.

The source even mentions that it “should speed up large¬†enumerations”, but provides NO evidence.

Sidenote: the USN journal is not viable as it stores only a log of changes. There is no structure information, and it isn’t guaranteed to hold information about every file on the drive. Reconstructing the filesystem tree¬†from the USN journal is an extremely complex task, and I doubt it could be done quickly, if at all.

The truth is: no matter what flags you set, you will see roughly the same picture:

Note the "Functions Doing Most Individual Work" table.

Note the “Functions Doing Most Individual Work” table.

Nearly all of the program’s time is spent in NtOpenFile & NtQueryDirectoryFile, and a single core (I have 8 logical) is pegged at 100%.

std::async as a force multiplier

Before I get to the hard numbers, I want to mention that there’s a¬†parallel option. God damn do I love C++11.

Continue reading ‘How fast are FindFirstFile/FindFirstFileEx, and CFileFind – actually?’

Lucky's Notes

Notes on math, coding, and other stuff


Abandoned places and history from the five boroughs and beyond.

Open Mind

Science, Politics, Life, the Universe, and Everything

I learned it. I share it.

A software engineering blog by György Balássy

Untapped Cities

Rediscover your city: Urban discovery and exploration in NYC and around the world

Bit9 + Carbon Black Blog


The Electric Chronicles: Power in Flux

If someone ever tells you that you don't need more power, walk away. You don't need that kind of negativity in your life.

Ted's Energy Tips

Practical tips for making your home more comfortable, efficient and safe

love n grace

feel happy, be happy

Recognition, Evaluation, Control

News and views from Diamond Environmental Ltd.

greg tinkers

Sharing the successes and disasters.

Sam Thursfield's Blog

I want music in my life not questions!

Cranraspberry Blog

Sharing the things I love


Advances in biological systems.

The Embedded Code

Designing From Scratch

Sean Heelan's Blog

Program analysis, verification and security


Connecting Research, Policy, and Practice in Education


A Group Complaint about Law, Liberty, and Leisure

Warners' Stellian Appliance

Home & Kitchen Appliance Blog

Bad Science Debunked

Debunking dangerous junk science found on the Internet. Non-scientist friendly!

4 gravitons

The trials and tribulations of four gravitons and a postdoc

Strange Quark In London

A blog about physics, citylive and much procastination

The Lumber Room

"Consign them to dust and damp by way of preserving them"

In the Dark

A blog about the Universe, and all that surrounds it

andrea elizabeth

passionate - vibrant - ambitious

Probably Dance

I can program and like games

a totally unnecessary blog

paolo severini's waste of bandwidth

Musing Mortoray

Programming and Life

PJ Naughter's space

Musings on Native mode development on Windows using C++

  Bartosz Milewski's Programming Cafe

Category Theory, Haskell, Concurrency, C++

Brandon's Thoughts

Thoughts on programming

David Crocker's Verification Blog

Formal verification of C/C++ code for critical systems

10 Minute Astronomy

Stargazing for people who think they don't have time for stargazing.

One Dev Job

notes of an interactive developer

Enterprise Architect, IoT, Cloud, Mobile Apps, Technology Evangelist, Technical Pre-Sales, Business Evangelist, Speaker

Coder/Architect for IoT, Cloud Technologies and Mobile Apps, Azure Cloud, Amazon Cloud, Windows Phone 10 Apps, iPhone Apps, Scrum Master, Business Evangelist, Mobile apps developer in iOS and Windows 10 UWP, Azure IoT Hub, Machine Learning, Stream Analytics, Azure Mobile Service, APM Tools

The Angry Technician

No, the Internet is not broken.

Kenny Kerr

Author ‚ÄĘ Systems programmer ‚ÄĘ Creator of C++/WinRT ‚ÄĘ Engineer on the Windows team at Microsoft ‚ÄĘ Romans 1:16

IT affinity!

The Ultimate Question of Life, the Universe, and Everything is answered somwhere else. This is just about IT.


The ramblings of a crazed mind

Molecular Musings

Development blog of the Molecule Engine