camelofdk
1 day ago
My background is in software development and system administration, not aviation, and my development area was specifically telecommunications (writing software that administered setup changes on telecom switches/exchanges). As this involved inventing new, unusual technology in the DSL (domain specific language) realm that depended on an advanced understanding of the host servers' processing of program/function modules and run-time treatment of object code, I became intimately familiar with diagnosing ways software can be plagued by programming errors that ocasionally lead to dormant bugs.

I'll give an example. You code a function call that involves using a tricky pointer construction to pass a data value. Done correctly, this kind of setup allocates a place in memory to hold the data being passed and a location reference (the pointer) through which the data can be accessed. In some programming environments, a commonly-made mistake is bungling the way the pointer is created, leaving it pointing not to the intended data-holding element(s) but instead to an incorrect memory location whose contents and their nature and usage are unknown. This is a kind of "wild pointer".

If you're lucky, your wild pointer will corrupt vital data during testing, leading to detectable and traceable errors. But programs in executable (machine-language) form always contain "wasteland" space that's not actually being used. If your wild pointer corrupts something there, the code using it may appear to work normally - sometimes for years. But then one day, something (a recompilation, for example) can cause the arrangement of things in the executable to shift around, and you may suddenly be corrupting not waste space but something vital.

When I hear that a suspected bad sensor is thought to have supplied a faulty value, as a veteran "bitsmith" I have to wonder whether a good sensor value hasn't been hosed by a program bug.

[. . .]
And, BTW, yes, guys, this can happen in Ada-based avionics software.

[. . .]
WmJBean
1 day ago
...or it may just be Boeing's sensor supplier failed to do adaquate Q.A.

[. . .]
camelofdk
1 day ago
I hope the angle-of-attack sensor data supplied directly by the sensor is something that can be recovered from the FDR and correlated timewise with the other flight data. If the data show level- or nose-down readings being "corrected" by the avionics system attempting to pull out of a perceived stall threat by lowering the nose, I would strongly suspect corruption of the sensor's value in memory by a software bug.

As a programmer, I would want a series of memory dumps, but I have no idea whether that sort of thing has become a standard part of what the "black" box equipment does. If not, it really should be.

And the human pilots should be able to more practicably override the software if they suspect it's gone off its rocker.

camelofdk
1 day ago
(Edited)
A colleague and I once programmed, tested, delivered and installed a 6,000-line software package involving several hundred "tricky pointer" uses of the kind my example mentions. This doesn't sound like much until you realize that the code was ultra-compact because it literally had no ordinary variables. It comprised a compiler, virtual machine, and administration utility handling all item-specific data elements via elaborately-structured data tables maintained by our software's users, who were telecom switching engineers. Our QA (test) people declined to test it, citing its monumental level of complexity. So aside from our client company's cursory checks (we actually created "their" data tables for them), the whole package went into production as was. It ran for 2½ years and was never "QDR'd" (found to have a bug).

There are clear reasons why this seemingly "risky" piece of code behaved itself, and they had to do with how it was designed, laid out in code, and tested. I can still remember some important points:

- Complexity that could be encapsulated in clean (though sometimes complicated) data structures was, minimizing procedural code lines, and making testing more certain.

- I wrote a 50+ page code-level test script as I wrote the code and exhaustively carried it out. Every conceivable "boundary condition" (e.g., array [sub-]dimension limit) was identified and tested.

- The code itself was intentionally visually laid out for rational proofreading and systematic testing.

- "Risky" stuff like potentially confusing pointer usages was verified, for each type of case, to be producing correct object code and the code formats were catalogued into a finite set of approved patterns.

- No operation with the potential to overrun a data structure went unchecked.

- Sanity checks were programmed to detect "impossible" logic errors (but none ever did).

This is the real-world way to make software as close to error-free as possible. Alas, it seems to be seldom used!

Early on, employees of the F.A.A. and Boeing decided how to divide up the certification work. But, partway through the process, a former F.A.A. safety engineer told the Seattle Times, “we were asked by management to re-evaluate what would be delegated. Management thought we had retained too much at the FAA.” The engineer said that “there was constant pressure to re-evaluate our initial decisions,” and “even after we had reassessed it … there was continued discussion by management about delegating even more items down to the Boeing Company.”

Even the work that was retained, such as reviewing technical documents provided by Boeing, was sometimes curtailed. “There wasn’t a complete and proper review of the documents,” the former engineer added. “Review was rushed to reach certain certification dates.”

The new revelations don’t stop there. “Federal prosecutors and Department of Transportation officials are scrutinizing the development of Boeing Co.’s 737 MAX jetliners,” the Wall Street Journal reported on Monday. “A grand jury in Washington, D.C., issued a broad subpoena dated March 11 to at least one person involved in the 737 MAX’s development, seeking related documents, including correspondence, emails and other messages,” a source told the paper. (The Justice Department and Department of Transportation declined to comment on the Journal’s reporting.)

The criminal investigation began well before the crash of the Ethiopian Airlines Flight. It’s not clear yet whether it is focussing on the mcas system, the report in the Journal said. But, that article added, “In the U.S., it is highly unusual for federal prosecutors to investigate details of regulatory approval of commercial aircraft designs, or to use a criminal probe to delve into dealings between the FAA and the largest aircraft manufacturer the agency oversees. Probes of airliner programs or alleged lapses in federal safety oversight typically are handled as civil cases, often by the DOT inspector general.”