The live feed buries the only useful information at the very bottom of the article:
> The plane manufacturer says it has found that intense radiation from the Sun could corrupt data crucial to flight controls.
> It’s thought most will be able to undergo a simple software update.
> The issue was discovered after a JetBlue aircraft en-route from Mexico to the United States in October experienced a ‘sudden drop in altitude’.
> The plane made an emergency landing, with reports at the time suggesting 15 to 20 people suffered minor injuries.
> It’s thought the incident was caused by intense solar radiation, which corrupted data in a computer used to help control the aircraft.
On the Qantas 72 flight (2008), the ATSB report showed the same power spike that upset the ADIRU also left tidy 1-word corruptions in the flight data recorder. Those aligned with the clock cycle, shared the same amplitude and were confined to single ARINC words. That is pretty much exactly the signature of a failing solid state relay or contactor on the shared avionics power bus (upstream of both FDR and fly by wire).
Radation-driven bit flips would be Poisson distributed in time and energy. So that is one way to find out
Do you think they're using the guise of "its solar radiation" as cover to do a software update to fix a more problematic "bug", and perhaps tangentially there are some changes in said-update to improve some error correcting type code (eg: related to detecting spurious bit flips).
Not in aviation.
No, that would be straight to jail.
Yeah I don't buy it either.
If it was really 'solar radiation' there would be more small details.
After reset, it went away. If it was this kind of hw issue, it should still be present.
Considering those units were designed back when they did not have EDAC mandated, I can believe it could have been a bit flip (along with some other stuff they will probably address to take into consideration this failure mode). Nowadays, most MCU's have ECC on them so the time of this excuse is mostly gone now. :)
What do you think it could be ?
"That is pretty much exactly the signature of a failing solid state relay or contactor on the shared avionics power bus (upstream of both FDR and fly by wire)."
Thanks. I didn’t realise what that meant in context.
The software update is actually a rollback, apparently.
https://www.pprune.org/rumours-news/669424-airbus-a320-recal...
Interesting how radiation issues could be solved in software.
To give you a bit of insight, around the same timeframe (late October/early November) I directly observed two high-accuracy RTK GPS receivers reporting high accuracy (2cm), full 3D DGPS lock with carrier phase, and positions wandering within about a 5m circle horizontally. The altitude was staying pretty consistent (within about 1m, which was outside of the reported accuracy but not bad) until there was a sudden 60m altitude shift. This was all while they were sitting static on the ground, verified both by the crew and the accelerometer, gyro, and RADAR data.
There wasn’t a software fix per se, but we were able to quickly add a check to verify that the Kalman Filter’s position variance estimate was on the same order of magnitude as the accuracy level that the receivers were reporting and put a big red warning up. This wasn’t a flight-critical system, but it is the first time we’d ever seen that behaviour from those receivers and we’ve used them for 5 years.
i would expect a huge shift like that to violate the gaussian assumption of the kalman filter? (which i guess is what you're checking, sort of?). regardless i would expect the kalman filter to smooth the shift over some substantial time at least?
Not my area at all, but I'm extremely surprised that a fly-by-wire system would use GPS as an altitude reference. Is that really the case?
Perhaps it's improving the checksum algorithm on network packets, or even ... adding one.
Makes you wonder, if/how _passengers_ are directly protected against the radiation
They're not. Excessive high altitude flight increases your chance of developing melanoma.
Ok, I'll take an aisle seat more often now instead of a window seat.
Unless you fly as often as pilots and other onboard staff, it's unlikely to be significant.
It comes down to voting algorithms and memory persistence. Sometimes there is a threshold before data are "voted out".
I don't work on the A320 but solar radiation is a well-known issue in avionics, generally speaking.
Edit: deleted some speculation
Finally turning on the ECC RAM option?
Maybe there's a range that requires a change?
Now imagine, if it was over the air update, then maybe there would be no disruption?
Agreed, I expected additional shielding or something physical like that.
s/solved/mitigated/
i wonder how definitive that is and how well they were able to reproduce the issue under controlled conditions and how strong the evidence is that there was particularly strong solar radiation in play. it would probably be a good thing if they published technical details for investigations like this that impact public safety.
i believe it could be solar radiation, but i also believe that solar radiation could be a catch-all for unexplained phenomena.
Note that the software update (it actually looks like a roll-back to an older version?) will only fix 4,500 newer aircraft, another older 2,000 (not sure what these are, they can't be pre-NEO, the ratios seem wrong?) will also need a hardware fix.
Also: > The radiation corrupted data in the ELAC - a computer used to operate control surfaces on the wings and horizontal stabilizer.
It's unclear to me how a software update is supposed to help this component with radiation shielding
I'm amazed airlines haven't put up press releases detailing what is happening with their fleets yet. It has been a few hours so presumably they know and in the US at least this is a crazy busy weekend for travel.
At least they didn’t wait for a crash before doing this :/
The proper reaction when you have a potential issue in your engineering
Sorry forgot to add “unlike that other aerospace company”
Airbus is not immune to design & manufacturing issues with fatal consequences, they’re just not too-of-mind these days. A similar issue seems to have ‘cropped up’ on this flight: https://en.wikipedia.org/wiki/Qantas_Flight_72
There was a television show (episode) about another design issue (which was fatal) some time ago: https://en.wikipedia.org/wiki/Air_France_Flight_447
> https://en.wikipedia.org/wiki/Air_France_Flight_447
Quoting your link, "Final Report" section:
> Temporary inconsistency between the measured speeds, likely as a result of the obstruction of the pitot tubes by ice crystals, caused autopilot disconnection and [flight control mode] reconfiguration to "alternate law (ALT)".
- The crew made inappropriate control inputs that destabilized the flight path.
- The crew failed to follow appropriate procedure for loss of displayed airspeed information.
- The crew were late in identifying and correcting the deviation from the flight path.
- The crew lacked understanding of the approach to stall.
- The crew failed to recognize the aircraft had stalled, and consequently did not make inputs that would have made recovering from the stall possible.
Note the numerous "the crew"
It's often easy to blame the humans in the loop, but if the UX is poor or the procedures too complicated, then it's a systems fault even if the humans technically didn't "follow procedure".
The relief second officer basically pulled up when the stall protection had been disabled and by the time the other pilot and captain realized what was happening it was too late to save the plane.
There is a design flaw though: the sidesticks in modern Airbus planes are independent, so the other pilot didn’t get any tactile feedback when the second officer was pulling back.
Crews saved multiple 737-MAXs, but the public has focused on the aircraft whose crews were less effective.
Faulting the crew is a common thing in almost all air incidents. In this case the crew absolutely could have saved the plane, but the plane did not help them at all.
Part of the sales pitch of the Airbus is that the computer does A LOT of handholding for the pilots. In many configurations, including the one that the plane was flying in before the incident, the inputs that caused the crash would have been harmless.
In that incident the airspeed feed was lost to the computer and it literally changed the flight controls and turned off the safety limits, and none of the three people in the cockpit noticed. When an Airbus changes flight control modes, it does not keep inputs idempotent. Something harmless under one set of "laws" could crash the plane under another set of laws. In this case, what the pilot with the working control stick was doing would not have caused a crash, except that the computer had taken off the training wheels without anyone noticing.
As a result of changing the primary controls one pilot was able to unintentionally place the plane in an unrecoverable state without the other pilots even noticing that he was making control inputs.
So yes, the pilots flew the aircraft into the ground, but the computer suffered a partial failure and then changed how the primary flight controls operated.
Imagine if the brake pedal, steering wheel, and accelerator all started responding to inputs differently when your car had a sensor issue. That causes the cruise control to fail. Add in that the cruise control failure turns off ABS, auto-brakes, lane assist, and stability control for some reason. Oh yeah, there's a steering control on the other side of the car on the armrest and the person sitting there can now make steering inputs, but it won't give feedback in your steering wheel, and also your steering wheel still can be manipulated when the other guy is steering, but it is completely disconnected from the tires while the other guy is steering. Also, its a storm in the middle of the night, so you don't have a good visual reference for speed. So now your car is slipping, at night, in a storm, lights are flashing everywhere, nothing makes sense since the instruments are not reading correctly. However, the car is working exactly as described in the manual. When the car ends up in a ditch, the investigation will find that the cause of the crash was driver error since the car was operating exactly as it was designed.
Worth noting that Boeing (and just about every other aircraft on earth) has linked flight controls between the two pilot's positions that always behave in the exact same way so this type of failure could have never happened on a 737 for example.
At the end of the day, this was pilot error, but more in a "You're holding it wrong, I didn't design it wrong" kind of way. After all, there were three people with a combined 20k flying hours, including thousand of hours in that design.
If three extremely qualified pilots that have literal years of experience in that cockpit, who are rigorously trained and tested on a regular basis for emergencies in that cockpit, can fly the thing into the ground... maybe the design of the user interface needs a look.
You also conveniently skipped over the parts of the wikipedia article where they charged the manufacturer with manslaughter, and documented dozens of similar incidents.
As if Airbus hasn't suffered horrific crashes of their airplanes killing hundreds of people
It was only discovered after a flight experienced the issue, though. It could have been much more serious.
But what does it say about their QA or lack of it?
Just to be clear, I’m not faulting Airbus. I take issues with the shallow snark at Boeing. The JetBlue incident was serious.
Airbus isn’t immune to controversies , like AF447 or Habsheem air show crash in 1988
As software developers, we should perhaps refrain from criticizing aeronautical engineers' QA standards.
a problem on 1 flight in a gazillion, and you complain about QA?
Yes? How do you know it’s not? They roll back to a previous version. How do they know that version isn’t prone to the same issue?
In most capitalist organizations QA begs for more time. "getting to market" and "this years annual reports" are what help cause situations not here, not the working class, who want to do a good job.
Well "I ain't going" didn't let a mere crash or two stop them.
Boeing?
More discussion: https://news.ycombinator.com/item?id=46082296
Is this the graph of the root cause?
IMO saying that predictable solar cycles are a “root cause” is like calling Bobby Tables’ mom (https://xkcd.com/327/) the root cause of a SQL injection vulnerability.