The most popular questions posed to Linux kernel developer Matthew Garrett during his Reddit AMA this week related to kernel hacking and hardware issues. But Garrett, a senior security engineer at Nebula, answered frankly on a variety of subjects that ranged from technical issues in the kernel, to his workstation setup, to how to kill fruit flies and why he likes the movie Hackers. Here is a digest of some of the more kernel-related questions and answers (plus a fruit fly question, for more flavor.) Visit Reddit for the full AMA.
On being a Linux kernel developer
How did you get into kernel development?
I got started because my laptop didn’t suspend and resume properly, and anybody who knew anything about it would just tell me that it wasn’t expected to. I was doing a PhD at the time, so basically anything that wasn’t what I was supposed to be doing was an attractive option. I spent some time hacking on things and finally ended up with a laptop that had working ACPI suspend/resume. Then other people asked me to help them with their laptop, and it kind of went from there.
But part of that was that I was lucky – I stumbled into a corner of kernel development that people were interested in, but not many people were actively working on. It was pretty easy to become a subject matter “expert” when nobody else knew anything about it! These days it’s harder because most of the interesting bits of the kernel are already well-explored, and almost all of the easy work has already been done.
What was your most favorite moment in kernel development?
Every single time I’ve fixed a bug.
Has Nebula been supportive of the work you do that is not necessarily core to their business or do you have to mainly do your misc firmware/kernel hacking in your spare time?
Completely. Say we have a bug in the VM layer of the kernel. I’m not an expert in that field – it’d take me ages to make progress. But because I helped get a VM developer’s laptop working, they’re happy to take a look at the problem and give me feedback. The work I do in the broader community benefits us.
DE/WM/distro/favorite applications?
GNOME 3 with no extensions, Fedora 20, pretty much exclusively Firefox, Evolution (work email), Empathy, Liferea and gnome-terminal. And a lot of evince for spec reading.
Any special setup/method/tools when it comes to kernel hacking?
Special setup? Nope. Basically just emacs and git.
What does the inbox of a kernel dev looks like?
My inbox looks like a train carrying email crashed into another train carrying email, and then an email plane crashed into that.
On becoming a kernel developer
How difficult is it to get to be a kernel developer? I’ve been hearing about the Linux foundation trying to get some young people into the kernel development process.
http://eudyptula-challenge.org/ is probably a good place to start. The kernel is complicated code, and it’s a complicated community, but it’s still fundamentally just software. Becoming a kernel developer isn’t really any harder than becoming a developer on any other project, but getting your code into mainline – that’s rather harder.
I’m thinking on being an operating system developer, would you recommend to hack the kernel as a good start? And what programming languages do you use to hack the kernel?
The kernel is pretty much entirely in C, with a small amount of assembly language for each architecture. I wouldn’t recommend starting with the kernel as a way to learn operating systems, to be honest – Linux as it is now is a giant complicated thing with a lot of intricacies, and it’s not terribly well suited as a learning exercise. Something like Minix is a much better codebase to look at to figure out how things fit together and ought to help you figure out whether you’re more interested in the kernel or userspace.
On the Linux and open source communities
In which areas of GNU/Linux do you think there is still room for improvement?
Security. Privacy. Hardware support. Usability. Power management. I could probably go on.
What are today’s biggest challenges for the Linux platform and Open Source?
I think the biggest challenge is ensuring user freedom in the face of a rapidly changing computing climate. The move to online services means we’re losing many of our effective freedoms to control the software we depend on, and it’s going to be difficult to do something meaningful about that.
Many people have talked before about what we can do as a community (in FOSS) or as an industry (tech in general) to promote diversity and social justice. What are things that you think everyone should do about this?
Pay attention to what people are telling you. The work of the Ada Initiative is important here. Read their blog posts. Read their publications. Search for presentations and discussion of the OPW and listen to what people found helpful. I have opinions on this stuff, but I’m not the one doing the work – there are subject matter experts out there, and they’ve got much more to say on the subject than I do.
On hardware
Does Intel hardware still offer the best support? (condensed)
Eh. Intel CPU and graphics are still your best bet. Atheros wifi may well be reasonable. I’m disappointed at how much Intel won’t tell us these days – there are various integration specs they won’t release which means (for instance) backlight hotkeys are broken on some systems. The Thunderbolt situation is especially disappointing.
AMD have done a lot to improve things, but the GPU driver team is still significantly smaller than Intel’s. I understand some of the reasons for this, so I don’t want to give the impression that I don’t appreciate AMD’s work.
Least – Broadcom wireless is a disaster. They released a driver for their then-current wifi chipsets a few years back, so everybody gave up on reverse engineering their hardware. And then they never updated it to drive anything they released after that. Avoid like the plague. And nvidia, well. The enablement work they’re doing on Tegra is great, and I hope some of it bleeds over to the x86 side. But right now, you’d have to say that they’re at the back of the pack for good kernel support.
What about the requirement for the user to control their own computer by having the ability to actually boot it with freedom? I heard that this is an important factor in trustworthiness.
That kind of depends on what you trust. All x86 machines with Windows 8 certification will allow the users to control what their machine will boot – including shutting out the ability to boot Microsoft code. If you want control of your firmware then things are more limited. Modern Intel systems tend to require firmware for the management engine in the chipset, which is signed – it’s not currently possible to replace that, so even if you’re running Coreboot you still need that blob. AMD have been more helpful in providing documentation and assistance in that respect, but the firmware for the GPUs is still all closed.
You’ve spent quite some time doing investigation and archaeology to essentially reverse-engineer how and why various bits of hardware do what they do. Why not sit down (possibly with other people) and define what an actual good machine would look like?
There’s some work happening in that respect, but it’s not a short-term project… I hope that there’ll be some public announcements soon.
Other interesting opinions
What’s your personal opinion about systemd?
I like it! We ship Upstart in our product and, while clearly better than sysvinit, it’s honestly just not very good – the version in 12.04 can’t even reexec itself without losing state, which means you can’t load new selinux policy (for example). That did get fixed later, but spending years in that state isn’t a great advert. systemd is more reliable, more functional, has developed a significantly larger development community and doesn’t have a CLA (these points may be related)
What are your thoughts on Lennart Poettering’s recent blog post?
I’m a little worried about the reliance on btrfs, but it’s a sufficiently restricted set of functionality that we might be able to get away with it. Other than that, it’s a better proposal than I’ve seen from anyone else.
What is your opinion on the Android platform and its openness?
It’s an open OS, but not necessarily an open platform. There’s a bunch of technical decisions in Android that I disagree with, but it’s succeeded in getting free software into the hands of millions of people and it’s enabled projects like Cyanogen to exist. I think that’s an overall win.
I’m having a problem with fruit flies. Do you recommend a method for killing them? We have regular wild type fruit flies, but also see all combinations of white/brown bodies and red/brown eyes.
Those combinations are pretty normal. You’ll catch more fruit flies with vinegar than with honey – add a little yeast to some vinegar, dilute it with some water, put it in a bottle with a funnel in it. Maybe add some kind of surfactant so they’ll fall through the surface.