This year I had the amazing experience of traveling to Sweden 🇸🇪 (and Denmark 🇩🇰 ) for the first time to present at the Lund Linux Conference (https://lundlinuxcon.org/?page=current). 🐧 🗣️ 🎙️
It’s a small but very neat conference. Both the audience and the organizers were awesome, they made me feel right at home, and I even made some cool new friends! I really hope to return next year. 😃
The videos of the presentations a were uploaded a few days ago, so here’s my talk about upstream Linux kernel hardening. 🐧 ⚔️ 🛡️ Thanks!
The maintainer has already taken this patch, and it will soon land in mainline and a couple of stable trees. 😃🐧
Here’s a link to the slides and video from my latest presentation at Linux Plumbers Conference, where I discuss the ongoing efforts to globally enable the -Wflex-array-member-not-at-end compiler option in the upstream Linux kernel:
I’m really excited to share that my first presentation of 2025 will be in Adelaide, Australia. 🙌🏽😃
This will be my second time speaking at the Everything Open conference, and I’m really happy to be invited back. As always, feel free to say hi if you see me around!
I’ll be speaking about some of the recent progress on upstream Linux kernel hardening. ⚔️🛡️🐧
Here’s a link to the slides and video from my latest presentation on the ongoing efforts to globally enable the -Wflex-array-member-not-at-end compiler option in the upstream Linux kernel:
A few months ago, I had the wonderful experience of presenting as an invited speaker at Symposium sur la Sécurité des Technologies de l’Information et des Communications (SSTIC) in Rennes, France. 🇨🇵
From what my French friends have told me, this is one of the largest and most relevant information security conferences in France, and this year marked its 22nd edition.
The conference is typically held in French, with mine being the only talk in English this year. So, I’m really excited to share the video of the presentation with you all. 🙂 🙌🏽🐧
Enhancing spatial safety: Better array-bounds checking in C (and Linux) — Gustavo A. R. Silva
The C language has historically suffered from a lack of proper bounds-checking on all kinds of arrays. The Kernel Self-Protection Project has been addressing this issue for several years. In this presentation, we will learn about the most recent hardening efforts to resolve the problem of bounds-checking, particularly for fixed-size and flexible arrays.
We will explore the different mechanisms being used to harden key APIs like memcpy() against buffer overflows, which includes the use of some interesting built-in compiler functions. We will also talk about a couple of recent compiler options like -fstrict-flex-arrays and -Wflex-array-member-not-at-end, as well as the new __counted_by__ attribute released in Clang-18 a few weeks ago, which helps us gain run-time bounds-checking coverage on flexible arrays.
Overall, we will discuss how various challenges have been overcome and highlight the innovations developed to solve the problem of array bounds-checking in both C and the Linux kernel once and for all.
I gave a presentation at Kernel Recipes in September this year. 🇨🇵🗣️🐧
My slides are already up, and as always, I hope people find them useful. 🙂
I had some insightful discussions with the audience about the problem I’m currently working on. They asked questions and shared their thoughts. It turned out to be a very productive and interesting session. I really enjoyed it!
This conference is truly one of a kind, and for a number of reasons, it has a special place in my heart. ❤️
Motivated by the series of events I describe below, I decided to write this short blog post about how fixing Coverity issues can open the door to your first meaningful contributions to the Linux kernel. I hope people find it both inspiring and useful. 🙂
Kernel Newbies and Kernel Janitors
In October last year, I replied the following to an email sent to the kernel-janitors mailing list.
> Yesterday someone on my lists just sent an email looking for kernel > tasks. This was a university student in a kernel programming class. > We also have kernel-janitors and outreachy and those people are always > asking for small tasks.
We have tons of issues waiting to be audited and fixed here:
You will never run out of fun. :) People just need to sign up.
That's really a great way to learn and gain experience across the whole kernel tree.
-- Gustavo
At the time, my response didn’t gain much traction. However, early this month, I came across a familiar message on the kernel newbies mailing list asking for guidance on how to contribute to the Linux kernel.
Hi all,
I am an embedded software engineer. I use Linux every day, and I appreciate its neatness and simplicity.
One day, I watched a video from Greg: https://youtu.be/LLBrBBImJt4, and I started wondering if maybe I could contribute to the Linux kernel. So, I sent a very simple (and maybe stupid) patch to the community:
[...]
It turns out that the patch was rejected.
So, my question is: how can I start contributing to the Linux kernel? Maybe I could start by fixing some small bugs?
> One day, I watched a video from Greg: https://youtu.be/LLBrBBImJt4, and I started wondering if maybe I could contribute to the Linux kernel.
If you are interested in security, fixing Coverity issues is a great way to contribute to the kernel. Here are some presentations that you might find useful:
Later that day, I received a couple of notifications informing me that someone was requesting access to the Linux kernel Coverity scans. I granted the access and forget about it.
Fixing Coverity issues
Then, early last week, an email from that same thread landed my inbox:
Hi,
Thank you all for the good advice. I have now successfully submitted some small changes to the kernel:
Contributing to the Linux kernel is not that hard, all we need is patience and persistence.
I definitely will do more work on the Linux kernel!
-- Best, Qianqiang Liu
I was so happy to discover that those patches looked quite similar to the ones I used to submit back in the day when I was trying to land my first Coverity fixes. Then in a subsequent email, this was confirmed:
Hi Malatesh,
> Can you help me to contribute, what I needs to do ?
You can refer to this mail thread. The advice from Gustavo is pretty useful.
Even though, at the time I started working in the Linux kernel I already had solid experience in embedded systems, C programming, and had taken a Linux kernel development training, I was totally new to the Linux kernel community and all the nuances around upstream contributions in particular. As I was about to learn over the years, this is what is actually crucial for becoming a successful kernel contributor. However, gaining this experience takes time, and the only way to become an experienced contributor, as you might have guessed, is to send tons of patches.
Bug fixing presentations
So, for those interested in landing their first kernel patches, one simple way to start gaining experience contributing to the Linux kernel is by fixing as many Coverity issues as possible. I promise you’ll learn a lot in the process. 😉
Check out the following presentations, where I dive deep into fixing Coverity issues and other problems in the Linux kernel:
And of course, sign up for linux-next (despite being named “linux-next weekly scan”, these are actually daily scans) and -rc Coverity scans. It’d be helpful if you could briefly mention this blog post in your request message when signing up, though it’s not required. Also, I encourage you to make sure you know how to submit a kernel patch. 🙂
Learn from existing contributions
Back in the day, when I started fixing issues in the kernel, it was not uncommon for me to feel a bit lost from time to time. One of the best things you can do in those situations is to look at what others are currently working on in your areas of interest.
In this case, one simple approach is to check the commits addressing Coverity issues that have recently landed in linux-next. The link below will take you to a list of all kernel patches in linux-next that contain the keyword Coverity in their changelog text.
I recommend studying them. If you don’t understand how people concluded that that was the right fix for the issue, take it a step further and look up the email thread that initiated the discussion and read it thoroughly to understand what is going on. This can be a great learning experience. 😉
Below is a link to all the Linux kernel mailing lists, including of course The Linux Kernel Mailing List or LKML:
You can start by checking LKML. However, it’s not uncommon for developers to omit that list and send their patches only to the relevant subsystem mailing lists. So, if you don’t find the thread on LKML, look it up on the other lists. Depending on the driver the patch affects, it should be obvious which lists to check.
Lastly, when you send a patch addressing Coverity issues, please briefly mention that in the changelog text. A simple Reported by Coverity is enough. This way, others can easily find your commits in the future and learn from your contributions as well. 🙂
Try in staging first
As a final piece of advice, I recommend starting by fixing issues in drivers/staging/. After landing several patches and gaining some experience from the feedback provided by the staging maintainers, you will feel more comfortable moving on to other areas of the kernel.
Each subsystem and driver in the kernel is usually maintained by different groups of people, each with their own way of doing things and their own idiosyncrasies. Adapting to these different methods is one the most important pieces of experience you will gain as you continue submitting patches and paying attention to the feedback you receive along the way. Always remember: upstream Linux kernel development is highly social. 😉
I’m really happy to share that I will be traveling to Paris to speak at Kernel Recipes in the week after the Open Source Summit Europe. ✈️🇨🇵🗣️🎙️ This will be my 6th consecutive edition speaking at one of the most unique Linux kernel conferences. I’m really excited about this opportunity, and as always, feel free to say hi if you see me around. 🙂👋🏽
My talk will cover the work I’ve been doing in the Kernel Self-Protection Project over the last few months to fix thousands of -Wflex-array-member-not-at-end warnings. It can also be considered a sequel to my presentation last year, where I introduced this GCC compiler option to the audience:
You can see the description of my upcoming presentation below.
Enhancing spatial safety: Fixing thousands of -Wflex-array-member-not-at-end warnings
The introduction of the new -Wflex-array-member-not-at-end compiler option, released in GCC-14, has revealed approximately 60,000 warnings in the Linux kernel. Among them, some legitimate bugs have been uncovered.
In this presentation, we will explore in detail the different strategies we are employing to resolve all these warnings. These methods have already helped us resolve about 30% of them. Our ultimate goal in the Kernel Self-Protection Project is to globally enable this option in mainline, further enhancing the security of the kernel in the spatial safety domain.
The counted_by attribute was introduced in Clang-18 and will soon be available in GCC-15. Its purpose is to associate a flexible-array member with a struct member that will hold the number of elements in this array at some point at run-time. This association is critical for enabling runtime bounds checking via the array bounds sanitizer and the __builtin_dynamic_object_size() built-in function. In user-space, this extra level of security is enabled by -D_FORTIFY_SOURCE=3. Therefore, using this attribute correctly enhances C codebases with runtime bounds-checking coverage on flexible-array members.
Here is an example of a flexible array annotated with this attribute:
In the above example, count is the struct member that will hold the number of elements of the flexible array at run-time. We will call this struct member the counter.
In the Linux kernel, this attribute facilitates bounds-checking coverage through fortified APIs such as the memcpy() family of functions, which internally use __builtin_dynamic_object_size() (CONFIG_FORTIFY_SOURCE). As well as through the array-bounds sanitizer (CONFIG_UBSAN_BOUNDS).
The __counted_by() macro
In the kernel we wrap the counted_by attribute in the __counted_by() macro, as shown below.
ceba9725fb45 (“cxgb4: Annotate struct sched_table with …”)
However, as we are about to see, not all __counted_by() annotations are always as straightforward as the one above.
__counted_by() annotations in the kernel
There are a number of requirements to properly use the counted_by attribute. One crucial requirement is that the counter must be initialized before the first reference to the flexible-array member. Another requirement is that the array must always contain at least as many elements as indicated by the counter. Below you can see an example of a kernel patch addressing these requirements.
In the patch above, datalen is the counter for the flexible-array member data. Notice how the assignment to the counterevent->datalen = datalen had to be moved to before calling memcpy(event->data, data, datalen), this ensures the counter is initialized before the first reference to the flexible array. Otherwise, the compiler would complain about trying to write into a flexible array of size zero, due to datalen being zeroed out by a previous call to kzalloc(). This assignment-after-memcpy has been quite common in the Linux kernel. However, when dealing with counted_by annotations, this pattern should be changed. Therefore, we have to be careful when doing these annotations. We should audit all instances of code that reference both the counter and the flexible array and ensure they meet the proper requirements.
In the kernel, we’ve been learning from our mistakes and have fixed some buggy annotations we made in the beginning. Here are a couple of bugfixes to make you aware of these issues:
6dc445c19050 (“clk: bcm: rpi: Assign ->num before accessing…”)
9368cdf90f52 (“clk: bcm: dvp: Assign ->num before accessing…”)
Another common issue is when the counter is updated inside a loop. See the patch below.
34c34c242a1b (“wifi: wil6210: cfg80211: Use __counted_by…”)
The patch above does a bit more than merely annotating the flexible array with the __counted_by() macro, but that’s material for a future post. For now, let’s focus on the following excerpt.
Notice that in this case, num_channels is our counter, and it’s set to zero before the for loop. Inside the for loop, the original code used this variable as an index to access the flexible array, then updated it via a post-increment, all in one line: cmd.cmd.channel_list[cmd.cmd.num_channels++]. The issue is that once channel_list was annotated with the __counted_by() macro, the compiler enforces dynamic array indexing of channel_list to stay below num_channels. Since num_channels holds a value of zero at the moment of the array access, this leads to undefined behavior and may trigger a compiler warning.
As shown in the patch, the solution is to increment num_channels before accessing the array, and then access the array through an index adjustment below num_channels.
Another option is to avoid using the counter as an index for the flexible array altogether. This can be done by using an auxiliary variable instead. See an excerpt of a patch below.
ea9e148c803b (“Bluetooth: hci_conn: Use __counted_by() and…”)
Again, the entire patch does more than merely annotate the flexible-array member, but let’s just focus on how aux_num_cis is used to access flexible array pdu->cis[].
In this case, the counter is num_cis. As in our previous example, originally, the counter is used to directly access the flexible array: &pdu.cis[pdu.cp.num_cis++]. However, the patch above introduces a new variable aux_num_cis to be used instead of the counter: &pdu->cis[aux_num_cis++]. The counter is then updated after the loop: pdu->num_cis = aux_num_cis.
Both solutions are acceptable, so use whichever is convenient for you. 🙂
Here, you can see a recent bugfix for some buggy annotations that missed the details discussed above:
[PATCH] wifi: iwlwifi: mvm: Fix _counted_by usage in cfg80211_wowlan_nd*
In a future post, I’ll address the issue of annotating flexible arrays of flexible structures. Spoiler alert: don’t do it!
