16 years ago today, after watching a presentation on YouTube titled “Write and Submit your first Linux Kernel Patch” by Greg Kroah-Hartman, I submitted my first patch1 to the Linux kernel. 🐧🙂
I didn’t really have much idea of what I was doing, and after submitting a few more similar patches that same year (2010), I stopped.
Years later, in 2017, I picked it up again, and haven’t stopped since!🐧⚔️🛡
If you’ve contributed a few patches to the kernel but haven’t submitted in a while, give it another go. You never know what opportunities the future might bring. 🙌🏼
If you’ve never contributed to the kernel before and have some experience with C, you’d probably want to start here.
During one of my presentations at Open Source Summit Japan1 🇯🇵 the past year, I talked about a bug I found while addressing -Wflex-array-member-not-at-end issues in the Linux kernel. The warning reported by the compiler was the following.
drivers/net/virtio_net.c:429:46: warning: structure containing a flexible array member is not at the end of another structure [-Wflex-array-member-not-at-end]
According to the compiler, struct virtio_net_rss_config_trailer at line 429 above is a flexible structure; this is a structure that contains a flexible-array member (FAM), and the issue is that the FAM in this structure is not at the end of struct virtnet_info.
Let’s remember that not-at-end FAMs are a compiler extension that may cause undefined behavior, and compilers do not handle the sizes of objects containing them consistently. For this reason, they are now deprecated:
struct flex { int length; char data[]; };
struct mid_flex { int m; struct flex flex_data; int n; };
In the above, accessing a member of the array mid_flex.flex_data.data[] might have undefined behavior. Compilers do not handle such a case consistently. Any code relying on this case should be modified to ensure that flexible array members only end up at the ends of structures.2
flexible-array members and tail padding
Now, let’s take a look at the internals of struct virtio_net_rss_config_trailer using the Poke a Hole (pahole) tool.
As we can see above, the FAM hash_key_data[] is located at offset 3 and, of course, its size is 0 bytes. However, the size of the structure is 4 bytes, which means that between the end of the struct and the address of the FAM there is 1 byte of padding. As shown above, this is also displayed by pahole: /* padding: 1 */.
This type of padding between the last member and the end of a structure is called tail padding, and it’s quite common. However, it can be problematic if people are not familiar with the nuances of flexible-array members. This is one of those subtle details developers should be acutely aware of when working with FAMs, in particular.
Based on the work over the past few years making flexible-array members safer3 and addressing -Wflex-array-member-not-at-end issues across the whole kernel tree, I’ve come to notice that it’s not uncommon for people to assume that the offset of a FAM is always at the very end of the flexible structure. That is, people tend to assume that offsetof(struct flex, fam) == sizeof(struct flex) is always true. However, due to tail padding, this is not always the case and, as we’re about to see, it should not be taken for granted.
Key takeaway: offsetof(struct flex, fam) != sizeof(struct flex) when tail padding is present.
The bug
Okay, now let’s take a closer look at struct virtnet_info with the help of pahole. Note how the next member after the flexible structure is a fixed-size array: u8 rss_hash_key_data[40].
Aside from the tail padding in struct virtio_net_rss_config_trailer, two more things caught my attention here:
a) The name of the fixed-size array: rss_hash_key_data, is suspiciously similar to the FAM’s name: rss_trailer.hash_key_data[].
b) The type of the fixed-size array is basically the same as the element type of the FAM.
These two things tell me that, probably, the original intention here was to overlap both the FAM and the fixed-size array; what could also be thought of as an implicit union between the two arrays.
So we effectively have an implicit union between rss_trailer.hash_key_data[] and rss_hash_key_data[VIRTIO_NET_RSS_MAX_KEY_SIZE]. However, due to the tail padding in struct virtio_net_rss_config_trailer previously described, these arrays do not actually share the same address in memory: hash_key_data being at offset 83, and rss_hash_key_data at offset 84 in struct virtnet_info. See below.
Therefore, this results in a misalignment bug that must be fixed. 🐛
flexible-array members and implicit unions
Before getting into the details of how to properly fix this issue, let me note that implicit unions between a FAM and a fixed-size array are not uncommon in kernel code. I’ve addressed several -Wflex-array-member-not-at-end warnings when this was the case before (however, those did not manifest any misalignment bug). This is actually one of the categories of not-at-end FAMs I often discuss at conferences:
Upstream Kernel Hardening: Progress on enabling -Wflex-array-member-not-at-end (OSSJP2025)4
What’s crucial for implicit unions of this kind is that the arrays are correctly aligned. That is, both arrays must share the same address in memory. We also need to ensure that this alignment is not inadvertently changed in the future.
The bugfix
So, going back to the bug in question, we need to address two issues:
1) We need to find a way to make the FAM in struct virtio_net_rss_config_trailer appear at the end of struct virtnet_info. With this, we’ll be fixing the -Wflex-array-member-not-at-end warning.
2) We need to fix the misalignment bug between the FAM and the fixed-size array.
Over the past couple of years we, in the Linux Kernel Self-Protection Project5, have implemented different methods and strategies to address these FAM not-at-end issues. The most recent innovation we’ve come up with for this is the TRAILING_OVERLAP()6 helper macro.
The idea behind this helper is to address the common pattern of implicit unions between a FAM and a number of following adjacent members of any type in a structure by creating an explicit union between them. I plan to talk more in detail about this macro, and show different scenarios where it can be used in a future post. For now, I’ll just show the bugfix7 below and describe the fixed up memory layout.
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index ca92b4a1879ce..db88dcaefb20b 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -425,9 +425,6 @@ struct virtnet_info {
u16 rss_indir_table_size;
u32 rss_hash_types_supported;
u32 rss_hash_types_saved;
- struct virtio_net_rss_config_hdr *rss_hdr;
- struct virtio_net_rss_config_trailer rss_trailer;
- u8 rss_hash_key_data[VIRTIO_NET_RSS_MAX_KEY_SIZE];
/* Has control virtqueue */
bool has_cvq;
@@ -484,7 +481,16 @@ struct virtnet_info {
struct failover *failover;
u64 device_stats_cap;
+
+ struct virtio_net_rss_config_hdr *rss_hdr;
+
+ /* Must be last as it ends in a flexible-array member. */
+ TRAILING_OVERLAP(struct virtio_net_rss_config_trailer, rss_trailer, hash_key_data,
+ u8 rss_hash_key_data[VIRTIO_NET_RSS_MAX_KEY_SIZE];
+ );
};
+static_assert(offsetof(struct virtnet_info, rss_trailer.hash_key_data) ==
+ offsetof(struct virtnet_info, rss_hash_key_data));
struct padded_vnet_hdr {
struct virtio_net_hdr_v1_hash hdr;
After using TRAILING_OVERLAP(), and moving both struct virtio_net_rss_config_trailer rss_trailer; and u8 rss_hash_key_data[VIRTIO_NET_RSS_MAX_KEY_SIZE]; at the end of struct virtnet_info, we end up with the following memory layout.
After changes, those members are correctly aligned at offset 795:
struct virtnet_info {
...
union {
struct virtio_net_rss_config_trailer rss_trailer; /* 792 4 */
struct {
unsigned char __offset_to_hash_key_data[3]; /* 792 3 */
u8 rss_hash_key_data[40]; /* 795 40 */
}; /* 792 43 */
}; /* 792 44 */
...
};
As mentioned above, rss_trailer.hash_key_data and rss_hash_key_data are now correctly aligned at offset 795 (the 1-byte padding in between is gone), and FAM rss_trailer.hash_key_data is seen by the compiler as being at the end of struct virtnet_info. With this, both the warning and the bug are effectively resolved.
Lastly, the static_assert() line ensures that the alignment does not inadvertently change.
Conclusion
To conclude, let me just note that this is yet another subtle bug that -Wflex-array-member-not-at-end has helped us uncover in the Linux kernel. As I shared in a previous post8, we’ve recently completed 85% of the work toward enabling this compiler option in mainline. Once it is globally enabled, it’s unlikely that these and other classes of memory corruption bugs9101112 will make their way into the upstream Linux kernel going forward.
Today I woke up to the great news that we now have “only” 100 -Wflex-array-member-not-at-end issues left to be addressed in linux-next. I started fixing these issues a couple of years ago after running into a memory corruption bug1 caused by a flexible-array member in the middle of a structure.
I discovered this bug while reviewing my slides a couple of days before presenting at Kernel Recipes 2023 in Paris 🇫🇷. Naturally, I didn’t miss the opportunity to share the story during my presentation2 and talk about my plans to enable this compiler option upstream.
A few months before Kernel Recipes that year, I had experimented a bit with the, at the time, under-development version of -Wflex-array-member-not-at-end. Honestly, given the number of warnings and issues reported by the compiler, I didn’t pay much attention to it at the time. However, I knew it would eventually become one of the next great challenges to tackle in the Linux Kernel Self-Protection Project3.
It wasn’t until I confirmed that the bug I had incidentally found while reviewing my slides was also reported by -Wflex-array-member-not-at-end that I began actively addressing these issues and thinking about how to solve the problem as a whole.
In a future post, I’ll dive deeper into the effort and innovations behind this work, but for now I’ll just say that I’m really happy that about 85% of it is complete. As shown in a slide above, we started with 650 unique issues (accounting for roughly 60,000 warnings in total), and after a lot of effort, we’re now in the final phase before we can finally enable this compiler option in mainline Linux. 🙂
I’ve talked about this work at multiple conferences. Below are a couple of presentations for those who want to learn more.
Lastly, a shoutout to Qing Zhao4, who developed this compiler option and other hardening features in GCC5 that we use extensively to harden the Linux kernel. Qing retired this year, and all of us in the Linux Kernel Self-Protection Project are glad we had the opportunity to meet and work with her. All the best, Qing.
Thank you!🙂🐧🛡⚔️
P.S. The slides shown in this post are from the following presentation.
While reviewing one of the patches sent to the linux-hardening list this afternoon, I spotted a bug I introduced in the same code two years ago (on March 5 2024, to be precise).
Over nearly a decade contributing to the Linux kernel, I’ve discovered and fixed many years-old bugs, and it’s always a special feeling. This time is no different. Here to fix what I break! 🙌🏼
“struct nx842_crypto_header is declared with the __packed attribute, however the fields grouped with struct_group_tagged() were not packed. This caused the grouped header portion of the structure to lose the packed layout guarantees of the containing structure.
Fix this by replacing struct_group_tagged() with __struct_group(…, …, __packed, …) so the grouped fields are packed, and the original layout is preserved, restoring the intended packed layout of the structure.
I received the following request this evening and it absolutely made my day. 🙂
I’m a mechatronics student and I discovered the project from this block post: “https://embeddedor.com/blog/2024/09/28/one-simple-and-rewarding-way-to-contribute-to-the-linux-kernel-fix-coverity-issues/” and as such I’m instersted in contributing as well.
More than a year ago, I wrote a blog post to guide people to resources on how to start contributing to the Linux kernel by fixing Coverity issues. Every few weeks, I receive a request like the one above for access to the Coverity scans, and it makes my day every single time. It makes me really happy that more and more newcomers are finding inspiration in that blog post. 🙂
Someone might ask ‘Why Coverity in particular?’ The answer is quite simple: that’s how I started my career in the Linux kernel almost ten years ago. (That’s not to say this is the best possible way to start contributing to the kernel. This often depends on people’s previous experience and background. 😉)
I can therefore attest that it’s an effective way to gain experience contributing across the entire kernel tree, because it exposes people to all sorts of technical and social challenges, both of which are essential for learning to navigate the, at times, wild waters of the Linux kernel community.
You can check out the blog post via the following links. And if you think someone else might be interested or get inspired by it, please feel free to share it with your network. ✌🏼:
Now, for someone asking, ‘Why do people even have to ask for access to the Coverity scans for the kernel?’ The answer is straightforward: Black Duck manages the Coverity project directly through a request-for-access scheme, and others and I are only admins for the -rc and linux-next scans. See this for more information: https://scan.coverity.com/about, or contact Black Duck directly. 🙂
First time in South Korea. Three talks in two days. Over 200 minutes of public speaking. Two packed rooms. Made new connections. (My luggage arrived four days after me. 😅)
This week was very intense, and I’ll never forget this first visit to Seoul. I’m a bit exhausted right now, but really grateful.
Thanks, Korea! 🙏🏼🇰🇷♥️
See the abstracts and slides from my presentations below.
Enhancing spatial safety: Better array-bounds checking in C (and Linux)
The C language has historically suffered from a lack of proper bounds-checking on all types of arrays. The Linux Kernel Self-Protection Project has been addressing this issue for several years. In this presentation, we’ll learn about the most recent hardening efforts to resolve the problem of bounds-checking, particularly for fixed-size and flexible arrays.
We’ll explore the different mechanisms being used to harden key APIs like memcpy() against buffer overflows, which includes the use of some interesting built-in compiler functions. We’ll also talk about a couple of recent compiler options like -fstrict-flex-arrays and -Wflex-array-member-not-at-end, as well as the new counted_by attribute introduced in Clang 18 and GCC 15, which helps us gain run-time bounds-checking coverage on flexible arrays.
Overall, we’ll discuss how various challenges have been overcome, and highlight the innovations developed to solve the problem of array bounds-checking in both C and the Linux kernel once and for all.
I delivered the above (90-minute) presentation on November 4 and 5. The slides are basically the same for both sessions.
Below is the video of the presentation I gave on Nov 5. They haven’t uploaded the recording of the presentation on Nov 4, but as soon as it’s up, I’ll add it to my Presentations page. I personally liked that presentation better because the room was packed and people asked a lot of questions and made some comments.
Upstream Kernel Hardening: Progress on enabling -Wflex-array-member-not-at-end
The -Wflex-array-member-not-at-end compiler option was introduced in GCC 14. It warns about flexible-array members in the middle of composite structures. At the time, it revealed around 60,000 warnings in the upstream Linux kernel. While the vast majority of these are duplicates, about 650 are unique and require individual auditing and resolution. These issues fall into various categories and differ in complexity, which adds to the challenge of globally enabling this flag upstream.
In this presentation, we’ll share the progress we’ve made on this work as part of the Linux Kernel Self-Protection Project (KSPP) over the last year. We’ll go over the challenges we’ve encountered, show concrete code examples, and demonstrate how to fix these kinds of problems. We’ll also discuss why enabling this option is important for the kernel, and how we plan to complete this work in the near future.
Whether you’re a seasoned kernel developer or someone looking to start contributing upstream, this presentation will introduce useful helpers and strategies you can use to fix existing code or implement new functionality, and in doing so, help us harden the Linux kernel for the benefit of everyone.
