GrantSeltzer

bpf-next llm summaries

Wed, 01 Apr 2026 00:00:00 +0000

I try to follow the bpf-next mailing list closely, but I always miss things. LWN is incredible, I’m a long time subscriber, but there’s still gaps I’d like to fill. So I built a small automated system to summarize the mailing list for me and publish the results here on my site.

It works in two stages. First, a GitHub Actions workflow runs every morning at 5am ET. It queries the Patchwork REST API for recent bpf-next patches and commits the raw patch data to a staging directory in this blog’s repository. An hour later at 6am ET, a Claude Code scheduled agent picks up the staging files, reads the raw patch data, generates structured JSON summaries, writes them to data/bpf_next/daily/YYYY-MM-DD.json, cleans up the staging files, and pushes. That push triggers CI to rebuild and deploy the site to grantseltzer.github.io. On Mondays the workflow also generates a weekly rollup, and on the 1st of each month a monthly one. Each summary is kept as a dated file so the page serves as a running archive.

Replacing procfs with bpf

Sat, 21 Jun 2025 00:00:00 +0000

You can find all the code displayed in this post in my repo here.

The proc file system provides to users information about the processes on a running linux system via a file system interface. The directory structure is organized such that every process has its own directory. The files in the directory are useful for finding out permissions, state, open files, and more. I wrote a small post about it in 2019.

Global No-Inline BPF Functions

Sun, 11 May 2025 00:00:00 +0000

The bpf verifier is tasked with ensuring that bpf programs that are loaded will predictably halt and run safely. It follows all branches of bpf instructions, counting each instruction as another permutation of the program’s state. As it counts, there’s a limit of 1 million instructions.

If you’ve been writing bpf programs for a long time or have been looking at outdated examples then you’re probably familiar with using the static modifier and __always_inline macro.

Building Go Stack Traces from BPF

Wed, 06 Dec 2023 00:00:00 +0000

In a couple of my posts from a few years ago (1, 2) I explored the idea of attaching bpf programs to Go functions via uprobes. The second post dived into how to extract values of parameters. This can be seen as part 3 of the series as I’m going to demonstrate how to get a stack trace from bpf code. The work described in this post is in contribution to my work at Datadog on the Dynamic Instrumentation product, allowing users to hook specific functions and get snapshots of parameter values and stack traces.

How Go slices work

Thu, 30 Nov 2023 00:00:00 +0000

I recently ran into a hard to find bug in my Go code when using slices. Since I’ve been using Go for over 8 years now, I figured this is something that others will run into and want to share what I had forgotten about how slices work.

Take a look at the following Go code:

x := []int{1,2,3}
y := append([]int{}, x...)
z := x

We have three separate slices. What I want to briefly explore is the difference between y, and z. The difference has to do with how slices are implemented by Go. This can sometimes make it hard to track down bugs, so it’s important to know.

BPF Attach Cookies

Wed, 09 Aug 2023 00:00:00 +0000

BPF has a helper function called bpf_get_attach_cookie. It’s available in bpf programs like kprobes, uprobes, and tracepoints. It can be very useful for many applications so i’ll be exploring how to use it, and going over a few examples from real code i’ve written.

A cookie is just an unsigned 64-bit integer. That’s it. You can assign a cookie to a bpf program when you attach it. For example, here’s code using the go cilium/ebpf library to assign a cookie to a uprobe:

Memory Execution Override With ebpf

Fri, 08 Jul 2022 00:00:00 +0000

BPF Map Concurrency Techniques

Thu, 16 Jun 2022 00:00:00 +0000

There are times when developing your BPF projects when you need to ensure safe access to shared memory. You may have a counter that you’re updating from various different BPF programs and reading in userspace. You may also be updating map values from both userspace and from BPF. In this post i’ll demonstrate a couple different techniques for handing these scenarios safely.

Per-cpu maps

Per-cpu maps are types of maps where each possible CPU has its own copy of underlying memory. Since BPF programs can’t be preempted, when you access a value inside one of these maps from your BPF program, you know that it’s the only program touching that value. The userspace program can read these values at any time safely.

Handling errors with CGO

Wed, 16 Mar 2022 00:00:00 +0000

One of the main projects that I work on is libbpfgo. This is a Go wrapper around libbpf, a userspace library for dealing with bpf objects. The wrapper uses CGO to reference individual functions and data types in libbpf, which is written in C.

For example, BPFLoadObject() is a libbpfgo API function which calls bpf_object__load(), a libbpf API function.

func (m *Module) BPFLoadObject() error {
	ret := C.bpf_object__load(m.obj)
	if ret != 0 {
		return fmt.Errorf("failed to load BPF object")
	}

	return nil
}

The goal for libbpfgo is to fully implement the libbpf API (which is pretty big). One complication with this is that libbpf has very inconsistent error handling. Some APIs will return an integer error code directly, some will return a NULL pointer and set the error code in errno, some will return an error code inside of a pointer, and some return an error code which is also in errno. As a result of trying to maintain libbpfgo’s error handling sanity, I’ve picked up some lessons on how you can use CGO to your advantage.

Portable BPF with CO:RE

Mon, 11 Oct 2021 00:00:00 +0000

Basic Guide to Linux Mailing Lists

Sun, 10 Oct 2021 00:00:00 +0000

The Linux kernel is the biggest open source project in existence. It has tens of thousands of contributors from all around the world, from many different companies and communities.

Linux development does not happen in a single github repository, and likely never will. Instead there are many forks of the linux kernel. Development happens in these forks for very specific subsystems, only a fraction of the overall code. Contributors submit relevant patches to the ‘mailing list’ that is used for each of these forks. The fork maintainers review these patches, and at regular intervals submit groups of patches ‘upstream’ to more generalized forks. All these patches eventually make their way up to the ‘mainline’ kernel which a single maintainer presides over. Eventually the forks pull down the changes from other downstream forks.

The libbpf Documentation Site

Sun, 10 Oct 2021 00:00:00 +0000

libbpf is the standard implementation of the userspace library for loading and interacting with bpf programs. It defines the bpf ELF file format and CO-RE features. It’s a crucial part of the bpf ecosystem and is soon approaching its first major release. With this in mind, it’s important to have proper documentation.

Introducing libbpf.readthedocs.org. By going to the API docs you can see that each libbpf API function, type, enum and macro are listed. This is generated from the libbpf header files. Comments can be placed in code above the relevant API code and displayed on the documentation site.

Go Time #201

Tue, 14 Sep 2021 00:00:00 +0000

Listen on Changelog.com

Compile Once Run Everywhere, the eBPF Dream!

Thu, 19 Aug 2021 00:00:00 +0000

Skip to 3:11:00 for my talk (or watch the whole thing, it was a great conference!)

Supporting CO:RE in Tracee

Fri, 23 Jul 2021 03:32:37 +0000

If you’re used to developing software in a high level language like Go or Java you likely haven’t thought about supporting different operating system versions. Sure you want to be able to run on Linux, Macos, or Windows, but that’s abstracted away for you.

When developing eBPF software you must think about hundreds of Linux kernel versions. When your eBPF programs are reading in kernel data structure fields, there’s no guarantee of backwards compatibility or stable APIs. Take a look at this example:

Tracing Go Programs with eBPF!

Thu, 27 May 2021 00:00:00 +0000

Getting started with bpf and libbpfgo

Thu, 18 Mar 2021 00:00:00 +0000

In my previous posts on the subject of bpf I used a project called BCC to compile, load, and interact with my bpf programs. I, and many other developers, have recently heard about a better way to build ebpf projects called libbpf. There are a few good resources to use when developing libbpf based programs but getting started can still be a quite overwhelming. The goal of this post is to provide a simple and effective explanation of what libbpf is and how to start using it.

What is vmlinux.h?

Thu, 11 Mar 2021 00:00:00 +0000

A version of this post was also uploaded to Aqua's blog here

If you’ve been reading much bpf code recently, you’ve probably seen this:

#include "vmlinux.h"

vmlinux.h is generated code. it contains all of the type definitions that your running Linux kernel uses in it’s own source code. This is an important concept to wrap your head around so let me explain.

When you build Linux one of the output artifacts is a file called vmlinux. It’s also typically packaged with major distributions. This is an ELF binary that contains the compiled kernel inside it.

Parsing Go Binary DWARF Info

Sat, 02 Jan 2021 03:32:37 +0000

In my very first blog post, Dissecting Go Binaries, I began to explore ELF files. That is the default format of executable binaries that Go produces on unix-like operating systems like Linux and MacOS. My most recent project, weaver, has me exploring a related object file format that Go leverages, DWARF.

The DWARF specification is, of course, not Go specific. The DWARF object file specification describes the functions, variables, and types of a compiled program. Entries of a DWARF file are organized in a tree structure where each node can have children and siblings. For example, an entry node that describes a function would have children that describe the parameters. An entry node that describes a struct would have children that describe the fields of it.

Tracing Go Programs with eBPF!

Fri, 13 Nov 2020 00:00:00 +0000

Code can be found here

Go Time #159

Tue, 10 Nov 2020 00:00:00 +0000

Listen on Changelog.com

Tracing Go Functions with eBPF Part 2

Sun, 10 May 2020 03:32:37 +0000

In part 1 of this series we learned about how to attach uprobes and eBPF programs to specific functions in Go programs. We went through an example where we attached our probe to the handler of a webserver. Everytime the probe was triggered we simply printed out a log saying that the handler was called. This could be adapted to record metrics, perhaps using counters instead of log lines.

In this post we’re going to delve a bit deeper. We’re going to use the capabilities that eBPF has for traversing through the memory of the program we’re tracing.

Fermentation Experimentation

Thu, 05 Mar 2020 03:32:37 +0000

In the past year I’ve decided that I haven’t perpetuated enough Brooklyn stereotypes and have taken up fermentation as a hobby. This post goes through a few of the experiments that I’ve run with varying success.

My set up is pretty basic. I have a small cabinet/closet in the corner of my kitchen that i’ve designated as the ‘fermentation station’. I have a thermometer but it mostly stays at room temperature. In the winter it gets warmer as it sits directly in front of a radiator which is always on. I also affixed a curtain to avoid direct sunlight as the setup is next to a window.

Tracing Go Functions with eBPF Part 1

Fri, 14 Feb 2020 03:32:37 +0000

eBPF is a virtual machine, similar in concept to the JVM, except it’s inside the Linux kernel. It lets you write C-like code, compile it, and load the byte code into the kernel. You can then attach hooks to your loaded eBPF program to trigger it to run. Those hooks could be things like system calls, kprobes, or uprobes.

There’s many use cases for eBPF. Considering eBPF programs have full system visibility there are very few limits on what you can do. You can write an eBPF program which logs every time certain files are modified. You use eBPF to profile performance of your other programs. You can implement host-based networking rules, or use it for writing malware.

Using Karn for Seccomp Enforcement

Wed, 11 Dec 2019 03:32:37 +0000

Karn aims to provide for Linux what entitlements provide for iOS, or what pledge provides for OpenBSD. It does this by translating a high level set of intuitive ’entitlements’ into complex seccomp profiles.

For example, A developer using Karn can simply specify that their application needs to make network connections, or exec other processes and karn will handle granting it permission to do those things (and nothing else!).

So, how does it work?

The Problem with Sandboxing Solutions

Fri, 06 Dec 2019 00:00:00 +0000

The Problem with Sandboxing Solutions

Mon, 18 Nov 2019 00:00:00 +0000

Linux Procfs

Wed, 02 Oct 2019 03:32:37 +0000

I love the procfs because it provides an intuitive and easy to use interface for interacting with the kernel. In the same way you’d normally use the command line for exploring files, you can control and find out information about currently running processes. All without the need for system calls.

A procfs is mounted at /proc by default. If you go into /proc you’ll see something like this:

You’ll notice a lot of the directories that are just named by numbers. There is one for each individual process. The directory names correspond to the process ID. One directory for each process.

Linux Overlayfs

Wed, 18 Sep 2019 03:32:37 +0000

Remember back to when your teacher in grade school would write on sheets of clear plastic, stack them, and display them with an overhead projector? That is exactly how overlayfs works… except with filesystems.

Overlay filesystems allow you to take multiple directory trees and view them as if they were merged together. The added benefit is that changes in the merged view do not affect the underlying directories. There are many use cases for this, mostly related to containers, but more on that at the end.

Debugging Go tests

Thu, 15 Aug 2019 03:32:37 +0000

This post discusses building go test binaries and walking through them with delve

I recently was working on debugging a unit test I wrote in Go. I couldn’t figure out why one of my test cases was causing a runtime error that never happened when running my actual program. I was using a runtime directive so I suspected there may be some difference between doing a go test and a go run. I was looking through Go build mode documentation while wondering if I could step through it with a debugger. Lo and behold you can compile your tests into an ELF executable!

Standardized Streams and Shells

Tue, 21 May 2019 03:32:37 +0000

In this post we discuss how standardized streams (`stdin`/`stdout`/`stderr`) work on Linux, especially related to shells.

Every well known operating system has a concept of standardized streams. These consist of standard input, output, and error. As their name purports, they are the standard places for proccesses to read input, and send output. They’re a crucial concept that allows various running processes to easily communicate.

Standardized streams enable you to string together multiple commands with a pipe (|) in your shell like this:

Using dynamic libraries in static Go binaries

Tue, 09 Apr 2019 03:32:37 +0000

This post highlights a linker directive in Go. It allows us to import functions from a dynamic library even when CGO is disabled. I use the example of a contribution I recently made to the net and runtime packages to demonstrate its use case.

Go has many little known features that allow you to give instructions to the compiler, linker, and other parts of the toolchain using special comments. Dave Cheney wrote an excellent post on them here. One such ‘pragma’ as Cheney calls them is //go:cgo_import_dynamic. This is a linker directive. It tells the linker to pull in a specific function from a dynamic library such as libc.

The Beauty of io.Writer

Thu, 04 Apr 2019 03:32:37 +0000

In this post we explore best practices of defining interfaces in Go. We use `io.Writer` to break down patterns and antipatterns.

A perfect example of a properly designed Go interface is io.Writer:

type Writer interface {
 Write(p []byte) (n int, err error)
}

It is short, simple, and powerful. The Write method takes a very generic slice of bytes and writes it to something.

In Rob Pike’s Go Proverbs talk he talks about how in Go, interfaces are not declared to be satisfied, they’re satisfied implicitly. An interface should be a way of classifying types, not a blueprint for declaring them. In practical terms, an interface should not at all care about how it’s implemented.

Understanding cgroups

Tue, 20 Nov 2018 03:32:37 +0000

This post will introduce you to cgroups. The goal is to give a comprehensive enough explanation of cgroups and subsystems to broadly understand what they accomplish and how. The major focus will be on the CPU cgroup. The [second section](#using-cgroups) and [third section](#practically-using-cgroups) will explore using cgroups from the command line.

Control groups (or cgroups) are a feature of the Linux kernel by which groups of processes can be monitored and have their resources limited. For example, if you don’t want a google chrome process (or it’s many child processes) to exceed a gigabyte of RAM or 30% total CPU usage, cgroups would let you do that. They are an extremely powerful tool by which you can guarentee limits on performance, but understanding how they work and how to use them can be a little daunting.

Dissecting Go Binaries

Fri, 14 Sep 2018 03:32:37 +0000

This post will introduce you to the basic concepts of disassembly and walk through writing a simple disassembler in Go. The goal is to make the whole concept of disassembly as accessible as possible and hopefully give you, the reader, a takeaway or two about how compiled Go binaries work.

Assembly code cannot lie. By reading assembly we become as close as possible to knowing what’s being run on our computer chips. This is exactly why disassembly is important! If you have a binary with malicious intentions, disassembling it will expose them. If you can’t figure out a performance bottleneck in your code, you can disassemble it for clarity.

Mon, 01 Jan 0001 00:00:00 +0000

Bialys

Bialys are a traditional roll that are fundamental to Ashkenazi Jewsish cuisine. They were a staple at bakeries in New York throughout the 20th century but are hard to find outside of areas with large Ashkenazi populations. Bagels simply get all the glory.

Whereas bagels are supposed to be tough and chewy, Bialys are soft, light, and have an even airy crumb structure. Instead of having a hole, they have a divet in the center which has caramelized onions.

Mon, 01 Jan 0001 00:00:00 +0000

Miami rolls + Challah

Miami rolls are a challah based onion pocket roll. They’re very soft, sweet, flavorful, and delicious. Pretty traditional to jewish bakeries in New York. I worked at a bakery all throughout high school and these were my favorite things there. When they closed down in 2020 I tried to get the recipe but the owner wouldn’t share it with me. Through 15+ iterations I have engineered a recipe that is exactly how I remember them. This recipe is truly worth sharing to keep the tradition of them alive considering so many classic New York bakeries continue to go out of business. I have no clue why they’re named Miami rolls.

Mon, 01 Jan 0001 00:00:00 +0000

Recipes:

Mon, 01 Jan 0001 00:00:00 +0000

Split Pea Soup

Ingredients

2 cups green split peas
7 cups vegetable broth
1 cup coconut milk
1 cup dried potato flakes
2 large yellow onions
2 tbs minced garlic
2 tbs diced fresh ginger
1 bunch of parsely
3 tbs olive oil

1 tbs herbs de provence
2 tbs curry herb blend
1 tbs salt
2 tbs carraway seed

Instructions

Chop the yellow onions. Mince the garlic. Dice the ginger. Chop the parsely.

About Me

Mon, 01 Jan 0001 00:00:00 +0000

My name is Grant Seltzer Richman. I currently work at Datadog. I have worked as a software engineer at Aqua, Oscar Health, Capsule8, and Red Hat. My interests include systems programming, cycling, triathlon, security, video games, baseball, and reading. I mostly will be writing blog posts about linux, security, and various hobbies.

Feel free to reach out if you want to say hi!

Bird Search

Mon, 01 Jan 0001 00:00:00 +0000

A quick tool I use for birding. If I’m searching for a specific species I use this to see where other birders have reported sightings on eBird. It gives all checklists within the specified parameters.

Try it yourself! Get an eBird API Token. Enter your region, look for the bird of your dreams! Birding is fun!