Greg Kurtzer's Ginux Is Now a Userspace Kernel: Architecture, Syscall Brokering, and Production Trade-offs

Greg Kurtzer's fingerprints are on some of the most widely deployed infrastructure in high-performance computing: Warewulf for stateless provisioning, Singularity (now Apptainer) for unprivileged HPC containers, and Rocky Linux as the post-CentOS rebuild the community rallied behind. All three share a design instinct — take a privileged capability away from the user and hand it to a trusted supervisor that does the dangerous part on their behalf.

Ginux is the logical terminus of that instinct. It has moved from containing a process to being the kernel that process talks to. That is a categorically different engineering problem, and it is why the project lit up Hacker News. This analysis walks the architecture, the syscall interposition machinery, the memory and I/O fast paths, and the honest performance ledger you inherit when you adopt it.

Scope note: the design described here reflects the published architecture and public discussion. Where a detail is still in motion, it is flagged explicitly rather than smoothed over. Always validate against upstream before you design around it.

Why Turn a Container Runtime Into a Kernel?

The traditional container stack has a structural weakness: runc plus namespaces plus cgroups gives you isolation by configuration. The container's payload still exercises the host kernel's full syscall surface, and the only thing standing between a compromised workload and a privilege escalation is whether the kernel's own checks held. Every CVE in the syscall layer is your CVE.

The userspace-kernel answer is to invert that: make the payload's syscalls land in your code. Now the attack surface is whatever your broker implements — typically a few hundred syscalls instead of ~400 with years of accumulated driver-adjacent cruft. This is the same thesis gVisor's Sentry has pursued, and it is the thesis Ginux now implements, but with an HPC-shaped bias toward unprivileged operation, no daemon root requirement, and predictable data-plane throughput rather than cloud multi-tenancy.

The Ginux Architecture at 30,000 Feet

The system splits cleanly into a control plane and a data plane, and almost every performance or security property falls out of where that line sits.

┌──────────────────────────────────────────────────────────────┐
│  Host kernel (Linux)                                          │
│  seccomp-bpf  ·  userfaultfd  ·  io_uring  ·  cgroups v2      │
└────────────┬─────────────────────┬──────────────────────────┘
             │ notifications       │ shared rings
┌────────────▼─────────────────────▼──────────────────────────┐
│  Ginux userspace kernel                                       │
│  ┌────────────┐  ┌───────────┐  ┌───────────┐  ┌──────────┐ │
│  │  Syscall   │  │  Memory   │  │    VFS    │  │  Netdev  │ │
│  │  broker    │  │  manager  │  │  layer    │  │  (eBPF)  │ │
│  └────────────┘  └───────────┘  └───────────┘  └──────────┘ │
│          scheduler · capability model · audit ring           │
└────────────┬─────────────────────────────────────────────────┘
             │ trapped syscalls
┌────────────▼─────────────────────────────────────────────────┐
│  Unmodified payload (ELF, script, MPI rank, Python worker)    │
└──────────────────────────────────────────────────────────────┘

Control Plane vs Data Plane

The control plane is the broker: accept a syscall notification, decide, reply. It is latency-bound and sensitive to scheduler wakeup cost. The data plane is everything that must not round-trip through the broker — bulk file I/O, network packets, page faults. Ginux pushes those onto shared-memory rings so the broker only handles setup, not steady-state traffic.

Syscall Interposition: The Heart of the Design

There are three viable mechanisms for intercepting a foreign process's syscalls, and the choice dominates the entire performance profile.

Why SECCOMP_RET_USER_NOTIF Beats ptrace

ptrace gives you PTRACE_SYSCALL, which stops the tracee twice per call. Each stop is a signal delivery plus two scheduler transitions, and the tracee becomes unkillable-by-design in awkward ways while stopped. It is also historically a privilege-escalation magnet.

seccomp with SECCOMP_RET_USER_NOTIF (Linux 5.0+, listener fd via SECCOMP_FILTER_FLAG_NEW_LISTENER) is a different animal: no signals, no SIGSTOP, no priority inversion, and the notification is a plain blocking ioctl your broker goroutine can sit in. Add SECCOMP_IOCTL_NOTIF_ADDFD (5.9+) and you can inject file descriptors into the tracee's table, which is the thing ptrace never did cleanly.

// Package ginux implements the syscall broker that gives the userspace
// kernel authority over a sandboxed payload.
package ginux

import (
	"fmt"
	"runtime"
	"unsafe"

	"golang.org/x/sys/unix"
)

// Mirrored from include/uapi/linux/seccomp.h so the broker compiles against
// x/sys/unix releases that predate these constants.
const (
	seccompSetModeFilter        = 0x1
	seccompFilterFlagNewListener = 1 << 3
	seccompRetUserNotif         = 0x7fc00000

	ioctlNotifRecv    = 0xC0502100 // _IOWR('!', 0, struct seccomp_notif)
	ioctlNotifSend    = 0xC0182101 // _IOWR('!', 1, struct seccomp_notif_resp)
	ioctlNotifIDValid = 0x40082102 // _IOW('!', 2, __u64)
	ioctlNotifAddFD   = 0xC0042103 // _IOWR('!', 3, struct seccomp_notif_addfd)
)

// InstallBroker traps every syscall from the calling thread and returns the
// listener fd the supervisor should poll. Must be called in the child, after
// fork/exec but before handing control to untrusted code.
func InstallBroker() (int, error) {
	filter := []unix.SockFilter{{
		Code: unix.BPF_RET | unix.BPF_K,
		K:    seccompRetUserNotif,
	}}
	prog := &unix.SockFprog{Len: uint16(len(filter)), Filter: &filter[0]}

	// Irreversible: no setuid binary may widen privileges after this point.
	if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil {
		return -1, fmt.Errorf("prctl(PR_SET_NO_NEW_PRIVS): %w", err)
	}

	// seccomp state is per-thread. The tracee must stay pinned to this OS thread.
	runtime.LockOSThread()
	defer runtime.UnlockOSThread()

	fd, _, errno := unix.Syscall(unix.SYS_SECCOMP,
		uintptr(seccompSetModeFilter),
		uintptr(seccompFilterFlagNewListener),
		uintptr(unsafe.Pointer(prog)),
	)
	if errno != 0 {
		return -1, fmt.Errorf("seccomp(SET_MODE_FILTER): %w", errno)
	}
	return int(fd), nil
}

The Broker Loop, Done Correctly

The subtlety nobody mentions in the tutorial posts: a notification can go stale between receive and reply. The tracee may have been killed, or it may have re-entered the syscall after an interrupted signal. You must validate the notification ID before acting, or you will open the wrong file on behalf of a recycled task.

func (b *Broker) Serve() error {
	for {
		req := unix.SeccompNotif{}
		if err := ioctlPtr(b.listener, ioctlNotifRecv, unsafe.Pointer(&req)); err != nil {
			if errors.Is(err, unix.EINTR) || errors.Is(err, unix.ENOENT) {
				continue
			}
			return fmt.Errorf("notif recv: %w", err)
		}

		// TOCTOU guard: cheap, mandatory, and the #1 thing reviewers miss.
		id := req.ID
		if err := ioctlPtr(b.listener, ioctlNotifIDValid, unsafe.Pointer(&id)); err != nil {
			continue // tracee is gone; drop silently
		}

		resp := unix.SeccompNotifResp{ID: req.ID}
		switch req.Data.Nr {
		case unix.SYS_OPENAT:
			// You cannot return a raw fd number. It must be injected into the
			// tracee's descriptor table; the kernel picks the number.
			f, err := b.vfs.OpenAt(int(req.Data.Args[0]), req.Data.Args[1], req.Data.Args[2])
			if err != nil {
				resp.Error = -int32(errnoOf(err))
				break
			}
			guestFD, err := b.addFD(req.ID, f, seccompADDFdFlagsCloseOnExec)
			if err != nil {
				resp.Error = -int32(unix.EMFILE)
				break
			}
			resp.Val = int64(guestFD)
		default:
			resp.Error = -int32(unix.ENOSYS)
		}

		if err := ioctlPtr(b.listener, ioctlNotifSend, unsafe.Pointer(&resp)); err != nil {
			return fmt.Errorf("notif send for syscall %d: %w", req.Data.Nr, err)
		}
	}
}

Memory: userfaultfd-Driven Demand Paging

The broker cannot be in the path of every page fault — that would be fatal. Ginux registers guest address ranges with userfaultfd in MISSING mode and serves faults from a fault-handler thread that never touches the syscall broker at all. The backing image is immutable and shared across sandboxes, so the common case is a UFFDIO_COPY from the page cache with zero decode work.

#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <linux/userfaultfd.h>
#include <poll.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <unistd.h>

static int uffd_create(void) {
    int fd = (int)syscall(SYS_userfaultfd, O_CLOEXEC | O_NONBLOCK);
    if (fd < 0) return -1;
    struct uffdio_api api = { .api = UFFD_API, .features = 0 };
    if (ioctl(fd, UFFDIO_API, &api) < 0) { close(fd); return -1; }
    return fd;
}

static int region_register(int uffd, char *base, size_t len) {
    struct uffdio_register reg = {
        .range = { .start = (unsigned long)base, .len = len },
        .mode  = UFFDIO_REGISTER_MODE_MISSING,
    };
    if (ioctl(uffd, UFFDIO_REGISTER, &reg) < 0) return -1;
    if (!(reg.ioctls & (1UL << _UFFDIO_COPY))) { errno = ENOTSUP; return -1; }
    return 0;
}

/* One fault handler thread per region. Never call this from the broker. */
static void *serve_faults(void *arg) {
    struct region *r = arg;
    for (;;) {
        struct pollfd pfd = { .fd = r->uffd, .events = POLLIN };
        if (poll(&pfd, 1, -1) <= 0) continue;

        struct uffd_msg msg;
        if (read(r->uffd, &msg, sizeof msg) != sizeof msg) continue;
        if (msg.event != UFFD_EVENT_PAGEFAULT) continue;

        unsigned long fault = msg.arg.pagefault.address & ~(r->page - 1);
        struct uffdio_copy cp = {
            .dst  = fault,
            .src  = (unsigned long)(r->base + (fault - (unsigned long)r->base)),
            .len  = r->page,
            .mode = 0,
        };
        if (ioctl(r->uffd, UFFDIO_COPY, &cp) < 0 && errno != EEXIST) {
            /* EEXIST means another thread won the race: benign. */
        }
    }
    return NULL;
}

The important architectural point: UFFDIO_COPY also supports the UFFDIO_COPY_MODE_DONTWAKE flag, which lets Ginux batch wakeups — one UFFDIO_WAKE per fault batch rather than per page. That single change is often worth double-digit percentages on multi-threaded startup.

The I/O Fast Path: io_uring and Shared Rings

For a userspace kernel, the naive design — broker every read(2) — collapses immediately. Ginux routes bulk block I/O through an io_uring instance in the guest's own memory, with the broker acting only as the ring's provisioner. The guest submits directly; the kernel-side completion path never touches a syscall notification.

#include <liburing.h>

#define QD 256

int vbd_submit_read(struct io_uring *ring, int imgfd, void *buf, size_t len, off_t off)
{
    struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
    if (!sqe) return -EAGAIN;                 /* ring full: push backpressure upstream */

    io_uring_prep_read(sqe, imgfd, buf, len, off);
    io_uring_sqe_set_data64(sqe, (uint64_t)buf); /* opaque completion tag */

    int ret = io_uring_submit(ring);
    return ret < 0 ? ret : 0;
}

int vbd_reap(struct io_uring *ring, void **buf_out, int *err_out)
{
    struct io_uring_cqe *cqe;
    int ret = io_uring_wait_cqe(ring, &cqe);
    if (ret < 0) return ret;

    *buf_out = (void *)io_uring_cqe_get_data64(cqe);
    *err_out = cqe->res;                       /* short read or -errno */
    io_uring_cqe_seen(ring, cqe);
    return 1;
}

Networking: eBPF, Not Proxies

The same principle applies on the wire. A userspace TCP stack in the broker is correct but slow; Ginux pairs the ring buffer with AF_XDP for the zero-copy path and falls back to vhost-user-style shared memory for peer sandboxes on the same node. eBPF programs enforce the policy at the socket layer, so filtering stays O(1) per packet instead of O(calls) through the broker.

Scheduling and the Interposition Tax

A userspace kernel does not own the CPU — it borrows it. Ginux uses a hybrid model: a small number of pinned host threads run a cooperative scheduler for the guest's green threads, and any syscall that must block is handed to the broker, which blocks the host thread and returns a -EAGAIN-equivalent to the guest runtime.

That means your workload's concurrency model interacts with the interposition layer. Go workloads with a large GOMAXPROCS will pin threads aggressively and increase contention on the notification fd; a single-threaded Python worker will barely notice. Budget for measuring this, not guessing.

Security Posture: Smaller Blast Radius, Bigger Broker

Ginux's security model has one genuinely strong property and one genuinely dangerous one.

Strong: the payload cannot reach a syscall the broker does not implement. Deny-by-default is structural, not policy-configured. Capability dropping happens in the supervisor, inside a new user namespace:

func DropToPayloadIdentity(uid, gid uint32) error {
	// Caller has already unshared CLONE_NEWUSER; CAP_SETUID only exists there.
	if err := unix.Setgroups([]int{}); err != nil {
		return fmt.Errorf("setgroups: %w", err)
	}
	if err := unix.Setresgid(int(gid), int(gid), int(gid)); err != nil {
		return fmt.Errorf("setresgid: %w", err)
	}
	if err := unix.Setresuid(int(uid), int(uid), int(uid)); err != nil {
		return fmt.Errorf("setresuid: %w", err)
	}

	// All-zero data clears effective, permitted, inheritable and bounding sets.
	hdr := unix.CapUserHeader{Version: unix.LINUX_CAPABILITY_VERSION_3, Pid: 0}
	data := [2]unix.CapUserData{}
	if err := unix.Capset(&hdr, &data[0]); err != nil {
		return fmt.Errorf("capset: %w", err)
	}
	if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil {
		return fmt.Errorf("no_new_privs: %w", err)
	}
	return nil
}

Dangerous: the broker is now a single, monolithic, syscall-facing surface with the privileges of the supervisor. A path-traversal bug in your OpenAt emulation is a host compromise, not a sandbox escape. Treat the broker as Tier-0 code: fuzz it, review it, and never let it run with more capability than the payload legitimately needs.

Performance Reality Check

Order-of-magnitude figures, measured on modern x86-64; treat them as a starting hypothesis, not a benchmark.

The actionable conclusion: keep the syscall rate low, not the syscall cost. strace -c your payload before you adopt Ginux. A workload doing 50k syscalls/sec will lose roughly 10–15% CPU on brokering; a workload doing 500 syscalls/sec will lose noise-level amounts. Vectorized HPC solvers and gRPC servers sit in completely different risk categories.

Operational Checklist Before You Ship Ginux

• Pin your kernel version. SECCOMP_IOCTL_NOTIF_ADDFD landed in 5.9; SECCOMP_IOCTL_NOTIF_ID_VALID earlier. Mixed node fleets will behave differently.
• Verify `CONFIG_USERFAULTFD` and `vm.unprivileged_userfaultfd`. On many distro kernels this is 0, and your fault handler silently fails to start.
• Set `RLIMIT_MEMLOCK` explicitly. io_uring rings and AF_XDP UMEM are locked memory; the default soft limit of 64 KiB will bite you at QD 256.
• Run the broker in its own cgroup v2 slice. When a payload misbehaves, you want the broker to survive the OOM killer's first pass.
• Instrument the notification fd. Export recv-to-send latency as a histogram. Broker tail latency is your new p99.
• Audit `pidfd` lifecycle. Use pidfd_open/pidfd_send_signal rather than raw PIDs; a recycled PID during teardown is a classic cross-sandbox signal injection bug.
• Fuzz the VFS emulation independently. It is the highest-value target in the entire system.

Pitfalls That Will Bite You

Trusting the notification payload's pointers. req.Data.Args[] are guest virtual addresses. Reading them from the broker without process_vm_readv and a validity window is a TOCTOU hole. Always re-validate after ID_VALID.

Forgetting that seccomp is per-thread. If the payload creates threads before the filter is installed, those threads are unfiltered. Install in the child after exec boundaries, or use SECCOMP_FILTER_FLAG_TSYNC.

Assuming `UFFDIO_COPY` wakes the waiter. It does by default — which is why batching with DONTWAKE requires an explicit matching UFFDIO_WAKE. Get this wrong and threads hang in D state with no diagnostics.

Benchmarking with warm caches. The whole point of demand paging is that the first fault is expensive. Measure cold start separately from steady state or you will report a number that never occurs in production.

Choosing Between Ginux, gVisor, Kata, and runc

Ginux's differentiator is not raw isolation strength — Kata wins on that — it is operating without daemon root and without hardware virtualization. On an HPC login node where users cannot be granted CAP_SYS_ADMIN and nested virt is unavailable, that combination is the entire argument.

Sources & Technical References

• Linux kernel documentation — seccomp filter / user notification
• seccomp(2) manual page — man7.org
• Linux kernel documentation — Userfaultfd
• Linux kernel documentation — io_uring
• Linux kernel documentation — AF_XDP
• Linux kernel documentation — BPF and XDP reference guide
• Linux kernel documentation — KVM API
• Linux kernel documentation — User Mode Linux HOWTO
• Linux kernel documentation — Control Group v2
• capabilities(7) manual page — man7.org
• namespaces(7) manual page — man7.org
• pidfd_open(2) manual page — man7.org
• gVisor — Architecture Guide
• Kata Containers — project documentation
• Firecracker — microVM design documentation
• Apptainer — User Guide (Greg Kurtzer's container lineage)
• Rocky Linux — official documentation
• DPDK — Programmer's Guide
• Saltzer, J. H. & Schroeder, M. D., The Protection of Information in Computer Systems, Proceedings of the IEEE, 1975 — doi:10.1109/PROC.1975.9939
• Bershad, B. N. et al., The Case for Lightweight Remote Procedure Call, ACM TOCS, 1990 — foundational analysis of the user/kernel boundary crossing cost modeled above.