The diagram below is a visual representation of the work ahead, as of the tail end of 2022.

These items add up to multiple decades of work.  I am working on them intermittently at present.

This page will be revised occasionally.

roadmap

Roadmap Themes Over Time

2005-2008: Observability

(in FreeBSD™) I had been trying to figure out what made interpreters “slow” compared to native code when I came across the paper "Continuous Profiling: Where Have All the Cycles Gone?".  This paper described how Digital’s Systems Research Center had used in-CPU hardware counters for system-wide performance measurements.  This appeared to be just the kind of thing I needed, so I wrote PmcTools (i.e., hwpmc(4) , pmcstat(8) & pmccontrol(8)) for FreeBSD — please see my post ‘PmcTools: Motivation and Future Steps’.

My initial commit supported counting-mode access to in-CPU hardware performance counters — sufficient for instrumenting interpreters.[1]  Subsequently, I added support for profiling the kernel and user processes using sampling, for profiling dynamically loaded objects, and for capturing callchains.

In order to implement profiling in PmcTools, I needed a way to “look inside” ELF objects, to map the raw machine addresses captured by hwpmc(4) to source code locations.  Because there were no BSD-licensed libraries available at that time for manipulating ELF objects, I wrote libelf a BSD-licensed implementation of the SVR4 ELF(3) API (FreeBSD commit).

2008-present: Code Sharing

(in Elftoolchain) In 2008, my FreeBSD mentee Kai Wang and I started the Elftoolchain project (SVN-r1).  This project’s goals were:

  • to offer liberally licensed libraries and tools for program development,

  • and to enable sharing of toolchain development effort across the open-source BSD operating system projects.

I also had a long-term goal to write tools to patch running processes to add PmcTools instrumentation (libmc(3), isa(1)) on the fly.

The Elftoolchain project’s code is known to be present in FreeBSD, NetBSD, OpenBSD, RTEMS and Minix3.[2]

The Next Steps: Integration, Refactoring, New Build and Measurement Tools

(in Elftoolchain & NetBSD) The work ahead would fall into the following broad categories:

  1. Improving code quality — improved tests, code reviews, refactoring etc.

  2. Changes to ease integration into BSD base systems: e.g., tooling to automate code imports, making Elftoolchain ELF definitions compatible with the kernel build environment, and so on.

  3. Integration: replacing GNU binutils utilities with their Elftoolchain equivalents.

  4. Improving compatibility with GNU binutils.

  5. New tools and libraries, as described in the next section.

The Work Ahead: Elftoolchain

Improving Quality

Documentation review

Study the guidelines at the Good Docs Project, and revise the project’s documentation accordingly.  This would be an ongoing task.

GNU binutils compatibility review

A periodic check of option and behavior-level compatibility of Elftoolchain’s utilities with their GNU binutils equivalents.

Coverity® Scan

Periodic Coverity® Scan over Elftoolchain code, and fixes for any regressions reported.

Refactoring

Code reviews, refactoring shared code across utilities (e.g., ticket #578), changing the code to work at a higher level of abstraction (#609).

Change Test Framework

The current set of tests were written using the TET test framework from the OpenGroup™.  TET is not BSD-licensed.  Porting the project’s tests to ATF or other BSD-licensed test framework would remove a dependency on non-BSD software (ticket: #270).

elfc(1) rework

elfc(1) is an YAML-based test tool that I wrote that generates ELF objects from textual descriptions.  I would prefer a more descriptive notation than YAML for describing complex ELF objects.

Enhancements

libelf enhancements

Support compressed sections (ticket #594), new APIs (#591), better MIPS64 support (#559), etc.

-lsymtab

Symbol table handling, a common need for many tools.

-lpeg/peg

PEG parser generator for general parsing needs.  Intended to be an easier-to-use alternative to GNU bison(1), although my real motivation here is to understand how PEG parsers work.

Configuration/data notation

A notation to describe data, for use with isa(1), elfc(1) etc.

-ldemangler

Correct C++ demangling offered as a standalone library, driven off a formal grammar (ticket #595).

isa(1)

A tool to describe machine instruction sets.

ld(1) refactor, enhance

Use PEG-based parsing instead of bison(1)-generated parsers for parsing linker scripts.  This removes a dependency on GNU bison(1).  PEG-based parsers are usually also easier to read.

as(1)

A machine assember, an as(1) replacement.

libmc

Machine code parsing.

-lryu (Floating point printing)

Use (or implement afresh) a correct floating point to text converter for displaying floating point quantities. See "Ryū: fast float-to-string conversion", Ulf Adams, PLDI 2018.

-ltask

A library for managing tasks with inter-task dependencies.  This library would be useful for tools that need to sequence their actions.

Build in src/

Changes to ease integration of Elftoolchain code into the NetBSD® base system.

Update libelf/libdwarf

Update to a recent Elftoolchain revision.

Kernel Build

Integrating Elftoolchain’s definitions into NetBSD kernel source.

Incorporate Utilities

Add a WITH_ELFTOOLCHAIN (or similar) build knob and incrementally add Elftoolchain components to the NetBSD base system.

Add tests to src/

This is to allow imported Elftoolchain tests to be tested along with the other tests present in /usr/tests in NetBSD.

Infrastructure

Change Review

Deployment of a change review tool for pre-commit reviews.

Code Browser

A Trac-like code browser, but one that understands both Trac and Allura markup styles.

Trial builds

A way for changes to be built and tested across all of a project’s OSes prior to commit.

Build Tool

This is a tool that takes the best ideas of Bazel, Buck and similar build tools and adapts these for (cross-)building a \*BSD operating system, while allowing sharing of build cycles across developers.

Please see: BuildAutomation (Elftoolchain Wiki).

Auditable Builds

This is an enhancement to the build tool to allow end-users to cryptographically verify that:

  1. A given binary (or kernel, or dynamically loadable object) was built from a set of vetted sources …​

  2. using a toolchain that was built in turn from vetted sources …​

  3. and which ran on a similarly vetted kernel while building the binary …​

  4. and that every byte of code or data the final executable or dynamically loaded object can be accounted for.

Please get in touch if this sounds interesting!

The Work Ahead: PmcTools

Implementation Portability

Port the kernel module for PmcTools (i.e. hwpmc(4)) to other \*BSD kernels.

BSD project kernels share a common lineage, but their implementation has diverged over the years. Kernel code is consequently no longer portable ‘as-is’ between these kernels. Nevertheless, it seems worthwhile to define the ‘surface of interaction’ between hwpmc(4) and the rest of the kernel in an kernel-agnostic way.  Doing so would bring the following benefits:

  1. Tightly defining the interfaces used by the revised kernel module paves the way for developing and testing it in userspace, akin to how NetBSD’s Rump Kernel framework allows drivers to be written and tested outside the kernel.  We then get:

    • easier fuzz-testing,

    • the ability to profile the driver using ‘native’ PmcTools,

    • the ability to study its behavior in an instruction simulator,

    • and so on.

  2. Rewriting the code in this way should make it easier to port the module to microkernels like Minix3 and OSes like OpenHarmony.

    There is little value, IMHO, in having every open-source OS project maintain their own performance analysis tools.

Compatibility With Emerging Standards

Standards for file formats and observability are being actively discussed in the open-source community (e.g. the Common Trace Format from the DiaMon working group, and the observability APIs being defined by the Open Telemetry project).

Being compatible with these emerging standards allows users to use their favorite visualization and analysis tools with PmcTools’ output.

Reuse of Functionality

Split up the implementation of PmcTools, particularly the kernel driver (hwpmc(4)), into independently reusable (and independently testable) modules.  Specifically:

  1. The functionality that captures callchains is useful in other contexts.  This functionality could be made accessible to other parts of the kernel, and perhaps even to userspace.

  2. The functionality that captures the program counter on a performance counter interrupt could be generalized to capturing additional aspects of ‘system state’ — please see the section on User Extensibility below.

  3. The functionality that writes samples to a log could be reused to support system tracing.

  4. We could allow the capture of ‘system state’ to be triggered by subsystems other than in-CPU hardware performance counters.  This would allow us to sample system state based on behavior visible to off-CPU counters (such as the performance counters in some motherboard chipsets).  We could even connect up a power meter that raises an interrupt for every ‘X’ millijoules of energy consumed and directly profile a system for its power use.

User Extensibility

Allow a performance counter event to trigger the execution of user-supplied scripts.

This facility would be akin to D scripts in Solaris and EBPF scripts in the Linux kernel.

Some BSD kernels already offer scriptability today, e.g., Lua-based scripting in NetBSD.  However, using WebAssembly (WASM) would allow users a wider choice of scripting languages, since many languages now compile to WASM byte code.

We would need to make PmcTools’ log format user-extensible, so that these scripts can add their own data to the log.  It also goes without saying that this feature would need to be designed from the start for security and reliability.

Support Asymmetric Multiprocessing

Support multi-processor systems using multiple classes of CPUs.

Examples of such systems are those using ARM®'s big.LITTLE CPUs.  The individual CPUs in such systems could have differing performance counter implementations capable of monitoring differing subsets of hardware events.

Many aspects of the PmcTools toolkit would need to change in order to support such systems: for example, the user API, the kernel driver and the documentation.

Additional Hardware Counter Classes

  • Support hardware-assisted branch tracing.

  • Support chainable/conditional performance counters present in a few modern CPUs.

  • Support off-CPU performance counters, such as those in motherboard chipsets, package-level counters, and counters in peripherals.

Support Standard System Metrics

Extend the approach taken by Fabien Thomas’s pmc.soft(3) functionality to make additional system information through PmcTools’s APIs.

Whole-system performance analyses would be made easier if the standard system metrics (e.g. CPU time, network statistics, power rail information, etc.) could be made available alongside the information from hardware performance counters.

Integration With Standard Profiling Tools

Integrate PmcTools with standard profiling tools (i.e., cc -pg).

PmcTools was designed from the outset to allow cheap userspace access to per-process counters.  This feature allows a profiling runtime (e.g. _mcount) to profile applications against their behavior on hardware.

Tracing Support

Augment system traces with hardware and software counts.

This would allow developers to map the behavior of hardware to higher-level constructs in their code (e.g., associating a trace span for database transaction with a count of memory bus cycles seen).

Improve the PMC Configuration Language

PMCs are currently configured using a comma-separated list of name/value pairs that are translated to the configuration values expected by the hardware.  Extend this to support some of the more sophisticated PMC features in newer CPUs, such as chainable hardware counters and conditional counting.

While here improve error reporting for invalid PMC configurations.

Governance Structures

Set up a way to give the relevant stakeholders a voice in the future of the code.

This would be important for long-term sustainability.

Carry-overs From PmcTools v1

I am inclined to preserve the following:

  1. The basic life-cycle of a PMC: i.e. allocation, use, release.

  2. Keeping PMC handles distinct from file descriptors.

  3. The notion of PMC scopes (process vs system).

  4. The notion of PMC operational modes (sampling vs counting).

  5. The ability to cheaply read process-scope PMCs from userspace (using RDPMC on x86, or its equivalent on other platforms).

  6. Logging traces to a file descriptor.

  7. Comprehensive documentation for the operation of the system.

Future

PmcTools + Elftoochain

Use libmc (the planned library for machine code parsing/process instrumentation) for writing utilities to instrument running processes, etc.

Profile Guided Layout

Profile guided layout of object code, to improve its cache behavior at runtime.

Profiling Code For Energy Use

(Research project) As mentioned in passing in my 2009 ACM Bangalore presentation (slide 46), it should be possible to profile code for energy use with a reasonable level of accuracy, even on stock hardware.  The applications for such a profiler are varied — for example, such a profiler could help schedule workloads with high energy usage at the times where energy is cheap, or it could find code ‘hot spots’ (no pun intended) that consume significant energy in embedded devices, and so on.  If this kind of thing interests you, please get in touch.

PmcTools-aware self profiling

Replace _mcount() (used by cc -pg) with a version that uses the RDPMC instruction (or equivalent) to accumulate PMC values instead of call counts.

1. Please see hwpmc(4) for more information on the counting-mode PMCs.
2. Interestingly, Minix3 may well be one of the world’s most widely used OSes, because it is apparently present (embedded) in the ‘management engine’ in recent Intel® CPUs.