Wander Lairson Costa

Modernizing stalld with eBPF

Fri, 07 Nov 2025 00:00:00 +0000

Last year I started working on stalld, a daemon that prevents thread starvation in real-time Linux systems. On systems with isolated CPUs running high-priority RT tasks, essential kernel threads can starve indefinitely. stalld monitors CPU run queues and temporarily boosts starving threads using SCHED_DEADLINE (or SCHED_FIFO as fallback).

The main goal was switching from the old sched_debug backend to a new queue_track backend. The old approach reads /sys/kernel/debug/sched/debug to get task information — file I/O, parsing, relatively high overhead. The new approach uses BPF programs to track tasks in real-time. Lower overhead, better precision, but a complete rewrite of the core monitoring logic.

This post covers what I learned along the way: eBPF verifier fights, kernel portability headaches, race conditions, and production bugs.

eBPF and the verifier

Building the queue_track backend meant diving deep into eBPF. The old sched_debug backend had to read and parse text files on every iteration. With BPF, I track tasks in the kernel as events happen and aggregate the results before copying to userspace. The overhead reduction is significant, especially on systems with many CPUs¹.

The eBPF verifier presents its own challenges. It exists for good reasons — arbitrary code in the kernel would be a security nightmare. But when it rejects a program, the error messages can be difficult to diagnose.

I encountered this with logging. On some kernel versions, the verifier refused to load the program with “Argument list too long”. The fix was dropping one parameter from the logging macro:

// Original: rejected by verifier on some kernels
log_task_prefix("enqueue", p->comm, p->pid, p->tgid, "rt", cpu);
// Fixed: passes verification
log_task_prefix("enqueue", p->comm, p->pid, p->tgid, cpu);

With eBPF, writing correct code is not enough. The code must also convince the verifier it is correct. And verifier behavior changes between kernel versions.

Then there is the kernel itself changing under you. Data structures evolve. Fields move or disappear entirely. The cpu field in struct task_struct was removed in recent kernels.

This is where BPF CO-RE becomes essential. You define legacy struct variants, use bpf_core_field_exists() to check at runtime what fields exist, and adapt accordingly:

struct task_struct___legacy {
    int cpu;
};

static __always_inline int task_cpu(const struct task_struct *p)
{
    const struct task_struct___legacy *legacy = (void *)p;

    if (bpf_core_field_exists(legacy, cpu))
      return BPF_CORE_READ(legacy, cpu);

    return BPF_CORE_READ(p, thread_info.cpu);
}

I used this pattern again for __state vs state when checking if tasks are running. Once you’ve written the adaptation logic, the BPF loader does the rest. Different kernels, same binary.

Tracepoints

The queue_track backend needed a way to hook into the scheduler and track task state changes. Early on, I used fentry probes on scheduler functions like enqueue_task_rt() and dequeue_task_rt(). This worked in testing, but fentry requires a synchronize_rcu_tasks() call during attachment. On a busy system with a high-priority RT task monopolizing a CPU, that call can block indefinitely.

The result was ironic: the daemon designed to prevent starvation was itself getting starved during startup.

// fentry: requires synchronize_rcu_tasks(), can block
SEC("fentry/enqueue_task_rt")
int BPF_PROG(enqueue_task_rt_enter, struct rq *rq, struct task_struct *p)

// tracepoint: no blocking synchronization required
SEC("tp_btf/sched_wakeup")
int BPF_PROG(handle__sched_wakeup, struct task_struct *p)

Moving to tracepoints resolved the issue. Tracepoints are stable kernel APIs and do not require blocking synchronization. Startup became immediate, even on heavily loaded systems.

However, I then discovered incomplete task tracking. Tasks do more than wake up and sleep — they migrate between CPUs, get created, and exit. Missing any of these events causes stalld to accumulate stale entries (tasks that no longer exist on a given CPU) and lose track of tasks that migrated.

I added handlers for the remaining scheduler events. sched_wakeup_new catches newly created tasks from their first wakeup. sched_migrate_task handles CPU migration — dequeue from the source CPU and re-enqueue on the destination:

SEC("tp_btf/sched_migrate_task")
int BPF_PROG(handle__sched_migrate_task, struct task_struct *p, int dest_cpu)
{
    int src_cpu = task_cpu(p);

    if (dequeue_task(src_cpu, p))
        enqueue_task(dest_cpu, p);

    return 0;
}

The most subtle problem? stalld only tracked tasks that changed state after it started. Anything already on a runqueue was invisible until it rescheduled. The fix was a BPF task iterator that walks all tasks at startup:

SEC("iter/task")
int dump_task(struct bpf_iter__task *ctx)
{
    struct task_struct *task = ctx->task;
    if (!task)
        return 0;

    if (task_is_running(task))
        enqueue_task(task_cpu(task), task);

    return 0;
}

Now stalld has a complete system snapshot from the moment it starts.

Bug fixes

Some bugs manifest clearly. Consider this scenario: a task starts starving, stalld detects it and attempts to boost it, but the task exits before the boost completes. The get_current_policy() syscall fails because the task no longer exists. However, stalld was not cleaning up the stale entry, causing it to retry the same dead PID on every iteration:

[stalld] Failed to get policy for PID 12345
[stalld] Failed to get policy for PID 12345
[stalld] Failed to get policy for PID 12345
...

The fix was straightforward once identified. If the syscall fails, the task has likely exited. Remove the entry and continue:

if (get_current_policy(task->pid, &policy) < 0) {
    cleanup_starving_task_info(task);
    continue;
}

if (boost_with_deadline(task->pid, &policy) < 0) {
    cleanup_starving_task_info(task);
    continue;
}

Other bugs were more subtle. The merge_taks_info() function should clear the starving vector and rebuild it on every iteration. However, it only cleared the vector when there were zero new tasks. If new tasks existed but none were starving, the vector retained stale data. stalld would then boost tasks that were no longer starving:

// Bug: only clears when no new tasks
if (new_nr_rt_running == 0) {
    update_cpu_starving_vector(cpu, 0);
}

// Fix: unconditionally clear before rebuilding
update_cpu_starving_vector(cpu, 0);

A one-line fix, but finding it required understanding the state machine.

DL-server integration

Recent kernels include the DL-server, which provides built-in starvation handling. This raises the question: is stalld obsolete?

The answer is no. stalld is now “DL-server aware.” It monitors tasks system-wide but avoids interfering with tasks that DL-server handles. Instead, stalld addresses what DL-server cannot: starving SCHED_FIFO tasks, and starving tasks on CPUs where DL-server was manually disabled. The two mechanisms complement each other.

My initial approach was conservative. If the kernel has DL-server, stalld would switch to log-only mode automatically — observe and log, but do not boost. This avoided potential conflicts and deferred to the built-in solution. Later, I refined this to the complementary approach described above.

Summary

The migration from sched_debug to queue_track is complete. Instead of parsing debugfs files, stalld now tracks tasks in real-time with BPF. Lower overhead, better data, and it scales better on systems with lots of CPUs.

Multiple bugs resolved along the way. Four major eBPF improvements: tracepoint migration, task iterator, CO-RE portability, and verifier compatibility.

Going deeper

1: Tested on kernels 5.14 and later. The queue_track backend requires BPF CO-RE support, available in kernels 5.2+ with appropriate libbpf versions.

Debugging the Linux Kernel through tracing

Mon, 24 Jun 2024 00:00:00 +0000

Disclaimer: this is a mental note in the form of a blog post for future references. Most of it is a summary of a debug session to solve this issue.

Kernel is one of hardest piece of software to debug. Although it has some limited support for a classical debug session, it is often impractical. Not surprisingly, over the years, kernel hackers developed a lot of tools to aid the art of debugging the Linux kernel. Tracing tool are among the most useful techniques.

Tracing is a technique used to monitor the live kernel in real-time, involving a logging mechanism to record kernel activity. In this post, we will provide an overview of two kernel features that enable tracing: tracepoints and kprobes. We’ll interface with these through the trace-cmd command, though alternatives like bpftrace can also be used.

Tracepoints

A tracepoint is a predefined static hook to call a function (probe) that allow tracing tools to log system activity. Tracepoints are listed inside the category events in the tracing filesystem, under sys/kernel/tracing/events/. You can also list them using the trace-cmd list command:

$ trace-cmd list -e | head
qrtr:qrtr_ns_service_announce_new
qrtr:qrtr_ns_service_announce_del
qrtr:qrtr_ns_server_add
qrtr:qrtr_ns_message
sunrpc:rpc_xdr_sendto
sunrpc:rpc_xdr_recvfrom
sunrpc:rpc_xdr_reply_pages
sunrpc:rpc_clnt_free
sunrpc:rpc_clnt_killall
sunrpc:rpc_clnt_shutdown

Let’s use trace-cmd to trace timer:hrtimer_expire_entry, which triggers every time an hrtimer callback executes:

trace-cmd record -e timer:hrtimer_expire_entry sleep 0.01
CPU0 data recorded at offset=0x2a5000
    197 bytes in size (4096 uncompressed)
CPU1 data recorded at offset=0x2a6000
    212 bytes in size (4096 uncompressed)
CPU2 data recorded at offset=0x2a7000
    132 bytes in size (4096 uncompressed)
CPU3 data recorded at offset=0x2a8000
    163 bytes in size (4096 uncompressed)
CPU4 data recorded at offset=0x2a9000
    211 bytes in size (4096 uncompressed)
CPU5 data recorded at offset=0x2aa000
    209 bytes in size (4096 uncompressed)
CPU6 data recorded at offset=0x2ab000
    189 bytes in size (4096 uncompressed)
CPU7 data recorded at offset=0x2ac000
    189 bytes in size (4096 uncompressed)

trace-cmd record captures events while the given command runs. In this example, we ran sleep 0.01 to capture events for just 10ms, as hrtimer generates many entries.

The logs are saved in a binary file named trace.dat. To convert these logs to a text format, use trace-cmd report:

trace-cmd report | head 
cpus=8
          <idle>-0     [002] 21403.059823: hrtimer_expire_entry: hrtimer=0xffffacffe3163cf8 now=21402099965347 function=hrtimer_wakeup/0x0
          <idle>-0     [007] 21403.059855: hrtimer_expire_entry: hrtimer=0xffffacffe0fcfb20 now=21402099998095 function=hrtimer_wakeup/0x0
           Timer-34807 [002] 21403.059858: hrtimer_expire_entry: hrtimer=0xffff90d2cc326158 now=21402100001643 function=tick_nohz_handler/0x0
 IPDL Background-4695  [003] 21403.059859: hrtimer_expire_entry: hrtimer=0xffff90d2cc3a6158 now=21402100001860 function=tick_nohz_handler/0x0
           sleep-45035 [001] 21403.059859: hrtimer_expire_entry: hrtimer=0xffff90d2cc2a6158 now=21402100001874 function=tick_nohz_handler/0x0
 Isolated Web Co-29032 [004] 21403.059859: hrtimer_expire_entry: hrtimer=0xffff90d2cc426158 now=21402100002003 function=tick_nohz_handler/0x0
          <idle>-0     [006] 21403.059861: hrtimer_expire_entry: hrtimer=0xffff90d2cc526158 now=21402100004124 function=tick_nohz_handler/0x0
          <idle>-0     [005] 21403.059863: hrtimer_expire_entry: hrtimer=0xffff90d2cc4a6158 now=21402100006647 function=tick_nohz_handler/0x0
    dav1d-worker-44563 [000] 21403.059864: hrtimer_expire_entry: hrtimer=0xffff90d2cc226158 now=21402100006686 function=tick_nohz_handler/0x0

The logs are dominated by the tick_nohz_handler function, but we can filter it out using the -f option:

$ grep -w tick_nohz_handler /proc/kallsyms | cut -f1 -d' '
ffffffff86232650
$ trace-cmd record -e timer:hrtimer_expire_entry -f 'function != 0xffffffff86232650' sleep 0.01

The function parameter is a machine-sized integer containing the address of the callback function. We must translate the function name to its address using the /proc/kallsyms file.

$ trace-cmd report
cpus=8
          <idle>-0     [006] 21953.174222: hrtimer_expire_entry: hrtimer=0xffffacffc6b43d10 now=21952218640837 function=hrtimer_wakeup/0x0
          <idle>-0     [001] 21953.174606: hrtimer_expire_entry: hrtimer=0xffffacffc1e17c18 now=21952219021472 function=hrtimer_wakeup/0x0
        Renderer-4729  [001] 21953.175599: hrtimer_expire_entry: hrtimer=0xffff90cf80307c98 now=21952220003877 function=intel_uncore_fw_release_timer/0x0
        Renderer-4729  [001] 21953.175602: hrtimer_expire_entry: hrtimer=0xffff90cf8191d418 now=21952220003877 function=intel_uncore_fw_release_timer/0x0
          <idle>-0     [007] 21953.177506: hrtimer_expire_entry: hrtimer=0xffffacffe3163a58 now=21952221925981 function=hrtimer_wakeup/0x0
        Renderer-4729  [001] 21953.179598: hrtimer_expire_entry: hrtimer=0xffff90cf80307c98 now=21952224002950 function=intel_uncore_fw_release_timer/0x0
    dav1d-worker-45360 [002] 21953.182593: hrtimer_expire_entry: hrtimer=0xffffacffe0fcfb88 now=21952227002482 function=hrtimer_wakeup/0x0
          <idle>-0     [001] 21953.183062: hrtimer_expire_entry: hrtimer=0xffffacffcd667c58 now=21952227481587 function=hrtimer_wakeup/0x0
    dav1d-worker-45358 [007] 21953.184810: hrtimer_expire_entry: hrtimer=0xffffacffe062fc10 now=21952229231336 function=hrtimer_wakeup/0x0
           sleep-45667 [000] 21953.184852: hrtimer_expire_entry: hrtimer=0xffffacffc2ae7b10 now=21952229272526 function=hrtimer_wakeup/0x0
           sleep-45667 [000] 21953.184944: hrtimer_expire_entry: hrtimer=0xffffacffc143f810 now=21952229364853 function=hrtimer_wakeup/0x0

To view the fields available for filtering and their types, print the event format:

$ trace-cmd list -e timer:hrtimer_expire_entry -F
system: timer
name: hrtimer_expire_entry
ID: 388
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:void * hrtimer;   offset:8;       size:8; signed:0;
        field:s64 now;  offset:16;      size:8; signed:1;
        field:void * function;  offset:24;      size:8; signed:0;

The fields prefixed with common_ are common to all events from the same system. Say, you want to trace all timer events but only for a specific PID:

$ trace-cmd record -e 'timer:*' -f 'common_pid == 1' sleep 1

To get the stack trace for each event, use -T:

$ trace-cmd record -e timer:hrtimer_expire_entry -f 'function != 0xffffffff86232650' -T sleep 0.01
$ trace-cmd report
cpus=8
          <idle>-0     [002] 24703.375192: hrtimer_expire_entry: hrtimer=0xffffacffc616fdc0 now=24702442127437 function=hrtimer_wakeup/0x0
          <idle>-0     [002] 24703.375200: kernel_stack:         <stack trace >
=> trace_event_raw_event_hrtimer_expire_entry (ffffffff86218921)
=> __hrtimer_run_queues (ffffffff8621c491)
=> hrtimer_interrupt (ffffffff8621d1aa)
=> __sysvec_apic_timer_interrupt (ffffffff8608db25)
=> sysvec_apic_timer_interrupt (ffffffff8715a20c)
=> asm_sysvec_apic_timer_interrupt (ffffffff8720160a)
=> cpuidle_enter_state (ffffffff8715c2f9)
=> cpuidle_enter (ffffffff86de023d)
=> do_idle (ffffffff8619ef17)
=> cpu_startup_entry (ffffffff8619f179)
=> start_secondary (ffffffff8608bacb)
=> common_startup_64 (ffffffff8603ddad)

Static probes are invaluable for understanding system behavior, but they require kernel developers to make them available. What if you need to trace a part of the kernel without tracepoints?

Kernel probes

Kernel probes, also known as kprobes, allow you to dynamically create tracepoints to log information. These probes can be attached to almost any kernel address, including functions and their entry/exit points.

To create a probe, you can write directly to the tracing filesystem, but I rather prefer to use perf-probe¹.

A few weeks ago I was debugging a memory leak reported in the RT kernel. At some point I realized there was something odd between the deadline scheduler and the hrtimer (now you know why I previously picked the hrtimer as a tracepoint example). The entry point to start the timer is the function start_dl_timer. Unfortunately, there is no tracepoint we can use to track its call. Let’s create one:

$ perf probe start_dl_timer
Added new event:
  probe:start_dl_timer (on start_dl_timer)

$ trace-cmd record -e probe:start_dl_timer sleep 1

Notice that, by default, perf-probe put the probe inside the “probe” group. This is all good but doesn’t give much information. We would like to know the PID, task_struct reference count and the timer object to relate to the hrtimer tracepoints. start_dl_timer receives a pointer to struct sched_dl_entity, containing the timer object. perf-probe allows you to dereference struct members, so adding the timer object to the trace is a piece of cake. Firstly, we need to remove the previously defined probe:

$ perf probe --del probe:start_dl_timer
Removed event: probe:start_dl_timer

Let’s see which lines we can probe:

$ perf probe -L start_dl_timer
      0  static int start_dl_timer(struct sched_dl_entity *dl_se)
         {
      2         struct hrtimer *timer = &dl_se->dl_timer;
                struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
                struct rq *rq = rq_of_dl_rq(dl_rq);
                ktime_t now, act;
                s64 delta;
         
                lockdep_assert_rq_held(rq);
         
                /*
                 * We want the timer to fire at the deadline, but considering
                 * that it is actually coming from rq->clock and not from
                 * hrtimer's time base reading.
                 */
                act = ns_to_ktime(dl_next_period(dl_se));
     16         now = hrtimer_cb_get_time(timer);
     17         delta = ktime_to_ns(now) - rq_clock(rq);
                act = ktime_add_ns(act, delta);
         
                /*
                 * If the expiry time already passed, e.g., because the value
                 * chosen as the deadline is too small, don't even try to
                 * start the timer in the past!
                 */
     25         if (ktime_us_delta(act, now) < 0)
                        return 0;
         
                /*
                 * !enqueued will guarantee another callback; even if one is already in
                 * progress. This ensures a balanced {get,put}_task_struct().
                 *
                 * The race against __run_timer() clearing the enqueued state is
                 * harmless because we're holding task_rq()->lock, therefore the timer
                 * expiring after we've done the check will wait on its task_rq_lock()
                 * and observe our state.
                 */
     37         if (!hrtimer_is_queued(timer)) {
     38                 if (!dl_server(dl_se))
     39                         get_task_struct(dl_task_of(dl_se));
     40                 hrtimer_start(timer, act, HRTIMER_MODE_ABS_HARD);
                }
         
                return 1;

The -L option lists the source code line numbers where we can install the probe hook².

Line 16 seems a good candidate, since the line 2 defines a local timer variable that holds the timer address. This is useful because we don’t have a syntax in perf-probe to take the address of a struct member. Just to illustrate member dereference, we will also print the timer function. Let’s confirm which variables are available to use:

$ perf probe -V start_dl_timer:16
Available variables at start_dl_timer:16
        @<start_dl_timer+67>
                ktime_t act
                struct dl_rq*   dl_rq
                struct hrtimer* timer
                struct rq*      rq
                struct sched_dl_entity* dl_se

The syntax to specify a function and line number is <function>:<line>. The -V option shows the available variables.

We mentioned we want to print the PID and the task_struct reference count. But where are they? dl_se is a member of task_struct (called dl inside the struct), so we can get the pointer to the task_struct using the container_of macro. There is only one problem: we don’t have container_of! But we can do it manually if we know the offset of dl_se inside task_struct. We can do it with the help of gdb:

 gdb /usr/lib/debug/lib/modules/$(uname -r)/vmlinux
Reading symbols from /usr/lib/debug/lib/modules/5.14.0-469.el9.x86_64/vmlinux...
(gdb) p &((struct task_struct *)0)->dl
$1 = (struct sched_dl_entity *) 0x280
(gdb) p &((struct task_struct *)0)->pid
$2 = (pid_t *) 0xa98
(gdb) p &((struct task_struct *)0)->usage
$3 = (refcount_t *) 0x30

This little trick works by casting 0 to struct task_struct * and then taking the address of the of field we are interested. This will give the field offset. Now we know the layout of the struct:

struct task_struct {
  // ...
  refcount_t usage;             // offset 0x30
  // ...
  struct sched_dl_entity dl;    // offset 0x280
  // ...
  pid_t pid;                    // offset 0xa98
};

From the function argument, we know the address of the field dl. The task_struct address is obtained by subtracting 0x280 from the dl_se pointer parameter. From there, to get a field address, just add the respective offset:

usage   = -0x280 + 0x030 = -0x250
pid     = -0x280 + 0xa98 = +0x818

We can finally define our custom probe:

$ perf probe 'start_dl_timer:16 hrtimer=timer timer->function pid=+0x818(%di) usage=-0x250(%di)'
Added new event:
  probe:start_dl_timer_L16 (on start_dl_timer:16 with hrtimer=timer function=timer->function pid=+0x818(%di) usage=-0x250(%di))

Notice we can give a name to the probe parameters, and also we can dereference structures to get their members. The field name becomes the probe parameter name if we don’t give one explicitly. Also notice the usage of the rdi register here². According to the x86_64 calling convention, it holds the function first argument. In our case, dl_se. Also, we had to specify the type s32 for pid and usage parameters. If you don’t specify a type, the default type is x64. s32 means signed 32 bits. Let’s use our new custom probe:

$ trace-cmd record -e probe:start_dl_timer_L16 stress-ng --cyclic 5 --timeout 10s --minimize --quiet
$ trace-cmd report
cpus=1
 stress-ng-cycli-51628 [000] 79677.069484: start_dl_timer_L16:   (ffffffff84b6c843) hrtimer=0xffff9f9dcaa649d8 function=0xffffffff84b72150 pid=51628 usage=1
 stress-ng-cycli-51628 [000] 79677.069708: start_dl_timer_L16:   (ffffffff84b6c843) hrtimer=0xffff9f9dcaa649d8 function=0xffffffff84b72150 pid=51628 usage=1

We can again consult the /proc/kallsyms to check what function is the hrtimer callback:

$ grep ffffffff84b72150 /proc/kallsyms 
ffffffff84b72150 t dl_task_timer

Summary

In this post, we explored the fundamentals of tracing the Linux kernel using static tracepoints and dynamic probes. These tools offer a powerful and flexible means of diagnosing and understanding kernel behavior. If you have any questions or comments, feel free to leave them below.

Going deeper

This post only scratches the surface. If you’re eager to delve deeper, here are some valuable resources:

1: perf probes are not exactly the same as kernel probes, but in practice the differences don’t matter.

1: you need the kernel debug symbols installed. For RHEL base systems, install it using dnf install -y kernel-debuginfo.

2: the probe syntax does not make a difference between rdi and edi. You just specify di. The same thing applies to the other registers.

Improving the Kernel TTY throughput

Wed, 31 Aug 2022 00:00:00 +0000

It is incredible how we can find room for improvement even in the eldest and battle-tested codebases out there. This post is about one of these cases.

I was investigating a soft lockup bug report which only happened in an HP Proliant DL380 Gen9. The trigger for the issue is when we try to load the scsi_debug with a few hundred virtual devices:

$ modprobe scsi_debug virtual_gb=1 add_host=2 num_tgts=600
Message from syslogd@storageqe-25 at Jul 15 10:29:35 ...
 kernel:watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [modprobe:35056]

Message from syslogd@storageqe-25 at Jul 15 10:30:23 ...
 kernel:watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [migration/25:140]

Message from syslogd@storageqe-25 at Jul 15 10:30:23 ...
 kernel:watchdog: BUG: soft lockup - CPU#18 stuck for 23s! [modprobe:35056]

And below, we show the dmesg output:

watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [modprobe:35056]
rcu: 	14-...!: (1 GPs behind) idle=24a/1/0x4000000000000002 softirq=12191/12192 fqs=5030
Modules linked in:
rcu: 	33-...!: (20094 ticks this GP) idle=f9e/1/0x4000000000000000 softirq=8891/8891 fqs=5031
 scsi_debug
	(detected by 14, t=80246 jiffies, g=346621, q=270790)
 loop
NMI backtrace for cpu 14
 dm_service_time
CPU: 14 PID: 0 Comm: swapper/14 Tainted: G               X --------- ---
 dm_multipath
Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 02/17/2017
 rfkill
Call Trace:
 intel_rapl_msr
 <IRQ>
 intel_rapl_common sb_edac
 dump_stack+0x64/0x7c
...
...
CPU: 18 PID: 35056 Comm: modprobe Tainted: G             L X --------- ---
Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 02/17/2017
RIP: 0010:number+0x21b/0x340
Code: 4c 8d 44 28 01 48 39 c3 76 03 c6 00 20 48 ...
RSP: 0018:ffffb03ec3650b98 EFLAGS: 00000046
RAX: 0000000000000000 RBX: ffffb03f43650cff RCX: 0000000000000038
RDX: ffffb03ec3650baf RSI: 0000000000000001 RDI: 0000000000000000
RBP: 00000000fffffffd R08: ffffb03ec3650d06 R09: 0000000000000004
R10: ffffb03ec3650c98 R11: ffffb03ec3650d06 R12: 0000000000000020
R13: 0000000000000000 R14: 0000000000004a0e R15: 0000000000000a00
FS:  00007f519cfb1740(0000) GS:ffff9b096fa00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a8b65190b8 CR3: 0000000550bec004 CR4: 00000000001706e0
Call Trace:
 <IRQ>
 vsnprintf+0x363/0x560
 sprintf+0x56/0x70
 info_print_prefix+0x7b/0xd0
 record_print_text+0x52/0x150
 console_unlock+0x144/0x330
 vprintk_emit+0x14d/0x230
 printk+0x58/0x6f
 print_modules+0x62/0xaf
 watchdog_timer_fn+0x190/0x200
 ? lockup_detector_update_enable+0x50/0x50
 __hrtimer_run_queues+0x12a/0x270
 hrtimer_interrupt+0x110/0x2c0
 __sysvec_apic_timer_interrupt+0x5f/0xd0
 sysvec_apic_timer_interrupt+0x6d/0x90
 </IRQ>

Notice printk() and (in special) console_unlock() in the stack trace. This is not a coincidence. It happened every single time.

I found out that scsi_debug sends a lot of information to the console output, causing the kernel to spend most of its time flushing data to the serial TTY. vprintk_emit() disables preemption before calling console_unlock()¹. Because the preemption is disabled and the serial TTY driver holds a spinlock while writing to the port, the RCU stall detector reports a stall. A slow serial console causing an RCU stall is well known and documented:

"Booting Linux using a console connection that is too slow to keep up with the boot-time
console-message rate. For example, a 115Kbaud serial console can be way too slow to keep
up with boot-time message rates and will frequently result in RCU CPU stall warning
messages. Especially if you have added debug printk()s."

In the hrtimer interrupt handler, the kernel calls the check_rcu_stall() function, which will call printk() to dump the stall information. At this time, the interrupt handler becomes the console owner and keeps flushing the remaining data to the serial console¹. The CPU will then get stuck in the interrupt routine, and triggers a soft lockup.

I suspected there was something odd with the serial port. It is configured with 115200 bps, 8 bits per data. Considering one start bit and one stop bit, it gives an expected throughput of approximately 11520 bytes per second. Let’s use bpftrace while loading the scsi_debug driver to verify if it is indeed the case:

$ bpftrace -e '
BEGIN {
	@ = 0;
}

kfunc:serial8250_console_putchar {
	@++;
}

interval:s:1 {
	printf("%d B/s\n", @);
	@ = 0;
}'
Attaching 3 probes...
2522 B/s
2517 B/s
2518 B/s
2507 B/s
2510 B/s
2414 B/s
...

serial8250_console_putchar() is the function that effectively writes a character to the serial port. We count how many times we call it per second.

Maybe the tracing machinery is causing an overhead, jeopardizing the serial performance? Let’s try another approach:

$ bpftrace -e '
BEGIN {
	@ = (uint32) 0;
}

kfunc:uart_console_write {
	@ += args->count;
}

interval:s:1 {
	printf("%u B/s\n", @);
	@ = 0;
}'
Attaching 3 probes...
1838 B/s
2460 B/s
2536 B/s
2415 B/s
2461 B/s
2247 B/s
2521 B/s
2441 B/s
2409 B/s
2378 B/s
2353 B/s
2398 B/s
2376 B/s
2414 B/s
...

Which has less overhead, as we don’t call the probe function for each character sent. Again the throughput is not what we would expect.

The scripts above report the overall system throughput. The next step is determining if the suboptimal performance is due to system overhead or the serial TTY code itself. The following bpftrace script measures more precisely the throughput of the serial console driver:

$ bpftrace -e '
kfunc:uart_console_write {
  @c[cpu] = args->count;
  @t[cpu] = nsecs;
}

kretfunc:uart_console_write {
  $elapsed = nsecs - @t[cpu];
  $rate = @c[cpu] * 1000000000 / $elapsed;
  printf("%ld B/s\n", $rate);
}'
Attaching 2 probes...
2206 B/s
2502 B/s
2585 B/s
2553 B/s
2559 B/s
2537 B/s
2555 B/s
2557 B/s
2554 B/s
2557 B/s
2561 B/s
2542 B/s
2534 B/s
2255 B/s
2555 B/s
2482 B/s
...

No luck. To facilitate my tests, I created a small driver + app that calls the console driver the same way printk() does. After loading the serco driver, use sertest -n <N> <driver-path> to send N bytes to the console. Let’s test it:

$ time ./sertest -n 2500 /dev/serco

real    0m0.997s
user    0m0.000s
sys     0m0.997s

Ok, 2500 bytes sent in approximately 1 second. That matches our previous results. With the help of the function tracer, we can see what is going on in more detail:

$ trace-cmd record -p function_graph -g serial8250_console_write \
   ./sertest -n 1 /dev/serco

$ trace-cmd report

            |  serial8250_console_write() {
 0.384 us   |    _raw_spin_lock_irqsave();
 1.836 us   |    io_serial_in();
 1.667 us   |    io_serial_out();
            |    uart_console_write() {
            |      serial8250_console_putchar() {
            |        wait_for_xmitr() {
 1.870 us   |          io_serial_in();
 2.238 us   |        }
 1.737 us   |        io_serial_out();
 4.318 us   |      }
 4.675 us   |    }
            |    wait_for_xmitr() {
 1.635 us   |      io_serial_in();
            |      __const_udelay() {
 1.125 us   |        delay_tsc();
 1.429 us   |      }
...
...
...
 1.683 us   |      io_serial_in();
            |      __const_udelay() {
 1.248 us   |        delay_tsc();
 1.486 us   |      }
 1.671 us   |      io_serial_in();
 411.342 us |    }

So, it takes 411 us to send just one single byte. The function serial8250_console_write() is the one responsible to dispatch the TTY data to the serial port (it calls uart_console_write()). It writes a character and waits for the chip controller to process the data before sending the next. Modern serial controllers have a FIFO of at least 16 bytes. Maybe we can exploit that and improve the situation. Well, that is what I did. Let’s how things go now:

$ time ./sertest -n 2500 /dev/serco

real    0m0.112s
user    0m0.056s
sys     0m0.055s

Wow! Much better! Now, if we send 11500 bytes, it should take approximately one second:

$ time ./sertest -n 11500 /dev/serco

real    0m3.142s
user    0m0.000s
sys     0m3.142s

Let’s run trace-cmd one more time, but I will only show the piece of the trace output that interests us:

sertest-21649 [008]  2360.845784: funcgraph_exit:         1.942 us   |        }
sertest-21649 [008]  2360.845784: funcgraph_entry:        1.710 us   |        io_serial_in();
sertest-21649 [008]  2360.845786: funcgraph_entry:                   |        __const_udelay() {
sertest-21649 [008]  2360.845786: funcgraph_entry:                   |          delay_tsc() {
sertest-21649 [008]  2360.845786: funcgraph_entry:        0.128 us   |            preempt_count_add();
sertest-21649 [008]  2360.845787: funcgraph_entry:        0.130 us   |            preempt_count_sub();
sertest-21649 [008]  2360.845787: funcgraph_entry:        0.130 us   |            preempt_count_add();
sertest-21649 [008]  2360.845787: funcgraph_entry:        0.131 us   |            preempt_count_sub();
sertest-21649 [008]  2360.845787: funcgraph_entry:        0.131 us   |            preempt_count_add();
sertest-21649 [008]  2360.845788: funcgraph_entry:        0.130 us   |            preempt_count_sub();
sertest-21649 [008]  2360.845788: funcgraph_exit:         1.666 us   |          }
sertest-21649 [008]  2360.845788: funcgraph_exit:         1.915 us   |        }
sertest-21649 [008]  2360.845788: funcgraph_entry:      + 12.787 us  |        io_serial_in();
sertest-21649 [008]  2360.845801: funcgraph_entry:                   |        __const_udelay() {
sertest-21649 [008]  2360.845801: funcgraph_entry:                   |          delay_tsc() {
sertest-21649 [008]  2360.845801: funcgraph_entry:        0.152 us   |            preempt_count_add();
sertest-21649 [008]  2360.845802: funcgraph_entry:        0.151 us   |            preempt_count_sub();
sertest-21649 [008]  2360.845802: funcgraph_entry:        0.151 us   |            preempt_count_add();
sertest-21649 [008]  2360.845802: funcgraph_entry:        0.143 us   |            preempt_count_sub();
sertest-21649 [008]  2360.845803: funcgraph_entry:        0.145 us   |            preempt_count_add();
sertest-21649 [008]  2360.845803: funcgraph_entry:        0.147 us   |            preempt_count_sub();
sertest-21649 [008]  2360.845803: funcgraph_exit:         1.909 us   |          }
sertest-21649 [008]  2360.845803: funcgraph_exit:         2.204 us   |        }
sertest-21649 [008]  2360.845803: funcgraph_entry:        1.743 us   |        io_serial_in();
sertest-21649 [008]  2360.845805: funcgraph_entry:                   |        __const_udelay() {

io_serial_in() usually executes in less than 2 us, but sometimes something odd happens, and the controller takes more than 10 us to reply. That is the reason we can’t get the expected speed of 115200 bps for such an amount of data.

We see a 25% throughput boost when working with the original modprobe sci_debug scenario. Even though we could not reach maximum performance, we improved things considerably.

1: console_unlock() will flush the printk() ring buffer before unlocking the console.

1: This behavior is vital to ensure essential information flushes from the ring buffer in the case of fatal errors.

A crash course on debugging kernel crashes using the crash utility

Thu, 24 Jun 2021 00:00:00 +0000

The other day I was working on a bug where kernel-rt-debug reported some warnings in the middle of the rcutorture test followed by a kernel hang:

rcu_torture_writer: Testing asynchronous GPs. 
rcu_torture_writer: Testing normal GPs. 
------------[ cut here ]------------ 
DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)) 
WARNING: CPU: 2 PID: 7996 at kernel/softirq.c:169 __local_bh_disable_ip+0x4ae/0x800 
Modules linked in: rcutorture torture rpcsec_gss_krb5 auth_rpcgss nfsv4 ....
CPU: 2 PID: 7996 Comm: rcu_torture_rea Kdump: loaded Not tainted 4.18.0-240.10.rt14.5.el8.x86_64+debug #1 
Hardware name: Hewlett-Packard ProLiant DL388eGen8, BIOS P73 06/01/2012 
RIP: 0010:__local_bh_disable_ip+0x4ae/0x800 
Code: 08 84 d2 0f 85 c7 02 00 00 8b 15 bd f7 73 07 85 d2 0f 85 37 fc ff ff 48 c7 c6 80 fd e7 8e 48 c7 ...
RSP: 0018:ffff8883d243fab0 EFLAGS: 00010286 
RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000 
RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000246 
RBP: ffff8883d243fae8 R08: fffffbfff1f08f59 R09: fffffbfff1f08f58 
R10: 0000000000000000 R11: ffffffff8f847ac3 R12: ffff888104d88000 
R13: 0000000000000010 R14: ffff8883d243fb98 R15: 0000000000000064 
FS:  0000000000000000(0000) GS:ffff8883d3800000(0000) knlGS:0000000000000000 
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
CR2: 000056094f904f44 CR3: 00000002c8228005 CR4: 00000000000606e0 
Call Trace: 
 ? rcutorture_one_extend+0x5af/0xa30 [rcutorture] 
 rcutorture_one_extend+0x5c0/0xa30 [rcutorture] 
 rcu_torture_one_read+0x562/0x900 [rcutorture] 
 ? rcutorture_one_extend+0xa30/0xa30 [rcutorture] 
 ? lock_timer_base+0x112/0x260 
 ? lock_downgrade+0x6f0/0x6f0 
 rcu_torture_reader+0x240/0x53d [rcutorture] 
 ? rcu_torture_timer+0x1c0/0x1c0 [rcutorture] 
 ? lock_downgrade+0x6f0/0x6f0 
 ? lock_contended+0xde0/0xde0 
 ? lock_acquire+0x134/0x4b0 
 ? rcu_torture_one_read+0x900/0x900 [rcutorture] 
 ? trace_hardirqs_on_caller+0x309/0x4d0 
 ? _raw_spin_unlock_irqrestore+0x45/0xf0 
 ? __kthread_parkme+0xb6/0x180 
 ? rcu_torture_timer+0x1c0/0x1c0 [rcutorture] 
 kthread+0x30c/0x3d0 
 ? kthread_create_on_node+0xc0/0xc0 
 ret_from_fork+0x3a/0x50 
irq event stamp: 10175619 
hardirqs last  enabled at (10175619): [<ffffffff8e8c8c74>] _raw_spin_unlock_irq+0x24/0xd0 
hardirqs last disabled at (10175618): [<ffffffff8e8baabc>] __schedule+0x1bc/0x22f0 
softirqs last  enabled at (10175616): [<ffffffff8cbefc15>] __local_bh_enable_ip+0xb5/0x1a0 
softirqs last disabled at (10175613): [<ffffffffc18d6aef>] rcutorture_one_extend+0x5af/0xa30 [rcutorture] 
---[ end trace 0000000000000002 ]--- 

The log is bigger than that but for this post only this part is worthy. The warning reported is from this code fragment ¹:

void __local_bh_disable_ip(unsigned long ip, unsigned int cnt)
{
  unsigned long flags;
  int newcnt;

  WARN_ON_ONCE(in_hardirq());

  /* First entry of a task into a BH disabled section? */
  if (!current->softirq_disable_cnt) {
    if (preemptible()) {
      local_lock(&softirq_ctrl.lock);
      /* Required to meet the RCU bottomhalf requirements. */
      rcu_read_lock();
    } else {
*     DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt));
    }
  }

  ...

As shown in the log above, __local_bh_disable_ip is called by rcutorture_one_extend:

static void rcutorture_one_extend(int *readstate, int newstate,
          struct torture_random_state *trsp,
          struct rt_read_seg *rtrsp)
{
  unsigned long flags;
  int idxnew = -1;
  int idxold = *readstate;
  int statesnew = ~*readstate & newstate;
  int statesold = *readstate & ~newstate;

  WARN_ON_ONCE(idxold < 0);
  WARN_ON_ONCE((idxold >> RCUTORTURE_RDR_SHIFT) > 1);
  rtrsp->rt_readstate = newstate;

  /*
   * First, put new protection in place to avoid critical-section gap.
   * Disable preemption around the ATOM disables to ensure that
   * in_atomic() is true.
   */
  if (statesnew & RCUTORTURE_RDR_BH)
*   local_bh_disable();
  if (statesnew & RCUTORTURE_RDR_RBH)
    rcu_read_lock_bh();
  if (statesnew & RCUTORTURE_RDR_IRQ)
    local_irq_disable();
  if (statesnew & RCUTORTURE_RDR_PREEMPT)
    preempt_disable();
  if (statesnew & RCUTORTURE_RDR_SCHED)
    rcu_read_lock_sched();
  preempt_disable();
  if (statesnew & RCUTORTURE_RDR_ATOM_BH)
*   local_bh_disable();
  if (statesnew & RCUTORTURE_RDR_ATOM_RBH)
    rcu_read_lock_bh();
  preempt_enable();
...
...

As shown, rcutorture_one_extend calls __local_bh_disable_ip in more than one location. As the functions offsets also appear in the log, finding the source code line is straightforward. I will show you how to do it later. What I am really interested in is the values of the variables statesnew and statesold, as they control several conditional calls this function makes. Knowing the value of these variables will help us have a better idea about the context in which __local_bh_disable_ip was called when it triggered the warning.

In the rest of this post, I am going to use the crash utility to reveal the value of these variables. Crash is an extension of gdb so we can utilize it to debug kernel and kdump crash files.

I am not going to explain how to set up kdump and generate the vmcore ² file since there are plenty of docs around explaining how to do so. Our focus here is to exemplify a use case for crash. You need to use a debug kernel to show the warning and you must run sysctl panic_on_warn=1 to generate a crash dump on warning.

Assuming you have a vmlinux image and a vmcore dump, you start crash like so:

# crash /usr/lib/debug/lib/modules/4.18.0-311.rt7.92.el8.test.x86_64+debug/vmlinux vmore

The dmesg command shows messages from the kernel ring buffer in case you have the vmcore file but no log dump. Let’s use the bt command to see the call stack trace:

crash> bt -s
PID: 31010  TASK: ffff8882024a3500  CPU: 0   COMMAND: "rcu_torture_rea"
 #0 [ffff8881d2067620] machine_kexec+888 at ffffffff8a920d28
 #1 [ffff8881d2067740] __crash_kexec+171 at ffffffff8ac938bb
 #2 [ffff8881d2067858] panic+476 at ffffffff8aa01605
 #3 [ffff8881d2067918] __warn.cold.6+27 at ffffffff8aa018aa
 #4 [ffff8881d2067958] report_bug+412 at ffffffff8c788c8c
 #5 [ffff8881d2067988] do_error_trap+306 at ffffffff8a854542
 #6 [ffff8881d20679d0] do_invalid_op+54 at ffffffff8a854f86
 #7 [ffff8881d20679f0] invalid_op+20 at ffffffff8ca00e04
    [exception RIP: __local_bh_disable_ip+1198]
    RIP: ffffffff8aa1b37e  RSP: ffff8881d2067aa0  RFLAGS: 00010282
    RAX: 0000000000000000  RBX: 0000000000000200  RCX: ffffffff8adbc90e
    RDX: 0000000000000000  RSI: 0000000000000004  RDI: 0000000000000246
    RBP: ffff8881d2067ad8   R8: fffffbfff1d271e2   R9: 0000000000000000
    R10: ffffffff8e938f0b  R11: fffffbfff1d271e1  R12: ffff8882024a3500
    R13: 0000000000000010  R14: ffff8881d2067b98  R15: 0000000000000044
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffff8881d2067ae0] rcutorture_one_extend+1483 at ffffffffc19f745b [rcutorture]
 #9 [ffff8881d2067b40] rcu_torture_one_read+1763 at ffffffffc19f8033 [rcutorture]
#10 [ffff8881d2067dc8] rcu_torture_reader+578 at ffffffffc19f8762 [rcutorture]
#11 [ffff8881d2067f10] kthread+976 at ffffffff8aa80ed0
#12 [ffff8881d2067f50] ret_from_fork+58 at ffffffff8ca0026a
crash> 

In this example, I am using the linux image provided by the kernel-rt-debug-debuginfo ³ file. The -s option displays the offsets of the functions. By the way, typing help shows a summary of the available commands, and typing help <cmd> shows the documentation for the command cmd. You can use any gdb commands while using crash. Therefore, to display the variables we are interested in, set the stack frame to the rcutorture_one_extend function and print the variables:

crash> frame 8
crash: prohibited gdb command: frame
crash>  gdb frame 8
crash: prohibited gdb command: frame

Err, you didn’t think it would be that easy, did you? Sadly, we will need to do it manually. Before anything else, let’s instruct crash to load the debug symbols for the loadable modules with the mod command:

crash> mod -S                                                                                                                                                                                                                                                     
     MODULE       NAME                    SIZE  OBJECT FILE                                                                                                                                                                                                       
ffffffffc04678c0  dm_mod                397312  /usr/.../modules/.../kernel/drivers/md/dm-mod.ko.debug
ffffffffc047ee00  dm_log                 36864  /usr/.../modules/.../kernel/drivers/md/dm-log.ko.debug
ffffffffc0497a80  dm_region_hash         36864  /usr/.../modules/.../kernel/drivers/md/dm-region-hash.ko.debug
ffffffffc04ae300  dm_mirror              65536  /usr/.../modules/.../kernel/drivers/md/dm-mirror.ko.debug
ffffffffc04c6f80  ip_tables              69632  /usr/.../modules/.../kernel/net/ipv4/netfilter/ip_tables.ko.debug
ffffffffc0545900  sysfillrect            28672  /usr/.../modules/.../kernel/drivers/video/fbdev/core/sysfillrect.ko.debug
ffffffffc054c000  syscopyarea            24576  /usr/.../modules/.../kernel/drivers/video/fbdev/core/syscopyarea.ko.debug
ffffffffc055b340  crc32c_intel           24576  /usr/.../modules/.../kernel/arch/x86/crypto/crc32c-intel.ko.debug
ffffffffc056ee40  mgag200                69632  /usr/.../modules/.../kernel/drivers/gpu/drm/mgag200/mgag200.ko.debug
ffffffffc057adc0  ata_generic            20480  /usr/.../modules/.../kernel/drivers/ata/ata_generic.ko.debug
ffffffffc0583680  t10_pi                 20480  /usr/.../modules/.../kernel/block/t10-pi.ko.debug
ffffffffc0595200  mdio                   28672  /usr/.../modules/.../kernel/drivers/net/mdio.ko.debug
ffffffffc05a55c0  ata_piix               61440  /usr/.../modules/.../kernel/drivers/ata/ata_piix.ko.debug
ffffffffc05aa180  libcrc32c              16384  /usr/.../modules/.../kernel/lib/libcrc32c.ko.debug
...

Now we can use the dis command to find out the source line where we call __local_bh_disable_ip:

crash> dis -l rcutorture_one_extend
/usr/src/debug/kernel-rt-4.18.0-311.rt7.92.el8.test/linux-4.18.0-311.rt7.92.el8.test.x86_64/kernel/rcu/rcutorture.c: 1235
<rcutorture_one_extend>:     data32 data32 data32 xchg %ax,%ax [FTRACE NOP]
/usr/src/debug/kernel-rt-4.18.0-311.rt7.92.el8.test/linux-4.18.0-311.rt7.92.el8.test.x86_64/kernel/rcu/rcutorture.c: 1238
<rcutorture_one_extend+5>:   movabs $0xdffffc0000000000,%rax
<rcutorture_one_extend+15>:  push   %rbp
...
...
...
/usr/src/debug/kernel-rt-4.18.0-311.rt7.92.el8.test/linux-4.18.0-311.rt7.92.el8.test.x86_64/./include/linux/bottom_half.h: 19
<rcutorture_one_extend+1444>:        mov    $0x200,%esi
<rcutorture_one_extend+1449>:        mov    $0xffffffffc19f7434,%rdi
<rcutorture_one_extend+1456>:        callq  0xffffffff8aa1aed0 <__local_bh_disable_ip>
<rcutorture_one_extend+1461>:        jmpq   0xffffffffc19f6f44 <rcutorture_one_extend+180>
<rcutorture_one_extend+1466>:        mov    $0x200,%esi
<rcutorture_one_extend+1471>:        mov    $0xffffffffc19f744a,%rdi
<rcutorture_one_extend+1478>:        callq  0xffffffff8aa1aed0 <__local_bh_disable_ip>
/usr/src/debug/kernel-rt-4.18.0-311.rt7.92.el8.test/linux-4.18.0-311.rt7.92.el8.test.x86_64/kernel/rcu/rcutorture.c: 1264
<rcutorture_one_extend+1483>:        testb  $0x80,-0x30(%rbp)
...
...
...

The -l option provides the source code line number information. We see that the 1483 offset is at rcutorture.c:1264 and the instruction above is the call we are looking for⁴. Now we can look at the source code to see the corresponding line:

crash> dis -s rcutorture_one_extend
FILE: kernel/rcu/rcutorture.c
LINE: 1235

 * change, do a ->read_delay().
 */
static void rcutorture_one_extend(int *readstate, int newstate,
                                  struct torture_random_state *trsp,
                                  struct rt_read_seg *rtrsp)
{
        unsigned long flags;
        int idxnew = -1;
        int idxold = *readstate;
        int statesnew = ~*readstate & newstate;
        int statesold = *readstate & ~newstate;

        WARN_ON_ONCE(idxold < 0);
        WARN_ON_ONCE((idxold >> RCUTORTURE_RDR_SHIFT) > 1);
        rtrsp->rt_readstate = newstate;

        /*
         * First, put new protection in place to avoid critical-section gap.
         * Disable preemption around the ATOM disables to ensure that
         * in_atomic() is true.
         */
        if (statesnew & RCUTORTURE_RDR_BH)
                local_bh_disable();
        if (statesnew & RCUTORTURE_RDR_RBH)
                rcu_read_lock_bh();
        if (statesnew & RCUTORTURE_RDR_IRQ)
                local_irq_disable();
        if (statesnew & RCUTORTURE_RDR_PREEMPT)
                preempt_disable();
        if (statesnew & RCUTORTURE_RDR_SCHED)
                rcu_read_lock_sched();
        preempt_disable();
        if (statesnew & RCUTORTURE_RDR_ATOM_BH)
*                local_bh_disable();
        if (statesnew & RCUTORTURE_RDR_ATOM_RBH)
                rcu_read_lock_bh();
...

The line 1263 is where we make the call.

As promised at the beginning, we will now recover the values of the local variables statesnew and statesold. Let’s start by looking at the start of the rcutorture_one_extend function assembly code:

crash> dis -l rcutorture_one_extend
/usr/src/debug/kernel-rt-4.18.0-311.rt7.92.el8.test/linux-4.18.0-311.rt7.92.el8.test.x86_64/kernel/rcu/rcutorture.c: 1235
<rcutorture_one_extend>:     data32 data32 data32 xchg %ax,%ax [FTRACE NOP]
/usr/src/debug/kernel-rt-4.18.0-311.rt7.92.el8.test/linux-4.18.0-311.rt7.92.el8.test.x86_64/kernel/rcu/rcutorture.c: 1238
<rcutorture_one_extend+5>:   movabs $0xdffffc0000000000,%rax
<rcutorture_one_extend+15>:  push   %rbp
<rcutorture_one_extend+16>:  mov    %rsp,%rbp
<rcutorture_one_extend+19>:  push   %r15
<rcutorture_one_extend+21>:  mov    %esi,%r15d
<rcutorture_one_extend+24>:  push   %r14
<rcutorture_one_extend+26>:  mov    %rdi,%r14
<rcutorture_one_extend+29>:  push   %r13
<rcutorture_one_extend+31>:  push   %r12
<rcutorture_one_extend+33>:  push   %rbx
<rcutorture_one_extend+34>:  sub    $0x28,%rsp
<rcutorture_one_extend+38>:  mov    %rdx,-0x48(%rbp)
<rcutorture_one_extend+42>:  mov    %rdi,%rdx
<rcutorture_one_extend+45>:  shr    $0x3,%rdx
<rcutorture_one_extend+49>:  mov    %rcx,-0x38(%rbp)
<rcutorture_one_extend+53>:  movzbl (%rdx,%rax,1),%edx
<rcutorture_one_extend+57>:  mov    %rdi,%rax
<rcutorture_one_extend+60>:  and    $0x7,%eax
<rcutorture_one_extend+63>:  add    $0x3,%eax
<rcutorture_one_extend+66>:  cmp    %dl,%al
<rcutorture_one_extend+68>:  jl     0xffffffffc19f6ede <rcutorture_one_extend+78>
<rcutorture_one_extend+70>:  test   %dl,%dl
<rcutorture_one_extend+72>:  jne    0xffffffffc19f7856 <rcutorture_one_extend+2502>
<rcutorture_one_extend+78>:  mov    (%r14),%r13d
/usr/src/debug/kernel-rt-4.18.0-311.rt7.92.el8.test/linux-4.18.0-311.rt7.92.el8.test.x86_64/kernel/rcu/rcutorture.c: 1239
<rcutorture_one_extend+81>:  mov    %r15d,%r12d
<rcutorture_one_extend+84>:  not    %r12d
<rcutorture_one_extend+87>:  mov    %r13d,%eax
<rcutorture_one_extend+90>:  and    %r13d,%r12d
<rcutorture_one_extend+93>:  not    %eax
<rcutorture_one_extend+95>:  and    %r15d,%eax
<rcutorture_one_extend+98>:  mov    %eax,-0x30(%rbp)

The only parameters we care about are readstate and newstate. According to the x86_64 calling convention, they are passed, respectively, in the rdi and rsi registers. Below is a commented version of the rcutorture_one_extend function assembly code up to the part that assigns values to both variables:

movabs $0xdffffc0000000000,%rax
push   %rbp
mov    %rsp,%rbp  # setup the function stack frame
push   %r15       # save register r15
mov    %esi,%r15d # move newstate to r15d
push   %r14       # save r14
mov    %rdi,%r14  # move readstate to rdi
push   %r13       # save r13
push   %r12       # save r12
push   %rbx       # save rbx
sub    $0x28,%rsp # allocate space for the local variables

#######################################################################
# this part doesn't interest us
mov    %rdx,-0x48(%rbp)
mov    %rdi,%rdx
shr    $0x3,%rdx
mov    %rcx,-0x38(%rbp)
movzbl (%rdx,%rax,1),%edx
mov    %rdi,%rax
and    $0x7,%eax
add    $0x3,%eax
cmp    %dl,%al
jl     0xffffffffc19f6ede <rcutorture_one_extend+78>
test   %dl,%dl
jne    0xffffffffc19f7856 <rcutorture_one_extend+2502>
#######################################################################

mov    (%r14),%r13d     # move *readstate to r14d
mov    %r15d,%r12d      # move newstate to r12d
not    %r12d            # r12 = ~newstate
mov    %r13d,%eax       # move *readstate to eax
and    %r13d,%r12d      # r12d = *readstate & ~newstate = statesold
not    %eax             # eax = ~*readstate
and    %r15d,%eax       # eax = ~*readstate & newstate = statesnew
mov    %eax,-0x30(%rbp) # statesnew = (int) rbp[-6]

From the code above, statesold ends up in the register r12d and statesnew ends in the position rbp-6 (in 64 bits sized chunks) of the stack frame. We shall start deducing the value of statesnew. We can take advantage of the option -f of the bt command to display the content of the stack frame:

crash> bt -f
...
 #8 [ffff8881d2067ae0] rcutorture_one_extend at ffffffffc19f745b [rcutorture]
    ffff8881d2067ae8: ffffffff8abc8c10 ffff8881d2067e00
    ffff8881d2067af8: 000000003a40cf78 ffff8881d2067c00
    ffff8881d2067b08: ffff888300000044 ffff8881d2067bd8
    ffff8881d2067b18: ffffffffc1a5f300 ffff8881d2067c28
    ffff8881d2067b28: ffff8881d2067b98 ffff8881d2067d40
    ffff8881d2067b38: ffff8881d2067e00 ffffffffc19f8033
...

The stack frame starts at 0xffff8881d2067b38 (remember the stack grows downwards). The first position corresponds to the rbp push at the function preamble. We then count 6 positions upwards to the right and reach the address 0xffff8881d2067b08 whose value is 0xffff888300000044. Since statesnew is an integer, we only count the first 32 bits and we finally find the value 0x44. If we look at the kernel/rcu/rcutorture.c source file, we conclude that statesnew = RCUTORTURE_RDR_PREEMPT | RCUTORTURE_RDR_ATOM_BH.

statesold is stored in the r12d register, but we can’t rely on the register value because it probably has already been clobbered by one of the rcutorture_one_extend callees. However, it is saved in some function’s stack frame. Here is the prologue of the __local_bh_disable_ip, the first rcutorture_one_extend callee:

crash> dis __local_bh_disable_ip
<__local_bh_disable_ip>:     data32 data32 data32 xchg %ax,%ax [FTRACE NOP]
<__local_bh_disable_ip+5>:   push   %rbp
<__local_bh_disable_ip+6>:   mov    %rsp,%rbp
<__local_bh_disable_ip+9>:   push   %r15
<__local_bh_disable_ip+11>:  push   %r14
<__local_bh_disable_ip+13>:  push   %r13
<__local_bh_disable_ip+15>:  push   %r12
...

Indeed statesold is saved in the __local_bh_disable_ip stack frame, more precisely in rbp-4. Below is the function stack frame:

crash> bt -f
...
 #7 [ffff8881d20679f0] invalid_op at ffffffff8ca00e04
    [exception RIP: __local_bh_disable_ip+1198]
    RIP: ffffffff8aa1b37e  RSP: ffff8881d2067aa0  RFLAGS: 00010282
    RAX: 0000000000000000  RBX: 0000000000000200  RCX: ffffffff8adbc90e
    RDX: 0000000000000000  RSI: 0000000000000004  RDI: 0000000000000246
    RBP: ffff8881d2067ad8   R8: fffffbfff1d271e2   R9: 0000000000000000
    R10: ffffffff8e938f0b  R11: fffffbfff1d271e1  R12: ffff8882024a3500
    R13: 0000000000000010  R14: ffff8881d2067b98  R15: 0000000000000044
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    ffff8881d20679f8: 0000000000000044 ffff8881d2067b98 
    ffff8881d2067a08: 0000000000000010 ffff8882024a3500 
    ffff8881d2067a18: ffff8881d2067ad8 0000000000000200 
    ffff8881d2067a28: fffffbfff1d271e1 ffffffff8e938f0b 
    ffff8881d2067a38: 0000000000000000 fffffbfff1d271e2 
    ffff8881d2067a48: 0000000000000000 ffffffff8adbc90e 
    ffff8881d2067a58: 0000000000000000 0000000000000004 
    ffff8881d2067a68: 0000000000000246 ffffffffffffffff 
    ffff8881d2067a78: ffffffff8aa1b37e 0000000000000010 
    ffff8881d2067a88: 0000000000010282 ffff8881d2067aa0 
    ffff8881d2067a98: 0000000000000018 ffff8881d2067e00 
    ffff8881d2067aa8: ffffffffc19f744a ffff8881d2067bd8 
    ffff8881d2067ab8: 0000000000000010 0000000000000010 
    ffff8881d2067ac8: ffff8881d2067b98 0000000000000044 
    ffff8881d2067ad8: ffff8881d2067b38 ffffffffc19f745b
...

I guess you can see that statesold location is 0xffff8881d2067ab8 and its value is 0x10. If we do the same exercise of looking for the corresponding macro(s) definition(s) for the value, we discover that statesold = RCUTORTURE_RDR_SCHED.

Thanks to Julia Denham for taking the time to review this post!

1: the watchful reader may find some inconsistencies among the line numbers reported in the log and in the git tree. This is because I am debugging the RHEL 8 kernel.

2: the name of the crash file kdump generates.

3: notice that I am using the rt-debug kernel to reproduce the problem. This bug only manifests in the realtime kernel. Be sure to use the debuginfo package corresponding to your kernel.

4: bear in mind that the call stack entries indicate the return addresses.

5: local_bh_disable is an inline function that calls __local_bh_disable_ip:

static inline void local_bh_disable(void)
{
        __local_bh_disable_ip(_THIS_IP_, SOFTIRQ_DISABLE_OFFSET);
}

NUIPreview: a Visual Studio extension to build Tizen.NUI graphical interfaces

Tue, 22 Dec 2020 00:00:00 +0000

As the last project of 2020, I and some other co-workers were assigned the task of creating a prototype for a graphical user interface preview Visual Studio extension for the NUI graphics toolkit. Notice the extension is for Visual Studio, not Visual Studio Code.

The idea is the programmer create the user interface in a XAML file and the plugin displays the preview of the user interface. NUI already has some support for the XAML standard.

For parsing the XAML file, @lauromoura found the OmniXaml library. Although not maintained anymore, the library is pretty complete. Our initial idea was to make the window show the result integrated with Visual Studio, but as NUI creates its own Window and message loop, we had to take another approach. The basic architecture is as follows: when the user click in the menu to show the preview, we start a thread in which we create the NUI Window and run the message loop. We also start a timer that periodically polls the active document; if it is a XAML file and we can parse it successfully, we update the UI.

One particular detail is that Visual Studio is a 32 bits executable, so all the DALi libraries and their dependencies must be compiled in 32 bits. As DALi is contantly evolving, we created a branch in our own fork to make sure we can keep the plugin development without being surprised by bugs introduced by new commits.

Building DALi for 32 bits Windows

First, you have to make sure all vcpkg dependencies are installed with their 32 bits variant. That’s the default in vcpkg last time I checked, but you can also add the :x86-windows suffix for each package name.

That said, download our fork of the DALi libraries:

$ git clone -b vsix git://github.com/expertisesolutions/dali-core
$ git clone -b vsix git://github.com/expertisesolutions/dali-adaptor
$ git clone -b vsix git://github.com/expertisesolutions/dali-toolkit
$ git clone -b vsix git://github.com/expertisesolutions/dali-csharp-binder
$ git clone -b vsix git://github.com/expertisesolutions/tizenfx-stub

You also need the windows-dependencies repository. To build the libraries, you can use the following batch script:

set base_dir=c:\work
set common_args=-g "Visual Studio 16 2019" -A Win32 -DCMAKE_INSTALL_PREFIX=%base_dir%\dali-env \
-DCMAKE_TOOLCHAIN_FILE=%base_dir%\vcpkg\scripts\buildsystems\vcpkg.cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_DEBUG=ON -Wno-dev

REM window-dependencies
cd %base_dir%\windows-dependencies\build\
rd /Q /S build
mkdir build
cd build
cmake %common_args% .. || goto :error
cmake --build . --target install --config debug || goto :error

REM dali-core
cd %base_dir%\dali-core\build\tizen
rd /Q /S build-windows
mkdir build-windows
cd build-windows
cmake %common_args% -DENABLE_PKG_CONFIGURE=OFF -DENABLE_LINK_TEST=OFF -DINSTALL_CMAKE_MODULES=ON .. || goto :error
cmake --build . --target install --config debug || goto :error

REM dali-adaptor
cd %base_dir%\dali-adaptor\build\tizen
rd /Q /S build-windows
mkdir build-windows
cd build-windows
cmake %common_args% -DENABLE_PKG_CONFIGURE=OFF -DENABLE_LINK_TEST=OFF -DINSTALL_CMAKE_MODULES=ON -DPROFILE_LCASE=windows .. || goto :error
cmake --build . --target install --config debug || goto :error

REM dali-toolkit
cd %base_dir%\dali-toolkit\build\tizen
rd /Q /S build-windows
mkdir build-windows
cd build-windows
cmake %common_args% -DENABLE_PKG_CONFIGURE=OFF -DENABLE_LINK_TEST=OFF -DINSTALL_CMAKE_MODULES=ON -DUSE_DEFAULT_RESOURCE_DIR=OFF .. || goto :error
cmake --build . --target install -j1 --config debug || goto :error

REM dali-csharp-binder
cd %base_dir%\dali-csharp-binder\build\tizen
rd /Q /S build-windows
mkdir build-windows
cd build-windows
cmake %common_args% .. || goto :error
cmake --build . --target install --verbose --config debug || goto :error

tizenfx-stub
cd %base_dir%\tizenfx-stub
rd /Q /S build-windows
mkdir build-windows
cd build-windows
cmake %common_args% .. || goto :error
cmake --build . --target install --verbose --config debug || goto :error

:error
cd %base_dir%

Notice it assumes you have the vcpkg and the repositories under C:\work. You can tweak the script for your own directory structure.

You also need to build angle. To build it you can follow the official documentation, but before running the autoninja command to compile it, you have to run gn args <output directory> and set the following variables:

angle_enable_d3d11 = false
target_cpu = "x86"

To make redistribution of the plugin easier, we ship the NUIPreview repository with all DLL dependencies, so you don’t need to worry about compiling them.

Running the plugin

Here is a demo of how the plugin behaves:

Of course, this is just a prototype. You check the source code in the repository.

Highway to hell: C++ enums and bit fields

Sun, 25 Oct 2020 00:00:00 +0000

C++ enums historically gave me some headaches due to the fact that compilers are free to choose the size of the type for whatever criteria they think. This makes particularly hard to write portable code among compilers from ABI point of view (Visual C++ vs C++ Builder, anyone?).

Some days ago I was debugging a problem with dali-toolkit and the enum story scalated to another level.

Let me give you some background: dali-toolkit loads textures using a background worker thread that is responsible to load the image files to memory. I was looking why some textures didn’t load in Windows. To make the asynchronous loading possible it implements a small state machine and it keeps all information about the texture in a type called TextureInfo.

Here is a partial definition of the structure:

struct TextureInfo {
		// ....
    LoadState loadState:4;         ///< The load state showing the load progress of the Texture
    FittingMode::Type fittingMode:3; ///< The requested FittingMode
    Dali::SamplingMode::Type samplingMode:3; ///< The requested SamplingMode
    StorageType storageType:2;     ///< CPU storage / GPU upload;
		// ....
};

All these custom types are enums defined elsewhere. Yes, you read right, enums + bit fields, what possibly can go wrong? To illustrate the root cause of the bug, let’s write a toy sample:

enum Animal {
    Cat,
    Dog,
    Horse,
    Turtle,
};

enum Breed {
    BorderCollie,
    GoldenRetriever,
    GermanShepard,
    Husky,
};

enum Adopted {
    No,
    Yes,
};

struct AnimalInfo {
    Animal animal:2;
    Breed breed:2;
    Adopted adopted:1;

    AnimalInfo(Animal a, Breed b, Adopted ad)
        : animal(a), breed(b), adopted(ad) {}
};

int main(void) {
    AnimalInfo animal(Animal::Dog, Breed::GermanShepard, Adopted::Yes);

    auto breed = animal.breed;

    return sizeof(AnimalInfo);
}

Now, using godbolt, we can view the assembly code generated by different compilers. In particular, let’s look at the assembly code for the main function. Firstly, the the gcc generated code. I added comments to the parts interesting to us:

main:
        push    rbp
        mov     rbp, rsp
        sub     rsp, 16
        lea     rax, [rbp-8]
        mov     ecx, 1
        mov     edx, 2
        mov     esi, 1
        mov     rdi, rax
        call    AnimalInfo::AnimalInfo(Animal, Breed, Adopted)
        movzx   eax, BYTE PTR [rbp-8] // load the information
        shr     al, 2   	// breed is in the bits 2 and 3, so shift right 2 bits
        and     eax, 3  	// breed is only two bits long
        movzx   eax, al  	// eax = breed value
        mov     DWORD PTR [rbp-4], eax // store it in our variable
        mov     eax, 4    // AnimalInfo is 4 bytes long
        leave
        ret

Here is clang. It uses different registers but is essentially the same code:

main:
        push    rbp
        mov     rbp, rsp
        sub     rsp, 16
        mov     dword ptr [rbp - 4], 0
        lea     rdi, [rbp - 8]
        mov     eax, 1
        mov     esi, eax
        mov     edx, 2
        mov     ecx, eax
        call    AnimalInfo::AnimalInfo(Animal, Breed, Adopted)
        mov     r8b, byte ptr [rbp - 8]
        shr     r8b, 2
        and     r8b, 3
        movzx   eax, r8b
        mov     dword ptr [rbp - 12], eax
        mov     eax, 4
        add     rsp, 16
        pop     rbp
        ret

Now the msvc compiler:

main PROC
$LN3:
  sub rsp, 56 ; 00000038H
  mov r9d, 1
  mov r8d, 2
  mov edx, 1
  lea rcx, QWORD PTR animal$[rsp]
  call AnimalInfo::AnimalInfo(Animal,Breed,Adopted) ; AnimalInfo::AnimalInfo
  mov eax, DWORD PTR animal$[rsp]
  shl eax, 28
  sar eax, 30 // OH NO! SIGNED SHIFT!!!
  mov DWORD PTR breed$[rsp], eax
  mov eax, 4
  add rsp, 56 ; 00000038H
  ret 0
main ENDP

Before commenting about the bug generated by msvc, let me praise the compiler optimizer, it got the breed value in two instructions, while clang and gcc needed three. Very smart! Too bad that code is wrong. I think it is better to undestand the issue if we view the bit layout of the word holded by AnimalInfo:

	-------------------------------------
	|  31-5  |    4    |  3-2  |  1-0   |
	-------------------------------------
	| Unused | Adopted | Breed | Animal |
	-------------------------------------

So, the code shifts 28 bits to left, moving Breed to the bits 31 and 30, then it shifts 30 bits to the right, but it does a signed shift, filling the shifted bits with the value of the most significant bit. The Breed value is GermanShepard, which is 2. So, when we shift left, the bit 31 will hold the value 1, and when we shift right, we end up with the final value 0xfffffffe! The reason this happens is because msvc assumes the enum is a signed integer by default and, combined with the use of bit fields, yields the bug. This is precisely what happened in dali-toolkit.

Once we understood the problem, we can solve it easily by forcing the enum to an unsigned integer:

enum class Animal: unsigned int {
    Cat,
    Dog,
    Horse,
    Turtle,
};

enum class Breed: unsigned int {
    BorderCollie,
    GoldenRetriever,
    GermanShepard,
    Husky,
};

enum class Adopted: unsigned int {
    No,
    Yes,
};

struct AnimalInfo {
    Animal animal:2;
    Breed breed:2;
    Adopted adopted:1;

    AnimalInfo(Animal a, Breed b, Adopted ad)
        : animal(a), breed(b), adopted(ad) {}
};

int main(void) {
    AnimalInfo animal(Animal::Dog, Breed::GermanShepard, Adopted::Yes);

    auto breed = animal.breed;

    return sizeof(AnimalInfo);
}

Now the generated code does what it is supposed to do:

main PROC
$LN3:
  sub rsp, 56 ; 00000038H
  mov r9d, 1
  mov r8d, 2
  mov edx, 1
  lea rcx, QWORD PTR animal$[rsp]
  call AnimalInfo::AnimalInfo(Animal,Breed,Adopted) ; AnimalInfo::AnimalInfo
  mov eax, DWORD PTR animal$[rsp]
  shr eax, 2
  and eax, 3
  mov DWORD PTR breed$[rsp], eax
  mov eax, 4
  add rsp, 56 ; 00000038H
  ret 0
main ENDP

Building DALi library for Windows

Wed, 05 Aug 2020 00:00:00 +0000

Since I left Mozilla I came back to my old days as a C/C++ developer. One of the projects I am working on is porting the Tizen.NUI API to Windows. Tizen.NUI itself is written in C#, so it doesn’t require any major effort to run on Windows, but it is heavily based on DALi, and there where the job lies. DALi kind of works on Windows already, but I caught several bugs affecting 64 bits. You can refer to the pull requests to get details on what was wrong.

DALi, doesn’t belong to a single repo, actually. On top of dali-core we have dali-adaptor and dali-toolkit, not to mention the sample applications. This post is a step by step guide to build DALi on Windows.

Dependencies

To build and install DALi you will need:

Visual Studio 2019 (Community Edition is fine)
git
vcpkg

Building Windows dependencies

After the dependencies installed, you need the windows-dependencies repository. Run the following commands in a subfolder that will host the DALi repos:

C:\DALi>git clone https://github.com/dalihub/windows-dependencies
C:\DALi>set VCPKG_FOLDER=C:\DALi
C:\DALi>set DALI_ENV_FOLDER=C:\DALi\dali-env
C:\DALi>cd windows-dependencies\build
C:\DALi\windows-dependencies\build>mkdir build
C:\DALi\windows-dependencies\build>cd build
C:\DALi\windows-dependencies\build\build>cmake -g Ninja . -DCMAKE_TOOLCHAIN_FILE=%VCPKG_FOLDER%/vcpkg/scripts/buildsystems/vcpkg.cmake -DCMAKE_INSTALL_PREFIX=%DALI_ENV_FOLDER% ..
C:\DALi\windows-dependencies\build\build>cmake --build . --target install

DALI_ENV_FOLDER is the output directory where the artifacts generated are installed.

Building the DALi libraries

Building the DALi libraries (dali-core, dali-adaptor and dali-toolkit) is straightforward and the README of each library describes the steps in more detail. Here I am just giving you the sequence of commands to build them. If you want to know more details, you can refer to each library documentation.

dali-core

C:\DALi>git clone https://github.com/dalihub/dali-core/
C:\DALi>cd dali-core\build\tizen
C:\DALi\dali-core\build\tizen>mkdir build
C:\DALi\dali-core\build\tizen>cd build
C:\DALi\dali-core\build\tizen\build>cmake -g Ninja . -DCMAKE_TOOLCHAIN_FILE=%VCPKG_FOLDER%/vcpkg/scripts/buildsystems/vcpkg.cmake -DENABLE_PKG_CONFIGURE=OFF -DENABLE_LINK_TEST=OFF -DCMAKE_INSTALL_PREFIX=%DALI_ENV_FOLDER% -DINSTALL_CMAKE_MODULES=ON -Wno-dev ..
C:\DALi\dali-core\build\tizen\build>cmake --build . --target install

dali-adaptor

C:\DALi>%VCPKG_FOLDER%\vcpkg\vcpkg.exe install pthreads:x64-windows
C:\DALi>git clone https://github.com/dalihub/dali-adaptor/
C:\DALi>cd dali-adaptor\build\tizen
C:\DALi\dali-adaptor\build\tizen>mkdir build
C:\DALi\dali-adaptor\build\tizen>cd build
C:\DALi\dali-adaptor\build\tizen\build>cmake -g Ninja . -DCMAKE_TOOLCHAIN_FILE=%VCPKG_FOLDER%/vcpkg/scripts/buildsystems/vcpkg.cmake -DENABLE_PKG_CONFIGURE=OFF -DENABLE_LINK_TEST=OFF -DCMAKE_INSTALL_PREFIX=%DALI_ENV_FOLDER% -DINSTALL_CMAKE_MODULES=ON -DPROFILE_LCASE=windows -Wno-dev ..
C:\DALi\dali-adaptor\build\tizen\build>cmake --build . --target install

dali-toolkit

C:\DALi>git clone https://github.com/dalihub/dali-toolkit/
C:\DALi>cd dali-toolkit\build\tizen
C:\DALi\dali-toolkit\build\tizen>mkdir build
C:\DALi\dali-toolkit\build\tizen>cd build
C:\DALi\dali-toolkit\build\tizen\build>cmake -g Ninja . -DCMAKE_TOOLCHAIN_FILE=%VCPKG_FOLDER%/vcpkg/scripts/buildsystems/vcpkg.cmake -DENABLE_PKG_CONFIGURE=OFF -DENABLE_LINK_TEST=OFF -DCMAKE_INSTALL_PREFIX=%DALI_ENV_FOLDER% -DINSTALL_CMAKE_MODULES=ON -DUSE_DEFAULT_RESOURCE_DIR=ON -Wno-dev ..
C:\DALi\dali-toolkit\build\tizen\build>cmake --build . --target install

dali-demo

dali-demo contains a series of sample applications that you can try to test if your installation is ok. The procedure to build and install these apps are the same as for the libraries:

C:\DALi>git clone https://github.com/dalihub/dali-demo/
C:\DALi>cd dali-demo\build\tizen
C:\DALi\dali-demo\build\tizen>mkdir build
C:\DALi\dali-demo\build\tizen>cd build
C:\DALi\dali-demo\build\tizen>cmake -g Ninja . -DCMAKE_TOOLCHAIN_FILE=%VCPKG_FOLDER%/vcpkg/scripts/buildsystems/vcpkg.cmake -DENABLE_PKG_CONFIGURE=OFF -DINTERNATIONALIZATION=OFF -DCMAKE_INSTALL_PREFIX=%DALI_ENV_FOLDER% -Wno-dev ..
C:\DALi\dali-demo\build\tizen\build>cmake --build . --target install

Running

If everything was ok, your binaries should reside inside inside the DALI_ENV_FOLDER directory.

DALi depends on libANGLE, an OpenGL ES library that translates OpenGL calls to one of several supported backends. On Windows, the default backend is Direct3D 11. I was having problems with the D3D11 backend on my machine when trying to run the DALi sample applications. What I did was build libANGLE disabling the D3D11 backend, falling back to Direct X 9. The build instructions tell how to disable the D3D11 backend.

With all set, you can run a sample application:

C:\DALi\dali-env\bin>hello-world.example.exe

On the day I write this post, I have some patches merged in the DALi repos and others are pending to review. If you are experiencing crashes you may want to give a look at the open pull requests and apply some of them.

Running packet.net images in qemu

Fri, 14 Dec 2018 00:00:00 +0000

For the past months, I have been working on adding Taskcluster support for packet.net cloud provider. The reason for that is to get faster Firefox for Android CI tests. Tests showed that jobs run up to 4x faster on bare metal machines than EC2.

I set up 25 machines to run a small subset of the production tasks, and so far results are excellent. The problem is that those machines are up 24/7 and there is no dynamic provisioning. If we need more machines, I have to manually change the terraform script to scale it up. We need a smart way to do that. We are going to build something similar as aws-provisioner. However, we need a custom packet.net image to speed up instance startup.

The problem is that if you can’t ssh into the machine, there is no way to get access to it to see what’s wrong. In this post,l I am going to show how you can run a packet image locally with qemu.

You can find documentation about creating custom packet images here and here.

Let’s create a sample image for the post. After you clone the packet-images repo, run:

$ ./tools/build.sh -d ubuntu_14_04 -p t1.small.x86 -a x86_64 -b ubuntu_14_04-t1.small.x86-dev

This creates the image.tar.gz file, which is your packet image. The goal of this post is not to guide you on creating your custom image; you can refer to the documentation linked above for this. The goal here is, once you have your image, how you can run it locally with qemu.

The first step is to create a qemu disk to install the image into it:

$ qemu-img create -f raw linux.img 10G

This command creates a raw qemu image. We now need to create a disk partition:

$ cfdisk linux.img

Select dos for the partition table, create a single primary partition and make it bootable. We now need to create a loop device to handle our image:

$ sudo losetup -Pf linux.img

The -f option looks for the first free loop device for attachment to the image file. The -P option instructs losetup to read the partition table and create a loop device for each partition found; this avoids we having to play with disk offsets. Now let’s find our loop device:

$ sudo losetup -l
NAME        SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                         DIO LOG-SEC
/dev/loop1          0      0         1  1 /var/lib/snapd/snaps/gnome-calculator_260.snap      0     512
/dev/loop8          0      0         1  1 /var/lib/snapd/snaps/gtk-common-themes_818.snap     0     512
/dev/loop6          0      0         1  1 /var/lib/snapd/snaps/core_5662.snap                 0     512
/dev/loop4          0      0         1  1 /var/lib/snapd/snaps/gtk-common-themes_701.snap     0     512
/dev/loop11         0      0         1  1 /var/lib/snapd/snaps/gnome-characters_139.snap      0     512
/dev/loop2          0      0         1  1 /var/lib/snapd/snaps/gnome-calculator_238.snap      0     512
/dev/loop0          0      0         1  1 /var/lib/snapd/snaps/gnome-logs_45.snap             0     512
/dev/loop9          0      0         1  1 /var/lib/snapd/snaps/core_6034.snap                 0     512
/dev/loop7          0      0         1  1 /var/lib/snapd/snaps/gnome-characters_124.snap      0     512
/dev/loop5          0      0         1  1 /var/lib/snapd/snaps/gnome-3-26-1604_70.snap        0     512
/dev/loop12         0      0         0  0 /home/walac/work/packet-images/linux.img            0     512
/dev/loop3          0      0         1  1 /var/lib/snapd/snaps/gnome-system-monitor_57.snap   0     512
/dev/loop10         0      0         1  1 /var/lib/snapd/snaps/gnome-3-26-1604_74.snap        0     512

We see that our loop device is /dev/loop12. If we look in the /dev directory:

$ ls -l /dev/loop12*
brw-rw---- 1 root   7, 12 Dec 17 10:39 /dev/loop12
brw-rw---- 1 root 259,  0 Dec 17 10:39 /dev/loop12p1

We see that, thanks to the -P option, losetup created the loop12p1 device for the partition we have. It is time to set up the filesystem:

$ sudo mkfs.ext4 -b 1024 /dev/loop12p1 
mke2fs 1.44.4 (18-Aug-2018)
Discarding device blocks: done                            
Creating filesystem with 10484716 1k blocks and 655360 inodes
Filesystem UUID: 2edfe9f2-7e90-4c35-80e2-bd2e49cad251
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409, 663553, 
        1024001, 1990657, 2809857, 5120001, 5971969

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done     

Ok, finally we can mount our device and extract the image to it:

$ mkdir mnt
$ sudo mount /dev/loop12p1 mnt/
$ sudo tar -xzf image.tar.gz -C mnt/

The last step is to install the bootloader. As we are running an Ubuntu image, we will use grub2 for that.

Firstly we need to install grub in the boot sector:

$ sudo grub-install --boot-directory mnt/boot/ /dev/loop12
Installing for i386-pc platform.
Installation finished. No error reported.

Notice we point to the boot directory of our image. Next, we have to generate the grub.cfg file:

$ cd mnt/
$ for i in /proc /dev /sys; do sudo mount -B $i .$i; done
$ sudo chroot .
# cd /etc/grub.d/
# chmod -x 30_os-prober
# update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-3.13.0-123-generic
Found initrd image: /boot/initrd.img-3.13.0-123-generic
done

We bind mount the /dev, /proc and /sys host mount points inside the Ubuntu image, then chroot into it. Next, to avoid grub creating entries for our host OSes, we disable the 30_os-prober script. Finally we run update-grub and it creates the /boot/grub/grub.cfg file. Now the only thing left is cleanup:

# exit
$ for i in dev/ sys/ proc/; do sudo umount $i; done
$ cd ..
$ sudo umount mnt/

The commands are self explanatory. Now let’s run our image:

$ sudo qemu-system-x86_64 -enable-kvm -hda /dev/loop12

And that’s it, you now can run your packet image locally!

A tale of my first Go patch

Sat, 11 Mar 2017 00:00:00 +0000

For the last year I have been using the Go programming language, working on the implementation of the taskcluster-worker project. If you want to learn more about taskcluster-worker, you can read this post.

In the taskcluster-worker task payload, there is a field called command, that specifies the command the task must run. Through a configuration file, you can specify either the spawned command should run as the current user or to run the command with a newly created user. The pseudo code bellow illustrates how this works:

var usr *User
if createUser {
  usr = CreateUser()
} else {
  usr = CurrentUser()
}

RunCommand(command, usr)

if createUser {
  DeleteUser(usr)
}

Our first production use for taskcluster-worker it to run Firefox automated tests inside the OSX environment. We began configuring taskcluster-worker to run each task with a new user to provide some kind of task isolation, but it didn’t work well. Some tests, as you might wonder, require we run the browser UI inside a desktop session. To allow this, the taskcluster-worker daemon runs as a Launch Agent. The problem is that the test itself runs as a different user, under the desktop session owned by the taskcluster-worker user. As the user that runs Firefox is different from the user owning the desktop session, the tests weren’t allowed to execute several operations, like clipboard copy/paste. Because of this that we introduced the option to run the test as the current user.

When implementing the option to run tasks as the current user, I found a subtle problem with Go process creation package, described next.

The problem

The os/exec package allows you to spawn new processes. For example, the following code will spawn the echo command and print the “hello-world” string:

package main

import (
  "fmt"
  "os"
  "os/exec"
)

func main() {
  cmd := exec.Command("echo", "hello-world")
  cmd.Stdout = os.Stdout
  cmd.Stderr = os.Stderr
  if err := cmd.Run(); err != nil {
    fmt.Println(err)
  }
}

Here we use the Command function to create a Cmd structure. We then connect the stdout and stdin channels of the child process to the parent process and spawn the new process. It runs as expected:

$ go run echo.go
hello-world

So far so good. You can also set the user in which the process will run. Let’s demonstrate it by expliciting setting the current user as the user account of the child process:

package main

import (
  "fmt"
  "os"
  "os/exec"
  "os/user"
  "strconv"
  "syscall"
)

func main() {
  usr, err := user.Current()
  if err != nil {
    panic(err)
  }

  uid, err := strconv.Atoi(usr.Uid)
  if err != nil {
    panic(err)
  }

  gid, err := strconv.Atoi(usr.Gid)
  if err != nil {
    panic(err)
  }

  cmd := exec.Command("echo", "hello-world")
  cmd.Stdout = os.Stdout
  cmd.Stderr = os.Stderr
  cmd.SysProcAttr = &syscall.SysProcAttr{
    Credential: &syscall.Credential{
      Uid: uint32(uid),
      Gid: uint32(gid),
    },
  }
  if err = cmd.Run(); err != nil {
    fmt.Println(err)
  }
}

The exec.Cmd structure accepts a system specific SysProcAttr structure pointer, which has a Credential field used to set the user and group ids of the child process. This is a no-op operation, as by default the child process runs with the same user and group ids of its parent. But when we run it:

$ go run echo.go
fork/exec /bin/echo: operation not permitted

Wait! Can’t we run a process with our own account??? This is non-sense. Let’s investigate the root cause of this weird behavior. A system call must be returning EPERM and causing the whole thing to collapse. We can use dtrace to discover exactly what system call is failing:

$ sudo dtrace -n 'syscall:::return/execname == "echo" && arg0 < 0 && errno == EPERM/{}'
dtrace: description 'syscall:::return' matched 522 probes
CPU     ID                    FUNCTION:NAME
  0    338                 setgroups:return

The above dtrace script logs all system calls executing in the context of our process that return EPERM error. Explaining the details of dtrace is beyond the scope of this post, please refer to the dtrace website to find the documentation about it.

The output shows that the setgroups system call is the guilty one. If you look at the system call manual page you see that it requires admin privileges to succeed. But, why is Go runtime calling it? Below you see the relevant piece of code inside Go syscall package (as I am running a macOS, this is the code specific for BSD kernels, Linux and Solaris specific source codes are similar):

// User and groups
if cred := sys.Credential; cred != nil {
	ngroups := uintptr(len(cred.Groups))
	groups := uintptr(0)
	if ngroups > 0 {
		groups = uintptr(unsafe.Pointer(&cred.Groups[0]))
	}
	_, _, err1 = RawSyscall(SYS_SETGROUPS, ngroups, groups, 0)
	if err1 != 0 {
		goto childerror
	}
	_, _, err1 = RawSyscall(SYS_SETGID, uintptr(cred.Gid), 0, 0)
	if err1 != 0 {
		goto childerror
	}
	_, _, err1 = RawSyscall(SYS_SETUID, uintptr(cred.Uid), 0, 0)
	if err1 != 0 {
		goto childerror
	}
}

The reason is that the Credential struct has a Groups field representing an array of supplementary group ids to pass to setgroups, and when Credential != nil, setgroups gets always called, even if didn’t set the Groups property. And that’s where the problem lies, you can’t set process user and group id without issuing a call to setgroups. The solution is to find a way to say “don’t bother with supplementary groups”.

Attempt #1: don’t call `setgroups` if `len(Groups) == 0`

My first solution was the obvious one, don’t call setgroups if the length of Groups is zero. It works fine, except for one detail: we now can’t clear the supplementary groups of the child process, which is, above all, a security problem.

Attempt #2: don’t call `setgroups` if `Groups == nil`

After the security concern, my idea was to distinguish a empty group ([]uint32{}) from a nil group. If Groups == nil, we skip setgroups, otherwise we call it. This removes the previous issue, but introduces another: it subtly breaks backward compatibility. Previously, if you left Groups intact, the child process would always try to clear supplementary groups, now it doesn’t.

Attempt #3: call `setgroups` only if `Groups` and `getgroups` mismatch

One of the code reviewers suggested to call getgroups to get the current set of supplementary groups and only call setgroups if the current and new groups set mismatch. Implementing this was trickier than you might think. Here is a fairly straightforward implementation:

func groupsMismatch(g1, g2 []uint32) bool {
	if len(g1) != len(g2) {
		return true
	}
	table := map[uint32]struct{}
	for _, g := range g1 {
		table[g] = struct{}{}
	}
	for _, g := range g2 {
		if _, ok := table[g]; !ok {
			return true
		}
	}

	return false
}

// ...

// User and groups
if cred := sys.Credential; cred != nil {
	ngroups := uintptr(len(cred.Groups))
	groups := uintptr(0)
	if ngroups > 0 {
		groups = uintptr(unsafe.Pointer(&cred.Groups[0]))
	}
	var groupsize uintptr
	groupsize, _, err1 = RawSyscall(SYS_GETGROUPS, 0, 0, 0)
	if err1 != 0 {
		goto childerror
	}
	cgroups := make([]uint32, groupsize)
	groupsize, _, err1 = RawSyscall(SYS_GETGROUPS, groupsize, uintptr(unsafe.Pointer(&cgroups[0])), 0)
	if err1 != 0 {
		goto childerror
	}
	if groupsMismatch(cgroups, cred.Groups) {
		_, _, err1 = RawSyscall(SYS_SETGROUPS, ngroups, groups, 0)
		if err1 != 0 {
			goto childerror
		}
	}
	_, _, err1 = RawSyscall(SYS_SETGID, uintptr(cred.Gid), 0, 0)
	if err1 != 0 {
		goto childerror
	}
	_, _, err1 = RawSyscall(SYS_SETUID, uintptr(cred.Uid), 0, 0)
	if err1 != 0 {
		goto childerror
	}
}

This is a classic linear time array mismatch algorithm implemented in the groupsMismatch function. By definition, the array of groups are unique, so we don’t have to worry about duplicated values. When the current supplementary group set is different from the new one, we call setgroups to apply the new group set. When we run it:

wcosta@Wanders-MBP:~ $ work/golang/bin/go run test.go
fatal error: stack growth after fork

runtime stack:
runtime.throw(0x10d65ca, 0x17)
	/Users/wcosta/work/golang/src/runtime/panic.go:596 +0x95
runtime.newstack(0x0)
	/Users/wcosta/work/golang/src/runtime/stack.go:965 +0xcea
runtime.morestack()
	/Users/wcosta/work/golang/src/runtime/asm_amd64.s:398 +0x86

goroutine 1 [running]:
runtime.makeslice(0x10b1940, 0xe, 0xe, 0x0, 0xe, 0x0)
	/Users/wcosta/work/golang/src/runtime/slice.go:39 +0xf6 fp=0xc4200499d8 sp=0xc4200499d0
syscall.forkAndExecInChild(0xc4200103d0, 0xc42000c380, 0x3, 0x3, 0xc420082000, 0x24, 0x24, 0x0, 0x0, 0xc420049cd8, ...)
	/Users/wcosta/work/golang/src/syscall/exec_bsd.go:171 +0x707 fp=0xc420049ad0 sp=0xc4200499d8
syscall.forkExec(0xc420010360, 0x9, 0xc42000c260, 0x2, 0x2, 0xc420049cd8, 0x20, 0xc42000c360, 0xc42001c000)
	/Users/wcosta/work/golang/src/syscall/exec_unix.go:193 +0x368 fp=0xc420049bf0 sp=0xc420049ad0
syscall.StartProcess(0xc420010360, 0x9, 0xc42000c260, 0x2, 0x2, 0xc420049cd8, 0x2, 0x4, 0xc420016180, 0xc420049ca8)
	/Users/wcosta/work/golang/src/syscall/exec_unix.go:240 +0x64 fp=0xc420049c48 sp=0xc420049bf0
os.startProcess(0xc420010360, 0x9, 0xc42000c260, 0x2, 0x2, 0xc420049e80, 0xc420012480, 0x23, 0x23)
	/Users/wcosta/work/golang/src/os/exec_posix.go:45 +0x1fb fp=0xc420049d30 sp=0xc420049c48
os.StartProcess(0xc420010360, 0x9, 0xc42000c260, 0x2, 0x2, 0xc420049e80, 0x0, 0x0, 0xc42006e750)
	/Users/wcosta/work/golang/src/os/exec.go:94 +0x64 fp=0xc420049d88 sp=0xc420049d30
os/exec.(*Cmd).Start(0xc420080000, 0xc42000c201, 0xc42000c300)
	/Users/wcosta/work/golang/src/os/exec/exec.go:359 +0x502 fp=0xc420049ed8 sp=0xc420049d88
os/exec.(*Cmd).Run(0xc420080000, 0xc42000c300, 0xc420049f68)
	/Users/wcosta/work/golang/src/os/exec/exec.go:277 +0x2b fp=0xc420049f00 sp=0xc420049ed8
main.main()
	/Users/wcosta/test.go:36 +0x21d fp=0xc420049f88 sp=0xc420049f00
runtime.main()
	/Users/wcosta/work/golang/src/runtime/proc.go:185 +0x20a fp=0xc420049fe0 sp=0xc420049f88
runtime.goexit()
	/Users/wcosta/work/golang/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420049fe8 sp=0xc420049fe0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/Users/wcosta/work/golang/src/runtime/asm_amd64.s:2197 +0x1
exit status 2

Before diving into the error, let’s talk a bit about the history of the fork system call. When fork was created, there was no concept of threads, forking the process was the way multiprocessing was achieved. As you probably know, when fork is called, a child process is spawned which is identical to the parent process, like memory values and files opened. But what if a multithreaded process calls fork? The answer is that only the thread that invoked the system call is copied to the child process, but the state of the global memory remains the same, and that includes the state of mutexes and semaphores held in another thread. That means you can execute a very restrict set of operations in the child process. Allocating memory from the heap is dangerous, as the heap is often protected by a mutex, and another thread could be in the middle of a heap allocation when the process was forked. Trying to allocate memory from the heap in this circumstance will deadlock the child process.

Another thing you need to know is that the goroutine stack is resizable, and they are quite small when goroutines are created. If you are interested in the details of goroutines stack management, I recommend this blog post.

Ok, with this background I can now explain what happened. The stack trace points to the line cgroups := make([]uint32, groupsize), which tries to allocate a new array in the stack, the stack limit is reached and then the go runtime tries to allocate more memory to the stack, and the go runtime detects it is a forked process and panics. The conclusion is that even a function call may cause a runtime panic.

In the end, to implement this patch I had to declare a global []uint32 large enough to store the groups returned by getgroups (in environments I looked at, it doesn’t need to be more than 16 entries large) and implemented an $O(n^2)$ but constant space matching algorithm.

It turns out it actually didn’t reach its goal, because often a new child process inherits the parent’s supplementary groups. Therefore, even so the caller doesn’t care about supplementary groups, setgroups can be called anyway.

Accepted solution: define a new member in the `Credential` struct

After all previous attempts, I admitted defeat and added a NoSetGroups member to the Credential struct so that when it is true, it skips supplementary groups management. In a ideal world it would be called SetGroups with the logic inverted, but I had to do so in order to preserve backward compatibility.

If you are curious about the final patch, just look at the git commit.

By the end of the day, it is just astonishing how a subtle issue with a so simple patch yielded so much trouble. Finding the perfect solution is often almost impossible, but even finding a good one isn’t that simple.

Yet another "Java faster than C++" claim

Tue, 23 Aug 2016 00:00:00 +0000

Some days ago, a friend emailed me desperated because a Java code was faster than his C++ implementation. These Java faster than C++ claims were very common when Java was the tech companies fashion language. I never liked Java because I always thought it was quite verbose (however, I never tried the new functional constructions in Java 8) and used to be very heavy. I mean, I could recognize that an app was written in Java in a blink of eyes, just by looking how slow it was. Things have changed since, but as I was never a professional Java programmer, I didn’t follow the language evolution.

Back to the post, the code in question is a Huffman code implementation. He based his C++ implementation on Sedgewick’s Java version. If you want to profile on your own, you can clone his repo (as he already merged my patches, you’ll need to rewind git history a bit to profile the old code). You can find instructions on how install the Java code here.

Let’s get it started by using the simple time command. In this test, we are measuring the time to compress and decompress the sample mobydick.txt file. First let’s profile the compression and decompression times for the Java version:

$ time java edu.princeton.cs.algs4.Huffman - < mobydick.txt > modydick.dwc

real  0m0.223s
user  0m0.255s
sys 0m0.034s

$ time java edu.princeton.cs.algs4.Huffman + < mobydick.dwc > mobydick.txt

real  0m0.186s
user  0m0.196s
sys 0m0.035s

Now, the C++ implementation:

$ time ./Huffman -e < mobydick.txt > mobydick.dwc

real  0m0.380s
user  0m0.364s
sys 0m0.009s

$ time ./Huffman -d < mobydick.dwc > mobydick.txt

real  0m0.352s
user  0m0.330s
sys 0m0.010s

Just for the record, I profiled the second execution of each command to make sure results won’t differ because of file caching.

Ok, C++ implementation is indeed slower, the question is why? After a rough look at the source code, the heap allocations were quite suspicious, but after some experiments I realized this was not the case, then I started to profile the code. If you know me, you also know I am a very keen Linux user, but I had to switch to Mac OS X for a project I am working on at Mozilla. As I have no much experience on Mac development, and that includes native tools, I made some google research and came across Instruments. After some painful time trying to use it inside Xcode, I came back to google and found iprofiler. That was a life saver, I could profile from command line and see the results on Instruments GUI. Instruments come with a set of Profiler templates, the one of interest here is the time profiler, which as its name says, profiles the time spent on function calls. Let’s start by profiling the compression operation:

$ iprofiler -timeprofiler -I 100us bin/Huffman -e < mobydick.txt > mobydick.bwc

And now let’s look at the results:

The program spends most of the time reading the input file. Why that? Here is a simplified version of the code that reads the input file:

std::string data(
    std::istream_iterator<char>(std::cin),
    std::istream_iterator<char>()
);

The problem lies on using std::cin to read the file. ISO C++11 says that standard streams objects must be thread safe, that means a mutex lock/unlock operation is executed each time the internal I/O buffer is accessed. In the code above, a mutex is locked for each character read, which makes things painfully slow. The solution is to use a std::fstream object, as it has no imposed lock contention. Obviously, all this applies to file writing as well, and we take the same path of replacing std::cout by std::fstream.

Now let’s see how much we improved compression and decompression performance:

$ time bin/Huffman -e < mobydick.txt > mobydick.bwc

real  0m0.108s
user  0m0.095s
sys 0m0.007s

$ time bin/Huffman -d -o mobydick.txt -i mobydick.bwc

real 0m0.215s
user 0m0.201s
sys  0m0.007s

250% faster for compression and 63% for decompression!!!

Can we make this better? Hrm, lets profile it one more time, but this time, we will profile decompression, since it is still slower than its Java counterpart:

Wait! What are those __shared_weak_count::__release_shared/__add_shared calls consuming almost half of the running time? If we take a look at the implementation, we see it uses std::shared_ptr to allocate the nodes for the Huffman tree. As you might know, shared_ptr provides a smart pointer with a thread safe, reference counted, copy semantics. The code is single threaded, so this thread safe reference counting is a waste of resources, but the point is that shared_ptr is thread safe, and not going to suggest to implement a single thread version of it. What do we do? Well, a careful review on the code shows that we actually don’t need a smart pointer with copy semantics, the pointer is never shared, but transferred from one owner to another, so what we need is moving semantics, and we have a perfect smart pointer with this characteristic: std::unique_ptr.

By replacing std::shared_ptr with std::unique_ptr, we are free from the reference counting contention. Let’s see how fast we got with this change:

$ time bin/Huffman -e -i mobydick.txt -o mobydick.bwc

real  0m0.107s
user  0m0.092s
sys 0m0.008s

$ time bin/Huffman -d -o mobydick.txt -i mobydick.bwc

real  0m0.120s
user  0m0.109s
sys 0m0.007s

Wow! Decompression now, compared to the original code, is 193% faster!

C++ code got a lot faster, and we achieved this with only intelligent use of the standard C++ library, no need to implement lock free data structures, or fancy algorithms or either special processor features. Sometimes, less is more. If you want to take a look at the actual source code, you can visit my pull request (there is an extra commit reversing a flawed attempt to optimizes the code preserving std::cin). If I was asked to try to make the code even faster, I would give file mapping a try, since file I/O seems to be the bottleneck here. Maybe I do that in the future, and publish the results here.

The taskcluster-worker Mac OSX engine

Mon, 27 Jun 2016 00:00:00 +0000

In this quarter, I worked on implementing the taskcluster-worker Mac OSX engine. Before talking about this specific implementation, let me explain what a worker is and how taskcluster-worker differs from docker-worker, the currently main worker in Taskcluster.

The role of a Taskcluster worker

When a user submits a task graph to Taskcluster, contrary to the common sense (at least if you are used on how OSes schedulers usually work), these tasks are submitted to the scheduler first, which is responsible to process dependencies and enqueue them. In the Taskcluster manual page there is a clear picture ilustrating this concept.

The provisioner is responsible for looking at the queue and determine how many pending tasks exist and, based on that, it launches worker instances to run these tasks.

Then comes the figure of the worker. The worker is responsible for actually executing the task. It claims a task from the queue, runs it, upload the generated artifacts and submits the status of the finished task, using the Taskcluster APIs.

docker-worker is a worker that runs task command inside a docker container. The task payload specifies a docker image as well as a command line to run, among other environment parameters. docker-worker pulls the specified docker image and runs task commands inside it.

taskcluster-worker and the OSX engine

taskcluster-worker is a generic and modularized worker under active development by the Taskcluster team. The worker delegates the task execution to one of the available engines. An engine is a component of taskcluster-worker responsible for running a task under a specific system environment. Other features, like environment variable setting, live logging, artifact uploading, etc., are handled by worker plugins.

I am implementing the Mac OSX engine, which will mainly be used to run Firefox automated tests in the Mac OSX environment. There is a macosx branch in my personal Github taskcluster-worker fork in which I push my commits.

One specific aspect of the engine implementation is the ability to run more than one task at the same time. For this, we need to implement some kind of task isolation. For docker-worker, each task ran in its own docker container so tasks were isolated by definition. But there is no such thing as a container for OSX engine. Our earlier tries with chroot failed miserably, due to incompatibilities with OSX graphic system. Our final solution was to create a new user on the fly and run the task with this user’s credentials. This not only provides some task isolation, but also prevents privilege escalation attacks by running tasks with different user than the worker.

Instead of dealing with the poorly documented Open Directory Framework, we chose to spawn the dscl command to create and configure users. Tasks usually takes a long time to execute, spawning loads of subprocess, so a few spawns of the dscl command won’t have any practical performance impact.

One final aspect is how we bootstrap task execution. A tasks boils down to a script that executes task duties. But where does this script come from? It doesn’t live in the machine that executes the worker. OSX engine provides a link field in task payload that a task can specify an executable to download and execute.

Running the worker

OSX engine will primarily be used to execute Firefox tests on Mac OSX, and the environment is expected to have a very specific tools and configurations set. Because of that, I am testing the code on a loaner machine. To start the worker, it is just a matter of opening a terminal and typing:

$ ./taskcluster-worker work macosx --logging-level debug

The worker connects to the Taskcluster queue, claims and execute the tasks available. At the time I am writing, all tests but Firefox UI functional tests” were green, running on optimized Firefox OSX builds. We intend to land Firefox tests in taskcluster-worker as Tier-2 on next quarter, running them in parallel with Buildbot.

Overcoming browser same origin policy

Fri, 01 Apr 2016 00:00:00 +0000

One of my goals for 2016 Q1 was to write a monitoring dashboard for Taskcluster. It basically pings Taskcluster services to check if they are alive and also acts as a feed aggregator for services Taskcluster depends on. One problem with this approach is the same origin policy, in which web pages are only allowed to make requests to their own domain. For web servers which is safe to make these cross domain requests, they can either implement jsonp or CORS. CORS is the preferred way so we will focus on it for this post.

CORS is a mechanism that allows the web server tell the browser that is safe to accomplish a cross domain request. It consists of a set of HTTP headers with details for the conditions to accomplish the request. The main response header is Access-Control-Allow-Origin, which contains either a list of allowed domains or a *, indicating any domain can make a cross request to this server. In a CORS request, only a small set of headers is exposed to the response object. The server can tell the browser to expose additional headers through the Access-Control-Expose-Headers response header.

But what if the web server doesn’t implement CORS? The only solution is to provide a proxy that will make the actual request and add the CORS headers.

cors-proxy

To allow the monitoring dashboard make requests for status state on remote services that do not implement CORS, we created the cors-proxy. It exports a /request endpoint that allows you to make requests to any remote host. cors-proxy redirects it to the remote URL and sends the responses back, with appropriate CORS headers set.

Let’s see an example:

$.ajax({
  url: 'https://cors-proxy.taskcluster.net/request',
  method: 'POST',
  contentType: 'application/json',
  data: {
    url: 'https://queue.taskcluster.net/v1/ping',
  }
}).done(function(res) {
  console.log(res);
});

The information about the remote request is sent in the proxy request body. All parameter fields are shown in the project page.

Before you think on using the hosted server to bypass your own requests, cors-proxy only honors requests from a whitelist. So, only some subdomains under Taskcluster domain can use cors-proxy.

In tree tasks configuration

Fri, 09 Oct 2015 00:00:00 +0000

This post is about our plans for representing Taskcluster tasks inside the gecko tree. Jonas, Dustin and I had a discussion in Berlin about this, here I summarize what we have so far. We currently store tasks in an yaml file and they translate to json format using the mach command. The syntax we have now is not the most flexible one, it is hard to parameterize the task and very difficulty to represents tasks relationships.

Let us illustrate the shortcomings with two problems we currently have. Both apply to B2G.

B2G (as in Android) has three different build variants: user, userdebug and eng. Each one has slightly different task configurations. As there is no flexible way to parameterize tasks, we end up with one different task file for each build variant.

When doing nightly builds, we must send update data to the OTA server. We have plans to run a build task, then run the test tasks on this build, and if all tests pass, we run a task responsible to update the OTA server. The point is that today we have no way to represent this relationship inside the task files.

For the first problem Jonas has a prototype for json parameterization. There were discussions on Berlin work week either we should stick with yaml files or use Python files for task configuration. We do want to keep the syntax declarative, which favors yaml, but storing configurations in Python files brings much more expressiveness and flexibility, but this can result in the same configuration hell we have with Buildbot.

The second problem is more complex, and we still haven’t reached a final design. The first question is how we describe task dependencies, top-down, i.e., we specify which task(s) should run after a completed task, or ground up, a task specifies which tasks it depends on. In general, we all agreed to go to a top-down syntax, since most scenarios beg for a top down approach. Other either should put the description of tasks relationship inside the task files or in a separated configuration file. We would like to represent task dependencies inside the task file, the problem is how to check what’s the root task for the task graph. One suggestion is having a task file called root.yml which only contain root tasks.

Running phone builds on Taskcluster

Mon, 05 Oct 2015 00:00:00 +0000

In this post I am going to talk about my work for phone builds inside the Taskcluster infrastructure. Mozilla is slightly moving from Buildbot to Taskcluster. Here I am going to give a survivor guide on Firefox OS phone builds.

Submitting tasks

A task is nothing more than a json file containing the description of the job to execute. But you don’t need to handle the json directly, all tasks are written in YAML, and it is then processed by the mach command. The in tree tasks are located at testing/taskcluster/tasks and the build tasks are inside the builds/ directory.

My favorite command to try out a task is the mach taskcluster-build command. It allows you to process a single task and output the json formatted task ready for Taskcluster submission.

$ ./mach taskcluster-build \
    --head-repository=https://hg.mozilla.org/mozilla-central 
    --head-rev=tip \
    --owner=foobar@mozilla.com \
    tasks/builds/b2g_desktop_opt.yml

Although we specify a Mercurial repository, Taskcluster also accepts git repositories interchangeably.

This command will print out the task to the console output. To run the task, you can copy the generated task and paste it in the task creator tool. Then just click on Create Task to schedule it to run. Remember that you need Taskcluster Credentials to run Taskcluster tasks. If you have taskcluster-cli installed, you can the pipe the mach output to taskcluster run-task.

The tasks are effectively executed inside a docker image.

Mozharness

Mozharness is what we use for effectively build stuff. Mozharness architecture, despite its code size, is quite simple. Under the scripts directory you find the harness scripts. We are specifically interested in the b2g_build.py script. As the script name says, it is responsible for B2G builds. The B2G harness configuration files are located at the b2g/config directory. Not surprisingly, all files starting with “taskcluster” are for Taskcluster related builds.

Here are the most common configurations:

default_vcs: This is the default vcs used to clone repositories when no other is given. [tc_vcs](https://tc-vcs.readthedocs.org/en/latest/) allows mozharness to clone either git or mercurial repositories transparently, with repository caching support.
default_actions: The actions to execute. They must be present and in the same order as in the build class `all_actions` attribute.
balrog_credentials_file: The credentials to send update data to the OTA server.
nightly_build: `True` if this is a nightly build.
upload: Upload info. Not used for Taskcluster.
repo_remote_mappings: Maps externals repository to [mozilla domain](https://git.mozilla.org).
env: Environment variables for commands executed inside mozharness.

The listed actions map to Python methods inside the build class, with - replaced by _. For example, the action checkout-sources maps to the method checkout_sources. That’s where the mozharness simplicity comes from: everything boils down to a sequence of method calls, just it, no secret.

For example, here is how you run mozharness to build a flame image:

python <gecko-dir>/testing/mozharness/scripts/b2g_build.py \
  --config b2g/taskcluster-phone.py \
  --disable-mock \
  --variant=user \
  --work-dir=B2G \
  --gaia-languages-file locales/languages_all.json \
  --log-level=debug \
  --target=flame-kk \
  --b2g-config-dir=flame-kk \
  --repo=https://hg.mozilla.org/mozilla-central \

Remember you need your flame connected to the machine so the build system can extract the blobs.

In general you don’t need to worry about mozharness command line because it is wrapped by the build scripts.

Hacking Taskcluster B2G builds

All Taskcluster tasks run inside a docker container. Desktop and emulator B2G builds run inside the builder docker image. Phone builds are more complex, because:

Mozilla is not allowed to publicly redistribute phone binaries.
Phone build tasks need to access the Balrog server to send OTA update data.
Phone build tasks need to upload symbols to the crash reporter.

Due to (1), only users authenticated with a @mozilla account are allowed to download phone binaries (this works the same way as private builds). And because of (1), (2) and (3), the phone-builder docker image is secret, so only authorized users can submit tasks to it.

If you need to create a build task for a new phone, most of the time you will starting from an existing task (Flame and Aries tasks are preferred) and then make your customizations. You might need to add new features to the build scripts, which currently are not the most flexible scripts around.

If you need to customize mozharness, make sure your changes are Python 2.6 compatible, because mozharness is used to run Buildbot builds too, and the Buildbot machines run Python 2.6. The best way to minimize risk of breaking stuff is to submit your patches to try with “-p all -b do” flags.

Need help? Ask at the #taskcluster channel.

Mercurial for git lovers

Sat, 03 Jan 2015 00:00:00 +0000

So far I have been a heavy git user, but since I joined Mozilla I have given Mercurial a try. I must say that migrating from Subversion to git was far less painful than from git to Mercurial. I think there are a few reasons for that:

When I started to learn git, I started it with PyUSB, a small personal project that was on its youth at that time. This means I needed just the most basic commands to start using it. When I started on Mercurial, I was working on Gecko, a very big and complex project, with thousands of contributors, so I had to learn more advanced commands from day one.
When you install git, you have all you need, all commands you need are there. Mercurial, in the other hand, operates in a very different way. It just ships with some basic commands, delegating most of the more advanced stuff to extensions. So you have to dig to find the extensions that enable the Mercurial approach to commands you love in git.
There is nothing like git branches parallel in Mercurial, and 9/10 of git users’ workflow rely heavily on branches (more on that later).

Most of the day to day commands in Mercurial are quite similar to git, like clone and commit. Some others have very different names, like git revert, which in Mercurial is called backout. hg revert is used to discard non-committed changes in your repository.

I prepared a list of some Mercurial extensions that will make your life easier when coming from git. These extensions will make you feel more comfortable while moving from git to Mercurial.

git feature	Mercurial extension
color	Color
git rebase	Rebase
Interactive rebase	Histedit
git stash	Shelve
git clean	Purge
git add -i	Record
gitk	Hgk
__git_ps1	Prompt
git cherry-pick	Transplant
git send-email	PatchBomb
git am	Mbox
Topic branches	Bookmarks
Paged log	Pager

This link has a lot more information about differences between git and Mercurial, including Mercurial counterparts for several git actions.

Mercurial comes with an important feature that there is no equivalent in git at all, called phases. I mention it here because you may have problems with it if you (like me) do a lot of rebase and history editing. Every commit you push or pull to/from a remote repository is considered public, which makes it immutable. This happens all the time to me because I often push commits to Mozilla Try and reviewboard. Thus, I generally face errors like this:

    hg rebase -d bookmark0
    abort: can't rebase immutable changeset 35c25b97fca2
    (see hg help phases for details)

That’s because I have pushed this commit and Mercurial made it public (aka immutable). The solution for this is make it a draft commit again:

    hg phase --draft --force bookmark3

I find this feature very annoying and I think it would be better implemented as an extension. It sounds Java telling me what I can and can’t do. You can disable making commits public after a push by adding these entries in your hgrc file:

    [phases]
    publish = False

Special note on Bookmarks

Since Mercurial version 1.8, Bookmarks are now part of Mercurial core. They are often advertised as git branches on Mercurial. They are not! The most difficult part for me was to understand that Mercurial has no branch support like git. Period. Bookmarks are just a hack that tries to mimic topic branches, but, technically, a bookmark is not a branch, it is more like a tag. I am not going to explain how Bookmarks work, you can learn more on it here.

The most irritating thing regarding Bookmarks is that changing the history in one bookmark can affect other bookmarks as well. In some cases, when you create multiple heads descending from a bookmark, you cannot edit its history at all. Let’s try an example. Imagine you fix a bug and submit it for review. While you wait for feedback, you create more bookmarks to work on other product features. You eventually end up with a tree like this:

    @ bookmark 3
    |
    | o bookmark 2
    |/
    |
    | o bookmark 1
    |/
    |
    o bookmark 0 (bug 137463)

The @ indicates the commit representing the current directory.

After a while you receive feedback for bug 137463 and you are requested some changes. You then move back to bookmark 0:

    o bookmark 3
    |
    | o bookmark 2
    |/
    |
    | o bookmark 1
    |/
    |
    @ bookmark 0 (bug 137463)

And start to apply the requested changes, making a new commit:

    o bookmark 3
    |
    | o bookmark 2
    |/
    |
    | o bookmark 1
    |/
    |
    | @ bookmark 0 (bug 137463)
    |/
    |
    o commit 1 (original bug 137453 commit)

bookmark 0 moved to the new commit. What you want now is to squash commit 1 and bookmark 0, but you can’t, because you will mess up bookmarks 1-3. One solution would be to rebase bookmarks 1-3 on top of bookmark 0:

    o bookmark 3
    |
    | o bookmark 2
    |/
    |
    | o bookmark 1
    |/
    |
    @ bookmark 0 (bug 137463)
    |
    o commit 1

You can now histedit from commit 1 and squash it with bookmark 0, right? Wrong! This happens because the structure you see is just an illusion from branching point of view. In git, when you modify a branch, it doesn’t affect descendant branches. In Mercurial this is not true simply because all the commits are in the same branch.

Conclusion

If your git workflow relies heavily on git branching capabilities and history editing, you will have some trouble to adapt yourself to Mercurial (like me).

If I could request just one feature to Mercurial developers, that would be lightweight branches like git. That would give Mercurial a big boost.

There are other problems I had with bookmarks regarding remote repositories, but I will save that for a future post.

Profiling Firefox OS apps

Thu, 16 Oct 2014 00:00:00 +0000

B2G has a built-in profiler that you can use to find hot spots in your app. The process is made easy by the profile.sh script, which helps you to generate a profile file for upload to the Cleopatra application.

Earlier, you had to build your own Firefox OS image to use profile.sh tool, but that’s no longer the case. Thanks to Snappy symbolication server, you can grab a Mozilla nightly image and start to investigate where your app is wasting time.

profile.sh communicates with the device through adb, so you need to enable it in the developer menu. You also need an engineering build image of your device (images with -eng suffix).

In this post, I am going into the steps to get a profile stack trace for your app. In a few words, you have to follow a some steps:

Start the b2g process and your application with profile support.
Perform the actions you want to profile.
Capture the profiles from the device.
Upload profile to Cleopatra.

Starting with profile support

In general, you want to start profile support in the B2G parent process’ compositor thread and your app. The template for this procedure is ./profile.sh start -p b2g -t Compositor && ./profile.sh start -p <your-app>. If you don’t know the app name, you can use the profile.sh ps command to list the B2G processes:

$ ./profile.sh ps
  PID Name
----- ----------------
  208 b2g              profiler not running
  399 (Nuwa)           profiler not running
  896 Built-in Keyboa  profiler not running
 1059 Homescreen       profiler not running
 1251 Usage            profiler not running
 2238 (Preallocated a  profiler not running

For this example, we are going to start profile for Homescreen application:

$ ./profile.sh start -p b2g -t Compositor && ./profile.sh start -p Homescreen
Process: b2g
Threads: Compositor
Using default features js,leaf,threads
Starting profiling PID 208..
Profiler started

Process: Homescreen
Using default features js,leaf
Starting profiling PID 1059..
Profiler started

You should now take your phone and perform the actions you would like to profile.

Getting profile files

Pulling profile data from the devices is performed by running profile.sh capture command. One of the steps performed is symbol resolution. profile capture provides the -s option which allows you to pass the URL of the symbolication server, that perform symbol resolution.

For Firefox OS oficial builds, you can use the Mozilla symbolication server:

$ ./profile.sh capture -s https://symbolapi.mozilla.org
Symbols: https://symbolapi.mozilla.org
Signaling Profiled Processes: 208 1059
Stabilizing 208 b2g ...
Pulling /data/local/tmp/profile_0_208.txt into profile_208_b2g.txt
Adding symbols to profile_208_b2g.txt and creating profile_208_b2g.sym ...
Stabilizing 1059 Homescreen ...
Pulling /data/local/tmp/profile_2_1059.txt into profile_1059_Homescreen.txt
Adding symbols to profile_1059_Homescreen.txt and creating profile_1059_Homescreen.sym ...
Merging profile:  profile_208_b2g.sym profile_1059_Homescreen.sym
./gecko/tools/profiler/merge-profiles.py profile_208_b2g.sym profile_1059_Homescreen.sym

Results: profile_captured.sym
Removing old profile files (from device) ... done

The format of the pulled files is profile_PID_NAME.txt and the files with symbols resolved have the same name but with .sym extension. It also creates a merged file called profile_captured.sym. After that, you can upload the results to Cleopatra:

Symbol resolution through breakpad symbols

That’s not the end of the story. Symbolication server makes use of the google-breakpad symbol files for symbol resolution. Firefox already uses breakpad for the crash reporting system. When you build your own B2G image, you use the build.sh script, and it can also be used to generate breakpad symbols:

$ ./build.sh buildsymbols

The symbols will be generated in the $GECKO_OBJDIR/dist/crashreporter-symbols directory. The -s of the profile.sh also accepts the path to the breakpad symbols directory for local symbol resolution, so you can also profile using breakpad symbols:

$ ./profile.sh capture -s $GECKO_OBJDIR/dist/crashreporter-symbols/

And the profile_captured.sym file will be generated as usual.

If you distribute the breakpad symbols with your custom Firefox OS image anyone can profile it.

Conclusion

This post was intented to be a brief tutorial on how to get profiles for your app without building your own Firefox OS image. Currently, profile.sh is shipped with private builds only, and it will eventually ships with Nightly too.

If you have any doubts, ping me on #fxos-automation room.

Wander Lairson Costa

Modernizing stalld with eBPF

eBPF and the verifier

Tracepoints

Bug fixes

DL-server integration

Summary

Going deeper

Debugging the Linux Kernel through tracing

Tracepoints

Kernel probes

Summary

Going deeper

Improving the Kernel TTY throughput

A crash course on debugging kernel crashes using the crash utility

NUIPreview: a Visual Studio extension to build Tizen.NUI graphical interfaces

Building DALi for 32 bits Windows

Running the plugin

Highway to hell: C++ enums and bit fields

Building DALi library for Windows

Dependencies

Building Windows dependencies

Building the DALi libraries

dali-core

dali-adaptor

dali-toolkit

dali-demo

Running

Running packet.net images in qemu

A tale of my first Go patch

The problem

Attempt #1: don’t call setgroups if len(Groups) == 0

Attempt #2: don’t call setgroups if Groups == nil

Attempt #3: call setgroups only if Groups and getgroups mismatch

Accepted solution: define a new member in the Credential struct

Yet another "Java faster than C++" claim

The taskcluster-worker Mac OSX engine

The role of a Taskcluster worker

taskcluster-worker and the OSX engine

Running the worker

Overcoming browser same origin policy

Cross-origin resource sharing

cors-proxy

In tree tasks configuration

Running phone builds on Taskcluster

Submitting tasks

Mozharness

Hacking Taskcluster B2G builds

Mercurial for git lovers

Special note on Bookmarks

Conclusion

Profiling Firefox OS apps

Starting with profile support

Getting profile files

Symbol resolution through breakpad symbols

Conclusion

Attempt #1: don’t call `setgroups` if `len(Groups) == 0`

Attempt #2: don’t call `setgroups` if `Groups == nil`

Attempt #3: call `setgroups` only if `Groups` and `getgroups` mismatch

Accepted solution: define a new member in the `Credential` struct