Hung Task Sysctl: Enhanced System Info Dumping

Let's dive into a cool update for the Linux kernel! This article breaks down patch 2/3, focusing on how it enhances the hung_task functionality. We'll explore how this patch makes debugging easier when a task gets stuck. Get ready for an easy-to-understand explanation of the technical details!

1. Patch Overview: Making Debugging Easier

What problem does this patch solve, guys?

When a process in the kernel gets stuck in an uninterruptible state (a “hung task”), developers need as much system context as possible to figure out what's going on. Previously, the hung task detector could only print some fixed information, like lock status or call stacks from all CPUs. These features were controlled by separate switches, which wasn't very flexible. If you wanted to automatically get memory usage or timer lists when a task got stuck, there wasn't a straightforward way to do it. This made diagnosing these tricky issues a real pain.

The core changes introduced in this patch provide a flexible and extensible way to dump system info when a hung task is detected. The introduction of a new sysctl configuration item called hung_task_sys_info is the centerpiece. This allows users to specify exactly which types of system information they want to be automatically dumped when a hung task occurs, using a simple string like "tasks,mem,locks". This patch streamlines the debugging process.

Technically, the patch unifies the original, scattered boolean flags (like hung_task_show_lock) into a single "bitmask". When a hung task is detected, the kernel calls a generic sys_info() function, passing in this mask. This function then handles printing all the requested information. This makes the debugging information output highly configurable and extensible, a total game-changer for kernel devs.

2. Diving Deep: Technical Background and Concepts

sysctl: The Kernel's Configuration Interface

sysctl is a cool mechanism provided by the Linux kernel that lets system admins dynamically read and modify kernel parameters while the system is running. These parameters show up as files in a virtual file system, usually under /proc/sys/. You or programs can read and write to these files to check or tweak how the kernel behaves. This patch adds a new configurable parameter by creating a hung_task_sys_info file in the /proc/sys/kernel/ directory. It’s like giving you a remote control for your kernel's debugging features.

hung task detector: The Watchdog for Unresponsive Processes

The hung task detector is a background monitoring feature in the kernel that finds processes that have been stuck in an uninterruptible sleep state (TASK_UNINTERRUPTIBLE, or D state) for a long time. A task stuck in this state usually means there's a bug in the kernel, like a driver waiting for a hardware response that's never coming. When the detector finds such a task, it prints warning messages, including the task's call stack, to help developers pinpoint the problem. Think of it as a vigilant watchdog, barking when something goes terribly wrong.

bitmask: Packing Multiple Options into One Variable

A bitmask is a programming trick that uses each binary bit in an integer variable to represent a separate boolean state (on/off). By using bitwise operations (like OR, AND), you can efficiently store and manage multiple options in one variable. In this patch, the hung_task_si_mask variable is a bitmask where each bit corresponds to a type of system information (e.g., SYS_INFO_LOCKS represents lock information, SYS_INFO_ALL_BT represents all CPU backtraces). This approach saves space and makes it easier to expand compared to defining a bunch of boolean variables. It's like having a super-efficient way to manage multiple settings at once.

3. Code Review: A Deep Dive into the Changes

Review Results and Comments

File: kernel/hung_task.c

--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -60,12 +61,23 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 static int __read_mostly sysctl_hung_task_warnings = 10;
 
 static int __read_mostly did_panic;
-static bool hung_task_show_lock;
 static bool hung_task_call_panic;
-static bool hung_task_show_all_bt;
 
 static struct task_struct *watchdog_task;
 
+/*
+ * A bitmask to control what kinds of system info to be printed when
+ * a hung task is detected, it could be task, memory, lock etc. Refer
+ * include/linux/sys_info.h for detailed bit definition.
+ */
+static unsigned long hung_task_si_mask;
+
+/*
+ * There are several sysctl knobs, and this serves as the runtime
+ * effective sys_info knob
+ */
+static unsigned long cur_si_mask;
+
 #ifdef CONFIG_SMP
 /*
  * Should we dump all CPUs backtraces in a hung task event?
@@ -260,10 +273,10 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout,
  " disables this message.\n");
  sched_show_task(t);
  debug_show_blocker(t, timeout);
- hung_task_show_lock = true;
+ cur_si_mask |= SYS_INFO_LOCKS;
 
  if (sysctl_hung_task_all_cpu_backtrace)
- hung_task_show_all_bt = true;
+ cur_si_mask |= SYS_INFO_ALL_BT;
  if (!sysctl_hung_task_warnings)
  pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
 }
@@ -313,7 +326,6 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
  if (test_taint(TAINT_DIE) || did_panic)
  return;
 
- hung_task_show_lock = false;
  rcu_read_lock();
  for_each_process_thread(g, t) {
 
@@ -329,12 +341,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
  }
  unlock:
  rcu_read_unlock();
- if (hung_task_show_lock)
- debug_show_all_locks();
+ if (unlikely(cur_si_mask)) {
+ sys_info(cur_si_mask);
+ cur_si_mask = 0;
  }
 
  if (hung_task_call_panic)

Code Interpretation:

The core of this diff is refactoring the information dumping mechanism in the hung task detector. The change is all about making the process of gathering and reporting debugging info more flexible and efficient.

Before the change: The code used two separate boolean variables, hung_task_show_lock and hung_task_show_all_bt. After detecting a hung task, these flags were set to true. After scanning all processes, the code checked these flags with two if statements and called the corresponding dump functions, debug_show_all_locks() and trigger_all_cpu_backtrace(). It was like having separate switches for each debugging feature.

After the change: The code removes these boolean variables and introduces two unsigned long bitmasks: hung_task_si_mask (for storing the global sysctl configuration) and cur_si_mask (for the temporary state of the current detection cycle). Instead of setting boolean values to true, the corresponding flag bit (like SYS_INFO_LOCKS) is set in cur_si_mask using the bitwise OR operator |=. After the scan, a single if (unlikely(cur_si_mask)) check replaces the multiple if statements. If cur_si_mask is not zero (meaning there's a request to dump information), the new generic function sys_info(cur_si_mask) is called, passing all requests to it at once. Finally, cur_si_mask is cleared, ready for the next detection. This change consolidates the scattered control logic into a unified, data-driven framework, greatly improving code readability and extensibility.

Checklist Review & Assessment:

1. Logic & Functional Correctness:

   The code logic is correct. It successfully replaces the old boolean flag system with a bitmask mechanism. The check_hung_task function retains the checks for older sysctl configurations like sysctl_hung_task_all_cpu_backtrace, but their action changes from setting boolean flags to setting the corresponding bit in cur_si_mask (cur_si_mask |= SYS_INFO_ALL_BT). This ensures backward compatibility. cur_si_mask is cleared after each detection cycle, preventing state pollution between cycles.

2. Coding Style & Readability:

   The code follows standard Linux kernel coding style. The newly introduced global variables, hung_task_si_mask and cur_si_mask, have clear comments explaining their purpose. Replacing multiple if checks at the end of the check_hung_uninterruptible_tasks function with a single if (unlikely(cur_si_mask)) structure and a sys_info() call significantly simplifies the control flow, making the logic clearer and easier to understand.

3. Potential Risk Assessment:

   The risk is low. This patch primarily modifies how debugging information is reported and doesn't touch the core hung task detection algorithm. The introduced sys_info framework might cause brief performance hiccups when handling hung task events if the user requests time-consuming information dumps (like complete memory information). However, this is user-initiated behavior and an expected trade-off for debugging. The use of unlikely() is appropriate because cur_si_mask is mostly zero in normal operation.

4. Architecture & Maintainability:

   This is an excellent architectural improvement. It transforms the hung task detector from a fixed-function module into an extensible information reporting platform. To add a new debugging information output in the future, developers only need to add the corresponding implementation in the sys_info framework, with almost no need to modify the hung_task.c code. This decoupling greatly improves code maintainability, making future feature extensions simple and clear.

5. Nit-picking:

   The comment for the cur_si_mask variable, /* There are several sysctl knobs, and this serves as the runtime effective sys_info knob */, is correct but could be more precise. It could be improved to: /* The effective sys_info mask for the current detection cycle. It aggregates the base 'hung_task_si_mask' and any flags triggered by other conditions within this cycle. */, better explaining its role as a temporary aggregation variable.

4. Community Review & Discussion

(No replies or public discussions were found in the email data for this patch.)