momo zone

调核人的blog

linux SMP 关于cpu的几个变量和宏

最近有点迷惑kernel关于cpu枚举的几个宏和全局变量,看了一下文章感觉清醒一点

Overview

This page is a completely impromptu coverage of managing multiple CPUs, cpumasks, iterating through all possible CPUs and so on. I didn’t think it was going to be that big a deal, but there’s a lot more going on here than you might think. And plenty more to come, depending on how ambitious I get.

If you want to keep up with notifications on new Linux kernel tutorials, you’re welcome to follow me on Twitter athttps://twitter.com/rpjday.

Files

A tentative list of kernel source files related to this topic (list will probably change over time as I run across additional entries to add or delete some of these that aren’t really relevant):

  • include/linux/
    • cpu.h
    • cpumask.h
    • smp.h
    • smpboot.h
  • arch/x86/
    • configs/x86_64_defconfig
    • kernel/smpboot.c
    • include/asm/
      • smp.h
      • cpumask.h
  • kernel/
    • cpu.c
    • smp.c
    • smpboot.h
    • smpboot.c
  • lib/cpumask.c

What am I missing?

Your current setup

All this will be demonstrated on my quad-core ASUS laptop so, technically, I will have eight processors.

First, let’s verify that:

$ grep ^processor /proc/cpuinfo
processor	: 0
processor	: 1
processor	: 2
processor	: 3
processor	: 4
processor	: 5
processor	: 6
processor	: 7
$

Also, check under sysfs:

$ ls -1 /sys/devices/system/cpu
cpu0
cpu1
cpu2
cpu3
cpu4
cpu5
cpu6
cpu7
cpufreq
cpuidle
... snip ...

So far, so good. Onward.

NR_CPUS versus nr_cpu_ids

The variable NR_CPUS scattered throughout the kernel source is not your actual number of CPUs, it’s the maximumnumber set by the config process found in arch/x86/Kconfig:

config NR_CPUS
        int "Maximum number of CPUs" if SMP && !MAXSMP
        range 2 8 if SMP && X86_32 && !X86_BIGSMP
        range 2 512 if SMP && !MAXSMP
        default "1" if !SMP
        default "4096" if MAXSMP
        default "32" if SMP && (X86_NUMAQ || X86_SUMMIT || X86_BIGSMP || X86_ES7000)
        default "8" if SMP
        ---help---
          This allows you to specify the maximum number of CPUs which this
          kernel will support.  The maximum supported value is 512 and the
          minimum value which makes sense is 2.

          This is purely to save memory - each supported CPU adds
          approximately eight kilobytes to the kernel image.

In my case, I can verify that my .config file contains:

CONFIG_NR_CPUS=256

That’s different from the global variable nr_cpu_ids initialized in kernel/smp.c:

/* Setup number of possible processor ids */
int nr_cpu_ids __read_mostly = NR_CPUS;
EXPORT_SYMBOL(nr_cpu_ids);

/* An arch may set nr_cpu_ids earlier if needed, so this would be redundant */
void __init setup_nr_cpu_ids(void)
{
        nr_cpu_ids = find_last_bit(cpumask_bits(cpu_possible_mask),NR_CPUS) + 1;
}

You can see the initialization scattered throughout the dmesg file at boot time (the following lines are not sequential in that file):

[    0.000000] smpboot: Allowing 16 CPUs, 8 hotplug CPUs
[    0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:16 nr_node_ids:1
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
[    0.000000]  RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=16.
[    0.344852] smpboot: Booting Node   0, Processors  #1 #2 #3 #4 #5 #6 #7
[    0.437322] Brought up 8 CPUs
[    0.437326] smpboot: Total of 8 processors activated (31925.90 BogoMIPS)

The default value for NR_CPUS from the x86_64 defconfig file is:

CONFIG_NR_CPUS=64

I just used an overly generous, explicit value of 256. No reason.

Command-line parameters controlling SMP

From kernel/smp.c, there appear to be three kernel command-line parameters you can play with related to this topic:

  • nosmp
  • nr_cpus=
  • maxcpus=
/* Setup configured maximum number of CPUs to activate */
unsigned int setup_max_cpus = NR_CPUS;
EXPORT_SYMBOL(setup_max_cpus);

/*
 * Setup routine for controlling SMP activation
 *
 * Command-line option of "nosmp" or "maxcpus=0" will disable SMP
 * activation entirely (the MPS table probe still happens, though).
 *
 * Command-line option of "maxcpus=<NUM>", where <NUM> is an integer
 * greater than 0, limits the maximum number of CPUs activated in
 * SMP mode to <NUM>.
 */

void __weak arch_disable_smp_support(void) { }

static int __init nosmp(char *str)
{
        setup_max_cpus = 0;
        arch_disable_smp_support();

        return 0;
}

early_param("nosmp", nosmp);

/* this is hard limit */
static int __init nrcpus(char *str)
{
        int nr_cpus;

        get_option(&str, &nr_cpus);
        if (nr_cpus > 0 && nr_cpus < nr_cpu_ids)
                nr_cpu_ids = nr_cpus;

        return 0;
}

early_param("nr_cpus", nrcpus);

static int __init maxcpus(char *str)
{
        get_option(&str, &setup_max_cpus);
        if (setup_max_cpus == 0)
                arch_disable_smp_support();

        return 0;
}

early_param("maxcpus", maxcpus);

/* Setup number of possible processor ids */
int nr_cpu_ids __read_mostly = NR_CPUS;
EXPORT_SYMBOL(nr_cpu_ids);

/* An arch may set nr_cpu_ids earlier if needed, so this would be redundant */
void __init setup_nr_cpu_ids(void)
{
        nr_cpu_ids = find_last_bit(cpumask_bits(cpu_possible_mask),NR_CPUS) + 1;
}

Bringing up the CPUs

Also from kernel/smp.c:

/* Called by boot processor to activate the rest. */
void __init smp_init(void)
{
        unsigned int cpu;

        idle_threads_init();

        /* FIXME: This should be done in userspace --RR */
        for_each_present_cpu(cpu) {
                if (num_online_cpus() >= setup_max_cpus)
                        break;
                if (!cpu_online(cpu))
                        cpu_up(cpu);
        }

        /* Any cleanup work */
        printk(KERN_INFO "Brought up %ld CPUs\n", (long)num_online_cpus());
        smp_cpus_done(setup_max_cpus);
}

which explains the line you saw in dmesg (in my case):

[    0.437322] Brought up 8 CPUs

The cpumask for identifying CPUs

Pretty much everything you need is declared/defined in the header file include/linux/cpumask.h:

/*
 * The following particular system cpumasks and operations manage
 * possible, present, active and online cpus.
 *
 *     cpu_possible_mask- has bit 'cpu' set iff cpu is populatable
 *     cpu_present_mask - has bit 'cpu' set iff cpu is populated
 *     cpu_online_mask  - has bit 'cpu' set iff cpu available to scheduler
 *     cpu_active_mask  - has bit 'cpu' set iff cpu available to migration
... snip ...
#if NR_CPUS > 1
#define num_online_cpus()       cpumask_weight(cpu_online_mask)
#define num_possible_cpus()     cpumask_weight(cpu_possible_mask)
#define num_present_cpus()      cpumask_weight(cpu_present_mask)
#define num_active_cpus()       cpumask_weight(cpu_active_mask)
#define cpu_online(cpu)         cpumask_test_cpu((cpu), cpu_online_mask)
#define cpu_possible(cpu)       cpumask_test_cpu((cpu), cpu_possible_mask)
#define cpu_present(cpu)        cpumask_test_cpu((cpu), cpu_present_mask)
#define cpu_active(cpu)         cpumask_test_cpu((cpu), cpu_active_mask)
#else

and you can use these macros to iterate through your CPUs:

#define for_each_cpu(cpu, mask)                         \
        for ((cpu) = -1;                                \
                (cpu) = cpumask_next((cpu), (mask)),    \
                (cpu) < nr_cpu_ids;)

#define for_each_possible_cpu(cpu) for_each_cpu((cpu), cpu_possible_mask)
#define for_each_online_cpu(cpu)   for_each_cpu((cpu), cpu_online_mask)
#define for_each_present_cpu(cpu)  for_each_cpu((cpu), cpu_present_mask)

and so on, and so on. There is a ridiculous amount of detail you can get into here, but this is a start until I decide how to rewrite it. Feel free to scan the kernel source tree to see the numerous calls to for_each_cpu()for_each_online_cpu(),for_each_possible_cpu(), etc.

关于cpumask的一些解释:

/*
* The following particular system cpumasks and operations manage
* possible, present, active and online cpus.
*
* cpu_possible_mask- has bit ‘cpu’ set iff cpu is populatable
* cpu_present_mask – has bit ‘cpu’ set iff cpu is populated
* cpu_online_mask – has bit ‘cpu’ set iff cpu available to scheduler
* cpu_active_mask – has bit ‘cpu’ set iff cpu available to migration
*
* If !CONFIG_HOTPLUG_CPU, present == possible, and active == online.
*
* The cpu_possible_mask is fixed at boot time, as the set of CPU id’s
* that it is possible might ever be plugged in at anytime during the
* life of that system boot. The cpu_present_mask is dynamic(*),
* representing which CPUs are currently plugged in. And
* cpu_online_mask is the dynamic subset of cpu_present_mask,
* indicating those CPUs available for scheduling.
*
* If HOTPLUG is enabled, then cpu_possible_mask is forced to have
* all NR_CPUS bits set, otherwise it is just the set of CPUs that
* ACPI reports present at boot.
*
* If HOTPLUG is enabled, then cpu_present_mask varies dynamically,
* depending on what ACPI reports as currently plugged in, otherwise
* cpu_present_mask is just a copy of cpu_possible_mask.
*
* (*) Well, cpu_present_mask is dynamic in the hotplug case. If not
* hotplug, it’s a copy of cpu_possible_mask, hence fixed at boot.
*
* Subtleties:
* 1) UP arch’s (NR_CPUS == 1, CONFIG_SMP not defined) hardcode
* assumption that their single CPU is online. The UP
* cpu_{online,possible,present}_masks are placebos. Changing them
* will have no useful affect on the following num_*_cpus()
* and cpu_*() macros in the UP case. This ugliness is a UP
* optimization – don’t waste any instructions or memory references
* asking if you’re online or how many CPUs there are if there is
* only one CPU.
*/

Advertisements

发表评论

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / 更改 )

Twitter picture

You are commenting using your Twitter account. Log Out / 更改 )

Facebook photo

You are commenting using your Facebook account. Log Out / 更改 )

Google+ photo

You are commenting using your Google+ account. Log Out / 更改 )

Connecting to %s

%d 博主赞过: