Can't install 64b version on Dual Xeon setup

[bmx34]

Indeed, their is a 3.5. I did not noticed it because this version does not have a changelog file here : https://mirrors.edge.kernel.org/pub/linux/kernel/v3.x. I am not sure why.
But the code is definitly available, and I built it. If fails.

So the problem comes from a changelist between 3.4.113 and 3.5

[p.H]

I don't know why, but changelogs for major x.y kernel versions (x.y.z are only stable bugfix releases) are not available any more at kernel.org. This is one of the reasons why I maintain my own clone of the kernel git repository and build the missing changelogs myself.
If you are interested, I uploaded ChangeLog-3.5 here : http://www.plouf.fr.eu.org/bazar/ChangeLog-3.5
It contains all commits between 3.4 (not 3.4.113) and 3.5.

[bmx34]

Hi

Here are some news about this issue :
I have found the faulty commit : 8e029fcdd8702719c9179317cae9ef84ebe7027e, on branch 'x86-trampoline-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

The problem is that with this change, the NX flag is being activated. In my case, it appears that my 2 CPUs are slightly different :

Code: Select all

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	: Intel(R) Xeon(TM) CPU 3.40GHz
stepping	: 1
microcode	: 0x5
cpu MHz		: 2800.000
cache size	: 1024 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl est cid cx16 xtpr
bogomips	: 6800.52
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 3
model name	: Intel(R) Xeon(TM) CPU 3.40GHz
stepping	: 4
microcode	: 0xe
cpu MHz		: 2800.000
cache size	: 1024 KB
physical id	: 3
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl est tm2 cid xtpr
bogomips	: 6800.71
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

and while the first one supports the NX flag, the second does not... Seems that I got those CPUs just when Intel started to support the NX flag, and unfortunately, I have 2 different revisions.

I tried to rebuild a kernel without setting this flag, and it works fine. However, it is probably not great for security, as NX is all about malicious softwares...
Another workaround I found is to start with nosmp. In that case, I only use the first cpu, which supports NX, but I loose all the benefit of my dual cpu. Maybe physically switching my 2 CPU would also help, as the CPU 0 would then be the one not supporting NX... I did not had a look where the CPU flags are tested in the code.

I filled a bug report here : https://bugzilla.kernel.org/show_bug.cgi?id=207919

[CwF]

Intersting, good work. I've intentionally mixed some mult-sockets and as you found it is the lowest common denominator match if it works. Some more modern stuff should run asymmetrically with respect to clocks and multipliers, but features need to match.

The correct cpu should be cheap!

[p.H]

I tried to rebuild a kernel without setting this flag, and it works fine.

You should be able to disable the NX bit without compiling a kernel, either in the BIOS settings, either by adding "noexec=off" to the kernel command line.

However, it is probably not great for security, as NX is all about malicious softwares...

Not really about malicious software. Rather about software vulnerable to buffer overflow attacks.

Another workaround I found is to start with nosmp. In that case, I only use the first cpu, which supports NX, but I loose all the benefit of my dual cpu.

You cannot worry about disabling the NX bit and want to use a processor which does not support it at the same time. You must choose between security and performance.

Note that nosmp will also disable multicore and hyperthreading on the active CPU. Probably not the best choice.

[Head_on_a_Stick]

Note that nosmp will also disable multicore and hyperthreading on the active CPU. Probably not the best choice.

SMP is a potential source of *many* side-channel vulnerabilities and disabling it can actually increase performance for non-hyperthreaded applications.

[CwF]

Note that nosmp will also disable multicore and hyperthreading on the active CPU. Probably not the best choice.

SMP is a potential source of *many* side-channel vulnerabilities and disabling it can actually increase performance for non-hyperthreaded applications.

Two different things with two different settings. Disabling hyperthreading can improve performance and does not disable multcore. nosmp does. In either case I use the bios, which can also limit the number of cores.

[Head_on_a_Stick]

Two different things with two different settings.

D'oh! Thanks for the correction. I was confusing SMP with SMT (again).

[CwF]

I use the bios

Come to think of it you can likely disable NX in the bios?! Then the cpu's would kinda match...

[bmx34]

Unfortunatly, I do not have any option in the BIOS to deactivate NX, but I found in some Intel documents that "Mixing processors of different steppings but the same model (as per CPUID instruction) is supported". And I am not in this situation, as my CPUs are model 4/stepping 1 and model 3 stepping 4.

Still, I tried to physically swap my CPUs, so that CPU 0, that seems to be used to detect the capabilities of the system is the one without NX. The nice thing is that it seem to work fine ! In this configuration, I managed to start MX Linux, *once*...
The sad part is that during this swap, I must have damaged something on that old guy, as I now have huge unstatility and randomness : sometimes I even don't get the BIOS beep, sometimes it reboots by itself, crash, and sometimes I manage to boot... I checked memory, graphic card, swap again the CPUs, use a single CPU, etc... but for now, I did not managed to find were the problem is. I'll give it a few more hours of work until I declare it as RIP, but I am quite pessimist to manage to find a solution.

One thing for sure : for now, I have more than a Linux issue, and it seem that swapping the CPUs was a key workaround.