Retracting the gcc security hole

In my last blog post, I wrote about a potential security hole
in the way GCC allocates stack frames. After much research on how to get
close to this error, I will happily retract the claim, and I can now
sleep more easily at night. (Although I need a new impossible-problem
to solve!)

http://eli.thegreenplace.net/2011/09/06/stack-frame-layout-on-x86-64/

And I thank my colleague, Luke, for finding this (he actually pointed
to the wikipedia page). The low down is this. If you have a leaf
function - one that calls no other function, then GCC will emit
an assembly optimisation where instead of decreasing the RSP register
by the required space, it will reduce it by the amount needed, minus
128 bytes. This can allow short positive and negative addressing
offsets from the RSP register.

int func(void)
{	char	buf[4096] = {0};
	int	i, s;

	for (i = 0; i < 4096; i++)
		s += buf[i];
	return s;
}


A function like the above will reveal the strangeness, if you compile
to assembler. 

My assertion was that whilst RSP is not at the bottom of the stack,
that something like a signal or some other callback, could corrupt
the stack and lead to indeterminate results. I tried various things
to trigger this, and havent succeeded. The link above, demonstrates
that the 128 "red zone" is what saves the day.

So, until I can uncover someone not honoring the ABI protocol, we can
all sleep more soundly at night.

Now I can go back to the line profiling/tracing tool...

Post created by CRiSP v11.0.32a-b6738