When gcc 0.x came out - it was so refreshing. A free C compiler.
GCC evolved over the years, got slower and used more memory. I used
to use gcc on a 4MB RAM system (no typo), and wished I had 5MB RAM.
Today, memory is cheap, and a few GB to compile code is acceptable.
(The worst I have seen is 30+GB to compile a C++ piece of code - not
mine!)
One of the powerful features of gcc was that "gcc -g" and "gcc -O"
were not exclusive. And gdb came about as a free debugger, complimenting
gcc.
Over recent years, gdb has become closer to useless. It is a powerful
and complex and featureful debugger. But I am fed up single stepping
my code, and watching the line of execution bounce back and forth
because the compiler emits strange debug info where we move back
and forth over lines of code and declarations.
Today, in debugging fcterm - my attempt to place a breakpoint
on a line of code, puts the breakpoint *miles* away from the place I
am trying to intercept. This renders "gcc -g" close to useless, unless
I turn off all optimisations, and pray the compiler isnt inlining code.
Shame on gcc. Maybe I should switch to clang/llvm.
Post created by CRiSP v12.0.3a-b6801
Thursday, 19 February 2015
Sunday, 15 February 2015
address: 0000f00000000000
Strange. Continue to keep finding why dtrace is not passing my tests.
I have narrowed it down to a strange exception. If the user script
accesses an invalid address, we either get a page fault or a GPF.
DTrace handles this and stubs out the offending memory access. Heres
a script
This script will examine all syscalls and try and access the string
for arg0/1/2 - and for most syscalls, there isnt one. So we end up
dereferencing a bad pointer. But only some pointers cause me pain.
Most are handled properly. The address in the title is one such address.
I *think* what we have is the difference between a page fault and a GPF.
Despite a lot of hacking to the code - I cannot easily debug, since
once this exception happens the kernel doesnt recover. I have modified
the script above to only do syscall::chdir: which means I can manually
test via a shell, doing a "cd" command. On my 3-cpu VM, I lose one of the
CPUs and the machine behaves erratically. Now I need to figure out if
we are getting a GPF or some other exception.
I tried memory addresses: 0x00..00f, 0x00..0f0, 0x00..f00, ... in order
to find this. I suspect there is no page table mapping here or its
special in some other way. May need to dig into the kernel GDT or
page table to see what is causing this.
UPDATE: 20150215
After a bunch of digging I found that the GPF interrupt handler had
been commented out. There was a bit more to this than that, because
even when I re-enabled it, I was getting some other spurious issues.
All in all, various bits of hack code and debugging had got in the way
of a clear message.
I have been updating the sources to merge back in the fixes
for the 3.16 kernel, but have a regression on syscall tracing which
can cause spurious panics. I need to fix that before I do a next
release.
Post created by CRiSP v12.0.3a-b6801
I have narrowed it down to a strange exception. If the user script
accesses an invalid address, we either get a page fault or a GPF.
DTrace handles this and stubs out the offending memory access. Heres
a script
build/dtrace -n '
BEGIN {
cnt = 0;
tstart = timestamp;
}
syscall::: {
this->pid = pid;
this->ppid = ppid;
this->execname = execname;
this->arg0 = stringof(arg0);
this->arg1 = stringof(arg1);
this->arg2 = stringof(arg2);
cnt++;
}
tick-1s { printf("count so far: %d", cnt); }
tick-500s { exit(0); }
'
This script will examine all syscalls and try and access the string
for arg0/1/2 - and for most syscalls, there isnt one. So we end up
dereferencing a bad pointer. But only some pointers cause me pain.
Most are handled properly. The address in the title is one such address.
I *think* what we have is the difference between a page fault and a GPF.
Despite a lot of hacking to the code - I cannot easily debug, since
once this exception happens the kernel doesnt recover. I have modified
the script above to only do syscall::chdir: which means I can manually
test via a shell, doing a "cd" command. On my 3-cpu VM, I lose one of the
CPUs and the machine behaves erratically. Now I need to figure out if
we are getting a GPF or some other exception.
I tried memory addresses: 0x00..00f, 0x00..0f0, 0x00..f00, ... in order
to find this. I suspect there is no page table mapping here or its
special in some other way. May need to dig into the kernel GDT or
page table to see what is causing this.
UPDATE: 20150215
After a bunch of digging I found that the GPF interrupt handler had
been commented out. There was a bit more to this than that, because
even when I re-enabled it, I was getting some other spurious issues.
All in all, various bits of hack code and debugging had got in the way
of a clear message.
I have been updating the sources to merge back in the fixes
for the 3.16 kernel, but have a regression on syscall tracing which
can cause spurious panics. I need to fix that before I do a next
release.
Post created by CRiSP v12.0.3a-b6801
Monday, 9 February 2015
no dtrace updates
People have been questioning why there are no dtrace updates.
I hope to be in a position to properly respond shortly. Just before
Christmas, I started work on Debian Jessie (3.16 kernel) and hit a number
of issues. Although I made good progress fixing issues on x32 syscalls on
a x64 system, and systematically fixing other issues, I had to hack the
driver tremendously. These hacks are experiments to figure out why
I could so easily crash the kernel. The usual means of panicing the
kernel did not hold - normally a stray issue causes a kernel message
and I can debug around the issue to isolate the cause.
The issues I hit were all very low level - the cross-cpu calls, the
worker interrupt thread, and the current issue - relating to invalid
pointers when accessed via a D script. I have a "hard" test which wont
pass without crashing the kernel - crashing the kernel really hard,
requiring a VM reboot. This is nearly impossible to debug. The first
thing I had to do was increase the console mode terminal size - when
the panic occurs, the system is totally unresponsive and all I have
is the console output to look out, with no scrolling ability. Having
a bigger console helps - but it seems like the GPF or PageFault interrupt,
when occuring inside the kernel, does not work the same way as
it has on all prior Linux kernels. Looking closely at the
interrupt routines shows some changes in the way this works - enough
to potentially cause a paniccing interrupt to take out the whole
kernel; this makes life tough to debug.
If I am lucky, the area of concern is related to the interrupt
from kernel space. If I am unlucky, it is not this, but something else.
(Am hypothesing that the kernel stacks may be too small).
I have been saving up putting out any updates, despite some
pull requests from people, because I am not happy the driver is in
a consistent state to release. When I have finished this area
of debugging, I can cross-check the other/older kernels, and see if
I have broken anything.
It is very painful dealing with hard-crashing kernels - almost nothing
helps in terms of debugging, so am having to try various tricks to
isolate the instability. These instabilities in theory, exist on other
Linux releases - but I will only know when I have gotten to the bottom
of the issue.
Post created by CRiSP v12.0.3a-b6801
I hope to be in a position to properly respond shortly. Just before
Christmas, I started work on Debian Jessie (3.16 kernel) and hit a number
of issues. Although I made good progress fixing issues on x32 syscalls on
a x64 system, and systematically fixing other issues, I had to hack the
driver tremendously. These hacks are experiments to figure out why
I could so easily crash the kernel. The usual means of panicing the
kernel did not hold - normally a stray issue causes a kernel message
and I can debug around the issue to isolate the cause.
The issues I hit were all very low level - the cross-cpu calls, the
worker interrupt thread, and the current issue - relating to invalid
pointers when accessed via a D script. I have a "hard" test which wont
pass without crashing the kernel - crashing the kernel really hard,
requiring a VM reboot. This is nearly impossible to debug. The first
thing I had to do was increase the console mode terminal size - when
the panic occurs, the system is totally unresponsive and all I have
is the console output to look out, with no scrolling ability. Having
a bigger console helps - but it seems like the GPF or PageFault interrupt,
when occuring inside the kernel, does not work the same way as
it has on all prior Linux kernels. Looking closely at the
interrupt routines shows some changes in the way this works - enough
to potentially cause a paniccing interrupt to take out the whole
kernel; this makes life tough to debug.
If I am lucky, the area of concern is related to the interrupt
from kernel space. If I am unlucky, it is not this, but something else.
(Am hypothesing that the kernel stacks may be too small).
I have been saving up putting out any updates, despite some
pull requests from people, because I am not happy the driver is in
a consistent state to release. When I have finished this area
of debugging, I can cross-check the other/older kernels, and see if
I have broken anything.
It is very painful dealing with hard-crashing kernels - almost nothing
helps in terms of debugging, so am having to try various tricks to
isolate the instability. These instabilities in theory, exist on other
Linux releases - but I will only know when I have gotten to the bottom
of the issue.
Post created by CRiSP v12.0.3a-b6801
Monday, 1 December 2014
DTrace & Debian/Jessie
Someone reported a bug in dtrace whereby execve() wasnt tracing.
I created a VM and started testing, and can confirm this. Looks
like dtrace is getting confused by the new rewritten syscall assembler.
I have a working version for this in my testbed, but I found that
changes to the IPI code in the kernel are making any dtrace probes
extremely unreliable - looks like a 1:N chance of seeing output (where
N is the number of cpus you have).
I have some similar issues in Ubuntu 14.04 - hopefully similar issues.
Hope to have a new release shortly in a few days.
Post created by CRiSP v12.0.3a-b6801
I created a VM and started testing, and can confirm this. Looks
like dtrace is getting confused by the new rewritten syscall assembler.
I have a working version for this in my testbed, but I found that
changes to the IPI code in the kernel are making any dtrace probes
extremely unreliable - looks like a 1:N chance of seeing output (where
N is the number of cpus you have).
I have some similar issues in Ubuntu 14.04 - hopefully similar issues.
Hope to have a new release shortly in a few days.
Post created by CRiSP v12.0.3a-b6801
Saturday, 8 November 2014
CRiSP/crtags Optimisation
The tagging facility in CRiSP has always worked reasonably well - it
is based on a series of custom parsers for each programming language.
Over the years, the list of languages supported has grown and is
now a huge repository of nearly every common language and file
format out there. (See: "crtags -help" for details).
The initial implementation is getting on for nearly 20y old.
The goal originally was to provide the "ctags" facility of vi,
but better. Machines of the day were looking like 4-16MB of RAM rather
than 4-16GB of RAM which is common today, so effort was made to
optimise space usage. The crtags file format is a series of sections
of files and items which are tagged. It has been optimised - right
from the beginning, to avoid optimise space usage, and avoid bad
paging behavior. (Now, a distant artifact ! Systems rarely page or
are memory constrained). This attempt to optimise memory goes back to the
1MB machines that CRiSP was originally built on. These optimisations
are no longer necessary - but removing them would only offer a small
change in performance.
crtags is designed to work with reasonably sized projects and directories.
It takes a few seconds to scan the nearly 8000 files in the CRiSP source
tree, and I regularly use it on the Linux kernel. The 3.16.1 kernel
has 47426 files in it. Scanning that takes a little while. I have benchmarked
this over the years.
Recently I did some more work to look at the performance and optimisation
facilities in crtags. I collected a series of Linux kernels - so
I would have a good/large test case - about 500MB of source files,
67356 files in all. On my i7 laptop (crtags is single threaded), it was
taking about 2m20s to scan the files. On investigating the performance,
I could see that we had an O(n^2) algorithm on filename matching. This
is silly, given the complexity of the language parsers - that the mere
filenames were using a lot of the CPU.
I modified the code to put in a hash table for filename matching,
and this gave a huge win - down to about 23s for scanning the same files -
about a 7x improvement in speed.
In looking at crtags, most of the processing is a constant per file -
and each file is handled, one after the other. This opens up a huge
win by multithreading the code. Potentially an Nx speedup, on an N cpu system.
The code is difficult to convert to multithreading - it would require
a lot of edits and refactoring, to ensure each thread is
avoiding global state - a common reason why converting a non-threaded
application to multithreaded is so difficult.
Its depressing how little attention the C standards bodies and
compiler writers have, for converting non-threaded code to threaded.
Really, there are a set of transformations (refactoring) and one would
think that tools could help identify the major areas at issue
(use of global variables, and use of "static" variables). I
may create a refactoring macro in CRiSP to handle this.
On Unix, using a fork/join model of operation, one can create
the equivalent of a multithreaded code, by use of fork() and wait()
system calls. For example, divide all the files to be processed
up, into separate groups, processed by individual CPUs. Then the
issue of global state and locking disappears - at the expensive of
more work on the "join" or merge at the end of processing.
I have modified crtags to use this fork/join model (it is a command
line option, and not enabled by default), and reran my test. The above
test went from 23s to 4s by using "crtags -j 8" - using the 4 real and
4 hyperthread cpu's on my i7. About a 6x performance increase.
(The final code will be slower due to the lack of a merge).
So we went from 2m20s to 4s - a 35x speed up, with just a handful of lines
of code.
The depressing thing about Windows (and this is late 2014) is that
it still does not support fork(). It does support threads. So the above
code will have no effect on Windows, and real threading will have to
be implemented or an alternate way of achieving the same result.
I do fail to understand why Windows cant implement fork() - from
the user space point of view, there is almost nothing to implement.
From the kernel point of view, its not a huge amount of code.
Granted, Windows processes may carry more state and forking may
be more expensive if Windows could do it, but that would be such
a big benefit when writing portable code or porting code to Windows.
Oh well. (Cygwin supports fork(), and it is hugely expensive
in operation, since it relies on software to copy huge blocks
of memory around, rather than relying on the CPU's MMU to do
copy-on-write (COW) operation - the key to why fork() is so efficient
on Unixes).
Having said that, fork() on Linux is not brillliantly fast. Despite
processors being so much faster than years of old - fork() seems
to be getting slower, possibly related to the need for all
CPUs to synchronise MMU and other state, rather than fork() itself
getting more complicated.
To end users of CRiSP, they may see the initial performance optimisation,
but unless they are working on extremely huge projects, they may
hardly ever notice the change put in place. Also note, that
in CRiSP, one rarely tags an entire project - CRiSP does incremental
updates to the tag database, as you are editing/saving files.
Post created by CRiSP v12.0.3a-b6801
is based on a series of custom parsers for each programming language.
Over the years, the list of languages supported has grown and is
now a huge repository of nearly every common language and file
format out there. (See: "crtags -help" for details).
The initial implementation is getting on for nearly 20y old.
The goal originally was to provide the "ctags" facility of vi,
but better. Machines of the day were looking like 4-16MB of RAM rather
than 4-16GB of RAM which is common today, so effort was made to
optimise space usage. The crtags file format is a series of sections
of files and items which are tagged. It has been optimised - right
from the beginning, to avoid optimise space usage, and avoid bad
paging behavior. (Now, a distant artifact ! Systems rarely page or
are memory constrained). This attempt to optimise memory goes back to the
1MB machines that CRiSP was originally built on. These optimisations
are no longer necessary - but removing them would only offer a small
change in performance.
crtags is designed to work with reasonably sized projects and directories.
It takes a few seconds to scan the nearly 8000 files in the CRiSP source
tree, and I regularly use it on the Linux kernel. The 3.16.1 kernel
has 47426 files in it. Scanning that takes a little while. I have benchmarked
this over the years.
Recently I did some more work to look at the performance and optimisation
facilities in crtags. I collected a series of Linux kernels - so
I would have a good/large test case - about 500MB of source files,
67356 files in all. On my i7 laptop (crtags is single threaded), it was
taking about 2m20s to scan the files. On investigating the performance,
I could see that we had an O(n^2) algorithm on filename matching. This
is silly, given the complexity of the language parsers - that the mere
filenames were using a lot of the CPU.
I modified the code to put in a hash table for filename matching,
and this gave a huge win - down to about 23s for scanning the same files -
about a 7x improvement in speed.
In looking at crtags, most of the processing is a constant per file -
and each file is handled, one after the other. This opens up a huge
win by multithreading the code. Potentially an Nx speedup, on an N cpu system.
The code is difficult to convert to multithreading - it would require
a lot of edits and refactoring, to ensure each thread is
avoiding global state - a common reason why converting a non-threaded
application to multithreaded is so difficult.
Its depressing how little attention the C standards bodies and
compiler writers have, for converting non-threaded code to threaded.
Really, there are a set of transformations (refactoring) and one would
think that tools could help identify the major areas at issue
(use of global variables, and use of "static" variables). I
may create a refactoring macro in CRiSP to handle this.
On Unix, using a fork/join model of operation, one can create
the equivalent of a multithreaded code, by use of fork() and wait()
system calls. For example, divide all the files to be processed
up, into separate groups, processed by individual CPUs. Then the
issue of global state and locking disappears - at the expensive of
more work on the "join" or merge at the end of processing.
I have modified crtags to use this fork/join model (it is a command
line option, and not enabled by default), and reran my test. The above
test went from 23s to 4s by using "crtags -j 8" - using the 4 real and
4 hyperthread cpu's on my i7. About a 6x performance increase.
(The final code will be slower due to the lack of a merge).
So we went from 2m20s to 4s - a 35x speed up, with just a handful of lines
of code.
The depressing thing about Windows (and this is late 2014) is that
it still does not support fork(). It does support threads. So the above
code will have no effect on Windows, and real threading will have to
be implemented or an alternate way of achieving the same result.
I do fail to understand why Windows cant implement fork() - from
the user space point of view, there is almost nothing to implement.
From the kernel point of view, its not a huge amount of code.
Granted, Windows processes may carry more state and forking may
be more expensive if Windows could do it, but that would be such
a big benefit when writing portable code or porting code to Windows.
Oh well. (Cygwin supports fork(), and it is hugely expensive
in operation, since it relies on software to copy huge blocks
of memory around, rather than relying on the CPU's MMU to do
copy-on-write (COW) operation - the key to why fork() is so efficient
on Unixes).
Having said that, fork() on Linux is not brillliantly fast. Despite
processors being so much faster than years of old - fork() seems
to be getting slower, possibly related to the need for all
CPUs to synchronise MMU and other state, rather than fork() itself
getting more complicated.
To end users of CRiSP, they may see the initial performance optimisation,
but unless they are working on extremely huge projects, they may
hardly ever notice the change put in place. Also note, that
in CRiSP, one rarely tags an entire project - CRiSP does incremental
updates to the tag database, as you are editing/saving files.
Post created by CRiSP v12.0.3a-b6801
Saturday, 6 September 2014
Firefox, GMail and cookies (solved)
I reported a while back, a problem with Firefox and inability to
login to gmail. After a restart of firefox, you get a cookie error
page. I found a solution was to tediously delete all google cookies - then
I could login. However, logging into my other account would, sooner or later,
give rise to the same cookie mismatch error. It was annoying
and infuriating.
I had resorted to using my phone/tablet to handling email, but that seems
silly (except when on the road).
After repeated scouring the web for solutions or solved solutions,
nothing worked. But I bumped into a reference regarding potential
issue with plugins.
So I tried disabling all my firefox plugs, and it worked !
So I then narrowed down which is the offending culprit.
And the culprit is:
Flash Video Downloader - YouTube Full HD Download
I dont know why, but I assume these plugins act as private
proxies, to intercept web requests, and somehow, it is mishandling
cookies.
So I can relax that I hope this issue is finally over.
I hope this is useful to others out there when faced with a similar
issue.
Post created by CRiSP v11.0.35a-b6772
login to gmail. After a restart of firefox, you get a cookie error
page. I found a solution was to tediously delete all google cookies - then
I could login. However, logging into my other account would, sooner or later,
give rise to the same cookie mismatch error. It was annoying
and infuriating.
I had resorted to using my phone/tablet to handling email, but that seems
silly (except when on the road).
After repeated scouring the web for solutions or solved solutions,
nothing worked. But I bumped into a reference regarding potential
issue with plugins.
So I tried disabling all my firefox plugs, and it worked !
So I then narrowed down which is the offending culprit.
And the culprit is:
Flash Video Downloader - YouTube Full HD Download
I dont know why, but I assume these plugins act as private
proxies, to intercept web requests, and somehow, it is mishandling
cookies.
So I can relax that I hope this issue is finally over.
I hope this is useful to others out there when faced with a similar
issue.
Post created by CRiSP v11.0.35a-b6772
Tuesday, 26 August 2014
Galaxy Tab S 8.4 / LTE
I recently bought one of these devices. I didnt really need it,
but I wanted a larger version of my phone (Galaxy Note 2), for
video watching. I dislike my mini ipad - it is a great device to hold,
but the whole iOS experience leaves me very cold. That and the puny amount
of RAM. Switching between 3 apps will force restart the apps since
iOS cannot juggle the demands of only 3 apps in the 512MB of memory.
My phone has 2GB and can easily keep 5-6 apps open. So app switching
is faster, because of the lack of a need to restart.
The Galaxy Tab S has a screen of 2560x1600. Although brilliant from
a bragging rights, its difficult to appreciate what this means. What
it means to me is that when I split screen apps, I can read the smaller
fonts of info from web browser, IMDB, etc.
There is definitely some lag in the device. I attribute this to code which
will synchronously try and connect somewhere. I rooted my device, put
in an adblocker, but sometimes the delays are much more than just bad
OS coding. Theres too many services all fighting to wake up and do
things, and even with a quad core cpu, it hangs. I run a tool to monitor
the CPU speed and when it is being sluggish, the CPUs are at their lowest
clock speed (250MHz) which suggests externally waiting for something
and not a CPU issue. I need to run more stuff to see whats going on, e.g.
a network monitoring.
My major complaint with the Tab-S is its so big. I can easily hold
the Galaxy phones in my hand comfortably, in such a way that my fingers
cannot touch the screen. The same is not true of an 8" tablet. You either
gingerly hold it from the sides, e.g. the end where the Samsung label
is, because theres more room for your fingers, or the other end
where the home/back/menu button is located. That end of the tab is
bad news as the system sees you frantically pressing any/all of those buttons.
If you attempt to grasph the device with your fingers mid-screen, chances
you are going to do something you regret (such as click on a link, or
fast-forward a video). Very strangely, the one advantage of iOS is
that it is such an unfunctional OS, that this does not happen
with the ipads. If you are watching a video and use your fingers
to grasp the middle of the screen, then nothing happens, because apple
doesnt attribute any activity to touching the middle of the view area.
So, the one advantage that Android has (and MX-Player - brilliant video
player) is actually a problem on a large screen device, because there
is nowhere safe to hold the device.
I really dont know what is happening with the edge-less phones
currently being touted. I dont know how you can hold one without
having a very erratic user experience from the edges
of your palm or fingers, just trying to hold the device.
Only time will tell.
Post created by CRiSP v11.0.34a-b6771
but I wanted a larger version of my phone (Galaxy Note 2), for
video watching. I dislike my mini ipad - it is a great device to hold,
but the whole iOS experience leaves me very cold. That and the puny amount
of RAM. Switching between 3 apps will force restart the apps since
iOS cannot juggle the demands of only 3 apps in the 512MB of memory.
My phone has 2GB and can easily keep 5-6 apps open. So app switching
is faster, because of the lack of a need to restart.
The Galaxy Tab S has a screen of 2560x1600. Although brilliant from
a bragging rights, its difficult to appreciate what this means. What
it means to me is that when I split screen apps, I can read the smaller
fonts of info from web browser, IMDB, etc.
There is definitely some lag in the device. I attribute this to code which
will synchronously try and connect somewhere. I rooted my device, put
in an adblocker, but sometimes the delays are much more than just bad
OS coding. Theres too many services all fighting to wake up and do
things, and even with a quad core cpu, it hangs. I run a tool to monitor
the CPU speed and when it is being sluggish, the CPUs are at their lowest
clock speed (250MHz) which suggests externally waiting for something
and not a CPU issue. I need to run more stuff to see whats going on, e.g.
a network monitoring.
My major complaint with the Tab-S is its so big. I can easily hold
the Galaxy phones in my hand comfortably, in such a way that my fingers
cannot touch the screen. The same is not true of an 8" tablet. You either
gingerly hold it from the sides, e.g. the end where the Samsung label
is, because theres more room for your fingers, or the other end
where the home/back/menu button is located. That end of the tab is
bad news as the system sees you frantically pressing any/all of those buttons.
If you attempt to grasph the device with your fingers mid-screen, chances
you are going to do something you regret (such as click on a link, or
fast-forward a video). Very strangely, the one advantage of iOS is
that it is such an unfunctional OS, that this does not happen
with the ipads. If you are watching a video and use your fingers
to grasp the middle of the screen, then nothing happens, because apple
doesnt attribute any activity to touching the middle of the view area.
So, the one advantage that Android has (and MX-Player - brilliant video
player) is actually a problem on a large screen device, because there
is nowhere safe to hold the device.
I really dont know what is happening with the edge-less phones
currently being touted. I dont know how you can hold one without
having a very erratic user experience from the edges
of your palm or fingers, just trying to hold the device.
Only time will tell.
Post created by CRiSP v11.0.34a-b6771
Subscribe to:
Posts (Atom)