Monday, 1 December 2014

DTrace & Debian/Jessie

Someone reported a bug in dtrace whereby execve() wasnt tracing.
I created a VM and started testing, and can confirm this. Looks
like dtrace is getting confused by the new rewritten syscall assembler.
I have a working version for this in my testbed, but I found that
changes to the IPI code in the kernel are making any dtrace probes
extremely unreliable - looks like a 1:N chance of seeing output (where
N is the number of cpus you have).

I have some similar issues in Ubuntu 14.04 - hopefully similar issues.

Hope to have a new release shortly in a few days.

Post created by CRiSP v12.0.3a-b6801

Saturday, 8 November 2014

CRiSP/crtags Optimisation

The tagging facility in CRiSP has always worked reasonably well - it
is based on a series of custom parsers for each programming language.
Over the years, the list of languages supported has grown and is
now a huge repository of nearly every common language and file
format out there. (See: "crtags -help" for details).

The initial implementation is getting on for nearly 20y old.
The goal originally was to provide the "ctags" facility of vi,
but better. Machines of the day were looking like 4-16MB of RAM rather
than 4-16GB of RAM which is common today, so effort was made to
optimise space usage. The crtags file format is a series of sections
of files and items which are tagged. It has been optimised - right
from the beginning, to avoid optimise space usage, and avoid bad
paging behavior. (Now, a distant artifact ! Systems rarely page or
are memory constrained). This attempt to optimise memory goes back to the
1MB machines that CRiSP was originally built on. These optimisations
are no longer necessary - but removing them would only offer a small
change in performance.

crtags is designed to work with reasonably sized projects and directories.
It takes a few seconds to scan the nearly 8000 files in the CRiSP source
tree, and I regularly use it on the Linux kernel. The 3.16.1 kernel
has 47426 files in it. Scanning that takes a little while. I have benchmarked
this over the years.

Recently I did some more work to look at the performance and optimisation
facilities in crtags. I collected a series of Linux kernels - so
I would have a good/large test case - about 500MB of source files,
67356 files in all. On my i7 laptop (crtags is single threaded), it was
taking about 2m20s to scan the files. On investigating the performance,
I could see that we had an O(n^2) algorithm on filename matching. This
is silly, given the complexity of the language parsers - that the mere
filenames were using a lot of the CPU.

I modified the code to put in a hash table for filename matching,
and this gave a huge win - down to about 23s for scanning the same files -
about a 7x improvement in speed.

In looking at crtags, most of the processing is a constant per file -
and each file is handled, one after the other. This opens up a huge
win by multithreading the code. Potentially an Nx speedup, on an N cpu system.
The code is difficult to convert to multithreading - it would require
a lot of edits and refactoring, to ensure each thread is
avoiding global state - a common reason why converting a non-threaded
application to multithreaded is so difficult.

Its depressing how little attention the C standards bodies and
compiler writers have, for converting non-threaded code to threaded.
Really, there are a set of transformations (refactoring) and one would
think that tools could help identify the major areas at issue
(use of global variables, and use of "static" variables). I
may create a refactoring macro in CRiSP to handle this.

On Unix, using a fork/join model of operation, one can create
the equivalent of a multithreaded code, by use of fork() and wait()
system calls. For example, divide all the files to be processed
up, into separate groups, processed by individual CPUs. Then the
issue of global state and locking disappears - at the expensive of
more work on the "join" or merge at the end of processing.

I have modified crtags to use this fork/join model (it is a command
line option, and not enabled by default), and reran my test. The above
test went from 23s to 4s by using "crtags -j 8" - using the 4 real and
4 hyperthread cpu's on my i7. About a 6x performance increase.
(The final code will be slower due to the lack of a merge).

So we went from 2m20s to 4s - a 35x speed up, with just a handful of lines
of code.

The depressing thing about Windows (and this is late 2014) is that
it still does not support fork(). It does support threads. So the above
code will have no effect on Windows, and real threading will have to
be implemented or an alternate way of achieving the same result.

I do fail to understand why Windows cant implement fork() - from
the user space point of view, there is almost nothing to implement.
From the kernel point of view, its not a huge amount of code.
Granted, Windows processes may carry more state and forking may
be more expensive if Windows could do it, but that would be such
a big benefit when writing portable code or porting code to Windows.
Oh well. (Cygwin supports fork(), and it is hugely expensive
in operation, since it relies on software to copy huge blocks
of memory around, rather than relying on the CPU's MMU to do
copy-on-write (COW) operation - the key to why fork() is so efficient
on Unixes).

Having said that, fork() on Linux is not brillliantly fast. Despite
processors being so much faster than years of old - fork() seems
to be getting slower, possibly related to the need for all
CPUs to synchronise MMU and other state, rather than fork() itself
getting more complicated.

To end users of CRiSP, they may see the initial performance optimisation,
but unless they are working on extremely huge projects, they may
hardly ever notice the change put in place. Also note, that
in CRiSP, one rarely tags an entire project - CRiSP does incremental
updates to the tag database, as you are editing/saving files.

Post created by CRiSP v12.0.3a-b6801

Saturday, 6 September 2014

Firefox, GMail and cookies (solved)

I reported a while back, a problem with Firefox and inability to
login to gmail. After a restart of firefox, you get a cookie error
page. I found a solution was to tediously delete all google cookies - then
I could login. However, logging into my other account would, sooner or later,
give rise to the same cookie mismatch error. It was annoying
and infuriating.

I had resorted to using my phone/tablet to handling email, but that seems
silly (except when on the road).

After repeated scouring the web for solutions or solved solutions,
nothing worked. But I bumped into a reference regarding potential
issue with plugins.

So I tried disabling all my firefox plugs, and it worked !

So I then narrowed down which is the offending culprit.

And the culprit is:

Flash Video Downloader - YouTube Full HD Download

I dont know why, but I assume these plugins act as private
proxies, to intercept web requests, and somehow, it is mishandling

So I can relax that I hope this issue is finally over.

I hope this is useful to others out there when faced with a similar

Post created by CRiSP v11.0.35a-b6772

Tuesday, 26 August 2014

Galaxy Tab S 8.4 / LTE

I recently bought one of these devices. I didnt really need it,
but I wanted a larger version of my phone (Galaxy Note 2), for
video watching. I dislike my mini ipad - it is a great device to hold,
but the whole iOS experience leaves me very cold. That and the puny amount
of RAM. Switching between 3 apps will force restart the apps since
iOS cannot juggle the demands of only 3 apps in the 512MB of memory.

My phone has 2GB and can easily keep 5-6 apps open. So app switching
is faster, because of the lack of a need to restart.

The Galaxy Tab S has a screen of 2560x1600. Although brilliant from
a bragging rights, its difficult to appreciate what this means. What
it means to me is that when I split screen apps, I can read the smaller
fonts of info from web browser, IMDB, etc.

There is definitely some lag in the device. I attribute this to code which
will synchronously try and connect somewhere. I rooted my device, put
in an adblocker, but sometimes the delays are much more than just bad
OS coding. Theres too many services all fighting to wake up and do
things, and even with a quad core cpu, it hangs. I run a tool to monitor
the CPU speed and when it is being sluggish, the CPUs are at their lowest
clock speed (250MHz) which suggests externally waiting for something
and not a CPU issue. I need to run more stuff to see whats going on, e.g.
a network monitoring.

My major complaint with the Tab-S is its so big. I can easily hold
the Galaxy phones in my hand comfortably, in such a way that my fingers
cannot touch the screen. The same is not true of an 8" tablet. You either
gingerly hold it from the sides, e.g. the end where the Samsung label
is, because theres more room for your fingers, or the other end
where the home/back/menu button is located. That end of the tab is
bad news as the system sees you frantically pressing any/all of those buttons.

If you attempt to grasph the device with your fingers mid-screen, chances
you are going to do something you regret (such as click on a link, or
fast-forward a video). Very strangely, the one advantage of iOS is
that it is such an unfunctional OS, that this does not happen
with the ipads. If you are watching a video and use your fingers
to grasp the middle of the screen, then nothing happens, because apple
doesnt attribute any activity to touching the middle of the view area.

So, the one advantage that Android has (and MX-Player - brilliant video
player) is actually a problem on a large screen device, because there
is nowhere safe to hold the device.

I really dont know what is happening with the edge-less phones
currently being touted. I dont know how you can hold one without
having a very erratic user experience from the edges
of your palm or fingers, just trying to hold the device.

Only time will tell.

Post created by CRiSP v11.0.34a-b6771

Tuesday, 12 August 2014

Intel NUC

My NUC decided to die tonight. After only a few weeks. Luckily, google
helped me find the problem.

It had lost its mind/marbles. It couldnt find the root hard drive.

After fiddling in the BIOS, I found that resetting the defaults fixed
the problem.

It is 2014. Issues with BIOS date back to around about 1990 or so - that
was when one bothered to go into the BIOS and configure pointless items.
Since then, stuff works. Thanks Intel. That really is poor software
development - a 24x7 machine suddenly needing a screen to reset the BIOS

Yes, I need to update the BIOS, but go search for BIOS updates for NUC,
and be startingly confused by how many versions there are and advice
not to use the latest one.

Why is the software industry going backwards? Its not just Intel. But
every major software manufacturer is adding functionality and usability
by actively removing it.

Post created by CRiSP v11.0.33a-b6757

Monday, 4 August 2014

Keep at 'em

I keep watching the stats on requests to my rss feed. Google seems
to like me, which is encouraging, and so do some other bots/trackers.

I see the occasional nasty connection trying to break stuff, and
enjoy myself watching some of the attacks. I assume they are attacks
but there may be a reason for some of them. Eg a Japanese person
may be trying out a Unicode request, based on some indirect link
from google, for example.

It did break a record today, and if this keeps up, I can start
figuring out how to really handle high load. At the moment,
the load is low, and even although the feed is written for speed of
development, and not optimal cpu usage - its a great exercise for
some future optimisations to really decide how to sustain higher

I ought to give it a real name, since the P and Q pages are not
really indicative of what the service is.

Definitely an interesting experiment.

BTW I have used 250MB of mobile bandwidth in 3-4 months (to my phone).
I cannot believe how low my bandwidth is, now that all adverts and
other HTML window dressing is removed from the equation.

Who needs 4G? This works at 9600 baud really well :-)

Post created by CRiSP v11.0.33a-b6757

Monday, 21 July 2014

Problems with RSS

RSS Feed

Not sure what did it - maybe I ran as root, accidentally when I didnt mean
to, but nothing would update, despite changing permissions and resetting
some internal state. In the end decided just to clean it all out and
start again. If you are reading the feed, you likely wont notice much
and in a few hours, the differing RSS feeds will race ahead with their
usual updates.

Despite the simplicity of the code, theres always a surprise lurking
in the code - but then, thats the fun of the fair !

Post created by CRiSP v11.0.33a-b6757