Enjoy symlinks and hardlinks on NTFS

Can’t believe I didnt come across this before - if you’ve gotten used taming your hdd by creating links to folders and have been annoyed with the lack of symlinks and hardlinks on NTFS, then despair no more. I’ve been using Mark Russinovich’s (of sysinternals fame) tool - junction.exe all this while and though it works great, have always wanted something that would integrate with Explorer too. For an in-depth discussion - read http://shell-shocked.org/article.php?id=284 Anyways, I’m extremely happy with NTFS Link - this will surely go into my list of “Must have tools - install immediately on a new machine” list

VPN into Windows VPN Server from Ubuntu Hardy

Ok - this was easy - and while there’s some resources on google, I had to figure out a few itty bitty things for my work VPN setup.

install

• network-manager-pptp
• pptp-linux

Restart network manager with

killall nm-applet
sudo /etc/init.d/dbus restart
nm-applet –sm-disable &

Configure VPN settings

Click on the network manager applet and click on VPN connections

1. Create a new VPN connection
2. Ensure that you select Refuse CHAP  in the authentication tab.
3. In the routing tab, you can give netmasks that need to go through VPN - for my work network, I have: 10.10.5.0/24 172.16.106.0/24

That’s it. Now click on the Network applet, and connect to your VPN. In the authentication dialog, use <domain>\username and your windows domain password.

Free subversion hosting - What’s the best?

All - I’ve just signed up for an Assembla account - these folks provide free subversion hosting with a 500 meg space and unlimited spaces.

Will see how it goes.

Gnuplot, dstat - easy graphing on Linux

Recently, started fiddling around with how to monitor and graph performance data on linux boxes. Other than the usual tools like top and vmstat, which are either interactive (top) or too textual to do anything much.

First off, vmstat, doesnt lend itself well to graphing without additional scripts to lay out the data so tools like gnuplot can be used. Secondly, and more seriously, it doesn’t include a timestamp in the output.

Looking around a bit found that dstat seems to be a good replacement to vmstat (and iostat) - and the generated data is consumable with gnuplot.

Here’s a quick example of generating graphs for CPU user, system and idle times

dstat -tc 5 500 > dstat.raw

now fire up gnuplot and go ahead and plot it

gnuplot> set xdata time gnuplot> set timefmt "%s" gnuplot> set format x "%M:%S" gnuplot> plot "dstat.raw" using 1:2 title "User" with lines, "dstat.raw" using 1:3 title "Sys" with lines, "dstat.raw" using 1:4 title "Idle" using lines

To make gnuplot generat an output file, you need

gnuplot> set term png

gnuplot> set output “dstat.png”

gnuplot> replot

And you’re done. here’s the graph generated on my machine. There’s loads more that you can do - and admittedly, you can do everything by dumping your file to excel. However, that doesn’t lend itself well to a completely automated process. When you’re doing performance testing and such like, you will likely repeat this enough number of times. Not having to do it manually helps big time!

Working with huge XML files - tools of the trade.

XMLStarlet is great for slicing and dicing huge XML files. Had a run in recently - had a 80 Mb XML file in a single line :D. Guess what, most editors that I tried balked and fell over. This was on a 2Gig Core2 Duo machine.

XMLSpy, vi, emacs, notepad++ all died - and trying to do something with a 80 Gig XML where the 80 gigs are on a single line isnt much fun. So the first order of business was to pretty print the XML. XMLstarlet worked great -

xmlstarlet fo file.xml > output.xml

and you’re done.

The next order of business was that we needed to validate the XML document against a schema. Our first attempt was with Sun’s multi schema validator (MSV). MSV does not validate the whole document but instead stops after a certain number of failures. So, MSV - out, XMLStarlet in. XMLStarlet can validate documents again W3C schema, DTD  or a RELAXNG schema.

xmlstarlet val --err --xsd schema.xsd input.xml >  errors.txt

And presto! - you get an error report that you can slice and dice with sed/awk or anything else at all.

XMLStarlet also allows you to write Xpaths to query the xml - however, I found the syntax too weird and round about. A better alternative is a perl based solutions - XSH2 - a command line xml editing shell. You can install it under cygwin and it supports basic command pipelining and redirection.

So go ahead and launch XSH. At your cygwin prompt

[~]xsh
—————————————
 xsh - XML Editing Shell version 2.1.1
—————————————

Copyright (c) 2002 Petr Pajas.
This is free software, you may use it and distribute it under
either the GNU GPL Version 2, or under the Perl Artistic License.
Using terminal type: Term::ReadLine::Gnu
Hint: Type `help’ or `help | less’ to get more help.
$scratch/>

Now, lets load up our document, type

$scratch/>$x:=open formatted.xml

Your prompt changes to

$x/>

So go ahead and try a few xpaths

$x/> ls /path/to/node

and XSH prints out the matching nodes. Now what if you need to create a document fragment of nodes matching a certain xpath? Piece of cake - do ahead

$x/> ls /path/to/node | tee fragment.xml

XSH2 has many, many more features - but this should be good enough to get you off the ground.

Piclens - full screen slideshows with flickr (and others)

discovered Piclens

Its a great addin for firefox - and integrates with Flickr to give you full screen slideshows a’la Picasa slideshows on your machine!! Its a bit tricky to figure out how to get it to work - Just hover any picture on any page  and click on the blue bubbly overlay button that appears

Maven Emma plugin - filters dont work

Using v0.5 of the Maven emma plugin. Found that if you specify an include/exclude filter, no classes are instrumented….On digging a little deeper under the covers, it seems that there’s a bug with the Maven emma plugin - I’ve edited the plugin’s jelly script and using it here.

Posted the bug and the fix to Maven’s sourceforge developer forum. You can find it here

The fixed plugin.jelly’s here - Maven emma plugin 0.5 fixed

Ant - Debugging classpath

Here’s a nice tip on debugging classpaths in Ant: http://www.javalobby.org/java/forums/t71033.html

Essentially,

<?xml version="1.0"?>
<project name="project" default="default">
	<property name="lib" value="web/WEB-INF/lib"/>
	<property name="src" value="src"/>
	<property name="dist" value="dist"/>

	<path id="classpath">
	  <fileset dir="${lib}">
		  <include name="**/*.jar"/>
	  </fileset>
	</path>

    <target name="default" description="--> description">
    	<javac srcdir="${src}" destdir="${dist}">
    		<classpath refid="classpath"/>
	</javac>
    </target>

</project>

Cobertura vs EMMA

This is a rant against the Cobertura maven integration - and to a certain extent on Cobertura also - and hopefully an objective one ;). Hopefully this’ll help someone select between the two big (open source) coverage tools out there.

We’ve been using Cobertura at work till now and its done its job nicely - the reports look great and the maven 1.x integration, while not neat, is functional. While we knew that someday we were going to have to merge coverage data and have a single report across multiple test methods (junit, selenium tests and manual), cobertura documentation stated that this was possible and so we weren’t really bothered.

Thought I’d give it a whirl and set it up - and that’s when the trouble started. Atleast, with the maven integration.

First of all - there’s no way to just instrument code. You can generate the report (which will instrument classes and run the tests) but if you just want to instrument classes so that the final deployable contains instrumented classes, its a no go with the maven plugin tools.

Obviously, no point giving it up there - so thought I’d just include the ant tasks and go the ant way in my maven goal. Turns out that there’s no ‘plugin init’ kind of goal that can be called post build:start that will set up dependencies and import the cobertura ant tasks. You have to do it all yourself.  Fine - went that way too - so now my maven.xml uses the cobertura ant tasks and finally I’m able to generate an instrumented build. YIPPPPPPPEEEEEE…. or wait…lets’ just make sure that this thing works…

Does it?

Turns out - no - it doesnt - so I dropped the WAR into tomcat and accessed the login page of the application and then shut down tomcat nicely. There’s even a cobertura.ser created in the tomcat bin folder and I’m thinking that probably this will all work together finally…

So I go ahead, tweak my Maven.xml further with a coverage task that will merge the data from the junit runs and servlet container runs. Turn the switch on… and lo and behold…Exception reading the merged data file. Back to google and after hunting around for sometime found this Bug while merging reports

So finally I’m ready to give up cobertura and give Emma a try…and it couldnt have been better…

1. Goals are nicely setup

2. You can init the emma system with the emma:init goal and then use ant tasks if you want flexibility for doing things like merging reports.

3. The merging works :))

One sticky issue that I did come up  with was that the for the same source and test cases the coverage reported by cobertura and emma differ widely. With Cobertura, we were at 40% coverage while with EMMA, the number’s up to 60% coverage - and while EMMA has some literature on how it does things - I’d be glad if someone did explain why or how the reported numbers could be so different for the same base code and unit test suite?