Nifty tidbits

Nifty tidbits and random thoughts on technology and anything else that catches my fancy

Hudson for CI – Tips, Tricks and insights

Posted by Raghu on August 28, 2009

Just started using Hudson recently and I’m wowed! It’s head and shoulders above CruiseControl and things that I like a lot are

  1. Snappy web based config – felt great that I could set up a CI build with essentially the repo path alone
  2. Plugin system!
  3. Deep maven2 integration (though read on below that this isnt always what works)
  4. Trending data OOB – essentially giving you nice charts about how your build is doing over time

Now that I’ve said all the very nice things about it, here’s a few things that were hard to figure out/or weren’t immediately apparent. If your maven builds aggregates modules then you’ll find the experience a bit challenging

  1. The generated site doesnt work: Basically, the link is to one of the modules’ site instead of a link to the parent project. This apparently is a known issue and the solution on hudson user list is to run the site:deploy goal and have a link in the project description to point to that url
  2. Code coverage: none of the coverage tools (EMMA, clover etc) support code coverage over a multi module build. Since coverage is very important to me, I eventually resorted to having separate build jobs instead of using the default multi module support. Here’s how my svn structure looks

    /trunk/basebuild #contains the parent pom
    /trunk/project1 # pom refers to ../basebuild/pom.xml
    /trunk/project2 # ditto here

    With the directory structure above, there are build jobs for project1 and project2. Each build job checks out both the project folder (/trunk/project1) and the basebuild folder so that the POM references work.
    One undesirable effect of this set up is that if project 2 depends on project 1, then project 1 build will have to install the artifact to the local repo for the project2 build to work.

  3. Findbugs plugin – Running maven builds with findbugs configured did a Out of Memory (OOM) and failed the build. I tried setting MAVEN_OPTS to -Xmx512M at a bunch of places and nothing worked. Eventually, it turned out that the right place to specify it is in the Hudson COnfigure job page in the build section!
  4. Violations plugin – This is a great little hudson plugin. However, I couldnt get this to work with a inherited POM setup above. Eventually resorted to using Findbugs and PMD hudson plugins individually.

I should mention that I’m running hudson 1.321 with the latest plugins. If you have any tips to share on running hudson – please do drop a link in the comments. Overall, a great big ‘thank you’ to the Hudson folks!

Posted in Java | Tagged: , | Leave a Comment »

Recipe: Unit testing Apache CXF RESTful services

Posted by Raghu on August 26, 2009

Recently, decided to use Apache CXF to expose a service with a RESTful API. Part of the reason for choosing REST had more to do with the fact that the client is going to be a mobile client. These days, though mobile devices stacks have come a long way and provide SOAP clients, it still seems prudent to not depend on a whole slew of technologies where plain ‘ole HTTP and JSON might do the trick.
As I started exploring CXF, I liked the JAX-RS implementation and decided to go ahead with it – however, almost immediately, hit a snag when I went on to write test cases. Apache CXF documentation is not quite there and things do require some investigation – at least initially till you get a hang of the framework. As it took time to figure out the solution, it makes sense to share it on blogosphere. Here’s how to go about writing unit tests:

Firstly, the service and the service implementation:

package com.aditi.blackberry.web;

import javax.ws.rs.FormParam;
import javax.ws.rs.GET;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Response;

@Path("/chat")
@Produces("application/json")
public interface ChatWebService {

 @POST
 @Path("connect")
 public Response connect(@FormParam("user")String username, @FormParam("pass")String password);
}

The service implementation:

package com.aditi.blackberry.web;

import javax.ws.rs.Produces;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;

@Produces("application/json")
public class ChatWebServiceImpl implements ChatWebService {

 public Response connect(String username, String password) {
 if(username ==null || "".equals(username) ||
 password ==null || "".equals(password)) {
 return Response.status(Status.BAD_REQUEST).build();
 }
 String[] response = {username, password};
 return Response.ok(response).build();
 }
}

The corresponding spring context xml (applicationContext.xml) is:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
	xmlns:aop="http://www.springframework.org/schema/aop" xmlns:tx="http://www.springframework.org/schema/tx"
	xmlns:jaxrs="http://cxf.apache.org/jaxrs" xmlns:cxf="http://cxf.apache.org/core"
	xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
       http://www.springframework.org/schema/aop
       http://www.springframework.org/schema/aop/spring-aop-2.5.xsd
       http://www.springframework.org/schema/tx
       http://www.springframework.org/schema/tx/spring-tx-2.5.xsd
       http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd
       http://cxf.apache.org/jaxrs http://cxf.apache.org/schemas/jaxrs.xsd">

	<import resource="classpath:META-INF/cxf/cxf.xml" />
	<import resource="classpath:META-INF/cxf/cxf-extension-jaxrs-binding.xml" />

	<bean id="logInbound" class="org.apache.cxf.interceptor.LoggingInInterceptor" />
	<bean id="logOutbound" class="org.apache.cxf.interceptor.LoggingOutInterceptor" />
	<!--
     logging
     -->
	<cxf:bus>
		<cxf:inInterceptors>
			<ref bean="logInbound" />
		</cxf:inInterceptors>
		<cxf:outInterceptors>
			<ref bean="logOutbound" />
		</cxf:outInterceptors>
		<cxf:inFaultInterceptors>
			<ref bean="logOutbound" />
		</cxf:inFaultInterceptors>
	</cxf:bus>
	<jaxrs:server id="jaxrsservice" address="${server.address}" >
		<jaxrs:serviceBeans>
			<ref bean="chatwebservice" />
		</jaxrs:serviceBeans>
		<jaxrs:providers>
			<ref bean="flexjsonprovider"/>
		</jaxrs:providers>
	</jaxrs:server>
	<bean id="flexjsonprovider" class="com.aditi.blackberry.web.FlexJsonMessageBodyWriter" />
	<bean id="chatwebservice" class="com.aditi.blackberry.web.ChatWebServiceImpl" />
</beans>

A few things to note here – logging is turned on using interceptors and the jaxrs server is defined. I’m also using flexJson to convert arbitrary objects to json – so a MessageBodyWriter bean is also injected into the jaxrs server node. The most important thing is that we havent included either the cxf-servlet.xml config for the cxf-extension-http-jetty.xml. Essentially, what we want to do is for the actual build, include cxf-servlet.xml and for the test runs, run the service on the bundled jetty server.

So, go ahead and define a applicationContext-web.xml:

        <bean id="propertyConfigurer"
		class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:/build.properties" />
	</bean>
	<import resource="classpath:META-INF/cxf/cxf-servlet.xml" />
	<import resource="classpath:/applicationContext.xml" />

This is the context xml that we’ll provide to the ContextLoaderListener in our web.xml.

For the test cases, define applicationContext-test.xml – this is the context xml which we’ll load from the test cases.

	<bean id="propertyConfigurer"
		class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:/test.properties" />
	</bean>
	<import resource="classpath:META-INF/cxf/cxf-extension-http-jetty.xml" />
	<import resource="classpath:/applicationContext.xml" />
	<jaxrs:client id="chatclient" address="${server.address}" serviceClass="com.aditi.blackberry.web.ChatWebService">
		<jaxrs:headers>
			<entry key="Accept" value="application/json"></entry>
		</jaxrs:headers>
	</jaxrs:client>

As you see, we also define a jaxrs:client for the test context xml.

There’s one final issue to address – which is that we would ideally like the urls we use to access the service to be the same. The spring jaxrs:server binding takes an address attribute which defines the url the service is hosted on. For deployment onto an external container, this takes the form of “/myservice” – a path element relative to the context location. For the internal jetty hosted service, it takes the full http path (http://localhost:port/my/path/to/service). The easiest way is to have this set using a property reference in spring and have the applicationContext-web.xml and applicationContext-test.xml load different property files as shown in above.

For completeness, here’s the web.xml:

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">

	<display-name>CXF REST Example</display-name>
	<description>CXF REST Example</description>

	<context-param>
<param-name>contextConfigLocation</param-name>
<param-value>classpath:/applicationContext-web.xml</param-value>
	</context-param>
	<listener>
	<listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
	</listener>

	<servlet>
		<servlet-name>CXFServlet</servlet-name>
		<servlet-class>org.apache.cxf.transport.servlet.CXFServlet</servlet-class>
		<load-on-startup>1</load-on-startup>
	</servlet>

	<servlet-mapping>
		<servlet-name>CXFServlet</servlet-name>
		<url-pattern>/*</url-pattern>
	</servlet-mapping>

</web-app>

And finally, here’s a junit test case:

base class:

package com.aditi.blackberry.web;

import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "classpath:/applicationContext-test.xml" })
public abstract class AbstractApiTest {

	@Autowired
	@Qualifier("chatclient")
	protected ChatWebService proxy;
}

A test case for the connect API:

package com.aditi.blackberry.web;

import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;

import org.junit.Assert;
import org.junit.Test;

public class ConnectApiTest extends AbstractApiTest{
	@Test
	public void testConnect() {
		Response resp = proxy.connect("raghu", "password");
		Assert.assertTrue(resp.getStatus() == 200);
		System.out.println(resp.getEntity().toString());
	}
}

Posted in Java, Web | Tagged: , , , , , | Leave a Comment »

PIL vs Imagemagick

Posted by Raghu on January 1, 2009

Decided that I want to timestamp my photo collection with the date from the exif data. Many digicams have an option to do this – unfortunately, my Panasonic DMC-LZ8 doesn’t seem to do this. I knew imagemagick would do the trick, but thought it would be a good time to play around with PIL and python.

Here’s my PIL effort – functional, but one that came with quite some amount of googling and trying to make sense of the PIL documentation which is inadequate at best.


from PIL import Image
from PIL import ImageFont, ImageDraw
from PIL.ExifTags import TAGS
from os.path import basename, dirname,join
import logging
import sys
import datetime
import time

# Important: I set out to write the image annotation in PIL - there's one serious drawback though. When saving
# the image, the exif data is'nt preserved.

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(levelname)s %(message)s')
logger = logging.getLogger()
logger.level = logging.DEBUG

def readExif(image):
    info = image._getexif()
    ret ={}
    for tag,value in info.items():
        ret[TAGS.get(tag,tag)] = value
    dt = datetime.datetime (*time.strptime (ret['DateTime'],"%Y:%m:%d %H:%M:%S")[0:6])
    ret['DateTime'] = dt
    return ret

def annotateImage (file):
    i = Image.open(file)
    font = ImageFont.truetype("/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans-Bold.ttf", 36)
    exif = readExif(i)
    draw = ImageDraw.Draw(i)
    width, height = i.size
    draw.text((width * 0.7, height - 100),exif['DateTime'].strftime("%a %d-%b-%Y  %l:%M %p"), font=font, fill='orange')
    outfile = join(dirname(file), "Ann_" + basename(file))
    i.save (outfile, quality=98)
    logger.debug (outfile + " saved")

if __name__== "__main__":
    logger.debug ("getting exif for " + sys.argv[1])
    for file in sys.argv[1:]:
        logger.debug ("Annotating " + file)
        annotateImage(file)

Unfortunately, PIL has a fatal flaw – you can annotate the image and save it – but then the saved image doesn’t retain the original image’s exif metadata. I also tried the exiv2 library, but couldnt figure out a way to load the image, annotate it and then copy over the metadata. Googling around didn’t turn up any intersting solutions – so if any of you have any ideas, please share.

Meanwhile, as I was getting tired of coaxing PIL to do what I want, I just wrote a a little bash script to do the same in imagemagick. Its as painless as it can be, comes with excellent documentation, hardly any gotchas, a world of options in case you feel creative and the job gets done in 10 mins. Here’s the bash script below.

#! /bin/bash
# script adds a black 18px bottom border to the pic with the Exif datetime tag
# no safety checks :) . Original pics are left untouched.
while [ "x$*" != "x" ]
do
file=$1;
shift;
outfile="$(dirname "$file")/Ann_$(basename "$file")"
echo $outfile
echo $file
date=$(identify -verbose "$file" | grep 'DateTime:'| sed 's/ Exif:DateTime: //;s/:/-/;s/:/-/')
date="$(date -d "$date" +"%a %d-%b-%Y %l:%M %p")"
convert "$file" -size 1x18 xc:Black -fill White -background Black -append -gravity Southeast -draw "text 0,0 '$date'" "$outfile"
done

Overall, the experience left me dissappointed and dissatisfied with PIL.

Posted in Python, Rant, Tools | Tagged: , , | 3 Comments »

andLinux with Hardy Heron

Posted by Raghu on October 7, 2008

andLinux is  built on top of co-linux (co operative linux) and basically runs side by side with Windows. andLinux packages the whole thing better (coLinux bundled with Xming and a nice systray app allowing you to launch Linux apps right in windows).

Here’s details on getting off the ground – and the reason that I have this post is that though andLinux comes with an installer application, it still needs some amount of fiddling under the hood to make it work. This post is just to make sure I can go through the process again when the time comes

  1. When installing andlinux, choose the COFS option for making your hard drive visible in Linux
  2. Install with the command line option to launch andLinux (do not install it as a service just yet)
  3. Post installation, tweak andlinux’s network setup – set up a couple of virtual TAP adapter . You will have to tweak things both on the linux side and on the windows side. Basically, you create a 2 TAP adapters – one is a loopback and another for sharing your LAN connection. Your wireless network is shared via Slirp (doesnt need a TAP adapter setup).
  4. Keep in mind a gotcha – slirp wont allow you to ping – so if you have only slirp working, then try a wget www.google.com to check if you have network connectivity.
  5. Start the andlinux server (or if its already running) make sure that your c drive is shared – on the bash prompt you should be able to do ls /mnt/windows
  6. do a apt-get update to update your package list. run an update. As of this time, the only prebuilt images on andlinux.org is gutsy.
  7. do a apt-get install update-manager-core
  8. run do-release-upgrade – and you should see apt running and updating your system to hardy.

Posted in Linux, Tips | Tagged: | Leave a Comment »

Compact Ubuntu

Posted by Raghu on June 23, 2008

I’ve always hated the fact that on Ubuntu with the default themes, there’s far too much space wasted. The buttons are too tall, the treeview wastes too much space so that if you’re on eclipse or some other ide, you see a precious few items on the screen.

I’ve been trying to tweak it to no end – even looking to see if there are any ~/.gtkrc-2.0 tweaks. Found a few links such as this Making Eclipse look good on Linux – Max’s blog - however, didn’t really satisfy my need.

And so it stayed until today when I came across Clearlooks Compact Gnome Theme.

I love it – one more for my list of must-haves!

Posted in IDE, Linux, Rant, Tips, Troubleshooting, Utilities | Tagged: , , , , | Leave a Comment »

Enjoy symlinks and hardlinks on NTFS

Posted by Raghu on June 18, 2008

Can’t believe I didnt come across this before – if you’ve gotten used taming your hdd by creating links to folders and have been annoyed with the lack of symlinks and hardlinks on NTFS, then despair no more. I’ve been using Mark Russinovich’s (of sysinternals fame) tool – junction.exe all this while and though it works great, have always wanted something that would integrate with Explorer too. For an in-depth discussion – read http://shell-shocked.org/article.php?id=284 Anyways, I’m extremely happy with NTFS Link – this will surely go into my list of “Must have tools – install immediately on a new machine” list :-)

Posted in Tips, Tools, Utilities, Very Cool | Leave a Comment »

Upgrade blues – upgrading to Firefox 3 final from Firefox RC 3

Posted by Raghu on June 17, 2008

As evident from other posts here – have been keenly waiting for the FF 3 final. Imagine my surprise when the “Check updates” didnt find an upgrade! (I’m on FF3 rc3).

Anyway, so off I went to Mozilla.org and downloaded a copy of the final – and did my bit towards FF download day. Happily installed it – all defaults as usual. Install told me that it was installing into the same location as my current installation (c:\program files\mozilla firefox 3 beta 1 – that’s where my FF3 install have been going  – all the way from b1 to b5 and then from rc1 to rc3 – so no surprise).

Well, installation completed successfully, and I started FF 3 – but my title bar still says Build 2008052906 – even the file version has the same build ID.

Something’s up – don’t know what yet – but has anyone else had a similar experience?

Posted in Firefox, Web | 2 Comments »

Desultory Monday…

Posted by Raghu on June 16, 2008

This entry was posted using Its all text on Firefox 3.0 RC2 on Ubuntu Hardy heron, with emacs 23 snapshot as the editor. I love it :-)

Well, Its all Text is great if you hate typing into webforms with textboxes that make editing such a big pain in the butt.

Its great to see that Its All text has been updated to work with FF 3.0 now. The fun would be to see if this works on Windows with cygwin emacs as the editor. Had problems the last time I tried that – but that’s been sometime ago now.

Today’s been a desultory Monday. Spent sometime getting emacs snapshot with pretty fonts on my hardy. Its beautiful.

The next thing has been mostly scratching my head on hadoop. What I’d like to do is parse an access log and generate multiple outputs – ie single input of gobs of web access logs and multiple outputs – with say requests by country, popular pages, % of client browser and so on.

  1. parse web log
  2. pull out remote ips and use geo ips to find the originating country
  3. pull out user agent field and figure out browser distribution.
  4. Filter the requested resource and pull out only pages – find pages by popularity

Now there seem to be quite a number of ways of doing this -

  • Code the whole thing in Java – and this is where I’m getting into analysis paralysis.
    Look at ways to generate multiple outputs from MapRed and then use Job and JobControl to setup the pipeline.
  • Use Pig – Pig examples on the Pig overview page seem to suggest that this should be trivial with Pig.
  • Use Cascading – seems to be doing the same thing – will need to do this in JRuby or Groovy though.

Will post an update once I get through the java route

Posted in Java | Tagged: | Leave a Comment »

VPN into Windows VPN Server from Ubuntu *Hardy* Intrepid

Posted by Raghu on June 12, 2008

** Update 2008/11/17 **: Networkmanager is broken in intrepid. To get it  working had to install network manager from ppa as given here – http://www.ubuntu-forums.com/showpost.php?s=e0d93c09b8c340976477456593ac4cf7&p=6094870&postcount=5

Ok – this was easy – and while there’s some resources on google, I had to figure out a few itty bitty things for my work VPN setup.

install

  • network-manager-pptp
  • pptp-linux

Restart network manager with

killall nm-applet
sudo /etc/init.d/dbus restart
nm-applet --sm-disable &

Configure VPN settings

Click on the network manager applet and click on VPN connections

  1. Create a new VPN connection
  2. Ensure that you select Refuse CHAP  in the authentication tab.
  3. In the routing tab, you can give netmasks that need to go through VPN – for my work network, I have: 10.10.5.0/24 172.16.106.0/24

That’s it. Now click on the Network applet, and connect to your VPN. In the authentication dialog, use <domain>\username and your windows domain password.

Posted in Linux, Tips, Tools, Utilities | 5 Comments »

Scalability principles: Lessons from eBay

Posted by Raghu on June 5, 2008

InfoQ: Scalability Best Practices: Lessons from eBay

Great article on Ebay scalability principles.

 

Posted in Java | Leave a Comment »