Downloading over an unreliable connection with Wget

Rant – BSNL!!!

This is a part rant, part tip – so bear with me… My broadband connection absolutely sucks over the past week. I upgraded from 2Mbps with a download limit to a 4Mbps with unlimited downloads and since then it has been nothing but trouble… Damn BSNL!! I’ve probably registered about 30 odd complaints with them to no avail. If there was a Nobel for bad customer service, BSNL would probably win it by a mile. Some examples:

1. They’ll call to find out what the complaint it and even when I explain what’s happening, they hardly hear me out at all.
2. Either they call up and say ‘We have fixed it at the Exchange’ and nothing has changed
3. They automatically close the complaints

Guess they find it too troublesome that someone who’s paying for broadband actually expects the said broadband connection to work reliably!

Anyway, Airtel doesn’t seem to be any better – they need 10 days to set up a connection and when I was on the phone with them, they didn’t seem too interested in increasing their customer count by 1 .

I also tried calling an ISP called YouBroadband after searching some of the Bangalore forums for good ISP providers. They promised a call in 24 hours to confirm if they have coverage in my area and it was feasible for them to set up the connection and that was 48 hours ago!

At work, I’ve heard good things about ACTBroadband and they have some ads in TOI as well, but they said they don’t have coverage in my area .

So how do you download

Today I needed to download something and doing it from the browser failed each time since my DSL connection would blink out in between!

After ranting and raving and writing the first part above and still mentally screaming at BSNL, decided to do something about it… Time for trusty old wget – surely, it’ll have something?

Turns out that guess was a 100% on the money… it took a few tries experimenting with different options, but finally worked like a charm

wget -t0 --waitretry=5 -c -T5 url
    where
    -t0 - unlimited retries
    --waitretry - seconds to wait between retries
    -c resume partially downloaded files
    -T5 - set all timeouts to 5 seconds. Timeouts here are connect timeout, read timeout and dns timeout

Yoohooo!! Successfully compiled Android from source

Finally!!!

So my last weekend project had been to compile Android ICS from source. Given that the size of the repo itself is in excess of 6Gigs, just getting it down itself took the better part of Friday night and Saturday night. When I got down to running make on it, it was Sunday afternoon.

Needless to say, things didn’t work too well. I’m running this on a 32 bit Ubuntu 10.04 Virtualbox with a piddly 1GB RAM. When make failed the first time, realized that swap was a measly 300Mb. First steps first, went on to increase memory to 2GB (that’s all I can spare) and increased swap to 2Gs.

Compilation next round started and that failed too – ran out of disk space – and this was Sunday night. Things kind of stayed there and finally this evening, resized the disk in virtualbox to 50Gigs. Again started the compilation and this time ran into linker errors when building webcore. One more round of troubleshooting involved deleting the previously built static library and then running make again. Surprisingly, this time make completed successfully – to the point where I wasn’t sure if it had succeeded or just failed silently on something else.

The next step was to run the emulator to see if it really would boot up. Over at source.android.com, they oversimplify it when they say that you just run emulator from the android root folder. That didn’t work for me – and this time it was because I hadn’t sourced the envSetup.sh file… this thread http://groups.google.com/group/android-platform/browse_thread/thread/91ff18e034acf951 helped in tracking that one down.

So finally, after all that trouble, I have my very own ICS build running!!!!

For now, its onward ahoy to setting up Eclipse and starting with a fix I’ve been mulling about for sometime now..

Signing off from cloud nine
R

Unit testing Apache CXF RESTful services – code available

So, the original post on the topic written about two and a half years ago had code snippets, but there’s been comments and PMs for the complete code. So last week, as I resurrected this blog, decided to get that code out on github. Unfortunately, that was easier said than done; it has been quite some time and frankly, I’d lost the code. I must’ve switched machines about 3 times in the interim and gone from SVN to github for personal projects. Some hunting around ensued and thankfully, I was able to find the actual code we wrote based on the sample I’d posted. So cleaned that up – and just extracted the unit testing example out of it and pushed it to github – get it here. I haven’t updated any of the dependencies – so this is still running against spring 2.5 and cxf 2.2.3 (I think) and things might’ve changed quite a bit since then (I haven’t used the JAXRS bits of CXF much after that)

Running tests:

        mvn test

Running the server:

        mvn jetty:run

Creating an interstitial login page with JqueryMobile

So, at work, we’re building a mobile website using JqueryMobile. The app has a bunch of publicly visible pages however, other pages require the user to be authenticated. We didn’t want the user to be forced to login on the first page. Instead, whenever a protected page is accessed, and if the user insn’t logged into the app, we’d like to take him to the login page. Once he’s successfully authenticated, then take him to the page he was navigating to. Doing this in a normal webapp is quite standard – however, with JqueryMobile, query params meddle with the hash navigation model. Also, the page that the user tries to access could be a div in the same physical page or a different url that needs to be fetched.

Trying to solve this was interesting as we were all really just getting started with JqueryMobile – so finding the ideal solution required a few tries. The solution takes a leaf out of JqueryMobile’s approach. The outline of the solution is:

1. Any page div that’s a protected resource is marked with a data-needs-auth="true" attribute
2. We hook into the document level pagebeforechange event to see if the user is trying to transition to a page requiring authentication. If so, then check if we have the user’s authenticated context available.
3. if the said context isnt available,
   1. Cancel default event handling since we’re now going to navigate the user to the login page.
   2. save the toPage object – so once the user is logged in, we know where to take him.
   3. navigate to the login page.
4. In the login page, the page can call the server apis to autheticate the user. Once the user is authenticated, then
   1. See if there’s a valid returnTo object, if so, take the user to the page.
   2. If not, take the user to a ‘default’ page – in our case, this is the app dashboard page.

Code below

    var pageVars = {}
    $(document).bind("pagebeforechange", function (event, data) {
        if (typeof data.toPage == 'object' && data.toPage.attr('data-needs-auth') == 'true') {
            if (!sessionStorage.getItem("TokenSSKey")) {
                if (!localStorage.getItem("TokenLSKey")) {
                    pageVars.returnAfterLogin = data;
                    event.preventDefault();
                    $.mobile.changePage("#Login_Page", { changeHash: false });
                }
                else {
                    sessionStorage.setItem('TokenSSKey', localStorage.getItem("TokenLSKey"));
                }
            }
        }
    });

The login event handler that handles the server response that’s received once we pass the username and password

    function SuccessLogin(data) {
        if (data != null && data.LoginResult != null) {
            if (data.LoginResult.Code === 0) {
                localStorage.setItem('UNameLSKey', data.LoginResult.User.AccountName);
                if ($("#RememberMeChkBx").is(":checked")) {
                    ErrorPanel.html("");
                    localStorage.setItem('TokenLSKey', data.LoginResult.Token);
                    sessionStorage.setItem('TokenSSKey', data.LoginResult.Token);
                }
                else {
                    ErrorPanel.html("");
                    sessionStorage.setItem('TokenSSKey', data.LoginResult.Token);
                }
                if (pageVars && pageVars.returnAfterLogin) {
                    $.mobile.changePage(pageVars.returnAfterLogin.toPage);
                }
                else {
                    $.mobile.changePage("#DashBoard_Page", { changeHash: false });
                }
            }
        }
    }

Learning Vim

Is it worth it?

Definitely seems to be. I’ve looked at VIM in the past, tried it out too a couple of times or more, failed miserably(mostly within a day or two) and then wondered Why nutheads use VI. This would usually be followed with going back to the comfort of Emacs. I think over the years, I’ve spent more time customizing Emacs than actually getting any work done with it. And somewherethat felt wrong. In light of that, the minimalistic VIM looked attractive and worth another try.

So what about this time?

So this time things worked out a bit better. Rather than firing up VIM, spent some time reading through other’s experiences on picking up VIM. And the first thing I did right was to disable the arrow keys in normal mode (I still have them in insert mode)

    " disable arrow keys
    noremap   <Up>     <NOP>
    noremap   <Down>   <NOP>
    noremap   <Left>   <NOP>
    noremap   <Right>  <NOP>

Once you have that bit, you’re forced to use h/j/k/l. And while h/j/k/l muscle memory is built up within a week, the nice thingthat really happens is that you dont use h/j/k/l much – instead you move to using more efficient movement commands. There’re aton of resources/cheatsheets on the web – but the approach I followed was to figure out some small keystroke when I needed it.What that mean’t was that I could get work done – but at the same time get more efficient gradually.

customizations

VIM out of the box is pretty badly configured – and that’s part of the reason that people seem to shy away from it. In fact, all the times that I tried out VIM before, I didnt even come close to cusotimizing my .vim. There are folks who have curated vim dotfiles on github etc – but my advice is to stay away from them. You should know what goes in your .vim and be in control of thatrather than getting a bunch of things in your .vim that you dont understand. Just so you know, looking at the github history for my vimfiles repo – the initial commit was 3 months ago – but after that, all the commits have come in only in the last 4 weeks.What that means is that while I put in a vim file initially, I didnt do much with it initially since I was just getting a hang of the basics. Once one becomes comfortable with the basics, one moves to customizing the vim environment more and more.

Parting words

To summarize, VIM definitely seems nice once you invest into it. It’s easy to drop off in the initial stage and not go any further – and I believe this is what happens to the vast majority of folks who try it out. However, once you build that initial comfort level,it feels light, fast and easy.Start easy, persist, and customize bit by bit – you’ll feel yourself going from struggling with Vim to feeling comfortableand then to customizing your environment for an even better experience with VIM.I’ve definitely been more productive with VIM than I ever felt I was with Emacs – and these posts to my blog from Vim part of that.Besides that, I’ve used VIM effectively with a decent sized js code, html markup etc and felt the speed of editing inspite of still beinga noob in Vim terms.

Blogging with Vim

So now I’m in Vim land and this is the first time I’ve gotten far enough to feel a bit comfy. Decided to dust off my blog and start at it again – what better to do it in than in VIM.

So – TA-DA – here’s the first post – courtsey VIM on ubuntu. However, as usual, it was rougher than it’s supposed to be. IN any case, I’ll forget how I got this far the next time so the next few posts will be around recording how to get VIM to post to WP.com blogs.

But before that – the first thing to to is to get the VimRepress plugin. Better if you have pathogen installed, in which case you can do

    cd .vim
    git submodule add https://github.com/raghur/VimRepress bundle/VimRepress.git

That’s my fork on Github of https://github.com/connermcd/VimRepress.git which fixes a few things:

• Makes VimRepress work properly through a proxy
• Changes the attachment filename to a ‘.odt’ since WordPress.com doesn’t allow a text file attachment.

I still dont have a clue if doing this will break the plugin – but nevertheless, basic case of posting to my blog works and at this stage that seems good enough for me.

PS as you can see from this post – I’ve not yet got a hang of markdown syntax

Dec 29th – PPS a couple of posts and one more tip for WordPress.com. WP.com does <br/> for hardbreaks in the markdown text. Obviously, this doesnt leave the post looking very good. I have the following in my .vimrc to get around this

        augroup Markdown
            autocmd FileType markdown set wrap 
                                    \ linebreak
        augroup END

PPS
You will also need to have python markdown installed once you have VimRepress running.

    easy_install markdown

Gnuplot, dstat – easy graphing on Linux

Recently, started fiddling around with how to monitor and graph performance data on linux boxes. Other than the usual tools like top and vmstat, which are either interactive (top) or too textual to do anything much.

First off, vmstat, doesnt lend itself well to graphing without additional scripts to lay out the data so tools like gnuplot can be used. Secondly, and more seriously, it doesn’t include a timestamp in the output.

Looking around a bit found that dstat seems to be a good replacement to vmstat (and iostat) – and the generated data is consumable with gnuplot.

Here’s a quick example of generating graphs for CPU user, system and idle times

dstat -tc 5 500 > dstat.raw

now fire up gnuplot and go ahead and plot it

gnuplot> set xdata time gnuplot> set timefmt "%s" gnuplot> set format x "%M:%S" gnuplot> plot "dstat.raw" using 1:2 title "User" with lines, "dstat.raw" using 1:3 title "Sys" with lines, "dstat.raw" using 1:4 title "Idle" using lines

To make gnuplot generat an output file, you need

gnuplot> set term png

gnuplot> set output “dstat.png”

gnuplot> replot

And you’re done. here’s the graph generated on my machine. There’s loads more that you can do – and admittedly, you can do everything by dumping your file to excel. However, that doesn’t lend itself well to a completely automated process. When you’re doing performance testing and such like, you will likely repeat this enough number of times. Not having to do it manually helps big time!

Working with huge XML files – tools of the trade.

XMLStarlet is great for slicing and dicing huge XML files. Had a run in recently – had a 80 Mb XML file in a single line . Guess what, most editors that I tried balked and fell over. This was on a 2Gig Core2 Duo machine.

XMLSpy, vi, emacs, notepad++ all died – and trying to do something with a 80 Gig XML where the 80 gigs are on a single line isnt much fun. So the first order of business was to pretty print the XML. XMLstarlet worked great -

xmlstarlet fo file.xml > output.xml

and you’re done.

The next order of business was that we needed to validate the XML document against a schema. Our first attempt was with Sun’s multi schema validator (MSV). MSV does not validate the whole document but instead stops after a certain number of failures. So, MSV – out, XMLStarlet in. XMLStarlet can validate documents again W3C schema, DTD  or a RELAXNG schema.

xmlstarlet val --err --xsd schema.xsd input.xml >  errors.txt

And presto! – you get an error report that you can slice and dice with sed/awk or anything else at all.

XMLStarlet also allows you to write Xpaths to query the xml – however, I found the syntax too weird and round about. A better alternative is a perl based solutions – XSH2 – a command line xml editing shell. You can install it under cygwin and it supports basic command pipelining and redirection.

So go ahead and launch XSH. At your cygwin prompt

[~]xsh
---------------------------------------
 xsh - XML Editing Shell version 2.1.1
---------------------------------------

Copyright (c) 2002 Petr Pajas.
This is free software, you may use it and distribute it under
either the GNU GPL Version 2, or under the Perl Artistic License.
Using terminal type: Term::ReadLine::Gnu
Hint: Type `help' or `help | less' to get more help.
$scratch/>

Now, lets load up our document, type

$scratch/>$x:=open formatted.xml

Your prompt changes to

$x/>

So go ahead and try a few xpaths

$x/> ls /path/to/node

and XSH prints out the matching nodes. Now what if you need to create a document fragment of nodes matching a certain xpath? Piece of cake – do ahead

$x/> ls /path/to/node | tee fragment.xml

XSH2 has many, many more features – but this should be good enough to get you off the ground.

HOWTO: Access your machine from the internet without a static IP

For machines to be accessible on the internet, usually you need a static IP that’s leased from your ISP so that when someone types in your IP address, so that packets can be routed over to your machine. However, getting a static ip is costly and for the most part, internet users have dynamic IP address that the ISP allocates each time an end user connects to the internet. Since the ip address keeps changing on each connection, there’s no straightforward way to connect to the machine without knowing the IP address that’s been allocated – or so it was at least till Dynamic DNS came along (it isnt new – has been around for ages, but for some reason isn’t that well known)

Typically, when you type in http://www.google.com in your browser, your machine performs a DNS (Domain name service) lookup with the DNS servers from your ISP to find out the IP address corresponding to http://www.google.com. With DDNS (dynamic DNS) this is made to work with your dynamically allocated IP address also. Here’s how it works

1. Register with a DDNS service provider. Service provider provide free accounts for personal use – go to www.dyndns.org
2. Once you’ve created your account, go ahead and set up your hostname. DDNS service providers will have some domains that you can choose from and you get to choose the host part. For a fee, you can also use a domain name of your choice.
3. If your set up has a router at your end, check your router administration page if it supports dynamic DNS. If it does, you need to enter the hostname, account and password. Everytime your router connects to the internet, it sends an update notification to the DDNS service notifying the new IP obtained from your ISP. The DDNS service takes care of sending update notifications to routers on the internet.
4. If you dont have a router, then download the DDNS client software from the service provider. Most DDNS providers have windows, mac and linux clients. These run on your machine and do the same thing – notify the DDNS service provider of your new IP whenever you establish a connection with your ISP.
5. If you’ve got all this set up, then you can reach your machine from the net – try ping <your host name>

If you’re running Linux/Ubuntu, make sure your’re running SSH service and try ssh <your host name>. If you have a router setup, then you will need an additional step – basically the DDNS name refers to your router IP – and not the machine behind the router that you wish to reach. You will also need to make sure that your machine has a static IP from your router. To set up your router, go to your router administration page.

1. Go to the LAN section and give a range of IPs outside of the static IP. Most routers have lan addresses like 192.168.x.y – 192.168.x.z. If you want your host to have an IP address of 192.168.1.100, then give a LAN range that does not include this IP – say 192.168.1.110 – 192.168.1.200.
2. Save and reboot your router.
3. Now go to your network settings and enter your static IP (192.168.1.100), netmask 255.255.255.255, gateway (usually 192.168.1.1).
4. Go to your router administration page and look for a section like virtual server – your router will allow you to forward packets received on a particular port to a host and port within your LAN. You will have to enter the external port (we’ll use 22), the internal machine to forward (192.168.1.100) and the port to forward to (22). With this in place, any packets received on port 22 (ssh) on your router will be forwarded to the 192.168.1.100 machine on the ssh port.
5. Save and reboot your router.
6. Give it a spin.

From a different machine (or from the same one -doesnt matter), try out ssh <your host> and you should be able to login to your machine – via the internet.