Continuous Integration for Python

After beavering away at some ideas for Knowsis over the last 3 weeks and admittedly not really doing it test first , I spent this weekend finally getting round to setting up a CI server and some builds to run the pitiful number of tests that I have actually written to try and make me write more. It’s been bugging me all along but as the only developer at the moment it’s not been at the top of the priority list. However, my previous experience of setting this kind of thing up for legacy projects tells me that if I don’t get round to it soon, it will be infinitely more painful in the long run.

At 7digital we used Teamcity as  our CI build server, but knowing how much their build agent licensing can cost I thought I would look at the open source alternatives seeing as we’re bootstrapping. After some research a nailed it down to the either BuildBot or Jenkins (formerly Hudson) and digging a bit deeper it seems that people with experience of both would suggest using Jenkins first until you realise it can’t do something that you really need BB for as it can be quite painful to get set up; Jenkins on the other hand is very simple to get set up.

Jenkins CI Dashboard

Installing Jenkins

One of my favourite things about using Ubuntu, coming from a Windows background, is the ease of installing things using apt-get. These instructions are taken from the Jenkins site:

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo aptitude update
sudo aptitude install jenkins

Jenkins gets installed and set up to run as a daemon at startup under a newly created user Jenkins. It’s now usable at http://127.0.0.1:8080, however I wanted to be able to skip the need for a port number so set up nginx to proxy requests for me. This isn’t a necessary step so I won’t go into it here but there are some simple guides available on line if you haven’t used nginx before (just remember to restart nginx after you change the config, it’ll save you hours of head scratching!).

Setting up Jenkins to work with git

This step isn’t necessary if you don’t use git, but i’ll go into it as I do and it took me a bit of figuring out, plus there wasn’t a huge amount of info out there on how to do it.

From the home screen of Jenkins go to the plugins section:

Manage Jenkins -> Manage Plugins

In the Available tab find the “Jenkins GIT plugin” and check the install checkbox. If you use github you can also install the “GitHub plugin” which creates a link from your project page to your github repository and also allows you to use GitHub’s post receive hooks to notuify Jenkins when code has been committed (not necessary as you can use polling to check for changes). Your Jenkins instance will need to be exposed publicly for this to work, so make sure you set up user authentication properly; there’s also a plugin to allow you to use your GitHub logins as authentication if you want to use that.

Once you have selected the required plugins click ‘Download now and install after restart’ which will install the plugins and restart Jenkins, should take no more than a minute to complete.

Create SSH keys for Jenkins

You now need to set up your ssh keys for the Jenkins user. Open up a terminal window and switch to the Jenkins user

sudo -su jenkins

You can run through the creation of your public private key pairs as normal which will be created in the Jenkins user home directory (/var/lib/jenkins). If you want a guide for this, I have always found the one on the GitHub site help pages to be the easiest to follow.

Now set up a user with your git repository for your build slaves to run as and copy the contents of the public key to it. You can use your own account if you wish but I would recommend using a separate one.

Creating Jobs

Creating a Job in Jenkins is really simple. From the Jenkins dasboard click “New Job”. Enter the name of your job and select “Build a freestyle software project“. Click Ok.

For now you can ignore the options at the top of the next screen, head  down to the source control section.

Source Code Management

If you installed the git plugin you should see git as an option here.  Select the option that’s relevant and point it to the location of your repository.

Build Triggers

Further down is the “Build Triggers” section, you should select the option ‘Poll SCM‘ option, this will then present you with a schedule box that will allow you to enter the frequency to poll your SCM, it uses the cron format. A few examples

Every minute:

* * * * *

Every 10 minutes:

*/10 * * * *

Every hour :

@hourly

At 15 mins past every hour:

15 * * * *

Build Steps

So far the job will just checkout when there are any changes to your code, so now you need to make the job actually do something interesting. You can set up one of more build steps to run for your unit tests, tp deploy your code to a test environment, run your system tests, deploy your code to live etc.

At this point i’ll just run the unit tests. If you have written some tests using unitest syntax you can use the nose test runner (nosetests) to automatically discover and run these tests. You can also get it to output the test results in Junit report format so that Jenkins can display your test results. You will need to make sure that nose is installed on your build server and slaves for this to work.

Select “Excute Shell” in the “Add Build Step” dropdown and add the following line:

nosetests --with-xunit

The shell script will run from the top level of your project (known as the workspace root) so if nose cannot auto discover your tests because they are buried in a folder tree structure you can always add a cd command to switch to that directory first. The  –with-xunit switch will output an xml report in the Junit format called nosetests.xml into the folder under which nosetests ran.

In the Jenkins set up there is a “Post-build Actions” section under which you should select “Publish Junit test report results” and enter nosetests.xml

If you use some other test format or want to use another test runner enter the shell command that would execute those tests remembering to install anything required onto your build server as well.

Now Go And Write Some Tests

That’s it, you now have a build set up to run your tests every time you check in changes, so there is no excuse not to write any. This is only a start and there a plenty of other things you might want to set up like failure notifications, test reports, dashboards etc so the best thing to do is explore the Jenkins site.

Advertisements

Lovefilm vs. Netflix vs. Sky Movies vs. Blinkbox

I love watching movies. I could quite happily sit and watch 2 or 3 back to back on a Sunday as long as the F1 isn’t on. I’ve been a Sky Movies subscriber since getting Sky about 5 years ago and was always pretty satisfied with the service. In late 2010 Sky launched a partnership with Xbox which changed the game completely as it meant I now had access to 100’s of on demand movies at no extra cost.

The improvements to the Xbox dashboard released in December (2011) to include apps from third party movie streaming services such as Blinkbox, Lovefilm and as of today Netflix has drawn to my attention to the possibility that I could get rid of my £16 per month Sky Movies Subscription if it could replace the quality of service currently offered by Sky. Whilst the £4.99/£5.99 price point seems pretty convincing I was interested to see exactly what movies each of the services had available and which movies they were missing. In a recent article I read that Netflix had negotiated exclusivity deals on new movies with Lionsgate and MGM so it would be interesting to see how much the catalogues actually differed.

I don’t have time to do a full cross reference (although I was tempted to script something up to scrape them) so I’ve chosen a selection of more recent films that I know have been made available for online streaming and a selection of much older films. I have checked each film on Sky Go, Lovefilm and Netflx as well as on Blinkbox which is not a subscription service but I thought was a useful comparison for streaming library completeness. I have ignored Lovefilm offerings where they are available on DVD but not for instant streaming.

Lovefilm Netflix Sky Movies
Blinkbox (pay per view)
New/Recent Movies
Super 8 NO NO YES YES(£3.49)
Cowboys and Aliens NO NO NO * YES(£3.49)
Rise of Plant of the Apes NO NO NO * YES(£3.49)
Twliight NO NO YES YES (£1.99)
Harry Potter (8) YES(£3.49) NO NO * YES(£3.49)
The Hangover YES NO NO ** YES (£2.49)
The Hangover 2 YES(£3.49) NO NO ** YES(£3.49)
Four Lions YES NO NO** NO
Old Movies
Harry Potter (1) NO NO NO ** YES (£2.49)
Plant of the Apes (1968) YES NO NO NO
The Godfather NO NO YES YES(£1.99)
The 39 Steps NO YES NO YES (FREE)
Blair Witch Project NO YES NO NO
Con Air YES YES YES NO
Home Alone NO NO YES NO
Being John Malkovich YES YES YES NO
An American Werewolf in London NO YES NO YES(FREE)
Unlimited Streaming Subscription Price per month £9.99 £5.99 £16.00
N/A
* available on demand. £3.99 each
** were previously available

As we can see Netflix has pretty poor coverage at the moment compared to Lovefilm in terms of newer releases while Netflix seems to be better in terms of back catalogue. Out of the three subscription services Sky has the best coverage across the board, one limitation of the Sky Go service is the total number of movies that they have available at any one time which does change regularly, I have marked those movies which I know that I have watched on Sky Go on my Xbox previously. The Blinkbox pay per view model would work out expensive to watch 10-15 movies a month although they do have a large selection of free movies available – the quality of which does vary. Blinkbox also announced today that they were making streaming available for free on some titles if you buy the DVD in Tesco stores and provide your clubcard details (Tesco own Blinkbox).

I’m not planning on moving away from Sky yet, but it will be interesting to see how this pans out over the next few months and whether Netflix start to strengthen their catalogue with more studio deals. Meanwhile Lovefilm’s new owner, Amazon, will be sure to strengthen their streaming business in light of the Kindle Fire potentially getting a UK release later this year.

UPDATE – Blinkbox have got in touch to say that some of the movies that were marked as unavailable are actually available. I’ve updated those that are available to stream (some are only available to purchase)


My Datsusara Journey Begins

Datsusara (脱サラ) is the Japanese term used to describe leaving life as a salaryman and moving on to do something completely different. It’s not used in the case of moving for more money, out of boredom or due to termination of employment; it’s used to exemplify the act of pursuing a childhood dream or having a moment of inspiration that leads to a completely different career path.

As you may have worked out by now, this post is about me moving on from my current role as Product Manager at 7digital after almost 5 years there to start a company, Knowsis, with two friends . The reasons for me leaving are very much in line with the principles of Datsusara – starting a company is something I’ve wanted to do for a long time and a few things have happened recently which have made me realise that there is never going to be a right time to do these things and we need to get out there and Just Do It™ 

Oh, and if anyone has a spare meeting room or some desk space in Central London that we can borrow for the whole day on Wednesday this week it would be much appreciated.


Goodbye wallet – the phone will take it from here

Just announced right now at their NY offices,  Google’s NFC payments system – Google Wallet. Expected to launch in September this year, the service will allow Google Andoird users to pay using their phone at a number of retail stores.

Google Wallet

Google have partnered with Citibank, MasterCard, FirstData and Sprint for launch meaning that only Sprint users with a Citbank credit card will be able to use this from the start. This is being billed as an open platform and other payment providers and banks are being encouraged to get involved. Launch retail partners include Subway, Duane Reade, Footlocker and ToysRus starting in New York and moving San Francisco shortly afterwards.

One of the more interesting features is the tie in with Google Offers and rewards which has also just been announced. This allows you to redeem offers at the same time as paying and also collect loyalty rewards electronically rather than carrying a myriad of cards to be stamped manually.

Let’s see if Apple make their announcement at WWDC and whether its an “open” platform


Music Beta by Google

It’s official – just announced now at Google IO.

As per all the rumours it allows you to upload your music via a desktop manager, then play in the browser, tablets and mobiles. It’s also able to create 25 track playlists based on audio data and not artist similarity. They are allowing you to upload up to 20,000 songs. Launch will be invitation only for US users. Free during beta.

But, there’s no music for sale….yet.


It’s Google Music … oh, no it’s Movies

The press has been awash with details of the Google music service since late last night. Predictions have been around the introduction of an unlicensed cloud locker service that allows users to upload 50GB of music for free. Just announced now at Google IO was something that no one had predicted, Movies.

Google have just announced a Google Movies service within the marketplace in the next few weeks that allows users to link movie rentals to their Google account meaning that they can rent anywhere and watch anywhere. Works online, on tablets and on mobiles.

Users will be able to “pin” movies to a device to allow them to watch offline when out of streaming range. Rentals will cost about $2 and be standard streaming rental terms of 30 days (reduced to 24 hours after first viewing commences).


Cloud Wars Episode II – The Swedes Strike Back

Back in April I correctly predicted that Spotify would soon offer their mobile service to all customers, including free accounts, following their restrictions placed on non paying users; this is a huge step for them in the ongoing battle to get a full cloud service out there. To be clear, this update does not allow non-paying users to stream music that they do not own – only the tracks they have synced from their personal computer to their device. The desktop version of Spotify has been able to sync a users owned music over WiFi to a the mobile device for some time, a smart move on Spotify’s part to save streaming bandwidth costs. This new update now includes iPod syncing, something that has required the owners to always be tied to iTunes. Today’s announcement is really significant in terms of getting users into the Spotify ecosystem and in my opinion really puts Lorentzon & Ek’s company out there as the leaders in the cloud race.

Apple, Amazon and Spotify already account for 90% of  worldwide digital music revenues for and I think this move could prove to be a significant shift in Spotify’s publicised revenue problems if they can get customers to move away from Apple. This is all the more likely given their pending US launch and the hype around the lack of launch for the last yeat. Having worked on the original purchasing implementation, which was powered by 7digital, it was actually quite interesting to see how many purchases were made through it when users could still listen for free – it really points to users’ preference to own their music over the “rental” streaming model.

I think the most interesting outcome from all of this will be the backlash reaction from Apple. In the past, Apple have changed the way that music is stored on iPods during software updates to prevent third-party software from being able to move music onto the device. It wouldn’t take much for Apple to do this and I wouldn’t be surprised if the next software updates will cause problems for the engineers  over in Gothenberg & Stockholm.

The other question surrounds the iOS app and whether Apple will allow future updates to it or if they will even just remove it from the app store entirely. At 7digital we’ve struggled to get our iOS app through the submission process; an app that does considerably less to compete with iTunes and the iPod compared to that from Spotify. Unless Spotify has an agreement with Apple – which I find highly unlikely, I wouldn’t be surprised if the app store guidelines change to say if your app name contains the word ‘Spot’  and your logo is green then it can’t exist in the app store.

On top of this there is the well documented case of Apple wanting 30% of any app subscriptions which would effectively ruin Spotify’s model as they will not be able to operate on these terms. I guess the ball is now in Apple’s court.