Building an API for Mokocharlie’s Community photos #1

After a few years of Mokocharlie creating communities around images, it has become necessary to open up this data to partners who want to provide galleries of their own but don’t have the time to go out to curate those images themselves.

To this end the most logical path would be to develop an API that leverages the data we already have, I have chosen Laravel for this project. Over the next few weeks I will be blogging about the progress made on this project, from architecture to design decisions etc. The outline will be as follows:

  • Basic Architecture (How clients will connect to this service, authentication protocols, etc)
  • Data Formats/Structures
  • Content provision and distribution
  • Logging and Monitoring
  • Case Studies
  • Further work.

A standalone java/jar application from Scala sources using Proguard – Part 1

After about an hour of digging around and looking for a way to package a simulation I wrote in Scala for a coleague, I came across this process of doing it. I have reduced it down to a bash script which will be at the bottom of this post but I thought I should go through the steps a bit as the script makes certain assumptions.

The first assumption is that you have the necessary libraries and such to run it they are as follows:

  • Scala Library (That goes without saying if you’re going to be writing Scala code) I have a custom path for mine as I want to manually sync my libraries, compilers and so on with the latest versions. I have extracted my copy to ~/Bin/Scala-2.9.1.final this will be the path my bash script will assume, you must change this to your path if it’s different.
  • Proguard: I am using version 4.7, version 4.4 comes with ubuntu but, seeing as I like to keep it at the latest and I am impatient my proguard lives in ~/Bin/Proguard4.7
  • Java (This also goes without saying. You will probably need the JDK as well as the JRE so install them both) Proguard uses the java libraries extensively so you need that.

My simulation is in the following directory structure
engine
| – src
| — | —  Engine.scala
| – build-app.sh
| – scala-config.pro

The engine->src->Engine.scala is my scala application with the object Engine defined as my entry point this is important for the MANIFEST.MF file the script will be creating

Now the steps

  • First we can create a little folder to hold the output of the scala compiler so we create  a folder called bin inside of the engine folder itself so that the folder structure now looks like this
    engine
    | – src
    | — | — Engine.scala
    | – bin
    | – build-app.sh
    | – scala-config.pro
    by running

    mkdir bin

    then run the following while you’re inside engine which from now on I will refer to as root

    scalac -sourcepath src -d bin src/Engine.scala

    this will compile the Engine.scala code into java classes inside of the bin folder

  • This completes the scala bit, everything else following is run with Java so we need a MANIFEST.MF file to compile a jar
    echo "Main-Class: Engine" > MANIFEST.MF

    this will do it for you. Bear in mind you are still at the root level

    Now at this point we change directory to the bin folder where scalac has created our classes and compile our jar using the manifest we have just created so this should do it:

    cd bin
    jar -cfm ../engine-no-scala.jar ../MANIFEST.MF *.*

    You can obviously do this a different way by staying in the root and making sure your path is correct, personally I found this easier.

    This above code will create our engine called engine-no-scala.jar with the MANIFEST.MF file
    bear in mind at this point if you ran java -jar engine-no-scala.jar it may fail if your scala code required the scala library to run
    mine does so now I need to package this WITH the scala library into a complete self-contained java application
    now earlier I did say that my scala home was in a custom location so we need to keep that in mind for the next section

    PROGUARD

  • Not only does proguard include the required libraries and so on, it also obfuscates and discards any unused classes and packages this is very nifty for distribution as it cuts down by orders of magnitude the size of your application.When I did this without proguard my application was over 8 MB because the entire scala library was in my jar
    after I finished with proguard I am looking at an 8 KB application, and I think I can even get it smaller if I optimise my code a little bit morebut that’s for another time. Proguard however needs a configuration file to work properly in my case, I went to their site and downloaded a sample scala-config file (click on the Scala link) I just made the following changes to the vanilla one they provided

    -injars engine-no-scala.jar
    -injars /home/kwabena/Bin/scala-2.9.1.final/lib/scala-library.jar
    -outjars engine.jar
    -libraryjars <java.home>/lib/rt.jar
    
    -dontwarn scala.**
    -verbose

    I just changed the path to my jar file and my scala library and I also turned on verbosity (I like to know what’s going on) my build script now has the following lines

    cd ../
    echo "Adding Scala library, obfuscating..."
    #proguard
    proguard.sh @scala-config.pro

    after that you can remove any side effects from the script with

    rm -rf bin MANIFEST.MF engine-no-scala.jar

    bear in mind that we changed the our current workind directory to the root level
    after a few minutes of compiling, obfuscating, optimisation and shrinking you should have a nice jar to run by itself on a computer with ONLY the JRE installed

The complete script follows you can just copy this, save it into a file called “build-app.sh” as it is in my folder structure change the permissions to execute with

chmod +x build-app.sh

and run it

#!/bin/bash

echo "Creating required directories"
mkdir bin

echo "Compiling scala sources"
scalac -sourcepath src -d bin src/Engine.scala

echo "Creating Manifest"
#build manifest
echo "Main-Class: Engine" > MANIFEST.MF

echo "Building java archive from scala classes"
#create jar
cd bin
jar -cfm ../engine-no-scala.jar ../MANIFEST.MF *.*

cd ../

echo "Adding Scala library, obfuscating..."
#proguard
proguard.sh @scala-config.pro

#remove fluff
echo "Removing fluff..."
rm -rf bin MANIFEST.MF engine-no-scala.jar

I hope you find it as useful as I would have should this have been available earlier.

Web frameworks for Scala

The best way for me to learn a language generally is to build a project using that language. I have decided to start on this Scala journey and so far it’s been all theory and very little practice. In any case I have always been a web developer and that is where I would like to stay. So a web project in Scala is naturally the path I would want to go with. I kept stumbling across the Lift framework, I like Lift but it’s a bit tricky for me right now at this level and so I will perhaps visit it later.

I then came across the parsimonious Scalatra. I love it it’s light elegant and doesn’t require you to do too much to get started. I have not used it long enough to say that this is going to be my weapon of choice for any future developments but it’s a very strong candidate.

I only just now discovered the Play framework and on first glance it’s a brilliant looking MVC framework, honing in on my Zend experience with PHP. It feels like Zend, I have not actually started writing anything in it yet but I am going to attempt a full blown web project in it and see how that goes. Perhaps something for an EC2 instance.

Setting Up SBT on Ubuntu

So I have had to re-install my OS several times over the past month mainly due to my impatience (installing 11.10 beta and not having it working properly) and I have found that my SBT configurations generally evaporate with every re-installation so I wrote a little script to automate the whole process. A bit like my previous get Scala script so once again just copy and paste this into a text file, make it executable with chmod +x <filename> and run it

#!/bin/bash

cd ~

echo --------------------------------------------------------------------------------------------
echo - Getting SBT Launcher
echo --------------------------------------------------------------------------------------------
wget http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-tools.sbt/sbt-launch/0.11.0/sbt-launch.jar

printf 'java -Xmx512M -jar `dirname $0`/sbt-launch.jar "$@"' > sbt

chmod +x ./sbt

echo --------------------------------------------
echo - Move sbt and sbt-lauch to share
echo --------------------------------------------

sudo mv sbt /usr/share/sbt
sudo mv sbt /usr/share/sbt-launch.jar

echo --------------------------------------------------------------------------------------------
echo - symlinking sbt and sbt-lauch to /usr/bin/{sbt, sbt-lanch.jar}
echo --------------------------------------------------------------------------------------------
sudo ln -s /usr/share/sbt /usr/bin/sbt
sudo ln -s /usr/share/sbt-launch.jar /usr/bin/sbt-launch.jar

I suppose, when I get time I can make this accept parameters so you can specify the version of SBT you want and the file name you want to use etc but for now this should suffice…
comments welcome.

Installing Scala on Debian Based Systems (e.g Ubuntu)

Having followed instructions from here, which I thought was really easy, I put together a little bash script to do exactly that. And being a sucker for the latest I went with the 2.9.0.1 installation. I am honestly not sure why Ubuntu still has 2.7.7 in its repository. Ubuntu 11.10 (oneiric)  now has scala 2.9.x so you may want to use that instead of this script but if you want to keep up to date with the latest Scala release then:

Simply copy this and paste into a file, name it whatever you want, and make it executable with chmod +x

#!/bin/bash

cd ~

echo --------------------------------------------------------------------------------------------
echo - Getting Scala 2.9.0.1
echo --------------------------------------------------------------------------------------------
wget http://www.scala-lang.org/downloads/distrib/files/scala-2.9.0.1.tgz

echo --------------------------------------------------------------------------------------------
echo - Decompressing
echo --------------------------------------------------------------------------------------------
tar zxf scala-2.9.0.1.tgz

echo --------------------------------------------------------------------------------------------
echo - moving scala to /usr/share/scala
echo --------------------------------------------------------------------------------------------
sudo mv scala-2.9.0.1 /usr/share/scala

echo --------------------------------------------------------------------------------------------
echo - Symlinking executables
echo --------------------------------------------------------------------------------------------
sudo ln -s /usr/share/scala/bin/scala /usr/bin/scala
sudo ln -s /usr/share/scala/bin/scalac /usr/bin/scalac
sudo ln -s /usr/share/scala/bin/fsc /usr/bin/fsc
sudo ln -s /usr/share/scala/bin/sbaz /usr/bin/sbaz
sudo ln -s /usr/share/scala/bin/sbaz-setup /usr/bin/sbaz-setup
sudo ln -s /usr/share/scala/bin/scaladoc /usr/bin/scaladoc
sudo ln -s /usr/share/scala/bin/scalap /usr/bin/scalap