Simon Palmer’s blog

I recently posted about really simple file upload in my app and once I had that working I moved onto my next problem, which was in two parts, 1) making the persistence location flexible – and scalable – and 2) providing a public url to the images.

Taking the second problem first, I thought I was going to need to find some clever way of mapping a URI to a folder on the server and I investigated about 25 ways of doing that in tomcat, jetty, apache, and anywhere else I thought might work. There are solutions, but they normally mean an additional web server and subdomain, or some such configuration, to serve static content. There is also a grails plugin for serving static images. All of this felt like a lot of infrastructure and complexity, not to mention server load, which runs against my basic “do it elsewhere, not on the server” philosophy.

For the first issue (scalable persistence) I looked at where in the standard AWS AMI I was using I could place content and the choices are limited because I wanted sub-folders for each of my users and giving rights to processes on the server instance to create folders was also looking like a lot of non-standard config.

In steps Amazon S3. The solution is really very simple. Sign up for an AWS S3 account, create a bucket with a unique name for the app, add buckets for each user (they already have a UID to identify them) and back the asynchronous upload with a simple write to S3 from my controller. If you make the assets publicly readable then you can expose them through the regular s3.amazonaws.com url and you don’t need any other server infrastructure for your static file serving (*jumps up and down with glee*). Of course, there’s a Grails plugin for S3, so that’s where I started, and before I go much further I should say that I think that does the job very thoroughly and you should try that before you take my hack – at the very least download the code and study it closely, it is well written.

But… I didn’t want to involve my database and my file system in the upload process and I never anticipate needing a background process for doing the upload to the buckets, so it was breaking my self-imposed KISS rules.

So, “how hard could it be”? Here’s how.

First it is a good idea to grab the Jets3t library, which is brilliant. I found the install all a bit weird, but there is a jar file in the jets3t-XXX/applets folder – at least there was in the download I grabbed – and you basically need to stick that in your lib folder under your Grails app. If you are using the STS Eclipse plug-in you’ll need to add the jar as an external library to your project properties to have it compile, run, work, etc.

It’s a good idea to write a Grails Service for this, so you start by doing that

grails create-service AmazonS3

And here’s the code for the service… this is admittedly very simplistic and it needs some error handling, but you’ll figure that out…

import org.jets3t.service.S3Service
import org.jets3t.service.security.AWSCredentials
import org.jets3t.service.impl.rest.httpclient.RestS3Service
import org.jets3t.service.model.S3Bucket
import org.jets3t.service.model.S3Object
import org.jets3t.service.acl.AccessControlList

class AmazonS3Service {
	
	static String accessKey="PUTYOUROWNONEINHERE"
	static String secretKey="PUTYOUROWNONEINHERE"
	static RestS3Service s3 = new RestS3Service(new AWSCredentials(accessKey, secretKey))
        boolean transactional = false
	String rootBucketPath="yourbucket/subbucket/whatever/"
	String defaultBucketLocation=S3Bucket.LOCATION_EUROPE
	
	Map mimeExtensionMap = [
			"png" : "image/png",
			"jpg": "image/jpeg",
			"gif": "image/gif",
			"tiff": "image/tiff",
			"pdf": "application/pdf",
			"mpeg": "video/mpeg",
			"mp4": "video/mp4",
			"mov": "video/quicktime",
			"wmv": "video/x-ms-wmv",
			"html": "text/html",
			"xml": "text/xml",
			"mp3": "audio/mpeg",
			"flv": "application/octet-stream"
	]
	
	S3Bucket makeBucket(uid)
	{
		S3Bucket bucket = s3.getOrCreateBucket((rootBucketPath + uid), defaultBucketLocation)
		bucket.setAcl AccessControlList.REST_CANNED_PUBLIC_READ
		return bucket
	}
	
	void put(inputstream, name, uid, ext, length)
	{
		if (mimeExtensionMap.containsKey(ext.toLowerCase()))
		{
			String mime = mimeExtensionMap[ext.toLowerCase()]; 
			S3Bucket bucket = makeBucket(uid)
			S3Object up = new S3Object()
			up.setAcl AccessControlList.REST_CANNED_PUBLIC_READ
			up.setContentLength length
			up.setContentType mime
			up.setDataInputStream inputstream
			up.setKey name
			up.setBucketName bucket.getName()
			s3.putObject bucket, up
		}
	}
	void putXML(text, name, uid)
	{
		String mime = mimeExtensionMap["xml"]; 
		S3Bucket bucket = makeBucket(uid)
		S3Object up = new S3Object(bucket, name, text)
		up.setAcl AccessControlList.REST_CANNED_PUBLIC_READ
		up.setContentLength text.length()
		up.setContentType mime
		s3.putObject bucket, up
	}
}

This gives you a single API entry point (well, two, put and putXML) and will give you read-only assets on your S3 bucket that follow this sort of path…

yourbucket/subbucket/whatever/an-identifier-of-your-coice/image1.jpg
yourbucket/subbucket/whatever/an-identifier-of-your-coice/image2.jpg
yourbucket/subbucket/whatever/an-identifier-of-your-coice/image3.jpg
yourbucket/subbucket/whatever/82757c2a-cfa9-49bf-89fa-9efdaf9bf418/image1.jpg
yourbucket/subbucket/whatever/82757c2a-cfa9-49bf-89fa-9efdaf9bf418/image2.jpg
yourbucket/subbucket/whatever/82757c2a-cfa9-49bf-89fa-9efdaf9bf418/image3.jpg

These can then be got at via urls which reference these buckets via the normal s3 url…

To call this from a controller used to manage the asynchronous file upload you just need to know about one thing, which is how to get from a Spring MultipartFile to something which the JetS3t API can consume.

def amazonS3Service;
def uploadfile = {

	if(request.method == 'POST') 
	{

		Iterator itr = request.getFileNames();

		String vloc = "";
		while(itr.hasNext()) 
		{
			MultipartFile mpFile = request.getFile(itr.next());
			if (!mpFile.isEmpty()) 
			{
				// success
				String _file = mpFile.getOriginalFilename().replace(" ", "_");
				vloc += h.uid + "/" + _file;
				String ext = _file.substring(_file.lastIndexOf(".")+1);
				amazonS3Service.put (mpFile.getInputStream(), _file, h.uid, ext, mpFile.getSize())

			}
		}

		render vloc
	}
}

Basically mpFile.getInputStream() will give something which can be sent directly to S3 without being persisted to the local file system – which is what pretty much every other example I came across does. A couple of things to point out at this point, first is that h.uid is not visible in this code, it is just a unique value I ascribe to each user in my system and you should choose your own. It forms the rightmost bucket name in my S3 key structure. Second is that I use the file name as the key and I modify it to replace the spaces with underscores to avoid any nasty problems with filenames.

Eh voila, you have static image persistence to an almost infinitely scalable server infrastructure (at reasonable cost) and serving via the cloud. Normal caveats apply, and this is far from perfect code, but it works and I hope it helps you get started.

OK, after A MONTH of trying to figure out how to get the Grails mail plugin to talk to authsmtp, and *utterly* failing, in a puddle of tears this afternoon at my 1,745th attempt at finding out anything about this on the web, I went into deep hack mode and cracked it in 15 minutes flat.

The answer… write my own f*!%$ing email service.

If you can bear to read to the end I have a bigger Grails lesson, but here’s the solution for any poor person who may be following me down this path.

First off, why AuthSMTP.com? Well, in a production envronment, for a real world application which is going to be generating email in response to user actions (such as notifications), then your gmail account will not cut it and your local ISP will almost certainly block the IP address of your server – especially if you deploy into the cloud at either Google or AWS. That means you have to have a real SMTP host to take care of the relaying and send. AuthSMTP is one of many out there, but their pricing policy is reasonable and they are production strength. I have used them for personal accounts for POP3 and they have been good, although their technical support demonstrated the worst sort of brainless responses possible and was deeply disappointing.

So, following the mantra which got me here, of “how hard can it be?”, here’s the answer…

First off go to your grails prompt and create a service with a name of your choice using

grails create-service Mymail

Next in the scaffolded code, put the following – I decided that I would just use old fashioned Java since I sort of felt I know what I was doing and I am still not comfortable enough with Groovy that I would risk chasing my own stupid errors for another month.

import java.util.Properties;
import javax.mail.internet.*;
import javax.mail.*;

class MymailService 
{
    boolean transactional = false

    public boolean sendMessage(String to, String msgSubject, String msgText) 
    {
		String host = "mail.authsmtp.com";
		String username = "ac99999"; // your authsmtp username
		String password = "xxxxxxxxxx" // your authsmtp password
		String from = "no-reply@yourdomain.com";
		
		Properties props = System.getProperties();
		props.put("mail.smtp.host", host);
		props.put("mail.smtp.user", username);
		props.put("mail.smtp.password", password);
		props.put("mail.smtp.port", "2525"); // thish is the port recommended by authsmtp
		props.put("mail.smtp.auth", "true");
				
		Session session = Session.getDefaultInstance(props, null);
		MimeMessage message = new MimeMessage(session);
		message.setFrom(new InternetAddress(from));
		
		InternetAddress to_address = new InternetAddress(to);
		message.addRecipient(Message.RecipientType.TO, to_address);
		
		message.setSubject(msgSubject);
		message.setText(msgText);
		Transport transport = session.getTransport("smtp");
		transport.connect(host, username, password);
		transport.sendMessage(message, message.getAllRecipients());
		transport.close();
		return true;
	}
}

It’s worst fault is that the thread waits for a response form the server and I confess the error handling could be added improved, and OK, you can’t attach files and you can’t send to multiple recipients, although adding that would be really simple, and you can’t do back-flips and set properties on the fly and blah blah. BUT… you can send a basic message from one email address to another, which I am betting is what 99% of automatically generated email does.

Now for the lesson, and look away if you have already been baptised by a Grails evangelist. The conclusion I am quickly coming to is that all the benefit I got on the swings of a rapid start with Grails I have more than lost on the ugly, nasty, dizzying, vomity, downright dangerous roundabout of debugging it when something doesn’t work.

Even having gone as far as downloading and installing the STS Eclipse IDE – which remains the slowest piece of software on my desktop by a factor of 5 – and figuring out how to actually debug a session, stepping through the code I found myself in a completely impenetrable marass of codeless call stacks as deep as the mariana trench. It remains about as bad a development experience as I have had in the 20-some years I’ve been doing it. The only worse thing is the silent fail of a Javascript library as the browser refuses it. It’s about time the browser was replaced with something better, but that’s a whole other rant.

I spent ages trying to decide which was the best way for me to implement file upload in my grails app. What I wanted was the ability for a user to select an image from their local machine and for it to be uploaded into a tag on my page. The biggest issue I faced was that the upload button, and in fact the whole DIV that contained the image and other stuff, is generated at runtime. The other mandatory condition was that it had to happen asynchronously and without re-loading the page.

I tried out 3 or 4 solutions, including several grails plug-ins (google for “grails file upload” and you’ll find them), several ajax javascript solutions, which I have now lost, and a couple of Flash uploader implementations. The trouble with all of them was a combination of complexity and size. In the end I found a really simple alternative which seems to work just fine and builds on adding the Grails UI plugin into my application.

My solution was in two parts, first was to give myself the ability to have a modal dialog box on my page and second was to utilise the Yahoo GUI library to make an asynchronous connection back to the server.

The modal dialog came directly from the Grails UI plugin, which has a very rich set of excellent features and I recommend it. Installing that plugin gave me a dialog box, which was as simple as including a resource reference and a GSP tag in a hidden div on my page…

        <gui:resources components="['dialog']"/>
        <div>
            <gui:dialog
                id="dlgFileUpload"
                title="Upload Image"
                draggable="false"
                modal="true"
                buttons="[[text:'Upload', handler: 'onUploadButtonClick', isDefault: true],[text:'Cancel', handler: 'function() {this.cancel();}', isDefault: false]]">
                <form action="uploadfile" enctype="multipart/form-data" method="post" id="uploadForm">
                <input type="file" name="testFile"/>
                </form>
            </gui:dialog>
        </div>

The important thing to notice in here is that the dialog box contains a form which defines the multipart enctype, the method as POST. The action refers to the code in my Grails controller which handled persistence of the file. The gui:resources tag is exploded by Grails into a set of script includes for the Yahoo UI libraries. This is important because the YUI library is very large and there are many interdependencies, and this single tag hides all that, which is a major time saver.

The other thing that the Grails UI plugin gave me was the whole of the YUI library, so the next bit was to write some code to handle the file upload itself. Rather than lots of fancy components I realised that I could just use the YAHOO.util.Connect object to make an asynchronous call back to the server, in much that same way that I might otherwise have used an XMLHTTPRequest object in Javascript. Because the whole YUI library is included with the plugin I already had the code. To get at the Connect object I had to include the appropriate script because the gui:resources tag seemed not to add it as a dependence…

        <script type="text/javascript" src="/js/yui/2.7.0/utilities/utilities.js" ></script>

From that point it was a simple matter of writing a function to handle the click on the upload button in my dialog which made the request back to the server, and handled the response…

        function onUploadButtonClick(e)
        {
            var uploadHandler =
            {
                upload: function(o)
                {
                    refreshActiveImage(o.responseText);
                    setDirty();
                    hideWaitCursor();
                }
            };
            showWaitCursor();
            //the second argument of setForm is crucial,
            //which tells Connection Manager this is an file upload form
            YAHOO.util.Connect.setForm('uploadForm', true);
            YAHOO.util.Connect.asyncRequest('POST', 'uploadfile', uploadHandler);
            this.cancel();
        }

The important thinhg to notice in here is that the setForm call requires the second argument to be set to true so it recognises the form settings for the POST. Otherwise, the two lines of code invoking YAHOO connect do everything you need to send a file back to the server.

My server side code processes the multipart file upload and persists the file into Amazon S3 (see my other post about that). I then render the file as a virtual location relative to the current page back into my page so I can simply update the src property of the right IMG tag on my page – that’s what refreshActiveImage(o.responseText); does.

Eh voila! A common plug-in, a great utility library, some really basic Javascript and a form and I have the simplest file upload I could imagine.

The grails-ui plugin has a very rich feature set, being based on the Yahoo’s YUI library.  I have been trying to avoid it, but eventually I have caved and I need a modal dialog box for my web page.  I say caved because I think that it is not an intuitive element of a web application and there are enough cross-browser coding issues to make the introduction of a modal window on a web page a bit of a nightmare.  Of course that’s all taken care of by libraries such as YUI, but they come at a price; there’s a lot of code to download.

Anyhow, I caved and decided I would install the ui plugin to my project and use the dialog box.  It still feels like a bit of a sledgehammer to crack my little nut, but it works and it is not as buggy as the modal DIV I started writing myself.

The trouble is that out of the box you get none of the default styling from the YUI skins.  According to the grails documentation it should be sufficient to enclose your dialog in a tag and set the class of that tag to refer to the YUI skins…

    <body>
        <div class="yui-skin-sam">
            <gui:dialog blah blah></gui:dialog>
        </div>
    </body>

However, this doesn’t work. And neither does this…

    <body class="yui-skin-sam">
        <div>
            <gui:dialog blah blah></gui:dialog>
        </div>
    </body>

In fact the only way I could get this to work was by adding the “yui-skin-sam” class reference to the body tag in layout/main.gsp.

But that comes with a set of problems of its own, not least of which is that styling the body tag in the main layout means that every generated page gets that styling. If, like me, you had gone to some lengths to define your own styling, that is a real pain because the chance of you not having weird clashes in styles – and yui-skin-sam not overriding them – is almost nil for any semi-serious web app.

So, some words to the wise:

• If you are starting out on your new shiny Grails app, and you want to style it yourself, imply a namespace in your style names when you are making them by prefixing them all with a code.
• If you think you might want some exoteric UI feature from the grails UI plug-in – and you almost certainly will – then install it early and declare the style class on the main layout body tag from the start. That way you won’t have a lot of re-factoring of style code to do later.