The answer… write my own f*!%$ing email service.

If you can bear to read to the end I have a bigger Grails lesson, but here’s the solution for any poor person who may be following me down this path.

First off, why AuthSMTP.com? Well, in a production envronment, for a real world application which is going to be generating email in response to user actions (such as notifications), then your gmail account will not cut it and your local ISP will almost certainly block the IP address of your server – especially if you deploy into the cloud at either Google or AWS. That means you have to have a real SMTP host to take care of the relaying and send. AuthSMTP is one of many out there, but their pricing policy is reasonable and they are production strength. I have used them for personal accounts for POP3 and they have been good, although their technical support demonstrated the worst sort of brainless responses possible and was deeply disappointing.

So, following the mantra which got me here, of “how hard can it be?”, here’s the answer…

First off go to your grails prompt and create a service with a name of your choice using

grails create-service Mymail

Next in the scaffolded code, put the following – I decided that I would just use old fashioned Java since I sort of felt I know what I was doing and I am still not comfortable enough with Groovy that I would risk chasing my own stupid errors for another month.

import java.util.Properties;
import javax.mail.internet.*;
import javax.mail.*;

class MymailService
{
    boolean transactional = false

    public boolean sendMessage(String to, String msgSubject, String msgText)
    {
		String host = "mail.authsmtp.com";
		String username = "ac99999"; // your authsmtp username
		String password = "xxxxxxxxxx" // your authsmtp password
		String from = "no-reply@yourdomain.com";

		Properties props = System.getProperties();
		props.put("mail.smtp.host", host);
		props.put("mail.smtp.user", username);
		props.put("mail.smtp.password", password);
		props.put("mail.smtp.port", "2525"); // thish is the port recommended by authsmtp
		props.put("mail.smtp.auth", "true");

		Session session = Session.getDefaultInstance(props, null);
		MimeMessage message = new MimeMessage(session);
		message.setFrom(new InternetAddress(from));

		InternetAddress to_address = new InternetAddress(to);
		message.addRecipient(Message.RecipientType.TO, to_address);

		message.setSubject(msgSubject);
		message.setText(msgText);
		Transport transport = session.getTransport("smtp");
		transport.connect(host, username, password);
		transport.sendMessage(message, message.getAllRecipients());
		transport.close();
		return true;
	}
}

It’s worst fault is that the thread waits for a response form the server and I confess the error handling could be added improved, and OK, you can’t attach files and you can’t send to multiple recipients, although adding that would be really simple, and you can’t do back-flips and set properties on the fly and blah blah. BUT… you can send a basic message from one email address to another, which I am betting is what 99% of automatically generated email does.

Now for the lesson, and look away if you have already been baptised by a Grails evangelist. The conclusion I am quickly coming to is that all the benefit I got on the swings of a rapid start with Grails I have more than lost on the ugly, nasty, dizzying, vomity, downright dangerous roundabout of debugging it when something doesn’t work.

Even having gone as far as downloading and installing the STS Eclipse IDE – which remains the slowest piece of software on my desktop by a factor of 5 – and figuring out how to actually debug a session, stepping through the code I found myself in a completely impenetrable marass of codeless call stacks as deep as the mariana trench. It remains about as bad a development experience as I have had in the 20-some years I’ve been doing it. The only worse thing is the silent fail of a Javascript library as the browser refuses it. It’s about time the browser was replaced with something better, but that’s a whole other rant.

I tried out 3 or 4 solutions, including several grails plug-ins (google for “grails file upload” and you’ll find them), several ajax javascript solutions, which I have now lost, and a couple of Flash uploader implementations. The trouble with all of them was a combination of complexity and size. In the end I found a really simple alternative which seems to work just fine and builds on adding the Grails UI plugin into my application.

My solution was in two parts, first was to give myself the ability to have a modal dialog box on my page and second was to utilise the Yahoo GUI library to make an asynchronous connection back to the server.

The modal dialog came directly from the Grails UI plugin, which has a very rich set of excellent features and I recommend it. Installing that plugin gave me a dialog box, which was as simple as including a resource reference and a GSP tag in a hidden div on my page…

        <gui:resources components="['dialog']"/>
        <div>
            <gui:dialog
                id="dlgFileUpload"
                title="Upload Image"
                draggable="false"
                modal="true"
                buttons="[[text:'Upload', handler: 'onUploadButtonClick', isDefault: true],[text:'Cancel', handler: 'function() {this.cancel();}', isDefault: false]]">
                <form action="uploadfile" enctype="multipart/form-data" method="post" id="uploadForm">
                <input type="file" name="testFile"/>
                </form>
            </gui:dialog>
        </div>

The important thing to notice in here is that the dialog box contains a form which defines the multipart enctype, the method as POST. The action refers to the code in my Grails controller which handled persistence of the file. The gui:resources tag is exploded by Grails into a set of script includes for the Yahoo UI libraries. This is important because the YUI library is very large and there are many interdependencies, and this single tag hides all that, which is a major time saver.

The other thing that the Grails UI plugin gave me was the whole of the YUI library, so the next bit was to write some code to handle the file upload itself. Rather than lots of fancy components I realised that I could just use the YAHOO.util.Connect object to make an asynchronous call back to the server, in much that same way that I might otherwise have used an XMLHTTPRequest object in Javascript. Because the whole YUI library is included with the plugin I already had the code. To get at the Connect object I had to include the appropriate script because the gui:resources tag seemed not to add it as a dependence…

        <script type="text/javascript" src="/js/yui/2.7.0/utilities/utilities.js" ></script>

From that point it was a simple matter of writing a function to handle the click on the upload button in my dialog which made the request back to the server, and handled the response…

        function onUploadButtonClick(e)
        {
            var uploadHandler =
            {
                upload: function(o)
                {
                    refreshActiveImage(o.responseText);
                    setDirty();
                    hideWaitCursor();
                }
            };
            showWaitCursor();
            //the second argument of setForm is crucial,
            //which tells Connection Manager this is an file upload form
            YAHOO.util.Connect.setForm('uploadForm', true);
            YAHOO.util.Connect.asyncRequest('POST', 'uploadfile', uploadHandler);
            this.cancel();
        }

The important thinhg to notice in here is that the setForm call requires the second argument to be set to true so it recognises the form settings for the POST. Otherwise, the two lines of code invoking YAHOO connect do everything you need to send a file back to the server.

My server side code processes the multipart file upload and persists the file into a known location which suits my application. I then render the file as a virtual location relative to the current page back into my page so I can simply update the src property of the right IMG tag on my page – that’s what refreshActiveImage(o.responseText); does.

Eh voila! A common plug-in, a great utility library, some really basic Javascript and a form and I have the simplest file upload I could imagine.

I’m a bit disappointed that there is not an HTML language definition, but that’s OK, I write XHTML in any case so XML works just fine.

function makeRooms(edit)
{
	var t = getTag("roomsBody", "div");
	if (t != null)
	{
		var div = "<div id='rooms'>";

		//if (rooms === undefined) rooms = [];
		for (var i = 0; i &lt; rooms.length; i++)
		{
			if (edit)
			{
				div += makeRoomDivEdit(rooms[i], imgpath, i);
			}
			else
			{
				div += makeRoomDiv(rooms[i], imgpath);
			}
		}
		div += &quot;</div>";
		if (edit) div += "<a style='padding:5px;' class='instruction' href='addRoom()'>Add Room</a>";
		t.innerHTML = div;
	}
}

Anyhow, I caved and decided I would install the ui plugin to my project and use the dialog box.  It still feels like a bit of a sledgehammer to crack my little nut, but it works and it is not as buggy as the modal DIV I started writing myself.

The trouble is that out of the box you get none of the default styling from the YUI skins.  According to the grails documentation it should be sufficient to enclose your dialog in a tag and set the class of that tag to refer to the YUI skins…

    <body>
        <div class="yui-skin-sam">
            <gui:dialog blah blah></gui:dialog>
        </div>
    </body>

However, this doesn’t work. And neither does this…

    <body class="yui-skin-sam">
        <div>
            <gui:dialog blah blah></gui:dialog>
        </div>
    </body>

In fact the only way I could get this to work was by adding the “yui-skin-sam” class reference to the body tag in layout/main.gsp.

But that comes with a set of problems of its own, not least of which is that styling the body tag in the main layout means that every generated page gets that styling. If, like me, you had gone to some lengths to define your own styling, that is a real pain because the chance of you not having weird clashes in styles – and yui-skin-sam not overriding them – is almost nil for any semi-serious web app.

So, some words to the wise:

• If you are starting out on your new shiny Grails app, and you want to style it yourself, imply a namespace in your style names when you are making them by prefixing them all with a code.
• If you think you might want some exoteric UI feature from the grails UI plug-in – and you almost certainly will – then install it early and declare the style class on the main layout body tag from the start. That way you won’t have a lot of re-factoring of style code to do later.

Here it is for anyone else who might find it useful. Because WordPress don’t allow upload of the .syn file extension I have renamed it gsp.pdf. Right-click on the link save it to your hard disk and rename to .syn file to have it work. It needs to go into your samples folder underneath your Textpad install location.

I have been building it as I have gone along, so it is likely to be incomplete. However, it has most of the commonly used tags so it works for me. I based it on an existing HTML template which also has reasonably support for Javascript, so it is a pretty good general Grails View syntax hilighter.

This seems particularly important for Grails since the IDE’s available are pretty bad (or expensive) and it sort of works better to have a favourite text editor in the mix as your primary editor. I have also added some of the common grails commands to Textpad and it is starting to function as a pretty good alternative to the STS Eclipse plug-in which Spring provide – which isn’t worth it in my opinion.

I have another one I am building for Groovy which I’ll post just as soon as I get it in a state where it does what I feel is the minimum to get by.

I got some great responses, although none of them led me to a real answer. I ended up logging a bug at Adobe about the issue. Lord knows what will happen to that and I still have a broken Flex socket server, but the experience was good enough that I have overcome my objections and re-joined the Stackoverflow community.

It is easy to spend (read waste) too much time there chasing rep points, and generally in response to questions that the posters ought really to be able to figure out for themselves. And my criticisms remain the same as they were, I have just got over them. Mostly.

There is a definite increase in the sorts of questions which start “I downloaded this code from the internet”, followed by “I have no idea what this bit does, but I changed it and now it doesn’t work”, followed by “can you tell me why”. Basically I think this is lazy and dangerous and I hope these ar students rather than professional programmers. Stackoverflow encourages this sort of behaviour because someone – probably in pursuit of rep points – will normally take the time to post an answer, sometimes a very good answer.

I’m not bothered about the rep any more, although it remains a nice idea and a very good way to get people hookes for long enough to see the real value underneath it. The self-regulation seems to be working reasonably well too and the excesses of the early days are all but gone and there are fewer people trying to be the SO police than there were – although that remains a problem.

So back on and getting and adding value. SO is now part of my technical infrastructure, which I suppose means it is succeeding.

If you think about it a sense of humour is a very sophisticated layer of communication which sits above mastering the mechanics of speaking, the detail of language and the meaning of words.

I was amazed. 6 months ago he was in nappies and barely able to talk. Now we are playing word games and he is deciding what is funny. That sort of transformation must belie a huge change inside his head.

I think I subscribe to the Noam Chomsky, Stephen Pinker view of the world that we have an innate ability to master language, and I suspect that the real key to it all is the development of memory. If you consider everything that is necessary to be able to communicate, vocabulary, syntax, grammar etc. it is all predicated on being able to remember things. I suspect that the transformation I have witnessed in Jack is the development of his memory.

He suddenly has an amazing vocabulary. Yesterday as he was struggling with unzipping his coat and he said quite clearly “absolutely ridiculous”. Later that day he was sitting on the loo and he looked at me and said “concentrate”. These are very sophisticated ideas and there is a complex context necessary for them to be applied appropriately, which they were.

Memory development is supported by the other amazing things he has started to do which is remember directions and locations. If we drive out from our home towards the North of London he can now tell whether we are going to the shopping mall, the chinese grocery store or Nanny’s house by the route we take right at the last moment.

So I’m obviously a proud dad – he’s only 2 1/2 – but I am also an amazed observer of the rate of development of the human brain.

I was talking to a software friend based in California at the start of this week. She has people developing software in pretty much all the major centres in the world. I asked whether she either used, or intended to use Google Wave and she didn’t. Not because it may not be the best thing since sliced bread, but primarily because she already had processes and systems in place which worked perfectly well and which were embedded into her teams.

Of course that presumes that she had concluded what it was that Google Wave does, and when I asked she waffled a bit but basically said “collaboration”. That’s a big space, but it seems to be where most people place Google Wave when thinking about mapping it into their world.

It is generally a big ask to have all your teams switch working practices, especially core traits like how they communicate with peers and document that communication. Contemplating such a switch generally requires a backdrop of discontinuous change or profound dysfunction, to make it worthwhile, and I guess Google may claim ground as agents of discontinuous change with Wave. I’m not convinced though.

I’ll keep my account of course, but until I find a need it remains a Faberge egg. A fascinating, if useless, enigma.

Ouch, scrollbars don’t work properly in Firefox 3.0.15, pretty fundamental and visible feature and surprised they don’t have a commodity widget like that working. Even more surprisingly, when I try to run it in IE to see if it works there I’m asked to install the Chrome Frame plug-in. Small things perhaps, but shop-window problems which make it feel pre-beta.

Layout is simple and clean and in the Google mould. Bits of my gmail profile, picture and contacts for instance, have made it across. It is obvious how to create a new Wave and add people to it and I have started a couple with the contacts I have which appear to have wave accounts. I have a simon.palmer@googlewave.com address, which mirrors my gmail address, so I try and send an email to it to see what happens. So far nothing has happened. It didn’t bounce and nothing arrived.

Weirdly I was expecting to see my gmail inbox in here. I don’t know why really, I was just expecting it. And that leads me to the obvious question. If this isn’t email, then what is it? What can I wave that I can’t email?

I have a couple of waves already set up, several welcome ones from Dr Wave (c’mon, are we really patients? or does it need a PhD to figure it out) containing video clip tutorials and explanations. After watching a few the idea seems to be a cross between multi-party email and chat, with sharing of documents, pictures etc. and the whole in a chronological sequence of evolution of the interaction. You can (apparently, and eventually) use it for collaboration on documents, creating events and the like. I’m beginning to get the picture.

Trying to map it into the MS Office world, I think it sits somewhere between Sharepoint, BPM, MSN, Word and Outlook. It is certainly a different take. I think my main issue at the moment is that the benefit is not clearly articulated. How is it better than working the old way, because let’s face it, most of us do most of these things today, and there is overlap between them already?

It may make corporate buyers look again before investing in collaboration infrastructure like Sharepoint, although it is the smaller players in that market, and the open source projects that are likely to feel the ripples from the wave, not Microsoft.

So those are my first immediate impressions and I’m going to hedge my bets at this point. There may yet be something very illuminating underneath it which I am yet to see. Maybe, as we all take to smaller devices, there is some new way of working which wave is anticipating.

However, it has more than a little of the segway about it, and unfortunately a bit of the Sinclair C5 too. Innovative and clever? Undoubtedly. Challenging to established norms? Certainly. Paradigm-shifting? Hmm, maybe not. I have the feeling that we are all still going to be emailing in 5 years time. Google Wave will almost certainly find a niche in some CaliTechy communities, as has the segway, but I’m still walking and driving rather than segwaying, and I can’t see my wavebox displacing my inbox. Yet at least.

I say dialogue, but the reality is that communication happening in LinkedIn is not a dialogue at all. At best it is a didactic monologue conducted in a forum where everyone is shouting. At worst it is a set of very thinly veiled adverts for people or companies who clearly think they are operating at the bleeding edge of guerilla marketing. And unfortunately the worst is also the most common.

After years of patience with it, repeatedly giving it another chance, and following the updates where my first circle of contacts are connecting with, well, my first circle of contacts (with whom I know they already have alternative connections) I have finally had the veil of web 2.0 drawn back from my eyes and asked myself, objectively, what’s the point?

I’ve done some evaulation of the number of outbound vs. inbound contacts I have had in the last 4 years, and the extent of those which have resulted in a contacts for which I have no other contact than LinkedIn, or would otherwise not have made contact with those people. My measure of its value as a networking tool is the extent to which it has extended my network, rather than just confirmed it in another medium. I have subtracted anyone who I found elsewhere and thereafter connected with on LInkedIn.

It makes dismal reading.

Of my 70 contacts…
21 are friends
26 are ex-colleagues
14 I met elsewhere
8 are current colleagues

and

1 I found on LinkedIn (I have not heard from him since the initial contact in 2007)

I have been a member of 23 groups, all but one of which I have left because they were thinly veiled self-publicisation for the founder and members. I’m about to leave the remaining one for the same reason, just as soon as I stop ranting.

Maybe my version of networking is out of touch with what everyone else calls networking. Maybe I’m just using it as a version of Facebook without the silly games and photos. But as far as I can tell, so is everyone else. If I look at the second circle beyond mine, which is indeed a large number of people (9,500) they have a similar profile of connections to my connections, namely friends and colleagues.

The groups seem to be populated by people who believe that if they word their advertisement as a question the rest of us will be completely fooled by it and will follow the link to their web site. How stupid do you think we are, and how far from networking can you get? For an answer please go to my web site http://www.WhatSortOfMoronsAskQuestionsLikeThatAndThinkItIsClever.com

The questions section is no better. The questions are asked by people who clearly think they have the answer already and conveniently provide you with a link to their web site to supply it. They are then answered by other people who think they too have the perfect solution which, surprise surprise, you can find by clicking on the link to their web site.

In what possible way can this be construed as networking? It’s not even advertising or marketing or PR. It’s just aimless posting. If you want to do that why not have a blog?

Lastly to the jobs. I have optimistically bought the notion that people who advertise open positions on LinkedIn are genuinely interested in people responding who have the right skills and experience. It may be true, but I am yet to see evidence to support the assertion.

I wanted to test out the efficacy of LinkedIn as a tool to help in a job search. I am genuinely in a position where I would benefit from having a network of people look at my CV and I am in the job seeking market. I brushed up my CV, updated my profile, cleaned up my blog and links and then set about applying for positions for which I thought I would be a suitable candidate, and for which I am eminently qualified.

So far I have I applied for 46 positions. I have done research into each company and written a cover letter with each application, and have a named individual to whom I am writing. I’m an experienced guy who knows how to write and knows a bit about business, especially the business I have been in for the last 20-something years.

I have had ZERO responses. Not even a single courtesy note saying “no thank you”. The closest I got was an out of office.

Maybe I’ve got an unappealing profile and I am applying for things that are inappropriate given my background. Maybe my cover letters are not good enough and my CV doesn’t make me stand out. Maybe the prevailing economic conditions mean that the job market is bad. Maybe.

But maybe, just maybe, LinkedIn doesn’t work. Maybe people use it as either a last resort or as a way of pretending they are doing something when they are not, or as a lazy way of avoiding making a real network. Maybe there are some mavens for whom their 1,435 direct contacts are the source of valuable business, but maybe peoples’ contacts are little more than their personal phone book or contact list tapped into a web site. Maybe, like mine and pretty much everyone else I know and am connected to, they are little more than a online map charting our recent business past.

For me the promise has definitely not delivered. I have had absolutely no return for the investment of time and effort I have made in attempting to conjure a network on LinkedIn. I think I have used it in a fairly typical way and at a fairly typical level of activity. I think my job search was probably characteristic of what someone might do if they were serious about using it as a tool for job hunting. If it can’t deliver benefit in that domain then what exactly is the point?

In spite of this I will keep my profile. The somewhat un-dis-provable argument of it not doing any harm will mean that I may as well. I want it to work, I really do, however it doesn’t, and I doubt I am alone in being disenchanted with the reality.

LinkedIn. Sigh.