Building an assessment app in two days without rushing (hopefully)

At the University of Queensland, I'm helping to teach a software engineering studio course ("Design Computing Studio 2") where students are all collaborating, in groups, on 170-person project. It's a somewhat unique course, as it has a very large amount of collaboration between groups, and we've presented papers on it at two well-regarded international conferences: ICSE and ITiCSE.

This year, my colleague Jim has introduced peer critiques into the course — something that was missing in previous iterations. Groups will be asked to present their plans, and students will be asked to critique them. The groups in turn get to rate whether or not the critique feedback was useful.

I've also been writing the interactive teaching software "Impressory" for use on the course, and have a number of other software projects on the go. So, I wrote handy to make it easy and concise to write interactive, functional, webapps (generally, with Angular.js and Play).

It's time to find out how easy it really is.

The challenge

This week, I'm going to write the critique tool, as the first piece of functionality for an open source assesement app. And I'm going to blog it as I go — each post describing what code I've written, and how and why it does what it does.

This isn't going to be a hackathon — I'm not planning on staying up nights. In fact the next thing I have to do after posting this is head out the door to a meet-up about something completely different. And then I've got another meeting tomorrow morning. (So you might not see the next post until tomorrow afternoon.)

But the next lecture for the course is at 8am on Thursday, and I'd like to have it done by then so I can show it in class. Right now, it's 6pm on Monday. So that leaves Tuesday and Wednesday to get it done (in between other things like writing lecture content).

Let's see how we go.

For the students on the course

On the course, we teach highly collaborative ways of building software — part of our ethos is that even when we think we're working individually, we are still collaborating because we work with each others' code. There are tests, continuous integration, ticket management, collaboration tools, and all the fun of the fair.

If I'm writing this app in public and blogging it, then the chances are at least one or two students will glance at my code and the blog posts. And, if they're anything like I was as a student, they will be particularly delighted to see what I get wrong! (I'm sure there will be some things.)

So, I'd better to make the disclaimer up front that there will be some short-cuts.

Things that will be the same:

• Version control, using git on github
• There will be some tests (though using Specs2 rather than JUnit)
• Automated builds (though using sbt rather than gradle). At each stage, you should be able to run it at home if you want.

Things that will be different:

• Scala, not Java.

  Later in the course, we introduce students (a little) to Scala. But for the moment, it might look a little alien.

  Using Scala means a lot of the code I write can be much more concise.

  It can also be written in a functional style that makes it easy to use some powerful abstractions, and to avoid some mistakes (by making it less likely that a mistake will compile without errors).

• Some unfamiliar libraries.

  In handy, I hopefully have a bit of a head start.

  handy-appbase-core contains a few classes that have been hived off from Impressory to mean I don't need to repeat myself about functionality I've already written.

  And that includes dealing with security, asynchronous databases, returning model objects as JSON, log in with GitHub, dealing with whether it's a JSON request for data or a browser hitting a data URL, etc.

• Probably fewer tests

  On the course, having fewer tests would be a very bad habit — there's 169 other programmers who could break your code at any moment, and we'd like a test to show they haven't please.

  But over the next few days I'm not going to have 169 collaborators to keep me honest.

  The style I'm writing this in, however, tries to make sure that the type checker is less likely to let a mistake through than for a typical Java program.

• No CI server.

  I haven't yet set up a continuous integration server for the project. As I'm a lone developer, so there's nobody to continuously integrate with, I might not.

Play, Angular.js, and Handy

For a few recent projects, I've been using Angular.js at the client and the Scala Play framework at the server.

I also have a handy library that helps make a lot of things quick, easy, and tidy.

Normally, every framework has its good points but the occasional gotcha. But in this case, I found them to be a very virtuous combination, each making the others more useful.

Play and Handy

Play is a non-blocking framework, and with my handy library, it's nice and easy to write code that can include as many asynchronous calls as it likes and returns the result to the client. Even if the JSON conversion involves more asynchronous calls to fetch other information you'd like to include.

def get(contentId:String) = DataAction.one  { 
  implicit request => 
    for (
      content <- contentRef(contentId);
      approved <- request.approval ask 
                  Permissions.ReadContent(content.itself)
    ) yield content
}

In our hypothetical example here, DataAction.one will call content.toJsonFor(request.approval) to produce JSON to return. In our hypothetical example the "content" is sometimes a sequence — a playlist of other pieces of content to show in order — and converting it to JSON fetches and embeds them in the response. And it all happens aynschronously in a compact monadic style.

(This example is a slightly butchered version of what happens with content sequences in Impressory)

I'm gradually improving the documentation at handy, if you want to find out how it makes things easier.

Play's snag with HTML

If you want to Play's templating system for HTML, it has the annoying gotcha that while you can embed an Option straight into the template,

<span>@{object.myOptionValue}</span>

you can't embed a Future straight into the template.

<!-- not supported -->
<span>@object.myFutureValue</span> 

Of course that's where "gotcha" means "has the same limitation that every other templating system on the planet has too".

Play's templating, like just about every other templating framework I know of, doesn't have the smarts to turn the template itself into a Future if it's got a Futures embedded into it.

This is perfectly fair enough, but means that if you're working in a non-blocking way, you end up needing to resolve all your Futures and only then call the template, passing them in as explicit parameters.

Which always made me grumpy, as it was unnecessary overhead making things look messy. Thanks to my handy library, I could convert to JSON in an asynchronous way, with embedded Futures in the conversion, but not to HTML.

Angular.js to the rescue

The obvious way that Angular.js helps Play is that the templating happens on the client. You're no longer embedding data into the HTML, so you don't end up needing to resolve those futures.

A problem not encountered is as good as a problem solved!

A snag with Angular.js

The first gripe I had about Angular.js was perhaps an unusual one. Not the documentation (that I've coped ok with), but that inlining reusable snippets is a little clunky.

There are directives, and there's ngInclude, but the template to include either has to either:

• be fetched in another http request,
• or embedded into a <script> element in the HTML. (Which doesn't itself solve how to split it into a different file.)

Play's templating to the rescue

Well, this is of course is where Play's templating solves the problem.

An Angular.js template can be broken up into as many fragments as you like, and pieced together on the server in a single request.

So for instance, Impressory has an includeDirectives.scala.html file, whose sole responsibility is to pull in all the directive templates

<-- etc -->
<script type="text/ng-template" 
        id="directive_ce_edit_tags.html"
>
  @views.html.partials.viewcontent.directiveEditTags()
</script>
<script type="text/ng-template" 
        id="directive_ce_edit_settings.html"
>
  @views.html.partials.viewcontent.directiveEditSettings()
</script>
<-- etc --> 

In the end

In the ends, this gives me a way of writing Single Page Apps that is:

• very compact
• scalable and non-blocking
• typesafe
• looks neat and clean (a simple monadic style)
• separates concerns well (database, security, JSON conversion, etc are all handled tidily)
• above all, fast to write

This last part is particularly important — I'm employed as a research engineer, and there's only so long I can get away with cranking out code. Most of the time I want to be deciding what to build and inventing interesting solutions to interesting problems, rather than spending too much time writing the code.

Turning into a gibbering idiot at interview...

I am really bad at interviews. Suddenly, my brain flips into a mode where I'm worrying about my potential answers will appear to each of the panelists and their competing opinions and interests, and while I'm making up my mind what's coming out of my mouth is a long stream of woolly waffle.

Asked what are the emerging trends in software engineering research that we ought to be teaching students, my brain runs at a mile a minute thinking the following:

• Most of the things courses are struggling to adjust to -- cloud computing, platform as a service, polyglot programming, etc -- aren't "emerging", they've long since bloomin' well emerged. Some courses have even been struggling to get agile practices in there, and that's been around for ten years now.
• Actually the bigger problem I see students having isn't the lack of "emerging research" on their course, it's that most of their studies are devoid of context and in software engineering context can change everything (the real-world problem we're trying to solve and all the real-world constraints on the team).  So they leave uni still strangers to many of the real problems of software development.
• For your course? I'm not sure what to answer yet, because I'm still getting to know your teaching program. For ours, I know the problems the course I helped redesign had before my colleague and I redesigned it, which were...
• The answer's probably going to be different for each of the students -- what are the paths your cohort of students tend to take? Do they try to form the next great start-up, go into big data, build safety-critical software for railway signalling?
• And of course trying to think of what the favourite research topics of the panel might be.

Unfortunately, while it's thinking that (mostly thinking "would that be an ok answer?"), what's actually coming out of my mouth is a time-filling ramble, and I never quite say any of it.

Does Oracle v Google put a hole in the GPL?

I had an odd thought (a nagging worry) that with Oracle v Google's API argument resolved, there might be a gaping hole in the GPL, even version 3. I'm not a lawyer, so I hope someone will correct me on this and allay my concern. Let me tell it as a story....

One day, NastyCo decided it wanted to use nicelibrary in a proprietary software product. Nicelibrary is open source under GPLv3, but NastyCo don't want to give their source code away.

So NastyCo decide to use Maven to build their product. NastyCo don't give their customers a compiled binary; they give them a build script, and tell them "all you need to do is run the script; you can even just double click on it". When a customer runs that build script, their computer then fetches a compiled jar of NastyCo's code from NastyCo's servers, but fetches all the open source GPL libraries from Maven Central Repository and other public open source repositories. And their computer running the build script assembles the proprietary product automatically there and then.

Mr Nasty, CEO of NastyCo, reasons thusly:

• NastyCo's proprietary source code isn't a "derived work" of nicelibrary because after Oracle v Google it seems to have been made clear that there's no copyright in an API.
• The compiled and linked program together is a "derived work", but NastyCo are not distributing that -- they are just distributing the binary of their own code, and a build script. Their customers (running the build script) are fetching the rest from somewhere else and building the "derived work" themselves.
• Because NastyCo aren't distributing ("propagating") a derived work, the GPL doesn't require them to publish their source code, so they're not in breach of the GPL.
• Although the customers are running a derived work, they built it themselves (their computer built it using the build script), and the GPL explicitly allows them to run and modify private derived works, so they're not in breach of the GPL either.
• Because those privately-run derived works are allowed under the GPL, NastyCo can't even get in trouble for condoning or enabling infringement.

So, thinks Mr Nasty, neither NastyCo nor their customers are breaching the GPL, but nicelibrary has now ended up in NastyCo's proprietary product that can end up on millions of customers' machines without NastyCo ever having to publish a line of sourcecode.

So, I dearly hope there is a flaw in Mr Nasty's reasoning, but what is it?

Stanford AI Class

I was one of the many thousands who took part in the online Stanford AI class -- in my case as much to find out about how they'd make the class work as in order to learn some of the AI topics I missed out on as an undergrad way back when.  Now that it's over, here are a few thoughts:

I'll put my conclusion first. Large online classes like these won't replace local university courses; they will transform them. More and more, university lecturers are going to become content curators and facilitators, and they're going to need to write less and less of their own presentation material.

Of course, mine is a slightly biased view as the Intelligent Book, the interactive cloud teaching software I've been developing, makes it very easy to incorporate third party material like this into a lecture course.  And as you read through this, you'll sense a certain "this is why we need Intelligent Books" theme in my comments!

Anyway, into detail on what I thought of the course...

The video lectures, which were like video-recorded personal tutorials, worked very well indeed. They were clear, concise, engaging, and had the feel of being in a small class rather than a large one. Thrun and Norvig are excellent communicators and very interesting to listen to. The fact that it was an ongoing course (everyone working to a schedule), was good motivation to make time to watch the videos and do the quizzes.  That's the good news, and it really is very very good news indeed.

But every class has its flaws.  So what were this one's?

Well, the class interaction and quizzes were simplistic, both in style and content.  For instance, some of the final exam's questions on computer vision weren't about artificial intelligence at all, but were simple early high school physics questions about the optics. An object that's yay big is yay distant from a camera with a focal length of such-and-such, what's the size of the image on the image plane? Here are three objects in a scene; this camera sees them in this order, what order do they appear to be in to these other cameras that are looking at the scene from different angles?

I tend to think that while the videos are very effective for presenting a topic, they aren't so efficient for quizzes and reference.  For reference, looking up that formula just to check you've got it right, seeking within a video to find the point it was on-screen is much slower than flicking back through text.  For quizzes, the format they used only supported tick-the-box and fill-in-the-box questions, but nonetheless required the lecturers to spend time recording a video introduction for each question.

(So, this is already one area where I see the Intelligent Book bringing an advantage -- it helps courses to use a plurality of different kinds of content.  Hop from the video to the notes, to the quiz, to the advice...)

The interaction between class-members was essentially limited to forums and whatever students organised off-line. The videos were pre-recorded, so of course there wasn't much in the way of to-and-fro between the lecturers and the class, except in the "office hours" on Google Hang-outs.

This is unfortunate, as interactive teaching is very beneficial and is starting to gain traction in universities.  Eric Mazur, Bob Beichner, Rich Felder, and others in science and engineering education have been trying to encourage lecturers to interact with their classes more, and move beyond simple one-way transmission of material.  Having taught a class last semester using the Intelligent Book, with the students chatting, discussing, and giving feedback live on the lecture screen, and answering and discussing questions as a class, I genuinely missed the interaction.

So what do I think will happen next -- how do I think/hope this will change university engineering and science education?

Well, the videos really are excellent. So the first thing that will happen is that other universities will want to use these videos and others like them in their courses. Rather than Dr Joe Bloggs spend another two hours working on his PowerPoint slides for a class, he might be better off finding and showing an excellent video by famous presenters, and then spending his energy interacting with the class to further their understanding.

And I think that trend -- to use more third party prerecorded material and spend more time interacting with the class rather than preparing material -- will grow very quickly. Lecturers won't just enjoy easy access to good material; they'll realise that the lecturers who recorded these videos get a great deal of exposure and can become famous teachers -- producing the next great teaching video will become another route to increasing your academic profile. I think we'll quickly see lecturers competing to get their videos used in other people's classes.

And that, I think, means that traditional lectures will change. Short videos punctuated by class discussions and exercises, and linked to rich sets of notes and social material, will become far more common than they are now. But then, I'm biased, because that's just the sort of thing that the Intelligent Book makes easy.

Metrics for Intelligent Books

There's a post on metrics in Intelligent Books gone up over on the Intelligent Book's blog. 

Intelligent Book, Australian Innovation Challenge

I'm delighted to have been picked as a finalist in the education category of The Australian Innovation Challenge 2011 for the Intelligent Book.  (See the Weekend Australian today.)

The use of it on the Software Studio course this semester has been pretty successful, and the next step is to encourage others to use it on their couses too. More info is over on theintelligentbook.com and the About the Intelligent Book blog.