Full stack web development

Over the next few months, I plan to write a series of posts about full-stack web development. This first post is a rough, but sufficiently accurate guide to how web applications have evolved over the last 15 or so years – and I’ll end with the set of technologies I plan to write-about over the next few  months.

In the good old days, we had simple web pages (with animated gifs!). Apple, that paragon of beautiful design, had a website that looked like this:

apple_1473503a

Things were simple back then – often, the developer would administer the web server (often his only machine), and he would write HTML pages and place it in a folder (/www) on his server machine. These HTML pages would then be served when a browser requested the page…

1

But there was a problem – you could only get static content. What if you wanted your visitors to see how many other people had visited this web-site (remember those spinning visitor count images?!). Or what if you wanted people to fill out a form with their name and email address? That was when you turned to CGI and Perl scripts – simple code that often was run by the web server itself, and that could interface with the file-system or databases.

2

But organizing code in CGI/Perl scripts was rough. CGI didn’t scale well (a new process for each request), was often insecure (accessed the file system / environment variables), and it didnt provide any structured way of building dynamic applications. Things got confusing for a few years (till about 2005) – there was Java Server Pages (JSP), Microsoft’s ASP and then there was PHP! I like to think of the reference architecture at this time as being IIS & ASP.NET – you could build scalable, secure applications, and development was quite easy with Visual Studio.

3

Things got interesting though once we had AJAX (2005) – till now, web servers mostly sent back an entire page or document. AJAX allowed for JavaScript code on the client side to request for a fragment of a page – and then dynamically (i.e – without going back to the server) refresh a part of the page (by updating the document in the browser – called DOM, or document object model).

What was sent back was still HTML, but the code on the browser could stick in this HTML fragment inline to the current page. This meant that web apps could be more responsive (I believe that this was when we really had a web-app, instead of a web-page). Google’s GMail and Google Maps were the killer demos for AJAX, and suddenly, everyone was using AJAX to refresh a portion of the page.

4

For a couple of years, things were all about AJAX, using the existing technologies on the server. And then, in about 2007, there was a new kid on the block – Ruby on Rails, from 37signals. The “5 minute blog” demo for Ruby on Rails took the developer world by storm, and suddenly everyone was talking about Rails! What made Rails different was that there was a prescribed way to lay out your web application – using a development pattern that had been commonly used in desktop applications, but that had been forgotten in the move to web apps. This was Model (data) – View (templates) – Controller (business logic). Rails was opinionated – “this is the way to do it”, had lots of plugins and made building web apps sane once again.

5

Between 2007 and 2010, there were 3 developments that prompted another change:

The first was the rise of smart phones and mobile apps. Now, as was often the case, many applications had a web version and a mobile phone app for it. However, the server was sending back HTML pages – not something the mobile phone app could consume. So, you needed structured data from the server, not HTML.

The second development was the rise of jQuery – a JavaScript library that became really popular and made it easy to build very dynamic, beautiful web applications – and that made AJAX easy!

The third development was the release of nodejs – for the first time, you could have high performance JavaScript on the server, thus potentially ending the nightmare of having “client side developers” who knew HTML/JavaScript and “server side developers” who knew .NET/C#/Ruby.

6

This was a good architecture – but we could re-use the learnings on the client side to simplify the spaghetti of jQuery code that was happening on the client side. What was needed was an “opinionated” way to assemble HTML pages on the client from data that was fetched from the server – similar in spirit to Rails. So, in the last 2 years, we’ve had  number of frameworks that simplify the client side development – Backbone, Ember, Derby and Meteor – and my favorite one, AngularJS.

7

So, this is where we are today – and my reference architecture is mongodb for the database server, node/express as the web/application server, AngularJS on the client side (along with Bootstrap for styling). I find that this architecture allows me to build web services really quickly that can have a client interface build fast using AngularJS, and then one can start development of mobile apps against the service using PhoneGap or native mobile development tools.

8

Over the next several weeks, I’ll have posts on each of these: MongoDB, Node/ExpressJS, JSON and REST interfaces, AngularJS, Testing with Karma & mocha and Bootstrap for styling pages.

64 thoughts on “Full stack web development

    • arunr Post author

      I have looked at NuoDB (and RethinkDB and TaffyDB and others… :)). As compared to Nuo (which if I’m not mistaken is not open-source), the fact that Mongo is open is a big draw. Technology options are evolving fast, and for me, documentation and support of a technology is another big criteria: Mongo does great there too, with great documentation on patterns in which it should be used.

      Reply
    • Declan de Wet

      User registration with Node is particularly easy with things like Express/Connect, Mongoose and Passport. Express has built-in support for setting up sessions in roughly 2-5 lines of code, depending on how you want your sessions to work (SSL, custom memory store, etc.). Mongoose is an ODM for MongoDB and it exposes before and after middleware on certain events that happen in the database, like a model’s save event, which makes things like hashing the user’s password fairly straightforward (UserSchema.pre(‘save’, callback)). Registration can be coded from scratch if it’s simple but Passport offers hundreds of authentication strategies – local (username & password), twitter, github, OAuth 1 & 2, HTTP bearer tokens, Basic Auth and they all work alongside each other, but are all optional too.

      Reply
    • arunr Post author

      Hi Jacob – thanks, I do plan to write an article about that. I’ve found is that passportjs works awesome & passport-local-mongoose (look on mongoosejs at plugins) makes it even easier to configure a user authentication system. Stay tuned…

      Reply
    • dimitrk

      Well, in that case scroll uuuuup to the first stack diagram / image you will see in the page. When someone designs a road, do you ask the engineer who designed it if horses can ride on it? Lets drop such questions or similar (IE6 support) and move forward on the 2013 era where browser vendors remove options such as “disable javascript” from their products… ;)

      Reply
    • Declan de Wet

      Node.js is a server-side architecture platform, so everything Node handles will still work for users with no JavaScript. It’s merely the client side stuff that starts breaking. Some web applications are built with JavaScript-Only users in mind, and any smart front-end developer will cater to this situation by informing the user to turn on JavaScript if they want to use the webapp.

      However, if the web app isn’t too complicated and can still serve the same content (page reloads being the only difference), then it is rather trivial to detect if the request came from an X-Requested-With header set to the value of XMLHttpRequest in Express and use the result to determine whether to send JSON or render a server-side view, like so:

      app.get(‘/posts’, function(request, response) {
      if (req.xhr) { // this request came from AJAX
      res.send({some: ‘json’}); // send JSON object, could be a database call
      } else { // not from AJAX
      res.render(‘some_view’, { some: ‘json’ }); // render a view with the same data
      }
      });

      Combine this with something like Backbone’s pushState API and write a top-level JQuery click handler that disables the default link click event on JavaScript-enabled browsers and you’re essentially serving the exact same content to both users with the exception that the user who has JavaScript enabled will benefit from no page refreshes (There are SEO benefits too, but I go the extra step and use the _escaped_fragment_ query to cater to crawlers)

      Reply
  1. Bill

    I liked the summary in the beginning (possibly because most of that is history to me, and I didn’t know anything about it), up to and including the part about Rails. After that, though, the technologies chosen seem a little arbitrary and difficult to back up. They’re still being developed in the present and it will be hard to judge their impact for another few years at least.

    Reply
  2. Abel Muíño

    Good summary, although I have to disagree with MVC having been forgotten until Rails. The java world was using it a lot earlier with Struts, for instance. JSP was (mostly) the template language for the views.

    Reply
  3. Ruturaj K. Vartak

    I’ve gone through almost-ALL the phases of this web-development cycle… Now slightly trailing behined (not yet using AngularJS ;) ).

    Why did you find mongoDB as preferred ? I’ve used it multiple times, and everytime it has either not scaled well, or not helped to get the desired result.

    Reply
  4. S

    > Backbone, Ember, Derby and Meteor – and my favorite one, AngularJS.

    How did you even compare them, “simplify the client side development”? Derby and Meteor are full stack frameworks, simplify the whole life of “full stack developer”

    Reply
  5. pht

    The next thing is to tackle the biggest issue : how much data is transfered *in total* to the client (hence bandwith and latency issues with getting megabytes of code to the browser before anything can happen…)

    Reply
  6. Paul

    Hi

    Great overview, as I started in 1992 I remember all of this :-) I moved into mobile development some years back, I kind of lost what happened since 2006. I was a ColdFusion developer in the good old days.

    What you should also consider doing is creating a page with some sort of chart / grid / flow / overview of the various technologies (Javascript libraries, scripting, database etc.).

    Part of the problem I see is that there are so many open source solutions that’s very hard to work out which one you should be going with as your stack. There are all these terms like Bootstrap, Frameworks, etc – how do they all relate to each other ? If only someone would put together some way to explain all of them you would have a killer site on your hands for newbies and oldies! :-)

    Thanks

    Reply
  7. Pingback: Full Stack Web Development – act2.me | Mark Provan

  8. ubersoldat

    You lost me when implying that RoR brought to web development MVC or anything new in fact. The only thing that RoR brought was Ruby, that’s it, everything was already been done in other platforms: MVC, convention over configuration, ORM, out-of-the-box authentication, i18n, etc.
    On the other hand, I’ve been developing web pages since the CGI and Java Applets days and my reference architecture these days is pretty similar to yours, except that I’m not so sure about AngularJS since I’m loving Meteor A LOT!

    Reply
  9. kevin

    great article…i will be following ur blog…
    you did not explain what json is…
    also how does browser use the json data to do inline change of the page in the first node picture

    Reply
  10. Alex Bee

    So basically you’re suggesting do everything in JS and you’ll be fine. All-js approach works fine for small to mid-sized projects. And, as always, there is no one tool for the job. So some-js, some more clean and beautiful languages and technologies work much better IMHO.

    Reply
  11. hosting

    You can certainly see your enthusiasm in the work you write. The arena hopes for more passionate writers like you who are not afraid to say how they believe. At all times follow your heart.

    Reply
  12. arunr Post author

    Thanks for all the comments – I’ve added the ability to subscribe by email and the RSS feed, as many of you requested. I’ve also set up Akismet to get rid of the spam. Thanks!

    Reply
    • Declan de Wet

      *Sort of* is the answer to that question. While it is a little complicated for search engines to do and requires a little configuration, there are easier ways.

      One of those ways is useful if you’re running a server like nginx or Apache. The concept is that you have a headless browser like PhantomJS and use that along with a server config to reroute the crawler to a path, on which the PhantomJS takes the requested URL, parses it and prints a snapshot of the state of the JS web app to the crawler, which the crawler can then index as if it were a regular page.

      I personally think that’s a bit much, so for trivial web applications that can have their content rendered server-side as well as client-side, I simply enable HTML5′s pushState in the frontend framework I’m using and have an equivalent server-side route that serves the same content. Search engines can then index that as normal.

      For web applications that are a little bit more than trivial, most search engines that follow Google’s AJAX specifications or similar will be able to detect an AJAX application if they see this in the of an HTML page:

      From there, on the server side you can listen for the route appended with an _escaped_fragment_ query appended to the end (/route?_escaped_fragment_) and serve just the content of what you want the crawler to index.

      Reply
      • Declan de Wet

        Forgive the above comment, as the HTML I entered after the second last paragraph was stripped from the comment body. Basically, what I meant to say was that in the HEAD tag of the HTML, if the crawler sees the tagname “meta” with name “fragment” and content “!”, the crawler will then know to use the _escaped_fragment_ query parameter.

        Reply
  13. Pingback: Weekly Design News (N.195)

  14. Pingback: Weekly Design News – Resources, Tutorials and Freebies (N.195) | Wordpress Webdesigner

  15. Pingback: Weekly Design News – Resources, Tutorials and Freebies (N.195) | Joe Garde

  16. Pingback: Messy Web: Angular.js, Ruby on Rails, Dart, GWT! Almost forgot Closure « Giant Elk

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>