I've been working for the past few months learning serverside Javascript development using Node.js to build a new Events platform for Focus.com. We had been running our Roundtable events over phone systems, where listeners would dial in at a specified time to listen to the panels speak, with very little interactivity on the part of the listeners to the panelists. We were looking for a way to improve the accessibility and interactivity of the event experience. Our goals for the project were to:

• Provide a single stream for consuming all event related content (tweets, chat, and Q&A being the 3 primary content types)
• Provide some democratization of the content of the events, allowing users to vote on the issues they wanted to see the panelists address, during the event. 
• Allow users to provide feedback on the current topic of discussion, as well as gather that data for later use in highlighting the best parts of the event as decided by attendees for those who later want to listen to a replay of the talk.

With these objectives in mind, I began digging around in search of the best technologies to leverage to accomplish these goals. To give you some history on our webstack, we've been rocking Django since 0.96, which has met all of our previous needs and then some. However, Django is not really suitable for the sort of web app experience we had in mind, where chat and other actions need to be broadcast to anywhere between tens to hundreds of users (while being scalable to thousands if need be) in as close to real time as possible. In our search for figuring out the best tools to use for this, we had several requirements:

• Fast- This needed to be able to handle communication with hundreds of simultaneous users without noticeable delays in sending and receiving data in real-time. 
• Scalable- We needed to be able to offer HA (High Availability) scaling for this service, ensuring that we did not introduce any single point of failure that could disrupt the experience.
• Capable of sending and receiving data in real time - Needed to support a large number of concurrent connections at relatively low latency.
• Easily integratable into our current Django site- Using the Django ORM (talking to MySQL) for managing all of the persistence of data was our best option for using that content elsewhere on our site, and preventing any additional points of failure from being introduced.
• No new languages- We didn't want to have to do any significant development with any language our developers were not already familiar with. This basically limited us to Python and Javascript. 

In the end, we resolved to add two new pieces of tech to our stack: RabbitMQ and Node.js.

RabbitMQ is an Erlang implementation of AMQP (Advanced Message Queuing Protocol). "AMQP is middleware to provide a point of rendezvous between back-end systems (data stores and services) and front end systems such as end user applications."[1] Node.js is pretty much the hotness right now, recently taking the crown from Rails for being the cool new tech on the block (if Github traffic is any indication). Node.js offered us the opportunity to use Javascript both serverside and clientside for the events application. 

So, technologies chosen, I set out learning how to develop using Node.js's event-driven model. After a couple prototype iterations we found an architecture that would accomplish our goals. Here's an early diagram of it:

(Disclaimer: I don't know UML. I am terrible at flowcharts. Sorry)

Since that early diagram, the only real change was that the majority of content-related data is now sent by the Nodejs layer to the client after the client handshakes with Node. The content that is static per event is cached at the page level in the Django layer. 

Architecture within Django

We have 8 models defined in the Django ORM that contain data needed by the live event app. These include all of the primary forms of content in the first bullet point in this post, as well as all secondary forms of content that relate to those primary ones (ie: things like votes, answers to questions, etc.). I added to each of these models a 'to_json_serializable' method, which returns a Python dictionary object representing all relevant data for an object of that model that can be JSON serialized.

I bound the following post_save signal handler to all 8 models:

(EDIT: Link for if the embed doesn't appear)

I leverage memcached here to determine if the output from to_json_serializable has changed since last save to reduce unnecessary data transfer (ie: if some object attributes change, but the relevant attributes do not).

The amqp_transmit function sends the JSON package through to RabbitMQ to a fanout exchange that all of the individual Node.js instances have unique queues bound to. Each instance bound to the fanout gets a copy of all data pushed from Django to the fanout.

The last architecture change within Django was building a full RESTful API for accessing and modifying all of these models. The GET requests to the API leverage the to_json_serializable methods, while the POST requests are able to use a user's existing Django user account and authentication state for managing access control. 

Architecture within Node.js

Working with Node.js was a ton of fun, and introduced me to an event-driven programming style. We used socket.io for managing connections from the browser to Node.js, a web transport library with both client and serverside Javascript, designed specifically for use in real time web applications, It manages the transfer of data between those entities. 

When a Node instance is instantiated, the first thing it does is bind a queue to the RabbitMQ fanout exchange that Django is pushing data to. Once that connection is established, it begins listening for socket.io connections from clients, and requests a list of events from Django. At this point, any event-specific data requests from clients or data-updates from Django's post_save signals are deferred. Once Django returns to Node a list of events, separate objects are created for each event, and all of the currently deferred functions are moved from the global events queue to the relevant queue inside of the event object representing the event which they are attending. Upon instantiation of a new event object, Node makes a request to Django for all of the relevant data. Once it gets and processes all of that data, the deferred functions for that event are evaluated in FIFO order. 

Depending on the browser, you'll get a users current sessionid cookie (originally set by Django) in the initial socket.io-facilitated connection from the client to Node.js. If this value is not provided, Node.js makes an additional request to the client for this value. That value is then verified with Django to associate identity with each client. We ended up needing to change our SESSION_COOKIE_DOMAIN value in Django's settings.py from www.focus.com to .focus.com before launching the platform, in order to launch our node boxes behind node.focus.com and allow sessionid cookies to be shared between the two subdomains.

Once a user is connected and Node has finished gathering data about the event the user is attending, Node has a fairly simple job to do. It only has to listen for updates about data changing via RabbitMQ, update its internal representation of the event, and ship the diff to all connected clients attending the event tied to that content.

Overall, I'm extremely happy with how the finished product has come together, and believe that we made the right architecture decisions here. Please leave any questions or comments, I'd be happy to go into further detail about any aspect that I may have glossed over. 

[1]: http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol#Overview

PS: I'll be speaking at one of our events tomorrow Wednesday 9/21/11 at 2:00 PT about the new platform, feel free to drop in and check it out. Behind Focus.com: Unveiling our new Live Event App

We need to do away with Secret Questions. You know, when you sign up for an account online, and it asks you to answer a seemingly innocuous question. And therein, lies half of the problem, the innocuous nature of the question that they ask. The other half, is that you're never quite sure what someone can do with knowledge of the secret question and its answer. 

Maybe I'm jaded, but I remember the days of several youthful indiscretions on my part where a combination of Google Fu and a very small amount of social engineering resulted in full access to others' email accounts. Back in my early teens, I used to play an online game (Asheron's Call) with account management and authentication provided by Microsoft's The Zone. There was a thriving community on IGN's VNBoards forums for the game, including server specific individual forums. There was a thread where people posted where they lived in real life. On it's own, this would be no problem. However, accounts on VNBoards had an optional publically visible email field. Zone accounts were tied to an MSN or Hotmail email address. In many cases, the very same email that was publically visible on the user's VN account. Hotmail was the most popular email provider of the day, and it was the easiest to reset the password of.

The first step of gaining full access to the email account was scanning the entire thread and seeing which users had a Hotmail email address tied to their account. Then trying the password reset functionality for the email address in the cases where it was a Hotmail account. The reset was a two step process. Step One involved entering the user's zip code (and Country and State, but those are a product of one's zip). Google made it pretty easy to turn the posted string describing a user's location from the VN thread into a list of zip codes, which could then be tried one by one until it accepted one and brought you to Step Two.

The second step would ask you for the answer to the user's secret question. Who honestly remembers exactly which secret question they picked for a service they signed up for months or even years ago? These questions are designed to be innocuous. And there's the problem. A stranger can ask you these sorts of questions and without even thinking about the security implications you'll probably respond honestly. 

Succeeding with my social engineering experiment in some cases was as simple as Googling the user's zip code plus the words 'high school' to get the name of the schools in the area and trying different permutations of capitalizing or abbreviating the school's name. In other cases, due to either a slightly more personal question, or due to weak Google Fu, approaching the user was necessary.

Approaching them and getting the answer was pretty simple. Create a new character on the online game with a very feminine name, send them a private message stating that you're a girl who thinks they know them (Saw you live in my town from the 'Where do you live?' thread on VNBoards!), build some rapport and then ask them the secret question. Wait for them to log off, rest the password, and BAM! Full account access to their email. Despite the questionable morality of gaining full access to the user's email, I thankfully had enough moral fiber as a pre-teen (Thanks Mom and Dad!) to not be a dick and abuse the power. I politely left them an email from themselves explaining how I did it, and that they should probably change their secret answer to something a little harder to social engineer out of them.

Now, while these secret questions may have seemed like a good idea back in the early days of the Internet, I'd hope that we would have done away with them entirely by now. The above use-case is an explanation of the worst-case scenario for how secret questions can be exploited. However, when required to put a secret question on a new account, you're left totally in the dark as to how and when a secret question will be posed to you, or whether someone can take over your account with nothing but these bits of information. What site have you ever signed up for that tells you how and when they use the secret questions? Personally, I don't know that I've encountered any.

The biggest problem with Secret Questions as a standard, is that there is no standard.

 Ever since my little experiment, I've answered every single Secret Question question by mashing random buttons on the keyboard and submitting. This has only ever caused me problems once, and an email to the support team solved the issue. 

What I'm proposing is that we get rid of these things entirely. I'm not here with a perfect alternative. Off the top of my head, maybe sending the user a text message with a one-time passcode, or asking extremely personal questions instead of harmless ones, or asking for characters 12-16 of your favorite credit card; I'm betting that a better option exists than any of these. However, the total removal from the web of Secret Questions in their current incarnation would be a better alternative than continuing to use them as they exist today.

As some of you may know, I recently began working at Focus Research as a Software Engineer, shipping Django, jQuery, Python, etc. Part of the process of ramping up to own the parts of the site I was hired to handle, particularly the site's News Feed, has been to fix all the existing bugs in PivotalTracker that relate to the Feed as a way to dig in and get a better sense of the existing code. 

Recently I was tasked with fixing a bug where on specific portions of the feed, a particular user's profile image did not display. Trying to replicate the bug was difficult without the ability to create a piece of content originating from that user's account, so I started looking into what might be the easiest way to create content from that user's account on my local environment. My boss, Dan, forwarded me a link to Impostor mentioning that he'd read the description and bookmarked it, but had gone no further with it than that.

The installation instructions claimed it was easy as adding a line to AUTHENTICATION_BACKENDS in settings.py and a line to INSTALLED_APPS. Unfortunately, AUTHENTICATION_BACKENDS seemed to be missing from our settings file. This may be an artifact of the way our code was originally designed to run on Django 0.96, but I'm not really sure, I wasn't writing Django code back then. I added a line for that setting, hoping that getting Impostor to work would be as easy as advertised. 

Once installed, a user with the staff flag enabled on their account can log into another user's account by using the following login string as their username, with their usual password.

staff_username as user_username

Attempted logging in through our interstitial login form. No dice. Attempted logging in through a secondary login form. Attempted logging in through the django admin panel. All of these returned errors suggesting that they had failed form validation when comparing a string involving two email addresses and whitespace to validation rules expecting a single email address.

After realizing that this was not going to be a drag and drop solution, I hacked up the following solution to get Impostor working. The noteworthy changes to the normal authentication process are as follows:

I had to remove the lines from the clean() method that call request.session.test_cookie_worked() because that method was returning False despite the fact that cookies seemingly work on my development box, using Impostor or the default authentication backend. 

I had to add the backend attribute to the user object, setting it to 'impostor.backend.AuthBackend' after retrieving it from the form, before passing it to django.contrib.auth.login(request, user).

Below I've embedded a Github Gist containing the view method and form that we used to get Impostor working at Focus.