Sunday, September 30

Mistakes that cost lives

From The Economist Plenty of blame to go around, Sep 27th 2007:


"On September 5th Mattel had told an American Congressional committee that
its recall of 17.4m toys containing a small magnet that could be swallowed by
children was due to a flaw in the toys' design, rather than production
flaws in China.
"



From a BBC article on the 13 August 2007:


"The boss of a Chinese toy firm involved in a huge safety recall has committed suicide, Chinese media has said. Zhang Shuhong, who co-owned the Lee Der Toy Company, was reportedly found dead at his factory in southern China.
About 1.5 million toys made for Fisher Price, a subsidiary of US giant Mattel, were withdrawn from sale earlier this month. Many were made by Lee Der."


(emphasis all mine)
This seems to suggest the suicide was a consequence of the recall and the accusations.
If only the (late, reluctant and shameless) apology could bring Zhag Shuhong's life back...

Sources:
Plenty of blame to go around
Chinese toy boss 'kills himself'

Saturday, September 29

The eight rules of Dean Carney

From the [very entertaining] book Ugly Americans by Ben Mezrich, the tale of Ivy League traders playing the Asian markets:

  1. Never get into something you can’t get out by the closing bell. Every trade you make, you’re looking for the exit point. Always keep your eye on the exit point.
  2. Don’t ever take anything at face value. Because face value is the biggest lie of any market. Nothing is ever priced at its true worth. The key is to figure out the real, intrinsic value — and get it for much, much less.
  3. One minute, you have your feet on the ground and you’re moving forward. The next minute, the ground is gone and you’re falling. The key is to never land. Keep it in the air as long as you fucking can.
  4. You walk into a room with a grenade, and your best-case scenario is walking back out still holding that grenade. Your worst-case scenario is that the grenade explodes, blowing you into little bloody pieces. The moral of the story: don’t make bets with no upside.
  5. Don’t overthink. If it looks like a duck and quacks like a duck — it’s a duck.
  6. Fear is the greatest motivator. Motivation is what it takes to find profit.
  7. The first place to look for a solution is within the problem itself.
  8. The ends justify the means, but there’s only one end that really matters: Ending up on a beach with a bottle of champagne.


Vaguely Related:

Quotes Entirely Relevant to Software Engineers

Thursday, September 20

5 Business Ideas

Share my ISA (of dubious legality)
On the one hand there is a number of people constrained by the ISA limits but keen on investing their money in a reduced risk, tax efficient way; on the other hand, only a small number of people uses or maximises their ISA.
I see an opportunity to restore the supply-demand balance: investors could agree to "lend" money to the non-investors and maximize their ISAs every year, effectively using the ISA allowance on their behalf. The "borrower" would take a (very) small commission for his ISA allowance rental, a lot better than the nothing they would have received had their ISA allowance remained unused.
As the amount accumulates, the compounded interest may allow a renegotiation more favorable to the borrower.

Twitter Services
Services on the move in an area (e.g.: plumbers) update their status and location; interested parties subscribe to the service they need and request a visit if the professional is available and in their neighbourhood.

Rent My Garage
A garage rental marketplace: matchmaking between people with a garage and no car, and people with a car but no garage.

Paper-Holding Notebook Screen
Computer lids that also serve as page holders by either having a pressure system on the side of the screen to hold the pages in between, or having a supporting frame that slides outwards and becomes a vertical paper tray.
This would simplify tremendously the process of typing while reading from paper notes.

The Chosen 1
Unlike current online dating and matchmaking services, The Chosen 1 "game" goes beyond simple string matching.
Male and female participants are plotted on the screen as blue and pink dots resembling stars in the sky. The game challenges participants over time with questions, puzzles, activities, and preferences, some of which are introduced by the participants themselves, others are created by the system. Traits, tastes, preferences, characteristics, and relationships, can change and evolve throughout a season.
Participants can decide whether they want to attract opposites or like-minded personalities.
The Chosen 1 clusters participants creating "galaxies" and "systems" of attraction. Once certain thresholds are broken, participants within a constellation are allowed to chat online. If two participants of opposite sexes (or same sex if they so selected) are so close that they end up in a cluster of two, they are each others' Chosen 1 and given each others' contacts.
Game Over.

Because ideas are easy, doing stuff is hard. True, true.

Vaguely Related:
Business need: Digg for Consumer Electronics

Wednesday, September 19

Facebook is the new MySpace (definitely)

Morning chat I overheard on the tube:


...
Little Girl 1: I wanna get into facebook
Little Boy: yeah, everyone's in it
Little Girl 2: wha' is it?
Little Girl 1: it's kinda like myspace...my sister's in it too. I really want to join
...

Tuesday, September 18

Saturday, September 15

Barclays needs central bank loan too

From the BBC Business News, 31 August 2007:

Barclays says that a "technical breakdown" in the UK's clearing system forced it to borrow £1.6bn from the Bank of England.

It is the second time this month that the bank has tapped into the central bank's emergency credit line, sparking fears it is facing a cash crisis.
...
On 20 August, Barclays was forced to take out a £314m loan from the Bank of England after HSBC was unable to process a last-minute request for the money.


No bank run for Barclays, though.

Wednesday, September 5

Better living through (unit) testing and advice

A few weeks ago I (accidentally) found out that a pet-project of mine, the cross-lingual instant messaging, wasn't working anymore. Debugging showed the "3rd party translation sub-system" was always returning an empty translation.

The fix revealed my development and maintenance process was flawed:

  1. It was time consuming to identify the root cause;
  2. My unit tests didn't address that particular scenario;
  3. I wasn't notified when the problem first occurred, and so might have been unaware of the existence of a problem for God knows how long;
  4. There was no user friendly message given to the user, who was left with emptiness where a translated message was expected;

The course of action required addressing each of the above listed issues:
  1. Identify and fix the class of problems observed;
  2. Improve prevention and identification of similar problems in the future, by expanding the coverage and improving the quality of the unit-tests;
  3. Introduce a prevention and notification mechanism;
  4. Improve the user experience by providing user-friendly error messages;
1) Identify and fix the class of problems observed
I just had to find out what changed in the 3rd party API and adapt my parser accordingly. Thanks to Spring, this involved a small change on a XML file and didn't require recompiling the code.
Now the next step forward is trying to automate this detection and adaption process. This is still work in progress to merit a post of its own.

2) Improve coverage and quality of unit-tests
Unit-tests that don't address particular scenarios are text-book cases of code smells. Here's the signature for my translation services:


public interface ITranslationEngine {
/**
* Translates 'text' from a given 'fromLanguage' into a given 'toLanguage'
* @param fromLanguage
* @param toLanguage
* @param text
* @return
* @throws Exception
*/
public String translate(String fromLanguage, String toLanguage, String text) throws TranslationException;
}


Implementations of the method translate above were responsible for the 3 tasks below:
  1. send request to the 3rd party translation sub-system with the original message to translate, the source language, and the destination language;
  2. fetch response from the translation sub-system;
  3. parse response, extract and return translated string;

There weren't any unit-tests for any of these tasks because these multiple tasks were encapsulated under the one single method translate. When a method is long and/or does too much, it is harder to test. The method wasn't fine-grained enough to allow proper unit-test coverage, and there were at least two anti-patterns present in the tests:
  • The One, where a unit-test contains one test method which tests an entire set of functionality;
  • The Superficial Coverage, where exceptional conditions are missing from the test cases;

The fix involved extracting methods to simplify testing. The original translate method was refactored and each of the tasks listed above extracted into a well-named method. I added unit-tests that focused on each extracted method and considered input and output data in valid and exceptional scenarios. Loose coupling and testability go hand in hand.

3) Introduce a notification mechanism
To address this I implemented an Advice, a Spring schema-based AOP, to intercept the translate method, and check its arguments and return String. If the returned string is empty, a notification email is sent. The code snippet is shown below:


...
<aop:config>
<aop:aspect id="translationEngineInterceptor" ref="translationEngineAdvice">
<aop:after-returning
method="notify"
returning="translatedString"
pointcut="execution(* translate(String, String, String)) and args(from, to, text)" />
</aop:aspect>
</aop:config>

<bean id="translationEngineAdvice" class="org.jiim.translation.TranslationEngineAdvice">
<property name="mailSender" ref="mailSender"/>
<property name="mailMessage" ref="mailMessage"/>
</bean>

...

public class TranslationEngineAdvice {
protected final Log logger = LogFactory.getLog(getClass());
private MailSender mailSender;
private SimpleMailMessage mailMessage;
public static final String NEW_LINE = System.getProperty("line.separator");

...

public boolean notifyByEmail(JoinPoint jp, Object translatedString, String from, String to, String text) {
String s = (String) translatedString;
// if message string returns empty translation
if (!"".equals(text) && "".equals(s.trim())) {
SimpleMailMessage msg = new SimpleMailMessage(this.mailMessage);
StringBuffer txt = new StringBuffer(msg.getText());
txt.append(NEW_LINE).append("Date [").append(new Date()).append("]");
txt.append(NEW_LINE).append("From language [").append(from).append("]");
txt.append(NEW_LINE).append("To language [").append(to).append("]");
txt.append(NEW_LINE).append("Original Text [").append(text).append("]").append (NEW_LINE);
msg.setText(txt.toString());
try {
this.mailSender.send(msg);
return true;
} catch (MailException me) {
logger.error("Error sending email notification ", me);
}
}
return false;
}
}
...



4) Improve the user experience
I made a more considerate application by fetching a localized user friendly message to display if the translated message is empty.

References
Extracting Methods to Simplify Testing
Making Considerate Software
JUnit Anti-Patterns
TDD Anti-Pattern Catalogue
Reusable Advice - Part II: Unobtrusive Notification

Sunday, September 2

Rhino + Maven for JavaScript compression @ build time

AJAX-based Rich Internet Applications (RIAs) make heavy use of JavaScript. To improve the user experience of RIAs we can minimize the number of (JavaScript) file requests and reduce their size.

These concatenation and compression operations can be done on-the-fly or at build time.
On-the-fly techniques intercept requests and perform merging and/or compression in real-time. They are simple(r) to implement, often involving URL rewriting combined with a compress script, but:

  • don't scale, since compression costs grow proportionately to the number and size of the files;
  • add latency, by consuming CPU resources during (precious) server response time;
  • are harder to test and debug;
  • rely heavily on caching for continued performance improvements, when research indicates 40-60% of users browse with an empty cache;
  • don't work on all browsers

Compression at build time allows the use of more robust, often slower, compression methods (e.g.: a proper JavaScript interpreter instead of a regular expression) and/or combination of different methods (e.g.: run YUI Compressor after Rhino).
At build time we're unconstrained by the "real-time performance pressures".

Here's a step-by-step guide for a Maven/Ant build file using Mozilla's Rhino to serve pre-compressed JavaScript and significantly reduce application load time.

1.Setup
  1. Download the rhino.jar from Dojo ShrinkSafe
  2. Download, install, and configure Maven
  3. Copy the rhino.jar to your Maven repository
  4. Have a Maven-based Web-application ready

2.Add rhino.jar as a dependency to the POM

<dependency>
<groupId>rhino</groupId>
<artifactId>custom_rhino</artifactId>
<version>0.1</version>
<properties>
<war.bundle>true</war.bundle>
</properties>
</dependency>

3.Setup some useful variables on maven.xml


<!-- destination for the compressed JavaScript files -->
<j:set var="js.compression.dir" value="${typically_the_war_src_dir}"/>
<j:set var="js.compression.skip" value="false"/> <!-- enable/disable compression -->
<j:set var="js.compressor.lib.path" value=""/> <!-- path to the compressor jar -->

4.Fetch compressor and make it globally available

    <goal name="get-compressor" description="Set path to JavaScript compressor library">
<!-- get compressor from dependencies, to allow standalone use of goal -->
<j:if test="${empty(js.compressor.lib.path)}">
<ant:echo message="Fetching JavaScript compressor from repository" />
<j:forEach var="lib" items="${pom.artifacts}">
<j:if test="${lib.dependency.artifactId == 'custom_rhino'}">
<lib dir="${maven.repo.local}">
<include name="${lib.path}"/>
<j:set var="js.compressor.lib.path" value="${lib.path}"/>
</lib>
</j:if>
</j:forEach>
</j:if>
<ant:echo>
Path to JavaScript compressor library is ${js.compressor.lib.path}
</ant:echo>
</goal>


This way, get-compressor can be reused as a pre-requisite of all compression tasks.

5.Aggregate compression sub-tasks

<goal name="compress-js" description="Compress JavaScript across all site components">
<!-- Different sections might have different compression requirements -->
<j:if test="${js.compression.skip == 'false'}">
<attainGoal name="compress-site-js"/>
<attainGoal name="compress-forum-js"/>
...
</j:if>
</goal>


6.Compression task (to compress a single file)

<goal name="compress-site-js" prereqs="get-compressor" description="Compress JavaScript files for 'site'">
<ant:echo message="Compressing JavaScript files for 'site'" />
<j:set var="stripLinebreaks" value="true" />
<j:set var="js.dir" value="${path_to_javascript_files}"/>
<concat destfile="${js.dir}/site-concat.temp" force="yes">
<filelist dir="${js.dir}"
files="file_to_merge.js, another_file_to_merge.js, etc.js"/>
</concat>
<ant:java
jar="${js.compressor.lib.path}"
failonerror="true"
fork="yes"
output="${js.dir}/site-breaks.temp">
<ant:arg value="-c"/>
<ant:arg value="${js.dir}/site-concat.temp"/>
</ant:java>
<!-- move compressed files back from dest to src dir -->
<ant:move file="${js.dir}/site-breaks.temp" tofile="${js.dir}/site_c.js" filtering="true">
<j:if test="${stripLinebreaks == 'true'}">
<ant:echo message="Removing line breaks" />
<filterchain>
<striplinebreaks/>
</filterchain>
</j:if>
</ant:move>
<delete file="${js.dir}/site-concat.temp"/>
</goal>


This goal concatenates the 3 JavaScript files file_to_merge.js, another_file_to_merge.js,
and etc.js into a single (temporary) file site-concat.temp.
It then compresses site-concat.temp into site-breaks.temp (-breaks suffix indicates compressed file still contains line brakes).
Finally, it moves the content of site-breaks.temp into site_c.js (optionally removing line breaks) and cleans-up any temporary files and directories.

7.Compression task (to compress many independent files)

<goal name="compress-forum-js" prereqs="get-compressor" description="Compress JavaScript files for 'forum'">
<j:set var="stripLinebreaks" value="true"/>
<j:set var="src.dir" value="${path_to_javascript_files}"/>
<!-- Delete previously compressed files -->
<ant:delete>
<ant:fileset dir="${src.dir}" includes="*_c.js"/>
</ant:delete>
<ant:echo message="Compressing 'forum' JavaScript files from ${src.dir}" />
<!-- create temp dir for compressed files -->
<ant:mkdir dir="${src.dir}/_compressedjs"/>
<j:set var="dest.dir" value="${src.dir}/_compressedjs"/>
<!-- compile JavaScript files to compress -->
<ant:fileScanner var="forumJSFiles">
<ant:fileset dir="${src.dir}" casesensitive="yes">
<ant:include name="file_to_compress.js"/>
<ant:include name="another_file_to_compress.js"/>
<ant:exclude name="*_c.js"/>
</ant:fileset>
</ant:fileScanner>
<!-- loop through files and compress using compressor set in 'get-compressor' goal -->
<j:forEach var="jsFile" items="${forumJSFiles.iterator()}">
<ant:echo message="Compressing ${jsFile.name}" />
<ant:java
jar="${js.compressor.lib.path}"
failonerror="true"
fork="yes"
output="${dest.dir}/${jsFile.name}">
<ant:arg value="-c"/>
<ant:arg value="${src.dir}/${jsFile.name}"/>
</ant:java>
</j:forEach>
<!-- move compressed files back from dest to src dir -->
<ant:move todir="${src.dir}" filtering="true">
<ant:fileset dir="${dest.dir}" casesensitive="yes">
<ant:include name="*.js"/>
</ant:fileset>
<j:if test="${stripLinebreaks == 'true'}">
<ant:echo message="Removing line breaks" />
<filterchain>
<striplinebreaks/>
</filterchain>
</j:if>
<ant:mapper type="glob" from="*.js" to="*_c.js"/>
</ant:move>
<!-- delete temp dir -->
<ant:delete dir="${dest.dir}"/>
</goal>


This goal loops through a list of JavaScript files compressing them one by one.

8.Caveats

Rhino compression removes all comments, so beware of IE-specific conditional compilation statements such as the snippet below (taken from http://jibbering.com/2002/4/httprequest.html):


...
var xmlhttp=false;
/*@cc_on @*/
/*@if (@_jscript_version >= 5)
// JScript gives us Conditional compilation, we can cope with old IE versions.
// and security blocked creation of the objects.
try {
xmlhttp = new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (E) {
xmlhttp = false;
}
}
@end @*/
...


There are many possible ways around this issue, namely:
  • Use Ant to concatenate the critical code sections with the compressed files after compression
  • Move all critical sections to a separate, uncompressed file
  • Move all critical sections inline
  • Fix the compressor and submit your patch to Dojo :)


9.Improvements

When I implemented the solution above, more than a year ago, a common complaint from fellow developers was that the compression task had the undesirable side-effect of slowing down the build -biiig time. A colleague pointed me towards Ant's Uptodate task, which I then used to implement conditional compression. This means files were only compressed if they had been changed. Conditional compression reduced the automated compression time from about 1 minute to 5 seconds on average.

To use conditional compression, replace the ellipsis in the script below with the content of any of the compression goals above.


<goal name="conditional-compress-js" prereqs="get-compressor"
description="Conditional compression of JavaScript files">
<j:set var="src.dir" value="${path_to_javascript_files}"/>
<!-- check timestamps to see if compression is required -->
<ant:echo message="Checking timestamps of JavaScript files from ${src.dir}" />
<ant:fileScanner var="jsFiles">
<ant:fileset dir="${src.dir}" casesensitive="yes">
<ant:include name="**/*.js"/>
<ant:exclude name="**/*_c.js"/>
</ant:fileset>
</ant:fileScanner>
<j:forEach var="jsFile" items="${jsFiles.iterator()}">
<ant:echo message="Checking last-modified-date of ${jsFile.name}" />
<uptodate property="js.modified" targetfile="${src.dir}/${jsFile.name}">
<srcfiles dir="${src.dir}" includes="**/*_c.js" />
</uptodate>
</j:forEach>
<j:if test="${js.modified}">
<j:set var="compression.required" value="true"/>
</j:if>
<j:if test="${compression.required}">
<ant:echo message="Modified JavaScript files. Compression required." />

<!-- insert compression block here -->
...
</j:if>
</goal>


10.Further (potential) Improvements
  • Clever use of Caching, to avoid unnecessary downloading of unmodified resources.
  • Request parameter trigger to alternate between compressed and uncompressed JavaScript in real-time, a feature that is very useful for testing and debugging.
  • Very clever use of Versioning to aid with caching; my friend and former colleague Robert, responsible for an ingenious solution, can write about it in a post of his own.
  • In specific and controlled situations JavaScript namespaces can be shortened with tools like Ant, e.g.: <replace file="${file.js}" token="the.long.api.namespace" value="__u._a"/>.
  • Clever use of HTTP compression


Comments

  • I dislike the verbosity (by the very nature of XML) and obtrusiveness in the build configuration file.
  • I believe the Maven plugin approach (in combination with Julien Lecomte's excellent YUI Compressor) is a better solution.
  • With a few changes the scripts above can also be used to compress CSS resources at build time.

(Vaguely) Related Posts:
Measuring client-side performance of Web-apps
CSS clean-up @ build time


References
Custom on-the-fly compression
Make your pages load faster by combining and compressing javascript and css files
Minify
pack:tag

On-the-fly server compression
gzip, where have you been all my life..?

Compression @ build time
YUI compressor
Maven plugin for the YUI Compressor

Relevant Articles
YUI Performance Research - Part 1
YUI Performance Research - Part 2
Minification v Obfuscation
Serving JavaScript Fast
Using The XML HTTP Request
Response Time: Eight Seconds Plus Or Minus Two
Optimizing Page Load Time
Speed Web delivery with HTTP compression