Vicky Cristina Barcelona

Is Woody Allen at his best.

The story and cinematic style are reminiscent of Rohmer's Pauline à la plage and Conte d'été.

5/5 stars

Eccentric Java: oddities and corner cases

Java Puzzlers is a worthy complement to Effective Java. It provides a collection of puzzles that illustrate obscure (and some times bizarre) behavior of the Java programming language and its core libraries.

I've highlighted some of the book's puzzles below, grouped by Numerical, API, and OOP-related traps & solutions.

§ Numerical Traps

• Use the bitwise AND (&) instead of the MODULO (%) operator. The example in the book is a particular case of the more general equivalence: i % 2n = i & (2n - 1), which I first read about in Hacker's Delight. The bitwise expression on the right is likely faster and works safely with negative operands. [#1, page 5]



• Use integrals types (such as int or long) or BigDecimal for precise floating-point calculations, e.g.: 200 - 110 = 90 instead of 2.00 - 1.10 = 0.89999...
  The impossibility of representing 0.1 in binary floating-point is explained with all the detail one can handle in What Every Computer Scientist Should Know About Floating-Point Arithmetic. [#2, page 8]



• Beware of hidden integer overflows. When working with large numbers add the suffix 'L' (uppercase to improve legibility) to promote every literal to a long, e.g.: final long MICROS_PER_DAY = 24L * 60 * 60 * 1000;. [#3, page 9]



• Watch out for loop guards <= Integer.MAX_VALUE. All integers are <= Integer.MAX_VALUE and when Integer.MAX_VALUE is incremented it silently wraps around to Integer.MIN_VALUE. [#26, page 57]



• Beware of math operations at integer boundaries, e.g.: Math.abs(Integer.MIN_VALUE); If the argument is Integer.MIN_VALUE or Long.MIN_VALUE, Math.abs returns its argument. [#64, page 149]

§ API Traps

• Pattern.quote(String) converts a String into a regex pattern that will match the string. Use String.replaceAll(Pattern.quote("."), ","); instead of String.replaceAll("\\."), ",");, for example, to prevent the accidental omission of escape characters. [#20, page 43]



• The finally block is always executed, even if the try block returns. Err...except if the try block calls System.exit(0). @See #39 below. I first read about this in Peter Norvig's Java Infrequently Answered Questions. [#36, page 77]



• The set of checked exceptions that a method can throw is the intersection of the set of checked exceptions that is declared to throw in all applicable types. [#37, page 79]



• The System.exit(0) method stops the execution of the current thread & all others, but does not cause finally blocks to execute.
  We can use shutdown hooks to add behavior that must occur before the VM exits, e.g.:
  
  

  
  public class HelloWorld {
   public static void main(String[] args) {
       System.out.println("Hello");
       Runtime.getRuntime().addShutdownHook(new Thread() {
           public void run() {
               System.out.println(" world");
           }
       });
       System.exit(0);
   }
  }
  

  
  [#39, page 83]



• The close method of input & output streams can throw IOException too; from JSE5, input and output streams implement the Closeable interface. This lets us write a generic close method for streams, e.g.:
  
  

  
  ...
  InputStream in = null;
  try {
      in = new FileInputStream(src);
      ...
  } finally {
      closeAndIgnoreException(in);
  }
  
  private static void closeAndIgnoreException(Closeable c) {
   if (c != null) {
       try {
           c.close();
       } catch(IOException ioe){ // ignored }
   }
  }
  

  
  [#41, page 87]



• The JVM has a stack depth limit; when the limit is exceeded a StackOverflowError is thrown, regardless of the infinite nature of the recursion. [#45, page 101]



• Use Method m = <object variable>.class.getMethod(<method name>); instead of Method m = <object variable>.getClass().getMethod(<method name>);to prevent possible runtime exceptions caused by the runtime type returned by reflection. [#79, page 189]

§ OOP Traps

• Java's overload resolution process selects the most specific of the methods or constructor available. Supposing we declare the methods:
  
  

  
      void f(float) { ... }
      void f(double) { ... }
  

  
  
  Calling f with f(42), the parameter types for the first method are assignable to the parameter types of the second method through a widening primitive conversion. That is, double = float. By contrast, float = double is not valid without a cast. Thus, f(double) is removed from the set of possible methods to call and f(float) is called. [#46, page 105]



• A static member, such as a thread-safe counter, must be synchronized on the class object itself, e.g.:
  
  

  
  public class SynchronizedStatic {
    private static Long counter;
    public SynchronizedStatic() {
        synchronized (SynchronizedStatic.class) {
            counter++;
        }
    }
  }
  

  
  [#55, page 127]



• When a variable and a type have the same name and are both in scope, the variable name takes precedence, e.g. (from the book):
  
  

  
  public class ShadesOfGray {
      public static void main(String[] args){
          System.out.println(X.Y.Z); // prints 'White'
      }
  }
  class X {
      static class Y {
          static String Z = "Black";
      } 
      static C Y = new C(); // variable Y takes precedence over type Y
  }
  class C {
      String Z = "White";
  }
  

  
  [#68, page 163]



• Package-private methods cannot be directly overridden by a method in a different package (hence the name, I'd say.) [#70, page 168]



• The final modifier means different things on methods & fields: final instance methods cannot be overridden; final static methods may not be hidden; final fields may not be assigned more than once. [#72, page 171]



• Overriding = replacement with same name, signature, & exceptions thrown -without reducing visibility;
  Hiding = static override; Overloading = supplement with same name but different signature & exceptions thrown;
  Shadowing (textual scope) = enclosing scope is shadowed by local variables, methods, and types with the same name;
  Obscuring = static shadowing. [#Glossary, page 180]

Resources:

What Every Computer Scientist Should Know About Floating-Point Arithmetic
IEEE Standard 754 Floating Point Numbers
Why integer overflow "wraps around"
Hacker's Delight
Peter Norvig's Java Infrequently Answered Questions
JVM Overview
Overload Resolution
Algorithms for Overload Resolution
Function Overloading With Partial Ordering
The Java Language Specification
Effective Java: Second Edition
Java Puzzlers: Traps, Pitfalls, and Corner Cases
Advanced Topics in Programming Languages: Java Puzzlers, Episode VI

More underground map obsession

London Tube Journey Planner (TubeJP) offers detailed public transport maps for London



On NY Turf offers routes for New York City


On both services, a layer with the geographic transport routes (not their equivalent schematic diagrams) overlays the city maps.


Related:
A Taste of a Moscow Tube Station
Tube Map variations (great stuff!)
Moscow Tube
LondonTown Tube - The Real Thing

Not bad

Jia (staring intently at my face):

"Each part of your face is not perfect but when you put them all together
is not bad"

:D

Hey! My life is O.K.! (happy, happy!)

My Complete Far Side (50% off the original price!) arrived Friday morning and made my weekend in a way that can only be described by Zita Swoon's wonderful Bananaqueen below

Some classics:

Gary Larson is brilliant.
One thing, though: the 2 volumes are way too big and heavy -4 volumes would have been much better :)

(More) Resources For Learning Mandarin

Another gem for my growing list of Resources For Learning Mandarin:

Nciku contains a neat dictionary, conversations (text-to-speech, optionally annotated with hànzì and pīnyīn), Q & A's, tools, an awesome handwriting recognition tool that let's you draw Chinese characters, and plenty more cool stuff!

Related:
Resources For Learning Mandarin

RE: Where Are the Software Engineers of Tomorrow?

I found the article by Emeritus Professors Robert B.K. Dewar and Edmond Schonberg anecdotal and weakly supported.
I describe the argument flaws by quoting and commenting in-line in the rest of this post.

"For example, Bjarne Stroustrup reports from Texas A & M University that the
industry is showing increasing unhappiness with the results of this approach."

Here, the industry as represented by Bjarne Stroustrup is hardly impartial evidence.
Represented by James Gosling the evidence from the same industry might have been different.

"It [Texas A&M] did [teach Java as the first language]. Then I started
teaching C++ to the electrical engineers and when the EE students started to
out-program the CS students, the CS department switched to C++. [5]"

Hmmm, the conclusion here may be wrong due to many types of selection bias:

1. The creator of C++ is likely to attract C++ inclined candidates.
2. Students applying to an University where the creator of C++ teaches C++ will more likely be those keen on the programming language, and thus more likely to be good at it.
3. It is likely that students taught by Bjarne Stroustrup, the creator of the C++ language himself, are going to end up better C++ programmers than your average programmer in any language.
   

It is conceivable that a Computer Science department where Guido van Rossum teaches Python will attract and create lots of above average Python programmers, beating the crap out of their C++ and Java colleagues (the company of 'stars' raises the level of everyone else).

"At AdaCore, we are certainly aware of many universities that have adopted Ada
as a first language because of similar concerns."

Again, selection bias: on the one hand, AdaCore will obviously look more actively for such Universities; on the other hand, such Universities are more likely to establish a relationship with [an important?] Ada company and make AdaCore aware of their adoption.

This is surely not a rigorous scientific analysis.


The above paragraph, with no concrete evidence to support it, also seems to follow Bjarne Stroustrup's excerpt before it and suggest that, just like the switch to C++, switching to Ada results in students who out-program their colleagues programming in other languages.
No evidence of this is given in the article, and there are no scientific studies published on this.

"Ada is the language of software engineering par excellence. Even when it is not
the language of instruction in programming courses, it is the language chosen to
teach courses in software engineering. This is because the notions of strong
typing, encapsulation, information hiding, concurrency, generic programming,
inheritance, and so on, are embodied in specific features of the language."

This is a conceptual jump from personal views to universal ones. Plus, all the notions listed are features of Java, C# and plenty of other languages besides Ada.

"From our experience and that of our customers, we can say that a real
programmer writes Ada in any language."

Oh. Selection bias and anecdotal evidence again. Their experience and that of customers of AdaCore is likely of an Ada-centric universe.

"For example, an Ada programmer accustomed to Ada’s package model, which
strongly separates specification from implementation, will tend to write C in a
style where well-commented header files act in somewhat the same way as package
specs in Ada. The programmer will include bounds checking and consistency checks
when passing mutable structures between subprograms to mimic the strong-typing
checks that Ada mandates [6]. She will organize concurrent programs into tasks
and protected objects, with well-defined synchronization and communication
mechanisms."

There is no necessary causality relationship between being an Ada programmer and writing good C code. Although it is likely that [programming] languages determines the way we think, the relationship shown is more like that of being a good programmer and writing good code in either Ada or C.

From earlier statements in the article, it is difficult to hire good Ada programmers. When they find them, these [good] Ada programmers also write good C code. No surprises there.

In the end, there is also no evidence given that Ada is a better replacement for Java as an entry level language.


Sources:
Professors Slam Java As "Damaging" To Students
Computer Science Education: Where Are the Software Engineers of Tomorrow?

Types and Programming Languages Revisited

My previous brute-force approach to list all the 4-dimension combinations of S = {(latent, manifest), (static, dynamic), (weak, strong), (nominal, structural)} typing -by filtering the desired tuples from the powerset of S - wasn't very efficient.
It was a perfect example of The Danger of Naïveté: an inefficient algorithm that was nevertheless simple to devise and implement.

Moreover, quoting The eight rules of Dean Carney, I should have observed that "the first place to look for a solution is within the problem itself". If the problem involves 4 tuples of 2 elements each to begin with, the efficient solution is not likely to require the powerset of all 8 elements.

Thus, I figured I should give it another go.

First, for simplicity and better readability, let me replace the values of S with the character-tuples on the left hand side:

(A, B) = (latent, manifest)
(C, D) = (static, dynamic)
(E, F) = (weak, strong)
(G, H) = (nominal, structural)

S = {(A, B), (C, D), (E, F), (G, H)}

Next, I'll write down some 4-size combinations. After a short while, a pattern emerges:

ACEG
ACEH
ACFG
ACFH
ADEG
ADEH
...

The pattern becomes clear when I look at the elements like this:

X₁=(A, B) x X₂=(C, D) x X₃=(E, F) x X₄=(G, H)

This is a Cartesian product of X₁ x X₂ x ... x X_n, the set containing all possible combinations of one element from each set.

With this knowledge, implementing an efficient algorithm is easy. First, a method to return the Cartesian product of two sets:


# arguments must be lists, e.g.: cartesian([1, 2], [3, 4])
def cartesian(list1, list2):
     result = []
     for el in list1:
         for item in list2:
             if isinstance(el, list):
                 result.append(el + [item])
             else:
                 result.append([el] + [item])
     return result


Testing the method with input X=[1, 2] x Y=[3, 4] yields:


     [[1, 3], [1, 4], [2, 3], [2, 4]]


Which looks good.
Next, a method to process the Cartesian product of any number of sets. Here's a candidate implementation:



def types_cartesian(aList):
     head = aList[0]
     tail = aList[1:]
     for item in tail:
         head = cartesian(head, item)
     return head


The output of method types_cartesian(aList) given input aList=S is:

S={['static', 'dynamic'], ['weak', 'strong'], ['latent', 'manifest'], ['nominal', 'structural']}

S₁ x S₂ x ... S_n =


     ['static', 'weak', 'latent', 'nominal']
     ['static', 'weak', 'latent', 'structural']
     ['static', 'weak', 'manifest', 'nominal']
     ['static', 'weak', 'manifest', 'structural']
     ['static', 'strong', 'latent', 'nominal']
     ['static', 'strong', 'latent', 'structural']
     ['static', 'strong', 'manifest', 'nominal']
     ['static', 'strong', 'manifest', 'structural']
     ['dynamic', 'weak', 'latent', 'nominal']
     ['dynamic', 'weak', 'latent', 'structural']
     ['dynamic', 'weak', 'manifest', 'nominal']
     ['dynamic', 'weak', 'manifest', 'structural']
     ['dynamic', 'strong', 'latent', 'nominal']
     ['dynamic', 'strong', 'latent', 'structural']
     ['dynamic', 'strong', 'manifest', 'nominal']
     ['dynamic', 'strong', 'manifest', 'structural']


This is the same list as the one obtained earlier with the powerset approach.
The big difference is that with the powerset implementation the number of possible combinations explodes exponentially ( O(2ⁿ) ), whereas the Cartesian product is a quadratic ( O(n²) ) algorithm.

Here's the code.


Related:
Types and Programming Languages
http://en.wikipedia.org/wiki/Cartesian_product
http://programming-misc.googlecode.com/svn/trunk/python/types_cartesian.py

Anchoring & Brainstorming

From Quote of the week: Why brainstorming is a bad idea:

"To their surprise, the researchers found that virtual groups, where
people brainstormed individually, generated nearly twice as many
ideas as the real groups.

The result, it turned out, is not an anomaly. In a 1987 study,
researchers concluded that brainstorming groups have never
outperformed virtual groups.
Of the 25 reported experiments by psychologists all over
the world, real groups have never once been shown to be
more productive than virtual groups.
In fact, real groups that engage in brainstorming consistently
generate about half the number of ideas they would have
produced if the group's individuals had worked alone."

Fascinating stuff.
Could this be related to [Nobel laureate] Daniel Kanheman's work on the Anchoring bias, where [apparently individual] contributions during a brainstorming session overly rely (anchor) on each others' ideas?




Useful Links:
Tversky, A. & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases
http://en.wikipedia.org/wiki/Daniel_Kahneman
http://www.princeton.edu/~kahneman/
The Medici Effect

Types and Programming Languages

This excellent overview of type systems by nostrademons @ reddit describes latent, manifest, static, dynamic, weak, strong, nominal, and structural typing, and places a number of popular programming languages on some of these dimensions.

I got curious about listing all the possible combinations and then find programming languages to fill in the slots.

I cooked up some quick and dirty Python code to do the job.

Let's call S = {static, dynamic, latent, manifest, weak, strong, nominal, structural}
The powerset of S has 2⁸ = 256 combinations, only some of them useful but I can fix that later.

# return a list with 2len(l) elements (definition of powerset)
def powerset(l):
     r = []
     if l:
         head = l[:1]
         tail = l[1:]
         for item in powerset(tail):
             r.append(item)
             r.append(head + item)
     else:
         r.append([])
     return r


The powerset of S returns the complete list of subsets of S from sizes 1 to 8 plus the empty set {}.
This means subsets {static}, {dynamic}, {static, dynamic}, ..., all the way to {static, dynamic, latent, manifest, weak, strong, nominal, structural} are included too.

However, there are 4 pairs of [kind of] mutually exclusive types and I am only interested in the subsets of size 4. Thus, I need only consider the unique 4-dimension combinations.

The maths tells me I should expect n!/k!(n-k)! = 8!/4!(8-4)! = 70 such combinations (see binomial coefficient or look up the middle number of the 8^th row of Pascal's triangle).

I can obtain this by keeping only those subsets of size 4 from the list returned by the powerset. The function below will help:

# filter list keeping only n-tuples where n=size
def len_filter(l, size=1):
     return [item for item in l if len(item) == size]


Still, the set of 70 size-4 combinations contain lots of invalid members because of the 4 pairs of [kind of] mutually exclusive types (read original comment for details):

    ("static", "dynamic"),
    ("latent", "manifest"),
    ("weak", "strong"),
    ("nominal", "structural")


A simple solution is to just remove any combination that contains these "invalid" pairs. The 2 functions below will do that:


# retain n-tuples that don't contain any of the invalid pairs
def invalidpairs_filter(l):
     return [item for item in l if not is_invalid(INVALID_PAIRS, item)]

# returns True if any of the invalid tuples is contained in the given n-tuple,
# False otherwise
def is_invalid(invalid_pairs, ntuple):
     s = set(ntuple)
     for invalid_pair in invalid_pairs:
         si = set(invalid_pair)
         if si.issubset(s):
             return True
     return False



Et voilá! Running the Python program prints the 16 combinations we want:


    ['latent', 'nominal', 'static', 'weak']
    ['dynamic', 'latent', 'nominal', 'weak']
    ['manifest', 'nominal', 'static', 'weak']
    ['dynamic', 'manifest', 'nominal', 'weak']
    ['latent', 'nominal', 'static', 'strong']
    ['dynamic', 'latent', 'nominal', 'strong']
    ['manifest', 'nominal', 'static', 'strong']
    ['dynamic', 'manifest', 'nominal', 'strong']
    ['latent', 'static', 'structural', 'weak']
    ['dynamic', 'latent', 'structural', 'weak']
    ['manifest', 'static', 'structural', 'weak']
    ['dynamic', 'manifest', 'structural', 'weak']
    ['latent', 'static', 'strong', 'structural']
    ['dynamic', 'latent', 'strong', 'structural']
    ['manifest', 'static', 'strong', 'structural']
    ['dynamic', 'manifest', 'strong', 'structural']


Should there be a programming language for all the above combinations? nostrademons and grauenwolf already filled in a few of these slots (apologies if my copy & paste introduced errors):

latent, nominal, static, weak =
dynamic, latent, nominal, weak = PHP
manifest, nominal, static, weak = C, Access SQL, Java, C++, C#,
dynamic, manifest, nominal, weak =
latent, nominal, static, strong = Haskell
dynamic, latent, nominal, strong = Scheme
manifest, nominal, static, strong = Java, C++, C#, VB.NET (strict), T-SQL
dynamic, manifest, nominal, strong =
latent, static, structural, weak = VB.NET, VBScript
dynamic, latent, structural, weak = JavaScript, PHP, Assembly
manifest, static, structural, weak =
dynamic, manifest, structural, weak =
latent, static, strong, structural = Ocaml, Haskell
dynamic, latent, strong, structural = Erlang, Scheme, Common Lisp, Python, Ruby
manifest, static, strong, structural = Haskell, C++ Templates
dynamic, manifest, strong, structural = Common Lisp


Repetitions occur because most languages support multiple forms of typing.

My next task (for another night, another post) is to clean-up the table and find languages that fit in the 5 unused dimensions from the list above:


    ['latent', 'nominal', 'static', 'weak'] =
    ['dynamic', 'manifest', 'nominal', 'weak'] =
    ['dynamic', 'manifest', 'nominal', 'strong'] =
    ['manifest', 'static', 'structural', 'weak'] =
    ['dynamic', 'manifest', 'structural', 'weak'] =


Possible Questions:

1. No programming languages for a given combination? Why is that?


2. Once most known programming languages found their place in the matrix, it will be interesting to understand why some combinations are more popular than others

I've created project programming-misc on Google Code to hold these and other programming exercises I might engage in. The code shown above is available in the SVN repository and downloads page.


Resources:
3-31-04 I'm Over It (the article by Bruce Eckel that started the conversation @ reddit)
Comment by nostrademons
Comment bt grauenwolf


[Update]
It is quite bizarre that the otherwise very polished Google Analytics has a bug known since (at least) Nov 2005!
The URL considered invalid by Google Analytics is actually the one generated by and taken from Google Code. The issue is well documented with easy work-arounds, though.

One-liners that made me lol

(original theme by the-one-and-only Niniane)

"Don't let your beard get in the way of your typing"

A comment on this Reddit thread.

Mistakes that cost lives

From The Economist Plenty of blame to go around, Sep 27th 2007:

"On September 5th Mattel had told an American Congressional committee that
its recall of 17.4m toys containing a small magnet that could be swallowed by
children was due to a flaw in the toys' design, rather than production
flaws in China."

From a BBC article on the 13 August 2007:

"The boss of a Chinese toy firm involved in a huge safety recall has committed suicide, Chinese media has said. Zhang Shuhong, who co-owned the Lee Der Toy Company, was reportedly found dead at his factory in southern China.
About 1.5 million toys made for Fisher Price, a subsidiary of US giant Mattel, were withdrawn from sale earlier this month. Many were made by Lee Der."

(emphasis all mine)
It seems clear the suicide was a consequence of the recall and the accusations.
If only the (late, reluctant and shameless) apology could bring Zhag Shuhong's life back...

Sources:
Plenty of blame to go around
Chinese toy boss 'kills himself'

The eight rules of Dean Carney

From the [very entertaining] book Ugly Americans by Ben Mezrich, the tale of Ivy League traders playing the Asian markets:

1. Never get into something you can’t get out by the closing bell. Every trade you make, you’re looking for the exit point. Always keep your eye on the exit point.
2. Don’t ever take anything at face value. Because face value is the biggest lie of any market. Nothing is ever priced at its true worth. The key is to figure out the real, intrinsic value — and get it for much, much less.
3. One minute, you have your feet on the ground and you’re moving forward. The next minute, the ground is gone and you’re falling. The key is to never land. Keep it in the air as long as you fucking can.
4. You walk into a room with a grenade, and your best-case scenario is walking back out still holding that grenade. Your worst-case scenario is that the grenade explodes, blowing you into little bloody pieces. The moral of the story: don’t make bets with no upside.
5. Don’t overthink. If it looks like a duck and quacks like a duck — it’s a duck.
6. Fear is the greatest motivator. Motivation is what it takes to find profit.
7. The first place to look for a solution is within the problem itself.
8. The ends justify the means, but there’s only one end that really matters: Ending up on a beach with a bottle of champagne.

Vaguely Related:

Quotes Entirely Relevant to Software Engineers

5 Business Ideas

Share my ISA (of dubious legality)
On the one hand there is a number of people constrained by the ISA limits but keen on investing their money in a reduced risk, tax efficient way; on the other hand, only a small number of people uses or maximises their ISA.
I see an opportunity to restore the supply-demand balance: investors could agree to "lend" money to the non-investors and maximize their ISAs every year, effectively using the ISA allowance on their behalf. The "borrower" would take a (very) small commission for his ISA allowance rental, a lot better than the nothing they would have received had their ISA allowance remained unused.
As the amount accumulates, the compounded interest may allow a renegotiation more favorable to the borrower.

Twitter Services
Services on the move in an area (e.g.: plumbers) update their status and location; interested parties subscribe to the service they need and request a visit if the professional is available and in their neighbourhood.

Rent My Garage
A garage rental marketplace: matchmaking between people with a garage and no car, and people with a car but no garage.

Paper-Holding Notebook Screen
Computer lids that also serve as page holders by either having a pressure system on the side of the screen to hold the pages in between, or having a supporting frame that slides outwards and becomes a vertical paper tray.
This would simplify tremendously the process of typing while reading from paper notes.

The Chosen 1
Unlike current online dating and matchmaking services, The Chosen 1 "game" goes beyond simple string matching.
Male and female participants are plotted on the screen as blue and pink dots resembling stars in the sky. The game challenges participants over time with questions, puzzles, activities, and preferences, some of which are introduced by the participants themselves, others are created by the system. Traits, tastes, preferences, characteristics, and relationships, can change and evolve throughout a season.
Participants can decide whether they want to attract opposites or like-minded personalities.
The Chosen 1 clusters participants creating "galaxies" and "systems" of attraction. Once certain thresholds are broken, participants within a constellation are allowed to chat online. If two participants of opposite sexes (or same sex if they so selected) are so close that they end up in a cluster of two, they are each others' Chosen 1 and given each others' contacts.
Game Over.

Because ideas are easy, doing stuff is hard. True, true.

Vaguely Related:
Business need: Digg for Consumer Electronics

Facebook is the new MySpace (definitely)

Morning chat I overheard on the tube:

...
Little Girl 1: I wanna get into facebook
Little Boy: yeah, everyone's in it
Little Girl 2: wha' is it?
Little Girl 1: it's kinda like myspace...my sister's in it too. I really want to join
...

Fight for Kisses

Barclays needs central bank loan too

From the BBC Business News, 31 August 2007:

Barclays says that a "technical breakdown" in the UK's clearing system forced it to borrow £1.6bn from the Bank of England.

It is the second time this month that the bank has tapped into the central bank's emergency credit line, sparking fears it is facing a cash crisis.
...
On 20 August, Barclays was forced to take out a £314m loan from the Bank of England after HSBC was unable to process a last-minute request for the money.

No bank run for Barclays, though.

Better living through (unit) testing and advice

A few weeks ago I (accidentally) found out that a pet-project of mine, the cross-lingual instant messaging, wasn't working anymore. Debugging showed the "3rd party translation sub-system" was always returning an empty translation.

The fix revealed my development and maintenance process was flawed:

1. It was time consuming to identify the root cause;
2. My unit tests didn't address that particular scenario;
3. I wasn't notified when the problem first occurred, and so might have been unaware of the existence of a problem for God knows how long;
4. There was no user friendly message given to the user, who was left with emptiness where a translated message was expected;
   

The course of action required addressing each of the above listed issues:

1. Identify and fix the class of problems observed;
2. Improve prevention and identification of similar problems in the future, by expanding the coverage and improving the quality of the unit-tests;
3. Introduce a prevention and notification mechanism;
4. Improve the user experience by providing user-friendly error messages;
   

1) Identify and fix the class of problems observed
I just had to find out what changed in the 3rd party API and adapt my parser accordingly. Thanks to Spring, this involved a small change on a XML file and didn't require recompiling the code.
Now the next step forward is trying to automate this detection and adaption process. This is still work in progress to merit a post of its own.

2) Improve coverage and quality of unit-tests
Unit-tests that don't address particular scenarios are text-book cases of code smells. Here's the signature for my translation services:

public interface ITranslationEngine {
  /**
    * Translates 'text' from a given 'fromLanguage' into a given 'toLanguage'
    * @param fromLanguage
    * @param toLanguage
    * @param text
    * @return
    * @throws Exception
  */
  public String translate(String fromLanguage, String toLanguage, String text) throws TranslationException;
}

Implementations of the method translate above were responsible for the 3 tasks below:

1. send request to the 3rd party translation sub-system with the original message to translate, the source language, and the destination language;
2. fetch response from the translation sub-system;
3. parse response, extract and return translated string;
   

There weren't any unit-tests for any of these tasks because these multiple tasks were encapsulated under the one single method translate. When a method is long and/or does too much, it is harder to test. The method wasn't fine-grained enough to allow proper unit-test coverage, and there were at least two anti-patterns present in the tests:

• The One, where a unit-test contains one test method which tests an entire set of functionality;
• The Superficial Coverage, where exceptional conditions are missing from the test cases;
  

The fix involved extracting methods to simplify testing. The original translate method was refactored and each of the tasks listed above extracted into a well-named method. I added unit-tests that focused on each extracted method and considered input and output data in valid and exceptional scenarios. Loose coupling and testability go hand in hand.

3) Introduce a notification mechanism
To address this I implemented an Advice, a Spring schema-based AOP, to intercept the translate method, and check its arguments and return String. If the returned string is empty, a notification email is sent. The code snippet is shown below:

...
<aop:config>
   <aop:aspect id="translationEngineInterceptor" ref="translationEngineAdvice">
      <aop:after-returning
          method="notify"
          returning="translatedString"
          pointcut="execution(* translate(String, String, String)) and args(from, to, text)" />
   </aop:aspect>
</aop:config>

<bean id="translationEngineAdvice" class="org.jiim.translation.TranslationEngineAdvice">
    <property name="mailSender" ref="mailSender"/>
    <property name="mailMessage" ref="mailMessage"/>
</bean>

...

public class TranslationEngineAdvice {
  protected final Log logger = LogFactory.getLog(getClass());
  private MailSender mailSender;
  private SimpleMailMessage mailMessage;
  public static final String NEW_LINE = System.getProperty("line.separator");

  ...

  public boolean notifyByEmail(JoinPoint jp, Object translatedString, String from, String to, String text) {
    String s = (String) translatedString;
    // if message string returns empty translation
    if (!"".equals(text) && "".equals(s.trim())) {
      SimpleMailMessage msg = new SimpleMailMessage(this.mailMessage);
      StringBuffer txt = new StringBuffer(msg.getText());
      txt.append(NEW_LINE).append("Date [").append(new Date()).append("]");
      txt.append(NEW_LINE).append("From language [").append(from).append("]");
      txt.append(NEW_LINE).append("To language [").append(to).append("]");
      txt.append(NEW_LINE).append("Original Text [").append(text).append("]").append (NEW_LINE);
      msg.setText(txt.toString());
      try {
        this.mailSender.send(msg);
        return true;
      } catch (MailException me) {
        logger.error("Error sending email notification ", me);
      }
    }
    return false;
  }
}
...

4) Improve the user experience
I made a more considerate application by fetching a localized user friendly message to display if the translated message is empty.

References
Extracting Methods to Simplify Testing
Making Considerate Software
JUnit Anti-Patterns
TDD Anti-Pattern Catalogue
Reusable Advice - Part II: Unobtrusive Notification

Rhino + Maven for JavaScript compression @ build time

AJAX-based Rich Internet Applications (RIAs) make heavy use of JavaScript. To improve the user experience of RIAs we can minimize the number of (JavaScript) file requests and reduce their size.

These concatenation and compression operations can be done on-the-fly or at build time.
On-the-fly techniques intercept requests and perform merging and/or compression in real-time. They are simple(r) to implement, often involving URL rewriting combined with a compress script, but:

• don't scale, since compression costs grow proportionately to the number and size of the files;
• add latency, by consuming CPU resources during (precious) server response time;
• are harder to test and debug;
• rely heavily on caching for continued performance improvements, when research indicates 40-60% of users browse with an empty cache;
• don't work on all browsers
  

Compression at build time allows the use of more robust, often slower, compression methods (e.g.: a proper JavaScript interpreter instead of a regular expression) and/or combination of different methods (e.g.: run YUI Compressor after Rhino).
At build time we're unconstrained by the "real-time performance pressures".

Here's a step-by-step guide for a Maven/Ant build file using Mozilla's Rhino to serve pre-compressed JavaScript and significantly reduce application load time.

1.Setup

1. Download the rhino.jar from Dojo ShrinkSafe
   
2. Download, install, and configure Maven
3. Copy the rhino.jar to your Maven repository
4. Have a Maven-based Web-application ready
   

2.Add rhino.jar as a dependency to the POM

<dependency>
 <groupId>rhino</groupId>
 <artifactId>custom_rhino</artifactId>
 <version>0.1</version>
 <properties>
   <war.bundle>true</war.bundle>
 </properties>
</dependency>

3.Setup some useful variables on maven.xml

<!-- destination for the compressed JavaScript files -->
<j:set var="js.compression.dir" value="${typically_the_war_src_dir}"/>
<j:set var="js.compression.skip" value="false"/> <!-- enable/disable compression -->
<j:set var="js.compressor.lib.path" value=""/> <!-- path to the compressor jar -->

4.Fetch compressor and make it globally available

    <goal name="get-compressor" description="Set path to JavaScript compressor library">
<!-- get compressor from dependencies, to allow standalone use of goal -->
<j:if test="${empty(js.compressor.lib.path)}">
 <ant:echo message="Fetching JavaScript compressor from repository" />
 <j:forEach var="lib" items="${pom.artifacts}">
   <j:if test="${lib.dependency.artifactId == 'custom_rhino'}">
     <lib dir="${maven.repo.local}">
       <include name="${lib.path}"/>
       <j:set var="js.compressor.lib.path" value="${lib.path}"/>
     </lib>
   </j:if>
 </j:forEach>
</j:if>
<ant:echo>
 Path to JavaScript compressor library is ${js.compressor.lib.path}
</ant:echo>
</goal>

This way, get-compressor can be reused as a pre-requisite of all compression tasks.

5.Aggregate compression sub-tasks

<goal name="compress-js" description="Compress JavaScript across all site components">
<!-- Different sections might have different compression requirements -->
<j:if test="${js.compression.skip == 'false'}">
 <attainGoal name="compress-site-js"/>
 <attainGoal name="compress-forum-js"/>
 ...
</j:if>
</goal>

6.Compression task (to compress a single file)

<goal name="compress-site-js" prereqs="get-compressor" description="Compress JavaScript files for 'site'">
<ant:echo message="Compressing JavaScript files for 'site'" />
<j:set var="stripLinebreaks" value="true" />
<j:set var="js.dir" value="${path_to_javascript_files}"/>
<concat destfile="${js.dir}/site-concat.temp" force="yes">
 <filelist dir="${js.dir}"
   files="file_to_merge.js, another_file_to_merge.js, etc.js"/>
</concat>
<ant:java
  jar="${js.compressor.lib.path}"
  failonerror="true"
  fork="yes"
  output="${js.dir}/site-breaks.temp">
  <ant:arg value="-c"/>
  <ant:arg value="${js.dir}/site-concat.temp"/>
</ant:java>
<!-- move compressed files back from dest to src dir -->
<ant:move file="${js.dir}/site-breaks.temp" tofile="${js.dir}/site_c.js" filtering="true">
 <j:if test="${stripLinebreaks == 'true'}">
   <ant:echo message="Removing line breaks" />
   <filterchain>
      <striplinebreaks/>
   </filterchain>
 </j:if>
</ant:move>
<delete file="${js.dir}/site-concat.temp"/>
</goal>

This goal concatenates the 3 JavaScript files file_to_merge.js, another_file_to_merge.js,
and etc.js into a single (temporary) file site-concat.temp.
It then compresses site-concat.temp into site-breaks.temp (-breaks suffix indicates compressed file still contains line brakes).
Finally, it moves the content of site-breaks.temp into site_c.js (optionally removing line breaks) and cleans-up any temporary files and directories.

7.Compression task (to compress many independent files)

<goal name="compress-forum-js" prereqs="get-compressor" description="Compress JavaScript files for 'forum'">
<j:set var="stripLinebreaks" value="true"/>
<j:set var="src.dir" value="${path_to_javascript_files}"/>
<!-- Delete previously compressed files -->
<ant:delete>
 <ant:fileset dir="${src.dir}" includes="*_c.js"/>
</ant:delete>
<ant:echo message="Compressing 'forum' JavaScript files from ${src.dir}" />
<!-- create temp dir for compressed files -->
<ant:mkdir dir="${src.dir}/_compressedjs"/>
<j:set var="dest.dir" value="${src.dir}/_compressedjs"/>
<!-- compile JavaScript files to compress -->
<ant:fileScanner var="forumJSFiles">
 <ant:fileset dir="${src.dir}" casesensitive="yes">
   <ant:include name="file_to_compress.js"/>
   <ant:include name="another_file_to_compress.js"/>
   <ant:exclude name="*_c.js"/>
 </ant:fileset>
</ant:fileScanner>
<!-- loop through files and compress using compressor set in 'get-compressor' goal -->
<j:forEach var="jsFile" items="${forumJSFiles.iterator()}">
  <ant:echo message="Compressing ${jsFile.name}" />
  <ant:java
      jar="${js.compressor.lib.path}"
      failonerror="true"
      fork="yes"
      output="${dest.dir}/${jsFile.name}">
      <ant:arg value="-c"/>
      <ant:arg value="${src.dir}/${jsFile.name}"/>
  </ant:java>
</j:forEach>
<!-- move compressed files back from dest to src dir -->
<ant:move todir="${src.dir}" filtering="true">
  <ant:fileset dir="${dest.dir}" casesensitive="yes">
    <ant:include name="*.js"/>
  </ant:fileset>
  <j:if test="${stripLinebreaks == 'true'}">
    <ant:echo message="Removing line breaks" />
    <filterchain>
      <striplinebreaks/>
    </filterchain>
  </j:if>
  <ant:mapper type="glob" from="*.js" to="*_c.js"/>
</ant:move>
<!-- delete temp dir -->
<ant:delete dir="${dest.dir}"/>
</goal>

This goal loops through a list of JavaScript files compressing them one by one.

8.Caveats

Rhino compression removes all comments, so beware of IE-specific conditional compilation statements such as the snippet below (taken from http://jibbering.com/2002/4/httprequest.html):

...
var xmlhttp=false;
/*@cc_on @*/
/*@if (@_jscript_version >= 5)
// JScript gives us Conditional compilation, we can cope with old IE versions.
// and security blocked creation of the objects.
try {
 xmlhttp = new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
 xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (E) {
 xmlhttp = false;
}
}
@end @*/
...

There are many possible ways around this issue, namely:

• Use Ant to concatenate the critical code sections with the compressed files after compression
• Move all critical sections to a separate, uncompressed file
• Move all critical sections inline
• Fix the compressor and submit your patch to Dojo :)
  

9.Improvements

When I implemented the solution above, more than a year ago, a common complaint from fellow developers was that the compression task had the undesirable side-effect of slowing down the build -biiig time. A colleague pointed me towards Ant's Uptodate task, which I then used to implement conditional compression. This means files were only compressed if they had been changed. Conditional compression reduced the automated compression time from about 1 minute to 5 seconds on average.

To use conditional compression, replace the ellipsis in the script below with the content of any of the compression goals above.

<goal name="conditional-compress-js" prereqs="get-compressor"
                 description="Conditional compression of JavaScript files">
<j:set var="src.dir" value="${path_to_javascript_files}"/>
<!-- check timestamps to see if compression is required -->
<ant:echo message="Checking timestamps of JavaScript files from ${src.dir}" />
<ant:fileScanner var="jsFiles">
 <ant:fileset dir="${src.dir}" casesensitive="yes">
   <ant:include name="**/*.js"/>
   <ant:exclude name="**/*_c.js"/>
 </ant:fileset>
</ant:fileScanner>
<j:forEach var="jsFile" items="${jsFiles.iterator()}">
 <ant:echo message="Checking last-modified-date of ${jsFile.name}" />
 <uptodate property="js.modified" targetfile="${src.dir}/${jsFile.name}">
   <srcfiles dir="${src.dir}" includes="**/*_c.js" />
 </uptodate>
</j:forEach>
<j:if test="${js.modified}">
 <j:set var="compression.required" value="true"/>
</j:if>
<j:if test="${compression.required}">
 <ant:echo message="Modified JavaScript files. Compression required." />

 <!-- insert compression block here -->
 ...
</j:if>
</goal>

10.Further (potential) Improvements

• Clever use of Caching, to avoid unnecessary downloading of unmodified resources.
• Request parameter trigger to alternate between compressed and uncompressed JavaScript in real-time, a feature that is very useful for testing and debugging.
• Very clever use of Versioning to aid with caching; my friend and former colleague Robert, responsible for an ingenious solution, can write about it in a post of his own.
• In specific and controlled situations JavaScript namespaces can be shortened with tools like Ant, e.g.: <replace file="${file.js}" token="the.long.api.namespace" value="__u._a"/>.
• Clever use of HTTP compression
  

Comments

• I dislike the verbosity (by the very nature of XML) and obtrusiveness in the build configuration file.
• I believe the Maven plugin approach (in combination with Julien Lecomte's excellent YUI Compressor) is a better solution.
• With a few changes the scripts above can also be used to compress CSS resources at build time.
  

(Vaguely) Related Posts:
Measuring client-side performance of Web-apps
CSS clean-up @ build time

References
Custom on-the-fly compression
Make your pages load faster by combining and compressing javascript and css files
Minify
pack:tag

On-the-fly server compression
gzip, where have you been all my life..?

Compression @ build time
YUI compressor
Maven plugin for the YUI Compressor

Relevant Articles
YUI Performance Research - Part 1
YUI Performance Research - Part 2
Minification v Obfuscation
Serving JavaScript Fast
Using The XML HTTP Request
Response Time: Eight Seconds Plus Or Minus Two
Optimizing Page Load Time
Speed Web delivery with HTTP compression

Reusable Advice - Part II: Unobtrusive Notification

Besides encapsulating behaviors that affect multiple classes into reusable modules, AOP can also be used for unobtrusive notification tasks.
Instead of browsing application logs looking for exceptions, for example, I like to have the application itself email me the exception stack-trace and the arguments that caused the error when something goes wrong.

Here's an example advice that sends an email after an exception is thrown. The date of occurrence and the exception stack-trace are sent in the body of the email.

The Java code for the advice:

...
public class ThrowableAdvice {
 private MailSender mailSender;
 private SimpleMailMessage mailMessage;
 public static final String NEW_LINE = System.getProperty("line.separator");

 // Setters and getters omitted for clarity
 ...

 /**
 * Send email notification with exception stacktrace
 *
 * @param throwable
 * @return boolean frue if email message is sent, false otherwise
 */
 public boolean notifyByEmail(Throwable throwable) {
   SimpleMailMessage msg = new SimpleMailMessage(this.mailMessage);
   StringBuffer txt = new StringBuffer(msg.getText());
   txt.append(NEW_LINE).append(new Date());
   txt.append(NEW_LINE).append(ExceptionUtils.stackTraceToString(throwable));
   msg.setText(txt.toString());
   try {
     this.mailSender.send(msg);
     return true;
   } catch (MailException me) {
     logger.error("Error sending email notification ", me);
   }
   return false;
 }
}
...

A useful method to "print" a stack-trace onto a String:

...
public static String stackTraceToString(Throwable throwable) {
 StringWriter sw = new StringWriter();
 PrintWriter pw = new PrintWriter(sw, true);
 throwable.printStackTrace(pw);
 return sw.getBuffer().toString();
}
...

And a snippet of the Spring XML configuration:

...
<aop:config>
   <aop:aspect id="appThrowableInterceptor" ref="throwableAdvice">
      <aop:after-throwing
         method="notifyByEmail"
         throwing="throwable"
         pointcut="execution(* some.package.*.*(..))" />
   </aop:aspect>
</aop:config>

<bean id="throwableAdvice" class="ThrowableAdvice">
   <property name="mailSender" ref="mailSender"/>
   <property name="mailMessage" ref="throwableMessage"/>
</bean>

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">
   <property name="host" value="localhost"/>
</bean>

<bean id="throwableMessage" class="org.springframework.mail.SimpleMailMessage">
   <property name="subject" value="ALERT! Exception thrown @ some app module"/>
       <property name="text" value="ALERT! Exception thrown from some module"/>
       <property name="from" value="the.notifier@application.org"/>
       <property name="to">
       <list>
           <value>developers@application.org</value>
       </list>
   </property>
</bean>
...

This is as flexible and unobtrusive as it gets -the throwableAdvice advice above can be:

• easily reused by specifying a different pointcut (above in bold);
• removed altogether without needing to touch the advised (target) code;

(Vaguely) Related Posts:
Better Living Through (Unit) Testing and Advice
Reusable Advice - Part I


References:
AOP@Work: Dependency injection with AspectJ and Spring
AOP@Work: Check out library aspects with AspectJ 5
AOP@Work: AOP myths and realities
Improve modularity with aspect-oriented programming
Spring AOP Reference documentation

Compulsive Attention Seeker

A few weeks ago, I approached a TfL staff member (let's call him SM) after failing to exit the gate with my Oyster card. Chat transcript below.

Me: "Hi, I can't get out and the machine keeps telling me to seek attention..."

SM (grinning and slowly printing each word in my head): "seek attention? Are you sure the machine told you to seek attention"?

Me (mildly annoyed and a tad confused): "Yes, it displayed the words seek attention in flashing red letters!"

This goes on a few more rounds until SM, visibly amused, opens the gate and let's me through.

...

Today's note to self: the gate screens actually display the words "Seek Assistance"...