One vexing issue I ran into is that often the ssh connection would fail while running the script, yet I would have no problem ssh'ing manually afterwards. I spent some time googling the problem and found some people having similar mysterious failures, but no concrete solutions.

The failure mode for me during script execution looked exactly like what you see when you've screwed up your keys somehow: running with -v, you can see it tries public-key, has no luck, and moves on to ask for a password. That definitely is not going to work because a) it's not interactive and b) there is no root password on a fresh EC2 instance.

So, what is the problem? Well, I have not really figured it out, but I have come to realize that there is a period of time after the Amazon says the instance is 'running' during which it will still reject all SSL connections. The period of time seems to vary between 30 and 60 seconds, and seems to increase during what I would assume are peak usage hours.

I speculate that somewhere in the cloud is a little elf that runs around distributing Amazon's half of the keypairs to all newly-started EC2 instances. The lag in SSL connectivity is due to the elf getting tired and/or busy, and the monitoring machinery is not designed to wait for him to finish before it declares that an instance is 'running'. I've not been able to confirm that this is the case, but I'm sure Occam would favor explanations involving lazy elves.

A crude way around the problem is to simply sleep for a minute or so before trying to establish a connection. A slightly less crude way is to repeatedly attempt to SSH in after the instance starts.

A lot of my work lately has relied on generating XML using templates (I'm using Velocity). The XML is mostly used to either drive an ETL process, specify some XHTML to get rendered, or to configure some internal subsystem.

It gets complicated, though:

• Sometimes that XML gets consumed outside the VM via HTTP, and sometimes it gets consumed inside the VM.
• In either case, I need to be able to embed template parameters in a URL (i.e., query stirng).
• I want a uniform method for addressing the templates (inside or outside the VM)
• I want a uniform method for processing and organizing the templates in the webapp
• Some of the templates I want locked down from the outside - I don't want them served out over HTTP
• ...except that sometimes I do. For debugging, it turns out to be really handy to be able to peek at template instantiations in my browser.

My solution was to create a backdoor URL dispatcher for internal clients. So, from outside I can access a template as

http://localhost/foo/bar/baz.vm?name=roger

whereas from inside the VM (and only inside the VM) I can access it as

myapp:/foo/bar/baz.vm?name=roger

One nice thing about this is that the myapp URL will bypass the container security. This means that I can be free of having to authenticate when calling internally without having to do unnatural things to my web.xml security configuration to lock it down from outside.

Another nice benefit is that the servlet request gets processed in the caller's thread. This lets me propagate any Exceptions encountered while directly back to the caller, which makes debugging and error handling a lot simpler when there are problems in a template.

(more...)

So, what is a self-respecting hacker to do besides write their own? Here's the nutshell:

SQLSheet is a JDBC driver which allows you to interact with Microsoft Excel spreadsheets using SQL statements.

Features:

• Pure Java (no native components)
• Fast (operates directly on the spreadsheet, does not rely on an in-memory database)
• Read and write operations
• PreparedStatement support

Now, if you just want to manipulate spreadsheets, just use Apache's POI - it's great. Why is SqlSheet useful, then? Mainly for manipulating spreadsheets inside other tools that are database-oriented. You can just give them a jdbc:xls URL and start doing whatever you normally do. In my case, I specifically wanted to be able to manipulate spreadsheets from inside the Scriptella ETL framework.

(more...)

I just switched from MovableType 4 to WordPress and am overall very pleased. It's faster, cleaner, simpler, and seems to have a stronger community. It's PHP based, which I much prefer to MT's weird template tags. A couple of pitfalls:

If you are like me and like to write your entries in raw HTML, there are a couple of things you need to do:

• Go to 'Users', click 'Edit' on your username, and uncheck 'Use the visual editor when writing'
• Download the Raw HTML plugin.
• Or, better yet, use my hacked version which is always on (i.e., doesn't require the magic 'raw' comment delimiters).

Also, don't mess around with symlinks or .htaccess - this seems to somehow cause problems with deleting comments and posts. If you come from MT, it's tempting to do this because WordPress wants to live under your document root, whereas MT publishes static pages into a separate directory. My advice is to just put WordPress in your public_html dir where it wants to be - it feels weird but it's ok.

Finally, I highly recommend downloading the Persistent Styles Plugin so that you can non-destructively tweak wordpress templates.

• I like to use .properties files for configuration unless there is a compelling reason to use something more complicated.
• I'm pretty uptight about using constants in my code. This includes using constants to refer to the names of properties in a .properties file.
• Over time, the constants and the .properties file can drift apart - you have to keep them both in sync.
• Similarly, properly documenting them both becomes a painful

So, I proceeded to finally do something about it. I spent about 15 minutes writing the code and 45 minutes writing the documentation, go figure why. Maybe it's because The Wire is over and I think I just needed to do something to say goodbye. At any rate, I have found it has made my life just a little bit easier.

With that, I give you, PropJoe.

docs | download

My applications run inside Tomcat, and I should note also that Tomcat does include a native Windows launcher that may be sufficient for many purposes. In my case, however, I don't want my users to know that they are running Tomcat. I want to have complete control over the startup environment and the outward appearance of the app. I also don't want my users to have access to the configuration options that come with the Tomcat system tray icon.

Fortunately, there are some reasonable options out there, and the best I have found is Launch4j. I'm going to describe the broad strokes of how to create a Launch4j exe that starts up Tomcat for you. Over all, I've been very pleased with the results, and would recommend this to anyone who is deploying primarily to Windows users and wants to deploy their app in a professional-feeling package.

I should also note that this entry by Roshan Shrestha from several years back provided a very helpful starting point. I've updated and simplified what he did there, and I also opted to use Launch4j instead of JSmooth as it seems to be more up to date.

Overview

In short, my requirements were:

• Provide a native .EXE that launches Tomcat
• Appear in the taskbar like a regular .EXE
• Take the user to a download page if they don't have a JRE installed
• Open the user's browser after the server is launched from the EXE
• Allow me to provide a custom icon and splash screen
• In general, just 'feel' as much like a native Windows app as possible

There are 5 basic steps you need to do to get this going:

1. Create a main class to start Tomcat
2. Ensure that the Catalina environment gets initialized
3. Create a Launch4j Configuration
4. Create an icon and a splash screen
5. Set up an Ant script to put it all together

I describe these steps in more detail below.

Create a main class to start Tomcat

At the end of the day, Launch4j just invokes a main method and then dies. In our main method, we want to start Tomcat and then start the user's browser pointed at the new Tomcat instance.

The listing of TomcatLauncher.java below shows what you need to do. Note the two key dependencies: on Tomcat's Bootstrap class for starting tomcat, and on ejalber's very useful BrowserLauncher for starting the user's browser. (You'll need to download the BrowserLauncher source and integrate it into your build).

Ensure that the Catalina environment gets initialized

The default Tomcat startup scripts do some mucking around to set various system properties when Tomcat is actually launched. We need to ensure that our launcher emulates this mucking around. For the most part, the two main things you need to worry about are setting catalina.home and logging system properties.

I've found the easiest and most flexible way to set the required properties is to use Launch4j's ini file feature, which lets you specify arguments to the launched JVM in an .ini file. This file is read every time the launcher is executed (NOT when it is built), so it also allows you to tweak VM settings (such as heap size) in the wild. The file has to be in the same directory as the lancher .exe file and have the same name but with an extension of .l4j.ini rather than .exe.

The .ini file can use a number of useful variables such as EXEDIR that we can use to ensure that the right directories get found. See the mylauncher.l4j.ini listing below for an example.

Create a Launch4j Configuration

Launch4j needs an XML file that describes how to create the launcher at buildtime. There isn't a whole lot to this - it basically tells it where to find jars and icons and the name of the class to run. Launch4j comes with a lot of documentation and even a fancy GUI for setting this up, so I won't belabor this part of it. See the appended launcher.xml for an example and a few notes on potential gotchas.

Create an icon and a splash screen

The splash screen is completely native and for this reason (apparently), it *has* to be a bloated .BMP. The .BMP

The icon file has to be a Windows .ico file. There are lots of ways to make this. On MacOS, I found a very handy one called IcoMaker.

Set up an Ant script to put it all together

Launch4j includes a nice ant task that lets you automate the building of your .exe (even on MacOS!). Again, I won't belabor the details here, but you need to:

• Compile your classes and jar them up.
• Run the Launch4jTask, pointing it at the launch4j binaries, splash BMP, and ICO files
• Copy your jar, the ini file, and whatever else into your deployment directory

That's pretty much all there is to it. As I say, so far I have been very pleased with they way this has worked. Users don't even know they are running a Java application, and I retain a lot of control over the environment that they're running in.

Another thing that I've done that completes the illusion is to create a windows tray icon using the com.jeans.trayicon package - I'll add an entry about this later.

Code Listings

launcher.xml

<launch4jConfig>

<!-- Maybe just me, but I like keeping the jar outside -->
<dontWrapJar>true</dontWrapJar>
<headerType>gui</headerType>

<outfile>mylauncher.exe</outfile>
<errTitle>An error has occurred</errTitle>
<chdir>.</chdir>
<priority>normal</priority>
<downloadUrl>http://java.com/download</downloadUrl>
<supportUrl>http://www.mycompany.com/support</supportUrl>

<!-- This is resolved at BUILDTIME -->

<icon>images/desktop_icon.ico</icon>
<classPath>
<mainClass>com.mycompany.TomcatLauncher</mainClass>

<!-- You need to use backslashes here -->
<!-- These are resolved at RUNTIME -->

<cp>%EXEDIR%\lib\mylauncher.jar</cp>
<cp>%EXEDIR%\tomcat\bin\bootstrap.jar</cp>
</classPath>
<jre>
<path>jre</path>
<minVersion>1.5.0</minVersion>
<maxVersion></maxVersion>
<dontUsePrivateJres>true</dontUsePrivateJres>
</jre>
<splash>
<!-- This is resolved at BUILDTIME -->

<file>images/splash.bmp</file>
<waitForWindow>true</waitForWindow>
<timeout>60</timeout>
<timeoutErr>false</timeoutErr>
</splash>
</launch4jConfig>

build.xml

...
<target name="exe">

<taskdef name="launch4j"
classname="net.sf.launch4j.ant.Launch4jTask"
classpath="${launch4j.dir}/launch4j.jar:${launch4j.dir}/lib/xstream.jar" />

<launch4j configFile="launcher.xml"
outfile='${build.dir}/mylauncher.exe'
bindir='${launch4j.dir}/bin'/>
</target>

mylauncher.l4j.ini

-Dcatalina.home="%EXEDIR%/tomcat"
-Djava.util.logging.config.file="%EXEDIR%/tomcat/conf/logging.properties"
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Xms512m
-Xmx512m

TomcatLauncher.java

package com.mycompany;

import org.apache.catalina.startup.Bootstrap;
import edu.stanford.ejalbert.BrowserLauncher;
import javax.swing.JOptionPane;

public class TomcatLauncher implements Runnable {


// =========================================================================
// Constants

// url that the launcher will open in a browser.  you might need to make
// this more flexible (e.g., passed in via args or configuration).
private static final String START_URL = "http://localhost:8080";

// =========================================================================
// main() - start Tomcat in a separate thread so launcher thread can die

public static void main(String[] args) {
TomcatLauncher tl = new TomcatLauncher();
Thread t = new Thread(tl);
t.start();
}


// =========================================================================
// Runnable imlementation - launch Tomcat


public void run() {
registerShutdownHook();
startTomcat();
showStartPage();
}

// =========================================================================
// Private methods that do the real work

/**
* In case Ctrl-C or other unexpected shutdown...
*/
private void registerShutdownHook() {
this.shutdownThread = new Thread("shutdownHook") {
public void run() {
if (readyToExit == false) {// abnormal exit
readyToExit = true;
try {
if (bootStrap != null) bootStrap.stop();
} catch (Exception e) {
e.printStackTrace();
}
// do any other cleanup work, e.g. closing log files
} catch (Exception e) {
e.printStackTrace();
}
}
Runtime.getRuntime().addShutdownHook(this.shutdownThread);
}

/**
* Start Tomcat
*/
private void startTomcat() {
try {
log("Starting Tomcat...");
bootStrap = new Bootstrap();
bootStrap.init();
bootStrap.start();
log("...Tomcat started.");
} catch (Throwable t) {
t.printStackTrace();
JOptionPane.showMessageDialog(this,
"Error starting Ledgerscape server.", title,
JOptionPane.ERROR_MESSAGE);
}
}

/**
* Open the browser
*/
private void showStartPage() throws Exception {
try {
BrowserLauncher.openURL(START_URL);
} catch (Exception e) {
e.printStackTrace();
JOptionPane.showMessageDialog(this, "Error starting browser.", title,
JOptionPane.ERROR_MESSAGE);
}
}

private void log(String msg) {
// FIXME you will probably want to make this write to a file
System.out.println(msg);
}
}

I blogged about Scriptella a while ago. It's a simple and elegant framework for declaratively manipulating result-set style data (typically but not necessarily from a database). Fyodor also seems to have updated the reference section with some nice diagrams that explain the concepts very clearly.

A quick google search turned up Rob Mayhew's AntSpell task. It works as advertised, no muss no fuss. 10 minutes later, I had it integrated into my build with a custom dictionary. Thanks Rob!

Fortunately, I did find one which is meeting my needs in a very elegant way: Scriptella

Their website says

"Our primary focus is simplicity. You don't have to study yet another complex XML-based language"

I second that emotion; the last thing I need to do is deal with another misguided attempt to stuff procedural logic into XML.

At first glance, though, Scriptella looks like nothing if not "yet another complex XML-based language:"

<etl>
<connection driver="$driver" url="$url" user="$user"
password="$password"/>
<script>
<include href="PATH_TO_YOUR_SCRIPT.sql"/>
-- And/or directly insert SQL statements here
</script>
</etl>

Do not be fooled. That is just a shell, and it's about all of the XML you will have to write.

The Rules

For the impatient, here is a sketch of a spec for how you write a Scriptella script:

• You use XML to build a simple skeleton for nesting chunks of declarative languages (typically SQL but not always)
• Outer chunks are responsible for generating a result set - (a list of hashmaps, essentially)
• Inner chunks are processed iteratively for each row in the result set generated by its enclosing chunk
• Inner chunks do something useful with each row (such as perform an INSERT). They can refer to members of the current result set row by name.
• Chunks can be nested to any depth (i.e., inner chunks can also be outer chunks)

A Simple Example

That may sound a little confusing, but it's actually quite elegant and powerful. A simple example goes a long way.

Say you have an Employee table in Database A that you want to copy into Database B, row by row. In scriptella, it would look something like this:

...
<query connection-id='DatabaseA'>
SELECT ID, FIRST, LAST FROM EMP
  <script connection-id='DatabaseB'>
INSERT INTO EMPCOPY VALUES (?ID, ?FIRST, ?LAST)
 </script>
</query>

When processed, this Scriptella script would select each row out of EMP in Database A and insert them into EMPCOPY in Database B. The nested 'script' element gets processed once for each row, and the '?' parameters get filled in with values from each row.

It's as if you wrote Java code that opened two JDBC connections, executed a query on one, iterated through the result set, and dumped each row into a prepared INSERT statement opened on the other connection.

At this point, you can knock yourself out - create tables from scratch, write arbitrarily complex SELECT statements, create and drop tables as needed, whatever. You can also use this as a way to write queries that generate queries - a more dialect-neutral alternative to the various flavors of dynamic SQL (i.e., queries that generate queries). You have the full power of SQL at your disposal and Scriptella won't get in the way unless you ask it to.

Beyond SQL

Things start to get really interesting when you consider that Scriptella is designed to accommodate any language, not just SQL. For example, something like this can be used to dump data from a table to System.out:

...
<query connection-id='DatabaseA'>
SELECT ID, FIRST, LAST FROM EMP
   <script connection-id='janino-java-connection'>
String employee = get("FIRST");
System.out.print("Found an employee named "+employee);
    </script>
</query>

Any language that you have a JSR-223 driver for can be used. You can have scripts that generate data for processing by enclosed XML chunks as well as processing data from enclosing chunks

Wrapping Up

There are a lot more (and better) examples on the Scriptella website. The short of it, though is that Scriptella gives you a simple, elegant, declarative mechanism for:

• Moving data between databases
• Writing dynamic queries
• Sprinkling in procedural logic when you need it

Most importantly, it does all of this with as little intrusion as possible. Most of the time, I want to do the heavy lifting with SQL, and Scriptella doesn't get in the way of that. But when I need to add some procedural logic or do something trickier, Scriptella is there.

JPA FAIL! So, Mr. Patrick Linskey, who knows something about such things, points out that the standard actually does provide a framework for dynamic PUs: javax.persistence.Persistence.createEntityManagerFactory.  You still have to plug in vendor-specific properties, but this is a much better way to go.  I'm not actually sure how I could have been led so far astray - what I knew at the time I knew from perusing the hibernate forums...

I gotta say, again, that I really think JPA is great. It took almost a decade, but Sun got pretty much everything right. I love just putting annotations everywhere, and anyone who thinks that is a bad idea is a dodo-head. Die XML, die!

import org.hibernate.ejb.Ejb3Configuration;
import java.sql.DataSource;
import java.util.Properties;

// ...

@SuppressWarnings("deprecation")
public static EntityManagerFactory
createEMF(DataSource ds,
Class[] entityClasses,
Properties props)
{
Ejb3Configuration ejb3conf =
new Ejb3Configuration();
ejb3conf.setDataSource(ds);
//
// set any fixed properties you want
ejb3conf.setProperty("hibernate.dialect",
org.hibernate.dialect.HSQLDialect);
ejb3conf.setProperty("hibernate.show_sql",
"false");
ejb3conf.setProperty("hibernate.hbm2ddl.auto",
"update");

if (props != null) {
Set<Entry<Object, Object>> entries = props.entrySet();
for (Entry e : entries) {
ejb3conf.setProperty(
e.getKey().toString(),
e.getValue().toString());
}
}
for (int i = 0; i < entityClasses.length; i++) {
assert entityClasses[i] != null;
ejb3conf.addAnnotatedClass(entityClasses[i]);
}
return ejb3conf.createEntityManagerFactory();
}

Scary, huh? It's you on the internet! Well, not really - its just looping back locally. But still. I blatantly stole this cool trick from this guy.

(For those not on an iSight equipped Mac: the above box shows the video feed from the Mac's built-in webcam).

But seriously, I am a little concerned about the iSight. It's fun and has the geek factor and everything, but there also are security and privacy issues that I think Apple seems to have pretty much ignored.

This morning, I was on a WebEx. The WebEx software and the iSight both decided that video of me should be transmitted. The WebEx software provides no way to disable this. Apple provides no way to disable the camera. A control panel? A shutter? Better yet, a removable camera? A BTO option to skip it entirely? Something? Please?

And yes, I do know I could fix this with a 1 sq cm of electrical tape. But then how am I going to get that consulting gig at NORAD if they won't let me through the 20-foot-thick door with my machine?

I chose these because they seem to be the projects which have the most momentum. (I'm not proud - it pays to follow the herd in this kind of thing). I can't claim to have written 'real' applications with any of these yet, but in each case I at least got so far as prototype that presents master-detail views that update a database.

A quick note about me: I've been a longtime skeptic of the whole Ajax thing. My experience writing the early console for WebLogic server left me traumatized by javascript. The latest generation of tools may be changing my mind.

So, without further adieu, the nutshells:

At the end of the day, the answer is the same as it almost always is: it depends on the app. Seam if you have to do traditional pages, GWT if you're building GoogleMaps, and Echo if you're doing mid-size/J2EE apps.

At the moment, I'm mostly in the last category, so my choice is clear: Echo all the way

You can get it here. Lemme know if it works for you.

Props to thenopp and especially Ralf for helping me track down some bugs.

Well, it seems to have happened again. Not sure exactly when it happened, but it may well have been right around the time I upgraded to 1.4.8. But it also has been running hot for long stretches of a time, lately - maybe that's the cause. Ugh, who knows?. Back you go to Apple for a new logic board, little PowerBook.

This all got me wondering how I could make my machine make an assertion at startup about how much memory it has. That in turn got me googling, which in short order produced a handy answer courtesy of one Matt Comi. Thanks, Matt.

And yes, I do need to get a new machine. I have needed one for months, but just can't bring myself to do it knowing that the Core2 Duos are just around the corner. Any day now, right?

And to top it all off, he admits mistakes. At this year's 'puzzlers' quiz at JavaOne last week, he paused to offer everyone a big mea culpa for a design decision he made which later turned out to be wrong.*

How often do you see a THOUGHT LEADER do that? And in front of thousands of people, no less? I'm a fan for life.

* The mistake was that java.util.Arrays.asList() was updated to take varargs. This means it's now ok to pass an array of primitives to asList, but you will get a (perhaps) astonishing result: a List containing a single entry (your array of primitives).

• Overall, and as reported elsewhere, the energy level was quite a bit higher than in recent years. Attendance was quite good.
• I saw an inordinate number of 'Java couples' strolling about with matching orange backpacks. About 30 by my count. Was this just me? Male & female checking out the latest on NetBeans and Ajax hand-in-hand or in an otherwise not very we're-just-co-workers kind of way.
• Crowd control for the sessions sucked even worse than usual.
• I guess scripting language support is the next big thing. Seems useful, but only to a point. The only use case I buy is to make it easier to provide applications with end-user scriptability. But now they're talking about VB. They're talking about inline XML. I am a little afraid.
• Ajax was all the rage. This still just gets a big WTF from me.

If I were to hazard a guess as to W exactly that last F is, I would say this: Java is done. I don't mean that in a bad way. I mean that in the sense of all the big mountains having been climbed in J(2?)[S|E]E

They've finally cleaned up most all of the crap out of the language. They've cleaned the crap out of the persistence APIs. Actually, most all of the EE frameworks seem to be in pretty good shape now. We're finally starting to get mature development tools. Even Swing is becoming a viable way to write desktop applications.

The fact that a crock of hype like Ajax can be the hot thing at JavaOne speaks volumes to how far Java has come. The sense I get is that there just isn't that much to do to the Java platform. Maybe we don't need the JCP anymore, not because it's too slow, but because it's going to run out of useful work to do?

There is one voice of reason (with a lame handle, "Dude Guy") amidst the madness, though:

This is just [the] good old problem of representing unknown information on a database, on which many people have written a fair amount, of which many, many more have read nothing. One classic problem: there is a difference between using a third value to represent "the real-world entity represented by this record has a value for this, but it is unknown," "the entity represented could have a value for this in the real world, but it actually happens not to have one," and "the real world is such that the entity described can't have a value for this." In short, people tend to systematically confuse data that represents facts about the real-world entities [with] questions and metadata about the completeness of the representation.

Thank you, Dude Guy, whoever you are.

For an added challenge, three movies in each list have been whited-out. Give up? Click-drag your mouse over the lists to reveal the hidden answers!

	10 Best Popcorn Movies	10 Best Pictures
1	Harry Potter and the Goblet of Fire	Harry Potter and the Goblet of Fire
2	King Kong	Brokeback Mountain
3	Star Wars: Episode III	King Kong
4	Batman Begins	Sin City
5	Serenity	Star Wars: Episode III
6	Sin City	Serenity
7	Mr. & Mrs. Smith	Batman Begins
8	Wedding Crashers	Crash
9	War of the Worlds	Walk the Line
10	The Chronicles of Narnia	The Chronicles of Narnia

Yeah, that's what I'm talkin' about! The best thirteen bucks I've spent in a while - a portable laptop pad that

• helps keeps a powerbook cool
• protects surface of the table and the PB's underside
• provides a handy swivel motion
• produces a slight tilt that i think is actually more comfortable for typing