These are tumultuous times for lisp. Once the bastion of language academics and hackers and shunned by those of industry, it's received much lip service from successful dot commers, most notably Paul Graham. Lisp has been around a long, long time (its the second oldest language still in use, behind fortran), and its gotten very mature in that time. Industry languages continue to steal features from lisp, and some modern languages like python and ruby are more lisp than C. Still, lisp does not have the popularity one might expect. What is it that's holding it back while python prospers?
Lets start with what it gets right. It is said that lisp is very expressive, allowing extremely complicated ideas to be expressed succinctly (and hopefully quickly) by those fluent in the language. That's a rather heady idea, especially considering it is being applied to a practical industry. What this means in practice is that a great programmer is less limited by lisp than C++ or Java. This is really important because there are no natural limits on the productivity of a programmer. Unlike artisans and construction workers, there are no physical processes that limit how fast he can accomplish a task. The best person of another trade might be hundreds of times better than the novice, but they are limited by nature. Take away that limitation, and a handful of the best people could accomplish amazing things.
What makes lisp expressive aren't complex features like Object Orientation, but powerful simple ones like closures. The simplicity of these features let you build new complex ones as times change. Aspect Oriented Programming was part of the original Object Oriented lisp system based on closures. Continuations are a great way to deal with the non-linear flow of web applications.
Many of these powerful features have been adopted by new languages: Python can generate closures and has list comprehensions. Still, they don't contain all of lisp's power, and yet a search for open-source lisp applications on sourceforge returns 345 results to python's 6,571. Where are the hordes of elite programmers looking to change the world? Its easier to write things in lisp, right?
What lisp is getting wrong, opening the door for other lisp-like languages, is the new user experience. This means documentation, discoverability, sample code, libraries, and books. Its forums with solutions and smiling people. Unfortunately the lisp community is confident that their horse will win in the end, and they're just not willing to put in the effort. Other languages whose success is not so certain are more motivated to please. Python is one guy's foolish attempt to change the world, and you know that he spent a lot of time working to that end.
What lisp needs is is dedicated people behind it, laying the foundation of what people want. Specifically, people who are not currently lisp users. Given a python executable, some syntax, and knowledge of "import", "dir", and "help", anyone can learn python. Lisp should also be easy for newbies; there is even less syntax! SLIME is a great package, but please, stop hurting the cause... people who don't use Emacs are terrified by it.
Even the first step of a newbie has too many decisions: picking a lisp interpreter. There are simply too many of them, and their features are often inconsistent and incomplete. Perl, python, and ruby all managed to get one interpreter running on many platforms fairly uniformly. I don't worry about the networking or threading model inconsistencies between python on my mac and python on my linux server.
They also all have package managers that just work. This is a critical feature that just needs to work. CPAN really set the bar for package mangers in general. Asdf-install is okay, but it can be inconsistent across runtimes and packages don't always install correctly.
There are simple things as well. I am constantly irritated by the lack of prompt features in lisp interpreters. I'm lucky if they even recognize control characters... command history is a blessing. I haven't seen one that has any manner of auto-complete (something based on list-all-packages and do-symbols would be fine, thank you).
Its the unfortunate nature of lispers to work on the really long term things which are totally useless to their contemporaries. While I really appreciate that Paul Graham has decided to work on his own version of the language, I suspect that his lisp oriented incubator company would benefit more from a simple, usable IDE for sbcl and openmcl.
Ultimately, I am hopeful. Lisp is something that I believe in, and I'll keep trying to do more with it. If any language is ready to transcend old habits, this one is. Most things have already been tried in it once (distributed computing, hardware implementation, graphical editing), the next big thing is probably already in there somewhere waiting to be harnessed.
I've been working at a new gig in Boulder, Colorado. We're writing an earth rendering engine, a la Google Earth. Why bother? Unlike the competition, we're going to be putting lots of our imagery on hard drives so that you can take it off the net: in your car, on a plane, in your backpack. You can sync changes when you get home, or use a cellular connection for low bandwidth data like local searching and geolocating your friends. If you've wanted it, hopefully we're on it (but feel free to send suggestions or leave comments).
