Often when someone says a particular programming language is bad, they are referring more to the common practice associated with that language than the language itself. Many times they are really complaining about their own poor programming habits more than the specific language. Sometimes these habits are shared by the entire culture built around a particular language.

Perl is a good example of this. People complain about how difficult it is to read and then proceed to write awful unreadable code. Perl can be very readable, but its terseness makes it easy for people to write huge lines of code that do 10 or 11 different things. You can do the same thing in Java, but most people try to avoid a single line that is 500 characters long because it is a pain to scroll back and forth sideways to read the code.

Sometimes the lack of a particular restriant can inspire horrible code.

Based on the number of edits, Wikipedia appears to be written by a small number of people.  Aaron Swartz did some testing and came to the conclusion most content in the final revision comes from people who don’t even have a login.

I have been looking for a way to determine the percentage each author contributed to a particular page in Wikipedia or MediaWiki, something I assumed would be trivial, but it turns out is much more complicate. For performance reasons, MediaWiki (the software that powers Wikipedia) saves the entire text of a new revision–not just what has changed so every edit results in a completely new copy of the page being saved.  This means to see who contributed what, requires going back through each edit and comparing it to see what was included in the final version.

I started looking at Aaron’s method to see if it might be useful.  What he did is briefly described here. To the best of my understanding this is the basic process he used:

Find the longest matching string between the first revision and the final revision from the Wikipedia dumpfile.  Mark that string in the final revision as having come from the author of the first revision.  Continue this with the first revision and next largest matching string until there are no more matching strings that haven’t been marked.  Move to the second revision and repeat the process.  When you finish, you should have a version where every character is marked based on where it came from.

So lets look at an example:

Bob Revision 1: The fox jumped the hound.
Joe Revision 2: The quick brown fox jumped over the lazy hound.

So our final version is:

The quick brown fox jumped over the lazy hound.

Now lets find the longest matching string between the final version and Bob’s initial edit.  We’ll mark Bob’s contributions in Red:

The fox jumped the hound.
The quick brown fox jumped over the lazy hound.

Ok so that is the longest string, now lets find the second longest:

The fox jumped the hound.
The quick brown fox jumped over the lazy hound.

Repeat:

The fox jumped the hound.
The quick brown fox jumped over the lazy hound.

And once more:

The fox jumped the hound.
The quick brown fox jumped over the lazy hound.

Going through the same process marking Joe’s revision in blue produces:

The quick brown fox jumped over the lazy hound.

So we can easily see that Joe contributed 18 non space characters and Bob contributed 21 non space characters that made it into the final revision.  This is just fine if you are simply adding information.  It gets a bit more tricky when you are removing words because parts of words that have been removed will match words that have been added later.

Consider this scenario:

Bob Revision 1: The icky quaint rowdy fox jumped over the hound.
Joe Revision 2: The quick brown fox jumped over the lazy hound.

Now if we do the same process we get:

The quick brown fox jumped over the lazy hound.

Even though the words “icky”, “quaint” and “rowdy” added by Bob have been removed in the final version, they are still matching parts of words.  ICK of “icky” is matching the last part of quICK.  The QU of “quaint” is matching the first part of QUick, etc.

This approach is strongly biased toward the person who started the article or contribute early on–particularly if they added a lot of text that was later removed.  What would happen if the person who originally edited the article also pasted in a copy of the alphabet several hundred times at the bottom of their text?  It would match everything added later regardless of who added it. Now people probably aren’t doing that, but if they add a bunch of text that eventually gets removed, their removed text will still match a lot of text in subsequent revisions.

Still, this approach isn’t unreasonable and probably gives fair results as long as someone isn’t specifically trying to game the system.  Aaron’s method of recursively looking for the longest string is important if you need to see who did what.  If you just need to know how many characters each person contributed (and you are fine with the level of accuracy discussed above), there is a much more efficient approach.

More efficient method

The trick is to realize that this method is going to attribute any character in the final revision to the earliest revision to introduce that character.  So if revision 1 has three a’s, three a’s of the final revision will be credited to the author of the first revision–regardless of where they occur. Because of this, there is no advantage of recursively matching the longest string unless you are trying to produce an annotated version showing who wrote what block of text. Even then  you run into the problem shown above where subsequent edits don’t get full credit.

In other words, we can get the same results simply by counting the number of times each letter appears in each revision starting with the earliest revision.  That revision then receives credit for the occurrence of those letters in the final version as long as those letters haven’t been already credited to another revision.

Here is a simple example:

Revision 1: AB BA BAB
Revision 2: AB BAB BAB BABA

Revision 1: A = 3 B = 4
Revision 2: A = 5 B = 7

So the first version gets credit for:
A = 3 B = 4 total: 7 characters or 7/12th of the final version

Revision 2 gets credit for:
A = 2 B = 3 total: 5 characters or 5/12ths of the final version

More accurate methods

As shown, this approach gives an extraordinary amount of weight to early contributors–particularly if they are verbose–regardless of how much of their actual content makes its way into the final version. Its greatest strength is that it handles cases where text is moved from one position to another.  It gets this strength at the expense of giving “false credit” to people who contribute in earlier revisions.

Levenshtein Distance

Another possibility would be to use the Levenshtein Distance.  This basically counts the number of changes necessary to convert one string into another. I did some testing using Levenshtein Distance doing the following:

Starting at the oldest revision, calculate the Levenshtein Distance from the final revision.  This number represents how many characters from the revision appear in the final.  Moving to the next oldest revision, calculate the Levenshtein distance, but discount any credit already given to previous revisions.

This handles inserts, where words are inserted between existing words, but it doesn’t handle situations where words, sentences or paragraphs have their positions substituted.  If you have ABC, the revision that changes it to CBA gets credit for the contents of C and A even though all it did was move text around.

History Flow and the sentenced based method

To get a more accurate picture you have to use a slightly longer unit than characters.  The two simplest ways would be to use words or sentences instead of individual characters.   IBM has done some analysis using a tool called History Flow that uses the sentence as the fundamental unit. As they point out, a revision that adds a comma will get credit for the entire sentence that contains the comma.

Line based approach

Jeff Atwood uses the “line” as his fundamental unit. This is a very reasonable approach if you are working with code or something where you are likely to have a lot of new lines.  However, for long paragraphs it is a bit problematic.  Either they get treated as a single line and you have the same issue with the comma as IBM’s method applied to an entire paragraph or you break the paragraph into lines at specific points and adding in content can reorder the entire paragraph making it all appear new.

Word based approach

A good balance might be to use individual words as the fundamental unit being compared.  This drastically reduces the “false credit” problem associated with character based matching while minimizing the “comma problem”. There is still going to be a bit of “false credit” especially for common words. If someone writes 500 words to start an article, all of their original text is deleted and new text added, they are going to get credit for a number of words like “the”, “and”, etc.  If what they wrote was one topic, they will get credit for even more because they are likely to have used a lot of keywords that will be in the final revision.  Simply pasting in the dictionary a few times would give them significant credit for text that will not appear in the final revision. Still it represents a very reasonable approach, particularly if people aren’t trying to game the system.

Spam and methods

In this type of analysis of a Wikipedia or a different MediaWiki, one crucial thing to consider is spam. Character based and word based analysis is going to be heavily skewed by spam entries–even if they are immediately reverted.  Sentence based approaches are probably going to be more accurate if revisions contain spam while word based methods are likely to be more accurate in closed systems where spam isn’t an issue.

People Found This When Looking For:

• wikipedia (1)
• wikipedia algorithm (1)

I was using Selenium to automate some tasks beyond testing and needed to set it up to run with a cron job. A shell script calls the appropriate Maven command, but I kept getting the error:
Error: no display specified

The fix was to add this to the script before calling Maven:
export DISPLAY=:0

Evidently when Selenium is started from cron, it doesn’t know what display to use. This code tells it to use display 0 and it runs normally.

People Found This When Looking For:

• Error: no display specified (17)
• selenium Error: no display specified (13)
• selenium no display specified (5)
• selenium display (3)
• error no display specified selenium (3)
• selenium cron jobs (2)
• no display specified (2)
• selenium DISPLAY Error: no display specified (1)
• selenium display=:1 (1)
• selenium error (1)

If you are trying to use Rome and Rome Feed Fetcher, you may want to change the user agent.  However, the following will not change the default user agent:

FeedFetcher feedFetcher = new HttpURLFeedFetcher();
feedFetcher.setUserAgent("User Agent 007");
SyndFeed feed = null;
feedURL = new URL(rssUrl);
feed = feedFetcher.retrieveFeed(feedURL);
List entries = feed.getEntries();

To change the user agent you must use the InfoCache as shown:

FeedFetcherCache feedInfoCache = HashMapFeedInfoCache.getInstance();
FeedFetcher feedFetcher = new HttpURLFeedFetcher(feedInfoCache);
feedFetcher.setUserAgent("User Agent 007");
SyndFeed feed = null;
feedURL = new URL(rssUrl);
feed = feedFetcher.retrieveFeed(feedURL);
List entries = feed.getEntries();

Otherwise the User agent is set to “Java/1.5.0_04″. This is odd because the default client for Rome is “Rome Client (http://tinyurl.com/64t5n) Ver: 0.7″. It seems like an attempt to change the user agent without having a HashMapFeedInfoCache will result in changing the user agent, but somehow it reverts to “Java/1.5.0_04″ instead of whatever you set it to.

People Found This When Looking For:

• rome client http://tinyurl com/64t5n (1)
• user agent rome (1)
• user agent Rome Client (1)

To invent, you need a good imagination and a pile of junk.

– Thomas Edison

This is what is so facinating about programming. Your “pile of junk” consists of digital assest instead of physical matterial, so the raw materials are not limited by the normal laws of supply and demand. In software, you are limited only by your imagination.

Brett Porter has hacked together a tool that will let you use a CVS or Subversion repository as your maven repository.

Brett Porter – Storing your Maven Repository in CVS/Subversion
It’s pretty rough, but is a working prototype that makes Maven 1.1/2.0 downloads a checkout/update, and deploy is an add/commit. I see this would be useful for snapshot repositories, where you could use one filename instead of transforming the version, so getting the latest would literally be an svn update.

If you are using Subversion with Apache, it is pretty easy to achieve most of this. The problem that I’m faced with is the fact that Maven can’t handle repositories that use SSL and a login.

Currently, I’m using a separate server to host our Maven repository because the Subversion server is using SSL. I hope that Maven will eventually come up with a way to work around this, but right now it looks like most of their efforts are being spent on Maven 2.

People Found This When Looking For:

• maven subversion repository (3)
• maven repository in svn (2)
• maven svn (2)
• using a subversion repo as a maven repo (1)
• use subversion as maven repository (1)
• svn post commit maven (1)
• subversion maven repository (1)
• storing maven repository in svn (1)
• maven using svn as maven repository (1)
• check out from CVS using maven version 1 1 (1)

I ran across this blog post that is probably typical of many people who are managing software projects.

Musings of a Software Development Manager » Blog Archive » CruiseControl Warnings
I get about 48 email messages from Cruisecontrol each day for one of our projects. This is not something I’m proud of since this situation has existed for at least 4 weeks now, we’ve had a broken build. The problem stems from some nasty functional tests that no one wants to investigate and we’ve sort of let our process slip.

There is a simple solution to this. Turn off the tests that are failing. People’s first reaction to this is “Oh no, we can’t turn off the tests! They indicate that something is wrong. Eventually we’ll have time to fix it.”

If you are actually going to fix it go ahead, but if something has been broken for more than a week, chances are no one is going to fix it any time soon. You should turn it off so it starts building without errors again.

Why is this better? If your team gets 10 emails each day saying that something is broken, they are going to ignore it. No one is really responsible for all of the problems, so no individual really works on fixing it. However, if the build is working correctly and someone checks in code that breaks a unit test and everyone gets and email, that person is probably going to try to fix it because it shows that he is responsible for the problem.

Think of it another way. Lets say I have 3 smoke alarms, 1 gas alarm, 1 CO2 alarm, and a flooded basement alarm in my house and they all sound pretty much the same. Now lets say that the flooded basement alarm goes off and I decide that it isn’t important enough to fix the cause…. So I just let the alarm go off. How likely do you think I am to notice if another alarm goes off once I get used to ignoring the first alarm.

If I’m not going to fix the problem, the best thing I can do is disable the flooded basement alarm until I have a chance to fix it. After a week of ignoring the alarm and nothing bad happening, I’m not suddenly notice it and decide I should do something about it.

One of the first things I did when I started at my current job, is go through and renamed every test that failed our automatic build process as “pending”. By the time the test would run, I had disabled about 2/3 of the tests. Since they were failing we ignored them anyway, so marking them as pending didn’t change anything. Before they were turned off, it would have been impossible to notice if one of the tests that were previously working broke because of a change.

Over time we’ve turned most of the pending tests back on one at a time as we’ve had more time to fix the code or fix the test.

When your tests fail, it should be unusual. I setup our builds to break if any test fails. I’ve got a lava lamp above my cubicle and everyone in the company know what it means. If something breaks people start asking the developers about it until it gets fixed.

This guy says that using Subversion and Cruisecontrol cut their costs by 92%. It would be interesting to see how he calculated this. Regardless, it is amazing how many quality tools are available for free now.

Why Subversion Rocks
In a recent study I performed on my development groups process improvement over the past 5 years, we found that we had cut the cost of managing our build and release process by approximately 92% by incorporating Subversion and related tools like CruiseControl, ViewCVS, and other custom software (and major process changes that accompany them) to integrate and automate our release management processes.

Here is an interesting discussion about the dangers of design patterns. Design patterns can be a great way to solve problems, but many times they become an excuse to over-architect solutions. Design patterns should reduce complexity. If they introduce more complexity, you are doing something wrong.

The myth of design patterns is that they will make it easy to solve problems that you haven’t through of yet. This sounds great in theory, but it is easy to end up “gold plating” the software by building in flexibility that will never be used.

The real power of design patterns is in solving existing problems. A lot of the times this means they will be introduced when you are re-factoring code. When the code starts becoming complex and a design pattern makes things simpler, then it might be a good time to use one.

I’ve inherited a project that makes extensive use of design patterns. In some places the patterns make the code simpler. For example the chain of command pattern is used in several places, so once you understand how the authentication piece works, you understand the principle behind several other key components.

Another pattern that shows up a lot is the factory pattern. While this is a great way to decouple code, it is only beneficial if you gain some type of benefit by having the code loosely coupled. I ran into an piece of code the other day where a factory created an object and I needed to know what it was. I had to look through 5 other java classes and two configuration files to figure it out. In this case, using a pattern made things much more complicated and there weren’t any significant benefits.

Patterns need to be used to solve real problems. If you try to use them to solve every potential future problem, they become a liability.

Technorati Tags: programming, software

Here is a scary scenario for a programmer. Unfortunately it is probably not too uncommon. Imagine you are starting a new job. Before accepting the position you ask all the right questions, but on your first day you discover the following:

1. The lead programer left two months ago. (You already knew this.) The only form of documentation is a few comments and javadocs. (You didn’t know this.)
2. While the project has it’s own CVS server, the backups were done without shutting down CVS and they had never been tested. The server crashed and all the backups were corrupt. There is a copy of the source code that someone had on their local hard drive, but no history information and there isn’t code of the last stable release.
3. The 30% of the unit tests fail. Another 30% have errors. Some of the tests are just wrong, others rely on having resources configured in a specific way. There are also some tests that aren’t actually testing code, but instead do things that populate the databases, etc.
4. The core framework used in the project is based on a 6 year old proprietary binary library that doesn’t have any documentation.
5. The bug tracking data was lost in the server crash. Since you don’t have the source code from a stable release, there are a bunch of little bugs, but none of them are documented.

Sound like a nightmare? Definitely. But it is also an opportunity. Here are the steps I would take under the above circumstances. I’d enjoy hearing what others have to say as well.

1. Implement Version Control. Personally I prefer Subversion, but regardless of what you use, getting it up and running should be your first priority. And just having it functional isn’t adequate. It needs to be running, automatically backing up, and you must test doing a restore from the backup data. If you haven’t tested your backup you are begging disaster to strike. (more on version control)
2. Setup an Issue Tracker. You need a place to keep track of the problems you find. You don’t want to remember a nasty bug when a customer calls to tell you about it. The issue tracker should be used for more than just bugs. It should be your central repository for everything the developers are working on. That means it should hold features, tasks, ideas for improvements, etc. Set milestones and assign issues to those milestones. If you are tracking estimated time for each issue, you’ll be able to set realistic schedules.
3. Document your setup. You need to know how to setup the build and run environment from scratch. I would suggest starting with a clean machine and setting everything up. If your project builds fine on the old developer’s machine, but not on any new machines you setup, you have a problem. You may find yourself spending several days hunting down all the dependencies and configuration settings.
4. Fix the Unit Tests. If tests are failing the developers will ignore them. All of your active tests have to pass every time you run them. If this means you go from 400 tests down to 50 that is fine. Tests that don’t pass can be prefixed with “pending” so they won’t run and you can re-implement them over time. If a test fails you need to consider the build to be broken.
5. Setup Automatic Builds. This doesn’t need to be anything fancy, but if code compiles and tests fine on your machine, but fails on others, you need to know about it right away. Ideally the code should recompile whenever there are changes in the version control system. Also you should make the build fail if the tests fail. When a build fails, you can have it send email, flash the lights, or turn on the overhead sprinklers, but you should not ignore it.
6. Document the code. Digging through someone else’s code and trying to make sense of it can be extremely painful. But you really can’t start writing your own code until you understand how the old code works. Even if you want to replace parts of the old code, you will need to understand what the code you plan to replace is actually doing. Make sure you record your findings, so they can benefit others. You may get called away to rescue another troubled project and you want to make sure you successors don’t have to start over.

Technorati Tags: programming, software, version control