Background

Most software projects rely on other software projects to function. For example, Nitrogen depends on SimpleBridge, Coverize, and Mochiweb or Yaws. Riak depends on Webmachine and Mochiweb.

In the name of simplicity and ease of use, it's generally a good idea for the parent repo to contain the source code of any sub-projects it uses.

But then you are faced with a decision:

Should the parent project include the full history of the sub-project as well?

You currently have three options, all with tradeoffs:

1. Remove revision history in your sub-projects by deleting the .git/.hg directory, and experience pain when you want to pull the latest updates or commit a patch on the sub-project.

2. Track the entire .git/.hg directory for each sub-project, and accept that the .git/.hg directory of your parent project will now be huge.

3. Use Git submodules (or Mercurial subrepos), and hope that you never have to include a Git project inside of a Mercurial project, or vice versa. Also, accept the unfortunate fact that your build process now requires a working Internet connection.

The Search for a Better Way

What do I really want out of sub-repo support?

• I want the sub-repos to seem like part of the parent project to the user, while still seeming like distinct repositories to me.

• I want to be able to work with the full history of the sub-repo, but I don't want this included in the parent repo, or sent out to anyone who downloads the code.

• I want an easy process for contributors to get the history of the sub-repos, so that they can commit patches.

Furthermore:

• I only want to use tested, core features of Git or Mercurial.

• I want the solution to be "cross platform", so that I can stick Git repos in Mercurial and vice versa.

• I want the solution to be simple to use and easy to understand for the most common use case. (In other words, hide complexity from the non power-users.)

Introducing subgit and subhg

After much thought and frustration, I think I've finally found a solution that meets all of my needs. It lets me work with the parent repo as I normally would, using the git or hg command. Furthermore, it gives me a different command to work with the sub-repos. Finally, it is cross platform, allowing me to mix and match Git and Mercurial projects. The only downside is that a contributor needs to jump through an extra hoop or two in order to get the history of a sub-repo.

In practice, the solution looks like this:

Change to the directory of a sub-project...

> cd ParentProject/SubProject1

Operate on the parent project...

> git status
...status info...

> git commit
...commit code...

Operate on the sub-project. Notice the use of the 'subgit' command...

> subgit status
...sub-project status info...

> subgit commit
...commit sub-project code...

Best of all, the subgit and subhg commands are just thin wrappers around git and hg. Each is about 25 lines of shell script.

Installation

To try this out on your computer:

1. Save the following scripts to a location in your PATH: subgit and subhg. (Remember to run chmod 755 to make them executable.)

2. Create a global excludes file for both Git and Mercurial, and add .subgit and .subhg to it.

   ~/.gitconfig

   [core]
   excludesfile = "~/.gitignore"
   

   ~/.gitignore

   .subgit
   .subhg
   

   ~/.hgrc

   [ui]
   ignore=~/.hgignore
   

   ~/.hgignore

   .subgit
   .subhg
   

That's all!

How Does It Work?

The core concept behind this approach is to store the version history of your sub-project in a non-standard directory, and then use special wrapper scripts when you want Git or Mercurial to operate against that directory.

In other words, your projects won't have a .git or .hg directory. Instead, they will have a .subgit or .subhg directory, which is not tracked by the parent repo.

|--ParentProject    <-- a Git repository
   |
   |--.git
   |
   |--SubProject1   <-- a Git sub-repo
   |  |--.subgit    
   |
   |--SubProject2
   |  |--.subhg     <-- a Mercurial sub-repo
   |
   |--src

This tricks Git or Mercurial into tracking the files inside of your sub-repo, even though the files actually belong to a different repository. (Normally Git won't track a Git repo nested inside of another Git repo.)

The wrapper scripts--subgit and subhg--do the heavy lifting to make Git or Mercurial use the .subgit and .subhg directories.

subgit simply searches upward for the closest parent directory that contains a .subgit directory. Once found, it calls git, telling git to use the .subgit directory for repository information. subhg works the same way.

Usage

To create a sub-repo that can be managed with subgit or subhg:

1. Inside of an existing, clone a repository like normal:

   git clone git://hostname.com/repository.git sub_project
   

2. Then, change to the new repository's directory and run subgit setup. This simply renames .git to .subgit:

   cd sub_project
   subgit setup
   

3. Now, test that it worked by viewing the parent's Git log and the sub-repo's git log:

   git log
   ...print out log for the parent project...
   
   
   subgit log
   ...print out log for the sub-project...
   

subhg works the same way.

Some Final Thoughts

First, this approach is intended for all of the projects out there using GitHub, BitBucket, Google Code, etc. as their main distribution channel. Most of these projects have a small group of contributors, and a much larger group of users.

If you distribute your project via a tar'd, gzip'd file, then this blog post is not for you.

Second, in order for other contributors to submit patches to the sub-project code, they will first need to obtain the full history of the sub-project. (Which makes sense, because the whole point of this was to NOT transfer the full history during a clone.)

As far as I know, the best approach to get the history is to just pull it from the sub-project's remote URL into a tmp directory:

git clone git://hostname.com/repository.git tmp
mv tmp/.git sub_project/.subgit
rm -rf tmp

-or-

hg clone http://hostname.com/repo/path/ tmp
mv tmp/.hg sub_project/.subhg
rm-rf tmp

Then, switch to the sub_project directory and checkout the right version. (This assumes your sub-project is in a directory named sub_project.)

Downloads:

• Source code for subgit.
• Source code for subhg.

• Watch the Video
• See the slides

Check out SlideBlast.com, a tool I created that lets you share and control a slide presentation on the web. SlideBlast was built using Nitrogen and Riak, and is an example of exactly how much you can do with the right tools and 1,000 lines of code. (Ok, it's more like 1,130 lines, but who's counting?)

The full source code is available on GitHub: http://github.com/rklophaus/SlideBlast

How it Works

1. Upload a .pdf or .zip file.
2. Share a link with your remote audience.
3. Start presenting. As you flip through slides, your attendees' slides change, too.

Components

SlideBlast runs on Erlang and uses the following projects:

• Nitrogen to create a slick, Comet-based user interface.
• Riak to store slide show data (including the images) with a flexible schema.
• Mochiweb as the underlying HTTP server.
• Ghostscript to split and convert .pdf files into images.
• Imagemagick to create image thumbnails.
• SyntaxHighlighter to transform uploaded code files into beautiful HTML.
  (Try uploading an .erl, .cs, .cpp, .js, .java, .sh, or .sql file!)

Why I Built SlideBlast.com

I built SlideBlast for my talk Nitrogen and Riak by Example, presented at the Erlang User Conference 2009 in Stockholm, Sweden. In the presentation, I briefly cover both Nitrogen and Riak, and then describe some of the techniques used to build SlideBlast. The video should be online soon, check it out.

    Error: Error parsing request: Malformed request: \
    com.squarespace.framework.ResourceNotFoundException: Invalid Object Reference (2)

After a little debugging, I realized that it works if you include the ID of your SquareSpace blog in the endpoint XML-RPC URL. For SquareSpace, your XML-RPC endpoint URL should look like this:

    http://www.squarespace.com/do/process/external/PostInterceptor#BLOG_ID

To figure out the correct value for BLOG_ID, go to the url of your SquareSpace journal (for example, mine is http://rklophaus.com/blog), view source, and search for "CURRENT_MODULE_ID". You will see a line that says:

    Squarespace.Constants.CURRENT_MODULE_ID = "<YOUR BLOG ID>";

This is the ID that you should use. The ID for this blog is 3142087, so my endpoint looks like this:

    http://www.squarespace.com/do/process/external/PostInterceptor#3142087

On November 9th, the Supreme Court heard oral arguments in re Bilski. (In re Bilski means "in the matter of Bilski." I looked it up for you.)

The outcome of this case determines the future of software patents.

What's a "Bilski"?

Back in 1997, Bilski and Warsaw filed a patent for managing risk in commodities trading through hedging. Using their process, for example, an oil company would be able to offer a locked-in rate to customers, and do some fancy purchasing behind the scenes to cover their asses so that if oil prices spiked, they would still turn a reasonable profit.

The US Patent Office (USPTO) denied the patent because it did not pass the "machine-or-transformation" test. This test says that for a business process to be patentable, it must either be implemented with a particular machine designed or adapted to carry out the process, OR it must transform/reduce an article into a different state.

The problem with the USPTO's decision is that it contradicted precedent set by the Federal Circuit Court.

"Ruh Roh"

Ruh Roh indeed, Scoob. The Federal Circuit Court previously held in State Street, that an invention is patentable if "it produces a useful, concrete and tangible result" and applies to a specific application, tests that Bilski's patent should have passed.

The Bilski team appealed to the Board of Patent Appeals, with no luck. So they appealed again to the Federal Circuit Court, who you would think would quickly overrule the Board's decision, citing State Street. The Circuit Court, however, agreed with the USPTO, saying in effect that they were wrong before, that the State Street ruling no longer applied, and that the machine-or-transformation test is now the one true test.

Interestingly, the Circuit Court included language in the decision stating more or less "holy crap guys, ever since the State Street decision we've been overwhelmed with business process patents, these damn patents are getting more and more abstract, and we need to streamline this process lest we get buried in paperwork."

When all else fails, talk to "Los Jefes"

Bilski appealed again, to the Supreme Court, which granted certiorari in June. (Certiorari means that they would hear the case. Again, I looked it up for you.) The surprisingly readable oral arguments occurred on November 9th, which brings us to the present day.

(See a timeline of these events.)

So, how does this relate to software?

Software was seen as patentable as a business process because it produced a useful, concrete and tangible result. By discarding the State Street precedent, the Federal Circuit Court questioned the foundation upon which business process patents, and thus software patents, are built.

If the Supreme Court upholds the Federal Circuit Court's decision, it means that all existing US software patents will be very, very suspect, and RMS will be a happy, happy man.

If the Supreme Court completely overrules the Federal Circuit Court's decision, it will mean we keep the status quo, a continuation of the muddy quagmire that surrounds software patents today.

This is a black-and-white summary of the outcomes, of course. It is likely that the Supreme Court will shoot for the gray area in between.

Don't Forget The Economy

As exciting as it may be, the Supreme Court won't shake things up too much. Technology is one of the few industries in the United States that hasn't been gutted by the recession. At best, sweeping away software patents will plant some serious concerns in the minds of investors. At worst, their concerns will be justified. I suspect that of all of the factors the Supreme Court considers, the health of the economy will be what guides them the most.

It's Broke, Fix It

Rather than rehash all of the arguments for and against software patents, I'll provide you with a Google search for software patent debate.

To be honest, I'm still not sure which side I support on the issue. I do know that software patents--in their current form--are broken, because:

• Software patents alone don't do much. You need a patent plus gobs of money.
• A good chunk of innovation in software industry comes from teams of smart people at startup companies.
• Startup companies don't have gobs of money, and if they did, they'd prefer to spend it building things, not fiddling with patents.

Can't wait to hear what the Supreme Court decides.

The Erlang Process Registry

Very often in Erlang, you want to create a process and give it a name. This allows you to send messages to that process from other processes without passing the Pid around. Erlang supports named processes using erlang:register(Name, Pid), but there are limitations.

• The Name must be an atom.
• A process can only have one, and a name can be associated with only one process.
• The local and global registries are separate. (A process is registered globally via global:register_name(Name, Pid).)
• There can be a lag between when a process is registered, and when it can receive messages under its registered name. ie: calling "register(myprocess, self())", and then immediately calling "myprocess ! message", can cause an error.

These limitations constrain the way Erlang developers architect and debug applications. With a more robust process registry capable of storing additional metadata about each registered process, a developer can better decouple the responsibilities of an application, or gather information about running processes for debugging or reporting purposes.

Some concrete examples:

Riak spins off a new process for each get, put, and delete operation. There is currently no way to tell how many data operations are running at any one time, or to look inside a process once it is running to see what bucket it will modify. Ideally, this could be attached as metadata to the process, and a separate monitoring application could fetch this data and display it on the Riak Web dashboard.

Nitrogen spins off a process for a user's session. Ideally, on subsequent requests Nitrogen could look up the session process in the registry based on the user's cookie. For now, a separate lookup structure is used.

GProc

Ulf Wiger, back in his days at Ericcson (he is currently the CTO of Erlang Training and Consulting, noticed that he and other engineers were repeatedly (re)creating their own process registry structures in the form of lists and lookup tables as a way of working around the limitations described above.

To solve the problem in a more general fashion, Ulf created gproc. (Well, he first created proc, and then--with the insight he gained--created gproc.)

gproc lets you:

• associate one or more names with a process, where a name can be any Erlang term
• attach one or more arbitrary properties to a process, where the key or value can be any Erlang term
• associate a counter with a process (with an optional rollup to aggregate counters across multiple processes), where the process name can be any Erlang term
• retrieve any of these keys and values using ETS style wildcard matching

gproc can be confusing at first because naming processes, applying properties, and creating counters are all accomplished using gproc:reg(GProcKey, Value). This was done in the name of speed, consistency, code-reuse -- all of these operations store data in the same ETS table, and so gproc makes functions double up.

Furthermore, to ensure thread safety, a process can only set it's own values. To put it another way, it's as if erlang:register(Name, Pid) became erlang:register(Name) and can only act on the current process.

Finally, it helps to think of "registering" a process as instead "the process sets a globally unique key". Similarly, "setting a property" means "the process sets a non-unique key".

You'll see what I mean in the examples below.

Start a shell with GProc

To get a gproc playground, run the following:

    svn checkout http://svn.ulf.wiger.net/gproc/branches/experimental-0906/gproc
    cd gproc
    erlc -I include -o ebin src/*.erl
    erl -pa ebin
    > application:start(sasl).
    > application:start(gproc).

Storing a value

So let's play. To store a value in gproc, call "gproc:reg({n, l, Key}, Value)"

Again, both Key and Value can be any Erlang term. With a type of n (for name) you can only associate one value with a key. Try it again, and gproc will throw an exception.

Removing a Value

To remove something from gproc, call "gproc:unreg({n, l, Key})". Alternatively, kill the process. gproc monitors all processes that have stored values. When a process dies, gproc automatically removes any values associated with that process. This can lead to some confusion when testing gproc in the Erlang shell. If you store some values in gproc, and then run a gproc operation that throws an exception, Erlang restarts your shell process, so you lose any gproc values associated with that process.

Though this can be confusing in the shell, this is exactly the behaviour we want from a process registry.

Retrieving a Value

To get your stored value, call "gproc:get_value({n, l, Key})". Note that this will only retrieve values stored by the current process. You can also use "gproc:get_value({n, l, Key}, Pid)" to get values from another pid.

Who put the cookie in the cookie jar?

In addition to your value, gproc stores the Pid responsible for the value. You can retrieve this by calling "gproc:lookup_pid({n, l, Key})" or "gproc:lookup_pids({p, l, Key})". (You use the former when working with registered properties and aggregated counters, and the latter when working with properties and local counters.)

Taking a step back

So you are probably wondering about the significance of the 3-tuple that we are passing to these functions. This tuple, of the form {Type, Scope, Key} tells gproc a few things about your value:

• Type is either 'n', 'p', 'c', or 'a' (n = name, p = property,  c = counter, a = aggregate counter)
• Scope is either 'l' or 'g' ('l' = local, 'g' = global)
• Key is any Erlang term.

So far, we have been using a type of 'n', a globally unique key, which gives us a good mechanism for uniquely naming a process. This is similar to erlang:register/2.

Specifying 'p', for property, lets us set a key/value pair for a process, but other processes can also set a name and value of their own.

Specifying 'c', for counter, lets you set and increment a local counter for a process.

And specifying 'a' for aggregated counter, lets you access the sum of all local counters by the same name.

Note: we're always using 'l' for local scope. See the 'G is for Global' note below on why I haven't written anything about global scope.

Registering a Process

To register a process in gproc, call "gproc:reg({n, l, Key}, ignored)". The 'ignored' atom is irrelevant here, but could be used to store some extra metadata about the process.

Now you can use:

• gproc:lookup_pid({n, l, Key}) or gproc:where({n, l, Key}) to return the Pid of the process.
• gproc:send({n, l, Key}, Msg) to send a message to the process.

These functions are also available for registered processes, but less useful:

• gproc:get_value({n, l, Key}) gets the value from the process that set it.
• gproc:select/1 to gets the value from any process (more below).

Setting a Property

To set a property, call "gproc:reg({p, l, Key}, Value)".

Now, you can use:

• gproc:get_value({p, l, Key}) to get the value from the process that set it.
• gproc:select/1 to get the value from any process  (more below).
• gproc:lookup_pids({p, l, Key}) to return a list of processes matching the key.
• gproc:send{p, l, Key}) to send a message to all processes matching the key.

Creating a Local Counter

To create a local counter, call "gproc:reg({c, l, Key}, Integer)".

Now, you can use:

• gproc:update_counter({c, l, Key}, Integer) to increment or decrement the counter.
• gproc:select/1 to get the counter value from any process  (more below).
• gproc:lookup_pids({c, l, Key}) to return a list of processes matching the key.
• gproc:send{c, l, Key}) to send a message to all processes matching the key.

Creating an Aggregated Counter

To create an aggregated counter, call "gproc:reg({a, l, Key}, Ignored)

Now you can use:

• gproc:get_value({a, l, Key}) to get the counter value from the process that set it.
• gproc:select/1 to get the counter value from any process (more below).
• gproc:lookup_pid({n, l, Key}) or gproc:where({n, l, Key}) to return the Pid of the process that set the counter.
• gproc:send({n, l, Key}, Msg) to send a message to the process that set the counter.

It's Really a Shared Dictionary

As you can see, gproc is really a shared dictionary with some limitations on who can store what when. Again, this leads to some confusion at first, but means that gproc can be applied to a wider variety of problems.

gproc:select/1

So far, we've glossed over gproc:select/1, and for good reason: it's complicated. But, it's also where gproc derives most of its power, so it pays to understand it.

With gproc:select/1, you can query the gproc process registry using roughly the same semantics as an ETS MatchSpec. Here is a quick (and far from complete) tutorial:

The basic form is gproc:select(Scope, MatchSpec).

Scope can either be 'all', 'names', 'props', 'counters', or 'aggr_counters'. MatchSpec is of the form [{MatchHead, Guard, Result}]. (Yes, it must be wrapped within a list.)

The MatchHead is a 3-tuple of the form {GProcKey, Pid, Value}. GProcKey is the 3-tuple key we've seen earlier, ie: "{n, l, Key}". By leaving an empty Guard ([]) for now, and using a catch-all Result (['$$']), we can already do some interesting things.

This will pull out every record in the process registry:

    MatchHead = '_',
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

This will pull out every record where value == value1:

    MatchHead = {'_', '_', value1},
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

You can specify a more complicated keys. This example will pull out any entry whose key would match "{myrecord, _}"

    Key = {myrecord, '_'},
    GProcKey = {'_', '_', Key}
    MatchHead = {GProcKey, '_', '_'},
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

MatchHead can also use '$1' variables, which lets you start using guards to shape your select. For example, this will pull out any gproc entry whose key is a list:

    GProcKey = {'_', '_', '$1'},
    MatchHead = {GProcKey, '_', '_'},
    Guard = [{is_list, '$1'}],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

See this gist for more examples. http://gist.github.com/188032.

Normally, you would be able to use the Result property of the MatchSpec to select out a subset of the data you want to capture. In gproc this is slightly broken. You can choose which result you want to pull out by setting result to a '$1' variable (for example, ['$1']), but currently you can only pull out one result from each match. In other words, ['$1', '$2'] does not work. (It only returns '$2'.)

GProc and QLC

It appears that GProc began to have support for QLC, which is pretty friggin' sweet, but unfortunately I could not seem to make it work.

G is for Global

GProc automatically detects whether gen_leader is available on the system. If it is, then gproc will run in "global mode" meaning that all of the functions above work across your entire Erlang cluster rather than on a single node, which is also pretty friggin' sweet. Unfortunately, gproc currently uses an obsolete version of gen_leader, so running gproc in global mode is not recommended.

Evaluation

GProc is extremely powerful, but the interface could use simplification. Convenience methods around gproc:reg/2 specifically shaped for registering processes would go far to help smooth the learning curve for newcomers.

In addition, gproc:select/1 has a few small bugs. The MatchSpec must be passed as a list, even though it will only work with one MatchSpec, and the Result portion of the MatchSpec can only be used with one '$N' variable.

In less than a month I'll be heading out to London to speak about Nitrogen at the Erlang Factory. I've booked a little extra time before and after the trip to go sip some tea, see Big Ben, high five the Queen, and ride in a London cab.

In the meantime, Nitrogen is undergoing some big changes. The improvements were inspired by conversations and presentations at Erlang Factory San Francisco as well as Growing a Language by Guy Steele.

The main goals are to:

• Allow Nitrogen to cooperate with--rather than compete against--some of the other great work out there, including Webmachine, Erlang Web, ErlyWeb, etc.
• Make it easier for the Nitrogen community to customize caching, sessions, state management, and many of the other things that get more complicated as a site grows.

Early prototypes are looking good. Hopefully I'll have some great stuff to show off by the time the 'Factory starts.

Some sample pages written in Nitrogen:

• Ajax and Comet page updates
• Form Validation
• Streaming Http File Uploads
• Dragging and Dropping
• Sorting

This clip did the trick - http://www.kk.org/thetechnium/archives/2009/02/so_amazing_but.php

(By comedian Louis C.K.)