We are moving this site to http://appdelegateinc.com/, for our newly formed company, App Delegate Inc.

Moved to http://appdelegateinc.com/blog/2010/05/17/browser-based-map-reduce/

I've recently upgraded to OS X 10.6 Snow Leopard and I was quite happy to know that it came with recent versions of Apache, PHP and Python. The first thing I did was enable PHP in Apache by editing http.conf

and then I uncommented the PHP line by changing

Next, I checked out my company's source code and browsed to the website at http://localhost/~mwilliamson/website and tried to login. Unfortunately, I came across this lovely error:

So I Googled around and saw various people with the same issue where none was able to be fixed. I checked to see if the MySQL database was using old passwords-- it wasn't. I changed the password on the database user. No dice. Then I decided to recompile PHP without mysqlnd and woo! It worked! Naturally, I'm making it sound much quicker than it really was. This took several days.

Here's the commands to recompile PHP and restart Apache.

We can't make and install yet. There's a bug in the PHP sources which messes up iconv. We'll need to change line 186 in ext/iconv/iconv.c:

change:

Save, then run:

Calling `Make` at this point will yield the following error:

We'll need to change the Makefile first. Add "-lresolv" to the EXTRA_LIBS line on line 102 like so:

to

Then we can build and install:

Hope it helps!

I've noticed it can be hard to find the mochiweb docs online, so I compiled them and I'm now hosting them at mochiweb documentation. Feel free to ask in the comments if you'd like me to host anything else at http://dawsdesign.com/drupal/man-pages.

While thinking about ErlFS, I realized I'd need at least two subsystems. One for finding a node with the data I am looking for and one to actually store that data. The first will be covered by Chordial. For the storage portion, I was originally going to use a homegrown method where I would just store the data in files named after their key. I decided I should explore the various possiblities and so I did a little research and decided to take a crack at writing a port driver for Erlang. Thanks go to Kevin Smith for his article on writing linked-in drivers for Erlang.

There are several way to write an Erlang driver. One way is to spawn a process and communicate over a pipe to your program and another is to bring your program in the same memory space and communicate directly with in-memory vectors. The first is safe; if your program crashes, your Erlang application can recover using standard OTP principles, such as supervisors. The in-vm-memory model is much more dangerous; if your program crashes, the whole Erlang VM crashes with it. This method, however, is much faster, as it does not have to copy memory and then have the OS switch context for each call. We will be covering this method.

Writing your first Erlang driver can be very difficult as there are very limited examples and the documentation can be very obscure. In this article, I will walk you through writing one for a common purpose-- writing to a database.

All sources can be found at the project's GitHub page.

Build Directory and Makefiles

First let's start with a simple OTP build directory structure. I'll create a new folder called erl_bdb_store and in that make the following folders: ebin, include, priv, and src. This is what it should look like:

• erl_bdb_store/
  • ebin (to store the erlang bytecode/binaries)
  • include (to store erlang include files)
  • priv (to store non erlang projects/code)
  • src (to store erlang source code)

Next, we'll need a couple simple makefiles. Please note that these currently only compile under linux (tested on Ubuntu). The first one just goes to the /priv and /src directories and runs the Makefiles in each.

./Makefile:

---

The next one compiles the C source (the meat-and-potatoes of the driver) into a shared library. This is different on each operating system and will need to be changed for each. I will do this sometime in the future.

./priv/Makefile:

---

The erlang Makefile is simple
./src/Makefile

---

Erlang Source

Now we will make our Erlang wrapper for the driver. It is a standard OTP gen_server behavior callback.
./src/bdb_store.erl

The two important parts are the init function and the handle_call function. The init function loads and starts the driver.

---

The handle_call function communicates with the driver to put, get and delete records from the database file. This just forwards and translates the commands to the actual driver, where all of the real work happens:

---

Notice the primary difference between each call is the first byte of each Message binary. This byte is inspected inside the driver and there it is determined which function to perform:

---

C Source

The C code does most of the work. It handles creating all of the error messages and performing all of the commands the Erlang source communicates to it.

Let's take a look at the C source piece-by-piece.

First we'll create a header file with settings and function prototypes.
./priv/bdb_drv.h:

At the top we include necessary Erlang headers followed by a couple standard C headers and then the Berkeley DB header. Next we define the path where we want our database to store data, and then we define constants for the byte values of the commands sent to the driver from the Erlang VM. The _bdb_drv_t struct is very important, as we cannot use global variables and so we must keep our state in a struct. The reason is that the Erlang VM can and will start a new instance of the driver many times for performance and we want to keep our state localized. The last entries in the header file are our function prototypes.

---

Now to our C implementation file (./priv/bdb_drv.c).
Include our header file:

---

Specify the callbacks we will be implementing:

This array defines the callback functions that will be invoked at various points by the Erlang VM. In this example, we are implementing start, stop and outputv. This means that when the Erlang VM starts an instance of our driver, it will call our function start, when it sends a message, it will call outputv and when it stops the instance, it will call stop. outputv is only used if output is not defined in the array. output uses a buffer whereas outputv does not, meaning there is no overhead in copying the data, so it is faster. See driver_entry documentation for more detail.

---

Next is boilerplate which tells the VM which struct to use as the state holder:

---

Here, we define the start function and open up the database for reading and writing.

If Berkeley DB returns an error, it is propagated to the Erlang VM as a typical {error, Reason} tuple.

---

The stop function closes the database and releases the driver:

---

outputv will serve as our entry point when a message is sent to the driver:

It interprets the first byte of the message to determine which function to call.

---

As it's name implies, the put function inserts a record into the database:

The function grabs the first 20 bytes after the first command byte and uses it as the key. This leaves room for 160 bits-- enough for a SHA1 hash. All of the remaining bytes are used as the value to be stored. This function returns the atom ok or the typical {error, Reason} tuple.

---

The get function is similar to put, except we have to free the memory allocated by Berkeley after the record has been retrieved:

First we call an Erlang function named driver_alloc_binary (see erl_driver#driver_alloc_binary) which keeps a reference count, starting with 1. We then return the value with driver_output_term which increases the reference count to 2 and then call driver_free_binary which brings it to 1 again. When the Erlang VM has finished with it, it will also call driver_free_binary which brings the reference count to 0 and it will be freed from memory.

---

The del function deletes a record and is the most simple of the database functions:

---

Finally we have our catch-all function which will be called if an unrecognized command byte is sent. It returns the tuple {error, unkown_command}:

---

Usage

Let's compile it and jump into an Erlang shell to test it out:

---

Now, let's start the gen_server we wrote and perform a couple tests:

---

That's it for the walkthrough. If you have any questions, please post a comment or use the contact form. You can also email me at the email address listed in the source. I hope it helps you on a future project! Please let me know if it does!

All sources can be found at the project's GitHub page.

I've been stalled with ErlFS for a while now, but my interest has been flared up due to someone cloning my SVN repo into GitHub, where it ended up on the front page for interesting projects.

So, I had actually been scheming a bit and was thinking about using a distributed hash table instead of a mesh network, due to the ability to scale massively. This is one of the goals of ErlFS, so it seemed natural.

I'm dead set on it now and I've started a project called chordial, which is my open source _interpretation_ of MIT's chord algorithm. Once it is finished, it should greatly simplify ErlFS and I will likely start from scratch.

You can find chordial at http://github.com/dawsdesign/chordial/.

I can currently add multiple nodes to the ring, but I am having some issues with the finger function on the nodes. It will just take some time to get sorted out, but I am feeling great about this project and it is going quite well.

If you want to get your manager to try out Erlang, but they are hesitant (probably an understatement) you might be able to get some leverage if you use a relational database rather than mnesia. So, here's a quick sample of how to connect to and query a MySQL database.

First, you'll need to install the MySQL ODBC driver if you haven't already. You can find it at http://dev.mysql.com/downloads/connector/odbc/5.1.html. Then you'll need a database to connect to and a table to query. Here's the SQL script I used to create my test database:

And here's the code to connect and query in erlang:

This is what my interactive session looks like:

It's that simple. Post a comment if you want more samples or description.

Erlang docs: http://www.erlang.org/doc/apps/odbc/index.html

I came across the most impressive Erlang tutorial I have seen yet. Check it out. Now.

http://www.vimeo.com/2007411

I'm working on a tile editor in Flash CS4 and opted to use the TileList class for the palette. Once a swatch is selected, I need to create a copy of that swatch and attach it to the stage. I wasn't quite sure how polymorphism in AS3 worked but I ended up with this little snippet:

Let me know if it helps!

I recently got an iPod Touch and it is the best gadget I've ever had. One of the features of the powerful little machine is the location service. This service uses the wifi radio to listen for wifi beacons, the signal an access point sends out every second or two to let clients know the AP is there. When you refresh your list of wireless networks, it gathers these signals to make the list. Anyway, the utility of this feature is actually quite extensive. You can use it to find nearby restaurants, movie showtimes, and make getting directions a breeze. Unfortunately in my home area, there is no coverage.

I began searching and found an article on MacRumors that mentioned Apple uses a company called Skyhook to provide their database of AP MAC addresses and their coordinates http://www.macrumors.com/2008/01/23/skyhooks-wi-fi-location-technology-f.... Skyhook, according to their website, keeps a database of MAC addresses and associates them with their GPS location. Using this database is claimed to be accurate within 10-20 Meters.

After visiting Skyhook's website, I found the form where you can add your Wifi Access Point to their database. The page is at http://www.skyhookwireless.com/howitworks/submit_ap.php. Unfortunately it can take up to a few weeks to have the data updated with all of their partners. I will repost here if it ends up working for me.

P. S. If you can't find your MAC address, this article might help http://getsatisfaction.com/skyhookwireless/topics/how_do_i_find_the_mac_....

UPDATE 12/22/2008
It worked! Now when I push the current location icon in Google Maps it centers right on my street. 12 days later ain't bad.