Hello again world, it's been awhile. Sorry it's been so long since we last talked...

While it is true that I've been busy, we both know that's just an excuse. So much of my early career development centered around blogging, and was spurred by trying to articulately communicate technical ideas. The practice of self-reflection and thoughtful communication is pivotal in the progression of every software engineer.

So I'm back, ready to reboot and create a space where I can log lessons learned and battles won.

Let's catch up! Since I last blogged...

• I moved home to Missouri from Colorado 🌄
• I spent the last 5 years in remote positions, working on product teams full of smart and talented people 👨🏻‍💻
• I lost almost 100 pounds 📉
• I went entirely vegan ⓥ
• I started rock climbing 🧗🏻‍♂️
• I built my own little product for webpack users 📊
• I spoke at a conference 🎤
• I bought 10 acres of woods 🌳🌲🌲🌳🌲
• I traveled to 6 different countries 🇨🇦🇵🇹🇮🇪🇪🇸🇫🇷🇨🇿

I've got quite the backlog of experiences to write about, and I've also decided to bring along the least egregious and evergreen posts from my pre-2015 self.

Apologies for the young opinions and PHP 😂

Hopefully it won't be so long until you see another post from me.

Cheers!

We're all too familiar with this term and stigma. I have enjoyed my limited time working with node.js professionally, but I can't have a conversation with someone outside the ecosystem without callback hell coming up.

The idea is simple. With JavaScript, it is canonical to handle all I/O in an asynchronous way. The lowest level tool being callbacks passed to functions that perform I/O.

So routinely, we get an arbitrary example that looks something like this...

DBfind(function(err, thing) {
    thing.doSomething(function(err) {
        thing.doAnother(function(err) {
            thing.doYetAnother(function(err) {
                // :(((((
            });
        });
    });
});

The primary problem presented with these types of examples boils down to the fact that it is ugly.¹ With the universal counter point, Promises or flow control libraries, focused on fixing nesting depth.

Is that the primary problem though, or is it deeper?

We have stacked 4 sequential I/O operations. Is this a good thing even in a synchronous I/O environment? Imagine the same logic stuffed into an object method that blocks awaiting the return. On platforms that prefer synchronous I/O, these expensive operations are easy to bury.

The concept of callback hell should be treated first as a symptom that could point to a deeper design issue. Is this method's single responsibility I/O? I have come to view methods accepting callbacks as a potential code smell. Causing us to determine whether the I/O can be made more explicit to the developer.

A common offense being what I will call tail call I/O. The object method mutates the object, and then calls out to I/O proxying the callback passed to the original method call.

Imagine the following example.

Thing.prototype.updateFoo = function(newFoo, callback) {
    this.somethingComplicated = newFoo;
    this.save(callback);
};

Thing.prototype.updateBar = function(newBar, callback) {
    this.somethingMoreComplicated = newBar;
    this.save(callback);
};

var thing = new Thing();

thing.updateFoo('Foo', function(err) {
    thing.updateBar('Bar', function(err) {
        // :((((((
    };
};

In these cases, you can easily flatten that out like so.

Thing.prototype.updateFoo = function(newFoo) {
    this.somethingComplicated = newFoo;
};

Thing.prototype.updateBar = function(newBar) {
    this.somethingMoreComplicated = newBar;
};

var someThing = new Thing();
someThing.updateFoo('Foo');
someThing.updateBar('Bar');
someThing.save(function(err) {
    // :)
});

Not only is this nicer, but more importantly it is a design improvement. Reaching out to I/O land only once, when the developer consciously decides it is required. This approach also makes .updateFoo() and .updateBar() easy to test. Previously, they would be beholden to the external services performing the I/O operation, needing to be mocked or integrated into the test runner.

There are situations that require a waterfall of I/O operations, but our first instinct should not be to accept them as is, targeting cosmetic defects.

Instead of bemoaning this distinctive property of JavaScript and slapping some paint on it, use it to better assess the quality and performance of our api designs.

In transitioning to the Node ecosystem, I quickly recognized many strengths npm brings to the table as a package manager. As someone who has used rvm/bundler to routinely manage gemsets, an immediate win I noticed was how every install of a project is localized within the project by default.

Really enjoying the #node package ecosystem… everything is local by default. Global packages being mere convenience.

— Jonathan Ⓓ Johnson (@jondavidjohn) March 8, 2014

This is not only true about code level library dependencies, but also command line tooling that the project may rely on for automation. These tools are linked into ./node_modules/.bin/*

For example, after an npm install of a project that declares gulp (a popular task runner) as a devDependency, you would get your task runner installed at ./node_modules/.bin/gulp. This means you don't actually have to install gulp globally as long as you're ok with running gulp with

./node_modules/.bin/gulp <task>

Because this is a bit verbose, most projects also recommend you install a global gulp runner that enables you to shorten this up with a familiar

gulp <task>

But are these global installs necessary to reap the more succinct benifit?

This is where npm's script feature comes in.

npm scripts

Another feature of npm is the ability to define both lifecycle tasks for your package and custom package scripts, giving it some limited ability as a task runner itself.

In your package.json you can define these scripts like so.

{
  "name": "my_package",
  "version": "0.0.1",
  "description": "My Awesome Project.",
  "scripts": {
    "<task>": "gulp <task>"
  }
}

Then you can

npm run <task>

Which achieves our succinct interface while divorcing the global install requirement.

This is possible because npm run <...> runs with ./node_modules/.bin included in its path.

consitency

Another benifit of this approach is having a consistent task interface across multiple projects that might have differing task running needs. One project might particularly benifit from a stream based task runner like gulp, while another might not need an additional task runner at all.

Both projects in this case would use the same npm scripts interface and would be immediately familiar for team members that might not be familiar with one or both projects.

  var testForA = RegExp.prototype.test.bind(/[aA]/);
  // function () { [native code] }

  testForA('hello');
  // false

  testForA('hella');
  // true

  ['hello', 'hella', 'holla'].filter(testForA);
  // ['hella', 'holla']

Oh the lengths we'll go to avoid typing 'function'

Node is much more than JavaScript on the Server.

That's Node's gimmick though right? You can now write both your server and client in JavaScript.

No longer will you toil away, learning and bouncing between multiple environments. Painfully switching contexts, excreting blood and tears when you are forced to move from the client to the server or vice versa.

Hopefully you picked up on my sarcasm.

The fact that Node uses JavaScript is but a tertiary point (a benefit to some, a downside to others). It most definitely is a contributor to it's success, but so much so that the primary point and purpose of Node is often missed. If this is obvious and apparent to you, you probably can stop reading.

Saying that you like Node because you can write JavaScript on the server side is like saying you like red Lotus Elises because they are painted red. Not everyone likes red, and even if everyone did, it's hardly the most important/objective thing that makes this car "good".

Node is fundamentally different than the dynamic language platform status quo. It was born from an obsession with efficiency. A dissatisfaction with the classic 1-to-1 relationship between threads and connections drove it's ability to handle a significantly increased level of concurrency for web applications written with dynamic languages.

Blocking and the I/O Problem

The classic bottleneck of most applications, web or otherwise, is I/O.

The application is regularly transferring information to and from a database, the file system, the network, or a combination of these. Generally, the transfer of this information is far slower than the speed of the CPU operating on this information in memory. Productive CPU over-utilization is likely not your problem.

The second problem is the way we have handled the first. Historically understanding computers, we think sequentially. This is how Computers think right? You provide a set of instructions and they are run in sequence from top to bottom. Your first program generally looks something like this.

// print
console.log("Enter your name: ");

// block execution to retrieve data
// keeping the process tied up awaiting the
// data's return
var name = console.read();

// manipulate data and report
console.log("Hello " + name + "!");

But what if you could do this in a non-blocking way

// print
console.log("Enter your name: ");

// make a request for the data, but allow the process to
// be free to do other things, providing a bit of code
// to run when the data is retrieved.
console.read(

  // When we are notified we have a
  // response, manipulate data and report
  function(err, name) {
    console.log("Hello " + name + "!");
  }
);

Making your process block and wait around for millions of clock cycles is a waste of time and resources. To handle additional requests, you must use a process manager of some kind to keep around multiple processes. As the web moves to be a more interactive medium, we are continually reducing the size of each request, while multiplying the number of requests generated by every single client. This makes it obvious why the 1-to-1 relationship between a request and a process is hard to scale.

Historically, we have taught ourselves to more easily reason our programs in the first style, but I would argue that the second style is equally, if not more, intuitive as a human being.

Thinking like a Human

As a human being you understand the asynchronous nature of information flow. You intuitively know that it's just not efficient or practical to stop everything you are doing every time you need to retrieve or communicate information.

As a developer, every time you ask a manager or stake holder of a project for clarification, do you simply sit and wait quietly for a reply? No, of course you don't. You move on to the next thing that you can make progress with, juggling a hand full of important tasks. Once the information comes back, you continue with the related task.

For a more rudimentary anecdote, you are driving in your car with a friend of yours and you need to know the weather report for tomorrow. Would you slam on the breaks and stop traffic to turn to your friend and ask what the weather will be like tomorrow? Waiting still and silent (except for the honking behind you) until he looks it up for you? Of course you wouldn't, you would continue having other conversations and driving while he retrieves the information for you.

In this way, Node's non-blocking/asynchronous model does not need to be as foreign as it would seem at first pass.

Why not threads?

Threads are hard, heavy, and generally overkill

They do solve the rudimentary problem of parallelism, but they present an all new set of problems around thread-safety and mutability. They are generally very complex and difficult for dynamic languages to implement, if implemented at all.

In Node, everything but your code, runs in parallel.

Everything that can be handled via parallelism for you, is. But because your code is never run in parallel and always in a single thread, issues of thread-safety and mutability fall away. This provides many of the benefits of parallelism for the common use case without presenting the developer with near the additional complexity.

Again, I would argue that the way Node handles parallelism again mirrors how we as humans process thoughts, not in parallel, but with highly efficient multi-tasking.

Language X has a non-blocking server too...

Non-blocking I/O is not a new thing, but what makes Node different is the platform is built entirely around this principle. Non-blocking is philosophically adopted as the rule, not the exception. So when you go to grab that Node MySQL library, you can expect it will be fully compatible with your non-blocking application. Generally this cannot be said of other event driven frameworks written in other dynamic languages.

DISCLAIMER: This is actually a really terrible idea, and mostly just a thought experiment!

This is a pattern you can use to extend JavaScript Functions, including native functions.

For example, we’ll extend the join() function of the Array type and add some logging before passing control back to the native join() function.

Array.prototype.join = (function(_super) {

    // return our new `join()` function
    return function() {
        console.log("Hey, you called join!");
        return _super.apply(this, arguments);
    };         // Pass control back to the original join()
               // by using .apply on `_super`

})(Array.prototype.join);
//                            
// Pass the original function into our
// immediately invoked function as `_super`
// which remains available to our new
// function thanks to JavaScript closures.

You can form more complex and useful git aliases when you realize you can write aliases like you would write a standard shell function. Prepending your alias with an exclamation mark tells git to expand the alias to your shell, enabling the use of standard unix tools to do some fun things.

Simple Example

[alias]
    ...
    sayhello = !(echo "Hello World")
    ...

This shows you how to run a simple single line bash command as a git alias. You first tell git you are going to run a shell command with !() then you give it your single line of bash.

Multi line aliases and passing parameters

The next step is to be able to run more of a shell script. Multiple lines and be able to pass in arguments.

[alias]
    ...
    say = "!f() { msg=${1-Hello World}; echo $msg;  }; f"
    ...

This illustrates the structure of a basic shell script git alias that allows for multiple lines and arguments. You first tell git you are going to run a shell command with !, then you define a shell function named f with f(){ };, then run it with f;. This enables you to accept arguments into your functions like $1 in our case.

Also, in case you are wondering, ${1-Hello World} assigns "Hello World" to $msg if no value is present in $1 which is how I chose to provide a default value.

On the heels of my responsive bash prompt, I've made another few moves to bring more utility to my git bash prompt.

Dirty?

Yes, so we have our git branch showing in our bash prompt as most do these days. That's a really helpful piece of information to be readily available, but lets do more.

I wanted the name of my branch to also let me know if my working directory is clean or not. If I have uncommitted changes or untracked files, I want to be able to know that just by looking at my bash prompt. So I've now made it so the branch name is red when the working tree is dirty and green when it's clean.

Here's how.

Out of Sync?

How many times have you committed changes and forgotten to push them? Or been working away on a feature and didn't notice you were behind by a number of commits? Wouldn't it be nice to also know if you're ahead or behind the remote tracking branch at a glance?

Here's what I've got

Here's how I did it

All of these tweaks and modifications are tracked in my dotfiles repository so take a look.

A commitment to polyglot mindset was an intentional decision I made early in my career as a developer.

(so, earlier than now...)

What did this mean at the time?

I'm going to know all the languages, so when a problem arises that needs solving, I'll solve it with the most perfect (obscure) technology choice. Achieving a solution with the clearest and correctest code base.

During my last round of job interviews, I can hear myself speaking like this. I can just hear "the right tool for the job" rolling off my tongue.

Maybe it's just me, but this seems like pervasive wisdom propagating through our industry right now, especially to new comers in the field. That it's important to know a lot of languages and technologies so that you can use the best of all the things, to solve problems perfectly.

The Polyglot Mindset

Sometimes this mantra can easily reek of our generations fear of commitment. Jumping from language to language, framework to framework.

I wrote a rails app, therefore I know rails.

I wrote a node app, therefore I know node.

I wrote a python script, therefore I know python.

This sort of aggressive polyglot mindset is like a boy scout in search of merit badges. A classic jack of all trades, and a master of none.

The little bit of experience I've gained has shown me that while it's good to be comfortable learning new things, it's far more valuable to gain mastery in a few things. To spend a significant amount of time with a single technology, or family of technologies. Learn the quirks and demystify the magic. To not be satisfied by breadth, but to seek out depth.

"The right tool for the job"

I've also learned that "the right tool for the job" seldom exists, and when it does it slaps you so hard in the face you cannot ignore its existence.

The truth (and beauty) of our field is that you can solve the same problem 10 different ways.

Within any successful organization there is a core technology focus, and it's highly likely that 1 of the 10 possible solutions involves a technology everyone on the team is familiar with. As much as "the right tool for the job" is a unicorn, as is a master of all trades, let alone a team of them.

What happens when you (are allowed to) choose a technology that you know (dabbled in), and the perfect solution starts breaking?

You go find the resident expert on ... oh wait... you are the resident expert.

There are many places in life where the theoretical world and the real world clash so harshly. This is one of those places. This is one of those fields.

This resounds so clearly in the talk I posted recently about Scaling Pintrest . In which the overarching lesson learned is that a simplified, familiar, mature architecture will likely be more resilient and maintainable, than one that contains a grab bag of the latest and greatest (immature and unfamiliar) solutions.

So be ignorant of other technologies?

Having said all this I still believe in a polyglot mindset that a polyglot mindset is useful in its place.

Running with weekend projects in foreign languages teaches you a lot. Not just about the technology in use, but it breaks you out of the logical rut path your everyday technology ingrains in you. You will begin to use your everyday technologies in ways you hadn't yet thought of.

Gain exposure to general programming idioms and patterns that you wouldn't otherwise be exposed to. Learn new ways to solve problems with your everyday language by learning the idioms of others.

Enjoy the new, but remember to respect mastery, familiarity, and maturity.

The thread-first and thread-last clojure threading macros are a really neat feature of Clojure, and somewhat difficult to explain to someone who has never had a use for them.

These two tools enable you to unwrap deeply nested function calls in a more readable way, without creating a bunch of single use variables in the process.

I'm going to create a few functions to make this seem more useful...

function sum(a, b) { return a + b; }
function diff(a, b) { return a - b; }
function str(a) { return a + ""; }

Consider this

var result = str(diff(10, sum(3, parseInt("3")))); // "4"

We could make it a bit more clear like this

var inted  = parseInt("3");
var sumed  = sum(3, inted);
var diffed = diff(10, sumed);
var result  = str(diffed); // "4"

But what if we could thread the values through like this? (->> is the thread-last operator in Clojure)

var result = thread("->>", "3"
                           parseInt,
                           [sum, 3],
                           [diff, 10],
                           str); // "4"

Looking at this you can start to see what the thread-last macro does. It starts with the value "3" and passes it as the last argument to parseInt(). The result of that, gets passed as the last argument to sum() (with 3 being the first argument). this goes on and on until the last function returns the result.

You can imagine what the thread-first macro (->) does...

var result = thread("->", "3"
                          parseInt,
                          [sum, 3],
                          [diff, 10],
                          str); // "-6"

The only call that changes result is diff() because with the thread-first macro, the result is passed as the first argument. Diff becomes diff(6, 10).

Here is my implementation of these Clojure threading macros in Javascript.

var thread = function() {
    var i, type, func, value;
    var args = Array.prototype.slice.call(arguments);

    type = args.shift();
    value = args.shift();

    switch (type) {
        case '->;': // thread-first
            while (args.length) {
                arg = args.shift();
                if (arg instanceof Array) {
                    func = arg.shift();
                    arg.unshift(value);
                    value = func.apply(this, arg);
                }
                else {
                    value = arg(value);
                }
            }
            break;

        case '->>': // thread-last
            while (args.length) {
                arg = args.shift();
                if (arg instanceof Array) {
                    func = arg.shift();
                    arg.push(value);
                    value = func.apply(this, arg);
                }
                else {
                    value = arg(value);
                }
            }
            break;
    }
    return value;
};

You are not your code

But this process is how we move the shared infrastructure forward. In order to advance the state of the art, we have to be willing not only to try new ideas, but to retreat when those ideas prove untenable or when something better comes along. And we have to be able to speak candidly about problematic code without fear of offending the egos behind it.

Software Engineering is an interesting art, one in which you pour over the smallest details of something that you are crafting into existence, but optimally are detached from the work as if you didn't. It's a very tricky balance to strike, and one that I believe comes with time and experience. Time and experience that will continue to show you that the beautiful solution you create today will be flawed and ugly by the standards 6 months or a year (if you are lucky) in the future.

I think there is a second lesson here, one that comes from the other side of the equation. Learning how to critique and question the work of others.

Just as you are not your code, remember that they are not their code.

Always seek to build up in collaboration instead of attacking in competition. The enlightened solution you provide today will be the next subject of critique.

UPDATE: I've moved this code into my dotfiles repo as I continue to improve and update it.

The Problem

This may be news to some, but json_encode() respects property visibility.

Meaning that if you have a class that looks like this...

class Thing {

    public $something;
    protected $protected_something;
    private $private_something;

    public __construct($pub, $prot, $priv) {
        $this->something = $pub;
        $this->protected_something = $prot;
        $this->private_something = $priv;
    }

    public get_private() {
        return $this->private_something;
    }

    public get_proctected() {
        return $this->protected_something;
    }
}

And you run one of these guys through json_encode()

$thing = new Thing("One", "Two", "Three");
echo json_encode($thing);

You'll end up with the following

{
    "something": "One"
}

json_encode() only serializes the publicly visible properties of an object. If you follow the getter/setter pattern (which I don't particularly recommend), this can be a neusense.

PHP < 5.4.0

The solution to this problem for those running PHP 5.3 or less (most of us), is to implement a json encoding function within your class. Which gives all your properties the visibility they need, and allows you to control the exact json representation of your object.

Using the example class above...

class Thing {

    ...
    public function to_json() {
        return json_encode(array(
            'something' => $this->something,
            'protected' => $this->get_protected(),
            'private' => $this->get_private()
        ));
    }
    ...
}

Now, you'll generate the json representation with your method (which uses json_encode() internally)...

$thing = new Thing("One", "Two", "Three");
echo $thing->to_json();

You'll end up with the following

{
    "something": "One",
    "protected": "Two",
    "private": "Three"
}

PHP >= 5.4.0

For all you lucky ones out there working in a PHP 5.4 environment, your solution is a bit more elegant

Simply implement the JsonSerializable interface and add the jsonSerialize function which should return the data structure you want to be represented in JSON.

Using the example class above...

class Thing implements JsonSerializable {

    ...
    public function jsonSerialize() {
        return [
            'something' => $this->something,
            'protected' => $this->get_protected(),
            'private' => $this->get_private()
        ];
    }
    ...
}

Now, when you run one of these guys through json_encode()

$thing = new Thing("One", "Two", "Three");
echo json_encode($thing);

You'll end up with the following

{
    "something": "One",
    "protected": "Two",
    "private": "Three"
}

This is the preferable solution, because now you don't have to write documentation for the users of your library/code how to convert your object to JSON, they simply use the native PHP JSON API.

Bonus: I want to just encode EVERY property

Just loop over the object inside any of the json methods above...

...
    $json = array();
    foreach($this as $key => $value) {
        $json[$key] = $value;
    }
    return $json; // or json_encode($json)
...

I intended to set up a repository (hosted on BitBucket) to initiate a pull on a dev server when new commits are pushed up.

It seemed like a simple enough process. BitBucket has a service that will fire off a POST request as a post-receive hook. So I set up a receiving php script to check a randomized token and then initiate the git pull. Looking something like this...

define('PRIVATE_KEY', 'XXXXXXXXXXXXXXXXxxx');

if ($_SERVER['REQUEST_METHOD'] === 'POST'
        && $_REQUEST['thing'] === PRIVATE_KEY)
{
    echo shell_exec("git pull");
}

Didn't end up being as simple as I had anticipated...

There were a few considerations that I did not take into account. Documenting them here will hopefully help you avoid some obstacles in trying to get something like this set up.

the binary and the $PATH

The user that is attempting to execute git pull is the apache user (www in our case). This user did not happen to have git in their path.

This took a while to track down because the exec() family of functions simply fail silently because they only report STDOUT and not STDERR. To get the function to report STDERR you can route it into STDOUT by adding 2>&1 at the end of your command.

After I realized this I logged in and found the full path of the git binary with which git, which is /full/path/to/bin/git.

...
    echo shell_exec("/full/path/to/bin/git pull 2>&1");
...

Now it was reporting the next issue...

permissions

error: cannot open .git/FETCH_HEAD: Permission denied

The apache user also needs read and write access to the entire repository.

chown -R ssh_user:www repository/

It's also a good idea to make sure any files/directories inherit this ownership if being created by others by setting the group sticky bit.

chmod -R g+s repository/

"Host key verification failed"

Next, you need to do an intial git pull with the apache user to make sure the remote is added to the apache user's known_hosts file

sudo -u www git pull

ssh key

Another consideration created by this command being run by the apache user is the ssh key it uses to communicate with the remote repository.

First, I went down the path of attempting to use the GIT_SSH environment variable to set the ssh -i option to tell it to use a specific ssh key I had generated with the ssh user. I never got this to work, most likely because there are a lot of rules ssh uses to determine the safety of a given key. It requires some specific permissions regarding the user that is attempting to use the key.

An easier way I discovered was to give the apache user a home directory (via /etc/passwd) and a .ssh directory and then run the ssh-keygen command as the apache user (www)

sudo -u www ssh-keygen -t rsa

This creates the keys and puts them in their expected location with the proper permissions applied.

Then I added the key as a read-only key for the BitBucket repository and everything worked as expected.

Yesterday I found an interesting quirk while working with a code base that uses json_encode and json_decode to serialize data moving in and out of the database.

The benifits of using JSON over standard PHP serialization (for basic objects) are fairly obvious:

• Data portability
• Easier to parse by hand (manual database updates)
• Lighter weight in general

But there is one gotcha... Consider this example:

$php_struct = array(
    "key" => 5,
    "another" = "data",
);

$json_string = json_encode( $php_struct );

//...

$php_struct = json_decode( $json_string );

// array? Nope, this is an object
echo gettype($php_struct);

Expected Unexpected Behavior

The problem here is that most often when using a serialization method in/out of the database, it should be hidden and abstracted into the boilerplate database interaction. The expectation is that if you put one thing in, you get the same thing out. Which is not the case when feeding these json functions associative arrays.

The Array Flag

json_decode() does provide a second parameter to tell it you want objects decoded as associative arrays, but again, this simply flips the problem on it's head. You encode an object and when it's decoded, you get an array back. It is still possible to put one thing in, and get a different thing out.

Solution?

The only solution I can think of is something like this...

Serialize

// $php_struct is our data

// Create a wrapper object
$obj = new stdClass;

// encode the actual data
$obj->data = json_encode( $php_struct );

// Store whether or not it is an array
$obj->is_array = is_array( $php_struct );

// encode for storage
$obj_json = json_encode( $obj );
save( $obj_json );

De-serialize

$obj_json = retrieve();
$obj = json_decode( $obj_json );
$php_struct = json_decode( $obj->data, $obj->is_array );