unified

#Creating an editor

I’m not entirely sure how to call this thing. “Demo” is too generic, “dingus” too vague, and I think editor is pretty apt. Anyway, drop a line on Gitter if you know a better name.

This guide shows how to create an interactive online editor with unified. In it we’ll visualise syntactic properties of text by “syntax highlighting” them. The editor will run in a browser. It’ll be fast as we’re using virtual-dom (but you could use React and the like too).

For this example we’ll create an app that visualises sentence length. It’s based on a tip by Gary Provost, and the visualisation is based on a tweet by @gregoryciotti.

You can also view this project with some more features online.

Stuck? A good place to get help fast is Gitter. Have an idea for another guide? Share it on Gitter!

#Case

Before we start, let’s first outline what we want to make. We want to highlight sentences in text based on how many words they have. The user should be able to change text, and it should highlight live.

We’ll use xo as a linter, and browserify as a bundler to compile our JavaScript with require calls to JavaScript that works in the browser (you could of course swap those out for your favourite linter and bundler).

#Project structure

Let’s first outline our project structure:

demo/
├─ index.js
├─ build.js
├─ index.html
├─ index.css
└─ package.json

...where demo/ is our directory, and build.js is the JavaScript generated by compiling index.js.

Keep index.js, index.html, and index.css empty for now, and fill package.json with the following.

package.json
{
  "name": "demo",
  "private": true,
  "dependencies": {},
  "devDependencies": {
    "browserify": "^14.0.0",
    "xo": "^0.18.0"
  },
  "scripts": {
    "build": "browserify index.js > build.js",
    "lint": "xo",
    "test": "npm run build && npm run lint"
  },
  "xo": {
    "space": true,
    "esnext": false,
    "envs": [
      "browser"
    ],
    "ignore": [
      "build.js"
    ]
  }
}

private: true means you can’t accidentally publish your package to npm.

Now, after running npm install and npm test you’ll see build.js appear too. The above package sets up xo as the linter and browserify as the bundler.

Now, fill index.html with the following:

index.html
<!doctype html>
<meta charset="utf8">
<title>demo</title>
<link rel="stylesheet" href="index.css">
<div id="root"></div>
<script src="build.js"></script>

This links index.css and build.js, and adds an element (#root) which we’ll add our editor to later. Oh, did you know that <html>, <head>, and <body> are optional? For this example we’ll keep the HTML clean, but feel free to add them if you prefer them.

#Setting up JavaScript

Alright! Now, let’s set up our JavaScript. Start by adding the following to index.js:

index.js
var h = require('virtual-dom/h');
var createElement = require('virtual-dom/create-element');
var diff = require('virtual-dom/diff');
var patch = require('virtual-dom/patch');

var root = document.getElementById('root');
var tree = render('The initial text.');
var dom = root.appendChild(createElement(tree));

function onchange(ev) {
  var next = render(ev.target.value);
  dom = patch(dom, diff(tree, next));
  tree = next;
}

function render(text) {
  var node = parse(text);

  return h('div', {className: 'editor'}, [
    h('div', {key: 'draw', className: 'draw'}, highlight(node)),
    h('textarea', {
      key: 'area',
      value: text,
      oninput: onchange
    })
  ]);

  function parse() {}

  function highlight() {}
}

Don’t forget to npm install --save virtual-dom.

That’s going a bit fast, I can imagine, if you’ve never seen virtual-dom in use before. If that’s the case, please take some time to peruse the virtual-dom docs at your leisure. This guide will wait!

To summarise what all these things in the code mean:

  • h creates “virtual” nodes
  • createElement turns them into DOM nodes
  • diff finds the difference between two virtual nodes
  • patch applies diff to a DOM node
  • root is our anchor into the document
  • tree is the current virtual tree
  • dom is the current DOM tree
  • onchange handles any state change (the text in our case)
  • render creates a new virtual tree based on that state
  • parse transforms the state into a natural language syntax tree
  • highlight transforms that syntax tree into a virtual tree

In render, we’re creating two elements: a <div> that we’ll draw our syntax highlighting in, and a <textarea> that the user can edit. Both are wrapped in a parent <div>. We’ll style the text area and the drawing area exactly the same, and position the text above the drawing area, with the following styles.

index.css
html {
  font-size: 16px;
  line-height: 1.5;
}

.editor {
  position: relative;
  max-width: 37em;
  margin: auto;
  overflow: hidden;
}

textarea, .draw {
  margin: 0;
  padding: 0;
  width: 100%;
  border: none;
  outline: none;
  resize: none;
  overflow: hidden;
  /* Can’t use a nice font: kerning renders differently in textareas. */
  font-family: monospace;
  line-height: inherit;
  font-size: inherit;
  background: transparent;
  white-space: pre-wrap;
  word-wrap: break-word;
  font-size: inherit;
  line-height: inherit;
}

textarea {
  color: inherit;
  position: absolute;
  top: 0;
}

.draw {
  min-height: 100vh;
}

That’s quite a bit of code: mainly to enforce the same styles on our text and drawing areas.

#Natural language syntax tree

Now, let’s set up our natural language syntax tree parsing. We’ll of course use unified, and retext-english to parse English natural language.

Change index.js like so:

index.js
@@ -2,7 +2,10 @@ var h = require('virtual-dom/h');
 var createElement = require('virtual-dom/create-element');
 var diff = require('virtual-dom/diff');
 var patch = require('virtual-dom/patch');
var unified = require('unified');
var english = require('retext-english');

var processor = unified().use(english);
 var root = document.getElementById('root');
 var tree = render('The initial text.');
 var dom = root.appendChild(createElement(tree));
@@ -25,7 +28,9 @@ function render(text) {
     })
   ]);

  function parse() {}
  function parse(value) {
    return processor.runSync(processor.parse(value));
  }

   function highlight() {}
 }

Don’t forget to npm install --save unified retext-english.

Sweet, now we have access to a lot of info on the text. It still doesn’t do anything yet though. Let’s add some usefulness.

#Virtual DOM

Our next task is to go from a natural language syntax tree to a virtual DOM. We already have highlight for that, but it’s empty, so let’s add code to fill it:

index.js
@@ -32,5 +32,21 @@ function render(text) {
     return processor.runSync(processor.parse(value));
   }

  function highlight() {}
  function highlight(node) {
    var children = node.children;
    var length = children.length;
    var index = -1;
    var results = [];

    while (++index < length) {
      results = results.concat(one(children[index]));
    }

    return results;
  }

  function one(node) {
    var result = 'value' in node ? node.value : highlight(node);
    return result;
  }
 }

highlight searches all children in the given node, and one returns either the “text content” of a node, or the result of searching its children for text content.

If you’d now run npm test again, and open index.html in your browser, you’ll see that the drawing area already has our text (it’s hidden with styles, but you should be able to see it in your web inspector).

We need one more thing before we can start highlighting: we need to detect sentences, and apply styles to them. Change index.js like so:

index.js
@@ -18,6 +18,7 @@ function onchange(ev) {

 function render(text) {
   var node = parse(text);
  var key = 0;

   return h('div', {className: 'editor'}, [
     h('div', {key: 'draw', className: 'draw'}, highlight(node)),
@@ -47,6 +48,19 @@ function render(text) {

   function one(node) {
     var result = 'value' in node ? node.value : highlight(node);

    if (node.type === 'SentenceNode') {
      key++;
      result = h('span', {
        key: 's-' + key,
        style: {backgroundColor: color(count(node))}
      }, result);
    }

     return result;
   }

  function count() {}

  function color() {}
 }

key is needed for virtual-dom to be performant.

We don’t colour sentences yet, but there’s <span> elements wrapping them now. You can see that in action by running npm test again and using your web inspector to inspect the drawing area.

We’ve also set up two functions to highlight sentences. count will count the number of words of a given sentence, and color will pick a corresponding colour.

#Highlight

Now, let’s add colours. Update index.js like so:

index.js
@@ -4,6 +4,11 @@ var diff = require('virtual-dom/diff');
 var patch = require('virtual-dom/patch');
 var unified = require('unified');
 var english = require('retext-english');
var visit = require('unist-util-visit');

var hues = [
  0
];

 var processor = unified().use(english);
 var root = document.getElementById('root');
@@ -60,7 +65,20 @@ function render(text) {
     return result;
   }

  function count() {}
  function count(node) {
    var value = 0;

    visit(node, 'WordNode', add);

    return value;

    function add() {
      value++;
    }
  }

  function color() {}
  function color(count) {
    var val = count < hues.length ? hues[count] : hues[hues.length - 1];
    return 'hsl(' + [val, '93%', '85%'].join(', ') + ')';
  }
 }

Don’t forget to npm install --save unist-util-visit.

The count function searches node for all occurrences of words, through unist-util-visit, and returns that count.

color takes a number, and returns a nice colour in HSL for it. It does so based on if there’s a corresponding hue for it in hues (now just one value). If there’s no corresponding hue, it uses the last specified hue.

It’s not much, but it’s something. Try it out by running npm test again, and viewing index.html in your browser. If everything went okay, you should see each sentence highlighted in red.

#Colour

One colour isn’t that cool, and we’re trying to recreate that visualisation by @gregoryciotti. We need some more colours. From that image, I deducted the following hues. But you could of course use any hues you like!

To match that image, change hues like so:

index.js
@@ -7,7 +7,20 @@ var english = require('retext-english');
 var visit = require('unist-util-visit');

 var hues = [
  60,
  60,
  60,
  300,
  300,
   0,
  0,
  120,
  120,
  120,
  120,
  120,
  120,
  180
 ];

 var processor = unified().use(english);

#Squashing bugs

💃 After running npm test again, and reopening index.html in your browser, you should now see The initial text in purple! If you add more sentences, they each should receive colours based on how many words they have.

If you add more text, you’ll notice that our drawing area grows nicely, but our text area does not. That’s because this example positions the <textarea> absolutely on top of the drawing area. The easiest way to get both areas the same height, is with the following slightly hacky code:

index.js
@@ -28,10 +28,13 @@ var root = document.getElementById('root');
 var tree = render('The initial text.');
 var dom = root.appendChild(createElement(tree));

setTimeout(resize, 4);

 function onchange(ev) {
   var next = render(ev.target.value);
   dom = patch(dom, diff(tree, next));
   tree = next;
  setTimeout(resize, 4);
 }

 function render(text) {
@@ -95,3 +98,10 @@ function render(text) {
     return 'hsl(' + [val, '93%', '85%'].join(', ') + ')';
   }
 }

function resize() {
  dom.lastChild.rows = Math.ceil(
    dom.firstChild.getBoundingClientRect().height /
    parseInt(window.getComputedStyle(dom.firstChild).lineHeight, 10)
  ) + 1;
}

This updates the rows attribute on the text area to correspondent with the size of the drawing area.

#Further exercises

The above code has a few issues:

  • onchange is not debounced, which leads to performance issues
  • input events are not supported in some older browsers
  • The styles aren’t perfect
  • and probably some other things!

...maybe you could solve some? Other than those issues, it’s a pretty cool little demo.

If you haven’t already, check out the other guides!