unified

unified is an interface for processing text with syntax trees and transforming between them.

Core

The unified library itself is a small module. It’s a rather small API. Plugins do everything else: minify HTML, lint markdown, check indefinite articles (“a”, “an”), and more.

Three syntaxes are connected to unified, each coming with a syntax tree definition, and a parser and stringifier: mdast with remark for markdown, nlcst with retext for prose, and hast with rehype for HTML.

unified defers part of its logic to vfile, which is a virtual file format representing documents being processed, and unist, a schema for syntax trees.

Syntaxes

remark for markdown

rehype for HTML

retext for natural language

Virtual files

vfile stores metadata about documents being processed (often, but not always, from the file system). Mainly, it houses a path to files, and their contents. Additionally, it tracks messages associated with files and where they occurred. This powers code linting, shown below with remark-cli, remark-validate-links, and remark-preset-lint-consistent.

sh
remark -u preset-lint-consistent -u validate-links example.md
example.md
        1:3  warning  Incorrect list-item indent: add 2 spaces
  1:11-1:18  warning  Emphasis should use `*` as a marker
   3:1-3:10  warning  Found reference to undefined definition

 3 warnings

Read more about vfile in it’s readme.

Linting with unified

Open Source Guides by GitHub (and you) uses unified to check markup and prose style

Gatsby uses unified to process markdown for blazing fast static site generation

debugger by Mozilla uses unified to check their markup and prose

Syntax tree

unist discloses documents as syntax trees. Syntax trees come in two flavours: Concrete (CST) and Abstract (AST). The first has all information needed to restore the original document completely, the latter does not. But, ASTs can recreate an exact syntactic representation. For example, CSTs house info on style such as tabs or spaces, but ASTs do not. This makes ASTs often easier to work with.

For example, say we have the following HTML element:

example.html
<a href="https://github.com/unifiedjs/unified" class="highlight">
  Fork on GitHub
</a>

Yields (in hast, an abstract syntax tree):

tree.json
{
  "type": "element",
  "tagName": "a",
  "properties": {
    "href": "https://github.com/unifiedjs/unified",
    "className": ["highlight"]
  },
  "children": [
    {"type": "text", "value": "\n  Fork on GitHub\n"}
  ]
}

These trees also come with positional information (not shown above), so every node knows where it originates from.

Read more about unist in it’s readme.

Built with unified

write-music

write-music visualises sentence length with unified. Varying sentence length can make reading more enjoyable.

wooorm.com/write-music

This sentence has five words. Here are five more words. Five word sentences are fine. But several together become monotonous. Listen to what is happening. The writing is getting boring. The sound of it drones. It’s like a stuck record. The ear demands some variety.

Now listen. I vary the sentence length, and I create music. Music. The writing sings. It has a pleasant rhythm, a lilt, a harmony. I use short sentences. And I use sentences of medium length. And sometimes when I am certain the reader is rested, I will engage him with a sentence of considerable length, a sentence that burns with energy and builds with all the impetus of a crescendo, the roll of the drums, the crash of the cymbals—sounds that say listen to this, it is important.

awesome-lint

awesome-lint uses unified to make it easier to create and maintain Awesome lists.

sh
awesome-lint
readme.md
      1:1  Missing Awesome badge after the main heading      awesome-badge
     12:1  Marker style should be -                          unordered-list-marker-style
    199:3  Remove trailing slash (https://sindresorhus.com)  trailing-slash

3 errors

how-to-markdown

how-to-markdown, a core NodeSchool workshopper, uses unified to teach markdown

sh
how-to-markdown
How To Markdown Markdown is awesome! ──────────────────── » HELLO WORLD » HEADINGS » EMPHASIS » LISTS » LINKS » IMAGES » BLOCKQUOTES » CODE

documentation.js

documentation.js uses unified to generate formatted docs

sh
documentation build ex.js -f md api.md && head -n 10 api.md
# documentation

Generate JavaScript documentation as a list of parsed JSDoc
comments, given a root file as a path.

**Parameters**

-   `indexes` **Array<string> or string** files to process
-   `options` **Object** options
    -   `options.external` **Array<string>** a string regex / glob match pattern

readability

readability measures ease of reading of text. It uses several formulas. Green means text should be readable for your target audience.

wooorm.com/readability

While many writers and speakers since ancient times have used plain language, the 20th century brought more focus to reading ease. Much research has focused on matching prose to reading skills. This has used many successful formulas: in research, government, teaching, publishing, the military, medicine, and business. Many people in many languages have been helped by this.

By the year 2000, there were over 1,000 studies on readability formulas in professional journals about their validity and merit. The study of reading is not just in teaching. Research has shown that much money is wasted by companies in making texts hard for the average reader to read.

Guides

Use unified

example.js
var unified = require('unified');
var vfile = require('to-vfile');
var report = require('vfile-reporter');
var markdown = require('remark-parse');
var toc = require('remark-toc');
var remark2retext = require('remark-retext');
var english = require('retext-english');
var indefiniteArticle = require('retext-indefinite-article');
var remark2rehype = require('remark-rehype');
var doc = require('rehype-document');
var html = require('rehype-stringify');

Learn how to use unified by transforming markdown to HTML. It’ll also show how to use plugins, add a table of contents, and check prose.

Create a plugin

plugin.js
var visit = require('unist-util-visit');
var is = require('unist-util-is');

module.exports = attacher;

function attacher() {
  return transformer;

  function transformer(tree, file) {
    visit(tree, 'ParagraphNode', visitor);

    function visitor(node) {
      var children = node.children;

Learn how to create a plugin for unified by creating a plugin for retext to check the amount of spaces between sentences.

Create an online editor

wooorm.com/write-music

This sentence has five words. Here are five more words. Five word sentences are fine. But several together become monotonous.

Now listen. I vary the sentence length, and I create music. Music. The writing sings. It has a pleasant rhythm, a lilt, a harmony. I use short sentences. And I use sentences of medium length. And sometimes when I am certain the reader is rested, I will engage him with a sentence of considerable length, a sentence that burns with energy and builds with all the impetus of a crescendo, the roll of the drums, the crash of the cymbals—sounds that say listen to this, it is important.

Learn how to create an interactive online editor with unified. This guide creates a demo visualising syntactic properties of text.