Syntax highlighter for HTML

The SyntaxHighlighter project looks nice. It has a "copy to clipboard" feature (implemented in Flash). With some additional hacking, this would save from keeping Novelang's nasty zero-width spaces added for correct line wrapping.

Zipper for faster tree modifications

Novelang uses immutable trees to represent a document and transform it. While immutable data structure have well-known advantages, Novelang's tree library requires to update every parent node on each change on any child node. Clever guys found how to save those changes when performing multiple local modifications. This relies on a tree structure called Zipper. Here is a very clear explaination, the original paper (site currently down), the Scala implementation and the Clojure one.

Google's font directory

This is an amazing initiative from Google: a directory for Web fonts. Fonts are available under SIL Open Font License 1.1. There are some beautiful fonts available, with a nice and clear browsing interface. One can download font sources from here.

Novelang-0.43.0 released!

Download Novelang-0.43.0 here !

• Added nohead option to insert command.

• Fixed some bugs around identifiers.

• Introduced detection of colliding explicit identifiers. This has no useful purpose for now but will serve as a basis for implementing internal links.

• Small performance enhancement on HTML document rendering in a Web browser: don’t use JavaScript to set collapsible descriptors hidden.

MinorThird's Mixup

Could this be useful in Novelang? The Mixup language performs complex queries on pure text. It's "like a regex query, but while regex operates at character level, Mixup operates at token level." Mixup is part of the MinorThird suite and available under BSD license.

Novelang-0.42.0 released!

Download Novelang-0.42.0 here !

• Now requires Java 6.

• New Nhovestone report: Novelang has its own benchmark!

• Added stylesheet html-FR.xsl for French punctuation.

• Performance enhancement on rendered HTML page: when containing many tags it should load faster. Instead of dynamically computing styles on the Web browser, HTML rendered by the server directly includes those styles.

• Various performance enhancements on document generation. With the same amount of memory (-Xmx parameter), Novelang handles documents twice bigger and serves them 20 % faster than previous version. Benchmark ran against version 0.41.0 and 0.38.1. This includes buffered reading of Part files, multithreaded Part rendering, and reduced memory consumption when dealing with AST (Abstract Syntax Tree).

Nhovestone

“Nhovestone” is the name of Novelang’s dedicated benchmark tool, and also a geeky pun .

Nhovestone aims to highlight performance variations across versions using only a few (carefully selected) measurements:How does response time evolve when increasing the number of documents aggregated in a single Book?How does response time evolve when increasing the size of one single document?

Nhovestone doesn’t try to generate an absolute performance index. This is because such an index makes sense only when computed from always the same source documents and the same hardware.

How it works

Nhovestone focuses on HTML generation using default stylesheet, because HTML is great for fast edit-and-review roundtrips. It uses the Novelist to generate pseudo-random text with a realistic structure. For each benchmarked Novelang version, Nhovestone starts a JVM with a small amount of memory (currently -Xmx32M). With few memory the breaking point appears sooner. Nhovestone increases the size of the source document(s) in a linear fashion, and after each increasing, measures how long takes the call of a Novelang instance.

Performance degradation

Response time start to increase exponentially as document becomes fairly big in regard of available memory. This triggers a lot of CPU-intensive garbage collection consuming a lot of time. Nhovestone detects that a running Novelang HTTP daemon gets “strained” when response time gets above a dynamically-computed threshold. The threshold comes from the straight line drawn from a linear regression on the first half of the measurements, with a slope made steeper by a fixed coefficient. When a response time appears above this straight line, the Novelang HTTP daemon got strained and it’s not worth any further measurement.

Adding Parts

This is the first scenario: for each new measurement, there is an additional Part file. All Parts are more or less equal in size and complexity (including level depth). The graph below shows that performance degradation stays linear until the 300^th call. Then, version 0.41.0 starts suffering before older versions. It’s likely that new features require additional memory so starvation occurs sooner.

Increasing the size of the same Part

This is the second scenario: the generated document comes from a single Part file of a size increasing before each call. Each fragment added to the Part has the same size and structure as in the previous test, but all 3 versions show fatigue much sooner (at least 7.5 times). This shows that creating a Part takes much more temporary memory than the finished Part itself.

Tuning

These figures are strongly connected to the volume and the structure of underlying document. Experience shows that small increments generate more measurements (before the fatal strain) and therefore show a more readable trend. They also reduce measurement artefacts that could fool strain detection.

Report generation

JFreeChart generates those graphs. JFreeChart is probably the best charting library for Java at this time, at least on the OSS marketplace. It is stable and highly configurable.

The next step: embed those graphs in a Novelang-generated PDF and publish it as a complement of existing documentation.

The Novelist: random text generation

Novelang already does all the typesetting for you. What’s next? Writing text, of course! The just-started Novelist subproject, which aims to generate big documents for Novelang testing under heavy load.

Based on French metrics, random text looks like this:

Uomuecto eaufues xuner ig ocanerr, ebanu otpaa. Uuse, on eian aibtd, rttaintlufe elvettarrh, yrn enemlcmlun, ebcazepuer madscg, êiiovemtt teeost eseeerde? Fetn eearréetcs emrseoss icia ntmvesrud. Aoasro cênit ctainetda aèugedet css eali, unero aaie eneoden, nrortio. Oovlod; tfsmenco méttsna, eesdis uoeaeanao rcuent, desungtt av au oneerao, dxuaste umeinétniu lccdeiilne rliùearde veyiritisac yàslu. Iinmseuo odiapqied cmiiapearlo ebnjtus uauueis, libginmasa edrc emaèi sllieyr sode!

It bases on simplistic distribution algorithm. Word count and letter count from uniform distribution in a pre-defined range (something like 5-20 for words and 2-12 for letters). Letters come from a frequency table giving the percentage of appearance for each letter.

While the result doesn’t look much like real text, it’s good enough to stress basic parsing and typesetting.

There has been a lot of research about text analysis, first for cryptography, next for natural language analysis and Web crawling. Among all of them, there is a nifty one: the n-grams , which describe all the different letter sequences of a fixed length in a given text. The demo on Wolfram Alpha is gorgeous. It shows how combinations grow fast: a simple sentence like “ceramics come from” contains 69 3-grams. Google’s n-grams database (ranging from 1-grams to 5-grams) weights 24 GiB gzip’ed and contains near 1 billion of 3-grams. Amazingly, this number doesn’t increase so much for 4-grams and 5-grams.

Novelang-0.41.1 released!

Download Novelang-0.41.1 here !

• Fixed bug with Promoted Tags, not detected under some circumstances.

Novelang-0.41.0 released!

Download Novelang-0.41.0 here !

• New feature: Promoted Tags. Implicit Tags matching Explicit Tags become Promoted Tags.

• Support lines of literal inside paragraphs inside angled bracket pairs.

• Minor enhancements on HTML default stylesheet.

Novelang-0.40.1 released!

Download Novelang-0.40.1 here !

• Fixed display bug on generated documentation.

Novelang-0.40.0 released!

Download Novelang-0.40.0 here !

• Brand new stylesheet for HTML.

HTML default stylesheet improvements

A new default HTML stylesheet will be available soon. It should improve Novelang usability a lot. Key features are:

• A better look.
• Scaling up with metadata-oriented features.

If a picture is worth a thousand words:

Fluid layout

New layout supports horizontal resize. The column for rendered text may span from 500 to 1000 pixels.

Lines of literal (<pre> tag) wrap if they are too long. Because wrapping only occurs with the white-space : pre-line style, which discards indentation by default. To prevent this, some JavaScript replaces every space character inside a <pre> by a non-brekable space, immediately followed by a zero-width space. This causes a clean-looking wrapping, but text copied in the clipboard has unwanted character.

Overall look

Titles are indented. This is a compromise with the Descriptor feature (described later).

Line spacing is constant, even between two paragraphs, or between a paragraph and an embedded list. There is a slight loss of information (it may be hard to see where a paragraph begins) but this globally increases readability.

Fonts

Font choice has a huge impact on overall look. Chosing fonts is hard stuff, because fonts rendering is hardly the same across Web browsers. Font readability also changes a lot, depending on line spacing, contrast, and other fonts around.

The convention is: serif font for rendered document, sans-serif for extra information like actions and tags. Literal (<pre> and <code>) shows with a fixed-with font, which is serif, too.

After experimenting with a lot of combinations, finally, the winners are:

—  Palatino Linotye for the rendered document. This gorgeous font is a bit more readable than Times New Roman. It’s available on all platform, and looks gorgeous with appropriate contrats (dark grey over light gray instead of black over white).

—  Lucida Grande with Tahoma as second choice. Lucida Grande is highly readable (was chosen as default for Mac OS X), but sophisticated enough to not look “poor” aside Palatino.

—  Courier New is not new at all, but it mixes harmoniously across text in Palatino. Those fonts display much better on Mac OS X, or with Safari on Windows XP.

Descriptors

Descriptors appeared in Novelang-0.39.0, as an experimental feature. They now display with a nice fade and animation, in order to preserve user’s visual landmarks. Descriptor have a vertical bar that helps to see the scope of the descriptor. This vertical bar only shows when Descriptor is discloed.

Descriptor disclosers now appear close to Tag column. This avoids polluting the left margin.

Scalable lists for metadata

On big documents, there can be so many tags they don’t fit in the height on a Web browser’s window. But most of time, they all fit so it’s convenient to have all of them at a fixed position. How to deal with the exception without hurting common case? Having a 2^nd scrollbar in a browser’s frame looks confusing. But the scrollbar has a great feature: it shows that some items are out of sight. One trick could be displaying a huge popup, but this probably means a lot of work for a poor result.

Finally, the solution comes with a fade to grey at the end of the list to show that all items don’t show. A tiny button “unpins” the tag list from the top of Web browser’s window and lets it go to the document’s beginning. So, when entering the “1 % case” we still have a standard behavior.

Here is the Tag tab in its default pinned state (note the fade at the bottom and scrollbar position):

Unpinning causes it to scroll with the rest of the document:

In addition to Tags, there will be, in a (hopefully near) future, more metadata like Identifiers. Tags show up under a tab bar where it’s easy to add new tabs.

Novelang-0.39.2 released!

Download Novelang-0.39.2 here !

• Reject diacritics in tags.

• Filter on implicit tags as on explicit tags.

Novelang-0.39.1 released!

Download Novelang-0.39.1 here !

• Fixed documentation generation.

Novelang-0.39.0 released!

Download Novelang-0.39.0 here !

• Experimental feature: descriptors in default HTML stylesheet.

• Support Unicode files starting with a BOM (Byte Order Mark).

• Reject TAB character.

• Display Unicode name of a rejected character (as long as it belongs to the set of 16 bit Unicode characters as defined in Unicode 5.2 standard). This feature is experimental and may wreck existing error messages.

Descriptors in HTML

Designators are identifiers and tags. Source document may contain explicit designators, but since version 0.37.0 Novelang is smart enough to generate implicit designators from the text of source documents. Rules for implicit designators are as intuitive as possible, but it is helpful to show somewhere the implicit designators Novelang generated for you.

So default HTML stylesheet introduce a new artefact called “descriptor”. It’s a text area surrounding a paragraph or a level title, that unfolds for displaying implicit descriptors and maybe other useful things in the future, like location in the source document.

Looks like this:

The descriptor shows:

• Implicit identifier: \\ThisIsASectionWithATitle_andStyleInsideTheTitle
• Implicit tag: ThisIsASectionWithATitle
• Implicit tag: andStyleInsideTheTitle

While working on layout and animation I found those links useful:

JQuery , the must-have JavaScript framework for doing everything with browser’s DOM in an concise and elegant fashion.

Stuff About CSS floats. (As a bonus, found this one about floatless layout , may become useful one day.)

CSS popups are fine but I gave up this way as the popup only appears with mouse pointer over a drop zone, prevents from copy-pasting.

This one about null HTML links also helped to drop bad ideas.

Novelang-0.38.1 released!

Download Novelang-0.38.1 here !

• Fixed occasional crash caused by Implicit Tags.

Novelang-0.38.0 released!

Download Novelang-0.38.0 here !

• Experimental support for Implicit Tags, deduced from level’s title.

Generating human-friendly designators

A Designator helps to locate text fragments. This is a generic term for Tags and Identifiers. With Novelang-0.37.0 come Implicit Identifiers, that make a level title behave as an Identifier. With explicit Identifiers, to reference a level from an insert command you decorate the level with an Identifier like this:

  \\Preamble
== Preamble

This is a preamble, blah blah blah...

And this is how to insert only the Part with “Preamble” title in some Novelang book:

insert file:my-document.nlp \\Preamble

But why duplicating the “Preamble” word? As long it doesn’t collide with another Identifier we should be able to write:

== Preamble

This is a preamble, blah blah blah...

… And use the insert command the same way.

Now with this feature available, it makes sense to support implicit Tags, too. When requesting a document containing only fragments tagged with @Preamble one could expect to see our level with “Preamble” title. The need for Implicit Tags and Identifier came out from documents looking like this:

  \\Preamble
  @Preamble
== Preamble

...

Quite not good, for a typing-savvy tool, is it? So now we need a common rule to create Implicit Tags and Implicit Identifiers out from legal Novelang level titles.

There are some differences between Implicit Tags and Implicit Identifiers.

— Implicit Tags don’t appear in the list of explicitely-defined Tags (in the n:meta/n:tags element).

— One given level title generates only one Implicit Identifier, but it may generate several Implicit Tags. This makes sense for long titles; the longer they are the less likely they are to appear several times in the rendered document. With a simple rule – like breaking on punctuation signs – a long title may generate several meaningful Tags.

Here are some generic rules for crafting Implicit Designators:

— Generate something as close as possible of what a human could write.

— Resolve to a limited set of characters that comply with the specification of a URL . By now, Tags appear in the URL-like document request as parameters. There is a chance to support Identifiers as document request parameters, too.

To make a long story short, the RFC lists diacriticless letters, digits and "$-_.+!*'()," characters as legal part of a URL. We can note there is non-uniform support of punctuation signs (! supported but not ? and :). For this reason, we exclude punctuation signs. Same for paired delimiters. The asterisk, plus sign, and dollar sign don’t appear as document construct (they may only appear under some escaped form), so we exclude them too. Only remain low line _ and hyphen minus -.

Implicit Tags split on punctuation signs, while Implicit Identifiers must keep them by some mean. By disallowing the low line in Tag syntax, we save it for punctuation sign replacement for Implicit Identifiers.

The hyphen minus may replace space character. But forcing character case to camel case makes shorter Designators, while keeping them quite readable. Camel case only happens for whitespace stripping between two adjacent words.

Samples:

Document source   Implicit Designator
aéœ               aeoe
x, yz             x_yz      -> 2 Tags: @x  @yz 
X, yz             X_yz      -> 2 Tags: @X  @yz 
v `0.1.2`         v0-1-2
Foo bar           FooBar
foo bar           fooBar
foO BAR           foOBAR
w (x yz)          w_xYz     -> 3 Tags: @w  @x  @yz