The Novelang blog

Good article on O’Reilly Radar: Why an ebook still needs an index

2011-09-03T06:41:00.001+02:00

Read this excellent post, Why an ebook still needs an index on O’Reilly Radar.

We learn why full text search won’t replace a good old-fashioned book index. We also learn that most ebooks don’t have one, because it’s horribly difficult to create a useful semantic index.

Yes, there are plans to include book index into Novelang.

iA Writer

2011-06-26T22:49:00.000+02:00

iA Writer is the most clever text editor I’ve ever seen. Serious. Just watch its 1-minute presentation. It’s a lot of fun. I watched it three times already. They say:

iA Writer for Mac is a digital writing tool that makes sure that all your thoughts go into the text instead of the program.

That’s a lot of shared mindset with Novelang. While both tools leverage on a Wiki syntax, iA Writer tackles the problem with a dedicated graphical interface. It’s a minimalistic text editor that dynamically adjusts its display according to a Wiki syntax ( Markdown ). It has no menus, no preferences, and, well… almost no features. You pay for getting rid of the crap. It’s a paying application, only available on Mac and iPad, through the AppStore. Yes, it screws GPL lovers, but that’s not a reason to not look closely at what it does.

Markdown

iA Writer relies on the Markdown syntax. I don’t like Markdown syntax , for various reasons.

But Markdown is simple enough to be extremely fast to learn (especially with HTML left out). By sticking to this minimalistic level, iA Writer becomes easy to learn and keeps focused on the text being written.

I should have invented it first!

During a presentation of Novelang for corporate use, a guy asked me: “Do you plan some graphical editor for Novelang?” I answered: “Never, it would lose the purpose of the tool. Maybe I’ll package some syntax highlighting configuration for text editors one day but nothing more.”

This was a decent answer, but iA Writer is more clever. I didn’t envisage dynamic formatting, just because it seems hard to program. One author if IntelliJ IDEA explained that they fully rewrote a Java parser by hand to be fault-tolerant. The rule about parsers is: don’t write them by hand because you’re not clever enough for that. And Novelang syntax is more complex than Java.

I didn’t play with iA Writer so I’ve no hint on how its parser works. Probably the display asks for a best effort given a paragraph, and when the parsing fails there is just no styling. Because of the interactive nature of the application, the user quickly learns how to fix broken text.

Novelang’s strict parser offers strong guarantees about correctness, and that’s a strong point when it comes to generate the final document. But this all-or-nothing approach greatly reduces interactivity. (Interactivity happens later in the Web browser, with some JavaScript to show metadata.)

iA Writer found a new path to the ultimate ease-of-use, while still relying on the Wiki syntax that’s at the heart of Novelang philosophy. I’m just disappointed I didn’t think first about an “augmented” text editor for aiding input of a Wiki text. On the other hand, this wouldn’t have helped me to keep focused on a restrained feature set.

Outsmarting iA Writer?

Spending hundreds of hours of effort to mimick iA Writer sounds like a waste. On the other hand, iA Writer only runs on the Mac/iOS platform and doesn’t offer more than a text editor when inserted in a publication chain. By now, a simple mix of Notepad++ , Novelang and git already offers a complete collaborative solution.

But iA Writer is not the end of the story. By now it works with a full-size keyboard, or with iPad’s keyboard emulation – not the most comfortable thing you can dream about. There are things to invent for a post-iPad world, when handheld devices will have something like a decent virtual keyboard . Starting experiments on a graphical interface with a dedicated parser could be a step in the future.

Conclusion

By now, Novelang is good at what it does, but lacks a few features: intralinks, embeddability, ePub, and desktop integration . It’s important to finish that first.

Novelang could support Markdown as optional syntax. This would make iA Writer the editor of choice and keep Novelang project focused on document agregation and rendering. Two different syntaxes trigger the “double feature alarm”. The source parser could become a pluggable thing, letting other people implement Markdown support. By now, the very few users of Novelang are happy with the out-of-the-box experience and didn’t ask for such geeky features.

Hacking a special text editor component that behaves like iA Writer’s one could be a nice experiment, but it shouldn’t be part of Novelang’s core before years.

AsciiFlow: WYSIWYG ASCIIArt

2011-06-15T05:54:00.002+02:00

AsciiFlow is a drawing tool for producing ASCII art. Currently supports lines and boxes, plus an eraser tool. Looks like the ultimate tool for UML diagrams – no kidding.

Thanks to Julien Kirch who reported this to me first.

Jekyll: a static website generator

2011-03-23T08:51:00.003+01:00

Quoting its presentation, "Jekyll is a simple, blog aware, static site generator. It takes a template directory (representing the raw form of a website), runs it through Textile or Markdown and Liquid converters, and spits out a complete, static website suitable for serving with Apache or your favorite web server."

Jekyll is It’s github's Wiki engine. This is a clever integration: the VCS tracks Wiki page changes, which means the project hosted by github has a wiki and versioned documentation along with sourcecode.

Jekyll is a batch-only generator, so no interactive preview. Markdown and Textile are also more lenient about grammar validation. But Jekyll is probably much lighter than Novelang, which makes it a better choice for github.

Novelang-0.56.0 released!

2011-03-12T01:28:00.000+01:00

Just released Novelang-0.56.0!

Summary of changes:

Added Romanian characters, as listed on Wikipedia. This includes diacritics made obsolete by the spelling reform of 1904, minus the d/D letter with a comma below, for which Unicode offers no precomposed characters.

Download it from here.

Enjoy!

Novelang-Desktop

2011-02-05T13:13:00.000+01:00

This is the result of a conversation with another Novelang user, coming with ideas for a better integration with the desktop environment.

First, as strange as it seems, Novelang is a desktop application. While it embeds a Web server it’s not meant to serve content to remote clients. The support of a Web client is just the fastest approach to give quick preview of files edited in a desktop-integrated text editor. (For serving Novelang documents on the World Wide Web, it’s much faster to generate them as static files first.)

By now the desktop integration relies on a java -jar Novelang-...jar typed in a command-line console. That’s a bit harsh.

The yet-to-develop Novelang-desktop project would be fancier. Novelang-desktop is a graphical application where user sets content roots and associated TCP ports. With this, editing or viewing Novelang documents from multiple content directories doesn’t require to start several JVMs anymore.

Here is a skeletal view of the main window of Novelang-desktop:

 ___________________________________________
|  __________________________               |
| | C:\projects\docs  | 8081 |  [ Add...  ] |
| | I:\shared\foo     | 8082 |  [ Edit... ] |
| |___________________|______|  [ Remove  ] |
|                                           |
| [?]                      [ Configure... ] |
|___________________________________________|

Double-clicking on a table line opens a Web browser with the directory listing of the content root.

The directory listing comes with great improvements. It shows every files, and each Novelang source file has several associated links:

Open in current Web browser (one link per rendition MIME type, like PDF, HTML, FO, XML).
Open in default editor.
Batch generation.

For the last two there is a trick: the link loops back to a special service of Novelang-desktop, which has all the powers to execute arbitrary code and scan local filesystem.

For editor opening it executes a pre-configured command with the file path as an argument.
For batch generation first it opens a dialog requesting target directory.

Novelang-desktop’s configuration goes in user’s home directory by default.

Novelang-0.55.0 released!

2011-02-04T21:15:00.001+01:00

Just released Novelang-0.55.0!

Summary of changes:

New --temporary-dir option. For better error messages, Novelang now buffers the whole document before sending it to the HTTP client. When the document is too big Novelang buffers it into a temporary file under this directory. If an error occurs then its not too late to send an HTTP redirection.
Fixed SVG embedding for PDF. Now the image appears as true vector image inside the PDF. Correct reference to the SVG resource implied adding a $content-directory parameter passed to XSL stylesheets.
Fixed loss of request parameters when issuing error page.
Less verbose logging of Logback configuration at startup.
Fixed various cases of bad problem reporting, where location in origin file was missing.
Plenty of other small fixes.

Download it from here.

Enjoy!

Secrets for embedding SVG images in PDF documents

2011-01-23T00:41:00.005+01:00

Nhovestone reports don’t look good when zooming, because the charts are bitmap images. While trying to render them as vector images, a few interesting things did appear.

(Most of them relate to FOP-1.0 and Batik-1.7 which are the latest versions at this time.)

There was a bug (fixed in HEAD) preventing from embedding SVG images in PDF documents. Novelang resolves the image path (in n:vector-image/n:resource-location) as relative to the content directory. But most of times the stylesheet doesn’t reside right under the content root, so FOP (the PDF renderer) couldn’t find the SVG file. Now XSL stylesheets receive a $content-directory parameter to solve this.

There are several manners to embed an SVG file into a PDF. In order to keep the vector graphics (means: avoiding an ugly conversion to a bitmap) FOP needs a rather twisted declaration of the form:

<fo:instream-foreign-object>
  <svg:svg width="123px" height="456px" >
    <svg:image width="123px" height="456px" xlink:href="file:/my/image.svg" />
  </svg:svg>

The documentation explains that width and the height must appear two times, with exactly the same values.

It’s important to note that an SVG image must always have width and height. Moreover, they must be in pixel unit. Pixel unit is bad, as it introduces a dependency to document’s resolution. But units other than pixel don’t work.

Knowing this, the graphs started to look really good. Unfortunately, the grid becomes bigger when zooming in. In theory, the solution is to set line with to a zoom-independant value with vector-effect="non-scaling-stroke" property (thanks to StackOverflow for that hint). Unfortunately, this is SVG 1.2. Batik doesn’t support this yet and some archived mails show this is somewhat tricky to implement.

Conclusion: while SVG embedding in PDF is far from perfect, it’s much better than rasterized images that seemed to be FOP’s approach. In order to get best vector images we'll have to wait for non-scaling stroke feature in Batik, and maybe some FOP upgrade in order to propagate this feature to generated PDF.

Aggregated javadoc with Maven

2011-01-18T21:04:00.003+01:00

This generates a javadoc covering every subproject. From Novelang root project:

mvn -Dnovelang.javadoc.skip=false javadoc:aggregate

Safari 5.0 (Mac OS X 10.6.5, Windows XP 32bits SP3).
Firefox 3.6 (Mac OS X 10.6.5, Windows XP 32bits SP3).
Google Chrome 8.0 (Windows XP 32bits SP3).
Internet Explorer 8 (Windows XP 32bits SP3).

The website also passed W3C validation for HTML 4.0 Transitional.

Some links about CSS layout

2010-12-27T20:20:00.002+01:00

Three Column Stretch : strech isn’t that good, makes too long lines for main text.

Piefecta – A superb 3-col tableless layout – long right col looks good on Safari and zooms well. Tells it deals with various browser bugs. Probably the finest piece of engineering but too many fixes are making it unreadable. We don’t care about supporting IE6. License: unknown.

CSS Fixed Layout #3.1 (Fixed-Fixed-Fixed) zooms correctly. Much simpler. License: unknown.

Elastic-fluid hybrid got it right. It scales up and down in with, staying in fixed limits. Read author’s comments .

Also check this tutorial about elastic layout.

The Holy Grail 3 column Liquid Layout has great explainations. (Only 1 error on W3C validator but the element reported to be missing appears as it should – validator bug?) License: free to use, linkback appreciated. After a close look, it turned out that forcing min-width and max-width for the column (worst case: nesting another div) and using proper text alignment does the job.

Novelang-0.53.2 released!

2010-12-26T20:37:00.002+01:00

Just released Novelang-0.53.2!

Summary of changes:

Indicating error location when something goes bad during XSL transformation.
Minor fixes on HTML documentation.

Download it from here.

Enjoy!

Novelang-0.53.1 released!

2010-12-23T21:53:00.002+01:00

Just released Novelang-0.53.1!

Summary of changes:

Minor cosmetic changes for HTML generation.

Download it from here.

Enjoy!

Novelang-0.53.0 released!

2010-12-23T19:01:00.001+01:00

Just released Novelang-0.53.0!

Summary of changes:

Experimental support for Multipage. See the result in Novelang documentation.

Small logging enhancements.

More restrictive rules when applying XSL stylesheets. Generation now breaks on warnings. This might break existing incorrect stylesheets.

Changed default representation of Fragment Identifiers, both Implicit and Explicit. Removed leading double reverse solidus \\ when rendering (still required in document sources).

Download it from here.

Enjoy!

Rule-based number spelling

2010-11-19T21:47:00.000+01:00

Novelang comes with a Numbering class which formats an integer value in words. This adds a bit of magic when the stylesheet writes "Chapter fourty-two" from a stupid counter.

Currently the Numbering class only supports French and English, and values from 0 to 50 (all values are hardcoded). The ICU project offers the RuleBasedNumberFormat which supports rule-based formatting. This makes easy to support much greater ranges.

XSL mockup for multipage rendering

2010-11-17T22:53:00.005+01:00

Here is how an XSL would render a multipage document.

First, let’s consider the whole document defining the opus:

== One

Some text of level one.

== Two

Some text of level two.

=== Two-one

Some text of level two-one.

The XML form of the document above is:

<?xml version="1.0" encoding="UTF-8" ?>
<opus>
  <level>
    <title>One</title>
    <paragraph>Some text of level one.</paragraph>
  </level>
  <level>
    <title>Two</title>
    <paragraph>Some text of level two.</paragraph>
    <level>
      <title>Two-one</title>
      <paragraph>Some text of level two-one.</paragraph>
    </level>
  </level>
</opus>

Let’s take for granted that Novelang supports XSL metadata. Our multipage-enabled stylesheet would define an embedded stylesheet that transforms a whole opus into a simple map of page names and page paths. A path is whatever the stylesheet may reprocess, but an XPath expression is quite good. For the document above, here is how our map could look like, if we want to support 2 levels:

page1 -> /opus/level[1]
page2 -> /opus/level[2]
page3 -> /opus/level[2]/level[1]

Please note that, at this point, the decisision to support a given depth, or exclude some tagged levels, entirely belongs to the page-extracting stylesheet.

By merging the page map with the opus, we get the XML input for the rendering of one page. Novelang knows which page it is either because it is iterating over all known pages of the map (batch mode), or because the page name is a part of the request issued to the HTTP dæmon.

<op>us>
  <meta>
    <page>
      <name>page2</name>
      <path>/opus/level[2]</path>
    </page>
  </meta>

  <level>
    <title>One</title>
    <paragraph>Some text of level one.</paragraph>
  </level>
  <level>
    <title>Two</title>
    <paragraph>Some text of level two.</paragraph>
    <level>
      <title>Two-one</title>
      <paragraph>Some text of level two-one.</paragraph>
    </level>
  </level>
</opus>

(Note: the n: namespace prefix doesn’t appear here for brevity.)

The stylesheet gets this whole document as input for every page. All what changes is the name, path pair in the meta/page element. The stylesheet needs to know which page it is rendering, and the whole document tree as well, in order to create a navigation bar or any kind of header or footer corresponding to a specially-titled or tagged level of the document.

This involves some XSL trickery: evaluating an XPath expression at runtime. While it’s not part of XPath 1.0 specification, it is a part of semi-official EXSLT communitiy initiative. The dyn:evaluate http://www.exslt.org/dyn/functions/evaluate does that for us. It works well with Xalan-2.7.1 which is the XSLT engine bundled with Novelang (it works a slightly better than JDK’s one).

In the stylesheet below, we save useful expressions into variables.

The root template prints those variables, then a pseudo-navigation bar made of nested lists.

The nested loop for iterating over level elements is rather ugly but it makes sense as we don’t want infinite deph of titles in a navigation bar.

The title-with-locator template just adds bold on the title in the navigation bar that corresponds to current page.

All other templates mimic Novelang’s standard rendering.

<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:dyn="http://exslt.org/dynamic"
    extension-element-prefixes="dyn"
>
  <!-- Be sure to use Xalan-2.7.1 (not JDK's default). -->

  <!--
    Here, expect a meta section, embedding a stylesheet 
    that extracts the pages we'll find in the meta section 
    of input document.
  -->

  <xsl:output method="html" />

  <xsl:variable name="page-name" select="/opus/meta/page/name" />
  <xsl:variable name="page-path" select="/opus/meta/page/path" />
  <xsl:variable name="page-nodeset" 
      select="dyn:evaluate( $page-path )" />
  <xsl:variable name="page-id" 
      select="generate-id( $page-nodeset )" />

  <xsl:template match="meta/page" >
    $page-name=<xsl:value-of select="$page-name" />
    $page-path=<xsl:value-of select="$page-path" />
    $page-id=<xsl:value-of select="$page-id" />
  </xsl:template>

  <xsl:template match="/opus" >
    <html>
      <xsl:apply-templates select="meta" />

      <!-- Navigation bar -->
      <ul>
        <xsl:for-each select="level">
          <li>
            <xsl:call-template name="title-with-locator"/>
          </li>
          <xsl:if test="level">
            <ul>
              <xsl:for-each select="level">
                <li>
                  <xsl:call-template name="title-with-locator"/>
                </li>
              </xsl:for-each>
            </ul>
          </xsl:if>
        </xsl:for-each>
      </ul>

      <!-- Document body, same templates as usual -->
      <xsl:apply-templates select="$page-nodeset" />

    </html>

  </xsl:template>


  <xsl:template match="paragraph" >
    <p>
      <xsl:value-of select="." />
    </p>
  </xsl:template>

  <xsl:template match="title" />

  <xsl:template match="level" >
    <h2><xsl:value-of select="title" /></h2>
    <xsl:apply-templates/>
  </xsl:template>

  <xsl:template match="level/level" >
    <h3><xsl:value-of select="title" /></h3>
    <xsl:apply-templates/>
  </xsl:template>

  <xsl:template name="title-with-locator" >
    <xsl:text>
    </xsl:text>
    <xsl:choose>
      <xsl:when test="generate-id( . ) = $page-id" >
        <b><xsl:call-template name="title-alone" /></b>
      </xsl:when>
      <xsl:otherwise>
        <xsl:call-template name="title-alone" />
      </xsl:otherwise>
    </xsl:choose>
  </xsl:template>

  <xsl:template name="title-alone" >
    Title: <xsl:value-of select="title" />
  </xsl:template>

</xsl:stylesheet>

Finally, this is how the rendering looks like:

Novelang-0.52.0 released!

2010-11-13T22:51:00.001+01:00

Just released Novelang-0.52.0!

Summary of changes:

Added n:block-inside-asterisk-pairs. Default stylesheet render it as bold.

Download it from here.

Enjoy!

Grammar pattern: twin delimiters

2010-11-13T21:16:00.002+01:00

This post describes a tricky point of Novelang’s grammar design: how to handle twin delimiters like // in a non-ambiguous manner for an ANTLR grammar. It’s a useful refresh before adding long-awaited ** (asterisk pair) delimiter.

The problem

For paired delimiters like ( and ) or [ and ] it’s easy to know when to “open” or “close” a block, and support nested blocks. In contrast, a twin delimiter is an opening one if not preceded by a closing one inside the same block, regardless of what happens in subblocks. This is a complicated way to say we support this kind of nesting:

// block-1 ( block-2 //block-3// ) //

+ block-inside-solidus-pairs
    block-1
  + block-inside-parenthesis
      block-2
    + block-inside-solidus-pairs
        block-3

We also support this:

block-1 // block-2 // block-3 // block-4 //

  block-1
+ block-inside-solidus-pairs
    block-2
  block-3
+ block-inside-solidus-pairs
    block-4

(We have only one level of nesting here. 2 levels of nesting is counter-intuitive and would have required very complex lookahead.)

The pattern

The pattern is to define special grammatical elements when inside a block defined by a twin delimiter, to propagate this element cannot appear again, unless inside some other subblock.

Taking “XXX” for the name of some twin delimiter, here is a simplified version of the grammar for spreadblocks. The term “spreadblock” stands for a block that may spread on several lines (containing single line breaks).

paragraph
  : ... mixedDelimitedSpreadblock
  ;

mixedDelimitedSpreadblock
  : word ( punctuationSign | delimitedSpreadblock ) ...

delimitedSpreadblock
  : xxxSpreadblock
  : parenthesizedSpreadblock
  | squareBracketsSpreadblock
  | doubleQuotedSpreadblock
  | hyphenPairSpreadblock
  ;

parenthesizedSpreadblock  
  : '(' spreadblockBody ')' // Same for other paired delimiters.
  ;

spreadblockBody
  : ... mixedDelimitedSpreadblock
  ;

xxxSpreadblock
  : XXX spreadblockBodyNoXxx XXX
  ;

spreadblockBodyNoXxx
  : ... mixedDelimitedSpreadblockNoXxx ...
  ;

mixedDelimitedSpreadblockNoXxx
  : ... delimitedSpreadblockNoXxx ...
  ;

delimitedSpreadblockNoXxx
  : parenthesizedSpreadblock
  | squareBracketsSpreadblock
  | doubleQuotedSpreadblock
  | hyphenPairSpreadblock
  ;

This is more or less the same for tightblocks. “Tightblocks” stand for blocks containing no line breaks, like cells and embedded lists.

acell  // Same for embedded list items.
  : ... mixedDelimitedTightblock ..
  ;

mixedDelimitedTightblock
  : word ( punctuationSign | delimitedTightblock | ... ) ...
  : word ( punctuationSign | delimitedSpreadblock | ... ) ...
  ;

delimitedTightblock
  : xxxTightblock
  | parenthesizedTightblock
  | squareBracketsTightblock
  | doubleQuotedTightblock
  | hyphenPairTightblock
  ;

xxxTightblock
  : XXX tightblockBodyNoXxx XXX
  ;

tightblockBodyNoXxx
  : ... mixedDelimitedTightblockNoXxx ...
  ;

mixedDelimitedTightblockNoXxx
  : word ( punctuationSign | delimitedTightblockNoXxx ) ...
  ;

delimitedTightblockNoXxx
  : parenthesizedTightblock 
  | squarebracketsTightblock
  | doubleQuotedTightblock
  | hyphenPairTightblock
  ; // That's all.

Thought it is over? There is another kind of block, the delimitedTightblockNoSeparator used inside the subblockAfterTilde which reflects each block inside ~x~y~z! But at this point you probably got the idea.

Yes this makes the grammar quite verbose, but factoring it would reduce ANTLR’s ability to check for inconsistencies. Anyways, the slightest addition brings the need of writing test cases for every logical path inside each ANTLR grammar rule.

Novelang-0.51.1 released!

2010-11-07T19:01:00.001+01:00

Just released Novelang-0.51.1!

Summary of changes:

Upgraded from FOP-0.95 to FOP-1.0. FOP is the library for generating PDF documents.

Various other library upgrades that shouldn’t affect normal users.

Download it from here.

Enjoy!

Novelang-0.51.0 released!

2010-11-06T21:00:00.001+01:00

Just released Novelang-0.51.0!

Summary of changes:

Fixed: list with double hyphen and number sign was using a “plus sign” everywhere (source documents and XML elements). This might break existing documents and stylesheet using this brand new feature.

Download it from here.

Enjoy!

Novelang-0.50.2 released!

2010-11-06T15:30:00.000+01:00

Just released Novelang-0.50.2!

Summary of changes:

Fixed: support paragraphs as lists (n:list-with-triple-hyphen and n:list-with-double-hyphen-and-plus-sign) inside n:paragraphs-inside-angled-bracket-pairs.

Download it from here.

Enjoy!

Novelang-0.50.1 released!

2010-11-06T13:54:00.001+01:00

Just released Novelang-0.50.1!

Summary of changes:

Minor fix on JavaShell for cleaner shutdown when there is no default JmxKit. This only may affect users of Novelang-attirail subproject.

Fixed documentation generation where release notes for SNAPSHOT versions appeared for non-SNAPSHOT versions.

Download it from here.

Enjoy!

Novelang-0.50.0 released!

2010-11-05T22:48:00.001+01:00

Just released Novelang-0.50.0!

Summary of changes:

Embedded numbered lists (n:embedded-list-with-number-sign).

Paragraphs as numbered lists (n:list-with-double-hyphen-and-plus-sign).

Switched to Maven 3. This required no change but future build features may not work with formerly-used Maven 2.2.1.

Download it from here.

Enjoy!

Novelang-0.49.0 released!

2010-10-30T20:59:00.001+02:00

Just released Novelang-0.49.0!

Summary of changes:

In default stylesheet for HTML and PDF, the first n:cell-row element renders as a table header, if there is more than one. This might break existing documents.

Download it from here.

Enjoy!

Maven cheat sheet (update)

2010-10-30T20:36:00.006+02:00

This is an update of previous Maven cheat sheet.

Release

mvn -e --batch-mode clean release:prepare -Dnovelang.build.distribution -DreleaseVersion=M.m.f > build-release-prepare.log

mvn release:perform -Dnovelang.build.distribution -DreleaseVersion=M.m.f > build-release-perform.log

Novelang-0.48.0 released!

2010-10-25T05:47:00.001+02:00

Just released Novelang-0.48.0!

Summary of changes:

Fixed startup option in documentation.

Tags and location for lines of literal. Required an intermediate n:raw-lines element nested inside n:lines-of-literal. This might break existing stylesheets.

Location for cell rows with vertical line.

Download it from here.

Enjoy!

Random text generator for French

2010-10-13T10:42:00.000+02:00

This is a cool one. As it takes random phrases from classical French litteracy, punctuation signs and "typographic grey" look natural.

Cheatsheet template

2010-10-07T12:00:00.000+02:00

Wikipedia definitely offers the right template for a cheat sheet. Short and easy to render with Novelang.

Novelang-0.47.0 released!

2010-10-04T17:56:00.001+02:00

Just released Novelang-0.47.0!

Summary of changes:

Feature removal: relative identifier. Never used, and would make multipage rendering much more complicated.

Small enhancements to Novelang-attirail, the reusable library.

Download it from here.

Enjoy!

Technical study: multi-page HTML rendering

2010-09-20T05:59:00.002+02:00

What we need

How hard would that be to render a single Novelang document over multiple HTML pages? Better ask: how cool would that be? Think about Novelang documentation taking one single huge page. This makes non-linear reading quite uncomfortable. Of course, multi-page rendering should work for both batch and interactive mode.

Technical implications

For batch rendering, there can be a simple approach. Xalan (XSLT rendering engine) offers the redirect extension for redirecting output into a given file.

<xsl:template match="/doc/foo">
  <redirect:write select="@file">
    <foo-out>
      <xsl:apply-templates/>
    </foo-out>
  </redirect:write>
</xsl:template>

Unfortunately, this is not suitable for interactive rendering. For interactive rendering, the endering process must known both:

The requested page, through a URL aware of the page (as sub-part of the whole document).
The whole document, because we may need to render links to other chapters or whatever.

The same need arises for batch rendering but with Xalan’s Redirect extension mentioned above, the whole logic gets buried inside the XSLT (which probably makes it quite complex).

Obviously, we need a Renderer to work the same way for batch and interactive rendering, e. g. there should be no special handling of interactive or batch rendering in the XSL stylesheet. (But multi-page rendering would require a special stylesheet anyways, at least for generating navigation.)

While XSLT-based rendering is the most common case in Novelang, it’s better to think about the general contract of a org.novelang.rendering.Renderer. As it already does, the Renderer should spit bytes into a java.io.OutputStream with no knowledge wether it is a file or a socket. The job of creating the output (which means chosing a file name in the case of batch rendering) is left to some upstream object opening the OutputStream. Currently, this is done by org.novelang.batch.DocumentGenerator or org.novelang.daemon.DocumentHandler which both end by calling DocumentProducer, passing it the OutputStream.

So rendering stage needs additional logic. Interactive rendering implies to extract the requested page from the URL. Batch rendering implies to find the list of pages to create corresponding files on the filesystem.

New Renderer contract

There can’t be unique way to split a document into pages, so we have new responsabilties for our Renderer:Given a document tree (as a org.novelang.common.SyntacticTree) it calculates a list of page identifiers.Given the same document tree, plus a page identifier, it renders the corresponding page to an OutputStream.With something like a single empty page identifier, we should get the same single-page rendering as we have now.

For an XSLT-based Renderer, we should embed page identifiers generation in the same XSL stylesheet (as a part of already-discussed stylesheet metadata ):

<xsl:stylesheet [namespaces blah blah] >

  <nlm:multipage>
    <!-- 
      Some XSL tranformations here,
      starting from  element. 
    -->
  </nlm:multipage>

  ...

Before page rendering occurs, Novelang asks the Renderer for page identifiers. The default XSL-based Renderer applies the content of the element (if there is one) as a stylesheet on the whole document tree. Then it obtains a list of page identifiers as follows:

<pages>
  <page name="Home" >/n:opus
  <page name="ChapterOne" >/n:opus/n:level[1]
  <page name="ChapterTwo" >/n:opus/n:level[2]
<pages>

Of course each page name is unique. In order to achieve this with no tweak, the document tree may embed unique identifiers by extending the semantic of n:implicit-identifier or by adding a new n:unique-identifier element. Node paths seem easy to generate .

Now for each page, Novelang creates the corresponding file out of the page name. If the stylesheet in the did chose filesystem-friendly names, those will be used verbatim (otherwise we may apply some variant of URL encoding). And, for each page, Novelang calls the Renderer with the whole document tree again, and passes additional metadata elements to tell the Renderer which page it is rendering. Input XML looks like this:

<n:opus>
  <n:meta>
    <n:page-name>ChapterOne
    <n:page-path>/n:opus/n:level[1]
  </n:meta>
</n:opus>

This should be enough for the Renderer to figure how to render only the page of interest. It might need to peek elsewhere in the document tree (like for a footer with a copyright notice, or find other chapter names for a navigation bar).

Mix with other features (present or future)

There is an additional role for node identifiers: they might help to “enhance” internal links by adding the prefix corresponding to the target page. (The internal link feature is yet in inception phase. It just seems easier to implement it right after multipage rendering.)

Unique page names

Novelang’s Fragment Identifier is the perfect candidate to generate page identifers. Unfortunately, composite identifier contain the \ character. Should we escape it, or mix it with some weird pseudo-directory feature? But maybe it’s time to remove relative identifiers which never proved useful, and don’t guarantee identifer uniqueness, anyways.

It’s easy to create a new element by adding a simple counter to a colliding identifier. The value for some given document fragment may change across several generations, when adding fragments with colliding identifiers. This won’t be a problem for internal links (links defined by the document itself) prohibit usage of unique identifier. Remember: unique identifiers are only for pure HTML links.

If there is a chance that a foreign HTML documents links to the HTML anchor defined by the unique identifier (in a pure WWWW – World Wide Web Way) then document author should use explicit identifiers.

New URL scheme

With single-page rendering, the rendered document has the same name as the source document (with the difference of the extension). Multi-page adds a new “dimension”. Because the name of the page may collide with another document’s name, the name of the originating document prefixes the page name. Let’s look at different options:

/main/documentation~syntax.html
/main/documentation!syntax.html
/main/documentation,syntax.html
/main/documentation^syntax.html
/main/documentation--syntax.html

Let’s see which character we could use (only checked on Mac OS X, to do: check on Windows):

Character	Escaped?	Comments
`~`	No	Already used for Novelang meta pages.
`-`	No	Already used for Novelang identifiers.
`^`	No	Meaningless in that context.
`#`	No	Fragment in URL.
`,`	No	Hard to distinguish from full stop `.` character.
`!`	No	Hard to read.
`_`	No	Too common in file names.
`+`	No	Used in URL encoding. Usage unrelated to “plus” meaning.
`%`	No	Used in URL encoding.
`=`	Yes	Used in URL encoding. Usage unrelated to “equality” meaning.
`$`	Yes	Overused.
`;`	Yes	Hard to read.
`\|`	Yes	Hard to read.
`'`	Yes	Hard to read.
`&`	Yes	Already used for URL parameters.
`?`	Yes	Already used for URL parameters, DOS wildcard.
`@`	Yes	Inverted meaning if page name appears second.
`{`	Yes	Weird because unpaired. Meaningful otherwise.
`§`	Yes	Mac OS X console doesn’t like it.
`:`	-	Path separator on Unix.

The “Escaped?” column means, it requires escaping on Mac OS X console.

Finally, it turns out that -- looks the best, especially with a variable-width font like in Mac OS X Finder or Windows Explorer.

Special case: if the page identifier was blank, the page separator doesn’t appear so we would still have:

/main/documentation.html

This naming scheme also implies that all pages appear flatly in the same directory. This should help when resolving resource names.

Novelang-0.46.1 released!

2010-09-04T17:45:00.001+02:00

Just released Novelang-0.46.1!

Summary of changes:

Added source packaging for Novelang-attirail subproject.

Download it from here.

Enjoy!

Novelang-0.46.0 released!

2010-08-29T10:10:00.002+02:00

Just released Novelang-0.46.0!

Summary of changes:

New experimental features for code reuse:

Novelang-attirail subproject aggregating various tools. It’s not part of standard distribution, by now it requires separate rebuild.

Pluggable logging implementation.

Java code all under org.novelang package (was novelang).

Download it from here.

Enjoy!

Novelang-0.45.0 released!

2010-08-27T12:16:00.001+02:00

Just released Novelang-0.45.0!

Summary of changes:

Added Greek and Polish characters to the grammar.

Download it from here.

Enjoy!

Novelang-0.44.5 released!

2010-08-08T18:08:00.001+02:00

Just released Novelang-0.44.5!

Summary of changes:

Fixed release notes generation.

Download it from here.

Enjoy!

Novelang-0.44.4 released!

2010-08-08T17:12:00.003+02:00

Just released Novelang-0.44.4!

Summary of changes:

Fixed a few references to old "Part" and "Book" terms, and file suffixes as well.

Download it from here.

Enjoy!

Script for renaming to new extensions

2010-08-08T16:02:00.002+02:00

Here is a Bash script (tested on Mac OS X) that renames every .nlp into .novella and .nlb into .opus. It also changes file content. Use with care.

#!/bin/sh

SED='s/\.nlp/\.novella/g;s/\.nlb/\.opus/g'
for file in `find src modules \( -name *.nlp -o -name *.nlb \) `
do
  newfile=` echo "$file" | sed $SED `
  echo "$file -> $newfile"
  sed $SED < $file > $newfile
  rm $file
done

Novelang-0.44.3 released!

2010-08-07T10:51:00.001+02:00

Just released Novelang-0.44.3!

Summary of changes:

Fixed Nhovestone report generation.

Download it from here.

Enjoy!

Maven cheat sheet (0.44.2)

2010-07-25T22:49:00.005+02:00

There is an updated version of this post.

This is a list of useful Maven commands. They work with Novelang-0.44.2. Later version will probably make some of them less verbose, using some default parameters.

Convention: the Novelang/$ represents the command prompt, with working directory being Novelang’s home directory. Subdirectories appear when needed.

Plugin versions

Stay up-to-date by listing more recent plugins (there is another goal for dependencies):

Novelang/$ mvn versions:display-plugin-updates

Show dependency tree:

Novelang/$ mvn dependency:tree

Feed local repository with fresh artifacts

Novelang/$ mvn clean install

Force child modules version

Force the version of every child module to the one of the parent:

Novelang/$ mvn -N versions:update-child-modules

Performing a release (may be specific to Novelang-0.44.2)

First, clean previous POM backup files:

Novelang/$ mvn release:clean

Then prepare the release. This does the following:

Check VCS state. Includes: no uncommitted file; remote repository sync’ed with local.
Change the POM versions to release version (shown as M.m.f in the snippet below).
Run the build, using declared .
Commit changed POMs to local SCM.
Tag the SCM locally.
Pushes the changes on remote repository, including tags (failing on a conflict).
Revert SCM versions to development version.

Novelang/$ mvn -e --batch-mode release:prepare -Drelease=false -DlocalCheckout=true -DreleaseVersion=M.m.f -DdevelopmentVersion=SNAPSHOT -Dtag=release-M.m.f > build-release-prepare.log

This part is likely to fail. If something goes wrong:

Reset git in the --hard way, to the version immediately before Maven’s changes.
Delete release-M.m.f tag local git repository.
Delete release-M.m.f tag on remote git repository: git push -v github :refs/tags/release-M.m.f
Force synchronization between local git repository and remote one. This may be done by committing an innocuous change, then pushing it with --force option (better idea, anyone?).
Call again: Novelang/$ mvn release:cleanGet sure that’s everything OK with gitk.

Might be useful to reset all POM version (like after some POM or branch hacking): set root pom.xml version to SNAPSHOT and run mvn versions:update-child-modules.

Once this is done, our git repositories contain good, tagged stuff. Last step is to perform the final build.

Novelang/$ mvn release:perform > build-release-perform.log

(There is no additional parameter to pass; the release.prepare did create some POM copies with relevant information.)

The release.perform goal performs a fresh checkout in Novelang/target/checkout where all the pom.xml contain expected M.m.f version. The build calls the deploy:deploy on Novelang-documentation and Novelang-distribution which upload relevant files on SourceForge and send email notifications.

Useful links

Using master password.

Mini-guide about Maven release plugin.

Untested

Resume from a give module folder instead of restarting the build since the beginning:

Novelang/$ mvn reactor:resume -Dfrom=bar

Novelang-0.44.2 released!

2010-07-25T21:43:00.001+02:00

Just released Novelang-0.44.2!

Summary of changes:

Another fix for a build problem. Now the deploy:deploy goal should work properly when called from release:perform.

Download it from here.

Enjoy!

Novelang-0.44.1 released!

2010-07-25T21:08:00.001+02:00

Just released Novelang-0.44.1!

Summary of changes:

Fixed build problem when deploying files and sending annoucements.

Download it from here.

Enjoy!

Novelang-0.44.0 released!

2010-07-24T23:11:00.001+02:00

Just released Novelang-0.44.0!

Summary of changes:

Renamed Part into Novella and Book into Opus. Nicer, clearer. New recommended file suffixes are .novella and .opus. Old .nlp and .nlb suffixes still supported.

Switched build system from Ant to Maven. This should be transparent for users.

Download it from here.

Enjoy!

Syntax highlighter for HTML

2010-06-26T00:03:00.000+02:00

The SyntaxHighlighter project looks nice. It has a "copy to clipboard" feature (implemented in Flash). With some additional hacking, this would save from keeping Novelang's nasty zero-width spaces added for correct line wrapping.

Zipper for faster tree modifications

2010-06-19T08:55:00.000+02:00

Novelang uses immutable trees to represent a document and transform it. While immutable data structure have well-known advantages, Novelang's tree library requires to update every parent node on each change on any child node. Clever guys found how to save those changes when performing multiple local modifications. This relies on a tree structure called Zipper. Here is a very clear explaination, the original paper (site currently down), the Scala implementation and the Clojure one.

Google's font directory

2010-05-30T19:24:00.002+02:00

This is an amazing initiative from Google: a directory for Web fonts. Fonts are available under SIL Open Font License 1.1. There are some beautiful fonts available, with a nice and clear browsing interface. One can download font sources from here.

Novelang-0.43.0 released!

2010-04-26T23:04:00.001+02:00

Download Novelang-0.43.0 here !

Added nohead option to insert command.

Fixed some bugs around identifiers.

Introduced detection of colliding explicit identifiers. This has no useful purpose for now but will serve as a basis for implementing internal links.

Small performance enhancement on HTML document rendering in a Web browser: don’t use JavaScript to set collapsible descriptors hidden.

MinorThird's Mixup

2010-04-25T17:40:00.003+02:00

Could this be useful in Novelang? The Mixup language performs complex queries on pure text. It's "like a regex query, but while regex operates at character level, Mixup operates at token level." Mixup is part of the MinorThird suite and available under BSD license.

Novelang-0.42.0 released!

2010-04-22T21:57:00.000+02:00

Download Novelang-0.42.0 here !

Now requires Java 6.

New Nhovestone report: Novelang has its own benchmark!

Added stylesheet html-FR.xsl for French punctuation.

Performance enhancement on rendered HTML page: when containing many tags it should load faster. Instead of dynamically computing styles on the Web browser, HTML rendered by the server directly includes those styles.

Various performance enhancements on document generation. With the same amount of memory (-Xmx parameter), Novelang handles documents twice bigger and serves them 20 % faster than previous version. Benchmark ran against version 0.41.0 and 0.38.1. This includes buffered reading of Part files, multithreaded Part rendering, and reduced memory consumption when dealing with AST (Abstract Syntax Tree).

Nhovestone

2010-04-05T16:01:00.002+02:00

“Nhovestone” is the name of Novelang’s dedicated benchmark tool, and also a geeky pun .

Nhovestone aims to highlight performance variations across versions using only a few (carefully selected) measurements:How does response time evolve when increasing the number of documents aggregated in a single Book?How does response time evolve when increasing the size of one single document?

Nhovestone doesn’t try to generate an absolute performance index. This is because such an index makes sense only when computed from always the same source documents and the same hardware.

How it works

Nhovestone focuses on HTML generation using default stylesheet, because HTML is great for fast edit-and-review roundtrips. It uses the Novelist to generate pseudo-random text with a realistic structure. For each benchmarked Novelang version, Nhovestone starts a JVM with a small amount of memory (currently -Xmx32M). With few memory the breaking point appears sooner. Nhovestone increases the size of the source document(s) in a linear fashion, and after each increasing, measures how long takes the call of a Novelang instance.

Performance degradation

Response time start to increase exponentially as document becomes fairly big in regard of available memory. This triggers a lot of CPU-intensive garbage collection consuming a lot of time. Nhovestone detects that a running Novelang HTTP daemon gets “strained” when response time gets above a dynamically-computed threshold. The threshold comes from the straight line drawn from a linear regression on the first half of the measurements, with a slope made steeper by a fixed coefficient. When a response time appears above this straight line, the Novelang HTTP daemon got strained and it’s not worth any further measurement.

Adding Parts

This is the first scenario: for each new measurement, there is an additional Part file. All Parts are more or less equal in size and complexity (including level depth). The graph below shows that performance degradation stays linear until the 300^th call. Then, version 0.41.0 starts suffering before older versions. It’s likely that new features require additional memory so starvation occurs sooner.

Increasing the size of the same Part

This is the second scenario: the generated document comes from a single Part file of a size increasing before each call. Each fragment added to the Part has the same size and structure as in the previous test, but all 3 versions show fatigue much sooner (at least 7.5 times). This shows that creating a Part takes much more temporary memory than the finished Part itself.

Tuning

These figures are strongly connected to the volume and the structure of underlying document. Experience shows that small increments generate more measurements (before the fatal strain) and therefore show a more readable trend. They also reduce measurement artefacts that could fool strain detection.

Report generation

JFreeChart generates those graphs. JFreeChart is probably the best charting library for Java at this time, at least on the OSS marketplace. It is stable and highly configurable.

The next step: embed those graphs in a Novelang-generated PDF and publish it as a complement of existing documentation.

The Novelist: random text generation

2010-03-20T17:07:00.000+01:00

Novelang already does all the typesetting for you. What’s next? Writing text, of course! The just-started Novelist subproject, which aims to generate big documents for Novelang testing under heavy load.

Based on French metrics, random text looks like this:

Uomuecto eaufues xuner ig ocanerr, ebanu otpaa. Uuse, on eian aibtd, rttaintlufe elvettarrh, yrn enemlcmlun, ebcazepuer madscg, êiiovemtt teeost eseeerde? Fetn eearréetcs emrseoss icia ntmvesrud. Aoasro cênit ctainetda aèugedet css eali, unero aaie eneoden, nrortio. Oovlod; tfsmenco méttsna, eesdis uoeaeanao rcuent, desungtt av au oneerao, dxuaste umeinétniu lccdeiilne rliùearde veyiritisac yàslu. Iinmseuo odiapqied cmiiapearlo ebnjtus uauueis, libginmasa edrc emaèi sllieyr sode!

It bases on simplistic distribution algorithm. Word count and letter count from uniform distribution in a pre-defined range (something like 5-20 for words and 2-12 for letters). Letters come from a frequency table giving the percentage of appearance for each letter.

While the result doesn’t look much like real text, it’s good enough to stress basic parsing and typesetting.

There has been a lot of research about text analysis, first for cryptography, next for natural language analysis and Web crawling. Among all of them, there is a nifty one: the n-grams , which describe all the different letter sequences of a fixed length in a given text. The demo on Wolfram Alpha is gorgeous. It shows how combinations grow fast: a simple sentence like “ceramics come from” contains 69 3-grams. Google’s n-grams database (ranging from 1-grams to 5-grams) weights 24 GiB gzip’ed and contains near 1 billion of 3-grams. Amazingly, this number doesn’t increase so much for 4-grams and 5-grams.

Novelang-0.41.1 released!

2010-03-15T23:50:00.001+01:00

Download Novelang-0.41.1 here !

Fixed bug with Promoted Tags, not detected under some circumstances.

Novelang-0.41.0 released!

2010-03-14T12:09:00.000+01:00

Download Novelang-0.41.0 here !

New feature: Promoted Tags. Implicit Tags matching Explicit Tags become Promoted Tags.

Support lines of literal inside paragraphs inside angled bracket pairs.

Minor enhancements on HTML default stylesheet.

Novelang-0.40.1 released!

2010-03-08T22:42:00.001+01:00

Download Novelang-0.40.1 here !

Fixed display bug on generated documentation.

Novelang-0.40.0 released!

2010-03-08T22:22:00.001+01:00

Download Novelang-0.40.0 here !

Brand new stylesheet for HTML.

HTML default stylesheet improvements

2010-03-03T22:11:00.004+01:00

A new default HTML stylesheet will be available soon. It should improve Novelang usability a lot. Key features are:

A better look.
Scaling up with metadata-oriented features.

If a picture is worth a thousand words:

Fluid layout

New layout supports horizontal resize. The column for rendered text may span from 500 to 1000 pixels.

Lines of literal (<pre> tag) wrap if they are too long. Because wrapping only occurs with the white-space : pre-line style, which discards indentation by default. To prevent this, some JavaScript replaces every space character inside a <pre> by a non-brekable space, immediately followed by a zero-width space. This causes a clean-looking wrapping, but text copied in the clipboard has unwanted character.

Overall look

Titles are indented. This is a compromise with the Descriptor feature (described later).

Line spacing is constant, even between two paragraphs, or between a paragraph and an embedded list. There is a slight loss of information (it may be hard to see where a paragraph begins) but this globally increases readability.

Fonts

Font choice has a huge impact on overall look. Chosing fonts is hard stuff, because fonts rendering is hardly the same across Web browsers. Font readability also changes a lot, depending on line spacing, contrast, and other fonts around.

The convention is: serif font for rendered document, sans-serif for extra information like actions and tags. Literal (<pre> and <code>) shows with a fixed-with font, which is serif, too.

After experimenting with a lot of combinations, finally, the winners are:

— Palatino Linotye for the rendered document. This gorgeous font is a bit more readable than Times New Roman. It’s available on all platform, and looks gorgeous with appropriate contrats (dark grey over light gray instead of black over white).

— Lucida Grande with Tahoma as second choice. Lucida Grande is highly readable (was chosen as default for Mac OS X), but sophisticated enough to not look “poor” aside Palatino.

— Courier New is not new at all, but it mixes harmoniously across text in Palatino. Those fonts display much better on Mac OS X, or with Safari on Windows XP.

Descriptors

Descriptors appeared in Novelang-0.39.0, as an experimental feature. They now display with a nice fade and animation, in order to preserve user’s visual landmarks. Descriptor have a vertical bar that helps to see the scope of the descriptor. This vertical bar only shows when Descriptor is discloed.

Descriptor disclosers now appear close to Tag column. This avoids polluting the left margin.

Scalable lists for metadata

On big documents, there can be so many tags they don’t fit in the height on a Web browser’s window. But most of time, they all fit so it’s convenient to have all of them at a fixed position. How to deal with the exception without hurting common case? Having a 2^nd scrollbar in a browser’s frame looks confusing. But the scrollbar has a great feature: it shows that some items are out of sight. One trick could be displaying a huge popup, but this probably means a lot of work for a poor result.

Finally, the solution comes with a fade to grey at the end of the list to show that all items don’t show. A tiny button “unpins” the tag list from the top of Web browser’s window and lets it go to the document’s beginning. So, when entering the “1 % case” we still have a standard behavior.

Here is the Tag tab in its default pinned state (note the fade at the bottom and scrollbar position):

Unpinning causes it to scroll with the rest of the document:

In addition to Tags, there will be, in a (hopefully near) future, more metadata like Identifiers. Tags show up under a tab bar where it’s easy to add new tabs.

Novelang-0.39.2 released!

2010-02-21T22:46:00.001+01:00

Download Novelang-0.39.2 here !

Reject diacritics in tags.

Filter on implicit tags as on explicit tags.

Novelang-0.39.1 released!

2010-02-17T21:39:00.001+01:00

Download Novelang-0.39.1 here !

Fixed documentation generation.

Novelang-0.39.0 released!

2010-02-17T21:18:00.001+01:00

Download Novelang-0.39.0 here !

Experimental feature: descriptors in default HTML stylesheet.

Support Unicode files starting with a BOM (Byte Order Mark).

Reject TAB character.

Display Unicode name of a rejected character (as long as it belongs to the set of 16 bit Unicode characters as defined in Unicode 5.2 standard). This feature is experimental and may wreck existing error messages.

Descriptors in HTML

2009-12-27T18:17:00.002+01:00

Designators are identifiers and tags. Source document may contain explicit designators, but since version 0.37.0 Novelang is smart enough to generate implicit designators from the text of source documents. Rules for implicit designators are as intuitive as possible, but it is helpful to show somewhere the implicit designators Novelang generated for you.

So default HTML stylesheet introduce a new artefact called “descriptor”. It’s a text area surrounding a paragraph or a level title, that unfolds for displaying implicit descriptors and maybe other useful things in the future, like location in the source document.

Looks like this:

The descriptor shows:

Implicit identifier: \\ThisIsASectionWithATitle_andStyleInsideTheTitle
Implicit tag: ThisIsASectionWithATitle
Implicit tag: andStyleInsideTheTitle

While working on layout and animation I found those links useful:

JQuery , the must-have JavaScript framework for doing everything with browser’s DOM in an concise and elegant fashion.

Stuff About CSS floats. (As a bonus, found this one about floatless layout , may become useful one day.)

CSS popups are fine but I gave up this way as the popup only appears with mouse pointer over a drop zone, prevents from copy-pasting.

This one about null HTML links also helped to drop bad ideas.

Novelang-0.38.1 released!

2009-12-22T20:42:00.000+01:00

Download Novelang-0.38.1 here !

Fixed occasional crash caused by Implicit Tags.

Novelang-0.38.0 released!

2009-12-21T21:40:00.001+01:00

Download Novelang-0.38.0 here !

Experimental support for Implicit Tags, deduced from level’s title.

Generating human-friendly designators

2009-12-14T20:57:00.002+01:00

A Designator helps to locate text fragments. This is a generic term for Tags and Identifiers. With Novelang-0.37.0 come Implicit Identifiers, that make a level title behave as an Identifier. With explicit Identifiers, to reference a level from an insert command you decorate the level with an Identifier like this:

  \\Preamble
== Preamble

This is a preamble, blah blah blah...

And this is how to insert only the Part with “Preamble” title in some Novelang book:

insert file:my-document.nlp \\Preamble

But why duplicating the “Preamble” word? As long it doesn’t collide with another Identifier we should be able to write:

== Preamble

This is a preamble, blah blah blah...

… And use the insert command the same way.

Now with this feature available, it makes sense to support implicit Tags, too. When requesting a document containing only fragments tagged with @Preamble one could expect to see our level with “Preamble” title. The need for Implicit Tags and Identifier came out from documents looking like this:

  \\Preamble
  @Preamble
== Preamble

...

Quite not good, for a typing-savvy tool, is it? So now we need a common rule to create Implicit Tags and Implicit Identifiers out from legal Novelang level titles.

There are some differences between Implicit Tags and Implicit Identifiers.

— Implicit Tags don’t appear in the list of explicitely-defined Tags (in the n:meta/n:tags element).

— One given level title generates only one Implicit Identifier, but it may generate several Implicit Tags. This makes sense for long titles; the longer they are the less likely they are to appear several times in the rendered document. With a simple rule – like breaking on punctuation signs – a long title may generate several meaningful Tags.

Here are some generic rules for crafting Implicit Designators:

— Generate something as close as possible of what a human could write.

— Resolve to a limited set of characters that comply with the specification of a URL . By now, Tags appear in the URL-like document request as parameters. There is a chance to support Identifiers as document request parameters, too.

To make a long story short, the RFC lists diacriticless letters, digits and "$-_.+!*'()," characters as legal part of a URL. We can note there is non-uniform support of punctuation signs (! supported but not ? and :). For this reason, we exclude punctuation signs. Same for paired delimiters. The asterisk, plus sign, and dollar sign don’t appear as document construct (they may only appear under some escaped form), so we exclude them too. Only remain low line _ and hyphen minus -.

Implicit Tags split on punctuation signs, while Implicit Identifiers must keep them by some mean. By disallowing the low line in Tag syntax, we save it for punctuation sign replacement for Implicit Identifiers.

The hyphen minus may replace space character. But forcing character case to camel case makes shorter Designators, while keeping them quite readable. Camel case only happens for whitespace stripping between two adjacent words.

Samples:

Document source   Implicit Designator
aéœ               aeoe
x, yz             x_yz      -> 2 Tags: @x  @yz 
X, yz             X_yz      -> 2 Tags: @X  @yz 
v `0.1.2`         v0-1-2
Foo bar           FooBar
foo bar           fooBar
foO BAR           foOBAR
w (x yz)          w_xYz     -> 3 Tags: @w  @x  @yz

Novelang-0.37.0 released!

2009-12-13T22:04:00.001+01:00

Download Novelang-0.37.0 here !

Implicit identifiers, deduced from level’s title. See “Identifiers” chapter in Part syntax.

Novelang-0.36.0 released!

2009-09-27T17:41:00.001+02:00

Download Novelang-0.36.0 here !

New --style-dirs command line parameter (superceding --style-dir) for multiple style directories.

Minor identifier-related fixes and internal refactorings.

Novelang-0.35.0 released!

2009-09-20T19:10:00.001+02:00

Download Novelang-0.35.0 here !

Experimental support for identifiers. See “Identifiers” chapter in Part syntax, and “Insert Command” in “Book Files”.

Novelang-0.34.1 released!

2009-09-09T23:15:00.002+02:00

Download Novelang-0.34.1 here !

Fixed: bug preventing from starting a Novelang release with a numbered version.

Novelang-0.34.0 released!

2009-09-09T22:36:00.001+02:00

Download Novelang-0.34.0 here !

New levelabove option for insert book command.

New sort option for insert book command.

New explodelevel batch command for splitting one document’s levels into several parts.

New --content-root command line argument for setting the directory where content files reside.

Fixed: paragraph as list did not support indented embedded list items.

Novelang-0.33.1 released!

2009-08-23T18:10:00.001+02:00

A new release of Novelang is available! Download it here and see documentation for details.

Apostrophe and quotation marks

2009-08-21T20:15:00.002+02:00

Finally, people getting serious about this.

Novelang-0.33.0 released!

2009-08-17T08:15:00.001+02:00

A new release of Novelang is available! Download it here and see documentation for details.

Novelang-0.32.1 released!

2009-08-08T08:31:00.001+02:00

A new release of Novelang is available! Download it here and see documentation for details.

Novelang-0.32.0 released!

2009-08-02T18:38:00.000+02:00

A new release of Novelang is available! Download it here and see documentation for details.

Novelang-0.31.1 released!

2009-07-26T16:00:00.003+02:00

This release brings minor fixes around new block-after-tilde feature. Can be downloaded here. See documentation for details.

Idiom: custom blocks

2009-07-26T15:26:00.000+02:00

Novelang has no semantic markup, insteads it creates an AST (Abstract Syntax Tree) to feed a stylesheet with. This allows creating document-specific idioms, to be handled at stylesheet level. Here is one.

Starting from source document like this:

<<
[INFO] This is an info block.
>>

<<
[WARNING] Beware of "this" paragraph.

(This warning spreads on several paragraphs.)
>>

We want lines of literal to appear in a special manner (like within a frame and with a special icon in the margin). Here is how to achieve this:

<?xml version="1.0"?>
<xsl:stylesheet
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"
   xmlns:n="http://novelang.org/book-xml/1.0"
>
 <xsl:import href="default-html.xsl" />
 <xsl:import href="punctuation-FR.xsl" />

 <xsl:template match="/" >
   <xsl:apply-imports/>
 </xsl:template>

 <xsl:template match="n:paragraphs-inside-angled-bracket-pairs" >
   <xsl:choose>
     <xsl:when 
         test="n:paragraph-regular[1]/n:block-inside-square-brackets[1]='WARNING'" 
     >
       <blockquote>
         <b>WARNING</b><br/>
         <xsl:apply-templates />
       </blockquote>
     </xsl:when>
     <xsl:when 
         test="n:paragraph-regular[1]/n:block-inside-square-brackets[1]='INFO'" 
     >
       <blockquote>
         <b>INFO</b><br/>
         <xsl:apply-templates />
       </blockquote>
     </xsl:when>
     <xsl:otherwise>
       <blockquote>
         <xsl:apply-templates />
       </blockquote>
     </xsl:otherwise>
   </xsl:choose>

 </xsl:template>

 <xsl:template 
     match="n:block-inside-square-brackets[ text()='WARNING' or text()='INFO' ]" 
 />

</xsl:stylesheet>

Rendering document source samples

2009-07-26T14:59:00.001+02:00

A nice feature in the documentation would be to show the Novelang source and the rendering result at the same time. There are several ways to achieve this:

— Duplicate the source code in the Novelang document. One is escaped, one is not. The latter gets rendered in the document itself, in a n:paragraphs-inside-angled-bracket-pairs element with a special tag. For now this won’t work in many cases, like levels or lines of literal.

— Reference a screenshot of a previous rendering. This is the most stupid solution because it’s boring to do and hard to keep up-to-date.

— Be clever and generate the image dynamically from the source snippet.

Rendering tools

How to render a PDF fragment into an embeddable image?

IcePDF claims to be open source but the license doesn’t appear on the Web site and downloading the product requires registration. Anyway, the Java WebStart’ed demo doesn’t display anything except a pair of messages telling it’s a trial version. This behavior was observed on Mac OS X 10.5 and Java 6.

PDFRenderer is available under LGPL. The project seems a bit asleep for now; it looks like a dump-everything-to-the-community effect of Sun’s policy last years. PDFRenderer does a nice job with many PDF, but Novelang-generated ones appear severely broken!

PDFBox is licensed under the Apache License, but contains license notices from Adobe (for AFM fonts) and Sun (for JAI). A close look at PDFBox-7.3.jar shows it embeds those AFM fonts.

Since PDFBox-7.3 doesn’t work (spits an exception), let’s check a snapshot out! This is revision 795516 or something. The build goes well, and image generation doesn’t crash. But the text in images appears seriously damaged! And the font doesn’t look correct. The original was created using Linux Libertine; images contain a Helvetica-like which may not have the same metrics. And all text in non-proportional fonts doesn’t appear at all.

Should I give up my dream of finding an OSS solution for rendering images out from FOP-generated PDF documents? Debugging FOP or PDFRenderer looks like a lot of work. And, while it’s easier to get perfect control on PDF rendering, HTML rendering may be enough for creating the samples.

So here comes Flying Saucer to the rescue. It’s pure Java XHTML renderer which supports CSS 2.1. I’ve used it already and I know it works. The “inheritable” nature of CSS means I can tweak the output a bit (reducing margins and page width) while reusing the default CSS stylesheet.

Finally, all this product review turns to be nonsense, because FOP is supposed to generate images directly ! Insanely great!

Integration to Novelang

Here comes hard stuff. Including external resources depends if the document is self-contained (PDF) or multipart (HTML), and if document is generated by generator (batch) or HTTP dæmon (interactive). As a self-contained document, PDF is generated the same way wether it’s a batch or interactive context.

The FO stylesheet may manage image embedding into the PDF, thus avoiding to spread complexity elsewhere. For SVG, the fo:instream-foreign-object allows direct inclusion of the XML. For images, the architecturally-simple approach would be to write a FOP extension taking the code snippet as parameter, then inserting the rendered image into the Area Tree .

Using external files only makes sense when generating an HTML documents, because we’re pretty sure in this case that user agent won’t request the image before it can read its address from the HTML. For PDF documents, the temporary file must exist before running the FO stylesheet, so it would require some kind of ugly pre-processing.

External files are generated “once-for-all” in batch mode. But, in interactive mode, how long should they live? And does it make sense to write files on the filesystem while the resource could be dynamically generated?

Dynamic resources could be kept in some session-scoped cache. This is how it would work:No need to cache the generated image, only the source snippet. This allows deferred generation.The HTTP session contains several cache areas, one per document name.When a fresh document is generated, reset the whole cache area for this document name.During XSLT processing, call an XSL extension that feeds the cache with snippets.Given a snippet, the cache returns some kind of identifier to be inserted as a link in resulting HTML.A special resource handler (at HTTP dæmon level) queries the cache with the identifiers.If the cache has such a snippet, then it returns it for rendering.

The XSL extension called by the stylesheet could trigger two different behavior, wether it’s dæmon or interactive mode:Use the caching stuff as above.Just write the image file on the filesystem.

How to invalidate the cache? Session expiration is not enough: if several documents exist in the same session, some may become unused therefore causing excessive memory consumption. To avoid this, turn reference to “old” documents to soft references so the JVM would clean them upon memory demand.

Bad behavior would occuring when trying to load an image inside an HTML page after the cache got cleaned by some way (session expiration or memory reclaim) and prior to refreshing the whole page. This sounds like a tolerable annoyance.

Conclusion(s)

I wanted to avoid coding whatever looks like a cache for as long as possible. Now there is a case where caching is linked to a feature out of the performance scope. Anyways, the cache described above is a “toy” cache. Real caching would take the whole resource graph (source documents, images, stylesheets and so one) in account.

Dynamically-generated images could also make sense for rendering ASCII Math for Web browsers which don’t support SVG.

As often, a bit of additional comfort requires a lot of work.

First Novelang demo in the enterprise world!

2009-07-26T14:55:00.001+02:00

Last week I gave an introduction to Novelang at a software publisher’s who’s looking at a collaborative tool for writing it’s product’s documentation. The attenders were the internal IT manager, two technical writers and and an IT intern. Technical writers were enthusiastic, since Novelang is a huge leap from FrameMaker and Robohelp – that’s what they’re working with for now. They had a look at Scenari and Nuxeo . While both are open-source product, the setup fees seemed hugely overpriced.

(In my own humble opinion, Scenari is overengineered crap with excessively complex graphical editor. But it has nice slides to explain the “What You See Is What You Mean” concept. I spent no time looking at Nuxeo for now.)

The technical writers really liked that Novelang never asks for more information that the very minimum required (I guess FrameMaker doesn’t do that). I had a chance to show off with how whitespaces, non-breakable spaces, zero-width spaces, indentation, line breaks, separators, and automatic handling of punctuation typography through a customizable stylesheet. After spending countless hours on those obscure cases that’s great to discover that I’m not the only guy to who this matters (oh, by the way both tech writers are girls).

Now they’re evaluating the product but I already got some feedeback.

With somebody looking over my should, I realized how Novelang installation looks ugly. I unzipped the Novelang archive, did set a pair of system properties, and wrote a .bat file at the root of their working directory, all with my bare hands.

The “local webserver” concept is confusing. For most of people, a webserver is a remote host. This leads to some confusion on collaborative features. They were tempted to store the source document on a shared network drive. On the other hand, they’re not used to source control tools like CVS (the one in use in their company).

The lack of graphical editor looks strange to people who are not accustomed to technical writing.

There is no “guided tour” document nor “cheat sheet” to give an overview of every feature and how to use it best.

I had a deep look at the documents they’re producing for now. The content seems to fit in Novelang syntax. They have some very big tables with one column full of content like lists. For this, a level-based structure seems more appropriate than Novelang’s cell rows.

Having image resolution for PDF hardcoded to 300 dpi won’t work here. At least they need a command-line option until stylesheet metadata gets implemented.

A big requirement is the already-discussed index feature.

To be continued!

Liberation fonts

2009-07-22T20:23:00.003+02:00

Liberation is a superb font set under the GPL+ license, which allows redistribution while not extending the GPL licence to the documents produced with the fonts. Definitely a must-have. It's a 3-family set (Serif, Sans, Mono) with the 4 combinations with bold and italics. At the first glance they look more elegant than Times + Arial + Courier.

Novelang-0.31.0 released!

2009-07-12T22:50:00.003+02:00

So sweet! This release brings better control over whitespace suppression. See documentation for details.

Removing unwanted spaces (continued)

2009-07-12T13:34:00.002+02:00

This is about some kind of brand new operator: it groups all words an blocks and punctuation signs which are not separated by space.

A great feature of Novelang is to apply standard typographic rules, especially when there is punctuation. The problem is, sometimes you can’t apply those rules in a blunt manner.

Consider these cases: on the left, what’s in the source document, and on the right default rendering.

Source document	Default rendering	Hack
`imprimé(e)s`	imprimé (e) s	`imprimé(e)s`
`F.B.I.`	F. B. I. (superfluous spaces)	`F.B.I`
`computer//ing//`	computer ing	No hack available

Default space insertion makes it all wrong. I tried to fix it by detecting proximity (lack of spaces) between casual words and blocks inside grave accents. But, if adding other cases like full stops, blocks inside solidus pairs and blocks inside parenthesis, we end up with many complex tranformations which just break existing whitespace addition for the common case.

The solution is something more generic. I’m thinking about a special character which groups everything that follows until there is a space, a line break or the end of the document. This character would be the tilde ~ because it looks like a kind of elastic ligature.

So, with source document like this:

~computer//ing//

We get an AST (Abstract Syntax Tree) like this:

+ block-after-tilde
  + word "computer"
  + block-inside-pair-of-solidus "ing"

But we still miss the feature of adding zero-width spaces when needed. How to express this? Since zero-width spaces only make sense inside a group with no space, we can reuse the tile character safely.

This:

~A.L.L.~O.F~'E.M.

… becomes:

+ block-after-tilde
  + subblock
    + word "A"
    + punctuation-sign full-stop
    + word "L"
    + punctuation-sign full-stop
    + word "L"
    + punctuation-sign full-stop
  + subblock
    + word "O"
    + punctuation-sign full-stop
    + word "F"
    + punctuation-sign full-stop
  + subblock
    + apostrophe-wordmate
    + word "E"
    + punctuation-sign full-stop
    + word "M"

And this is enough for the stylesheet to find where to insert zero-width spaces.

Foreign characters

2009-06-25T23:06:00.002+02:00

I like seeing all of those strange letters.

Autotagging

2009-06-21T11:18:00.000+02:00

The Tag feature is great. It turns out that I’m mainly using it to tag levels, and too often it leads to duplicate information like this:

  @Graphics
== Graphics

...

What I want is to simply write:

== Graphics

...

And then Novelang should guess that “Graphics” matches the Graphics tag. I call this “Autotagging”.

We need a few simple tranformations rules to turn titles into tags.

“foo”	`@foo`
“Foo”	`@Foo`
“Foo bar”	`@Foobar`
“Foo, bar”	`@Foo @bar`
“Foo. Bar”	`@Foo @Bar`
“Foo-bar”	`@Foo-bar`

Automatically-generated tags wouldn’t be part of the tag list, but they would be used for the matching of a known tag with existing level titles.

Depending on the document, this behavior may produce a lot of noise so it requires explicit activation.

Of course, the default tag list (with checkboxes) has a new option to enable autotagging, with some JavaScript adding an autotag parameter in the URL.

Novelang-0.30.0 released!

2009-06-19T22:39:00.000+02:00

Latest release available here.

This release fixes a few bugs and brings some minor enhancements. See documentation for details.

Book index

2009-06-07T09:13:00.006+02:00

Start small, think big: while I don’t even provide a decent default stylesheet, I’m not afraid to blog about a tough subject: book index.

An index is a table at the end of a book, referencing pages or chapters containing a pertinent usage of a key word.

A very simple kind of index could look like this:

icons, 21, 136, 138

You can have ranges:

gender in language, 4-6

References avoid duplicates:

GUIs, see graphical user interfaces (GUIs)

Words can group on several levels (3 being the maximum):

keys
  capitalizing name of, 68
  typographic conventions for, 144, 147
  writing about, 142

Simplest representation

How could Novelang help to represent this?

The simplest thing to do is to use some kind of delimiter to tell that a word is an index entry. (As usual the meaning of the delimiter would be a matter of stylesheet.)

The {icon} is displayed on the corner.

Obviously, this doesn’t work, because we want the index entry to show a plural. So we invent some new kind of syntactic form representing a tuple (which symbols are used doesn’t matter at this stage).

The { icon | icons } is displayed in the corner.

Great, but how to model several levels of entries? We could make the source document look like this:

In this case we must
{ capitalize | { keys | capitalizing name of } }
the name of a key.

This feels just unreadable!

External index entry declaration

The trick: split index entry declaration from its complete definition, using several files. Complete definition would rely on a subset of Novelang grammar for source files. Example above becomes:

%% source document:

In this case we must
{ capitalize | capitalizing-name-of-key }
the name of a key.


%% index definition file:

capitalizing-name-of-key
- capitalizing name of
  - keys

This deserves a few explainations. The first capitalize in the source document is still what’s displayed. The capitalize-name-of-key is the entry name that is not supposed to be read by anybody else than the document writer – could be 123456 as well. The capitalizing name of in the index definition file is what to display in the index. The keys subitem is the parent item.

Because name of the index entry has no semantic meaning, we can let Novelang generate it using simple replacement rules (spaces becoming hyphen minus…). Explicit names are useful for special cases (like homonyms) but now we expect to be able to write:

%% source document:

In this case we must {capitalize} the name
of a key.


%% index definition file:

capitalize
- keys
  - capitalizing name of

Because for the same index entry we may have another pertinent words which are not exactly “capitalize” we let the index entry support several names.

%% source document place 1:

In this case we must {capitalize} the name
of a key.


%% source document place 2:

Sometime the name of a key should not be
{in capitals}.


%% index definition file:

in-capitals
capitalize
- keys
  - capitalizing name of

The index definition file could support lots of features.Some styling. There are rare cases (like latin names) where italics are required.A kind of markup to tell which words to take in account in alphabetical sort.Multiple posting: the same keyword in the source document has several index entries. This can happen using several embedded list items.“See” and “See Also” references.

Reusing tags and identifiers

Now, entry names may give some feeling of déjà vu. We already have machine-processed names with tags (implemented) and identifiers (not implemented yet). Are entry names just redundant? As tags and identifier apply on a whole paragraph or level, they don’t have the same level of precision when it comes to refer to the exact location of a word. And it turns out that referencing a range of paragraphs (or even chapters) is what we need for supporting page ranges.

The obvious way of handling page range would be to add a new “end of index entry range” here polluting the source document and making it look like LaTeX. This is a convention of the Novelang grammar: avoid end delimiters whenever possible. Tags and identifiers provide a nice extension.

Here is how we use a tag in the index definition file. Instead of tagging, it refers to the tagged text as index entry name:

%% source document:

@gender
== Stylistic principles

=== Avoid jargon

 ...

=== Avoid sexist language

 ...

=== Common gender

 ...


%% index definition file:

@gender
- gender in language

This is not yet perfect because page ranges are bound to the scope of a tag/identifier. But it seems possible to Some tree-mangling could detect that several tagged nodes appear consecutively:

%% source document:

== Stylistic principles

=== Avoid jargon

 ...

@gender
=== Avoid sexist language

 ...


@gender
=== Common gender

 ...

Tree-mangling could add a special marker as the last child of the last node of the consecutive tagged ones. Our tree would look like:

level
  + level-title     "Stylistic principles"
  + level
  |   + level-title "Avoid jargon"
  + level
  |   + level-title "Avoid sexist language"
  |   + tag         "gender"
  |   + paragraph
  |   |   + start-of-range "gender"
  |   |   + ...
  |   + paragraph
  + level
      + level-title "Common gender"
      + tag         "gender"
      + paragraph
      + paragraph
         + ...
         + end-of-range    "gender"

Foreseen weaknesses

This is still unperfect because page ranges are bound to the scope of tags/identifiers but this looks like a pretty good approximation.

Another possible drawback is the ability to have overlapping ranges or page numbers with several different tags/identifiers, or index entries in the middle. Because FOP provides no hook to deal with page numbers once they are known, Novelang could not fix a serie of page numbers like 14-17, 15-18, 15, 16. It’s a long way to perfection.

Bibliography

All examples from A Style Guide for the Computer Industry, Sun Technical Publications, 1996.

Novelang-0.29.0 released!

2009-06-03T08:34:00.002+02:00

Latest release available here.

It brings various enhancements to whitespace handling, and the pretty color palette for tags.

Image processing

2009-05-30T00:39:00.002+02:00

In my endless quest of nice libraries to integrate into Novelang, I’ve been wandering about image processing. This makes sense for technical documentation with screen captures; often it is useful to do some rescale of fade. Ccompression is useful, too, but it should be probably be left to the rendering stage.

When updating the captured image, you need to process the image again with your Gimp or whatever. This should be done automatically! I’m looking for a scripting language to do clever things. I’ve no idea on how to integrate it to Novelang – maybe some special files to avoid messing macro-instructions with content.

The language itself could be something like this:

  {
    rescale( 40% )
    fade( SOUTHWEST, 3px )
  }
  ./my-image.png

I want something clever with an explicit representation of pipeline processing. And, yes, it should be all in Java and with a GPL-compatible license. Am I asking too much here?

I’ve found an amazing software piece: ImageJ, a public domain tool for image processing with huge amount of macros and plugins. It seems widely used for science. Bad news, image transparency doesn't look like a great concern. The Alpha Channel plugin is the best I’ve found so far with its rough edges.

NetKernel has a pipeline image processing feature that looks like what I want. But I don’t like their everything-is-a-String approach.

Maybe I shouldn’t be so ambitious, and just start hacking “the smallest thing that could possibly work” for solving my own problem instead of looking for a save-the-world solution.

By the way, this is an interactive rendering effect editor based on BeanShell that may ease some pain while hacking image filters.

Space character and related stuff

2009-05-28T14:43:00.001+02:00

Blocks of literal

Most of times, the text inside blocks of literal should be kept in one piece. A blatant example is a numeric value and its unit.

Compact several spaces into one for the same reason as above.Trim leading and trailing spaces. Otherwise they offer a suspicious mean to override text layout.Replace spaces by non-break spaces.

With the low line character _ figuring the no-break space we’d like to obtain such transformation:

` 20   m  ` -> `20_m`

This means a long block of literal with several spaces (transformed into no-break spaces) could become very cumbersome and mess the layout. So we need a hint to allow line breaks at some places. This can be done by splitting the big block of literal into several small ones, which are not separated by spaces.

With the vertical bar character | figuring the zero-width space we have such transformation:

   `Y.O.U.``A.R.E.``B.E.A.U.T.I.F.U.L` 
-> `Y.O.U.|A.R.E.|B.E.A.U.T.I.F.U.L`

See more about the zero-width space here. A quick test shows that FOP supports it.

Implementation will be done at tree-mangling level. A whitespace between two consecutive blocks of literal will be replaced by a special node meaning that a break is allowed here. The special node will be replaced by a no-width space at rendering time.

Apostrophe

This technique could be useful to keep apostrophe character stuck to a word when in last position. By now, Novelang does not take care of the whitespace after or before the apostrophe.

he's here     -> he’s here
houses' roofs -> houses’roofs
during '60    -> during’60

This is because whitespaces are used as separators, but don’t cary “real” information (except in a few cases, like indentation for embedded lists). Before discarding WHITESPACE nodes, the ones immediately preceding or following an apostrophe could become an EXPLICIT_WHITESPACE to be rendered as, yes, a space character.

Pretty color palette for tags

2009-05-27T00:00:00.003+02:00

Default representation of tags attempts to help locating them at a glance, with nice colors. “Nice colors” means a lot of care.

Defining a color palette from scratch is tricky. Colors must be dinstiguishable one from the other. They must spread evenly on the visible spectrum; but this is not easy because the visual effect depends on the display. For this reason, I use the 140 colors of the SVG specification (the same are used in the CSS spec). Much of hard work is done here, including finding pretty names.

But that’s not all. Because the small rectangle of the tag has text, too, there must be a foreground color. First I tried to compute it, using a simple algorithm (increasing Red, Green and Blue of 50% each and applying modulus 255). The text was always barely readable. Not really good.

Another problem was the choice of the color for each tag. I’ve chosen to pick the color of each tag in a predefined list. When all the colors have been set, we start from the start again. This round-robin algorithm for chosing colors is ok, but inside the 140 colors, many look quite the same. Colors like mistyrose and lavenderblush are very close, and if we have only 10 tags, it’s a pity to see two tags looking the same. So it makes sense to edit the color list in order to make the first one look very different. In addition, because those first colors will be picked up the most often, they must be in the same tone (mild saturation).

If there are more than 10 or 20 tags, similar colors will be unavoidable, finally. But, since we display text (and a thin border) there is a foreground color to chose. This gives (140 × 139) 19460 possibilities! Of course background and foreground cannot be the same (hence the 139) and many possibilities are unreadable. But, given a color like white, those colors look quite similar: mintcream, honeydew, ghostwhite, floralwhite, seashell, azure, linen, aliceblue, cornsilk, oldlace, ivory, snow, whitesmoke. Wow!

Maybe there is a clever algorithm to detect which foreground colors give best contrast and distinguishability, but I didn’t find it. It seems much more convenient to let a human do the job.

Since editing some lines of code would require to switch back-and-forth between the code editor and the web browser, I wrote a palette editor based on a HTML page. It looks like this:

It’s easy to change the order of appearance of a color with a drag and drop:

And, after clicking on one color, you set the foreground with an alt-click on wished color.

Don’t forget to save using the Save button (File > Save in Web browser’s menu won’t work). Yes, the color palette editor only runs on Firefox by now.

The color palette is located in:

src/main-resources/style/javascript/colors.htm

This new feature (and the beautiful color palette) will be available in the next release of Novelang (0.29.0).

Novelang-0.28.0 released!

2009-05-24T15:53:00.002+02:00

Latest release available here.

Now tags are handled as query parameters. This is much faster on big documents, and it works for every kind of document.

See documentation for details, and the list of other enhancements.

Missing closing delimiters

2009-05-17T22:26:00.003+02:00

By now, a block with a missing closing delimiter was properly detected as an error, but the error message was ugly. See, for this:


Something -- missing

You got:


line 0:-1 mismatched input '' expecting HYPHEN_MINUS

Not a great deal here, but pretty annoying in a 1000-line long source document.

After a close look, it looked very complex to determine where the error was coming from. Considering this case:


There " is ( something " missing

… The problem is obviously with the unclosed parenthesis. It’s easy to see (for a human) because parenthesis are paired delimiters: there is an opening and a closing one. The double quotes " is single in the sense it may be used for both opening and closing a block, depending on the context. In the example above, the Novelang parser started evaluating a parenthesized block, and the double quote looked like an unclosed block. How to handle this correctly?

— In order to avoid grammar bloat, the grammar emits some kind of events, telling it started parsing a block with such or such delimiter. The position of every token for a start delimiter is kept. If something goes wrong, the error message(s) will report the position of the unclosed delimiter.

— Event consistency check is scoped: if an unclosed delimiter is detected inside a paragraph, this should have no influence on the way unclosed delimiters are handled inside another paragraph.

— When trying to figure where is the opening delimiter with no closing counterpart, the trick is, to look at paired delimiters first. If something went wrong with paired delimiters, just report the errors about them. Otherwise, report errors with single delimiters.

I just checked this new feature into Github and the results are pretty good. Given source document like this (line numbers added for clarity):

1   ( s
2  t -- u
3  v )
4  
5  // w
6  x [ y
7  z //

Instead of a bunch of nonsense, Novelang now returns following problems:


2:2: Missing delimiter. For '--' there should be a matching '--' or '-_'
7:4: no viable alternative at input '\n'
6:2: Missing delimiter. For '[' there should be a matching ']'

This will be available in the next version (0.28.0). Keep informed reading this blog!

Novelang-0.27.1 released!

2009-05-10T20:16:00.003+02:00

Just a fix after I messed MIME type for rendered documents. As usual, available here.

Novelang-0.27.0 released!

2009-05-10T19:20:00.004+02:00

Latest release available here.

Novelang-0.27.0 enhances the tag feature with standard HTML stylesheet displaying the list of user-defined tags. In a source document, tags are words preceded by an arrobas @. Levels, paragraphs, paragraphs inside angled bracket pairs (aka blockquotes) and cell rows (aka tables) may be tagged.

  @javascript @performance
By now this feature all relies on Javascript 
running inside the Web browser.

HTML generated using default stylesheet renders tags like this, with a nice color set making tags distinguishable at a glance:

It is now possible to hide all the text which is not tagged, selecting tags in a list which appear on topright corner of HTML document, with a fixed position that keeps it always visible and a disclosure box which hides the list by default:

If a level or a set of paragraphs inside angled bracket pairs do have at least one of requested tags, it is displayed with all of its content. If a paragraph has at least one of requested tags, it is displayed, as all its parents (levels or set of paragraphs).

By now this feature all relies on Javascript running inside the Web browser. This doesn’t scale on big documents (with lots of paragraphs and levels). For some big document with HTML generation taking about 13 s, selecting one tag takes more than 70 s and triggers several “slow script” warnings.

A more suitable approach would be to trim the AST (Abstract Syntax Tree) server-side. This requires passing parameters to the query. Because of pre-rendering processing, tag-based filtering would work for any other other format than HTML for free.

There would be less to do in Javascript; it would just update the tag list in order to reflect document’s state.

Novelang-0.26.0 released!

2009-04-25T07:58:00.002+02:00

Latest release available here.

Outstanding new feature: tags. Now you can tag some pieces of the source documents with arbitrary labels:

  @my-tag  @foo
This is a tagged paragraph.

Novelang's default HTML stylesheet generates pretty colorful tags in the margin, and a tag summary at the end. It's intended to be for humans but stylesheets may take advantage of tags, too.

See documentation for details.

Novelang-0.25.0 released!

2009-04-14T10:14:00.002+02:00

Latest release available here.

This version brings relaxed syntax to associate names to URL:

Go to the "website"
http://novelang.sourceforge.net

See documentation for details.

Novelang-0.24.0 released!

2009-04-12T23:45:00.001+02:00

Latest release of Novelang available here. This version brings embedded lists. See documentation for details.

Named URL: try again!

2009-04-09T10:29:00.002+02:00

With Novelang-0.23.0 comes a new feature: named URL. The purpose is to associate some text to a URL, in order to let the stylesheet display something nicer than the URL itself. With source document like this:

This is a 
  "url"
http://url.net/my-very-long-path
.

… we get:

This is a url.

The rationale of this syntax is the consistency with decorations. Decorations are source metadata that is conveniently before decorated source, appearing on its own single line and with some indentation for the visual comfort. It seemed a good idea to follow the same scheme.

But the syntax described above has many drawbacks.

First, it’s very space-consuming. The scarce resource is the vertical space inside the text editor, because you cannot stretch the display device in height (and long horizontal lines are hard to read). So text like this seems to waste space:

This is
  "url one"
http://url.net/1
and here is 
  "url two"
http://url.net/2
.

URL must appear on their very own line, for already discussed reasons, so the full stop character at the end must stay as it is. The problem is with the blocks inside double quotes: it’s supposed to remain short, so reserving a whole line for it is obviously a waste. Finally I’d like to write text like this:

This is "url one"
http://url.net/1
and here is "url two"
http://url.net/2
.

One question then arises: how to distinguish a block associated to a URL from one which is not? After all you may need to display some text in double quotes right before some URL. First I thought about a new “attach” operator which would tell explicitely that some block inside double quotes is related to the following URL:

Stupid: "url" ~
http://url.net

This is not a good idea because experience shows that, 99 % of times, the block is related to the following URL. So it makes no sense to make the most common case a special thing which breaks the consistency of the grammar. And what if the text of the URL should appear inside double quotes? Is there some new clever escape mechanism to invent?

Corresponding Abstract Syntax Tree is also broken, in the sense where n:external-link and n:link-title nodes do carry semantic meaning, while I claim everywhere that such meaning is confusing when stylesheet defines alternate meaning. The n:external-link was a clever idea to wrap the URL and the title in one single element, but I should find something else.

The solution

When a block inside double quotes, or a block inside square brackets, are located right before a URL, they become URL children. Considering such source document:

This is a ["url"]
http://url.net
.

… we get something like this (consistent with stylesheet’s rendering of block inside double quotes):

This is a “url”.

In the rare case where a block inside double quotes must appear verbatim, we “break” the proximity with some “invisible” character which is an empty block of literal inside grave accents. That’s a little weird but it’s not a problem as it should remain the exception:

That's a "url" ``
http://url.net

… so it renders like this:

That’s a “url” http://url.net.

Finally, the n:external-link disappears in favor of n:url. The n:link-title becomes a n:block-inside-double-quotes or n:block-inside-square-brackets. The text of the URL gets wrapped inside a n:url-literal and that’s all.

n:url
  + n:url-literal
  + n:block-inside-double-quotes