2009-06-25
2009-06-21
Autotagging
The Tag feature is great. It turns out that I’m mainly using it to tag levels, and too often it leads to duplicate information like this:
@Graphics == Graphics ...
What I want is to simply write:
== Graphics ...
And then Novelang should guess that “Graphics” matches the Graphics tag. I call this “Autotagging”.
We need a few simple tranformations rules to turn titles into tags.
| “foo” | @foo |
| “Foo” | @Foo |
| “Foo bar” | @Foobar |
| “Foo, bar” | @Foo @bar |
| “Foo. Bar” | @Foo @Bar |
| “Foo-bar” | @Foo-bar |
Automatically-generated tags wouldn’t be part of the tag list, but they would be used for the matching of a known tag with existing level titles.
Depending on the document, this behavior may produce a lot of noise so it requires explicit activation.
Of course, the default tag list (with checkboxes) has a new option to enable autotagging, with some JavaScript adding an autotag parameter in the URL.
2009-06-19
Novelang-0.30.0 released!
Latest release available here.
This release fixes a few bugs and brings some minor enhancements. See documentation for details.
2009-06-07
Book index
Start small, think big: while I don’t even provide a decent default stylesheet, I’m not afraid to blog about a tough subject: book index.
An index is a table at the end of a book, referencing pages or chapters containing a pertinent usage of a key word.
A very simple kind of index could look like this:
icons, 21, 136, 138
You can have ranges:
gender in language, 4-6
References avoid duplicates:
GUIs, see graphical user interfaces (GUIs)
Words can group on several levels (3 being the maximum):
keys capitalizing name of, 68 typographic conventions for, 144, 147 writing about, 142
Simplest representation
How could Novelang help to represent this?
The simplest thing to do is to use some kind of delimiter to tell that a word is an index entry. (As usual the meaning of the delimiter would be a matter of stylesheet.)
The {icon} is displayed on the corner.
Obviously, this doesn’t work, because we want the index entry to show a plural. So we invent some new kind of syntactic form representing a tuple (which symbols are used doesn’t matter at this stage).
The { icon | icons } is displayed in the corner.
Great, but how to model several levels of entries? We could make the source document look like this:
In this case we must
{ capitalize | { keys | capitalizing name of } }
the name of a key.
This feels just unreadable!
External index entry declaration
The trick: split index entry declaration from its complete definition, using several files. Complete definition would rely on a subset of Novelang grammar for source files. Example above becomes:
%% source document:
In this case we must
{ capitalize | capitalizing-name-of-key }
the name of a key.
%% index definition file:
capitalizing-name-of-key
- capitalizing name of
- keys
This deserves a few explainations. The first capitalize in the source document is still what’s displayed. The capitalize-name-of-key is the entry name that is not supposed to be read by anybody else than the document writer – could be 123456 as well. The capitalizing name of in the index definition file is what to display in the index. The keys subitem is the parent item.
Because name of the index entry has no semantic meaning, we can let Novelang generate it using simple replacement rules (spaces becoming hyphen minus…). Explicit names are useful for special cases (like homonyms) but now we expect to be able to write:
%% source document:
In this case we must {capitalize} the name
of a key.
%% index definition file:
capitalize
- keys
- capitalizing name of
Because for the same index entry we may have another pertinent words which are not exactly “capitalize” we let the index entry support several names.
%% source document place 1:
In this case we must {capitalize} the name
of a key.
%% source document place 2:
Sometime the name of a key should not be
{in capitals}.
%% index definition file:
in-capitals
capitalize
- keys
- capitalizing name of
The index definition file could support lots of features.Some styling. There are rare cases (like latin names) where italics are required.A kind of markup to tell which words to take in account in alphabetical sort.Multiple posting: the same keyword in the source document has several index entries. This can happen using several embedded list items.“See” and “See Also” references.
Reusing tags and identifiers
Now, entry names may give some feeling of déjà vu. We already have machine-processed names with tags (implemented) and identifiers (not implemented yet). Are entry names just redundant? As tags and identifier apply on a whole paragraph or level, they don’t have the same level of precision when it comes to refer to the exact location of a word. And it turns out that referencing a range of paragraphs (or even chapters) is what we need for supporting page ranges.
The obvious way of handling page range would be to add a new “end of index entry range” here polluting the source document and making it look like LaTeX. This is a convention of the Novelang grammar: avoid end delimiters whenever possible. Tags and identifiers provide a nice extension.
Here is how we use a tag in the index definition file. Instead of tagging, it refers to the tagged text as index entry name:
%% source document: @gender == Stylistic principles === Avoid jargon ... === Avoid sexist language ... === Common gender ... %% index definition file: @gender - gender in language
This is not yet perfect because page ranges are bound to the scope of a tag/identifier. But it seems possible to Some tree-mangling could detect that several tagged nodes appear consecutively:
%% source document: == Stylistic principles === Avoid jargon ... @gender === Avoid sexist language ... @gender === Common gender ...
Tree-mangling could add a special marker as the last child of the last node of the consecutive tagged ones. Our tree would look like:
level
+ level-title "Stylistic principles"
+ level
| + level-title "Avoid jargon"
+ level
| + level-title "Avoid sexist language"
| + tag "gender"
| + paragraph
| | + start-of-range "gender"
| | + ...
| + paragraph
+ level
+ level-title "Common gender"
+ tag "gender"
+ paragraph
+ paragraph
+ ...
+ end-of-range "gender"
Foreseen weaknesses
This is still unperfect because page ranges are bound to the scope of tags/identifiers but this looks like a pretty good approximation.
Another possible drawback is the ability to have overlapping ranges or page numbers with several different tags/identifiers, or index entries in the middle. Because FOP provides no hook to deal with page numbers once they are known, Novelang could not fix a serie of page numbers like 14-17, 15-18, 15, 16. It’s a long way to perfection.
Bibliography
All examples from A Style Guide for the Computer Industry, Sun Technical Publications, 1996.
2009-06-03
Novelang-0.29.0 released!
Latest release available here.
It brings various enhancements to whitespace handling, and the pretty color palette for tags.
2009-05-30
Image processing
In my endless quest of nice libraries to integrate into Novelang, I’ve been wandering about image processing. This makes sense for technical documentation with screen captures; often it is useful to do some rescale of fade. Ccompression is useful, too, but it should be probably be left to the rendering stage.
When updating the captured image, you need to process the image again with your Gimp or whatever. This should be done automatically! I’m looking for a scripting language to do clever things. I’ve no idea on how to integrate it to Novelang – maybe some special files to avoid messing macro-instructions with content.
The language itself could be something like this:
{
rescale( 40% )
fade( SOUTHWEST, 3px )
}
./my-image.png
I want something clever with an explicit representation of pipeline processing. And, yes, it should be all in Java and with a GPL-compatible license. Am I asking too much here?
I’ve found an amazing software piece: ImageJ, a public domain tool for image processing with huge amount of macros and plugins. It seems widely used for science. Bad news, image transparency doesn't look like a great concern. The Alpha Channel plugin is the best I’ve found so far with its rough edges.
NetKernel has a pipeline image processing feature that looks like what I want. But I don’t like their everything-is-a-String approach.
Maybe I shouldn’t be so ambitious, and just start hacking “the smallest thing that could possibly work” for solving my own problem instead of looking for a save-the-world solution.
By the way, this is an interactive rendering effect editor based on BeanShell that may ease some pain while hacking image filters.
2009-05-28
Space character and related stuff
Blocks of literal
Most of times, the text inside blocks of literal should be kept in one piece. A blatant example is a numeric value and its unit.
Compact several spaces into one for the same reason as above.Trim leading and trailing spaces. Otherwise they offer a suspicious mean to override text layout.Replace spaces by non-break spaces.
With the low line character _ figuring the no-break space we’d like to obtain such transformation:
` 20 m ` -> `20_m`
This means a long block of literal with several spaces (transformed into no-break spaces) could become very cumbersome and mess the layout. So we need a hint to allow line breaks at some places. This can be done by splitting the big block of literal into several small ones, which are not separated by spaces.
With the vertical bar character | figuring the zero-width space we have such transformation:
`Y.O.U.``A.R.E.``B.E.A.U.T.I.F.U.L` -> `Y.O.U.|A.R.E.|B.E.A.U.T.I.F.U.L`
See more about the zero-width space here. A quick test shows that FOP supports it.
Implementation will be done at tree-mangling level. A whitespace between two consecutive blocks of literal will be replaced by a special node meaning that a break is allowed here. The special node will be replaced by a no-width space at rendering time.
Apostrophe
This technique could be useful to keep apostrophe character stuck to a word when in last position. By now, Novelang does not take care of the whitespace after or before the apostrophe.
he's here -> he’s here houses' roofs -> houses’roofs during '60 -> during’60
This is because whitespaces are used as separators, but don’t cary “real” information (except in a few cases, like indentation for embedded lists). Before discarding WHITESPACE nodes, the ones immediately preceding or following an apostrophe could become an EXPLICIT_WHITESPACE to be rendered as, yes, a space character.
2009-05-27
Pretty color palette for tags
Default representation of tags attempts to help locating them at a glance, with nice colors. “Nice colors” means a lot of care.
Defining a color palette from scratch is tricky. Colors must be dinstiguishable one from the other. They must spread evenly on the visible spectrum; but this is not easy because the visual effect depends on the display. For this reason, I use the 140 colors of the SVG specification (the same are used in the CSS spec). Much of hard work is done here, including finding pretty names.
But that’s not all. Because the small rectangle of the tag has text, too, there must be a foreground color. First I tried to compute it, using a simple algorithm (increasing Red, Green and Blue of 50% each and applying modulus 255). The text was always barely readable. Not really good.
Another problem was the choice of the color for each tag. I’ve chosen to pick the color of each tag in a predefined list. When all the colors have been set, we start from the start again. This round-robin algorithm for chosing colors is ok, but inside the 140 colors, many look quite the same. Colors like mistyrose and lavenderblush are very close, and if we have only 10 tags, it’s a pity to see two tags looking the same. So it makes sense to edit the color list in order to make the first one look very different. In addition, because those first colors will be picked up the most often, they must be in the same tone (mild saturation).
If there are more than 10 or 20 tags, similar colors will be unavoidable, finally. But, since we display text (and a thin border) there is a foreground color to chose. This gives (140 × 139) 19460 possibilities! Of course background and foreground cannot be the same (hence the 139) and many possibilities are unreadable. But, given a color like white, those colors look quite similar: mintcream, honeydew, ghostwhite, floralwhite, seashell, azure, linen, aliceblue, cornsilk, oldlace, ivory, snow, whitesmoke. Wow!
Maybe there is a clever algorithm to detect which foreground colors give best contrast and distinguishability, but I didn’t find it. It seems much more convenient to let a human do the job.
Since editing some lines of code would require to switch back-and-forth between the code editor and the web browser, I wrote a palette editor based on a HTML page. It looks like this:
It’s easy to change the order of appearance of a color with a drag and drop:
And, after clicking on one color, you set the foreground with an alt-click on wished color.
Don’t forget to save using the Save button (File > Save in Web browser’s menu won’t work). Yes, the color palette editor only runs on Firefox by now.
The color palette is located in:
src/main-resources/style/javascript/colors.htm
This new feature (and the beautiful color palette) will be available in the next release of Novelang (0.29.0).
2009-05-24
Novelang-0.28.0 released!
Latest release available here.
Now tags are handled as query parameters. This is much faster on big documents, and it works for every kind of document.
See documentation for details, and the list of other enhancements.
2009-05-17
Missing closing delimiters
By now, a block with a missing closing delimiter was properly detected as an error, but the error message was ugly. See, for this:
Something -- missing
You got:
line 0:-1 mismatched input '' expecting HYPHEN_MINUS
Not a great deal here, but pretty annoying in a 1000-line long source document.
After a close look, it looked very complex to determine where the error was coming from. Considering this case:
There " is ( something " missing
… The problem is obviously with the unclosed parenthesis. It’s easy to see (for a human) because parenthesis are paired delimiters: there is an opening and a closing one. The double quotes " is single in the sense it may be used for both opening and closing a block, depending on the context. In the example above, the Novelang parser started evaluating a parenthesized block, and the double quote looked like an unclosed block. How to handle this correctly?
— In order to avoid grammar bloat, the grammar emits some kind of events, telling it started parsing a block with such or such delimiter. The position of every token for a start delimiter is kept. If something goes wrong, the error message(s) will report the position of the unclosed delimiter.
— Event consistency check is scoped: if an unclosed delimiter is detected inside a paragraph, this should have no influence on the way unclosed delimiters are handled inside another paragraph.
— When trying to figure where is the opening delimiter with no closing counterpart, the trick is, to look at paired delimiters first. If something went wrong with paired delimiters, just report the errors about them. Otherwise, report errors with single delimiters.
I just checked this new feature into Github and the results are pretty good. Given source document like this (line numbers added for clarity):
1 ( s 2 t -- u 3 v ) 4 5 // w 6 x [ y 7 z //
Instead of a bunch of nonsense, Novelang now returns following problems:
2:2: Missing delimiter. For '--' there should be a matching '--' or '-_' 7:4: no viable alternative at input '\n' 6:2: Missing delimiter. For '[' there should be a matching ']'
This will be available in the next version (0.28.0). Keep informed reading this blog!
2009-05-10
Novelang-0.27.1 released!
Just a fix after I messed MIME type for rendered documents. As usual, available here.
Novelang-0.27.0 released!
Latest release available here.
Novelang-0.27.0 enhances the tag feature with standard HTML stylesheet displaying the list of user-defined tags. In a source document, tags are words preceded by an arrobas @. Levels, paragraphs, paragraphs inside angled bracket pairs (aka blockquotes) and cell rows (aka tables) may be tagged.
@javascript @performance By now this feature all relies on Javascript running inside the Web browser.
HTML generated using default stylesheet renders tags like this, with a nice color set making tags distinguishable at a glance:
It is now possible to hide all the text which is not tagged, selecting tags in a list which appear on topright corner of HTML document, with a fixed position that keeps it always visible and a disclosure box which hides the list by default:
If a level or a set of paragraphs inside angled bracket pairs do have at least one of requested tags, it is displayed with all of its content. If a paragraph has at least one of requested tags, it is displayed, as all its parents (levels or set of paragraphs).
By now this feature all relies on Javascript running inside the Web browser. This doesn’t scale on big documents (with lots of paragraphs and levels). For some big document with HTML generation taking about 13 s, selecting one tag takes more than 70 s and triggers several “slow script” warnings.
A more suitable approach would be to trim the AST (Abstract Syntax Tree) server-side. This requires passing parameters to the query. Because of pre-rendering processing, tag-based filtering would work for any other other format than HTML for free.
There would be less to do in Javascript; it would just update the tag list in order to reflect document’s state.
2009-04-25
Novelang-0.26.0 released!
Latest release available here.
Outstanding new feature: tags. Now you can tag some pieces of the source documents with arbitrary labels:
@my-tag @foo This is a tagged paragraph.
Novelang's default HTML stylesheet generates pretty colorful tags in the margin, and a tag summary at the end. It's intended to be for humans but stylesheets may take advantage of tags, too.
See documentation for details.
2009-04-14
Novelang-0.25.0 released!
Latest release available here.
This version brings relaxed syntax to associate names to URL:
Go to the "website" http://novelang.sourceforge.net
See documentation for details.
2009-04-12
Novelang-0.24.0 released!
Latest release of Novelang available here. This version brings embedded lists. See documentation for details.
2009-04-09
Named URL: try again!
With Novelang-0.23.0 comes a new feature: named URL. The purpose is to associate some text to a URL, in order to let the stylesheet display something nicer than the URL itself. With source document like this:
This is a "url" http://url.net/my-very-long-path .
… we get:
This is a url.
The rationale of this syntax is the consistency with decorations. Decorations are source metadata that is conveniently before decorated source, appearing on its own single line and with some indentation for the visual comfort. It seemed a good idea to follow the same scheme.
But the syntax described above has many drawbacks.
First, it’s very space-consuming. The scarce resource is the vertical space inside the text editor, because you cannot stretch the display device in height (and long horizontal lines are hard to read). So text like this seems to waste space:
This is "url one" http://url.net/1 and here is "url two" http://url.net/2 .
URL must appear on their very own line, for already discussed reasons, so the full stop character at the end must stay as it is. The problem is with the blocks inside double quotes: it’s supposed to remain short, so reserving a whole line for it is obviously a waste. Finally I’d like to write text like this:
This is "url one" http://url.net/1 and here is "url two" http://url.net/2 .
One question then arises: how to distinguish a block associated to a URL from one which is not? After all you may need to display some text in double quotes right before some URL. First I thought about a new “attach” operator which would tell explicitely that some block inside double quotes is related to the following URL:
Stupid: "url" ~ http://url.net
This is not a good idea because experience shows that, 99 % of times, the block is related to the following URL. So it makes no sense to make the most common case a special thing which breaks the consistency of the grammar. And what if the text of the URL should appear inside double quotes? Is there some new clever escape mechanism to invent?
Corresponding Abstract Syntax Tree is also broken, in the sense where n:external-link and n:link-title nodes do carry semantic meaning, while I claim everywhere that such meaning is confusing when stylesheet defines alternate meaning. The n:external-link was a clever idea to wrap the URL and the title in one single element, but I should find something else.
The solution
When a block inside double quotes, or a block inside square brackets, are located right before a URL, they become URL children. Considering such source document:
This is a ["url"] http://url.net .
… we get something like this (consistent with stylesheet’s rendering of block inside double quotes):
This is a “url”.
In the rare case where a block inside double quotes must appear verbatim, we “break” the proximity with some “invisible” character which is an empty block of literal inside grave accents. That’s a little weird but it’s not a problem as it should remain the exception:
That's a "url" `` http://url.net
… so it renders like this:
That’s a “url” http://url.net.
Finally, the n:external-link disappears in favor of n:url. The n:link-title becomes a n:block-inside-double-quotes or n:block-inside-square-brackets. The text of the URL gets wrapped inside a n:url-literal and that’s all.
n:url + n:url-literal + n:block-inside-double-quotes
2009-03-22
Novelang-0.23.0 released!
Latest release of Novelang available here. This version brings named URL (called "external-link"). See documentation for details.
2009-03-20
Wiki Creole grammar
Finally I’ve found a Wiki grammar for ANTLR: http://www.riehle.org/2008/01/09/an-ebnf-grammar-for-wiki-creole-10
That’s interesting to compare with Novelang’s one.
— Paragraphs are the same central thing.
— Rule names embed rendering-oriented meaning (like text_boldcontent).
— Some rules embed their own terminator (a list may end by a end_of_list).
— Some predicates look like hand-coded lookahead:
{ input.LA(1) != STAR || (input.LA(1) == STAR && input.LA(2) == STAR) }?
This grammar is just a skeleton, it doesn’t produce an AST tree or whatever.
2009-03-05
Novelang-0.22.0 released!
Latest release of Novelang available here.
This version brings various fixes related to images. See documentation for details.
2009-03-03
SVG support
I knew it, I knew it… SVG support cannot be seamless, at least not at the first try. First I thought that Safari beta 4 was not SVG-enabled but it appears that it displays whole SVG files (with .svg extension in the URL). And, when embedded in a <object> tag, SVG may work, too, as shown here:
http://labs.silverorange.com/archive/2006/january/howtoinclude
So why are Novelang-generated pages not displaying embedded SVG? I noticed that everytime I refreshed the page, Safari downloaded the .svg file. Could that be something about the MIME type? For the page above, Safari’s Web Inspector tells that Content-Type is text/xml. For the Novelang-generated page, there is no Content-Type. I should fix that first.
Camino (same rendering engine as Firefox) understands SVG well, but adds some ugly scrollbars. They disappear when the size is set manually (maybe adding a few pixels). Adding image size will require some SVG parsing, the same way raster images are loaded by the Part to get their true size.
Aside of this, the article on silverorange.com gives a nice trick for replacing SVG by a raster image: just embed the reference to the raster image inside the <object> element. So when Novelang is requested a non-existing .png file, it could try to locate the same file but with a .svg extension and return the rasterized image. As a consequence, the batch mode should anticipate this and produce .png files for all .svg ones.
Yes, the Web server is something to be invented again and again.
Subscribe to:
Posts (Atom)