Draft proposal for a Reveal Codes Pane

Table of Contents

1 Introduction	1

2 Description of the feature	1

3 Implementation concept	2

4 Two notes on notation	2

5 Behavior of the Reveal Codes pane	3

6 Detailed description of the tokenization/detokenization process	4

6.1 In-paragraph attributes	4

6.2 Cross-paragraph attributes	4

6.3 Overlapping boundaries	5

7 Visual features of the Reveal Codes pane	5

Introduction

There are some important reasons why WordPerfect is still much loved by its users; first of all, its typesetting power, much of which comes specifically from its stream-based content/formatting model. Strongly tied to this model is one of the most beloved features of WordPerfect: the Reveal Codes pane. Due to its particular nature, this feature allowed WordPerfect to give important visual feedback on where and how attributes and styles where applied, allowing user to fine-tune formatting and correct problems with almost unparalleled speed and power (only programs with similar features, like XyWrite, could match this).

This is the reason why the possibility for a Reveal Codes-like functionality pops up now and then in discussions about development of other word processors, often pushed by disgruntled WordPerfect users. Personally, I have always been of the opinion that something similar, even if maybe not as easy to implement and/or possibly not as powerful and flexible, can be developed for essentially any text content/formatting model. In particular, after having a look at the draft document http://sw.openoffice.org/drafts/text_formatting.html I came to the conclusion that Reveal Codes functionality is indeed feasible in OpenOffice.org; the purpose of this document is to propose some possible ideas on its implementation.

Description of the feature

The purpose of Reveal Codes is to provide the user with:

full feedback on all attribute changes which may influence formatting; this includes the obvious “visual” attributes (changes in font, size, style, color, etc) as well as less obvious ones, deriving from non-directly-attributive features (active language, hyphenation status, outline/bullets/numbering, etc);

full visual feedback on non-printing and other non-showing elements of the document without cluttering the editing window;

full control on text/attribute/insertion point relative position; in other words, the user should be able to determine and decide with absolute precision and ease whether the text he's going to paste/type is going to be inserted within the scope of any particular active attribute or not (this is especially important at attribute boundary);

full control on the attributes, including but not limited to:

possibility to remove an attribute with a single keyboard action;

possibility to change an attribute by “activating” its visual representation.

The proposed way to achieve this is that of creating a parallel representation of the text which emulates the streamlined text/formatting model which supports Reveal Codes visualization in a more natural way. As will be briefly shown, this can be implemented quite straightforwardly, at least for some attributes; such a streamline emulation representation can probably be implemented without changing the actual internal memory representation of the paragraph or the usual WYSIWYG screen representation.

Implementation concept

I'd like to start with an example, viz. the same available at the afore mentioned address. For what one can deduce from the description there, a text like

This is a paragraph with different character attributes is internally represented as a text string (viz. This is a paragraph with different character attributes) plus an array of attributes; in the specific case:

Attribute	Start	End
Bold	0	19
Size(16pt)	11	28
Italic	17	40
Color(light red)	35	50

Such a representation can be straightforwardly converted into a streamlined one by inserting appropriate on/off virtual tokens before the starting position and after the ending position of each attribute:

(bold on)This is a p(size change)aragra(italic on)ph(bold off) with
diff(size revert)erent (color change)chara(italic off)cter attributes(color revert)

Since, as it was mentioned above, we want to show as much as possible, we may also want to show spaces with some particular symbol (this would be akin to activating View Nonprinting Characters, without cluttering the editing window):

(bold on)This·is·a·p(size change)aragra(italic on)ph(bold off)·with
diff(size revert)erent·(color change)chara(italic off)cter·attributes(color revert)¶

It is also quite immediate to see that the above process of inserting virtual tokens can be fully reversed. While in general a streamlined text/formatting model cannot be cleanly and straightforwardly converted to the representation used by OpenOffice.org, this bidirectional conversion is possible for models created starting from the OpenOffice.org representation: in mathematical terms, there is an injective mapping of the OpenOffice.org model to the streamlined model, but it is not onto. This limitation, while important for the development of import/export filters between the two models, does not influence the implementation of the Reveal Codes pane, since the latter only has to do with virtual tokens created from the OpenOffice.org model.

Two notes on notation

First of all, I'd like to point out that what I say about attributes (manual color/size/font changes, manual activation of bold/italic/underline, etc) also happily applies to styles. In particular, character styles behave as paired attributes. So everything that will be said about attributes must be considered as applicable to styles as well, unless where otherwise specified.

Paragraph style changes (both hard-coded or obtained by applying a paragraph style) can be displayed in the Reveal Codes pane as well: each paragraph that doesn't have the “Default” style would begin with some virtual tokens showing the paragraph style:

(Paragraph Style: Text body)(Alignment: center)Some text.¶

Of course such formatting changes won't have “revert” codes since their scope is automatically the whole paragraph.

The second point I'd like to discuss is that of “flip-flop” attributes. Grossly, it is possible to group character formatting into two classes: that of multi-selection changes (for example: font, font size, font color) and that of two-selection changes (for example: bold, italic, all character styles). Some attributes have a dual nature, though: underline and strike-through are among these: conceptually they can be switched on and off (and are therefore similar to flip-flop attributes), but their “on” value can have multiple (sub)values (underline or strike-through style, in the examples mentioned).

As it may have been noticed in the previous examples, I used different tokens for the begin/end position in the two cases: “revert” in one case, “off” in the other. (The reason for this initial choice of representation is WordPerfect's heritage). A more consistent and appropriate behavior, on the other hand, is that of setting the second code (attribute change end) to be “revert” in all cases. For flip-flop attributes, the first code (attribute change begin) can be either on or off (an initial off code makes sense if for example the surrounding text already has the attribute set: for example a forced “italic off” within text with character style “Emphasis”).

Actually, it would be appropriate to have a third value for the initial token for flip/flop attributes: “switch”; this would start the opposite of the current status for the attribute; it would for example make sense for nested emphasis: within roman text, emphasis would be in italic; within italic text, it makes sense to have emphasis in roman. As an example, consider the attribute “italic” (bold would do as well) and the snippet

text before (italic start change)text inbetween(italic revert) text after

The following table shows the outcome in the six cases given by the three different values for the start code (on, off, switch) and the two “context” values, i.e. whether the snippet is inserted in a text in roman or already italicized:

	Value of start code
Context	On	Off	Switch
Non-italic	Text before text inbetween text after	Text before text inbetween text after	Text before text inbetween text after
Italic	Text before text inbetween text after	Text before text inbetween text after	Text before text inbetween text after

In my opinion the “switch” value, if not already present in the current OpenOffice.org implementation, should be added, because of its power and importance.

Behavior of the Reveal Codes pane

Virtual tokenization of the attributes should of course only happen when the Reveal Codes feature gets activated. In this context, cursor movement and text changing commands are supposed to behave slightly differently from the normal Writer behavior. Noticeable differences:

virtual tokens behave as normal character tokens:

cursor must take them into consideration when moving;

they can be deleted; deleting a virtual token is internally mapped to removal of the corresponding attribute;

they can be moved (dragged):

dragging a virtual token to another place is internally mapped to extending/shortening its range, possibly swapping start/end point if necessary;

dragging it outside the Reveal Codes pane should delete it;

[TODO: point has to be reconsidered] shortcuts to flip/flop attributes creation should create switch/revert pairs:

in the standard OpenOffice.org behavior some shortcuts (namely, those that change attributes like bold, italic, underline, etc) work differently depending on whether they act on selected text or not:

if selection is active, the shortcuts create an attribute change that spans the selected text; one obtains something like:
Some (italic on)italic(italic revert) text

if selection is not active, the first activation of the shortcut calls the appropriate style, the second call turns off the attribute and explicitly turns on the “lack of attribute”; one obtains something like:
Some (italic on)italic(italic revert)(italic off) text(italic revert)

(The examples show what happens when the context is non-italic; “off” and “on” would be swapped if the context was italic.)

when Reveal Codes is on, the saner and more coherent behavior of always creating simple pairs should be followed. The precise behavior depends on where the cursor is when the shortcut is called:

Cursor position	Shortcut action
Away from initial or terminal code for the specific attribute	Create switch/revert pair, put cursor between the two
Left of initial code Right of terminal code	Move cursor inside scope (right of on code or left of off code)
Right of initial code Left of terminal code	Move cursor outside scope (left of on code or right of off code)

Observe that the second and third case can only happen when Reveal Codes is on, since otherwise there is no (virtual) token representing the attribute boundary, and therefore the precise position of the cursor with respect to the boundary cannot be determined. These two cases can be summarized as follows: if the cursor is next to an code relative to the specific attribute, and the relative shortcut is called, switch side of the cursor with respect to the code.

the behavior described in the previous points could leave adjacent initial/terminal codes (e.g. when a pair is created, but the user then inserts nothing); in these cases adjacent tokens relative to the same attribute will simply be removed (as soon as the cursor leaves; i.e., they should be kept while the cursor is between them, so as to allow the user to enter text in the newly created pair; a timeout could be set after the user moves the cursor from between the pair before the pair is removed).

Detailed description of the tokenization/detokenization process

In-paragraph attributes

[TODO; I need to know more details about the internal structures.]

Cross-paragraph attributes

[TODO; in particular, should paired attributes that span multiple paragraphs be “split” on a par-by-par basis? If yes, we can make the whole tokenization process local to paragraphs, and thus much more simple; otherwise (which would seem like the best idea to me, but I need to know how they are treated internally) it would be slightly more complicated, and memory and time consuming; again, more knowledge of the internal workings is needed.]

Overlapping boundaries

There may happen cases when some formatting changes boundaries happen at the same character position; the virtual tokens would therefore appear all in the same position, which defeats one of the purposes of the feature: precise positioning of pointer relatively to formatting changes. A certain order should therefore be assumed by default in the tokenization process, and such order should be (at least locally, i.e. on a case-by-case basis) modifiable by the user. My suggestion for the default order is the following:

put all revert (formatting change end) codes first; if more than one are present, they should be put in “nesting order” (which requires some look-behind): formatting changes started first should end last;

put all formatting change start codes next; if more than one are present, they should be put in “nesting order” (which requires some look-ahead): formatting changes ended last should begin first;

if two (or more) formatting changes both start and end at the same position(s) (viz. imagine a word which is bolded and underlined), they could be represented in the same order as they are represented in memory, in the stack of attribute changes (styles first, hard-coded formatting changes next; both in memory order) [TODO, more knowledge of the internals needed]

[TODO: are examples needed, or is it clear enough?]

The reason why “nesting order” is suggested as default is that “well-formed”, appropriately formatted documents will usually have nested formatting changes.

Of course, users should be allowed to change the order. A means to do this is dragging: if the user wants one style to come before another, he could just drag the appropriate token. [TODO: other ways to do it?]

Visual features of the Reveal Codes pane

[TODO]

In the old DOS days WordPerfect used to represent the codes in a way not much different from the one shown in the previous sections, except that square brackets were used instead of round parentheses, and explicit codes where shown for line and paragraph terminators (in WordPerfect lingo, “soft” and “hard” returns). With the advent of graphical interfaces, the codes were turned into fully-functional buttons: both their display and their behavior was improved; codes became button-shaped, aesthetically providing a more pleasant display, but also active elements: double-clicking on them would call an appropriate dialog (style editor, font selector, etc). Line and paragraph terminators are still explicitly shown, to avoid confusion for cases when the lines in the Reveal Codes pane would contain so much material to wrap before the “real” line would. A screenshot follows.

Reveal Codes: basic example

For those who missed it: this is a quote from the title of a paper written by Donald Ervin Knuth

Giuseppe Bilotta

2003-04-15T13:12:10