Xtext's new Formatter API

Moritz Eysholdt, itemis AG
XTEXT’S NEW
FORMATTER API

AGENDA
• motivation
• use cases
• ITextRegionAccess
• IFormattableDocument
• example formatter
• testing

DEFINITION
“improve readability and emphasize structure
of a document
without changing its meaning”

DEFINITION
“improve readability and emphasize structure
of a document
without changing its meaning”
“prettify spaces, line wraps, tabs and indentation
without changing the AST”

…if it does change the AST,
it’s probably a clean-up action or refactoring

I’m lazy
readability
diff & merge
bug prevention
MOTIVATION

I’m lazy
readability
diff & merge
bug prevention
With Formatter:
CTRL+SHIFT+F
MOTIVATION

I’m lazy
readability
diff & merge
bug prevention
With Formatter:
CTRL+SHIFT+F
Without Formatter:
→→ __↓
→→ __↓
¶
↑↑ ↓↓ ¶
MOTIVATION

bad formatting…
- obscures the code’s structure
- unnecessarily consumes brain
cycles to ﬁlter out anomalies
I’m lazy
readability
diff & merge
bug prevention
MOTIVATION

• whitespace-only changes in diffs
are annoying
I’m lazy
readability
diff & merge
bug prevention
MOTIVATION

are annoying
• merge conﬂicts because of
whitespace-only super annoying
I’m lazy
readability
diff & merge
bug prevention
MOTIVATION

are annoying
• merge conﬂicts because of
whitespace-only super annoying
• only commit formatted code!
I’m lazy
readability
diff & merge
bug prevention
MOTIVATION

wrong indentation!
proper formatting would have
increased the chances of
spotting this bug!
I’m lazy
readability
diff & merge
bug prevention
MOTIVATION

on demand
serialization
semantic
quickﬁx
USE CASES

on demand
serialization
semantic
quickﬁx
CTRL+SHIFT+F
….and via context menu
USE CASES

on demand
serialization
semantic
quickﬁx
Use an Xtext grammar to convert an
EMF model to text.
Formatter contributes indentation,
newlines, space, etc.
USE CASES

on demand
serialization
semantic
quickﬁx
Use an Xtext grammar to convert an
EMF model to text.
Formatter contributes indentation,
newlines, space, etc.
Serializer Use Cases:
- non-textual (e.g. graphical) editors
- model migration
USE CASES

on demand
serialization
semantic
quickﬁx
USE CASES
@Fix(IssueCodes.INVALID_FEATURE_NAME)
public void fixName(final Issue issue, IssueResolutionAcceptor acceptor) {
acceptor.accept(issue, "Uncapitalize name", "", "",
new ISemanticModification() {
@Override
public void apply(EObject element, IModificationContext context) {
((Feature) element).setName(toFirstLower(issue.getData()[0]));
}
});
}

messy
formatter
pretty
developer
creates
developer creates formatter, NoEngine!

messy
formatter
pretty
developer
creates
text
model
(AST)
text-
regions
(CST)
in
in
in
Input: Text, AST and CST

messy
formatter
pretty
developer
creates
text
model
(AST)
text-
regions
(CST)
in
in
in
text
replacements
apply
Output: TextReplacements

messy
formatter
pretty
grammar
developer
creates
text
model
(AST)
text-
regions
(CST)
in
in
in
text
replacements
apply
references
access to grammar

fragment = formatting2.Formatter2Fragment auto-inject {}
fragment = formatting.FormatterFragment auto-inject {}
GeneratorFragment for MWE2

import org.eclipse.xtext.formatting2.AbstractFormatter2
!
class StatesFormatter extends AbstractFormatter2 {
!
def dispatch void format(Model model, extension IFormattableDocument document) {
model.regionFor.keyword(";").prepend[noSpace]
}
}
Model:
"machine" name=QualifiedName “;";

!
!
}
}
Model:
extends AbstractFormatter2

!
!
}
}
Model:
dispatch over AST elements

!
!
}
}
1. start with the EObject
Model:

!
!
}
}
2. ﬁnd desired semantic text region using ITextRegionAccess
Model:

!
!
}
}
3. register formatting information on IFormattableDocument
Model:

text:String
Document
getText():String
offset:int
length:int
TextRegion1 *
TextRegions…
…represent a substring
…can overlap
…can be empty

regionForOffset(int offset, int length)
regionForDocument()
resource:XtextResource
ITextRegionAccess
regionForEObject(EObject object)
regionForRootEObject()
the document.

created from node
model or via serializer

getText():String
offset : int
length: int
ITextRegion
regionForDocument()
ITextRegionAccess
returns
regions for anything

non-hidden token 0..n hidden tokens
ISequentialRegion
getText():String
offset : int
length: int
ITextRegion
undefined:boolean
IHiddenRegion
grammarElement:EObject
ISemanticRegion nextprevious
previousnext
regionForDocument()
ITextRegionAccess
returns
strictly alternating
linked list of
hidden/semantic regions

whitespace token comment token
ISequentialRegion
getText():String
offset : int
length: int
ITextRegion
undefined:boolean
IHiddenRegion
previousnext
IHiddenRegionPartparts
IWhitespace IComment
regionForDocument()
ITextRegionAccess
returns

eObject:EObject
IEObjectRegion ISequentialRegion
getText():String
offset : int
length: int
ITextRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
regionForDocument()
ITextRegionAccess
returns
returns
leading
trailing
for AST element

eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing

eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing
/* Copyright */
machine my.X;
!
// First State
state
1. Take Document
1.

eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing
/* Copyright */
machine my.X;
!
// First State
state
Model:
"machine"
name=QualifiedName
";"
elements+=State*;
!
State:
{State} ‘state';
!
QualifiedName:
ID ('.' ID)*;
1. Take Document
2. Parse it using Grammar
1.
2.

!
println(textRegionAccess.toString())
}
}
eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing
/* Copyright */
machine my.X;
!
// First State
state
Model:
"machine"
name=QualifiedName
";"
elements+=State*;
!
State:
{State} ‘state';
!
QualifiedName:
ID ('.' ID)*;
1. Take Document
3. Dump it via textRegionAccess.toString()
1.
2.
3.

!
println(textRegionAccess)
}
}
eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing
/* Copyright */
machine my.X;
!
// First State
state
Model:
"machine"
name=QualifiedName
";"
elements+=State*;
!
State:
{State} ‘state';
!
QualifiedName:
ID ('.' ID)*;
1. Take Document
3. Dump it via textRegionAccess.toString()
4. See what we get (next slide)
1.
2.
3.

/* Copyright */
machine my.X;
!
// First State
state
Model:
"machine"
name=QualifiedName
";"
elements+=State*;
!
State:
{State} ‘state';
!
QualifiedName:
ID ('.' ID)*;
Columns: 1:offset 2:length 3:kind 4: text 5:grammarElement
Kind: H=IHiddenRegion S=ISemanticRegion B/E=IEObjectRegion
!
0 H "/* Copyright */" Comment:TerminalRule'ML_COMMENT'
16 "n" Whitespace:TerminalRule'WS'
B Model'my.X' Model
16 7 S "machine" Model:'machine'
23 1 H " " Whitespace:TerminalRule'WS'
24 4 S "my.X" Model:name=QualifiedName
28 0 H
28 1 S ";" Model:';'
29 H "nn" Whitespace:TerminalRule'WS'
17 "// First Staten" Comment:TerminalRule'SL_COMMENT'
B State Model:elements+=State path:Model'my.X'/elements[0]
46 5 S "state" State:'state'
E State Model:elements+=State path:Model'my.X'/elements[0]
E Model'my.X' Model
51 0 H
eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing

!
B Model'my.X' Model
28 0 H
28 1 S ";" Model:';'
E Model'my.X' Model
51 0 H
/* Copyright */
machine my.X;
!
// First State
state
Model:
"machine"
name=QualifiedName
";"
elements+=State*;
!
State:
{State} ‘state';
!
QualifiedName:
ID ('.' ID)*;
eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing
keyword
datatype

!
B Model'my.X' Model
28 0 H
28 1 S ";" Model:';'
E Model'my.X' Model
51 0 H
/* Copyright */
machine my.X;
!
// First State
state
Model:
"machine"
name=QualifiedName
";"
elements+=State*;
!
State:
{State} ‘state';
!
QualifiedName:
ID ('.' ID)*;
eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing
empty
ﬁrst region in document
last region in document
1 part
2 parts

!
B Model'my.X' Model
28 0 H
28 1 S ";" Model:';'
E Model'my.X' Model
51 0 H
/* Copyright */
machine my.X;
!
// First State
state
Model:
"machine"
name=QualifiedName
";"
elements+=State*;
!
State:
{State} ‘state';
!
QualifiedName:
ID ('.' ID)*;
eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing

!
B Model'my.X' Model
28 0 H
28 1 S ";" Model:';'
E Model'my.X' Model
51 0 H
/* Copyright */
machine my.X;
!
// First State
state
Model:
"machine"
name=QualifiedName
";"
elements+=State*;
!
State:
{State} ‘state';
!
QualifiedName:
ID ('.' ID)*;
eObject:EObject
IEObjectRegion
undeﬁned:boolean
IHiddenRegion
previousnext
IHiddenRegionPart
children
parts
leading
trailing
begin
end
grammarElement
path

(Recap)
!
!
}
}
Model:

FIND SEMANTIC REGIONS
// in local eObject only
eObject.regionFor
textRegionAccess.regionForEObject(eObject).regionFor
!
// in eObject or any of its children
eObject.allRegionsFor
textRegionAccess.regionForEObject(eObject).allRegionsFor
!
// in whole document
textRegionAccess.regionForRootEObject.allRegionsFor
}
// first matched semantic region
.keyword(",")
.feature(StatesPackage.Literals.STATE__NAME)
.keyword(actionsKeyword_2_1_0)
.assignment(actionsAssignment_2_1_1)
.ruleCall(actionsINTTerminalRuleCall_2_1_1_0)
!
// all matched semantic region
.keywords(",")
.features(StatesPackage.Literals.STATE__NAME)
.keywords(actionsKeyword_2_1_0)
.assignments(actionsAssignment_2_1_1)
.ruleCalls(actionsINTTerminalRuleCall_2_1_1_0)
!
.keywordPairs("(", ")")
{

eObject.regionFor
!
!
}
.keyword(",")
!
.keywords(",")
!
{ITextRegionAccess

eObject.regionFor
!
!
}
.keyword(",")
!
.keywords(",")
!
{ITextRegionExtensions

eObject.regionFor
!
!
}
.keyword(",")
!
.keywords(",")
!
{Keywords (String)

eObject.regionFor
!
!
}
.keyword(",")
!
.keywords(",")
!
{EMF EStructuralFeature (EAttribute, EReference)

eObject.regionFor
!
!
}
.keyword(",")
!
.keywords(",")
!
{Grammar Element (from MyLanguageGrammarAccess)

TEXT REGION RECAP
• ITextRegionAccess (for Xtend, ITextRegionExtensions)

TEXT REGION RECAP
• linked list of strictly alternating ISemanticRegion and
IHiddenRegion

TEXT REGION RECAP
IHiddenRegion
• IHiddenRegion can be empty

TEXT REGION RECAP
IHiddenRegion
• IHiddenRegion contains parts, e.g. IComment and
IWhitespace

TEXT REGION RECAP
IHiddenRegion
IWhitespace
• IEObjectRegion for elements from AST

TEXT REGION RECAP
IHiddenRegion
IWhitespace
• IEObjectRegion for elements from AST
• use toString() during debugging!

!
!
}
}
Model:
"machine" name=QualifiedName “;"; (Recap)

interface IFormattableDocument {
!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!
// ...
}
API:
specify formatting information for hidden regions
Implementation:
store text replacers for text regions

!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
// ...
}
specify
for one
or two
offset : int
length: int
IHiddenRegion
autowrap()
setPriority(int priority)
indent()
newLine()
setNewLines(int newLines)
setNewLines(int min, int default, int max)
noSpace()
oneSpace()
setSpace(String space)
IHiddenRegionFormatter

!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
// ...
}
specify
for one
or two
offset : int
length: int
IHiddenRegion
autowrap()
indent()
newLine()
noSpace()
oneSpace()
!
}
}

!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
// ...
}
specify
for one
or two
offset : int
length: int
IHiddenRegion
autowrap()
indent()
newLine()
noSpace()
oneSpace()
space, newLines, indent, autowrap

!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
// ...
}
specify
for one
or two
offset : int
length: int
IHiddenRegion
autowrap()
indent()
newLine()
noSpace()
oneSpace()
IHiddenRegion ISemanitcRegion IHiddenRegion ISemanitcRegion IHiddenRegion ISemanitcRegion IHiddenRegion
IHiddenRegionPart
1
*
remember the
strictly alternating
list of
hidden/semantic regions

!
!
!
!
!
!
!
// set for provided hidden region
def set(IHiddenRegion hiddenRegion, (IHiddenRegionFormatter)=>void init)
!
// append after the provided semantic region
def append(ISemanticRegion appendAfter, (IHiddenRegionFormatter)=>void init)
def append(EObject appendAfter, (IHiddenRegionFormatter)=>void init)
!
// prepend before the provided semantic region
def prepend(ISemanticRegion prependBefore, (IHiddenRegionFormatter)=>void init)
def prepend(EObject prependBefore, (IHiddenRegionFormatter)=>void init)
!
!
!
!
!
!
!
!
!
!
!
// ...
}
specify for oneIHiddenRegionFormatter IHiddenRegion

!
!
!
!
!
!
!
// set for provided hidden region
def set(IHiddenRegion hiddenRegion, (IHiddenRegionFormatter)=>void init)
!
// append after the provided semantic region
def append(ISemanticRegion appendAfter, (IHiddenRegionFormatter)=>void init)
def append(EObject appendAfter, (IHiddenRegionFormatter)=>void init)
!
// prepend before the provided semantic region
def prepend(ISemanticRegion prependBefore, (IHiddenRegionFormatter)=>void init)
def prepend(EObject prependBefore, (IHiddenRegionFormatter)=>void init)
!
!
!
!
!
!
!
!
!
!
!
// ...
}
specify for one
IHiddenRegionFormatter IHiddenRegion

!
!
!
!
!
!
!
// set for both provided hidden regions
def set(IHiddenRegion r1, IHiddenRegion r2, (IHiddenRegionFormatter)=>void init)
!
// before and after provided semantic region
def surround(ISemanticRegion semanticRegion, (IHiddenRegionFormatter)=>void init)
def surround(EObject owner, (IHiddenRegionFormatter)=>void init)
!
// set for two inwards-bound hidden regions
def interior(ISemanticRegion r1, ISemanticRegion r2, (IHiddenRegionFormatter)=>void init)
def interior(EObject object, (IHiddenRegionFormatter)=>void init)
!
!
!
!
!
!
!
!
!
!
!
// ...
}
specify for twoIHiddenRegionFormatter IHiddenRegion

!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
// ...
}

!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
// ...
}
Why specify formatting for two hidden regions at once?

!
!
!
!
!
!
!
!
!
!
!
!

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
// ...
}
Why specify formatting for two hidden regions at once?
- indent() spans from ﬁrst to second hidden region!

Xtext's new Formatter API

Xtext's new Formatter API

More Related Content

What's hot

Viewers also liked

Similar to Xtext's new Formatter API

Xtext's new Formatter API