Xtext @ Google

Last week I had the chance to speak at the EclipseDay at the GooglePlex in Mountain View, CA, an event organized by the Eclipse Foundation and hosted by Google’s Open Source Programs Office. Google is a truly amazing company and the GooglePlex is a very exciting place to be at. As I was one day early and needed a creative place to put finishing touches on my slides, I took the chance to visit Robert Konigsberg who was kind enough to host me for that day. The organizers put together a great agenda with lots of interesting talks. Looking at the agenda, you will see that there actually are three major topics, Eclipse in the Enterprise, Modeling and Runtime: The talks

all dealt with how Eclipse can be used in the Enterprise or which challenges you will face if you start using Eclipse in large-scale scenarios. The talks

either directly had modeling as their topic or used modeling technology to solve their problems. I especially liked the NASA talk which featured a number of computer animated videos of recent or future missions. The NASA mission control software is heavily based on Eclipse and technology form the Eclipse Modeling Project. Finally, there were some talk on runtime technology:

Attending great talks of course is one of the major benefits of going to a conference like this. To me, meeting other people and connecting has become even more important than attending great talks. Of course, this is nothing new to event organizers and so there were many chances to grab a drink and chat with the other attendees. With more than 170 people attending, it was hard to have a chat with everyone, but nevertheless I managed to talk to a number of people, many of which already are Xtext users or are now planing to have a detailed look at it. Even Google is using Xtext, but ssshh! Overall, I really enjoyed my two days at Google and look forward to more events like this. Speaking of which, Eclipse Summit Europe is approaching fast, so make sure you register on time. Also, if you’re interested in Modeling and DSLs, we’re organizing a Modeling / DSL Day in North America this coming fall – see my previous post for more info. And finally, here are the slides and the video of my talk:

Eclipse Modeling Day in North America

We (i.e. the Eclipse Foundation, itemis and Cloudsmith) are looking into arranging one more Eclipse Modeling Day in North America this fall and would like to gather feedback from folks interested in attending. Please use this poll to help us tailor a location and program to suit your particular interests:

http://spreadsheets.google.com/viewform?hl=en&formkey=dF ZQcGh3RDhrUUdyNTAtMXlVU2pKQ1E6MA..

Our goal is to organize an event especially for people new to modeling or who are curious what this modeling fuss is all about and how it can help them. So, this will not be an experts meeting, but you can meet experts talking about their topics. We also are interested in case studies, so ideally you will get first-hand experience from people who use modeling in real life to meet real challenges.

We welcome any feedback, so please fill out our little poll.

Getting started with Xtext, part 2

Last week, I showed you how easy it is to create a DSL with Xtext. In this installment, we will have a look at how to leverage the models created with the DSL.

Goal

Let’s imagine we want to create an application for orders. People can sign in to the system, place orders for various items, check out and have them sent to their address. Very simple, but we can show a lot of things here.

As we expect to be writing more than one application of this type and as we also would like to be able to express the structure of the application on a business level (one of the major drivers for DSLs and MDSD for that matter), we come up with the idea of using a DSL to describe what the application does. Defining the DSL is what we did last week. This week, we need to map the concepts of the DSL to some code and some APIs we’re going to program against.

So, we’re going to create a set of code templates for a code generator that can then read our DSL models and create persistence code for us.

Prepare

In order to draw any benefit from this post, you need to have a running version of the DSL we created last time round. If you’re new to this series, please go back one week and follow along the steps outlined in last week’s article. It shouldn’t take you longer than 30 minutes, and I promise we’ll wait here for you.

Create a main template

As we’re probably going to create more than one kind of class (we’ll have Entity beans, DAOs and maybe some interfaces), it is a wise idea to separate all those templates into separate files. The code generator then needs to invoke all those template files in order to generate our application. To make things more maintainable, every code generator should have a main entry point, so let’s create a main template file which serves as this main entry point.

First, please delete everything in org.xtext.example.entity.generator/templates. Then, create a new Xpand template in this directory by selecting File -> New -> Other … -> Xpand Template. This is going to be our main template, so let’s call it Main.xpt. An empty editor will open.

Insert the following line:

«IMPORT entity»

In case you wonder how to insert the special quotation marks, use code completion (press CTRL + Space, or CMD + Space on a Mac) to insert them. Upon activating code completion, a dialog will appear, asking you whether to add the Xpand nature to the project. Please confirm this dialog.

The IMPORT statement imports the metamodel of your DSL, thereby making it known to Xpand, which will enable us to use the metamodel in our templates.

Next, add the following lines:

«DEFINE main FOR Model»
  «EXPAND Entity::entity FOREACH this.types.typeSelect(Entity)»
«ENDDEFINE»

This will add a new template called main to your template file. By specifying FOR Model, we specify that this template will later be bound to a variable representing your model. The EXPAND…FOREACH statement will invoke a template called entity in a template file called Entity for all elements of the collection types in your model. Remember, your model contains a number of entities and type definitions. These are stored in the collection types. The typeSelect(Entity) statement makes sure the generator only considers elements which are of type Entity (we do not want to create Java beans for the plain data types).

Create a template for Java beans

Every data-oriented application needs a bunch of classes to hold the data. Usually referred to as entities, these classes are POJOs in our case. So, let’s now create a template which helps us to create POJOs from our model.

Please add another Xpand template to your project by selecting File -> New -> Other … -> Xpand Template. Name the new file Entity.xpt, making sure to save it to the same folder as Main.xpt.

Again, start the file by importing the meta model:

«IMPORT entity»

Next, add the following lines:

«DEFINE entity FOR entity::Entity»
  «FILE this.name + ".java"»
    public class «this.name» {
      «EXPAND attribute FOREACH this.attributes»
    }
  «ENDFILE»
«ENDDEFINE»

This template will create a new Java file for each entity instance it is invoked for. This is achieved by the FILE statement, which takes the name of the output file as it’s first parameter.

As you can guess, we also need a template to create the attributes, so please add the following lines:

«DEFINE attribute FOR Attribute»
  private «this.type.name» «this.name»;

  public void set«this.name.toFirstUpper()»(«this.type.name» «this.name») {
    this.«this.name» = «this.name»;
  }

  public «this.type.name» get«this.name.toFirstUpper()»() {
    return this.«this.name»;
  }
«ENDDEFINE»

This template will create a private field, a setter and a getter for each attribute it is invoked for.

Create a template for a DAO

To load entities from a database and save them back, we will need to write some entities, but again, this is something the generator can do for us, so let’s write a template:

Create a template DAO.xpt and insert this text:

«IMPORT entity»

«DEFINE dao FOR entity::Entity»
  «FILE this.name + "DAO.java"»
    import java.util.Collection;
    import org.springframework.orm.hibernate3.support.HibernateDaoSupport;
    public class «this.name»DAO
      extends HibernateDaoSupport {
      «EXPAND crud FOR this»
    }
  «ENDFILE»
«ENDDEFINE»

«DEFINE crud FOR Entity»
  public «this.name» load(Long id) {
    return («this.name»)getHibernateTemplate().get(«this.name».class, id);
  }

  @SuppressWarnings("unchecked")
  public Collection loadAll() {
    return getHibernateTemplate().loadAll(«this.name».class);
  }

  public «this.name» create(«this.name» entity) {
    return («this.name») getHibernateTemplate().save(entity);
  }

  public void update(«this.name» entity) {
    getHibernateTemplate().update(entity);
  }

  public void remove(«this.name» entity) {
    getHibernateTemplate().delete(entity);
  }
«ENDDEFINE»

Do not forget to invoke this template from the master template file Main.xpt:

«IMPORT entity»

«DEFINE main FOR Model»
	«EXPAND Entity::entity FOREACH this.types.typeSelect(Entity)»
	«EXPAND DAO::dao FOREACH this.types.typeSelect(Entity)»
«ENDDEFINE»

Create a model

In your workspace, open org.xtext.example.entity.generator/src/model/MyModel.entity, making sure it contains the following model:

typedef String
typedef Integer
typedef Double
typedef Date mapsto java.util.Date

entity Customer {
  String firstName
  String lastName
  String email
  Address shippingAddress
  Order* orders
}

entity Address {
  String line1
  String line2
  String city
  String postcode
  String state
  String country
}

entity Product {
  String description
  Double price
}

entity Book extends Product {
  String ISBN
  String author
}

entity Order {
  Date orderDate
  OrderLine* orderLines
}

entity OrderLine {
  Product item
  Integer amount
}

Setting up the generator

Before we can actually start the generator and transform our model, we need to adjust some aspects of the generator project.

First, please open the generator workflow in /org.xtext.example.entity.generator/src/workflow/EntityGenerator.mwe. We need to change the name of the root template so it matches the name of the root template we chose. We also would like the generator to format the output properly, so we’re going to add an output beautifier.

Change the generator component in the workflow file to match the following:

  <component class="org.eclipse.xpand2.Generator">
    <metaModel class="org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel"/>
    <fileEncoding value="MacRoman"/>
    <expand value="templates::Main::main FOR model"/>
    <genPath value="src-gen"/>
    <beautifier class="org.eclipse.xpand2.output.JavaBeautifier"/>
  </component>

Note: you need to leave the fileEncoding attribute as-is. The above fragment is valid only on a Mac.

As the formatter is based on the Eclipse java formatter (see also this post on the Eclipse code formatter which I wrote some years ago and which still drives 200 reader/day to my blog), we have to add some bundle dependencies to the generator project.

Do to so, please open /org.xtext.example.entity.generator/META-INF/MANIFEST.MF, go to the Dependencies and add the following bundles:

  • org.eclipse.jface.text
  • org.eclipse.core.runtime
  • org.eclipse.jdt.core
  • org.eclipse.xtext.log4j
  • com.ibm.icu

Running the generator

To run the generator, simply select the workflow file /org.xtext.example.entity.generator/src/workflow/EntityGenerator.mwe and choose Run as… -> MWE Workflow from the context menu. You’ll see a bunch of log messages in the console view, and after a few seconds, you’ll find a bunch of just generated files in /org.xtext.example.entity.generator/src-gen.

Using the DSL to drive the generator

In the above sample, we wrote the DSL model by hand (well, you probably just copied it from this post), but this certainly is not the normal usage pattern. So, how can you use the DSL editor to create models that are then being read by the generator?

Actually, it is very easy if you follow these steps:

  1. Start the DSL in a runtime workbench (refer to the last installment to see how)
  2. In the runtime workspace, create a new Eclipse Plug-in Project. Choose any name you like, e.g. org.xtext.example.entity.mymodel. Make sure you deselect Generate an activator… on the second page of the wizard – we don’t need one.
  3. Add all of the following dependencies to the newly created project’s manifest file:
    • org.xtext.example.entity
    • org.eclipse.xpand
    • org.eclipse.xtend
    • org.eclipse.xtext
    • org.eclipse.emf.mwe.core
    • org.eclipse.emf.mwe.utils
    • org.eclipse.xtend.typesystem.emf
    • org.eclipse.jface.text
    • org.eclipse.core.runtime
    • org.eclipse.jdt.core
    • org.eclipse.xtext.log4j
    • com.ibm.icu
    • org.xtext.example.entity.generator
  4. Create a new workflow file in /org.xtext.example.entity.mymodel/src/workflow/EntityGenerator.mwe and insert the following text:
    <workflow>
      <cartridge file="workflow/EntityGenerator.mwe" model="classpath:/model/MyModel.entity"/>
    </workflow>
    		
  5. Patch the workflow file in the generator: Go back to the original workspace and change the line containing <component class=”org.eclipse.xtext.MweReader” uri=”… to <component class=”org.eclipse.xtext.MweReader” uri=”${model}”>
  6. Go back to the runtime workspace and create a new file MyModel.entity in /org.xtext.example.entity.mymodel/src/model/MyModel.entity.

You can now use the DSL editor to create your model. The generator can be started by running the generator workflow file EntityGenerator.mwe in your newly created model project /org.xtext.example.entity.mymodel in the runtime workspace.

Conclusion

I hope the last section didn’t bother you too much. The good news is that we’re going to provide a wizard which will help you setting up DSL model projects – see bug 281214.

Due to space limitations, this example is not complete in a sense that you can generate a fully functional CRUD application from it. You should, however be able to see what’s possible with textual DSLs and code generation.

If you’re interested in seeing the whole generator, drop me a line (peter dot friese [at] itemis dot de) or DM me on Twitter (my Twitter ID is @peterfriese).

Download the code

Download the code from my SVN repository.

Getting Started with Xtext

Xtext has been released as a part of the Eclipse Galileo release train on June 24th, 2009. Xtext is a framework for building DLSs (domain specific languages). In fact, it can be seen as a DSL for defining DSLs.

In this article, we will develop a small DSL for defining entities.

Download and Install

Hop over to http://xtext.itemis.com and download an Xtext distribution matching your platform. We’ve got all major platforms: Windows, Mac OSX Carbon, Mac OSX Cocoa 64, Mac OSX Cocoa 32, Linux Gtk 64, Linux Gtk 32.

To install, unzip the distribution file to a directory of your liking.

Windows users, please make sure you unzip to a directory near to your filesystem root! Eclipse contains files and folders with long names that might be hard to handle for Windows.

Create a new project

After launching Eclipse and creating a new workspace (or using an existing one), create a new Xtext project by selecting File -> New… Project… -> Xtext Project. In this article, we’re creating a DSL for describing entities, so let’s go with the following settings:

  • Main project name: org.xtext.example.entity
  • Language name: org.xtext.example.Entity
  • DSL-File extension: entity
  • Create generator project: yes

Click Finish to let Xtext create the three projects that make up your DSL:

  • org.xtext.example.entity – this project contains the DSL itself, including all back end infrastructure like the parser and the meta model.
  • org.xtext.example.entity.ui – as the name implies, this project contains all UI centric code of the DSL: the editor, the outline, content assist and so forth
  • org.xtext.example.entity.generator – this project contains a code generator which will transform the DSL scripts (aka models) you write in your DSL into something useful. In our example, we will create some POJOs and DAOs from our models.

Upon finishing creating these three projects, Xtext opens a file Entity.xtext, which contains a sample grammar, which we will change in a second.

Define the grammar for your DSL

Next up, we need to define the grammar for our DSL. To make things easier for us, let’s first write down a sample model. Please open org.xtext.example.entity.generator/src/model/MyModel.entity and key in the following text:

typedef String
typedef Integer
typedef Date mapsto java.util.Date

entity Person {
  String name
  String surName
  Date birthDay
  Address home
  Address work
}

entity Boss extends Person {
  Person* employees
}

entity Address {
  String street
  String number
  String city
  String ZIP
}

Most of this model should be pretty obvious, but there are two things worth noting:

  • We define our own list of data types to gain a certain degree of flexibilty, i.e. to map them to different types in the host language, as can be seen for the Date data type, which gets mapped to java.util.Date (we might also decide to map it to java.sql.Date)
  • The star (*) denotes multiplicity. We might also have chosen square brackets (Person[] employees) or something completely different – it’s largely a matter of taste.

Let’s derive the grammar for this model. Open org.xtext.example.entity/src/org/xtext/example/Entity.xtext, erase its entire contents and enter the following:

grammar org.xtext.example.Entity with org.eclipse.xtext.common.Terminals
generate entity "http://www.xtext.org/example/Entity"

The first line indicates we want the new grammar to be derived from the (built-in) grammar Terminals, which defines some basic terminal rules (like STRING, ID and so forth). If you’re interested, CTRL+Click on the language name to navigate to its definition.

The second line defines the name and namespace URI for our own grammar.

Let’s now define that our DSL supports types. Add the following lines:

Model:
  (types+=Type)*;

Type:
  TypeDef | Entity;

This tells Xtext that our Model contains any number (i.e. 0..N, as declared by the *) of Types. What exactly a Type is needs to be specified. Apparently, a Type can be either a TypeDef or an Entity:

TypeDef:
  "typedef" name=ID ("mapsto" mappedType=JAVAID)?;

A TypeDef starts with the keyword typedef, followed by an ID making up its name. Following the name, we can optionally (hence the question mark at the end) add a mapsto clause. The fragment mappedType=JAVAID specifies that the TypeDef will later have an attribute named mappedType of type JAVAID. As JAVAID is not yet defined, we need to do so:

JAVAID:
  name=ID("." ID)*;

So, a JAVAID is a sequence of IDs and dots, making up a qualified Java type name, such as java.util.Date.

Next, let’s define how to model entities:

Entity:
  "entity" name=ID ("extends" superEntity=[Entity])?
  "{"
    (attributes+=Attribute)*
  "}";

As you might have guessed, Entitys start with the keyword entity, followed by an ID as their name. They may optionally extends another entity. Surrounding a rule call with square brackets means “this is a cross reference”, i.e. a reference to an already existing element.

Entitys do have Attributes (any number, to be precise), thus we have to define how Attributes look like:

Attribute:
  type=[Type] (many?="*")? name=ID;

By now, you should be able to read this rule: an Attribute has a type which is a cross reference to a Type (which is either a TypeDef or an Entity), it has an optional multiplicity indicator (the star) and – of course – if has a name.

Your grammar should look like this by now:

grammar org.xtext.example.Entity with org.eclipse.xtext.common.Terminals

generate entity "http://www.xtext.org/example/Entity"

Model:
  (types+=Type)*;

Type:
  TypeDef | Entity;

TypeDef:
  "typedef" name=ID ("mapsto" mappedType=JAVAID)?;

JAVAID:
  name=ID("." ID)*;

Entity:
  "entity" name=ID ("extends" superEntity=[Entity])?
  "{"
    (attributes+=Attribute)*
  "}";

Attribute:
  type=[Type] (many?="*")? name=ID;

Compiling the DSL

Now it is time to see the fruits of our labor. But first, we need to compile our grammar. Xtext will create:

  • a parser
  • a serializer
  • an Ecore meta model
  • a full blown Eclipse editor

from this grammar. To make this happen, please select org.xtext.example.entity/src/org/xtext/example/GenerateEntity.mwe and select Run As -> MWE Workflow from the context menu. Xtext will now generate the entire infrastructure for your DSL and after a few seconds you should have a shiny new DSL including a great editor.

Taking it for a spin

Seeing is believing, so let’s take the DSL editor for a test drive. Select the DSL project org.xtext.example.entity and, from the context menu, select Run As -> Eclipse Application. A second instance of Eclipse will be started.

In this new instance, create a new, empty project (File -> New -> Project… -> General -> Project. Create a new file Sample.entity in the new project.

You can now insert the model we designed above or enter a new model:

Getting Started With Xtext

Leveraging the model

Now that we’ve got a fancy editor for our DSL, we want to transform the models we can create with this editor into something meaningful. This, however, will be the topic of the next installment.

More info

Feel free to discuss this article in the comments section of my blog. Should you have any technical questions regarding Xtext, we’ve got an excellent newsgroup where the committers answer your questions. We (itemis) also offer professional (commercial) support, i.e. customized trainings to get your team up to speed. Just drop us a note, we’re happy to discuss the details with you.

Download the code

You can check out the code from this SVN repository location.

Xtext does London

Marking the end of this years series of DemoCamps, Neil Bartlett and SkillsMatter are organizing the final DemoCamp in London on Monday 29th, 2009. Yes, that’s next monday!

Neil was kind enough to invite someone from the “modeling/oAW folk”, so Heiko Behrens will be giving an Xtext demo.

If you haven’t had the chance to attend Code Generation 2009 or if you’re curious what all this DSL talk is all about, I urge you to sign up for the DemoCamp now – it is free of charge.

I was fortunate enough to hear Heiko speak at the DemoCamp in Hamburg. Not only is Heiko a very good speaker (with a very convincing, deep voice), but also did he manage to come up with some very good examples of what a DSL is and why you should consider using them in your projects.

By the way, if you want to play around with Xtext before the DemoCamp, you can get a fresh copy from our download site at http://xtext.itemis.com.

Noisy DSLs

I recently came across some DSLs which had some defects. Let’s look at a sample I quickly hacked together with Xtext:

Resource - Demo/demo.mydsl - Eclipse SDK

Here are two recommendations I’d like to give when designing a textual DSL:

  1. Don’t use uppercase letters in the keywords of your DSL, unless you either really like them. In the sample DSL, the keywords (Element, EndElement, Child, Endchild all start with an uppercase letter, which makes it cumbersome to edit the DSL script (you need to press the shift key a lot). Back in the days when we didn’t have syntax highlighting, UPPERCASE keywords made a lot of sense as they helped to draw a better distinction between the keywords, the variables and the constants (strings and numbers) of a program. Nowadays, it is hard to find an IDE which does not offer syntax highlighting, so this argument does not hold any more.
  2. Do not use noisy beginend syntax constructions. They do not increase readability of your DSL script and they add to the amount of code to be typed.

Here is an improved version of the DSL script:

Resource - Demo/demo.mydsl1 - Eclipse SDK

By removing the Endelement and Endchild keywords, we have improved readability a lot. Using lowercase keywords has lowered the effort you need to invest to key in the DSL script. And – honestly – don’t we all like to be lazy at times? After all, one of the goals of DSLs is to make your work easier and more efficient.

That being said, have a great weekend!

By the way, if you need help designing a DSL with Xtext – I and the entire Xtext team are happy to assist you. Just drop me a line (peter at itemis dot com) or hop on to our professional support site

Following Eclipse Milestones

With the Galileo Release coming up, you might find yourself having a hard time updating to the latest milestones AND keeping your favorite plug-ins up-to-date.

Did you know that you can migrate your additional plug-ins from one Eclipse install to another one? This can be a huge time-saver, especially for people who like to live on the bleeding edge.

Here’s how:

  1. Install a fresh copy of Eclipse. Let’s assume you install Eclipse 3.5 RC4 Cocoa 64bit (you’re feeling lucky)
  2. Let’s further assume you’ve got an existing install of Eclipse 3.5 RC3 Cocoa 32bit with some additional plug-ins, like FindBugs, WindowBuilder Pro, etc.
  3. After installing, start your newly installed instance of Eclipse
  4. Select Help -> Install New Software…
  5. In the Install dialog, click the Add… button to add a new update site:
    Eclipse Install Dialog

  6. In the next dialog, click on Local… to add a local update site:
    Eclipse: Add Site

  7. using the file chooser, browse to <path_to_your_OLD_eclipse_instance>/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile/ and click Choose to select that directory.
  8. Click OK to add the update site:
    Eclipse: Add Local Update Site

  9. The Install dialog will now list all plug-ins installed in the old location (i.e. your old Eclipse instance), clearly highlighting the ones that are not already installed in the new instance:
    Eclipse: Install from existing Eclipse install

  10. Check all features that you want to transport to the new location and continue the installation by clicking Next>.
  11. After confirming the license terms and clicking Finish, Eclipse will install the selected features from the old location into the new location. After the obligatory workbench restart you’re good to go.

The only thing that I am wondering about is: why is there no first-class UI action (e.g. an import wizard) to do this?

Follow

Get every new post delivered to your Inbox.