CS 496 - Assignment 2

Lexical Analysis

In this assignment, you will implement a lexical analyzer for the SPL ("Simple Procedural Language") programming language.

Getting Started

You will need to have Java (any JDK version 5.0 or higher) and Eclipse (any recent version) installed.

Download CS496_Assign2.zip.  Save the file somewhere you will remember it (e.g., the Desktop).

Start Eclipse.  Choose File->Import...->General->Existing Projects into Workspace.  Choose Select archive file, and click Browse.  In the file browser dialog, select the zip file you just saved.  Close the file browser.  In the Import dialog, the spl project should be shown.  Click Finish.

Your Task

Your task is to edit the lexical analyzer specification, which is in a file called Lexer.jflex in the edu.ycp.cs496.spl.fe package in the src folder.

A JFlex specification is a collection of regular expressions defining lexical structures (tokens, whitespace, and comments), and lexical rules specifying how to handle each kind of structure.

Your goal is to add new rules to the lexical analyzer specification so that the generated lexical analyzer will be able to handle the SPL programs in the three files in the test directory.

In general, when you add a rule defining a kind of token, the action for that rule should call one of the makeSymbol methods to create a new Symbol object representing the token.

Note that you will need to add new terminal directives in the CUP parser specification (Parser.cup) to define new kinds of tokens.

Use the already-defined lexical rules and token types as a guide as you add new ones.


JFlex works by generating Java source code.  So, any time you change the lexical analyzer specification (Lexer.jflex), you will need to rebuild the project.

The first time you want to build the lexical analyzer, open the build.xml file in the root of the project.  In the Outline window, right-click on the build target.  Choose Run As->Ant Build....  A dialog will pop up.  Choose the tab labeled Refresh.  Click the checkbox labeled Refresh resources upon completion.  Choose The project containing the selected resources in the list of radio buttons.  The dialog should now look something like this:

Click run.  You should see some messages in the Console window indicating that the project is being built (including running JFlex).

From now on, when you need to build the project you can click on build.xml, right click the build target, and choose Run As->Ant Build.

Testing Your Lexical Analyzer

The Tokenize program in the edu.ycp.cs496.spl package is a test driver for your lexical analyzer.  When you run it (right-click on Tokenize.java, choose Run As->Java Application) it reads a file (or console input if no file is specified), and uses your lexical analyzer to tokenize the input characters.

To run the Tokenize program on an input file, first right-click on Tokenize.java, choose Run As->Java Application.  The program is now running in the Console window, reading from the keyboard.  Click the red square to kill the program:

Now, right-click on Tokenize.java again, and choose Run As->Open Run Dialog....  When the dialog opens, choose the Arguments tab.  In the box labeled Program arguments, type test/t0.spl.  The dialog should look something like this:

Now click Run.  (Wasn't that a lot of work?)  The Tokenize program runs, using test/t0.spl as its input.  Initially, you will see an error message about an illegal character at line 1.  When you add new lexical rules to handle all of the lexical constructs (tokens, whitespace, comments) in the input file, you should see the following output:

Reading input from test/t0.spl
[func] #4
[main] #5: main
[(] #7
[)] #8
[:] #9
[void] #10
[begin] #2
[end] #3

Each line of output (other than the first) shows the lexeme of one of the tokens recognized in the input file, along with its type.  The numbers assigned to your token types may be different.

Don't forget to test your lexical analyzer with the other input files (test/t1.spl and test/t2.spl) as well.


To submit, first export your project by right-clicking on the name of the project (spl) in the Package Explorer, and then choosing Export->General->Archive File.  Save the project as a zip file.  Upload the zip file to the Marmoset server as Project 2:


You should have received your Marmoset username and password in an email.