In the previous part of the article, we’ve started writing our own implementation of the esoteric programming language Rockstar. We went through the basics of parsing and transforming text with Parslet and ended up transforming some simple Rockstar lyrics into equally simple Ruby code.
Today we’ll make the implementation a bit more complete so that by the end of this article, we will be able to actually execute a basic Rockstar program.
This is what we’re going to run:
Devilish Secret takes a whisper
The spell is abracadabra. complete
A demoneye is eye unforgivable
Put a whisper of the spell into the grimoire
Give back a demoneye with the grimoire
Listen to a whisper
Scream Devilish Secret taking a whisper
This is a forbidden magic spell that converts a temperature from Celsius to Fahrenheit. Sounds great, right?
So, turn on some appropriate music and let’s get to it. And as before, all of this code is available on Github with all steps in separate commits, so you can follow along.
First, a Warmup
Before we start doing any coding, let’s take a look at the program we’re going to run and go through it line by line, comparing it with the Rockstar language definition so that we know what we already have and what needs to be implemented to make it work.
Devilish Secret takes a whisper
This is a function definition. We’ll leave that for later because a function needs to have some contents in a block, so this will be a bit more complicated to implement.
The spell is abracadabra. complete
A poetic number literal—we already have these working—but the tricky part is that it has a . inside, meaning this should be treated as a decimal, not as an integer.
A demoneye is eye unforgivable
Another poetic number literal, this time with just an integer value. This should already work as it is, nothing to do here.
Put a whisper of the spell into the grimoire
An assignment—which we also implemented in the first part—but a whisper of the spell
looks suspicious. That’s because it’s not a variable name but a multiplication of two variables.
Give back a demoneye with the grimoire
A return statement. Sounds easy enough, except like in the line above, it returns a math operation (this time an addition) instead of a single variable.
Listen to a whisper
A variable with its value coming from the user’s input. Should be easy as well.
Scream Devilish Secret taking a whisper
And finally, a print statement that prints the output of the above function called with an argument.
Now that we know what to do, let’s start with the easy stuff.
Enter STDIN
First off, we need to get the user to input some value. This is what the Listen to a whisper
line does—it waits for the user to enter something and assigns it to a whisper value.
Parsing this is pretty simple. We can make a basic parser rule that will take a variable and add this new rule to the helper rule.
rule(:input) do
str('Listen to ') >> variable_names.as(:input_variable)
end
rule(:string_input) { input | print_function | basic_assignment_expression | poetic_number_literal | proper_variable_name | common_variable_name }
And in the RockstarTransform
class we add another rule that makes it work in Ruby.
This is a bit more complicated than a simple STDIN.gets.chomp
because by default, STDIN.gets
returns a string, and we want the contents of the variable to be a number if the user inputs a number—after all, the goal is to convert temperatures. If what the user entered can’t be converted into an integer, we just leave it as a string with the rescue __input
part.
rule(input_variable: simple(:var)) do
"print '> '\n__input = STDIN.gets.chomp\n#{var} = Integer(__input) rescue __input"
end
And we’re done. Well, we’re not—we forgot to write a test for this.
At first glance, this seems complicated. After all, testing input from STDIN in RSpec is a bit tricky because the tests should run automatically and there’s no reason (or if we run them in some remote service like Travis or CircleCI, also no way to actually do it) to wait for the user to manually type something. But we’re lucky, we’re not actually running the code, we’re just transforming text so that we don’t have to care about it at all.
context 'input from STDIN' do
let(:input) do <<~END
Listen to the news
Shout the news
END
end
it 'transforms into ruby' do
expect(KaiserTutorial.transpile(input)).to eq <<~RESULT
print '> '
__input = STDIN.gets.chomp
the_news = Integer(__input) rescue __input
puts the_news
RESULT
end
end
Call of the Wild
Next on our list of easy things is the function call with an argument. Since it’s just a function named like a proper variable that takes exactly one variable name as an argument, we already did this kind of work before. So the three components we need to write aren’t complicated either.
context 'function call' do
it 'tranforms function name and argument' do
expect(KaiserTutorial.transpile("Superman taking a whooping")).to eq 'superman(a_whooping)'
end
end
A simple parser rule—and don’t forget we have to add it to the :string_input
helper rule as with everything, else it won’t be parsed:
rule(:function_call) do
(
variable_names.as(:function_name) >> str(' taking ') >> variable_names.as(:argument_name)
).as(:function_call)
end
rule(:string_input) do
input | print_function | function_call | basic_assignment_expression | poetic_number_literal | proper_variable_name | common_variable_name
end
And a transformer rule:
rule(function_call: { function_name: simple(:function_name), argument_name: simple(:argument_name) }) do
"#{function_name}(#{argument_name})"
end
And we’re done with this part.
Return to the Sea
Similar to the above we can do a return statement. For now, we’ll make it consume only a variable name, not the math operation we need. This is easy enough, so I’ll just put the new parser rule here, as the rest is analogous to the function call, just with different names.
You can write the transform rule and tests by yourself as an exercise, or you can cheat a little by looking at the linked commit in the Github repository.
rule(:return_statement) do
str('Give back ') >> variable_names.as(:return_statement)
end
Better Living Through Better Testing
So far we’ve been pretty good with following the TDD principles, or at least writing tests for each feature we’re implementing. However, there are two problems with how exactly we were doing it.
The first one is that we’ve been putting all our tests into one file, which, just like with models and other objects, isn’t really the best way to do such things. This is easy to fix, though—all we need to do is split the tests into separate files and subdirectories so they make more sense when someone looks at them. Because the tests themselves don’t change at this step, I’ll just list the new spec/
subdirectory structure here:
functions/
function_spec.rb # function calls, returns and body
general/
general_spec.rb # general tests, for example handling multiple lines properly
statements_spec.rb # all the statements that don't have their own place - print, input and so on
variables/
assignment_spec.rb # assignment of values to variables
variable_names_spec.rb # generating proper variable names
The second problem is unfortunately much worse—all our tests are quite useless right now. Well, maybe not completely useless, but each only tests the so-called “happy path.” This means we have no tests that can catch problems but that only test for some of the positive outcomes, making our test suite not very reliable.
Let’s take as an example the only test for the print statement we have right now:
context 'print statement' do
it "prints a variable" do
expect(KaiserTutorial.transpile("Shout Tommy")).to eq "puts tommy"
end
end
It’s an okay test and it passes like all the others, but the only thing we can learn from it is that when we put a valid proper variable into the statement, it will transpile to an equivalent Ruby code. That’s good to know, but this test doesn’t answer any questions about our code.
Does it work with common variables (the world
, for example), too? It should, but we don’t test that, so it’s only a guess. What if we use something else instead of a variable, like a function call? Will it break with an exception? Will it output some garbage instead or crash everything entirely? What if we pass something completely wrong?
This single test doesn’t answer any of these questions, and if we want our tests to be useful, they should be greatly extended. Let’s upgrade them, starting with this example:
context 'print statement' do
it 'handles proper variables' do
expect(KaiserTutorial.transpile('Shout Tommy')).to eq 'puts tommy'
end
it 'handles common variables' do
expect(KaiserTutorial.transpile('Shout the world')).to eq 'puts the_world'
end
it 'throws a syntax error if passed garbage' do
expect(KaiserTutorial.transpile('Shout the Rock')).to eq ''
end
it 'throws a syntax error if passed a function call' do
expect(KaiserTutorial.transpile('Shout Joker taking Hostages')).to eq ''
end
end
This looks much better. We test both of our possible positive cases (a valid proper and common variable), and some cases that should result in an error (the Rock
is not be a valid variable name at all). We still could do even better—for example, the common variables in our tutorial accept not only the but also The
, a
, and A
, so all of this should also be tested.
However, our work is not done yet because running this test will show two failures. Let’s deal with the second test first.
2) KaiserTutorial print statement throws a syntax error if passed a function call
Failure/Error: expect(KaiserTutorial.transpile('Shout Joker taking Hostages')).to eq ''
expected: ""
got: nil
(compared using ==)
In addition to this failure summary, we also get a Parslet failure cause tree (which I won’t paste here because it will take too much space) from the rescue block below, but the exception itself gets consumed, which is why we simply get back a nil
. That’s not very useful, right?
def self.parse(input)
KaiserTutorial::RockstarParser.new.parse(input)
rescue Parslet::ParseFailed => failure
puts failure.parse_failure_cause.ascii_tree
end
We can fix this easily by adding a raise SyntaxError, failure.message
in that rescue block, which will still consume the Parslet exception but will also replace it with another one.
The reasoning behind this is that the users of our transpiler shouldn’t be expected to even know what Parslet is, so a SyntaxError is much more useful and easier to understand for them. In the future, we could probably handle it even better by looking at the cause tree and providing a meaningful message together with the error.
We have to also change our test slightly, as RSpec only works properly with raise_error
if we pass a block to expect
. It’s a very easy mistake to make and later waste time wondering what’s wrong with our test.
expect { KaiserTutorial.transpile('Shout Joker taking Hostages') }.to raise_error SyntaxError
This test finally passes and clears up one of our failures. Time to deal with the other one. Here we can see we passed the Rock
to the print statement, which should result in a syntax error because, according to the Rockstar language definition, it’s not a valid variable name at all. But we don’t get an exception this time. Instead, we get something completely wrong in return because the parser managed to split the variable into two lines for whatever reason.
1) KaiserTutorial print statement throws a syntax error if passed garbage
Failure/Error: expect(KaiserTutorial.transpile('Shout the Rock')).to eq ''
expected: ""
got: "puts the_\nrock\n"
(compared using ==)
Diff:
@@ -1 +1,3 @@
+puts the_
+rock
We can try fixing the print statement itself, but let’s remember how parsing this with Parslet works. We’re using the following rule for the print statement:
rule(:print_function) do
(str('Shout') >> space >> variable_names.as(:output)).as(:print)
end
Where variable_names
is parsed separately, as that’s yet another rule in our parser. This points us to the :common_variable_name
rule (or its transformer rule in the RockstarTransform
class) as the real culprit here. So let’s add more examples to the variable names spec to be sure it works correctly. If we’re right and this is the problem, the original problem will be fixed as well.
it "doesn't convert mixed case words" do
expect(KaiserTutorial.transpile('the World')).to eq ''
end
This test fails with a similar failure as we had in the print statement, confirming our suspicions. The fix for this error is pretty simple—we just have to ensure the word after the keyword will always be a lowercase word.
We can do this by adding an argument to the repeat
atom in the :common_variable_name
rule. This makes Parslet expect at least one instance of the repeated match, whereas without an argument, it can pass with zero matches and that’s what is happening in our test.
rule(:common_variable_name) do
(str('A ') | str('a ') | str('The ') | str('the ')) >> match['[[:lower:]]'].repeat(1)
end
You might notice I’ve also done a bit of other refactoring here in the rockstar_parser.rb
file, moving the .as(:variable_name)
out of this rule. I like this accidental approach to refactoring—I’m already touching the method (or rule, in this case), so there’s nothing wrong in leaving it in a better state than I found it. Otherwise, I would have to spend much more time refactoring everything later.
Number of the Beast
After fixing all our tests by adding even more negative cases, let’s get back to implementing new features. Next on our list are two mathematical operations: addition and multiplication. Adding them is pretty similar to what we did in the previous part for the assignment—we have a variable name at both sides of the operation and a keyword in the middle.
rule(:addition) do
(
variable_names.as(:left) >> str(' with ') >> variable_names.as(:right)
).as(:addition)
end
rule(:multiplication) do
(
variable_names.as(:left) >> str(' of ') >> variable_names.as(:right)
).as(:multiplication)
end
We could do it in a better way by introducing another rule for the math operator keyword, but for the sake of the tutorial, there’s nothing wrong with spelling everything out so it’s easier to understand.
Now that we have the operations working, we need to get back to our return and assignment rules. Right now they only allow variables, but our source program says—for example, Give back a demoneye with the grimoire
.
Let’s add a case for this in our function_spec.rb
file.
it 'returns a result of a math operation' do
expect(KaiserTutorial.transpile('Give back a song with a message')).to eq 'return a_song + a_message'
end
As expected, right now this test throws a SyntaxError at us, so to make it work we need to modify a few of our parsing rules.
rule(:variable_names) { (common_variable_name | proper_variable_name).as(:variable_name) }
rule(:math_operations) { addition | multiplication }
rule(:operation_or_variable) { math_operations | variable_names }
These three are just support rules so that the parser doesn’t get out of hand. Having these, we can fix our return rule, replacing the reference to variable_names
with operation_or_variable
. We can modify the assignment and print function’s rules in a similar way.
rule(:return_statement) do
str('Give back ') >> operation_or_variable.as(:return_statement)
end
rule(:basic_assignment_expression) do
(str('Put ') >> operation_or_variable.as(:right) >> str(' into ') >> variable_names.as(:left)).as(:assignment)
end
rule(:print_function) do
(str('Shout') >> space >> operation_or_variable.as(:output)).as(:print)
end
This brings us much closer to translating the whole program.
We’re All Floating Down Here
Next up we need to fix our poetic number literals to deal with decimal numbers properly. A decimal number written in a poetic way is, for example, eye. a rock
, which should be translated to 3.14
, and world peace. not a war anywhere
should result in 55.4138
. Since the parsing of the whole expression remains the same, we only need to modify the :string_as_number
rule in the RockstarTransform
class. Here’s what we’ll do:
rule(string_as_number: simple(:str)) do |context|
if context[:str].to_s.include?('.')
context[:str].to_s.gsub(/[^A-Za-z\s\.]/, '').split('.').map do |sub|
str_to_num(sub.strip)
end.join('.').to_f
else
str_to_num(context[:str]).to_i
end
end
def self.str_to_num(string)
string.to_s.split(/\s+/).map { |e| e.length % 10 }.join
end
If there’s no dot in the string, we continue with str_to_num
as before. If there is a dot, we need to split the whole string and convert the parts separately, and then join it back, converting the resulting string to a float. We also need to move the .to_i
from our helper method, since it would mess with the decimals: to_i
automatically strips the leading zeroes, so 1.00001
would not get converted properly if we left it there.
All these cases should be of course reflected in our test suite:
context 'poetic number literal' do
it "converts an integer" do
expect(KaiserTutorial.transpile("Tommy was a lean mean wrecking machine")).to eq "tommy = 14487"
end
it "strips leading zeroes from integers" do
expect(KaiserTutorial.transpile("Jack was carjacking a nice car")).to eq "jack = 143"
end
it "converts a decimal number" do
expect(KaiserTutorial.transpile("Mary was looking. smooth")).to eq "mary = 7.6"
end
it "keeps leading zero in decimal part" do
expect(KaiserTutorial.transpile("Jack was busy. Carjacking a nice car")).to eq "jack = 4.0143"
end
end
Madness to the Method
The last thing we have to implement in our Rockstar parser is a function definition. This is a bit trickier because we need to somehow tell Parslet that some of the following lines will have a different scope and that it should treat them as the contents of the function. We also need to inform the parser to treat an empty line as the end of the function’s body. Fortunately, Parslet provides us with a tool that can do exactly that.
Let’s start with writing a test to know what to expect:
context 'function definition' do
let(:one_argument_function) do <<~END
Midnight takes Hate
Shout Desire
Give back Desire
END
end
it 'makes a function definition' do
expect(KaiserTutorial.transpile(one_argument_function)).to eq <<~RESULT
def midnight(hate)
puts desire
return desire
end # enddef
RESULT
end
end
Next, the parser rules:
rule(:function_definition) do
(
variable_names.as(:function_name) >> str(' takes ') >> variable_names.as(:argument_name) >> eol >>
scope {
line.repeat.as(:function_block) >>
(eol | eof)
}
).as(:function_definition)
end
rule(:eof) { any.absent? }
rule(:statements) { return_statement | input | print_function | function_call | basic_assignment_expression | poetic_number_literal }
rule(:string_input) { function_definition | math_operations | statements | variable_names }
The latter ones are mostly cosmetic—we’ve added an end of file rule, added the function definition to the list of rules to parse, and moved out the statements to a separate rule, so it takes less horizontal space.
The first one is easy apart from the new scope
atom we haven’t used before—this atom is the Parslet’s way of making it all work as we expect. It will run the Parser on the scoped block first, similarly as it does for things like variable names.
Parsing the example from the test will result in the following parse tree:
lyrics: [
{
line: {
function_definition: {
argument_name: "hate",
function_block: [
"puts desire",
"return desire"
],
function_name: "midnight"
}
}
}
]
We can see that the whole rule, together with the scoped block, is still treated as a single line. This is good because we can then plug it into the rest of the rules that transform multiple lines and we don’t have to do anything more, but we already have the lines inside the function parsed as an array.
Transforming it into Ruby code is a formality at this point:
rule(function_definition: {
function_name: simple(:function_name),
argument_name: simple(:argument_name),
function_block: sequence(:function_block_lines),
enddef: simple(:_)
} ) do |context|
output = "def #{context[:function_name]}(#{context[:argument_name]})\n"
output += context[:function_block_lines].map { |l| " #{l}\n" }.join
output += "end # enddef\n"
output
end
Come Together
And with that done, we have all the pieces required to run our initial program. We should see if it actually works. We could simply copy/paste it into a test example between the squiggly heredocs, but that’s boring, so let’s make it a separate file.
spec/fixtures/c_to_f.rock
should be a nice name and placement for it.
Now all we have to do is to read it. If we were testing a Rails app, it would be trivial—just use a file_fixture("c_to_f.rock").read
and be done with it, but we don’t have this helper here. We could just add rspec-rails
to our gemspec, but it would most likely come with some parts of Rails itself, which we don’t need at all. So what do we do?
Well, we’re rockstars, aren’t we? After all, good artists copy and great artists steal.
So we can steal that file_fixture
helper from Rails and, with some minor modifications, use it without needing the rest of Rails. This goes into our spec_helper.rb
file, after the configuration block:
def file_fixture(fixture_name)
file_fixture_path = File.dirname __FILE__
path = Pathname.new(File.join(file_fixture_path, 'fixtures', fixture_name))
if path.exist?
path
else
msg = "the directory '%s' does not contain a file named '%s'"
raise ArgumentError, msg % [file_fixture_path, fixture_name]
end
end
This is mostly the same as the original helper, but we scrapped all the module code around it (we don’t need it anyway), and we don’t have all the ActiveSupport methods to provide the file_fixture_path
for us, so we have to figure out the path ourselves.
Fortunately, we’re including it in the spec/spec_helper.rb
file, which is loaded everytime we run RSpec so we know where the spec/
directory is – File.dirname __FILE__
will point us at it. Then it’s just the matter of adding fixtures
and the file name after it.
We can now write a test that will use our new fixture file. Let’s make a failing one first to see the output:
context 'celsius to fahrenheit example' do
let(:input) { file_fixture "c_to_f.rock" }
it 'transpiles code' do
expect(KaiserTutorial.transpile(input.read)).to eq ''
end
end
The result looks promising—the function got defined properly, the input from STDIN is there, all the variable names look okay…
Diff:
@@ -1 +1,12 @@
+def devilish_secret(a_whisper)
+ the_spell = 1.8
+ a_demoneye = 32
+ the_grimoire = a_whisper * the_spell
+ return a_demoneye + the_grimoire
+end # enddef
+
+print '> '
+__input = STDIN.gets.chomp
+a_whisper = Integer(__input) rescue __input
+scream_devilish_secret(a_whisper)
Or do they? If you look closely, you’ll notice that the last line is transpiled into scream_devilish_secret(a_whisper)
. Wasn’t that supposed to be a print statement? Looks like we forgot something on the way.
Back in Black
It’s not a big deal, really. These kinds of things happen all the time in software development. At least we caught it early in our tests, so all’s well and good, we just need to fix it.
Looking back at the parser code, we made a simple mistake. In the first part we defined a print statement with only Shout
as the keyword, so Scream
is treated as part of a proper variable. Additionally, we only added math operations to the print statement rule and forgot about function calls, so we have to fix that too.
it 'handles different keywords' do
expect(KaiserTutorial.transpile('Scream Really Loud')).to eq 'puts really_loud'
end
it 'prints the result of a function call' do
expect(KaiserTutorial.transpile('Shout Joker taking Hostages')).to eq 'puts joker(hostages)'
end
Now that we have new tests, we should upgrade our parser rule as well:
rule(:print_function) do
((str('Shout') | str('Scream')) >> space >> (function_call | operation_or_variable).as(:output)).as(:print)
end
We can now run the test that tries to transpile our whole program again, and it will fail once more, but this time the print statement, too, is correct:
Diff:
@@ -1 +1,12 @@
+def devilish_secret(a_whisper)
+ the_spell = 1.8
+ a_demoneye = 32
+ the_grimoire = a_whisper * the_spell
+ return a_demoneye + the_grimoire
+end # enddef
+
+print '> '
+__input = STDIN.gets.chomp
+a_whisper = Integer(__input) rescue __input
+puts devilish_secret(a_whisper)
Since this output will not change, we can keep it in our test. We could also save it in another fixture file, if we wanted to keep the RSpec example cleaner. Your choice, really—this is short enough that both approaches are valid.
context 'celsius to fahrenheit example' do
let(:input) { file_fixture "c_to_f.rock" }
it 'transpiles code' do
expect(KaiserTutorial.transpile(input.read)).to eq <<~PROGRAM
def devilish_secret(a_whisper)
the_spell = 1.8
a_demoneye = 32
the_grimoire = a_whisper * the_spell
return a_demoneye + the_grimoire
end # enddef
print '> '
__input = STDIN.gets.chomp
a_whisper = Integer(__input) rescue __input
puts devilish_secret(a_whisper)
PROGRAM
end
end
And with this, our whole test suite is green, so we can say we now can successfully transpile a whole program from Rockstar to Ruby.
Run for the Hills
We’re not done yet, though. At the start, I’ve promised we’ll be able to actually run the program, not just transpile it. To do so, we need to have an executable command in our gem. Luckily it’s a very simple process thanks to the Thor gem that allows us to quickly build a really nice command line interface.
Let’s add it to our kaiser-tutorial.gemspec
and define what our executable file will be:
spec.executables = ['kaiser-tutorial']
spec.add_dependency "thor"
We should also define what we want Thor to do for us. We can do it in lib/kaiser_tutorial/cli.rb
:
require 'thor'
module KaiserTutorial
class CLI < Thor
desc "execute FILE", "transpiles and runs a .rock FILE"
def execute(filename)
file = File.read filename
output = KaiserTutorial.transpile(file)
instance_eval output
say
end
end
end
This is the simplest Thor setup that can be made in it. We define a command which will take a filename as an argument, read the file into a string, transpile it to Ruby, and then simply run instance_eval
on the transpiled output to execute it. The last say
is just putting an empty line at the end.
The last part we need is the executable file itself, exe/kaiser-tutorial
, which is pretty simple as well, as it just starts our CLI class and lets Thor do all the work for us. Note that the file is missing the .rb
extension—it will get added as a proper shell command when we install the gem, and we want to be able to run kaiser-tutorial
, not kaiser-tutorial.rb
, right?
#!/usr/bin/env ruby
require "rubygems"
require "bundler/setup"
require "kaiser_tutorial"
require "kaiser_tutorial/cli"
KaiserTutorial::CLI.start(ARGV)
We should probably also update the gem’s version in lib/kaiser_tutorial/version.rb
. After all, we’ve added quite a lot of features.
Gimme the Prize
And now that we’re done with the CLI, it’s time to see how it all works together. Running rake install:local
will install our new gem’s version, so we can make sure our CLI works as expected:
$ kaiser-tutorial
Commands:
kaiser-tutorial execute FILE # transpiles and runs a .rock FILE
kaiser-tutorial help [COMMAND] # Describe available commands or one specific command
And finally, we can run the program that we set out to write:
$ kaiser-tutorial execute spec/fixtures/c_to_f.rock
> 36
96.8
It might not be very impressive, but here’s the thing, now that you’ve successfully run a whole program written in the Rockstar language, you can now officially call yourself a Rockstar programmer.
Congratulations!
Polcode is an international full-cycle software house with over 1,300 completed projects. Propelled by passion and ambition, we’ve coded for over 800 businesses across the globe. If you share our passion and want to become a part of our team, contact our HR department. We’ll be happy to answer all your questions and even happier to welcome you aboard 🙂 Or maybe you have an interesting project in mind? If so, drop us an email and let’s talk over the details.
On-demand webinar: Moving Forward From Legacy Systems
We’ll walk you through how to think about an upgrade, refactor, or migration project to your codebase. By the end of this webinar, you’ll have a step-by-step plan to move away from the legacy system.
Latest blog posts
Ready to talk about your project?
Tell us more
Fill out a quick form describing your needs. You can always add details later on and we’ll reply within a day!
Strategic Planning
We go through recommended tools, technologies and frameworks that best fit the challenges you face.
Workshop Kickoff
Once we arrange the formalities, you can meet your Polcode team members and we’ll begin developing your next project.