Let's build a (nano) web framework

What?

Recently I was diging into the codebase of Ruby on Rails to check a routing issue I was having.

It all started when I was handed a project... Uhh, let's build a (nano) web framework!

Introducing Ruby on Lane

Here's how our hello world project will look like:

require './ruby_on_lane.rb'

# Schema
Schema.define('hello_app_database') do
  table :users do
    column :string, :name
    column :int, :age
  end
end

# Models
class User < Model
  def self.adult?
    self.age > 18
  end
end

# Seed
User.create(name: "John Doe", age: 42)
User.create(name: "Tipsy Lala", age: 25)
user = User.create(name: "Lorem Ipsum", age: 19)

user.name == "Lorem Ipsum"

# Routes
get '/' do
  'Hello :)'
end

get '/filter' do
  filter_age = params['age'] || 20

  count = User.where('age', '>', filter_age).count

  "Number of users who are older than #{filter_age}: #{count}"
end

How deep are we gonna go?

We're going to abstract the web server layer and focus on the application side.

For this, we'll be building on top of Rack. Rack provide an interface that all compliant should follow:

It must respond to call
It takes one argument: environment that contains informations about the request
It must returns an Array of exactly three values: The status, the headers and the body

You can find more information about each part in the Rack specification.

Here's a simple Rack compliant application:

require 'rack'
require 'rack/server'

class HelloApp
  def call(env)
    [200, {"Content-Type" => "text/html"}, ["<h1>Hello</h1>"]]
  end
end

Rack::Server.start :app => HelloApp.new

Running it from the terminal:

$ ruby hello-world.rb
[2020-03-06 15:25:22] INFO  WEBrick 1.4.2
[2020-03-06 15:25:22] INFO  ruby 2.5.7 (2019-10-01) [x86_64-darwin17]
[2020-03-06 15:25:22] INFO  WEBrick::HTTPServer#start: pid=61969 port=8080

Here's how it look like when we naviagte to http://localhost:8080/:

Introducing the Database Schema

In this section we'll be focusing on these lines:

Schema.define('hello_app_database') do
  table :users do
    column :string, :name
    column :int, :age
  end
end

The above code should do the following:

Create the database hello_app_database if it doesn't exists
Create the table users if it doesn't exists
Create two columns with name and types respectevly: name: String, age: Int if they don't exists

We won't be considering the case where the columns already exists but have different types.

Note: We'll structure the code in a way that makes it simple to support new databases. Altought for simplicity, we'll be using SQLite in this example.

First comes the DSL

Let's get the syntax working

First, let's get the syntax working for:

Schema.define('hello_app_database') do
  # ...
end

We can get this syntax working by defining Schema module, that have the define method which takes 2 arguments: database name and a block to execute:

module Schema
  def self.define(db_name, &block)
    # ...
  end
end

Next comes table and column.

These operations (creating database, tables & columns) will be provided by a class that we'll create called: SQLiteAdapter.

At this stage, the methods will be noop; we'll implement them after we get the syntax that we want.

Here's how SQLiteAdapter will look like:

class SQLiteAdapter
  def initialize(db_name)
    puts "db_name: #{db_name}"
  end

  def table(_name, &block)
    puts "table: #{_name}"
  end
end

Now let's bridge the connection between SQLiteAdapter and Schema: what we want is to excude the block provided to Schema.define in the context of SQLiteAdapter. Ruby have exactly a method for this: instance_eval

From the official documentation:

Evaluates a string containing Ruby source code, or the given block, within the context of the receiver (obj). In order to set the context, the variable self is set to obj while the code is executing, giving the code access to obj’s instance variables and private methods. When instance_eval is given a block, obj is also passed in as the block’s only argument.

So what we'll do is creating an instance of SQLiteAdapter and call instance_eval on the created object:

module Schema
  def self.define(db_name, &block)
    adapter = SQLiteAdapter.new(db_name)
    adapter.instance_eval(&block)
  end
end

So far we have supported this syntax:

Schema.define('test.db') do
  table :users do
    # ...
  end
end

What's left is to support the column part. Adding it is very similar to table part: execute instance_eval from table method:

class SQLiteAdapter
  def initialize(db_name)
    puts "db_name: #{db_name}"
  end

  def table(_name, &block)
    puts "table: #{_name}"

    self.instance_eval(&block)
  end

  def column(_type, _name)
    puts "\tcolumn: #{_name} -> #{_type}"
  end
end

So the final code for adding the DSL syntax is:

class SQLiteAdapter
  def initialize(db_name)
    puts "db_name: #{db_name}"
  end

  # create table if it doesn't exists
  def table(_name, &block)
    puts "table: #{_name}"
    self.instance_eval(&block)
  end

  # create column if it doesn't exists (no matter the type)
  def column(_type, _name)
    puts "\tcolumn: #{_name} -> #{_type}"
  end
end

module Schema
  def self.define(db_name, &block)
    adapter = SQLiteAdapter.new(db_name)
    adapter.instance_eval(&block)
  end
end

Executing this snippet:

Schema.define('test.db') do
  table :users do
    column :string, :name
    column :int, :age
  end
  table :lorem do
    column :string, :ipsum
  end
end

will generate:

db_name: test.db
table: users
  column: name -> string
  column: age -> int
table: lorem
  column: ipsum -> string

Now, let's get the semantics

All the database logic resides in SQLiteAdapter, here's what we'll be changing:

initialize: create database if it doesn't exists
table: create table if it doesn't exists
column: create column if it doesn't exists with the provided type

We'll be using the sqlite3 gem to support the SQLite operations.

To achieve this, first let's change column method to save the columns to be created in @columns instance variable. This will get reset for each created table:

def column(_type, _name)
  @columns << {
    :type => map_type[_type],
    :name => _name.to_s
  }
end

map_type here is adapter specific method, that convert the column types that we support to the database at hand (SQLite in this case). As we only support 2 column types currently the implementation for it is quite simple:

def map_type
  {
    :string => "TEXT",
    :int => "INTEGER"
  }
end

Note: The type mapping is from the SQLite documentation.

Second comes the table method. It's functionality could be as follow:

Get the columns defined for the table
Check if a table exists:
- If it does, then we check each column, if it doesn't exsits then we create it (To keep this simple we won't be supporting migrations, though they can be added as needed).
- If it doesn't, then we create the table with the columns

The code for this is straightforward as written above:

def table(_name, &block)
  @columns = []

  self.instance_eval(&block)

  if table_exists?(_name)
    @columns.each do |column|
      unless column_exists?(_name, column[:name])
        add_column(table: _name, column: column[:name], type: column[:type])
      end
    end
  else
    create_table(table: _name, columns: @columns)
  end
end

Now we have to define some methods that are specific for SQLite: table_exists?:

def table_exists?(_name)
  table_exists = @db.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", _name.to_s)

  return !table_exists.empty?
end

column_exists?:

def column_exists?(_table, _name)
  @db
    .execute(
      "SELECT COUNT(*) FROM pragma_table_info(?) WHERE name=?",
      _table.to_s,
      _name.to_s
    )
    .flatten
    .all? { |n| n == 1}
end

add_column

def add_column(table:, column:, type:)
  @db.execute("ALTER TABLE #{table} ADD COLUMN #{column} #{type}")
end

create_table

Here the query is as follow: CREATE TABLE table_name column1 typeA, column 2 typeB; we use a little helper method inline to simplify generating this query.

def create_table(table:, columns:)
  @db.execute("CREATE TABLE #{table} (#{inline(columns)})")
end

# convert [{
#   :type => "string"
#   :name => "test"
# }, ...]
# into: name type, name type, name type,
def inline(columns)
  columns
    .map { |column| "#{column[:name]} #{column[:type]}" }
    .join(", ")
end

@db is defined in the initialization:

def initialize(db_name)
  @db = SQLite3::Database.new db_name
end

Now is a good time to split the adapter agnostic code into it's own class: Adapter where we'll currently have 2 methods: table and column. Then SQLiteAdapter will have all the other adapter specific methods.

The final code for the Schema is as follow:

class Adapter
  def table(_name, &block)
    @columns = []

    self.instance_eval(&block)

    if table_exists?(_name)
      @columns.each do |column|
        unless column_exists?(_name, column[:name])
          add_column(table: _name, column: column[:name], type: column[:type])
        end
      end
    else
      create_table(table: _name, columns: @columns)
    end
  end

  def column(_type, _name)
    @columns << {
      :type => map_type[_type],
      :name => _name.to_s
    }
  end
end

class SQLiteAdapter < Adapter
  def initialize(db_name)
    @db = SQLite3::Database.new db_name
  end

  def table_exists?(_name)
    table_exists = @db.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", _name.to_s)

    return !table_exists.empty?
  end

  def column_exists?(_table, _name)
    @db
      .execute(
        "SELECT COUNT(*) FROM pragma_table_info(?) WHERE name=?",
        _table.to_s,
        _name.to_s
      )
      .flatten
      .all? { |n| n == 1}
  end

  def add_column(table:, column:, type:)
    @db.execute("ALTER TABLE #{table} ADD COLUMN #{column} #{type}")
  end

  def create_table(table:, columns:)
    @db.execute("CREATE TABLE #{table} (#{inline(columns)})")
  end

  # convert [{
  #   :type => "string"
  #   :name => "test"
  # }, ...]
  # into: name type, name type, name type,
  def inline(columns)
    columns
      .map { |column| "#{column[:name]} #{column[:type]}" }
      .join(", ")
  end

  def map_type
    {
      :string => "TEXT",
      :int => "INTEGER"
    }
  end
end

module Schema
  def self.define(db_name, &block)
    adapter = SQLiteAdapter.new(db_name)
    adapter.instance_eval(&block)
  end
end

The above is good enough for our example, and there are many things that can be added to improve developer experience:

Add restrictions and error messages
Add logging
Execute queries in transactions
Support migrations
...

Let's move on to the next part.

Introducing the Models

In this section, we'll be focusing on this part of the code:

# Models
class User < Model
  def self.adult?
    self.age > 18
  end
end

# Seed
User.create(name: "John Doe", age: 42)
User.create(name: "Tipsy Lala", age: 25)
user = User.create(name: "Lorem Ipsum", age: 19)

user.name == "Lorem Ipsum"


User.where(:age, '>', 20).count

Our Model will have 2 methods:

create: will persist the record into database.
where: will filter based on the conditions that we provide. The supported operations are: == and != for both String and Int types and these operations only for Int type: > and <.

Here are the things that things that we need to address to get the above working:

Know which table a model references (based on name convention)
Add the ability to create a record and return it (create method)
Add the ability to filter/search a record (where method)

We find the table

Let's have a defined name convention that the users of this framework would use: Model name is the camel case of the singular of the table name. For example, to find the table name for the User model: Convert to underscore case (User -> user) -> then get the plural of it: users. First we add String.to_underscore method:

# source https://stackoverflow.com/a/1509939
class String
  def to_underscore
    self.gsub(/::/, '/').
    gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
    gsub(/([a-z\d])([A-Z])/,'\1_\2').
    tr("-", "_").
    downcase
  end
end

Then we add the ability to get the plural form of a name:

require "linguistics"

Linguistics.use :en

"user".en.plural == "users"

Now to get the table name from the model we define a new method that get the class name and do the transformation above on it:

class Model
  def self.table
    @table ||= self.name.to_underscore.en.plural
  end
end

Let's create some records (`Model.create`)

The create method have 2 functions:

Insert the record into the database
Return the created record as an object

To keep things simple, we won't add the ability to modify a record attributes.

Inserting the record into the database

Inserting the record into the database is straightforward:

User.create(name: "John Doe", age: 42)
# should execute this query
db.execute("INSERT INTO users (name, age) VALUES (?, ?)", ["John Doe", 42])

For User.create, the parameters will be passed as a single hash called: params. Let's see how to get each part:

users: that's the table name, Model.table will return it
(name, age): that's the joined params keys: params.keys.join(', ')
(?, ?): that's a ? for each params: (['?'] * params.count).join(', ')
["John Doe", 42]: these are params

We don't actually want to deal with the database from Model; Instead will be calling a method on the SQLiteAdapter, that in turns will execute the query. Though before this, let's move the database connection @db from an instance to a class variable @@db. With this the SQLiteAdapter.insert method will take the table name as param and the other fields and another one. As follow:

def self.insert(table, params)
  @@db.execute("INSERT INTO #{table} (#{params.keys.join(', ')}) 
              VALUES (#{(['?'] * params.count).join(', ')})", params.values)
end

And Model.create will just pass the table name along side with the arguments:

class Model
  def self.table
    @table ||= self.name.to_underscore.en.plural
  end

  def self.create(params)
    SQLiteAdapter.insert(self.table, params)
  end
end

Note: We'll be refactoring the calls to SQLiteAdapter to a constant in a later stage to make it easier to configure which database adapter to use.

Let's return the record on creation

This will get this part to work:

user = User.create(name: "Lorem Ipsum", age: 19)

user.name == "Lorem Ipsum"

We can achieve this by dynamically defining methods with column names that return the provided values. To achieve this we need:

Get table columns (SQLiteAdapter.get_columns)
Define the methods (Model.create_attributes)

The implementation would be as follow:

def self.get_columns(_table)
  @@db
    .execute(
      "SELECT name FROM pragma_table_info(?)",
      _table.to_s
    )
    .flatten
end

def self.create_attributes(params)
  SQLiteAdapter.get_columns(self.table).each do |column|
    self.class.send(:define_method, column.to_sym) do
      params.fetch(column.to_sym, nil)
    end
  end
end

Now we modify Model.create to call create_attributes and return the object itself:

def self.create(params)
  SQLiteAdapter.insert(self.table, params)
  self.create_attributes(params)
  self
end

Let's find these records (`Model.where`)

Here we'll get this part to work:

User.create(name: "John Doe", age: 42)
User.create(name: "Tipsy Lala", age: 25)
User.create(name: "Lorem Ipsum", age: 19)

User.where(:age, '>', 20).count # 2

Limitations that we'll take into account:

We can only filter by one attribute
The operators are directly mapped into the database (i.e. should use common ones e.g. >, ==, >= ...)

First we add User.where that forward the call to SQLiteAdapter:

def self.where(field, op, value)
  SQLiteAdapter.search(self.table, field, op, value)
end

Now SQLiteAdapter.search will work as follow:

Get all columns by executing the "SELECT #{columns} FROM #{table} WHERE #{CONDITION}
Map the results from an array (e.g. ["John Doe", 42] => {"name" => "John Doe", "age" => 42}

def self.search(table, field, op, value)
  columns = self.get_columns(table)

  results = []

  @@db
    .execute("SELECT #{columns.join(', ')} FROM #{table} WHERE #{field} #{op} #{value}")
    .map do |result|
        record = {}
        result.each_with_index do |val, idx|
          record[columns[idx]] = val
      end

      results << record
  end

  return results
end

Now we move on to routing.

Now comes the routing

In this section we'll get this part working:

get '/' do
  "Hello 👋"
end

get '/filter' do
  filter_age = params['age'] || 20

  count = User.where('age', '>', filter_age).count

  "Number of users who are older than #{filter_age}: #{count}"
end

We need to define:

get: define get route and provide a response
params: store parameters as hash

Let's start by creating our basic Rake app with the default mapping "/" returning "Ruby On Lane":

class RubyOnLane
  def self.app
    @app ||= begin
      Rack::Builder.new do
        map "/" do
          run ->(env) {[200, {'Content-Type' => 'text/plain'}, ['Ruby on Lane']] }
        end
      end
    end
  end
end

Next we want to define get, what it should be doing:

Define the path
Define that response

But first let's define a core part of the app, the request handler i.e. the class to be called on each request. This handler will:

Get the response body from the block passed to get by evaludating it and storing its return value
Define the params helper
Build out the request using Rack::Request

Here's the code for it:

class Handler
  def initialize(&block)
    @block = block
  end

  def params
    @request.params
  end

  def call(env)
    @request = Rack::Request.new(env)

    @body = self.instance_eval(&@block)

    [200, {"Content-Type" => "text/plain"}, [@body]]
  end
end

Now get can be defined as:

def get(path, &block)
  RubyOnLane.app.map(path) do
    run Handler.new(&block)
  end
end

And then we can run the app directly using: Rack::Server.start :app => RubyOnLane.app, here's an example app:

require 'rack'
require 'rack/server'


class Handler
  def initialize(&block)
    @block = block
  end

  def params
    @request.params
  end

  def call(env)
    @request = Rack::Request.new(env)

    @body = self.instance_eval(&@block)

    [200, {"Content-Type" => "text/plain"}, [@body]]
  end
end

class RubyOnLane
  def self.app
    @app ||= begin
      Rack::Builder.new do
        map "/" do
          run ->(env) {[200, {'Content-Type' => 'text/plain'}, ['Ruby on Lane']] }
        end
      end
    end
  end
end

def get(pattern, &block)
  RubyOnLane.app.map(pattern) do
    run Handler.new(&block)
  end
end

get "/hello" do
  "Hello #{params['name'] || "World"}!"
end

Rack::Server.start :app => RubyOnLane.app

Visiting http://localhost:8080/hello?name=Test will return: Hello Test!

Finaly we define a runner

This will let us run the app as is without having to explicitly use: Rack::Server.start :app => RubyOnLane.app

if ARGV.length > 1
  puts "Usage: ruby runner.rb app_name"
end

if ARGV.length == 1
  app_name = ARGV[0]

  unless app_name.end_with?('.rb')
    app_name = app_name + '.rb'
  end
end

require File.expand_path(app_name)

Rack::Server.start :app => RubyOnLane.app

Usage is as follow: ruby runner.rb app where the app is stored in 'app.rb'.

Conclusion

Fig.2 - Proof of concept Ruby on Lane application.

You can find the full source code in this gist.

This was just a demostration, I purposely left things out to keep this as short as possible while covering this topic.

I had lots of fun writing this and I learned more about Rack and metaprogramming in Ruby.

BTW if you enjoyed this and want me to build something in a similar topic, please feel free to send me an email.