Tech Corner: The Crux Web Application Server

Back in the early 2000s, blogs were becoming a thing and WordPress was becoming quite popular as a blog software platform. It was some time in 2004 or so that I was reading a technical article on a piece of technology called XmlRPC. It was a software specification which enabled WordPress to do its thing and allow authors to write and post blogs using their browser.

After reading the article, I thought to myself, “This thing is a security nightmare.” And, sure enough, within weeks of the article’s publication, and not due to any “secrets” being released, WordPress and the various blog sites relying on its software were being hacked and/or brought down due to DoS (Denial of Service) attacks.

Some Technical Tidbits

I don’t know how WordPress does its particular brand of magic now. What I can talk about is XmlRPC circa 2004. And to showcase the security issues I discovered in the article, I need to walk you through some technical nitty-gritty. If this is not your bag, I would suggest moving to the next section.

What is RPC?

RPC stands for Remote Procedure Call and it is an overarching term for one computer process talking to another computer process to do something for it. Maybe it’s your web browser asking a computer server for the current weather in Phoenix, for instance. But how does it do that? Here we get a bit technical and we need to dig deeper.

What is a Procedure Call?

In very, very general terms, a procedure call is a way provide functionality via software. For instance, let’s build a procedure called “sum” which will add two numbers together and returns the result.

Procedure calls can receive data as “arguments”. An argument is a piece of data passed to a procedure call, or function. For our example, we’re going to define a procedure call named “sum” which takes two arguments, which we’ll add together. Arguments usually need names, so we’ll name them “a” and “b”.

Different programming languages use different syntaxes to do their work. As an example, I’ll show how “sum” can be defined using three different languages: C, Ruby and Javascript.

int sum(int a, int b) { return a+b; } // This is C code
def sum(a,b) a+b  end # Ruby code
function sum(a,b) { return a+b } // Javascript code

I’m not going to go into the details of the languages; that’s a different sort of discussion. But I will focus on the ‘C’ language version. If you take a look at the C language version, you will see these “int” words. These words tell the C compiler, the tool used to convert code into programs, that the type of data involved are integers. This is important for C code and a reason why C code is much faster than Ruby and Javascript.

In compiled languages, there is quite a bit of communication taking place among the various code modules so that the compiler can do its job. In the C language example above, the compiler is going to generate code that assumes that there will be two integers placed at specific points in memory and that the function will look for those two integer pieces of data in memory, load them into the CPU, add them, and finally place them in a specific piece of memory. (I’m purposely glossing over such things as stack management and frame pointers and the like. You’re welcome.) The point though, here, is that enough information is provided so that the compiler can make sure that any piece of code which uses the sum procedure know where to put the data and where in memory the result will be placed.

So, lets say that some C code contained a statement like the following:

x = sum(2,5)

The variable ‘x’ would contain the value 7 when this code was compiled run. Under the covers, what would actually happen is that the value ‘2’ would be placed in one memory location, the value ‘5’ in another, the processor’s program counter would jump to the code for the sum procedure, load those values from the special places, add them and place ‘7’ into another special memory location, jump back to where it started and pull ‘7’ from the special memory location. It’s faster than it sounds, but it is fast and the code only needs to know about the special memory locations defined in the language specification.

What was wrong with XmlRPC?

The compilation method that I described only works for procedures embedded within one computer process. But, RPC is all about talking with different computer processes, probably on another computer. How is this done?

There is a language called XML, of which HTML, the language of the web, is a subset. One way for different computer process to “talk” with each other is to send formatted data back and forth. Around 2004, that language was XML.

Like HTML, XML uses tags. The format looks similar to this:

<tag attribute="value">data</tag>

The data of some tags can also contain other bits of tagged data, forming a giant tree. For RPC, this would be overkill, but most web page do contain a nested tree of tagged data bits.

So, what was XmlRPC doing? Well, let’s assume that a remote server has defined the sum procedure and is waiting for some XML to do its work. Perhaps, something like the following:

<function name="sum">
  <argument>2</argument>
  <argument>5</argument>
</function>

This looks very similar to the C code version, and it would probably work. However, the devil is in the details.

When C code is compiled, those “int” words are telling the compiler about the nature of the data being passed to the procedure. If you pass incorrect data to it via code, you will get a compiler error and the program won’t build. This is called type-safety. In other words, don’t pass me shit I either don’t know about or don’t care about. I add two integers together. That’s it. Do anything else and the program won’t build.

So, given the XmlRPC specification at the time, it was possible to send this as an RPC call:

<function name="sum">
  <argument>Your mother</argument>
  <argument>was a hamster!</argument>
</function>

Those aren’t numbers. At the very least, the RPC call should return an error, if coded properly. Maybe we can assume that much.

However, let’s consider a function which rolls dice. What if one of the arguments was an integer which specified the number of dice rolls and a malicious entity sent a message where a billion dice rolls were requested? Perfectly legal and one billion is an integer so we’re not violating the type of the argument. It’s simply a malicious request to send your server on a wild goose chase and waste time.

Or, taking maliciousness out of the equation, what if you simply put the arguments in the wrong order? After all, considering the XML written above, there is no knowledge of what the data is to be used for.

Contract Negotiations

After reading the XmlRPC specification and seeing the security issues within (I’m sure they’ve been fixed by now), I thought to myself, “How could I make something better?” And that’s how I spent years of my free time building the Crux Web Application Server.

For the sake of brevity, believe it or not, I’m going to fast forward through all of the various experiments I went through. For data transmission, like XmlRPC, I initially used XML. Then I added a YAML variant. Then I added a JSON variant. All three could be supported, until I asked myself, “Why?” For whom am I building this? So now, JSON is the data transmission format.

Before any code is written, the RPC (or service call or API) author must write the specification. The specification defines the following:

• the name of the call
• the credentials it needs, if any
• the name of the module and function to be called after passing the contract negotiation
• the list of inputs, or arguments, and their types and their optional constraints
• the output and its type

Once the specification is written, the specification compiler takes that information and writes two Ruby methods (Crux is written in Ruby): one which will receive the initial code request and one which contains the contractual obligation of the service call. If the incoming data does not pass muster, it is rejected before it even gets to the “meat” of the functionality, which is in a third human-editable Ruby module.

This core is the beating heart of the Crux system.

The IMDE

But any good system needs good tools. So I wrote a Web interface called IMDE, for Integrated Management / Development Environment. The IMDE is how service call can be tested before any other application gets hooked up to them.

Once I’m logged in, I’m greeted with the following screen:

There are six tabs of information:

• Service Manifest: the list of defined services
• Service Calls: an interactive way to test the calls
• Development Make: the method to compile service specifications
• Crux User Management: a way to manage Crux users across all Crux applications, including the management of rights and permissions
• Crux Site Logs: a way to review generated logs and/or cull old ones
• Crux Site Map: a way to display the listed Crux applications and their access URLs

The Demo

Let’s build a new Crux service call. Let’s call it reverse. However, it needs to be part of a Crux application and a service within that application. For the purposes of this demo, we’ll use the ‘tocs’ service group and the AService service within that group, which I use for throwaway test code.

First, let’s take a look at the list of existing services within the tocs.AService domain:

The contents of the current list is not important. As I said, it contains throwaway test code. The primary things to note are that the methods list is in alphabetical order and that there is no method named reverse. Let’s change that.

First we need to write the specification. The specifications are contains in a subdirectory called ‘specs’ and a further subdirectory named after the applications suite, ‘tocs’ for our example. Within that directory is a file named AService.crux. To add the specification, we need to edit this file, like so:

Here you can see the specification, which is English-based (though no AI language models are used). The specifications file has a heading section noting the name of the service, the author(s) for the file as well as library requirements. For the AService, the AManager library is a requirement.

After the heading, each method has its own specification area. Here, we define the ‘reverse’ service and describe its functionality, its interface and its parameters. This tells us that AService.reverse will call AManager.reverse with one supplied argument named string and will return the reversed string in a field called outcome. The ‘noclient’ option to the interface tells the specification compiler that an argument for database connectivity will not be needed.

With the specification written, we need to compile it using the IMDE:

This tells us that the AService specification has been compiled to the generated service code, the generated contracts code and has emitted specification metadata to be referenced by the system.

After restarting the server (process not shown), and adding the functionality to the AManager module, we can use the IMDE to test it.

The code:

class AManager

  def self.reverse(string)
    string.reverse
  end

  ## More code below...

end

The following images show: the new method added to the list of available methods, the test harness after the method is selected, the results of calling the method.

Finally, the new reverse method can be seen in the services manifest window, showing its specification:

With this system in place, I can build web-based services in mere minutes. Back in 2018 or so, the on-line dice roller that I built back in 2000 was ported to this system and is still being used today. It has proven to be pretty robust.

The Future: WebSockets

A transition or extension to support WebSockets is needed to support live network connections and push notifications. Right now, web-based apps which use this system use polling to determine if time-consuming work is finished and/or whether a user has received notifications. Supporting a methodology which can support “fire-and-forget” service requests and push notifications will add quite a bit of value to this system.

In my next technical post, I will dive a bit deeper into the contract negotiation capabilities of the system.

Stay tuned.