Redstone Dev-Log #1

This post is currently in a draft state.

Redstone is a new learning project I started about a week back. The goal is to implement a language server for Java in Rust, along with a VSCode extension to support it. I plan to write regular dev-logs about the progress, and this is the very first one.

Why Redstone?

The short answer is because I am both curious and bored. Through this project, I plan to explore the internals of rust-analyzer as a reference implementation of a language server in Rust, and eclipse.jdt.ls as a reference for a Java language server. I also want to learn more about parsers, abstract syntax trees (ASTs), concrete syntax trees (CSTs), grammars, and the process of building a semantic model out of raw source code. Error-resilient parsing and full-fidelity syntax trees—common in code editors—are another area I intend to study.

This will be an opportunity to get serious hands-on experience with Rust, especially in writing large projects, performance and memory optimizations, and learning common patterns. I also aim to deepen my understanding of the Language Server Protocol, figure out how to write a VSCode extension, and implement semantic analysis for Java. Even if the project never becomes production-ready, I’ll consider it a success if I manage to learn all of the above.

With that in mind, let’s look at what’s been done so far.

Basics of LSP

The Language Server Protocol (LSP) standardises how editors communicate with a language server.

^ Diagram from VSCode Documentation

It uses the JSONRPC protocol. While I will go deeper into LSP in a later post, some basic concepts are worth noting now. There are three types of messages—Request, Response, and Notification. Requests and Responses are tied together with an ID, while a Notification is more like an event and does not have a response. Requests support cancellation through the $/cancelRequest notification. Finally, there is a handshake process during initialization, where both the client and server share the capabilities they support.

VSCode Extension

The VSCode extension now has its boilerplate setup, starts a new language client and language server, and sends requests from the client to the server in response to editor events.

VSCode provides tooling to generate extension boilerplate, as described in their guide. The extension here acts as a language client, sending requests to the language server based on editor events. This is done using the LanguageClient implementation from the vscode-languageclient/node package:

let client = new LanguageClient(
	"redstone",
	"Redstone Java Language Server",
	serverOptions,
	clientOptions
);

The client is given serverOptions—in this case, an Executable { command } pointing to the redstone-language-server binary. For now, I’ve hardcoded the path to Cargo’s build target for the server, but eventually the binary will be bundled with the extension. The client runs this command in a separate process and communicates with it via stdio.

Once connected, the Language Client sends requests such as textDocument/DidOpen or textDocument/DidChange to the language server. These are sent only if the server has indicated support for them in the serverCapabilities shared during the handshake.

Redstone Language Server

The language server can now accept events and log the incoming requests. Some initial boilerplate is in place for AST and CST structures, code generation for SyntaxKind, and a Java UnGrammar based on the Oracle Java Language Specification. Partial code generation of AST nodes has also been implemented.

The server is built using the lsp-types crate, which provides all the LSP types, and the lsp-server crate from rust-analyzer, which contains their own language server implementation. These make it easy to get a basic server running quickly. After initializing the stdio connection and completing the handshake, the server runs an event loop using Crossbeam channels and the select! macro to receive and handle messages. An exit notification will break this loop and shut down the server.

Abstract Syntax Tree and Concrete Syntax Tree

This is a complex topic I will cover in depth in a future post. For now, it’s worth noting that unlike compilers, editors and language servers require full-fidelity ASTs and error-resilient parsing. One approach is to use an untyped Concrete Syntax Tree and then layer an AST as a view on top of it. The rowan crate from rust-analyzer provides an efficient CST implementation that supports this.

Java UnGrammar and Code Generation

UnGrammar from rust-analyzer lets you describe your AST in an EBNF-like format, then gives programmatic access to it. This is particularly useful for generating AST abstractions.

For example, the Java UnGrammar defines:

ImportDeclaration =
  SingleTypeImportDeclaration
| TypeImportOnDemandDeclaration
| SingleStaticImportDeclaration
| StaticImportOnDemandDeclaration

SingleTypeImportDeclaration =
  'import' TypeName ';'

TypeImportOnDemandDeclaration =
  'import' PackageOrTypeName '.' '*' ';'

SingleStaticImportDeclaration =
  'import' 'static' TypeName '.' Identifier ';'

StaticImportOnDemandDeclaration =
  'import' 'static' TypeName '.' '*' ';'

In UnGrammar, ImportDeclaration becomes a node with an alt rule:

"ImportDeclaration": {
  "alt": [
    { "node": "SingleTypeImportDeclaration" },
    { "node": "TypeImportOnDemandDeclaration" },
    { "node": "SingleStaticImportDeclaration" },
    { "node": "StaticImportOnDemandDeclaration" }
  ]
}

This can be used to generate a Rust enum representing the ImportDeclaration AST node:

pub enum ImportDeclaration {
	SingleTypeImportDeclaration(SingleTypeImportDeclaration),
	TypeImportOnDemandDeclaration(TypeImportOnDemandDeclaration),
	SingleStaticImportDeclaration(SingleStaticImportDeclaration),
	StaticImportOnDemandDeclaration(StaticImportOnDemandDeclaration),
}

What’s next?

TBA