Why nim?

I first encountered nim at a PyGotham talk about optimizing hotpaths in Python code by actually rewriting the logic in nim and importing it back into your Python application. The language's Python-like syntax, static typing, easy-to-use FFI, and focus on performance appealed to me. Since that talk, nim was in the back of my mind as a new low-ish level language to pick up.

Fast forward to a few months ago, I felt the familiar urge to learn a new programming language. First I tried Rust due to its recent inclusion into the kernel, but I ended up being fairly unproductive in it after a few nights of hacking. I realized that it would take a lot more work to get comfortable with the compiler before I could write anything substantial. Not to knock on Rust, I'm still spending time learning it! But with limited brain capacity after a long day of work, I just wanted to sling some code and explore some new computing concepts that interested me.

Enter nim. After my first night of coding, I was actually able to write working code that implemented basic CRDT data types. I was happy since my code compiled, I could quickly iterate on it due to its python-like syntax, and I was quickly getting the hang of the type system.

QuestDB

In September, I joined QuestDB as a Senior Cloud Engineer and became fully immersed in the project and its ecosystem. One of the ingestion methods that QuestDB supports is the Influx Line Protocol (ILP). Even though we have a bunch of mature ILP clients written in Rust, C++, Python, and Java, I'm a hands-on learner and wanted some experience working with the database. I figured that writing an ILP client in nim would be a fantastic project to improve my skills while also learning more about the about the protocol and some QuestDB fundamentals.

questdb-nim

After some hacking, I ended up with a general purpose ILP client that supports both synchronous and async execution. Nim made my job easy with a few nice features:

Single file execution

Unlike other compiled languages, nim makes it easy to compile and execute any file with a .nim extension. Similar to python's if __name__ == "__main__":, you can run code under if isMainModule: simply by adding the -r flag to the nim compile command. This lets you iterate quickly and easily when working on smaller features instead of setting up an entire test harness and/or framework.

Autodocumentation

Nim comes with an autodocumentation tool, nim doc, that makes it incredibly easy to generate clean html docs for your module. For a production-grade project, I would implement this in a CI pipeline using GitHub actions or a similar CI/CD platform. But for the initial stages of the library, I decided to just include a git pre-commit hook and installation command in my Makefile. While this does make commits larger and clutter up the git history slightly, all documentation-related changes are in a single directory so they should be easy to ignore when looking at code-related changes.

Compile-time support

Nim has a strong sense of compile-time support including metaprogramming, easy-to-use compiler pragmas, and even the ability to execute code at compile time. For example, adding async support to the library was simple using the {.async.} pragma. All I had to do was mark methods with that pragma and they would be eligible for use with the std/async* packages, which make it very easy to instantiate objects and execute async code.

The Library

You can find the finished product on Github: https://github.com/sklarsa/questdb-nim/. I've tested it against a local QuestDB instance, and have some simple unit tests to validate basic functionality.

Future work

  1. Add ILP authentication support to the library
  2. Improve ILP-line parsing and error-handling. There are still many edge cases that would not be parsed correctly by the logic that I implemented.
  3. E2E testing and benchmarking

Nim Resources