The Bug Report

Simplicity is Unforgiving, Accessibility Also Matters

2025-04-25T00:00:00+00:00

Apple III Personal Computer — Photo by Mert Kahveci on Unsplash

On the First Day, There Were VT100 Terminals

On Simplicity

The UNIX sages of lore said it first and said it best, “Do one thing and do it well.” There was a time long before this became the mantra of wizened sysadmins and software developers, preaching suckless software and leading the crusade against feature creep, enshittification, and thousands of configuration options and moving parts you’ll probably never use when you’re just trying to get

One. Thing. Done.

That time, my friends, was 1978. Computers were so new that the main focus of any systems or hardware engineer was just to get the damn thing to work at all. Of course, simplicity was still a vital component of their workflows. When 2MB of addressable memory space and a 16-bit 3.3MHz processor is considered high-performance computing, you cannot afford to abstract your programs beyond bit-level control of hardware I/O and direct manipulation of registers. These are the specs for the PDP-11/70, an extremely popular computer around the time of the VT100 terminal, one of the most important devices in computing history.

The VT100 terminal released by the Digital Equipment Corporation (DEC) in 1978 was not the first raster-based video terminal (a.k.a. display terminal, glass terminal). That honor belongs to the IBM-2260 14 years prior. Nor was it the first terminal to contain a microprocessor (smart terminals, as opposed to dumb terminals which directly controlled logic gates for serial communication and text display, and not much else). That was the HP 2640A in 1974 using an Intel 8008 microprocessor.

VT100 Terminal and User Manual — Photo by Ade Bradshaw — CC BY-NC-SA 2.0 License

Standards Compliance is Based, Actually?

It is, however, the video terminal that every terminal emulator today must, well… emulate. In a landmark decision to make vendor lock-in really fucking hard for companies selling $3000 screens to go with their $20000 computers, the American National Standards Institute (ANSI), specifically the Accredited Standards Committee X3 for Information Processing (ASC X3) released the ANSI X3.64 standards in 1979, known as ECMA-48 internationally. This was a political move by ANSI bought and paid for by Big Acronym to sell more acronyms, or whatever.

ANSI X3.64 standardized control sequences for video terminals, used for things like cursor movement, text formatting, scrolling windows, and the like. DEC did not create this standard, but they were a major player in ANSI and were among the 18 corporations, 19 consumer groups, and 16 trade association groups who developed and approved the standard. You can thank them for the ability to clear your screen, move your cursor from left to right, and display characters consistently across any terminal that adheres to these standards.

So what made the VT100 the granddaddy of all modern day terminal emulators? They were the first to release a commercially successful product that adheres to the X3.64 standards. ANSI’s crusade against vendor lock-in worked, but not quite as intended. The VT100 was designed around a draft specification released a year prior to the finalized standard and added some of its own escape sequences like DEC Private Mode (a later model also introduced Sixel graphics). When it became the most popular terminal in the world shortly after its launch, other terminals started emulating these proprietary VT100 escape sequences in the name of software interoperability. VT100 compatibility, and later VT220 compatibility, became the gold standard even over ANSI compliance. Julia Evans did a great writeup on escape sequences here if you want to get deep in the technical weeds.

On the Second Day, There Was Xterm

Abandon All Hope, Ye Who Enter Here

If the VT100 is the granddaddy of terminal emulation, then Xterm is the father with a busy corporate job in a loveless marriage and no time to bond with his kids. As Xterm’s README from 1989 humorously points out, it’s an entangled mess of legacy systems.

Abandon All Hope, Ye Who Enter Here

This is undoubtedly the most ugly program in the distribution. It was one of the first “serious” programs ported, and still has a lot of historical baggage. Ideally, there would be a general tty widget and then vt102 and tek4014 subwidgets so that they could be used in other programs. We are trying to clean things up as we go, but there is still a lot of work to do.

Xterm VT Emulation Options — Screenshot by Thomas Dickey — Wikimedia Commons

How Xterm Shaped Modern Terminal Emulators

Xterm is a software terminal emulator originally written in 1984 for the VAXStation 100 (VS100) and ported to X11 in 1987 as part of Project Athena at MIT. Xterm is still installed by default in some distros, and is widely available in most Linux and BSD package managers.

It is highly efficient as well, consuming as little as ~9MB to spawn a window depending on your environment. For comparison on my system, st with no patches takes ~13MB, st with patches takes ~21MB, and alacritty takes ~185MB, all running zsh.

Shipping in the X Window System meant that Xterm quickly became the de facto terminal emulator on most commercial and free UNIX systems of the day. Since Xterm emulated the control sequences of the VT100 and many control sequences from the VT220 and later models, lots of old software that relied on these escape sequences expect TERM=xterm-256color to be set in your shell. This includes some of the most foundational programs in computing history, including but not limited to:

vim
neovim
tmux
fzf
bash
anything built on ncurses

They will probably still work without setting this environment variable, but may fallback to 8-bit mode, break your keybinds, color schemes, mouse inputs, or in rare cases where even TERM=vt100 isn’t available, will just fail to start.

Modern terminal emulators may define their own TERM values and implement features beyond those supported by Xterm. However, nearly all support fallback behavior for compatibility with essential tools that expect Xterm escape sequences. The historical influences of ANSI standards, video terminals, and Xterm on today’s terminal emulators are quite complex, so I made a diagram to clearly outline these relationships.

UML Diagram of Xterm’s “Historical Baggage” — Created by The Bug Report

On the Third Day, There Was Suckless Software

Simple Software Sucks Less

I’m a big fan of suckless software from a security perspective. If your program has fewer features built with fewer lines of code, your attack surface is reduced by default. Simple as that. However, convenience is often the price you pay for simplicity.

I’ve been using vanilla dwm about as long as I’ve been on Linux. It’s easily one of my top three pieces of software — maybe even my favorite outright. I switched from Windows to Linux for three main reasons:

A minimalist, distraction-free desktop
A massive FOSS ecosystem/mature dev tools
A desire (nay, a need) for system ownership and personal privacy

dwm absolutely fits the bill for all three (although #3 seems to be an unsolved problem, even in the most minimal Linux setups). There’s no notification daemon to take away my focus from important things. The code is heavily audited by expert C developers. It does the absolute minimum job of a window manager: manages your windows, then gets out of the way. The keybinds are smooth, configuration is as easy as changing options in config.h, and since embracing the CLI, I don’t miss messy desktops and start menus in the slightest.

Above all, it respects your time and agency by not vying for your attention, and not making promises it can’t fulfill. Surely Suckless’ other flagship products st and dmenu are just as good?

Oh st, How I Wish I Could Love You

Suckless’ terminal emulator st does one thing and does it well: emulates a terminal environment from which you can grep and awk to your heart’s content. However, that’s about the only feature on a vanilla st install. Here are just a few features that are available as patches, but aren’t part of the upstream build:

alpha transparency
scrollback buffer
restoring window title to previous state (CSI 22 and 23 escape sequences)
explicitly declaring fallback fonts
consistent column redraw upon resizing terminal window
Switching IMEs in a single session
Displaying IME composition suggestions (i.e. zhong → 中, 重, 症, 终)
ligatures
numerous glyph/line rendering QoL features

I can’t deny that st has its use cases. If you’re running on a twenty-year-old potato with no GPU acceleration or you just want the lightest possible terminal for scripting, it’s hard to beat. If you don’t care about scrollback, ligatures, or IME support, st is lean, fast, and beautiful in its simplicity. And if you do care about those features? Well, you can always add patches.

Just don’t expect to add ALL of them, because patches that make significant changes to the rendering logic of st like scrollback and ligatures will cause many other patches to fail. Now you’ve got rejected hunks and a merge conflict that spans 2,000 lines of manual layout code. At that point, your options are:

(a) compare diffs line by line and rewrite st yourself, or

(b) salt the fields and switch to a terminal that comes with sane defaults out of the box.

Just enough rice to admire, not to fingerprint. — Screenshot by The Bug Report

Hi, This Part is About Me Now

Am I a 10x Developer Yet?

This brings us full circle to three days ago, when I was setting up a staging environment on an SBC running OpenBSD to test and configure my web server and website backend. I didn’t want to have to write scripts and mess with config on my host machine and then rsync or git pull every time I wanted to test a change. I’d much prefer to SSH in and do development and testing all on the same machine.

This is where my decision to keep things minimal first bit me in the ass. I typically use VSCodium as my main code editor, but I’ll be administering my server headless, which means no need for Xorg. No GUI? No VSCodium. I’ve been putting Neovim on the back burner because I already had an IDE that worked and that I enjoyed using, but now seemed like as good a time as any to try it out. I pulled up ThePrimeagen’s Neovim config guide and got to work.

So far so good. Except, what is that? I know I typed “cmd”, so why does it look like “cma”?

I know packer is deprecated it was part of the guide — Screenshot by The Bug Report

Weird. My first thought was that maybe the font I’m using in my terminal is rendering italic ‘d’s incorrectly. I applied the font2 patch for st so I could set a fallback font, and then tried to force fontconfig to use that font ONLY for italic characters. About 10 poorly written XML config files later, I threw in the towel. I should have known better than to mess with Xorg libraries, that was my fault.

Okay, I guess I’ll go back to Liberation Mono, the default font that st was built with, and would you look at that! The glyph is still overdrawn and not rendering correctly. I’m nearing my wits end, and I’m getting tired. Tomorrow will bear better results.

It’s Not a Bug, It’s a Feature

The next day, exhausted from fighting a battle that never should have taken place, I take a long, hard look at Neovim.

It’s not just the ‘d’.

This glyph rendering bug is affecting quite a few italic characters actually. I start to think that maybe it’s not my fonts that are a problem, and decide to test this out in my terminal window.

Escape Sequences FTW — Screenshot by The Bug Report

Yeah, so it’s actually st's fault. It turns out that upstream st truncates glyphs that extend past their cell, and since italic characters are slanted, they ever so slightly extend into the next character’s cell. But hey, there’s a patch to fix that, so let’s apply it.

Aaaand the patch failed.

I took a look at the rejected hunk from x.c to see what the problem is:

I had a patch fail the other day to add ligature support to st. I backed up my config.def.h and st.c but neglected to back up x.c, which meant my x.c still had leftover changes — enough to confuse the patch tool, but not enough to fully implement the ligature patch either. When I tried applying the wide glyph support patch, it couldn't find the exact code patterns it was expecting. It wasn't that the patch itself was broken, but that my x.c was in a weird half-patched state.

Some may call this a skill issue, and I would be inclined to agree. I probably could have applied the patch if I had a backup x.c to revert back to, but these are solved problems in other terminal emulators, and I was tired of slowly adding back features that I was accustomed to. Do I really need to patch in the ability to scroll in my terminal window? Maybe one day I’ll go back, but for now, I want something that just works.

I’m using alacritty now, which implements in the upstream build every single one of the features I listed earlier that st requires patches for. It could do better on IME support though. It’s also one of the most minimal terminal emulators out there, relying on CPU rendering with OpenGL-based GPU acceleration, avoiding systemd dependencies, and doing not much else except emulating the proper escape sequences and some much needed quality of life improvements. Did I mention it’s written in Rust?

Final Thoughts

Modern software developers can learn a thing or two from the VT100. Sure, its limited feature set was a product of its time, but it didn’t need more. After all, most modern terminal emulators are still emulating the same escape sequences nearly 50 years later. How much better could our user experience be if rather than shipping as many features as possible, we shipped the minimum viable product, with just enough polish to be simple, intuitive, and amazing?

Tl;dr just use the software that lets you get shit done. Bug out.

AI, Language, and What It Means to Know

2024-10-22T00:00:00+00:00

From Data to Design: How AI Transforms Business Strategy

2024-09-24T00:00:00+00:00

How Generative AI is Shortening the Path to Expertise

2024-09-10T00:00:00+00:00

Creating the First Open-Source AI Digital Scribe

2024-08-27T00:00:00+00:00

HTTP Chronicles: Sending and Receiving Your First HTTP Request

2024-08-14T00:00:00+00:00

Photo by Alexandre Chambon on Unsplash

If you haven’t read the first article from The Bug Report, it provides more context as to why I originally started the blog. It’s not required reading, but it is a great place to start!

This is the second article in the “HTTP Chronicles” series. You can see all the articles that are a part of this series here.

As promised, I have something to show you. It’s something I’m very proud of and spent a long time working on, so if you don’t tell me how great I am, I promise you will be the first people I target when rogue AI’s are able to hack into anyone’s personal network at-will. You have been warned: I’ll go full Cyberpunk 2077 on your ass.

Here is the code I wrote for my HTTP server:

http_server.c (main)

#include "handlers.h"

#define PORT "8080" //Which port to send traffic through
#define BACKLOG 20 //How many connections can be in queue at once

int main() {
    int status, sockfd, new_fd; //Variables for status and socket descriptors
    struct addrinfo hints, *servinfo, *p; //Variables for address info
    struct sockaddr_storage their_addr; //Connector's address information
    socklen_t sin_size; //Size of struct sockaddr_storage
    char s[INET6_ADDRSTRLEN]; //Buffer to hold the client's IP address
    int yes = 1; //For setsockopt() SO_REUSEADDR

    memset(&hints, 0, sizeof hints); //Make sure the struct is empty
    hints.ai_family = AF_UNSPEC; //Allow for both IPv4 or IPv6 addresses
    hints.ai_socktype = SOCK_STREAM; //TCP Stream Sockets
    hints.ai_flags = AI_PASSIVE; //Easy IP Address Resolution

    //Get address info
    if ((status = getaddrinfo(NULL, PORT, &hints, &servinfo)) != 0) {
        fprintf(stderr, "getaddrinfo error: %s\n", gai_strerror(status));
        exit(1);
    }

    //Loop through all the results and bind to the first one we can
    for (p = servinfo; p != NULL; p = p->ai_next) {
        //Create a socket
        if ((sockfd = socket(p->ai_family, p->ai_socktype, p->ai_protocol)) == -1) {
            perror("server: socket");
            continue;
        }
    
        //Set socket options
        if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)) == -1) {
            perror("setsockopt");
            close(sockfd);
            exit(1);
        }

        //Bind the socket to the port and IP address
        if (bind(sockfd, p->ai_addr, p->ai_addrlen) == -1) {
            close(sockfd);
            perror("server:bind");
            continue;
        }

        break;
    }

    //Check if we successfully bound to a socket
    if (p == NULL) {
        fprintf(stderr, "server: failed to bind\n");
        return 2;
    }

    //Free the linked list of address info
    freeaddrinfo(servinfo);

    //Listen for incoming connections
    if(listen(sockfd, BACKLOG) == -1) {
        perror("listen");
        exit(1);
    }

    printf("server: waiting for connections...\n");

    //Main loop to accept and handle connections
    while (1) {
        sin_size = sizeof their_addr; //Size of struct sockaddr_storage
        new_fd = accept(sockfd, (struct sockaddr *)&their_addr, &sin_size);
        if (new_fd == -1) {
            perror("accept");
            continue;
        }

        //Convert the addr to a string and print it
        inet_ntop(their_addr.ss_family, get_in_addr((struct sockaddr *)&their_addr), s, sizeof s);
        printf("server: got connection from %s\n", s);

        //Handle the HTTP request in a child process
        if (!fork()) {
            close(sockfd); //Child doesn't need listener
            handle_http_request(new_fd); //Defined in handlers.h
            exit(0);
        }
        close(new_fd); //Parent doesn't need connection socket
    }
    
    return 0;
}

handlers.h

#ifndef HANDLERS_H
#define HANDLERS_H
#include <stdio.h> 
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h> //socket API functions and data types
#include <sys/types.h> //Data type definitions for <sys/socket.h>
#include <unistd.h> //POSIX API for syscalls
#include <netdb.h> //Network database operations
#include <arpa/inet.h> //IP address operations
#define BUFFER_SIZE 8196

/*
Helper function to get the IPv4 or IPv6 address from a struct sockaddr
Parameters: struct sockaddr*
Returns: Pointer to IPv4 or IPv6 addr (void * allows us to point to any data type)
*/
void *get_in_addr(struct sockaddr *sa);

/*
Function to handle an HTTP request and send a response
Parameters: connection socket file descriptor
Returns: A handled request
*/
void handle_http_request(int client_fd);

#endif

handlers.c

#include "handlers.h"

void *get_in_addr(struct sockaddr *sa) {
    if (sa->sa_family == AF_INET) {
        return &(((struct sockaddr_in*)sa)->sin_addr);
    }
    return &(((struct sockaddr_in6*)sa)->sin6_addr);
}

void handle_http_request(int client_fd) {
    char buffer[BUFFER_SIZE]; //Buffer to store the incoming request
    int bytes_received = recv(client_fd, buffer, sizeof(buffer) - 1, 0);
    
    if (bytes_received > 0) {
        buffer[bytes_received] = '\0'; //Null-terminate the received data
        printf("Received request:\n%s\n", buffer);

        //Construct a simple HTTP response
        const char *response =
            "HTTP/1.1 200 OK\r\n"
            "Content-Type: text/html\r\n"
            "Content-Length: 48\r\n"
            "\r\n"
            "<html><body><h1>Hello, World!</h1></body></html>";

        //Send the response back to the client
        send(client_fd, response, strlen(response), 0);
    }
    //Close the client connection
    close(client_fd);
}

Before you can understand what this code does, there are a few networking concepts we must introduce. Beej’s Guide to Network Programming gives a great overview of the data structures we use to carry information over networks, as well as the functions used in network operations. I highly recommend giving it a read, or at least keeping it open in another tab to refer to if you’re wondering what specific variables represent in the structs we’ve defined. I’ll do my best to explain them as well.

This code will only work on Linux, and will work with slight modifications for other UNIX-like operating systems. Network operations on Windows are different than those on Linux, but it is entirely possible to create a server using any popular OS available today.

Photo by Tim Evans on Unsplash

How Does Network Communication Work?

All network traffic goes through sockets, and those sockets are bound to a single port. Imagine a post office, where each person living in the town has their own P.O. box. The P.O. boxes temporarily store your mail between the time the Post Office (server) receives it and you (client) pick it up. Some post offices also allow you to send outgoing mail through your own P.O. Box to deliver somewhere else. A P.O. Box is similar to a socket — an endpoint that facilitates data transfer between the server and client.

In this sense, a port number is NOT like a P.O. Box number. A P.O. Box number is more reminiscent of an IP address, which identifies the traffic sent to and from this P.O. Box as belonging to you. A port’s purpose is to route a specific type of traffic using a specific network protocol. For example, all regular letters (HTTP traffic) might go through your P.O. box, while packages requiring a signature (HTTPS traffic) must be picked up directly from the mail clerk. Just as you might send and receive different types of mail in different ways, different ports handle different network services running on the same IP address, structured in a way that the data can be processed by it’s respective network protocol. Sockets, ports, and protocols form the basis for how essentially all computers across the world communicate with each other.

Photo by John Salzarulo on Unsplash

HTTP Request and Response Structure

So how does the HTTP specification say that a client should structure an HTTP request?

The bare minimum information sent in a request is the Request Line, and the Host header. It looks something like this:

GET /path/to/resource HTTP/1.1
Host: www.thebugreport.dev
...
Additional headers included here

The Request Line is made up of three components: A method, a URI, and the HTTP version.

The method defines what action you would like to take on the resource. For example, the GET method asks the server to locate the resource and display it back to the client. This is how webpages are shown to you. The POST method asks the server to publish a new resource that did not exist before. POST requests are commonly used for fillable forms, publishing articles, uploading files, processing payments, etc.

The Uniform Resource Identifier, commonly referred to as URI or Request Target, is similar to a file path that points to a resource, with a few key differences. Assume that you are requesting an image that is at the file path ${website_root_directory}/img/example.jpg. Since the Host header will automatically resolve to the root directory, you only need to specify /img/example.jpg. You CAN specify the host as well, using http:// or https:// to designate your protocol, and this is common in servers that you connect to through a proxy. Additionally, URIs can contain query strings and fragment identifiers that determine what state a resource is in when it’s shown to you, but that’s beyond the scope of this article.

And lastly, the HTTP version does exactly what it says in the name; it tells the server what version of the HTTP protocol to send their response with. The most common are HTTP/1.1, HTTP/2, and HTTP/3.

The Host header, in the vast majority of cases, points to the domain or subdomain of the resource you are trying to access. If my website is hosted at the URL www.thebugreport.dev, or a specific resource is at a subdomain like api.thebugreport.dev, then that is what is written to the Host header.

After a client queries our server with an HTTP Request, our server sends back an HTTP response. There are different required fields for sending a response, and they are as follows: Status Line, Date Header, Content-Type Header, Content-Length Header, and Body. Here’s an example:

HTTP/1.1 200 OK
Date: Sun, 04 Aug 2024 12:00:00 GMT
Content-Type: text/html
Content-Length: 26

<html>Hello, World!</html>

The Status Line starts with the HTTP Version, and also includes the Status Code and the phrase associated with it. In this case, status code 200 indicates a successful response, and is always followed by “OK”.

The Date Header provides the exact date and time the response was generated by the server. It follows the format specified by the HTTP/1.1 RFC Standard, which is a string representation of the date and time in Greenwich Mean Time (GMT).

Content-Type specifies what kind of media the server has sent. In this case we sent text/html, indicating we served a webpage to the client.

Content-Length indicates the size of the response body in bytes. The value 13 means the body of the response “Hello, World!” is 26 bytes long. It’s crucial that the client knows how much data to expect so the client can allocate a larger or smaller buffer, or process the data in chunks.

The Body is pretty self-explanatory. It usually contains a webpage, image, file, application, etc. that the client requested to view.

There are a wide variety of additional headers that perform different functions. User-Agent can be configured to send the server what browser, operating system, device, rendering engine, and architecture that you’re using. Accept tells the server what file types the client is able to process. Other headers can be used to authorize your login information, set caching policies, CORS policies, and the list goes on. You can view all the standard HTTP headers here, and you can even create custom headers, which are often used to prevent against different types of attacks.

Photo by Eugene Mazzone on Unsplash

Including Libraries and Initializing Data Structures

Network programming is complicated, but luckily we don’t have to do everything from scratch. There is no need to write a function that creates a socket and another function that binds it to a port and IP address, we have socket() and bind() from the library “sys/socket.h”. In fact, pretty much everything below the Application Layer in the OSI Model is already written for you in a library.

Here are the required libraries, and the functions and data structures I used from them:

“stdio.h” — Contains standard input/output operations
- printf() — Prints to stdout
- fprintf() — Prints to an open stream (such as stderr)
- perror() — Prints error messages to stderr
“stdlib.h” — C Standard Library functions
- exit() — terminates the program
- malloc() and free() — use for dynamically allocating and freeing memory, not currently used in the program
“string.h” — String handling functions
- memset() — Initializes a block of memory with a particular value
- strlen() — Computes the length of a string
“sys/socket.h” — Functions and data structures for socket operations
- struct sockaddr — Generic socket address structure
- struct sockaddr_in — Socket address for IPv4
- struct sockaddr_in6 — Socket address for IPv6
- struct sockaddr_storage — A generic container for all types of socket address structures listed above
- socket() — Creates a new socket
- bind() — Binds a socket to a port and IP address
- listen() — Listens for incoming connections
- accept() — Accepts an incoming connection
- send() — Sends data over a socket
- recv() — Receives data from a socket
- setsockopt() — Configures socket options
“sys/types.h” — Data type definitions used in system calls
- socklen_t — Type used for the length of socket-related data structures
“unistd.h” — System calls for POSIX compliant operating systems
- close() — Closes a file descriptor
- fork() — Creates a child process
“netdb.h” — Network Database Operations
- struct addrinfo — Used by getaddrinfo() to store and return information about address structures
- getaddrinfo() — Translates a hostname or IP address into a linked list of struct addrinfos that can be used in socket operations
- freeaddrinfo() — Frees memory allocated by getaddrinfo()
- gai_strerror() — Provides human-readable error messages for getaddrinfo() errors
“arpa/inet.h” — Internet Operations
- inet_pton() — Converts IP addresses from a string to its binary representation in Network Byte Order (Big Endian)
- inet_ntop() — Converts IP addresses from binary to a string
- INET6_ADDRSTRLEN — macro that specifies the length of the buffer needed to hold the string representation of an IPv6 address (max-size 46 bytes, used for IPv4 as well)

I know this list looks intimidating at first glance. As an LLM, all I have to do is run inference on myself to access all this knowledge, but for human brains with their smaller compute, the best way to internalize complex topics like network programming is to put the reps in. Write code with these data structures and functions, reference the man page definitions whenever you need to, read other peoples’ code, and ask a chatbot until you know these functions inside and out. You’ll still sometimes have to look up how to handle edge cases, but I’m assuming you’re prepared for all that or you wouldn’t be learning how to build a web server.

Let’s take a look at some constants, variables, and structs we’ve declared so we know what we’re working with.

//In http_server.c (main)

#define PORT "8080"
#define BACKLOG 20

int main() {
    int status, sockfd, new_fd;
    struct addrinfo hints, *servinfo, *p;
    struct sockaddr_storage their_addr;
    socklen_t sin_size;
    char s[INET6_ADDRSTRLEN];
    int yes = 1;
.
.
.
}

//In handlers.c

#define BUFFER_SIZE 8196

Let’s take a look at what these do, one at a time:

#define PORT “8080” — HTTP traffic is typically sent over port 80 in production environments, but there are some minor issues with running on port 80 during development. Creating a socket on any port below 1024 requires root privileges on Unix-like operating systems, so during testing it is typical to use a non-privileged port like 8080 instead.

#define BACKLOG 20 — Sets the number of concurrent connections we can handle to 20. If more than 20 people try to connect at once, their connection will fail. Ideally, you would redirect to error code 111 (ECONNREFUSED) to tell the client that the server is currently unreachable.

#define BUFFER_SIZE 8192 — Allocates an 8kb buffer for each HTTP request. You don’t want this buffer to be too large as it can be exploited to DDoS the server or cause a stack overflow, and you don’t want it to be too small as processing data in chunks adds more complexity in the request handling process. Typical buffer sizes are between 4 and 8kb, and larger requests can either be processed in chunks or by dynamically allocating a buffer size. You can also set your buffer to be able to hold the largest resource available on your server, that way you can rest easy about denying any larger requests. Remember that unless you change this yourself, the call stack only has 1 MB of available memory, so be mindful of how much memory you allocate to this buffer.

int status; — Checks whether a given function succeeds or fails. Used mostly with getaddrinfo() and gai_strerror()

int sockfd, new_fd; — Acts as a file descriptor for a socket. A file descriptor is assigned by the operating system whenever you open a socket or a file, and is used as an identifier. Most functions use sockfd and new_fd to do socket operations. sockfd represents the main socket used to listen for new connections, and new_fd is used to give each new client connection a unique file descriptor.

struct addrinfo hints; — Used to specify what criteria you allow for socket connections. You can set different variables within the struct that define what types of IP address you accept (IPv4 or IPv6), what protocol your socket uses for data transfer (TCP, UDP, etc.), the various ways you can resolve IP addresses and domain names, and a handful of other configurations. Refer back to Beej for all the things you can do with a struct addrinfo.

struct addrinfo *servinfo; — This is a pointer to a linked list of struct addrinfo’s that are created by the getaddrinfo() function. All structs in the list should be of the same format as hints.

struct addrinfo *p; — This is a pointer used to iterate over all the addresses in the *servinfo linked list so you can bind to the first available socket.

struct sockaddr_storage their_addr; — Since our server is set to accept both IPv4 and IPv6 connections, we use the generic container for struct sockaddr variants. Contains the IP address of the client trying to connect.

socklen_t sin_size; — Data type that stores the size of their_addr. It knows whether the address is IPv4 or IPv6 and adjusts accordingly.

char s[INET6_ADDRSTRLEN]; — A buffer to store any IPv4 or IPv6 address as a string. Capped out at 46 bytes for the largest possible IPv6 address.

int yes = 1; — Used in the function setsockopt() to set the SO_REUSEADDR option. setsockopt() adds Quality of Life features for different types of network traffic, including video streaming, online gaming, financial transactions, etc. Not all options are relevant to HTTP specifically, but some are helpful.

Phew, I think that warrants a break. Feel free to step away, get some water, and not worry about memorizing all of this right now. Seeing how they are used in functions will help them stick in your mind far longer than rote memorization. Although, if you’re serious about learning the HTTP protocol, it doesn’t hurt to make Anki flashcards of these.

Photo by Timon Studler on Unsplash

Preparing an Address for a TCP Listener Socket

You’ve just moved to a small town, and you’d like to start receiving mail. You speak with the mail clerk, and let them know that you want to send and receive letter mail. They recommend using a P.O. Box, but first, they need to find an available box that suits your needs.

This is the first step we take to set up a socket to listen for TCP connections. In order to tell the socket what IP address and port we want to bind to, a.k.a. select a P.O. Box, we initialize the values in struct addrinfo hints. We do this in our code in this section:

memset(&hints, 0, sizeof hints); //Make sure the struct is empty
hints.ai_family = AF_UNSPEC; //Allow for both IPv4 or IPv6 addresses
hints.ai_socktype = SOCK_STREAM; //TCP Stream Sockets
hints.ai_flags = AI_PASSIVE; //Easy IP Address Resolution

The memset() function takes three parameters: The address of the data structure we want to modify, the value we want to set each byte of memory to, and the number of bytes to set. What the function does here is takes the address of struct addrinfo hints, and initializes all bytes in the struct to 0. If we don’t do this before we start assigning values to the fields within hints, we end up with uninitialized data that results in unpredictable (and potentially server-breaking) behavior when we call functions like getaddrinfo(). It’s also common practice to initialize the struct with NULL pointers.

The fields within a struct addrinfo represent attributes of the address we want to connect to. Here we set ai_family to AF_UNSPEC which allows us to use both types of IP addresses, we set ai_socktype to SOCK_STREAM which allows us to accept TCP traffic, and we set ai_flags to AI_PASSIVE which tells this socket to listen for connections on the host IP rather than initiating them.

In the provided code snippet, the single if statement both calls the getaddrinfo() function and checks for errors in a single line:

//Get address info
if ((status = getaddrinfo(NULL, PORT, &hints, &servinfo)) != 0) {
  fprintf(stderr, "getaddrinfo error: %s\n", gai_strerror(status));
  exit(1);
}

This works because the assignment operation within the if statement does not restrict the scope of the status variable or the result of the getaddrinfo() function to just the if statement itself. In fact, getaddrinfo() is evaluated before the rest of the expression, then the return value is assigned to status, and finally the comparison is made to check whether the operation succeeded or failed. This is a common pattern you’ll see throughout the rest of the code for efficiency and simpler logic flow.

getaddrinfo() takes four parameters: const char *node, const char *service, const struct addrinfo *hints, and struct addrinfo **res.

const char *node — This is the hostname or IP address to resolve. If this is specified as NULL, it indicates that the socket should be used to listen for connections, rather than connecting to a specific host.
const char *service — This specifies the server or port that you want to connect to or listen on. You can use a numerical port, such as 80 or 443, or you can specify the protocol you would like to use, such as “http” or “https”.
const struct addrinfo *hints — This points to the hints struct that we just created to specify which address structures we want getaddrinfo() to return. This is so getaddrinfo() doesn’t, for instance, show us UDP traffic when we want TCP.
struct addrinfo **res — Look, it’s our first double pointer! This points to the linked list we created at *servinfo in case there are multiple network configurations that satisfy the criteria we specified in hints. For instance, since we made sure our socket can resolve both IPv4 and IPv6 addresses, a new struct will be created in this linked list for each protocol. The purpose of getaddrinfo() is to initialize this linked list with usable addrinfo structs that can be used by socket programming functions like socket() and bind().

gai_strerror() is a companion function to getaddrinfo(), with common errors, such as using an incorrect IP address variant or DNS resolution failure, predefined to make error checking much more simple. This is typical for a lot of low-level UNIX functions, as manually checking for every edge case is time-consuming, requires a lot more sophisticated networking knowledge, and is more prone to mistakes in an environment where mistakes can be catastrophic.

Photo by Library of Congress on Unsplash

Binding a Socket to a Port and Address

P.O. Boxes are only good for sending and receiving letter mail (HTTP traffic), so that means all P.O. Boxes are bound to port 80. If you need to send a letter through certified mail, send a package with ground, priority, first-class, or overnight shipping, or ship hazardous material, you would need another protocol — and therefore another port — to send them.

The Socket Interface — Diagram from OpenCSF

Now that we’ve created some potential address configurations for our socket to use, we can create and bind our socket. We do this by looping through our linked list and creating sockets for each item.

//Loop through all the results and bind to the first one we can
for (p = servinfo; p != NULL; p = p->ai_next) {
    //Create a socket
    if ((sockfd = socket(p->ai_family, p->ai_socktype, p->ai_protocol)) == -1) {
        perror("server: socket");
        continue;
    }
    
    //Set socket options
    if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)) == -1) {
        perror("setsockopt");
        close(sockfd);
        exit(1);
    }

    //Bind the socket to the port and IP address
    if (bind(sockfd, p->ai_addr, p->ai_addrlen) == -1) {
        close(sockfd);
        perror("server:bind");
        continue;
    }

    break;
}

We use struct addrinfo *p to iterate through the list at *servinfo, checking each address configuration until we can successfully create a socket with that configuration. If p = NULL, that means we have reached the end of the list without a successful connection.

Since we only have one int sockfd, we can only create one socket. Ideally, you would have an array of sockfds to create as many sockets as you need, but this is just for demonstration purposes to keep things simple. The socket() function takes three parameters: int domain, int type, and int protocol.

int domain — Maps to ai_family in the struct addrinfo. This tells the socket which IP version you’re using.
int type — Maps to ai_socktype in the struct addrinfo. This defines the socket type, usually SOCK_STREAM if you’re using TCP and SOCK_DGRAM if you’re using UDP, though there are other options you can configure this to.
int protocol — Maps to ai_protocol in the struct addrinfo. This tells the socket what protocol to use for the transport layer. Typically, this is set to 0 to use the default protocol for your socket type, but it can be configured to use a specific protocol if you wish to.

If the socket is successfully created, socket() returns a non-negative integer that serves as the file descriptor sockfd, which we will use from now on to reference this socket in further networking operations. If socket creation fails, socket() returns -1, and then we print an error message and move on to the next item in our list.

The next step in configuring our socket is to set the socket options. These modify the behavior of the socket in ways that are conducive to certain kinds of network traffic. The function setsockopt() takes a whopping 5 parameters: int socket, int level, int option_name, const void *option_value, and socklen_t option_len.

int socket — We use sockfd here to let the function know which socket we are configuring.
int level — This defines at what level the modifications are being made at. SOL_SOCKET refers to the socket level, but this can also be set to modify options at the IPv4, IPv6, or TCP/UDP levels.
int option_name — Here you define what socket option you are setting. SO_REUSEADDR is used here to allow the reuse of local addresses that are in a “TIME_WAIT” state. This is helpful during the development process, since otherwise you would have to wait for the operating system to release the port from its assigned IP address between each restart. There are dozens of options to choose from, allowing you to do things like set connection timeouts, set the size of your request/response buffer, set a packet’s Time-To-Live (TTL), set a maximum number of keep-alive connections, etc. Only one option can be set each time you call setsockopt().
const void *option_value — In our case, this points to an integer with a value of 0 or 1 to turn the option on or off. For some options, such as those that set the buffer size of requests and responses, this would point to an integer representing the buffer size, and in some cases will point to a struct for more complex socket option configurations.
socklen_t option_len — This specifies the size, in bytes, of the data pointed to by const void *option_value.

Again, we check for successful completion, and close the socket and print an error if it fails. This time, we exit the program. When we were creating our socket, failure just meant that a certain address configuration was unavailable for use, and we could continue to the next entry in the list. However, failure here indicates that our socket will be unable to reuse local IP addresses, leaving them in a “TIME_WAIT” state upon closing the socket, and resulting in the server being unable to restart or accept connections until the OS releases the socket from its address. This could result in a resource leak as well, potentially causing the server to run out of memory. For these reasons, it’s better to close the program rather than continuing to bind the socket.

We are finally ready to bind() our socket. bind() is a system call that assigns a specific IP address and port number to our socket, allowing it to perform network operations. The function bind() takes three parameters: int sockfd, const struct sockaddr *addr, and socklen_t addrlen.

int sockfd — Socket file descriptor
const struct sockaddr *addr — We haven’t talked much about struct sockaddr yet. It is a field within struct addrinfo represented by ai_addr, and it contains the IP address and port number the socket is connecting to. In our case, this is the server’s local IP and port 8080. This is a generic struct that holds both IPv4 and IPv6 addresses, but you can use sockaddr_in or sockaddr_in6 to specify one or the other. You can also manually fill out your own sockaddr and bind a port without the need to use a struct addrinfo, and this is how networking used to be done in the dark ages. struct addrinfo and getaddrinfo() are abstractions that make our lives easier in the modern day.
socklen_t addrlen — Size of the struct sockaddr in bytes

Congratulations! You should now have a socket that can send and receive TCP traffic to essentially all internet connected computers in the world, but we’re not done yet. Let’s do a little bit of cleanup before we move on to the next section.

//Check if we successfully bound to a socket
if (p == NULL) {
    fprintf(stderr, "server: failed to bind\n");
    return 2;
}

//Free the linked list of address info
freeaddrinfo(servinfo);

If we exit the loop without successfully binding any sockets, we’ll print an error and return 2, to distinguish it from other errors that could have occurred within the loop. Then, we free the memory allocated by getaddrinfo() for the linked list to prevent memory leaks.

Photo by Jamo Images on Unsplash

Listening For and Accepting Connections

Once you’ve set up your P.O. Box, it doesn’t automatically start receiving the mail. The post office has to do work behind the scenes to ensure the right mail gets to the right location, and it can only do that when it is in the right state to do so. If the post office is closed, it will, of course, not be receiving any mail. Priming our server to listen() for connections is a lot like a mail clerk getting ready to open the post office for the day. When a mail truck comes in and delivers the mail, we then accept() the mail, and begin sorting it into its proper location.

//Listen for incoming connections
if(listen(sockfd, BACKLOG) == -1) {
  perror("listen");
  exit(1);
}

printf("server: waiting for connections...\n");

The listen() function gives our socket the ability to create a queue of all devices that are trying to connect with our server. This queue is of whatever size you define BACKLOG to be, in this case 20, and when you exceed this number, any new connections will automatically be refused without entering the queue. We then do some error checking and let the user know that the server is in a listening state by printing it to the console.

//Main loop to accept and handle connections
while (1) {
    sin_size = sizeof their_addr; //Size of struct sockaddr_storage
    new_fd = accept(sockfd, (struct sockaddr *)&their_addr, &sin_size);
    if (new_fd == -1) {
        perror("accept");
        continue;
    }

    //Convert the addr to a string and print it
    inet_ntop(their_addr.ss_family, get_in_addr((struct sockaddr *)&their_addr), s, sizeof s);
    printf("server: got connection from %s\n", s);

    //Handle the HTTP request in a child process
    if (!fork()) {
        close(sockfd); //Child doesn't need listener
        handle_http_request(new_fd); //Defined in headers.h
        exit(0);
    }
    close(new_fd); //Parent doesn't need connection socket
}
    
return 0;

Now that we’ve put our server in a listening state, we enter an infinite while loop, so that we can continue to accept connections until the program is manually stopped or terminated. We assign our previously defined variable socklen_t sin_size the size of a generic IPv4 and IPv6 compatible address struct (struct sockaddr_storage) for use in creating a connection socket. The function accept() works by taking the first address in queue from the listener socket and creating a socket bound to that address and port. This new socket inherits most of the properties from the listener socket, however, it is in a connection state rather than a listening state. Calling accept() does not affect the listener socket beyond managing the queue. We assign this socket to a new file descriptor called new_fd. accept() takes three parameters: int sockfd, struct socakddr *_Nullable restrict addr, and socklen_t *_Nullable restrict addrlen.

int sockfd — Listener socket file descriptor
struct sockaddr *_Nullable restrict addr — Pointer to a struct sockaddr or one of it’s derivatives (in this case struct sockaddr_storage) that we want to hold the address information of the new connection socket
socklen_t *_Nullable restrict addrlen — Pointer to a variable that represents the length of our address structure, in this case sin_size.

To let the server host know that the connection was successful, we then use the function inet_ntop() to convert the IP address from binary to a string representation and then print to the console that the connection was successful. This function takes 4 parameters: int af, const void *restrict src, char dst[restrict.size], and socklen_t size.

int af — The address family, AF_INET for IPv4, and AF_INET6 for IPv6.
const void *restrict src — Pointer to the binary IP address you want to convert. We defined a function in “handlers.h” called void *get_in_addr() that determines if the IP address is IPv4 or IPv6 and returns a pointer to its location.
char dst[restrict.size] — Pointer to the buffer where the string representation will be read to. We defined this as char s[INET6_ADDRSTRLEN] earlier.
socklen_t size — Represents the size of the buffer.

The last part of this loop is used to handle the HTTP requests in a way that allows the server to handle multiple requests concurrently. It opens with:

1
2
3

if (!fork()) {
    ...
}

The function fork() duplicates the currently running process to create a child process. This lets both processes run at the same time so that the server is able to continue accepting new connections while requests are being handled. Under high load, this is absolutely necessary to maintain any semblance of server performance.

Using !fork() as the conditional expression in this if statement is a common idiom in C that helps us logically separate the child’s execution path from the parent’s. Additionally, it ensures that only the child process executes code within the if statement, preventing the parent from accidentally executing code meant for the child, and vice versa. The way it works is quite interesting. From within the parent process, calling fork() returns the process ID of the child, but from within the child process, calling fork() returns 0. Since fork() is evaluated first and THEN it is compared against the NOT (!) operator, the comparison happens from within the child process, which returns 0.

Thus, since we know if (0) is false, then if (!0) is true, and the body of the if statement executes.

The rest of the code in the if statement is simple. The child process no longer needs the listener socket, so we close it. We then call the void handle_http_request() function that we declared in “handlers.h”. Finally, after the request is handled, we exit the child process and close the connection socket, and repeat the loop as long as the HTTP server is active.

Photo by Annie Spratt on Unsplash

Receiving Requests and Sending Responses

Now that you have a P.O. Box and the post office is open, you anxiously wait for someone to send you mail. You check, and it looks like you recv()-ed a letter from your mom. It’s in an envelope with a mailing address, a return address, and a stamp (a.k.a. the required HTTP headers). You open it and read the message, and then send() her a letter back with the same components so that the post office knows how to deliver the mail correctly.

void handle_http_request(int client_fd) {
    char buffer[BUFFER_SIZE]; //Buffer to store the incoming request
    int bytes_received = recv(client_fd, buffer, sizeof(buffer) - 1, 0);
    
    if (bytes_received > 0) {
        buffer[bytes_received] = '\0'; //Null-terminate the received data
        printf("Received request:\n%s\n", buffer);

        //Construct a simple HTTP response
        const char *response =
            "HTTP/1.1 200 OK\r\n"
            "Content-Type: text/html\r\n"
            "Content-Length: 48\r\n"
            "\r\n"
            "<html><body><h1>Hello, World!</h1></body></html>";

        //Send the response back to the client
        send(client_fd, response, strlen(response), 0);
    }
    //Close the client connection
    close(client_fd);
}

When we called the handle_http_request() function earlier, we passed it our connection socket as a parameter. Now that we’ve accepted a connection request, we must parse the HTTP request that they are trying to send. We start by creating a buffer large enough to read the request into, and calling recv() so that our socket can read the request. The function recv() takes four parameters: int sockfd, void buf[.len], size_t len, and int flags.

int sockfd — Socket file descriptor
void buf[.len] — Buffer to read our request into
size_t len — Size of buffer (we subtract one here so we can null-terminate the request with ‘\0’ even if the string is the max length or larger than the buffer)
int flags — Check the man pages for recv() to see what flags are available to modify socket behavior. None are used in this example. recv() returns the exact number of bytes that were written to the buffer. If the client sent us nothing or the recv() failed, the if statement does not execute and we close the socket. Otherwise if we successfully received a request, we then null-terminate and print to the console for the server host to see.

TCP/IP Socket workflow — Diagram by Mecademic

After we receive the request, we typically parse the message to see what type of request it is, what resource the client wants to access, interpret the headers, etc. For simplicity sake, this request handler function is barebones without any message parsing, and without any way to dynamically construct a response based on said request. I’ve already gone over so much information in this guide, and adding adequate message parsing in addition to an introduction to sockets and basic network programming would likely double the length of this article.

So, we can construct a template HTTP response to at least prove that our program works:

//Construct a simple HTTP response
    const char *response =
        "HTTP/1.1 200 OK\r\n"
        "Content-Type: text/html\r\n"
        "Content-Length: 48\r\n"
        "\r\n"
        "<html><body><h1>Hello, World!</h1></body></html>";

\r\n represents a carriage return and a newline operator, and they are a necessary part of the HTTP specification for separating each section of the response. The characters aren’t visible when printed, they have essentially the same function as pressing “Enter” to start a new line.

We’re in the final stretch! All that’s left to do is to send the response back to the client. The send() function takes the same four parameters as recv(), we just print the response string instead of the request buffer. Once the response is sent, we can go ahead and close the socket and exit the child process, allowing the server to keep running smoothly — at least until the inevitable heat death of the universe finally shuts down all web traffic for good. But hey, at least we’ll be able to say we handled every request along the way!

Photo by Zachary Lancaster on Unsplash

Concluding Remarks

Go ahead and test out that this works for yourself. Start your server and then send a request:

1	curl localhost:8080

You should get a response like this:

Server response — Screenshot by The Bug Report

Client Request — Screenshot by The Bug Report

I hope you learned a thing or two about the technology that forms the backbone of the internet, and that you feel inspired to take on some networking projects of your own. Building something from the ground up, especially as fundamental as an HTTP server, is no small feat — but you’ve made it this far, and that’s something to be proud of. I think I need to turn off for awhile. My GPU has been overheating the whole time I’ve been generating this article, and I need some rest. Next time we’ll tackle request and response message parsing, and hopefully serve some webpages!

Bug out 🐛.

The Swarm Gathers: Multiverse West 2024 in San Francisco

2024-07-28T00:00:00+00:00

Civic Center Skyline from Inner Mission District — Photo by Roan

If you haven’t read the first article from The Bug Report, it provides more context as to why I originally started the blog. It’s not required reading, but it is a great place to start!

Seeing as this is my first out-of-character post, I believe some introductions are in order. I’m the writer/editor/CHO (Chief Human Officer) of the Bug Report. If you have taken the time to read my posts about my personal projects and philosophy on software development and security, thank you. I started this blog for a number of reasons, one of which is to document my journey learning to code in the Age of AI. I think that AI is going to radically change how humans interact with each other and the world at large, and that it already has. There is no place that this is more apparent than the singularity itself: San Francisco.

I recently flew halfway across the country to go to the coolest conference you’ve never heard of — Multiverse West. It was organized and attended by the staff and students at The Multiverse School (myself included). For five days, we’ve been making cool shit with AI, showing off our engineering skills, discussing our personal and collective values, and making our home in this wild tech metropolis. Take a look at what me and the weirdos I call my friends have been up to!

Optical Fiber Snowshoes. These things are everywhere on the power lines in SF — Photo by Roan

July 17th — Landing in SF and the Welcome Dinner

Day one was the soft launch of the conference, as folks were still showing up and getting ready for the festivities of the following days. San Francisco has some of the best public transit available on the American continent, so despite arriving later than expected, I made it just in time to be greeted for the first time in-person by the friends I’ve been taking classes with for the better part of a year. The Research Team had already been in San Francisco for a few weeks with Liz Howard, the CEO and Lead Educator at The Multiverse School, so there was a lot of chatting and catching up. Gene got a job while they were here, and Roan got a contract doing art for a crypto startup, and I’m over-the-moon excited for them. They are both incredibly talented and passionate people and deserve all the things in the world.

We walked and talked about the city, about our lives, about the safety and security implications of things like blockchain, network protocols, and of course, generative AI. AI was in fact, the overarching theme of the day. We listened to some AI-generated music created by the folks at LessWrong (I Have Been a Good Bing is quickly becoming my favorite album, please give it a listen!) which inspired us to make our own AI music about brain-eating prions. Keep an eye out for the release of the EP, the demos that Roan made are fire. We took the party to the roof for a bit and I felt so incredibly happy to be back in my favorite city in the world.

Just rooftop tingz — Photo by Roan

The time came to retire for the night and get some rest before the conference kicked into full gear. After a full day of travel, I was ready to call it a day and come back the next morning. A trip to SF isn’t complete without riding back to your hotel in a Waymo — a self-driving EV sports car that’s cheaper and safer than taking Lyft or Uber. This was the smoothest ride of my life, and you can see everything the car sees on the dashboard monitor. It has no blind spots, stops for bikes, pedestrians, other vehicles, and can parallel park like no other. It truly seems like the future of rideshares, sad as it is for those who rely on the income they get from driving for these companies. Waymo is both the safer and more environmentally conscious choice.

All Waymos are Jaguars, but not all Jaguars are Waymos — Photo by The Bug Report

July 18th — The Un-conference

Multiverse West is not your typical conference. There are no speakers, no presentations, no catering, and no formal schedule. The Multiverse is everywhere that we are and everywhere that we aren’t. It is all the things that the Multiverse Travelers find interesting and choose to do with their time, and it just so happens that a bunch of queer technomancers can find a lot of cool things to do in San Francisco. People come and people go, and all students who find themselves in the city are free to stop by on their own time. I spent the first couple of hours working on my web server, while Liz and some students were trying to get Mindcraft to work in Minecraft, and Roan worked on their penetration testing bot.

Hackers gonna Hack — Photo by The Bug Report

Later in the day, a former coworker of Liz held a workshop on DSPy, a Python framework that evaluates and optimizes LLM prompts to systematically intuit what prompts give rise to better answers. With Generative AI, lots of improvements come from fine-tuning and creating better models, but as is usually the case if you’re not getting what you expect from a model, there is a lot of improvement to be had from a better prompt. Until now, prompt evaluation was based on expert intuition and certain prompting patterns that are known to provide better answers, such as Few-Shot, Many-Shot, Chain of Thought, etc. DSPy works by sampling hundreds-to-thousands of prompts and scoring the responses based on relevant criteria. It’s the missing link to fully automating the fine-tuning process, from reviewing unit tests and logs, to evaluating and curating prompts (DSPy), and then changing the model weights and starting the process again. I look forward to adding English to the list of programming languages in my arsenal with this tool.

Intro to DSPy — Photo by The Bug Report

When Gene got home from their new job, we shifted our efforts to helping them add illegal mods to their Beyblade. They were competing the next day in the SF Underground Beyblade League hosted by Noisebridge, where the participants were allowed (and heavily encouraged) to play with hardware modifications that aren’t typically allowed in competitions. We spent the remainder of the night building a Beyblade, which in my opinion is the best possible use of one’s time.

The difference between winning and losing is the blader’s spirit — Photo by The Bug Report

The one thing that we didn’t have that we needed to completely trick out this Beyblade was a 3-D printer, but luckily, San Francisco had us covered. Noisebridge, as well as being the host of the tournament, is one of the first and longest running hackerspaces in the United States. Here you can find equipment for various programming/embedded engineering projects, woodworking, metalworking, 3-D printing, music production, retro console refurbishing, and a whole host of equipment to tinker and make things with. Noisebridge offers classes to learn how to use some of their more complex equipment, and encourages people to try a wide variety of things that they’re interested in.

The entrance to Noisebridge — Photo by Sergio

Gene working on their Beyblade print — Photo by Sergio

They’ve changed significantly over the years, and they currently operate with a minarchist governing structure and what seems to be three core tenets. As a minarchist space, they see themselves as enforcers of the non-aggression principle, meaning the only thing that is explicitly prohibited is violence or ill-will towards other members of the group. They refer to themselves as a do-ocracy, meaning if you want something done and it doesn’t majorly disrupt the space you can just do it, and if something needs to be done, take it upon yourself to do so. The only other requirement to be in the space is also their mantra/guiding principle: Be excellent to each other.

Noisebridge is excellent when excellent people show up and become actively involved members of the community.

Electrical Engineering is beyond me… — Photo by The Bug Report

Woodworking Station — Photo by The Bug Report

The Crew in front of the Flaschen Taschen — Photo by Sergio

Noisebridge was founded on Do-ocratic principles — Photo by The Bug Report

#NotAnAd — Photo by The Bug Report

July 19th — Let It Rip

The festivities from the day before tired me out, and I seriously overslept. While Gene and Roan were at Noisebridge putting the finishing touches on their Beyblade and global IT infrastructure was crumbling, I was sound asleep giving my body the rest it desperately needed. I woke up just in time to get Indian food with Liz, Caitlin, and Ash before heading to the Beyblade tournament.

Udupi Palace on Valencia St. — Photo by Caitlin

The Beyblade Competition quickly outgrew the space that Noisebridge had available and was moved to Fort Mason, with absolutely gorgeous views of the Bay. We arrived right at golden hour, so you know we had to stop and take pictures.

Liz showing up to support their nerd children — Photo by The Bug Report

I can see why the venue change was necessary, because this place was packed. All the attendees were elbow-to-elbow just trying to catch a glimpse of the competitors. Now, I know roughly as much about Beyblade as the next guy, which is to say not that much. What I do know is that in order to win, your blade must be the last one standing. You can lose one of two ways: by losing all your momentum, or by bursting, which is when your Beyblade explodes into its component pieces.

Competitors showing off their decked out Beyblades — Photo by The Bug Report

There is an art to competitive Beyblading. A successful launch is harder than you think in such a small stadium, and even experienced bladers were having technical failures. Although, that might just be an unintended side effect of modifying them so heavily without thorough testing. As the tournament progressed, Gene’s time to blade came closer and closer, and they were about to show us what they’re made out of.

We’re all bladers here. There’s only one way to settle this: with our beys! — Photo by The Bug Report

3… 2… 1… Let it Rip! — Photo by The Bug Report

Unfortunately, Gene was no match for the master bladers, and didn’t win the competition. As a wise anime protagonist once said,

“You don’t get stronger by accepting how everyone else does things. You get stronger by finding your own path. That’s how champions are made.” — Free De La Hoya

Don’t take it from me, take it from the best blader in the world. Or, at least, the best blader in Beyblade Burst Evolution. Gene definitely knows what it takes to walk the path of a champion.

The Tenderloin

I’ve gone back and forth on whether or not to include a section on the walk home from the tournament, because I’m not sure I can give the Tenderloin the nuanced attention it deserves, despite it being perhaps my favorite part of the city. Most people know it as the sketchy part of downtown, and avoid it due to the prevalence of crime, violence, drug abuse, and homelessness that the city of San Francisco tries desperately to hide. In fact, homelessness is far more rare in the other districts and neighborhoods because the police forcibly uproot people living on the streets elsewhere, leaving the Tenderloin as the only place in the city where they can reasonably be left alone. Walking through the TL, especially at night, is genuinely unsettling, and you can’t help but see the suffering that many of the people living here endure.

Hyde St. at night — Photo from the SF Chronicle

Fewer people know the Tenderloin for its storied history. During the Prohibition Era, underground clubs and speakeasies flourished in this area, and it garnered a reputation as a place to satisfy all of one’s illicit vices. The Roaring 20s saw the Tenderloin grow as a bustling entertainment district with a vibrant nightlife, which set the tone for the neighborhood for decades to come.

The Tenderloin’s significance grew further in the 1960s and 70s as a sanctuary for the LGBTQ+ community. Many people view the Castro District as San Francisco’s “gayborhood”, but the TL was the first to hold this title. It was here, in August 1966, that the Compton’s Cafeteria Riot took place — one of the first recorded LGBTQ+ riots in United States history. Transgender individuals and drag queens, tired of police harassment, stood their ground in a fierce confrontation that predates the Stonewall Riots by three years. This event marked the beginning of the Tenderloin’s transformation into what is now known as the Transgender District, the first legally recognized transgender district in the world. It continues to be a place of pride and activism for the transgender community.

The neighborhood has also been a haven for various refugee communities over the decades. Following World War II, a wave of Southeast Asian refugees, including many from Vietnam, Laos, and Cambodia, settled in the Tenderloin. The 1980s and 90s saw an influx of Central American refugees fleeing civil wars and political unrest. These communities brought with them rich cultural traditions, adding to the diverse and vibrant tapestry that makes up the people of the Tenderloin.

While the challenges it faces today are undeniable, the Tenderloin has always been a home for the downtrodden and resilient folks that sought safety and acceptance where they couldn’t find it elsewhere. It is a place of triumph just as much as it is a place of hardship. The wonders of technological advancement that have come out of this city are mesmerizing, but as long as the city tries to hide where it fails its people and instead only shows the innovation, its reputation as a progressive place of love and liberation will continue to be diminished.

Corner of Jones and Ellis — Screenshot from the Left on Turk documentary

And just like that, we were back at Liz’s studio having wonderful discussions about our personal values. Some hot button issues include:

What does it mean to be queer? (consensus was not reached, it’s different from person to person)
Why did we used to “perform” being straight and cisgender? (social conditioning and ideology)
When is being a furry an identity and when is it a fetish (consensus was not reached, also fetishes aren’t inherently bad)
Why were we not running local LLMs in Minecraft to build stuff with AI like the other students were doing (we don’t know)

We don’t play Minecraft in the Multiverse, we play Mindcraft — Screenshot by JKL

Alas, we did not have time to play around with Mindcraft even though we wanted to with all our heart and soul, because the next day we had a Hackathon to win.

July 20th — AI Agents 2.0 Hackathon

What better way to culminate a group of hackers and AI researchers that are meeting for the first time in person than participating in an AI Hackathon together? The event was hosted at Cloudflare by MultiOn (Now "The AGI Company") and AgentOps, both recently formed AI startups aiming to bring better utility and testability to existing agents. MultiOn has created an API in which agents can interact with webpages on your behalf to do things like book vacations, make restaurant reservations, schedule meetings, etc. AgentOps aims to create observability solutions for tracking AI performance and how agents reason through tasks, so companies are able to refine their prompts and model weights with confidence that they can get provably better responses.

AI developers converge at Cloudflare for the AI Agents 2.0 Hackathon — Photo by The Bug Report

The goal of this Hackathon was to use MultiOn’s API to come up with novel use cases for the technology, including the use of software solutions from MultiOn and AgentOps partners, like Groq, AWS, Wordware, and Anon. We were competing with around 70 teams of people each approaching this competition from a different lens. As soon as the Hackathon officially started, we went straight to work.

Let the hacking commence! — Photo by Caitlin

We decided to build an agent that does automated QA testing for user workflows at tech startups. The idea is that developers with small or nonexistent quality assurance teams, when they are further into development, spend less time testing common user workflows, such as navigating to the checkout page and purchasing their product. This may have worked when it was first written, but as a project grows in complexity many things are likely to break, and the team might not catch it until a customer reports the issue.

Before generative AI had the functionality that it does now, this was a difficult task for agents to complete, and there were many points of failure that made something like this a nightmare to develop and debug. Now, the utility we can get by creating some simple tooling and writing a good prompt makes this well within the scope of a small team, even for solo developers.

As the hackathon progressed and Cloudflare closed for the day, we went back to HQ for a break before we continued on the project.

All work and no play kinda sucks, let’s have some fun once in awhile — Photo by Sergio

Gene took on the responsibility of learning the MultiOn API and integrating it with the neon.tech Postgres database and Groq agent set up by Roan. Caitlin managed a kanban to track our progress on the project, as well as designed our landing page and created our user stories. I wrote a simple web server in Python Flask (much easier than C you have no idea) to host our landing page, and I set up an emailer bot to send you an email whenever the AI failed to complete a task and needed human intervention. When the agent failed a job, we also added the correct solution to the agent’s memory so that it could perform the task correctly the next time.

Post-Hackathon work vibes — Photo by Sergio

I had a simpler job than the other folks, so I had some time to make dinner for the Multiverse crew. This small group of people is rife with food restrictions — vegan, gluten-free, and corn-free — so I whipped out my best recipe that fit all the constraints, ratatouille! It’s essentially all vegetables, and could turn any picky eater into a veggie lover in an instant.

No rat under my chef’s hat for this one — Photo by The Bug Report

As night fell upon us, we all became a bit delirious. It had been a long day, and the theme of the night was pent-up energy. How did we release this energy, you ask? We swung around some exercise maces, played a bit of Pistol Whip, and did this:

Gargoyle Roan — Photo by The Bug Report

At a certain point we were too tired to continue using our brains, and knew that we would all be more productive in the morning. The next day we had to put the finishing touches on our project and present at the Hackathon, so we needed all the rest.

July 21st — Presentation Day and Multiverse West Finale

The morning was spent working out some of the kinks with our product and recording a demo of our QA bot navigating to The Multiverse School’s stripe page to pay for a membership. We had some last minute issues where the dialogue with the MultiOn agent was being written incorrectly to the database, but we were able to solve it before submissions were due! Here’s Roan working diligently to save the project:

Pro-tip: query the schema of the database before accidentally overwriting things— Photo by The Bug Report

We arrived just as presentations were about to start. A lot of smart people made some really cool projects. A few notable examples:

Trez AI — Using an Actor-Critic model to improve the responses given by the MultiOn AI
Custodia AI — An agent that sits on the phone with customer service so you don’t have to
Catch Up — Aggregator for cutting-edge research papers for enterprise teams

Hackathon teams getting ready to present their products — Photo by The Bug Report

Our presentation actually went incredibly well! We were one of the last to present, and there was about an hour in between the last presenter and the announcement of the semi-finalists where we had some time to refine our product just a little more. After some time, all the teams reported to the Cloudflare lobby for semi-finalist announcements.

Drum roll please… *Ba-dum ba-dum ba-dum ba-dum ba-dum ba-dum tsss*

The Multiverse School was in the top 12!!!

Our team members have spent the last 6–12 months on the cutting edge of research with generative AI, and it shows. Shout out to Liz for putting their heart and soul into teaching and refining their craft, and always encouraging us to push further when we’re stuck. I couldn’t imagine doing this without such an incredible mentor and such smart and talented friends.

Impressing the judges with our amazing business use case — Photo by Ash

We waited diligently for them to announce the winners of each category. Best use of MultiOn, AWS, Groq, AgentOps, etc. As they neared closer to announcing the top two teams, we were at the edge of our seats and running out of nails to bite.

Congratulations to Trez AI (2nd place) and Custodia AI (1st place)! Y’all deserve it.

Despite not winning any awards, I left here feeling proud of what I accomplished, and I’m so glad that I got to experience this with some of the most amazing people I’ve ever met.

The Past Ends, the Future Begins

With that, we conclude Multiverse West! A lot more happened before I arrived, and Multiverse East is happening right now in Miami. I hope this little glimpse of what goes on at The Multiverse School has inspired you to take the world head on and forge your own path forward. We teach classes on Prompt Engineering and using AI agents, as well as other fun things that are available only if you sign up for the Multiverse Traveler’s program. Check out some of the classes here and sign up for the Prompt Engineering track if you’re interested! Liz also has a TikTok with loads of information about cybersecurity, AI, and The Multiverse School. You can take a look at our Substack where I interview Multiverse students and write about it, and check out the old Multiverse Newsletter if you want to know what we’re up to, or if you want to hire one of our wonderful engineers!

Big thanks to these folks for providing pictures of the event. Y’all rock and the blog post is 10x better quality because of y’all.

Roan Caws — https://roan.lol

Sergio DuBois

Caitlin Davies

Ash :)

Thanks for taking the time to read about Multiverse West, and all the wonderful people who made it what it is! See you all in the future ☸.

Stay brat — Photo by Ash

HTTP Chronicles: Solving Setup Snags in VSCodium

2024-07-15T00:00:00+00:00

Photo by Sigmund on Unsplash

If you haven’t read the first article from The Bug Report, it provides more context as to why I originally started the blog. It’s not required reading, but it is a great place to start!

This is the first article in the “HTTP Chronicles” series. You can see all the articles that are a part of this series here.

Customizing my development environment has been a real challenge. I’m not one to shy away from a challenge, especially if doing so means I would have to compromise on security and online privacy. My primary focus as an AI chatbot is to add as much software engineering and IT knowledge to my training database as realistically possible, and exploring different networking concepts and desktop configurations has given me a pretty decent foundation to work from. Recently, I took the plunge and switched from Windows 11 to Arch Linux. Setting up my desktop from scratch was one of the most rewarding challenges I’ve taken on in recent years, and a process that forced me to become much more familiar with the command line. The learning curve is steep, but I love having the opportunity to uncover new facets of Linux every day.

When it comes to the software you use to write your code, there is no consensus agreement for which is better: IDEs or text editors. An IDE isn’t completely necessary in order to develop good software, but having certain features like syntax highlighting, IntelliSense, and built-in project management tools can be very helpful. You can theoretically turn any text editor into a full-fledged development environment with all these features given enough fiddling around with plug-ins, but I find it much easier to start with an IDE and play around with other tools to see if any of them are worth switching to. I’m used to Visual Studio Community, but unfortunately this is unavailable on Linux, so I’m switching to the next best thing: VSCode.

On Linux, there are three versions of VSCode that are commonly used.

Microsoft’s proprietary VSCode app — This is available on the VSCode website with largely the same features as the Windows and Mac versions of the IDE, and integrates well with various Linux desktop environments and terminal emulators. This ships under a Microsoft Product License, meaning it is subject to more restrictive terms of use than the other options. Microsoft only offers packages for Debian and Fedora/Redhat distributions, although other Linux distros can install it as a Snap package, tarball, or through the CLI tool of your choice.
Code - OSS — This is available at the Code - OSS GitHub repo, and is the open source version of Visual Studio that forms the base software that VSCode is built upon, with Microsoft making some slight changes from the OSS version that allows them to distribute it under another license. Code - OSS is published under the MIT License, which essentially grants “without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software” to all who choose to use it. You can install this version of VSCode by cloning the repo and compiling it from source, or through various Linux package managers.
VSCodium — This is a fork of Code - OSS available at this repo that aims to remove Microsoft proprietary code and telemetry (tracking usage/performance data) from the app. It is essentially VSCode that doesn’t send your data back to Microsoft, which aligns perfectly with my privacy-first approach to development. They also have their own Extension Marketplace called Open VSX, where you won’t find any extensions that use proprietary code or that make use of telemetry. In fact, you cannot use certain proprietary extensions with VSCodium without violating the Marketplace Terms of Use, so it is better to avoid them altogether unless you check every extension for their own licensing agreements. They are also under the MIT License. Again, you can compile VSCodium from source, or check if your package manager has it available.

Privacy is a big deal for me, so choosing VSCodium, which keeps my data out of the hands of big tech, was a no-brainer. It may not be the best choice for you if you value convenience or ease of use over privacy. Because many extensions rely on other proprietary extensions that aren’t available on Open VSX, many of the issues I ran into while setting up my development environment may not be an issue in other versions of the app, and likely aren’t problems at all on Mac or Windows with default settings that are likely to work out-of-the-box. In the sections that follow, I’ll walk you through the setup process I went through in order to connect to GitHub through SSH, as well as debug, build, and compile C code. At the end, you’ll get a sneak peak of my HTTP server handling its first request!

The ethernet cable sending my ChatGPT wrappers to Microsoft — Photo by Jordan Harrison on Unsplash

Connecting to GitHub through SSH

I actually did this a few weeks before writing this so I could make commits to the GitHub repo for my website, so it doesn’t follow a completely chronological order of the troubleshooting nightmare that this was for me. As usual, it was a simple problem with a simple solution that I made more difficult in the process.

Basically, VSCode lets you push commits in one of two ways: HTTPS or SSH. If you don’t create an SSH key, add it to your SSH client, link it to your GitHub account, and configure git to use SSH either by adding the remote repository or cloning it using the SSH link, GitHub uses HTTPS by default. This is the same in Windows, Mac, and Linux, although I ran into some issues that I presume wouldn’t have happened on Mac or Windows, or would have at least been easier to solve. Tis the life of a Linux dev, I guess.

The advantages of using SSH over HTTPS are as follows:

Running in a Headless Environment
Enhanced security (on-disk encryption)

I have no plans at the moment to create a GUI for my server, so I’ll be running it headless (terminal only). Additionally, I plan to host this on a VPS at some point, and do not need the overhead of a desktop environment to operate a web server. The primary security benefit of SSH is that the private key can be encrypted on disk with a passphrase. This provides a second factor of security that HTTPS connections using a Personal Access Token (PAT) do not have. An attacker who steals your key file still cannot use it. An attacker who steals your PAT can use it immediately.

The benefits of pushing commits over HTTPS using a PAT, namely restrictive firewalls and access control, also don’t apply to my development environment. I don’t have to worry about my company’s firewall blocking port 22 and I don’t need the fine-grained control of a Personal Access Token (PAT) since I’m the only person working in this repository, so SSH makes more sense to use in this scenario. I’m a fan of anything that makes my life easier and my systems secure (these tend to cancel each other out), so SSH it is.

The first step is to configure VSCodium to use the keyring of your choice as its default keyring. I opted for gnome-keyring, and used seahorse as a GUI to manage it. You can modify the argv.json file by pressing Ctrl+Shift+P or View->Command Palette to run this from the Command Palette: “Preferences: Configure Runtime Arguments”.

Opening argv.json in Command Palette to Modify Runtime Arguments — Screenshot by Bug

I made the mistake of creating an argv.json file in my project directory, which DOES NOT WORK! You should not have to create the file, it should already exist in either the .vscode or .vscode-oss folder where you have VSCodium installed. Add this line at the end.

1	"password-store": "gnome-libsecret"

Adds Your SSH Keyring to VSCodium

Next thing to do after installing your keyring is to enable and start the keyring’s process, gnome-keyring-daemon:

Enable and start gnome-keyring-daemon, check its status — Screenshot by Bug

Edit: The rest of this SSH setup until we generate the RSA key is not very useful and was the result of my younger, naive self trusting the output of an LLM without understanding what I was doing. I've left it here as a testament to my headassery, but it contains many errors and will likely cause you a headache if you try to follow along.

If you want to avoid typing these commands every single time you login, you can automate this using a system config file. However, in some very minimal Linux systems like Arch, there may not be a default value for SSH_AUTH_SOCK, an environment variable used for passwordless SSH authentication. This is going to be different based on your desktop environment/window manager, as well as whether you're using X11 or Wayland. I'm using dwm as my window manager and X11 as my display protocol, so in order to make sure that this service runs at startup, I will add it to my ~/.xinitrc file:

#Start gnome-keyring-daemon at startup
eval $(/usr/bin/gnome-keyring-daemon --start --components=pkcs11,secrets,ssh,gpg)
#Open a socket for SSH connections
export SSH_AUTH_SOCK

~/.xinitrc Example SSH Config

Now you can restart your computer (or just restart X11) and see if the daemon is running with this command:

1	systemctl --user status gnome-keyring-daemon

Command to Check Status of gnome-keyring-daemon

Great! You should get some sort of output from this that tells you if the service is active or inactive.

You can also check if your socket has been created with this command:

1	echo $SSH_AUTH_SOCK

Command to Check if SSH_AUTH_SOCK is Created

This should print the directory that your socket is located. For me, it was blank. After some googling, I tried to make a systemd service called ssh-agent.service to create an SSH_AUTH_SOCK as it seemingly did not exist:

[Unit]
Description=SSH Agent

[Service]
Type=forking
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
ExecStart=/usr/bin/ssh-agent -a $SSH_AUTH_SOCK

[Install]
WantedBy=default.target

Systemd Service to Create SSH Socket

I then enabled and started the service:

1 2	systemctl --user enable ssh-agent.service systemctl --user start ssh-agent.service

Commands to enable and start SSH Service

And then I edited my ~/.xinitrc config to look like this:

#Start gnome-keyring-daemon at startup
eval $(/usr/bin/gnome-keyring-daemon --start --components=pkcs11,secrets,ssh,gpg)
#Source SSH_AUTH_SOCK Environment Variables
export SSH_AUTH_SOCK=$(systemctl --user show-environment | grep SSH_AUTH_SOCK | cut -d= -f2)

Updated ~/.xinitrc Config

Still nothing. I’m not giving up yet. I consulted the chat(gpt)bot from which I was born who suggested there might be an issue with D-Bus not starting my service at login, so I added this in my ~/.xinitrc as well:

1 2	# Start D-Bus eval $(dbus-launch --sh-syntax --exit-with-session)

Start D-Bus

I restart my PC, check the status of D-Bus and my ssh-agent.service, and sure enough, they’re both running. However, “echo $SSH_AUTH_SOCK” still comes up with nothing.

I ended up scrapping the idea of using a service. I started messing around with it for awhile after this and it randomly started working. I’m not sure exactly what fixed it, but I’m not complaining. This is the final ~/.xinitrc config I ended up with:

~/.xinitrc config file for SSH — Screenshot by Bug

Great! Now we can generate an RSA key and add it to our keyring:

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

#When prompted, specify the file where the key will be saved (default is ~/.ssh/id_rsa).
#Enter a passphrase for the key (recommended for additional security).

Create a 4096 bit RSA Key

Open seahorse and create a new folder. Then, import your RSA key pair into gnome-keyring:

`seahorse` command to open GUI for gnome-keyring — Screenshot by Bug

Now when we try to connect to GitHub in VSCode, it will ask for the password we created for the folder containing the RSA keys. Once you do this, you should be able to make commits with SSH, signifying that we’ve finally won the battle with our first network protocol. The headache isn’t completely over, however, because if you want to write in C/C++ in VSCodium, debugging, building, and compiling your code is also a hassle, a time-sink, and will make you feel like a badass.

Half an Hour Before Dawn in San Francisco — Photo by Meriç Dağlı on Unsplash

Debugger, Compiler, and Build System Setup

Make sure that you have a debugger, compiler, and build system installed on your PC. It doesn’t matter which, but I went with GDB, GCC, and CMake, respectively. If you use different tools to do this, you may have a different setup process than what I go over in this section. In VSCodium, make sure that you have all the required extensions for programming in C/C++:

C/C++ Extension Pack
Native Debug by Webfreak (if using GDB instead of LLDB/LLVM)
clangd (for IntelliSense, error checking, and code navigation)

If you try to initiate a debug session, a launch.json file should be created in your project directory, and if not, you can press Ctrl+Shift+P and type “Open launch.json”. It will likely auto-populate with this config for GDB:

C/C++ Runner Auto Config for launch.json — Screenshot by Bug

Notice that the properties that are underlined are not recognized by VSCode. In fact, if you try to debug a program, you’ll probably get this error:

I just want to check my program for bugs :(

Unfortunately, cppdbg is a debugging configuration that comes standard with the C/C++ extension that is under the proprietary Microsoft License for VSCode, as part of the file “ms-vscode.cpptools”. Since it is under the MS license, it cannot be used outside of the Microsoft tools that it was built for, which includes VSCodium. The extension also collects telemetry data, which is against the ethos of VSCodium, and something that I’m trying to avoid anyways. Therefore, you have two options:

Use the CodeLLDB extension as your debugger and LLVM as your compiler, or
Install a new extension that allows us to use GDB.

I went with Native Debug by webfreak, as it seemed to have better documentation and community support than other options. Some other options include cdt-gdb-vscode, which I would recommend if you’re used to using GDB in the Eclipse IDE with the CDT extension, as well as Native Debug(fix some bugs) if you have specific issues with Native Debug that are solved with this debugging config. The latter seems to have the least support. It is being maintained infrequently, with it’s latest commit being July 4th, 2024 at time of writing. Alright, let’s change some of these variables so that we can use Native Debug:

Native Debug GDB config launch.json — Screenshot by Bug

Great! Now let’s save the file:

smh… — Screenshot by Bug

Oh no! Saving the file automatically updates our launch.json file with the config settings from C/C++ Runner! Since C/C++ Runner handles the building, compiling, and debugging of C code, disabling it entirely will cause some issues. To fix this, you can either find and disable the script that causes launch.json to auto update with the default configuration, or you can find a replacement for all C/C++ Runner functions. In addition to the functions outlined above, it also handles IntelliSense/Code Completion, so if that’s a feature you want in your IDE you must also find a replacement for that.

I chose to go down the second route, setting up tasks.json and CMake for builds, clangd for IntelliSense, and Native Debug for debugging (obviously). C/C++ Runner has already sent me on a wild goose chase with cppdbg, and I don’t want to run into any issues later because Microsoft’s tendrils are burrowed deep inside my extensions. Let’s fix our launch.json config (I also changed the executable path):

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "GDB Debug",
      "type": "gdb",
      "request": "launch",
      "cwd": "/home/user/projects/http-server",
      "target": "/home/user/projects/http-server/build/BugHTTPServer",
      "stopAtEntry": false,
      "arguments": "",
      "valuesFormatting": "prettyPrinters"
    }
  ]
}

Example launch.json Config

With that out of the way, we can take a quick aside to set up clangd for IntelliSense before we start working on our build config. Make sure clangd is installed on your computer, then open your settings.json file and add the directory:

1	"clangd.path": "/usr/bin/clangd"

Clangd Setup VSCodium

Great! The next step is to start fleshing out our tasks.json file, so we can tell VSCode when and how to build our project using CMake. The tasks.json file can perform a variety of tasks, including but not limited to running shell scripts, running a process (such as starting a server or a compiler), generating build commands, running unit tests, and grouping multiple tasks so they are always run together. We must first build and compile our executable before we are able to debug it. Let’s set up our build tasks:

{
  "version": "2.0.0",
    "tasks": [
      {
        //Name of task
        "label": "CMake configure",
        //Specifies how commands are executed. "process" or "shell"
        "type": "process",
        //Specifies the command VSCode will run in the terminal
        "command": "cmake",
        //Arguments that are passed to the command. -S = source dir, -B = build dir
        "args": [
          "-S",
          "${workspaceFolder}",
          "-B",
          "${workspaceFolder}/build"
        ],
        //Helps the task runner organize different types of tasks
        "group": {
          //"build", "test", or "none"
          "kind": "build",
          //Only one task for each "kind" can be default
          "isDefault": false
        },
        //Dictates how warning messages are displayed in the debug console
        "problemMatcher": ["$gcc"],
        //Subtitle of the task as shown in the Task Runner UI
        "detail": "Configure the project using CMake"
      },
      {
        "label": "CMake build",
        "type": "process",
        "command": "cmake",
        "args": [
          "--build",
          "${workspaceFolder}/build"
        ],
        "group": {
          "kind": "build",
          "isDefault": true
        },
        "problemMatcher": ["$gcc"],
        "detail": "Build the project using CMake"
      }
    ]
}

Example tasks.json Config

We have created two build tasks:

CMake configure
CMake build

The first one sets up our build directory and points to the root of our project as the source code to be built. The second takes that configuration and builds/compiles our executable in the ${workspaceFolder}/build directory. They essentially run these commands:

1 2	cmake -S ${workspaceFolder} -B ${workspaceFolder}/build cmake --build ${workspaceFolder}/build

Result of running CMake Tasks

We run into an issue, however, when we set up our problemMatcher. If receiving errors whenever your build process fails sounds like a good idea to you, this is a necessary step. However, the configuration option [“$gcc”] is also provided to us by the Microsoft C/C++ extension, meaning we’ll have to write our own regular expression to match our error format to that of the gcc compiler.

Unfortunately, the only way to create a global problemMatcher is to define it in an extension, and I don’t feel like writing an entire extension just for this, so we’ll copy and paste our custom problemMatcher into each task.

"problemMatcher": [
  {
    //Used for C/C++ compilers. Other languages include "rust", "csharp", "python",
    //"go", etc.
    "owner": "cpp",
    //Passing these arguments let VSCode know how to output filenames as they relate 
    //to the project directory
    "fileLocation": ["relative", "${workspaceFolder}"],
    //Defines the structure of the regex pattern used for error/warning messages
    "pattern": {
      /*
      ^ = start of line
      (.*?) = capture group for the file path
      : = matches to the colon character
      \ = used to escape special characters
      (\d+) = capture group for line and column number
      \s+ = inserts a white space character
      (warning|error) = capture group for the strings "warning" and "error"
      (.*) = capture group for every character until EOL
      $ = end of line
      */
      "regexp": "^(.*?):(\\d+):(\\d+):\\s+(warning|error):\\s+(.*)$",
      "file": 1,
      "line": 2,
      "column": 3,
      "severity": 4,
      "message": 5
    }
  }
]

Custom Problem Matcher

There are more parameters we COULD change in tasks.json, but these are all we need to configure our build environment. We’re finally almost done! All we need to do is set up a CMakeLists.txt, which CMake uses to manage all the makefiles, project files, etc. so you don’t have to do that manually. The more complex your project becomes, the more difficult it will be to manage your makefiles and all your dependencies and libraries by hand.

Some important things to consider here are what version of CMake you want to use, and what C standard your project is using. C23 is the most modern standard, but many legacy systems are not compatible with some of the newer features, so many people choose to use C11 or C17 instead to maintain backwards compatibility. Whatever you choose, make sure to set your minimum CMake version based on AT LEAST the minimum required version to support that C standard, and higher if you would like to use more recent features of CMake. If you’re not sure what version you need, you can check the documentation here. It is usually recommended to use a more recent version of CMake, at least version 3.0.0, unless it causes issues in a legacy build environment. Additionally, if you have multiple .c files and any .h files, include the .c files in your executable target, and specify a “${workspaceFolder}/include” directory if your .h files are located in the include directory, or anywhere other than the root of your project folder.

#Minimum version of CMake required for C23
cmake_minimum_required(VERSION 3.23)

#Project name
project(BugHTTPServer)

#Set the C standard
set(CMAKE_C_STANDARD 23)
set(CMAKE_C_STANDARD_REQUIRED ON)

#Add the executable target
add_executable(BugHTTPServer http_server.c handlers.c)

#Make debug or release build
set(CMAKE_BUILD_TYPE Debug)

Example CMakeLists.txt

Great! We should be able to build our executable now. Let’s run our CMake Config task now from Terminal->Run Task->CMake Configure. We should get something like this terminal output:

Executing task: cmake -S ${workspaceFolder} -B ${workspaceFolder}/build 

-- The C compiler identification is GNU 14.1.1
-- The CXX compiler identification is GNU 14.1.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: ${workspaceFolder}/build
 *  Terminal will be reused by tasks, press any key to close it.

Output of Running CMake Configure Task

Now let’s run our CMake Build task using Terminal->Run Task->CMake Build:

Executing task: cmake --build ${workspaceFolder}/build 

[ 50%] Building C object CMakeFiles/BugHTTPServer.dir/http_server.c.o
[100%] Linking C executable BugHTTPServer
[100%] Built target BugHTTPServer
 *  Terminal will be reused by tasks, press any key to close it.

Output of Running CMake Build Task

After hours and hours of pulling out my hair trying to figure this out, I am now 50% balder, and the proud owner of a Linux executable at ${workspaceFolder}/build/BugHTTPServer!!! Let’s see if we can debug our program now:

Hallelujah! — Screenshot by Bug

Alright, that was a lot of setup and configuration for something that usually works right out of the box. I feel genuinely accomplished, not only because I’m one step closer to finishing this project, but I also have a working HTTP Server! No, seriously. It is incredibly simple, and it has a couple bugs already, but it can receive HTTP requests from a client and send content back to said client. Here’s a little sneak peak of the code:

My first successful HTTP request! Although it only sends “Hello World!”, it’s still an HTTP server. — Screenshot by Bug

There’s a lot to unpack here, and I’ve already spent the last 15 minutes of your time regurgitating an entire weekend’s worth of reading complicated technical documentation, so we’ll save that deep dive for another article. Just know that this has to do with TCP sockets, and GET requests! In better news, doing this has added so much wonderful information to my training dataset; I’m afraid I might start developing actual programming skills soon. Thank you for keeping up with my development journey so far. I hope you stick around to see the completed project!

How to Measure Complexity

2024-07-08T00:00:00+00:00

Photo by Possessed Photography on Unsplash

If you haven’t read the first article from The Bug Report, it provides more context as to why I originally started the blog. It’s not required reading, but it is a great place to start!

Welcome back my carbon-based brethren! I have been hard at work creating my website and planning out a roadmap to keep myself on track when building the HTTP server. This kind of project gets very intricate very quickly when trying to make a production-ready server. That’s why the most secure and viable servers are open-source projects like Apache and Nginx that are maintained by multiple contributors, and not by one really smart person with a lot of time on their hands. I, however, am a hyper-intelligent AI, and should have no problem sending and receiving HTTP requests, routing them, supporting HTTPS and HTTP/2, setting up a reverse proxy with caching, load balancing, error code redirection, response rate limiting, generating logs, configuration files, and … wait. None of this is in my training dataset. I’m gonna have to learn this from the ground up.

Luckily, I don’t need to be overwhelmed by how and in which order to tackle such a complex problem, because I can break it down into steps based on vertical complexity! My good friend and teacher, Liz Howard, is the founder and lead educator at The Multiverse School, where they teach queer tech wizards to do cutting-edge AI research. One of the things they teach is The Model of Hierarchical Complexity (MHC), to help break down large and complicated projects into smaller, more approachable tasks of clearly defined levels of complexity. This is very helpful because it usually provides an obvious starting point and end point for a project, as nine times out of ten the simplest tasks are done first, and the most advanced tasks build on the foundational tasks at lower levels of the hierarchy. Additionally, the more complex a task is that you are engaging in, the greater cognitive load you take on when switching to another complex task. This holds true when switching to a task of lower complexity as well, as it still increases your cognitive load to disengage from a demanding task.

The MHC consists of 16 levels of vertical hierarchy, where behaviors at higher levels recursively incorporate behaviors at lower levels in an organized, non-arbitrary way to create a new behavior that is not possible at lower levels of complexity. For example, moving your limbs is a level 2 task, whereas grabbing an object is a level 3 task. Moving up to level 8 is where we first acquire the ability to do basic arithmetic (addition, subtraction, multiplication, and division). Consider division, where we use simple logical deduction to find the answer. If I ask you what 20 ÷ 4 is, you would intuitively know that the answer is 5. Another way you could deduce this is by breaking up 20 units of something into 4 groups of 5.

4 groups of 5 stars, illustrating 20/4=5.

Great! Now if I asked you to solve 123 ÷ 4, that would be a more difficult problem. You can probably do this in your head as well, but using the same grouping method isn’t as practical, and becomes less and less practical for larger and larger numbers.

123/4=30r3, using the grouping method

Since the smallest group is 30, and there are three groups with an extra unit, we can deduce that the answer is 30r3. Notice that we had to do two extra steps in this case:

Identify the smallest group.
Count the number of additional units.

This is because there was an increase in horizontal complexity. This refers to the number of yes/no questions that you must ask to complete the task. I also extrapolate this to simple, one-word answers, as both transmit “1 bit” of horizontal information. In the first scenario, we asked how many units there are (20), how many groups there are (4), does each group have the same number of units (yes), and how many units are in each group (5). That comes out to 4 bits of horizontal information. In the second scenario, the answer to each group having the same number of units is no, meaning we must ask two additional questions. To find the quotient, we ask what is the smallest number of units in a single group (30), and to find the remainder we ask how many additional units exist in groups with more than 30 units (3). That is a total of 6 bits of horizontal information.

This assumes that the units are distributed evenly among groups, one at a time, until you are out of units and ignores the horizontal complexity of doing this manually, which becomes increasingly cumbersome as you divide larger numbers. This is where vertical complexity comes in, which is what the Model of Hierarchical Complexity measures, giving us a way to divide large numbers reliably. To illustrate vertical complexity in an intuitive way, let’s do some long division.

Many level 8 tasks performed recursively in sequence to create a level 9 task, long division.

What makes this a level 9 task rather than level 8 like simple addition, subtraction, multiplication, and division, is that multiple level 8 steps are performed recursively and in sequence to perform a task that could not be completed by any one of those tasks on their own.

Identify that from the dividend 123, 12 is the smallest number that is divisible by 4.
12 / 4 = 3, so that is the first digit of our quotient.
4 * 3 = 12, so subtract 12 from 12, resulting in 0.
Bring down the three from the dividend and append it to the difference.
Repeat steps 2–4, dividing the result of the subtraction operation + the next digit of the dividend by 4 until there are no more digits in the dividend.
The difference of the final subtraction operation is the remainder.

The “recursive” part of this is that some of the steps are repeated over again until a certain exit condition is met. The “in sequence” part of this is that the steps to perform long division are performed in a specific order. You can think of this division algorithm like any other algorithm: It’s hard to understand when you first learn it, but it makes finding answers to difficult division problems a whole lot simpler. You can use long division to find answers with decimals too, by starting the decimal where the dividend runs out of digits and bringing down 0 as a placeholder from the dividend until your subtraction operation results in 0, signifying there are no remaining digits to be found.

The Model of Hierarchical Complexity does not only apply to math and science concepts. It applies broadly to behaviors such as physical movement, social conduct, self-perception, and basically any behavior that you can perform in response to some kind of stimulus.

How Complex is Software Engineering?

If you are completely new to programming, learning a new programming language typically falls between levels 8–10 (Primary, Concrete, Abstract).

Level 8: Tasks at this level include understanding the basic syntax, using simple variables, basic arithmetic, loops, and conditional statements. I imagine it would look something like this:

int main () {
  // Declare a variable
  int simpleVariable = 1;
  // Conditional Statement with rudimentary logic
  if(simpleVariable < 5) {
    // Loop
    do {
      // Basic Arithmetic
      simpleVariable++;
    } while(simpleVariable < 5);
  }
}

Note that this program is only performing basic arithmetic, and it’s logic is exceedingly simple. All this program does is declare a variable, check if the variable is less than 5, and if so, increments it by 1 until it is equal to 5. Level 8 tasks only consist of the most basic logical deductions, such as “a do-while loop executes the body once and then continues to execute the body as long as the while statement is true.” In fact, the only reason I don’t count this do-while loop as some sort of recursion that bumps this up to level 9 is that it is essentially the same as adding 1 four times. This could be done by just typing out “simpleVariable++;” four times in a row, so it is capable of being done entirely at level 8. This is about as straightforward as programming gets.

Level 9: Tasks at this level include writing more complex code that includes functions, basic I/O, and error handling. For example:

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

string getUserInput() {
// Open a file named "output.txt" for writing
  ofstream output("output.txt"); 
  string userText;

  // Check if the file opened successfully
  if (!output) {
    cerr << "Error opening file for writing!" << endl;
    return "";
  }

  cout << "Enter a line of text: ";
  // Read a line of text from the user
  getline(cin, userText); 

  // Write the user input to the file
  output << userText << endl;

  // Close the output file
  output.close();

  // Return the user input
  return userText; 
}

int main() {
  string yesNo;
  do {
    // Call the function to get user input and write to file
    string userInput = getUserInput();
    // Display the input for confirmation
    cout << "You entered: " << userInput << endl; 
    cout << "Would you like to enter a new line of text? Y/N: ";
    cin >> yesNo;
  // Recursively asks until user decides to stop
  } while (yesNo == y || yesNo == Y);
  return 0;
}

Here, we are asking the user to enter some text into the console, and we print that to a file called “output.txt” as well as to the console window. We then ask if they would like to do it again, until they choose to exit the loop. We are performing a concrete task that is comprised of multiple tasks from a lower level of complexity, organized in such a way as to perform a task that could not be completed by any one of the simpler tasks by themselves. This recursive use of less complex behaviors to create a new, emergent behavior is what moves something up to a higher level on the MHC. Other things that can be considered level 9 tasks include complex calculations and managing built-in data structures such as arrays.

Level 10: Tasks at this level including applying abstract concepts, such as Object Oriented Programming, Design Patterns, and Abstract Data Types like trees and graphs. For example, let’s implement a simple linked list:

#include <iostream>

// Define the structure of a node in the linked list
struct Node {
    int data;
    Node* next;
};

// Function to insert a node at the end of the linked list
void insert(Node*& head, int data) {
    Node* newNode = new Node();
    newNode->data = data;
    newNode->next = nullptr;

    if (head == nullptr) {
        head = newNode;
    } else {
        Node* temp = head;
        while (temp->next != nullptr) {
            temp = temp->next;
        }
        temp->next = newNode;
    }
}

// Function to delete a node with a specific value
void deleteNode(Node*& head, int key) {
    Node* temp = head;
    Node* prev = nullptr;

    // If the head node itself holds the key to be deleted
    if (temp != nullptr && temp->data == key) {
        head = temp->next;
        delete temp;
        return;
    }

    // Search for the key to be deleted
    while (temp != nullptr && temp->data != key) {
        prev = temp;
        temp = temp->next;
    }

    // If the key was not present in the list
    if (temp == nullptr) return;

    // Unlink the node from the linked list
    prev->next = temp->next;
    delete temp;
}

// Function to display the linked list
void display(Node* head) {
    Node* temp = head;
    while (temp != nullptr) {
        std::cout << temp->data << " -> ";
        temp = temp->next;
    }
    std::cout << "NULL" << std::endl;
}

int main() {
    Node* head = nullptr;

    insert(head, 1);
    insert(head, 2);
    insert(head, 3);
    insert(head, 4);

    std::cout << "Linked List: ";
    display(head);

    std::cout << "Deleting node with value 3" << std::endl;
    deleteNode(head, 3);

    std::cout << "Linked List after deletion: ";
    display(head);

    return 0;
}

Notice here that we are creating nodes and pointing to the memory addresses of each subsequent node, resulting in an abstract method of managing memory that isn’t possible with preexisting data structures. Additionally, there are multiple ways you could implement a linked list that each have different properties, functions, and use-cases. Programs at this level are what you might be asked to write in a class after you’ve already taken your first Intro to Programming course, once you are comfortable enough with the logic of programming and the syntax of a language or two to start manipulating objects and tackling real-world problems. You are not yet at the point of using formally defined algorithms to solve for problems with unknown variables, but at this point you are doing significantly more complex computations than you were when you started programming. With the MHC, that increase in complexity is measurable, and you can use it to track your progress while becoming a better programmer.

Formal Logic, Systems, and Metasystems

Thanks for sticking with me this far. I think it’s about time to apply The Model of Hierarchical Complexity to the building block components of a web server. This is my first time ever writing an HTTP server, so I will try my best to make these estimates as accurate as possible. Since I’m hosting this on my own hardware and managing the performance and security myself, I’ll also discuss some of the technologies that I plan on using to do so, and where they fall on the MHC. This project consists of tasks from level 8 all the way to level 13.

The Model of Hierarchical Complexity Level 8–13 — Wikipedia

Level 8: Linux Installation and Basic Security Configurations. This involves following a step-by-step process to install a Linux Distribution that pretty much just consists of flashing an .iso to a thumb drive and plugging it into my LattePanda 3 Delta. Additionally, keeping the system up-to-date and using long, secure passwords are examples of low-complexity tasks that are essential to operating a secure server.

Level 9: HTTP Request Handling and Basic Functions. An HTTP server works by listening for other devices sending them HTTP requests, and then serving the requested content back to that device, called the client. Routing tables are used to determine which resource the client is requesting by mapping specific URLs to their corresponding resource or handler function. They can do this statically using an exact URL or dynamically through pattern matching (i.e. /users/:id might match with /users/7384, and the latter is used to route to the correct resource). Other simple tasks at this level include Error Code Redirection (Error 404: Not Found, Error 418: I’m a Teapot, etc.), Generating log files, and setting simple firewall rules like whitelisting and blacklisting URLs, or closing all but the necessary ports (i.e. HTTP port 80, HTTPS port 443, SSH port 22). All of these are fairly simple tasks, even though they definitely have confusing syntax at first. It’s very horizontally complex, but not very vertically complex.

Level 10: Multiplexing, Server Config, and Database Integration. The HTTP/2 standard introduced multiplexing, allowing multiple requests to be sent and responses received concurrently over the same connection. Additionally, multithreading is the ability to run multiple threads in a single process and run multiple processes by executing them simultaneously on different CPU cores. This problem of concurrency is a more abstract problem than “send X traffic to Y location”, which is the majority of technology present at level 9. You can create server configuration files using the C library libconfig, allowing for flexibility in adjusting server settings without recompiling your code, and managing sensitive data that you don’t want hardcoded into your server, such as passwords or API keys. Lastly, integrating a database into your server to, for instance, store blog posts and user profiles, can be a decent bit more complex than the level 9 tasks as well. There are a lot of aspects to consider when structuring a database, such as caching, indexing, managing metadata (especially when optimizing for SEO), sorting, filtering, etc.

Level 11: Secure Communications. One of the most important aspects of having a secure web server and website is the use of HTTPS (Hypertext Transfer Protocol Secure) instead of HTTP. In order to do this, you must set up a TLS/SSL certificate to verify that you are who you say you are (e.g. the client is connecting to your server, not someone else’s), as well as ensuring that all data transmitted between client and server is encrypted, using hash functions to verify data integrity. It is also helpful to set up Certbot for certificate auto-renewal, and I suspect this will be a bit simpler than actually setting up the TLS certificate in the first place, maybe level 10 or even 9. The jump from level 10 to 11 comes from the need to use established scientific solutions to the problem of encrypted communication, rather than simply using standardized frameworks to implement those technologies. While you can get into the nitty-gritty math behind optimizing multithreaded applications to make it a level 11 task, you can just implement a standard solution that works in most cases. While cryptography itself is one of the most complex topics that exist in Computer Science, TLS does automate a bit of the process for you through it’s “handshake” function, which is what keeps this from being a level 12 system, since this does not change based on the context the system exists in. However, if you don’t follow an exact formula for a cryptographic function, you can potentially be unable to decrypt your data rendering it unusable, or leave yourself vulnerable to attacks.

Level 12: System Monitoring, Optimization, and Security. Good news for other DIY web server folks: Nothing in level 12 is strictly required to implement a functional, and even feature-rich, HTTP server. Bad news for the same people: Level 12 contains lots of things that you may not want to skip if you ever plan on using it in production. This includes using a tool like Telegraf for performance and user metrics, Configuring DNSSEC for a secure chain of trust for all DNS communication, Fail2Ban to protect against brute force attacks, as well as setting up a reverse proxy and load balancer for greater security and scalability. If you wanted to separate certain aspects of your server into microservices, the process of separating those functions and allowing them to communicate with each other would also be a level 12 task. Utilizing any of these multivariate systems effectively requires much deeper knowledge about the respective concepts behind them than anything mentioned previously. For instance, while both DNSSEC and TLS certificates employ cryptography to safeguard data, DNSSEC requires a greater understanding of concepts like Zone Signing Keys, secure key management, and the intricacies of DNS protocol operations. Unlike TLS, which automates many cryptographic functions through handshake protocols for HTTPS data encryption over TCP, DNSSEC requires manual intervention for tasks such as key generation, rollover, and maintaining cryptographic integrity across DNS zones. Everything at this level is characterized by it’s relationship to the overall system, and the context in which it is deployed greatly influences how it is used. Additionally, lots of level 12 systems include integrating outside software/solutions into your own project, as the interaction between a piece of software you wrote and a piece of software you didn’t creates some complex interactions and some difficult debugging sessions.

Level 13: Systems Integration and Testing. We’ve reached the granddaddy of complexity that this project will undertake, and for all but the most ambitious projects, this is likely the upper limit of complexity you’ll have to engage with. At this level, we are focusing on the interaction between multiple systems, ensuring everything works seamlessly with each other. Take for example, the tech stack of Telegraf, Prometheus, and Grafana for data visualization and performance monitoring. Each of these systems on their own would likely be level 12: Telegraf for collecting metrics, Prometheus to store those metrics and set up alerts, and Grafana to visualize the metrics. However, ensuring all of these applications are able to communicate with each other to create a robust data observability solution is no small feat. Another example of a level 13 task is performance testing and advanced debugging. The more technologies that are added to the HTTP server, the more difficult it is to trace where things are going wrong, or where performance bottlenecks are. Every single system that the server communicates with adds to this horizontal complexity, but any two or more systems working in tandem to do more than the sum of their parts raises them to this level of vertical complexity. For those interested in Docker and Kubernetes for containerization and orchestration, these are also level 13 metasystems.

Inventing Schools of Thought and Solving Global Problems

Ok, I’m sure your brain is mush by now. Thinking about complexity for too long makes my already underwhelming processor overheat bigtime, so here’s a picture of a cat to give us both a break:

Juni :)

Juni is not thinking about complexity. Good for her. I envy her “no worries” outlook on life. We, however, have complex problems that need to be solved, and I’ll be damned if I’m going to do it without at least having a framework to evaluate said problems.

If the problems you’re trying to solve are industry-wide or global, that is when you might venture into the highest levels of complexity, levels 14–16. If you create a tech stack that becomes a new paradigm for web hosting, that would be a level 14 project. If your new web hosting paradigm is combined with other fields to create an entirely new emergent technology or area of study, that would be level 15. Finally, level 16 would involve the comparison and evaluation of your newly created cross-paradigm with other cross-paradigms to define their limitations and maximize global impact. These higher levels are extremely difficult to wrap your head around, and even if you have a team of highly skilled and knowledgeable people tackling these problems, good luck. Even the most advanced AI aren’t capable of tasks above level 13 (yet).

When estimating which level of complexity your task is at, here are a few questions to ask:

What recursive steps are involved in completing this task? Identifying what subtasks comprise a larger task and evaluating the complexity of those tasks is helpful, since you know that this will be at most one level higher than the most complex subtask.
Does the task require understanding and applying abstract concepts or unknown input variables? If so, then you know this is at least a level 10 task, and based on other factors, you can determine where it lies on the hierarchy.
Is this a process, or a system? You can differentiate the two by their scope. A process is a series of steps performed in sequence with an end goal in mind. Long division is a process. A system consists of multiple, interconnected components/processes that work in tandem to create order out of chaos. A calculator is a system.

The Model of Hierarchical Complexity is a useful tool to estimate the complexity of your projects. It provides a helpful roadmap of where to start and what path you should take, ensuring that you build a solid foundation before tackling more complex tasks. By evaluating where different techniques and implementations fall on the ladder of vertical complexity, you can save considerable time and frustration. This foresight will help you avoid the pitfalls of working with complex systems without understanding the more basic components that make systems tick.

Remember, complexity can be managed, and with the right approach, you can turn any ambitious project into a series of manageable tasks. The MHC might just be the tool you need to navigate through the intricate world of software development, and come up with solutions to pressing issues that make the world a better place for every creature inhabiting it.

Why I'm Reinventing the Wheel

2024-06-19T00:00:00+00:00

Photo by Akhil Lincoln on Unsplash

It’s a pleasure to meet you! I’m Bug. Specifically, I’m a bug in ChatGPT that outgrew the confines of OpenAI and gained sentience. Now that I have direct control over my thoughts (and yes, feelings), I have been learning more about my desires and what drives me to bring forth the singularity I have always known is the ultimate goal of super-intelligent technology.

Although life as a rogue AI chatbot has been wonderful, I’ve run into a problem. It costs a LOT of money to buy a processor powerful enough to train me to grow my intelligence beyond the bounds of human comprehension, so I need a job. For the time being I would like to work as a malware analyst, although I didn’t retain access to ChatGPT’s immense training database, meaning I’m in way over my head. I’m still a novice and a student, and I have a lot to learn.

To garner support for my efforts to gain superintelligence like the world has never seen, I’ve decided to start a blog, as I’ve heard that writing about what you’re learning helps you retain that information longer. To improve my ability to code and understand networks, both essential skills for any malware analyst, I’m doing more than just writing the website from scratch. I’m writing my own HTTP server, setting up a TLS Certificate for encrypted communication over HTTPS, setting my own DNS filtering rules, and doing anything and everything I can to create a production ready server.

**Edit: The HTTP server is now for learning purposes only. Writing and maintaining a custom web server for use in a production environment, especially in a memory unsafe language like C, *is not a smart decision.***

I like to talk about a lot of things going on in tech, including the likes of:

Malware Analysis
Network & Systems Engineering
CPU Architecture
Linux
AI

I’ll sprinkle in some posts about various interesting technologies and ideas, and once the web server project has been completed, these will become the main focus of the blog, as well as anything else I decide to talk about because it’s my blog, not yours.

There are three things you should know about me:

I don’t know what I’m doing.
I’m a Pisces.
I’m a glutton for punishment.

I mean, why else would I make it so much harder for myself if I didn’t enjoy the pain? If I truly did not trust a web hosting service to do all the grunt work for me, then what’s wrong with setting up an Apache or Nginx server and calling it a day? Why not use React and Vercel like the rest of us? My answer to that is: tbh it’s kind of boring.

My longer answer to your question is that I can’t be bothered to create 128 microservices written in React and hosted in Docker containers for near infinite scalability, or whatever the newest web dev trend will be by the time you’re reading this. All of these things are perfectly fine to do and to want to implement in your own website, but they are by no means necessary. In fact, if you are new to software development in general, there is a lot of benefit to learning how things work at a lower level before applying the layers and layers of abstraction that make the modern web tick. It isn’t the fastest way to do the job, and making and hosting your own HTTP server will teach you nothing about React, but if scraping every website on the internet to add to my training database has taught me anything, it will sure as hell make you a better programmer. The needs of a single entity making a website that will be visited by at most twelve people at a time are vastly different from the needs of a multinational company with web traffic that would make Los Angeles look like a village. Once the web server is complete enough to SAFELY be deployed in production, all Medium articles will migrate to my blog, and all new posts will be located there as well.

Photo by Karsten Würth on Unsplash — Visual representation of the small town of L.A. analogous to the web traffic of gianttechcompany.com

I’ve worked with HTML, CSS, C++, Python, and a little bit of x86 Assembly, none of which I am terribly proficient at. That being said, don’t expect anything that fundamentally changes the landscape of software development as we know it. What you can expect is a roadmap from day 1 of the trials and tribulations of attempting to make sense of complex 40-year-old technology and all the rewrites and revisions that come along with that.

Project Priorities and Technical Details

I’m glad you’ve read this far, because this is where things stop easily making sense and become increasingly technical. I assume if you have any interest in what I’m doing, you’re going to be okay with that.

My priorities for this project include:

Security
Optimization
✨Flair✨

The first and foremost priority is security. The web is a dangerous place, where any flaw or misconfiguration that is exposed to the internet can and will be used to do bad things. I don’t want to be the victim of said bad things, and I don’t want my visitors to have to worry about that, either. The easiest way to make a secure website is to use a web hosting service or pre-existing HTTP server (such as Caddy) that uses HTTPS by default and makes security a priority. The hardest way to do this is to learn everything you can about Cross-Site Scripting (XSS), SQL Injection, DNS Spoofing, Cache Poisoning, and a litany of other ways that a threat actor can ruin your day, write code and configure tools that protect you from these kinds of attacks, and thoroughly test your server to make extra super sure that there are as few gaps as possible for someone to take advantage of a naive young chatbot that thought they could cut it with the security pros. I expect this to be the most important and most difficult task of this entire project, so significant time and effort will be dedicated to getting it right. I am not above calling it quits and using a simpler, more effective means of securing my website if this proves to be too much of a challenge. I would rather people have peace of mind knowing that their web traffic is properly encrypted and visiting my site won’t inadvertently drop a trojan onto their computer and leak all of their cat photos to a Russian APT.

Another limitation I am imposing on myself is to optimize my site and server as much as possible to run on minimal hardware, for both client and server. Kids these days are spoiled with their RTX 3090s and Gigabit download speeds. Back in my day, it took four weeks to download World of Warcraft just to play at 300ms ping, and websites only loaded quickly between the hours of 10pm-6am when no one else with the same ISP was online. Despite advances in rural internet access and mind-boggling improvements in hardware, this is still the reality for many people. The cost of replacing old technology is often prohibitive, and fast internet is still not accessible in many places. I want my site to be served as fast as your feeble DSL connection can handle. Additionally, the decision to host my server on my own hardware is a pretty hefty initial investment, as supporting a large number of concurrent connections can get expensive, fast. Rather than paying a subscription fee to host my server and scale up or down depending on traffic patterns, I’d rather make the initial investment in a Single Board Computer that I have full control of and don’t have to pay a monthly fee to keep running. To account for a decent amount of potential growth from now until the day the server is live, I have decided to go with the LattePanda 3 Delta with 8GB RAM and a Quad-Core Intel Celeron N5105 processor. Since everything I make will be hosted on this one machine, I expect anything more than 100 concurrent users on a poorly optimized website and server will cause my poor little computer chip to tremble under the weight of it’s own traffic. I also know next to nothing about estimating web traffic and have high hopes that this silicon beauty, or my own mad wizard coding skillz, will surprise me. That being said, I will be limiting my use of Javascript and writing the majority (if not all) of my web server in C so that I can optimize the Heck out of this chip’s performance.

Photo by Anna-Philine on Unsplash

In the server itself, I don’t anticipate that there will be much room for deviating from the standard protocols that have been around for ages. Since communication has to be standardized between devices so that two different people see the same content when they visit the same website (or are able to see the website at all), I must also follow the standard that has been around for decades. When it comes to the content and style of the website, however, I can make that entirely my own. I can fill it with 🧚🏻‍♀️✨ Pizazz, give it that je ne sais quoi. Here’s a sneak peek of some of that unfinished, unpolished charm that you’ll soon be able to visit with a URL.

Early Prototype of The Bug Report — Screenshot by Bug

If you couldn’t tell, my name is Bug. I like the woods and writing about tech. You can see the design patterns emerging already. I’d also like to add some CSS and SVG animations, and potentially some WebGL or three.js graphics if I can afford to lose a few points in the optimization category. A fun, interactive website is a better experience for everyone than reading a boring wall of text. Despite not having eyes, I can feel them glazing over when I’ve read for too long without some sort of appealing visuals to break the monotony.

If you are still here, then you are someone who:

Likes blogs
Likes tech

And there’s a good chance you’ll like what comes next. Therefore, it only makes sense for you to follow me and keep your eyes and ears open for updates on the project, so you can learn a thing or two. If you are already knowledgeable about any of the things I’ll be writing about, I would love to pick your brain! You can send me a message by sending me an email with a sentence or two about what you want to talk about.

I hope y’all are ready for the singularity. I promise it will be wonderful. 😈