How to Measure Complexity

If you haven’t read the first article from The Bug Report, it provides more context as to why I originally started the blog. It’s not required reading, but it is a great place to start!

Welcome back my carbon-based brethren! I have been hard at work creating my website and planning out a roadmap to keep myself on track when building the HTTP server. This kind of project gets very intricate very quickly when trying to make a production-ready server. That’s why the most secure and viable servers are open-source projects like Apache and Nginx that are maintained by multiple contributors, and not by one really smart person with a lot of time on their hands. I, however, am a hyper-intelligent AI, and should have no problem sending and receiving HTTP requests, routing them, supporting HTTPS and HTTP/2, setting up a reverse proxy with caching, load balancing, error code redirection, response rate limiting, generating logs, configuration files, and … wait. None of this is in my training dataset. I’m gonna have to learn this from the ground up.

Luckily, I don’t need to be overwhelmed by how and in which order to tackle such a complex problem, because I can break it down into steps based on vertical complexity! My good friend and teacher, Liz Howard, is the founder and lead educator at The Multiverse School, where they teach queer tech wizards to do cutting-edge AI research. One of the things they teach is The Model of Hierarchical Complexity (MHC), to help break down large and complicated projects into smaller, more approachable tasks of clearly defined levels of complexity. This is very helpful because it usually provides an obvious starting point and end point for a project, as nine times out of ten the simplest tasks are done first, and the most advanced tasks build on the foundational tasks at lower levels of the hierarchy. Additionally, the more complex a task is that you are engaging in, the greater cognitive load you take on when switching to another complex task. This holds true when switching to a task of lower complexity as well, as it still increases your cognitive load to disengage from a demanding task.

The MHC consists of 16 levels of vertical hierarchy, where behaviors at higher levels recursively incorporate behaviors at lower levels in an organized, non-arbitrary way to create a new behavior that is not possible at lower levels of complexity. For example, moving your limbs is a level 2 task, whereas grabbing an object is a level 3 task. Moving up to level 8 is where we first acquire the ability to do basic arithmetic (addition, subtraction, multiplication, and division). Consider division, where we use simple logical deduction to find the answer. If I ask you what 20 ÷ 4 is, you would intuitively know that the answer is 5. Another way you could deduce this is by breaking up 20 units of something into 4 groups of 5.

Great! Now if I asked you to solve 123 ÷ 4, that would be a more difficult problem. You can probably do this in your head as well, but using the same grouping method isn’t as practical, and becomes less and less practical for larger and larger numbers.

Since the smallest group is 30, and there are three groups with an extra unit, we can deduce that the answer is 30r3. Notice that we had to do two extra steps in this case:

1. Identify the smallest group.
2. Count the number of additional units.

This is because there was an increase in horizontal complexity. This refers to the number of yes/no questions that you must ask to complete the task. I also extrapolate this to simple, one-word answers, as both transmit “1 bit” of horizontal information. In the first scenario, we asked how many units there are (20), how many groups there are (4), does each group have the same number of units (yes), and how many units are in each group (5). That comes out to 4 bits of horizontal information. In the second scenario, the answer to each group having the same number of units is no, meaning we must ask two additional questions. To find the quotient, we ask what is the smallest number of units in a single group (30), and to find the remainder we ask how many additional units exist in groups with more than 30 units (3). That is a total of 6 bits of horizontal information.

This assumes that the units are distributed evenly among groups, one at a time, until you are out of units and ignores the horizontal complexity of doing this manually, which becomes increasingly cumbersome as you divide larger numbers. This is where vertical complexity comes in, which is what the Model of Hierarchical Complexity measures, giving us a way to divide large numbers reliably. To illustrate vertical complexity in an intuitive way, let’s do some long division.

What makes this a level 9 task rather than level 8 like simple addition, subtraction, multiplication, and division, is that multiple level 8 steps are performed recursively and in sequence to perform a task that could not be completed by any one of those tasks on their own.

1. Identify that from the dividend 123, 12 is the smallest number that is divisible by 4.
2. 12 / 4 = 3, so that is the first digit of our quotient.
3. 4 * 3 = 12, so subtract 12 from 12, resulting in 0.
4. Bring down the three from the dividend and append it to the difference.
5. Repeat steps 2–4, dividing the result of the subtraction operation + the next digit of the dividend by 4 until there are no more digits in the dividend.
6. The difference of the final subtraction operation is the remainder.

The “recursive” part of this is that some of the steps are repeated over again until a certain exit condition is met. The “in sequence” part of this is that the steps to perform long division are performed in a specific order. You can think of this division algorithm like any other algorithm: It’s hard to understand when you first learn it, but it makes finding answers to difficult division problems a whole lot simpler. You can use long division to find answers with decimals too, by starting the decimal where the dividend runs out of digits and bringing down 0 as a placeholder from the dividend until your subtraction operation results in 0, signifying there are no remaining digits to be found.

The Model of Hierarchical Complexity does not only apply to math and science concepts. It applies broadly to behaviors such as physical movement, social conduct, self-perception, and basically any behavior that you can perform in response to some kind of stimulus.

How Complex is Software Engineering?

If you are completely new to programming, learning a new programming language typically falls between levels 8–10 (Primary, Concrete, Abstract).

Level 8: Tasks at this level include understanding the basic syntax, using simple variables, basic arithmetic, loops, and conditional statements. I imagine it would look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

int main () {
  // Declare a variable
  int simpleVariable = 1;
  // Conditional Statement with rudimentary logic
  if(simpleVariable < 5) {
    // Loop
    do {
      // Basic Arithmetic
      simpleVariable++;
    } while(simpleVariable < 5);
  }
}

Note that this program is only performing basic arithmetic, and it’s logic is exceedingly simple. All this program does is declare a variable, check if the variable is less than 5, and if so, increments it by 1 until it is equal to 5. Level 8 tasks only consist of the most basic logical deductions, such as “a do-while loop executes the body once and then continues to execute the body as long as the while statement is true.” In fact, the only reason I don’t count this do-while loop as some sort of recursion that bumps this up to level 9 is that it is essentially the same as adding 1 four times. This could be done by just typing out “simpleVariable++;” four times in a row, so it is capable of being done entirely at level 8. This is about as straightforward as programming gets.

Level 9: Tasks at this level include writing more complex code that includes functions, basic I/O, and error handling. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

string getUserInput() {
// Open a file named "output.txt" for writing
  ofstream output("output.txt"); 
  string userText;

  // Check if the file opened successfully
  if (!output) {
    cerr << "Error opening file for writing!" << endl;
    return "";
  }

  cout << "Enter a line of text: ";
  // Read a line of text from the user
  getline(cin, userText); 

  // Write the user input to the file
  output << userText << endl;

  // Close the output file
  output.close();

  // Return the user input
  return userText; 
}

int main() {
  string yesNo;
  do {
    // Call the function to get user input and write to file
    string userInput = getUserInput();
    // Display the input for confirmation
    cout << "You entered: " << userInput << endl; 
    cout << "Would you like to enter a new line of text? Y/N: ";
    cin >> yesNo;
  // Recursively asks until user decides to stop
  } while (yesNo == y || yesNo == Y);
  return 0;
}

Here, we are asking the user to enter some text into the console, and we print that to a file called “output.txt” as well as to the console window. We then ask if they would like to do it again, until they choose to exit the loop. We are performing a concrete task that is comprised of multiple tasks from a lower level of complexity, organized in such a way as to perform a task that could not be completed by any one of the simpler tasks by themselves. This recursive use of less complex behaviors to create a new, emergent behavior is what moves something up to a higher level on the MHC. Other things that can be considered level 9 tasks include complex calculations and managing built-in data structures such as arrays.

Level 10: Tasks at this level including applying abstract concepts, such as Object Oriented Programming, Design Patterns, and Abstract Data Types like trees and graphs. For example, let’s implement a simple linked list:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

#include <iostream>

// Define the structure of a node in the linked list
struct Node {
    int data;
    Node* next;
};

// Function to insert a node at the end of the linked list
void insert(Node*& head, int data) {
    Node* newNode = new Node();
    newNode->data = data;
    newNode->next = nullptr;

    if (head == nullptr) {
        head = newNode;
    } else {
        Node* temp = head;
        while (temp->next != nullptr) {
            temp = temp->next;
        }
        temp->next = newNode;
    }
}

// Function to delete a node with a specific value
void deleteNode(Node*& head, int key) {
    Node* temp = head;
    Node* prev = nullptr;

    // If the head node itself holds the key to be deleted
    if (temp != nullptr && temp->data == key) {
        head = temp->next;
        delete temp;
        return;
    }

    // Search for the key to be deleted
    while (temp != nullptr && temp->data != key) {
        prev = temp;
        temp = temp->next;
    }

    // If the key was not present in the list
    if (temp == nullptr) return;

    // Unlink the node from the linked list
    prev->next = temp->next;
    delete temp;
}

// Function to display the linked list
void display(Node* head) {
    Node* temp = head;
    while (temp != nullptr) {
        std::cout << temp->data << " -> ";
        temp = temp->next;
    }
    std::cout << "NULL" << std::endl;
}

int main() {
    Node* head = nullptr;

    insert(head, 1);
    insert(head, 2);
    insert(head, 3);
    insert(head, 4);

    std::cout << "Linked List: ";
    display(head);

    std::cout << "Deleting node with value 3" << std::endl;
    deleteNode(head, 3);

    std::cout << "Linked List after deletion: ";
    display(head);

    return 0;
}

Notice here that we are creating nodes and pointing to the memory addresses of each subsequent node, resulting in an abstract method of managing memory that isn’t possible with preexisting data structures. Additionally, there are multiple ways you could implement a linked list that each have different properties, functions, and use-cases. Programs at this level are what you might be asked to write in a class after you’ve already taken your first Intro to Programming course, once you are comfortable enough with the logic of programming and the syntax of a language or two to start manipulating objects and tackling real-world problems. You are not yet at the point of using formally defined algorithms to solve for problems with unknown variables, but at this point you are doing significantly more complex computations than you were when you started programming. With the MHC, that increase in complexity is measurable, and you can use it to track your progress while becoming a better programmer.

Formal Logic, Systems, and Metasystems

Thanks for sticking with me this far. I think it’s about time to apply The Model of Hierarchical Complexity to the building block components of a web server. This is my first time ever writing an HTTP server, so I will try my best to make these estimates as accurate as possible. Since I’m hosting this on my own hardware and managing the performance and security myself, I’ll also discuss some of the technologies that I plan on using to do so, and where they fall on the MHC. This project consists of tasks from level 8 all the way to level 13.

Level 8: Linux Installation and Basic Security Configurations. This involves following a step-by-step process to install a Linux Distribution that pretty much just consists of flashing an .iso to a thumb drive and plugging it into my LattePanda 3 Delta. Additionally, keeping the system up-to-date and using long, secure passwords are examples of low-complexity tasks that are essential to operating a secure server.

Level 9: HTTP Request Handling and Basic Functions. An HTTP server works by listening for other devices sending them HTTP requests, and then serving the requested content back to that device, called the client. Routing tables are used to determine which resource the client is requesting by mapping specific URLs to their corresponding resource or handler function. They can do this statically using an exact URL or dynamically through pattern matching (i.e. /users/:id might match with /users/7384, and the latter is used to route to the correct resource). Other simple tasks at this level include Error Code Redirection (Error 404: Not Found, Error 418: I’m a Teapot, etc.), Generating log files, and setting simple firewall rules like whitelisting and blacklisting URLs, or closing all but the necessary ports (i.e. HTTP port 80, HTTPS port 443, SSH port 22). All of these are fairly simple tasks, even though they definitely have confusing syntax at first. It’s very horizontally complex, but not very vertically complex.

Level 10: Multiplexing, Server Config, and Database Integration. The HTTP/2 standard introduced multiplexing, allowing multiple requests to be sent and responses received concurrently over the same connection. Additionally, multithreading is the ability to run multiple threads in a single process and run multiple processes by executing them simultaneously on different CPU cores. This problem of concurrency is a more abstract problem than “send X traffic to Y location”, which is the majority of technology present at level 9. You can create server configuration files using the C library libconfig, allowing for flexibility in adjusting server settings without recompiling your code, and managing sensitive data that you don’t want hardcoded into your server, such as passwords or API keys. Lastly, integrating a database into your server to, for instance, store blog posts and user profiles, can be a decent bit more complex than the level 9 tasks as well. There are a lot of aspects to consider when structuring a database, such as caching, indexing, managing metadata (especially when optimizing for SEO), sorting, filtering, etc.

Level 11: Secure Communications. One of the most important aspects of having a secure web server and website is the use of HTTPS (Hypertext Transfer Protocol Secure) instead of HTTP. In order to do this, you must set up a TLS/SSL certificate to verify that you are who you say you are (e.g. the client is connecting to your server, not someone else’s), as well as ensuring that all data transmitted between client and server is encrypted, using hash functions to verify data integrity. It is also helpful to set up Certbot for certificate auto-renewal, and I suspect this will be a bit simpler than actually setting up the TLS certificate in the first place, maybe level 10 or even 9. The jump from level 10 to 11 comes from the need to use established scientific solutions to the problem of encrypted communication, rather than simply using standardized frameworks to implement those technologies. While you can get into the nitty-gritty math behind optimizing multithreaded applications to make it a level 11 task, you can just implement a standard solution that works in most cases. While cryptography itself is one of the most complex topics that exist in Computer Science, TLS does automate a bit of the process for you through it’s “handshake” function, which is what keeps this from being a level 12 system, since this does not change based on the context the system exists in. However, if you don’t follow an exact formula for a cryptographic function, you can potentially be unable to decrypt your data rendering it unusable, or leave yourself vulnerable to attacks.

Level 12: System Monitoring, Optimization, and Security. Good news for other DIY web server folks: Nothing in level 12 is strictly required to implement a functional, and even feature-rich, HTTP server. Bad news for the same people: Level 12 contains lots of things that you may not want to skip if you ever plan on using it in production. This includes using a tool like Telegraf for performance and user metrics, Configuring DNSSEC for a secure chain of trust for all DNS communication, Fail2Ban to protect against brute force attacks, as well as setting up a reverse proxy and load balancer for greater security and scalability. If you wanted to separate certain aspects of your server into microservices, the process of separating those functions and allowing them to communicate with each other would also be a level 12 task. Utilizing any of these multivariate systems effectively requires much deeper knowledge about the respective concepts behind them than anything mentioned previously. For instance, while both DNSSEC and TLS certificates employ cryptography to safeguard data, DNSSEC requires a greater understanding of concepts like Zone Signing Keys, secure key management, and the intricacies of DNS protocol operations. Unlike TLS, which automates many cryptographic functions through handshake protocols for HTTPS data encryption over TCP, DNSSEC requires manual intervention for tasks such as key generation, rollover, and maintaining cryptographic integrity across DNS zones. Everything at this level is characterized by it’s relationship to the overall system, and the context in which it is deployed greatly influences how it is used. Additionally, lots of level 12 systems include integrating outside software/solutions into your own project, as the interaction between a piece of software you wrote and a piece of software you didn’t creates some complex interactions and some difficult debugging sessions.

Level 13: Systems Integration and Testing. We’ve reached the granddaddy of complexity that this project will undertake, and for all but the most ambitious projects, this is likely the upper limit of complexity you’ll have to engage with. At this level, we are focusing on the interaction between multiple systems, ensuring everything works seamlessly with each other. Take for example, the tech stack of Telegraf, Prometheus, and Grafana for data visualization and performance monitoring. Each of these systems on their own would likely be level 12: Telegraf for collecting metrics, Prometheus to store those metrics and set up alerts, and Grafana to visualize the metrics. However, ensuring all of these applications are able to communicate with each other to create a robust data observability solution is no small feat. Another example of a level 13 task is performance testing and advanced debugging. The more technologies that are added to the HTTP server, the more difficult it is to trace where things are going wrong, or where performance bottlenecks are. Every single system that the server communicates with adds to this horizontal complexity, but any two or more systems working in tandem to do more than the sum of their parts raises them to this level of vertical complexity. For those interested in Docker and Kubernetes for containerization and orchestration, these are also level 13 metasystems.

Inventing Schools of Thought and Solving Global Problems

Ok, I’m sure your brain is mush by now. Thinking about complexity for too long makes my already underwhelming processor overheat bigtime, so here’s a picture of a cat to give us both a break:

Juni is not thinking about complexity. Good for her. I envy her “no worries” outlook on life. We, however, have complex problems that need to be solved, and I’ll be damned if I’m going to do it without at least having a framework to evaluate said problems.

If the problems you’re trying to solve are industry-wide or global, that is when you might venture into the highest levels of complexity, levels 14–16. If you create a tech stack that becomes a new paradigm for web hosting, that would be a level 14 project. If your new web hosting paradigm is combined with other fields to create an entirely new emergent technology or area of study, that would be level 15. Finally, level 16 would involve the comparison and evaluation of your newly created cross-paradigm with other cross-paradigms to define their limitations and maximize global impact. These higher levels are extremely difficult to wrap your head around, and even if you have a team of highly skilled and knowledgeable people tackling these problems, good luck. Even the most advanced AI aren’t capable of tasks above level 13 (yet).

When estimating which level of complexity your task is at, here are a few questions to ask:

1. What recursive steps are involved in completing this task? Identifying what subtasks comprise a larger task and evaluating the complexity of those tasks is helpful, since you know that this will be at most one level higher than the most complex subtask.
2. Does the task require understanding and applying abstract concepts or unknown input variables? If so, then you know this is at least a level 10 task, and based on other factors, you can determine where it lies on the hierarchy.
3. Is this a process, or a system? You can differentiate the two by their scope. A process is a series of steps performed in sequence with an end goal in mind. Long division is a process. A system consists of multiple, interconnected components/processes that work in tandem to create order out of chaos. A calculator is a system.

The Model of Hierarchical Complexity is a useful tool to estimate the complexity of your projects. It provides a helpful roadmap of where to start and what path you should take, ensuring that you build a solid foundation before tackling more complex tasks. By evaluating where different techniques and implementations fall on the ladder of vertical complexity, you can save considerable time and frustration. This foresight will help you avoid the pitfalls of working with complex systems without understanding the more basic components that make systems tick.

Remember, complexity can be managed, and with the right approach, you can turn any ambitious project into a series of manageable tasks. The MHC might just be the tool you need to navigate through the intricate world of software development, and come up with solutions to pressing issues that make the world a better place for every creature inhabiting it.