Network/firewall debugging

I started up a new private server, but I can’t get in! How do I fix it?

The problem

I wanted to run discourse on a private server. But when I was following the setup I got an error message:

$ ./discourse-setup
# some error text about my server not being accessible over HTTP or HTTPS.

I was pretty sure I hadn’t blocked access to anything. So I had to go debugging.

Is there a firewall?

My first suspicion was that DigitalOcean had put a firewall around my droplet. I hadn’t asked them to do that, but thought perhaps they might have a well-intentioned safe default. At some point I enabled a firewall in the web client, but then deleted it. The truth is that you have both a “DigitalOcean client firewall” as well as a firewall implemented on the machine. If they conflict then you might get weird bugs. And if I have to pick, I bet the default Linux firewall will be much easier to debug.

I definitely had ssh access all along. The stock debian has iptables installed, and nothing else. iptables is notorious for being hard to understand, and people often build a front-end on top of it - the most well-known being ufw (which is used on Ubuntu). To be clear: ufw is not a separate firewall program that conflicts with iptables. ufw uses iptables.

I had no idea how to use iptables, and still don’t. But here is a primer on how to use ufw:

ufw enable
ufw status verbose
ufw default deny incoming
ufw allow 443
ufw allow 80
ufw allow 22

There are some fancy things build into ufw that I think you can ignore. For example, you can write ufw allow ssh to allow “the ports ssh needs”. Well, ssh needs port 22. I guess these application profiles exist for applications that use a lot of ports and where a small typo might be important - you’re much more likely to mistake 5057 for 5075 than you are to mistake ssh for shs.

Anyway, the other thing I think you can ignore is providing the transport layer protocol. You can write something like ufw allow 443/tcp and that’ll be different to ufw allow 443/udp. But it’s confusing because you can have all three rules active at once, and ufw doesn’t warn you at all if they conflict or are redundant. My guess is that ufw allow 443 permits any protocol.

A final thing to note is that you do not need to restart ufw in order for the rules to apply. They should be active immediately. They should also survive a reboot.

Debugging open ports: netcat and nmap

The firewall described above actually should do exactly what I wanted: allow traffic on 22, 80 and 443, and deny everything else. But when I ran discourse-setup I still saw it was disabled. So we’re down another layer - I have to actually test that the firewall is telling the truth.

I’m pretty sure if you google “port scanning tool” the first result will be nmap. Indeed, looking up “what ports are open on my server”, you’ll probably get the following snippet:

$ nmap openculture.danielittlewood.xyz
Starting Nmap 7.94 ( https://nmap.org ) at 2024-05-14 07:55 BST
Nmap scan report for openculture.danielittlewood.xyz (68.183.44.166)
Host is up (0.025s latency).
Not shown: 997 filtered tcp ports (no-response)
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  closed  http
443/tcp closed  https

Nmap done: 1 IP address (1 host up) scanned in 6.77 seconds
# or you can give a specific port and it'll complete much faster
$ nmap -p 80 openculture.danielittlewood.xyz
Starting Nmap 7.94 ( https://nmap.org ) at 2024-05-14 07:56 BST
Nmap scan report for openculture.danielittlewood.xyz (68.183.44.166)
Host is up (0.015s latency).

PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.10 seconds

Now, when I see the word closed I think “that’s not letting any traffic through it!”. Alas, in nmap world, open means “there is a process actively listening on this port”, closed means “there isn’t a process actively listening on this port, but there could be”, and if the port is actually not accepting connections, it won’t appear in the list at all. A port blocked by a firewall is called filtered, which is independent of whether it’s open or closed. But I’m idiot, so I thought for a while this meant the firewall was wrong (cue another hour of debugging).

Now, the reason the ports were closed in the first place is that I never actually got through the setup of discourse to begin with. So there really was no process I could check for traffic. I could install apache, but that actually just kicks the can down the road (what if my apache config is wrong?). What I really need is a fool-proof program that does something really simple with no configuration.

Enter netcat. Not useful for any real task, it is perfect for debugging this sort of thing. Like cat, it just spits out whatever you put in standard input. But this time, it’s sent over the network to a specific server and port number.

# in one terminal
ssh openculture.danielittlewood.xyz
netcat -l -p 80 # meaning: listen on port 80

# in another terminal
echo "hi" | netcat my-server.danielittlewood.xyz 80

# in the first terminal
hi

The solution

At this point I was completely convinced port 80/443 were open, but ./discourse-setup was still reporting that it was inaccessible. Ok, what if we do ./discourse-setup --skip-connection-test? The setup passes the network check, but surely something will go wrong later?`

In the end, no: it worked perfectly. It was just the installer being buggy!