Self-hosting with tunnels

Published on 16 March, 2025

I have been using [Nextcloud](https://nextcloud.com/) for a long time, running on a publicly-accessible VPS. The machine it is running on is not very powerful, but the server runs fine and I've never had much trouble with it.

However, I want to scale up the storage on it. I have a 4TB hard drive, which I bought with the intention of using it to store backups. I am much too paranoid to store backups of my personal laptop in the cloud. And even if I did, it would be very expensive: just 320GB of attached storage costs over $30 USD per month at Hetzner, which is nearly triple what I'm paying at the moment.

The obvious suggestion is to host these services at home. But I don't have a static IP at home, so I never really understood how to route traffic to it.

## What's tunnelling?

A *tunnel* is a long-running connection between two computers that are unable to establish the connection directly. Often this works by opening a connection in the reverse direction first - for instance, your local machine sends a request to a public server, and the connection is kept open. When the public server receives requests from other people, some of those requests get forwarded to the local machine, it sends a response back over the open channel, and the public server forwards them to the initial requestor.

There is a list of tunnelling solutions at [awesome-tunnelling](https://github.com/anderspitman/awesome-tunneling).

They recommend [Cloudflare Tunnel](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/). In this case, you run a program called `cloudflared` locally, which maintains an open connection to cloudflare's infrastructure. You register your domain name with them, and they connect any remote requests for your domain to the local service. This is configurable, so you can have e.g. `nextcloud.danielittlewood.xyz` pointed to `localhost:8993` and `mastodon.danielittlewood.xyz` pointed to `localhost:9967`. [ngrok](https://github.com/inconshreveable/ngrok) (now proprietary) works on a similar principle.

That's all well and good, `cloudflared` is even open source. But I don't really understand how it works, and it seems to be tied to cloudflare's services (which are generally non-free). Mac Chaffee (McAfee?) seems to have struggled with the same issue in his (apparently very similar) article [Flouting the Internet Protocols with Tunnels](https://www.macchaffee.com/blog/2024/tunneling/).

As documented there, you can achieve a simple compromise using [ssh tunnelling](https://stackoverflow.com/a/76741677/4681998). In this case, as long as two computers can connect via ssh to a trusted third party, you can maintain a tunnel between them. Here is an example snippet:

``` # on client 1 ssh -N -T -R 22222:localhost:22 your-remote-server # on client A ssh -p 22222 your-remote-server # client A is now logged into client 1 ```

This is very simple; so simple I even tried to implement it in the past. But it means maintaining a very small server running just to proxy connections, which is a crazy waste of resources. It turns out someone beat me to writing the software, and wrote [sish](https://github.com/antoniomika/sish). I found it via [Awesome-Selfhosted](https://awesome-selfhosted.net/). In fact, the maintainers of `sish` even run a cheap managed service call `tuns.sh`. It is part of [pico.sh](https://pico.sh/), which costs about $2 USD per month. There are some other cute things they will give you as well, like static site hosting. Giving them money feels playing my small part in breaking up the homogeneity of the web.

## Example tuns.sh configuration

To get an account on <https://pico.sh>, I ran `ssh pico.sh` and used their terminal client to register (very unusual, but also very cute!). Then I gave $30 away for a year of the premium service. To set up the reverse tunnel seems very easy:

``` ssh -R ssh:22:localhost:22 tuns.sh ```

The effect is that another client can run `ssh -J tuns.sh danielittlewood-ssh` to connect. `danielittlewood` is my username, and `ssh` is just a name for this particular tunnel. Port assignment is (username, tunnel-name)-local, so you can have multiple services exposed on port 443, for instance.

This is actually enough for me to get a simple network attached storage. At home, I have two laptops, the "client" (which I want to back up) and the "server" (which I want to run services from). I copied my public ssh key by hand from the client into the server's `.ssh/authorized_keys` file. I also created a new ssh key pair on the server with the following snippet:

``` ssh-keygen -t ed25519 -C "dan-selfhosting" ```

On the client, I ran `ssh pico.sh` and added the new public key by hand. This means both machines will be able to connect to <https://tuns.sh>, which is necessary for the tunnel to work. I run that `ssh -R` line above on the server, and on the client:

``` ssh -J nue.tuns.sh danielittlewood-ssh Connection to nue.tuns.sh closed by remote host. Connection closed by UNKNOWN port 65535 ```

It didn't work! The reason is that when you set up a "[private alias](https://docs.ssi.sh/cheatsheet#https-private-access)" you have to specify on the *server* command line all the SSH fingerprints of the people who are allowed to connect. In this case, I want everyone in my `~/.ssh/authorized_keys` file to be able to connect:

``` ssh -R ssh:22:localhost:22 nue.tuns.sh \ tcp-aliases-allowed-users=$(ssh-keygen -lf ~/.ssh/authorized_keys \ | awk '{ print $2; }' | paste -sd ",") ```

Note that I also specified `nue.tuns.sh` rather than `tuns.sh` - that was deliberate too. I spoke to <hello@pico.sh> about it:

> When you use `ssh tuns.sh`, it selects the datacenter closest to you > (Nuremberg DE, nue.tuns.sh or Ashburn VA-US, ash.tuns.sh). You can select the > server which is closest to you manually by using `ssh {ash or nue}.tuns.sh`. > That will ensure that your tunnel server is "durable" as we don't use a global > routing tunnel for tuns (at least not yet).

P.S. if you want to use [sshfs](https://github.com/libfuse/sshfs), here is the snippet for that:

``` sshfs -o ssh_command="ssh -J tuns.sh" danielittlewood-ssh:/media mount ```

## Automating with guix

This is all well and good -- it works, you can use it. But it's a bit brittle. The tunnel is frequently closed, and to restart it I have to go fetch my self-hosting machine to reconnect. I also have trouble remembering the command. I am running GNU Guix on this server, so the obvious way to make the connection robust was to write a [shepherd user service](https://www.gnu.org/software/shepherd/manual/html_node/Managing-User-Services.html) to reopen the tunnel whenever it closed. This was surprisingly easy!

### Reopening the tunnel

After following the recommended setup I linked above, I created a service with the following definition. It is very simple, and cribbed from an example in the manual:

```scm ; ~/.config/shepherd/init.d/tuns.sh.scm

(use-modules (shepherd support))

(define tuns-sh (service '(tuns-sh) #:documentation "Maintain open ssh tunnel to tuns.sh" #:start (lambda () (fork+exec-command `("ssh" "-R" "ssh:22:localhost:22" "nue.tuns.sh" "tcp-aliases-allowed-users=SHA256:...") #:log-file (string-append %user-log-dir "/tuns.sh.log"))) #:stop (make-kill-destructor) #:respawn? #t #:respawn-limit '(3 . 5)))

(register-services (list tuns-sh)) (start-service tuns-sh) ```

There is nothing mysterious here. The arguments to `fork+exec-command` are just the space-separated command line arguments you would write in the shell. I have had to copy the public ssh key fingerprints out of the `authorized-keys` file, but that can be fixed. The variable `%user-log-dir` by default points to `~/.local/share/shepherd`. `(make-kill-destructor)` means that when you call `shepherd stop` it'll just kill whatever process was started. And the `respawn` arguments mean that if the command exits for some reason, `shepherd` will resume it by just rerunning the command. The `(3 . 5)` notation just means "do not try more than 3 times in 5 seconds", in case your command is really broken.

By logging out and logging in, or rebooting, or running `shepherd start tuns-sh`, I can see the service running. It gets logged to a sensible place, and all is well.

### Other ports

After using this service successfully for a couple of days I am looking for a way to host other services. Obviously I don't want to have to replicate all that boilerplate just to add another nearly identical service. Luckily, since shepherd is configured in scheme, it is completely trivial to write a function that creates services. Literally all you have to do is identify the pieces of data that are variable, and interpolate them. In the example below I need the name of the service, its local port, and the remote port `tuns` will expose.

```scm (use-modules (shepherd support))

(define (tuns-service name local-port public-port) (service (list name) #:documentation (string-append "Maintain an open ssh tunnel for service " (symbol->string name)) #:start (lambda () (fork+exec-command `("ssh" "-R" ,(string-append (symbol->string name) ":" (number->string public-port) ":" "localhost" ":" (number->string local-port)) "nue.tuns.sh" "tcp-aliases-allowed-users=SHA256:g8XCuncz8RrAd7vv0EQK6ze7fM1fY7wp/Y3/GJTLv4M,SHA256:RN7XhizoH1JzUNARFagrqzj6VR8Hi/8pDKS/UeVoyPc")) #:log-file (string-append %user-log-dir "/" (symbol->string name) ".log")) #:stop (make-kill-destructor) #:respawn? #t #:respawn-limit '(3 . 5)))

(define ssh (tuns-service 'ssh 22 22))

(register-services (list ssh)) (start-service ssh) ```

Now it is trivial to add new services - almost a single line! While I was debugging this I hit a couple of annoying problems.

1. At some point I accidentally ran multiple copies of shepherd simultaneously. This seems like it's just an error, but I didn't get warned about it at all. I had to kill the shepherd process manually by doing `ps -aux | grep shepherd` and doing `kill $PID`. 2. Once I killed the extra shepherd processes, I discovered I actually *had* screwed up my ssh tunnel, and got booted out. So I had to find the IP address on my local network using `ifconfig` to find my IP address `192.168.4.81` and doing `sudo nmap -p 22 192.168.4.0/24` to find the other machine. 3. I couldn't figure out how to reload my init script without doing the tedious `kill shepherd` snippet. Also this would have killed my tunnel again if I didn't have the direct-IP tunnel open. It would be nice for shepherd to be able to validate scripts before actually running them. 4. Sometimes the error messages I got from shepherd were not super helpful. For example at one point I gave the service a name which was a string rather than a symbol, and the error said `Throw to key `%exception' with args `("#<&message message: \"invalid service provision list\">")'.`. I didn't actually know what a provision list was until I read the manual -- it would have been nice to see something like `expected symbol but got string "ssh"` or something like this.

### Getting the `authorized_keys` automatically

I think this should be easy, but I haven't done it yet.