Self-hosting
with tunnels
16th March 2025
I have been using Nextcloud for a long time, running on a publicly-accessible VPS. The machine it is running on is not very powerful, but the server runs fine and I’ve never had much trouble with it.
However, I want to scale up the storage on it. I have a 4TB hard drive, which I bought with the intention of using it to store backups. I am much too paranoid to store backups of my personal laptop in the cloud. And even if I did, it would be very expensive: just 320GB of attached storage costs over $30 USD per month at Hetzner, which is nearly triple what I’m paying at the moment.
The obvious suggestion is to host these services at home. But I don’t have a static IP at home, so I never really understood how to route traffic to it.
What’s tunnelling?
A tunnel is a long-running connection between two computers that are unable to establish the connection directly. Often this works by opening a connection in the reverse direction first - for instance, your local machine sends a request to a public server, and the connection is kept open. When the public server receives requests from other people, some of those requests get forwarded to the local machine, it sends a response back over the open channel, and the public server forwards them to the initial requestor.
There is a list of tunnelling solutions at awesome-tunnelling.
They recommend Cloudflare
Tunnel. In this case, you run a program called
cloudflared
locally, which maintains an open
connection to cloudflare’s infrastructure. You register your
domain name with them, and they connect any remote requests for
your domain to the local service. This is configurable, so you
can have e.g. nextcloud.danielittlewood.xyz
pointed
to localhost:8993
and
mastodon.danielittlewood.xyz
pointed to
localhost:9967
. ngrok (now
proprietary) works on a similar principle.
That’s all well and good, cloudflared
is even
open source. But I don’t really understand how it works, and it
seems to be tied to cloudflare’s services (which are generally
non-free). Mac Chaffee (McAfee?) seems to have struggled with
the same issue in his (apparently very similar) article Flouting
the Internet Protocols with Tunnels.
As documented there, you can achieve a simple compromise using ssh tunnelling. In this case, as long as two computers can connect via ssh to a trusted third party, you can maintain a tunnel between them. Here is an example snippet:
# on client 1
ssh -N -T -R 22222:localhost:22 your-remote-server
# on client A
ssh -p 22222 your-remote-server
# client A is now logged into client 1
This is very simple; so simple I even tried to implement it
in the past. But it means maintaining a very small server
running just to proxy connections, which is a crazy waste of
resources. It turns out someone beat me to writing the software,
and wrote sish. I found it
via Awesome-Selfhosted.
In fact, the maintainers of sish
even run a cheap
managed service call tuns.sh
. It is part of pico.sh, which costs about $2 USD
per month. There are some other cute things they will give you
as well, like static site hosting. Giving them money feels
playing my small part in breaking up the homogeneity of the
web.
Example tuns.sh configuration
To get an account on https://pico.sh, I ran ssh pico.sh
and used their terminal client to register (very unusual, but
also very cute!). Then I gave $30 away for a year of the premium
service. To set up the reverse tunnel seems very easy:
ssh -R ssh:22:localhost:22 tuns.sh
The effect is that another client can run
ssh -J tuns.sh danielittlewood-ssh
to connect.
danielittlewood
is my username, and
ssh
is just a name for this particular tunnel. Port
assignment is (username, tunnel-name)-local, so you can have
multiple services exposed on port 443, for instance.
This is actually enough for me to get a simple network
attached storage. At home, I have two laptops, the “client”
(which I want to back up) and the “server” (which I want to run
services from). I copied my public ssh key by hand from the
client into the server’s .ssh/authorized_keys
file.
I also created a new ssh key pair on the server with the
following snippet:
ssh-keygen -t ed25519 -C "dan-selfhosting"
On the client, I ran ssh pico.sh
and added the
new public key by hand. This means both machines will be able to
connect to https://tuns.sh, which is necessary for the
tunnel to work. I run that ssh -R
line above on the
server, and on the client:
ssh -J nue.tuns.sh danielittlewood-ssh
Connection to nue.tuns.sh closed by remote host.
Connection closed by UNKNOWN port 65535
It didn’t work! The reason is that when you set up a “private
alias” you have to specify on the server command
line all the SSH fingerprints of the people who are allowed to
connect. In this case, I want everyone in my
~/.ssh/authorized_keys
file to be able to
connect:
ssh -R ssh:22:localhost:22 nue.tuns.sh \
tcp-aliases-allowed-users=$(ssh-keygen -lf ~/.ssh/authorized_keys \
| awk '{ print $2; }' | paste -sd ",")
Note that I also specified nue.tuns.sh
rather
than tuns.sh
- that was deliberate too. I spoke to
hello@pico.sh
about it:
When you use
ssh tuns.sh
, it selects the datacenter closest to you (Nuremberg DE, nue.tuns.sh or Ashburn VA-US, ash.tuns.sh). You can select the server which is closest to you manually by usingssh {ash or nue}.tuns.sh
. That will ensure that your tunnel server is “durable” as we don’t use a global routing tunnel for tuns (at least not yet).
P.S. if you want to use sshfs, here is the snippet for that:
sshfs -o ssh_command="ssh -J tuns.sh" danielittlewood-ssh:/media mount
Automating with guix
This is all well and good – it works, you can use it. But it’s a bit brittle. The tunnel is frequently closed, and to restart it I have to go fetch my self-hosting machine to reconnect. I also have trouble remembering the command. I am running GNU Guix on this server, so the obvious way to make the connection robust was to write a shepherd user service to reopen the tunnel whenever it closed. This was surprisingly easy!
Reopening the tunnel
After following the recommended setup I linked above, I created a service with the following definition. It is very simple, and cribbed from an example in the manual:
; ~/.config/shepherd/init.d/tuns.sh.scm
(use-modules (shepherd support))
define tuns-sh
(
(service
'(tuns-sh)"Maintain open ssh tunnel to tuns.sh"
#:documentation lambda ()
#:start (
(fork+exec-command"ssh" "-R" "ssh:22:localhost:22" "nue.tuns.sh" "tcp-aliases-allowed-users=SHA256:...")
`(string-append %user-log-dir "/tuns.sh.log")))
#:log-file (
#:stop (make-kill-destructor)#t
#:respawn? 3 . 5)))
#:respawn-limit '(
list tuns-sh))
(register-services ( (start-service tuns-sh)
There is nothing mysterious here. The arguments to
fork+exec-command
are just the space-separated
command line arguments you would write in the shell. I have had
to copy the public ssh key fingerprints out of the
authorized-keys
file, but that can be fixed. The
variable %user-log-dir
by default points to
~/.local/share/shepherd
.
(make-kill-destructor)
means that when you call
shepherd stop
it’ll just kill whatever process was
started. And the respawn
arguments mean that if the
command exits for some reason, shepherd
will resume
it by just rerunning the command. The (3 . 5)
notation just means “do not try more than 3 times in 5 seconds”,
in case your command is really broken.
By logging out and logging in, or rebooting, or running
shepherd start tuns-sh
, I can see the service
running. It gets logged to a sensible place, and all is
well.
Other ports
After using this service successfully for a couple of days I
am looking for a way to host other services. Obviously I don’t
want to have to replicate all that boilerplate just to add
another nearly identical service. Luckily, since shepherd is
configured in scheme, it is completely trivial to write a
function that creates services. Literally all you have to do is
identify the pieces of data that are variable, and interpolate
them. In the example below I need the name of the service, its
local port, and the remote port tuns
will
expose.
(use-modules (shepherd support))
define (tuns-service name local-port public-port)
(
(servicelist name)
(string-append "Maintain an open ssh tunnel for service " (symbol->string name))
#:documentation (lambda ()
#:start (
(fork+exec-command"ssh" "-R" ,(string-append (symbol->string name) ":" (number->string public-port) ":" "localhost" ":" (number->string local-port)) "nue.tuns.sh" "tcp-aliases-allowed-users=SHA256:g8XCuncz8RrAd7vv0EQK6ze7fM1fY7wp/Y3/GJTLv4M,SHA256:RN7XhizoH1JzUNARFagrqzj6VR8Hi/8pDKS/UeVoyPc"))
`(string-append %user-log-dir "/" (symbol->string name) ".log"))
#:log-file (
#:stop (make-kill-destructor)#t
#:respawn? 3 . 5)))
#:respawn-limit '(
define ssh (tuns-service 'ssh 22 22))
(
list ssh))
(register-services ( (start-service ssh)
Now it is trivial to add new services - almost a single line! While I was debugging this I hit a couple of annoying problems.
- At some point I accidentally ran multiple copies of shepherd
simultaneously. This seems like it’s just an error, but I didn’t
get warned about it at all. I had to kill the shepherd process
manually by doing
ps -aux | grep shepherd
and doingkill $PID
. - Once I killed the extra shepherd processes, I discovered I
actually had screwed up my ssh tunnel, and got booted
out. So I had to find the IP address on my local network using
ifconfig
to find my IP address192.168.4.81
and doingsudo nmap -p 22 192.168.4.0/24
to find the other machine. - I couldn’t figure out how to reload my init script without
doing the tedious
kill shepherd
snippet. Also this would have killed my tunnel again if I didn’t have the direct-IP tunnel open. It would be nice for shepherd to be able to validate scripts before actually running them. - Sometimes the error messages I got from shepherd were not
super helpful. For example at one point I gave the service a
name which was a string rather than a symbol, and the error said
Throw to key
%exception’ with args("#<&message message: \"invalid service provision list\">")'.
. I didn’t actually know what a provision list was until I read the manual – it would have been nice to see something likeexpected symbol but got string "ssh"
or something like this.
Getting the
authorized_keys
automatically
I think this should be easy, but I haven’t done it yet.