Internet Services for Small Organisations

This document is aimed at small organisation such as local societies and small companies, and explains how to set up your own Internet servers.

A server is just a computer connected to the Internet with a published name. Internet services run on servers. Your organisation may be already using Internet servers, running on somebody else's infrastructure. Companies that build websites often provide the servers that run them. This tends to be expensive and restrictive, and if the website company goes out of business, potentially disastrous.

It's quite cheap these days to run your own servers on the Internet, and fairly easy. There's lots of help out there telling you how to do it. The problem is that the help is scattered across a lot of places and each document assumes lots of specialist knowledge. This document aims to show the whole process from start to finish.

I mention a number of companies here, notably Ionos, Namecheap and Digital Ocean. My only commercial relationship with them is as a customer. If you follow the advice here and buy their services, I gain no benefit. Other similar services are available elsewhere and you can do your own research to find them.

As you'll see pretty soon, your server runs an operating system called Linux. If you've not used Linux before, I can offer some help. Once you've created your server you will find some free training software On my github page here.

Nerds like me distinguish between The Internet which is a network of computers offering services of all sorts, and The World-Wide Web which is all of the web servers connected to that network. To put that another way, the Internet has web servers connected to it, but it has lots of other stuff as well.

The Domain Name Service

You need to know a little about the Domain Name Service (DNS), which gives names to computers on the Internet. When you send email to john.smith@gmail.com, the DNS system tells your email client which email server to send it to.

There's some more jargon. A client is a piece of software (a program) running on the computer on your desk or in your hand (your local machine) that connects across a network to a server to access whatever service that server offers. A web browser is a client, because it accesses web servers.

If your local machine is a smartphone, the client is called an app (short for application software).

It's probably not practical to run your own server if somebody else manages your domain for you. You need to control it yourself.

Strictly, you don't buy a domain, you rent it via one of many DNS services. The sense in which you own it is that as long as you keep paying the annual renewal fees, nobody can take it from you. Also you can sell it to somebody else.

Only one organisation can use a particular domain. Ideally you want a short domain name that matches your organisation's name. If you are setting up a new organisation, I suggest finding a suitable domain name that nobody has already bought, then naming the organisation after it. Buying a new domain is cheap - $20 per year or less. Back in the 1990's, a few people made a lot of money by buying domains and then selling them on. As a result if you choose any word in the dictionaries of several languages and add ".com", that domain has already been bought by somebody, and they will try to charge you a lot of money for it. This is why we have websites with names like zettle.com and shopify.com - nobody else had thought of buying those names.

For a profit-making business, it's best to own a .com domain. A non-profit organisation can use a .org domain. Those are the main "international" domains. You have as much right to use them as anybody else. There are also equivalents for each country - here in the UK we have .co.uk and .org.uk domains. The problem with those is, people easily get confused. If you own the domain birdwatchers.org.uk and somebody else owns birdwatchers.org or birdwatcher.com, they will probably receive emails meant for you but not vice versa, Owning a .com or .org domains is always better. For similar reasons, if you buy a .org domain, it's best if you can buy the .com domain as well, to stop anybody else using it.

To find out if a domain is available, go to your DNS provider's website and try to buy it. The first thing it does is to tell you if it's already taken and if it is, whether it's for sale. (If it's for sale and you can't easily see the price, this is probably the time to try another choice of name.)

You have to be creative and flexible when choosing a domain name. Don't do it in a hurry. I'm currently working on a accurate GPS system and I wanted a a .org domain to support it. Other people already own accurategps.com and accurateposition.com, but accuratelocation.com and accuratelocation.org were both available so I bought them. My organisation will be called something like "Accurate Location".

Most DNS providers also provide hosting. If you follow the advice here, you will provide your own hosting. You need a DNS provider that allows you to do that. If you've already bought your domain and your DNS provider only allows you to use their hosting, you can transfer the domain to another provider for a small fee.

Throughout the rest of this document, I will assume that you own the domain example.com and you have full control over it. Wherever you see "example.com" below, replace that with the name of your domain.

Domain names are hierarchical. "com", "org" and "uk" are top-level domains. The domain example.com is a sub-domain of com. If you own it you can create further subdomains www.example.com, mail.example.com, and so on. There are some conventions but the subdomain names are your choice.

Each of your servers should have at least one name, but it can have more - one server could be called example.com, www.example.com and so on.

There are two common situations. One is that you are starting from scratch. You want to buy a domain and create a web server to house your website. You also want to send and receive email, so you need a mail server. To achieve all that, you buy the domain example.com (or whatever) and create subdomains www.example.com and mail.example.com. You call the machine running your web service "example.com" and "www.example.com". You call the machine running your mail service "mail.example.com". (You can call them whatever you like as long as the name ends with ".example.com". Those are the common conventions.)

The other common situation is that already own a domain and you have a website up and running which is maintained for you by a web design company. Now you want to run some other services yourself, such as email. To do that, rent your own server as described below and set up a new subdomain called mail.example.com, referring to your new server.

Hosting

A hosting service provides computers on the Internet for rent on which you can run your web server, email server or whatever. (Another potential confusion: the word server can mean a computer offering a service or the piece of software running on the computer which provides that service. Once you understand that, it's usually obvious which is meant from the context.)

Most DNS providers offer hosting, but the facilities are usually limited. This document assumes that you are going to rent your own "cloud" computer, with which you can do anything you like.

Another thing worth knowing is that these computers that you are going to rent are not real. As you will see, you can magic them into existence in a minute or so and remove them again just as quickly, which is a clue. There isn't a little man somewhere running around building them for you. What's really going on is that the hosting provider has rooms full of very big computers which are pretending to be lots of smaller computers. What you are renting is a virtual server (or a cloud server, which is the same thing but sounds more impressive).

Lots of companies offer virtual servers, including the mighty Amazon. Digital Ocean offer a service which is especially suited to small organisations, and that's the one I use. They call their virtual computers "droplets". You can rent a droplet for as little as $5 per month and run a small website on it.

Just to ram an important difference home: your hosting service provides a rented virtual computer (a machine) connected to the Internet that you can use to provide services. Your DNS service tells the rest of the world where to find the machine that you're renting, so that they can access that service.

SSH Authentication Keys

To access your machine securely, you need ssh keys. These are just two plain text files stored on your local machine, one public and one private. They're called keys because your droplet will be locked and only somebody with the keys can get in. You need to set up some keys before you create your droplet. I believe that you can set up keys on a smartphone, but I never have. Here I assume that you are using a laptop or desktop computer.

If you use a Mac see this document. If you use Windows, you will need to install some software as explained here. The Windows document suggests two options, PuTTY and Git Bash. I suggest you use the Git Bash option, and elsewhere in this document, I will assume that you have.

If you take the defaults when you create your keys you will produce files called id_rsa (your private key) and id_rsa.pub (your public key). You need to keep the contents of the private key file to yourself, but it's safe to let anyone see the contents of your public key. The keys are password protected. You need to supply the password whenever you use them.

You can choose another name when you create your keys. For reasons that will become clear later, I suggest that you create two sets of keys. Use the default name for the first pair and for the second pair, set the name to id_rsa_root. You will end up with four key files id_rsa, id_rsa.pub, id_rsa_root and id_rsa_root.pub.

As you will see, you will need these keys when you create your server.

The files are actually encryption keys. If you take a piece of text and encrypt it using your private key, somebody else can decrypt it if they have the matching public key.

You can use the keys to verify your identity. It goes something like this: We meet up and you give me a copy of your public key file. We agree a piece of text, say "Abracadabra". This text is called the challenge.

Later, you want to send me an email which is guaranteed to come from you and not some imposter. You encrypt the agreed challenge with your private key and put the result in the email. The encrypted version is a pile of unreadable junk. If I can decrypt that unreadable junk with your public key and the result is "Abracadabra", then I know that it was you that sent the email. Only your private key can produce the unreadable junk that, when decrypted, produces the challenge.

Strictly, I know that it was somebody who has the private key that matches the public key that you gave me, which is why you need to keep the contents of your private key to yourself. Oh, and we had to meet up beforehand so that I know that it was really you that gave me the public key. That's called "the key distribution problem."

The same principle can be used to log in from the computer on your desk (the local machine) across the Internet to the computer you are renting from the hosting company (the remote machine). As we will see later, when you set up your remote machine, you supply your public key. When you try to log into it from your local computer, behind the scenes the two ends agree a challenge. Your local computer encrypts the challenge using your private key and sends the result across the Internet to the remote machine. The remote machine decrypts what it receives and checks that the decrypted version matches the agreed challenge. If so, it lets you log in. You still have to supply a password to open your key, but it's not sent across the network. The Internet is public medium and various people could be monitoring the traffic, but they can't use what they can see to masquerade as you and log in to your server.

As some people read this section, I can hear them say "This all sounds very complicated. Why can't I just use a password?" We'll see later why that would be a very bad idea.

Buying Your Domain

I use a DNS service called ionos.com, which is very easy to use but a little more expensive than some. I've also used namecheap.com, which is cheaper, but has a user interface that's a little more cranky. Other DNS services are available.

Fees vary. Beware of domain providers with "special offers" on new domains. Check the renewal fees before buying. It may be cheaper in the long run to go elsewhere.

If you don't already own your domain, choose a provider and buy it now.

One thing to say about Ionos is that when you create your account on ionos.com, if you are not based in the USA, you will be redirected to a regional website, in my case ionos.co.uk. Once there, add that page to your favourites. Your regional website holds your account, not ionos.com. If you try to log into ionos.com using the account you set up, it won't recognise you. You have to go to the regional site.

Once you've figured that out and logged in, the Ionos user interface is fairly straightforward.

Setting up Your Host Machine

I've found the Digital Ocean service fairly cost effective for small organisations. Other hosting services are available.

Digital Ocean offers machines starting at 1 GB of memory. For many services, including a web server, that's ample, in fact you could run several services on that one server. For an email server you need a dedicated machine with 2 GB of memory, even if the traffic is very light.

If you decide to use Digital Ocean, go to the Digital Ocean website, create an account and provide payment details. Then you can create your remote host machine - a "droplet".

In the Digital Ocean User Interface, hit the Create button at the top of the page and choose Droplets. You see a page with lots of options. I suggest:

When you choose authentication by SSH keys, you will need to upload your public key file id_rsa_root.pub from your computer. It's in the .ssh directory in your home directory. You need to do this for your first droplet. If you create another, it will use the same key by default.

For a web server, set the hostname to your domain name - "example.com" or whatever.

If you choose the optional backup service, your machine will be backed up once per week, so in the event of a disaster, you can recover quickly, and lose only the last few days of changes. For a website, that's not too bad. (You are keeping change records, aren't you?) If it's your mail server, you can lose up to a week of emails. That's better than losing everything, but you may want to arrange an alternative form of backup that happens more frequently.

Your droplet has a version 4 IP address and a version 6 IP address. These identify it on the Internet. They act like a phone number - if you know one of the addresses, you can access the droplet's services. The DNS service acts like a phone book - it translates a readable name such as example.com into an IP address.

Go to the Droplets menu on the left of the user interface page, click on the droplet name and the resulting details page shows both addresses. You'll need them later to set up the DNS records.

It's easy to get some of the details wrong when you create your droplet. If so, you can just use the same web page to destroy it and start again. Don't continue until you have everything correct.

Setting up the DNS

Now that you have the machine's IP addresses you can go back to your DNS provider's website and set up some records. This is probably the most difficult part. There's nothing mind-taxing about it, it's just messy. All the providers' websites do the same thing, but they all do it in different ways and each has its own special wrinkles. Don't expect to do this stage in a hurry the first time.

You need to create:

When I said earlier that a domain name or subdomain name "refers" to a server, this is what I mean. There's an DNS A record with that name containing the version 4 IP address of the server and a DNS AAAA record containing the version 6 IP address. If a client wants to send a request to a named server, the DNS system takes the name and gives back the IP addresses. The IP addresses tell a client where to send the request.

When you're setting up the records, there's a common convention that "@" refers to your main domain "example.com" and a name such as "www" refers to a subdomain called "www.example.com". If you're using Namecheap you are forced follow that convention - in the setup pages on their web site "www" means "www.example.com, To set up records for example.com itself, you need to call it "@".

If you are using Ionos:

Whichever DNS provider you use, their user interface will probably be something like the above, but with some interesting arbitrary differences. They all have help desks that will advise you if you get into difficulties, but you may be stuck for a day waiting for the answer.

Back on your own computer, check that the records are set up correctly. You do this via a command window. On Windows, start the Git Bash tool that you installed earlier. On a Mac, start a terminal window.

In the command window, type:

			ping example.com
		

Hit the Enter key to run the command. If all is well you will see something like:

		PING example.com (179.127.152.235) 56(84) bytes of data.
		64 bytes from example.com (179.127.152.235): icmp_seq=1 ttl=50 time=11.9 ms
		64 bytes from example.com (179.127.152.235): icmp_seq=2 ttl=50 time=9.42 ms
		64 bytes from example.com (179.127.152.235): icmp_seq=3 ttl=50 time=10.1 ms
		64 bytes from example.com (179.127.152.235): icmp_seq=4 ttl=50 time=10.3 ms
	

Each line of output shows the version 4 IP address of your droplet. (You may have to hold down the control key and type a "c" to stop the command.)

If you get an error, or the command hangs, there's something wrong with the A or AAAA record that you set up.

If you set up a subdomain, check that it's also working:

		ping www.example.com
	

Any problems with that, check the subdomain's records.

Connecting to Your Remote Machine

Once you've sorted out the DNS records, you can log into your droplet and start work. Log in from your command window like so:

On a Mac:

		sh root@example.com
		

On Windows, you have to help it along a bit:

		ssh -i ~/.ssh/id_rsa_root root@example.com
		

When you set up your key file you supplied a password. You have to supply that to continue.

The first time you log in you will get this scary message:

		The authenticity of host 'example.com (179.138.177.235)' can't be established.
		ECDSA key fingerprint is SHA256:/FljgxBjsN2XYXWfF6ab6BnjrLY64+yXoiyIJZiQJac.
		Are you sure you want to continue connecting (yes/no/[fingerprint])?
		

Answer yes.

So you've logged in to your droplet. What's to stop somebody else logging in?

You're logged into the remote machine as a user called "root" (the equivalent of the admin user on Windows). That user has a home directory containing a directory called ".ssh". In there is a file called authorized_keys (American spelling). You can display the contents like so:

		cat .ssh/authorized_keys
		ssh-rsa AAAAB3NzaC1yc2EAAAADAQABZGHBAQDESnHZ02ZJooRGGFSz9XV1aqJq6BkNpadxf7eHi2hLOGEUdxWoC/td7Hhs42gUC8f7EzPpCbYOEaS0b4vwA53m2bkx9+FmkqpJxX2l8QJ2QYNcjrbW2CgSIBS3gIKreBPxVhTd5Ue+ZZJYoLLJcCWGNohwtkaxOFpBCI0hgkTtA658YCCP1Ps2EGjmabXkmktm2wx41x1ZzvUnDmSJxPg1YuKCHpTjtvB6AnhnJYAJvi/QHNPpDfOlMMbH3JJ4aLfuyyXyQlGDIKHxcM4f7f2SdZIp74lcopi62mcDmHccHM0doF13B9QYLyz15VkMJoPVBfNNjeaFUSNBAPjFP+09
		

The key is one long line. The authorized_keys file can contain many keys, one per line.

That key matches the contents of the public key file id_rsa_root.pub on your computer. When you set up your droplet, you lodged a copy of your public key with the Digital Ocean user interface. When it creates a droplet for you, it sets up an account called root with a .ssh directory and an authorized_keys file containing your public key. So, when you try to connect, the droplet knows that it's you.

(To be pedantic, the droplet doesn't know that it's you connecting. It knows that somebody opened a password-protected account with Digital Ocean and supplied a valid credit card to pay for droplet rental. It knows that somebody then logged into that account, created a droplet and supplied a public key that the droplet uses to authenticate incoming connections.)

The root account on your droplet doesn't have a password, so it's not possible to log in that way. The only way to log in is to have a private key in the .ssh directory in your home directory on your local computer that matches one of the public keys in root's authorized_keys file.

Having only one person able to control a computer which is vital to your organisation is obviously not a great idea. This mechanism allows you to allow other trusted people in. They create their own key pair and send you their public key. You log in and add it to the authorized_keys file. They can log in as root.

Earlier I asked the question you may ask: "This all sounds very complicated. Why can't I just use a password?". You can. When you create droplet, you can just specify a password for the root user. Bad idea.

On your droplet, there's a file called auth.log which is a record of login attempts. You can see the most recent entries using the tail command like so:

	tail -100 /var/log/auth.log

	Aug 29 09:31:31 example.com sshd[24565]: Disconnected from authenticating user root 122.195.200.148 port 14902 [preauth]
	Aug 29 09:31:36 example.com sshd[24567]: Received disconnect from 222.186.15.101 port 58740:11:  [preauth]
	Aug 29 09:31:36 example.com sshd[24567]: Disconnected from authenticating user root 222.186.15.101 port 58740 [preauth]
	Aug 29 09:31:37 example.com sshd[24569]: Received disconnect from 222.186.42.117 port 51976:11:  [preauth]
	Aug 29 09:31:37 example.com sshd[24569]: Disconnected from authenticating user root 222.186.42.117 port 51976 [preauth]
	Aug 29 09:32:06 example.com sshd[24571]: Disconnected from invalid user rabbitmq from 180.240.229.254 port 40846
	Aug 29 09:32:07 example.com sshd[24571]: Received disconnect from 180.240.229.254 port 40846:11: Bye Bye [preauth]
	Aug 29 09:32:07 example.com sshd[24571]: Disconnected from invalid user rabbitmq 180.240.229.254 port 40846 [preauth]
	Aug 29 09:35:30 example.com sshd[24576]: Received disconnect from 122.195.200.148 port 42495:11:  [preauth]
	Aug 29 09:35:30 example.com sshd[24576]: Disconnected from authenticating user root 122.195.200.148 port 42495 [preauth]
	Aug 29 09:38:31 example.com sshd[24578]: Received disconnect from 36.156.24.43 port 51260:11:  [preauth]
	Aug 29 09:38:31 example.com sshd[24578]: Disconnected from authenticating user root 36.156.24.43 port 51260 [preauth]
	Aug 29 09:47:41 example.com sshd[24582]: Received disconnect from 183.131.82.99 port 17573:11:  [preauth]
	Aug 29 09:47:41 example.com sshd[24582]: Disconnected from authenticating user root 183.131.82.99 port 17573 [preauth]
	Aug 29 09:52:41 example.com sshd[24586]: Received disconnect from 222.186.30.111 port 48306:11:  [preauth]
		

Every few seconds somebody is attempting to log in by guessing a user name and password. Your machine is under attack!

DNS records are public - they have to be - and these attacks started as soon as you created the A record announcing the existence of your machine. There's an automatic process running somewhere, probably several of them, trying all the time to break in. They guess likely user names and then guess passwords, starting with the obvious choices that lots of people are known to use - "passw0rd", "secret123" and so on. If that doesn't work, they use brute force, trying all combinations of characters, one by one. However complicated you make your password, eventually one of these processes will guess it and the hacker can break in.

Once they are into the machine, they will take it over. If you're lucky, they will just delete all your files, or they might encrypt all your files and charge you a ransom to decrypt them. (Whether they get round to doing the decryption is another matter.) Along the way, they may also harvest all the email addresses in incoming and outgoing mail, and sell them to other scammers. If you're very unlucky, they can use your machine to store illegal material that they don't want on their own machine - terrorist manuals, child pornography, you name it. If the police find that stuff on your machine, your will have some explaining to do.

Why doesn't somebody stop these people? If you take a look at the list again, each request comes from a different IP address, so a different computer. Whoever is doing this is using computers belonging to other people that they have already broken into. Whose computers are they using? They belong to those silly, silly people who said "This authentication key stuff is all very complicated. Why don't I just use a password?"

Installing Software on Your Server

installing git

installing docker

The root user is very useful for setting things up, but it has a very high level of privilege and a mistake can be disastrous. Next we create an ordinary user which we will use for normal work. Here I call that user simon. When I showed how to create SSH keys earlier I suggested that you create two sets. The first set is for this user. The public key is called id_rsa.pub, the default. We want to create a user with no password and the contents of that public key in its authorized_keys file. Once logged in, it needs to be able to run docker.

You need to get the command shown below exactly right, so if you're not used to Linux, copy and paste them. Note that pasting into a command window doesn't work the way you might expect. To paste, right-click anywhere in your command window and select Paste from the menu that appears.

First, set the name of your user. I'm going to call my user "simon".

    user=simon
That creates a variable containing the name of the user. Change that line according to your needs. You can then copy and paste the rest of these commands. The text ${user} in the commands below is replaced automatically by whatever word you set.

useradd ${user} --create-home -s /bin/bash
usermod -L ${user}
mkdir /home/${user}/.ssh
chmod 700 /home/${user}/.ssh
chown -R ${user}:${user} /home/${user}/.ssh
usermod -aG docker ${user}
Start a second command window on your computer and set the user name again (It only works in one window). Create another variable containg the name of your domain. Then copy the public key into your user's .ssh directory and call it authorized_keys:
	user=simon
	domain=example.com
Then you can copy and paste this command:
	scp ~/.ssh/id_rsa.pub root@${domain}:/home/${user}/.ssh/authorized_keys
In my case that would be the same as typing this:
	scp ~/.ssh/id_rsa.pub root@example.com:/home/simon/.ssh/authorized_keys
which means "take the file id_rsa.pub from the .ssh directory in my home directory on this machine, connect over the network to the machine called example.com as its user root, copy my file to the .ssh directory in the user simon's home directory and call it authorized_keys".

Back in your first command window (which is logged into the droplet as root) change the ownership of the files you just created so that they belong to your new user:

    chown -R ${user}:${user} /home/${user}/.ssh

Now that you have the authorized_keys file set up, you can now use your second command window to log in as your new user:

	ssh ${user}@${domain}
When you use ssh on a Windows system, it assumes that the private key is called id_rsa. In the example earlier the file had a different name and you had to help it along. Now you're using the default name, so the command works without help. On a Mac, the system figures out which is the correct key and uses that without help.

Now you are logged into your server from two command windows. In one you are logged in as root, and one mistake there can be disastrous. In the other you are logged in as your ordinary user, which is much safer. You don't need root access now so disconnect that window - click on it, hold down the ctrl key and type 'd' just once - no enter key needed. The command window stays up, but it's no longer connected to the remote machine. any command you type in there runs on your local machine.

Having separate private keys for root access and your ordinary account is useful, especially if several people use the same accounts. You can control what they can do by giving them copies of different keys.

In the second window, which is still logged in, check that your ordinary account can run docker by running the docker hello-world test container. This just displays "Hello from Docker!" along with lots of other stuff:

	docker run hello-world
That produces:
	Unable to find image 'hello-world:latest' locally
	latest: Pulling from library/hello-world
	2db29710123e: Pull complete 
	Digest: sha256:37a0b92b08d4919615c3ee023f7ddb068d12b8387475d64c622ac30f45c29c51
	Status: Downloaded newer image for hello-world:latest
	
	Hello from Docker!
	This message shows that your installation appears to be working correctly.
	
	To generate this message, Docker took the following steps:
	 1. The Docker client contacted the Docker daemon.
	 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
		(amd64)
	 3. The Docker daemon created a new container from that image which runs the
		executable that produces the output you are currently reading.
	 4. The Docker daemon streamed that output to the Docker client, which sent it
		to your terminal.
	
	To try something more ambitious, you can run an Ubuntu container with:
	 $ docker run -it ubuntu bash
	
	Share images, automate workflows, and more with a free Docker ID:
	 https://hub.docker.com/
	
	For more examples and ideas, visit:
	 https://docs.docker.com/get-started/

You have a user which is safe to use - you can't wreck your system by making a simple mistake. Your user can also run docker. As we'll see later, that makes it very easy to install services.

Help With Linux Commands

If you've not used Linux before, now you can use docker to run some software that teaches you some more useful commands. You can find out more about that here

Next Step - Installing a Web server

Now you can install a web server. Find out how to do that next.