Ubuntu VPS install

Initial Setup

Determine the IP address of the new server and ssh into it as the root user. This information should be provided by the hosting company that provisioned the server.

$ ssh -l root -p <ip address>

Set a new root password. Use a password generator to create a complex password. Save the password in the vault and paste it to the session.

# passwd root

Apply all system updates.

# sudo apt update
# sudo apt upgrade

User setup and authentication

Add users for administration.

# adduser <user>

Set the password for the user. Generate a new password for the user and save it in the password vault.

# usermod -aG sudo <user>

Disable the firewall.

# ufw disable

Change the default ssh configuration.

# nano /etc/ssh/sshd_config

Change the port, passwordauthentication (dont forget to add ssh key first), disable root login. Store the port in the password vault. When done, restart the service.

# service sshd restart

Log out. Confirm the new port is active. Try to log in as root and confirm that fails. Log in with your privileged account. If desired, go ahaed with Ssh_keypair_setup.

Firewall setup

Install the required prerequisites.

$ sudo apt-get install -y openssl
$ sudo apt install perl -y
$ sudo apt-get install libwww-perl liblwp-protocol-https-perl libgd-graph-perl libio-socket-ssl-perl libcrypt-ssleay-perl libnet-libidn-perl libio-socket-inet6-perl libsocket6-perl curl nginx apache2-utils snapd -y

Download the alternative to the default ubuntu firewall and install it.

$ cd /usr/src
$ sudo wget https://download.configserver.com/csf.tgz
$ sudo tar -xzf csf.tgz
$ cd csf
$ sudo sh install.sh

Confirm the new firewall default configuration is working.

$ sudo perl /usr/local/csf/bin/csftest.pl

Stop and disable the default firewall.

$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld

Make customizations the csf configuration.

$ sudo nano /etc/csf/csf.conf

Be sure to set the following:

• Testing to 0 (set to 1 if needing further testing)
• RESTRICT_SYSLOG=3
• If it is a jisti server, add the UDP port 10000 to "UDP_IN"
• SYSLOG_CHECK = "1800"
• LF_ALERT_TO = "" # this needs to be set to shared notifications email address
• LF_POP3D = "1"
• LF_IMAPD = "1"
• LF_WEBMIN = "1" # not enabled in other system, but confirm web facing instance is restricted to certain IPs
• UI = "1"
• UI_PORT = "xxxx" # this should be unique for each server and stored in the password vault
• UI_USER = "<user>" # this should be unique for each server and stored in the password vault
• UI_PASS = "<password>" # this should be unique for each server and stored in the password vault

Save the file. Next, configure the UI to only allow specific IPs.

$ nano /etc/csf/ui/ui.allow

Save the file with the list of allowed IP addresses, one on each line. Next, start the service.

$ sudo service lfd restart

This may kill your ssh session. Log back in. Set the hostname to the client instance.

$ sudo nano /etc/hostname

Monitoring setup

Download the kickstart script.

$ cd 
$ wget https://my-netdata.io/kickstart.sh

Run the script.

$ sudo bash kickstart.sh --disable-telemetry

While the application is installing, configure a DNS record for stat.domain.com

nginx web server

Create a simple authentication.

$ sudo htpasswd -c /etc/nginx/.htpasswd3 server

When prompted, enter the password "status". Next, Set the configuration file for this instance.

$ sudo nano /etc/nginx/sites-available/stat.<instance>

Use the previously known good configuration:

upstream backend {
   # the Netdata server
    server 127.0.0.1:19997;
#    server unix:/var/run/netdata/netdata.sock;
    keepalive 64;
}

server {

    allow 75.157.219.105;
    allow 209.205.88.122;
    allow 127.0.0.1;
    deny  all;

    # nginx listens to this

    # the virtual host name of this
    server_name stat.domain.ext;

    location / {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_set_header Connection "keep-alive";
        proxy_store off;


        auth_basic "Restricted Content";
        auth_basic_user_file /etc/nginx/.htpasswd3;
        

    }

    listen 80;
}

Create the symlink to the sites-enabled.

$ sudo ln -s /etc/nginx/sites-available/stat.<instance> /etc/nginx/sites-enabled/stat.instance

Customze the netdata installation configuration.

$ sudo nano /etc/netdata/netdata.conf

• Increase the value of "history" to 604800 (7 days).
• Delay the update to 5 seconds.
• Change the default port to 19997.
• Add other recommended defaults as follows:

[global]
    run as user = netdata

    # the default database size - 1 hour
    history = 604800
    update every = 5

    # some defaults to run netdata with least priority
    process scheduling policy = idle
    OOM score = 1000
    access log = none

[web]
        default port = 19997
        bind to = 127.0.0.1

Restart the netdata service.

$ sudo systemctl restart netdata

Make some networking security changes

$ sudo nano /etc/sysctl.conf

Enable spoof protection and make some network security changes.

#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3

##############################################################3
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1


###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
net.ipv4.conf.all.log_martians = 1

net.ipv4.icmp_ignore_bogus_error_responses = 1


net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Block SYN attacks
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5

#

###################################################################
# Magic system request Key
# 0=disable, 1=enable all, >1 bitmask of sysrq functions
# See https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html
# for what other values do
#kernel.sysrq=438

# Ignore Directed pings
net.ipv4.icmp_echo_ignore_all = 1

fs.file-max = 100000
net.core.somaxconn = 65535
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 3240000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_slow_start_after_idle = 0

Change what is shown when people login

$ sudo nano /etc/issue.net

===============================================================
This system is for the use of authorized users only.

Your activity is being monitored. This login attempt and
your identifying information have been recorded.

Anyone using this system expressly consents to such monitoring
and is advised that if there is evidence of criminal activity,
said evidence will be provided to law enforcement officials.
===============================================================

Install certbot

$ sudo snap install core; sudo snap refresh core
$ sudo snap install --classic certbot
$ sudo ln -s /snap/bin/certbot /usr/bin/certbot
$ sudo certbot --nginx

For the email address, enter status@librti.com and follow the prompts. Certbot should make changes to the nginx configuration files and issue the certificates.

Install postfix

Postfix is required for server level email functionality.

$ sudo apt install mailutils postfix

In the menu that appears, choose "Internet Site". Select the appropriate FQDN. When done, edit the postfix main.cf.

$ sudo nano /etc/postfix/main.cf

Set the following:

inet_interfaces = loopback-only

Restart postfix

$ sudo systemctl restart postfix

command line test

$ echo "This is the body of the email" | mail -s "This is the subject line" your_email_address

wordpress mail test

Drop this file into a web enabled directory on the server and run it.

<?php
// Set $to as the email you want to send the test to.
$to = "romtim@pm.me";


// Email subject and body text.
$subject = 'wp_mail function test';
$message = 'This is a test of the wp_mail function: wp_mail is working';
$headers = '';

// Load WP components, no themes.
define('WP_USE_THEMES', false);
require('wp-load.php');

// send test message using wp_mail function.
$sent_message = wp_mail( $to, $subject, $message, $headers );

//display message based on the result.
if ( $sent_message ) {
// The message was sent.
echo 'The test message was sent. Check your email inbox.';
} else {
// The message was not sent.
echo 'The message was not sent!';
}

?>

Backups

Create the backup folder and filesystem.

$ sudo mkdir -p /opt/librti/{backups,log,s3,ssl}
$ sudo chown -R root:sudo /opt/librti
$ sudo chmod -R 775 /opt/librti/

Generate an ecryption key. Be sure to save it in the password vault for this server.

$ cd /opt/librti/ssl
$ openssl rand -hex 32 > enc.key

Change to the s3 directory.

$ cd /opt/librti/s3

Retrieve and save the access and security keys, and bucket names. Be sure to save them in the password vault for this server.

$ nano acc.key
$ nano bucket
$ nano qbucket
$ nano sec.key

Install the s3 client.

$ sudo apt-get install s3cmd

Connect to the Wasabi storage.

$ s3cmd --configure

When done, copy the s3cfg file to root.

$ sudo cp ~/.s3cfg /root/

Configure the backup file list. Replace example.com with the FQDN of the server. Comment out not applicable sections as appropriate.

$ sudo nano /opt/librti/filelist.txt

File: /opt/librti/filelist.txt

# file list for this server

# system
/etc/ssh/sshd_config
/etc/hostname
/etc/sysctl.conf
/etc/issue.net
/etc/hosts

# firewall
/etc/csf/csf.conf
/etc/csf/ui/ui.allow

# monitoring
/etc/netdata/netdata.conf

# nginx
/etc/nginx

# jitsi
/etc/systemd/system.conf
/etc/jitsi/meet/example.com-config.js
/usr/share/jitsi-meet/interface_config.js
/usr/bin/jitsicustomizations
/usr/share/jitsi-meet/css/all.css
/usr/share/jitsi-meet/libs/app.bundle.min.js
/etc/prosody/conf.avail/example.com.cfg.lua
/etc/jitsi/jicofo/jicofo.conf

# mysql 
#/opt/librti/backups/mysql.sql.gz

# nextcloud
/var/www/nextcloud/config
/var/www/nextcloud/data

# php
/etc/php

# certificates
/etc/letsencrypt

If using mysql, configure the mysqlbackup file.

$ sudo nano /opt/librti/backupmysql

File: /opt/librti/backupmysql

#!/bin/sh
datum=`/bin/date +%Y%m%d-%H`
ENCRYPTIONKEY=`cat /opt/librti/ssl/enc.key`
/usr/bin/mysqldump --user=root --lock-all-tables --all-databases | \
gzip -9 - | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.sql.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

$ sudo chmod +x /opt/librti/backupmysql

Configure the script that deleted files older than X days from the S3 storage.

$ sudo nano /opt/librti/deleteOldS3

File: /opt/librti/deleteOldS3

#!/bin/bash

# Usage: ./deleteOld "bucketname" "30 days"

s3cmd ls s3://$1 | while read -r line;
  do
    createDate=`echo $line|awk {'print $1" "$2'}`
    createDate=`date -d"$createDate" +%s`
    olderThan=`date -d"-$2" +%s`
    if [[ $createDate -lt $olderThan ]]
      then 
        fileName=`echo $line|awk {'print $4'}`
        echo $fileName
        if [[ $fileName != "" ]]
          then
            s3cmd del "$fileName"
        fi
    fi
  done;

Make the file executable

$ sudo chmod +x /opt/librti/deleteOldS3

Configure the master backup file. Comment out sections as appropriate based on the server role.

$ sudo nano /opt/librti/librtibackup

File: /opt/librti/librtibackup

#!/bin/bash

# variables
datum=`/bin/date +%Y%m%d-%H`
ENCRYPTIONKEY=`cat /opt/librti/ssl/enc.key`
s3bucket=`cat /opt/librti/s3/bucket`

# back up, compress and encrypt mysql
/usr/bin/mysqldump --user=root --lock-all-tables --all-databases | \
gzip -9 - | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.sql.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# back up, compress and encrypt file list
tar czvpf - --exclude='^#' --verbatim-files-from -T /opt/librti/filelist.txt | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# remove backup files older than 30 days
find /opt/librti/backups/* -mtime +30 -exec rm {} \;

# do the same from the s3 storage older than 60 days
bash /opt/librti/deleteOldS3 "$s3bucket" "60 days"

# shuttle the files to s3 storage
s3cmd sync --acl-private --recursive --skip-existing --multipart-chunk-size-mb=256 /opt/librti/backups/ s3://$s3bucket

Make the file executable

$ sudo chmod +x /opt/librti/librtibackup

Create a cron job to run the backups. General backups can run weekly. Websites can be backed up more frequently, etc.

$ sudo crontab -e

Syntax to run at 8:30 PM Tuesdays (right before maintenance window):

0 20 * * 2 /opt/librti/librtibackup >> /opt/librti/log/librti.log 2>&1

During the next maintenance window, check and confirm that the backup

• generated a logfile
• created encrypted backup files in /opt/librti/backups
• shuttled those files to the s3 storage
• you can download the backups locally and decrypt them

Configure the quarterly backup file.

$ sudo nano /opt/librti/quarterlybackup

File: /opt/librti/quarterlybackup

#!/bin/bash

# variables
datum=`/bin/date +%Y%m%d-%H`
ENCRYPTIONKEY=`cat /opt/librti/ssl/enc.key`
s3bucket=`cat /opt/librti/s3/qbucket`

# back up, compress and encrypt mysql
/usr/bin/mysqldump --user=root --lock-all-tables --all-databases | \
gzip -9 - | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.sql.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# back up, compress and encrypt file list
tar czvpf - --exclude='^#' --verbatim-files-from -T /opt/librti/filelist.txt | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# shuttle the files to s3 storage
s3cmd sync --acl-private --multipart-chunk-size-mb=256 /opt/librti/backups/$HOSTNAME-${datum}*.librti s3://$s3bucket

# remove the backup files created
rm /opt/librti/backups/$HOSTNAME-${datum}.sql.librti /opt/librti/backups/$HOSTNAME-${datum}.librti

Make the file executable

$ sudo chmod +x /opt/librti/quarterlybackup

Create a cron job to run the quarterly backups.

$ sudo crontab -e

50 20 15 */3 * /opt/librti/quarterlybackup >> /opt/librti/log/librti.log 2>&1

Optional: database only backups (webservers only)

Create the backup folder.

$ sudo mkdir -p /opt/librti/dbbackups

File: /opt/librti/librtibackup-dbonly

$ sudo nano /opt/librti/librtibackup-dbonly

#!/bin/bash

# variables
datum=`/bin/date +%Y%m%d-%H`
ENCRYPTIONKEY=`cat /opt/librti/ssl/enc.key`
s3bucket=`cat /opt/librti/s3/bucket`

# back up, compress and encrypt mysql
/usr/bin/mysqldump --user=root --lock-all-tables --all-databases | \
gzip -9 - | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/dbbackups/$HOSTNAME-${datum}.sql.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# remove backup files older than 30 days
find /opt/librti/dbbackups/* -mtime +30 -exec rm {} \;

# todo: remove files on the s3 older than 30 days

# shuttle the files to s3 storage
s3cmd sync --acl-private --recursive --skip-existing --multipart-chunk-size-mb=256 /opt/librti/dbbackups/ s3://$s3bucket

Set the file as executable.

$ sudo chmod +x /opt/librti/librtibackup-dbonly

Configure the cron. Modify the filesystem backup to once a week as required.

$ sudo crontab -e

21 17 * * 2 /opt/librti/librtibackup >> /opt/librti/log/librti.log 2>&1
21 17 * * 1,3,4,5,6,7 /opt/librti/librtibackup-dbonly >> /opt/librti/log/librti.log 2>&1

Revisit

Get csf webmin up and running with remote