Gr0x0rd at 06:08, 3 February 2023

2023-02-03T06:08:42Z

Gr0x0rd: Created page with "= Initial Setup = Determine the IP address of the new server and ssh into it as the root user. This information should be provided by the hosting company that provisioned the..."

2023-02-03T06:06:50Z

Created page with "= Initial Setup = Determine the IP address of the new server and ssh into it as the root user. This information should be provided by the hosting company that provisioned the..."

New page

= Initial Setup =

Determine the IP address of the new server and ssh into it as the root user. This information should be provided by the hosting company that provisioned the server.
$ ssh -l root -p <ip address>

Set a new root password. Use a password generator to create a complex password. Save the password in the vault and paste it to the session.
# passwd root

Apply all system updates.
# sudo apt update
# sudo apt upgrade

== User setup and authentication ==

Add users for administration.
# adduser <user>
Set the password for the user. Generate a new password for the user and save it in the password vault.
# usermod -aG sudo <user>
Disable the firewall.
# ufw disable
Change the default ssh configuration.
# nano /etc/ssh/sshd_config
Change the port, passwordauthentication (dont forget to add ssh key first), disable root login. Store the port in the password vault. When done, restart the service.
# service sshd restart
Log out. Confirm the new port is active. Try to log in as root and confirm that fails. Log in with your privileged account. If desired, go ahaed with [[Ssh_keypair_setup]].

= Firewall setup =
Install the required prerequisites.
$ sudo apt-get install -y openssl
$ sudo apt install perl -y
$ sudo apt-get install libwww-perl liblwp-protocol-https-perl libgd-graph-perl libio-socket-ssl-perl libcrypt-ssleay-perl libnet-libidn-perl libio-socket-inet6-perl libsocket6-perl curl nginx apache2-utils snapd -y

Download the alternative to the default ubuntu firewall and install it.
$ cd /usr/src
$ sudo wget https://download.configserver.com/csf.tgz
$ sudo tar -xzf csf.tgz
$ cd csf
$ sudo sh install.sh
Confirm the new firewall default configuration is working.
$ sudo perl /usr/local/csf/bin/csftest.pl
Stop and disable the default firewall.
$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld
Make customizations the csf configuration.
$ sudo nano /etc/csf/csf.conf
Be sure to set the following:
* Testing to 0 (set to 1 if needing further testing)
* RESTRICT_SYSLOG=3
* If it is a jisti server, add the UDP port 10000 to "UDP_IN"
* SYSLOG_CHECK = "1800"
* LF_ALERT_TO = "" # this needs to be set to shared notifications email address
* LF_POP3D = "1"
* LF_IMAPD = "1"
* LF_WEBMIN = "1" # not enabled in other system, but confirm web facing instance is restricted to certain IPs
* UI = "1"
* UI_PORT = "xxxx" # this should be unique for each server and stored in the password vault
* UI_USER = "<user>" # this should be unique for each server and stored in the password vault
* UI_PASS = "<password>" # this should be unique for each server and stored in the password vault
Save the file. Next, configure the UI to only allow specific IPs.
$ nano /etc/csf/ui/ui.allow
Save the file with the list of allowed IP addresses, one on each line. Next, start the service.
$ sudo service lfd restart
This may kill your ssh session. Log back in. Set the hostname to the client instance.
$ sudo nano /etc/hostname

= Monitoring setup =
Download the kickstart script.
$ cd
$ wget https://my-netdata.io/kickstart.sh
Run the script.
$ sudo bash kickstart.sh --disable-telemetry
While the application is installing, configure a DNS record for stat.domain.com

== nginx web server ==
Create a simple authentication.
$ sudo htpasswd -c /etc/nginx/.htpasswd3 server
When prompted, enter the password "status". Next, Set the configuration file for this instance.
$ sudo nano /etc/nginx/sites-available/stat.<instance>
Use the previously known good configuration:
<pre>
upstream backend {
# the Netdata server
server 127.0.0.1:19997;
# server unix:/var/run/netdata/netdata.sock;
keepalive 64;
}

server {

allow 75.157.219.105;
allow 209.205.88.122;
allow 127.0.0.1;
deny all;

# nginx listens to this

# the virtual host name of this
server_name stat.domain.ext;

location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_pass_request_headers on;
proxy_set_header Connection "keep-alive";
proxy_store off;

auth_basic "Restricted Content";
auth_basic_user_file /etc/nginx/.htpasswd3;

}

listen 80;
}
</pre>
Create the symlink to the sites-enabled.
$ sudo ln -s /etc/nginx/sites-available/stat.<instance> /etc/nginx/sites-enabled/stat.instance

Customze the netdata installation configuration.
$ sudo nano /etc/netdata/netdata.conf
* Increase the value of "history" to 604800 (7 days).
* Delay the update to 5 seconds.
* Change the default port to 19997.
* Add other recommended defaults as follows:
<pre>
[global]
run as user = netdata

# the default database size - 1 hour
history = 604800
update every = 5

# some defaults to run netdata with least priority
process scheduling policy = idle
OOM score = 1000
access log = none

[web]
default port = 19997
bind to = 127.0.0.1
</pre>
Restart the netdata service.
$ sudo systemctl restart netdata

== Make some networking security changes ==
$ sudo nano /etc/sysctl.conf
Enable spoof protection and make some network security changes.
<pre>
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3

##############################################################3
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
# Enabling this option disables Stateless Address Autoconfiguration
# based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1

###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
net.ipv4.conf.all.log_martians = 1

net.ipv4.icmp_ignore_bogus_error_responses = 1

net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Block SYN attacks
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5

#

###################################################################
# Magic system request Key
# 0=disable, 1=enable all, >1 bitmask of sysrq functions
# See https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html
# for what other values do
#kernel.sysrq=438

# Ignore Directed pings
net.ipv4.icmp_echo_ignore_all = 1

fs.file-max = 100000
net.core.somaxconn = 65535
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 3240000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_slow_start_after_idle = 0
</pre>

== Change what is shown when people login ==
$ sudo nano /etc/issue.net
<pre>
===============================================================
This system is for the use of authorized users only.

Your activity is being monitored. This login attempt and
your identifying information have been recorded.

Anyone using this system expressly consents to such monitoring
and is advised that if there is evidence of criminal activity,
said evidence will be provided to law enforcement officials.
===============================================================
</pre>

= Install certbot =
$ sudo snap install core; sudo snap refresh core
$ sudo snap install --classic certbot
$ sudo ln -s /snap/bin/certbot /usr/bin/certbot
$ sudo certbot --nginx
For the email address, enter '''status@librti.com''' and follow the prompts. Certbot should make changes to the nginx configuration files and issue the certificates.
= Install postfix =
Postfix is required for server level email functionality.
$ sudo apt install mailutils postfix
In the menu that appears, choose "Internet Site". Select the appropriate FQDN. When done, edit the postfix main.cf.
$ sudo nano /etc/postfix/main.cf
Set the following:
<pre>
inet_interfaces = loopback-only
</pre>
Restart postfix
$ sudo systemctl restart postfix
== command line test ==
$ echo "This is the body of the email" | mail -s "This is the subject line" your_email_address
== wordpress mail test ==
Drop this file into a web enabled directory on the server and run it.
<pre>

<?php
// Set $to as the email you want to send the test to.
$to = "romtim@pm.me";

// Email subject and body text.
$subject = 'wp_mail function test';
$message = 'This is a test of the wp_mail function: wp_mail is working';
$headers = '';

// Load WP components, no themes.
define('WP_USE_THEMES', false);
require('wp-load.php');

// send test message using wp_mail function.
$sent_message = wp_mail( $to, $subject, $message, $headers );

//display message based on the result.
if ( $sent_message ) {
// The message was sent.
echo 'The test message was sent. Check your email inbox.';
} else {
// The message was not sent.
echo 'The message was not sent!';
}

?>

</pre>

= Backups =

Create the backup folder and filesystem.
$ sudo mkdir -p /opt/librti/{backups,log,s3,ssl}
$ sudo chown -R root:sudo /opt/librti
$ sudo chmod -R 775 /opt/librti/
Generate an ecryption key. Be sure to save it in the password vault for this server.
$ cd /opt/librti/ssl
$ openssl rand -hex 32 > enc.key
Change to the s3 directory.
$ cd /opt/librti/s3
Retrieve and save the access and security keys, and bucket names. Be sure to save them in the password vault for this server.
$ nano acc.key
$ nano bucket
$ nano qbucket
$ nano sec.key
Install the s3 client.
$ sudo apt-get install s3cmd
[https://wiki.librtisolutions.com/index.php?title=Backups#Connecting_to_Wasabi_storage Connect to the Wasabi storage].
$ s3cmd --configure
When done, copy the s3cfg file to root.
$ sudo cp ~/.s3cfg /root/
Configure the backup file list. Replace example.com with the FQDN of the server. Comment out not applicable sections as appropriate.
$ sudo nano /opt/librti/filelist.txt
File: '''/opt/librti/filelist.txt'''
<pre>
# file list for this server

# system
/etc/ssh/sshd_config
/etc/hostname
/etc/sysctl.conf
/etc/issue.net
/etc/hosts

# firewall
/etc/csf/csf.conf
/etc/csf/ui/ui.allow

# monitoring
/etc/netdata/netdata.conf

# nginx
/etc/nginx

# jitsi
/etc/systemd/system.conf
/etc/jitsi/meet/example.com-config.js
/usr/share/jitsi-meet/interface_config.js
/usr/bin/jitsicustomizations
/usr/share/jitsi-meet/css/all.css
/usr/share/jitsi-meet/libs/app.bundle.min.js
/etc/prosody/conf.avail/example.com.cfg.lua
/etc/jitsi/jicofo/jicofo.conf

# mysql
#/opt/librti/backups/mysql.sql.gz

# nextcloud
/var/www/nextcloud/config
/var/www/nextcloud/data

# php
/etc/php

# certificates
/etc/letsencrypt
</pre>
If using mysql, configure the mysqlbackup file.
$ sudo nano /opt/librti/backupmysql
File: '''/opt/librti/backupmysql'''
<pre>
#!/bin/sh
datum=`/bin/date +%Y%m%d-%H`
ENCRYPTIONKEY=`cat /opt/librti/ssl/enc.key`
/usr/bin/mysqldump --user=root --lock-all-tables --all-databases | \
gzip -9 - | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.sql.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -
</pre>
$ sudo chmod +x /opt/librti/backupmysql
Configure the script that deleted files older than X days from the S3 storage.
$ sudo nano /opt/librti/deleteOldS3
File: '''/opt/librti/deleteOldS3'''
<pre>
#!/bin/bash

# Usage: ./deleteOld "bucketname" "30 days"

s3cmd ls s3://$1 | while read -r line;
do
createDate=`echo $line|awk {'print $1" "$2'}`
createDate=`date -d"$createDate" +%s`
olderThan=`date -d"-$2" +%s`
if [[ $createDate -lt $olderThan ]]
then
fileName=`echo $line|awk {'print $4'}`
echo $fileName
if [[ $fileName != "" ]]
then
s3cmd del "$fileName"
fi
fi
done;
</pre>
Make the file executable
$ sudo chmod +x /opt/librti/deleteOldS3
Configure the master backup file. Comment out sections as appropriate based on the server role.
$ sudo nano /opt/librti/librtibackup
File: '''/opt/librti/librtibackup'''
<pre>
#!/bin/bash

# variables
datum=`/bin/date +%Y%m%d-%H`
ENCRYPTIONKEY=`cat /opt/librti/ssl/enc.key`
s3bucket=`cat /opt/librti/s3/bucket`

# back up, compress and encrypt mysql
/usr/bin/mysqldump --user=root --lock-all-tables --all-databases | \
gzip -9 - | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.sql.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# back up, compress and encrypt file list
tar czvpf - --exclude='^#' --verbatim-files-from -T /opt/librti/filelist.txt | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# remove backup files older than 30 days
find /opt/librti/backups/* -mtime +30 -exec rm {} \;

# do the same from the s3 storage older than 60 days
bash /opt/librti/deleteOldS3 "$s3bucket" "60 days"

# shuttle the files to s3 storage
s3cmd sync --acl-private --recursive --skip-existing --multipart-chunk-size-mb=256 /opt/librti/backups/ s3://$s3bucket
</pre>
Make the file executable
$ sudo chmod +x /opt/librti/librtibackup
Create a cron job to run the backups. General backups can run weekly. Websites can be backed up more frequently, etc.
$ sudo crontab -e
Syntax to run at 8:30 PM Tuesdays (right before maintenance window):
<pre>
0 20 * * 2 /opt/librti/librtibackup >> /opt/librti/log/librti.log 2>&1
</pre>
During the next maintenance window, check and confirm that the backup
* generated a logfile
* created encrypted backup files in /opt/librti/backups
* shuttled those files to the s3 storage
* you can download the backups locally and decrypt them

Configure the quarterly backup file.
$ sudo nano /opt/librti/quarterlybackup
File: '''/opt/librti/quarterlybackup'''
<pre>
#!/bin/bash

# variables
datum=`/bin/date +%Y%m%d-%H`
ENCRYPTIONKEY=`cat /opt/librti/ssl/enc.key`
s3bucket=`cat /opt/librti/s3/qbucket`

# back up, compress and encrypt mysql
/usr/bin/mysqldump --user=root --lock-all-tables --all-databases | \
gzip -9 - | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.sql.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# back up, compress and encrypt file list
tar czvpf - --exclude='^#' --verbatim-files-from -T /opt/librti/filelist.txt | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/backups/$HOSTNAME-${datum}.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# shuttle the files to s3 storage
s3cmd sync --acl-private --multipart-chunk-size-mb=256 /opt/librti/backups/$HOSTNAME-${datum}*.librti s3://$s3bucket

# remove the backup files created
rm /opt/librti/backups/$HOSTNAME-${datum}.sql.librti /opt/librti/backups/$HOSTNAME-${datum}.librti
</pre>
Make the file executable
$ sudo chmod +x /opt/librti/quarterlybackup
Create a cron job to run the quarterly backups.
$ sudo crontab -e
<pre>
50 20 15 */3 * /opt/librti/quarterlybackup >> /opt/librti/log/librti.log 2>&1
</pre>

== Optional: database only backups (webservers only) ==
Create the backup folder.
$ sudo mkdir -p /opt/librti/dbbackups
File: '''/opt/librti/librtibackup-dbonly'''
$ sudo nano /opt/librti/librtibackup-dbonly
<pre>
#!/bin/bash

# variables
datum=`/bin/date +%Y%m%d-%H`
ENCRYPTIONKEY=`cat /opt/librti/ssl/enc.key`
s3bucket=`cat /opt/librti/s3/bucket`

# back up, compress and encrypt mysql
/usr/bin/mysqldump --user=root --lock-all-tables --all-databases | \
gzip -9 - | \
gpg -c --batch --yes -z 0 --cipher-algo AES256 -o /opt/librti/dbbackups/$HOSTNAME-${datum}.sql.librti --passphrase-fd 9s 9< <(printf '%s' "$ENCRYPTIONKEY") -

# remove backup files older than 30 days
find /opt/librti/dbbackups/* -mtime +30 -exec rm {} \;

# todo: remove files on the s3 older than 30 days

# shuttle the files to s3 storage
s3cmd sync --acl-private --recursive --skip-existing --multipart-chunk-size-mb=256 /opt/librti/dbbackups/ s3://$s3bucket
</pre>
Set the file as executable.
$ sudo chmod +x /opt/librti/librtibackup-dbonly
Configure the cron. Modify the filesystem backup to once a week as required.
$ sudo crontab -e
<pre>
21 17 * * 2 /opt/librti/librtibackup >> /opt/librti/log/librti.log 2>&1
21 17 * * 1,3,4,5,6,7 /opt/librti/librtibackup-dbonly >> /opt/librti/log/librti.log 2>&1
</pre>
== Revisit ==

Get csf webmin up and running with remote

← Older revision		Revision as of 06:08, 3 February 2023
Line 558:		Line 558:

	Get csf webmin up and running with remote		Get csf webmin up and running with remote
		+
		+	=Increasing disk space=
		+
		+	Not all hosting providers have the ability to increase the storage on a disk and apply it to the appropriate partition (typically root). As such the following process is required.
		+
		+	==Confirm the additional space==
		+	Check the status of the system partitions using
		+	$ sudo df -h
		+	Check the type of block device and file system using
		+	$ sudo lsblk
		+	Run fdisk to examine the status of the disk that's been extended
		+	$ sudo fdisk /dev/sdx
		+	(where x is the virtual disk that's been increased, typically a).
		+	<br>
		+	Likely you will note a warning that there is a GPT PMBR size mismatch, and this will be corrected upon write.
		+	<br><br>
		+	Make a note of the sector that the partition that is to be extended begins at.
		+
		+	==Delete the partition to be extended==
		+	Command (m for help): p
		+	View the partitions so you know which one to delete.
		+	Command (m for help): d
		+	Enter the correct partition to be deleted.
		+
		+	==Recreate the partition==
		+	Command (m for help): n
		+	Partition number: this should be the same as the partition you deleted.
		+	First sector: this needs to match the first sector of the partition you deleted.
		+	Last sector: this can be customized, or just go with default to use all space.
		+	If using LVM, it will mention a signature was found. DO NOT remove the signature.
		+	Command (m for help): w
		+	This will sync the disks and exit fdisk.
		+
		+	==Extend the filesystem==
		+
		+	===Optional: LVM===
		+
		+	If the system is using LVM, the volume will need to be extended. Determine the group name
		+	$ sudo vgdisplay
		+	Note the name. Extend the volume
		+	$ sudo lvextend -L +XG /dev/mapper/<volume_name>
		+	(where X is the size in GB to extend the volume).
		+	<bR><br>
		+	To extend the filesystem, run the resize2fs command.
		+	$ sudo resize2fs /dev/<device or partition>
		+
		+	==Update the kernel==
		+	LVM should update automatically. If not using LVM, update the kernel manually.
		+	$ sudo partprobe
		+	Confirm the new storage has been added successfully:
		+	$ sudo df -h

Ubuntu VPS install - Revision history

Gr0x0rd at 06:08, 3 February 2023

Gr0x0rd: Created page with "= Initial Setup = Determine the IP address of the new server and ssh into it as the root user. This information should be provided by the hosting company that provisioned the..."