Welcome to the UCCNetsoc SysAdmin Handbook

This handbook is a collection of useful information for SysAdmins and others who are interested in the UCC Netsoc infrastructure. It goes over the topics of networking, security, and administration of our services.

Command Line

The command line is a vital tool in any SysAdmin's toolkit. It is vital to have a good understanding of how to work with the command line to adminstrate a system and develop supporting software for it.

• Files & Directories
• Permissions
• Git
• SSH & SCP

Files & Directories

In linux, everything is a file or directory. All files and directories have a path. There is a relative and a full path, with relative being a path relative to the current working directory.

To see your current working directory, run the pwd (Print Working Directory) command.

$ pwd
/home/sysadmin

Listing Files and Directories

The ls (LiSt) command lists the contents of a directory. If no arguments are specified, it lists the contents of the current directory.

$ ls
development
Documents
Downloads
Pictures

$ ls Pictures
picture-of-a-cat.png
picture-of-a-dog.jpg

Files that begin with a . are "hidden" files, and do not normally appear the output of an ls command. To show all files, use the -a (All) flag.

$ ls -a
.bash_history
.bash_logout
.bash_profile
.bashrc
.profile
.vimrc
development
Documents
Downloads
Pictures

Changing Your Working Directory

The cd command changes your working directory to the path you specify. Paths can be relative or absolute.

Absolute Paths

Absolute paths are paths that begin from / (root).

$ cd /home/sysadmin/Documents
$ # or
$ cd ~/Documents

Note: The ~ (tilde) character is a shortcut for the current user's home directory.

Relative Paths

Relative paths are paths relative to the current working directory.

The simplest relative path is a child directory of the current working directory.

$ ls
Documents
Pictures

$ cd Documents
$ ls
Guide.pdf
Resources.txt

. is the current directory.

$ pwd 
/home/sysadmin
$ cd .
$ pwd
/home/sysadmin

.. is the parent directory.

$ pwd 
/home/sysadmin/Pictures
$ cd ..
$ pwd
/home/sysadmin

And more complex...

$ pwd
/home/sysadmin/Pictures
$ cd ../Documents
$ pwd
/home/sysadmin/Documents

Copying and Moving Files & Directories

Copying

Use the cp (CoPy) command to copy files and directories.

$ cp <source> <destination>

To copy a directory AND its contents, use the -r (Recursive) flag.

$ cp -r <source directory> <destination directory>

Moving

Moving files and directories works much the same with the mv (MoVe) command.

$ mv <source> <destination>

You can also use the mv command to rename a file or directory

$ mv <oldname> <newname>

Links

A link works like a file to point from one place to another. There are two types of links:

Hard Links

A hard link is a file that points to the inode of another file. If this "pointed-to" file is deleted, the data persists until all hard links are removed (a normal file acts as a hard link to itself). A hard link can only be created for files, not directories.

Create a hard link with the ln (LiNk) command.

ln <source> <destination>

$ cat srcFile
hello i am a file.
$ ln srcFile hardLink
$ rm srcFile
$ cat hardLink
hello i am a file.

In the permissions section, we look at the ls -l command. We can see the number of hard links to a file in the output.

$ ls -l
-rw-rw-r--  1 sysadmin  users  0    Dec  1  2021    srcFile
-rw-r--r--  1 sysadmin  users  584  Mar  10 2022    script.sh

The number before sysadmin is the number of hard links.

$ ln srcFile hardLink
$ ls -l
-rw-rw-r--  2 sysadmin  users  0    Dec  1  2021    hardlink
-rw-rw-r--  2 sysadmin  users  0    Dec  1  2021    srcFile
-rw-r--r--  1 sysadmin  users  584  Mar  10 2022    script.sh

As we can see here, after adding a hard link, the number of hard links increases for the source file, and the number is reflected in the hard link too.

Soft/Symbolic Link

A symbolic link (symlink for short) is a file that points to another file or directory's path. If the "pointed-to" file or directory is deleted, the link is no longer valid, and points to a non-existent file or directory.

The -s (Soft) flag specifies a symbolic link.

To create a link called newLink to a file called srcFile, run the following command:

$ ln -sf srcFile newLink

$ cat srcFile
this is the source file
$ ln -sf srcFile newLink
$ cat newLink
this is the source file

The cat (conCATenate) command allows you to read the contents of a file.

Permissions

Each file and directory have a set of permissions associated with them. These permissions can be seen in the format of drwxrwxrwx. The first character is to specify if the resource is a directory, the next three characters are for the owner, the second three characters are for the group, and the third three characters are for others.

To see the permissions, owner and group of a file or directory run the ls -l command.

$ ls -l
drwxr-xr-x  2 sysadmin  users  4096 May  18  2022 Documents
drwxr-xr-x  2 sysadmin  users  4096 May  20  2022 Downloads
drwxr-xr-x  2 sysadmin  users  4096 May  20  2022 Pictures
-rw-r--r--  1 sysadmin  users  584  Mar  10  2022 README.md
-rw-r--r--  1 sysadmin  users  584  Mar  10  2022 script.sh

The permisssions are in the leftmost column.

To take Documents for example, it is a directory, where the owner (sysadmin) has read, write and execute permissions (execute permissions on a directory means you can cd into it). The users group has permission to read and execute the directory (not allowed to write or delete it). Every other user outside of that group has read and execute permissions. However, the parent directory of Documents must allow the user to execute into it, before allowing the user to read or execute into the Documents directory

Changing Permissions of a file

Changing the Owner & Group

To change the owner, use the chown (CHange OWNer) command.

$ chown <user>:<group> <file or directory>

Changing the Group

To change the group, use the chgrp (CHange GRouP) command.

$ chgrp <group> <file or directory>

Changing RWX Permissinos

To change permission modifiers of a file or directory, use the chmod (CHange MODifiers) command.

The first flag specifies the person(s) to change permissions for.

The second flag specifies what to do with the third parameter

The third set of flags specifies the permission modifier to apply.

For example, to change the permission of the file cat.png to allow everyone in the group to read:

$ chmod g+r cat.png

Git

Git is a VCS (Version Control System) that operates by calculating diffs (differeneces between files) between two versions of a project.

To begin using git, install the git command line tool. Git is often preinstalled on most linux distributions, and OSX, for windows or exceptions, see instructions at git-scm.org.

To begin using Netsoc-as-Code, you must first add an SSH key to the setup-control-host.yml file. Instructions on how to generate an SSH key and can be found here, and instructions on how to add the public key to Github can be found here.

Git Basic Training

To create a project with Git, you must first create a directory, and initialize it as a git repository.

$ mkdir <project>
$ cd <project>
$ git init

If you have created a repository on Github, you can set the origin to the Github repository.

$ git remote add origin git@github.com:<your username>/<your project>.git

Alternatively, to begin working on a pre-existing project, we first have to clone the project. As you should have already set up an SSH key, you can use SSH to authenticate yourself to Github.

$ git clone git@github.com:<org or user>/<project name>.git

To commit (save) changes, you must first add changed files to the staging area.

$ git add <file or directory name>

and you can use git status to see what files are staged for commit.

$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes staged for commit:
    modified: newfile.txt

It is a good idea to use git status to make sure you've added all of the needed files, but also to ensure you don't commit something you shouldn't like a secret file (like a .env).

Now, finally to commit changes staged:

$ git commit -m "<message briefly explaining changes>"

And to push to the remote repository:

$ git push origin <branch name>

How we develop code

We use git for every project we develop. A feature of Github that we take advantage of is Pull Requests. Pull Requests allow us to merge changes from one version of a project into another. This allows us to branch off from the 'main' branch and work on a new feature, before creating a Pull Request to merge the new feature into the main branch.

Git Branching

The git checkout command allows us to switch between and create new branches.

To go to <branch name>

git checkout <branch name>

To create and go to a new branch

git checkout -b <new branch name>

After pushing commits and finishing work on a branch, you can open a Pull Request on Github to merge in your changes, by going to the 'Pull Requests' tab on the repository page on Github and selecting your branch.

SSH & SCP

SSH (Secure SHell) is a way of accessing the command line (or cli for short) of any linux machine connected to a network. You need to either have a user and password on the remote machine, or a public key in the ~/.ssh/authorized_keys file on the remote machine.

Your user and the authorized_keys file gets populated with your public key once you add it to the NaC repo, as outlined in the git handbook and an already existing SysAdmin runs the playbook.

Using Remote SSH with VSCode

There is a Visual Studio Code extension that allows you to edit files on a remote machine and open terminals etc. It is very handy for editing and running Ansible playbooks from the control host.

• Install the extension "Remote SSH"

• If prompted, create the config file in .ssh/config

• Create a new SSH Target with the following:

  ssh <username you added in setup-control-host.yml>@control.netsoc.co -p 2222

• When prompted, select Linux as the remote OS

Then, you can connect & login with your Github Account to clone Netsoc as Code and to author and push commits.

Contact a SysAdmin to provide you with ansible SSH keys to run playbooks on the infrastructure machines.

.ssh/config

Your SSH targets will be written to this file. If you need to create more complex SSH target configs (for example using a Jump Host), you can open this file directly and add the necessary configuration.

On eduroam, it is likely that your SSH connection to control.netsoc.co will be blocked, as it is not using port 22. To fix this, you can add a Jump Host similar to the following in your .ssh/config file.

Host csgate
    HostName csgate.ucc.ie
    User <your csgate username>
    Port 22

Host control.netsoc.co
    HostName control.netsoc.co
    User <your control host username>
    Port 2222
    ProxyJump csgate

Note: This config sets it so that you first need to SSH into csgate, requiring your csgate password, before then SSHing to control. Be careful not to use the wrong password for either csgate or your SSH key on control.

Logging in with username & password

$ ssh <username>@<hostname> -p <port>

port is generally going to be 22 and will normally not need the -p flag

Logging in with private key

$ ssh -i <path to private key> <username>@<hostname> -p <port>

path to private key will generally be ~/.ssh/id_<signing method>. Port rules apply here too

SCP (Secure CoPy)

SCP uses SSH to copy files from one machine to another.

To copy from local machine to remote:

$ scp -P 22 <source file> <username>@<hostname>:<destination relative to users home>

To copy from remote machine to local:

$ scp -P 22 <username>@<hostname>:<source relative to users home> <destination file>

To copy from one remote host to another remote host:

scp -P 22 <username>@<source hostname>:<source> <username>@<destination hostname>:<destination>

To copy a directory with all its contents, use the -r (Recursive) flag immediately after scp.

Disks & Partitions

A partition is a section of a disk, which can be mounted to your machine. A disk can, and often does have multiple partitions.

For example, the following is a typical partition layout

lsblk -k

# <device>        <dir>        <type>        <options>        <dump> <fsck>
/dev/sda1         /boot        vfat          defaults         0      2
/dev/sda2         /            ext4          defaults         0      1
/dev/sda3         /home        ext4          defaults         0      2
/dev/sda4         none         swap          defaults         0      0

/dev/sda is the disk, such as a solid state drive or hard drive, and each device with a suffix of a number refers to a partition on that disk.

• The <dir> portion refers to the mountpoint of that partition. So from this, all data in your home folder /home/<YOUR_USERNAME> will be stored on the /dev/sda3 partition.
• The <type> portion refers to the filesystem type. So from this, you can see that the /dev/sda3 partition is an ext4 filesystem, which is a commonly used filesystem for general storage on linux.
• The <options> portion refers to the filesystem options. Options can include mounting the filesystem as readonly (ro), or as readwrite (rw).
• The <dump> and <fsck> portions are used to determine if the filesystem is in a good state on boot/mount.

The "swap" partition is of a special type that acts as virtual memory or as "swap space", allowing the disk to store data when your RAM isn't enough.

Mounting a partition

Generally, unless you have a specific reason not to, partitions are often mounted somewhere from the /mnt directory.

mount /dev/sda3 /mnt/service_data

If you want this mount to be permanent, you can add it to your /etc/fstab file.

UUID=<UUID> /mnt/service_data ext4 rw,relatime,data=ordered 0 0

Use blkid to find the UUID of the partition you want to mount.

Disks & Partitions

A partition is a section of a disk, which can be mounted to your machine. A disk can, and often does have multiple partitions.

For example, the following is a typical partition layout

lsblk -k

# <device>        <dir>        <type>        <options>        <dump> <fsck>
/dev/sda1         /boot        vfat          defaults         0      2
/dev/sda2         /            ext4          defaults         0      1
/dev/sda3         /home        ext4          defaults         0      2
/dev/sda4         none         swap          defaults         0      0

/dev/sda is the disk, such as a solid state drive or hard drive, and each device with a suffix of a number refers to a partition on that disk.

• The <dir> portion refers to the mountpoint of that partition. So from this, all data in your home folder /home/<YOUR_USERNAME> will be stored on the /dev/sda3 partition.
• The <type> portion refers to the filesystem type. So from this, you can see that the /dev/sda3 partition is an ext4 filesystem, which is a commonly used filesystem for general storage on linux.
• The <options> portion refers to the filesystem options. Options can include mounting the filesystem as readonly (ro), or as readwrite (rw).
• The <dump> and <fsck> portions are used to determine if the filesystem is in a good state on boot/mount.

The "swap" partition is of a special type that acts as virtual memory or as "swap space", allowing the disk to store data when your RAM isn't enough.

Mounting a partition

Generally, unless you have a specific reason not to, partitions are often mounted somewhere from the /mnt directory.

mount /dev/sda3 /mnt/service_data

If you want this mount to be permanent, you can add it to your /etc/fstab file.

UUID=<UUID> /mnt/service_data ext4 rw,relatime,data=ordered 0 0

Use blkid to find the UUID of the partition you want to mount.

ZFS

ZFS is a Linux-based, user-space filesystem that provides a simple, flexible, and powerful way to manage and manage ZFS pools. On Netsoc infra, we use ZFS to more easily set up disks. ZFS also uses an Arc Cache to speed up the process of reading and writing data to and from the disk. While this uses a lot of memory (on Scalper, it is ~8GB), it provides read speed benefits.

ZFS Pool

A ZFS pool is a collection of ZFS virtual devices (vdevs) that appear as a single storage device accessible to the filesystem.

ZFS Mirror

A ZFS mirror is a setup where 2 or more disks are used to store the same data. It allows for good IO performance (due to being able to read from two disks at the same time) and is also a good way to protect your data from failure (as to lose any data, all disks need to fail).

VLANs

A VLAN is a layer of network abstraction that allows you to isolate a network from the rest of the network. At Netsoc, we use this to stop traffic from going places it shouldn't, and to create different subnets for different uses.

Our VLAN definitions can be seen on Nac here.

We use dot1q to tag our VLANs with a VLAN ID, to ensure no traffic crosses VLANs except for the ones we want.

VLAN Tagging

Our VyOS router uses the dot1q VLAN tagging protocol to tag our VLANs. This means that each ethernet (OSI Model Layer 2) packet has a VLAN ID header field inserted, that is supported by the dot1q protocol. This packet is then sent to every machine on that VLAN.

For simplicity, we match a VLAN ID with a subnet. This allows us to easily keep track of which VLANs are on which IP address range.

Notice that the cloud VLAN has /16 subnet mask, so as to be able to support up to 65k IPs, so that Netsoc Cloud IPs can grow (almost) indefinitely.

Web request scenario

Say, for instance a tcp request is sent to Netsoc's Web IP address 84.39.234.53:

VyOS will catch this packet, and looks through its NAT (Network Address Translation) table to find the correct internal IP address (in this case the web VM's IP address) to send the packet to. This can be found here on NaC.

managerN in the config on NaC is a placeholder. In the applied config, the placeholder is replaced by web whose definition is stored in ansible vars/network.yml here.

VLAN Trunking

VLAN trunking is when a machine can send packets to multiple VLANs. At Netsoc, all Proxmox hosts have a trunk connection (not connected to WAN for obvious reasons - we don't want to send arbitrary traffic to the outside world accidentally).

VyOS Router

Be VERY careful when editing the VyOS router config, as it can break all network traffic in Netsoc Cloud.
Contact Oisin Canty (ocanty on GitHub) if you are going to be editing the VyOS router.

VLANs

A VLAN is a layer of network abstraction that allows you to isolate a network from the rest of the network. At Netsoc, we use this to stop traffic from going places it shouldn't, and to create different subnets for different uses.

Our VLAN definitions can be seen on Nac here.

We use dot1q to tag our VLANs with a VLAN ID, to ensure no traffic crosses VLANs except for the ones we want.

VLAN Tagging

Our VyOS router uses the dot1q VLAN tagging protocol to tag our VLANs. This means that each ethernet (OSI Model Layer 2) packet has a VLAN ID header field inserted, that is supported by the dot1q protocol. This packet is then sent to every machine on that VLAN.

For simplicity, we match a VLAN ID with a subnet. This allows us to easily keep track of which VLANs are on which IP address range.

Notice that the cloud VLAN has /16 subnet mask, so as to be able to support up to 65k IPs, so that Netsoc Cloud IPs can grow (almost) indefinitely.

Web request scenario

Say, for instance a tcp request is sent to Netsoc's Web IP address 84.39.234.53:

VyOS will catch this packet, and looks through its NAT (Network Address Translation) table to find the correct internal IP address (in this case the web VM's IP address) to send the packet to. This can be found here on NaC.

managerN in the config on NaC is a placeholder. In the applied config, the placeholder is replaced by web whose definition is stored in ansible vars/network.yml here.

VLAN Trunking

VLAN trunking is when a machine can send packets to multiple VLANs. At Netsoc, all Proxmox hosts have a trunk connection (not connected to WAN for obvious reasons - we don't want to send arbitrary traffic to the outside world accidentally).

VyOS Router

Be VERY careful when editing the VyOS router config, as it can break all network traffic in Netsoc Cloud.
Contact Oisin Canty (ocanty on GitHub) if you are going to be editing the VyOS router.

VLANs

A VLAN is a layer of network abstraction that allows you to isolate a network from the rest of the network. At Netsoc, we use this to stop traffic from going places it shouldn't, and to create different subnets for different uses.

Our VLAN definitions can be seen on Nac here.

We use dot1q to tag our VLANs with a VLAN ID, to ensure no traffic crosses VLANs except for the ones we want.

VLAN Tagging

Our VyOS router uses the dot1q VLAN tagging protocol to tag our VLANs. This means that each ethernet (OSI Model Layer 2) packet has a VLAN ID header field inserted, that is supported by the dot1q protocol. This packet is then sent to every machine on that VLAN.

For simplicity, we match a VLAN ID with a subnet. This allows us to easily keep track of which VLANs are on which IP address range.

Notice that the cloud VLAN has /16 subnet mask, so as to be able to support up to 65k IPs, so that Netsoc Cloud IPs can grow (almost) indefinitely.

Web request scenario

Say, for instance a tcp request is sent to Netsoc's Web IP address 84.39.234.53:

VyOS will catch this packet, and looks through its NAT (Network Address Translation) table to find the correct internal IP address (in this case the web VM's IP address) to send the packet to. This can be found here on NaC.

managerN in the config on NaC is a placeholder. In the applied config, the placeholder is replaced by web whose definition is stored in ansible vars/network.yml here.

VLAN Trunking

VLAN trunking is when a machine can send packets to multiple VLANs. At Netsoc, all Proxmox hosts have a trunk connection (not connected to WAN for obvious reasons - we don't want to send arbitrary traffic to the outside world accidentally).

VyOS Router

Be VERY careful when editing the VyOS router config, as it can break all network traffic in Netsoc Cloud.
Contact Oisin Canty (ocanty on GitHub) if you are going to be editing the VyOS router.

External IPs

At Netsoc, we have access to a number of IP addresses.

* Used be used for mapping NAT ports to Proxmox hosts & VMs.

Important Internal IPs

These IPs can be verified or changed on NaC.

Switch Config

pants#show vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi0/2, Gi0/4, Gi0/5, Gi0/6, Gi0/7, Gi0/8, Gi0/9, Gi0/10, Gi0/11, Gi0/12, Gi0/15, Gi0/16, Gi0/17
                                                Gi0/18, Gi0/25, Gi0/26, Gi0/27, Gi0/28
10   wan                              active    Gi0/1, Gi0/19, Gi0/20, Gi0/21, Gi0/22, Gi0/23, Gi0/24
20   proxmox                          active
30   infra                            active
40   cloud                            active
70   router                           active
80   oob                              active    Gi0/3
90   mgmt                             active
1002 fddi-default                     act/unsup
1003 token-ring-default               act/unsup
1004 fddinet-default                  act/unsup
1005 trnet-default                    act/unsup

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1    enet  100001     1500  -      -      -        -    -        0      0
10   enet  100010     1500  -      -      -        -    -        0      0
20   enet  100020     1500  -      -      -        -    -        0      0
30   enet  100030     1500  -      -      -        -    -        0      0
40   enet  100040     1500  -      -      -        -    -        0      0
70   enet  100070     1500  -      -      -        -    -        0      0
80   enet  100080     1500  -      -      -        -    -        0      0
90   enet  100090     1500  -      -      -        -    -        0      0
1002 fddi  101002     1500  -      -      -        -    -        0      0
1003 tr    101003     1500  -      -      -        -    -        0      0
1004 fdnet 101004     1500  -      -      -        ieee -        0      0
1005 trnet 101005     1500  -      -      -        ibm  -        0      0

Remote SPAN VLANs
------------------------------------------------------------------------------


Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

Subnets

A subnet is a segment of an IP address space.

A subnet looks like the following:

Subnet Mask

The mask in this sense is like multiplying by 1 (aka this bit exists)

So, in essence the first 3 bytes from the top IP address will always be the same, however, the subnet mask has no effect on the last byte.

From the CIDR notation, note the /24 at the end, this signifies 24 bits for the network prefix of the IP address, leaving 8 bits for the host identifier (32bits - 24bits).

In simpler terms, you could give a machine an ip address anywhere between 10.20.0.0 and 10.20.0.255.

Servers

Over the years, Netsoc have had a number of servers with varying capabilities and uses.

Previously, some of servers were named after famous scientists or mathematicians (Boole, Lovelace, Feynman), but this tradition fell off with the introduction of Netsoc Cloud.

Servers

Over the years, Netsoc have had a number of servers with varying capabilities and uses.

Previously, some of servers were named after famous scientists or mathematicians (Boole, Lovelace, Feynman), but this tradition fell off with the introduction of Netsoc Cloud.

Containerization