How to Set Up SSH on Linux: Keys, Authentication, and Server Security

SSH, or Secure Shell, is the standard protocol for remote server access — and knowing how to set up SSH correctly is one of the most important skills for any Linux administrator. Whether you are managing a single VPS or a fleet of production machines, the difference between a secure connection and a compromised server often comes down to how carefully you configured SSH from the start.

This guide covers the full setup process: installing the SSH server, generating cryptographic keys, enabling key-based authentication, and applying the hardening steps that most tutorials skip entirely.

Installing and Starting the SSH Server

On Debian and Ubuntu systems, the SSH daemon is provided by the openssh-server package. Install it with:

sudo apt update && sudo apt install openssh-server

On RHEL-based systems, the package name is openssh-server as well, but the package manager differs:

sudo dnf install openssh-server

After installation, start and enable the service so it survives reboots:

sudo systemctl enable --now ssh

To verify it is running, use systemctl status ssh. A green “active (running)” line confirms the daemon is ready to accept connections.

What Are SSH Keys and How SSH Keys Work

SSH keys are cryptographic key pairs — a private key that stays on your local machine and a public key that gets placed on the remote server. When you connect, the server issues a challenge that only the holder of the matching private key can answer, making password guessing attacks completely irrelevant.

The most common algorithm today is Ed25519. It produces shorter keys than RSA while offering equivalent or better security:

ssh-keygen -t ed25519 -C "[email protected]"

The command prompts for a file path (the default ~/.ssh/id_ed25519 is fine for most users) and a passphrase. Always set a passphrase — it is the last line of defense if your private key file is ever stolen.

The Anatomy of a Key Pair

After generation, you will have two files. The file without an extension is your private key — treat it like a password and never share it. The .pub file is the public key, and you can freely copy it to any server you want to access.

The public key is a single line of text beginning with ssh-ed25519 followed by a base64-encoded string and your comment. It belongs in the ~/.ssh/authorized_keys file on the remote server.

Copying Your Public Key to the Server

The cleanest way to deploy a public key is with ssh-copy-id:

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server_ip

This appends the key to the server’s authorized_keys file and sets the correct permissions automatically. If ssh-copy-id is not available, copy the key manually by appending the public key content to ~/.ssh/authorized_keys on the server, then set permissions with chmod 600 ~/.ssh/authorized_keys.

Enabling SSH Key Authentication and Disabling Passwords

With the key in place, open the SSH daemon configuration file:

sudo nano /etc/ssh/sshd_config

Locate and set these three directives:

Directive	Recommended Value	Purpose
PubkeyAuthentication	yes	Enable key-based login
PasswordAuthentication	no	Block password login attempts
PermitRootLogin	no	Prevent direct root access
AuthorizedKeysFile	.ssh/authorized_keys	Specify key file location

After making changes, reload the daemon without dropping existing connections:

sudo systemctl reload ssh

Before logging out, open a second terminal and verify you can still log in using your key. Locking yourself out of a server by prematurely disabling password authentication is a common and entirely avoidable mistake.

Linux Server Hardening: Beyond the Basics

Disabling password authentication is the most impactful hardening step, but secure Linux server configuration goes further. The following directives belong in your sshd_config as well.

Limiting Access by User and Port

Changing the default SSH port from 22 reduces automated scan noise, though it does not stop a determined attacker. More meaningful is restricting which users can connect at all:

AllowUsers deploy admin
Port 2222

You can also restrict login to a specific group by using AllowGroups instead. Combined with a firewall rule that only allows connections from known IP ranges, this shrinks your attack surface considerably.

Pair this with automatic blocking of repeated failed attempts using fail2ban:

sudo apt install fail2ban
sudo systemctl enable --now fail2ban

The default fail2ban configuration monitors /var/log/auth.log and bans IPs that trigger five failed login attempts within ten minutes.

Tightening Idle Session Limits

Sessions left open indefinitely are a liability. Set these values in sshd_config:

ClientAliveInterval 300
ClientAliveCountMax 2

This sends a keepalive packet every 300 seconds and terminates the session if the client fails to respond twice — effectively disconnecting idle sessions after about ten minutes.

Managing files over SSH pairs naturally with other Linux automation tasks. If you are already using shell scripts for remote administration, the approach described in the guide to how to rename files in Linux translates directly to remote sessions once your SSH connection is secure.

Configuring the SSH Client for Multiple Servers

Most guides focus entirely on the server side. The client-side ~/.ssh/config file is just as important for day-to-day usability. It lets you define aliases, specify which key to use per host, and set connection options without typing them every time:

Host myserver
    HostName 203.0.113.10
    User deploy
    IdentityFile ~/.ssh/id_ed25519
    Port 2222

With this configuration saved, connecting is as simple as ssh myserver. The config file supports multiple Host blocks, making it the right way to manage access to dozens of servers cleanly.

If you are operating a lightweight or resource-constrained environment, pairing a hardened SSH setup with one of the best lightweight Linux distros gives you a lean, secure server foundation without excess overhead.

Verifying Your SSH Configuration Is Secure

Run the sshd test mode before every restart to catch syntax errors:

sudo sshd -t

A clean output (no messages) means the configuration file is valid. For a more comprehensive audit, tools like ssh-audit scan your server from the outside and flag weak algorithms, outdated key exchange methods, and misconfigured settings.

Check the authentication log in real time to confirm key-based logins are working as expected:

sudo tail -f /var/log/auth.log

A successful key login shows “Accepted publickey” in the log. A failed password attempt after disabling PasswordAuthentication shows “Failed password” — meaning the setting is active and blocking the attempt correctly.

Making SSH Part of a Larger Security Strategy

SSH configuration is not a one-time task. As your server evolves, revisit sshd_config after major OS upgrades, review authorized_keys files to remove stale entries, and rotate key pairs on a regular schedule. Key rotation is especially important when team members leave or when a machine that stored private keys is decommissioned.

Two-factor authentication adds another layer on top of key-based login using packages like libpam-google-authenticator. It requires a valid key plus a time-based one-time password, making unauthorized access nearly impossible even if a private key is somehow compromised.

A properly configured SSH setup is the foundation of every other remote administration task. Get it right once, document the configuration, and your servers will be significantly harder to compromise than the majority of internet-facing Linux machines.