Category: Networking and Internet

Introducing Netshoot: A Powerful Network Troubleshooting Tool for Docker

2023/08/21 No Comments

Reading time: 20 – 34 minutes

Networking issues can be a real headache, especially when dealing with containerized applications. Whether it’s latency, routing problems, DNS resolution, firewall issues, or incomplete ARPs, network problems can significantly degrade application performance. Fortunately, there’s a powerful tool that can help you troubleshoot and resolve these issues: netshoot.

What is Netshoot?

Netshoot is a Docker container equipped with a comprehensive set of networking troubleshooting tools. It’s designed to help you diagnose and fix Docker and Kubernetes networking issues. With a proper understanding of how Docker and Kubernetes networking works and the right tools, you can troubleshoot and resolve these networking issues more effectively.

Understanding Network Namespaces

Before diving into the usage of netshoot, it’s essential to understand a key concept: Network Namespaces. Network namespaces provide isolation of the system resources associated with networking. Docker uses network and other types of namespaces (pid,mount,user, etc.) to create an isolated environment for each container. Everything from interfaces, routes, and IPs is completely isolated within the network namespace of the container.

The cool thing about namespaces is that you can switch between them. You can enter a different container’s network namespace, perform some troubleshooting on its network stack with tools that aren’t even installed on that container. Additionally, netshoot can be used to troubleshoot the host itself by using the host’s network namespace. This allows you to perform any troubleshooting without installing any new packages directly on the host or your application’s package.

Using Netshoot with Docker

Container’s Network Namespace

If you’re having networking issues with your application’s container, you can launch netshoot with that container’s network namespace like this:

$ sudo docker run -it --net container:<container_name> nicolaka/netshoot

Host’s Network Namespace

If you think the networking issue is on the host itself, you can launch netshoot with that host’s network namespace:

$ sudo docker run -it --net host nicolaka/netshoot

Network’s Network Namespace

If you want to troubleshoot a Docker network, you can enter the network’s namespace using nsenter. This is explained in the nsenter section below.

Using Netshoot with Docker Compose

You can easily deploy netshoot using Docker Compose using something like this:

version: "3.6"
services:
  tcpdump:
    image: nicolaka/netshoot
    depends_on:
      - nginx
    command: tcpdump -i eth0 -w /data/nginx.pcap
    network_mode: service:nginx
    volumes:
      - $PWD/data:/data

  nginx:
    image: nginx:alpine
    ports:
      - 80:80

Included Packages

Netshoot includes a wide range of powerful tools for network troubleshooting. Here’s a list of the included packages along with a brief description of each:

apache2-utils: Utilities for web server benchmarking and server status monitoring.
bash: A popular Unix shell.
bind-tools: Tools for querying DNS servers.
bird: Internet routing daemon.
bridge-utils: Utilities for configuring the Linux Ethernet bridge.
busybox-extras: Provides several stripped-down Unix tools in a single executable.
conntrack-tools: Tools for managing connection tracking records.
curl: Tool for transferring data with URL syntax.
dhcping: Tool to send DHCP requests to DHCP servers.
drill: Tool similar to dig.
ethtool: Tool for displaying and changing NIC settings.
file: Tool to determine the type of a file.
fping: Tool to ping multiple hosts.
grpcurl: Command-line tool for interacting with gRPC servers.
iftop: Displays bandwidth usage on an interface.
iperf: Tool for measuring TCP and UDP bandwidth performance.
iperf3: A newer version of iperf.
iproute2: Collection of utilities for controlling TCP/IP networking.
ipset: Tool to manage IP sets.
iptables: User-space utility program for configuring the IP packet filter rules.
iptraf-ng: Network monitoring tool.
iputils: Set of small useful utilities for Linux networking.
ipvsadm: Utility to administer the IP Virtual Server services.
jq: Lightweight and flexible command-line JSON processor.
libc6-compat: Compatibility libraries for glibc.
liboping: C library to generate ICMP echo requests.
ltrace: A library call tracer.
mtr: Network diagnostic tool.
net-snmp-tools: Set of SNMP management tools.
netcat-openbsd: Networking tool known as the “Swiss army knife” of networking.
nftables: Successor to iptables.
ngrep: Network packet analyzer.
nmap: Network exploration tool and security scanner.
nmap-nping: Packet generation and response analysis tool.
nmap-scripts: Scripts for nmap.
openssl: Toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols.
py3-pip: Package installer for Python.
py3-setuptools: Python Distutils Enhancements.
scapy: Packet manipulation tool.
socat: Relay for bidirectional data transfer.
speedtest-cli: Command-line interface for testing internet bandwidth.
openssh: OpenSSH client and server.
strace: System call tracer.
tcpdump: Packet analyzer.
tcptraceroute: Traceroute implementation using TCP packets.
tshark: Network protocol analyzer.
util-linux: Miscellaneous system utilities.
vim: Highly configurable text editor.
git: Distributed version control system.
zsh: Unix shell.
websocat: Simple WebSocket client.
swaks: Swiss Army Knife for SMTP.
perl-crypt-ssleay: Perl module for OpenSSL.
perl-net-ssleay: Perl module for using OpenSSL.

With this extensive set of tools, netshoot is a powerful ally in diagnosing and resolving network issues in your Docker and Kubernetes environments. Whether you’re dealing with latency, routing problems, DNS resolution, firewall issues, or incomplete ARPs, netshoot has the tools you need to troubleshoot and fix these issues.

If you’re interested in trying out netshoot for yourself, you can find the project on GitHub at https://github.com/nicolaka/netshoot. It’s a powerful tool that can help you troubleshoot and resolve network issues in your Docker and Kubernetes environments.

Serving Static Files with Docker and Darkhttpd

2023/08/18 No Comments

Reading time: 12 – 20 minutes

In this blog post, we’ll explore how to use Docker and the lightweight HTTP server, Darkhttpd, to serve static files. This setup is particularly useful when you need a simple web server for sharing files or hosting a static website. We’ll also discuss how to use a reverse proxy like Traefik to route external traffic to the Darkhttpd service.

Docker Compose Configuration

Below is the docker-compose.yml file that defines the Darkhttpd service:

version: '3.3'
services:
  darkhttpd:
    image: p3terx/darkhttpd
    container_name: darkhttpd
    restart: unless-stopped
    volumes:
      - './site:/www:ro'
    entrypoint: ["/darkhttpd","/www"]
    networks:
      your_network:
        ipv4_address: your_ipv4_address
networks:
  your_network:
    external:
      name: your_network_name

Here’s a brief overview of the configuration:

The image field specifies the Docker image to use for the service.
The container_name field sets the name of the container.
The restart field configures the restart policy for the container.
The volumes field defines the volume mounts for the service.
The entrypoint field overrides the default entrypoint of the image.
The networks field specifies the networks that the service is connected to.

Setting Up the Service

Create a directory named site in the same directory as the docker-compose.yml file. Place the static files you want to serve in this directory.
Replace your_network, your_ipv4_address, and your_network_name in the docker-compose.yml file with the appropriate values for your setup.
Run the following command to start the Darkhttpd service:

docker-compose up -d

Access the static files by navigating to the IP address specified in the docker-compose.yml file.

Using a Reverse Proxy

To route external traffic to the Darkhttpd service, you can use a reverse proxy like Traefik. Configure the reverse proxy to forward requests to the IP address specified in the docker-compose.yml file.

Conclusion

Using Docker and Darkhttpd to serve static files is a simple and efficient solution for sharing files or hosting a static website. By adding a reverse proxy, you can easily route external traffic to the Darkhttpd service. This setup is ideal for scenarios where you need a lightweight web server without the overhead of a full-fledged web server like Apache or Nginx.

Enhancing SSH Security with StealthSSHAccess

2023/08/17 No Comments

Reading time: 4 – 7 minutes

In today’s interconnected world, maintaining the security of your server infrastructure is paramount. One critical point of vulnerability is the SSH (Secure Shell) service, which allows remote administration of servers. Despite using a non-default port, many administrators still find their servers bombarded with brute-force and denial-of-service attacks. To address this challenge, I’ve developed a solution called StealthSSHAccess.

The Problem

Attackers often employ brute force attacks to gain unauthorized access to servers via SSH. Even if you’ve changed the default SSH port, determined attackers can still discover the new port and target it. These attacks can lead to service disruption, unauthorized data access, and potential breaches of sensitive information.

The Solution: StealthSSHAccess

StealthSSHAccess is an innovative approach to managing remote SSH access while mitigating the risks associated with brute-force attacks. Let’s delve into how it works and why it’s an effective solution:

Dynamic Access Control

StealthSSHAccess takes a dynamic and personalized approach to SSH access control. It operates as a smart gateway between potential attackers and your SSH service. Here’s a simplified breakdown of how it functions:

Monitoring for Intent: Instead of directly exposing the SSH port, StealthSSHAccess monitors a non-SSH TCP port for connection attempts. Attackers, unaware of this, can’t target the SSH port directly.
Capture and Response: When an attempt is made on the monitored port, StealthSSHAccess captures the IP address of the requester. This initial connection attempt fails, serving as a signal of intent to access SSH.
Secure Access Window: Based on this signal, StealthSSHAccess temporarily opens the SSH port exclusively for the captured IP address. This allows for a secure connection from that specific source.
Time-Bound Access: Access is granted for a predetermined duration. If SSH access isn’t established within this timeframe, the port is automatically closed for that specific IP. This tightens the window of exposure and bolsters security.
Automatic Closure: If the port remains unused during the allowed time, StealthSSHAccess automatically revokes access and closes the port. A continuous monitoring mechanism controls this process.

Benefits and Features

1. Enhanced Security: By hiding the SSH port from attackers, StealthSSHAccess reduces the attack surface and minimizes exposure to potential threats.

2. Selective Accessibility: With StealthSSHAccess, you control who gains access by simply attempting a connection to a specific port. This provides an additional layer of security.

3. Minimal Configuration: Implementing StealthSSHAccess is easy thanks to its Docker-based deployment. This means you can integrate it seamlessly into your existing system.

4. Persistence Across Restarts: StealthSSHAccess ensures continuity by persisting IP timer information across service interruptions or restarts. This keeps the system aware of pending access requests.

Getting Started with StealthSSHAccess

To deploy StealthSSHAccess, follow these steps:

Requirements: Ensure you have Docker and Docker Compose installed.
Configuration: Set up environment variables using the provided .env file. Customize parameters like LOGLEVEL, IFACE, PORT_TO_MONITOR, and more to match your environment.
Building and Running: Build the images using docker-compose build, and then launch the services with docker-compose up -d.
Data Persistence: IP timer data is stored in the ./data directory, so make sure it’s writable by the Docker user.
Security Note: Be aware that these services run with privileged access due to their interaction with the system’s network configuration. Understand the security implications before deployment.

Conclusion

In the ongoing battle against cybersecurity threats, StealthSSHAccess stands as a beacon of innovative protection for your servers. By intelligently managing SSH access and responding dynamically to legitimate requests, this solution offers heightened security without sacrificing convenience. Whether you’re an administrator or a security-conscious user, consider integrating StealthSSHAccess into your infrastructure to safeguard your servers from the persistent threats of the digital landscape.

To explore the project, access the source code, and learn more about its implementation, visit the StealthSSHAccess GitHub repository. Remember, security is a journey, and with StealthSSHAccess, you’re taking a proactive step toward a more resilient and secure server environment.

Deploying gotop with Ansible

2022/12/14 No Comments

Reading time: 5 – 8 minutes

Gotop is a terminal based graphical activity monitor inspired by gtop and vtop; it’s available at:

https://github.com/xxxserxxx/gotop/

I published a role in Ansible Galaxy for deploying gotop in Linux servers. The role page in Ansible Galaxy is at:

https://galaxy.ansible.com/oriolrius/install_gotop

Role installation command and deployment command:

ansible-galaxy install oriolrius.install_gotop

# change SERVER_IP, for the IP address where you want to deploy gotop
ansible -i SERVER_IP, -u root -m include_role -a name=oriolrius.install_gotop all

Get the IP addresses of local Docker containers

2022/11/16 No Comments

Reading time: 13 – 21 minutes

We have Docker running with containers that are connected to their own private network. To efficiently manage and monitor these containers, it’s often useful to retrieve their private IP addresses.

With the following command, you can easily obtain the private IP addresses of all running Docker containers:

sudo docker inspect $(docker ps -q) --format='{{ printf "%-50s" .Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' | sort -t. -k2,2n -k3,3n -k4,4n

Output example:

$ sudo docker inspect $(docker ps -q ) --format='{{ printf "%-50s" .Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' | sort -t. -k2,2n -k3,3n -k4,4n
/rproxy                                            10.3.10.2
/n8n                                               10.3.10.4
/semaphore                                         10.3.10.6
/code                                              10.3.10.7
/ssh                                               10.3.10.9
/nodered                                           10.3.10.11
/pihole_opendns                                    10.3.10.23
/pihole_googledns                                  10.3.10.24

OpenSSH public key fingerprint

2022/10/25 No Comments

Reading time: 8 – 14 minutes

Quick and easy, how to get the fingerprint of your SSH RSA key.

# syntax:
openssl pkey -in PATH/PRIVATE_RSA_KEY -pubout -outform DER | openssl md5 -c

# example:
$ openssl pkey -in ~/.ssh/id_rsa -pubout -outform DER | openssl md5 -c
MD5(stdin)= a6:26:23:d9:c1:d3:d5:e5:c0:38:ab:3c:c1:6a:3f:ea

Mikrotik passwordless SSH with public key

2022/10/17 No Comments

Reading time: 18 – 30 minutes

Following the instructions described in the official documentation:

https://wiki.mikrotik.com/wiki/Use_SSH_to_execute_commands_(public/private_key_login)

The process is as always as easy as:

# upload the id_rsa.pub file
# then import the public key file for the user used for connecting via SSH
user ssh-keys import public-key-file=id_rsa.pub user=admin-ssh
# and it's done.

Everything was OK with my WSL Ubuntu 20.04. (I added WSL at the beginning of the versions because it runs in Windows Subsystem Linux).

But, with the newest WSL Ubuntu 22.04 I was unsuccessful.

Being precise, the SSH versions are:

# WSL Ubuntu 20.04
$ ssh -V
OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020

# WSL Ubuntu 22.04
$ ssh -V
OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022

After connecting with verbose details, I found this message, that was the key for solving the problem:

debug1: Offering public key: /home/my_user/.ssh/id_rsa RSA SHA256:2******************************Y agent
debug1: send_pubkey_test: no mutual signature algorithm

Then, I discovered that newest SSH versions aren’t compatible with Mikrotik SSH version. It seems that version isn’t enough newest and are incompatible with how public keys are negotiated at the beginning of the connection.

Finally, the solution was to use an extra parameter for establishing the connection:

ssh -o 'PubkeyAcceptedAlgorithms +ssh-rsa' THE_USER@THE_HOST

Of course, an alternative is using ~/.ssh/config file or the system file: /etc/ssh/ssh_config and add this parameter for everything, or specific hosts. For instance, like this:

Host JUST_A_NAME_OF_THE_CONNECTION
  Hostname THE_IP_ADDRESS_OR_HOSTNAME_OF_THE_TARGET_HOST
  user THE_USER
  PubkeyAcceptedAlgorithms +ssh-rsa

Get the IP address of the WSL2 in Windows 10

2022/09/15 No Comments

Reading time: 6 – 9 minutes

Nothing else than what the title says. Simple PowerShell script for dumping the IP address:

wsl -- ip -o -4 -json addr list eth0 `
| ConvertFrom-Json `
| %{ $_.addr_info.local } `
| ?{ $_ }

socat: publish a port only available in localhost

2022/08/25 No Comments

Reading time: 11 – 18 minutes

Assume that we have a service only available in localhost (127.0.0.1/8) and we want to expose this port temporarily. Of course, you can use iptables for redirecting the port. But take care, this is not a simple DNAT because packets will not be evaluated by PREROUTING (-t nat) rules.

Another option is using an old-powerful Swiss knife tool: socat (github).

# binds public port to any local interface
socat TCP-LISTEN:<public_port>,fork TCP:127.0.0.1:<internal_port>
# binds only to an IP address
SOCAT_SOCKADDR=<interface_IP> socat TCP-LISTEN:<public_port>,fork TCP:127.0.0.1:<internal_port>

# examples:

# binds to all interfaces:
socat TCP-LISTEN:1880,fork TCP:127.0.0.1:1880
# just for an IP address of one interface:
SOCAT_SOCKADDR=10.2.0.110 socat TCP-LISTEN:1880,fork TCP:127.0.0.1:1880

WSL2: upgrade from Ubuntu 18.04 to 20.04

2022/08/24 No Comments

Reading time: 16 – 27 minutes

I was afraid for missing my files, configurations and much other staff that I had in Linux filesystem of WSL2 (Windows 10). But I had to upgrade because of compatibility with an application that I need. So, finally, I decided to upgrade the Ubuntu 18.04 that I installed in WSL2 to a newer version 20.04. Yes, I know that there is 22.04 available nowadays, but I wasn’t comfortable jumping to many versions.

I followed simple steps that I found at:

How To Upgrade Existing WSL/WSL2 Ubuntu 18.04 to 20.04

The steps in a nutshell were:

sudo apt update
sudo apt upgrade -y
sudo apt --purge autoremove
sudo apt install update-manager-core
sudo do-release-upgrade

When I answered all the questions and after stopping the WSL2 VM with:

# command that I ran from 'cmd.exe' (Windows console)
wsl --shutdown

I had and issue rebooting, WSL2 didn’t boot and it gave this message:

wsl/usr/sbin/enter-systemd-namespace: line 10: /usr/sbin/daemonize: No such file or directory

I resolved the problem, shutting down the WSL and running the command:

wsl -u root -e bash --noprofile --norc

It gave me a root console where I could modify the file:

/usr/sbin/enter-systemd-namespace

specifically, in line 10, I changed:

/usr/sbin/daemonize
to
/usr/bin/daemonize

And when I accessed to WSL2 everything worked perfectly. I used to access WSL2 from Windows Terminal which uses the command:

C:\WINDOWS\system32\wsl.exe -d Ubuntu-18.04

But it has small issues with bybou, finally I changed this command to:

C:\WINDOWS\system32\bash.exe ~ -login

I love to use a console with byobu enabled when access the shell, but with my former command, the command ‘byobu-enable’ was ignored. I didn’t find why, in the end the solution was to change the command that I use for accessing Linux console (WSL2).

Likewise, I hope these notes can help someone.