Debugging container network traffic using Wireshark SSH Remote Capture

Intro

Hi all, It has been a while since I’ve posted something on this blog. . I do have some self hosted applications running and docker networking always got to me. With my history in network monitoring I have an urge to see what is going on and what traffic is going through my applications and docker containers. With docker you can create bridged networks to separate applications, which is awesome, but once you cluster containers inside a network and make them talk directly to each other some visibility is lost. Personally I use 4 networks.

Proxy_net (All applications that are/should be externally accessible via NGINX)
Download_net (All applications related to downloading legal stuff, all this traffic is routed through a VPN docker)
Development_net (All things related to development applications such as Gitea and other work in progress scripts)
Database_net (All applications that make use of my MariaDB instance)
Utilities_net (All utilities such as Nextcloud/Paperless etc.)

The main reason I created these separate networks is security, I do not want my Nextcloud to talk to my development tools or downloading applications. In docker-compose you can easily add one or multiple networks to a container.

networks:   
  - proxy_net 
  - database_net

Docker networks also have their own internal DNS resolver, so you can use the container name as hostname for container to container communication. A great example for this is the link between MariaDB and Nextcloud, who both are in the Database_net. Instead of pointing the database host to 192.168.x.x in the Nextcloud config, you can just enter mariadb as database host. Since all the containers in the Database_net can communicate inside their own space, there is no need for exposing ports on the host running docker, which is a good thing, security wise. Internal routing is also very useful for NGINX configs, in fact, the configuration templates for NGINX already use this method.

The following image is a very basic overview of a network. where eth1 is the physical adapter in the machine.

But what if you’re curious what actually happens in that network. Reasons for this could be debugging, troubleshooting or even malware analysis. You could install tcpdump or a variant to capture the traffic, transfer it over to your desktop and start your analysis. This method is pretty cumbersome if you ask me. Redeploying the container will remove tcpdump again, some containers do not have apt installed and a few other reasons can be mentioned why it sucks. So lets try if we can find a lazier way to achieve this 😉

The Setup

So the goal is set, we want to analyze network traffic of a specific docker container, preferably on a machine on another place in the network using WireShark, as I do prefer the WireShark GUI over tcpdump and other cli network analysis methods. We can split this setup in 2 parts, the Docker side of things (on the server), and the Client side of thing (I’m using a Windows Desktop).

Server setup

Lets take it back a small step. On the server side we need the following:

Something to tap into the specific docker containers network traffic
(Some variant of) tcpdump to actually capture the network
Something to ship the traffic information to another place, which is Wireshark on another system

To achieve this we will make use of the dockers ability to share the ‘network’ of a single container with another one. More info on this specific topic can be found in the docker documentation. This can be done by using the --network container:<containername> option when launching a container. This will put the container inside another dockers network space and make them share the same ‘docker IP’. (be aware for port conflicts in the applications used inside a container)

If I was having another container inside the Database_net, and I wanted to connect to the NextCloud instance, I would have to use the mariadb hostname, and all of nextclouds traffic would be routed through the ‘mariadb network’. There are many use-cases for this, such as forcing various containers to connect through a VPN container. In our use-case we will put another container under the MariaDB container network which contains the tools we need for our network capture. Still get it?

For transferring network traffic we will make use of the SSH Protocol. Most docker containers do not come with SSH (enabled by default), so that is something we need to create. Another tool we need for this to work is tcpdump, also a tool that is not common to be shipped in containers.

You can create your own DockerFile to create something that contains both SSH and tcpdump. For the sake of ease, we slightly alter a DockerFile made by the awesome people at linuxserver.io which contains an SSH server with almost everything set up. This container does not come with tcpdump, so we add tcpdump to the apt-get sequence already existing in the original DockerFile.

git clone https://github.com/linuxserver/docker-openssh-server

You can add tcpdump to the apt-get command yourself, or replace the full Dockerfile it with the following contents:

https://gist.github.com/wessel145/031aafcf8a166967dc0ed056cfc50bcf

In addition, tcpdump requires elevated rights to capture network traffic of network interface inside the docker container. Unfortunately, Wireshark is not able to enter a sudo password for the execution of tcpdump. So you have to be very specific with the docker-compose file, use below as reference.

This docker-compose.yml is not final, but good for testing the setup. Make sure you place this file in the cloned repository from linuxserver.io.

version: '3.7'
services:
  openssh-server:
    build:
      # Directory where you cloned the gist to, leave . if this file is in the same directory
      context: .
    container_name: openssh-server
    network_mode: container:mariadb
    environment:
      - SUDO_ACCESS=true
      - USER_NAME=wessel
    volumes:
      - ./ssh-config:/config

Make sure to set your DOCKERCONFIDIR in your .env file, this is a good practice, you can also remove the first three, the configdir you can redirect to any directory, but you will need to be able to access this directory to put in your public key if that is your preferred way of authenticating (I recommended you to do so).

Please refer to the official linuxserver documentation for other key authorization options. Do not use a password, as it will break the rest of the structure and is less safe 🙂

For this tutorial I will use a keypair generated with PuTTYgen in openSSH format and put the public key in ./ssh-config/.ssh/authorized_keys
(Note that the ssh-config folder will be created after the first time running the container)

If everything is set up correctly you probably do not have to test your setup at this phase, if you really like to you can alter the docker-compose file a bit to make it function standalone:

version: '3.7'
services:
  openssh-server:
    build:
      # Directory where you cloned the gist to, leave . if this file is in the same directory
      context: .
    container_name: openssh-server
    # network_mode: container:mariadb
    environment:
      - SUDO_ACCESS=true
      - USER_NAME=wessel
    volumes:
      - ./ssh-config:/config
    # expose port for debug purposes
    ports:
      - 2222:2222

If all is well you should be able to login with your favorite SSH client or any terminal and be able to execute sudo tcpdump without a password prompt.

Later in this post we’ll discuss connecting this ssh container to another container to capture its data, but for now we’ll continue with setting up WireShark.

WireShark setup

On the client side we’ll use a Windows desktop with WireShark => 3.5.0. Feel free to use a Linux distro or other OS if you’d like, important here is the Wireshark version number, the current stable release contains some bugs and will give some errors as it comes with an older version of extcap. Newer builds can be found here.

Alright, assuming you’ve set up the server part, have your ssh key pair ready it is time to fire up WireShark and click the option next to SSH remote capture:

Which will open the following window:

In here we fill in the IP of the server our modified ssh server is running on, and its corresponding port.

Under Authentication we we fill in our username as specified in the environment variable for the docker container. We select our private key (make sure it is in open-ssh format! otherwise you’ll face an error). If your private key is password protected, also enter it in the corresponding field.

WireShark does not remember the passphrase for security reasons, so make sure to fill it in every time you fire up a session

Under capture we set information such as the interface to monitor. Active interfaces can be found with ifconfig.

$ docker exec -it openssh-server ifconfig -a

In my case the interface is eth0 .

Also select use sudo on the remote machine. The filter is prefilled with a basic filter which will filter out traffic between WireShark and the docker container. You can modify this filter to your liking but for now we leave it on default and hit Start. Just do a quick check if port 2222 is excluded.

If everything is set up correctly you should see no traffic flowing, as the container is not doing anything. To force some traffic through the open-ssh container you can execute the following command:

$ docker exec -it openssh-server curl linuxserver.io

Some traffic should pop up now 🙂

Awesome, it works! and we monitor traffic flowing through a docker container on a remote machine!

Connecting to a docker container

The traffic of the docker container itself isn’t that interesting. We want to monitor other ones, so we need a way to bind them. Docker offers a compose option to use the network of another docker container, so they will share the same network interface, which we can capture. I will not go into details on how it exactly works in this blog.

We modify the docker-compose file to (in my case mariadb) connect the two containers, but basically it can be any other container

version: '3.7'
services:
  openssh-server:
    build:
    # Directory where you cloned the gist to, leave . if this file is in the same directory
      context: .
    container_name: openssh-server
    network_mode: container:mariadb
    environment:
      - SUDO_ACCESS=true
      - USER_NAME=wessel
    volumes:
      - ./ssh-config:/config

See how there’s no port 2222 mentioned here? Since we bind the container to the network of the other one, we cannot expose any ports on this one. And there we have our only caveat. We need to expose port 2222 on the container we want to monitor and rebuild it. So for our mariadb docker we need to make sure the port is exposed in the compose file:

ports:
- 2222:2222

We docker-compose up the file where the mariadb container is located and then compose up the openssh-server container we’ve used before. If all is correct, this should return no errors!

Now we fire up a Wireshark connection again (don’t forget the key passphrase, if you have one) and…:

And we see traffic flowing through the mariadb container! In my case we see some statements related to my DSMR meter.

We have reached our goal and are now able to capture traffic between docker containers! Traffic which does not reach the physical interface of the server.

Final words

At first I created this guide for personal usage as I couldn’t find someone explaining this on the internet. I learned a lot about docker networking and am happy to share that with you, and hope you learned a thing or two by reading this blog. Using remote packet capture in docker networks can be useful for debugging but I can also see setups where it can be used for malware analysis, the sky is the limit.

Do you have any feedback or comments? Leave a comment or contact me via twitter: @WesSec_

Wessel’s Blog