1

I'm using podman on RHEL8.6 to deploy a pod composed of two containers, zabbix-server and postgres-server. To deploy the containers, podman play kube is used on a .yaml file to import and run the pod configuration. This is a rootless setup.

The containers need to communicate over an internal network using DNS. I use podman create network mon to create a bridged network called 'mon' that has DNS enabled.

With podman play kube monitoring.yaml --network mon, the containers are not able to communicate with each other over DNS. I've tested the setup without using podman play kube by creating the pod and containers manually. In the manual setup, the containers in the pod are able to find each other using DNS.

manual setup:

podman pod create --name monitoring
podman network create mon
podman run -d --rm --network=mon --pod=monitoring --name postgres-server docker.io/timescale/timescaledb:latest-pg14
podman run -d --rm --network=mon --pod=monitoring --name zabbix-server docker.io/zabbix/zabbix-server-pgsql

mon network config:

[
    {
        "cniVersion": "0.4.0",
        "name": "mon",
        "plugins": [
            {
                "bridge": "cni-podman1",
                "hairpinMode": true,
                "ipMasq": true,
                "ipam": {
                    "ranges": [
                        [
                            {
                                "gateway": "10.89.0.1",
                                "subnet": "10.89.0.0/24"
                            }
                        ]
                    ],
                    "routes": [
                        {
                            "dst": "0.0.0.0/0"
                        }
                    ],
                    "type": "host-local"
                },
                "isGateway": true,
                "type": "bridge"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "portmap"
            },
            {
                "backend": "",
                "type": "firewall"
            },
            {
                "type": "tuning"
            },
            {
                "capabilities": {
                    "aliases": true
                },
                "domainName": "dns.podman",
                "type": "dnsname"
            }
        ]
    }
]

monitoring.yaml file:

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-3.4.7
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2022-05-24T13:40:30Z"
  labels:
    app: monitoring
  name: monitoring
spec:
  containers:
  - args:
    - postgres
    image: docker.io/timescale/timescaledb:latest-pg14
    name: postgres-server
    resources: {}
    securityContext:
      capabilities:
        drop:
        - CAP_MKNOD
        - CAP_NET_RAW
        - CAP_AUDIT_WRITE
  - args:
    - /usr/sbin/zabbix_server
    - --foreground
    - -c
    - /etc/zabbix/zabbix_server.conf
    image: docker.io/zabbix/zabbix-server-pgsql:latest
    name: zabbix-server
    resources: {}
    securityContext:
      capabilities:
        drop:
        - CAP_MKNOD
        - CAP_NET_RAW
        - CAP_AUDIT_WRITE
    volumeMounts:
    - mountPath: /var/lib/zabbix/export
      name: bdc06ccb2b092148bb78c353fccc104255a81a8735eb3a14974f6ede0e6516e8-pvc
    - mountPath: /var/lib/zabbix/snmptraps
      name: 196d7983e1b62995bae07beb7942e18ac46dcf6d3346e43fff2604c9a7bc30ec-pvc
  restartPolicy: Never
  volumes:
  - name: bdc06ccb2b092148bb78c353fccc104255a81a8735eb3a14974f6ede0e6516e8-pvc
    persistentVolumeClaim:
      claimName: bdc06ccb2b092148bb78c353fccc104255a81a8735eb3a14974f6ede0e6516e8
  - name: 196d7983e1b62995bae07beb7942e18ac46dcf6d3346e43fff2604c9a7bc30ec-pvc
    persistentVolumeClaim:
      claimName: 196d7983e1b62995bae07beb7942e18ac46dcf6d3346e43fff2604c9a7bc30ec
status: {}

Troubleshooting

The podman docs indicate that attaching a network created with podman network create is supported: https://docs.podman.io/en/latest/markdown/podman-play-kube.1.html#network-mode-net

Opening a shell ( podman exec -it <container> /bin/bash ) on the containers created manually, I can ping both zabbix-server and postgres-server:

$ podman exec -it zabbix-server /bin/bash
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.89.0.3       c61fdf070064 zabbix-server
10.89.0.1 host.containers.internal
bash-5.1$ ping postgres-server
PING postgres-server.dns.podman (10.89.0.2) 56(84) bytes of data.
64 bytes from postgres-server.dns.podman (10.89.0.2): icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from postgres-server.dns.podman (10.89.0.2): icmp_seq=2 ttl=64 time=0.062 ms
64 bytes from postgres-server.dns.podman (10.89.0.2): icmp_seq=3 ttl=64 time=0.100 ms
^C
--- postgres-server.dns.podman ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.051/0.071/0.100/0.021 ms

On the play kube containers, pinging either monitoring-<name>-server or <name>-server fails. Pinging the DNS resolver on both the manual and the play kube setups returns successfully.

bash-5.1$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.89.0.2       monitoring be3df46ca68d-infra
10.89.0.1 host.containers.internal

bash-5.1$ ping postgres-server
ping: postgres-server: Name does not resolve
bash-5.1$ ping monitoring-postgres-server
ping: monitoring-postgres-server: Name does not resolve
bash-5.1$ cat /etc/resolv.conf
search dns.podman
nameserver 10.89.0.1

Using podman inspect <container>, both the manual and kube play setups have the 'mon' network attached. However, on the play kube setup, the output for network setup is the following:

"NetworkMode": "container:49e7b0bafd4619b9bcb50b1b841aeee3cb910bf3c555a9788dc297e71c948092"

This differs from the manual network setup:

"NetworkMode": "bridge"

It almost appears as if 'play kube' is ignoring the --network flag and instead creating an internal container network. Looking at the containers network setup with kube play, the ips on both containers are the same. On the manual setup, the IPs are different:

"Networks": {              
                "mon": {            
                    "EndpointID": "",
                    "Gateway": "10.89.0.1",
                    "IPAddress": "10.89.0.3",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "3e:30:b7:e2:1d:00",
                    "NetworkID": "mon",

Networks": {              
                "mon": {            
                    "EndpointID": "",
                    "Gateway": "10.89.0.1",
                    "IPAddress": "10.89.0.2",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "fa:1a:6b:f3:02:3a",
                    "NetworkID": "mon",

I've tried podman kube play with and without SELinux in Enforcing mode. I've also tried running this setup as root to no avail.

I've tested this setup with the following versions:

  • podman: 3.4.7, 4.0.2
  • RHEL: 8.5, 8.6

Any help would be much appreciated!

1 Answer 1

2

I found the solution. Containers created using podman play kube share a network namespace. Consequently, they can access each other using localhost.

1
  • Is there currently no way to DNS resolve using container names with play kube. While not a huge problem, each container in a pod has to expose an unique port.
    – alexm
    Nov 6, 2022 at 9:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .