1

I'm using an NFS mount to provide the /etc/prometheus/prometheus.yml (default) config file to prom/prometheus docker image all via Ansible. When the container is deployed, I'm getting the following error in the container logs, and the container restarts after a few seconds.

level=error ts=2020-10-28T16:01:04.432Z caller=main.go:290 msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" err="open /etc/prometheus/prometheus.yml: permission denied"

I can browse the mounted filesystem on my docker host (a Raspberry Pi 4), touch files and read prometheus.yml as the user that launches the container.

Below are the relevant tasks from my playbook, and the issue is the same when deploying the container from the CLI without the playbook with the remote filesystem mounted to at /mnt/prometheus, and passed to the container as a volume at /etc/prometheus

docker run -p 9090:9090 -v /mnt/prometheus:/etc/prometheus prom/prometheus

prometheus/tasks/main.yml (become: yes is set in the playbook that calls this role)

  - name: "Create mountpoint"
    file: 
        path: "{{ prometheus_mount_path }}"
        state: directory
        mode: 0777
        owner: root
        group: users
        

  - name: "Mount nfs drive for prometheus filesystem"
    mount: 
        path: "{{ prometheus_mount_path }}"
        src: "{{ nfs_server }}:{{ prometheus_nfs_path }}"
        state: mounted
        fstype: nfs
        
  - name: "Create prometheus.yml in mountpoint from template"
    template: 
        src: prometheus.yml.j2
        dest: "{{ prometheus_mount_path }}/prometheus.yml"

        
  - name: "Deploy prometheus container"
    docker_container:
        name: prometheus
        image: prom/prometheus:latest
        restart_policy: always
        state: started
        network_mode: host
        hostname: prometheus
#        exposed_ports: 9090
        published_ports: 9090:9090
        user: 995:1002
        mounts: 
        volumes:
            - "{{ prometheus_mount_path }}:/etc/prometheus"
        comparisons:
            '*': ignore
            env: strict

Any idea what would cause or how to resolve the permission denied issue from the container?

Edit: I tested by providing docker a local directory to share with the container instead of an NFS mount. This was shared successfully with the container, and the container started. Points to NFS issue, but I haven't figured that out yet.

2
  • What are the permissions of /etc/prometheus/prometheus.yml file inside the docker container? Does the user that runs the prometheus service have access to read that file? Oct 28, 2020 at 19:58
  • @GracefulRestart The container doesn't stay up for long enough that I can check. I have this config (mounting an NFS volume for data that's expected to persist container creation and destruction) running for another service, and it's working w/ identical playbooks, and only changed variables for mount points, image, etc. Oct 28, 2020 at 20:04

1 Answer 1

1

I found my solution and posted the answer here. Slightly more detail in my blog post

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .