Skip to content

Regression introduced in the 10.2.7 builds with updates within Splunk-Ansible. #737

Description

@brzuniga-lab

When we have to restart a Docker Splunk 10.2.7 instance, sometimes we see the following error:

TASK [splunk_common : Check for previous ConfigMap-owned values in indexes.conf] ***
ok: [localhost]
Tuesday 01 September 2026 21:24:09 +0000 (0:00:00.257) 0:00:44.298 *****
TASK [splunk_common : Load previous ConfigMap-owned values in indexes.conf] ****
fatal: [localhost]: FAILED! => {
"ansible_facts": {
"previous_config_map_stanzas": {}
},
"ansible_included_var_files": [],
"changed": false,
"message": "an error occurred while trying to read the file '/opt/splunk/etc/system/local/.indexes.conf.splunk-ansible-managed.yml': [Errno 13] Permission denied: b'/opt/splunk/etc/system/local/.indexes.conf.splunk-ansible-managed.yml'. [Errno 13] Permission denied: b'/opt/splunk/etc/system/local/.indexes.conf.splunk-ansible-managed.yml'"
}

Observed behavior:

The first initialization creates a Splunk-Ansible state file such as:

  • /opt/splunk/etc/system/local/.inputs.conf.splunk-ansible-managed.yml
    The file is created with the following ownership and permissions:
  • -rw-rw---- 1 41812 41812 25 .inputs.conf.splunk-ansible-managed.yml
    The numeric ownership corresponds to:
  • splunk:splunk
  • UID 41812
  • GID 41812
  • mode 0660
    The container initialization process runs as the ansible user:
  • uid=999(ansible) gid=999(ansible) groups=999(ansible),1000(sudo)
  • The ansible user is not a member of group splunk and therefore cannot read the persisted state file on the next initialization.
    The subsequent startup fails with:
    TASK [splunk_common : Load previous ConfigMap-owned values in inputs.conf]
    fatal: [localhost]: FAILED! => {
    "ansible_facts": {
    "previous_config_map_stanzas": {}
    },
    "ansible_included_var_files": [],
    "changed": false,
    "message": "an error occurred while trying to read the file '/opt/splunk/etc/system/local/.inputs.conf.splunk-ansible-managed.yml': [Errno 13] Permission denied: b'/opt/splunk/etc/system/local/.inputs.conf.splunk-ansible-managed.yml'. [Errno 13] Permission denied: b'/opt/splunk/etc/system/local/.inputs.conf.splunk-ansible-managed.yml'"
    }

Splunk-Ansible task behavior:

In both the 9.4.15 and 10.2.7 images, the state file is loaded using this task:

  • name: Load previous ConfigMap-owned values in {{ conf_file }}
    include_vars:
    file: "{{ conf_directory }}/.{{ conf_file }}.splunk-ansible-managed.yml"
    name: previous_config_map_stanzas
    when: config_map_managed_values_file.stat.exists
    no_log: "{{ hide_password }}"
    This task does not appear to use privilege escalation or become_user: splunk.

The file is created using:

  • name: Record ConfigMap-owned values in {{ conf_file }}
    copy:
    content: "{{ conf_stanzas | to_nice_yaml }}"
    dest: "{{ conf_directory }}/.{{ conf_file }}.splunk-ansible-managed.yml"
    owner: "{{ splunk.user }}"
    group: "{{ splunk.group }}"
    mode: '0660'
    when:
    • reconcile_config_map_values | default(false) | bool
    • conf_stanzas | length > 0
      become: yes
      become_user: "{{ splunk.user }}"
      no_log: "{{ hide_password }}"

This creates an apparent mismatch:

  • The file is written as splunk:splunk with mode 0660.
  • The next initialization attempts to read the file as ansible.
  • The ansible user is not a member of the splunk group.
  • The read fails with EACCES.

Example default.yml:

splunk:
http_enableSSL: true
http_port: 8443
conf:
- key: inputs
value:
directory: /opt/splunk/etc/system/local
content:
default:
host: test-host

Simplified reproduction procedure:

Start splunk/splunk:10.2.7 with a persistent volume mounted at /opt/splunk/etc.
Supply a default.yml containing a splunk.conf entry for inputs.conf.
Allow the first initialization to complete.
Confirm that this file exists:

  • ls -ln /opt/splunk/etc/system/local/.inputs.conf.splunk-ansible-managed.yml
    Expected result:
  • -rw-rw---- 1 41812 41812 ...
    Confirm the container runtime identity: docker exec id
    Observed result:
  • uid=999(ansible) gid=999(ansible) groups=999(ansible),1000(sudo)
  • Restart the container:
  • docker restart
  • Splunk-Ansible initialization fails while loading the persisted managed-values file.

Deyault.yml

The current Splunk Operator documentation continues to support default.yml through the defaults and defaultsUrl CR fields. For example, a ConfigMap can be mounted and referenced as:
spec:
volumes:
- name: defaults
configMap:
name: splunk-defaults
defaultsUrl: /mnt/defaults/default.yml

Because SOK uses the same Splunk Enterprise image, the same code path may be reachable when default.yml contains splunk.conf entries.

Actual Kubernetes exposure may depend on the effective Pod security context. A Pod configured with fsGroup: 41812 or supplementalGroups: [41812] may not experience the failure.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions