New snmp-integration docs not being picked up for Logstash Reference (LSR)

[karenzone]

Integration-snmp docs are not being picked up in LSR docgen.

Some context

• New integration-snmp plugin was added to rakelib/plugins-metadata.json: Update plugins-metadata.json with integration-snmp: set skiplist to false logstash#16137 on 2024-05-17, Friday.
• Integration-snmp was bumped to v4.0.2 and published on 2024-05-20, Monday, and was picked up for the Versioned Plugin Reference (VPR).
• @karenzone kicked off a LSR docgen job. The job completed successfully, but it didn't pick up any changes.
  
• This message in the output indicates that integration-snmp is getting skipped because it is tagged as non-default integration. (You can see the same messaging for integration-zeromq, which we DO NOT want getting picked up at this time.
  
• We regularly publish non-default input, output, filter, and codec plugins, but the tooling treats integrations differently (https://github.com/elastic/docs-tools/blob/main/plugindocs.rb#L66-L69):
  

How do we want to handle this?

• It appears that we have two other non-default integration plugins.
  
  • It seems weird that the other two non-default plugins are being picked up for the docgen job, but they aren't mentioned in the skiplist. So obviously, there's some way to get plugins building that I'm not aware of. Can we figure out what that is, and then perhaps add those two in-process plugins to the skiplist with default-plugins: false and "skip-list": true? And then change docs tooling to allow non-default integration plugins such as integration-snmp is in the interim?

[karenzone]

Another possible solution: Could we add those to this skiplist to skip the whole repo?

[jsvd]

Another possible solution: Could we add those to this skiplist to skip the whole repo?

Indeed, the skip list in docs-tools should be only for repos that aren't plugins, so keeping:

skip:
  - logstash-core-plugin-api
  - logstash-patterns-core
  - logstash-devutils

is fine, and the rest should be moved to plugins-metadata.json, and we should be able to remove the condition to automatically exclude non default integrations, and exclude them if they have "skip-list":true https://github.com/elastic/docs-tools/blob/main/plugindocs.rb#L66-L69

[karenzone]

PRs:

• Add incomplete integration plugins to metadata.json as skip logstash#16174
• Adjust docs tooling for integration plugins #107

[karenzone]

UPDATE: The root cause still exists, but this particular issue was resolved when integration-snmp became a default plugin with the merge of elastic/logstash#16180.