Handling of metadata shipped with Tape REST API for staging

[ArturAkh]

Dear dCache developers,

As was mentioned in the already-closed issue:

#8011

which mainly was for the use-case of archival metadata, a question from our side remains open on staging metadata and corresponding plans on dCache side.

I understand, that this is less carved in stone as it is the case for archival metadata, but at least there was a concrete suggestion from Julien Leduc - the current implementation in rucio (for ATLAS):

rucio/rucio#7451

As far as I understand, also CMS was considering this a good solution.

So all in all, I'll keep this second part on staging metadata open - unless this is also already covered and implemented in dCache. In that case, I'd like to ask you for some example on how to use and test that.

Thanks,
Artur

[DmitryLitvintsev]

Hi Artur,

The way tape rest API works - submits bulk pin requests. As such, the staging metadata has to propagate to PinManager and then to PoolManager and then to pool.

None of that yet exists in dCache.

Thanks,
Dmitry

P.S.: Based on my understanding of this feature, the intent is to pass some sort of mount policy hints to CTA that would play role in tape mount prioritization. I believe KIT uses HPSS. How are you going to use this metadata if it were to be passed by dCache?

[ArturAkh]

Hi @DmitryLitvintsev,

Thanks for the clarification! Yes, indeed this would involve quite a lot of pass-through logic down to the pool and its HSM provider. So rather something mid-/long-term to consider - hopefully well before HL-LHC ;).

Yes, the feature is mainly meant to have a definition of priorities. In case of our system at KIT we have a self-written interface between dCache and HPSS, which anyhow has a sorting and grouping mechanism for requests based on cartridges the files are on. Nothing speaks against introducing a prioritization logic in addition to that in our interface. The values for priority would then be picked up from a (custom and self-written) HSM provider that can talk to the interface.

Cheers,
Artur