diff --git a/Dataproducts-Summary-table.tex b/Dataproducts-Summary-table.tex index 1870500..3919224 100644 --- a/Dataproducts-Summary-table.tex +++ b/Dataproducts-Summary-table.tex @@ -10,20 +10,28 @@ \subsubsection{Summary Table} \sptablerule \textbf{Term} & \textbf{Label} & \textbf{Description} &\textbf{Parent}\cr \sptablerule -\multicolumn{4}{|r|}{\bf Added to \emph{product-type} Vocabulary \url{https://www.ivoa.net/rdf/product-type/}} \\ \hline +\noalign{\vspace{8pt}} +\hline +\multicolumn{4}{|r|}{\bf Added to the IVOA Data Product Type Vocabulary} (\url{https://www.ivoa.net/rdf/product-type/}) \\ \hline +\noalign{\vspace{2pt}} {\bf hea-event-bundle} & Event Bundle & A compounded dataset containing containing an {\bf hea-event-list} and multiple files or other substructures that are products necessary to analyze the {\bf hea-event-list} & none \cr -{\bf hea-event-list} & Event list & A dataset that records a collection of observed particle-detection events, such as incoming high-energy particles, where an event is typically characterized by a spatial position, a time, and a spectral value ({\em e.g.\/}, an energy, a channel, a pulse height) & none \cr - \hline -\multicolumn{4}{|r|}{\bf Added to \emph{response-type} vocabulary \url{https://www.ivoa.net/rdf/response-type/} } \\ \hline -{\bf response-function} & Response Function & A dataset that maps a physical quantity to an observable quantitiy. This term is mainly intended for retrieval. To annotate datasets, use a narrower term Narrower terms are preferred to indicate more precisely the type of {\bf response-function} & \cr -{\bf rmf} &\raggedright Redistribution Matrix File & A dataset that records the probability density function mapping from energy space into detector pulse height (or position) space & \#response-function \cr +{\bf hea-event-list} & Event List & A dataset that records a collection of observed particle-detection events, such as incoming high-energy particles, where an event is typically characterized by a spatial position, a time, and a spectral value ({\em e.g.\/}, an energy, a channel, a pulse height) & \#temporally-resolved-dataset \cr +\noalign{\vspace{8pt}} +\hline +\multicolumn{4}{|r|}{\bf Added to the IVOA Response Function Types Vocabulary} (\url{https://www.ivoa.net/rdf/response-type/}) \\ \hline +\noalign{\vspace{2pt}} +{\bf response-function} & Response Function & A dataset that records the mapping from a physical quantity to an observable quantitiy. Narrower terms are preferred to indicate more precisely the type of {\bf response-function} & \cr +{\bf rmf} &\raggedright Redistribution Matrix File & A dataset that records the probability density function mapping from energy space into detector pulse height (or position) space & \#response-function, \#pdf \cr {\bf aeff} & Effective Area & A dataset that records the ``effective area'' of a telescope and/or instrument. The effective area is the geometric area of the telescope and/or instrument reduced by efficiency factors such as reflectivity and vignetting, among other effects & \#response-function \cr {\bf arf} &\raggedright Ancillary Response File & A dataset that records the combined telescope/instrument effective area and detector quantum efficiency as a function of energy & \#response-function \cr -{\bf bkgrate} & Background Rate & A dataset that models the rate of residual events that are not from the expected source type ({\em e.g.\/}, for gamma-ray instrument {\bf bkgrate} measures residual non-gamma-ray events coming from charged cosmic rays) & \#response-function \cr -{\bf edisp} & Energy Dispersion & A dataset that records the probability density of detecting an event with an energy estimator (proxy) given the true energy of the event & \#response-function \cr -{\bf psf} &\raggedright Point Spread Function & A dataset that records the probability density function of spatial/angular spreading of incident particles from a point source caused by the instrument (detector and/or mirror and/or analysis) & \#response-function \cr \hline -\multicolumn{4}{|r|}{\bf Added to \emph{analysis-product} vocabulary \url{https://github.com/ivoa-std/VEPs}} \\ \hline -{\bf draws} & Draws & A dataset that records statistical draws computed from a probability distribution, for example Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable & none \cr +{\bf bkgrate} & Background Rate & A dataset that models the rate of residual events that are not from the expected source type ({\em e.g.\/}, for gamma-ray instruments {\bf bkgrate} measures residual non-gamma-ray events coming from charged cosmic rays) & \#response-function \cr +{\bf edisp} & Energy Dispersion & A dataset that records the probability density of detecting an event with an energy estimator (proxy) given the true energy of the event & \#response-function, \#pdf \cr +{\bf psf} &\raggedright Point Spread Function & A dataset that records the probability density function of spatial/angular spreading of incident particles from a point source caused by the instrument (detector and/or mirror and/or analysis) & \#response-function, \#pdf \cr +\noalign{\vspace{8pt}} +\hline +\multicolumn{4}{|r|}{\bf Considered for addition to an IVOA Analysis Data Product Vocabulary\footnote{The term ``Advanced Data Product,'' which is used in the ObsCore Recommendation, Version 1.1 is preferred rather than ``Analysis Data Product'' since the latter suggests that additional steps ({\em i.e.\/} some type of analysis) have taken place to construct these data products, which may not be the case.}} \\ \hline +\noalign{\vspace{2pt}} +{\bf draws} & Draws & A dataset that records statistical draws computed from a probability distribution or a sample population, for example Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable, or the DeltaTS associated with a quantity from a frequentist analysis & none \cr {\bf pdf} &\raggedright Probability Density Function & A dataset that records the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable & none \cr {\bf region} & Region & A dataset that encodes (one or more) regions of parameter space, for example a spatial region or a region of phase space covered by a dataset. The set of dimensions represented by the region can be arbitrary & none \cr %\sptablerule diff --git a/HighEnergyObsCoreExt.bib b/HighEnergyObsCoreExt.bib index 565f2ff..8a04ff1 100644 --- a/HighEnergyObsCoreExt.bib +++ b/HighEnergyObsCoreExt.bib @@ -32,6 +32,17 @@ @INPROCEEDINGS{2006SPIE.6270E..1VF adsnote = {Provided by the SAO/NASA Astrophysics Data System} } +@article{2026ApJ..1005..116F, + author = {{Fruscione}, Antonella and {McDowell}, Jonathan~C. and {Burke}, Douglas~J. and {Cresitello-Dittmar}, Mark and {Evans}, Ian~N. and {Evans}, Janet~D. and {Glotfelty}, Kenny and {G\"unther}, Hans Moritz and {Huenemoerder}, David and {joye}, William and {Lee}, Nicholas~P. and {McLaughlin}, Warren and {Miller}, Joseph~B. and {Nynka}, Melania and {Principe}, David~A. and {Siemiginowska}, Aneta}, + title = "{CIAO: Chandra's Data Analysis System for X-Ray Astronomy and Beyond}", + journal = {\apj}, + year = 2026, + month = jul, + volume = {1005}, + pages = {116}, + doi = {10.3847/1538-4357/ae6db4} +} + @article{gammapy:2023, author = {{Donath}, Axel and {Terrier}, R\'egis and {Remy}, Quentin and {Sinha}, Atreyee and {Nigro}, Cosimo and {Pintore}, Fabio and {Kh\'elifi}, Bruno and {Olivera-Nieto}, Laura and {Ruiz}, Jose Enrique and diff --git a/HighEnergyObsCoreExt.tex b/HighEnergyObsCoreExt.tex index 8d2406c..b7d7dfc 100644 --- a/HighEnergyObsCoreExt.tex +++ b/HighEnergyObsCoreExt.tex @@ -2,7 +2,7 @@ \input tthdefs \IfFileExists{./gitmeta}{\input gitmeta }{\typeout{NOTICE: gitmeta.tex not found}} -\title{IVOA ObsCore Extension and Discovery of High Energy Astrophysics Data} +\title{{\em Roadmap to\/} IVOA ObsCore Extension and \hbox{Discovery of High Energy Astrophysics Data}} % see ivoatexDoc for what group names to use here; use \ivoagroup[IG] for % interest groups. @@ -128,6 +128,8 @@ \section*{Conformance-related definitions} The \gls{VO} is a general term for a collection of federated resources that can be used to conduct astronomical research, education, and outreach. The \href{https://www.ivoa.net}{International Virtual Observatory Alliance (IVOA)} is a global collaboration of separately funded projects to develop standards and infrastructure that enable VO applications. +% Glossary entry expansion should restart in full at this point +\glsresetall \section{Introduction} @@ -138,18 +140,18 @@ \section{Introduction} \section{High Energy Astrophysics Data} -\gls{HEA} data include observations obtained using photon detectors covering X-ray (from $\sim$0.1 keV to $\sim$120 keV) through gamma-ray (from 120 keV up to $\gtrsim$ PeV) energies, as well as cosmic-ray and astrophysical neutrino ($\gtrsim$ GeV) detectors, or other messengers related to \gls{HEA} phenomena. The domain is now sufficiently mature to provide open data that are science-ready and work with open analysis tools ({\em e.g.\/}, CIAO, \citealt{2006SPIE.6270E..1VF}, or Gammapy, \citealt{gammapy:2023}). The science output of the \gls{HEA} domain already includes advanced products such as images, cubes, spectra, and time series such as light curves and time-resolved spectra. Additional data products include fitted sky models with spatial, spectral, and/or temporal component(s), along with their confidence intervals or confidence limits, and covariance matrices. Finally, multiple \gls{HEA} instruments produce source catalogs and surveys covering up to the full the sky, which include maps of photon or particle flux, exposure, sensitivity, and aperture-photometry likelihood profiles. +\gls{HEA} data include observations obtained using photon detectors covering X-ray (from $\sim$0.1 keV to $\sim$120 keV) through gamma-ray (from 120 keV up to $\gtrsim$ PeV) energies, as well as cosmic-ray and astrophysical neutrino ($\gtrsim$ GeV) detectors, or other messengers related to \gls{HEA} phenomena. The domain is now sufficiently mature to provide open data that are science-ready and work with open analysis tools ({\em e.g.\/}, CIAO, \citealt{2026ApJ..1005..116F}, or Gammapy, \citealt{gammapy:2023}). The science output of the \gls{HEA} domain already includes advanced products such as images, cubes, spectra, and time series such as light curves and time-resolved spectra. Additional data products include fitted sky models with spatial, spectral, and/or temporal component(s), along with their confidence intervals or confidence limits, and covariance matrices. Finally, multiple \gls{HEA} instruments produce source catalogs and surveys covering up to the full the sky, which include maps of photon or particle flux, exposure, sensitivity, and aperture-photometry likelihood profiles. Observations of the universe at the highest energies are based on techniques that are radically different compared to the UV through radio domains. \gls{HEA} observatories\footnote{For example, Chandra, XMM-Newton, Fermi, H.E.S.S., MAGIC, VERITAS, HAWC, LHAASO, IceCube, ANTARES, Auger, and soon CTAO, KM3NeT, and SWGO.} are generally designed to detect particles ({\em e.g.\/}, individual photons, cosmic-rays, or neutrinos) with the ability to estimate multiple observables for those particles. These detection techniques all rely on {\em event counting\/}\footnote{As opposed to signal integrating ({\em e.g.\/}, using a detector that accumulates the total photon signal during an exposure).}, where an event has some probability of being due to the interaction of a particle from an astrophysical source with the detectors, but also has some probability of being from instrumental or background effects. The data corresponding to an event are first an instrumental signal, which is then calibrated and processed to estimate physical quantities such as a time of arrival, point-of-origin on the sky, and an energy proxy associated with the event. Several other intermediate and qualifying characteristics may be associated with a detected event, depending on the detection technique. The ensemble of events detected over a given time interval and spatial field-of-view is referred to as an {\em event list\/}, which we designate an {\bf hea-event-list} in this document. -Though {\bf hea-event-list}s {\em may\/} include estimators for calibrated physical values, they typically still have to be corrected for the photometric, spectral, spatial, and/or temporal responses of the telescope and detector combination to yield scientifically interpretable information. The mappings between physical measurements of the source properties and the observables are called Instrument Response Functions (\glspl{IRF}\footnote{We try to avoid using the term \gls{IRF} in a normative sense since historical usage across the broad \gls{HEA} community (and from facility to facility) varies. In some cases, \gls{IRF} has been used to mean specifically the product of the \gls{ARF} and \gls{RMF}, whereas in other cases \gls{IRF} has been used more generally to mean any instrumental response function regardless of type.}). Some \glspl{IRF} are probabilistic in nature\footnote{For example, the energy matrix is a probability density function.}, and in addition may depend on the set of events selected for analysis by the end user. They are usually not invertible, so methods such as forward-folding fitting (using source models with any combination of spectral, spatial, temporal, and/or polarization components that are estimated) are needed to estimate physical properties, such as the true flux of particles from a source arriving at the instrument, given the measured observable quantities. The \glspl{IRF} generally evolve over time with the instrument and observation characteristics, and are usually defined for a specific time interval and may be decomposed into a standard set of independent components (see \S~3.1.5 of \citealt{2024ivoa.note.heig}), such as the spatial point-spread function or the energy-migration matrix or different messenger particle types, where each component may be stored or computed separately. Since both \glspl{IRF} and {\bf hea-event-list}s are required to analyze \gls{HEA} data, some \gls{IVOA} standards must be modified to expose both of them via the \gls{VO}. +Though {\bf hea-event-list}s {\em may\/} include estimators for calibrated physical values, they typically still have to be corrected for the photometric, spectral, spatial, and/or temporal responses of the telescope and detector combination to yield scientifically interpretable information. The mappings between physical measurements of the source properties and the observables are called \glspl{IRF}\footnote{We try to avoid using the term \gls{IRF} in a normative sense since historical usage across the broad \gls{HEA} community (and from facility to facility) varies. In some cases, \gls{IRF} has been used to mean specifically the product of the \gls{ARF} and \gls{RMF}, whereas in other cases \gls{IRF} has been used more generally to mean any instrumental response function regardless of type.}. Some \glspl{IRF} are probabilistic in nature\footnote{For example, the energy redistribution matrix is a probability density function.}, and in addition may depend on the set of events selected for analysis by the end user. They are usually not invertible, so methods such as forward-folding fitting (using source models with any combination of spectral, spatial, temporal, and/or polarization components that are estimated) are needed to estimate physical properties, such as the true flux of particles from a source arriving at the instrument, given the measured observable quantities. The \glspl{IRF} generally evolve over time with the instrument and observation characteristics, and are usually defined for a specific time interval and may be decomposed into a standard set of independent components (see \S~3.1.5 of \citealt{2024ivoa.note.heig}), such as the spatial point-spread function or the energy-migration matrix or different messenger particle types, where each component may be stored or computed separately. Since both \glspl{IRF} and {\bf hea-event-list}s are required to analyze \gls{HEA} data, some \gls{IVOA} standards must be modified to expose both of them via the \gls{VO}. -In the following, the current ObsCore standard will be discussed in \S~\ref{sec:obscore}, focusing on attributes that need to be modified. Then, we propose the creation of a \gls{HEA} extension of ObsCore in \S~\ref{sec:obscoreext}, as some attributes are very specific to our domain. In these two sections, the discussion focuses on the attribute definitions rather than on the attribute values. In \S~\ref{sec:voc}, enhancement of vocabulary is proposed for some ObsCore attributes, DataLink semantics, UCDs, and MIME-types. +In the following, the current ObsCore standard will be discussed in \S~\ref{sec:obscore}, focusing on attributes that need to be modified. Then, we propose the creation of a \gls{HEA} extension of ObsCore in \S~\ref{sec:obscoreext}, as some attributes are very specific to our domain. In these two sections, the discussion focuses on the attribute definitions rather than on the attribute values. In \S~\ref{sec:voc}, vocabulary enhancements are proposed for some ObsCore attributes, DataLink semantics, UCDs, and MIME-types. Finally, in \S~\ref{sec:accessoptions} we discuss various options for accessing \gls{HEA} data products using ObsTAP services. \section{ObsCore Attribute Definitions for High Energy Astrophysics Data} \label{sec:obscore} -The ObsCore representation of any \gls{HEA} \textbf{hea-event-list} data products is described in terms of curation, coverage, and access. However, given the \gls{HEA} data specificities, several properties, including resolutions, observable axis descriptions, and polarization states would be simply set to ``NULL'', and data axis lengths set to ``$-1$''. Therefore, for these data products and associated \glspl{IRF}, the definitions of some ObsCore attributes should be adjusted so that they better represent the content of the data from the perspective of data discovery. We note that many properties, including spatial and spectral coverage and resolution can vary strongly with energy and off-axis angle. These adjustments will also typically apply to advanced, high-level data products derived from \textbf{hea-event-list} data. +The ObsCore representation of any \gls{HEA} \textbf{hea-event-list} data products is described in terms of curation, coverage, and access. However, given the \gls{HEA} data specificities, several properties, including resolutions, observable axis descriptions, and polarization states would be simply set to ``NULL'', and data axis lengths set to ``$-1$''. Therefore, for these data products and associated \glspl{IRF}, the definitions of some ObsCore attributes should be adjusted so that they better represent the content of the data from the perspective of data discovery. We note that many properties, including spatial and spectral coverage and resolution can vary strongly with energy and off-axis angle. These adjustments will also typically apply to higher-level advanced data products derived from \textbf{hea-event-list} datasets. Currently, some ObsCore attributes ({\em dataproduct\_type\/} and {\em calib\_level\/}) are formally defined in the ObsCore Recommendation Version 1.1 \citep{2017ivoa.spec.0509L} and also in the vocabularies documents \citep{2023ivoa.spec.0206D, 2021ivoa.spec.0525D}\footnote{Primarily the Data Product Type Vocabulary, \url{https://www.ivoa.net/rdf/product_type}.}, which may be referenced in future versions of the ObsCore Recommendation. For completeness, we are proposing in this document modifications to both the existing ObsCore Recommendation and IVOA vocabularies. @@ -166,29 +168,29 @@ \subsection{{\em dataproduct\_type}} {\bf event}: an event-counting ({\em e.g.\/}, X-ray or other high energy) dataset of some sort. Typically this is instrumental data, {\em i.e.\/}, ``event data''. An event dataset is often a complex object containing multiple files or other substructures. An event dataset may contain data with spatial, spectral, and time information for each measured event, although the spectral resolution (energy) is sometimes limited. Event data may be used to produce higher level data products such as images or spectra. \end{quote} -We propose to add the following {\em dataproduct\_type\/} terms to ObsCore to better define a \gls{HEA} {\bf hea-event-list} and an {\bf hea-event-bundle} that includes the {\bf hea-event-list} and associated data: +We propose to add the following {\em dataproduct\_type\/} terms to ObsCore to better define an \gls{HEA} event list ({\bf hea-event-list}) and an \gls{HEA} ``event bundle'' ({\bf hea-event-bundle}) that includes the {\bf hea-event-list} and associated data products: \begin{quote} {\bf hea-event-list}: a dataset that records a collection of observed particle-detection events, such as incoming high-energy particles, where an event is typically characterized by a spatial position, a time, and a spectral value ({\em e.g.\/}, an energy, a channel, a pulse height). -{\bf hea-event-bundle}: a compounded dataset containing an {\bf hea-event-list} and multiple files or other substructures that are products necessary to analyze the hea-event-list. Data in an {\bf hea-event-bundle} may thus be used to produce higher level data products calibrated in physical units when containing \glspl{IRF} or other data products that can be used to construct \glspl{IRF}. +{\bf hea-event-bundle}: a compounded dataset containing a {\bf hea-event-list} and multiple files or other substructures that are products necessary to analyze the hea-event-list. Data in a {\bf hea-event-bundle} may thus be used to produce higher level data products calibrated in physical units when containing \glspl{IRF} or other data products that can be used to construct \glspl{IRF}. \end{quote} -We note that the term {\bf event} has caused confusion in the past, since ``event'' may also be used to describe notifications ({\em e.g.\/}, as in ``VOEvent'') of astrophysical events such as supernova explosions. Such ``events'' are quite different from the particle detection events being described herein. Using {\bf hea-event-list} will help to resolve this ambiguity. +We note that the term {\bf event} has caused confusion in the past, since ``event'' may also be used to describe notifications ({\em e.g.\/}, as in ``VOEvent'') of astrophysical events such as supernova explosions. Such ``events'' are quite different from the particle detection events being described herein. Use of the terms {\bf hea-event-list} and {\bf hea-event-bundle} will help to resolve this ambiguity. -In addition to {\em dataproduct\_type\/} terms that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of ``advanced data products'' (typically with {\em calib\_level\/} $\ge$ 3) that may be generated from astronomical observations by users or observatories. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and observations, or Frequentist adjustment of models of electron populations for multi-wavelength data spanning from X-rays to PeV gamma rays) means that data providers may choose to provide such analysis products directly to the end user. For example, the Chandra Source Catalog includes 38 types of advanced data products (for a total of $\sim\!90$ million files) and $\sim\!50$\% of these data product types are not well represented by a {\em dataproduct\_type\/} value that allows for meaningful data discovery. Users will certainly want to discover these data products independently from the associated progenitor observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional {\em dataproduct\_type\/} (or {\em dataproduct\_subtype\/}) terms for these advanced data products, and note that these terms will certainly be useful independent of waveband ({\em i.e.\/}, they can be equally applicable to UV/optical, IR, and radio datasets): +In addition to {\em dataproduct\_type\/} terms that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of ``advanced data products'' (typically with {\em calib\_level\/} $\ge$ 3) that may be generated from astronomical observations by users or observatories. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and observations, or frequentist adjustment of models of electron populations for multi-wavelength data spanning from X-rays to PeV gamma rays) means that data providers may choose to provide such advanced data products\footnote{We choose to use the term ``advanced data products'' for these types of data products because that is the term for such data products, such as {\bf sed}, used in the ObsCore Recommendation document.} directly to the end user. For example, the Chandra Source Catalog includes 38 types of advanced data products (for a total of $\sim\!90$ million files) and $\sim\!50$\% of these data product types are not well represented by a {\em dataproduct\_type\/} value that allows for meaningful data discovery. Users will certainly want to discover these data products independently from the associated progenitor observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional {\em dataproduct\_type\/} (or {\em dataproduct\_subtype\/}) terms for these advanced data products, and note that these terms will certainly be useful independent of waveband ({\em i.e.\/}, they can be equally applicable to UV/optical, IR, and radio datasets): \begin{quote} -{\bf draws}: a dataset that records statistical draws computed from a probability distribution, for example Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable. The draws can be interpreted to provide a robust estimation of the probability distribution of variable, and correlations between the draws provide information about how well the draws converge to the parent probability distribution.\footnote{As an example, within the standard $\Lambda CDM$ cosmological model, estimates of the cosmological density parameters $\Omega M$ and $\Omega\Lambda$ can be derived from the intersection of confidence contours from Hubble diagram of quasars with those from the Type Ia supernovae \citep{2019adds.book..283C}. These contours are {\bf draws}.} +{\bf draws}: a dataset that records statistical draws computed from a probability distribution or a sample population, for example Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable, or the DeltaTS associated with a quantity from a frequentist analysis. The draws can be interpreted to provide a robust estimation of the probability distribution of variable, and correlations between the draws provide information about how well the draws converge to the parent probability distribution.\footnote{As an example, within the standard $\Lambda CDM$ cosmological model, estimates of the cosmological density parameters $\Omega M$ and $\Omega\Lambda$ can be derived from the intersection of confidence contours from Hubble diagram of quasars with those from the Type Ia supernovae \citep{2019adds.book..283C}. These contours are {\bf draws}.} -{\bf pdf}: a dataset that records the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable, or the DeltaTS associated with a quantity from a Frequentist analysis. The probability density function provides a robust estimation of the variable and allows arbitrary confidence intervals to be computed directly from the distribution. +{\bf pdf}: a dataset that records the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable, or the DeltaTS associated with a quantity from a frequentist analysis. The probability density function provides a robust estimation of the variable and allows arbitrary confidence intervals to be computed directly from the distribution. {\bf region}: a dataset that includes an encoding of (one or more) regions of parameter space, for example a spatial region or a region of phase space covered by a dataset. The set of dimensions represented by the region can be arbitrary.\footnote{One possible encoding is a \gls{MOC}; however the vast majority of pre-existing region data products in \gls{HEA} data archives currently use other encodings.} -{\bf response-function}: a dataset that records a mapping from a physical quantity to an observable quantity. For \gls{HEA}, this may be the components of the composite \gls{IRF} such as an Auxiliary Response File ({\bf arf}), Redistribution Matrix File ({\bf rmf}), Effective Area ({\bf aeff}), Energy Dispersion ({\bf edisp}), the Background Rate ({\bf bkgrate}). The Point Spread Function ({\bf psf}) is a response function that is generally applicable across multiple wavebands. While these datasets may generally be represented as an N-dimensional data cube, designating them as {\bf response-function}s enhances data discovery for very common types of \gls{HEA} dataset (see the use cases in Appendix~\ref{sec:uc}). +{\bf response-function}: a dataset that records a mapping from a physical quantity to an observable quantity. For \gls{HEA}, this may be the components of the composite \gls{IRF} such as an Auxiliary Response File ({\bf arf}), Redistribution Matrix File ({\bf rmf}), Effective Area ({\bf aeff}), Energy Dispersion ({\bf edisp}), or the Background Rate ({\bf bkgrate}). The Point Spread Function ({\bf psf}) is a response function that is generally applicable across multiple wavebands. While these datasets may generally be represented as an $N$-dimensional data cube, designating them as {\bf response-function}s enhances data discovery for very common types of \gls{HEA} datasets (see the use cases in Appendix~\ref{sec:uc}). \end{quote} -%\TODO{ mireille: Rewrite this paragraph to avoid measurements: -%no agreement exist for this term among ObsCore Implementors in practice. } +%\TODO{ mireille: Rewrite this paragraph to avoid measurements: no agreement exist for this term among ObsCore Implementors in practice. } +%%% There is no way to rewrite this paragraph without referring to measurements. The term "measurements" is defined in the ObsCore Recommendation and this recommendation modifies the wording of that definition somewhat. The measurements concept is actually quite important as it's use avoids having to dramatically increase the the number of data product type terms one would otherwise require (some of which may be facility dependent). The {\bf measurements} {\em dataproduct\_type\/} is quite useful for many different types of advanced data products (which may be derived from multiple observations). But users of those products often may not be interested in the progenitor datasets, especially as multiple advanced data products may be extracted from the same single progenitor or a few progenitors ({\em e.g.\/}, measurements associated with multiple sources detected in a single observation field). We propose to delete the caveat associated with {\em dataproduct\_type\/} = ``measurements'' in the ObsCore IVOA Recommendation (\S~4.1.1) that requires the derived data products be exposed ``{\bf together} with the progenitor observation dataset''. The recovery of progenitor observation datasets may be achieved using provenance information, if desired. @@ -196,21 +198,23 @@ \subsection{{\em dataproduct\_subtype}} The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify more precisely the scientific nature of a data product. Although no vocabulary is defined for {\em dataproduct\_subtype\/}, we recommend that data providers formulate and use a standardized vocabulary for this attribute for data products that are commonly used in \gls{HEA}\null. We have proposed several terms in \S~5 for commonly used \gls{HEA} {\bf response-function} types ({\em e.g.\/}, {\bf aeff}, {\bf edisp}, {\bf psf}), but additional terms could be standardized for other common data products. For example, standardizing using {\bf exposure-map} for an exposure map would enable queries such as ({\em dataproduct\_type\/} = {\bf image}) AND ({\em dataproduct\_subtype\/} = {\bf exposure-map}) to work across multiple facilities. Other possible terms could include (but are not limited to) {\bf significance-map} for a significance map, {\bf probability-map} for a probability map, and {\bf exclusion-map} for an exclusion map ({\em e.g.\/}, as used to adjust TeV background models). -\TODO{ Propose a small table of definitions for all these maps : could be added in the analysis-product vocabulary} +% \TODO{ Propose a small table of definitions for all these maps : could be added in the analysis-product vocabulary} +%%% Commented out for the submitted document. +%%% Such a table would include at least exposure-map, instrument-map, significance-map, probability-map, exclusion-map, background-map, sensitivity-map but there are a host of other possibilities, for example background-subtracted image, pha spectrum, aperture-photometry pdf, detection region, field-of-view region, extended-detection region etc. etc. Which of these are common across facilities (or might be in the future) and which are unique to a single facility is unclear. For example, maybe we're the only ones that produce a sensitivity-map (limiting sensitivity vs. location) at the moment, but should we define this product name since it seems reasonable that other facilities will want to create a similar product in the future? \subsection{{\em calib\_level}} ObsCore defines calibration {\bf Level 1} as ``Instrumental data in a standard format (FITS, VOTable, SDFITS, ASDM, etc.) which could be manipulated with standard astronomical packages.'' and {\bf Level 2} as ``Calibrated, science ready data with the instrument signature removed.'' -However, some \gls{HEA} {\bf hea-event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. This is typically done because the {\bf response-function}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf hea-event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. +However, some {\bf hea-event-list}s include spatial and time axes that are calibrated physical quantities, but the spectral axis is instrumental and requires application of the IRFs to remove this signature. This is typically done because the {\bf response-function}s can depend on the choice of region (spatial/time) from which the events are extracted (especially for telescope/detector combinations where the telescope position dithers on the sky during the exposure), which depends on the specific science case and therefore cannot be determined {\em a priori\/}. Such {\bf hea-event-list}s fall ``between'' {\em calib\_level\/} 1 and 2. On the other hand, other {\bf hea-event-list}s may not have any calibrated axes or may have all axes calibrated, and it is important to be able to differentiate between these for data discovery. While the value for {\em calib\_level\/} for any data product is left for the data provider to determine, we suggest that individual data providers set {\em calib\_level\/} = 1 if an {\bf hea-event-list} is considered to be ``uncalibrated'' according to normal usage for their data products, and set {\em calib\_level\/} = 2 if an {\bf hea-event-list} is considered to be ``calibrated'' according to normal usage for their data products. -Also, we propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis ObsCore {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). +We also propose that the calibration status of the spatial/spectral/time data axes be identified using the appropriate axis ObsCore {\em calib\_status\/} keyword ({\em s\_calib\_status\/} for the spatial axes, {\em em\_calib\_status\/} for the spectral axis, and {\em t\_calib\_status\/} for the time axis). \subsection{{\em access\_url}} -Given the complexity and number of \gls{HEA} data products, the {\em access\_url\/} may point either directly to a file ({\em e.g.\/}, to the {\bf hea-event-list} or an {\bf hea-event-bundle}), or to a DataLink service that will provide links to the data and to associated data ({\em e.g.\/}, {\bf response-function}s). +Given the complexity and number of \gls{HEA} data products, the {\em access\_url\/} may point either directly to a file ({\em e.g.\/}, to an {\bf hea-event-list} or an {\bf hea-event-bundle}), or to a DataLink service that will provide links to both primary and associated ({\em e.g.\/}, {\bf response-function}) data products. If a DataLink is provided, {\em access\_format\/} should be set to ``application/x-votable+xml;content=datalink'' to indicate that the URL points to a Data\-Link service. @@ -252,9 +256,9 @@ \subsection{{\em em\_calib\_status}} \subsection{{\em o\_ucd}} -For an {\bf hea-event-list}, we can consider that all measures stored in column values are observables. This is {\em the\/} fundamental difference between \gls{HEA} {\bf hea-event-list}s and typical pixelated datasets. The current ObsCore Recommendation suggests that {\em o\_ucd\/} be set to ``NULL'' for event lists. However this significantly hampers data discovery for \gls{HEA} datasets. Since the data content of {\bf hea-event-list}s may vary significantly from facility to facility, meaningful discovery of \gls{HEA} datasets {\em requires\/} the user be able to query the UCDs of the set of observables included in an {\bf hea-event-list}. +For an {\bf hea-event-list}, we can consider that all measures stored in column values are observables. This is {\em the\/} fundamental difference between {\bf hea-event-list}s and typical pixelated datasets. The current ObsCore Recommendation suggests that {\em o\_ucd\/} be set to ``NULL'' for event lists. However this significantly hampers data discovery for \gls{HEA} datasets. Since the data content of {\bf hea-event-list}s may vary significantly from facility to facility, meaningful discovery of \gls{HEA} datasets {\em requires\/} the user be able to query the UCDs of the set of observables included in an {\bf hea-event-list}. -A natural way of doing this that is consistent with current usage would be to extend {\em o\_ucd\/} to allow specification of {\em multiple\/} observables for {\bf hea-event-list}s (and {\bf hea-event-bundle}s), for example, {\em o\_ucd\/} = \newline {\em `pos.eq\#time\#instr.pulse;arith.sum'\/}. We propose using the {\em hash symbol\/} (`\#') to separate UCDs for the multiple observables to distinguish from the case where multiple UCD words separated by semicolons may be needed to define the UCD for a single observable. This follows a suggestion from the EPN-TAP Recommendation \citep{2022ivoa.spec.0822E} to use the hash symbol as a separator. Doing so can simplify ADQL queries since ADQL includes a {\tt ivo\_hashlist\_has} IVOA-standardized user defined function that can be used to validate if a particular UCD is included. One can also perform an ADQL query similar to ``o\_ucd LIKE `\%string\%'\null'' if all that is desired is to verify the presence of a specific UCD `string'. +A natural way of doing this that is consistent with current usage would be to extend {\em o\_ucd\/} to allow specification of {\em multiple\/} observables for \gls{HEA} data products, including at least {\bf hea-event-list}s and {\bf hea-event-bundle}s, for example, {\em o\_ucd\/} = {\em `pos.eq\#time\#instr.pulse;arith.sum'\/}. We propose using the {\em hash symbol\/} (`\#') to separate UCDs for the multiple observables to distinguish from the case where multiple UCD words separated by semicolons may be needed to define the UCD for a single observable. This follows a suggestion from the EPN-TAP Recommendation \citep{2022ivoa.spec.0822E} to use the hash symbol as a separator. Doing so can simplify ADQL queries since ADQL includes a {\tt ivo\_hashlist\_has} IVOA-standardized user defined function that can be used to validate if a particular UCD is included. One can also perform an ADQL query similar to ``o\_ucd LIKE `\%string\%'\null'' if all that is desired is to verify the presence of a specific UCD `string'. We note that extending {\em o\_ucd\/} to allow specification of multiple observables would require similar adjustments to the other observable axis attributes {\em o\_unit\/}, {\em o\_calib\_status\/}, and {\em o\_stat\_err\/}. @@ -262,12 +266,12 @@ \subsection{{\em o\_ucd}} In the example {\em o\_ucd\/} above, the UCD {\em instr.pulse;arith.sum\/} is used to represent the detector Pulse Height Amplitude (PHA)\null. There is currently no UCD defined for a raw measure like PHA, but we propose the addition of {\em instr.pulse\/} to the UCDs list vocabulary, together with other UCDs that are relevant for \gls{HEA} data, in \S~\ref{sec:UCDs}. Several additional UCDs, including electromagnetic spectrum, physical quantities, and statistical parameters UCDs, are also proposed in \S~\ref{sec:UCDs} that are relevant for \gls{HEA} data products but could also be of use for other domains such as cosmology. -Advanced data products may similarly record multiple observables that can only be differentiated through their UCDs. For example, a Chandra Source Catalog {\bf pdf} dataset for a detection may include multiple marginalized probability density functions computed using a Bayesian X-ray aperture photometry algorithm in units of net counts, net count rates, photon fluxes, and energy fluxes in multiple apertures. The observables recorded in the different marginal probability density functions (MPDF) may be distinguished by their UCDs which then become relevant for data discovery when a user is searching for specific aperture photometry datasets. +Advanced data products may similarly record multiple observables that can only be differentiated through their UCDs. For example, a Chandra Source Catalog {\bf pdf} dataset for a detection may include multiple marginalized probability density functions computed using a Bayesian X-ray aperture photometry algorithm with various observables including net counts, net count rates, photon fluxes, and energy fluxes, in multiple apertures. The observables recorded in the different marginal probability density functions (MPDF) may be distinguished by their UCDs which then become relevant for data discovery when a user is searching for specific aperture photometry datasets. \subsection{{\em proposal\_id}} -To support advanced data products that may be constructed using data from multiple progenitor observations, we propose to modify the ObsCore Recommendation for {\em proposal\_id\/} to allow multiple values, similar to {\em facility\_name\/} and {\em instrument\_name\/}. We propose using the {\em hash symbol\/} (`\#') to separate the different values, like in the {\em o\_ucd\/} field. +To support advanced data products that may be constructed using data from multiple progenitor observations, we propose to modify the ObsCore Recommendation for {\em proposal\_id\/} to allow multiple values using a list of comma separated strings, or the word ``Many'', similar to the specifications for the {\em facility\_name\/} and {\em instrument\_name\/} provenance keywords. \section{Extensions to ObsCore Specific to High Energy Astrophysics Data} @@ -275,7 +279,7 @@ \section{Extensions to ObsCore Specific to High Energy Astrophysics Data} \subsection{{\em ev\_xel}} -The lengths of each data axis (spatial, spectral, time, polarization) captured in attributes {\em s\_xel1\/}, {\em s\_xel2\/}, {\em em\_xel\/}, {\em t\_xel\/}, {\em pol\_xel\/} do not apply non-pixelated data including {\bf hea-event-list}s, and ObsCore recommends that these attributes be set to $-1$. However, the dimensionality of an event list is an important property for data discovery, as the number of events often scales with signal-to-noise (and also data volume scales with number of events). We propose to add a new, optional attribute {\em ev\_xel\/} that records the number of events in an {\bf hea-event-list} (effectively, the length of the ``events'' axis in the {\bf hea-event-list}'s table). +The lengths of each data axis (spatial, spectral, time, polarization) captured in attributes {\em s\_xel1\/}, {\em s\_xel2\/}, {\em em\_xel\/}, {\em t\_xel\/}, {\em pol\_xel\/} do not apply to non-pixelated data including {\bf hea-event-list}s, and ObsCore recommends that these attributes be set to $-1$. However, the dimensionality of an event list is an important property for data discovery, as the number of events often scales with signal-to-noise (and also data volume scales with number of events). We propose to add a new, optional attribute {\em ev\_xel\/} that records the number of events in an {\bf hea-event-list} (effectively, the length of the ``events'' axis in the {\bf hea-event-list}'s table). \subsection{{\em s\_ref\_energy\/}/{\em em\_ref\_energy\/}/{\em s\_ref\_oaa\/}/{\em em\_ref\_oaa}} @@ -287,7 +291,7 @@ \subsection{{\em t\_intervals}} The global time bounds described by {\em t\_min\/}/{\em t\_max\/} in general are not sufficiently flexible when representing \gls{HEA} datasets or advanced data products from any waveband. The former are typically composed of many \glspl{STI}/\glspl{GTI}, where data are only valid during the stable or good intervals, while advanced data products may be constructed from multiple progenitor observations that can span decades from the start time of the first observations to the stop time of the last observation (albeit very sparsely). For both cases, data queries using only {\em t\_min\/}/{\em t\_max\/} will not be adequate to determine whether useful scientific data coincide with a transient cosmic phenomenon. In such cases, a more detailed knowledge of the observation time coverage is necessary. We propose to add a new optional attribute {\em t\_intervals\/} that would contain the list of observation intervals or \glspl{STI}/\glspl{GTI} as a TMOC description following the \gls{MOC} IVOA standard \citep{2022ivoa.spec.0727F}. This element could then be compared across data collections to make the data set selection via simple intersection or union operations in TMOC representation. -We recognize that performing such queries will require enhancements to ADQL, but this capability is sufficiently important for some \gls{HEA} data discovery scenarios that we have chosen to add {\em t\_intervals\/}, in anticipation that ADQL will eventually provide this functionality. +We recognize that performing such queries will require enhancements to ADQL, but this capability is sufficiently important for some \gls{HEA} data discovery scenarios that we have chosen to add {\em t\_intervals\/}, in anticipation that ADQL will eventually provide this functionality.\footnote{The use of a prototype User-Defined Function that extended ADQL to compute the intersection of TMOC such as {\em t\_intervals} and a second TMOC or a time-period defined by an MJD pair was demonstrated at the Northern Spring 2026 IVOA Interop in Strasbourg, France.} \subsection{{\em energy\_min\/}/{\em energy\_max\/}} @@ -297,7 +301,7 @@ \subsection{{\em energy\_min\/}/{\em energy\_max\/}} \subsection{{\em obs\_mode}} -Many \gls{HEA} instruments may be configured using multiple observing modes and these observing modes may significantly impact the structure and characteristics ({\em e.g.\/}, calibration accuracy) of the resulting observation datasets. For example, the Chandra ACIS instrument typically produces {\bf hea-event-list}s with 2-dimensional spatial coordinates ({\em i.e.\/}, imaging) but has an observation mode that continuously reads-out the detector, effectively producing an {\bf hea-event-list} with a single spatial dimension (the other spatial dimension is collapsed); users looking only for imaging data may want to restrict their queries to exclude the latter observing mode. +Many \gls{HEA} instruments may be configured using multiple observing modes and these observing modes may significantly impact the structure and characteristics ({\em e.g.\/}, calibration accuracy) of the resulting observation datasets. For example, the Chandra ACIS instrument typically produces {\bf hea-event-list}s with 2-dimensional spatial coordinates ({\em i.e.\/}, imaging) but has a high time resolution observation mode that continuously reads-out the detector, effectively producing an {\bf hea-event-list} with a single spatial dimension (the other spatial dimension is collapsed); users looking only for imaging data may want to restrict their queries to exclude the latter observing mode. We propose to add an optional attribute {\em obs\_mode\/} that allows the data provider to specify the observation mode for an observation. Constraints on observation mode can provide a simple way to discover data sets for a specific facility/instrument combination. We note that permissible {\em obs\_mode\/} values will vary from facility to facility and from instrument to instrument. @@ -321,7 +325,7 @@ \subsection{{\em scan\_mode}} \begin{itemize} \item \texttt{on-source}: pointed observation; \item \texttt{on-off}: switched observations between two spatial positions (source and background); - \item \texttt{raster-map}: observations on a predefined spatial mesh (generally regular and rectangular ({\em e.g.\/}, a grid observation for \glspl{IACT}); + \item \texttt{raster-map}: observations on a predefined spatial mesh (generally regular and rectangular [{\em e.g.\/}, a grid observation for \glspl{IACT}]); \item \texttt{on-the-fly-cross-scan}: observations along a predefined spatial pattern; \item \texttt{on-the-fly-cross-map}: observations along parallel directions ({\em e.g.\/}, a wobble observation for \glspl{IACT}); \item \texttt{slew} : observations taken while the telescope is slewing. @@ -343,12 +347,12 @@ \subsection{{\em analysis\_mode}} Most \gls{HEA} instruments employ significant software processing to transform raw data into the {\bf hea-event-bundle} data exposed to users, including algorithms for calibration and event property reconstruction. The way in which this processing is configured therefore has a potentially large impact on the content of the reduced datasets; indeed the same observation processed with two different configurations may result in different scientific performance. In some cases, multiple processing configurations within the same observation collection are used to provide users with a wider range of scientific coverage. -We propose to add an optional attribute {\bf analysis\_mode} that allows the data provider to specify the data reduction/analysis mode for an observation, in case more than one is applied. Constraints on analysis mode can provide a simple way to discover data sets for a specific facility/instrument combination. We note that permissible {\bf analysis\_mode} values may vary from facility to facility and from instrument to instrument. +We propose to add an optional attribute {\em analysis\_mode\/} that allows the data provider to specify the data reduction/analysis mode for an observation, in case more than one is applied. Constraints on analysis mode can provide a simple way to discover data sets for a specific facility/instrument combination. We note that permissible {\em analysis\_mode\/} values may vary from facility to facility and from instrument to instrument. \subsection{{\em event\_type}} \label{sec:evttype} -Some \gls{HEA} instruments allow particle events to be partitioned into separate subsets based on some set of defined criteria. This is typically based on a data analysis quality associated with the reconstruction and discrimination of the events, and analyses can flag each event by a quality label, effectively partitioning the dataset into strictly disjoint event subsets. For each subset, a set of associated {\bf response-function}s can be separately computed\footnote{For example, the Fermi-LAT collaboration produces separate \glspl{IRF} for each event type; see \url{https://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_LAT_IRFs/IRF_overview.html}.}. +Some \gls{HEA} instruments allow particle events to be partitioned into separate subsets based on some set of defined criteria. This is typically based on a data analysis quality associated with the reconstruction and discrimination of the events, and analyses can flag each event by a quality label, effectively partitioning the dataset into strictly disjoint event subsets. For each subset, a set of associated {\bf response-function}s can be separately computed.\footnote{For example, the Fermi-LAT collaboration produces separate \glspl{IRF} for each event type; see \url{https://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_LAT_IRFs/IRF_overview.html}.} We propose to add an optional attribute {\em event\_type\/} that specifies the data quality flag for an observation. This attribute will allow the data provider to split the event list into several event lists labelled by an unique {\em event\_type\/} for a given observation, and if appropriate to distribute their associated \glspl{IRF}. Constraints on event type can provide a simple way to discover data sets for a specific facility/instrument combination and to reduce the downloaded data volume. We note that permissible {\em event\_type\/} values may vary from facility to facility and from instrument to instrument. @@ -358,21 +362,79 @@ \subsection{{\em messenger}} While many \gls{HEA} facilities detect photons ({\em e.g.\/}, most space-based X-ray and gamma-ray facilities), other facilities can detect alternate messenger particles, such as cosmic-rays, neutrinos, or massive particles. The end user will therefore need the ability to select the messenger when querying for data products using ObsCore. -We propose to add an optional attribute {\em messenger\/} that specifies the messenger type for an observation. Constraints on {\em messenger\/} can provide a simple way to discover data sets for a specific messenger. While the entries in the IVOA Messengers Vocabulary\footnote{\url{https://www.ivoa.net/rdf/messenger}} provide a useful starting point for populating this attribute, we propose that the predefined {\em messenger\/} values for \gls{HEA} data include at least the following: +We propose to add an optional attribute {\em messenger\/} that specifies the {\em name} of the messenger type for an observation. Constraints on {\em messenger\/} can provide a simple way to discover data sets for a specific messenger. While the entries in the IVOA Messengers Vocabulary\footnote{\url{https://www.ivoa.net/rdf/messenger}} provide a useful starting point for populating this attribute, we propose that the predefined {\em messenger\/} values for \gls{HEA} data include at least the following: \begin{itemize} \item \texttt{cosmic-ray}: the messenger is a cosmic-ray; \item \texttt{muon}: the messenger is a muon; \item \texttt{neutrino}: the messenger is a neutrino (any type); \item \texttt{photon}: the messenger is a photon; - \item \texttt{proton}: the messenger is a proton or anti-proton; - \item \texttt{pdgid{\em $\pm$nn\/}}: the messenger is a particle whose PDG ID\footnote{Particle Data Group Identifier, see \url{https://www.phy.bnl.gov/twister/bee/particles/}} is {\em $\pm$nn\/}. + \item \texttt{proton}: the messenger is a proton or anti-proton. \end{itemize} -The value ``pdgid{\em $\pm$nn\/}'' (where the sign is mandatory) allows a messenger particle not listed in the table to be specified by reporting the particle's Particle Data Group Identifier as {\em $\pm$nn\/}. For example, a muon, $\mu^{-}$, may be specified as ``pdgid+13''. We note that permissible {\em messenger\/} values will vary from facility to facility and from instrument to instrument. +We note that permissible {\em messenger\/} values will vary from facility to facility and from instrument to instrument. + +\subsection{{\em messenger\_pdgid}} +\label{sec:messenger_pdgid} + +The optional attribute {\em messenger\/} encodes the {\em name\/} of the detected messenger particle. More generally, the particle astrophysics community, like the high energy particle physics community, use the Particle Data Group Identifier (PDG ID)\footnote{See \url{https://www.phy.bnl.gov/twister/bee/particles/}} to identify the precise messenger particle that is detected. + +We propose to add an optional attribute {\em messenger\_pdgid\/} that specifies the PDG ID of the messenger type for an observation. Constraints on {\em messenger\_pdgid\/} can provide a simple way to discover data sets for a specific messenger with a finer granularity than is possible with the {\em messenger} attribute alone. The value should be a string that specifies the messenger particle's PDG ID as `{\em $\pm$nn\/}' (where the sign is mandatory). For example, a muon, $\mu^{-}$, may be specified as `+13'. + +The difference between {\em messenger\/} and {\em messenger\_pdgid} is that ({\em e.g.\/}) a data discovery query that includes {\tt messenger = `neutrino'} would enable the user to identify datasets with a neutrino (any kind) messenger particle, whereas a data discovery query that includes {\tt messenger\_pdgid = `+16' OR messenger\_pdgid = `-16'} would restrict the search to include only $\tau$ neutrinos ($\nu_\tau$) or antineutrinos ($\nu_\tau^{-}$). + +The downside of {\em messenger\_pdgid} is that PDG ID very unlikely to be recognized outside of the particle astrophysics and high energy particle physics communities. We choose to include both {\em messenger\/} and {\em messenger\_pdgid} because the majority of astrophysicists --- even experienced high-energy astrophysicists --- are unlikely to recognize PDG ID\null. While any astrophysicist is likely to be able to write a query such as {\tt messenger = `photon'}, very few would be able to write that query as {\tt messenger\_pdgid = `+22'} from memory. + +We note that permissible {\em messenger\_pdgid\/} values will vary from facility to facility and from instrument to instrument. \subsection{Additional Columns} Similar to the ObsCore Recommendation, service providers may include additional columns to the ObsCore \gls{HEA} Extension table to expose additional metadata. While the service provider may choose to add additional columns to either the main ObsCore table or the ObsCore \gls{HEA} Extension table, we recommend that \gls{HEA}-data specific metadata columns be added to the ObsCore \gls{HEA} Extension table. +\subsection{Summary} + +Table~\ref{tab:obscore_hea} summarizes the proposal for the \gls{HEA} extension of ObsCore. We use the table name {\em ivoa.obscore\_hea\/} to describe the extension table here and in Appendix~\ref{sec:uc}. In addition to the table columns identified herein, we note that the {\em ivoa.obscore\_hea\/} table must also include the {\em obs\_publisher\_did\/} column that is shared exactly with the {\em ivoa.obscore\/} table so that a {\tt NATURAL JOIN} will yield one-to-one matched per-dataset rows of ObsCore and \gls{HEA} extension metadata. + +\begin{landscape} +\begin{center} +\begin{longtable}{ | m{0.15\linewidth} | m{0.23\linewidth} | m{0.07\linewidth} | m{0.07\linewidth} | m{0.4\linewidth} | m{0.05\linewidth} |} +\hline + +{\centering \bf Column Name} &{\centering \bf UCD} &{\centering \bf Unit} &{\centering \bf Type} &{\centering \bf Description} &{\centering \bf MAN}\\ +\hline +{\em ev\_xel\/} & \ucd{meta.number;instr.detection;phys.particle} & unitless & int & {Number of events in an {\bf hea-event-list}}& NO \\ +\hline +{\em s\_ref\_energy\/} & \ucd{meta.ref;em.energy;pos} & eV & float & {Energy at which the ObsCore spatial characterization attributes {\em s\_fov\/} , {\em s\_region\/}, {\em s\_resolution\/} are defined} & NO \\ +\hline +{\em em\_ref\_energy\/} & \ucd{meta.ref;em.energy;em} & eV & float & {Energy at which the ObsCore spectral characterization attributes {\em em\_res\_power\/}, {\em em\_resolution\/} are defined} & NO \\ +\hline +{\em s\_ref\_oaa\/} & \ucd{pos.posAng;instr.offset;pos} & deg & float & {Off-axis angle ({\em i.e.\/}, the angular separation of the target or source from the telescope optical axis) at which the ObsCore spatial characterization attributes {\em s\_fov\/} , {\em s\_region\/}, {\em s\_resolution\/} are defined} & NO \\ +\hline +{\em em\_ref\_oaa\/} & \ucd{pos.posAng;instr.offset;em} & deg & float & {Off-axis angle ({\em i.e.\/}, the angular separation of the target or source from the telescope optical axis) at which the ObsCore spectral characterization attributes {\em em\_res\_power\/}, {\em em\_resolution\/} are defined} & NO \\ +\hline +{\em t\_intervals\/} & \ucd{TBD}& unitless & TMOC & {List of observation intervals or stable/good time intervals describing the exact observation time coverage} & NO \\ +\hline +{\em energy\_min\/} & \ucd{em.energy;stat.min} & float & eV & {Energy associated to the ObCcore attribute {\em em\_max\/}, describing the minimal energy of the dataset} & NO \\ +\hline +{\em energy\_max\/} & \ucd{em.energy;stat.max} & float & eV & {Energy associated to the ObsCore attribute {\em em\_min\/}, describing the maximal energy of the dataset} & NO \\ +\hline +{\em obs\_mode\/} & \ucd{meta.code;obs.param} & unitless & string &{Observation mode of the observation ({\em e.g.\/}, TBU)} & NO \\ +\hline +{\em tracking\_mode\/} & \ucd{meta.code;obs.param} & unitless & string & {Tracking mode of an observation ({\em e.g.\/}, sidereal rate, moving target [solar system] tracking, drift scans)} & NO \\ +\hline +{\em analysis\_mode\/} & \ucd{meta.code;obs.param} & unitless & string &{Data reduction/analysis mode}& NO \\ +\hline +{\em event\_type\/} & \ucd{meta.code.qual;instr.detection;phys.particle} & unitless & string &{Data quality flag of the events ({\em e.g.\/}, ``good psf'', ``good rejection'', ``Nhit (100,200)''} & NO \\ +\hline +{\em messenger\/} & \ucd{meta.id;phys.particle} & unitless & string &{Messenger particle type ({\em e.g.\/}, ``photon'', ``cosmic-ray'', ``neutrino'')} & NO \\ +\hline +{\em messenger\_pdgid\/} & \ucd{meta.ref.pdgid;phys.particle} & unitless & string &{Messenger particle PDG ID ({\em e.g.\/}, ``+13'')} & NO \\ +\hline +\caption{\gls{HEA} ObsCore Extension Table Attributes. } +\label{tab:obscore_hea} +\end{longtable} +%\end{center} +\end{center} +\end{landscape} + \section{Vocabulary Enhancements} \label{sec:voc} @@ -381,7 +443,7 @@ \subsection{Evolution of the Data Product Type Vocabulary} The \gls{IVOA} Data Product Type Vocabulary\footnote{See \url{http://www.ivoa.net/rdf/product-type}.} provides terms, labels, and descriptions for many types of astronomical data products. However, there are some additions and changes that are appropriate to better support \gls{HEA} datasets. -We propose to add vocabulary entries for the new data product types outlined in \S~\ref{sec:dataproduct_type} and also propose to slightly modify the existing definition of {\bf hea-event-list} so that it aligns more accurately with the definition in that section. +We propose to add vocabulary entries for the new data product types outlined in \S~\ref{sec:dataproduct_type} and also propose to slightly modify the existing definition of {\bf event-list}, which we redesignate as {\bf hea-event-list}, so that it aligns more accurately with the definition in that section. %explain here the 2 vocabularies Additionally, we propose to add several more specific entries to the data product type vocabulary that specialize these types (especially {\bf response-function}). @@ -397,10 +459,10 @@ \subsubsection{Event List} \subsubsection{Event Bundle} -Some use cases require access to a bundle of datasets that includes the {\bf hea-event-list} and associated data products. We define an {\bf hea-event-bundle}: +Some use cases require access to a bundle of datasets that includes an {\bf hea-event-list} and associated data products. We define an {\bf hea-event-bundle}: \begin{quote} -{\bf hea-event-bundle}: A compounded dataset containing an {\bf hea-event-list} and multiple files or other substructures that are products necessary to analyze the hea-event-list. Data in an {\bf hea-event-bundle} may thus be used to produce higher level data products calibrated in physical units when containing \glspl{IRF} or other data products that can be used to construct \glspl{IRF}. +{\bf hea-event-bundle}: A compounded dataset containing an {\bf hea-event-list} and multiple files or other substructures that are products necessary to analyze the {\bf hea-event-list}. Data in an {\bf hea-event-bundle} may thus be used to produce higher level data products calibrated in physical units when containing \glspl{IRF} or other data products that can be used to construct \glspl{IRF}. \end{quote} An {\bf hea-event-bundle} might for example consist of an {\bf hea-event-list} and the associated {\bf response-function}s used to calibrate the dataset, and may also contain provenance information, data quality time-series, and preview images or plots. @@ -412,7 +474,7 @@ \subsubsection{Response Functions} We then propose to add the following data product type to define response functions (\glspl{IRF}). \begin{quote} - {\bf response-function}: A dataset that records the mapping from a physical quantity to an observable quantity. Narrower terms are preferred to indicate more precisely the type of {\bf response-function}. + {\bf response-function}: A dataset that records a mapping from a physical quantity to an observable quantity. Narrower terms are preferred to indicate more precisely the type of {\bf response-function}. \end{quote} The following data product types specialize {\bf response function}. Note that while most of these are primarily used in \gls{HEA}, the point spread function ({\bf psf}) is a {\bf response-function} that is generally applicable across multiple wavebands. @@ -435,12 +497,12 @@ \subsubsection{Response Functions} \subsubsection{Advanced Data Products} -In addition to data product types that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of advanced data products (typically with {\em calib\_level\/}${}\ge 3$) that may be generated from astronomical observations. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and observations) means that data providers may choose to provide such analysis products directly to the end user. +In addition to data product types that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of advanced data products (typically with {\em calib\_level\/}${}\ge 3$) that may be generated from astronomical observations. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and observations) means that data providers may choose to provide such advanced data products directly to the end user. Users will certainly want to discover these data products independently from the associated progenitor observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional data product types for these advanced data products, and note that these data product types will certainly be useful independent of waveband ({\em i.e.\/}, they can be equally applicable to UV/optical, IR, and radio datasets): \begin{quote} -{\bf draws}: A dataset that records statistical draws computed from a probability distribution, for example Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable. The draws can be interpreted to provide a robust estimation of the probability distribution of variable, and correlations between the draws provide information about how well the draws converge to the parent probability distribution. + {\bf draws}: A dataset that records statistical draws computed from a probability distribution or a sample population, for example Markov chain Monte Carlo (MCMC) draws used when computing the Bayesian marginal probability density function for a random variable, or the DeltaTS associated with a quantity from a frequentist analysis. The draws can be interpreted to provide a robust estimation of the probability distribution of variable, and correlations between the draws provide information about how well the draws converge to the parent probability distribution. {\bf pdf}: A dataset that records the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable. The probability density function provides a robust estimation of the variable and allows arbitrary confidence intervals to be computed directly from the distribution. @@ -494,28 +556,25 @@ \subsubsection{Instrument-related Quantities} We propose to add a new UCD {\em instr.detection\/} as the base of the hierarchy to describe instrument-related properties of particle events detected by \gls{HEA} detectors. To be more precise, an event in the event list will be tagged as {\em instr.detection;phys.particle\/}. Initially, we propose a small set of event-related UCDs that identify key properties that are particularly important for \gls{HEA} data analysis. -\paragraph{Event Grade} - -For imaging X-ray instruments (especially those based on CCD detectors), detected events typically deposit charge into more than a single detector pixel. The events are assigned a ``grade'' based on how charge is deposited into the central pixel and surrounding pixels, and the grade information is essential for data analysis since typically only a subset of grades will correspond to valid events. We propose to use these combined UCD terms {\em meta.code.class;instr.detection;phys.particle\/} to provide the UCD string for the {\bf event\_grade} column. - \paragraph{Pulse Height} For many X-ray and gamma-ray instruments, the signal observed in a given detector spectral channel is the result of event counting and would typically be recorded as a Pulse Height Amplitude (PHA), or perhaps a Pulse Invariant (PI) value that is calculated from PHA by applying an appropriate gain calibration. The PHA (or PI) can be related to the incident particle energy by applying the appropriate {\bf response-function}, and higher data calibration level products may replace or augment these values with quantities such as energy, or perhaps particle or energy flux. There is currently no UCD defined for a raw pulse height amplitude measure like PHA (or PI). PHA is such an important quantity to \gls{HEA} datasets that we propose adding a new UCD {\em instr.pulse\/} for these raw data values. {\em instr.pulse\/} can be combined with more general UCD terms to describe pulse parameters recorded by the pulse height analyzer electronics. Typically, PHA corresponds to the integrated signal detected by the electronics, which could therefore be described using the UCD {\em instr.pulse;arith.sum}. For some detectors, additional pulse parameters such as the peak pulse height or pulse width are important, and could be described by UCDs {\em instr.pulse;stat.max} and {\em instr.pulse;stat.fwmh}, respectively, and so on. We note that the background signal (both of instrumental and cosmological origin) may be significant for many \gls{HEA} detectors and so the detected events may be unrelated to any observed source on the sky. For Cherenkov neutrino detectors, this quantity might refer to the number of observed photon hits from secondary particle photon generation. +\paragraph{Event Grade} + +For imaging X-ray instruments (especially those based on CCD detectors), detected events typically deposit charge into more than a single detector pixel. The events are assigned a ``grade'' based on how charge is deposited into the central pixel and surrounding pixels, and the grade information is essential for data analysis since typically only a subset of grades will correspond to valid events. The combined UCD terms {\em meta.code.class;instr.detection;phys.particle\/} may be used to provide the UCD string for data columns encoding event grade information. \paragraph{Event Type} -For \gls{VHE} (and GeV) gamma-ray data, there is the notion of event type (see \S~\ref{sec:evttype}) that can be mandatory for some data releases. We propose to use a combined UCD \\{\em meta.code.qual;instr.detection;phys.particle\/} \\ to tag the proposed {\bf event\_type} parameter. +For \gls{VHE} (and GeV) gamma-ray data, there is the notion of event type (see \S~\ref{sec:evttype}) that can be mandatory for some data releases. The combined UCD terms {\em meta.code.qual;instr.detection;phys\-.particle\/} may be used to provide the UCD string for data columns encoding event type information. \subsubsection{Physical Quantities} The messengers for \gls{HEA} observations may include particles other than the ones currently described in the UCD list. Because some instruments can now distinguish electrons from positrons\footnote{For example, the Fermi-LAT instrument.}, as well antiprotons from protons\footnote{For example, the AMS-2 experiment.}, we also propose to add {\em phys.particle.positron\/} and {\em phys.particle.antiproton\/}, as well as {\em phys.particle.cosmicray\/} and unify them all under the {\em phys.particle\/} UCD hierarchy. -%Mireille clarify the usage of pdgid , as a value in the messenger column. -For particle detectors, a wide range of different particles might have to be described. As is customary in particle physics, we propose to facilitate the use of the Particle Data Group Identifier\footnote{see \url{https://www.phy.bnl.gov/twister/bee/particles/}} as reference for any particle. For instance, we can populate the {\em messenger\/} column describing {\em e.g.\/}, $\nu_{\tau}$ and $\bar{\nu}_{\tau}$ with {\em pdgid+16\/} and {\em \hbox{pdgid-16}\/} respectively. -% $\nu_{\tau}$ and $\bar{\nu}_{\tau}$ as {\em pdgid+16\/} and {\em \hbox{pdgid-16}\/} +For particle detectors, a wide range of different particles might have to be described. As is customary in particle physics, we propose to facilitate the use of the Particle Data Group Identifier (PDG ID)\footnote{see \url{https://www.phy.bnl.gov/twister/bee/particles/}} as reference for any particle. A data column that lists the PDG ID of the messenger particle would be described by the UCD string {\em meta.ref.pdgid;phys.particle\/}. One should note that electrons are denoted by the UCD {\em phys.electron\/} in the current version of the UCD list \citep{2024ivoa.spec.1218C} and are not grouped under the {\em phys.particle\/} hierarchy. Marking {\em phys.electron\/} (and {\em phys.electron.degen\/}) as obsolete or not recommended and adding the term {\em phys.particle.electron\/} to the UCD list would improve the consistency of the {\em phys.particle\/} branch. @@ -555,7 +614,8 @@ \subsubsection{Evolution of UCD list} Q & {\em instr.detection\/} & Particle event detection \cr %Q & {\em instr.event.grade\/} & Particle event grade \cr Q & {\em instr.pulse\/} & Pulse height amplitude measure \cr -%Q & {\em instr.event.type\/} & Particle event type \cr +% Q & {\em instr.event.type\/} & Particle event type \cr +P& {\em meta.ref.pdgid\/} & Particle Data Group identifier encoding a type of particle \cr E & {\em phot.count.density\/} & Count flux density (dimensionality: $\rm [L^{-2}\,T^{-1}\,E^{-1}]$) \cr E & {\em phot.count.density.sb\/} & Count flux density surface brightness (dimensionality: $\rm [L^{-2}\,T^{-1}\,E^{-1}\,\hbox{sr}^{-1}]$) \cr E & {\em phot.count.radiance\/} & Count flux radiance (dimensionality: $\rm [L^{-2}\,T^{-1}\,\hbox{sr}^{-1}]$) \cr @@ -575,7 +635,6 @@ \subsubsection{Evolution of UCD list} S & {\em phys.particle.photon\/} & Related to photon \cr S & {\em phys.particle.positron\/} & Related to positron \cr %S & {\em phys.particle.pdgid\/} & Particle Data Group Identifier \cr -P& meta.ref.pdgid &Particle data group identifier encoding a type of particle \cr % we stated with semantics to use meta.ref.pdgdid;phys.particle for messenger %S & {\em phys.particle.pdgid$\pm$XX\/} & Related to a particle with PDG ID $\pm$XX \cr %mireille @@ -595,13 +654,13 @@ \subsubsection{Evolution of UCD list} \textbf{} & \textbf{UCD word} & \textbf{Description}\cr \sptablerule S & {\em em.gamma.hard\/} & Hard gamma ray (500 keV -- 100 MeV) \cr -Q & {\em stat.confidenceLevel\/} & Level of confidence for a statistical measure, detection, or classification process \cr E & {\em phot.count\/} & Count flux (dimensionality: $\rm [L^{-2}\,T^{-1}]$) \cr E & {\em phot.fluence\/} & Radiant photon energy received by a surface per unit area or irradiance of a surface integrated over time of irradiation (dimensionality: $\rm [L^{-2}]$) \cr Q & {\em phot.flux.bol\/} & Bolometric flux (dimensionality: $\rm [M\,T^{-3}]$) \cr E & {\em phot.radiance\/} & Radiance as energy flux per solid angle (dimensionality: $\rm [M\,T^{-3}\,\hbox{sr}^{-1}]$) \cr %mir the case of electron would be in a VEP to discuss the backward compatibility of this change S & {\em phys.electron\/} & Electron (not recommended/deprecate) \cr +Q & {\em stat.confidenceLevel\/} & Level of confidence for a statistical measure, detection, or classification process \cr S & {\em stat.min\/} & Minimum value \cr S & {\em stat.max\/} & Maximum value \cr \sptablerule @@ -619,7 +678,7 @@ \subsection{MIME-type Enhancements}\label{sec:mimetypes} Many \gls{HEA} FITS format data products will comply with existing MIME-types discussed in the ObsCore Recommendation, such as {\bf application/fits} or {\bf application/x-fits-bintable}, but specialized FITS formats have been established for high energy data. -Many data collections distributed for instance at HEASARC adopted the OGIP format in FITS. +Many data collections distributed for instance at HEASARC adopted the OGIP format in FITS\null. The gamma-ray community has developed two additional data format standards, so we propose to add the following MIME-types: \begin{itemize} @@ -628,48 +687,6 @@ \subsection{MIME-type Enhancements}\label{sec:mimetypes} \item {\bf x-fits-vodf}: for FITS files following the Very-high-energy Open Data Format (VODF) specification \citep{2023arXiv230813385K}. \end{itemize} -\section{Proposed ivoa.obscore\_hea Table Attributes}\label{sec:ibscoreext} - -This section summarizes the proposal for the \gls{HEA} extension of ObsCore. We use the term {\em ivoa.obscore\_hea\/} to described the extension here and in Appendix~\ref{sec:uc}. - -\begin{landscape} -\begin{center} -\begin{longtable}{ | m{0.15\linewidth} | m{0.23\linewidth} | m{0.07\linewidth} | m{0.07\linewidth} | m{0.4\linewidth} | m{0.05\linewidth} |} -\hline - -{\centering \bf Column Name} &{\centering \bf UCD} &{\centering \bf Unit} &{\centering \bf Type} &{\centering \bf Description} &{\centering \bf MAN}\\ -\hline - ev\_xel & \ucd{meta.number;instr.detection;phys.particle} & unitless & int & {Number of events in an {\bf hea-event-list}}& NO \\ -\hline - s\_ref\_energy & \ucd{meta.ref;em.energy;pos} & eV & float & {Energy at which the ObsCore spatial characterization attributes s\_fov , s\_region, s\_resolution are defined} & NO \\ -\hline - em\_ref\_energy & \ucd{meta.ref;em.energy;em} & eV & float & {Energy at which the ObsCore spectral characterization attributes em\_res\_power, em\_resolution are defined} & NO \\ -\hline - s\_ref\_oaa & \ucd{pos.posAng;instr.offset;pos} & deg & float & {Off-axis angle ({\em i.e.\/}, the angular separation of the target or source from the telescope optical axis) at which the ObsCore spatial characterization attributes s\_fov , s\_region, s\_resolution are defined} & NO \\ -\hline - em\_ref\_oaa & \ucd{pos.posAng;instr.offset;em} & deg & float & {Off-axis angle ({\em i.e.\/}, the angular separation of the target or source from the telescope optical axis) at which the ObsCore spectral characterization attributes em\_res\_power, em\_resolution are defined} & NO \\ -\hline - t\_intervals & \ucd{TBD}& unitless & TMOC & {List of observation intervals or stable/good time intervals describing the exact observation time coverage} & NO \\ -\hline - energy\_min & \ucd{em.energy;stat.min} & float & eV & {Energy associated to the ObCcore attribute em\_max, describing the minimal energy of the dataset} & NO \\ -\hline - energy\_max & \ucd{em.energy;stat.max} & float & eV & {Energy associated to the ObsCore attribute em\_min, describing the maximal energy of the dataset} & NO \\ -\hline - obs\_mode & \ucd{meta.code;obs.param} & unitless & string &{Observation mode of the observation ({\em e.g.\/}, TBU)} & NO \\ -\hline - tracking\_mode & \ucd{meta.code;obs.param} & unitless & string & {Tracking mode of an observation ({\em e.g.\/}, sidereal rate, moving target [solar system] tracking, drift scans)} & NO \\ -\hline - analysis\_mode & \ucd{meta.code;obs.param} & unitless & string &{Data reduction/analysis mode}& NO \\ -\hline - event\_type & \ucd{meta.code.qual;instr.detection;phys.particle} & unitless & string &{Data quality flag of the events ({\em e.g.\/}, ``good psf'', ``good rejection'', ``Nhit (100,200)''} & NO \\ -\hline - messenger & \ucd{meta.name;phys.particle} & unitless & string &{Messenger particle type ({\em e.g.\/}, ``photon'', ``cosmic-ray'', ``neutrino'', ``pdgid-13'')} & NO \\ -\hline -\end{longtable} -%\end{center} -\end{center} -\end{landscape} - \input{extendedAccessTonewtypesOfproducts.tex} diff --git a/Makefile b/Makefile index a4deef9..80c3173 100644 --- a/Makefile +++ b/Makefile @@ -8,7 +8,7 @@ DOCNAME = HighEnergyObsCoreExt DOCVERSION = 1.0 # Publication date, ISO format; update manually for "releases" -DOCDATE = 2026-06-24 +DOCDATE = 2026-07-04 # What is it you're writing: NOTE, WD, PR, REC, PEN, or EN DOCTYPE = PEN diff --git a/UseCases.tex b/UseCases.tex index d62db61..22c7aa2 100644 --- a/UseCases.tex +++ b/UseCases.tex @@ -289,7 +289,8 @@ \subsubsection{Use Case --- Search for all ANTARES neutrino data products for a NATURAL JOIN ivoa.obscore_hea WHERE (CONTAINS(POINT('ICRS', 98.24, 5.81), CIRCLE('ICRS', s_ra, s_dec, s_fov)) = 1) -AND (dataproduct_type IN ('hea-event-bundle', 'hea-event-list', 'response-function')) +AND (dataproduct_type IN ('hea-event-bundle', 'hea-event-list', +'response-function')) AND (obs_collection = 'ANTARES-2017-PS') \end{verbatim} @@ -358,7 +359,7 @@ \subsubsection{Use Case --- Calculate the probability for a source class to be e \noindent Find all neutrino datasets satisfying: \begin{enumerate}[(i)] \item dataproduct\_type = ``response-function'', dataproduct\_subtype = ``aeff`` - \item messenger contains ``pdgid-16'' or ``pdgid+16'', + \item messenger\_pdgid contains ``$-16$'' or ``$+16$'', \item obs\_mode = ``wide-array'', \item analysis\_mode = ``pointsource''. \end{enumerate} @@ -369,7 +370,7 @@ \subsubsection{Use Case --- Calculate the probability for a source class to be e WHERE (dataproduct_type = 'response-function') AND (dataproduct_subtype = 'aeff') -AND (messenger = 'pdgid-16' OR messenger = 'pdgid+16') +AND (messenger_pdgid = '-16' OR messenger_pdgid = '+16') AND (obs_mode LIKE '%wide-array%') AND (analysis_mode LIKE '%pointsource%') \end{verbatim} @@ -457,7 +458,7 @@ \subsubsection{Use Case --- Search for M31 source light curves and aperture phot AND ((dataproduct_type = 'light-curve') OR (dataproduct_type = 'pdf')) AND (calib_level = 4) AND (energy_min <= 300.0) AND (energy_max >= 7000.0) -AND (INTERSECTS(TMOC(17, t_intervals), TMOC(17, 56320.0, 56325.0)) = 1) +AND (INTERSECTS(TMOC(27, t_intervals), TMOC(27, 56320.0, 56325.0)) = 1) \end{verbatim} \subsubsection{Use Case --- Search for the CTAO flux light curves of PKS 2155-304 in 2030} diff --git a/extendedAccessTonewtypesOfproducts.tex b/extendedAccessTonewtypesOfproducts.tex index 48f9593..49ba526 100644 --- a/extendedAccessTonewtypesOfproducts.tex +++ b/extendedAccessTonewtypesOfproducts.tex @@ -2,94 +2,56 @@ %\newpage \newcommand{\blinks}{\{links\}} % François Version 2: -\section{Extending access to new types of data products in ObsTAP services} +%\section{Extending access to new types of data products in ObsTAP services}\label{sec:accessoptions} +\section{Possible Approaches for Accessing \gls{HEA} Data Products Using ObsTAP Services}\label{sec:accessoptions} %\section{Access methods for data related to sky datasets discovered with ObsTAP} %(Various implementation strategies for data access) -\subsection{Difference between hea-event-list, hea-event-bundle, response function and analysis data products} -As discussed in the previous section, high energy data distribution requires a special focus on event lists and associated response functions as well as analysis data products. -Event lists without corresponding response functions do not allow data interpretation. That’s the reason why hea-event-lists are often gathered in the same package with the response function in an “hea-event-bundle” with a specific format (OGIP, GADF or VODF). In addition, the High Energy astronomical community has developed a lot of probabilistic analysis methods to analyse and extract information from the event lists. They cannot all be described by ObsCore parameters due to the richness and complexity of their parameters. They are listed in section~\ref{sec:voc_product_type}. Access to this category of data can be managed similarly to response functions in some cases, as proposed in section \ref{sec:analysis-dp}. +%\subsection{Difference between hea-event-list, hea-event-bundle, response function and analysis data products} +%As discussed in the previous section, high energy data distribution requires a special focus on event lists and associated response functions as well as analysis data products. +%Event lists without corresponding response functions do not allow data interpretation. That’s the reason why hea-event-lists are often gathered in the same package with the response function in an “hea-event-bundle” with a specific format (OGIP, GADF or VODF). In addition, the High Energy astronomical community has developed a lot of probabilistic analysis methods to analyse and extract information from the event lists. They cannot all be described by ObsCore parameters due to the richness and complexity of their parameters. They are listed in section~\ref{sec:voc_product_type}. Access to this category of data can be managed similarly to response functions in some cases, as proposed in section \ref{sec:analysis-dp}. -\subsection{Direct access to an hea-event-bundle} - The simplest method to access hea-event-list and associated response function is to provide a link to an hea-event-bundle package; this can be done by providing an URL to the package in the {\em access\_url \/} FIELD of the ObsCore (or ObsCore + extension) table. The {\em access\_format \/} FIELD will contain the appropriate MIME type for the bundle package (for example x-fits-ogip, x-fits-gadf, or x-fits-vodf). Table~\ref{tab:bundle} shows an excerpt of an ObsCore result and illustrates this method. This method can be combined with methods described below in~\ref{sec:datalink} for a common access to the bundle and other resources such as previews or some specific analysis data products. +As discussed previously, \gls{HEA} data distribution requires special focus on event lists and their associated \glspl{IRF}, as well as on advanced data products. In most cases, corresponding \glspl{IRF} are required to enable physical interpretation of event list data, at least for some physical axes. That is one reason why data providers may prefer to package an {\bf hea-event-list} together with the associated {\bf response-function}s, or with other ancillary data products that enable end-users to construct appropriate \glspl{IRF}, in a single package. We have proposed an {\bf hea-event-bundle} data product type to represent such packages, which may be recorded in a specific format such as OGIP FITS, GADF FITS, or VODF FITS, or may simply consist of a set of data products in a common archive file format such as a tarball. +In other cases, recognizing the computation complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime, \gls{HEA} data providers may choose to provide advanced data products, such as probability density functions or statistical draws, directly to the user community. In most cases, these advanced data products have spatial, spectral, and/or temporal coverage that can be directly described using standard ObsCore attributes, and therefore these data products can be discovered directly. In other cases, these data products may not be describable using standard ObsCore parameters ({\em e.g.\/}, if they depend on non-sky based parameters such as telescope altitude and azimuth). +%%- Access to this category of data may be managed similarly to {\bf response-function}s in some cases, as proposed in section \ref{sec:analysis-dp}. + +In this section, we first discuss direct access to individual \gls{HEA} data products (including an {\bf hea-event-bundle}), and then focus on several possible approaches that use DataLink \citep{2023ivoa.spec.1215B} to facilitate access to additional datasets that may be associated with a primary datasets ({\em e.g.\/}, {\bf response-function}s associated with a primary {\bf hea-event-list}). + +The choice of access method is entirely up to the data provider. + +If a data collection consists primarily of individual observations, each with a small set of associated data products such as \glspl{IRF} that are required for further data analysis (this is a very common use case), then either direct access to the {\bf hea-event-bundle}s or access via DataLink is very likely preferable. + +However, if the data collection includes data products such as spectra, light-curves, or advanced data products ({\em e.g.\/}, aperture photometry probability density functions) that are extracted from the primary {\bf hea-event-list} and that can be used independently of the primary dataset, then direct access to these products is likely preferable from the user's perspective. Since \gls{HEA} event lists typically simulateously encode spatial, spectral, and temporal information, a very large amount of data may be associated with a single {\bf hea-event-list}. For example, roughly 4,600 individual detectable X-ray sources are located in a single Chandra observation field of view centered on the Galactic center. Using DataLink to the primary {\bf hea-event-list} to access these spectra, light-curves, and advanced data products would be unwieldy, especially when only a small subset of the products are required. + + +\subsection{Direct Access to a Data Product or {\bf hea-event-bundle}} +% The simplest method to access hea-event-list and associated response function is to provide a link to an hea-event-bundle package; this can be done by providing an URL to the package in the {\em access\_url \/} FIELD of the ObsCore (or ObsCore + extension) table. The {\em access\_format \/} FIELD will contain the appropriate MIME type for the bundle package (for example x-fits-ogip, x-fits-gadf, or x-fits-vodf). Table~\ref{tab:bundle} shows an excerpt of an ObsCore result and illustrates this method. This method can be combined with methods described below in~\ref{sec:datalink} for a common access to the bundle and other resources such as previews or some specific analysis data products. + +The simplest method to directly access any data product is to simply provide a direct link to the data product. This can be done by providing a URL to the product in the {\em access\_url\/} attribute of the ObsCore table. The {\em access\_format\/} attribute will contain the appropriate MIME type for the data product ({\em e.g.\/}, {\tt x-fits-bintable}). In the case of an {\bf hea-event-list} together with associated data products such as \glspl{IRF}, a link to an {\bf hea-event-bundle} package may be provided. In this case, the {\em access\_format \/} FIELD will contain the appropriate MIME type for the bundle package ({\em e.g.\/}, {\tt x-fits-ogip}, {\tt x-fits-gadf\/}, {\tt x-fits-vodf}, {\tt application/x-tar}). Table~\ref{tab:bundle} shows an excerpt of an ObsCore result and illustrates this method. This method can be combined with methods described below in~\ref{sec:datalink} for a common access to the bundle and other resources such as previews or some specific advanced data products. %\begin{landscape} -\begin{longtable}{|p{4.1cm}|p{4.4cm}|p{4.4cm}|} -\sptablerule -\textbf{\em Column name\/}&\textbf{\em Row 1 value\/}&\textbf{\em Row 2 value\/}\\ -\sptablerule -\sptablerule - \textbf{dataproduct\_type}& hea-event-list&hea-event-list\\ -\sptablerule - \textbf{dataproduct\_subtype}& GADF DL3&GADF DL3\\ -\sptablerule -\textbf{calib\_level}& 2&2\\ -\sptablerule - \textbf{obs\_collection}& HESS-DL3-DR1&HESS-DL3-DR1\\ -\sptablerule - \textbf{obs\_id}&20136&20317\\ -\sptablerule - \textbf{obs\_title}& & \\ -\sptablerule - \textbf{obs\_publisher\_did}&ivo:\/\/padc.obspm\/hess\#20136&ivo:\/\/padc.obspm\/hess\#20137\\ -\sptablerule - \textbf{obs\_creator\_did}&&\\ -\sptablerule -\textbf{access\_url}& \footnotesize{https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz}&\footnotesize{https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020137.fits.gz}\\ -\sptablerule - \textbf{access\_format}&x-fits-gadf&x-fits-gadf\\ -\sptablerule -\textbf{access\_estsize}&414720&216000\\ -\sptablerule - \textbf{target\_name}&MSH15-52&MSH15-52\\ -\sptablerule - \textbf{target\_class} &&\\ -\sptablerule - \textbf{s\_ra}&228.6125&228.6125\\ -\sptablerule - \textbf{s\_dec}&-58.771667&-59.77771677\\ -\sptablerule -\textbf{s\_fov}&5.0&5.0\\ -%\sptablerule \newpage -\sptablerule - \textbf{s\_region}&&\\ -\sptablerule - \textbf{s\_resolution}&0.10000000149011612&0.10000000149011612\\ -\sptablerule - \textbf{t\_min}&53090.123451203704&53090.144735925925\\ -\sptablerule - \textbf{t\_max}&53090.14291879629&53090.155175740736\\ - \sptablerule - \textbf{t\_exptime}&1521.02685546875&819.2053833007812\\ -\sptablerule -\textbf{t\_resolution}&9.999999974752427E-7&9.999999974752427E-7\\ -\sptablerule -\textbf {em\_min}&1.1889202653543914E-20&1.208503847135941E-20\\ -\sptablerule - \textbf{em\_max}&5.5549210638422004E-18&5.2809212082467665E-18\\ -\sptablerule -\textbf {em\_res\_power}&&\\ -\sptablerule - \textbf{o\_ucd}&time;pos;phys.energy&time;pos;phys.energy\\ -\sptablerule - \textbf{pol\_states}& &\\ -\sptablerule - \textbf{facility\_name}&H.E.S.S.&H.E.S.S\\ -\sptablerule - \textbf{instrument\_name}& 1,2,3,4&1,2,3,4\\ -\sptablerule - \textbf{s\_xel1}&-1&-1\\ -\sptablerule - \textbf{s\_xel2}& -1&-1\\ -\sptablerule - \textbf{t\_xel}&-1&-1\\ -\sptablerule - \textbf{em\_xel}& -1&-1\\ -\sptablerule -\textbf{pol\_xel}& -1&-1\\ -\sptablerule -\caption{ObsCore response for hea-event-bundles (inspired from the H.E.S.S. ObsTAP prototype at Paris Observatory)} +\begin{longtable}{|m{0.22\linewidth}|m{0.34\linewidth}|m{0.34\linewidth}|} +\hline +{\bf Attribute}&{\bf Row 1 Value}&{\bf Row 2 Value}\\ +%\hline +\hline +{\em dataproduct\_type\/}& {\tt hea-event-bundle}&{\tt hea-event-bundle}\\ +\hline +%{\em dataproduct\_subtype\/}&{\tt GADF DL3}&{\tt GADF DL3}\\ +%\hline +%{\em calib\_level\/}& 2&2\\ +%\hline +{\em obs\_collection\/}& HESS-DL3-DR1&HESS-DL3-DR1\\ +\hline +{\em obs\_id\/}&20136&20317\\ +\hline +{\em access\_url\/}& \footnotesize{https://hess-dr.obspm.fr/\newline retrieve/hess\_dl3\_dr1\_obs\newline\_id\_020136.fits.gz}&\footnotesize{https://hess-dr.obspm.fr/\newline retrieve/hess\_dl3\_dr1\_obs\newline\_id\_020137.fits.gz}\\ +\hline +{\em access\_format\/}&x-fits-gadf&x-fits-gadf\\ +\hline +%{\em access\_estsize\/}&414720&216000\\ +\caption{Excerpt of the ObsCore Response for {\bf hea-event-bundle}s (Inspired by the \gls{HESS} ObsTAP Prototype at Paris Observatory).} \label{tab:bundle} \end{longtable} %\end{landscape} @@ -97,31 +59,34 @@ \subsection{Direct access to an hea-event-bundle} \subsection{Access via DataLink} \label{sec:datalink} -DataLink \citep{2023ivoa.spec.1215B} allows to relate a main record provided by a service (ObsTAP, SSA/SIA or a catalogue service) to a various number of links which complement the information contained in the record. An implementation note \citep{2024ivoa.note.datalink} gives some hints on how to use this standard. -There are at least three ways to implement this standard for hea-event-lists depending how many steps are used in the access method to the associated datasets.\\ + +DataLink \citep{2023ivoa.spec.1215B} allows a data provider to relate a main record provided by a service (ObsTAP, SSA/SIA, or a catalog service) to a arbitrary set of links that complement the information contained in the record. An implementation note \citep{2024ivoa.note.datalink} provides some hints on how to use this standard. +There are at least three ways to implement this standard for {\bf hea-event-list}s depending how many steps are used in the access method to the associated datasets: \begin{itemize} - \item Subsection~\ref{sec:datalinkinobs} is a full 2-step approach, - \item Subsection~\ref{sec:datalinksd} allows both direct access to hea-event-list and 2-step access to the additional material, - \item Subsection~\ref{sec:datalinktap} is not described in the DataLink standard but is fully consistant with TAP and the DataLink \blinks\ endpoint response specification. It allows direct access to both hea-event-list and additional material, like response files, or analysis data products. + \item Subsection~\ref{sec:datalinkinobs} outlines the standard two-step approach; + \item Subsection~\ref{sec:datalinksd} demonstrates both direct access to the {\bf hea-event-list} and two-step access to the additional material; + \item Subsection~\ref{sec:datalinktap} allows direct access to both the {\bf hea-event-list} plus additional data products, such as \glspl{IRF} or advanced data products. This approach is not described in the DataLink standard but is fully consistant with TAP and the DataLink \blinks\ endpoint response specification. \end{itemize} - \subsubsection{From an hea-event-list in ObsCore: DataLink to access the event list itself and response functions via {\em access\_url \/} } +\subsubsection{From an {\bf hea-event-list} in ObsCore: DataLink to Access the Event List and Response Functions via {\em access\_url\/}} \label{sec:datalinkinobs} - The { \em data\_product type \/} exposed in ObsCore is “hea-event-list” and the DataLink \blinks\ endpoint response points to the event list itself and to the various response functions individually. This solution allows retrieval of data with explicit access to the various components of a “bundle” which doesn't need to be packaged in that case. In addition to response functions, analysis data products can also be linked to the event list this way. Figure~\ref{fig:obscoretodl} shows an ObsCore response where the selected record points to a \blinks\ endpoint response illustrating this method. Table~\ref{tab:datalink} shows this DataLink response content. -\begin{figure} - -\includegraphics[width=1.5\textwidth, angle=90]{Obscore+datalink.png} -\caption{Selection of H.E.S.S. data from Aladin Desktop: In the galactic center area, an ObsTAP query has been performed and H.E.S.S. datasets are discovered. One of the ObsCore record has been selected and points to a DataLink response, with 8 links provided in the white box. The content of this DataLink table is described in Table~\ref{tab:datalink}. } -\label{fig:obscoretodl} -\end{figure} +The {\em dataproduct\_type\/} exposed in ObsCore is {\bf hea-event-list} and the Data\-Link \blinks\ endpoint response points to the event list and to the various {\bf response-function}s individually. This solution allows retrieval of data with explicit access to the various components of a ``bundle,'' which doesn't need to be packaged in that case. In addition to \glspl{IRF}, additional data products, including advanced data products can also be linked to the {\bf hea-event-list} this way. +% Figure~\ref{fig:obscoretodl} shows an ObsCore response where the selected record points to a \blinks\ endpoint response illustrating this method. +Table~\ref{tab:datalink} provides example DataLink response content to a query for \gls{HESS} Data Level 3 Public Test Data. +%\begin{figure} +% +%\includegraphics[width=1.5\textwidth, angle=90]{Obscore+datalink.png} +% \caption{Selection of H.E.S.S. data from Aladin Desktop: In the galactic center area, an ObsTAP query has been performed and H.E.S.S. datasets are discovered. One of the ObsCore record has been selected and points to a DataLink response, with 8 links provided in the white box. The content of this DataLink table is described in Table~\ref{tab:datalink}. } +%\label{fig:obscoretodl} +%\end{figure} \begin{landscape} \begin{center} -\begin{longtable}{|p{0.08\linewidth}|p{0.3\linewidth}|p{0.08\linewidth}|p{0.22\linewidth}|p{0.1\linewidth}|p{0.14\linewidth}|p{0.13\linewidth}|} +\begin{longtable}{|m{0.08\linewidth}|m{0.3\linewidth}|m{0.08\linewidth}|m{0.22\linewidth}|m{0.1\linewidth}|m{0.14\linewidth}|m{0.13\linewidth}|} \hline%\sptablerule \textbf{ID} &\textbf{\footnotesize access\_url} &\textbf{\footnotesize semantics}&\textbf{\footnotesize description} &\textbf{\footnotesize content\_type} &\textbf{\footnotesize content\_qualifier}\cr \hline%\sptablerule -{\footnotesize 20136} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz\#EVENTS} & {\footnotesize \#this} & {\footnotesize Event list dataset (HDU=EVENTS) } & {\footnotesize x-fits-gadf} & {\footnotesize hea-event-list} \cr +{\footnotesize 20136} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz\#EVENTS} & {\footnotesize \#this} & {\footnotesize Event list dataset\newline (HDU=EVENTS) } & {\footnotesize x-fits-gadf} & {\footnotesize hea-event-list} \cr \hline%\sptablerule {\footnotesize 20136} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136\_image.png} & {\footnotesize \#preview-image} & {\footnotesize Preview image of the observation} & {\footnotesize image/png} & {\footnotesize preview} \cr \hline%\sptablerule @@ -129,25 +94,26 @@ \subsection{Access via DataLink} \hline%\sptablerule {\footnotesize 20136} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz\#AEFF } & {\footnotesize \#calibration} & {\footnotesize Effective Area (HDU=AEFF)} & {\footnotesize x-fits-gadf} & {\footnotesize aeff} \cr \hline%\sptablerule -{\footnotesize 20316} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz\#EDISP} & {\footnotesize \#calibration} & {\footnotesize Energy Dispersion (HDU=EDISP)} & {\footnotesize x-fits-gadf} & {\footnotesize edisp} \cr +{\footnotesize 20316} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz\#EDISP} & {\footnotesize \#calibration} & {\footnotesize Energy Dispersion\newline (HDU=EDISP)} & {\footnotesize x-fits-gadf} & {\footnotesize edisp} \cr \hline%\sptablerule -{\footnotesize 20316} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz\#PSF} & {\footnotesize \#calibration} & {\footnotesize Point Spread Function (HDU=PSF) } & {\footnotesize x-fits-gadf} & {\footnotesize psf} \cr +{\footnotesize 20316} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz\#PSF} & {\footnotesize \#calibration} & {\footnotesize Point Spread Function\newline (HDU=PSF) } & {\footnotesize x-fits-gadf} & {\footnotesize psf} \cr \hline%\sptablerule {\footnotesize 20316} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz\#BKG} & {\footnotesize \#calibration} & {\footnotesize Background Rate (HDU=BKG) } & {\footnotesize x-fits-gadf} & {\footnotesize bkgrate} \cr \hline%\sptablerule -{\footnotesize 20316} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz} & {\footnotesize \#package} & {\footnotesize Event bundle (including HDU=EVENTS,GTI,AEFF,\newline EDISP,PSF,BKG) } & {\footnotesize x-fits-gadf} & {\footnotesize hea-event-bundle } \cr +{\footnotesize 20316} & {\footnotesize https://hess-dr.obspm.fr/retrieve/hess\_dl3\newline \_dr1\_obs\_id\_020136.fits.gz} & {\footnotesize \#package} & {\footnotesize Event bundle (including\newline HDU=EVENTS,GTI,AEFF,\newline EDISP,PSF,BKG) } & {\footnotesize x-fits-gadf} & {\footnotesize hea-event-bundle } \cr \hline%\sptablerule -\caption{DataLink response table attached to an hea-event-list record in ObsCore. Inspired from Paris Observatory H.E.S.S. ObsTAP prototype} +\caption{DataLink Response Table Attached to an {\bf hea-event-list} record in ObsCore. Inspired from Paris Observatory \gls{HESS} ObsTAP Prototype.} \label{tab:datalink} \end{longtable} \end{center} \end{landscape} - \subsubsection{Datalink access using a service descriptor} +\subsubsection{Datalink Access Using a Service Descriptor} \label{sec:datalinksd} - To avoid preventing a direct access to the hea-event-list while keeping the explicit access to the various response functions, it is possible to add a “DataLink” service descriptor in the ObsTAP result as can be seen in the example below. +To avoid preventing a direct access to the {\bf hea-event-list} while keeping the explicit access to the various response functions, it is possible to add a ``DataLink'' service descriptor in the ObsTAP result as can be seen in the example below. +{\small \begin{verbatim} Datalink service @@ -162,139 +128,180 @@ \subsection{Access via DataLink} value="application/x-votable+xml;content=datalink"/> \end{verbatim} +} + +In this case the service descriptor provides the root URL to the DataLink \blinks\ endpoint. The full \blinks\ endpoint URL for a specific row (or group of rows) is built by adding the content of the referred ID FIELD this way: + +\noindent http://provider.org/dataservice/datalink?ID={\em referred\_ID\_FIELD\_content}. - In this case the service descriptor provides the root URL to the DataLink \blinks\ endpoint. The full \blinks\ endpoint URL for a specific row (or group of rows) is then built by adding the content of the referred ID FIELD this way: \\ http://provider.org/dataservice/datalink?ID={\it referred\_ID\_FIELD\_content}. - It is important to notice that it is not mandatory that the {\it referred\_ID\_FIELD} be {\em obs\_publisher\_did/}. It can be {\em obs\_id}, or any free identifier FIELD allowing to group several rows together. For example, referring to {\em obs\_id} will allow all the datasets belonging to the same observation to be bound to the same \blinks\ endpoint response. +\noindent Note that the {\it referred\_ID\_FIELD} is not required to be {\em obs\_publisher\_did/}. It can be {\em obs\_id}, or any free identifier FIELD allowing grouping of several rows together. For example, referring to {\em obs\_id} will allow all the datasets belonging to the same observation to be bound to the same \blinks\ endpoint response. - This method is implemented in many archive services such as ESO, CADC, GAVO, HEASARC and even VizieR. +This method is implemented in many archive services such as ESO, CADC, GAVO, HEASARC and even VizieR. - \subsubsection{Datalink table in a TAP service / join with ObsCore table} + \subsubsection{Datalink Table in a TAP Service / Join with ObsCore Table} \label{sec:datalinktap} -The \blinks\ endpoint table can also be implemented as a TAP table of its own. This table will contain columns describing all the links (hea-event-list, response functions, analysis data products) desirable for any dataset record in the \texttt{ivoa.obscore} table of ObsTAP service. -A direct access to the ObsCore record and the links provided by the \blinks\ endpoint response table is then possible. An identifier column would be common in the ObsCore table and the DataLink table and used as a keyreference. Suppose we have {\em dataproduct\_type} = ‘hea-event-list’ in the \texttt{ivoa.obscore} table, and we name this DataLink table “datalink.response”, we can then query the TAP service this way in order to access directly to the hea-event-list product: +The \blinks\ endpoint table can also be implemented as a TAP table of its own. This table will contain columns describing all the links ({\bf hea-event-list}, {\bf response-function}s, additional data products) desirable for any dataset record in the {\tt ivoa.obscore} table of ObsTAP service. +A direct access to the ObsCore record and the links provided by the \blinks\ endpoint response table is then possible. An identifier column must be present in both the ObsCore table and the DataLink table to be used as a key reference. Suppose we have {\em dataproduct\_type} = `hea-event-list' in the {\tt ivoa.obscore} table, and we name this DataLink table ``datalink.response''. we can then query the TAP service this way in order to access directly to the {\bf hea-event-list} data product: +{\small \begin{verbatim} SELECT * FROM ivoa.obscore NATURAL JOIN ivoa.obscore_hea JOIN datalink.response ON - ivoa.obscore.obs_publisher_did = datalink.response.ID + ivoa.obscore.obs_publisher_did = datalink.response.ID WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6) - AND (dataproduct_type = ’hea-event-list’ ) - AND ( semantics = '#this') + AND (dataproduct_type = `hea-event-list') + AND semantics = `#this') \end{verbatim} -While to access directly the psf we can select this way : +} + +\noindent while to access directly the associated psf we can select this way : + +{\small \begin{verbatim} SELECT * FROM ivoa.obscore NATURAL JOIN ivoa.obscore_hea JOIN datalink.response ON ivoa.obscore.obs_publisher_did = datalink.response.ID WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6) - AND (dataproduct_type = ’hea-event-list’ ) - AND ( semantics = '#calibration') - AND (content_qualifier = 'psf') + AND (dataproduct_type = `hea-event-list') + AND (semantics = `#calibration') + AND (content_qualifier = `psf') \end{verbatim} -and to access directly a background image : +} + +\noindent and to access directly a background image : + +{\small \begin{verbatim} SELECT * FROM ivoa.obscore NATURAL JOIN ivoa.obscore_hea JOIN datalink.response ON - ivoa.obscore.obs_publisher_did = datalink.response.ID - WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6) - AND (dataproduct_type = ’hea-event-list’ ) - AND ( semantics = '#auxiliary') - AND (content_qualifier = 'bkgimage') -\end{verbatim} -One caveat: if response functions and other auxiliary or analysis data are common to a full observation (instead of dataset) or even worse to several observations, then the DataLink-like table will repeat the information several times. To avoid that it's possible to create an additional index column (for example {\em dl\_association\_id \/} or whatever appropriate term) and modify the first query this way (for others it would be similar of course) : -\begin{verbatim} -SELECT * FROM ivoa.obscore - NATURAL JOIN ivoa.obscore_hea - JOIN datalink.response ON ivoa.obscore.response_id = datalink.response.ID + ivoa.obscore.obs_publisher_did = datalink.response.ID WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6) - AND (dataproduct_type = ’hea-event-list’ ) - AND ( semantics = '#this') + AND (dataproduct_type = `hea-event-list') + AND (semantics = '#auxiliary') + AND (content_qualifier = `bkgimage') \end{verbatim} +} -\subsection{Accessing Response datasets via a specific table joined with the \texttt{ivoa.obscore} table} - -For the discovery of response data products, the use cases recorded show that selection criteria are based on 1) the type of response 2) the observed data set properties, and on the observation, like the observing position and field of view, the observing date and time, the energy and time bounds. However, the cardinality of the relationship between a response dataset and an hea-event-list dataset varies according to projects and may be 1 to 1, like in CTAO project, or 1 to N, N to 1 or N to N. - -Compared to the \texttt{ivoa.obscore} table, it can happen that less attributes are needed to select a response file. The response details could be into the response files, in the metadata encoded according to the file format (OGIP, GADF, VODF, etc). In this case, response data products can be described by a specific response table, with a short list of attributes as proposed in Tab. \ref{tab:response_table}. - -In order to handle the various cardinality between response and observation datasets, the foreign keys are defined in the response table and will allow JOIN operations between the \texttt{ivoa.obscore} and \texttt{ivoa.response} table. - -\section*{Response table proposal} -%\TODO{ check fields for this new table } -%\begin{table}[htbp] -%\begin{center} -\begin{longtable}{|p{0.25\linewidth}|p{0.12\linewidth}|p{0.1\linewidth}|p{0.39\linewidth}|} -\hline -Column Name & Unit & Type & Description\\\hline -%obs\_collection & unitless & String & Name of the data collection \\\hline -% to bind a response with an observation data set -obs\_id & unitless & String & Related Observation ID (foreign key) \\\hline -obs\_publisher\_did & unitless & String & Related dataset ID (foreign key) \\\hline -% copied from related observation if needed -target\_name & unitless & String & Astronomical object observed, if any\\\hline -facility\_name & unitless & String & Facility Name used for observed data products \\\hline -instrument\_name & unitless & String & Instrument Name used for observed data products \\\hline -% response specific fields -resp\_product\_type & unitless & String & Product type as defined in response-type vocabulary\\\hline -resp\_publisher\_did & unitless & String & Dataset identifier given by the publisher\\\hline -resp\_access\_url & unitless & String & URL used to access (download) response dataset\\\hline -resp\_access\_format & unitless & String & File content format (see in ObsCore MIME Types)\\\hline -% response spatial coverage -resp\_s\_ra & deg & double & ICRS Central right ascension, for the region where the response-product applies \\\hline -resp\_s\_dec & deg & double & ICRS Central declination, for the region where the response-product applies\\\hline -resp\_s\_region & unitless & String & Region of interest for response use (expressed in ICRS frame)\\\hline -resp\_t\_intervals & unitless& String (TMOC) & time coverage for response use \\\hline -%response energy coverage -resp\_energy\_min & m & double & Energy band minimal value for response use \\\hline -resp\_energy\_max & m & double & Energy band maximal value for response use \\\hline - -\caption{Possible mandatory fields of the Response table proposed in this HEIG ObsCore extension serialisation} -\label{tab:response_table} -\end{longtable} -%\end{center} -%\end{table} - -Implementing this extra response table will allow to write queries like: \\ - -\noindent Find all datasets satisfying: -\begin{enumerate}[(i)] - \item Position inside 3 arcmin from (83.84358, $-5.43639$) - \item Response product\_type = ``psf'' - \item obs\_id = ``1527'' - \item obs\_collection = ``HESS'' -\end{enumerate} - -When using the response table defined in Table \ref{tab:response_table}, -we can join the \texttt{ivoa.obscore} table with the \texttt{ivoa.response} table on -the two keys {\em obs\_id \/} and {\em obs\_collection \/} for instance, as in: -\begin{verbatim} -SELECT obs_publisher_did, s_ra, s_dec, s_fov, t_min, t_max, -energy_min, energy_max, access_url, access_format, -resp_publisher_did, resp_access_url, resp_access_format, -resp_energy_max, resp_energy_min -FROM ivoa.obscore -NATURAL JOIN ivoa.response -WHERE -(target_name = ’Crab’ OR target_name = ’M1’ OR -CONTAINS(POINT(s_ra, s_dec), CIRCLE, 83.6324, +22.0174, 0.083333) = 1) -AND (resp_product_type = 'psf') -AND (obs_id = '1527') -AND (obs_collection = 'HESS') -\end{verbatim} - -If a time constraint is needed, then we can use {\em resp\_t\_min \/}, and {\em resp\_t\_max \/} and constrain the interval like: -\begin{verbatim} -AND (resp_t_min > 56000.0 and resp_t_max < 56001.5) -\end{verbatim} - -\section*{Querying for Analysis Data products} -\label{sec:analysis-dp} - -The same strategy can apply for searching analysis data products for one observation or for a hea-event-list data product, using a dedicated \texttt{ivoa.obscore.adp} table . -The benefice is to describe more specific properties of these data products in the optional columns, without overloading the main \texttt{ivoa.obscore} table. - +One caveat: if response functions and other associated data are common to a full observation (instead of dataset) or even worse to several observations, then the DataLink-like table will repeat the information several times. %To avoid that it's possible to create an additional index column (for example {\em dl\_association\_id\/} or whatever appropriate term) and modify the first query this way (for others it would be similar of course): +% +%%% +%%% THIS DOESN'T SEEM TO MATCH THE TEXT +%%% +%{\small +%\begin{verbatim} +%SELECT * FROM ivoa.obscore +% NATURAL JOIN ivoa.obscore_hea +% JOIN datalink.response ON +% ivoa.obscore.response_id = datalink.response.ID +% WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6) +% AND (dataproduct_type = ’hea-event-list’) +% AND (semantics = '#this') +%\end{verbatim} +%} +%%- \subsection{Accessing {\bf response-function} Datasets via a Specific Table Joined with the {\tt ivoa.obscore} Table}\label{sec:respaccess} +%%- +%%- \begin{center} +%%- \begin{tabular}{|m{0.9\linewidth}|} +%%- \hline +%%- \center{\bf Note} \cr +%%- \vspace{6pt} +%%- This section describes work-in-progress that has been proposed as a conceptual general approach that may be appropriate for facilitating access to {\bf response-function}s, but that requires further consideration by the \gls{IVOA} \gls{HEIG}. This section is included to highlight a possible approach that the \gls{HEIG} may endorse in the future. \cr +%%- \hline +%%- \end{tabular} +%%- \end{center} +%%- +%%- %For the discovery of response data products, the use cases recorded show that selection criteria are based on 1) the type of response 2) the observed data set properties, and on the observation, like the observing position and field of view, the observing date and time, the energy and time bounds. However, the cardinality of the relationship between a response dataset and an hea-event-list dataset varies according to projects and may be 1 to 1, like in CTAO project, or 1 to N, N to 1 or N to N. +%%- +%%- %Compared to the \texttt{ivoa.obscore} table, it can happen that less attributes are needed to select a response file. The response details could be into the response files, in the metadata encoded according to the file format (OGIP, GADF, VODF, etc). In this case, response data products can be described by a specific response table, with a short list of attributes as proposed in Tab. \ref{tab:response_table}. +%%- +%%- %In order to handle the various cardinality between response and observation datasets, the foreign keys are defined in the response table and will allow JOIN operations between the \texttt{ivoa.obscore} and \texttt{ivoa.response} table. +%%- +%%- For the discovery of {\bf response-function} data products, the use cases record\-ed in Appendix~\ref{sec:uc} include selection criteria that are based on the type of {\bf response-function}, the observation data set properties, and on the observation properties such as the observing position and field-of-view, the observation date and time, the energy, and time bounds. +%%- +%%- Compared to the set of queryable attributes included in {\tt ivoa.obscore} table, more or fewer attributes may be required to select an appropriate {\bf response-function} dataset. In such a case, {\bf response-function} data products could be described by a specific {\tt ivoa.response} table. One possible definition of such a table is presented in Table~\ref{tab:response_table}. +%%- +%%- Additonally, the cardinality of the relationship between a {\bf response-function} dataset and an {\bf hea-event-list} dataset will vary according to the facility and may be $1:1$, as is the case for the CTAO project, or $1:N$, $N:1$, or $N:N$. +%%- +%%- In order to handle the various possible cardinality relationships between {\bf response-function} and {\bf hea-event-list} datasets, foreign keys must be defined in the {\tt ivoa.response} table that will allow JOIN operations between that table and the {\tt ivoa.obscore} table. +%%- +%%- \subsection*{Response Table Concept} +%%- %\TODO{ check fields for this new table } +%%- %\begin{table}[htbp] +%%- %\begin{center} +%%- {\small +%%- \begin{longtable}{|m{0.25\linewidth}|m{0.1\linewidth}|m{0.10\linewidth}|m{0.41\linewidth}|} +%%- \hline +%%- {\bf Column Name} & {\bf Unit} & {\bf Type} & {\bf Description}\\\hline +%%- %obs\_collection & unitless & String & Name of the data collection \\\hline +%%- % to bind a response with an observation data set +%%- {\em obs\_id} & unitless & String & Related observation ID (foreign key) \\\hline +%%- {\em obs\_publisher\_did} & unitless & String & Related dataset ID (foreign key) \\\hline +%%- % copied from related observation if needed +%%- {\em target\_name} & unitless & String & Astronomical object observed, if any\\\hline +%%- {\em facility\_name} & unitless & String & Facility name used for observed data products \\\hline +%%- {\em instrument\_name} & unitless & String & Instrument name used for observed data products \\\hline +%%- % response specific fields +%%- {\em resp\_product\_type} & unitless & String & Product type as defined in the response-type vocabulary\\\hline +%%- {\em resp\_publisher\_did} & unitless & String & Dataset identifier given by the publisher\\\hline +%%- {\em resp\_access\_url} & unitless & String & URL used to access (download) the response dataset\\\hline +%%- {\em resp\_access\_format} & unitless & String & File content format (see in Obs\-Core MIME Types)\\\hline +%%- % response spatial coverage +%%- {\em resp\_s\_ra} & deg & double & ICRS reference right ascension, for the region where the response-product applies \\\hline +%%- {\em resp\_s\_dec} & deg & double & ICRS reference declination, for the region where the response-product applies\\\hline +%%- {\em resp\_s\_region} & unitless & String & Region of interest for response use (expressed in ICRS frame)\\\hline +%%- {\em resp\_t\_intervals} & unitless& String (TMOC) & Time coverage for response use \\\hline +%%- %response energy coverage +%%- {\em resp\_energy\_min} & eV & double & Energy band minimal value for response use \\\hline +%%- {\em resp\_energy\_max} & eV & double & Energy band maximal value for response use \\\hline +%%- +%%- \caption{Possible Mandatory Fields of the Response Table Concept.} +%%- \label{tab:response_table} +%%- \end{longtable} +%%- } +%%- %\end{center} +%%- %\end{table} +%%- +%%- \noindent Implementing an {\tt ivoa.response} table similar to this would allow queries such as the following: \\ +%%- +%%- \noindent Find all datasets satisfying: +%%- \begin{enumerate}[(i)] +%%- \item Position inside 3 arcmin from (83.6324, $+22.0174$) +%%- \item Response product\_type = ``psf'' +%%- \item obs\_id = ``1527'' +%%- \item obs\_collection = ``HESS'' +%%- \end{enumerate} +%%- +%%- When using the response table defined in Table \ref{tab:response_table}, we can join the {\tt ivoa.obscore} table with the {\tt ivoa.response} table on +%%- the two keys {\em obs\_id\/} and {\em obs\_publisher\_did\/} for instance, as in: +%%- \begin{verbatim} +%%- SELECT obs_publisher_did, s_ra, s_dec, s_fov, t_min, t_max, +%%- energy_min, energy_max, access_url, access_format, +%%- resp_publisher_did, resp_access_url, resp_access_format, +%%- resp_energy_max, resp_energy_min +%%- FROM ivoa.obscore +%%- NATURAL JOIN ivoa.response +%%- WHERE +%%- (target_name = ’Crab’ OR target_name = ’M1’ OR +%%- CONTAINS(POINT(s_ra, s_dec), CIRCLE, 83.6324, +22.0174, 0.083333) = 1) +%%- AND (resp_product_type = 'psf') +%%- AND (obs_id = '1527') +%%- AND (obs_collection = 'HESS') +%%- \end{verbatim} +%%- +%%- If additional constraints are required, then we can use ({\em e.g.\/}) {\em resp\_t\_min\/}, and {\em resp\_t\_max\/} and constrain the time interval by adding a clause like: +%%- \begin{verbatim} +%%- AND (resp_t_min > 56000.0 and resp_t_max < 56001.5) +%%- \end{verbatim} +%%- +%%- \subsection*{Querying for Advanced Data products} +%%- \label{sec:analysis-dp} +%%- +%%- A similar strategy to \S~\ref{sec:respaccess} {\em could\/} also apply for discovering advanced data products for an observation or for a {\bf hea-event-list} data product, using a dedicated {\tt ivoa.obscore.adp} table. This might allow more specific properties of these data products to be listed in optional table columns, without overloading the main {\tt ivoa.obscore} table. +%%- +%%- +%%- \ No newline at end of file