CDAWEB 500 with DSCOVR_ORBIT_PRE

[jbfaden]

Results in 500:

wget -O - 'http://localhost:8280/HapiServer/hapi/data?id=DSCOVR_ORBIT_PRE&time.min=2025-04-15T00:00Z&time.max=2025-04-15T23:00Z'

[jbfaden]

This is the problem where the listing is out of sync with the files which are found on the server. This wants to find https://cdaweb.gsfc.nasa.gov/sp_phys/data/dscovr/orbit/pre_or/2025/dscovr_orbit_pre_20250402_v02.cdf but dscovr_orbit_pre_20250402_v03.cdf is found.

[rweigel]

Would mind posting non-localhost URLs?

I have a cron updating everything every three hours and it appears to be up-to-date. This

https://cdaweb.gsfc.nasa.gov/WS/cdasr/1/dataviews/sp_phys/datasets/DSCOVR_ORBIT_PRE/orig_data/20150211T233907Z,20250416T120000Z

shows

<FileDescription>
<Name>https://cdaweb.gsfc.nasa.gov/sp_phys/data/dscovr/orbit/pre_or/2024/dscovr_orbit_pre_20240414_v02.cdf</Name>
<MimeType>application/x-cdf</MimeType>
<StartTime>2024-04-14T00:00:00.000Z</StartTime>
<EndTime>2024-04-14T23:59:00.000Z</EndTime>
<Length>194837</Length>
<LastModified>2024-04-12T04:54:58.191Z</LastModified>
</FileDescription>

[jbfaden]

I think the problem there is the https://cottagesystems.com/server/cdaweb/hapi address didn't show the problem because it had the file downloaded already. Sorry for all the noisy channels here, I figure it's better to document something than nothing.

I couldn't find the email you sent about updating the database, so could you do that here?

If the service is still saying dscovr_orbit_pre_20240414_v02.cdf is the file, maybe there's another problem going on.

Catch up with you in the morning...

[rweigel]

dscovr_orbit_pre_20240414_v02.cdf appears to be the correct file:

https://cdaweb.gsfc.nasa.gov/pub/data/dscovr/orbit/pre_or/2024/

[rweigel]

I put the database update on a cron this weekend. See ~/cdawmeta/etc. It writes a log to ~/cdawmeta/data/crontab The full update takes ~40 minutes, but most of the time is for Cluster and THEMIS orig_data. We may want to go back to crawling their HTTP directories to generate orig_data. Bob

…

On Sun, Mar 23, 2025 at 8:06 PM Jeremy Faden ***@***.***> wrote: I think the problem there is the https://cottagesystems.com/server/cdaweb/hapi address didn't show the problem because it had the file downloaded already. Sorry for all the noisy channels here, I figure it's better to document something than nothing. I couldn't find the email you sent about updating the database, so could you do that here? If the service is still saying dscovr_orbit_pre_20240414_v02.cdf is the file, maybe there's another problem going on. Catch up with you in the morning... — Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUQ57W2377FEHFHMY7BD432V5EARAVCNFSM6AAAAABZTO2JN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBWGU3DOMRSGM> . You are receiving this because you commented.Message ID: ***@***.***> [image: jbfaden]*jbfaden* left a comment (hapi-server/server-java#62) <#62 (comment)> I think the problem there is the https://cottagesystems.com/server/cdaweb/hapi address didn't show the problem because it had the file downloaded already. Sorry for all the noisy channels here, I figure it's better to document something than nothing. I couldn't find the email you sent about updating the database, so could you do that here? If the service is still saying dscovr_orbit_pre_20240414_v02.cdf is the file, maybe there's another problem going on. Catch up with you in the morning... — Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUQ57W2377FEHFHMY7BD432V5EARAVCNFSM6AAAAABZTO2JN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBWGU3DOMRSGM> . You are receiving this because you commented.Message ID: ***@***.***>

[jbfaden]

Something is confusing here, note I was getting a sample time range of 2025-04-15T00:00Z/2025-04-15T23:00Z, but you are looking at 2024.

[jbfaden]

Another thing is we should make sure we use if-modified-since in our web service calls. I'd be surprised if Bernie isn't checking that.

[jbfaden]

I think this is the gist of the bug. Here you have "sampleStartDate" of 2024-04-15 in the info, but then in the orig_data response you have 2025-04-02 as the last day:

/home/weigel/cdawmeta/data
spot10> more hapi/info/DSCOVR_ORBIT_PRE.json |grep sample
    "sampleStartDate": "2025-04-15T00:00:00.000Z",
    "sampleStopDate": "2025-04-15T23:59:00.000Z",

/home/weigel/cdawmeta/data
spot10> grep Time  orig_data/info/DSCOVR_ORBIT_PRE.json | tail -6
        "StartTime": "2025-03-31T00:00:00.000Z",
        "EndTime": "2025-03-31T23:59:00.000Z",
        "StartTime": "2025-04-01T00:00:00.000Z",
        "EndTime": "2025-04-01T23:59:00.000Z",
        "StartTime": "2025-04-02T00:00:00.000Z",
        "EndTime": "2025-04-02T23:59:00.000Z",

/home/weigel/cdawmeta/data
spot10> 

One is based on the file listing service and the other is based on all.xml, right?

[rweigel]

See my next email forward for why this happened. It should be fixed soon.

[jbfaden]

With the ETAGS and last headers, I see. How about using the source file listing to create the range? That way all the "all.xml" is used for is the catalog?

[rweigel]

This seems to be the best option. Although this should not be an issue anymore, it is still possible for all.xml to be out-of-sync with the orig_data response when things are in flight or depending on how they do things on their end.