performance issues when browsing core docs

[emillon]

Hi,

Loading pages such as https://ocaml.org/p/core/v0.17.1/doc/index.html is very slow, more than 10s.

I investigated and the server is GET'ing https://docs-data.ocaml.org/live/p/core/v0.17.1/index.js which is a massive file.

This is the search index. Disabling by stubbing Ocamlorg_package.search_index will make the problem go away.

Now, for a proper fix, I had a look at how this function is used in the source code (more precisely, in Handler):

• in most cases, it's only hashed.
• in package_search_index, it's served as a response.

For the first one, I suggest adding a new endpoint on docs-data to query just the hash. This will already cut the time in 3 or 4 (several resources on the page consume the hash).

The second one is trickier. Querying directly from the browser will allow efficient caching (docs-data.ocaml.org seems to reply with meaningful cache headers), but will hit into same origin policy. This can be solved by setting CORS headers on docs-data.ocaml.org, but has security implications.

[emillon]

(this could also be solved by keeping the proxy and adding a HTTP cache on ocaml.org's side, but this doesn't seem like the end solution)

[sabine]

I'll do a quick fix by caching the package search index hash, however, this hash should be generated by the docs pipeline when it builds the search indexes.