pageserver: add direct io config to virtual file

[yliang412]

Problem

We need a way to incrementally switch to direct IO. During the rollout we might want to switch to O_DIRECT on image and delta layer read path first before others.

Summary of changes

• Added a wrapper around VirtualFile, current implementation become VirtualFileInner
• Use open_v2, create_v2, open_with_options_v2 when we want to use the IO mode specified in PS config.

As an example, we are using open_v2 with virtual_file::IoMode::Direct in #9169

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

5013 tests run: 4855 passed, 0 failed, 158 skipped (full report)

---

Flaky tests (5)

Postgres 17

• test_timeline_archive[4]: debug-x86-64

Postgres 16

• test_slots_and_branching: release-x86-64
• test_subscriber_restart: release-x86-64

Postgres 14

• test_hot_standby_feedback: release-arm64
• test_cli_start_stop_multi: release-x86-64

Code coverage* (full report)

• functions: 31.5% (7535 of 23931 functions)
• lines: 49.6% (60231 of 121410 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
a85bd88 at 2024-10-01T04:55:03.861Z :recycle:}