turbo: speed up rebuildSegments

[stevemilk]

1. do not reopen segments/indexed if already opened during rebuildSegments
2. accelerate scanning files by avoiding doubly nested loop

[AskAlexSharov]

looks good. but seems we have corner-case: we don't know "is download of file finished or not". i think it's possible to open some file (like .idx). i feel - it's possible to open .idx file - which was bad (metadata section was not downloaded) - and still get non-nil object in this case (no error may happen - if validation will not catch it).

In all other cases i think your fix will work.

[stevemilk]

find a way to cover the corner case:
if size of downloaded file is equal to the size stored in corresponding torrent file, download is considered as finished.
And only if file is complete(download finished || generate locally) and not opened yet, we reopen the file.

[mh0lt]

find a way to cover the corner case: if size of downloaded file is equal to the size stored in corresponding torrent file, download is considered as finished. And only if file is complete(download finished || generate locally) and not opened yet, we reopen the file.

Unfortunately this check won't work. The torrent lib truncates the mmap to its full size on initiation so it can randomly write to the full extent. So this will always be true once the download starts.

We have added a callback from the downloader when download completes - which can be used to know when the file is downloaded:

TorrentCompleted(ctx context.Context, in *TorrentCompletedRequest, opts ...grpc.CallOption) (Downloader_TorrentCompletedClient, error)

Making this call available to the snapshot code is part of a bigger refactor.

[stevemilk]

Thx for reminding. Will try implementing the callback and other possible alternatives.