Rough ETA/roadmap?

[josephrocca]

Just wondering what the roadmap roughly looks like at the moment? I.e. flagged and unflagged guesstimates for Chrome? Even a really rough estimate would be handy - e.g. might we see a flagged MVP some time this year?

[jbingham]

Thanks for asking! A dev trial will come first, which will be behind a flag. The first dev trial is likely to be available on Chrome OS only, sometime this year.

In order for the model loader API to be available without a flag, it will have to move through the web standards process, which is a bit unpredictable. The progression from dev trial to origin trial to web standard can easily take over a year.

We expect that the Web NN API will be available first. TensorFlow.js will support it.

In the meantime, TensorFlow.js and the WASM runner for TF Lite are your best bets for TensorFlow models.

[josephrocca]

Thanks! Useful info. I had assumed that WebNN would take much longer due to the larger surface area of the API and potential interaction with WebGPU.

I'm looking forward to testing ModelLoader on my Chromebook during the Dev trial 👍

[yuhonglin]

Hi, sorry for being a bit late. Now this API is ready for dev-trial in M102. Please notice that the API is currently chromeOS only. It is available after turning on two flags: "#enable-experimental-web-platform-features" and "#enable-machine-learning-model-loader-web-platform-api". A simple demo is https://false-shy-event.glitch.me/ which classifies images by MobileNet-v2 model based on the TFJS-TFLite runner and "loader api" (when available) respectively and compares their inference times.

[josephrocca]

@yuhonglin Just tested it on ChromeOS and it's fast! Thanks for your work on this. A few questions if that's okay:

• I'm wondering if you can add the COOP/COEP headers and set tflite to use multiple threads, simd, etc. so we can compare the max tfjs-tflite performance with the new ModelLoader performance?
• Does it support TF Select ops? tfjs-tflite currently doesn't support them, so I'm just curious (I know tflite isn't the final web-standard format)
• Is there an ETA on the dev trial being available on Chrome (including Linux)?
• Is there an ETA on an origin trial, or is it too early to guess at that?

[yuhonglin]

Thanks for trying it! Please see my comments inline below.

• I'm wondering if you can add the COOP/COEP headers and set tflite to use multiple threads, simd, etc. so we can compare the max tfjs-tflite performance with the new ModelLoader performance?

Yes, there is another simple demo for multithreaded tfjs-tflite (https://amber-hilarious-wire.glitch.me/). But it only contains tfjs-tflite and will not call the model loader API. One can use the "time_used" value from dev-tools to obtain the time.

On my Pixelbook, model loader is still faster than multithreaded tfjs-tflite.

• Does it support TF Select ops? tfjs-tflite currently doesn't support them, so I'm just curious (I know tflite isn't the final web-standard format)

I am not sure and need to check it. It just uses the tflite backend in chromeOS (build config: https://source.chromium.org/chromiumos/chromiumos/codesearch/+/main:src/third_party/chromiumos-overlay/sci-libs/tensorflow/). I think for any feature, if we really need it and there is no security/privacy concerns with it, we can add it to the chromeOS's tflite --- this is the power of model loader API :)

• Is there an ETA on the dev trial being available on Chrome (including Linux)?

There is no concrete plan yet. Doing that needs lots of effort and there are other things we may do first (e.g. hardware acceleration support, currently this API is CPU-only). I think we should first collect user's feedback, e.g., if there is strong needs of this API, maybe we can put more resources on it then speed up the progress.

• Is there an ETA on an origin trial, or is it too early to guess at that?

Similar as above, I feel it is still a bit early to have a concrete plan for it.

[josephrocca]

Hey @yuhonglin, wondering if there are any updates on the status/progress of this proposal?

[jbingham]

@josephrocca the Model Loader work is on pause for now. toward the end of last year, both @yuhonglin and I moved to other projects. The TF Lite WASM runner and TensorFlow.js are your best bets for now. There's still a lot of performance benefit from the WASM and WebGPU runners that power those.

[josephrocca]

No worries, thanks for the update!