You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some accelerators use non-standard float point types (e.g. bfloat16 and TF32). They are important to achieve high performance (e.g. by using Nvidia's tensor cores), and/or reduce resource usage (e.g. FP32->FP16 reduces memory usage by half).
How could MLLoader leverage these types? Some ideas:
Do it transparently, auto convert based on the accelerator
Should the API allow JS code to specify acceptable quantization levels (e.g. use bf16 but not fp16)
What if the chip doesn't support the model's declared data type (e.g. BF16 chip + FP32 model)
The text was updated successfully, but these errors were encountered:
Another factor is download time. IIUC, the current tfjs format (for example) doesn't support float16, and so tfjs-converter converts weights to float32. This isn't ideal because it doubles the model size. I think it makes more sense to always optimistically serve the model in its "native" floating point format and for conversion to be done at run time based on the device's hardware.
anssiko
changed the title
Open question: support for non-IETF 754 float point types
Open question: support for non-IEEE 754 float point types
Mar 9, 2022
Relates to webmachinelearning/webnn#252
Some accelerators use non-standard float point types (e.g. bfloat16 and TF32). They are important to achieve high performance (e.g. by using Nvidia's tensor cores), and/or reduce resource usage (e.g. FP32->FP16 reduces memory usage by half).
How could MLLoader leverage these types? Some ideas:
The text was updated successfully, but these errors were encountered: