win-mf: Add Windows Media Foundation Capture Device and Intel NPU AI features #10471

thyintel · 2024-04-02T21:29:24Z

Description

PR is linked with: obsproject/obs-deps#242 (Requires wil headers to build)
Windows 11 is also required for this PR to build and Windows 11 SDK (10.0.22000.0).

This is a draft implementation of Intel AI features using the NPU(=”Intel AI Boost”) on Intel Core Ultra(a.k.a Meteor Lake).
It enables Background Blur(Standard and Portrait), Background Removal, Auto Framing and Eye Contact correction over new Microsoft Effect Package and Media Foundation.
Because this capture device is based on Media Foundation, we created separate “Video Capture Device 2”. We are thinking that it’s better to integrate it into the existing DShow based Video Capture Device in order to not to make end-users confused.
Thus, we need help for (1) overall design guidance how we can integrate - how we can kick off MF based device instead of DShow base, (2) Any missing features to get merged? Because we focus on prototyping new AI features, wondering if we’re missing something on the MF device.
BTW, this feature only works on OEM integrated web-cam, external USB camera is planned to be supported for the future, but not yet supported as of March 2024, no ETA announced.

Screen shots:
GPU 99%, NPU 30% over 3D Mark, w/ AI features are enabled

New UI for AI – Background Blur, Removal, Auto Framing, Eye Contact by NPU(=Intel AI Boost)
they only appear on MTL, otherwise this section would be skipped.

Motivation and Context

Utilize Intel NPU for OBS and be able to add AI capabilities in OBS for Intel devices.

How Has This Been Tested?

This has been tested on:

MTL system containing the Intel NPU
ADL system without NPU

Types of changes

New feature (non-breaking change which adds functionality)

Checklist:

[x ] My code has been run through clang-format.
[x ] I have read the contributing document.
[x ] My code is not on the master branch.
[x ] The code has been tested.
[x ] All commit messages are properly formatted and commits squashed where appropriate.
[x ] I have included updates to all appropriate documentation.

RytoEX

Please use a more descriptive commit message.

In C++ for variables/functions/identifiers that remain in C++, we prefer camelCase over snake_case.

Is it possible to, at minimum, split this into two commits, or perhaps two PRs:

Add Media Foundation
Add the NPU features

plugins/win-mf/CMakeLists.txt

cmake/Modules/FindWIL.cmake

cmake/finders/FindWIL.cmake

plugins/win-mf/cmake/windows/obs-module.rc.in

plugins/win-mf/win-mf.cpp

plugins/win-mf/plugin-macros.h.in

plugins/win-mf/mf-plugin.cpp

RytoEX · 2024-04-24T21:00:13Z

plugins/win-mf/data/placeholder.png is only used for the DirectShow-based Virtual Camera. It doesn't seem to have a purpose here?

RytoEX · 2024-05-16T17:56:04Z

The deps updated with obsproject/obs-deps#242 was merged, so this is no longer a draft, but it will need to be rebased.

WizardCM · 2024-06-16T00:41:13Z

So I was able to get this building with current deps, however it does not work on Windows 10. This is due to usage of IMFCameraControlMonitor. I assume this is expected, but I'm noting it for posterity.

10:31:38.979: Module '../../obs-plugins/64bit/win-mf.dll' not loaded

WizardCM · 2024-06-16T01:13:17Z

I launched this PR on a Windows 11 Insider build, and got a crash when adding the source.

Snippets:

11:04:53.494: Video Capture Device 2 - MF: DecodeDeviceId failed
11:04:53.494: Video Capture Device 2 - MF: Video configuration failed
11:04:53.501: User added source 'Video Capture Device 2 - MF' (MediaFoundationsource_input) to scene 'Scene'

win-mf.dll!PhysicalCamera::GetBlur+0x55
win-mf.dll!PhysicalCamera::GetMepSetting+0x29
win-mf.dll!PhysicalCamera::SaveSettingsToDefault+0x11c
win-mf.dll!PhysicalCamera::Stop+0x2d
win-mf.dll!PhysicalCamera::Uninitialize+0x2d
win-mf.dll!PhysicalCamera::~PhysicalCamera+0x2e
win-mf.dll!PhysicalCamera::Release+0x22
win-mf.dll!MF_Create+0x4f
win-mf.dll!DeviceEnumerator::EnumerateCameraCallback+0xc2
win-mf.dll!MF_EnumerateCameras+0x93
win-mf.dll!DeviceEnumerator::Enumerate+0x7c
win-mf.dll!GetMediaFoundationSourceProperties+0xb2
obs.dll!obs_source_properties+0x8d
obs64.exe!OBSPropertiesView::ReloadProperties+0x81

Full logs (session and crash):

2024-06-16 11-04-39.txt

Crash 2024-06-16 11-04-54.txt

thyintel · 2024-06-28T21:38:01Z

I launched this PR on a Windows 11 Insider build, and got a crash when adding the source.

Snippets:

11:04:53.494: Video Capture Device 2 - MF: DecodeDeviceId failed
11:04:53.494: Video Capture Device 2 - MF: Video configuration failed
11:04:53.501: User added source 'Video Capture Device 2 - MF' (MediaFoundationsource_input) to scene 'Scene'

win-mf.dll!PhysicalCamera::GetBlur+0x55
win-mf.dll!PhysicalCamera::GetMepSetting+0x29
win-mf.dll!PhysicalCamera::SaveSettingsToDefault+0x11c
win-mf.dll!PhysicalCamera::Stop+0x2d
win-mf.dll!PhysicalCamera::Uninitialize+0x2d
win-mf.dll!PhysicalCamera::~PhysicalCamera+0x2e
win-mf.dll!PhysicalCamera::Release+0x22
win-mf.dll!MF_Create+0x4f
win-mf.dll!DeviceEnumerator::EnumerateCameraCallback+0xc2
win-mf.dll!MF_EnumerateCameras+0x93
win-mf.dll!DeviceEnumerator::Enumerate+0x7c
win-mf.dll!GetMediaFoundationSourceProperties+0xb2
obs.dll!obs_source_properties+0x8d
obs64.exe!OBSPropertiesView::ReloadProperties+0x81

Full logs (session and crash):

2024-06-16 11-04-39.txt

Crash 2024-06-16 11-04-54.txt

It looks like the Crash logs are from the Windows 10 build, are there any from your attempt at adding the source on Windows 11?

WizardCM · 2024-06-29T00:53:04Z

@thyintel the crash log is definitely from Windows 11, per the build number 26236 - not sure why it says 10. As I mentioned here, the MediaFoundation source is entirely hidden/disabled on on Windows 10 due to using Windows 11-only APIs.

RytoEX · 2024-07-10T21:04:54Z

This builds and runs on Windows 11 23H2 with the Windows 11 SDK (10.0.22621.0). The "Video Capture Device 2" is available and correctly loads my webcam. However, in Debug configuration, it spams the log with the following:

log spam

source 'Video Capture Device 2' (MediaFoundationsource_input) created
Video Capture Device 2: DecodeDeviceId failed
Video Capture Device 2: Video configuration failed
User added source 'Video Capture Device 2' (MediaFoundationsource_input) to scene 'Scene'
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(1363)\win-mf.dll!00007FFA13A03EB7: (caller: 00007FFA13A04389) ReturnHr(3003) tid(4df8) 80070490 Element not found.
    [PhysicalCamera::GetMepSetting(GetBlur(blur, shallowfocus, mask))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(1401)\win-mf.dll!00007FFA13A043E9: (caller: 00007FFA13A06055) ReturnHr(3004) tid(4df8) 80070490 Element not found.
    [PhysicalCamera::SaveSettingsToDefault(GetMepSetting(setting))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(1363)\win-mf.dll!00007FFA13A03EB7: (caller: 00007FFA13A04389) ReturnHr(3005) tid(3978) 80070490 Element not found.
    [PhysicalCamera::GetMepSetting(GetBlur(blur, shallowfocus, mask))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(1401)\win-mf.dll!00007FFA13A043E9: (caller: 00007FFA13A06055) ReturnHr(3006) tid(3978) 80070490 Element not found.
    [PhysicalCamera::SaveSettingsToDefault(GetMepSetting(setting))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(393)\win-mf.dll!00007FFA13A04567: (caller: 00007FFA13A04CDB) ReturnHr(3007) tid(5490) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(pSample->GetUnknown(MFSampleExtension_CaptureMetadata, IID_PPV_ARGS(&spMetadataAttri)))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(393)\win-mf.dll!00007FFA13A04567: (caller: 00007FFA13A04CDB) ReturnHr(3008) tid(3fcc) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(pSample->GetUnknown(MFSampleExtension_CaptureMetadata, IID_PPV_ARGS(&spMetadataAttri)))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3009) tid(5490) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3010) tid(3fcc) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3011) tid(5490) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3012) tid(5994) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3013) tid(5490) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]
C:\dev\obsproject\obs-studio\plugins\win-mf\libmfcapture\source\PhysicalCamera.cpp(397)\win-mf.dll!00007FFA13A0462E: (caller: 00007FFA13A04CDB) ReturnHr(3014) tid(5994) C00D36E6 The requested attribute was not found.    [PhysicalCamera::FillSegMask(spMetadataAttri->GetBlobSize( MF_CAPTURE_METADATA_FRAME_BACKGROUND_MASK, &cbBlobSize))]

Given the log spam, I wonder if something is not appropriately gated.

Appropriately, since the hardware I'm testing on at the moment does not have an NPU, I do not see the NPU features (background removal, eye gaze correction, etc.). I will re-check on capable hardware at a later time.

tytan652 · 2024-07-31T14:03:26Z

plugins/win-mf/mf-plugin.cpp

+	obs_data_t *obs_settings = obs_data_create();
+	obs_apply_private_data(obs_settings);
+	obs_data_release(obs_settings);


Suggested change

obs_data_t *obs_settings = obs_data_create();

obs_apply_private_data(obs_settings);

obs_data_release(obs_settings);

Unnecessary line of code, since the settings instance is not even modified.

Thank you for pointing this out; it should be removed now.

WizardCM · 2024-07-31T14:07:44Z

The current implementation of this PR will also need to bump the minimum Windows SDK version to 22621 due to the usage of IMFCameraControlNotify.

obs-studio/cmake/windows/compilerconfig.cmake

Lines 45 to 47 in 8b975c4

    
           if(CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION VERSION_LESS 10.0.20348) 
        
             message(FATAL_ERROR "OBS requires Windows SDK version 10.0.20348.0 or more recent.\n" 
        
                                 "Please download and install the most recent Windows SDK.")

WizardCM · 2024-07-31T14:17:18Z

Ran the build on a debugger on Windows and confirmed the crash was not due to building on Windows:

WizardCM · 2024-07-31T14:29:46Z

plugins/win-mf/win-mf.cpp

+	obs_property_set_long_description(p,
+					  obs_module_text("Buffering.ToolTip"));


This configures the "Buffering.ToolTip" text onto the "FPS" dropdown. If there is no plan to keep Buffering, I recommend going through and deleting related buffering code, otherwise re-introduce the Buffering toggle and assign this tooltip to it.

thyintel · 2024-08-08T21:54:40Z

Ran the build on a debugger on Windows and confirmed the crash was not due to building on Windows:

Unfortunately, I could not reproduce this crash.

WizardCM · 2024-08-10T03:39:04Z

Ran the build on a debugger on Windows and confirmed the crash was not due to building on Windows 10:

Unfortunately, I could not reproduce this crash.

@thyintel I was able to track down the crash to the fact I have NDI Virtual Input installed. I expect the crash is not limited to this particular virtual camera device, but rather any device where the extended controls aren't implemented/available.

This was my workaround, which fixes the crash but still allows the camera to appear in the list:

diff --git a/plugins/win-mf/libmfcapture/source/PhysicalCamera.cpp b/plugins/win-mf/libmfcapture/source/PhysicalCamera.cpp
index 65c4a8815..48b7052df 100644
--- a/plugins/win-mf/libmfcapture/source/PhysicalCamera.cpp
+++ b/plugins/win-mf/libmfcapture/source/PhysicalCamera.cpp
@@ -1360,6 +1360,9 @@ HRESULT PhysicalCamera::GetMepSetting(MepSetting &setting)

        bool blur, shallowfocus, mask, autoframing, eyegaze;

+       if (m_spExtController == nullptr)
+               return S_OK;
+
        RETURN_IF_FAILED(GetBlur(blur, shallowfocus, mask));
        setting.Blur = blur;
        setting.ShallowFocus = shallowfocus;

thyintel · 2024-08-14T22:00:38Z

Ran the build on a debugger on Windows and confirmed the crash was not due to building on Windows 10:

Unfortunately, I could not reproduce this crash.

@thyintel I was able to track down the crash to the fact I have NDI Virtual Input installed. I expect the crash is not limited to this particular virtual camera device, but rather any device where the extended controls aren't implemented/available.

This was my workaround, which fixes the crash but still allows the camera to appear in the list:
diff --git a/plugins/win-mf/libmfcapture/source/PhysicalCamera.cpp b/plugins/win-mf/libmfcapture/source/PhysicalCamera.cpp
index 65c4a8815..48b7052df 100644
--- a/plugins/win-mf/libmfcapture/source/PhysicalCamera.cpp
+++ b/plugins/win-mf/libmfcapture/source/PhysicalCamera.cpp
@@ -1360,6 +1360,9 @@ HRESULT PhysicalCamera::GetMepSetting(MepSetting &setting)

        bool blur, shallowfocus, mask, autoframing, eyegaze;

+       if (m_spExtController == nullptr)
+               return S_OK;
+
        RETURN_IF_FAILED(GetBlur(blur, shallowfocus, mask));
        setting.Blur = blur;
        setting.ShallowFocus = shallowfocus;

Thank you for the suggestion, I changed the code to include this :)

RytoEX · 2024-08-22T00:09:14Z

I think I see what is happening with the build failure. While the Windows runners have CMake 3.30.2 installed, our CMake specifies:

cmake_minimum_required(VERSION 3.22...3.25)

This sets the CMake policy_max to 3.25, which will use the OLD behavior of CMake Policy 0149. CMP0149 controls which version of the Windows SDK is selected:

The OLD behavior for this policy is to use the exact value of CMAKE_SYSTEM_VERSION if possible. The NEW behavior for this policy is to ignore it.

This means that, because the runners are currently running Windows 10.0.20348, CMake running on them will select the Windows10SDK.20348 by default, because the CMAKE_SYSTEM_VERSION exactly matches an available SDK on the runner.

To get CI to build this PR, we would have to do one of the following:
A. Set the max_policy value to 3.27 or higher
B. Explicitly enable the policy with cmake_policy()
C. Explicitly set the Windows SDK version in the CMAKE_GENERATOR_PLATFORM CMake variable (see CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION for details)

cc @PatTheMav

PatTheMav · 2024-08-22T12:51:14Z

I think I see what is happening with the build failure. While the Windows runners have CMake 3.30.2 installed, our CMake specifies:
cmake_minimum_required(VERSION 3.22...3.25)
This sets the CMake policy_max to 3.25, which will use the OLD behavior of CMake Policy 0149. CMP0149 controls which version of the Windows SDK is selected:

The OLD behavior for this policy is to use the exact value of CMAKE_SYSTEM_VERSION if possible. The NEW behavior for this policy is to ignore it.

This means that, because the runners are currently running Windows 10.0.20348, CMake running on them will select the Windows10SDK.20348 by default, because the CMAKE_SYSTEM_VERSION exactly matches an available SDK on the runner.

To get CI to build this PR, we would have to do one of the following: A. Set the max_policy value to 3.27 or higher B. Explicitly enable the policy with cmake_policy() C. Explicitly set the Windows SDK version in the CMAKE_GENERATOR_PLATFORM CMake variable (see CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION for details)

cc @PatTheMav

I have a CMake 3.27 update ready, but given that Ubuntu 24.04 provides CMake 3.28 it might make more sense to get straight to that (macOS default would be CMake 3.30, but most recent Visual Studio-provided version is also 3.28).

RytoEX · 2024-08-22T19:27:49Z

I have a CMake 3.27 update ready, but given that Ubuntu 24.04 provides CMake 3.28 it might make more sense to get straight to that (macOS default would be CMake 3.30, but most recent Visual Studio-provided version is also 3.28).

A minor correction: Visual Studio 17.11 (the current stable release) provides CMake 3.29.5.

thyintel marked this pull request as draft April 2, 2024 21:29

WizardCM added the New Feature New feature or plugin label Apr 2, 2024

RytoEX self-assigned this Apr 3, 2024

thyintel force-pushed the obs_intel_ai branch from 4b745e5 to 165b26a Compare April 5, 2024 18:56

thyintel mentioned this pull request Apr 5, 2024

deps.windows: Add WIL headers obsproject/obs-deps#242

Merged

thyintel force-pushed the obs_intel_ai branch 3 times, most recently from fc46b24 to 0584cda Compare April 5, 2024 20:49

RytoEX requested changes Apr 9, 2024

View reviewed changes

thyintel changed the title ~~win-mf: Add win-mf project~~ win-mf: Add Windows Media Foundation Capture Device and Intel NPU AI features Apr 15, 2024

thyintel force-pushed the obs_intel_ai branch 6 times, most recently from c8280fd to dcea1bb Compare April 23, 2024 03:06

thyintel force-pushed the obs_intel_ai branch from dcea1bb to d72b83c Compare May 7, 2024 22:23

RytoEX marked this pull request as ready for review May 16, 2024 17:55

thyintel force-pushed the obs_intel_ai branch from d72b83c to 4085cf5 Compare May 17, 2024 00:09

thyintel force-pushed the obs_intel_ai branch from 4085cf5 to 411f873 Compare May 28, 2024 22:02

tytan652 reviewed Jul 31, 2024

View reviewed changes

WizardCM reviewed Jul 31, 2024

View reviewed changes

thyintel force-pushed the obs_intel_ai branch from 411f873 to beae20e Compare August 8, 2024 19:34

thyintel force-pushed the obs_intel_ai branch from beae20e to 1eae32a Compare August 14, 2024 21:57

thyintel force-pushed the obs_intel_ai branch from 1eae32a to 74ebb74 Compare August 19, 2024 18:38

thyintel force-pushed the obs_intel_ai branch 2 times, most recently from 84577d8 to 677be88 Compare September 3, 2024 20:20

thyintel added 2 commits September 13, 2024 11:58

win-mf: Add Windows Media Foundation Capture Device

dbcc5a4

win-mf: Add Intel NPU AI Features

2b117ff

thyintel force-pushed the obs_intel_ai branch from 677be88 to 2b117ff Compare September 13, 2024 18:59

RytoEX added the Seeking Testers Build artifacts on CI label Sep 13, 2024

RytoEX mentioned this pull request Sep 13, 2024

CI: Use Windows 11 SDK 10.0.22621.0 #11283

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

win-mf: Add Windows Media Foundation Capture Device and Intel NPU AI features #10471

win-mf: Add Windows Media Foundation Capture Device and Intel NPU AI features #10471

thyintel commented Apr 2, 2024 •

edited

Loading

RytoEX left a comment

RytoEX commented Apr 24, 2024

RytoEX commented May 16, 2024 •

edited

Loading

WizardCM commented Jun 16, 2024

WizardCM commented Jun 16, 2024

thyintel commented Jun 28, 2024

WizardCM commented Jun 29, 2024

RytoEX commented Jul 10, 2024

tytan652 Jul 31, 2024

thyintel Aug 14, 2024

WizardCM commented Jul 31, 2024

WizardCM commented Jul 31, 2024

WizardCM Jul 31, 2024

thyintel commented Aug 8, 2024

WizardCM commented Aug 10, 2024

thyintel commented Aug 14, 2024

RytoEX commented Aug 22, 2024 •

edited

Loading

PatTheMav commented Aug 22, 2024

RytoEX commented Aug 22, 2024

	obs_data_t *obs_settings = obs_data_create();
	obs_apply_private_data(obs_settings);
	obs_data_release(obs_settings);

		obs_property_set_long_description(p,
		obs_module_text("Buffering.ToolTip"));

win-mf: Add Windows Media Foundation Capture Device and Intel NPU AI features #10471

Are you sure you want to change the base?

win-mf: Add Windows Media Foundation Capture Device and Intel NPU AI features #10471

Conversation

thyintel commented Apr 2, 2024 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

RytoEX left a comment

Choose a reason for hiding this comment

RytoEX commented Apr 24, 2024

RytoEX commented May 16, 2024 • edited Loading

WizardCM commented Jun 16, 2024

WizardCM commented Jun 16, 2024

thyintel commented Jun 28, 2024

WizardCM commented Jun 29, 2024

RytoEX commented Jul 10, 2024

tytan652 Jul 31, 2024

Choose a reason for hiding this comment

thyintel Aug 14, 2024

Choose a reason for hiding this comment

WizardCM commented Jul 31, 2024

WizardCM commented Jul 31, 2024

WizardCM Jul 31, 2024

Choose a reason for hiding this comment

thyintel commented Aug 8, 2024

WizardCM commented Aug 10, 2024

thyintel commented Aug 14, 2024

RytoEX commented Aug 22, 2024 • edited Loading

PatTheMav commented Aug 22, 2024

RytoEX commented Aug 22, 2024

thyintel commented Apr 2, 2024 •

edited

Loading

RytoEX commented May 16, 2024 •

edited

Loading

RytoEX commented Aug 22, 2024 •

edited

Loading