Skip to content

Commit 46b2240

Browse files
drewbatgitdrewbatgit
authored
Merge pull request #5268 from MicrosoftDocs/drewbat/copilot-updates-3D
Drewbat/copilot updates 3D
2 parents 3072e19 + 4a4633f commit 46b2240

File tree

3 files changed

+272
-1
lines changed

3 files changed

+272
-1
lines changed

‎hub/apps/develop/windows-integration/copilot-key-state.md

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
---
2+
title: Handle Microsoft Copilot hardware key state changes
3+
description: Learn how to register to be activated and receive notifications when the Microsoft Copilot hardware key or Windows key + C is pressed.
4+
ms.topic: article
5+
ms.date: 10/25/2024
6+
ms.localizationpriority: medium
7+
---
8+
9+
10+
11+
# Handle Microsoft Copilot hardware key state changes
12+
13+
This article describes how apps can register to be activated and receive notifications when the Microsoft Copilot hardware key or Windows key + C is pressed, pressed and held, and released. This feature enables apps to perform different actions depending on which key state change is detected. For example, an app may perform normal activation when the key is single-pressed, but take a screenshot when the key is pressed and held. Or, an app may begin recording audio and show a status indicator that audio is being recorded when the key is pressed and held, and then stop recording audio when the key is released. The key must be pressed and held for at least 300 ms to move into the held state.
14+
15+
This feature extends the features of a basic Microsoft Copilot hardware key provider, which simply registers to be launched when the hardware key is pressed. For more information, see [Microsoft Copilot hardware key providers](microsoft-copliot-key-provider.md).
16+
17+
The rest of this article will walk through creating a simple C# WinUI 3 app that responds to activation initiated by a single press or a press and hold and release of the Microsoft Copilot button.
18+
19+
## Create a new project
20+
21+
In Visual Studio, create a new project. For this example, in the **Create a new project** dialog, set the language filter to C# and the project type to WinUI 3 and then select the "Blank App, Packaged (WinUI 3 in Desktop).
22+
23+
## Add a property to track the Microsoft Copilot key pressed state
24+
25+
In this example, we will create a property called **State** that we will use to display the current activation state in the UI. In `MainWindow.xaml.cs`, inside the definition of **MainWindow** add the following code to create a string property that we can bind to in our XAML file.
26+
27+
```csharp
28+
// MainWindow.xaml.cs
29+
public event PropertyChangedEventHandler? PropertyChanged;
30+
31+
private void OnPropertyChanged([CallerMemberName] string propertyName = "State")
32+
{
33+
PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName));
34+
}
35+
36+
public void SetState(string state)
37+
{
38+
State = state;
39+
}
40+
41+
private string _state;
42+
public string State
43+
{
44+
get => _state;
45+
set
46+
{
47+
if (_state != value)
48+
{
49+
_state = value;
50+
OnPropertyChanged();
51+
}
52+
}
53+
}
54+
```
55+
56+
Add a **TextBox** control to the UI to show the current activation state of the app. Replace the default **StackPanel** element in MainPage.xaml with the following code.
57+
58+
```xaml
59+
<!-- MainWindow.xaml -->
60+
<StackPanel Orientation="Horizontal" HorizontalAlignment="Center" VerticalAlignment="Center">
61+
<TextBlock Name="KeyStateText" Text="{x:Bind State, Mode=OneWay}" />
62+
</StackPanel>
63+
```
64+
65+
Finally, update the **MainWindow** constructor to take an argument that will set the **State** property when the window is created.
66+
67+
```csharp
68+
// MainWindow.xaml.cs
69+
public MainWindow(string state)
70+
{
71+
this.InitializeComponent();
72+
73+
_state = state;
74+
}
75+
```
76+
77+
78+
## Register for URI activation
79+
80+
The system launches Microsoft Copilot hardware key providers using URI activation. Register a launch protocol by adding the [uap:Protocol](/uwp/schemas/appxpackage/uapmanifestschema/element-uap-protocol) element to your app manifest. For more information about how to register as the default handler for a URI scheme, see [Handle URI activation](/windows/apps/develop/launch/handle-uri-activation).
81+
82+
The following example shows the **uap:Extension** registering the URI scheme "myapp-copilothotkey".
83+
84+
```xml
85+
<!-- Package.appxmanifest -->
86+
...
87+
xmlns:uap3="http://schemas.microsoft.com/appx/manifest/uap/windows10/3"
88+
...
89+
90+
<Extensions>
91+
...
92+
<uap:Extension Category="windows.protocol">
93+
<uap:Protocol Name="myapp-copilothotkey"> <!-- app-defined protocol name -->
94+
<uap:DisplayName>SDK Sample URI Scheme</uap:DisplayName>
95+
</uap:Protocol>
96+
</uap:Extension>
97+
...
98+
```
99+
100+
## Microsoft Copilot hardware key app extension
101+
102+
An app must be packaged in order to register as a Microsoft Copilot hardware key provider. For information on app packaging, see [An overview of Package Identity in Windows app](/windows/apps/desktop/modernize/package-identity-overview). Microsoft Copilot hardware key providers declare their registration information within the [uap3:AppExtension](/uwp/schemas/appxpackage/uapmanifestschema/element-uap3-appextension-manual). The **Name** attribute of the extension must be set to "com.microsoft.windows.copilotkeyprovider". To support the key state changes, apps must provide some additional entries to their **uap3:AppExtension** declaration.
103+
104+
Inside of the **uap3:AppExtension** element, add a [uap3:Properties](/uwp/schemas/appxpackage/uapmanifestschema/element-uap3-properties-manual) element with child elements **PressAndHoldStart** and **PressAndHoldStop**. The contents of these elements should be the URI of the protocol scheme registered in the manifest in the previous step. The query string arguments specify whether the URI is being launched because the user pressed and held the hot key or because the user released the hot key. The app uses these query string values during app activation to determine the correct action to take.
105+
106+
```xml
107+
<!-- Package.appxmanifest -->
108+
109+
<Extensions>
110+
...
111+
<uap3:Extension Category="windows.appExtension">
112+
<uap3:AppExtension Name="com.microsoft.windows.copilotkeyprovider"
113+
Id="MyAppId"
114+
DisplayName="App display name"
115+
Description="App description"
116+
PublicFolder="Public">
117+
<uap3:Properties>
118+
<PressAndHoldStart>myapp-copilothotkey:?state=Down</PressAndHoldStart>
119+
<PressAndHoldStop>myapp-copilothotkey:?state=Up</PressAndHoldStop>
120+
</uap3:Properties>
121+
</ uap3:AppExtension>
122+
</uap3:Extension>
123+
...
124+
```
125+
126+
## Handle URI activation
127+
128+
To detect whether the app was activated via URI activation, call [AppInstance.GetActivatedEventArgs](/windows/windows-app-sdk/api/winrt/microsoft.windows.applifecycle.appinstance.getactivatedeventargs) and check to see if the value of the [AppActivationArguments.Kind](/windows/windows-app-sdk/api/winrt/microsoft.windows.applifecycle.appactivationarguments.kind) property is [Protocol](/windows/windows-app-sdk/api/winrt/microsoft.windows.applifecycle.extendedactivationkind). If the app was launched via protocol activation, check to see if the URI scheme is the same as the protocol name you specified in your app manifest. If all of these tests pass, then you know that your app was activated by the user pressing the Copilot hardware key. At this point you can parse the URI query string and get the *state* parameter, which will have the values you specified in the **PressAndHoldStart** and **PressAndHoldStop** elements in the app manifest.
129+
130+
```csharp
131+
// App.xaml.cs
132+
133+
protected override void OnLaunched(Microsoft.UI.Xaml.LaunchActivatedEventArgs args)
134+
{
135+
var eventargs = AppInstance.GetCurrent().GetActivatedEventArgs();
136+
string state = "";
137+
if ((eventargs != null) && (eventargs.Kind == ExtendedActivationKind.Protocol))
138+
{
139+
var protocolArgs = (Windows.ApplicationModel.Activation.ProtocolActivatedEventArgs)eventargs.Data;
140+
WwwFormUrlDecoder decoderEntries = new WwwFormUrlDecoder(protocolArgs.Uri.Query);
141+
state = Uri.UnescapeDataString(decoderEntries.GetFirstValueByName("state"));
142+
}
143+
state = (state == "") ? "Launched" : state;
144+
145+
m_window = new MainWindow(state);
146+
m_window.Activate();
147+
}
148+
```
149+
150+
> [!IMPORTANT]
151+
> Note that, by default, WinUI 3 apps are multi-instanced, which means that a new instance will be launched whenever the Microsoft Copilot hot key is pressed or released. This may be the desired behavior for many providers, but if you would prefer, you can update you app to use a single instance. For more information, see [Create a single-instanced WinUI app with C#](/windows/apps/windows-app-sdk/applifecycle/applifecycle-single-instance).
152+
153+
## Handle fast path invocation
154+
155+
In addition to URI activation, apps can register to support fast path invocation in which a running app receives messages about Copilot hardware app through window messages. For a currently-running app, this invocation method is faster than URI activation and will provide a better user experience, since the app can begin listening for speech more quickly after the key is pressed and held.
156+
157+
### Update the app manifest file to support fast path invocation
158+
159+
To add support for fast path invocation, update the "com.microsoft.windows.copilotkeyprovider" extension to add the *MessageWParam* attribute to the **SingleTap**, **PressAndHoldStart**, and **PressAndHoldStop** elements. Each *MessageWParam* value must be a unique 32-bit integer, but the values used are chosen by the app. This example uses values of 0, 1, and 2, respectively. These values will be used later in the example when they are passed in the *wParam* parameter of a Windows message to determine the current pressed state of the Windows Copilot hardware key.
160+
161+
```xml
162+
<!-- Package.appxmanifest -->
163+
164+
<uap3:Extension Category="windows.appExtension">
165+
<uap3:AppExtension Name="com.microsoft.windows.copilotkeyprovider"
166+
Id="MyAppId"
167+
DisplayName="App display name"
168+
Description="App description"
169+
PublicFolder="Public">
170+
<uap3:Properties>
171+
<SingleTap FastPathValue="0"/>
172+
<PressAndHoldStart MessageWParam="1">myapp-copilothotkey://?state=Down</PressAndHoldStart>
173+
<PressAndHoldStop MessageWParam="2">myapp-copilothotkey://?state=Up</PressAndHoldStop>
174+
</uap3:Properties>
175+
</uap3:AppExtension>
176+
</uap3:Extension>
177+
```
178+
179+
### Access win32 APIs for window registration
180+
181+
Fast path activation is enabled by setting a property on the [IPropertyStore](/windows/win32/api/propsys/nn-propsys-ipropertystore) associated with one of the app's windows. To do this requires access to some native Win32 APIs. This walkthrough will use the CsWin32 library, which automates the generation of C# bindings and is available as a NuGet package.
182+
183+
In Visual Studio, in **Solution Explorer**, right-click on your project file and select **Manage NuGet packages...**. On the **Browse** tab of the NuGet package manager, search for "cswin32" and select the "Microsoft.Windows.CsWin32" package and click **Install*.
184+
185+
After the package is installed, add a new text file in your project directory and name it "NativeMethods.txt". The CsWin32 tool will look in this file for a list of the Win32 APIs that it will generate bindings for. Put the following API names in "NativeMethods.txt".
186+
187+
`SUBCLASSPROC`
188+
189+
`SHGetPropertyStoreForWindow`
190+
191+
`IPropertyStore`
192+
193+
`SetWindowSubclass`
194+
195+
`DefSubclassProc`
196+
197+
### Register the window for Microsoft Copilot fastpath invocation
198+
199+
Next we will update the **MainWindow** class to register the window to receive fastpath invocations from the Copilot hardware key.
200+
201+
First, call [GetWindowHandle](/windows/windows-app-sdk/api/win32/microsoft.ui.xaml.window/nf-microsoft-ui-xaml-window-iwindownative-get_windowhandle) to get an [HWND](/windows/win32/winprog/windows-data-types) handle to the **MainWindow**. Call [SHGetPropertyStoreForWindow](/windows/win32/api/shellapi/nf-shellapi-shgetpropertystoreforwindow) to get the **IPropertyStore** for the window. Create a new [PROPERTYKEY](/windows/win32/api/wtypes/ns-wtypes-propertykey) and set the *fmtid* member to the GUID for Windows Copilot fastpath activation. Set the value of the property to an app-defined value, that will be passed back to the app from the system when the hardware key state changes. The app-defined value is the windows message ID which must be in the WM_APP range. For more information, see [WM_APP](/windows/win32/winmsg/wm-app). Call [SetValue](/windows/win32/api/propsys/nf-propsys-ipropertystore-setvalue) and then call [Commit](/windows/win32/api/propsys/nf-propsys-ipropertystore-commit) to commit the change to the property store.
202+
203+
Finally, create a [SUBCLASSPROC](/windows/win32/api/commctrl/nc-commctrl-subclassproc) callback that will be called when the hardware key state changes. **WindowSubClass** is the callback implementation that will be shown in the next step. Call [SetWindowSubclass](/windows/win32/api/commctrl/nf-commctrl-setwindowsubclass) to register the callback.
204+
205+
```csharp
206+
private HWND hWndMain;
207+
private Windows.Win32.UI.Shell.SUBCLASSPROC SubClassDelegate;
208+
public const int WM_COPILOT = 0x8000 + 0x0001;
209+
210+
public MainWindow(string state)
211+
{
212+
this.InitializeComponent();
213+
214+
hWndMain = (HWND)WinRT.Interop.WindowNative.GetWindowHandle(this);
215+
Microsoft.UI.Windowing.AppWindow appWindow = AppWindow;
216+
217+
218+
var propertyStoreGUID = new Guid("886D8EEB-8CF2-4446-8D02-CDBA1DBDCF99");
219+
var hr = PInvoke.SHGetPropertyStoreForWindow((HWND)this.AppWindow.Id.Value, in propertyStoreGUID, out var propertyStore);
220+
var key = new PROPERTYKEY();
221+
var copilotFastpathGUID = new Guid("38652BCA-4329-4E74-86F9-39CF29345EEA");
222+
key.fmtid = copilotFastpathGUID;
223+
key.pid = 0x00000002;
224+
var value = new PROPVARIANT();
225+
value.Anonymous.Anonymous.vt = VARENUM.VT_UINT;
226+
value.Anonymous.decVal = WM_COPILOT;
227+
((IPropertyStore)propertyStore).SetValue(in key, in value);
228+
((IPropertyStore)propertyStore).Commit();
229+
230+
SubClassDelegate = new Windows.Win32.UI.Shell.SUBCLASSPROC(WindowSubClass);
231+
bool bRet = PInvoke.SetWindowSubclass((HWND)appWindow.Id.Value, SubClassDelegate, 0, 0);
232+
233+
_state = state;
234+
}
235+
```
236+
237+
### Implement the window subclass callback
238+
239+
The last step in this example is implementing the window subclass callback that will be called whenever the app is running and the state of the Windows Copilot hardware key changes. In this example, we check that the window message is the **WM_COPILOT** value that we specified when setting the property store value in the previous step. Then we check the value of the *wParam* argument to see which of the values we specified with the **MessageWParam** attributes in the app manifest has been passed in. **SetState** is called to update the UI with the current state.
240+
241+
```csharp
242+
private LRESULT WindowSubClass(HWND hWnd, uint uMsg, WPARAM wParam, LPARAM lParam, nuint uIdSubclass, nuint dwRefData)
243+
{
244+
switch (uMsg)
245+
{
246+
case WM_COPILOT:
247+
{
248+
switch (wParam.Value)
249+
{
250+
case 0:
251+
SetState("SingleTap");
252+
break;
253+
case 1:
254+
SetState("PressAndHold START");
255+
break;
256+
case 2:
257+
SetState("PressAndHold END");
258+
break;
259+
}
260+
}
261+
break;
262+
263+
}
264+
return PInvoke.DefSubclassProc((HWND)hWnd, uMsg, wParam, lParam);
265+
266+
}
267+
```
268+

‎hub/apps/develop/windows-integration/microsoft-copliot-key-provider.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ ms.localizationpriority: medium
1010

1111
# Microsoft Copilot hardware key providers
1212

13-
Starting with Windows Build 22621, apps can register to be included in the picker UI that allows users to select the app that is launched when the Microsoft Copilot hardware key is pressed.
13+
Starting with Windows Build 22621, apps can register to be included in the picker UI that allows users to select the app that is launched when the Microsoft Copilot hardware key or the Windows key + C is pressed.
1414

1515
> [!NOTE]
1616
> It is recommended that apps that register to be a Microsoft Copilot hardware key provider be implemented as single-window apps.

‎hub/apps/toc.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -360,6 +360,9 @@ items:
360360
href: develop/feeds/feed-provider-manifest.md
361361
- name: Microsoft Copilot hardware key providers
362362
href: develop/windows-integration/microsoft-copliot-key-provider.md
363+
items:
364+
- name: Handle Microsoft Copilot hardware key state changes
365+
href: develop/windows-integration/copilot-key-state.md
363366
- name: Search providers
364367
items:
365368
- name: Overview

0 commit comments

Comments
 (0)