You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Complete the "device support" section
Closesgh-39
* Add device keywords to the creation functions
* Add the device object and a device array attribute
* Update for review comments, narrow scope of syntax and semantics
* Remove device object from API specification
* Address review comments about control method priority, and `*_like`
* A small tweak on wording for `.device`
Assumes a bit less about implementation. A string like `'cpu'` should meet the requirements, and it doesn't have `__neq__`.
Copy file name to clipboardExpand all lines: spec/design_topics/device_support.md
+99-1Lines changed: 99 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -2,5 +2,103 @@
2
2
3
3
# Device support
4
4
5
-
TODO. See https://github.com/data-apis/array-api/issues/39
5
+
For libraries that support execution on more than a single hardware device - e.g. CPU and GPU, or multiple GPUs - it is important to be able to control on which device newly created arrays get placed and where execution happens. Attempting to be fully implicit doesn't always scale well to situations with multiple GPUs.
6
6
7
+
Existing libraries employ one or more of these three methods to exert such control:
8
+
1. A global default device, which may be fixed or user-switchable.
9
+
2. A context manager to control device assignment within its scope.
10
+
3. Local control via explicit keywords and a method to transfer arrays to another device.
11
+
12
+
This standard chooses to add support for method 3 (local control), because it's the most explicit and granular, with its only downside being verbosity. A context manager may be added in the future - see {ref}`device-out-of-scope` for details.
13
+
14
+
15
+
## Intended usage
16
+
17
+
The intended usage for the device support in the current version of the
18
+
standard is _device handling in library code_. The assumed pattern is that
19
+
users create arrays (for which they can use all the relevant device syntax
20
+
that the library they use provides), and that they then pass those arrays
21
+
into library code which may have to do the following:
22
+
23
+
- Create new arrays on the same device as an array that's passed in.
24
+
- Determine whether two input arrays are present on the same device or not.
25
+
- Move an array from one device to another.
26
+
- Create output arrays on the same device as the input arrays.
27
+
- Pass on a specified device to other library code.
28
+
29
+
```{note}
30
+
Given that there is not much that's currently common in terms of
31
+
device-related syntax between different array libraries, the syntax included
32
+
in the standard is kept as minimal as possible while enabling the
33
+
above-listed use cases.
34
+
```
35
+
36
+
## Syntax for device assignment
37
+
38
+
The array API will offer the following syntax for device assignment and
39
+
cross-device data transfer:
40
+
41
+
1. A `.device` property on the array object, which returns a `Device` object
42
+
representing the device the data in the array is stored on, and supports
43
+
comparing devices for equality with `==` and `!=` within the same library
44
+
(e.g., by implementing `__eq__`); comparing device objects from different
45
+
libraries is out of scope).
46
+
2. A `device=None` keyword for array creation functions, which takes an
47
+
instance of a `Device` object.
48
+
3. A `.to_device(device)` method on the array object, with `device` again being
49
+
a `Device` object, to move an array to a different device.
50
+
51
+
```{note}
52
+
The only way to obtain a `Device` object is from the `.device` property on
53
+
the array object, hence there is no `Device` object in the array API itself
54
+
that can be instantiated to point to a specific physical or logical device.
55
+
```
56
+
57
+
58
+
## Semantics
59
+
60
+
Handling devices is complex, and some frameworks have elaborate policies for
61
+
handling device placement. Therefore this section only gives recommendations,
62
+
rather than hard requirements:
63
+
64
+
- Respect explicit device assignment (i.e. if the input to the `device=` keyword
65
+
is not `None`, guarantee that the array is created on the given device, and
66
+
raise an exception otherwise).
67
+
- Preserve device assignment as much as possible (e.g. output arrays from a
68
+
function are expected to be on the same device as input arrays to the
69
+
function).
70
+
- Raise an exception if an operation involves arrays on different devices
71
+
(i.e. avoid implicit data transfer between devices).
72
+
- Use a default for `device=None` which is consistent between functions
73
+
within the same library.
74
+
- If a library has multiple ways of controlling device placement, the most
75
+
explicit method should have the highest priority. For example:
76
+
1. If `device=` keyword is specified, that always takes precedence
77
+
2. If `device=None`, then use the setting from a context manager, if set.
78
+
3. If no context manager was used, then use the global default device/strategy
79
+
80
+
81
+
(device-out-of-scope)=
82
+
83
+
## Out of scope for device support
84
+
85
+
Individual libraries may offers APIs for one or more of the following topics,
86
+
however those are out of scope for this standard:
87
+
88
+
- Identifying a specific physical or logical device across libraries
89
+
- Setting a default device globally
90
+
- Stream/queue control
91
+
- Distributed allocation
92
+
- Memory pinning
93
+
- A context manager for device control
94
+
95
+
```{note}
96
+
A context manager for controlling the default device is present in most existing array
97
+
libraries (NumPy being the exception). There are concerns with using a
98
+
context manager however. A context manager can be tricky to use at a high
99
+
level, since it may affect library code below function calls (non-local
100
+
effects). See, e.g., [this PyTorch issue](https://github.com/pytorch/pytorch/issues/27878)
101
+
for a discussion on a good context manager API.
102
+
103
+
Adding a context manager may be considered in a future version of this API standard.
0 commit comments