Define syntax and format of REUSE.yaml

[mxmehl]

As discussed in spdx/spdx-spec#502, the SPDX project plans to support a "metadata, pre-document file" that contains specific information about files relative to its position. This follows a request to implement something called REUSE.yaml, first discussed here. This issue is to discuss the exact format and syntax of the file.

Proposed YAML options

In the original discussion, we proposed four different syntaxes. One of them (also disliked by the REUSE team) has been turned down in a SPDX call. I removed two others as they are rather unintuitive and clumsy. Also, I changed the format a bit to comply with the YAML syntax (using * as key name is invalid), and added another option.

Option 1: list

Each list item is a SPDX tag as used in file headers. Easy to read thanks to the -, but all items must be wrapped in " to escape the : which would separate a key from a value – we cannot have multiple keys!

- files: "src/*"
  info:
    - "SPDX-FileCopyrightText: 2020 Me"
    - "SPDX-FileCopyrightText: © 2017 You"
    - "SPDX-License-Identifier: MIT"

Option 2: multi-line string

SPDX tags are just separated by new lines. No - or escaping of : are required. However, indentation must be preserved for all lines!

- files: "src/*"
  info: |
    SPDX-FileCopyrightText: 2020 Me
    SPDX-FileCopyrightText: © 2017 You
    SPDX-License-Identifier: MIT

Option 3: license and copyright as separate keys

We could also separate the two information items. Downside: the keys must be wrapped in " to escape the - in the key name.

- files: "src/*"
  "SPDX-FileCopyrightText":
    - "2020 Me"
    - "© 2017 You"
  "SPDX-License-Identifier": MIT

Background on the YAML keys

Unlike the SPDX YAML format, we would like to avoid copyrightText and licenseDeclared as key names. In REUSE, the SPDX-License-Identifier and SPDX-FileCopyrightText (or alternatively traditional, varying copyright statements) are common and understood by the users.

This was also accepted in the SPDX call.

Possible targets

REUSE.yaml is intended to target files that are relative to its position, and only those that are "below".

Statements like files: "../../src/*" should not be possible.

Supporting traditional copyright statements?

A related question is whether we should only support SPDX-FileCopyrightText as indicator for files' copyright, or also "traditional" statements like "Copyright © 2021 Jane Doe".

REUSE recommends the SPDX tag, but also supports the traditional statements. My suggestion would be to do the same in REUSE.yaml to reduce friction, but in SPDX this could lead to conflicts. Happy to collect opinions here!

Globbing

DEP-5 uses a simple glob syntax. In this, */Makefile would include any Makefile in all paths below. I am not sure whether this globbing is represented in any native Python module. The benefit of sticking with the DEP-5 glob is that we could more easily convert existing DEP-5 files to REUSE.yaml.

Another possibility would be using the Python-native glob. */Makefile would only match a Makefile in one level below, while **/Makefile would match all Makefiles.

We could also use pathspec, supporting the same globbing as gitignore.

Conflict resolution

As in DEP-5, I would suggest that the last match of a file wins. So if the file foo.txt is first matched by * and then *.txt, the last statement would count.

The dependecy resolution within REUSE and its different options – including REUSE.yaml – is discussed in #70.

[silverhook]

When it comes to YAML flavours I think all should be OK – I guess we would use an external parser and linter anyway, right?

For files that reuse.yaml should target, I agree it should only affect its siblings and children. Parents etc. should be out of scope.

Regarding traditional copyright statements, I think it is reasonable to expect an SPDX tag, but after it, it should be free text form. Non-SPDX-tag statements were accepted before for legacy reasons. The YAML file is going to be new, so no legacy exists for it. Even if someone has a preferred format, they can just prepend it with SPDX tag.

Globbing – no preference, as long as it’s something that is in common practice and coherent.

Conflict resolution – I agree with your proposal.

[Jayman2000]

I think that the syntax should avoid the strings “SPDX-License-Identifier:” and “SPDX-<tagname>:”. Those strings are likely to cause false positives. Tools that aren’t REUSE.yml aware will mistakenly assume that the data applies to REUSE.yml. Here’s my proposal:

Option 1: list

- files: "src/*"
  info:
    - "FileCopyrightText: 2020 Me"
    - "FileCopyrightText: © 2017 You"
    - "License-Identifier: MIT"

Option 2: multi-line string

- files: "src/*"
  info: |
    FileCopyrightText: 2020 Me
    FileCopyrightText: © 2017 You
    License-Identifier: MIT

Option 3: license and copyright as separate keys

- files: "src/*"
  "FileCopyrightText":
    - "2020 Me"
    - "© 2017 You"
  "License-Identifier": MIT

If we do decide to drop the “SPDX-”, then I would recommend option 3. That way, if someone makes a mistake and includes the “SPDX-”, they have to do less to fix it.

I would also recommend making the REUSE Tool give a helpful error when this mistake happens. For example, it could say “Found ‘SPDX-License-Identifier’ in REUSE.yml. In REUSE.yml, use ‘License-Identifier’ instead (no ‘SPDX-’).”

[silverhook]

Great catch, @Jayman2000! What you write makes sense to me. It does provide some extra complication, but seems worth it to me in order to avoid future issues.

[andrewshadura]

Why not rename FileCopyrightText to copyrights and License-Identifier to license? A similar format is already used by scan-copyrights.

[mxmehl]

Why not rename FileCopyrightText to copyrights and License-Identifier to license? A similar format is already used by scan-copyrights.

We would like to have the SPDX project make this part of their spec, too, in order to not create conflicts with other compliance tools and practices (see: spdx/spdx-spec#502).

In SPDX, there are multiple "license" fields for instance, e.g. the concluded or declared license. I am afraid that this unclear terminology would not pass SPDX. However, main goal is to avoid confusion: so either we stick with the tags that are already used in REUSE (except in DEP5) or we make them really simple (as you suggested).

[andrewshadura]

We would like to have the SPDX project make this part of their spec, too, in order to not create conflicts with other compliance tools and practices (see: spdx/spdx-spec#502 <spdx/spdx-spec#502>).

Why I can see why you might want that, I'm not sure that's a goal worth pursuing. One of the reasons I keep my usage of SPDX to the minimum is its verbosity. I fear if and when your proposal is merged into SPDX, it's going to become yet another verbose way of specifying licensing information people will avoid. I'm also unsure why you want to deprecate DEP-5, which in my view is superior to many other similar formats. If something isn't quite right in it, I'd personally try to evolve it into a machine-readable copyright format 2.0 rather than abandon it completely.

[floriansnow]

Most of this looks good to me. I would like to add my two cents in regards to two things:

• I am in favor of globbing expressions that work with Python glob because that makes handling much easier and less error prone.
• I am still prefer JSON over YAML for better Python support; legibility is not an issue when it's formatted nicely

[andrewshadura]

I don’t think YAML vs JSON is an issue with Python: there are multiple YAML libraries for Python (pyyaml, ruamel, strict-yaml), so YAML is quite well-supported.
JSON is much less readable even when pretty-printed, it requires commas between list elements but not after, and I wouldn’t count on anything that generates it to actually pretty-print it. In my experience most generated JSON was dumped onto a single endless line, and most generated YAML was formatted and human-readable.

[floriansnow]

JSON is in the standard library and json.dump() supports decent printing with the indent parameter. Perhaps strict-yaml could serve a similar purpose, but most of the time, JSON is the stricter, more well defined version of YAML IMHO.

[mxmehl]

Why I can see why you might want that, I'm not sure that's a goal worth pursuing. One of the reasons I keep my usage of SPDX to the minimum is its verbosity. I fear if and when your proposal is merged into SPDX, it's going to become yet another verbose way of specifying licensing information people will avoid.

The files we intent to use have not much in common with a full SPDX SBOM, for which I agree that they are impossible to parse for humans. However, making REUSE's labelling compatible with an ISO standard has the great advantage that the likelihood of being compatible with other tools and best practices is much higher.

I see the advantage of creating own specs, but following the practice of "not invented here" even if there are somewhat good alternatives has only seldomly advanced technology.

I'm also unsure why you want to deprecate DEP-5, which in my view is superior to many other similar formats. If something isn't quite right in it, I'd personally try to evolve it into a machine-readable copyright format 2.0 rather than abandon it completely.

Please read the full discussion and proposal that I've linked in the first post. There are good reasons why DEP-5 is not ideal for our purpose: https://lists.fsfe.org/pipermail/reuse/2020q3/000085.html

[nicorikken]

Reading the discussion on spdx/spdx-spec#502 one thing stands out to me, the desire to align with the SPDX YAML. I think the current thoughts best align with the files section. The packages section apparently is of interested to the community as listed in the same thread, but that might be out of scope for now. So I think we need to look closer at
https://github.com/spdx/spdx-spec/blob/e25d183ade64c123770412297b9bf5086a7ed0bf/examples/SPDXYAMLExample-2.2.spdx.yaml#L241

Based on that I would consider a file like:

---
spdxVersion: "SPDX-2.3" # mandatory to allow future spec changes
creationInfo: # optional
  comment: "Easily add metadata to image files."
  created: "2022-05-25"
  # and other metadata if desired
# FIXME: perhaps needs information that this is to be considered input, not output
files:
# In line with SPDX YAML output
- copyrightText: "Copyright Photographer X"
  fileContributors: ["Photographer X"] # optional
  licenseConcluded: "CC-BY-4.0"
  fileName: "./images/other-author.jpg"
# My main proposal for simplicity
- fileGlob: "./images/*.jpg" #or another term, but to differentiate from 'fileName'
  copyrightText: |
    Copyright 2022 Photographer X
    Copyright © 2022 Image editor Y
  fileContributors:
    - "Photographer X"
    - "Image editor Y"
  licenseConcluded: "CC-BY-4.0" # I don't see a reason to change the key, or is there?

I know the format is quite different from earlier proposals:

• It is not in a a key hierarchy with the source although it is still an ordered list to help determine ordering when evaluating.
• Copyright information is just treated as text, without SPDX tags (does help avoid false positive scans)
• New term for fileGlob, another idea I have is the term filePath.

I step into this discussion quite late, so feel free to point out my false reasoning.