Skip to content

Commit 40f8ee6

Browse files
rvermeulenrvermeulen
committed
Add minimal test case for deviation report generation
1 parent 5571453 commit 40f8ee6

File tree

6 files changed

+138
-0
lines changed

6 files changed

+138
-0
lines changed

‎scripts/reports/analysis_report_test.py

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
from pathlib import Path
33
import sys
44
from guideline_recategorizations import generate_guideline_recategorizations_report
5+
from deviations import generate_deviations_report
56

67
script_path = Path(__file__)
78
# Add the shared modules to the path so we can import them.
@@ -34,4 +35,28 @@ def test_guideline_recategorizations_report(tmp_path):
3435
expected = expected.replace("$codeql-version$", codeql.version).replace("$database-path$", str(db_path))
3536
actual = (tmp_path / "guideline_recategorizations_report.md").read_text()
3637

38+
assert(expected == actual)
39+
40+
def test_deviations_report(tmp_path):
41+
42+
db_path = tmp_path / 'test-db'
43+
src_root = TEST_DATA_DIR / 'deviations'
44+
codeql = CodeQL()
45+
46+
compile_src_command = "clang -fsyntax-only test.cpp"
47+
index_coding_standards_config_command = f"python3 {SCRIPTS_DIR}/configuration/process_coding_standards_config.py"
48+
49+
try:
50+
codeql.create_database(src_root, 'cpp', db_path, compile_src_command, index_coding_standards_config_command)
51+
except CodeQLError as err:
52+
print(err.stdout)
53+
print(err.stderr)
54+
raise err
55+
56+
generate_deviations_report(db_path, REPO_ROOT, tmp_path)
57+
58+
expected = (TEST_DATA_DIR / 'deviations' / 'deviations_report.md.expected').read_text()
59+
expected = expected.replace("$codeql-version$", codeql.version).replace("$database-path$", str(db_path))
60+
actual = (tmp_path / "deviations_report.md").read_text()
61+
3762
assert(expected == actual)

‎scripts/reports/test-data/deviations/coding-standards.yml

Whitespace-only changes.

‎scripts/reports/test-data/deviations/deviations_report.md.expected

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Deviations report
2+
3+
## Overview
4+
5+
- Report generated with supported CodeQL version $codeql-version$
6+
- Database path: $database-path$
7+
- 3 valid deviation records and 14 invalid deviation records found in the database
8+
- 3 valid deviation permits and 2 invalid deviation permits found in the database
9+
10+
## Deviation Records
11+
12+
| Rule ID | Query ID | Automated Scope | Scope | Justification | Background | Requirements
13+
| --- | --- | --- | --- | --- | --- | --- |
14+
| A0-1-1 | cpp/autosar/useless-assignment | Applies to the following file paths: invalid | | | | |
15+
| A0-1-1 | cpp/autosar/useless-assignment | Applies to the following file paths: valid | | This useless assignment is required. | | |
16+
| A0-4-2 | cpp/autosar/type-long-double-used | Identified by the use of the code-identifier: a-0-4-2-deviation | | long double is required for interaction with third-party libraries. | | |
17+
18+
## Invalid Deviation Records
19+
| Path | Reason |
20+
| --- | --- |
21+
| invalid/coding-standards.xml:5:7:5:26 | No rule-id and query-id specified for this deviation record. |
22+
| invalid/coding-standards.xml:6:7:8:26 | The rule-id `bad rule id` for this deviation matches none of the available queries. |
23+
| invalid/coding-standards.xml:9:7:11:26 | A query-id of `bad rule id` is specified for this deviation, but not rule-id is specified. |
24+
| invalid/coding-standards.xml:15:7:17:26 | A query-id of `cpp/autosar/useless-assignment` is specified for this deviation, but not rule-id is specified. |
25+
| invalid/coding-standards.xml:22:7:26:26 | A deviation `raised-by` is specified without providing an `approved-by`. |
26+
| invalid/coding-standards.xml:22:7:26:26 | A deviation `raised-by` is specified without providing both a `name` and `date`. |
27+
| invalid/coding-standards.xml:27:7:33:26 | A deviation `raised-by` is specified without providing an `approved-by`. |
28+
| invalid/coding-standards.xml:27:7:33:26 | A deviation `raised-by` is specified without providing both a `name` and `date`. |
29+
| invalid/coding-standards.xml:34:7:41:26 | A deviation `raised-by` is specified without providing an `approved-by`. |
30+
| invalid/coding-standards.xml:42:7:50:26 | A deviation `approved-by` is specified without providing both a `name` and `date`. |
31+
| invalid/coding-standards.xml:51:7:61:26 | A deviation `approved-by` is specified without providing both a `name` and `date`. |
32+
| invalid/coding-standards.xml:74:7:78:26 | There is no deviation permit with id `non-existing-permit`. |
33+
| invalid/coding-standards.xml:79:7:81:26 | No rule-id and query-id specified for this deviation record. |
34+
| invalid/coding-standards.xml:85:7:88:26 | The deviation is applied to a query with the rule category 'mandatory' that does not permit a deviation. |
35+
36+
## Deviation Permits
37+
38+
| Permit ID | Rule ID | Query ID | Automated Scope | Scope | Justification | Background | Requirements
39+
| --- | --- | --- | --- | --- | --- | --- | --- |
40+
| DP1 | | | Application depends on the associated deviation records | | foo bar baz | | |
41+
| DP2 | A0-1-1 | cpp/autosar/useless-assignment | Application depends on the associated deviation records | | | | |
42+
| DP3 | | | Application depends on the associated deviation records | | | | |
43+
44+
## Invalid Deviation Permits
45+
| Path | Reason |
46+
| --- | --- |
47+
| invalid/coding-standards.xml:100:7:103:33 | Deviation permit does not specify a permit identifier. |
48+
| invalid/coding-standards.xml:104:7:107:33 | Deviation permit specifies unknown property `invalid-property`. |

‎scripts/reports/test-data/deviations/invalid/coding-standards.yml

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
deviations:
2+
-
3+
- rule-id: bad rule id
4+
- query-id: bad rule id
5+
- rule-id: A0-1-1
6+
- query-id: cpp/autosar/useless-assignment
7+
- rule-id: A0-1-1
8+
query-id: cpp/autosar/useless-assignment
9+
- rule-id: A0-1-1
10+
query-id: cpp/autosar/useless-assignment
11+
raised-by:
12+
- rule-id: A0-1-1
13+
query-id: cpp/autosar/useless-assignment
14+
raised-by:
15+
name: foo1
16+
- rule-id: A0-1-1
17+
query-id: cpp/autosar/useless-assignment
18+
raised-by:
19+
name: foo2
20+
date: 1970-01-01Z
21+
- rule-id: A0-1-1
22+
query-id: cpp/autosar/useless-assignment
23+
raised-by:
24+
name: foo3
25+
date: 1970-01-01Z
26+
approved-by:
27+
- rule-id: A0-1-1
28+
query-id: cpp/autosar/useless-assignment
29+
raised-by:
30+
name: foo4
31+
date: 1970-01-01Z
32+
approved-by:
33+
name: bar1
34+
- rule-id: A0-1-1
35+
query-id: cpp/autosar/useless-assignment
36+
raised-by:
37+
name: foo5
38+
date: 1970-01-01Z
39+
approved-by:
40+
name: bar2
41+
date: 1970-01-01Z
42+
- rule-id: A0-1-1
43+
query-id: cpp/autosar/useless-assignment
44+
permit-id: non-existing-permit
45+
- permit-id: DP1
46+
- permit-id: DP2
47+
- rule-id: RULE-13-6
48+
query-id: c/misra/sizeof-operand-with-side-effect
49+
deviation-permits:
50+
- permit-id: DP1
51+
justification: foo bar baz
52+
- permit-id: DP2
53+
rule-id: A0-1-1
54+
query-id: cpp/autosar/useless-assignment
55+
- rule-id: A0-1-1
56+
query-id: cpp/autosar/useless-assignment
57+
- permit-id: DP3
58+
invalid-property: invalid-property

‎scripts/reports/test-data/deviations/test.cpp

Whitespace-only changes.

‎scripts/reports/test-data/deviations/valid/coding-standards.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
deviations:
2+
- rule-id: A0-1-1
3+
query-id: cpp/autosar/useless-assignment
4+
justification: This useless assignment is required.
5+
- rule-id: A0-4-2
6+
justification: long double is required for interaction with third-party libraries.
7+
code-identifier: a-0-4-2-deviation

0 commit comments

Comments
 (0)