Skip to content

Add run-tests.php --context [n] option. #5968

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

TysonAndre
Copy link
Contributor

Mentioned in #5965 (comment)

This PR proposes 3 lines of context so the impact can be seen in tests.
Other diff programs show around 3 lines of context.
(This helps indicate exactly which position a test should be updated
to add a new expected line at)

Are there reasons to continue not printing lines of context for test failures?
The lack of context seems inconvenient when investigating CI failures or when the line order wasn't obvious.

@TysonAndre
Copy link
Contributor Author

TysonAndre commented Aug 15, 2020

Example output can be seen at https://travis-ci.org/github/php/php-src/jobs/717434379 . Lines starting with -- were used as a placeholder in this PR to indicate where common lines of context end.

The line numbers will take up more columns for additions or removals in all places if either the expected or actual output take up more than 999 lines.

This PR was created before the PR to colorize diff output was merged.

@nikic @Girgias

========DIFF========
     test_time_spring_type3_prev_type2_prev: DIFF: 2010-03-13 18:38:28 EST - 2010-02-11 02:18:48 EST = **P+0Y1M2DT16H19M40S**
     test_time_spring_type3_prev_type2_st: DIFF: 2010-03-14 00:10:20 EST - 2010-03-13 18:38:28 EST = **P+0Y0M0DT5H31M52S**
     test_time_spring_type3_prev_type2_dt: DIFF: 2010-03-14 03:16:55 EDT - 2010-03-13 18:38:28 EST = **P+0Y0M0DT7H38M27S**
004+ test_time_spring_type3_prev_type2_post: DIFF: 2010-03-15 19:59:59 EDT - 2010-03-13 18:38:28 EST = **P+0Y0M2DT0H21M31S**
004- test_time_spring_type3_prev_type2_post: DIFF: 2010-03-15 19:59:59 EDT - 2010-03-13 18:38:28 EST = **P+0Y0M2DT1H21M31S**
     test_time_spring_type3_st_type2_prev: DIFF: 2010-03-13 18:38:28 EST - 2010-03-14 00:10:20 EST = **P-0Y0M0DT5H31M52S**
     test_time_spring_type3_st_type2_st: DIFF: 2010-03-14 00:15:35 EST - 2010-03-14 00:10:20 EST = **P+0Y0M0DT0H5M15S**
     test_time_spring_type3_st_type2_dt: DIFF: 2010-03-14 03:16:55 EDT - 2010-03-14 00:10:20 EST = **P+0Y0M0DT2H6M35S**
--
     test_time_spring_type3_dt_type2_st: DIFF: 2010-03-14 00:10:20 EST - 2010-03-14 03:16:55 EDT = **P-0Y0M0DT2H6M35S**
     test_time_spring_type3_dt_type2_dt: DIFF: 2010-03-14 05:19:56 EDT - 2010-03-14 03:16:55 EDT = **P+0Y0M0DT2H3M1S**
     test_time_spring_type3_dt_type2_post: DIFF: 2010-03-15 19:59:59 EDT - 2010-03-14 03:16:55 EDT = **P+0Y0M1DT16H43M4S**
013+ test_time_spring_type3_post_type2_prev: DIFF: 2010-03-13 18:38:28 EST - 2010-03-15 19:59:59 EDT = **P-0Y0M2DT0H21M31S**
013- test_time_spring_type3_post_type2_prev: DIFF: 2010-03-13 18:38:28 EST - 2010-03-15 19:59:59 EDT = **P-0Y0M2DT1H21M31S**
     test_time_spring_type3_post_type2_st: DIFF: 2010-03-14 00:10:20 EST - 2010-03-15 19:59:59 EDT = **P-0Y0M1DT18H49M39S**
     test_time_spring_type3_post_type2_dt: DIFF: 2010-03-14 03:16:55 EDT - 2010-03-15 19:59:59 EDT = **P-0Y0M1DT16H43M4S**
     test_time_spring_type3_post_type2_post: DIFF: 2010-03-15 19:59:59 EDT - 2010-03-15 18:57:55 EDT = **P+0Y0M0DT1H2M4S**
017+ test_time_spring_type3_stsec_type2_dtsec: DIFF: 2010-03-14 03:00:00 EDT - 2010-03-14 01:59:59 EST = **P+0Y0M0DT0H0M1S**
018+ test_time_spring_type3_dtsec_type2_stsec: DIFF: 2010-03-14 01:59:59 EST - 2010-03-14 03:00:00 EDT = **P-0Y0M0DT0H0M1S**
017- test_time_spring_type3_stsec_type2_dtsec: DIFF: 2010-03-15 03:00:00 EDT - 2010-03-13 01:59:59 EST = **P+0Y0M0DT0H0M1S**
018- test_time_spring_type3_dtsec_type2_stsec: DIFF: 2010-03-15 01:59:59 EST - 2010-03-15 03:00:00 EDT = **P-0Y0M0DT0H0M1S**
========DONE========
XFAIL DateTime::diff() -- spring type3 type2 [ext/date/tests/DateTime_diff-spring-type3-type2.phpt]   XFAIL REASON: Various bugs exist

The expectation of DateTime_diff-spring-type3-type2.phpt

--EXPECT--
test_time_spring_type3_prev_type2_prev: DIFF: 2010-03-13 18:38:28 EST - 2010-02-11 02:18:48 EST = **P+0Y1M2DT16H19M40S**
test_time_spring_type3_prev_type2_st: DIFF: 2010-03-14 00:10:20 EST - 2010-03-13 18:38:28 EST = **P+0Y0M0DT5H31M52S**
test_time_spring_type3_prev_type2_dt: DIFF: 2010-03-14 03:16:55 EDT - 2010-03-13 18:38:28 EST = **P+0Y0M0DT7H38M27S**
test_time_spring_type3_prev_type2_post: DIFF: 2010-03-15 19:59:59 EDT - 2010-03-13 18:38:28 EST = **P+0Y0M2DT1H21M31S**
test_time_spring_type3_st_type2_prev: DIFF: 2010-03-13 18:38:28 EST - 2010-03-14 00:10:20 EST = **P-0Y0M0DT5H31M52S**
test_time_spring_type3_st_type2_st: DIFF: 2010-03-14 00:15:35 EST - 2010-03-14 00:10:20 EST = **P+0Y0M0DT0H5M15S**
test_time_spring_type3_st_type2_dt: DIFF: 2010-03-14 03:16:55 EDT - 2010-03-14 00:10:20 EST = **P+0Y0M0DT2H6M35S**
test_time_spring_type3_st_type2_post: DIFF: 2010-03-15 19:59:59 EDT - 2010-03-14 00:10:20 EST = **P+0Y0M1DT18H49M39S**
test_time_spring_type3_dt_type2_prev: DIFF: 2010-03-13 18:38:28 EST - 2010-03-14 03:16:55 EDT = **P-0Y0M0DT7H38M27S**
test_time_spring_type3_dt_type2_st: DIFF: 2010-03-14 00:10:20 EST - 2010-03-14 03:16:55 EDT = **P-0Y0M0DT2H6M35S**
test_time_spring_type3_dt_type2_dt: DIFF: 2010-03-14 05:19:56 EDT - 2010-03-14 03:16:55 EDT = **P+0Y0M0DT2H3M1S**
test_time_spring_type3_dt_type2_post: DIFF: 2010-03-15 19:59:59 EDT - 2010-03-14 03:16:55 EDT = **P+0Y0M1DT16H43M4S**
test_time_spring_type3_post_type2_prev: DIFF: 2010-03-13 18:38:28 EST - 2010-03-15 19:59:59 EDT = **P-0Y0M2DT1H21M31S**
test_time_spring_type3_post_type2_st: DIFF: 2010-03-14 00:10:20 EST - 2010-03-15 19:59:59 EDT = **P-0Y0M1DT18H49M39S**
test_time_spring_type3_post_type2_dt: DIFF: 2010-03-14 03:16:55 EDT - 2010-03-15 19:59:59 EDT = **P-0Y0M1DT16H43M4S**
test_time_spring_type3_post_type2_post: DIFF: 2010-03-15 19:59:59 EDT - 2010-03-15 18:57:55 EDT = **P+0Y0M0DT1H2M4S**
test_time_spring_type3_stsec_type2_dtsec: DIFF: 2010-03-15 03:00:00 EDT - 2010-03-13 01:59:59 EST = **P+0Y0M0DT0H0M1S**
test_time_spring_type3_dtsec_type2_stsec: DIFF: 2010-03-15 01:59:59 EST - 2010-03-15 03:00:00 EDT = **P-0Y0M0DT0H0M1S**

Mentioned in php#5965 (comment)

This PR proposes 3 lines of context so the impact can be seen in tests.
Other `diff` programs show around 3 lines of context.
(This helps indicate exactly which position a test should be updated
to add a new expected line at)

Use the mapping for choosing order to display diffs

Properly include context in cases where the expected output had more lines than
the actual output, e.g.

```
--FILE--
A
A1
A
C
NEARBY
--EXPECTF--
A
B
A1
B
A
B
A
B
NEARBY
```
@Girgias
Copy link
Member

Girgias commented Aug 16, 2020

LGTM and can indeed help while debugging tests failures on CI.

Just FYI Nikita is currently on vacation so if you want his input, you will need to wait.

@php-pulls php-pulls closed this in 0461c15 Aug 16, 2020
@TysonAndre TysonAndre deleted the see-context-lines branch October 22, 2020 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants