Skip to content

Commit 404538b

Browse files
committed
Do not recognize docstrings in the tokenizer
In Python, docstrings seem to be the first item in the body of a function. They are not specifically recognized in the parser, and not in the tokenizer either. Thus we should follow the same approach. Tests were updated. This also fixed the warning: + re2c -W -b tokenizer.re -o tokenizer.cpp tokenizer.re:684:20: warning: unreachable rule (shadowed by rule at line 677) [-Wunreachable-rules]
1 parent 44b247e commit 404538b

File tree

3 files changed

+18
-6
lines changed

3 files changed

+18
-6
lines changed

src/lpython/parser/tokenizer.re

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -674,7 +674,7 @@ int Tokenizer::lex(Allocator &al, YYSTYPE &yylval, Location &loc, diag::Diagnost
674674
return yytokentype::TK_EOLCOMMENT;
675675
}
676676
}
677-
docstring { RET(TK_DOCSTRING) }
677+
//docstring { RET(TK_DOCSTRING) }
678678
679679
// Include statements are ignored for now
680680
'include' whitespace string1 { continue; }

tests/reference/tokens-docstring1-1355fbb.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
"outfile": null,
77
"outfile_hash": null,
88
"stdout": "tokens-docstring1-1355fbb.stdout",
9-
"stdout_hash": "8002935592daf4c952a5edb182c847a27e183ec70c3239fd060545f8",
9+
"stdout_hash": "4b40183da86ff53034cdbf2b7a70333443f5a233a9086caec0b81d69",
1010
"stderr": null,
1111
"stderr_hash": null,
1212
"returncode": 0

tests/reference/tokens-docstring1-1355fbb.stdout

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,24 +3,36 @@
33
(TOKEN "(") 9:9
44
(TOKEN ")") 10:10
55
(TOKEN ":") 11:11
6-
(TOKEN "docstring") 12:54
6+
(NEWLINE) 12:12
7+
(TOKEN "string" """A multi-line
8+
docstring.
9+
""") 17:54
710
(NEWLINE) 55:55
811
(NEWLINE) 56:56
912
(KEYWORD "def") 57:59
1013
(TOKEN "identifier" test2) 61:65
1114
(TOKEN "(") 66:66
1215
(TOKEN ")") 67:67
1316
(TOKEN ":") 68:68
14-
(TOKEN "docstring") 69:116
17+
(NEWLINE) 69:69
18+
(TOKEN "string" """
19+
A multi-line
20+
docstring.
21+
""") 74:116
1522
(NEWLINE) 117:117
1623
(NEWLINE) 118:118
1724
(KEYWORD "def") 119:121
1825
(TOKEN "identifier" test2) 123:127
1926
(TOKEN "(") 128:128
2027
(TOKEN ")") 129:129
2128
(TOKEN ":") 130:130
22-
(TOKEN "docstring") 131:166
29+
(NEWLINE) 131:131
30+
(TOKEN "string" """ A single-line docstring.""") 136:166
2331
(NEWLINE) 167:167
24-
(TOKEN "docstring") 168:199
32+
(NEWLINE) 168:168
33+
(TOKEN "string" """
34+
A multi-line
35+
docstring.
36+
""") 169:199
2537
(NEWLINE) 200:200
2638
(EOF) 201:201

0 commit comments

Comments
 (0)