feat: re-use pyright config (#77)

In addition, added a few smaller features related to this
as described in the CHANGELOG.md entry that was created.

Author: tjdevries

.github/workflows/scip-snapshot.yml

@@ -1,8 +1,5 @@
 name: scip-snapshots
 
-env:
-  NODE_VERSION: '16.7.0'
-
 on:
   push:
     branches:
@@ -16,15 +13,20 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
-      - uses: actions/setup-node@v2
+      - name: Install asdf.
+        uses: asdf-vm/actions/setup@v2.1.0
+      - name: Cache asdf and asdf-managed tools.
+        uses: actions/cache@v3.3.1
+        id: asdf-cache
         with:
-          node-version: ${{ env.NODE_VERSION }}
-          registry-url: 'https://registry.npmjs.org'
-
+          path: ${{ env.ASDF_DIR }}
+          key: asdf-${{ runner.os}}-${{ hashFiles('**/.tool-versions') }}
+      - name: Install asdf tools (if not cached).
+        if: steps.asdf-cache.outputs.cache-hit != 'true'
+        uses: asdf-vm/actions/install@v2.1.0
       - name: Get npm cache directory
         id: npm-cache
-        run: |
-          echo "::set-output name=dir::$(npm config get cache)"
+        run: echo "dir=$(npm config get cache)" >> $GITHUB_OUTPUT
       - uses: actions/cache@v2
         with:
           path: ${{ steps.npm-cache.outputs.dir }}
@@ -33,9 +35,6 @@ jobs:
             ${{ runner.os }}-node-
 
       - run: npm install
-
       - run: cd ./packages/pyright-scip/ && npm install && npm run build
-
       - run: python --version
-
       - run: cd ./packages/pyright-scip/ && npm run check-snapshots

.tool-versions

@@ -0,0 +1,2 @@
+python 3.10.11
+nodejs 18.16.0

README.md

@@ -13,19 +13,87 @@ $ npm install -g @sourcegraph/scip-python
 
 scip-python requires Node v16 or newer. See the [Dockerfile](https://github.com/sourcegraph/scip-python/blob/scip/Dockerfile.autoindex) for an exact SHA that is tested.
 
+scip-python uses `pip` to attempt to determine the versions and names of the packages available in your environment. If you do not use pip to install the packages, you can instead use the `--environment` flag to supply a list of packages to use as the environment. This will skip any calls out to pip to determine the state of your env. See [Environment](##-environment) for more information.
+
+
 ## Usage
 
 ```
 $ npm install @sourcegraph/scip-python
 
 $ # NOTE: make sure to activate your virtual environment before running
-$ scip-python index . --project-name $MY_PROJECT
+$ scip-python index . --project-name=$MY_PROJECT
 
 $ # Make sure to point towards the sourcegraph instance you're interested in uploading to.
 $ #     more information at https://github.com/sourcegraph/src-cli
 $ src code-intel upload
 ```
 
+### target-only
+
+To run scip-python over only a particular directory, you can use the `--target-only` flag. Example:
+
+```
+$ scip-python index . --project-name=$MY_PROJECT --target-only=src/subdir
+```
+
+### project-namespace
+
+Additionally, if your project is loaded with some prefix, you can use the `--project-namespace` to put a namespace before all the generated symbols for this project.
+
+```
+$ scip-python index . --project-name=$MY_PROJECT --project-namespace=implicit.namespace
+```
+
+Now all symbols will have `implicit.namespace` prepended to their symbol, so that you can use it for cross repository navigation, even if the directory structure in your current project does not explicitly show `implicit/namespace/myproject/__init__.py`.
+
+## Environment
+
+The environment file format is a JSON list of `PythonPackage`s. The `PythonPackage` has the following form:
+
+```json
+{
+    "name": "PyYAML",
+    "version": "6.0",
+    "files": [
+      "PyYAML-6.0.dist-info/INSTALLER",
+      ...
+      "yaml/__init__.py",
+      "yaml/composer.py",
+      "yaml/tokens.py",
+      ...
+    ]
+},
+```
+
+Where:
+- `name`:
+  - The name of the package. Often times this is the same as the module, but is not always the case.
+  - For example, `PyYAML` is the name of the package, but the module is `yaml` (i.e. `import yaml`).
+- `version`:
+  - The vesion of the package. This is used to generate stable references to external packages.
+- `files`:
+  - A list of all the files that are a member of this package.
+  - Some packages declare multiple modules, so these should all be included.
+
+The environment file should be a list of these packages:
+
+```json
+[
+  { "name": "PyYAML", "version": "6.0", "files": [...] },
+  { "name": "pytorch", "version": "3.0", "files": [..] },
+  ...
+]
+```
+
+To use the environment file, you should call scip-python like so:
+
+```
+$ scip-python index --project-name=$MY_PROJECT --environment=path/to/env.json
+```
+
+If you're just using pip, this should not be required. We should calculate this from the pip environment. If you experience any bugs, please report them. The goal is that we support standard pip installation without additional configuration. If there is other python tooling that can generate this information, you can file an issue and we'll see if we can support it as well.
+
 ## Sourcegraph Example Configuration
 
 Using the usage example above may be quite simple to add a CI pipeline (perhaps using the `sourcegraph/scip-python:autoindex`) image

packages/pyright-scip/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Release v0.4
+
+- remove: `--include` and `--exclude`. Instead use `pyproject.toml` and pyright configuration.
+- add: `--target-only` to only emit and parse information related to some subdirectory of your project. Should still be run from root of project.
+- add: `--project-namespace` to prefix any definitions in your current project. This can be useful when your package gets installed in some non-standard way and there doesn't have the appropriate prefix that other python packages would import from.
+- Now respects pyright config by default (and discovers applicable pyright configuration).
+- Updated pyright internal library
+- Attempt to capture possible failures in pyright library so some indexing can still be completed.

packages/pyright-scip/package-lock.json

@@ -8,7 +8,7 @@
     "clean": "shx rm -rf ./dist ./out README.md LICENSE.txt",
     "prepack": "npm run clean && shx cp ../../README.md . && shx cp ../../LICENSE.txt . && npm run build",
     "check-snapshots": "npm run update-snapshots -- --check",
-    "update-snapshots": "node ./index.js snapshot-dir snapshots --environment snapshots/testEnv.json --no-progress-bar",
+    "update-snapshots": "node ./index.js snapshot-dir snapshots --environment snapshots/testEnv.json --quiet",
     "test": "jest --forceExit --detectOpenHandles",
     "webpack": "webpack --mode development --progress",
     "watch": "webpack --mode development --progress --watch"
@@ -42,6 +42,7 @@
     "webpack-cli": "^4.9.1"
   },
   "dependencies": {
+    "@iarna/toml": "2.2.5",
     "commander": "^9.2.0",
     "diff": "^5.0.0",
     "glob": "^7.2.0",

packages/pyright-scip/package.json

@@ -11,7 +11,7 @@
 #                             > ```
 
 print(A.SOME_CONSTANT)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,

packages/pyright-scip/snapshots/output/aliased_import/actual.py

@@ -38,7 +38,7 @@
 #                                        > ```
 
 print(re, Callable, Optional)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,

packages/pyright-scip/snapshots/output/builtin_imports/builtin_imports.py

@@ -53,7 +53,7 @@ def something(self):
 #                   > ```
 #                 ^^^^ definition  snapshot-util 0.1 class_nohint/Example#something().(self)
         print(self.x)
-#       ^^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#       ^^^^^ reference  python-stdlib 3.11 builtins/print().
 #       external documentation ```python
 #                   > (function) def print(
 #                   >     *values: object,
@@ -66,7 +66,7 @@ def something(self):
 #             ^^^^ reference  snapshot-util 0.1 class_nohint/Example#something().(self)
 #                  ^ reference  snapshot-util 0.1 class_nohint/Example#x.
         print(self.y)
-#       ^^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#       ^^^^^ reference  python-stdlib 3.11 builtins/print().
 #             ^^^^ reference  snapshot-util 0.1 class_nohint/Example#something().(self)
 #                  ^ reference  snapshot-util 0.1 class_nohint/Example#y.
 

packages/pyright-scip/snapshots/output/class_nohint/class_nohint.py

@@ -75,7 +75,7 @@ def something(x):
 #   ^^^ definition  snapshot-util 0.1 comp/var.
 #          ^^^^ reference  snapshot-util 0.1 comp/asdf.
     print(var)
-#   ^^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#   ^^^^^ reference  python-stdlib 3.11 builtins/print().
 #   external documentation ```python
 #               > (function) def print(
 #               >     *values: object,

packages/pyright-scip/snapshots/output/comprehensions/comp.py

@@ -8,7 +8,7 @@
 #              > __name__: str
 #              > ```
     print("main")
-#   ^^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#   ^^^^^ reference  python-stdlib 3.11 builtins/print().
 #   external documentation ```python
 #               > (function) def print(
 #               >     *values: object,

packages/pyright-scip/snapshots/output/dunder_vars/__main__.py

@@ -8,7 +8,7 @@
 #            > ```
 
 print(f"var: hello {var}")
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,

packages/pyright-scip/snapshots/output/f_string/fstring.py

@@ -6,7 +6,7 @@
 #               ^^^^^^^^^^^ reference  snapshot-util 0.1 `xyz.nested_file`/__init__:
 
 print(nested_file.X)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,

packages/pyright-scip/snapshots/output/file_from_module_import/abc/file.py

@@ -6,7 +6,7 @@
 #               ^^^^^^^^^^^ reference  snapshot-util 0.1 `xyz.nested_file`/__init__:
 
 print(nested_file.X)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,

packages/pyright-scip/snapshots/output/file_from_module_import/file_from_module.py

@@ -10,7 +10,7 @@
 #                                              ^^^^^^^^^^^^^^^^^^^^ reference  snapshot-util 0.1 `src.foo.bar.baz.mod`/AnotherNestedMuchWow#
 
 print(SuchNestedMuchWow().class_item)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,
@@ -23,11 +23,11 @@
 #     ^^^^^^^^^^^^^^^^^ reference  snapshot-util 0.1 `src.foo.bar.baz.mod`/SuchNestedMuchWow#
 #                         ^^^^^^^^^^ reference  snapshot-util 0.1 `src.foo.bar.baz.mod`/SuchNestedMuchWow#class_item.
 print(AnotherNestedMuchWow().other_item)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #     ^^^^^^^^^^^^^^^^^^^^ reference  snapshot-util 0.1 `src.foo.bar.baz.mod`/AnotherNestedMuchWow#
 #                            ^^^^^^^^^^ reference  snapshot-util 0.1 `src.foo.bar.baz.mod`/AnotherNestedMuchWow#other_item.
 print(InitClass().init_item)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #     ^^^^^^^^^ reference  snapshot-util 0.1 `src.foo.bar`/InitClass#
 #                 ^^^^^^^^^ reference  snapshot-util 0.1 `src.foo.bar`/InitClass#init_item.
 

packages/pyright-scip/snapshots/output/nested_items/src/importer.py

@@ -5,7 +5,7 @@
 #      ^^^^^^^^^^^^^^^ reference  snapshot-util 0.1 `foo.bar.baz.mod`/__init__:
 
 print(foo.bar.baz.mod.SuchNestedMuchWow)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,

packages/pyright-scip/snapshots/output/nested_items/src/long_importer.py

@@ -5,7 +5,7 @@
 #      ^^^^^^^^ reference  requests 2.0.0 requests/__init__:
 
 print(requests.get("https://sourcegraph.com"))
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,

packages/pyright-scip/snapshots/output/request_goofiness/goofy.py

@@ -32,7 +32,7 @@
 #                              > ```
 
 print(Any)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,

packages/pyright-scip/snapshots/output/unique/builtin_import_refs.py

@@ -86,7 +86,7 @@ def nested():
 #           ^ definition local 0
 #                ^^ reference  snapshot-util 0.1 property_access/usage().(xs)
             print(x.prop_ref)
-#           ^^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#           ^^^^^ reference  python-stdlib 3.11 builtins/print().
 #           external documentation ```python
 #                       > (function) def print(
 #                       >     *values: object,

packages/pyright-scip/snapshots/output/unique/property_access.py

@@ -6,7 +6,7 @@
 #      documentation (module): this_is_not_real [unable to re...
 
 print(this_is_not_real.x)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #external documentation ```python
 #            > (function) def print(
 #            >     *values: object,
@@ -18,7 +18,7 @@
 #            > ```
 #     ^^^^^^^^^^^^^^^^ reference local 0
 print(this_is_not_real.x)
-#^^^^ reference  python-stdlib 3.11 builtins/__init__:print().
+#^^^^ reference  python-stdlib 3.11 builtins/print().
 #     ^^^^^^^^^^^^^^^^ reference local 0
 
 

packages/pyright-scip/snapshots/output/unresolved_import/unresolved.py

@@ -25,6 +25,8 @@ checkIndexParser([], {
 });
 
 checkIndexParser(['--cwd', 'qux'], { cwd: 'qux' });
-checkIndexParser(['--no-progress-bar'], { quiet: false });
+checkIndexParser(['--quiet'], { quiet: true });
 checkIndexParser(['--show-progress-rate-limit', '120'], { showProgressRateLimit: 120 });
 checkIndexParser(['--show-progress-rate-limit', '0.5'], { showProgressRateLimit: 0.5 });
+checkIndexParser(['--target-only', 'foo'], { targetOnly: 'foo' });
+checkIndexParser(['--project-namespace', 'LSP'], { projectNamespace: 'LSP' });