deps: Remove Python 3.7 and 3.8 as supported runtimes (#2133)

* Initial batch of changes to remove 3.7 and 3.8

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* more updates to remove 3.7 and 3.8

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* updates samples/geography/reqs

* updates samples/magics/reqs

* updates samples/notebooks/reqs

* updates linting

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* updates conf due to linting issue

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* updates reqs.txt, fix mypy, lint, and debug in noxfile

* Updates owlbot to correct spacing issue in conf.py

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* updates owlbot imports

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* removes kokoro samples configs for 3.7 & 3.8

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* removes owlbots attempt to restore kokoro samples configs

* removes kokoro system-3.8.cfg

* edits repo sync settings

* updates assorted noxfiles for samples and pyproject.toml

* update test-samples-impl.sh

* updates install_deps template

* Edits to the contributing documentation

* deps: use pandas-gbq to determine schema in `load_table_from_dataframe` (#2095)

* feat: use pandas-gbq to determine schema in `load_table_from_dataframe`

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* fix some unit tests

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* bump minimum pandas-gbq to 0.26.1

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* drop pandas-gbq from python 3.7 extras

* relax warning message text assertion

* use consistent time zone presense/absense in time datetime system test

* Update google/cloud/bigquery/_pandas_helpers.py

* Update google/cloud/bigquery/_pandas_helpers.py

Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>

* remove pandas-gbq from at least 1 unit test and system test session

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>

* Feat: Adds foreign_type_info attribute to table class and adds unit tests. (#2126)

* adds foreign_type_info attribute to table

* feat: Adds foreign_type_info attribute and tests

* updates docstrings for foreign_type_info

* Updates property handling, especially as regards set/get_sub_prop

* Removes extraneous comments and debug expressions

* Refactors build_resource_from_properties w get/set_sub_prop

* updates to foreign_type_info, tests and wiring

* Adds logic to detect non-Sequence schema.fields value

* updates assorted tests and logic

* deps: updates required checks list in github (#2136)

* deps: updates required checks list in github

* deps: updates snippet and system checks in github to remove 3.9

* changes the order of two items in the list.

* updates linting

* reverts pandas back to 1.1.0

* Revert changes related to pandas <1.5

* Revert noxfile.py changes related to pandas <1.5

* Revert constraints-3.9 changes related to pandas <1.5

* Revert test_query_pandas.py changes related to pandas <1.5

* Revert test__pandas_helpers.py changes related to pandas <1.5

* Revert test__versions_helpers.py changes related to pandas <1.5

* Revert tnoxfile.py changes related to pandas <1.5

* Revert test__versions_helpers.py changes related to pandas <1.5

* Revert test_table.py changes related to pandas <1.5

* Update noxfile changes related to pandas <1.5

* Update pyproject.toml changes related to pandas <1.5

* Update constraints-3.9.txt changes related to pandas <1.5

* Update test_legacy_types.py changes related to pandas <1.5

* Updates magics.py as part of reverting from pandas 1.5

* Updates noxfile.py in reverting from pandas 1.5

* Updates pyproject.toml in reverting from pandas 1.5

* Updates constraints.txt in reverting from pandas 1.5

* Updates test_magics in reverting from pandas 1.5

* Updates test_table in reverting from pandas 1.5

* Updates in tests re: reverting from pandas 1.5

* Updates pyproject to match constraints.txt

* updates pyproject.toml to mirror constraints

* remove limit on virtualenv

* updates owlbot.py for test-samples-impl.sh

* updates to owlbot.py

* updates to test-samples-impl.sh

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* further updates to owlbot.py

* removes unneeded files

* adds presubmit.cfg back in

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Tim Sweña (Swast) <swast@google.com>

Author: 3 people

.coveragerc

@@ -9,6 +9,6 @@ omit =
     google/cloud/bigquery_v2/*  # Legacy proto-based types.
 exclude_lines =
     # Re-enable the standard pragma
-    pragma: NO COVER
+    pragma: (no cover|NO COVER)
     # Ignore debug-only repr
     def __repr__

.kokoro/presubmit/snippets-3.8.cfg renamed to .kokoro/presubmit/snippets-3.9.cfg

@@ -3,5 +3,5 @@
 # Only run this nox session.
 env_vars: {
     key: "NOX_SESSION"
-    value: "snippets-3.8"
+    value: "snippets-3.9"
 }

.kokoro/presubmit/system-3.8.cfg renamed to .kokoro/presubmit/system-3.9.cfg

@@ -3,5 +3,5 @@
 # Only run this nox session.
 env_vars: {
     key: "NOX_SESSION"
-    value: "system-3.8"
-}
+    value: "system-3.9"
+}

.kokoro/samples/python3.7/common.cfg

@@ -33,8 +33,7 @@ export PYTHONUNBUFFERED=1
 env | grep KOKORO
 
 # Install nox
-# `virtualenv==20.26.6` is added for Python 3.7 compatibility
-python3.9 -m pip install --upgrade --quiet nox virtualenv==20.26.6
+python3.9 -m pip install --upgrade --quiet nox virtualenv
 
 # Use secrets acessor service account to get secrets
 if [[ -f "${KOKORO_GFILE_DIR}/secrets_viewer_service_account.json" ]]; then

.kokoro/samples/python3.7/continuous.cfg

@@ -22,7 +22,7 @@ In order to add a feature:
   documentation.
 
 - The feature must work fully on the following CPython versions:
-  3.7, 3.8, 3.9, 3.10, 3.11, 3.12 and 3.13 on both UNIX and Windows.
+  3.9, 3.10, 3.11, 3.12 and 3.13 on both UNIX and Windows.
 
 - The feature must not add unnecessary dependencies (where
   "unnecessary" is of course subjective, but new dependencies should
@@ -143,13 +143,12 @@ Running System Tests
    $ nox -s system
 
    # Run a single system test
-   $ nox -s system-3.8 -- -k <name of test>
+   $ nox -s system-3.9 -- -k <name of test>
 
 
   .. note::
 
-      System tests are only configured to run under Python 3.8.
-      For expediency, we do not run them in older versions of Python 3.
+      System tests are configured to run under Python 3.9, 3.11, 3.12.
 
   This alone will not run the tests. You'll need to change some local
   auth settings and change some configuration in your project to
@@ -195,11 +194,11 @@ configure them just like the System Tests.
 
    # Run all tests in a folder
    $ cd samples/snippets
-   $ nox -s py-3.8
+   $ nox -s py-3.9
 
    # Run a single sample test
    $ cd samples/snippets
-   $ nox -s py-3.8 -- -k <name of test>
+   $ nox -s py-3.9 -- -k <name of test>
 
 ********************************************
 Note About ``README`` as it pertains to PyPI
@@ -221,16 +220,12 @@ Supported Python Versions
 
 We support:
 
--  `Python 3.7`_
--  `Python 3.8`_
 -  `Python 3.9`_
 -  `Python 3.10`_
 -  `Python 3.11`_
 -  `Python 3.12`_
 -  `Python 3.13`_
 
-.. _Python 3.7: https://docs.python.org/3.7/
-.. _Python 3.8: https://docs.python.org/3.8/
 .. _Python 3.9: https://docs.python.org/3.9/
 .. _Python 3.10: https://docs.python.org/3.10/
 .. _Python 3.11: https://docs.python.org/3.11/

.kokoro/samples/python3.7/periodic-head.cfg

@@ -52,11 +52,11 @@ dependencies.
 
 Supported Python Versions
 ^^^^^^^^^^^^^^^^^^^^^^^^^
-Python >= 3.7
+Python >= 3.9
 
 Unsupported Python Versions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Python == 2.7, Python == 3.5, Python == 3.6.
+Python == 2.7, Python == 3.5, Python == 3.6, Python == 3.7, and Python == 3.8.
 
 The last version of this library compatible with Python 2.7 and 3.5 is
 `google-cloud-bigquery==1.28.0`.

.kokoro/samples/python3.7/periodic.cfg

@@ -126,12 +126,12 @@
 
 if sys_major == 3 and sys_minor in (7, 8):
     warnings.warn(
-        "The python-bigquery library will stop supporting Python 3.7 "
-        "and Python 3.8 in a future major release expected in Q4 2024. "
+        "The python-bigquery library no longer supports Python 3.7 "
+        "and Python 3.8. "
         f"Your Python version is {sys_major}.{sys_minor}.{sys_micro}. We "
         "recommend that you update soon to ensure ongoing support. For "
         "more details, see: [Google Cloud Client Libraries Supported Python Versions policy](https://cloud.google.com/python/docs/supported-python-versions)",
-        PendingDeprecationWarning,
+        FutureWarning,
     )
 
 __all__ = [

.kokoro/samples/python3.7/presubmit.cfg

@@ -336,8 +336,13 @@ def types_mapper(arrow_data_type):
             ):
                 return range_date_dtype
 
-            elif range_timestamp_dtype is not None and arrow_data_type.equals(
-                range_timestamp_dtype.pyarrow_dtype
+            # TODO: this section does not have a test yet OR at least not one that is
+            # recognized by coverage, hence the pragma. See Issue: #2132
+            elif (
+                range_timestamp_dtype is not None
+                and arrow_data_type.equals(  # pragma: NO COVER
+                    range_timestamp_dtype.pyarrow_dtype
+                )
             ):
                 return range_timestamp_dtype
 

.kokoro/samples/python3.8/common.cfg

@@ -21,8 +21,6 @@
 
 from typing import Any
 
-from packaging import version
-
 try:
     import pyarrow  # type: ignore
 except ImportError:
@@ -101,14 +99,10 @@ def pyarrow_timestamp():
         pyarrow.decimal128(38, scale=9).id: "NUMERIC",
     }
 
-    # Adds bignumeric support only if pyarrow version >= 3.0.0
-    # Decimal256 support was added to arrow 3.0.0
-    # https://arrow.apache.org/blog/2021/01/25/3.0.0-release/
-    if version.parse(pyarrow.__version__) >= version.parse("3.0.0"):
-        _BQ_TO_ARROW_SCALARS["BIGNUMERIC"] = pyarrow_bignumeric
-        # The exact decimal's scale and precision are not important, as only
-        # the type ID matters, and it's the same for all decimal256 instances.
-        _ARROW_SCALAR_IDS_TO_BQ[pyarrow.decimal256(76, scale=38).id] = "BIGNUMERIC"
+    _BQ_TO_ARROW_SCALARS["BIGNUMERIC"] = pyarrow_bignumeric
+    # The exact decimal's scale and precision are not important, as only
+    # the type ID matters, and it's the same for all decimal256 instances.
+    _ARROW_SCALAR_IDS_TO_BQ[pyarrow.decimal256(76, scale=38).id] = "BIGNUMERIC"
 
 
 def bq_to_arrow_scalars(bq_scalar: str):

.kokoro/samples/python3.8/continuous.cfg

@@ -835,8 +835,10 @@ def schema(self):
         See
         https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#ExternalDataConfiguration.FIELDS.schema
         """
-        prop = self._properties.get("schema", {})
-        return [SchemaField.from_api_repr(field) for field in prop.get("fields", [])]
+        # TODO: The typehinting for this needs work. Setting this pragma to temporarily
+        # manage a pytype issue that came up in another PR. See Issue: #2132
+        prop = self._properties.get("schema", {})  # type: ignore
+        return [SchemaField.from_api_repr(field) for field in prop.get("fields", [])]  # type: ignore
 
     @schema.setter
     def schema(self, value):

.kokoro/samples/python3.8/periodic-head.cfg

@@ -13,7 +13,7 @@
 # limitations under the License.
 
 import copy
-from typing import Dict, Optional
+from typing import Dict, Optional, Union
 
 
 class AvroOptions:
@@ -106,7 +106,7 @@ def enable_list_inference(self, value: bool) -> None:
         self._properties["enableListInference"] = value
 
     @property
-    def map_target_type(self) -> str:
+    def map_target_type(self) -> Optional[Union[bool, str]]:
         """Indicates whether to simplify the representation of parquet maps to only show keys and values."""
 
         return self._properties.get("mapTargetType")

.kokoro/samples/python3.8/periodic.cfg

@@ -415,7 +415,9 @@ def __init__(self, job_id, client):
     @property
     def configuration(self) -> _JobConfig:
         """Job-type specific configurtion."""
-        configuration = self._CONFIG_CLASS()
+        # TODO: The typehinting for this needs work. Setting this pragma to temporarily
+        # manage a pytype issue that came up in another PR. See Issue: #2132
+        configuration = self._CONFIG_CLASS()  # pytype: disable=not-callable
         configuration._properties = self._properties.setdefault("configuration", {})
         return configuration
 

.kokoro/samples/python3.8/presubmit.cfg

@@ -55,7 +55,6 @@
 except ImportError:
     bigquery_magics = None
 
-
 IPYTHON_USER_AGENT = "ipython-{}".format(IPython.__version__)  # type: ignore
 
 

.kokoro/test-samples-impl.sh

@@ -58,7 +58,7 @@ def __init__(self, model_ref: Union["ModelReference", str, None]):
         # semantics. The BigQuery API makes a distinction between an unset
         # value, a null value, and a default value (0 or ""), but the protocol
         # buffer classes do not.
-        self._properties = {}
+        self._properties: Dict[str, Any] = {}
 
         if isinstance(model_ref, str):
             model_ref = ModelReference.from_string(model_ref)