Add example 5 with Nautobot to PeeringDB example

Author: chadell

examples/05-nautobot-peeringdb/README.md

@@ -0,0 +1,65 @@
+# playground-diffsync
+
+## Description
+
+This repository is just a set of examples around [diffsync](https://github.com/networktocode/diffsync) in order to understand how it works and how you could use it.
+
+## Install dependencies
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip3 install -r requirements.txt
+```
+
+## Context
+
+The goal of this example is to synchronize some data from [PeeringDB](https://www.peeringdb.com/), that as the name suggests is a DB where peering entities define their facilities and presence to facilitate peering, towards [Nautobot Demo](https://demo.nautobot.com/) that is a always on demo service for [Nautobot](https://nautobot.readthedocs.io/), an open source Source of Truth.
+
+In Peering DB there is a model that defines a `Facility` and you can get information about the actual data center and the city where it is placed. In Nautobot, this information could be mapped to the `Region` and `Site` models, where `Region` can define hierarchy. For instance, Barcelona is in Spain and Spain is in Europe, and all of them are `Regions`. And, finally, the actual datacenter will refer to the `Region` where it is placed.
+
+Because of the nature of the demo, we will focus on syncing from PeeringDB to Nautobot (we can assume that PeeringDB is the authoritative System of Record) and we will skip the `delete` part of the `diffsync` library.
+
+## Show me the code
+
+We have 3 files:
+
+- `models.py`: defines the reference models that we will use: `RegionMode` and `SiteModel`
+- `adapter_peeringdb.py`: defines the PeeringDB adapter to translate via `load()` the data from PeeringDB into the reference models commented above. Notice that we don't define CRUD methods because we will sync from it (no to it)
+- `adapter_nautobot.py`: deifnes the Nautobot adapter with the `load()` and the CRUD methods
+
+```python
+from IPython import embed
+embed(colors="neutral")
+
+# Import Adapaters
+from adapter_nautobot import NautobotRemote
+from adapter_peeringdb import PeeringDB
+
+# Initialize PeeringDB adapter, using CATNIX id for demonstration
+peeringdb = PeeringDB(ix_id=62)
+
+# Initialize Nautobot adapter, pointing to the demo instance (it's also the default settings)
+nautobot = NautobotRemote(
+    url="https://demo.nautobot.com",
+    token="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+)
+
+# Load PeeringDB info into the adapter
+peeringdb.load()
+
+# We can check the data that has been imported, some as `site` and some as `region` (with the parent relationships)
+peeringdb.dict()
+
+# Load Nautobot info into the adapter
+nautobot.load()
+
+# Let's diffsync do it's magic
+diff = nautobot.diff_from(peeringdb)
+
+# Quick summary of the expected changes (remember that delete ones are dry-run)
+diff.summary()
+
+# Execute the synchronization
+nautobot.sync_from(peeringdb)
+```

examples/05-nautobot-peeringdb/adapter_nautobot.py

@@ -0,0 +1,201 @@
+"""Diffsync adapter class for Nautobot."""
+import os
+import requests
+from diffsync import DiffSync
+
+from .models import RegionModel, SiteModel
+
+NAUTOBOT_URL = os.getenv("NAUTOBOT_URL", "https://demo.nautobot.com")
+NAUTOBOT_TOKEN = os.getenv("NAUTOBOT_TOKEN", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
+
+
+class RegionNautobotModel(RegionModel):
+    """Implementation of Region create/update/delete methods for updating remote Nautobot data."""
+
+    @classmethod
+    def create(cls, diffsync, ids, attrs):
+        """Create a new Region record in remote Nautobot.
+        Args:
+            diffsync (NautobotRemote): DiffSync adapter owning this Region
+            ids (dict): Initial values for this model's _identifiers
+            attrs (dict): Initial values for this model's _attributes
+        """
+        data = {
+            "name": ids["name"],
+            "slug": attrs["slug"],
+        }
+        if attrs["description"]:
+            data["description"] = attrs["description"]
+        if attrs["parent_name"]:
+            data["parent"] = str(diffsync.get(diffsync.region, attrs["parent_name"]).pk)
+        diffsync.post("/api/dcim/regions/", data)
+        return super().create(diffsync, ids=ids, attrs=attrs)
+
+    def update(self, attrs):
+        """Update an existing Region record in remote Nautobot.
+        Args:
+            attrs (dict): Updated values for this record's _attributes
+        """
+        data = {}
+        if "slug" in attrs:
+            data["slug"] = attrs["slug"]
+        if "description" in attrs:
+            data["description"] = attrs["description"]
+        if "parent_name" in attrs:
+            if attrs["parent_name"]:
+                data["parent"] = {"name": attrs["parent_name"]}
+            else:
+                data["parent"] = None
+        self.diffsync.patch(f"/api/dcim/regions/{self.pk}/", data)
+        return super().update(attrs)
+
+    def delete(self):
+        """Delete an existing Region record from remote Nautobot."""
+        # self.diffsync.delete(f"/api/dcim/regions/{self.pk}/")
+        return super().delete()
+
+
+class SiteNautobotModel(SiteModel):
+    """Implementation of Site create/update/delete methods for updating remote Nautobot data."""
+
+    @classmethod
+    def create(cls, diffsync, ids, attrs):
+        """Create a new Site in remote Nautobot.
+        Args:
+            diffsync (NautobotRemote): DiffSync adapter owning this Site
+            ids (dict): Initial values for this model's _identifiers
+            attrs (dict): Initial values for this model's _attributes
+        """
+        diffsync.post(
+            "/api/dcim/sites/",
+            {
+                "name": ids["name"],
+                "slug": attrs["slug"],
+                "description": attrs["description"],
+                "status": attrs["status_slug"],
+                "region": {"name": attrs["region_name"]} if attrs["region_name"] else None,
+                "latitude": attrs["latitude"],
+                "longitude": attrs["longitude"],
+            },
+        )
+        return super().create(diffsync, ids=ids, attrs=attrs)
+
+    def update(self, attrs):
+        """Update an existing Site record in remote Nautobot.
+        Args:
+            attrs (dict): Updated values for this record's _attributes
+        """
+        data = {}
+        if "slug" in attrs:
+            data["slug"] = attrs["slug"]
+        if "description" in attrs:
+            data["description"] = attrs["description"]
+        if "status_slug" in attrs:
+            data["status"] = attrs["status_slug"]
+        if "region_name" in attrs:
+            if attrs["region_name"]:
+                data["region"] = {"name": attrs["region_name"]}
+            else:
+                data["region"] = None
+        if "latitude" in attrs:
+            data["latitude"] = attrs["latitude"]
+        if "longitude" in attrs:
+            data["longitude"] = attrs["longitude"]
+        self.diffsync.patch(f"/api/dcim/sites/{self.pk}/", data)
+        return super().update(attrs)
+
+    def delete(self):
+        """Delete an existing Site record from remote Nautobot."""
+        # self.diffsync.delete(f"/api/dcim/sites/{self.pk}/")
+        return super().delete()
+
+
+class NautobotRemote(DiffSync):
+    """DiffSync adapter class for loading data from a remote Nautobot instance using Python requests.
+    In a more realistic example, you'd probably use PyNautobot here instead of raw requests,
+    but we didn't want to add PyNautobot as a dependency of this plugin just to make an example more realistic.
+    """
+
+    # Model classes used by this adapter class
+    region = RegionNautobotModel
+    site = SiteNautobotModel
+
+    # Top-level class labels, i.e. those classes that are handled directly rather than as children of other models
+    top_level = ("region", "site")
+
+    def __init__(self, *args, url=NAUTOBOT_URL, token=NAUTOBOT_TOKEN, **kwargs):
+        """Instantiate this class, but do not load data immediately from the remote system.
+        Args:
+            url (str): URL of the remote Nautobot system
+            token (str): REST API authentication token
+            job (Job): The running Job instance that owns this DiffSync adapter instance
+        """
+        super().__init__(*args, **kwargs)
+        if not url or not token:
+            raise ValueError("Both url and token must be specified!")
+        self.url = url
+        self.token = token
+        self.headers = {
+            "Accept": "application/json",
+            "Authorization": f"Token {self.token}",
+        }
+
+    def load(self):
+        """Load Region and Site data from the remote Nautobot instance."""
+        # To keep the example simple, we disable REST API pagination of results.
+        # In a real job, especially when expecting a large number of results,
+        # we'd want to handle pagination, or use something like PyNautobot.
+        region_data = requests.get(f"{self.url}/api/dcim/regions/", headers=self.headers, params={"limit": 0}).json()
+        regions = region_data["results"]
+        while region_data["next"]:
+            region_data = requests.get(region_data["next"], headers=self.headers, params={"limit": 0}).json()
+            regions.extend(region_data["results"])
+
+        for region_entry in regions:
+            region = self.region(
+                name=region_entry["name"],
+                slug=region_entry["slug"],
+                description=region_entry["description"] or None,
+                parent_name=region_entry["parent"]["name"] if region_entry["parent"] else None,
+                pk=region_entry["id"],
+            )
+            self.add(region)
+            # self.job.log_debug(message=f"Loaded {region} from remote Nautobot instance")
+
+        site_data = requests.get(f"{self.url}/api/dcim/sites/", headers=self.headers, params={"limit": 0}).json()
+        sites = site_data["results"]
+        while site_data["next"]:
+            site_data = requests.get(site_data["next"], headers=self.headers, params={"limit": 0}).json()
+            sites.extend(site_data["results"])
+
+        for site_entry in sites:
+            site = self.site(
+                name=site_entry["name"],
+                slug=site_entry["slug"],
+                status_slug=site_entry["status"]["value"] if site_entry["status"] else "active",
+                region_name=site_entry["region"]["name"] if site_entry["region"] else None,
+                description=site_entry["description"],
+                longitude=site_entry["longitude"],
+                latitude=site_entry["latitude"],
+                pk=site_entry["id"],
+            )
+            self.add(site)
+            # self.job.log_debug(message=f"Loaded {site} from remote Nautobot instance")
+
+    def post(self, path, data):
+        """Send an appropriately constructed HTTP POST request."""
+        response = requests.post(f"{self.url}{path}", headers=self.headers, json=data)
+        response.raise_for_status()
+        return response
+
+    def patch(self, path, data):
+        """Send an appropriately constructed HTTP PATCH request."""
+        response = requests.patch(f"{self.url}{path}", headers=self.headers, json=data)
+        response.raise_for_status()
+        return response
+
+    def delete(self, path):
+        """Send an appropriately constructed HTTP DELETE request."""
+        response = requests.delete(f"{self.url}{path}", headers=self.headers)
+        response.raise_for_status()
+        return response

examples/05-nautobot-peeringdb/adapter_peeringdb.py

@@ -0,0 +1,55 @@
+"""Diffsync adapter class for PeeringDB."""
+import requests
+from slugify import slugify
+import pycountry
+
+from diffsync import DiffSync
+from diffsync.exceptions import ObjectNotFound
+
+from .models import RegionModel, SiteModel
+
+PEERINGDB_URL = "https://peeringdb.com/"
+
+
+class PeeringDB(DiffSync):
+    """DiffSync adapter using pysnow to communicate with PeeringDB."""
+
+    # Model classes used by this adapter class
+    region = RegionModel
+    site = SiteModel
+
+    # Top-level class labels, i.e. those classes that are handled directly rather than as children of other models
+    top_level = ("region", "site")
+
+    def __init__(self, *args, ix_id, **kwargs):
+        """Initialize the PeeringDB adapter."""
+        super().__init__(*args, **kwargs)
+        self.ix_id = ix_id
+
+    def load(self):
+        """Load data via from PeeringDB."""
+        ix_data = requests.get(f"{PEERINGDB_URL}/api/ix/{self.ix_id}").json()
+
+        for fac in ix_data["data"][0]["fac_set"]:
+            # PeeringDB has no Region entity, so we must avoid duplicates
+            try:
+                self.get(self.region, fac["city"])
+            except ObjectNotFound:
+                region = self.region(
+                    name=fac["city"],
+                    slug=slugify(fac["city"]),
+                    parent_name=pycountry.countries.get(alpha_2=fac["country"]).name,
+                )
+                self.add(region)
+
+            site = self.site(
+                name=fac["name"],
+                slug=slugify(fac["name"]),
+                status_slug="active",
+                region_name=fac["city"],
+                description=fac["notes"],
+                longitude=fac["longitude"],
+                latitude=fac["latitude"],
+                pk=fac["id"],
+            )
+            self.add(site)

examples/05-nautobot-peeringdb/models.py

@@ -0,0 +1,56 @@
+"""DiffSyncModel subclasses for Nautobot-PeeringDB data sync."""
+from typing import Optional, Union
+from uuid import UUID
+
+from diffsync import DiffSyncModel
+
+
+class RegionModel(DiffSyncModel):
+    """Shared data model representing a Region."""
+
+    # Metadata about this model
+    _modelname = "region"
+    _identifiers = ("name",)
+    _attributes = (
+        "slug",
+        "description",
+        "parent_name",
+    )
+
+    # Data type declarations for all identifiers and attributes
+    name: str
+    slug: str
+    description: Optional[str]
+    parent_name: Optional[str]  # may be None
+
+    # Not in _attributes or _identifiers, hence not included in diff calculations
+    pk: Optional[UUID]
+
+
+class SiteModel(DiffSyncModel):
+    """Shared data model representing a Site in either of the local or remote Nautobot instances."""
+
+    # Metadata about this model
+    _modelname = "site"
+    _identifiers = ("name",)
+    # To keep this example simple, we don't include **all** attributes of a Site here. But you could!
+    _attributes = (
+        "slug",
+        "status_slug",
+        "region_name",
+        "description",
+        "latitude",
+        "longitude",
+    )
+
+    # Data type declarations for all identifiers and attributes
+    name: str
+    slug: str
+    status_slug: str
+    region_name: Optional[str]  # may be None
+    description: Optional[str]
+    latitude: Optional[float]
+    longitude: Optional[float]
+
+    # Not in _attributes or _identifiers, hence not included in diff calculations
+    pk: Optional[Union[UUID, int]]