diff --git a/README.md b/README.md index 1aa22f34..eba32fb8 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ Available source types: * VMware vCenter Server * AWS EC2, EBS, and network interfaces, with opt-in S3, Lambda, and EKS inventory (see [AWS source](docs/source_aws.md)) * [bb-ricardo/check_redfish](https://github.com/bb-Ricardo/check_redfish) inventory files +* OpenStack (optional `openstacksdk` dependency; see [OpenStack source](docs/source_openstack.md)) **IMPORTANT: READ INSTRUCTIONS CAREFULLY BEFORE RUNNING THIS PROGRAM** @@ -43,6 +44,9 @@ This ensures stale objects are removed from NetBox keeping an accurate current s * Optional boto3 SDK, AWS read-only credentials, and automatic or explicit region discovery * Optional AWS Organizations discovery with collector permissions and a trusted inventory role in each member account * Existing VPC-specific NetBox VRFs for IP import; see [AWS setup](docs/source_aws.md) +#### Source: OpenStack (if used) +* Identity v3, Nova and Cinder APIs; `openstacksdk>=4.0,<5` +* See [OpenStack installation and configuration](docs/source_openstack.md) for credentials, permissions, and inventory mapping # Installing * here we assume we install in ```/opt``` diff --git a/docs/source_openstack.md b/docs/source_openstack.md new file mode 100644 index 00000000..f00869e1 --- /dev/null +++ b/docs/source_openstack.md @@ -0,0 +1,194 @@ +# OpenStack source + +The `openstack` source reads availability zones, compute hypervisors, servers, +flavors, and Cinder volumes, then reconciles their inventory into NetBox. +OpenStack is queried read-only; inventory writes are made to NetBox. +All identifiers and credentials below are fictitious placeholders. + +## Installation + +The SDK is optional; other sources do not require it: + +```sh +pip install -r requirements-openstack.txt +# Alternatively: +uv sync --extra openstack +``` + +The supported SDK range is `openstacksdk>=4.0,<5`. Selecting an OpenStack source +without the SDK logs an error and skips that source. + +## Authentication and permissions + +The current source uses Keystone Identity v3 username/password authentication +with an explicit project, region, user domain, and project domain. It passes +these source settings directly to `openstacksdk`; it does not expose +`clouds.yaml` cloud selection or application-credential authentication. + +The account must be allowed to read: + +* Detailed Nova availability zones, including their host information. +* Detailed Nova hypervisors and servers. +* Nova flavors when server responses do not include their full specifications. +* Detailed Cinder volumes, including sizes for attached volumes. + +`all_projects = True` is the default for server and volume discovery and +requires cross-project access, normally administrative or suitably delegated +read permissions. Exact roles depend on the cloud's Nova and Cinder policy. +`all_projects = False` requests servers and volumes within the authenticated +project, but does not remove the availability-zone or hypervisor queries; +project-only credentials may still lack the permissions needed by this source. + +Use TLS certificate validation (the default). Configure the runtime's trusted +CA certificates for private endpoints rather than disabling verification. +The sync host needs outbound access to Keystone and the Nova/Cinder endpoints +advertised in the service catalog. No guest SSH access or guest agent is required +for addresses returned by Nova. + +## Configuration + +Add this section to the application's existing NetBox settings: + +```ini +[source/example-openstack] +type = openstack +auth_url = https://identity.example.org:5000/v3 +project = inventory +username = inventory-reader +password = REPLACE_WITH_PASSWORD +region = RegionOne +user_domain = Default +project_domain = Default +validate_tls_certs = True +all_projects = True +group_name = Example OpenStack +cluster_site_relation = .* = Example Cloud Site +permitted_subnets = 10.0.0.0/8, fd00::/8 +dns_name_lookup = False +``` + +All seven authentication/scope fields (`auth_url`, `project`, `username`, +`password`, `region`, `user_domain`, and `project_domain`) are mandatory. +`group_name` defaults to `Openstack`. Use separate named sources for other +regions or clouds; there is no automatic region enumeration. + +Keep real passwords out of version control and restrict access to configuration +files. The application also supports its standard source environment settings; +for example, a source defined entirely through environment variables can use: + +```sh +NBS_SOURCE_1_NAME=example-openstack +NBS_SOURCE_1_TYPE=openstack +NBS_SOURCE_1_AUTH_URL=https://identity.example.org:5000/v3 +NBS_SOURCE_1_PROJECT=inventory +NBS_SOURCE_1_USERNAME=inventory-reader +NBS_SOURCE_1_PASSWORD=REPLACE_WITH_PASSWORD +NBS_SOURCE_1_REGION=RegionOne +NBS_SOURCE_1_USER_DOMAIN=Default +NBS_SOURCE_1_PROJECT_DOMAIN=Default +``` + +These are example variable assignments; supply secrets through your runtime's +secret mechanism rather than storing this block in a repository. +Generate the complete supported configuration with: + +```sh +python netbox-sync.py -g +``` + +### Site, network, and tenant relations + +`cluster_site_relation` maps availability-zone names to NetBox sites using +regular expressions. `host_site_relation` can override sites for hypervisors. +Correct site mapping matters for prefix, VLAN, and VRF selection. + +Only addresses permitted by `permitted_subnets` are imported. Pre-create the +NetBox prefixes, VRFs, and VLANs appropriate to your cloud's networks. Address +prefix matching prefers the mapped site and can fall back to other matching +prefixes; without a matching prefix, a bare IPv4/IPv6 address retains its +host-length mask. Carefully inspect overlapping tenant address spaces before +permitting IP writes. + +Interfaces and addresses come from Nova server address data, not a separate +Neutron topology inventory. Fixed and floating addresses may both be imported +when permitted; fixed addresses are preferred for primary IP selection. +`set_primary_ip`, `preserve_primary_ips`, `skip_fhrp_group_ips`, +`ip_tenant_inheritance_order`, interface exclusion filters, and +`disable_vlan_sync` control the shared network reconciliation behavior. + +Tenants and tags are configured through relations rather than automatically +mirroring Keystone project names: + +```ini +cluster_site_relation = ^zone-a$ = Example Cloud Site +vm_tenant_relation = ^production-.* = Example Tenant +vm_tag_relation = .* = OpenStack +``` + +Host, VM, and cluster site/tenant/role/tag options and platform mappings are +described in the generated configuration and [example settings](../settings-example.ini). + +## Object mapping + +| OpenStack | NetBox | +| --- | --- | +| Configured group name | Cluster group | +| Availability zone | Cluster | +| Hypervisor | Device and reported host interface/IP | +| Server | Virtual machine | +| Flavor vCPUs / RAM | VM CPU / memory | +| Flavor name | VM platform unless disabled or mapped | +| Image-backed root / ephemeral disks | VM disk capacity or virtual disks | +| Attached Cinder volume | VM disk capacity or virtual disk named by volume ID | +| Server network/MAC/address data | VM interfaces, MACs, and permitted IP addresses | + +Servers with status `ACTIVE` are active in NetBox; other states map to offline. +By default VM names are used, with duplicate names within an availability zone +logged and skipped. Consider `set_vm_name_to_uuid = True` for cross-project +inventories where names repeat; the original name is then stored in the +`openstack_vm_name` custom field. Changing naming settings can affect matching, +so review a dry run before applying them to existing inventory. + +`skip_vm_platform` disables flavor-to-platform mapping. `skip_vm_comments` +disables VM UUID comments. `strip_host_domain_name` and +`strip_vm_domain_name` control domain stripping. +`host_custom_object_attributes` and `vm_custom_object_attributes` copy selected +SDK resource attributes into custom fields; dotted paths are supported. + +NetBox 3.7+ uses separate virtual disks; older versions retain aggregate VM +disk capacity. OpenStack reports flavor and volume sizes in gigabytes. NetBox +versions before 4.1 store the reported value as gigabytes; from 4.1, the source +converts it to megabytes by multiplying by 1000. Missing attached-volume sizes +are logged rather than replaced with fabricated zero sizes. NetBox 4.2+ uses +scoped cluster sites and separate MAC address objects. + +## Filters and lifecycle + +Cluster, host, and VM include/exclude filters are regular expressions matched +from the start of the name. Include checks run before exclude checks. +Excluding an availability zone excludes its hosts and VMs; excluding a host +does not by itself exclude its VMs. + +Objects participate in the application's normal source tagging, orphaning, and +pruning lifecycle. Changing project scope, filters, naming, or permitted networks +can change which objects are considered discovered. Avoid overlapping sources +and inspect the dry-run log before allowing writes or pruning. + +## Validation + +Start with a dry run using credentials with the intended scope: + +```sh +python netbox-sync.py -n -c settings.ini +``` + +Check authentication, availability-zone coverage, hypervisor access, server and +volume counts, site mappings, duplicate names, and address/VRF assignments. +Unit tests use a fake SDK and the real in-memory NetBox models: + +```sh +python -m pytest tests/test_openstack_source.py +``` + +These tests do not replace validation against your cloud's API versions and +authorization policies. diff --git a/module/config/file_output.py b/module/config/file_output.py index 9acf4e5a..9e8ab3c7 100644 --- a/module/config/file_output.py +++ b/module/config/file_output.py @@ -17,6 +17,7 @@ from module.sources.vmware.config import VMWareConfig from module.sources.check_redfish.config import CheckRedfishConfig from module.sources.hetzner.config import HetznerConfig +from module.sources.openstack.config import OpenStackConfig from module.sources.aws.config import AWSConfig from module.common.logging import get_logger from module.config import default_config_file_path, source_config_section_name @@ -37,7 +38,8 @@ class ConfigFileOutput(DescriptionFormatterMixin): VMWareConfig, CheckRedfishConfig, HetznerConfig, - AWSConfig + AWSConfig, + OpenStackConfig ] header = f"Welcome to the {__description__} configuration file." diff --git a/module/sources/__init__.py b/module/sources/__init__.py index 5cdc0f1b..7a2e18fa 100644 --- a/module/sources/__init__.py +++ b/module/sources/__init__.py @@ -10,6 +10,7 @@ # define all available sources here from module.sources.vmware.connection import VMWareHandler from module.sources.hetzner.connection import HetznerHandler +from module.sources.openstack.connection import OpenStackHandler from module.sources.aws.connection import AWSHandler from module.sources.check_redfish.import_inventory import CheckRedfish @@ -20,7 +21,7 @@ from module.config import source_config_section_name # list of valid sources -valid_sources = [VMWareHandler, CheckRedfish, HetznerHandler, AWSHandler] +valid_sources = [VMWareHandler, CheckRedfish, HetznerHandler, AWSHandler, OpenStackHandler] def validate_source(source_class_object=None, state="pre"): diff --git a/module/sources/common/source_base.py b/module/sources/common/source_base.py index 7538e5de..1966fabd 100644 --- a/module/sources/common/source_base.py +++ b/module/sources/common/source_base.py @@ -510,7 +510,8 @@ def add_update_interface(self, interface_object, device_object, interface_data, # skip handling of IPs for VMs with not installed/running guest tools skip_ip_handling = False - if type(device_object) == NBVM and grab(vmware_object,'guest.toolsRunningStatus') != "guestToolsRunning": + if type(device_object) == NBVM and vmware_object is not None and \ + grab(vmware_object, 'guest.toolsRunningStatus') != "guestToolsRunning": log.debug(f"VM '{device_object.name}' guest tool status is 'NotRunning', skipping IP handling") skip_ip_handling = True diff --git a/module/sources/openstack/config.py b/module/sources/openstack/config.py new file mode 100644 index 00000000..63238580 --- /dev/null +++ b/module/sources/openstack/config.py @@ -0,0 +1,394 @@ +# -*- coding: utf-8 -*- +# Copyright (c) 2020 - 2023 Ricardo Bartels. All rights reserved. +# +# netbox-sync.py +# +# This work is licensed under the terms of the MIT license. +# For a copy, see file LICENSE.txt included in this +# repository or visit: . + +import re +from ipaddress import ip_address + +from module.common.misc import quoted_split +from module.config import source_config_section_name +from module.config.base import ConfigBase +from module.config.option import ConfigOption +from module.config.group import ConfigOptionGroup +from module.sources.common.config import * +from module.sources.common.permitted_subnets import PermittedSubnets +from module.common.logging import get_logger + +log = get_logger() + + +class OpenStackConfig(ConfigBase): + + section_name = source_config_section_name + source_name = None + source_name_example = "my-openstack-example" + + def __init__(self): + self.options = [ + ConfigOption(**config_option_enabled_definition), + + ConfigOption(**{**config_option_type_definition, "config_example": "openstack"}), + + ConfigOption("auth_url", + str, + description="OpenStack Identity v3 authentication URL", + config_example="https://api.openstack-instance.local:5000/v3", + mandatory=True), + + ConfigOption("project", + str, + description="Name of the OpenStack project", + config_example="MyProject", + mandatory=True), + + ConfigOption("username", + str, + description="username to use to log into OpenStack", + config_example="openstackuser", + mandatory=True), + + ConfigOption("password", + str, + description="password to use to log into OpenStack", + config_example="supersecret", + sensitive=True, + mandatory=True), + + ConfigOption("region", + str, + description="Which OpenStack region to use", + config_example="RegionOne", + mandatory=True), + + ConfigOption("user_domain", + str, + description="Which OpenStack user domain to use", + config_example="Default", + mandatory=True), + + ConfigOption("project_domain", + str, + description="Which OpenStack project domain to use", + config_example="Default", + mandatory=True), + + ConfigOption("group_name", + str, + description="OpenStack group name", + default_value="Openstack"), + + ConfigOption("validate_tls_certs", + bool, + description="""Enforces TLS certificate validation. + Disable only for an OpenStack endpoint with a self-signed certificate.""", + default_value=True), + + ConfigOption("all_projects", bool, default_value=True, + description="Discover servers and volumes across projects (requires administrative access)."), + + ConfigOption(**config_option_permitted_subnets_definition), + ConfigOption(**config_option_ip_tenant_inheritance_order_definition), + ConfigOption("preserve_primary_ips", bool, default_value=False, + description="Keep existing primary IP assignments when an address is no longer discovered."), + ConfigOption("skip_fhrp_group_ips", bool, default_value=True, + description="Do not assign addresses belonging to an existing FHRP group."), + ConfigOption("host_interface_exclude_filter", str, + description="Regular expression matching host interface names to ignore."), + ConfigOption("vm_interface_exclude_filter", str, + description="Regular expression matching VM interface names to ignore."), + ConfigOption("disable_vlan_sync", bool, default_value=False, + description="Do not synchronize VLANs discovered through existing prefixes."), + + ConfigOptionGroup(title="filter", + description="""filters can be used to include/exclude certain objects from importing + into NetBox. Include filters are checked first and exclude filters after. + An object name has to pass both filters to be synced to NetBox. + If a filter is unset it will be ignored. Filters are all treated as regex expressions! + If more then one expression should match, a '|' needs to be used + """, + config_example="""Example: (exclude all VMs with "replica" in their name + and all VMs starting with "backup"): vm_exclude_filter = .*replica.*|^backup.*""", + options=[ + ConfigOption("cluster_exclude_filter", + str, + description="""If a cluster is excluded from sync then ALL VMs and HOSTS + inside the availability zone will be ignored!"""), + ConfigOption("cluster_include_filter", str), + ConfigOption("host_exclude_filter", + str, + description="""This will only include/exclude the host, + not the VM if Host is part of a multi host cluster"""), + ConfigOption("host_include_filter", str), + ConfigOption("vm_exclude_filter", + str, description="simply include/exclude VMs"), + ConfigOption("vm_include_filter", str) + ]), + ConfigOptionGroup(title="relations", + options=[ + ConfigOption("cluster_site_relation", + str, + description="""\ + This option defines which OpenStack availability zone is part of a NetBox site. + This is done with a comma separated key = value list. + key: defines the cluster name as regex + value: defines the NetBox site name (use quotes if name contains commas) + This is a quite important config setting as IP addresses, prefixes, VLANs + and VRFs are site dependent. In order to assign the correct prefix to an IP + address it is important to pick the correct site. + A VM always depends on the cluster site relation + """, + config_example="nova = Cloud Site"), + ConfigOption("host_site_relation", + str, + description="""Same as cluster site but on host level. + If unset it will fall back to cluster_site_relation""", + config_example="nyc02.* = New York, ffm01.* = Frankfurt"), + ConfigOption("cluster_tenant_relation", + str, + description="""\ + This option defines which cluster/host/VM belongs to which tenant. + This is done with a comma separated key = value list. + key: defines a hosts/VM name as regex + value: defines the NetBox tenant name (use quotes if name contains commas) + a cluster can be specified as "Cluster-name" or + "availability-zone-name" + """, + config_example="Cluster_NYC.* = Customer A"), + ConfigOption("host_tenant_relation", str, config_example="compute.* = Infrastructure"), + ConfigOption("vm_tenant_relation", str, config_example="grafana.* = Infrastructure"), + ConfigOption("vm_platform_relation", + str, + description="""\ + This option defines custom platforms if the VMWare created platforms are not suitable. + Pretty much a mapping of VMWare platform name to your own platform name. + This is done with a comma separated key = value list. + key: defines a VMWare returned platform name + value: defines the desired NetBox platform name""", + config_example="centos-7.* = centos7, microsoft-windows-server-2016.* = Windows2016"), + ConfigOption("host_role_relation", + str, + description="""\ + Define the NetBox device role used for hosts. The default is + set to "Server". This is done with a comma separated key = value list. + key: defines host(s) name as regex + value: defines the NetBox role name (use quotes if name contains commas) + """, + default_value=".* = Server"), + ConfigOption("vm_role_relation", + str, + description="""\ + Define the NetBox device role used for VMs. This is done with a + comma separated key = value list, same as 'host_role_relation'. + key: defines VM(s) name as regex + value: defines the NetBox role name (use quotes if name contains commas) + """, + config_example=".* = Server"), + ConfigOption("cluster_tag_relation", + str, + description="""\ + Define NetBox tags which are assigned to a cluster, host or VM. This is + done with a comma separated key = value list. + key: defines a hosts/VM name as regex + value: defines the NetBox tag (use quotes if name contains commas) + Match the availability zone name to assign cluster tags.""", + config_example="nova = Infrastructure"), + ConfigOption("host_tag_relation", str, config_example="compute.* = Infrastructure"), + ConfigOption("vm_tag_relation", str, config_example="grafana.* = Infrastructure") + ]), + ConfigOption("dns_name_lookup", + bool, + description="""Perform a reverse lookup for all collected IP addresses. + If a dns name was found it will be added to the IP address object in NetBox + """, + default_value=True), + ConfigOption("custom_dns_servers", + str, + description="use custom DNS server to do the reverse lookups", + config_example="192.168.1.11, 192.168.1.12"), + ConfigOption("set_primary_ip", + str, + description="""\ + define how the primary IPs should be set + possible values: + + always: will remove primary IP from the object where this address is + currently set as primary and moves it to new object + + when-undefined: + only sets primary IP if undefined, will cause ERRORs if same IP is + assigned more then once to different hosts and IP is set as the + objects primary IP + + never: don't set any primary IPs, will cause the same ERRORs + as "when-undefined" + """, + default_value="when-undefined"), + ConfigOption("skip_vm_comments", + bool, + description="do not set notes to the UUID or name of the VM", + default_value=False), + ConfigOption("skip_vm_platform", + bool, + description="Do not use the OpenStack flavor name as the NetBox VM platform", + default_value=False), + ConfigOption("strip_host_domain_name", + bool, + description="strip domain part from host name before syncing device to NetBox", + default_value=False), + ConfigOption("strip_vm_domain_name", + bool, + description="strip domain part from VM name before syncing VM to NetBox", + default_value=False), + ConfigOptionGroup(title="custom object attributes", + description="""\ + add arbitrary host/vm object attributes as custom fields to NetBox. + multiple attributes can be defined comma separated. + use OpenStack SDK resource attribute names; dotted paths are supported. + """, + options=[ + ConfigOption("host_custom_object_attributes", + str, + config_example="id"), + ConfigOption("vm_custom_object_attributes", + str, + config_example="id, project_id") + ]), + ConfigOption("set_vm_name_to_uuid", + bool, + description="Set the name in Netbox to the VM UUID instead of name", + default_value=False), + + # removed settings + ConfigOption("netbox_host_device_role", + str, + deprecation_message="You need to switch to 'host_role_relation'.", + removed=True), + ConfigOption("netbox_vm_device_role", + str, + deprecation_message="You need to switch to 'vm_role_relation'.", + removed=True), + ConfigOption("sync_tags", + bool, + deprecation_message="You need to switch to 'host_tag_relation', " + + "'vm_tag_relation' or 'cluster_tag_relation'", + removed=True), + ConfigOption("sync_parent_tags", + bool, + deprecation_message="You need to switch to 'host_tag_relation', " + + "'vm_tag_relation' or 'cluster_tag_relation'", + removed=True) + ] + + super().__init__() + + def validate_options(self): + + for option in self.options: + + if option.value is None: + continue + + if "filter" in option.key: + + re_compiled = None + try: + re_compiled = re.compile(option.value) + except Exception as e: + log.error(f"Problem parsing regular expression for '{self.source_name}.{option.key}': {e}") + self.set_validation_failed() + + option.set_value(re_compiled) + + continue + + if "relation" in option.key: + + relation_data = list() + + relation_type = option.key.split("_")[1] + + for relation in quoted_split(option.value): + + object_name = relation.split("=")[0].strip(' "') + if "=" not in relation: + log.error(f"Config relation '{self.source_name}.{option.key}' must contain '='.") + self.set_validation_failed() + continue + relation_name = relation.split("=", 1)[1].strip(' "') + + if len(object_name) == 0 or len(relation_name) == 0: + log.error(f"Config option '{relation}' malformed got '{object_name}' for " + f"object name and '{relation_name}' for {relation_type} name.") + self.set_validation_failed() + continue + + try: + re_compiled = re.compile(object_name) + except Exception as e: + log.error(f"Problem parsing regular expression '{object_name}' for '{relation}': {e}") + self.set_validation_failed() + continue + + relation_data.append({ + "object_regex": re_compiled, + "assigned_name": relation_name + }) + + option.set_value(relation_data) + + continue + + if "custom_object_attributes" in option.key: + + option.set_value(quoted_split(option.value)) + + continue + + if option.key == "ip_tenant_inheritance_order": + order = quoted_split(option.value) + if any(value not in ("device", "prefix", "disabled") for value in order): + log.error(f"Invalid IP tenant inheritance order: '{option.value}'") + self.set_validation_failed() + option.set_value(order) + continue + + if option.key == "set_primary_ip": + if option.value not in ["always", "when-undefined", "never"]: + log.error(f"Primary IP option '{option.key}' value '{option.value}' invalid.") + self.set_validation_failed() + + if option.key == "custom_dns_servers": + + dns_name_lookup = self.get_option_by_name("dns_name_lookup") + + if not isinstance(dns_name_lookup, ConfigOption) or dns_name_lookup.value is False: + continue + + custom_dns_servers = quoted_split(option.value) + + tested_custom_dns_servers = list() + for custom_dns_server in custom_dns_servers: + try: + tested_custom_dns_servers.append(str(ip_address(custom_dns_server))) + except ValueError: + log.error(f"Config option 'custom_dns_servers' value '{custom_dns_server}' " + f"does not appear to be an IP address.") + self.set_validation_failed() + + option.set_value(tested_custom_dns_servers) + + continue + + permitted_subnets_option = self.get_option_by_name("permitted_subnets") + + if permitted_subnets_option is not None: + permitted_subnets = PermittedSubnets(permitted_subnets_option.value) + if permitted_subnets.validation_failed is True: + self.set_validation_failed() + + permitted_subnets_option.set_value(permitted_subnets) diff --git a/module/sources/openstack/connection.py b/module/sources/openstack/connection.py new file mode 100644 index 00000000..01122ac4 --- /dev/null +++ b/module/sources/openstack/connection.py @@ -0,0 +1,1299 @@ +# -*- coding: utf-8 -*- +# Copyright (c) 2020 - 2021 Ricardo Bartels. All rights reserved. +# +# netbox-sync.py +# +# This work is licensed under the terms of the MIT license. +# For a copy, see file LICENSE.txt included in this +# repository or visit: . + +import pprint +from ipaddress import ip_interface +from importlib import import_module +from urllib.parse import unquote + +from packaging import version + +from module.sources.common.source_base import SourceBase +from module.sources.openstack.config import OpenStackConfig +from module.common.logging import get_logger, DEBUG3 +from module.common.misc import grab, dump, get_string_or_none +from module.common.support import normalize_mac_address +from module.netbox.inventory import NetBoxInventory +from module.netbox import * + +log = get_logger() + + +# noinspection PyTypeChecker +class OpenStackHandler(SourceBase): + """ + Source class to import data from a Openstack instance and add/update NetBox objects based on gathered information + """ + + dependent_netbox_objects = [ + NBTag, + NBManufacturer, + NBDeviceType, + NBPlatform, + NBClusterType, + NBClusterGroup, + NBDeviceRole, + NBSite, + NBCluster, + NBDevice, + NBVM, + NBVirtualDisk, + NBVMInterface, + NBInterface, + NBIPAddress, + NBPrefix, + NBTenant, + NBVRF, + NBVLAN, + NBVLANGroup, + NBMACAddress, + NBFHRPGroupItem, + NBCustomField + ] + + source_type = "openstack" + + # internal vars + session = None + site_name = None + + def __init__(self, name=None): + + if name is None: + raise ValueError(f"Invalid value for attribute 'name': '{name}'.") + + self.inventory = NetBoxInventory() + self.name = name + self.session = None + self.init_successful = False + + # parse settings + settings_handler = OpenStackConfig() + settings_handler.source_name = self.name + self.settings = settings_handler.parse() + + self.set_source_tag() + self.site_name = f"OpenStack: {name}" + + if self.settings.enabled is False: + log.info(f"Source '{name}' is currently disabled. Skipping") + return + + self.create_openstack_session() + + if self.session is None: + log.info(f"Source '{name}' is currently unavailable. Skipping") + return + + self.init_successful = True + self.permitted_clusters = dict() + self.cluster_host_map = dict() + self.volume_map = dict() + self.processed_host_names = dict() + self.processed_vm_names = dict() + self.object_cache = dict() + + def create_openstack_session(self): + """ + Initialize session with OpenStack + + Returns + ------- + bool: if initialization was successful or not + """ + + if self.session is not None: + return True + + log.debug(f"Starting OpenStack connection to '{self.settings.auth_url}'") + + try: + openstack = import_module("openstack") + except ImportError: + log.error("OpenStack source requires openstacksdk. Install with " + "'pip install -r requirements-openstack.txt' or 'uv sync --extra openstack'.") + return False + + try: + self.session = openstack.connect( + auth_url=self.settings.auth_url, + project_name=self.settings.project, + username=self.settings.username, + password=self.settings.password, + region_name=self.settings.region, + user_domain_name=self.settings.user_domain, + project_domain_name=self.settings.project_domain, + verify=self.settings.validate_tls_certs, + app_name='netbox-sync', + app_version='0.1', + ) + self.session.authorize() + except Exception as e: + log.error(f"Unable to connect to OpenStack instance '{self.settings.auth_url}': {e}") + self.finish() + return False + + log.info(f"Successfully connected to OpenStack '{self.settings.auth_url}'") + + return True + + def finish(self): + if self.session is not None: + self.session.close() + self.session = None + + def apply(self): + """ + Main source handler method. This method is called for each source from "main" program + to retrieve data from it source and apply it to the NetBox inventory. + + Every update of new/existing objects fot this source has to happen here. + """ + + log.info(f"Query data from OpenStack: '{self.settings.auth_url}'") + self.permitted_clusters.clear() + self.cluster_host_map.clear() + self.volume_map.clear() + self.processed_host_names.clear() + self.processed_vm_names.clear() + self.object_cache.clear() + + self.inventory.add_update_object(NBClusterGroup, data={"name": self.settings.group_name}, source=self) + + availability_zones = self.session.compute.availability_zones(details=True) + for availability_zone in availability_zones: + self.add_cluster(availability_zone) + + hypervisors = self.session.compute.hypervisors(details=True) + for hypervisor in hypervisors: + self.add_host(hypervisor) + + volumes = self.session.block_storage.volumes(details=True, all_projects=self.settings.all_projects) + for volume in volumes: + self.add_volume(volume) + + servers = self.session.compute.servers(details=True, all_projects=self.settings.all_projects) + # Prefer an already-associated server before using status to break ties. + for server in sorted(servers, key=self.get_server_priority): + self.add_virtual_machine(server) + + self.update_basic_data() + + @staticmethod + def passes_filter(name, include_filter, exclude_filter): + """ + checks if object name passes a defined object filter. + + Parameters + ---------- + name: str + name of the object to check + include_filter: regex object + regex object of include filter + exclude_filter: regex object + regex object of exclude filter + + Returns + ------- + bool: True if all filter passed, otherwise False + """ + + # first includes + if include_filter is not None and not include_filter.match(name): + log.debug(f"Object '{name}' did not match include filter '{include_filter.pattern}'. Skipping") + return False + + # second excludes + if exclude_filter is not None and exclude_filter.match(name): + log.debug(f"Object '{name}' matched exclude filter '{exclude_filter.pattern}'. Skipping") + return False + + return True + + def get_site_name(self, object_type, object_name, cluster_name=""): + """ + Return a site name for a NBCluster or NBDevice depending on config options + host_site_relation and cluster_site_relation + + Parameters + ---------- + object_type: (NBCluster, NBDevice) + object type to check site relation for + object_name: str + object name to check site relation for + cluster_name: str + cluster name of NBDevice to check for site name + + Returns + ------- + str: site name if a relation was found + """ + + if object_type not in [NBCluster, NBDevice]: + raise ValueError(f"Object must be a '{NBCluster.name}' or '{NBDevice.name}'.") + + log.debug2(f"Trying to find site name for {object_type.name} '{object_name}'") + + # check if site was provided in config + relation_name = "host_site_relation" if object_type == NBDevice else "cluster_site_relation" + + site_name = self.get_object_relation(object_name, relation_name) + + if object_type == NBDevice and site_name is None: + site_name = self.get_site_name(NBCluster, cluster_name) + if site_name is not None: + log.debug2(f"Found a matching cluster site for {object_name}, using site '{site_name}'") + + # set default site name + if site_name is None: + site_name = self.site_name + log.debug(f"No site relation for '{object_name}' found, using default site '{site_name}'") + + return site_name + + def get_object_based_on_macs(self, object_type, mac_list=None): + """ + Try to find a NetBox object based on list of MAC addresses. + + Iterate over all interfaces of this object type and compare MAC address with list of desired MAC + addresses. If match was found store related machine object and count every correct match. + + If exactly one machine with matching interfaces was found then this one will be returned. + + If two or more machines with matching MACs are found compare the two machines with + the highest amount of matching interfaces. If the ration of matching interfaces + exceeds 2.0 then the top matching machine is chosen as desired object. + + If the ration is below 2.0 then None will be returned. The probability is too low that + this one is the correct one. + + None will also be returned if no machine was found at all. + + Parameters + ---------- + object_type: (NBDevice, NBVM) + type of NetBox device to find in inventory + mac_list: list + list of MAC addresses to compare against NetBox interface objects + + Returns + ------- + (NBDevice, NBVM, None): object instance of found device, otherwise None + """ + + object_to_return = None + + if object_type not in [NBDevice, NBVM]: + raise ValueError(f"Object must be a '{NBVM.name}' or '{NBDevice.name}'.") + + if mac_list is None or not isinstance(mac_list, list) or len(mac_list) == 0: + return + + interface_typ = NBInterface if object_type == NBDevice else NBVMInterface + + objects_with_matching_macs = dict() + matching_object = None + + for interface in self.inventory.get_all_items(interface_typ): + + mac_address = grab(interface, "data.primary_mac_address.data.mac_address") or \ + grab(interface, "data.mac_address") + if mac_address in mac_list: + + matching_object = grab(interface, f"data.{interface.secondary_key}") + if not isinstance(matching_object, (NBDevice, NBVM)): + continue + + log.debug2("Found matching MAC '%s' on %s '%s'" % + (grab(interface, "data.mac_address"), object_type.name, + matching_object.get_display_name(including_second_key=True))) + + if objects_with_matching_macs.get(matching_object) is None: + objects_with_matching_macs[matching_object] = 1 + else: + objects_with_matching_macs[matching_object] += 1 + + # try to find object based on amount of matching MAC addresses + num_devices_witch_matching_macs = len(objects_with_matching_macs.keys()) + + if num_devices_witch_matching_macs == 1 and isinstance(matching_object, (NBDevice, NBVM)): + + log.debug2("Found one %s '%s' based on MAC addresses and using it" % + (object_type.name, matching_object.get_display_name(including_second_key=True))) + + object_to_return = list(objects_with_matching_macs.keys())[0] + + elif num_devices_witch_matching_macs > 1: + + log.debug2(f"Found {num_devices_witch_matching_macs} {object_type.name}s with matching MAC addresses") + + # now select the two top matches + first_choice, second_choice = \ + sorted(objects_with_matching_macs, key=objects_with_matching_macs.get, reverse=True)[0:2] + + first_choice_matches = objects_with_matching_macs.get(first_choice) + second_choice_matches = objects_with_matching_macs.get(second_choice) + + log.debug2(f"The top candidate {first_choice.get_display_name()} with {first_choice_matches} matches") + log.debug2(f"The second candidate {second_choice.get_display_name()} with {second_choice_matches} matches") + + # get ratio between + matching_ration = first_choice_matches / second_choice_matches + + # only pick the first one if the ration exceeds 2 + if matching_ration >= 2.0: + log.debug2(f"The matching ratio of {matching_ration} is high enough " + f"to select {first_choice.get_display_name()} as desired {object_type.name}") + object_to_return = first_choice + else: + log.debug2("Both candidates have a similar amount of " + "matching interface MAC addresses. Using NONE of them!") + + return object_to_return + + def get_object_based_on_primary_ip(self, object_type, primary_ip4=None, primary_ip6=None): + """ + Try to find a NBDevice or NBVM based on the primary IP address. If an exact + match was found the device/vm object will be returned immediately without + checking of the other primary IP address (if defined). + + Parameters + ---------- + object_type: (NBDevice, NBVM) + object type to look for + primary_ip4: str + primary IPv4 address of object to find + primary_ip6: str + primary IPv6 address of object to find + + Returns + ------- + + """ + + def _matches_device_primary_ip(device_primary_ip, ip_needle): + + ip = None + if device_primary_ip is not None and ip_needle is not None: + if isinstance(device_primary_ip, dict): + ip = grab(device_primary_ip, "address") + + elif isinstance(device_primary_ip, NBIPAddress): + ip = device_primary_ip.data.get("address") + elif isinstance(device_primary_ip, int): + ip = self.inventory.get_by_id(NBIPAddress, nb_id=device_primary_ip) + ip = grab(ip, "data.address") + + if ip is not None and ip.split("/")[0] == ip_needle: + return True + + return False + + if object_type not in [NBDevice, NBVM]: + raise ValueError(f"Object must be a '{NBVM.name}' or '{NBDevice.name}'.") + + if primary_ip4 is None and primary_ip6 is None: + return + + if primary_ip4 is not None: + primary_ip4 = str(primary_ip4).split("/")[0] + + if primary_ip6 is not None: + primary_ip6 = str(primary_ip6).split("/")[0] + + for device in self.inventory.get_all_items(object_type): + + if _matches_device_primary_ip(grab(device, "data.primary_ip4"), primary_ip4) is True: + log.debug2(f"Found existing host '{device.get_display_name()}' " + f"based on the primary IPv4 '{primary_ip4}'") + return device + + if _matches_device_primary_ip(grab(device, "data.primary_ip6"), primary_ip6) is True: + log.debug2(f"Found existing host '{device.get_display_name()}' " + f"based on the primary IPv6 '{primary_ip6}'") + return device + + def get_object_relation(self, name, relation, fallback=None): + """ + + Parameters + ---------- + name: str + name of the object to find a relation for + relation: str + name of the config variable relation (i.e: vm_tag_relation) + fallback: str + fallback string if no relation matched + + Returns + ------- + data: str, list, None + string of matching relation or list of matching tags + """ + + resolved_list = list() + for single_relation in grab(self.settings, relation, fallback=list()): + object_regex = single_relation.get("object_regex") + match_found = False + if object_regex.match(name): + resolved_name = single_relation.get("assigned_name") + log.debug2(f"Found a matching {relation} '{resolved_name}' ({object_regex.pattern}) for {name}") + resolved_list.append(resolved_name) + match_found = True + + # special cluster condition + if match_found is False and grab(f"{relation}".split("_"), "0") == "cluster": + + stripped_name = "/".join(name.split("/")[1:]) + if object_regex.match(stripped_name): + + resolved_name = single_relation.get("assigned_name") + log.debug2(f"Found a matching {relation} '{resolved_name}' ({object_regex.pattern}) " + f"for {stripped_name}") + resolved_list.append(resolved_name) + + if grab(f"{relation}".split("_"), "1") == "tag": + return resolved_list + + else: + resolved_name = fallback + if len(resolved_list) >= 1: + resolved_name = resolved_list[0] + if len(resolved_list) > 1: + log.debug(f"Found {len(resolved_list)} matches for {name} in {relation}." + f" Using first on: {resolved_name}") + + return resolved_name + + def get_cluster_for_host(self, hostname): + for cluster, hosts in self.cluster_host_map.items(): + if hostname in hosts: + return cluster + return None + + def get_server_priority(self, server): + name = get_string_or_none(grab(server, "name")) + if name is not None and self.settings.strip_vm_domain_name: + name = name.split(".")[0] + if self.settings.set_vm_name_to_uuid: + name = get_string_or_none(grab(server, "id")) + + cluster_name = get_string_or_none(grab(server, "availability_zone")) + cluster = self.get_object_from_cache(NBCluster, cluster_name) + if name is not None and cluster is not None and not self.settings.skip_vm_comments: + existing_vm = self.inventory.get_by_data( + NBVM, data={"name": name, "cluster": cluster} + ) + if grab(existing_vm, "data.comments") == get_string_or_none(grab(server, "id")): + return 0, server.status != "ACTIVE" + + return 1, server.status != "ACTIVE" + + @staticmethod + def get_compute_host_names(hosts): + compute_host_names = [] + for hostname, services in (hosts or {}).items(): + if not isinstance(services, dict) or not services: + compute_host_names.append(hostname) + continue + + service_names = {str(name).lower() for name in services} + service_names.update( + str(grab(service, "binary", fallback="")).lower() + for service in services.values() + ) + has_nova_services = any(name.startswith("nova-") for name in service_names) + if not has_nova_services or "nova-compute" in service_names: + compute_host_names.append(hostname) + + return compute_host_names + + def object_synced_by_other_source(self, nb_object): + if nb_object.source is not None: + return nb_object.source is not self + other_tags = [source.source_tag for source in self.inventory.source_list if source is not self] + return bool(set(nb_object.get_tags()).intersection(other_tags)) + + def add_custom_attributes(self, object_type, obj, data): + attributes = self.settings.host_custom_object_attributes if object_type == NBDevice else \ + self.settings.vm_custom_object_attributes + for attribute in attributes or []: + value = grab(obj, attribute) + if value is None: + continue + field = self.add_update_custom_field({ + "name": f"openstack_{attribute}", "label": attribute[:50], + "object_types": [object_type.object_type], "type": "text", + "description": f"OpenStack resource attribute '{attribute}'", + }) + data.setdefault("custom_fields", {})[field.data["name"]] = str(value) + + def add_device_vm_to_inventory(self, object_type, object_data, pnic_data=None, vnic_data=None, + nic_ips=None, p_ipv4=None, p_ipv6=None): + """ + Add/update device/VM object in inventory based on gathered data. + + Try to find object first based on the object data, interface MAC addresses and primary IPs. + 1. try to find by name and cluster/site + 2. try to find by mac addresses interfaces + 3. try to find by primary IP + + IP addresses for each interface are added here as well. First they will be checked and added + if all checks pass. For each IP address a matching IP prefix will be searched for. First we + look for longest matching IP Prefix in the same site. If this failed we try to find the longest + matching global IP Prefix. + + If a IP Prefix was found then we try to get the VRF and VLAN for this prefix. Now we compare + if interface VLAN and prefix VLAN match up and warn if they don't. Then we try to add data to + the IP address if not already set: + + add prefix VRF if VRF for this IP is undefined + add tenant if tenant for this IP is undefined + 1. try prefix tenant + 2. if prefix tenant is undefined try VLAN tenant + + And we also set primary IP4/6 for this object depending on the "set_primary_ip" setting. + + If a IP address is set as primary IP for another device then using this IP on another + device will be rejected by NetBox. + + Setting "always": + check all NBDevice and NBVM objects if this IP address is set as primary IP to any + other object then this one. If we found another object, then we unset the primary_ip* + for the found object and assign it to this object. + + This setting will also reset the primary IP if it has been changed in NetBox + + Setting "when-undefined": + Will set the primary IP for this object if primary_ip4/6 is undefined. Will cause a + NetBox error if IP has been assigned to a different object as well + + Setting "never": + Well, the attribute primary_ip4/6 will never be touched/changed. + + Parameters + ---------- + object_type: (NBDevice, NBVM) + NetBoxObject sub class of object to add + object_data: dict + data of object to add/update + pnic_data: dict + data of physical interfaces of this object, interface name as key + vnic_data: dict + data of virtual interfaces of this object, interface name as key + nic_ips: dict + dict of ips per interface of this object, interface name as key + p_ipv4: str + primary IPv4 as string including netmask/prefix + p_ipv6: str + primary IPv6 as string including netmask/prefix + + """ + + if object_type not in [NBDevice, NBVM]: + raise ValueError(f"Object must be a '{NBVM.name}' or '{NBDevice.name}'.") + + pnic_data = pnic_data or {} + vnic_data = vnic_data or {} + nic_ips = nic_ips or {} + + if log.level == DEBUG3: + + log.debug3("function: add_device_vm_to_inventory") + log.debug3(f"Object type {object_type}") + pprint.pprint(object_data) + pprint.pprint(pnic_data) + pprint.pprint(vnic_data) + pprint.pprint(nic_ips) + pprint.pprint(p_ipv4) + pprint.pprint(p_ipv6) + + # check existing Devices for matches + log.debug2(f"Trying to find a {object_type.name} based on the collected name, cluster, IP and MAC addresses") + + device_vm_object = self.inventory.get_by_data(object_type, data=object_data) + + if device_vm_object is not None: + log.debug2("Found a exact matching %s object: %s" % + (object_type.name, device_vm_object.get_display_name(including_second_key=True))) + + # keep searching if no exact match was found + else: + + log.debug2(f"No exact match found. Trying to find {object_type.name} based on MAC addresses") + + # on VMs vnic data is used, on physical devices pnic data is used + mac_source_data = vnic_data if object_type == NBVM else pnic_data + + nic_macs = [x.get("mac_address") for x in mac_source_data.values() if x.get("mac_address") is not None] + + device_vm_object = self.get_object_based_on_macs(object_type, nic_macs) + + if device_vm_object is not None: + log.debug2("Found a matching %s object: %s" % + (object_type.name, device_vm_object.get_display_name(including_second_key=True))) + + # keep looking for devices with the same primary IP + else: + + log.debug2(f"No match found. Trying to find {object_type.name} based on primary IP addresses") + + device_vm_object = self.get_object_based_on_primary_ip(object_type, p_ipv4, p_ipv6) + + if device_vm_object is None: + object_name = object_data.get(object_type.primary_key) + log.debug(f"No existing {object_type.name} object for {object_name}. Creating a new {object_type.name}.") + device_vm_object = self.inventory.add_object(object_type, data=object_data, source=self) + else: + device_vm_object.update(data=object_data, source=self) + + # update role according to config settings + object_name = object_data.get(object_type.primary_key) + role_name = self.get_object_relation(object_name, + "host_role_relation" if object_type == NBDevice else "vm_role_relation") + + if object_type == NBDevice: + if role_name is None: + role_name = "Server" + device_vm_object.update(data={"device_role": {"name": role_name}}) + if object_type == NBVM and role_name is not None: + device_vm_object.update(data={"role": {"name": role_name}}) + + # compile all nic data into one dictionary + if object_type == NBVM: + nic_data = vnic_data + else: + nic_data = {**pnic_data, **vnic_data} + exclude_filter = self.settings.vm_interface_exclude_filter if object_type == NBVM else \ + self.settings.host_interface_exclude_filter + if exclude_filter is not None: + nic_data = {name: data for name, data in nic_data.items() if not exclude_filter.match(name)} + + # map interfaces of existing object with discovered interfaces + nic_object_dict = self.map_object_interfaces_to_current_interfaces(device_vm_object, nic_data) + + if object_data.get("status", "") == "active" and (nic_ips is None or len(nic_ips.keys()) == 0): + log.debug(f"No IP addresses for '{object_name}' found!") + + primary_ipv4_object = None + primary_ipv6_object = None + + if p_ipv4 is not None: + try: + primary_ipv4_object = ip_interface(p_ipv4) + except ValueError: + log.error(f"Primary IPv4 ({p_ipv4}) does not appear to be a valid IP address (needs included suffix).") + + if p_ipv6 is not None: + try: + primary_ipv6_object = ip_interface(p_ipv6) + except ValueError: + log.error(f"Primary IPv6 ({p_ipv6}) does not appear to be a valid IP address (needs included suffix).") + + for int_name, int_data in nic_data.items(): + existing_interface = nic_object_dict.get(int_name) + if existing_interface is not None: + existing_name = grab(existing_interface, "data.name") + if existing_name is not None: + int_data["name"] = existing_name + if object_type == NBDevice: + int_data.pop("type", None) + + # add/update interface with retrieved data + nic_object, ip_address_objects = self.add_update_interface(nic_object_dict.get(int_name), device_vm_object, + int_data, nic_ips.get(int_name, list())) + + # add all interface IPs + for ip_object in ip_address_objects: + + if ip_object is None: + continue + + ip_interface_object = ip_interface(grab(ip_object, "data.address")) + + # continue if address is not a primary IP + if ip_interface_object not in [primary_ipv4_object, primary_ipv6_object]: + continue + + # set/update/remove primary IP addresses + set_this_primary_ip = False + ip_version = ip_interface_object.ip.version + if self.settings.set_primary_ip == "always": + + for object_type in [NBDevice, NBVM]: + + # new IPs don't need to be removed from other devices/VMs + if ip_object.is_new is True: + break + + for devices_vms in self.inventory.get_all_items(object_type): + + # device has no primary IP of this version + this_primary_ip = grab(devices_vms, f"data.primary_ip{ip_version}") + + # we found this exact object + if devices_vms == device_vm_object: + continue + + # device has the same object assigned + if this_primary_ip == ip_object: + devices_vms.unset_attribute(f"primary_ip{ip_version}") + + set_this_primary_ip = True + + elif self.settings.set_primary_ip != "never" and \ + grab(device_vm_object, f"data.primary_ip{ip_version}") is None: + set_this_primary_ip = True + + if set_this_primary_ip is True: + + log.debug(f"Setting IP '{grab(ip_object, 'data.address')}' as primary IPv{ip_version} for " + f"'{device_vm_object.get_display_name()}'") + device_vm_object.update(data={f"primary_ip{ip_version}": ip_object}) + + return device_vm_object + + def add_object_to_cache(self, obj_type, key, netbox_object): + + if obj_type is None or key is None or netbox_object is None: + return + + # noinspection PyBroadException + try: + vm_class_name = obj_type.__name__ + except Exception: + return + + if self.object_cache.get(vm_class_name) is None: + self.object_cache[vm_class_name] = dict() + + self.object_cache[vm_class_name][key] = netbox_object + + def get_object_from_cache(self, obj_type, key): + + if obj_type is None or key is None: + return + + # noinspection PyBroadException + try: + vm_class_name = obj_type.__name__ + except Exception: + return + + if self.object_cache.get(vm_class_name) is None: + return + + return self.object_cache[vm_class_name].get(key) + + def add_cluster(self, obj): + """ + Add a OpenStack Availability Zone as a NBCluster to NetBox. Cluster name is checked against + cluster_include_filter and cluster_exclude_filter config setting. Also adds + cluster and site_name to "self.permitted_clusters" so hosts and VMs can be + checked if they are part of a permitted cluster. + + Parameters + ---------- + obj: openstack.compute.v2.availability_zone.AvailabilityZone + cluster to add + """ + + name = get_string_or_none(grab(obj, "name")) + group = self.inventory.get_by_data(NBClusterGroup, data={"name": self.settings.group_name}) + + if name is None or group is None: + return + + group_name = grab(group, "data.name") + + log.debug(f"Parsing OpenStack AZ: {name}") + + if self.passes_filter(name, self.settings.cluster_include_filter, + self.settings.cluster_exclude_filter) is False: + return + + site_name = self.get_site_name(NBCluster, name) + + data = { + "name": name, + "type": {"name": "Openstack"}, + "group": group, + "site": {"name": site_name} + } + if version.parse(self.inventory.netbox_api_version) >= version.parse("4.2.0"): + data.pop("site") + data["scope_type"] = NBSite.object_type + data["scope_id"] = self.inventory.add_update_object(NBSite, data={"name": site_name}, source=self) + tags = self.get_object_relation(name, "cluster_tag_relation") + if tags: + data["tags"] = tags + + tenant_name = self.get_object_relation(name, "cluster_tenant_relation") + if tenant_name is not None: + data["tenant"] = {"name": tenant_name} + + # try to find cluster including cluster group + log.debug2("Trying to find a matching existing cluster") + cluster_object = None + for cluster_candidate in self.inventory.get_all_items(NBCluster): + if grab(cluster_candidate, "data.name") != name: + continue + if cluster_candidate.get_site_name() not in (None, site_name) and \ + self.object_synced_by_other_source(cluster_candidate): + continue + + # try to find a cluster with matching site + candidate_group = grab(cluster_candidate, "data.group.data.name") + if candidate_group not in (None, group_name): + continue + + if cluster_candidate.get_site_name() == site_name: + cluster_object = cluster_candidate + log.debug2("Found an existing cluster where 'name' and 'site' are matching") + break + + if grab(cluster_candidate, "data.group") is not None and \ + grab(cluster_candidate, "data.group.data.name") == group_name: + cluster_object = cluster_candidate + log.debug2("Found an existing cluster where 'name' and 'cluster group' are matching") + break + + if grab(cluster_candidate, "data.tenant") is not None and \ + tenant_name is not None and \ + grab(cluster_candidate, "data.tenant.data.name") == tenant_name: + cluster_object = cluster_candidate + log.debug2("Found an existing cluster where 'name' and 'tenant' are matching") + break + + if cluster_object is not None: + cluster_object.update(data=data, source=self) + else: + cluster_object = self.inventory.add_update_object(NBCluster, data=data, source=self) + + self.add_object_to_cache(NBCluster, name, cluster_object) + + self.cluster_host_map[name] = self.get_compute_host_names(grab(obj, "hosts")) + log.debug2(f"Found compute hosts for availability zone '{name}': " + f"{self.cluster_host_map[name]}") + + self.permitted_clusters[name] = site_name + + def add_host(self, obj): + """ + Parse a Openstack host to NetBox once all data is gathered. + + First host is filtered: + host has a cluster and is it permitted + was host with same name and site already parsed + does the host pass the host_include_filter and host_exclude_filter + + Then all necessary host data will be collected. + + Primary IPv4/6 will be determined by 'host_ip' value + + Parameters + ---------- + obj: Hypervisor + host object to parse + """ + + name = get_string_or_none(grab(obj, "name")) + if name is None: + return + + if name is not None and self.settings.strip_host_domain_name is True: + name = name.split(".")[0] + + # parse data + log.debug(f"Parsing Openstack host: {name}") + + # + # Filtering + # + + # manage site and cluster + short_name = get_string_or_none(obj.service_details["host"]) + cluster_name = self.get_cluster_for_host(short_name) + + if cluster_name is None: + # Nova omits disabled compute services from availability zone host lists + if get_string_or_none(grab(obj, "status")) == "disabled": + log.info(f"Host '{name}' is disabled in Nova and not part of any availability zone. Skipping.") + else: + log.error(f"Unable to find an availability zone for host '{name}'. Skipping.") + return + + if log.level == DEBUG3: + try: + log.info("Cluster data") + dump(obj.service_details.to_dict()) + except Exception as e: + log.error(e) + + if self.permitted_clusters.get(cluster_name) is None: + log.debug(f"Host '{name}' is not part of a permitted cluster. Skipping") + return + + # get cluster object + nb_cluster_object = self.get_object_from_cache(NBCluster, cluster_name) + + # get a site for this host + site_name = self.get_site_name(NBDevice, name, cluster_name) + + if name in self.processed_host_names.get(site_name, list()): + log.warning(f"Host '{name}' for site '{site_name}' already parsed. " + "Make sure to use unique host names. Skipping") + return + + # add host to processed list + if self.processed_host_names.get(site_name) is None: + self.processed_host_names[site_name] = list() + + self.processed_host_names[site_name].append(name) + + # filter hosts by name + if self.passes_filter(name, self.settings.host_include_filter, self.settings.host_exclude_filter) is False: + return + + # + # Collecting data + # + + # collect all necessary data + manufacturer = None + model = None + platform = get_string_or_none(obj.hypervisor_type) + + # if the device vendor/model cannot be retrieved (due to problem on the host), + # set a dummy value so the host still gets synced + if manufacturer is None: + manufacturer = "Generic Vendor" + if model is None: + model = "Generic Model" + + # get status + status = "offline" + if get_string_or_none(obj.status) == "enabled" and grab(obj, "state", fallback="up") == "up": + status = "active" + + # get host_tenant_relation + tenant_name = self.get_object_relation(name, "host_tenant_relation") + + # get host_tag_relation + host_tags = self.get_object_relation(name, "host_tag_relation") + + # prepare host data model + host_data = { + "name": name, + "device_type": { + "model": model, + "manufacturer": { + "name": manufacturer + } + }, + "site": {"name": site_name}, + "cluster": nb_cluster_object, + "status": status + } + + # add data if present + if platform is not None: + host_data["platform"] = {"name": platform} + if tenant_name is not None: + host_data["tenant"] = {"name": tenant_name} + if len(host_tags) > 0: + host_data["tags"] = host_tags + self.add_custom_attributes(NBDevice, obj, host_data) + + pnic_data_dict = dict() + pnic_data = { + "name": "eth0", + "enabled": True, + "type": "virtual" + } + pnic_data_dict["eth0"] = pnic_data + + ip_addr = grab(obj, "host_ip") + if ip_addr is not None and not self.settings.permitted_subnets.permitted(ip_addr, interface_name="eth0"): + ip_addr = None + prefix = None + matched_prefix = None + if ip_addr is not None: + matched_prefix = self.return_longest_matching_prefix_for_ip(ip_interface(ip_addr), site_name) + if matched_prefix is None: + matched_prefix = self.return_longest_matching_prefix_for_ip(ip_interface(ip_addr)) + if matched_prefix is not None: + prefix = matched_prefix.data["prefix"].prefixlen + ip_addr = f"{ip_addr}/{prefix}" + + vnic_ips = dict() + vnic_ips["eth0"] = list() + if ip_addr is not None: + ip_addr = str(ip_interface(ip_addr)) + vnic_ips["eth0"].append(ip_addr) + + host_primary_ip4 = ip_addr if ip_addr is not None and ip_interface(ip_addr).version == 4 else None + host_primary_ip6 = ip_addr if ip_addr is not None and ip_interface(ip_addr).version == 6 else None + + # add host to inventory + host = self.add_device_vm_to_inventory(NBDevice, object_data=host_data, pnic_data=pnic_data_dict, + vnic_data=dict(), nic_ips=vnic_ips, + p_ipv4=host_primary_ip4, p_ipv6=host_primary_ip6) + self.add_object_to_cache(NBDevice, short_name, host) + + return + + def add_volume(self, obj): + """ + Parse OpenStack volume and store in in a map. + """ + + id = obj.id + size = obj.size + + self.volume_map[id] = size + + def add_virtual_machine(self, obj): + """ + Parse a OpenStack VM add to NetBox once all data is gathered. + + Parameters + ---------- + obj: openstack.compute.v2.server.Server + virtual machine object to parse + """ + + name = get_string_or_none(grab(obj, "name")) + if name is None: + return + + if name is not None and self.settings.strip_vm_domain_name is True: + name = name.split(".")[0] + + if self.settings.set_vm_name_to_uuid: + name = get_string_or_none(obj.id) + + log.debug(f"Parsing OpenStack VM: {name}") + + # get VM power state + status = "active" if get_string_or_none(obj.status) == "ACTIVE" else "offline" + + cluster_name = get_string_or_none(obj.availability_zone) + + # check VM cluster + if cluster_name is None: + log.error(f"Requesting cluster for Virtual Machine '{name}' failed. Skipping.") + return + + elif self.permitted_clusters.get(cluster_name) is None: + log.debug(f"Virtual machine '{name}' is not part of a permitted cluster. Skipping") + return + + nb_cluster_object = self.get_object_from_cache(NBCluster, cluster_name) + + if name in self.processed_vm_names.get(cluster_name, list()): + log.warning(f"Virtual machine '{name}' for cluster '{cluster_name}' already parsed. " + "Make sure to use unique VM names. Skipping") + return + + # add host to processed list + if self.processed_vm_names.get(cluster_name) is None: + self.processed_vm_names[cluster_name] = list() + + self.processed_vm_names[cluster_name].append(name) + + # filter VMs by name + if self.passes_filter(name, self.settings.vm_include_filter, self.settings.vm_exclude_filter) is False: + return + + # + # Collect data + # + + site_name = self.get_site_name(NBCluster, cluster_name) + flavor = dict(grab(obj, "flavor") or {}) + if not all(key in flavor for key in ("vcpus", "ram", "disk")) and flavor.get("id"): + flavor = self.session.compute.get_flavor(flavor["id"]).to_dict() + # first check against vm_platform_relation + platform = None + if self.settings.skip_vm_platform is False: + platform = get_string_or_none(flavor.get("original_name") or flavor.get("name")) + + if platform is not None: + platform = self.get_object_relation(platform, "vm_platform_relation", fallback=platform) + + disks = {} + if grab(obj, "image") and flavor.get("disk"): + disks["root"] = int(flavor["disk"]) + if flavor.get("ephemeral"): + disks["ephemeral"] = int(flavor["ephemeral"]) + disk_complete = True + for volume in grab(obj, "attached_volumes", fallback=[]) or []: + size = self.volume_map.get(volume["id"]) + if size is None: + log.warning(f"Volume '{volume['id']}' of VM '{name}' was not discovered; preserving disk size.") + disk_complete = False + disks[volume["id"]] = int(size) if size is not None else None + + annotation = None + if self.settings.skip_vm_comments is False: + if not self.settings.set_vm_name_to_uuid: + annotation = get_string_or_none(obj.id) + + # assign vm_tenant_relation + tenant_name = self.get_object_relation(name, "vm_tenant_relation") + + vm_data = { + "name": name, + "cluster": nb_cluster_object, + "status": status, + } + if flavor.get("vcpus"): + vm_data["vcpus"] = float(flavor["vcpus"]) + if flavor.get("ram") is not None: + vm_data["memory"] = int(flavor["ram"]) + if disk_complete and version.parse(self.inventory.netbox_api_version) < version.parse("3.7.0"): + vm_data["disk"] = sum(disks.values()) + tags = self.get_object_relation(name, "vm_tag_relation") + if tags: + vm_data["tags"] = tags + host = self.get_object_from_cache(NBDevice, grab(obj, "compute_host")) + if host is not None and version.parse(self.inventory.netbox_api_version) >= version.parse("3.3.0"): + vm_data["device"] = host + + # Add adaption for change in NetBox 3.3.0 VM model + # issue: https://github.com/netbox-community/netbox/issues/10131#issuecomment-1225783758 + if version.parse(self.inventory.netbox_api_version) >= version.parse("3.3.0"): + vm_data["site"] = {"name": site_name} + if platform is not None: + vm_data["platform"] = {"name": platform} + if annotation is not None: + vm_data["comments"] = annotation + if tenant_name is not None: + vm_data["tenant"] = {"name": tenant_name} + + if self.settings.set_vm_name_to_uuid: + custom_field = self.add_update_custom_field({ + "name": "openstack_vm_name", + "label": "name", + "object_types": ["virtualization.virtualmachine"], + "type": "text", + "description": f"Openstack '{self.name}' synced object attribute 'name'" + }) + vm_data["custom_fields"] = { + grab(custom_field, "data.name"): get_string_or_none(grab(obj, "name")) + } + self.add_custom_attributes(NBVM, obj, vm_data) + + vm_primary_ip4 = None + vm_primary_ip6 = None + vm_primary_ip4_is_fixed = False + vm_primary_ip6_is_fixed = False + vm_nic_dict = dict() + nic_ips = dict() + count = 0 + + for network, addresses in (grab(obj, "addresses") or {}).items(): + default_mac = next((address.get("OS-EXT-IPS-MAC:mac_addr") for address in addresses + if address.get("OS-EXT-IPS-MAC:mac_addr")), None) + interfaces = {} + for address in addresses: + mac = address.get("OS-EXT-IPS-MAC:mac_addr") or default_mac + if mac not in interfaces: + count += 1 + full_name = unquote(f"vNIC{count} ({network})") + interfaces[mac] = full_name + vm_nic_dict[full_name] = { + "name": full_name, + "description": full_name, + "enabled": True, + } + if mac: + vm_nic_dict[full_name]["mac_address"] = normalize_mac_address(mac) + nic_ips[full_name] = [] + full_name = interfaces[mac] + try: + ip_addr = str(ip_interface(address["addr"])) + except (KeyError, ValueError): + log.warning(f"Invalid address on OpenStack VM '{name}'; skipping.") + continue + if not self.settings.permitted_subnets.permitted(ip_addr, interface_name=full_name): + continue + + matched_prefix = self.return_longest_matching_prefix_for_ip(ip_interface(ip_addr), site_name) + if matched_prefix is None: + matched_prefix = self.return_longest_matching_prefix_for_ip(ip_interface(ip_addr)) + if matched_prefix is not None: + prefix = matched_prefix.data["prefix"].prefixlen + ip_addr = f"{ip_interface(ip_addr).ip}/{prefix}" + + nic_ips[full_name].append(ip_addr) + is_fixed = address.get("OS-EXT-IPS:type") == "fixed" + if ip_interface(ip_addr).version == 4 and ( + vm_primary_ip4 is None or (is_fixed and not vm_primary_ip4_is_fixed)): + vm_primary_ip4 = ip_addr + vm_primary_ip4_is_fixed = is_fixed + if ip_interface(ip_addr).version == 6 and ( + vm_primary_ip6 is None or (is_fixed and not vm_primary_ip6_is_fixed)): + vm_primary_ip6 = ip_addr + vm_primary_ip6_is_fixed = is_fixed + + # add VM to inventory + vm = self.add_device_vm_to_inventory(NBVM, object_data=vm_data, vnic_data=vm_nic_dict, + nic_ips=nic_ips, p_ipv4=vm_primary_ip4, p_ipv6=vm_primary_ip6) + if version.parse(self.inventory.netbox_api_version) >= version.parse("3.7.0"): + for disk_name, disk_size in disks.items(): + data = {"name": disk_name, "virtual_machine": vm} + if disk_size is None: + existing = self.inventory.get_by_data(NBVirtualDisk, data=data) + if existing is not None: + existing.update(data=data, source=self) + continue + multiplier = 1000 if version.parse(self.inventory.netbox_api_version) >= version.parse("4.1.0") else 1 + data["size"] = disk_size * multiplier + self.inventory.add_update_object(NBVirtualDisk, data=data, source=self) + + return + + def update_basic_data(self): + """ + + Returns + ------- + + """ + + # add source identification tag + self.inventory.add_update_object(NBTag, data={ + "name": self.source_tag, + "description": f"Marks objects synced from Openstack '{self.name}' " + f"({self.settings.auth_url}) to this NetBox Instance." + }) + + # update virtual site if present + this_site_object = self.inventory.get_by_data(NBSite, data={"name": self.site_name}) + + if this_site_object is not None: + this_site_object.update(data={ + "name": self.site_name, + "comments": "A default virtual site created to house objects " + "that have been synced from this Openstack instance " + "and have no predefined site assigned." + }) + + server_role_object = self.inventory.get_by_data(NBDeviceRole, data={"name": "Server"}) + + if server_role_object is not None: + role_data = {"name": "Server", "vm_role": True} + if server_role_object.is_new is True: + role_data["color"] = "9e9e9e" + + server_role_object.update(data=role_data) + +# EOF diff --git a/pyproject.toml b/pyproject.toml index 0cf28b08..be1a3c97 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,6 +33,9 @@ dependencies = [ aws = [ "boto3>=1.40,<2", ] +openstack = [ + "openstacksdk>=4.0,<5", +] test = [ "pytest>=9.1.1", ] diff --git a/requirements-openstack.txt b/requirements-openstack.txt new file mode 100644 index 00000000..99abdacb --- /dev/null +++ b/requirements-openstack.txt @@ -0,0 +1,2 @@ +-r requirements.txt +openstacksdk>=4.0,<5 diff --git a/settings-example.ini b/settings-example.ini index bf8ab5af..6b338dab 100644 --- a/settings-example.ini +++ b/settings-example.ini @@ -748,4 +748,196 @@ regions = auto ;organization_account_ids = 111122223333 ; Leave organization_external_id unset unless the member trust policy requires it. +[source/my-openstack-example] + +; Defines if this source is enabled or not +;enabled = True + +; type of source. This defines which source handler to use +type = openstack + +; OpenStack Identity v3 authentication URL +auth_url = https://api.openstack-instance.local:5000/v3 + +; Name of the OpenStack project +project = MyProject + +; username to use to log into OpenStack +username = openstackuser + +; password to use to log into OpenStack +password = supersecret + +; Which OpenStack region to use +region = RegionOne + +; Which OpenStack user domain to use +user_domain = Default + +; Which OpenStack project domain to use +project_domain = Default + +; OpenStack group name +;group_name = Openstack + +; Enforces TLS certificate validation. Disable only for an OpenStack endpoint with a self- +; signed certificate. +;validate_tls_certs = True + +; Discover servers and volumes across projects (requires administrative access). +;all_projects = True + +; IP networks eligible to be synced to NetBox. If an IP address is not part of this +; networks then it WON'T be synced to NetBox. To excluded small blocks from bigger IP +; blocks a leading '!' has to be added +;permitted_subnets = 172.16.0.0/12, 10.0.0.0/8, 192.168.0.0/16, fd00::/8, !10.23.42.0/24 + +; define in which order the IP address tenant will be assigned if tenant is undefined. +; possible values: +; * device : host or VM tenant will be assigned to the IP address +; * prefix : if the IP address belongs to an existing prefix and this prefix has a tenant assigned, then this one is used +; * disabled : no tenant assignment to the IP address will be performed +; the order of the definition is important, the default is "device, prefix" which means: +; If the device has a tenant then this one will be used. If not, the prefix tenant will be used if defined +;ip_tenant_inheritance_order = device, prefix + +; Keep existing primary IP assignments when an address is no longer discovered. +;preserve_primary_ips = False + +; Do not assign addresses belonging to an existing FHRP group. +;skip_fhrp_group_ips = True + +; Regular expression matching host interface names to ignore. +;host_interface_exclude_filter = + +; Regular expression matching VM interface names to ignore. +;vm_interface_exclude_filter = + +; Do not synchronize VLANs discovered through existing prefixes. +;disable_vlan_sync = False + +; filter options + +; filters can be used to include/exclude certain objects from importing into NetBox. +; Include filters are checked first and exclude filters after. An object name has to pass +; both filters to be synced to NetBox. If a filter is unset it will be ignored. Filters +; are all treated as regex expressions! If more then one expression should match, a '|' +; needs to be used +; +; Example: (exclude all VMs with "replica" in their name and all VMs starting with +; "backup"): vm_exclude_filter = .*replica.*|^backup.* + +; If a cluster is excluded from sync then ALL VMs and HOSTS inside the availability zone +; will be ignored! +;cluster_exclude_filter = +;cluster_include_filter = + +; This will only include/exclude the host, not the VM if Host is part of a multi host +; cluster +;host_exclude_filter = +;host_include_filter = + +; simply include/exclude VMs +;vm_exclude_filter = +;vm_include_filter = + +; relations options + +; This option defines which OpenStack availability zone is part of a NetBox site. +; This is done with a comma separated key = value list. +; key: defines the cluster name as regex +; value: defines the NetBox site name (use quotes if name contains commas) +; This is a quite important config setting as IP addresses, prefixes, VLANs +; and VRFs are site dependent. In order to assign the correct prefix to an IP +; address it is important to pick the correct site. +; A VM always depends on the cluster site relation +;cluster_site_relation = nova = Cloud Site + +; Same as cluster site but on host level. If unset it will fall back to +; cluster_site_relation +;host_site_relation = nyc02.* = New York, ffm01.* = Frankfurt + +; This option defines which cluster/host/VM belongs to which tenant. +; This is done with a comma separated key = value list. +; key: defines a hosts/VM name as regex +; value: defines the NetBox tenant name (use quotes if name contains commas) +; a cluster can be specified as "Cluster-name" or +; "availability-zone-name" +;cluster_tenant_relation = Cluster_NYC.* = Customer A +;host_tenant_relation = compute.* = Infrastructure +;vm_tenant_relation = grafana.* = Infrastructure + +; This option defines custom platforms if the VMWare created platforms are not suitable. +; Pretty much a mapping of VMWare platform name to your own platform name. +; This is done with a comma separated key = value list. +; key: defines a VMWare returned platform name +; value: defines the desired NetBox platform name +;vm_platform_relation = centos-7.* = centos7, microsoft-windows-server-2016.* = Windows2016 + +; Define the NetBox device role used for hosts. The default is +; set to "Server". This is done with a comma separated key = value list. +; key: defines host(s) name as regex +; value: defines the NetBox role name (use quotes if name contains commas) +;host_role_relation = .* = Server + +; Define the NetBox device role used for VMs. This is done with a +; comma separated key = value list, same as 'host_role_relation'. +; key: defines VM(s) name as regex +; value: defines the NetBox role name (use quotes if name contains commas) +;vm_role_relation = .* = Server + +; Define NetBox tags which are assigned to a cluster, host or VM. This is +; done with a comma separated key = value list. +; key: defines a hosts/VM name as regex +; value: defines the NetBox tag (use quotes if name contains commas) +; Match the availability zone name to assign cluster tags. +;cluster_tag_relation = nova = Infrastructure +;host_tag_relation = compute.* = Infrastructure +;vm_tag_relation = grafana.* = Infrastructure + +; Perform a reverse lookup for all collected IP addresses. If a dns name was found it will +; be added to the IP address object in NetBox +;dns_name_lookup = True + +; use custom DNS server to do the reverse lookups +;custom_dns_servers = 192.168.1.11, 192.168.1.12 + +; define how the primary IPs should be set +; possible values: +; +; always: will remove primary IP from the object where this address is +; currently set as primary and moves it to new object +; +; when-undefined: +; only sets primary IP if undefined, will cause ERRORs if same IP is +; assigned more then once to different hosts and IP is set as the +; objects primary IP +; +; never: don't set any primary IPs, will cause the same ERRORs +; as "when-undefined" +;set_primary_ip = when-undefined + +; do not set notes to the UUID or name of the VM +;skip_vm_comments = False + +; Do not use the OpenStack flavor name as the NetBox VM platform +;skip_vm_platform = False + +; strip domain part from host name before syncing device to NetBox +;strip_host_domain_name = False + +; strip domain part from VM name before syncing VM to NetBox +;strip_vm_domain_name = False + +; custom object attributes options + +; add arbitrary host/vm object attributes as custom fields to NetBox. +; multiple attributes can be defined comma separated. +; use OpenStack SDK resource attribute names; dotted paths are supported. +;host_custom_object_attributes = id +;vm_custom_object_attributes = id, project_id + +; Set the name in Netbox to the VM UUID instead of name +;set_vm_name_to_uuid = False + ;EOF diff --git a/tests/test_openstack_source.py b/tests/test_openstack_source.py new file mode 100644 index 00000000..2a80bfc8 --- /dev/null +++ b/tests/test_openstack_source.py @@ -0,0 +1,454 @@ +"""OpenStack discovery against fake SDK resources and the real NetBox models.""" + +import sys +from types import ModuleType, SimpleNamespace +from unittest.mock import Mock + +import pytest + +from module.config.file_output import ConfigFileOutput +from module.config.files import ConfigFileINI, ConfigFileYAML +from module.config.parser import ConfigParser +from module.netbox import ( + NBCluster, NBClusterGroup, NBCustomField, NBDevice, NBInterface, NBIPAddress, NBMACAddress, + NBPrefix, NBSite, NBTenant, NBVirtualDisk, NBVM, NBVMInterface, NBVRF, +) +from module.sources import instantiate_sources, valid_sources +from module.sources.openstack.config import OpenStackConfig +from module.sources.openstack.connection import OpenStackHandler + + +def resource(**values): + obj = SimpleNamespace(**values) + obj.to_dict = lambda: values.copy() + return obj + + +def server(**overrides): + values = dict( + name="app.example.org", id="vm-uuid", status="ACTIVE", availability_zone="zone.example", + flavor={"original_name": "m1.small", "vcpus": 2, "ram": 2048, "disk": 10}, + image={"id": "image"}, attached_volumes=[{"id": "volume-1"}], compute_host="compute1", + addresses={"private": [ + {"addr": "10.0.0.8", "version": 4, "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:00:00:01", + "OS-EXT-IPS:type": "fixed"}, + {"addr": "fd00::8", "version": 6, "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:00:00:01", + "OS-EXT-IPS:type": "fixed"}, + {"addr": "192.0.2.8", "version": 4, "OS-EXT-IPS:type": "floating"}, + ]}, + ) + values.update(overrides) + return resource(**values) + + +@pytest.fixture +def cloud(monkeypatch, inventory): + settings = dict( + type="openstack", auth_url="https://identity.example/v3", project="admin", + username="sync", password="test-only", region="RegionOne", user_domain="Default", + project_domain="Default", permitted_subnets="10.0.0.0/8, fd00::/8", + dns_name_lookup=False, cluster_site_relation=".* = Cloud", vm_tenant_relation=".* = Operations", + cluster_tag_relation=".* = OpenStack", vm_tag_relation=".* = Production", + ) + parser = ConfigParser() + monkeypatch.setattr(parser, "content", {"source": {"cloud": settings}}) + monkeypatch.setattr(parser, "parsing_finished", True) + sdk = ModuleType("openstack") + session = Mock() + session.compute.availability_zones.return_value = [ + resource(name="zone.example", hosts={"compute1": {}}), + resource(name="empty", hosts=None), + ] + session.compute.hypervisors.return_value = [ + resource(name="compute1.example", service_details={"host": "compute1"}, status="enabled", + state="up", host_ip="10.0.0.2", hypervisor_type="QEMU", hypervisor_version=8000000), + ] + session.block_storage.volumes.return_value = [resource(id="volume-1", size=20)] + session.compute.servers.return_value = [server()] + sdk.connect = Mock(return_value=session) + monkeypatch.setitem(sys.modules, "openstack", sdk) + return SimpleNamespace(settings=settings, sdk=sdk, session=session, inventory=inventory) + + +@pytest.mark.parametrize("netbox_version", ["3.6.0", "3.7.0", "4.0.0", "4.1.0", "4.2.0", "4.3.0"]) +def test_discovery_uses_current_inventory_apis(cloud, netbox_version): + inventory = cloud.inventory + inventory.netbox_api_version = netbox_version + site = inventory.add_object(NBSite, data={"id": 1, "name": "Cloud"}, read_from_netbox=True) + tenant = inventory.add_object(NBTenant, data={"id": 1, "name": "Network"}, read_from_netbox=True) + vrf = inventory.add_object(NBVRF, data={"id": 1, "name": "Cloud VRF"}, read_from_netbox=True) + prefix_data = {"prefix": "10.0.0.0/24", "tenant": tenant, "vrf": vrf} + if netbox_version >= "4.2.0": + prefix_data.update(scope_type=NBSite.object_type, scope_id=site) + else: + prefix_data["site"] = site + prefix = inventory.add_object(NBPrefix, data=prefix_data, read_from_netbox=True) + # Model the resolved relations returned by the real NetBox inventory loader. + prefix.data.update(tenant=tenant, vrf=vrf) + prefix.data["scope_id" if netbox_version >= "4.2.0" else "site"] = site + inventory.add_object(NBPrefix, data={"prefix": "fd00::/64"}, read_from_netbox=True) + handler = OpenStackHandler("cloud") + assert handler.init_successful + assert cloud.sdk.connect.call_args.kwargs["verify"] is True + handler.apply() + cluster = handler.get_object_from_cache(NBCluster, "zone.example") + assert cluster.get_site_name() == "Cloud" + if netbox_version >= "4.2.0": + assert cluster.data["scope_id"] is site + assert cluster.data["scope_type"] == "dcim.site" + assert "site" not in cluster.data + host = inventory.get_all_items(NBDevice)[0] + vm = inventory.get_all_items(NBVM)[0] + assert host.data["primary_ip4"].data["address"] == "10.0.0.2/24" + assert host.data["platform"].data["name"] == "QEMU" + assert vm.data["device"] is host + assert vm.data["status"] == "active" + assert vm.data["vcpus"] == 2 + assert vm.data["memory"] == 2048 + if netbox_version < "3.7.0": + assert vm.data["disk"] == 30 + assert not vm.get_virtual_disks() + else: + assert "disk" not in vm.data + disks = {disk.data["name"]: disk.data["size"] for disk in vm.get_virtual_disks()} + multiplier = 1000 if netbox_version >= "4.1.0" else 1 + assert disks == {"root": 10 * multiplier, "volume-1": 20 * multiplier} + assert vm.data["platform"].data["name"] == "m1.small" + assert "Production" in [tag.data["name"] for tag in vm.data["tags"]] + assert "OpenStack" in [tag.data["name"] for tag in cluster.data["tags"]] + nic = inventory.get_all_items(NBVMInterface)[0] + assert len(inventory.get_all_items(NBVMInterface)) == 1 + assert nic.data["name"] == "vNIC1 (private)" + assert {ip.data["address"] for ip in nic.get_ip_addresses()} == {"10.0.0.8/24", "fd00::8/64"} + assert vm.data["primary_ip4"].data["vrf"].data["name"] == "Cloud VRF" + assert vm.data["primary_ip4"].data["tenant"].data["name"] == "Operations" + assert vm.data["primary_ip6"].data["address"] == "fd00::8/64" + if netbox_version >= "4.2.0": + assert len(inventory.get_all_items(NBMACAddress)) == 1 + assert nic.data["primary_mac_address"].data["mac_address"] == "FA:16:3E:00:00:01" + else: + assert nic.data["mac_address"] == "FA:16:3E:00:00:01" + handler.apply() + assert len(inventory.get_all_items(NBVM)) == 1 + assert len(inventory.get_all_items(NBVMInterface)) == 1 + assert len(inventory.get_all_items(NBCluster)) == 2 + if netbox_version >= "3.7.0": + assert len(inventory.get_all_items(NBVirtualDisk)) == 2 + handler.finish() + cloud.session.close.assert_called_once() + + +def test_registration_authentication_and_project_scope(cloud): + assert OpenStackHandler in valid_sources + assert OpenStackConfig in ConfigFileOutput.source_config_list + cloud.settings.update(validate_tls_certs=False, all_projects=False) + handlers = instantiate_sources() + assert len(handlers) == 1 + handler = handlers[0] + cloud.sdk.connect.assert_called_once_with( + auth_url=cloud.settings["auth_url"], project_name="admin", username="sync", + password="test-only", region_name="RegionOne", user_domain_name="Default", + project_domain_name="Default", verify=False, app_name="netbox-sync", app_version="0.1", + ) + cloud.session.authorize.assert_called_once() + handler.apply() + cloud.session.compute.servers.assert_called_once_with(details=True, all_projects=False) + cloud.session.block_storage.volumes.assert_called_once_with(details=True, all_projects=False) + + +def test_missing_optional_sdk_is_explicit_and_disabled_sources_do_not_import(cloud, monkeypatch, caplog): + def missing_sdk(name): + raise ModuleNotFoundError(name) + monkeypatch.setattr("module.sources.openstack.connection.import_module", missing_sdk) + handler = OpenStackHandler("cloud") + assert not handler.init_successful + assert "requirements-openstack.txt" in caplog.text + assert handler.session is None + cloud.settings["enabled"] = False + import_mock = Mock(side_effect=AssertionError("disabled source imported SDK")) + monkeypatch.setattr("module.sources.openstack.connection.import_module", import_mock) + assert not OpenStackHandler("cloud").init_successful + import_mock.assert_not_called() + + +def test_authentication_failure_without_legacy_msg_attribute(cloud, caplog): + cloud.session.authorize.side_effect = RuntimeError("identity unavailable") + handler = OpenStackHandler("cloud") + assert not handler.init_successful + assert handler.session is None + assert "identity unavailable" in caplog.text + cloud.session.close.assert_called_once() + + +def test_filters_active_duplicates_and_uuid_custom_field(cloud): + cloud.settings.update( + strip_host_domain_name=True, strip_vm_domain_name=True, + vm_exclude_filter="ignore.*", cluster_exclude_filter="empty", + vm_role_relation=".* = Application", + ) + cloud.session.compute.servers.return_value = [ + server(status="SHUTOFF", id="inactive"), server(), + server(name="ignore.example", id="ignored"), + ] + handler = OpenStackHandler("cloud") + handler.apply() + vms = cloud.inventory.get_all_items(NBVM) + assert len(vms) == 1 + assert vms[0].data["name"] == "app" + assert vms[0].data["status"] == "active" + assert vms[0].data["role"].data["name"] == "Application" + assert cloud.inventory.get_all_items(NBDevice)[0].data["name"] == "compute1" + assert len(cloud.inventory.get_all_items(NBCluster)) == 1 + cloud.settings["set_vm_name_to_uuid"] = True + cloud.session.compute.servers.return_value = [server()] + handler = OpenStackHandler("cloud") + handler.apply() + vm = cloud.inventory.get_all_items(NBVM)[0] + assert vm.data["name"] == "vm-uuid" + assert vm.data["custom_fields"]["openstack_vm_name"] == "app.example.org" + field = cloud.inventory.get_all_items(NBCustomField)[0] + assert field.data["object_types"] == ["virtualization.virtualmachine"] + handler.apply() + assert field.data["object_types"] == ["virtualization.virtualmachine"] + + +def test_duplicate_vm_names_keep_the_existing_server_identity(cloud): + cloud.session.compute.servers.return_value = [ + server(id="server-a", status="SHUTOFF"), + server(id="server-b", status="ACTIVE"), + ] + handler = OpenStackHandler("cloud") + handler.apply() + vm = cloud.inventory.get_all_items(NBVM)[0] + assert vm.data["comments"] == "server-b" + + cloud.session.compute.servers.return_value = [ + server(id="server-a", status="ACTIVE"), + server(id="server-b", status="SHUTOFF"), + ] + handler.apply() + assert vm.data["comments"] == "server-b" + assert vm.data["status"] == "offline" + assert len(cloud.inventory.get_all_items(NBVM)) == 1 + + +def test_duplicate_zone_hosts_are_assigned_to_the_nova_compute_zone(cloud): + cloud.session.compute.availability_zones.return_value = [ + resource(name="internal", hosts={ + "compute1": {"nova-scheduler": {}, "nova-conductor": {}}, + }), + resource(name="nova", hosts={"compute1": {"nova-compute": {}}}), + ] + cloud.session.compute.servers.return_value = [server(availability_zone="nova")] + + handler = OpenStackHandler("cloud") + handler.apply() + + host = cloud.inventory.get_all_items(NBDevice)[0] + vm = cloud.inventory.get_all_items(NBVM)[0] + assert host.data["cluster"].data["name"] == "nova" + assert vm.data["cluster"].data["name"] == "nova" + assert vm.data["device"] is host + + +def test_interface_names_and_host_interface_type_are_preserved(cloud): + addresses = { + "private": [{ + "addr": "10.0.0.8", "version": 4, + "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:00:00:01", + }], + "secondary": [{ + "addr": "10.0.0.9", "version": 4, + "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:00:00:02", + }], + } + cloud.session.compute.servers.return_value = [server(addresses=addresses)] + handler = OpenStackHandler("cloud") + handler.apply() + vm_nic = next(nic for nic in cloud.inventory.get_all_items(NBVMInterface) + if nic.data["name"] == "vNIC2 (secondary)") + host_interface = cloud.inventory.get_all_items(NBInterface)[0] + host_interface.data["type"] = "1000base-t" + + cloud.session.compute.servers.return_value = [ + server(addresses={"secondary": addresses["secondary"]}), + ] + handler.apply() + + assert vm_nic.data["name"] == "vNIC2 (secondary)" + assert host_interface.data["type"] == "1000base-t" + + +def test_network_ports_invalid_addresses_and_floating_primary_preference(cloud): + cloud.settings["permitted_subnets"] += ", 192.0.2.0/24" + vm_resource = server() + vm_resource.addresses["private"].extend([ + {"addr": "invalid", "version": 4}, + {"addr": "10.0.0.9", "version": 4, "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:00:00:02"}, + ]) + vm_resource.addresses["without-mac"] = [{"addr": "10.0.1.9", "version": 4}] + cloud.session.compute.servers.return_value = [vm_resource] + OpenStackHandler("cloud").apply() + vm = cloud.inventory.get_all_items(NBVM)[0] + nics = cloud.inventory.get_all_items(NBVMInterface) + assert len(nics) == 3 + assert vm.data["primary_ip4"].data["address"] == "10.0.0.8/32" + assert {ip.data["address"] for ip in nics[0].get_ip_addresses()} == { + "10.0.0.8/32", "fd00::8/128", "192.0.2.8/32", + } + + +def test_primary_ip_uses_first_fixed_address(cloud): + cloud.settings["permitted_subnets"] += ", 192.0.2.0/24" + cloud.session.compute.servers.return_value = [server(addresses={ + "private": [ + {"addr": "192.0.2.8", "version": 4, "OS-EXT-IPS:type": "floating"}, + {"addr": "10.0.0.8", "version": 4, "OS-EXT-IPS:type": "fixed"}, + {"addr": "10.0.0.9", "version": 4, "OS-EXT-IPS:type": "fixed"}, + ], + })] + + OpenStackHandler("cloud").apply() + + vm = cloud.inventory.get_all_items(NBVM)[0] + assert vm.data["primary_ip4"].data["address"] == "10.0.0.8/32" + + +def test_flavor_lookup_unknown_volume_and_ipv6_hypervisor(cloud): + cloud.session.compute.servers.return_value = [server(flavor={"id": "flavor-1"}, + attached_volumes=[{"id": "missing"}])] + cloud.session.compute.get_flavor.return_value = resource(name="large", vcpus=4, ram=4096, disk=20) + cloud.session.compute.hypervisors.return_value[0].host_ip = "fd00::2" + handler = OpenStackHandler("cloud") + handler.apply() + vm = cloud.inventory.get_all_items(NBVM)[0] + assert vm.data["memory"] == 4096 + assert vm.data["vcpus"] == 4 + assert "disk" not in vm.data + host = cloud.inventory.get_all_items(NBDevice)[0] + assert "primary_ip4" not in host.data + assert host.data["primary_ip6"].data["address"] == "fd00::2/128" + cloud.session.compute.get_flavor.assert_called_once_with("flavor-1") + + +@pytest.mark.parametrize("status, level, message", [ + ("disabled", "INFO", "is disabled in Nova"), + ("enabled", "ERROR", "Unable to find an availability zone"), +]) +def test_host_without_availability_zone_is_skipped(cloud, caplog, status, level, message): + cloud.session.compute.hypervisors.return_value.append( + resource(name="compute2.example", service_details={"host": "compute2"}, status=status, + state="up", host_ip="10.0.0.3", hypervisor_type="QEMU", hypervisor_version=8000000)) + handler = OpenStackHandler("cloud") + with caplog.at_level("INFO"): + handler.apply() + names = [device.data["name"] for device in cloud.inventory.get_all_items(NBDevice)] + assert "compute2.example" not in names + records = [r for r in caplog.records if "compute2.example" in r.getMessage() and message in r.getMessage()] + assert [r.levelname for r in records] == [level] + + +def test_cluster_matching_does_not_reassign_another_cloud(cloud): + other_group = cloud.inventory.add_object(NBClusterGroup, data={"name": "Other"}) + other_site = cloud.inventory.add_object(NBSite, data={"name": "Other Site"}) + foreign = cloud.inventory.add_object(NBCluster, data={ + "name": "zone.example", "group": other_group, "type": {"name": "Openstack"}, + "scope_type": NBSite.object_type, "scope_id": other_site, + }) + handler = OpenStackHandler("cloud") + handler.apply() + assert foreign.data["group"] is other_group + assert foreign.get_site_name() == "Other Site" + assert handler.get_object_from_cache(NBCluster, "zone.example") is not foreign + + +def test_clusters_with_same_default_group_are_isolated_by_source_tags(cloud): + other_source = SimpleNamespace(source_tag="Source: other") + cloud.inventory.source_list = [other_source] + other_site = cloud.inventory.add_object(NBSite, data={"name": "Other Site"}) + foreign = cloud.inventory.add_object(NBCluster, data={ + "name": "zone.example", "group": {"name": "Openstack"}, "type": {"name": "Openstack"}, + "scope_type": NBSite.object_type, "scope_id": other_site, "tags": ["Source: other"], + }) + handler = OpenStackHandler("cloud") + handler.apply() + assert foreign.get_site_name() == "Other Site" + assert handler.get_object_from_cache(NBCluster, "zone.example") is not foreign + + +def test_missing_volume_retains_existing_virtual_disk_size(cloud): + handler = OpenStackHandler("cloud") + handler.apply() + vm = cloud.inventory.get_all_items(NBVM)[0] + disk = next(disk for disk in vm.get_virtual_disks() if disk.data["name"] == "volume-1") + cloud.session.block_storage.volumes.return_value = [] + handler.apply() + assert disk.data["size"] == 20000 + assert disk.source is handler + assert len(vm.get_virtual_disks()) == 2 + + +@pytest.mark.parametrize("setting,value", [ + ("vm_include_filter", "["), ("vm_tenant_relation", "missing separator"), + ("set_primary_ip", "sometimes"), ("ip_tenant_inheritance_order", "unknown"), +]) +def test_invalid_settings_fail_cleanly(cloud, setting, value): + cloud.settings[setting] = value + with pytest.raises(SystemExit): + OpenStackHandler("cloud") + + +def test_mac_and_primary_ip_matching_use_current_netbox_objects(cloud): + handler = OpenStackHandler("cloud") + handler.apply() + vm = cloud.inventory.get_all_items(NBVM)[0] + assert handler.get_object_based_on_macs(NBVM, ["FA:16:3E:00:00:01"]) is vm + assert handler.get_object_based_on_primary_ip(NBVM, primary_ip4="10.0.0.8") is vm + assert handler.get_object_from_cache(NBVM, "zone.example") is None + + +def test_interface_exclusions_and_resource_custom_attributes(cloud): + cloud.settings.update( + vm_interface_exclude_filter="vNIC1.*", host_interface_exclude_filter="eth0", + vm_custom_object_attributes="id, flavor.vcpus", host_custom_object_attributes="name", + ) + OpenStackHandler("cloud").apply() + assert not cloud.inventory.get_all_items(NBVMInterface) + assert not cloud.inventory.get_all_items(NBIPAddress) + vm = cloud.inventory.get_all_items(NBVM)[0] + assert vm.data["custom_fields"]["openstack_id"] == "vm-uuid" + assert vm.data["custom_fields"]["openstack_flavor_vcpus"] == "2" + host = cloud.inventory.get_all_items(NBDevice)[0] + assert host.data["custom_fields"]["openstack_name"] == "compute1.example" + + +def test_source_neutral_vm_ip_updates_remove_undiscovered_addresses(cloud): + handler = OpenStackHandler("cloud") + handler.apply() + vm = cloud.inventory.get_all_items(NBVM)[0] + old_ip = vm.data["primary_ip4"] + cloud.session.compute.servers.return_value = [server(addresses={ + "private": [{"addr": "10.0.0.9", "version": 4, + "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:00:00:01"}], + })] + handler.settings.set_primary_ip = "always" + handler.apply() + assert vm.data["primary_ip4"].data["address"] == "10.0.0.9/32" + assert "assigned_object_id" in old_ip.unset_items + assert len(cloud.inventory.get_all_items(NBVMInterface)) == 1 + + +@pytest.mark.parametrize("file_type", [ConfigFileINI, ConfigFileYAML]) +def test_config_generation_does_not_require_sdk(monkeypatch, file_type): + monkeypatch.setattr("module.sources.openstack.connection.import_module", + Mock(side_effect=AssertionError("config generation imported SDK"))) + output = ConfigFileOutput.__new__(ConfigFileOutput) + output.config_file_type = file_type + output.comment_prefix = file_type.comment_prefix + output.lines = [] + output.format() + text = "\n".join(output.lines) + assert "my-openstack-example" in text + assert "openstack" in text + assert "all_projects" in text + assert "ip_tenant_inheritance_order" in text