Skip to content

load_entity raises instead of logging when a category page has no usable schema title #202

Description

@LukasGold

OSW.load_entity reads the schema title outside the block that handles a failure per page. One unusable category page therefore ends the whole call, and the remaining titles are not loaded.

osw-python/src/osw/core.py

Lines 1355 to 1372 in d7f0116

for category in jsondata["type"]:
schema = (
self.site
.get_page(
WtSite.GetPageParam(
titles=[category], offline_pages=param.offline_pages
)
)
.pages[0]
.get_slot_content("jsonschema")
)
schemas.append(schema)
# generate model if not already exists
cls_name: str = schema["title"]
# If a schema_to_use is provided, we do not need to check if
# the model exists
if not param.model_to_use:
if not hasattr(model, cls_name):

for category in jsondata["type"]:
    schema = (
        self.site.get_page(...).pages[0].get_slot_content("jsonschema")
    )
    schemas.append(schema)
    cls_name: str = schema["title"]          # line 1368
    if not param.model_to_use:
        if not hasattr(model, cls_name):     # line 1372

Three inputs raise here:

Input Exception
the category page has no jsonschema slot, so get_slot_content returns None TypeError: 'NoneType' object is not subscriptable (line 1368)
the schema has no title key KeyError: 'title' (line 1368)
title is present but not a string TypeError: attribute name must be string (line 1372)

get_slot_content returns None for a slot the page does not have:

osw-python/src/osw/wtsite.py

Lines 1700 to 1701 in d7f0116

if slot_key not in self._slots:
return None

A page reaches this state when its type names a page that does not exist, is not a category, or has an empty jsonschema slot.

Expected

Report the fault for that page and continue with the next one, as the rest of the loop does.

  • The entity construction below catches every exception, logs it with the page title, and the loop continues:

    osw-python/src/osw/core.py

    Lines 1407 to 1408 in d7f0116

    except Exception as e:
    _logger.error(f"Error creating entity from page {page.title}: {e}")
  • The missing-model case four lines lower does the same: it logs with _logger.error, sets schemas_fetched = False and the loop continues with the next page:

    osw-python/src/osw/core.py

    Lines 1381 to 1388 in d7f0116

    if not hasattr(model, cls_name):
    schemas_fetched = False
    _logger.error(
    f"Model {cls_name} not found. Schema {category} "
    f"needs to be fetched first."
    )
    if not schemas_fetched:
    continue

Today the exception passes through the try at line 1340, whose finally only restores the cache state, and leaves load_entity. The message names neither the page nor the category, so the caller cannot tell which page is wrong.

Notes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions