Repository navigation
feat: add ZVG-Portal provider for forced-sale auctions (Zwangsversteigerungen) - #499
Conversation
…cription and area
… skip cancelled dates
orangecoding
left a comment
There was a problem hiding this comment.
Nice idea, forced sales are a source nobody else covers. Can't merge it yet though. The object type filter doesn't do anything on the portal side (details inline), so every job pulls every auction of the whole state. The rest is smaller stuff.
| if (name === 'obj_liste') { | ||
| for (const id of value.split(',')) if (id.trim()) body.append('obj_liste', id.trim()); |
There was a problem hiding this comment.
obj_liste is only the left select in the portal form, the list of types you can pick from. What actually gets submited as the filter is obj_arr[]: insertObj() copies the options over from obj_liste and checkFormular() selects everything in obj_arr right before submit. The server just ignores obj_liste. And without [] PHP would only keep the last value anyway, you can see that on the result page, it echoes <input type=hidden name=obj_liste value='16'>.
Tried it against the live portal with NRW:
- no type filter: 910 Termine
obj_liste=3&obj_liste=15&obj_liste=16: 910obj_arr[]=3&obj_arr[]=15&obj_arr[]=16: 238
So right now every job gets every object type of the whole state. Flats, garages, parking spots, all of it, and on the first run all of that is new, gets geocoded and notifed. The SH test only passes because SH has 3 auctions at the moment and they're all Einfamilienhäuser. Can't tell a working filter from a broken one there.
Please send the ids as obj_arr[]. Whatever the URL parameter ends up beeing called, the doc block (l. 14/17) and the PR description should match it. A small test on the POST body would have caught this btw.
| const body = new URLSearchParams({ ger_id: '0', order_by: '2', art: '' }); | ||
| for (const [name, value] of params) { | ||
| if (name === 'button') continue; | ||
| if (name === 'obj_liste') value.split(',').forEach((id) => body.append('obj_liste', id)); |
There was a problem hiding this comment.
This is a second copy of parseSearchUrl, incl. the same obj_liste bug. The betterhomes downloader imports buildSearchPayload from its provider for exactly that reason. Can you export parseSearchUrl and use it here? Then the fixture gets recorded with the same request the provider actualy sends.
| return { | ||
| ...listing, | ||
| size: listing.size ?? extractNumber(livingArea ?? plotArea ?? ''), | ||
| description: [listing.description, description, place ? `Ort der Versteigerung: ${place}` : null] |
There was a problem hiding this comment.
fetchDetails appends to listing.description, so running it twice gives a different result than running it once. That bites with the manual "fetch details" button (refetchListingDetails): it passes in the stored row, and after the first fetch (or a run with details enabled) that row already has the court text in it. Every click appends Beschreibung and Ort again and isImprovement happyly calls it updated.
Simulated it with a live detail page: 560 → 1046 → 1532 chars, court text in there 1x, 2x, 3x. A second call should give the same description, e.g. just dont append text that's already there.
| const rows = parseResults(await response.text()); | ||
| return plzPrefixes.length === 0 | ||
| ? rows | ||
| : rows.filter((row) => plzPrefixes.some((p) => new RegExp(`\\b${p}\\d*\\s`).test(row.address ?? ''))); |
There was a problem hiding this comment.
A few things with the prefix match. \b${p}\d*\s also matches house numbers followed by a space. With prefix 4 on the live NRW list I get 6 adresses that aren't in a 4xxxx postcode, for example Seydlitzstr. 48 a, 58840 Plettenberg or Wintersheide 4 - 14, 33689 Bielefeld. The prefix also goes into new RegExp unescaped, so a typo like plz_prefix=2( throws and the whole run fails. And the regex is rebuilt for every row.
Matching against the actual postcodes gets rid of all three:
(row.address?.match(/\b\d{5}\b/g) ?? []).some((zip) => plzPrefixes.some((p) => zip.startsWith(p)))| .text() | ||
| .replace(/\s+/g, ' '); | ||
| const description = /Beschreibung:\s*(.*?)\s*Verkehrswert/.exec(text)?.[1]; | ||
| const place = /Ort der Versteigerung:\s*(.*?)\s*(?:Informationen zum|Gericht:)/.exec(text)?.[1]; |
There was a problem hiding this comment.
Small one: on pages without a court link Ort der Versteigerung is followed by GeoServer: instead (zvg_id=168706, nw, AG Essen), so place stays empty there. Adding GeoServer: to the alternatives shuold do it.
… fetchDetails idempotent - the portal filters on obj_arr[]; obj_liste is only the picker list and was ignored - postcode prefixes are compared with the 5-digit postcodes of the address (no regex from user input) - fetchDetails no longer appends text the description already has - place of the auction also ends at GeoServer: - the fixture downloader reuses the provider's parseSearchUrl - tests for the POST body, the postcode filter, repeated fetchDetails and GeoServer
…living-area match, request timeout, docs - the id is the court's record alone; a corrected Verkehrswert no longer stores and announces the auction twice - the 1 EUR the court enters before it sets a Verkehrswert is no price - living area also from 'Wohnfl.', 'Wohnfläche: ca.', 'rd.', 'EG' - 30s timeout on every request; warn when the portal answers a bare error - empty address parts ', ,' are dropped - tests assert the POST actually sent, the id, the placeholder, the living-area forms and, live, that obj_arr narrows a large state - doc/providers.md: ZVG-Portal section
|
Thanks for the thorough review, you're right on all points. I reproduced the type filter issue against the live portal: NRW returns 772 auctions with no filter and with Pushed two commits. The first addresses your comments:
The second is from a review of my own against the live portal: the listing id no longer includes the price (a corrected Verkehrswert would otherwise have been stored and announced twice), the 1 EUR placeholder isn't taken for a price, more living-area spellings are read, requests have a timeout, and I added a doc section. One limitation I'd rather state than hide: the portal's detail page only answers with a portal Referer, so the notification link can only open the search page. Tests now assert the POST that is sent, and a live-only test checks that the type filter really narrows a large state. |
orangecoding
left a comment
There was a problem hiding this comment.
All good now, thanks for fixing everything so quickly. The NRW live test is a nice touch.
What does this PR do?
Adds a provider for the ZVG-Portal (zvg-portal.de), the German justice portal that lists forced-sale auctions. Auctions rarely appear on the commercial portals, so this finds houses and plots nobody else lists.
The portal has no GET search: a search is a form POST answered with a server-rendered table. The job URL only describes the search and the provider replays it:
https://www.zvg-portal.de/index.php?button=Suchen&land_abk=sh&obj_arr=3,15,16&plz_prefix=22,25
land_abk: state code (required by the portal, lowercase)obj_arr: comma separated object type ids, numbered as in the portal's form (3 Einfamilienhaus, 15 Baugrundstück, 16 unbebautes Grundstück, ...). Sent asobj_arr[], the field the portal filters on; leave it out and every type of the state comes backplz,ort, ...) is passed throughplz_prefix(not a portal field): keeps only results with a postcode starting with one of the prefixes, for a state-wide search where only a region is wantedNotes:
button=showZvg) answers "error" without a portal Referer, so a link from a notification cannot open it. Listings link to the portal's search page instead (unique via an extrazvg_idparameter), andfetchDetailsreads the detail page server-side for the court's description, the living area and the auction location.fetchwith a timeout,needsBrowser: false.Related issue
none
AI disclosure (required)
Fredy accepts AI assisted contributions, but they have to be declared. An
undeclared AI PR will be closed. Tick exactly one box:
ai:none- I wrote this myself. No AI generated code, text or commit messages.ai:assisted- AI helped with parts of it (autocomplete, refactoring, tests, docs). I reviewed and understand every line.ai:generated- AI produced most or all of this PR. I reviewed it, but it is largely machine written.Which AI: Claude Code, Sonnet 5.5
How much: the provider, its test, the fixture plumbing and the downloader were drafted with Claude Code; I tested it against the live portal, reviewed every line and ran the checks.
Why: a forced-sale source for houses and plots in my area, and faster scaffolding of a provider with an unusual POST search.
Checklist
yarn test:offlinepasses (oryarn testif the change touches a live provider)yarn lintandyarn format:checkpass