Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,12 @@ scheduler (every N minutes) or manual trigger via POST /api/jobs/:id/run
geocoder searches within the resolved countries; the map's `maxBounds` is the union of their
boxes from `ui/src/components/map/countryBounds.js`. Where no provider exists to ask - home
addresses, the listings map, the listing detail - the answer is the union across the jobs the user
can see, and where the job form is open it is the providers ticked in it
can see, and where the job form is open it is the providers ticked in it. An optional `currency`
(ISO 4217, absent means `EUR`) and `currencyByHost` (the exceptions, for a provider whose national
sites price differently) are read once, when the pipeline stores a listing; after that
`listings.currency` is the only source, because comparisons happen in SQL. `currency.js` in the
same directory holds the resolver. A row without one fails on the `NOT NULL` column by design, so
every writer has to set it
- `config` - the **static** `ProviderConfig` template: `requiredFieldNames`, `crawlContainer`, `crawlFields`, `sortByDateParam`, `normalize()`, optional `getListings()`, `fetchDetails()`, `activityProbe()`, and `needsBrowser: false` on a provider whose search and details never touch the browser (ImmoScout's mobile API, flatfox, the tecnocasa network), so a job run or detail fetch with only such providers starts no Chromium. `url` is `null` here and there is no bound `filter`. A portal whose pages only answer a browser session (DataDome answers node with 403 whether an advert is online or not - immowelt, SeLoger) declares `browserActivityProbe(link, browser)` and `priceTracking.browserProbe(listing, browser)` instead; the alive-checker and the price tracker start one browser per run for them, and only when a due listing needs it.
- `createConfig(sourceConfig, blacklist)` - returns a **fresh** `ProviderConfig` per job run: the template plus this run's `url`, `enabled`, and a `filter` closed over this run's blacklist.
- Optional `fetchListingByUrl(url, browser)` on the static template - reads one advert's page into a complete `ParsedListing`, or `null` when the page is no single advert or could not be read. It is what "Add listing" in the listings view calls. Build the provider's own search-card shape from the exposé and pass it through its `normalize`, so `link` and hash come out as a search run would produce them: a later run of the job recognises the stored row by both, and a different spelling stores the advert twice. The `/api/jobs/provider` response carries `addByUrl` per provider so the dialog can say "not supported yet" before anything is sent.
Expand Down
21 changes: 15 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,12 +108,21 @@ The picker orders providers by country and then by size, from
behind the ranked ones of its own country. Add it to `PROVIDER_SIZE_ORDER` only if it belongs
somewhere specific. A country new to `COUNTRY_ORDER` sorts behind the ones already listed.

The same shape is where anything else that varies by country belongs. `currency` is the obvious
next one and is deliberately not implemented: `€` appears in around forty places, twelve of them in
the finance module, which assumes considerably more than a symbol, and three locale files carry it
inside translated strings. Whoever arrives with a provider quoting in kroner should add
`currency: 'DKK'` here, default it to `EUR`, and resolve it exactly the way `countries` is
resolved in `lib/services/providers/`.
A provider that quotes prices outside the euro declares `currency` on `metaInformation`, for
example `currency: 'DKK'`. The default is `EUR`. A provider whose national sites price differently
lists the exceptions by host, for example `currencyByHost: { 'betterhomes.ch': 'CHF' }`. That is
data rather than a function like `countryOf`, because the job form has to read it in the browser.

The pipeline copies the answer onto each listing as it stores it, and from then on
`listings.currency` is the only source: comparisons happen in SQL, and everything else reads the
row. Declaring a currency therefore does not reach listings already stored. A provider that has
any needs a migration to backfill them, as `54.listing-currency.js` does for flatfox.

Amounts stay in the portal's currency; Fredy does not convert them. Listing views, notifications
and MCP carry that currency. Monetary dashboard aggregates are hidden for mixed currencies, and
a numeric price range or job price ceiling needs sources that share one currency. The finance
module still models EUR and German purchase costs, so non-EUR listings receive no finance result
or affordability verdict. Do not add a country-specific currency map elsewhere in the app.

### How to write new notification adapter?

Expand Down
13 changes: 13 additions & 0 deletions doc/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,19 @@ export const metaInformation = {
Addresses are then geocoded in those countries, and the map opens on them so you can draw a search
area there. A provider spanning several is fine: `countries: ['de', 'at', 'ch']`.

## Currencies

Providers declare `metaInformation.currency`, with `EUR` as the default. Flatfox uses `CHF`;
BETTERHOMES prices in EUR and lists its Swiss site under `currencyByHost`. Each listing stores its
currency when it is found. Prices stay in their original currency throughout listing views,
notifications, price history and MCP. Fredy does not convert between currencies.

A dashboard covering several currencies hides its monetary medians, and price ranges need a
selection with one currency. A job can only be saved with a price ceiling if its sources share a
currency. An older job that mixes them keeps filtering with its ceiling until it is next edited,
and then needs the ceiling cleared to save. The finance calculator retains its EUR and German
purchase-cost assumptions: it does not score non-EUR listings.

## 🛡️ Bot detection & proxies

Most browser-based providers (kleinanzeigen, wg-gesucht, ohnemakler, ...) are scraped through a
Expand Down
19 changes: 11 additions & 8 deletions lib/FredyPipelineExecutioner.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ import { distancesToAddresses } from './services/listings/distanceCalculator.js'
import { updateTravelTimesForListings } from './services/listings/travelTimeSweeper.js';
import { getSettings, getUserSettings, getAddresses } from './services/storage/settingsStorage.js';
import booleanPointInPolygon from '@turf/boolean-point-in-polygon';
import { currencyForCountries, formatListing } from './utils/formatListing.js';
import { formatListing } from './utils/formatListing.js';
import { getCurrencyForListing } from './services/providers/providerCurrency.js';
import { addressesWithBudget, exceedsCommuteBudget, normalizeCommuteFilter } from './utils/commuteBudget.js';
import { sleep } from './utils.js';
import { enqueuePreviewCapture } from './services/listings/previewCaptureQueue.js';
Expand Down Expand Up @@ -584,11 +585,7 @@ class FredyPipelineExecutioner {
// longer gets "3 Zimmer" in every notification.
const userId = getJob(this._jobKey)?.userId;
const language = getUserSettings(userId)?.language ?? 'en';
const formattedListings = await Promise.all(
newListings.map(async (listing) =>
formatListing(listing, language, currencyForCountries(await getCountriesForListing(this._providerId, listing))),
),
);
const formattedListings = newListings.map((listing) => formatListing(listing, language));
const settings = await getSettings();
const baseUrl = settings?.baseUrl ?? '';
const sendNotifications = notify.send(
Expand All @@ -604,11 +601,17 @@ class FredyPipelineExecutioner {
/**
* Persist new listings and pass them through.
*
* The currency is settled here, once: `storeListings` is synchronous and the provider metadata is
* not, and every later step reads it off the listing or the row.
*
* @param {ParsedListing[]} newListings Listings to store.
* @returns {ParsedListing[]} The same listings, unchanged.
* @returns {Promise<ParsedListing[]>} The same listings, now carrying their currency.
*/
_save(newListings) {
async _save(newListings) {
logger.debug(`Storing ${newListings.length} new listings (Provider: '${this._providerId}')`);
for (const listing of newListings) {
listing.currency = await getCurrencyForListing(this._providerId, listing);
}
storeListings(this._jobKey, this._providerId, newListings);
return newListings;
}
Expand Down
2 changes: 2 additions & 0 deletions lib/api/routes/dashboardRouter.js
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ export default async function dashboardPlugin(fastify) {
numberOfListings,
numberOfActiveListings,
medianPriceOfListings,
currency,
medianPriceSampleSize,
medianPricePerSqm,
} = getListingsKpisForJobIds(jobIds);
Expand Down Expand Up @@ -131,6 +132,7 @@ export default async function dashboardPlugin(fastify) {
numberOfListings,
numberOfActiveListings,
medianPriceOfListings,
currency,
medianPriceSampleSize,
medianPricePerSqm,
},
Expand Down
9 changes: 8 additions & 1 deletion lib/api/routes/financeRouter.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
* Licensed under Apache-2.0 with Commons Clause and Attribution/Naming Clause
*/

import { FINANCE_CURRENCY } from '../../services/providers/currency.js';
import * as listingStorage from '../../services/storage/listingsStorage.js';
import { isAdmin as isAdminFn } from '../security.js';
import logger from '../../services/logger.js';
Expand Down Expand Up @@ -114,6 +115,9 @@ export default async function financePlugin(fastify) {
const stored = getUserSettings(userId)?.finance_profile ?? null;
const summary = summarizeProfile(stored);
const price = Number(listing.price);
if (listing.currency !== FINANCE_CURRENCY) {
return { dealType: listing.dealType ?? null, scored: null, result: null, ...summary, unsupportedCurrency: true };
}
if (!Number.isFinite(price) || price <= 0) {
return { dealType: listing.dealType ?? null, scored: null, result: null, ...summary };
}
Expand Down Expand Up @@ -178,8 +182,11 @@ export default async function financePlugin(fastify) {

try {
// Reuse the normal listings query so this endpoint inherits the exact same user
// scoping - a user can never score listings belonging to someone else.
// scoping - a user can never score listings belonging to someone else. The currency is part
// of the query rather than a skip below: those rows would still use up `pageSize`, and euro
// listings past the cap would go unscored.
const { result: listings } = listingStorage.queryListings({
currencyFilter: FINANCE_CURRENCY,
page: 1,
pageSize: MAX_SCORED_LISTINGS,
jobIdFilter: nullIfEmpty(filter.jobId),
Expand Down
4 changes: 3 additions & 1 deletion lib/api/routes/jobRouter.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import { canUseChannel } from '../../services/security/channelAccess.js';
import * as channelStorage from '../../services/storage/configuredAdapterStorage.js';
import { canAccessJob, canModifyJob } from '../../services/security/access.js';
import { normalizeCommuteFilter } from '../../utils/commuteBudget.js';
import { jobCurrencyProblem } from '../../services/providers/currency.js';
import { providerListProblem } from '../../services/jobs/providerUrl.js';
import { getProviders } from '../../utils.js';

Expand Down Expand Up @@ -244,7 +245,8 @@ export default async function jobPlugin(fastify) {
const providerMeta = new Map(
(await getProviders()).map((module) => [module?.metaInformation?.id, module?.metaInformation]),
);
const providerProblem = providerListProblem(provider, providerMeta);
const providerProblem =
providerListProblem(provider, providerMeta) ?? jobCurrencyProblem(provider, providerMeta, specFilter);
if (providerProblem != null) {
return reply.code(400).send({ error: providerProblem });
}
Expand Down
8 changes: 6 additions & 2 deletions lib/api/routes/listingsRouter.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ import { canAccessJob } from '../../services/security/access.js';
import { updateDistancesForListing } from '../../services/geocoding/distanceService.js';
import { geocodeAddress, isGeocodingPaused } from '../../services/geocoding/geoCodingService.js';
import { getCountriesForListing } from '../../services/providers/providerCountries.js';
import { FINANCE_CURRENCY } from '../../services/providers/currency.js';
import { MixedCurrencyError } from '../../errors.js';
import { SCAM_OVERRIDES } from '../../services/listings/scamSignals.js';
import { buildApplicationLetter } from '../../services/application/applicationService.js';
import { refetchListingDetailsOnce } from '../../services/listings/detailRefetchService.js';
Expand Down Expand Up @@ -47,7 +49,9 @@ const NO_ACCESS_MESSAGE = 'You are trying to access a listing that is not associ
* @returns {'affordable'|'stretch'|'unaffordable'|null}
*/
const verdictForRow = (listing, thresholds) =>
verdictForListing(listing.price, listing.dealType, thresholds, { chargesIncluded: rentIncludesCharges(listing) });
listing.currency === FINANCE_CURRENCY
? verdictForListing(listing.price, listing.dealType, thresholds, { chargesIncluded: rentIncludesCharges(listing) })
: null;

/**
* A single listing with its stored gallery. Only the single read carries it: at most twenty urls,
Expand Down Expand Up @@ -653,7 +657,7 @@ export default async function listingsPlugin(fastify) {
return reply.send({ deleted: changes });
} catch (error) {
logger.error(error);
return reply.code(500).send({ error: error.message });
return reply.code(error instanceof MixedCurrencyError ? 400 : 500).send({ error: error.message });
}
});

Expand Down
9 changes: 8 additions & 1 deletion lib/errors.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,14 @@ class ExtendableError extends Error {
}
}
class NoNewListingsWarning extends ExtendableError {}
export { NoNewListingsWarning };
/** A price bound was put on listings priced in several currencies, where it has no one unit. */
class MixedCurrencyError extends ExtendableError {
constructor() {
super('Select one currency before filtering by price.');
}
}
export { NoNewListingsWarning, MixedCurrencyError };
export default {
NoNewListingsWarning,
MixedCurrencyError,
};
9 changes: 8 additions & 1 deletion lib/mcp/jobDraftContext.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,14 @@ export async function buildDraftContext(user) {
const providers = (await getProviders())
.map((provider) => provider.metaInformation)
.filter((meta) => meta?.id != null && meta?.baseUrl != null)
.map(({ id, name, baseUrl, hosts }) => ({ id, name, baseUrl, ...(hosts == null ? {} : { hosts }) }));
.map(({ id, name, baseUrl, hosts, currency, currencyByHost }) => ({
id,
name,
baseUrl,
currency,
currencyByHost,
...(hosts == null ? {} : { hosts }),
}));

// `fields` is dropped here rather than at the point of rendering: a channel object that still
// carries its token cannot leak from a place it never reaches.
Expand Down
16 changes: 14 additions & 2 deletions lib/mcp/jobDraftStore.js
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@

import { hostsOf, normalizeHost, resolveProviderForUrl, validateProviderUrl } from '../services/jobs/providerUrl.js';
import { DEAL_TYPES, normalizeDealType, detectDealTypeForJob } from '../services/dealType.js';
import { currencyForLink, jobCurrencyProblem } from '../services/providers/currency.js';
import { normalizeCommuteFilter } from '../utils/commuteBudget.js';

/**
Expand Down Expand Up @@ -340,6 +341,12 @@ export function applyDraftUpdate(draft, patch, ctx) {
next.enabled = patch.enabled === true;
}

const currencyProblem = jobCurrencyProblem(
next.providers,
new Map(ctx.providers.map((p) => [p.id, p])),
next.specFilter,
);
if (currencyProblem) problems.push(currencyProblem);
if (problems.length > 0) return { ok: false, draft, problems };

if (touchedRefinement) next.refinementsAnswered = true;
Expand Down Expand Up @@ -402,7 +409,12 @@ export function describeDraft(draft, ctx) {
spec == null
? '–'
: [
spec.maxPrice != null ? `max ${spec.maxPrice} EUR` : null,
spec.maxPrice != null
? `max ${spec.maxPrice} ${currencyForLink(
ctx.providers.find((p) => p.id === draft.providers[0]?.id),
draft.providers[0]?.url,
)}`
: null,
spec.minSize != null ? `min ${spec.minSize} m²` : null,
spec.minRooms != null ? `min ${spec.minRooms} rooms` : null,
]
Expand Down Expand Up @@ -478,7 +490,7 @@ export function nextStep(draft, ctx) {
return {
key: 'refinements',
question:
'Optional refinements - answer with any of these, or say "skip": maximum price in EUR, minimum size in m², ' +
'Optional refinements - answer with any of these, or say "skip": maximum price in the selected portals’ currency, minimum size in m², ' +
'minimum number of rooms, words that should exclude a listing, ' +
(addresses.length > 0
? `travel-time limits in minutes for your saved addresses (${addresses.join(', ')}), `
Expand Down
Loading