# Migration and import Bringing the catalogue over from the previous Drupal site, repeatedly, without losing what has been curated here — and handing it to the client as a workbook they correct. ## Two generations | | 2025 | 2026 | |---|---|---| | What | `data/migration/*.sql` + `commands/LegacyController.php` | `commands/ImportController.php` | | How | `DELETE … WHERE legacyId IS NOT NULL` then re-insert, ids hard-coded | upsert keyed on a legacy id, field by field | | Images | every image, downloaded over HTTP on every run | only the delta; local file first, HTTP fallback | | Runs | once | as often as needed; a second run is a no-op | **The 2025 generation is history and is never run again.** It stays in the repository as the record of the first migration. Running it now would erase a year of editorial work — see [ADR 0035](../adr/0035-the-drupal-re-import-is-incremental.md). ## Who owns which field The rule the whole import turns on. `ImportDiffService::STORAGE_FIELDS` and `ARTICLE_FIELDS` are the machine-readable version, and the same structures drive the report, every `--dry-run` and the write. | Owner | Fields | |---|---| | **The source** | `price1M/3M/6M/12M`, the property values, `description` (hu), `locationId`, `storageTypeId`, existence | | **Us, never overwritten** | `slug`, `name`, `nameInContract`, `shortName`, `svgNodeId`, `areaId`, `isPublished`, `isFeatured`, `isNew`, `isPremium`, `translations`, everything on `location` and `address` | | **The workbook** | whatever the client corrects, applied **after** the source import | That last row is a one-way gate: **once the corrected workbook is in, the source import must not run again.** Two mechanical consequences worth knowing: - **`ActiveRecord::beforeSave()` regenerates the slug from the name on every save**, and a number of units carry a hand-tuned slug that no longer matches — unit `01` is served under `kk-01`. Both importers blank `$model->createSlugFrom` before saving, so the URL survives. Everything else about the save still happens, including the search index and the translation queue. - **The importer never deletes.** A unit the source no longer has gets `isPublished = 0` and a line in the report. It may carry photos, property values, rentals and payments. - **A new site is created unpublished, its units published.** Publishing the site is then the single switch that puts them on the public listing, because a unit is public only if its site is — see [STORAGE-AND-LOCATIONS.md](STORAGE-AND-LOCATIONS.md#a-unit-is-public-only-if-its-site-is). ## The three databases [`bin/import-db`](../../bin/import-db) sets them up. All three live on the local MariaDB. | Database | Role | |---|---| | `drupal_src` | the Drupal dump — read-only source, only ever `SELECT`ed from | | `r24_prod` | a pristine copy of the production dump — the restore point, never written | | the application's own | the working copy the importers run against | ```shell bin/import-db drupal [dump.sql] # load data/import/dumps/drupal.sql into drupal_src, grant SELECT bin/import-db prod [dump.sql] # load data/import/dumps/prod.sql into r24_prod bin/import-db reset [--force] # replace the application database with the r24_prod copy bin/import-db restore-dev # put back what reset last overwrote bin/import-db status ``` `reset` is what makes the idempotence check cheap: restore, run the chain twice, diff the dumps. It backs the application database up to `data/import/dumps/app-before-reset.sql` first, every time. The source is read with **cross-schema queries on the application connection** — `drupal_src.node` — not through a second `Connection` component. `LEGACY_DB_NAME` in `env.php` names it; empty disables every importer. The application user needs `SELECT` on it, which `bin/import-db drupal` grants. ## The commands Every one of them takes `--dryRun=1`. | Command | Does | |---|---| | `import/report` | what an import would change, and the data-quality problems for the client. Writes nothing. | | `import/catalogue` | reference data, new sites, the units, the articles | | `import/photos` | the photos: local file tree first, the old site over HTTP second | | `import/contents-manifest` | a checksum manifest of `web/contents/` — the upload's verification step | | `import/occupancy` | seeds the occupancy overrides from the source's availability calendar | | `import/export-xlsx` | writes the workbook | | `import/import-xlsx ` | reads a corrected workbook back | The order for a full run: ```shell bin/import-db drupal # a fresh source dump docker exec -ti raktar24-php-fpm ./yii import/report # send this to the client first docker exec -ti raktar24-php-fpm ./yii import/catalogue docker exec -ti raktar24-php-fpm ./yii import/photos docker exec -ti raktar24-php-fpm ./yii import/occupancy docker exec -ti raktar24-php-fpm ./yii search-index/build tar -czf contents-new.tar.gz -C web/contents $( date +%Y/%m ) # upload today's folder docker exec -ti raktar24-php-fpm ./yii import/contents-manifest --against=… # verify it all arrived docker exec -ti raktar24-php-fpm ./yii import/export-xlsx # hand it over docker exec -ti raktar24-php-fpm ./yii import/import-xlsx … # when it comes back ``` ## The mapping tables [`data/import/maps.php`](../../data/import/maps.php) holds what the source cannot express, reviewed by hand once. Keyed by **short name**, not by id, so it survives a database that numbers its rows differently. | Map | What | |---|---| | `locations` | `helyszin` taxonomy term → location short name. **The reliable site key.** 16 terms. | | `newLocations` | the definition of a site the map names and the database does not have yet | | `storageTypes` | `tipus` taxonomy term → `storage_type.legacyId` | | `locationPhotoNodes` | `helyszinalaprajz` node → location short name, for the floor-plan photos. Many-to-one; `null` means the node is not a site. | > **The site's key is the `helyszin` taxonomy term, not the floor-plan node.** `location.legacyId` used to > hold the node id, which is wrong three ways: several nodes describe one site, two sites have no node, and > the Szentendre sites had the wrong node — `SZTE` (Kalászi út) claimed node 177, the Dózsa György út > container yard, and inherited its photos. `m260907_100000_location_legacy_keys` repoints it. ## Reading the source [`services/LegacySourceService.php`](../../services/LegacySourceService.php) is the **only** place that knows Drupal's shape. Everything downstream sees plain arrays keyed by legacy id with our column names on them. The five property-value conversions, chosen by `storage_property.legacyModule` and our own `type` — the same five the 2025 migration spelled out in thirty hand-written `UNION` branches: | legacyModule | type | value | |---|---|---| | `taxonomy` | SELECT / MULTISELECT | the term id, resolved to an option | | `taxonomy` | NUMBER | the digits in the term name (`"15 m"` → 15) | | `taxonomy` | BOOLEAN | `false` for `Nem` and `Nincs`, `true` for anything else | | `number` | NUMBER | the integer column | | `list` | BOOLEAN | `1` is true, `2` is false | > **The boolean conversion is lossy, on purpose.** `Igényelhető` ("available on request") becomes `true`, > like `Igen`, because the target column is a boolean. Kept identical to 2025 so a re-import does not churn > fourteen thousand rows. Occupancy comes from the **Availability Calendars** module: `availability_calendar_availability` is one row per calendar per day, states `2` free, `3` booked, `4` provisionally booked, and `field_data_field_foglaltsag` links a calendar to a unit. [`helpers/DateRanges.php`](../../helpers/DateRanges.php) run-length encodes the days into bookings; `--gap=N` bridges the stray unmarked day in an otherwise unbroken one. ## Photos Keyed on **(owner column, owner id, Drupal file id)**. A photo already present is skipped without reading, copying, downloading or resizing anything — which is what makes the command runnable every day and what keeps the articles' images from being fetched again. Photos with no `legacyId` were uploaded by hand and are invisible to the importer; photos it no longer finds in the source are **reported, not touched**. ### Where a source image comes from **The local tree first, the old site second.** A file present under `data/import/files/` is copied from there; anything else is fetched from `LegacySourceService::FILES_URL` — and **fetched into the local tree**, not into `runtime/`. So the tree is the importer's cache and fills itself in: one source image used on twelve units costs one download, and a re-run costs none. That makes the `rsync` of the previous server's `sites/default/files` (742 files, ~350 MB) **optional rather than a prerequisite**. The measured delta at go-live was 102 photos from **63 distinct files, 41 MB, 32 seconds** — cheaper than keeping 350 MB in step. What the `rsync` still buys is independence: the crawl stops working the day the old site is switched off, a local copy does not. `--localOnly=1` refuses the fallback. Nothing downloaded is trusted: both routes end in `Picture::createFromFile()`, which checks the MIME type, and `Picture::addExtraAttributes()` throws if `getimagesize()` cannot read it. An error page served with a 200 fails there rather than becoming a photo. ### Getting the files onto the server `web/contents/` is not in Git and is not deployed, so newly imported files have to be copied over by hand. **They are all in one directory.** `File::getPath()` buckets by the record's `createdAt`, and an import creates its records today, so everything it wrote is under `web/contents//`. Verified at go-live: 102 new files, all in `2026/09/07/`, and **zero** files touched anywhere else in the tree. ```shell tar -czf contents-new.tar.gz -C web/contents 2026/09 # 41 MB # on the server, from the application root: tar -xzf contents-new.tar.gz -C web/contents ``` Only the originals matter — production generates its own image versions on request ([ADR 0017](../adr/0017-image-versions-generated-on-request.md)), so a version that happens to be in the archive is a bonus, not a requirement. `import/contents-manifest` is the **verification** step rather than the transport: run it here and on the server, and `--against=` lists every file the server is missing or holds a different copy of. Use it after the upload to confirm a truncated archive did not go unnoticed. ## The workbook [`services/SpreadsheetService.php`](../../services/SpreadsheetService.php) — see [ADR 0036](../adr/0036-the-excel-round-trip-is-xlsx.md). Three sheets: **Helyszínek**, **Raktárak**, and a read-only **Segédlet** listing what every dropdown accepts. - `id` identifies the row and is locked. No id on the units sheet is a new unit; on the sites sheet it is an error. - An empty cell **clears** the value. Columns the client must not touch are read-only and never read back. - **One unreadable cell rejects the whole workbook**; the write is one transaction. - `description` is not in the workbook at all. - The location sheet carries all 38 location properties, which are **empty in the database** — utility suppliers, meter numbers, gate codes, the community charge. Collecting those is the main reason the workbook exists. [`helpers/SpreadsheetValue.php`](../../helpers/SpreadsheetValue.php) converts cells, free of the spreadsheet library so it is unit tested: `igen` / `Igen` / `1` / `yes` are all true, `99 900` and `35 900 Ft` are numbers, an option matches its label ignoring case and space. ## Traps 1. **`Storage::getValues()` memoizes every property value row in a process-wide static.** So `Storage::updateComputedProperties()`, which maintains `area` and `volume` from `StoragePropertyValue::afterSave()`, computes from a snapshot taken before the import wrote anything — and a newly created unit ends up with `area = 0`. Both importers write those two columns themselves, with `updateAll()`, after the property values. See the same trap in [CONSOLE.md](CONSOLE.md). 2. **A console run leaves the caches stale.** Every write command flushes at the end; `search-index/build` is still yours to run. 3. **`ImportDiffService` memoizes its lookups in class properties, not method statics**, precisely so `reset()` can drop them — creating a site invalidates every map keyed on one, inside the same process. `import/catalogue` calls it after the sites and after the options. 4. **A duplicate in the source is a duplicate here.** The client has three pairs of BP VAR units with the same name at the same site (nodes 649/731, 650/726, 706/711). The report lists them; nothing merges them. 5. **`import/occupancy` seeds, it does not synchronise.** It refuses to run when overrides already exist unless `--reset` clears them, because after go-live the office maintains occupancy here. See [ADR 0033](../adr/0033-occupancy-overrides-live-on-the-storage.md). 6. **Six units at BP VAR were deleted and re-created in Drupal under new ids.** The importer unpublishes the old records and creates the new ones, so the site has both until someone deletes the old ones by hand. 7. **The property values are rebuilt wholesale from the source**, which is only safe because nothing has ever edited them here — all 14 040 original rows still carry the migration's timestamp and no `updatedAt`. If that stops being true, the rule has to change. 8. **A source unit whose `helyszin` term maps nowhere is skipped, not guessed.** It shows up as *not routable* in the report; add the term to `maps.php`.