# Adapting a gallery concept to take the viewer's data

`vizs/shared/dp-import.js` lets a concept redraw itself with the viewer's own
file. The concept supplies its built-in sample (already in one of six shapes)
and a `render(data, meta)` function. The library does everything else: the
"Use your data" button, drag and drop, `?data=`, same-origin `postMessage`,
parsing CSV/TSV/JSON, validation with line-and-column errors, the "Showing your
data · Reset to sample" banner, and reset.

Nothing is ever uploaded. Files are read with the File API in the page; the only
network request the library can make is a `?data=` fetch from the same origin.

The public explainer is `import/index.html`. The reference adapter is
`vizs/calendar-earthquakes.html` (shape `daily`).

## Checklist

1. Pick the one shape that fits the concept (table below). If none fits, do not
   adapt the concept.
2. Load the library after d3 (or whatever the page already loads). From a page in
   `vizs/`: `<script src="shared/dp-import.js"></script>`. From `vizs/<dir>/`:
   `<script src="../shared/dp-import.js"></script>`. Always a relative path: the
   same tree is served at `datapoems.io/datavis/` and `dr.eamer.dev/datavis/`.
   Never use root paths like `/data/...`, and never depend on `/datavis/dev/*`
   (blocked in production).
3. Turn the concept's built-in data into the shape. Keep the generator or the
   embedded data exactly as it was; only change what it returns.
4. Turn the draw code into `render(data, meta)`. It must be safe to call many
   times: clear the previous SVG/canvas, cancel timers, `requestAnimationFrame`
   loops and simulations it started, and rebuild legends, axes and scales.
5. Replace the page's "draw once" call with `DPImport.register({...})`, at the end
   of the body (or after the concept's own async data load resolves).
6. With `meta.isSample === true`, the page must look exactly as before: same
   title, text, color domain, legend labels and source line. With user data,
   derive scales from the data, and swap out claims that no longer hold
   (title, units, "Data: USGS ..."); say "Data: your file, read in this browser
   and never uploaded." Never present the viewer's numbers under the original
   dataset's name.
7. Add `"dataShape": "<shape>"` to that concept's entry in
   `datavis/gallery/catalog.json` (only for adapted concepts). The gallery badge,
   the "Your data" filter and the list on `import/index.html` all read it.
8. Test with the samples in `import/samples/` (below) and read the screenshots.

## API

```js
DPImport.register({
  shape: 'daily',            // required: daily | panel | dayhour | weekhour | profile24 | weekly
  sample: { start: '2024-01-01', values: [...] }, // required: the concept's own data, in that shape
  render: function (data, meta) { ... },           // required: draws data; called for sample and user data
  title: 'Earthquake Calendar Heatmap',            // optional: used in the button's accessible name
  maxLength: 7320,           // optional: max days (or weeks for weekly) the concept can draw; default 36600
  position: 'bottom-left',   // optional: bottom-left (default) | bottom-right | top-left | top-right
  mount: someElement,        // optional: put the control inline in this element instead of fixed to a corner
  docsUrl: '...',            // optional: override the "Formats" link (defaults to ../../import/ from the script)
  onError: function (msg) {} // optional: told about load errors (the banner already shows them)
});
```

`register()` can be called once per page. It validates `sample` against the shape
(and throws in development if it does not match), calls `render(sample, meta)`
immediately, injects the control once the body exists, then loads `?data=` if
present. The sample you pass is normalized first, so `render` always receives
normalized data (numbers or `null`, ISO date strings).

`render(data, meta)`:

- `data`: normalized data in the registered shape (formats below).
- `meta`:
  - `isSample` (boolean): true for the built-in sample.
  - `source`: `"sample" | "file" | "drop" | "url" | "message" | "api"`.
  - `name`: file name (or the `name` from a posted message), `""` for the sample.
  - `shape`: the registered shape. `sourceShape`: the shape the file was in before
    conversion (see "Conversions").
  - `start`, `end`, `length`: ISO first and last date and number of entries
    (days, or weeks for `weekly`); `null` for `weekhour` and `profile24`.
  - `warnings`: array of strings already shown in the banner (gaps filled, sums).

If `render` throws, the library redraws what was shown before and shows the error
in the banner. Throw a plain `Error` with a short reason when data is unusable
(for example, every value is null).

Other members of `window.DPImport`: `version`, `shapes`,
`normalize(shape, input, {name, maxLength}) -> {data, meta}` (throws
`DPImport.Error` with a user-facing message), `parseText(text, name)`,
`describeShape(shape)`, `load(input, name)`, `loadUrl(url)`, `reset()`,
`current() -> {data, meta}`, and `util`:

| util | does |
| --- | --- |
| `addDays(iso, k)` | `"2024-02-28", 2` -> `"2024-03-01"` (UTC day math, DST-safe) |
| `daysBetween(a, b)` | whole days from a to b |
| `dateRange(start, n)` | array of n ISO dates |
| `weekdayIndex(iso)` | 0 = Monday ... 6 = Sunday |
| `mondayOf(iso)` | the Monday on or before the date |
| `toLocalDate(iso)` | `new Date(y, m-1, d)` at local midnight, for d3.time* scales and formats |
| `isValidDate(iso)` | true for a real `YYYY-MM-DD` |
| `weekdays` | `["Mon","Tue","Wed","Thu","Fri","Sat","Sun"]` |

Keep dates as ISO strings in your own state. Convert with `toLocalDate` only at the
point you hand a date to d3. Do not use `new Date("2024-03-07")` (that is UTC
midnight and shows as the previous day west of Greenwich).

## The six shapes (normalized form handed to `render`)

All numbers are finite numbers or `null` (no data). Never treat `null` as 0 when
drawing: show it as a gap or a neutral "no data" color. Dates are `"YYYY-MM-DD"`
strings.

### `daily`: one value per consecutive day

```json
{ "start": "2024-01-01", "values": [6200, 8104, null, 7450] }
```

`values[i]` is the day `addDays(start, i)`. Accepted input: that object; JSON rows
`[{"date": "2024-01-01", "value": 6200}, ...]`; CSV/TSV with columns `date,value`.
Missing days in rows become `null`.

### `panel`: several aligned daily series

```json
{ "start": "2024-03-01", "series": { "steps": [5519, 5360], "sleep_hours": [6.6, 9.2] } }
```

Every series has the same length; `series[name][i]` is day `addDays(start, i)`.
Accepted input: that object; `{ "start": ..., "cols": {...} }` (alias emitted by
an existing personal-data export); CSV/TSV with a `date` column plus one numeric
column per series (`date,steps,sleep_hours,coffee_cups`); JSON rows with the same
keys. Series order follows the file's column order. Shorter series are padded with
`null`.

### `dayhour`: 24 values for each consecutive day

```json
{ "start": "2024-09-02", "hours": [[3, 2, 1, 0, 0, 0, 0, 0, 4, 2, 7, 10, 11, 13, 8, 10, 8, 5, 3, 7, 9, 8, 6, 7], ...] }
```

`hours[i][h]` is hour `h` (0-23, local clock) of day `addDays(start, i)`. Every row
has exactly 24 entries. Accepted input: that object; long CSV `date,hour,value`
(hour as `0`-`23`, `7:00` or `07:00:00`). Missing cells become `null`.

### `weekhour`: a 7 x 24 weekday-by-hour grid

```json
{ "rows": ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"], "matrix": [[24 numbers], ... 7 rows] }
```

Normalized output always has `rows` in Monday-first order and `matrix[0]` is
Monday. Input `rows` may be in any order (Sunday-first is reordered) and may be
omitted (Monday-first assumed). Accepted input also: CSV `weekday,hour,value`,
weekday as `Mon`..`Sun`, full names, or ISO numbers `1` (Monday) to `7` (Sunday).
`0` is rejected as ambiguous.

### `profile24`: named hour-of-day profiles

```json
{ "series": { "weekdays": [24 numbers], "weekends": [24 numbers] } }
```

No dates. Each series has exactly 24 entries, index = hour 0-23. Accepted input:
that object (no `start` key); CSV with an `hour` column plus one column per
profile (`hour,weekdays,weekends`).

### `weekly`: one value per consecutive week

```json
{ "shape": "weekly", "start": "2023-01-02", "values": [45, 40, 39, 51] }
```

`values[i]` is the week starting `addDays(start, 7 * i)`. `start` is the first
day of the first week (any weekday; ISO `"2024-W05"` is turned into its Monday).
The JSON form needs `"shape": "weekly"`; without it, a `start` + `values` object
is daily data and a weekly concept sums it into Monday-to-Sunday weeks. Accepted
input: that object; rows/CSV `week,value` (or `date,value` when the dates are
whole weeks apart). Missing weeks become `null`.

### Common input rules (all shapes)

- JSON object, JSON array of row objects, CSV, or TSV. The format is sniffed from
  the file extension, then from the content (`{`/`[` = JSON; tab, comma or
  semicolon delimiter).
- Column names are case-insensitive. Date column: `date`, `day`, `ds`,
  `iso_date`. Hour: `hour`, `hr`, `hour_of_day`. Weekday: `weekday`, `dow`,
  `day_of_week`, `wday`. Week: `week`, `week_start`, `week_of`. Value: `value`,
  `count`, `total`, `y`, `n` (or the single remaining column).
- Dates must be `YYYY-MM-DD`; a trailing time (`T...` or ` 12:00`) is ignored.
  Impossible dates (`2024-02-30`) are rejected.
- Empty cells, `NA`, `N/A`, `NaN`, `null`, `none`, `-` mean `null`. `1,234` is
  rejected with "Remove the thousands separators."
- Lines starting with `#` above the header are comments. Extra JSON keys (such as
  `"_note"`) are ignored. A `"shape"` key, if present, must be a valid shape name;
  weekly JSON uses `"shape": "weekly"` to distinguish it from daily `start` +
  `values` data.
- Duplicate dates/cells, rows with more fields than the header, and spans longer
  than `maxLength` are errors. Every error names the field or the line and column,
  for example `Line 3, column "value": "abc" is not a number.`
- Files over 10 MB are refused.

### Conversions (done for you, reported in `meta.warnings`)

| file is | concept takes | conversion |
| --- | --- | --- |
| daily | panel | one series named `value` |
| panel (one series) | daily | that series; two or more series is an error |
| dayhour | daily, panel | daily sums |
| dayhour | weekhour | mean per weekday and hour |
| dayhour | profile24 | mean per hour, series `average` |
| weekhour | profile24 | mean of the 7 weekdays, series `average` |
| daily, panel (one series), dayhour | weekly | Monday-to-Sunday sums (`start` = a Monday) |

Anything else is an error that names both shapes.

## Load sources (all automatic once registered)

- **Button**: a real `<button>` "Use your data" plus a "Formats" link to
  `import/index.html#<shape>`, fixed bottom-left by default. Opens a file picker
  (`.json,.csv,.tsv,.txt`).
- **Drag and drop**: dropping a file anywhere on the page. An overlay shows while
  dragging.
- **`?data=`**: `page.html?data=../import/samples/daily-synthetic.json`. Relative
  or same-origin absolute URLs only; cross-origin URLs are refused without a
  request. HTML responses and non-2xx responses are errors.
- **postMessage** (same origin only; other origins are ignored silently):

  ```js
  frame.contentWindow.postMessage({
    type: 'dp-import', shape: 'daily', name: 'My steps',
    data: { start: '2024-01-01', values: [6200, 8104, 7450] }  // any accepted JSON form, or CSV text
  }, location.origin);
  // reply to the sender: { type: 'dp-import-result', ok: true, meta } or { ok: false, error }
  ```

The banner (`role="status"`, `aria-live="polite"`) reads "Showing your data:
"name", start to end." with warnings, and a "Reset to sample" button. Errors
appear in the same banner and keep whatever was drawn before. After reset, focus
returns to the "Use your data" button.

## Sample files for testing

`import/samples/` has one synthetic file per shape in JSON and CSV
(`daily-synthetic.json`, `daily-synthetic.csv`, `panel-...`, `dayhour-...`,
`weekhour-...`, `profile24-...`, `weekly-...`). They are made-up numbers, labeled
as such. Do not present them, or anything generated, as real data.

## Worked example: the reference adapter (shape `daily`)

Before (`vizs/calendar-earthquakes.html`):

```js
function generateEarthquakeData() { /* ... */ data.push({ date: date, count: finalCount }); return data; }
function createCalendarHeatmap() {
    const data = generateEarthquakeData();
    const colorScale = d3.scaleSequential(d3.interpolateYlOrRd).domain([15, 80]);
    // ... 53 week columns of 2024 ...
}
createCalendarHeatmap();
```

After:

```html
<script src="https://d3js.org/d3.v7.min.js"></script>
<script src="shared/dp-import.js"></script>
```

```js
function generateEarthquakeData() {
    const values = [];
    // ... same generator ...
    return { start: '2024-01-01', values: values };   // the daily shape
}

const SAMPLE_DOMAIN = [15, 80];                        // sample keeps its old look
const sampleText = { title: document.getElementById('title').textContent /* ... */ };

function createCalendarHeatmap(data, meta) {
    const isSample = !meta || meta.isSample;
    // 1. Page text: restore the original for the sample, neutral text for user data.
    // 2. Days from the shape, dates kept as ISO strings:
    const days = data.values.map((v, i) => {
        const iso = DPImport.util.addDays(data.start, i);
        return { iso: iso, date: DPImport.util.toLocalDate(iso), count: v };
    });
    // 3. One row of weeks per calendar year (d3.groups by year), so multi-year input stacks.
    // 4. Color domain: SAMPLE_DOMAIN for the sample, else min..max of the data
    //    (top capped at the 99th percentile when a few spikes would wash it out).
    // 5. Clear and redraw: d3.select('#chart').selectAll('*').remove(); legend too.
    // 6. null -> neutral "no data" fill (#2a2a2a), tooltip "No data".
}

DPImport.register({
    shape: 'daily',
    title: 'Earthquake Calendar Heatmap',
    sample: generateEarthquakeData(),
    render: createCalendarHeatmap,
    maxLength: 7320   // 20 years of stacked calendar rows
});
```

Catalog entry gets `"dataShape": "daily"`.

## Testing an adapter

With a browser automation tool such as Playwright:

1. Load the page with no parameters: it must look as before; no banner.
2. Button: `page.waitForEvent('filechooser')` + `page.click('text=Use your data')`,
   then `setFiles('<repo>/datavis/gallery/import/samples/<shape>-synthetic.json')`
   and the `.csv`. Banner text starts "Showing your data".
3. Drag and drop: build a `DataTransfer` in the page with a `File`, dispatch
   `dragenter`, `dragover`, `drop` on `body`.
4. `?data=../import/samples/<shape>-synthetic.csv` (adjust `../` for subfolders).
   `?data=https://example.com/x.csv` must show an error and make no request.
5. A wrong-shape file shows an error and keeps the current drawing.
6. Focus "Reset to sample" and press Enter: the sample returns exactly.
7. Record `page.on('request')`: after load, the only new requests are the
   same-origin `?data=` fetch.
8. Screenshot each state and look at it. Also check a 390 px wide viewport: the
   control must not cover the concept's own controls (use `position` or `mount`).