# plan-for-hackmd-airtable-sync
## command line tool
Great—here are two **implementation plans** for your locally running Next.js app that (a) enrich every note with YAML front‑matter, and (b) push notes + metadata into Airtable. I’ll keep UI light and focus on **API routes**, **file layout**, and **Airtable interactions**.
> **Key facts we’ll rely on**
>
> * HackMD API quotas: **2,000 calls/month** (10,000 on Prime) and **100 calls / 5 minutes**. ([HackMD][1])
> * Notes list & note-content endpoints (`GET /v1/notes`, `GET /v1/notes/:noteId`). ([HackMD][2])
> * “Download all notes” is available in Settings (manual ZIP). ([HackMD][3])
> * Historical **`/download`** path returns raw Markdown for a note URL (works mainly for **public/published** links; it’s undocumented—use as best-effort fallback only). ([GitHub][4])
> * Airtable REST API rate limit: **5 req/sec per base**; bulk update/upsert up to **10 records per request**. ([Airtable][5])
> * New interactive HackMD **Swagger docs** link (for endpoint shapes). ([HackMD][6])
---
## Shared groundwork (used by both plans)
**Environment variables**
```
HACKMD_TOKEN=... # Bearer token
HACKMD_TEAM_PATH=... # optional (e.g., '@bok-center'); empty => personal
NOTES_DIR=./content/hackmd # where enriched .md files land
AIRTABLE_TOKEN=... # PAT
AIRTABLE_BASE_ID=appXXXXXXXX # your Base
AIRTABLE_TABLE=Notes # table name (or tblXXXXXXXX)
```
**Directory layout**
```
/app/api/...
/lib/hackmd.ts # API wrapper + rate limit handling
/lib/files.ts # read/write .md, hashing, safe filenames
/lib/yaml.ts # add/merge YAML via gray-matter
/lib/airtable.ts # batched upsert (10 per request)
/content/hackmd/ # enriched markdown output
/export/manual/ # (Plan 2) unzipped manual download of .md files
/state/index.json # cache: { noteId -> { lastChangedAt, sha256, path } }
```
**Airtable schema (suggested)**
Table **Notes** (primary field = `HackMD ID`):
* `HackMD ID` (single line) — **unique key**
* `Title` (single line)
* `Short ID` (single line)
* `Tags` (multiple select)
* `Created At` (date-time)
* `Last Changed At` (date-time)
* `Publish Type` (single select)
* `Read Permission` / `Write Permission` (single select)
* `Team Path` / `User Path` (single line)
* `Permalink` (URL), `Publish Link` (URL)
* `Local Path` (single line)
* `SHA256` (single line)
* `YAML` (long text)
* `Content` (long text) — optional; include if you want raw MD in Airtable
* `Status` (single select: New/Updated/Unchanged/Failed)
* `Last Sync At` (date-time)
Use **bulk upsert** (≤10 records/request) with `performUpsert` on field `HackMD ID`. ([Airtable][7])
**YAML front‑matter shape (example)**
```yaml
---
title: "<derived from content or API>"
tags: ["tag1", "tag2"]
created: "2025-01-11T15:31:56.000Z"
updated: "2025-03-07T19:14:22.000Z"
source: "hackmd"
slug: "<permalink or shortId>"
hackmd:
id: "<noteId>"
shortId: "<shortId>"
readPermission: "owner|signed_in|guest"
writePermission: "owner|signed_in|guest"
publishType: "view|slide|book"
publishedAt: "2024-... (optional)"
permalink: "<string|null>"
publishLink: "<string|null>"
teamPath: "<@team or null>"
userPath: "<userPath>"
lastChangeUserName: "<string>"
url: "https://hackmd.io/<shortId or permalink or id>"
file:
path: "content/hackmd/<safe-title>--<shortId>.md"
sha256: "<hash>"
size: 12345
apiDownloadedAt: "2025-08-27T..."
---
```
(*HackMD titles are derived from content; don’t be surprised if the first `#` heading is the title.*) ([HackMD][2])
---
# Plan 1 — **List notes via API, try `/<...>/download` first, fall back to API content**
### Flow
1. **List metadata**
* Call `GET https://api.hackmd.io/v1/notes` (and optionally `GET /v1/teams/{teamPath}/notes` if you need a team; these older “Teams” list endpoints exist but are marked deprecated—prefer Swagger docs to confirm your variant). Cache `id`, `shortId`, `title`, `tags`, `createdAt`, `lastChangedAt`, `readPermission`, `publishType`, `permalink`, `publishLink`, `teamPath`, `userPath`. ([HackMD][2])
2. **For each note, attempt a direct Markdown download**
Build a **candidate download URL**:
* If `publishLink` exists (published): use `publishLink + '/download'`.
* Else if `readPermission==='guest'` and `shortId` exists: try `https://hackmd.io/{shortId}/download` (or `https://hackmd.io/@{userOrTeam}/{shortId}/download`).
* This is **best‑effort**, not guaranteed; if 4xx/other, fall back to step 3. ([GitHub][4])
3. **Fallback to API for content**
`GET https://api.hackmd.io/v1/notes/:noteId` → `.content` (raw MD). Rate‑limit to **≤100 calls per 5 minutes**. ([HackMD][1])
4. **Enrich with YAML**
Use `gray-matter` to parse/merge. Prepend or merge the YAML block above (avoid overwriting any existing YAML; nest HackMD‑specific fields under `hackmd:`). ([GitHub][8])
5. **Write to disk**
* File path: `${NOTES_DIR}/${safeTitle}--${shortId || id}.md`
* Compute `sha256` of final file; store/update `/state/index.json` (`id -> {lastChangedAt, sha256, path}`) to avoid reprocessing unchanged notes.
6. **Upsert in Airtable**
Batch records in chunks of 10; include: `HackMD ID`, `Title`, `Short ID`, timestamps, perms, links, `Local Path`, `SHA256`, and optionally `YAML` + `Content`. Respect **5 req/sec**. ([Airtable][5])
7. **Attachments (optional)**
If you’ll need embedded images later, HackMD exposes an **Attachments API**—requests must hit `hackmd.io` (not `api.hackmd.io`) and **follow redirects**. Keep that in mind for a Phase 2. ([HackMD][9])
### Next.js routes (suggested)
* `GET /api/hackmd/list` → returns note metadata cache.
* `POST /api/hackmd/sync-plan1` → orchestrates: list → download/fallback → YAML → write → Airtable.
* `POST /api/hackmd/rebuild-yaml` → re‑merges YAML for all local files (no network).
### Minimal code skeletons (TypeScript)
**`/lib/hackmd.ts`**
```ts
// fetch with Bearer auth; simple throttling (1 call / 3s ≈ 100/5min)
const sleep = (ms:number)=>new Promise(r=>setTimeout(r,ms));
export async function listNotes() {
const res = await fetch('https://api.hackmd.io/v1/notes', {
headers: { Authorization: `Bearer ${process.env.HACKMD_TOKEN}` }
});
if (!res.ok) throw new Error(`List failed: ${res.status}`);
return res.json();
}
export async function getNoteContent(noteId: string) {
await sleep(3000);
const res = await fetch(`https://api.hackmd.io/v1/notes/${noteId}`, {
headers: { Authorization: `Bearer ${process.env.HACKMD_TOKEN}` }
});
if (!res.ok) throw new Error(`Get note failed: ${res.status}`);
const j = await res.json();
return j.content as string;
}
export function candidateDownloadUrls(n: any): string[] {
const urls: string[] = [];
if (n.publishLink) urls.push(`${n.publishLink}/download`);
if (n.readPermission === 'guest' && n.shortId) {
urls.push(`https://hackmd.io/${n.shortId}/download`);
if (n.userPath || n.teamPath) {
const owner = n.teamPath?.replace(/^@/,'') || n.userPath;
urls.push(`https://hackmd.io/@${owner}/${n.shortId}/download`);
}
}
return urls;
}
```
**`/lib/yaml.ts`**
```ts
import matter from 'gray-matter';
import crypto from 'node:crypto';
export function enrichMarkdown(rawMd: string, meta: any, outPath: string) {
const parsed = matter(rawMd);
const base = parsed.data || {};
// merge without clobbering user's keys
const merged = {
...base,
title: base.title ?? meta.title ?? parsed.content.match(/^#\s+(.*)$/m)?.[1],
tags: base.tags ?? meta.tags ?? [],
created: base.created ?? new Date(meta.createdAt).toISOString(),
updated: new Date(meta.lastChangedAt).toISOString(),
source: 'hackmd',
slug: base.slug ?? meta.permalink ?? meta.shortId ?? meta.id,
hackmd: { ...(base.hackmd || {}), ...pickHackmd(meta) },
file: { ...(base.file || {}), path: outPath },
apiDownloadedAt: new Date().toISOString()
};
return matter.stringify(parsed.content, merged);
}
function pickHackmd(m:any){ // trim to useful fields
const {id, shortId, readPermission, writePermission, publishType,
publishedAt, permalink, publishLink, teamPath, userPath,
lastChangeUser} = m;
return {
id, shortId, readPermission, writePermission, publishType,
publishedAt, permalink, publishLink, teamPath, userPath,
lastChangeUserName: lastChangeUser?.name
};
}
export function sha256(s: string){
return crypto.createHash('sha256').update(s).digest('hex');
}
```
**`/lib/airtable.ts`** (batched upsert)
```ts
const BASE = process.env.AIRTABLE_BASE_ID!;
const TABLE = process.env.AIRTABLE_TABLE!;
const API = `https://api.airtable.com/v0/${BASE}/${encodeURIComponent(TABLE)}`;
async function batchUpsert(records: any[]) {
// Up to 10 per request; respect 5 r/s
const body = {
performUpsert: { fieldsToMergeOn: ['HackMD ID'] },
records: records.map(r => ({ fields: r }))
};
const res = await fetch(API, {
method: 'PATCH',
headers: {
Authorization: `Bearer ${process.env.AIRTABLE_TOKEN}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(body)
});
if (!res.ok) throw new Error(`Airtable upsert failed ${res.status}`);
return res.json();
}
export async function upsertInChunks(rows:any[]) {
for (let i=0;i<rows.length;i+=10) {
await batchUpsert(rows.slice(i,i+10));
await new Promise(r=>setTimeout(r,220)); // ~4.5 r/s
}
}
```
**`/app/api/hackmd/sync-plan1/route.ts`** (orchestrator)
```ts
import { NextRequest, NextResponse } from 'next/server';
import fs from 'node:fs/promises';
import path from 'node:path';
import { listNotes, getNoteContent, candidateDownloadUrls } from '@/lib/hackmd';
import { enrichMarkdown, sha256 } from '@/lib/yaml';
import { upsertInChunks } from '@/lib/airtable';
export async function POST(_req: NextRequest) {
const notes = await listNotes();
const outDir = process.env.NOTES_DIR || './content/hackmd';
await fs.mkdir(outDir, { recursive: true });
const toAirtable:any[] = [];
for (const n of notes) {
let content: string | null = null;
// 1) try /download (best-effort for public notes)
for (const u of candidateDownloadUrls(n)) {
try {
const r = await fetch(u, { redirect: 'follow' });
if (r.ok) { content = await r.text(); break; }
} catch {}
}
// 2) fallback to API
if (!content) content = await getNoteContent(n.id);
// 3) write enriched MD
const safe = (s:string)=>s.replace(/[^\w\-]+/g,'-').replace(/-+/g,'-').replace(/^-|-$/g,'');
const fname = `${safe(n.title || 'untitled')}--${n.shortId || n.id}.md`;
const outPath = path.join(outDir, fname);
const md = enrichMarkdown(content, n, outPath);
await fs.writeFile(outPath, md, 'utf8');
const hash = sha256(md);
// 4) prepare Airtable row
toAirtable.push({
'HackMD ID': n.id,
'Title': n.title,
'Short ID': n.shortId,
'Tags': (n.tags || []).map((t:string)=>t),
'Created At': new Date(n.createdAt).toISOString(),
'Last Changed At': new Date(n.lastChangedAt).toISOString(),
'Publish Type': n.publishType,
'Read Permission': n.readPermission,
'Write Permission': n.writePermission,
'Team Path': n.teamPath,
'User Path': n.userPath,
'Permalink': n.permalink,
'Publish Link': n.publishLink,
'Local Path': outPath,
'SHA256': hash,
'Status': 'Updated'
});
}
await upsertInChunks(toAirtable);
return NextResponse.json({ ok: true, processed: toAirtable.length });
}
```
---
# Plan 2 — **List notes via API; pair with a folder from manual “Download all notes”; enrich + Airtable**
This minimizes per‑note API **content** calls (you still make **one** list call) and then merges metadata with files from the manual export. Manual export instructions live in HackMD Settings → **Download all notes**. ([HackMD][3])
### Flow
1. **User action**: Download the ZIP from HackMD Settings and unzip it to `${PROJECT_ROOT}/export/manual/`. (The export provides the raw `.md` for your notes; names typically reflect titles—ID presence is not guaranteed.)
2. **List metadata**
Same `GET /v1/notes` call as Plan 1; cache `id`, `shortId`, `title`, `lastChangedAt`, etc. ([HackMD][2])
3. **Pairing heuristic (file ↔ note)**
Because exported filenames may not contain IDs, pair using a cascade:
* **Exact title match** (normalized): first `# Heading` from file vs. API `title` (HackMD titles derive from content) ([HackMD][2])
* **Fuzzy title match** (Levenshtein) if needed.
* **Content prefix similarity**: compare first 300–1,000 chars of file to `GET /v1/notes/:id` **only** for the small subset still unpaired (keeps API calls low).
* **Last resort**: manual review list for any still-unmatched files (emit a JSON report).
4. **YAML enrichment + write**
For each **paired** file, read file content, merge YAML like in Plan 1 (do **not** fetch content via API unless necessary). Write the **enriched file** to `${NOTES_DIR}` with a stable filename `${safeTitle}--${shortId || id}.md`.
5. **Airtable upsert**
Same as Plan 1 (batched 10 at a time).
6. **Report**
Return a JSON payload listing: `{ pairedCount, unmatchedFiles: [...], unmatchedNotes: [...] }` so you can quickly handle edge cases.
### Next.js routes (suggested)
* `POST /api/hackmd/sync-plan2` → expects nothing; it reads the folder `${PROJECT_ROOT}/export/manual/`, lists API notes, pairs, enriches, and upserts.
* `GET /api/hackmd/pairing-report` → returns the last pairing report.
**Pairing helper (sketch)**
```ts
// /lib/pair.ts
import fs from 'node:fs/promises';
import path from 'node:path';
export async function loadManualFiles(dir:string) {
const entries = await fs.readdir(dir, { withFileTypes: true });
const files = entries.filter(e=>e.isFile() && e.name.endsWith('.md'));
return Promise.all(files.map(async f => {
const p = path.join(dir, f.name);
const text = await fs.readFile(p,'utf8');
const h1 = text.match(/^#\s+(.*)$/m)?.[1]?.trim();
return { path: p, name: f.name, h1, text };
}));
}
export function matchNotesToFiles(notes:any[], files:any[]) {
const norm = (s:string)=>s?.toLowerCase().replace(/\s+/g,' ').trim() || '';
const byTitle = new Map(files.map(f => [norm(f.h1), f]));
const pairs:any[] = [], unmatchedNotes:any[] = [], unmatchedFiles = new Set(files.map(f=>f.path));
for (const n of notes) {
const f = byTitle.get(norm(n.title));
if (f) { pairs.push({ note:n, file:f }); unmatchedFiles.delete(f.path); }
else unmatchedNotes.push(n);
}
return { pairs, unmatchedFiles: [...unmatchedFiles], unmatchedNotes };
}
```
---
## Rate limiting and durability
* **HackMD**: throttle to **≤100/5 min**; remember each **note content fetch** (Plan 1 fallback) is **one request per note**. Use your metadata list + `lastChangedAt` to skip unchanged notes on subsequent runs and stay under the **2,000 calls/month** cap. ([HackMD][1])
* **Airtable**: batch updates (10/req) and sleep \~200–250ms between requests to remain under **5 req/sec per base**. ([Airtable][5])
* **Direct `/download`**: treat as opportunistic; works best for **public/published** notes. If it fails, fall back to the API (reliable). ([GitHub][4])
---
## Nice-to-haves you can add later
* **Local index** (`/state/index.json`) to skip any `id` whose `lastChangedAt` hasn’t advanced—turns N calls into “only changed notes.”
* **Attachments pass** using HackMD Attachments API (remember to use `hackmd.io` domain and follow redirects). ([HackMD][9])
* **Team workspaces**: if you use Teams, list team notes using the relevant endpoint (see Developer Portal + Swagger to confirm the current path; older “Teams API” pages are marked deprecated). ([HackMD][6])
---
### Why two plans?
* **Plan 1** gives you a fully automated pipeline end‑to‑end, leaning on `/download` when possible to reduce API content calls, with a robust API fallback for private notes.
* **Plan 2** minimizes content API calls by reusing the **manual ZIP**, while still capturing authoritative **metadata** via the API and pushing a clean, YAML‑enriched corpus into Airtable.
If you want, I can adapt these skeletons to your exact repo layout (e.g., monorepo with `apps/web`), wire in your **pnpm** scripts, and add a tiny admin page to trigger either plan and show the pairing report.
[1]: https://hackmd.io/%40hackmd-api/api-policy?utm_source=chatgpt.com "Policy"
[2]: https://hackmd.io/%40hackmd-api/user-notes-api?utm_source=chatgpt.com "User Notes API"
[3]: https://hackmd.io/%40hackmd-blog/batch-download-all-your-notes?utm_source=chatgpt.com "Batch Download All Your Notes :package"
[4]: https://github.com/hackmdio/codimd/issues/448?utm_source=chatgpt.com "Download URL for raw Markdown file - hackmdio/codimd"
[5]: https://www.airtable.com/developers/web/api/rate-limits?utm_source=chatgpt.com "Rate limits - Airtable Web API"
[6]: https://hackmd.io/%40hackmd-api/developer-portal "HackMD Developer Portal - HackMD"
[7]: https://www.airtable.com/developers/web/api/update-multiple-records "Airtable Web API"
[8]: https://github.com/jonschlinkert/gray-matter?utm_source=chatgpt.com "GitHub - jonschlinkert/gray-matter: Smarter YAML front ..."
[9]: https://hackmd.io/%40hackmd-api/attachments-api?utm_source=chatgpt.com "User Notes API"
Sign in with Wallet
Connect another wallet