Feeds: hosts and snippets from a URL; README demo

CHANGELOG.md

@@ -29,6 +29,13 @@ prompt. Write them for users.
   versions of the app keep syncing alongside this one.
 - Install and restart waits for an update check that's still running, so it installs the update
   that check found.
+- Feeds: hosts and snippets from an address your team publishes (Hosts → Add from… → Feed URL…),
+  read with a token. They stay in step with it on their own, every 15 minutes; your renames, keys,
+  tags and notes stay yours. A feed can list each server's host key, so new servers connect
+  without asking you to trust them, and a server with another key is refused. Its snippets are
+  read-only, and when the feed changes one, Run and Paste show you the new commands first.
+- Two devices refreshing the same cloud source at once no longer make duplicate hosts or
+  "(conflict)" copies.
 
 ## 0.2.0
 

README.md

@@ -20,6 +20,10 @@ encrypted sync you can self-host.
 Docs: [developer docs](docs/README.md) · [sync & crypto spec](docs/sync-crypto-spec.md) · [self-host](https://morphgate.com/docs/self-hosting) · [roadmap](ROADMAP.md) · [releasing](RELEASING.md) ·
 [changelog](CHANGELOG.md)
 
+<p align="center">
+  <img src="assets/demo/morphgate.svg" alt="MorphGate: the Hosts page with hosts in groups; a click on web-1 opens its session, where uptime, git log and ls run">
+</p>
+
 ## Install
 
 Download from the [Releases](../../releases) page:

ROADMAP.md

@@ -40,6 +40,9 @@ xterm.js in the webview avoids writing a terminal renderer.
 - [x] Terminal appearance: built-in themes plus themes imported from Ghostty, Alacritty, iTerm2 and Windows
       Terminal files (vault items, synced), a theme per host, and this device's font, size and cursor
 - [x] Jump hosts (ProxyJump chains via direct-tcpip, per-hop host-key pinning)
+- [x] Cloud sources: hosts kept in step with Vallic Cloud (its API), AWS, Google Cloud, Azure, Hetzner Cloud and
+      DigitalOcean (their CLIs), and feeds: hosts and snippets from a JSON document a team publishes at a URL,
+      read with a token, refreshed every 15 minutes, with the host keys it lists trusted for its hosts
 - [x] Port forwarding: local / remote / dynamic SOCKS5 tunnels saved in the vault, start/stop from the sidebar
 - [x] SFTP tabs (`russh-sftp`): local + remote panes, drag and drop (also from the OS), recursive upload/download
       with progress, cancel and replace, mkdir/rename/chmod/delete

apps/desktop/e2e/README.md

@@ -4,10 +4,11 @@ These run the real AppImage, deb and rpm binaries, not a dev build, and drive th
 
 | Scenario | What it checks |
 | --- | --- |
-| `appimage.mjs` | Creates a vault; SSHes to a throwaway local sshd and runs a command; imports two hosts and their key from a `~/.ssh/config` (one behind the other as its jump host) and connects through it; imports a PuTTY session with a new key in a `.ppk` file, a Termius-style CSV file with a group and a Royal TS document, and signs in with the PuTTY key; connects Vallic Cloud through a stand-in console (`vallic-console.mjs`: device sign-in, rotating tokens, an environment that is this sshd) and signs in to its host; signs in with a key that only has a certificate from the run's user CA (which sshd trusts), from the vault and from an agent; reaches a host by a command its jump host runs (with a login command instead of the shell), and another by a proxy command on this computer (both the relay in `crates/mg-ssh/tests/relay.py`); selects two hosts, moves them into a group and deletes them; signs in to a host with one key from an `ssh-agent` (its own socket) and forwards it (`ssh-add -l` on the host lists it); splits a tab into three panes (beside and below, one through a jump host), broadcasts a command to all of them and then types into one, drags a border and closes them; opens a `morphgate://connect` link from a second launch (which hands it over and exits) and connects from the filled-in quick connect, and refuses an unknown link; refuses a tampered update twice without touching the AppImage; installs the signed update, restarts, unlocks the same vault as the new version, and finds nothing newer |
+| `appimage.mjs` | Creates a vault; SSHes to a throwaway local sshd and runs a command; imports two hosts and their key from a `~/.ssh/config` (one behind the other as its jump host) and connects through it; imports a PuTTY session with a new key in a `.ppk` file, a Termius-style CSV file with a group and a Royal TS document, and signs in with the PuTTY key; connects Vallic Cloud through a stand-in console (`vallic-console.mjs`: device sign-in, rotating tokens, an environment that is this sshd) and signs in to its host; adds AWS and DigitalOcean through stand-in CLIs (`clouds/`); adds a feed (`host-feed.mjs`) whose host connects with the host key it lists, whose wrongly listed host is refused and whose changed snippet asks before it runs; signs in with a key that only has a certificate from the run's user CA (which sshd trusts), from the vault and from an agent; reaches a host by a command its jump host runs (with a login command instead of the shell), and another by a proxy command on this computer (both the relay in `crates/mg-ssh/tests/relay.py`); selects two hosts, moves them into a group and deletes them; signs in to a host with one key from an `ssh-agent` (its own socket) and forwards it (`ssh-add -l` on the host lists it); splits a tab into three panes (beside and below, one through a jump host), broadcasts a command to all of them and then types into one, drags a border and closes them; opens a `morphgate://connect` link from a second launch (which hands it over and exits) and connects from the filled-in quick connect, and refuses an unknown link; refuses a tampered update twice without touching the AppImage; installs the signed update, restarts, unlocks the same vault as the new version, and finds nothing newer |
 | `package.mjs` (deb, rpm) | The binary knows how it was installed, finds the update, and warns that installing asks for an administrator password (installing needs root, so it stops there) |
 | `teams.mjs` | Two apps side by side (Alice and Bob) with a local reference sync server: both turn on sync; Alice creates an organization and a vault and invites Bob; Bob joins only after the owner's fingerprint matches, and Alice confirms him only after his matches; a host saved in the shared vault reaches Bob, his edit reaches Alice; Bob publishes his SSH key; Alice revokes him (the vault leaves his device) and rotates the vault key; members may create vaults; the activity log tells the story. Then Alice offboards Bob against the local sshd: removal and key rotation, his key found and removed from `authorized_keys`, and the host's shared key replaced (she still logs in) |
 | `layout.mjs` | Turns sync on from the offer a new vault gets, fills every page (hosts in a group, a key, a snippet, a tunnel, a team with an invite to a long address), and at 860, 1024, 1280 and 1600 px wide checks that nothing on a page overlaps or runs out of its column. Closing an editor returns to the page it was opened from, and closing a page to the session under it; with no session open, a page takes the tab bar's place |
+| `demo.mjs` | Only with `E2E_ONLY=demo`: the README's demo. The AppImage's Hosts page, a click on a host and its recorded session (in a demo shell), drawn by `tools/readme-demo/demo-svg.mjs` as `demo/morphgate.svg` (copy it to `assets/demo/morphgate.svg`) |
 | `mobile.mjs` | The phone layout in a 390 × 844 window: imports a key and saves a host on the phone screens, connects from the host's sheet, interrupts a command with Ctrl from the extra-key row and repeats one with ↑, keeps the session open behind the host list, browses into a folder over SFTP and downloads a file to Downloads, then locks and unlocks on the phone unlock screen |
 
 ```sh
@@ -24,10 +25,10 @@ Screenshots and logs go to `target/e2e/run-<time>/`.
   screen, and nothing reaches your session bus or keyring.
 - **Data:** `HOME` and the XDG directories point at the run directory, so your vault and settings are
   never read or written.
-- **Network:** everything is on loopback (sshd on 127.0.0.1 and 127.0.0.2), on ports free when the
+- **Network:** everything is on loopback (sshd on 127.0.0.1 to 127.0.0.4), on ports free when the
   run starts, so runs can share a machine: sshd (`E2E_SSH_PORT`), the update feed
   (`E2E_FEED_PORT`), the sync server (`E2E_SYNC_PORT`), the test identity provider
-  (`E2E_IDP_PORT`), and two per tauri-driver (`E2E_DRIVER_PORT`: from that port up). Set them to
+  (`E2E_IDP_PORT`), the stand-in feed of hosts (`E2E_HOST_FEED_PORT`), and two per tauri-driver (`E2E_DRIVER_PORT`: from that port up). Set them to
   pin a port.
 - **Files on the "server":** sshd's SFTP sessions start in the run's `sshd/home`, whose
   `.ssh/authorized_keys` is the file sshd reads, so the offboarding steps edit that copy and never

apps/desktop/e2e/appimage.mjs

@@ -622,6 +622,127 @@ try {
   await s.click(".tab.active .close");
   ok("added DigitalOcean through its CLI: its droplet became a host and connected");
 
+  // A feed (host-feed.mjs): Add from… → Feed URL…, read with its token; its host connects with
+  // the host key the feed lists (nothing asks), another listed with the wrong key is refused, an
+  // invalid entry is skipped and said; its snippet is read-only, and once the feed changes it,
+  // running it asks first.
+  await s.exec(`[...document.querySelectorAll(".sidebar-nav button")].find((b) => b.innerText.startsWith("Hosts")).click(); return true`);
+  await s.wait(() => s.exec(`return document.querySelector(".panel.page h1")?.innerText === "Hosts" || null`), "the Hosts page");
+  await s.addFrom("Feed URL…");
+  await s.waitText("Add a feed", ".cloud-adding", 30000);
+  await s.typeLabelled("Address", process.env.E2E_HOST_FEED);
+  await s.typeLabelled("Token", "wrong-token");
+  await s.clickButton("Add and read the feed");
+  await s.waitText("refused the token", ".cloud-page > .error", 30000);
+  await s.typeLabelled("Token", process.env.E2E_HOST_FEED_TOKEN);
+  await s.clickButton("Add and read the feed");
+  // Titles are uppercased by CSS (innerText follows it).
+  const feedCard = `[...document.querySelectorAll(".cloud-source")].find((x) => x.innerText.toLowerCase().includes("e2e operations"))`;
+  const feedAdded = await s
+    .wait(
+      () => s.exec(`return ${feedCard}?.innerText.includes("2 hosts added") ? "ok" : document.querySelector(".cloud-page > .error, .cloud-adding .error")?.innerText || null`),
+      "the feed source",
+      30000,
+    )
+    .catch(async (e) => {
+      await s.screenshot(`${shots}/feed-not-added.png`);
+      throw e;
+    });
+  assert.equal(feedAdded, "ok", `adding the feed: ${feedAdded}`);
+  page = await s.exec(`return ${feedCard}.innerText`);
+  assert.match(page, /1 snippet added/);
+  assert.match(page, /Skipped: broken: the host name/);
+  assert.match(page, /token …cdef/);
+  assert.ok(!page.includes(process.env.E2E_HOST_FEED_TOKEN), "the token isn't shown");
+  await s.screenshot(`${shots}/feed-source.png`);
+  await s.clickButton("Close");
+  await s.exec(
+    `[...document.querySelectorAll(".sidebar .list li")].find((l) => l.querySelector(".row-title")?.innerText === "e2e-feed-web").querySelector(".row-main").click(); return true`,
+  );
+  await s.wait(
+    () =>
+      s.exec(
+        `if (document.querySelector("dialog.hostkey[open]")) throw new Error("asked to trust the feed's host");
+         const r = document.querySelector(".terminal:not(.xterm):not([hidden]) .xterm-rows"); return r && /[$#]\\s*$/m.test(r.innerText) ? true : null`,
+      ),
+    "a shell on the feed's host, nothing asked",
+    60000,
+  );
+  await s.exec(
+    `[...document.querySelectorAll(".sidebar .list li")].find((l) => l.querySelector(".row-title")?.innerText === "e2e-feed-wrong").querySelector(".row-main").click(); return true`,
+  );
+  // The terminal wraps mid-word: compared without whitespace.
+  const refusedText = await s
+    .wait(
+      () =>
+        s.exec(
+          `const r = (document.querySelector(".terminal:not(.xterm):not([hidden]) .xterm-rows")?.innerText ?? "").replace(/\\s+/g, "");
+           return r.includes("isn'toneE2Eoperationslistsforit:refused") ? r : null`,
+        ),
+      "the wrong key refused",
+      60000,
+    )
+    .catch(async (e) => {
+      await s.screenshot(`${shots}/feed-wrong-key.png`);
+      throw e;
+    });
+  assert.ok(!refusedText.includes("$"), "no shell on the wrongly listed host");
+  await s.click(".tab.active .close");
+  // Back to the session on e2e-feed-web: where snippets go.
+  await s.exec(`[...document.querySelectorAll(".tab")].find((t) => t.querySelector(".label")?.innerText === "e2e-feed-web").click(); return true`);
+  ok("added a feed (a wrong token said first): its host connected with the key the feed lists, one listed with another key was refused, an invalid entry skipped");
+
+  // Its snippet, from the Snippets page into the session on e2e-feed-web.
+  await s.exec(`[...document.querySelectorAll(".sidebar-nav button")].find((b) => b.innerText.startsWith("Snippets")).click(); return true`);
+  await s.wait(() => s.hasText(".panel .lname", "E2E hello"), "the feed's snippet");
+  const feedRow = `[...document.querySelectorAll(".panel.page [role=row]")].find((r) => r.innerText.includes("E2E hello"))`;
+  assert.match(await s.exec(`return ${feedRow}.innerText`), /E2E operations/i);
+  const runFeedSnippet = () =>
+    s.exec(`[...${feedRow}.querySelectorAll("button")].find((b) => b.innerText === "Run").click(); return true`);
+  await runFeedSnippet();
+  await s.wait(
+    () => s.exec(`const r = document.querySelector(".terminal:not(.xterm):not([hidden]) .xterm-rows"); return r && /^e2e-feed-one\s*$/m.test(r.innerText) ? true : null`),
+    "the snippet's output",
+    30000,
+  );
+  // Read-only: no Save, a copy instead.
+  await s.exec(`[...document.querySelectorAll(".sidebar-nav button")].find((b) => b.innerText.startsWith("Snippets")).click(); return true`);
+  await s.wait(() => s.hasText(".panel .lname", "E2E hello"), "the Snippets page again");
+  await s.exec(`${feedRow}.querySelector(".lnamebtn").click(); return true`);
+  await s.waitText("it changes there, and is read-only here", ".panel", 30000);
+  assert.ok(!(await s.exec(`return [...document.querySelectorAll(".panel button")].some((b) => b.innerText.startsWith("Save"))`)), "no Save");
+  await s.clickButton("Close");
+
+  // The feed changes the snippet: a refresh brings it, and running it asks first.
+  await fetch(`${new URL(process.env.E2E_HOST_FEED).origin}/e2e/next`, { method: "POST" });
+  await s.exec(`[...document.querySelectorAll(".sidebar-nav button")].find((b) => b.innerText.startsWith("Hosts")).click(); return true`);
+  await s.wait(() => s.exec(`return document.querySelector(".panel.page h1")?.innerText === "Hosts" || null`), "the Hosts page");
+  await s.addFrom("Clouds…");
+  await s.waitText("E2E operations", ".cloud-source", 30000);
+  await s.exec(`[...${feedCard}.querySelectorAll("button")].find((b) => b.innerText === "Refresh").click(); return true`);
+  await s.wait(() => s.exec(`return ${feedCard}?.innerText.match(/1 gone from the feed, 1 snippet changed/) ? true : null`), "the feed's next version", 30000);
+  await s.clickButton("Close");
+  await s.exec(`[...document.querySelectorAll(".sidebar-nav button")].find((b) => b.innerText.startsWith("Snippets")).click(); return true`);
+  await s.wait(() => s.hasText(".panel .lname", "E2E hello"), "the Snippets page");
+  await runFeedSnippet();
+  await s.waitText("E2E hello changed", "dialog.snippet-check", 30000);
+  assert.match(await s.exec(`return document.querySelector("dialog.snippet-check").innerText`), /Changed by E2E operations[\s\S]*echo e2e-feed-two/);
+  await s.screenshot(`${shots}/feed-snippet-changed.png`);
+  await s.clickButton("Run it");
+  await s.wait(
+    () => s.exec(`const r = document.querySelector(".terminal:not(.xterm):not([hidden]) .xterm-rows"); return r && /^e2e-feed-two\s*$/m.test(r.innerText) ? true : null`),
+    "the changed snippet's output",
+    30000,
+  );
+  // Sent once as it is now: the next run doesn't ask.
+  await s.exec(`[...document.querySelectorAll(".sidebar-nav button")].find((b) => b.innerText.startsWith("Snippets")).click(); return true`);
+  await s.wait(() => s.hasText(".panel .lname", "E2E hello"), "the Snippets page");
+  await runFeedSnippet();
+  await new Promise((r) => setTimeout(r, 1000));
+  assert.ok(!(await s.exec(`return !!document.querySelector("dialog.snippet-check[open]")`)), "not asked again");
+  await s.click(".tab.active .close");
+  ok("the feed's snippet ran, read-only with a copy offered; once the feed changed it, running it asked first (and only once)");
+
   // Several hosts at once: the CSV and Royal TS ones (nothing later uses them) into one group,
   // then both deleted.
   await s.exec(`[...document.querySelectorAll(".sidebar-nav button")].find((b) => b.innerText.startsWith("Hosts")).click(); return true`);