diff --git a/docs/.gitbook/assets/add-template.png b/docs/.gitbook/assets/add-template.png new file mode 100644 index 0000000000..4b0db148fc Binary files /dev/null and b/docs/.gitbook/assets/add-template.png differ diff --git a/docs/.gitbook/assets/ai-help-on-echarts.png b/docs/.gitbook/assets/ai-help-on-echarts.png new file mode 100644 index 0000000000..4a78d70330 Binary files /dev/null and b/docs/.gitbook/assets/ai-help-on-echarts.png differ diff --git a/docs/.gitbook/assets/ai-help-on-jS.png b/docs/.gitbook/assets/ai-help-on-jS.png new file mode 100644 index 0000000000..bdc83d5f26 Binary files /dev/null and b/docs/.gitbook/assets/ai-help-on-jS.png differ diff --git a/docs/.gitbook/assets/ai-help-query-selection.png b/docs/.gitbook/assets/ai-help-query-selection.png new file mode 100644 index 0000000000..db900a5c6f Binary files /dev/null and b/docs/.gitbook/assets/ai-help-query-selection.png differ diff --git a/docs/.gitbook/assets/automator-ai-help-httpquery.png b/docs/.gitbook/assets/automator-ai-help-httpquery.png new file mode 100644 index 0000000000..088ec0a761 Binary files /dev/null and b/docs/.gitbook/assets/automator-ai-help-httpquery.png differ diff --git a/docs/.gitbook/assets/automator-ai-help-jsquery.png b/docs/.gitbook/assets/automator-ai-help-jsquery.png new file mode 100644 index 0000000000..c7120c1f90 Binary files /dev/null and b/docs/.gitbook/assets/automator-ai-help-jsquery.png differ diff --git a/docs/.gitbook/assets/automator-app-build.png b/docs/.gitbook/assets/automator-app-build.png new file mode 100644 index 0000000000..17d3a3f7a8 Binary files /dev/null and b/docs/.gitbook/assets/automator-app-build.png differ diff --git a/docs/.gitbook/assets/automator-query-selection.png b/docs/.gitbook/assets/automator-query-selection.png new file mode 100644 index 0000000000..c4ab4ad19c Binary files /dev/null and b/docs/.gitbook/assets/automator-query-selection.png differ diff --git a/docs/.gitbook/assets/template-list.png b/docs/.gitbook/assets/template-list.png new file mode 100644 index 0000000000..93f59d0e43 Binary files /dev/null and b/docs/.gitbook/assets/template-list.png differ diff --git a/docs/.gitbook/assets/use-template.png b/docs/.gitbook/assets/use-template.png new file mode 100644 index 0000000000..c3baff15d0 Binary files /dev/null and b/docs/.gitbook/assets/use-template.png differ diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index abada2fcce..a300dd331c 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -51,6 +51,10 @@ * [Modules](build-applications/create-a-new-app/modules.md) * [Version and Release Management](build-applications/create-a-new-app/version-and-release-management.md) * [App Editor](build-applications/app-editor/README.md) + * [Automator](build-applications/app-editor/automator.md) + * [AI Help](build-applications/app-editor/ai-help.md) + * [NPM Template Plugins](build-applications/app-editor/npm-template-plugins.md) + * [Build Collaborative Video Apps](build-applications/app-editor/collaborative-video-apps.md) * [Query & Editing Roundtrips](build-applications/app-editor/query-and-editing-roundtrips.md) * [Bulk Editing](build-applications/app-editor/bulk-editing.md) * [Keyboard shortcuts](build-applications/app-editor/keyboard-shortcuts.md) @@ -106,6 +110,9 @@ * [Toggle Button](build-applications/app-editor/visual-components/data-collection-and-forms/toggle-button.md) * [Radio](build-applications/app-editor/visual-components/data-collection-and-forms/radio.md) * [Meeting & Collaboration](build-applications/app-editor/visual-components/meeting-and-collaboration.md) + * [Chat Box](build-applications/app-editor/visual-components/chat-box.md) + * [AI Chat](build-applications/app-editor/visual-components/ai-chat.md) + * [Chat Controller](build-applications/app-editor/visual-components/chat-controller.md) * [Avatar & Avatar Group](build-applications/app-editor/visual-components/meeting-and-collaboration/avatar-and-avatar-group.md) * [Comment](build-applications/app-editor/visual-components/meeting-and-collaboration/comment.md) * [Project Management](build-applications/app-editor/visual-components/project-management.md) @@ -141,6 +148,7 @@ * [Customize Styles](build-applications/themes-and-styling/customize-styles.md) * [Component Styling Possibilities](build-applications/themes-and-styling/component-styling-possibilities.md) * [Video Calls in Lowcoder](build-applications/video-calls-in-lowcoder.md) +* [Realtime Shared State and Presence](build-applications/realtime-collaboration.md) ## πŸš€ Connect your Data diff --git a/docs/build-applications/app-editor/README.md b/docs/build-applications/app-editor/README.md index c08cf4b69b..adc3c70139 100644 --- a/docs/build-applications/app-editor/README.md +++ b/docs/build-applications/app-editor/README.md @@ -68,4 +68,6 @@ The query editor is flexible and adapts the options to build queries to the data The component and property pane (red) is located on the right of the window. Drag components onto the canvas from the **Insert** tab and edit the properties of the components in the **Properties** tab. +Under the **Insert** tab, switch to **Extensions** to access **Modules**, **Templates** (npm layout packages), and **Plugins**. See [NPM Template Plugins](./npm-template-plugins.md). + When a component is selected on the canvas, the **Properties** tab will be activated automatically and display the properties of that component. diff --git a/docs/build-applications/app-editor/ai-help.md b/docs/build-applications/app-editor/ai-help.md new file mode 100644 index 0000000000..6cb300c5bb --- /dev/null +++ b/docs/build-applications/app-editor/ai-help.md @@ -0,0 +1,466 @@ +# AI Help + +AI Help is an AI-assisted editor feature for focused help inside Lowcoder input fields, such as JavaScript, SQL, JSON, and chart option editors. + +Supported editor fields show an AI Help action that opens the AI Helper panel. This field-level helper currently does not require an AI Robot subscription. The AI Robot subscription applies to [Lowcoder Automator](automator.md), which creates and edits apps. Model-provider usage is billed separately under your provider's terms. + +## Bring Your Own Model + +Lowcoder does not require a specific AI model for AI Help. + +You can connect any model or provider of your choice through a Lowcoder query. This can be an OpenAI-compatible API, a private model gateway, a self-hosted model, or another LLM service. The selected query sends AI Help's request to your provider and converts the provider response into the message format Lowcoder expects. + +This lets each workspace choose its own model, endpoint, credentials, data policy, and cost profile. + +For the best AI Help experience, choose a model that can follow structured instructions and supports tool or function calling. AI Help can show a normal text explanation without a tool call, but the "apply" action appears only when the model returns the AI Help tool call. + +Examples of possible providers include: + +- OpenAI or any OpenAI-compatible API +- xAI Grok through an OpenAI-compatible chat completions endpoint +- Ollama with a local model that supports tools +- Anthropic Claude through a small adapter in the JavaScript bridge query +- your own backend that accepts Lowcoder's `messages`, `tools`, and `target` and returns the assistant message shape + +## How AI Help Works + +When you ask for help in a supported editor field, Lowcoder builds an AI payload and passes it to the selected query as: + +```js +ai.value +``` + +The payload contains: + +```js +{ + mode: "helper", + messages: [...], + tools: [...], + target: {...} +} +``` + +Inside the selected JavaScript query, read this request directly as `ai.value`, not `args.ai.value`. + +The `messages` include the helper system prompt and the conversation. The `tools` describe how the model can offer a result that Lowcoder can apply. The `target` describes the current field, for example whether it is SQL, JavaScript, JSON, an ECharts option, or another supported input. + +AI Help does not call the model directly. The selected query is the integration layer between Lowcoder and your AI provider. + +## Where AI Help Appears + +AI Help is shown inside supported Lowcoder code editors and structured input editors. In the current editor flow, this includes: + +- JavaScript queries +- transformers +- temporary state values +- app-level Scripts and Styles, including JavaScript, CSS, and Global CSS +- SQL query editors +- HTTP and Server-Sent Events request bodies +- GraphQL query editors +- MongoDB query fields +- Redis command editors +- Elasticsearch query editors +- AlaSQL query editors +- JSON editor and JSON explorer components +- ECharts option JSON for charts +- Table and Table Lite data JSON +- option-list data, such as Select, Multi Select, Radio, Checkbox, and similar option components +- Tree component data JSON +- JSON Schema Form schema, UI schema, and form data JSON +- JavaScript event-handler scripts, such as Run JavaScript actions + +When AI Help opens, it receives context for the exact field where it was opened. For example, in a JavaScript query it receives a JavaScript-focused target, and in an ECharts option field it receives an ECharts JSON-focused target. + +## Recommended Query Setup + +A common setup is to create two queries: + +1. an HTTP query that calls your model provider +2. a JavaScript query that acts as the bridge between AI Help and the HTTP query + +The same pair can serve both AI Help and Automator. Both features call the selected JavaScript query with `ai.value`; `ai.value.mode` identifies the caller as `"helper"` or `"automator"`. + +```text +AI Help ----\ + > shared JavaScript bridge -> provider HTTP query +Automator --/ +``` + +The JavaScript bridge is the query you select in the AI Helper and Automator panels. The provider-specific request and response conversion belongs in this bridge, not in either feature. + +This pattern is useful because the AI Helper UI does not need to know which model provider you use. The HTTP query handles the provider call. The JavaScript bridge normalizes the provider response for Lowcoder. + +Before you start, prepare: + +- a model endpoint +- any required API key or authentication headers +- a model name +- a Lowcoder HTTP query, for example `openAIResponses` or `llmHttp` +- a Lowcoder JavaScript query, for example `unifiedAIQuery` + +The OpenAI Responses example below uses `openAIResponses` and `unifiedAIQuery`. Other providers can use the same Lowcoder payload contract with a provider-specific HTTP query and bridge conversion. + +## HTTP Query + +Create an HTTP query that points to your model provider endpoint. Its URL, authentication, and body are provider-specific; the JavaScript bridge supplies the converted request values when it runs the HTTP query. For expressions such as `{{ instructions.value }}`, pass `{ instructions: { value: text } }` to `.run(...)`. These are runtime request values, so the query’s Variables list can remain empty. Run the JavaScript bridge from Automator or AI Help; running the HTTP query on its own does not supply these values. + +### OpenAI Responses API + +This is a tested provider example, not a requirement to use OpenAI. + +Create a REST query named `openAIResponses`: + +- **Triggered when:** manually +- **Method:** `POST` +- **URL:** `https://api.openai.com/v1/responses` +- **Parameters:** none +- **Variables:** empty +- **Body type:** JSON + +Headers: + +| Key | Value | +|---|---| +| `Authorization` | `Bearer YOUR_OPENAI_API_KEY` | +| `Content-Type` | `application/json` | + +Body: + +```json +{ + "model": "gpt-4.1-mini", + "instructions": {{ instructions.value }}, + "input": {{ input.value }}, + "tools": {{ tools.value }}, + "tool_choice": "auto", + "parallel_tool_calls": false +} +``` + +The bridge must flatten Lowcoder's `tool.function` objects for the Responses API and convert returned `function_call` items back into Lowcoder `tool-call` message parts. See the [OpenAI Responses API reference](https://developers.openai.com/api/reference/cli/resources/responses/methods/create). + +## Provider Starting Points + +Many providers can use the same HTTP body when they support the OpenAI chat completions format. + +### OpenAI-Compatible APIs + +Use this style for OpenAI and other providers that expose a compatible `/chat/completions` endpoint: + +```json +{ + "model": "your-model-name", + "stream": false, + "parallel_tool_calls": false, + "messages": {{ messages.value }}, + "tools": {{ tools.value }} +} +``` + +Use the OpenAI-compatible Chat Completions bridge shown later on this page. It reads the response from: + +```js +response.choices?.[0]?.message +``` + +### Grok + +Grok can be used in the same way when you call an OpenAI-compatible xAI chat completions endpoint. + +Use your xAI endpoint and API key in the HTTP query, then set the model to the Grok model you want to use: + +```json +{ + "model": "grok-...", + "stream": false, + "messages": {{ messages.value }}, + "tools": {{ tools.value }} +} +``` + +If the response follows the OpenAI-compatible shape, no change is needed in the JavaScript bridge. + +### Ollama + +Ollama can also be used when your Lowcoder instance can reach the Ollama server and the selected local model supports tools. + +For a local setup, the HTTP query usually points to an Ollama OpenAI-compatible endpoint such as: + +```text +http://localhost:11434/v1/chat/completions +``` + +Then use a local model name: + +```json +{ + "model": "llama3.1", + "stream": false, + "messages": {{ messages.value }}, + "tools": {{ tools.value }} +} +``` + +In hosted or containerized Lowcoder environments, `localhost` means the Lowcoder container or server, not your laptop. Use a reachable network hostname for Ollama in that case. + +### Claude + +Claude can be used too, but it does not use the exact same request and response shape as OpenAI-compatible APIs. Keep the same AI Help payload, but adapt it in the JavaScript bridge. + +For Claude, the bridge usually needs to: + +- move the system message into Claude's top-level `system` field +- convert OpenAI-style tools into Claude tools +- read Claude `text` and `tool_use` response blocks +- return Lowcoder's assistant message shape + +A simplified bridge shape looks like this: + +```js +const a = ai.value; +const system = a.messages.find((m) => m.role === "system")?.content || ""; +const messages = a.messages.filter((m) => m.role !== "system"); +const tools = a.tools.map((tool) => ({ + name: tool.function.name, + description: tool.function.description, + input_schema: tool.function.parameters, +})); + +return claudeHttp + .run({ system: { value: system }, messages: { value: messages }, tools: { value: tools } }) + .then((response) => { + const blocks = response.content || []; + const text = blocks + .filter((block) => block.type === "text") + .map((block) => block.text) + .join("\n"); + const toolUse = blocks.find((block) => block.type === "tool_use"); + + if (!toolUse) { + return { + role: "assistant", + content: text ? [{ type: "text", text }] : [], + }; + } + + const argsText = JSON.stringify(toolUse.input || {}); + + return { + role: "assistant", + content: [ + ...(text ? [{ type: "text", text }] : []), + { + type: "tool-call", + toolCallId: toolUse.id, + toolName: toolUse.name, + args: toolUse.input || {}, + argsText, + }, + ], + }; + }); +``` + +The exact HTTP query body depends on the Claude API version and the model you select. + +## JavaScript Bridge Query + +Create a JavaScript query that receives `ai.value`, calls the provider HTTP query, and returns a Lowcoder assistant message. Select the same bridge for AI Help and Automator when they use the same provider. + +### OpenAI Responses bridge + +Create a JavaScript query named `unifiedAIQuery`: + +```js +const request = ai.value; + +const systemMessage = request.messages.find( + (message) => message.role === "system" +); + +const input = request.messages + .filter((message) => message.role !== "system") + .map((message) => ({ + role: message.role, + content: message.content + })); + +const tools = request.tools.map((tool) => ({ + type: "function", + name: tool.function.name, + description: tool.function.description, + parameters: tool.function.parameters +})); + +return openAIResponses + .run({ + instructions: { value: systemMessage?.content || "" }, + input: { value: input }, + tools: { value: tools } + }) + .then((response) => { + const output = response.output || []; + const assistantText = output + .filter( + (item) => + item.type === "message" && + item.role === "assistant" + ) + .flatMap((item) => item.content || []) + .filter((part) => part.type === "output_text") + .map((part) => part.text) + .join("\n"); + + const content = assistantText + ? [{ type: "text", text: assistantText }] + : []; + + output + .filter((item) => item.type === "function_call") + .forEach((functionCall) => { + const argsText = functionCall.arguments || "{}"; + content.push({ + type: "tool-call", + toolCallId: functionCall.call_id, + toolName: functionCall.name, + args: JSON.parse(argsText), + argsText + }); + }); + + if (content.length === 0) { + throw new Error("OpenAI returned no text or function call"); + } + + return { + role: "assistant", + content + }; + }); +``` + +This single bridge handles both tool names because they arrive through `request.tools`: AI Help supplies `apply_ai_helper_result`, while Automator supplies `execute_automator_actions`. + +### OpenAI-compatible Chat Completions bridge + +For OpenAI-compatible Chat Completions providers, including compatible Grok and Ollama endpoints, use a bridge that preserves the nested tool definitions and reads `choices[0].message`: + +```js +const a = ai.value; + +return llmHttp + .run({ + messages: { value: a.messages }, + tools: { value: a.tools }, + }) + .then((response) => { + const msg = response.choices?.[0]?.message || {}; + const content = msg.content || ""; + const toolCall = msg.tool_calls?.[0]; + + if (!toolCall) { + return { + role: "assistant", + content: content ? [{ type: "text", text: content }] : [], + }; + } + + const argsText = toolCall.function?.arguments || "{}"; + const args = JSON.parse(argsText); + + return { + role: "assistant", + content: [ + ...(content ? [{ type: "text", text: content }] : []), + { + type: "tool-call", + toolCallId: toolCall.id, + toolName: toolCall.function.name, + args, + argsText, + }, + ], + }; + }); +``` + +This bridge can also be shared with Automator when both features use the same provider and response format. + +

JavaScript bridge query that forwards AI Help payloads to the HTTP query and normalizes the response.

+ +## Select the Query + +In the App Editor: + +1. open AI Help from a supported editor field +2. choose the JavaScript bridge query, such as `unifiedAIQuery`, in the AI query selector +3. ask for help, an explanation, or a generated replacement value +4. apply the suggestion if AI Help returns an apply action + +AI Help and Automator use the same saved AI-query preference. Selecting the shared bridge in either panel makes it the default when the other panel is opened. + +AI Help keeps the conversation focused on the field where it was opened. For example, in a SQL editor it should help write or improve SQL. In a JSON field it should return valid JSON. + +

Select the JavaScript bridge query in the AI Helper panel.

+ +## Example Prompts + +AI Help is designed for focused field-level work. Useful prompts include: + +- "Explain this JavaScript query." +- "Rewrite this SQL query to filter by the selected customer." +- "Generate valid ECharts option JSON for a bar chart." +- "Find the syntax error in this expression." +- "Replace this value with a safer version." + +When AI Help returns an apply action, Lowcoder can insert, append, or replace the current field value depending on the tool arguments returned by the model. + +

AI Help can generate and apply ECharts option JSON for chart configuration.

+ +

AI Help can generate, replace, or explain JavaScript query code.

+ +## Response Contract + +The selected query must return an assistant message: + +```js +{ + role: "assistant", + content: [ + { type: "text", text: "Here is a safer query..." } + ] +} +``` + +When the model offers a value that Lowcoder can apply to the current field, include the tool call part returned by the model: + +```js +{ + role: "assistant", + content: [ + { + type: "tool-call", + toolCallId: "call_123", + toolName: "apply_ai_helper_result", + args: { + label: "Replace SQL", + value: "select * from orders limit 100", + mode: "replace", + language: "sql" + }, + argsText: "{\"label\":\"Replace SQL\",\"value\":\"select * from orders limit 100\",\"mode\":\"replace\",\"language\":\"sql\"}" + } + ] +} +``` + +The exact tool arguments are generated from the `tools` definition passed to your model. + +## Notes + +- AI Help currently does not require AI Robot; the AI Robot subscription gates Automator. +- You can use any model that can accept the provided messages and tools, or any backend that can translate them. +- Keep provider API keys in datasource or query configuration, not directly in app-visible code. +- If AI Help explains an answer but does not show an apply action, check whether the model returned a tool call. +- If the query fails, test the HTTP query first, then test the JavaScript bridge query. diff --git a/docs/build-applications/app-editor/automator.md b/docs/build-applications/app-editor/automator.md new file mode 100644 index 0000000000..b176c317c4 --- /dev/null +++ b/docs/build-applications/app-editor/automator.md @@ -0,0 +1,539 @@ +# Automator + +Lowcoder Automator is an AI-assisted editor feature for creating and changing parts of an app from natural language instructions. + +Automator requires an active **AI Robot** subscription for the current workspace. A workspace admin subscribes in **Settings β†’ Subscriptions**. All admins and editors in that workspace are billable seats and can use Automator; app viewers are not billed. Each workspace needs its own subscription. + +Without an active subscription, the Automator panel shows a subscription prompt. With access, you can select the Lowcoder query that should handle AI requests. + +## AI Robot Subscription + +AI Robot currently uses the same monthly EUR prices as Lowcoder Support & SLA. These are separate subscriptions: Support does not unlock Automator. **Stripe is the source of truth for prices.** The table below reflects the live Stripe catalogue verified on 10 October 2026; prices exclude tax. + +| Admins and editors in the workspace | Monthly price per seat | +|---|---| +| 1–10 | €3.49 | +| 11–100 | €2.90 | +| 101–500 | €1.90 | +| 501–1,000 | €0.90 | +| 1,001–5,000 | €0.49 | +| 5,001+ | €0.39 | + +Stripe uses **volume pricing**: the total seat count selects one rate, and that rate applies to every seat. For example, 11 seats cost 11 Γ— €2.90 = €31.90 per month before tax. + +The seat count is set at checkout. On installations with seat synchronization enabled, changes to admins and editors are synchronized automatically while the workspace is open. Changes made while it is closed are reconciled on the next visit. Additions and removals are prorated on the next invoice. Viewers never count as billable seats. + +Seat synchronization requires deployment of the frontend and API changes and configuration of the private n8n relay. See the [server setup guide](https://github.com/lowcoder-org/lowcoder/blob/main/server/api-service/README.md#subscription-seat-synchronization). + +AI Robot provides access to Automator. Model-provider usage is billed separately under your provider's terms. + +## Guided setup in the editor + +Open the **Automator** panel, open **Connection** in its header, and choose **Set up AI connection**. If no query is selected (or the saved query is missing from this app), Automator shows the setup entry point instead of an empty chat. + +1. **Connect.** Choose the OpenAI example and enter your API key and model name, or choose your own provider / self-hosted model. You can also reuse an existing REST datasource. The model must support function/tool calling. +2. **Create queries.** Review the connection. The wizard saves a workspace REST datasource and adds a provider query plus a JavaScript bridge to this app, using available names. Existing queries are preserved, and both new queries run only when manually triggered. The JavaScript bridge (`automatorAI1`, or the next available name) is automatically selected for Automator; it supplies values to the provider HTTP query (`automatorHttp1`) and converts the reply. +3. **Try it.** Select **Test connection**. This makes one real provider request using a harmless `check_connection` tool. It sends no app context and does not apply canvas changes. Provider charges may apply. A text-only reply does not verify tool support. +4. After verification, select **Start building**. The bridge is selected for you. In a blank app, try: β€œCreate a simple to-do app with a task table and an add button.” Review the result before publishing. + +The API key is stored in the REST datasource's authentication header, outside the app's JavaScript. Datasource management permissions control who can manage that configuration. Do not embed keys in prompts or bridge scripts. + +If the connection test fails, the datasource and queries remain available. Fix the connection and retry without creating another pair. **Manage model connection** opens the datasource settings. If you finish later, the generated JavaScript bridge remains selected. Keep both queries. Returning to the browser may refresh subscription access, but the setup stays open and keeps your entries while that check completes. + +### Self-hosted models and private gateways + +Select **My own provider / self-hosted**, then supply the base URL, exact model name and optional API key. Choose **OpenAI-compatible Chat Completions** or **OpenAI Responses** to match the server. The API path is appended to the datasource URL: a base URL ending in `/v1` normally uses `/chat/completions` or `/responses`, rather than another `/v1` segment. + +The endpoint must be reachable from the Lowcoder server. In Docker, `localhost` is the container itself, not your laptop. Use the intended private network address; do not expose a local model publicly just to connect it. For custom authentication or extra headers, configure a REST datasource first, then select it in the wizard. + +Other API formats need a custom bridge. The examples below explain `ai.value`, provider requests and the expected assistant response. The generated Responses bridge explicitly uses `strict: false` because Automator's action parameters contain flexible property objects; see the [OpenAI function-calling guide](https://developers.openai.com/api/docs/guides/function-calling). + +### If setup does not work + +- **401 / authentication error:** check the datasource API key and authentication headers. +- **429 / quota or rate limit:** check the provider account's billing, quota and rate limits. +- **Connection or timeout error:** check the URL and reachability from the Lowcoder server. +- **Text reply, no verified tool call:** choose a model with tool support and check the selected API format. +- **No app changes after a real request:** inspect whether the model returned the supported Automator tool call. A text reply alone does not edit the canvas. + +Once connected, normal Automator requests send conversation and app context to your chosen provider and apply supported edits directly in the editor. + +## Bring Your Own Model + +Lowcoder does not require a specific AI model for Automator. + +You can use any model or provider of your choice, for example OpenAI-compatible APIs, self-hosted models, private gateways, or other LLM services. The selected Lowcoder query is responsible for sending Automator's request to your model and converting the model response back into the message format Lowcoder expects. + +This keeps the model, API key, endpoint, and provider configuration under your control. + +For Automator, choose a model that supports tool or function calling. Automator can still show a normal text reply without tool calls, but app changes are applied only when the model returns the Automator tool call. + +Examples of possible providers include: + +- OpenAI or any OpenAI-compatible API +- xAI Grok through an OpenAI-compatible chat completions endpoint +- Ollama with a local model that supports tools +- Anthropic Claude through a small adapter in the JavaScript bridge query +- your own backend that accepts Lowcoder's `messages` and `tools` and returns the assistant message shape + +## How Automator Works + +At runtime, Automator builds an AI payload from the current editor state and conversation. It passes this payload to the selected query as: + +```js +ai.value +``` + +The payload contains: + +```js +{ + mode: "automator", + messages: [...], + tools: [...] +} +``` + +Inside the selected JavaScript query, read this request directly as `ai.value`, not `args.ai.value`. + +The query should send `messages` and `tools` to your model. If the model returns a normal assistant reply, the query returns that text to Automator. If the model calls an Automator tool, the query returns the tool call so Lowcoder can apply the generated actions in the editor. + +Automator does not call the model directly. The selected query is the integration layer between Lowcoder and your AI provider. + +## Recommended Query Setup + +A common setup is to create two queries: + +1. an HTTP query that calls your model provider +2. a JavaScript query that acts as the bridge between Automator and the HTTP query + +The same pair can serve both Automator and AI Help. Both features call the selected JavaScript query with `ai.value`; `ai.value.mode` identifies the caller as `"automator"` or `"helper"`. + +```text +Automator --\ + > shared JavaScript bridge -> provider HTTP query +AI Help ----/ +``` + +The JavaScript bridge is the query you select in the Automator and AI Helper panels. The provider-specific request and response conversion belongs in this bridge, not in either feature. + +This pattern is useful because the Automator UI does not need to know which model provider you use. The HTTP query handles the provider call. The JavaScript bridge normalizes the provider response for Lowcoder. + +Before you start, prepare: + +- a model endpoint +- any required API key or authentication headers +- a model name +- a Lowcoder HTTP query, for example `openAIResponses` or `llmHttp` +- a Lowcoder JavaScript query, for example `unifiedAIQuery` + +The OpenAI Responses example below uses `openAIResponses` and `unifiedAIQuery`. Other providers can use the same Lowcoder payload contract with a provider-specific HTTP query and bridge conversion. + +## HTTP Query + +Create an HTTP query that points to your model provider endpoint. Its URL, authentication, and body are provider-specific; the JavaScript bridge supplies the converted request values when it runs the HTTP query. For expressions such as `{{ instructions.value }}`, pass `{ instructions: { value: text } }` to `.run(...)`. These are runtime request values, so the query’s Variables list can remain empty. Run the JavaScript bridge from Automator or AI Help; running the HTTP query on its own does not supply these values. + +### OpenAI Responses API + +This is a tested provider example, not a requirement to use OpenAI. + +Create a REST query named `openAIResponses`: + +- **Triggered when:** manually +- **Method:** `POST` +- **URL:** `https://api.openai.com/v1/responses` +- **Parameters:** none +- **Variables:** empty +- **Body type:** JSON + +Headers: + +| Key | Value | +|---|---| +| `Authorization` | `Bearer YOUR_OPENAI_API_KEY` | +| `Content-Type` | `application/json` | + +Body: + +```json +{ + "model": "gpt-4.1-mini", + "instructions": {{ instructions.value }}, + "input": {{ input.value }}, + "tools": {{ tools.value }}, + "tool_choice": "auto", + "parallel_tool_calls": false +} +``` + +The bridge must flatten Lowcoder's `tool.function` objects for the Responses API and convert returned `function_call` items back into Lowcoder `tool-call` message parts. See the [OpenAI Responses API reference](https://developers.openai.com/api/reference/cli/resources/responses/methods/create). + +## Provider Starting Points + +Many providers can use the same HTTP body when they support the OpenAI chat completions format. + +### OpenAI-Compatible APIs + +Use this style for OpenAI and other providers that expose a compatible `/chat/completions` endpoint: + +```json +{ + "model": "your-model-name", + "stream": false, + "parallel_tool_calls": false, + "messages": {{ messages.value }}, + "tools": {{ tools.value }} +} +``` + +Use the OpenAI-compatible Chat Completions bridge shown later on this page. It reads the response from: + +```js +response.choices?.[0]?.message +``` + +### Grok + +Grok can be used in the same way when you call an OpenAI-compatible xAI chat completions endpoint. + +Use your xAI endpoint and API key in the HTTP query, then set the model to the Grok model you want to use: + +```json +{ + "model": "grok-...", + "stream": false, + "messages": {{ messages.value }}, + "tools": {{ tools.value }} +} +``` + +If the response follows the OpenAI-compatible shape, no change is needed in the JavaScript bridge. + +### Ollama + +Ollama can also be used when your Lowcoder instance can reach the Ollama server and the selected local model supports tools. + +For a local setup, the HTTP query usually points to an Ollama OpenAI-compatible endpoint such as: + +```text +http://localhost:11434/v1/chat/completions +``` + +Then use a local model name: + +```json +{ + "model": "llama3.1", + "stream": false, + "messages": {{ messages.value }}, + "tools": {{ tools.value }} +} +``` + +In hosted or containerized Lowcoder environments, `localhost` means the Lowcoder container or server, not your laptop. Use a reachable network hostname for Ollama in that case. + +### Claude + +Claude can be used too, but it does not use the exact same request and response shape as OpenAI-compatible APIs. Keep the same Automator payload, but adapt it in the JavaScript bridge. + +For Claude, the bridge usually needs to: + +- move the system message into Claude's top-level `system` field +- convert OpenAI-style tools into Claude tools +- read Claude `text` and `tool_use` response blocks +- return Lowcoder's assistant message shape + +A simplified bridge shape looks like this: + +```js +const a = ai.value; +const system = a.messages.find((m) => m.role === "system")?.content || ""; +const messages = a.messages.filter((m) => m.role !== "system"); +const tools = a.tools.map((tool) => ({ + name: tool.function.name, + description: tool.function.description, + input_schema: tool.function.parameters, +})); + +return claudeHttp + .run({ system: { value: system }, messages: { value: messages }, tools: { value: tools } }) + .then((response) => { + const blocks = response.content || []; + const text = blocks + .filter((block) => block.type === "text") + .map((block) => block.text) + .join("\n"); + const toolUse = blocks.find((block) => block.type === "tool_use"); + + if (!toolUse) { + return { + role: "assistant", + content: text ? [{ type: "text", text }] : [], + }; + } + + const argsText = JSON.stringify(toolUse.input || {}); + + return { + role: "assistant", + content: [ + ...(text ? [{ type: "text", text }] : []), + { + type: "tool-call", + toolCallId: toolUse.id, + toolName: toolUse.name, + args: toolUse.input || {}, + argsText, + }, + ], + }; + }); +``` + +The exact HTTP query body depends on the Claude API version and the model you select. + +## JavaScript Bridge Query + +Create a JavaScript query that receives `ai.value`, calls the provider HTTP query, and returns a Lowcoder assistant message. Select the same bridge for Automator and AI Help when they use the same provider. + +### OpenAI Responses bridge + +Create a JavaScript query named `unifiedAIQuery`: + +```js +const request = ai.value; + +const systemMessage = request.messages.find( + (message) => message.role === "system" +); + +const input = request.messages + .filter((message) => message.role !== "system") + .map((message) => ({ + role: message.role, + content: message.content + })); + +const tools = request.tools.map((tool) => ({ + type: "function", + name: tool.function.name, + description: tool.function.description, + parameters: tool.function.parameters +})); + +return openAIResponses + .run({ + instructions: { value: systemMessage?.content || "" }, + input: { value: input }, + tools: { value: tools } + }) + .then((response) => { + const output = response.output || []; + const assistantText = output + .filter( + (item) => + item.type === "message" && + item.role === "assistant" + ) + .flatMap((item) => item.content || []) + .filter((part) => part.type === "output_text") + .map((part) => part.text) + .join("\n"); + + const content = assistantText + ? [{ type: "text", text: assistantText }] + : []; + + output + .filter((item) => item.type === "function_call") + .forEach((functionCall) => { + const argsText = functionCall.arguments || "{}"; + content.push({ + type: "tool-call", + toolCallId: functionCall.call_id, + toolName: functionCall.name, + args: JSON.parse(argsText), + argsText + }); + }); + + if (content.length === 0) { + throw new Error("OpenAI returned no text or function call"); + } + + return { + role: "assistant", + content + }; + }); +``` + +This single bridge handles both tool names because they arrive through `request.tools`: Automator supplies `execute_automator_actions`, while AI Help supplies `apply_ai_helper_result`. + +### OpenAI-compatible Chat Completions bridge + +For OpenAI-compatible Chat Completions providers, including compatible Grok and Ollama endpoints, use a bridge that preserves the nested tool definitions and reads `choices[0].message`: + +```js +const a = ai.value; + +return llmHttp + .run({ + messages: { value: a.messages }, + tools: { value: a.tools }, + }) + .then((response) => { + const msg = response.choices?.[0]?.message || {}; + const content = msg.content || ""; + const toolCall = msg.tool_calls?.[0]; + + if (!toolCall) { + return { + role: "assistant", + content: content ? [{ type: "text", text: content }] : [], + }; + } + + const argsText = toolCall.function?.arguments || "{}"; + const args = JSON.parse(argsText); + + return { + role: "assistant", + content: [ + ...(content ? [{ type: "text", text: content }] : []), + { + type: "tool-call", + toolCallId: toolCall.id, + toolName: toolCall.function.name, + args, + argsText, + }, + ], + }; + }); +``` + +Both bridge examples keep Automator independent from a specific provider. If your provider returns a different response shape, update only the JavaScript bridge. + +

JavaScript bridge query that forwards Automator payloads to the HTTP query and normalizes the response.

+ +## Select the Query + +In the App Editor: + +1. open the Automator panel +2. choose the JavaScript bridge query, such as `unifiedAIQuery`, in the query selector +3. send an instruction, such as creating components, adjusting layout, or modifying supported properties + +Automator and AI Help use the same saved AI-query preference. Selecting the shared bridge in either panel makes it the default when the other panel is opened. + +Automator will run the selected query and apply supported tool calls returned by the model. + +

Select the JavaScript bridge query in the Automator panel.

+ +## Example Instructions + +After selecting the bridge query, you can ask Automator for app-editor changes such as: + +- "Create a customer form with name, email, company, and submit button." +- "Add a table for orders and place it below the filters." +- "Change the selected button text to Save changes." +- "Create a simple dashboard layout with a chart and two summary cards." + +The model can only apply actions that are available in the tools passed by Lowcoder. If a request is outside the supported action set, the assistant should explain what it can do instead. + +

Automator can apply generated tool calls to build or update the app canvas.

+ +## Response Contract + +The selected query must return an assistant message: + +```js +{ + role: "assistant", + content: [ + { type: "text", text: "Done." } + ] +} +``` + +When the model calls an Automator tool, include the tool call part: + +```js +{ + role: "assistant", + content: [ + { + type: "tool-call", + toolCallId: "call_123", + toolName: "execute_automator_actions", + args: { + explanation: "No canvas changes are required.", + actions: [] + }, + argsText: "{\"explanation\":\"No canvas changes are required.\",\"actions\":[]}" + } + ] +} +``` + +The exact tool arguments are generated from the `tools` definition passed to your model. + +## Notes + +- Automator requires an active AI Robot subscription for the current workspace and admin or editor access. +- You can use any model that can accept the provided messages and tools, or any backend that can translate them. +- Keep provider API keys in datasource or query configuration, not directly in app-visible code. +- If Automator replies with text but does not change the app, check whether the model returned a tool call. +- If the query fails, test the HTTP query first, then test the JavaScript bridge query. + + +## Build with JSON, with or without an LLM + +Automator executes an ordered JSON recipe. An LLM is one way to write that recipe; +a JavaScript query can produce the same actions with loops, conditions, or your own data. +This is an editor automation interface, so the resulting components remain editable in Lowcoder. + +Open **Build with JSON** in the Automator header for a copyable example, the supported +action names, and a JavaScript wrapper. After a build, **See the JSON recipe** opens the +actions from that build so you can inspect or adapt them. + +To try a deterministic recipe: + +1. Start in a blank app. Create a JavaScript query in Data Queries and keep its trigger manual. +2. Paste the following code and select this query in the Automator query selector. +3. Send a message in the Automator conversation. That invokes the selected query and + applies the returned actions. The query's own **Run** button only returns the recipe. + +```javascript +const actions = [{ + action: typeof recipeTitle === "undefined" ? "place_component" : "set_properties", + component: "text", + component_name: "recipeTitle", + layout: { x: 0, y: 0, w: 12, h: 4 }, + action_parameters: { text: "## Built from JSON", type: "markdown" } +}]; +const args = { actions }; +return { + role: "assistant", + content: [{ + type: "tool-call", + toolCallId: `recipe_${Date.now()}`, + toolName: "execute_automator_actions", + args, + argsText: JSON.stringify(args) + }] +}; +``` + +The example updates `recipeTitle` on later runs. It ignores the message text and makes +no LLM call, so no model connection is needed. AI Robot access and a workspace admin +or editor role are still required. Put the full ordered recipe in one +`execute_automator_actions` call. Review copied recipes before running them again: +creation actions can create duplicates, and actions are not rolled back as a group. + +During a run, Automator shows elapsed time while the query prepares its response, then +progress based on actual processed actions. The finish card distinguishes successful, +partial, and failed builds, provides an app preview when changes succeeded, and keeps +the recipe with the conversation. It reports executor outcomes, not a guarantee that +all generated app logic works as intended; use the preview to test your app. diff --git a/docs/build-applications/app-editor/collaborative-video-apps.md b/docs/build-applications/app-editor/collaborative-video-apps.md new file mode 100644 index 0000000000..5792ed70ed --- /dev/null +++ b/docs/build-applications/app-editor/collaborative-video-apps.md @@ -0,0 +1,1052 @@ +# Build Collaborative Video Apps + +Lowcoder can combine a live Agora meeting with the rest of an application. Participants can talk, share a screen, chat, exchange files, ask an AI assistant for help, review a Google document, or explore a form or website together without leaving the meeting. + +This guide shows how to build and bind those features in Lowcoder. It focuses on the application setup: which components, queries, states, module inputs, module outputs, and events to create. You can build only the sections your app needs. + +## What You Can Build + +The same meeting shell can support several experiences: + +| Experience | What participants can do | +| --- | --- | +| **AI meeting room** | Meet by video, see who is online or typing, use persistent room chat, attach files, mention people, and explicitly ask an AI assistant about the conversation or attachments. | +| **Document workspace** | Meet by video, browse permitted Google Drive files, and invite everyone to open the same document. Google continues to provide the document editing and permissions. | +| **Explore Together** | Meet by video while one participant guides others through a Typeform, Google Form, or supported public website. Depending on the page, participants can share form values, navigation, pointers, or caret positions. | + +## Recommended App Structure + +You can build everything on one Lowcoder page. For a larger application, reusable modules make the app easier to understand and maintain. + +Create: + +* a main **Meeting App** containing the Agora Meeting Controller and meeting controls; +* an optional **Header** module for profile, navigation, notifications, and sign-out; +* a **Stage** module for chat, documents, forms, or websites; and +* a **Backstage** module for the local camera and participant strip. + +The main app owns the meeting. It passes meeting data into the modules: + +| Module input | Binding in the main app | +| --- | --- | +| `participants` | `{{ meetingParticipants.value }}` | +| `messages` | `{{ meetingController.messages }}` | +| `currentUser` | `{{ meetingUser.value }}` | + +The document and Explore Together modules return a selected document or session through a module output. Their module event tells the main app when to broadcast it. + +The examples use `meetingUser.value` when the components are on the main page. If you place them inside a module, create the `currentUser` input shown above and use `currentUser.value` inside that module. + +> **Tip:** Build the common meeting shell first and test it with two users. Add chat, AI, documents, or Explore Together only after the meeting works. + +## Understand What Stores Each Kind of Data + +| Part | Use it for | Do not use it for | +| --- | --- | --- | +| **Agora Meeting Controller** | Audio, video, screen sharing, meeting participants, and short meeting signals | Persistent chat or business records | +| **Chat Controller** | Online users, typing, AI-thinking state, and live change notifications | Permanent message storage | +| **Your datasource** | Users, rooms, messages, mentions, and notifications | Camera or microphone streams | +| **Object storage** | Uploaded images and documents | Chat presence | +| **AI query** | Generating an optional assistant response | Storing the conversation | +| **Lowcoder proxy** | Loading supported forms and websites into an Explore Together session | General unrestricted web browsing | + +Save anything users expect to find later in a durable datasource. + +β€œPresence” has three separate bindings in these apps: + +* meeting presence comes from `meetingController.localUser` and `meetingController.participants`; +* chat presence comes from `chatController.onlineUsers` and `chatController.typingUsers`; and +* page presence, such as pointers and carets, appears only inside a supported Explore Together page. + +Adding the Agora meeting does not automatically add persistent chat or shared form fields. + +## Before You Start + +Prepare the services needed by your chosen experience: + +1. An [Agora account and project](https://console.agora.io/) with an App ID and enabled App Certificate. +2. A trusted server that issues Agora RTC and RTM tokens. +3. Authentication that gives every participant a stable user ID and display name. +4. Hocuspocus when you need Chat Controller or Explore Together. +5. A datasource for durable users, messages, rooms, and notifications. +6. Object storage when users can attach files. +7. An AI provider when the app includes an assistant. +8. Google or Typeform authorization when users browse private provider content. + +Use HTTPS and WSS in production. Browsers require a secure context for camera, microphone, and screen capture. + +## 1. Bind the Signed-In User + +Authenticate the user before they enter the meeting. The pattern is not tied to one identity provider. + +After sign-in, make one user object available to the app. In the examples below, a Temporary State named `meetingUser` contains: + +```javascript +{ + id: "user-42", + name: "Alex Morgan", + email: "alex@example.com", + picture: "https://example.com/alex.png" +} +``` + +Adapt the field names to your identity provider. The important requirements are: + +* `meetingUser.value.id` is stable and unique; +* `meetingUser.value.name` is safe to show to other users; and +* the app redirects unauthenticated users to sign-in. + +Create or update the signed-in user in your datasource. That record can also populate mention suggestions and receive notifications. + +For a suite of meeting apps, add a home or navigation page linking to the AI room, document workspace, and Explore Together workspace. A reusable header can show: + +* `{{ meetingUser.value.picture }}` as the avatar; +* `{{ meetingUser.value.name }}` as the display name; +* `{{ (loadUnreadNotifications.data || []).length }}` as an unread badge; and +* a sign-out event that ends the meeting before closing the user session. + +Create `loadUnreadNotifications` with a filter for the signed-in recipient and unread status. You can run it periodically, for example every 15 seconds, and after a notification is marked as read. Bind a notification click to your mark-read query and then navigate to the destination stored in the notification record. + +## 2. Build the Common Agora Meeting + +### Add the Meeting Controller + +Drag an **Agora Meeting Controller** into the main app and rename it `meetingController`. + +Bind its properties: + +| Meeting Controller property | Example binding | +| --- | --- | +| **App ID** | Your public Agora App ID | +| **Meeting Name** | `{{ url.query.meetingId || "demo-room" }}` | +| **Local User ID** | `{{ meetingUser.value.id }}` | +| **Local User Name** | `{{ meetingUser.value.name }}` | +| **RTC Token** | `{{ getMeetingTokens.data.rtcToken }}` | +| **RTM Token** | `{{ getMeetingTokens.data.rtmToken }}` | + +Use a dynamic meeting ID from your booking, event, or URL instead of using one fixed room for every meeting. + +Never place the Agora App Certificate in a Lowcoder app, JavaScript query, browser storage, or application export. Generate tokens on a trusted server. + +Lowcoder's multi-container setup maps these deployment variables into its token service: + +```text +AGORA_APP_ID +AGORA_APP_CERTIFICATE +AGORA_CORS_ALLOW_ORIGIN +``` + +A meeting name routes users to a channel; it is not a password. The token service must check that the signed-in user is allowed to join the requested meeting. + +### Create the Token Query + +Create a REST query named `getMeetingTokens`. + +| Setting | Value | +| --- | --- | +| Method | `GET` | +| URL | `https://tokens.example.com/rte/{{ meetingController.meetingName }}/publisher/userAccount/{{ meetingController.localUserID }}/` | +| Run when page loads | Off | + +Replace `https://tokens.example.com` with the address of your token service. The response must contain: + +```json +{ + "rtcToken": "...", + "rtmToken": "..." +} +``` + +Add a success event to `getMeetingTokens` that runs: + +```javascript +meetingController.startMeeting() +``` + +Add a **Join meeting** button whose click event runs `getMeetingTokens`. This order matters: first fetch current tokens, then join. + +Add a **Leave meeting** button whose click event runs: + +```javascript +meetingController.endMeeting() +``` + +Also run `meetingController.endMeeting()` from a Before unload query so the participant leaves cleanly when the page closes. + +### Create a Combined Participant List + +`meetingController.participants` contains the other participants. The current participant is available separately as `meetingController.localUser`. + +Create a Transformer named `meetingParticipants`: + +```javascript +if (!meetingController.meetingActive) return []; + +return [ + { ...meetingController.localUser, localUser: true }, + ...(meetingController.participants || []), +]; +``` + +Use `{{ meetingParticipants.value }}` everywhere the app or a module needs the complete participant list. + +Each item can contain: + +```javascript +{ + user: "user-42", + userName: "Alex Morgan", + audiostatus: false, + streamingVideo: true, + streamingSharing: false, + speaking: false, + localUser: true +} +``` + +Only the current participant has `localUser: true`, because the transformer adds it. + +### Display Participant Cameras + +Add a **List View** to the stage or backstage module. + +Set its **No. of rows** to: + +```javascript +{{ participants.value }} +``` + +If you did not use a module, bind it directly to: + +```javascript +{{ meetingParticipants.value }} +``` + +Inside the first List View item, add a **Camera Stream** and bind **Video Stream ID** to: + +```javascript +{{ currentItem }} +``` + +Add a Text component for the participant name: + +```javascript +{{ currentItem.userName || "User" }} +``` + +Useful optional bindings include: + +| UI | Binding | +| --- | --- | +| Muted icon | `{{ currentItem.audiostatus ? "/icon:solid/microphone" : "/icon:solid/microphone-slash" }}` | +| Speaking border | `{{ currentItem.speaking ? "#1A237E" : "transparent" }}` | +| Camera-off indicator | `{{ !currentItem.streamingVideo }}` | +| β€œYou” label | `{{ currentItem.localUser ? "You" : "" }}` | + +Bind **Profile Image URL** to an avatar found by `currentItem.user`, or use a generated fallback: + +```javascript +{{ + "https://api.dicebear.com/7.x/initials/svg?seed=" + currentItem.userName +}} +``` + +The participant object does not automatically contain your user database profile. If you want stored profile images, add a `users` module input and look up the matching record by `currentItem.user`. + +### Display Screen Sharing + +Add a **Screen Share Stream** to the main content area. Bind **Video Stream ID** to: + +```javascript +{{ + meetingParticipants.value.find( + participant => participant.streamingSharing + ) || {} +}} +``` + +Hide the empty screen-share area when no participant is sharing: + +```javascript +{{ !meetingParticipants.value.some(participant => participant.streamingSharing) }} +``` + +### Add Meeting Controls + +Connect buttons to these methods: + +| Button | Click action | +| --- | --- | +| Mute | `meetingController.audioControl()` | +| Camera | `meetingController.videoControl()` | +| Share screen | `meetingController.startSharing()` | +| Leave | `meetingController.endMeeting()` | + +Disable or hide the controls while: + +```javascript +{{ !meetingController.meetingActive }} +``` + +### Bind Stage and Backstage Modules + +Create a `participants` data input on both modules, with an empty-array default. In the parent Meeting App, bind each Module component input to: + +```javascript +{{ meetingParticipants.value }} +``` + +If the stage responds to meeting signals, also create a `messages` data input and bind it to: + +```javascript +{{ meetingController.messages }} +``` + +This is the shared contract used by the document and Explore Together examples later in this guide. + +## 3. Add Persistent Chat and Live Presence + +Add one **Chat Controller** and one **Chat Box** to your chat stage. + +Rename them `chatController` and `chatBox`. + +### Prepare the Chat Queries + +Create datasource queries with these responsibilities: + +| Query | Responsibility | +| --- | --- | +| `loadRooms` | Return rooms the signed-in user may access | +| `loadMessages` | Return messages for `chatController.currentRoomId` | +| `saveMessage` | Save the user's message and attachment metadata | +| `loadUsers` | Return users for the mention picker | +| `loadInvites` | Optional private-room invitations | +| `saveMentionNotifications` | Optional notifications for mentioned users | + +A minimal room record looks like: + +```javascript +{ + id: "general", + name: "General", + type: "public" +} +``` + +An AI room can include: + +```javascript +{ + id: "assistant", + name: "Meeting Assistant", + type: "llm", + llmQueryName: "runMeetingAssistant" +} +``` + +A minimal message record looks like: + +```javascript +{ + id: "message-101", + roomId: "general", + text: "Please review the attached file.", + authorId: "user-42", + authorName: "Alex Morgan", + authorType: "user", + timestamp: 1710000000000, + files: [] +} +``` + +Suggested datasource fields are: + +| Record | Useful fields | +| --- | --- | +| User | `id`, `name`, `email`, `picture` | +| Room | `id`, `name`, `type`, optional `llmQueryName` | +| Message | `id`, `roomId`, `text`, `authorId`, `authorName`, `authorType`, `timestamp`, `files` | +| Notification | `id`, `recipientUserId`, `title`, `body`, `destination`, `read`, `createdAt` | + +### Bind Chat Controller + +| Chat Controller property | Binding | +| --- | --- | +| **Application ID** | A stable collaboration namespace such as `meeting-suite-v1` | +| **User ID** | `{{ meetingUser.value.id }}` | +| **User Name** | `{{ meetingUser.value.name }}` | + +Do not generate a different Application ID per browser. Users can see each other's presence only when they connect to the same Application ID. + +For a connection indicator, bind a Text component to `{{ chatController.connectionStatus }}`. Do not enable realtime-only actions until `{{ chatController.ready }}` is true. See [Realtime Shared State and Presence](../realtime-collaboration.md) if the controller remains offline. + +### Bind Chat Box + +| Chat Box property | Binding | +| --- | --- | +| **Messages** | `{{ loadMessages.data || [] }}` | +| **Current User ID** | `{{ chatController.userId }}` | +| **Current User Name** | `{{ chatController.userName }}` | +| **Rooms** | `{{ loadRooms.data || [] }}` | +| **Current Room ID** | `{{ chatController.currentRoomId }}` | +| **Pending Invites** | `{{ loadInvites.data || [] }}` | +| **Typing Users** | `{{ chatController.typingUsers }}` | +| **Online Users** | `{{ chatController.onlineUsers }}` | +| **AI Is Thinking** | `{{ !!chatController.aiThinkingRooms?.[chatController.currentRoomId] }}` | +| **Mention Candidates** | `{{ (loadUsers.data || []).map(user => ({ id: user.id, label: user.name })) }}` | + +Enable **Show Rooms Panel** when users should change rooms from Chat Box. + +If you do not support private-room invitations, bind **Pending Invites** to `{{ [] }}` instead of creating `loadInvites`. + +### Bind Chat Events + +| Chat Box event | Action | +| --- | --- | +| `startTyping` | Run `chatController.startTyping(chatController.currentRoomId)` | +| `stopTyping` | Run `chatController.stopTyping()` | +| `roomSwitch` | Run `chatController.switchRoom(chatBox.pendingRoomId)`, then `loadMessages` | +| `messageSent` | Run the `sendChatMessage` JavaScript query below | +| `fileUpload` | Run the attachment upload flow from the next section | + +Create a JavaScript query named `sendChatMessage`: + +```javascript +const sendMessageFlow = async () => { + const attachments = pendingAttachments.value || []; + + await saveMessage.run({ + roomId: chatController.currentRoomId, + text: chatBox.lastSentMessageText, + authorId: meetingUser.value.id, + authorName: meetingUser.value.name, + authorType: "user", + files: attachments, + }); + + chatController.setRoomData( + chatController.currentRoomId, + "messagePing", + { + roomId: chatController.currentRoomId, + authorId: meetingUser.value.id, + timestamp: Date.now(), + } + ); + + await loadMessages.run(); + chatController.stopTyping(); + pendingAttachments.setValue([]); +}; + +sendMessageFlow(); +``` + +The values passed to `.run(...)` must match the variables used by your `saveMessage` query. + +On Chat Controller's `roomDataChanged` event, run `loadMessages`. This lets another participant's saved message appear without refreshing the page. + +> **Important:** Chat Controller tells clients that room data changed; it is not the message database. `loadMessages` must still read from your datasource. + +### Add Mention Notifications + +After `saveMessage` succeeds: + +1. Compare the message text with the names in `loadUsers.data`. +2. Ignore the signed-in user and the AI mention. +3. Insert one unread notification for each matched user. +4. Let the header reload unread notifications. + +A notification record should contain at least the recipient user ID, message or room ID, title, preview text, read status, and creation time. Store an app or page destination if clicking a notification should reopen the room. + +## 4. Bind File Attachments + +In Chat Box: + +* enable **Allow File Upload**; +* set **Max Files** to a sensible limit; and +* optionally restrict **File Type**, for example `{{ [".png", ".jpg", ".pdf"] }}`. + +Create a Temporary State named `pendingAttachments` with the initial value `[]`. + +When `fileUpload` runs, Chat Box exposes two aligned arrays: + +* `chatBox.files` contains file names, types, and sizes; and +* `chatBox.value` contains the file contents. + +Upload each file with your storage query. Store the returned metadata in `pendingAttachments`: + +```javascript +[ + { + name: "requirements.pdf", + type: "application/pdf", + url: "https://storage.example.com/..." + } +] +``` + +Bind the Chat Box `fileUpload` event to a JavaScript query such as `uploadPendingFiles`. That query should: + +1. loop over `chatBox.files` and `chatBox.value` by index; +2. run the upload query for each file; +3. collect the authorized URLs returned by storage; and +4. call `pendingAttachments.setValue(uploadedFiles)`. + +For example, adapt this query to the variables and response returned by your storage query: + +```javascript +const uploadFiles = async () => { + const uploadedFiles = []; + + for (let index = 0; index < chatBox.files.length; index += 1) { + const file = chatBox.files[index]; + const result = await uploadChatFile.run({ + fileName: file.name, + contentType: file.type, + fileData: chatBox.value[index], + }); + + uploadedFiles.push({ + name: file.name, + type: file.type, + url: result.url, + }); + } + + pendingAttachments.setValue(uploadedFiles); +}; + +uploadFiles(); +``` + +`sendChatMessage` then saves `pendingAttachments.value` in the message's `files` field. + +> **Important:** Capture files during `fileUpload`. Do not wait until `messageSent` to read `chatBox.files`, because Chat Box clears its pending files when the message is sent. + +Chat Box does not store files. Use private storage or short-lived signed URLs for non-public content and recheck authorization when users download an attachment. + +## 5. Bind an AI Assistant + +Use an `llm` room and invoke the AI only when the user explicitly mentions it, for example `@AI`. + +Chat Box exposes: + +```javascript +{{ chatBox.lastSentMessageTagsLlm }} +``` + +This becomes true when the last sent message mentioned the AI. + +If the room includes an AI assistant, replace the basic `sendChatMessage` query from the previous section with this complete version. Keeping the user-message and AI-message steps in one async function also preserves attachment metadata until the AI request finishes: + +```javascript +const sendMessageWithAi = async () => { + const attachments = pendingAttachments.value || []; + + await saveMessage.run({ + roomId: chatController.currentRoomId, + text: chatBox.lastSentMessageText, + authorId: meetingUser.value.id, + authorName: meetingUser.value.name, + authorType: "user", + files: attachments, + }); + + chatController.setRoomData( + chatController.currentRoomId, + "messagePing", + { authorId: meetingUser.value.id, timestamp: Date.now() } + ); + + await loadMessages.run(); + chatController.stopTyping(); + + const currentRoom = (loadRooms.data || []).find( + room => room.id === chatController.currentRoomId + ); + + const shouldRunAi = + currentRoom?.type === "llm" && + chatBox.lastSentMessageTagsLlm === true; + + if (!shouldRunAi) { + pendingAttachments.setValue([]); + return; + } + + chatController.setAiThinking(chatController.currentRoomId, true); + + try { + const aiResult = await runMeetingAssistant.run({ + roomId: chatController.currentRoomId, + message: chatBox.lastSentMessageText, + history: loadMessages.data || [], + files: attachments, + }); + + await saveMessage.run({ + roomId: chatController.currentRoomId, + text: aiResult.text, + authorId: "__llm_bot__", + authorName: "AI", + authorType: "assistant", + files: [], + }); + + chatController.setRoomData( + chatController.currentRoomId, + "messagePing", + { authorId: "__llm_bot__", timestamp: Date.now() } + ); + + await loadMessages.run(); + } finally { + chatController.setAiThinking(chatController.currentRoomId, false); + pendingAttachments.setValue([]); + } +}; + +sendMessageWithAi(); +``` + +Adapt `aiResult.text` to the response shape returned by your AI query. + +Run the AI request on a trusted server. Keep the provider key out of the Lowcoder app. Send only the authorized room history and attachments the user intended to share with the assistant. + +If attachments are included, convert images and documents to the input format supported by your provider. Reject unsupported types before calling the AI. + +## 6. Build a Shared Google Document Stage + +Create a Module named `DocumentStage`. + +### Create the Module Interface + +Add these module inputs: + +| Input | Type | Default | +| --- | --- | --- | +| `participants` | Data | `[]` | +| `messages` | Data | `[]` | +| `currentUserId` | Data | `""` | + +Add: + +* a Temporary State named `selectedDocument`, initially `null`; +* a module output named `documentToShare` with value `{{ selectedDocument.value }}`; and +* a module event named `onShareDocument`. + +In the main app, bind the DocumentStage Module component: + +| Input | Binding | +| --- | --- | +| `participants` | `{{ meetingParticipants.value }}` | +| `messages` | `{{ meetingController.messages }}` | +| `currentUserId` | `{{ meetingUser.value.id }}` | + +### Authorize Google Drive + +Use Google OAuth or an authorized Google datasource. Request only the scopes required by your app. + +Create a REST query named `listDriveFiles`: + +| Setting | Value | +| --- | --- | +| Method | `GET` | +| URL | `https://www.googleapis.com/drive/v3/files` | +| Header | `Authorization: Bearer {{ googleAccessToken.value }}` | +| `q` parameter | `'{{ currentFolderId.value }}' in parents and trashed=false` | +| `fields` parameter | `files(id,name,mimeType,modifiedTime,webViewLink,thumbnailLink,parents)` | +| `orderBy` parameter | `folder,name` | + +Create `currentFolderId` with initial value `root`, and run `listDriveFiles` after Google sign-in and whenever the folder changes. + +Bind a Table or List View to: + +```javascript +{{ listDriveFiles.data.files || [] }} +``` + +When a user selects an item: + +* if its MIME type is `application/vnd.google-apps.folder`, set `currentFolderId` to that item ID and run `listDriveFiles`; +* if it has a `webViewLink`, set `selectedDocument` to the item; and +* otherwise open the file using the provider's supported download or view action. + +Bind an IFrame URL to: + +```javascript +{{ selectedDocument.value?.webViewLink || "" }} +``` + +If Google or the selected file type refuses to load in an IFrame, show an **Open in Google** button using the same `webViewLink` instead. + +Add a **View together** button. Its click event should first set `selectedDocument`, then use **Trigger module event** to trigger `onShareDocument`. + +### Broadcast the Selected Document + +On the DocumentStage Module component in the main app, bind `onShareDocument` to: + +```javascript +meetingController.broadCast({ + type: "documentShare", + senderId: meetingUser.value.id, + document: documentStage.documentToShare, +}); +``` + +The Meeting Controller places received channel messages in `meetingController.messages`. In `DocumentStage`, add a Data Responder watching: + +```javascript +{{ JSON.stringify(messages.value) }} +``` + +Run: + +```javascript +const allMessages = messages.value || []; +const received = allMessages[allMessages.length - 1]; +const signal = received?.channelmessage?.message; + +if ( + signal?.type === "documentShare" && + signal.senderId !== currentUserId.value +) { + selectedDocument.setValue(signal.document); +} +``` + +> **Important:** Lowcoder synchronizes which document is open. Google provides the actual document collaboration. Every participant must have the correct Google account and file permission. + +## 7. Build Explore Together Modules + +Explore Together works best as one parent `ExploreStage` module containing three child modules: + +* `TypeformStage`; +* `GoogleFormsStage`; and +* `WebsiteStage`. + +A Tabbed Container in `ExploreStage` lets the user choose the provider. + +### Give Every Provider Module the Same Interface + +On each provider module, create: + +| Item | Name | Value | +| --- | --- | --- | +| Input | `participants` | Data, default `[]` | +| Input | `messages` | Data, default `[]` | +| Output | `sessionToShare` | The driver query's `data.data.broadcast` | +| Event | `onExploreTogether` | Triggered after the driver session is created | +| Temporary State | `localParticipant` | `{}` | +| Temporary State | `sharedPageUrl` | `null` | +| Temporary State | `receivedSession` | `null` | + +Add a Data Responder watching: + +```javascript +{{ JSON.stringify(participants.value) }} +``` + +Run: + +```javascript +const current = (participants.value || []).find( + participant => participant.localUser +); + +localParticipant.setValue(current || {}); +``` + +Bind the provider module's IFrame URL to: + +```javascript +{{ sharedPageUrl.value || "" }} +``` + +### Create the Driver Session Query + +Create a REST query for the provider selected by the presenter: + +| Provider | POST endpoint | Provider URL field | +| --- | --- | --- | +| Typeform | `/proxy/typeform/session` | `typeformUrl` | +| Google Forms | `/proxy/google-forms/session` | `googleFormUrl` | +| Website | `/proxy/website/session` | `websiteUrl` | + +Set the method to `POST`, the body type to JSON, and the `Content-Type` header to `application/json`. + +Use your Lowcoder proxy service as the base URL. For example: + +```text +https://lowcoder.example.com/proxy/website/session +``` + +The driver request body follows this pattern: + +```javascript +{ + websiteUrl: selectedWebsite.value.url, + roomId: selectedWebsite.value.id, + role: "driver", + editorId: localParticipant.value.user, + username: localParticipant.value.userName +} +``` + +For Typeform, replace `websiteUrl` with `typeformUrl`. For Google Forms, replace it with `googleFormUrl`. + +For Google Forms, obtain the source URL from the selected Google Drive form's `webViewLink`. For Typeform, use the form's display URL returned by the Typeform API. For a website, use an approved HTTP(S) URL entered or selected by the user. + +When a user enters a website manually, create the selected record before running the driver query: + +```javascript +selectedWebsite.setValue({ + id: `website-${Date.now()}`, + url: websiteUrlInput.value.trim(), +}); +``` + +Run the driver session query after that state change. + +On driver query success: + +1. set `sharedPageUrl` to `{{ createDriverSession.data.data.proxiedUrl }}`; and +2. trigger the module event `onExploreTogether`. + +Bind the module output `sessionToShare` to: + +```javascript +{{ createDriverSession.data.data.broadcast }} +``` + +The exact query name may be `createTypeformSession`, `createGoogleFormsSession`, or `createWebsiteSession`; update the bindings accordingly. + +### Re-Emit the Event from ExploreStage + +Bind `participants` and `messages` from ExploreStage into all three child Module components: + +```javascript +{{ participants.value }} +{{ messages.value }} +``` + +Create an ExploreStage output named `sessionToShare`. Return the active child module's output: + +```javascript +{{ + providerTabs.selectedTabKey === "googleForms" + ? googleFormsStage.sessionToShare + : providerTabs.selectedTabKey === "website" + ? websiteStage.sessionToShare + : typeformStage.sessionToShare +}} +``` + +Add an ExploreStage event named `onExploreTogether`. When a child module triggers its event, use **Trigger module event** to re-emit `onExploreTogether` from ExploreStage. + +### Broadcast the Session from the Main App + +Bind the main app's ExploreStage Module event to: + +```javascript +meetingController.broadCast({ + type: "exploreTogether", + senderId: meetingUser.value.id, + provider: exploreStage.selectedProvider, + session: exploreStage.sessionToShare, +}); +``` + +Expose the selected provider from ExploreStage as an output named `selectedProvider`, bound to `{{ providerTabs.selectedTabKey }}`. + +### Join as a Follower + +Each provider module should watch its `messages` input with a Data Responder: + +```javascript +const allMessages = messages.value || []; +const received = allMessages[allMessages.length - 1]; +const signal = received?.channelmessage?.message; + +if ( + signal?.type === "exploreTogether" && + signal.provider === "website" && + signal.senderId !== localParticipant.value.user +) { + receivedSession.setValue(signal.session); +} +``` + +The example above is for `WebsiteStage`. Use `typeform` in TypeformStage and `googleForms` in GoogleFormsStage. This prevents hidden provider modules from joining the wrong session. + +Create a follower REST query using the provider's join endpoint: + +| Provider | POST endpoint | +| --- | --- | +| Typeform | `/proxy/typeform/session/join` | +| Google Forms | `/proxy/google-forms/session/join` | +| Website | `/proxy/website/session/join` | + +Example website follower body: + +```javascript +{ + websiteUrl: receivedSession.value.websiteUrl, + roomId: receivedSession.value.roomId, + collab: receivedSession.value.collab, + role: "follower", + editorId: localParticipant.value.user, + username: localParticipant.value.userName +} +``` + +Use `typeformUrl` or `googleFormUrl` for the other providers. + +Add another Data Responder watching: + +```javascript +{{ JSON.stringify(receivedSession.value) }} +``` + +When a valid session arrives, run the follower query. On success, set: + +```javascript +sharedPageUrl.setValue(joinSession.data.data.proxiedUrl) +``` + +Both presenter and follower must display the returned `proxiedUrl`. Do not replace it with the original provider URL, because the original URL does not contain the Explore Together connection. + +### Provider Selection Bindings + +For Typeform, create an authorized `GET https://api.typeform.com/forms` query. Bind a Table or Select component to the returned forms and store the selected form's `id` and `_links.display` URL. Keep the Typeform token in a server-side datasource or query. + +For Google Forms: + +1. authorize Google; +2. query `GET https://www.googleapis.com/drive/v3/files` with `q=mimeType='application/vnd.google-apps.form'`; +3. bind the results to a Table or List View; and +4. use the selected form's ID and `webViewLink` in the driver session query. + +For a generic website, use an Input component. Bind the **Disabled** property of the **Explore together** button to: + +```javascript +{{ + !/^https?:\/\//i.test(websiteUrlInput.value || "") +}} +``` + +Only enable URLs permitted by your server's website allowlist. + +### What Participants Can Share + +| Workspace | Shared experience | Important behavior | +| --- | --- | --- | +| **Typeform** | Current step, next/back actions, text and choice values, answers, and live caret presence | Test every form you publish because provider markup can change. | +| **Google Forms** | Text and choice values, page navigation, and live caret presence | Remote submit is not applied automatically. Decide who submits and explain it in the app. | +| **Supported website** | Standard text/select/checkbox/radio fields, presenter scrolling, pointers, text selections, click highlights, and navigation offers | Followers can choose **Follow** or **Stay** when the presenter navigates. Password and file fields are not synchronized. | + +Canvas content, media players, custom widgets, cross-origin content, and internal JavaScript state may not synchronize. Test each supported site before offering it to users. + +## 8. Configure the Proxy for Production + +External forms and websites must use the Lowcoder proxy for Explore Together. + +Configure: + +```text +LOWCODER_API_KEY_SECRET +LOWCODER_HOCUSPOCUS_URL +LOWCODER_HOCUSPOCUS_SECRET +LOWCODER_PROXY_ALLOWED_HOSTS +LOWCODER_GOOGLE_FORMS_ALLOWED_HOSTS +LOWCODER_WEBSITE_ALLOWED_HOSTS +``` + +Use the three allowlists for Typeform, Google Forms, and generic websites respectively. An empty generic website allowlist permits all public hosts, which is useful for local testing but usually too broad for production. + +Keep Google, Typeform, AI, storage, and proxy credentials in server-side datasources or environment variables. Never copy access tokens from a test app into published documentation or exported applications. + +## Complete Binding Checklist + +Before testing, confirm these connections: + +| From | To | +| --- | --- | +| Signed-in user | Meeting Controller user ID and name | +| Token query response | Meeting Controller RTC and RTM token properties | +| Token query success | `meetingController.startMeeting()` | +| Meeting Controller local user + participants | `meetingParticipants` Transformer | +| `meetingParticipants.value` | Stage and Backstage `participants` inputs | +| `meetingController.messages` | Document/Explore Stage `messages` input | +| Participant List View item | Camera Stream Video Stream ID | +| Sharing participant | Screen Share Stream Video Stream ID | +| Chat Controller presence | Chat Box typing and online properties | +| Chat Box `messageSent` | Save message, signal room, reload messages | +| Chat Box `fileUpload` | Upload files and fill `pendingAttachments` | +| Chat Controller `roomDataChanged` | Reload durable messages | +| AI mention flag | Run AI query only when explicitly mentioned | +| Document module event | Broadcast document session through Meeting Controller | +| Explore Stage event | Broadcast proxy session through Meeting Controller | +| Received meeting signal | Set selected document or run follower session query | +| Proxy response `proxiedUrl` | Provider IFrame URL | + +## Test with Two Users + +Use two browsers or one normal and one private window with different signed-in users. + +Test in this order: + +1. Both users receive tokens and join the same meeting. +2. Each user sees the other's camera, name, mute state, and speaking state. +3. Mute, camera, screen share, and leave controls work. +4. Chat online and typing presence updates. +5. Messages survive refresh and a Hocuspocus restart because they come from the datasource. +6. Attachments remain authorized and downloadable. +7. The AI runs only after an AI mention, shows thinking state, and saves its response. +8. Mentioned users receive the intended notifications. +9. Both users can access a shared Google document. +10. Explore Together creates one driver session and joins the same session for the follower. +11. Supported fields, pointers, steps, scrolling, and navigation synchronize as documented. +12. Form submission behavior is clear and does not create duplicate submissions. + +Also test denied camera permissions, expired tokens, failed uploads, AI errors, inaccessible documents, blocked websites, refreshes, and participant disconnects. + +## Troubleshooting Bindings + +| Problem | Check | +| --- | --- | +| Meeting does not start | App ID, RTC token, RTM token, meeting name, user ID, and token query success event | +| One browser replaces another | Every participant must have a different stable Local User ID | +| Local user is missing from a module | The `meetingParticipants` Transformer adds `localUser: true` and is bound to the module input | +| Remote cameras have no place to render | List View is bound to participants and Camera Stream receives `{{ currentItem }}` | +| Screen share is active but blank | Screen Share Stream receives the participant whose `streamingSharing` value is true | +| Typing or online users never update | Chat Controller uses the same Application ID and can reach Hocuspocus | +| Chat disappears after refresh | Save and reload messages from a datasource rather than relying on shared state | +| Attachment URLs are missing | Capture and upload files on `fileUpload`, before `messageSent` clears them | +| AI replies to every message | Gate the AI query with `chatBox.lastSentMessageTagsLlm` | +| Shared document does not open | Module event fires, output contains the document, `messages` is bound, and receiver reads `channelmessage.message` | +| Follower opens a separate Explore Together session | Pass the driver's `roomId` and `collab` into the `/session/join` query | +| IFrame opens but nothing synchronizes | Bind the proxy response's `proxiedUrl`, not the original page URL | +| Google document is forbidden | Every participant needs permission from Google | +| Website is rejected | URL must be public HTTP(S) and permitted by the website allowlist | + +## Production Checklist + +* Keep Agora certificates, AI keys, storage credentials, provider tokens, and proxy secrets on trusted servers. +* Authenticate and authorize Agora token, upload, AI, Google, proxy-session, and download requests. +* Use stable user IDs, unguessable meeting IDs, and scoped, short-lived tokens. +* Use HTTPS/WSS and restrict CORS and proxy host allowlists. +* Store durable messages, users, notifications, and business records in your datasource. +* Use Chat Controller and Agora RTM only for presence and live signals. +* Apply file size/type limits, malware scanning where appropriate, and short-lived file URLs. +* Confirm provider permissions before sharing a document or form. +* Make it clear who is presenting, who may edit, who submits forms, and what data is sent to AI. +* Define retention and privacy rules for messages, attachments, AI prompts, and shared sessions. + +## Related Documentation + +* [Chat Box](visual-components/chat-box.md) +* [Chat Controller](visual-components/chat-controller.md) +* [Realtime Shared State and Presence](../realtime-collaboration.md) +* [Agora secure token authentication](https://docs.agora.io/en/realtime-media/rtm/build/connect-and-authenticate/authentication-workflow) diff --git a/docs/build-applications/app-editor/npm-template-plugins.md b/docs/build-applications/app-editor/npm-template-plugins.md new file mode 100644 index 0000000000..2ab225dda4 --- /dev/null +++ b/docs/build-applications/app-editor/npm-template-plugins.md @@ -0,0 +1,181 @@ +# NPM Template Plugins + +NPM Template Plugins let developers package reusable Lowcoder layouts as npm packages. Lowcoder users can add those packages from the editor, drag a prepared layout onto the canvas, and then place normal Lowcoder components into the editable areas of that layout. + +## Why this feature exists + +Complex layouts can already be created directly inside Lowcoder. For many apps, that is still the right approach. The problem starts when the same layout needs to be reused often and contains many fixed structural parts, such as sidebars, headers, tab areas, dashboard shells, card grids, landing pages, or admin panels. + +Building those structures natively in Lowcoder can require a lot of nested containers and repeated layout setup. Over time, that can make the app harder to maintain, harder to copy between apps, and heavier for the editor and runtime. + +NPM Template Plugins solve this by moving the fixed layout structure into a developer-maintained npm package. The package provides the layout shell, while Lowcoder keeps the important editable parts available as drop-zones. Users still build visually in Lowcoder, but they start from a prepared structure instead of rebuilding the same nested layout every time. + +In short: + +- Developers prepare and publish reusable layout packages. +- Lowcoder users install those packages from the editor. +- The fixed layout comes from the package. +- The editable areas remain available as Lowcoder drop-zones. + +## Templates vs npm component plugins + +Both npm component plugins and npm template plugins are installed from npm, but they solve different problems. + + +| Type | Use it for | +| -------------------- | ------------------------------------------------------------------------------------ | +| NPM Component Plugin | Adding one custom component, such as a chart, widget, map, viewer, or input control. | +| NPM Template Plugin | Adding a reusable page or section layout that contains editable Lowcoder drop-zones. | + + +Use a template plugin when the main value is the layout structure. Use a component plugin when the main value is a single reusable component. + +## For Lowcoder users + +### What you need + +Before adding a template package, you need: + +- the npm package name or npm package URL +- access to the Lowcoder workspace where the package should be available +- private npm registry settings, if the package is not published publicly + +For testing, you can use the demo package: + +```text +lowcoder-comp-template-demos +``` + +### Add a template package + +1. Open your app in the Lowcoder editor. +2. Go to **Insert > Extensions**. +3. Find the **Templates** section. +4. Click **Add npm Template**. +5. Enter the npm package name or URL. +6. Confirm the dialog. + +
Add npm template package from the Extensions panel

Add an npm template package from the Extensions panel.

+ +Lowcoder stores the package in the workspace settings, so the templates can be reused by apps in the same workspace. + +### Choose a template + +After the package loads, Lowcoder shows the available template designs under the package name. Drag the template you want onto the canvas. + +
Template package with available template designs

Template designs exposed by the installed npm package.

+ +The demo package includes layouts such as dashboard, landing page, sidebar, card grid, and admin panel templates. + +### Use the drop-zones + +The template package provides the fixed layout. The drop-zones are the editable areas where you add Lowcoder components such as tables, charts, forms, buttons, containers, and custom components. + +
Using a template by dropping Lowcoder components into the prepared drop-zone

Add Lowcoder components inside the template drop-zone.

+ +Components inside a drop-zone work like normal Lowcoder components. You can configure them, bind queries, connect data, and style them as usual. + +## For template plugin developers + +Template plugin developers create the npm package that Lowcoder users install. The reference implementation uses React, TypeScript, Vite, `lowcoder-sdk`, and `lowcoder-cli`, but the idea is not limited to React as a design choice. A developer can use another UI technology if the final package is compatible with Lowcoder's plugin loading model and exposes the required Lowcoder component/container behavior. + +For most teams, React is still the recommended path because the Lowcoder SDK examples and the current template demo are React-based. + +Developers should prepare: + +- a reusable layout shell, such as a dashboard, portal, landing page, sidebar layout, or admin panel +- one or more editable drop-zones where users can place Lowcoder components +- template names, descriptions, and icons for the editor +- `package.json` metadata under `lowcoder.comps` +- an export map from `src/index.ts` +- an npm package published publicly or available through a configured private registry + +The reference implementation is available in the Lowcoder component plugin examples: + +```text +lowcoder-create-component-plugin/lowcoder-comp-templates-demo/lowcoder-templates/ +``` + +The published demo package is: + +```text +lowcoder-comp-template-demos +``` + +## Development model + +A template package usually has two parts: + + +| Part | Responsibility | +| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | +| Fixed layout | Built by the developer in the package. This can include navigation, headers, sidebars, cards, tabs, static sections, and other structure. | +| Drop-zones | Editable Lowcoder areas where users can drag components and continue building visually. | + + +The important rule is to keep the layout shell in code and expose only the parts that should remain editable as Lowcoder drop-zones. This avoids forcing users to manage a deeply nested native Lowcoder layout while still giving them control over the business-specific parts of the app. + +## Package registration + +Lowcoder discovers templates from the `lowcoder.comps` section in `package.json`. + +Example: + +```json +{ + "name": "my-template-package", + "version": "0.1.0", + "lowcoder": { + "description": "Reusable Lowcoder layout templates", + "comps": { + "dashboardTemplate": { + "name": "Dashboard Template", + "icon": "./icons/dashboard.svg", + "description": "Dashboard layout with a main content drop-zone", + "isContainer": true, + "layoutInfo": { + "w": 24, + "h": 80, + "delayCollision": true + } + } + } + } +} +``` + +Important fields: + + +| Field | Purpose | +| ------------- | -------------------------------------------------------- | +| `name` | The template name shown to Lowcoder users. | +| `icon` | The icon shown in the Templates list. | +| `description` | A short explanation of the layout. | +| `isContainer` | Required when the template contains editable drop-zones. | +| `layoutInfo` | Default canvas size and placement behavior. | + + +The key inside `lowcoder.comps`, for example `dashboardTemplate`, must match the component export in `src/index.ts`. + +Example: + +```ts +import DashboardTemplateComp from "./templates/dashboard/DashboardTemplateComp"; + +export default { + dashboardTemplate: DashboardTemplateComp, +}; +``` + +## Developer checklist + +Before publishing a template package, check that: + +- each template has a clear name, description, and icon +- each editable area is implemented as a Lowcoder drop-zone +- templates with drop-zones are marked with `"isContainer": true` +- every `lowcoder.comps` key is exported from `src/index.ts` +- the package builds successfully with the Lowcoder plugin build flow +- the package is published to npm or available from the configured private registry +- users know which package name or URL to add in Lowcoder diff --git a/docs/build-applications/app-editor/visual-components/README.md b/docs/build-applications/app-editor/visual-components/README.md index bd2df1d0b3..7f1345632b 100644 --- a/docs/build-applications/app-editor/visual-components/README.md +++ b/docs/build-applications/app-editor/visual-components/README.md @@ -28,6 +28,8 @@ We are proud to share that we also have unmatched, deep styling capabilities, so ### Meetings & Collaboration +See [Meeting & Collaboration](meeting-and-collaboration.md) for the Chat Box, AI Chat, Chat Controller, and video meeting guides. +
button
Screen Share StreamAccess and Displays the Stream of shared Screens in the Browser.screen-share-stream-l.svg
Camera StreamAccess and Displays the Stream of the Camera of the current User and Attendees.camera-stream-l.svg
Agora Meeting ControllerMeeting Component to manage and secure a Meeting Room.meeting-room-l.svg
AvatarDisplays user avatars or profile images for personalized identification.avatar-l.svg
Avatar GroupA group of avatars to represent multiple users or entities in a compact and visually appealing way.avatargroup-l.svg
CommentA component for adding and displaying user comments, supporting threaded replies and user interaction.comment-l.svg
MentionA component that supports mentioning users or tags within text content, typically used in social media or collaborative platforms.mention-l.svg
Icon ButtonA button for controlling functions like start, end, mute, or share.icon-button-l.svg
### Project Management diff --git a/docs/build-applications/app-editor/visual-components/ai-chat.md b/docs/build-applications/app-editor/visual-components/ai-chat.md new file mode 100644 index 0000000000..a89832a786 --- /dev/null +++ b/docs/build-applications/app-editor/visual-components/ai-chat.md @@ -0,0 +1,560 @@ +# AI Chat + +The **AI Chat** component (`chatComp`) provides the user interface and local thread state for assistant-style conversations. It is provider-neutral: the selected Lowcoder query decides whether the assistant uses OpenAI, Anthropic Claude, xAI Grok, Ollama, or another service. + +It includes: + +- local thread creation, switching, renaming, and deletion +- local conversation history +- a message composer +- optional attachments +- message editing and response regeneration +- a selected Lowcoder query as the assistant backend + +## How The Integration Works + +Use two queries for a normal provider integration: + +```text +AI Chat component + | + | runs the selected query with a current conversation snapshot + v +JavaScript adapter query + | + | converts conversationHistory to the provider's request schema + v +REST query for OpenAI, Claude, Grok, Ollama, or another provider + | + | provider-specific response + v +{ role: "assistant", content: "..." } +``` + +Select the **JavaScript adapter query**, not the REST query, in the component's **Chat Query** property. The adapter has two jobs: + +1. send the relevant conversation history to the REST query +2. convert the provider response to the message shape expected by AI Chat + +This separation keeps the component provider-neutral and makes it easy to change providers without changing the component. + +## Component Properties + +The main properties are: + +- **Chat Query** (`chatQuery`) β€” the adapter query that runs when the user sends or edits a message +- **System Prompt** (`systemPrompt`) β€” the instruction prepended to the exposed conversation history +- **Placeholder** (`placeholder`) β€” the composer placeholder text +- **Left Panel Width** (`leftPanelWidth`) β€” the thread sidebar width; the default is `250px` + +The component also supports auto or fixed height and separate styles for its container, sidebar, messages, input, buttons, thread items, and animations. + +## Query Inputs + +When AI Chat runs its selected query, the following values are available directly inside that query: + +```js +prompt.value +message.value +conversationHistory.value +``` + +- `prompt.value` is the current user's plain text. +- `message.value` is the complete current message object, including its content and any attachments. +- `conversationHistory.value` is a request snapshot containing the system prompt, previous messages, and the current user message. Use this value to build the provider request. + +Do not prefix these values with `args`. For example, use `prompt.value`, not `args.prompt.value`. + +The component also exposes its reactive history to the rest of the app: + +```js +chat1.conversationHistory +``` + +Replace `chat1` if your component has a different name. + +Inside the selected query, prefer `conversationHistory.value`. Unlike the exposed component value, the query argument is constructed synchronously and is guaranteed to include the message currently being processed. + +## Conversation History + +`conversationHistory` is an array containing the active thread's messages. It begins with a system message whose content comes from the **System Prompt** property: + +```js +[ + { + role: "system", + content: "You are a helpful assistant.", + timestamp: 1750000000000 + }, + { + role: "user", + content: "Hello", + timestamp: 1750000001000 + }, + { + role: "assistant", + content: "Hi! How can I help?", + timestamp: 1750000002000 + } +] +``` + +AI providers normally accept only selected fields. Map the history before sending it instead of passing the raw objects: + +```js +const history = conversationHistory.value.map(({ role, content }) => ({ + role, + content +})); +``` + +This removes Lowcoder-only fields such as `timestamp` and attachment metadata. + +The exposed array also supports the standard state methods: + +```js +chat1.setConversationHistory(newArray) +chat1.clearConversationHistory() +chat1.resetConversationHistory() +``` + +## Required Query Response + +The selected query must return the assistant message itself: + +```js +return { + role: "assistant", + content: "Assistant reply text" +}; +``` + +`content` may also be an array of Assistant UI content parts: + +```js +return { + role: "assistant", + content: [ + { type: "text", text: "Assistant reply text" } + ] +}; +``` + +The top-level `role` must be `"assistant"`. The component generates `id` and `createdAt` when they are omitted. + +Do not return a provider wrapper such as this: + +```js +return { + message: { + role: "assistant", + content: "Assistant reply text" + } +}; +``` + +The component reads `role` and `content` from the top-level returned object. + +## Recommended Setup + +The following examples assume: + +- the component is named `chat1` +- both queries are set to run manually +- the REST query's **Variables** tab is empty +- the JavaScript adapter is selected in `chat1` under **Chat Query** + +Keep API keys in a protected Lowcoder data source or another server-side secret store for production applications. Do not expose keys in public app values or client-side JavaScript. + +### OpenAI Responses API + +Create a REST query named `openAIResponses`. + +Configure it as follows: + +- **Method:** `POST` +- **URL:** `https://api.openai.com/v1/responses` +- **Triggered when:** manually +- **Body type:** JSON + +Headers: + +| Key | Value | +| --- | --- | +| `Authorization` | `Bearer YOUR_OPENAI_API_KEY` | +| `Content-Type` | `application/json` | + +JSON body: + +```json +{ + "model": "gpt-4.1-mini", + "instructions": {{ instructions.value }}, + "input": {{ input.value }} +} +``` + +Create a JavaScript query named `openAIChatAdapter`: + +```js +const history = conversationHistory.value; + +const system = history.find(({ role }) => role === "system"); +const input = history + .filter(({ role }) => role !== "system") + .map(({ role, content, attachments = [] }) => { + const images = attachments + .flatMap((attachment) => attachment.content || []) + .filter((part) => part.type === "image" && part.image) + .map((part) => ({ + type: "input_image", + image_url: part.image, + detail: "auto" + })); + + return { + role, + content: images.length + ? [ + ...(content + ? [{ type: "input_text", text: content }] + : []), + ...images + ] + : content + }; + }); + +return openAIResponses + .run({ + instructions: system?.content || "", + input + }) + .then((response) => { + const content = (response.output || []) + .filter((item) => item.type === "message" && item.role === "assistant") + .flatMap((item) => item.content || []) + .filter((part) => part.type === "output_text") + .map((part) => part.text) + .join("\n"); + + if (!content) { + throw new Error("OpenAI returned no assistant text"); + } + + return { role: "assistant", content }; + }); +``` + +Select `openAIChatAdapter` in **Chat Query**. + +The Responses API accepts conversation input through `input` and separate system/developer guidance through `instructions`. Its response contains an `output` array, so the adapter extracts `output_text` rather than assuming the first output item always contains the final text. See the [OpenAI Responses API reference](https://developers.openai.com/api/reference/resources/responses/methods/create). + +### Anthropic Claude Messages API + +Create a REST query named `claudeMessages`. + +Configure it as follows: + +- **Method:** `POST` +- **URL:** `https://api.anthropic.com/v1/messages` +- **Triggered when:** manually +- **Body type:** JSON + +Headers: + +| Key | Value | +| --- | --- | +| `x-api-key` | `YOUR_ANTHROPIC_API_KEY` | +| `anthropic-version` | `2023-06-01` | +| `Content-Type` | `application/json` | + +JSON body: + +```json +{ + "model": "YOUR_CLAUDE_MODEL", + "max_tokens": 1024, + "system": {{ system.value }}, + "messages": {{ messages.value }} +} +``` + +Replace `YOUR_CLAUDE_MODEL` with a model available to your Anthropic account. + +Create a JavaScript query named `claudeChatAdapter`: + +```js +const history = conversationHistory.value.map(({ role, content }) => ({ + role, + content +})); + +const system = history.find(({ role }) => role === "system"); +const messages = history.filter(({ role }) => role !== "system"); + +return claudeMessages + .run({ + system: system?.content || "", + messages + }) + .then((response) => { + const content = (response.content || []) + .filter((block) => block.type === "text") + .map((block) => block.text) + .join("\n"); + + if (!content) { + throw new Error("Claude returned no assistant text"); + } + + return { role: "assistant", content }; + }); +``` + +Select `claudeChatAdapter` in **Chat Query**. + +Anthropic's Messages API receives prior user and assistant turns in `messages`; the system instruction is supplied separately. Text replies are returned as blocks in `response.content`. See the [Claude Messages API reference](https://platform.claude.com/docs/en/api/http/messages/create). + +### xAI Grok Responses API + +Create a REST query named `grokResponses`. + +Configure it as follows: + +- **Method:** `POST` +- **URL:** `https://api.x.ai/v1/responses` +- **Triggered when:** manually +- **Body type:** JSON + +Headers: + +| Key | Value | +| --- | --- | +| `Authorization` | `Bearer YOUR_XAI_API_KEY` | +| `Content-Type` | `application/json` | + +JSON body: + +```json +{ + "model": "YOUR_GROK_MODEL", + "input": {{ input.value }} +} +``` + +Replace `YOUR_GROK_MODEL` with a model available to your xAI account. + +Create a JavaScript query named `grokChatAdapter`: + +```js +const input = conversationHistory.value.map(({ role, content }) => ({ + role, + content +})); + +return grokResponses + .run({ input }) + .then((response) => { + const content = (response.output || []) + .filter((item) => item.type === "message" && item.role === "assistant") + .flatMap((item) => item.content || []) + .filter((part) => part.type === "output_text") + .map((part) => part.text) + .join("\n"); + + if (!content) { + throw new Error("Grok returned no assistant text"); + } + + return { role: "assistant", content }; + }); +``` + +Select `grokChatAdapter` in **Chat Query**. + +xAI recommends its Responses API for new integrations. It accepts the conversation in `input` and returns assistant text in `output_text` content blocks. See the [xAI text generation guide](https://docs.x.ai/developers/model-capabilities/text/generate-text). + +### Ollama OpenAI-Compatible API + +First, make sure Ollama is running and the model has been pulled. Then create a REST query named `ollamaChat`. + +Configure it as follows: + +- **Method:** `POST` +- **URL:** `http://localhost:11434/v1/chat/completions` +- **Triggered when:** manually +- **Body type:** JSON + +Header: + +| Key | Value | +| --- | --- | +| `Content-Type` | `application/json` | + +JSON body: + +```json +{ + "model": "llama3.2", + "messages": {{ messages.value }}, + "stream": false +} +``` + +Create a JavaScript query named `ollamaChatAdapter`: + +```js +const messages = conversationHistory.value.map(({ role, content }) => ({ + role, + content +})); + +return ollamaChat + .run({ messages }) + .then((response) => { + const content = response.choices?.[0]?.message?.content; + + if (!content) { + throw new Error("Ollama returned no assistant text"); + } + + return { role: "assistant", content }; + }); +``` + +Select `ollamaChatAdapter` in **Chat Query**. + +Ollama provides an OpenAI-compatible `/v1/chat/completions` endpoint. Replace `llama3.2` with a model installed on your Ollama server. See the [Ollama OpenAI compatibility guide](https://docs.ollama.com/api/openai-compatibility). + +> **Network note:** `localhost` is resolved from where the Lowcoder REST query executes. If Lowcoder runs in Docker or on another server, `localhost` is not your laptop's Ollama process. Use a hostname or network address reachable from the Lowcoder backend, such as a Docker service name when both services share a Docker network. + +## Why The JavaScript Adapter Is Needed + +Each provider returns a different response shape: + +| Provider | Assistant text location | +| --- | --- | +| OpenAI Responses | `response.output[*].content[*].text` for `output_text` parts | +| Claude Messages | `response.content[*].text` for text blocks | +| xAI Responses | `response.output[*].content[*].text` for `output_text` parts | +| Ollama Chat Completions | `response.choices[0].message.content` | + +AI Chat always expects the same result: + +```js +{ + role: "assistant", + content: "..." +} +``` + +The adapter isolates those provider differences. + +## `currentMessage` + +The component exposes: + +```js +chat1.currentMessage +``` + +This value is updated after the selected query succeeds. Use `prompt.value` inside the selected query for the message currently being processed. Use `currentMessage` in post-response app logic and events. + +## Attachments + +The composer accepts files up to 10 MB. The current message's attachments are available inside the selected query through: + +```js +message.value.attachments +``` + +Attachments are not automatically converted to an AI provider's multimodal request schema. The OpenAI example above demonstrates how to convert image attachments; other providers require their corresponding content-part format. + +- Images are represented in the current message with browser-generated base64 data URLs. +- Other files use browser-local `blob:` URLs, which a server-side REST query cannot fetch directly. +- In `conversationHistory`, image content is retained, while non-image file content is not retained as provider-ready data. + +For multimodal or file analysis, add provider-specific conversion or upload logic to the adapter/backend. Do not pass Lowcoder attachment objects directly to a provider API. + +## Send And Edit Flow + +When the user sends a message, AI Chat: + +1. validates and stores the user message in the active thread +2. runs the selected query with `prompt.value` and `message.value` +3. validates the returned assistant message +4. stores the assistant reply in the same thread + +When the user edits an earlier message, the component truncates the later messages, inserts the edited message, reruns the selected query, and stores the newly generated reply. + +## Events + +The component emits: + +- `componentLoad` when the component initializes +- `messageSent` after the selected query succeeds for a new or edited user message +- `messageReceived` when an assistant message is stored, including the component's generic error reply +- `threadCreated` when a thread is created from the thread list +- `threadUpdated` when an initial title is generated or a thread is renamed +- `threadDeleted` when a thread is deleted + +Because `messageSent` occurs after a successful query, it is not an immediate before-request hook. + +## Errors And Debugging + +If the selected query fails or returns an invalid message, the chat displays a generic assistant error: + +```text +Sorry, I encountered an error. Please try again. +``` + +The original provider or adapter error is not displayed in the chat bubble. Inspect the failed REST/JavaScript query result and the browser console while developing. + +Common causes are: + +- the REST query, rather than the adapter, was selected in **Chat Query** +- the adapter did not return a top-level `role: "assistant"` +- the REST body used `args.prompt.value` or `args.message.value` +- provider history included Lowcoder-only fields such as `timestamp` +- an API key, model ID, URL, or required header is incorrect +- Ollama is not reachable from the Lowcoder backend + +The current component waits for the selected query to finish and then appends the complete reply. Token-by-token streaming into the chat is not currently implemented. + +## Local Storage + +Threads and messages are stored in an AlaSQL database backed by the browser's `localStorage`. The exposed database name has this form: + +```js +chat1.databaseName // ChatDB_ +``` + +The database contains `threads` and `messages` tables. It can be queried with AlaSQL, for example: + +```sql +SELECT * FROM ChatDB_chat1234.messages WHERE threadId = 'thread-123' +``` + +This storage is local to the browser and device. It is not shared between users, and clearing site data removes it. If transcripts must be durable or shared, persist them to an external database using component events and `conversationHistory`. + +## AI Chat Versus Chat Box + +Use **AI Chat** for: + +- one user conversing with an assistant +- multiple local assistant threads +- prompt/response workflows +- provider-backed AI interfaces + +Use **Chat Box** for room-based collaboration between multiple human participants. + +## Summary + +- AI Chat is provider-neutral. +- Select a JavaScript adapter as **Chat Query**. +- Read the current query input from `prompt.value` or `message.value`. +- Map the guaranteed `conversationHistory.value` query snapshot to the provider's schema. +- Make the adapter return `{ role: "assistant", content: "..." }`. +- Add provider-specific handling before sending attachments. +- Persist transcripts externally if browser-local storage is not sufficient. diff --git a/docs/build-applications/app-editor/visual-components/chat-box.md b/docs/build-applications/app-editor/visual-components/chat-box.md new file mode 100644 index 0000000000..7619f8bc83 --- /dev/null +++ b/docs/build-applications/app-editor/visual-components/chat-box.md @@ -0,0 +1,874 @@ +# Chat Box + +The component **Chat Box** provides a ready-made chat interface for room-based conversations. + +It includes: + +- a message list with per-role styling for your own, other people's, and AI messages +- a message composer with `@`-mentions and file attachments +- a rooms sidebar with public, private, and AI rooms +- typing indicators +- online user indicators +- room creation and leave actions +- invite actions for private rooms + +It is designed to work best together with **Chat Controller**, while your own queries handle the actual room and message records. + +> **Before you start:** the realtime features β€” typing indicators, online users, live room switching β€” come from **Chat Controller**, which needs a Hocuspocus server running and reachable from the browser. Without it the component still renders, but nothing syncs between users. +> +> See [Realtime Shared State and Presence](../../realtime-collaboration.md) for setup, deployment, and troubleshooting. + +## What Chat Box Does + +`Chat Box` is the UI layer for a chat application. + +It renders: + +- the currently active room +- the messages for that room +- the sidebar of rooms +- typing users +- room-level online presence +- create, leave, and invite interactions + +It does **not** persist messages or rooms by itself. + +Instead, the usual architecture is: + +- **Chat Box** handles chat UI and emits events +- **Chat Controller** handles realtime collaboration state +- **Queries** load and save rooms, invites, and messages + +## Recommended Architecture + +The most common pairing is: + +- `chatBox1` +- `chatController1` +- `loadRooms` +- `loadMessages` +- `saveMessage` +- optional room and invite queries + +In that setup: + +- `chatController1.currentRoomId` is the active room source of truth +- `chatBox1` is bound to the room and message data from your queries +- `chatBox1` events trigger controller methods and data queries + +## Core Bindings + +Bind the component like this. + +### Content Bindings + +`Messages` + +```js +{{ loadMessages.data || [] }} +``` + +`Current User ID` + +```js +{{ chatController1.userId }} +``` + +`Current User Name` + +```js +{{ chatController1.userName }} +``` + +### Realtime Bindings + +`Typing Users` + +```js +{{ chatController1.typingUsers }} +``` + +`Online Users` + +```js +{{ chatController1.onlineUsers }} +``` + +`AI Is Thinking` + +```js +{{ !!chatController1.aiThinkingRooms?.[chatController1.currentRoomId] }} +``` + +### Rooms Bindings + +`Rooms` + +```js +{{ loadRooms.data || [] }} +``` + +`Current Room ID` + +```js +{{ chatController1.currentRoomId }} +``` + +`Pending Invites` + +```js +{{ loadInvites.data || [] }} +``` + +`Show Rooms Panel` + +```js +true +``` + +### Optional Bindings + +`Mention Candidates` β€” the `@`-mention list, when you want it to include people who are not currently online: + +```js +{{ loadTeamMembers.data.map(u => ({ id: u.id, label: u.fullName })) }} +``` + +## Room Types + +The component supports these room types: + +- `public` +- `private` +- `llm` + +### Public Rooms + +Use `public` for standard shared chat spaces. + +Typical examples: + +- General +- Announcements +- Team coordination channels + +### Private Rooms + +Use `private` when access depends on membership. + +Typical examples: + +- project sub-teams +- manager-only rooms +- rooms where invite flows matter + +These rooms usually work together with: + +- `pendingInvites` +- `inviteSend` +- `inviteAccept` +- `inviteDecline` + +### LLM Rooms + +Use `llm` when the room is associated with an AI workflow. + +The room can store an `llmQueryName` field so your app knows which query or backend flow should respond in that room. + +The Chat Box only records whether the user **@-mentioned the AI** on the last send (`chatBox1.lastSentMessageTagsLlm`). Your app should **run the mapped LLM query only when that flag is true**, so normal messages in an LLM room stay human-to-human until someone tags the assistant. + +Typical examples: + +- GPT assistant +- support bot room +- internal knowledge assistant room + +The UI can then show AI thinking state through: + +```js +{{ !!chatController1.aiThinkingRooms?.[chatController1.currentRoomId] }} +``` + +## Expected Data Shape + +### Messages + +Each message should look like: + +```json +{ + "id": "msg_123", + "text": "Hello", + "authorId": "user_1", + "authorName": "Alice", + "timestamp": 1710000000000, + "authorType": "user" +} +``` + +For assistant responses, `authorType` can be: + +```json +{ + "authorType": "assistant" +} +``` + +A message renders as an AI message when `authorType` is `"assistant"` **or** when `authorId` is `"__llm_bot__"`. Otherwise it renders as your own message when `authorId` matches `Current User ID`, and as someone else's message in every other case. + +#### Field Aliases + +You rarely need to reshape your query result, because the renderer accepts common alternative field names. The first one present wins: + +| Purpose | Accepted keys | +| --- | --- | +| Message id | `id`, `_id` | +| Message body | `text`, `message`, `content` | +| Author id | `authorId`, `userId`, `author_id`, `sender` | +| Author name | `authorName`, `userName`, `author_name`, `senderName` | +| Role | `authorType`, `role` | +| Attachments | `files`, `attachments` | + +A message with no id falls back to its index, and a message with neither text nor attachments renders a placeholder rather than an empty bubble. + +#### Attachments On A Message + +To show files on a stored message, add a `files` (or `attachments`) array. Each entry needs a URL and a name: + +```json +{ + "id": "msg_124", + "text": "Here is the report", + "authorId": "user_1", + "authorName": "Alice", + "timestamp": 1710000000000, + "files": [ + { "name": "q3-report.pdf", "url": "https://files.example.com/q3.pdf", "type": "application/pdf" } + ] +} +``` + +- `url` is required β€” entries without one are skipped. `href`, `link`, and `src` are accepted as aliases. +- `name` defaults to `file`; `title` and `fileName` are accepted as aliases. +- Entries that look like images β€” by `type` or by file extension β€” render as inline thumbnails. Everything else renders as a file chip. + +These are **links**, so you need somewhere to host the file. Uploading it is your app's job; see [Sending Attachments](#sending-attachments) below. + +### Rooms + +Each room should look like: + +```json +{ + "id": "room_123", + "name": "General", + "type": "public", + "description": "Team chat", + "members": ["user_1", "user_2"], + "createdBy": "user_1", + "createdAt": 1710000000000, + "llmQueryName": null +} +``` + +### Pending Invites + +Each invite should look like: + +```json +{ + "id": "invite_123", + "roomId": "room_123", + "roomName": "Design Team", + "fromUserId": "user_1", + "fromUserName": "Alice", + "toUserId": "user_2", + "timestamp": 1710000000000 +} +``` + +## Exposed Variables + +`chatBox1` exposes state created by user interaction. These values are the bridge between the UI and your query/controller logic. + +### Message State + +- `{{ chatBox1.lastSentMessageText }}` +- `{{ chatBox1.lastSentMessageTagsLlm }}` +- `{{ chatBox1.messageText }}` + +Use `lastSentMessageText` when saving a message after `messageSent`. + +Use `lastSentMessageTagsLlm` when deciding whether to run the room's LLM query β€” see [Mentions](#mentions). + +Use `messageText` when you want the current draft value. It updates on every keystroke, so it is the live draft, not the sent message. + +### Title And Attachment State + +- `{{ chatBox1.chatTitle }}` +- `{{ chatBox1.files }}` +- `{{ chatBox1.value }}` + +`files` and `value` are the composer's pending attachments; see [Attachments](#attachments). + +### Room Action State + +- `{{ chatBox1.pendingRoomId }}` +- `{{ chatBox1.newRoomName }}` +- `{{ chatBox1.newRoomType }}` +- `{{ chatBox1.newRoomDescription }}` +- `{{ chatBox1.newRoomLlmQuery }}` + +These are populated when the user interacts with the rooms UI. + +### Invite State + +- `{{ chatBox1.inviteTargetUserId }}` +- `{{ chatBox1.pendingInviteId }}` + +These are populated when the user sends or handles invites. + +## Mentions + +The composer has an `@`-mention picker. Typing `@` opens a list; picking an entry inserts the display name as plain text, so a sent message reads naturally: + +``` +@Alice can you check this? @AI summarize the thread +``` + +### Where The Candidate List Comes From + +By default the picker is assembled automatically: + +1. **AI** β€” always offered, so any room can tag the assistant +2. members of the current room, resolved to their display name when they are online +3. other online users in the same room + +Set the **Mention Candidates** property to override that list with your own. Each entry is an object; `id` (or `userId`) and `label` (or `userName` / `name`) are read, and AI is appended to whatever you supply: + +```js +{{ loadTeamMembers.data.map(u => ({ id: u.id, label: u.fullName })) }} +``` + +This is the property to use when you want to `@`-mention people who are not currently connected. + +### Detecting An AI Mention + +`{{ chatBox1.lastSentMessageTagsLlm }}` is `true` when the message that was just sent tagged the assistant. It recognizes both forms: + +- the plain display name inserted by the picker, e.g. `@AI` +- the serialized token `@[AI](u:__llm_bot__)`, kept for messages written before the plain form + +This flag is the gate for LLM rooms: a message in an AI room stays human-to-human until someone tags the assistant. See [LLM Room Flow](#llm-room-flow). + +## Attachments + +The composer can carry file attachments, using the same model as the [File Upload](document-and-file-management/file-upload.md) component. + +### Properties + +| Property | Purpose | +| --- | --- | +| **Allow File Upload** | Shows the paperclip button in the composer. On by default. | +| **Max Files** | Cap on pending attachments. Older ones drop off once the cap is passed. Defaults to `10`. | +| **File Type** | Accepted types, as an array, e.g. `[".png", ".pdf"]`. Empty means anything. | + +### Exposed Variables + +- `{{ chatBox1.files }}` β€” metadata per pending attachment: `uid`, `name`, `type`, `size`, `lastModified` +- `{{ chatBox1.value }}` β€” the base64 contents, aligned index-for-index with `files` + +Both arrays hold the composer's **pending** attachments and are cleared automatically once the message is sent. + +### Method + +```js +chatBox1.openMessageFilePicker() +``` + +Opens the same native file picker as the paperclip button β€” useful for a drag-and-drop zone or a custom "Attach" button elsewhere in your app. + +### Event + +`fileUpload` fires after files finish being read into `files` / `value`. Use it to upload them somewhere and keep the resulting URLs. + +### Sending Attachments + +Chat Box does not host files. The flow is yours to wire: + +1. the user picks files β€” `fileUpload` fires, and `chatBox1.files` / `chatBox1.value` fill in +2. on `fileUpload`, upload them to your storage and **store the returned URLs in a temporary state**, for example a temporary state named `pendingAttachments` +3. on `messageSent`, save the message with a `files` array built from `pendingAttachments` +4. clear `pendingAttachments`, then re-run `loadMessages` β€” the message renders with its attachments + +> **Warning:** Do not read `chatBox1.files` inside the `messageSent` handler. The composer clears its pending attachments as part of sending, so that value is being emptied at the same moment your handler runs. Capture what you need during `fileUpload` instead. + +> **Note:** A message with attachments but no text is valid β€” the send button accepts it, and `messageSent` fires with `lastSentMessageText` empty. If your save query requires text, check your captured attachment state rather than rejecting every empty-text send. + +## Appearance + +### Layout Properties + +| Property | Purpose | +| --- | --- | +| **Chat Title** | Header title. Overridden by the room name whenever a room is active. | +| **Show Header** | Hides the header bar entirely. | +| **Show Rooms Panel** | Hides the sidebar β€” use this when your app drives room selection with its own UI. | +| **Panel Width** | Sidebar width, e.g. `240px`. | +| **Allow Room Creation** | Shows the `+` button and the create-room modal. | +| **Allow Room Search** | Shows the search field above the room list. | +| **Auto Height** | Standard auto / fixed height switch. | + +### Style Sections + +Each region of the component is themed separately: **Style** (container), **Sidebar**, **Header**, **Message Area**, **Input Area**, **Input Field**, **Send Button**, **Attach Button**, and **Animation**. + +**Message Style** and **Avatar Style** each have a three-way switch for **Own**, **Other**, and **AI**, so the three kinds of message bubbles and avatars can be styled independently. + +## Events + +The component emits: + +- `messageSent` +- `startTyping` +- `stopTyping` +- `roomSwitch` +- `roomLeave` +- `roomCreate` +- `inviteSend` +- `inviteAccept` +- `inviteDecline` +- `fileUpload` + +These events are the main integration points for your queries and Chat Controller methods. + +## Typical Message Flow + +The normal message flow is: + +1. the user types in the chat input +2. `chatBox1` emits `startTyping` +3. the user sends the message +4. `chatBox1.lastSentMessageText` and `chatBox1.lastSentMessageTagsLlm` become available +5. `chatBox1` emits `messageSent` +6. your `saveMessage` query stores the message +7. your app tells connected users to refresh +8. `loadMessages` runs again + +### Example Save Payload + +Typical message payload: + +```json +{ + "id": "{{ Date.now() }}_{{ Math.random().toString(36).slice(2, 8) }}", + "roomId": "{{ chatController1.currentRoomId }}", + "text": "{{ chatBox1.lastSentMessageText }}", + "authorId": "{{ chatController1.userId }}", + "authorName": "{{ chatController1.userName }}", + "timestamp": {{ Date.now() }}, + "authorType": "user" +} +``` + +### Recommended `messageSent` Wiring + +```js +saveMessage + .run() + .then(() => { + chatController1.setRoomData( + chatController1.currentRoomId, + "messagePing", + { + roomId: chatController1.currentRoomId, + ts: Date.now(), + authorId: chatController1.userId + } + ); + return loadMessages.run(); + }) + .then(() => chatController1.stopTyping()) +``` + +This pattern works well because: + +- the real message is persisted by your query +- the lightweight room signal tells other users to reload + +## Typing Indicator Flow + +`Chat Box` already emits: + +- `startTyping` +- `stopTyping` + +The simplest wiring is: + +### `startTyping` + +```js +chatController1.startTyping(chatController1.currentRoomId) +``` + +### `stopTyping` + +```js +chatController1.stopTyping() +``` + +Other users then see the indicator through: + +```js +{{ chatController1.typingUsers }} +``` + +## Room Switching Flow + +When a user clicks a room, the room id is exposed through: + +```js +{{ chatBox1.pendingRoomId }} +``` + +Recommended wiring: + +```js +chatController1.stopTyping() +chatController1.switchRoom(chatBox1.pendingRoomId) +``` + +Then on `chatController1 -> roomSwitched`, reload messages: + +```js +loadMessages.run() +``` + +This keeps the active room state centralized in `chatController1.currentRoomId`. + +## Room Creation Flow + +When the user creates a room, `chatBox1` exposes: + +- `chatBox1.newRoomName` +- `chatBox1.newRoomType` +- `chatBox1.newRoomDescription` +- `chatBox1.newRoomLlmQuery` + +Typical create-room payload: + +```json +{ + "id": "{{ Date.now() }}_{{ Math.random().toString(36).slice(2, 8) }}", + "name": "{{ chatBox1.newRoomName }}", + "type": "{{ chatBox1.newRoomType }}", + "description": "{{ chatBox1.newRoomDescription }}", + "members": ["{{ chatController1.userId }}"], + "createdBy": "{{ chatController1.userId }}", + "createdAt": {{ Date.now() }}, + "llmQueryName": "{{ chatBox1.newRoomLlmQuery }}" +} +``` + +Recommended event handling: + +```js +createRoom.run().then(() => loadRooms.run()) +``` + +## Leave Room Flow + +When a user leaves a room, the target room is exposed through: + +```js +{{ chatBox1.pendingRoomId }} +``` + +Typical handling: + +1. update that room's membership in your data store +2. reload rooms +3. clear or switch the current room if needed + +## Invite Flow + +For private rooms, `chatBox1` can support invites. + +### Invite Send + +Use: + +```js +{{ chatBox1.inviteTargetUserId }} +``` + +to create an invite record or trigger your backend flow. + +### Invite Accept / Decline + +Use: + +```js +{{ chatBox1.pendingInviteId }} +``` + +to resolve the invite record, update room membership, and reload invites. + +## LLM Room Flow + +For `llm` rooms, the room record can carry: + +```js +llmQueryName +``` + +This gives your app a room-level mapping to the AI query or workflow that should run there. + +A common pattern is: + +1. user sends a message in an LLM room +2. your app detects the room type or `llmQueryName`, and **`chatBox1.lastSentMessageTagsLlm === true`** (user @-mentioned the AI) +3. you trigger the AI query +4. set thinking state: + +```js +chatController1.setAiThinking(chatController1.currentRoomId, true) +``` + +5. save the assistant message after the AI response +6. clear thinking state: + +```js +chatController1.setAiThinking(chatController1.currentRoomId, false) +``` + +## Running The LLM Query + +For `llm` rooms, the important detail is that `chatBox1` stores the room metadata, but it does not automatically execute the query named in `llmQueryName`. + +That execution flow is still something you wire yourself in the app. + +The usual pattern is: + +1. save the user's message first +2. resolve the current room +3. check whether it is an `llm` room +4. if it is, run the mapped AI query +5. save the assistant response as a normal chat message +6. notify connected users to reload + +### 1. Store The Query Name On The Room + +When creating an LLM room, save: + +```js +{{ chatBox1.newRoomLlmQuery }} +``` + +into the room's `llmQueryName` field. + +This should be the exact Lowcoder query name you want the room to use. + +Examples: + +- `askSupportBot` +- `runKnowledgeAssistant` +- `generateAiReply` + +### 2. Resolve The Current Room + +After the user sends a message, resolve the active room from your room data: + +```js +{{ (loadRooms.data || []).find(r => r.id === chatController1.currentRoomId) }} +``` + +Then inspect: + +- `currentRoom.type` +- `currentRoom.llmQueryName` + +### 3. Decide Whether AI Should Run + +Typical condition: + +```js +currentRoom?.type === "llm" && + currentRoom?.llmQueryName && + chatBox1.lastSentMessageTagsLlm === true +``` + +If that condition is false, stop after the normal save/reload flow (the message is still saved like any other chat message). + +If it is true, continue into the AI flow. + +### 4. Choose How To Run The Query + +There are two common approaches. + +#### Fixed AI Query + +If all AI rooms use the same logic, you can keep one explicit AI query such as: + +- `runLlmReply` + +In that case: + +1. save the user message +2. run `runLlmReply` +3. save the assistant response + +This is the simplest option. + +#### Mapped AI Queries Per Room + +If different LLM rooms should trigger different AI workflows, use `llmQueryName` as the room-level mapping. + +In that case: + +1. save the user message +2. read `currentRoom.llmQueryName` +3. route to the matching query +4. save the assistant response + +Because event handlers usually call explicit queries, the cleanest implementation is often a small dispatcher flow that decides which AI query to run based on `llmQueryName`. + +### 5. What The AI Query Should Receive + +Useful inputs usually include: + +- the active room id +- the current room record +- the latest user message +- optional conversation history for that room + +Typical values: + +```js +{{ chatController1.currentRoomId }} +{{ chatBox1.lastSentMessageText }} +{{ chatBox1.lastSentMessageTagsLlm }} +{{ (loadRooms.data || []).find(r => r.id === chatController1.currentRoomId) }} +{{ loadMessages.data || [] }} +``` + +### 6. Save The Assistant Reply + +When the AI query returns, save the assistant response as a normal chat message. + +Typical assistant payload: + +```json +{ + "id": "{{ Date.now() }}_{{ Math.random().toString(36).slice(2, 8) }}", + "roomId": "{{ chatController1.currentRoomId }}", + "text": "{{ llmQuery.data.message?.content || llmQuery.data.content || llmQuery.data.text }}", + "authorId": "__llm_bot__", + "authorName": "AI", + "timestamp": {{ Date.now() }}, + "authorType": "assistant" +} +``` + +The message will render as an assistant message when: + +- `authorType === "assistant"` + +or when: + +- `authorId === "__llm_bot__"` + +### 7. Notify Connected Users Again + +After saving the assistant reply, notify the room again: + +```js +chatController1.setRoomData( + chatController1.currentRoomId, + "messagePing", + { + roomId: chatController1.currentRoomId, + ts: Date.now(), + authorId: "__llm_bot__" + } +) +``` + +Then clear AI thinking: + +```js +chatController1.setAiThinking(chatController1.currentRoomId, false) +``` + +### Recommended Full LLM Room Sequence + +The full flow usually looks like this: + +1. user sends a message in `chatBox1` +2. save the user message +3. notify the room to reload +4. resolve the current room +5. if the room is an `llm` room **and** `chatBox1.lastSentMessageTagsLlm` is true: +6. set AI thinking to `true` +7. run the AI query mapped by `llmQueryName` +8. save the assistant response +9. notify the room again +10. set AI thinking to `false` +11. reload messages + +If the room is `llm` but the user did not @-mention the AI, skip steps 6–10 and only keep the normal chat persistence/reload path. + +### Practical Note + +If you only need one AI behavior, a single fixed LLM query is easier to maintain. + +If you need several AI room behaviors, `llmQueryName` is the right place to store the mapping, but it is usually best combined with a dispatcher layer rather than trying to dynamically call arbitrary queries directly from the event handler. + +## Working With Chat Controller + +`Chat Box` is the UI. `Chat Controller` is the realtime collaboration layer. + +That means: + +- `chatController1.currentRoomId` is the active room source of truth +- `chatController1.typingUsers` drives typing state +- `chatController1.onlineUsers` drives online presence +- `chatController1.sharedState` is app-wide shared data +- `chatController1.roomData` is room-scoped shared data + +For a full controller reference, see [Chat Controller](chat-controller.md). For the server it needs, see [Realtime Shared State and Presence](../../realtime-collaboration.md). + +## Building A Custom Chat With Standard Components + +If you want a completely custom UI, you can still use the same room and realtime model. + +Typical setup: + +1. use a `List View` to render rooms +2. use `currentItem.id` inside the room row to select a room +3. call: + +```js +chatController1.switchRoom(currentItem.id) +``` + +4. load messages filtered by: + +```js +{{ chatController1.currentRoomId }} +``` + +5. show typing users and online users from `chatController1` +6. use `chatController1.setRoomData(...)` for lightweight room notifications + +This gives you the same realtime collaboration layer even if you do not use the built-in Chat Box layout. + +## Summary + +- **Chat Box** is the room-based chat UI component +- it supports `public`, `private`, and `llm` rooms +- it exposes the key user interactions as variables and events +- `@`-mentions gate the AI: `lastSentMessageTagsLlm` tells you when to run the LLM query +- attachments are picked up by the composer, but hosting and linking them is your app's job +- it works best together with **Chat Controller**, which needs a running realtime server β€” see [Realtime Shared State and Presence](../../realtime-collaboration.md) +- your own queries remain responsible for loading and saving rooms, messages, and invites diff --git a/docs/build-applications/app-editor/visual-components/chat-controller.md b/docs/build-applications/app-editor/visual-components/chat-controller.md new file mode 100644 index 0000000000..5aefa6c9eb --- /dev/null +++ b/docs/build-applications/app-editor/visual-components/chat-controller.md @@ -0,0 +1,626 @@ +# Chat Controller + +The component **Chat Controller** provides the realtime state layer for chat experiences. + +It is designed to work with: + +- **Chat Box** +- a custom room-based chat UI built from standard components such as `List View`, `Container`, `Input`, and `Button` + +> **Before you start:** this component needs a Hocuspocus server running and reachable from the browser. Without it, `ready` stays `false` and every shared value stays empty. +> +> See [Realtime Shared State and Presence](../../realtime-collaboration.md) for setup, deployment, and troubleshooting. + +## What Chat Controller Does + +`Chat Controller` manages the realtime collaboration layer of a chat application. + +That includes: + +- connection state +- current user presence +- active room state +- typing indicators +- online users +- room-scoped shared data +- app-wide shared data +- AI thinking indicators + +It does **not** replace your message or room queries. + +Instead, it works alongside them: + +- your queries persist and load the real room/message data +- `Chat Controller` synchronizes the live collaborative state across connected clients + +## Core Idea + +All users connected with the same `applicationId` share the same realtime collaboration channel. + +Inside that channel, the controller keeps synchronized: + +- who is online +- who is typing +- which room each user is currently viewing +- app-level shared data +- room-level shared data +- AI thinking state + +This makes it a good fit for: + +- live room switching +- presence indicators +- typing indicators +- lightweight notifications like β€œroom changed, reload messages” + +## Before You Start: The Realtime Server + +The controller talks to a **Hocuspocus** WebSocket server. Without that server running and reachable from the browser, the component loads but never connects: `ready` stays `false` and every shared value stays empty. + +In a default Docker install the service is already there. For local development, start it with: + +```bash +node client/packages/lowcoder/hocuspocus-server.js +``` + +and make sure the frontend was built with `REACT_APP_HOCUSPOCUS_URL` pointing at it (it defaults to `ws://localhost:3006`). + +Full setup, deployment, and troubleshooting instructions are in [Realtime Shared State and Presence](../../realtime-collaboration.md). + +## How It Works Under The Hood + +Internally, the controller uses **Hocuspocus** and **Yjs** for synchronization. + +In practice, that means: + +- presence β€” online users and typing flags β€” is shared through Yjs **awareness**, which is ephemeral and vanishes when a client disconnects +- app-wide shared objects are synchronized through a shared Yjs map +- room-scoped shared objects are synchronized through another shared Yjs map + +Each app gets its own Yjs document, named `signal_`. Apps with different `applicationId` values never see each other's state. + +The server-side transport is implemented in [`hocuspocus-server.js`](https://github.com/lowcoder-org/lowcoder/blob/main/client/packages/lowcoder/hocuspocus-server.js). + +> **Warning:** The bundled server holds documents **in memory only**. On a server restart, `sharedState` and `roomData` are lost. Use them for live signalling, and keep anything that must survive a restart in your own database. + +## Main Properties + +Configure: + +- `applicationId` +- `userId` +- `userName` + +### `applicationId` + +This scopes the shared collaboration space. Users with the same `applicationId` participate in the same realtime channel. Defaults to `lowcoder_app`. + +> **Warning:** Leaving the default means every app that also left the default shares one collaboration space on the same server. Set a value that is unique to your app. + +### `userId` + +This identifies the current user. It must be **unique per person** β€” two browsers using the same `userId` are indistinguishable to the presence layer. Binding it to the logged-in user is the usual choice: + +```js +{{ currentUser.id }} +``` + +### `userName` + +This is the display name used in presence and typing indicators. + +```js +{{ currentUser.name }} +``` + +Both can also be changed at runtime with `setUser(userId, userName)`. + +### Identity Is Not Verified + +The controller trusts whatever `userId` and `userName` the client sends. Anyone who can open the app can present themselves as any user to the realtime layer. Keep authorization decisions in your queries and backend β€” never rely on `onlineUsers` as proof of who someone is. + +## Exposed State + +The controller exposes the following state values. + +### Connection State + +- `{{ chatController1.ready }}` +- `{{ chatController1.error }}` +- `{{ chatController1.connectionStatus }}` + +Use these when you want to: + +- show whether the realtime layer is online +- display connection status +- surface authentication or transport errors + +`ready` is a boolean β€” `true` only while the WebSocket is open. `connectionStatus` is the human-readable label `Connecting...`, `Online`, or `Offline`. `error` is `null` while healthy, and carries the failure reason after a failed authentication (which also fires the `error` event). + +Gate anything that writes shared state on `ready`, otherwise the write silently does nothing: + +```js +{{ chatController1.ready }} +``` + +### Presence State + +- `{{ chatController1.onlineUsers }}` +- `{{ chatController1.typingUsers }}` +- `{{ chatController1.currentRoomId }}` + +### AI State + +- `{{ chatController1.aiThinkingRooms }}` + +### Shared Data State + +- `{{ chatController1.sharedState }}` +- `{{ chatController1.roomData }}` + +### Identity State + +The controller also re-exposes its own identity settings, which is convenient when other components need them without duplicating the bindings: + +- `{{ chatController1.userId }}` +- `{{ chatController1.userName }}` +- `{{ chatController1.applicationId }}` + +## State Shapes + +### `onlineUsers` + +`onlineUsers` is an array of currently connected peers. It lists **other** users only β€” the current user is not included, so add yourself when rendering a presence list. + +Shape: + +```js +[ + { + userId: "user_1", + userName: "Alice", + currentRoomId: "room_123" + } +] +``` + +This is useful for: + +- online user sidebars +- room presence counts +- showing whether another user is active in the same room + +### `typingUsers` + +`typingUsers` is scoped to the current room and excludes the current user. + +Shape: + +```js +[ + { + userId: "user_2", + userName: "Bob", + roomId: "room_123" + } +] +``` + +This is useful for: + +- β€œBob is typing...” +- β€œ2 people are typing...” + +### `currentRoomId` + +This is the active room for the current user. + +It is typically the room id used by: + +- `loadMessages` +- room highlighting +- typing presence +- AI thinking indicators + +### `aiThinkingRooms` + +This is a map of room ids to thinking state. + +Example: + +```js +{ + room_123: true, + room_456: false +} +``` + +This is useful when you have `llm` rooms or an AI assistant integrated into a room-based chat flow. + +### `sharedState` + +`sharedState` is app-level shared data. + +Example: + +```js +{ + activeAnnouncement: "Maintenance at 7 PM", + featureFlagAiRooms: true +} +``` + +This is for values that belong to the application as a whole rather than to a single room. + +### `roomData` + +`roomData` is room-scoped shared data. + +Its shape is: + +```js +{ + room_123: { + messagePing: { ts: 1710000000000, authorId: "user_1" }, + lastMessage: { text: "Hello", authorId: "user_1" } + } +} +``` + +This is for values that should be scoped to a specific room only. + +## Shared State vs Room Data + +This is the most important conceptual distinction in the controller. + +### Use `sharedState` For App-Level Collaboration + +Typical examples: + +- app-wide announcement banners +- feature flags +- collaborative state not tied to a room +- global session metadata + +Write with: + +```js +chatController1.setSharedState(key, value) +``` + +### Use `roomData` For Room-Level Collaboration + +Typical examples: + +- `messagePing` telling room clients to reload messages +- room-specific last message preview +- room-scoped draft metadata +- room-specific AI assistant state + +Write with: + +```js +chatController1.setRoomData(roomId, key, value) +``` + +## Events + +The controller emits: + +- `userJoined` +- `userLeft` +- `roomSwitched` +- `connected` +- `disconnected` +- `error` +- `aiThinkingStarted` +- `aiThinkingStopped` +- `sharedStateChanged` +- `roomDataChanged` + +These events are the main hooks for reacting to realtime updates. + +## Methods + +The controller exposes the following methods. + +### Presence Methods + +`startTyping(roomId?)` + +Marks the current user as typing in the given room or the current room. + +`stopTyping()` + +Clears the typing flag for the current user. + +`switchRoom(roomId)` + +Updates the active room and moves the current user's presence to that room. + +### AI Methods + +`setAiThinking(roomId, isThinking)` + +Marks whether an AI is currently thinking in the given room, for every connected user. + +`isThinking` is read as true only for the boolean `true` or the string `"true"`; anything else counts as false. + +### App-Level Shared State Methods + +`setSharedState(key, value)` + +Sets an app-wide shared value. + +`deleteSharedState(key)` + +Removes an app-wide shared value. + +### Room-Level Shared State Methods + +`setRoomData(roomId, key, value)` + +Sets a room-scoped shared value. + +`deleteRoomData(roomId, key?)` + +Removes a single room-scoped key or clears the entire room entry. + +### User Method + +`setUser(userId, userName)` + +Updates the identity used by the controller. + +## Typical Patterns + +### Typing Indicator Pattern + +On input change: + +```js +chatController1.startTyping(chatController1.currentRoomId) +``` + +On blur, send, room switch, or inactivity: + +```js +chatController1.stopTyping() +``` + +### Room Switch Pattern + +When a user selects a room: + +```js +chatController1.switchRoom(roomId) +``` + +Then on `roomSwitched`, reload the messages for: + +```js +{{ chatController1.currentRoomId }} +``` + +### Message Reload Pattern + +After saving a message, notify the room: + +```js +chatController1.setRoomData( + chatController1.currentRoomId, + "messagePing", + { + roomId: chatController1.currentRoomId, + ts: Date.now(), + authorId: chatController1.userId + } +) +``` + +Then on `roomDataChanged`, reload messages. + +### Room Presence Pattern + +To show users currently in the active room: + +```js +{{ + (chatController1.onlineUsers || []).filter( + u => u.currentRoomId === chatController1.currentRoomId + ) +}} +``` + +### LLM Room Pattern + +If you support `llm` rooms, use: + +```js +chatController1.setAiThinking(roomId, true) +``` + +when the AI starts generating, and: + +```js +chatController1.setAiThinking(roomId, false) +``` + +when the AI finishes. + +This makes the thinking state visible to everyone in that room. + +### LLM Query Execution Pattern + +`Chat Controller` does not execute the room's LLM query by itself. + +Instead, it supports the realtime pieces around that AI execution: + +1. a user sends a message +2. the app saves the user message +3. the app checks whether the current room is an `llm` room +4. the app runs the AI query mapped to that room +5. the app saves the assistant response +6. the controller shares thinking and refresh state with everyone else in the room + +In other words: + +- your query layer executes the AI logic +- `Chat Controller` keeps the room state synchronized while that happens + +Typical sequence: + +set AI thinking: + +```js +chatController1.setAiThinking(chatController1.currentRoomId, true) +``` + +run the AI query + +save the assistant message + +notify the room: + +```js +chatController1.setRoomData( + chatController1.currentRoomId, + "messagePing", + { + roomId: chatController1.currentRoomId, + ts: Date.now(), + authorId: "__llm_bot__" + } +) +``` + +clear AI thinking: + +```js +chatController1.setAiThinking(chatController1.currentRoomId, false) +``` + +This lets every connected user in that room see: + +- that the AI is currently working +- that a new assistant reply has arrived + +## Using Chat Controller With Chat Box + +Recommended pairing: + +- `chatBox1.currentRoomId` -> `{{ chatController1.currentRoomId }}` +- `chatBox1.typingUsers` -> `{{ chatController1.typingUsers }}` +- `chatBox1.onlineUsers` -> `{{ chatController1.onlineUsers }}` +- `chatBox1.isAiThinking` -> `{{ !!chatController1.aiThinkingRooms?.[chatController1.currentRoomId] }}` + +Typical event handling: + +- `chatBox1 -> roomSwitch` -> `chatController1.switchRoom(chatBox1.pendingRoomId)` +- `chatBox1 -> startTyping` -> `chatController1.startTyping(chatController1.currentRoomId)` +- `chatBox1 -> stopTyping` -> `chatController1.stopTyping()` + +For the UI-side details, see [Chat Box](chat-box.md). + +## Using Chat Controller With A Custom Chat UI + +When building your own room-based chat interface, the controller becomes the realtime backbone. + +Typical flow: + +1. use a `List View` to display rooms +2. select a room with: + +```js +chatController1.switchRoom(currentItem.id) +``` + +3. load messages filtered by: + +```js +{{ chatController1.currentRoomId }} +``` + +4. show typing users from `chatController1.typingUsers` +5. show online users from `chatController1.onlineUsers` +6. use `roomData` to signal room refreshes or store room-scoped metadata + +## Beyond Chat: General Realtime Features + +Nothing in `sharedState` and `roomData` is chat-specific. Once a Chat Controller is on the canvas, any component in the app can use it as a live channel between users. + +### Live Presence On Any Screen + +Show who else has the app open, whether or not you use rooms at all: + +```js +{{ (chatController1.onlineUsers || []).map(u => u.userName).join(", ") }} +``` + +### "Who Is Looking At What" + +Presence carries a fixed set of fields, so anything custom belongs in `sharedState` rather than presence. Write the current user's focus on selection: + +```js +chatController1.setSharedState("viewing_" + chatController1.userId, { + userName: chatController1.userName, + recordId: table1.selectedRow.id, + ts: Date.now() +}) +``` + +Then read every peer's entry back out of `sharedState`, and clear your own key when the user leaves the screen: + +```js +chatController1.deleteSharedState("viewing_" + chatController1.userId) +``` + +> **Warning:** Unlike presence, `sharedState` is **not** cleaned up when a user disconnects. A key written by someone who then closed the tab stays until something deletes it. Store a timestamp alongside the value and ignore stale entries when you read them. + +### Broadcast A Refresh + +The cheapest way to keep several users' tables in sync is a ping rather than the data itself: + +```js +chatController1.setSharedState("dataPing", { ts: Date.now(), by: chatController1.userId }) +``` + +On `sharedStateChanged`, re-run the query that loads the table. Every connected client reloads from your real data source, so there is one source of truth and the realtime layer only carries the signal. + +## Limits To Know About + +- **No live cursors, no shared text editing.** The controller synchronizes structured key/value data and presence. It does not share caret positions or merge concurrent edits to a document. +- **Presence fields are fixed** β€” `userId`, `userName`, `currentRoomId`, `typing`. Custom per-user data goes in `sharedState`. +- **Shared maps are last-write-wins per key.** Two users writing the same key at the same moment do not merge; one value survives. Give each user their own key when they each own a piece of the state. +- **Nothing is persisted.** A server restart empties `sharedState` and `roomData`. +- **Writes before `ready` are dropped** silently. +- **`typingUsers` is filtered to the current room** and excludes the current user; `onlineUsers` spans the whole app and also excludes the current user. + +## When To Use Shared Objects + +Use the controller's shared objects when you need realtime collaboration data that should not itself be the canonical persisted chat record. + +Good use cases: + +- ephemeral collaboration metadata +- room refresh signals +- live status indicators +- app-wide flags visible to all connected users + +Less suitable use cases: + +- full persistent message history +- canonical room records +- long-term audit storage + +Those are usually better handled by your own queries and datastore. + +## Summary + +- **Chat Controller** is the realtime collaboration component β€” for chat features and for any other live shared state +- it needs a running Hocuspocus server; see [Realtime Shared State and Presence](../../realtime-collaboration.md) +- it scopes collaboration by `applicationId` +- it exposes presence, room state, app-wide shared state, and room-scoped shared state +- use `sharedState` for app-level data +- use `roomData` for room-level data +- pair it with **Chat Box** or with your own custom chat UI diff --git a/docs/build-applications/app-editor/visual-components/meeting-and-collaboration.md b/docs/build-applications/app-editor/visual-components/meeting-and-collaboration.md index 7473561ce3..e5a7cde59d 100644 --- a/docs/build-applications/app-editor/visual-components/meeting-and-collaboration.md +++ b/docs/build-applications/app-editor/visual-components/meeting-and-collaboration.md @@ -1,2 +1,17 @@ # Meeting & Collaboration +### Chat Components + +- [Chat Box](chat-box.md) β€” room-based chat UI for several people +- [AI Chat](ai-chat.md) β€” thread-based assistant chat for a single user +- [Chat Controller](chat-controller.md) β€” presence, typing, and shared live state + +Chat Box and Chat Controller need a running realtime server; see [Realtime Shared State and Presence](../../realtime-collaboration.md). + +### Video Meeting Components + +- **Agora Meeting Controller** β€” joins and controls an Agora meeting +- **Camera Stream** β€” displays one participant's camera or profile image +- **Screen Share Stream** β€” displays one participant's shared screen + +See [Build Collaborative Video Apps](../collaborative-video-apps.md) for meeting setup and patterns for presence, chat, AI, documents, forms, and shared websites. diff --git a/docs/build-applications/realtime-collaboration.md b/docs/build-applications/realtime-collaboration.md new file mode 100644 index 0000000000..ff7c11a7a9 --- /dev/null +++ b/docs/build-applications/realtime-collaboration.md @@ -0,0 +1,135 @@ +# Realtime Shared State and Presence + +Lowcoder ships a realtime collaboration layer that lets several people using the **same app** see each other and share live state. + +It powers: + +* [Chat Box](app-editor/visual-components/chat-box.md) β€” the room-based chat UI +* [Chat Controller](app-editor/visual-components/chat-controller.md) β€” the presence and shared-state component +* any custom UI you build yourself out of standard components + +This page explains what the layer actually does, what it does **not** do, and how to run it in development and in production. + +> **Note:** This is different from [collaborative video apps](app-editor/collaborative-video-apps.md). Agora provides audio, video, screen share, meeting participants, and temporary meeting signals. This page covers the Hocuspocus WebSocket service used by Chat Controller for chat presence and shared data. Explore Together modules can also use Hocuspocus through a provider-specific proxy bridge. + +## What The Layer Provides + +Everyone connected to the same collaboration space shares: + +| Capability | Where it appears | +| --- | --- | +| Who is online, and what room they are viewing | `chatController1.onlineUsers` | +| Who is typing | `chatController1.typingUsers` | +| Which room the current user is in | `chatController1.currentRoomId` | +| Whether an AI is generating a reply in a room | `chatController1.aiThinkingRooms` | +| App-wide shared key/value data | `chatController1.sharedState` | +| Room-scoped shared key/value data | `chatController1.roomData` | + +That is the full set provided by **Chat Controller itself**. In particular: + +* Chat Controller does not expose live cursors or shared-text editing. It synchronizes structured key/value data and chat presence. An [Explore Together](app-editor/collaborative-video-apps.md#6-add-explore-together) proxy bridge can add provider-specific pointers, carets, form values, or navigation to a supported external page. +* Presence carries a **fixed** set of fields β€” `userId`, `userName`, `currentRoomId`, `typing`. You cannot add your own presence fields from the property panel. +* It is **not a message store**. Chat messages, rooms, and invites are loaded and saved by your own queries. The realtime layer only tells other clients that something changed. + +If you want to share something else live β€” a selected row, a "user X is viewing record Y" indicator, a shared filter β€” use `setSharedState` / `setRoomData` instead of presence. See [Chat Controller](app-editor/visual-components/chat-controller.md). + +## Architecture + +``` +Browser A ─┐ +Browser B ─┼─ WebSocket ──▢ Hocuspocus server ──▢ Yjs document per app +Browser C β”€β”˜ (port 3006) ("signal_") +``` + +* The transport is [Hocuspocus](https://tiptap.dev/docs/hocuspocus), a WebSocket server for [Yjs](https://yjs.dev) documents. +* Each `applicationId` on the Chat Controller maps to one Yjs document, named `signal_`. Two apps with different ids never see each other's state. +* Online users and typing flags travel over **Yjs awareness**, which is ephemeral β€” it disappears the moment a client disconnects. +* `sharedState`, `roomData`, and AI thinking state are stored in **Yjs shared maps** on the document. + +The server source lives at [`client/packages/lowcoder/hocuspocus-server.js`](https://github.com/lowcoder-org/lowcoder/blob/main/client/packages/lowcoder/hocuspocus-server.js). + +> **Warning:** The bundled server keeps documents **in memory only** β€” no persistence extension is configured. If the server restarts, everything in `sharedState` and `roomData` is gone. Treat those maps as live signalling, and keep anything you actually need to survive a restart in your own database. + +## Running The Server + +The server is a small Node process. It needs no database. + +### Environment Variables (server side) + +| Variable | Default | Purpose | +| --- | --- | --- | +| `PORT` | `3006` | HTTP / WebSocket port | +| `HOST` | `0.0.0.0` | Bind address | +| `HOCUSPOCUS_SECRET` | _(empty)_ | Shared secret. When empty, **any client may connect.** | + +It answers two HTTP routes, which are handy for health checks: + +* `GET /health` β†’ `{"status":"ok","auth":"enabled"}` +* `GET /` β†’ service info + +### Local Development + +From the repository: + +```bash +node client/packages/lowcoder/hocuspocus-server.js +``` + +You should see `[hocuspocus] listening on ws://0.0.0.0:3006`. + +### Docker + +An image is published as `lowcoderorg/lowcoder-hocuspocus`. The multi-container compose file already includes it: + +```yaml +hocuspocus: + image: lowcoderorg/lowcoder-hocuspocus:latest + container_name: hocuspocus + restart: unless-stopped + environment: + # Empty is convenient for local development only. + HOCUSPOCUS_SECRET: "" +``` + +See [`deploy/docker/docker-compose-multi.yaml`](https://github.com/lowcoder-org/lowcoder/blob/main/deploy/docker/docker-compose-multi.yaml). + +> **Important:** The multi-container compose file leaves `HOCUSPOCUS_SECRET` empty for local development. Set a matching server and frontend value before exposing the service. With an empty secret the server accepts every connection and every document name, so anyone who can reach the port can read and write the shared state of any app. + +## Pointing The Client At The Server + +The frontend reads two build-time variables: + +| Variable | Default | Purpose | +| --- | --- | --- | +| `REACT_APP_HOCUSPOCUS_URL` | `ws://localhost:3006` | WebSocket URL the browser connects to | +| `REACT_APP_HOCUSPOCUS_SECRET` | _(empty)_ | Token sent on connect; must match the server's `HOCUSPOCUS_SECRET` | + +These are baked into the client bundle at build time, so they must be set when the frontend is built or served, not only on the API service. + +Behind TLS, use `wss://` and terminate the WebSocket upgrade at your reverse proxy: + +``` +REACT_APP_HOCUSPOCUS_URL=wss://collab.example.com +``` + +> **Warning:** `REACT_APP_HOCUSPOCUS_SECRET` ends up in the JavaScript bundle that browsers download. It is a deployment-level gate to keep strangers off the port β€” not a per-user credential. Do not treat it as an authorization boundary between your own app users. + +## Verifying The Setup + +1. Place a **Chat Controller** component in an app. +2. Open the app in two browser windows, using two different `userId` values. +3. Check `{{ chatController1.ready }}` β€” it should be `true`, and `{{ chatController1.connectionStatus }}` should read as online. +4. Check `{{ chatController1.onlineUsers }}` in one window β€” it should list the other window's user. + +If `ready` stays `false`: + +* confirm the server is running and `GET /health` responds; +* confirm `REACT_APP_HOCUSPOCUS_URL` is reachable from the **browser**, not just from the server; +* if the server has a secret set, confirm the client was built with the same value β€” a mismatch fires the controller's `error` event and sets `{{ chatController1.error }}`. + +## Where To Go Next + +* [Chat Controller](app-editor/visual-components/chat-controller.md) β€” full reference for presence, shared state, methods, and events +* [Chat Box](app-editor/visual-components/chat-box.md) β€” the ready-made room chat UI +* [Build Collaborative Video Apps](app-editor/collaborative-video-apps.md) β€” combine Agora meetings with chat, AI, documents, forms, or shared websites +* [AI Chat](app-editor/visual-components/ai-chat.md) β€” single-user assistant chat, which needs none of this infrastructure diff --git a/docs/build-applications/video-calls-in-lowcoder.md b/docs/build-applications/video-calls-in-lowcoder.md index 6ccd741fe7..036ba5d6f4 100644 --- a/docs/build-applications/video-calls-in-lowcoder.md +++ b/docs/build-applications/video-calls-in-lowcoder.md @@ -1,5 +1,7 @@ # Video Calls in Lowcoder +> For the Lowcoder 3.0 guide, see [Build Collaborative Video Apps](app-editor/collaborative-video-apps.md), including meeting setup, chat, AI, documents, and shared forms or websites. For Chat Controller deployment, see [Realtime Shared State and Presence](realtime-collaboration.md). + Collaboration features, especially interactive video calls, in customer-facing and internal employee apps are crucial for modern Applications. For customer-facing apps, this enhances engagement, provides immediate support, and personalizes the user experience, fostering customer satisfaction and loyalty.