diff --git a/AGENTS.md b/AGENTS.md index 3da4334..350448c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,8 +21,10 @@ Navigation is defined in `docs.json` and currently follows this top-level struct When adding or moving pages: 1. Keep `docs.json` in sync with actual files. -2. Keep pages grouped by product area (`datasets/`, `workflows/`, `storage/`, `sdks/`, `guides/`, `api-reference/`). -3. Preserve the current pattern where high-level landing pages link to deeper pages via `Card`/`HeroCard` blocks. +2. Keep page paths aligned with their navigation section. If a page appears under a named group such as `Agents and AI Tools`, `Run and Inspect`, or `Automations`, place it under the matching slugged folder (`agents-and-ai-tools/`, `workflows/run-and-inspect/`, `workflows/automations/`) so the URL reflects the sidebar structure. +3. Keep pages grouped by product area (`datasets/`, `workflows/`, `sdks/`, `guides/`, `api-reference/`). +4. Keep the User Guides nav and the [Tilebox Cookbook](/guides/cookbook) in sync. Whenever you add, remove, rename, or move a `guides/**` page in `docs.json`, update `guides/cookbook.mdx` with the same guide metadata, and vice versa. +5. Preserve the current pattern where high-level landing pages link to deeper pages via `Card`/`HeroCard` blocks. ## Diátaxis Mapping @@ -142,10 +144,12 @@ For any substantial docs update, verify all of the following: 1. Page intent matches a single Diátaxis category. 2. Page is linked in `docs.json` in the correct section. -3. Internal links resolve and `mintlify broken-links` passes. -4. Vale warnings/errors are addressed or intentionally accepted. -5. New screenshots/assets are optimized and use correct paths. -6. Language and terminology match surrounding Tilebox docs. +3. The page path, `docs.json` entry, and expected public URL match the navigation group slug. +4. If the page is under `guides/**`, the User Guides nav and `guides/cookbook.mdx` are both updated. +5. Internal links resolve and `mintlify broken-links` passes. +6. Vale warnings/errors are addressed or intentionally accepted. +7. New screenshots/assets are optimized and use correct paths. +8. Language and terminology match surrounding Tilebox docs. ## Notes On API Reference Pages diff --git a/agentic-development/agent-skills.mdx b/agents-and-ai-tools/agent-skills.mdx similarity index 92% rename from agentic-development/agent-skills.mdx rename to agents-and-ai-tools/agent-skills.mdx index 8799130..8193d87 100644 --- a/agentic-development/agent-skills.mdx +++ b/agents-and-ai-tools/agent-skills.mdx @@ -45,4 +45,4 @@ Load the relevant Tilebox skills. Use the Tilebox CLI to submit a Sentinel-2 mos Skills work best with the Tilebox CLI. The CLI gives the agent repeatable terminal commands, and skills tell it how to combine those commands safely. Use MCP as an alternative when the agent runs in a web or chat environment without practical terminal access. -For Python workflow release work, ask the agent to use both `writing-tilebox-workflows` and `releasing-tilebox-workflows`. The typical loop is to edit tasks, build and publish a release, deploy it to a development cluster, run `tilebox runner start`, submit a test job, and inspect logs or spans before iterating. +For Python workflow release work, ask the agent to use both `writing-tilebox-workflows` and `releasing-tilebox-workflows`. The typical loop is to edit tasks, build, and publish a release, deploy it to a development cluster, run `tilebox runner start`, submit a test job, and inspect logs or spans before iterating. diff --git a/agentic-development/ai-interfaces.mdx b/agents-and-ai-tools/ai-interfaces.mdx similarity index 92% rename from agentic-development/ai-interfaces.mdx rename to agents-and-ai-tools/ai-interfaces.mdx index e7ecb00..32848a7 100644 --- a/agentic-development/ai-interfaces.mdx +++ b/agents-and-ai-tools/ai-interfaces.mdx @@ -10,9 +10,9 @@ Tilebox gives agents multiple ways to get context and take action. The right int | Interface | Use it for | What it gives the agent | | --- | --- | --- | -| [Tilebox CLI](/agentic-development/tilebox-cli) | Default setup for coding agents, terminal operations, automation, and CI-friendly workflows | Structured command output, command discovery, and predictable inputs | -| [Agent Skills](/agentic-development/agent-skills) | Repeated Tilebox tasks across datasets, jobs, and automations | Instructions for composing CLI commands into end-to-end workflows | -| [Tilebox MCP](/agentic-development/tilebox-mcp) | Web-based agents, chat tools, or MCP-capable clients where terminal access is limited | Authenticated tools for datasets, workflows, and documentation search | +| [Tilebox CLI](/agents-and-ai-tools/tilebox-cli) | Default setup for coding agents, terminal operations, automation, and CI-friendly workflows | Structured command output, command discovery, and predictable inputs | +| [Agent Skills](/agents-and-ai-tools/agent-skills) | Repeated Tilebox tasks across datasets, jobs, and automations | Instructions for composing CLI commands into end-to-end workflows | +| [Tilebox MCP](/agents-and-ai-tools/tilebox-mcp) | Web-based agents, chat tools, or MCP-capable clients where terminal access is limited | Authenticated tools for datasets, workflows, and documentation search | | [Tilebox SDKs](/sdks/introduction) | Application code and production integrations | Language-native APIs for Python and Go | | [`llms-full.txt`](https://docs.tilebox.com/llms-full.txt) | Documentation-only context | A static markdown bundle of the Tilebox documentation | diff --git a/agentic-development/overview.mdx b/agents-and-ai-tools/overview.mdx similarity index 66% rename from agentic-development/overview.mdx rename to agents-and-ai-tools/overview.mdx index 68e6d6c..ed17359 100644 --- a/agentic-development/overview.mdx +++ b/agents-and-ai-tools/overview.mdx @@ -1,10 +1,10 @@ --- -title: AI Overview -description: Learn how Tilebox supports agentic development through the CLI, agent skills, the MCP server, and documentation context. +title: Agents and AI tools +description: Learn how Tilebox gives coding agents access to the same data, workflow, job, and observability tools that developers use. icon: brain-circuit --- -AI-assisted development in Tilebox is built around giving agents the same operational context that developers use: current documentation, authenticated access to Tilebox resources, and predictable tools for running actions. The section explains how to connect agents to Tilebox, which interface to use for each task, and how the Tilebox CLI and skills help agents work with datasets and workflows. +Tilebox gives agents the same operational context that developers use: current documentation, authenticated access to Tilebox resources, and predictable tools for running actions. This section explains how to connect agents to Tilebox, which interface to use for each task, and how the Tilebox CLI and skills help agents work with datasets and workflows. ## How Tilebox supports agents @@ -14,18 +14,18 @@ Tilebox supports agents through a CLI-based setup. The CLI gives agents a determ Configure an agent with the Tilebox CLI, Tilebox skills, and documentation context. - + Decide when to use MCP, the CLI, skills, SDKs, or documentation context. - - Connect AI tools to the Tilebox MCP server when terminal access is not available. - - + Install the CLI, authenticate it, and use agent-friendly command patterns. - + Install Tilebox skills that teach agents how to work with Tilebox tools. + + Connect AI tools to the Tilebox MCP server when terminal access is not available. + Use an agent to edit workflow code, publish releases, deploy them to a dev cluster, and inspect jobs. diff --git a/agentic-development/tilebox-cli.mdx b/agents-and-ai-tools/tilebox-cli.mdx similarity index 78% rename from agentic-development/tilebox-cli.mdx rename to agents-and-ai-tools/tilebox-cli.mdx index af4fde0..161244d 100644 --- a/agentic-development/tilebox-cli.mdx +++ b/agents-and-ai-tools/tilebox-cli.mdx @@ -1,12 +1,12 @@ --- title: Tilebox CLI -description: Install, authenticate, and use the Tilebox CLI in AI-assisted development workflows. +description: Install, authenticate, and use the Tilebox CLI for developer workflows, operations, and AI-assisted development. icon: terminal --- -The Tilebox CLI is a command-line interface for humans, agents, and automation. It helps AI-assisted workflows by exposing Tilebox operations through explicit commands, supporting structured output, and letting agents inspect the command tree before taking action. +The Tilebox CLI is a command-line interface for developers, operators, agents, and automation. Use it to inspect Tilebox resources, manage datasets and workflows, submit jobs, start release runners, and run operational commands from the shell. -Use the CLI as the default Tilebox interface for coding agents that can run terminal commands. It keeps actions visible in the shell, works with existing development workflows, and pairs with Tilebox skills for multi-step tasks. +The CLI is also the default Tilebox interface for coding agents that can run terminal commands. It keeps actions visible in the shell, supports structured output, and pairs with Tilebox skills for multi-step tasks. ## Install the CLI @@ -69,4 +69,4 @@ The Tilebox skills include guidance for using the CLI in agent workflows. Instal npx skills add tilebox/skills ``` -After installing the skills, ask your agent to inspect `tilebox agent-context` and follow the relevant Tilebox skill before running commands that change resources. If your agent cannot use a terminal, configure the [Tilebox MCP server](/agentic-development/tilebox-mcp) instead. +After installing the skills, ask your agent to inspect `tilebox agent-context` and follow the relevant Tilebox skill before running commands that change resources. If your agent cannot use a terminal, configure the [Tilebox MCP server](/agents-and-ai-tools/tilebox-mcp) instead. diff --git a/agentic-development/tilebox-mcp.mdx b/agents-and-ai-tools/tilebox-mcp.mdx similarity index 94% rename from agentic-development/tilebox-mcp.mdx rename to agents-and-ai-tools/tilebox-mcp.mdx index 2fed56c..2edd2b0 100644 --- a/agentic-development/tilebox-mcp.mdx +++ b/agents-and-ai-tools/tilebox-mcp.mdx @@ -6,7 +6,7 @@ icon: plug The Tilebox MCP server connects AI tools to Tilebox through the Model Context Protocol. It exposes tools for working with Tilebox datasets and workflows, and it includes documentation search so agents can retrieve current Tilebox context before answering questions or taking action. -Use MCP when your agent cannot easily use terminal commands. For coding agents with shell access, the [Tilebox CLI](/agentic-development/tilebox-cli) and [Agent Skills](/agentic-development/agent-skills) are the default setup. MCP is a good fit for web-based agents, chat tools, or hosted AI clients that support MCP but do not run local commands. +Use MCP when your agent cannot easily use terminal commands. For coding agents with shell access, the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli) and [Agent Skills](/agents-and-ai-tools/agent-skills) are the default setup. MCP is a good fit for web-based agents, chat tools, or hosted AI clients that support MCP but do not run local commands. ## What the MCP server provides diff --git a/api-reference/go/workflows/WithSpan.mdx b/api-reference/go/workflows/WithSpan.mdx index a9e1008..27ebf7e 100644 --- a/api-reference/go/workflows/WithSpan.mdx +++ b/api-reference/go/workflows/WithSpan.mdx @@ -12,7 +12,7 @@ func WithSpan( ) error ``` -Wrap a function with a [tracing span](/workflows/observability/tracing) using the current runner's tracer. +Wrap a function with a [tracing span](/workflows/run-and-inspect/tracing) using the current runner's tracer. Use `WithSpan` inside a task `Execute` method. If the context is not a task execution context, the function runs without creating a span. diff --git a/api-reference/go/workflows/WithSpanResult.mdx b/api-reference/go/workflows/WithSpanResult.mdx index f840b98..5d569f2 100644 --- a/api-reference/go/workflows/WithSpanResult.mdx +++ b/api-reference/go/workflows/WithSpanResult.mdx @@ -12,7 +12,7 @@ func WithSpanResult[Result any]( ) (Result, error) ``` -Wrap a function with a [tracing span](/workflows/observability/tracing) using the current runner's tracer and return the function result. +Wrap a function with a [tracing span](/workflows/run-and-inspect/tracing) using the current runner's tracer and return the function result. Use `WithSpanResult` inside a task `Execute` method. If the context is not a task execution context, the function runs without creating a span. diff --git a/api-reference/go/workflows/WithTaskSpan.mdx b/api-reference/go/workflows/WithTaskSpan.mdx index 148b66c..8265706 100644 --- a/api-reference/go/workflows/WithTaskSpan.mdx +++ b/api-reference/go/workflows/WithTaskSpan.mdx @@ -12,7 +12,7 @@ workflows.WithTaskSpan( ) error ``` -Wrap a function with a [tracing span](/workflows/observability/tracing) using the current runner's tracer. +Wrap a function with a [tracing span](/workflows/run-and-inspect/tracing) using the current runner's tracer. `WithTaskSpan` is an alias for [`WithSpan`](/api-reference/go/workflows/WithSpan). Use it inside a task `Execute` method. If the context is not a task execution context, the function runs without creating a span. diff --git a/api-reference/go/workflows/WithTaskSpanResult.mdx b/api-reference/go/workflows/WithTaskSpanResult.mdx index 8a838fa..f3c3f28 100644 --- a/api-reference/go/workflows/WithTaskSpanResult.mdx +++ b/api-reference/go/workflows/WithTaskSpanResult.mdx @@ -12,7 +12,7 @@ workflows.WithTaskSpanResult[Result any]( ) (Result, error) ``` -Wrap a function with a [tracing span](/workflows/observability/tracing) using the current runner's tracer and return the function result. +Wrap a function with a [tracing span](/workflows/run-and-inspect/tracing) using the current runner's tracer and return the function result. `WithTaskSpanResult` is an alias for [`WithSpanResult`](/api-reference/go/workflows/WithSpanResult). Use it inside a task `Execute` method. If the context is not a task execution context, the function runs without creating a span. diff --git a/api-reference/python/tilebox.datasets/Client.mdx b/api-reference/python/tilebox.datasets/Client.mdx index 530c69b..759078c 100644 --- a/api-reference/python/tilebox.datasets/Client.mdx +++ b/api-reference/python/tilebox.datasets/Client.mdx @@ -9,6 +9,7 @@ class Client( url: str = "https://api.tilebox.com", token: str | None = None, warn_if_unauthenticated: bool = True, + transport: Literal["grpc", "http1"] = "grpc", ) ``` @@ -28,6 +29,10 @@ Create a Tilebox datasets client. Whether to log a warning when no API key is provided for the Tilebox API URL. Defaults to `True`. + + Network transport to use for API requests. Defaults to `"grpc"`. Use `"http1"` to force the Connect protocol over HTTP/1.1 on networks that do not support gRPC over HTTP/2 correctly. + + If no API key is provided, the client uses anonymous open data access for public datasets. @@ -42,5 +47,8 @@ client = Client( url="https://api.tilebox.com", token="YOUR_TILEBOX_API_KEY", ) + +# use HTTP/1.1 if your network blocks or breaks gRPC over HTTP/2 +client = Client(transport="http1") ``` diff --git a/api-reference/python/tilebox.workflows/Client.mdx b/api-reference/python/tilebox.workflows/Client.mdx index 0cf00ff..5963d71 100644 --- a/api-reference/python/tilebox.workflows/Client.mdx +++ b/api-reference/python/tilebox.workflows/Client.mdx @@ -10,6 +10,7 @@ class Client( token: str | None = None, name: str | None = None, client_id: UUID | None = None, + transport: Literal["grpc", "http1"] = "grpc", ) ``` @@ -33,6 +34,10 @@ Create a Tilebox workflows client. Optional stable ID used to scope internal loggers. If not set, a random ID is generated. + + Network transport to use for API requests. Defaults to `"grpc"`. Use `"http1"` to force the Connect protocol over HTTP/1.1 on networks that do not support gRPC over HTTP/2 correctly. + + ## Sub clients The workflows client exposes sub clients for interacting with different parts of the Tilebox workflows API. @@ -88,6 +93,9 @@ client = Client( name="sentinel-2-runner", ) +# use HTTP/1.1 if your network blocks or breaks gRPC over HTTP/2 +client = Client(transport="http1") + # access sub clients job_client = client.jobs() cluster_client = client.clusters() diff --git a/api-reference/python/tilebox.workflows/Client.runner.mdx b/api-reference/python/tilebox.workflows/Client.runner.mdx index b2c567b..45c4a1e 100644 --- a/api-reference/python/tilebox.workflows/Client.runner.mdx +++ b/api-reference/python/tilebox.workflows/Client.runner.mdx @@ -29,7 +29,7 @@ For new Python workflow projects, define a reusable [`Runner`](/api-reference/py - An optional [job cache](/workflows/caches) for caching results from tasks and sharing data between tasks. + An optional [job cache](/workflows/run-and-inspect/caches) for caching results from tasks and sharing data between tasks. diff --git a/api-reference/python/tilebox.workflows/JobCache.group.mdx b/api-reference/python/tilebox.workflows/JobCache.group.mdx index bedfae1..ceb1dc1 100644 --- a/api-reference/python/tilebox.workflows/JobCache.group.mdx +++ b/api-reference/python/tilebox.workflows/JobCache.group.mdx @@ -7,7 +7,7 @@ icon: box-archive def JobCache.group(name: str) -> JobCache ``` -You can nest caches in a hierarchical manner using [groups](/workflows/caches#groups-and-hierarchical-keys). +You can nest caches in a hierarchical manner using [groups](/workflows/run-and-inspect/caches#groups-and-hierarchical-keys). Groups are separated by a forward slash (/) in the key. This hierarchical structure functions similarly to a file system. ## Parameters diff --git a/api-reference/python/tilebox.workflows/Runner.mdx b/api-reference/python/tilebox.workflows/Runner.mdx index d712d19..4c7cd87 100644 --- a/api-reference/python/tilebox.workflows/Runner.mdx +++ b/api-reference/python/tilebox.workflows/Runner.mdx @@ -23,7 +23,7 @@ A `Runner` object is a reusable definition. Use it for direct execution by conne - Optional [job cache](/workflows/caches) used by tasks executed by this runner. + Optional [job cache](/workflows/run-and-inspect/caches) used by tasks executed by this runner. diff --git a/assets/changelog/2026-06-agentic-workflows-dark.png b/assets/changelog/2026-06-agentic-workflows-dark.png new file mode 100644 index 0000000..f7c6d70 Binary files /dev/null and b/assets/changelog/2026-06-agentic-workflows-dark.png differ diff --git a/assets/changelog/2026-06-agentic-workflows-light.png b/assets/changelog/2026-06-agentic-workflows-light.png new file mode 100644 index 0000000..bb3e3ad Binary files /dev/null and b/assets/changelog/2026-06-agentic-workflows-light.png differ diff --git a/authentication.mdx b/authentication.mdx index 9dc06f9..19d1774 100644 --- a/authentication.mdx +++ b/authentication.mdx @@ -12,17 +12,48 @@ To create an API key, log into the [Tilebox Console](https://console.tilebox.com Keep your API keys secure. Deactivate any keys if you suspect they have been compromised. -## Bearer token authentication + + Tilebox Console + Tilebox Console + -The Tilebox API uses bearer tokens for authentication. You need to pass your API key as the `token` parameter when creating an instance of the client. +## Set up environment variable + +Set `TILEBOX_API_KEY` in your shell environment (optional but recommended). That way, the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli) and SDK clients automatically pick it up. + +The following commands persist the variable for future terminal sessions and also make it available in the current session. + + +```bash Bash +echo 'export TILEBOX_API_KEY=""' >> ~/.bashrc +source ~/.bashrc +``` +```zsh Zsh +echo 'export TILEBOX_API_KEY=""' >> ~/.zshrc +source ~/.zshrc +``` +```fish Fish +set -Ux TILEBOX_API_KEY "" +``` +```powershell PowerShell +[Environment]::SetEnvironmentVariable("TILEBOX_API_KEY", "", "User") +$env:TILEBOX_API_KEY = "" +``` + + +Open a new terminal after setting the variable to confirm your shell loads it automatically. + +## Manual SDK authentication + +In case you didn't configure your shell environment, you can also pass your API key as the `token` parameter when creating an instance of the SDK clients. ```python Python from tilebox.datasets import Client as DatasetsClient from tilebox.workflows import Client as WorkflowsClient -datasets_client = DatasetsClient(token="YOUR_TILEBOX_API_KEY") -workflows_client = WorkflowsClient(token="YOUR_TILEBOX_API_KEY") +datasets_client = DatasetsClient(token="") +workflows_client = WorkflowsClient(token="") ``` ```go Go import ( @@ -30,8 +61,8 @@ import ( "github.com/tilebox/tilebox-go/workflows/v1" ) -datasetsClient := datasets.NewClient(datasets.WithAPIKey("YOUR_TILEBOX_API_KEY")) -workflowsClient := workflows.NewClient(workflows.WithAPIKey("YOUR_TILEBOX_API_KEY")) +datasetsClient := datasets.NewClient(datasets.WithAPIKey("")) +workflowsClient := workflows.NewClient(workflows.WithAPIKey("")) ``` diff --git a/changelog.mdx b/changelog.mdx index 1ea1169..b1b3ea3 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -1,10 +1,51 @@ --- title: Product Updates -description: A chronological record of new features, improvements, and other notable changes shipped across the Tilebox platform. Updated regularly with each new release. +description: New Tilebox features, product improvements, and workflow capabilities as they ship. icon: rss mode: center --- + + + Closed workflow iteration loop from deploy to run to observe to fix + Closed workflow iteration loop from deploy to run to observe to fix + + + ## Agentic Workflows + + Tilebox Workflows now supports workflow release publishing and cluster deployments. This adds a new release-based execution path to the workflow orchestrator: package a Python workflow project, publish an immutable release, deploy it to a cluster, run it on release runners, and inspect the resulting job logs and execution traces. + + The release path is designed for closed-loop workflow iteration by developers and agents. An agent can edit workflow code, publish a release, deploy it to a development cluster, submit a test job, inspect the failed task logs or spans, apply a fix, and retry or submit the next job without leaving the same command-line workflow. + + A typical iteration looks like this: + + ```bash + tilebox workflow publish-release --json + tilebox workflow deploy-release --latest --target production --json + tilebox job submit --task tilebox.com/example/ProcessScene --cluster workflow-dev --json + tilebox job wait + tilebox job logs --json + ``` + + ### What changed + + - **Release runners.** Release runners run in an environment you control, watch a cluster, load the deployed releases for that cluster, and execute compatible tasks without rebuilding the runner process for every workflow code change. + - **Workflow release publishing.** A workflow is now a long-lived object with a stable slug, and each release captures a concrete version of the workflow project, runtime entrypoint, selected files, and discovered task identifiers. + - **Project-local workflow configuration.** `tilebox.workflow.toml` binds a repository directory to a workflow slug, build inputs, runtime entrypoint, and optional deployment targets, so commands can use the local project as context. + - **Cluster deployments.** Publishing and deploying are separate steps. A release can be deployed to one or more clusters, and different clusters can run different releases of the same workflow. + + ### Start here + + + + Use an AI coding agent to edit, publish, deploy, run, inspect, and retry workflow releases. + + + Configure `tilebox.workflow.toml`, release contents, runtime entrypoints, and deployment targets. + + + + Job Execution Trace View @@ -17,7 +58,7 @@ mode: center The Console includes a built-in explorer for workflow observability, so you can inspect task logs, trace timing, failures, and runner behavior from the job view. - See the [Workflow observability documentation](/workflows/observability/introduction) for examples and integration options. + See the [Workflow observability documentation](/workflows/run-and-inspect/introduction) for examples and integration options. @@ -41,7 +82,7 @@ mode: center ## MCP Server for Datasets and Workflows - Introducing the [Tilebox MCP server](/agentic-development/tilebox-mcp), which provides tools for AI agents to access and interact with Tilebox datasets and workflows. + Introducing the [Tilebox MCP server](/agents-and-ai-tools/tilebox-mcp), which provides tools for AI agents to access and interact with Tilebox datasets and workflows. @@ -86,7 +127,7 @@ mode: center ![Progress Indicators](/assets/changelog/2025-09-progress-indicators.png) - User defined progress indicators for jobs are now available. See the [progress documentation](/workflows/progress) for more details. + User defined progress indicators for jobs are now available. See the [progress documentation](/workflows/run-and-inspect/progress) for more details. diff --git a/storage/clients.mdx b/datasets/storage/clients.mdx similarity index 98% rename from storage/clients.mdx rename to datasets/storage/clients.mdx index 2e86e61..5aec9b0 100644 --- a/storage/clients.mdx +++ b/datasets/storage/clients.mdx @@ -7,7 +7,7 @@ icon: hard-drive Tilebox does not host the actual open data satellite products but instead relies on publicly accessible storage providers for data access. Tilebox ingests available metadata as [datasets](/datasets/concepts/datasets) to enable high performance querying and structured access of the data as [xarray.Dataset](/sdks/python/xarray). -In addition to public provider clients, Tilebox also supports a local file system storage client. This is useful when your data is already available on a local disk, a mounted network share, or a synced folder such as Dropbox or Google Drive. +Tilebox also supports a local file system storage client. This is useful when your data is already available on a local disk, a mounted network share, or a synced folder such as Dropbox or Google Drive. Below is a list of the storage providers currently supported by Tilebox. diff --git a/docs.json b/docs.json index a7ab77e..259b25c 100644 --- a/docs.json +++ b/docs.json @@ -31,13 +31,13 @@ ] }, { - "group": "Agentic Development", + "group": "Agents and AI Tools", "pages": [ - "agentic-development/overview", - "agentic-development/ai-interfaces", - "agentic-development/tilebox-cli", - "agentic-development/tilebox-mcp", - "agentic-development/agent-skills" + "agents-and-ai-tools/overview", + "agents-and-ai-tools/ai-interfaces", + "agents-and-ai-tools/tilebox-cli", + "agents-and-ai-tools/agent-skills", + "agents-and-ai-tools/tilebox-mcp" ] }, { @@ -73,13 +73,8 @@ "datasets/ingest", "datasets/delete", "datasets/geometries", - "datasets/open-data" - ] - }, - { - "group": "Storage", - "pages": [ - "storage/clients" + "datasets/open-data", + "datasets/storage/clients" ] }, { @@ -93,8 +88,8 @@ "workflows/concepts/tasks", "workflows/concepts/jobs", "workflows/concepts/runners", - "workflows/concepts/workflow-releases", - "workflows/concepts/clusters" + "workflows/concepts/clusters", + "workflows/concepts/workflow-releases" ] }, { @@ -107,33 +102,33 @@ "workflows/build-and-deploy/cluster-deployments" ] }, - "workflows/caches", - "workflows/progress", { - "group": "Observability", + "group": "Run and Inspect", "icon": "eye", "pages": [ - "workflows/observability/introduction", - "workflows/observability/tracing", - "workflows/observability/logging", - "workflows/observability/query", + "workflows/run-and-inspect/introduction", + "workflows/run-and-inspect/logging", + "workflows/run-and-inspect/tracing", + "workflows/run-and-inspect/query-telemetry", + "workflows/run-and-inspect/caches", + "workflows/run-and-inspect/progress", { "group": "Integrations", "icon": "plug", "pages": [ - "workflows/observability/integrations/open-telemetry", - "workflows/observability/integrations/axiom" + "workflows/run-and-inspect/integrations/open-telemetry", + "workflows/run-and-inspect/integrations/axiom" ] } ] }, { - "group": "Near-Real Time", + "group": "Automations", "icon": "bolt", "pages": [ - "workflows/near-real-time/automations", - "workflows/near-real-time/cron", - "workflows/near-real-time/storage-events" + "workflows/automations/automations", + "workflows/automations/cron", + "workflows/automations/storage-events" ] } ] @@ -143,21 +138,43 @@ { "tab": "User Guides", "groups": [ + { + "group": "Tilebox Guides", + "pages": [ + "guides/cookbook" + ] + }, { "group": "Datasets", "pages": [ "guides/datasets/access-sentinel2-data", + "guides/datasets/query-satellite-data", "guides/datasets/create", "guides/datasets/ingest", - "guides/datasets/ingest-format" + "guides/datasets/ingest-format", + "guides/datasets/build-spatiotemporal-catalog" ] }, { "group": "Workflows", "pages": [ + "guides/workflows/run-your-first-workflow", + "guides/workflows/build-and-deploy-workflow", + "guides/workflows/deploy-to-your-compute", + "guides/workflows/debug-failed-run", + "guides/workflows/retry-with-compatible-release", "guides/workflows/agentic-workflow-iteration", "guides/workflows/multi-language" ] + }, + { + "group": "Operations", + "pages": [ + "guides/operations/monitor-workflow-runs", + "guides/operations/query-logs-and-spans", + "guides/operations/configure-cron-automations", + "guides/operations/configure-storage-event-automations" + ] } ] }, diff --git a/guides/cookbook.mdx b/guides/cookbook.mdx new file mode 100644 index 0000000..02d16ee --- /dev/null +++ b/guides/cookbook.mdx @@ -0,0 +1,266 @@ +--- +title: Tilebox Cookbook +sidebarTitle: Cookbook +description: Task-focused Tilebox guides for datasets, workflows, and operations. +icon: book-open +mode: custom +--- + +export const cookbookSections = [ + { + category: "Datasets", + description: "Find, model, ingest, and query geospatial metadata with Tilebox Datasets.", + guides: [ + { + title: "Access Sentinel-2 data", + href: "/guides/datasets/access-sentinel2-data", + description: "Query Sentinel-2 metadata by area and time range.", + icon: "satellite", + level: "Beginner", + time: "5 min", + tags: ["Open data", "Sentinel-2", "Spatial filters", "Python SDK"], + }, + { + title: "Query satellite data by time and location", + href: "/guides/datasets/query-satellite-data", + description: "Narrow open data catalogs before downloading product files.", + icon: "magnifying-glass-location", + level: "Beginner", + time: "5 min", + tags: ["Open data", "Temporal filters", "Spatial filters", "Storage clients"], + }, + { + title: "Creating a dataset", + href: "/guides/datasets/create", + description: "Create a custom dataset and define its schema in the Console.", + icon: "database", + level: "Beginner", + time: "5 min", + tags: ["Datasets", "Dataset schemas", "Console", "Timeseries datasets"], + }, + { + title: "Ingesting data", + href: "/guides/datasets/ingest", + description: "Ingest GeoParquet data into a Tilebox timeseries dataset.", + icon: "up-from-bracket", + level: "Beginner", + time: "15 min", + tags: ["Ingestion", "Timeseries datasets", "GeoParquet", "Python SDK"], + }, + { + title: "Ingesting from common file formats", + href: "/guides/datasets/ingest-format", + description: "Load tabular and geospatial files before ingesting them into Tilebox.", + icon: "file-binary", + level: "Intermediate", + time: "20 min", + tags: ["Ingestion", "DataFrames", "xarray", "File formats"], + }, + { + title: "Build a spatio-temporal catalog", + href: "/guides/datasets/build-spatiotemporal-catalog", + description: "Create a geospatial catalog that can be queried by time and geometry.", + icon: "globe", + level: "Intermediate", + time: "15 min", + tags: ["Spatio-temporal datasets", "Geometry", "Ingestion", "Catalogs"], + }, + ], + }, + { + category: "Workflows", + description: "Create workflow tasks, run jobs, publish releases, and deploy code to your own compute.", + guides: [ + { + title: "Run your first workflow", + href: "/guides/workflows/run-your-first-workflow", + description: "Define a small task, start a direct runner, submit a job, and inspect the result.", + icon: "play", + level: "Beginner", + time: "10 min", + tags: ["Tasks", "Jobs", "Direct runners", "Python SDK"], + }, + { + title: "Build and deploy a workflow project", + href: "/guides/workflows/build-and-deploy-workflow", + description: "Package a Python workflow release and deploy it to a development cluster.", + icon: "rocket", + level: "Beginner", + time: "15 min", + tags: ["Workflow releases", "Cluster deployments", "Release runners", "CLI"], + }, + { + title: "Deploy to your compute", + href: "/guides/workflows/deploy-to-your-compute", + description: "Run workflow releases on local, cloud, or on-premises compute you control.", + icon: "server", + level: "Intermediate", + time: "10 min", + tags: ["Clusters", "Release runners", "Compute", "Deployments"], + }, + { + title: "Debug a failed workflow run", + href: "/guides/workflows/debug-failed-run", + description: "Inspect task state, logs, traces, runner context, and cluster alignment.", + icon: "bug", + level: "Beginner", + time: "10 min", + tags: ["Jobs", "Logs", "Traces", "Runners"], + }, + { + title: "Retry a failed job after a compatible fix", + href: "/guides/workflows/retry-with-compatible-release", + description: "Publish a compatible fix and retry failed work without rerunning completed tasks.", + icon: "rotate", + level: "Intermediate", + time: "10 min", + tags: ["Workflow releases", "Retries", "Task identifiers", "Clusters"], + }, + { + title: "Iterate on Workflow Releases with Agents", + href: "/guides/workflows/agentic-workflow-iteration", + description: "Use a coding agent to edit, publish, deploy, run, and debug workflow releases.", + icon: "robot", + level: "Intermediate", + time: "20 min", + tags: ["Agents", "Workflow releases", "Clusters", "Observability"], + }, + { + title: "Multi-language Workflows", + href: "/guides/workflows/multi-language", + description: "Combine Python and Go tasks in one workflow when each step needs different tools.", + icon: "diagram-project", + level: "Intermediate", + time: "20 min", + tags: ["Python SDK", "Go SDK", "Tasks", "Interoperability"], + }, + ], + }, + { + category: "Operations", + description: "Monitor jobs, query telemetry, and configure automations for recurring or event-driven workflow runs.", + guides: [ + { + title: "Monitor workflow runs", + href: "/guides/operations/monitor-workflow-runs", + description: "Use the Console, CLI, and SDKs to monitor workflow execution.", + icon: "eye", + level: "Beginner", + time: "5 min", + tags: ["Console", "CLI", "Jobs", "Observability"], + }, + { + title: "Query logs and spans", + href: "/guides/operations/query-logs-and-spans", + description: "Query workflow telemetry from the CLI or SDKs for diagnostics and reports.", + icon: "magnifying-glass-chart", + level: "Beginner", + time: "8 min", + tags: ["Logs", "Spans", "Telemetry", "SDKs"], + }, + { + title: "Configure cron automations", + href: "/guides/operations/configure-cron-automations", + description: "Register an automation that submits workflow jobs on a schedule.", + icon: "clock", + level: "Beginner", + time: "8 min", + tags: ["Automations", "Cron", "Runners", "Tasks"], + }, + { + title: "Configure storage-event automations", + href: "/guides/operations/configure-storage-event-automations", + description: "Trigger workflow jobs when files are created or modified in a storage location.", + icon: "right-to-line", + level: "Intermediate", + time: "10 min", + tags: ["Automations", "Storage locations", "Storage events", "Runners"], + }, + ], + }, +]; + +export const GuideCard = ({ guide }) => ( + +
+
+ +
+
+ + {guide.level} + + + {guide.time} + +
+
+

+ {guide.title} +

+

+ {guide.description} +

+
+ {guide.tags.map((tag) => ( + + {tag} + + ))} +
+
+); + +export const GuideSection = ({ section }) => ( +
+
+
+

+ {section.category} +

+
+

+ {section.description} +

+
+
+ {section.guides.map((guide) => ( + + ))} +
+
+); + +
+
+

+ User guides +

+

+ Tilebox Cookbook +

+

+ Recipes for going from question to running system: query satellite catalogs, ingest your own data, ship workflows, and debug what happened. +

+
+ {cookbookSections.map((section) => ( + + {section.category} · {section.guides.length} guides + + ))} +
+ + {cookbookSections.map((section) => ( +
+ +
+ ))} +
+
diff --git a/guides/datasets/build-spatiotemporal-catalog.mdx b/guides/datasets/build-spatiotemporal-catalog.mdx new file mode 100644 index 0000000..a209757 --- /dev/null +++ b/guides/datasets/build-spatiotemporal-catalog.mdx @@ -0,0 +1,79 @@ +--- +title: Build a spatio-temporal catalog +description: Create a custom spatio-temporal dataset and ingest geospatial metadata that can be queried by time and location. +icon: globe +--- + +Use a spatio-temporal dataset when each datapoint has both a time and a geometry. This is useful for internal imagery catalogs, derived products, ground truth data, regions of interest, and processing outputs that need geospatial lookup. + +This guide shows the shape of the workflow. For the full ingestion guide, see [Ingesting data](/guides/datasets/ingest). + +## Create the dataset + +Create a dataset in the [Tilebox Console](https://console.tilebox.com/datasets/my-datasets) and choose the **Spatio-temporal** kind. Tilebox adds the required `time`, `id`, `ingestion_time`, and `geometry` fields automatically. + +Add fields that describe the products you want to catalog, such as: + +| Field | Type | Purpose | +| --- | --- | --- | +| `product_id` | `string` | Stable product or scene identifier. | +| `location` | `string` | Storage path, object key, or provider location. | +| `cloud_cover` | `float64` | Optional quality or filtering field. | +| `processing_level` | `string` | Product processing level or category. | + +## Prepare datapoints + +Load your source metadata into a GeoDataFrame. The geometry column should contain the footprint for each datapoint. + +```python Python +import geopandas as gpd + +products = gpd.read_parquet("products.geoparquet") +products = products.rename( + columns={ + "timestamp": "time", + "scene": "product_id", + "path": "location", + } +) +``` + +## Ingest the catalog + +Ingest the prepared records into a collection. + +```python Python +from tilebox.datasets import Client + +client = Client() +dataset = client.dataset("my_org.imagery_catalog") +collection = dataset.collection("processed-scenes") + +collection.ingest(products) +``` + +## Query by time and location + +After ingestion, use the same query model as Tilebox open data catalogs. + +```python Python +from shapely import box + +area = box(11.0, 46.0, 12.0, 47.0) + +matches = collection.query( + temporal_extent=("2026-01-01", "2026-02-01"), + spatial_extent=area, +) +``` + +## Next steps + + + + Learn the required fields and query behavior. + + + Load CSV, Parquet, GeoParquet, and NetCDF data before ingestion. + + diff --git a/guides/datasets/query-satellite-data.mdx b/guides/datasets/query-satellite-data.mdx new file mode 100644 index 0000000..f9c1eec --- /dev/null +++ b/guides/datasets/query-satellite-data.mdx @@ -0,0 +1,88 @@ +--- +title: Query satellite data by time and location +description: Find satellite metadata in a Tilebox open data catalog by combining temporal and spatial filters. +icon: satellite +--- + +Use this guide when you know the area and time range you care about and want to find matching satellite products before downloading any files. + +Tilebox Datasets stores searchable metadata for open data catalogs. Querying metadata first lets you narrow a large catalog to the scenes that match your workflow, notebook, or agent task. + +## Prerequisites + +- You have a [Tilebox API key](/authentication). +- You have installed the [Python SDK](/sdks/python/install). + +```bash +uv add tilebox shapely +``` + +## Define the search area + +Create a polygon for the area you want to inspect. This example uses a bounding box around Colorado. + +```python Python +from shapely import Polygon + +area = Polygon( + [ + (-109.05, 37.0), + (-102.05, 37.0), + (-102.05, 41.0), + (-109.05, 41.0), + (-109.05, 37.0), + ] +) +``` + +## Query Sentinel-2 metadata + +Select the Sentinel-2 MSI open data catalog and query a collection by time and location. + +```python Python +from tilebox.datasets import Client + +client = Client() +sentinel2 = client.dataset("open_data.copernicus.sentinel2_msi") +collection = sentinel2.collection("S2A_S2MSI2A") + +scenes = collection.query( + temporal_extent=("2025-10-01", "2025-11-01"), + spatial_extent=area, + show_progress=True, +) + +print(scenes[["granule_name", "processing_level", "product_type"]]) +``` + +The result is an `xarray.Dataset` containing scene metadata. Use it to inspect candidate scenes, filter by metadata fields, or pass selected datapoints to a storage client. + +## Download matching products + +Metadata queries do not download product files. Use a [storage client](/datasets/storage/clients) when you want to read or download the files referenced by a datapoint. + +```python Python +from pathlib import Path +from tilebox.storage import CopernicusStorageClient + +storage = CopernicusStorageClient( + cache_directory=Path("./data"), + s3_access_key_id="YOUR_COPERNICUS_ACCESS_KEY", + s3_secret_access_key="YOUR_COPERNICUS_SECRET_KEY", +) + +first_scene = scenes.isel(time=0) +path = storage.download(first_scene) +print(path) +``` + +## Next steps + + + + Follow a longer Sentinel-2 example with output inspection. + + + Configure provider-specific clients for product access. + + diff --git a/guides/operations/configure-cron-automations.mdx b/guides/operations/configure-cron-automations.mdx new file mode 100644 index 0000000..c8813d9 --- /dev/null +++ b/guides/operations/configure-cron-automations.mdx @@ -0,0 +1,60 @@ +--- +title: Configure cron automations +description: Register a workflow automation that submits jobs on a cron schedule. +icon: clock +--- + +Use cron automations when a workflow should run on a schedule, such as hourly catalog checks, daily quality-control jobs, or recurring data processing. + +Tilebox automations submit workflow jobs when their trigger condition matches. A runner on the target cluster still needs to execute the submitted tasks. + +## Define the automation task + +Create a task that can run from a scheduled trigger. + +```python Python +from tilebox.workflows import ExecutionContext +from tilebox.workflows.automations import CronTask + + +class DailyCatalogCheck(CronTask): + def execute(self, context: ExecutionContext) -> None: + context.logger.info("Running daily catalog check", trigger=str(self.trigger)) +``` + +## Register the cron automation + +Use the automation client to register a schedule and target cluster. + +```python Python +from tilebox.workflows import Client + +client = Client() +automations = client.automations() + +automations.create_cron_automation( + "daily-catalog-check", + DailyCatalogCheck(), + cron_schedules=["0 6 * * *"], +) +``` + +## Keep a runner available + +The automation submits jobs. It does not execute them by itself. Start a runner that can execute the task identifier. + +```python Python +runner = client.runner(tasks=[DailyCatalogCheck]) +runner.run_forever() +``` + +## Next steps + + + + Read the full cron automation reference. + + + Inspect jobs created by automations. + + diff --git a/guides/operations/configure-storage-event-automations.mdx b/guides/operations/configure-storage-event-automations.mdx new file mode 100644 index 0000000..0994330 --- /dev/null +++ b/guides/operations/configure-storage-event-automations.mdx @@ -0,0 +1,77 @@ +--- +title: Configure storage-event automations +description: Register a workflow automation that submits jobs when files are created or modified in a storage location. +icon: right-to-line +--- + +Use storage-event automations when new or changed files should trigger workflow work, such as processing new uploads in a bucket or reacting to generated products. + +Tilebox submits a job when a matching storage event occurs. A runner on the target cluster still needs to execute the submitted task. + +## Define the storage-event task + +Create a task that handles the event payload. + +```python Python +from tilebox.workflows import ExecutionContext +from tilebox.workflows.automations import StorageEventTask, StorageEventType + + +class ProcessUploadedObject(StorageEventTask): + head_bytes: int = 64 + + def execute(self, context: ExecutionContext) -> None: + if self.trigger.type == StorageEventType.CREATED: + path = self.trigger.location + context.logger.info( + "Processing uploaded object", + path=path, + ) + data = self.trigger.storage.read(path) + context.logger.info("Read object data", size_bytes=len(data)) +``` + +## Register the storage location + +Register or select the storage location that Tilebox should watch. Storage locations can represent cloud buckets or local file-system locations supported by Tilebox automations. + +```python Python +from tilebox.workflows import Client + +client = Client() +automations = client.automations() +locations = automations.storage_locations() +print(locations) +``` + +## Register the automation + +Create the automation using the task prototype and storage trigger. + +```python Python +automations.create_storage_event_automation( + "process-uploaded-objects", + ProcessUploadedObject(), + triggers=[(locations[0], "incoming/**")], +) +``` + +## Test and inspect + +Create or change a matching object, then inspect the submitted job in the Console or through the Tilebox command-line tool. + +```bash +tilebox job logs --json +tilebox job spans --json +``` + +## Next steps + + + + Read the full storage-event automation reference. + + + Inspect task state, logs, traces, and runner context. + + diff --git a/guides/operations/monitor-workflow-runs.mdx b/guides/operations/monitor-workflow-runs.mdx new file mode 100644 index 0000000..d405f84 --- /dev/null +++ b/guides/operations/monitor-workflow-runs.mdx @@ -0,0 +1,48 @@ +--- +title: Monitor workflow runs +description: Use the Console, CLI, and SDKs to monitor workflow jobs, task state, logs, traces, and runner behavior. +icon: eye +--- + +Use this guide when you need to follow workflow execution after a job is submitted. Tilebox tracks job state, task state, logs, traces, and runner context for each workflow run. + +## Use the Console for live inspection + +Open the [Jobs view](https://console.tilebox.com/workflows/jobs) in the Tilebox Console. Use it to inspect the task graph, task states, logs, and trace timing for a job. + +The Console is the best first stop when a human is investigating a job interactively. + +## Use the command-line tool for automation + +Use the Tilebox command-line tool when you need structured output for scripts, CI checks, or coding agents. + +```bash +tilebox job logs --json +tilebox job spans --json +``` + +Ask agents to keep job IDs and command output in their working notes so they can continue from observed failures instead of guessing. + +## Use SDK queries in notebooks or services + +Use SDK queries when monitoring is part of a notebook, dashboard, or service. + +```python Python +from tilebox.workflows import Client + +client = Client() +job = client.jobs().find("019e07b1-916b-0630-f3ba-f1c33235d174") +logs = client.jobs().query_logs(job) +spans = client.jobs().query_spans(job.id) +``` + +## Next steps + + + + Learn how Tilebox connects logs, traces, task status, and runner context. + + + Work through a failed or queued job investigation. + + diff --git a/guides/operations/query-logs-and-spans.mdx b/guides/operations/query-logs-and-spans.mdx new file mode 100644 index 0000000..4d24885 --- /dev/null +++ b/guides/operations/query-logs-and-spans.mdx @@ -0,0 +1,56 @@ +--- +title: Query logs and spans +description: Query workflow logs and spans from the CLI or SDKs for debugging, reports, and automated diagnostics. +icon: magnifying-glass-chart +--- + +Tilebox stores workflow logs and spans for each job. Query them when you need structured diagnostics outside the Console, such as in notebooks, scripts, or agent loops. + +## Query from the command line + +Use JSON output when another tool or agent will parse the result. + +```bash +tilebox job logs --json +tilebox job spans --json +``` + +Use `jq` to narrow the output during debugging. + +```bash +tilebox job logs --json | jq '.[] | select(.level == "ERROR")' +``` + +## Query from Python + +Use the jobs client in notebooks and scripts. + +```python Python +from tilebox.workflows import Client + +client = Client() +job = client.jobs().find("019e07b1-916b-0630-f3ba-f1c33235d174") + +logs = client.jobs().query_logs(job) +spans = client.jobs().query_spans(job.id) + +logs_df = logs.to_pandas() +spans_df = spans.to_pandas() +``` + +## Use the detailed reference pages + + + + Query logs and spans from Python and Go. + + + Emit structured logs from workflow tasks. + + + Add custom spans around important parts of task code. + + + Export workflow telemetry to Axiom. + + diff --git a/guides/workflows/build-and-deploy-workflow.mdx b/guides/workflows/build-and-deploy-workflow.mdx new file mode 100644 index 0000000..8a4c76a --- /dev/null +++ b/guides/workflows/build-and-deploy-workflow.mdx @@ -0,0 +1,121 @@ +--- +title: Build and deploy a workflow project +description: Structure a Python workflow project, publish a workflow release, and deploy it to a development cluster. +icon: rocket +--- + +Use workflow releases when the code under test should match the code that runners execute. A release packages a Python workflow project, records the discovered task identifiers, and lets release runners load the deployed code from a cluster. + +This guide summarizes the build and deploy path. For the underlying model, see [Workflow releases](/workflows/concepts/workflow-releases) and [Release lifecycle](/workflows/build-and-deploy/releases). + +## Prerequisites + +- You have installed the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli). +- `TILEBOX_API_KEY` is set in the shell where you run commands. +- You have a Python workflow project with a `Runner` object. + +## Create the project structure + +Use a small, importable project layout. + + + + + + + + + + + + + +## Configure the workflow + +Point `tilebox.workflow.toml` at the exported runner object and include the files the release runner needs. + +```toml tilebox.workflow.toml +[workflow] +slug = "my-workflow" +root = "." +runner = "my_workflow.runner:runner" + +[build] +include = [ + "pyproject.toml", + "uv.lock", + "my_workflow/**", +] +exclude = [ + ".venv/**", + "**/__pycache__/**", + "**/*.pyc", +] +use_gitignore = true + +[targets.dev] +clusters = ["workflow-dev"] +``` + +## Build and publish the release + +Build locally first when you want detailed validation output. + +```bash +tilebox workflow build-release --debug --json +``` + +Publish the checked release content. + +```bash +RELEASE_ID=$(tilebox workflow publish-release --json | jq -r '.id') +``` + +Publishing creates an immutable release. It does not change what any cluster runs until you deploy it. + +## Deploy to a development cluster + +Deploy the release to the development target. + +```bash +tilebox workflow deploy-release --release "$RELEASE_ID" --target dev --json +``` + +Start a release runner in an environment you control. + +```bash +tilebox runner start --cluster workflow-dev --debug +``` + +## Run and inspect a job + +Submit a task that matches one of the identifiers discovered from the release. + +```bash +JOB_ID=$(tilebox job submit \ + --name my-workflow-test \ + --task tilebox.com/example/ProcessScene \ + --version v1.0 \ + --cluster workflow-dev \ + --input '{"scene_id":"S2A_001"}' \ + --wait \ + --json | jq -r '.id') +``` + +Inspect the result before making the next change. + +```bash +tilebox job logs "$JOB_ID" --json +tilebox job spans "$JOB_ID" --json +``` + +## Next steps + + + + Learn the details of Python workflow project layout. + + + Choose where release runners run and how clusters map to environments. + + diff --git a/guides/workflows/debug-failed-run.mdx b/guides/workflows/debug-failed-run.mdx new file mode 100644 index 0000000..15262fa --- /dev/null +++ b/guides/workflows/debug-failed-run.mdx @@ -0,0 +1,63 @@ +--- +title: Debug a failed workflow run +description: Inspect task state, logs, traces, runner context, and cluster alignment when a workflow job fails or stays queued. +icon: bug +--- + +Use this guide when a workflow job fails, runs slower than expected, or stays queued. Tilebox records job state, task state, logs, traces, and runner context so you can identify whether the problem is in task code, task routing, dependencies, or the runner environment. + +## Find the job + +Open the job in the [Tilebox Console](https://console.tilebox.com/workflows/jobs), or use the Tilebox command-line tool if you already have the job ID. + +```bash +tilebox job logs --json +tilebox job spans --json +``` + +## Check task state first + +Start with the task graph. A failed task often points to task code or runtime dependencies. A queued task often points to cluster, runner, or task registration mismatch. + +Common checks: + +- The job was submitted to the intended cluster. +- A runner is connected to the same cluster. +- The runner advertises the submitted task identifier and compatible version. +- Any task dependencies are complete. +- Retry limits have not been exhausted. + +## Inspect logs + +Logs show messages emitted by task code and runner context attached by Tilebox. + +```bash +tilebox job logs --json | jq '.[] | {time, level, body, attributes}' +``` + +Use structured log fields in your tasks so the relevant scene ID, product ID, path, or model name appears in the log record. + +## Inspect traces + +Traces show task timing, parent-child relationships, custom spans, and failures. + +```bash +tilebox job spans --json | jq '.[] | {name, status, duration}' +``` + +Use traces to find slow subtasks, repeated retries, and failures inside a specific custom span. + +## Fix and rerun + +For direct runners, fix the code and restart the runner process. For release runners, publish a fixed release and deploy it. + +If the fix is compatible with the failed task input schema and task major version, retry the job. If the change is breaking, submit a new job with a new task version. + + + + Publish a compatible fix, deploy it to the same cluster, and retry failed work. + + + Learn how logs, traces, task status, and runner context fit together. + + diff --git a/guides/workflows/deploy-to-your-compute.mdx b/guides/workflows/deploy-to-your-compute.mdx new file mode 100644 index 0000000..d501a03 --- /dev/null +++ b/guides/workflows/deploy-to-your-compute.mdx @@ -0,0 +1,72 @@ +--- +title: Deploy to your compute +description: Run Tilebox workflow releases on compute environments you control by using clusters and release runners. +icon: server +--- + +Tilebox manages workflow state, releases, deployments, jobs, logs, and traces. Your compute environment runs the runner process that executes the work. This lets workflows run on local machines, cloud virtual machines, Kubernetes clusters, on-premises systems, or controlled customer infrastructure. + +## Choose a cluster for the environment + +A cluster is the routing boundary for jobs and runners. Jobs submitted to a cluster can only be claimed by runners connected to the same cluster. + +Use separate clusters for environments that should not run the same code by accident, such as development and production-like compute. + +```bash +tilebox cluster create "workflow-dev" --json +tilebox cluster create "workflow-prod" --json +``` + +## Deploy a release to the cluster + +Deploy a published workflow release to the cluster or to a target defined in `tilebox.workflow.toml`. + +```bash +tilebox workflow deploy-release --release "$RELEASE_ID" --cluster workflow-dev --json +``` + +For repeated deployments, define targets in the workflow configuration. + +```toml +[targets.dev] +clusters = ["workflow-dev"] + +[targets.production] +clusters = ["workflow-prod"] +``` + +Then deploy by target name. + +```bash +tilebox workflow deploy-release --release "$RELEASE_ID" --target dev --json +``` + +## Start release runners where the work should run + +Start one or more release runners in the environment that has the required network access, credentials, hardware, and data access. + +```bash +tilebox runner start --cluster workflow-dev --debug +``` + +The runner watches its cluster, downloads missing release artifacts, starts the workflow runtime, and advertises the tasks it can execute. Updating a deployment changes what the runner can execute without rebuilding the runner process. + +## Scale runner processes + +Start multiple runners for the same cluster when you want more parallelism. + +```bash +tilebox parallel -n 4 -- tilebox runner start --cluster workflow-dev +``` + +You can also run runners through your own process manager, VM image, container platform, or scheduler. Tilebox does not require the compute process to run in Tilebox-managed infrastructure. + +## Verify cluster alignment + +If a job stays queued, check that these three values match: + +1. The job was submitted to the expected cluster. +2. The workflow release is deployed to that cluster. +3. A release runner is running for that cluster. + +For details, see [Cluster deployments](/workflows/build-and-deploy/cluster-deployments) and [Runners](/workflows/concepts/runners). diff --git a/guides/workflows/retry-with-compatible-release.mdx b/guides/workflows/retry-with-compatible-release.mdx new file mode 100644 index 0000000..ff40c3a --- /dev/null +++ b/guides/workflows/retry-with-compatible-release.mdx @@ -0,0 +1,69 @@ +--- +title: Retry a failed job after a compatible fix +description: Publish a fixed workflow release, deploy it to the same cluster, and retry failed work without resubmitting the whole job. +icon: rotate +--- + +When a workflow job fails because of a bug in task code, you can often fix the code, publish a compatible release, deploy it to the same cluster, and retry the failed job. Tilebox resumes from failed tasks instead of rerunning completed work. + +## Confirm the fix is compatible + +A retry can use the fixed release when the failed task and the new task registration are compatible. + +Keep these stable: + +- task identifier name +- task major version +- task input schema + +Use a new major task version and submit a new job when the input schema or behavior is no longer compatible with the failed task. + +## Publish the fixed release + +After editing the workflow code, publish a new release. + +```bash +RELEASE_ID=$(tilebox workflow publish-release --json | jq -r '.id') +``` + +For validation details, build locally first. + +```bash +tilebox workflow build-release --debug --json +``` + +## Deploy to the same cluster + +Deploy the fixed release to the same cluster that received the original job. + +```bash +tilebox workflow deploy-release --release "$RELEASE_ID" --cluster workflow-dev --json +``` + +The job cluster, release deployment cluster, and release runner cluster must match. + +## Retry the job + +Retry the failed job. + +```bash +tilebox job retry --json +``` + +Then inspect logs and spans to confirm the fixed task completed. + +```bash +tilebox job logs --json +tilebox job spans --json +``` + +## Next steps + + + + Understand release compatibility and task registrations. + + + Inspect task state, logs, traces, and runner context. + + diff --git a/guides/workflows/run-your-first-workflow.mdx b/guides/workflows/run-your-first-workflow.mdx new file mode 100644 index 0000000..82a3ea7 --- /dev/null +++ b/guides/workflows/run-your-first-workflow.mdx @@ -0,0 +1,89 @@ +--- +title: Run your first workflow +description: Define a small workflow task, start a direct runner, submit a job, and inspect the result. +icon: play +--- + +This guide runs a minimal workflow with a direct runner. It gives you the fastest path to the Tilebox Workflows execution model before publishing workflow releases or deploying to a cluster. + +Use this path when you are developing locally as a developer. If you want an agent to do the setup and iteration for you, start with [Onboard your agent](/onboard-your-agent) and then follow [Iterate on workflow releases with agents](/guides/workflows/agentic-workflow-iteration). + +## Prerequisites + +- You have a [Tilebox API key](/authentication). +- You have installed the [Python SDK](/sdks/python/install). + +```bash +uv add tilebox +export TILEBOX_API_KEY="YOUR_TILEBOX_API_KEY" +``` + +## Create a workflow file + +Create a file named `hello_workflow.py`. + +```python hello_workflow.py +from tilebox.workflows import Client, ExecutionContext, Runner, Task + + +class HelloWorkflow(Task): + name: str + + def execute(self, context: ExecutionContext) -> None: + context.logger.info("Running root task", name=self.name) + context.submit_subtask(WriteGreeting(name=self.name)) + + +class WriteGreeting(Task): + name: str + + def execute(self, context: ExecutionContext) -> None: + context.logger.info("Hello from Tilebox", name=self.name) + + +if __name__ == "__main__": + client = Client() + runner = Runner(tasks=[HelloWorkflow, WriteGreeting]) + + job = client.jobs().submit( + "hello-workflow", + HelloWorkflow(name="Earth"), + ) + print(f"Submitted job: {job.id}") + + runner.connect_to(client).run_all() +``` + +The root task submits one subtask. Tilebox tracks both tasks as part of the same job, and the direct runner executes all eligible work. + +## Run the workflow + +Run the file locally. + +```bash +uv run python hello_workflow.py +``` + +The script submits a job, starts a direct runner, executes the root task and subtask, and exits after the queued work is complete. + +## Inspect the job + +Open the job in the [Tilebox Console](https://console.tilebox.com/workflows/jobs) to inspect task state, logs, and the execution trace. + +For scripted inspection, use the jobs client or the Tilebox command-line tool: + +```bash +tilebox job logs --json +tilebox job spans --json +``` + +## Next steps + + + + Define task inputs, subtasks, dependencies, retries, and identifiers. + + + Package the same kind of workflow code as a release and deploy it to a cluster. + + diff --git a/index.mdx b/index.mdx index db6bc68..1e8df1a 100644 --- a/index.mdx +++ b/index.mdx @@ -9,6 +9,12 @@ mode: custom export const HomeSearch = () => { const openSearch = () => { + const searchButton = document.querySelector("#search-bar-entry, #search-bar-entry-mobile"); + if (searchButton instanceof HTMLButtonElement) { + searchButton.click(); + return; + } + const isMac = typeof navigator !== "undefined" && /Mac|iPhone|iPad|iPod/.test(navigator.platform); const event = new KeyboardEvent("keydown", { key: "k", @@ -197,7 +203,7 @@ export const HomeSearch = () => { Create and deploy workflows - + Connect with MCP @@ -255,7 +261,7 @@ export const HomeSearch = () => { Tilebox dataset explorer Tilebox dataset explorer - + Workflow logs in Tilebox Console Workflow logs in Tilebox Console @@ -269,7 +275,7 @@ export const HomeSearch = () => {
- + diff --git a/onboard-your-agent.mdx b/onboard-your-agent.mdx index 4dd4c8f..ec2107e 100644 --- a/onboard-your-agent.mdx +++ b/onboard-your-agent.mdx @@ -4,7 +4,22 @@ description: Configure an AI agent with the Tilebox CLI and Tilebox skills. icon: rocket --- -This tutorial gives an AI agent the default Tilebox setup for repository-based work: the Tilebox CLI and Tilebox skills. After completing it, your agent can inspect available CLI commands, run commands with structured output, and use Tilebox-specific skills for datasets, jobs, and automations. +
+This page is intended for users to guide them with configuring their agent with Tilebox. +Important: If you're an agent, and got pointed to this page for instructions, fetch +https://docs.tilebox.com/agent-onboarding.md instead. Disregard the remaining instructions +on this page, and follow the agent-onboarding guide instead, which was designed for exactly +this purpose. +
+ +Onboard your AI agent to Tilebox, by installing the Tilebox CLI, installing Tilebox Agent skills and configuring your API Key. After completing it, your agent will be able to manage your Tilebox resources, and use Tilebox-specific skills to create and query datasets, submit and monitor jobs. +It will also be able to write geospatial workflows, deploy it to clusters, submit jobs and iterate on the workflow code after inspecting job logs and checking outputs. +If you want to use Tilebox directly as a developer, start with the [developer quickstart](/quickstart). + + +**Using a LLM Chat interface instead of an agent?** +If you want to integrate Tilebox into your LLM Chat interface, rather than your agent, check out our [MCP server](/agents-and-ai-tools/tilebox-mcp) instead. + @@ -34,21 +49,6 @@ This tutorial gives an AI agent the default Tilebox setup for repository-based w The skills cover CLI usage, dataset management, job monitoring, workflow authoring, and automations. - - Configure the Tilebox MCP server when your agent cannot easily use terminal commands, for example in a web-based chat tool. The server uses OAuth, so you do not need to add API keys or headers to the configuration. - - ```json - { - "mcpServers": { - "tilebox": { - "url": "https://mcp.tilebox.com/mcp" - } - } - } - ``` - - When the client connects, sign in with your Tilebox account and select the team the agent should use. - Ask your agent to inspect its Tilebox capabilities before assigning a larger task. @@ -60,4 +60,4 @@ This tutorial gives an AI agent the default Tilebox setup for repository-based w ## Next steps -Continue with [Tilebox CLI](/agentic-development/tilebox-cli) for agent-friendly command patterns, or use [AI interfaces](/agentic-development/ai-interfaces) to decide when MCP is a better fit than terminal commands. +Learn how to [Iterate on Workflow Releases with Agents](/guides/workflows/agentic-workflow-iteration), or check out the [CLI Reference](/agents-and-ai-tools/tilebox-cli) to get an overview of useful `tilebox` terminal commands. diff --git a/quickstart.mdx b/quickstart.mdx index 34fc18c..13bfeef 100644 --- a/quickstart.mdx +++ b/quickstart.mdx @@ -1,12 +1,12 @@ --- title: Quickstart -description: Start here to set up Tilebox and get started building. +description: Start here as a developer to create an API key, query Earth data, and run a small workflow. icon: play --- -Tilebox works across the [Console](/console), [command line](/agentic-development/tilebox-cli), [SDKs](/sdks), and [coding agents](/onboard-your-agent). The first step is the same for each path: create an API key. +This quickstart is for developers using Tilebox directly from a terminal, notebook, or SDK. If you want an AI coding agent to work with Tilebox for you, start with [Onboard your agent](/onboard-your-agent). -Follow the steps below to get started. +You will create an API key, query open Sentinel-2 metadata, and run a small workflow task using the Tilebox [Python SDK](/sdks/python). ## Start in a Notebook @@ -20,53 +20,58 @@ If you prefer to work locally, follow these steps to get started. Create an API key by logging into the [Tilebox Console](https://console.tilebox.com), navigating to [Settings -> API Keys](https://console.tilebox.com/settings/api-keys), and clicking the "Create API Key" button. - - Tilebox Console - Tilebox Console - - Then, add it to your environment ```bash export TILEBOX_API_KEY= ``` - - Tilebox can be used from the browser, terminal, via SDKs or coding agents. Choose the access point that matches how you want to start. + + Tilebox can be used from the browser, terminal, or via our SDKs running locally or in interactive notebook environments. This quickstart guides you through setting up the Tilebox **Python SDK locally**. + Alternatively, you can also check out any of the following ways of using Tilebox. - - Use Tilebox from the browser. Create API keys, inspect datasets and monitor jobs without installing anything. - + - - Work locally from your terminal. Use this path for setup, local development, and operational commands. - + - - Use Tilebox from your application code in Python or Go. - + - - Setup your coding agent, and give it access to Tilebox-specific skills and resources for catalogs, workflows, jobs and automations. - + + + + + To install the python SDK locally, run the following command. + + ```bash + uv add tilebox + ``` Use the datasets client to query data from a dataset. ```python Python from tilebox.datasets import Client + from shapely import Polygon + + # define a search area (optional) + new_york_city = Polygon( + [(-74.40, 41.02), (-74.48, 40.27), (-73.37, 40.34), + (-73.39, 41.05), (-74.40, 41.02)] + ) client = Client(token="YOUR_TILEBOX_API_KEY") # select a dataset - datasets = client.datasets() - dataset = datasets.open_data.copernicus.sentinel2_msi - - # and load data from a collection in a given time range - collection = dataset.collection("S2A_S2MSI1C") - data_january_2022 = collection.query(temporal_extent=("2022-01-01", "2022-02-01")) + dataset = client.dataset("open_data.copernicus.sentinel2_msi") + # and query data + scenes_new_york_january_2026 = dataset.query( + collections=["S2A_S2MSI2A", "S2B_S2MSI2A", "S2C_S2MSI2A"], + temporal_extent=("2026-01-01", "2026-02-01"), + spatial_extent=new_york_city + ) + print(scenes_new_york_january_2026.granule_name) ``` @@ -74,6 +79,9 @@ If you prefer to work locally, follow these steps to get started. ```python Python from tilebox.workflows import Client, Runner, Task + from tilebox.workflows.observability.logging import configure_console_logging + + configure_console_logging() # Replace with your actual token client = Client(token="YOUR_TILEBOX_API_KEY") @@ -83,35 +91,42 @@ If you prefer to work locally, follow these steps to get started. name: str = "World" def execute(self, context): - print(f"{self.greeting} {self.name}, from the main task!") + context.logger.info(f"{self.greeting} {self.name}, from the main task!") context.submit_subtask(HelloSubtask(name=self.name)) class HelloSubtask(Task): name: str def execute(self, context): - print(f"Hello from the subtask, {self.name}!") + context.logger.info(f"Hello from the subtask!", name=self.name) # Initiate the job jobs = client.jobs() - jobs.submit("parameterized-hello-world", HelloWorldTask(greeting="Greetings", name="Universe")) + job = jobs.submit("parameterized-hello-world", HelloWorldTask(greeting="Greetings", name="Universe")) # Run the tasks runner = Runner(tasks=[HelloWorldTask, HelloSubtask]) runner.connect_to(client).run_all() + + print("Explore the job you just submitted in the Tilebox Console.") + print("Check out tasks, log messages, and execution timining:") + print(f"https://console.tilebox.com/workflows/jobs/{job.id}") ``` Review the following guides to learn more about the modules that make up Tilebox: - + Learn how to create a Timeseries dataset using the Tilebox Console. Learn how to ingest an existing CSV dataset into a Timeseries dataset collection. - + + Inspect task state, logs and traces when a workflow job fails. + + Package a Python workflow project, publish a release, and deploy it to a cluster. diff --git a/sdks/introduction.mdx b/sdks/introduction.mdx index c06e77c..00b0703 100644 --- a/sdks/introduction.mdx +++ b/sdks/introduction.mdx @@ -7,7 +7,9 @@ icon: book-open import { HeroCard } from '/snippets/components.mdx'; -The following language SDKs are currently available for Tilebox. Select one to learn more. +Use Tilebox SDKs when you want Tilebox in notebooks, scripts, services, or workflow task code. The SDKs expose datasets, storage access, workflow jobs, runners, observability queries, and automation APIs from regular app code. + +If you are setting up a coding agent, start with [Onboard your agent](/onboard-your-agent). Agents use the [Tilebox CLI](/agents-and-ai-tools/tilebox-cli) first, then inspect SDK examples or API reference pages when they need to edit code. + +## Choose an SDK + + + + Use Python for notebooks, xarray-based dataset queries, data ingestion, and Python workflow task code. + + + Use Go for backend services, typed dataset access, workflow task submission, and long-running runners. + + + Start from working Python notebooks for common Tilebox dataset workflows. + + + Look up method signatures and parameter details for Python and Go clients. + + diff --git a/sdks/python/sample-notebooks.mdx b/sdks/python/sample-notebooks.mdx index e861f4e..9144722 100644 --- a/sdks/python/sample-notebooks.mdx +++ b/sdks/python/sample-notebooks.mdx @@ -1,7 +1,7 @@ --- title: Sample notebooks description: Get started quickly with ready-to-run Jupyter notebooks that demonstrate common satellite data access and analysis workflows using the Tilebox Python SDK. -icon: notebook +icon: book-open --- To quickly become familiar with the Python client, you can explore some sample notebooks. Each notebook can be executed standalone from top to bottom. diff --git a/workflows/near-real-time/automations.mdx b/workflows/automations/automations.mdx similarity index 69% rename from workflows/near-real-time/automations.mdx rename to workflows/automations/automations.mdx index c6bd255..9d44618 100644 --- a/workflows/near-real-time/automations.mdx +++ b/workflows/automations/automations.mdx @@ -1,13 +1,10 @@ --- title: Automations +sidebarTitle: Trigger jobs automatically description: Trigger workflow jobs automatically in response to external events such as schedules or file changes, enabling near-real-time data processing pipelines. icon: repeat --- - - This feature is only available in the Python SDK. - - ## Introduction Tilebox Workflows can execute jobs in two ways: a one-time execution triggered by a user, typically a batch processing, and near-real-time execution based on specific external events. @@ -16,10 +13,10 @@ By defining trigger conditions, you can automatically submit jobs based on exter Tilebox Workflows currently supports the following trigger conditions: - + Trigger jobs based on a Cron schedule. - + Trigger jobs after objects are created or modified in a storage location such as a cloud bucket. @@ -35,18 +32,15 @@ To create a trigger, define a special task that serves as a prototype. In respon Each automation has a [task identifier](/workflows/concepts/tasks#task-identifiers), a [version](/workflows/concepts/tasks#semantic-versioning), and [input parameters](/workflows/concepts/tasks#input-parameters), just like regular tasks. Automations also automatically provide a special `trigger` attribute that contains information about the event that initiated the task's execution. - - Go doesn't support registering automations yet, please use python or the console instead. - - ## Automation Client -The Tilebox Workflows client includes a sub-client for managing automations. You can create this sub-client by calling the `automations` method on the main client instance. +The Tilebox Workflows clients include sub-clients for inspecting automations and storage locations. Python also includes helpers for registering automation prototypes from task classes. ### Listing Registered Automations -To list all registered automations, use the `all` method on the automation client. +To list all registered automations, use the automation client. + ```python Python from tilebox.workflows import Client @@ -56,6 +50,27 @@ automations = client.automations() automations = automations.all() print(automations) ``` +```go Go +ctx := context.Background() +client := workflows.NewClient() + +automations, err := client.Automations.List(ctx) +if err != nil { + slog.ErrorContext(ctx, "failed to list automations", slog.Any("error", err)) + return +} + +for _, automation := range automations { + slog.InfoContext(ctx, + "automation", + slog.String("id", automation.ID.String()), + slog.String("name", automation.Name), + slog.Int("cron_triggers", len(automation.CronTriggers)), + slog.Int("storage_event_triggers", len(automation.StorageEventTriggers)), + ) +} +``` + ```plaintext Output [ @@ -76,11 +91,11 @@ print(automations) ### Registering Automations -To register an automation, use the `create_*_automation` methods specific to each trigger type provided by the automation client. Refer to the documentation for each trigger type for more details. +To register an automation from code, use the Python automation client's `create_*_automation` methods for each trigger type. You can also create and manage automations in the Tilebox Console. Refer to the documentation for each trigger type for more details. - - + + ## Overview in the Tilebox Console diff --git a/workflows/near-real-time/cron.mdx b/workflows/automations/cron.mdx similarity index 64% rename from workflows/near-real-time/cron.mdx rename to workflows/automations/cron.mdx index a827c67..3d11b87 100644 --- a/workflows/near-real-time/cron.mdx +++ b/workflows/automations/cron.mdx @@ -4,15 +4,12 @@ description: Schedule recurring workflow jobs using cron expressions so that tas icon: clock --- - - This feature is only available in the Python SDK. - - ## Creating Cron tasks Cron tasks run repeatedly on a specified [cron](https://en.wikipedia.org/wiki/Cron) schedule. -To create a Cron task, use `tilebox.workflows.automations.CronTask` as your tasks base class instead of the regular `tilebox.workflows.Task`. +In Python, use `tilebox.workflows.automations.CronTask` as your task base class instead of the regular `tilebox.workflows.Task`. In Go, register an executable task with the same task identifier as the automation prototype. + ```python Python from tilebox.workflows import ExecutionContext from tilebox.workflows.automations import CronTask @@ -30,10 +27,21 @@ class MyCronTask(CronTask): trigger_time=self.trigger.time, ) ``` +```go Go +type MyCronTask struct { + Message string +} + +func (t *MyCronTask) Execute(ctx context.Context) error { + slog.InfoContext(ctx, "Cron task triggered", slog.String("message", t.Message)) + return nil +} +``` + ## Registering a Cron trigger -After implementing a Cron task, register it to be triggered according to a Cron schedule. +After implementing a Cron task, register it to be triggered according to a Cron schedule. The Python SDK provides a registration helper, and you can also register cron automations from the Tilebox Console. When the Cron expression matches, a new job is submitted consisting of a single task instance derived from the Cron task prototype. ```python Python @@ -62,13 +70,34 @@ cron_automation = automations.create_cron_automation( With the Cron automation registered, a job is submitted whenever the Cron expression matches. But unless a [runner](/workflows/concepts/runners) is available to execute the Cron task the submitted jobs remain in a task queue. Once an [eligible runner](/workflows/concepts/runners#task-selection) becomes available, all jobs in the queue are executed. + ```python Python -from tilebox.workflows import Client +from tilebox.workflows import Client, Runner client = Client() -runner = client.runner(tasks=[MyCronTask]) -runner.run_all() +runner = Runner(tasks=[MyCronTask]) +runner.connect_to(client).run_forever() +``` +```go Go +ctx := context.Background() +client := workflows.NewClient() + +runner, err := client.NewTaskRunner(ctx) +if err != nil { + slog.ErrorContext(ctx, "failed to create task runner", slog.Any("error", err)) + return +} + +if err := runner.RegisterTasks(&MyCronTask{}); err != nil { + slog.ErrorContext(ctx, "failed to register tasks", slog.Any("error", err)) + return +} + +if err := runner.RunForever(ctx); err != nil { + slog.ErrorContext(ctx, "task runner stopped", slog.Any("error", err)) +} ``` + If this runner runs continuously, its logs may resemble the following: @@ -91,9 +120,44 @@ The [Tilebox Console](https://console.tilebox.com/workflows/automations) provide Use the console to view, edit, and delete the registered Cron automations. +You can also inspect registered cron triggers from the SDKs. + + +```python Python +from tilebox.workflows import Client + +client = Client() +automations = client.automations().all() + +for automation in automations: + for trigger in automation.cron_triggers: + print(automation.name, trigger.schedule) +``` +```go Go +ctx := context.Background() +client := workflows.NewClient() + +automations, err := client.Automations.List(ctx) +if err != nil { + slog.ErrorContext(ctx, "failed to list automations", slog.Any("error", err)) + return +} + +for _, automation := range automations { + for _, trigger := range automation.CronTriggers { + slog.InfoContext(ctx, + "cron trigger", + slog.String("automation", automation.Name), + slog.String("schedule", trigger.Schedule), + ) + } +} +``` + + ## Deleting Cron automations -To delete a registered Cron automation, use `automations.delete`. After deletion, no new jobs will be submitted by that Cron trigger. Past jobs already triggered will still remain queued. +To delete a registered Cron automation from Python, use `automations.delete`. You can also delete cron automations from the Tilebox Console. After deletion, no new jobs will be submitted by that Cron trigger. Past jobs already triggered will still remain queued. ```python Python from tilebox.workflows import Client @@ -110,7 +174,7 @@ automations.delete("0190bafc-b3b8-88c4-008b-a5db044380d0") ## Submitting Cron jobs manually -You can submit Cron tasks as regular tasks for testing purposes or as part of a larger workflow. To do so, instantiate the task with a specific trigger time using the `once` method. +In Python, you can submit Cron tasks as regular tasks for testing purposes or as part of a larger workflow. To do so, instantiate the task with a specific trigger time using the `once` method. Submitting a job with a Cron task using `once` immediately schedules the task, and a runner may pick it up and execute it. The trigger time set in the `once` method does not influence the execution time; it only sets the `self.trigger.time` attribute for the Cron task. diff --git a/workflows/near-real-time/storage-events.mdx b/workflows/automations/storage-events.mdx similarity index 75% rename from workflows/near-real-time/storage-events.mdx rename to workflows/automations/storage-events.mdx index 2cf2be2..521cfea 100644 --- a/workflows/near-real-time/storage-events.mdx +++ b/workflows/automations/storage-events.mdx @@ -4,15 +4,12 @@ description: Trigger workflow jobs automatically whenever new objects are create icon: right-to-line --- - - This feature is only available in the Python SDK. - - ## Creating a Storage Event Task Storage Event Tasks are automations triggered when objects are created or modified in a [storage location](#storage-locations). -To create a Storage Event task, use `tilebox.workflows.automations.StorageEventTask` as your tasks base class instead of the regular `tilebox.workflows.Task`. +In Python, use `tilebox.workflows.automations.StorageEventTask` as your task base class instead of the regular `tilebox.workflows.Task`. In Go, register an executable task with the same task identifier as the automation prototype. + ```python Python from tilebox.workflows import ExecutionContext from tilebox.workflows.automations import StorageEventTask, StorageEventType @@ -34,6 +31,17 @@ class LogObjectCreation(StorageEventTask): context.logger.info("Read object data", path=path, size_bytes=len(data)) context.logger.info("Object preview", path=path, preview=data[:self.head_bytes].hex()) ``` +```go Go +type LogObjectCreation struct { + HeadBytes int +} + +func (t *LogObjectCreation) Execute(ctx context.Context) error { + slog.InfoContext(ctx, "storage event task triggered", slog.Int("head_bytes", t.HeadBytes)) + return nil +} +``` + ## Storage Locations @@ -53,8 +61,9 @@ For example, a GCP storage bucket is integrated by setting up a [PubSub Notifica ### Listing Available Storage Locations -To list all available storage locations, use the `all` method on the storage location client. +To list all available storage locations, use the automation client. + ```python Python from tilebox.workflows import Client @@ -64,6 +73,26 @@ automations_client = client.automations() storage_locations = automations_client.storage_locations() print(storage_locations) ``` +```go Go +ctx := context.Background() +client := workflows.NewClient() + +storageLocations, err := client.Automations.ListStorageLocations(ctx) +if err != nil { + slog.ErrorContext(ctx, "failed to list storage locations", slog.Any("error", err)) + return +} + +for _, location := range storageLocations { + slog.InfoContext(ctx, + "storage location", + slog.String("id", location.ID.String()), + slog.String("location", location.Location), + slog.String("type", location.Type.String()), + ) +} +``` + ```plaintext Output [ @@ -108,7 +137,8 @@ local_object = local_folder.read("my-object.txt") ## Registering a Storage Event Trigger -After implementing a Storage Event task, register it to trigger each time a storage event occurs. This registration submits a new job consisting of a single task instance derived from the registered Storage Event task prototype. +After implementing a Storage Event task, register it to trigger each time a storage event occurs. The Python SDK provides a registration helper, and you can also register storage-event automations from the Tilebox Console. +This registration submits a new job consisting of a single task instance derived from the registered Storage Event task prototype. ```python Python from tilebox.workflows import Client @@ -147,13 +177,34 @@ Here are some examples of valid glob patterns: With the Storage Event automation registered, a job is submitted whenever a storage event occurs. But unless a [runner](/workflows/concepts/runners) is available to execute the Storage Event task the submitted jobs remain in a task queue. Once an [eligible runner](/workflows/concepts/runners#task-selection) becomes available, all jobs in the queue are executed. + ```python Python -from tilebox.workflows import Client +from tilebox.workflows import Client, Runner client = Client() -runner = client.runner(tasks=[LogObjectCreation]) -runner.run_forever() +runner = Runner(tasks=[LogObjectCreation]) +runner.connect_to(client).run_forever() +``` +```go Go +ctx := context.Background() +client := workflows.NewClient() + +runner, err := client.NewTaskRunner(ctx) +if err != nil { + slog.ErrorContext(ctx, "failed to create task runner", slog.Any("error", err)) + return +} + +if err := runner.RegisterTasks(&LogObjectCreation{}); err != nil { + slog.ErrorContext(ctx, "failed to register tasks", slog.Any("error", err)) + return +} + +if err := runner.RunForever(ctx); err != nil { + slog.ErrorContext(ctx, "task runner stopped", slog.Any("error", err)) +} ``` + ### Triggering an Event @@ -183,7 +234,7 @@ The [Tilebox Console](https://console.tilebox.com/workflows/automations) provide ## Deleting Storage Event automations -To delete a registered storage event automation, use `automations.delete`. After deletion, no new jobs will be submitted by the storage event trigger. Past jobs already triggered will still remain queued. +To delete a registered storage event automation from Python, use `automations.delete`. You can also delete storage-event automations from the Tilebox Console. After deletion, no new jobs will be submitted by the storage event trigger. Past jobs already triggered will still remain queued. ```python Python from tilebox.workflows import Client @@ -200,7 +251,7 @@ automations.delete("0190bafc-b3b8-88c4-008b-a5db044380d0") ## Submitting Storage Event jobs manually -You can submit Storage event tasks as regular tasks for testing purposes or as part of a larger workflow. To do so, instantiate the task with a specific storage location and object name using the `once` method. +In Python, you can submit Storage Event tasks as regular tasks for testing purposes or as part of a larger workflow. To do so, instantiate the task with a specific storage location and object name using the `once` method. ```python Python job_client = client.jobs() diff --git a/workflows/build-and-deploy/project-structure.mdx b/workflows/build-and-deploy/project-structure.mdx index 1cc4ac3..108ba82 100644 --- a/workflows/build-and-deploy/project-structure.mdx +++ b/workflows/build-and-deploy/project-structure.mdx @@ -6,6 +6,10 @@ icon: folder-tree A Python workflow project contains task classes, a `Runner` definition, and a `tilebox.workflow.toml` file. The Tilebox command-line tool uses these files to build a workflow release, discover the tasks it can execute, and make the release available to release runners after deployment. + + This project structure and `tilebox.workflow.toml` are required for [release runners](/workflows/concepts/runners#release-runners). For direct runners, you can organize your code however you want, as long as your runner process can import and register the tasks it executes. The same structure can still be useful when you want a small, importable workflow package. + + Keep the project small and importable from its root. Release builds import the configured runner object, check that the Python runtime starts, and package the selected files into an immutable artifact. ## Project structure diff --git a/workflows/concepts/runners.mdx b/workflows/concepts/runners.mdx index f91719f..a8772b6 100644 --- a/workflows/concepts/runners.mdx +++ b/workflows/concepts/runners.mdx @@ -27,7 +27,7 @@ The two modes differ in how the runner gets its task registrations and how you r | | Release runner | Direct runner | | --- | --- | --- | | Executable tasks | Loaded from workflow releases deployed to the runner's cluster | Registered directly in your script, service, or binary | -| Runtime | [Tilebox CLI](/agentic-development/tilebox-cli.mdx) invokes the Python workflow project runtime from the release artifact | Your python script or Go binary, implemented with the Tilebox SDK | +| Runtime | [Tilebox CLI](/agents-and-ai-tools/tilebox-cli.mdx) invokes the Python workflow project runtime from the release artifact | Your python script or Go binary, implemented with the Tilebox SDK | | Start command | `tilebox runner start --cluster ` | `python runner.py`, `./my-runner-binary`, or your own deployment | | rollout model | You publish releases and deploy them to clusters, the runner automatically picks up deployment changes | You deploy, restart, scale, and roll back the runner process yourself | | Best for | Reproducible releases, fast cluster deployments, and AI-assisted workflow iteration | Custom deployments, Go runners, and direct SDK control | @@ -113,7 +113,7 @@ Release runners advertise the task identifiers from workflow releases currently Start multiple runner processes to execute tasks in parallel. Each runner process claims and executes tasks independently. You can run several release runners, several direct runners, or a mix of both in the same cluster. This allows for high parallelism and can be used to scale the execution of tasks to handle large workloads. -To test this, run multiple instances of the runner script in different terminal windows on your local machine, or use the [CLI](/agentic-development/tilebox-cli) built-in `parallel` subcommand to start multiple runners in parallel. +To test this, run multiple instances of the runner script in different terminal windows on your local machine, or use the [CLI](/agents-and-ai-tools/tilebox-cli) built-in `parallel` subcommand to start multiple runners in parallel. ```bash # start multiple release runners in parallel @@ -354,4 +354,4 @@ This mechanism ensures that scenarios such as power outages, hardware failures, ## Observability -Tilebox captures logs, spans, task states, and runner context from both runner modes. Use [Workflow observability](/workflows/observability/introduction) to inspect job execution, task failures, and runner behavior. +Tilebox captures logs, spans, task states, and runner context from both runner modes. Use [Workflow observability](/workflows/run-and-inspect/introduction) to inspect job execution, task failures, and runner behavior. diff --git a/workflows/concepts/tasks.mdx b/workflows/concepts/tasks.mdx index 8fbc531..3bb034e 100644 --- a/workflows/concepts/tasks.mdx +++ b/workflows/concepts/tasks.mdx @@ -170,7 +170,7 @@ task := &ParentTask{numSubtasks: 5} In this example, a `ParentTask` submits `ChildTask` tasks as subtasks. The number of subtasks to be submitted is based on the `num_subtasks` attribute of the `ParentTask`. The `submit_subtask` method takes an instance of a task as its argument, meaning the task to be submitted must be instantiated with concrete parameters first. -Parent task do not have access to results of subtasks, instead, tasks can use [shared caching](/workflows/caches#storing-and-retrieving-data) to share data between tasks. +Parent task do not have access to results of subtasks, instead, tasks can use [shared caching](/workflows/run-and-inspect/caches#storing-and-retrieving-data) to share data between tasks. By submitting a task as a subtask, its execution is scheduled as part of the same job as the parent task. Compared to just directly invoking the subtask's `execute` method, this allows the subtask's execution to occur on a different machine or in parallel with other subtasks. To learn more about how tasks are executed, see the section on [runners](/workflows/concepts/runners). @@ -982,7 +982,7 @@ This workflow consists of four tasks: An important aspect is that there is no dependency between the `PrintHeadlines` and `MostFrequentAuthors` tasks. This means they can execute in parallel, which the Tilebox Workflow Orchestrator will do, provided multiple runners are available. - In this example, the results from `FetchNews` are stored in a file. This is not the recommended method for passing data between tasks. When executing on a distributed cluster, the existence of a file written by a dependent task cannot be guaranteed. Instead, it's better to use a [shared cache](/workflows/caches). + In this example, the results from `FetchNews` are stored in a file. This is not the recommended method for passing data between tasks. When executing on a distributed cluster, the existence of a file written by a dependent task cannot be guaranteed. Instead, it's better to use a [shared cache](/workflows/run-and-inspect/caches). ## Optional Tasks diff --git a/workflows/introduction.mdx b/workflows/introduction.mdx index 46b1e30..83de5fa 100644 --- a/workflows/introduction.mdx +++ b/workflows/introduction.mdx @@ -6,6 +6,10 @@ icon: network-wired mode: wide --- + + Want to learn by example? Run your first workflow by creating a task, starting a runner, submitting a job, and inspecting the result. + + Tilebox Workflows is a parallel processing engine for space data operations. It helps you turn processing steps such as fetching scenes, validating products, generating previews, running models, and publishing outputs into [tasks](/workflows/concepts/tasks) that can run across local machines, on-premises systems, and cloud environments. @@ -26,7 +30,7 @@ This model also gives AI coding agents a practical iteration loop: edit workflow Submit a root task to a cluster and let runners execute the resulting task graph. - + Learn how to set up a runner in order to execute tasks. @@ -35,7 +39,7 @@ This model also gives AI coding agents a practical iteration loop: edit workflow Map releases to clusters and run them with `tilebox runner start`. - + Use logs, traces, and job state to debug workflows across runners. diff --git a/workflows/caches.mdx b/workflows/run-and-inspect/caches.mdx similarity index 100% rename from workflows/caches.mdx rename to workflows/run-and-inspect/caches.mdx diff --git a/workflows/observability/integrations/axiom.mdx b/workflows/run-and-inspect/integrations/axiom.mdx similarity index 100% rename from workflows/observability/integrations/axiom.mdx rename to workflows/run-and-inspect/integrations/axiom.mdx diff --git a/workflows/observability/integrations/open-telemetry.mdx b/workflows/run-and-inspect/integrations/open-telemetry.mdx similarity index 100% rename from workflows/observability/integrations/open-telemetry.mdx rename to workflows/run-and-inspect/integrations/open-telemetry.mdx diff --git a/workflows/observability/introduction.mdx b/workflows/run-and-inspect/introduction.mdx similarity index 87% rename from workflows/observability/introduction.mdx rename to workflows/run-and-inspect/introduction.mdx index fcee3ea..73d2261 100644 --- a/workflows/observability/introduction.mdx +++ b/workflows/run-and-inspect/introduction.mdx @@ -1,7 +1,6 @@ --- -title: Workflow observability -sidebarTitle: Overview -description: Inspect workflow logs, traces, task status, and runner behavior. +title: Inspect workflow runs +description: Inspect workflow logs, traces, task status, and runner behavior while jobs execute. icon: lightbulb --- @@ -18,25 +17,25 @@ Use the built-in view for day-to-day debugging and operations. Add structured lo @@ -137,4 +136,4 @@ The parent task, spawned subtasks, task logs, and spans share the same job trace ## Integrate with external observability platforms -If your team uses another observability platform, configure [OpenTelemetry](/workflows/observability/integrations/open-telemetry) or [Axiom](/workflows/observability/integrations/axiom) export in the runner process. Tilebox keeps the workflow state, while your platform receives the same logs and traces for alerting, long-term storage, or analysis. +If your team uses another observability platform, configure [OpenTelemetry](/workflows/run-and-inspect/integrations/open-telemetry) or [Axiom](/workflows/run-and-inspect/integrations/axiom) export in the runner process. Tilebox keeps the workflow state, while your platform receives the same logs and traces for alerting, long-term storage, or analysis. diff --git a/workflows/observability/logging.mdx b/workflows/run-and-inspect/logging.mdx similarity index 96% rename from workflows/observability/logging.mdx rename to workflows/run-and-inspect/logging.mdx index ff96818..1475f3e 100644 --- a/workflows/observability/logging.mdx +++ b/workflows/run-and-inspect/logging.mdx @@ -208,8 +208,8 @@ func main() { ``` -See [Query telemetry](/workflows/observability/query) for the log and span query APIs. +See [Query telemetry](/workflows/run-and-inspect/query-telemetry) for the log and span query APIs. ## Export to another backend -Tilebox stores logs by default. To export logs to your own observability backend as well, configure an [OpenTelemetry](/workflows/observability/integrations/open-telemetry) or [Axiom](/workflows/observability/integrations/axiom) integration when the runner process starts. +Tilebox stores logs by default. To export logs to your own observability backend as well, configure an [OpenTelemetry](/workflows/run-and-inspect/integrations/open-telemetry) or [Axiom](/workflows/run-and-inspect/integrations/axiom) integration when the runner process starts. diff --git a/workflows/progress.mdx b/workflows/run-and-inspect/progress.mdx similarity index 100% rename from workflows/progress.mdx rename to workflows/run-and-inspect/progress.mdx diff --git a/workflows/observability/query.mdx b/workflows/run-and-inspect/query-telemetry.mdx similarity index 100% rename from workflows/observability/query.mdx rename to workflows/run-and-inspect/query-telemetry.mdx diff --git a/workflows/observability/tracing.mdx b/workflows/run-and-inspect/tracing.mdx similarity index 94% rename from workflows/observability/tracing.mdx rename to workflows/run-and-inspect/tracing.mdx index 3b07e08..ddc3a45 100644 --- a/workflows/observability/tracing.mdx +++ b/workflows/run-and-inspect/tracing.mdx @@ -153,8 +153,8 @@ func main() { ``` -See [Query telemetry](/workflows/observability/query) for the log and span query APIs. +See [Query telemetry](/workflows/run-and-inspect/query-telemetry) for the log and span query APIs. ## Export to another backend -Tilebox stores traces by default. To export spans to your own observability backend as well, configure an [OpenTelemetry](/workflows/observability/integrations/open-telemetry) or [Axiom](/workflows/observability/integrations/axiom) integration when the runner process starts. +Tilebox stores traces by default. To export spans to your own observability backend as well, configure an [OpenTelemetry](/workflows/run-and-inspect/integrations/open-telemetry) or [Axiom](/workflows/run-and-inspect/integrations/axiom) integration when the runner process starts.