Allow remote dataset access

[dentiny]

Closes #12

This PR adds remote storage support, so data viewer could load lance dataset from not only local filesystem.
The core change is pretty simple actually, since lance dataset already support remote access natively.

Initial UI

After a successful load

Followup item

• Remote storage access is much slower than local access, I have a few optimizations on parallelism and cutting down IO requests in my own fork, will send out PR shortly: https://github.com/dentiny/lance-data-viewer

AI usage disclaimer:
GPT-5.5 helped me make the code change, I deployed my own fork to k8s and verified it worked

[gordonmurray]

Thanks for taking on remote dataset support. This addresses a real project need, but I’m requesting changes before merge.

There are two project-level requirements I need to settle as maintainer:

1. The existing local mounted-dataset workflow must remain available. This PR currently replaces the local dataset list with a remote URI form. Remote access should be additive rather than removing the primary local workflow.

2. The security model for arbitrary user-supplied URIs needs to be explicit. The viewer has no authentication, so allowing any browser user to submit s3://, filesystem, or other URIs could expose datasets accessible through the server’s credentials. Remote access needs a defined restriction/allowlist or a clearly trusted deployment model.

Please hold the implementation at this point while we agree those two design decisions. Once agreed, the revision should include:

• tests for the supported URI schemes and access policy;
• preservation of local dataset browsing.

The capability is valuable, but the current UI regression and unrestricted URI access mean this is not safe to merge yet.

[dentiny]

Thanks for the mention!

On the security model, there are two separate boundaries:

• Access to the web application: Each Lance Data Viewer instance is deployed as a Kubernetes service within a single cluster and namespace. The Service is private (ClusterIP) and exposed through Cloudflare Tunnel. Cloudflare Access authenticates users through Google SSO and controls who may access that viewer’s hostname.
• Access to storage: Each viewer deployment has one dedicated storage identity, provided through a Kubernetes Secret or workload identity in its namespace. Lance uses these ambient credentials when opening datasets. The identity must be read-only and restricted to the approved buckets or prefixes.

All users admitted to one viewer share that deployment’s storage permissions. Different access domains should use separate viewer deployments, namespaces, credentials, and hostname access policies.

This is a trusted-deployment model rather than per-user authorization inside Lance Data Viewer. Deployments must not expose a credentialed viewer without external authentication.

[gordonmurray]

Your deployment model makes sense, and I agree that per-user authorization does not belong in this viewer. We will go with it.

The mode has to be opt-in, so a default container never shows a location box. Someone who runs the documented docker run with a :ro mount must get exactly what they get today. Remote access turns on when the operator asks for it, which is what your Kubernetes setup describes anyway.

There is a concrete reason for the guard beyond access control. lancedb.connect() creates the directory when the path does not exist. In 0.36.0 that is Path(uri).mkdir(parents=True, exist_ok=True) in db.py, and 0.33.0 does the same. So an unvalidated location box lets a typo write to the server filesystem. This project promises never to write to Lance data, so I want that closed in code rather than documented.

What that means in practice:

• the local mount stays the default and keeps working with no new flags
• the location field appears only when the operator enables it
• a location that does not resolve to an existing dataset returns an error instead of creating a directory

[gordonmurray]

One more thing before you rework this.

The PR adds pylance as a runtime dependency, through lancedb[pylance] in requirements.txt and pylance==0.8.17 in pyproject.toml. Those two disagree with each other, and pylance is not pinned in any of the eight constraints
files. I resolved both ends of the matrix in a container:

• with constraints-0.36.0.txt it installs pylance 10.0.0, the current latest. Nothing holds it there, so it moves on the next pylance release.
• with constraints-0.3.1.txt pip prints WARNING: lancedb 0.3.1 does not provide the extra 'pylance'. Old lancedb depends on pylance anyway, so it still arrives, but the extra itself does nothing.

Every other dependency here is pinned per matrix entry, so an unpinned major across all eight images stands out.

Worth settling before you spend time on it: this PR and #84 take different approaches to the same problem. Here you open one dataset by full URI with lance.dataset(), behind separate /remote/* endpoints. In #84 you point
lancedb.connect() at a location and list tables, which reaches object storage without pylance and without new endpoints. Both are reasonable, but I would rather ship one of them than both. My preference is the #84 shape, because
it keeps a single code path with a single guard on it, and the local mount already works that way.

Rework against current main first, then merge. Most of the conflicting regions are in code the rework touches anyway, so they largely resolve themselves.

[gordonmurray]

This endpoint accepts arbitrary user-supplied URIs and passes them directly to lance.dataset(). Because local filesystem paths are also valid Lance URIs, a user can bypass the configured DATA_PATH//data boundary and read any
Lance dataset accessible to the server process.

I’m not suggesting removing remote storage support, but the trust boundary needs to be good. Possible approaches:

• allow local paths only when they resolve beneath a configured data root;
• restrict remote access to explicitly enabled schemes/hosts or an allowlist;
• keep local /data mode as the default and require an explicit configuration flag for user-supplied locations;
• add tests proving paths outside the allowed root are rejected.

This is important because the viewer is unauthenticated by default, so arbitrary URI access could expose server-side data to anyone who can reach the UI.