databricks-mlflow-evaluation: fix four UX gaps surfaced by a user study - #589
Open
adamgurary wants to merge 1 commit into
Open
databricks-mlflow-evaluation: fix four UX gaps surfaced by a user study#589adamgurary wants to merge 1 commit into
adamgurary wants to merge 1 commit into
Conversation
A user study ran a coding agent through the databricks-mlflow-evaluation skill end to end and hit four gaps: 1. (highest impact) Inspecting a dataset via get_dataset()/.to_df() imports databricks-agents, which pulls databricks-connect (pinned numpy<2). Installing it downgraded numpy in an env that had numpy>=2 and broke every downstream import. Add a GOTCHA + inline note: inspect a dataset's columns with w.tables.get() or DESCRIBE TABLE over a SQL warehouse (databricks-sdk only), never pip/uv install databricks-agents at inspection time. 2. The databricks-agents dependency was not stated up front. Expand the install section and note the numpy<2 pin. 3. openai.autolog() was the default example even for LangGraph agents, which drops node spans. Add an autolog selector table + GOTCHA (langchain.autolog for LangChain/LangGraph, needs only langchain-core). 4. set_experiment by numeric ID was not shown. Add the experiment_id= form to setup and a GOTCHA for the numeric-name failure. All changes are skill markdown (GOTCHAS.md, CRITICAL-interfaces.md, patterns-datasets.md). No code paths affected. Co-authored-by: Isaac Signed-off-by: Adam Gurary <guraryadam@gmail.com>
adamgurary
force-pushed
the
mlflow-eval-skill-ux-fixes
branch
from
July 30, 2026 23:32
d6a3edc to
51085c4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Running the
databricks-mlflow-evaluationskill end to end during a user study surfaced four gaps in the skill guidance. One of them broke the user's Python environment. All four fixes are self-contained edits to the skill markdown (GOTCHAS.md,CRITICAL-interfaces.md,patterns-datasets.md).1 (highest impact): inspecting a dataset's schema installs
databricks-agentsand can break the environmentThe skill teaches
mlflow.genai.datasets.get_dataset()/.to_df()to inspect a persistent dataset. On a Databricks backend these importdatabricks-agents, which depends ondatabricks-connect(pinnednumpy<2). When a coding agent hits the import error and runspip install databricks-agents, the install downgrades numpy in an environment that already has numpy 2.x and breaks every downstream import (AttributeError: module 'numpy' has no attribute 'long'). Added a GOTCHA plus an inline note: inspect a dataset's columns withw.tables.get()orDESCRIBE TABLEover a SQL warehouse (needs onlydatabricks-sdk, already installed), neverpip install databricks-agentsat inspection time.2: the
databricks-agentsdependency was not stated up frontExpanded the install section in
CRITICAL-interfaces.mdto namedatabricks-agentsand itsnumpy<2pin.3:
mlflow.openai.autolog()was the default even for LangGraph agentsopenai.autolog()captures only the raw LLM call and drops LangGraph node spans. Added an autolog selector table and a GOTCHA: LangChain and LangGraph usemlflow.langchain.autolog()(which needs onlylangchain-core).4: attaching to an experiment by numeric ID was not shown
Added the
mlflow.set_experiment(experiment_id="<id>")form to the setup example and a GOTCHA for the numeric-name failure.Testing
Docs-only change to skill markdown. Verified the three files have balanced code fences and that the new Table of Contents anchors resolve to their headings. No code paths affected.