[Feature] Add PPL makeresults command#5622
Conversation
PR Reviewer Guide 🔍(Review updated until commit 9dc6277)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 9dc6277 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 58ab1c4
Suggestions up to commit 1257c59
Suggestions up to commit 3a8654b
Suggestions up to commit 8e6ecc8
|
8e6ecc8 to
3a8654b
Compare
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 3a8654b.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
|
Persistent review updated to latest commit 3a8654b |
3a8654b to
1257c59
Compare
|
Persistent review updated to latest commit 1257c59 |
Add the makeresults leading command on the Calcite (v3) path. It generates in-memory rows with no index scan: - count=N produces N rows, each with a single _time timestamp set to query time - format=csv|json data=... parses an inline literal into typed rows, with column types synthesized following OpenSearch dynamic-mapping semantics and surfaced through the same type path an index scan uses (JSON int->long, decimal->float, string->keyword; typed CSV name:type via cast's vocabulary; bare CSV->string) Grammar lives in the ppl/ copies only. Adds the MakeResults AST node, AstBuilder wiring with MakeResultsDataParser, CalciteRelNodeVisitor.visitMakeResults building LogicalValues + Project, a V2 Analyzer reject stub, and anonymizer rendering. Includes unit tests, AstBuilder and anonymizer tests, integration tests, and the user doc. Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
1257c59 to
58ab1c4
Compare
|
Persistent review updated to latest commit 58ab1c4 |
| if (v.isNull()) { | ||
| return null; | ||
| } | ||
| if (v.isObject() || v.isArray()) { |
There was a problem hiding this comment.
Do we expect to reject those nested field types? Or just fallback to serialized JSON strings? I think spath command could resolve the json string later.
| public void testJson() throws IOException { | ||
| JSONObject result = | ||
| executeQuery( | ||
| "makeresults format=json data='[{\"name\":\"John\",\"age\":35,\"score\":3.5}," | ||
| + "{\"name\":\"Sarah\",\"age\":39,\"score\":4.0}]'"); | ||
| verifySchema( | ||
| result, schema("name", "string"), schema("age", "bigint"), schema("score", "float")); | ||
| verifyDataRows(result, rows("John", 35, 3.5), rows("Sarah", 39, 4.0)); | ||
| } |
There was a problem hiding this comment.
We should generate @timestamp default time fields as well for specified json data.
| @Test | ||
| public void testTypedCsv() throws IOException { | ||
| JSONObject result = | ||
| executeQuery("makeresults format=csv data='name:string,age:int\nJohn,35\nSarah,39'"); | ||
| verifySchema(result, schema("name", "string"), schema("age", "int")); | ||
| verifyDataRows(result, rows("John", 35), rows("Sarah", 39)); | ||
| } |
| @Test | ||
| public void testBareCsv() throws IOException { | ||
| JSONObject result = executeQuery("makeresults format=csv data='name,age\nJohn,35\nSarah,39'"); | ||
| verifySchema(result, schema("name", "string"), schema("age", "string")); | ||
| verifyDataRows(result, rows("John", "35"), rows("Sarah", "39")); | ||
| } |
| if (data.length() > 29999) { | ||
| throw new SyntaxCheckException("makeresults data must not exceed 29999 characters"); | ||
| } |
There was a problem hiding this comment.
Nit: Claim this constraint in the md doc.
| case STRING: | ||
| return s; | ||
| case BOOLEAN: | ||
| return value instanceof Boolean ? value : Boolean.parseBoolean(s); |
There was a problem hiding this comment.
Boolean.parseBoolean() will treat any invalid boolean value string to false. For example, Boolean.parseBoolean("not true") -> false.
Also, these edge cases have some point as well. |
| String[] header = splitCsvLine(lines[0]).toArray(new String[0]); | ||
| List<String> names = new ArrayList<>(); | ||
| List<ExprCoreType> types = new ArrayList<>(); | ||
| for (String token : header) { | ||
| String t = token.trim(); | ||
| int colon = t.lastIndexOf(':'); | ||
| if (colon > 0 && colon < t.length() - 1) { | ||
| String maybeType = t.substring(colon + 1).trim(); | ||
| ExprCoreType declared = resolveType(maybeType); | ||
| if (declared != null) { | ||
| names.add(t.substring(0, colon).trim()); | ||
| types.add(declared); | ||
| continue; | ||
| } | ||
| } | ||
| names.add(t); | ||
| types.add(ExprCoreType.STRING); | ||
| } |
There was a problem hiding this comment.
How about dedupe the field names while processing csv headers? Or we can check if Project can handle them. I remember SqlValidatorUtil.uniquify could append identity numbers to avoid them. For example, 'field, field' names will be uniquified as 'field, field0'
There was a problem hiding this comment.
And how about the blank field handling? Like ',field'?
Signed-off-by: Louis Chu <lingzhichu.clz@gmail.com>
|
Persistent review updated to latest commit 9dc6277 |
Description
Adds the
makeresultsleading command on the Calcite (v3) path. It generates in-memory rows with no index scan.Count path
| makeresults/| makeresults count=Nproduces N rows (default 1), each with a single@timestamp(TIMESTAMP) column set to query time (now(), one value for the whole query).@timestampis OpenSearch PPL's implicit time field (OpenSearchConstants.IMPLICIT_FIELD_TIMESTAMP), so the generated rows compose with the time-aware command family (sort,stats ... by span,timechart,reverse,timewrap). This intentionally uses@timestamprather than a synthetic_timecolumn.format/data path
format=csv|json data="..."parses an inline literal into typed rows. Column types are synthesized at plan time following OpenSearch dynamic-mapping semantics and surfaced through the same type path an index scan uses:long(bigint), JSON decimal →float(REAL), JSON boolean →boolean, JSON string →keyword/string.name:typeuses the same vocabulary ascast(e.g.age:int); a bare CSV header token defaults tostring.Grammar lives in the
ppl/copies only. The command adds theMakeResultsAST node,AstBuilderwiring withMakeResultsDataParser,CalciteRelNodeVisitor.visitMakeResults(buildsLogicalValues+Project), a V2Analyzerreject stub (V2 returns the standard "supported only whenplugins.calcite.enabled=true" error), and anonymizer rendering.Robustness / hardening
count=0and header-only CSV build an empty typedLogicalValues(tuples=[[]])with the parsed rowType (previously threw a raw Calcite "Value count must be a positive multiple of field count"). A negativecountalso yields zero rows.SyntaxCheckException(previously NPE viaMap.merge).longkeep full precision as a string column instead of silently wrapping."").""escape).count≤ 1,000,000;data≤ 29,999 chars.Notes / intentional divergences
bigintin the response schema, because makeresults rows carry no index mapping to mapbigintback to the PPL type namelong.date/time/timestamp/ip/jsoninline CSV types are not supported on this path (usestring+cast).@timestamp = now()) is non-deterministic, so it is not registered for doctest; the deterministicdata=examples can be added after cluster validation.Related Issues
Closes #3629
RFC: #5626
Check List
docs/user/ppl/cmd/makeresults.md).--signoff.spotlessCheckpassed.CalcitePPLMakeResultsTest13/13,AstBuilderTest,PPLQueryDataAnonymizerTest).CalcitePPLMakeResultsIT6/6,NewAddedCommandsIT.testMakeResults).By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.