Skip to content

Graphql loaders#2674

Open
wyatt-joyner-pometry wants to merge 11 commits into
db_v4from
graphql-loaders
Open

Graphql loaders#2674
wyatt-joyner-pometry wants to merge 11 commits into
db_v4from
graphql-loaders

Conversation

@wyatt-joyner-pometry

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add load_nodes_from_parquet and load_edges_from_parquet GraphQL endpoints. It is specifically load_nodes_from_parquet rather than load_nodes, because I only written for this capacity at first.

Note that I am also not allowing shared_metadata, since we'll have to come up with a format for how to interpret types.

Claude mostly wrote the AppConfig changes for the allowed_parquet_paths, but there are tests for negative and positive cases of writing to allowed parquet paths.

Why are the changes needed?

We need endpoints for loading parquet files over GraphQL.

Does this PR introduce any user-facing change? If yes is this documented?

I've copied over the descriptions from the corresponding Python API.

How was this patch tested?

Added tests alongside the other existing GraphQL tests.

Are there any further changes required?

add support for shared_metadata
add support for other input formats like csv and raw blob
(more generally) add support for column name-mapping and casting

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite Current: b44fbb7 Previous: 9823ef7 Ratio
lotr_graph/num_edges 4 ns/iter (± 0) 0 ns/iter (± 0) +∞
lotr_graph/num_nodes 4 ns/iter (± 0) 1 ns/iter (± 0) 4
lotr_graph/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph/graph_latest 3 ns/iter (± 0) 0 ns/iter (± 0) +∞
lotr_graph_materialise/materialize 8064596 ns/iter (± 26120) 1564816 ns/iter (± 35303) 5.15
lotr_graph_window_100/num_nodes 14 ns/iter (± 0) 5 ns/iter (± 0) 2.80
lotr_graph_window_100_materialise/materialize 8126806 ns/iter (± 21229) 1669150 ns/iter (± 10700) 4.87
lotr_graph_window_10/has_node_existing 142 ns/iter (± 9) 62 ns/iter (± 11) 2.29
lotr_graph_window_10_materialise/materialize 3313396 ns/iter (± 14928) 971980 ns/iter (± 4278) 3.41
lotr_graph_subgraph_10pc/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph_subgraph_10pc_materialise/materialize 1980515 ns/iter (± 14471) 334634 ns/iter (± 1287) 5.92
lotr_graph_subgraph_10pc_windowed/has_node_existing 147 ns/iter (± 10) 62 ns/iter (± 14) 2.37
lotr_graph_subgraph_10pc_windowed_materialise/materialize 1200531 ns/iter (± 16188) 230399 ns/iter (± 2617) 5.21
lotr_graph_window_50_layered/num_edges_temporal 149761 ns/iter (± 2678) 70121 ns/iter (± 7586) 2.14
lotr_graph_window_50_layered/has_node_existing 413 ns/iter (± 23) 129 ns/iter (± 12) 3.20
lotr_graph_window_50_layered/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph_window_50_layered/graph_latest 87112 ns/iter (± 1704) 36649 ns/iter (± 916) 2.38
lotr_graph_window_50_layered_materialise/materialize 30561603 ns/iter (± 527418) 3488825 ns/iter (± 24948) 8.76
lotr_graph_persistent_window_50_layered/num_edges_temporal 645419 ns/iter (± 6750) 192686 ns/iter (± 1569) 3.35
lotr_graph_persistent_window_50_layered/has_node_existing 448 ns/iter (± 403) 174 ns/iter (± 83) 2.57
lotr_graph_persistent_window_50_layered/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph_persistent_window_50_layered/iterate_exploded_edges 3476880 ns/iter (± 28692) 1659940 ns/iter (± 19402) 2.09
lotr_graph_persistent_window_50_layered/graph_latest 136183 ns/iter (± 3302) 57549 ns/iter (± 4809) 2.37
lotr_graph_persistent_window_50_layered_materialise/materialize 53423837 ns/iter (± 149257) 5298035 ns/iter (± 147912) 10.08

This comment was automatically generated by workflow using github-action-benchmark.

@fabianmurariu fabianmurariu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but There are a few changes needed here and there

Comment thread raphtory-graphql/src/model/mod.rs Outdated
Comment thread raphtory-graphql/src/model/mod.rs Outdated
Comment thread raphtory-graphql/src/model/mod.rs Outdated
Comment thread raphtory-graphql/src/model/mod.rs Outdated
Comment thread raphtory-graphql/src/model/mod.rs Outdated
Comment thread raphtory-api/src/core/entities/properties/prop/prop_type.rs Outdated
Comment thread raphtory-graphql/src/model/mod.rs Outdated
Comment thread raphtory-graphql/src/lib.rs
Comment thread raphtory-graphql/src/lib.rs
Comment thread raphtory-graphql/src/lib.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants