Add an adapter for benchmark.oss_ci_benchmark_v3 #5921
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
facebook-github-bot
added
the
CLA Signed
|
@clee2000 Do you know if the file on S3 needs to be public accessible for the replication to work? i.e. https://ossci-benchmarks.s3.us-east-1.amazonaws.com/v3/pytorch/pytorch/11831781500/33022921916/add_loop_eager_dynamic.json needs to be public |
It needs either public access or for the clickhouse role to have access. You can test if it works by performing the query in the query console or by running the script locally |
|
Yeah, I have been able to run this locally and add some records to that table from S3 |
| extra_info Map(String, String) | ||
| ), | ||
| """ | ||
| general_adapter(table, bucket, key, schema, ["none"], "JSONEachRow") |
There was a problem hiding this comment.
don't you use gzip in the upload?
There was a problem hiding this comment.
Ok, I thought that this is the format of the source files on S3, they aren't gzipped, but let me add gzip here
The schema comes from https://github.com/pytorch/test-infra/blob/main/torchci/clickhouse_queries/oss_ci_benchmark_v3/query.sql. An example S3 path is
s3://ossci-benchmarks/v3/pytorch/pytorch/11850871071/33027181871/add_loop_eager_dynamic.jsonI think we should figure out how to test changes to these replicator lambdas. Otherwise, we might lose some data if they break. Any thoughts?