Add labels to jobs_by_organization#9719
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR adds a labels column to monitoring_derived.jobs_by_organization_v1, selecting it from INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION and adding a matching REPEATED RECORD field (key/value STRINGs) to schema.yaml.
The change looks correct: the schema type matches the source column's ARRAY<STRUCT<key STRING, value STRING>> shape, the field ordering in the SELECT matches its position in the schema, and the trailing comma before FROM is valid BigQuery. There is no published view over this table and downstream consumers don't require the new field, so no additional schema updates are needed. I have no findings to raise.
Note: since this only appends a column, historical partitions won't be backfilled with label data — new values will populate going forward (and for the previous day, which main() reprocesses). If backfilled label history is needed, that would require a separate backfill.
whd
left a comment
whd
left a comment
There was a problem hiding this comment.
Labels are explicit metadata the intended use of which is monitoring so this isn't something that requires additional policy discussion.
| bi_engine_statistics.bi_engine_mode AS bi_engine_mode, | ||
| bi_engine_statistics.acceleration_mode AS acceleration_mode, | ||
| bi_engine_statistics.bi_engine_reasons AS bi_engine_reasons, | ||
| labels, |
There was a problem hiding this comment.
suggestion: With labels now being included in this ETL, the bigquery_usage_v2 ETL should probably be refactored to get labels from this table rather than the JOBS_BY_PROJECT view, because it only queries the JOBS_BY_PROJECT view for the default projects.
This comment has been minimized.
This comment has been minimized.
Integration report
|
4 participants
Description
There are useful labels for auditing where we don't have access to the source projects https://github.com/mozilla/webservices-infra/pull/11890
Reviewer, please follow this checklist