How to Detect Jagged Rows in Bigquery?
We Have a Data Issue Where We'd Like to Take a Backup of a Particular Kind and Determine Which Rows Are "Jagged", So Effectively I'm Trying to Detect Which...
We have a data issue where we'd like to take a backup of a particular Kind and determine which rows are "jagged", so effectively I'm trying to detect which rows are missing a certain column (meaning the field does not exist on that row, which I'm distinguishing from being null). Is there a way to do this in BigQuery?
1 Answer
From docs:
configuration.load.allowJaggedRows boolean [Optional] Accept rows that are missing trailing optional columns. The missing values are treated as nulls. Default is false which treats short rows as errors. Only applicable to CSV, ignored for other formats.
This means missing values from jagged rows will be treated as null. You might need to try a different approach if preserving these values is important, like ingesting the whole row and parsing inside BigQuery - when possible.
James Sterling reports on renewable energy developments, climate policy, ecological conservation, and green tech innovations around the globe.
