Features to upload historical data into BigQuery
When uploading data to Big Query, we can find different options depending on our needs. It is not the same to upload data from a couple of months than from several years, so we are going to see the different options we have when transferring our information using Dataslayer.
If you are not familiar with our Big Query product we explain everything you need to know in this FAQ.
To upload historical data, the manually option is the most indicated, but depending on the quantity of data you have, it may sometimes be better to choose manually splitted.
Once the query is created, we must pay attention to the date range we have selected and the amount of data we have for this date range.
If you have a small volume of data (one month, or one year with a small amount of information): You can use the manually option for a immediate upload of your data. The manually works in such a way that it uploads all the data at once.
Manually Splitted option
If you have a large volume of data (a month with a lot of data and many accounts, or more than a year): You can use the manually splitted option. To avoid timeouts, interruptions and errors, this function divides the queries by groups and executes them periodically, so that the data will be uploaded little by little in an uninterrupted way.
Important: Note that this function can ONLY be executed when the query has TEMPORAL DIMENSIONS added.
When this option is used, the query can take a long time, so once executed, even if the browser is closed, the query will continue to run.
Once you have your historical data, the best practice is to modify the transfer's date range to yesterday and set the daily schedule option.
Warning: Make sure that your dataset has the default expiration date set to never otherwise the tables will be deleted automatically. We explain it here.
Select this option to schedule periodic uploads. Ideal for keeping the data updated periodically.
As always, please contact us via our live chat on our website or via email if you still have doubts or questions. We are happy to help!
Updated on: 01/09/2023