Features to upload historical data into BigQuery
When uploading data to Big Query, we can find different options depending on our needs. It is not the same to upload data from a couple of months than from several years, so we are going to see the different options we have when transferring our information using Dataslayer.
Schedule, to schedule periodic uploads. Ideal for keeping the data updated periodically.
Manual (ideal for upload Historical Data), for medium data volume and immediate upload.
Manual (ideal for loading large volume historical data). This is the best option for uploading large historical data. To avoid timeouts, interruptions and errors, this function divides the queries by groups and executes them periodically, so that the data will be uploaded little by little in an uninterrupted way. Important: note that this function can only be executed when the query has temporal dimensions added.
When this option is used, the query can take a long time, so once executed, even if the browser is closed, the query will continue to run.
During the execution of the query, no other queries can be performed in Big Query.
To upload historical data, the manual option is the most indicated, and depending on the quantity, it will be convenient to choose the first or the second option (when in doubt, it is better to use the second option; the duration will be longer, but it ensures a successful transfer).
After uploading all the historical data, you can use the "schedule" function to program a daily update that uploads the previous day's data every day, so that your data is always up to date.
Schedule, to schedule periodic uploads. Ideal for keeping the data updated periodically.
Manual (ideal for upload Historical Data), for medium data volume and immediate upload.
Manual (ideal for loading large volume historical data). This is the best option for uploading large historical data. To avoid timeouts, interruptions and errors, this function divides the queries by groups and executes them periodically, so that the data will be uploaded little by little in an uninterrupted way. Important: note that this function can only be executed when the query has temporal dimensions added.
When this option is used, the query can take a long time, so once executed, even if the browser is closed, the query will continue to run.
During the execution of the query, no other queries can be performed in Big Query.
To upload historical data, the manual option is the most indicated, and depending on the quantity, it will be convenient to choose the first or the second option (when in doubt, it is better to use the second option; the duration will be longer, but it ensures a successful transfer).
After uploading all the historical data, you can use the "schedule" function to program a daily update that uploads the previous day's data every day, so that your data is always up to date.
Updated on: 03/11/2022
Thank you!