Add ability to select a subset of assets for backfill/Update in a view.
The current 'Update' feature backfills all assets matching the asset shape defined in the view.
However, when there is a need to back-fill only a small set of assets in a view there is no option to select such assets.
Adding a select option would prevent updating all assets in the view and user could select only those assets which require a back-fill
Suhas Gundoor
shared this idea
Thank you for your feedback! We would need more discussion with supporters to understand the problem.
How does the lack of this functionality impact your work? How often does this happen? How many assets are typically in your view? And how many assets are typically affected?
Does the new automatic updates through persistent signups (introduced in 2018) mitigate this problem at all?
Please feel free to email me or post a comment to help us understand the pain point.
2 comments
-
Bo Sun
commented
We're feeling the same pain because of lacking "selective" backfill/update. We don't have a complete AF tree in place, which means we would keep adding more AF elements which would match Integrator views. These views are streaming data into Azure Eventhub, auto-captured to Blob store, being processed by Databricks, and finally writing results to Postgresql.
Without "selective" backfill for only new matches, we have to backfill all in the views, which would take a much longer time, and what's even worse is introducing duplicates for all downstream processes and extra cost for storage and computation in Azure.
-
kbobeck
commented
We use the Integrator to help cleanse utility billing data. In our Electric Meter View, which publishes every 15 minutes, we have 122 accounts. Needing to republish for a couple of accounts means republishing for all of them and, depending on the time period, could mean millions of rows being republished unnecessarily and hours (yes I do mean hours) of time for the Integrator to carry out the task. As far as the automatic updates go, I'm unsure of the full scope of it. If I've added Manual data to an account for 2016, but have never published 2016 for that account, will it still update even though there's technically nothing to "overwrite" in the target? What if my time frame is only 1 day (as it is for the Electric accounts), is it only checking for updates across that one day? The intricacies of the automatic updates are unclear to me, so I'm not willing to risk it and not have my data be published.