Allow bi-temporal data archiving
PI DA Server should allow for more flexibility in terms of the format of archived data. A big advantage would be if PI could archive two timestamps per event. One timestamp will come from whatever the data source is (PI Interfaces, fiels devices, etc) - this is effectively what PI has now. In addition to that, PI Should be able to add a second timestamp recording when the event got into PI i.e. when either PI Snapshot or Archive Subsystem saw the event coming in their respective queue. Benefits and use cases:
- any PI Client could make sure it gets all new pi events since client last talked to PI. Keep in mind that PI can have lots of backfills going on at the same time as realtime data so it is impossible for a PI Client app to know that since last time it talked to PI, there was a backfill correcting some data for 3 months ago. I know there are consumer queues, but they are not working properly all the time, they can overflow, etc. The more robust way would be if PI knew when it actually got the events.
- PI admins would easily detect backfill operations and asses data latency and anomalous delays. Questions like : when do we get most data into our PI during the day, or when did these duplicate events made into PI, or what is the data latency from field sensors to the PI archive, would all easily be answered. Now it's virtually impossible.
Please upvote if you like.
Other use cases for bi-temporal data include
- delayed data entry (lab analysis of physical sample sent to PI hours after the sample was bottled)
- data correction (data entered in real time from DCS, but later corrected by engineer after recalibrating instrument)
To add to the initial ideas, I believe data analyses would also benefit from knowing when the events were archived. Better overall updates management as well, better analyses reprocessing triggers. For example, one would know which backfilled events caused a calculation to re-evaluate its result and have a better picture of the data overall.