Easily deploying and maintaining AF objects in multiple remote AF databases
Originally posted on PI Square at https://pisquare.osisoft.com/ideas/5538-easily-deploying-and-maintaining-af-objects-in-multiple-remote-af-databases
Today, one of the method that can be used to selectively reconcile object changes (including templates, analyses, UOM, etc.) between source and destination servers is using Excel (in order to keep track of object changes and/or object deletions). However, it is very time-consuming and prone to errors.
Inspired in the current project I am working on (containing a central AF server and remote AF servers), it would be great to have a tool that allowed us to deploy and maintain all types of AF objects in a selective way in multiple remote AF servers.
Here is a use case:
The user selects the desired source objects (templates, elements, analyses, enumerations, UOMs, etc.) and export (or "publish") them to several destinations, at once, after of the press of a button (instead of “live” sync). The user selects what to be exported or synchronized to each destination, and not necessarily the same group of objects would be exported/sync to all destinations: each destination may contain different set of elements and templates.The synchronization would support name changes, object additions and deletions (similar to what is available in PI System Connector today)The user should be warned if he tries to export an element that depends on an enumeration or UOM () that is not included in the object set being exported.The user should be optionally warned about the attributes, analyses, etc. that will be deleted from the destination databases, if the corresponding ones have been deleted from the source templates or elements.The user should be able reconcile tag name changes: if the tag name convention is changed in the source attribute set with auto-tag creation, the tags created at the destination should be renamed as per new substitution parameters string.User should be able to save/load different export scenarios.
() You may ask: why not export the entire UOM? The answer is because the source server may be a development server and may contain UOMs that were added for other type of industries, which would be irrelevant in some production environments.
echoing sentiments, a simple drag and drop works for the initial build, but subsequent updating is tedious.
We currently have no plans to provide GIT like versioning control with AF, but we are looking into providing ways to move AF databases and AF objects more easily between systems. As we continue to evaluate this, please continue to provide your input and use cases.
Thomas McCarthy commented
A pharma customer would like to manage their AF Templates globally and enable the distribution of these templates to their site systems. The site would probably want to have an opt in to these updates, so that they could evaluate the impact and install them when they are able to.
Laurie Dieffenbach commented
Here is some further information from a discussion with Stephen Kwan:
I believe the goal of my customers is to “publish” changes to standards across many sites rather than to have continuous synchronization. It seems likely to me that a “publish” operation would be well coordinated, but would make it much easier to deploy standard solutions across an enterprise. The following are questions from Steve and my responses:
• One way sync or two-way sync? Who is the master if there are conflicts in a two-way sync?
o For my customers, it would be one-way sync (or publication, if you will); the “corporate” source would be considered the master in these cases.
• In a one way sync, do we overwrite and delete differences at the destination? If so, what if there were analytics that are generating calculation results? Overwrite them?
o We could lead customers to allow updating of the base templates, while leaving any derived templates alone. That is a best practice we advocate for allowing site customization of standard enterprise templates.
[SK] I would have to disagree with you on this. I can see the use case of a fixed base template and making changes to derived templates. Derived templates have a heavy dependency on the base template, therefore a change in the base template potentially has much higher impact than derived templates. As an example, if the derived template uses Attribute1, but a subsequent update of the base template no longer has Attribute1, what should we do?
• It's likely that source and destination AF structure would have different Data Archives for their PI Point attributes. Would customer demand that this mapping would occur automatically?
o I think the main goal is publishing standard templates and UOMs. So, the tag references shouldn’t be as much of a consideration.
[SK] All templates have attributes and at least one attribute references a PI Point. That’s the reason behind my question.
• In the case of templates, once a PI Point attribute is "locked in" (via Create or Update DR), it would need to be "reset to template" when it's sync from A to B. This resetting would undo any manual changes or overrides. Is that OK?
o I’m not sure I understand the question. Can you “lock in” a tag reference in a template? Isn’t the purpose of a template to be generic?
[SK] Actually this is a behavior of a PI Point Data Reference. As an example, an attribute template may have a configuration of \\%Server%\%Element%.%Attribute%. The % delimits a substitution parameter. When the user instantiate this configuration to actually set it to a PI Point, the configuration is “locked”. You have to do this for AF to automatically create PI Points. When this attribute is pushed out to the site, one of three things may have happened. 1) the configuration was not locked, thus %Server% is dynamic and would be reflective of the server it’s connected to. 2) this was locked and thus no longer dynamic and when it’s pushed out the site, we need to know what to do with it. 3) at the source, this configuration string was overwritten at the instance level and thus “locked” in. If this is pushed out to site (instance, not template), we need to know what to do with it.
• What if there are conflicts with UOM database between source and destination? Who wins?
o In the case of one customer, I think the corporate UOM list would win. I don’t know that all customers care about this so much.
[SK] Ok, so there is danger here because UOM changes unbeknownst to the site is a bad thing. Changes to conversion factors, i.e. custom UOM, could have happened without end users knowing that it has.
• When syncing AFTables, can we assume the login credential to external databases will be the same? What if it's different? What should we do?
o I can’t answer this one. It hasn’t come up for my customers.
Steve Edwards commented
The highest priority I see is a Master library for an enterprise. The value is to propagate the latest analytics across sites faster. This means a one-way sync of library templates, tables, and enum sets from master server to slave servers. The master always wins. Attr/Tag mappings are preserved when the master template is edited. The UOM database can be sync'd across all AF servers, again master wins, one-way. Slaves could subscribe themselves or a master could subscribe a slave.
Steve Moore commented
From my perspective this is a vital function; one that our homegrown systems have always had, and that I'm rather surprised that OSI PI does not. As the OP states, manually managing standardization across AF servers is a bit of an oxymoron, as it is prone to errors.
Managing across a large corporation with lots of plants. This will be a great benefit to central administration to keep standard.
Steve Edwards commented
i have this need now in o&g where assets are decentralized but must be administered remotely from a central library. Primary focus for us is element and event frame templates.as these are enhanced centrally, all subscribing AF libraries underneath are synchronized. this seems to be a recurring enterprise feature we will see much more and more.
Robert Raesemann commented
The generalized ability to automate the replication of data from one AF database to another seems to be needed. While there are many ways to grow your own solution, there is no configurable way to do this.
- Replicating templates and template changes from a centralized server to site servers
- Replicating elements from a centralized server to site servers
Scott Robertson commented
In response to Fabiano Batista, "Interesting use case. It looks like yo..."
I'm thinking that a scheduled update would be best. For the continuous use case, as in ML, I would say "Daily at 3 AM". For the Template updates, I would set a day and time when the update would occur, which could be days or weeks in advance. The ideal model, from a security standpoint, is an outbound connection from the remote sites, that checks periodically for updates, downloads the updates, and schedules locally for execution.
Fabiano Batista commented
In response to Scott Robertson, "I have a very similar use case. Potent..."
Interesting use case.
It looks like your use case requires a continuous AF synchronization (instead of "on-demand" one, like the one I suggested). PI System Connector would be the best product for this, but unfortunately the current version of PI System Connector cannot push analysis objects to the destination servers (i.e., your remote servers).
Scott Robertson commented
I have a very similar use case. Potentially 1,000s of remote PI Servers will need to be kept in sync. A typical deployment pattern is to use Machine Learning to update analytics for real-time alerting. ML would update the central AF server, and the remote AF servers would check in periodically for their updates.
Hans Bleijendaal commented
Tool should also support DTAP-street development, making it easy to move AF-parts easily from dev->test->acc->prod servers. Automical synchronization (add/update/delete).
Santosh Koirala commented
Looking for a tool that would compare two databases (that originated from the same parent) then gives user option to keep or discard changes from each and eventually merge them to form a working production database.