As a process engineer I would like to be able to integrate complex specific calculations, such as from a 3rd party analytics engine, so that I can use the Asset Analytics and its context with all the available data and scheduling to run those without needing to do custom heavy development.
Reopening this suggestion to collect additional feedback.
Steve Edwards commented
This is needed today to protect the intellectual property of what is inside in the analytic. This would be useful for "community" partners (equipment suppliers, engr service providers, consultants, etc..) to be able to distribute their expertise to PI System owners in a PI System analytic template. Users need see only inputs, outputs. Their logic is proprietary. An example case would be vibration monitoring service providers triggering EFs at plant site for site operators to qualify the incident and interact with the community partner. Community PI AF authors would greatly enhance existing pi system investments and be a welcome boost to site activity, strengthening data leaving the site to cloud based sharing and research.
Rick Davin commented
Another user ( Dave Johnson ) has posted an interesting discussion on this topic at:
PI AF Extensible Analytics
Jørgen Foss commented
I find Dave Johnsen’ post excellent.
There are probably several ways to achieve “Extensible analytics”.
To give another customer’s view on this:
We use a Python Flask server hosting our Python code and a custom DR to call the functions. As Dave points out the CDR has some draw backs.
We can also call our Python Flask server using the REST possibility in Notifications but creating an event frame for each calculation is over kill.
One possible extension for Osisoft that would be more generic is to let the analysis service be able to send web requests with a setup much like the one in Notifications, but where the trigger is the same as a built-in analysis. Then it would be up to the hosting server to able to handle the web requests. The hosted functions could essentially do all that is needed i.e. retrieving data from PI and writing data back to PI. If a response to an attribute is required, then Osisoft should define the response format. (Mathworks or other calculation server providers would have to adapt the Osisoft format)
Dave Johnson commented
In my opinion, the lack of a robust calculation engine is the largest gap in OSIsoft's current product offering. There is a huge opportunity for OSIsoft to add significant value to what is already otherwise an excellent platform.
Please read the background information below to get up to speed on the incredible usefulness of this feature and then be sure to upvote this user voice idea!
The PI AF/MATLAB architecture (now deprecated) was elegantly designed and provided a glimpse into the possibilities of PI AF extensible analytics. While few companies expressed interest in MATLAB integration per se, many customers would petition for a more generic offering that would enable the integration of PI AF with a server hosting a catalog of functions using ASP.NET, Python Flask, etc. if they truly understood the value proposition.
Here are the benefits PI AF extensible analytics would provide:
People can leverage PI System Explorer, the standard OSIsoft tool for creating AF templates and elements, to:
- Invoke functions that reside in external DLLs and libraries beyond the built-in PI AF analytics functions.
- Overcome the inherent limitations of PI AF analytics which do not support programming loops and generally lacks the expressiveness to construct and maintain complex calculations.
- Use external functions residing in DLLs as first-class citizens just like the built-in AF functions. This brings full IntelliSense/autocomplete capabilities so people can create their calculations as AF analyses.
- Manage calculation code using robust software engineering tools including unit testing, source control, etc. (since the code resides in an external DLL)
This PI AF extensibility point would unlock other scenarios that could add value to companies with JSON/HTTP as the lingua franca. For example, a Python Flask server hosting various Python data science packages, R function hosting, etc.
Extensible analytics would provide a decoupled architecture resulting in these benefits:
- Externally hosted ASP.NET calculation server (or other technology platform) makes it less likely that the PI Analysis service will crash and burn if something goes awry in the calculations.
- Clearer separation of responsibilities for OSIsoft to pinpoint the problem as residing with customer code rather than custom data references, PI ACE, etc.
- Easier to maintain and update calculations through a pure functions DLL and a published catalog/manifest on an external calculation server.
- Local hosting of calculations for critical operations so facilities can function as islands rather than relying on a third-party calculation engine in the cloud.
- Happy, productive customers! People can accomplish their complex calculation objectives using one tool (PI System Explorer).
Why not just use a Custom Data Reference (CDR)?
- The CDR and PETools functions run in the same memory space as the PI AF analytics engine which increases the likelihood of crashing the AF analytics service if something goes awry with the calculations versus the decoupled extensible analytics design.
- If nested functions are needed (i.e. calling one function inside another function), using the CDR will be challenging. It will require some advanced knowledge for people building calculations to ensure the correct function execution order. This contrasts with the extensible analytics option which enables IntelliSense and nesting of functions in AF analyses just like built-in AF functions.
- Utilizing the functions via the CDR in PI AF templates will be more clunky for the end users who are building AF structures, leading to a suboptimal UX versus extensible analytics.
- When using PI System Explorer (PSE) to build PI AF templates/elements, PSE will download the CDR/engineering functions DLL file to the user’s local machine behind the scenes, presenting an opportunity to reverse engineer the DLL code. This is a concern if the functions in the DLLs contain "secret sauce" and particularly a concern if the DLLs are .NET-based which can be trivially decompiled.
- Calculations are more costly to build and maintain since one must build and deploy CDR .NET “glue” code on the PI AF server versus updating a catalog of functions on a decoupled server.
Richard Armstrong commented
AF Analytics has been a great step forward but personally I'd like to see the ability to have functions developed and added so they were available to all AF users.
An addition to the AF Library for function templates was my thought.
I've talked to customers that have needed to write external AF SDK Programs to solve problems that have not been overly complex. (eg requiring loops, arrays)
If we had a function editor/builder that allowed for these to be done in AF they would be context aware and able to be recalculated within the AF environment. Ideally this would move away from the PE syntax required in AF Analytics and be closer to C, C# or Python.
Without going into to great detail, I agree with Samuel Clark's comments around the ease of integration. We deployed the Matlab production server and were getting significant value out of the ability to write custom functions to complete the transfer of excel based monitoring to AF- everything from simple solutions like excel's SumProduct, iterative solvers or ML algorithms. These can all be done individually by sending data out with the Pi integrator but they cannot be deployed efficiently or easily by average users
My organization also uses ACE calculations to perform complex calculations (Utilization, Furnace Efficiency) and would welcome the introduction of a third party analytics engine as an add-on and means to expand the functionality and robustness of AF Analytics. Initial thought was AF Analytics was set to be the heir-apparent to ACE; unfortunately without the infusion of third party analytics into AF I don't see that happening.
Steve Edwards commented
Customers doing pattern recognition and prognostics often need multivariable, nonlinear functions to model the physical behavior they see. Extensible analytics keep projects rooted in PI AF and provide feedback to us as to what functions need to be included in our product.
Sasha Krivonosova commented
In many cases customers need to run complex calculations using python scripts, for example:
K-means clustering is used for tank leak detection to eliminate false positives;
Drilling engineers run models against (near)real time data using python code.
It would be beneficial to have a capability to trigger these calculations (external library) from Asset Analytics and write results to pi tags natively within the same tool, rather then use two different data egress and ingress technologies (ex. combination of Integrator and an Interface).
Samuel Clark commented
IMO the biggest advantages with the way Matlab Production Server (MPS) integration was implemented were the self-serve user experience of working in PI System Explorer and being able to leverage the existing PI Analysis Service scheduler. Further the use of function calls greatly simplifies the calculation logic as contextual info /metadata doesn't need to be handled in the calculation. For example the context of the calculation is the element whose analysis called the function. The output of the calculation is the output attribute. There is no need to recursively discover or hard code the paths of the inputs and outputs. There is no shortage of ways to extract PI data, run calculations, and write the results back to PI but most of them do not address the above considerations.
It looked like we had plans to support other generic extensibility points in a similar fashion to the MPS integration and I think it was a very elegant approach that would have provided a true successor to PI ACE.
In response to Rick Davin, "Hi Tim Carmichael Data References alwa..."
Thank you! Well explained and I agree with your reasoning.
In response to John Messinger, "We would like to be able to add custom a..."
Since I haven't had an opportunity to use MATLAB functions, how does the requested implementation differ from custom data references? And, please excuse my ignorance of the process... I am truly asking to understand.
Rick Davin commented
In response to Tim Carmichael, "Since I haven't had an opportunity to us..."
Hi Tim Carmichael
Data References always work on-demand, but an Analysis has the benefit of persisting the output to PI point for history and faster retrieval. The issue is there are some things that can't be done in Asset Analytics since it lacks a true looping mechanism or conditional branching. A data reference may access a 3rd party library but configured analytics cannot.
The reason for the proposed idea is to have a standard hooks from Analytics to a calculation service of the customer's choosing (not just MATLAB). In this use case, Analytics is regulated to be the scheduling service but it shells the actual calculation to an external service.
since the integration solution is still up and explained on the MATLAB web site (!), it would be interesting to learn more about why the integration was withdrawn from PI AF 2018 SP2.
We were only a couple of weeks away from buying the MATLAB Production Server...
We would like to be able to add custom analytics that can be exposed in the same way the Matlab functions were, effectively passing input data to an external calculation engine and then return results to AF Analytics. This would allow us to migrate many legacy ACE calculations to AF Analytics for a number of clients, where we are currently performing complex calculations that are effectively contextual to a specific AF Element. In quite a few of these cases, we have used ACE as a scheduling container, but the code is typically AF SDK in order to interact with the asset hierarchy.
Can this suggestion be re-opened? Now that the MATLAB Production Server is being pulled back.
Now that Matlab is off the table, any new directions for extensibility of Analytics?
I appreaciate the step back! While Matlab provides scaleability performance and capabilities on the high-end of the spectrum, there is still a demand on the lower end.
I do understand the complexities and pitfalls involved (remember Custom DR's), but there should / could be a midway, just like PI Vision seems to strike a midway with it's extensibility. E.g. leaving the data delivery to the custom function to the analysis service, but having the capability to use e.g. C# to implement custom functions within that limited frame.After all we don't want to go back to the days where we open, render and parse Excel files in an ACE module...
Diego Curras commented
Customer would like to have the following possibility:
During the installation of a new Analyses service kit pointing to a PI AF server where an analysis service is already configured, he would like to prevent the current PI analyses service from stopping during the installation so that no real-time data calculation is lost.
He suggested this because he would like to keep his current analysis service running without forcing it to stop despite this new installation.
Brent Bregenzer commented
I've worked with customers who have expressed the same desire. They want to do some basic pre-calculations on many of the incoming data streams across the AF database, e.g., quality checks like badval or range limits and/or some basic pre-processing like moving average with a time window defined in an attribute. It can get tedious having to copy paste all of these lines of expressions from one template to another for many attributes. It would be easier to build a predefined function of their own where they just need to add the attribute name and few parameters like time range.
There have been requests for both user defined functions that only use a set of the included Analysis functions (this might be a good first step) and custom functions written in some programmatic language similar what was done in ACE.