As a process engineer I would like to be able to integrate complex specific calculations, such as from a 3rd party analytics engine, so that I can use the Asset Analytics and its context with all the available data and scheduling to run those without needing to do custom heavy development.
Reopening this suggestion to collect additional feedback.
We want to be able to execute Python scripts from AF analyses.
I support this feature request. In 2015, I developed a PI System-MATLAB integration layer and wrote a generic analysis framework that could accommodate not only PI System but any other such asset framework that had a .NET code base.
The integration layer allowed the user to define an AF Analysis with AF Attributes as inputs and output. However, the actual calculation would be left out and done in MATLAB instead. I wrote a parser in MATLAB for the AF Analysis definition, which would then isolate the input AF Attributes, read their data with AFSDK calls over the MATLAB-NET API.
The framework allowed any number of analyses to be implemented in MATLAB as a library of analyses, using the MATLAB OOP semantics. The AF tree would store the name of the MATLAB analysis in an AF Attribute, which could be passed into the AF Analysis, parsed by the MATLAB framework and dispatched to the delegated analysis object in MATLAB. Results would be displayed in MATLAB figures.
The framework entry point was from the MATLAB side, since there was no facility to spawn the MATLAB framework from an AF Analysis. Ideally, one would want to spawn the MATLAB code from a custom user AF Analysis. It would save my many lines of MATLAB framework code, although it was fun creating the bridge pattern and object factories in MATLAB OOP semantics.
A similar thing may be done with Python.
Todd McQuiston commented
We are huge users of ACE, and we would love not to be anymore. Moving all of this to AF and out of MDB is a huge "pie in the sky" goal for us, but we use a great deal of custom code, making AF Analytics unable to take over at the moment.
Amit Vankudre commented
Is it also possible to develop the capability of writing array results to a pi point ? Currently, array functions are available only for reading data or carrying out any operations on it, but the outputs need to be written one by one. It would be nice to have a feature where an array is being pushed to pi point in a single go (or if its going to be a per event basis, then we at least need to have some kind of looping functions available as in any standard programming language)
Amit Vankudre commented
I agree with others on this forum. Its important to develop an advanced analytics engine (either integrate PI analysis with an external engine or develop the capability within PI Analysis itself). I am working on an advanced analytics project or which the customer wants to develop many complex calculations that involve using statistical methods to arrive at an output. Doing this is quite easy in Python or MATLAB, but if integrated with PI analysis can prove really useful. Currently, the solutions need to be developed using multiple engines and there is no integrated platform available
Steve Edwards commented
This is needed today to protect the intellectual property of what is inside in the analytic. This would be useful for "community" partners (equipment suppliers, engr service providers, consultants, etc..) to be able to distribute their expertise to PI System owners in a PI System analytic template. Users need see only inputs, outputs. Their logic is proprietary. An example case would be vibration monitoring service providers triggering EFs at plant site for site operators to qualify the incident and interact with the community partner. Community PI AF authors would greatly enhance existing pi system investments and be a welcome boost to site activity, strengthening data leaving the site to cloud based sharing and research.
Rick Davin commented
Another user ( Dave Johnson ) has posted an interesting discussion on this topic at:
PI AF Extensible Analytics
Jørgen Foss commented
I find Dave Johnsen’ post excellent.
There are probably several ways to achieve “Extensible analytics”.
To give another customer’s view on this:
We use a Python Flask server hosting our Python code and a custom DR to call the functions. As Dave points out the CDR has some draw backs.
We can also call our Python Flask server using the REST possibility in Notifications but creating an event frame for each calculation is over kill.
One possible extension for Osisoft that would be more generic is to let the analysis service be able to send web requests with a setup much like the one in Notifications, but where the trigger is the same as a built-in analysis. Then it would be up to the hosting server to able to handle the web requests. The hosted functions could essentially do all that is needed i.e. retrieving data from PI and writing data back to PI. If a response to an attribute is required, then Osisoft should define the response format. (Mathworks or other calculation server providers would have to adapt the Osisoft format)
Dave Johnson commented
In my opinion, the lack of a robust calculation engine is the largest gap in OSIsoft's current product offering. There is a huge opportunity for OSIsoft to add significant value to what is already otherwise an excellent platform.
Please read the background information below to get up to speed on the incredible usefulness of this feature and then be sure to upvote this user voice idea!
The PI AF/MATLAB architecture (now deprecated) was elegantly designed and provided a glimpse into the possibilities of PI AF extensible analytics. While few companies expressed interest in MATLAB integration per se, many customers would petition for a more generic offering that would enable the integration of PI AF with a server hosting a catalog of functions using ASP.NET, Python Flask, etc. if they truly understood the value proposition.
Here are the benefits PI AF extensible analytics would provide:
People can leverage PI System Explorer, the standard OSIsoft tool for creating AF templates and elements, to:
- Invoke functions that reside in external DLLs and libraries beyond the built-in PI AF analytics functions.
- Overcome the inherent limitations of PI AF analytics which do not support programming loops and generally lacks the expressiveness to construct and maintain complex calculations.
- Use external functions residing in DLLs as first-class citizens just like the built-in AF functions. This brings full IntelliSense/autocomplete capabilities so people can create their calculations as AF analyses.
- Manage calculation code using robust software engineering tools including unit testing, source control, etc. (since the code resides in an external DLL)
This PI AF extensibility point would unlock other scenarios that could add value to companies with JSON/HTTP as the lingua franca. For example, a Python Flask server hosting various Python data science packages, R function hosting, etc.
Extensible analytics would provide a decoupled architecture resulting in these benefits:
- Externally hosted ASP.NET calculation server (or other technology platform) makes it less likely that the PI Analysis service will crash and burn if something goes awry in the calculations.
- Clearer separation of responsibilities for OSIsoft to pinpoint the problem as residing with customer code rather than custom data references, PI ACE, etc.
- Easier to maintain and update calculations through a pure functions DLL and a published catalog/manifest on an external calculation server.
- Local hosting of calculations for critical operations so facilities can function as islands rather than relying on a third-party calculation engine in the cloud.
- Happy, productive customers! People can accomplish their complex calculation objectives using one tool (PI System Explorer).
Why not just use a Custom Data Reference (CDR)?
- The CDR and PETools functions run in the same memory space as the PI AF analytics engine which increases the likelihood of crashing the AF analytics service if something goes awry with the calculations versus the decoupled extensible analytics design.
- If nested functions are needed (i.e. calling one function inside another function), using the CDR will be challenging. It will require some advanced knowledge for people building calculations to ensure the correct function execution order. This contrasts with the extensible analytics option which enables IntelliSense and nesting of functions in AF analyses just like built-in AF functions.
- Utilizing the functions via the CDR in PI AF templates will be more clunky for the end users who are building AF structures, leading to a suboptimal UX versus extensible analytics.
- When using PI System Explorer (PSE) to build PI AF templates/elements, PSE will download the CDR/engineering functions DLL file to the user’s local machine behind the scenes, presenting an opportunity to reverse engineer the DLL code. This is a concern if the functions in the DLLs contain "secret sauce" and particularly a concern if the DLLs are .NET-based which can be trivially decompiled.
- Calculations are more costly to build and maintain since one must build and deploy CDR .NET “glue” code on the PI AF server versus updating a catalog of functions on a decoupled server.
Richard Armstrong commented
AF Analytics has been a great step forward but personally I'd like to see the ability to have functions developed and added so they were available to all AF users.
An addition to the AF Library for function templates was my thought.
I've talked to customers that have needed to write external AF SDK Programs to solve problems that have not been overly complex. (eg requiring loops, arrays)
If we had a function editor/builder that allowed for these to be done in AF they would be context aware and able to be recalculated within the AF environment. Ideally this would move away from the PE syntax required in AF Analytics and be closer to C, C# or Python.
Without going into to great detail, I agree with Samuel Clark's comments around the ease of integration. We deployed the Matlab production server and were getting significant value out of the ability to write custom functions to complete the transfer of excel based monitoring to AF- everything from simple solutions like excel's SumProduct, iterative solvers or ML algorithms. These can all be done individually by sending data out with the Pi integrator but they cannot be deployed efficiently or easily by average users
Steven Paine commented
My organization also uses ACE calculations to perform complex calculations (Utilization, Furnace Efficiency) and would welcome the introduction of a third party analytics engine as an add-on and means to expand the functionality and robustness of AF Analytics. Initial thought was AF Analytics was set to be the heir-apparent to ACE; unfortunately without the infusion of third party analytics into AF I don't see that happening.
Steve Edwards commented
Customers doing pattern recognition and prognostics often need multivariable, nonlinear functions to model the physical behavior they see. Extensible analytics keep projects rooted in PI AF and provide feedback to us as to what functions need to be included in our product.
Sasha Krivonosova commented
In many cases customers need to run complex calculations using python scripts, for example:
K-means clustering is used for tank leak detection to eliminate false positives;
Drilling engineers run models against (near)real time data using python code.
It would be beneficial to have a capability to trigger these calculations (external library) from Asset Analytics and write results to pi tags natively within the same tool, rather then use two different data egress and ingress technologies (ex. combination of Integrator and an Interface).
Samuel Clark commented
IMO the biggest advantages with the way Matlab Production Server (MPS) integration was implemented were the self-serve user experience of working in PI System Explorer and being able to leverage the existing PI Analysis Service scheduler. Further the use of function calls greatly simplifies the calculation logic as contextual info /metadata doesn't need to be handled in the calculation. For example the context of the calculation is the element whose analysis called the function. The output of the calculation is the output attribute. There is no need to recursively discover or hard code the paths of the inputs and outputs. There is no shortage of ways to extract PI data, run calculations, and write the results back to PI but most of them do not address the above considerations.
It looked like we had plans to support other generic extensibility points in a similar fashion to the MPS integration and I think it was a very elegant approach that would have provided a true successor to PI ACE.
Tim Carmichael commented
In response to Rick Davin, "Hi Tim Carmichael Data References alwa..."
Thank you! Well explained and I agree with your reasoning.
Tim Carmichael commented
In response to John Messinger, "We would like to be able to add custom a..."
Since I haven't had an opportunity to use MATLAB functions, how does the requested implementation differ from custom data references? And, please excuse my ignorance of the process... I am truly asking to understand.
Rick Davin commented
In response to Tim Carmichael, "Since I haven't had an opportunity to us..."
Hi Tim Carmichael
Data References always work on-demand, but an Analysis has the benefit of persisting the output to PI point for history and faster retrieval. The issue is there are some things that can't be done in Asset Analytics since it lacks a true looping mechanism or conditional branching. A data reference may access a 3rd party library but configured analytics cannot.
The reason for the proposed idea is to have a standard hooks from Analytics to a calculation service of the customer's choosing (not just MATLAB). In this use case, Analytics is regulated to be the scheduling service but it shells the actual calculation to an external service.
Tomas Stark commented
since the integration solution is still up and explained on the MATLAB web site (!), it would be interesting to learn more about why the integration was withdrawn from PI AF 2018 SP2.
We were only a couple of weeks away from buying the MATLAB Production Server...
We would like to be able to add custom analytics that can be exposed in the same way the Matlab functions were, effectively passing input data to an external calculation engine and then return results to AF Analytics. This would allow us to migrate many legacy ACE calculations to AF Analytics for a number of clients, where we are currently performing complex calculations that are effectively contextual to a specific AF Element. In quite a few of these cases, we have used ACE as a scheduling container, but the code is typically AF SDK in order to interact with the asset hierarchy.