As a member-owned electric cooperative serving 170,000 customers in Virginia, REC has long known that the kilowatt-hours it purchases are not equal to the kilowatt-hours it sells. Rather than commissioning a study that would identify losses at a single point in time, REC had the insight to build a software system that would enable REC to automate the required loss analytics, creating repeatability.

Both operational and customer systems serve as repositories for large volumes of data that can be difficult to access, let alone provide insights for meaningful analysis. The path forward is to combine this massive data volume, sometimes called Big Data, into a single integrated data platform. These platforms can be organized to discover useful patterns in the data that may not have previously been apparent. In addition, they can provide information on customer behavior and choices they can be expected to make in response to market conditions.

REC had the vision to build a cloud-native data platform to ingest, organize, manage and secure the data that was needed for loss analytics, knowing it would serve as the data foundation for future automated and self-service analytics needs.

The data platform needed an architecture that could be deployed incrementally and allow data to be ingested in its source format, then later be transformed. This model enabled early use-cases to be deployed while providing support for more complex future use-cases not yet envisioned.

In addition, the platform needed to support the following operational capabilities to enable long-term viability:

• Scalability — The platform must ingest and manage REC’s largest datasets. It must be able to grow to meet storage requirements, while performing analytics and other data management operations in a performant manner.
• Elasticity — The system must scale up to meet performance requirements when performing analytics or machine learning.
• Useability — The solution must be easily used by the REC analytics and engineering teams and support the existing visualization and data access tools used by REC.
• Self-service — The platform must enable the organization to access data using industry standard tools. It must be usable by REC employees with little existing data knowledge.
• Manageable — The REC IT and analytics team must be able to manage the infrastructure with minimal uplift or assistance from external vendors.
• Cost-effective — The solution should not require a significant initial capital investment and should support a pay-for-use model.
• Extensible — Additional sources and analytics must be available with minimal rework.
• Fast deployment — The platform must be deployed in a timely fashion to support immediate project needs.

REC engaged 1898 & Co. to assist with designing and building this platform.

REC’s organizational needs showed this was a natural fit for a cloud-based solution architecture — specifically the platform as a service (PaaS) model. PaaS enables the solution to use system-level building blocks — such as data storage engines (databases), messaging systems and analytics engines, all pre-deployed by a cloud provider. PaaS services are fast to deploy and managed by the cloud provider so there are no upgrades or maintenance activities. They also provide an existing security model and are integrated to work easily with our cloud tools. The use of PaaS was important for REC as it fit several of the key requirements related to quick deployment, ease of management and future scalability.

The solution is built on a data lakehouse architecture for managing operational data. A lakehouse incorporates the best features of a data lake — such as scalability and simplicity of data ingestion — with the strengths of a data warehouse, such as ease of access to structured data (SQL) and support for integrations with industry-leading business intelligence tools. The lakehouse model simplified the overall solution design and system management needs, reducing initial and ongoing cost. Data stored in the lakehouse is also accessible from industry-standard data science notebooks, such as Juypter, analytics engines and machine learning frameworks.

Building Data Infrastructure

Sustainability was a key objective for REC as part of this effort. This was accomplished by minimizing the need for active management of the system so that it could be managed by REC’s current team. The cloud services used were deployed using infrastructure as a code (IaaC) to provide repeatability and the ability to detect changes.

Each data source was modeled after a standard template to create consistency and simplify production support.

Workshops covering the technology and concepts selected have helped the REC team gain understanding of the platform. REC is now supporting and building new capabilities upon the initial deployment.

Putting It All Together

With the cloud infrastructure in place, the operational data being collected allowed scientists and engineers to use the data platform for analytics.

This required an investigation that involved writing algorithms to analyze bulk data from substation SCADA and meter data. This step was key to yielding insights into whether electricity was being delivered to unmetered loads or was being stolen. The algorithms used AMI data generated for each circuit along with power flow estimates based on data from the SCADA systems. These loading estimates were incorporated into the algorithm to further refine the portion of energy load that was not accounted for by metering data. These estimates were derived by other methods as well, including AMI to phase matching, AMI to transformer associations and other methods aimed at approximating the source SCADA levels.

A set of dashboards use the data platform to provide REC with a number of actionable insights into the different functions of load and losses indicative of its system. This results in the alignment of core systems for a precision approach to reducing losses in the future.

This is the first of many analytics use-cases that are anticipated to use the data platform. Featuring standard ingestion processes, data organization, integrated analytics engines for big data and machine-learning tools for model development and production execution, the platform is now in place to provide insight into future system losses while enabling future projects.

This enabling technology is already paying dividends and enabling new capabilities for REC. For example, the platform provides the data infrastructure for electric vehicle impact analysis. These analytics will enable REC to plan for the adoption of electric vehicles by its members by predicting where possible EV adoption might occur at future points in time. The data is expected to yield insights into what equipment might require upgrades as well as where the grid could best support deployments of charging infrastructure.

The industry has identified many applications for meter and substation operations data to support system reliability planning and forecasting, operational awareness and other functions. The REC data platform will enable these capabilities and those yet to be envisioned.