Virtual Power Plants for Grid Resilience: A Concise Overview of Research and Applications

Yijing Xie ,,, Yichen Zhang , Senior,, Wei-Jen Lee ,,,Zongli Lin ,,, and Yacov A.Shamash ,,

Abstract—The power grid is undergoing a transformation from synchronous generators (SGs) toward inverter-based resources(IBRs).The stochasticity, asynchronicity, and limited-inertia characteristics of IBRs bring about challenges to grid resilience.Virtual power plants (VPPs) are emerging technologies to improve the grid resilience and advance the transformation.By judiciously aggregating geographically distributed energy resources(DERs) as individual electrical entities, VPPs can provide capacity and ancillary services to grid operations and participate in electricity wholesale markets.This paper aims to provide a concise overview of the concept and development of VPPs and the latest progresses in VPP operation, with the focus on VPP scheduling and control.Based on this overview, we identify a few potential challenges in VPP operation and discuss the opportunities of integrating the multi-agent system (MAS)-based strategy into the VPP operation to enhance its scalability, performance and resilience.

I.INTRODUCTION

THE power grid, once dominated by traditional synchronous generators (SGs) such as nuclear, coal, gas and hydropower systems, is experiencing a shift toward inverterbased resources (IBRs) due to the increased penetration of renewable energy resources (RESs) such as wind and solar energy [1]-[3].IBRs powered by solar photovoltaic (PV),wind, fuel cell (FC) and battery storage resources use power electronic devices to convert direct current power to alternating current power to be transmitted on the bulk-power system.North American Electric Reliability Corporation (NERC) estimates that approximately 860 gigawatts (GW) of IBRs will be added to the grid over the next decade [4].More than 67 GW of IBRs are planned to be connected to Electric Reliability Council of Texas (ERCOT) by 2023 [5].Road maps of reaching 100% IBRs have been set up by system operators such as Hawaiian Electric and National Grid Electricity System Operator [6].

TABLE I DATABASES

Power systems with high IBR penetration levels face reliability and resilience challenges.With SGs being replaced by IBRs completely, power systems face significant challenges to maintain proper inertia.The low inertia may lead to a larger frequency deviation and a higher rate of change of frequency(RoCoF) caused by similar disturbances and supply/demand imbalances [7], which in turn can trigger undesired underfrequency load shedding or even jeopardize grid reliability.Power systems with high IBR penetration levels may also experience larger voltage dips following disturbances.Fluctuation of distributed PVs in a certain geographical area may result in more oscillations as the level of PV penetration increases [8].Extreme weather events, cybersecurity threats and volatility in energy markets at all levels put intense strain on the power grid [9].As observed by NERC, the current trends indicate the potential for more frequent and more serious long duration reliability disruptions, including the possibility of national consequence events, unless reliability and resilience are appropriately prioritized [10].

Virtual power plants (VPPs), aggregations of distributed energy resources (DERs), have been recognized as new and promising technologies to improve the grid reliability and resilience and advance the grid transformation [9], [11]-[13].A VPP that leverages DER flexibility can perform as reliably as conventional resources and contribute to resource adequacy at a similar scale [13].Many countries around the world such as the United States, Australia, Germany and China have initiated VPP pilot projects.The global VPP market was valued at $3.37 billion in 2022 and is expected to grow to over$12.27 billion in revenue by 2030 [14].

Research efforts have been made on VPP operation to facilitate grid transformation.Depending on the operation timescale and the functionality of the VPP, these efforts mainly fall into two categories: VPP scheduling and VPP control.VPP scheduling focuses on the steady-state operation and management for power market in the hourly and daily timescales,while VPP control focuses on the dynamic nature and transients in a faster timescale.

Despite recent progresses in research on VPP operation, significant challenges still exist.The uncertainty and fluctuation of RESs and the dynamic characteristics of VPP components make it difficult to identify accurate and appropriate models for VPP scheduling and control.The growing number of DERs and the increasing capacity of VPPs have rendered the centralized architecture inadequate in meeting the scalability requirements of VPPs.As the VPP market attracts more VPP aggregators, the need for cooperative and competitive performance-guaranteed VPP operation has arisen.Additionally, the cyber-physical nature of VPPs poses great challenges to the resilience of VPP operation.

The past few decades have witnessed the development of the multi-agent system (MAS)-based strategy in distributed optimization, control and learning of large-scale systems[15]-[21].MAS-based theory is envisioned to empower VPP operation with scalability, performance and resilience.In this paper, we will review the recent literature on VPP concept and development and the latest progresses in VPP scheduling and control.Based on this review, we identify a few potential challenges in VPP operation and discuss the opportunities of integrating the MAS-based strategy with VPP operation to enhance its scalability, performance and resilience.

The references we selected mainly come from the databases listed in Table I.The remainder of this paper is organized as follows.Section II introduces the concept and development of VPPs.Sections III and IV review the problems and solution methods in VPP scheduling and VPP control, respectively.In Section V, we identify a few challenges in VPP operation and propose a distributed framework for VPP operation using the MAS-based strategy.Opportunities for enhancing scalability,performance and resilience of VPP operation are discussed.Section VI concludes the paper.

II.VIRTUAL POWER PLANTS

A. Concept

A virtual power plant (VPP) is a virtual aggregation of heterogeneous DERs such as distributed generators (DGs),energy storage systems (ESSs), controllable loads (CTs), and electrical vehicles (EVs) that acts as a single entity to provide capacity and ancillary services to grid operations and participates in electricity wholesale markets.A VPP can be regarded as an aggregation of DER technologies.With a centralized control system, aggregators, utilities or grid operators can remotely and automatically adjust DERs to provide clean energy, reliability and grid services while maintaining customer comfort and productivity.Through a combination of software and hardware, VPPs not only open the grid to a whole new utility-scale behind-the-meter supply, but also coordinate geographically distributed and heterogeneous DERs into holistic demand-flexible resources [22].We summarize the evolution of the VPP concept and scope based on Reference [23] and the latest literature as follows.

● 1997: Virtual utility [24];

● 2003: DER combination participating in the energy market as a unique unit [25];

● 2007: Method for technical and commercial management[26];

● 2008: Characteristics of DGs and CTs to characterize the VPP [27];

● 2009: Ancillary services through the management of DERs [28];

● 2011: VPP control with EVs [29;]

● 2017: Coordination between VPPs and system operators[30];

● 2018: Management of a set of DERs [31];

● 2019: Dispatchable virtual unit offering ancillary services to the network [32];

● 2020: Management based on the virtual cloud concept[33];

● 2021: Dynamic virtual power plant (DVPP) concept [34];

● 2022: Multiple VPPs volt/var control [35];

● 2023: VPPs as hybrid dynamical systems [36].

B. Differences Between Microgrid and VPP

Simlar to VPP, a microgrid (MG) is another choice to aggregate, manage and deploy DERs, particularly during a grid outage [37]-[44].The U.S.Department of Energy defines an MG as a group of interconnected loads and DERs within clearly defined electrical boundaries that act as a single controllable entity with respect to the grid.An MG can operate in either the grid-connected mode or the islanded mode [23].An MG can be a component of a VPP [45].Several distinctions between VPPs and MGs are shown in Fig.1 and listed as follows.

Fig.1.The relationship between VPPs and MGs.

● VPPs are not limited by geography and a static set of resources.

● VPPs cannot operated in islanded mode.

● VPPs generally aim at enhancing the competitiveness of the resources that are integrated by maximizing its profit,while MGs focus on stable, reliable and resilient operation of the resources within MGs.

● An MG can be connected to the main grid through a single point of common coupling (PCC) but a VPP can use cloud communication to control DERs located at multiple PCCs[46].Therefore, the VPP could be used to aggregate DGs,ESSs, EVs, entire MGs, demand response units and even entire distribution stations across an interconnection [47].

● MGs typically participate in retail distributions but VPPs can participate in wholesale markets.

C. VPP Components

In general, a VPP contains distributed energy resources(DERs), information and communication technologies (ICTs)and a management center as shown in Fig.2.

Fig.2.VPP components.

1)DERs: According to NERC, a distributed energy resource(DER) is any resource on a distribution system that produces electricity [48].In a VPP, DERs can be any controllable or dispatchable power resource that can respond to orders from the management center, adjusting its operational status such as changing the active power output setpoint.Common DERs in a VPP include distributed generators (DGs) such as combined heat and power (CHP) units, fuel cells (FCs), microturbines(MTs), diesel generators, photovoltaics (PVs), wind turbines(WTs), energy storage systems (ESSs) such as batteries, flywheels and superconducting magnetic energy storage, controllable loads (CLs) such as electric vehicles (EVs), heating,ventilating and air-conditioning (HVAC) units, and microgrids (MGs).

2)Information and Communication Technologies(ICTs):Information Technology Industry Council defines ICTs as the communications networks that connect all parts of the grid including operations, service providers, customers, distribution, and transmission by facilitating communications between machines, between humans, and between humans and machines [49].In a VPP, ICTs mainly provide real-time monitoring and communication functionalities including data measurements at DERs, data communications technologies and communications networks that transmit measurements, data and control signals between devices/DERs and the management center.Reference [50] provides an overview of VPP communication architectures and protocols.Reference [51]proposes the integration of a VPP into grid services using IEC 61850 standard.Applications of 5G technology in VPP can be found in [52], [53].

3)Management Center: The management center serves as the central hub of a VPP for information processing and decision making.It integrates the request from the system operator and all the data collected at DERs via ICTs, performs data analysis and decision making, and sends optimized orders such as active power setpoints to DERs via ICTs to ensure that the VPP operates efficiently, economically and securely.

D. VPP Services

VPPs have the capability to incorporate DERs into electricity market operations, ancillary services market, and provide management and support services for the distribution and transmission grids [54].Ancillary services are services provided by generation, transmission and control equipments which are necessary to support the transmission of electric power from the generator to the consumer [55].These services help grid operators to maintain a reliable electricity system.Examples of VPP ancillary services are active power and frequency control and reactive power and voltage control [56].The system frequency depends on active power balance.VPP can fulfill the system operator’s request by providing an active power reserve or absorbing extra active power.The voltages of the system’s critical buses should stay within certain limits,which depend on reactive power balance.VPP can support voltage stability via providing or absorbing reactive power.

E. VPP Projects

Many countries around the world, including the United States, Australia, Germany and China, have initiated VPP projects.The global VPP market was valued at $3.37 billion in 2022 and is expected to grow to over $12.27 billion in revenue by 2030 [14].

1)United States: Wood Mackenzie, a global energy research firm, has identified 563 VPPs either operating or in development in the United States in 2023, with California having the highest concentration of VPPs in the country [57].

● California: Stem is one of the leading VPP operators in California and has 2.5 GW of contracted storage assets under management across 14 different VPPs.In 2021, Stem dispatched 86 megawatts (MW) of stored energy, which is enough to power 103000 Homes, during 5-hour flex alert in California [58].Pacific Gas & Electric is testing VPPs with Tesla and Sunrun to ease grid stress during hot summer evenings [59].AutoGrid deploys a utility-grade VPP of residential batteries in Southern California to support grid resilience in Southern California Edison’s service area by optimizing the operations of nearly 400 kilowatts (KW) of battery capacity [60].San Diego Gas & Electric has been launching a VPP pilot project since 2022 [61].

● New York: Swell is working with Public Service Enterprise Group Long Island and contractors to use battery management software to remotely discharge home batteries and provide relief to the electricity grid during times of stress [62].A $110 million VPP project facility for multifamily residential buildings was completed in New York and will be used for VPP projects [63].

● Texas: Texas regulators approved the Aggregate Distributed Energy Resource (ADER) Pilot Project.The pilot allows up to 80 MW of capacity from assets in homes and businesses to bid into wholesale markets, run by grid operator ERCOT.ADER consists of many individual sites that can inject or withdraw power from the grid in response to an ERCOT instruction.Tesla will control the individual Powerwall and use individual data such as home energy usage and data produced by Powerwall [64].

● Puerto Rico: Sunrun has been establishing Puerto Rico’s first VPP.It will aggregate the solar and battery storage systems of more than 7000 customers to form a 17 MW VPP.The company is aiming to dispatch the VPP in 2024 [65].

● Hawaii: Shifted Energy has equipped smart-water-heater control modules in over 3000 multifamily condos and apartment buildings on the islands of Oahu and Maui.This VPP provides the utility’s aggregated capacity of up to 2.5 MW,delivering rapid grid support [66].

2)Australia: Australia Energy Market Operator has demonstrated the capability of VPPs for contingency frequency ancillary services using 8 VPPs with a total capacity of 31 MW.It is anticipated to operate a VPP with 700 MW capacity to verify the potential of the storage systems for energy management and ancillary services [67].Simply Energy VPP implemented over 1200 batteries in homes in South Australia and managed up to 6.5 MW of residential energy storage to provide ancillary services [68].

3)Germany: Next Kraftwerke has establised a VPP combining the flexibility of energy producers and consumers.The Next Pool connects more than 2900 medium-scale and smallscale power-producing and power-consuming units with a capacity of 1.9 GW using a fully automated centralized control system [69].

4)China: China is accessing its VPP capacity through its first VPP management center in Shenzhen with a capacity of 870 MW [70].

III.VPP SCHEDULING

The aim of VPP scheduling is to coordinate DERs to provide a certain amount of power resource subject to their capacity constraints in a collective way, while optimizing objectives such as minimization of the generation, transmission, distribution and maintenance cost, maximization of the profit of VPP participation in energy market, maximization of power system performances and maximization of social welfare such as minimization of greenhouse gases.

VPP scheduling problems are mainly formulated as constrained optimization problems.Take the VPP scheduling problem that focuses on profit maximization in a day-ahead market as an example [71].In the day-ahead market, VPP makes decision one day ahead on the levels of power of DERs to maximize the profit of the VPP.The objective function can be defined in terms of the revenues and costs of DERs as follows:

TABLE II LITERATURE ON VPP SCHEDULING

Optimization-based methods and reinforcement learningbased methods are two categories of methods for solving VPP scheduling problems.Literature on VPP scheduling is summarized in Table II.

A. Optimization-Based Methods for VPP Scheduling

Optimization-based methods for VPP scheduling include linear programming (LP), nonlinear programming (NLP),mixed-integer linear programming (MILP), stochastic optimization (SO), robust optimization (RO), bilevel programming (BP) and heuristic algorithm (HA).

MILP is the mostly adopted method for VPP scheduling.MILP is an optimization problem with both integer decision variables and continuous decision variables.Two categories of VPP scheduling problems can be solved via MILP.One category involves integer decision variables in the original optimization problem.For example, binary variables may indicate a link status between two specified buses, charge/discharge status of an ESS, and on/off status of an MT [76]-[81].The other category involves reformulation of the original optimization problem into an MILP problem.Certain nonlinearities can be effectively managed by employing a combination of binary and real variables [75], [82], [95].For example, the cost of a DG is described by a quadratic function

whereai,biandciare coefficients of theith DG cost function[45], [96].This cost function can be linearized as a piecewise linear function [97].If valve-point effects of power generation units and a nonlinear power flow model are incorporated in VPP scheduling, the optimization problem (1) becomes nonconvex [98].Relaxation and linearization methods can be employed to convert the nonconvex problem into a convex one [35], [71], [92], [99].

To address the uncertainties such as intermittent renewable energy output and market price, robust optimization (RO) and stochastic optimization (SO) are two common methods.In RO, uncertainty is modeled as a bounded set.Then the scheduling problem is expressed as a minmax problem.The optimal scheduling is made under the worst-case realization.Reference [83] formulates the VPP scheduling problem as an RO, which is transformed into a computationally tractable MILP problem based on the piecewise linearization technique,affine decision rule and duality theory.It is, however, often observed that the results of RO is too conservative for practical operations.To address this issue, SO is proposed.In SO,uncertainties can be modeled as stochastic variables, the statistical features of which can be obtained based on historic data.As an example, wind speed is commonly represented using the Weibull distribution [100].In [84], uncertainties are represented by a finite collection of scenarios, constituting a discrete approximation of the probability distribution of uncertain variables.More recently, a combination of RO and SO, referred to as stochastic robust optimization (SRO), is used for the VPP scheduling problem via incentive demand response and solved by a tailored column-and-constraint generation algorithm [86].

A bilevel programming (BP) problem pertains to the case where an optimization problem incorporates another optimization problem as a constraint.VPP scheduling with participation in multiple markets can be formulated as a BP problem[71], [87], which can be reformulated and solved as a singlelevel mathematical programming problem.

Finding optimal solutions for some scheduling problems is challenging.Reference [89] points out that certain VPP dispatch problem is NP-complete.Heuristic algorithms (HA) to solve the VPP scheduling problems include teaching-learningbased-optimization [88], hill climber [89] and particle swarm optimization [90].

B. Reinforcement Learning-Based Methods for VPP Scheduling

Because of the model inaccuracy, uncertainties of RESs and the nonlinear characteristics of DGs, traditional optimizationbased methods for VPP scheduling have high complexity and may not be applicable to real-world situations.Reinforcement learning (RL) has been adopted in VPP scheduling to deal with these challenges.Comparative studies have shown that RL methods provide better performance and incur lower time costs [94], [101].RL is used for solving the markov decision process (MDP) problem without the knowledge of the environment model.The MDP problem is a traditional representation of sequential decision-making, where actions influence not only immediate rewards but also subsequent situations, or states, and through those future rewards.

By appropriate setting of state, action and reward, the VPP scheduling problem can be modeled as an MDP and solved via RL.Reference [91] proposes an RL framework to solve the demand response problem of a commercial building-based VPP, where the state is the current load commitment, action is the demand adjustment ratio, and the reward is the utility function, andQ-learning is adopted to solve the problem.Motivated by the fact that the traditional RL method cannot guarantee the safety of the action, [92] proposes safe RL by including safety constraints.As a result, the MDP is transformed as a constrained MDP and solved by constrained soft actor-critic algorithm.Reference [93] studies the incentivebased demand response to reduce the bias between the dayahead forecast and the actual output of a VPP.The system state includes consumer’s power load, load elasticity, realtime price and the deviation amount between RES actual output and day-ahead bidding.The action is the incentive rate and the reward is the net profit.To cope with large-scale datasets created by the continuous state, a deep neural network is adopted to parameterize the state value function and the stateaction value function, resulting in Deep RL [93], [94].Compared with the traditional heuristic methods, Deep RL with offline training incurs lower time costs [94].

TABLE III LITERATURE ON VPP CONTROL

IV.VPP CONTROL

Once a VPP is scheduled for power delivery, its control system is responsible for sending commands to DERs to produce the desired aggregated power.Nevertheless, existing literature lacks emphasis on the development and implementation of real-time control for VPP operations [47], [102].To ensure efficient and reliable VPP control, the transient behaviors of VPPs need to be investigated.Transients following faults,large setpoint adjustments, or fluctuations in loads can potentially push the system beyond operational thresholds, resulting in instability, triggering protective devices, or violation of equipment ratings.This scenario becomes more critical in power systems with high IBR levels [103].However, only few papers consider the dynamics and transients of DERs in VPP control.

Literature on VPP control can mainly be categorized into three aspects: power setpoint tracking, frequency control and voltage control.Literature on VPP control is summarized in Table III.

A. Power Setpoint Tracking

The goal of power setpoint tracking is to make sure the total power output of the VPPtracks the VPP power setpointP∗.Ideally, the output of each DERPi(t) should track its reference setpointP∗i.These setpoints are obtained via VPP scheduling.Sandia National Laboratory has been developing and validating a real-time platform for VPP optimization and control with the aim at providing ancillary services [47], [102].Reference [47] focuses on designing feedback controllers for VPPs that align with energy market requirements and reserve objectives.If there exists an error between the VPP active power setpointP∗and the actual power, each DER will receive a signal that is proportionally to the error and its contribution ratio, that is,

The new setpoint will be

whereKP,KIandKDare respectively the proportional, integral and derivative gains.Another power setpoint tracking method called setpoint modulation is proposed in [104].The power tracking error ofith DER (P∗i(t)-Pi(t)) is used for the new setpoint.Under the assumption that each DER is stable,the setpoint modulation approach ensures accurate tracking of the setpoint.Note that in the design of controller for power setpoint tracking [47], [102], [104], the dynamics of the VPP is not considered and the transients are not investigated.

B. Frequency Control

Frequency response of a power system pertains to the ability to balance the generation and the load to maintain system frequency within acceptable limits (around 60 Hertz in North America).Conventionally, frequency response is separated into four categories: inertial response, primary control, secondary control and tertiary control that operate over multiple time scales.Fig.3 illustrates the frequency response following a contingency event that causes a frequency deviation.Once the contingency occurs, the energy stored in rotating masses of all SGs is immediately extracted as the inertial response to reduce the declining in frequency.The lowest point of frequency is referred to as the frequency nadir.The governor response serves as the primary control that occurs within the first few seconds and aims at arresting the frequency decline by increasing the active power output.The automatic generation control serves as the secondary control that occurs within tens of seconds to minutes, restoring the frequency back to the nominal value by adjusting the operating set-point of the governors.The tertiary control involves the reserve deployment, which typically operates within minutes to hours and supports the system operator in optimizing resources and managing larger disturbances or fluctuations.

Fig.3.Frequency response following a contingency.

Reference [105] studies a VPP containing ESSs, WTs and PVs and the influence of ESS primary frequency control on the transient performance of power systems.For basic frequency control, the frequency deviation is adopted as the measurement of the feedback control of ESSs, WTs and PVs.For coordinated control of ESSs and other DERs, a signal proportional to the mismatch between the setpoint and the actual output of the VPP is integrated into the local control of ESS.Coordinated control achieves better dynamic responses.Reference [106] extends the coordinated control of [105] to all DERs.Reference [107] studies a VPP comprising EVs for the primary frequency response and demonstrates enhanced primary frequency response through the involvement of EVs.The frequency model of EV is described by a first-order lag transfer function and the setpoint of EV is adjusted based on the frequency deviation Δf.Reference [108] formulates the secondary frequency control problem as an optimal control problem and solves it by using model predictive control.In[109], the dynamics of DER is modeled by a first-order system with input delay.Motivated by the fact that exact models of DERs are hard to obtain, the regulation problem is formulated as an MDP and solved via Deep RL.In the MDP formulation, the state contains the power tracking error, the actual power output, the residual energy and the bounds of regulation command, the action corresponds to the adjustment of regulation command, and the reward reflects the tracking error and profit.

To address the decreasing in inertial response for low-inertia systems with high RoCoF, a fast-acting response to changing frequency, fast frequency response (FFR), is proposed to enhance the grid resilience.FFR pertains to the capability of certain resources to increase the energy supply at a much faster pace than the conventional mechanical-based primary control.References [110]-[113] focus on the DVPP design for FFR.Reference [34] describes the goal of DVPP as the coordination of individual devices at the transmission grid level to collectively meet the system operator’s criteria, while considering the individual limitations of the devices involved.DVPPs emphasizes the dynamic ancillary services, including fast frequency and fast voltage control.As a key enabling concept for DVPPs, heterogeneity of the DERs plays a crucial role.Heterogeneous DERs are able to provide aggregated response beyond the capabilities of any individual DER alone[114].Reference [110] introduces a model-matching method to design a variable-speed feedback controller for WTs such that WTs satisfy a desired frequency response.Reference[111] proposes a decentralized control scheme utilizing dynamic participation factors.The desired frequency response is transformed to a Bode diagram of a desired functionFFCR(s).The controller for theith DER is designed as

C. Voltage Control

Analogous to frequency control, voltage control includes primary, secondary and tertiary voltage control.Primary control refers to automatic voltage regulators of individual devices such as generators, synchronous condensers, static var compensators.These controls rely on local information and measurements to act within few seconds.Given that primary control only relies on local data, the role of secondary control is to enhance voltage stability at the transmission level by utilizing the reactive power capabilities of the main generators within the region.This is achieved by adjusting their terminal voltage setpoints over a span of minutes.Tertiary control commonly leverages an optimal power-flow program and acts either on a 15-minute basis or in response to triggering events[115].

References[112] and [113] incorporate fast voltage control with fast frequency control design of a dynamic VPP via a decentralized multivarible control method.A desired multiinput multi-output specification of the DVPP is defined by a target transfer matrix linking frequency and voltage to active and reactive power.This multi-input multi-output specification is then disaggregated within the DERs by adopting adaptive dynamic participation matrices.

V.CHALLENGES AND OPPORTUNITIES

A. Challenges

Most existing results on VPP operation are within a centralized framework, where the information of all DERs is sent to the central hub of a VPP for information processing and decision making.However, the centralized VPP operation framework faces significant challenges.

First, there is no scalable model for the dynamic response of a VPP.A VPP is composed of several heterogeneous DERs,each possessing its own capacity and dynamics.Traditionally,a VPP acts as the representative for DERs, providing the information of individual DERs such as costs and dynamics parameters to the system operator.The increasing number of DERs adds complexity to the integration and dispatch of DERs.From the perspective of the system operator, it is not viable to model each DER.Understanding the capabilities for frequency and voltage support, as well as transient response at the PCC, is necessary.Although the aggregation models for DERs within a VPP have attracted research interest in the energy market and system operation in steady-state mode[116]-[120], few results aim at finding the equivalent models of the dynamic responses of DERs [121]-[124].However, all these aggregated models [121]-[124] are formed in a centralized way which may not reflect the real-time change of local DERs and may be unable to meet the scalability need of a VPP.Second, as the VPP market attracts more aggregators,the need for cooperative and competitive performance-guaranteed VPP operation has arisen.Currently, no results are available on VPP operation with performance guarantees.Third,the cyber-physical nature of VPPs poses great challenges to the resilience of VPP operation.The communication network may not be reliable due to time delay, attacks and failures.Reference [105] shows that VPP control cannot tolerate a large time delay (＞ 100 ms) in low-bandwidth (＜ 5 Mbps) networks.Thus, research gaps exist in VPP opeation with scalability, performance and resilience guarantees.

B. MAS-Based Distributed VPP

A distributed framework for VPP operation using the multiagent system (MAS)-based strategy is promising to address those challenges, as shown in Fig.4.When contingency happens, the system operator will estimate the initial power imbalance and send it to the VPP center.DERs within a VPP are connected via physical links as well as cyber links.The physical network may differ from the cyber network.A traditional centralized VPP requires all DERs to send information such as voltage, frequency, and power to the control center.The control center then performs centralized computing and broadcasts signals to all DERs.However, the centralized VPP is inefficient for VPPs with hundreds of thousands of DERs.Instead, in a fully distributed VPP framework, only a subset of DERs exchange information with the VPP center and other DERs only exchange information with neighboring DERs.Such a distributed framework for VPP operation with scalability and extendibility is modeled as an MAS.

Fig.4.A distributed VPP operation framework.

An MAS is composed of multiple intelligent agents operating in an environment to address challenges that are beyond the individual agents’ capabilities.With the increase of the quantity and the complexity of components and the amount of data produced in an MAS, the distributed scheme is more desirable than the centralized scheme.In a distributed framework of MASs, agents only utilize the local information obtained through the underlying communication network to complete global tasks.Various distributed algorithms have been introduced for MASs to achieve diverse objectives such as consensus [15], flocking [125], coverage control [126], formation control [127], optimization [128], [129] and learning[130].

Distributed optimization and control methods have been studied and applied to various problems in power systems.Reference [131] provides an overview of the existing research on distributed algorithms applied to the optimization and control of power systems.Reference [132] discusses the artificial intelligent applications in microgrid energy management systems and introduces future work areas of applying artificial intelligent techniques to VPPs.Reference [133] discusses the online optimization methods in power systems.However, only a few results are available on distributed VPP operation, as summarized in Table IV.

For distributed VPP scheduling, the economic dispatch and voltage regulation problems in a direct-current distribution system are formulated as a multi-objective optimization problem [134].A distributed primal-dual subgradient method is proposed to solve this multi-objective optimization problem and determine the current injection.Each DER obtains the global information in a distributed way via forward and backward communication.A double-consensus based distributed optimization method is introduced in [135] to deal with the VPP scheduling problem.The global variables (the multiplier and the power mismatch) are determined locally and shared among their neighbors using consensus algorithms.Reference[136] studies distributed dispatch of DERs for frequency supports via active power sharing and voltage regulation.Because of the high resistance/reactance ratio of lines in distribution networks, changes in active power of DERs can also give rise to voltage concerns.Two optimization problems are solved,where one aims at minimizing supplying active power and the other aims at minimizing the overall voltage mismatch and the cost associated with the reactive power support.In [137], voltage control is achieved through the manipulation of reactive power supplied by distributed VPPs within a subtransmission network.Individual load buses independently monitor their bus voltages.Should a voltage violation identified, all VPPs are simultaneously activated to provide the required reactive power assistance.Suppose that each bus has a VPP and the reactive power change for each VPP is equal.Distributed algorithm is proposed to calculate each bus sensitivity.Then,an average consensus mechanism is employed to determine the reactive power adjustment required for each VPP, based on the calculated sensitivities.Reference [35] proposes a fully distributed control solution for active distribution networks consists of several VPPs with the aim of network loss minimization and voltage profile optimization.The fully distributed method relies on a consensus-based alternating direction method of multipliers, where global variables are updated locally via consensus.

TABLE IV LITERATURE ON DISTRIBUTED VPP OPERATION

For distributed VPP control, [138] proposed distributed control algorithms for DGs in a VPP based on the consensus mechnisim such that the total power outputs of DGs tracks the desired power while minimizing the cost.The dynamics of the DG is a first-order stable linear time-invariant system.Reference [139] designs a dynamic distributed clustering algorithm to cluster batteries into different VPPs based on their capacities and demands.Then, SoC balancing is applied within VPPs.Such a dynamical VPP is better than fixed VPPs in terms of power loss and battery life.A similar method is adopted in [140] to form heterogeneous VPPs based on power requirements.

The distributed clustering of batteries in [139] and [140]belongs to distributed VPP formation.The goal of VPP formation is to form a VPP considering its sitting and sizing (see,for example, [141]-[143] for centralized VPP formations).More specifically, distributed VPP formation involves determining the suitable size of a VPP and the optimal placement of DERs for operation in a distributed manner.

C. Opportunities

The study on distributed VPP operation is still in its infancy.Few results are available on distributed VPP control and distributed VPP formation.Theories of distributed optimal control, differential graphical game, resilient control, delay-mitigation control and multi-agent reinforcement learning(MARL) all offer opportunities to promote scalable, optimal and resilient VPP operation.

Distributed operations of VPPs rely on the consensus method [35], [135]-[140].However, consensus does not necessarily impose optimality.For a cooperative MAS where agents have a common interest, distributed optimal control theory is proposed to solve optimal decision-making problems [16].On the other hand, differential graphical game theory provides solutions for agents that have conflicts of interest among themselves, for example, when individual agents aim to optimize their own performance indices [17].Performance-guaranteed distributed operation of VPPs still need investigation.

Although the lack of central coordination brings about robustness against single-point failures, the high dependence on local coordination may render an MAS vulnerable to attacks and failures in the cyber and physical layers including Byzantine agents, denial-of-service, actuator faults, sensor noises and communication delays.The challenge lies in the design of resilient algorithms to identify attacks, suppress the impact and then prevent attacks.Mean-subsequence reducedtype algorithms have been developed to improve the resilience of MASs to withstand adversarial attacks [18].Delay-mitigation control methods such as low gain feedback [144] have been validated to be effective in dealing with MASs subject to time delay [145].A distributed Cauchy-kernel-based maximum correntropy filter is designed for state estimation of large-scale systems suffering from hybrid attack model composed of denial-of-service attacks and deception attacks [146].Distributed proportional-integral-observer-based control is proposed to achieve vehicle platooning in the presence of replay attacks [147].Distributed VPP operation with resilience assurance should be considered.

The growing complexity, heterogeneity, uncertainty and volatility of components in a VPP pose new challenges to model-based control and operation methods.RL-based mechanisms are promising to mitigate the shortcomings of modelbased methods [148].However, most RL-based methods are in a centralized scheme.The multi-agent reinforcement learning (MARL) framework provides a distributed scheme.MARL endows agents with the intelligence of learning optimal behaviors by interacting with an unknown and complex environment.MARL refers to both theoretical developments and solution methods.Research on MARL theory focuses on developing scalable, robust, efficient and safe algorithms with theoretical guarantees.On the other hand, the implementation of MARL focuses on formulating problems that can be solved using off-the-shelf MARL methods.Research on MARL for VPPs is still in an early stage because of two reasons.First,MARL theory has many unsolved problems and most existing results on MARL lack theoretical guarantees.Second,MARL has not been well-customized for VPP applications.The formulation of control and operation problems in VPPs in a MARL setting still needs investigation.

The design of MARL methods for VPPs should take resilience and efficiency into consideration.Reference [149]reveals the vulnerability of consensus-based MARL algorithms to adversary attacks.To mitigate the influence of adversary attacks, a resilient distributed Q-learning algorithm is proposed.Then, an event-triggered communication strategy is combined with resilient distributed Q-learning to guarantee the resilience and communication efficiency of MARL [21].However, research is needed to investigate efficient and resilient MARL algorithms for VPP operations.A well-developed MARL theory tailored for VPP operation has the potential to overcome the limitations of model-based methods in dealing with model inaccuracy and uncertainties, while ensuring scalability, performance, and resilience.

VI.CONCLUSIONS

VPPs are developing at a fast pace to meet the grid transformation requirements and to cope with climate change challenges.This paper reviews the concept, development and global markets of VPPs.With the increasing penetration of DERs and the advances in ICTs, the concept of VPPs has emerged to coordinate DERs in a more flexible and manageable way.

Global developments of VPPs, along with practical projects,drive research efforts towards achieving optimal, reliable and efficient VPP operation.These research efforts in VPP operation are categorized into VPP scheduling and VPP control.The former focuses on steady-state operation and management for power markets in hourly and daily timescales, while the latter focuses on the dynamic nature and transients in a faster timescale.

It is observed that the majority of efforts on VPP operation have fallen into a centralized framework.This paper identifies several challenges faced in the centralized VPP operation.In particular, there is a lack of scalable modeling for VPP operation, and the centralized framework lacks scalability,performance and resilience guarantees.

Prompted by this observation, we discusses the opportunities for enhancing the scalability, performance and resilience of VPP operation by integrating MAS-based strategies such as distributed optimal control, differential graphical game,resilient control, delay-mitigation control and MARL.