Survivability Evaluation of Gas, Water and Electricity Infrastructures

The infrastructures used in cities to supply power, water and gas are consistently becoming more automated. As society depends critically on these cyber-physical infrastructures, their survivability assessment deserves more attention. In this overview, we first touch upon a taxonomy on survivability of cyber-physical infrastructures, before we focus on three classes of infrastructures (gas, water and electricity) and discuss recent modelling and evaluation approaches and challenges.

Introduction
More and more aspects of our daily life depend heavily on large-scale infrastructural systems, think of rail and road networks, but also about telecommunication networks (internet, wired and wireless telephony). Many governments have recently issued reports on the importance (and vulnerability) of their so-called critical infrastructures, e.g., [1,2,3]; an overview can be found in [7]. Over the last few years, the infrastructure systems and networks that provide gas, water and electricity have become much more "ICT-based", implying that their well-operation is becoming dependent on the correct operation of the supporting ICT. And although the embedded ICT does provide more functionality, it is also often a source of failures, or the victim of attacks. Nevertheless, it is essential for all these critical infrastructural systems to survive catastrophic events. In this paper we address approaches towards so-called "survivability evaluation" of infrastructural systems; our focus thereby lies on water, gas and electricity infrastructures, infrastructures that used to be run by municipalities, but now are mostly run by large internationally operating companies.
We note here that the concept of survivability is not restricted to just this class of infrastructural systems. It is also known for military devices, for example, aircraft combat survivability, and even in agriculture [39]. The literature is abundant with different definitions of survivability. For an overview see for example [33,41]. Distinct definitions stress different aspects of survivability, be it the detection of faults, the defence against attacks or the recovery from various types of disasters. We will focus on the behaviour of a system after a disaster has occurred. Note that we do not introduce a new definition of survivability but state a slightly generalised version of the one in [19]; it reflects an intuitively appealing view on survivability of systems but is therefore also quite informal: Survivability is the ability of a system to recover predefined service levels in a timely manner after the occurrence of disasters.
A disaster might be any kind of severe disturbance of the infrastructural system, for example, a power breakdown, a complete or partial cut of communication lines, a flood, heavy rain or a thunderstorm. The possible causes are manifold and include purposeful attacks as well as natural disasters like earthquakes or thunderstorms.
A system is survivable if it includes mechanisms to return to normal service within an acceptable time even though a disaster occurred. What kind of mechanisms are used and how they are implemented is not part of the survivability definition. One possible mechanism to achieve survivability is fault tolerance or any other form of redundancy [51].
The above definition of survivability does not give at all a precise recipe how to decide whether a system is survivable or not. To overcome this, many approaches have been followed in the literature for the quantitative determination of survivability [38,40,41,45,63]. Most of them are model-based and suggest some measure on the system (model) behaviour and study its evolution after the occurrence of a disaster. It, thus, is the deliberate decision of the person performing the survivability evaluation to choose an appropriate measure.
Note that the definition of survivability in essence addresses the evolution of the system of interest after the occurrence of a disaster. This implies that the process leading to a disaster, does not have to be included in the evaluation of the survivability. This is actually very favourable, as the exact occurrence process and probabilities are mostly very difficult to establish. In this context, we speak of so-called GOOD models for survivability, for Given the Occurrence Of Disasters [63]. In contrast, models in which the disaster occurrence is explicitly modelled, are called ROOD models, for Random Occurrence Of Disasters.
What is typical for the approaches 11 presented in this paper, is that the application field requires some form of hybrid model, taking into account discrete state components (e.g., for the up/down state of various components, or their mode of operation), continuous state components (e.g., for the physical issues playing a role), in combination with both deterministic (e.g., fixed time-outs or deterministic system evolution) and stochastic behaviour (e.g., for restoration or repair actives with random length). This combination makes analytical approaches very challenging, however, there is a clear need for these, as purely simulation-based approaches are very costly, sometimes even overly costly, to use in practice. In the remaining part of the paper, we give a brief introduction into recent approaches on survivability evaluation of infrastructures for smart water, gas and electricity networks.
The three approaches have quite a lot in common, however, also have remarkable differences. Two of them (that for gas and electricity) are based on a form of behavioural decomposition [56] in which the failure (or disaster) handling process is modelled separately from the performance of the system, through a combination of a stochastic process describing the failure handling mechanism and steady-state performance measures of interest, much the same as done in performability evaluation using Markov-reward models [28]. In contrast, in the approach taken for the water system is truly hybrid, in that the failure handling process and the water transportation and storage are combined in a single integrated model.

Water
Water infrastructures include the production and distribution of drinking water, as well as the collection and cleaning of sewage water. The main goal of drink water companies is the reliable supply of high quality water, whereas sewage facilities have to guarantee that a predefined maximum amount of water can be taken from the community sewage system and be cleaned and released with acceptable quality. SCADA (Supervisory Control and Data Analysis) systems are used to remotely manage treatment and distribution facilities in all phases of operation [52]. They are used for real-time monitoring and control of the substance (water) quality, optimising pumps, maintaining reservoir levels, managing distribution systems pressures, detection of leakages and to ensure the security of facilities. Improperly managed water networks can result in increased cost and insufficient supply of drinking water. Currently, water cleaning facilities are migrating towards unmanned operation, as human operation can not be guaranteed due to labor laws. The trend towards unmanned operation requires even more dependable and survivable systems.

Modelling approaches
To make the above more concrete, we now focus wastewater-management systems. Such systems clean water in several chemical and physical cleaning steps, before it is released. A suitable modelling formalism for such systems needs to take into account continuous (water tanks, pumps, etc.) and discrete (the setting of valves, the state of the SCADA system, etc.) quantities, as well as random events (failure occurrences, repair times, etc.). So-called stochastic hybrid models (SHMs) combine discrete and continuous variables with stochastics, hence, allow to model water treatment facilities in a natural way. However, the water treatment plants we want to consider are by far too large for state-of-the-art approaches that feature general SHMs. Several formalisms supporting SHMs have been defined [18,27,31], where each of them is suitable only in some very specific domain, and suffers from limitations that prevent it from being used in other applications.
Water management systems are characterised by deterministic fluid transportation, however, with rates that change according to a stochastic process. Hence, Fluid Stochastic Petri Nets (FSPNs) [27,31] and Piece-wise Deterministic Markov Processes (PDMPs) [18] appear to be suitable. However, the memory of continuous variables in PDMPs is lost upon stochastic transitions. Hence, they are not suitable to model the physical behaviour of fluid critical infrastructures. First and second order Fluid stochastic Petri nets (FSPNs), cf. [27,61], have a sound mathematical basis allowing for a completely formalised characterisation of the state-evolution in terms of differential equations. However, such equations can be solved only when there are at most one or two continuous variables. Simulation is the only available alternative when considering larger models [14,26].

Hybrid Petri-nets with General one-shot Transitions
To tackle the issue of scalability, a new approach based on Hybrid Petri nets [17] has recently been proposed, where the deterministic evolution is separated from the stochastic evolution of the system [25], by exploiting the quasi-deterministic behaviour of the system under study, given that failure and repair events are stochastic. Therefore, there are relatively few stochastic transitions, which allows for separating the deterministic from the stochastic evolution of the system, using a conditioningdeconditioning argument. This will speed up the reachability analysis and will allow for a large number of continuous variables in the model, as opposed to previous approaches.
The Hybrid Petri Net formalism with General one-shot transitions (HPNG) as proposed in [25] is specifically tailored towards fluid critical infrastructures. It allows for an arbitrary number of continuous variables ("tanks") that can be connected via fluid transitions ("pumps"). These transitions can be controlled by discrete places that can be connected via deterministic and generally distributed transitions; these can be used to model the ICT part of the system. Generally distributed transitions must respect the constraint that they can fire only once during the evolution of the model: for this reason we call them one-shot transitions. They can be used well to model one-time disasters, or repairs.
[24] introduces a new and efficient algorithm that maps the underlying statespace onto a plane for all possible firing times of the general transition s and for all possible system times t. The key idea of the proposed method is that instead of dealing with infinitely many points in the so-called t − s plane, we can partition the state space into several regions, such that all points inside one region are associated with the same system state. To compute the probability to be in a specific system state at time τ , it suffices to find all regions intersecting the line t = τ and decondition the firing time over the intersections.
To compute more complex measures of interest over time, the so-called Stochastic Time Logic (STL) has been proposed in [23], together with efficient model checking procedures that recursively traverse the underlying state space of the hybrid Petri net model. STL allows to formulate intricate state-based and time-boundeduntil-based properties; the notion of survivability can easily be expressed using the until operator. Even though the current analysis approach is limited to a single general one-shot transition, it has been shown in [22] that one can effectively model and analyse a real sewage treatment facility, as will be shown next.

Case study
Waste water treatment facilities clean sewage water from households and industry in several cleaning steps. Such facilities are dimensioned to accommodate a maximum intake. However, in the case of very bad weather conditions or failures of system components the system might not suffice to accommodate all waste water. We show the model of a real waste water treatment facility, situated in the city of Enschede, the Netherlands, as HPNG and analyse under which circumstances the existing  Figure 1 models the various stages of the sewage treatment process in a simplified manner. We are mainly interested in the capacity of each phase and the average amount of time the waste water stays in the different phases. We, however, do not aim at modelling the physical, chemical and biological processes in detail. Then, for a given failure of the system (at a certain time), we analyse the survivability of the system for changing weather conditions. Fixing the failure to a specific time of the day results in a so-called "Given the Occurrence Of Disaster" (GOOD) model, allows us to model the repair of the system with the single general one-shot transition. Since the evaluation method at hand is so quick, it is easily possible to parametrize the failure time and hence analyse the system thoroughly.
The capacity of the community sewerage system is modelled by an overflow place denoted P c (the leftmost "box" in Figure 1), of which the input rate may differ, depending on the weather conditions. From this tank the water is pumped into the treatment facility with a maximum rate 12 (transition T z ); in case the input exceeds the capacity of the place and the intake of the treatment facility, waste water flows into place P o which models the amount of water in the streets. The primary stage of the sewage treatment consists of two phases, namely the sand interceptor and the primary sedimentation tank. The sand interceptor is responsible for filtering solids like sand from the water. After that, the sewage flows into a large tank, which is used to settle the sludge, while the lighter material rises to the surface and is removed, and the remaining water overflows. In the model the sand interceptor is abstracted by the pump T z , and the primary sedimentation tank is modelled by the overflow place P ps .
A sedimentation tank physically separates suspended solids from water using gravity [5]. While the dirt settles at the ground, cleaned water is forwarded to the second cleaning stage. This stage consists of several phases for removing chemical and biological contaminations, modelled by a sequence of continuous transitions and places, before a secondary sedimentation tank separates the biological material from the now environment friendly sewage water, that can safely be disposed to surface water. The second sedimentation tank is modelled by overflow place P ss . The sludge that settles at the primary and secondary sedimentation tank is accumulated and forwarded to the sludge treatment stage. There it is thickened to reduce its volume for easier off-site transport. The sludge from the primary tank is pumped out and forwarded to the fresh sludge thickener. This is also modelled by an overflow place, denoted P ft . Sludge is pumped out of the place with a small rate and discharged to the digestion tank which is considered a very large tank. The overflow is directed to the filtrate basement. The same procedure is repeated for the accumulated sludge in the second sedimentation tank.
We now consider a failure in the sand interceptor, T z , modelled by the deterministic transition T f , firing at deterministic time α (which again could be parametrized for any value). After the occurrence of a failure, a repair crew will repair the pump, which takes, on average 2 hours (but that actually follows an exponential distribution). For this case we now investigate the following survivability property Φ (expressed in the logic STL): where, m Pr = 1, means that the sand interceptor pump is repaired. This equation expresses the probability that the overflow tank P o will have a very low level, that is, there is no overflow, during the 30 hours following the failure, or until it is repaired, whichever comes first. Here, we have chosen the time bound [α, α + 30] for the Until operator, since the pump is supposed to be repaired within 30 hours after its failure. The above formula (1) is typical for a wide variety of survivability measures of interest. The first term, before the Until operator is called the safety condition, whereas the one after the Until operator is called therecovery condition.
For this scenario, we consider two parameters, the time of failure and the intake rate. The result is shown in Figure 2. On the x-axis the overall intake rate (leftmost transition) is varied from 6 to 13, and the y-axis represents different failure occurrence times, from α = 30 minutes (0.5 hours) to α = 5 hours (after model start). As expected, for larger intake rates, the probability for Φ to hold decreases. However, it is interesting that for a late occurrence of the failure, the probability is lower, especially for high intake rates. The reason for this is that the effective capacity of the system is equal to the sum of the cleaning street rate (rate 4), and the fresh sludge thickener pump rate (rate 1.25), that is, 5.25 in total. Initially, the buffer P c is fully filled (capacity 20). Therefore, the buffer is filling up for intake rates greater than 5.25, hence, a late failure will cause a quicker violation of the safety condition. For intake rates smaller than 5.25, the buffer P c is actually emptied. On the other hand, when failures take place early, we have a non-zero survivability probability, even for high intake rates.

Conclusions
The case study clearly shows the strength of HPNGs in both modelling capabilities and efficiency of computations, for the application field of critical water infrastructures, even with the current restriction to a single general one-shot transition. Using the underlying stochastic time diagram and the new algorithms for model checking the logic STL, it is possible to analyse the survivability of the system very quickly, so that full parametric studies can easily be performed.

Gas
Long-range gas transportation is performed through transmission networks (publiclyowned in some countries), which operate at high-pressure and usually feature redundancy and storage capacity (pipeline, underground, liquefied natural gas) to make shortage a very unlikely event. Gas delivery to customers is mostly achieved by distribution networks (owned by municipalities or private investors), which operate at lower pressure due to safety issues and leakage control.
In the past, gas transmission, distribution, and retail were usually performed by a single "vertical" company. Nowadays, the liberalised regulation has produced a number of independent companies which manage customer service but have no role in network operation. Hence, the new role of a gas network operator includes the calculation and publication of technical and available capacity, the allocation of capacity rights, and the contractual and physical congestion management.
Survivability evaluation has been less investigated for gas networks compared to electric and telecommunication network systems. However, recently, the subject is receiving increased attention due to competitive challenges raised by demandresponse control applications, smart monitoring and actuation devices, novel industrial organisation of utilities, and an emphasis on homeland security and serviceability [4].

Modelling approaches
Most of the literature on the analysis of gas networks focuses on the fluid-dynamics perspective, mainly oriented to assess flow rates and pressures across network elements [30,16,49]. Optimisation of operations has been addressed in various ways, notably to favour efficient integration within multi-carrier systems combining provisioning of electric and gas power [48,37,34,43]. Stochastic modelling has been applied in [9] to consider different rates of leakage that may occur in a pipe fault and thus predict the impact on pressures and flow rates, supporting the planning of appropriate actions to mitigate risks. In [54], fluid-dynamic analysis of a section of a real gas network is repeated for different configurations of demand, thus reflecting the statistics of usage at different hours of the day and in different seasons. The effects of sequential restoration and constrained network capacity are considered in [29] to support reliability assessment by deriving average measures of interruption rate and outage time experienced by end-users, exemplifying the approach on a small-sized gas network.

An approach for derivation of transient survivability metrics
The recently proposed approach in [11] addresses quantitative evaluation of the transient behaviour of a gas distribution networks after the failure of a network element, i.e., again addressing a so-called GOOD model. Notice that the HPNG model cannot directly be used in the context of gas networks; the HPNG model does only address the fluid volume, and (piecewise) constant pump rates; for gas networks, next to the volume also the pressure should betaken into account. Based on pressure, also the pumps speeds can change. Temperature is taken to be constant. Overall, a more advanced modelling and analysis approach is needed for modelling gas distribution networks.
As a relevant assumption, changes of the operating conditions of the network due to daily and seasonal demand variations or demand-response mechanisms are considered independent of the actions taken in reaction to a component failure. This permits a decoupling of the fluid dynamic behaviour of gas from the stochastic temporal behaviour of recovery actions, yielding two distinct models, while allowing their separate analyses to provide feedback to each other ("co-modelling"). On the one hand, the fluid dynamics model follows a relatively conventional graphtheoretical representation of the network topology, supporting well-known techniques for the evaluation of pressures and flow rates under a given configuration of components and parameters. On the other hand, the failure management model provides a representation of the different functional behaviours that may occur when a network component fails.
The fluid dynamic analysis consists of solving a system of non-linear equations that describes the gas behaviour across the network in terms of pressures at nodes and mass flow rates in pipes. This is performed through an iterative procedure based on the Newton-Raphson method, cf. [15]. Setting up and solving such system of equations has a complexity of O(N +M ) and O(N 3 ), respectively, where N is the number of nodes and M is the number of pipes. The number of sectioning valves does not affect fluid dynamic analysis. The failure management and recovery process model is defined as a so-called stochastic Time Petri Net (sTPN) [57] extended with enabling and flush functions [55] (see Figure 3, lower part), which augment the modelling convenience without changing the model expressivity nor disrupting the subsequent analysis. As shown in Figure 3 (upper part), the model structure can be visualised using the UML activity diagram of recovery actions and turns out to be independent of the network topology, which makes the model general and almost guarantees a constant level of complexity of stochastic analysis. In contrast, stochastic distributions associated with temporal parameters of the model depend on the specific network under analysis, notably on the failure localisation and the consequent pressure regulation within the network.
The failure management and recovery process model describes the successive steps to be taken to recover from a disaster (or failure), in Figure 3 (lower part) visualised as an sTPN. This model may include concurrently enabled transitions with non-exponential distributions (possibly with bounded support), which goes well beyond the limits of the so-called enabling restriction and motivates the use of the solution technique proposed in [32] to perform transient stochastic analysis. The analysis method yields the transient probability of each logical state of the model, which actually corresponds to a specific operating condition of the network. The complexity of the analysis largely depends on the number of concurrent timers and on the length of paths between subsequent regenerations. As the structure of the survivability model is independent of the network topology, also the complexity of stochastic analysis turns out to be independent of the network topology. Subsequently, these probabilities are aggregated on the basis of the results of fluid dynamic analysis, which is in fact repeated under different boundary conditions to assess the service level experienced by each load node in each operating condition of the network, i.e., after each step of the failure management and recovery process that either changes the network topology or the pressure at the supply node. Hence, for each tangible state in the failure management and recovery process, a fluid dynamic analysis has to be performed; the results of these are combined with the transient state probabilities, in much the same way as done for performability evaluation using Markov-reward models [28]. This finally allows us to derive transient and average availability measures for end-users.

Case study
To provide a proof of concept of the overall methodology, the approach has been applied on a small-sized network taken from the literature and shown in Figure 4(a) [29]. The network is made of a supply node, four load nodes (marked as A through D) and nine pipes (numbered from 4 to 12). The gas is provided by the supply node, while the sectioning valve belonging to pipe 9 is kept closed in ordinary operating conditions. According to this, the gas is supplied radially, so that load nodes A and B are served by pipes 4 and 6, and 4, 7 and 8, respectively (the "upper branch"), while load nodes C and D are served by pipes 5, 11 and 12, and 5 and 10, respectively (the "lower branch"). Without loss of generality, we now focus on failures of pipe 5, as failures of pipes belonging to the so-called network ring leave more load nodes not served than failures of radial pipes. Table 1 illustrates the results of fluid dynamic analysis, whereas Figures 4(b,c,d) show the transient metrics derived for each end-user. Although not reported here, the stochastic analysis also supports the derivation of the average outage time experienced by load nodes after a pipe failure. If failure statistics are known, such average availability measures could also be derived over a longer period of time.  Table 1 Service level of each load node after each step of the failure management procedure, that changes either the network topology or the pressure at the supply node, after a failure of pipe 5.

Conclusions
The approach of [11] supports modelling and evaluation of the transient behaviour of a gas distribution network after a pipe failure. As a salient feature, the approach allows the use of non-Markovian transitions that overcome the limits of previous modelling approaches. This section only shows a small example, however, larger cases have been addressed in [32,12]. Future work includes relaxing the assumption that recovery actions do not overlap with variations of the network operating conditions.

Smart Grid
The Smart Grid vision for the generation and distribution of electric power is sustained by favourable tradeoff between the ratio of the increasing power generation, transmission and distribution costs to the decreasing costs of deploying computer and communication technologies. Therefore, utilities are embarking onto capital investments that deploy computer and communication technologies to the power grid with the objective of increasing the overall reliability of power systems. One objective of these investments is fortification of the grid against inevitable disruptions caused by weather, as, for instance, exemplified by the very large storm (Sandy) that recently hit the northeast coast of the United States. Regardless of the source of disruptions, which could also be the result of attacks, the goal is to mitigate the effects of failure and prevent cascading blackouts. Smart grids aim to deploy proven ICT and internet services to the power system. For example, one approach to power reliability improvement is to dynamically route  power. This is equivalent to dynamic routing protocols in the internet that detect failed links and automatically re-route over them. This is referred to in smart grid terminology as failure detection, isolation and restoration (FDIR), whereby faulty sections of a feeder line are located and isolated, and power is restored to sections outside the faulty region. Dynamic routing protocols in the Internet are complemented by dynamic flow control algorithms. In power systems, flow control is referred to as demand/response. The demand/response feature in power systems are activated to manage transient variations in the supply-demand power balance or as a failure recovery mechanism.
Therefore, the opportunity for improvement of the reliability of power systems by the deployment of computer and communication technologies has been a topic of interest to system dependability researchers, see, e.g., [10,20,60,47,21,42]. The survivability assessment of power grids was first performed in [8,35,46].

Modelling approaches
Most existing approaches focus on steady-state analysis of power distribution systems. For example, Brown et al. [10] use an hierarchical Markovian model to derive classical metrics such as the system average interruption duration index (SAIDI). Pievatolo et al. [50] presents a model where the components fail according to independent semi-Markov processes and the restoration times can follow nonexponential distributions. Using the model, the authors obtain the steady-state outage duration distribution. Elmakias [20] reviews available applications of Markov models in power system reliability assessment, focused on steady-state metrics.
Several studies [60,64,58,62] study the impact of adding Distributed Generation (DG) as a backup source in a power system on reliability metrics such as SAIDI. Martins and Borges [44] present a model for active distribution systems expansion planning and assess expansion alternatives using steady state metrics such as SAIDI. Chopade and Bikdash use graph-theoretic techniques to carry out structural and functional vulnerability analysis for a smart grid [13]. This study indirectly addresses survivability through the analysis of vulnerability. On the communication technology side, Wang et al. [59] evaluate the reliability of wide-area measurement systems (as part of the monitoring infrastructure of the smart grid) using Markovian models. However, none of these approaches considers transient measures for survivability. Resilience, defined as the ability of a system to bounce back from a failure, is a quantitative metric related to survivability. Decision support for phased recovery of a power grid, based on an analysis of the resilience of the grid throughout restoration efforts, has been presented in [6].

A phased-recovery model
Our recently introduced approach [8,46] targets assessment of transient properties of the power systems accounting for the implications of electro-mechanical and computer-based strategies to address failures in an integrated manner. In this approach, we quantify the effect of FDIR behaviour and demand/response functionality on survivability metrics, based on extended SAIDI metrics. We assume a topology as shown in Figure 5, where a feeder line between two substations is partitioned into sections that can be isolated by opening recloser devices. In case of a failure, parts of the feeder line can be powered by the backup substation by opening the tie switch.
The assessment of complex systems such as the smart grid with numerous elements and many possible states is highly challenging. The key steps to make our analysis feasible have been: (i) initial state conditioning: considering survivability instead of overall reliability metrics, thus conditioning the initial state of our model to be a failure state, i.e., the use of GOOD models; (ii) state space factorisation: modelling the system behaviour after the failure of a single given section; and (iii) state aggregation: aggregating the state of the sections outside the faulty section. Figure 5 visualises the state aggregation principle. Consider a failure in section i = 5. Then, sections 1 to 4 are treated together as the downstream sections (denoted as i−) and sections 6 to 9 are considered together as the upstream sections (denoted as i+).
Our approach models the recovery of the system as a Markov chain with reward rates as illustrated in Figure 6. The states of the model correspond to the different recovery phases, indicated in the upper part of the figure: failure at section i (blue), isolation of section i and automated restoration of downstream sections i− (black), automated restoration of upstream sections i+, and full repair. Each state has a reward rate, which models the survivability metric of interest. In the following, we use energy not supplied per hour as reward rate. Parameters of our model are p, the probability that the communication network is still operational after a section failure; q, the probability that there is sufficient backup power to supply energy for sections i+; and r, the probability that load can be successfully reduced in case of insufficient backup power. The time-related parameters in our model are α, the time to restore the upstream sections, β, the time to call for demand/response; δ, the time to manually repair the faulty section,  Table 2 Choice of reward rates: lower bound on energy not supplied per hour (ENS/h) (from [46]). and γ, the time to restore communication.
Finally, we denote with the mean time to detect the failure and isolate the faulty section. Because is an order of magnitude smaller than the other intervals of time considered in this paper, we assume its value to be = 0. The average time spend in state 0 (cf. Figure 6) is therefore also 0; the initial probabilities for states 1,3 and 4 are then equal to pq, p(1 − q) and 1 − p, respectively.
A final assumption in our model is that the failure of communication and the load in the sections are independent of the failed section. Additionally, the model presented here supports a single faulty section only. In ongoing work [46], we are extending this to multiple failures.

Case study
In the following, we show a typical case study of applying our method taken from [46]. The load per section in the topology (cf. Figure 5, based on a feeder line in Virginia, USA [60]) is input to compute the values of the reward rates at the different model states, as shown in Table 2. The reward rates are computed based on data provided by the engineers of the power grid about its topology, taking into account the load and supply at each section. As the worst-case scenario, we consider a situation in which section 1 fails, i.e., i = 1, thereby maximising the demand placed on the backup substation to supply the i+ sections. Figure 7 (cf. [8]) shows the expected accumulated energy not supplied (EAENS) by time t, for two cases, namely the case when demand-response is not enabled (left graphs; r = 0) and demand-response being enabled (right graph; r = 1). Using our method, changes in different parts of the system, i.e., due to investments, can be assessed. In the following, we analyse the relation between q and r, as an example. If q = 0.9, that is, there is a high probability that the backup power suffices for sections i+, demand response does not have a significant impact on EAENS. In contrast, if q = 0.1, demand response does play a key role, because sections i+ can be automatically restored when demand response is effective. The corresponding plots in Figure 7 (left and right) also demonstrate the significant impact of integrated demand response on EAENS.
The presented method allows us to quantify how various input parameters affect the EAENS. In other work, cf. [36,35], we show how these input parameters can be derived for existing power grids and how investments can be optimised. We have also applied the approach to a distribution automation benchmark derived from a German medium voltage distribution network [35].

Conclusions
The presented method supports the evaluation of different investment alternatives to improve survivability in distribution automation power grids.
The results obtained in the case study indicate that the integration of demand response with failure recovery results in a significant reduction in the amount of energy not supplied after a failure. In the future our models can serve to quantify the trade-offs between investment cost and reliability gains. The presented method is a step towards an holistic approach to guide investment decisions on different parts of a smart grid infrastructure.

Summary and future work
In this paper we provide an overview of three approaches towards the modelling and analysis of the survivability of smart infrastructures, in particular, gas, water and electricity networks. As more and more citizens rely on such infrastructures, adequate means to address such infrastructures, in an efficient model-based way, become more important. Such means help to make important design trade-offs and to see whether and where investments are needed to guarantee continuous operation.
From a modelling and analysis perspective, the challenges to tackle lie in: • handling of discrete and continuous quantities, next to deterministic and random behaviour; • dealing with large-scale systems, that is, the ability to deal with large models, large state spaces, and still provide computationally attractive numerical procedures; • bridging the gap between the viewpoint of an application-engineer (who focusses on the application, that is, the gas, water or electricity network), and that of the modeller and analyser of the models.
The three presented approaches have quite a lot in common, however, also have remarkable differences. The approaches for smart gas and electricity infrastructures are based on a form of behavioural decomposition [56] in which the failure (or dis-aster) handling process is modelled separately from the performance of the system, much the same as done in performability evaluation using Markov-reward models [28]. In contrast, in the approach taken for the water system is truly hybrid, in that the failure handling process and the water transportation and storage are combined in a single integrated model. This fully integrated approach has the advantage of avoiding the approximation due to the behavioural decomposition, however, this comes at the price of a more limited use of stochastic variables (only one general stochastic event can be modelled).
In this paper we have only shown small-scale applications of the three recently developed methods for survivability evaluation. A practical challenge is to team up with true application engineers, that is, from gas, water and electricity operators, to come to models for real systems. The final proof of the pudding is in the eating!


6 Seconds of Sound and Vision: Creativity in Micro-Videos

The notion of creativity, as opposed to related concepts such as beauty or interestingness, has not been studied from the perspective of automatic analysis of multimedia content. Meanwhile, short online videos shared on social media platforms, or micro-videos, have arisen as a new medium for creative expression. In this paper we study creative micro-videos in an effort to understand the features that make a video creative, and to address the problem of automatic detection of creative content. Defining creative videos as those that are novel and have aesthetic value, we conduct a crowdsourcing experiment to create a dataset of over 3,800 micro-videos labelled as creative and non-creative. We propose a set of computational features that we map to the components of our definition of creativity, and conduct an analysis to determine which of these features correlate most with creative video. Finally, we evaluate a supervised approach to automatically detect creative video, with promising results, showing that it is necessary to model both aesthetic value and novelty to achieve optimal classification accuracy.

Introduction
Short online videos, or micro-videos, have recently emerged as a new form of user-generated content on social media platforms such as Vine, Instagram, and Facebook 1 . The Vine platform, in particular, has become associated with the notion of creativity, as it was launched with the goal of allowing users to create 6-second videos whose time constraint "inspires creativity" 2 . Some commentators have even claimed of Vine in particular that "its constraints were allowing digital videos to take on entirely new forms" 3 , and interest in Vine videos has prompted the creation of a specific 6-second film category at major film festivals such as the Tribeca Film Festival in New York.
Not all micro-videos uploaded on social media platforms are creative in nature (1.9% of randomly sampled videos were annotated as creative in our study), and quality can vary widely. This motivates the need for automatic approaches to detect and rank the best, and in particular the most creative, micro-video content on social media platforms. Such applications can increase the visibility of video authors, and replace or augment current features of social-media platforms such as "Editors Picks", which showcases the best content on Vine.
Micro-videos provide a unique opportunity to address the study of audio-visual creativity using computer vision and audio analysis techniques. The very short nature of these videos means that we can analyze them at a micro-level. Unlike short video sequences within longer videos, the information required to understand a micro-video is contained within the video itself. This allows us to study audio-visual creativity at a fine-grained level, helping us to understand what, exactly, constitutes creativity in micro-videos.
In this paper we study the audio-visual features of creative vs non-creative videos 4 and present a computational framework to automatically classify these categories. In particular, we conduct a crowdsourcing experiment to annotate over 3,800 Vine videos, using as guidelines: (1) a widely accepted definition of creative artifacts as those that are novel and valuable, and (2) insights from the philosophy of aesthetics about the judgements of aesthetic value (i.e. sensory, emotional/affective, and intellectual). We go on to use this dataset to study creative micro-videos and to evaluate approaches to automatic detection of creative micro-videos.
The main contributions of this paper are: • We create a new dataset of creative micro-videos, and make the vine video ids and annotations publicly available to the research community 5 .
• We propose and implement a new set of features to model the novelty and aesthetic value of micro-videos.
• We analyze the extent to which each of these features, and other existing features, correlate with creativity, giving insights into the audio-visual features most associated with creative video. We also classify videos as creative/non-creative, with promising results, and we show that combining aesthetic value and novelty features gives highest accuracy.
Unlike previous work in computational aesthetics [5,7], which mainly focuses on assessing visual beauty using compositional features, we explore here the more complex and subtle concept of creativity. Focusing on creative content allows us to analyze audio-visual content from a different perspective, allowing us to model the fact that creative content is not always the most beautiful-looking (in the conventional sense) or visually interesting. To the best of our knowledge, this is the first work to address creativity in micro-videos.
In the next Section we present related work, and we define video creativity in Section 3. In Section 4 we describe a crowdsourced annotation of Vine videos. Section 5 presents computational features for modeling creativity. In Section 6 we correlate these features with, and evaluate the automatic classification of, creative content. We conclude in Section 7.

Related Work
Our work is closely related to computational approaches to studying concepts such as beauty [5], interestingness [7], memorability [10], or emotions [17]. In particular, we are influenced by recent work in computational aesthetics for the automatic assessment of visual beauty. The earliest work [5,12] distinguishes between high-quality (professional) and low-quality (amateur) photos based on features inspired by photographic rules, with applications in image quality enhanchement [3] and automatic aesthetic feedback for photographers [32]. Nishiyama et al. [25] propose more complex visual features based on color harmony, and combine them with low-level features for aesthetic image classification. Other work has investigated generic local features for modeling beauty, showing that they outperform systems based on compositional features [19]. Several researchers have included the semantics of the image in the aesthetic evaluation, labeling images according to their scene content and building category-based beauty models [16,23].
The main difference between visual aesthetic research and our work is that the notion of creativity is more complex than visual photographic beauty, in addition to the fact that we also focus on audio. We argue that creative videos may not be always considered 'beautiful' in the conventional sense, and may even be 'ugly'. While we incorporate and re-elaborate many of the mentioned approaches for detecting creative videos, by using sensory (including aesthetic), and visual affect features, we also design a new set of features to model audio-visual creativity.
Moreover, while much related work focuses on still images, in our work we analzye video data, and we build specific video features for micro-videos. The few previous works on video aesthetics build video features based on professional movie aesthetics [4,2], or simply aggregate frame-level features [21], with limited success.
Also different from much of the work in computational aesthetics, we use a croudsourced groundtruth, allowing us to create a high quality labelled dataset using a set of annotation guidelines tailored for creativity. Crowdsourcing was previously used to build a corpus for image memorability [10], but most computational aesthetics research exploits online professional photo websites such as dpchallenge.com [5,12,7,23,16], photo.net [16], or Flickr [7].

Defining Video Creativity
Although the precise definition of creativity has been the subject of debate in many disciplines, one of the most common observations is that creativity is connected with imagination and innovation, and with the production of novel, unexpected solutions to problems [24]. However, "All who study creativity agree that for something to be creative, it is not enough for it to be novel: it must have value, or be appropriate to the cognitive demands of the situation" [31], an idea that is shared by many researchers [8,22,31]. Based on these observations, we define a creative artifact as one that is novel (surprising, unexpected) and has value.
As applied to micro-videos, we interpret by novelty that the video is unique in a significant way, or that it expresses ideas in an unexpected or surprising manner. Value is a more complex notion, however, and in this context it is best equated with aesthetic value. Most definitions of aesthetic value incorporate the maxim that beauty is in the eye of the beholder: Immanuel Kant, for example, in his Critique of Judgement [11], argues that aesthetic judgements involve an emotional response (e.g., pleasure) to a sensory input (i.e. the audio-visual signal from the video) that also provokes "reflective contemplation". At the risk of oversimplifying, judgements of aesthetic value involve sensory, emotional and intellectual components.
In the following sections, we will use this definition to: (1) provide a definition of creative video as part of our guidelines for crowd workers to annotate videos as creative or non-creative (Section 4), and (2) inform our choice of computational features for modeling creative videos.

Dataset
To create a corpus of micro-videos annotated as creative, we first identified a set of candidate videos that were likely to be creative. This was necessary because our preliminary analysis showed that only a small fraction of videos are creative, meaning that a random sampling would need an extremely large annotation effort to collect a reasonable number of positive creative videos to analyze. With this in mind, we defined a set of sampling criteria likely to return creative videos. We started by sampling 4,000 videos. Specifically, we took (a) 1, 000 videos annotated with hashtags that were associated to creative content by 3 different blogs about Vine: #vineart, #vineartist, #artwork, and #vineartgallery (b) 200 videos mentioned in 16 articles about Vine creativity on social media websites, (c) 2, 300 videos authored by the 109 creators of the videos identified in criteria b, based on the assumption that these authors are likely to author other creative micro-videos, and (d) 500 randomly selected videos from the Vine streamline, for the purpose of estimating the true proportion creative videos on Vine. The results of the labeling experiment summarized in Table 3 confirm the validity of this sampling strategy: while only 1.9% of the random sample has been labeled as creative (D-100), our sampling strategy yielded 25% creative videos, giving a corpus that is large enough to be useful. In total, after discarding invalid urls, we annotated 3,849 candidate videos, created and shared between November 2012 to July 2013.
We annotate these videos using Crowdflower 6 , a large crowdsourcing platform. To ensure quality annotations, the platform enables the definition of Gold Standard data where workers are assigned a subset of pre-labelled 'jobs', allowing the known true label to be compared against the contributor label. This mechanism allows worker performance to be tracked, and can ensure that only judgements coming from competent contributors are considered. It also presents an opportunity to give feedback to workers on how to improve their annotations in response to incorrect answers.
In the experiment, a contributor looks at a 6-second video and judges if it is creative. According to Section 3, a creative video is defined as a video that: (1) has aesthetic value, or evokes an emotion (happy, sad, angry, funny, etc), and (2) has interesting or original/surprising video/audio technique. The worker is advised to listen to the audio, and can watch a pair of exemplar creative and non-creative videos before performing the job. After watching the target video the contributor answers the question "Is this video creative?" with "positive", "negative" or "don't know". In the first two cases, the user can give more details of the motivation of their choice according to the criteria in Table 1, phrased in a simple language appropriate to crowdsourcing platforms, where workers typically do not take time to read complex definitions and guidelines [20]. To ensure that the job could be easily understood by crowd workers, in a preliminary survey we collected feedback on the interface from 15 volunteers.
The experiment ran for 5 days and involved 285 active workers (65 additional workers were discarded due to the low

Sensory
The audio is appealing/striking The visuals are appealing/striking Emotional The video evokes an emotion Intellectual The video suggests interesting ideas

Novelty
The audio is original/surprising The visuals are original/surprising The story or content is original/surprising Table 1. Criteria for labeling a video as creative quality of their annotations) located in USA (88%), United Kingdom (8%), and Germany (4%). 7 No time constraint was set on the task, and each video was labeled by 5 independent workers. The final annotations reached a level of 84% worker agreement (82% for creative, 85% for non-creative), which we consider high for this subjective task. Looking at pervideo agreement, summarized in Table 2, 48% of videos have 100% agreement (i.e. all 5 independent annotators agreed), 77% show an 80% consensus. These levels of agreement represent different criteria for labeling a video as (non) creative, and in Section 6 we consider 3 different labelled ground-truth datasets, D-100, D-80, and D-60, based on 100%, 80% and 60% agreement. From Table 2 we can also see that 25-30% of videos were annotated as creative.  Table 3. Creative vs non-creative videos per sampling strategy, for the D-100 dataset (100% agreement). Table 3 shows the distribution of creative and non-creative videos according to the strategy used to sample the videos. As expected, the videos specifically mentioned in blogs about Vine (b) have the highest proportion of creative videos, while the vast majority of randomly sampled videos (d) are non-creative, justifying the need for our sampling strategies.

Features for Modeling Creativity
In this Section we describe novel and existing features for modeling creative micro-videos, which we group based on the two components of our definition of creative videos: novelty and value. We re-use existing features from computational aesthetics, semantic image analysis, affective image classification, and audio emotions modeling, and propose

Aesthetic Value Features
We use a set of features to model the aesthetic value of a video based on two of the three components of aesthetic value identified in Section 3: the sensory component and the emotional affect of the video. The third, intellectual, component is, to the best of our knowledge, not modeled by any existing computational approaches, so we do not model it in this work.

Sensory Features
Sensory features model the raw sensory input perceived by the viewer, which can be approximated by the raw signal output by the video. Such features cover all aspects of the signal, i.e. visual, audio, movement, filmmaking techniques, etc. We implement existing features for semantic image classification and aesthetic image analysis, and we design new descriptors to capture the structural characteristics of short-length online videos.
Video Scene Content. We extract the 462-dimensional namely the Saliency Moments feature [26] from video frames, a holistic representation of the content of an image scene based on the shape of the salient region, which has proven to be extremely effective for semantic image categorization and retrieval.
Composition and Photographic Technique. In computational aesthetics, several compositional descriptors describing the photographic and structural properties of images and video frames have been proposed. Other features attempt to model the visual theme of images and videos [29]. We use some of the most effective frame-level compositional features, such as the Rule of Thirds and Low Depth of Field [5], the Michelson Contrast [6], a measure of Symmetry [27], and a Uniqueness [27] measure indicating the familiarity of the spatial arrangement. Finally we implement a feature describing the Image Order using information theory-based measurements [28].
Filmmaking Technique Features. We design a set of new features for video motion analysis, inspired by movie theory and tailored to model the videomaking techniques of short on-line videos. General Video Properties. We compute the number of frames N f and the number of shots N s in the video. In the current setting, the number of frames is a proxy for frame rate, as almost all videos are exactly 6 seconds in length, whereas the frame rate tends to vary. Stop Motion. Many popular creative short videos are stopmotion creations, where individual photos are concatenated to create the illusion of motion. In such videos the frequency of changes in the scene is lower than traditional videos. We capture this technique by computing the Euclidean distance δ(F i , F i+1 ) between the pixels of neighboring frames F i and F i+1 and then retaining as a stop motion measure S the ratio between N f and the number of times such difference is not null (the scene is changing), namely . (1) Loop. Many popular videos in Vine are shared with the hashtag #loop. A looping video carries a repeatable structure that can be watched repeatedly without perceiving where the beginning/end of the sequence is. To capture this, we compute the distance between the first and the last frames of the video, namely L = δ(F 1 , F N f ) Movement. similar to previous works, [4,2], we compute the amount of motion in a video using a feature that can describe the speed of the main objects in the image regardless of their size. We first compute a saliency map of each frame and then retain, as a movement feature, the average of the distances between the maps of neighboring frames: where SM (·) is the saliency map computed on the frame using the Spectral Residual technique [9]. Camera-Shake. Typical micro-videos are not professional movies, and often contain camera shake introduced by handheld mobile phone cameras. Artistic video creators, however, often carefully produce their videos, avoiding camera-shake. We compute the average amount of camera shake in each frame using an approach based on the directionality of the Hough transform computed on image blocks [1].

Emotional Affect Features
In this section we separately introduce sets of visual and audio features known to correlate with emotional affect.

Novelty
The novelty of an artifact can be represented by its distance from a set of other artifacts of the same type. One way to compute such distance is to first divide the attribute space into K clusters, and then calculate the distance the between the artifact and its nearest cluster [18]. In our approach, we compute an improved novelty feature that takes into account the distances between the artifact attribute and all the clusters in the attribute space, thus measuring not only the distance to the most similar element, but the detailed position of the attribute in the space.
We measure novelty for both the visual and the audio channel of the video, using as attributes the aesthetic values features from Section 5.1. We take a random set of videos, independent of our annotated corpus, and extract the 4 groups of visual attributes (Scene Content (SC), Filmmaking Techniques, Composition and Photographic Technique and Visual Affect), and the Audio Affect attributes. We cluster the space resulting from each attribute into 10 clusters using K-means, obtaining 40 clusters for the visual attributes (10 clusters each for 4 attributes) and 10 for the audio attribute.
To calculate the novelty score for a given video, we extract the visual and audio attributes, and we then compute the Audio Novelty as the collection of the distances between the Audio Affect attribute of the video and all the clusters of the corresponding space (giving a 10 dimensional feature). Similarly, we compute the video Visual Novelty as the set of distances between each visual attribute of the video and the corresponding cluster set (40 dimensions).

Experimental Results
In this Section we explore the extent to which audiovisual features correlate with creative video content, and then evaluate the approaches for creative video classification.

What Makes a Video Creative?
To analyze which features correlate most with creative micro-videos, we consider videos with 100% agreement (i.e. D-100 from Table 2), as we are interested in the correlations for the cleanest version of our dataset. We extract 7 groups of features for each video: Scene Content, Composition/Photographic Technique, Filmmaking Technique, Visual Emotional Affect, Audio Emotional Affect, Visual Novelty, and Audio Novelty. For frame-level features, we consider the features of middle frame of the video.
We first analyze to what extent each group of features correlates with video creativity, using the Multiple Correlation Coefficient (M P C), which measures how well a multidimensional variable fits a monodimensional target variable, given the reconstructed signal after regression. In our context, the elements of the multidimensional variable are the individual features within a feature group.   In Figure 1(a), we report MPC values for all features, testing how well each of our groups of features fits the label vector of our data. The results show that Scene Content is most strongly correlated with creative videos, followed by Composition/Photographic features and Video Novelty, showing that both novelty and aesthetic value features are crucial for understanding creativity. Emotional affect features are less strongly correlated than sensory features, suggesting that the raw sensory output of the audiovisual stream is more useful than features that attempt to model emotional affect. The correlation for audio features is much lower than for visual features: it seems likely that micro-video authors and annotators both place more emphasis on the visual aspect.
To determine, within each group of features, the most important individual features, we calculate the Pearson correlation coefficient ρ between individual features and creative labels. For this analysis, we exclude the Scene Content features, whose elements are non-separable, and the Novelty features, which require all features to represent the position of the video in the attribute space.
In Figure 1(b) we show the ρ values for the most highly correlated features. Among the Composition/Photographic Technique and Visual Affect features, we can see that creative videos strongly correlate with visual uniformity (positive correlation with Energy), and order (negative correlation with Entropy and the Complexity measure), suggesting that videos with homogenous frames are more likely to be perceived as creative. Moreover, a negative correlation with the Hue statistic in both the whole image (Visual Affect -Hue) and in the inner quadrant (Composition/Photographic -Hue (RoT)) and a highly positive ρ for Visual Affect -Brightness and Composition/Photographic -Brightness (RoT) shows that creative videos are related to warm and bright colors. The Uniqueness feature shows a positive ρ, indicating videos with a less familiar layout are more likely to be labeled as creative. Surprisingly, important properties for visual aesthetics such as Symmetry (ρ = 0.04) and Low Depth of Field (ρ = 0.02) do not play a key role for creative video detection (these are not shown in Figure 1(b), which only includes features with a high correlation). Also, Skin color (Visual Affect) is negatively correlated with creativity, showing that the presence of people is not so common in creative videos, unlike most other popular videos.
Creative videos are associated with highly pleasant emotions, and dominant, non-overwhelming, controllable emotions (ρ = 0.24 for Pleasure and ρ = 0.23 for Dominance).
In terms of Audio Affect, Onset Rate and loudness (Short Time Energy) are negatively correlated with creativity, meaning that less frenetic, low-volume sound is preferred.
Regarding Filmmaking Techniques, the most highly correlated feature is the presence of the loop technique, a common form of expression in micro-videos (it is negatively correlated because a high score indicates low likelihood of a loop). Camera Shake, which can be seen as an inverse quality measure, is a negative indicator of creativity, suggesting that more 'polished' videos are likely to be seen as creative.
As we can see from our findings, creativity involves a variety of dimensions beyond photographic beauty. Photographic features traditionally used in visual aesthetic frameworks [5,7] are equally or even less correlated with creativity than other, complementary, features. Moreover, we can see that single Filmmaking Technique features such as Loop, or Audio Affect features such as Onset Rate, are better indicators of creativity, compared to single photographic features.

Classifying Creative Videos
We now evaluate methods to automatic classify videos as creative, using the same features.
Experimental Setup. In Table 2 in Section 4 we described three different versions of our annotated corpus, based on 60%, 80% and 100% per-video annotation agreement. These datasets give a natural tradeoff between dataset size and label accuracy. To measure the effect of this tradeoff on classification, we report results for each dataset.
For each version of the corpus we use 2/3 of the positive examples for training, and the rest for testing. For training and testing, we subsample an equal amount of negative ex-amples, to ensure a balanced set. We train a separate Support Vector Machine with Radial Basis Function (RBF) kernel for each of the 7 groups of features. For groups of features that are calculated for a single video frame, at the training stage we sample 12 frames for the video, and create a separate training instance for each sampled frame, each given the label of the parent video. We use the trained models to classify the creative videos in the test set. For each video, the classifier outputs a label and a classification score. For the frame-level features, we sample 12 frames as in training, classify each, and retain as overall classification of the video the rounded average of the single frame scores. We use classification accuracy as our evaluation measure.
For the novelty features, we use 1000 non-annotated videos for the clustering. To check that this number does not introduce any bias in our experiment, we re-compute clustering on an increasing number of videos, from 500 to 5000, and obtained similar results as those presented in Table 5.
To test the complementarity of the groups of features and the improvement obtained by combining them, we also combine the classification scores of different classifiers using the median value of the scores of all the classifiers, previously shown to perform well for score aggregation [13].
Results. The classification results are shown in Table 5. Similar to the correlations, we can see that the best feature group is Composition/Photographic Technique, with 77% accuracy (D-100 dataset), followed by Scene Content and Filmmaking Technique features. We can also see that Emotional Affect features are outperformed by Sensory features. Our new, 6-dimensional, Video Technique feature achieves comparable classification accuracy to the performance of the 462 dimension Scene content feature. Combining emotional and sensory features improves classification accuracy to 79%, showing the complementarity of these features. Although the Novelty features carry some discriminative power for creative video classification, Aesthetic Value features are still more discriminative. However, when we combine novelty and value features, we can see their complementarity, with the classification accuracy increased from 79% to 80% for the D-100 dataset.
Overall, we can notice the importance of using a diversity of features for creativity prediction, since classifiers based on traditional photographic features or generic scene features, typical of visual aesthetic frameworks, benefit from the combination with other cues, justifying a tailored framework for creative video classification.
Finally, we can also see that the quality of the annotations is crucial: classification accuracy is always much higher for the cleanest dataset, D-100, even though this dataset is only 60% the size of the D-80 dataset, and less than half the size of the D-60 dataset.

Conclusions
In this paper, we study creativity in short videos, or microvideos, shared in online social media platforms such as Vine or Instagram. Defining creative videos as videos that are novel (i.e., surprising, unexpected) and have aesthetic value, we run a crowdsourcing experiment to label more than 3,800 micro-videos as creative or non-creative. We obtain a high level of inter-annotator agreement, showing that, with appropriate guidelines, it is possible to collect reliable annotations for a subjective task such as this. From this annotation we see that a small, but not insignificant, 1.9% of randomly sampled videos are labeled as creative.
We propose a number of new and existing computational features, based on aesthetic value and novelty, for modeling creative micro-videos. We show that groups of features based on scene content, video novelty, and composition and photographic technique are most correlated with creative content. We show that specific features measuring order or uniformity correlate with creative videos, and that creative videos tend to have warmer, brighter colors, and less frenetic, low volume sounds. Also, they tend to be associated with pleasant emotions, and dominant, non-overwhelming, controllable emotions. Loop and Camera Shake features, specifically designed for modeling creativity in micro-videos, also show high correlation with creativity. Several features traditionally associated with beauty or interestingness show low correlations with creative micro-video, underlining the difference between creativity and those concepts. Specifically, skin color, symmetry and low depth, which are widely used in modeling beauty and interestingness, are not correlated with creative micro-videos.
Finally, we evaluate approaches to the automatic classification of creative micro-videos. We show promising results overall, with a highest accuracy of 80% on a balanced dataset. The best results are achieved when we combine novelty fea-tures with aesthetic value features, showing the usefulness of this twofold definition of creativity. We also show that high quality ground truth labels are essential to train reliable models of creative micro-videos.
In future work, we plan to enlarge the set of features for modeling creativity. We will design features to model the intellectual aspect of aesthetic value through semantic visual cues such as specific visual concept detectors. Moreover, we plan to include non-audiovisual cues such as the metadata related to the video (tags, tweets, user profile), the comments about it, and its' popularity in the social media community.
Furthermore, we would like to apply our model, or a modified version of it, to other micro-video platforms and also to a broader spectrum of multimedia content, such as images and longer videos, etc., and to study the differences and commonalities between their creative features.