extfieldsofstudy:
Business
Computer Science

text:
Developing a Trust Domain Taxonomy for Securely Sharing Information Among Others

In any given collaboration, information needs to flow from one participant to another. While participants may be interested in sharing information with one another, it is often necessary for them to establish the impact of sharing certain kinds of information. This is because certain information could have detrimental effects when it ends up in wrong hands. For this reason, any would-be participant in a collaboration may need to establish the guarantees that the collaboration provides, in terms of protecting sensitive information, before joining the collaboration as well as evaluating the impact of sharing a given piece of information with a given set of entities. The concept of a trust domains aims at managing trust-related issues in information sharing. It is essential for enabling efficient collaborations. Therefore, this research attempts to develop a taxonomy for trust domains with measurable trust characteristics, which provides security-enhanced, distributed containers for the next generation of composite electronic services for supporting collaboration and data exchange within and across multiple organisations. Then the developed taxonomy is applied to possible scenarios (e.g. Health Care Service Scenario and ConfiChair Scenario), in which the concept of trust domains could be useful.

INTRODUCTION
Why trust is so important in online activities? Trust is an element that entails in online exchange relationships characterized by uncertainty, anonymity, a lack of control and potential opportunism [1]. In online relationships, such as online transactions often require sharing of sensitive information (e.g. username, password, emailing address or telephone number), corporate information (e.g. airline reservation or inventory data) and financial information (e.g. credit or debit card numbers) among the transacting parties. Therefore, trust helps to ensure that one does not take an advantage of others during or after the particular transaction.
There has been an increasing interest in the imperative of trust in societies in general and in organizations in particular [2], [3], [4], [5], [6], [7], [8], [9], [10], [19], [20]. However, there is no common integrative taxonomy to interpret the extant literature and to guide future research regarding online trust related issues [4], [19], [20] . Li [11] offers a systematic review; other authors [12], [13] present a meta-analysis. Clearly, some aspects of trust are subjective; moreover there are many contextual factors in play. It is often related to and also affected by individuals' differences and their situational factors [9]. It is true that different people interpret the role of trust quite differently in various situations and have a different scale of trust towards different trustees. The Trusted Computing Group defines trust by saying "an entity can be trusted if it always behaves in the expected manner for the intended purpose" [14].
Information sharing is an essential part of any business process. Every day, organizations and individuals exchange information for the purpose of service delivery, communication and collaboration [21]. For example, a researcher wishing to set-up an experiment may write an email to an infrastructure provider, describing the experiment and specifying the resource requirements for the experiment. This experiment could be of a sensitive nature, such as development of a vaccine or stem cell research and therefore requires that its content, intention or existence be known only to a selected list of individuals and organizations. Similarly, two law enforcement agencies working on similar cases may require sharing information about the evidence of a crime. However, each agency may need to share such information with only a selected list of other agencies or individuals from these agencies. In this case, how does one on the selected email list ensure the information shared by someone else hasn't been altered or contaminated by intruders? In general, we tend to trust people on our email list [21] [22]. However, there might be a hidden risk of contaminating the information (e.g. email message) during online transmission. A popular maxim captures the essence: "Trust is good, but control is better" [15]. There is, therefore, a need for mechanisms that allow parties to share such information while restricting its flow to the selected entities.
The research work reported in this paper introduces an integrative taxonomy for trust domains. The concept of trust domains is meant to be used for proving a foundation (evidence) for securely sharing information (how, when and with whom) among a group of entities. This enables the parties involved and the observers to understand the level of trust before going ahead with sharing data.
The reminder of this paper is structured in the following manner. Section II describes the elements of trust domains. Section III discusses the useful infrastructure models to develop our proposed taxonomy for trust domains. In section IV, we introduce a taxonomy for trust domains, which enables to securely share the information among the others. Then the introduced taxonomy is applied to a possible scenario in Section V. Section VI provides conclusions and opens up opportunities for future work that may extend the research work reported in this paper.

II. ELEMENTS OFTRUST DOMAINS
In this paper, we are interested in providing these mechanisms in a way that enables the parties involved in information sharing to understand: i) the level of security that will be achieved together with the risks mitigated; ii) the mechanisms used to achieve it; and iii) economic, performance and scalability implications. Most importantly, these mechanisms should allow these parties to evaluate the above characteristics before they go ahead with sharing the information. To achieve this, we propose the notion of a trust domain as a concept that makes this possible -enabling organizations and individuals to maximize their business profitability through efficient and secure information sharing.
To start, we need to identify and define concepts that could help us capture the notion of a trust domain. In this paper, we are interested in identifying and characterizing these concepts in order to: i) provide perspectives from which to discuss the concept of a trust domain; ii) enable instantiation of trust domains with varying properties; and iii) provide integration and interoperability mechanisms. We build a model that captures the concepts that could be used to describe a trust domain and relationships among those concepts.
The model assumes that a trust domain will require some social structure as well as infrastructure support. The social structure provides regulatory and monitoring mechanisms for individuals and organizations involved in a trust domain as well as a means for assigning responsibility and appropriate penalty in the event that information flows to unwanted parties. The infrastructure, on the other hand, serves to provide mechanisms that ensure that technical systems involved in or used as a means of processing, communicating or sharing information observe and enforce the required information flow constraints. Therefore, we take an approach in which we identify some of the existing mechanisms that could serve as fundamental building blocks for a trust domain and then we find possible integration points to enable the conceptualization of a trust domain.

III. USEFUL INFRASTRUCTURE MODELS
Our taxonomy for trust domains embraces systems deployed on diverse infrastructures. Each infrastructure may employ different technologies to serve the needs of a trust domain. However, at the architectural level, several aspects can be identified to be common among the various types of infrastructure. Of interest to develop taxonomy for trust domains are the Web Services Architecture [16] and the Trusted Multi-tenant Infrastructure (TMI) [17].

A. Web services Architectural Model
The Web Services (WS) Architecture [16]defines functional components, relations among them and constraints upon which they operate. The architecture defines four models (message-oriented, policy, service-oriented and resource) that capture concepts and relations among them as viewed from a certain perspective. We collect the concepts defined in these models into an integrated architectural model as illustrated in Figure 1. The message-oriented model defines Message as a first class concept that has a Sender and one or more Receivers. A message is delivered by some MessageTransport mechanism, which is constrained by a DeliveryPolicy. The Delivery Policy is a subclass of Policy, defined in the policy model, which defines MessageReliability properties of the MessageTransport mechanism. Messages are used as the main mechanisms for communicating requests for services and responses to the requests -where a service is defined, in the service-oriented model, as an abstract resource capable of performing one or more tasks. This definition of a service allows us to link a service to a Resource defined in the resource model through a subclass relation.
A resource has State and is owned by a person or organization that defines a Policy that governs how a Resource is used or behaves. This policy is also defines the scope of a domain and is defined in a way that captures the desired goal state. Policies capture permissions that are required by certain agents in order to enable certain resources attain some state and obligations that agents have on the resources. Permissions are enforced by a PermissionGuard while Obligations are enforced by an Audit Guard.

B. Trusted System Domain Model
The TCG TMI Working Group [17] aims to build a set of specifications for trustworthy operation in multi-tenant infrastructures. As part of this effort, the working group has released the first version of the use cases for a TMI. We use these use cases to build a model that includes concepts and relations among the concepts defined in the use cases. The concepts capture the parts of a TMI while the relations capture the dependencies, causalities and interactions among the concepts necessary to capture the functionality of a TMI as described in the use cases.
The model, illustrated in Figure 2, is built around three main entities, Policy, Asset and Role. An Asset is an entity that is of value to a person or organization that participates in a TMI. Its value is defined by the entity that has an interest in it. Implying that the same asset could have different values depending on whose point of view this value is being computed. A Policy is a piece of data that defines the constraints on how the assets are used or accessed by entities within a TMI as well as the expected behavior of such entities. A Role is a set of responsibilities assumed by an entity in the system. It also defines the activities that any entity with such a role can perform. TMI distinguishes between two main roles, Consumer and Provider, and defines several concepts some of which are generic across domains of the two roles while others are specific to each domain. A DomainAuditAgent is defined as a generic agent that forwards AuditEvent to the CentralAuditStore -a component that stores audit events collected from various parts of a TMI. The ConsumerAuditAgent and ProviderAuditAgent are subclasses of DomainAuditAgent that are specific to Consumer and Provider domains respectively.
Each Role establishes a Policy that is published to a DomainPolicyStore and also creates a DomainManagementAgent to manage the domain according to the policy they define. This Policy is expected to be enforced by a PolicyEnforcementPoint in the domain in which the policy is setup, and is monitored by the specific type of DomainAuditAgent for the particular domain. Consumer DomainManagementAgent and ProviderDomainManagement Agent are subclasses of DomainManagementAgent that are specific to Consumer and Provider domains respectively.
A Resource is defined as a subclass of Asset, which has State and can be provisioned or de-provisioned in the Provider domain by the ProviderDomainManagementAgent. To ensure that the correct policy is being enforced on a resource, the Consumer can validate the state of the resource using the ConsumerDomainManagementAgent.

IV. PROPOSED TRUST DOMAINS TAXONOMY
In this section of the paper, we discuss how the models identified above are combined to create trust domain taxonomy. We illustrate the concepts that can be used to integrate the models and discuss how the semantic gap in the usage of these concepts can be bridged.

A. Fundamental Concepts and Relations
The proposed model consists of a number of concepts, such that each concept captures a class of things that may exists in a trust domain, be used to build a trust domain and used within a trust domain. Though all these concepts may be used in different instances of a trust domain, a few of them can be identified as being fundamental to the existence of a trust domain. We identify the Data, Policy, Controls, Roles, Actions and Evidence as being fundamental concepts necessary to build a trust domain.
As depicted in Figure 3, a Role owns Data that will exist within a trust domain and establishes a Policy that constrains Actions. As mentioned in the fundamental model, a Role establishes one or more policies within the domain. However, any given policy can only be established by one role. This means that if two roles establish identical policies, then both policies are treated as a unique entity, which can be linked through the equivalence property.
Actions are performed by a given role or by some agent that represents a particular role. These actions are monitored by Controls to ensure that the policy is being upheld. These controls produce Evidence to indicate that actions have been performed in accordance to the policies. Both Evidence and Policy can be considered to be a form of data, which can be manipulated in the same way as other data and may be subject to the same information flow restrictions. We describe each of these concepts in the following subsections.

1) Asset
Our conceptualization of a trust domain is based on the idea of enabling secure information flow among a set of entities.
Such entities may each have a set of devices through which they share the data. Furthermore, these entities may provide access to the information stored on the devices or other media to other members of the domain. For this reason, we define the concept of an Asset as being a fundamental element of a trust domain. An asset is something of value to the owner, but could also be valuable to other entities such as attackers or competitors. One example of an asset is data. Data may exist in many different forms and indeed each form is faced with various kinds of challenges, with respect to controlling how it flows. For example, printed material could be prevented from being taken out of the building. However, it is still possible that someone could scan the materials and send it over a network or copy it to USB stick. For this reason, different types of protections maybe required to overcome each type of challenge. To reduce the scope of our work, we limit the model to data that exists in digital form. Other examples of assets include: resources such as computers; services such as web services; and communication infrastructure such as networks. 2

) Policy
Policies are a means of specifying the behavior of entities within a trust domain and how data flows within or outside of a trust domain. Each policy specifies the expected relationship/dependency between one or more entities within a domain. For example, a policy can be used to specify the data allowed to into or out of a trust domain, the characteristics of the controls that should exist in a trust domain and the kind of evidence that these controls should provide. To satisfy the requirements for a trust domain, the policy being enforced, how it is being enforced and the decisions made in relation to this policy must be made explicit.

3) Controls
Controls are a set of mechanisms, processes or procedures that enforce the policies within a trust domain. These controls could be accomplished through social, e.g. penalties, or technical means e.g. trusted computing. Controls monitor activities that occur within a trust domain and produce evidence, described below, that can be used to determine the properties of a trust domain or its constituents.

4) Roles
Roles are used to specify the level of participation in a trust domain. Each role defines the kinds of activities that an entity assuming that role can do as well as the kinds of things that entities would be accountable for. Roles are also a way of separating concerns in a trust domain and isolating the activities that participants can perform in a trust domain.

5) Actions
Actions are a series of functionalities performed by components and agents in a trust domain. An action typically either consumes or produces some data or results in the change of state of a particular component or system. Depending on the level of abstraction desired, an action can be divided into subactions where each sub-action contributes to the overall outcome of the parent action.

6) Evidence
Evidence is data that is produced by the controls within a trust domain to indicate the kinds of activities that have occurred in a trust domain. These activities are captured by monitoring the actions that are performed by or on behalf of roles that exists within a trust domain. Examples of such evidence include; provenance -records of how data came to be; audit logs -log of events that occur during the lifetime of a trust domain; integrity measurement lists -a record of the binary hashes of software components and data in a trust domain; and digital certificates -specify the cryptographic identities of components that perform certain actions.

B. Integrating Infrastructure Models
The fundamental concepts discussed above exist in more than infrastructure model. To build a taxonomy for trust domains that makes use of these models, we need to understand how the models can be integrated together. To achieve this, we use the fundamental concepts as integration points, so that each concept that exists in more than one model is linked using equivalence. This allows all the relations that apply to a concept in a particular model to apply to an equivalent concept in another model. As an example, Policy, which is defined in the fundamental model, exists in the trusted system domain model as well as in the policy model of the Web Service (WS)-architecture. This allows us to relate the policy concept in the Trusted System Domain (TSD) model to the Permission concept in the WS policy model, through the equivalence property between the two policy concepts and the captures property.
Using this approach, however, is faced with the challenge of a possibility of conflicts and inconsistencies. For example, in the WS policy model, the policy is defined by a person or organization while the TSD model specifies that a policy be established by a Role. To address this issue, we first identify that is a possible relationship between person or organization and role. Using this relationship, we can then determine which the appropriate concept to be linked directly to the common concept, i.e. Policy. In this particular case, we decided on Role because it is possible that other entities such as autonomous systems might be able to set-up a policy.

C. Integrated Model
In this section of the paper, we describe an integrated model to form a taxonomy for trust domains resulting from the integration effort described above, also illustrated in Figure 4. We define a DomainEntity as a first class entity in a trust domain. It is defined as an entity that has a memberOf relation to the Domain and has exactly one of the following types: Person, Organization, System, Process, Resource or Agent. Furthermore, each Domain EntityhasRole within the domain. In this model, we do not restrict the membership and the role relationship, so that a domain entity may participate in more than one domain at any given time and may assume more than one role. We define a Policy as data whose scope is limited to a Domain. In other words, a policy is only effective within the domain. It does not directly in the behavior or properties of entities outside the domain. This, however, raises a question of how to define identical domains (i.e. domains that enforce exactly the same kinds of policies). To answer this question, we define a Policy to be singleton object but allow it to be cloned; for example, copies can be made for use in other domains. The cloned policy can be linked to the original policy through the equivalence property.
A policy is publishedTo the DomainPolicyStore by a DomainManagementAgent and is consumed by the PolicyDecisionPoint, which creates PolicyDecisions. Policy Decisions exist in the form of Permissions or Obligations and are enforced by a type of control called PolicyEnforcementPoint. The policy decisions can be considered as a type of part of evidence, which together with some policy meta-data can be used to support how the policy has been broken down into enforceable constraints.
An important aspect in trust domains is the ability to relate the decisions to the policies that triggered these decisions. For example, when a constraint specifies that a certain action be permitted, trust domains must be able to demonstrate how that constraint (i.e. to permit an action) was reached and the policies, which influenced this decision. For this reason, we define influenced as a relation between the set of policies and the set of decisions, which were used to arrive at the decisions. For this to work, we require a policy to be consumed by a PolicyDecisionPoint that creates the PolicyDecision.
A Role may also own Agents, which may act on their behalf, and Assets, which are shared with other members of the domain. Assets have a type, i.e. AssetType, which could either be Resource, Data or Service. Furthermore, the model specifies that individual assets and agents must be owned by exactly one Role.
Actions are performed on Assets by Domain Entities and controls within the domain. However, before an action can be performed, certain constraints determined by the policy must be satisfied. For this reason, we specify that Policy constrainsAction. To enable checking that appropriate constraints are satisfied before an action is performed, controls within the domain monitor actions.
Resources are an important aspect of trust domains. In this model, we define a resource as an asset type, which has state. A resource can be provisioned or de-provisioned by controls within the trust domain. When a resource is provisioned, it becomes available for use by agents and domain entities that request for Assets (including Resources) in the domain. The state of the resource also plays an important role in determining the properties of a trust domain. We therefore allow state to be validated by DomainManagementAgent. This validation determined whether or not the behavior or state of resources is in-line with domain policies.
A special type of control, referred to as DomainAuditAgent, is responsible for generating audit events by monitoring the activities in the domain. The DomainAuditAgent forwards AuditEvent to a CentralAuditStore where they can be analyzed as part of evidence to determine the properties of a trust domain and alerts the DomainManagementAgent when certain critical events are observed.
We define messages as the main mechanisms for accessing services provided within the domain. Each message has a Sender and one or more Receivers; both defined as agents and are delivered through some MessageTransport mechanisms such as Remote Procedure Call (RPC) or Web Services, which is constrained by a DeliveryPolicy. The model defines a service as a Resource, which may be providedBy another Resource.

V. APPLICATION SCENARIOS
In this section of the paper, we attempt to define possible scenarios such as Health Care Scenario and ConfiChair Scenario. Then the developed taxonomy is applied to possible scenarios, in which the concept of trust domains could be useful.

A. Health Care Service Scenario
We consider a scenario, in which, a number of health institutions (i.e. hospitals or clinics) provide specialist services to their clients. Each service uses one or more databases to store data collected from interactions with their clients. The data collected maybe shared with other services, restricted to a particular service or isolated from other data stores within a single service. The specialist service providers are monitored for quality and financial purposes by Monitoring Services. For this reason, monitoring services may require access to certain parts of the data collected at a particular site. Furthermore, various monitoring services may need to share part of the data and keep some of it private.
We illustrate, in Figure 5, an example setup in which three specialist services share some parts of their data among themselves and two monitoring services that may access all or part of the data collected at each site. Each specialist service has a Demographics database and a number of other databases. In addition to the Demographics database, each service has the following other databases: Specialist Service 1 (SS1) has a Diagnostics database; Specialist Service 2 (SS2) has Scans and Treatment databases; and Specialist Service 3 (SS3) has Births database. Specialist Service 1 (SS1) and Specialist Service 2 (SS2) are monitored by Monitoring Service 1 (MS1) while Specialist Service 3 (SS3) is monitored by Monitoring Service 2 (MS2). Monitoring Service 1 (MS1) and Monitoring Service 2 (MS2), each have some databases where the monitoring data is stored.

Data Flow Relations & Trust Domains Application
In this section of the paper, we attempt to apply the developed taxonomy for trust domains to the above-mentioned scenario (in this case Health Care Service Scenario). This is achieved by identifying the boundaries that exist with respect to data flows within the setup. SS1 and SS3 have agreed on a policy to allow data to flow between their Demographics databases. SS3 has, additionally, agreed on a similar policy but with SS2. These results in two intersecting trust domains, SS1-SS3-Demo-TDom and SS2-SS3-Demo-TDom, are shown in Figure 6. According to the above-scenario, patients' demographics information (in these case "Assets") can only be shared between the blood bank and the cancer clinic. The maternity clinic has agreed on a similar policy but with only the cancer clinic. Therefore, this can be accomplished by defining policies, specifying the behaviour of entities within the trust domain and how data flows (in this case bi-directional) within the trust domain. Penalties or technical means (e.g. email warnings messages or systems generated warning messages) can be used to control activities that occur within the trust domain. These activities captured by monitoring the actions, can be used to produce evidence such as records of how demographics data came to be, audit logslog of events that occur during the lifetime of a trust domain. Therefore, these controls produce evidence to ensure that actions have been performed according the policies defined as represented in the trust domains taxonomy. Figure 6: Trust Domains involving demographics data flow SS1 and SS2 agree to share diagnostic and scan information between them. This agreement defines a bi-direction data flow between the two databases resulting in a trust domain, SS1-SS2-Findings-TDom, as shown in Figure 6. According to the above-scenario, patients' diagnostic and scan information (in this case "Assets") can only be shared between the blood bank and the cancer clinic. Therefore, this can also be accomplished by defining policies, specifying the behaviour of entities within the trust domain and how data flows (in this case bidirectional) within the trust domain. Penalties or technical means (e.g. email warnings messages or systems generated warning messages) can be used to control activities that occur within the trust domain. These activities captured by monitoring the actions can be used to produce evidence such as records of how diagnostic and scan information came to be, audit logs-log of events that occur during the lifetime of a trust domain. Therefore, these controls produce evidence to ensure that actions have been performed according to the policies defined as represented in the proposed trust domains taxonomy. SS3 has an internal one-directional data flow from the Demographics database to the Births database. Data is not allowed to flow in the other direction. This creates a trust domain, SS3-Internal-TDom, as shown in Figure 7(a). According to the above-scenario, patients' demographics and birth information (in these case "Assets") can only be accessible by the maternity clinic. Additionally, the data is only allowed to flow from the Demographic database to the Birth database. Therefore, this can be accomplished by defining policies, specifying the behaviour of entities within the trust domain and how data flows (in this case onedirectional) within the trust domain. Penalties or technical means (e.g. email warnings messages or systems generated warning messages) can be used to control activities that occur within the trust domain. These activities captured by monitoring the actions that can be used to produce evidence such as records of how demographics and birth information came to be, audit logs-log of events that occur during the lifetime of a trust domain. Therefore, these controls produce evidence to ensure that actions have been performed according the policies defined as represented in the proposed trust domains taxonomy. Finally, MS1 and MS2 have a mutual agreement to allow data to flow between their statistics databases, resulting in a trust domain, MS1-MS2-Stats-TDom illustrated in Figure 7 (b). According to the above-scenario, patients' birth as well as treatment and their visit statistics information (in this case "Assets") can only be shared between the two monitoring databases. Therefore, this can be accomplished by defining policies, specifying the behaviour of entities within the trust domain and how data flows (in this case bi-directional) within the trust domain. Penalties or technical means (e.g. email warnings messages or systems generated warning messages) can be used to control activities that occur within the trust domain. These activities captured by monitoring the actions that can be used to produce evidence such as records of how patients' birth as well as their treatment and visit statistics information came to be, audit logs-log of events that occur during the lifetime of a trust domain. Therefore, these controls produce evidence to ensure that actions have been performed according the policies defined as represented in the proposed trust domains taxonomy.
This case illustrates that trust domains will differ in their nature. Some will exist within a single entity such as an organisation or department while some will crossorganisational boundaries. Furthermore, some trust domains will place restrictions on the direction of data flow and others (e.g. the Demographics trust domains) will have some level of transitivity in their data flows. One thing common to all the domains, however, is that there is an agreement between the parties involved, which defines what data can flow within the domain as well as the direction of its flow. Other properties will definitely exist and this warranties the need for further investigations into the nature and expected properties of trust domains.

B. ConfiChair Scenario
ConfiChair is a cloud-based conference management protocol developed by researchers from Birmingham University [18]. It is proposed to address a set of privacy and confidentiality requirements for existing cloud-based conference management systems such as EasyChair or the Editor's Assistant (EDAS). In ConfiChair, as shown in Figure  8, authors, reviewers and the conference chair interact through their browsers with the cloud, to perform the usual tasks of uploading and downloading papers and reviews. It is noteworthy that privacy and the data confidentiality issues concern the cloud-based conference system administrator who administrates the system for all conferences (not the conference chair, who is concerned with a single conference). For example, the cloud based conference system administrator could accidentally or deliberately disclose some sensitive information (e.g. strongest and weakest reviewers) because the service provider has access to all the information related to conferences hosted by the conference management system.

Data Flow Relations & Trust Domains Application
In this section of the paper, we attempt to apply the developed taxonomy for trust domains to the above-mentioned scenario (in this case ConfiChair Scenario). This is achieved by identifying the boundaries that exist with respect to data flows within the setup.
According to the ConfiChair scenario shown in Figure 8, authors, reviewers, and the conference chair interact through their browsers with the cloud, to perform the usual tasks of uploading and downloading papers and reviews. Additionally, the conference system administrator is responsible of administrating the ConfiChair system for all conferences. Therefore, the role element in the taxonomy is represented in the ConfiChair scenario. Various roles carry some actions developed by the ConfiChair system. Author upload/download papers, reviewer reviews papers, conference chair referees the process and the conference system administrator administrates all conferences are a few examples. Therefore, the action element in the taxonomy is represented in the ConfiChari scenario. A set of policies developed within the ConfiChair system. For example, (a) a reviewer doesn't see other reviewers of a paper before writing her own, (b) the Conference System Administrator does not have access to the content of papers or reviews, or the numerical scores give by the reviewers to papers and (c) the conference system administrator does have access to the names of authors and the names of reviewers; however, he does not have ability to tell if a particular author was reviewed by a particular reviewer. The policy element in the taxonomy is, therefore, represented in the ConfiChair scenario. The following controls also developed in the ConfiChair system in order to monitor actions: (a) the login procedure implemented relies on each user having an identity id and a secret password pswid; (b) secrecy of papers, reviews and scores: conference system administrator does not have access to the content of papers or reviews, or the numerical scores given by reviewers to papers; (c) unlinkability of author-reviewer: conference system administrator does not have access to the names of authors and the names of reviewers and (d) however, all sensitive data (e.g. encryption of each review or score) is seen by the conference system administrator only in encrypted form. Therefore, the control element in the taxonomy is also represented in the ConfiChair scenario. The following evidence such as information produced by uploading/downloading papers, reviewing papers, refereeing the ConfiChair system process and also administrating all conferences through the ConfiChair system is maintained by the ConfiChair system. The evidence element in the taxonomy is therefore represented in the ConfiChari scenario. Finally, data such as authors, reviewers, conference chairs and conference system administrator information; paper upload/download/submission information; paper review information; and referee process information exist in digital form within the ConfiChair system. Therefore, the asset element in the taxonomy is represented in the ConfiChair scenario.
The author uploads/downloads (in this case bi-directional) papers and the reviewers reviews (in this case bi-directional) those uploaded papers using the ConfiChair system. Therefore, this creates a trust domain, Author-ConfiChairSystem-Reviewer. According to the ConfiChair scenario shown in Figure 8, the conference chair referees the process. This creates two trust domains, Author-ConfiChairSystem-ConferenceChair as well as ConferenceChair-ConfiChairSystem-Reviewer.
Finally, the conference system administrator is responsible of administrating the ConfiChair system for all conferences where the conference chair referees the conference process. The conference system administrator does not have access to the names of authors and the names of reviewers and as well as all sensitive data (e.g. encryption of each review or score) is seen by the conference system administrator only in encrypted form. Therefore, this creates a trust domain, ConferenceChair-ConfiChairSystem-ConferenceSystemAdministrator.

VI. CONCLUSIONS AND FUTURE WORK
This paper has presented a taxonomy for Trust Domains which enables individuals and organizations to securely collaborate across functions, geographies and corporate boundaries by providing collaborating parties (or participants) the means to create online environments designed to prevent information from leaking and where their resources can be shared as they specify. For this, Trust Domains relay on mechanisms ensuring that information flow to the right entities and that information is only accessed by entities capable of protecting it as specified. Furthermore, for future research, to help participants decide whether a trust domain is able to protect their resources, additional mechanisms enable participants to verify whether the entities composing the trust domain behaved and currently behave as required. By developing a mechanism to represent Trust Domains and reasons about their properties, we expect to support tool integration and decision-making based on the integration of evidence from disparate sources.


Finding Action Tubes

We address the problem of action detection in videos. Driven by the latest progress in object detection from 2D images, we build action models using rich feature hierarchies derived from shape and kinematic cues. We incorporate appearance and motion in two ways. First, starting from image region proposals we select those that are motion salient and thus are more likely to contain the action. This leads to a significant reduction in the number of regions being processed and allows for faster computations. Second, we extract spatio-temporal feature representations to build strong classifiers using Convolutional Neural Networks. We link our predictions to produce detections consistent in time, which we call action tubes. We show that our approach outperforms other techniques in the task of action detection.

Introduction
In object recognition, there are two traditional problems: whole image classification, "is there a chair in the image?", and object detection, "is there a chair and where is it in the image?". The two problems have been quantified by the PASCAL Visual Object Challenge [11,10] and more recently the ImageNet Challenge [8,7]. The focus has been on the object detection task due to its direct relationship to practical, real world applications. When we turn to the field of action recognition in videos, we find that most work is focused on video classification,"is there an action present in the video", with leading approaches [39,40,34] trying to classify the video as a whole. In this work, we address the problem of action detection, "is there an action and where is it in the video".
Our goal is to build models which can localize and classify actions in video. Inspired by the recent advances in the field of object detection in images [13], we start by selecting candidate regions and use convolutional networks to classify them. Motion is a valuable cue for action recognition and we utilize it in two ways. We use motion saliency to eliminate regions that are not likely to contain the action. This leads to a big reduction in the number of regions being processed and subsequently in compute time. Additionally, we incorporate kinematic cues to build powerful models for action detection. Figure 2 shows the design of our action models. Given a region, appearance and motion cues are used with the aid of convolutional neural networks (CNNs) to make a prediction. Our experiments indicate that appearance and motion are complementary sources of information and using both leads to significant improvement in performance (Section 4). Predictions from all the frames of the video are linked to produce consistent detections in time. We call the linked predictions in time action tubes. Figure 1 outlines our approach.
Our detection pipeline is inspired by the human vision system and, in particular, the two-streams hypothesis [14]. The ventral pathway ("what pathway") in the visual cortex responds to shape, color and texture while the dorsal pathway ("where pathway") responds to spatial transformations and movement. We use convolutional neural networks to computationally simulate the two pathways. The first network, spatial-CNN, operates on static cues and captures the appearance of the actor and the environment. The second network, motion-CNN, operates on motion cues and captures patterns of movement of the actor and the object (if any) involved in the action. Both networks are trained to discriminate between the actors and the background as well as between actors performing different actions.
We show results on the task of action detection on two publicly available datasets, that contain actions in real world scenarios, UCF Sports [32] and J-HMDB [17]. These are the only datasets suitable for this task, unlike the task of action classification, where more datasets and of bigger size (up to 1M videos) exist. Our approach outperforms all other approaches ( [15,41,37,25]) on UCF sports, with the biggest gain observed for high overlap thresholds. In particular, for an overlap threshold of 0.6 our approach shows a relative improvement of 87.3%, achieving mean AUC of 41.2% compared to 22.0% reported by [41]. On the larger J-HMDB, we present an ablation study and show the effect of each component when considered separately. Unfortunately, no other approaches report numbers on this dataset.
Additionally, we show that action tubes yield state-of-theart results on action classification on J-HMDB. Using our action detections we are able to achieve an accuracy of 62.5% on J-HMDB, compared to 56.6% reported by [39], the previous state-of-the-art approach on video classification.
The rest of the paper is organized as follows. In Section 2 we mention related work on action classification and action detection in videos. In Section 3 we describe the details of our approach. In Section 4 we report our results on the two datasets.

Related Work
There has been a fair amount of research on action recognition. We refer to [1,29,42] for recent surveys in the field. For the task of action classification, recent approaches use features based on shape (e.g. HOG [5], SIFT [28]) and motion (e.g. optical flow, MBH [6]) with high order encodings (e.g. Bag of Words, Fischer vectors) and train classifiers (e.g. SVM, decision forests) to make action predictions. More specifically, Laptev et al. [26] extract local features at spatio-temporal interest points which they encode using Bag of Words and train SVM classifiers. Wang et al. [39] use dense point trajectories, where features are extracted from regions which are being tracked using optical flow across the frames, instead of fixed locations on a grid space. Recently, the authors improved their approach [40] using camera motion to correct the trajectories. They estimate the camera movement by matching points between frames using shape and motion cues after discarding those that belong to the humans in the frame. The big relative improvement of their approach shows that camera motion has a significant impact on the final predictions, especially when dealing with real world video data. Jain et al. [16] make a similar observation.
Following the impressive results of deep architectures, such as CNNs, on the task of handwritten digit recognition [27] and more recently image classification [23] and object detection in images [13], attempts have been made to train deep networks for the task of action classification. Jhuang et al. [18] build a feedforward network which consists of a hierarchy of spatio-temporal feature detectors of predesigned motion and shape filters, inspired by the dorsal stream of the visual cortex. Taylor et al. [36] use convolutional gated RBMs to learn features for video data in an unsupervised manner and apply them for the task of action classification. More recently, Ji et al. [19] build 3D CNNs, where convolutions are performed in 3D feature maps from both spatial and temporal dimensions. Karpathy et al. [21] explore a variety of network architectures to tackle the task of action classification on 1M videos. They show that operating on single frames performs equally well than when considering sequences of frames. Simonyan & Zisserman [34] train two separate CNNs to explicitly capture spatial and temporal features. The spatial stream operates on the RGB image while the temporal stream on the optical flow signal. The two stream structure in our network for action detection is similar to their work, but the crucial difference is that their network is for full image classification while our system works on candidate regions and can thus localize the action. Also, the way we do temporal integration is quite different since our work tackles a different problem.
Approaches designed for the task of action classification use feature representations that discard any information regarding the location of the action. However, there are older approaches which are figure centric. Efros et al. [9] combine shape and motion features to build detectors suitable for action recognition at low resolution and predict the action using nearest neighbor techniques, but they assume that the actor has already been localized. Schüldt et al. [33] build local space-time features to recognize action patters using SVM classifiers. Blank et al. [3] use spatio-temporal volume silhouettes to describe an action assuming in addition known background. More recently, per-frame human detectors have been used. Prest et al. [30] propose to detect humans and objects and then model their interaction. Lan et al. [25] learn spatio-temporal models for actions using figure-centric visual word representation, where the location of the subject is treated as a latent variable and is inferred jointly with the action label. Raptis et al. [31] extract clusters of trajectories and group them to predict an action class using a graphical model. Tian et al. [37] extend the deformable parts model, introduced by [12] for object detection in 2D images, to video using HOG3D feature descriptors [22]. Ma et al. extract segments of the human body and its parts based on color cues, which they prune using motion and shape cues. These parts serve as regions of interest from which features are extracted and subsequently are encoded using Bag of Words. Jain et al. [15] produce space-time bounding boxes, starting from super-voxels, and use motion features with Bag of Words to classify the action within each candidate. Wang et al. [41] propose a unified approach to discover effective action parts using dynamical poselets and model their relations. Figure 1 outlines our approach. We classify region proposals using static and kinematic cues (stage a). The classifiers are comprised of two Convolutional Neural Networks (CNNs) which operate on the RGB and flow signal respectively. We make a prediction after using action specific SVM classifiers trained on the spatio-temporal representations produced by the two CNNs. We link the outputs of the classifiers across the frames of the videos (stage b) to produce action tubes.

Regions of interest
Given a frame, the number of possible regions that contain the action is enormous. However, the majority of these candidates are not descriptive and can be eliminated without loss in performance. There has been a lot of work on generating useful region proposals based on color, texture, edge cues ( [38,2]). We use selective search [38] on the RGB frames to generate approximately 2K regions per frame. Given that our task is to localize the actor, we discard the regions that are void of motion, using the optical flow signal. As a result, the final regions we consider are those that are salient in shape and motion.
Our motion saliency algorithm is extremely simple. We view the normalized magnitude of the optical flow signal f m as a heat map at the pixel level. If R is a region, then For α = 0.3, approximately 85% of boxes are discarded, with a loss of only 4% in recall on J-HMDB, for an overlap threshold of 0.5. Despite the small loss in recall, this step is of great importance regarding the algorithm's time complexity. To be precise, it takes approximately 11s to process an image with 2K boxes, with the majority of the time being consumed in extracting features for the boxes (for more details see [13]). This means that a video of 100 frames would require 18min to process! This is prohibitive, especially for a dataset of thousands of videos. Eliminating regions which are unlikely to contain the action reduces the compute time significantly.

Figure 2:
We use action specific SVM classifiers on spatiotemporal features. The features are extracted from the fc7 layer of two CNNs, spatial-CNN and motion-CNN, which were trained to detect actions using static and motion cues, respectively.

Action specific classifiers
We use discriminative action classifiers on spatiotemporal features to make predictions for each region. The features are extracted from the final layer of the CNNs which are trained to discriminate among different actions as well as between actions and the background. We use a linear SVM with hard negative mining to train the final classifiers. Figure 2 shows how spatial and motion cues are combined and fed into the SVM classifier.

CNNs for action detection
We train two Convolutional Neural Networks for the task of action detection. The first network, spatial-CNN, takes as input RGB frames and captures the appearance of the actor as well as cues from the scene. The second network, motion-CNN, operates on the optical flow signal and captures the movement of the actor. Spatio-temporal features are extracted by combining the output from the intermediate layers of the two networks. Action specific SVM classifiers are trained on the spatio-temporal features and are used to make predictions at the frame level. Figure 2 schematically outlines the procedure. Subsequently, we link the detections in time to produce temporarily consistent action predictions, which we call action tubes.
We train spatial-CNN and motion-CNN similar to R-CNN [13]. Regions of interest are computed at every frame of the video, as described above. At train time, the regions which overlap more than 50% with the ground truth are considered as positive examples, and the rest are negatives. The networks are carefully initialized to avoid overfitting.
The architecture of spatial-CNN and motion-CNN is identical and follows [23] and [43]. Assume C(k, n, s) is a convolutional layer with kernel size k × k, n filters and a stride of s, P (k, s) a max pooling layer of kernel size k × k and stride s, N a normalization layer, RL a rectified linear unit, F C(n) a fully connected layer with n filters and D(r) a dropout layer with dropout ratio r. The architecture of our networks follows: The final fully connected layer has number of outputs as many as the action classes plus one for the background class. During training a softmax loss layer is added at the end of the network.
Network details The architecture of our CNNs is inspired by two different network designs, [23] and [43]. Our network achieves 17% top-5 error on the ILSVRC-2012 validation set for the task of classification.
Weight initialization Proper initialization is a key for training CNNs, especially in the absence of data. spatial-CNN: We want spatial-CNN to accurately localize people performing actions in 2D frames. We initialize spatial-CNN with a model that was trained on the PASCAL VOC 2012 detection task, similar to [13]. This model has learned feature representations necessary for accurately detecting people under various appearance and occlusion patterns, as proven by the high person detection AP reported on the VOC2012 test set. motion-CNN: We want motion-CNN to capture motion patterns. We train a network on single frame optical flow images for the task of action classification. We use the UCF101 dataset (split 1) [35], which contains 13320 videos of 101 different actions. Our single frame optical flow model achieves an accuracy of 72.2% on split 1, similar to 73.9% reported by [34]. The 1.7% difference can be attributed to the differences in the network's architectures.
Indeed, the network used in [34] yields 13.5% top-5 error on the ILSVRC-2012 validation set, compared to the 17% top-5 error achieved by our network. This model is used to initialize motion-CNN when trained on smaller datasets, such as UCF Sports and J-HMDB.
Processing of input data We preprocess the input for each of the networks as follows spatial-CNN: The RGB frames are cropped to the bounds of the regions of interest, with a padding of 16 pixels, which is added in each dimension. The average RGB values are subtracted from the patches. During training, the patches are randomly cropped to 227 × 227 size, and are flipped horizontally with a probability of 0.5. motion-CNN: We compute the optical flow signal for each frame, according to [4]. We stack the flow in the x-, ydirection and the magnitude to form a 3-dimensional image, and scale it by a constant (s = 16). During training, the patches are randomly cropped and flipped.
Parameters We train spatial-CNN and motion-CNN with backpropagation, using Caffe [20]. We use a learning rate of 0.001, a momentum of 0.9 and a weight decay of 0.0005. We train the networks for 2K iterations. We observed more iterations were unnecessary, due to the good initialization of the networks.

Training action specific SVM classifiers
We train action specific SVM classifiers on spatio-temporal features, which are extracted from an intermediate layer of the two networks. More precisely, given a region R, let φ s (R) and φ m (R) be the feature vectors computed after the 7th fully connected layer in spatial-CNN and motion-CNN respectively. We combine the two feature vectors φ(R) = [φ s (R) T φ m (R) T ] T to obtain a spatio-temporal feature representation for R. We train SVM classifiers w α for each action α ∈ A, where ground truth regions for α are considered as positive examples and regions that overlap less than 0.3 with the ground truth as negative. During training, we use hard negative mining.
At test time, each region R is a associated with a score vector score(R) = {w T α φ(R) : α ∈ A}, where each entry is a measure of confidence that action α is performed within the region.

Linking action detections
Actions in videos are being performed over a period of time. Our approach makes decisions on a single frame level. In order to create temporally coherent detections, we link the results from our single frame approach into unified detections along time.
Assume two consecutive frames at times t and t + 1, respectively, and assume R t is a region at t and R t+1 at t + 1. For an action α, we define the linking score between those regions to be where ov(R,R) is the intersection-over-union of two regions R andR and λ is a scalar. In other words, two regions are strongly linked if their spatial extent significantly overlaps and if they score high under the action model.
For each action in the video, we seek the optimal path whereR α = [R 1 , R 2 , ..., R T ] is the sequence of linked regions for action α. We solve the above optimization problem using the Viterbi algorithm. After the optimal path is found, the regions inR * α are removed from the set of regions and Eq. 2 is solved again. This is repeated until the set of regions is empty. Each path from Eq. 2 is called an action tube. The score of an action tubeR α is defined as

Results
We evaluate our approach on two widely used datasets, namely UCF Sports [32] and J-HMDB [17]. On UCF sports we compare against other techniques and show substantial improvement from state-of-the-art approaches. We present an ablation study of our CNN-based approach and show results on action classification using our action tubes on J-HMDB, which is a substantially larger dataset than UCF Sports.
Datasets UCF Sports consists of 150 videos with 10 different actions. There are on average 10.3 videos per action for training, and 4.7 for testing 1 . J-HMDB contains about 900 videos of 21 different actions. The videos are extracted from the larger HMDB dataset [24], consisting of 51 actions. Contrary to J-HMDB, UCF Sports has been widely used by scientists for evaluation purposes. J-HMDB is more interesting and should receive much more attention than it has in the past.
Metrics. To quantify our results, we report Average-Precision at a frame level, frame-AP, and at the video level, video-AP. We also plot ROC curves and measure AUC, a metric commonly used by other approaches. None of the AP metrics have been used by other methods on this task. However, we feel they are informative and provide a direct link between the tasks of action detection and object detection in images. 1 The split was proposed by [25]   • frame-AP measures the area under the precision-recall curve of the detections for each frame (similar to the PASCAL VOC detection challenge [11]). A detection is correct if the intersection-over-union with the ground truth at that frame is greater than σ and the action label is correctly predicted.
• video-AP measures the area under the precision-recall curve of the action tubes predictions. A tube is correct if the mean per frame intersection-over-union with the ground truth across the frames of the video is greater than σ and the action label is correctly predicted.
• AUC measures the area under the ROC curve, a metric previously used on this task. An action tube is correct under the same conditions as in video-AP. Following [37], the ROC curve is plotted until a false positive rate of 0.6, while keeping the top-3 detections per class and per video. Consequently, the best possible AUC score is 60%.

Results on UCF sports
In Figure 3, we plot the ROC curve for σ = 0.2 (red). In Figure 4 we plot the average AUC for different values of σ. We plot the curves as produced by the recent state-ofthe-art approaches, Jain et al. [15], Wang et al. [41], Tian et al. [37] and Lan et al. [25]. Our approach outperforms all other techniques by a significant margin for all values of σ, showing the most improvement for high values of overlap, where other approaches tend to perform poorly. In particular, for σ = 0.6, our approach achieves an average AUC of 41.2% compared to 22.0% by [41]. Table 1 shows frame-AP (second row) and video-AP (third row) for an interestion-over-union threshold of σ = 0.5. Our approach achieves a mean AP of 68.1% at the    Table 1: AP on the UCF Sports dataset for an intersection-overunion threshold of σ = 0.5. frame-AP measures AP of the action detections at the frame level, while video-AP measures AP of the predicted action tubes.
frame level and 75.8% at the video level, with excellent performance for most categories. Running is the only action for which the action tubes fail to detect the actors (11.7 % video-AP) , even though our approach is able to localize them at the frame level (54.9% frame-AP). This is due to the fact that the test videos for Running contain multiple actors next to each other and our simple linking algorithm fails to consistently associate the detections with the correct actors, because of the proximity of the subjects and the presence of camera motion. In other words, the action tubes for Running contain the action but the detections do not always correspond to the same person. Indeed, if we make our evaluation agnostic to the instance, video-AP for Running is 83.8%. Tracking objects in a video is a very interesting but rather orthogonal problem to action localization and is beyond the scope of this work. Figure 7 shows examples of detected action tubes on UCF sports. Each block corresponds to a different video. The videos were selected from the test set. We show the highest scoring action tube for each video. Red boxes indicate the detections in the corresponding frames. The predicted label is overlaid.

Results on J-HMDB
We report frame-AP and video-AP for the 21 actions of J-HMDB. We present an ablation study of our approach by evaluating the performance of the two networks, spatial- CNN and motion-CNN. Table 2 shows the results for each method and for each action category.
As shown in the ablation study, it is apparent that the combination of spatial and motion-CNN performs significantly better for almost all actions. In addition, we can make some very useful observations. There are specific categories for which one signal matters more than the other. In particular, motion seems to be the most important for actions such as Clap, Climb Stairs, Sit, Stand and Swing Baseball, while appearance contributes more for actions such as Catch, Shoot Gun and Throw. Also, we notice that even though motion-CNN performs on average a bit worse than spatial-CNN at the frame level (24.3% vs. 27.0% respectively), it performs significantly better at the video level (45.7% vs. 37.9% respectively). This is due to the fact that the flow frames are not very informative when considered separately, however they produce a stronger overall prediction after the temporal smoothing provided by our linking algorithm. Figure 5 shows the AUC for different values of the intersection-over-union threshold, averaged over the three splits on J-HMDB. Unfortunately, comparison with other approaches is not possible on this dataset, since no other approaches report numbers or have source code available. Figure 8 shows examples of action tubes on J-HMDB. Each block corresponds to a different video. The videos are selected from the split 1 test set. We show the highest scoring action tube for each video. Red boxes indicate the detections in the corresponding frames. The predicted label is overlaid.
Action Classification Our approach is not limited to action detection. We can use the action tubes to predict an action label for the whole video. In particular, we can predict the label l for a video by picking the action with the maximum frame-AP (%) brush hair catch clap climb stairs golf jump kick ball pick pour pullup push run shoot ball shoot bow shoot gun sit stand swing baseball throw walk wave mAP spatial-CNN 55. 8 Table 2: Results and ablation study on J-HMDB (averaged over the three splits). We report frame-AP (top) and video-AP (bottom) for the spatial and motion component and their combination (full). The combination of the spatial-and motion-CNN performs significantly better under both metrics, showing the significance of static and motion cues for the task of action recognition. Figure 6: The confusion matrix on J-HMDB for the classification task, when using action tubes to predict a label for each video.
action tube score where S α (R) is the score of the action tubeR as defined by Eq. 2.
If we use Eq. 3 as the prediction, our approach yields an accuracy of 62.5%, averaged over the three splits of J-HMDB. Figure 6 shows the confusion matrix.
In order to show the impact of the action tubes in the above result, we adapt our approach for the task of action classification. We use spatial and motion-CNNs in a classification setting, where full frames are used as input instead of regions. The weights of the CNNs are initialized from networks trained on UCF 101 (split1) for the task of action classification. We average the class probabilities as produced by the softmax layers of the CNNs, instead of training SVM classifiers (We observed major overfitting problems when training SVM classifiers on top of the com-  Table 3: Classification accuracy on J-HMDB (averaged over the three splits). CNN (third column) shows the result of the weighted average of spatial and motion-CNN on the whole frames, while Action Tubes (fourth column) shows the result after using the scores of the predicted action tubes to make decisions for the video's label.
bined fc7 features). We average the outputs of spatial-and motion-CNNs, with weights 1/3 and 2/3 respectively, and pick the action label with the maximum score after averaging the frames of the videos. Note that our pipeline for classification is similar to [34]. This approach yields an accuracy of 56.5% averaged over the three splits of J-HMDB. This compares to 56.6% achieved by the state-of-the-art approach [39]. Table 3 summarizes the results for action classification on J-HMDB. It is quite evident that focusing on the actor is beneficial for the task of video classification, while a lot of information is being lost when the whole scene is analyzed in an orderless fashion.

Conclusions
We propose an approach to action detection using convolutional neural networks on static and kinematic cues. We experimentally show that our action models perform stateof-the-art on the task of action localization. From our ablation study it is evident that appearance and motion cues are complementary and their combination is mandatory for accurate predictions across the board.
However, there are two problems closely related to action detection that we did not tackle. One is, as we mention in Section 4, the problem of tracking. For example, in a track field it is important to recognize that the athletes are running but also to be able to follow each one throughout the race. For this problem to be addressed, we need compelling datasets that contain videos of multiple actors, unlike the existing ones where the focus is on one or two actors. Second, camera motion is a factor which we did not examine, despite strong evidence that it has a significant impact on performance [40,16]. Efforts to eliminate the effect of camera movement, such as the one proposed Figure 7: Examples from UCF Sports. Each block corresponds to a different video. We show the highest scoring action tube detected in the video. The red box indicates the region and the predicted label is overlaid. We show 4 frames from each video. The top example on the right shows the problem of tracking, while the 4th example on the right is a wrong prediction, with the true label being Skate Boarding. Figure 8: Examples from J-HMDB. Each block corresponds to a different video. We show the highest scoring action tube detected in the video. The red box indicates the region and the predicted label is overlaid. We show 4 frames from each video. The 2nd example on the left and the two bottom ones on the right are wrong predictions, with true labels being catch, sit and run respectively. by [40], might further improve our results.


s2fieldsofstudy:
Computer Science
Biology

text:
Proper Scoring and Sufficiency

Logarithmic score and information divergence appear in both information theory, statistics, statistical mechanics, and portfolio theory. We demonstrate that all these topics involve some kind of optimization that leads directly to the use of Bregman divergences. If a sufficiency condition is also fulfilled the Bregman divergence must be proportional to information divergence. The sufficiency condition has quite different consequences in the different areas of application, and often it is not fulfilled. Therefore the sufficiency condition can be used to explain when results from one area can be transferred directly from one area to another and when one will experience differences.

INTRODUCTION
The use of scoring rules has a long history in statistics. An early contribution was the idea of minimizing the sum of square deviations that dates back to Gauss and works perfectly for Gaussian distributions. In the 1920's Ramsay and de Finetti proved versions of the Dutch book theorem where determination of probability distributions were considered as dual problems to maximizing a payoff function. Later it was proved that any consistent inference corresponds to optimizing with respect to some payoff function. A more systematic study of scoring rules was given by McCarthy [1] and has recently been studied by Dawid, Lauritzen and Parry [2] where the notion of a local scoring rule has been extended. The basic result is that the only strictly local proper scoring rule is logarithmic score.
Thermodynamics is the study of concepts like heat, temperature and energy. A major objective is to extract as much energy from a system as possible. Concepts like entropy and free energy play a significant role. The idea in statistical mechanics is to view the macroscopic behavior of a thermodynamic system as a statistical consequence of the interaction between a lot of microscopic components where the interacting between the components are governed by very simple laws. Here the central limit theorem and large deviation theory play a major role. One of the main achievements is the formula for entropy as a logarithm of a probability.
One of the main purposes of information theory is to compress data so that data can be recovered exactly or approximately. One of the most important quantities was called entropy because it is calculated according to a formula that mimics the calculation of entropy in statistical mechanics. Another key concept in information theory is information divergence (KL-divergence) that was introduced by Kullback and Leibler in 1951 in a paper entitled information and sufficiency. The link from information theory back to statistical physics was developed by E.T. Jaynes via the maximum entropy principle. The link back to statistics is now well established [3,4,5].
The relation between information theory and gambling was established by Kelly [6]. Logarithmic terms appear because we are interested in the exponent in an exponential growth rate of of our wealth. Later Kelly's approach has been generalized to training of stocks although the relation to information theory is weaker [7].
Related quantities appear in statistics, statistical mechanics, information theory and finance, annd we are interested in a theory that describes when these relations are exact and when they just work by analogy. First we introduce some general results about optimization on convex sets. This part applies exactly to all the topics under consideration and lead to Bregman divergences. Secondly, we introduce a notion of sufficiency and show that this leads to information divergence and logarithmic score. This second step is not always applicable which explains when the different topics are really different.
Proofs of the theorems in this short paper can be found in an appendix that is part of the arXiv version of the paper.

STATE SPACE
The present notion of a state space is based on [8], and is mainly relevant for quantum systems.
Before we do anything we prepare our system. Let P denote the set of preparations. Let p 0 and p 1 denote two preparations. For t ∈ [0, 1] we define (1 − t) · p 0 + t · p 1 as the preparation obtained by preparing p 0 with probability 1−t and t with probability t. A measurement m is defined as an affine mapping of the set of preparations into a set of probability measures on some measurable space. Let M denote a set of feasible measurements. The state space S is defined as the set of preparations modulo measurements. Thus, if p 1 and p 2 are preparations then they represent the same state if m (p 1 ) = m (p 2 ) for any m ∈ M.
In statistics the state space equals the set of preparations and has the shape of a simplex. The symmetry group of a simplex is simply the group of permutations of the extreme points. In quantum theory the state space has the shape of the density matrices on a complex Hilbert space and the state space has a lot of symmetries that a simplex does not have. For simplicity we will assume that the state space is a finite dimensional convex compact space.

OPTIMIZATION
Let A denote a subset of the feasible measurements M such that a ∈ A maps S into a distribution on the real numbers i.e. a random variable. The elements of A may represent actions like the score of a statistical decision, the energy extracted by a certain interaction with the system, (minus) the length of a codeword of the next encoded input letter using a specific code book, or the revenue of using a certain portfolio. For each s ∈ S we define F (s) = sup a∈A E [a (s)]. We note that F is convex but F need not be strictly convex. We say that a sequence of actions (a n ) n is asymptotically optimal for the state s if E [a n (s)] → F (s) for n → ∞.
If the state is s 1 but one acts as if the state were s 2 one suffers a regret that equals the difference between what one achieves and what could have been achieved.
where the supremum is taken over all sequences (a n ) n that are asymptotically optimal over s 2 .
Proposition 2. The regret D F has the following properties: If the state space is finite dimensional and there exists a unique action a 2 such that F ( If unique optimal actions exists for any state then F is differentiable which implies that the regret can be written as a Bregman divergence in the following form In the context of forecasting and statistical scoring rules the use of Bregman divergences dates back to [9]. We note that D F1 (s 1 , s 2 ) = D F2 (s 1 , s 2 ) if and only if F 1 (s) − F 2 (s) is an affine function of s. If the state s 2 has the unique optimal action a 2 then so the function F can be reconstructed from D F except for an affine function of s 1 . The closure of the convex hull of the set of functions s → E [a (s)] is uniquely determined by the convex function F.

SUFFICIENCY
Let (s θ ) θ denote a family of states and let Φ denote a completely positive transformation S → T where S and T denote state spaces. Then Φ is said to be sufficient for (s θ ) θ if there exists a completely positive transformation We say that the regret D F on the state space S satisfies the sufficiency property if D F (Φ (s 1 ) , Φ (s 2 )) = D F (s 1 , s 2 ) for any completely positive transformation S → S that is sufficient for (s 1 , s 2 ) . The notion of sufficiency as a property of divergences was introduced in [10]. The crucial idea of restricting the attention to transformations of the state space into itself was introduced in [11].
Theorem 3. Assume that S is a state space. If the divergence D F satisfies the sufficiency property then for any state s and any completely positive transformation Φ : If the alphabet size is two the above condition on F is sufficient to conclude that Theorem 4. Assume that the state space S is a classical or quantum state space on three or more letters. If the regret D F satisfies the sufficiency property, then F is proportional to the entropy function and D F is proportional to information divergence (relative entropy).
This theorem can be proved via a numer of partial results as explained in the next section.

Statistics
Consider an experiment with X = {1, 2, . . . , } as sample space. A scoring rule f is defined as a function with domain X × M + 1 (X ) → R such that the score is f (x, Q) when the prediction was given by Q and x ∈ X has been observed. A scoring rule is proper if for any probability measure P ∈ M + 1 (X ) the score x∈X P (x) · f (x, Q) is minimal when Q = P. Lemma 7. On a finite space a Bregman divergence that satisfies the sufficiency condition gives a strictly local scoring rule.
The following theorem was given in [11] with a much longer proof.
Theorem 8. On a finite alphabet with at least three letters a Bregman divergence that satisfies the sufficiency condition is proportional to information divergence.
Proof. Since any strictly local proper scoring rule corresponds to separable divergence a divergence that is Bregman and satisfies sufficiency must also be separable. If the alphabet size is at least three the only separable divergences that are Bregman divergences are the ones proportional to information divergence [10].

Information theory
Let b 1 , b 2 , . . . , b n denote the letters of an alphabet and let (κ (b i )) denote the length of the codeword κ (b i ) according to some code book κ. If the code is uniquely decodable then 2 − (κ(bi)) ≤ 1. Note that (κ (b i )) is an integer. If only integer values of are allowed then h is piecewise linear and sufficiency is not fulfilled. If arbitrary real numbers are allowed then it obvious we get a proper local scoring rule.

Statistical mechanics
Statistical mechanics can be stated based on classical mechanics or quantum mechanics. For our purpose this makes no difference because Theorem 4 can be applied for both classical systems and quantum systems.
Proof of Theorem 4. If we restrict to any commutative subalgebra the divergence is proportional to information divergence as stated in Theorem 8 so that F is proportional to the entropy function H restricted to the sub-algebra. Any state generates a commutative sub-algebra so the function F is proportional to H on all states and the divergence is proportional to information divergence.
Assume that a heat bath of temperature T is given and that all the states are close to the state of the heat bath. An action a ∈ A is some interaction with the thermodynamic system that extracts some energy from the system. In thermodynamics the quantity F (s) = sup a∈A E [a (s)] is normally called the free energy. If the temperature is kept fixed under all interactions F is called Helmholtz free energy. Any sufficient transformation Φ for s 1 and s 2 is quasi-static and can be approximately realized by a physical process Ψ that is reversible in the thermodynamic sense of the word.
≤ a 2 (s 2 ) = a 2 (Ψ (Φ (s 2 ))) = (a 2 • Ψ) (Φ (s 2 )) ≤ a Φ(s2) (Φ (s 2 )) . (6) Hence a Φ(s2) = a 2 • Ψ so that The amount of extractable energy Ex is proportional to information divergence. The quotient between extractable energy and information divergence depends on the temperature and one may even define the absolute temperature via the formula where k = 1.381 · 10 −23 J /K is Boltzmann's constant. Equation (8) was derived already in [12] by a similar argument. According to Equation (8) any bit of information can be converted into an amount of energy! One may ask how this is related to the mixing paradox (a special case of Gibbs' paradox). Consider a container divided by a wall with a blue and a yellow gas on each side of the wall. The question is how much energy can be extracted by mixing the gasses?
We loose one bit of information about each molecule by mixing the gasses, but if the color is the only difference no energy can be extracted. This seems to be in conflict with Equation (8), but in this case different states cannot be converted into each other by reversible processes. For instance one cannot convert the blue gas into the yellow gas. To get around this problem one can restrict the set of preparations and one can restrict the set of measurements. For instance one may simply ignore measurements of the color of the gas. What should be taken into account and what should be ignored, can only be answered by an experienced physicist. Formally this solves the mixing paradox but from a practical point of view nothing has been solved. If for instance the molecules in one of the gasses are much larger than the molecules in the other gas then a semi-permeable membrane can be used to create an osmotic pressure that can be used to extract some energy. It is still an open question which differences in properties of the two gasses that can be used to extract energy.

Portfolio theory
Let X 1 , X 2 , . . . , X k denote price relatives for a list of stocks. For instance X 5 = 1.04 means that stock no. 5 increases its value by 4 %. A portfolio is a probability vector b = (b 1 , b 2 , . . . , b k ) where for instance b 5 = 0.3 means that 30 % of your money is invested in stock no. 5. The total price relative is X 1 ·b 1 +X 2 ·b 2 +· · ·+X k ·b k = X · b. We now consider a situation where the stocks are traded once every day. For a sequence of price relative vectors X 1 , X 2 , . . . X n and a constant re-balancing portfolio b the wealth after n days is According to law of large numbers Here E log X, b is proportional to the doubling rate and is denoted W b, P where P indicates the probability distribution of X. Our goal to maximize W b, P by choosing an appropriate portfolio b. Let b P denote the portfolio that is optimal for P . As proved in [7] Theorem 9. The Bregman divergence satisfies the equation if and only if the measure P on k distinct vectors of the form (a 1 , 0, 0, . . . 0), (0, a 2 , 0, . . . 0) , until (0, 0, . . . a k ) .

CONCLUSION
On the level of optimization the theory works out in exactly the same way in statistics, information theory, statistical mechanics, and portfolio theory. The sufficiency condition is more complicated to apply. It requires that we restrict to a certain class of mappings of the state space into itself. In the case where the state space can be identified with a set of density matrices one should restrict to completely positive maps. In case the state space has a different structure it is not obvious which mappings one should restrict to. The basic problem is that we have to introduce a notion of tensor product for convex sets and it is not obvious how to do this, but this will be the topic of further investigations and results on this topic may have some impact on our general understanding of quantum theory. The original paper of Kullback and Leibler [13] was called "On Information and Sufficiency". In the present paper we have made the relation between information divergence and the notion of sufficiency more explicit. The idea of sufficiency has different consequences in different applications but in all cases information divergence prove to be the quantity that convert the general notion of sufficiency into a number. For specific applications one cannot identify the sufficient variables without studying the specific application in detail. For problems like the the mixing paradox there is still no simple answer to the question about what the sufficient variables are, but if the sufficient variables have been specified we have the mathematical framework to develop the rest of the theory in a consistent manner.

Proof of Theorem 5
If f is given in terms of a regret function D F then Assume that f is proper. Then we may define a divergence by (15) Since f is assumed to be proper D (P, Q) ≥ 0 with equality if and only if P = Q. The equality t i ·D F (P i , Q) = t i · D F P i ,P + D F P , Q follows by straight forward calculations. With these two results we see that D equals a Bregman divergence D F and that

Proof of Lemma 7
Let D F denote a regret function that satisfies the sufficiency condition. Then where we have made a cyclic permutation of indices. Next we use the sufficient transformation that projects a mixture of δ 1 and a uniform distribution.
Note that the projection can be obtained by taking a mixture of all permutations of the extreme points that leave the first extreme point unchanged. Hence the scoring rule is given by the local scoring rule

Proof of Theorem 9
We have Next we use Jensen's inequality to get is constant P -almost surely. Equivalently dQ dP is proportional to X, b Q X, b P for any probability measure Q on the support of X. The set of vectors b Q lie in a k − 1 dimensional convex set. Therefore the set of probability measures on the support of P is at most k − 1 dimensional. Hence P is supported on at most k vectors in R k 0,+ . The inequalitŷ holds with equality if b Q