Security of OS-level virtualization technologies: Technical report

The need for flexible, low-overhead virtualization is evident on many fronts ranging from high-density cloud servers to mobile devices. During the past decade OS-level virtualization has emerged as a new, efficient approach for virtualization, with implementations in multiple different Unix-based systems. Despite its popularity, there has been no systematic study of OS-level virtualization from the point of view of security. In this report, we conduct a comparative study of several OS-level virtualization systems, discuss their security and identify some gaps in current solutions.

Introduction
During the past couple of decades the use of different virtualization technologies has been on a steady rise. Since IBM CP-40 [19], the first virtual machine prototype in 1966, many different types of virtualization and their uses have been actively explored both by the research community and by the industry. A relatively recent approach, which is becoming increasingly popular due to its light-weight nature, is Operating System-Level Virtualization, where a number of distinct user space instances, often referred to as containers, are run on top of a shared operating system kernel. A fundamental difference between OS-level virtualization and more established competitors, such as Xen hypervisor [24], VMWare [48] and Linux Kernel Virtual Machine [29] (KVM), is that in OS-level virtualization, the virtualized artifacts are global kernel resources, as opposed to hardware. This allows multiple virtual environments to share a common host kernel and utilize underlying OS interfaces. As a result, OS-level virtualization incurs less CPU, memory and networking overhead, which is important not only for High Performance Computing (HPC), such as dense cloud configurations, but also for resource constrained environments such as mobile and embedded devices. The main disadvantage of OS-level virtualization is that each container can only contain a system of the same type as the host environment, e.g. Linux guests on a Linux host.
An important factor to take into account in the evaluation of the effectiveness of any virtualization technology is the level of isolation it provides. In the context of OS-level virtualization isolation can be defined as separation between containers, as well as the separation between containers and the host. In order to systematically compare the level of isolation provided by different OS-level virtualization solutions, one first needs to establish a common system model.
The goal of this study is to propose a generic model for a typical OS-level virtualization setup, identify its security requirements, and compare a selection of OS-level virtualization solutions with respect to this model. While other technologies as HW supported secure storage, various encryption primitives and specific CPU/memory features can enhance the security of OS-level virtualization solutions, they are left out of the scope of this paper and present the potential future work. To the best of our knowledge this is the first study of this kind that focuses on the security aspects of OS-level virtualization technologies. We base our analysis on information collected from the documentation and/or wherever possible the source code of the respective systems. As a result of this comparison section 9 identifies a number of gaps in the current implementation of Linux OS-level virtualization solutions.

Usage Scenarios
We identify the following common usage scenarios as motivation for OS-level virtualization in general. The first three originate from use cases in the context of warehouse scale computing. The latter two stem from security needs.
In Server consolidation, a set of distinct physical servers are substituted with a single physical server running a number of distinct virtual environments. Solutions based on hardware virtualization often require that the guest OS be modified; either to support the virtualization solution itself (as in the case of paravirtualization) or to facilitate interaction between the guest and host OSs by installing special-purpose components into the guest (as in full virtualization solutions such as VMWare, Virtual Box etc.) [47]. In contrast, one of the goals for OS-level virtualization is to provide a set of tools integrated into the OS to allow the creation and management of virtual environment without modifications to the software components placed inside a container. In Virtual Private Server (VPS) and cloud computing environments, where service providers grant superuser-level access in the rented virtual environments to customers, strict isolation between environments of different customers and the hosting provider is important unlike in server consolidation where all virtual environments are managed by the same entity.
Resource and application state management emerged from the need to run a number of distinct applications or multiple instances of a single application which require access to the same resources on a single machine, e.g. binding to the same network port. In addition by placing an application into a self-contained compartment it is possible to provide Checkpoint and Restart (CR) functionality [17,47,30]. CR allows processes to be moved between different physical or virtual environments. This can be useful for load-balancing or in high-availability environments, as well as software development and testing on different UNIX platforms.
A Multi-OS experience allows end-users the ability to use applications and services from different operating systems on the same device by the means of virtualization technology. While the OS-level virtualization is limited to systems sharing a common kernel, it can provide a way for the user to run a number of different OS variants on the same system. Since there are many new mobile operating systems on the rise such as Android, Tizen, FirefoxOS and the like, feature is likely to be useful for many experienced users. The need to share certain data, like the user's contacts or calendar, across the different OSs installed on the same device brings in an additional challenge for this use case.
Application or service isolation places critical and externally exposed services into separate sandbox environments that are able to contain damage in case sandboxed services become compromised. Sandboxing also makes it possible to delegate the administration of these services to third, possibly less trusted, parties [28].
The Bring Your Own Device (BYOD) policy [37] allows one physical device to be used simultaneously for personal and business needs resulting in a need of rigid separation between these two environments in order to guarantee user privacy while conforming to enterprise policies. Presenting separate environments to the end-user can also improve the usability of the solution [2], compared to domain separation by means of access control mechanisms alone.

System model
In Figure 1(a) we present a system model for a typical container setup that can support the types of usage scenarios we discussed in Section 2. There are a number of containers C 1 . . . C n that run on a single physical host machine. The OS kernel is shared among all the containers, but the extent of shared host user space depends on a concrete setup (see Table 1): Full OS installation & management corresponds to the most common case when the host user space layer comprises a complete OS installation with the container management layer on top. In this case some host resources may be shared between the host and one or more containers via bind-mounts [25] or overlay filesystems [20]. Each container can be one of two types: -Application containers have a single application or service instance running inside. They are commonly used for application isolation or resource management referred to in Section 2. -System containers have an entire OS user space installation and are commonly used for server consolidation.
Lightweight management corresponds to the case where the host user space layer consists of merely a light-weight management layer used to initialize and run containers. This setup can be argued to be more secure, as it exhibits a reduced attack surface compared to a complete underlying host system. Again, each container can be one of two types: -Direct application/service setup refers to the case when only a single application or service is installed in the container. It is more suitable for application isolation scenarios in which, for instance, a banking application is run in a separate container isolated from the rest of a less trusted OS running in another container.
-Direct OS setup refers to the case when a container runs an entire OS user space installation. It can provide an end-user the appearance of simultaneously running multiple OS instances, and is therefore well suited for Multi-OS and BYOD environments.   The system model described above intentionally omits cases where containers C 1 . . . C n are not independent, but arranged in a hierarchical structure. While some systems, such as FreeBSD jails [28], allow such setups, they are rarely used in practice and are therefore left beyond the scope of this report.

Attacker model and security requirements
The attacker is assumed to have full control over a certain subsetC of containers. The remaining set C is assumed to be in the control of legitimate users. The goals of the attacker can be classified as follows: -Container compromise: compromise C k ∈ C by means of illegitimate data access, Man-in-the-Middle (MitM) attacks or by affecting the control flow of instructions executed in C k ∈ C. -Denial of Service: disturb normal operation of the host or C k ∈ C.
-Privilege escalation: obtain a privilege not originally granted to a C j ∈C.
The above goals can be achieved via different types of attacks that can be roughly classified into distinct groups based on the interfaces available in a typical Single UNIX Specification compliant OS [45]. These attack groups can be further arranged into two classes based on the type of underlying mechanism: attacks via communication mechanisms and attacks via storage mechanisms (see Figure 1(b)). From this classification we derive a set of security requirements that each OS-level virtualization solution needs to fulfill. In the description below, numbers in parenthesis refer to arrows in Figure 1 Separation of processes is a fundamental requirement that aims to isolate processes running in distinct containers to prevent C j ∈C from influencing C k ∈ C using interfaces provided by the operating system for process management, such as signals and interrupts (1a). In addition, it might be possible to directly access the memory of a process running in C k ∈ C by using special system calls, e.g. the ptrace() system call allows a debugger process to attach and monitor the memory of a debugged process (1b).
Filesystem isolation is required in order to prevent illegitimate access to filesystem objects belonging to C k ∈ C or the host (2).
Device isolation should protect device drivers shared between different containers and a host. Such drivers present another significant attack vector because they expose interfaces (3) to code running into the kernel space, which may be abused to gain illegitimate data access, escalate privileges or mount other attacks.
IPC isolation is needed in order to prevent C j ∈C from accessing or modifying data belonging to C k ∈ C being transmitted over different IPC channels (4). Such channels include traditional System V IPC primitives, such as semaphores, shared memory and message queues as well as POSIX message queues.
Network isolation aims to prevent attacks by C j ∈C via available network interfaces (5). In particular, an attacker can attempt to eavesdrop on or modify network traffic of the host or C j ∈C, perform MitM attacks etc.
Resource management provides a way to limit the amount of resources available to each container depending on the system load. This is needed in order to prevent an attacker from exhausting physical resources available on a device, such as disk space or disk I/O limits, CPU cycles, network bandwidth and memory (6).

Terminology
We define the following terms that are going to be used through the remaining of this report and that might be not familiar for a reader without a Unix/Linux background: -Kernel space refers to the part of the virtual memory that is used to run the OS kernel code, modules, drivers and extensions. -User space denotes the part of the virtual memory that includes the user applications, system processes (daemons) and services. -Kernel resource is a kernel structure referring to shared physical or virtual devices, system resources, such as memory or cpu time, or a set of identifiers used through the kernel. -Superuser/privileged user is a special user in UNIX-like systems that is allowed to perform privileged system operations. UNIX classical root user is an example of such user. -Linux capabilities is a set of predefined capabilities implemented in the Linux kernel for performing different privileged operations, such as mounting a filesystem or overwriting the system security policies. Such capabilities are often referred as POSIX capabilities. -Filesystem root denotes the top-most directory in the UNIX filesystem hierarchy as it is visible to running processes. -Upstream/mainline Linux kernel refers to the Linux kernel source code tree maintained at kernel.org. This is the official Linux kernel source that contains all the released and upcoming features.
6 Overview of technologies

FreeBSD Jails
The pioneering notion of Jails was first introduced in FreeBSD 4.0 in 2000 [28]. The motivation behind Jails was the need to have separate virtual compartments on a single host, combined with the ability to delegate a subset of the traditional superuser privileges to the root user for each compartment. A number of changes were introduced to the FreeBSD kernel in order to implement Jails. These include the hardening of the chroot(2) system call, basic isolation that would restrict a jailed process from communicating with processes outside the Jail, the ability to limit the visibility of processes via the procfs pseudo filesystem and sysctl interfaces, restrictions on TCP/IP networking, Jail-aware device drivers, and the ability to restrict a root user inside a Jail from performing certain system calls. Later on the ability to have multiple IP-addresses per Jail, more powerful Jail management facilities and the ability to create hierarchical Jails were added.

Linux-VServer
The need for a mechanism similar to FreeBSD Jails in Linux led to the Linux-VServer project [8]. The first official release of the Linux-VServer occurred in 2003. In Linux-VServer, separate virtual environments are referred to as Virtual Private Servers (VPSs). They can be managed with the help of user space tools provided by the util-vserver package. Each VPS has its own context that contains all the information regarding the VPS; its name, allowed limits, bounded capabilities, scheduler information etc. In addition, the behavior of each VPS can be further adjusted by specifying context capabilities and flags that allow a VPS to modify its host name or hide network interfaces that a certain VPS is not permitted to access. The biggest downside with regards to deployability is the need to apply the Linux-VServer patches to kernel source code and recompile the kernel as the Linux-VServer changes are currently not integrated into the mainline Linux kernel development branch.

Solaris Zones
Solaris Zones/Containers project [35] was started in 2004 in order to provide a commercial OS-level virtualization solution. Sun engineers analyzed FreeBSD Jails and Linux-VServer solutions available at the time and concluded, that while the goals of the projects are similar, the depth of OS integration, quality of administrative tools and overall maturity of the aforementioned projects were not at a level needed to support commercial solutions. In addition, they also wanted to create a set of usable zone management tools that would enable the delegation of zone setup and configuration whenever possible to the administrators of a zone. The isolation provided by Solaris Zones is based on attaching a zone identifier to a process, and using it to restrict the visibility of the process across zone boundaries. The zone identifier is also used to determine process privileges inside non-global zones, System V IPC communication etc. Resource management is implemented using standard mechanisms provided in Solaris, such as entitlements, limits and partitions. When used together, they are able to ensure a minimal level of service, bound resource consumption and even the dedication of certain resources only to specific zones.

OpenVZ
The OpenVZ project [11] is another open source OS-level virtualization solution for Linux begun in 2005. It is currently part of the commercial Parallels Cloud Server solution [12]. OpenVZ uses the term Virtual Environments (VEs) [47] to refer to containers. The implementation consists of set of kernel changes and user space tools. The OpenVZ kernel is based on the Red Hat Enterprise Linux kernel, which in turn is based on the relatively old 2.6.32 upstream Linux kernel. However, OpenVZ developers have integrated many of their kernel modifications into the upstream kernel. Hence, the project's main user space tool, vzctl, can be used with both upstream and OpenVZ kernels, but the developers "recommend using the OpenVZ kernel for security, stability and features". The OpenVZ project was also the first one to implement the Checkpoint and Restart (CR) functionality for for VEs [30]. CR allows processes to be moved between different physical or virtual environments. This can be useful for load-balancing or in high-availability environments, as well as software development and testing on different UNIX platforms.

LxC
The Linux Containers (LxC) project [9] is the only currently available OS-level virtualization solution for Linux that consists only of a set of user space tools. This is possible because LxC utilizes only those virtualization features integrated into the upstream Linux kernel. This gives LxC an advantage over other Linuxbased projects, because the process of applying patches and compiling a specific version of the Linux kernel can become a non-trivial task even for experienced Linux users. Another differentiating feature of LxC is the ability to use Linux Security Modules (LSMs) [41] to harden a container setup. Apparmor [1] and SELinux [38] profiles are officially supported, but in principle, other existing LSMs, such as Smack [15] could be used as well.

Cells/Cellrox
The Cells architecture [22] and the commercial Cellrox solution [2] built on the Cells architecture are the only open source OS-level virtualization solutions for smartphones. The primary design goal behind Cells is to support the Bring Your Own Device (BYOD) policy [37] on the Android platform. BYOD allows one physical device to be used simultaneously for personal and business needs resulting in a need of rigid separation between these two environments in order to guarantee user privacy while conforming to enterprise policies. Cells allows a user to have two or more virtual phones on the single Android device, e.g. one for personal use and another, which can be controlled by the user's employer and can contain confidential company data and applications. The user can switch between the virtual phones using a special icon on the Android home screen. Similarly to LxC, Cells utilizes upstream kernel features to isolate virtual phones. However, since Android has some non-standard Linux extensions, the developers of Cells had to implement a number of additional isolation mechanisms. The primary example are the changes done to the Binder driver in order to support IPC isolation on Android.

Comparison
Following [17], we define the notion of a kernel namespace as a set of identifiers representing a class of global kernel resources, such as process and user ids, IPC objects or filesystem mounts. The OS-level virtualization in the upstream Linux kernel is based on the usage of different kernel namespaces. The subsections below introduce the relevant namespaces and compare selected OS-level virtualization solutions, highlighted in bold in the previous paragraph, based on the security requirements listed in section 4.

Separation of processes
The primary isolation mechanism required from any OS-level virtualization solution is that it is able to distinguish processes running in different containers from those running on the host, limit cross-container process visibility and to prevent memory and signaling-level attacks described in the section 4. The simplest solution to this problem is to embed a container identifier C i into the process data structure and to check the scope and the permissions of all syscall invocations. FreeBSD Jails, Solaris Zones, OpenVZ and Linux-VServer implementations follow this approach by linking a structure describing the container to the process data structure. However, unlike FreeBSD and Solaris, the data structures describing OpenVZ and Linux-VServer containers are not used to achieve process separation. They only store related container data such as resource limits and capabilities. Instead, OpenVZ, Linux-VServer, LxC and Cells use process id (pid) namespaces that are part of the mainline Linux kernel. A pid namespace is a mechanism to group processes in order to control their ability to see (for example via proc pseudo-filesystem) and interact (for example by sending signals) with one another. The pid namespaces also provide pid virtualization: two processes in different pid namespaces may have the same pid.
Having a separate structure describing a container and storing a pointer to it in the process task structure is a convenient way to have all the relevant information concerning the container in one place. However, the upstream Linux kernel has followed a different approach of grouping different kernel resources into separate namespaces and using these namespaces to build containers. This approach incurs additional complexity, but adds the flexibility to choose a combination of namespaces that best fits the desired use case. It also allows gradual introduction of namespaces to an existing system, like the upstream Linux kernel, which also helps in testing and verification of the implementation [17]. Furthermore, it avoids information duplication when both the process and the container structures have similar information. The pros and cons of these two approaches are summarized in Table 2.
In addition to the ability to isolate and virtualize process ids, the upstream Linux kernel also allows virtualization and isolation of the user and group identifiers with the help of user namespaces. Typically the root user has all the privileges to perform various system administration tasks and is able to override all access control restrictions. However, it is not desired that a root user running inside a container would be given the privileges of the host root user. Therefore, the Linux user namespace implementation interprets a given Linux capability as authorizing an action within that namespace: for example, the CAP SYS BOOT capability inside a container grants the authority to reboot that container and not the host. Moreover, many capabilities such as CAP SYS MODULE cannot be safely granted for container in any meaningful manner. When a process attempts to perform an action guarded by such capability, the kernel always checks if the process possesses this capability in the host user namespace. All Linux OS-level virtualization solutions support the option of starting a new user namespace for each container, but all the related configuration such as mapping the user identifiers between the host and the container must be done manually.

Filesystem isolation
The filesystem is one of the most important OS interfaces that allows processes to store and share data as well as to interact with one another. In order to prevent filesystem-based attacks described in section 4, it should be possible to isolate the filesystem between containers and to minimize the sharing of the data. The amount of sharing needed between the host and each container depends on the usage scenario. In the case of application isolation, it is not worthwhile to completely duplicate the OS setup inside a container and therefore some parts of the filesystem, such as common libraries, need to be securely shared with the host. On the other hand in the case of server consolidation, quite often it is best to completely separate the filesystems and create container filesystems from scratch.
All Linux-based OS virtualization solutions utilize a mount namespace that allows separation of mounts between the containers and the host. The design of upstream Linux mount namespaces [17] has been influenced by private namespaces [34] in Plan 9 from Bell Labs [33]. Namespaces in Plan 9 are file-orientated, and the principal purpose is to facilitate the customization of the environment visible to users and processes. Since all Linux based systems create each container within a new mount namespace, all the internal mount events are only effective inside the given container. However, it is important to underline that the mount namespace by itself is not a security measure. Running a container in a separate mount namespace does not give any additional guarantees concerning the data isolation between the containers since containers inherit the view of filesystem mounts from their parent and thus are able to access all parts of the filesystem similarly.
A typical approach for process filesystem access containment is by using the chroot() system call where process is bound within a subtree of the filesystem hierarchy. If desired, resources may be shared with the host by mounting them within the subtree visible inside the container. Since the chroot() system call [7] only affects pathname resolution, privileged processes (i.e. processes with the CAP SYS CHROOT privilege) can escape the chroot jail. This can be done for example by changing the root directory again via chroot() to a subdirectory relative to their current working directory. Of the virtualization solutions under comparison, only Cells relies on chroot() alone. Some systems, such as Linux-VServer utilize a Secure chroot barrier [8] to prevent processes in a VPS from escaping the modified environment.
Another approach, utilized by for instance LxC, is to not only modify the root directory for processes in a container, but modify the root filesystem as well. This can be achieved with the Linux specific pivot root() system call [7], which is typically used during boot to change from a temporary root filesystem (e.g. an initrd) to the actual root filesystem. As its name suggests, the pivot root() system call moves the mountpoint of the old root filesystem to a directory under the new root filesystem, and puts the new root filesystem at its place. When done inside a mount namespace, the old root filesystem can be unmounted, thus rendering the host root filesystem inaccessible for processes inside the container, without affecting processes belonging to the root mount namespace on the host system. At the time of writing, the implementation of pivot root() also changes the root directory and current working directory of the process to the mountpoint of the new root filesystem if they point to the old root directory. OpenVZ relies on this behavior and uses the pivot root() system call alone. However, as the behavior with regards to the current root directory and the current working directory remains unspecified, proper usage dictates that the caller of pivot root() must ensure that processes with root directory or current working directory at the old root operate correctly regardless of the behavior of pivot root(). To ensure this, LxC changes the root directory and current working directory to the mountpoint of the new root before invoking pivot root().
FreeBSD and Solaris also provide a sandbox-like environment for each jail/zone using similar chroot()-like calls that are claimed to avoid above mentioned security vulnerabilities [28], [35]. Mounting and unmounting of filesystems is prohibited by default for a process running inside a jail unless different allow.mount.* options are specified.
A separate user namespace per container can further strengthen the filesystem isolation by mapping the user and group ids to a less privileged range of host uids and groups. Together with a mount namespace and a pivot root environment it strengthens protection against filesystem-based attacks described in 4.

Device isolation
In Unix, device nodes are special files that provide an interface to the host device drivers. In classical Unix configurations, the device nodes are separated from the rest of the filesystem and their inodes are placed in the /dev directory. In the case of Linux, this task is usually performed by the udevd daemon process issuing the mknod system call upon receiving the event from the kernel. Device nodes are security-sensitive since an improperly exposed or shared device inside a container can lead to a number of easy attacks (see section 4). In the simplest example, if a container has an access to /dev/kmem and /dev/mem nodes, it is able to read and write all the memory of the host. Thus, in order to isolate containers from one another it is important to prevent containers from creating new device nodes and to make sure that containers are only allowed to access a "safe" set of devices listed below: 1. Purely virtual devices, such as pseudo-terminals and virtual network interfaces. The security guarantee comes from the fact that these devices are explicitly created for each container and not shared. 2. Stateless devices, such as random, null and others. Sharing these devices among all containers and the host is safe because they are stateless. 3. User namespace-aware devices. If a device supports verifying process capabilities in the corresponding user namespace, then it is safe to expose such device to a container, because the specified limitations will be enforced. The current 3.14-rc2 upstream kernel does not have any physical devices supporting this feature, but they are expected to appear in the future.
All compared systems allow the system administrator to define a unique set of device nodes for each container and by default create only a small set of stateless and virtual devices. In Linux, creation of new device nodes within containers can be controlled by limiting access to the CAP SYS MKNOD Linux capability and by ensuring that all mountpoints inside containers have the nodev flag set.
The biggest difference of the Cells implementation is the addition of a "device namespace" that attempts to make the Linux input/output devices namespaceaware. Cells assumes the host to have a single set of input/output devices and multiplexes access to the physical host device via virtual devices created in each container. One virtual device at a time is allowed to access physical devices, based on whether an application from a given container is "on the foreground" (ie. visible on the screen) or not. Security-wise such an exclusive-access solution is comparable to the "purely virtual" devices category mentioned above and can be considered safe.
As mentioned above, Linux device drivers controlling physical devices are currently not namespace-aware and thus cannot be securely used inside containers. Quite commonly these devices assume only one controlling master host and require privileges that are hard to grant for a unprivileged container securely (unless the device is used exclusively by a single container). In other words, namespace support inside the device drivers would require extensive modifications to the existing driver code base.

IPC isolation
In order to achieve IPC isolation between containers, processes must be restricted to communicate via certain IPC primitives only within their own container. If the filesystem isolation is done correctly (see section 7.2), then filesystem-based IPC mechanisms (such as UNIX domain sockets and named pipes) are automatically isolated because the processes are not able to access filesystem paths outside of their own container. However, the isolation of the rest of the IPC objects (such as System V IPC objects and POSIX message queues) requires additional mechanisms. In Linux these IPC objects are isolated with the help of the IPC namespaces that allow the creation of a completely disjoint set of IPC objects. Linux-VServer, OpenVZ, LxC and Cells all spawn a new IPC namespace for each container in order to achieve the required isolation.
In addition to using IPC namespaces, Cells also has to implement namespace support for the Binder system since it is the primary IPC mechanism on the Android OS. The solution [10] includes having a separate Context Manager for each IPC namespace that is able to resolve Binder addresses only in that namespace and therefore provide isolation of Binder addresses between different containers.
Solaris Zones follow a different approach to isolate IPC objects that are not filesystem path-based. A zone ID is attached to each object based on the zone ID of the process that creates it, and processes are not able to access objects from other zones. An exception is made only for an administrator in the global zone that can access and manage all the objects. FreeBSD simply blocks SysV IPC object-related system calls if such calls are issued from within a jail. The allow.sysvipc option allows SysV IPC mechanisms for jailed processes but lacks any isolation between jails.

Network isolation
The main goal of network isolation is to prevent network-based attacks described in section 4. Moreover, in order to fulfill the server consolidation and resource management use cases, it also needs to provide a virtualized view of the network stack.
Network isolation methods differ in terms of the OSI layer of the TCP/IP stack where the isolation is implemented (see Table 3 for a comparison between these implementations). FreeBSD and Linux-VServer implement network isolation on Layer 3 with the help of bind filtering. They restrict a bind() call made from within a container to a set of specified IP addresses and therefore processes are only allowed to send and receive packets to/from these addresses. The benefit of such an approach is the small amount of code that needs to be modified in the network implementation and a minimal performance overhead. However, the downside is that a lot of the standard networking functionality is not accessible for a process inside a container such as obtaining an address from the Dynamic Host configuration Protocol (DHCP), acting as a DHCP server or the usage of routing tables.
Another approach, supported by Solaris Zones and OpenVZ, provides a Layer 3 virtualized network interface (VNI) for each container. Compared to bind filtering this implementation is more flexible since it allows the configuration of different traffic control settings, such as traffic shaping and policing, from within the container. The Layer 3 implementation provided by OpenVZ is called venet, while Solaris uses the term shared-IP zone.
The third approach includes providing a Layer 2 virtualized network interface for each container with a valid Link layer address. This gives containers the ability to use many features that are not supported by the previous two solutions, such as DHCP autoconfiguration, separate routing information and filtering rules. This approach can also support a broader set of network configurations. However, the primary downsides include a performance penalty and the inability to control the container networking setup from the host. The latter can be important for the server consolidation case if the host administrator needs to be in the control of the overall network configuration. OpenVZ, Solaris, LxC and Cells all support the creation of the Layer 2 virtualized interfaces. On Linux platforms this feature is called virtual Ethernet (veth). On Solaris a similar configuration is named exclusive-IP zone.
The Linux Layer 2 network isolation is based on the concept of a network namespace that allows the creation of a number of networking stacks that appear to be completely independent. The simplest networking configuration for a container running in a separate network namespace includes a pair of virtually linked Ethernet (veth) interfaces and assigning one of them to the target namespace while keeping the other one in the host namespace. After the virtual link is established, interfaces can be configured and brought up [6].
Linux provides multiple ways for connecting containers to physical networks. One option is connecting the veth interface and the host physical interface by using a virtual network bridge device. Another option is to utilize routing tables to forward the traffic between virtual and physical interfaces. When a virtual bridge device is used, all container and host interfaces are attached to the same link layer bridge and thus receive all link layer traffic on the bridge. However, in the case of route configuration, containers are not able to communicate with each other unless a network route is explicitly provided. Also in the latter case,

Resource limiting
A good virtualization solution needs to provide support for limiting the amount of primary physical resources allocated to each container in order to prevent containers from carrying out denial of service attacks described in section 4. Since the 9.0 release FreeBSD utilizes Hierarchical Resource Limits (RCTL) to provide resource limitation for users, processes or jails [13]. RCTL supports defining an action in case a specified limit is reached: deny new resource allocation, log a warning, send a signal (for example SIGHUP or SIGKILL) to a process that exceeded the limit or to send a notification to the device state change daemon.
Solaris implements resource management for zones using a number of techniques that can be either applied to a whole zone or to a specific process inside a zone. Resource partitioning, called resource pools, allows defining a set of resources, such as a physical processor set, to be exclusively used by a zone. A dynamic resource pool allows to adjusting the pool allocations based on the system load. Resource capping is able to limit the amount of the physical memory used by a zone.
The traditional way of managing resources on BSD-derived systems is the rlimits mechanism that allows specifying soft and hard limits for system resources for each process. Cells and Linux-VServer utilize rlimits to do resource management for containers. However, the main problem of rlimits is that it does not allow specifying limits for a set of processes or to define an action when a limit is reached. Also the CPU and memory controls are very limited and do not allow specifying the relative share of CPU time, number of virtual pages resident in RAM or physical CPU or memory bank allocations.
In an attempt to address some of these limitations, OpenVZ and Linux-VServer have implemented custom resource management extensions, such as new limits for the maximum size of shared and anonymous memory or new CPU scheduler mechanisms. In addition both virtualization solutions added the possibility to specify resource limits per container.
Linux Control Groups (cgroups) [3] is a relatively new mechanism that aims to address the downsides of rlimits. It allows arranging a set of processes into hierarchical groups and performs resource management for the whole group. The CPU and memory controls provided by cgroups are rich, and in addition it is possible to implement a complex recovery management in case processes exceed their assigned limits. LxC, Linux-VServer, OpenVZ and Cells provide a way to use cgroups as a container resource management mechanism. Table 4 presents a comparison of different aspects between rlimits and cgroups. A combined use of these mechanisms allows protecting the container from a set of DoS attacks directed towards the CPU, memory, disk I/O and filesystem (rlimits combined with filesystem quotas). However, the future direction is to aggregate all resource management to cgroups, and allow rlimits to be changed by a privileged user inside a container 5 .

Related work
A number of previous studies have compared different aspects of the OS-level virtualization to other virtualization solutions. Padala et al. [32] analyze the performance of Xen vs. OpenVZ in the context of server consolidation. Chaudhary et al. [26], Regola et al. [36] and Xavier et al. [42] perform comparisons of different virtualization technologies for HPC. Yang et al. [43] study the impact of different virtualization technologies for the performance of the Hadoop framework [46].
The Capsicum sandboxing framework [39] introduced in FreeBSD 9 isolates processes from global kernel resources by disabling system calls which address resources via global namespaces. Instead, resources are accessed via capabilities which extend Unix file descriptors. Linux has a similar mechanism, called seccomp [18], that allows a process to restrict a set of systems calls that it can execute. Both Capsicum and seccomp require modifications to existing applications.
While there are OS-level virtualization solutions such as ICore [4] and Sandboxie [14] in existence for Microsoft Windows as add-on solutions, we have left separation of processes  [23] to do fine-grained resource management led to the creation of a new facility for resource management in server systems called Resource Containers. Zap [31] allows the grouping of processes into Process Domains (PODs) that provide a virtualized view of the system and support for CR. An OS-level virtual machine architecture for Windows is proposed by Yu et al. [44]. A partial OS-level virtualization is provided by the PDS environment by Alpern et al. [21]. Wessel et al. [40] propose a solution for isolating user space instances on Android similar to the Cells/Cellrox. The solution by Wessel et al. has a special focus on security extensions, such as remote management, integrity protection and storage encryption.

Discussion and Conclusions
All compared systems implement core container separation features in terms of the memory, storage, network and process isolation. However, while the initial innovation around containers happened on FreeBSD and Solaris, the mainline Linux has caught up in terms of features and the flexibility of the implementation. Linux is likely to have a complete user space process environment virtualization in course of time. Given the scale of deployment of Linux and the maturity of its OS-level virtualization features, we focus on Linux in the rest of this section. Table 5 summaries the state of the OS-level virtualization supported by the current upstream Linux kernel. The first row shows how each type of isolation discussed in section 7 can be achieved using the currently available techniques. The second row presents a number of gaps that are briefly described below.
Security namespaces. In order to reduce security exposure and adhere to the principle of least privilege, many OSs provide an integrated mandatory ac-cess control (MAC) mechanism. MACs can be used to strengthen the isolation between different containers and the host, as well as to enforce MAC policies for processes inside containers. The latter is especially important when the container has a full OS installation, because it usually comes with pre-configured MAC policies. Therefore, OS-level virtualization solutions should support the ability to use the common MAC mechanisms in the underlying host kernel to enforce independently defined (container-specific) MAC policies. However, currently none of the compared solutions fulfills this requirement. Linux kernel developers plan to address this limitation in the future by introducing a security namespace that would make LSMs container-aware.
IPC extensions. While IPC namespaces and filesystem isolation techniques cover most of the inter-process communication methods available on Linux, exceptions exist. For example Transparent Inter-process Communication (TIPC) [16] is not currently covered. TIPC is a network protocol that is designed for an intercluster communication. Usage of such methods would break the IPC isolation borders between containers and if the given features are not needed, they should be disabled from the kernel configuration.
Device namespaces. As discussed in section 7.3, secure access to device drivers from within a container remains an open problem. One way to approach it would be to create a new namespace class (a device namespace) and group all devices to belong in their own device namespaces in hierarchical manner, following the generic namespace design pattern. Given this, only processes within the same device namespace would be allowed to access devices belonging in it. However, since the core of such functionality would resemble more access/resource control than a fully featured namespace, it was initially decided to implement the functionality as a separate cgroups device controller. The discussions defining the full notion of the device namespace and its functionality continue in the kernel community [5].
(Pseudo)random number generator devices. In section 7.3 we stated that using stateless devices such as /dev/random or /dev/urandom are secure within containers due to their stateless nature. This means that even if two containers share the same device, they cannot predict or influence the output from another device node within another container. However, it is important to note that exposing blocking devices, such as /dev/random, poses a Denial-of-Service possibility. A malicious container can exhaust all available entropy and block the /dev/random from being used in all other containers and the host, making it impossible to perform cryptographic operations requiring random input. Even if only non-blocking /dev/urandom is exposed, there is a theoretical possibility that a malicious container can predict the random output for another container or a host. For example in [27] Dodis et al. give an assessment of both /dev/random and /dev/urandom showing that these devices do not accumulate entropy properly. A complete solution would be to implement a separate random device per namespace or even introduce a namespace for (pseudo)random number generators.
Hotplug support. Desktop Linux relies heavily on the dynamic nature of device nodes. Once new devices are plugged in to the system, the kernel generates an uevent structure notifying the user space of the new hardware. As briefly explained in section 7.3, Uevent is typically handled by the udevd daemon which configures the device for system use. Traditionally it has also created the corresponding device node after device setup. As far as containers are concerned, this setup is risky and complicated -containers should not be allowed to configure hardware and/or have permissions for creating the new device nodes. As a result, safe device hotplug for containers remains an open problem.
Incomplete implementation of cgroups. As was mentioned in the section 7.6, the current goal of the upstream Linux is to integrate all features supported by rlimits into the cgroups resource management. However this has not been done yet and currently remains as work in progress.


PDF/A-3u as an archival format for Accessible mathematics

Including LaTeX source of mathematical expressions, within the PDF document of a text-book or research paper, has definite benefits regarding `Accessibility' considerations. Here we describe three ways in which this can be done, fully compatibly with international standards ISO 32000, ISO 19005-3, and the forthcoming ISO 32000-2 (PDF 2.0). Two methods use embedded files, also known as `attachments', holding information in either LaTeX or MathML formats, but use different PDF structures to relate these attachments to regions of the document window. One uses structure, so is applicable to a fully `Tagged PDF' context, while the other uses /AF tagging of the relevant content. The third method requires no tagging at all, instead including the source coding as the /ActualText replacement of a so-called `fake space'. Information provided this way is extracted via simple Select/Copy/Paste actions, and is available to existing screen-reading software and assistive technologies.

Introduction
PDF/A is being adopted by publishers and Government agencies for the longterm preservation of important documents in electronic form. There are a few variants, which pay more or less regard to Accessibility considerations; i.e., 'a' for accessible, 'b' for basic, 'u' for (presence of) unicode mappings for all font characters. Later versions [3,4] of this ISO standard [2] allow for other file attachments in various data formats. In particular, the PDF/A-3u variant allows the inclusion of embedded files of arbitrary types, to convey supplementary descriptions of technical portions of a document's contents.
'Accessibility' is more relevant for reports and text-books than for research outputs. In fact in some countries it is a legal requirement that when a visuallyimpaired student enrols in unit of study for which a text-book is mandated as 'Required', then a fully accessible version of the contents of that book must be made available. Anecdotally, visually-impaired students of mathematics and related fields much prefer mathematical material to be made available as L A T E X source, to any other format. With a Braille reader, this is text-based and sufficiently compact that expressions can be read and re-read with ease, until a full understanding has been achieved. This is often preferable to having an audio version [13,14], which is less-easy to navigate. Of course having both a wellstructured audio version, as well as textual source, is even more useful. The PDF example [12] accompanying this paper 1 in fact has both, though here we concentrate on how the latter is achievable within PDF documents.
Again anecdotally, the cost of reverse-engineering 2 all the mathematical expressions within a complete textbook is typically of the order of £10,000 or AUD 30,000 or CAD 10,000. This cost would have been dramatically reduced if the PDF had originally been created to include a L A T E X or MathML description of each expression 3 , attached or embedded for recovery by the PDF reader or other assistive technology. How to do this in PDF is the purpose of this paper.
The method of Associated Files, which is already part of the PDF/A-3 standard [4], is set to also become part of the ISO 32000-2 (PDF 2.0) standard [6], which should appear some time in 2014 or 2015. In Sect. 3 this mechanism is discussed in more detail, showing firstly how to include the relevant information as attachments, which can be extracted using tools in the PDF browser. The second aspect is to relate the attachments to the portion of content as seen onscreen, or within an extractable text-stream. This can be specified conveniently in two different ways. One way requires structure tagging to be present (i.e., a 'Tagged PDF' document), while the other uses direct tagging with an /AF key within the content stream. In either case a PDF reader needs to be aware of the significance of this /AF key and its associated embedded files.
With careful use of the /ActualText attribute of tagged content, L A T E X (or other) source coding of mathematical expressions can be included within a PDF document, virtually invisibly, yet extractable using normal Select/Copy/Paste actions. A mechanism, using very small space characters inserted before and after each mathematical expression, is discussed in Sect. 4. This is applicable with any PDF file, not necessarily PDF/A. It is important that these spaces not interfere with the high-quality layout of the visual content in the document, so we refer to them as 'fake spaces'.
The various Figures in this paper illustrate the ideas and provide a look at the source coding of a PDF document 1 that includes all the stated methods, thus including the L A T E X source of each piece of mathematical content. (Where explicit PDF coding is shown, the whitespace may have been massaged to conserve space within the pages of this paper.) Indeed the example document includes as many as 7 different representations of each piece of mathematical content: the visual form, as typically found in a PDF document; the L A T E X source, in two different ways; i.e, an attachment associated with a /Formula structure tag and also associated directly to the (visual) content, and as the /ActualText replacement of a 'fake space'.
a MathML version as an attachment, also associated to the /Formula structure tag and also associated directly to the (visual) content; a MathML representation through the structure tagging; words for a phonetic audio rendering, to be spoken by 'Read Out Loud'; the original L A T E X source of the complete document, as a file attachment associated with the document as a whole.
In practice not all these views need be included to satisfy 'Accessibility' or other requirements. But with such an array of representations, it is up to the PDF reading software to choose those which it wants to support, or which to extract according to particular requirements of end-users. It is remarkable that a single document can be so enriched, yet still be conforming with a standard such as PDF/A-3u, see [4]. Indeed, with all content being fully tagged, this document 1 would also validate for the stricter PDF/A-3a standard, apart from the lack of a way to specify the proper rôle of MathML structure tagging, so that tags and their attributes are preserved under the 'Save As Other ... XML 1.0' export method when using Adobe's 'Acrobat Pro' software. This deficiency will be addressed in PDF 2.0 [6]. Methods used to achieve the structure tagging in the example document 1 have been the subject of previous talks and papers [10,9] by the author. It is not the intention here to promote those methods, but rather to present the possibilities for mathematical publishing and 'Accessibility' that have been opened up by the PDF/A-3 and PDF/UA standards [4,7], and the 'fake spaces' idea. The example document [12] is then just a 'proof-of-concept' to illustrate these possibilities.
Since the PDF/A-3 standard [4] is so recent, and with PDF 2.0 [6] yet to emerge, software is not yet available that best implements the 'Associated Files' concept. The technical content of the Figures is thus intended to assist PDF software developers in building better tools in support of accessible mathematics. It details (i) exactly what kind of information needs to be included; (ii) the kind of structures that need to be employed; and (iii) how the information and structures relate to each other. For those less familiar with PDF coding, the source snippets have been annotated with high-lighting 4 and extra words indicating the ideas and intentions captured within each PDF object. Lines are used to show relationships between objects within the same Figure,  2 Overview of the PDF file format PDF files normally come employing a certain amount of compression, to reduce file-size, so appear to be totally intractable to reading by a human. Software techniques exist to undo the compression, or the PDF file may have been created without using any. The example document 1 was created without compression, so can be opened for reading in most editing software.
The overall structure of an uncompressed PDF file consists of: (a) a collection of numbered objects: written as ¡num¿ 0 obj . . . endobj where the ' . . . ' can represent many, many lines of textual (or binary) data starting on a new line after obj and with endobj on a line by itself. The numbering need not be sequential and objects may appear in any order. An indirect reference sequence of the form ¡num¿ 0 R 5 is used where data from one object is required when processing another. A cross-reference Thus the data in a PDF file is contained within the collection of objects, using the cross-reference table to precisely locate those objects. A PDF browser uses the /Catalog object (e.g., object 2081 in Fig. 3c) to find the list of /Page objects (e.g., object 5 in Fig. 3b), each of which references a /Contents object. This provides each page's contents stream of graphics commands, which give the details of how to build the visual view of the content to be displayed. A small portion of the page stream for a particular page is shown in Figures 1b, 3a, 5a.
Character strings are used in PDF files in various ways; most commonly for ASCII strings, in the form ( . . . ); see Figures 1a, 1b, 2b, 2c, 3a, 3c, and 5a. Alternatively, a hexadecimal representation with byte-order mark ¡FEFF. . . ¿ can be used, as in Figures 1b, 3a, 5a. This is required particularly for Unicode characters above position 255, with 'surrogate pairs' used for characters outside the basic plane, as with the variable name in those figures. Below 255 there is also the possibility of using 3-byte octal codes within the ( . . . ) string format; see footnote 11 in Sect. 4. For full details, see §7.3.4 of PDF Specifications [1,5].
PDF names of the form /⟨name⟩, usually using ordinary letters, have a variety of uses, including (i) tag-names in the content stream (Figures 1b, 3a, 5a); (ii) identifiers for named resources (Fig. 3b within object 20 and in the /AF tagging shown in Fig. 3a); and extensively as (iii) dictionary keys (in all the Figures 1, 2, 3, 5) and frequently as dictionary values (see below).
Other common structures used within PDF objects are as follows.
The key-value pairs may occur in any order, with the proviso that if the same ⟨key⟩ occurs more than once, it is the first instance whose ⟨value⟩ is used. A /Type key, having a PDF name as value, is not always mandatory; but when given, one refers to the dictionary object as being of the type of this name. See Figures 1a, 2b, 2c, 3b, 3c and 5b for examples. (iii) stream objects consist of a dictionary followed by an arbitrarily-long delimited stream of data, having the form ¡¡ ... ¿¿ stream ... endstream, with the stream and endstream keywords each being on a separate line by themselves (see objects 26 and 28 in Fig. 2c). The dictionary must include a /Length key, whose value is the integer number of bytes within the data-stream. With the length of the data known, between the keywords on separate lines, there is no need for any escaping or special encoding of any characters, as is frequently needed in other circumstances and file-formats. See §7.3.8 of [1,5] for more details; e.g., how compression can be used. Dictionaries and arrays can be nested; that is, the ⟨value⟩ of a dictionary item's ⟨key⟩ may well be another dictionary or array, as seen in objects 20 and 90 within Fig. 3b. Similarly one or more ⟨item⟩s in an array could well be a dictionary, another array, or an indirect reference (regarded as a 'pointer' to another object).
With the use of PDF names, objects, and indirect references a PDF file is like a self-contained web of interlinked information, with names chosen to indicate the kind of information referenced or how that information should be used.
The use of objects, dictionaries (with key-value pairs) and indirect references makes for a very versatile container-like file format. If PDF reader software does not recognise a particular key occurring within a particular type of dictionary, then both the key and its value are ignored. When that value is an indirect reference to another object, such as a stream object, then the data of that stream may never be processed, so does not contribute to the view being built. Thus PDF producing or editing software may add whatever objects it likes, for its own purposes, without affecting the views that other PDF reading software wish to construct. This should be contrasted with HTML and XML when a browser does not recognise a custom tag. There that tag is ignored, together with its attributes, but any content of that tag must still be handled.
It is this feature of the PDF language which allows different reader software to support different features, and need not use all of the information contained within a PDF file. For example, some browsers support attachments; others do not. A PDF format specification now consists mostly of saying which tags and dictionary keys must be present, what others are allowed, and how the information attached to these keys and tags is intended to be used. Hence the proliferation of different standards: PDF/A, PDF/E, PDF/VT, PDF/UA, PDF/X, perhaps with several versions or revisions, intended for conveying different kinds of specialised information most relevant within specific contexts.
. Two types of tagging can be employed within PDF files. 'Tagged PDF' documents use both, with content tags connected as leaf-nodes of the structure tree.
Tagging of content is done as /⟨tag⟩ ⟨dict⟩ BDC . . . EMC within a contents stream. Here the BDC and EMC stand for 'Begin Dictionary Content' and 'End Marked Content' respectively, with the ⟨dict⟩ providing key-value pairs that specify 'properties' of the marked content, much like 'attributes' in XML or HTML tagging 6 . The ⟨tag⟩ can in principle be any PDF name; however, in §14.6.1 of the specifications [1,5] it stipulates that "All such tags shall be registered with Adobe Systems (see Annex E) to avoid conflicts between different applications marking the same content stream." Thus one normally uses a standard tag, such as /Span, or in the presence of structure tagging (see below) choose the same tag name as for the parent structure node. Figures 1b, 3a, 5a show the use of Presentation-MathML content tag names, which are expected to be supported in PDF 2.0 [6]. Typical attributes are the /ActualText and /Alt strings, which allow replacement text to be used when content is extracted from the document using Copy/Paste or as 'Accessible Text' respectively. The /MCID attribute allows marked content to be linked to document structure, as discussed below. A variant of this tagging uses a named resource for the ⟨dict⟩ element. This is illustrated with /AF content tagging in Sect. 3.2.
Tagging of structure requires building a tree-like structural description of a document's contents, in terms of Parts, Sections, Sub-sections, Paragraphs, etc. and specialised structures such as Figures, Tables, Lists, List-items, and more [1,5, §14.8.4]. Each structure node is a dictionary of type /StructElem having keys /S for the structure type, /K an array of links to any child nodes (or Kids) including marked content items, and /P an indirect reference to the parent node. Optionally there can be a /Pg key specifying an indirect reference to a /Page dictionary, when this cannot be deduced from the parent or higher ancestor. Also, the /A key can be used to specify attributes for the structure tag when the    Fig. 1a. It is through the /MCID numbers that the association is made to the /Formula structure tag for the corresponding piece of mathematical content.
document's contents are exported in various formats; e.g., using 'Save As Other ... XML 1.0' export from Adobe's 'Acrobat Pro' browser/editor. Fig. 1a shows the MathML tagging of some inline mathematical content. The tree structure is indicated with lines connecting nodes to their kids; reverse links to parents are not drawn, as this would unduly clutter the diagram. Other keys, such as /ID and /T can provide an identifier and title, for use primarily in editing software to locate specific nodes within appropriately ordered listings.
The link between structure and marked content (as leaf-nodes to the structure tree, say) is established using the /MCID number attribute. A numeric integer entry in the /K Kids array corresponds to an /MCID number occurring within the contents stream for that page specified via a /Pg entry, either of the structure node itself or the closest of its ancestors having such a key. Fig. 1b shows this linking via /MCID with lines drawn to the corresponding structure nodes shown in Fig. 1a. The interplay of structure with content was addressed in the author's paper [10], with Figure 1 of that paper giving a schematic view of the required PDF structural objects.

'Associated Files', carrying L A T E X and Math views of mathematical content.
There are several ways in which file attachments may be associated with specific portions of a PDF document, using the 'Associated Files' technique [4, Annex E]. The file is embedded/attached and then associated, by a method, either to:      2a shows how attachments are presented within a separate panel of a browser window, using information from an array of filenames; see Fig. 2b. This is independent of the page being displayed, so the array must be referenced from the document level. This is seen in Fig. 3c using the /Names key of the /Catalog dictionary, which references object 2080, whose /EmbeddedFiles key then references the filenames array (object 1860 in Fig. 2b). One can also see in Fig. 2b how each filename precedes an indirect reference to the /Filespec dictionary [6, §7.11.3, Tables 44 and 45] for the named file; see Fig. 2c. This dictionary contains a short description (/Desc) of the type of content as well as the filename to use on disk, and a link via the /EF key to the actual EmbeddedFile stream object. That a file is 'Associated' is indicated by the /AFRelationship key, whose value is a PDF name indicating how the file is related to visible content. Options here are /Source as used with the L A T E X source coding, or /Supplement as used with the MathML description. Other possibilities are /Data (e.g., for tabular data) and /Alternative for other representations such as audio, a movie, projection slides or anything else that may provide an alternative representation of the same content. /Unspecified is also available as a non-specific catch-all.
Not all attachments need be 'Associated' and conversely not all 'Associated Files' need be displayed in the 'Attachments' panel, so there is another array (object 1859) as shown in Fig. 3c, linked to the /MarkInfo sub-dictionary of the /Catalog dictionary. Files associated with the document as a whole, as in method (i) above, link via the /AF key in the /Catalog dictionary (see Fig. 3c).
For the L A T E X source of a mathematical expression method (iv) is preferred, provided structure tagging is present within the PDF. This is discussed below in Sect. 3.1. Method (iii) also works, provided the expression is built from content confined to a single page. This is described in Sect. 3.2.
As 'Associated Files' have only been part of published PDF/A standards [4] since late 2012, it may be some time before PDF readers provide a good interface for 'Associated Files', beyond using the 'Attachments' pane. This ought to include interfaces to view the contents of attached files, do searching within the files, and make the file's contents available to assistive technology. One possible way to display this association is apparent in earlier work [11], whereby a bounding rectangle appears as the mouse enters the appropriate region.

. fi
With an understanding of how structure tagging works, as in Sect. 2.1, then associating files to structure is simply a matter of including an /AF key in the structure node's dictionary, as shown in Figures 1a and 2b. The value for this key is an array of indirect references to /Filespec objects for the relevant files. There is nothing in the content stream in Fig. 1b to indicate that there is a file associated with this structure node. Rather the browser, knowing that 'Associated' files are present, needs to have gone through some pre-processing to first locate the node (if any) to which it is associated, then trace down the structure tree to the deepest child nodes (objects 114, 116, 118). array of references to Associated embedded files dictionary of named resources -associated file arrays dictionary of name types (a) A portion of the PDF content stream associating content with embedded files for the mathematical expression indicated in Fig. 2a, using a marked content tag /AF to refer to a named resource /inline-1. All content down to the final EMC is associated.
(b) A portion of the /Properties dictionary (upper, object 20) which is linked to a /Page object (lower right, object 5) via its /Resources key (see lower left, object 90). Thus a name (such as /inline-1) is associated with an array of /Filespec references (viz. [27 0 R 29 0 R]), which lead to the L A T E X and MathML files seen in Fig. 2c.
(c) The document's /Catalog (object 2081) indicates presence of embedded files via the /Names key (object 2080). This references the array (object 1860 in Fig. 2b), to establish the correspondence between filenames and /Filespec dictionaries. Embedded files which are 'Associated' to content portions are also listed in an array (object 1859) referenced from the /AF key in the /MarkInfo dictionary. Fig. 3. Embedded files associated with specific content.

. fi
With an understanding of how content tagging works, as in Sect. 2.1, and the fact that marked content operators may be nested, then associating files to content is also quite simple. One simply uses an /AF tag within the page's content stream with BDC . . . EMC surrounding the content to be marked, as shown in Fig. 3a. This employs the named resource variant (here /inline-1) to indicate the array of 'Associated' files. Fig. 3b shows how this name is used as a key (in dictionary object 20) having as value an array of indirect references to /Filespec objects (27 and 29). These resources can be specific to a particular page dictionary (object 5), but in the example document 1 the named resources are actually made available to all pages, since this accords with not including multiple copies of files when a mathematical expression is used repeatedly. Finally Fig. 3c shows the coding required when embedded files, some of which may also be associated to content or structure, are present within a PDF document. One sees that the array (object 1859) of indirect object references in the lower part of Fig. 3c refer to the same /Filespec objects (27 and 29) as the named resources (object 20) in the upper part of Fig. 3b. These are the same references using /AF keys seen in Fig. 1b and Fig. 2b to the objects themselves in Fig. 2c.
This mechanism makes it easier for a PDF reader to determine that there are files associated to a particular piece of content, by simply encountering the /AF tag linked with a named resource. This should work perfectly well with a PDF file that is not fully tagged for structure. However, if the content is extended (e.g., crosses a page-boundary) then it may be harder for a PDF writer to construct the correct content stream, properly tagging two or more portions.
4 Access-tags: attaching L A T E X source to 'fake' spaces.
A third method allows inclusion of the L A T E X source of mathematics so that it may be readily extracted, using just the usual Select/Copy/Paste actions. This works with some existing PDF reader applications, including the freely available 'Adobe Reader'. It is achieved by making use of the /ActualText attribute [5, §14.9.4] for a piece of marked content, whether or not structure tagging is present. It can be done by existing PDF-writing software that supports tagging of content, as in Sect. 2.1, and specification of a value for the /ActualText attribute. Fig. 4 shows how this works, by tagging a 'fake space' character immediately before mathematical content, and another immediately afterwards. By selecting (see Fig. 4a) then Copy/Paste the content into text-editing software, the result should be similar to Fig. 4b. The PDF reader must recognise 9 /Actual-Text and replace the copied content (e.g. a single font character) with its value.   4. This shows how the selection in (a), when copied and pasted into a text file, recovers the L A T E X source (b) that was used to specify the visual appearance of the mathematical content. In (c) we see structure within a /Formula node, (see also Fig. 5b) with leaf-nodes of /accesstag structure nodes being marked content of type /AccessTag. This consists of a single space character carrying an /ActualText attribute which holds the replacement text; as seen explicitly in the coding shown in Fig. 5a. The 'fake spaces' are very narrow; when selected they can be seen very faintly in (c) within the ovals indicated, at the outer edge of the the bounding rectangles of the outermost math symbols.  Fig. 3b see Fig. 2a see Fig. 1b see Fig. 1b see Fig. 1b portion of page stream, see Fig. 3b <accesstag> <accesstag> see Fig. 2b,c (a) Complete portion of the content stream corresponding to the mathematics shown as selected in Fig. 4a. This is the same content as in Figures 1b and 3a but with the ' ...' parts there now showing the /AccessTag coding of a 'fake space' with /ActualText attribute. The /AF ... BDC ... EMC wrapping of Fig. 3a is also shown. Being part of the document's content, these space characters are also assigned /MCID numbers to be linked to structure nodes, as in (b) below.
(b) Portion of the structure tree as in Fig. 1a, but now showing how the 'fake spaces' can be linked to structure nodes, here /accesstag. The missing portions of Fig. 1a, indicated there by ' ...' are now filled-in, but leaving out other parts whose purpose has already been explained. Fig. 4c, shows the tagging opened out within the 'Tags' navigation panel, with the /accesstag structure nodes selected. The PDF name /AccessTag tags a single 'space' character [( )]Tf as marked content, having replacement text in the /ActualText attribute; see Fig. 5a. An /MCID number is not needed for this technique to work; these are included in the example document 1 which is fully tagged 10 .
We refer to these tagged 'fake spaces' as 'Access-tags', since a motivation for their use is to allow Assistive Technology (e.g., a Braille reader) access to the L A T E X source of mathematical content. The spaces are 'fake' in the sense that they are just 1pt in height and have nearly zero width. This makes them hard to select by themselves, but nearly impossible to separate from the mathematical content which they accompany; see Fig. 4c. They act as ordinary spaces when copied, but this is substituted with the /ActualText replacement, if supported. Another aspect of their 'fakeness' is that they take no part in the typesetting, when using pdf-L A T E X (post-2014). Of course the same idea could be implemented in different ways with different PDF-producing software.
One places into the initial 'Access-tag' text of the L A T E X source -with care given to encode special characters 11 -as the value of its /ActualText attribute. The source coding is preceded by the string ¡latex¿ and followed by ¡/latex¿ and ¡content¿, with return-characters (in octal "015) to allow these 'delimiters' to ultimately copy onto lines by themselves. The trailing 'Access-tag' just takes ¡/content¿. As a result, the eventual Paste gives text as shown in Fig. 4b.
Assistive Technology (e.g., a Braille reader) works either by (a) emulating Copy/Paste of on-screen portions of the document's window; or (b) by directly accessing the 'Accessible Text' view of the PDF document's contents. In both cases the /ActualText replacements are extracted. (The 'Accessible Text' view can be exported directly using Adobe's 'Acrobat Pro' software, see [12].) For mathematical symbols, where Figures 1b, 3a and 5a show the presence of both /Alt and /ActualText attributes, then the /Alt contributes to the 'Accessible Text', whereas /ActualText supplies what is copied to the Clipboard for Copy/Paste. In either case, a human reader when encountering ¡latex¿ on a line by itself can choose whether to read (or listen to) the following lines of L A T E X coding, else use a Find action to skip down to where the next ¡/latex¿ occurs. This is followed by a line containing ¡content¿. Similarly the human can read/listen or skip down to where ¡/content¿ occurs.