N3FS' Not NFS

by Giovanni Chiola, DISI, University of Genoa, Italy

Copyright © 2000 - 2001 Giovanni Chiola.
This work is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of merchantability or fitness for a particular purpose.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU Free Documentation License".


Index


Main Goals and Functionalities

The idea of the N3FS Project is to design and prototype a really distributed, ubiquitous file system whose implementation can be hidden and spread over several nodes of the network. We would have liked to call this a "Network File System." Unfortunately we could not, as NFS already refers to a very well known product developed long time ago by Sun Microsystems. N3FS will be completely different from the Network File System originally introduced by SUN Microsystems, and this justifies its name.

There will be no centralized server to provide this service, only a (possibly dynamic) set of co-operating server nodes. In this respect it will also be very different as compared to CODA (developed at CMU and recently integrated into the Linux Kernel), the Andrew File System, and all other (perfectly respectable) prototypes developed so far as part of various research projects worldwide. The main difference between N3FS and most other distributed file system prototypes derives exactly from the explicit avoidance of one or more "trusted servers." This change has a strong impact on the security and fault-tolerance models, that will be based on client-side algorithms. Such a drastic change substantially contributes to the robustness of the system in a truly distributed, dynamically changing environment.

Individual users will have to register for the service and be identified by appropriate digital certificates. Also "server" nodes will be identified by certificates and appropriate cryptographic protocols will be adopted to guarantee both users and system against "denial of service" kind of attacks. Integrity of data will be supported by digital signature of the distributed data, so that the client will be able to recognise original data from counterfaited data. Confidentiality of data will be provided by local encryption at "client" side based on symmetric random keys, so that clear-text data will never be reconstructed in other nodes.

The security architecture will be based on complete trust of the local client operating environment only. Each user will only access his/her own virtual system, without any possibility of decyphering any other users' files. Sharing of information will be explicitly handled through the adoption of appropriate group keys and certificates. However information sharing among users will not be addressed in the first Beta implementation of N3FS.

Availability will be provided by redundancy, implemented by appropriate K-out-of-N erasure recovery codes based on Galois Fields arithmetic. The proper amount of redundancy will be chosen by the user, depending on the desired level of availability for a particular file (or set of files). Properly functioning servers will support data availability by accepting "write" or "delete" requests only if digitally signed by the owner of the file system. A moderate amount of servers might become unavailable (or unreliable) due to security attack as well as to hardware, software, and communication link failures. In any case the redundancy of the implementation will allow to retrieve data provided that a minimum amount of K servers will still be functional and reachable from the client (independently of the cause of mulfunction of the other servers). Directories and private keys will be stored with maximum level of redundancy and of encryption strength (based on hashing of one or more pass-phrases).

The location of data and their possible migration from a "hidden server" to the other will be totally transparent to the user (except for latency time). A proper, distributed diffusion algorithm based on monitoring of performance and availability will be implemented. Data will be automatically reconstructed if one or more of redundant pieces will remain inaccessible for long time (assuming that those pieces of data were lost). A concept of physical location of the client will also be studied in order to help efficient migration and relocation of data, in order to improve availability, average response time, and fault-tolerance, all at the same time. A user planning to move from a location to another should also have the possibility (but not the obligation) to notify the system in advance of his/her plans, so that the system can re-distribute copy locations in order to adjust for these travelling plans and reduce latency and overhead in data access from the new users's location.

Users will contribute to the implementation by bringing disks and processing nodes to the system. In the long term each user will receive a share of available storage capacity related to the amount of storage capacity provided to the system, although such quotas can be passed for relatively short time, allowing for temporary storage of larger data sets. The idea of sharing disk space with other users in a secure, efficient, and reliable way is to obtain new characteristics of the file system (fault-tolerance, high availability, ubiquity, etc.) rather than to increase the total storage capacity with respect to the disk space that a single user could buy. This zero-cost resource sharing could bootstrap the deployment of the system, but of course will not prevent the addition of a "free market economy," in which somebody could sell or lease or rent disk space to users willing to buy it without providing servers to the system.

Acceptable levels of performance both in terms of average latency to access data and of required network bandwidth to support access to the file system by real applications will be achieved by appropriate Caching and/or Prefetching techniques. Part of the locally available disks will be used to cache and mirror frequently and/or most recently accessed data, so that most actual (read) accesses will be resolved locally. In case of link failure, synchronization will be automatically implemented upon re-establishment of the network connection. In order to implement such features in an easy way, a proxy agent will be run on each client machine, even in case the small amount of disk storage is not worthy to be shared with the rest of the servers.

Transparent management, ubiquity, high security, and high availability will be the main benefits of this disk sharing service as compared to currently available technologies.

The implementation will be based on an Internet-like protocol definition. In particular, the transport protocol will be implemented by HTTP 1.1 GET, PUT, HEAD, and DELETE methods. Encryption and signature capabilities will be provided by software taken from the implementation of the GNU Privacy Guard. Digital certificates will be handled using the OpenSSL toolkit. The client side of the protocol will be implemented as a modification of the Linux file system calls, offering transparent access to an "n3fs" personal file system to each user of the local machine. In principle, each machine permanently connected to the system could run a "server daemon" that contributes to the implementation of the "set of hidden servers" and with one or more "client modules". Portable machines with sporadic network connections will run only a proxy server that does not contribute to the distributed server set.

Last but not least, the prototype will be developed and distributed under the GNU GPL. An incremental approach will be followed. The first Beta release will include all the main functionalities of the system as outlined above, however the main concern will be the production of a robust and stable infrastructure with minimal re-coding efforts. Subsequent steps in the evolution of the software will include the study of new algorithms to improve performance, availability, and long-term reliability.

General Organization and Data Encoding

Users will be uniquely identified by means of a single 160 bit number called a User_Id. The User_Id will be constructed by applying a standard cryptographic hash function (Message Digest) to an ASCII text describing the identity of the user according to a format like:

Such a user description will be stored in a file resident in the user's home directory in the client. The idea is that the user can always reconstruct the file manually with minimal effort in case of loss. The user will therefore be able to access data even after changing the client machine (as only locally cached data will be lost). The application of the Message Digest results in an extremely low probability that two different users may be associated with the same User_Id, even if their User_Id is generated in a totally independent way, and even in case of several billions of users trying to register for the service world-wide.

Each user will have a number of signature keys guaranteed by standard certificates:

main key
Signed by a recognized Certification Authority.
Used to sign all other certificates produced by the user. High security is required.
load key
Signed by the user using his/her main key.
Used to sign read requests. Low security requirements, as the main purpose of the key is to avoid congestion as a consequence of "denial of service" type of attacks. Confidentiality is guaranteed by the encryption of each file content with randomly generated one-time symmetric keys, not by refusing delivery of information to unauthorized customers.
store keys
Signed by the user using his/her main key.
Used to sign write (and delete) requests. Intermediate/high security requirements, as the main purpose of the key is to keep integrity of data by reducing the chance of undue removal of information by attacker. Attacks to integrity in the sense of modification of data will be detected using document signature verified at client side. More than one store key can be used, related to various degree of integrity needed. For instance, higher requirements for directories and keys, intermediate requirements for normal files, lower requirements for temporary copies of files. The main certificate of a user or a server must be signed by a central N3FS authority that guarantees the identity and integrity of each user and each server of the system. Therefore, the system deployment will require the availability of an N3FS portal that each prospective user or prospective server must contact in order to obtain the certification of its main key.

Each user will have have the possibility of addressing files using a single 48 bit (6 bytes) identifier called the File_Id. The File_Id takes the place of the "inode" number in a Unix-like file system. It will be computed by applying a cryptographic digest function to the pathname of the file relative to the mounting point of the N3FS file system. N3FS will not support multiple "physical links" to the same file, so that each file will be uniquely identified by its pathname. Only "logical links" to existing files will be supported in order to simplify the assignment of File_Id. In the (extremely unlikely) case of clash of two or more pathnames to the same File_Id, a retry algorithm will be implemented that increments by 1 the File_Id value until a "free" File_Id is found.

Each file will be characterized by a logical clock (version number) of 40 bits (5 bytes). A newly created file will automatically be assigned version number 0. Version number will be incremented by one each time a change is committed (the file is closed after an update). Upon successful installation of a higher version number, data corresponding to a lower version number of the same file may be erased from servers in order to reclaim storage space. Proper distributed garbage collection algorithms will be run by the servers to remove outdated information and restore free disk space.

Each file will be associated with a randomly generated 128 bit key for symmetric block encryption. The key will be stored in an encrypted form into the "inode" structure, that will also be redundantly stored on servers. Privacy of the file encryption key will be enforced by a double encryption scheme (using two independent algorithms and two independent keys computed from the digest of two independent pass-phrases).

Each file will be split in fixed size, 64KB blocks, that will be individually compressed and encrypted by the user agent module. Integrity will be checked by storing the MD5 digest of the compressed block before encryption (that will be checked for as a signature of each block on reading). Blocks will be enumerated starting from 0, using a 32 bit (4 bytes) counter (so that each file will be limited to 256TB in size). The proxy or local server daemon will eventually split each block in N "parts" using a 4-out-of-N erasure restore encoding based on Galois fields arithmetic. The number N of parts will range from a minimum of 4 to a maximum of 255 (due to the use of 8 bits in the Galois field encoding). The choice of the value N will be left to the configuration of the user agent, depending on the required availability of each file.

The encryption key and the parameter N will be kept constant for all blocks of a given file, while they will typically differ for different files.

Data Distribution and Retrieval

The user agent will take care of sending the inode and all pieces of blocks to at least one N3FS server in case of file close after a write open, and to request the inode and some of the pieces of blocks to at least one N3FS server in case of file read or modify open. The N3FS server(s) directly contacted by a user agent will take the responsibility of forwarding requests to other N3FS servers when they cannot satisfy the request directly. An extension of the HTTP 1.1 protocol will be adopted to support the communication between user agents and servers as well as among servers.

All inodes and pieces of blocks will be uniquely identified by standard URLs, of the form:

    n3fs:/<userid>/<fileid>/.inode.<versno>.<partno>
    n3fs:/<userid>/<fileid>/<blockno>.<versno>.<partno>
where:
<userid>
is a 27 character string that encodes the User_Id number using a base64 code;
<fileid>
is a 8 character string that encodes the File_Id number using a base64 code;
<blockno>
is a 6 character string that encodes the block number using a base64 code;
<versno>
is a 7 character string that encodes the version number using a base64 code;
<partno>
is a 2 character string that encodes the part number using a base64 code;

Retrieve Protocol

In order to retrieve data from any N3FS server, the user agent will send it a request of the form:

    GET <n3fs_url> HTTP/1.1
    Host: <client_name>:<port>
In case the server has the data referred to by the URL it will send the following reply message to the address and port specified in the Host Header of the request message:
    200 OK HTTP/1.1
    Location: <n3fs_url>
    Content-Length: <nobytes>
    <actual_data>
Otherwise, the server will reply to the user agent with the following message:
    202 Request Accepted HTTP/1.1
    Location: <n3fs_url>
and then it will forward the original request to another server in the form:
    GET <n3fs_url> HTTP/1.1
    Host: <client_name>:<port>
    Forwarded: <server_name>:<port>
A server that receives a forwarded request, will always check for the presence of its own address in one of the "Forwarded" fields, and discard the request in case it finds it (in order to avoid cycles). In case the request is not discarded, the server will check the presence of the requested URL and send the data to the client with a "200 OK" code. If the server does not contain the requested data, it will forward the request to yet another server, after having appended its own "Forwarded" header.

In case the server realizes that the data requested is outdated (i.e., it was marked as possibly outdated or has a version number inferior than the current one, the "200 OK" code is substituted by the "203 Non-Authoritative Information" and possibly by the version number that is believed to be the currently valid one.

Save Protocol

In order to store data in any N3FS server, the user agent will send it a request of the form:

    PUT <n3fs_url> HTTP/1.1
    Host: <client_name>:<port>
    Content-Length: <nobytes>
    <actual_data>
The server may decide to store the data locally, and in that case it will send back an acknowledgement message to the address specified in the "Host:" header of the form:
    201 Created HTTP/1.1
    Location: <n3fs_url>
Alternatively, the server may decide to store the data temporarily, but then to forward the request to another server. In this case it will send an acknowledgement to the client of the form:
    202 Request Accepted HTTP/1.1
    Location: <n3fs_url>
Upon receipt of the "202" answer, the client knows that the data is stored in some server, but not necessarily the one that it sent the request to.

In case the server runs out of storage space, it may be unable to satisfy the request, and in that case it will answer with a negative acknowledgement of the form:

    503 No More Space HTTP/1.1
    Location: <n3fs_url>

In case the server accepted to store temporarily the data, it will eventually forward the request to another server in the form:

    PUT <n3fs_url> HTTP/1.1
    Host: <client_name>:<port>
    Forwarded: <server_name>:<port>
    Content-Length: <nobytes>
    <actual_data>
Then it will wait for the acknowledgement before removing the temporarily buffered data. A server that receives a "Forwarded PUT" request will check for the presence of its own address in the "Forward" list. In case it finds its address in the list, it will send the following negative answer:
    406 Not Acceptable HTTP/1.1
    Location: <n3fs_url>
Similarly, it will send the same negative answer in case it already permanently stored another part of the same block of the same file for the same user. If the forwarded request would be acceptable but the server has not enough memory to satisfy it, it will reply to the forwarding server:
    503 No More Space HTTP/1.1
    Location: <n3fs_url>
A server that receives an acceptable forwarded request may decide to forward it to yet another server, even in case it would have enough memory resources to store it locally. In that case it sends the reply:
    202 Request Accepted HTTP/1.1
    Location: <n3fs_url>
to the forwarding server, and then it will take care of forwarding the request again to another server. The server that decides to store locally a piece of data, sends the reply:
    201 Created HTTP/1.1
    Location: <n3fs_url>
both to the server that last forwarded the request to it and to the client that originated the request (so that it can keep track of the final destination of the data if it has enough memory to store the information).

In case the server discovers that the version number of the block part to be stored is less than or equal to the one of an already stored one, then it will answer with the error message:

    409 Conflict HTTP/1.1
    Location: <n3fs_url>

Caching on Clients

In order to increase performance in the access to files, a caching mechanism will be implemented in the client. User processes will access data stored in a partition of a local disk, and blocks of files distributed over the network will be reconstructed in the local cache before access is allowed to a user process. In case of modifications to data in cache, consistency with the copy distributed over the network will be enforced upon closure of the file. In order to make the maximum file length independent of the size of the disk partition used for local caching, the caching itself will be implemented on a block-by-block basis. Hence only a subset of the blocks constituting a file that is currently open to be accessed by a user process will need to be present in the local cache.

The management strategy for the local cache will mimic demand-paged virtual memory, where file blocks will take the role of pages. LRU will be used to select file blocks to be discarded from the local cache to make room for a newly requested one. Prefetching strategies will be implemented in order to anticipate requests of new blocks that could possibly come from the user process that opened the file, based on the knowledge of the "opening mode" (e.g., prefetch the first block upon read open, and prefetch the last block upon append open).

Caching on a local disk partition will be implemented as an extension/modification of the Linux "ext2" file system. The Inode of each cached file will be present in the local (modified) file system. The modification will consist in defining some of the pointers to data blocks to assume an "invalid" value. Seeking an invalid data block will result in generating a "load" request for a new block to be added to the partial representation of the file. Conversely, a data block may be "invalidated" (i.e., detached from the inode) if it is not the block an open file pointer is currently positioned to. This modified local file system will run in "user mode" instead of as a kernel module. Indeed the disk partition will not be shared with other users, so that there would be no reason to have the OS kernel to manage the partition. System calls will only occur as a result of block accesses by means of the "block device" interface. The implementation will therefore consist in an "extraction" of the relevant code from the kernel and its inclusion as part of the user proxy code. A (smaller) cache will also be implemented in RAM, trying to mimic the "buffer cache" Linux structure, always at the user level.

Traditional file system calls will be intercepted at the level of standard runtime library, thus eliminating any need for actual kernel modifications. Access to "n3fs" files will be redirected to the proxy process running at user level using standard IPC calls, while regular file accesses will trap to kernel level in the usual way.

File versions, Consistency and Repair

Consistency and updates will be handled by a proper use of the version number associated with each file. The version number is zero when a file is created. The version number will not change on read accesses. The version number is automatically incremented by one if an already existent file is opened in write or append mode and subsequently closed.

The inode as well as each individual block parts are labelled with the current version number of the file. In case of an update of the file (open in write mode followed by a close), the block parts and the inode of the previous version (let's say version V) will not be removed from the server until the complete set of new data (version V+1) is installed in the servers. Only upon successful installation of version V+1 will the servers be allowed to reclaim space by removing previous versions.

The last version of the inode (version V) contains the list of all version numbers for all data blocks. The current version of a data block will therefore be less than or equal to the version number of the inode. The version number of a data block will be incremented only in case of change in that block, while the previous version will be kept in case of changes limited to some other data blocks. The current version number of an inode will be stored in the current version of the directory that refers to it, so that if the client starts with the current version of its root directory, it will retrieve only the current version of each file following the directory structures. Following this approach, keeping obsolete versions of any file except the root directory will be armless (and it would even allow the deployment of an "undo" facility that could deliberately retrieves previous versions of a file even after modifications are committed, provided that servers have enough space to keep multiple versions of the same files).

Special care must be devoted to the management of the root directory in order to keep consistency even in the face of adverse network states. For instance, consider the case when a user first starts working in location A, then moves to location B where he changes his root directory, and then returns to location A to continue the work. It might happen that all orginal files are located in servers near to location A. It might also happen that a partitioning of the network takes place after he retrieved the current version from location B, and before the updated version is stored. In that case, version V+1 will be stored in different servers near to location B, disconnected (due to the net partition) from servers near location A where the original file system resided. If the user returns to location A and attempts to access the file system before the network partitioning is removed, he can only find a consistent representation of the previous version, rather than the new one that was created from location B. Moreover, if the user updates the file system, it might create a new version that has the same number as the (different) one that was created from location B, that would result in a disaster if at that point the network partition is removed.

The problem outlined above can be solved in two different ways.

  1. Whenever the inode of any file is opened for "write", it is marked as possibly inconsistent by the server that stores it before submitting a copy to the client. If the client opens a version of file that is marked as possibly inconsistent, a warning message is issued. The client will implement a special "manual repair" mode in case the inode is opened for write, so that the user will be asked to take personal responsibility in the definition of the new version number, with the system suggesting that inconsistency problems could arise in future due to the choice of a version number that is too low.
  2. Whenever the client closes the inode after a write open, it will wait the acknowledgement from a minimum number of servers holding the previous version of the inode, so that only a number of servers insufficient to reconstruct a valid previous version of the inode are allowed not to have marked their copy as outdated. In case not enough servers acknowledge the change within a reasonable time-out, the user is notified of the problem, and asked whether the change should be considered failed or committed anyway.
The two approaches are not exclusive. We can therefore implement both for greater robustness of the versioning system.

The current version of a file may also be subject to repair actions in case of detected data losses. Such repair actions are needed in order to restore the original level of redundancy, and be able to stand additional data losses without degradation of availability. Indeed the probability of data loss increases with time, so that eventually all data blocks would be lost even with high levels of redundancy without the introduction of such repair actions.

Servers will periodically exchange information on their content with other servers. When a server ceases to communicate with its neighbors for a long time, their neighbors will suspect that its data have been lost. They will therefore start a check and recovery action. For each data block that is suspected to be lost, the other servers will put together 4 independent parts in order to reconstruct its content, and then will assign it to another available server. Repair actions will not be started for blocks that are marked as outdated or possibly outdated.

Interface with the Linux File System

The client side of the file system will be implemented as the cooperation of two "entities": the filter/dispatcher in the standard library that recognizes and re-directs system calls to "n3fs" files, and a proxy implemented as a user-level processes running in the background. The proxy will implement the cache on a local disk partition as well as in RAM (using two levels of block caching, managed by an LRU replacement algorithm). The proxy process will fork appropriate "retrieve proxy" and the "store proxy" processes that will contact one or more remote servers in order to manage cache misses and write-backs. They will implement the encoding, splitting and send of individual blocks to be stored on servers and the request, receive, merging and decoding of individual bloks to be retrieved from servers. The proxy processes will operate on 64KByte buffers.

The proxy process will act as a "user-level daemon", launched upon the "login" operation of the user, and terminated by the "logout" of the user.

TO BE COMPLETED



GNU Free Documentation License

Version 1.1, March 2000

Copyright (C) 2000  Free Software Foundation, Inc.
59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.

0. PREAMBLE

The purpose of this License is to make a manual, textbook, or other written document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

1. APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you".

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (For example, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, whose contents can be viewed and edited directly and straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup has been designed to thwart or discourage subsequent modification by readers is not Transparent. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML designed for human modification. Opaque formats include PostScript, PDF, proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.

2. VERBATIM COPYING

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

3. COPYING IN QUANTITY

If you publish printed copies of the Document numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a publicly-accessible computer-network location containing a complete Transparent copy of the Document, free of added material, which the general network-using public has access to download anonymously at no charge using public-standard network protocols. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

4. MODIFICATIONS

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.

You may add a section entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

5. COMBINING DOCUMENTS

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections entitled "History" in the various original documents, forming one section entitled "History"; likewise combine any sections entitled "Acknowledgements", and any sections entitled "Dedications". You must delete all sections entitled "Endorsements."

6. COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

7. AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, does not as a whole count as a Modified Version of the Document, provided no compilation copyright is claimed for the compilation. Such a compilation is called an "aggregate", and this License does not apply to the other self-contained works thus compiled with the Document, on account of their being thus compiled, if they are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one quarter of the entire aggregate, the Document's Cover Texts may be placed on covers that surround only the Document within the aggregate. Otherwise they must appear on covers around the whole aggregate.

8. TRANSLATION

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License provided that you also include the original English version of this License. In case of a disagreement between the translation and the original English version of this License, the original English version will prevail.

9. TERMINATION

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

10. FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.