Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Open Database Connectivity wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Entity–attribute–value model wikipedia , lookup
Concurrency control wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Functional Database Model wikipedia , lookup
Clusterpoint wikipedia , lookup
Relational model wikipedia , lookup
Backend Architecture Version Description 0.1 Creation of initial document. 0.2 Description of the public API to the backend, and the design of the database. 0.2.5 Converted the document to new documentation standard. 1.0 1.1 Updated document to reflect final release. Added list of Backend files Author Aaron Cottle Aaron Cottle Date 10/18/2007 10/18/2007 Felipe Serrano Aaron Cottle M. Freeburg 10/25/2007 12/21/2007 12/26/2007 Contents Introduction .................................................................................................................................................. 4 Public API ...................................................................................................................................................... 4 Data Flow .................................................................................................................................................. 4 Backend Object ..................................................................................................................................... 4 Content ................................................................................................................................................. 5 Entities .................................................................................................................................................. 5 Searching................................................................................................................................................... 5 Searching....................................................................................................................................................... 6 Tags and Comments ...................................................................................................................................... 6 Database Access............................................................................................................................................ 6 Backend Database Tables ............................................................................................................................. 7 Category .................................................................................................................................................... 7 Content ..................................................................................................................................................... 7 ContentProperties..................................................................................................................................... 7 Data ........................................................................................................................................................... 8 Entity ......................................................................................................................................................... 8 EntityProperties ........................................................................................................................................ 8 IdType........................................................................................................................................................ 8 Notifications .............................................................................................................................................. 8 PageViews ................................................................................................................................................. 9 Permissions ............................................................................................................................................... 9 Relation ..................................................................................................................................................... 9 SystemSettings .......................................................................................................................................... 9 Tag ............................................................................................................................................................. 9 TagProperties ............................................................................................................................................ 9 TransactionBuyer .................................................................................................................................... 10 TransactionContents ............................................................................................................................... 10 Backend Files............................................................................................................................................... 11 Access Control ......................................................................................................................................... 11 IAccessControl.cs & AccessControl.cs ................................................................................................. 11 IAccessControlList.cs & AccessControlList.cs ...................................................................................... 11 AccessControlSettingsHelper.cs.......................................................................................................... 11 IAccessControlObject.cs ...................................................................................................................... 11 RelationsPermissionCalculator.cs ....................................................................................................... 11 Backend................................................................................................................................................... 11 IBackend.cs, IBackendInternal.cs, & Backend.cs ................................................................................ 11 BackendFactory.cs .............................................................................................................................. 11 DatabaseConnection.cs....................................................................................................................... 11 GetBackendObjectVisitor.cs ............................................................................................................... 11 BackendObject ........................................................................................................................................ 11 IBackendObject.cs & BackendObject.cs.............................................................................................. 11 IBackendObjectCollection.cs............................................................................................................... 11 BackendQuery ......................................................................................................................................... 12 BackendQuery.cs................................................................................................................................. 12 IQueryValidator.cs & QueryValidator.cs ............................................................................................. 12 IStatementMember.cs & StatementMember.cs ................................................................................ 12 SparqlTranslator.cs ............................................................................................................................. 12 Statement.cs ....................................................................................................................................... 12 BackendRelation ..................................................................................................................................... 12 IBackendRelation.cs & BackendRelation.cs ........................................................................................ 12 Category .................................................................................................................................................. 12 ICategory.cs & Category.cs ................................................................................................................. 12 Collections ............................................................................................................................................... 12 IDBDictionary.cs, DBDictionary.cs, & ADBDictionary.cs ..................................................................... 12 IBackendDictionary.cs & BackendDictionary.cs .................................................................................. 12 GroupCollection.cs .............................................................................................................................. 12 Content ................................................................................................................................................... 12 IContent.cs & Content.cs .................................................................................................................... 12 ContentCollection.cs ........................................................................................................................... 13 DB Configuration ..................................................................................................................................... 13 CreateTables.sql .................................................................................................................................. 13 DevDB.reg & ProdDB.reg .................................................................................................................... 13 Entity ....................................................................................................................................................... 13 IEntity.cs & Entity.cs............................................................................................................................ 13 EntityCollection.cs............................................................................................................................... 13 IUser.cs & User.cs ............................................................................................................................... 13 Exceptions ............................................................................................................................................... 13 BackendDisposedException.cs ............................................................................................................ 13 BannedUserException.cs..................................................................................................................... 13 IllegalAccessException.cs .................................................................................................................... 13 IncorrectPasswordException.cs .......................................................................................................... 13 ObjectNotCreatedException.cs ........................................................................................................... 13 ObjectNotDeletedException.cs ........................................................................................................... 13 ObjectNotFoundException.cs ............................................................................................................. 13 ObjectNotInitializedException.cs ........................................................................................................ 13 UserAlreadyExistsException.cs ........................................................................................................... 14 UserNotValidatedException.cs ........................................................................................................... 14 Notifications ............................................................................................................................................ 14 Relationships ........................................................................................................................................... 14 Tags ......................................................................................................................................................... 14 ITag.cs & Tag.cs ................................................................................................................................... 14 Transactions ............................................................................................................................................ 14 IPaymentDelegate.cs .......................................................................................................................... 14 ITransaction.cs & Transaction.cs......................................................................................................... 14 ITransactionMember.cs & TransactionMember.cs............................................................................. 14 Miscellaneous ......................................................................................................................................... 14 AuthenticationSystem.cs .................................................................................................................... 14 BackendConstants.cs .......................................................................................................................... 14 NpgsqlStream.cs ................................................................................................................................. 14 Introduction The backend provides a display- independent way to store, retrieve, and search for data. This document broadly describes both the public-facing API and the internal architecture hidden from the outside, including database schemas. For a detailed description of all methods available to be called, consult the actual code comments. IBackend.cs contains all of the methods that can be called from the backend and their documentation. Public API There are a number of public data interfaces exposed, each representing an object in the system. The top-level interface is IBackendObject which provides a number of attributes common to all objects in the backend. The IEntity, IUser, and IContent classes derive from this, each adding methods specific to users and content. Another interface, IBackendObjectCollection, is also exposed, which not only derives from IBackendObject but also implements ICollection. This means that in addition to being able to be treated as a backend object it contains a number of backend objects. In this way, groups of, say, users can be treated in exactly the same way as a single object. No concrete implementations of any of these interfaces are exposed. One adapter interface is exposed which houses all methods which can be called on the backend to store or retrieve data: IBackend. No concrete implementations are exposed, but an additional class, BackendFactory, has a public static method which creates one. The only access to the backend must go through the IBackend implementation provided by BackendFactory. IBackend controls all access to database connections and opens a new connection when it’s instantiated, so it’s necessary to enclose all instances of it in a “using” block in C# to make sure it’s properly disposed of and connections aren’t left open. IBackend also manages transactions, allowing multiple changes to happen to the database and be committed concomitantly with a call to SaveChanges(). Any changes made to the database which are not eventually followed by a call to SaveChanges() will be rolled back when the instance of the backend is disposed. Data Flow Backend Object There are a number of features common to all objects stored in the backend. To create a new object, parameters are passed into the proper method on IBackend. The backend then allocates a new Guid based upon what type of object is being created, saves the given information to the database, and returns the created object. All mutator methods on a backend object write directly to the database, but of course do not actually change the database until SaveChanges() is called on the IBackend. Each backend object internally retains a reference to the backend which created it in order to change the database when needed. This means that once the backend is disposed (such as when the using block has closed) any backend objects, even if still in scope, cannot change themselves. It’s best to think of a backend object as a reader on a stream: a reader can be set to retrieve information from a stream, but if the stream is closed, the reader becomes worthless. Most backend objects locally cache data values, so it may be possible to read information about a backend object after the backend is closed, but trying to set information will throw an exception. Backend object can be retrieved from IBackend by passing a certain amount of uniquely identifying information to the proper method. All backend objects have a Guid which uniquely identifies them, and it is even possible to determine the type of an object given only its Guid. Content Content objects represent any type of user-uploaded data. Content objects, once created, can produce any number of different data streams which can be written to or read. Each stream can be retrieved by a key. For example, a newly uploaded video will have the “default” stream of the actual video content itself, but it may also have a “thumbnail” stream, which provides a small snapshot of a frame of the video. Entities Any agent viewing or creating content in the system is classified as an IEntity. The IEntity interface uses three types of uniquely identifying information: the Guid universal to all backend objects, plus an email address and a username. Each entity has various permissions on types of content. For example, the author of a piece of content would have permission to view and edit it, while an entity who has purchased content could view but not edit it, while an entity who has not purchased the content cannot view the content in full, but could perhaps view a preview of it. Each entity also has a password associated with it. This password can be verified with the stored one in the database to determine if the entity can successfully log in. Entities can also be “banned” from the system temporarily for violating terms of use, at the discretion of an administrator, which prevents them from logging in. One entity can also “sponsor” one or more other entities, showing that the sponsor favors the sponsored entity’s content and wishes to promote him or her. The backend provides methods for obtaining both the list of all sponsors of an entity and all entities whom this entity is sponsoring. Searching Searching goes through several stages. For a search starting from a user’s typed text, the search module parses and translates the query into a search tree. The relations module then translates the search tree into a series of Sparql statements which are bundled together inside a BackendQuery. Backend queries are accepted in IBackend, turned into Sql, and executed, returning all information which matched the query. Backend queries can also be formed manually, without starting from user input. Searching Searching pulls information primarily from the Relations table in the database. The relations table is an RDF-style database table, formed based on the “subject-predicate-object” form. In brief, two backend objects form the subject and the object; these can be thought of as nouns. The predicate describes the relation between the two backend objects; this can be thought of as a verb. The RDF language, which is a W3C standard, is excellent for showing that different types of relationships between objects exist, but searching also needs the ability to store the “strength,” or value, of a relationship. The value can be thought of as an adverb. As an example, the simple sentence “Matt trusts Brad 10%” can thus be stored in the relation table. Sparql is a language designed to query RDF-style databases. We again have extended this slightly to incorporate the notion of the strength of a relationship. Sparql allows for sentences as above to be formed, but also allows for variables to take the place of some literals, representing unknown information. The pseudo-Sparql sentence “?truster Trusts Brad > 10%” is a simple query to find all people who trust Brad at least 10%. Joining this to the sentence “?truster MemberOf BradLikers” will further require that all returned results must not only trust Brad at least 10% but must also be a member of the group “BradLikers.” More of these Sparql statements can be bundled together in a single BackendQuery to precisely specify what results are desired. Backend queries are passed into the backend through the IBackend interface. Tags and Comments Tags can be used to provide additional information about a piece of content. Since tags are backend objects, each time a tag is created it is allocated a new Guid, regardless of whether or not the tag’s title has been used before. The relations between tags and the thing they describe are stored as relations in the relations table in the format “TagId Tags ContentId.” This general method of storing tags allows a wide number of things to be stored in the same fashion, such as comments and messages. Database Access The database connection, username, and password are stored in the Windows registry. The system has been designed to be agnostic with respect to the type of database used. An adapter class, DatabaseConnection, abstracts all of the implementation-specific details and serves as a factory for connections, commands, and parameters. Backend Database Tables There is a visitor on IContent objects which performs different actions based on the content type of the object. Specifically, this visitor is used to determine how to store and retrieve different types of data. While there are some general tables which handle all of the common properties of content, the additional information which is specific to different content types is stored in separate tables. Each content type, in essence, knows how to store and retrieve itself. To add a new content type, add a new entry to the ContentType enum, and add new entries to the extended visitors for the new content type; everything else remains the same. One alternative to the extended visitor pattern would be to have different public sub-interfaces of IContent, one for each type of content. Implementations of these interfaces would contain the contentspecific knowledge about how to store and retrieve data. The benefit of this is that a general purpose dictionary of key-value pairs would not be necessary, and different types of data would have methods specifically for retrieving it. Another alternative to having multiple tables is to have one large table which serves as a list of key-value pairs for all types of content. The benefit of this that once the backend is written it would never need to be changed, even when adding new types of content. The downside is that the key-value model may be too inflexible for some types of data. Category Stores the hierarchy of all categories. Id Name CreationDate Parent uuid char varying(200) timestamp uuid Guid to uniquely identify this category. Name of this category. Time this category was created. The parent category. Content Stores all universal properties of content objects. Id Type Name Published Description CreationDate Deleted Price uuid smallint char varying(200) boolean char varying(4000) timestamp boolean numeric(8,2) Guid to uniquely identify this content object. Specification of the type of content. The name to display of the content object. Flag to determine if this content should be viewable or not. Description of the type of content. The time this content was created. Flag to determine if this content has been deleted. Price at which this should be sold. 0 if content is free. ContentProperties A dictionary of additional non-searchable properties about a piece of content. Id Name Value uuid char varying(50) char varying(200) The Id of the piece of content. The key of the dictionary. The value of the dictionary. Data A mapping of postgre sql data streams to content objects. Id Data VersionKey uuid oid char varying(30) The Id of the piece of content. The postgre sql data stream id. The key to know what version of data this is. Entity Stores all universal properties of entities. Id Email Password Name Validated Deleted CreationDate Username Banned uuid char varyring(150) bytea char varying(500) boolean boolean timestamp char varying(32) boolean The Id of this entity. The email address of this entity. Also used as a login. MD5 hash of password appended to user id. The display name of this entity. Flag for if this entity has had its real-world identity validated. Flag for if this entity has been deleted. The time this entity was added to the system. The username of this entity. Flag for if this entity has been banned from the system. EntityProperties A dictionary of additional non-searchable properties about an entity. Id Name Value uuid char varying(50) char varying(200) The Id of the entity. The key of the dictionary. The value of the dictionary. IdType A mapping of what type of object every Guid represents. Id Type uuid integer The Id of the object. The SearchTypes value of the type of object. Notifications A listing of all notifications the system has generated. QueueOrder ObjectNotified Handled Notification serial uuid boolean bytea The order of the notification. The object being notified. Flag for whether this notification has been handled. The data of the notification. PageViews A list of the number of times each entity has viewed each content. UserId ContentId ViewCount LastView uuid uuid integer timestamp The Id of the entity. The content being viewed. The number of times the content has been viewed. The time the user last viewed this piece of content. Permissions Listing of all users and the objects they have permission for. ObjectId EntityId Action uuid uuid integer The Id of the object the user has permission on. The Entity who has permission. The type of permission the entity has. Relation The RDF table containing all searchable relations between objects. Subject Predicate Object Value CreationDate uuid integer uuid integer timestamp The Id of the subject. The RelationTypes value of the type of relation. The id of the object. A measure of the “strength” of the relationship. The time the relation was created. SystemSettings A dictionary of system-wide properties. Name Value char varying(25) char varying(25) The key of the dictionary. The value of the dictionary. Tag Stores the information about a tag. Id Name CreationDate TagType Description uuid char varying(200) timestamp integer char varying(4000) The id of this tag. The title of the tag. The time the tag was created. The TagType value of the type of tag it is. A longer description of the tag. TagProperties A dictionary of additional non-searchable properties about a tag. Id Name Value uuid char varying(50) char varying(200) The Id of the tag. The key of the dictionary. The value of the dictionary. TransactionBuyer A listing of all of the unique information about a single transaction, including the buyer. TransactionId BuyerId Date Delegate uuid uuid timestamp bytea The Id of the transaction. The id of the buyer. The date this transaction occurred. The action to perform once the payment has gone through. TransactionContents A listing of all the pieces of content within a particular transaction. TransactionId ContentId Price uuid uuid numeric(9, 2) The Id of the transaction. The content item. The price this content was sold for in this transaction. Backend Files Access Control IAccessControl.cs & AccessControl.cs Interface and implementation for querying access permissions of Entities upon BackendObjects. IAccessControlList.cs & AccessControlList.cs Interface and implementation to model the set of access permissions upon a single BackendObject. AccessControlSettingsHelper.cs Services for changing access permissions upon a BackendObject. IAccessControlObject.cs Definition of the interface implemented by BackendObjects which supports access control functionality. RelationsPermissionCalculator.cs Service that determines which members (subject and object) of a specific relation type need what permissions in order to conduct various access operations on that relationship. Backend IBackend.cs, IBackendInternal.cs, & Backend.cs IBackend.cs contains the interface provided by the Backend to other modules. IBackendInteral.cs contains extra interface methods available only within the Backend module. Backend.cs contains the combined implementation of these interfaces. BackendFactory.cs Provides a safe access point for other modules to create a Backend object for use. DatabaseConnection.cs Provides a generic database interface for other Backend classes, to encapsulate and limit the necessary changes for using different database solutions. GetBackendObjectVisitor.cs Visitor that creates an appropriate specific object type from a generic BackendObject. BackendObject IBackendObject.cs & BackendObject.cs IBackendObject.cs describes the interface used by other modules to interact with generic Backend objects, and BackendObject.cs contains the implementation of Backend objects. IBackendObjectCollection.cs Defines a Collection<BackendObject> interface, which is implemented for User and Content collections in their respective sections. BackendQuery BackendQuery.cs Implementation of a model of a Sparql-like search query for data modeled by the Backend module. Other modules build a BackendQuery, and then send it to the Backend for processing. IQueryValidator.cs & QueryValidator.cs Interface and abstract class which provide a standard way to validate and get other information about the various query classes (BackendQuery, Statement, and StatementMember). IStatementMember.cs & StatementMember.cs Interface and implementation of one part of a Sparql-like statement (subject, predicate, object, or value), including properties that define how it is used in the query (whether it is a variable, etc.). SparqlTranslator.cs Contains the code to convert a BackendQuery object into an actual database query. Statement.cs Implementation which models a single Sparql-like statement. BackendRelation IBackendRelation.cs & BackendRelation.cs Contain the interface and implementation for the Backend’s model of a single relationship. Category ICategory.cs & Category.cs Contain the interface and implementation for the Backend’s model of a single content category. Collections IDBDictionary.cs, DBDictionary.cs, & ADBDictionary.cs Interface, abstract class, and implementation of the database code which supports the properties dictionary defined in the BackendDictionary class. IBackendDictionary.cs & BackendDictionary.cs Contains the interface and implementation for a dictionary, which as a member of an object provides an easy way to add new properties to the object. Sometimes referred to as a “properties dictionary.” GroupCollection.cs Framework for handling groups of users using the relationships model. Content IContent.cs & Content.cs Interface and implementation for the model of a single piece of content. ContentCollection.cs Implementation of Collection<Content> interface, for handling groupings of content. DB Configuration CreateTables.sql SQL scripts for creating all the database tables. DevDB.reg & ProdDB.reg Registry scripts used to set up database connection constants on the server. Entity IEntity.cs & Entity.cs Interface and implementation of the generic model of a human entity in the system. EntityCollection.cs Implementation of Collection<Entity> interface, to model a group of users. IUser.cs & User.cs Extension of the entity classes to represent a single user. Exceptions BackendDisposedException.cs Thrown when there is an attempt to use a Backend object that has already been closed. BannedUserException.cs Thrown when a banned user attempts to log in. IllegalAccessException.cs Thrown when an access to some content, etc. is attempted without the necessary permissions. IncorrectPasswordException.cs Thrown when a user enters a password that does not match the password on file. ObjectNotCreatedException.cs Thrown when creation of an object fails. ObjectNotDeletedException.cs Thrown when deletion of an object fails. ObjectNotFoundException.cs Thrown when a matching object is not found during a get request. ObjectNotInitializedException.cs Thrown when an object fails to initialize. UserAlreadyExistsException.cs Thrown when there is an attempt to create a user with an already existing username or email. UserNotValidatedException.cs Thrown when a user that hasn’t been validated attempts to log in. Notifications Described in the Notifications documentation. Relationships Described in the Relationships documentation. Tags ITag.cs & Tag.cs Interface and implementation used to model a single tag. Transactions IPaymentDelegate.cs Framework for the use of delegates in the payment system. ITransaction.cs & Transaction.cs Interface and implementation of the model of a single monetary transaction in the payment system. ITransactionMember.cs & TransactionMember.cs Interface and implementation of the representation of a piece of content that is purchased as part of a transaction. Miscellaneous AuthenticationSystem.cs Contains methods used by the WebApp module to log users and check if they are logged in. BackendConstants.cs Contains constants used only by the Backend module, including table column names. NpgsqlStream.cs Contains methods specific to interacting with a PostgreSql database.