Download 02 Backend Architecture

Backend Architecture
Version Description
0.1
Creation of initial document.
0.2
Description of the public API to the backend, and the design of
the database.
0.2.5
Converted the document to new documentation standard.
1.0
1.1
Updated document to reflect final release.
Added list of Backend files
Author
Aaron Cottle
Aaron Cottle
Date
10/18/2007
10/18/2007
Felipe
Serrano
Aaron Cottle
M. Freeburg
10/25/2007
12/21/2007
12/26/2007
Contents
Introduction .................................................................................................................................................. 4
Public API ...................................................................................................................................................... 4
Data Flow .................................................................................................................................................. 4
Backend Object ..................................................................................................................................... 4
Content ................................................................................................................................................. 5
Entities .................................................................................................................................................. 5
Searching................................................................................................................................................... 5
Searching....................................................................................................................................................... 6
Tags and Comments ...................................................................................................................................... 6
Database Access............................................................................................................................................ 6
Backend Database Tables ............................................................................................................................. 7
Category .................................................................................................................................................... 7
Content ..................................................................................................................................................... 7
ContentProperties..................................................................................................................................... 7
Data ........................................................................................................................................................... 8
Entity ......................................................................................................................................................... 8
EntityProperties ........................................................................................................................................ 8
IdType........................................................................................................................................................ 8
Notifications .............................................................................................................................................. 8
PageViews ................................................................................................................................................. 9
Permissions ............................................................................................................................................... 9
Relation ..................................................................................................................................................... 9
SystemSettings .......................................................................................................................................... 9
Tag ............................................................................................................................................................. 9
TagProperties ............................................................................................................................................ 9
TransactionBuyer .................................................................................................................................... 10
TransactionContents ............................................................................................................................... 10
Backend Files............................................................................................................................................... 11
Access Control ......................................................................................................................................... 11
IAccessControl.cs & AccessControl.cs ................................................................................................. 11
IAccessControlList.cs & AccessControlList.cs ...................................................................................... 11
AccessControlSettingsHelper.cs.......................................................................................................... 11
IAccessControlObject.cs ...................................................................................................................... 11
RelationsPermissionCalculator.cs ....................................................................................................... 11
Backend................................................................................................................................................... 11
IBackend.cs, IBackendInternal.cs, & Backend.cs ................................................................................ 11
BackendFactory.cs .............................................................................................................................. 11
DatabaseConnection.cs....................................................................................................................... 11
GetBackendObjectVisitor.cs ............................................................................................................... 11
BackendObject ........................................................................................................................................ 11
IBackendObject.cs & BackendObject.cs.............................................................................................. 11
IBackendObjectCollection.cs............................................................................................................... 11
BackendQuery ......................................................................................................................................... 12
BackendQuery.cs................................................................................................................................. 12
IQueryValidator.cs & QueryValidator.cs ............................................................................................. 12
IStatementMember.cs & StatementMember.cs ................................................................................ 12
SparqlTranslator.cs ............................................................................................................................. 12
Statement.cs ....................................................................................................................................... 12
BackendRelation ..................................................................................................................................... 12
IBackendRelation.cs & BackendRelation.cs ........................................................................................ 12
Category .................................................................................................................................................. 12
ICategory.cs & Category.cs ................................................................................................................. 12
Collections ............................................................................................................................................... 12
IDBDictionary.cs, DBDictionary.cs, & ADBDictionary.cs ..................................................................... 12
IBackendDictionary.cs & BackendDictionary.cs .................................................................................. 12
GroupCollection.cs .............................................................................................................................. 12
Content ................................................................................................................................................... 12
IContent.cs & Content.cs .................................................................................................................... 12
ContentCollection.cs ........................................................................................................................... 13
DB Configuration ..................................................................................................................................... 13
CreateTables.sql .................................................................................................................................. 13
DevDB.reg & ProdDB.reg .................................................................................................................... 13
Entity ....................................................................................................................................................... 13
IEntity.cs & Entity.cs............................................................................................................................ 13
EntityCollection.cs............................................................................................................................... 13
IUser.cs & User.cs ............................................................................................................................... 13
Exceptions ............................................................................................................................................... 13
BackendDisposedException.cs ............................................................................................................ 13
BannedUserException.cs..................................................................................................................... 13
IllegalAccessException.cs .................................................................................................................... 13
IncorrectPasswordException.cs .......................................................................................................... 13
ObjectNotCreatedException.cs ........................................................................................................... 13
ObjectNotDeletedException.cs ........................................................................................................... 13
ObjectNotFoundException.cs ............................................................................................................. 13
ObjectNotInitializedException.cs ........................................................................................................ 13
UserAlreadyExistsException.cs ........................................................................................................... 14
UserNotValidatedException.cs ........................................................................................................... 14
Notifications ............................................................................................................................................ 14
Relationships ........................................................................................................................................... 14
Tags ......................................................................................................................................................... 14
ITag.cs & Tag.cs ................................................................................................................................... 14
Transactions ............................................................................................................................................ 14
IPaymentDelegate.cs .......................................................................................................................... 14
ITransaction.cs & Transaction.cs......................................................................................................... 14
ITransactionMember.cs & TransactionMember.cs............................................................................. 14
Miscellaneous ......................................................................................................................................... 14
AuthenticationSystem.cs .................................................................................................................... 14
BackendConstants.cs .......................................................................................................................... 14
NpgsqlStream.cs ................................................................................................................................. 14
Introduction
The backend provides a display- independent way to store, retrieve, and search for data. This document
broadly describes both the public-facing API and the internal architecture hidden from the outside,
including database schemas. For a detailed description of all methods available to be called, consult the
actual code comments. IBackend.cs contains all of the methods that can be called from the backend and
their documentation.
Public API
There are a number of public data interfaces exposed, each representing an object in the system. The
top-level interface is IBackendObject which provides a number of attributes common to all objects in the
backend. The IEntity, IUser, and IContent classes derive from this, each adding methods specific to users
and content. Another interface, IBackendObjectCollection, is also exposed, which not only derives from
IBackendObject but also implements ICollection. This means that in addition to being able to be treated
as a backend object it contains a number of backend objects. In this way, groups of, say, users can be
treated in exactly the same way as a single object. No concrete implementations of any of these
interfaces are exposed.
One adapter interface is exposed which houses all methods which can be called on the backend to store
or retrieve data: IBackend. No concrete implementations are exposed, but an additional class,
BackendFactory, has a public static method which creates one. The only access to the backend must go
through the IBackend implementation provided by BackendFactory. IBackend controls all access to
database connections and opens a new connection when it’s instantiated, so it’s necessary to enclose all
instances of it in a “using” block in C# to make sure it’s properly disposed of and connections aren’t left
open. IBackend also manages transactions, allowing multiple changes to happen to the database and be
committed concomitantly with a call to SaveChanges(). Any changes made to the database which are
not eventually followed by a call to SaveChanges() will be rolled back when the instance of the backend
is disposed.
Data Flow
Backend Object
There are a number of features common to all objects stored in the backend. To create a new object,
parameters are passed into the proper method on IBackend. The backend then allocates a new Guid
based upon what type of object is being created, saves the given information to the database, and
returns the created object. All mutator methods on a backend object write directly to the database, but
of course do not actually change the database until SaveChanges() is called on the IBackend.
Each backend object internally retains a reference to the backend which created it in order to change
the database when needed. This means that once the backend is disposed (such as when the using block
has closed) any backend objects, even if still in scope, cannot change themselves. It’s best to think of a
backend object as a reader on a stream: a reader can be set to retrieve information from a stream, but if
the stream is closed, the reader becomes worthless. Most backend objects locally cache data values, so
it may be possible to read information about a backend object after the backend is closed, but trying to
set information will throw an exception.
Backend object can be retrieved from IBackend by passing a certain amount of uniquely identifying
information to the proper method. All backend objects have a Guid which uniquely identifies them, and
it is even possible to determine the type of an object given only its Guid.
Content
Content objects represent any type of user-uploaded data. Content objects, once created, can produce
any number of different data streams which can be written to or read. Each stream can be retrieved by a
key. For example, a newly uploaded video will have the “default” stream of the actual video content
itself, but it may also have a “thumbnail” stream, which provides a small snapshot of a frame of the
video.
Entities
Any agent viewing or creating content in the system is classified as an IEntity. The IEntity interface uses
three types of uniquely identifying information: the Guid universal to all backend objects, plus an email
address and a username. Each entity has various permissions on types of content. For example, the
author of a piece of content would have permission to view and edit it, while an entity who has
purchased content could view but not edit it, while an entity who has not purchased the content cannot
view the content in full, but could perhaps view a preview of it.
Each entity also has a password associated with it. This password can be verified with the stored one in
the database to determine if the entity can successfully log in. Entities can also be “banned” from the
system temporarily for violating terms of use, at the discretion of an administrator, which prevents them
from logging in.
One entity can also “sponsor” one or more other entities, showing that the sponsor favors the
sponsored entity’s content and wishes to promote him or her. The backend provides methods for
obtaining both the list of all sponsors of an entity and all entities whom this entity is sponsoring.
Searching
Searching goes through several stages. For a search starting from a user’s typed text, the search module
parses and translates the query into a search tree. The relations module then translates the search tree
into a series of Sparql statements which are bundled together inside a BackendQuery. Backend queries
are accepted in IBackend, turned into Sql, and executed, returning all information which matched the
query. Backend queries can also be formed manually, without starting from user input.
Searching
Searching pulls information primarily from the Relations table in the database. The relations table is an
RDF-style database table, formed based on the “subject-predicate-object” form. In brief, two backend
objects form the subject and the object; these can be thought of as nouns. The predicate describes the
relation between the two backend objects; this can be thought of as a verb. The RDF language, which is
a W3C standard, is excellent for showing that different types of relationships between objects exist, but
searching also needs the ability to store the “strength,” or value, of a relationship. The value can be
thought of as an adverb. As an example, the simple sentence “Matt trusts Brad 10%” can thus be stored
in the relation table.
Sparql is a language designed to query RDF-style databases. We again have extended this slightly to
incorporate the notion of the strength of a relationship. Sparql allows for sentences as above to be
formed, but also allows for variables to take the place of some literals, representing unknown
information. The pseudo-Sparql sentence “?truster Trusts Brad > 10%” is a simple query to find all
people who trust Brad at least 10%. Joining this to the sentence “?truster MemberOf BradLikers” will
further require that all returned results must not only trust Brad at least 10% but must also be a
member of the group “BradLikers.” More of these Sparql statements can be bundled together in a single
BackendQuery to precisely specify what results are desired. Backend queries are passed into the
backend through the IBackend interface.
Tags and Comments
Tags can be used to provide additional information about a piece of content. Since tags are backend
objects, each time a tag is created it is allocated a new Guid, regardless of whether or not the tag’s title
has been used before.
The relations between tags and the thing they describe are stored as relations in the relations table in
the format “TagId Tags ContentId.” This general method of storing tags allows a wide number of things
to be stored in the same fashion, such as comments and messages.
Database Access
The database connection, username, and password are stored in the Windows registry. The system has
been designed to be agnostic with respect to the type of database used. An adapter class,
DatabaseConnection, abstracts all of the implementation-specific details and serves as a factory for
connections, commands, and parameters.
Backend Database Tables
There is a visitor on IContent objects which performs different actions based on the content type of the
object. Specifically, this visitor is used to determine how to store and retrieve different types of data.
While there are some general tables which handle all of the common properties of content, the
additional information which is specific to different content types is stored in separate tables. Each
content type, in essence, knows how to store and retrieve itself. To add a new content type, add a new
entry to the ContentType enum, and add new entries to the extended visitors for the new content type;
everything else remains the same.
One alternative to the extended visitor pattern would be to have different public sub-interfaces of
IContent, one for each type of content. Implementations of these interfaces would contain the contentspecific knowledge about how to store and retrieve data. The benefit of this is that a general purpose
dictionary of key-value pairs would not be necessary, and different types of data would have methods
specifically for retrieving it.
Another alternative to having multiple tables is to have one large table which serves as a list of key-value
pairs for all types of content. The benefit of this that once the backend is written it would never need to
be changed, even when adding new types of content. The downside is that the key-value model may be
too inflexible for some types of data.
Category
Stores the hierarchy of all categories.
Id
Name
CreationDate
Parent
uuid
char varying(200)
timestamp
uuid
Guid to uniquely identify this category.
Name of this category.
Time this category was created.
The parent category.
Content
Stores all universal properties of content objects.
Id
Type
Name
Published
Description
CreationDate
Deleted
Price
uuid
smallint
char varying(200)
boolean
char varying(4000)
timestamp
boolean
numeric(8,2)
Guid to uniquely identify this content object.
Specification of the type of content.
The name to display of the content object.
Flag to determine if this content should be viewable or not.
Description of the type of content.
The time this content was created.
Flag to determine if this content has been deleted.
Price at which this should be sold. 0 if content is free.
ContentProperties
A dictionary of additional non-searchable properties about a piece of content.
Id
Name
Value
uuid
char varying(50)
char varying(200)
The Id of the piece of content.
The key of the dictionary.
The value of the dictionary.
Data
A mapping of postgre sql data streams to content objects.
Id
Data
VersionKey
uuid
oid
char varying(30)
The Id of the piece of content.
The postgre sql data stream id.
The key to know what version of data this is.
Entity
Stores all universal properties of entities.
Id
Email
Password
Name
Validated
Deleted
CreationDate
Username
Banned
uuid
char varyring(150)
bytea
char varying(500)
boolean
boolean
timestamp
char varying(32)
boolean
The Id of this entity.
The email address of this entity. Also used as a login.
MD5 hash of password appended to user id.
The display name of this entity.
Flag for if this entity has had its real-world identity validated.
Flag for if this entity has been deleted.
The time this entity was added to the system.
The username of this entity.
Flag for if this entity has been banned from the system.
EntityProperties
A dictionary of additional non-searchable properties about an entity.
Id
Name
Value
uuid
char varying(50)
char varying(200)
The Id of the entity.
The key of the dictionary.
The value of the dictionary.
IdType
A mapping of what type of object every Guid represents.
Id
Type
uuid
integer
The Id of the object.
The SearchTypes value of the type of object.
Notifications
A listing of all notifications the system has generated.
QueueOrder
ObjectNotified
Handled
Notification
serial
uuid
boolean
bytea
The order of the notification.
The object being notified.
Flag for whether this notification has been handled.
The data of the notification.
PageViews
A list of the number of times each entity has viewed each content.
UserId
ContentId
ViewCount
LastView
uuid
uuid
integer
timestamp
The Id of the entity.
The content being viewed.
The number of times the content has been viewed.
The time the user last viewed this piece of content.
Permissions
Listing of all users and the objects they have permission for.
ObjectId
EntityId
Action
uuid
uuid
integer
The Id of the object the user has permission on.
The Entity who has permission.
The type of permission the entity has.
Relation
The RDF table containing all searchable relations between objects.
Subject
Predicate
Object
Value
CreationDate
uuid
integer
uuid
integer
timestamp
The Id of the subject.
The RelationTypes value of the type of relation.
The id of the object.
A measure of the “strength” of the relationship.
The time the relation was created.
SystemSettings
A dictionary of system-wide properties.
Name
Value
char varying(25)
char varying(25)
The key of the dictionary.
The value of the dictionary.
Tag
Stores the information about a tag.
Id
Name
CreationDate
TagType
Description
uuid
char varying(200)
timestamp
integer
char varying(4000)
The id of this tag.
The title of the tag.
The time the tag was created.
The TagType value of the type of tag it is.
A longer description of the tag.
TagProperties
A dictionary of additional non-searchable properties about a tag.
Id
Name
Value
uuid
char varying(50)
char varying(200)
The Id of the tag.
The key of the dictionary.
The value of the dictionary.
TransactionBuyer
A listing of all of the unique information about a single transaction, including the buyer.
TransactionId
BuyerId
Date
Delegate
uuid
uuid
timestamp
bytea
The Id of the transaction.
The id of the buyer.
The date this transaction occurred.
The action to perform once the payment has gone through.
TransactionContents
A listing of all the pieces of content within a particular transaction.
TransactionId
ContentId
Price
uuid
uuid
numeric(9, 2)
The Id of the transaction.
The content item.
The price this content was sold for in this transaction.
Backend Files
Access Control
IAccessControl.cs & AccessControl.cs
Interface and implementation for querying access permissions of Entities upon BackendObjects.
IAccessControlList.cs & AccessControlList.cs
Interface and implementation to model the set of access permissions upon a single BackendObject.
AccessControlSettingsHelper.cs
Services for changing access permissions upon a BackendObject.
IAccessControlObject.cs
Definition of the interface implemented by BackendObjects which supports access control functionality.
RelationsPermissionCalculator.cs
Service that determines which members (subject and object) of a specific relation type need what
permissions in order to conduct various access operations on that relationship.
Backend
IBackend.cs, IBackendInternal.cs, & Backend.cs
IBackend.cs contains the interface provided by the Backend to other modules. IBackendInteral.cs
contains extra interface methods available only within the Backend module. Backend.cs contains the
combined implementation of these interfaces.
BackendFactory.cs
Provides a safe access point for other modules to create a Backend object for use.
DatabaseConnection.cs
Provides a generic database interface for other Backend classes, to encapsulate and limit the necessary
changes for using different database solutions.
GetBackendObjectVisitor.cs
Visitor that creates an appropriate specific object type from a generic BackendObject.
BackendObject
IBackendObject.cs & BackendObject.cs
IBackendObject.cs describes the interface used by other modules to interact with generic Backend
objects, and BackendObject.cs contains the implementation of Backend objects.
IBackendObjectCollection.cs
Defines a Collection<BackendObject> interface, which is implemented for User and Content collections
in their respective sections.
BackendQuery
BackendQuery.cs
Implementation of a model of a Sparql-like search query for data modeled by the Backend module.
Other modules build a BackendQuery, and then send it to the Backend for processing.
IQueryValidator.cs & QueryValidator.cs
Interface and abstract class which provide a standard way to validate and get other information about
the various query classes (BackendQuery, Statement, and StatementMember).
IStatementMember.cs & StatementMember.cs
Interface and implementation of one part of a Sparql-like statement (subject, predicate, object, or
value), including properties that define how it is used in the query (whether it is a variable, etc.).
SparqlTranslator.cs
Contains the code to convert a BackendQuery object into an actual database query.
Statement.cs
Implementation which models a single Sparql-like statement.
BackendRelation
IBackendRelation.cs & BackendRelation.cs
Contain the interface and implementation for the Backend’s model of a single relationship.
Category
ICategory.cs & Category.cs
Contain the interface and implementation for the Backend’s model of a single content category.
Collections
IDBDictionary.cs, DBDictionary.cs, & ADBDictionary.cs
Interface, abstract class, and implementation of the database code which supports the properties
dictionary defined in the BackendDictionary class.
IBackendDictionary.cs & BackendDictionary.cs
Contains the interface and implementation for a dictionary, which as a member of an object provides an
easy way to add new properties to the object. Sometimes referred to as a “properties dictionary.”
GroupCollection.cs
Framework for handling groups of users using the relationships model.
Content
IContent.cs & Content.cs
Interface and implementation for the model of a single piece of content.
ContentCollection.cs
Implementation of Collection<Content> interface, for handling groupings of content.
DB Configuration
CreateTables.sql
SQL scripts for creating all the database tables.
DevDB.reg & ProdDB.reg
Registry scripts used to set up database connection constants on the server.
Entity
IEntity.cs & Entity.cs
Interface and implementation of the generic model of a human entity in the system.
EntityCollection.cs
Implementation of Collection<Entity> interface, to model a group of users.
IUser.cs & User.cs
Extension of the entity classes to represent a single user.
Exceptions
BackendDisposedException.cs
Thrown when there is an attempt to use a Backend object that has already been closed.
BannedUserException.cs
Thrown when a banned user attempts to log in.
IllegalAccessException.cs
Thrown when an access to some content, etc. is attempted without the necessary permissions.
IncorrectPasswordException.cs
Thrown when a user enters a password that does not match the password on file.
ObjectNotCreatedException.cs
Thrown when creation of an object fails.
ObjectNotDeletedException.cs
Thrown when deletion of an object fails.
ObjectNotFoundException.cs
Thrown when a matching object is not found during a get request.
ObjectNotInitializedException.cs
Thrown when an object fails to initialize.
UserAlreadyExistsException.cs
Thrown when there is an attempt to create a user with an already existing username or email.
UserNotValidatedException.cs
Thrown when a user that hasn’t been validated attempts to log in.
Notifications
Described in the Notifications documentation.
Relationships
Described in the Relationships documentation.
Tags
ITag.cs & Tag.cs
Interface and implementation used to model a single tag.
Transactions
IPaymentDelegate.cs
Framework for the use of delegates in the payment system.
ITransaction.cs & Transaction.cs
Interface and implementation of the model of a single monetary transaction in the payment system.
ITransactionMember.cs & TransactionMember.cs
Interface and implementation of the representation of a piece of content that is purchased as part of a
transaction.
Miscellaneous
AuthenticationSystem.cs
Contains methods used by the WebApp module to log users and check if they are logged in.
BackendConstants.cs
Contains constants used only by the Backend module, including table column names.
NpgsqlStream.cs
Contains methods specific to interacting with a PostgreSql database.