Download Portals, Portlets, and Clients to Grid Services

Portals, Portlets, and
Clients to Grid Services
Marlon Pierce
Community Grids Lab
Indiana University
[email protected]
Useful Links

These slides are available from here:
• http://www.servogrid.org/slide/iSERVO/
IRIS_WebServices_2005/

General portlet tutorial information is
available from here:
• http://www.servogrid.org/slide/iSERVO/
NMI/OGCE2TutorialMaterial/

Grid portlets are available from here:
• http://www.collab-ogce.org
Outline



Globus toolkit review
Computational Portals and Portlets
Building Grid Portlets with the Java
CoG Kit
Purpose of Talk

I am not affiliated with Globus and do not develop
Globus Web Services.
• I just use Globus services as-is.
• My slides are from this point of view.



My emphasis is on building Web Portals from
portlets as clients to Grid services using the Java
COG kit.
The Java CoG kit project is lead by Gregor von
Laszewski at Argonne National Labs
I work with Gregor through the Open Grid
Computing Environments project.
• We build portlets using the Java CoG

I also lead portlet and Web Service development for
the NASA SERVOGrid project.
• Grid services, GIS services
SERVOGrid Summary

Portlet-based portal

• Use Geographical Information
System standards.
• Web Feature Service
• With portal extensions to
support secure remote
portlets.

“Execution Grid” Web
Services


• Run remote tasks using
Ant job managers.
• Move files around (upload,
download, crossload).
• Maintain persistent user
session state


Provide data portal
applications.
• Monitor jobs
quakesim.jpl.nasa.org
Encodes GPS, Seismic, and
Fault data archives as GML.
Working on high performance
web services.
• Web Map Service


Generates interactive web
maps using features and other
map servers.
Streaming video version in
development.
• Sensor Grid Services
“Context”
• Application metadata to
describe applications.
Data Grid Web Services



Real time streaming GPS
Data mining on streams
Information Grid Services
• Extended UDDI
• WS-Context for session
metadata.
www.crisisgrid.org
Globus Toolkit Overview


The Globus Toolkit provides foundations for
building “Virtual Organizations.”
Single Sign-On Authentication
• Log in once, access all machines on your Grid.

Grid Security Infrastructure
• Public Key Infrastructure
• Generic Security Service programming interfaces

Secure Remote Resource Access
• Remotely execute tasks.
• Interact with both shells and queuing systems.


Secure, high performance file transfer
Information services for your grid resources.
Globus Toolkit 2.x

GT 2 services implemented as Unix services.
• You quite commonly manage them through /etc/xinetd.d or
similar.

Standard services in xinetd.d:
• GRAM gatekeeper for running remote tasks.
• GridFTP for file transfers

Upload, download, and third party.
• MDS information services
• MyProxy: a remote credential repository so that you don’t
need to keep lots of copies of your private key.


Jim Basney at NCSA is the MyProxy guy.
Support for both remote shell operations and
queuing systems.
• PBS and LSF tend to be best supported.
• You may be able to find others.

Many other Globus-related tools have come and
gone.
Globus Toolkit and Web Services

Globus and others in the Global Grid Forum realized that they
needed to provide more than just a bag of services
•
•
•
•



Needed a framework for building new services.
Needed standard languages for writing service APIs
Standard messaging envelopes.
Web Services were the natural fit.
The Open Grid Services Architecture (OGSA) outlines the
basic premise of the Globus approach to Web Services.
Globus Toolkit 3.x was the initial implementation of these
ideas.
But it was found to be unsustainable.
• Initial spec (“OGSI”) was too monolithic, took to long to approve,
made custom extensions to WSDL, had portions that were
superseded by the main stream Web Service community, and was
disconnected from the general community (OASIS, W3C).
• See http://www-128.ibm.com/developerworks/library/wsresource/ogsi_to_wsrf_1.0.pdf for some discussion.

GT 4 release (April 2005) attempts to correct these problems.
Some Globus 2 Services v. Globus 4
Web Services


PKI X.509 certificates used for
mutual authentication
GSI Security


• GSSAPI-based socket
connections


• Your certificates work in both
GT versions.
GRAM
• Uses Resource Specification
Language (RSL)
• Binds to PBS, LSF, and possibly
other schedulers with third
party plugins (support may
vary).

WS-GRAM
• Replaces RSL with XML Job
Descriptors.
• Uses Reliable File Transfer and
GridFTP to support file staging
and multiple jobs (which is a
pain to install).
• Binds to PBS, LSF
GridFTP
• Note GT 4 includes GridFTP 2.0,
which is a different codebase
from GridFTP 1.17 bundled in
previous Globus releases.
• We have reported some
incompatibilities, which I think
have been addressed in GT
4.0.1.
Still PKI X.509 based mutual
authentication…
But used now with Web Service
Security: SSL and WS-Security

Reliable File Transfer (RFT)
• Web Service-based
• Uses “scheduling” techniques to
do batch file transfers that don’t
require persistent client
connections.
• Builds on GridFTP.
Grid (and Web Service) Clients

Web and Grid Services run in service
hosting environments.
• Apache Axis, Globus WS container, etc.

Clients likewise need to run in client hosting
environments.
• Graphical Use Interfaces, Desktops, Web
Browser Portals
• But also grid and web service shells.


Gridshell, HPSearch
We will focus on portals, since these are the
most common and have standards.
Building Grid Portals
A Famous Web Portal
A Famous Portal, After Login
What to Notice

After logging in, my colors, layouts, and
content all changed.
• I get my stock list, my Bloomington weather, my
news stories, etc.
• I got rid of “Garfield”

As we will see later, each of these content
fragments (encircled) is managed by a thing
called a portlet…
• … I don’t guarantee that this is true for Yahoo’s
web site but it is true for a large class of
enterprise Java portal systems.

Portlets are the key to portal software reuse.
Let 10,000 Flowers Bloom

Many portal projects have been launched since late
’90s.
• HotPage from SDSC, NCSA efforts, DOD, DOE Portals, NASA
IPG
• 2002 Special Issue of Concurrency and Computation:
Practice and Experience.

The field continues to be active
• An upcoming issue of Concurrency will survey the current
field.
• Based on Global Grid Forum 14 Science Gateway workshop
in June 2005.
• GCE2005 workshop at Supercomputing 05.


http://www.ggf.org/ggf_events_lodging_ggf15.htm
Global Grid Forum’s Grid Computing Environments
(GCE) Research Group
• Community forum

How do we share and reuse all of this work?
Three-Tiered Architecture
Portal User Interface
Grid and
Web
Protocols
JDBC,
Local, or
Remote
Connection
Portal
Client
Stub
Database
Service
Database
Portal
Client
Stub
Grid Resource
Broker Service
HPC
or
Compute Cluster
Portal
Client
Stub
Information
and
Data Services
Grid Information
Services, SRB
Three-tiered architecture is accepted standard for
accessing Grid and other services
What Is a Portlet?


A portlet is a piece of Java code that manages the
content of one section of a web portal’s HTML.
It can do anything else that a Java web application
can do.
• You can connect a portlet to a database, invoke a web
service, download an RSS feed, etc.


It lives in a portlet container, which creates,
manages, and destroys all the portlets of the portal.
Portlet containers are part of portals.
• Portals must do other things like manage login, users,
groups, layouts, etc.

JSR 168 standardizes two main things:
• How the portlet container manages portlet lifecycles
• How the portlets are programmed.
JSR 168 Overview
A General Review of JSR 168
What is JSR 168?

From the portlet development point of view,
it is really very simple:
• You write a java class that extends
GenericPortlet.
• You override/implement several methods
inherited from GenericPortlet.
• You use some supporting classes/interfaces



Many are analogous to their servlet equivalents
Some (portletsession) actually seem to be trivial
wrappers around servlet equivalents in Pluto.
I have a complete example in the extended
slides.
• See also tutorial slides.
Some GenericPortlet.java Methods
Method
Description
Init
Called when the portlet is created.
Override if you need to set initial params.
doView
Controls what happens immediately
before the portlet is displayed in view
mode. Normally you override this.
doHelp, doEdit
Other portlet display modes
processAction
Place for handling any <form> actions
before turning over to the display mode
method (like doView). You should override
this for web forms.
Some Supporting Classes/Interfaces
Class
Description
PortletContext
Similar to servlet context; get context info and
the RequestDispatcher from here.
PortletSession
Stores attribute information for a single portlet
application across multiple requests.
RenderRequest,
RenderResponse
The request and response objects available to
the doView() method. Similar to the normal
servlet request
ActionRequest,Actio
nResponse
The request and response objects available to
the processAction() method. Similar to the
servlet request and response objects.
PortletURL
Use this to create URLs that reference the portal.
PortletRequestDispa
tcher
Use this to include/forward to a JSP or servlet in
the same portlet app.
WindowState
See if you are in minimized, maximized, normal
state.
The Big Picture


As a portlet developer, the previous set of classes are all you
normally touch.
The portlet container (Pluto) is responsible for running your
portlets.
• Init, invoke methods, destroy.

Portlets have a very limited way of interacting with the
container.
• It is a black box->black hole.
• The API is basically one-way.
A Comment on Portlet Coding

The extended example makes some important and
dubious assumptions
• Developers would want to write a GenericPortlet extension
for every single portlet they develop.

And write really complicated processAction() and doView()
methods.
• Developers will like the specific JSR 168 portlet-style
Model-View-Controller that it forces on them.
• Developers will gladly ignore other development
methodologies/frameworks like Velocity, Struts, and Java
Server Faces.

Fortunately, these other development environments
can be mapped to portlet actions.
• In the OGCE project, we have developed support for
Velocity portlets.
• We are transitioning to Java Server Faces
Some Free JSR 168 Containers
Container
Comments
GridSphere
Supports JSF. I recommend. Popular container in
scientific portal community.
uPortal
Supports WSRP; JSF support planned. I have used
extensively, but prefer GridSphere.
LifeRay
Depends heavily on EJB. Interesting set of portlets
but too non-portable.
ExoPortal
I found this difficult to use, had some
incompatibilities with CoG jars.
StringBeans
Supports WSRP, JSF. I have not yet evaluated.
The Java CoG Kit
Gregor von Laszewski
Argonne National Laboratory
University of Chicago
[email protected]
http://www.cogkit.org
The Story So Far…




Globus has several interesting
services that can be used out of the
box.
Portlets can be used to build portals
out of reusable parts.
Portlets (as Java code) can also be
used to invoke remote services.
Now we will look at how to do this
for Grid applications.
CoG Kits


CoG Kits make Grid programming simple
and new technologies are easy to integrate
We focus on a CoG Kit for Java
• Python also available (K. Jackson, LBNL)
• Availability: Java CoG Kit since 1997

The CoG provides two important things
• A higher level programming client programming
environment than stubs.
• A shield against different versions of the Globus
toolkit

Same high level API works with GT 2.4, GT 3.0.2, GT
3.2.0, GT 3.2.1, GT 4.0.0
CoG Abstraction Layers
Nano
materials
BioDisaster
Informatics
Management
Applications
Portals
Development
Support
CoGGridfaces
GridfacesLayer
Layer
CoG
CoG
CoG GridIDE
GridIDE
CoGData
Dataand
andTask
TaskManagement
ManagementLayer
Layer
CoG
CoGAbstraction
AbstractionLayer
Layer
CoG
CoG
CoG
GT2
CoG
CoG
GT3
OGSI
classic
CoG
CoG
GT4
WS-RF
CoG
CoG
CoG
CoG
Condor Unicore
CoG
CoG
SSH
CoG
CoG
Others
Avaki
SETI
Grid Portlet Overview


In the tutorial examples, we look at writing JSR
168 portlets using JSP and Velocity.
Grid Portlets are just portlets, and portlets are just
Java applications.
• Use the Java CoG to make calls from the portlet code to
the desired Grid service.
• Can also make non-COG calls, such as to GridPort API.

These APIs are not be portlet-specific, since you
should be able to reuse the Grid APIs from other
Java clients.
• You can also build Web Services out of them.
• NCSA’s OGRE (general), Trebuchet (application specific)
are examples
• NCSA Tupelo also makes use of COG in its metadata
management services.
COG and Portlets


We use the COG to illustrate some
basic portlet programming features.
We use the COG4 APIs in our
ProxyManager, Job Submission, and
GridFTP portlets
• Hides differences between Globus Toolkit
versions.
• Is also structured to handle task-based
workflow and job composition.

For more detailed examples, see the
extended version of these slides.
Task
Handler
The class diagram
is the
same for all grid
tasks (running jobs,
modifying files,
moving data).
Task
Task
Specification
Service
Security
Context
Classes also abstract toolkit
provider differences. You set
these as parameters: GT2,
GT4, etd.
Service
Contact
Setting Up Task and Specification
Task task=new TaskImpl(“mytask”,
Task.JOB_SUBMISSION);
task.setProvider(“GT2”);
JobSpecification spec=
new JobSpecificationImpl();
spec.setExecutable(“rm”);
spec.setBatchJob(true);
spec.setArguments(“-r”);
…
task.setSpecification(spec);
Setting Up the Service and Security
Context
Service service=new
ServiceImpl(Service.JOB_SUBMISSION);
service.setProvider(“GT2”);
SecurityContext securityContext=
CoreFactory.newSecurityContext(“GT2”);
//Use cred object from ProxyManager
securityContext.setCredentials(cred);
service.setSecurityContext(
(SecurityContext)securityContext);
Set Up Service Contact and
Finish
ServiceContact serviceContact=
new ServiceContact(“myhost.myorg.org”);
service.setServiceContact(serviceContact);
task.setService(
Service.JOB_SUBMISSION_SERVICE,
service);
TaskHandler handler=new
GenericTaskHandler();
handler.submit(task);
Summary and Future Directions




Portlets provide a way to build reusable Web components.
The Java COG kit provides a nice abstraction layer for building
Grid portlets.
Portlets can be written using JSP, Velocity, and JSF.
In our OGCE project, we previously concentrated on Velocity
but are now moving to Java Server Faces.
• Provides a clear way to build reusable widgets, not just portlets.


AJAX techniques (such as Google Maps) provide a very way to
build interactive Web Service clients.
There is a larger problem: can we define a general user
interface component?
• Portlets would be a subclass
• But also include shells and desktop clients.

These components would also require standard supporting
services:
• Login, access control, layout management, ….
Basic Portlet
Programming Guide
In Action: Get started.
public class JunkPortlet extends
GenericPortlet {
public void init(){
//Do any initialization.
}
//Rest of the methods on following
slides go here.
…
}
Override doView()
protected void doView( RenderRequest req,
RenderResponse res)
throws PortletException, IOException {
//Include the desired JSP or HTML page.
//We could also use out to write directly to the
response.
WindowState state=req.getWindowState();
if(!state.equals(WindowState.MINIMIZED)) {
res.setContentType("text/html");
PortletRequestDispatcher rd=
getPortletContext().getRequestDispatcher(“MyJ
SP.jsp”);
rd.include(req,res);
}
}
The JSP Page
<portlet:defineObjects/>
<%
PortletSession
portletSession=renderRequest.getPortletSession();
portletSession.setAttribute("localattname","localattval");
PortletURL url=renderResponse.createActionURL();
String theActionString=url.toString();
%>
HTML Content is here.
A form is below.
<form method=post action="<%=theActionString%>">
<input type=…>
</form>
Some Notes

Include the <%portlet:definedObjects/%> tag, which will
instantiate renderRequest, renderResponse, and portletConfig
objects.
• You can then just use them, as with request, response, and other
JSP implicit objects.

The renderRequest gives you access to the PortletSession, if
you want to store session variables.
• One of the trouble points.


The renderResponse gives you access to the PortletURL object.
Use the PortletURL to generate a URL for the <form action>
• So that it points to portlet container and gets handled by the
processAction() method, rather than going of into space.
• Handle href URLs similarly.
• This is one of the sticking points.
Lastly, Override processAction()




When you invoke the form on
the previous JSP, the portlet
container will pass the action
handling to the processAction
method.
The ActionRequest can be
used to get any of the
<input> parameters in a way
similar to the usual
HttpServletRequest.
When the processAction
method returns, the container
then invokes the appropriate
do method (usually doView).
If you need to pass <form>
parameters on to doView, add
them to the ActionResponse.
• This will give them to the
RenderRequest.
• The example shows how to
add ALL parameters.
public void processAction
(ActionRequest request,
ActionResponse
actionResponse) throws
PortletException,
java.io.IOException {
//Process request parameters
…
//Add any other request
params
// to the renderRequest
actionResponse.setRenderPar
ameters(request.getParamete
rMap());
}
Deploying Portlet Applications

The portlet container (i.e. uPortal) runs as a
distinct web application.
• That is, it has its own directory in tomcat.
• Moreover, it runs as a separate context, with its own
classloader, session management, etc.

Portlet applications are deployed as distinct war
files/web applications.
• You go through the container webapp to get to the portlet
webapp.
• Portlets in the same application share jars, classes, and
runtime stuff like request and session variables.
• Portlets in different portlet apps do not share anything.

JSR 168 containers vary in the specifics of their
deployment procedures.
Writing Grid Portlets
Extended Example
Grid Portlet Overview


Previously we looked at writing JSR 168
portlets using JSP and Velocity.
Grid Portlets are just portlets
• Use the Java CoG to make calls from the portlet
code to the desired Grid service.
• Can also make non-COG calls, such as to
GridPort API.

These APIs should not be portlet-specific,
since you should be able to reuse the Grid
APIs from other Java clients.
• NCSA’s OGRE (general), Trebuchet (application
specific) are examples
• NCSA Tupelo also makes use of COG in its
metadata management services.
COG and Portlets


We will use the COG to illustrate some basic portlet
programming features.
We use the COG4 APIs in our ProxyManager, Job
Submission, and GridFTP portlets
• Hides differences between Globus Toolkit versions.
• Is also structured to handle task-based workflow and job
composition.

But for in-depth COG questions, we defer to
Gregor’s group.
• Available through AG session today.
• Or docs available through www.cogkit.org

OGCE is an open consortium, and there are other
alternatives.
• OGRE job management builds on COG for Workflow
• GridPort uses the GGF Community Scheduling Framework
and higher level tools (job sequencers, for instance).
• Use what you like
Getting Proxy Credentials


The key ingredient is getting proxy credentials into
the portal and sharing between different portlets.
We use MyProxy for this
• Advantage: you don’t have to store users’ long term keys
on the web server.
• Disadvantage: you have to run a separate MyProxy server;
users have to put keys there outside the portal.


You must include the OGCE2 MyProxy portlet (or a
variation of it) in your Grid portal to do any other
Grid stuff.
OGCE1 also had support for grid-proxy-init.
• Would be able to add this, but you would need to manage
your users’ long term keys.


Proxy Manager Under the
Hood
See
ogce2/indiana/extreme/ProxyManager/mod
ules/portlet/src for templates and code.
Templates illustrate several Velocity
scripting concepts.
• Control structures, variables

But the action code is more interesting.
• This is where you do your Grid coding.
• Use Velocity’s Context object to communicate
strings back to your templates.
Proxy Manager Code





Key part here is the
doGet_proxy()
method.
Abbreviated code on
the right.
MyProxy class:
import from CoG kit.
GSSCredential class:
standard Java
security class.
ProxyManager class:
used by us to pass
credentials between
portlets.
public void
doGet_proxy() {
MyProxy myproxy=
new MyProxy(…);
GSSCredential
proxy=
myproxy.get(…);
ProxyManager.addPr
oxy(…);
}
Writing Simple Job Submission
Portlets

For portlets, you have to deal with two main Grid
issues:
• Getting the credential from the proxy store.
• And then using it.

The CoG APIs provide an additional nice layer of
Grid abstraction for Grid tasks.
• We will look at this for wrapping GRAM calls.

We will assume the following:
• You have your project set up to be warred with maven.
• You have chosen a development technology (like JSP +
portlets or Velocity portlets).
• You will be collecting the necessary input information from
an HTML <form>.
• You will be putting the action code in either the portlet
(perhaps as a JavaBean) or Velocity action handler.
• All code snippets below go in this portlet/action code.
Write the Method Skeleton

In your action code,
write a method to do
job submission.
• For Velocity, this maps
to the “eventSubmit_”
form parameter in your
template.
• In portlets, this code is
called by your
processAction method.

You will need to pass
it, minimally, the
ActionRequest and
ActionResponse
objects.
public void
doSubmitJob(
ActionRequest req,
ActionResponse
resp,…) {
//Next, get proxy
//Then, run command.
}
Get the Proxy



Now get the proxy from the
local store.
The ProxyManager class
calls the static method
getDefaultProxy to look up
your credential using your
portlet session.
You need these imports
• org.ietf.jgss.GSSCredential
• xportlets.proxymanager.Pro
xyManager.
• javax.portlet.* for
everything else.

This assumes that you have
retrieved a credential from
the MyProxy server.
public doSubmitJob(
ActionRequest req,
ActionResponse res) {
//Get the proxy
PortletSession sess=
req.getPortletSession()
GSSCredential cred=
ProxyManager.getDefaultPr
oxy(sess);
//Next, run command
}
Run the Command


The CoG4 interfaces now come into play. You should
first understand the general CoG approach.
The COG4 interfaces attempt to abstract things at
several layers
• Tasks: a generic Grid function



Remote command execution, file operation, or information
inquiry.
Has a specification and service
Is run by a handler
• Specification: set the type of task (job submit, etc.) and its
specific parameters.
• Service: sets the Globus Toolkit (or other) provider, security
context, and service contact information.
• TaskHandler: runs the task.

Note this scheme is intended to shield you from
programming differences of different Globus
toolkits.
Making Contact with the Portal


The remaining issue is getting information from
the HTML forms to the appropriate CoG calls.
This is done through the PortletRequest object.
• Typically, it is actually the ActionRequest subclass.

In your doSubmitJob() command, just use
• String param=request.getParameter(“Param-Name”);
• “Param-Name” is replaced with the input parameter name
in your HTML form.


<input type=text name=“Param-Name” value=“rm -r *.*”>
In following code snippets, “quoted strings” are
usually what you would pass in from the Web form.
An Extended Code Example: Setting
Up Task and Specification
Task task=new TaskImpl(“mytask”,
Task.JOB_SUBMISSION);
task.setProvider(“GT2”);
JobSpecification spec=
new JobSpecificationImpl();
spec.setExecutable(“rm”);
spec.setBatchJob(true);
spec.setArguments(“-r”);
…
task.setSpecification(spec);
Extended Code Example: Setting Up
the Service and Security Context
Service service=new
ServiceImpl(Service.JOB_SUBMISSION);
service.setProvider(“GT2”);
SecurityContext securityContext=
CoreFactory.newSecurityContext(“GT2”);
//Use cred object from ProxyManager
securityContext.setCredentials(cred);
service.setSecurityContext(
(SecurityContext)securityContext);
Extended Example: Set Up
Service Contact and Finish
ServiceContact serviceContact=
new ServiceContact(“myhost.myorg.org”);
service.setServiceContact(serviceContact);
task.setService(
Service.JOB_SUBMISSION_SERVICE,
service);
TaskHandler handler=new
GenericTaskHandler();
handler.submit(task);
Monitoring Tasks

Monitoring running tasks submitted by the COG is done
through a status listener.
• Write a class that implements StatusListener interface
• Create an instance of your listener and add it to the task:


task.addStatusListener(myListener);
There is a small problem with doing this in a portal application
• The listener must be instantiated in the action code, along with
the rest of the COG classes.
• But these are lost after the action completes.

Simple solution: store it in the PortletSession
• This will keep the listener object alive after the action completes
• Get the session from the PortletRequest (more typically, its
subclass, ActionRequest)
• Sample code


PortletSession session=request.getPortletSession();
session.setAttribute(“mytask”,myListener);
• When you are curious about your task’s status, fetch it out of the
session by name.