Portal Syndication: Embedding One Web Site's Functionality in Another

April 29, 2003

Ivelin Ivanov

What Is Web Syndication?

Web site syndication has gotten more popular as sites reference each other not only by a single hyperlink but also by embedding content. The idea was pioneered by Netscape's Rich Site Summary (RSS) XML format. RSS was developed in early 1999 to populate Netscape's My Netscape portal with external newsfeeds ("channels"). Since then RSS has taken on a life of its own and now thousands of sites use it as a "what's new" mechanism.

RSS is an example of an organically grown and widely accepted standard. It was not endorsed by any of the popular standards committees. Even so it quickly became popular and found a large number of creative uses. Lately, however, it has reached its limits. There is a demand for more advanced portal syndication which RSS cannot satisfy.

The latest generation web portals demand more than posting news stories. Embedding and personalizing rich content and behavior from remote portals is becoming a necessity. Limited success has been achieved through complex and sophisticated backend integration via proprietary or web services compliant protocols. Recognizing the growing demand, influential organizations have attempted to develop new languages like the Web Services Expression Language (WSXL) from IBM, Web Services Inspection Language (WSIL) from Microsoft, and Web Services for Remote Portals (WSRP) from the OASIS Group.

While these efforts are certainly worthwhile and promising, it will most likely take years before they pass the filters of real life trial and failure. All of them require a thick infrastructure layer to support implementations. While possible, it is again unlikely that mainstream deployment will be achieved instantly.

Fortunately there is way to satisfy a large portion of the syndication requirements by applying established technologies and tools. We will illustrate the architecture of a possible solution using Apache Cocoon.

First, let's describe a typical use case scenario: a user logins at a familiar portal and happily surfs about. At some point she clicks on a link which leads to a strange page. It has the portal logo, even shows the same login id but still looks very different and unfriendly. After some time and frustration she gets used to switching back and forth between the two faces of the portal, while looking for another provider which offers both services in a coherent graphical interface.

For those who have never had this experience, we will give a popular example. Yahoo! Autos offers an easy-to-use interactive catalog of cars. However when it comes to insuring an automobile, applying for a loan, or buying, the site links to a co-branded page of another company. For example, Lending Tree will show Yahoo! Autos logo at the top of the screen, however the rest of the page looks very different than any other Yahoo! page. All the personalization niceties that a Yahoo! user enjoys are lost as soon as the application for a loan begins. A pop-up window appears when switching between the two sites, which reads "You are about to view pages over a secure connection...". When added up, these "negligible" inadequacies lead to an overall poor experience, which is certainly not the original intent of the Yahoo! content producers.

Now we have an idea of how things are not supposed to work, we will show that outsourcing interactive components to a third party site, while preserving the look and feel of the original portal, is still possible when done right. As we mentioned, Cocoon offers a solution. (For an introduction to Cocoon, see the Apache Cocoon site, "Getting Started With Cocoon 2", "Introducing Cocoon 2", or "Building XML Portals with Cocoon".)

A new Web Service Proxy component is available in Cocoon 2.1. Combined with the XMLForm component (see my earlier article, XML Forms, Web Services and Apache Cocoon) and XSLT, it allows vendors to share interactive content with little effort. The Web Service Proxy takes advantage of the fact that a Cocoon web application produces XML content, which is later translated into multiple presentation formats. Once the proxy is added to the Cocoon sitemap, it transparently pipes browser requests to a remote web application and returns the response back to the sitemap for local styling. Receiving a client independent XML format allows the local site to pull content and style it with XSLT with the desired look and feel.

How the Web Services Proxy Works

Let's look at how data flows between the user and the two sites involved. First, how is a form submitted to the original site? The XMLForm component is the answer.

XMLForm allows the end user to directly interact with the remote server through the embedding site. The form markup in the XML content of an embedded page uses a relative URL address for the target action. When the end user submits, the form data is sent to the containing site, which captures the form data and the relative URL. The Web Service Proxy then takes this information and resubmits it to the original site. It then reads the XML response and makes it available to the sitemap for styling again.

Typical web applications maintain user session data. The Web Service Proxy solves the problem of propagating session data to the embedded site by hooking into the end user session, automatically starting its own session with the remote site. If the remote site requires authentication, then the developer of the local web site has to pass the user credentials as parameters to the WebServiceProxyGenerator.

Below we illustrate the architecture of the solution with some example code and figures.

Figure 1. Architecture of the Web Service Proxy Solution. As opposed to a traditional proxy server, the Web Services Proxy captures user input and allows the web site to remain coherent even when the functionality for some of its components is delivered remotely.
Figure 2. Illustration of the data flow for a composite page. Some of the content is locally constructed, the rest is obtained remotely. Finally the same styling is applied and the user facing page appears consistent.

Now we will show a (very short) snippet of the sitemap which employs the Web Service Proxy. The Web Service Proxy completely handles the content and navigation logic between the two portals. Only stylesheets are additionally required to translate the remotely retrieved documents into a user friendly format.

<?xml version="1.0"?>

<map:sitemap xmlns:map="">

<!-- =========================== Components ================== -->


  <map:generators default="file">

  <map:generator name="wsproxy"






<!-- =========================== Pipelines =================== -->



     <!-- Interactive Web Application Syndication -->

     <map:match pattern="*">

       <map:generate type="wsproxy"



       <map:transform src="stylesheets/newWizard2html.xsl" />



       <map:serialize type="html"/>





Figure 3. The interaction between the key participants in a syndication session.

Figure 4. Screenshot from the remotely enabled application as it appears standalone.

Figure 5. Sample screenshot from the same application, embedded in another web application.

For more information on the markup used with the XMLForm component to generate the user interfaces shown above, please refer to my previous article.


The Web Service Proxy component is tightly integrated with the Cocoon framework and is particularly convenient to use in combination with XMLForm to enable syndication of web site functionality. We only scratched the surface of the possible applications. Although the solution we offered is conveniently applied with Cocoon, the concepts are generally applicable outside the framework as well.

The author is working on a new book which will cover this and other advanced topics in greater detail. Check your local book store in 2003 for "Extreme XML Publishing with Open Source Tools" from Manning Publications. You can also follow the weblog which posts use cases, best practices, and success stories on advanced Cocoon applications.