Menu

Web Services Integration Patterns, Part 2

June 30, 2004

Massimiliano Bigatti

In the first installment of this article, I introduced some design patterns applied to the problem area of web services integration. This article presents more patterns:

  1. Configuration Driven Service
  2. Data Logger
  3. Flow Logger
  4. State Logger
  5. Input/Output Validator

Configuration Driven Service

Input to the service isn't hardcoded, but driven by external configuration data

Sometimes you can't hardcode the parameters used to call a service because they can quickly change based on business rules. The input could then be dynamically constructed, using a configuration table. These parameters could be stored in a database schema, an XML file, a properties file, or in other storage.

With the simple services you encounter in the tutorials, demos, or web services technologies documentation, the interface appears immediately clear, is usually made up of few methods, and embeds a fair simple logic. However, in the real world web services may be very complex.

For example, we had to integrate a service for customer registry census, which required many input parameters, as it needs all the personal information about the customer, like name and surname, addresses, social security number, and other data collected by the application. We also integrated external data banks, which provide financial information about the customers; again, we had to pass a few hundred parameters.

Consider, too, that most web services are stateless: if they relate to some specific data, each time they require even more parameters to identify the primary key of the data being processed.

In such cases we found it useful to externalize the name of the parameters to pass and the logic used to obtain the values of that parameters.

Take for example an e-commerce application that should store the information about the order; maybe the customer is new, so the user had to introduce some personal information. In the case that the store operation is done by a web service, the configuration for that service could be present in a database table like this:

SOAP parameter EL like expression
Name session.currentOrder.requestingCustomer.name
Surname session.currentOrder.requestingCustomer.surname
Street session.currentOrder.requestingCustomer.address.street
City session.currentOrder.requestingCustomer.address.city
Country session.currentOrder.requestingCustomer.address.country
ssn session.currentOrder.requestingCustomer.ssn

The first column is the name of the SOAP element (the Service Parameter); the second one contains an EL-like expression that navigates the domain tree until it finds the getter that returns the required value.

The service proxy relies then on the Configuration Manager to load this table and to evaluate each EL expression against the domain entities to obtain the actual values of such data.

The Dynamic Service Proxy represents the service, but its interface doesn't contain all the setters that would be required to replicate the service interface as in the Service Proxy pattern. Instead, the Dynamic Service Proxy has a generic interface. The Configuration Manager manages the configuration data, maybe delegating data access to a DAO. The Configuration stores all the information needed to perform the parameter-value mapping.

When the client requires a call() to the Dynamic Service Proxy, it calls the Service Manager, requesting the loading of the configuration for such service; this data, often cached in memory, is used by the Manager to create the parameter and values lists and passes them to the proxy.

Looking at the Dynamic Service Proxy, you don't see clearly the service interface, so it gives no immediate feedback of the business rules that drive that particular service. The client code is decoupled from the service proxy, allowing the two to evolve in an independent way.

Sample code

CustomerStoreServiceAbstractDynamicServicecall()setParameter()

public class CustomerStoreService extends AbstractDynamicService {
  
  public void call() {
    ConfigurationManager manager = 
      new ConfigurationManager( "CustomerStoreService" );
      
    //add parameters and values
    manager.prepare( this, session );
    
    //calls the channel
    super.call();
  }
}

public class ConfigurationManager {

  String serviceName;
  Map cache;
  
  public ConfigurationManager( String serviceName ) {
    this.serviceName = serviceName;
    
    initDAO();
  }
  
  public void prepare( DynamicService service, ApplicationSession session ) {
    //loads the service configuration
    ConfigurationData conf = loadConfiguration();
    
    //obtain the values of the parameters, based on data session
    Map values = loadValues( session );
    
    //values the service parameters
    prepareService( service, conf, values );
  }
  
  //load cache-enabled configuration data
  ConfigurationData loadConfiguration() {
    ...
  }
  

  //obtains data from the domain entities
  Map loadValues( ApplicationSession session ) {
    ...
  }
  
  //sets the parameter calling the setParameter() method on the service
  prepareService( DynamicService, ConfigurationData conf, Map values ) {
    ...
  }
}

Data Logger

Input and output from the service is logged in a database table

Logging is one of the most used techniques to identify bugs in the software, but when it comes to the seb services, a simple text file isn't sufficient.

Usually the developer starts logging some key information about the process, to identify specific problems. Quickly, it became necessary to have the request and response XML message in the logs. But one problem is that often the services have many input and output, resulting in long log files.

Text files are also difficult to read and, although logging packages like log4j have nice formatting options, they don't directly support XML.

When debugging a web service it's useful to have the request and the response for a single call in front of each other. The multithreaded and parallel architecture of a J2EE container doesn't guarantee that the information is mantained in a contiguous way, so you'll end struggling in finding the correct response for a specific call.

The situation is worse in production, where many users use the system, performing many calls to the services and generating huge logging data. This amount is so huge that sometimes it overhelms the available storage and CPU in the production machines.

For these reasons, it's useful to log the data flow in a database table, so the input and output of the same call are always coupled. The table could use a pair of CLOB columns, one for the request and the other for the response.

The Logger incorporates all the logic required to store the XML documents in the database table. The Abstract Service implements a common base for services that are aware of the logging capabilities of the framework.

The client invokes the service as usual, but the code contained in the Abstract Service class passes the request (i.e. the SOAP request Envelope) to the Logger, which in turns stores it to the database, obtaining a lock on the row just created. When the response arrives, it is stored in the same row of the request.

Sample code

After the Channel.call(), the response XML is fetched from the channel object and passed over the logger, with the token previously obtained; the logger then store the response on the database.


public abstract class AbstractService implements Service {
  Channel channel;
  Logger logger;
  
  public void call() throws ServiceException {
    channel.init();
    
    String request = channel.getRequest();
    int token = logger.storeRequest( request );
    
    channel.call();
    
    String response = channel.getResponse();
    logger.storeResponse( token, response );
  }
}

Flow Logger

Input and output from the services involved in the process is logged in a database table

The flow logger works at the coordination level, storing the status of a process that uses several services.

When it comes to coordination of network resources, even the simpler process could raise execution failures. The following simple process is modeled after a real-world services coordination case (in original, of about a dozen services call):

State Operation/Service
1 CREATE
2 INIT
3 ADD A
4 ADD B
5 ADD C
6 MODIFY B
7 MODIFY C
8 CLOSE

The CREATE service create a new account, which is therefore INITialized; the program then ADD some elements, editing some others, and close the process with the CLOSE method. The correlation between calls is obtained by the application, passing a token from one call to another.

When an exception is thrown during a service call, the coordination could call a ROLLBACK service which clean the intermediate state of the account being created; the flow logger stores the calling sequence and the point where the problem occourred.

Note that transactional behavior is not comprised in base technologies used for Web Services like SOAP, WSDL, UDDI, XML-RPC or REST (but standards are emerging, like WS-Transaction), so commit/rollback features has to be developed at the application level.

The Flow Logger manages the access to the table used to track the operation flow, maybe using a DAO. The Service Coordinator organize the services in a process. Every time a service returns correctly, with a complete response or a Fault, the Flow Logger writes a record in the log table.

Sample code

Below you can see an example FlowLogger interface. It specifies a track() method that is used to store the current state in the database. The implementation will create a row for each call to this method. The integer key distinguishes different calls to the same process, when executed concurrently by different users on different data.


public interface FlowLogger {
  public int track( int key, String currentState );
}

The AccountCreateCoordinator class models the process shown in the table afterwards. Each single call is wrapped inside a method, that logs the specific call. Each method could raise a ServiceException, breaking the process in the middle and returing the control to the caller.


public class AccountCreateCoordinator {
  FlowLogger flowLogger;
  int accountId;
   ...
  public void createAccount( AccountData data ) throws ServiceException {
    this.data = data;
    
    create();
    init();
    addA();
    ...
    close();
  }
  
  public void create() throws ServiceException {
        CreateService createService = new CreateService();
        
        createService.setKey( data.key );
        createService.setType( data.type );
        createService.setName( data.name );
        createService.setSurname( data.surname );
        
        create.call();
        
        accountId = createService.getAccountId();
        flowLogger.track( data.key, "CREATE" );
  }
    ...
}

State Logger

It stores the state of a transaction to restart in a sequent call from that point.

As an evolution of the Flow Logger, the State Logger not only tracks the operation flow, but uses that information to perform only the calls not correctly done in previous invocations.

Think, for example, of a process that involves numerous service calls, like the one outlined in the previous pattern, but that is safe to remain in an inconsisent state between several calls.

The typical scenario is that the user has a function that calls a coordinator, whose logic comprises several services. In the middle of the process, a service raises an exception, maybe due to network errors. The error message is returned to the user, who can eventually take some action, like checking the configuration properties or call the IT support.

Subsequently, the user retry calling that function, that will skip the calls already done, resuming the work from the point where interrupted before.

The State Logger manages the access to the table used to track the operation flow. The Service Coordinator organizes the services in a process, eventually calling only the services not successfully terminated before.

Every time a service returns correctly, with a complete response or a fault, the State Logger write a record in the log table. The Service Coordinator that implement checks before any call that that call wasn't performed before, using the data contained in the logs.

Sample code

The StateLogger interface specifies a track() method that is used to store the current state in the database. The implementation will create a row for each call to this method. The wasDone() method identifies if a specific state was previously done, looking it up in the database. For each method there is an integer key that distinguishes different calls to the same process, maybe executed by different users on different data:


public interface FlowLogger {
  public int track( int key, State currentState );
  public boolean wasDone( int key, State whatState );
}

The AccountCreateCoordinator class models the process shown in the table afterward. Each single call is wrapped inside a method, that checks using the FlowLogger object that the specific call wasn't done before, for this session. Each method could raise a ServiceException, breaking the process in the middle and returning the control to the caller.


public class AccountCreateCoordinator {
  State CREATE = new State("CREATE");
  State INIT = new State("INIT");
  State ADD_A = new State("ADD_A");
  State ADD_B = new State("ADD_B");
  State ADD_C = new State("ADD_C");
  State MODIFY_B = new State("MODIFY_B");
  State MODIFY_C = new State("MODIFY_C");
  State CLOSE = new State("CLOSE");
  
  ...
  
  FlowLogger flowLogger;
  int accountId;
  
  public void createAccount( AccountData data ) throws ServiceException {
    this.data = data;
    
    create();
    init();
    addA();
    ...
    close();
  }
  
  public void create() throws ServiceException {
    if( !flowLogger.wasDone( data.key, CREATE ) ) {
      CreateService createService = new CreateService();
      
      createService.setKey( data.key );
      createService.setType( data.type );
      createService.setName( data.name );
      createService.setSurname( data.surname );
      
      create.call();
      
      accountId = createService.getAccountId();
      flowLogger.track( data.key, CREATE );
    }
  }
  
  ...
}

Input/output Validator

It validates input and output from the service to reduce garbage data

One of the key elements of web services and SOA Architectures is the loose coupling between the services versus the application and between different services. This can be true at the wire level, but the coupling has to be strengthened at the application level, to assure the correct operation of the logic implemented in the program.

For example, consider a simple application that would exchange the value of a sum of money expressed in a specific currency to another currency, using the web service shown before. Imagine that the country codes are manually introduced by the user using a web page and the output rate is used to calculate the converted value.

To work properly, this simple application has to perform a series of checks: first, it should verify that the country codes provided by the user are supported by the service; secondly, it should check that the resulting rate hasn't invalid values, like zero, or NaN.

Embedding those checks in the Service Proxy allows to maintain proximity between the service and related controls. When a problem is found, the Service Proxy should throw an exception, providing the caller with field that caused the error failing the check (data missing, bad length, wrong format, and so on).

The check implemented in the Service Proxy usually has an incrementally approach: each check raise a single exception, describing that single problem; an alternative is to perform all the checks and then throwing a single exception that describes all the problems encountered.

The Abstract Services defines the abstract methods that will check the input and output of the service. The Service Proxy implements such methods, providing specialized checks to input and output data. The Service Exception represents the problem encountered during verification.

Sample code

The following class represents the usual ExchangeService, but comprises the checkParameters() and checkResponse methods. The first one ensures that the country provided by the caller is valid, checking against null and against the list of possible countries. The second method verifies that the result value is a valid number, not infinite and more than zero.


public class ExchangeService extends AbstractService {
    String country1;
    String country2;
    float rate;
    
    //valid country list
    List countryList;
    ...
    public void checkParameters() throws ServiceException {
        if( country1 == null || countryList.get( country1 ) == null ) {
            throw new ServiceException("Country1 " + country 1 + " not valid");
        }
        if( country2 == null || countryList.get( country2 ) == null ) {
            throw new ServiceException("Country2 " + country 2 + " not valid");
        }
    }
    
    public void checkResponse() throws ServiceException {
        if( rate == null || rate == Float.NaN || 
                rate == Float.NEGATIVE_INFINITY  ||
                rate == Float.POSITIVE_INFINITY ||
                rate < 0 ) {
            throw new ServiceException("The service returned an invalid value");
        }
    }
        
    public void setCountry1( String country ) {
        country1 = country;
    }
    
    public void setCountry2( String country ) {
        country2 = country;
    }
    
    public float getRate() {
        return rate;
    }
}