Pragmatic API design

In a comment on a previous post, Carfield Yim suggests an alternative style for API requests. We could dub this the URL-encoded style, since the content type associated with the request format is application/x-www-form-urlencoded. (Carfield's example involves updating a user's name, so the hypothetical service ought to require that the request be submitted using HTTP POST method rather than GET, since the action is unsafe.)

There is eternal debate in the software and web communities on the question of what APIs should look like, involving an acronym soup of terms such as SOAP, REST, SOA, POX and ROA. Without invoking too many buzzwords, we can observe that the following are both desirable properties of real-world APIs.

1. Support for invoking an operation, and viewing the response, directly in a browser. Typically the requests would need to be submitted using an HTML form. This can be very useful to a developer trying to make use of the API, since it becomes incredibly easy to test its behavior. Learning by experiment is often easier and more effective than reading a specification.
2. Reuse of existing data formats. For example, one of the examples in the original post was concerned with creating a user account in EKP. The body of the request message contained CSV data in the format EKP expects for files uploaded using the User Data Loader. Hence, if an organization has already written code to generate data in that format (extracted from their HR system for example), it needs relatively little work to switch between either generating files for manual upload, or submitting the data directly using the API. On the other hand, the application/x-www-form-urlencoded content type isn't normally used outside of HTTP requests.

(In passing, it's worth noting that SOAP does not satisfy either of these requirements particularly well. It fails completely with the first. As for the second, it's reasonably easy to reuse an existing format if it is XML-based and literal encoding is used rather than SOAP encoding. On the other hand, using a format that is not XML-based—because it predates XML for example—typically requires sending the data as some kind of SOAP attachment, which raises a whole other set of issues.)

In some cases, support for batch operations might also be a requirement. For example, if I need to update 1,000 user records, this will be considerably more efficient if the data can be sent in a single request. If 1,000 separate requests are required, then network latency will become a serious concern. (However, HTTP pipelining could help if supported.)

As I see it, the URL-encoded request style meets the first requirement (it supports invocation from a browser), but fails to meet the second (it fails to reuse an existing data format for the request) and third (it is not well suited to batch operations). On the other hand, the CSV request style described in the original post fails to meet the first requirement, but meets the second and third. This seems like a reasonable trade-off in this case. However, the URL-encoded style seems like a reasonable approach in cases where support for large batch updates isn't a critical requirement, and there isn't an existing data format that obviously matches the desired content of the request message.

Code samples: invoking APIs using Java

Following on from my previous post, I thought it might be useful to post some sample code that demonstrates how to invoke EKP's APIs, in this case using Java.

Every API request must include an HTTP basic authentication header. If you are using Java's built-in HttpURLConnection class, you would need to construct and add this header manually. This is not hard, and the EKP APIs and Web Services Overview explains how to do it. However, it's even easier to use a library that supports HTTP basic authentication directly.

One such library is Jakarta Commons HttpClient. HttpClient is free, open source, and distributed under version 2.0 of the Apache License, meaning that it can be used with minimal obligations in software that is not itself open source. (Note that HttpClient has dependencies on both Commons Logging and Commons Codec, so you will also need both of those to run this sample code.)

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.StringRequestEntity;

We begin by creating an instance of HttpClient, which is straightforward.

HttpClient httpClient = new HttpClient();

We need to initialize the HttpClient object with the appropriate user name and password. The user name is ignored by EKP, so we simply use the empty string. The password must match the value of the authentication.key property configured in the file WEB-INF/conf/ekp.properties.

String userName = "";
String key = "mysecretkey";
httpClient.getParams().setAuthenticationPreemptive(true);
httpClient.getState().setCredentials(AuthScope.ANY,
        new UsernamePasswordCredentials(userName, key));

As described in the previous post, we can create a user account (here, joestudent) by sending an HTTP POST request to /ekp/contentHandler/usersCsv, and including CSV-formatted data in the body of the request.

PostMethod postMethod1 = new PostMethod(
        "http://ekp.example.com/ekp/contentHandler/usersCsv");
String content1 = "Action,UserID,Password,FamilyName,GivenName\r\n"
        + "A,joestudent,dummypass,Student,Joe\r\n";
postMethod1.setRequestEntity(new StringRequestEntity(content1,
        "text/plain", "UTF-8"));
httpClient.executeMethod(postMethod1);

Similarly, we can enroll joestudent in a course with ID Derivatives_101 by sending an HTTP POST request to /ekp/enrollmentHandler, again with appropriately-formatted CSV data in the body of the request.

PostMethod postMethod2 = new PostMethod(
        "http://ekp.example.com/ekp/enrollmentHandler");
String content2 = "USERID,LEARNINGID\r\n"
        + "joestudent,Derivatives_101\r\n";
postMethod2.setRequestEntity(new StringRequestEntity(content2,
        "text/plain", "UTF-8"));
httpClient.executeMethod(postMethod2);

API 101

We just finished a fairly intensive training session here in Hong Kong with a couple of partners. The subject was ostensibly our customization toolkit, but we also spent quite a bit of time discussing APIs and how EKP can be integrated with existing web sites, portals and applications. Every time we conduct these training sessions, this topic expands.

Here's an outline of what we covered, with links to the relevant documentation.

1. The first step in seamlessly combining EKP functionality with an existing site is understanding how to implement web-based single sign-on.
2. With a single sign-on mechanism in place, it's possible to have external pages link directly to specific pages in EKP, and, in particular, to launch courses directly from external pages. (See also: How can I link directly to a page in EKP without losing the navigation frames?)
3. Data can be manipulated in, or extracted from, EKP using the APIs described in the EKP APIs and Web Services Overview. Examples:
   • To ensure that EKP has a user account for a specific user ID (as part of the single sign-on process for example), have the external site send a direct server-to-server HTTP POST request to /ekp/contentHandler/usersCsv, with CSV-formatted data (following the User Data Loader format) in the body of the request. Note that most of the columns are optional when the User Data Loader format is used, so the following is valid for example:

     Action,UserID,Password,FamilyName,GivenName
     A,joestudent,mypassword,Student,Joe

   • To ensure that a specific user is enrolled in a specific course (prior to launching the course for example), have the external site send a direct server-to-server HTTP POST request to /ekp/enrollmentHandler with CSV-formatted data in the body of the request. For example:

     USERID,LEARNINGID
     joestudent,Derivatives_101

   • To retrieve a complete training history, in XML format, for user joestudent (to display within an external page for example), have the external site send a direct server-to-server HTTP GET request to /ekp/contentGenerator/trainingHistoryXml?userId=joestudent.

Update: I posted Java code samples for creating a user account and enrolling a user in a course.

Update 2: Musings on the invocation style of the API operations discussed above, prompted by Carfield's comment.

Directory services and hosted EKP sites

Consider the following scenario, which is not uncommon. An organization wishes to make use of an EKP site that is hosted outside of their network. The organization has already implemented a directory service (Active Directory for example) that is most likely accessible via LDAP. Naturally, the organization would like their users to be able to access EKP using the usernames and passwords stored in the directory service. Below are three ways in which this could be accomplished.

1. Usernames and passwords could be exported from the directory service (using a tool such as CSVDE for Active Directory for example) and imported into EKP. The import could be performed manually using EKP's CSV User Data Loader, or it could be automated using EKP's APIs or import scheduler. Users would log into EKP using EKP's standard login page.
2. The EKP site could be configured to connect directly to the organization's directory service using LDAP to perform authentication. Again, users would log into EKP using EKP's standard login page.
3. A login page could be created on a web server in the organization's DMZ that connects to the organization's directory service and implements an authentication service for EKP as described in this previous post.

The first approach has a few disadvantages.

• It is not real-time. For example, a password change in the directory service will not take effect on the EKP site until the next time the usernames and passwords are synchronized.
• For extra security, systems that support password-based authentication might store a cryptographic hash of a password, from which it is infeasible to recover the original password, instead of storing the original password. (In fact EKP does exactly this, which is why it is not possible to export passwords from EKP.) Thus, it might not be possible to export passwords from the directory service.
• Even if it is technically possible to export passwords from the directory service, replicating passwords on a system outside of the organization's network might violate the organization's security policies.

The second approach avoids these problems. However, it does require that the organization's directory service be accessible from outside the organization's network, which often will not be the case. (This would of course be the preferred approach if the EKP site was hosted inside of the organization's network.)

The third approach has several advantages.

• It is real-time. For example, when a password is changed in the directory service, the new password will immediately be effective for accessing EKP.
• It does not require passwords to be exported from the directory service or replicated outside of the organization's network.
• It does not require that the directory service be accessible outside of the organization's network via LDAP.
• In cases where the organization already has a web site or portal that authenticates against the directory service, the authentication service for EKP can be combined with the existing site, thus providing single sign-on between the existing site and EKP.

This third approach should be the preferred approach in most cases where the EKP site is hosted outside of the organization's network.

Can I launch courses directly from a portal so that learners don't have to log into EKP, but still have EKP track the learners' progress?

In order to link to a course so that the course launches immediately, and is tracked by EKP, you can link to the launchByCourseId transaction, e.g.:

/ekp/servlet/ekp/launchByCourseId?courseId=My_Course

(Note that this requires that the learner already be enrolled in My_Course, so you might want to pre-enroll learners using a Group Enroll or Auto-Enroll.)

If you link directly to the URL above, and the learner does not yet have a session with EKP, she will be prompted to log in as normal. In order to achieve "transparent login", you can use one of the two mechanisms described below.

1. Implement the "external authentication service" mechanism. To implement this, the portal would need to be modified to generate "authentication tokens" (cryptographic hashes) when requested by EKP. Then, EKP's configuration is modified so that, when it needs to authenticate a user, it redirects to the portal to request an authentication token instead of showing the standard login page.
   
   Note that this mechanism has the significant advantage that it is not required that users' passwords be synchronized in the two systems. In fact, when using this mechanism, the passwords stored in EKP's database are never used.
2. Use the standard verify transaction. This is the transaction that is used to validate a user ID and password submitted using the normal login page. It is possible to add a target parameter to specify the page to which the user should be redirected after a successful login (instead of the normal home page). In particular, if you want to start a course after login, you can specify the launchByCourseId transaction (with appropriate courseId parameter value) as the value of the target parameter. E.g.:

   /ekp/servlet/ekp?TX=VERIFY&UID=joestudent&PWD=password&target=%2Fekp%2Fservlet%2Fekp%2FlaunchByCourseId%3FcourseId%3DMy_Course

   Note that the value of the target parameter has been percent-encoded.
   
   The above example includes the parameters in the query string—however, for security reasons you should avoid including the user ID and password in the URL—therefore, the portal should be coded to use the POST method rather than GET to invoke the verify transaction.

Note that, unlike method 1, method 2 requires passwords to be synchronized between the two systems, which could be difficult to achieve if passwords are changed regularly (a good security practice). (This problem can be avoided by configuring both systems to use the same directory server, i.e. Active Directory or LDAP.) In addition, method 2 requires embedding the user's password in an HTML page (or HTTP redirect header), which might be cached. For these reasons, we recommend using method 1 if possible.

Can I use an external authentication service so that users don't need to log into EKP directly?

(Note: this post contains technical details that will likely be of most interest to developers.)

In a typical organization, EKP will be one of many applications that employees or other stakeholders use on a regular basis. In this situation, it's typically desirable to provide some kind of single (or reduced) sign on, so that users don't have to perform separate logins to different applications.

EKP provides a number of mechanisms to help solve this problem. Some of those most commonly-used are listed below.

• Support for LDAP and Active Directory services: Use of an LDAP or Active Directory service avoids the needs to store and update users' passwords in EKP's database, because users' credentials are checked against the centralized directory service when they log in. This works even if the user is accessing EKP over the internet. However, it does not actually eliminate the need for the user to perform a login to access EKP. (For details of LDAP- and Active Directory-based authentication, see the LDAP Authentication Integration Configuration Guide and the Active Directory Authentication Integration Configuration Guide respectively.)
• Support for Integrated Windows Authentication: Use of Integrated Windows Authentication avoids not only the need to store and update users' passwords in EKP's database, but also the need for users to explicitly log into EKP, because their identity is automatically propagated to EKP based on their Windows login. However, Integrated Windows Authentication typically only works within a Windows-based intranet; it typically will not work for access over the internet. (For details on using Integrated Windows Authentication with EKP, see the Single Sign-On Integration with Windows document.)

However, it's often desirable to provide an integrated authentication mechanism that will work over the internet, even if the authentication service is not physically co-located with EKP. This post describes such a mechanism.

For the sake of example, let's suppose that you are running a company portal alongside EKP, and that you want your portal users to be able to access EKP. However, you do not want users to have to log into EKP separately; rather, you want all authentication to occur via the portal, with users' identities being propagated to EKP once they have authenticated through the portal.

In order to enable the integrated authentication mechanism, it's necessary to define a couple of properties in the ekp.properties configuration file (under the WEB-INF/conf/ directory), as shown below.

authentication.key=mysecretkey
authentication.service.url=http://portal.abc.com/login.asp

The significance of these properties and their values is explained below. In addition, another optional property may be specified, as shown below.

authentication.digestAlgorithm=SHA

The value of this property specifies the cryptographic hash function that will be used as part of the authentication process, as explained below. Permitted values are MD5 and SHA. If this property is not specified, a default value of MD5 is assumed.

Normally, when an unauthenticated user tries to access a page in EKP that requires authentication, she will be redirected to EKP's standard login page. With the integrated authentication mechanism enabled, she will instead be redirected to the URL specified by the value of the authentication.service.url property. (Note that it does not matter how the user came to access the EKP page. She might have accessed the page by following a link from the portal, but that's not necessary for the mechanism to work.)

EKP will also append a query string parameter named salt to the URL that is the target of the redirect. The value of this parameter is a random sequence of bytes, converted to characters using Base64 encoding. The purpose of the salt value is to prevent replay attacks—which is to say that if an attacker was able to intercept the URL generated by the portal as described below, they would not be able to reuse it authenticate against EKP after the salt value expired.

An example of a possible redirect target URL is shown below.

http://portal.abc.com/login.asp?salt=OqQ1uao%3D

(N.B. In this example, the raw Base64-encoded salt value is the character string OqQ1uao=. However, the equals character "=" has special meaning when used in URLs, so the value has been URL-encoded as OqQ1uao%3D.)

On receiving the request from the redirected client, the portal needs to perform the following steps.

• Recover the original bytes of the salt by Base64-decoding the URL parameter value.
• Authenticate the user if she does not already have a session with the portal.
• Using the function configured as described previously, calculate a cryptographic hash of the user ID, secret key (this is the value of the authentication.key property described above) and salt value.
• Convert the bytes of the cryptographic hash to a character string using Base64.
• Redirect the client back to EKP's authenticationTokenVerifier transaction, passing both the user ID and the Base64-encoded cryptographic hash as query string parameters.

For example, assuming that the user is authenticated by the portal as joestudent, and the secret key is mysecretkey, then the MD5 cryptographic hash (after Base64 encoding) would be vf1nZ7R2YSoso+g+BLLVog==, and the EKP URL to which the portal would redirect the client would look as shown below.

/ekp/servlet/ekp/authenticationTokenVerifier?userId=joestudent&digest=vf1nZ7R2YSoso%2Bg%2BBLLVog%3D%3D

(Note that the value of the digest parameter is the URL-encoded cryptographic hash.)

On receiving the request from the redirected client, EKP will perform its own computation of the cryptographic hash and compare it with the value passed by the portal. If the values match, this is accepted as proof that the redirect was actually generated by the portal and was not "spoofed". EKP then considers the client to be authenticated and establishes a session with the client.

The following sample code provides an outline of how the mechanism described above might be implemented as a Java servlet.

String saltStr = req.getParameter("salt");
byte[] salt
  = new sun.misc.BASE64Decoder()
      .decodeBuffer(saltStr);
  
String key = "mysecretkey";
String userId = "joestudent";
  
// Calculate a digest
java.security.MessageDigest md
  = java.security.MessageDigest.getInstance("MD5");
  
// Convert user ID and key to bytes--need to
// specify a character encoding here in case the
// default encodings are different on the two
// systems.
md.update(userId.getBytes("UTF-8"));
md.update(key.getBytes("UTF-8"));
md.update(salt);
byte[] digest = md.digest();
  
// Use Base64 encoding to turn the digest into a
// string, so we can pass it in the URL.
String digestStr
  = new sun.misc.BASE64Encoder().encode(digest);
  
// Encode the digest again using URL encoding to
// escape any special characters.
String url
  = "http://ekp.abc.com/"
  + "ekp/servlet/ekp/authenticationTokenVerifier"
  + "?userId="
  + java.net.URLEncoder.encode(userId, "UTF-8")
  + "&digest="
  + java.net.URLEncoder.encode(digestStr, "UTF-8");
  
response.sendRedirect(url);

The mechanism described in this post works in EKP 4.6 Gold build 116 or higher, and in EKP 4.7 Gold build 52 or higher.

How can I link directly to a page in EKP without losing the navigation frames?

Sometimes it's convenient to link directly to pages in EKP from external systems. For example, if you have a course with ID MY_COURSE, you could link to the course catalog page from a page on your company intranet by using a URL like the one below.

/ekp/servlet/ekp?TX=FORMAT1&CID=MY_COURSE

However, if you create a link in this way then the visitor will see only the course catalog page, not the usual EKP navigation frames, so she will not be able to explore EKP, or even log out.

With a little more work, it's possible to create a URL that will let you link to the same page in such a way that it's displayed with the standard navigation frames. You do this by using a URL that follows the pattern shown below.

/ekp/servlet/ekp/pageLayout?mainSrc=URL-of-target-page

You need to replace URL-of-target-page with the actual URL of the target page. However, this URL needs to be URL-encoded. It's beyond the scope of this blog post to fully explain how URL-encoding works, but basically it involves replacing special characters with an encoded form that consists of a percentage symbol ('%') followed by two letters or digits. For example, the slash character ('/') is replaced by the percentage symbol followed by the digit '2' and the letter 'F'. Here is a more detailed explanation of URL-encoding.

To return to our original catalog page example, the URL-encoded form of the target URL is as shown below.

%2Fekp%2Fservlet%2Fekp%3FTX%3DFORMAT1%26CID%3DMY_COURSE

Once this is substituted into the URL that displays the navigation frames, the final full URL is as shown below.

/ekp/servlet/ekp/pageLayout?mainSrc=%2Fekp%2Fservlet%2Fekp%3FTX%3DFORMAT1%26CID%3DMY_COURSE

The same technique can be used for any internal EKP page that can be linked to directly.