HttpPost requests executed multiple times (Apache HttpClient)

This is something I noticed on Android, but from what I read it also involves the desktop Java version.

I was sending POST requests to an API server, and I was getting some random 400 Bad Request responses from time to time. I wish Apache provided an easy way to log the plain text version of Http requests, but I couldn’t find a better way to see what the app was sending than sending the same request to my PC when failing.

So to log requests I start netcat (sudo nc -l 80 on a mac) or a very minimal server in python (it’s more or less the same as the example on Twisted’s front page) and route them there whenever an error occurs.

try {
   response = client.execute(post,
                  new BasicResponseHandler());
} catch (IOException e) {
   if (DEBUG_FAILED_REQUESTS) {
      post.setURI(URI.create(DEBUG_FAILED_REQUESTS_SERVER));
      try {
         client.execute(post, new BasicResponseHandler());
      } catch (IOException e1) {
         e1.printStackTrace();
      }
   }
}

I don’t know if it’s my router, but sometimes connections from the Android device to my PC get blocked: to make them work I just open a browser on the Android, go to some website and then try again with my internal IP (192.168.0.whatever). It always works, no idea why.

Using this code I discovered that my post requests were executed 4 times each, nearly at the same time. I discovered that it’s the default behavior, and you must provide your own RetryHandler if you want the HttpClient to work otherwise.

In my case, my calls are sent to Google’s shortener service, and for some reason sometimes it just rejects requests. If you wait a little bit between attempts you increase your chance of getting valid responses. So this is what I did:

HttpPost post = new HttpPost(SHORTENER_URL);
String shortURL = null;
int tries = 0;
try {
    post.setEntity(new StringEntity(String.format(
            "{\"longUrl\": \"%s\"}",
            getURL(encodedID, encodedAssignedID))));
    post.setHeader("Content-Type", "application/json");
    DefaultHttpClient client = new DefaultHttpClient();
    // disable default behavior of retrying 4 times in a burst
    client.setHttpRequestRetryHandler(new DefaultHttpRequestRetryHandler(
            0, false));
    String response = null;
    while (response == null && tries < RETRY_COUNT) {
        try {
            response = client.execute(post,
                    new BasicResponseHandler());
        } catch (IOException e) {
            // maybe just try again...
            tries++;
            Utils.debug("attempt %d failed... waiting", tries);
            try {
                // life is too short for exponential backoff
                Thread.sleep(RETRY_SLEEP_TIME * tries);
            } catch (InterruptedException e1) {
                e1.printStackTrace();
            }
        }
    }
    Utils.debug("response is %s", response);
    if (response != null) {
        JSONObject jsonResponse = new JSONObject(response);
        shortURL = jsonResponse.getString("id");
    } else if (DEBUG_FAILED_REQUESTS) {
        Utils.debug("attempt %d failed, giving up", RETRY_COUNT);
        debugPost(post, client);
    }
} catch (JSONException e) {
    e.printStackTrace();
} catch (UnsupportedEncodingException e) {
    e.printStackTrace();
}

where debugPost() is a method that calls my PC to log the request, and Utils.debug() is just a small utility method I wrote to log messages with logcat using String.format() if format args are passed to it (it also takes care of splitting messages that would be truncated by logcat itself).

You could choose to implement exponential backoff very easily, but since it’s a blocking operation for the user in my case I preferred not to.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s