Here’s the code:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Text;
using System.Text.RegularExpressions;
using System.Net;

namespace ConsoleApplication1
{
class Program
{
static int daysToDownload = 1;
static readonly String urlBase = “https://itts.apple.com”;
static String fieldNameReportType = “”;
static String fieldNameReportPeriod = “”;
static String fieldNameDayOrWeekSelection = “”;
static String fieldNameSubmitTypeName = “”;
static String fieldNameDayOrWeekDropdown = “”;

static void Main(string[] args)
{

Console.WriteLine(“fetching data”);

// First start up a session and get the sessionpage
String SessionPage = GetSessionPage();

// After this navigate to the vendor page
String VendorPage = GetVendorPage( SessionPage );

// After this get the ID for the dayOrWeekDropdwon
String UrlDownload = GetDayOrWeekUrl(VendorPage);

DownloadFiles( UrlDownload );

Console.WriteLine(“done fetching data, press enter to continue”);
Console.ReadLine();
}

public static String GetSessionPage()
{
WebClient client = new WebClient();
String data = client.DownloadString(urlBase + “/cgi-bin/WebObjects/Piano.woa”);

Match match = Regex.Match(data, “\” action=\”(.*)\”");

WebClient c2 = new WebClient();
NameValueCollection myQueryStringCollection = new NameValueCollection();
myQueryStringCollection.Add(“theAccountName”, “itunesaccountname“);
myQueryStringCollection.Add(“theAccountPW”, “itunespassword“);
myQueryStringCollection.Add(“1.Continue.x”, “0″);
myQueryStringCollection.Add(“1.Continue.y”, “0″);
c2.QueryString = myQueryStringCollection;

String downloadUrl = urlBase + match.Groups[1];
Console.WriteLine(“Logging in on”);
Console.WriteLine( downloadUrl );
return c2.DownloadString( downloadUrl );
}

public static String GetVendorPage( String SessionPage )
{
Match match = Regex.Match(SessionPage, “name=\”frmVendorPage\” action=\”(.*)\”");

String UrlDownload = urlBase + match.Groups[1];

MatchCollection matchCollection = Regex.Matches(SessionPage, “name=\”(.*?)\”");

fieldNameReportType = matchCollection[6].Groups[1].ToString();
fieldNameReportPeriod = matchCollection[7].Groups[1].ToString();
fieldNameDayOrWeekSelection = matchCollection[10].Groups[1].ToString();
fieldNameSubmitTypeName = matchCollection[11].Groups[1].ToString();

WebClient client = new WebClient();
NameValueCollection myQueryStringCollection = new NameValueCollection();
myQueryStringCollection.Add(fieldNameReportType, “Summary”);
myQueryStringCollection.Add(fieldNameReportPeriod, “Daily”);
myQueryStringCollection.Add(fieldNameDayOrWeekSelection, “Daily”);
myQueryStringCollection.Add(fieldNameSubmitTypeName, “ShowDropDown”);

client.QueryString = myQueryStringCollection;
Console.WriteLine(“fetching vendor page on”);
Console.WriteLine(UrlDownload);
return client.DownloadString(UrlDownload);
}

public static String GetDayOrWeekUrl(String VendorPage)
{
// Get the new session Url
Match match = Regex.Match(VendorPage, “name=\”frmVendorPage\” action=\”(.*)\”");
String UrlDownload = urlBase + match.Groups[1];

// Also fill in the DayOrWeek variable
MatchCollection matchCollection = Regex.Matches(VendorPage, “name=\”(.*?)\”");
fieldNameDayOrWeekDropdown = matchCollection[8].Groups[1].ToString();

Console.WriteLine(“fetching dayOrWeek variable: ” + fieldNameDayOrWeekDropdown);

return UrlDownload;
}

public static void DownloadFiles( String UrlDownload )
{
ArrayList reportDates = new ArrayList();
for (int i = 1; i <= daysToDownload; i++)
{
TimeSpan timespan = TimeSpan.FromDays(i);
DateTime reportDate = DateTime.Today.Subtract(timespan);
reportDates.Add(reportDate);
}

foreach (DateTime date in reportDates)
{
WebClient client = new WebClient();
NameValueCollection myQueryStringCollection = new NameValueCollection();
myQueryStringCollection.Add(fieldNameReportType, “Summary”);
myQueryStringCollection.Add(fieldNameReportPeriod, “Daily”);
myQueryStringCollection.Add(fieldNameDayOrWeekDropdown, appleDate(date));
myQueryStringCollection.Add(“download”, “Download”);
myQueryStringCollection.Add(fieldNameDayOrWeekSelection, appleDate(date));
myQueryStringCollection.Add(fieldNameSubmitTypeName, “Download”);
client.QueryString = myQueryStringCollection;

String filename = “file.” + date.ToString(“dd_MM_yyyy”).Replace(“/”, “_”) + “.txt.gz”;
Console.WriteLine(“fetching file: ” + filename);
client.DownloadFile(UrlDownload, “c:/” + filename );
Console.WriteLine(“fetched file: ” + filename + ” – apple filename: ” + client.ResponseHeaders.Get(“content-disposition”) );
}
}

public static String appleDate( DateTime date )
{
return System.Web.HttpUtility.UrlEncode( date.ToString(“MM/dd/yyyy”) );
}
}
}

We used to be able to compile an .ear in about 2 hours (which I think was long already). This project has about roughly 130 JSPs and quite some tag libraries. After a redesign of the site the compilation times started to increase, upto 4 hours, 8 hours, … This was making me nervous as it meant we had to stop our sprints sooner … to test sooner … to start building sooner. This is were my quest for the initial problem began.

Why the hell does ojspc take so long to translate my JSPs??

Roughly .. translating a JSP could take upto 2 minutes, with about 130 JSPs, well that easily takes you 2 hours. I started messing around with ojspc.jar seeing if I could call it natively from code, or if I could override a class and maybe play with the internals of ojspc. Not much success here. After monitoring the ojspc process, I saw it was spawning a javac process every once in a while … probably compiling stuff – right? If only I could have a peak there and see what exactly it was doing … and so I did.

I replaced the standard javac binary with a javac bash script which looked more or less like this:

#!/bin/bash
echo “===============================================================” >> /tmp/compile.log
echo “$$ $PPID : $*” >> /tmp/compile.log

This game me some interesting information on what ojspc was trying to do, however the ojspc process died after trying to translate the first JSP. I assume it wasn’t finding the necessary classes to continue the compilation process.

So playing on with the workaround script – we included the original call the javac binary (now bjavac):

#!/bin/bash
echo “===============================================================” >> /tmp/compile.log
echo “$$ $PPID : $*” >> /tmp/compile.log
jout=`/opt/app/oracle/product/as10g/jdk/bin/bjavac -J-Djavac.pipe.output=true $1 $2 $3 $4 $5 $6 $7 $8 2>>/tmp/compile.log`

This actually compiled the necessary classes from the tag libs, and I noticed several tag libraries were coming back from time to time. For Example we have a <main:header /> tag which includes our common header. This comes back in each JSP. For every JSP this tag library was recompiled. Even worse – if you would have a tag library which formats numbers <main:numberformatter /> and used if several times within the same JSP, it would be recompiled for each occurence of the tag. Since we have custom tag libraries which implements HTML blocks on several parts of the site, the overhead was clear and something had to be done about keeping recompiling tag libraries over and over. The first attempt was the following script:

#!/bin/bash
echo “===============================================================” >> /tmp/compile.log
echo “$$ $PPID : $*” >> /tmp/compile.log
filename=`echo $4| cut -d’@’ -f 2`
javafile=`tail -1 $filename`
filet=`echo $javafile| cut -d’.’ -f 1`
classfile=”$filet.class”
ffname=`echo $classfile| awk -F/ ‘{print $NF}’`
echo “Checking if $classfile exists” >> /tmp/compile.log
if [ ! -s "$classfile" ]
then
echo “Must compile: $classfile”>>/tmp/compile.log
jout=`/opt/app/oracle/product/as10g/jdk/bin/bjavac -J-Djavac.pipe.output=true $1 $2 $3 $4 $5 $6 $7 $8 2>>/tmp/compile.log`
else
echo “Skipping $classfile” >>/tmp/compile.log
fi
echo “Done, exit code: $?”>>/tmp/compile.log

However Ojspc always terminated with a JspParseException. It didn’t like me skipping class files when it was necessary, pitty. After looking further through the Ojspc.jar files I noticed timestamps of the class files were important to please Ojspc, so we wrote another little script that fiddles with the timestamp of the .class file. The “else” branch in our above script became:

echo “Skipping $classfile” >>/tmp/compile.log
java  FileUtil $classfile>>/tmp/compile.log

The java FileUtil program fiddled with the timestamps of the already generated .class file and added 10000 msecs to the lastmodified time. Hence ojspc was always happy. The FileUtil tool looked like this:

import java.io.File;
import java.util.Date;

public class FileUtil {

public void changeFiletime(String filename) {

File fileToChange = new File(filename);

//Check time of the file/directory
Date filetime = new Date(fileToChange.lastModified());
System.out.println(filetime.toString());

if (fileToChange.setLastModified(fileToChange.lastModified()+10000))
System.out.println(“Successfile+10000!”);
else
System.out.println(“Failed!”);

//Check time again
filetime = new Date(fileToChange.lastModified());
System.out.println(filetime.toString());

}

public static void main(String[] args) {
FileUtil fileutil = new FileUtil();
fileutil.changeFiletime(args[0]);
}
}

Ojspc noticed the timestamp of the skipped .class file had changed and it was happy again, no more errors and the compilation times for our project dropped from 2 hours to 20 minutes. With all the documentation in hand and a workaround ready … it was time to open up a support request at Oracle Support. Following SR was created:

SR 7395808.994: OJSPC Batch translation keeps recompiling tag libraries was opened at Oracle Metalink.

Several weeks later a bug acknowledgement was released and after another while I received a temporary patch for ojspc.jar which works against 10.1.3.3.0. You can follow the bug process in following Bug report:

BUG 8418503 – OJSPC COMPILES THE SAME TAG CLASSES REPEATEDLY WITHOUT REASON

The patched ojspc.jar skips the compilation of .class files which already have been compiled, with the overhead gone from my workaround scripts, compile times dropped further to 7 minutes.

We have been working with these Oracle compiled builds now for over several months and have not encountered any problems. If you need to try the temporary patch contact me.

The fix is expected to be released in the patch-set for 10.1.3.5.

Quite some preparation work had been done by the developers and the database administrators to ensure a (temporary) replication could be setup between the old RAC and the new RAC, so all data would be transferred more or less “automatically” into the new RAC.

And then the big day came. We stopped the OC4J containers, we changed the datasources while the database administrators synced the RACs a last time and when ready we started the OC4J containers back to run on the new RAC.

Uh-oh…blocked threads

We were happy the upgrade went smooth and the websites were running more crispier than ever until … the containers started to lock up for some non-apparent reason. Nothing we hadn’t done before, so I took a thread dump of the OC4J to see what was going on.
Thread t@788: (state = BLOCKED)

- oracle.jdbc.pool.OracleImplicitConnectionCache.retrieveCacheConnection(java.lang.String, java.lang.String, java.util.Properties) @bci=0, line=651 (Compiled frame; information may be imprecise)
- oracle.jdbc.pool.OracleImplicitConnectionCache.getCacheConnection(java.lang.String, java.lang.String, java.util.Properties) @bci=4, line=549 (Compiled frame)
- oracle.jdbc.pool.OracleImplicitConnectionCache.getConnection(java.lang.String, java.lang.String, java.util.Properties) @bci=65, line=430 (Interpreted frame)
- oracle.jdbc.pool.OracleDataSource.getConnection(java.lang.String, java.lang.String, java.util.Properties) @bci=31, line=432 (Interpreted frame)
- oracle.jdbc.pool.OracleDataSource.getConnection(java.lang.String, java.lang.String) @bci=16, line=203 (Interpreted frame)
- oracle.jdbc.pool.OracleDataSource.getConnection() @bci=33, line=179 (Compiled frame)

This thread was actually deadlocked by

Thread t@29: (state = BLOCKED)
- oracle.jdbc.driver.PhysicalConnection.closeLogicalConnection() @bci=0, line=1581 (Compiled frame; information may be imprecise)
- oracle.jdbc.driver.LogicalConnection.cleanupAndClose(boolean) @bci=31, line=267 (Interpreted frame)
- oracle.jdbc.pool.OracleImplicitConnectionCache.closeCheckedOutConnection(oracle.jdbc.pool.OraclePooledConnection, boolean) @bci=45, line=1503 (Interpreted frame)
- oracle.jdbc.pool.OracleImplicitConnectionCacheThread.runAbandonedTimeout(long) @bci=155, line=258 (Interpreted frame)
- oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run() @bci=93, line=81 (Interpreted frame)

Also we were seeing a lot of connections to the OC4J TCP port which were on TIME_WAIT.

netstat -an | grep 12501 | wc -l
—> 444

A ticket was opened at Oracle in which we explained the situation. The first deadlock was caused by the abandoned-connection-timeout property which we had put to 60. Oracle advised us to remove the entry or put it on -1. We had this in our datasource just for security reason, if ever a query took more than 60 seconds we would break it off rather than keep it running for possibly much longer.

I found it a strange solution since we had not changed anything to the application level, never did we upgrade JDBC drivers or anything else … but ok … after setting the abandoned-connection-timeout property to -1 or just removing it, we started getting another behaviour. Now we would get Connection Timeout exceptions from the JDBC driver … well … kinda logica if you don’t kill anything anymore, I guess the pool got used up for some reason. The stacktrace would look somewhat like this

08/12/23 03:32:26 java.sql.SQLException: Io exception: Io exception: Connection timed out
08/12/23 03:32:26 at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:77)
08/12/23 03:32:26 at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:132)
08/12/23 03:32:26 at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:200)
08/12/23 03:32:26 at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:264)
08/12/23 03:32:26 at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:575)
08/12/23 03:32:26 at oracle.jdbc.driver.ClobAccessor.getString(ClobAccessor.java:296)
08/12/23 03:32:26 at oracle.jdbc.driver.T4CClobAccessor.getString(T4CClobAccessor.java:77)
08/12/23 03:32:26 at oracle.jdbc.driver.OracleResultSetImpl.getString(OracleResultSetImpl.java:2138)

Ok we do heavily use Clobs, as sometimes was not adviced by some people who also run sister-sites. But avoiding a feature in Oracle because you know it might give you problems – we thought that was too far fetched. We saw this exception when using one of the latest JDBC-drivers available. As Oracle told us to “try this one out”.

Meanwhile we started writing restart scripts, using a bit of python, when the socket would not answer after a decent period of time, we’d issue an OC4J restart so at least we could serve client request more or less decently. And since christmas and new year was coming closer, everything went just a tad slower.

What followed was a whole list of snoops and database traces. If you care to follow this SR just let me know your email-address as a comment. Meanwhile I wrote a wrapper for our DAO and noticed that queries that were using CLOBs were sometimes not finishing. This means that some of those queries stayed “running” and thus eventually we ran out of connections in the connection pool.

We were told to fiddle around with the sqlnet.expire_time. After experimenting a bit we found out that this value was set to 10 minutes. Setting the sqlnet.expire_time to 4 minutes helped us quite a bit, as it took much longer for JDBC to fill all te connections in the connection pool. At least our website was more stable (and needing less restarts).

Quite some days related we encountered the following snoop (IP addresses obfuscated)

61035 173.904541111.11.111.11222.22.22.222 TCP ingreslock > 46083 [PSH, ACK] Seq=115508 Ack=30515 Win=49680 [TCP CHECKSUM INCORRECT] Len=911
61073 174.318971111.11.111.11222.22.22.222 TCP [TCP Retransmission] ingreslock > 46083 [PSH, ACK] Seq=115508 Ack=30515 Win=49680 [TCP CHECKSUM INCORRECT] Len=911
61274 175.158981111.11.111.11222.22.22.222 TCP [TCP Retransmission] ingreslock > 46083 [PSH, ACK] Seq=115508 Ack=30515 Win=49680 [TCP CHECKSUM INCORRECT] Len=911
61998 176.839470111.11.111.11222.22.22.222 TCP [TCP Retransmission] ingreslock > 46083 [PSH, ACK] Seq=115508 Ack=30515 Win=49680 [TCP CHECKSUM INCORRECT] Len=911
63332 180.198899111.11.111.11222.22.22.222 TCP [TCP Retransmission] ingreslock > 46083 [PSH, ACK] Seq=115508 Ack=30515 Win=49680 [TCP CHECKSUM INCORRECT] Len=911

This was said to be a retransmission problem by Oracle. Liking this explanation more we opened a ticket at Sun to see if they had noticed any issues like this. Sun coordinated with Oracle to look at the snoops and proposed us the following kernel parameter:

set ip:dohwcksum=0

And with this kernel parameter … all problems went away. Now I really am not a Solaris expert to know what this kernel parameter does. I read it’s a hardware checksum. I personally think it’s weird to remove a “check” which is supposed to say if things are correct or not … but hey … Sun and Oracle told us to do it, and the problems are gone.

And after 3 months the end of this SR reached a Soft Close state and our website ran smoothly again. Never think a database upgrade might not impact your site obviously.

It all started back in November 2007 when I was asked by my colleagues at Foreach if I could come and help out to do a performance review. The site in question was the site of De Tijd (or L’Echo in french) which was being ported from Microsoft SQL Server and ASP to a mix of Oracle Database (for articles) and Microsoft SQL Server 2005 (for quotations). The frontend would be handled by Oracle Application Servers (OAS) – with the usual Oracle Containers for Java (OC4J – version 10.1.3.3.0) and Apache servers above. In front of that there’s the usual setup of load balancers and Oracle WebCache servers.

Performance review

Of course you start by the usual performance bottlenecks. Check primary keys, integrities, add unobstrusive logging to the DAO objects (log4j), start caching relevant objects with reasonable expire times (ehcache) … do more stress tests, and so on … Performance was reasonably satisfactory for launch and then that day finally came. The site was launched.

Blocked threads

It wasn’t before long until we started seeing strange behaviours on the origin servers (the servers behind the webcaches). At random times they would just lock-up, needing a restart of the OC4J container about every day. After doing a thread dump we would notice several blocked threads like these:

frame: 0: com.evermind.server.http.HttpSite.getApplication(HttpSite.java:418)
frame: 1: com.evermind.server.http.AJPRequestHandler.initAJP(AJPRequestHandler.java:1011)
frame: 2: com.evermind.server.http.AJPRequestHandler.initRequest(AJPRequestHandler.java:615)
frame: 3: com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:243)
frame: 4: com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:187)
frame: 5: oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
frame: 6: com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
frame: 7: java.lang.Thread.run(Thread.java:595)

Most of them were also pointing to the URLs of the detail of an article. After finding no relevant bottlenecks in our application. We did notice a lot of BLOCKED threads when we did a stress on the article page. If only I could look at HttpSite.getApplication(). I tried finding the source code for Orion, but of course that failed. So I reverted to look at the bytecode from oc4j-internal.jar *coughs*.

Once you head over there you will see that this method looks for a contextCache which is based on the path. The path is actually just the name of the URI in this case (without the querystring). You will also notice there’s a JVM lock there – when the path is not found, creating a new entry in the contextCache. So euhm wait … we have pretty URLs … does that mean we are “poluting” the contextCache with entries to our pretty URLs and locking the whole thing down for no reason ?
Seems like it was … we needed a quick fix so we changed the backend to work with non-pretty URLs and used a ProxyPass to put the logic of turning pretty URLs in non-pretty URLs.

This means you end up with 2 virtual hosts. One that does the frontend serving with pretty URLs and ProxyPasses to … the same machine (other vhost) with non-pretty URLs.

Result

No more blocked threads. contextCache size was kept to a minimum. Apache handles the ProxyPasses quite well. Stable running sites since then.
Are there drawbacks ? Yes:

• You are doing things more than once by ProxyPassing
• You loose your pretty URLs and need to rebuild them when generating page … but hey that can be done pretty quick performance wise
• You have to be careful how you configure your applications and mount points.
• You have to be careful how you redirect … You don’t want to redirect to a non-pretty URL (which might only resolve internally anyway).

So seems like having your pretty URLs handled by OC4J was not a good idea. I can’t keep but wonder if Tomcat would have had any issue with this…