Marc "Slyoldfox" Vanbrabant

We needed to pump (or pimp) our BI environment with some sales data from the downloads of a recently released iPhone App. In order to do that I found the follow python script which worked just fine: http://code.google.com/p/appdailysales/. Now since we have a Microsoft SSIS environment, I ported (part) of this code to C# so it can be run in a .NET environment. Note that you might have to change the bold parts to suit your needs.

Here’s the code: Continue reading iTunes Connect sales report script in c#

So we use ojspc (Oracle Java Compiler) to compile our JSPs and tag libraries. This saves us the overhead when loading the sites that it needs to parse the JSPs and Tags and thus slows down the initial first hit on a page.

We used to be able to compile an .ear in about 2 hours (which I think was long already). This project has about roughly 130 JSPs and quite some tag libraries. After a redesign of the site the compilation times started to increase, upto 4 hours, 8 hours, … This was making me nervous as it meant we had to stop our sprints sooner … to test sooner … to start building sooner. This is were my quest for the initial problem began.

Why the hell does ojspc take so long to translate my JSPs??

Roughly .. translating a JSP could take upto 2 minutes, with about 130 JSPs, well that easily takes you 2 hours. I started messing around with ojspc.jar seeing if I could call it natively from code, or if I could override a class and maybe play with the internals of ojspc. Not much success here. After monitoring the ojspc process, I saw it was spawning a javac process every once in a while … probably compiling stuff – right? If only I could have a peak there and see what exactly it was doing … and so I did.

I replaced the standard javac binary with a javac bash script which looked more or less like this:

Continue reading ojspc slow compilation and tags being recompiled

And thus it became christmass and the new year 2009 was coming. When an upgrade was planned to the Oracle RAC so we could benefit from a new and faster storage.

Quite some preparation work had been done by the developers and the database administrators to ensure a (temporary) replication could be setup between the old RAC and the new RAC, so all data would be transferred more or less “automatically” into the new RAC.

And then the big day came. We stopped the OC4J containers, we changed the datasources while the database administrators synced the RACs a last time and when ready we started the OC4J containers back to run on the new RAC.

Uh-oh…blocked threads

We were happy the upgrade went smooth and the websites were running more crispier than ever until … the containers started to lock up for some non-apparent reason. Nothing we hadn’t done before, so I took a thread dump of the OC4J to see what was going on.
Continue reading OC4J blocked threads … part 2

Infrastructure Setup

It all started back in November 2007 when I was asked by my colleagues at Foreach if I could come and help out to do a performance review. The site in question was the site of De Tijd (or L’Echo in french) which was being ported from Microsoft SQL Server and ASP to a mix of Oracle Database (for articles) and Microsoft SQL Server 2005 (for quotations). The frontend would be handled by Oracle Application Servers (OAS) – with the usual Oracle Containers for Java (OC4J – version 10.1.3.3.0) and Apache servers above. In front of that there’s the usual setup of load balancers and Oracle WebCache servers.

Performance review

Of course you start by the usual performance bottlenecks. Check primary keys, integrities, add unobstrusive logging to the DAO objects (log4j), start caching relevant objects with reasonable expire times (ehcache) … do more stress tests, and so on … Performance was reasonably satisfactory for launch and then that day finally came. The site was launched.

Blocked threads

It wasn’t before long until we started seeing strange behaviours on the origin servers (the servers behind the webcaches). At random times they would just lock-up, needing a restart of the OC4J container about every day. After doing a thread dump we would notice several blocked threads like these:

Continue reading OC4J Blocked threads