Odi's astoundingly incomplete notes

While researching about the scalability of the Wicket framework, I came across what seems like a widespread myth. In a couple of mailinglists and forums people were basically saying: "Scalability is basically limited by how much data is stored in a session. But memory is cheap, so don't worry". Sad, that this just slightly misses the point.

Scalability is limited by 4 things:

• CPU
• Memory
• Disk Throughput
• Network Bandwidth

Real applications typically hit only one of these limits. If we just look at the web framework, then the DB is out of our focus and we assume that this is not the bottleneck. If you're not doing video streaming then network bandwidth is not likely to be an issue for you. And if you're doing video streaming you better have plenty of spare bandwidth anyway. If you are not doing file serving (or again streaming) disk throughput will also not be an issue. What remains is CPU and memory. If CPU is your limit (the webapp does number crunching), then there is not much you can do: optimize your algorithms. So the only really interesting limiting factor is memory.

Now in Java we have a garbage collected heap. So memory is not completely independent from CPU. GC can consume a fair amount of CPU and it causes latency. So it is really important that the application does as little garbage collection as possible. In a generational GC heap (Concurrent Mark and Sweep - CMS) there is also cheap and fast garbage collection of the relatively small Eden space (the youngest generation) and expensive and slow garbage collection of the Old generation (the bulk of the heap). It's key that you prevent the big collections from happening. They really hurt. Rather have 10'000 small collections instead of 1 big one. That means that you should not "churn" objects, be conservative with what you allocate. And it means that the ones that you allocate temporarily (during a request or shorter) should have the shortest possible lifetime. Each object that survives the Eden collection is a performance killer: it has to be moved to the next generation and will have to be collected from there eventually by a big collection!

So if each request produces 1 MB of garbage and your Eden space is 100 MB, each 100 requests will cause a small collection. Fine if these are the only requests on the system. The requests are over, so all objects are unreferenced and can be collected. The collection will be quick and efficient.
If you now make 200 requests in parallel, the Eden space will be full half-way through the time. If those objects are still referenced now they can not be collected and will be moved to the next generation and you have hit the scalability limit: the Old generation will quickly fill up and big collections will have to run, comsuming CPU and causing latency. If however most of these objects are no longer referenced, they can be collected quickly and your webapp still scales.

What about session state now? Session state doesn't change much usually. So it is more or less constant. Its objects will live in the Old (or even permanent) generation. It's certainly a bad idea to constantly add and remove objects to/from the session, because these objects will have to be collected in the Old generation by a big collection. Session state is also really small usually. Just a few KB. Do the maths: a 2 GB heap can hold 20'000 sessions of 100 KB. That's more than enough and will not be the bottleneck. Of course don't store huge data in sessions.

To sum up this posting: if you want a scalable Java webapp:

• reduce object allocation
• no object churning
• prefer short-lived objects
• clear references as soon as possible
• keep response times as short as possible: longer response time means that request scoped objects may survive a small GC
• keep session state small
• rarely add/remove objects to/from session
• use the CMS garbage collector
• size the eden, old and permanent generations sensibly (100MB, 2GB, 64MB is a good example) and inspect their use with JConsole

For frameworks that means:

• careful with state save/restore: this creates a lot of garbage for every request
• careful with parsers: they can create a lot of garbage
• careful with HTML formatting: string ops easily create garbage
• careful how much you keep during the lifecycle of a request (request, response, context objects)
• careful with filter/interceptor architectures (streams): they tend to copy data with every added layer, plus add to the stack
• careful with (stream) buffering: buffers are relativly big chunky objects likely to end up in the Old generation or exhaust Eden space quickly.

Java objects that are infamous for a lot of overhead in terms of object allocation:

• Iterator (Java 5 shorthand for loops!): when used heavily e.g. to recursively iterate over tree structures
• HashSet, HashMap, Hashtable, LinkedList: wrap each inserted object
• ByteArrayOutputStream.toByteArray(): copies the internal array
• ArrayList: allocates new array when growing
• XML parsers and transformers (XSLT)
• String operations
• some JDBC drivers like Oracle
• reporting libraries like JasperReports

posted on 2009-03-27 01:03 UTC in Code | 1 comments | permalink