I Put A Spell On You, Because you’re mine: Aka Why is TomCat Holding Onto Jars?

I don’t normally like turning to the blogosphere for help, but I can’t for the life of me figure out why TomCat 5.5 refuses to release 4-5 jars from some web-apps when they’re undeployed. It’s like the old CCR song is talking about jars:

I put a spell on you
Because you’re mine.
You better stop
The things that you’re doin’.
I said “Watch out!
I ain’t lyin’, yeah!
I ain’t gonna take none of your
Foolin’ around;
I ain’t gonna take none of your
Puttin’ me down;
I put a spell on you
Because you’re mine.
All right!

This issue happens to me on a couple of machines here, and it’s happened at a client site too. I drop a WAR file into TomCat’s deploy dir, and it deploys the web-app ok (creating a dir with the same name as the WAR file). I then kill the WAR file, or drop in a newer version, and for both cases, TomCat tries to undeploy the old version by deleting the expanded directory. But it continues to hold onto 3-6 of the jars in the WEB-INF/lib directory, so can’t actually kill everything completely. Then it actually gets confused and tries to deploy that directory as a web-app, complaining that there’s no web.xml, which is true since there’s nothing there but some jars in the lib dir.

This is unbelievable frustrating as it essentially kills hot-deploy. I would think it’s something in my environment, however, as I said, it’s happened at a couple of client sites too.

Has anybody else had this issue with TomCat 5.5? This essentially kills hot-redeploy. Instead I have to stop TomCat, kill the left-over expanded dir (if I try while it’s still running Windows complains that the jars in WEB-INF/lib are still in use), and re-start. I can’t find any bugs in the TomCat bug tracker that seem related to this.

Updated 2005-8-23: Carlos Sanchez suggested trying the antiJARLocking config property on the TomCat Context, as described here. This actually seems to do the trick. One way to apply it is to modify the file
    conf/context.xml
in your TomCat install and change
     ...  
to
     ...

Of course this affects all apps, but I don’t necessarilly see this as a big deal. While it’s supposed to slow things down a bit, I really think this should have been the default.

Alternately, one of the other ways you can set this is inside a
   META-INF/context.xml
file inside your app itself. I guess the only thing that rubs me the wrong way about that is the fact that this is a pretty generic filename that might conflict with another lib that expects a file by that name. On the other hand, the advanatage is that if you know your app is going to have this issue in TomCat, you can just take care ahead of it ahead of time. In any case, this is mostly a dev-time issue (I doubt anybody trusts hot-redeploy in TomCat in production), so the first solution is fine for most people.

(Note when looking at the above XML and that in the comments that the braindead WordPress edit box sanitizer makes all xml tags lowercase. The context element above has an initial uppercase.

 

25 responses

  1. Guillaume Poirier says:


    My guess is the webapp is undeploy, but its ClassLoader hasn’t been garbage collected, so it stills has a lock on some jar files. There’s many reason that could cause the ClassLoader to be retained when the webapp is undeployed.

  2. Guillaume Poirier says:


    Bye the way, you can look at my posts on the Spring’s mailling list for more details on that problem (assuming it is indeed what I think it is).

    http://search.gmane.org/search.php?query=&email=gpoirier16%40globetrotter.net&group=gmane.comp.java.springframework.devel&sort=date


  3. My guess is that you are on Windows. Java and Windows (specifically NTFS) have interesting interactions with regards to file locking. This is a reason that JBossAS actually twiddels around with Tomcat and does a different deployment semantic that works better with regards to hot deployment. Dunno if you can swing that but it would be one way to get it working fast. If the classloader is the issue as the commenter above says then maybe make sure you’re on the latest 1.4.2_xx as there were problems fixed in this area or upgrade to Java 5.0_xx which seems to handle classloading (and garbage collection) much better.

    -Andy

  4. Colin Sampaleanu (blog author) says:


    It’s a tough call as to what’s going on. It’s definitely Windows that’s preventing me from manually deleting the directory while TomCat is running, but presumably that’s just because they’re still in use by TomCat, and locked.

    I just tried a test now with one app, which uses Spring and Hibernate. Although other times, I’ve seen as many as 5 or 6 jars held onto, this time it’s just ehcache.jar, and this is consistent across multiple attempts.

    When I do an undeploy (or hot-redeploy) in JBoss 4.0.2, the same jar is also not released (and JBoss even puts out a warning that the temp expanded dir is only going to be killed on the next restart). But because JBoss _does_ use a separate differently named location for each deploy/redeploy, the redeploy doesn’t fail, as it doesn’t have to write on top of the old jar.

    What is interesting is that Resin has absolutely no problem redeploying the app, and it even re-uses the same location for the expanded web-app location it dumps the contents of the WAR file to, not resorting to another dir every time like JBoss. I’m not really surprised here; Resin even back in 1999 when I first started using it had excellent hot-redeploy abilities. Resin is in fact even very amenable to not even deploying your app to it as a WAR file. Just point it to your web-app in expanded format in your target build directory, and it will pick up changed classes automatically, and generally being intelligent enough to restart the app itself as needed.


  5. Have you tried the antiJARLocking and antiResourceLocking properties in the context? IIRC that forces tomcat to copy files to a temp directory and use that ones instead.
    http://jakarta.apache.org/tomcat/tomcat-5.5-doc/config/context.html#Standard%20Implementation

  6. Colin Sampaleanu (blog author) says:


    Thanks Carlos, antiJARLocking=true seems to do the trick! I think that should be the default, but it’s better than nothing…


  7. The other option that will do the trick as META-INF/context.xml is putting the file under “TOMCAT_HOME/conf/Catalina/localhost/yourapplicationname.xml”, and you have

    <context docBase=”yourapplicationname” antiJARLocking=”true”> ... </context>

    You may need to change docBase if the application is not under TOMCAT_HOME/webapps, and maybe put the file in a subdir other than localhost eg. if you’re using virtual hosts in Tomcat config.

  8. Guillaume Poirier says:


    Preventing the jar files to be locked is better than nothing I guess, but the real problem is that the ClassLoader is not garbage collected, which means multiple hot-deploy will provoke an OutOfMemoryError, and possibly hold external resources such has TCP/IP connections or file locks, if any are opened by the webapp rather than the container. If the ClassLoader is not collected, all its classes definition, their static members and dependencies will stay in memory.

    On a developement server I guess it’s usually not too much of a problem though, but I would personally avoid hot-deploy on a production server, unless the webapp was carefully tested not to cause such leaks.


  9. AFAIK, the memory leak problems still in J2SE 5.0 are:

    o ThreadLocal, typically when a static references a ThreadLocal that reference an instance loaded by a class of the web app’s class loader.
    o java.beans. I believe Apache Commons Digester triggers this problem. I can’t remember the details off hand.
    o java.sql.DriverManager. But probably not something to use in web apps.
    o ResourceBundle keeps a soft reference if using the ListResource technique rather than .properties.

    I’ve submitted patches for the first three to Sun, but they don’t seem very keen thus far.

  10. Taras Tielkes says:


    The antiJARLocking/antiResourceLocking will only cause a copy of the files to be locked, not the actual files themselves. If you’re doing many redeployments, prepare for 100s of megabytes of (locked) Jars in the Tomcat work directory.

    Perhaps you’re using the “classpath*:” pseudo URL prefix to load resources from JAR files. This is broken (in practice) on Win32. (One of the JVM classes, JarURLConnection IIRC will keep a file lock open indefinately)

  11. Christoph Henrici says:


    This is one of the most useful blogs since long…..;-)
    This issue has been following me the last half year …
    I’m on Windows XP and Tomcat 5.0.x and have tried all sort of things, which never added to any help.
    And worst i felt terrible alone with my problem ;-)
    Looking forward to try out the proposed solutions.


  12. thanks! really useful… had the same problem with ehcache stopping redeploys. now tomcat redeploys like a dream.

  13. Arron Ferguson says:


    Thank you for this solution. Unfortunately it breaks something else:

    ServletContext ctx = getServletContext();
    webAppRoot = ctx.getRealPath(”/”);

    After changing:


    to

    Yes, it does allow your ANT build file to go ahead about its business by hot deployment but because it’s not locking, Tomcat has to rely on temporary copies which it so gleefully places in the catalina/temp folder naming each successive version n+1.

    Because of this, your Web app getRealPath returns something different each time (assuming you use this method call which is reasonable to assume). It looks as though Apache has made hot deployment a pinata that is just out of reach.

  14. Arron Ferguson says:


    Sorry that the above post did not include the:


    to

    It should read:

    <context> … </context>
    to
    <context antiJARLocking="true" antiResourceLocking="true"> … </context>

  15. Colin Sampaleanu (blog author) says:


    Keep in mind that
    ctx.getRealPath(”/”)
    is not even guaranteed to work in all servlet containers. Some containers don’t actually extract out the entire contents of WAR files anyway, and return null for this method.

    WebLogic for example, uses a special zip file url handler it registers in its environment (referring to resources as ‘zip:my/war/file/path#the/internal/resource.ext’), to load all simple files directly from the war. Mostly the only thing it extracts is jar files.

    So a portable app should not really rely on that method anyway…

    The biggest defect of the above mechanism with TomCat is that it does consume a decent amount of space. Above and beyond that, it still doesn’t work for quite all situations. If you attach to the app with a debugger, TomCat will still end up locking the files for some reason.

  16. Arron Ferguson says:


    Colin,

    You have a good point. Both the Servlet Spec (2.2+) and Javadocs basically point to what you just said:

    Servlet Spec (2.2+):
    The getRealPath method takes a String argument and returns a String representation of a file on the local file system to which that path corresponds. The getPathTranslated method computes the real path of the pathInfo of this request.

    In situations where the servlet container cannot determine a valid file path for these methods, such as when the web application is executed from an archive, on a remote file system not accessible locally, or in a database, these methods must return null.

    Javadocs:
    This method returns null if the servlet container cannot translate the virtual path to a real path for any reason (such as when the content is being made available from a .war archive).

    This is a real condundrum. Do you know of an alternative to using getRealPath() that *all* servlet/jsp containers will honor?

    *downloads CCR: I Put A Spell On You*
    *listens*
    *sees article relevance*

    :/

  17. Colin Sampaleanu (blog author) says:


    Realistically the only portable solution is to rely only on loading resources through the classloader, off the classpath, and never expect to be able to call getRealPAth.

    Now you do come across web-apps which expressly say they must be installed in expanded form (not WARed), exactly because they do expect to be able to call this method and get a proper result.

  18. Arron Ferguson says:


    Thanks for the reply Colin. But the path is not just for classes. I have a Web app that references XML, XSLT templates and media (e.g. images) from the file system and so knowing where it these files are is imperative.

    This is the part of Web app development that I love to hate with a passion.

  19. Colin Sampaleanu (blog author) says:


    Unfortunately that may well mean that your app is one such app that needs to always be deployed in expanded form, or on a container that fully expands WAR files…


  20. Actually, XML and XSLT should live on the classpath and be read as resources… you shouldn’t really be reading anything contained in the war, and fortunately, there’s never a time where you actually need it explicitly on the file system anyway… Java’s InputStream is flexible enough.

    At any j2ee release, they may mandate that wars are not exploded (they’ve hinted at it so far) at which point any code like this breaks, unless you manually explode (and therefore deploy)

  21. Victor Ott says:


    Tomcat hot redeployment: you shouldn’t rely on that very much on productive systems. As others stated before, the memory leakage is tremendous. Even with PermGen memory space set to astronomic values (max=256MB) certain applications can’t be hot redeployed more than 15-17 times in Tomcat. Maybe it is Bill’s revenge? ;-)


  22. thanks alot guys very good job.

  23. Larry Singer says:


    I have just been investigating this exact problem. I found the reason one of my JARs was being held was that a library was not closing the InputStream to a resource it was loading. I now have ehcache only stuck. I’ll try the context switche suggestion to see if it fixes it.

  24. Larry Singer says:


    Creating the context file didn’t work for me to remove ehcache.jar. What did work was placing an ehcache.xml in the classes directory.

  25. Al says:


    Best title for a blog entry I’ve seen in some time. BTW, it’s actually a Screamin’ Jay Hawkins song, covered by CCR. Clearly there’s a little bit of the devil spirit of SJH in Tomcat :)

One trackback

Leave a Reply