Giving more memory to the Tomcat Service in Windows

I've been setting up our super-awesome new server this week and last night I ran into an issue that I thought I should blog about.

This is a Windows Server 2008 R2 64-bit machine with 8GB RAM and I am installing ColdFusion 9 running on Tomcat 6. And, as with everything Tomcat related, it was a pain in the ass and the documentation sucked.

To add Tomcat as a Windows service is pretty easy.

You just run:


<%CATALINA_HOME%>/bin/service.bat install

For those that don't understand the notation above, CATALINA_HOME is the directory where Tomcat is. So in my case, we are going to N:\tomcat\bin and running service.bat.

This sets up Tomcat as a service, but by default it only gives it some tiny bit of memory to work with (256m or something, like it's 2002 again). So, of course, the moment I deployed a second instance of ColdFusion 9 on this container, I started getting "java.lang.OutOfMemoryError: PermGen space" errors.

So I Googled, and I Googled, and I Googled. There are SO MANY "solutions" for this problem that it is ridiculous. Unfortunately, none of them worked (for me). First it seems like most people suggested setting either a JAVA_OPTS or a CATALINA_OPTS environment variable with the new settings, but that doesn't seem to work. I saw at least one person suggest that the Tomcat service will NOT use the environment variables, even if they are set. My experience says that that is correct. When I set those variables, the Tomcat service would not use them (even after a reboot).

The next solution I saw was to add the settings to a setenv.bat file in the %CATALINA_HOME%/bin directory. So I tried it. Again the service did not pick up the settings (NOTE: When I ran Tomcat manually using startup.bat it DID pickup changes in both of the above named locations).

The last thing I tried was changing the settings of the service itself using the incredible cryptic command line options that I won't even get into. You can see them here: http://tomcat.apache.org/tomcat-6.0-doc/windows-service-howto.html

Finally, I gave up. It was 1:15AM (this morning) and I was pissed and tired. So i went to bed and figured I would look again today. Of course, I dreamed about Tomcat troubleshooting, so that sucked.

This morning, I sat down and looked at it again. Then, I saw this little bit of text on the above referenced HOW-TO page.

Tomcat6w is a GUI application for monitoring and configuring Tomcat services
(Bold added for emphasis)

Everywhere else that they mention Tomcat6w they refer to it as the "Monitoring" program. This is the FIRST place I have seen it mentioned that it has a DAMN UI for changing the service.

So I ran it:


<%CATALINA_HOME%>/bin/tomcat6w.exe

And I got an error like "The service Tomcat6 does not exist" or something. Great! Well, that was helpful. Do some more digging.

Here are the rest of the instructions for Tomcat6w.exe:

The available command line options are:

//ES// Edit service configuration This is the default operation. It is called if the no option is provided but the executable is renamed to servicenameW.exe //MS// Monitor service Put the icon in the system try

Awesome. WTF does this crap mean? I've never seen command line options like this before. And, of course, there are NO examples (The open source way of doing things).

So I tried passing in the //ES// param, and I got the same error message. Then I saw the part at the end of the explanation that says that renaming the executable to servicenameW.exe would do something. So, i tried it. Of course, servicenameW.exe didn't do a thing, because then didn't mean to literally name it that (I feel stupid about it now, but I was ready to try anything then). They mean you should give it the same name as the service and append w.exe to the end. So I went to look for the name of the service.

By default it looks like the service.bat file mention above gives the Tomcat service the name "Apache Tomcat 6". So I tried that. Of course, it didn't work, that is just the "display name". If you go into the service config you'll find the actual name of the service here:

So I renamed the file to tomcatServicew.exe (I had given the service a custom name while following some other instructions. Later I found out that if you don't specify a name it will automatically call it tomcat6 and all of this will work).

And then, BOOOM-shaka-laka.

At first I thought it had just reopened the other service config window. But then I saw the tabs highlighted above. So I clicked on the Java tab.

Here were some of the options I was looking for. I knew I wanted to increase heap memory and such anyway. But there was not a separate field for the MaxPermSize. But the was another arguments field.

So here is what I ended up with.

Now my Tomcat service starts just fine and has the appropriate amount of memory. All my CF instances start as expected and are REALLY fast.

I hope this helps someone, because I could have used a post like this last night.

Comments
Matt Woodward's Gravatar Bummer Jason--sorry this was such a hassle! If you run into issues with Tomcat in the future feel free to email me. I'm not an expert but have quite a bit of experience with Tomcat on both Linux and Windows so I'm happy to help if you need it.

Not sure I covered the MaxPermSize setting in this post, but it certainly would have saved you some time. ;-)
http://mattwoodward.com/machblog/index.cfm?event=s...

It's a long post but at the very end under "one last thing" I cover the memory settings.

Also, I'm not sure you want to be changing thread stack size unless you have a specific reason to do so. Most of the docs I've read indicate that setting is fine at the default unless you're having specific problems related to thread stack size, which I've personally never run into.

This is a case where the GUI makes things more obtuse than simply adding these settings to the startup script or a config file directly.

Lastly, not that this seemed to be your issue, but they do have a service installer for Windows so you don't need to run the bit to install Tomcat as a service if you don't want/need to. You can grab the service installer and it handles all that for you.
# Posted By Matt Woodward | 10/8/10 9:26 AM
Jason Dean's Gravatar Matt,

Thanks for the comment (and the additional help you gave me through IM this morning).

Yeah, I knew about the service installer option for Tomcat, but I had had a bad experience with it once in the past where I could not get the service removed after something had gone wrong with the installer. It was likely just my lack of experience that was the real problem, so I wanted to take the more calculated/educational route. I think I am now better off for it.

Yeah, I wish I had seen your post. Hopefully one of the two can help others in the future.
# Posted By Jason Dean | 10/8/10 11:45 AM
Brad's Gravatar I am glad it's not just me that has nights like this ;) And yes, dreaming about troubleshooting all night is the pits. I have had similar bouts with J2EE servers on many platforms.
# Posted By Brad | 10/8/10 12:33 PM
RAJ's Gravatar I was banging my head to make the tomcat6w.exe to work.
Thanks a lot for sharing your experience, helped me a lot..
# Posted By RAJ | 11/16/10 12:09 PM
Darren's Gravatar Thanks - Been wondering how to get this sorted this morning, yours was the best info and worked.
# Posted By Darren | 7/27/11 6:29 AM
Brian Swift's Gravatar This was extremely helpful. Many thanks
# Posted By Brian Swift | 11/10/11 6:34 AM
cfusiondev's Gravatar Have you tried to setup debug via cfeclipse or cfbuilder? I have tomcat 7 as windows service and cfusion webapp setup, and the RDS connection works fine. For Debug, in CF Administrator, I have set port number as 5005, but whenever I try to test debugger connection in Eclipse, I get message that debugger is running on 51309 (port keeps changing all the time). I added the -Xdebug -Xrunjdwp:transport=dt_socket,address=5005,server=y,suspend=n line to tomcat7w.exe (java options basically) but I still get same random port numbers. Let me know if you have any idea why tomcat wont pick 5005 from options I have set. Needless to say, I have restarted several times without any luck.
# Posted By cfusiondev | 11/24/11 2:02 AM
Pranay's Gravatar Hi,
I have a similar problem in LINUX. How do you get this done on LINUX?
# Posted By Pranay | 8/11/12 3:45 AM
Jason Dean's Gravatar In Linux I believe you need to look for, or create a file called setenv.sh and put the JVM arguments in there.

So in <CATALINA HOME>/bin/ look for setenv.sh, if it isn't there, then create it. It's just a text file. Then add something like this to it.

export JAVA_OPTS="-Xms1024m -Xmx1024m -XX:MaxPermSize=128m"
# Posted By Jason Dean | 8/11/12 10:20 AM
Matt Woodward's Gravatar Minor point on using JAVA_OPTS vs. CATALINA_OPTS -- if you use CATALINA_OPTS instead of JAVA_OPTS then the settings will only apply to your Tomcat instance and not impact anything you may have that uses the JAVA_OPTS setting.
# Posted By Matt Woodward | 8/11/12 10:31 AM
Jason's Gravatar Thanks for that addition Matt. Great advice. I didn't know that.
# Posted By Jason | 8/11/12 11:13 AM
Kristiaan's Gravatar Thank you. Thank you! THANK YOU!
for posting this on your blog. It saved my day.
# Posted By Kristiaan | 8/13/12 11:11 AM
JDubya's Gravatar In Windows ... adjust the following Reg_dword

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\PortaServer\Parameters\Java\JvmMx REG_DWORD from 256 to ????
# Posted By JDubya | 9/19/12 1:43 PM
Sri's Gravatar There is a line in service.bat:

"%EXECUTABLE%" //US//%SERVICE_NAME% ++JvmOptions "-Djava.io.tmpdir=%CATALINA_BASE%\temp" --JvmMs 128 --JvmMx 256

This means that the service will be installed with these default JvmOptions, you can either update the existing service or delete it, add -DXX:MaxPermSize=256m to JvmOptions in service.bat and install it again.
# Posted By Sri | 9/25/12 7:42 PM
Marian's Gravatar Fantastic!!! Thank you very much.
My new motto of the day is BOOOM-shaka-laka :)
# Posted By Marian | 11/13/12 2:50 PM
jaime's Gravatar Thanks!
# Posted By jaime | 3/5/13 4:26 PM
Polgarine's Gravatar Thanks so much! You saved my day :)
# Posted By Polgarine | 5/6/13 4:35 AM
Rajesh's Gravatar Hi Jason,

Where do we find this service config?
My client has a Tomcat Service running on his PROD server
and under the installed folder (D:\Program files\Apache ...\Tomcat..\bin\)
there are no .bat or env files.

And since its a PROD server, I cannot afford to keep that
service down for too long. Their App is getting this PermGen Error.
& We need to handle it.

After 100s of posts that get stuck on the catalina.bat or setenv.bat or something (which are not in client's setup at all) I came to this post.
Which seems very useful.

But please tell me how exactly to find the service name. Basically how u got that "Apache Tomcat 6 Properties (Local Computer)" dialog.

Regards,
Raj (Distressed)

Regards,
Rajesh J
# Posted By Rajesh | 8/14/13 12:43 PM
Francis's Gravatar Many thanks, so helpful
# Posted By Francis | 11/9/13 8:54 AM
Balaji's Gravatar I have tried all the options listed above. Butstill the error occurs. But the error I m getting is "java.lang.OutOfMemoryError: Java heap space" Even after setting Initail memory pool and maximum memory pool to 4096MB in the tomcat configuration, the error exists. Its a Windows Server 2003 with 4GB RAM which uses Tomcat5.5 . I have also tried setting JAVA_OPTS environment variables in system properties.But it dint help. I m not sure if java picks up the environment variables set in the system properties. Is tre any possible way to check the current heap size?
Kindly help me.
# Posted By Balaji | 12/26/13 2:43 AM
Jason Dean's Gravatar You can't give 4096mb of RAM to Java if you only have 4gb of RAM in the machine. The operating system needs RAM too as do other applications, even Java needs more memory because Permgen space is not part of the heap.

Set that to a more reasonable amount, like 1024 or, at most, 2048. Then try it again. If you are still running out of heap space then check and see how much RAM is being used by other applications. If you don't have even 1024mb free, without Java running, then you may need to remove some software services or consider a RAM upgrade.

On a final note, I hope that is a Windows 2003 64-bit server, not a 32-bit server.
# Posted By Jason Dean | 12/26/13 12:05 PM
Balaji's Gravatar I have tried giving the values such as 512,1024,2048 etc... till 4072.Still the same error...FYI It's a Windows 2003 32 bit server...Pls find the RAM usage which I checked under performance tab of task manager

Physical Memory
Total : 4193492
Available : 2555308
System Cache : 2192252

Commit Charge
Total : 1540160
Limit : 8210328
Peak : 1933664

From the above info. I guess there is available memory of 2GB. But still I couldn't figure out wre the problem actually is...
# Posted By Balaji | 12/27/13 12:43 AM
Jason Dean's Gravatar What are you setting maxpermsize to?
# Posted By Jason Dean | 12/27/13 12:30 PM
Balaji's Gravatar I have set the maximum heap size to 2048
# Posted By Balaji | 1/2/14 12:21 AM
Balaji's Gravatar MaxPermSize I have used is 2048m
# Posted By Balaji | 1/2/14 12:28 AM
Jason Dean's Gravatar Like I said earlier, Permgen space is separate from Heap space. So if you are giving 2048mb to the Heap *and* 2048mb to Permgen you are giving 4GB to Java. What is the OS supposed to use.

You said you have about 2GB free. Try giving 1024mb to the heap (Max and Min) and 512mb to MaxPermSize.

If that doesn't work, try MaxPermSize 256mb or 128mb. You can also try lowering Max/Min heap to 512mb. At some threshold it should work.

If you continue running out of memory at those lower settings, then there is a problem with whatever you are trying to run on Java using up too much memory.

Is there a reason you are trying to give so much memory to Java on a sever that doesn't have much memory to start with? If you need Java to have a 2GB heap then you need to, at least, double the RAM on that machine. Even with the settings I just suggested, you are going to be using up much of your remaining available memory, you should be looking at going to 8GB anyway.
# Posted By Jason Dean | 1/2/14 9:24 AM
Balaji's Gravatar Hi,
I have tried 1024mb to the heap (Max and Min) and 512mb to MaxPermSize. I also tried lowering the MaxPermSize to 256mb and128mb and Max/Min heap to 512mb. None of the option worked out...
At last I have increased the RAM to 8 GB and tried with the following heap and Perm Size

Internal memory pool : 4096 MB
Maximum memory pool : 4096 MB
Max Perm Size : 1024m

Still the problem exists. I have also tried with max perm size of 2048m. But I'm getting the same error.

Is it because of 32 bit server? I came across certain sites which states that heap space cannot be increased beyond a certain limit of 1.5GB in 32 bit Windows server...Is it so? will this be the reason for the issue?

Kindly advise...
# Posted By Balaji | 1/17/14 8:14 AM
Jason Dean's Gravatar I'm sorry, somehow I COMPLETELY missed that you are using 32-bit windows. I'm afraid we've been wasting some time. I've been working under the assumption that this was 64-bit Windows. Sorry.

32-bit Windows can't even use the memory upgrade you installed. 32-bit Windows can only address 4GB of memory.

As for why you keep running out of memory even you set it to a lower reasonable value (MinMax: 512mb and MaxPerm: 256m), I do not know. Does it run out of memory while the JVM is spinning up? Or is it not until you run your application? If the later, then I would begin to suspect that there is something wrong with your application. If the former then I would wonder if you just have SO MUCH running on your server that you don't even have 750mb to spare.

Either way, you are now beyond my ability to troubleshoot from blog post comments. I recommend you bring in a JVM expert to help you figure out what's wrong.

Good luck
# Posted By Jason Dean | 1/17/14 10:17 AM
David's Gravatar Well, I used the service installer and ended up with the same issue as you anyway and found all those other answers before this post too(yesterday evening and this morning). I dreamt about zombies, not tomcat, but I believe it was an allegory. Very glad I found this post before executing a tentative plan to go live in the woods have have nothing to do with electricity ever again.
# Posted By David | 3/2/14 7:40 AM
Parvez's Gravatar Some JVMs need contiguous memory. So may you have more memory but not contiguous. Try with the minimum memory and if possible stop other applications that you can.

Some Windows 32 bit platform also have a limit of total memory that can be allocated which somewhere around 1576MB.
# Posted By Parvez | 3/14/14 10:32 PM
James Campbell's Gravatar Does doing tomcat6w //ES//<service-name> achieve the same thing?
# Posted By James Campbell | 3/19/14 2:10 PM
Kandarp Pandya's Gravatar Thanks alot buddy... Thanks a million... I am going through same situation... I got direction for how to solve... Thanks once again...
# Posted By Kandarp Pandya | 5/30/14 5:11 AM
jeff's Gravatar Honestly I'm not pretty sure if this solution will work, I was using tomcat7 and this sick permgen error occurs, I was reading your solution, but i cant find the tomcat7w on my bin folder, luckily I have a fresh tomcat 8 version, where it has a tomcat8w in it, so I changed to tomcat8 and modified those same options that you did. lets just see what will happen, but as of now, the deployment is REALLY FAST :))) .
# Posted By jeff | 8/12/14 12:27 AM
jeff's Gravatar btw, its damn 2014 already, I just typed at google "tomcat permgen space out of memory 2014", and I saw alot of the same solution, there is eve one I saw who modified exactly what you did, you might have started a very informative, well defined and easy work-around solution well I just want to say thanks to you!!
# Posted By jeff | 8/12/14 12:32 AM
jeff's Gravatar and btw, i just noticed the title "Giving more memory to the Tomcat Service in Windows", google will not give this link to me if i type "permgen space shakala bla bla".. well I'm really glad I bumped into your blogged!

Thanks! God Bless!
# Posted By jeff | 8/12/14 12:35 AM
Jason Dean's Gravatar @Jeff,

Glad it worked out. I suspect there was a tomcatw.exe in your Tomcat install, but it may have had a different name. For it to work it needs to be named after your Tomcat service. So if your service is named Hippoplatapus, then your file would need to be called Hippoplatapusw.exe. So depending on how your TomCat was installed, the tomcatw.exe may have had a different name.
# Posted By Jason Dean | 8/12/14 3:25 PM
BlogCFC was created by Raymond Camden. This blog is running version 5.9.1. Contact Blog Owner