Greetings Chris, Florian!
This has been a tricky issue to diagnose, as there is nothing in the log
files that will indicate what is going wrong. We were seeing similar
behavior on our TDS (where we enable all of the services), but it was only
after analyzing the .hprof file generated on the OOM error that I was able
to diagnose the issue. What I have seen is that the memory is consumed by
objects created by the edal-java library, which is what powers/supplies the
WMS service (see attached hprof analysis image). Having seen that (25 GiB
of memory related to WMS requests), I was able to match up the OOM
generation time with WMS requests in our log files. I found that a single
WMS request (even a simple GetCapilibities request) can consume gigabytes
of RAM, even for modestly sized datasets. In this case, it was two
GetCapilibities requests that needed 25 GiB of memory to build the initial
WMS objects. Further analysis has revealed that this memory is freed up
pretty quickly once the initial internal WMS objects are built, but if you
have a few overlapping WMS requests generating these objects at the same
time, you can quickly exhaust memory. Since blocking WMS requests to the
problematic datasets, our TDS has not encountered an OOM error. That's just
a temporary fix, but truly addressing this will require a fix to the
edal-java library.
Sean
[image: OOM.png]
On Thu, Aug 28, 2025 at 1:38 AM Florian Obersteiner <
florian.obersteiner@xxxxxxx> wrote:
> Same issue here, with a slightly different setup (THREDDS 5.6 docker image
> running on an Ubuntu 24.04 VM behind a reverse proxy).
>
> It basically doesn't matter how much memory I give the thing - if I have
> all the services enabled, it crashes after about a day. I'm still in the
> process of figuring out what service (+file) exactly causes the crash. At
> the moment, I have only HTTP fileserver, ISO and NCML enabled and this
> seems to run fine. The thing is, you need to block services at the server
> (proxy) level, see also <https://github.com/Unidata/tds/issues/609>
> <https://github.com/Unidata/tds/issues/609>. Otherwise, all the LLM
> crawlers that nowadays populate the web will "find" the service, call it
> and potentially crash the THREDDS.
>
> Florian
>
>
> On 2025-08-27 20:33, Chris Woelkers - NOAA Federal wrote:
>
> Since I've gotten our public THREDDS server running I am having to restart
> it more often than expected, currently about once every two weeks. Every
> time I check the error logs I see the same message:
> java.lang.OutOfMemoryError:
> Java heap space
>
> I have increased the heap space sizes multiple times and am currently
> running with 4096m of initial space and 16384m of max space. Obviously that
> is not enough but I would rather not increase it again as this server is
> running out of unused memory space and just giving the application more
> memory hasn't worked yet. Having it set at 16G is already high for a Java
> application, at least from what I can gather from Internet resources, so I
> am wondering if anyone has any recommendations on what might be done.
> The server is running Debian 11 and THREDDS 5.6 is running on top of
> Tomcat 10.1.36 with an upgrade planned to the latest version of Tomcat and
> a further upgrade in the future to Debian 12.
>
> Thanks,
>
> Chris Woelkers
> IT Specialist
> National Oceanic and Atmospheric Administration
> Great Lakes Environmental Research Laboratory
> 4840 S State Rd | Ann Arbor, MI 48108
> Office: 734-741-2446
> Cell: 734-604-0059
> Voice: 734-219-3815
>
> _______________________________________________
> thredds mailing listthredds@xxxxxxxxxxxxxxxx
> To subscribe: thredds-join@xxxxxxxxxxxxxxxx
> To unsubscribe: thredds-leave@xxxxxxxxxxxxxxxx
>
> --
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Dr. Florian Obersteiner (he/him)
>
> Karlsruhe Institute of Technology (KIT)
> Hermann-von-Helmholtz-Platz 1
> D-76344 Eggenstein-Leopoldshafen, Germany
>
> Institute of Meteorology and Climate Research,
> Atmospheric Trace Gases and Remote Sensing (IMKASF)
> Data Management & IT
> Building 435, Room 408
> Phone: +49 721 608 28353
> Web: https://www.imk-asf.kit.edu/
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> To subscribe: thredds-join@xxxxxxxxxxxxxxxx
> To unsubscribe: thredds-leave@xxxxxxxxxxxxxxxx
>
