mythbackend memory leak

[anonymous]

I seem to have a problem with a memory leak. The backend runs fine for a while but after a day or two it starts to eat up all of the swap space on my machine. Valgrind gives me the following after running it for a couple hours:

==22603==    definitely lost: 7,483,036 bytes in 7,906 blocks.
==22603==      possibly lost: 54,448 bytes in 55 blocks.
==22603==    still reachable: 13,296,516 bytes in 287,277 blocks.
==22603==         suppressed: 0 bytes in 0 blocks.

I've never used valgrind before but it seems to imply that around 7 megs was lost.

I'm running ubuntu edgy (64 bit) on an Athlon 64 X2 4200+ Qt is version 3.3.6-3u Gcc is version 4.1.2 20060928 (ubuntu 4.1.1-13ubuntu5)

I configured using the following: ./configure --enable-xvmc --enable-dvb --enable-dts

My system has 2 gb of ram and 3.5 gb of swap. I have a pchdtv hd5500 and a wintv pvr-150 The video card is a a GeForce? 7300GS

The backend and frontend are on the same machine. I have attached the valgrind log.

[jaggroups@…]

valgrind log

[jaggroups@…]

backend log

[jaggroups@…]

new valgrind log

[jaggroups@…]

new backend log

[Stuart Auchterlonie]

I'll have a look at this, although it looks like it's in the upnp code.

For better backtraces within the valgrind logs you should rebuild myth with --compile-type=debug

Stuart

[Jeremy Green <jaggroups@…>]

I'm attaching a valgrind log file with debugging turned on. Thanks!

[Jeremy Green <jaggroups@…>]

valgrind log file with backend compiled with --compile-type=debug

[Stuart Auchterlonie]

Mem leak in upnp code

[Stuart Auchterlonie]

Can you try the patch i've attached to the ticket? The patch should fix the issue.

Stuart

[Jeremy Green <jaggroups@…>]

valgrind log (after patch was applied)

[Jeremy Green <jaggroups@…>]

It looks like the patch did something but that there are still some leaks in there. Or am I reading the valgrind log wrong? I'm running it for a little bit longer test (while it records a show) to see what happens. It seems to eat up most of the systems swap after about two days.

-Jeremy

[Stuart Auchterlonie]

(In [13317]) Refs #3317. Plugs a memory leak in the upnp code.

[Stuart Auchterlonie]

There's nothing wrong in the latest valgrind output you've sent. Please run it for a few days and report back. Also try doing what ever else it was you were doing when you made your first report. That exercised the upnp subsytem more extensively. Perhaps there were other media clients running on the network at the time (such as games consoles etc.)

Otherwise i'll close this in a few days...

Stuart

[Jeremy Green <jaggroups@…>]

Well, I don't know if this is abnormal or not but after 2 days of running top reports the following: 6339 mythtv 15 0 3002m 1.4g 14m S 0 69.1 8:49.08 1.6g mythbackend

So, mythbackend is using 3Gb of memory and 1.6Gb of swap. Is that normal?

Unfortunately I haven't been running valgrind because the backend becomes useless for HD content when running in valgrind.

[Stuart Auchterlonie]

Replying to Jeremy Green <jaggroups@brownout.com>:

Well, I don't know if this is abnormal or not but after 2 days of running top reports the following: 6339 mythtv 15 0 3002m 1.4g 14m S 0 69.1 8:49.08 1.6g mythbackend

So, mythbackend is using 3Gb of memory and 1.6Gb of swap. Is that normal?

No definitely not. We'll need some more information about what devices you have on your network (like media players, that sort of thing...)

Stuart

[Jeremy Green <jaggroups@…>]

I have a Denon AVR 4306 that is the receiver for my AV system. It supports upnp but I don't use it at all. I just played around with it a little. It sees the myth backend on the network. I'll attach another set of logs from when I was messing around with it over the network. I have an ADS Media-Link but I don't think that supports upnp. I also have a buffalo Linktheater that is currently not on the network.

[Jeremy Green <jaggroups@…>]

valgrind log while attempting to connect from upnp avr-4306

[Jeremy Green <jaggroups@…>]

I'm going to try running the backend for a while with --noupnp on the command line to see if it makes any difference to the memory usage. I'll let you know if I notice anything.

[Jeremy Green <jaggroups@…>]

Well, the backend seems to still be using an ungodly amount of memory:

10970 mythtv 15 0 1577m 1.3g 14m S 0 64.0 4:44.61 289m mythbackend

This less than a day after I started it with --noupnp So, I don't know how much of the upnp code this disables but it disables it all then that seems to imply that the problem is elsewhere. Right?

[spam@…]

Not sure if this is related or not, but there seems to be a leak somewhere in the code that runs the built in webserver on a master backend. I was monitoring my backend via nagios and after a re-install using monit, and memory usage climbing extremely fast.

I've updated the monit config on my master backend to not do the http check, only a PID and memory resource check about a week ago and since then it has not grown in size.

To confirm, I ran 'links -source ​http://localhost:6544/ > /dev/null' in a 100 loop for loop and memory jumped from 241virt/21res to 349virt/123res in less than a minute.

I am running monit (checking the http port) on a slave backend and it does not have this leak.

[Darryl Ross (spam@…]

Sorry, forgot to mention, this is on SVN 13468.

[anonymous]

I see the same problem running r13445. I recently installed monit to restart in the event of a crash. After monitoring the system overnight, mythbackend was using 70% of the memory. Eventually it crashes.

[postmaster@…]

same problem here r13181...
probably memory leak introduced here:
​http://svn.mythtv.org/trac/changeset/13022/trunk/mythtv/programs/mythbackend/httpstatus.cpp

[Stuart Auchterlonie]

Replying to postmaster@profesionalweb.com:

same problem here r13181...
probably memory leak introduced here:
​http://svn.mythtv.org/trac/changeset/13022/trunk/mythtv/programs/mythbackend/httpstatus.cpp

The only way to prove that's at fault is to update back to 13021 and try that, if that doesn't leak then move to 13022 and try that and observe the leak. *Probably* isn't of much use.

Anyway, I can now reproduce this to some degree. I'm just trying to work out exactly who is the culprit. It's not leaking as such, somebody is holding onto memory.

Stuart

[postmaster@…]

Hi stuart, if it really helps you i can do the revision (r13021 and r13022) test to be 100% sure that revision its the faulty one, that will maybe save you some time... If you are already on the track that may not be necessary.

ismael

[billymacdonald]

I recently upgrade to SVN 13790 and have a process that queries the http server on 6544. That seems to be eating memory still.

[anonymous]

I get a lot of these using pmap:

b20b4000    8188 rwx-- 00000000b20b4000 000:00000   [ anon ]

in fact:

pmap -d `pidof mythbackend` | grep 8188 | wc -l
84

I believe these are stack allocations from each time a new thread is created, so it implies that maybe we're not joining/terminating threads properly (this is after about 3 days with mythbackend using 774MB of virtual memory) Is there a quick way to determine which thread a block of memory belongs to, and where that thread got created?

[anonymous]

A followup to my last post.... I checked the number of running threads, and found it to be significantly smaller than the number of stack allocations reported by pmap.

I wrote a wrapper around libpthread to keep track of stack allocations and do a backtrace of where the 'pthread_create' call came from

It turns out that all of the cases where there is allocated stack but no running thread came from dvbrecorder.cpp::StartDummyVideo?() (there is a pthread_create in that function that is initiating this memory usage)

At this point I run out of steam. I don't see anything obvious in run_dummy_video() or RunDummyVideo?() that would prevent libpthread from reclaiming the stack-space, yet it appears that is what is happening at least in my case. But maybe this will be useful to someone more familiar with this code.

[anonymous]

Sorry for the rapid-fire posts, but here is an answer to the issue I found above. I found that simply doing:

pthread_detach(_video_thread);

right after the 'pthread_create' in dvbrecorder.cpp:StartDummyVideo() allows the stack allocation to be cleaned up properly.

[billymacdonald@…]

Patch to resolve Memory Leak

[billymacdonald@…]

Attached is a patch to fix this leak I believe.

The ConfigurationGroup? class has a virtual function for deleteLater() which does not call it's base class's deleteLater(). Therefore, the EventLoop? never knows to delete this derived class of QOBject. I haven't done a lot of C++ coding so I don't know if what I did is the "proper" way to call the base class.

This would effect any derived class of ConfigurationGroup?, which ScheduledRecording? is derived from, which is a member of the ProgramInfo? class.

[Janne Grunau]

(In [13823]) Closes #3317. Fixes Memleak by applying patch from billymacdonald <AT> gmail <DOT> com

ConfigurationGroup::deleteLater wasn't calling QObject's deletaLater resulting in never actually deleting the object