Increase sleep_time in truncate-delete patch to accomodate remote mounts

[ylee@…]

As someone who uses a NAS that uses EXT3 (which can't be changed) to store his MythTV recordings, I'm looking forward to the truncate-deletion patch to go live as part of 0.20. However, in my experience when using various iterations of it on my 0.19-fixed system, sleep_time in mainserver.cpp needs to be significantly increased for those of us with remotely-mounted recording directories. That is,

const size_t sleep_time = 500;

should be

const size_t sleep_time = 10000;

This is because the initial latency with any remote writes (at least on my Infrant ReadyNAS 600) is fixed. Any deletion between 0K and about 1GB or so take about the same amount of time, about two seconds or so. After about 1GB, the elapsed times increase linearly based on the size of the file being deleted, but the initial latency remains. Thus, depending on the remote filesystem being able to properly respond to repeated truncations each one half-second apart is not practical.

Enlarging sleep_time to 10000 (ten seconds) should a) be more than sufficient to take care of the initial network latency and will b) still truncate at the same pace (e.g., 4MB every one-half second is the same as 80MB every ten seconds).

[ylee@…]

I've had 0.20 running for a day and a half and here's what I've seen regarding the truncate deletions on my CIFS-mounted NAS:

• Without 'mythbackend -v file', truncations don't happen at all. That is, files mythbackend tries to delete do get renamed to cifsxxxx (see ​http://www.gossamer-threads.com/lists/mythtv/commits/208981#208981) but never shrink in size.
• With 'mythbackend -v file', I see the following:

2006-09-13 14:05:45.948 About to unlink/delete file: '/misc/cube/mythtv/store/xyz.mpg' 2006-09-13 14:05:46.925 About to delete file: /misc/cube/mythtv/store/xyz.mpg.png 2006-09-13 14:05:51.085 AutoExpire?: 1 file truncates are pending 2006-09-13 14:05:51.085 Truncating '/misc/cube/mythtv/store/xyz.mpg' by 4.00 MB every 500 milliseconds 2006-09-13 14:05:51.086 Reschedule requested for id 0. 2006-09-13 14:05:51.176 Event socket closed. No connection to the backend.

Yes, '-v file' consistently crashes mythbackend.

So, there appear to be two issues here:

• The issue I discussed in my opening ticket regarding too-fast network deletion requests resulting in crashes. I think changing sleep_time to a larger value will indeed fix this.
• A separate issue of truncations not proceeding at all (and mythbackend thus not crashing) when '-v file' isn't used.

[cpinkham]

If you're getting a crash then you need to follow the instructions in the HOWTO to get a backtrace so it can be debugged. Obviously it doesn't crash for everyone with "-v file" since we tell people to turn on debugging all the time. "-v file" is just a debugging option, it doesn't affect what code gets executed or not, it only enables some verbose debug messages, so it doesn't make a bit of sense that the truncate would not happen without the "-v file" option.

I can probably agree with you on the sleep_time issue, but the rest of this is sounding a big off-base.

[ylee@…]

I've attached, as truncate-debug.txt, the debugging log from mythbackend as it dies after attempting to delete a file with '-v file' and truncation turned on. To recap, this is 0.20 (ATrpms) with a cifs mount to an Infrant ReadyNAS 600.

[sphery <mtdean@…>]

The attached patch (mythtv-stat_before_unlink.patch) fixes the crash described in the attached backtrace by getting the file size before the delete operation and passing it to TruncateAndClose?(). With the CIFS drivers, it seems that st_size is set to 0 immediately upon unlinking--even if the file is open--so we cannot wait until TruncateAndClose?() to get the size. The patch also simplifies TruncateAndClose?() by removing all the st_blksize calculations and simply truncating based on file size, per the request of several developers.

The patch does not address the sleep time issue in the ticket. However, due to the behavior of the CIFS drivers that caused the crash, it's possible the sleep time issues Yeechang observed were actually side effects of an invalid file size estimation that is corrected by this patch. Yeechang, please test with this new patch and see if it fixes the other issues described in the ticket.

Thanks to jwestfall, janneg, and Chutt for helping me figure out what the CIFS drivers were doing and to Captain_Murdoch for helping come up with a plan for fixing the issue relatively cleanly.

[ylee@…]

I am pleased to report that, with Mike Dean's patch, truncated deletes work again on my 0.20 (ATrpms) system, with a couple of minor caveats.

The three standards I'm using to gauge the patch's behavior are 1) mythbackend no longer crashing (even with '-v file'), 2) properly truncating the recording (renamed to cifsxxxx or .nfsxxxx), and 3) eliminating the remaining stub file after truncation is done. Using NFS on my NAS always properly resulted in 1), 2), and 3). Back when I was successfully using Boleslaw's truncation patch on 0.19-fixes with the same cifs NAS filesystem, ​only 1) and 2) were true, and the same holds for this patch; however, it's easy to ​work around (although the phrase is now "All file truncates have finished," not "are finished.") for the time being. (By the way, ​I believe a firmware upgrade on the NAS broke both Boleslaw's 0.19-fixes patch and 0.20's pre-patch truncation method.)

As mentioned, the only outstanding issue is 3), having the cifsxxxx stub sticking around. My current cifs mount options in /etc/auto.misc are:

-fstype=cifs,noperm,mapchars,brl,sfu,serverino,directio,iocharset=utf8,rsize=32768,wsize=32768 :NAS/RECORDINGS

Perhaps this will give someone a clue as to fixing 3), but, again, I can easily work around this peculiarity for the time being.

Also note that I have not yet tested a) whether this patch breaks the previous 100%-correct truncation behavior seen under NFS and b) whether mythtranscode will also truncate properly. (FYI, mythtranscode in 0.20 apparently attempts to use the truncation method for deleting old versions of recordings even when truncation was turned off in mythtv-setup; that is, I'd find the old versions renamed to cifsxxxx but otherwise intact. Should I open another ticket?)

In conclusion, my sincerest thanks to Mike and to the other contributors in getting this issue resolved.

[cpinkham]

(In [11312]) Grab the filesize before unlinking the file when performing the truncating delete code in mythbackend. Some filesystems seem to lose the filesize when the file is unlinked so trying to get the size after the initial unlink would previously fail.

Closes #2364 using patch by Michael T Dean.

[cpinkham]

(In [11313]) Carry over [11312] from trunk.

Grab the filesize before unlinking the file when performing the truncating delete code in mythbackend. Some filesystems seem to lose the filesize when the file is unlinked so trying to get the size after the initial unlink would previously fail.

References #2364.