User jobs delayed by transcode delay

[Scott.Atkins@…]

I have a user job and transcoding set up on a recording. Additionally, I have setup a 5 day delay on the transcoding. When the program is recorded, the user job will queue up, but it will not run. If I delete the transcoding job, or transcode immediately, the user job will then run. I have not found a way to change this behavior, which is very, very wrong. If I have a user job which exports the video to (for example) a Tivo, why can't I use the mpeg before it is transcoded?

For completeness this is user job 3.

[Scott.Atkins@…]

Jobqueue patch

[Scott.Atkins@…]

Looking at /trunk/mythtv/libs/libmythtv/jobqueue.cpp line 576 it is somewhat obvious what needs to happen. There is a second bug here, if AutoTranscodeBeforeAutoCommflag? is set, the delay setting is completely ignored. Third, the delayed jobs all start at the same time, making it impossible to tell which job goes with which recording and possibly straining the backends (since there is no staggering).

So I'm submitting this patch, it is completely untested (I don't have a build environment set up, but if I understand the code it should work +/- a syntax error).

[Scott.Atkins@…]

Even with my changes, there's still the matter of newly added jobs (meaning jobs the user explicitly runs) will still be queued behind the transcode job. I haven't dug in far enough to fix this.

[anonymous]

I believe the proper solution is to just make JobQueue::GetJobsInQueue?() sort the list by schedruntime instead of inserttime. JobQueue::QueueJob?() uses the current time for the schedruntime if a runtime is not passed in, so it's the same as if we set schedruntime to now() in the SQL like we do for inserttime.

If you want to test this, just change the "ORDER BY j.inserttime, j.chanid, j.id;" in the first SQL statement in JobQueue::GetJobsInQueue?() to "ORDER BY j.schedruntime, j.chanid, j.id;" and that should fix the issue.

[cpinkham]

last comment was from me. must... remember... to.... log.... in....

[Scott.Atkins@…]

Replying to anonymous:

I believe the proper solution is to just make JobQueue::GetJobsInQueue?() sort the list by schedruntime instead of inserttime. JobQueue::QueueJob?() uses the current time for the schedruntime if a runtime is not passed in, so it's the same as if we set schedruntime to now() in the SQL like we do for inserttime.

If you want to test this, just change the "ORDER BY j.inserttime, j.chanid, j.id;" in the first SQL statement in JobQueue::GetJobsInQueue?() to "ORDER BY j.schedruntime, j.chanid, j.id;" and that should fix the issue.

This would require a change to the comm flag after transcoding so that a date and time were used to ensure proper processing order; and of course my patch gets at the two other issues I mentioned (or at least it should, like I said, I don't have a dev environment set up).

[cpinkham]

If a user uses delayed transcoding then commflagging should still occur now, not after the 2-week delayed transcode. The JobQueue? lets you transcode now before flagging (ie, for something like transcoding from HD to SD before flagging so flagging runs faster) or it lets you run flagging now and delay transcoding. Since inserttime == now() always and schedruntime == now() effectively except for delayed transcodes, then sorting by schedruntime instead of inserttime should solve the issue. Test SQL statement seems to work correctly here using that theory. I may test it a bit further and commit a change in the next day or so when I get a chance.

[Scott.Atkins@…]

Replying to cpinkham:

If a user uses delayed transcoding then commflagging should still occur now, not after the 2-week delayed transcode. The JobQueue? lets you transcode now before flagging (ie, for something like transcoding from HD to SD before flagging so flagging runs faster) or it lets you run flagging now and delay transcoding. Since inserttime == now() always and schedruntime == now() effectively except for delayed transcodes, then sorting by schedruntime instead of inserttime should solve the issue. Test SQL statement seems to work correctly here using that theory. I may test it a bit further and commit a change in the next day or so when I get a chance.

Fair enough, I guess that would simplify my patch some. I just hope my stuff helped some, I've always want to something I use so much.

[Scott Atkins <Scott.Atkins@…>]

Yes, changing the sql statement works great for me (I had to create a build environment to track down another bug).

[cpinkham]

(In [16944]) Process JobQueue? jobs in schedruntime order instead of inserttime. This solves the issue of deferred autotranscode jobs holding up other jobs.

Closes #5042.

[cpinkham]

(In [16945]) Carry over [16944] from -fixes. Fixes JobQueue? ordering to run jobs in schedruntime order instead of inserttime. This was holding up other jobs when a deferred transcode was also scheduled.

References #5042.

[patrickhdonnelly@…]

Just a comment; I submitted part of the original patch that was used to setup the transcoding delay; the reason the jobs all start at the same time was actually to decrease I/O load on the backends during "peak" times. Rather than defer the jobs in exactly 24hour increments, and possibly start up an expensive transcode operation right in the middle of prime-time, when presumably several other recording, commflagging, and viewing operations are taking place, the jobs all start at midnight, where the JobQueueMaxSimultaneousJobs? setting should serve to stagger them.

[Scott Atkins <Scott.Atkins@…>]

Who's to say that midnight isn't "primetime" for some people?

Wouldn't it be just as valid to wait using 6/12/18 hours? And remember, this is based on stop time, not start time (at least that's what I remember). So wouldn't your method more likely hit the max jobs at a time, where as my method would not (provided you could transcode in less time than the recording)?

[ebenblues@…]

I have a user job that compresses a recording using mencoder and uploads it to a different server. I generally setup cut points, then start the transcode job to remove the cutpoints, then run my user job to compress and upload it. Right now, I can select the transcode job, then immediately after, select the user job and the user job will not start until after the transcode job is finished. Is this fix going to break this? If so, is there a way to specify dependencies between jobs? Thanks.