I spent Monday running more tests with ffmpeg to get better data.  I am now forcing a specific number of prediction frames over a regular interval.