ICE causes rebooting on 7100+ recorders beginning today

[DeltaMikeCharlie]

I waited 30 minutes.  Reconnected the network cable and forced an ICE login from the menu.  The EPG seems to have reloaded and it is stable.  Strange.

UPDATE: It occurred to me that my investigation last night could not point to ICE as being the definitive cause of the reboots.

My PVR rebooted 5 times, but each time my wife tried to restart the recordings that we were missing.

When I removed the network cable, we also did not restart the recordings.

Unless ICE "fixed" the problem between 20:30-21:00 last night, then another cause could be likely.

[IanL-S]

I have had the problem on a 7100+ running 18 November 2013 firmware and a 2400 (now) running 29 Jan 2016 firmware. In each case, this is the latest firmware. Four other Toppys (7100+, 2400, 5300) running latest firmware do not appear to have had the problem.

The TRF-2400 seems to have recovered completely (I think I did a new firmware installation) but the 7100+ got so bad I disabled IceTV. I will look more closely at the 7100+ today to see if there is some other cause, although rebooting when trying to make device selection does seem telling.

Ian

[IanL-S]

It is definitely a problem with the IceTV servers; see my latest post in the thread below:

http://forum.icetv.com.au/iceforum/topfield/9/trf-2400-plus-tf7100hdpvrt-reboot-when-logging-in-to-icetv-melbourne/5051/

Ian

[Dave at IceTV]

Now that it also seems to be affecting Topfield 2400's in Sydney, I checked to see what FW I'm running on as so far I've not had any issues with this.

I'm still on:

July 2012 FW,
System ID302020,
Loader vs 1.02,
Device Vs 1.01
Application Vs TF-BCPF 1.18.00

Hopefully this might help resolve whether later FW version(s) are at fault. I have nearly all Sydney stations in my EPG, Inc ABC2 of course.

Version information of my Sydney Topfield PVRs - which are not rebooting.

TRF-2460
System ID   32026
Loader Version   v1.03
Device Version   v1.01
Application   TF-BCPF 2.02.00
Last Update   Aug 4 2014

TF7100pvrt
System ID   32050
Loader Version   v0.48
Device Version   v1.0.01
Application   v1.11.42
Last Update   May 6 2010

[Dave at IceTV]

It does not appear to be an IceTV problem. I have never experienced it here with my 2460 or 7100pvrt. We've only had about 5 emails from customers having the rebooting issue. Plus the 5 or 6 people posting here equals about a dozen (from different regions), which is a tiny percentage of total Topfield owners using IceTV.

Topfield PVRs have always been memory limited when it comes to processing guide data - especially the 7100. As new channels came online over the years IceTV has needed to implement hacks to work around the Topfield's limited memory. Hacks like sending 50% of the schedules in the first fetch and the remaining schedules in the next fetch. IceTV support has also needed to advise customers to limit the number of channels enabled in their account. Setting channels with multiple LCNs to 1 specific LCN instead of leaving the channel set to 'All' became a requirement 3 or 4 years ago to help the Topfield PVRs cope with their limited memory (or limited memory allocated to IceTV processing).

Note: This is different to Topfields in regional areas that will always reboot if all channels in the IceTV account are enabled and set to 'All' because there are duplicate/conflicting channels if they are all enabled. Conflicting channels like 7mate on LCN 70 and 7 Digital on LCN 70. And duplicate channels like Prime Northern and Prime Southern both on LCNs 6, 60 and 61.

This hardware or firmware limitation is more obvious after a factory reset or service scan when the Topfield needs to process the full 7 day EPG for all channels plus process all schedules for the whole week. In normal use the Topfield only needs to process a couple of hours of new EPG data and a few new schedules, as it already has the other data.

On my 2460 and 7100 I have all Sydney channels enabled but zero schedules setup and my Topfields have never rebooted since the new channels came online. Too many schedules has never caused rebooting before.

- Too many schedules causes a Topfield to stop updating schedules.
- Too much guide data causes Topfields to reboot.
- Conflicting channels enabled causes Topfields to reboot.

There are other things that can cause a Topfield to reboot but they aren't relevant here - unless people are running TAPs (Topfield Applications) or their Topfield has a hardware fault that only causes the Topfield to reboot when stressed to the limit.


As a test, someone with the rebooting issue could limit the number of LCNs and channels in My Account on the IceTV website to see if the rebooting stops. If it does, then re-enable those channels to confirm that this was the cause. Then disable them again to double check. Disabling channels with more guide data (more programs) like some of the ABC and SBS channels might get a result quicker.

Note: Channels need to be disabled on the right hand side (unticking on the left hand side just hides that channel from the website).

If we can be sure that too much guide data is the cause of the rebooting we can implement a work-around for Topfield PVRs.

[IanL-S]

I reduced the number of LCNs and the rebooting may be fixed. At any rate it immediately downloaded the EPG data, something it refused to do before. Will report progress.

I disabled LC 9, 93, 44, 3.

Ian

[DeltaMikeCharlie]

My 2400 is rebooting again this afternoon.  I will try to limit the LCNs and see what happens.

Update:  I removed some duplicate channels and have had no further reboots.  I will try adding channels tomorrow and see what happens.

[Jimmyd]

It does not appear to be an IceTV problem. I have never experienced it here with my 2460 or 7100pvrt. We've only had about 5 emails from customers having the rebooting issue. Plus the 5 or 6 people posting here equals about a dozen (from different regions), which is a tiny percentage of total Topfield owners using IceTV.

Topfield PVRs have always been memory limited when it comes to processing guide data - especially the 7100. As new channels came online over the years IceTV has needed to implement hacks to work around the Topfield's limited memory. Hacks like sending 50% of the schedules in the first fetch and the remaining schedules in the next fetch. IceTV support has also needed to advise customers to limit the number of channels enabled in their account. Setting channels with multiple LCNs to 1 specific LCN instead of leaving the channel set to 'All' became a requirement 3 or 4 years ago to help the Topfield PVRs cope with their limited memory (or limited memory allocated to IceTV processing).

Note: This is different to Topfields in regional areas that will always reboot if all channels in the IceTV account are enabled and set to 'All' because there are duplicate/conflicting channels if they are all enabled. Conflicting channels like 7mate on LCN 70 and 7 Digital on LCN 70. And duplicate channels like Prime Northern and Prime Southern both on LCNs 6, 60 and 61.

This hardware or firmware limitation is more obvious after a factory reset or service scan when the Topfield needs to process the full 7 day EPG for all channels plus process all schedules for the whole week. In normal use the Topfield only needs to process a couple of hours of new EPG data and a few new schedules, as it already has the other data.

On my 2460 and 7100 I have all Sydney channels enabled but zero schedules setup and my Topfields have never rebooted since the new channels came online. Too many schedules has never caused rebooting before.

- Too many schedules causes a Topfield to stop updating schedules.
- Too much guide data causes Topfields to reboot.
- Conflicting channels enabled causes Topfields to reboot.

There are other things that can cause a Topfield to reboot but they aren't relevant here - unless people are running TAPs (Topfield Applications) or their Topfield has a hardware fault that only causes the Topfield to reboot when stressed to the limit.


As a test, someone with the rebooting issue could limit the number of LCNs and channels in My Account on the IceTV website to see if the rebooting stops. If it does, then re-enable those channels to confirm that this was the cause. Then disable them again to double check. Disabling channels with more guide data (more programs) like some of the ABC and SBS channels might get a result quicker.

Note: Channels need to be disabled on the right hand side (unticking on the left hand side just hides that channel from the website).

If we can be sure that too much guide data is the cause of the rebooting we can implement a work-around for Topfield PVRs.

We have a 2460 and a 2400 and both of them had the rebooting constantly. As per your comments we disabled the channels we didn't need, made sure channels showing all were adjusted to their specific channel number. This appears to have fixed the issue. Both PVRs have stopped rebooting.

As for your comment of not many people having this issue, our normal course when having issues is to read threads like this to see if there has been a solution but we don't normally post anything, this is the first time. So, just because you don't have a large amount of complaints doesn't automatically mean that there aren't that many people effected as I'm sure there are many people like us that don't normally post.

Thanks for the final solution.

[Dave at IceTV]

As a test, someone with the rebooting issue could limit the number of LCNs and channels in My Account on the IceTV website to see if the rebooting stops. If it does, then re-enable those channels to confirm that this was the cause. Then disable them again to double check. Disabling channels with more guide data (more programs) like some of the ABC and SBS channels might get a result quicker.

Note: Channels need to be disabled on the right hand side (unticking on the left hand side just hides that channel from the website).

If we can be sure that too much guide data is the cause of the rebooting we can implement a work-around for Topfield PVRs.

We have a 2460 and a 2400 and both of them had the rebooting constantly. As per your comments we disabled the channels we didn't need, made sure channels showing all were adjusted to their specific channel number. This appears to have fixed the issue. Both PVRs have stopped rebooting.

As for your comment of not many people having this issue, our normal course when having issues is to read threads like this to see if there has been a solution but we don't normally post anything, this is the first time. So, just because you don't have a large amount of complaints doesn't automatically mean that there aren't that many people effected as I'm sure there are many people like us that don't normally post.

Thanks for the final solution.

Thanks for the feedback Jimmy.

You are right. If there are 5 people posting the same problem on a forum we can be sure there are many more people that have the same problem but don't post.

[DeltaMikeCharlie]

You are right. If there are 5 people posting the same problem on a forum we can be sure there are many more people that have the same problem but don't post.

Perhaps you could analyse the logs of your PIMP server and see how many closely timed EPG fetches there are from the same PVR.

From memory, the PVR should attempt a PIMP fetch every 15 minutes, if you find the same PVR fetching every 5 minutes or so before being throttled, you could have a PVR stuck on a reboot cycle.

It's up to you, but if you were to detect such a PVR, would you take any action like possibly emailing the user to advise them of the solution or even drop a few channels on their behalf?

You may not even need to check the log, you could just email every user that has got a PVR registered as a Topfield telling them that there MAY be a problem and to check on the Topfield forum for a solution.

[Dave at IceTV]

Perhaps you could analyse the logs of your PIMP server and see how many closely timed EPG fetches there are from the same PVR.

From memory, the PVR should attempt a PIMP fetch every 15 minutes, if you find the same PVR fetching every 5 minutes or so before being throttled, you could have a PVR stuck on a reboot cycle.

That's actually a good idea.

Humax PVRs will fetch every 1 or 2 minutes if their network connection is too slow or dropping packets. TMS Topfields are fetching exactly every 2 minutes if there are too many channels enabled.

Detecting the problem and proactively emailing the user to advise them of the solution would reduce our support workload.

or even drop a few channels on their behalf?

The problem with this is how do pick which channels a customer doesn't need without asking them first.

You may not even need to check the log, you could just email every user that has got a PVR registered as a Topfield telling them that there MAY be a problem and to check on the Topfield forum for a solution.

That is also a good idea.

Though I feel it would be better, for Topfield PVRs, if we automatically send half the channel's EPG data in one fetch and send the rest in the next fetch. So each alternate fetch gets a different half of their guide data. We already do the same thing with timers for Topfield PVRs. We may even be able to tell the Topfield to do it's second fetch sooner than 15 minutes.

- Fetch 1, send 1st half of EPG data and 1st half of timers (plus inform PVR to do next fetch in 3 minutes).
- Fetch 2, send 2nd half of EPG data and 2nd half of timers (plus inform PVR to do next fetch in 15 minutes).
- Repeat step 1 at next fetch.
- Repeat step 2 at next fetch, etc.

The potential complication is that we can't send new timers for shows whose guide data has not been sent yet.

The simple and reliable solution may be to send today, tomorrow and the next day in the 1st fetch, then the other 4 days in the 2nd fetch. This may be how we already split the amount of timers that we send to Topfields.

[DeltaMikeCharlie]

I did some testing today, and the problem does not seem to be related to the total amount of memory available to the PVR.  Both the PVR's memory reporting and the Linux OS showed ample RAM available.

My guess would be that the PVR does the fetch directly to RAM with a fixed buffer allocated.  Too much EPG data returned from the fetch overruns the buffer and reboots result.  I have a TAP that implements the basic PIMP protocol, but it writes fetched data to disk first and it does not reboot.

From my understanding of the PIMP protocol, the PVR provides a date range and channel list for each EPG fetch.  Too many requests within a certain period results in throttling that PVR for a period of time.

You would need to not only ignore the date range, but keep track of alternating calls to ensure that over 2 calls, you provide one whole set of EPG data.  If you sent half the channels instead, the logic is similar.

The most pragmatic action may be to just notify Topfield users to restrict their channel lists.  You could tell Topfield Korea too, but a resolution may take some time to materialise.

I have also noticed that the channel list is global to your user account.  Users with multiple PVR models may not want to restrict certain channels for all of their PVR makes/models.

[IanL-S]

The most pragmatic action may be to just notify Topfield users to restrict their channel lists.  You could tell Topfield Korea too, but a resolution may take some time to materialise.

That is an option, however given their response to the MPEG4 playback navigation issue they will not fix the 7100HDPVRt Plus firmware. However, there is no harm in trying. If Dave and Delta Mike Charlie could collaborate in putting together an explanation of the problem and the potential solution I will send a bug report to Topfield Australia. As the Chief Moderator of the Topfield Australia Forum I do this from time to time, but these days I get no response, although they did fix one problem I reported.

I have also noticed that the channel list is global to your user account.  Users with multiple PVR models may not want to restrict certain channels for all of their PVR makes/models.

Yes, in retrospect I have my PVRs which are across two account, poorly located. The SKIPPA and T2, neither of which has this problem, are on different accounts. Being able to configure each device would be great, but it may confuse some people. Probably you would have a default that could be modified for each device (if needed).

Ian

[Dave at IceTV]

I have a TAP that implements the basic PIMP protocol, but it writes fetched data to disk first and it does not reboot.

If you have access to the TAP's source code can you change it to use a fixed amount of RAM and see if you can work out approximately how much (little) RAM the pimp process is allocated? I'd like to ask Topfield to increase the allocated RAM by 50% but need to know if that is feasible with the amount of free RAM available.

My guess would be that the PVR does the fetch directly to RAM with a fixed buffer allocated.  Too much EPG data returned from the fetch overruns the buffer and reboots result.

That's what I suspect too, now that you have verified that the PVR has plenty of free RAM available.

Another effect of the 'not enough memory allocated' issue is that even when there is enough memory to process the EPG without the PVR crashing and rebooting it can often fail to set any new reservations.

The most pragmatic action may be to just notify Topfield users to restrict their channel lists.  You could tell Topfield Korea too, but a resolution may take some time to materialise.

Requesting a solution (firmware update) from Topfield Korea is the plan. But as any firmware update always takes time to arrive notifying all customers with Topfield PVRs is the immediate plan.

[vincent]

Just to add some more to the discussion, since I disabled ABC2 I have had no issues. The information given above does help to explain the issue, I will check what channels I have enabled but both times I have had issues it has been because I go chasing the new channels TENHD and then 7Flix. I think the second time I pressed the red button when in the ICETV setting on the 2400 to refresh the EPG data and bang it started the rebooting loop. I was playing around trying to get stuff sorted, I am also guessing a full reset will cause issues as that will trigger a complete refresh and collection of lots of data to process. This has to happen from time when timers stop being set automatically, look forward to that day.

Now that I have it fully working and no reboot since my last post I don't want to touch the 2400 as its working exactly like I want it (with no issues)! I am not missing ABC2 as I don't watch it but it does leave a big hole in JustEPG.