EPG Blanks SBS 1

[simoncasey]

My EPG for SBS 1 is blank from 14:00 today until 01:35 tomorrow (Sunday)

Beyonwiz T4 in Melbourne

[csutak40]

My EPG for SBS 1 is blank from 14:00 today until 01:35 tomorrow (Sunday)

Beyonwiz T4 in Melbourne

Same at my house - T2 as well

[grampus]

My EPG for SBS 1 is blank from 14:00 today until 01:35 tomorrow (Sunday)

Beyonwiz T4 in Melbourne

Same at my house - T2 as well

+ one more

[prl]

There's a hole in the SBS HD data for Canberra, too, but a different one.

The displayed EPG for SBS ONE HDMon Sep 11 on my test Beyonwiz T4 is missing SBS World News Late Edition, The World Game and Ouro S1E1. 24 Hours in Emergency and Ouro E1S2 are displayed OK (i.e. the data for 22:35-00:35 is missing).

However, in the IceTV data sent from the server, all the entries are present:

Code: [Select]

2017-09-11T11:40:00+00:00 2017-09-11T12:35:00+00:00 24 Hours In Emergency Carpe Diem
2017-09-11T12:35:00+00:00 2017-09-11T13:05:00+00:00 SBS World News - Late Edition 2017 - Episode 180
2017-09-11T13:05:00+00:00 2017-09-11T13:35:00+00:00 The World Game 2017 - Episode 28
2017-09-11T13:35:00+00:00 2017-09-11T14:35:00+00:00 Ouro Series 1, Episode 1
2017-09-11T14:35:00+00:00 2017-09-11T15:35:00+00:00 Ouro Series 1, Episode 2

This seems to point to a problem on the Beyonwiz end, at least for that data. I should be able to replay the load of the capture of the data with more debugging enabled in the firmware to try to see where the problem lies.

I don't know whether this problem has any relation to the other EPG issues reported in this topic and elsewhere.

[Daniel Hall at IceTV]

Hi Guys,

I have just had a look at my T2 here and it also has a gap on SBS ONE HD from now up until 1:35am tomorrow morning (Movie: Melancholia is the next entry), but checking on the 2Tune here as well and it does not show a gap.

These both use the same API on the IceTV server and I have forced the T2 to perform a full refresh.

This is pointing to something in the T2, if you can shed some more light on it prl that would be awesome.

[prl]

I think I have a handle on this, but it would be nice to have some JSON dumps of IceTV EPG data for periods on regions that have this problem to confirm.

Daniel, are you able to get a JSON snapshot of the Melbourne EPG data for the period that people were having problems with recently?

I've also got most of a Python program written that can test for the problem on any full EPG JSON dump.

The root of the problem seems to be that the IceTV event ids are 32 bits, and the Beyonwiz EPG has a-bit-less-than-16 bit event ids (0x0001-0xFFF8). The conversion is done by taking the mod-0xFFF7 reduction of the IceTV eventid, but that modulus has too many factors for my liking (7, 11, 23, 37). I think it might work better using mod 0xFFF1, which is slightly smaller, but prime. The folding of event ids seems to be creating event id collisions, and that in turn is causing some events to be "terminated with extreme prejudice".

Using a non-prime modulus may make that more frequent than necessary.

[Daniel Hall at IceTV]

Hi prl,

No problems, will email you some files in a sec.

[prl]

As Daniel posted, he sent me some more EPG sample files that have this problem, and I've verified that the issue is with the mapping of the 11-digit (decimal) IceTV event ids to the 16-bit event ids used in the Beyonwiz EPG (and elsewhere on the Beyonwiz side, like in timers).

It boils down to the Birthday Problem (and is similar to the Birthday Attack).

With a 16-bit hash used to reduce the IceTV event ids to EIT event id size, and about 200 events/channel in the EPG (the problem only occurs when the same short event id is used for two different programs on the same channel), there's about a 25% probability that at least two different events will have the same hash, if the hashes are uniformly distributed. If that seems an unreasonably high probability, that's why the Birthday Problem is also called the Birthday Paradox.

When a collision occurs, the event already in the EPG with that EIT event id is removed, and the event that collided is added to the EPG. The EPG allows lookup of an event by its EIT event id, so multiple events with the same EIT event id can't co-exist in the EPG.

The IceTV event ids are not uniformly distributed, they tend to run in sequences, and the simple modulo hash used in the Beyonwiz IceTV plugin tends to preserve those sequences. That means that the collisions tend to cluster: if one collision occurs in a channel, there's a fairly high probability that a run of several collisions will occur, and that removes a block of entries from the EPG on the Beyonwiz side, even though all the data has been correctly sent and received. However, there seems to be about the same overall probability that collisions occur as in the uniform distribution case.

I have experimented with re-hashing the IceTV event id to remove runs (e.g. by using the IceTV event id's MD5 hash) before reducing it by the modulo hash. That stops blocks of missing entries in the EPG from happening, but it doesn't significantly change the total number of collisions and missing entries in the EPG. They're just scattered over more channels and are less obtrusive.

I don't yet have a solution to this problem that appeals to me in any way