Topics

How does IceTV calculate its "Most Popular" rankings?

Code Select Expand

{
    "series_id": "121",
    "num_votes": "1088",
    "popularity_ratio": "1.00"
},
{
    "series_id": "26455",
    "num_votes": "1025",
    "popularity_ratio": "0.94"
},
In other words, how is the "num_votes" value calculated?

Is it the number of times that series is scheduled to record by IceTV users?

From what I can tell, the "popularity_ratio" is the "num_votes" for the series divided by the "num_votes" for the most popular series.  Is this correct?

Is the "Most Popular" data presented Australia-wide or for the user's local region?

How often is the "Most Popular" raking recalculated?

On the IceTV web site, there are filtering options for 'next 7 days', 'today', 'last 7 days' ... 'new series'.  Is there any way to apply these filters when using the JSON API?

During a JSON EPG request, would there be any benefit to adding the popularity data directly into any shows whose series also appear in the "Most Popular" list?  If an application wished to flag shows that are in the "Most Popular" list, as the system currently stands, it would need to fetch that list and match incoming EPG shows to the list.

I am not currently asking for any action to be taken.  I am trying to understand how this mechanism works.

What is ICETV's position regarding EPG data for '10 HD +1'?

Hi Daniel,

Can you please check the API documentation?

http://developer.icetv.com.au/#!/Shows/post_shows_timers_timer_id

The request '/shows/timers{timer_id}' may be missing a '/'.  I think that it should be '/shows/timers/{timer_id}'.

Also, I think that the HTTP request method should be PUT, not POST.

Apologies in advance if I have misread something.

Do IceTV still have any of their DVB tuners in stock?

[Would IceTV consider adding a feature that allows IceBox users to customise the format their own EPG content?]

Would IceTV consider adding a feature that allows IceBox users to customise the format their own EPG content?

The IceTV Kodi addon is written in the Python programming language.  It should be possible to call an external Python module from within the existing addon.

In its default configuration, the existing Kodi addon would remain relatively unchanged.  However, at startup, it would look for a user-defined Python module and set some flags if that module is found and is valid.

In the 'output_xmltv()' function, if the flags are true, service.icetv would then call those pre-defined external functions after the XMLTV object has been created, but before that object is converted to XML and sent to TVH.

This will allow the user to intercept and modify the XMLTV data before it is sent to TVH.

Because some data is critical, such as start/end times, these fields should not be customisable.

I suggest 3 functions in the external module for formatting 3 XMLTV elements:  'title', 'sub-title' and 'desc'.  Each of these modules would accept a single 'show' object as input and return a string containing the user-defined content for that field.

Once the module is initially loaded, each of the 3 functions can be tested for individually and flags set accordingly.

There would also need to be some try-except-else protection against syntax errors and the like within the user-defined module.

The existing Kodi addon would remain read-only in '/usr/share/kodi/addons/service.icetv'.  However, the external module would be located in '/storage/.kodi/userdata/addon_data/service.icetv/epg_formatter.py' (or similar) where it can be modified by the user.

I can post some code samples if this is of interest.

A logged-in user could report a problem with the EPG directly from within the 'tv-guide.icetv.com.au' web page.

When the EPG details are visible, in addition to the 'Recording Options' operations, a 'Report Error' operation could be added.  This could present the user with a screen containing a list of error types like 'Repeat status', 'Missing episode number', 'Wrong start time', etc, with maybe a free form text field for extra comments.

Once the error has been reported, the EPG will be flagged with a message that an error has been reported with a timestamp and error type.  'This entry was flagged as having a <type> error at <timestamp>'.

This could initiate a workflow within ICE TV to investigate/resolve the error.  Once the error has been resolved, the error displayed in the EPG could either be removed or shown with the resolved timestamp.

The XML / JSON feeds could also carry a flag indicating that an error may be present.

Comskip is used for detecting and skipping ads and as far as I can tell, the current comskip configuration produces a number of files.  The 3 files of interest are: Recording.edl, Recording.txt and Recording.vdr.

When Kodi loads a recording to be played, it requests the 'cutpoints' from TVHeadend.

TVHeadend will first try to read the cutpoints from Recording.txt.  If it is successful, it will stop there.  If no Recording.txt file is found, TVH will then try to read the cutpoints from Recording.edl.  Recording.vdr is never referenced.

I have been doing some work on TVH that will write 'scene marker' records into an 'edl' file for the start and then end of the EPG event.  If, for example, there are 2 minutes of pre-padding a 30 minute programme and then 5 minutes of post-padding, scene markers will be inserted at 2 minutes and 32 minutes.  These scene markers are written immediately after the recording has finished.

Unfortunately, when comskip runs, it overwrites the scene marker edl records that were just created.

I would like IceTV to consider making a few changes so that these two features can coexist.

comskip.ini

Code Select Expand

output_default=0    ; Turn off the default comskip 'txt' output.
edl_skip_field=3    ; Use the ad break marker type (3) instead of the cut marker type (0).
output_vdr=0        ; Turn off the unneeded 'vdr' output.

In addition to this, I have written small proof-of-concept script that demonstrates the principle of saving the TVH scene markers before comskip runs and then reapplying them once comskip has created its edl file.

Code Select Expand

#!/bin/bash
#Get the file name only
BASEONLY=`basename "$1" mkv`
#Get the directory name only
DIRONLY=`dirname "$1"`
#Create the EDL file name
FULLNAME="$DIRONLY/$BASEONLY"edl
#If the EDL file exists, save the scene marker records (type = 2)
if test -f $FULLNAME; then
    cat $FULLNAME | awk '$3 == 2 {print $0}' > "$FULLNAME"_saved
fi

#Run comskip
/usr/share/kodi/addons/service.tvheadendicetv/bin/comskip --ini=dmc_comskip.ini sample.mkv

#If a saved EDL file exists, add that to the end of the new EDL
if test -f "$FULLNAME"_saved; then
    cat "$FULLNAME"_saved >> $FULLNAME
    rm "$FULLNAME"_saved
fi

With a combination of these two features, the user will be able to jump to near the beginning of the programme and then also skip the ads during playback.  Here is what the final result looks like, with annotations.



Thoughts and comments welcome.

If anyone is interested in getting FTA EPG for the Radio channels, I can post some detailed instructions.



Please note:  This will likely void your warranty.

I'm not sure if this is a known issue, or indeed how many users this could impact.

I have set my test/dev IceBox to prepend the repeat flag to the title field.

This causes a timer creation to fail.  Here is an excerpt from my log:

Code Select Expand

Title is not matching, do not schedule the show
Got [[R-DEV] Hard Quiz] but was expecting [Hard Quiz]
Show:
{'eventId': 1165, 'channelName': 'ABC TV', 'channelUuid': 'c80811a5c49a725d0bb2f3e6eb447f39', 'channelNumber': '0', 'channelIcon': 'imagecache/2', 'start': 1706686200, 'stop': 1706688000, 'title': '[R-DEV] Hard Quiz', 'subtitle': 'Series 8, Episode 18', 'description': 'Tom Gleeson is hanging on to the Big Brass Mug until one Hard Quiz expert can prove they deserve it - but with which topic? Australian parrots, cricket bats, bones, or WikiLeaks?', 'repeat': 1, 'subtitled': 1, 'image': 'http://images.icetv.com.au/17a6-dfb1-1fa9-d0e4.jpg', 'ageRating': 7, 'first_aired': 1672531680, 'copyright_year': 2023, 'nextEventId': 1166}
Timer:
{'id': '1706682258', 'device_id': 188516, 'channel_id': 2, 'name': 'Hard Quiz', 'show_id': 180966489, 'start_time': '2024-01-31T07:30:00+00:00', 'duration_minutes': 30, 'action': 'record', 'state': 'waiting', 'message': 'Created with Interactive web interface.', 'series_recording_id': 0, 'keyword_id': 0, 'rescheduled_by_tca': False}
Could not get event ID
Missing information from tvheadend, can't proceed
No timer changes, local cache was not updated
End timer update
I have not tested appending the repeat flag, however, I suspect that the result will be the same.

Matching the names is probably the best that can be achieved because AFAICR the XMLTV specification does not support an eventID field (something that I am looking into for TVH).  Perhaps that match could be a substring search rather than strict equivalence.

[1)]

1) I was experimenting with the Ice TV Kodi addon over the weekend and I found that regardless of the host name that I provided during the LibreELEC portion of the setup process, the device was always named 'IceBox' under 'My Recorders' on the ICE web site.

After a quick search, I found that in 'icetv.py' in the 'login_via_setup_wizard()' function, the device label seems to be hard-coded rather than using the Linux host name (uname -n).

Code Select Expand

    if (region > 0):
        data = {'member': {'email_address': email_address,
                           'password': password,
                           'region_id': region},
                'device': {'uid': mac_address,
                           'label': 'IceBox',   <====== Hard-coded
                           'type_id': 40}}
    else:
        data = {'member': {'email_address': email_address,
                           'password': password},
                'device': {'uid': mac_address,
                           'label': 'IceBox',   <====== Hard-coded
                           'type_id': 40}}
Perhaps this is not the right area of the code, however, there are a number of Python calls that can get the hostname.


2) I also noticed that the addon is hard-coded to use 'service.tvheadend42'.  LibreELEC are currently working on 'tvheadend43'.  It might be handy to search for the loaded Kodi services and match any version of TVH.


3) Are there any plans to release this addon (or a variation thereof) as a generic Kodi-installable addon?

I would like to investigate paying via direct credit.  Are the following instructions still valid?

https://support.icetv.com.au/hc/en-us/articles/227082547-Can-I-renew-my-subscription-by-direct-deposit-

Has there been any thought about incorporating IceTv account management into the IceBox?  Things like creating/editing/deleting searches and favourites and getting recommendations, etc.

The last time I checked, timers created on the IceBox were not reflected on the Ice web site.  Is this still the case?  Synchronising timers would be good to see.

Instead of a weekly email, perhaps Kodi could deliver a 'What's new on ICE this week' screen that would have links to EPG and the ability to set timers.

With LibreELEC/IceBox, tvheadend is not provided as a stand-alone package, but as a Kodi addon that launches the tvheadend executable on startup.  This Kodi Addon reports its version number as being 4.2.8-36.

The WebUI reflects this as well stating: Build: 4.2.8-36 ~ LibreELEC Tvh-addon v11.0.0.0 (2023-04-26T23:18:34+0000)

On the TVH GitHub repository, however, the latest published version seems to be 4.2.7.  Furthermore, the last release of TVH seems to have been over 5 years ago.

It would seem that perhaps LibreELEC build their own TVH for inclusion with the Kodi distribution that they bundle.  The higher version number plus the suffix plus the build date suggest that LibreELEC may also have done some patching and/or improving somewhere along the way.

My question for IceTV is:  Do you rebuild LibreELEC from source or do you take a pre-built LibreELEC image and rebrand it?

If the IceBox 'firmware' is built from source every time, then the possibility of modifying TVH in a meaningful timeframe is much greater.

I was wondering if IceTV could issue us with a bit of a roadmap.

• What bugs/issues are being tracked and what are the current statuses?
• What new features are being considered/under development?
• When is the next release of the Kodi addon expected and what updates will it contain?

Stuff like that.

[adjusted]

I have noticed that the parental rating does not get into the TVH EPG database.  This is because TVH does not understand the 'ACMA' codes.

Code Select Expand

<rating system="ACMA">
<value>PG</value>
</rating>
In my investigation of the TVH source code, I discovered that it will also accept a raw number if it is provided and that number will be used as the parental rating value.

Code Select Expand

<rating system="ACMA">
<value>11</value>
</rating>
Normally, the EIT rating provided is the recommended age minus 3.  TVH, on the other hand, expects the actual recommended age to be provided via XMLTV.

Based on experiments long ago on a Topfield PVR feeding in EIT data, here are the codes and adjusted age values that it uses:

6 = "P"
7/8 = "C"
9/10 = "G"
11/12 = "PG"
13/14 = "M"
15/16 = "MA"
17 = "AV"
18 = "R"

The rules may have changes since then.

I wanted to investigate if I could have an alternate/backup EPG source in case my internet access was ever interrupted for an extended period.  The ICE EPG is great, however, if your internet goes down, then it is not updated.

I'm looking into this because many, many years ago, I came home after several weeks away to find that although my Topfield PVR had completed all of my timer-based recordings expected, they all lacked EPG information.  This was because during my absence, my ISP connection hardware had gone belly-up and my Topfield PVR could no longer talk to the ICE server.

My eventual goal is to set keyword searches in TVH that only use the FTA EPG and create a low-priority scheduled recording.  These recordings will be given a short-ish expiry date.  All of my 'normal' recordings will still be managed by Kodi/ICE.

Caveats:

• Doing this may void your warranty.
• This is only my recollection.
• I probably should have kept better notes.
• I may have forgotten a step or two.

This is what I have discovered so far.

• Created a new 'Network' called 'DVB-T Free-To-Air'.  This is so that 2 sets of muxes (TV frequencies to tune to) can be created, but with different EPG rules.
• Go into 'TV Adapters' and ensure that the new network is included in the 'Networks' list for all adapters.  This is so the new network can attach to a tuner when required.
• Created a new duplicate mux for every original ICE mux delivering services, but using the new network, not the default ICE network.  Tvheadend also has lists of pre-defined muxes for many locations.
• The new FTA muxes need 'EPG Scan' changed to 'Only EIT'.  This is so that they read the EPG from the FTA feed.
• The existing muxes created by ICE need 'EPG Scan' changed to 'Only Open TV Sky Ausat'.  This is to prevent them from using the FTA EPG, however, it does not seem to stop them from received the XMLTV EPG data that ICE sends.  Failure to do this will result in a mixture of ICE and FTA EPG data for channels on this mux.
• Once the new FTA muxes have been created, they will automatically scan for new services.
• Create a new 'Channel' for each of the new services found on the new muxes making sure to give them a name (I prepended 'FTA-') and/or LCN (I added 1,000) that will clearly identify them as having a separate EPG source.  Ensure that 'Automatically map EPG source' is enabled and that 'EPG source' is empty.  The 'EPG source' for the existing ICE channels should always have the XMLTV grabber set.
• Make sure that the 'Over-the-air EIT: DVB Grabber' is enabled in Configuration -> Channel / EPG.
• Ensure that the OTA Cron multi-line contains a line containing '*/15 * * * *'.  This will run an FTA EPG search every 15 minutes if a tuner is available.

If nothing else, this process may be handy for getting EPG information for 'radio' channels not provided by ICE.

I've done some testing over the previous months regarding getting the IceBox to wake using the RCU.  So far, I have had no success.  However, here is what I tried and what I found.  Perhaps someone else may have some other ideas.

Lets start with a positive - Timed Waking

The IceBox supports Linux rtcwake capabilities.  This means that the unit can be woken from a powered-off state in order to perform recordings.  I had to write directly to the RTC device in order for this to happen, but at least it is possible.  Perhaps Kodi can already utilise this feature, more investigation is required.

And now for the rest - Sleeping / waking, etc.

I have tested the IceTV remote with its USB dongle in a Windows 10 PC and a PC running Ubuntu.  On both systems, the ICE RCU was able to generate arrow keys, etc, and the power key was able to put those systems to sleep/hibernate.

Unfortunately, the IceTV RCU was unable to wake either system.  Both systems would wake on USB keyboard input and I even tried plugging the ICE dongle into the exact same port that a known good waking USB keyboard was in plugged into, however, the systems could not be woken with the ICE RCU.

Given that the IceBox can not be woken with the ICE RCU, perhaps a real USB keyboard can wake it.  Unfortunately not.  I have been unable to make a USB keyboard wake the IceBox.

Perhaps the IceBox can't be woken because it can't get to sleep in the first place.

Most modern computers can 'sleep' where memory is kept active but the rest of the computer is shut down and 'hibernate' where the memory is saved to disk before the unit is powered off.

I could get neither of these modes to work.

During my tests, I discovered that LibreELEC (the OS that the IceBox is based on) has 'swapping' disabled by default.  This feature is required for systems to be able to hibernate.

I followed the instruction and got a swapfile working, however, I was still unable to get the unit to hibernate.  Kodi would sort of freeze in a dimmed state and a full power reset was required to return the unit to operation.

I have noticed that a significant number of programmes are missing genre descriptions that should have them.



I checked on the IceTV web site, that the EPG clearly shows genre descriptions.



This is also reflected in the raw ICE EPG JSON feed.

Code Select Expand

        }, {
            "id": "176785103",
            "series_id": "9892",
            "episode_id": "216252",
            "channel_id": "38",
            "date": "2016",
            "season": "5",
            "episode-num": "4",
            "start": "2023-07-10T08:57:00+00:00",
            "stop": "2023-07-10T09:05:00+00:00",
            "title": "Shaun The Sheep",
            "subtitle": "Baa-d Hair Day",
            "desc": "The flock don't recognise Shaun when he loses one his most distinctive features. His old pal Bitzer helps him track down the lost thatch, but not before Shaun becomes very attached to one of the chickens",
            "icon": {
                "src": "http://images.icetv.com.au/ee37-9443-b346-38c3.jpg",
                "width": "831",
                "height": "467"
            },
            "category": [{
                    "name": "Children",
                    "eit": "0x50"
                }, {
                    "name": "Animation",
                    "eit": "0x55"
                }
            ],
            "language": "English",
            "country": "United Kingdom",
            "video": {
                "aspect": "16:9",
                "colour": "YES",
                "quality": "SDTV"
            },
            "previously-shown": {
                "start": "2016-01-13",
                "channel_id": ""
            },
            "subtitles": {
                "onscreen": "English"
            },
            "part_of_series": "Yes",
            "rating": "G",
            "external_ids": {
                "tvdb_series": "79890",
                "tvdb_episode": "5544592",
                "imdb_series": "tt0983983",
                "imdb_episode": "tt6226402"
            }
        }, {
I did some experimentation with XMLTV importing and I found that "Children's / Youth programs" was accepted, but that "Children" was not excepted.

Having a quick look at the tvheadend source code, it would seem that it is doing some text matching between the received genre description and the ETSI genre descriptions.

https://github.com/tvheadend/tvheadend/blob/14298acb6a8e3a83ed1091fab1f3a924077ddfea/src/epg.c#L1820C26-L1820C26

More investigation is obviously required, however, it would appear that IceTV needs to pass the ETSI description, or at least TVH's version of the ETSI description, for the genre description to be accepted.

The JSON feed already has the ETSI code, perhaps the ICE Kodi addon needs a lookup table to find a value that will work with TVH.

[Playing Recordings]

I have a number of Topfield PVRs and I have been experimenting with interoperability between Topfield and IceBox.  Here is a summary of my findings:

Playing Recordings

If you copy your Topfield recordings onto your Kodi/IceBox, they can be played without issue, however, the Topfield metadata in not available, even if you copy the INF file.

If you want to copy any Topfield recordings, you can do that via a USB flash drive, etc.

If, however, your Topfield PVR and IceBox are on the same network, you can go into the Kodi file manager and add a network source using FTP as the protocol.  This will enable you to copy the recordings directly from the Topfield to the IceBox.  I can add a step-by-step guide if anyone is interested.

Getting Metadata

To get metadata working, I firstly needed to copy the Topfield recordings into a Kodi 'Movie' library.  I wrote a script that will read the metadata from the Topfield INF file and create a Kodi 'NFO' file.  The Kodi movie library then needs to be rescanned and the metadata will appear.

The script is a once-off that needs to be run from the Linux command line on the IceBox to import the metadata.  If anyone is interested, I can make this available.

Playing recordings still on the Topfield PVR.

There are a set of TAPs for the Topfield called TMSServer and TMSClient.  These two TAPs work together to allow one Topfield PVR to play recordings residing on another Topfield PVR.  These TAPs use NFS as their data transport, but advertise via a proprietary protocol.

There is another TAP called WebControl that provides a JSON interface to the Topfield PVR and allows for control via a web browser and other systems that can read and process JSON data.

I have written an addon for Kodi that uses a combination of TMSServer advertisement broadcasts to learn what Topfield PVRs are available and WebControl to extract PVR-specific information.  I have called this addon 'TMSClient' and it is able to list the recordings on a Topfield PVR, with metadata, and play those recordings.  There is no copying involved, playback is achieved directly off the Topfield's HDD over the network.  The addon can also list the upcoming timers on the Topfield PVR read-only.

Timers

I have also written a proof-of-concept script that can copy all of the timers from a Topfield PVR onto the IceBox.  For now, every timer is copied as a once-off timer regardless of its frequency setting.  The script uses WebControl to extract configuration data from the Topfield so that it can match the DVB triplets of each channel to create the timers for the correct channels.

I have been investigating how tvheadend (TVH) stores its EPG data and from what I have found, TVH holds the EPG data in memory and only writes it to disk during a clean exit.  I need to do some more testing, however, I suspect that this means that any new EPG data received between system startup and a power failure will be lost.

For FTA EIT, this would not represent a huge problem because the EPG grabber task will run on startup and the EPG will be replenished.

For ICE, however, I believe that a power failure will result in a hole in the EPG data.  This hole will span all of the EPG data received between the last clean shutdown and the restart after the power failure.

As the ICE addon receives new EPG data, that data is processed, sent to TVH and then ICE saves its 'last update' time (I have confirmed that this is actually written to disk every minute).  For the next EPG cycle, ICE requests EPG changes since the 'last update', applies only those changes and the process repeats.

However, if a power failure has occurred, TVH will not have written any newly received EPG data to its database file.  ICE will not be aware of this and continue to request EPG data since its 'last update' received, not the last update retained by TVH.

I'd like to propose 2 possible resolutions:

1) ICE sets a flag in its local settings file on startup and then resets that flag when shutdown cleanly.  If the ICE addon starts and finds the flag set, there is a good chance that there was a power failure and it reloads the entire EPG into TVH.

2) ICE keeps track of the signature (eventid/channel/start/stop) of the last EPG entry successfully sent to TVH.  ICE also keeps track of the 'last update' value that the previous session started with.  On startup, ICE looks for the EPG entry signature and if it is not found, reloads the EPG from the 'last update' of the previous clean session.

Option 1 is probably easier to implement, however, it has a larger network impact for frequent power failures.  Option 2 has slightly more moving parts, but only re-requests the potentially missing data, not the whole EPG.

The one last thing that I have to investigate is whether the Kodi DVR function caches EPG information to disk, or if it obtains a fresh EPG data set via HTSP on startup.

If I have misinterpreted anything, I'm happy to be corrected.