Sky Q UPnP, REST and WebSocket API interfaces

It’s been a few years since we got into digging around Sky+HD’s UPnP control protocol, but having recently updated to Sky Q, there are a few changes that break the way we were talking to the box.

Backwards Compatibility

The Sky Q box contains pretty much the same UPnP advertised control system as Sky+HD, and includes a couple of new endpoints (service nodes) to do things that don’t seem too interesting. A “Pairing” service node and a “BasicManagement” service node.

There is one potentially breaking change though. The SkyServe2 service node now provides the service type urn:schemas-nds-com:service:SkyPlay:2 as well as the SkyControl2 service node. If you register a listener, or make any calls against the SkyServe2 node, you won’t get any useful data. This means if you’re using a library like dalhundal/sky-plus-hd, you’ll need to skip over the response from SkyServe2, and just use SkyControl2 like before.

Ultra HD

A couple of notes about UHD broadcasts. They currently broadcast on a channel with the ID 9000 (Hex 2328) (named “Ultra HD”) which isn’t tunable by number, only via EPG prompts when you’re on a station that is simulcast in UHD. Currently, if you query sky’s web services for the channel logo for 9000 you’ll get the Sky Sports 1 logo, so I think it’s safe to assume (for now at least) that if you’re watching UHD on channel 9000, you’re watching whatever is on Sky Sports 1 HD. This is important is you’re writing a logger for what you watch on TV, as there is no listings for 9000 available via tv.sky.com – so you’ll need to pretend you’re watching Hex FA2.

This will also (at least based on test broadcasts) be the case for Sky Sports F1 in Ultra HD. This is currently being broadcast as channel ID 9002 (Hex 232A) (named “Sky Sports F1 UHD”). You can rewrite that channel number to Hex EFB, Sky Sports F1 HD.

For UHD recordings, there’s a new flag on a recorded item simply called ‘uhd’ that will be 1 for UHD content.

The cool new stuff – RESTful and WebSocket interface

The new Sky Q app talks to the Sky Q box solely over a new RESTful interface (with a hint of WebSockets). This is cool, because the XML/SOAP of the UPnP interface was the worst thing ever and I hate SOAP forever.

The RESTful interface lives on port 9006 for the Sky Q box (it still uses UPnP for box IP address discovery). Once it’s in, there are a few endpoints it uses, all using /as/ as a base. There are /status urls that are available for each service, but these are only accessible if you pretend to be a WebSocket.

For example, you can get a full system overview at /as/system/status – but in order to access this URL, you need to send the upgrade headers, so:

curl "http://<sky_box_ip>:9006/as/system/status" -H "Connection: Upgrade" -H "Upgrade: websocket"

You’ll get a bunch of JSON back from that, which includes pretty much all the detail about your box, including a section called entitlements which details your package, and a few other things, such as trials. Apparently I’m a FIELD_TRIAL_1 and a GATEWAYENABLER. (I have no idea what any of those thing are) – There’s some stuff about hdmi which is a bit interesting – it talks about established HDCP connections, and if your display supports 2160p10bit. Also a bunch of things about your viewing card (that you probably don’t ever want to share on the internet)

This acts as a subscription, so any changes to that get pushed to you automatically.

Presumably, if/when they build the REST api out to support controlling the box via the Sky Q app, you’ll also be able to use a WebSocket to get notified of channel changes.

Currently, there are the following non-WS RESTful endpoints used by the Sky Q app:

GET /as/system/information – Returns a JSON array of a bunch of system information.
GET /as/pvr?limit=1000&offset=0 – Return the first 1000 recorded programs on the Sky Q box. This includes pushed on demand content, box sets and recordings.
GET /as/pvr/details/$pvrid – Return full details for an individual recording.

Live Streaming (and recording copies)

All of UPnP and RESTful endpoints expose streaming URLs for live channels and DVR content. They’re served on port 4730. There’s usually a URL for the source quality, and a transcoded version which I assume is used by the iOS and iPad clients when you download programs. These are just .ts muxed video files, but don’t get excited – the tracks are encrypted by whatever the broadcast medium was. For the vast majority of content, this will be Cisco’s (NDS) VideoGuard, the same for any VOD content.

Interestingly, Sky Go recently switched to use a native compiled version of Cisco’s VideoGuard on Mac and Windows because Silverlight was discontinued. This presumably improves Sky’s internal workflow and ingress, as they’re using one super secure encryption everywhere.

Cisco (formally NDS, who Cisco bought) also write all the software for the Sky boxes, speaking of which:

Sky Q EPG

The EPG is interesting. The vast majority of the content displayed is loaded over HTTP. It appears (based on the returned JSON) that Sky have a CMS that they use to curate the UI you see. It’s built around nodes, which load dynamically as you enter each section, with etags and last-modified headers to control caching. The CMS and base loading happens over HTTP from config.ethan.interactive.sky.com.

EPG personalisation is based on the Sky Q box deviceID. Again, unencrypted this is loaded from ethpersonalised.recs.sky.com.

The whole system uses a UUIDs to identify content. Specific media information is then loaded over UDP, which seems weird to me – and obviously makes life much harder to try and understand. I’ve never had to talk to a UDP service before. Anyway, that service runs on oig-e.sky.com.

The majority of thumbnail, images, and (all?) VOD is all powered from an Akamai CDN at cdn.sky.com.

You can manually form the URL to get an image for a DVR recording using the programmeuuid, and osid.

Activity Monitoring

Something that’s a little more concerning is that Sky log virtually every thing you do on the box, by POSTing data to bd01.sky.com, unencrypted.

That data includes some potentially sensitive information, like what you’re viewing, your box serial number and even things like your remote control battery level and serial number. That probably should at least have TLS protection.

Funnily enough, that service appears to often be overloaded, and will return a 503 error after about 5 seconds. The queue for activity monitoring continues though, and it’ll immediately send the next request after that 503. I do wonder if has potential for system instability if the queue gets too big…

Update – 21st March 2017

There are a few more endpoints: /as/services lists all the channels on the EPG, with /as/services/details/SID giving you more detail for a specific channel (but not that interesting). Even less interesting (for now) is /as/apps/status (WebSocket) that just tells you there’s an app called com.bskyb.epgui which is visible.

It looks like data for the EPG is still loaded over satellite – but it does specific program lookups over an EPG service that’s new and lives at awk.epgsky.com – annoyingly it doesn’t return EPG data for the Ultra HD channels, but it’s data includes something called an “eid” which is used to schedule recordings.

You just call /as/pvr/action/bookrecording?eid=<eid> to book it to record it. You’ll be given an internal pvrid that’s unique to your box, and then you can call /as/pvr/action/delete?pvrid=<pvrid> to remove it. You can also use that delete command with the list of PVR recordings you get from /as/pvr/ – This is how the android app works (and hopefully how the iOS app will from the next update)

It looks like none of the UPnP control methods we used on Sky+HD work on Sky Q. You’ll just be told “TRANSPORT IS LOCKED” whenever you try to do change channels, pause or fast forward.

  • Pingback: SkyQ Missing Thumbnails – Liam Gladdy()

  • Peter Williams

    Funny I’ve just found your old posts re sky HD and I’m trying to get it working via Alexa and Smartthingsa. Have you looked into any implementations of this for use with Samsung smartthings or Amazon Alexa?

    • Alexa requires external access to a device, so you’d need to write a web service to sit in-between your Sky box and Alexa.

      Certainly do-able though! This might be a fun one for the weekend 🙂

      • Peter Williams

        I think thats where smartthings and it’s hub comes in, as the intermediary device. I’m controlling my amp with it and some lights etc, loooooads more potential I think. I just dont have developments skills (yet) to really dig into it.

  • Dillan Pattni

    Hi Liam,

    We have tried to accomplish Sky HD/Q control via IP but have bumped in to numerous problems along the way. Is it possible to talk to you about it? We can hire your expertise. Follow me on @dillanpattni on twitter.

  • Steven Pressland

    Very interesting, had a play about and found a /as/services for the channel list. Do you have a full list of everything you have found on the REST service?

    I also have seen the reason why the status services use websockets, when you watch the output from the socket, the status changes when you do different things on the box, for example the HDMI section changes when you switch box in and out of standby, shows DHCP negotiation steps. The PVR status updates when you make any change to the recordings or playback position.

    • Yeh, I probably should have updated this. Where are a few more endpoints. /as/system/information gives you a shorter overview as a REST endpoint. /as/services lists all the channels on the EPG as you said, with /as/services/detail/SID giving you more detail for a specific channel (but not that interesting).

      It looks like data for the EPG is still loaded over satellite – but it does specific program lookups over an EPG service that’s new and lives at awk.epgsky.com – annoyingly it doesn’t return EPG data for the Ultra HD channels, but it’s data includes something called an “eid” which is used to schedule recordings.

      You just call /as/pvr/action/bookrecording?eid= to book it to record it. You’ll be given an internal pvrid that’s unique to your box, and then you can call /as/pvr/action/delete?pvrid= to remove it. You can also use that delete command with the list of PVR recordings you get from /as/pvr/ – This is how the android app works (and hopefully how the iOS app will from the next update)

  • Clive

    Has anyone found a way of getting any usable feedback from a SkyQ box yet?
    UPnP doesn’t return any status feedback at all, so no transport state or channel state…

    • Yeh it does… what are you looking for?

      I’m using it to get the currently playing channel so I can log it. It also returns DVR playing data too.

      The only “flaw” is that if you scan through the mini-guide, it’ll change “now playing” to the last mini-guide viewed channel, rather than what’s on the main screen.

      • Clive

        that’s what I’m after, the currently playing channel, transport state etc.. when I use the same commands I did on the skyhd box I get an xml back but it doesn’t contain any useful info.

        What commands are you using to get the channel info?
        GetMediaInfo?

        • Matt Hall

          Have they taken uPnP off Sky Q? I’m not seeing anything now on UDP other than Airplay, and no description2.xml on port 49153? I’ve got the remote stuff working based on dalhundal’s code – works really well – but channel FB would be great.

  • Clive

    Has anyone got an example url for awk.epgsky.com?
    Trying to figure out how to build the url to retrieve program info

  • Michael Hermanns

    Does Sky HD also have the REST api? If so, on which port is it? 9006 doesn’t seem to work…