Friday, June 14, 2013

Do we really need S-OFF?

Lately there has been a lot of confusion about if we - HTC users - really need S-OFF on our devices. I think it's time to make this case as clear as possible, and clear up any remaining doubts.

First of all, S-OFF stands for "Security OFF" and S-ON for "Security ON". It's a term specific to HTC devices (and refers to digital signature checking on the bootloader "hboot"). Retail devices always come with SHIP S-ON locked bootloaders. Hboot can also be found in an engineering version (ENG as opposed to SHIP), but it's not easy to get such a device.

You should also know the "fastboot" term: it is a diagnostic protocol used primarily to modify the flash filesystem via a USB connection from host computer. After enabling the protocol on the device itself (entering "fastboot" mode from inside the bootloader), it will accept a specific set of commands sent to it via USB using a command line, for example "fastboot flash boot boot.img" or "fastboot erase cache".


What's the main difference between S-ON and S-OFF from the end-user point of view?


With S-OFF you can:

  • Flash in fastboot original parts of the firmware like: Trust Zone (tz.img), Resource Power Manager (rpm.img), Advanced Digital Signal Processor (adsp.img), bootloader (hboot.img), Radio Config Data (rcdata.img), Splash Screen and others, very often device specific firmware like Consumer IR (cir.img) for the television remote controller in HTC One.
  • Flash in fastboot custom parts of the firmware above, however I've never seen in my life anyone compiling custom rpm.img or tz.img. I've seen custom bootloaders and Splash Screens only. You can also flash modified radio.img but there is rarely anyone out there who does this.
  • Use more advanced fastboot commands, for example you can change the CID (Carrier ID) of your device or even MID (Model ID). And this one is the most important one in the context of this article.
  • Reset the Tampered flag, so your device does not show up as "Relocked" if you relock your bootloader.

For about 2 years you have been able to unlock bootloaders of selected HTC devices on the htcdev.com webpage. Unlocking your bootloader results in an "UNLOCKED" message in the bootloader screen, and allows you to use some of the fastboot commands. For example system, boot and recovery partitions are no longer locked and you can flash a custom boot or recovery onto your device. This doesn't mean S-OFF, but it does give you some more control over your device.

Sometimes there are differences specific to the SoC ("System on a Chip") of each device. Both HTC One X and One X+ (nVidia Tegra 3) have locked out the capability to flash the boot partition from inside recovery, even if your bootloader is unlocked. It is possible to flash the boot partition only via the "fastboot flash boot <boot image name>" command. On the newest HTC smartphone - HTC One (Qualcomm Snapdragon S600) you can use either fastboot or adb shell (dd if=/... of=/...) to write the boot partition.

Do we really need S-OFF?

No, we don't. So what do we need? Because we surely need something. But to understand what we need, it's important to realize where the problem is first.

First of all, comparing HTC devices with Nexus devices is a pointless activity. Never do that. Why? Because they are all S-ON (they call it Secure Boot), and updates for Nexus devices contain the following (based on my experience with Samsung Galaxy Nexus):
  1. bootloader.img
  2. recovery.img
  3. GSM radio
  4. CDMA radio (in case of CDMA device)
That's all. On Nexus device you can flash the original bootloader or radio using the "package_extract_file" command in the updater-script. When HTC releases a major update, however, you will get:
  1. adsp.img
  2. cir.img
  3. dzdata_16g.hdr
  4. dzdata_16g.img
  5. dzdata_32g.hdr
  6. dzdata_32g.img
  7. dzdata_64g.hdr
  8. dzdata_64g.img
  9. bootloader.img
  10. radio.img
  11. recovery.img
  12. rpm.img
  13. sbl1-1.img
  14. sbl1-2.img
  15. sbl1-3.img
  16. sbl2.img
  17. sbl3.img
  18. tp.img
  19. tz.img
  20. more...
See the difference? This firmware images (if updated) are stored inside firmware.zip inside the OTA update. And without S-OFF you can manually update (using fastboot commands or command shell) only recovery, boot, system and sometimes radio. Other partitions are locked and you can't update firmware images other way then only with signed firmware.zip.

Content of HTC OTA update

However, very often, flashing only the content of the system and boot partitions is not enough to have the device fully working. For example, in the HTC One X it was necessary to use the new bootloader together with the official HTC Jelly Bean update, otherwise your device wouldn't boot with an older bootloader. This is why flashing a custom ROM for an HTC device is nowhere the same as flashing a custom ROM on a Nexus device. Apart from having the latest system files, you need to have the latest firmware.zip package flashed as well.

Because HTC sells their devices to different carriers around the world, they need to accept some requirements. For example carrier branding. Because of carrier branding, HTC has more than one version of the RUU (ROM Update Utility) for each device. To indicate the difference between the branded and un-branded versions of the same device, HTC used so called "CID" numbers. 

To find out your current CID number (together with some other useful info) you can use the "fastboot getvar all" command. Also, keep in mind that every OTA update checks CID/MID numbers before it will start to patch your system:




                         ifelse( is_ship_bootloader(getprop("ro.bootloader")) == "t" ,
                         assert(check_cid(getprop("ro.cid"), "00000000" , "11111111" ,
                         "22222222" , "33333333" , "44444444" , "55555555" , "66666666" ,
                         "77777777" , "88888888" , "99999999" , "HTC__001" , "HTC__E11" ,
                         "HTC__102" , "HTC__203" , "HTC__405" , "HTC__Y13" , "HTC__304" ,
                         "HTC__032" , "HTC__A07" , "HTC__J15" , "HTC__016") == "t"););
                         ifelse( is_ship_bootloader(getprop("ro.bootloader")) == "t" ,
                         assert(check_mid("full", "PN0710000") == "t");,
                         assert(check_mid("simple", "PN0710000") == "t"););

Obviously "check_cid" includes also SuperCIDs (00000000, 11111111, ...).
Content of android-info.txt
It's all in updater-script, so it can be easily edited anyway. But the real problem is different. As mentioned already, every OTA update contains firmware.zip - package with bootloader, radio, touch panel drivers, trust zone and other parts of important firmware. It also contains the "android-info.txt" file, where CIDs/MIDs are listed, so your S-ON bootloader won't let you flash an original firmware.zip if your CID number is not listed there. Yes, I'm not talking here about custom radio, bootloader or anything custom at all. Original, untouched firmware.zip from an OTA update can't be flashed onto the device if the CID number doesn't match. Is it a problem? Yes, this is the real problem we're dealing here with. Not S-ON/S-OFF, but CID restrictions and an inability to change the CID number.

How this can be resolved? "android-info.txt" is a plain text file, so it can be edited easily. If your CID number is not on the list, just add one more line with your CID. However, as long as your device is S-ON, you won't be able to flash it, because every firmware.zip is signed with a special key. Once firmware.zip is modified, the signature is broken and the bootloader will reject the request to update it. But there is a different method: you can change the CID number on your device with a fastboot command "fastboot oem writecid <cid number>". The best CID number to use is one of the WWE CIDs (for instance HTC__001). But wait - you can't use this particular fastboot command without S-OFF.

Is this problem a real one, or just some sort of users ill-informed craving? It's very real, because without the ability to flash firmware.zip from a WWE OTA update, every user from any carrier or different world region is forced to wait months to receive OTA updates customized to his CID. Everyone can de-brand his device easily by flashing a stock system image, but it won't be enough: because firmware.zip with corresponding parts of the firmware is needed at the same time. This isn't about the OTA itself, it's about the firmware.zip inside that OTA update.

Dangers:

So what are the dangers of obtaining S-OFF on your device? Some of the partitions in the device are extremely sensitive and can result in your device being bricked if they are even slightly corrupted. With S-OFF you can access all of these partitions and the slightest corruption during transfer (whether that be a power spike or you jiggled the cable slightly) can result in a bricked device as it does not check for signatures.

Here’s an example which has almost happened to me once on an S-OFF device: I was flashing a boot.img via fastboot, the command is: "fastboot flash boot boot.img". However I had made a small but significant typo: "fastboot flash hboot boot.img", simply by mis-hitting the B key; this command would be rejected by a device with S-ON as it is a protected partition, but would be accepted on a device with S-OFF. If I had pressed enter without checking the command, my device would have turned into a paperweight in seconds.
One of the most popular protected partitions the hacking community enjoy flashing is the radio partition. This is also a partition where the slightest corruption will cause your phone to brick. The FCC guidelines state the the radio must be booted with a separate processor (I guess to decrease the risk of it being tampered with), so what happens in a phone when it turns on is: radio is booted via a dedicated processor by the first stage loader, initialising the radio hardware (Wifi, Data, Bluetooth, etc.). Radio successfully boots and initiates the first stage loader to use the main CPU to load the second stage loader into RAM (also known as the SPL). Depending on the boot operation, it will either initiate the system or recovery. So without a functioning radio, the main CPU will not kick on and boot the phone.

Some other facts:
  1. You don't need S-OFF to root your device.
  2. You don't need S-OFF to be able to run Titanium Backup or other applications that requires root access. You just need root privileges for that.
  3. You don't need S-OFF to flash custom recovery image onto your device.
To summarize:

We don't need S-OFF, but we do need the ability to edit the CID number on the device (let's say at least on officially UNLOCKED devices), or the firmware.zip packages inside an OTA update should not be signed, so that "android-info.txt" can be easily edited, or the CID restrictions from android-info.txt should be removed (MID is enough to ensure that the right firmware gets to the right devices).

Something to re-think?

Even if we don't need S-OFF I'm quite worried about the policies of mobile companies and carriers. Their philosophy is "the more you are locked down, the more you are protected". That means Police should not fight with criminals, but everyone should just lock down their doors, windows and stay at home instead. It's far easier and cheaper to lock down mobile devices and not allow root access rather then improving the security in other areas.

Can you imagine that you just bought a brand new notebook for $3000 and:
  • you can login only as a Guest (no Administrator account available by default),
  • you can't change your operating system,
  • you can't use applications that requires Administrator privileges,
  • you can't browse freely the content of your hard drive.
You would say "Where the hell is my freedom?!" Here comes the answer from your notebook manufacturer - "For your own security, you don't have any freedom". Sounds like a George Orwell story to me.

I want the same freedom on my phone that I have on my PC.


This article was written in a cooperation with Shen Ye


Have any questions or comments? Feel free to share! Also, if you like this article, please use media sharing buttons (Twitter, G+, Facebook) down this post!


PS. I want thank to Tom Kelsall, my HTC Elevate companion for his help in a proper grammar redaction of the review! Thanks Tom!

47 comments:

  1. Nice article. Completely agree with you on that mate.

    ReplyDelete
  2. How is it possible that you don't even mention the fact that you still need to register at HTCDev for unlocking. I know your views about that in relation to warranty, but the truth is, I rather be on the safe side and don't want HTC to know I unlocked. That's the main reason for me for S-Off, it's the only way to root without unlocking at HTCDev.

    ReplyDelete
    Replies
    1. I mentioned this here "For about 2 years you have been able to unlock bootloaders of selected HTC devices on the htcdev.com webpage."

      But this article is not about unlocking the bootloader so I didn't focus much on the unlocking bootloader method. More information you can find here - http://android-revolution-hd.blogspot.com/2013/03/negative-effects-of-unlocked-bootloader.html

      Delete
    2. The most important part I mentioned is the fact that you have to register your device. That and S-Off are the only methods to be able to flash roms, so that's why it is so important. And while HTC shouldn't reject your device if something breaks that has nothing to do with unlocking, it is possible that they do say that. And then you arrive at the point that your phone should be repaired under warranty but they deny because of aforementioned reason, which could lead to lengthy discussions that are very hard to win.

      Delete
    3. In my experience, HTC has never rejected any repairs that were definitely not caused by unlocking. They've even fixed a phone I had which was completely bricked because of a bad flash.

      Delete
    4. In my country HTC rejected every warranty if phone is "relocked". I'm talking about One X.

      Delete
    5. Which country are you in? Is there information online that backs up your claim that EVERY warranty claim was rejected?

      Delete
  3. How you say S-Off is necessary first time for change brand cid whit no brand cid for me is the only reason for love s-off... Regards dfox

    ReplyDelete
    Replies
    1. Yes, right now we need S-OFF to change CID, but the meaning of this article is to show that we wouldn't need S-OFF if we were able to change CID number.

      Delete
  4. "Here’s an example which has almost happened to me once on an S-Off Eng device: I was flashing a boot.img via fastboot, the command is: "fastboot flash boot boot.img". However I had made a small but significant typo: "fastboot flash hboot boot.img", simply by mis-hitting the B key; this command would be rejected by a device with S-On as it is a protected partition, but would be accepted on a device with S-Off."

    you meant mis-hit H key?

    ReplyDelete
    Replies
    1. Nope .. he means he mis-hit the B key ... the mis-hit was that he hit the H as well ... If he never intended to hit the H key, then it wasn't a mis-hit H it was an eroneous hit on the H.

      Delete
  5. First of all, very nice article Mike. I followed you since ARHD 2.0 and is nice to see how the things turned out, with you being beside an respected developer on Xda an Android profesional, explaining on everyone's language this matter.
    Now back on track, the main issue, beside some risk that we are all aware or some features that we may or we may have not need is this policy that we see today so much often in every aspect of our life of restraining rights ,all in our 'best interest' and for our "safety".
    Practically is an intrusion in our life, other dictating what is best for us, not letting us to decide...... If we need or not.... S-OFF.:-) :-(

    ReplyDelete
  6. the key is the cid as being a hox+ owner for at&t and since I do not have lte in my area and would rather use an international Rom than use a north american rom. I feel that the phone would be worth more with the ability to change cid.

    ReplyDelete
  7. Nice article Mike, cheers for clearing that up. Hopefully more people will understand that they don't need S-OFF

    ReplyDelete
    Replies
    1. Well, but the problem is that one of the best methods to achieve SuperCID is doing S-OFF. So for now we sort of have no choice. I hope something will change in the near future.

      Delete
  8. Nice one. To me the most important part is the "freedom" part. I have absolutely no knowledge about this and I do trust DEV but knowing that I am locked in (or out) somewhere, even when there is no negative impact for me, drives me crazy.

    ReplyDelete
  9. Bro warranty is a big issue, One simply cannot loose warranty of a 840$ device just like that. S-Off is the only way we can root our devices n still not loose warranty.
    I had a very bad experience with HTC Sensation, had it not been for S-Off it would had been a very expensive paper weight.
    I know you say unlocking boot loader via htcdev.com wouldn't knock off the warranty, but that's ur view, HTC simply doesn't agree on that

    ReplyDelete
    Replies
    1. Do you really think you will loose warranty with "UNLOCKED" bootloader but you won't loose it after doing S-OFF? In my personal opinion unlocking bootloader or changing S-ON > S-OFF should not void the warranty, but doing S-OFF is something way more "invasive" then unlocking the bootloader via htcdev.com website.

      Delete
    2. Like in the days of the HTC Desire, just make the device S-On again and nobody will see that it was S-Off in the first place. In stark contrast to unlocking at HTCDev. I've been trough enough trouble with my HTC One X, had to send it back multiple times in the first few months, I didn't dare to unlock that thing.

      Delete
    3. "Do you really think you will loose warranty with "UNLOCKED" bootloader but you won't loose it after doing S-OFF? In my personal opinion unlocking bootloader or changing S-ON > S-OFF should not void the warranty, but doing S-OFF is something way more "invasive" then unlocking the bootloader via htcdev.com website."

      Pretty sure your device will lose warranty *if* HTC knows you have S-OFF...
      But as far as you can S-ON again, and remove the tamper flag, who can tell your phone was ever unlocked? (please correct me if I'm wrong) :)

      Delete
  10. so is there anything we can do to get the "freedom"? As for now, I would say the best way for me is to get rid of the HTC and never buy another one ever.
    Since I bought my HOX all I have is the frustration and no fun what so ever. I have a branded 3UK device so to get a decent ROM it is pain in .ss so I gave up long ago and I stick with the one I have flashed when I get it.
    I guess is a time to sell this rubbish and get some proper decent device, but definitely not from HTC

    ReplyDelete
    Replies
    1. This happens on every device, not just HTC. In fact, there is no Android device with root access out of the box (at least I'm not aware of any). This problem is a problem of Samsung, Sony, HTC and others. And in most cases carriers are to blame for that situation.

      Delete
    2. I have an Ainol Novo Spark and it comes rooted :P

      Delete
  11. First of all, let me congratulate you and thank you for your work with roms and for your articles. People pay attention and respect what you write, even if they don't agree. I have come to realise that you are the right source of information about these things, and I want to ask you something to see if I understand your explanation correctly (I think I do but I want to be sure). But please let me tell you my story first so that you understand the question better..

    I bought the international version of the HTC OneX last year when it was new and hot, I got it factory unlocked and completely unbranded. I used it until the last few months of 2012 with no problem and completely original, without even knowing anything about bootloaders or recoveries or roms. Then I accidentally dropped it, but it had a soft gel case and nothing happened.. nothing happened except for a wifi conectivity problem that started to appear. In november I received the OTA update from ICS to Jelly Bean. At the beginning of this year I sent it for repairs on warranty claim to the official HTC repair center for latinamerica (later I found out that they apparently outsource the repair center so it's another company that does the service). When I got it back it had ICS again and I discovered that I couldn't update it to Jelly Bean again. I asked them the reason for that and they told me that they had changed the 'motherboard' or the SoC (I don't know which is the correct term). So I started digging about these things, and found out that it had a 'ghost' CID (a very uncommon and weird BSTAR-301 CID). I decided to sell the device as it was (with ICS). When I was about to sell it (I had even found a buyer who didn't have the money in that moment), my boyfriend discovered a way to update it: apparently a carrier (personal argentina) that has the same 'ghost' CID was rolling out an OTA update. So I flashed an ICS ROM of that carrier, flashed the stock recovery and -just in case- I relocked my bootloader. I received the OTA and updated with no problem, and then I started flashing different roms until I found yours and decided to stay with it. I really don't fancy the new Sense 5 that much, and I think that the jellybean 4.2 features are not really a big deal for my needing or liking. But if in the future -for any reason- I decide I want to upgrade to jb 4.2 with sense 5, according to your explanation I cannot do it with the OTA (flashing the stock odexed rom you uploaded) because of my CID and having S-ON, right?.. So my question is this: won't I be able to do it in any way? not even manually flashing a custom rom based on the new update when it comes out?

    Thank you very much for your time and attention, and thank you in advance for your answer.

    ReplyDelete
  12. Hi,
    on hTC Tegra devices the Radio (i.e. 2G/3G) is not booted first. The baseband processor is flashless and depends on the NV memory of the AP.

    ReplyDelete
  13. A very well written article indeed!

    ReplyDelete
  14. Hey Mike, do you think there will ever be a way to get t-mobile hspa+ working on a dev edition htc one? since we have s-off and supercid now, any idea or speculation on if/when that would be possible?

    ReplyDelete
    Replies
    1. I think it is possible. You would just have to flash .401 UL firmware + .401 UL based ROM. However, I'm not 100% if that won't brick your device. You will have to ask first here - http://forum.xda-developers.com/showthread.php?t=2182823 if anyone already tried it.

      Delete
    2. I have a developer edition...i have flashed the new firmware (.401) and revolution hd 12.0 on it...just came from Detroit and tmobile lte AND hspda+ worked just fine there...28Mbits/second download speeds on one bar in LTE. Home in FL...hspda+ is working just fine on tmobile.

      Delete
    3. thanks mr sunshine. are you in a refarmed area? because i flashed 12.1 but am still on edge in certain areas where t-mobile hasn't been refarmed. thanks!

      Delete
  15. hi mike. ..can u help me??? i'm S-ON n HBOOT 1.18 firm 1.45 modid PG581200 . .. my phone is brick n idont have a back up from my stockrom. . . i've try pyramid RUU but its fail. . . any suggestion for my phone?

    ReplyDelete
  16. If I don't have s off and only an unlocked bootloader (with root), would I be able to flash android 5 roms? I am currently on ics/jb firmware 3.32-3.34. I mean I read the without s off you have to flash Ron's with the same firmware as stock. please clear this out.

    ReplyDelete
  17. can I s-off, do the supercid thing, and the go back s-on? I am really uncomfortable with the consequences of being s-off

    ReplyDelete
  18. hey i am completely new to android and got a htc sensation xe ... can u please guide me in knowing 1.what is S-off ? 2.what is unlocking ? 3.what is rooting and flashing cus.ROM ? sorry i tried learning them for other websites but its too technical which i cannot understand ...

    ReplyDelete
  19. Do relocked hox will get update through OTA??

    ReplyDelete
  20. Is it possible to update rooted desire HD 2.3.5 to 4.2 without s off

    ReplyDelete
  21. Thank you very much for such an informative article. I just rooted my one x white and installed twrp. I thought that s-off was the next step. I will not bother now.

    ReplyDelete
    Replies
    1. Ken, you can't S-OFF HTC One X anyway :)

      Delete
  22. Hi, so today we do need s-off so that we can change cid, mid and get to locked status. I take your point, and I agree its a pity that we have to take the risks to achieve s-off, but it is necessary today. Until HTC change this s-off, is still necessary, that why so many have it.
    Also HTC repair (at least in the UK) will see that your bootloader is/has been unlocked and deny warranty. It was the first thing they tried to do to me, although I eventually convinced them otherwise.
    In my opinion HTC would do well to re-architect its current security functions to allow the functions detailed to be user managed, along with bootloader unlocking. Save s-on for where it is really needed, but if it gets in the way of what users want/need, s-off IS necessary.
    Getting s-off, and having a phone that is s-off undoubtedly causes many problems for HTC repair ulitimately, so I have never understood why the complicated s-off dance with HTC is necessary. Its their design after all.

    ReplyDelete
  23. Class Article. Help me a lot To Clear my Base or Basic Concept..
    But Dear can you please Let me the One thing?
    I have a HTC ONE Sprint, Trying to Update the OTA via TWRP, but getting the msg Sign failed.. Assert Failed : getProp("CId")..00000,11111,22222,etc.
    How can i workout to Work the OTA to be updated Successfully in HTC ONE?
    as you mentioned that yo need to Change the CID, MID. but where can i find tht CID in Phone?

    Please Help.

    ReplyDelete
  24. Did you read the article? Man, i tell ya.....

    ReplyDelete
  25. why would you flash boot img on soff device ?

    ReplyDelete
    Replies
    1. Why you was reading this? :)

      Delete
  26. my htc sensation is stuck at htc logo coz of kernel bad flash .. so how can i repair my phone any advice

    ReplyDelete
  27. So, at least as it pertains to the one (m7), will those of us with s-on have issues upgrading to kitkat? I still have the original bootlaoder and firmware that shipped with my device (1.44, I believe) and have never had an issue flashing various ROMs and radio images. Is this because an issue hasn't arisen yet or is it something that comes up when upgrading to a new iteration of android? I haven't flashed a kitkat rom yet and after reading this, I am worried that having s-on ciuld start to become an issue for me as to this point I haven't had a need to s-off my phone and subsequently update my firmware package.

    ReplyDelete
  28. Hey if I want to flash the stock firmware and I am unlocked but s-on , can't I edit my cid and mid on that txt file ? Will it flash or not ?

    ReplyDelete
  29. Mine HTC Desire X RO.CIDLIST: HTC_*** RO.Cid: 11111111 Carrier: ALL Till now am not getting the Update (4.1.1/4.1.2) So, What can I Do?

    ReplyDelete