ZFS – Great Idea, if you have backups…

So this article is about the Zeta File System (ZFS) a relatively new file system and one that is supposedly very resilient to errors. It has some great features to prevent dataloss and is touted as the be-all and end-all of filesystems. Well my experience is varied, both good and bad, but the reality for me is that if you want to take advantage of some of the features ensure you have backups.. of everything because ultimately your data is not safe.

So a little history, many years ago I decided I needed a storage device to keep all my important stuff, and less important stuff and I was drawn to ZFS with its ‘self healing’ features so often touted, so I built a new server with 16x3T drives and configured it up using RAIDZ2 (the sorta equivalent of RAID6) with 15 disks and a hotspare. I started moving my data from the myriad of external drives to the storage array. I then suffered my first issues…. a drive died.. silently at first.. when I eventually spotted it I immediately issued a replace using the hotspare – which turned out to be not so ‘hot’ as you had to manually switch it in and the array was recovered following a week long resilver process.

Several years went past without incident, big 6kva UPSs kept power outages at bay from causing problems, regular scrubs seemed to work. A minor issue happened on another drive fail where the metadata was reported corrupt and to fix it I had to blow away the directory and it’s contents which were not important as could easily be recovered/rebuilt. This should have been the warning to me that all is not as safe as it should be, but I had bought into the hype and carried on blindly. Then came the undersea cable fire in Malta (well the fire wasn’t undersea, but it was on the end of the undersea cable directly under 4 substations which supplied my area) .. out went the power.. for some 12+ hours, the generator ran out of fuel, the UPSes all ran out of battery and the host died, hot and hard. It was not pretty, and on return of power things were not happy. The reboot was fine for the RAID set, but the ZFS was not happy it wouldn’t mount, infact it wouldn’t do anything especially import it. Calling one the FreeBSD Devs it was suggested I try, “zpool import -FfX storage” and a week of watching the drive activity I got the entire pool back without and visible errors.

I was sold, that was it this was the file system that was unbreakable. I threw caution to the wind and instead of keeping a server that was identical I started using the more useful features of ZFS. I created ZVOLs and built VMs, I created ‘timemachine’ drives for my Macs, and started to reap the short term benefits of ZFS.

Then I moved across the world from Malta to Australia, back home, and shipped the servers through an international removalists. The servers, despite being prepared correctly and packaged properly arrived damaged. Three (yes three) drives were damaged so as RAID6 (and ZRAID2) can only cope with two drives failed I decided to try something ….. I byte copied all three corrupt drives to new drives and put the new drives back into the server/array and ran the import as before. Several weeks of ‘rebuild’ and the array came back online – again without dataloss.. RESULT!

That’s not the end of it though, after all this you’d think I would be raving about it’s resilience. Well each time the issues have happened the critical data structures on disk have not been affected so recovery has been possible. Fast forward to March 2019, a drive died and I replaced it, then whilst resilvering on March 9, 2019 (just before midnight) a transformer blew up down the road, taking out all power. UPSes did not kick in and the generator did no good at all because the power was lost when the power went down. Power was resorted a short time later and the zpool was reporting it could not be imported without rolling back 5 seconds of data… this of course was not an issue as I had been minimising writes from the moment of the rebuild.

I went to bed leaving it to rebuild…

6am March 10, 2019 some idiot drunk or on drugs took out the power pole down the road from me, and the 11KV lines contacting the 240V lines ensured that the UPSes wouldn’t save a thing. Power went down and on return of power the problems of ZFS became very very apparent.

Here’s the issue (in sorta non technical terms)… The file system is a big database, one with lots of redundancy and checks, but it has a fundamental flaw built in. This flaw you can see in a multitude of posts where the devs state with resounding coherence of the party line, “The data on disk is always correct.” Well it is and isn’t… the data is all there, and it reports correct, but if one of the critical structures is corrupt, (eg a spacemap – in my case) the metadata (the stuff the makes sure that your data is right and stays right) is deemed corrupt and so ZFS in its wisdom pronounces the whole drive corrupt.

So lets reiterate that…

A small part of the data in the pool (drive).. just the right (or wrong – depending on your point of view) part got part written because of a power outage and now the entire pool (drive) cannot be mounted. All 36,964,899 files, some 21.2 terrabytes of data.. In fact according to “zpool status” there are just 3 errors in total and examination with “zdb” it appears they are all checksum errors of “metaslab 122” because of the spacemap corruption. So many weeks later I’m still trying to recover the data – I’ve just got myself another 36T of drive space after trying in place recovery, but still no luck.

I don’t have backups as I was moving stuff around and as previously stated had already thrown caution to the wind. The next step for me is to modify the code to ignore the checksum errors and see if I can ‘walk the dataset’ for all the files

I’ll let you know how I get on, but with all the ZFS devs posting that there is no need for a “fsck” in ZFS as the disk is always right I can only suggest anyone thinking of deploying ZFS to only do so if:

1/ You can make full backups, or

2/ You can afford to lose all the data

(and it should be noted, FreeBSD devs are advocating that the root file system should be on ZFS and is now actually the default when installing… good luck laptop owners on road…!)

Edit/Update: Added links to the news articles for the large power issues, changed data from 19th March -> 10th March as my initial post incorrectly detailed the date.

Update [2]: I posted this blog link to the FreeBSD mailing lists here: https://lists.freebsd.org/pipermail/freebsd-stable/2019-April/090988.html and unfortunately many chose to follow the same line of ‘ZFS is always right’ that I see elsewhere (eg: ZFS on Linux mailing lists) .. which is part of the problem. To their credit though a couple of the FreeBSD Devs contacted me onlist with helpful suggestions (you can see these in the links) and others contacted me offlist with really helpful information. One even (off list) pointed me at: Klennet Storage Software ZFS-Recovery which I have not had chance to test yet (being that I need to setup a Windows 7 image on an external USB drive) – but if it delivers what it promises it is the missing link that ZFS needs (in my opinion.)

Dive gear – The Do’s And Don’ts

Some of you will know I’ve been a diver for many years, the more astute of you will know of my love of underwater photography.

So a little about my policy on gear.. I tend to choose a manufacturer after doing a bit of research and stick with it, for everything. Its called brand loyalty…

Photographic equipment, I went with Nikon, and have gear worth in excess of €25,000, underwater housings, Sea and Sea worth a not insignificant amount. Dive gear, Oceanic, even my computer gear, all Apple (and no I’m not a “fan boy”.). I have just found if you stick to a brand everything “just works”.

Well unfortunately it seems I was wrong to trust brand loyalty is not a great thing for some brands as they have no customer loyalty.

This, therefore, is the story of Oceanic. Regulators, BCD, computers (three of them), masks, fins, even wetsuits, all of which I have despite certain items being better with other manufacturers I was sucked in by the “lifetime warranty” initially, and the deal was sealed when their “medium large” size for the wetsuit fit me perfectly.

Oceanic – Australia

Without fail in Australia I took my gear back to Nautilus SCUBA of Brisbane an authorized service center/dealer for Oceanic and all was fine. I then moved from Brisbane to Canberra and found myself visiting Norm Green from Indepth SCUBA who is both a good friend and great dive shop though this is where my problems started. They serviced my regulators one year and some mixup resulted in the Warranty being voided because I had supposedly no serviced the regulators one year… of course this I balked at and persisted in chasing Norman over the issue and after showing receipts and numerous emails from him to Oceanic the Warranty was reinstated due to me keeping to the service records over the years (turns out it was a late submission of paperwork that caused the problem.)

Oceanic – Malta

Then in 2009 I moved to Malta, and searched out a local Oceanic dealer.. world wide warranty? Pfft! From day one they told me there was no world wide warranty and I would have to pay in full for all servicing and parts, so I did, even when I had to stop diving because of a bout of cancer… Every year the regs, computer and BCD was serviced.

8 years later I returned to Australia and went to Dive Jervis Bay to get my gear serviced … especially after getting wet and finding my regs started free flowing. After waiting months for servicing and repair I was informed that the regulators were missing 2 parts, one of which was a critical O-ring and, in the words of Dive Jervis Bay, I was lucky to be alive as the regs could have failed at anytime.

The battery died on my Oceanic OC1 (not the first time), so I took it to Dive Jervis Bay and asked them to replace, test and service it. A couple of weeks and a few hundred dollars later it was returned to me and I booked a dive.

30 seconds into the dive I found the computer going into “calibrate compass” mode and buttons failing, then the dreaded water droplets. Dive aborted, and waited the first dive out, second dive I went with a backup. On return to shore I gave the computer back to the shop and asked them to look at it, they said they sent it back to Oceanic.

Weeks later (6-8 weeks) I was informed the computer was out of warranty and it was a write off as they were an obsolete model and $1000+ would need to be paid for a replacement. I suggested they should reconsider, and several weeks later received the reply that no, that was that, new computer at $1000 or I should go with another manufacturer. In shop I was asked to consider the Suunto range.

Well upshot of all this, after months of asking for the return of my now dead computer it was returned to me, and finally tonight I got around to opening it up. To my astonishment I found the computer very obviously had not even been opened, as it was still full of water, and the reason for the flood was the seal on the battery cover was both damaged and had debris on it.

So the do’s and don’ts …

Don’t trust a world wide warranty particularly by Oceanic, it’s not, and it will be cancelled at the drop of a hat, even if it is not your (the consumers) fault.

Don’t trust authorized service agents (particularly in Europe) to actually safely service your gear, let alone honor service agreements.

Don’t trust the manufacturer or their authorized service agents to care about you respecting brand loyalty (they don’t give a crap, it’s all money to them.)

Do research what you’re buying.

Do research “authorized service centers” to see if they have mandatory training.

Do learn how to service your own gear so you can at least check the work done by the agent.

Don’t assume because you are paying top dollar for gear you’re getting top quality.

Don’t bother with brand loyalty, it used to be worth something, but nowadays its worth nothing, the only thing brands care about are the number of greenbacks you can give up.

Footnote

So as I don’t expect to hear anything from Oceanic or any other Dive gear manufacturer, I’m now ridding myself of Oceanic stuff and going with what ever suits the purpose by which ever manufacturer I feel is not offering the best deal/value for money… Starting with a new air-integrated Computer.

Out with the Old, in with the New…

So as usual my New Year’s post, this one comes with a heavy heart, and hope for the future.  It comes with some reflexions and observations that I hope some will be able to take away and use to protect themselves from hurt, or just learn about happiness, and unlike previous posts has some clearly defined parts… Love, Government etc..  It is a little long, but worth the read (I hope.)

On the subject of of Love..

So this year started badly, I was in Canberra and was told I was going to get a divorce in 2017, well that’s happening now (and 0started in 2017), I should have known then the writing was on the wall, but instead I listened to my heart.  So to my first tips to a happy life, or at least getting out before you get hurt is watch for the signs.

First tip to something is wrong…

When you find, “Tinder” on the home screen of your partners phone, your partner is screwing around, or at least trying to (and no I didn’t go stalking her phone, she opened her phone to read a message whilst sitting next to me.)

Second tip…

When your partner spends all their time texting on their phone, but clicks the home button, clicks the power button to lock it, then puts it in its case and finally places it face down on the table on the opposite side from you (and it’s always the opposite side – ie change sides next time) when you walk into the room.  Another sign that your partner is screwing around.  The psychiatry of this is very simple, someone with nothing to hide may close their phone and put it away, but they will nearly always use the same hand to do it, they will nearly always place it in the same place (their left, their right, their handbag, their back pocket etc), they won’t do it when you walk into the room they’ll do it when they have finished speaking/texting.  If you’re unsure, leave the room, count to 30 and walk back in if they do it again, you know there is something they don’t want you to know about going on, the look on their face will tell you how bad it is.

Third tip…

When your partner dislikes everything you suggest, tells you they are bored all the time, but wants to go out ‘with their friends’ and when they do you note there are photos of their friends minus your partner put up on Facebook… you know the drill.  Of course the excuse will be that something happened, they had to go elsewhere, they got bored, went with a different friend etc…

Forth tip…

Love is a partnership, this means there are two (or more, but generally two) in the relationship.  If you’re the one doing all the work, giving all the love, there is an even better chance that that you are in a bad relationship. To break this down more, if you find yourself doing all the cooking, cleaning, organising of nights out, working, even initiating intimate contact you are in a one sided relationship, and it doesn’t matter how much you love the other person, they do not love you.  The person you are with cares nothing about you, they care about themselves and what they can get out of a relationship.

Fifth tip…

(This is a general one, not just when it comes to ‘love’.)

When you are to blame for everything, and facts are manipulated to make you the blame, the person is looking to absolve themselves of any responsibility rather than take responsibility.  Applying this to love, remember love is a partnership, you are responsible together.  An example in point, if you have an argument and the person you are with gets physically violent and you call the police and they get an arrest record that causes them to lose their job.  They have *NO RIGHT* to blame you for them losing their job.  Similarly, if they told you two weeks before they wanted a divorce, then similarly got violent and you email a immigration and they lose their visa or right to work they *CANNOT* lay blame on the writing of the email (especially where the legal requirement of a partner sponsor is to notify changes in a relationship.)

So the take away, don’t just accuse your partner/friend because something happens like above, however do remember if it feels wrong there is probably something wrong, trust your feelings not your heart.

Government…

USA

Donald Trump” … need I say more?  .. probably not but I will.

For those that are so smitten by his hard line on immigration and how he is making America great again, think on this.  My employer suspended all travel across the USA borders for non-essential trips.  My employer subsequently employed more staff around the world.  Net result:  United Airlines got less revenue from this American company, more staff were employed outside of the USA than within.  Say no more.

Malta

Well most of you I left Malta for good in the latter part of last year, well unfortunately Malta didn’t leave me.  There are things that I cannot talk about here because they will prejudice the court case that is going to start against those involved in the purchase of my house, but suffice it to say if you are a foreigner in Malta you are subject to all sorts of corruption and even when you have done everything you need to, they’ll find something to take more money from you without any apparent risk to themselves.

What I can tell you about Malta is after being there for eight years and making some really good friends, there are also some of the most two face back stabbing hypocrites I have ever met in the world living there, and for the size of the island the the proportion of genuine people vs bad people is highly disproportionate.  Worse, the selfishness and corruption I came across extends to a high percentage of the population and if you’re a foreigner you are a target.  Examples of this, just take a walk through Paceville, go into the stores and look for prices, where there are none, come back later with a friend that is Maltese and ask both of you separately ask how much for the same item.  The foreigner will nearly always see a higher price, you should also note that this practice is completely illegal in Malta.  Same thing happens when it comes to work on your house ask for your bathroom or kitchen to be tiled, you’ll get multiple quotes that turn out to be estimates depending on whether you speak Maltese, whether you are male or female etc.  Now don’t get me wrong, there are some honest builders, but not many.

This selfishness doesn’t just extend to services, it will also extend to personal relationships, many people I met only want to know you for what you can do for them, and don’t expect that your service to them will be in anyway reflected in their services to you.  I for example and a computer programmer, service technician, a network engineer and security engineer (ie I cover most things anyone can ever want or ask for at home.)  I have spent hours, sometimes days working to solve peoples’ computer troubles, and charged them very very little if anything as they are a ‘friend’, however when I needed help from them, I had to wait sometimes weeks, I would be quoted higher prices than *any other* quote and I wouldn’t always get the job I had asked for.  According to my ex, this is a very Italian trait and all Italians are worse than Maltese, and the Maltese get it from the Italians.

Corruption in government and their departments continue, many of you heard about the journalist Daphne Caruana Galizia being assassinated in the latter part of 2017.  Make no mistake, this was an assassination, not murder.  Her death was paid for by persons currently not disclosed, we may never know who paid for her assassination, we certainly will never see justice done.  

Personally I was the subject of corruption, I had my truck and bike stolen under the guise of a legal government transaction.  I had the government, the prime minister himself Dr Joseph Muscat, tell the police department to look into it.  I was told by the then Commissioner of Police, that they had seized my vehicles from the thieves they had inspected them and found that there were a number of immediate anomalies like engine and chassis numbers being altered (badly), false number plates fitted, the documentation said the vehicles had been scrapped (aka garaged), that they were batched up with other vehicles for sale as a job lot instead of being sold off individually…etc…  The then Commissioner of Police was then told to resign/fired just a few months later and that was almost the last I heard of the investigation.  The only further contact was with Detective Jonathan Ferris who refused point blank to consider my case consistently referring only to the illegal modification as the end of the investigation, and ignoring the first part of the case where the ADT Manager of Hall C, on the 20th June 2010 refused to sign the ‘ok to proceed’ customs form without a €1000 bribe.  This resulted in the start of the paperwork that resulted in the theft of my vehicles.  It should be noted that the vehicle that was refused was my English purchased Toyota Hilux (which is exempt import tax or VAT under EU rules) which I had also obtained an import tax exemption in place because I was told I needed to, and the same manager refused to sign the release even with these in hand.  The people of Malta know that they can keep any court case going for many years and that if at any point you miss a case they can permanently dismiss it in their favour, and that it costs in excess of €50.000 to get a case in to the European court so you will lose regardless.

When it came to ARMS this corrupt organisation is legalised mafia flouting EU rules, general decency and stealing from the public.  They over charge electricity ever single day for every single customer, they refuse to refund when you over pay saying that you can have the money off future bills, but they won’t pay interest to you even though they are fast to take it from you (which is actually questionable when it comes to European Credit law.)  However, my little story which I explained way back here the culmination of which was a €76,900 bill which resulted in them cutting power to my premises, stopping my business, and I still refused to pay as the courts had already told ARMS more than 3 years earlier that they had to change my tariff to the commercial tariff.  They hadn’t even changed my tariff from ‘holiday home’ to ‘normal household’ even though I had a certificate of residency. An email to Fredrick Azzopardi the CEO and a meeting was called.  At the meeting, it was not expected by them that there would be my ex, my lawyer and witness and myself.  They planned the same intimidation tactics they tried in court by filling the room full of lawyers and staff, however to his credit Fredrick Azzopardi realised I was not going to backdown.  He realised that my next step would be going back to court, so came to the table and arranged for the power to be reconnected immediately and the bill corrected on the the condition I paid some €23,000 within 3 days.  Power was restored and I paid a revised but still incorrect bill.  They refused to backdate the change to when I had originally requested the change, and they refused to acknowledge my residency to the start of the bill costing me an estimated extra €5,500 and charging me some extra €3,000 in interest.

Lastly, selling my house, Promise of Sale signed, however despite having all the paperwork the notary told my Power of Attorney that if they did not sign an extension both myself and they would be liable for damages by the buyer.  Legal advice has been taken, more on that later after court.

A general piece of advice if you are looking at moving to a new country, unless you have €500,000 to lose to corruption don’t even look at Malta.  It used to be a nice place, now it’s just continual replacement of all architecture, and anything green (eg trees) with concrete and tiles in the name of profits.

Australia

Only been here for 2 months, but I have already seen the true spirit of Australia, and oh how I have missed it.

Almost immediately on arrival I was presented with the opportunity to attend a meeting about the National Broadband Network (NBN) and challenge a number of people about the implementation.  On entering the hall I had the opportunity to speak to the local ABC radio reporter whom chose to interview me about the NBN as it was obvious to her and others that I knew a lot more about the NBN than even some locals.  This then led me to meet Debby and Michelle who are labour party supporters who subsequently introduced me to the local mayor, and various politicians, and Dawn an advocate and campaigner for the Same Sex Marriage bill.

The following day every news report seemed to contain my  interview and statements from reporters about events completely ignoring the politicians.  Seems they like my style of interview, and therefore on the announcement of the Same Sex Postal Survey (not allowed to call it a Vote or a Referendum 🙂 ) they brought the TV cameras and I had my say on ABC TV.  Of course those that missed it, the result was a big resounding ‘Yes!’ and the Same Sex Marriage bill was the subject of debate in Parliament and the Senate over the next few weeks eventually making it into Law with no amendments.

The sad fact of this development for me personally is that my ex-partner, soon to be ex-wife was so impatient, (selfish?) so without personal responsibility, didn’t wait for the result before calling for a divorce for the second time because she wanted to be in Australia, to have her visa.  This time I haven’t argued and won’t, despite us being legally married and despite the rules changing with immigration so she now cannot be refused a visa, despite me being in love with her… she has shown that she does not love me, she has shown that her life is a constant complaint about what she should be entitled to, and how she is so hard done by that I’m done running after her.  She has her bed, she has made it how she wants it, she is determined to lay in it, so I have given up trying to change that.

Personal thoughts for 2018…

So if you got here after reading it all, well done, it’s long and I can only apologise for that, what comes next for me you might think?

This year, I honestly don’t know, this year what I thought was going to happen won’t happen.  I was looking forward to moving into the new house with my now ex.  We were planning a family, to have a child, I was looking forward to this, even enquiring about the IVF that we would need.  I was looking forward to spending less time on the computers and more time relaxing with nature all around.  Well that all went down the proverbial, so now I’ll knuckle down with work, see if I can come up with a new idea/project/company that can generate money (my retirement plan) and I’ll get on with a couple of environmental projects that I have been thinking of over the years.

One thing is for sure, I won’t be leaving this area anytime soon as the people here have been super-friendly and supportive absolutely no discrimination or comments.  Everyone within days introduced themselves, the local club welcomed me and invited me to join their snooker club (I politely declined, though I do play against them in the Friday night pool competition.)  People wave and say, “Hello” all over the local area and greet me with a smile…  After 8 years of people talking behind my back and stares in supermarkets etc I must admit that I’ve been a little uneasy at how nice everyone is.

Regardless of everything, I wish everyone the New Year you deserve, look at yourself and think why did I say that?   It means if you are genuine to yourself and those around you, that you are kind to those around you, and you are selfless in your actions I wish you all the success and happiness in the year to come.  If you are selfish (and only you will really know, so be honest with yourself because I don’t care) then I hope you meet selfish people and you all experience the misery of your selfishness.  If you are greedy (particularly developers) I wish that the housing market collapses and you lose every penny you have so you have to live the misery that the ordinary people and their lives that you have destroyed… Unfortunately it will probably take someone else of greater greed to take that wealth off you.  You should however take note, you cannot take your wealth and properties with you, and people only remember people that are good, you will die like the rest of us, but you vanish into oblivion, some of us will be remembered for the good we do.

Happy New Year, friends, relatives and all the truly good people I have met.

The IoT should really be IoSI (Internet of Security Issues)

The Internet of Things

So here I am seeing issues, reading about issues and trying to stop issues in the Internet of Things…  Everyday someone seems to be publishing articles on the issues, people are getting more aware (you’d think!) but there seems to be no real movement.

Some of my readers will know what I do for my day job, for those that don’t I wrote the SORBS Anti-spam system.. not quite the most hated, but some who should know better have said they just want me dead, then SORBS dead, then me killed again just to be sure I’m actually dead.  Several years ago I spent Christmas sitting in front of my computers rewriting part of the system, particularly that part that finds “bad stuff” and reports it (eg Open-Relay Servers) and whilst scanning hosts that were actively trying to send spam and/or viruses to me I came across the web page of a fridge.  The page half loaded before it became completely unresponsive and tracing it I found it on an IP address that appeared to be in Rome (Italy)….  When I reported my finding of a ‘Fridge Spamming’ to my boss all hell broke loose, blog articles were written, front pages were held and suddenly the world knew about ‘Fridges Spamming‘.  Shortly there after we got debunked by our main competitor of the time who asserted it wasn’t possible, the article however sparked off massive research and watching of the technology from a security stance.

In July of the same year a bunch of researchers at a University found that the premise of the ‘debunking’ was actually false and that with a specific sequence of commands it was possible to get the fridge concerned into a system ‘admin/debug’ mode that allowed a remote attacker to use the device as a simple proxy server and install other “apps”.  This largely went unnoticed in IoT industry with respect to the original report, I never understood why… perhaps someone can explain that to me? 🙂

3 years later…

One would think we have learned something, we certainly have seen more of these types of attacks, not always for spam but just as a device to get into a network, to provide the door way.  Indeed the attackers have pretty much made an art out of it, using combinations of direct hacks, social engineering to gain access or persuade users to install things and even stealing devices…  The lists and lengths seems endless, especially when you consider who is doing this sort of thing and even who is paying who…   We’ve all heard about Trump and Russia and the controversy, well there are teams of hackers in Russia who’s sole income is to break into systems and steal secrets.  Its not a stretch to imagine that they are not unconnected…  Personally I don’t go into the conspiracy theories but I can tell you there are companies and persons of interest that do pay for services of such teams and not just Russian ones, there are European teams, Chinese teams and American etc..

The result is a lot more tech out there, all with security issues and all trying to keep market share, by innovating or by destroying the competition.

So why are we helping these people along?  Why are we allowing companies to circumvent privacy laws?  Why are they even trying?  Why are there more and more companies dealing with security remediation rather than companies dealing with the actual problem…?

All questions for you the reader (and hopefully some people that can effect change.)

So what is this blog post about? Why did you write it?

Well quite simply I chase down security patches for my services…  You see I still manage SORBS and recently we moved some of the servers around to a new Datacenter and as a consequence I changed a lot of security settings to make the systems more secure.  The fall out of this was I completely re-wired my home office network and the only thing on my network now that is not ‘secured’ (ie may have issues) was my wireless network.

Originally I had an OpenVPN connection for every service over the wireless that was an ‘authorised machine’ and a straight session login for controlling access.  I deliberately set the whole network to ‘Open’ (ie unencrypted) to remind people using it that everything can be watched so if it’s important, use HTTPS (or use the OpenVPN) etc.

I decided to switch the network to WPA2-Enterprise for authorised users, and to use a Juniper NAC to provide a captive portal and control the logins etc…  I didn’t account for the ridiculous cost of the licenses of the Juniper NAC so even though I picked up a brand new IC4500 for less than €70 I couldn’t use it because the most basic license (to allow 25 devices to login) is over €1200 and using the Captive Portal aspect (which is what I actually wanted) it was going to cost over €4500…   I pulled it apart… I found that the IC4500 is just a Dual Core, 1-RU server with a couple of gigs of RAM, an 80G hard drive and 2 Gigabit Ethernet ports… so changing the drive to something larger and a bit of fiddling and I put the OS I have been developing on it (BSD Server UNIX -BSDSUX for short) and now I have a captive portal of my own making…  so last thing was to get the Access Points able to do both Open Security and WPA2-Enterprise at the same time, and when logged in get forced off the open wireless and allowed onto the secure wireless.

So finally to the point…

The Internet of Security Issues

Not so long ago a number of security vulnerabilities were hitting the headlines, and in particular ‘ShellShock’ so running Amped Wireless AP20000G‘s around my home which I happen to know run Linux I was a little concerned.  I had the latest firmware on the devices and this was dated  few years earlier (13 Dec 2012) so I emailed Amped Wireless about the issue and wasn’t actually told anything about the issue except they’d review the bug.  Time went by and more and more issues came up, and still no firmware… the latest one is CVE-2017-6074 which was introduced to the Linux Kernel way back in 2006, in fact the vulnerability description states this:

The oldest version that was checked is 2.6.18 (Sep 2006), which is
vulnerable. However, the bug was introduced before that, probably in the first release with DCCP support (2.6.14, Oct 2005).

Now the clueful of you would know that this is a local privilege escalation issue and when it comes to routers, APs etc you’d actually have to get on the device to exploit it.  The same clueful will know that’s not as difficult as it might sound.

So figuring that I’m never going to get the firmware update I need/want I might as well go about hacking the router myself and building my own firmware that can indeed work with the IC4500 and finally finish securing my network to the level I want.

(and for those fed up with reading… if you haven’t worked it out… it’s 2017, the Access Point is classed as one of the ‘Internet of Things’ it is vulnerable to hacking on multiple fronts and 5 years later and I can’t get an update to the firmware – even though they are still selling these devices in shops!!!! … the gory horror for the techs is coming, so keep reading if you want…)

First things first when going down this path… Research the hardware and see what’s available… the Website ‘WikiDevi‘ is great for this and provides the following details

CPU1: Realtek RTL8198 (620 MHz)
FLA1: 8 MiB (Macronix MX25L6406EM2I-12G)
RAM1: 64 MiB (Hynix H5PS5162GFR-S6C)

WI1 chip1: Realtek RTL8192DR
WI1 802dot11 protocols: an
WI1 MIMO config: 2×2:2
WI1 antenna connector: RP-SMA
WI2 chip1: Realtek RTL8192CE
WI2 802dot11 protocols: bgn
WI2 MIMO config: 2×2:2
WI2 antenna connector: RP-SMA

ETH chip1: Realtek RTL8198
Switch: Realtek RTL8198
LAN speed: 10/100/1000
LAN ports: 4
WAN speed: 10/100/1000
WAN ports: 1

Which also tells me that normal OpenWRT support is not available (they don’t support RealTek devices mostly).. but more looking (and the WikiDevi page now says it) there is RealTek support by some authors.  Looking up the chips I also get information there is JTAG support (which is basically a serial port for debugging) so I got to work with my screwdriver and soldering iron and this was the result…

Which applying power produced the following in a minicom session.

Booting...?
========== SPI =============
SDRAM CLOCK:181MHZ
 ------------------------- Force into Single IO Mode ------------------------ 
|No chipID  Sft chipSize blkSize secSize pageSize sdCk opCk      chipName    |
| 0 c22017h  0h  800000h  10000h   1000h     100h   86   30   MX6405D/05E/45E|
 ---------------------------------------------------------------------------- 
Reboot Result from Watchdog Timeout!

---RealTek(RTL8198)at 2012.04.12-16:11+0800 version v1.2 [16bit](620MHz)
no sys signature at 00010000!
no sys signature at 00020000!
no sys signature at 00030000!
no sys signature at 00140000!
no rootfs signature at 000E0000!
no rootfs signature at 000F0000!
no rootfs signature at 00130000!
no rootfs signature at 00240000!
Jump to image start=0x80500000...
decompressing kernel:
Uncompressing Linux... done, booting the kernel.
done decompressing kernel.
start address: 0x80003640
RTL8192C/RTL8188C driver version 1.6 (2011-07-18)



Probing RTL8186 10/100 NIC-kenel stack size order[3]...
chip name: 8196C, chip revid: 0
NOT YET
eth0 added. vid=9 Member port 0x1...
eth1 added. vid=8 Member port 0x10...
eth2 added. vid=9 Member port 0x2...
eth3 added. vid=9 Member port 0x4...
eth4 added. vid=9 Member port 0x8...
[peth0] added, mapping to [eth1]...
init started: BusyBox v1.13.4 (2012-12-13 11:08:29 CST)
Init Start...
Init bridge interface...
killall: smbd: no process killed
killall: nmbd: no process killed
basename(1)
basename(2 /sys/block/sda)
basename(2 /block/sda)
basename(2 /sda)
basename(3 sda)
basename(1)
basename(2 /sys/block/sda)
basename(2 /block/sda)
basename(2 /sda)
basename(3 sda)
basename(1)
basename(2 /sys/block/sda/sda1)
basename(2 /block/sda/sda1)
basename(2 /sda/sda1)
basename(2 /sda1)
basename(3 sda1)
basename(1)
basename(2 /sys/block/sda/sda1)
basename(2 /block/sda/sda1)
basename(2 /sda/sda1)
basename(2 /sda1)
basename(3 sda1)
try_mount(1) sda1, /var/tmp/usb/sda1
CMD: /bin/ntfs-3g /dev/sda1 /var/tmp/usb/sda1 -o force

Error opening '/dev/sda1': No such device or address
Failed to mount '/dev/sda1': No such device or address
Either the device is missing or it's powered down, or you have
SoftRAID hardware and must use an activated, different device under
/dev/mapper/, (e.g. /dev/mapper/nvidia_eahaabcc1) to mount NTFS.
Please see the 'dmraid' documentation for help.
Init Wlan application...

WiFi Simple Config v2.3 (2011.11.08-13:04+0000).

Register to wlan0
Register to wlan1
route: SIOCDELRT: No such process
iwcontrol RegisterPID to (wlan0)
iwcontrol RegisterPID to (wlan1)
$$$ eth1 & eth0 up $$$
IEEE 802.11f (IAPP) using interface br0 (v1.7)
#

As one can see straight in at a root prompt (no login – but hey, needs to physically connect to it with a soldering iron…), and we can see it’s running BusyBox (which means it’s running ash not bash so not vulnerable to Shellshock – nice of the company to tell me!??!?!)…  But confirmed….

# x='() { :;}; echo VULNERABLE' ash -c : 
#

So what about the latest bug that goes back to 2006… well…

# cat /proc/version   
Linux version 2.6.30.9 (kevinlin@localhost.localdomain) (gcc version 3.4.6-1.3.6) #603 Thu Dec 13 15:14:20 CST 2012

That would be a yes then…  In fact we can see that this OS was made with the old version of the RealTek SDK

# cat /etc/version
RTL8198 v1.0 --  Thu Dec 13 15:13:43 CST 2012
The SDK version is: Realtek SDK v2.5-r7984
Ethernet driver version is: 7953-7929
Wireless driver version is: 7977-7977
Fastpath source version is: 7873-6572
Feature support version is: 7927-7480

So my next trick is to work out which GPIO pins I need to manipulate to get the power output control of the Skyworks (SiGe) SE5004L / 5004L power amplifiers under my control but that’s digressing from the topic of this post.  Poking around looking for the details and I found something else rather interesting…

# ps -ax
  PID USER       VSZ STAT COMMAND
    1 root      1576 S    init      
    2 root         0 SW<  [kthreadd]
    3 root         0 SW<  [ksoftirqd/0]
    4 root         0 SW<  [events/0]
    5 root         0 SW<  [khelper]
    8 root         0 SW<  [async/mgr]
   61 root         0 SW<  [kblockd/0]
   71 root         0 SW<  [khubd]
   88 root         0 SW   [pdflush]
   89 root         0 SW<  [kswapd0]
  649 root         0 SW<  [mtdblockd]
  870 root     13760 S    /bin/smbd -D -s /var/smb.conf 
  878 root     13808 S    /bin/smbd -D -s /var/smb.conf 
  882 root      6508 S    /bin/nmbd -D -s /var/smb.conf 
  902 root       960 S    iapp br0 wlan0 wlan1 
  913 root      1260 S    wscd -start -c /var/wsc-wlan1.conf -w wlan1 -fi /var/
  917 root       984 S    iwcontrol wlan0 wlan1 
  942 root      1008 S    dnrd --cache=off -s 168.95.1.1 
  951 root       956 S    reload -k /var/wlsch.conf 
  984 root      2168 S    webs 
  985 root      1584 S    -/bin/sh 
 1021 root      1576 R    ps -ax 
#

.. That little thing that says, “dnrd –cache=off -s 168.95.1.1” .. What this program is is a DNS relay server ie something to help you resolve addresses from the names we know and are used to like “www.microsoft.com” into the quad octet that the computers can deal with called an ‘IP Address’.  Now the reason I’m pointing it out is that 168.95.1.1 is not something I have configured and it is not something on my network, so it tweaked my curiosity.  Turns out it belongs to a Taiwanese company “Chunghwa Telecom Co., Ltd”

$ host 168.95.1.1
1.1.95.168.in-addr.arpa domain name pointer dns.hinet.net.
$ whois hinet.net

.
.
.

   Server Name: HINET.NET.TW
   Registrar: MELBOURNE IT, LTD. D/B/A INTERNET NAMES WORLDWIDE
   Whois Server: whois.melbourneit.com
   Referral URL: http://www.melbourneit.com.au


   Domain Name: HINET.NET
   Registrar: NETWORK SOLUTIONS, LLC.
   Sponsoring Registrar IANA ID: 2
   Whois Server: whois.networksolutions.com
   Referral URL: http://networksolutions.com
   Name Server: ANS1.HINET.NET
   Name Server: ANS2.HINET.NET
   Status: ok https://icann.org/epp#ok
   Updated Date: 02-feb-2017
   Creation Date: 19-mar-1994
   Expiration Date: 20-mar-2018

.
.
.

Domain Name: HINET.NET
Registry Domain ID: 2854475_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.networksolutions.com
Registrar URL: http://networksolutions.com
Updated Date: 2017-03-05T15:11:26Z
Creation Date: 1994-03-19T05:00:00Z
Registrar Registration Expiration Date: 2018-03-20T04:00:00Z
Registrar: NETWORK SOLUTIONS, LLC.
Registrar IANA ID: 2
Registrar Abuse Contact Email: abuse@web.com
Registrar Abuse Contact Phone: +1.8003337680
Reseller: 
Domain Status: ok https://icann.org/epp#ok
Registry Registrant ID: 
Registrant Name: Internet Dept., DCBG, Chunghwa Telecom Co., Ltd.
Registrant Organization: Internet Dept., DCBG, Chunghwa Telecom Co., Ltd.
Registrant Street: Data-Bldg, No. 21 Sec.1, Hsin-Yi Rd.
Registrant City: Taipei
Registrant State/Province: Taiwan
Registrant Postal Code: 100
Registrant Country: TW
Registrant Phone: +886.223444720
Registrant Phone Ext: 
Registrant Fax: +886.223960399
Registrant Fax Ext: 
Registrant Email: vnsadm@hinet.net
Registry Admin ID: 
Admin Name: Internet Dept., DCBG, Chunghwa Telecom Co., Ltd.
Admin Organization: Internet Dept., DCBG, Chunghwa Telecom Co., Ltd.
Admin Street: Data-Bldg, No. 21 Sec.1, Hsin-Yi Rd.
Admin City: Taipei
Admin State/Province: Taiwan
Admin Postal Code: 100
Admin Country: TW
Admin Phone: +886.223444720
Admin Phone Ext: 
Admin Fax: +886.223960399
Admin Fax Ext: 
Admin Email: vnsadm@hinet.net

So the not only is this Access Point vulnerable to hacking it’s also sending details of every site I’m going to back to a server in Taiwan…  Well not quite, because unlike most home users I am using my own DNS servers and have specifically blocked the access points from talking to the Internet… I am not your average home user though.  That leads me to the following conclusion that some will find scary…

The Conclusion…

The biggest current threat to our networks, our privacy, and our electronic identities (including funds) is the threat of the Internet of Things that have not been patched. 

This threat is massive as the clueful people out there often can’t patch because the companies selling the devices are not providing security fixes because their profit is about getting new devices out there, not fixing old devices. 

It’s even bigger because most of the world are not techs, they don’t even know how to update the firmware or where it would even be available if they did. 

…Yet we’re all connecting up to the Internet, we’re all buying these boxes from household temperature controls available on your phone to Smart TVs and Fridges… even ‘Smart Bulbs‘!

All of which have the ability to run code, all of which have potential security issues, and all of which can provide the unethical people out there, ‘doorways into you home’.

 

Social Engineering on Social Networking…

Its been a long while since I’ve posted anything, but in todays news how many of you out there are on Facebook and how many of you like to respond to these little gems:

“Red mushroom burger”

Or perhaps statuses that ask you to cut/paste in your answers:

Thursday Night fun… Six names.

Real Name: Michelle Sullivan

Soap opera name (middle name and street you live on): Isabelle Hemel

Star Wars name (first 3 letters of your last name, first 2 of middle, and last 2 of first): Sulisle

Superhero name (colour of shirt and item to your right): Grey mouse

Goth name (Black and pet name): Black Melody Pond

Rapper name (Lil and last thing you ate): Lil mushroom burger

Copy, paste and change if you wish to play too!

 

Both of these two things together often give enough information to people like me to enable me to steal from you, like stealing your Facebook account or your Hotmail account, or perhaps your iCloud account which then gives me access to your phone, location and photos… Maybe even your bank accounts….

Consider how many sites you visit and put in a load of details to ‘sign up’ and how many of those sites ask for additional security questions incase you forgot your password, then consider how much information you put in your Social Networking accounts that others can view…

One of the favourite questions always used to be ‘What is your mothers maiden name?’, and still is for many banks.  Here’s the problem in a nutshell: many people now have Facebook accounts and the parents of those people usually have Facebook accounts as well and how many of you have seen friends’ parents with names on Facebook such as, “Mary Johnson (nee Knowles)”.  Then how many look at the profile under the ‘About’ and see, “Sister, Dad, Mum” entries… because by default this information is viewable by fiends of friends….

Now taking all that in above along with friends posting “Happy Birthday, 40 today, can’t believe how old we all are now, congrats mate” etc (‘today’ being 24/2/2017) and that the ‘picture questions’ when shared from a page the answers are shared with the people on the page… consider what most people can see about you.  In the example I have written/posted above lets recap:

  • Real Name: Michelle Isabelle Sullivan
  • Date of birth: 24 Feb 1977
  • Favourite Colour: Red
  • My Address: Triq il-Hemel, Swieqi, Malta
  • My pets name: Melody Pond
  • What sort of pet: Cat
  • Mother’s Maiden name: Knowles
  • Brother’s Name: Stephen

Sound familiar?  What a bank might ask you on the phone for ‘confirming your identity’ by any chance?

How did I get all this you might ask… because not all is that obvious, well

“Name” is an easy one, but hey, need to know the full name, so my “Soap Opera name” gave me two details “Isabelle Hemel”, one of which was middle name.  That was verified because we also asked, the “Starwars Name” which use initials from each part of our name.

“Date of Birth”, easy but you might have missed it, I said I got wished a “Happy 40th Birthday” today (and probably got several hundred best wishes) all of which are posted with the permissions of the poster, *NOT* what permissions you have on your ‘Timeline’.  So being that I said “today” is 24th Feb 2017 and I’m getting “Happy 40” wishes that makes my Date of Birth 24 Feb 1977.

“My Favourite colour”, trivial, what was my “wand’s magic name” again? Oh that’s right, “Red mushroom burger“.

Social Engineering tip: ask other irrelevant details with the detail you want, people don’t spot they are giving something away that they might otherwise not, how many of you would answer truthfully if a stranger came up to you in the street and asked you, “What’s your favourite colour, and how old are you?”

“My Address” little more tricky this one, I’m sure you got the first one, my ‘Soap Opera name’ is “Isabelle Hemel but how did I get to “Triq il-Hemel, Swieqi, Malta”..?  Simple take a look at your timeline and look for the location information on most posts, on mine it says, “Swieqi” on a vast number of posts, if you look at Google maps and search for “Hemel Swieqi” that will give you the rest of the address (and even the postal code in many cases.)

How many of you have seen the “What’s your pet’s name?” as a ‘security question’ … well guess what… My “Goth Name” was “Black Melody Pond” and that just gave it to you!  How did I get to the pet being a “cat” though?  Well just go look at my photos, especially for ones with pets in them and you find I own a cat, and it takes very little to tie “Melody” to “a Cat“.

Mother’s Maiden name and Brother’s Name – well I told you that already, it’s on the “About” page of Facebook, not to mention that many people have mums that interact with them on Facebook and usually by sharing posts.  For example, how many of you have photos of you and your mum?  How many of those photos did you “tag” your mum in?

You might be asking, “But what about the ‘Rapper Name’ where does that fit in?” .. well remember what I said about, “ask other irrelevant details”.  It is bogus information, but it makes you feel better about giving me details you wouldn’t normally share…

Some of these details Facebook encourage setting better permissions on, but even with these ‘security checkups’ often the details are already leaked or are available to ‘Friends of Friends’… A study a while ago found most people in the world are 7 people away. What that means is if you go down 5 levels of ‘Friends of Friends’ (ie “Friends of Friends of Friends of Friends of Friends of Friends”) you will be linked to most people in the world