The IoT should really be IoSI (Internet of Security Issues)

The Internet of Things

So here I am seeing issues, reading about issues and trying to stop issues in the Internet of Things…  Everyday someone seems to be publishing articles on the issues, people are getting more aware (you’d think!) but there seems to be no real movement.

Some of my readers will know what I do for my day job, for those that don’t I wrote the SORBS Anti-spam system.. not quite the most hated, but some who should know better have said they just want me dead, then SORBS dead, then me killed again just to be sure I’m actually dead.  Several years ago I spent Christmas sitting in front of my computers rewriting part of the system, particularly that part that finds “bad stuff” and reports it (eg Open-Relay Servers) and whilst scanning hosts that were actively trying to send spam and/or viruses to me I came across the web page of a fridge.  The page half loaded before it became completely unresponsive and tracing it I found it on an IP address that appeared to be in Rome (Italy)….  When I reported my finding of a ‘Fridge Spamming’ to my boss all hell broke loose, blog articles were written, front pages were held and suddenly the world knew about ‘Fridges Spamming‘.  Shortly there after we got debunked by our main competitor of the time who asserted it wasn’t possible, the article however sparked off massive research and watching of the technology from a security stance.

In July of the same year a bunch of researchers at a University found that the premise of the ‘debunking’ was actually false and that with a specific sequence of commands it was possible to get the fridge concerned into a system ‘admin/debug’ mode that allowed a remote attacker to use the device as a simple proxy server and install other “apps”.  This largely went unnoticed in IoT industry with respect to the original report, I never understood why… perhaps someone can explain that to me? 🙂

3 years later…

One would think we have learned something, we certainly have seen more of these types of attacks, not always for spam but just as a device to get into a network, to provide the door way.  Indeed the attackers have pretty much made an art out of it, using combinations of direct hacks, social engineering to gain access or persuade users to install things and even stealing devices…  The lists and lengths seems endless, especially when you consider who is doing this sort of thing and even who is paying who…   We’ve all heard about Trump and Russia and the controversy, well there are teams of hackers in Russia who’s sole income is to break into systems and steal secrets.  Its not a stretch to imagine that they are not unconnected…  Personally I don’t go into the conspiracy theories but I can tell you there are companies and persons of interest that do pay for services of such teams and not just Russian ones, there are European teams, Chinese teams and American etc..

The result is a lot more tech out there, all with security issues and all trying to keep market share, by innovating or by destroying the competition.

So why are we helping these people along?  Why are we allowing companies to circumvent privacy laws?  Why are they even trying?  Why are there more and more companies dealing with security remediation rather than companies dealing with the actual problem…?

All questions for you the reader (and hopefully some people that can effect change.)

So what is this blog post about? Why did you write it?

Well quite simply I chase down security patches for my services…  You see I still manage SORBS and recently we moved some of the servers around to a new Datacenter and as a consequence I changed a lot of security settings to make the systems more secure.  The fall out of this was I completely re-wired my home office network and the only thing on my network now that is not ‘secured’ (ie may have issues) was my wireless network.

Originally I had an OpenVPN connection for every service over the wireless that was an ‘authorised machine’ and a straight session login for controlling access.  I deliberately set the whole network to ‘Open’ (ie unencrypted) to remind people using it that everything can be watched so if it’s important, use HTTPS (or use the OpenVPN) etc.

I decided to switch the network to WPA2-Enterprise for authorised users, and to use a Juniper NAC to provide a captive portal and control the logins etc…  I didn’t account for the ridiculous cost of the licenses of the Juniper NAC so even though I picked up a brand new IC4500 for less than €70 I couldn’t use it because the most basic license (to allow 25 devices to login) is over €1200 and using the Captive Portal aspect (which is what I actually wanted) it was going to cost over €4500…   I pulled it apart… I found that the IC4500 is just a Dual Core, 1-RU server with a couple of gigs of RAM, an 80G hard drive and 2 Gigabit Ethernet ports… so changing the drive to something larger and a bit of fiddling and I put the OS I have been developing on it (BSD Server UNIX -BSDSUX for short) and now I have a captive portal of my own making…  so last thing was to get the Access Points able to do both Open Security and WPA2-Enterprise at the same time, and when logged in get forced off the open wireless and allowed onto the secure wireless.

So finally to the point…

The Internet of Security Issues

Not so long ago a number of security vulnerabilities were hitting the headlines, and in particular ‘ShellShock’ so running Amped Wireless AP20000G‘s around my home which I happen to know run Linux I was a little concerned.  I had the latest firmware on the devices and this was dated  few years earlier (13 Dec 2012) so I emailed Amped Wireless about the issue and wasn’t actually told anything about the issue except they’d review the bug.  Time went by and more and more issues came up, and still no firmware… the latest one is CVE-2017-6074 which was introduced to the Linux Kernel way back in 2006, in fact the vulnerability description states this:

The oldest version that was checked is 2.6.18 (Sep 2006), which is
vulnerable. However, the bug was introduced before that, probably in the first release with DCCP support (2.6.14, Oct 2005).

Now the clueful of you would know that this is a local privilege escalation issue and when it comes to routers, APs etc you’d actually have to get on the device to exploit it.  The same clueful will know that’s not as difficult as it might sound.

So figuring that I’m never going to get the firmware update I need/want I might as well go about hacking the router myself and building my own firmware that can indeed work with the IC4500 and finally finish securing my network to the level I want.

(and for those fed up with reading… if you haven’t worked it out… it’s 2017, the Access Point is classed as one of the ‘Internet of Things’ it is vulnerable to hacking on multiple fronts and 5 years later and I can’t get an update to the firmware – even though they are still selling these devices in shops!!!! … the gory horror for the techs is coming, so keep reading if you want…)

First things first when going down this path… Research the hardware and see what’s available… the Website ‘WikiDevi‘ is great for this and provides the following details

CPU1: Realtek RTL8198 (620 MHz)
FLA1: 8 MiB (Macronix MX25L6406EM2I-12G)
RAM1: 64 MiB (Hynix H5PS5162GFR-S6C)

WI1 chip1: Realtek RTL8192DR
WI1 802dot11 protocols: an
WI1 MIMO config: 2×2:2
WI1 antenna connector: RP-SMA
WI2 chip1: Realtek RTL8192CE
WI2 802dot11 protocols: bgn
WI2 MIMO config: 2×2:2
WI2 antenna connector: RP-SMA

ETH chip1: Realtek RTL8198
Switch: Realtek RTL8198
LAN speed: 10/100/1000
LAN ports: 4
WAN speed: 10/100/1000
WAN ports: 1

Which also tells me that normal OpenWRT support is not available (they don’t support RealTek devices mostly).. but more looking (and the WikiDevi page now says it) there is RealTek support by some authors.  Looking up the chips I also get information there is JTAG support (which is basically a serial port for debugging) so I got to work with my screwdriver and soldering iron and this was the result…

Which applying power produced the following in a minicom session.

Booting...?
========== SPI =============
SDRAM CLOCK:181MHZ
 ------------------------- Force into Single IO Mode ------------------------ 
|No chipID  Sft chipSize blkSize secSize pageSize sdCk opCk      chipName    |
| 0 c22017h  0h  800000h  10000h   1000h     100h   86   30   MX6405D/05E/45E|
 ---------------------------------------------------------------------------- 
Reboot Result from Watchdog Timeout!

---RealTek(RTL8198)at 2012.04.12-16:11+0800 version v1.2 [16bit](620MHz)
no sys signature at 00010000!
no sys signature at 00020000!
no sys signature at 00030000!
no sys signature at 00140000!
no rootfs signature at 000E0000!
no rootfs signature at 000F0000!
no rootfs signature at 00130000!
no rootfs signature at 00240000!
Jump to image start=0x80500000...
decompressing kernel:
Uncompressing Linux... done, booting the kernel.
done decompressing kernel.
start address: 0x80003640
RTL8192C/RTL8188C driver version 1.6 (2011-07-18)



Probing RTL8186 10/100 NIC-kenel stack size order[3]...
chip name: 8196C, chip revid: 0
NOT YET
eth0 added. vid=9 Member port 0x1...
eth1 added. vid=8 Member port 0x10...
eth2 added. vid=9 Member port 0x2...
eth3 added. vid=9 Member port 0x4...
eth4 added. vid=9 Member port 0x8...
[peth0] added, mapping to [eth1]...
init started: BusyBox v1.13.4 (2012-12-13 11:08:29 CST)
Init Start...
Init bridge interface...
killall: smbd: no process killed
killall: nmbd: no process killed
basename(1)
basename(2 /sys/block/sda)
basename(2 /block/sda)
basename(2 /sda)
basename(3 sda)
basename(1)
basename(2 /sys/block/sda)
basename(2 /block/sda)
basename(2 /sda)
basename(3 sda)
basename(1)
basename(2 /sys/block/sda/sda1)
basename(2 /block/sda/sda1)
basename(2 /sda/sda1)
basename(2 /sda1)
basename(3 sda1)
basename(1)
basename(2 /sys/block/sda/sda1)
basename(2 /block/sda/sda1)
basename(2 /sda/sda1)
basename(2 /sda1)
basename(3 sda1)
try_mount(1) sda1, /var/tmp/usb/sda1
CMD: /bin/ntfs-3g /dev/sda1 /var/tmp/usb/sda1 -o force

Error opening '/dev/sda1': No such device or address
Failed to mount '/dev/sda1': No such device or address
Either the device is missing or it's powered down, or you have
SoftRAID hardware and must use an activated, different device under
/dev/mapper/, (e.g. /dev/mapper/nvidia_eahaabcc1) to mount NTFS.
Please see the 'dmraid' documentation for help.
Init Wlan application...

WiFi Simple Config v2.3 (2011.11.08-13:04+0000).

Register to wlan0
Register to wlan1
route: SIOCDELRT: No such process
iwcontrol RegisterPID to (wlan0)
iwcontrol RegisterPID to (wlan1)
$$$ eth1 & eth0 up $$$
IEEE 802.11f (IAPP) using interface br0 (v1.7)
#

As one can see straight in at a root prompt (no login – but hey, needs to physically connect to it with a soldering iron…), and we can see it’s running BusyBox (which means it’s running ash not bash so not vulnerable to Shellshock – nice of the company to tell me!??!?!)…  But confirmed….

# x='() { :;}; echo VULNERABLE' ash -c : 
#

So what about the latest bug that goes back to 2006… well…

# cat /proc/version   
Linux version 2.6.30.9 (kevinlin@localhost.localdomain) (gcc version 3.4.6-1.3.6) #603 Thu Dec 13 15:14:20 CST 2012

That would be a yes then…  In fact we can see that this OS was made with the old version of the RealTek SDK

# cat /etc/version
RTL8198 v1.0 --  Thu Dec 13 15:13:43 CST 2012
The SDK version is: Realtek SDK v2.5-r7984
Ethernet driver version is: 7953-7929
Wireless driver version is: 7977-7977
Fastpath source version is: 7873-6572
Feature support version is: 7927-7480

So my next trick is to work out which GPIO pins I need to manipulate to get the power output control of the Skyworks (SiGe) SE5004L / 5004L power amplifiers under my control but that’s digressing from the topic of this post.  Poking around looking for the details and I found something else rather interesting…

# ps -ax
  PID USER       VSZ STAT COMMAND
    1 root      1576 S    init      
    2 root         0 SW<  [kthreadd]
    3 root         0 SW<  [ksoftirqd/0]
    4 root         0 SW<  [events/0]
    5 root         0 SW<  [khelper]
    8 root         0 SW<  [async/mgr]
   61 root         0 SW<  [kblockd/0]
   71 root         0 SW<  [khubd]
   88 root         0 SW   [pdflush]
   89 root         0 SW<  [kswapd0]
  649 root         0 SW<  [mtdblockd]
  870 root     13760 S    /bin/smbd -D -s /var/smb.conf 
  878 root     13808 S    /bin/smbd -D -s /var/smb.conf 
  882 root      6508 S    /bin/nmbd -D -s /var/smb.conf 
  902 root       960 S    iapp br0 wlan0 wlan1 
  913 root      1260 S    wscd -start -c /var/wsc-wlan1.conf -w wlan1 -fi /var/
  917 root       984 S    iwcontrol wlan0 wlan1 
  942 root      1008 S    dnrd --cache=off -s 168.95.1.1 
  951 root       956 S    reload -k /var/wlsch.conf 
  984 root      2168 S    webs 
  985 root      1584 S    -/bin/sh 
 1021 root      1576 R    ps -ax 
#

.. That little thing that says, “dnrd –cache=off -s 168.95.1.1” .. What this program is is a DNS relay server ie something to help you resolve addresses from the names we know and are used to like “www.microsoft.com” into the quad octet that the computers can deal with called an ‘IP Address’.  Now the reason I’m pointing it out is that 168.95.1.1 is not something I have configured and it is not something on my network, so it tweaked my curiosity.  Turns out it belongs to a Taiwanese company “Chunghwa Telecom Co., Ltd”

$ host 168.95.1.1
1.1.95.168.in-addr.arpa domain name pointer dns.hinet.net.
$ whois hinet.net

.
.
.

   Server Name: HINET.NET.TW
   Registrar: MELBOURNE IT, LTD. D/B/A INTERNET NAMES WORLDWIDE
   Whois Server: whois.melbourneit.com
   Referral URL: http://www.melbourneit.com.au


   Domain Name: HINET.NET
   Registrar: NETWORK SOLUTIONS, LLC.
   Sponsoring Registrar IANA ID: 2
   Whois Server: whois.networksolutions.com
   Referral URL: http://networksolutions.com
   Name Server: ANS1.HINET.NET
   Name Server: ANS2.HINET.NET
   Status: ok https://icann.org/epp#ok
   Updated Date: 02-feb-2017
   Creation Date: 19-mar-1994
   Expiration Date: 20-mar-2018

.
.
.

Domain Name: HINET.NET
Registry Domain ID: 2854475_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.networksolutions.com
Registrar URL: http://networksolutions.com
Updated Date: 2017-03-05T15:11:26Z
Creation Date: 1994-03-19T05:00:00Z
Registrar Registration Expiration Date: 2018-03-20T04:00:00Z
Registrar: NETWORK SOLUTIONS, LLC.
Registrar IANA ID: 2
Registrar Abuse Contact Email: abuse@web.com
Registrar Abuse Contact Phone: +1.8003337680
Reseller: 
Domain Status: ok https://icann.org/epp#ok
Registry Registrant ID: 
Registrant Name: Internet Dept., DCBG, Chunghwa Telecom Co., Ltd.
Registrant Organization: Internet Dept., DCBG, Chunghwa Telecom Co., Ltd.
Registrant Street: Data-Bldg, No. 21 Sec.1, Hsin-Yi Rd.
Registrant City: Taipei
Registrant State/Province: Taiwan
Registrant Postal Code: 100
Registrant Country: TW
Registrant Phone: +886.223444720
Registrant Phone Ext: 
Registrant Fax: +886.223960399
Registrant Fax Ext: 
Registrant Email: vnsadm@hinet.net
Registry Admin ID: 
Admin Name: Internet Dept., DCBG, Chunghwa Telecom Co., Ltd.
Admin Organization: Internet Dept., DCBG, Chunghwa Telecom Co., Ltd.
Admin Street: Data-Bldg, No. 21 Sec.1, Hsin-Yi Rd.
Admin City: Taipei
Admin State/Province: Taiwan
Admin Postal Code: 100
Admin Country: TW
Admin Phone: +886.223444720
Admin Phone Ext: 
Admin Fax: +886.223960399
Admin Fax Ext: 
Admin Email: vnsadm@hinet.net

So the not only is this Access Point vulnerable to hacking it’s also sending details of every site I’m going to back to a server in Taiwan…  Well not quite, because unlike most home users I am using my own DNS servers and have specifically blocked the access points from talking to the Internet… I am not your average home user though.  That leads me to the following conclusion that some will find scary…

The Conclusion…

The biggest current threat to our networks, our privacy, and our electronic identities (including funds) is the threat of the Internet of Things that have not been patched. 

This threat is massive as the clueful people out there often can’t patch because the companies selling the devices are not providing security fixes because their profit is about getting new devices out there, not fixing old devices. 

It’s even bigger because most of the world are not techs, they don’t even know how to update the firmware or where it would even be available if they did. 

…Yet we’re all connecting up to the Internet, we’re all buying these boxes from household temperature controls available on your phone to Smart TVs and Fridges… even ‘Smart Bulbs‘!

All of which have the ability to run code, all of which have potential security issues, and all of which can provide the unethical people out there, ‘doorways into you home’.

 

Converting h.265 (HEVC) to h.264 (AVC)

Quick techie entry for anyone using the newer h265 codecs but unable to use them in players. (eg: Torrenting H265 encoded files then trying to play via PLEX and Roku)

Roku and other media players don’t support h.265 and as such any attempt to play h265 encoded files will result in an ‘Unable to play file’ error, so you might be wanting to convert the files to another format such as h.264.  To do this you need ffmpeg, however ffmpeg can be a little difficult to work, especially as it has so many options, so I wrote a little perl script to ‘mass convert’ all files in the current directory if they are h265 encoded to h264 encoding.  It is published here for those on UNIX systems (or those who know how to install Perl on Windows) to make life a little easier:

#!/usr/bin/perl

use strict;
use warnings;

open DIR, "ls -1 |";
while (<DIR>)
{
        chomp;
        next if ( -d "$_"); # skip directories
        next unless ( -r "$_"); # if it's not readable skip it!
        my $file = $_;
        open PROBE, "ffprobe -show_streams -of csv '$file' 2>/dev/null|" or die ("Unable to launch ffmpeg for $file! ($!)");
        my ($v, $a, $s, @c) = (0,0,0);
        while (<PROBE>)
        {
                my @streaminfo = split(/,/, $_);
                push(@c, $streaminfo[2]) if ($streaminfo[5] eq "video");
                $a++ if ($streaminfo[5] eq "audio");
                $s++ if ($streaminfo[5] eq "subtitle");
        }
        close PROBE;
        $v = scalar @c;
        if (scalar @c eq 1 and $c[0] eq "ansi")
        {
                warn("Text file detected, skipping...\n");
                next;
        }
        warn("$file: Video Streams: $v, Audio Streams: $a, Subtitle Streams: $s, Video Codec(s): " . join (", ", @c) . "\n");
        if (scalar @c > 1)
        {
                warn("$file has more than one video stream, bailing!\n");
                next;
        }
        if ($c[0] eq "hevc")
        {
                warn("HEVC detected for $file ...converting to AVC...\n");
                system("mkdir -p h265");
                my @params = ("-hide_banner", "-threads 2");
                push(@params, "-map 0") if ($a > 1 or $s > 1 or $v > 1);
                push(@params, "-c:a copy") if ($a);
                push(@params, "-c:s copy") if ($s);
                push(@params, "-c:v libx264 -pix_fmt yuv420p") if ($v);
                if (system("mv '$file' 'h265/$file'"))
                {
                        warn("Error moving $file -> h265/$file\n");
                        next;
                }
                if (system("ffmpeg -xerror -i 'h265/$file' " . join(" ", @params) . " '$file' 2>/dev/null"))
                {
                        warn("FFMPEG ERROR.  Cannot convert $file restoring original...\n");
                        system("mv 'h265/$file' '$file'");
                        next;
                }
        } else {
                warn("$file doesn't appear to need converting... Skipping...\n");
        }
}
close DIR;


Enjoy!

A history of my experience with FreeBSD and a warning to users….

So a rant about how a great project can go bad, and how it’s still doing stuff that should never be done and why I’ve decided to ‘fix it myself’ or move away from it completely.

Back in 2003 Paul Vixie forced me into using FreeBSD on one of my servers, it was not a welcome change for me, I was an avid Linux user until this point…. and it didn’t go well.. I started on 4.x, found there was no threading support, so “upgraded” the system to 5.x… which went badly…very badly… and every upgrade through 5.x was as bad…. Partly because of what I did, partly because of my lack of knowledge and partly because of system limitations.

Out came 6.0 and I started working with it and soon I had a whole slew of machines that were on 6.0 and with 6.1 and 6.2 things only got better. I had build servers, I had package servers, I could boot one of the servers off the network and have it completely re-install the server with the latest OS, Patches and Packages within 23 minutes (bare metal to built, configured and in production in 23 minutes…!)

Then came 7.0 and my ‘burnout’ – personal, profession life clashed, I ‘burnt out’ and my technical issues took a back burner, then before you knew it 7.3 and 7.4 were out and I had sold my company… and I was back working on getting things patched and upgraded… however some major changes had happened and the ‘ports’ tree no longer worked on 6.x hosts… so the entire system was frozen…. no new security patches, no upgrades, however along with the sale of the company came new hope… new hardware… and an opportunity to upgrade by replacing the hardware… New hardware was installed to 7.3 (as this was all that was available on Softlayer) and then my attention was diverted to getting my software upgraded to a new major revision and with it my attention and priorities changed from Sys-Admin work to developer and the older systems remained. Not long later the company that ‘bought me’, ‘sold me’ to another (my current) employer, Proofpoint Inc and new priorities were given… along with more new servers.. the result was 8.x systems being installed and with the advent of FreeBSD upgrading ‘bmake’ more stuff got changed in the ports tree, again making them non-working on pre 7.4 systems… more things got changed/patched on my servers and I ended up with new hardware again, this time running 9.0 and 9.1… at this point in time (2013) I had the following versions of FreeBSD in production:

  • FreeBSD 6.0
  • FreeBSD 6.1
  • FreeBSD 6.2
  • FreeBSD 7.2
  • FreeBSD 7.3
  • FreeBSD 8.1
  • FreeBSD 8.2
  • FreeBSD 8.3
  • FreeBSD 8.4
  • FreeBSD 9.0
  • FreeBSD 9.1

Which for any sys-admin you can guess would be a nightmare.  Further Proofpoint has policy and puppet, policies about how things are managed and puppet to manage everything.  It was suggested that my systems should be managed by puppet… so after Oct 2013 when the databases were finally migrated to the new hardware and then I could work on upgrading everything off old hardware and onto new OS’s and patches I setup a puppet server, a number of build servers and a test suite, all of my own creation and similar to what I had done in 2005… to take back control…  I also ended up with FreeBSD 9.2 on some servers, so I decided i would standardise on:

  • FreeBSD 8.4
  • FreeBSD 9.0
  • FreeBSD 9.1
  • FreeBSD 9.2

…at least until I could spend the time getting everything to a single OS level…  FreeBSD 10.0 came out, and later FreeBSD 9.3, but by that time I had the basic systems working and so adding these to the build and test suite was a matter of adding new build and test hosts… which just took a few hours.

As part of this build change I learned new tools:

  • Jenkins
  • Poudriere
  • Puppet
  • VirtualBox

I learned how to create my own ports, I learned how to patch my own ports privately.  I learned how to submit bugs back to FreeBSD ports maintainers.  I became a FreeBSD port maintainer myself.  I noted that as of 1st September 2014 the old pkg_* tools that had been around since day dot were about to be End-Of-Life’d in favour of a new ‘PKGNG‘ system.  I read the linked blog entry and decided that it was something I would have to look at, but later, because the EOL (as most sys-admins know) just means no new patches and something may start breaking that wouldn’t be supported by the developers.  At the end of July 2014 I spoke with the main protagonist of the change and was informed bluntly and to the point that they had already got a patch built and waiting to be applied, not to EOL the tools but to actually and deliberately break the existing tools thereby forcing people to use the new system.

Needless to say with less than 5 weeks of time before this was due to occur there was no chance of me converting all 57 servers, so I suggested that they shouldn’t I was told, its going to happen regardless… and that I should know that EOL means the product would no longer work, not that it would just not be supported anymore.  I guess all those years I had worked for the likes of Netscape, Oracle etc meant they all got it wrong… even Microsoft got it wrong, I mean Windows XP was ‘EOL’d a while back and well all those Window XP machines around the world just stopped working the same day… NOT!

So I continued with my build system and tried to get a stable patched repository of packages so I could at least continue my plan to get the servers to the standardised OS levels…  During testing of the packages I noted bugs, reported them to the developers, then pushed the maintainers (with mixed levels of success) to implement the fixes before the dead line (more appropriately named rather than EOL)… I failed.. several patches were not put into the ports tree until 7 days after the dead line (and that may have been deliberate on the developers aspect – though will never know.)  So the ports tree was patched on time, it rendered the old tools dead and my entire build, test and development system was broken.

I set about repairing it, for a while just copying pre-DeadLine files for building seemed to work with some local changes, so I continued to build out my systems to cope with this, and finally at the beginning of Dec 2014 I got a stable and complete repository.

Over Christmas 2014 I set myself the task of upgrading all servers to one of the standardised OS’s and at the same time patching all the existing OS’s on one of those versions.  Of 57 servers, 31 became un-usable in some way during the patch update process (freebsd-update)  Some became un-bootable, some couldn’t access the network, some (even going from 9.3-RELEASE to 9.3-P5) broke packages such as ‘sudo’ leaving me unable to gain increased privileges to finish the patch process…. after over 160 hours of work, only stopping Christmas day and New Years day, all systems were patched to 9.2 or 9.3 with all the security patches…as they had to be because of the NTPd remote root exploit…. only having to reinstall 2 of the systems from scratch as they were un-recoverable.

Early January 2015 the build system failed again when trying to patch new security issues and I found it was related to more changes by the same culprit so decided after seeing similar rants by other long standing advocates to ask for some help and got a working set of Mk/* files with the intention of fixing it again.  The files I got wouldn’t work so I merged the tree by hand (27900+ lines) only to find the system not quite working… a week later and I have a working build system for most of the ports.  I set it going and get a working repository and decide to re-run the build because of a failed patch, and it all broke again…

So for the warning to all FreeBSD Users:

IF YOU RUN PRODUCTION SERVERS THAT REQUIRE TESTING AND STABILITY BEFORE MAJOR CHANGES, YOU PROBABLY ARE STILL ON PKG_* TOOLS, DON’T UPGRADE, DON’T PATCH AND LOOK AT OTHER SOLUTIONS! Here’s why:

  • running ‘freebsd-update‘ the extra pass to “delete old” will delete all pkg_* tools (even if you haven’t converted to pkgng)
  • updating the ports tree and updating something will automatically convert the system to use pkgng (whether tested and working or not)
  • if you build your own packages using poudriere 3.1 or above it will also “upgrade” your system without confirmation or warning.

Basically whether tested or not, whether working or not, the FreeBSD developers (not the kernel devs as far as I know) will change your production systems to configurations that will probably render your automated systems completely ineffective, without warning and without notification.

What am I doing about it, well at the moment I have created a ports tree ( svn co http://svn.sorbs.net/repos/ports/head ) on http://svn.sorbs.net/repos/ports that can be put into poudriere (as SVN_HOST=svn.sorbs.net/repos ) and it will in theory build most packages for pkg_* tools – it’s not complete and is being changed on a daily basis currently as new changes go in, and with the latest “HEADSUP” announced on the FreeBSD Ports mailing list detailing another change in syntax that is not backward compatible with existing systems (even pkgng ones) I expect it won’t work for long….  My advice as the culprit seems hell bent on changing systems to the way Linux has been for years and ignoring all input from users of FreeBSD that does not agree with his vision, find an alternative.

After 12 years of promoting FreeBSD I am not any more, I’m not going to stop my employer moving everything to Linux, and I’m *NOT* going to upgrade anything to 10.x (and as 9.4 will probably not have pkg_* tools available, I won’t be going there either.)

Sadly, thinking about the whole issue, with a little work it could have been avoided, ensuring all variables in the ports are backwards compatible and having separate Mk/* repositories (even unmaintained/EOLd) would have made the whole process less painful an allowed the developers to continue their path, whether right or wrong, to completion, and allow us insignificant users to continue without pain.  In fact had someone had the for-sight I think even pre-bmake systems would still be patchable and working, even back to the 6.x tree! .. well at least until the new changes in the plist files… which most can be back-ported despite the claim that progress is impossible with the old pkg_* tools.

Spam in your fridge? Yeah sure, but what about spam from your fridge?

Well in light of the recent fascination and media-hype about spam from a new range of devices dubbed, the “Internet of Things”  I thought I post some information.

The media-hype is a little surprising in some ways as this ‘hack’ is old news, old technology and has been happening for years.  I first identified and tested for it publicly with the Spam and Open Relay Blocking System’s (SORBS) automated proxy tester.  The only thing that is different is now is the devices that can be exploited.  It used to be home routers, and computers directly connected to the Internet, but now it’s phones (Smart, VOIP and others), it’s Televisions, it’s Fridges, Cameras (usually security cameras, but not always), Digital Video Recorders, Set-Top boxes (Satellite receives, cable receivers, and Media Players etc), Audio Amplifiers, and many many more…

It was published by Proofpoint that ‘Thingbots” are sending spam.  Unfortunately it was taken by the media that “Thingbots” are the resulting robots from someone breaking into these devices and installing some software that sends spam, in a similar way that hackers try to trick people to download malicious software to their computers and laptops.  Whilst this is possible and undoubtably will happen in the near future, this is completely wrong at the present time.  Thingbots is a reference to the device being and ‘thing’ and being commanded to do something other than it’s designed purpose, regardless as to what that device or it’s purpose is.

For example, a ‘smart fridge’ is designed to keep food cool, keep track of the contents and alert the owner (maybe by email) if there is a problem with the fridge itself or with some of the contents (eg, like there being no milk left.)  The fridge is not designed evade security/anti-spam systems and to proxy or relay emails to a third party, but is is currently possible via a variety of devices.  How, you might ask, well I’ll get into that below, but first you might be asking why are these devices even connected to the internet?  Well it’s because people do not have a clue from a security perspective.  Neither the people that own the device nor (in a lot of cases) the designers.  The manufacturers are embedding computers into the devices and as a home appliance manufacturer (whether it be a fridge, a TV or other device) they are not experienced in IT Security, its not their job (yet) to be concerned with security, they want the functionality at the cheapest price.  To this end, they get people in their IT developer section (if they even have one, some just ask another supplier to provide them with the embedded software) straight from University, or school, with one very experienced manager most of which have no idea about security of the devices but they can code….  Worse, they make fatal management decisions giving the commands.. “Make it work!  Make it work quickly. Make it work cheaply!” When they have a developer that says, “Hey, what about security?” the answer comes back, don’t worry about it, it’s in the home, it’s safe behind the owners’ firewall, or “well put a password on it!”.. and so it begins..

Why this is bad..

Seems like a no brainer, the device is behind a firewall, it’s being NAT’d (Network Address Translation) so it’s not available on the Internet.. or is it?

Well most of these devices are running one of two Operating Systems, Windows or Linux, both operating system types have embedded versions, ‘Embedded Windows’ for Windows, or ‘BusyBox’ for Linux (amongst others, but that seems the most common.)

I’m not going to mention Embedded Windows here at all as I know nothing about it, and to be honest, at the moment, I don’t want to…  personal prejudice and all that..

Now in the case of Linux, the kernel itself is mostly secure and requires detailed and specialised knowledge to break into in the later versions.  The problem is they kernel is just part of the OS.  Linux is a UNIX variant and as such it relies on many applications as part of the OS for configuration, testing and usability. In the case of BusyBox these applications are often special versions that are cut down and trimmed as much as possible to save on space as embedded devices are usually limited on available memory, and they are all rolled into the same executable that operates as a multi-purpose tool “The Swiss Army knife of Embedded Linux” (it is given multiple names but only stored once in memory, and depending on the name you use to execute it, will depend on what function it performs.)  This is great, it makes devices very easy to build and makes it very versatile, for example, the Patriot Box Office, AC Ryan, Masscool and CinemaTube media players all run on the Realtek RTD1073 chipset BusyBox and a Linux kernel is an ideal OS for the device..  However, they are yet another example of the Internet of Things..  and in the case of the Masscool device a particularly good example of whats wrong…

My Masscool Media player…

I bought it in 2010 as it was one of the first HD capable media players that I could find with a good review, it also was one of the only ones with a HDMI port on the back..  Very quickly I found that it wouldn’t play some of the updated media formats so I went looking for firmware updates.  Non available, I relegated the box to the junk pile for a couple of years…  Recently I set up my games room and thought I’d get it out again and see if I could hack it to work with Plex as this is an XMBC fork that works well as a Media Server and has a DNLA server built in and the DNLA server can be hacked to transcode to various formats.

My first job was to see if I could get a firmware update, and went looking again.  What I found was two things, first, Masscool had not released a single update to the media player and in fact had stopped producing any Media Players.  Second, there is a sub-culture around the Patriot Box Office media players and firmware updates, and on the forums I found someone had posted the PBO unit was the same Chipset and board layout as a Masscool device. So I found a PBO firmware on their official site, downloaded it and started the firmware update to install it.. 30 seconds later I was told it was not made for this device and the update was aborted.  After this, the unit was a ‘brick’ .. it had killed it.  It could have been the boot code, it could have been something else.. don’t know but to many it would be dead and useless junk now, to me it was an opportunity to play.  I had no concerns about ‘bricking it’ as it was already bricked so I looked for hardware modifications and found with an old mobile phone sync cable I could interact with the Realtek chip directly and as such I could load just about any firmware I wanted on it.  The box within a few minutes was back operating in a really nice version of the Patriot Box Office software and playing all those newer media formats that I wanted to get in the first place.  Of course having “hacked” it I suddenly had all the details of what it’s running and how it could be easily attacked.  I had the default passwords, it was already listening for Telnet connections but I had been unable to get the information to logon to it until I hacked it… So I logged on and found that the Busybox installation had been complied with the ‘telnet’ option and therefore it is a device that can be a “ThingBot”.

More on how to exploit later…

My Dreambox DM500HD…

At home I have two Dreamboxes, a DM500HD and a DM800. both running ‘Enigma 2’, running OpenPLi 3.0 and OpenPLi 2.1 respectively… Running on an IBM STBx25xx Digital Set-Top Box Integrated Controller, and a 400 MHz Broadcom 7400 respectively.  Enigma 2 is another Busybox embedded device, however is significantly different from other media devices as it is being actively developed, supported, upgraded and patched.  All being said, by default DreamBoxes and the VU Plus, VU Duo devices and any other ‘Enigma 2’ device have a default password, login and open telnet port, in the case of Dreambox devices it is ‘root’ with a password of ‘dreambox’.  Worse, not only does it have telnet built in, it also has a web interface which by default is NOT password protected and allows you to get to all the system settings, including login security options and files with passwords…!

Further as this is a ‘feature rich’ device with whole store full of plugins and applications most of which are written in Python which is also embedded in the OS as an application.  This device is definitely a ‘ThingBot’, and it would be very easy to create a ‘Bot’ application that could be installed used to do a variety of other things – including compromise other devices.

So why would it be on the Internet?  Well simple (and in fact I put mine on the Internet) you can watch TV from anywhere in the world using the webinterface and if you had an Andriod or iOS based smart phone or tablet you can download an application to give you TV on the device anywhere in the world.. (and it works very well with Apple iPads, I use it when I visit the USA as all 600 channels over there seem to be full of rubbish and it works well on the hotel wireless!)

My D-Link DNS-325 Disk Arrays (Home NAS devices)…

I have two at home, one has two 2T drives, the other has two 3T drives, one is a backup for the other and is mass storage for me, it’s based on the 1.2 GHz Marvell® 88F6281 (Kirkwood) chipset..  I moved all my Music from my Macs to there, I moved all my Movies there, I moved all my photos there… and they are both full, so I bought a new Netgear this Christmas (more on that below.)  Now these devices are also Busybox based… what a surprise you may say.. no windows so far.. well that’s because I don’t buy windows anything, so this is all going to be something Linux based and mostly Busybox.. so anyhow, plugins available for it, eg, you can put a Database server (MySQL) on it, you can run a blog server on it, and of course you can put custom plugins on it by putting them in the root of the shared drive and rebooting it.. or using the web-interface. This includes adding an SSH server.. but why bother?  It has a telnet server built in, and you don’t even need to use a username and password (in fact you can’t even set it to have a username and password unless you are quite technical as any changes are automatically lost when you reboot as the password has to be saved to the boot flash and it is not using the provided tools.)

Again once you’re in you can telnet out elsewhere..

 My Netgear RN10400 NAS…

Built on the Marvell® Armada 370 1.2GHz chipset, it is also Linux though this is built on Debian Linux 7.1 (ReadyNAS version 6.x firmware) and unlike the others it’s secure(ish) by default.  You have to turn on shell access via the web-interface before you can get access to the shell.  However, unlike the others it has an extensive online plugin ‘store’ and you can install everything from a RADIUS server to a MySQL database server (by default with no credentials for root access) to your own blog and website.  However, like the Engima 2 systems it is also extensively supported by the manufacturer and community so firmware patches are forthcoming on a regular basis.

Would this be on the Internet, well yes, many of the applications are designed to give simple ‘SOHO’ services at low cost, so it is very likely this device will be placed on the Internet, either directly or by using the ‘DMZ’ or ‘Port Forward’ capabilities of most home routers.  Being a linux server under the hood, and not just a Busybox embedded device it also runs most if not all software that will run on a Debian Linux PC.. in fact it can even compile and run third party software that does not have packages capable of being installed on the NAS.

My Yamaha RX-A1030 A/V Receiver…

My latest toy, and very impressive one at that.. complete with it’s own gigabit network interface on the back.  As I only got it a month ago I haven’t had time to poke around with it above look at the web-interface, port-scan it, and install the ‘Tablet’ remote control application.  What I found was as follows:

michelle$ nmap -p1-65535 10.10.0.70

Starting Nmap 6.40 ( http://nmap.org ) at 2014-01-21 13:07 CET
Nmap scan report for 10.10.0.70
Host is up (0.0030s latency).
Not shown: 65529 closed ports
PORT STATE SERVICE
80/tcp open http
1040/tcp open netsaint
1900/tcp open upnp
8080/tcp open http-proxy
10200/tcp open unknown
50000/tcp open ibm-db2

Nmap done: 1 IP address (1 host up) scanned in 11.06 seconds

michelle$

Whoopsie!  More on that a little later, but I can tell you it has an iTunes streaming service , as well as ‘Net Radio’ and AirPlay.  (The important one to note here is the UPnP port.. more below)

My Thomson ‘Smart’ TV…

Another new addition (I got it at the same time as the Yamaha) and have not yet poked around with it, but it has it’s own App Store and Web Browser.. which is Embedded Opera and when you hit a webpage, with Flash content, it automatically attempts to download the flash plugin..  Do I really need to mention all the Flash vulnerabilities recently?  As for the scan it only has port 13000 open, which I have no idea about yet except it isn’t as webservice.

My LG Smart TV…

Another device (and one I haven’t poked around with) except I know it is embedded Linux and has various plugins available, including the Plex media client which historically has been written in Python.  LG have had a bad rap with their Smart TVs because of the fact they have been caught ‘calling home’, however like the Thomson and most other Smart TVs it is running embedded Linux.

So what about it…?

Well enough of the list of ‘ThingBots’ or devices that could be made to be ThingBots (Not even going to go into what on the LG Blu-Ray player and Melita HD Cable Set-top box – both Linux based)..  As you can see many of these devices are running an OS that is available in the mainstream and therefore compilers, software and plugins are available.  Some are securely setup by default, but most are not.

Some of these devices you can expect to see put on the Internet with out a security by the naive or experimenter, others you would not..  Or would you?

You see one of the problems with many of these devices is they all want to get access to the Internet, and even if you don’t give them access most of them are equipped to get access without you needing to know how.  Most people barely know how to setup a home router, so there is no way they would be able to configure port forwarding if needed, and certainly they would not know how to do that securely for protocols such as H.323 (video conferencing protocol that also used in MSN Messenger for example) so to get around this back in the late 1990s developers came up with UPnP aka Universal Plug and Play.

UPnP (Universal Plug and Play)…

This protocol/software is built into most routers, home-firewalls and devices.  It is a set of networking protocols that permits networked devices, such as personal computers, printers, Internet gateways, Wi-Fi access points and mobile devices to seamlessly discover each other’s presence on the network and establish functional network services for data sharing, communications, and entertainment. UPnP is intended primarily for residential networks without enterprise-class devices.

The UPnP technology is promoted by the UPnP Forum. The UPnP Forum is a computer industry initiative to enable simple and robust connectivity to stand-alone devices and personal computers from many different vendors. The Forum consists of over eight hundred vendors involved in everything from consumer electronics to network computing.

The concept of UPnP is an extension of plug-and-play, a technology for dynamically attaching devices directly to a computer, although UPnP is not directly related to the earlier plug-and-play technology. UPnP devices are “plug-and-play” in that when connected to a network they automatically establish working configurations with other devices.

The UPnP architecture allows device-to-device networking of personal computers, networked home appliancesconsumer electronics devices and wireless devices. It is a distributed, open architecture protocol based on established standards such as the Internet Protocol Suite (TCP/IP), HTTPXML, and SOAP. UPnP control points are devices which use UPnP protocols to control UPnP devices.

The UPnP architecture supports zero configuration networking. A UPnP compatible device from any vendor can dynamically join a network, obtain an IP address, announce its name, convey its capabilities upon request, and learn about the presence and capabilities of other devices. Dynamic Host Configuration Protocol (DHCP) and Domain Name System (DNS) servers are optional and are only used if they are available on the network. Devices can disconnect from the network automatically without leaving state information.

UPnP was published as a 73-part international standard, ISO/IEC 29341, in December, 2008.

What does this mean? well simply a UPnP device can tell the UPnP enabled firewall or router can open the ports without your knowledge – WITHOUT ANY AUTHENTICATION!

This is why back in 2002, Juniper Networks issued the following statement about their lack of support for UPnP in their devices:

SUMMARY:

Support for Universal Plug and Play (UPnP)

PROBLEM OR GOAL:

Universal Plug and Play Some chat programs are UPnP aware

SOLUTION:

NetScreen investigated UPnP, and have decided not to embrace this technology (as of mid 2002). Several factors went into this decision: a compromised host (say, with a trojan) could open the firewall entirely and permit other attacks and intrusions; all of the UPnP specs indicated that it is designed for the residential environment which is not NetScreen’s target market.

 

So how do they do it?

Follows is a series of images that show how it can be done.  I will *not* be showing how to manipulate a UPnP firewall remotely, I will also not be doing this remotely as my network is secure against things like UPnP especially as my border is protected with a Juniper SSG520.

First the D-Link DNS-325 NAS…

Spamming through a D-Link DNS-325

Now the Dreambox DM500HD (Remembering where the password is the default is ‘dreambox’ – which the malware “linux.darloz” is known to exploit):

Dreambox DM500HD proxy spam example.

The DM800 will work the same way, and in fact one could even install their own spamming program or proxy server using these set top boxes as this video will show:

 Conclusion…

Until both manufacturers and end users understand the security risks of devices on the Internet any device is is an attack vector for spammers and hackers, and as that device may not be monitored you could have the FBI, NSA, Interpol or Australian Federal Police (or other law enforcement) come knocking on your door to arrest and jail you for something you know nothing about…  Like trying to hack a nuclear reactor in Korea…

The final word…

I don’t have an Internet enabled fridge, and unless a manufacturer “donated” one to me I doubt I’ll ever have one (and no, I don’t need a new fridge, my Samsung 2009 side-by-side is perfect for my needs).. so no I can’t show you how to hack a fridge, just the same way I couldn’t show my employer.  All I know is, “yup there is at least one out there that is hacked” it might not be the fridge is hacked, it might be another device sharing the connection, but the spamming host is showing an Internet enabled fridge when querying it… one can only draw conclusions.

New Computer for Xmas? From Amazon? Watch out you might need a HazMat suit….!!

What is it with me, I seem to attract trouble at the moment, either that or I just don’t take s**t like others do…

So I’m not going to talk about the ripoff known as Ebay seller StuffUSell who sells stuff that they know doesn’t match the description… that’s Ebay and par for the course… No this is about someone you would think would know better… Amazon…!

Yeah the price of globalisation.. they’re so big in every country that when searching for stuff you don’t even see Ebay at the top of the list anymore, you just see 100’s of Amazon links leaving you little choice about where you can purchase items…  Even if they can’t/won’t deliver.

Many of you the readers know I live in Malta (Europe, not the town in the USA) it’s a small island in the middle of the Mediterranean sea and unfortunately getting stuff that is available to the rest of the world can be a task… and it’s not cheap (sometimes as much as double to RRP.)  For this reason I often use online services such as Amazon to get what I need at a reasonable price, paying extra for shipping.  Obviously because of Tax and VAT I prefer to order from Amazon EU/UK where ever possible.

So what is the subject about, you’re thinking.. well simple are you in Europe, are you thinking about ordering a computer/tablet for Christmas 2013…?  Well my advice is avoid Amazon at all cost as you might find yourself without what you are waiting for until after Christmas, with the excuse the Item you are ordering has a HAZMAT sticker on it and we can’t ship it to you…

Here’s the screenshot of the item I ordered over a week ago.. (click for hi-res version)

Thecus N4510UR 12TB NAS
Thecus N4510UR 12TB NAS

So as you can see ‘Ordered on 21 November 2013’ .. however lets take a look at ‘My Orders’ (click for hi-res)…

My Orders at Amazon
My Orders at Amazon

So I didn’t get any delivery, so I checked the order status, found it not yet dispatched so I got onto Customer Support (politely at first)… and after 24 hours I got this response:

 Hello,

I am writing to let you know about your order #202-2620275-0284318.

I have received an update from our fulfillment center stating that this item has been held up at JKPT this is because the item has been identified as having HAZMAT control on it and therefore can not be shipped to the address used as we can not ship this type of product to an overseas address.

I hope this helps you.

We loo forward seeing you again soon.

Warmest regards,

Ruban S.

It’s like ‘WFT?!?!?!’ HAZMAT?!?!??!  its a computer – it doesn’t even contain battery backup batteries!!

I got back to Customer support (again politly(ish) at first).. and couldn’t get a response as to what “JKPT” is … eventually I persuaded the Customer Support person to email me later what it meant, I got the following:

Hello,

I’m writing regarding your order #202-2620275-0284318.

Please be informed that, JKPT is a condition that an item is put into when we have no shipping method for the item due to HAZMAT regulations. It is usually to either an overseas address or a PO box address, locker or a parcel motel type place.

If we can be of further assistance, you can reply directly to this e-mail. You can reach us by chat or phone from this link:

http://www.amazon.co.uk/contact-us

Customer Service can be reached by phone and chat 7 days a week 06.00 to midnight, local UK time.

If you need to call us, we can be reached on Freephone (within the UK) 0800 496 1081. International customers can reach us on +44 207 084 7911.

We look forward to seeing you again soon.

Warmest regards,

Babuvignesh S.

At this point I got a little narcky and phoned them on the 0800 number for the UK and pointed out, that the address for delivery is a real address that they have delivered to previously, and that whilst they are correct ‘overseas’ pretty much everywhere in Europe could be classified as such if the origination point is Jersey as they previously indicated… and again the response:

 Hello,

Regarding your Order No: 202-2620275-0284318, we’ve got an update from our fulfilment team:

”  I’m sorry but this item has been held up at JKPT this is because the item has been identified as having HAZMAT control on it and therefore can not be shipped to the address used as we can not ship this type of product to an overseas address ”

Warmest regards,

Thangjam M

Then 24 hours later I get this:

Hello,

I’m sorry for the inconvenience caused to you with the restrictions to Malta.

I do understand your concern regarding the item being allowed to ship to Malta.

I’ve checked and can see that my colleague has already contacted appropriate department to investigate this issue.

As it is not yet possible to provide you with a resolution, we continue to work hard to provide an update and we still expect to be in contact with you on the date provided by my colleague, November 29, 2013.

Please accept my apologies for the inconvenience; we want to be sure to address this matter as thoroughly as possible.

If you don’t hear back from us by November 29, 2013, please contact us again by replying directly to this email.

I hope this helps. We look forward to seeing you again soon.

Warmest regards,

Imran A.

So the moral, if you want/need something quick (even as a business user buying business class items) don’t bother with Amazon, and certainly if it’s a computer or tablet (as tablets are computers).. go down the high street and buy over the counter – even if it costs more or takes your time, at least you’ll get it, and the shop is likely to be still there next time you need something!!

 

UPDATE [5th December 2013], this just in from Amazon:

Hello,

We’re writing about your Amazon.co.uk order 202-2620275-0284318 which included the following:

——————————————————

B009E0X9Q4

Thecus N4510UR 12TB (4 x 3TB) 4 Bay 1U Rackmount NAS with McAfee Antivirus Protection

——————————————————

Unfortunately, due to delivery restrictions on such items, we won’t be able to send you this item and have cancelled it from your order.  This is because this item contains flammable, pressurised, corrosive, environmentally hazardous or otherwise harmful substances classified as dangerous goods under the European Agreement concerning the International Carriage of Dangerous Goods by Air.

Although the amount of these substances in these products is usually quite limited, these products need to be transported in a certain way to ensure that they are handled with care and are therefore assigned to a specialist carrier.  Unfortunately this means that we can’t dispatch this to any destination outside of mainland UK.

We’re sorry for any inconvenience caused and hope to see you again soon.

Warmest regards

Customer Service Department

Amazon.co.uk

Please note: This e-mail was sent from a notification-only address that can’t accept incoming e-mail.  Please don’t reply to this message.

So there you have it, if you are buying a computer from Amazon (UK) and are not in the UK they cannot and will not ship the item – even if it is marked as being sold by Amazon Europe (S.a.r.L.) and even if it is marked as available for delivery to your country…

UPDATE 2:  Bit the bullet today, and went to one of the local computer stores and bought the non rackmount version of the NAS, found for €1123.00 (less than Amazon) I was able to get a 16TB version.. then I thought about it…  We’re on an island, everything is flown in.. but wait, Amazon said it was a HAZMAT marked item….!