Simple Effective Spam Blocking with Mikrotik

Let’s face it – reading every email that leaves your network is a daunting task.   We love the Spam Experts Folks and while we get ZERO dollars for referring them to you -  I still think they are worth looking at – much better than swimming in the  waters with a Barracuda – thats for sure !!!

BUT – what if you cannot afford Spam Experts – OR – don’t really run your own mail server – and simply are a small WISP that wants to ensure outgoing email is blocked ?

Welcome to the Mangle Firewall built right within the Mikrotik system –   this powerful firewall can help you along.

Simply limit the number of outbound SMTP connections that are leaving your network.   A great rule of thumb is to block anymore than 5 at a time.

Even a fairly decent size office generally does not send that much in a second –   (and hey there is always whitelisting ! )

If you have not been introduced to the Mikrotik Wiki – now is the time :-)







Another MiktoTik Certification Completed

Today I passed the wireless exam for Mikrotik which now certifies that I can work and understand not only Wireless Theory but also the Tik Interface.

This is the 4th certification for Mikrotik I have:

  • MTCNA – MikroTik Certified Network Associate
  • MTCRE – MikroTik Certified Routing Engineer
  • MTCWE – MikroTik Certified Wireless Engineer
  • MTCTCE – MikroTik Certified Traffic Control Engineer



Creating a Better Web Server Farm for HOAC

I love the guys at CloudFlare.   They know what they are doing – and I follow them fairly closely.    Our latest network is built on something they started.

At Hop Off A Cloud, we’re always looking for ways to eliminate bottlenecks…  Sometimes those bottlenecks are employees (training normally takes care of those kinds of issues and while we strive for perfection – something we can be sure of – we are not personally there yet…)

Hop Off A Cloud is only capable to deal with the large DOS and other network issues we see because we have partnered with folks like Agile Networks, CloudFlare, Hurricane Electric and other data center partners like DataCenter.BZ and OVH to build a resiliant network infrastructure.   Heck – we even utilize the work of a Local (small but powerful) Microwave Backhaul company  (www.CountryConnections.net ) to provide us an extra layer in our MPLS network just in case we need to get in another way to our network for management (and even transit in some cases).

Hey Glenn – Explain your Datacenter Rack

Sure, I would be happy to.    First off  Let’s start with a thank you to a friend that now works with SunGard in Philadelphia – BOB – you were an excellent resource in getting us some excellent RACKS!!!

Anyhow – A rack of equipment in  Hop Off A Cloud’s network has three core components: Servers, Switches and Routers.

In our own Datacenter, as well as those we also collocate within – We own and install all our own equipment because it’s impossible to have the flexibility and efficiency you need to do what we do running on someone else’s gear.    Over time, we’ve adjusted the specs of the gear we use based on the needs of our network and what we are able to cost effectively source from vendors.   We do however lease some backup servers within OVH’s datacenter however those are not part of this setup and may be described at a later date.

Most of the equipment in our network today is based on CloudFlare’s Generation 3 (G3) platform.    Focusing just on the network connectivity for our own  gear, our routers have multiple 10Gbps ports which connect out to the Internet as well as in to our switches. Our switches have a handful of 10Gbps ports that we use to connect to our routers and then 24 1Gbps ports that connect to the servers. Finally, our servers have 6 1Gbps ports, two on the motherboard (using Intel’s chipset) and four on an Intel PCI network card.    I had someone at a recent MikroTik training session tell me they thought this was overkill.   I told them I think in a year or two this will prove to be to little.

Unlike many of our competitors – EVERY CORE SERVER IN OUR DATACENTERS are able to run any of the Key Functions required by our customers (with the exception of our SAN Systems (as they are of course Storage and are dedicated to those functions specifically- but more on storage and DRDB later).  We believe, it is important to have our Core Systems be able to take on any service in an instant for any other system in the case that another systems utilization is going over 50%, or in the case of failure.   This means at any time – any CORE SYSTEM is able to Serve a Website, be a Caching Server, perform Logging, CLEA requests as well as a few other important services.     The power of providing MemCacheD on Every System – or even our Web Caching allows for us to simply increase the amount of storage available for such services by simply adding additional SSD drives to a chassis.     Since everything runs in our Highly Customized Atmosphere (called a HOAC Balloon) we are able to down a Balloon at any time to add RAM, CPU, Storage or do server maintenance with ZERO downtime for our end uses.   This also allows us to scale linear – where many of our competitors are stuck trying to figure out how to grow.

The Challenge – much like what Cloudflare – who we based our Balloon Philosophy on, is that this setup requires a heck of a backend network just to talk to itself !    Our DRDB chat between systems is pushing at some times close to 2.5gbps and a simple 1GB Fiber connection is not enough.
Using some tricks taught to us by Mike Delp and re-enforced by Dennis Burgess from Link Technologies in a few conversations, we upgraded the Network Cards to allow for Queuing as well as Bonding.     Mind you – it does take quite a bit of expensive hardware to BOND Four 1GBPS Fiber and two 1GBPS Copper ports together into a switch fabric – but we found the challenge to be both exciting and rewarding.
We then put this stinker to a test – and invited a large number of folks to test the switch fabric and initiate high levels of traffic just so we would know “does this work”    The answer is an outstanding YES.

Our next step is to upgrade each system to a 10GBPS backbone – and when that project begins we will most likely document the setup a bit more to help others build on what we have done.

For a customers individual server, or rack – we deploy Mikrotik 1100AH2 (soon to be CloudCore when they become stable and the hardware is no longer in Release candidate) or Dell 1950 Servers (which allow the customer to choose from Mikrotik x86, pfSense, MonoWall or even vYatta.    Our personal preference however for simplicity is the Mikrotik RouterOS unless the customer requires some other additional services- then most often we provide them with pFSense.   Chris who helps lead that project lives roughly 2 -3 hours from our Core Network and has even made a visit to us to help a Wireless ISP out of a Jam on July 4th.     He got to ride up the elevator of the customer’s grain elevator to watch the fireworks 150′ in the air however… (not all bad  I guess)

In any event – hopefully this helps you understand the Rack in the HOAC datacenter just a little bit more… and explains what we mean when we say -

“when your provider is in the middle of the perfect storm – come Hop Off A Cloud – the Weather is Better Over Here ! ”



Dont Choose that Vendor IF…

#1 – You have to talk yourself into using them!!! If you have to convince yourself it’s most likely going to turn out really bad!

#2 – You are more obsessed with finding a solution than you are finding the right solution. Take your time…putting the wrong solution or vendor in place  will cost you more in the long run!

#3 – They are not listening – a vendor who will not listen will become a lid to your organization – its growth and most possibly your own job.

#4 – You feel sorry for them – just because they are not doing well does not mean you are obligated to help their business out by hurting your own.

#5 – The thought of dealing with them makes you want to jump out of a window! Seriously, chemistry is essential to conquering the challenges that will be placed in front of you and your organization.   Why place yourself in a situation having to work with people you secretly wish you could get away from?

#6 – You can’t get past that “thing in your gut!” No matter how great their reputation is…no matter how great sales pitch is…you have to go with your gut.

#7 – They spend the majority of their blaming their VAR partners for lack of success of projects they have been part of.. If they blame them then it will be a matter of time before they are  blaming you.


Mac OS X Lion + USB = Bootable Stick

Recently I needed to re-intall MAC OS x Lion for some reason however Apple does not make this all to easy – especially since it comes from the Mac App store now.

In any event – here is how I got around it:


After you purchase Lion from the Mac App Store – let it download.

Once it has – browse on over to the /Applications folder – and then right click (in the viewer) on the Lion X app.


Choose – Show Package Contents – and then browse into the SharedSupport folder.

From here – you will see an “InstallESD.dmg” file.

Now is the cool part.

Open up your disk utility.

Choose your USB Stick (needs to be larger than 4GB )

highlight the usb stick and then choose restore.


The aforementioned dmg file will be the source – and the destination is the USB stick.

Keep in mind -your USB stick will be wiped during the process.


After about 5-20 minutes depending upon the speed of your system – your USB stick will be loaded and ready to go.




If you want to perform a clean install of Lion, or throw it on a Hackintosh, you’re going to need a physical disc—none of this Mac App Store craziness. Luckily, there’s an image file hidden inside the Lion installer package that you can burn do a DVD for just such an emergency. Obviously, you’ll need to wait until Lion is released, but once it is, just:

  1. Download Lion from the Mac App Store. The installer should show up in your Applications folder.
  2. Right-click on the installer and hit “Show Package Contents”. Navigate to Contents > SharedSupport and look for a file called “InstallESD.dmg”.
  3. Right-click on the DMG and burn it with Disk Utility, or use a program like Burn. When you’re done, you should be able to install it on a computer from a DVD like you normally would.

cent-os basics

Today I had someone reach out to me via chat and ask a basic question…

My VPS is based on CentOS 6 but it does not even have wget – how can I install the development tools?


Yum is a great friend to any CentOS system administrator –  I suggest beginning with the following few commands


1.  yum update -y

once complete add

2.  yum install nano screen wget -y

once complete then do

3.  yum groupinstall ‘Base’ -y

4.  yum groupinstall ‘Development Tools’ -y

5.  yum groupinstall ‘Perl Support’ -y

If you wish to know the other groupinstall groups available – simply use this command

yum grouplist



Use SSH to bypass a firewall OR Browse Securely

With applications on public wifi points like FireSheep, and corporate firewalls blocking access to most locations on the internet of interest – the question comes up quite regularly ” How can I keep secure while on public wifi” or ” How can I bypass my companies firewall.”

This is actually very simple – If you have linux running at home – or have a webserver with access on the internet – follow this brief tutorial.

ssh -C2qTnN -D 8080 username@remote_machine.com

Type this into your local putty or terminal console.

To explain what is happening – in short what we are doing is taking Port 8080 on your local system – and forcing all traffic on that port to your remote ssh server using the username chosen. I STRONGLY URGE YOU NOT TO USE THE ROOT USER !

The options in order we are using are: Compression, SSH2 (for security) Quite, Force, Pseudo-TTY Allocation, Redirection stdin away rom /dev/null – and finally placing the ssh client into “MASTER” for connection sharing.

The next step is quite simple – just go into your browser(s) and set them to use a proxy server – in this case it is simply localhost using port 8080.

If you find you need some tweaking for speed – please let me know as there are a ton of options we can help set in your browser’s about:config section.


Greater Love – a 9/11 memory

To many when we see the American Flag we see it as a National symbol – however on 9/11 and the days following while working at ground zero as a Paramedic involved with the search and rescue the flag became something more. Each time a fellow brother in Fire or Police were pulled from the rubble – they were placed in a stokes basket and the basket was draped with an American Flag.

Even now 10 years later each time I see the American Flag, be it as a sticker on the side of a police vehicle, waiving at the Ford dealership or even on someones shirt – I remember the many bodies I had the privileged to help carry to peace. For years I had worked in EMS and Fire – I have had to deal with many fatalities from innocent children to gang members who were in their own version of war – but 9/11 and the months following to this day still haunt my soul.

While driving in to the city our ambulance was rocked by people holding signs – and stopping us to hand us water, hugs – prayers and sharing tears – I had a rookie on board who was driving. He thought it was best to use the siren to get the people out of the way so we could get through the city to our staging area. I calmly reached over and turned the siren off – as I looked at the peoples faces I realized something – We, coming in from Southern New Jersey, were literally the only sign of hope in a city in despair -masked in acres of rubble where roads, cars and a concrete haze hid the scorched and scared faces that were looking at us empty – left without meaning. The siren only added to the shock – and our presence was supposed to be calming.

If you have ever been to New York – you will learn that even at the latest hour of the day – there are cars and buses – taxis and limo’s driving people all over the city – it was an eerie feeling as we drew closer to ground zero – We saw a sea of people all walking, some so drained mentally and emotionally you could see they were using every last ounce of their strength to drag themselves just to the next step. – and suddenly it was all quite. Hardly a sound. When we arrived just one block down from the collapse of the first tower – on Vesey street the realization of the devastation set in.

Pallets and Pallets of water and other items were just sitting in the street. We were assigned to go through the many surrounding buildings to search for survivors – sadly we did not find many. I can remember a secondary collapse in which I had to dive under a fire truck for safety – all the while thinking if the building is coming down there is no way this truck is going to save me. The first rule any EMS, Firefighter or Police officer learns is Scene Safety – look out for potential hazards. The truth is – the members of the FDNY, NYPD and the Port Authority Police knew they were running in to the most dangerous situation. As bodies were descending at the speed of gravity they were climbing as quickly as possible. Scene safety was not paramount upon arrival for these Hero’s but rather getting in – and saving as many lives as possible – even knowing that their own was not in jeopardy but already over.

A few mornings later I was exhausted and walking back from the Cafeteria – a fancy name for what really was a boat which until the terrorist attack was used to shuttle tourists around the water ways so they could “tour the city” and I found a priest ripping off his clothes screaming at God asking Him Why and Where. Where are you now God… he demanded to know. I walked over to help what obviously any EMT trained would could identify as a man who was mentally breaking down and in need of help.

I asked him his name – and he answered – I asked him why he doubted God’s presence and he pulled me by my collar over to a blue suv and read to me the note out loud -

“Dear NYPD – I am a nurse – Please don’t tow my car – AND if you find this please tell my husband and my children that I love them and if I do not make it let them know I am fulfilling John 15:13″

He demanded to know from me how God could let this happen. I sat with him for what seemed like hours – but was really only just a few minutes and felt the warm glow of the sun coming up over the horizon – as I looked up I saw a cross in the distance – this cross had tons of workers underneath it digging and working, struggling to find any one who could have possibly survived. I answered the priest and said – until now I never really could answer your question – or even that question for myself long before what happened yesterday but I do know one thing – Even as I walk through this Valley of Death the Lord is near me – He comforts Me and in the end I will lie down in Green Pastures. I pointed to the Cross – and he asked me if I would pray for him as he had “lost the words to speak to God”, I began with the Lord’s prayer – which he recited with me.

I never was able to find that priest again – even after searching – but I will never forget what he told me ” Glenn, I have a renewed faith and even in this tragedy I know that it is a promise SURELY goodness and mercy – the Love of our Father will follow us all the days of our lives – let me go and share the good news.”

With that I returned to the gator crew I was assigned to picking through the rubble.

For those who are struggling – I want to simply remind you also of that promise God has made us – He will Never Leave us or Forsake us – even when we are walking through the valley of the shadow of Death – His Rod and His staff are there to comfort us.

While driving back home I stopped at a small Baptist church in Hoboken NJ on the other side of the tunnel – ran to find a bible and opened it to John 15:13 I found something I learned as a child and had forgotten…

From the over 350 EMS, Fire and Police officers – as well as the countless other volunteers including building security and the other nameless who helped, from the folks who downed the plane in Shanksville PA, and from that nurse who most assuredly was killed we as a nation learned a lesson that most have already forgotten – except perhaps those who are still serving daily putting their lives in front of harms way day by day: “Greater love hath no man than this, that a man lay down his life for his friends.”


My Mind is a dump terminal… er……

The meaning of the cloud is somewhat…well…. Cloudy…


how to protect your MacBook Pro

This week while at the Greater NJ Annual Conference of the United Methodist Church something sad happened at the very end.

One of the key musicians, Mark’s laptop was stolen.   He had just purchased the laptop just a few days earlier (Tuesday I believe he said.)  This happened from what we can tell right at the end of the conference – and of course there were vendors and conference folks all over the place helping tear down – thus who knows where it is…  We are hoping it was just collected up by someone who mistook it for theirs…  (alas i think we are dreaming… anyhow)

I remember a posting from @ClifGuy who had his laptop stolen right out of another #CITRT members car in Florida – and at that time Windows PC security options were discussed – but not Apple/MAC.

So – here you go MAC Fans:

  1. Never EVER leave your MacBook unattended, not even briefly. Be aware of your laptop, as you would a purse, in airports, hotel rooms, restaurants, libraries, dorm rooms, and even @ a Church gathering.
  2. Always Back up your data regularly. You should do this anyway, in case of hardware failure or software bugs, but it is also critical in case or loss of theft.  Mac makes this easy with the Time Machine option – however a good online backup application would be wise to use
  3. Use a security cable. Simply put – its like a bicycle chain for your machine.   They simply can’t cut it and walk off w/o someone noticing…  we would hope… and even if they could – it stops the simple opportunist from jacking your mac
  4. Use motion sensors, either with hardware (Targus DEFCON, MicroSaver Alarmed Lock) or software (TheftSensor) .
  5. Be less conspicuous.  Carry your MacBook in a backpack instead of a laptop case…  people might not know whats in there.
  6. Choose appropriate passwords and make use of them. Don’t use guessable passwords. Log out when not using your MacBook.
  7. Set a firmware password. Use EFI (Intel) or Open Firmware (PPC) to set a password that prevents booting from another disk.
  8. Use encryption. Consider which data on your MacBook is most sensitive and take care to protect it. Use Apple’s FileVault feature on your home directory or utilize the Disk Utility or DropDMG for convenience.
  9. Install anti-theft software. Use a software package that “phones home” on the Internet or over a phone line (Undercover, LoJack for Laptops).
  10. Have separate logins. You might have one login for your routine documents but for important secure files – use FileVault on another login. By having a third login, with no password, you invite a thief to log in that way, making it more likely that they will connect to the Internet and activate the anti-theft software.
  11. Recordkeeping. Record your MacBook serial number and keep this information on paper somewhere- or even better take pictures and upload them to somewhere online you can always get to – and make sure that is password protected of course.   Register your purchase. Keep track of what personal information you have on your MacBook, so you know what you’ve lost, what passwords to change, etc.
  12. Insurance. Check if loss or theft of your MacBook is already covered under an insurance policy you have – such as your credit card.  If not, get renter’s insurance, a rider on a homeowner’s policy, or some other type of coverage- and make sure the deductible is low enough for it not to matter if it gets lost.
  13. Be sure to Avoid viruses/adware/spyware. Install all security updates to Mac OS X or other software.  At present you don’t need any special software for Macs – I suggest using the ClamAV for MAC located here:  http://www.clamxav.com/index.php .
  14. Keep your personal computer personal.In other words – NEVER LEND IT OUT.   And if you need to – use a separate login for them – thus insuring the security on your system.

Any other suggestions – please feel free to add them .