“It works, don’t touch it!”

2nd May, 2009 - Posted by Clifford Wagner -

The “It works, don’t touch it!” philosophy has two faces. On one hand the mindset can prevent unnecessary upgrades and modifications that break a system, but on the other hand it can manifest as a habit of getting things just “good enough”. Being content with a system that (finally!) works after hours or days of poking at it, without actually knowing what you did to get it working is a malady that plagues the IT industry as a whole.

In my many years of working with VoIP providers, maintaining or fixing customer systems, and general tinkering on my part, I have found numerous instances of “poke it with a stick until it works” implementations that eventually become production systems. There are many indications of this mentality implemented throughout ITSPs and VoIP termination providers. ViaTalk.com’s recurring, intermittent DTMF issues, reported by numerous customers using their BYOD (bring your own device) service with an Asterisk server is a very good example.

I think a large part of this philosophy stems from ITSPs and service providers hiring a consultant to build their network for them, pushing the project along to get it done as fast as possible, and then diving in head-first without actually planning for ongoing support and maintenance of the system. Over the years I have been approached by many individuals and companies looking to build the “next Vonage” on a shoe-string budget. I’m sorry, but this just doesn’t work.

I may be in the minority here, but when I tackle a project I like to understand how it works, know all of the features and options, and what they actually do. I start by skimming the documentation and reading the headers, and select paragraphs about features I plan to implement or modify from the defaults. Then as I get deeper into the configuration I reference anything that I don’t understand, or is not immediately obvious. This takes time.

The first time I implement a piece of software, or build a new system using technology I have not used before, it takes me about four times as long as it “should”. I’m not worried about that it just means next time I have to implement that same system it will take me less time, and as time goes on, I will learn more about that system and will be able to take less time implementing it, as well as implement features and settings that most people don’t realize are available.

I may not remember the syntax of every command or option available months later, but I do remember the capabilities of the system. Talking to potential customers or even other developers about a project while knowing the capabilities of the system will save time and development costs. I have had many conversations where the proposed solution to an issue is to write more code that duplicates built-in functionality of the software we are using.

Things change. For example, every version of Asterisk is different, even within minor versions the available commands and functionality can change. Keeping up on this can be daunting and time consuming. Many companies just resign to running the older version of the software for as long as possible, accepting potential security vulnerabilities along with limiting their available functionality. Upgrades should be planned, tested, and implemented on a rigorous schedule.

Just like backups, a well-planned upgrade schedule is essential to maintaining a healthy and efficient system. The cost of upgrades, both hardware and software, should be planned for and incorporated into your operating costs. With increased demand for your product or service, and with evolving software, especially open-source software, your business model should include the ongoing costs of not only adding capacity, but consistent software maintenance and upgrades.

These two concepts go hand-in-hand. Capacity upgrades are a great opportunity for software upgrades. When you add a new server, install the latest stable release of the software you are using. Test this platform thoroughly and put your own company account on this new system, or move a few clients over to the new system to test it out. Once you are satisfied with the performance and upgraded functionality of the new system start to migrate your existing customers over one server at a time. As the old servers are freed up test the hardware, replace any failing fans, power supplies, or other components, then reinstall them with the new software and rotate through the servers until everyone has been upgraded.

Rather than “It works, don’t touch it!” being the motto of your IT staff, try “It works, let’s start on version 2.0!” instead. Your first implementation is going to have issues. You will learn better ways of implementing things as you go. And to be honest, it is not practical to try and get a system “perfect” the first time around. You can launch with a “beta” program, version 1.0, whatever you want to call it. Get the customers, get revenue coming in, and then build a better system. A lot of the time you won’t even know all of the issues or shortcomings of your system until you have live customers on it, breaking it in ways you never could have imagined.