Archive for December, 2009

Lazy of late

Wednesday, December 30th, 2009

What… its the festive season. Get off my back.

So much food to consume – such little time. Been doing too much to have any time to add something new here. Hit a few movies:
– Sherlock Holmes – pretty cool. Downey just seems to go from strength to strength. Interesting film with a decent plot and some nice cinematography. As expected from Guy Richie – some pretty cool stuff.
– Avatar 3D – Pretty dam surprised by this one. Not big on Titanic guy (can’t remember his name and google is so dam far away). Good story and amazing CGI. First 3D flick and at the start it was a bit dodgy – but pretty much improved the whole experience. Would do it again.

Otherwise – working when it ain’t a public holiday and chilling out. Picking up the basketball more often with Jemma which is great fun.

Anyhoo – will get back soon with something more interesting.

Riding the Graduate Program Wave

Thursday, December 17th, 2009

Having gone through the hard yards of a graduate program I have learnt a fair amount of the elements I believe constitute a strong graduate program. Obviously things can look very different from being a graduate to the graduate manager (or creator of the graduate program) but my experiences and experiences of those around me have led me to believe the following are really important – so important they are more of a must than a want.

– Rotation scheme – graduates should have the opportunity to work within a number of teams across a number of business functions. This has a number of advantages – the graduate has visibility over a number of business functions as opposed to a blinker view and the opportunity to grow experience/skills is greatly increased. Both of which provide a positive return for the company AND graduate.0

– Individual performance review – appraise people on personal performance – not on market levels. I have always been irked that a number of graduate programs reward graduates based on market increases (fluctuations) and not on personal performance or improvement. Should a graduate with 2 years industry experience who functions effectively as a full time engineer not be paid more than a green graduate straight out of a CompSci degree who takes time to adjust.

– Do not mis-represent the program or your company. Correctly managing the face of your graduate program and company will lead to longer term graduate placements. In short – if you advertise your program as a very different beast to what it is you will have a number of unhappy graduates who feel a sense of betrayal and chances are will leave.

– Provide a strong framework for graduates to communicate. Graduate communication provides an outlet for graduates to share information on experiences and work. In many cases this will tie in with the rotation scheme as graduates will share thoughts on teams and future career moves. Providing a strong framework for graduates allows them to feel part of the company culture and provides a level of graduate support. Isolating graduates reduces the effectiveness of a graduate program.

– Build a program around keeping graduates and developing them for the future. Graduate programs provide a positive opportunity to both graduates and the company. Graduates can form a strong foundation for the future of a company – if treated correctly and fostered through the early stages of their career these employees will become crucial to the future of the company. The graduate can build their talents/skills along with experience within a company that appreciates the importance of graduates – offering vendor training, support and career development.

– Do not view graduates as a cheap and temporary pack horse. In many IT firms graduates are simply used as “churn and burn” materials – as in they are brought in cheap, worked to the bone and burnt out then discarded as a new intake are accepted. This has 2 negative effects on the company – it develops a bad name within the graduate community and negatively effects future intakes lowering the level of quality of graduates applying.

Simply put – if you are looking at a number of a graduate positions trying to determine which is best for you – I would be looking at the above points and analysing which program offers the most important to you.

Iron Man and Warhammer – *swoon*.

Monday, December 14th, 2009
Iron Man Vs Warhammer

Iron Man Vs Warhammer

I am a sucker for a good bit of sci-fi.

Outsourcing – a harsh task master

Friday, December 11th, 2009

Only last week was I reflecting on my past IT exploits and came to the realisation I have always been placed within the outsourcing sector. Often I’ve thought how different things must be when your boss is…. your boss. Instead of a multiple customers who you rarely deal with on a face to face basis.

Life in the outsourcing industry can be tough – as a client you always want to maximise your bang for your buck. You want the best infrastructure and highest SLAs for the lowest cost. Enter outsourcing. Essentially you are alleviating your direct responsibility over a number of facets of your IT – whether its the infrastructure, application or support. This allows a client to drastically downsize any internal IT staff requirements – saving costs.

As a provider you need to deliver a low cost platform or product that is scalable and can be adopted easily. In many instances an outsourcing supplier must leverage a single product across multiple clients – this means costs are spread. The company can build specialised teams delivering the same service to multiple clients without needing to employ multiple teams for each client. I gleaned a number of tricks of the trade from my time.

Lean is the key – running close to the line means a product can be delivered at a market competitive rate. Delivering a low cost solution with potential to build year on is a great method. Sadly running lean usually means staffing levels are limited and your current engineering team really feels the stress.

Staff turn over is high – due to the ‘lean’ methodology and higher stress placed on resources turn over can be high if not managed correctly. The trick is to always provide reward and recognition. If a resource performs work that saves the company money – they should be rewarded. Similarly – if a resource can attract more billable work from a client hence increasing the revenue from the account recognition is in order. Further to this point, company culture should be given plenty of intention – investing greatly in the work force can have good returns for productivity and the moral of teams.

International work force – more and more within the industry companies over products that offer use of an international work force – reducing labour costs. Partly this comes at a cost – I’ve seen this negatively effect the moral of both the clients technical resources and the providers onshore teams. Productivity can also be negatively impacted if language/culture barriers are not addressed and managed. I find if resources understand why a business is shifting to using offshore labour and the reasons are valid resistance will be greatly reduced.

From my outsource providing experience I have felt one notable oddity from the clients – a strong expectation of service quality increase as cost decreases. In some instances this is understandable but in most cases as platform/product requirements increase (due to business growth) you would expect costs to also increase (preferably a slow increase). Clients seem to expect an outsourcing provider to become ‘more efficient’ at providing a service as time passes and as such – the cost should reduce. While I agree somewhat with becoming more efficient – it is impossible to pass on a saving if the client is growing other elements.

I still dream of my boss being my boss though.

Applejuice! For half price!

Friday, December 11th, 2009

ESX CPU Monitoring – a hefty task

Wednesday, December 2nd, 2009

As virtualisation becomes a larger and larger part of the industry – so does the need to manage and monitor your infrastructure correctly. Working in one of the pioneers of dedicated virtual hosting within Australia has its benefits. The team here has been tuning and improving a number of elements over time. Some things are only learnt by experience in the industry.

As the VM fleet grows – so does the supporting infrastructure. We have had a few problems recently monitoring the underlying ESX CPU usage. Via the VMWare soap API we have had pretty good visibility over the guest CPU/MEM usage and we have tied this in really well with Netsaint (I know I know netsaint != nagios). Since I have come along I’ve kept an eye on the REAL CPU values provided by the API – I always felt something was off as we were hitting 90% – 95% REAL CPU but when we manually checked host usage via VIC the figures were much lower.

I took on the task of determining what was going wrong – and more important how to get around it. This would reduce false alerts and increase sleep for the oncall engineer.

The first trick was to determine that the API figures were not good to use – I concluded this when I built a new ESX host for our infrastructure and during provisioning alerting paged out that we had hit 99% REAL CPU. I confirmed immediately this wasn’t the case.

Now – how to monitor the figure accurately. Initially I wanted to snmpwalk to a MIB that contained the physical CPU usage – sadly after much searching I couldn’t find anything I wanted. I had to turn to scripting a solution. I discovered a file in ESX 3.5 that gave me some ideas:
/proc/vmware/sched/idle
This provides a break down of idlesec Vs usedsec for each core – but wasn’t what I was after.

The solution came after finding another similar fix for nagios monitoring (irk see above – still using nagios). Essentially the trick was to pull the figure from ‘esxtop’ – as this would give the most realistic and up-to-date figure available. To make this even easier and fit in with current monitoring requirements the decision was made to make this value available via an snmpwalk. The solution comes together from three elements:

1) Script to rip esxtop apart (awk figures will change depending on core numbers):

#!/bin/sh
/usr/bin/esxtop -d 2 -n 1 | /bin/awk -F "," '{print $24}' | /usr/bin/tail -1 | /bin/sed -e 's;";;g'

2) An snmp walkable script that will pass the output to the snmpwalk command – in our case this was the form of a perl script – here are some pieces:

my $miboid = 'snmp-walk-numbers-here';
my $command = '/home/user/bin/cpu-usage.sh';
my $return="Error: Cannot execute";


my $msg;
if($msg = qx("$command")){
$return = $msg;
}


# Needs to fit in a UDP packet
$msg = substr($msg, 0, 1300);


print "$miboid\n";
print "string\n";
print "$msg";

3) snmpwalk configured to point to the above

So far this has served us well – we have compared graphing and found the ESXTOP method to have far less spikes which is great for our needs. Easy to set up. Low load on the system to provide the result.

Big Fish to Small Fry – Part 2

Tuesday, December 1st, 2009

It took about 3 days for my decisions to be sealed – my original IT firm came back to me with an amazing offer and my current employers made no effort to keep me – sadly the decision had been made that as a graduate – the team could live without me.

My last month was great fun – I worked hard till the last day closing a lot of outstanding tasks but otherwise stress was declining. I completed a week of oncall in my last month as a sign of good faith (which also provided some more cash). During my exit interview I highlighted a lot of points about the shortfalls of the graduate program (I’ll cover these in a later post) in the hope future graduates wouldn’t go through similar hurdles I had.

I’ve now been with the smaller IT firm nearly one month – due to my previous experience with them it was much easier coming back than headed to a new company – a number of the systems had remained the same as had the staff members. I knew what I was in for.

The first month has been chaotic at times – I’ve stepped up a few times to cover oncall responsibilities without much knowledge of the specifics – but overall the transition has been smooth. I’ve been surprised at how mamy skills I learnt first time round have remained and just needed a slight kick to get back in gear. During the first 2 weeks I was inspired to start a blog to document a few things – technical tid bits I found interesting, general life within the industry and a sort of week by week play of how I developed. Its taken me these 3 -4 weeks to find the free time and motivation to finally kick it off – but this is the end result.

I had planned this post to mainly highlight the differences between large corporate life compared to a grass roots IT firm – but the content seems to have grown and gone elsewhere – I’ll have to touch on this comparison later as I feel its really important to see the differences.

Big Fish to Small Fry

Tuesday, December 1st, 2009

I’ve gone through a significant change recently – in this case the one change has largely impacted me professionally and personally. Here is a re-cap on my experiences.

Rewind to 2007 where I was just starting to look at career options – I had a superb job at a small IT firm who were looking to keep me post-graduation – I had proved my worth with close to two years casual service. I had also taken the step and approached a number of large firms with thoughts toward joining a graduate program – the world was promised – travel, career options and experience. After much deliberation I decided to accept an offer with a large IT outsourcing firm – for a number of reasons:

  • I felt I had been spoilt in my current role. Things were too good – I needed to taste the sting of the IT industry to truly appreciate what I had.
  • I wanted to experience big corporate life with all its glorious bureaucracy (as I would soon learn).
  • I was concerned about career progression and training.
  • I was under the impression things would lead down a very different path.

A brief description of the company I decided to side with would go something like this – Multi-national IT firm specialising in delivering managed IT solutions/applications. Rough employee figures within Aus sit around 3500. The company serviced a number of Industries including financial services (which I was soon to join) and natural resources and mining.

I had joined a graduate program that highly advertised travel and career progression including strong industry training. Things started strong – I was placed within a Server Operations team managing Wintel systems across a number of clients. The team was great, the location not so – plenty of distance between myself and it. Within my first year I grew a lot – I had come from a background where the entire company could go to a bar and have a drink – in my new role I was part of a strong team but there were layers and layers of management – not to mention multitudes of other teams servicing the same accounts. I soon begun to understand this was the basis of the outsourcing industry – core teams specialising in a single subject matter would provide a stronger service to the client while reducing costs. Another great measure is using offshore resources – this was another learning experience for me – half my team was based in multiple locations across India. I now had a strong grasp of the systems, the client and how the corporate system worked. One of the senior engineer’s had taken me under his wing and guided me through a number of tight situations – we were continuously under the microscope from our clients – waiting for us to slip up and be hit with financial penalties (another *feature* of the outsourcing industry).

At times I hated my job – the stress placed upon us was in some cases extreme. I felt as a graduate I was being abused and was graduate all by name – I was working as a full time engineer, part of an intense oncall roster, assisting in large projects but being paid the wage of an inexperienced graduate – needless to say this was a constant sore point.

By the 18 month mark I had learned a lot and was functioning as an Engineer without the assistance of others. I had assisted in developing a patching framework for a major client that was largely a success – I had also effectively assisted in our jobs being offshored by training and assisting offshore resources. A new contract was signed with our teams major client that pushed for larger offshoring and our team could sense the end was nearing. We begun to look elsewhere.

I was lucky in this case – I had still maintained friendships with my previous colleagues and an opportunity arose for me to interview for a Support Engineer role – effectively what I had been doing before I left for the graduate position – but full time.

As I waited for the results of my interview I advised my multi-national employers I would potentially be leaving – the gears started to move in the back ground as I assumed they would….

work calls- To be continued!