7/9/10

Agile Development: The quickstart guide to doing it right

Lots of places don’t do agile right, in fact most do it wrong. There are a number of factors why, chief among them I believe is that the percentage of people who have worked somewhere that nailed it completely right is so low that there aren’t enough to go around and lead by example. So here are a few of the tips I’ve been given or learnt over the years, credit has to be given to all the wonderful places I’ve worked over the years. The ones that have done it well, and the ones that have done it poorly… you can learn just as much from reflecting on where it all went wrong.

Back to basics
Are you already using agile, or are you planning on implementing it in your company? If you’re reading this chances are something isn’t working as well as you’d like and so it’s time to reset. First thing to do is throw out your preconceptions on agile, ignore almost everything you’ve read in practitioner manuals, because at it’s core it’s really quite simple. Most problems stem from people picking only certain features of agile or trying to let a tool guide the process for them.

Refocus on the manifesto
It’s worth going back and looking at the agile manifesto and then questioning each aspect of your process to ensure it is giving priority to the values on the left side. If it’s not, remove it.

Throw out the tools you’re using
Scrumworks? Urgh! VersionOne? Atlassian? They’re all cut from a similar cloth. If you’re not already doing agile correctly, they will just get in the way. So until you’re up and running, get them as far away as possible.

For planning, all you need are some index cards and a pen. If that doesn’t offer you enough protection for your DRP or wont work because you’ve got a distributed team then use a wiki. Again, avoid at all costs anything more complicated until you’ve got it all working.

Planning
The great thing about stripping it back to basics is that it re-aligns focus on the things that are important, you’re not distracted about calculating backlog points and predicting velocity. Instead you are collaborating with your customer to understand what they will consider working software.

You need to become the customer
I mentioned in my last post, nobody cares what tools you use, your customer is trusting you to make the right decisions to do the best job. You can only really do that when you understand the requirements completely, a full and deep appreciation for why the work is being done and not simply what needs to be implemented.

Your customer is unlikely to be a software developer so their scope, as broad as it is will be, is limited by their experience. They also can’t possibly know upfront all the questions you want answered and all the seemingly irrelevant detail that would feed in to giving the best user experience possible. Expecting them to bridge this gap is completely unrealistic, instead the gap needs to be bridged by having the developers move towards the customer. The developer needs to be completely bought in to the product, understand the problems it solving, and how they’d best be solved.

This is precisely the reason why tools like cucumber don’t work, they are bridging that gap from the wrong direction. And if we go back to the agile manifesto, that approach carries the potential to skew the priority back towards contract negotiation rather that customer collaboration.

Customers don’t write the stories
And neither do business analysts, development managers, or project managers. I’d go so far to say that if you want to be “agile” and you’ve got a team of BAs then you should fire them all. Anybody getting between the developer(s) and the customer is stopping the developer from becoming the customer. Crucial detail will be missed and misplaced assumptions will go unchallenged. Sure, you’ll still get a product out the door but it will either be what was agreed (which is often different to what is actually wanted) or it will not be as awesome as it could have been.

The people that get between developer and customer have the best of intentions, they’re trying to let developers focus on writing code. I’ll say it again though, they’re job isn’t to write code it’s to deliver a solution. It’s a false economy to think you’re saving the developer time by taking on the requirements analysis on their behalf, it will take longer for them to appreciate 3rd hand the requirements than the half day it would take to know them first hand and write them up themselves.

Pull up your sleeves, get out a set of index cards, some pens, and some A4 paper. No laptops, no ipads, no technology. Force the customer to explain the issue without computer aids as though you know nothing. Each of you draw up wireframes, compare the differences between them to see what assumptions have been made, write the card together. Make it a tactile experience.

The result should be a concise description or the business problem, and the end-user benefit you’re trying to deliver. Never more than a couple of sentences (maybe a few footnotes for pertinent implementation details that can’t be neglected).

The most important stories get done first
This part of it will be difficult, if not impossible, for the developer to make a judgement call on. It is often very difficult even for the customer, but it has to be done. That being said, it shouldn’t be as difficult as many people make it. If you’re working on a new app that’s not yet been released I’ve got a diagram to help you evaluate priorities. It’s also worth keeping a focus on getting the smallest usable feature set out the door as soon as possible. Eric Ries said a good way to get to that point is to outline the minimum features you think you need before you can release it publicly, and then you can probably reduce it by 90%. You always need much, much, much less than you ever expect.

If it’s an existing app and you’re having problems prioritising it might be worth speaking to some users. Just make sure you keep focus on doing the smallest and simplest thing possible to meet the requirement.

A good way to gauge if story is really the next most important thing to get completed is to double the estimate, if that changes the priority in the customer’s eyes it wasn’t as important as they thought.

If you’re not doing the work, you don’t get to estimate it
It took me a long time to appreciate this as a manager, I was often guilty of sizing up tasks based on how long I thought it would take me to complete it. Then someone else picks it up and I wonder why it took so long. Truth is it would always take me longer too but that is too easily forgotten, do you really have a load factor of 1? I thought not.

How easy or difficult a task is has so many variables: familiarity with the code base, how recently you’ve solved a similar problem, access to an existing solution, etc. and each of those come from a fairly personal experience. Sit down, with a pair to rationalise the decision if you like, and come to a conclusion you’re happy with. The important thing is that you are consistently optimistic/pessimistic on your approach, it’s less important to get the estimate right than it is to be wrong on a relatively consistent basis. Everything averages out very quickly when it comes to the planning.

Don’t mention time
Do everything you can to avoid estimating in hours or days. No matter how much you explain things like load factor when you see a story with “2 days” or “8 hours” written next to it people can’t help but think that is how long the task will take. So you can’t be entirely shocked when 3 days in they ask how that task that was meant to take 2 days is going. Avoid the awkward situations completely by not setting the expectation in the first place.

Estimate in points, or jelly beans, or anything other than hours or something that can be tracked by a watch or calendar.

Estimates aren’t guesses, but they’re not accurate either
Don’t take a look at a story, close your eyes, and come up with a number. Take the time to analyse in detail what is required, what systems you’ll need to talk to, read the 3rd party API/integration docs. Spend a whole day or more if you have to and write up a detailed list of tasks in the order you expect to tackle them, including any decisions on implementation you may come to, and put them at the bottom of your story card.

Time invested here pays for itself two-fold later in the iteration by maintaining focus on the minimum set of tasks that need to be completed and setting your direction each step of the way. Plus it gives you confidence that the task is achievable and unlikely to blowout.

When it comes to putting a number (points, jelly bean, whatever) on an estimate I now always advocate a 3 point system. 1 point roughly equates to a half day of effort (ssshhh, don’t tell the customer ;), 2 points a whole day, 3 points 2 days. The reason being that it’s too difficult to consistently estimate small tasks, if you estimate an hour for something and you get snagged and it takes you the first half of a day (not unusual) you’ve blown the estimate by 400%. At the other end of the spectrum you’ve got two problems, either a story that isn’t really implementing the most simple solution possible or a broad scope that it’s difficult to really sit down and estimate in the detail required to be confident in it.

If the tasks are obviously much smaller than 1 point, I’ll try and group a couple of similar smaller tasks into a single 1 point task. If a task is bigger than 3 points, I’ll split it into separate deliverable parts. “But it’s really a 5 point task, everything has to be delivered at once!”, Bollocks!

Estimates for work that isn’t scheduled are worthless
They might be useful for getting a high-level idea of when some story might be delivered, maybe, if it’s ever actually scheduled. Remember one of the core tenements of agile is responding to change, and nothing impacts priorities like actually giving software to users. So it’s not uncommon for that really-urgent-we’ve-gotta-have-it next feature to get bumped for a glaring hole a bunch of users pointed out. And then bumped again for something else. It comes back to expectation management again, and putting an estimate next to it adds a misplaced finality to where the story sits in the priority queue.

The other issue is that each iteration of development feeds back into next. Some previously completed work may make a later story easier to implement, or it may actually highlight that something is more difficult than originally expected. Either way there are enough factors that can completely invalidate whatever estimates you come up with.

Execution
Once the planning is done, it is down to actually starting the work. It’s not always a matter of ticking boxes off the work list though, the most effective teams I’ve worked in have shared some common traits.

Start mid-week
Few people love a Monday morning and getting motivated can sometimes be an issue. If you compound that difficulty by expecting everyone to be in the right mindset to sit down and write stories, well it doesn’t always work out for the best. There is no real long-term cost associated with moving the start of the iterations from a Monday to a Wednesday, but allowing me to be productive first thing after a hazy weekend by allowing me to look at a list of work and just crack on with it is a huge win.

Have a fixed iteration length
I prefer to work with 2 week iterations, 1 week feels too short and 4 too long.

There is a comfort that comes from having some order to the world, and the importance of the general psyche and morale of a team is often underestimated. On those times that a story does take longer than expected, it’s nice to know that you’ve still got a week or so to make it up. It’s also great to have a regular sense of completion in predictable intervals, the dreaded risk of being stuck on the same task that is going to take months never materialises (partly because you never schedule a story with more than 2 days of effort, right?). There is a lot of subtlety in the impact this predictability has on developers which makes it hard to pinpoint all the benefits, but when combined they all make for a happier environment.

From a management perspective, it’s great to have some clarity on when things are going to be delivered. And I’m not just talking about for the next fortnight, but you know on a specific day each and every fortnight something is going to be delivered. So does the rest of the team. It makes planning much easier, people know the most convenient windows to take holidays, and which days to stay home are particularly inconvenient because you’re planning with the customer.

Developer attention is a premium
Jason Fried from 37signals has a great video explaining why you can’t work at work that I’d recommend watching. When you’re deep in the coding zone nothing is more disrupting to your flow than someone coming and interrupting you to ask you for opinion on something completely unrelated. That 10 minute interruption can take 30 minutes to recover from fully to get you back into the same state. If they happen 2-3 times a day that is up to 25% of the workday productivity lost to distractions.

And as Jason mentions in the video the very action is basically a way of saying “Hey, my immediate needs are more important than whatever you’re focussing on, give me attention”. It’s selfish and unhealthy for productivity. Remove phones. Email people if you have to, but don’t expect a response that day. Inter-team or important but non-urgent questions should be posed via an instant messenger, chat program, or something that doesn’t demand immediate interruption (like phones and meetings do). What about things that are urgent? Well that needs to be assessed on a case-by-case basis, but the truth is truly urgent things happen very very rarely. Almost everything can wait at least 4 hours.

It’s here that a good development manager can work wonderfully, acting as bouncer and preventing direct access to developers while they’re busy coding and filtering the urgency of all requests.

Pair Programming
A fairly contentious part of agile is pair programming. When it works well, it’s hugely efficient and carries lots of additional benefits (higher quality code, quicker delivery, lower documentation requirements, less “single developer” business risk, etc.) but when it’s implemented poorly it’s just a waste of a resource. In most places I’ve worked it sadly falls into the latter camp, and it’s because people are just paying lip-service to the practice and working it as mentor/tutor type role rather than a pair actively writing code together.

That’s not to say it doesn’t also offer great potential as an approach to training, the important thing is both participants need to be as involved and active in writing the code.

Owning the workspace
The biggest problem when it doesn’t work is normally down to the setup of the workspace. Generally Person B brings their chair over to share Person A’s desk, and Person A stays almost exactly where they were to begin with. It’s guaranteed to fail. Here’s what you need to do:

•Share a single monitor, you both need to be looking at the same screen. They’re big enough these days that you can see everything you need with appropriate window management.
•Each person gets their own keyboard and mouse, no co-piloting on shared inputs and it’s not fine to have one person with a keyboard and the other person using the laptop it’s connected to.
•Divide the screen in half, then mark that virtual line on the desk with some tape or a marker… right down the desk. Each person gets their side and never should they encroach on the other half.
It might all seem a bit contrived and naff, but it makes a difference to the efficiency of the operation. Body position plays on the sub-conscious and determines whether you feel comfortable using the keyboard in front of you. Sitting too far to the side and you start to feel that you’re a spectator on somebody else’s computer, too far in front and you feel like you’re a pilot and co-pilot rather than peers.

Everyone is in their own context
You’re going to know what programs you need open to get the job done. For me it’s an IDE, a terminal or 3, and a web browser or the running application. Agree on a common screen layout where everything can be seen at the same time, and then mandate it across the entire team (I’ve found Divvy really useful to keep it consistent). It means that when you come to someone else’s computer it doesn’t feel foreign, it’s just like being on yours but at a different desk.

Most importantly though is that even though you’re both working on the same task together on the same machine, you’re probably both in slightly different head spaces and thinking about a slightly different aspect of the problem. Quickly switching between applications to satisfy your own curiosities can have a terrible impact on whatever it was your pair was thinking about. Being able to see everything at the one time prevents you from unexpectedly switching context on each other.

Make sure you get a monitor large enough to make it work. Given you’re going to spend ~8 hours each and every day staring at it, it’s not sensible to buy a cheap screen.

Silence is golden
This is difficult to nail until you’ve developed a good working relationship with your pair, but the best communication between the two of you is the non-verbal. For the same reasons above, you can never be sure what the head space of your pair is and interrupting them to mention a typo could completely break their train of thought. Instead wait for an obvious pause or context switch and simply point it out. Eventually it may get to the point where you know each other well enough that even more subtle queues like a shift in posture give away that you need to re-think the code you just wrote. But at least you can then do it once you’ve got your current thoughts committed to screen.

80 characters is enough for anyone
Like many of the tips on here, they’ve either been passed on to me by Graham Ashton or refined further by working with him for 3 years. This one seemed particularly pedantic to me, but I went along with it because it really wasn’t that difficult to adhere to and he was adamant that he’d accept nothing less. And it’s this, no line of code should ever extend beyond 80 characters.

I now try to enforce it wherever I go.

It took me a long time to fully appreciate how important it is, at least a year, but code readability is drastically improved and that is always a good thing. When pairing though, it is mandatory. You don’t have the luxury of scrolling indefinitely to the right to read the full line of code to see the intention because doing so will interrupt the flow of your pair. Worse still, it actually takes the bulk of the code off the screen.

It also prevents a nasty scenario where something important that drastically changes the intention of a line of code is cleanly hidden by the window edge and the real flow of the application is the opposite of what you thought was happening. Say something like the following:
def my_example_method
raise "Here is an example of something you would think gets raised!!!" unless
coder_reads_this_part?
end

Refactor any line that stretches over 80 characters. It only requires a little thought, makes the intention clearer, ensures you can see all the important code at once, and stops you from having to break someone else’s focus.

Test Driven Development
Tests first, then the code. It’s not just about ensuring you’ve got stable and working code, it’s about keep focussed on doing the smallest thing possible to make the tests pass.

This approach works great with pair programming, and here’s the formula: One person writes a test, the other makes it pass. Once it’s passing the person that just wrote the code writes the next test and hands it over. While one of you is trying to write a crafty test to keep the other busy for a while, your pair has already devised a cheeky way to have it always return the value you want and has devised a new test to catch you out.

It’s generally a quick back and forth iterative process where you’re each trying to outfox the other with edge cases or overly simplistic implementations that pass because the tests are robust enough. It’s almost game-like, and eventually you both come to a stalemate. At that point you sit back and realise the story is complete, and with excellent test coverage.

Don’t break the build
A continuous integration server is a must as is some form of reporting of any errors raised in production, and problems on both must be treated with a suitably high priority. Anything that is deemed “flakey” and raises an alert for non-legitimate reasons needs to be fixed immediately. As soon as the team starts to doubt the notifications, legitimate problems start to slip through too. It’s a bit like the broken windows theory, if you allow these things to go by without action then it breeds complacency and soon some failing tests become accepted as the norm.

That all means that before any developer commits any code back upstream, they run all the tests… all the time. They are there for a reason, and it’s inexcusable to not be running them. The CI box is there as a backstop only.

Deliver at the beginning of the end
If you’re starting your iterations every second Wednesday then code needs to be completed at the close of play the Monday immediately prior. That means you’ve got a full day to deploy the code to production and catch any unexpected errors. The other benefit of the mid-week approach is that on the rare occasions it all goes pear-shaped people are more inclined to stay back a little late and fix it. Trying to keep people back after hours on a Friday isn’t just difficult, they’ve often already mentally checked out and off on their weekend and you run the risk of actually making a bad situation worse.

If it all goes seamlessly and you roll-out and have a day free, fantastic! Time to look at that ever growing list of bugs your users have been sending back that haven’t been getting scheduled into an iteration. You can always find small tasks to fill the day that are incredibly useful but not super urgent.

Avoid bringing planning and unscheduled work forward, you’ll skew your workload for the next iteration and run the risk of over-stressing developers and/or over committing. The benefit of maybe squeezing in a day of development isn’t worth the additional risks.

Review
The code is released, people are using it, and everyone is happy. Now it’s time to look back and see how things went, what went really well and what could be improved upon.

Run a retrospective
Before the planning session, get all the team in the room to talk about the good and the bad of the previous iteration. Do it quickly, you should need an absolute max of 30mins but you can probably do it in much less. Get index cards or post-it notes of two different colours, one for “went well” and one for “needs improvement”, and everyone has to write at least one thing on a card of each colour. There are no limits to how many things people can list though, allow it as a cathartic forum for all issues to be brought out in the open.

Put all the cards up on a board (grouping similar topics), so everyone can see the balance of opinion on how things went. Ensure everyone understands what all of the cards mean and what they are referring to. Now everyone gets 3 points to spend on the “needs improvement” cards, they can spread the points out across 3 cards or put them all onto a single card (or the obvious option in between). The 3 cards that have the most points the team agrees to make a concerted effort to improve upon during the next iteration.

It’s only after a couple of iterations with consistently re-appearing top rated problem that I’d consider going back at looking at some of the tools I mentioned throwing out at the top of this article. And even then, only if you’re certain they’re the best way to fix the most pressing issue facing your team. If I’m honest, I don’t think it’s going to come up.

Split partially-completed stories
Occasionally a story only gets partly done, and there is much gnashing of teeth when people have to work out what to do with it. Because you’ve been taking an iterative test driven development you’ve got a bunch of code that is written, tested, and passing even though the story is incomplete. Fantastic!

Look at what is left to complete and write up a suitable story to reflect it, then go back and revise the previous card to reflect what was actually done. Now look at the original estimate and try and work out what percentage of the story you’ve completed, apply the appropriate number of points to each story (keeping in mind that if you’re following along with my 3 point/jelly bean system, 3 points = 2 days. So if you’ve done half you’ve got 1 day on each or 2 points on each).

That way you get an accurate reflection of both what was completed in the previous iteration and a fair idea of what remains, based on your original estimates (remember, we wanted continual optimism. No revising estimates retrospectively).

Calculate your velocity
Now you know how much work you got completed, you can with a fair degree of confidence predict how much you can do next iteration. After a few iterations, you can look back and calculate the average for the past two or 3 to flatten out any particular fast or slow ones and get a good feel for what is achievable.

Under-promise, over-deliver
No kid likes waking up Christmas morning to find the present they asked Santa for to be nowhere under the tree. Likewise, no customer likes being told you’re going to give them something in two weeks and for it to not materialise. So if you’re uncertain, you’re better off scheduling too little work and then pulling an extra story in later in the iteration than over committing. It’s not just about managing customer expectations though, it’s about limiting stress on developers by keeping the targets realistic.

Conclusion
These are the things that have contributed to making it work at places I’ve enjoyed working at. The most important thing is to measure overall success of your process as it’s ability to create a happy and productive environment, good things will naturally flow from it. At the very least you should apply the principles of agile to the implementation of the process itself. Nothing is set in stone, pause at regular intervals to review the success, adjust the most important aspects to give people what they want. Sphere: Related Content

Some random thoughts on programming

As I near the end of my third decade programming, I keep finding myself coming back to pondering the same kind of questions.

■ Why are some people good at this and some are not?
■ Can the latter be trained to be the former?
■ What is the role of language, development environment, methodology, etc. in
improving productivity?
I’ve always been partial to the “hacker”. The person who can just flat out code. For some people this seems to come easy. I have been lucky enough to work with a few people like this and it has been awesome. As you talk with them about a problem they are already breaking it down and have an idea of how the final system will work. I don’t mean that other aspects of professionalism, experience, etc. aren’t important its just that I believe in the end there is a raw kind of talent that you can’t teach. My wife can sing, I can’t, and not for lack of practice. She practices, but she can also belt out a perfect tune without hardly trying.

What are the core talents that makes someone a good hacker? I think there are two:
1.The ability to break down an idea/system into an abstract set of “concepts and
actions”
2.The ability to understand how these “concepts and actions” execute in a
particular environment.
Note that I didn’t say anything about coding. Coding is actually the least interesting part. If you can do the others well, coding is just another kind of typing. Coding is the thing your fingers do while your mind is flipping between 1) and 2). Learning a new language is less about learning the syntax than it is about learning the execution model.

Writing good, correct code involves not just conceiving of how something should be 1) but also understanding all that can happen 2).

I honestly don’t know if these things can be taught.

In 30 years I have only seen a few examples where a mediocre programmer became good or a good one became great and zero examples of a mediocre programmer becoming great.

What does this mean for people who want to build large software systems? The easy answer is to only hire great people. Which is fine when you can do it but not everyone can be or hire the best so what is left for the majority?

What should be done to make things easier for people aren’t in the top 5%?

There are lots of nice things that have been invented like source code control, higher level languages, garbage collection, etc. which reduce or abstract away some of the problems. And while sometimes these abstractions are leaky, they are really one of the few weapons we have against the beast called Complexity. Complexity is the key problem we face, especially if we are not wizards because:

A persons talent with regard to 1) and 2) above determine how much complexity they can deal with.

When I was in gradual school at UNC-CH Fred Brooks (the mythical man month guy) used to talk about Complexity as being a major thing which makes programming hard. Its not something easily cast aside:
Quoting Dr Brooks:
The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence.

I wish I had appreciated everything Fred said when I was a cocky grad student as much as I appreciate it today… sigh.

Some things are hard because they are hard and not much can be done about them. Still our goal needs to be to not try to make it worse than it is.

To this end, some people say: Do the simplest thing that will work.

I agree but prefer this formulation: Strive for the simplest thing that can’t possibly fail.

If you think about what can go wrong you are more likely to find the problems than if you think about what will go right.

I wish I could say we were making progress in reducing complexity but we seem to be adding it into our programming environments rather than removing it. One of the things I have thought about recently is the complexity of these abstractions themselves. Programming languages of today are so much more abstracted and complex than what I learned on (Assembler, Basic, Pascal, C).

For example the world (or at least the web) is currently engaged in writing or rewriting everything in Javascript. Not because JS is so awesome, but because 15 years ago it got stuck into browsers, then we all got stuck with browsers and thus we are all stuck with JS. Sure there’s lots of server stuff done in various other languages but really how long can that last? How long can one justify having to write 2 parts of an application in 2 different languages?

The future looks like node.js.

I am not a JS hater. Its just that I used to think Javascript was a simple language. As someone who wrote 3D Game engine code in C++, my thought was “if these webkids can do JS then it can’t be that complicated”. After building a webapp using Javascript/Dojo/svg to create a realtime browser for genetic information, I realize how wrong I was.

Sure its all nice and simple when you are doing this:
$(“#imhungry”).click(function() { alert(“food”);});

However a lot of complexity lurks just under hood. And its the way it all interacts that concerns me. Crockford called it Lisp in C’s clothing. This should excite a few people and scare the crap out of the rest.

JS has lots of opportunity for expressing yourself in interesting ways. Expressiveness is good, but too much of it can create more problems than it solves. Back when C++ was a bunch of source code you downloaded from AT&T and you had to compile the complier yourself before you could use it, we grad students stayed up late eating bad fried chicken and debating the merits of O-O programming. It was at this point I began to wonder if the productivity gains of O-O were more the offset by the time spent debating about what good O-O practice is. This tape has played in my head more than a few times in my career.

Its not all bad of course: On balance, garbage collection is a win and I see lots of benefits to using closures in JS, but the underlying complexities of things escape most people and their interactions are understood by even fewer. If you have to slog through this (and you should) to understand what happens when you call a function, you are cutting out alot of people from fully understanding what is going on when thier code executes.

I hope that some of this will be address by standards of practice. Things like Google’s guidelines seem to be a good start. As with C++, I think that the best hope for improving productivity will actually be to agree on limits on how we use these tools rather than seeing how far we can push the envelope.

Well except for wizards and we know who we are.


15 Responses to Some random thoughts on programming
Jason P Sage says:
September 7, 2010 at 7:49 am
Right On. Nice Article… I’m on year 28 of programming – and I really liked this article.

P.S. I am a JS hater and .Net hater too (though good at both) because P-Code and Compile On demand are Slower and Make it necessary to obfuscate your code if you have intellectual property concerns.

Jonathan Palethorpe says:
September 7, 2010 at 7:52 am
22 years ago when I started in programming I could mess about at home doing a bit of BASIC and at work I wrote green screen sales and invoicing apps in COBOL ’74 using ISAM files. Nowadays programming is many times more difficult- ISAM files have been replaces with all singing all dancing SQL databases with a host of frameworks designed to access them (no in-language stuff as in COBOL with ISAM) – the green screens have been replaced with name your weird and wonderful GUI framework of choice (WPF in my case) and the languages themselves are now many, varied and unreadable to those not in the know.

I much prefer doing development work now but there are times when I still hanker after a bit of green screen/ISAM/MOVE W01-FOO TO W02-BAR

Michael Fever says:
September 7, 2010 at 9:24 am
Yeah, I’ll have to admit that was pretty random. I am an ok programmer, not great, but better than mediocre. I approach every job the same however by looking at what needs to get done and then by what is going to be required later to maintain it. Sometimes being OK is all you need to be.

Bob Gibson says:
September 7, 2010 at 9:47 am
Very good points.

I can relate… I started programming, in Basic & Fortran, … 40 years ago.

Jeffrey Lanham says:
September 7, 2010 at 10:13 am
Right on. After working in computers and programming for 27+ years, I have to agree. Programming is, unfortunately, something you inheritly understand or you don’t. You can teach semantics, but not the thought processes that go along with it. That being said, great programmers start as mediocre programmers, but quickly rise. Mediocre programmers tend to rise to somewhat competent and then stay there. That’s, unfortunately, the way it is.

Great article.

Alan Balkany says:
September 7, 2010 at 10:15 am
Sometimes the “hacker” is just someone who’s very familiar with the problem domain. An average programmer may appear to be a “hacker” to their coworkers when they’re re-implementing a system very similar to what they implemented at two other jobs.

They already know what works best, and the mistakes they’ve made in previous implementations.

I think most people don’t program as well as their potential because the best practices haven’t been communicated to them. For example, I see many programmers who don’t seem to know that reducing function length is a good way to keep complexity manageable.

The best practices are still being worked out; the field of software development is still in its infancy, compared to more mature disciplines such as mathematics and mechanical engineering.

Oisín says:
September 7, 2010 at 10:47 am
Alan is spot on here. Perhaps the idea that many programmers overlook useful, basic heuristics for programming (such as “shorter functions imply more manageable code, so break down your functions until they are small”) ties in with one theme of the OP, that programming has become more complex in terms of frameworks and maybe language expressiveness.

When I started programming, it was on the horrible bare-bones no-frills Commodore 64 Basic interpreter, followed a couple of years later by a lovely language called GFA Basic on the Atari ST, and then by 68k assembly and C. Maybe those formative years of just playing about, writing programs with minimal focus on libraries, industry standards and “frameworks”, were helpful in learning how to solve problems and write code.
But these days I spend more time reading articles about flashy programming/library techniques than actually thinking about and writing programs.

It could be the case that novice programmers nowadays jump too early onto this wheel, without spending the time doing simple, pure programming, so that they don’t sufficiently develop these key abstractions and heuristic tricks for managing complexity.

Joel Wilson says:
September 7, 2010 at 10:54 am
Couldn’t agree more about programming skill or JS. The best programmers I’ve ever known generally don’t have a comp-sci education (myself included). In fact, the single best I’ve ever known personally was an English major a couple of papers away from his Doctorate. He found programming and never looked back. As far as JS, I agree that its “appeal” (for lack of a better word) is simply because it’s the only trick in town. As a language it sucks for anything more than basic functionality. Although I use jQuery professionally, I find it sad that we have to resort to bloated frameworks and layers of abstraction to try and hide the basic weakness of the underlying language.

Leon says:
September 7, 2010 at 11:39 am
Oh, I guess for that post you’ll earn a lot of flames from mediocre guys who think that they will be great one day. Especially nowadays when everyone and his mother see themselves as coders because they can hack up some javascript/ruby code

Maintenance Man says:
September 7, 2010 at 1:12 pm
Say it ain’t so. Us mediocre programmers need some hope for the future. Heheh.

CollegeGrad says:
September 7, 2010 at 1:38 pm
I majored in Mathematics (not applied) and only became interested in programming in my junior year of college. I only started to really learn quality programming skills after leaving college. I find that I often know exactly how to solve a problem before the veteran programmers at my job even have a chance to really think about it. The problem for me is spending the time making sure my code is well structured and maintainable. When working with other programmers, I end up being the one explaining the problem and how we’re going to solve it over and over. My own experience, which I’ll be the first to point out is very limited, has indicated that it’s my experience with the theoretical aspects of computer science and complexity theory, as well as all my time spent proving abstract theorems, that gives me an advantage over the people who just know how to code. Maybe if programmers focused less on languages and more on things like linear algebra, abstract algebra, graph theory, etc. then they would be much better at their job. just my novice opinion.

Ryan says:
September 7, 2010 at 2:29 pm
“Do the simplest thing that will work.”

Not sure if you will even read this, but:

That is a great statement *but* I think it is critical to mention what I consider a serious and way too often overlooked problem. Lets say you start with a great code base. Then you amend/change it ‘as simply as possible’ to add a new feature. And again for a new feature, and again. Every step being done ‘simply’ actually adds unnecessary to the complexity of the program. What really needs to happen is the simplest implementation that incorporates *all* features, including the ones that had previously been implemented. And this takes more time and more effort in the short term, but boy does it save on time and effort in the long term.

Giles Bowkett says:
September 7, 2010 at 2:46 pm
Since my comment disappeared into a black hole with no feedback, I can only assume it’s gone into some moderation zone, which means I should assume people will read it, which means I should probably be less obnoxious about it, so: Java was designed to prevent people from fucking up. It did not; instead it prevented people from writing great software. That’s in the nature of design approaches based on minimizing the damage that the incompetent can do. The incompetent can fuck up anything; the only thing you achieve with standards and practices is restraining the so-called “wizards,” who create all the advances that save everybody else so much time. Exercising control to prevent incompetent mistakes is a sucker’s game; freeing up the language to “give them enough rope” results in great libraries which make everybody more productive (or at least, everybody who uses them).

I’m rehashing a debate for you here from 2006. It comes from the days when people were saying Rails would never be accepted in corporations, and, more to the point, your advocacy of standards and practices to prevent people from fucking up is the same argument that was coming from the people who said Rails would never be accepted in corporations. Their logic was clean and their conclusions were false; to anyone who was reading those debates, your argument was, I’m sorry, completely discredited almost five years ago now.

Sorry to go off in such detail on your final paragraph, which was kind of a tangent to your overall post, it just drives me fucking crazy that people settle these arguments definitively, only to see them revived a few years later and started over completely from scratch. “Those who don’t study history are doomed to repeat it,” etc.

admin says:
September 7, 2010 at 2:47 pm
I think it depends on how draconian the limits are. I don’t think the google guidlines for instance are any kind of crushing blow to innovation. I just don’t think that just because a language allows something you should necessarily do it.

admin says:
September 7, 2010 at 3:03 pm
Ok. Thanks for the longer comment. I’m learning alot about how bad I am at making my point.

Putting people on rails (sorry couldn’t resist) isn’t going to make bad programmers good. Crap can come from anything. That wasn’t what i wanted to say.

I’m more interested in keep some kind of boundary so that what results is maintainable enough by other people.

I Actually pushed for doing a recent project in python/django rather than java etc because I value the benefits which some of these newer, better langauges give you.

I just think with great power comes great responsibility. The cool meta-programming trick you did is awesome and personally I will give you props for it. I just hope the next guys can figure it out too.

Also I think that not all “rope” as you say is equal. Some things allow for great expressiveness and freedom while not adding much to complexity. I don’t use ruby so I’m not an expert on that, but in python there are function decorators, its a cool syntax for people to add some functionality. But its pretty explicit when its used. @login_required etc. Javascripts closures seem to be able to happen almost by accident or at least its not as explicit to someone looking at the code for the first time. Doesn’t mean we shouldn’t use them. I did today, but we need to be aware of what we are doing. And not i am not trying to start a python/javascript debate. Sphere: Related Content

Are microkernels the future of secure OS design ?

MINIX 3, and microkernel OSs in general, might show us the way toward the future of secure OS design.

Date: September 6th, 2010
Author: Chad Perrin
Category: Application Security, Security

--------------------------------------------------------------------------------
Andrew S. Tanenbaum created the MINIX operating system in the 1980s as an instructional tool. He wanted to provide an aid to helping people learn about the design of operating systems, and did so by creating an incredibly simple — in some respects, too simple for any serious use — Unix-like OS from scratch, so that his students could examine and modify the source code and even run it on the commodity hardware of the time. He and Al Woodhull co-authored a textbook that focused on the MINIX OS.

Since then, 1.5 and 2.0 versions of MINIX have been created as well, with the same goal: to serve as a study aid. By contrast, the current iteration of the MINIX project, known as MINIX 3, is being developed with the aim of providing a production-ready, general purpose Unix-like operating system. In Tanenbaum-Torvalds debate, Part II, Andrew Tanenbaum describes the relationship of MINIX 3 to its forebears:

MINIX 1 and MINIX 3 are related in the same way as Windows 3.1 and Windows XP are: same first name. Thus even if you used MINIX 1 when you were in college, try MINIX 3; you’ll be surprised. It is a minimal but functional UNIX system with X, bash, pdksh, zsh, cc, gcc, perl, python, awk, emacs, vi, pine, ssh, ftp, the GNU tools and over 400 other programs.

Among Tanenbaum’s key priorities in the development of MINIX 3 is the desire to improve operating system reliability to the level people have come to expect from televisions and automobiles:

It is about building highly reliable, self-healing, operating systems. I will consider the job finished when no manufacturer anywhere makes a PC with a reset button. TVs don’t have reset buttons. Stereos don’t have reset buttons. Cars don’t have reset buttons. They are full of software but don’t need them. Computers need reset buttons because their software crashes a lot.

He acknowledges the vast differences in the types of software required by these different products, but is optimistic that the shortcomings imposed by the necessarily greater complexity of software in a desktop computer can be mitigated to the point where end users will no longer have to deal with the crashing behavior of operating system software. As he puts it, “I want to build an operating system whose mean time to failure is much longer than the lifetime of the computer so the average user never experiences a crash.”

There are already OSs that meet that goal under extremely limited, tightly constrained circumstances, but he intends to provide the same stability and reliability under the harsh, changeable circumstances of general-purpose consumer computer use. It is an ambitious goal, but his approach is well thought out and shows great promise. Given that many of the stability issues we see in operating systems using monolithic kernels result from the fact that a crashing driver can crash the OS kernel itself, his intention to solve the problem by moving drivers out of kernel space may be on the right track.

In fact, if a driver fails to respond to what he calls a “reincarnation server”, which monitors the status of active drivers, the reincarnation server can silently swap out the hung or otherwise failing driver process for a fresh process that picks up where the original left off. In most — if not all — cases, the user may not even know anything has happened, where a monolithic kernel OS would likely crash immediately.

While his primary interest is in reliability, rather than security per se, there are obvious security benefits to his microkernel design for MINIX 3. Without going into too much technical detail, there are mechanisms in place that restrict a driver’s access to memory addresses, limiting it to its own assigned address space in a manner that in theory cannot be circumvented by the driver. Drivers all run in user space, keeping them out of kernel space, whereas kernels loaded as part of a monolithic kernel design can typically touch any memory addresses at all.

A number of countermeasures have been invented and employed in various OS designs over the years, but keeping drivers out of kernel space entirely certainly seems like the most effective and inviolate protection against driver misbehavior. As a result, this may all but eliminate the possibility of buffer overruns and similar problems in kernel space.

Further, mediated contact between drivers and other user space systems in MINIX 3 extends the strict privilege separation of Unix-like systems to protect other processes than the kernel process, as well. In short, whole classes of common system vulnerabilities may become extinct within the family of OSs that may develop in the wake of MINIX 3.

MINIX 3 itself is still in development, but it is currently a working OS with many of Tanenbaum’s intended reliability assurance features already implemented. You can download it from the MINIX 3 Website and boot it from a LiveCD though, as Tanenbaum states, you should install it to a partition on the hard drive of a computer if you want to do anything useful with it.

It will not replace MS Windows, MacOS X, Ubuntu Linux, or FreeBSD right now, but it may well be the future of general purpose OS design — especially secure general
purpose OS design. Sphere: Related Content

12/8/10

The Future of the Internet

“In only a few short years, electronic computing systems have been invented and improved at a tremendous rate. But computers did not ‘just grow.’ They have evolved… They were born and they are being improved as a consequence of man’s ingenuity, his imagination… and his mathematics.” — 1958 IBM brochure

The Internet is a medium that is evolving at breakneck speed. It’s a wild organism of sweeping cultural change — one that leaves the carcasses of dead media forms in its sizeable wake. It’s transformative: it has transformed the vast globe into a ‘global village’ and it has drawn human communication away from print-based media and into a post-Gutenberg digital era. Right now, its perils are equal to its potential. The debate over ‘net neutrality’ is at a fever pitch. There is a tug-of-war going on between an ‘open web’ and a more governed form of the web (like the Apple-approved apps on the iPad/iPhone) that has more security but less freedom.
So what’s the next step in its evolution, and what’s the big picture? What does the Internet mean as an extension of human communication, of the human mind? And forget tomorrow — where will the web be in fifty years, or a hundred? Will the Internet help make the world look like something out of Blade Runner or Minority Report? Let’s just pray it doesn’t have anything to do with The Matrix sequels, because those movies really sucked.

This article will offer in-depth analysis of a range of subjects — from realistic expectations stemming from current trends to some more imaginative speculations on the distant future.

[By the way, did you know we have a free Email Newsletter? Subscribe now and get fresh short tips and tricks in your inbox!]

Security
“Death of the Open Web”?

Those words have an ominous ring for those of us who have a deep appreciation of the Internet as well as high hopes for its future. The phrase comes from the title of a recent New York Times article that struck a nerve with some readers. The article paints a disquieting picture of the web as a “haphazardly planned” digital city where “malware and spam have turned living conditions in many quarters unsafe and unsanitary.”

There is a growing sentiment that the open web is a fundamentally dangerous place. Recent waves of hacked WordPress sites revealed exploited PHP vulnerabilities and affected dozens of well-known designers and bloggers like Chris Pearson. The tools used by those with malicious intent evolve just as quickly as the rest of the web. It’s deeply saddening to hear that, according to Jonathan Zittrain, some web users have stooped so low as to set up ‘Captcha sweatshops’ where (very) low-paid people are employed to solve Captcha security technology for malicious purposes all day. This is the part where I weep for the inherent sadness of mankind.

“If we don’t do something about this,” says Jonathan Zittrain of the insecure web, “I see the end of much of the generative aspect of the technologies that we now take for granted.” Zittrain is a professor of Internet governance and regulation at Oxford University and the author of The Future of the Internet: and How to Stop It; watch his riveting Google Talk on these subjects.
The result of the Internet’s vulnerability is a generation of Internet-centric products — like the iPad, the Tivo and the XBOX — that are not easily modified by anyone except their vendors and their approved partners. These products do not allow unapproved third-party code (such as the kind that could be used to install a virus) to run on them, and are therefore more reliable than some areas of the web. Increased security often means restricted or censored content — and even worse — limited freedoms that could impede the style of innovation that propels the evolution of the Internet, and therefore, our digital future.

The web of 2010 is a place where a 17 year-old high school student can have an idea for a website, program it in three days, and quickly turn it into a social networking craze used by millions (that student’s name is Andrey Ternovskiy and he invented Chatroulette). That’s innovation in a nutshell. It’s a charming story and a compelling use of the web’s creative freedoms. If the security risks of the Internet kill the ‘open web’ and turn your average web experience into one that is governed by Apple or another proprietary company, the Andrey Ternovskiys of the world may never get their chance to innovate.

Security Solutions
We champion innovation on the Internet and it’s going to require innovation to steer it in the right direction. Jonathan Zittrain says that he hopes we can come together on agreements for regulating the open web so that we don’t “feel that we have to lock down our technologies in order to save our future.”

According to Vint Cerf, vice president and Chief Internet Evangelist at Google, “I think we’re going to end up looking for international agreements – maybe even treaties of some kind – in which certain classes of behavior are uniformly considered inappropriate.”

Perhaps the future of the Internet involves social structures of web users who collaborate on solutions to online security issues. Perhaps companies like Google and Apple will team up with international governmental bodies to form an international online security council. Or maybe the innovative spirit of the web could mean that an independent, democratic group of digital security experts, designers, and programmers will form a grassroots-level organization that rises to prominence while fighting hackers, innovating on security technology, writing manifestos for online behavior, and setting an example through positive and supportive life online.

Many people are fighting to ensure your ability to have your voice heard online — so use that voice to participate in the debate, stay informed, and demand a positive future. Concerned netizens and Smashing readers: unite!

Freedom
Net Neutrality

Some believe that the fate of the Internet has been up for grabs ever since the federal government stopped enforcing ‘network neutrality’ rules in the mid-2000’s. In a nutshell, net neutrality means equality among the information that travels to your computer: everyone has the right to build a website that is just as public, affordable, and easily accessible as any other. However, some companies like phone and internet service providers are proposing ‘pay tiers’ (web service where you need to pay premium fees in order to allow visitors to access your site quickly). These tiers of web service could kill net neutrality by allowing those who can afford premium service (particularly large media companies who don’t like sharing their audience with your blog) greater access to consumers than the average web user.

The debate over net neutrality reached a boiling point when Google and Verizon announced a ‘joint policy proposal for an open Internet’ on August 9th, 2010. Despite the proposal’s call for a “new, enforceable prohibition against discriminatory practices” amongst online content, many criticized it, citing leniency and loopholes.

Net neutrality needs to be made law. If the Internet were to have a slow lane and a fast lane, your average web user could lose many of his or her freedoms and opportunities online, thereby shattering the core values that make the Internet so profoundly valuable to society. However, that’s just the tip of the iceberg for this thorny issue. To learn more, read the full proposal or watch the Bill Moyers episode ‘The Net @ Risk.’

The World into the Web
Browser-based Everything

Google is developing a variety of applications and programs that exist entirely within the browser. Their PAC-MAN game was a preview of what’s to come because it allowed in-browser play of a simple, lightweight video game that required no downloads and relied on pure HTML, CSS, and Javascript. At the company’s 2010 I/O conference, Google laid out its plans to develop “rich multimedia applications that operate within the browser” (according to this New York Times report on the conference). The company plans to sell in-browser web applications like photo editing software (imagine using a Photoshop equivalent entirely within the browser) that it will sell in a web applications store called the Chrome Web Store.

If our programs and applications are about to be folded into the browser, what will exist within the browser in ten years? Currency? Education? Consciousness? Personally, I’m hopeful that my browser will be able to produce piping hot cheeseburgers sometime soon.
The Internet as a Collective Consciousness

The Internet is a medium, and philosopher Marshall McLuhan believed that all media are extensions of the human senses. The engine of our collective creative efforts is the force that’s causing the web to evolve more rapidly than any biological organism ever has.
The Internet is an extension of the collective human mind and it’s evolving into a medium of transcendence. By constructing a place where the collective human consciousness is both centralized in one location (on your computer) and globally accessible (for those with the means to reach or use a computer, that is), our human spirit is transcending the human body. Way back in 1964, McLuhan himself wondered, “might not our current translation of our entire lives into the spiritual form of information seem to make of the entire globe, and of the human family, a single consciousness?”

With the advent of trends including social media, ‘lifecasting,’ and ‘mindcasting,’ the Internet is being used as a real-time portal for the human consciousness. Perhaps those trends will be inverted by some web users of the future: instead of bringing offline life to the web (as so-called ‘lifecasters’ do when they stream live video of their attendance at an offline event), some web users will live their entire public lives online. Imagine a pop star who conducts her entire career online: every interview, live performance, music video or album release conducted solely through a browser window or mobile screen. Or a media theorist who exploited the platform of the web while discussing the theoretical ramifications of those actions. It’d be a great gimmick.

The Web into the World
The ‘Web of Things’

The ‘web of things’ or ‘Internet of things’ is a concept that will be a reality (at least in a rudimentary form) in the near future. The idea is that devices, appliances, and even your pets can all be tracked online. With Google Maps for iPhone, you can currently track your location on a digital map in relation to the streets and landmarks in your area. So it’s not hard to imagine a world where you can zoom in on your location and see detailed, 3D renderings of your surroundings: the cars on your block, the coffee machine in your kitchen, even Rover running around in your backyard! And it’s a good thing that you’re digitally tracking the location of poor Rover; he’s liable to hop the fence and make a run for it now that you’ve created a satellite computer out of everything you own (including his dog collar) by attaching a tracking device to it.

AT&T is betting big on the web of things. According to this Reuters article, the phone service provider is investing in tracking devices that could be installed in cars, on dog collars, and on the pallets used to move large shipments of products. The dog collar, for example, “could send text messages or emails to the owner of a pet when it strays outside a certain area, or the device could allow continuous tracking of the pet.”

Combine the concept of the ‘web of things’ with Second Life-style 3D imaging and you can imagine a web-based duplicate world — a virtual world that corresponds to the real one. But what are the implications of a world where every physical item has a corresponding digital existence online? Can we track the physical effects of climate change in the web of things? Will there be a digital avatar for every pelican carcass in the vicinity of the oil spill that’s devastating the Gulf of Mexico? It’s a tragic shame to develop a virtual world if we let the natural one go to waste in the meantime.

Interactive Landscapes

It has been said that today’s science fiction is tomorrow’s reality. Unfortunately, most good science fiction stories are cautionary tales set in dystopian nightmares.
Simon Mainwaring reports on the N building in Japan, where “the whole building facade has been transformed into a real time dialogue between smart phones and what’s going on inside the store.” The exterior of the building is layered with QR codes (an alternate form of bar code) that can deliver real-time information to your phone. In Stephen Spielberg’s film Minority Report (adapted from a short story by mad genius Philip K. Dick), Gap ads came alive to hawk khakis to Tom Cruise. Looks like we’re about one step away from this scenario.

Mr. Mainwaring imagines a future with “billboards that watch you shop and make targeted suggestions based on your age, location and past buying habits,” and “stores will effectively be turned inside out as dialogue and personalized interaction with customers begins outside the store.”
The technology is cool, but it sounds like a pretty annoying future if you ask me. Who wants to be accosted by a holographic salesperson? The web grants us a great opportunity to use our collective voices to speak out on topics that matter to us. Because there are no regulations yet for much of this technology, it may be up to concerned citizens to make themselves heard if Internet-based technology is used in intrusive or abrasive ways.
The ‘Innerweb’

Cyborgs are among us already — humans whose physical abilities have been extended or altered by mechanical elements built into the body (people who live with pacemakers are one example). What will happen when the Internet becomes available on a device that is biologically installed in a human? What will the first internal user interfaces look like?

Here’s one speculation.

In the near future, we may be capable of installing the Internet directly into the user’s field of vision via a tiny computer chip implanted into the eye. Sound far-fetched? I doubt that it would sound far-fetched for Barbara Campbell, whose sight has been partially restored by a digital retinal implant (CNN reports on Barbara’s artificial retina).

Ms. Campbell was blind for many years until she had a small microchip surgically implanted in her eye. A rudimentary image of Ms.Campbell’s surroundings is transmitted to the device, which stimulates cells in her retina, in turn transmitting a signal to her brain. It’s a miracle that the development of a bionic eye has begun to help the blind see.

How else might doctors and scientists take advantage of the internal microchip? Perhaps the user’s vision will be augmented with an Internet-based interface with capabilities including geolocation or object identification. Imagine if technology like Google Goggles (a web-based application that identifies images from landmarks to book covers) was applied inside that interface. The act of seeing could not only be restored but augmented; a user might be capable of viewing a landscape while simultaneously identifying web-based information about it or even searching it for physical objects not visible to the naked eye. Apply the concept of augmented sight with the idea of the ‘web of things’ — an environment where physical objects have a corresponding presence on the web — and you can imagine a world where missing people are located, theft is dramatically reduced, the blind can see, and ’seeing’ itself means something more than it used to.

If the web is an extension of our senses, it follows suit that the web may be capable of modifying those senses or even accelerating their evolution.

The Crown Jewels
“The next Bill Gates will be the deliverer of a highly technological solution to some of our climate change challenges.” — Lord Digby Jones of Birmingham

In preparation for this article, I considered a variety of wild ideas and fun speculations about the future. Could the Internet be used to solve the problem of climate change, generate tangible matter, or contact extraterrestrial life? Maybe those ideas sound like the stuff of imaginative fiction, but in a world where quantum teleportation has been achieved and researchers have created a living, synthetic cell, it almost seems as if the concept of science fiction is being eradicated while real technology brings our wildest fantasies to life. Here is the result of my most daring (absurd?) speculation.

Time Travel
I called on physics teacher Mark Stratil to answer my last burning question: could the Internet ever be capable of facilitating the development of time travel? Here’s Mark’s answer:

“The Internet is still based on computers, which make linear calculations. Right now, all computers are based on binary code, which is a series of yes and no questions. You can make something that’s incredibly complex with a series of yes and no questions, but it takes a certain amount of time. The Internet still has to go through those calculations and it still physically has to make one calculation that will lead to the next calculation that will lead to the next. So no matter how fast we can get our computers – they’re making billions of calculations, trillions of calculations per second – there’s still going to be some lag time. They’re still limited by time in that way. They still need some time to make that conversation or that calculation.

In that way, they’re kind of chained to time. Their whole existence is based on a linear sequence of things that happen. In order to create something else, something that goes outside of time, you would have to make it a non-linear system — something that that’s not based on a series of yes and no questions, because those have to be answered in a precise order. It would have to be some kind of system that was answering all the questions at once.”

So Mark’s short answer to my fundamental question was basically that the Internet, in its current state, would not be capable of facilitating time travel. However, if the Internet was liberated from the linear structure of binary code and migrated onto an operating system that ‘answered all questions at once,’ then maybe it could have the potential to manipulate time or transcend the boundaries of time.

Sounds unlikely at this point, but one of the Internet’s greatest capabilities is the opportunity to share and develop ideas like these!

Conclusion
Responsible Evolution
Through technology, we hold the reins to our own evolution.

For the first time in history, it might be said that there are moral implications in the act of evolution. The Internet is an extension of our senses and our minds, and its progress is propelled by our own creative and intellectual efforts. The future of the Internet will be shaped by millions of choices and decisions by people from all walks of life. Designers and programmers like us have the advantage of technical skill and specialized knowledge. Given the increasing presence of the Internet in our lives, our choices can have deep reverberations in human society.

We’ll face small choices like what color to use for a button and larger choices like which platforms to endorse and which clients to support with our work. But the real questions form broad patterns behind every media trend and every mini technological revolution. Can we use technology to develop solutions to environmental problems — or will we abandon the natural world in favor of a digital one and the ‘web of things’? Have we fully considered what it means to merge biology and technology? And finally, do we really need a digital tracking device on our coffee machines?

What a thrilling time to be alive! Let’s proceed with great enthusiasm and a commitment to designing a future that is meaningful, peaceful, and staggeringly exciting.

Partial Bibliography
* New York Times Magazine: “The Death of the Open Web”
* Simon Mainwaring: “The Future of Shopping: What Happens When Walls Start Talking”
* The Gutenberg Parenthesis: Parallels Between Pre-Printing Communication and the Digital Era
* New York Times: “Google Pitches a Web-Centric Future”
* Google Talks: The Future of the Internet
* Moyers on America: ‘The Net @ Risk’ (Video)
* Pew Internet Research Poll: Future of the Internet IV
* Google I/O: The Web is Killing Radio, Newspapers, Magazines, and TV (TechChrunch reports)

Related Posts
You may be interested in the following related posts:
* The Evolution of the Logo
* The Dying Art Of Design
* Lessons From Swiss Style Graphic Design
* Art Manifestos and Their Applications in Contemporary Design Sphere: Related Content

1/8/10

Linux and Windows Interoperability: On the Metal and On the Wire

by MichaelF on August 13, 2007 03:10PM

I had the opportunity to present at both OSCON in Portland and at LinuxWorld in San Francisco in the last three weeks – both O’Reilly and IDG were gracious enough to grant me a session on the work that Microsoft is doing with Novell, XenSource, and others on Linux and Windows interoperability.

Overall our focus is on three critical technology areas for the next-generation datacenter: virtualization, systems management, and identity. Identity in particular spans enterprise datacenters and web user experiences, so it’s critical that everyone shares a strong commitment to cross-platform cooperation.

Here are the slides as I presented them, with some words about each to give context, but few enough to make this post readable overall. I skipped the intro slides about the Open Source Software Lab since most Port 25 readers know who we are and what we do.


Why interoperability?

The market for heterogeneous solutions is growing rapidly. One visible sign of this is virtualization, an “indicator technology,” which by its nature promotes heterogeneity. Virtualization has become one of the most important trends in the computing industry today. According to leading analysts, enterprise spending on virtualization will reach $15B worldwide by 2009, at which point more than 50% of all servers sold will include virtualization-enabled processors. Most of this investment will manifest itself on production servers running business critical workloads.

Given the ever improving x86 economics, companies are continuing to migrate off UNIX and specialty hardware down to Windows and Linux on commodity processors.

So, why now?

First, customers are insisting on support for interoperable, heterogeneous solutions. At Microsoft, we run a customer-led product business. One year ago, we established our Interoperability Executive Customer Council, a group of Global CIOs from 30 top global companies and governments – from Goldman Sachs to Aetna to NATO to the UN. On the Microsoft side, this council is run by Bob Muglia, the senior vice president of our server software and developer tools division. The purpose of this is to get consistent input on where customers need us to improve interoperability between our platforms and others – like Linux, Eclipse, and Java. They gave us clear direction: “we are picking both Windows and Linux for our datacenters, and will continue to do so. We need you to make them work better together.”

Second, MS and Novell have established a technical collaboration agreement that allows us to combine our engineering resources to address specific interoperability issues.

As part of this broader interoperability collaboration, Microsoft and Novell technical experts are architecting and testing cross-platform virtualization for Linux and Windows and developing the tools and infrastructure necessary to manage and secure these heterogeneous environments.

I am often asked, “Why is the agreement so long?” as well as “Why is the agreement so short?” The Novell-Microsoft TCA is 5 years mutual commitment. To put this in context, 5 years from now (2012) is two full releases of Windows Server and 20 Linux kernel updates (given the 2.5 month cycle we’ve seen for the last few years). This is an eternity in technology. What’s important to me is that it’s a multi-product commitment to building and improving interoperability between the flagship products of two major technology companies. This means we can build the practices to sustain great interoperable software over the long term as our industry and customer needs continue to evolve.

This talk covers two major components of the future of Linux and Windows interoperability: Virtualization and Web Services protocols.

On the Metal focuses on the virtualization interoperability work being done between Windows Server 2008 and Windows Server virtualization, and SUSE Linux Enterprise Server and Xen.

On the Wire covers the details and challenges of implementing standards specifications, such as WS-Federation and WS-Management; and how protocol interoperability will enable effective and secure virtualization deployment and management.

These are the key components required for the next-generation datacenter. We know the datacenters of today are mixtures of Windows, Linux, and Unix, x86, x64 and RISC architectures, and a range of storage and networking gear. Virtualization is required to enable server consolidation and dynamic IT; it must be cross-platform. Once applications from multiple platforms are running on a single server, they need to be managed – ideally from a single console. Finally, they must still meet the demands of security and auditability, so regardless of OS they must be accessible by the right users at the right levels of privilege. Hence, cross-platform virtualization demands cross-platform management and identity.

In non-virtualized environments, a single operating system is in direct control of the hardware. In a virtualized environment a Virtual Machine Monitor manages one or more guest operating systems that are in “virtual” control of the hardware, each independent of the other.

A hypervisor is a special implementation of a Virtual Machine Monitor. It is software that provides a level of abstraction between a system’s hardware and one or more operating systems running on the platform.

Virtualization optimizations enable better performance by taking advantage of “knowing” when an OS is a host running on HW or a guest running on a virtual machine.

Paravirtualization , as it applies to Xen and Linux, is an open API between a hypervisor and Linux and a set of optimizations that together, in keeping with the open source philosophy, encourage development of open-source hypervisors and device drivers.

Enlightenment is an API and a set of optimizations designed specifically to enhance the performance of Windows Server in a Windows virtualized environment.

Hardware manfuacturers are interested in virtualization as well. Intel and AMD have independently developed virtualization extensions to the x86 architecture. They are not directly compatible with each other, but serve largely the same functions. Either will allow a hypervisor to run an unmodified guest operating system without incurring significant performance penalties.

Intel's virtualization extension for 32-bit and 64-bit x86 architecture is named IVT (short for Intel Virtualization Technology). The 32-bit or IA-32 IVT extensions are referred to as VT-x. Intel has also published specifications for IVT for the IA-64 (Itanium) processors which are referred to as VT-i; .

AMD's virtualization extensions to the 64-bit x86 architecture is named AMD Virtualization, abbreviated AMD-V.

There are three Virtual Machine Monitor models.

A type 2 Virtual Machine Monitor runs within a host operating system. It operates at a level above the host OS and all guest environments operate at a level above that. Examples of these guest environments include the Java Virtual Machine and Microsoft’s Common Language Runtime, which runs as part of the .NET environment and is a “managed execution environment” that allows object-oriented classes to be shared among applications.

The hybrid model, shown in the middle of the diagram has been used to implement Virtual PC, Virtual Server and VMWare GSX. These rely on a host operating system that shares control of the hardware with the virtual machine monitor.

A type 1 Virtual Machine Monitor employs a hypervisor to control the hardware with all operating systems run at a level above it. Windows Server virtualization (WSv) and Xen are examples of type 1 hypervisor implementations.

Development of Xen and the Linux hypervisor API paravirt_ops began prior to release of Intel and AMD’s virtualized hardware and were designed, in part, to solve the problems inherent in running a virtualized environment on non-virtualization-assisted hardware. They continue to support both virtualization-assisted and non-virtualization-assisted hardware. These approaches are distinct from KVM, or the Kernel-based Virtual Machine, supports only virtualization-assisted hardware; this approach uses the Linux kernel as the hypervisor and QEMU to set up virtual environments for Linux guest OS partitions.

In keeping with the open source community’s philosophy of encouraging development of open source code, the paravirt_ops API is designed to support open-source hypervisors. Earlier this year VMware’s VMI was added to the kernel as was Xen. Paravirt_ops is in effect a function table that enables different hypervisors – Xen, VMware, WSv – to provide implementation of a standard hypercall interface, including a default set of functions that write to the hardware normally.

Windows Server 2008 enlightenments have been designed to allow WS 2008 to run in either a virtualized or non-virtualized environment *unmodified*. WS 2008 recognizes when it is running as a guest on top of WSv and dynamically applies the enlightenment optimizations in such instances.

In addition to a hypercall interface and a synthethic device model, memory management and the WS 2008 scheduler are designed with optimizations for when the OS runs as a virtual machine.

The WSv architecture is designed so that a parent partition provides services to the child partitions that run as guests in the virtual environment. From left to right:

Native WSv Components:
* VMBus – Virtual Machine Bus – Serves as a synthetic bus for the system, enabling child partitions to access native drivers.
* VSP – Virtual Service Provider – Serves as an interface between the VMBus and a physical device
* HCL Drivers – “Hardware Compatibility List” Drivers (standard native Windows drivers that have passed WHQL certification)
* VSC – Virtual Service Consumer – Functions as a synthetic device. For example, a filesystem will talk to the VSC controller instead of an IDE controller. This in turn communicates with the VSP to dispatch requests through the native driver.

Interoperability Components:
* Linux VSC – Interoperability component that serves as a synthetic Linux driver. Functions like the VSC in a Windows partition. Developed by XenSource and published under a BSD-style license.
* Hypercall Adapter – Adapts Linux paravirt_ops hypercalls to WSv

Like the WSv architecture, the Xen architecture is designed so that a special partition, in this case Dom 0, provides services to guest partitions that run in a virtual environment.

Native Xen Components:
* paravirt_ops is a Linux-kernel-internal function table that is designed to support hypervisor-specific function calls. The default function pointers from paravirt_ops support running as a host on bare metal. Xen provides its own set of functions that implement paravirtualization.
* Native Drivers – standard set of drivers in the Linux kernel
* Xen/Linux ABI – having a consistent ABI enables long-term compatibility between guest operating systems and the Xen hypervisor

Interoperability Components:
* Xen Virtualized Drivers – Windows synthetic device drivers must be converted to Xen-virtualized drivers. These are developed using the Windows DDK and will be distributed as binary only per the DDK license.
* Xen/Windows ABI – The binary interface that integrates Windows with Xen, enabling Windows hypercalls to be executed through Xen instead of WSv. This will be licensed under the GPL and made available when the WSv top-level functional specification is made public.

The slide says it all… I couldn’t figure out a way to put this one in a graphic. ;)

Virtualization interoperability testing is very challenging. While the architecture may look similar at a high level, the devil is in the details – down at the API and ABI level, the technologies are quite different.

From a personnel standpoint, the expertise required to debug OS kernels is hard to find, let alone software engineers with these skills who are focused on writing test code. Microsoft has established a role known as “Software Design Engineer in Test” or “SDE/T” which describes the combination of skills and attitude required to test large-scale complex software rigorously through automated white-box test development.

The problem of testing Linux and Windows OSes across WSv and Xen requires these kernel-level skills, but on both operating systems. It’s a non-trivial challenge.

Next is the technical issue of the test matrix:
* Two full operating systems to test (Windows Server 2008 and SUSE Linux Enterprise Server 10)
* Single-core, dual-core, and quad-core CPUs
* Single-processor, dual-processor, and quad-processor boards
* Intel-VT and AMD-V chips
* Basic device configuration (NIC, HD, etc.)
To put this in context, we need a minimum of 40 server chassis to test this matrix – for each operating system.

On top of this, the software components that must be tested include:
* Linux VSC
* Windows PV hardware drivers
* Xen/Windows ABI
* Linux/WSv hypercall adapter
Since Windows and Linux are general-purpose operating systems, these components must be tested across a range of workloads which will guarantee consistent, high-performance operation regardless of usage (file serving, web serving, compute-intensive operations, networking, etc.).

Finally – and no less a challenge than the skills and technology aspects – is that of building a shared culture between two very different and mature engineering culture. What is the definition of a “Severity 1” or “Priority 1” designation for a defect? How do these defects compete for the core product engineering teams’ attention? How are defects tracked, escalated, processed, and closed across two different test organizations’ software tools? Most importantly, what is the quality of the professional relationships between engineers and engineering management of the two organizations? These are the critical issues to make the work happen at high quality and with consistency over the long term.

WS-Management is an industry standard protocol managed by the DMTF (Distributed Management Taskforce), whose working group members include HP, IBM, Sun, BEA, CA, Intel, and Microsoft among others. The purpose is to bring a unified cross-platform management backplane to the industry, enabling customers to implement heterogeneous datacenters without having separate management systems for each platform.

All Microsoft server products ship with extensive instrumentation, known as WMI. A great way to see the breadth of this management surface is to download Hyperic (an open source management tool) and attach it to a Windows server – all of the different events and instrumentation will show up in the interface, typically several screen pages long.

It is not surprising that the management tools vendors are collaborating on this work – and it’s essential to have not just hardware, OS, and management providers but application layer vendors like BEA as well – but to me the most important aspect of the work is the open source interoperability.

In the Microsoft-Novell Joint Interoperability Lab, we are testing the Microsoft implementation of WS-Management (WinRM) against the openwsman and wiseman open source stacks. This matters because the availability of proven, interoperable open source implementations will make it relatively easy for all types of providers of both management software and managed endpoints to adopt a technology that works together with existing systems out of the box. Regardless of development or licensing model, commercial and community software will be able to connect and be well-managed in customer environments.

So what does this all mean? We’ll see end-to-end interoperability, where any compliant console can manage any conforming infrastructure – and since the specification and the code are open, the barriers to entry are very low. It’s important that this capability extends to virtualized environments (which is non-trivial) so that customers can get the full potential of the benefits of virtualization – not just reducing servers at the cost of increased management effort.

Sometimes people challenge me with the statement “if you would just build software to the specification, you wouldn’t need to all this interoperability engineering!” This is in fact a mistaken understanding of interoperability engineering. Once you’ve read through a specification – tens to hundreds of pages of technical detail – and written an implementation that matches the specification, then the real work begins. Real-world interoperability is not about matching what’s on paper, but what’s on the wire. This is why it’s essential to have dedicated engineering, comprehensive automated testing, and multiple products and projects working together. A good example of this is the engineering process for Microsoft’s Web Services stack. The specifications (all 36 of them) are open, and licensed under the OSP (Open Specification Promise). In the engineering process, Microsoft tests the Windows Web Services implementation against the IBM and the Apache Axis implementations according to the WS-I Basic profile. A successful pass against all these tests is “ship criteria” for Microsoft, meaning we won’t ship our implementation unless it passes.

In the messy world of systems management, where multiple generations of technologies at a wide range of ontological levels (devices, motherboards, networking gear, operating systems, databases, middleware, applications, event aggregators, and so on) testing is complex. Adding virtualization into this mix adds another layer of complexity, necessitating methodical and disciplined testing.

Open ID is a distributed single sign-on system, primarily for websites. It’s supported by a range of technology providers including AOL, LiveJournal, and Microsoft.

WS-Federation is the identity federation web services standard which allows different identity providers to work together to exchange or negotiate information about user identity. It is layered on top of other Web Services specifications including WS-Trust, WS-Security, and WS-SecurityPolicy – many of which are lacking an open source implementation today.

ADFS is Active Directory Federation Services, a mechanism for identity federation built into Microsoft Active Directory.

Cardspace is an identity metasystem, used to secure user information and unify identity management across any internet site.

Project Higgins is an Eclipse project intended to develop open source implementations of the WS-Federation protocol stack as well as other identity technologies including OpenID and SAML.

Samba is a Linux/Unix implementation of Microsoft’s SMB/CIFS protocols for file sharing and access control information. It is widely deployed in Linux-based appliances and devices, and ships in every popular distribution of Linux as well as with Apple’s OS X.

This work is still in early phases, and you can expect more details here in the future. Mike Milinkovich of Eclipse has been a champion for improving the interoperability of Eclipse and Microsoft technologies, especially Higgins. Separately the Bandit Project has made significant progress in building technologies which support CardSpace. I appreciate the work of these teams and look forward to more progress here.

The slide says it all here. We’re committed to long term development and delivery of customer-grade interoperability solutions for Windows and Linux, and we’ll do it in a transparent manner. Tom Hanrahan, the Director of the Microsoft-Novell Joint Interoperability Lab, brings many years of experience in running projects where the open source community is a primary participant. I and my colleagues at Microsoft are excited to learn from him as he puts his experiences at the OSDL/Linux Foundation and at IBM’s Linux Technology Center into practice guiding the work of the lab.

You can expect regular updates from us on the progress and plans for our technical work, and I expect you to hold me and Tom accountable for this promise.

I hope you found the presentation valuable. I felt it was important to get this material out broadly since it will impact many people and essential to be clear about what we are building together with Novell, XenSource, and the open source community.

Email Digg It! Del.icio.us SlashDot It!
1.Concerned In Virtual Computer? new java.. CodeGear Pushes Java.. » The PC Info said:
PingBack from http://www.thepcinfo.com/2007/08/14/concerned-in-virtual-computer-new-java-codegear-pushes-java.html
posted at 07:19AM 08/14/2007
2.davemc said:
Sam, How does the acquisition of Xensource by Citrix fall into Microsoft's virtualization plans? davemc
posted at 12:34PM 08/15/2007
3.Sam Ramji said:
This is a positive event for Microsoft's virtualization work and our partnerships with both XenSource and Citrix. I've already been in contact with XenSource who have confirmed that this just takes a pair of existing strong technology relationships and adds Citrix's sales and marketing engine to XenSource's products. We'll continue with the joint engineering that we've been working on - establishing interoperability of Linux on Windows, promoting a common virtual hard disk format (VHD) and driving standards for management in this area.
It also demonstrates both how early the virtualization market is - there are many millions of servers still not virtualized, and that the demand for heterogenous virtualization management is very strong. Overall I see this as very positive.

Sam
posted at 08:58PM 08/17/2007
4.ChrisBradley said:
On the Metal - I haven't used MS Virtual Machine much, except at the beginning with 2004. I am impressed that you have released a 2007 version, and I intend to practice with that a bit in the coming days. The primary virtual machine program I used under vista due to Memory Constraints and other virtualization issues was called Virtual Box by Innotek. I also wrote an interview called http://pearpcfiles.blogspot.com which outlined the emergence of MAC OS X being able to run on PC's. Originally I ran it under Linux but later found that innovations enabled the emulator to run it under windows. With improvements in Processor and Memory, it looks like it will be a real possibility to cross all platform boundaries in 1 - 2 years. I hope my thoughts have been helpful. Man, would I love to have a job working with this stuff in realtime... :)
posted at 02:07PM 08/22/2007
5.communist said:
Interop - haha - good one. Your latest spin is far worse than Get the FUD. http://www.microsoft.com/windowsserver/compare/compare_linux.mspx "Interoperability by design" - insulting people's intelligence?
posted at 02:06AM 08/24/2007
6.Porte25: Open Source et Interop @ Microsoft said:
L'un des volets du partenariat passé entre Microsoft et Novell prévoyait la mise en place d'un laboratoire
posted at 09:08AM 10/21/2007
7.Ken Hess said:
I am not sure that interoperability is needed by either camp. Ford and Chevy aren't interoperable. You can't exchange parts between them yet they both take gasoline, oil, and have seats, tires, etc. They both sell well (even though GM posted a huge loss--probably from executive salaries that are too high) and yet they are not interoperable. They don't need to be.
Linux and Windows are different and Vive la Difference!
On another front, please keep Windows XP and just improve on it...dammit, it works. Vista is almost as big of a blunder as Windows Me. http://www.savexp.com.
Microsoft has the attitude that they need to keep changing something that works. It will be a cold day in hell before I "upgrade" to Vista. It constantly hits the hard drive, it is sluggish--even on a 2GB dual core machine. Who the hell released this piece of crap?
I was offered a co-authorship on Vista with a very famous tech author but after installing this rotten POS, I declined the very nice offer.
If you want interoperability, as unneccessary as it is, first make an OS that works correctly,...like XP.
posted at 09:30AM 02/13/2008
8.studious learner said:
I think that interoperability between vista(and xp) and linux is one of the best ideas yet. (In regards to the xp vs. vista matter that ken hess brought up, I believe that vista is by far a better OS than xp. though there are a few things that annoy me about vista(like slow startup times, useless things like user acount control, though that can be removed easily). I`ve used xp nearly my whole life, and visra is like a breath of fresh air. I`ve heard so many stories about vista bugs, that I`m starting to wonder if I happened to get one of the few ``stable`` copies of it.
Anyway, back to the interoperability issue. I believe that that this cooperation between OS supergiants Windows and Linux will have a huge impact on the computer world as a whole. If they can pull this off(and there`s no doubt that they will...eventually) than this world will be a better place.
posted at 02:29AM 03/15/2008
9.ChrisBradley said:
I think a lot of people are missing the point. Sun's MD can run 5000 virtualizations per blade from what I have heard. That means there is big potential for this sort of technology. Talk of Vista being a blunder is not necessarily true when you are looking at things from a Data Center Scale.
posted at 05:42AM 05/16/2008
10.Dan Foxwell said:
So with my home network I have Vista 64 Ultima and Mac OSX.4.11.
What do you recomemd? My Goal is to be able to use all of my Windows programs on the Mac; at work I use a thin client with the Mac to run windows.
Dan
posted at 11:25AM 01/24/2009 Sphere: Related Content