Archive for the ‘essay’ Category
The Stories Teams Tell Themselves
Leave a CommentTeams carry their work with a portent they assume outsiders cannot understand. An outsider in this instance can be a direct manager, a new team member, or, most likely, other teams within the organization. This is a common theme – but let’s start with examining it from within a development team first.
The development team may feel rushed to produce because the process for project prioritization is loose. Once this happens a development team can begin to form dissonant beliefs – for instance, that no one understands their plight.
Since development work is highly specialized (i.e. there isn’t a career path from help desk to software developer) development team members can have trouble expressing their frustrations to others outside the team. And when these producers of work become frustrated their level of dissonance can increase exponentially until they see it as them against everyone else. Or, really, the beginning of the end.
At the same time another team, like marketing/advertising, can tell themselves that the campaign they are working on is all-important. Maybe, in their minds, it is an immediate trump card to developer objections to time or effort. Many non-technical teams see software as being construction only. They fail to recognize that software, though analytical, is still just a communication game – like the rest of business. But, if the advertising team truly believes this, how can anyone argue a counter position without causing the other to be wrong.
Both teams spend too much time looking out and laying blame rather than looking in. At the same time, both teams feel that what they do is the most important – but how can the teams be the antithesis of one another? Wouldn’t that just lead to disaster? Yes.
They both miss business goals by focusing only on each other and not what they can improve. Is a new marketing campaign a success if they just sacrificed or compromised their architecture, personnel etc. Will that ROI get measured? On the other hand, is the platform upgrade a success if they missed two marketing goals in a row?
Sometimes, A occurs. And sometimes B occurs. Trying to figure out why is a losing game because narrating past experiences only takes into account a single viewpoint. It’s a fallacy. So, don’t waste time trying to figure out why you can’t execute. Focus on executing.
Martin Fowler is a Regular Guy
I didn’t think that before I met him, as brief of an encounter as it was. After all, he’s the author of Refactoring, Patterns of Enterprise Application Architecture and UML Distilled. He’s helped set the course of XP/Agile and regularly refers to my personal software hero, Kent Beck, as just “Kent”. Martin Fowler is respected, experienced and his name is dropped in regular software parlance to justify, or condemn, a given practice.
But – he’s still a regular guy. A software engineer. A developer. A coder. Here is why.
I attended a “Friends and Family” ThoughtWorks event at their headquarters in Chicago. The ThoughtWorks office is on the 25th floor of the ever imposing Aon Center building, which as some know, is a twin to the now gone World Trade Center buildings in New York City.
Their office space is designed to do two things: 1) induce violent jealousy of cube workers, and 2) show that they are hip coders and like Ruby. Since ThoughtWorks is a consultancy no one is ever there though. And if they are, you wonder what is wrong with them. However, they get to wear Puma’s, hoodies and generally expensive (a.k.a. designer) jeans. I wear “slacks”.
I attended a TW event before, so I had a general idea of the flow. For instance, I knew they would have some pizza so I didn’t cram my lunch in before I left. I also knew there would be mingling which I was ready for this time. I expected to schmooze with some developers at the CME or some random interactive shop. Instead, there he was – Martin Fowler – just walking along. So I engaged him and it went something like this:
Me: “I love your books. I live by Refactoring.” (I’m a fanboy.)
Fowler: “Thank you.” (What else could he say?)
Me: “What are you thinking about nowadays? Like today? What’s on your mind?”
Fowler: “Well, really, I have a problem with my website and I need to fix it. But I have to be here as well. It’s small, but it’s bothering me.”
Me: “Your website?”
Fowler: “Yes, there is a style issue.”
Me: “On your site?”
Fowler: “It’s minor, probably just CSS really.”
Me: “But annoying?”
Fowler: “Frustrating, yes.”
Me: “So, let me get this right, Martin Fowler is concerned about CSS on his website. So, in that way you’re like all the other software engineers here leaving some problem behind at their desk to be here…”
Fowler: “Very much, except, I don’t do any real work besides writing and talking.”
And so it came and went. I didn’t occupy him much longer with additional inane questions and politely excused myself so he could do his expected mingling with the crowd. After all, even though there was a panel, most people were probably there for him. Like me. I paid 8 bucks to take a cab across the Loop to see him and I hoped that shaking his hand would impart some type of wisdom via osmosis to my software skills.
So, he’s just a regular guy. I’m probably not that different in circumstances than him when he really started rolling in the 90’s. He has great ideas and the ability to make complex problems simple. But, at the end of the day, little things like CSS still weigh on him; a web page not displaying correctly or a function not being optimum. Stuff that software engineers are concerned about. Like me and like him.
Warning! Career Cancer!
I too got mixed up in the comments on StackOverflow that Joel and Jeff made about Robert Martin’s S.O.L.I.D. principles – to the point I was reading Coding Horror threads. (I leave out links and backstory to Joel and Jeff to spare you the wasted hours of reading.)
While on CodingHorror I was reminded of the vulgarity that anonymity creates and the attitude of so many people in our industry. They just don’t give a crap. They don’t care that they don’t know how to code, that there is a difference between awful design and good design. That objects actually help refactoring and they aren’t blob classes. They couldn’t care less. Here is an example:
From codinghorror comments
http://www.codinghorror.com/blog/archives/000856.html
There’s one thing you can do trying to keep up with progress or code like you always did. If you started in maybe Fortran, Basic, VB, ASP or maybe PHP – you continue to script your code in good old plain procedural programming. Too much OO slows you down…
New shiny thing,,, i don’t give a damn – I code it like I used to, get the machine obey my will, not the other way around…
And you don’t get more paid if you code spaghetti or nice code when your boss asks for results, time is money!!!
Ideals are good in a perfect world – I could say the search for programming utopia or something…
Would anyone want this guy on their team? He purports that he speaks for management…that they don’t care…well, they just don’t know he is sinking their software deeper and deeper into the mud.
So, I feel it’s my responsibility to warn you in Jeff Atwood fashion of this pariah. BEWARE!
They will leach onto your project, or you, and pull you down. They will undermine your work, the progress you make or anything you do in the name of progress, or heaven forbid, process. And it will all be stated in a backwards apologetic standard: “one design isn’t better than the other” or “200 line methods are fine, it can’t be done any other way”. And, one of my personal favorites “no time for TDD. No time, no time”.
It’s all stated in a pedestrian manner until you begin to peal away at the onion. “Hey, can’t we all get along. There is no black or white – just gray”. Unfortunately, as in real life, there is black and there is white. Billy is white and Sam is black. Deal with it. In software these folks will seriously suck the ever loving life out of you and then move on with their indecision, ignorance of the industry and belief that all things are equal.
Beware.
On Positive Affect and Software
A brief tale of software developer burnout:
Brandon sits down at his two year old HP workstation and scribbles his mouse across the plastic-laminate desktop. The twin monitors snap to attention and he logs in without thinking about it; finger memory taking over long ago. Visual Studio is already open, it’s been for the last month, because the project continues onward like warships of a 1984 Orwellian nightmare.
He’s worked at the same project for the last 11 months and management tells him the end is in sight. His inbox describes an alternate view:
Defect #234 -New- Wrong font color in IE 6
Defect #188 – Reopened- Pricing is incorrect
Brandon’s usually up beat, but he has seen nothing but build reports and defects for months. His stake-holders haven’t even been reviewing key features – they’re waiting for the magic release where all feature are included and everything works according to a document that some managers have signed.
Starting QA after 10 moths didn’t feel like a success. Nor does closing 30 defects in two weeks. And now, as the winter starts, he can only look forward to shipping the 50,000 line project on a cold weekend. It will feel good. Right? To be done with it. Right? Wash his hands. Move on. To the next…project…
This isn’t an unusual case. Even with supposed agile practices catching on, organizations like to hold scrums for the development team but the stake-holders choose to wait until it’s complete. Even with the idea of software being complete an oddity and undefinable end point. Yet, they’ll wait.
This outlook on software development has caused the most pain for the Brandon’s of the industry because there is a lack of incremental satisfaction.
Nassim Nicholas Taleb sites the idea of hedonic happiness in his book, the Black Swan, to examine the idea of hope in light of big singular rewards. Meaning, many small rewards have a much greater effect on our overall happiness then one or two big ones very far apart. This is called positive affect. Taleb’s example is financially driven, but excellent. Simply, a person will be happier making $100,000 a year as opposed to making $1,000,000 every ten years. The amount of tension and angst for those ten long years can not be recouped with the big payout – especially if it might be another ten years until it happens again. As humans, we’re simply not wired up like that.
This fits perfectly into the not so modern realm of software development that 90% of organizations participate in. Remember Brandon waited 11 months before a release. He waited 10 months until someone would even begin verifying his work. Therefore, there were 10 months of effort without any reward. If positive affect is indeed a true nature of humanity, then long development cycles are like a 30 story ledge that developers stand on crying week to week while looking at the cars below.
The Orwellian similitude’s carry on. The ruling “Party” of 1984 used doublespeek as a way to control the masses; simultaneously accepting as true two mutually contradictory beliefs. I’ve see this in places I’ve worked – and I’m sure I’m not alone. For example, the requirement is to be agile and to release all features at once. To do incremental work, but not test until all increments are complete. To look at iteration plans but not really care unless a feature might not make it at release day.
Are these contradictory? Indeed. Are they agile. Of course not. We don’t have to wait for a solution. It’s already been invented and bigger organizations can take advantage of it – if they will invest in a mindset change that iterations of the product and each one should be complete. It’s going to take more training and experience for a stakeholder to really understand that an iteration is planned to be a deployable piece of work and not the subgrouping of tasks on a Gantt chart.
I won’t belabor the incremental, agile, continuous integration points. The amount of literature and case studies is staggering. However, I’ll continue to make a case for the Brandon’s of the world. Give Brandon some wins. Schedule QA of the first iteration (even two iterations) of the project so that it’s in a deployable state. Then, and this is acrimonious to some, release it. Actually, physically release it. Then start the next set of features. Help our Brandon’s feel the little frequent successes that nature makes us crave and then watch the productivity of a motivated software team soar. Use positive affect to the advantage.
George Orwell’s 1984:
“To tell deliberate lies while genuinely believing in them, to forget any fact that has become inconvenient, and then, when it becomes necessary again, to draw it back from oblivion for just so long as it is needed, to deny the existence of objective reality and all the while to take account of the reality which one denies”
Wrong About TDD
Admitting you’re wrong is anything but easy. It’s a shot in the arm of the pride that we all hold dear – in our personal or professional lives – and often, for better or worse, in both. Five years of marriage taught me to admit fault or error when it’s valid. And lately, this humility, has spilled into my career as a software developer.
One can be wrong in myriad of ways. There is fault in attitude, fault in actions, and fault in opinion – with Test Driven Development I was wrong on all three. Gentle reader, following are the ways I wronged TDD and myself:
Ways I was wrong:
1. The way I write software is fine.
2. I write complicated software – this won’t work here. Object composition and encapsulation precludes TDD’s success.
3. This will slow me down.
4. It’s a fad.
Let’s dissect these. My attitude was prideful (1); my abilities do not need improvement I told myself. This is the sweet trap of stagnation we knowledge workers lay for ourselves. Spending so much time creating models in our heads and expressing ideas in a language for machines ripens the opportunity to see our reflection as Turing, Beck or Fowler.
The applications I build are anything but trivial (2). Is this a true statement? Sure. Not yes – aren’t most applications complicated? And how would one consistently define one system as being more complex than another? I could do it via lines of codes or cyclometric complexity – but that isn’t really what I told myself. In fact, I belittled TDD as too small for the job because of my boastfulness stemming from fault number one (1).
Asking the question “will this slow me down?” is a trivial scapegoat for avoiding doing any task better (3). This one is endemic of human nature because it’s difficult to argue the counterpoint – of course doing something new will be slower at first. So does drying the dishes before putting them away, but the average domesticate prefers their cabinets not moonlighting as mold spore factories, so they will wipe that Crate and Barrel flower pattern down.
A couple months before I noted to write an essay on the benefits of TDD I planned a counter thesis – TDD is a fad (and it sucks)(4). Yes, gentle reader, as egregious as it sounds I was quite comfortable in my anti-TDD stand; after all I was deluding myself with points one, two and three. In fact, I reacted from cognitive dissonance; if I don’t write software well then TDD can not possibly be of benefit. Stated inversely: if TDD works there is something wrong with the way I write software. That kind of conclusion is demoralizing and my brain rejected it before I could even fully consider the implications. I rejected TDD to keep my ego in place for the job I spend my days contributing to. This of course, led to confirmation bias (Nicholas Taleb terms this platonic confirmation in his book the Black Swan, http://www.fooledbyrandomness.com/) as I sought out evidence – again, the first three points – on how I was right and the TDD crowd was wrong.
If you can’t teach an old dog new tricks, like your author, it’s like pulling water from a rock to help a developer change a habit. Now, tell them to not write cod and then test, but to write tests that write the code. My struggle persisted. So, I looked at the advantages – which are numerous.
Advantages
Advantage #1 – Consumer Friendly
Aesthetics are more a part of coding than the secretly vain development community would like to admit. What is the class named? Does it take the correct parameters? Is it a subclass of something else? All these things are figured out iteratively because the test is the code. In order to write the test “GetAccountNumber” you write what you would like to see in your code.
public void Test_GetAccountNumber()
{
IAccount account = new Account(44444); //this class does not exist yet!
int a = account.GetAccountNumber(); //this method does not exist yet!
}
This idiom of works produces more readable code because it’s already been thought out and used before it ever hits the application.This is synonymous to building UI first for a UI app.
Start with the consumer.
Advantage #2 – Interface Based Development
Entire books are written on interface based development. Composition is preferred to inheritance and interfaces are the way to compose your objects, therefore interfaced based development is preferred. Writing your tests first give you the opportunity early on to program to an interface, saving much grief later on.
Advantage #3 – Prove it works the first time
As humans we look for ways to avoid pain and injury. Which hurts less: a) writing a method called GetAccountNumber and testing it to fill a drop down on a web page and watching the drop down render as empty, or b) run the unit test for the method and watch it pass or fail. It is infinitely easier to debug the unit test. The only parameters for the method GetAccountNumber in the unit test are what actually gets passed to the method; like the primitive number 413. The parameters for testing the method in the web page are the entire context of the web page – possibly the entire site – as any number of run time errors can occur while manipulating other elements.
With TDD, the chances of success the first time a method or class is used in an application greatly increase and application debugging goes down. They are inversely proportional.
Advantage #4 – Proof it works after changes.
No matter if the program is a flight simulator, a line of business app or a huge e-commerce platform, all programers struggle with using their classes after they’re written. Even with some up front class modeling, the first time a class is used is almost always a disastrous event.
For instance, let’s say the class built is called Mortgage. It looks like this:
class Mortgage
{
public Mortgage(datetime startYear, dateTime endYear, double loan)
{
….
public datetime GetStartYear()
{
…
….
This is all simple enough. The Mortgage class is plugged into an existing web app and the methods that it exposes are used by this application. Again, this is the normal course of events for software development the world over for the past, say, 30 years.
Just a week later a new application feature is needed to calculate the amount of interest paid for the length of the loan – which requires knowing the interest rate. Notice though, interest rate was not past in though the constructor. There are now two options – 1) overload the constructor with a new version that accepts the interest rate or 2) expose a public property to set the interest rate.
Option 2 does not sound very elegant and can lead to object state issues as the object is not created atomically with all the data it needs to perform it’s behaviors. So, the new GetInterest method could be called and throw an exception because it does not have the data it needs. (I’ve done this…it was a really, really bad idea.)
Option 1 sounds like a golden parachute over the later option. So, the constructor is overloaded with a new version:
public Mortgage(datetime startYear, dateTime endYear, double loan, double interest)
{
…
There are now two constructors for this object, and they need to stay this way for the time being because the Mortgage class is already used in 5 different places in the application. In fact, Joe Developer is using the Mortgage class with the first constructor and adding new methods to the class. Looking at that fact, it’s thought best not to change that first constructor because who knows what may break, or worse: misbehave at runtime.
The new Interest feature is now coded and approved by whomever approves these kinds of things and everyone is happy – until iteration 3. A routine that Joe Developer coded needs to slide that Interest number into a grid. Joey D thinks “no problem” and calls the GetInterest method: bang, dead. It blew up. There isn’t an interest value in the object because he constructed his object with the original constructor.
This could have been avoided by simply eliminating that first constructor which was leaving the object in an invalid state when the GetInterest method was added. But that was scary. Since there weren’t any tests on the methods of the Mortgage class it was deemed to risky to change the places where the first constructor was used and now every developer from this point on will run into the same problem. Of course they can now decide to alter all the instances of the first constructor in the application – but the damage was already done; defects.
Advantage #5 – Make the Brownfield easier to navigate.
As a professional we usually work in environments we did not bootstrap ourselves. The environments are preexisting and have characteristics of their own. Often, the code isn’t even legacy – it’s just there. Any environment like this is referred to as a Brownfield and means integration work. Integration is difficult because analyzing what may break with new or upgraded features can’t be seen until functional testing. Writing tests first in a Brownfield will make your life easier – and the guy who comes next.
Advantage #6 – Be a professional – take responsibility for your product
Simple economics dictate the likelihood of having a tester for every coder is the same as successfully writing software using a pencil. Writing tests that prove code works before testers touch it ups the quality dramatically. It’s a way to take responsibility for the product being created.
Advantage #7 – Have fun. Red, Green, Refactor.
Remember when writing software was fun. When you referred to it as “coding” or “hacking out some code”? Replace your TPS reports with TDD and bring a little fun back into the development cycle.
Doing Test Driven Development means that one small piece of functionality is being written and tested at a time. It’s written. It breaks. The code that the test is trying to run is written. It breaks again: Red. Tweak. It passes: Green. Now that it passes go and make it better. Improving the code is no longer a fantasy. It’s easily doable because it’s all covered by tests. Refactor those primitives into an object or replace conditionals with polymorphism. Or – this is fun- replace some meddlesome constructors with factory methods. It’s not a dream with TDD – changes can be made as long as the test go Green.
Advantage #8 - Sleep better at night
Turn off the monitor(s), grab the keys and go home assured that things are working. Pretend that it’s Monday and you cranked out the following code; four new classes and ten new methods. That’s a good amount of new code introduced into an application in one day. The build succeeded, good, but you’re still not sure that when your teammates get latest things will functionally work. Now, pretend you had tests over all that new code and they all were Green when you left. No worries. Better sleep.
Advantage #9 - End up with better software.
This is really what it’s all about. Test Driven Development makes your code refactorable, which means it’s easier to change, which means it’s easer to optimize, which means it runs better which is correlated to less defects, less QA and easier releases. And all that adds up to better software.
The Myth of Enterprise Development
The term enterprise is consistently shot across conference tables inside corporate IT and technology marketing departments. The term is most often ascribed to software and hardware systems running the core business; a reservation system for a hotel franchise, or a software package for payroll. However, this attribution excludes all the software that runs at smaller companies. So, what makes a system enterprise ready and which products are left looking in at the glowing warmth of Fortune 1000 server rooms while they desiccate.
Sadly, there isn’t much empirical evidence separating an enterprise ready app from any other piece of software. A development manager requiring an enterprise developer doesn’t make the role anymore real than an engineer claiming to be a “people person”. An HP blade server is no more enterprise ready being racked in a enormous corporate data center than in a startup’s cage at RackSpace. Let’s dissect this hypothesis by looking at two very popular web frameworks: ASP.NET and Ruby on Rails.
Microsoft sits in the interesting position of selling the same software that runs small businesses and large enterprises. This of itself should be enough to prove the term enterprise a loaded marking term – but let’s keep going: ASP.NET is Microsoft’s enterprise web platform. It, coupled with IIS, provide everything needed to build dynamic multitiered web applications. It has optimizations for Microsoft SQL Server, session and cache providers, error reporting, performance tuning, remote configuration, etc. It can do just about anything a web application in 2008 would require. Again, Microsoft, pundits and analysts say it is enterprise capable.
On the other extreme is Ruby on Rails. Note, not just Ruby the programming language, but the Rails web framework as well. What I’m embarking on is dangerous: I am not a Rails expert and provide this caveat up front. Rails is considered by popular opinion to not be enterprise ready. What does it lack? As far as I can tell, the only thing it really lacks is a support package, IDE and hordes of corporate programmers.
On it’s own, ROR is terribly productive. The Active Record design pattern is built right in as a OR Mapper with the friendly and obvious name of “Active Record”. So, it provides a concise model of objects over data right out of the box (if anything had a box these days). The framework is based off the of tried and true MVC, model-view-controller, design pattern to provide distinction between data, middle tier and presentation. This is a simple programming idiom used with GUI development, web applications or the myriad of systems that actually make’s power arrive to substations to power my MacBook (Pro, thank you very much).
I provide these contrasts between the frameworks to illustrate the futility of using the term enterprise for one group of software products but not for another. Because it gets better. Bigotry of platforms like ROR grows as corporate IT software developers feel more threatened. Hence their managers feel threatened and when Forester Research shows up all they hear is how platforms like ROR are not enterprise ready. But it’s not a technology stack that is not ready for development in these corporations. It’s the people.
In the end it’s that simple. When people are not ready they will search for whatever means to prove their disaffection, dissatisfaction or ignorance about a product to be real when it’s empirically false. This is why the enterprise is difficult to define: because it’s a constantly moving target through the emotions and minds of corporate software developers.
On the Failure to Build Tools
There is a misconception in non-product software development around getting things done; where accomplishing a given task only once is acceptable. The developers who fall prey to this most often are the ones who have never seen a unit test and whom “mocking” is akin to Daily Show jokes. I’m neither an expert at unit testing and couldn’t explain how to use a library like RhinoMocks properly if my feet were to the fire…it’s just an observation. However, getting things done for that one release, for that one build, for that one functional test is common in far too many shops. Mine is no different.
The failure to build tools isn’t just around software testing. It pervades other areas of the application life cycle; early research into a project, or, say, data transformation – reproducing a database is a big one. Other failures to build tools are build scripts, configuration repositories, post-build events, cache validators, deploy validators, version checking… and the list goes on. I won’t harp on the build process – I’ve done a lot of that in the past. However, the actual release process – that is another ball of wax, or, more like the soft underbelly of the software life cycle.
All but the simplest of applications require some level of configuration management. So, if you click and drag your way to better software the rest of this essay might bore the pants of you. Or if you write Visual Basic. That’s just a straight jab.
Applications are more than their code since any significant application written for business in the last 20+ years uses to some kind of data source. And that data source, usually, is not the same for the developers or testers as it is for the end user. Hence, this variable data needs to be stored somewhere. Simple enough then. Put it on disk. Somewhere. Modify it….at some point? The question is when and how – which leads to the the primary paradox of building or not building tools I’ll address.
As software developers, we are often faced with the situation to write a script to parse 20 odd lines of some odd piece of data. Additionally we often just need the data once, loaded up, or put into a CSV file. The angel on our right shoulder tells us to write a Python script to automate it so we can do it again tomorrow if necessary. The little red guy with horns on our left shoulder tells us to do it by hand; just crank it out, as we’ll be done in 30 minutes and can go home and play World of Warcraft or watch American Idol. Mr. Red doesn’t remind us that the last thousand times we cranked it out by hand it didn’t work out and we had to do it by hand a dozen more times. Hence, with limited cognitive abilities at our disposal, as we just write software not make cheese, we go with the choice of leisure: do it by hand.
If professional software developers, being us, often make the choice not to build the simplest of scripts for tooling, what do you think occurs with the folks in IT/QA/Systems? If you’re first answer is they find a big expensive software package that will do this for twice the hassle of everyone involved, you are right! However, as this is universally accepted and widely documented let us examine the second most obvious choice. Yes, they will hack out whatever it is that needs doing by hand.
Spending time with someone in IT working in a Windows environment is like must see TV. I consider it a cross of Lost and reruns of Friends. There is constant, really constant, clicking around. Click this OPEN, click that CLOSE! Drag files. Stop. Reset service. Click, click, click, click – ad infinity. If you are fortunate to watch these activities in the wild, like Lost, you’ll wonder what the heck is going on, and like the Friends reruns, feel like so much time is being wasted.
A software release process can get rather complicated; especially on the web. On the web, inevitably you’ll end up deploying, at the minimum, hundreds of files. In lots of cases it’s actually thousands of files. This is simply the nature of the web. Along with these application files, and their resources (html, images, video, pdfs) are their configuration files. And the configuration files required for the production release is not the same as the configuration files in your QA environment – still sound so simple?
Ok, probably in theory. But in practice it’s different, and when someone walks through the process and arrives at this point they understand the complexity, timing, and work required to pull off a release. But, sadly, the reason they see this is because there isn’t a tool built to abstract this away. A nice GUI tool for IT to use. And, when that doesn’t happen, it’s time for software developers to step up and take over. Because the failure to build tools isn’t with IT – it’s with us.
The Web Is Not Real; Web Applications Are Not Web Sites
I’ve decided that treating the web like it’s made up of HTTP requests, headers, images and markup is a bad idea. That the web, in all it’s HTML and javascript glory should not be treated as such. Instead, it should be treated as a big, fat, hairy delivery mechanism. Something that needs taming and requires a high level of abstraction to build anything useful on it. It’s like writing Win32 code to run on an old Commodore (which I’m sure someone has). Your HTML will never be in a native environment.
I’ve worked on the web for a long time. Not as long as some, but a good amount of time considering that we’re only about 13 years in. I started in HTML. Then DHTML and Javascript. Then VB Script with ASP. Then finally, ASP.NET, and where I’m currently treading water, with ASP.NET + CSS. Each of these progressions brought more sophistication to the “we’re all connected” yet annoyingly disconnected web. Static moves towards scripting, then on towards compilation then towards web UI control sets and finally CSS based layouts thrown on top. And with that, I assert the following statements to be false: over the years programming for the web has really become simpler. Our standard libraries make it easy to make applications run identically across the browser “environment” and across platforms. The web experience performs akin to a cougar chasing it’s lunch. We’re caching all our images on the client with a push of a button and decreased our time-to-last-byte counters on the server. I can’t see how any of these statement can be claimed for someone, or company, with out spoonful hubris. The web is the same promise of Java….write once, run anywhere. Yet, like Java, it can run anywhere broken, slow or unusable.
How many man hours have been spent across the world of web development trying to get some HTML to render properly in Internet Explorer? What could we have done with those computing cycles? What could they have done with their time? And it’s not just IE – Firefox is just as much a culprit, because as some have pointed out, IE became something of a standard by sheer market share – so the community programmed to it.
Now, let’s move beyond the simple web page example. Everyone knows that tweaking is a fact of life to get things working in a browser. The thing is, a whole cottage industry of consultants, designers and websites thrives off of it: A List Apart, CSS Beauty, etc. They boast the beauty and elegance of shiny CSS and the taming of the shrew; a.ka. HTML. Now you can move your presentation out of your presentation. What? To me, that’s appears to be the intent. Sure, I like using CSS to style, position and provide reuse, but it’s just moving the bits over a little. Skinning? Who does that?
But it’s not that simple. I got caught up in the sordidness. In the slickness. In the elegance. In the load of horse maneur I was being fed by reading web design sites and spending time with designers whose personal sites looked so cool and made me long to be just-like-them (wow, your style sheet is so organized!). I stood up for the purity of our HTML and for simple, open javascript libraries. I crowed about div’s and easy to read CSS files. I became the one cleaning the stables; shoveling the maneur.
And it’s not easy either. Consider the classic N-tier model: data layer, data access layer, business logic layer and presentation layer. This has been the thought pattern across most business applications for the last decade. Can we separate all these layers cleanly? At the end of the day can these layers really not know anything about each other? Nothing? MVC, as old (sorry, classic) a pattern as it is, won’t help when a business rule you never saw coming shows up. Or, some performance tweak must happen at the data access layer that is really implementing a tiny bit a business rule somewhere. How has our luck been with this? Is this even OOP? I don’t know what it is, exactly, but it’s not impressive.
Now, consider the presentation layer: your application needs to look a little more blue for some customers than others. That’s the requirement. So, where is this going to occur. You could apply a whole different style sheet with a few elements with a higher blue saturation. That would do it. But then you have duplication. You could heavily modularize your CSS files and make judicious use of “@import blue.css”. But now you don’t so much have an applicable style sheet(s) as you have style modules that are pieced together on the fly by the application or composed into more style sheets by a designer or developer. That doesn’t sound to bad to me – but I’m a software developer. But how would that sound to the UI designer/developer? Or, you application can replace the style for element A with the more blue style. My point, if you’re not catching it already, is that there isn’t and can’t be a clean separation of layers. That’s just not how the world works. Even in the food chain, an ant eater may eat ants, but those ants may have been surviving off ant eater excrement. There is a connection – maybe not a pure symbiosis, but it exists. You’re stored procedures are connected to your business logic in one way or another and your CSS is in cahoots with your HTML and (gasp!) your script/compiled code. That’s life.
On the flip side are the folks at all the web design shops whose biggest output is “sweet” CSS with mirrored background images that they show off to other designers and brag how it works in Safari. These are the marketing, branding and product sites. They aren’t concerned with reuse. They aren’t concerned about skinning. To them, CSS is a very flexible way for them to think and build web pages. It let’s them focus on making their site look as slick, as gentle, as in your face or as required by their client as possible. That is the non application side of using CSS. And it’s very different than what I, or the folks I associate with deal with.
Now, if we accept the hypothesis that clean separation of concerns is not attainable, if even just for arguments sake, then what is the course of action? That’s simple: don’t worry about it. This is probably enough to make the most staid of personalities somewhat indignant. But yes, the simplest answer is to not worry about it, or better stated, don’t worry about the details. Build your applications at such a level that your developers can worry about concepts not rendering. So that they don’t constantly have to make decisions on whether to use a span, a div, or a list (in horizontal mode none-the-less). Build libraries that render exactly the HTML you want and not what the web design community says you should have. It’s a different application of CSS or HTML or the web in general really. One of the reasons you shouldn’t have to worry about every detail of HTML and CSS as you create a new page in a web application is because the page doesn’t exist anyway.
I’m not saying there “is no spoon” (i.e. the Matrix), in that you can wish the page away. Instead, look back towards the origins of the web and our good friend Tim Berners-Lee. The original intent was for document navigation. Link from document A to document B. Not a generated on the fly document B, but a real document whose bits are on disk somewhere. Where if you were sitting at the console of said server, you could type “vi documentB” and be presented with the same thing that someone using browser would receive.
So, the page doesn’t exist. A representation of a page, generated in memory and streamed over HTTP via TCP/IP exists and then disappears. In that payload are a bunch of bits. These bits are then pulled together, parsed and rendered. A page isn’t sent to a browser – but the notion of one is. In the same way, you should write code and develop web applications towards the notion of a page. Once the page is considered an abstract concept it can be created as an abstract type with a modicum of work. And thus begins the business of abstracting the web away from the web application.
If you call a method on a business object to retrieve a list of widgets must you absolutely know that it’s talking to MySQL on the back end of the transaction? No. Not if it’s built even close to correct. Again, so why must we pay such attention if we’re receiving a POST or a GET. Heck, it seems a wasted effort to handle both (I know there are camps that don’t use POST) since it’s your software that dictates the communication request to your server. Your application receives a request. Plain and simple. Another example is web platforms that have a one to one mapping of a file to a URL. That just doesn’t make sense since that file (page) isn’t going to be sent back in the state it is on disk anyway.
Continue moving down the stack and we’re back at CSS. Ebay figured out a way to handle their CSS and images. Through some really slick Eclipse customization and use of software factories they compile all CSS and images with their source code; they’ve integrated instead of separating. It’s more complicated than I make it sound, but the idea is that the reuse of these assets is identifiable through their source control and build procedures. It’s not a secondary asset that is applied on top of all the engineering. It’s been sucked into the process. At the end of the day, the browser receives a bunch of CSS, but maybe some of it’s on the page and some of it’s in external files. It doesn’t matter that much. It’s more aesthetic than anything that all CSS should be tightly contained and able to be opened at any point in time in Dreamweaver. It’s trying to be separate when really you want it contained. You don’t want it to really exist. Just like the page, the notion of CSS files is great. And the rendering and positioning it provides is very helpful (plus it renders faster) but you don’t need to be explicit and draw lines in the sand of your application to make it so. A web application is software that delivers the notion of a document. The document should work extremely well in the requested browser. But the programming and design of a web application should be at a higher level of abstraction. The web, browsers and buzzword technology will outpace your development and features. This provides a way to either match that pace or ignore it.
From forward to Structure and Interpretation of Computer Programs; Abelso, Sussman and Sussman
“After a short time we forget about syntactic details of the language (because there are none) and get on with the real issues — figuring out what we want to compute, how we will decompose problems into manageable parts, and how we will work on the parts.”
We Could Have Built It Already
Somewhere in an IT department right now a team of people are choosing a software vendor. This vendor could be for just about anything in the corporate IT sphere. You can imagine the usual roundup: analysis packages, data providers, software components, UI controls – the list goes on and on nowadays. The people in the room are bright hard workers. And the problem they are solving is varied. Like how can the project succeed with the minimum work from in house staff or how can it be done without breaking the bank. And finally, wiping the sweat from the brow and mustering up the old college fighting spirit – how to deliver the best possible product.
Does this sound familiar? Bueler? Bueler?
These goals are altruistic and, some would say, commendable, but they aren’t always realistic. Why not? Because choosing a vendor is too often the default option when business is faced with a problem these days. And when that occurs, the We Could Have Built It Already principle is at work; otherwise known as WCHBIA (pronounced wa-chee-ba). In a nutshell, it’s a simple way to call out the business analysts, non-technical managers, technically challenged managers and the technical, but wet blanket managers, from wasting time on vendor product selection for systems that can be and more importantly should be built in house. Really, this is in hope of preventing Yet Another Integration Project, which we’ll touch on later. Don’t read this as all products should be built in house – they shouldn’t.
You can see WCHBIA all around you if you look close enough. It’s the multitiered, web front end, Ajax enabled, 14 pt font Web 2.0 web analytics package that tells you conversions of a shopping cart. It’s the “if you bought this, you’ll like this” recommendation platforms. It’s the five thousand dollar site licensed UI components that don’t scale on server farms. You’ve seen it or experienced it by now. Heck, if you work in corporate IT you’ve most likely integrated some third party (system, component, service, data feed).
Luckily there are ways to avoid WCHBIA. Let’s look at them.
–The Real Purchase–
Most vendors aren’t just offering product X, they are offering a solution. This solution involves downloads, installs, setup,configuration, documentation, maybe some scripts and a plethora of meetings. And that’s just for the technical staff. Project managers, lawyers, directors (up to CTO often) get involved because at the end of the day this is a capital investment with contracts to read and negotiate. This is a lot of people and consequently and lot of cycles at work across teams. Are you analyzing this time before heading down the vendor selection path? Have you really considered who the “third” in third-party means. Are you willing to actually estimate these hours? After all, multiple vendors will be investigated and probably tested in house which is an immediate time investment for the internal staff.
–Revenue Producing–
Next, is this system critical to your company? Is it used on a revenue producing product? Will it give your platform a leg-up? If you answer yes to these questions (or even two out of three) that is a big red flag to pay very close attention and tread carefully. Vendors go bankrupt and close down. They are sold and resold. They have their own internal priorities (and strife). They are very rarely plug-and-play no matter what Chaz, their outside sales guy, tells you.
At the end of the day, do you trust SimpleSoft LLC’s product to reliably be part of your revenue producing system. Do you trust it to fund payroll? And if this thing is so great, aren’t your prime competitors already using it or going to deploy it as well? Where is the market advantage in that? That’s just keeping an even playing field and not leveraging technology for market advantage. You won’t drown, but you won’t swim to shore either.
–Investing in Your Team–
There are significant advantages to investing in and improving a software platform. Solving the problem at hand is a key advantage. Although the vendor wants to sell a solution what’s usually required is a product – which is just software. So why even test drive a Ferrari when what you need is a Volvo?
Next, outsourcing interesting and challenging products to a vendor is a really good way to homogenize your development staff. It sends a signal that they are good at building houses but not skyscrapers. Sometimes this goes as far as turning them into glorified UI designers, that is, before the real UI designers are brought in. Management should know if their staff is up to building important revenue producing pieces of the platform. Most developers I know drool for the chance. That’s why they come to work. To build cool things and make an impact. It takes the same level of commitment to requirements and limiting scope – but if the focus is on building a functional product that fills the void it’s a win-win.
If you’re thinking that your teams software developers don’t want to build cool products and be challenged then you probably haven’t spent much time talking with them or listening to their conversations. If you know for an absolute fact that they would rather not build high impact features into your platform or deliver differentiating features to clients then you might have an unfortunate favor of “career” programmers on your team. And if that’s the case…there are other rivers your team will need to cross.
–YAIP–
Constantly looking to vendors is demoralizing for developers. If this is so important to spend 80, 100, 200K on why is it not entrusted to the development team to accomplish? Frankly, an investment in people is clearly lacking here and as outlined in the seminal book PeopleWare, this is as good a path as any toward Teamicide. Somewhere along the line managers have grown the thought in their head that software developers prefer integrating vendor software instead of writing software. Poll 100 developers and I’d hazard to guess that less than 10% of them say they’d like to spend their career integrating vendor solutions. This is where the term Yet Another Integration Project comes from; the constant churn of them in corporate IT software departments.
The churn of integration projects is justified by a claim to cost savings. The general thinking is that it would cost an in house team more to *build system X than it would to purchase it. But looking at the number of people involved, testing time, up front purchase cost, integration cost and maintenance cost of the “solution” – it’s tough to say it’s an accurate justification.
–Antipode Acronyms–
Remember when everyone was afraid of another acronym? NIH; Not Invented Here. For a time managers feared systems that were so complex that it couldn’t be maintained and someone would have to put a deugger on the equality operator because someone modified it’s behavior. That would be a terror to any reasonably sane person. But now the industry has swung hard the exact opposite way. Now, the fear should be systems built around vendor solutions and the lack of control. Who controls the roadmap for SuperMega EverywhereCache? SimploSoft LLC does, not you.
–Conclusion–
While looking for a vendor, testing their products and having lawyers read contracts, a decent set of software developers could often have built the technology in the same amount of time. Or maybe a little more time. It’s still software after all and integration projects are just as suspect to go dangerously off course as starting with a compiler. And given that developers (or system designers, architects) are roped into these vendor dealings, the time could simply be allocated towards building a working product. Basically, design and prototype sessions. The kind of things that really talented engineers want to do. In fact, the kind of thing that they were hired to do.
*If you work on a big CRM package and read this thinking “awesome, roll my own”, don’t show your employer this part of the article. Your type of software is what vendors are perfect for; unless your backoffice operations will generate more revenue or market share for your company. And if that’s the case, you are indeed in a very interesting industry.
